Storing real-time measurements in a CSV file

Storing real-time measurements in a CSV file

Although Yoctopuce sensors have a built-in data logger, it is often useful to store the data in a database or CSV file for local processing. Ideally, one would like to fetch the data in real time as much as possible, but reserving the possibility to retrieve the data stored by the module in case of temporary connection loss. And this is not so difficult to do...



An example of an application that uses this technique combining real-time acquisition and offline data retrieval from the module's datalogger is the Yocto-Visualizationtool. If you enable the use of the datalogger in the graph configuration, you will find that it is able to retrieve all intermediate measurement points in case of a network outage.

So today we offer you a small program in Python that illustrates the programming techniques used in Yocto-Visualization to achieve this result. In a few dozen lines, this program can monitor an arbitrary number of sensors and log their measurements in chronological order in a CSV file - one file per sensor -, automatically recovering the missing data in case of loss of connection or temporary program shutdown.

For people in a rush

You can find this program in our programming library for Python, in the directory Examples/Prog-SensorManager. If you want to monitor sensors in real time via CSV files and are really too rushed to do your own adaptation of this code, this can even be a ready-to-use solution. Just run it in a Unix shell or a Windows command window:

pi@raspberrypi ~ $ python3 SensorMonitor.py
Hit Ctrl-C to Stop
Device online : LIGHTMK1-956BE
Load missing data from LIGHTMK1-956BE.lightSensor : 100% done
18:36:23 Device offline : LIGHTMK1-956BE
18:36:28 Device online : LIGHTMK1-956BE
Load missing data from LIGHTMK1-956BE.lightSensor : 100% done
18:37:08 (clock ticking...)


As long as the program is running, you will find the real time data in the LIGHTMK1-956BE.lightSensor.csv directory, as you can check using tail -f for instance.

Remember that in order for this program to work, you must first:

  • Enable timed reports (regular interval measurements) at the frequency of your choice on each measurement function of interest
  • Activate the recording of measurements on these same functions
  • Activate the data logger on the module

These operations can be done using VirtualHub or Yocto-Visualization.

How does it work?

The program is structured in three parts:

  1. initialization of the Yoctopuce library, which connects to USB and/or YoctoHubs
  2. functions that manage the connection/disconnection of Yoctopuce modules
  3. the SensorManager class, of which we will create one instance per sensor to monitor

Initialisation


To initialize the Yoctopuce library in a temporary disconnection tolerant mode, we use the following lines:

if YAPI.PreregisterHub("usb", errmsg) != YAPI.SUCCESS:
    sys.exit("Init error: " + errmsg.value)

# Use Arrival/Removal callbacks to handle hot-plug
YAPI.RegisterDeviceArrivalCallback(deviceArrival)
YAPI.RegisterDeviceRemovalCallback(deviceRemoval)


To contact a YoctoHub or YoctoHubs, simply add more calls to PreregisterHub with the IP address of the YoctoHub(s). All YoctoHubs will be managed in parallel in a transparent way.

The body of the program is very simple: it just gives the library the hand to dispatch events, and display the time passing to know that it is still working:

while True:
    YAPI.UpdateDeviceList(errmsg)  # handle plug/unplug events
    YAPI.Sleep(500, errmsg)        # handle timed reports
    # display current time continuously
    now = datetime.datetime.now().strftime("%H:%M:%S")
    print("\b\b\b\b\b\b\b\b\b"+now, end=' ')


Connection/disconnection management


When a module announces itself to the program for the first time, the arrival callback is automatically called, with the corresponding YModule object as a parameter.

def deviceArrival(module):
    serial = module.get_serialNumber()
    print('Device online : ' + serial)


We'd like to enumerate the sensor type functions found there, to associate with each an instance of SensorManager responsible for storing its measurements. But if the module has already been seen once in the past, it should be done only the first time. So we will use a very useful method of the Yoctopuce library: get_userData(). This method allows you to associate an object of your choice to each Yoctopuce module. We will use it to store the list of SensorManager instances of each module. So here is the continuation of the deviceArrival function:

    # If this device is unknown, enumerate sensors on the device
    sensorManagers = module.get_userData()
    if sensorManagers is None:
        sensorManagers = []
        sensor = YSensor.FirstSensor()
        while sensor:
            if sensor.get_module().get_serialNumber() == serial:
                # For each sensor, create a SensorManager
                # and add it to the list
                handler = SensorManager(sensor)
                sensorManagers.append(handler)
            sensor = sensor.nextSensor()
        module.set_userData(sensorManagers)


All that remains is to call the SensorManager method that handles new module connections in all cases:

    # Notify the SensorManager about the arrival
    for handler in sensorManagers:
        handler.handleArrival()


The method that handles module disconnection only has to pass the call to the SensorManager instances:

def deviceRemoval(module):
    print('Device offline : ' + module.get_serialNumber())
    sensorManagers = module.get_userData()
    for handler in sensorManagers:
        handler.handleRemoval()


The SensorManager class


The two important things this class must do are:

  • Store data in real time;
  • Recover missing data in case of disconnection.

It all starts with the handleArrival method, which we called above.

def handleArrival(self):
    # register a callback for getting new measurements
    self.sensor.registerTimedReportCallback(self.sensorTimedReportCallback)
    # check the timestamp of the last measurement already known
    self.lastStamp = self.getLastTimestamp()
    # whether we need to recover any measurement from the datalogger
    print("Load missing data from %s :" % self.sensorName, end='     ')
    dataset = self.sensor.get_recordedData(self.lastStamp, 0)
    dataset.loadMore()
    progress = 0
    while progress < 100:
        progress = dataset.loadMore()
        print("\b\b\b\b%3d" % progress, end='%')
    details = dataset.get_measures()
    for measure in details:
        self.appendMeasureToFile(measure)
    print(' done')


These few lines are enough to recover the missing measurements in case of disconnection. For real time measurements, it is even easier:

def sensorTimedReportCallback(self, sensor, measure):
    # make sure we never go back in the past
    if measure.get_endTimeUTC() > self.lastStamp:
        self.appendMeasureToFile(measure)


The method of storing the values (appendMeasureToFile) depends on the application. In our case, we made a simple CSV file, but you may prefer to use a database:

def appendMeasureToFile(self, measure):
    self.lastStamp = measure.get_endTimeUTC()
    utc = measure.get_endTimeUTC_asDatetime()
    stamp = utc.strftime(SensorManager.DATEFORMAT)
    value = measure.get_averageValue()
    with open(self.dataFile, 'a') as file:
        file.write("%s;%.3f\n" % (stamp, value))


Similarly, the getLastTimestamp method must return the time of the last stored measurement, and will need to be adapted when using a database with a query of type SELECT MAX(`timestamp`) FROM `table`. In our case, we just read back the timestamp of the last line of the CSV file.

Conclusion

This shows that only a few lines of code are needed to build an application that makes optimal use of Yoctopuce sensors.

If you prefer to work in another language than Python, the translation of these few lines should not take you long: the principles demonstrated are applicable in all languages supported by our library, since we offer exactly the same functions for all languages. In case of doubt, don't hesitate to contact Yoctopuce support.




1 - harryg Tuesday,october 04,2022 13H59

Thanks for this great article.

Is there any way I can get an automatic update by email when new articles are published?

many thanks
Harry

2 - martinm (Yocto-Team)Wednesday,october 05,2022 11H16

@harryg: There is one new article published every week, usually on Friday around noon, Swiss time. We don't send notification emails, but you could probably use our RSS feed to get one: www.yoctopuce.com/EN/rssfeed.php

Yoctopuce, get your stuff connected.