Although Yoctopuce sensors have a built-in data logger, it is often useful to store the data in a database or CSV file for local processing. Ideally, one would like to fetch the data in real time as much as possible, but reserving the possibility to retrieve the data stored by the module in case of temporary connection loss. And this is not so difficult to do...
An example of an application that uses this technique combining real-time acquisition and offline data retrieval from the module's datalogger is the Yocto-Visualizationtool. If you enable the use of the datalogger in the graph configuration, you will find that it is able to retrieve all intermediate measurement points in case of a network outage.
So today we offer you a small program in Python that illustrates the programming techniques used in Yocto-Visualization to achieve this result. In a few dozen lines, this program can monitor an arbitrary number of sensors and log their measurements in chronological order in a CSV file - one file per sensor -, automatically recovering the missing data in case of loss of connection or temporary program shutdown.
For people in a rush
You can find this program in our programming library for Python, in the directory Examples/Prog-SensorManager. If you want to monitor sensors in real time via CSV files and are really too rushed to do your own adaptation of this code, this can even be a ready-to-use solution. Just run it in a Unix shell or a Windows command window:
Hit Ctrl-C to Stop
Device online : LIGHTMK1-956BE
Load missing data from LIGHTMK1-956BE.lightSensor : 100% done
18:36:23 Device offline : LIGHTMK1-956BE
18:36:28 Device online : LIGHTMK1-956BE
Load missing data from LIGHTMK1-956BE.lightSensor : 100% done
18:37:08 (clock ticking...)
As long as the program is running, you will find the real time data in the LIGHTMK1-956BE.lightSensor.csv directory, as you can check using tail -f for instance.
Remember that in order for this program to work, you must first:
- Enable timed reports (regular interval measurements) at the frequency of your choice on each measurement function of interest
- Activate the recording of measurements on these same functions
- Activate the data logger on the module
These operations can be done using VirtualHub or Yocto-Visualization.
How does it work?
The program is structured in three parts:
- initialization of the Yoctopuce library, which connects to USB and/or YoctoHubs
- functions that manage the connection/disconnection of Yoctopuce modules
- the SensorManager class, of which we will create one instance per sensor to monitor
Initialisation
To initialize the Yoctopuce library in a temporary disconnection tolerant mode, we use the following lines:
sys.exit("Init error: " + errmsg.value)
# Use Arrival/Removal callbacks to handle hot-plug
YAPI.RegisterDeviceArrivalCallback(deviceArrival)
YAPI.RegisterDeviceRemovalCallback(deviceRemoval)
To contact a YoctoHub or YoctoHubs, simply add more calls to PreregisterHub with the IP address of the YoctoHub(s). All YoctoHubs will be managed in parallel in a transparent way.
The body of the program is very simple: it just gives the library the hand to dispatch events, and display the time passing to know that it is still working:
YAPI.UpdateDeviceList(errmsg) # handle plug/unplug events
YAPI.Sleep(500, errmsg) # handle timed reports
# display current time continuously
now = datetime.datetime.now().strftime("%H:%M:%S")
print("\b\b\b\b\b\b\b\b\b"+now, end=' ')
Connection/disconnection management
When a module announces itself to the program for the first time, the arrival callback is automatically called, with the corresponding YModule object as a parameter.
serial = module.get_serialNumber()
print('Device online : ' + serial)
We'd like to enumerate the sensor type functions found there, to associate with each an instance of SensorManager responsible for storing its measurements. But if the module has already been seen once in the past, it should be done only the first time. So we will use a very useful method of the Yoctopuce library: get_userData(). This method allows you to associate an object of your choice to each Yoctopuce module. We will use it to store the list of SensorManager instances of each module. So here is the continuation of the deviceArrival function:
sensorManagers = module.get_userData()
if sensorManagers is None:
sensorManagers = []
sensor = YSensor.FirstSensor()
while sensor:
if sensor.get_module().get_serialNumber() == serial:
# For each sensor, create a SensorManager
# and add it to the list
handler = SensorManager(sensor)
sensorManagers.append(handler)
sensor = sensor.nextSensor()
module.set_userData(sensorManagers)
All that remains is to call the SensorManager method that handles new module connections in all cases:
for handler in sensorManagers:
handler.handleArrival()
The method that handles module disconnection only has to pass the call to the SensorManager instances:
print('Device offline : ' + module.get_serialNumber())
sensorManagers = module.get_userData()
for handler in sensorManagers:
handler.handleRemoval()
The SensorManager class
The two important things this class must do are:
- Store data in real time;
- Recover missing data in case of disconnection.
It all starts with the handleArrival method, which we called above.
# register a callback for getting new measurements
self.sensor.registerTimedReportCallback(self.sensorTimedReportCallback)
# check the timestamp of the last measurement already known
self.lastStamp = self.getLastTimestamp()
# whether we need to recover any measurement from the datalogger
print("Load missing data from %s :" % self.sensorName, end=' ')
dataset = self.sensor.get_recordedData(self.lastStamp, 0)
dataset.loadMore()
progress = 0
while progress < 100:
progress = dataset.loadMore()
print("\b\b\b\b%3d" % progress, end='%')
details = dataset.get_measures()
for measure in details:
self.appendMeasureToFile(measure)
print(' done')
These few lines are enough to recover the missing measurements in case of disconnection. For real time measurements, it is even easier:
# make sure we never go back in the past
if measure.get_endTimeUTC() > self.lastStamp:
self.appendMeasureToFile(measure)
The method of storing the values (appendMeasureToFile) depends on the application. In our case, we made a simple CSV file, but you may prefer to use a database:
self.lastStamp = measure.get_endTimeUTC()
utc = measure.get_endTimeUTC_asDatetime()
stamp = utc.strftime(SensorManager.DATEFORMAT)
value = measure.get_averageValue()
with open(self.dataFile, 'a') as file:
file.write("%s;%.3f\n" % (stamp, value))
Similarly, the getLastTimestamp method must return the time of the last stored measurement, and will need to be adapted when using a database with a query of type SELECT MAX(`timestamp`) FROM `table`. In our case, we just read back the timestamp of the last line of the CSV file.
Conclusion
This shows that only a few lines of code are needed to build an application that makes optimal use of Yoctopuce sensors.
If you prefer to work in another language than Python, the translation of these few lines should not take you long: the principles demonstrated are applicable in all languages supported by our library, since we offer exactly the same functions for all languages. In case of doubt, don't hesitate to contact Yoctopuce support.
