Exporting consolidated measures in CSV format

Exporting consolidated measures in CSV format

Among the questions that regularly come up for Yoctopuce support, we find in good position the question of how to generate a CSV file with data recorded by several sensors. So, to save you some time, we finally decided to add this function directly in the programming library.




We already discussed how to retrieve logged data sensor by sensor in a previous post in the tutorial series. So we won't discuss the basic concepts on how to turn the data logger on, and so on.

The part we are interested in today is the parallel retrieval of data from several sensors. In a simple world, you would only have to scan the measure arrays in parallel for each sensor, which is not rocket science. But if you want the code to be robust, you must take into account that the sensors haven't necessarily been turned on at the same time and don't necessarily record the data at the same frequency, so there might be missing data for a given time stamp.

The new function that we propose therefore properly manages the absence of data without shifting data. To keep the interface as simple as possible, it returns the measures through a simple array of real numbers, but in the case a measure is missing for a given sensor, the value is the special number Not a Number (NaN), which has the advantage of propagating naturally and is easily detected in all languages using the isnan (sometimes called isNan or is_nan) function.

The YConsolidatedDataSet class


We therefore created a new YConsolidatedDataSet class, that you can instantiate by giving as parameters:

  • the Unix time stamps corresponding to the beginning and end date of the period for which you want to retrieve the data, or 0 and 0 to retrieve all the data.
  • the array of YSensor objects (or YTemperature, YHumidity, and so on) for which you want to retrieve the consolidated data.

Then call in a loop the nextRecord() method, providing as parameter a list of real numbers (or an array depending on the language) by reference. As long as measures are found, the function returns true and fills out the list given as parameter with the next time stamp and the corresponding measures. When all the measures are exhausted, the function returns false.

Example


Here is a usage example of this new class, in Python, to generate a CSV file, with the date in the standardized ISO format:

# Enumerate all connected sensors
sensorList = []
sensor = YSensor.FirstSensor()
while sensor is not None:
    sensorList.append(sensor)
    sensor = sensor.nextSensor()

# Generate consolidated CSV output for all sensors
data = YConsolidatedDataSet(0, 0, sensorList)
record = []
while data.nextRecord(record):
    line = datetime.datetime.fromtimestamp(record[0]).isoformat()
    for idx in range(1, len(record)):
        line += ";%.3f" % record[idx]
    print(line)


You can find this same example in each of the Yoctopuce programming libraries, in the Prog-ExportCSV subdirectory.

Conclusion


This new class should make it easier to retrieve data for users who have little experience in programming. In order to keep the interface as simple as possible, we intentionally bypassed some advanced features, such as the possibility to read min/max values on top of average values, and the possibility to read the time stamp of the beginning of the measure on top of the time stamp of the end of the measure. But the source code is provided in the library, so you can also start from this class to code an improved version according to your own needs.

Add a comment No comment yet Back to blog












Yoctopuce, get your stuff connected.