Most of our USB sensors include an embedded data logger. It is trivial to activate this data logger. However, after using it for a while, we noticed that the programming interface used to retrieve the data is not very intuitive. We therefore decided to bring some improvements to it to facilitate its use. Here is an outline of what will probably change, to provide you with the opportunity to give us your feedback before the choices are definite.
Here is a summary of the comments we received concerning the data logger:
1. Recording frequency
The current method (one measure per second, then compressed to one measure per minute, then further compressed to one measure per hour) is "subtle" but not the most practical. A fixed frequency, selected by the user, would be simpler to use. For some sensors (such as the Yocto-Volt, Yocto-Thermocouple), a measure interval lower than the second would be good.
2. Splitting the measures into "streams"
Having to look up first the list of streams, then the data stream by stream renders the user task more complex. It would be good for the library to take charge of this transparently.
3. Measure reading time
Because of the large number of data stored, downloading them can take a noticeable time. Ideally, the library should provide a mechanism able to download these data as a background task, so as to relieve the developper from this task. The library should also ideally be able to quickly provide an overview of available data.
4. Combined use of the data logger and of current measures
The data logger interface is completely distinct from the interface to read current measures. When you want to represent in a single graph historical data and current data, you have to treat each type of data separately. It would be good to access historical and current data through a single interface.
5. Time-stamped data
To associate an actual hour to a measure flow, you have to make an explicit call to a configuration function when you start the data logger. Therefore, the measures are often not time-stamped, which is not practical.
6. Using the data logger with the HTTP callback API
Using the data logger with the API in HTTP callback mode is currently very difficult. It would however be useful to be able to retrieve by HTTP callback at least the most recently stored/averaged values in the module in an autonomous way.
What a program... here is therefore what we offer to change:
New item 1: Fixed but configurable recording frequency
By popular demand, our future firmwares will automatically use a fixed recording frequency. The frequency will be initialized to one measure per second to maintain compatibility with previous firmware. The recording frequency will be configurable through the programming interface and in the module configuration page, which you can access in the VirtualHub. The highest possible frequency will be 10 measures per second. To preserve space on the flash memory, it will be possible to select the specific sensors for which the measures must be recorded.
In the opposite to the current version, which only uses a discreet measure, the stored measure will now be the average of the observed values during the sampling period. You will also be able to store minimal and maximal values. The "subtle" data compression function will be suppressed in order to keep the complete flash memory for measures at the selected frequency.
If you need a firmware with the old behavior, let us know: we can provide you with one. However, updates proposed by the VirtualHub will probably always be firmwares with a fixed frequency recording.
New item 2: Reading the records through a callback function
To simplify how you retrieve measured data, from the data logger as well as for current measures, it will be possible to record on each function a callback notifying a periodic measure. This callback will look like this:
To retrieve the measures stored in the data logger for a given period, you will only need a few lines of code, such as for example:
var startTime = endTime - 86400;
tempSensor.registerTimedValueCallback(callback, startTime, endTime, userCtx);
humSensor.registerTimedValueCallback(callback, startTime, endTime, userCtx);
The registerTimedValueCallback method will look up all the available measures during the specified period, and call the specified callback functions for each existing record. Downloading the data will be performed as a background task so you won't need to worry about it. The callbacks will be invoked during calls to YAPI.HandleEvent() and YAPI.Sleep(), as it is currently done for the notification of current values. When the last known recorded value is sent, progress will be set to 100.
Moreover, the same callback will be called for all new measures as they appear, when the specified range include future time. You can therefore build a dynamically updated graph without additional effort.
The old blocking data reading functions (directly through the dataLogger object) are maintained for backward compatibility.
New item 3: Direct measure retrieval
For cases where an even simpler method is desired, we will provide a blocking function to retrieve the summary of available measures, and the nearest measure to a given time.
New item 4: Automatic time stamp
The data logger will now have a setting enabling it to automatically retrieve UTC time from the VirtualHub or the YoctoHub-Ethernet to which it is connected. This functionality will be set by default, but you can disable it if you want to use another time base which you manage yourself (such as local time for instance).
New item 5: Simplified access to the REST API and to HTTP callback
A new URL will be available on the VirtualHub (and on the YoctoHub-Ethernet) enabling you to retrieve in the JSON format with a simple REST request the 5 latest measures stored in the datalogger for each sensor. These same measures will also be provided to the HTTP callbacks.
A real example
To give you a taste of what will be possible to do with the new method to read data records by callback, we have prototyped it for the JavaScript API. The final version will require a new firmware to more efficiently load the new measures, but this "emulated" version works already with the current firmware.
The web page demonstrated below is written in vanilla JavaScript. You can find the page source code here. It can automatically detect newly connected devices (using the VirtualHub), and shows the measure history as well as live measures.
For the graphical representation, with used the Highstock JS library, one of the most flexible to draw graphs with JavaScript. It is free for non-commercial users, but not for companies like ourselves. Here is how we inject the data:
var now = +new Date() / 1000;
for(var i = 0; i < sensors.length; i++) {
sensors[i].registerTimedValueCallback(initialCallback, 0, now, [ ]);
}
// Callback used to setup chart from datalogger measures
function initialCallback(sensor, timeUTC, minVal, avgVal, maxVal, progress, userCtx)
{
// Store one measure
userCtx.push( [timeUTC*1000,avgVal] );
// Update progress bar
var serial = sensor.get_module().get_serialNumber();
document.getElementById(serial+"gauge").style.width = (2*progress)+"px";
// When all measures are received, build graph
if(progress == 100) {
var sensorId = sensor.get_hardwareId();
document.getElementById(serial+".progress").style.display = "none";
document.getElementById(sensorId).style.height = "400px";
chart = new Highcharts.StockChart({
chart : { type: 'line', zoomType: 'x', renderTo: sensorId },
scrollbar: { liveRedraw: true },
title: { text: sensorId },
rangeSelector: {
buttons: [{ count:5, type:'minute', text:'5min' },
{ count:1, type:'hour', text: 'hour' },
{ count:1, type:'day', text: 'day' },
{ type:'all', text: 'All' }],
selected: 2 },
xAxis : { minRange: 60 * 1000 }, // one minute
series : [{ name: sensor.functionId, data : userCtx }]
});
// enable live update with new data
sensor.registerTimedValueCallback(liveCallback, 0, -1, chart);
}
}
// Callback used for live update with new measures
function liveCallback(sensor, timeUTC, minVal, avgVal, maxVal, progress, userCtx)
{
userCtx.series[0].addPoint( [timeUTC*1000, avgVal], true );
}
What do you think?
Are you happy with the planned changes ? Let us know your comments and suggestions, by email or by leaving a comment on this post.
Of course all this will not happen within a week, but unless you change our mind, these changes should appear progressively within the next months.
