Logging and retrieving measures without network access

Logging and retrieving measures without network access

Even in the Internet of Things era, there are automation and measuring applications that require to work in a completely autonomous way, without any network connection of any kind. Someone asked us recently how to implement such an autonomous data logger using our sensors. The system should not require any network connection but use a USB key to retrieve the data. We are going to cover this example in details and at the same time take this opportunity to test a relatively new mini-computer that we recently discovered...

It isn't difficult to log the measures coming from a Yoctopuce sensor: all the sensors are able to do it on their own, thanks to the embedded flash memory. But there are two related issues that we need to solve for this application:

  1. To retrieve the data from the module flash memory and to ease their further treatment, we must be able to detect when a USB key is connected, to read the logged data for each sensor, and then to transcribe them on the USB key on a single table in the CSV format. It's the type of operations that you can perform on a mini-computer, in Python for example.
  2. For the data to receive a correct time stamp, we need an absolute time source. When no network is available, the only solution is to have a quartz clock with a battery: it's what is called a Real Time Clock (RTC). Any real computers has one, but it's not as frequent on the mini-computers that we want to use for this type of application.


The CuBox-i2eX: a mini-computer with an RTC


While looking for a mini-computer with an RTC, we discovered the SolidRun company, which manufactures several mini-computers which are a little neater than what the competition does. In the ARM low consumption mini-computer category, you can find several variants based on a NXP iMX6 processor, with 1 to 4 cores. You have the choice between a development board in the Raspberry Pi style and a small compact machine delivered in an enclosure with a power block.

The CuBox-i2eX from SolidRun is one of the few low consumption ARM mini-computers with an embedded RTC
The CuBox-i2eX from SolidRun is one of the few low consumption ARM mini-computers with an embedded RTC


The version we are most interested in, with the RTC, enclosure, and power block, costs $119. It's more expensive than a Raspberry Pi or a BeagleBone, but having an embedded RTC makes all the difference. SolidRun provides several supported Linux images that you can chose from, including a Debian Linux. The proposed installation system, called Ignition is very efficient. By following the instructions, we quickly and without trouble obtained a ready-to-use Debian on the machine. As the goal is to setup a system without keyboard or screen, we selected a distribution without graphical interface.

Implementation

Principle


The scenario is the following:

  • Normally, the Yoctopuce sensor registers the measures internally. The computer is used only to provide the exact time to the sensor for a correct time stamp.
  • From time to time, an operator comes to retrieve the data with a USB key. The USB key contains a file indicating the time period for which we want to retrieve the measures. The operator connects the key and can check, with a mini USB screen, that the transfer is performed as planned. The data are stored in a CSV file with a name specific to the saved sensor. The operator is informed when it is possible to remove the USB key

As a mini-computer doesn't have a screen, we had to introduce in the system a small USB screen (namely a Yocto-MiniDisplay) so that the operator gets some feedback on the process. The screen can be connected to the mini-computer, or the operator can bring it with the USB key, as illustrated below:

The components of our network-free logged data retrieval system
The components of our network-free logged data retrieval system



Implementation


To easily detect when a USB key is connected, you must install the usbmount package with the command:

sudo apt-get install usbmount


Thanks to this, the USB key magically appears under /media/usb when it is connected. We ask the operator to create on the key a text file called autoload.txt, which simply contains the date from which the data must be retrieved, in the YYYY-MM-DD format. Every second, the Python program checks if this file appears under /media/usb and if it is the case, the data retrieval process is triggered.

The autonomous data logger   Recovering data on a USB key
Our autonomous data logger system, with data retrieval through a USB key



Reading the data is performed one sensor after another, using the get_recordedData() method. As this operation can be time consuming, we display a global progress bar on the Yocto-MiniDisplay while downloading the data of all the sensors, using the mini-display drawing primitives. The code isn't very complex, but it is nevertheless too long to be commented one line after the other in this post. You can find it in its full version on GitHub, you can adapt it to your own need if you want to.

To fit all the data returned by all the sensors in a single CSV table, we must take into account that all the sensors may not be recording measures at the same rate. So we start by establishing the list of all the timestamps found among the logged measures, we sort it, and we then generate the CSV file with a distinct line for each timestamp, checking for each sensor whether it has a measure for this specific time. If not, the cell is left empty.

The file is written line by line, again with a global progress bar. At the end, a message asks the operator to remove the key. usbmount standard settings guarantee that the data are written on the USB disk without delay, so that no manual operation is required before disconnecting the key.

Demonstration


And here is the system in operation:

  



Conclusion


The CuBox + Yoctopuce module mix proved very efficient for this project, implemented in less than a day. Using the Yocto-MiniDisplay enabled us to add some user-friendliness, allowing the operator to use the mini-computer in a more comfortable setting.

We didn't encounter any USB issue, nor any other reliability issue with this small machine. Moreover, when the system is working in normal mode (autonomous data logging by the Yoctopuce sensors), the CuBox-i2eX is very energy efficient: it consumes only 200mA, of which 30mA are used to power the Yoctopuce sensor. This represents barely 1 Watt. It's better than most mini-computers that we tested. So we are certainly going to reuse it for other applications...

Add a comment No comment yet
Back to blog












Yoctopuce, get your stuff connected.