In an computer system, measure and event timestamps are considered an evidence and, nowadays, no one expects to encounter difficulties in this domain. However, knowing the exact time at any time is far from obvious, in particular if you work with autonomous sensors with limited resources.
Very concretely, we have been very recently made aware that the presence of time fluctuations in the history of the measures recorded by our modules gave rise to serious interpretation problems and that we should do something about it:
A measure history during which there is a rectification of the module clock
We can easily explain this phenomenon: the sensor internal clock is rather inaccurate and, when at regular intervals a rectification is received from the computer or from the network, this rectification is applied so that the time associated to the measure can remain as precise as possible for the following measures. So far, we were happy to associate to each measure the best known time at the time that the measure is logged. Simple and efficient, but... not very satisfactory as such.
Accuracy of a numerical clock
We can start by asking ourselves whether it makes sense to have to apply a rectification of several seconds to an internal clock. After all, a processor is a precision tool, isn't it? Well, it isn't, not when time is concerned because it's not really a requirement. For example, two processors can communicate by USB perfectly well even if the clock of the first one goes 2% faster than that of the second one. But 2%, over 5 minutes, it's already a 6 second drift in the time count. After a day, it's already a half-hour drift that you'd see.
Among the causes of internal clock variation we find not only temperature changes, but also a very significant influence of mechanical stress on the circuit: by pressing on the top or the bottom of the circuit, one can change the clock frequency by 1 to 2% in one direction or the other.
Mechanical overclocking, don't do this at home :-)
Fortunately, this can be improved by adding a quartz or an oscillator, which is typically accurate to 100 or 200ppm, for example. It's what we did for the clock which wakes up the YoctoHub-Wifi at predefined times. But it's still far from perfect: up to 8.5 second drift per day, so 5 minutes per month...
Some watch quartz are more accurate, to 20ppm for example. But there will always remain a significant drift. The only solution consists therefore in regularly correcting the drift from an absolute source, to the best of our abilities.
Absolute time sources
Thankfully, there are a few reliable references to obtain a non-drifting time. With Internet, the easiest way is to use an SNTP server (itself synchronized on a server with an up-to-date clock). It's the strategy that we used up to now with the YoctoHub-Ethernet and YoctoHub-Wireless-g. By the way, if you work with static IP addresses, make sure to configure a DNS server, as it is needed to locate the SNTP server nearest to you.
For devices connected by USB, one can simply use the time of the computer host, as this computer is very likely automatically synchronized on an NTP server. This is what Yoctopuce VirtualHub and API do automatically for you.
A potential issue of SNTP time synchronization is latency: depending on network conditions, there can be several seconds between when the SNTP server sends the time and when you receive it. To prevent this, there are SNTP servers everywhere, but one can't exclude a local network congestion.
To palliate this, there is an ultimate solution: using a GPS. Indeed, GPS signals contain very accurate absolute time information. As long as you can put an antenna in an adequate location, it's the most reliable manner to have the absolute time in autonomy.
The Yocto-GPS: your best real time clock
Using a corrected clock
As mentioned above, up to now the timestamps of Yoctopuce sensor measures were based on the sensor internal clock, with an immediate time rectification when connected, followed by a periodic correction, by SNTP for a network connection, or based on the computer clock for a USB connection. With the noted step-back-in-time effects during rectification.
However if we allow ourselves to modify the speed of the sensor clocks, we can do better than this. First, we can apply rectifications progressively, overtaking one second every 3 or 4 minutes, for example, by varying from a fraction of a percent the clock speed. Second, on the basis of receiving the absolute time every minute, the module can at any time compute the drift of its internal clock compared to the reference and correct the speed of its internal clock to stay as close as possible to the reference clock. The adaptation can be performed in a few tens of minutes and thus considerably reduce the need of future rectifications.
We published these two modifications today in the new firmware for our modules. You can therefore get rid of these annoying second skips in your data logs.
Icing on the cake, the YoctoHubs are now also able to detect the presence of a Yocto-GPS and to use the time obtained by GPS as a reference (instead of SNTP) on all the connected modules.