HTTP callback optimization

HTTP callback optimization

Some weeks ago, we published an update of our libraries which reduces the quantity of transferred data between the Yoctopuce modules and the API. However, this optimization wasn't available for HTTP callbacks. This week, we publish a new PHP library which enables you to take advantage of a similar improvement for HTTP callbacks.




In the latest version of our PHP library, we introduced a cache specific to the HTTP callback mode. When enabled, this cache allows you to reduce by 50% to 70% the quantity of data that the Yoctopuce modules send. This is particularly interesting if you use a YoctoHub-GSM, because usually the service providers make you pay dearly for each data byte that goes through their network.

To take advantage of this new cache, you must use a YoctoHub or a VirtualHub with version v27750 of the firmware or more recent, and version 1.10.27759 of the PHP library or more recent.

This new cache is completely backward compatible, that is whatever the version of PHP library of the YoctoHub or of the VirtualHub that you use, HTTP callback will work. However, the benefit is visible only if the two components are recent enough. This allows you to update the PHP library on a web server and to progressively update the YoctoHub firmware without service interruption.

Enabling the cache


To benefit from the cache, you must call the YAPI::SetHTTPCallbackCacheDir() method before any other function of our library. This method takes as parameter the path of a directory where the data is stored between two HTTP callbacks. This directory must already exist and the PHP script needs write privileges for this directory.

<?php
require_once('yocto_api.php');
// enable HTTP callback Cache
YAPI::SetHTTPCallbackCacheDir("../hidden/cache_dir");

$errmsg = "";
if(YAPI::RegisterHub('callback', $errmsg) != YAPI_SUCCESS) {
    print("Unable to start the API in callback mode ($errmsg)");
    die();
}
...



We recommend that you don't use a public directory because the library is going to store there data coming from the Yoctopuce modules. If you use a directory which can be accessed on the web site, some clever guy could guess the path and access these data.

How the cache works


During the first callback, the YoctoHub sends all the data to the PHP script. If the PHP script uses the cache with the YAPI::SetHTTPCallbackCacheDir() method, the PHP library stores the received data in the cache directory and tells the YoctoHub that it doesn't need to transfer part of the data during the next callbacks.

Concretely, the data are sent in JSON format. During the first callback, the JSON structure is fully sent, but during the following callbacks, only the values are sent and the structure keys are replaced by indices. To reconstruct the full JSON structure, the PHP library merges the received data and the data stored in the cache. This optimization enables you to usually reduce the traffic by 50% to 60% without overloading the YoctoHub CPU.

You must remember that the first callback of a YoctoHub always sends the whole data and only the following executions benefit from the "compression". If the YoctoHub restarts or goes to sleep, it goes back into "uncompressed" mode and sends the whole JSON structure again.

The Yocto-API-JZON callback

For scenarios in which the YoctoHub goes into sleeping mode between each callback, for example as when we tracked the shipping of a package, the cache is never used because the YoctoHub goes back into standard mode at each wake up. To solve this issue, we added a new callback type: the Yocto-API-JZON callback.

This mode is identical to the Yocto-API mode, with the difference that by default it sends the data in compressed mode. The full JSON structure is only sent the first connection to the PHP library. The cache is then valid for all the following callbacks, even if the YoctoHub restarts or goes into sleeping mode.

Important note: this mode is not backward compatible and works only if the PHP script has properly configured the cache. You are therefor required to use an up-to-date PHP library and to call the YAPI::SetHTTPCallbackCacheDir() method at the beginning of the PHP script.

For the YoctoHub to use this mode, you must simply select "Yocto-API-JZON" in the HTTP callback configuration panel. The parameters are the same as for "Yocto-API" callbacks.

The Yocto-API-JZON mode is available in the callback configuration window
The Yocto-API-JZON mode is available in the callback configuration window



Conclusion


If you use a YoctoHub-GSM, this new feature should easily cut your phone bill by half. For other YoctoHubs, the traffic decrease is the same but its impact is less as in most cases the connection is free :-)




1 - lorenzo @ valarm Wednesday,june 28,2017 5H59

Java version, s'il vous plaît, bitte schoen, per favore... - minus the dependency on the file system. Instead provide a way to pass a serializable "YoctoHubConfigCache" object that we can serialize/deserialize into our database. :-)

2 - seb (Yocto-Team)Wednesday,june 28,2017 13H37

@lorenzo: With Java you can use WebSocket Callbacks that's already use compressed format.

Yoctopuce, get your stuff connected.