New: a VirtualHub that works through the web

New: a VirtualHub that works through the web

When setting up an automation or measure system, it is essential to be able to monitor it remotely and easily. Today, we offer you a new solution to monitor and control your Yoctopuce-based systems via the Internet, which you can easily install on (almost) any web host: here is the VirtualHub for Web.




This is not the first solution that we have proposed for this purpose: in the last few years, we have shown you on this blog how to integrate your solutions with various third-party systems, such as EmonCMS, Valarm.net, Xively, InfluxDB, PRTG... But experience has shown us that in the long run, dependency on a third-party controlled service often becomes problematic: free or cheap systems rarely stay that way for long, and the price charged for a simple measure storage is often not comparable to the price one would pay to transfer and store the same amount of data on a standard host.

We have therefore chosen to offer you a new free tool to facilitate remote access to the data of your Yoctopuce modules. This tool is provided with the source code, and is therefore entirely in your hands.

VirtualHub for Web, what's that?


The purpose of the VirtualHub (for Web) is to provide a remote access to your Yoctopuce modules, through Internet. It is therefore a software that must be installed on a Web server. You can install it on any reasonably up-to-date standard PHP host, by following the procedure described below.

To connect a Yoctopuce system to the VirtualHub (for Web), you just need to configure an HTTP callback on the YoctoHub that controls it pointing to this Web server. During each callback, the VirtualHub (for Web) automatically downloads all the necessary information from the YoctoHub, including the configuration of the modules, the latest sensor data, the log messages from the modules, and even the files needed for the user interface.

When you establish a connection with a web browser to the VirtualHub (for Web) server, you get an interface very similar to that of a regular YoctoHub or VirtualHub.

The interface of the VirtualHub (for web)
The interface of the VirtualHub (for web)


You can go and view the status of each module, as it was the last time the module connected to the server. You can even change the module settings: the changes are applied the next time the module connects to the server.




The interface shows the last known status of the modules and allows you to perform a deferred configuration



The VirtualHub (for Web) emulates in a rather exhaustive way the behavior of Yoctopuce hubs and the modules connected to it. It is even able to emulate the module data logger (dataLogger function), since it stores exactly all the measurements transmitted by the sensors on the server, in the same format as they are stored on the flash memory. So you can even connect to it with applications based on the Yoctopuce API to retrieve the data. For example, if you load the latest version of Yocto-Visualization, you are able to add an instance of the VirtualHub (for Web) as a data source and see the data from remote hubs arrive in your graphs:




Yocto-Visualization can visualize the measures stored by the VirtualHub (for Web)



TheVirtualHub (for Web) can be used with a YoctoHub-Ethernet, a YoctoHub-Wireless-n, a YoctoHub-GSM-4G, as well as with all the previous versions of YoctoHubs we have been selling for 10 years. If you are controlling your Yoctopuce modules via USB, you can use the usual VirtualHub tool to perform HTTP callbacks equivalent to what a YoctoHub would do, but then you have to make sure your application goes through the VirtualHub to access the modules.

Installation

Installation on PHP server


To simplify the installation of this tool as much as possible, we have created a small PHP installer. You can find on our tools page a zip file containing a single PHP file, which you can put on your web server in the subdirectory where you want to see VirtualHub (for Web). If you do this using FTP, make sure to use binary mode. Then open a browser window pointing to this file and follow the few steps of the installation wizard.



  
Launching the VirtualHub (for Web) installer



It is not necessary to have a database on the Web server to use the VirtualHub (for Web): all the data retrieved from the Yoctopuce modules are stored by the PHP code directly in files on the server, in the form of a TAR archive, at the rate of one TAR file per Yoctopuce module.

The installer offers you to create several instances of the VirtualHub (for Web), if you wish. Without duplicating the code on the web server, this option allows you to separate several subsystems of Yoctopuce modules that are displayed separately, and their data are stored in separate directories.

As of today, we have tested the installer and the software on PHP 7.4, 8.0 and 8.1, running in the following modes: apache2handler, fpm-fcgi and lightspeed.

Configuration of HTTP callbacks on YoctoHub or VirtualHub


First, make sur to update your YoctoHub firmware (or update your VirtualHub) to version 51900 or higher, and the other Yoctopuce modules to firmware 50730 or higher.

You should then configure your outgoing callbacks as follow:

  1. Type of callback: Yocto-API callback
  2. Callback URL: _virtualhub-4web_instance_path_/HTTPCallback
  3. Type of security: MD5 signature
  4. Password: the same as you will set on the server for incoming callbacks


Transmission of measured values


One of the major goals of the VirtualHub (for Web) is to allow the precise transmission of the measures made by the Yoctopuce sensors to the web server, despite the fact that it is only possible to access the data logger of the modules in HTTP callback. To do this, YoctoHubs are now able to transmit periodic notifications (timed reports) to the server via HTTP callback. These are kept in memory by the hub during the entire time interval between two HTTP callbacks and sent in bulk to the PHP server at the time of connection, while preserving the exact time of the measurements.

With this, for example, you can take measures every hour during a whole day and send them via a single daily HTTP callback. Or conversely, you can configure a sensor to measure a signal at 10Hz, and not lose any measure despite an HTTP callback frequency of 0.3Hz.

Yocto-Visualization monitoring a sine wave signal sampled at 10Hz, using one callback every 3[s]
Yocto-Visualization monitoring a sine wave signal sampled at 10Hz, using one callback every 3[s]



However, this technique has several consequences that must be taken into account:

  1. It is not the logging frequency (logFrequency) configured on the sensor that is decisive, but the frequency of periodic notifications (reportFrequency) to know which measurements are logged by the VirtualHub (for Web).
  2. If you connect with the Yoctopuce library to the VirtualHub (for Web), you receive at the time of each HTTP callback the package of periodic notifications that have just been transmitted by the hub. So they are not transmitted one after the other, but the timestamp returned by get_endTimeUTC is correct.
  3. If we want to transmit the entire set of measures, we need to make HTTP callbacks often enough so that they can be kept in memory in the hub buffer, which can store about 150 measures.

If needed, the VirtualHub (for Web) web interface allows you to see if you are very close to saturating the periodic notification buffer or not.

Value change callbacks are not all memorized by the hub, due to lack of memory. Only one value change callback per function is emulated by the VirtualHub (for Web), when the HTTP callback is received.

Limitations


Implementing a Yoctopuce module emulator in PHP, based on simple HTTP callbacks, involved a lot of tricks, visiting never before exploited corners of our HTTP callback API. As a result, we had to make some fixes/improvements to the firmware and Yoctopuce libraries to make this magic possible.

So to be able to take advantage of it, you have to update your YoctoHub firmware (or update your VirtualHub) to version 51900 or higher, and the other Yoctopuce modules to firmware 50730 or higher. If you want to connect to a VirtualHub server (for Web) using Yocto-Visualization or Yoctopuce programming library, use version 51900 or higher.

When you use the VirtualHub (for Web), don't forget that the application of the settings is delayed to the next HTTP callback, which necessarily implies a slightly different behavior than a direct connection: for example, if you change an attribute and read back its value directly, you get the previous value until the time when the setting is actually applied to the module and transmitted back to the VirtualHub (for Web).

Some complex features are not yet emulated by the VirtualHub (for Web). It is the case for the remote update of modules firmware, and the access to the communication buffer of serial transmission modules (Yocto-Serial, Yocto-RS232, Yocto-RS485, ...). With a little luck, these features should appear progressively - as far as possible.

Comparison with GatewayHub


GatewayHub, which we have been offering since 2017, does not store data on hubs. It offers a direct connection to a single YoctoHub or VirtualHub at a time, but only acts as a gateway. Therefore, it does not enable your to retrieve data from hubs that use the sleep function to save power.

Also, because of the WebSocket technology it is based on, GatewayHub cannot be installed on standard PHP hosts: GatewayHub needs a Node.js host. We thought in 2017 that these would become more democratic with the rise of Node.js, but five years later, we see that PHP is still the most common and easiest type of host to implement. So better late than never, we set things right... Eventually, it is likely that VirtualHub (for web) will replace GatewayHub.




1 - mtf Saturday,december 03,2022 12H19

Thanks for this tool, it really looks awesome. Unfortunately, it seems to depend on being able to use a .htaccess file on the server, which as far as I understand is specific to Apache. Could vhub4web be made webserver-agnostic?

This is the error I am getting when attempting an installation on a server using Caddy webserver:

---

Oops, we have a problem...

installer-testdir/Subdir/Node/Test/RewriteRules caused an HTTP 404 Not Found tell me more
The file installer-testdir/.htaccess created by the installer did not produce the expected result. Most probably your Web server does not have mod_rewrite enabled, or AllowOverride is not set for this directory tree.This is a fatal error, as this software relies on URL rewriting to work properly.

---

2 - mtf Sunday,december 04,2022 0H49

I tried that in my Caddy configuration:

```
yoctovh.domain.tld, www.yoctovh.domain.tld {
tls email@domain.tld
root * /var/www/yoctopuce/virtualhub4web
encode gzip
php_fastcgi unix//run/php/php8.0-fpm.sock {
try_files /index.php?node={path}
}
file_server
}
```

With the `php_fastcgi` block being equivalent to the content of the `.htaccess` file created in the test directory by `vhub4web-installer.php`, but I am still getting the same issue. I am not sure which tests are being made by the installer, but this configuration should theoretically take care of the rewrites yet doesn't pass the tests.

I can maybe move my questions to an issue on the Github repository, perhaps it would be more appropriate.

3 - mvuilleu (Yocto-Team)Thursday,december 08,2022 17H28

@mtf: If you really intend to run the installer in the root folder of your Caddy server, the proper rewrite configuration should be:

@vhub4web {
  path_regexp static ^(/[^/]*)/(.*)$
}
rewrite @vhub4web {re.static.1}/index.php?{query}&node={re.static.2}

But as rewrite rules in Caddy are only global (no per-dir rewrite rule), this will not allow your server to be used for anything else. If want to use your server for other purposes as well, use the following rewrite rule that applies only to subdirectory VirtalHub-4web, and run the installer in that subdirectory:

@vhub4web {
  path_regexp static ^(/VirtualHub-4web/[^/]*)/(.*)$
}
rewrite @vhub4web {re.static.1}/index.php?{query}&node={re.static.2}

After many attempts, I finally managed to run Caddy+PHP on my Windows dev machine to verify that this works (using win-fpm). I had to make a small fix in VirtualHub-4web code as Caddy does not define SERVER_ADDR, but apart from this it works OK. The fix will be available in next build.

If you still experience issues using Caddy, drop a mail to support@yoctopuce.com

4 - mtf Monday,december 12,2022 15H23

Thanks a lot @mvuilleu for investigating this issue! I was monitoring the Github repository and saw the latest commits indeed, but could not spend much time trying the update yet.

I just tried very quickly with the rewrite rules you posted above, but got the following error "/installer-testdir/Subdir/Node/Test/RewriteRules did not return valid JSON tell me more - Response was:". I guess I just failed to properly integrate the rules you posted into my existing caddyfile, will have to have a closer look at it. If I still fail after more scrutiny, then I will contact the support by email instead of cluttering the comments here (or on Github so that our attempts can be useful to others?).

Thanks again, looking forward to using vhub4web on my server!

5 - mvuilleu (Yocto-Team)Thursday,december 15,2022 11H54

@mtf We just published a new version where the installer will automatically tell you in the error message the exact rewrite rule that you should add to your Caddyfile

6 - mtf Friday,december 16,2022 12H51

I took another look at it and my mistake was a minor (and silly) one: I just needed to change the "path_regexp static ^(/VirtualHub-4web/[^/]*)/.*$" part of the rule into "path_regexp static ^(/[^/]*)/.*$", because my caddyfile was already serving /var/www/yoctopuce/VirtualHub-4web to yoctovh.domain.tld. Now I can install Virtualhub-4web fine. Thank you very much for having added Caddy compatibility. Exciting!

For reference, this is my full vhub4web block in the caddyfile, only with redacted domain and email:

yoctovh.domain.tld, www.yoctovh.domain.tld {
tls email.domain.tld
root * /var/www/yoctopuce/Virtualhub-4web
encode gzip
php_fastcgi unix//run/php/php8.0-fpm.sock
file_server
@vhub4web {
path_regexp static ^(/[^/]*)/.*$
}
rewrite @vhub4web {re.static.1}/index.php
}

(And I most likely don't need the "file_server" line.)

Yoctopuce, get your stuff connected.