Five years after our first series of tests on the maximum number of USB sensors that we could use on a machine and on the efficiency of the different types of hubs to manage this type of traffic, we offer you a new series of tests to know whether the situation has evolved since 2016...
The key lessons from 2016 were:
- Single-TT hubs caused packet loss in case of heavy Full-Speed traffic
- The need to power all the hubs and to respect a tree topology
- The limit to 13 Yoctopuce modules for the Intel USB3 xHCI Intel Internet controller
- The limit to ~10 Yoctopuce modules on the Raspberry Pi 2
Let's now see how the context has evolved on these different points in five years...
The new hubs that we tested
The USB hubs in 2021
Nowadays, few people buy USB 2.0 hubs anymore, as 3.0/3.1 hubs became very cheap. Is the packet loss problem of Single-TT hubs a thing of the past? We bought different up-to-date hubs to test them:
We also included in our tests ExSys USB 3.0 hubs bought the last time round as they are still on the market and have a good reputation. As a reminder, they use the Genesys GL3520 chipset.
We note right away that there are only few different USB 3.0 chipsets: Genesys and VIA seems to have cornered the market. Microchip (previously SMSC) also commercializes a USB 3.0 chipset that we would have liked to test but, unfortunately, we haven't been able to find a commercially available product using it.
USB host controllers
Host controllers seem to have evolved more than the hubs themselves. In particular, the Intel XHCI implementation which dominated in 2016 and which drastically limited the number of USB devices that you could connect to a machine was replaced by new, more generous versions.
With the aim to test USB with a heavy load, we performed our tests with quite powerful machines, essentially a Lenovo Thinkpad X1 Extreme, 2nd generation, and a Fujitsu Lifebook U9310X. We also performed a few tests with a Fujitsu Celsius workstation, and on two much smaller machines: Lenovo ThinkCentre M75n, a small fanless computer, and on the Raspberry Pi.
One of the new features since 2016 is the generalization of the Thunderolt bus and of its USB C connectors. In the opposite to what you may think, it's not such good news for USB devices. Indeed, many manufacturers replaced root USB ports with Thunderbolt ports, on which you connect a "docking station" which is actually nothing more than a large hub with many devices. But often, such as with the Lenovo Thinkpad Thunderbolt 3 Dock Gen 2 docking station, this almost mandatory first hub stage already adds two levels of USB hubs. As USB allows no more than five levels of USB hubs, this limits the hubs that you can use to a maximum of three levels. Others must imperatively be connected on a USB A socket directly on the machine.
The first good news is that the evolution of USB controllers on the host globally increased reliability. The controller or the Windows drivers seem to better manage some communication or protocol errors. For example, when the Genesys GL850 chipset was unusable in 2016 with heavy Full-Speed USB traffic, it can almost go unnoticed these days. It's only in specific cases that we can make it fail, or by putting a USB analyzer to see that in reality errors occur that lead to transient disconnections that are then recovered by the controller in an almost transparent way.
By combining this reliability improvement with the increase of the number of USB endpoints on the host controllers, and the announced bandwidth of 5Gbps per port on USB 3.0, we hoped to pass all of our load tests with at least 50 modules without any difficulty anymore. Let's see if it's actually the case...
We used the same type of tests than five years ago: Full-speed USB modules (such as Yoctopuce modules) designed to send messages at each frame, and to which we send commands to make the led blink.
First, the test is run when all the modules have been listed, and we simply check that it lasts over time without reporting communication errors to the software stack.
Second, the software periodically triggers a restart of one of the modules to simulate a disconnection and reconnection, to check that the USB enumeration still works in load conditions, knowing that it was one of the weak points in the past.
The good new is that all the tested hubs passed the first load test. One of the most ambitious combinations that we tested contained:
- The Fujitsu Lifebook U9310X
- The Logilink UA0229 hub (10 ports)
- The Delock 64039 hub (13 ports)
- The IcyBox IB-AC6113 hub (13 ports)
- 34 Full-Speed test modules
In this configuration, the system declared 64 connected USB devices, among which 23 hubs. These 64 devices are the maximum limit supported by the USB controller of the Fujitsu Lifebook. It's already a lot better than in 2016, but you can see that the limit is not far away. The main reason is that each USB 3.0 hub actually counts as two devices: one USB 2.0 hub and one USB 3.0 hub. So when a 13 port hub is made of 4 USB 3.0 hubs, it costs 8 devices on its own. which means that for each 3 additional USB ports, you pay to extra devices... No wonder at this price that Intel was forced to raise the limit on the number of devices supported by its host controllers!
Note that on the Lenovo laptop, as there are two distinct USB host controllers the limit is twice as high. It's good to know that with some machines, you can still go higher.
With the same configuration, the second load test with module restarts shows the limits of USB 3.0. After a few seconds already, the USB stack starts to report communication errors and USB packets get lost. We could think that we were far from the limits: with our 34 modules which send 64 bytes each millisecond, we send 2MB per second, very far from the 5 Gbps announced by USB 3.0. But obviously, as was the case five years ago, there are resource allocation problems: the USB 3.0 system is able to stream high speed video, but not to manage 34 USB Full-Speed frames each millisecond.
Note that we weren't able to precisely determine where the weak link was between the controller and the hubs: whatever the combination of USB 3.0 hubs tested with about 30 modules, communication errors happen relatively quickly.
The only combination which resisted at full speed with more than 30 modules to our test with enumeration is the following:
- The Lenovo ThinkPad X1 Extreme 2nd gen, powered
- 12 USB 2.0 hubs in two trees (chipset SMSC USB2514BUI), powered, each connected to a separate USB host controller of the ThinkPad
- 36 Full-Speed test modules
The configuration which works best...
So the improvements of 2021 are not necessarily the USB 3.0, contrary to what we could have thought...
The Fujitsu Lifebook U9310X works very well with 20 Full-Speed test devices, but starts dropping packets at full load with 36 test devices. This is most probably due to the fact that it only has one USB host controller, as the same happens on the ThinkPad when the whole tree is put on a single USB host controller.
In the "small host" category, the ThinkCentre M75n, which only has one USB host controller, also works very well with 20 Full-Speed test devices.
We have run the same tests on a Raspberry Pi 3B+ using Linux armhf. The USB host controller is able to hold the load, but one can clearly see that the traffic is much slower than when running the same test on the Windows machines. The reason is that, in order to work around recurrent packet loss issues on small ARM machines, our support library automatically enables a packet acknowledgement mechanism for each packet sent. It is therefore significantly slower, but on the other hand, it works... even with 36 test devices!
We have not yet tried on the Raspberry Pi 4, as we did not have one available this week, but there is anyway not much hope. The USB Full-Speed support on this machine is a real pain, and even our packet acknowledge system cannot work around the issues. As of today, Raspberry Pi support team has not been able to fix the issue.
Improving host controllers enabled the considerable increase of Yoctopuce sensors that you can connect to a computer since our latest tests in 2016. But if you want a robust system able to manage many sensors, you must however take care of:
- the hub topology, to be as a tree
- the hubs power supply
- when using more than 20 devices, spreading them on multiple USB host controllers
- for more robustness, handling USB connections and disconnections in the software, with callbacks.
Finally, we can mention that you always have the option to overcome USB limitations with the connection of sensors through some YoctoHub-Ethernet, as an Ethernet bus better supports the load of many small frames than a USB bus. At the software level, it will make almost no difference to you to switch a sensor from USB to Ethernet since the Yoctopuce library supports both in an almost transparent way.