XMODEM and YMODEM are protocols that enable reliable file transfers over a serial connection (UART). They were designed 50 years ago to improve the reliability of data transfers over analog telephone lines, and one might have expected them to have disappeared completely by now. However, there are still a few specific cases where they are widely used, which is why we have just added native support for these protocols in our serial communication modules.
Why XMODEM in the 21st century?
Compared to all other reliable methods of transferring data today (USB, Ethernet, and so on), the advantage of using XMODEM/YMODEM over a serial line is that it's a very lightweight solution to implement: it requires only two data wires, uses few resources, and is fairly simple to code, especially on the receiving end. It is therefore still commonly used by the bootloader of microcontroller-based products for firmware updates. This is particularly the case for STM32 processors, for which open-source XMODEM and YMODEM bootloaders are available.
This is the primary application we've targeted with our XMODEM/YMODEM support: sending firmware files to the bootloader of microcontroller-based products.
Implementation
The reference implementation of these protocols was written in C for UNIX and is quite heavily dependent on it. It cannot be directly ported to all of today's programming languages. There are other, more recent open-source implementations, but their quality varies widely, and they do not always adhere to the established standard.
We chose a more optimal solution: implement the protocols directly in the Yoctopuce device. The user's code simply needs to provide the file as-is to the Yoctopuce serial interface (for example, a Yocto-Serial-C or a Yocto-RS232-C), and the device handles everything on its own, calculating checksums/CRC values, handling retransmissions, and managing the protocol's state machine. The application code is thus completely relieved of the XMODEM/YMODEM issues, particularly the timeout constraints.
Usage
Since transmitting firmware over a serial line can take several seconds (or even several tens of seconds, depending on the situation), we have chosen to provide an API that allows you to monitor the progress of the transmission. The transmission code therefore looks like this:
USE_1K_BLOCKS = true # use XMODEM-1K extension
progress = serial.xmodemUpload(firmwareImage, TIMEOUT, USE_1K_BLOCKS)
while progress < 100:
# update your progress indicator here, if desired
progress = serial.xmodemUploadMore()
If an abnormal situation occurs (timeout or transfer cancellation by the receiver), the xmodemUploadMore method raises an exception with a corresponding error message.
During an XMODEM or YMODEM transfer, the receiver takes the initiative to start the transfer when it is ready. In the case of a bootloader, two scenarios may arise depending on the manufacturer's design.
In the first case, a specific command must be sent over the serial line to request the firmware download. Upon receiving this command, the device switches to update mode and requests the start of the firmware transfer until it receives the firmware. The code for sending the firmware therefore has the following structure:
serial.writeLine("update_firmware")
# send the firmware image
progress = serial.xmodemUpload(firmwareImage, TIMEOUT, USE_1K_BLOCKS)
while progress < 100:
# update your progress indicator here, if desired
progress = serial.xmodemUploadMore()
In the second case, there is no specific update command, but the device quickly checks at startup to see if an update is available: it systematically attempts to initiate a transfer and aborts if it does not receive an immediate response. Sometimes, this check depends on the presence of a high or low signal on a specific pin. To ensure that the transfer starts within the allotted time, the best solution is therefore to initiate the transfer before restarting the device, so that the Yocto-Serial-C is immediately ready to send the beginning of the file as soon as it is requested:
progress = serial.xmodemUpload(firmwareImage, TIMEOUT, USE_1K_BLOCKS)
# pull RTS line, assuming it is connected the pin triggering the bootloader
serial.set_RTS(1)
# reboot the device to trigger the firmware update
RebootCustomDevice()
# wait until update is completed
while progress < 100:
# update your progress indicator here, if desired
progress = serial.xmodemUploadMore()
# update complete, release RTS line
serial.set_RTS(0)
XMODEM or YMODEM?
Between Ward Christensen's first implementation of XMODEM in 1977 and the standardization of YMODEM ten years later, a multitude of variants emerged, some of which were incompatible with one another. In 1987, Chuck Forsberg clarified what was meant by the terms XMODEM and YMODEM:
- XMODEM supports only the transfer of the content of a single file
- YMODEM allows for the transfer of multiple files (batch mode), including sending metadata (file name, size, attributes)
The XMODEM/CRC variant, which verifies data integrity using a CRC, rather than a checksum, poses no compatibility issues since the receiver specifies which type of validation it desires. Yoctopuce serial modules support both CRC and checksum modes transparently.
The XMODEM-1k variant, which uses 1-KB blocks instead of 128-byte blocks, requires both parties to enable it simultaneously. That is why you must explicitly specify whether you want to use 1KB blocks when calling xmodemUpload(). XMODEM-1k always operates in CRC mode.
YMODEM transfers file contents according to XMODEM-1k, but adds metadata packets around the content so that the receiver knows the file name and size in advance, and so that you can chain multiple transfers. Here's how to send multiple files using YMODEM with the Yoctopuce serial modules:
for i in range(nFiles):
isLast = (i + 1 >= nFiles)
progress = serial.ymodemUpload(files[i], TIMEOUT, names[i], "", isLast)
while progress < 100:
# update your file progress indicator here, if desired
progress = serial.ymodemUploadMore()
Please note that some vendors may misuse the term YMODEM to refer to XMODEM-1k. If you only need to transfer a single file and the vendor specifies YMODEM, it may actually be XMODEM-1k.
If your firmware transfer doesn't start as expected, remember that you can always open the VirtualHub web interface to observe the data exchange between the Yocto-Serial-C and your device in real time. This can help you understand what's happening...
Conclusion
If you're using microcontrollers with a serial-line bootloader, these new methods should make it easier for you to set up your automatic update and testing procedures.
Note that there are other, even more advanced, protocols that followed YMODEM: YMODEM-g, ZMODEM, and so on. However, since they were not adopted for bootloader technologies, due to their increased complexity and lack of real benefit, we have not yet implemented them in the Yoctopuce serial communication modules.
