In 2024, we shared a brief overview of our experience using AI to code an application that used our modules. Since then, AI has made significant strides and has become even more widely accessible. We thought it would be interesting to take stock of these developments.
In our previous post on the topic, we had used AI ourselves to generate the code. This time, we took advantage of having an intern on board to conduct a similar experiment, but with a more neutral approach. We were particularly interested in getting feedback from someone younger who learned to program at a time when AI was already widely available. He has a background in electronics and has taken programming courses, but doesn't yet have much experience in software development.
So we asked him to write a program that controls a kettle to maintain the water at 75° C using a relay and a Yoctopuce temperature sensor. Rest assured, that wasn't all he did during his internship, he was also able to observe and contribute to the design of a new product. We simply took advantage of a day which wasn't too busy to use him as a "guinea pig."
Our intern could use any AI as long as it was free and accessible to everyone. He was also free to choose whatever technical solutions he wanted. We just asked him for his chat history so we could assess the difficulties he encountered and the effectiveness of the AI systems.
The good news is that, in just one day, he succeeded in this challenge using Claude and ChatGPT. He built a graphical Python application using the Flask framework, but, as we'll explain, it wasn't perfect. We won't disclose the content of the conversations, but here are a few observations we drew from them.
The errors
We'll start with a few remarks on the results the AIs produced.
As is often the case, the first version of the app the AI created didn't work. This is a pattern we've also experienced. When you ask an AI to build an application, the first version contains errors that prevent the application from running. This is all the more frustrating because, 80% of the time, these are small, silly mistakes, ones that are obvious to someone familiar with the subject but which can confuse a novice programmer.
For example, the first version of the application generated by Claude contained the line "from yoctopuce.cdll.yocto_api import YAPI, YRefParam," which is completely incorrect. The two correct options are:
- "from yoctopuce.yocto_api import YAPI, YRefParam" if the Yoctopuce library is installed via pip
- "from yocto_api import YAPI, YRefParam" in all other cases.
No documentation or example for our library mentions "cdll"; the AI simply came up with an answer that seemed plausible.
When you're familiar with the topic, the error is glaring and easy to spot, but for someone who's never used our library, it can be a huge time-waster. That's exactly what happened. Claude eventually corrected this error, but it took four or five back-and-forth exchanges with questions that were often irrelevant.
Strange solutions
Once this error was corrected, we were quite impressed with the result. The application looked well-polished, and the AI had added a rather attractive graphical interface.
While the result was functional, it was far from optimal. The app wasn't very responsive. We understood the problem when we inspected the code. The AI hadn't sought to optimize interactions with the Yoctopuce modules.
Let's take the code that turned on the kettle for x seconds.
Yoctopuce relays can be switched from one state to another using the "setState()" method, but in this case, that means the application must manually handle both transitions (turning the kettle on and turning it off). This works correctly, but the code is more complicated and less efficient. One version of the application would freeze for a few seconds because it would turn the kettle on, wait X seconds, and then turn the kettle off. During that time, the graph and the interface were no longer updated...
Furthermore, if there's an error in the program and it crashes, the kettle might remain in the "on" position.
The correct solution is to use the "pulse()" method. The "pulse()" function allows the relay to be switched to state B for a specified length of time before it automatically returns to state A. The transition is managed directly by the relay board, not by the monitoring computer. This has two advantages: it simplifies the code and ensures that the relay does cut off the power at the end of the delay (even if the program is interrupted).
This method is documented, used in our example programs, and was even the topic of a post in 2014. However, the AIs never thought to use it.
When our intern asked the AI to solve this interface issue that was causing a deadlock, the AI created a new version that used threads, which made the code even more complicated. Threads allow code to run in parallel, but they are a complex topic and are often less efficient. Furthermore, our Python library is not "thread-safe," which means the developer must manage concurrency issues to ensure that no two threads are using our library at the same time.
The AI did not address the concurrency issues, which means that while the program worked, there was a high risk for the application to crash if left running long enough...
What we noticed
In the end, more than the final result, what's interesting is how our intern arrived at that result and the conclusions we drew from it.
Comments
First, the generated code is very verbose. By that, we mean that the application's source code contains a lot of comments explaining what's happening. That's good, but in our case, it's a bit excessive and becomes counterproductive.
Not necessarily efficient
Even though the application works, the code is often not very optimized. We mentioned relay switching earlier, but this is something we've noticed more generally. If we take a closer look at the code generated by AI systems, there's very often a way to achieve the same result more efficiently or more simply.
Too complex
Another characteristic of AI tools is that they often use solutions that are too complex. In our case, it involved threads, but often they use external libraries for a very small feature. In the end, we're left with a solution that depends on several libraries or frameworks we're unfamiliar with, when it would have been wiser to implement the functionality again differently.
Hard to get them to change course
Once an AI has chosen an approach, it's generally difficult to get it to change course. For example, when we saw that the AI had decided to use threads, we suggested to our intern that it might be wiser to use the "pulse()" method. The AI praised him for his idea and followed through, but kept using threads. In the end, we ended up with an illogical solution, where the code uses threads (with all the complexity that entails) even though they're no longer needed.
Experience
AI doesn't make up for a lack of experience. This might be reassuring for senior developers. Experience gained over the years is always useful.
An experienced developer will have far fewer problems with AI because, if it goes in the wrong direction, they'll know immediately how to steer it back on track. They'll quickly be able to identify where AI provides real added value and where it's better to continue working the old-fashioned way.
Conversely, for a junior developer, AI is a double-edged sword. The risk is placing too much trust in AI and failing to question its technical choices. The final result will work, but it will most likely be suboptimal.
Conclusion
We're very divided on the subject of AI. On the one hand, it's a very powerful tool, capable of incredible things, as the latest news show. On the other hand, it seems to stumble over obvious things. This is also what we notice in our daily use of it. Sometimes it's mind-blowing, and we wonder if our jobs will even exist in five years, and then, right after that, it trips over its own feet on a simple problem....
