Is the popular AI tool really worth the price?

AI models have only come about because of the great programmers who spend years writing code for individual systems. The question now is how well such AI code generators compare to human programmers.

Thanks to a new study published last month, we can learn more about how codes generated by well-known AI tools like ChatGPT compare to those generated by humans.

The authors looked at complexity and security and where they could be used. It is safe to say that the results had a wide range of success in terms of functional codes. Interestingly, the codes had accuracy ranging from 0.66% to 89%, all depending on the complexity of the task.

In most situations, an AI generator could produce better code than the average human mind, but this has not been without serious concerns from experts.

A leading expert from the University of Glasgow has shed light on how AI code generation can deliver important productivity benefits and automate the software development process.

That’s why it’s so important to realize how the advantages and disadvantages of such models change over time.

A comprehensive analysis revealed just how many problems are associated with the popular AI chatbot. And that’s where the authors sat down to reveal the flaws in detail.

What is interesting is that a programmer who spent decades creating code is now being replaced by an AI generator that can do it in a matter of seconds (but again, to a point).

The results of the study showed that the functional codes were very successful, and a more in-depth analysis showed that discovering these flaws could become easier.

Generally speaking, ChatGPT did a great job of solving problems in different coding languages. This was very true for solving problems related to LeetCode before 2021.

When algorithm problems were addressed after 2021, the tool’s ability to generate correct code was compromised. This resulted in a lack of understanding of the true meaning of such queries, even if they were for easy problems.

The coding world is still evolving, so ChatGPT hasn’t been exposed to new problems yet. Remember, it can’t think like a human and can only solve problems it’s familiar with. This may explain why it can handle old coding problems better than new ones.

Some experts fear that ChatGPT may generate bad codes because it does not understand the meaning of algorithmic problems. The study proved that the tool does a great job with shorter execution times and lower memory overhead compared to the other 50%.

Meanwhile, the researchers investigated ChatGPT’s ability to fix encoding problems after receiving feedback. They randomly selected 50 encoding situations in which the tool made incorrect encodings simply because it didn’t understand the content in front of it and the problems it was grappling with.

ChatGPT is usually considered to be great at fixing bugs, but in this case the tool didn’t do a very good job of fixing the errors it caused.

Researchers found that the codes implemented by ChatGPT have several security flaws, such as missing null tests, but many of them can be fixed.

We can therefore conclude by saying that developers using ChatGPT for coding need to take the error factor into account and provide additional data to the AI ​​chatbot to get the best results.

Photo: DIW-Aigen

Read the next article:

• Can AI tools like ChatGPT experience feelings and record memories? This new survey has the answer

• New technologies are revolutionizing business process automation strategies