Productivity and Patience: How GitHub Copilot Expands Your Development Horizons

GitHub, the shared repository of choice for developers of all types, has transformed the playing field for AI-powered development with its Copilot offering. The organization estimates that many developers use the tool to generate between 30% and 50% of the code they ultimately put into production.

Recently, researchers at Scalefocus embarked on a four-month experiment with three teams to measure the value and feasibility of GitHub Copilot. Their conclusion was that Copilot “speeds up the development process, reduces time spent on routine coding tasks, and increases productivity. On average, its adoption increases productivity by up to 20%.”

Also: From AI trainers to ethicists: AI may make some jobs obsolete, but it will generate new ones

However, the researchers added that “some limitations were also observed; Copilot sometimes struggled with more complex problems, requiring humans to steer it toward the desired solution.”

Understanding Copilot’s mechanics and best practices “is essential to maximizing its potential,” they warned. “Recognizing that Copilot may not be equally relevant to all tasks is key to avoiding inefficiencies. Identifying where Copilot excels can greatly increase its value; recognizing where it falls short can save a lot of wasted effort.”

Copilot excels at “implementing simple, well-defined components in terms of performance and other non-functional aspects. Its performance suffers when dealing with complex errors or tasks that require deep expertise.”

Also: US to award 12 regional ‘Technology Centers’ to fund more American Silicon Valleys

GitHub itself estimates that the increase in developer productivity is about 55%, according to Mario Rodriguez, senior vice president of product at GitHub, in a recent interview with Sam Ransbotham of Boston College and Shervin Khodabandeh of Boston Consulting Group, published in the MIT Sloan Management Review.

But productivity isn’t the only metric driving GitHub’s intentions at Copilot. “We even try to ask a lot of our customers to really measure overall developer satisfaction,” Rodriguez said. “If developers are unhappy in the organization, not much happens, right? You want them to be happy. Talent is scarce, so you definitely want your developers to be happy.”

The key to unlocking happiness is that Copilot is designed to be responsive and empathetic. “Copilot is very patient,” he explains. “It doesn’t get upset when you ask it, ‘Do you know how to actually do X, Y, and Z in this codebase?’ or ‘I forgot how to save this registry. Can you just tell me the best way to do it?’”

As a result, Rodriguez continued, “People who use it become much more productive. Sometimes you see a difference in the quality of the code, and then you see a difference in their satisfaction as well, because they don’t have to write 10,000 lines of tests when Copilot can help them with that.”

Also: Is AI in Software Engineering Achieving an ‘Oppenheimer Moment’? Here’s What You Need to Know

Copilot’s biggest challenge is context, he noted. “Code and code development have a lot to do with the context that you’re dealing with. Are you working in a legacy code base or not? Are you working in COBOL, C++, JavaScript, TypeScript? There’s a lot of context that has to happen for the quality of that code to be high and for you to accept it.”

GitHub Copilot, the first Copilot to emerge as an AI-powered developer assistant, has been merged into a division called GitHub Next. Rodriguez said this focus on AI is a very different proposition for the GitHub team. Typical GitHub code is verified, tested, and deterministic: “We know if something works or not. We know if there’s a bug or not. We know how to fix, multiple times, that bug. We’ve even seen the bug we’re making as we’re developing the feature.”

On the other hand, large language models “are not deterministic,” he continued. “And you can’t just expect them to be deterministic. You have to rethink how you think about quality. You have to rethink how you think about features.” At GitHub, “we have all these offline evaluations and assessments that we do for GitHub Copilot. And that’s how we define the quality of what it does in something like code completion.”

Also: AI is transforming organizations everywhere: How these 6 companies are leading the way

Ultimately, at some level, everyone will have the opportunity to be a programmer, Rodriguez said. “The definition of a software programmer is going to start changing in the future,” he said. “We probably all know how to cook, or we all know how to ride a bike, or we all know how to swim, but look, I’m not going to open a Michelin-starred restaurant tomorrow.”

AI’s impact on software development will be more subtle: “What if a text box was all they needed to be able to accomplish something that software creates and something they could derive value from?” For example, Rodriguez said, “If I could say very quickly into the phone, ‘Hey, I’m thinking about talking to my daughter about these topics. Can you give me the last three articles X, Y, and Z, and then just create a little program that we could play as a game?’ You can imagine Copilot being able to help you with that in the future.”

He goes on to say, “Imagine being able to share this little program that Copilot created for you. And you can remix it, and remix it, and remix it. Artists do this all the time, by the way. I think tools like Copilot will help you do that. It’s going to be incredible what our kids will experience as this technology evolves.”

Source link