Artificial intelligence is accelerating software development at breakneck speeds, but measuring it is difficult

Software development and deployment cycles continue to accelerate, thanks in large part to artificial intelligence (AI) that helps generate code and provides real-time suggestions. Even with this hyper-productivity, IT managers and business leaders still don’t know how to measure the impact of AI.

This is according to a new survey conducted by GitLab among 5,315 managers and IT specialists regarding software development productivity and DevSecOps. AI-powered development is now the norm, with 78% of respondents saying they currently use AI to build software or plan to do so in the next two years, up from 64% in 2023, the survey confirms. Additionally, 67% say their software lifecycle is now mostly or completely automated.

Also: From AI trainers to ethicists: AI may make some jobs obsolete but create new ones

The introduction of artificial intelligence can accelerate software development at breakneck speeds. Surprisingly, most managers (69%) say they deliver software twice as quickly as last year. Moreover, it takes more time to prepare IT specialists for current tasks. More than half, 52%, say it takes more than three months to onboard new developers – up from 42% a year ago.

Senior managers are much more distrustful of AI than their employees. The majority of managers (56%) say that introducing artificial intelligence into the software lifecycle is risky from a privacy and data security perspective. However, only 40% of professionals have such concerns.

Managers are also more concerned about AI skills, with 35% saying they lack the right skills to apply AI or interpret AI results as a barrier to using AI. Only 26% of IT specialists agree with this opinion.

Respondents currently using AI to build software (43%) were much more likely than respondents not using AI (20%) to say that onboarding developers typically takes less than a month. The same effect was seen when using the DevSecOps platform, with 44% of people currently using the platform saying it takes less than a month to onboard developers, compared to 20% of those not using the platform.

The survey shows that the most popular use of artificial intelligence in IT shops is generating code and providing explanations of how it works. In their future jobs, most would like AI to help them achieve forecasting and productivity metrics.

Plus: Meet your new IT superhero: Citizen developers flex their AI muscles

How artificial intelligence is used in development

• Code generation and code suggestion/completion, 47%
• Explanations of how the code snippet works, 40%
• Code Change Summaries, 38%
• Chatbots that enable users to ask questions in documentation using natural language, 35%
• Code review summaries, 35%

What IT professionals and managers want to see in AI

• Forecasting productivity metrics and identifying anomalies throughout the software lifecycle, 38%
• Explain how a vulnerability could be exploited and remedied, 37%
• Chatbots that allow users to ask questions in documentation using natural language, 36%
• Suggestions about who can review code changes, 34%
• Fixing failed pipeline jobs, 31%

Software supply chain security is a potential weakness – 67% of professionals say at least a quarter of the code they work on comes from open source libraries. At the same time, only 21% of organizations currently use a software bill of materials (SBOM) to document the composition of their software.

Also: Apple developers, rejoice! Your development tools just got a big, free AI boost

Management says developer productivity is a critical operational metric. However, many people are not sure how to measure it. Just over half of executives, 51%, say their current methods of measuring developer productivity are flawed or want to measure it but don’t know how. At least 45% admit they don’t even compare developer productivity to business results.

Most executives (55%) agree that developer productivity is important, and 57% agree that measuring developer productivity is key to growing their business. However, only 42% currently measure developer productivity in their organization and are satisfied with their approach. More than a third (36%) believe their methods of measuring developer productivity are flawed, and 15% want to measure developer productivity but are unsure how to do it.