Sensaka Data Center MonitoringSensaka
    Back to BlogAnalysis · AI-era IT jobs

    AI productivity, measured: −19 % to +22 %, not 10×

    Controlled studies of experienced IT people land between 19 % slower and 22 % faster. Vendor slides say 90 % and more. Here is what the measured numbers say, and what to plan with.

    October 2026 6 min readDa Li

    Part of the AI-era IT jobs series. Start with CIO and IT manager jobs grew 26 %. The managers under them got cut..

    Table of Contents
    1. What the controlled studies measured
    2. The gains go to the people who know least
    3. The jagged frontier: same tool, opposite results
    4. Why the vendor numbers are so much bigger
    5. What this means if you are an IT manager planning headcount
    6. Sources

    In early 2025 sixteen experienced open-source developers worked on 246 real issues in their own repositories, with and without AI tools, in a randomised trial. With AI they took 19 % longer. Before the study they expected to be 24 % faster, and afterwards they still believed they had been 20 % faster [METR, 2025, p.1]. That is the low end of the controlled evidence for experienced IT staff. The high end is Microsoft's trial of experienced security analysts, who were 22 % faster with Copilot [Microsoft, 2024, p.6]. Vendor claims start at 63 % and run to 98 %. Nobody has measured anything close to 10×.

    // 01

    What the controlled studies measured

    I spent 15 years at enterprise IT vendors and I know how a productivity number gets onto a slide. For this post I went back to the studies that had a control group, and to the pages where their numbers are printed.

    Range chart: measured effects of AI from controlled studies compared with vendor headline claims
    Measured effects (blue) against vendor claims (orange). Sources: METR; Humlum & Vestergaard; NBER w31161; Microsoft; Dell'Acqua et al.; Peng et al.; MSSP Alert; ServiceNow; TechInformed; Palo Alto Networks.

    The measured range is narrow once you look at who was studied and on what:

    • Customer-support agents at a Fortune 500 software company (5,179 of them) resolved 14 % more issues per hour with an AI assistant [NBER w31161, 2023, p.2, p.4].
    • Consultants at BCG finished tasks 25.1 % faster with GPT-4, on tasks picked to suit the model [Dell'Acqua et al., 2023, p.4].
    • Experienced security analysts were 22 % faster and 7 % more accurate on four Defender tasks. Microsoft ran that trial on its own product [Microsoft, 2024, p.4, p.6].
    • The famous Copilot number, 55.8 % faster, comes from a single greenfield task: writing an HTTP server in JavaScript. The 95 % confidence interval runs from 21 % to 89 % [Peng et al., 2023, p.1–2].

    The study I trust most for Europe comes from Denmark. Researchers linked surveys of about 25,000 workers in 11 AI-exposed occupations to the national registers [Humlum & Vestergaard, 2025, p.2]. Users reported saving 2.8 % of their working hours. The effect on earnings and recorded hours was zero, with confidence intervals ruling out anything larger than 1 % [Humlum & Vestergaard, 2025, p.2, p.4]. The authors point out that trials in the same occupations often showed gains above 15 % [p.4], so a lot gets lost between a trial and an ordinary working week.

    DORA's 2024 survey model points the same way for delivery teams: every 25 % increase in AI adoption went with an estimated 1.5 % drop in delivery throughput and 7.2 % in stability [DORA, 2024, p.40]. The 2025 edition reported throughput rising with adoption. It also found more than 80 % of people believing AI had made them more productive [DORA, 2025, infographic]. That is a self-assessment, the same kind the METR developers got wrong.

    // 02

    The gains go to the people who know least

    Every study here that split its sample by skill found the same pattern: AI helps novices a lot and experts a little.

    Dumbbell chart: AI effect for novices versus experienced workers in the same study
    Each row is one study, same task, two skill groups. Sources: NBER w31161; Microsoft Copilot for Security whitepaper; Dell'Acqua et al.; METR.

    Novice support agents gained 34 % in issues resolved per hour, while the most experienced agents saw "minimal impact" [NBER w31161, 2023, p.2]. Agents with two months of tenure using the tool performed as well as agents with more than six months who did not have it [p.4]. Among security analysts, novices became 35 % more accurate and professionals 7 % [Microsoft, 2024, p.10, p.4]. Among consultants, below-average performers improved 43 % and above-average performers 17 % [Dell'Acqua et al., 2023, p.4].

    That should worry anyone cutting junior hiring, because the tools do most for exactly the people companies have stopped hiring (junior IT jobs in Europe vs the US covers that squeeze). An agent who performs at two months like a six-month colleague is an argument for keeping the junior intake.

    // 03

    The jagged frontier: same tool, opposite results

    The BCG researchers coined the phrase "jagged technological frontier". AI is very good at some tasks and bad at others that look just as hard, and you cannot see the line until you cross it.

    Diagram of the jagged frontier: tasks where AI made people faster and tasks where it made them slower or wrong
    Controlled-study effects by task. Sources: Dell'Acqua et al.; Microsoft; NBER w31161; METR; Peng et al.

    On a business case built to mislead the model, consultants without AI got the right answer 84.5 % of the time and those with AI 60–70 %, a drop of 19 percentage points [Dell'Acqua et al., 2023, p.4, p.18]. In Microsoft's own trial, the experienced analysts wrote incident summaries 46.2 % faster but chose response steps 26.3 % more slowly [Microsoft, 2024, p.7]. METR's developers slowed down most on the issues they knew best, and the screen recordings show where the time went: prompting, waiting for output and reviewing it [METR, 2025, p.9, p.11].

    For an IT manager this means checking, task by task, which side of the line the work sits on. The METR result shows that asking people how it feels will not tell you.

    // 04

    Why the vendor numbers are so much bigger

    The vendor figures measure a different thing from the studies: the share of a queue or of the clock that a product took over, as the vendor counts it.

    ServiceNow says its agents now resolve 90 % of IT requests, inside ServiceNow's own operations [ServiceNow, 2025]. Klarna said its assistant did "the equivalent work of 700 full-time agents" in February 2024 [Klarna, 2024]. Fifteen months later the company was hiring human agents again [Entrepreneur, 2025]. Google says its triage agent cuts an alert review from 30 minutes to 60 seconds [TechInformed, 2026]. CrowdStrike claims its triage removes more than 40 hours of manual work a week [CrowdStrike, 2025], and Palo Alto Networks claims up to 98 % less time to respond [Palo Alto Networks, 2025]. None of these publishes a control group, an error rate or the time people spend checking the output.

    The one controlled study of an agent in a security operations centre is useful here. With a phishing-triage agent, 167 analysts found up to 6.5 times as many true positives per minute, and the authors put 83 % of that gain down to the agent ordering the queue rather than deciding the verdicts [Microsoft, arXiv 2511.13860, 2025, p.1–2]. The large gain came from one narrow job, putting the queue in order.

    Back at company level, only 13 % of CIOs say their enterprise has seen significant or transformational value from AI [Gartner, 2026]. In McKinsey's 2026 survey, 80 % of respondents said AI had improved their own productivity, while 37 % could point to any effect on profit, about the same share as a year earlier [McKinsey, 2026]. The engineers stay sceptical: among Stack Overflow respondents, the share who distrust AI output rose from 36 % to 58 % in Germany and from 38 % to 52 % in Poland between 2023 and 2025 [Stack Overflow survey microdata, 2023–25].

    // 05

    What this means if you are an IT manager planning headcount

    1. Plan with 15–25 % on the right tasks. That is the planning figure three of our chapters reached independently, for documentable, repetitive work done mostly by less experienced staff [synthesis report, 2026]. Put the cost of reviewing machine output in the same plan.
    2. Measure with a control group before you cut. Give the tool to half a team for a quarter and compare cycle time, reopened tickets and escalations with the other half. Self-reported time saving is the number that fooled the METR developers.
    3. Keep the junior intake and give it the tools. The largest measured gains are for novices. A team with no juniors gets the smallest effect and no one to grow into the senior seats.

    The measured gains favour narrow, repeatable work, and the narrowest in a data centre is checking hardware health, which hardware monitoring takes off the team's hands.

    // Sources

    Sources

    See hardware, power and cooling health from outside the operating system. Request an online trial and explore how Sensaka brings hardware, operations and business services into one platform.

    Request an Online Trial →
    Core Solution Pillar

    Sensaka AI & GPU Infrastructure Observability

    Full-stack thermal, liquid cooling, and BMC hardware monitoring for high-density NVIDIA H100/B200 GPU racks and AI clusters.

    Explore GPU & AI Observability

    Related articles & analysis

    What's next for help desk and IT support people, 2027–2029Analysis · AI-era IT jobsHelp desk jobs after AI: postings down 49 % in the US and 70 % in the UK, headcount upAnalysis · AI-era IT jobsThe IT job market 2022→2026: openings collapsed, jobs mostly didn'tAnalysis · AI-era IT jobsWhat's next for CIOs and IT managers, 2027–2029Analysis · AI-era IT jobs
    All articles