Insights lesenGespräch

AI skills

AI raises floors, not ceilings

AI Is a Tool: It Helps Experts and Makes Errors More Convincing

AI quickly brings beginners up to a decent level. Whether a result is actually right still depends on experience.

8 min read

Key takeaways

  • AI raises the floor: in studies, beginners and weaker performers benefit most, on tasks that AI handles well.
  • AI does not raise the ceiling: on tasks beyond its frontier, professionals using AI were correct far less often than those working without it.
  • The real risk: with AI, wrong work looks just as convincing as right work. Anyone who cannot judge the result will not notice.
  • Skill with AI means three things: knowing what is to be achieved, scoping the task properly, and checking the result.
  • For companies, this means: don't just hand out access, but set clear rules for review and sign-off.
Contents

A good camera does not make a good photographer. The idea is old, and for a long time it went without saying. Then came autofocus, automatic exposure and finally the smartphone. Today almost anyone can take sharp, correctly exposed pictures. The technology raised the floor.

Good photography still exists. But it no longer stands out through sharpness and exposure. It stands out through the idea behind the picture, the light, the moment and above all the selection: which of two hundred shots is the right one?

The same thing is happening with AI right now, only faster and in far more professions.

Why the tool comparison is only half right

People often say: a scalpel doesn’t make anyone a surgeon. A milling machine doesn’t make anyone an experienced machinist. That is true, but it misses part of the point. With traditional tools, a lack of skill usually shows during the work itself: the cut is off, the part doesn’t fit.

AI is different. It delivers a result that looks finished: a text, an analysis, a piece of code. That result can look professional even though it is wrong on the substance. This shifts the difficulty. Producing is no longer the hard part. Judging is.

A better comparison is therefore a camera on automatic mode, or a satnav. Both do a lot on their own. Both work well most of the time. And with both, only someone with experience notices when the result is wrong.

What the studies show: AI raises the floor

+43 %higher quality rating for the work of previously weaker participants, measured against their own baselineDell'Acqua et al., Harvard/BCG 2023
+17 %higher quality rating for previously stronger participants, on the same tasksDell'Acqua et al., Harvard/BCG 2023
+34 %more resolved inquiries per hour for newcomers in customer service, hardly any effect on experienced agentsBrynjolfsson, Li & Raymond, NBER

The best-known AI productivity study comes from researchers at Harvard Business School and other universities. They studied 758 consultants at Boston Consulting Group. On tasks that GPT-4 handles well, participants using AI completed more tasks, faster and at clearly higher quality. The biggest jump came from those who had previously performed below average: the rating of their work rose by 43 % compared with their own baseline, versus 17 % for the stronger performers.

A second study comes from customer service. More than 5,000 employees were given an AI assistant. Newcomers then resolved clearly more inquiries per hour. For very experienced employees, hardly anything changed. The researchers suspect that the AI passes the knowledge of the best agents on to the new ones.

At first glance, this contradicts the statement “AI doesn’t turn a layperson into an expert.” It evidently makes beginners better quickly. That is true, but only for part of the work.

Where it tips over: the invisible jagged frontier

The same Harvard study had a second part. Participants were given a task that lay outside the capabilities of the AI but looked similar to the others. The result: with AI, they were correct 19 percentage points less often than without it. They accepted answers that were convincingly worded but wrong.

The researchers call this the “jagged frontier.” AI is excellent at some tasks and weak at others that seem similar. Where exactly the line runs cannot be seen from the outside. You only see it if you already know what is right.

Two sides of the same technology

Inside the frontier
  • Routine and familiar tasks: texts, drafts, summaries
  • Good results likely
  • Big time savings, beginners catch up strongly
  • Review is still needed, but usually quick
Outside the frontier
  • New, complex or unusual tasks
  • Error risk rises, yet the result still looks convincing
  • Time savings uncertain
  • Thorough review needed: subject knowledge, sources, tests, plausibility
The frontier is invisible. Anyone who doesn't know where it is won't notice when it has been crossed.

Even experienced people are not safe from this. In early 2025, a study by the research group METR observed experienced software developers working on real tasks in their own projects. The study is small: 16 people and 246 tasks in large open-source projects, randomly assigned to be done with or without AI. With AI tools, they took 19 % longer on average. Afterward, they themselves estimated that the AI had sped them up by 20 %. The tools have evolved since then. The lesson remains: the feeling of being faster is not proof.

What generative AI skills and AI literacy really mean

So the question is not whether someone can operate AI. That can be learned in an hour. AI literacy means more: handing tasks over to AI sensibly, evaluating results critically, and taking responsibility for how they are used. Behind this are three skills that cannot be learned in an hour.

Three steps AI doesn't take off your hands

Clarify the goalWhat should be achieved in the end, and how can you tell?
Scope the taskWhat can the AI do well, and what stays with the human?
Judge the resultIs it right? Does it fit? What is missing?

Clarify the goal. AI answers the question that was asked, not the one that was meant. If you don’t know exactly what the end result should be, you get a lot of text and little outcome.

Scope the task. People who work well with AI divide up the work. The AI takes over drafts, variants, summaries and routine. The human keeps whatever requires experience and context. The Harvard researchers found exactly this way of working among the most successful participants.

Judge the result. This is the most important step. An AI-written text about payment methods can read fluently and still be wrong. An analysis can look clean and calculate with the wrong definition of “revenue.” Only someone who understands the subject sees this. A simple rule follows: if you can’t check a result yourself, you shouldn’t sign it off either.

AI can partly make up for missing expertise. But it does not replace the ability to reliably judge the quality of a result. People with little experience reach a convincing result faster with AI, without knowing for sure whether it is right. In short: AI shortens the path to a result. It does not shorten the path to judgment.

From hands-on experience

This website was itself built with AI, in a few days and with hundreds of changes. A behind-the-scenes report (in German) describes this in detail. Three experiences from it fit the thesis of this article exactly.

Cleanly built is not the same as fitting. For one project, a small animation was created on request. Technically it was flawless. One look made it clear: it felt dated and distracting. The AI could not tell. A human judgment settled it in seconds.

A second AI can be wrong too. An article was additionally reviewed by another AI system. The review criticized passages that had long since been revised. It had apparently read an older version. Anyone who had accepted the criticism without checking would have replaced correct sentences with worse ones.

The invisible errors are the most expensive. Despite a spam filter, unwanted messages kept arriving. The cause was not a gap in the filter but a forgotten test copy of the website that was still running. It was found only by closely examining the technical traces.

In this project, AI took over an enormous amount of manual work. What gets built, how it should come across and whether it is right remained human decisions.

What this means for e-commerce teams

In online retail, AI is already part of everyday work: product copy, translations, replies to customers, analyses, code for the shop. A few simple rules follow.

  • Review is work, not an afterthought. Anyone who signs off AI results needs time and expertise to do so. That time belongs in the plan.
  • Sign-off by impact. An internal draft needs less review than a text in the shop or a figure presented to senior management.
  • Support beginners. AI quickly brings new colleagues up to a good level. That is exactly why they need someone to proofread their results until they can recognize the limits themselves.
  • Skills are also an obligation. Since February 2025, Article 4 of the EU AI Act has required providers and deployers of AI systems to ensure a sufficient level of AI literacy among the people who work with these systems. Prior knowledge, experience and the specific use case all count. This is exactly what is meant: being able to assess opportunities and risks, not just press buttons. What data protection and the AI Act require beyond that is covered in the article AI tools in daily work.

Conclusion

AI is a powerful tool, but a special one. It takes over a lot of the craft and raises the floor for everyone. That makes the difference between decent and correct more important, not less. And that is precisely the difference the AI cannot recognize on its own.

Companies introducing AI should therefore not only ask who gets access. The more important question is who judges the results.

Frequently asked questions

Does AI make subject expertise unnecessary?

No. It shifts it. Less knowledge goes into producing, more into checking and deciding. Studies show that on difficult tasks in particular, a lack of expertise leads to more errors when AI is involved.

Do beginners or experienced people benefit more from AI?

On routine tasks that AI handles well, beginners measurably benefit more. On difficult or new tasks, only experience protects against accepting convincing-sounding errors.

How can you tell whether a task is suited to AI?

A simple rule of thumb: if the result can be checked quickly and reliably, the task is well suited. If nobody on the team can judge the result, AI should at most assist.

Is training in how to use AI tools enough?

It is a start. In the Harvard study, a short introduction to writing prompts added little. More important are clear rules for review and sign-off, and people who can judge results.

Sources

Inhalt

    InsightsIns Gespräch kommen