There is a point in an AI demonstration when a document appears almost instantly, somebody mentions productivity, and a job description begins to look nervous.
The document is impressive. The turnaround is impressive. The leap from ‘it produced that’ to ‘we no longer need the person who does this’ deserves a little more attention.
Preferably before somebody puts a savings figure in a spreadsheet. Once a number gets a colour and a formula, it can acquire a confidence its parents never intended.
I use AI extensively. It helps me write, research, build things and pursue ideas that probably should have remained a brief thought over coffee. I am interested in what it can genuinely take on, including work we currently regard as too complicated, too contextual or too human to automate.
But I want a test that can survive contact with the actual job.
My starting point is four questions: is the proposed process Better, Faster, Cheaper and Safer? Then come two less comfortable questions: compared with what, and under which conditions?
Start with the work you can describe
‘Can AI replace an analyst?’ is a very large question wearing a very small sentence.
Which analyst? Doing what? With which information? For whom? What happens when the request is ambiguous, two sources disagree, or somebody needs to be told that the answer they want is unsupported?
A role can include producing an artefact, finding the information behind it, negotiating what matters, spotting an exception and taking responsibility for what happens next. A function adds dependencies between people and systems. Completing one visible task gives us evidence about that task. We still have to assess the rest.
Start somewhere specific: extract the required fields from these documents; classify these incoming requests; prepare a first draft against this brief.
Then define a completed outcome. A draft sitting in a chat window is a different outcome from an approved response delivered to the right person.
That distinction is where I would begin the benchmark.
First, pass the requirements
Before comparing benefits, set the minimum standards the process must meet.
These might include required accuracy, permitted use of information, acceptable consequences of failure, a named owner and a workable recovery process. Some will be organisational choices. Others will come from obligations the organisation cannot simply mark down because the software is inexpensive.
I would treat these as gates. If the proposed process fails a required standard, it needs to change before proceeding.
This is different from giving safety a small slice of a weighted score. A sufficiently large saving should not be able to average away an unacceptable risk.
‘Yes, it discloses confidential information occasionally, but have you seen the turnaround?’ is not the argument I want on the approval paper.
Passing the gate does not mean the process is risk-free. It means we have defined an acceptable standard, tested against it and made the ownership clear.
Then, prove the performance
Compare the current process with the proposed AI process, including the people it needs. Give both the same outcome definition and acceptance standard. Use representative work, including awkward cases, and record the failures as carefully as the successes.
The baseline needs to be honest too. Comparing AI with a supposedly flawless human is convenient if you want AI to lose. Comparing it with a deliberately inefficient process is convenient if you want it to win. Neither tells you much about the decision in front of you.
Better: would you actually accept the result?
Measure accuracy, completeness, usefulness and consistency. Record how much correction is required before the output is fit for purpose.
Good formatting belongs somewhere in the assessment, but it should not be allowed to impersonate correctness. A beautifully structured answer can still contain a beautifully structured mistake.
Where judgement is involved, define the assessment criteria before reviewing the outputs. If practical, have reviewers assess them without knowing which process produced them. That gives the work a chance to be evaluated without the logo doing half the talking.
Faster: when is the work actually finished?
Time the journey to an accepted outcome. Include preparation, waiting, review, correction and the handling of exceptions.
Also distinguish elapsed time from human effort. A process can deliver sooner while requiring more staff attention, or reduce effort without changing the delivery time because the approval queue remains the same.
Five seconds to generate something is useful information. It is not the full measure if somebody then spends 20 minutes checking it.
The stopwatch needs to stay running after the applause.
Cheaper: what does a usable outcome cost?
Count the costs of the proposed process: tools, integration, information preparation, review, maintenance and dealing with failures. Compare them with the costs of today's process over a stated period and a realistic volume of work.
Use cost per accepted outcome, with unsuccessful attempts included in the costs. Otherwise, the process can appear wonderfully cheap by producing work nobody can use.
And distinguish capacity released from money saved. Giving a person time back can be valuable. It does not automatically remove that person's salary from the budget.
A business case should say what happens to the released time. More work completed? Better service? Less overtime? A redesigned role? Actual expenditure reduced?
‘Efficiency’ is doing quite a lot of unpaid explanatory work if nobody answers that question.
Safer: what happens when it goes wrong?
Measure failure frequency and severity separately. Consider who could be harmed, what information could be exposed, whether failures can be detected and whether the process can be stopped or corrected.
Fewer errors is encouraging. It is not sufficient if the remaining errors have more serious consequences.
Human oversight should be tested too. Can the reviewer see enough to recognise a problem? Do they have time and authority to intervene? Is escalation possible before the outcome becomes difficult to reverse?
A person appearing in the process diagram is a start. Their ability to do something useful is the part that earns the box.
These measures are not equal
Once the minimum requirements are satisfied, weight the benefits according to the task.
For an internal first draft, turnaround and reduced effort might carry substantial weight once quality is acceptable. For a consequential decision, accuracy, traceability and the consequences of error may deserve much more attention.
There is no universal percentage split I would trust across both.
Set the weights before seeing the results. Keep the underlying measures visible, even if you use an overall score. And show the variation: ordinary cases, difficult cases and the cases where the process failed.
An average can tell you where to look. It should not be allowed to hide the hole you are about to stand in.
The infographic is a framework for building that benchmark. It is not a validated scoring instrument or a forecast of job losses. The thresholds, sample, weights and evidence still have to come from the work.
Finally, check the whole job
Before making claims about a role or function, I would ask about three things.
Coverage: How much of the actual work can the proposed process complete? Count the preparation, coordination and exceptions, not just the outputs that are easy to demonstrate. Be clear whether coverage means a share of tasks, time or workload. Those are different denominators.
Reliability: Does performance hold across different inputs and changing conditions? Define when to reassess it, including after a material change to the model, workflow or source information.
Accountability: Who handles unresolved judgement, owns the outcome and recovers from failure? If substantial work moves to a reviewer, that work belongs in the proposed operating model and its costs.
An illustrative example: a team tests AI on routine customer enquiries. It produces strong drafts quickly and reduces the effort required to answer them. That could support an assisted workflow. With further evidence and suitable controls, some bounded requests might be handled automatically.
Neither result establishes that the entire customer service function can disappear. Complaints, unusual circumstances and disputed decisions still need an explicit plan. Equally, their existence does not prove that every routine request must remain manual.
The evidence should determine the scope.
Give the result somewhere honest to land
The assessment needs more than a yes or no.
Keep the work human-led where the requirements are unmet or the case is weak. Assist a person where AI improves the work but still needs substantial judgement or review. Automate a bounded task where the evidence and controls support it. Consider redesigning a role or function when the broader work, dependencies and ownership have been assessed.
‘Not enough evidence yet’ is also a legitimate finding. It should come with a useful next test, rather than becoming a permanent parking space.
None of this predicts what an organisation will choose to do. Demand, economics and organisational choices can lead to different staffing outcomes even where the technical results are similar.
My suggestion is to pick one recurring task and write down the current baseline, the acceptance standard and the consequences of getting it wrong. Test the proposed process against ordinary work and difficult exceptions. Include every person who has to make the result usable.
Then take the findings into the meeting.
The spreadsheet can come too. It just needs to show its working.