The assistant produces a supplier comparison in minutes, but someone still has to check it. How do you measure the benefit across the whole job, including errors, review effort, and the cost of keeping it running?
One Operational Ecosystem, Part 5
The purchasing assistant now produces a supplier comparison in minutes. Buyers use it regularly. The manager has seen several impressive demonstrations, and another department wants access.
Before extending it, the manager asks a straightforward question: how much has it improved the work?
One buyer says it saves time searching. An engineer says she spends longer checking the recommendations. Procurement still follows up with suppliers to confirm terms. The support team has added a recurring task to keep the supplier records synchronized.
All of those observations can be true at once.
Our hypothetical purchasing assistant has reached the point where the business needs to assess its everyday contribution. The earlier articles examined its information, authority, and behavior during an interruption. This time, I want to look at the evidence that would justify continuing or expanding it.
Return to the job you wanted to improve
The original problem was finding a suitable replacement supplier before production ran short of a component. A faster comparison helps, but the team still needs to reach a usable purchasing decision.
I would define the outcome in terms the purchasing manager can recognize: suitable suppliers identified, requirements checked, approvals obtained, and orders placed in time to meet the need.
Then I would choose a small set of measures that show whether the workflow is improving. For this example, that might include staff time spent on a completed comparison, elapsed time from request to decision, corrections required, and purchases that meet the agreed specification and deadline.
Some outcomes take longer to observe. An order can look successful when it is submitted and still result in the wrong component arriving. The review needs to follow enough of the purchasing process to catch that distinction.
Usage is useful context. It tells us whether people are trying the assistant and where it fits into their day. It needs to be considered alongside the results of that use.
Establish a fair comparison
Before claiming an improvement, I would examine how the team performed similar work without the assistant.
That comparison should include ordinary purchases and the difficult cases the system is expected to handle. A repeat order from an established supplier is a different task from finding a replacement for an unfamiliar component under time pressure.
For a trial, I would record those differences and compare similar cases. If the assistant receives the straightforward requests while experienced buyers handle the exceptions, the results should make that division visible.
Other changes matter too. The team may have cleaned up its supplier register, introduced a standard comparison form, or shortened an approval queue during the same period. Those changes may account for some of the improvement.
Where practical, I would compare the AI-assisted workflow with the improved process using ordinary search and existing tools. That helps the business judge how much additional benefit the assistant contributes.
Keep the dates, case counts, and limits of the comparison with the results. A few successful purchases can justify further testing. They provide a limited basis for predicting performance across a whole department.
Count the work that moves to someone else
Consider a simplified example using invented figures. Preparing a supplier comparison previously took a buyer 40 minutes and an engineer another 10 minutes to check. The combined effort was 50 minutes.
With the assistant, the buyer spends 15 minutes preparing and reviewing the comparison. The engineer now spends 25 minutes verifying it. Combined effort falls to 40 minutes.
The buyer has saved 25 minutes, while the total saving across those two roles is 10 minutes. Both figures are worth knowing. They describe different effects on the business.
That calculation still leaves out supplier follow-up, correction work, and ongoing support. It also says nothing about how long the request waits between steps. A shorter task can still sit in an engineer's queue for two days.
I would therefore track staff effort separately from elapsed time. I would also ask each person involved what checking or repair work the assistant has introduced. Some of that effort may never appear in the system's activity records.
Time released is useful capacity. Whether it becomes a financial saving depends on what happens next. The team might clear a backlog, improve supplier assessment, reduce overtime, or simply have more room to handle urgent requests. Report the outcome that actually occurs.
Look at the errors behind the average
A comparison can need correction for several reasons. The assistant might use an expired quote, miss a requirement, confuse quantities, or present an unresolved question as settled.
I would separate minor presentation corrections from errors that could change the purchasing decision. An incorrect delivery date can matter more than several formatting mistakes, even though a simple error count gives them equal weight.
The review should also include requests that were abandoned, handed back to a buyer, or completed through the fallback process. Counting only the comparisons the assistant finished would leave out an important part of its operating record.
For completed work, examine how much reviewers changed and why. A high acceptance rate deserves a closer look if people are approving recommendations without enough time to check them. Likewise, frequent escalation may be appropriate when the assistant encounters genuinely incomplete information.
A period with no reported incidents is encouraging, but it does not establish that all the controls work. The boundary tests from earlier articles still matter. Measurement should help explain performance in use while testing examines cases that may not have occurred yet.
Include the cost of keeping it useful
In Chapter 5 of my Applied AI Handbook, I discuss assessing value over time and including the work needed to operate a system. For our purchasing assistant, that means looking beyond the model or software bill.
Someone maintains access to the business applications. Someone checks changes to supplier records, investigates failures, tests updates, and answers users' questions. The team may also need training, backup procedures, and periodic reviews of the assistant's permissions.
I would distinguish the initial setup effort from recurring costs. Within recurring costs, I would separate costs that remain fairly steady from those that grow with usage or difficult cases.
Cloud services and local equipment both belong in this accounting. A local deployment can involve hardware, power, maintenance, and support. A hosted service can involve subscription or usage charges, integration work, and operational support. The relevant figures are the ones required to run this workflow to the agreed standard.
Record significant interruption costs as well. The manual work described in Part 4 takes time, and recovery may require specialist help. Use observed costs where available and label estimates clearly.
Agree on the decision the evidence will support
Before the trial begins, I would ask the outcome owner to define what would justify continuing, changing, expanding, or stopping it.
For this workflow, an acceptable result might be a sustained reduction in combined staff effort while maintaining purchasing quality and keeping support demands manageable. The business should set its own thresholds based on the job and its consequences.
Some requirements should remain firm conditions. Faster comparisons would not justify orders being submitted without the required approval or information being disclosed to the wrong person. A breach may require an immediate pause and investigation rather than waiting for the scheduled review.
At that review, the manager should have the operating results, costs, unresolved issues, and feedback from the people doing the work. The decision could be to retain the current scope, fix a particular weakness, or run a further bounded trial.
Expansion deserves its own assessment. Another department may have different suppliers, access rules, volumes, and review capacity. The current results provide useful evidence, but those differences still need to be tested.
If the assistant produces little benefit, narrowing or stopping its use should remain a practical option. The time already spent building it cannot tell the team whether the next month of operation is worthwhile.
Review a set of completed jobs
For a team already using AI, I would begin with a recent set of comparable requests. Follow each from the original need through the decision and, where possible, the resulting purchase.
Ask how much effort each role contributed, what had to be corrected, which requests fell back to manual handling, and what the assistant cost to support. Check whether the result met the business need. Record what you cannot yet establish.
The purchasing manager may find that the assistant helps most with routine comparisons and creates too much checking for specialist components. That would support a narrower scope and a specific improvement plan.
They may find a clear benefit across the workflow. In that case, they can explain what improved, for whom, and at what cost, with a basis for checking whether those gains persist.
Choose one AI-assisted task in your business. Follow the work far enough to see who benefits, who picks up additional effort, and whether the final result is better. That is the evidence I would want before asking the system to do more.
Further reading
- Dom Jocubeit, Applied AI Handbook: Implementing AI Without Breaking the Business, the Law, or Trust, Chapter 5, “Economics, ROI, and Cost Accounting,” pp. 47-52.