Before agentic AI, Vision Language Models (VLMs), and Computer-Using Agents (CUAs) became the centre of every enterprise automation conversation, I was running a fleet of more than 100 RPA bots at a global container shipping carrier. Their job was to solve one of the messiest reconciliation problems in the industry: detention and demurrage billing.
If you haven’t worked in shipping, detention and demurrage charges are essentially time-based penalties. A container sits too long at a terminal, or a customer keeps it beyond the agreed free time, and charges start accumulating by the day. Across thousands of containers, hundreds of terminals, multiple countries, and constantly changing operational events, the financial exposure becomes significant very quickly.
Looking back, what made this problem interesting wasn’t shipping. It was the shape of the data. The same pattern exists in almost every finance function today: invoices, supporting evidence, external systems, internal systems, and a reconciliation process sitting in the middle trying to determine which version of reality is correct.
What I didn’t realise at the time was that I wasn’t solving a single automation problem. I was solving three completely different automation problems under one business process.
Three Data Sources, Three Different Problems
Validating a single detention or demurrage invoice required data from three fundamentally different sources.
The first source was web portals. Terminal operators and carrier tracking websites all exposed operational events through browser-based interfaces. Every portal had its own layout, its own authentication flow, and its own way of presenting gate-in and gate-out timestamps. Most offered no public API. The challenge wasn’t processing data but finding it. In today’s language, this is Computer-Using Agent territory. The automation needs to understand what it sees on the screen and adapt when layouts inevitably change.
The second source was terminal systems, some of which were effectively green-screen applications accessed through terminal emulation. There was no visual complexity, no document understanding requirement, and no reasoning problem. Navigation was predictable and deterministic. You moved through fixed-position fields using predefined keystrokes. Classic RPA handled these systems extremely well because the environment rarely changed.
The third source was a collection of Windows-native transportation management and invoicing applications where disputes, adjustments, and billing actions were processed. These systems were structured but offered no API-first integration model. Automation interacted through the user interface, entering data, updating records, and creating financial adjustments.
Layered across all three were scanned invoices, rate sheets, supporting documentation, and customer correspondence. Every terminal produced different invoice formats. Some were PDFs, others scanned images. Tables appeared in different locations, terminology varied by operator, and exceptions were common. This wasn’t a navigation problem. It was a document understanding problem, which is exactly where Vision Language Models excel today.
When I look back now, the architecture can be described very simply:
- RPA handled deterministic structured systems.
- VLMs would have handled document understanding.
- CUAs would have handled unpredictable web portals.
The actual business process sat in the overlap between all three.
The Finance Pattern Hidden Inside the Shipping Problem
Remove the words container, terminal, and shipping and this immediately becomes a finance story.
At its core, detention and demurrage validation is an invoice audit process. A charge arrives. Finance must determine whether the amount being billed matches operational reality. The problem is that the invoice and the evidence supporting the invoice live in completely different systems owned by different organisations.
That pattern appears everywhere:
- Freight invoice reconciliation
- Accounts payable validation
- Cloud spend reconciliation
- Utility billing audits
- Procurement verification
- Expense management
- Supplier charge disputes
The shape is always the same:
- A document arrives.
- Supporting evidence exists elsewhere.
- Internal systems need updating.
- Someone must determine whether the charge is valid.
That single workflow spans documents, websites, and enterprise applications, which is exactly why a single automation technology rarely solves the entire problem.
What the 100-Bot Architecture Got Wrong
The original solution worked, but it scaled poorly. Every portal variation became a separate bot. Every invoice format became a separate bot. Every business process variation became a separate bot.
Before long, the estate consisted of more than 100 independent automations, each solving a tiny piece of the overall process. The maintenance burden became the real operational cost.
The lesson wasn’t that RPA was wrong. In fact, many of those automations were highly reliable. The problem was architectural. We treated every automation challenge as if it required a dedicated bot instead of recognising that multiple bots were really compensating for missing capabilities.
If I Rebuilt It Today
If I had access to today’s AI capabilities, I wouldn’t build a hundred separate bots. I’d build four capabilities and a single orchestrator.
- A document understanding capability powered by a Vision Language Model would replace dozens of OCR and template-specific invoice extraction bots. Instead of creating separate automations for every invoice format, one capability would extract the required financial data regardless of layout.
- One or two Computer-Using Agent tools would handle external portals that expose operational data but provide no API access. Their responsibility would be limited to gathering information and navigating systems, not making financial decisions.
- The deterministic RPA components would remain largely unchanged. Green-screen terminal navigation, structured data updates, and repeatable back-office actions are exactly where RPA continues to deliver the most value.
- Finally, an orchestration layer would coordinate everything. Rather than having 100 bots hardcoded to specific scenarios, an agent would determine which tool to invoke, collect evidence from multiple sources, and route exceptions to the appropriate human approver.
The result isn’t just fewer automation assets. It’s a fundamentally different maintenance model.
The Maintenance Math
The biggest lesson from running a large automation estate wasn’t about technology. It was about operational complexity. One hundred bots create one hundred points of brittleness. One orchestrator with a handful of specialised tools creates fewer moving parts, clearer ownership, and more reusable capabilities. That doesn’t automatically make the solution safer. In many ways, it makes governance more important.
As automation becomes more intelligent, every capability must be clear about:
- What it is allowed to do.
- What it is not allowed to do.
- When it should escalate to a human.
- How its decisions can be audited.
For finance processes, the goal shouldn’t be maximum autonomy. The goal should be appropriate autonomy. Use deterministic execution wherever deterministic execution is possible. Use reasoning only where reasoning is genuinely required. Most importantly, maintain a clear audit trail that explains what was observed, what evidence was gathered, and how a decision was reached.
Final Thoughts
Looking back, the lesson wasn’t that RPA was the wrong technology.
The lesson was that we were trying to solve fundamentally different problems with the same tool. Some tasks required structured execution. Some required document understanding. Some required the ability to navigate systems that were never designed for integration.
Today, we have names for those capabilities: RPA, VLMs, and Computer-Using Agents. What excites me isn’t that one replaces another. It’s that, for the first time, we have the ability to combine all three under a single orchestration layer. Ten years ago, that architecture would have saved me from managing a hundred bots.
Architecture Diagram: Modern Detention & Demurrage Invoice Validation
Simple takeaway: VLM understands the invoice, CUA gathers evidence, RPA updates systems, and the orchestrating agent brings everything together to make a finance decision.