Budget Data Architecture

Here's how a budget goes wrong at a mining company. Someone drafts a plan. It's half-baked, a what-if with placeholder numbers. Copies go out for comment, people tweak their copies, and somewhere in the shuffle one of them catches escape velocity: suddenly it's in hands it was never intended for, assumed to be correct in a way it never was. Corporate likes the numbers. The numbers go public. And now an entire company is being held to a plan nobody ever finished.

I've watched that happen at site after site after site. So had the colleague who called me in 2020. We'd worked together at Rio Tinto, on a special team that analyzed a wild number of mine plans, and we'd seen what plans do when there's no canonical version: operations runs one corner of the mine on one plan while finance forecasts from another, and corporate tells the public about another, and nobody can answer the only question that matters. Which one is the canonical one? Now he was at a major gold producer, and the question had followed him there.

The budgets themselves were good. Operations built them in XERAS, RPM Global's mining-specific financial modeling tool, which takes an engineering mine plan, the benches and trucks and shift schedules, and turns it into a full financial plan. But XERAS was a silo. The analysts who evaluated those budgets and greenlit the spending lived in a different world, on a modern Google BigQuery stack, and in the gap between the two worlds the plans multiplied. The fix was easy to state and hard to do: establish one canonical version of the budget, pipe it into the place where the downstream decisions got made, and get engineering, finance, and operations reading from the same page.

The client wasn't short on talent. They had internal experts on the planning tools, serious finance people, and a capable data engineering team. So often, the trouble internal teams have isn't capability. They just don't know the right things to ask for. My job was to know.

The first thing to ask for was the one thing nobody had: how XERAS actually stored the numbers. It sat on RPM's EPF platform, on SQL Server, and the schema underneath was undocumented, its original builders long gone from the vendor, so I had production access arranged and went at it empirically. Change one input in the planning tool. Open the database. Find what moved. Write it down. Change the next input. For a stretch of 2020 that loop was the job, and out the other end came documentation that had never existed: what each table held, how to interpret it, which queries extracted it, and when to run them so extractions caught the data at the right moments. There was one rule I couldn't break. This was a live production system, so I could read but not modify, which meant no new indices. That meant a bunch of fun shenanigans with virtual tables just to comb through that much data in any meaningful amount of time, with queries running against the tables exactly as they stood. This is literally why companies keep transactional and analytical databases separate. Unfortunately for me, in this case the transactional database had to serve analytical purposes.

Then came the decision I'm proudest of. The standard consultant move is to build the pipeline yourself. It's faster, it bills more, and it quietly leaves the client dependent on you, because the day you walk out, the understanding walks out too. I insisted on the opposite: their engineers would build it, and I would make them able to. We architected the ELT flow together, raw extractions landing in BigQuery on the cadences I'd mapped, transforms happening where the analysts lived. I wrote extractions and defined transforms alongside their team. It was my first time architecting a data pipeline in code rather than scoping one on a whiteboard, and every piece of it ended up in their hands, not mine.

Nobody should trust a new pipeline on the builder's word, so I reverse-engineered one more thing: XERAS's own reports, the numbers the planners already believed, and turned them into ground truth. An automated reconciliation harness compared the pipeline's output against production, run after run, and when the two disagreed we knew immediately and found out why. Data quality stopped being an argument and became a checklist.

The other half of a canonical plan is the people feeding it, so I coached the planners and forecasting teams on entering clean data at the source, across a portfolio that ran from operations with thousands of people down to a twelve-person seasonal pit. All of it landed during a period of consolidation, with layoff fears in the air, which meant a lot of the real work, across an engagement that ran over a year, was calling site personnel one by one and making a single thing clear before anything technical: I was there to help them, not to replace them.

What changed wasn't speed. It was direction. Before, the information diverged: every handoff spawned another version, and the versions drifted apart. After, it converged: one plan, extracted from one source, reconciled against ground truth, read by finance, engineering, and operations alike. The game of telephone ended because there was nothing left to pass around.

At least five years later, the system was still in use, run entirely by their own team. That's the outcome I designed for from the first week: not a pipeline with my fingerprints on it, but a client who understands their own data well enough to not need me. The best evidence the project worked is that they haven't.