Copper Mine Machine Learning

Machine learning and I got off to a rough start. When Sebastian Thrun launched his Introduction to Artificial Intelligence MOOC, I signed up like everyone else and quickly realized it was way over my head. So I stepped away and spent the next few years building the foundation I was missing.

By 2016 I had properly learned how to program, and machine learning was having its moment, a real swell of interest centered on the deep neural nets that were suddenly powering computer vision. I worked through courses like Andrew Ng's deeplearning.ai, and then stumbled onto Jeremy Howard and Rachel Thomas's fast.ai course and framework. That's when it finally clicked. I now had the lower-level knowledge to understand what was happening under the hood, but I didn't have time to build all the primitives it took to get something done. fast.ai was teaching a higher-level way to develop AI, with less data and less ceremony, and I was hooked. What I didn't have was a problem worth aiming it at.

Right around then my old friend Pratt reached out with an interesting data problem. Pratt had been my manager back in my MISOM days, we'd been friends and collaborators ever since, and he was now the principal investigator on a research project at the University of Utah. One of the copper majors wanted to take a boatload of operational data and see whether it could predict when the mine was at higher risk of safety incidents. If it could, supervisors could step in before someone got hurt instead of writing reports after. He was looking for someone to handle the engineering, and I jumped at the opportunity. The division of labor set itself. Pratt drove the science. I built the things that brought it to life, and I was the voice on what was and wasn't possible, how to validate a finding, how to justify it, how to structure it all. I told them from the start that we might not find strong predictors at all. I'm not a machine learning expert and I've never claimed to be. I was the best machine learning expert they could get, and part of that job was refusing to promise magic.

The first real job was getting my arms around the data. The mine sent over about twenty different sources covering five years of operations. Haul cycles, weather station readings, equipment time usage, fatigue monitoring systems, records of when operators clocked in and out. All told it came to more than twenty million rows, and much of it had been entered by hand at some point in its life, which meant it was full of surprises. My favorite was a field that had been silently broken for a year and a half without anyone noticing. The datasets were bigger than my machine's memory, so I built the processing pipelines in Dask and cached every transformation step, so that when I inevitably had to rerun things the iteration loop stayed fast. But the hardest problem wasn't size, it was shape. Operational logs are streams of events. An alarm fires here, a payload gets dumped there, and nothing happens on a nice even clock. Models want to know the state of the mine at a given moment, so a huge share of my time went into converting those events into states, doing things like reassembling individual haul-log events back into complete haul cycles, then rolling everything up to a day-and-shift level where the sources could finally line up with each other.

With the data in shape I could finally get to the modeling, and this is where 2018 shows its age. Deep learning was the hype of the moment, and fresh off fast.ai I wanted the neural nets to win. They didn't. Deep learning just wasn't ready for tabular data yet, and what actually performed was the older, less glamorous family of tools, random forests first and then gradient boosting. The decision I'm most proud of, though, wasn't a model at all. It was how we tested. Standard practice is to split your data randomly into training and test sets, but with time-based data like ours a random split lets the model peek into the future, and it will hand you back great-looking numbers that don't mean anything. Our whole question was whether the past could predict the future, so I made the test match the question. Train on the early years, test on the later ones. The scores came back weaker than a random split would have shown, but they were real.

And what those real numbers said was that every model kept landing on the same top predictor, the one nobody wanted to hear. It wasn't the weather, the equipment data, or even the fatigue monitors. It was labor hours. The more hours people worked, the more safety incidents the mine had. The client had trouble swallowing that, which is fair, since it sounds less like machine learning and more like a lecture about overwork. This is where the expectation-setting at the start of the project earned its keep. Because we'd agreed up front that we might not find strong predictors, I could report exactly what we found without it reading as failure. I ran partial-dependence plots to show how each factor moved the predictions, so the finding arrived as evidence instead of opinion, and the rest of the signal was weak enough that I told them plainly it wasn't something to build policy on. Could I have tuned my way to a more exciting story? Sure, and plenty of shops in that era did. We reported it straight instead, and Pratt presented the work at CIM in 2018.

Looking back, I think the honest result was also the true one. The idea that a model could see accidents coming was always a stretch, because too much of what causes an accident never makes it into a database. Did the operator have a fight with their wife that morning? Was there an undocumented microgeological event? What the data could support was something plainer and more useful. When people work more hours, more people get hurt. I eventually found some MSHA data that illustrated the issue beautifully, the long decline in coal mine incidents over time. Accidents at coal mines have gone down for decades even as production went up. A marvel of safety standards, right? Not quite. If you dig into the data, what you'll actually find is that incidents are almost perfectly linear with labor hours. We hurt fewer people in coal mines because we mechanized the mines. That's not the answer anyone hoped for, but it held up, and I'd give it again. The question didn't die there either. The same research line moved on to a coal operation with more data in better condition, and eventually made it into peer-reviewed print. That one gets its own page.