Food Hub Market Tracker

Every evening at 7:30, for the last two years, a robot has walked the aisles of the online food hub our farm sells through and written down everything: every product, every producer, every unit size, every price, and how many were left. About 1,570 snapshots so far. Nobody else in the market has this record. As far as I can tell, not even the hub keeps it.

Getting things to grow is one of the easiest parts of being a market gardener. The hard part is knowing what to grow and how to price it. For years I had been building up a repository of anecdata about what different things cost in different markets, and when we started selling through a local food hub I saw an opportunity to get real insight. The hub ran on a platform that functioned as a two-sided marketplace. We posted listings, buyers said what they wanted. But unlike the marketplaces everyone knows, where Amazon at least gives its sellers page views and traffic to stare at, we had no window into what a buyer was thinking beyond whether our product sold. I knew our absolute position and nothing about our relative one, who else was selling that week, what they charged, whether their salad mix moved while ours sat. In a marketplace of dozens of regional producers, relative position is most of the game, and pricing is already one of the hardest problems in farming.

So every time we listed a new product, I'd go onto the platform and look. Who else was selling it? At what price? How much inventory seemed to be around right now? At some point I realized I should systematize this, and that a static snapshot of today wasn't even what I wanted. The ideal was seeing how things changed over time. Who was getting to the end of a production run and who was just starting one, who was dropping prices and who was raising them, and who was actually selling. Being a data guy, I decided to make a data solution.

The first piece was getting the data out. I was hoping for a nice published API, and both the vendor and the hub told me there wasn't one, so I went in assuming I'd have to scrape. After a little reverse engineering it turned out there absolutely was an API, it just hadn't been made public, and with some ingenuity I could hit the right endpoints for exactly what I needed. Once I could do it once, I wanted to do it every day. A cron job on Vercel now runs the collection on a schedule, capturing both of the hub's sales periods, gzipping the results, and dropping them into blob storage. Hosting cost, zero. It has been quietly accumulating the market's history since May 2024.

Raw snapshots aren't answers, so I built the analytics pipeline in Jupyter notebooks, playing with what the data looked like and engineering it into the formats I needed. The messiest problem was that every listing was user-entered. Names and quantities were all over the board, and the same onions could show up as 316 three-pound bags, 189 five-pound bags, and 94 ten-pound cases, with "10 lb case," "10LB," and "5 pound" as three ways of writing one thing. I used agents to normalize all of it down to single units, which is something these tools are fantastic at. Hand them a big pile of messy data and say normalize this, and it works like a charm. The other trap was that those listings all drew from one shared pool of inventory, and the two sales periods were two views of the same stock, so naive sums double and triple counted the market. I had to reconstruct each product's true pool and aggregate against that instead of against the listings.

The dream of all this would be knowing what was listed versus what sold, but I don't have any insight into the hub's sales data. What I came up with was a proxy. I had inventory from one day to the next, so I could look at the delta between the two, and if a listing was lower one day than the day before, assume that inventory had been sold. I say assume because it's possible the inventory sold on another platform, or got discarded, or was a misestimate of how much was actually available in the first place. But when you're working with data like this, a decent approximation is often better than nothing.

That got me excited, because it meant I could answer a question nobody publishes. What actually sells on this platform, and where are the gaps in the market? I ran the analysis across a pile of products, and the honest finding was that there weren't any gaps. Nothing listed in the hub was supply constrained. Every product was demand constrained. The things that sold, sold for everyone, and the things that sat, sat for everyone too.

I didn't let that dissuade me, because the information was still incredibly useful. I built a Next.js dashboard on top of the database where I can pick any crop and see how its price has moved, how inventory has moved, and who the biggest sellers are, with prices weighted by inventory so one boutique listing can't skew the picture. I now know who the largest sellers in the metro are and exactly what they sell, and that has let us dial in our production and our sales effort. It's one of the big reasons the farm grew as large as it has.

The strategic answer followed from the finding. If we're demand constrained, the only levers we can push are quality and price, and the platform doesn't allow much quality signaling. So we went in hard on a few products where we could compete on price, and otherwise we largely exited wholesale and moved toward direct-to-customer channels, our farmers market and CSA, where we meet our buyers face to face, where they can see the quality we produce, and where we can build relationships. That has skyrocketed our sales and bought us a defensibility the food hub never could.

One of the things I love about data is that people think it will magically hand you an answer. It won't. You have to go in and massage it, find ways to leverage it toward the questions you came in with, and when the data doesn't support a question, evolve the question into something the data can answer. There's an art and a science to working with data, and it's one of the things that made me fall in love with programming in the first place. The robot still makes its rounds every night at 7:30. The storefront still only shows today, and I'm still the only one who can see yesterday.