Seeds You Need

Seed companies are unbelievably behind the times on e-commerce. Pick a vegetable variety and try to answer the questions a grower actually asks. Who sells it? Who carries something close to it? Is there an organic version anywhere? Who has it in stock right now? Most seed company websites can't answer those questions about their own catalog, and nothing anywhere answers them across catalogs. Every company publishes on its own site, in its own structure, with its own vocabulary. The data all exists. It has just never been in one place.

For me this was never abstract. Organic certification comes with a rule I lived with on my farm. When an organic version of a seed is commercially available, you're required to buy it, and when you can't find one, you document the search that came up empty. I ran that search by hand, vendor by vendor, season after season, and no corner of the industry is built to answer it. So for years I carried around a dream, a unified catalog for every seed company in North America, every variety from every vendor, searchable in one place. Seeds You Need is me finally building it.

It sat for years because the honest version of the old plan was more than 200 hand-built scrapers, one per company, each full of hand-tuned selectors that break the first time a vendor redesigns a page. For one person that isn't a project, it's a treadmill. LLMs changed the math. An agent can now reach out to a site I've never studied, figure out how to pull it down, and then pull it down. And instead of writing bespoke scrapers, you can use a cheap per-token model to parse through the data and pull out the critical information, at least until you grow to a scale where it makes sense to build bespoke scrapers. The only custom code per company is a small URL extractor.

So the pipeline is built in two separate stages. The scraper starts from a company's sitemap to discover seed URLs, but the sitemap is only a starting point. You'd be amazed how many sitemaps don't accurately reflect the website behind them, even from multimillion-dollar producers, so we genuinely crawl the pages rather than trusting the map, and companies with no sitemap at all get a crawl strategy that discovers and scrapes in a single respectful pass instead of hitting their server twice. Each page gets fetched, stripped down, and stored as raw HTML, the source of truth. Understanding happens later, in a replaceable stage where a model reads the stored HTML and extracts structured data, including the variant and packet-size combinations that make seed listings genuinely messy. If the schema changes, or a better model ships, I re-run parsing over HTML I already have and never touch anyone's website again. Onboarding a new company comes down to a config entry plus that small URL extractor, which is how one person scales to the 220-company target list I researched and catalogued, traffic data and sitemap availability included, before the first scrape ran.

The whole approach only works if the economics work, and LLM parsing has a reputation for being expensive, so I engineered the cost down. The first thing I did was use some genuinely amazing models like Gemini 2.5 Flash, which was one of the cheapest models available for a long time and was genuinely good at parsing information out of large documents without hallucinating. I could also strip the HTML down immensely before storing it and passing it to the model, so it received less in the first place, which shrinks the storage bill and the token bill both. The combination of the two brought my parsing cost down to about one fifth of a cent per listing, taking what would have been a five-figure catalog cost down to about the price of lunch. Add some hashing strategies that let us skip unchanged pages, which is most of the catalog on any weekly scan, and that's where the real long-term savings live. The goal was for the whole system to run for under $100 a week once it goes live across 200 sites, and I'm on pace to do that.

On top of the structured data sits the reason any of this matters, a search that understands how growers actually ask. "Heat tolerant lettuce that bolts slowly" is a real question, and vector embeddings over the parsed listings answer it by meaning rather than keywords, with plain filters on top for organic certification, price, and stock. The data is organized around varieties, not companies, so the questions I opened with have answers the moment listings flow through. Every supplier of Cherokee Purple in one view. The organic version of a conventional variety, if one exists anywhere. A documented empty search for the certifier when it doesn't. I proved that chain end to end with a tracer, raw HTML to parsed listing to semantic answer. Postgres and Drizzle hold the schema, and vector search rides on top.

What building it taught me is how hostile the internet already is to a bot that just wants to read it, and I hit that wall before the modern wave of anti-bot protection had even kicked in. Some of the hostility is deliberate, like Cloudflare challenges and login walls standing in front of pages that sell carrot seed. Some is neglect, like the missing sitemaps. Either way, scraping the open web politely is an operations problem, so the pipeline is built like an operations system. A feasibility script sizes up every new company before I touch it, checking robots.txt permissions, crawl delays, sitemap access, and Cloudflare and login walls, and hands back a verdict before I spend a token. A queue moves every URL through pending, processing, and completed states with full error tracking, throttled for testing and batched for production. And URL monitoring watches for strange swings in page counts between crawls, the early warning that a site changed shape underneath me.

This was a winter project in early 2026, and it ultimately got sidelined when I had to turn to the crop planning software that ran our 2026 season. But the scrapers are largely done, a cursory front end exists, and the raw HTML sits in storage as the source of truth, with parsing re-runnable any time and the models that do the understanding getting better and cheaper while it waits. The reality of a piece of software like this is that it's highly seasonal. Most people buy their seed from about February to May, so the goal is to finish it over the winter of 2026 and launch it for the 2027 growing season. Most of this was built with Opus 4.5, and I can't wait to see what 5.1 does against this codebase.