Building an Autonomous Market Intelligence Engine: How I Replaced My Job Search with AI Agents

Job searching is one of the last genuinely unautomated parts of a technical career. That is strange, because almost everything about it is mechanical.

You open the same four tabs. You type the same three keywords. You scroll past the same eighty postings you already rejected last week, because nothing remembers that you rejected them. You open a listing, skim it, decide in nine seconds that it is not a fit, and close it. Then you do it again tomorrow, and the day after, and you call this "being active on the market."

I did that for about two weeks before the obvious hit me: this is not a search problem. It is a data pipeline problem with a human stuck in the middle of it, doing the part a machine does better.

So I stopped searching and built the thing that searches. It's called Job Finder (github.com/peturganchev/Job-Finder), it runs on my machine, and what it actually produces is not a list of jobs. It is a continuously updated picture of what the market wants.

Why manual job searching is structurally broken

The failure is not effort. People searching for work are not lazy. The failure is that the process has three properties that make human attention the worst possible tool for the job.

It has no memory. Every session starts from zero. Job boards do not know what you saw yesterday. You re-read, re-evaluate, and re-reject the same postings constantly. A trivial deduplication layer eliminates most of the work, and no board gives you one.

It optimizes for recall, not precision. Boards want you to see many postings, because engagement is their business model. You want to see the three that matter. These goals are directly opposed, and the board controls the interface.

It gives you no aggregate view. You read listings one at a time, which means you never see the pattern. You cannot tell from reading thirty postings in sequence that twenty-two of them mention vector databases and only four mention the framework you spent last month learning. That signal exists in the data. Sequential human reading destroys it.

That last point is the important one. The individual job posting is almost worthless as information. The distribution across hundreds of postings is where the actual intelligence lives, and manual searching is specifically the method guaranteed to never surface it.

What the agents actually do

The architecture is deliberately unglamorous. Collection, normalization, analysis, delivery. Each stage is independent, and each one is allowed to fail without killing the others.

Collection runs against multiple sources with different personalities. dev.bg and jobs.bg for the Bulgarian market, LinkedIn for the broader pool, plus Himalayas, EURemoteJobs, and the HackerNews "Who is hiring" API for remote positions. Some of these are polite HTTP endpoints. Others are hostile: Cloudflare challenges, authwalls, 2FA prompts, rate limits designed specifically to stop what I was doing.

The solution was a persistent browser profile. The system logs in once, interactively, and keeps the session on disk. Subsequent runs reuse real cookies in a real browser context, with human-plausible timing. It is not clever. It is just patient, which turns out to matter more.

Normalization is where most of the value quietly happens. Every source has its own shape, so everything gets flattened into one schema and hashed for deduplication. A posting that appears on three boards under two titles is one record. The archive is permanent, so a listing I rejected in March never asks for my attention again in October.

This stage alone removed the majority of the manual work. No intelligence required. Just memory.

Analysis is the LLM layer, and this is where it stops being a scraper and starts being something else.

Keyword matching is the wrong abstraction

Here is the thing nobody says out loud: a keyword filter does not understand your career. It understands string equality.

Search "LangChain" and you miss the posting that says "experience building agentic workflows with modern orchestration frameworks." Search "Python" and you drown in eleven years of unrelated backend roles. Keywords are a blunt instrument pretending to be a filter, which is why everyone ends up with the two classic failure modes: too strict and you see nothing, too loose and you see everything.

The LLM does something categorically different. Each posting goes to Gemini together with my structured profile: skills, project history, target roles, learning roadmap. The model is asked to reason about fit rather than detect substrings, and it returns three things.

A match score from 0 to 100. Not a binary filter. A ranking signal, which means the inbox is sorted by relevance instead of recency. The twelve-percent matches still exist. They just live at the bottom where they belong.

The specific gaps. This is the part I did not expect to matter most. For every posting, the model names what I am missing. Not "you do not qualify," but "this role expects production RAG experience with a vector store, and your portfolio shows retrieval only in a prototype context."

A tailored cover letter draft. Generated against the actual posting and my actual background, so the starting point for an application is an editing task rather than a blank page.

The scoring is useful. The gap extraction is transformative, and for a reason that only becomes obvious in aggregate.

The skill gap analysis is the real product

One gap analysis tells you about one job. Two hundred gap analyses tell you about the market.

The system runs an --insights pass that synthesizes across everything collected: which technologies appear most often, which combinations cluster together, which requirements are genuinely common versus copy-pasted boilerplate that no hiring manager actually screens for. The output is a market report, written to disk, rebuilt as new data arrives.

That report answers a question job boards structurally cannot: not "what jobs can I apply to today," but "what should I build next so that more jobs become available to me in six weeks."

This inverts the entire relationship. The default job search is reactive. You take the market as fixed and sort yourself against it. A gap report makes it a feedback loop. The market tells you what it is short on, you build toward the gap, and your match scores move. I can watch that happen, because the scores are stored and the trend is visible.

The dashboard reflects this. Alongside the job pipeline with its application statuses, there is a skill roadmap view and a portfolio projects view. The roadmap is derived from what the market is actually demanding, not from a course catalog's opinion about what a curriculum should contain. The system suggests concrete portfolio projects weighted by how often their component skills appear in real postings.

I stopped thinking of it as a job finder somewhere around that point. It is a market intelligence engine that happens to emit job listings as a side effect.

What broke

Any account that only shows the working version is marketing, not engineering. So here is what went wrong.

Anti-bot defenses won repeatedly. Early scrapers were fast and obvious, and they got blocked within days. The fix was not a smarter evasion trick. It was accepting that a slow, patient, session-persistent scraper that looks like a tired human browsing at eleven at night is more durable than a fast one. Throughput was the wrong metric.

Per-posting LLM calls did not scale. Analyzing each listing in isolation was slow and wasteful, especially on a free API tier. Batched analysis fixed the economics, but introduced a new failure mode: batch calls fail as a unit, so one malformed posting could poison twenty good ones. That required per-item error isolation inside the batch.

The model was too generous at first. Early prompts produced scores clustered between 70 and 90 for almost everything, which is the LLM equivalent of a filter that passes all input. Useless. Forcing explicit justification for the gaps, and requiring the model to name what was missing before assigning a number, pulled the distribution apart into something with actual discriminating power.

Single-user SQLite became a ceiling. It was the right call for a local prototype and the wrong one the moment I wanted it running for anyone else. Moving to Supabase with PostgreSQL and real authentication, with a Cloudflare Worker fronting access, was the point where a personal script turned into a system other people could use.

What this generalizes to

The job search is a specific instance of a pattern that shows up everywhere: a high-volume, low-signal public data stream that humans monitor manually because no one has built the filter.

The recipe is not exotic. Collect from sources that do not want to be collected from. Normalize and remember, so nothing is evaluated twice. Score with a model that reasons about meaning instead of matching strings. Then, critically, aggregate the scores into a view of the whole, because the aggregate is where the real intelligence has been hiding the entire time.

The part I keep coming back to is the inversion. I did not build this to find a job faster. I built it to stop guessing about what the market wants, and the honest answer is that for months I had been guessing. Reading job postings one at a time felt like research. It was not. It was sampling a distribution at a rate far too low to ever see its shape.

The agents do the searching now. I spend my time on the only question that was ever worth my attention: given what the market is clearly asking for, what do I build next?

The code is on GitHub if you want to point it at your own profile.