Finding relevant candidates from public sources sounds like a perfect task for an autonomous AI agent.
Give the agent a browser, describe the target profile, and let it search the web, open pages, read resumes, analyze profiles, and build a shortlist. In theory, this looks like the most flexible approach. In practice, it is often too slow, too expensive, and too hard to control at scale.
In one of our pilot projects, we worked on a system for extracting structured candidate data from public and compliant online sources, including resumes, CVs, technical profiles, and search results. The goal was not to build a generic scraper. The goal was to show the client how different search and extraction approaches could be combined into a practical, scalable pipeline.
One of the client’s explicit requirements was to include resume search through Google and Bing. This became an important architectural decision. Instead of relying on a browser-based LLM agent as the first layer of the system, we built a search-first pipeline that collected public metadata from search results, enriched it with documents and API-based sources, and then used AI for normalization, deduplication, and ranking.
The result was a more predictable and cost-efficient workflow.
The Initial Assumption: Let the LLM Agent Browse
At first glance, a browser-based LLM agent seems more powerful than a search API.
It can open pages, follow links, read the page content, interpret context, and make decisions dynamically. For small-scale research tasks, this can be useful. If you need to inspect a handful of pages manually or semi-automatically, an agent can reduce routine work.
However, public candidate sourcing has a different structure. The system may need to process hundreds or thousands of potential results. It has to avoid sources that require login, respect access restrictions, handle inconsistent document formats, and produce a clean output instead of a collection of random links.
This is where the browser-agent approach becomes less attractive.
Every page visit adds latency. Every loaded website introduces uncertainty. Some pages are unavailable, some require authentication, some are blocked by anti-bot mechanisms, and some contain much more information than needed for the first filtering step. If the system only needs to know whether a search result is potentially relevant, opening the page is often unnecessary.
In our case, the browser agent was not the right tool for the first layer of the pipeline. It was more useful later, when the system already had a smaller set of relevant documents or records that required deeper interpretation.
Why the Search Layer Works Better for First-Pass Discovery
A search-first pipeline works differently.
Instead of trying to browse every website directly, the system queries Google and Bing through SerpApi and receives structured search result data: URL, title, snippet, ranking position, and source context.
For first-pass discovery, this is often enough.
A search result snippet may already contain a candidate's role, technology stack, location, current or previous company, document type, or a link to a resume. If we are looking for a Python backend developer with FastAPI and PostgreSQL experience, these signals often appear directly in the title or snippet. If we are looking for a machine learning engineer with computer vision experience, terms like PyTorch, OpenCV, detection, segmentation, or model training may also be visible before opening the page.
This gives the pipeline a fast and lightweight way to collect potential matches.
The search layer does not replace deeper analysis. It simply reduces the number of pages and documents that need to be processed later. Instead of sending an LLM to browse the entire web, we use search metadata to build a cleaner candidate pool first.
Public Profiles Without Platform Scraping
One important part of the pilot was working with publicly indexed professional profiles.
There are two very different ways to approach this. The first is to automate direct browsing of professional platforms: open profile pages, click through them, and extract data from the page. This approach is fragile, risky, and often conflicts with platform restrictions.
The second approach is more controlled: analyze only the information already available in public search results. In this setup, the system works with the URL, title, snippet, and source type. It does not automate login, bypass CAPTCHA, or crawl restricted areas.
This is not enough to make a final hiring decision, but it is enough for early-stage filtering. The system can identify potentially relevant profiles, separate personal profiles from company pages or job posts, and extract preliminary signals such as role, stack, location, or seniority.
The goal is not to replace a recruiter or sourcer. The goal is to create a structured first layer that removes irrelevant results and prepares a shortlist for further review.
Resume and CV Document Processing
Search results are only one part of the pipeline. The pilot also included public resume and CV documents.
Documents are valuable because they may contain much richer information than snippets: work history, skills, projects, education, certifications, links to portfolios, and references to technical profiles.
However, a found document is not a structured candidate record. It may be a PDF, DOCX, HTML page, or plain text file. It may follow a standard resume format or a completely custom layout. The relevant information may be spread across several sections.
After a document is discovered, the pipeline extracts text, identifies the document type, and uses AI to transform it into a structured candidate profile. The system can extract role, skills, companies, years of experience, education, links, and other relevant fields depending on the client's requirements.
This is where LLMs add real value. They are not used to blindly browse the web. They are used to interpret documents that have already passed the first relevance filter.
Adding API-Based Technical Sources
To make the pilot more representative, we also included technical sources such as GitHub and GitLab where applicable.
These sources provide a different type of signal. Search snippets and resumes describe a candidate. Technical profiles and repositories can show technologies used in practice, public projects, activity, and links to other profiles or personal websites.
API-based sources are also easier to structure than arbitrary web pages. They can return usernames, repositories, programming languages, profile descriptions, and other metadata in a more predictable format.
Combining search results, public documents, and technical APIs gave the client a broader view of what the system could do without turning the pilot into an oversized and expensive project.
The Pipeline Architecture
The final workflow had several layers.
First, the system receives the search criteria: target role, technology stack, location, source types, and document types. Based on this input, it generates search queries and API requests.
Then SerpApi returns structured results from Google and Bing. In parallel, API-based sources provide additional technical signals where available. Public resume documents are downloaded and parsed only when they match the source and relevance criteria.
After collection, all raw data is normalized into a common structure. This is a critical step because every source returns information differently. A search result gives URL, title, and snippet. A PDF resume gives unstructured text. A technical profile may give username, repositories, and programming languages. A personal website may contain a mix of biography, portfolio links, and project descriptions.
The normalization layer turns these fragments into a unified candidate record with fields such as name, role, skills, location, source URLs, document references, source type, and confidence level.
Only after this step does the system move to deduplication and ranking.
Modular AI Systems for Your Data
Deduplication: From Links to Candidate Records
Deduplication turned out to be one of the most important parts of the project.
When the system searches across multiple sources, the same person often appears several times. One candidate may be found in a search snippet, a PDF resume, a GitHub profile, a GitLab profile, a personal website, and a publicly indexed professional profile.
If these records are not merged, the client receives noise instead of insight.
The deduplication layer compares URLs, names, usernames, source domains, resume links, technology stacks, company mentions, and other matching signals. When multiple records likely refer to the same person, the system merges them into one candidate profile and attaches all supporting sources to that profile.
In one test run, the system processed 1,000 found people and merged 581 duplicate matches inside the pipeline.
This is exactly the kind of step that turns raw search output into a usable sourcing workflow. The client does not need another spreadsheet full of repeated links. They need a clean shortlist of people, with sources and relevance signals attached.
Ranking and Human Review
After deduplication, the remaining candidates are scored and ranked.
The scoring logic can include role match, skill match, location, number of independent sources, document quality, technical profile availability, recency, and confidence of extracted fields. The exact formula depends on the use case.
For example, a candidate found only in one weak snippet may receive a lower confidence score. A candidate with a resume document, a matching technical profile, and consistent skills across several sources may be ranked higher.
The output is not meant to be a black-box decision system. It is a structured review queue. Recruiters, researchers, or internal teams can see why a candidate was included, which sources support the match, and what signals were extracted.
This makes the workflow easier to audit and adjust.
Where LLMs Fit Best
The main takeaway from the pilot was not that LLM agents are useless. It was that they should be used in the right layer of the system.
LLMs are very useful for interpreting ambiguous snippets, extracting structured fields from resumes, normalizing job titles, matching similar skills, summarizing candidate relevance, and processing documents with inconsistent formatting.
But they do not need to be responsible for first-pass discovery across the entire web.
A search-first architecture uses the search layer for wide coverage, APIs for structured technical signals, document parsers for resumes and CVs, and LLMs for interpretation and normalization. Each component does the job it is best suited for.
This makes the system faster, more predictable, and easier to scale.
Looking for AI developers?
We create AI software — and we do it well. Talk to us to get your project started today
What We Deliberately Avoided
In this pilot, we deliberately avoided approaches that would make the system fragile or difficult to use in a real business context.
The system did not automate login, bypass CAPTCHA, imitate user behavior on restricted platforms, or crawl private areas of websites. It focused on public search results, public documents, and sources that could be accessed in a controlled and compliant way.
This boundary is important. A sourcing pipeline should not only find data. It should also be explainable, maintainable, and suitable for responsible business use.
Conclusion
For public resume data extraction, the most advanced-looking tool is not always the most effective one.
A browser-based LLM agent may be useful for deep analysis of selected pages or documents, but it is often too slow and unpredictable for large-scale first-pass discovery. A search-first pipeline provides a better foundation: it collects public metadata quickly, filters irrelevant results early, and passes only promising records to the more expensive AI layers.
The strongest architecture is not a single autonomous agent trying to browse everything. It is a layered system where search APIs, public documents, technical sources, deduplication logic, scoring, and LLM-based normalization work together.
For the client, this approach delivered exactly what a pilot should deliver: a practical demonstration of multiple data extraction methods, a clear architecture, and a usable path from raw public sources to a structured candidate shortlist.
Have an AI, automation, or custom software idea?
Request a free prototype and validate it before committing to a full build.