Online Database Scraping AI

Our client is a finance services company looking to expand their marketing efforts through the acquisition of clients from new channels, namely bankruptcy databases. We were tasked with the development of a data scraper which would go through these databases and extract data about companies who have recently filed for bankruptcy.

The scraper would have to integrate seamlessly into the company’s existing software infrastructure, connect with the CMS systems, and have a simple and intuitive design.

Data Scraper

The data scraper was built with Playwright to automatically access bankruptcy databases, navigate dynamic JavaScript-heavy pages, handle pagination, and extract structured company records from multiple online sources.

The system checks whether each record is new or has already been saved, helping avoid duplicates before the data enters the client’s workflow.

Before the extraction, the scraper processes the data to bring it to a uniform format, like phone numbers and addresses. The data can be extracted in various formats, including Excel sheets, or can be automatically added into a CMS our client uses.

To reduce maintenance, we added an LLM-assisted extraction layer with self-healing selectors. When page layouts changed, the system could identify equivalent fields based on labels, surrounding text, and page structure instead of failing immediately due to a broken CSS selector.

We ensured stability and continuous scraping by integrating Sentry for application monitoring, error tracking, and real-time alerts when the scraping flow was interrupted.

Human Behavior Emulation

The data scraping process closely emulates human behavior thus avoiding bans from database websites. Our data scraper can solve CAPTCHAs, handles pagination and performs page scrolling, changes IP dynamically, and rotates user agents.

The scraper uses a residential/mobile proxy pool, dynamic IP rotation, user-agent rotation, controlled scrolling, pagination handling, and CAPTCHA handling where permitted to improve scraping continuity and reduce interruptions.

The scraper can collect data continuously, adding new contacts as they show up in the databases, or work on a schedule, scanning the databases once a day or once a week.

Results

Our data scraper is already used daily by our clients, bringing them dozens of new potential clients daily. Its user-friendly design made it simple for the client's team to use the scraper, while CMS integration allowed new records to flow directly into the existing workflow.