Optimized N8N Pipeline for Large-Scale Product Catalog Processing

Challenge

Our client is a consumer electronics reseller managing a constantly updating catalog of products purchased from multiple suppliers. Their internal database stores all active items in stock, and every new supplier price list, often thousands of SKUs, must be matched against this database.

The challenge: supplier naming conventions varied significantly, even for identical products. As a result, automated matching frequently failed, forcing the client’s team to manually inspect mismatches, resolve naming conflicts, and add missing items to appropriate categories.

They already had a processing pipeline built in N8N, but it struggled with:

• Performance Bottlenecks: Daily imports contained 1,500+ product records, but N8N flows processed them far too slowly.
• Low Accuracy due To Unstructured Prompts: Existing prompts used in the LLM steps were free-form and inconsistent, causing classification errors.
• Inefficient Database Queries: Their SQL-based workflow executed one query per item, which is extremely time-consuming at scale.

The client approached us to optimize the entire pipeline, improve accuracy, and dramatically reduce execution time.

Solution

We redesigned the pipeline with a focus on performance, reliability, and scalable LLM usage.

Prompt Optimization and Consolidation

We reworked the LLM prompts responsible for product classification and similarity matching. The client’s original prompts were written in a free-form style and executed as separate steps, forcing multiple LLM calls for a single item.

We redesigned them into structured instructions and merged previously independent tasks into one cohesive prompt. This significantly improved response stability and reduced the number of model interactions required during processing, leading to a 2.5x faster LLM processing.

Pipeline Refinement in N8N

The existing N8N workflow contained unnecessary branching and operation sequences that slowed down processing of large supplier price lists.

We refactored the pipeline to minimize processing steps and focus on performance. As a result, the workflow is now capable of handling daily imports of thousands of records without the delays the client previously experienced.

Migration From SQL Lookups to Vector Search

A major bottleneck was the database: each product triggered an individual SQL query. With large catalogs, this approach scaled poorly.

We implemented the Qdrant vector database to enable semantic matching and bulk similarity search, allowing fast retrieval of relevant candidates and more accurate product matching despite varying supplier naming conventions.

Benefits:

• Compare hundreds of candidates at once,
• More accurate matching based on product semantics, not literal names,
• 5–10x faster retrieval compared to sequential SQL queries.

Results

The upgraded system delivered a substantial performance improvement:

• Processing speed improved by 2.5x,
• LLM-related steps executed 2.5x faster, thanks to prompt consolidation,
• Database lookup time improved by 5—10x with Qdrant,
• Higher product-matching accuracy, reducing manual review,
• Scalable architecture ready for increased daily data volumes.

The optimized workflow now processes large supplier price lists quickly and reliably, enabling the client to maintain an up-to-date product catalog with minimal manual intervention.