Data Pipeline with n8n + AI: End-to-end architecture for SMEs
A data pipeline isn't just "n8n pushing data around". Here's the 5-layer architecture our mid-market pipelines actually run on in production — and where they fail.
The 5 layers
Layers
Typical build
Error rate in prod
The numbers in this article are estimates from real projects for orientation only. Actual effort depends on the specific scope — for a binding quote, book a Discovery Call. Our official tier prices are on the Pricing page.
Every pipeline we build has the same base structure:
Ingest
Tap sources: webhooks, API polls, S3/storage watchers, email listeners, FTP drops.
Enrich
Enhance the data: lookups, geocoding, AI classification, document OCR.
Classify
Routing logic: where does each item go? Which needs human approval?
Store
Persist: Postgres, BigQuery, S3 — depends on volume and query pattern.
Notify
Who needs to know when something happens? Slack, email, CRM updates, dashboards.
Real stack: supplier-data pipeline
Client: Wholesale, 80 employees, 200+ suppliers, 30k inbound documents/month.
Docs / month
Time-to-insight
Freed up
Problem: Suppliers send invoices, delivery notes, price lists — as PDF, Excel, CSV, sometimes images. Manual handling: 3 full-time positions.
Architecture:
Layer by layer:
Layer 1: Ingest
Three sources:
- Email attachments → n8n IMAP trigger
- Supplier-portal uploads → n8n webhook
- FTP drops → 5-minute watch workflow
All land in an AWS S3 bucket with metadata tagging (source, supplier, date).
Layer 2: Enrich
- PDF → Mistral OCR API (beats Tesseract on tables)
- Image → Mistral OCR
- Excel → direct parsing in n8n
- Text extraction → Claude API for structured extraction (JSON schema prescribed)
Layer 3: Classify
Routing logic in n8n switch nodes:
- Invoice → accounting queue
- Delivery note → warehouse system (SAP integration)
- Price list → pricing engine + Slack alert to purchasing
Anomalies (price jump >15%, new supplier without master data) → human approval via Slack button.
Layer 4: Store
- Structured data → Postgres (for analytics)
- Original docs → S3 (10-year retention for compliance)
- Audit log → separate Postgres table
Layer 5: Notify
- Daily Slack summary: doc count, anomalies, backlog
- Anomaly → instant Slack DM to the relevant person
- Weekly PDF executive report (volume, top suppliers, issues)
Common pitfalls
n8n is powerful but not right for every job. Before each node, ask: would a standalone Python script do this better?
Pitfall 1: Everything in n8n
n8n is for orchestration, not heavy processing. If you're iterating 50k items, push it into a separate script — n8n stays the trigger.
Pitfall 2: No error handling
Default workflow stops on first failure. You need:
- "Continue On Fail" for non-critical nodes
- Error-trigger workflow with Slack alert
- Dead-letter queue in Postgres for manual retry
Pitfall 3: Database connection limits
n8n opens one connection per workflow run — at 100 concurrent runs you hit the pool limit fast. Fix: PgBouncer or external connection-reuse service.
Pitfall 4: Secrets in workflow JSON
n8n stores workflow definitions as JSON. Hard-coded API keys leak into backups, exports, git. Use n8n credentials, never plain text.
Hosting decision
n8n Cloud
Fast, managed, ~EUR 50-200/month. Data routes via US servers.
Self-hosted (EU)
Hetzner, Scaleway, AWS Frankfurt. Full control. Setup: 1-2 days. Maintenance: 1h/week.
For GDPR-sensitive workloads (HR data, patient data, financial data) we recommend self-hosted in the EU.
What does a pipeline cost?
Pipelines scale with complexity. Here are realistic ranges across three typical tiers:
Small
1 source, <5k docs/month
Build: €8-15k · 1-3 weeks Run-cost: €100-300/month
Medium
2-3 sources, OCR, AI classification
Build: €15-30k · 3-5 weeks Run-cost: €300-1,000/month
Complex
SAP/ERP integration, compliance, anomaly detection
Build: €30-50k · 5-8 weeks Run-cost: €1,000-2,000/month
The supplier use case described above (30k docs/month, SAP, 10-year retention) falls into Complex. A simple email-attachment pipeline with classification can start at Small and grow from there.
ROI typically in 6-9 months if the pipeline replaces ≥1 FTE.
Next step
Thinking about a specific pipeline? 30-minute call. We map your real sources onto the 5 layers and you walk away with a clear build plan.