Multi-Source Inventory Scraping & New Website Integration

Goals and Deliverables

Objectives & Deliverables

Core objectives:

• Scrape product details (name, description, specifications, SKU)
• Scrape pricing data (regular, discounted, currency)
• Scrape customer reviews (rating, review text, reviewer name, date)
• Create a central inventory database
• Build a responsive website from scratch to display the aggregated inventory
• Implement search, filters, and sorting for easy navigation
• Schedule automated scraping to keep data fresh

Key deliverables:

• Backend scraping system with modular scrapers for 50 sites
• Centralized product database
• Responsive, SEO-friendly website UI
• Admin dashboard for managing products and scraping schedules

Approach

Step 1: Site Mapping & Requirements Gathering

We audited all 50 target sites to determine:

• Content structure (HTML, JSON APIs, AJAX calls)
• Pagination and product loading methods
• Review availability and formatting
• Anti-bot protections

Step 2: Modular Scraper Development

• Built 50 separate scraper modules using Python, Playwright, and BeautifulSoup
• Extracted product details, prices, and reviews
• Normalized review ratings to a standard 5-star scale
• Stored images with URLs or downloaded as assets

Step 3: Data Normalization

• Standardized product fields (title, description, price, SKU)
• Converted currencies where needed
• Unified category taxonomy for consistent navigation on the new site

Step 4: Inventory Database

• Designed a PostgreSQL database to store scraped data
• Created relations for products, reviews, categories, and sources
• Enabled incremental updates to avoid full re-scrapes

Step 5: New Website Development

• Frontend: React + TailwindCSS for a clean, responsive design
• Backend: Django REST API to serve product data to the frontend
• Features: Search bar, category filters, sort by price/rating/date, product detail pages with reviews

Step 6: Automation & Scheduling

• Integrated Celery task queues for scheduled scraping jobs
• Set update frequency based on each site’s refresh rate (e.g., daily for fast-moving items)

Results

• Total products collected: 180,000+ across 50 websites
• Reviews aggregated: 1.2M+ customer reviews
• Website launch time: 8 weeks from project start
• Automated updates: Daily for high-change sites, weekly for stable inventories
• Search performance: <300ms response time for queries

Compliance

• Scraped only publicly available data
• Complied with each site’s legal terms and applicable laws
• Rate-limited scraping to avoid service disruption
• Provided opt-out mechanism if a source requested removal