Expedia — Dynamic Hotel Review Scraper & Control Dashboard

Overview & Dynamic Site Constraints

The goal of this project was to extract customer reviews, rating breakdowns, and stay metadata across a large queue of hotel listings on Expedia. Unlike static directories, Expedia is a highly dynamic platform where reviews and content are rendered on the client side via backend GraphQL APIs as the user scrolls, protected by anti-bot verification layers.

Standard headless automation failed to load review data reliably. To overcome this, I engineered an extraction pipeline using Playwright running in a headed browser environment, combined with network request interception to capture data payloads directly from underlying API responses as the page navigated.

Bandwidth Optimization via Hybrid Interception

Running headed browser automation at scale against thousands of listings introduces severe proxy bandwidth costs. If the browser downloads every image, tracking script, and marketing asset on every page load, proxy budgets exhaust rapidly.

To solve this, I implemented a hybrid extraction strategy. Instead of relying purely on heavy visual rendering or scraping rendered DOM trees, the engine intercepted internal API responses directly while blocking unnecessary static assets. This allowed the browser to satisfy dynamic verification checks while keeping raw proxy bandwidth consumption strictly focused on review and metadata payloads.

Building the Client Control Panel & Dashboard

Rather than delivering a background script that required technical intervention to operate, I built an interactive web dashboard so the client could manage and monitor the entire scraping lifecycle.

The control panel displayed real-time progress metrics, including collected reviews per hotel, overall run completion rates, and error logs. Through the interface, the client could start or pause extraction jobs, re-queue failed URLs with custom retry limits, export collected datasets automatically split by target file size, and trigger scheduled database backups at fixed milestones to prevent data loss.

Deployment on Windows VPS

Because the extraction engine relied on headed browser automation with specific display rendering dependencies that were inconsistent on headless Linux instances, I deployed the complete system directly to the client's Windows VPS environment. I configured the execution environment, local PostgreSQL storage, and the web-based control panel to ensure the client could operate the pipeline autonomously without terminal commands.