WalletInvestor — 90k Symbol Financial Scraping Service & Graph Capture

Overview & Scale Requirements

WalletInvestor is a major financial forecasting platform covering a massive directory of assets across cryptocurrencies, stocks, forex, and commodities. The goal of this project was to build a reliable data collection pipeline capable of tracking roughly 90,000 financial symbols on a daily recurring basis.

For every symbol, the system needed to collect current market prices, historical price series, future forecast milestones, and dynamically rendered forecast graphs. Scraping 90k assets daily while managing proxy bandwidth costs and preventing IP bans required building the system as an operational web service rather than a simple standalone script.

Dual-Mode Extraction Architecture

To balance extraction speed with proxy consumption costs across 90,000 targets, I designed the extraction core with two purpose-built modes:

The Lite pipeline was built using Python and BeautifulSoup to request and parse raw HTML directly. Configured with rotating proxy pools, custom user-agent headers, and adaptive request pacing, this mode quickly extracted tabular price data and historical records without the overhead of running a heavy browser.

The Graph pipeline was built using Playwright browser automation for asset pages where forecast graphs were rendered dynamically via client-side JavaScript. Because these interactive charts could not be retrieved from static markup, Playwright loaded the rendered page context and captured the interactive graph snapshots directly.

Building an API-Driven Service

Rather than requiring manual terminal execution, I built a lightweight web server around the scraping engine. The server exposed REST API endpoints that allowed downstream applications and other developers to trigger extraction jobs, query pipeline status, and access newly saved records stored in PostgreSQL.

Aside from proxy credentials securely stored in environment variables, the entire operational state was dynamically configurable via API. Target queues, request pacing thresholds, and runtime flags could be updated on the fly without restarting the service or modifying codebase files.

Scheduling, Persistence, and Telemetry

The ingestion pipeline was configured with automated cron jobs aligned with the Chicago timezone (US Central Time) to execute updates at optimal market close intervals without manual intervention. All extracted pricing data, historical points, and run metadata were structured cleanly in PostgreSQL for reliable downstream consumption.

To give the client full visibility over pipeline health, I built an automated reporting and alerting system. The service dispatched automated emails to the client's inbox whenever a scraping job started and finished, delivering run summaries that detailed total symbols processed, success rates, and execution times. Additionally, the system tracked proxy bandwidth usage and triggered threshold warnings before data allocations were exhausted.