Web Scraping Library with Proxy Rotation #176

Activity

1. armanist

   added

   new feature

   on Nov 5, 2024
2. armanist commented on Nov 5, 2024

   

   armanist

   on Nov 5, 2024

   Last edited by armanist

   CollaboratorAuthor

   More actions

   Summary
   Introduce a modular web scraper component for the Quantum PHP Framework that supports proxy rotation, flexible data extraction, and integration with the existing HttpClient.

   Use Case
   This module would be useful for tasks such as:

   • Collecting public data from websites
   • Monitoring prices, content changes, or SEO elements
   • Automating data collection in testing or admin tools

   Proposed Features

   • Proxy Rotation

   • Add a ProxyManager class to manage a list of proxies

   • Support:

     • Manual proxy configuration
     • Loading proxies from a file or array
     • Rotation rules (e.g., per request, on failure, round-robin)

   • Data Extraction

     • Allow extraction using flexible selector strategies:
     • CSS selectors (default)
     • XPath (optional)
     • Support reusable, modular selector definitions

   HTTP Requests

   • Leverage the existing HttpClient component

   • Include common scraping-related features:

     • Custom headers (e.g., User-Agent)
     • Cookie/session support
     • Timeout and retry logic

   Implementation Notes

   • This should be an optional module, not part of the core.
   • Follow Quantum’s DI and service patterns.
   • Keep components lightweight and composable.
   • Consider extensibility for future selector engines or proxy providers.

   Out of Scope

   • JavaScript rendering (e.g., headless browsers like Puppeteer)
   • Built-in scheduling or persistent scraping jobs

   Example Usage (TBD)

   $scraper = ServiceLocator::get(Scraper::class);
   
   $data = $scraper->withProxyRotation()
                   ->extractFrom('https://example.com/products', [
                       'title' => '.product-title',
                       'price' => '.price-tag',
                   ]);
   

   Related

   • Relies on: HttpClient
   • Potential new components: ProxyManager, ScraperEngine, SelectorFactory

   Status

   🟡 Open for discussion
3. armanist

   added this to the 2.9.9 milestone on Mar 13, 2025
4. quantum-php

   locked and limited conversation to collaborators on Jul 19, 2025
5. armanist

   converted this issue into a discussion #306 on Jul 19, 2025