ScriptsIntermediate★ 4.7 (23 reviews)

Python Web Scraping Script

Name: Python Web Scraping Script
Rating: 4.7 (23 reviews)
Author: WorkFreeWork Team

A robust Python script for scraping product data from e-commerce websites with rate limiting and error handling.

By WorkFreeWork Team•342 downloads•Added 1/15/2025

Overview

This comprehensive web scraping script uses BeautifulSoup and Requests to extract product information from e-commerce sites. It includes built-in rate limiting to avoid getting blocked, error handling for network issues, and data export to CSV/JSON formats.

Perfect for price monitoring, competitor analysis, or building product databases without manual data entry.

Get This Resource

View on GitHub →

Prerequisites

Python 3.8 or higher installed
Basic understanding of HTML structure
Familiarity with command line

Dependencies

beautifulsoup4==4.12.0
requests==2.31.0
pandas==2.1.0
python-dotenv==1.0.0

Installation

Download and extract the ZIP file
Navigate to the project directory: cd python-web-scraper
Create a virtual environment: python -m venv venv
Activate the virtual environment: source venv/bin/activate (Linux/Mac) or venv\Scripts\activate (Windows)
Install dependencies: pip install -r requirements.txt
Copy .env.example to .env and configure your settings

Usage Examples

Basic Usage

python

from scraper import ProductScraper

# Initialize the scraper
scraper = ProductScraper(
    base_url='https://example.com/products',
    rate_limit=2  # 2 seconds between requests
)

# Scrape products
products = scraper.scrape_products(max_pages=5)

# Export to CSV
scraper.export_to_csv(products, 'products.csv')

Advanced Configuration

python

from scraper import ProductScraper

scraper = ProductScraper(
    base_url='https://example.com/products',
    rate_limit=2,
    headers={
        'User-Agent': 'Mozilla/5.0 (compatible; ProductBot/1.0)'
    },
    retry_attempts=3,
    timeout=10
)

# Scrape with custom selectors
products = scraper.scrape_products(
    max_pages=10,
    selectors={
        'title': '.product-title',
        'price': '.product-price',
        'image': '.product-image img'
    }
)