How to Scrape Product Data from Lazada Using Python?

Let’s step into the world of online shopping for a moment. Every day, websites like Lazada are updated with thousands of products—new listings, changing prices, customer reviews, and more. Now imagine trying to gather all that information by visiting each product page one by one. Sounds exhausting, right? That’s exactly where web scraping comes in to save the day.

Web scraping is like teaching a computer how to browse a website and collect specific pieces of information—just like you would, but much faster. It helps turn scattered data from websites into clean, structured formats like spreadsheets. This is especially useful in fields like price tracking, market research, or keeping an eye on competitors.

For this project, we chose Lazada’s Skincare – Face Masks & Packs category. This section alone has a huge variety of products. Collecting details like prices, names, discounts, and seller info manually would take forever. So, we automated it. Using a script, we first collected all the product links by scrolling through the pages. Then, we visited each of those links to extract useful details- such as product name, price, any available offers, ratings, number of reviews, and seller information.

With everything neatly gathered, the data will be ready to analyze—whether that’s for understanding pricing trends, finding popular products, or studying how different sellers are performing.

Product data from Lazada: Product Links Scraping

Importing Required Libraries

import logging
import random
from playwright.sync_api import sync_playwright
from bs4 import BeautifulSoup
import sqlite3
import time

The script begins by loading all the tools it needs to do the job. Think of it like gathering all your ingredients before starting to cook—it helps everything run smoothly.

First, we bring in the logging library. This helps us keep track of what’s happening in the script—whether everything is going fine or if something goes wrong. Then we use random to add variety in things like selecting user-agents (these help us mimic different browsers) and setting small delays between actions. The time module is used to pause the script briefly between steps so we don’t overload the website with requests.

Next, we load sync_playwright from Playwright. This is the tool that helps us handle web pages that load content dynamically—just like when you scroll down an online store and more products appear. Playwright lets us control a browser behind the scenes, just like a person would.

We also use BeautifulSoup to read and understand the structure of the web page so we can find and extract the product links easily. And finally, sqlite3 is included so we can save those links into a database for later use.

Starting with these imports ensures the script has everything it needs before it begins. Playwright is especially important here, because most online stores use JavaScript to show content—and Playwright helps us handle that smoothly. Altogether, these libraries form a strong setup for collecting and storing product links efficiently.

Configuring Logging

# Configure logging
logging.basicConfig(
    filename="lazada_scraper.log",
    filemode="a",
    format="%(asctime)s - %(levelname)s - %(message)s",
    level=logging.INFO
)

Once the tools are set up, the script moves on to something very important—logging. This is like keeping a diary of everything the script does. It helps us know what went right, what went wrong, and when things happened.

The script sets up a log file called lazada_scraper.log. As the scraper runs, it keeps writing updates into this file. Each log entry includes the time, the type of message (like info, warning, or error), and a short message about what happened. This is really helpful when you want to go back and check if something went wrong—or just see how the script performed.

The logging level is set to INFO, which means it records general updates, any warnings, and errors—but skips very detailed debugging messages. This keeps the log clean and easy to read.

Logging becomes especially valuable in web scraping. If a page fails to load, if a user-agent is missing, or if there’s a problem saving data to the database, the log tells you where and why it happened. It’s like having a map to find where things went off track. And when the scraper is running through many pages, structured logging ensures you can spot and fix issues without guessing.

Loading User Agents

def load_user_agents(file_path):
    """
    Load a list of user agents from a text file for browser spoofing.

    Args:
        file_path (str): Path to the text file containing user agents, one per line

    Returns:
        list: List of user agent strings. Returns empty list if file reading fails.

    Raises:
        FileNotFoundError: If the specified file path doesn't exist
        IOError: If there are issues reading the file

    Notes:
        - Empty lines in the file are skipped
        - Logs a warning if no user agents are found in the file
        - File should contain one user agent string per line
    """
    try:
        with open(file_path, 'r') as file:
            user_agents = [line.strip() for line in file.readlines() if line.strip()]
        if not user_agents:
            logging.warning("No user agents found in the file.")
        return user_agents
    except Exception as e:
        logging.error(f"Failed to load user agents: {e}")
        return []

The next part of the script deals with something websites are very cautious about—bots. Many websites try to block scrapers, especially if they detect that the same “browser” is sending too many requests. To avoid this, we use something called a user-agent.

A user-agent is just a short message that tells the website what kind of browser or device is being used—like Chrome on Windows or Safari on an iPhone. The function load_user_agents(file_path) helps the scraper read a list of these user-agent strings from a text file. This way, the scraper can pretend to be a different browser each time it makes a request.

Here’s how it works: the function tries to open the file and read the user-agents line by line, storing them in a list. If the file is missing or empty, it doesn’t break the script—instead, it logs a warning and returns an empty list. That way, the script continues running, and you can check the logs to see what went wrong.

By rotating user-agents, the scraper blends in more naturally with normal web traffic, making it less likely to get blocked. And with proper error handling, it stays stable even when something unexpected happens.

Initializing the Browser

def initialize_browser(user_agents):
    """
    Initialize a Playwright browser session with a randomly selected user agent.

    Args:
        user_agents (list): List of user agent strings to choose from

    Returns:
        tuple: (playwright instance, browser instance, page instance)
            - playwright: The Playwright context manager
            - browser: The browser instance (Chromium)
            - page: The initialized page object

    Raises:
        Exception: If browser initialization fails
        IndexError: If user_agents list is empty

    Notes:
        - Launches browser in non-headless mode (visible browser window)
        - Randomly selects a user agent from the provided list
        - Logs the selected user agent for debugging purposes
    """
    try:
        user_agent = random.choice(user_agents)
        logging.info(f"Using user agent: {user_agent}")
        playwright = sync_playwright().start()
        browser = playwright.chromium.launch(headless=False)
        context = browser.new_context(
            user_agent=user_agent
        )
        page = context.new_page()
        return playwright, browser, page
    except Exception as e:
        logging.error(f"Failed to initialize browser: {e}")
        raise

Next, the script sets up the browser—the heart of the scraping process. The initialize_browser(user_agents) function uses Playwright to open a browser window that the script can control, just like a real user browsing the site.

To make things more realistic, it randomly picks one user-agent from the list we loaded earlier. This user-agent is logged so you know which identity the scraper is using for that session. Then, it opens a Chromium browser—not in headless mode, which means you can actually see the browser on your screen as it loads pages. This is especially helpful during testing or when something isn’t working and you want to watch what’s happening.

The function gives back three things: the Playwright instance (which manages everything), the browser object, and the page object (used to load and interact with websites).

If something goes wrong—like the browser fails to start—the function doesn’t just stop silently. It logs the issue and raises an error, making sure the script doesn’t continue with a broken setup. This way, the scraper acts more like a real person visiting the site, which helps avoid detection and makes debugging easier when needed.

Initializing the Database

def initialize_database():
    """
    Initialize SQLite database and create the required table structure.

    Returns:
        sqlite3.Connection: Database connection object

    Raises:
        sqlite3.Error: If database initialization or table creation fails

    Notes:
        - Creates a new database file 'lazada.db' if it doesn't exist
        - Creates a table 'lazada_products' with columns:
            * id (INTEGER PRIMARY KEY AUTOINCREMENT)
            * link (TEXT UNIQUE)
        - The UNIQUE constraint prevents duplicate product links
    """
    try:
        conn = sqlite3.connect("lazada.db")
        cursor = conn.cursor()
        cursor.execute("""
            CREATE TABLE IF NOT EXISTS lazada_products (
                id INTEGER PRIMARY KEY AUTOINCREMENT,
                link TEXT UNIQUE
            )
        """)
        conn.commit()
        return conn
    except sqlite3.Error as e:
        logging.error(f"Database initialization failed: {e}")
        raise

The initialize_database() function takes care of setting up a place to store the product links we’re collecting. It connects to an SQLite database, which is a simple, file-based database that works perfectly for small to medium projects like this.

Inside the database, it checks if a table called lazada_products exists. If not, it creates one with two columns: an id that automatically increases for each entry, and a link column to store the product URL. The link column is set to be unique, which means duplicate URLs won’t be added more than once.

Using a database like this is much more reliable than saving links in plain text files. It allows you to search, filter, and organize the data easily—especially helpful when the project grows. And if something goes wrong while setting up the database, the script doesn’t just fail quietly—it logs the problem so you’ll know exactly what needs fixing.

Fetching Page Content

def fetch_page_content(page, url):
    """
    Navigate to a URL and retrieve the page content using Playwright.

    Args:
        page (playwright.sync_api.Page): Initialized Playwright page object
        url (str): The URL to fetch content from

    Returns:
        str or None: HTML content of the page if successful, None if failed

    Notes:
        - Sets a high timeout (20 minutes) to handle slow-loading pages
        - Includes a 2-second delay after navigation to allow dynamic content to load
        - Logs errors if page fetch fails

    Rate Limiting:
        - Includes a 2-second fixed delay to prevent overwhelming the server
    """
    try:
        page.goto(url, timeout=1200000)  # Set a timeout to avoid indefinite waits
        time.sleep(2)  # Allow time for the page to load
        return page.content()
    except Exception as e:
        logging.error(f"Failed to fetch content from {url}: {e}")
        return None

The fetch_page_content(page, url) function handles the job of opening a web page and getting its full HTML content—just like what you'd see if you visited the page in your own browser.

It uses Playwright to open the given URL and sets a long timeout (up to 20 minutes) to make sure even slow-loading pages have enough time to load. After the page loads, the script waits an extra two seconds. This short pause is important because many e-commerce sites use JavaScript to load content after the page first appears. Without this wait, we might miss out on important product details.

Unlike simple HTTP requests that only grab the raw page source, Playwright gives us the fully rendered version, just like what an actual user would see. And if something goes wrong—like a slow network or a page that fails to load—an error is recorded in the logs. This way, the script won’t crash unexpectedly, and you’ll know where the problem occurred.

Parsing Product Links

def parse_product_links(html_content):
    """
    Extract product links from Lazada's HTML content using BeautifulSoup.

    Args:
        html_content (str): Raw HTML content of the page

    Returns:
        list: List of product URLs found on the page

    Notes:
        - Uses CSS selector to find product card elements
        - Handles relative URLs by prepending 'https:' when necessary
        - Returns empty list if parsing fails or no links found
    """
    try:
        soup = BeautifulSoup(html_content, "html.parser")
        product_links = []
        product_cards = soup.select("#root > div > div.ant-row.FrEdP.css-1bkhbmc.app > div:nth-child(1) > div > div.ant-col.ant-col-20.ant-col-push-4.Jv5R8.css-1bkhbmc.app > div._17mcb > div > div > div > div.ICdUp > div > a")  # Adjust selector for product links

        for card in product_cards:
            link = card.get("href")
            if link and not link.startswith("https:"):
                link = "https:" + link
            if link:
                product_links.append(link)

        return product_links
    except Exception as e:
        logging.error(f"Error parsing HTML content: {e}")
        return []

The parse_product_links(html_content) function is where the actual scraping happens. It takes the full HTML of a page and uses BeautifulSoup to look through the content and find product links.

It searches for specific parts of the page—called product cards—using a CSS selector. From each product card, it grabs the href value, which is the link to the individual product. Sometimes these links are incomplete (starting with just a /), so the function adds https: at the beginning to make them complete and usable.

This step helps us pull out only the links we care about from the rest of the page’s structure. Since websites often change their design, the CSS selector used here should be reviewed from time to time to make sure it's still accurate.

If anything goes wrong—like the expected structure isn’t found—the function won’t break the whole script. Instead, it logs an error and returns an empty list. All collected links are cleaned and standardized before storing them in the database, keeping everything consistent and easy to work with later.

Saving Links to the Database

def save_links_to_database(conn, links):
    """
    Save extracted product links to SQLite database.

    Args:
        conn (sqlite3.Connection): Database connection object
        links (list): List of product URLs to save

    Notes:
        - Uses INSERT OR IGNORE to handle duplicate links gracefully
        - Commits transaction after all insertions
        - Logs any errors during insertion process
        - Does not close the database connection (handled by caller)

    Database Schema:
        Table: lazada_products
        Columns: 
            - id (INTEGER PRIMARY KEY AUTOINCREMENT)
            - link (TEXT UNIQUE)
    """
    cursor = conn.cursor()
    for link in links:
        try:
            cursor.execute("INSERT OR IGNORE INTO lazada_products (link) VALUES (?)", (link,))
        except sqlite3.Error as e:
            logging.error(f"Error saving link {link}: {e}")
    conn.commit()

The save_links_to_database(conn, links) function takes the list of product links we’ve scraped and saves them into the SQLite database.

It uses an SQL command called INSERT OR IGNORE, which is helpful because it avoids saving the same link more than once. If a link is already in the database, the command simply skips it instead of causing an error. This keeps the database clean and free of duplicates.

Each time a link is added, the function commits the change—basically telling the database, “Save this now.” This way, even if the script suddenly stops running, the links that were already inserted won’t be lost.

If there’s any trouble while inserting a link—maybe due to a database issue—it gets logged. That way, we know something went wrong without the whole script crashing. In the bigger picture, this function makes sure every product URL we collect gets safely stored and is ready for any later steps, like analysis or further scraping.

Implementing a Random Delay

def random_delay(min_delay=2, max_delay=4):
    """
    Implement a random delay between web requests for rate limiting.

    Args:
        min_delay (float): Minimum delay in seconds (default: 2)
        max_delay (float): Maximum delay in seconds (default: 4)

    Notes:
        - Uses uniform distribution for randomization
        - Logs the actual delay duration for monitoring
        - Helps prevent detection of automated scraping
        - Recommended to adjust delay range based on website's terms of service
    """
    delay = random.uniform(min_delay, max_delay)
    logging.info(f"Sleeping for {delay:.2f} seconds.")
    time.sleep(delay)

The random_delay(min_delay, max_delay) function adds a randomly introduced delay between requests, simulating human web browsing behavior. It records the actual delay period and suspends execution with time.sleep().

Rate limiting is crucial in web scraping to avoid detection and IP blocks. By incorporating small, random pauses, the scraper makes it less likely to activate anti-bot measures. Varying the delay range can assist in finding a balance between efficiency and stealth, based on the target site's terms of use. The fact that randomized delays are included helps the scraping process seem more natural, making it less likely to be blocked.

Scraping Lazada Product Links

def scrape_lazada_links(base_url, start_page=1, end_page=102, user_agents_file="user_agents.txt"):
    """
    Main scraping function to extract product links from multiple Lazada pages.

    Args:
        base_url (str): URL template with {page_no} placeholder for pagination
        start_page (int): First page number to scrape (default: 1)
        end_page (int): Last page number to scrape (default: 102)
        user_agents_file (str): Path to file containing user agents (default: "user_agents.txt")

    Notes:
        - Initializes browser with random user agent rotation
        - Creates/connects to SQLite database for storing links
        - Implements rate limiting with random delays between requests
        - Handles errors gracefully and logs all activities
        - Ensures proper cleanup of resources in all scenarios

    Process Flow:
        1. Load user agents and initialize browser
        2. Set up database connection
        3. Iterate through pages and extract product links
        4. Save links to database
        5. Clean up resources

    Rate Limiting:
        - Implements random delays between requests
        - Adjustable through random_delay() function parameters
    """
    user_agents = load_user_agents(user_agents_file)
    if not user_agents:
        logging.critical("No user agents available. Exiting the scraper.")
        return

    playwright, browser, page = initialize_browser(user_agents)
    conn = initialize_database()

    try:
        for page_no in range(start_page, end_page+1):
            current_url = base_url.format(page_no=page_no)
            logging.info(f"Scraping page: {current_url}")
            html_content = fetch_page_content(page, current_url)

            if html_content:
                product_links = parse_product_links(html_content)
                if product_links:
                    save_links_to_database(conn, product_links)
                else:
                    logging.warning(f"No product links found on page {current_url}.")
            else:
                logging.error(f"Failed to fetch or parse content from page {current_url}. Skipping.")
            # Add a random delay between requests
            random_delay()
            
    except Exception as e:
        logging.critical(f"Critical error during scraping: {e}")
    finally:
        conn.close()
        browser.close()
        playwright.stop()
        logging.info("Scraping process completed.")

The scrape_lazada_links(base_url, start_page, end_page, user_agents_file) function is the heart of the scraper. It runs the entire scraping process from start to finish.

Here's what it does in order:

• Loads user agents from a file.

• Starts the Playwright browser and SQLite database.

• Loops through the specified page range.

• For each page, it:

  • Builds the URL.

  • Loads and waits for content.

  • Extracts product links.

  • Saves them to the database.

  • Waits randomly before the next request (to avoid detection).

• Handles any errors during scraping so the script doesn’t crash midway.

• Cleans up at the end by closing the browser, database, and stopping Playwright.

  
  

If no user agents are found, it logs a fatal error and stops early.

This function ties all parts together—link extraction, storage, and browser automation—into a controlled and reusable workflow. Its modular structure means it’s easy to adjust, extend, or debug when needed.

Running the Scraper

The script includes a main execution block:

if __name__ == "__main__":
    BASE_URL = "https://www.lazada.sg/skincare/?page={page_no}&spm=a2o42.tm80139881.cate_4.1.7d83yXEryXErvj"
    try:
        scrape_lazada_links(BASE_URL, start_page=1, end_page=102)
        logging.info("Scraping finished successfully.")
    except Exception as e:
        logging.critical(f"Scraper failed: {e}")

This final block ensures that the script only runs when you execute it directly—like running it from a terminal or command line. It acts like a gatekeeper, making sure the scraping process doesn't accidentally start when the file is imported somewhere else.

The BASE_URL here is a template that includes {page_no}, which gets replaced by actual page numbers as the scraper loops through them. In this case, it scrapes from page 1 to 102. Throughout the process, the script keeps an eye out for serious issues—logging any major failures that happen. If all goes well, it logs a success message once everything is done.

This setup makes the script clean and manageable. It starts only when you want it to and helps catch problems early.

Now that we’ve collected all the product links, the next big step is to extract detailed information from each of those product pages. In the upcoming sections, we’ll walk through how the script approaches that, breaking it down into clear steps to make the process easy to follow.

Product Data Scraping

Import Section

import logging
import sqlite3
import time
import random
from playwright.sync_api import sync_playwright, TimeoutError as PlaywrightTimeoutError
from bs4 import BeautifulSoup

# Configure logging
logging.basicConfig(
    filename="lazada_scraper_data.log",
    filemode="a",
    format="%(asctime)s - %(levelname)s - %(message)s",
    level=logging.INFO
)

Just like before, we start by importing the required libraries—only this time, they’re geared toward pulling detailed information from each product page. We reuse the same tools, but now they focus on extracting rich product data instead of just the links. As expected, logging is set up again to track the script’s activity, helping us catch issues and understand what’s happening behind the scenes.

initialize_browser Function

def initialize_browser(user_agents):
    """
    Initialize a Playwright browser instance with a randomly selected user agent.

    Args:
        user_agents (list): List of user agent strings to choose from

    Returns:
        tuple: (playwright instance, browser instance, page instance)
            - playwright: The Playwright context manager
            - browser: The Chromium browser instance
            - page: The initialized page object

    Raises:
        Exception: If browser initialization fails
        IndexError: If user_agents list is empty

    Notes:
        - Launches browser in non-headless mode for debugging
        - Uses random user agent for each session to avoid detection
        - Creates a new browser context for clean session management
    """
    try:
        user_agent = random.choice(user_agents)
        playwright = sync_playwright().start()
        browser = playwright.chromium.launch(headless=False)
        context = browser.new_context(user_agent=user_agent)
        page = context.new_page()
        return playwright, browser, page
    except Exception as e:
        logging.error(f"Failed to initialize browser: {e}")
        raise

This method sets up the browser environment where the scraper will run. It starts by picking a random user agent from the provided list. This user agent mimics real browsers and helps the scraper avoid detection by the website. Randomizing it reduces the chance of getting blocked.

Next, it initializes Playwright and opens a Chromium browser in non-headless mode, which means the browser window is visible. This is helpful for debugging, though in production it can be switched to headless mode to run in the background.

It then sets the browser context with the selected user agent and opens a new page for navigation. If anything goes wrong—like Playwright fails to launch or no user agent is available—it logs the error and raises an exception. This makes sure the scraping only continues when everything is properly set up.

load_user_agents Function

def load_user_agents(file_path):
    """
    Load user agent strings from a text file.

    Args:
        file_path (str): Path to the text file containing user agents

    Returns:
        list: List of user agent strings

    Raises:
        FileNotFoundError: If the specified file doesn't exist
        IOError: If there are issues reading the file

    Notes:
        - Each user agent should be on a separate line in the file
        - Empty lines are filtered out
        - Logs warning if file is empty
    """
    try:
        with open(file_path, "r") as f:
            user_agents = f.read().splitlines()
        if user_agents:
            return user_agents
        else:
            logging.warning("No user agents found in the file.")
            return []
    except FileNotFoundError as e:
        logging.error(f"User agents file not found: {e}")
        raise

The load_user_agents function loads the user-agent strings that help the scraper mimic different browsers and devices. These strings are used to make each request look like it’s coming from a real user, reducing the chance of getting blocked.

It reads from a text file given by file_path, where each line is a user agent. It filters out any empty lines and returns a clean list. If the file is empty or missing, it logs a warning or error. If something goes wrong during reading, it logs the issue and raises an exception. This setup ensures the scraper always has a pool of user agents to rotate through for safer scraping.

initialize_database Function

def initialize_database():
    """
    Initialize SQLite database and set up the required tables structure.

    Returns:
        sqlite3.Connection: Database connection object

    Raises:
        sqlite3.Error: If database operations fail

    Database Schema:
        Table: lazada_products
            - Existing table with added 'scraped' column (INTEGER DEFAULT 0)
        
        Table: lazada_product_details
            - id (INTEGER PRIMARY KEY AUTOINCREMENT)
            - link (TEXT UNIQUE)
            - title (TEXT)
            - brand (TEXT)
            - sale_price (TEXT)
            - price (TEXT)
            - discount (TEXT)
            - reviews (TEXT)

    Notes:
        - Checks for and adds 'scraped' column if missing
        - Creates product details table if it doesn't exist
        - Uses transactions for data consistency
    """
    try:
        conn = sqlite3.connect("lazada.db")
        cursor = conn.cursor()

        # Ensure `scraped` column exists in `lazada_products` table
        cursor.execute("PRAGMA table_info(lazada_products)")
        columns = [column[1] for column in cursor.fetchall()]
        if "scraped" not in columns:
            cursor.execute("ALTER TABLE lazada_products ADD COLUMN scraped INTEGER DEFAULT 0")
            logging.info("Added 'scraped' column to lazada_products table.")

        # Create new table for scraped data
        cursor.execute("""
            CREATE TABLE IF NOT EXISTS lazada_product_details (
                id INTEGER PRIMARY KEY AUTOINCREMENT,
                link TEXT UNIQUE,
                title TEXT,
                brand TEXT,
                sale_price TEXT,
                price TEXT,
                discount TEXT,
                reviews TEXT
            )
        """)
        logging.info("Initialized lazada_product_details table.")
        conn.commit()
        return conn
    except sqlite3.Error as e:
        logging.error(f"Database initialization failed: {e}")
        raise

The initialize_database function prepares the SQLite database that will store all the scraped product data. It opens a database file named lazada.db, creating it if it doesn't already exist. This database is where all links and product details will be safely stored for later analysis.

First, the function checks if the scraped column exists in the lazada_products table. It does this using a special SQL command (PRAGMA table_info) that looks at the table’s structure. If the scraped column isn’t found, it adds it and sets its default value to 0, meaning the product hasn’t been scraped yet. This step ensures the table is ready for the scraping process.

Next, it sets up a new table called lazada_product_details, where the actual product data will go. This table includes columns like id, link, title, brand, sale_price, price, discount, and reviews. These fields are important for tracking product information over time or generating reports.

If any errors happen while setting up the tables—like SQL mistakes or connection issues—they're logged so you can easily find and fix the problem. This function makes sure the database is properly structured before the scraper starts collecting product details.

fetch_unscraped_links Function

def fetch_unscraped_links(conn):
    """
    Retrieve all unscraped product links from the database.

    Args:
        conn (sqlite3.Connection): Database connection object

    Returns:
        list: List of tuples containing unscraped product links

    Raises:
        sqlite3.Error: If database query fails

    Notes:
        - Returns links where 'scraped' column is 0
        - Returns empty list if no unscraped links found
    """
    try:
        cursor = conn.cursor()
        cursor.execute("SELECT link FROM lazada_products WHERE scraped = 0")
        return cursor.fetchall()
    except sqlite3.Error as e:
        logging.error(f"Failed to fetch unscraped links: {e}")
        raise

The fetch_unscraped_links function is responsible for getting the list of product links that haven’t been scraped yet. It looks in the lazada_products table and selects all rows where the scraped column is set to 0. This means the product link is still new and hasn’t been processed.

By doing this, the scraper avoids repeating the same work or putting extra load on the database. It ensures that only fresh, unprocessed links are used during the scraping run.

The function returns the result as a list of tuples, which the main scraping loop goes through one by one. If something goes wrong—like a database error or a bad SQL query—the issue is logged, and an exception is raised so it doesn’t go unnoticed. This function helps keep the workflow clean and efficient by focusing only on products that are yet to be scraped.

fetch_page_content Function

def fetch_page_content(page, url):
    """
    Navigate to a product page and retrieve its content.

    Args:
        page (playwright.Page): Playwright page object
        url (str): Product page URL to fetch

    Returns:
        str or None: HTML content of the page if successful, None if failed

    Raises:
        PlaywrightTimeoutError: If page load exceeds timeout
        Exception: For other navigation errors

    Notes:
        - Sets 60-second timeout for initial page load
        - Includes 30-second delay for dynamic content loading
        - Returns None if any errors occur during fetch
    """
    try:
        page.goto(url, timeout=60000)
        time.sleep(30)  # Wait for the page to load
        return page.content()
    except PlaywrightTimeoutError:
        logging.error(f"Page timeout while fetching URL: {url}")
        return None
    except Exception as e:
        logging.error(f"Failed to fetch content from {url}: {e}")
        return None

The fetch_page_content function is used to load the HTML content of a product page. It uses Playwright’s page.goto method to open the provided URL and waits until the page has fully loaded. A timeout is set to 60 seconds—so if the page takes longer than that, the script stops trying and logs an error.

After the page finishes loading, the script waits an extra 30 seconds using time.sleep(30). This pause is important because many online stores load extra content—like images, prices, or reviews—after the initial page load using JavaScript. Waiting gives everything time to appear before we start scraping.

If anything goes wrong—like the page doesn’t load in time or another error occurs—it’s caught and logged so you can understand what happened. If everything works smoothly, the function returns the full HTML content of the page as a string. This content will later be parsed to pull out detailed product information. This step is key to making sure the scraper is actually seeing the same content a real user would.

parse_title, parse_brand, parse_prices, and parse_ratings Functions

def parse_title(soup):
    """
    Extract product title from the page content.

    Args:
        soup (BeautifulSoup): Parsed HTML content

    Returns:
        str: Product title or 'N/A' if not found

    Notes:
        - Uses CSS selector: 'h1.pdp-mod-product-badge-title'
        - Returns 'N/A' if title element not found
    """
    title = soup.select_one("h1.pdp-mod-product-badge-title") 
    return title.get_text() if title else "N/A"

def parse_brand(soup):
    """
    Extract product brand from the page content.

    Args:
        soup (BeautifulSoup): Parsed HTML content

    Returns:
        str: Brand name or 'N/A' if not found

    Notes:
        - Uses CSS selector for brand link element
        - Returns 'N/A' if brand element not found
    """
    brand = soup.select_one("a.pdp-link.pdp-link_size_s.pdp-link_theme_blue.pdp-product-brand__brand-link") 
    return brand.get_text() if brand else "N/A"


def parse_prices(soup):
    """
    Extract price information from the product page.

    Args:
        soup (BeautifulSoup): Parsed HTML content

    Returns:
        tuple: (sale_price, original_price, discount)
            - sale_price (str): Current sale price or 'N/A'
            - original_price (str): Original price or 'N/A'
            - discount (str): Discount percentage or 'N/A'

    Notes:
        - Extracts three price-related elements using specific CSS selectors
        - Returns 'N/A' for any missing price information
    """
    sale_price = soup.select_one("#module_product_price_1 > div > div > span") 
    price=soup.select_one("#module_product_price_1 > div > div > div > span.notranslate.pdp-price.pdp-price_type_deleted.pdp-price_color_lightgray.pdp-price_size_xs")
    discount=soup.select_one("#module_product_price_1 > div > div > div > span.pdp-product-price__discount")
    return sale_price.get_text() if sale_price else "N/A",price.get_text() if price else "N/A",discount.get_text() if discount else "N/A"

def parse_ratings(soup):
    """
    Extract product review information.

    Args:
        soup (BeautifulSoup): Parsed HTML content

    Returns:
        str: Review information or 'N/A' if not found

    Notes:
        - Uses CSS selector for review element
        - Returns 'N/A' if review information not found
    """
    reviews=soup.select_one("#module_product_review_star_1 > div > a")
    return reviews.get_text() if reviews else "N/A"

These functions are designed to pull specific details—like title, brand, price, and ratings—from each product page. They all use BeautifulSoup, a Python library that helps read and navigate the HTML content of a web page.

The parse_title function looks for the product’s name by targeting a specific HTML element: h1.pdp-mod-product-badge-title. If that element is found, it returns the text inside it. If not, it simply returns "N/A" so that the script doesn’t fail.

Similarly, the parse_brand function searches for the product’s brand using the brand link element. If it can’t find the brand, it also returns "N/A".

The parse_prices function collects three price-related values: the sale price, the original price, and the discount percentage. It uses CSS selectors to locate each of these. If any piece of this price data is missing, it substitutes "N/A" just for that part, allowing the rest of the data to be saved.

Lastly, the parse_ratings function tries to find review or rating information from the page. Again, if the expected element is missing, it returns "N/A" instead of crashing.

Together, these functions make sure the scraper collects all the available product details in a structured and dependable way. And when something is missing on the page, the default values ensure the scraper keeps running smoothly.

save_scraped_data Function

def save_scraped_data(conn, link, data):
    """
    Save extracted product data to the database.

    Args:
        conn (sqlite3.Connection): Database connection object
        link (str): Product URL
        data (dict): Dictionary containing product details:
            - title: Product title
            - brand: Product brand
            - sale_price: Current sale price
            - price: Original price
            - discount: Discount percentage
            - reviews: Review information

    Raises:
        sqlite3.Error: If database insertion fails

    Notes:
        - Uses parameterized query to prevent SQL injection
        - Logs successful data saves and errors
    """
    try:
        cursor = conn.cursor()
        cursor.execute("""
            INSERT INTO lazada_product_details (link, title, brand,sale_price, price, discount, reviews)
            VALUES (?, ?, ?, ?, ?, ?, ?)
        """, (link, data["title"], data["brand"], data["sale_price"], data["price"], data["discount"], data["reviews"]))
        conn.commit()
        logging.info(f"Saved scraped data for link: {link}.")
    except sqlite3.Error as e:
        logging.error(f"Error saving data for link {link}: {e}")

The save_scraped_data function takes the detailed product information we’ve collected and saves it into the SQLite database. It receives the data as a dictionary and uses a parameterized SQL query to insert it into the lazada_product_details table. Using this method helps keep the database secure by preventing SQL injection.

Inside the function, the data is passed into cursor.execute() for insertion. After that, conn.commit() is called to make sure the changes are saved permanently in the database.

If something goes wrong—like trying to insert a duplicate entry or if there’s an issue with the SQL query—it’s caught and logged without stopping the script.

This function ensures that every product’s data is stored reliably, so it can be used later for analysis, reporting, or any other tasks.

mark_link_as_scraped Function

def mark_link_as_scraped(conn, link):
    """
    Update the scraped status of a product link in the database.

    Args:
        conn (sqlite3.Connection): Database connection object
        link (str): Product URL to mark as scraped

    Raises:
        sqlite3.Error: If database update fails

    Notes:
        - Sets 'scraped' column to 1 for the specified link
        - Logs successful updates and errors
    """
    try:
        cursor = conn.cursor()
        cursor.execute("UPDATE lazada_products SET scraped = 1 WHERE link = ?", (link,))
        conn.commit()
        logging.info(f"Marked link as scraped: {link}.")
    except sqlite3.Error as e:
        logging.error(f"Error marking link as scraped: {e}")

The mark_link_as_scraped function helps the scraper keep track of its progress. After a product link has been successfully scraped, this function updates the scraped column in the lazada_products table to 1, meaning “this link has been processed.”

It does this by running an SQL UPDATE statement, then commits the change to make sure it's saved. This prevents the scraper from visiting the same product page again in future runs, saving time and avoiding duplicate work.

If there’s a problem—like a connection issue or update error—it gets caught and logged. This function is essential for keeping the scraping process organized and efficient.

random_delay Function

def random_delay():
    """
    Implement random delay between requests for rate limiting.

    Notes:
        - Generates random delay between 2 and 5 seconds
        - Logs the actual delay duration
        - Helps prevent detection of automated scraping
        - Catches and logs any errors during delay

    Raises:
        Exception: If sleep operation fails
    """
    try:
        delay = random.uniform(2, 5)  # Random delay between 2 and 5 seconds
        logging.info(f"Waiting for {delay:.2f} seconds.")
        time.sleep(delay)
    except Exception as e:
        logging.error(f"Error during delay: {e}")

The random_delay function adds a short, random pause between each request the scraper makes. This is important because sending too many requests too quickly can get the scraper blocked or trigger rate limits on the website.

The function picks a random number between 2 and 5 seconds using random.uniform() and then logs the delay so you can see how long the script is waiting. It then pauses the script using time.sleep().

This small delay helps the scraper behave more like a real person browsing the site, making it less likely to be flagged as a bot. And if anything goes wrong during the pause, like an error with the sleep function, it’s caught and logged.

scrape_product_data Function

def scrape_product_data():
    """
    Main function to orchestrate the product data scraping process.

    Process Flow:
        1. Load user agents and initialize browser
        2. Set up database connection
        3. Fetch unscraped product links
        4. For each link:
            - Fetch page content
            - Parse product details
            - Save data to database
            - Mark link as scraped
            - Apply rate limiting delay
        5. Clean up resources

    Error Handling:
        - Handles browser initialization failures
        - Manages database connection errors
        - Catches parsing and scraping exceptions
        - Ensures proper resource cleanup

    Notes:
        - Uses random user agents for each session
        - Implements rate limiting between requests
        - Logs all major operations and errors
        - Ensures proper cleanup in case of failures
    """
    user_agents = load_user_agents("user_agents.txt")
    if not user_agents:
        logging.critical("No user agents available. Exiting scraper.")
        return

    playwright, browser, page = None, None, None
    conn = None

    try:
        playwright, browser, page = initialize_browser(user_agents)
        conn = initialize_database()

        links = fetch_unscraped_links(conn)
        for (link,) in links:
            logging.info(f"Scraping link: {link}")
            html_content = fetch_page_content(page, link)

            if html_content:
                soup = BeautifulSoup(html_content, "html.parser")
                data = {
                    "title": parse_title(soup),
                    "brand": parse_brand(soup),
                    "sale_price": parse_prices(soup)[0],
                    "price": parse_prices(soup)[1],
                    "discount": parse_prices(soup)[2],
                    "reviews": parse_ratings(soup)
                }

                if data:
                    save_scraped_data(conn, link, data)
                    mark_link_as_scraped(conn, link)
                else:
                    logging.warning(f"No data found for link: {link}.")
            else:
                logging.error(f"Failed to fetch content for link: {link}.")

            random_delay()

    except Exception as e:
        logging.critical(f"Critical error during scraping: {e}")
    finally:
        if conn:
            conn.close()
        if browser:
            browser.close()
        if playwright:
            playwright.stop()
        logging.info("Scraping process completed.")

The scrape_product_data function is the core of the entire scraping workflow. It brings all the pieces together—from setting up the tools to collecting and saving the data.

It starts by loading the user agents and setting up both the browser and the database. Then, it pulls a list of product links that haven’t been scraped yet.

For each link in that list, it follows these steps:

1. Loads the page using fetch_page_content.

2. Parses the HTML with BeautifulSoup.

3. Extracts product details like title, brand, price, and ratings using the parsing functions.

4. Saves the extracted data into the database using save_scraped_data.

5. Marks the link as scraped using mark_link_as_scraped.

   
   

To avoid getting blocked by the site, the function also adds a random delay between each request using random_delay. If something goes wrong while processing a link—like a missing page or an unexpected structure—it logs the error and moves on to the next link.

This setup makes the scraper dependable and efficient, allowing it to continue running smoothly even if a few links fail along the way.

Running the Scraper

if __name__ == "__main__":
    try:
        scrape_product_data()
        logging.info("Scraping finished successfully.")
    except Exception as e:
        logging.critical(f"Scraper failed: {e}")

The if name == "__main__": block acts as the script’s starting point. It ensures that the scrape_product_data function runs only when the script is executed directly—not when it's imported into another script.

Inside this block, the script tries to run the scraping process using a try block. If everything goes well, a success message is logged to confirm that the scraping finished without issues. But if something goes wrong, the except block catches the error and logs it as a critical failure, along with the full error details.

This setup makes the script more reliable. It prevents crashes from going unnoticed, helps with debugging, and keeps the scraping process easy to manage and maintain.

Conclusion

In summary, this Lazada web scraping script offers a well-organized and reliable way to automate the collection of product links and detailed data. It smartly uses tools like Playwright to handle dynamic websites and BeautifulSoup to extract key information from the page—such as product titles, prices, brands, and reviews.

All collected data is stored neatly in an SQLite database, making it easy to access later for analysis or reporting. The script is built with thoughtful features like logging, error handling, and random delays to stay under the radar and avoid detection by the website.

What makes this setup strong is its attention to both functionality and stability. It’s designed to handle changes, prevent crashes, and run responsibly—respecting ethical scraping practices like pacing requests and managing system resources properly.

Altogether, the script creates a smooth, flexible, and beginner-friendly scraping experience that’s not just effective—but also dependable.

AUTHOR

I’m Shahana, a Data Engineer at Datahut, where I focus on building clean and scalable data pipelines that turn unstructured web content into valuable datasets—especially for use cases in e-commerce, retail intelligence, and product tracking.

At Datahut, we work closely with clients to automate data extraction from websites, even those that use dynamic elements like JavaScript and infinite scroll. In this blog, I shared a real-world example of how we scraped product data from Lazada using Playwright, BeautifulSoup, and SQLite. The solution was built to handle dynamic page content, structured storage, and responsible scraping practices—while keeping the code modular and easy to understand, especially for those just getting started.

If your team is looking to automate product data collection in the eyewear space or beyond, reach out to us through the chat widget on the right. We’d love to help you build a solution that fits your goals.

Frequently Asked Questions

1. Is it legal to scrape product pricing data from Noon?

Yes, scraping publicly available product information can be legal when done responsibly and in compliance with a website's Terms of Service and applicable laws. Always review the site's policies before collecting data at scale. If you're unsure, read our guide on Is Web Scraping Legal?:https://www.blog.datahut.co/post/is-web-scraping-legal

2. Why does this tutorial use Playwright instead of Requests?

Many modern e-commerce websites, including Noon, load content dynamically with JavaScript. Playwright renders the page like a real browser, making it easier to extract complete product information. You can learn more in the official Playwright documentation:https://playwright.dev/python/

3. Can I scrape other e-commerce websites using the same approach?

Yes. The same workflow—collecting product URLs first and then extracting product details—works for many online stores. You'll only need to update the CSS selectors and pagination logic to match the target website. You may also find our guide on How to Scrape H&M Product Data Using Python helpful:https://www.blog.datahut.co/post/how-to-scrape-h-m-product-data-using-python

4. How can I avoid getting blocked while scraping websites?

Some best practices include rotating user agents, adding random delays between requests, respecting rate limits, and avoiding unnecessary requests. For enterprise-scale projects, these techniques are often combined with proxy rotation and browser fingerprint management.

5. What can businesses do with scraped pricing data?

Pricing data can be used for competitor price monitoring, assortment analysis, dynamic pricing, market research, and trend analysis. Learn more about competitor price monitoring here:https://www.blog.datahut.co/post/monitor-competitor-prices-automatically-for-free-no-code-web-scraping-with-n8n

6. Why is SQLite used in this tutorial?

SQLite is lightweight, easy to set up, and ideal for learning or small scraping projects. As your scraper grows, you can migrate to databases such as PostgreSQL or MySQL for improved scalability and performance.

7. Can I scrape thousands of product pages with this script?

Yes, but you'll need additional features such as proxy rotation, retry mechanisms, browser session management, and distributed scraping infrastructure. If you're collecting data at enterprise scale, consider using a professional web scraping service:https://www.datahut.co/web-scraping-services

8. Where can I learn more about Python web scraping?

If you're just getting started, check out our beginner-friendly tutorials on How to Build a Web Crawler in Python from Scratch:https://www.blog.datahut.co/post/how-to-build-a-web-crawler-from-scratch and Top Python Web Scraping Libraries:https://www.blog.datahut.co/post/top-python-web-scraping-libraries

These guides explain the core concepts you'll use in projects like this one.