The unglamorous part of RAG

Everyone wants to talk about chunking strategies and which embedding model retrieves better. Almost nobody wants to talk about where the data came from in the first place, and in practice that is the part that decides whether the rest of the system is worth building at all. If the source data is stale, duplicated, or half broken, no amount of retrieval tuning fixes it downstream.

Part of my job at Workharu is exactly that: building the scraping pipelines that feed our RAG system with new data. It is less interesting to explain than a vector database, and more likely to break at three in the morning.

Sites do not want to be scraped consistently

The first lesson, and the one every scraping tutorial skips, is that the target site is not a stable API. It is a UI built for humans, and it changes without warning. A class name gets renamed during a redesign, pagination switches from page numbers to infinite scroll, or a piece of content that used to be server rendered starts loading in through a client side fetch after the fact.

None of these show up as errors most of the time. The scraper runs, returns a 200, and quietly hands you an empty list or half the fields you expect. Silent failure is worse than a crash, because a crash gets noticed.

What actually helps:

def extract_field(soup, selector, field_name):
    el = soup.select_one(selector)
    if el is None:
        logger.warning(f"missing field: {field_name} at {selector}")
        return None
    return el.get_text(strip=True)

Every extraction gets logged when it comes back empty, and a daily job checks the ratio of missing fields against a rolling baseline. A sudden jump means something upstream changed, long before it becomes a data quality problem someone else notices.

Static HTML, and when it stops being enough

Most of what I scrape is straightforward with requests and BeautifulSoup. But a meaningful chunk of sites render their actual content client side, and no amount of parsing the raw HTML response finds it because it simply is not there yet.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions()
options.add_argument("--headless")
driver = webdriver.Chrome(options=options)

driver.get(url)
WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, ".content-loaded"))
)

Selenium is slower and heavier than a plain HTTP request by an order of magnitude, so it is a fallback, not the default. The pipeline tries the fast path first and only reaches for a real browser when the fast path comes back with nothing useful. Reaching for Selenium everywhere because it always works is how a scraping job that should take ten minutes ends up taking three hours.

Rate limits are not a courtesy, they are survival

Hammering a site with concurrent requests gets you blocked, and getting blocked is far more expensive to recover from than scraping slower to begin with. A rotating pool of delays, a respectful concurrency limit, and honoring robots.txt where it exists are not optional extras, they are what keeps a pipeline running unattended for months instead of getting an IP range banned in a week.

import asyncio
import random

async def polite_fetch(session, url, semaphore):
    async with semaphore:
        await asyncio.sleep(random.uniform(0.5, 2.0))
        async with session.get(url) as resp:
            return await resp.text()

semaphore = asyncio.Semaphore(4)

Four concurrent requests with a randomized delay between them is not fast. It is fast enough, and it does not get flagged as a bot pattern the way a fixed interval does.

Deduplication is where most of the actual engineering lives

A scraper that runs on a schedule sees the same content again and again. Re-processing everything on every run wastes compute and, worse, can duplicate entries in whatever gets indexed downstream. The fix is a content hash checked before anything gets written:

import hashlib

def content_hash(text: str) -> str:
    normalized = " ".join(text.split()).lower()
    return hashlib.sha256(normalized.encode()).hexdigest()

Normalizing whitespace and case before hashing matters more than it sounds like it should. Two pages that differ only by a trailing space or inconsistent capitalization otherwise hash as different content and slip past the check entirely.

What actually goes into the RAG system

By the time scraped content reaches the parsing and chunking stage I wrote about separately, it has already been through field validation, deduplication, and a cleanup pass that strips navigation chrome, ads, and boilerplate that has nothing to do with the actual content. Skipping that cleanup and dumping raw scraped HTML straight into a retrieval pipeline is the fastest way to get a system that sounds confident while citing a cookie banner as its source.

The unglamorous conclusion: retrieval quality is mostly a function of how disciplined the pipeline feeding it is, not how clever the model on top is. I spend more time thinking about what could silently go wrong in a scraper than about which embedding model to use, and that ratio has only gotten more lopsided the longer I have done this.