Practice: Design Web Crawler

Web crawler in the style of the Mercator design used by search engines: a durable URL frontier, per-host politeness queues with robots.txt and DNS caches, distributed fetchers, Bloom-filter URL deduplication, SimHash content deduplication, and change-based recrawl scheduling. A teaching design.

Build this system yourself on the canvas below. Add components, connect them, and run traffic simulations to see where your design holds up and where it breaks.

Functional requirements

  • Seed initial URLs from a curated list of high-value domains
  • Fetch and render HTML pages (including JavaScript-rendered content)
  • Extract all outbound links from crawled pages
  • Filter extracted URLs — remove duplicates, invalid URLs, disallowed domains
  • Detect near-duplicate content using SimHash to avoid re-indexing
  • Store raw HTML in hot cache and cold permanent storage
  • Respect robots.txt crawl rules and enforce per-domain rate limits
  • Re-crawl high-value pages on a scheduled cadence

Non-functional requirements

  • Crawl 1 billion pages per month (~400 pages/sec sustained)
  • URL deduplication check < 1ms — bloom filter must fit in memory
  • No more than 1 request per second per domain (politeness constraint)
  • Raw HTML stored within 5s of fetch completion
  • 99.9% uptime — lost URL batches must be retryable from queue
  • Horizontal scaling: adding crawler workers should linearly increase throughput
PRISM
System Design Interview
Round 1 of 4 · Architecture Design · 60:00 remaining
PRISM logo
AI Interview
Interview Prep
Interview Challenges
Design Your Own System NEW
System Architectures
Design & Fix NEW
Interactive Roadmap System Design Guides
Notifications
  • No new notifications
Feedback
Signed in
Phase
Design Your Own System
Phase 01: Thinking in Systems
Upcoming

Components

User
CDN
Load Balancer
Server
Cache
Database
Blob Storage
Search Index
Queue
Worker
Rate Limiter
Service
API Gateway
Reverse Proxy
WebSocket Server
Third-party API

Inspector

Notes
Use clear names so your design intent is easy to understand.
Good
Name by business meaning
"Order API", "Restaurant Service"
Avoid
Generic names = zero signal
"Server 1", "API", "Queue"
A short description for each component makes feedback much better.
100%
Start by identifying:
  • Users & entry points
  • APIs & services
  • Databases & storage
  • Traffic flow & scale

Round 1 of 4 Architecture Design

Run a simulation to see results.

Time Remaining
60:00
System Design Interview

What are the core functional requirements?
What are the key non-functional constraints?

Questions

Start Evaluation to unlock questions.

Components Added
No components listed
Click "+ Add" to document components introduced in this stage.
Design Decisions

Click Simulate to run your design and see results here.

Internal notes — not shown to learners.

EVALUATE MODE

Test yourself like it's the real thing.

A structured 4-module evaluation that mirrors how top companies assess system design candidates.

Architecture Design
Draw your system on the canvas. Define components, connections, and data flow.
FR & NFR Requirements
Answer functional and non-functional requirement questions about your design.
MCQ Round
Multiple choice questions testing your depth on the chosen system.
Tradeoff Analysis
Justify your design decisions and defend your architectural tradeoffs.
AI Report Generated
A R S
Used by engineers preparing for FAANG & top-tier companies
Choose a Problem
No problem selected
  • 30 min
  • 45 min
  • 60 min
Round 2 of 4
MCQ Round
Answer multiple-choice questions based on your design.

Exit Interview?

You're in the middle of an interview session. Leaving now will end your current attempt.

Your progress will be saved.

Open a saved design

Select a design to load into the canvas.

My Evaluations

Your past evaluation sessions

Here’s a simple request flow that follows the expected layer order.

External User
→
Edge CDN → API Gateway → Load Balancer
→
Compute App Servers / Services
→
DataAccess Cache
→
Storage Database / Search Index
→
Async Queue → Worker

Tip: keep arrows moving forward through layers (Edge → Compute → Storage). Avoid sending storage back to compute.

Evaluation Instructions

Read the rules carefully before starting. The test auto-submits on refresh.

Before you start

  • Build your architecture on the canvas. The timer starts when you click Start Evaluation.
  • Don't forget to answer Functional Requirement and Non Functional Requirements.
  • When satisfied with your design, click Next to lock it and view the questions.
  • Please answer final step questions to complete the evaluation.

Dos

  • Do read each question carefully before answering.
  • Do include required components to maximize component coverage.
  • Do save a copy of your design if you want to keep it before submission.

Don'ts

  • Don't refresh or close the tab during an active evaluation — this will auto-submit your answers.
  • Don't switch app modes or open another tab while the evaluation is running.
  • Don't attempt to edit the design after clicking Next; the workspace will be locked.

All the best!!

Confirm

Input

Notice

Evaluation Report

Evaluation Complete

Generating Your Report

Hang tight — our AI is evaluating your design…

Did you know?

Loading…

Share feedback

Tell us what worked well and what we can improve.

Let's personalize this

Answer a couple of quick questions so we can tailor your journey and missions.

You can change this anytime from your Profile.

Your personalized missions are ready

We tailored these first steps based on your answers.

    PRISM Welcome Gift

    This is a personal welcome gift from PRISM.

    Congratulations.

    You explored PRISM.

    You earned Apprentice.

    As a welcome gift, unlock Full PRISM Access for the configured trial duration.

    This starts Trial. Trial timer begins only after you activate this gift.

    Welcome to PRISM

    We've prepared a personalized Apprentice Journey based on your goals and experience.

    This journey introduces you to the capabilities of PRISM that are most relevant to you.

    Complete all 8 missions to earn your Apprentice title. 8 MISSIONS

    PRISM Surprise Offer

    Complete your Apprentice Journey to unlock a special gift from PRISM.

    • No payment required
    • No credit card required
    • Just complete the journey
    MISSION CONTROL
    0 / 8 missions complete
    NEXT UP Continue your missions
    View full roadmap →
    Mission Complete 0 / 8 Completed Next: Keep going
    SYSTEM BRIEF

    ⬤ System Constraints

    What the system must do — every item is a user-facing behaviour your architecture must support.

      ↑ Engineering Constraints

      These are the failure modes you must design against — latency SLAs, durability targets, traffic ceilings.

        ⇆ Architecture Constraints

        ◈ Core Concepts to Master

        Your Journey
        PHASE – –
        0 / 0 0%
        0
        Mock Interview Checklist

        Pick a topic to start

        Explore concept overviews, real-system examples, key tradeoffs, and interview talking points for each roadmap section.

        Phase Mastered

        You've conquered this phase. These are the skills you now own:

          +500 XP

          Company Interview Paths

          Progress Summary