JOBKOREA Scraper Walkthrough

This project implements three techniques to scrape JOBKOREA, demonstrating the progression from manual scraping to AI-generated scraping code.

Prerequisites

Before starting, ensure you have:

Python 3.9+ installed
Node.js and npx (for Technique 2)
Bright Data account with API token (new users get free credits)
Basic familiarity with Python and JSON

Project Setup

Clone the repository and install dependencies:

python -m venv venv
source venv/bin/activate      # macOS / Linux
venv\Scripts\activate         # Windows

pip install -r requirements.txt

Required Python Libraries

requests - HTTP library for fetching web pages
BeautifulSoup4 - HTML/XML parsing library
Pydantic - Data validation using Python type annotations
mcp - Model Context Protocol Python SDK

1. Manual Scraper (Requests + BeautifulSoup)

File: manual_scraper.py

Attempts to scrape the site using standard HTTP requests with the requests library and BeautifulSoup for parsing.

Result: SUCCESS (via Advanced Parsing)

How It Works

JOBKOREA uses Next.js Client-Side Rendering (CSR), meaning job data is hidden in JSON scripts, not static HTML.
Problem: Standard BeautifulSoup fails to find list items in the rendered HTML.
Solution: The script uses Regex to extract job data directly from Next.js hydration scripts (self.__next_f.push).
Outcome: Successfully scrapes jobs without needing a full browser, though it's more fragile than the MCP approach.

Usage

python manual_scraper.py "https://www.jobkorea.co.kr/Search/?stext=python"

When to Use This Approach

Manual scraping is ideal for:

Learning how websites structure their data
Quick prototyping and experimentation
Sites with stable, predictable structures
Low-volume scraping needs

Limitations: Tightly coupled to JOBKOREA's current page structure and can break when the site changes.

2. Bright Data MCP Scraper

File: mcp_scraper.py

Uses the @brightdata/mcp server to bypass blocking and handle rendering automatically.

How It Works

Connects to the local MCP server via npx @brightdata/mcp
Calls scrape_as_markdown to fetch and render the page content
Saves raw markdown to scraped_data.md
Parses the markdown to extract structured data into jobs_mcp.json

Setup

Get your API token from Bright Data User Settings
Create a .env file in the project root:
```
BRIGHT_DATA_API_TOKEN=your_token_here
```

Usage

python3 mcp_scraper.py "https://www.jobkorea.co.kr/Search/?stext=python"

When to Use This Approach

MCP-based scraping is ideal for:

Production-ready, reliable scraping
Automated and scheduled jobs
Sites that frequently change their structure
Large-scale data collection
Integration into data pipelines

Benefits:

Handles rendering, networking, and anti-bot measures
Less sensitive to layout changes
Built on Bright Data's Web Unlocker technology
Easy to automate and integrate

3. AI-Generated Scraper (Bright Data Web Scraper IDE)

No local files needed - This technique uses Bright Data's cloud-based IDE with AI code generation.

Overview

Instead of writing scraping code from scratch, you use Bright Data's Web Scraper IDE to automatically generate production-ready JavaScript scraping code. The AI analyzes the target website and creates a fully functional scraper for you.

How It Works

Open the Web Scraper IDE in your Bright Data dashboard
Enter the target URL (JOBKOREA search page)
Click "Generate Code" - AI analyzes the site and creates the scraper
Review and customize the generated code if needed
Run the scraper directly from the IDE or via API
Download results in JSON, CSV, or other formats

Step-by-Step Guide

Step 1: Access the Web Scraper IDE

Log in to Bright Data Dashboard
Navigate to Data → My Scrapers from the left sidebar
Click New in the top-right corner
Select Develop your own web scraper

Step 2: Generate the Scraper Code

In the IDE, enter your target URL:

https://www.jobkorea.co.kr/Search/?stext=python

Click Generate Code
The AI will analyze the page structure and generate JavaScript code to:
- Navigate to the search results
- Extract job listings (title, company, location, date, URL)
- Handle pagination
- Structure the data

You'll receive an email notification when the code is ready

Step 3: Review and Customize

The generated code typically includes:

Interaction Code: JavaScript to navigate the site, click buttons, scroll, etc.
Parser Code: Logic to extract and structure the job data
Error Handling: Built-in retry logic and exception handling

You can customize:

Which fields to extract
How many pages to scrape
Filters and search parameters
Output format

Step 4: Test the Scraper

Click the Preview (play) button to test your scraper
Review the extracted data in the IDE
Make adjustments if needed
Save your scraper

Step 5: Run at Scale

You can now:

Option A: Run from the IDE

Click "Run" to execute the scraper
Monitor progress in real-time
Download results when complete

Option B: Use the API

import requests

response = requests.post(
    'https://api.brightdata.com/datasets/v3/trigger?dataset_id=YOUR_SCRAPER_ID',
    headers={
        'Authorization': 'Bearer YOUR_API_TOKEN',
        'Content-Type': 'application/json'
    },
    json=[{
        'url': 'https://www.jobkorea.co.kr/Search/?stext=python',
        'max_pages': 5
    }]
)

Option C: Schedule Regular Runs

Set up automated scraping schedules
Receive data via webhook, email, or cloud storage
Monitor via the dashboard

When to Use This Approach

AI-generated scraping is ideal for:

Fast setup - minutes instead of hours
Production-ready code without manual development
Maintaining scrapers - AI can regenerate when sites change
Non-developers who need reliable scrapers
Teams wanting to focus on data analysis, not scraping logic
Scaling to multiple websites quickly

Benefits

✅ AI-Powered Code Generation - Analyzes sites and writes scraping logic
✅ Pre-built Templates - Start from proven patterns for popular sites
✅ Fully Hosted - No infrastructure to manage
✅ Built-in Unblocking - Automatic CAPTCHA solving and proxy rotation
✅ Easy Maintenance - Regenerate code when sites change
✅ API Access - Integrate into your existing workflows
✅ Real-time Monitoring - Track scraper performance and costs

Key Features

JavaScript Environment: Full access to modern JS/Node.js
Browser Automation: Built-in headless browser capabilities
Utility Functions: Pre-built helpers for common scraping tasks
Debugging Tools: Console logs, network inspector, error tracking
Delivery Options: JSON, CSV, Amazon S3, webhooks, email
Scheduling: Cron-like scheduling for automated runs

Additional Resources

Web Scraper IDE Documentation - Complete guide
Develop a Scraper Guide - Step-by-step tutorial
Video Tutorials - Watch how to use the IDE
API Integration - Trigger scrapers programmatically

Comparing the Three Techniques

Technique	Setup Effort	Reliability	Automation	Code Required	Maintenance	Best Use Case
Manual Python Scraping	Low	Low-Medium	Limited	Yes (Python)	High	Learning, quick experiments
Bright Data MCP (Python)	Medium	High	High	Yes (Python)	Medium	Production scraping, scheduled jobs
AI-Generated Scraper (IDE)	Very Low	Very High	Very High	No (AI-generated)	Very Low	Fast setup, managed production scrapers

Project Structure

jobkorea_scraper/
│
├── manual_scraper.py        # Technique 1: Manual scraping
├── mcp_scraper.py           # Technique 2: MCP-based scraping
├── parsers/
│   └── jobkorea.py          # Shared parsing logic
├── schemas.py               # Pydantic models for validation
├── requirements.txt         # Python dependencies
├── .env                     # API token (create this)
├── jobs.json                # Output from manual scraper
├── jobs_mcp.json            # Output from MCP scraper
├── scraped_data.md          # Raw markdown from MCP
└── README.md                # This file

Note: Technique 3 doesn't require local files - it runs entirely in Bright Data's cloud IDE.

Output Files

After running the scrapers, you'll have:

jobs.json - Structured job data from manual scraper
jobs_mcp.json - Structured job data from MCP scraper
scraped_data.md - Raw markdown from MCP (for debugging)
debug.html - Raw HTML from manual scraper (for debugging)

From the Web Scraper IDE, you can download results in multiple formats:

JSON
CSV
NDJSON
Direct to Amazon S3, Google Cloud Storage, or Azure Blob Storage

Additional Resources

Bright Data Products

Web Scraper IDE - AI-assisted JavaScript scraper with code generation
Web Unlocker - Bypass CAPTCHAs and anti-bot systems
SERP API - Collect search engine results at scale
Proxy Networks - Residential, ISP, Mobile, and Datacenter proxies
Scraping Browser - Headless browser with unblocking

Documentation

Web Scraper IDE Overview - Complete IDE guide
MCP Server Documentation - Model Context Protocol
BeautifulSoup Integration - Using Bright Data with BeautifulSoup
Python SDK - Official Bright Data Python SDK
API Reference - Complete API documentation

Learning Resources

Bright Data Dashboard - Manage zones and monitor usage
Get API Token - User settings and API keys
Video Tutorials - Step-by-step guides
IDE Video Tutorials - Learn the Web Scraper IDE

Troubleshooting

Manual Scraper Issues

No jobs found: JOBKOREA structure may have changed. Check debug.html for the actual HTML structure.
Encoding errors: Ensure UTF-8 encoding is properly set for Korean text.
Connection errors: Add delays between requests or use Bright Data proxies.

MCP Scraper Issues

MCP server won't start: Ensure Node.js and npx are installed and updated.
API token errors: Verify your token in .env matches your Bright Data account.
Empty results: Check scraped_data.md to see what MCP returned.

Web Scraper IDE Issues

Code generation failed: Try a different URL or simplify the target page.
Generated code not working: Use the IDE debugger to inspect the page structure.
Can't find data: Customize the parser code to match the exact selectors.
Rate limiting: The IDE handles this automatically with built-in proxy rotation.

Wrapping Up

This project demonstrates three progressive approaches to scraping JOBKOREA:

Manual scraping helps you understand how the site works and teaches fundamental web scraping concepts
MCP-based scraping provides reliability and automation for production use with Python integration
AI-generated scraping offers the fastest path to production-ready, managed scrapers with zero code

Which Technique Should You Choose?

Choose Technique 1 if you're learning web scraping or need a quick prototype
Choose Technique 2 if you need Python integration, scheduled jobs, or custom data pipelines
Choose Technique 3 if you want the fastest setup, managed infrastructure, and automatic maintenance

For production applications scraping modern, client-rendered websites like JOBKOREA, Bright Data's Web Scraper IDE provides the most efficient path from idea to deployed scraper.

Next Steps

✅ Try all three techniques with different search queries
✅ Explore the Web Scraper IDE for AI-generated scrapers via AI Scraper Studio
✅ Build automated data pipelines using Bright Data's APIs
✅ Scale your scraping with Bright Data's Proxy Networks
✅ Learn more from video tutorials

Need help? Check out Bright Data Documentation or contact support.

Name		Name	Last commit message	Last commit date
Latest commit History 3 Commits
parsers		parsers
README.md		README.md
analyze_debug.py		analyze_debug.py
jobs_mcp.json		jobs_mcp.json
manual_scraper.py		manual_scraper.py
mcp_scraper.py		mcp_scraper.py
requirements.txt		requirements.txt
schemas.py		schemas.py

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

JOBKOREA Scraper Walkthrough

Prerequisites

Project Setup

Required Python Libraries

1. Manual Scraper (Requests + BeautifulSoup)

How It Works

Usage

When to Use This Approach

2. Bright Data MCP Scraper

How It Works

Setup

Usage

When to Use This Approach

3. AI-Generated Scraper (Bright Data Web Scraper IDE)

Overview

How It Works

Step-by-Step Guide

Step 1: Access the Web Scraper IDE

Step 2: Generate the Scraper Code

Step 3: Review and Customize

Step 4: Test the Scraper

Step 5: Run at Scale

When to Use This Approach

Benefits

Key Features

Additional Resources

Comparing the Three Techniques

Project Structure

Output Files

Additional Resources

Bright Data Products

Documentation

Learning Resources

Troubleshooting

Manual Scraper Issues

MCP Scraper Issues

Web Scraper IDE Issues

Wrapping Up

Which Technique Should You Choose?

Next Steps

About

Topics

Resources

Uh oh!

Stars

Watchers

Forks

Contributors

Uh oh!

Languages