RS-Paper-Hub

A curated collection of Remote Sensing & Earth Observation papers from arXiv, with automated scraping, cleaning, task tagging, topic filtering (VLM, Agent, UAV, SAR), and trend statistics.

English | 中文 | Live Demo

Overview

RS-Paper-Hub automatically scrapes remote sensing and earth observation papers from arXiv, extracts structured metadata, and provides a one-click pipeline for data cleaning, task tagging, topic filtering (VLM, Agent, UAV, SAR), classification, and trend statistics. Updated daily via GitHub Actions (Mon–Fri, synced with arXiv announcement schedule).

Key Features

Expanded Search Scope — Covers "remote sensing", "earth observation", and "geospatial" papers from arXiv (capturing GeoAI papers like GeoX, GeoCLIP that use geospatial terminology without explicitly saying "remote sensing")
Daily Automated Updates — GitHub Actions fetches the latest papers (last 7 days) Mon–Fri at 00:30 UTC (08:30 Beijing Time), aligned with arXiv's announcement schedule
Incremental Pipeline — Only new papers go through cleaning, classification, and tagging; existing data is preserved
Task Tagging — Auto-tags papers with 11 task types: Classification, Object Detection, Change Detection, Segmentation, VQA, Image Captioning, Visual Grounding, Image-Text Retrieval, Geolocation, Super-Resolution, 3D Reconstruction
Paper Classification — Labels papers as Method, Dataset, or Survey based on title keywords
VLM Filtering — Keyword-based filtering for Vision-Language Model related papers with context-aware rules (avoids false positives from non-VLM cross-modal or retrieval terms)
Agent Filtering — Keyword-based filtering for Agent / Autonomous Decision-Making related papers (multi-agent systems, RL-based agents, LLM agents, agentic workflows, etc.)
UAV Filtering — Keyword-based filtering for UAV / drone related papers (unmanned aerial vehicles, drone-based remote sensing, aerial photogrammetry, etc.)
SAR Filtering — Keyword-based filtering for SAR (Synthetic Aperture Radar) related papers (InSAR, PolSAR, Sentinel-1, despeckling, deformation monitoring, etc.)
Trends & Statistics — Interactive trend dashboards showing yearly/monthly paper distributions and top author rankings across all data sources, with incremental updates and click-to-drill-down author details
Five-Tab Web Viewer — Browse All Papers, VLM subset, UAV subset, Agent subset, or SAR subset; with search, multi-dimensional chart filtering, task/category/year filters, Google Scholar links, and mobile-friendly UI
Clickable Tag Filtering — Click any tag (date, type, category, task, VLM, etc.) on a paper card to filter by that value; stack multiple tags for progressive filtering, click again to remove
Paper Collection — Collect papers across multiple searches into a personal collection, then view or export them together
BibTeX Batch Export — Export filtered results, current page, custom range, or collection as timestamped .bib file with optional abstracts
Group Export — Export selected papers as a Group JSON file for sharing curated reading lists; submit via PR to make it available to all users
Code Discovery — Automatically extracts code repository URLs from abstracts
RSS/Atom Feeds — Auto-generated Atom feeds (All / VLM / Agent / UAV / SAR) for Zotero subscription, updated daily with the last 7 days of papers
PDF Download — Batch download with deduplication, organized by year

Quick Start

pip install -r requirements.txt

# Scrape all papers
python main.py

# One-click: clean + classify + tag tasks + filter VLM/Agent/UAV/SAR + trends
python pipeline.py

Daily Workflow

# 1. Grab latest papers (last 7 days, incremental by default)
python main.py --update

# 2. Run full pipeline (deduplicate → clean → classify → tag → filter VLM/Agent/UAV/SAR → trends → export)
python pipeline.py

Or use the one-liner:

bash run_daily.sh

That's it. All output files (papers.csv/json, papers_vlm.csv/json, papers_agent.csv/json, papers_uav.csv/json, papers_sar.csv/json) are updated in place.

Note: --incremental is enabled by default — existing papers are always skipped. Use --no-incremental to force a full re-fetch.

Web Scraping Fallback (when arXiv API is down)

The arXiv API can occasionally be unreachable. A web scraping fallback is provided that produces the same output format by scraping arXiv search pages directly:

# Use web scraper instead of API
bash run_daily.sh --web

# Or run directly
python main_web.py --update

The fallback uses main_web.py + web_scraper.py to scrape arxiv.org/search/ for "remote sensing" and "earth observation" papers. It is purely incremental — existing papers are never modified, only new papers are appended. The downstream pipeline (pipeline.py) works identically regardless of which fetch method is used.

Flag	Description	Default
`--update`	Quick update (latest 7 days)	off
`--days N`	Look back N days in update mode	7
`--max-results N`	Max papers to fetch	unlimited
`--with-code`	Query Papers With Code for repos	off
`-v, --verbose`	Verbose logging	off

Requires: beautifulsoup4 (included in requirements.txt)

Usage

Scraping

# Full scrape
python main.py

# Custom year range
python main.py --start-year 2023 --end-year 2025

# Limit results (for testing)
python main.py --max-results 100

# Quick update (latest 7 days)
python main.py --update

# Check progress
python main.py --status

Pipeline (Recommended)

# One-click: clean + classify + tag tasks + filter VLM/Agent/UAV/SAR + trends
python pipeline.py

# Custom input
python pipeline.py --input output/papers.json

pipeline.py runs the following 15 steps (incrementally — each step skips already-processed papers):

Load & Deduplicate — Remove duplicate papers by Paper_link
Clean — Extract code URLs from abstracts, fill code field
Classify — Label every paper as Method / Dataset / Survey (title-based)
Tag Tasks — Assign task labels (CLS, OD, CD, SEG, VQA, IC, VG, ITR, GeoLoc, SR, 3D)
Save — Write cleaned papers.csv + papers.json
Filter VLM — Select Vision-Language Model related papers, export papers_vlm.csv/json
Filter Agent — Select Agent-related papers, export papers_agent.csv/json
Filter UAV — Select UAV-related papers, export papers_uav.csv/json
Filter SAR — Select SAR-related papers, export papers_sar.csv/json
Update Groups — Auto-update author-based paper groups
Generate Atom Feeds — Produce feed.xml, feed_vlm.xml, feed_agent.xml, feed_uav.xml with the last 7 days of papers
Generate Trends — Compute/update trend statistics (incremental) to trends/trends.json

Standalone Filter Scripts

# VLM filter only
python filter_vlm.py --input output/papers.json

# Agent filter only
python filter_agent.py --input output/papers.json

# Preview without saving (dry-run)
python filter_vlm.py --dry-run
python filter_agent.py --dry-run

PDF Download

# Scrape + download
python main.py --download

# Download only (from existing data)
python main.py --download-only

CLI Reference

`main.py`

Flag	Description	Default
`--start-year`	Start year	2020
`--end-year`	End year	2026
`--max-results`	Max papers to fetch	unlimited
`--output-dir`	Output directory	`output`
`--update`	Quick update (latest 7 days)	off
`--no-incremental`	Disable incremental, re-fetch all	off
`--incremental`	Skip existing papers	on
`--download`	Download PDFs	off
`--download-only`	Download PDFs only (skip scraping)	off
`--with-code`	Query Papers With Code for repos	off
`--status`	Show progress and exit	—
`-v, --verbose`	Verbose logging	off

`pipeline.py`

Flag	Description	Default
`--input`	Input JSON file	`output/papers.json`
`--output-dir`	Output directory	`output`

`filter_vlm.py` / `filter_agent.py`

Flag	Description	Default
`--input`	Input JSON file	`output/papers.json`
`--output-dir`	Output directory	`output`
`--dry-run`	Preview matches without saving	off

Output Schema

All outputs are available in both CSV and JSON format.

Field	Description	Example
`Category`	Paper category	Method, Dataset, Survey
`Type`	arXiv primary category	Computer Vision
`Subtype`	Secondary categories	Image and Video Processing
`Date`	Exact publication date	2024-03-15
`Month`	Publication month	3
`Year`	Publication year	2024
`Title`	Paper title	Hybrid Attention Network for...
`abbr.`	Abbreviation from title	HMANet
`Paper_link`	arXiv URL	http://arxiv.org/abs/2301.12345
`Abstract`	Full abstract	...
`code`	Code repository URL	https://github.com/...
`Publication`	Venue (journal/conference)	CVPR 2024
`BibTex`	BibTeX citation	@article{...}
`Authors`	Author list	Alice, Bob, Charlie
`_tasks`	Task tags (semicolon-separated)	CLS;OD;SEG

Task Tags

Papers are automatically tagged with task types based on title and abstract keyword matching:

Tag	Task	Examples
CLS	Classification	scene classification, land use/cover classification
OD	Object Detection	object/vehicle/ship/building detection
CD	Change Detection	change detection, bi-temporal analysis
SEG	Segmentation	semantic/instance/panoptic/referring segmentation
VQA	Visual Question Answering	VQA, RSVQA
IC	Image Captioning	image captioning, caption generation
VG	Visual Grounding	visual grounding, phrase grounding
ITR	Image-Text Retrieval	cross-modal retrieval
GeoLoc	Geolocation	geolocation, place recognition
SR	Super-Resolution	super-resolution, image enhancement
3D	3D Reconstruction	3D reconstruction, point cloud, depth estimation

Project Structure

rs-paper-hub/
├── main.py              # Scraper CLI entry point
├── main_web.py          # Web scraping fallback (when arXiv API is down)
├── pipeline.py          # One-click: clean + classify + tag + filter VLM/Agent/UAV/SAR + trends
├── filter_vlm.py        # Standalone VLM filter script
├── filter_agent.py      # Standalone Agent filter script
├── filter_uav.py        # Standalone UAV filter script
├── rss_generator.py     # Atom feed generator for Zotero
├── config.py            # Search configuration
├── scraper.py           # arXiv API scraper
├── web_scraper.py       # arXiv web scraper (HTML fallback)
├── parser.py            # Metadata parser & BibTeX generation
├── downloader.py        # PDF downloader with resume support
├── progress.py          # Progress tracker
├── pwc_client.py        # Papers With Code client
├── cleaning/
│   ├── abstract_cleaner.py   # Abstract URL extraction
│   ├── classifier.py         # Paper classifier (Method/Dataset/Survey)
│   ├── task_tagger.py        # Task tagging (11 task types)
│   └── filter/
│       ├── vlm_filter.py     # VLM keyword rules
│       ├── agent_filter.py   # Agent keyword rules
│       ├── uav_filter.py     # UAV keyword rules
│       └── sar_filter.py     # SAR keyword rules
├── .github/workflows/
│   └── daily-update.yml      # Daily CI/CD pipeline (Mon-Fri, synced with arXiv)
├── index.html               # Interactive web viewer (5 tabs: All / VLM / UAV / Agent / SAR)
├── groups/                  # Paper groups for curated reading lists
│   ├── index.json           # Group registry (key, label, file; auto+authors for auto-update)
│   └── *.json               # Individual group files (arrays of arXiv URLs)
├── update_groups.py         # Auto-update groups by author matching
├── requirements.txt
└── output/
    ├── papers.csv/json              # All papers (cleaned + classified + tagged)
    ├── papers_vlm.csv/json          # VLM subset with categories
    ├── papers_vlm_annotated.json    # Full list with VLM flags
    ├── papers_agent.csv/json        # Agent subset with categories
    ├── papers_agent_annotated.json  # Full list with Agent flags
    ├── papers_uav.csv/json          # UAV subset with categories
    ├── papers_uav_annotated.json    # Full list with UAV flags
    ├── papers_sar.csv/json          # SAR subset with categories
    ├── papers_sar_annotated.json    # Full list with SAR flags
    ├── feed.xml                     # Atom feed — all papers (last 7 days)
    ├── feed_vlm.xml                 # Atom feed — VLM papers (last 7 days)
    ├── feed_agent.xml               # Atom feed — Agent papers (last 7 days)
    ├── feed_uav.xml                 # Atom feed — UAV papers (last 7 days)
    ├── feed_sar.xml                 # Atom feed — SAR papers (last 7 days)
    ├── progress.json                # Scraping progress
    └── trends/
        └── trends.json              # Trend statistics (yearly, monthly, top authors)

Search Scope

Papers are fetched from all arXiv categories where the title or abstract contains "remote sensing" or "earth observation". To customize, edit SEARCH_QUERY in config.py:

SEARCH_QUERY = 'ti:"remote sensing" OR abs:"remote sensing" OR ti:"earth observation" OR abs:"earth observation"'

Web Viewer

Visit rspaper.top or run locally:

python3 -m http.server 8080

Features include:

Five data tabs — Switch between All Papers, VLM subset, UAV subset, Agent subset, and SAR subset (with dropdown for overflow tabs)
Quick date filters — "Today" and "This Week" buttons with red badge counts
Relevance-ranked search — Title matches prioritized over abstract matches
Multi-dimensional chart filtering — Click year/type/category/task bars to filter, multi-select supported
Task distribution chart — Top 5 tasks shown with collapsible remaining tasks
Year range selection — Single year or custom range via dropdown
Paper classification — All papers labeled as Method, Dataset, or Survey
Paper collection — Collect papers across multiple searches, then view or export them together
BibTeX batch export — Export filtered results, current page, custom range, or collection as timestamped .bib file with optional abstracts
Group export & sharing — Export papers as a Group JSON file with custom name; groups appear in "All Groups" dropdown for quick filtering. Submit your group via PR to share with the community (learn more)
Auto-updated author groups — Define groups with "auto": true and "authors": [...] in groups/index.json; the pipeline automatically matches papers by author name and keeps group files up to date daily
New papers panel — Side panel showing today's and this week's papers
Google Scholar links — One-click search on Google Scholar for each paper
Bilingual UI — Switch between English and Chinese
Skills page — A curated collection of research skills (coding, writing, workflows and more) with community contributions welcome
Venues page — Quick-reference directory of key journals and conferences in remote sensing and related AI/CV fields, organized by category
Resources page — Community-driven collection of remote sensing datasets and tools, loaded from resources/ JSON files — contribute via PR (learn more)
Trends page — Interactive statistics dashboard with yearly/monthly paper distributions and top author rankings; click authors to see their individual publication timeline
Radar page — 3D globe word cloud of BERT-weighted keywords (drag to rotate, click to search) paired with weekly hot paper rankings in three scoring modes (Trend / Frequency / Comprehensive); switch between All Papers, VLM, UAV, Agent, and SAR sources
Mobile-friendly — Responsive layout with collapsible filters and wrapping navigation
LaTeX rendering — Math formulas rendered via KaTeX

Note: Category and task labels are rule-based and may contain inaccuracies. We are continuously improving them. This does not affect tracking the latest research trends.

RSS Feeds & Zotero Integration

The pipeline automatically generates Atom feeds containing the last 7 days of papers. These feeds are designed for Zotero but work with any feed reader.

Feed	URL	Content
All Papers	`https://rspaper.top/output/feed.xml`	All recent papers
VLM Papers	`https://rspaper.top/output/feed_vlm.xml`	VLM subset
Agent Papers	`https://rspaper.top/output/feed_agent.xml`	Agent subset
UAV Papers	`https://rspaper.top/output/feed_uav.xml`	UAV subset
SAR Papers	`https://rspaper.top/output/feed_sar.xml`	SAR subset

Subscribe in Zotero

Open Zotero → File → New Feed → From URL
Paste one of the feed URLs above
Zotero will automatically pull new papers with title, authors, abstract, arXiv link, and code URL

Feeds are updated daily alongside the main pipeline via GitHub Actions.

Rate Limits

Operation	Rate	Note
Metadata query	~3s / request	Returns up to 100 results per request
PDF download	~3s / file	Respect arXiv rate limits

Recommended workflow: scrape metadata first (python main.py), then download PDFs separately (python main.py --download-only).

Citation

If you find RS-Paper-Hub useful in your research or work, please consider citing this repository:

@software{rs_paper_hub,
  author       = {ML4Sustain},
  title        = {RS-Paper-Hub: A Curated Collection of Remote Sensing and Earth Observation Papers from arXiv},
  year         = {2026},
  url          = {https://rspaper.top},
  note         = {Automated scraping, cleaning, classification, task tagging, VLM filtering, and Agent filtering pipeline for remote sensing papers}
}

License

MIT

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Repository files navigation

RS-Paper-Hub

Overview

Key Features

Quick Start

Daily Workflow

Web Scraping Fallback (when arXiv API is down)

Usage

Scraping

Pipeline (Recommended)

Standalone Filter Scripts

PDF Download

CLI Reference

`main.py`

`pipeline.py`

`filter_vlm.py` / `filter_agent.py`

Output Schema

Task Tags

Project Structure

Search Scope

Web Viewer

RSS Feeds & Zotero Integration

Subscribe in Zotero

Rate Limits

Citation

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 327 Commits
.github/workflows		.github/workflows
asset		asset
cleaning		cleaning
docs		docs
groups		groups
output		output
resources		resources
trends		trends
venues		venues
wordcloud		wordcloud
.gitattributes		.gitattributes
.gitignore		.gitignore
CNAME		CNAME
README.md		README.md
README_zh.md		README_zh.md
backfill_dates.py		backfill_dates.py
backfill_geospatial.py		backfill_geospatial.py
clean.py		clean.py
config.py		config.py
downloader.py		downloader.py
filter_agent.py		filter_agent.py
filter_uav.py		filter_uav.py
filter_vlm.py		filter_vlm.py
generate_radar_weekly.py		generate_radar_weekly.py
html_inbox_import.py		html_inbox_import.py
index.html		index.html
main.py		main.py
main_web.py		main_web.py
parser.py		parser.py
pipeline.py		pipeline.py
progress.py		progress.py
pwc_client.py		pwc_client.py
requirements.txt		requirements.txt
rss_generator.py		rss_generator.py
run_daily.sh		run_daily.sh
scraper.py		scraper.py
update_groups.py		update_groups.py
web_scraper.py		web_scraper.py

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

RS-Paper-Hub

Overview

Key Features

Quick Start

Daily Workflow

Web Scraping Fallback (when arXiv API is down)

Usage

Scraping

Pipeline (Recommended)

Standalone Filter Scripts

PDF Download

CLI Reference

main.py

pipeline.py

filter_vlm.py / filter_agent.py

Output Schema

Task Tags

Project Structure

Search Scope

Web Viewer

RSS Feeds & Zotero Integration

Subscribe in Zotero

Rate Limits

Citation

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`main.py`

`pipeline.py`

`filter_vlm.py` / `filter_agent.py`