Skip to content

Commit 0bc54b0

Browse files
pon00050claude
andcommitted
docs: rewrite README to reflect sessions 43–80 (web API, MCP server, current data state)
- Add Web API + MCP Server section with 10 tool table and connection instructions - Expand CLI commands into full table (15 commands) - Update folder structure: app.py, .mcp.json, full src/ (17 files), tests/ additions - Update data table: bond_isin_map.parquet, cb_bw_events 11 cols, disclosures 271,504 rows - Update test count 168 → 301; update --stage values; update data coverage 2017–2023 - Remove stale "run extract_disclosures.py first" note (now wired into pipeline) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent c961b8a commit 0bc54b0

1 file changed

Lines changed: 115 additions & 36 deletions

File tree

README.md

Lines changed: 115 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -22,54 +22,75 @@ This project builds that infrastructure layer — so that researchers, journalis
2222

2323
## Current State / 현재 상태
2424

25-
**Milestone 1 complete. Milestones 2–4 data extraction complete — all four milestones runnable.**
25+
**Milestones 1–4 complete. All four analysis milestones runnable. FastAPI web/API layer and MCP server added.**
2626

27-
**마일스톤 1 완료. 마일스톤 2–4 데이터 수집 완료 — 4개 마일스톤 모두 실행 가능.**
27+
**마일스톤 1–4 완료. 4개 마일스톤 모두 실행 가능. FastAPI 웹/API 레이어 및 MCP 서버 추가.**
2828

2929
| Output | Location | EN | 한국어 |
3030
|---|---|---|---|
3131
| `beneish_scores.csv` | `03_Analysis/` | Ranked anomaly table with DART links — main deliverable | DART 링크 포함 이상 징후 순위표 — 주요 산출물 |
32-
| `beneish_scores.parquet` | `01_Data/processed/` | All 8 M-Score components, sector percentiles, CFS/OFS provenance | M-Score 8개 구성 요소, 섹터 백분위, CFS/OFS 출처 |
33-
| `company_financials.parquet` | `01_Data/processed/` | 5-year financials, all KOSDAQ companies | 5개년 재무제표, 코스닥 전 상장사 |
34-
| `cb_bw_events.parquet` | `01_Data/processed/` | CB/BW issuance events from DART DS005 | CB/BW 발행 이벤트, DART DS005 |
32+
| `beneish_scores.parquet` | `01_Data/processed/` | All 8 M-Score components, 2018–2023, sector percentiles, CFS/OFS provenance | M-Score 8개 구성 요소, 2018–2023, 섹터 백분위, CFS/OFS 출처 |
33+
| `company_financials.parquet` | `01_Data/processed/` | 2017–2023 financials, all KOSDAQ companies | 2017–2023 재무제표, 코스닥 전 상장사 |
34+
| `cb_bw_events.parquet` | `01_Data/processed/` | CB/BW issuance events — 11 cols including issue_amount, refixing_floor, maturity_date, board_date, warrant_separable | CB/BW 발행 이벤트 — 발행금액·리픽싱하한·만기일·이사회일·분리형 여부 포함 11개 컬럼 |
3535
| `price_volume.parquet` | `01_Data/processed/` | OHLCV price/volume windows around CB/BW events | CB/BW 이벤트 전후 OHLCV 주가/거래량 |
3636
| `corp_ticker_map.parquet` | `01_Data/processed/` | corp_code ↔ ticker mapping | corp_code ↔ 종목코드 매핑 |
3737
| `officer_holdings.parquet` | `01_Data/processed/` | Officer holding changes | 임원 보유 주식 변동 |
38-
| `disclosures.parquet` | `01_Data/processed/` | DART filing listings for timing analysis | DART 공시 목록 — 공시 시점 분석용 |
38+
| `disclosures.parquet` | `01_Data/processed/` | 271,504 DART filings across 921 corps — wired into pipeline automatically | DART 공시 목록 271,504건, 921개사 — 파이프라인에 자동 연결 |
3939
| `major_holders.parquet` | `01_Data/processed/` | 5%+ ownership threshold filings from DART majorstock.json | 대량보유상황보고서 — 5% 이상 지분 신고 이력 |
4040
| `bondholder_register.parquet` | `01_Data/processed/` | CB bondholder names and face values from 사채권자명부 sub-documents | CB 사채권자명부 — 권리자명·채권금액 |
4141
| `revenue_schedule.parquet` | `01_Data/processed/` | Revenue by customer/segment from 매출명세서 in 사업보고서 | 매출명세서 — 고객·품목별 매출 |
42+
| `bond_isin_map.parquet` | `01_Data/processed/` | 1,859 CB/BW ISINs mapped to 656 corp_codes via FSC API | FSC API로 수집한 CB/BW ISIN 1,859건 — 656개사 연결 |
4243
| `dart_xbrl_crosswalk.csv` | `tests/fixtures/` | XBRL element → variable mapping; audit trail | XBRL 요소 → 재무 변수 매핑; 감사 추적 |
4344
| [`beneish_viz.html`](https://raw.githack.com/pon00050/kr-forensic-finance/master/03_Analysis/beneish_viz.html) | `03_Analysis/` | Self-contained visual summary of Phase 1 results (5 Plotly charts) | Phase 1 결과 시각적 요약 — 5개 Plotly 차트, 단독 실행 가능 HTML |
4445
| `<corp_code>_report.html` | `03_Analysis/reports/` | Per-company forensic HTML report (all 4 milestones + AI synthesis) | 기업별 포렌식 HTML 보고서 |
4546

4647
**Visual summary (no Python required):** [beneish_viz.html — Phase 1 결과 보기](https://raw.githack.com/pon00050/kr-forensic-finance/master/03_Analysis/beneish_viz.html) — interactive Plotly charts, no Python required. / Python 없이 바로 보기.
4748

48-
**All four milestones are runnable now.** Run `extract_disclosures.py` first to populate `disclosures.parquet` for Milestone 3.
49-
50-
**4개 마일스톤 모두 실행 가능.** 마일스톤 3용 `disclosures.parquet``extract_disclosures.py`로 추출.
51-
5249
## Quickstart / 빠르게 시작하기
5350

5451
```bash
5552
git clone https://github.com/pon00050/kr-forensic-finance
5653
cd kr-forensic-finance
57-
uv sync
54+
uv sync # production dependencies
55+
uv sync --extra dev # + dev/test dependencies (needed to run tests)
5856
cp .env.example .env # add DART API key / DART API 키 입력 (free / 무료: opendart.fss.or.kr)
5957
```
6058

61-
**Option A — `krff` CLI (v1.5.0+):**
59+
**Option A — `krff` CLI:**
60+
61+
| Command | Description |
62+
|---|---|
63+
| `krff run` | ETL pipeline — DART extraction + transform |
64+
| `krff refresh` | Full pipeline + all 4 analysis milestones in one command |
65+
| `krff analyze` | Print beneish_scores.parquet summary |
66+
| `krff charts` | Write 03_Analysis/beneish_viz.html |
67+
| `krff status` | Artifact inventory (rows, sizes, dates) |
68+
| `krff status -v` | + DART run summary |
69+
| `krff quality` | Data quality metrics (null rates, coverage) |
70+
| `krff audit` | Pipeline freshness check — exits 1 if stale |
71+
| `krff stats` | Run stale statistical tests (14-node DAG) |
72+
| `krff report <corp_code>` | Generate per-company HTML report |
73+
| `krff batch_report` | Generate reports for all flagged companies |
74+
| `krff serve` | Start FastAPI + MCP server on :8000 |
75+
| `krff queue` / `krff surface` / `krff hide` | Review queue management |
76+
| `krff version` | Print version |
77+
| `krff --help` | List all commands |
78+
6279
```bash
80+
# Typical first run
81+
krff audit # check pipeline freshness before running
6382
krff run --market KOSDAQ --start 2019 --end 2023
6483
python 03_Analysis/beneish_screen.py # compute M-scores → beneish_scores.parquet
65-
krff analyze # print score summary
66-
krff charts # write 03_Analysis/beneish_viz.html
67-
krff status # show artifact inventory (rows, sizes, dates)
68-
krff run --stage cb_bw --backend fdr # use FinanceDataReader backend for OHLCV
69-
krff serve # start HTTP API on http://127.0.0.1:8000 (Swagger UI at /docs)
70-
krff monitor --once # poll data sources once (Phase 3 stub)
71-
krff alerts # show recent alerts (Phase 3 stub)
72-
krff --help # list all commands
84+
krff analyze # print score summary
85+
krff charts # write 03_Analysis/beneish_viz.html
86+
```
87+
88+
**MCP server (for Claude Code / Claude Desktop):**
89+
```bash
90+
krff serve # starts REST API + MCP at http://127.0.0.1:8000
91+
# Claude Code auto-connects via .mcp.json (included in repo)
92+
# Or add manually:
93+
claude mcp add --transport http kr-financial-statements http://localhost:8000/mcp/
7394
```
7495

7596
**Option B — direct scripts (unchanged):**
@@ -87,8 +108,7 @@ python 03_Analysis/beneish_screen.py
87108

88109
**Smoke test (5 companies, ~3 min):**
89110
```bash
90-
krff run --market KOSDAQ --start 2021 --end 2023 --sample 5 --sleep 0.1 --max-minutes 3
91-
python 03_Analysis/beneish_screen.py
111+
krff refresh --sample 5 --sleep 0.1 --max-minutes 3
92112
```
93113

94114
## Limitations and Disclaimer / 한계 및 면책 고지
@@ -122,6 +142,47 @@ All data is publicly available and free. 사용된 데이터는 모두 무료로
122142

123143
See also: [CONTRIBUTING.md](CONTRIBUTING.md) · [ROADMAP.md](ROADMAP.md)
124144

145+
### Web API + MCP Server
146+
147+
`krff serve` starts a FastAPI application at `http://localhost:8000` that exposes both a REST API and an MCP (Model Context Protocol) server — a standard interface that lets Claude and other AI clients call tools directly against live pipeline data.
148+
149+
**REST API (`http://localhost:8000`):**
150+
151+
| Endpoint | Description |
152+
|---|---|
153+
| `GET /api/status` | Artifact inventory (same as `krff status`) |
154+
| `GET /api/quality` | Data quality metrics |
155+
| `GET /api/companies/{corp_code}/summary` | All signals for one company |
156+
| `GET /api/companies/{corp_code}/report` | Trigger per-company HTML report generation |
157+
| `GET /api/alerts` | Recent anomaly alerts |
158+
| `GET /api/monitor/status` | Monitor pipeline health |
159+
160+
Swagger UI at `/docs`. Web routes at `/`, `/demo`, `/about`, `/datasets`, `/contact`, `/privacy`, `/terms`.
161+
162+
**MCP Server (`http://localhost:8000/mcp/`):**
163+
164+
10 tools are available to any MCP-compatible client (Claude Code, Claude Desktop, etc.):
165+
166+
| Tool | Description |
167+
|---|---|
168+
| `lookup_corp_code` | Name or ticker → corp_code. **Call this first** — all other tools require corp_code. |
169+
| `get_company_summary` | All signals for one company in one call |
170+
| `get_beneish_scores` | M-Score history with all 8 components (2018–2023) |
171+
| `get_cb_bw_events` | CB/BW events with manipulation flag counts |
172+
| `get_price_volume` | OHLCV for a date range (paginated) |
173+
| `get_officer_holdings` | Officer holding changes from DART |
174+
| `get_timing_anomalies` | Disclosure timing vs. price/volume anomalies |
175+
| `get_major_holders` | 5%+ block-holding filings (대량보유) |
176+
| `get_officer_network` | Cross-company officer centrality |
177+
| `search_flagged_companies` | Ranked anomaly screen by M-Score (paginated) |
178+
179+
Connect from Claude Code:
180+
```bash
181+
krff serve # must be running first
182+
# Auto-connects via .mcp.json (included in repo)
183+
# Or manually: claude mcp add --transport http kr-financial-statements http://localhost:8000/mcp/
184+
```
185+
125186
### Folder Structure
126187

127188
```
@@ -132,6 +193,8 @@ kr-forensic-finance/
132193
├── LICENSE
133194
├── pyproject.toml
134195
├── cli.py krff CLI entry point
196+
├── app.py FastAPI app — REST API + MCP server (krff serve)
197+
├── .mcp.json MCP client config (Claude Code / Claude Desktop)
135198
├── .env.example API keys + optional cloud storage template
136199
├── 00_Reference/ Local reference docs (not committed)
137200
├── 01_Data/
@@ -158,24 +221,42 @@ kr-forensic-finance/
158221
│ ├── beneish_viz.py Visual summary → beneish_viz.html
159222
│ ├── beneish_viz.html Generated output — open in any browser
160223
│ ├── phase1_research_questions.py Open analytical threads from Phase 1
161-
│ ├── cb_bw_timelines.py Milestone 2 — CB/BW event chains (Marimo app; run via run_cb_bw_timelines.py — Marimo UI optional)
162-
│ ├── timing_anomalies.py Milestone 3 — Disclosure timing (Marimo app; run via run_timing_anomalies.py — Marimo UI optional)
163-
│ ├── officer_network.py Milestone 4 — Officer graph (Marimo app; run via run_officer_network.py — Marimo UI optional)
224+
│ ├── cb_bw_timelines.py Milestone 2 — CB/BW event chains (Marimo app; run via run_cb_bw_timelines.py)
225+
│ ├── timing_anomalies.py Milestone 3 — Disclosure timing (Marimo app; run via run_timing_anomalies.py)
226+
│ ├── officer_network.py Milestone 4 — Officer graph (Marimo app; run via run_officer_network.py)
164227
│ ├── run_cb_bw_timelines.py Standalone runner → cb_bw_summary.csv
165228
│ ├── run_timing_anomalies.py Standalone runner → timing_anomalies.csv
166229
│ ├── run_officer_network.py Standalone runner → centrality_report.csv
230+
│ ├── _scoring.py Shared CB/BW scoring logic
231+
│ ├── statistical_tests/ 10 statistical validation scripts (local only)
232+
│ ├── officer_network/ Output: centrality_report.csv
233+
│ ├── reports/ Generated per-company HTML reports (gitignored)
167234
│ └── company_dives/ Per-company forensic scripts (local only, not committed)
168235
├── src/
169-
│ ├── __init__.py Package init
170-
│ ├── pipeline.py Pipeline wrapper for CLI/API callers
236+
│ ├── __init__.py
237+
│ ├── _paths.py Centralized path constants
171238
│ ├── analysis.py Beneish screen wrapper
239+
│ ├── audit.py Pipeline freshness DAG (krff audit)
172240
│ ├── charts.py Plotly chart generation
173-
│ └── report.py Per-company HTML report generator (krff report)
241+
│ ├── constants.py Threshold + flag literals
242+
│ ├── data_access.py Parquet/CSV loader layer
243+
│ ├── db.py DuckDB connection factory
244+
│ ├── mcp_server.py FastMCP server — 10 tools
245+
│ ├── mcp_utils.py JSON serialization helpers
246+
│ ├── models.py Pydantic response models
247+
│ ├── pipeline.py Pipeline wrapper for CLI/API
248+
│ ├── quality.py Data quality metrics (krff quality)
249+
│ ├── report.py Per-company HTML report generator
250+
│ ├── review.py Visibility queue (public/paid tiers)
251+
│ ├── stats_runner.py 14-node STATS_DAG orchestrator
252+
│ └── status.py Artifact inventory (krff status)
174253
└── tests/
175254
├── conftest.py Shared fixtures (sys.path setup)
176255
├── test_pipeline_invariants.py Schema/logic tests (run any time)
177256
├── test_acceptance_criteria.py End-to-end checks (after pipeline)
178257
├── test_cli.py CLI smoke tests
258+
├── test_e2e_synthetic.py End-to-end tests with synthetic data
259+
├── test_mcp_server.py MCP tool tests (skip when parquets absent)
179260
└── top50_spot_check.csv Spot-check reference data
180261
```
181262

@@ -214,16 +295,16 @@ All three support `--force`, `--sample N`, `--sleep S`, `--max-minutes M`. HTML
214295
| `--max-minutes N` | Hard deadline guard; exits cleanly after N minutes |
215296
| `--sleep S` | Inter-request sleep in seconds (default 0.5; use 0.1 for smoke tests) |
216297
| `--force` | Extract stage: re-fetch raw files (company_list.parquet, wics.parquet, etc.). Transform stage: delete and rebuild `company_financials.parquet`. |
217-
| `--stage dart\|transform\|cb_bw` | Run a single stage only (default: dart + transform) |
298+
| `--stage dart\|transform\|cb_bw\|beneish\|analysis` | Run a single stage only (default: dart + transform) |
218299

219300
### Testing
220301

221302
```bash
222-
pytest tests/ -v # Full suite (168 tests)
223-
pytest tests/test_pipeline_invariants.py tests/test_e2e_synthetic.py -v # No pipeline data needed
303+
uv run python -m pytest tests/ -v # 301 tests
304+
uv run python -m pytest tests/test_pipeline_invariants.py tests/test_e2e_synthetic.py -v # no pipeline data needed
224305
```
225306

226-
Test documentation is maintained locally in `00_Reference/`.
307+
Data-dependent tests auto-skip on CI (no parquets present); all 301 tests run locally after a full pipeline run.
227308

228309
### Output Schemas
229310

@@ -244,15 +325,14 @@ Test documentation is maintained locally in `00_Reference/`.
244325
| `dart_link` | Direct URL to company's annual report on DART |
245326
| `extraction_date` | Date this row's data was extracted from DART |
246327

247-
Full schema spec is maintained locally in `00_Reference/7_Projects/17_MVP_Requirements.md`.
248-
249328
**`top50_spot_check.csv`** (`tests/`) — Top 50 companies by M-Score with `corp_code`, `ticker`, `company_name`, `year`, `m_score`, `flag`.
250329

251330
### Generating Per-Company Reports
252331

253332
```bash
254333
krff report 01051092 # → 03_Analysis/reports/01051092_report.html
255334
krff report 01051092 --skip-claude # skip AI synthesis (no API key needed)
335+
krff batch_report # generate reports for all flagged companies
256336
```
257337

258338
Reports are self-contained HTML files. Run pipeline and analysis scripts first, then generate:
@@ -268,11 +348,10 @@ krff report <corp_code>
268348

269349
Set `ANTHROPIC_API_KEY` in `.env` to enable the AI synthesis section (claude-sonnet-4-6).
270350

271-
**Data coverage:** Reports reflect 2019–2023 data. Re-run the pipeline to update.
351+
**Data coverage:** Reports reflect 2017–2023 data. Re-run the pipeline to update.
272352

273353
### Further Reading
274354

275355
Architecture notes, API research findings, and methodology documentation are maintained locally in `00_Reference/` (not committed to the public repository). Clone the repo and run the pipeline — the code is self-documenting.
276356

277357
S3-compatible cloud storage is optional — all scripts fall back to local files.
278-

0 commit comments

Comments
 (0)