HailBerry

中文说明点这里

English

Overview

HailBerry is an open-source toolkit that turns chemistry reaction images into structured JSON annotations, then helps review, validate, and package them.

The repository includes workflow code, prompts, validators, a local review interface, method notes, and selected input-output examples.

Workflow Diagram

Start Here

In three sentences:

• HailBerry turns chemistry reaction images into structured JSON drafts.
• You can open the review tool and validate included examples without any API key.
• You only need a model API key when you want HailBerry to generate new drafts from your own images.

If you want to...	Go here first	You will use
See the tool without setup	Beginner Quick Start, steps 1-2	Included demo files and the local review tool
Review your own JSON files	Review Tool Data Layout	packages/review-tool/data/base_submissions/
Generate a model draft from an image	Beginner Quick Start, steps 3-5	packages/pipeline/.env.local and final_annotate.py
Check whether JSON files are valid	Beginner Quick Start, step 1	packages/validators/hailberry_validate.py
Understand licenses and example data	License Boundary	MIT for project-authored code; CC-BY-NC for selected example data

Before You Start

HailBerry is a local toolkit, not a hosted service. The review tool and validators can run immediately with the included demo files. Model-assisted extraction for your own chemistry images requires your own vision-model API key.

API keys are read from:

packages/pipeline/.env.local

Create that file by copying packages/pipeline/.env.example, then fill only the provider you plan to use. .env.local is ignored by Git; do not commit it.

For a first full annotation run, choose one of the built-in model API providers below. Other model services require adding a small provider wrapper in code; they cannot be enabled by adding only an API key.

Provider	Fill in .env.local	Run with
OpenAI	OPENAI_API_KEY	--providers openai
Anthropic	ANTHROPIC_API_KEY	--providers anthropic
DashScope / Qwen-VL	DASHSCOPE_API_KEY	--providers dashscope
Zhipu GLM	ZHIPU_API_KEY	--providers zhipu

Fill only one provider to start. You can add more providers later if you want to compare model outputs.

What The Tool Does

HailBerry takes a chemistry reaction-scheme image as input and produces a JSON file shaped like this:

{
  "reactions": [
    [
      {"type": "reactants", "text": "CC(=O)O", "relations": []},
      {"type": "reactants", "text": "CO", "relations": []},
      {"type": "products", "text": "COC(C)=O", "relations": ["methyl acetate"]}
    ]
  ]
}

The allowed entity types are:

• reactants
• reagent
• products
• temperature
• time
• other conditions

reactants, reagent, and products are usually SMILES. Conditions can be plain text. relations stores yield, ee, dr, equivalents, method labels, R-group notes, selected-example notes, uncertainty notes, or other visible metadata.

Visual Input-Output Examples

The examples below use three CC-BY-NC-4.0 images with paired HailBerry JSON outputs. They are demonstration data, not standard answers or benchmarks.

CC-BY-NC-4.0 license text:

https://creativecommons.org/licenses/by-nc/4.0/

ID	Input image	Output JSON	Reactions	Model / route
0001		output/0001.json	8	OpenAI ChatGPT web model label 5.5pro, followed by HailBerry validation and human review.
0059		output/0059.json	1	OpenAI ChatGPT web model label 5.5pro, followed by HailBerry validation and human review.
0118		output/0118.json	9	Anthropic Claude Fable 5 via Claude CLI (claude --model fable --effort max), followed by HailBerry validation and human review.

Repository Contents

HailBerry is kept as a monorepo because the tools share the same schema, file conventions, prompts, and validation logic.

• packages/pipeline: batch extraction, model/API calls, parsing, repair, consensus, reporting, RDKit checks, and packaging helpers.
• packages/review-tool: local web workbench for comparing source images with reaction JSON and making human corrections.
• packages/prompts: prompt files and prompt constants used for reaction-figure extraction and repair.
• packages/validators: standalone validators for JSON files, annotation directories, and flat annotation ZIP files.
• examples/synthetic: a tiny hand-made demo.
• examples/cc-by-nc: selected CC-BY-NC-4.0 input-output examples for non-commercial demonstration.

Features

• Model-assisted reaction extraction from chemistry figures.
• Prompt files for strict reaction-annotation JSON output.
• Parsers and repair utilities for model responses.
• Candidate comparison and consensus helpers.
• RDKit and schema validation tools.
• A local human-review web workbench.
• Synthetic examples for trying the tools.
• Small non-commercial visual examples showing input images and paired JSON outputs.

Software Components

HailBerry Pipeline handles model calls, parsing, refinement, consensus, validation, reports, and packaging. Historical run scripts are kept for transparency, while stable utilities and README commands are the recommended entry points.

HailBerry Review is a local browser workbench for human correction. It can load multiple draft versions, show matching source images, edit entities, preview SMILES with RDKit.js, optionally use Python RDKit on the server, mark samples as reviewed or uncertain, and export corrected JSON.

HailBerry Prompts contains the domain-specific instructions used by the workflow. The prompts encode rules for multi-reaction figures, selected examples, product-only examples, R-group tables, wildcard placeholders, stereochemistry, ring sizes, reagent-condition separation, and strict JSON output.

HailBerry Validators checks JSON syntax, top-level schema, allowed entity types, required fields, optional filename patterns, optional product presence, directories, and flat ZIP files. Validation does not prove full chemical correctness.

Review Tool Preview

HailBerry Review is the easiest way to understand the workflow because it runs locally and does not require any API key. It loads a source image, shows the current reaction JSON, lets you edit each entity, previews SMILES with RDKit, and exports corrected annotations.

Current web UI language: Chinese only for now. The English documentation explains the workflow, but the in-browser labels and buttons have not been localized yet.

The screenshots below use the CC-BY-NC-4.0 example image 0001 from examples/cc-by-nc/input/0001.png.

The editor is split into three working areas:

• Left: sample list, search, filters, and review progress.
• Middle: source image and structured reaction editor.
• Right: RDKit structure preview, live JSON preview, validation, and export buttons.

One-Click Review Launch

The review tool has one-click launch scripts for common desktop systems. These scripts start the local server and open the browser automatically. Keep the terminal window open while using the web page.

System	One-click file	How to use
macOS	packages/review-tool/start_mac.command	Double-click the file. If macOS says it cannot run, use chmod +x packages/review-tool/start_mac.command once.
Windows	packages/review-tool/start_windows.bat	Double-click the file. It tries py -3 server.py, then falls back to python server.py.
Linux	packages/review-tool/start_linux.sh	Run chmod +x packages/review-tool/start_linux.sh once, then double-click it or run it from a terminal.

Manual launch works on every system:

cd packages/review-tool
python3 server.py

The page usually opens at:

http://127.0.0.1:8765/

If that port is busy, HailBerry Review automatically picks another nearby port and prints the exact URL in the terminal.

To review your own annotations, put draft JSON files here:

packages/review-tool/data/base_submissions/my-run/sample-001.json
packages/review-tool/data/base_submissions/my-run/sample-002.json

Put matching images with the same file stem here:

packages/review-tool/data/images/sample-001.png
packages/review-tool/data/images/sample-002.svg

Then launch with that base version:

cd packages/review-tool
HAILBERRY_REVIEW_BASE=my-run python3 server.py

Basic review flow:

1. Choose a sample from the left list.
2. Check the source image against the structured reaction editor.
3. Click a SMILES text field to preview the molecule in the RDKit panel.
4. Edit incorrect entities, relations, or reaction counts.
5. Use 导出前检查 / Validate before export.
6. Use 导出 annotations zip to export corrected JSON files.

Prompt Assets

The main prompt files are:

• packages/prompts/prompts/base_reaction_annotation_prompt.txt
• packages/prompts/prompts/final_fable_prompt_exact.txt
• packages/prompts/prompts/pipeline_prompts.py
• packages/pipeline/scripts/prompting/base_prompt.txt
• packages/pipeline/scripts/prompting/prompts.py

The exact long-form final prompt is included because prompt wording is part of the method. Project-authored prompt material is released under MIT.

Method

The intended workflow is:

1. Prepare source images.
2. Run model-assisted extraction with chemistry-specific prompts.
3. Parse model output into the shared reactions JSON schema.
4. Normalize obvious formatting issues, such as fenced code blocks or trailing prose.
5. Run RDKit and schema checks where possible.
6. Use repair or refinement passes for invalid or uncertain SMILES.
7. Compare multiple model outputs when available.
8. Review uncertain or high-value samples in HailBerry Review.
9. Export clean JSON and validation reports.
10. Export clean JSON files or a validation report for downstream use.

The system treats model outputs as drafts, not truth. Human review is expected for difficult images, ambiguous arrows, selected examples, uncertain stereochemistry, R-group substitution tables, and cases where a visually plausible SMILES string fails to represent the intended chemistry.

Known Limits

• The tools do not guarantee chemically correct annotations.
• OCR and vision models can hallucinate, omit products, or misread stereochemistry.
• RDKit checks parseability and selected structural properties, not full reaction correctness.
• Literature or web-search helpers can suggest context but should not override visible evidence in the image without human judgment.
• Included examples are small demonstrations; production use should supply its own images and human review.

Beginner Quick Start

Step 1: clone or open the repository, then run the local checks that do not need any API key.

Validate the included synthetic demo:

cd /path/to/HailBerry
python3 packages/validators/hailberry_validate.py packages/review-tool/data/base_submissions/demo --require-product

Validate the included CC-BY-NC examples:

python3 packages/validators/hailberry_validate.py examples/cc-by-nc/output --require-product

Step 2: open the review tool. This also does not need an API key.

cd packages/review-tool
python3 server.py

Open the URL printed in the terminal. It is usually:

http://127.0.0.1:8765/

Step 3: set up the model pipeline. This is only needed if you want HailBerry to call a vision model on your own images.

cd /path/to/HailBerry/packages/pipeline
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env.local

Edit packages/pipeline/.env.local. For example, to use DashScope:

DASHSCOPE_API_KEY=
DASHSCOPE_VISION_MODEL=qwen-vl-max-latest

Or to use OpenAI:

OPENAI_API_KEY=
OPENAI_STRONG_MODEL=gpt-5.2

Or to use Anthropic:

ANTHROPIC_API_KEY=
ANTHROPIC_STRONG_MODEL=claude-sonnet-4.6

Or to use Zhipu:

ZHIPU_API_KEY=
ZHIPU_VISION_MODEL=glm-4.5v

Check that HailBerry can see your key:

python3 scripts/core/check_env.py

Step 4: prepare your own image. The default image folder is:

packages/pipeline/data/images/

Create it and put your image there with a numeric id:

mkdir -p data/images
cp /path/to/your/reaction-image.png data/images/0001.png

Step 5: run a direct final-annotation draft:

python3 -m scripts.api.final_annotate 0001 --providers dashscope

If you configured OpenAI instead, run:

python3 -m scripts.api.final_annotate 0001 --providers openai

If you configured Anthropic instead, run:

python3 -m scripts.api.final_annotate 0001 --providers anthropic

If you configured Zhipu instead, run:

python3 -m scripts.api.final_annotate 0001 --providers zhipu

The output is written to:

packages/pipeline/work/final_outputs/0001/<provider>.json

That file contains the raw model text and a parsed field. The parsed field is the reaction annotation JSON draft. Treat it as a draft and review it before using it.

To save only the annotation JSON from the parsed field, run this from packages/pipeline:

python3 - <<'PY'
import json
from pathlib import Path

provider = "dashscope"  # change to "openai", "anthropic", or "zhipu" if that is what you used
src = Path("work/final_outputs/0001") / f"{provider}.json"
dst = Path("work/annotations/0001.json")
dst.parent.mkdir(parents=True, exist_ok=True)
payload = json.loads(src.read_text(encoding="utf-8"))["parsed"]
dst.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
print(f"Wrote {dst}")
PY

Validate the exported annotation:

python3 ../validators/hailberry_validate.py work/annotations --require-product

Step 6: print the prompt index if you want to inspect the prompt files:

python3 scripts/prompt_summary.py

For a custom image or work directory, set these in packages/pipeline/.env.local:

HAILBERRY_IMAGE_DIR=/absolute/path/to/images
HAILBERRY_WORK_DIR=/absolute/path/to/work

Review Tool Data Layout

Draft JSON files:

packages/review-tool/data/base_submissions/my-run/sample-001.json
packages/review-tool/data/base_submissions/my-run/sample-002.json

Matching source images:

packages/review-tool/data/images/sample-001.png
packages/review-tool/data/images/sample-002.svg

Supported image suffixes are .png, .jpg, .jpeg, .webp, and .svg.

Common Questions

Question	Short answer
Do I need an API key?	No for the review tool, validators, prompt inspection, and included demos. Yes for model-assisted extraction on your own images.
Where do my images go?	For model extraction, use packages/pipeline/data/images/ or set HAILBERRY_IMAGE_DIR. For the review tool, use packages/review-tool/data/images/.
Where is the model output?	packages/pipeline/work/final_outputs/<image_id>/<provider>.json. The usable annotation draft is in the parsed field.
Is a RDKit warning always a chemistry mistake?	No. RDKit checks whether SMILES-like fields parse as molecules; text conditions, R-groups, placeholders, and ambiguous figures still need human judgment.
Is everything in this repository MIT?	No. Project-authored code, prompts, and documentation are MIT. Third-party dependencies keep their licenses. examples/cc-by-nc/ is CC-BY-NC-4.0 for non-commercial demonstration.
Why is the web UI in Chinese?	The review tool was first built for a Chinese-speaking review workflow. English documentation is included, but browser labels have not been localized yet.

Background

HailBerry was first built during the Bohrium Scientific Data Annotation Competition, Track 2. The public repository is maintained as a general reaction-annotation toolkit.

Competition page:

https://www.bohrium.com/competitions/76372386918?tab=introduce

License Boundary

Project-authored code, prompts, and documentation are released under the MIT License. MIT is an OSI-approved open-source license and is permissive enough for reuse, modification, distribution, and sublicensing.

HailBerry is not entirely third-party-free. The project workflow, prompts, review-tool application code, validators, examples written by the project, and documentation are project-authored unless a file says otherwise. The main third-party open-source components are listed below.

Component	Where used	Included in this repository?	License / notice
RDKit.js (@rdkit/rdkit)	Browser-side molecule preview in HailBerry Review	Yes, packages/review-tool/app/vendor/RDKit_minimal.js and .wasm	BSD-3-Clause; see packages/review-tool/app/vendor/RDKit_NOTICE.txt
RDKit Python package	Optional server-side SMILES parsing, rendering, and validation	No, installed by pip from packages/pipeline/requirements.txt	BSD-3-Clause
OpenAI Python SDK	Optional OpenAI API calls	No, installed by pip	Apache-2.0
Anthropic Python SDK	Optional Anthropic API calls	No, installed by pip	MIT
Google Gen AI SDK	Optional Google model API experiments and historical scripts	No, installed by pip	Apache-2.0
Hugging Face Hub	Optional local-model utilities	No, installed by pip	Apache-2.0
Pillow	Image loading, conversion, and report rendering helpers	No, installed by pip	MIT-CMU
Pydantic	Data validation helpers	No, installed by pip	MIT
python-dotenv	Loading .env.local files	No, installed by pip	BSD-3-Clause
NumPy	Numeric utilities used by chemistry/model helpers	No, installed by pip	BSD-3-Clause family with bundled dependency notices from NumPy wheels
Requests	HTTP API calls	No, installed by pip	Apache-2.0
tqdm	Progress bars in batch scripts	No, installed by pip	MPL-2.0 and MIT

The Python dependencies above are normal runtime dependencies; their source code is not copied into this repository. If you redistribute a packaged environment or binary build, keep the corresponding third-party license notices from those packages.

The CC-BY-NC example files under examples/cc-by-nc/ and review-tool screenshots under docs/assets/review-tool/ are separate data-license exceptions:

• examples/cc-by-nc/input/*.png: source images used here under CC-BY-NC-4.0.
• examples/cc-by-nc/output/*.json: HailBerry-generated annotation examples paired with those images, shared here under CC-BY-NC-4.0 for non-commercial demonstration.
• docs/assets/review-tool/*.png: screenshots of HailBerry Review that display the 0001 CC-BY-NC-4.0 example image.
• These example files may be copied, displayed, shared, and adapted only for non-commercial purposes, with attribution and a link to the CC-BY-NC-4.0 license.
• They must not be used for commercial products, paid datasets, commercial model training, commercial evaluation services, or any other use primarily intended for commercial advantage or monetary compensation.

Attribution for the example data:

Competition input images are from the Bohrium Scientific Data Annotation Competition, Track 2:

https://www.bohrium.com/competitions/76372386918?tab=introduce

Output JSON files are HailBerry-generated annotation examples produced by the Project HailBerry authors and paired with those source images for demonstration.

中文

项目概览

HailBerry 是一个开源工具箱，用来把化学反应图片转换成结构化 JSON 标注，并辅助人工复查、校验和打包。

这个仓库包含工作流代码、prompt、校验器、本地复查界面、方法说明和精选输入输出示例。

工作流图示

先看这里

三句话版：

• HailBerry 可以把化学反应图片变成结构化 JSON 草稿。
• 不填 API key 也可以打开复查工具、校验内置示例、查看 prompt。
• 只有当你想让 HailBerry 对自己的图片生成新草稿时，才需要准备模型 API key。

如果你想...	先看哪里	会用到
不配置模型，先看看工具长什么样	新手快速开始第 1-2 步	内置 demo 和本地复查工具
复查自己的 JSON 文件	Review Tool 数据目录	packages/review-tool/data/base_submissions/
让模型从图片生成草稿	新手快速开始第 3-5 步	packages/pipeline/.env.local 和 final_annotate.py
检查 JSON 是否合规	新手快速开始第 1 步	packages/validators/hailberry_validate.py
看清许可证和示例数据边界	许可证边界	项目自写代码是 MIT；精选示例数据是 CC-BY-NC

开始之前

HailBerry 是本地工具箱，不是一个托管好的在线服务。复查工具和校验器可以直接用仓库内置 demo 运行，不需要 API key。想让 HailBerry 对自己的化学反应图片调用视觉模型自动抽取，则需要准备自己的模型 API key。

API key 读取位置是：

packages/pipeline/.env.local

先复制 packages/pipeline/.env.example 生成 .env.local，然后只填写你准备使用的 provider。.env.local 已被 Git 忽略，不要提交真实 key。

第一次跑完整 annotation，可以先从下面几个内置支持的模型 API 里选一个。其他模型服务需要在代码里新增一个 provider wrapper，不是只填 API key 就能直接跑。

Provider	在 .env.local 填写	命令里使用
OpenAI	OPENAI_API_KEY	--providers openai
Anthropic	ANTHROPIC_API_KEY	--providers anthropic
DashScope / Qwen-VL	DASHSCOPE_API_KEY	--providers dashscope
Zhipu GLM	ZHIPU_API_KEY	--providers zhipu

一开始只填一个 provider 就可以。之后如果想比较多个模型输出，再继续增加其他厂商 key。

工具做什么

HailBerry 的输入是一张化学反应示意图，输出是类似下面的 JSON 文件：

{
  "reactions": [
    [
      {"type": "reactants", "text": "CC(=O)O", "relations": []},
      {"type": "reactants", "text": "CO", "relations": []},
      {"type": "products", "text": "COC(C)=O", "relations": ["methyl acetate"]}
    ]
  ]
}

允许的实体类型：

• reactants
• reagent
• products
• temperature
• time
• other conditions

reactants、reagent 和 products 通常是 SMILES。条件字段可以是普通文本。relations 用于记录收率、ee、dr、当量、方法标签、R 基说明、selected-example 说明、不确定性说明或其他图中可见元信息。

输入输出可视化示例

下面的表格展示了 3 张 CC-BY-NC-4.0 授权图片和配套 HailBerry JSON 输出。这些内容只用于演示，不是标准答案或 benchmark。

CC-BY-NC-4.0 许可证文本：

https://creativecommons.org/licenses/by-nc/4.0/

ID	输入图片	输出 JSON	反应条数	模型 / 流程
0001		output/0001.json	8	OpenAI ChatGPT 网页端模型标签 5.5pro，之后经 HailBerry 校验和人工复查。
0059		output/0059.json	1	OpenAI ChatGPT 网页端模型标签 5.5pro，之后经 HailBerry 校验和人工复查。
0118		output/0118.json	9	Anthropic Claude Fable 5，通过 Claude CLI (claude --model fable --effort max) 调用，之后经 HailBerry 校验和人工复查。

仓库内容

HailBerry 保持为单仓库 monorepo，因为这些工具共享同一个 schema、文件约定、prompt 和校验逻辑。

• packages/pipeline：批量抽取、模型/API 调用、解析、修复、共识合并、报告、RDKit 检查和打包辅助。
• packages/review-tool：本地网页复查工具，用于对照原图检查和修改 reaction JSON。
• packages/prompts：用于反应图抽取和修复的 prompt 文件和 prompt 常量。
• packages/validators：可独立使用的 JSON、目录和扁平 annotation ZIP 校验器。
• examples/synthetic：手工制作的小型演示。
• examples/cc-by-nc：精选 CC-BY-NC-4.0 输入输出示例，仅作非商业演示。

功能

• 从化学反应图片中进行模型辅助抽取。
• 提供严格 reaction annotation JSON 输出的 prompt 文件。
• 解析和修复模型响应。
• 对多个候选结果进行比较和共识合并。
• 使用 RDKit 和 schema 进行校验。
• 提供本地人工复查网页工具。
• 提供可直接试用的虚构示例。
• 提供少量非商业可视化示例，展示输入图片与配套 JSON 输出。

软件组件

HailBerry Pipeline 负责模型调用、解析、精修、共识合并、校验、报告和打包。历史运行脚本为了透明度保留；稳定工具和 README 中的命令是推荐入口。

HailBerry Review 是本地浏览器复查工具，用于人工修订。它可以读取多个底稿版本、显示同名源图片、编辑实体、使用 RDKit.js 预览 SMILES、在可用时调用服务端 Python RDKit、标记已复查或存疑，并导出修订后的 JSON。

HailBerry Prompts 包含工作流使用的领域专用指令。Prompt 中编码了多反应图、selected examples、product-only examples、R-group 表、通配符占位、立体化学、环大小、试剂/条件拆分和严格 JSON 输出等规则。

HailBerry Validators 检查 JSON 语法、顶层 schema、允许的实体类型、必需字段、可选文件名规则、可选产物存在性、目录和扁平 ZIP。校验不能证明完整化学正确性。

Review Tool 界面预览

HailBerry Review 是最直观的入口，因为它完全在本地运行，而且不需要 API key。它会加载源图片和当前 reaction JSON，让你逐个编辑实体，用 RDKit 预览 SMILES，并导出修订后的 annotation。

当前网页前端暂时只有中文。英文文档会说明使用流程，但浏览器里的按钮和提示还没有英文界面。

下面截图使用 examples/cc-by-nc/input/0001.png 中的 CC-BY-NC-4.0 示例图片。

界面分成三个主要工作区：

• 左侧：样本列表、搜索、筛选和复查进度。
• 中间：源图片和结构化 reaction 编辑器。
• 右侧：RDKit 结构预览、实时 JSON 预览、校验和导出按钮。

一键启动 Review Tool

Review tool 已经准备了常见桌面系统的一键启动脚本。这些脚本会启动本地服务，并自动打开浏览器。使用网页时不要关闭终端窗口。

系统	一键启动文件	使用方式
macOS	packages/review-tool/start_mac.command	双击文件即可。如果 macOS 提示无法运行，先执行一次 chmod +x packages/review-tool/start_mac.command。
Windows	packages/review-tool/start_windows.bat	双击文件即可。脚本会先尝试 py -3 server.py，失败后自动尝试 python server.py。
Linux	packages/review-tool/start_linux.sh	先执行一次 chmod +x packages/review-tool/start_linux.sh，之后可以双击或在终端运行。

所有系统也都可以手动启动：

cd packages/review-tool
python3 server.py

页面通常会打开在：

http://127.0.0.1:8765/

如果端口被占用，HailBerry Review 会自动换到附近可用端口，并在终端打印准确 URL。

如果要复查自己的 annotation，把底稿 JSON 放到：

packages/review-tool/data/base_submissions/my-run/sample-001.json
packages/review-tool/data/base_submissions/my-run/sample-002.json

把同名图片放到：

packages/review-tool/data/images/sample-001.png
packages/review-tool/data/images/sample-002.svg

然后指定这个底稿版本启动：

cd packages/review-tool
HAILBERRY_REVIEW_BASE=my-run python3 server.py

基本复查流程：

1. 从左侧列表选择样本。
2. 对照源图片检查中间的结构化 reaction 编辑器。
3. 点击 SMILES 文本框，在右侧 RDKit 面板预览分子。
4. 修改错误实体、relations 或 reaction 数量。
5. 点击 导出前检查。
6. 点击 导出 annotations zip 导出修订后的 JSON 文件。

Prompt 资产

主要 prompt 文件：

• packages/prompts/prompts/base_reaction_annotation_prompt.txt
• packages/prompts/prompts/final_fable_prompt_exact.txt
• packages/prompts/prompts/pipeline_prompts.py
• packages/pipeline/scripts/prompting/base_prompt.txt
• packages/pipeline/scripts/prompting/prompts.py

完整长 prompt 也放入仓库，因为 prompt 文本本身是方法的一部分。项目自写 prompt 以 MIT 开源。

方法

推荐工作流：

1. 准备源图片。
2. 使用化学专用 prompt 进行模型辅助抽取。
3. 将模型输出解析为统一的 reactions JSON schema。
4. 规范化明显格式问题，例如代码块包裹或尾部解释文本。
5. 尽可能运行 RDKit 和 schema 检查。
6. 对无效或不确定 SMILES 运行修复或精修流程。
7. 在有多个模型输出时进行对比。
8. 在 HailBerry Review 中复查不确定或高价值样本。
9. 导出干净 JSON 和校验报告。
10. 导出干净 JSON 文件或校验报告，供后续使用。

系统把模型输出视为底稿，而不是事实。对于困难图片、模糊箭头、selected examples、不确定立体化学、R-group 取代表，以及视觉上合理但不一定代表目标结构的 SMILES，都应进行人工复查。

已知局限

• 工具不能保证标注一定化学正确。
• OCR 和视觉模型可能幻觉、漏产物或误读立体化学。
• RDKit 主要检查可解析性和部分结构属性，不等于完整反应正确性。
• 文献或网络搜索辅助可以提供上下文，但不应在缺少人工判断时覆盖图片中的可见证据。
• 内置示例只是小型演示；实际使用时应提供自己的图片并进行人工复查。

新手快速开始

第 1 步：进入仓库，先跑不需要 API key 的本地校验。

校验内置虚构 demo：

cd /path/to/HailBerry
python3 packages/validators/hailberry_validate.py packages/review-tool/data/base_submissions/demo --require-product

校验内置 CC-BY-NC 示例：

python3 packages/validators/hailberry_validate.py examples/cc-by-nc/output --require-product

第 2 步：打开本地复查工具。这个步骤也不需要 API key。

cd packages/review-tool
python3 server.py

打开终端打印出来的 URL，通常是：

http://127.0.0.1:8765/

第 3 步：配置模型 pipeline。只有当你想让 HailBerry 调用视觉模型处理自己的图片时，才需要这一步。

cd /path/to/HailBerry/packages/pipeline
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env.local

编辑 packages/pipeline/.env.local。例如使用 DashScope：

DASHSCOPE_API_KEY=
DASHSCOPE_VISION_MODEL=qwen-vl-max-latest

或者使用 OpenAI：

OPENAI_API_KEY=
OPENAI_STRONG_MODEL=gpt-5.2

或者使用 Anthropic：

ANTHROPIC_API_KEY=
ANTHROPIC_STRONG_MODEL=claude-sonnet-4.6

或者使用 Zhipu：

ZHIPU_API_KEY=
ZHIPU_VISION_MODEL=glm-4.5v

检查 HailBerry 是否读到了 key：

python3 scripts/core/check_env.py

第 4 步：准备自己的图片。默认图片目录是：

packages/pipeline/data/images/

创建目录，并把图片按数字 id 命名放进去：

mkdir -p data/images
cp /path/to/your/reaction-image.png data/images/0001.png

第 5 步：运行直接生成 final annotation 草稿的命令：

python3 -m scripts.api.final_annotate 0001 --providers dashscope

如果你配置的是 OpenAI，则运行：

python3 -m scripts.api.final_annotate 0001 --providers openai

如果你配置的是 Anthropic，则运行：

python3 -m scripts.api.final_annotate 0001 --providers anthropic

如果你配置的是 Zhipu，则运行：

python3 -m scripts.api.final_annotate 0001 --providers zhipu

输出文件会写到：

packages/pipeline/work/final_outputs/0001/<provider>.json

这个文件里包含模型原始输出和 parsed 字段。parsed 字段就是 reaction annotation JSON 草稿。它仍然是草稿，使用前建议人工复查。

如果只想把 parsed 字段里的 annotation JSON 单独保存出来，可以在 packages/pipeline 目录运行：

python3 - <<'PY'
import json
from pathlib import Path

provider = "dashscope"  # 如果你用的是 openai、anthropic 或 zhipu，就改成对应 provider
src = Path("work/final_outputs/0001") / f"{provider}.json"
dst = Path("work/annotations/0001.json")
dst.parent.mkdir(parents=True, exist_ok=True)
payload = json.loads(src.read_text(encoding="utf-8"))["parsed"]
dst.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
print(f"Wrote {dst}")
PY

校验导出的 annotation：

python3 ../validators/hailberry_validate.py work/annotations --require-product

第 6 步：如果想查看 prompt 文件索引，可以运行：

python3 scripts/prompt_summary.py

如果想使用自定义图片目录或工作目录，可以在 packages/pipeline/.env.local 里设置：

HAILBERRY_IMAGE_DIR=/absolute/path/to/images
HAILBERRY_WORK_DIR=/absolute/path/to/work

Review Tool 数据目录

底稿 JSON 文件：

packages/review-tool/data/base_submissions/my-run/sample-001.json
packages/review-tool/data/base_submissions/my-run/sample-002.json

同名源图片：

packages/review-tool/data/images/sample-001.png
packages/review-tool/data/images/sample-002.svg

支持的图片后缀为 .png、.jpg、.jpeg、.webp 和 .svg。

常见问题

问题	简短回答
我一定需要 API key 吗？	不一定。复查工具、校验器、prompt 查看和内置 demo 都不需要；只有让模型处理自己的图片时才需要。
我的图片应该放在哪里？	模型抽取用 packages/pipeline/data/images/，也可以设 HAILBERRY_IMAGE_DIR。复查工具用 packages/review-tool/data/images/。
模型输出在哪里？	packages/pipeline/work/final_outputs/<image_id>/<provider>.json，可用的 annotation 草稿在 parsed 字段里。
RDKit warning 一定表示错了吗？	不一定。RDKit 主要检查 SMILES-like 字段能否解析；文字条件、R 基、占位符和模糊图片仍需人工判断。
整个仓库都是 MIT 吗？	不是。项目自写代码、prompt 和文档是 MIT；第三方依赖保持自身许可证；examples/cc-by-nc/ 是 CC-BY-NC-4.0 非商业演示。
为什么网页前端只有中文？	Review tool 最初按中文复查流程开发。仓库有英文文档，但浏览器按钮和提示还没有英文界面。

项目背景

HailBerry 最初开发于 Bohrium 科学数据标注大赛赛道 2。公开仓库按通用 reaction annotation 工具箱维护。

比赛页面：

https://www.bohrium.com/competitions/76372386918?tab=introduce

许可证边界

项目自有代码、prompt 和文档使用 MIT License 开源。MIT 是 OSI 认可的开源许可证，允许复用、修改、分发和再授权。

HailBerry 不是完全没有第三方开源组件。除非文件中另有说明，项目工作流代码、prompt、review tool 应用代码、校验器、项目自写示例和文档都由本项目作者编写。主要第三方开源组件如下。

组件	使用位置	是否随仓库一起包含	许可证 / 说明
RDKit.js (@rdkit/rdkit)	HailBerry Review 浏览器端分子预览	是，packages/review-tool/app/vendor/RDKit_minimal.js 和 .wasm	BSD-3-Clause；见 packages/review-tool/app/vendor/RDKit_NOTICE.txt
RDKit Python package	可选的服务端 SMILES 解析、渲染和校验	否，通过 packages/pipeline/requirements.txt 用 pip 安装	BSD-3-Clause
OpenAI Python SDK	可选 OpenAI API 调用	否，通过 pip 安装	Apache-2.0
Anthropic Python SDK	可选 Anthropic API 调用	否，通过 pip 安装	MIT
Google Gen AI SDK	可选 Google 模型 API 实验和历史脚本	否，通过 pip 安装	Apache-2.0
Hugging Face Hub	可选本地模型工具	否，通过 pip 安装	Apache-2.0
Pillow	图片读取、转换和报告渲染辅助	否，通过 pip 安装	MIT-CMU
Pydantic	数据校验辅助	否，通过 pip 安装	MIT
python-dotenv	读取 .env.local 文件	否，通过 pip 安装	BSD-3-Clause
NumPy	化学和模型辅助工具中的数值计算	否，通过 pip 安装	BSD-3-Clause 系列，并包含 NumPy wheel 自带的第三方依赖说明
Requests	HTTP API 调用	否，通过 pip 安装	Apache-2.0
tqdm	批处理脚本进度条	否，通过 pip 安装	MPL-2.0 和 MIT

上表中的 Python 依赖是普通运行时依赖，其源码没有复制进本仓库。如果你重新分发打包环境或二进制构建，请同时保留这些依赖自身的第三方许可证说明。

examples/cc-by-nc/ 下的 CC-BY-NC 示例文件，以及 docs/assets/review-tool/ 下的 review tool 截图，是单独的数据许可证例外：

• examples/cc-by-nc/input/*.png：源图片，在此按 CC-BY-NC-4.0 使用。
• examples/cc-by-nc/output/*.json：与这些图片配套的 HailBerry annotation 示例，在此按 CC-BY-NC-4.0 进行非商业演示分享。
• docs/assets/review-tool/*.png：HailBerry Review 的截图，其中展示了 0001 这张 CC-BY-NC-4.0 示例图片。
• 这些示例文件可以在非商业目的下复制、展示、分享和改编，但必须署名并附 CC-BY-NC-4.0 许可证链接。
• 它们不得用于商业产品、付费数据集、商业模型训练、商业评测服务，或任何主要以商业利益或金钱回报为目的的用途。

示例数据署名：

比赛输入图片来自 Bohrium 科学数据标注大赛赛道 2：

https://www.bohrium.com/competitions/76372386918?tab=introduce

输出 JSON 文件是 Project HailBerry 作者使用 HailBerry 生成的 annotation 示例，并与这些源图片配套用于演示。