Skip to content

Commit f2bd17a

Browse files
dale-wahlClaude Codestijn-uva
authored
Zeeschuimer auto map w/ CI (#593)
* add mapper helper and crazy CI * add streaming and timing and other stuff so i can see what's going on * no streaming here * use qwen; lib.js is global apparently, add some helper functions; add a bunch of DON'T Do's; clean up code block fences; do some surface lint tests * CI: use qwen, log warnings and add to PR * clear up AI generated block markers note * one PR per module * make platforms dynamic * add MapItemException to shared helpers * Update rules on regex and notify reviewers * 🤨 fix merge... again? * add extensions syslink to .gitignore * streamline translation helpers for prompt and linting * map_item_converter: fix _strip_js_comments * zeeschuimer_map_item_sync.yml: move python functions to new map_item_ci.py * lock create-pull-request github action (it has a ZS token!) * add tests for map_item_sync CI * add lint rules for better warnings * allow tests and linting without LLM packages * fix (🤞) regex escaping newlines and stuff * check for preexisting map_item * get rid of streaming (used for dev): easier extraction of response * don't hardcode packages: use them from setup.py * update CI tests * CI: write requirements w/ newlines * alllow LLM provider, base_url, and api_key to be inputs * llm.py add litellm provider * llm litellm adapter fix * add API endpoint for mapped JSON dataset items * map_item_converter: pass api key * delete unused item mapper API endpoint * llm litellm provider if at first you do not succeed or the one about the definition of madness * llm.py: set_structure pass additional langchain options. * map_item_converter: update token max and examine JSON parsing errors * update map_item_converter to allow prompt mode which does not use json_schema (gpt-oss-120b fails w/ schema) * pass structured arg in zeeschuimer CI * update zeeschuimer_map_item_sync.yml to use Github variables * Move map_item helper scripts into their own folder * Don't require a config reader in LLMAdapter * Update converter.py for LLMAdapter changes * Fix botched merge * Further cleanup * More repathing * Fix tests --------- Co-authored-by: Claude Code <claude-code@noreply.github.com> Co-authored-by: Stijn Peeters <stijn.peeters@uva.nl>
1 parent e612241 commit f2bd17a

13 files changed

Lines changed: 2985 additions & 224 deletions

File tree

‎.github/workflows/zeeschuimer_map_item_sync.yml‎

Lines changed: 243 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,252 @@
1-
# Bootstrap the Zeeschuimer map_item sync workflow
2-
# This is necessary to test workflow in PR (so far as I can tell)
1+
# Auto-translate Zeeschuimer datasource `map_item` functions from Python to JS
2+
# and open a draft PR per module against digitalmethodsinitiative/zeeschuimer.
3+
#
4+
# Triggers on pushes to master that touch any Zeeschuimer datasource (or the
5+
# helper script itself). Also exposes a `workflow_dispatch` trigger with a
6+
# `bootstrap` input for the initial run that translates every Zeeschuimer
7+
# datasource at once (single PR). Datasources without a matching Zeeschuimer
8+
# module — e.g. facebook — are skipped automatically.
9+
#
10+
# Architecture: a `detect` job groups changed files by module and emits a
11+
# matrix; a `sync` job fans out one parallel run per module, each opening
12+
# (or updating) its own PR on a stable per-module branch. The planning
13+
# and PR-body logic are in `helper-scripts/map_item/ci.py` (# unit-tested
14+
# in tests/test_map_item_sync.py); the LLM translation and JS
15+
# splicing live in `helper-scripts/map_item/converter.py`.
16+
#
17+
# Required secrets (configured in repo Settings -> Secrets and variables -> Actions):
18+
# ZEESCHUIMER_APP_ID - numeric App ID of the GitHub App installed on
19+
# digitalmethodsinitiative/zeeschuimer with permissions
20+
# contents:write + pull-requests:write (and nothing else)
21+
# ZEESCHUIMER_APP_PRIVATE_KEY - full PEM private key for that App (including BEGIN/END lines)
22+
# DMI_OLLAMA_KEY - API key for https://ollama.digitalmethods.net (legacy fallback)
23+
#
24+
# Optional overrides — set in repo Settings -> Secrets and variables -> Actions to change
25+
# the provider used by automatic (push-triggered) runs without editing this file.
26+
# Resolution order for each setting (first non-empty wins):
27+
# API key: workflow_dispatch input -> LLM_PROVIDER_API_KEY secret -> DMI_OLLAMA_KEY secret
28+
# provider/url/model/output_mode: workflow_dispatch input -> repo variable -> hardcoded default below
29+
#
30+
# LLM_PROVIDER_API_KEY (secret) - generic key for the active provider; swap when switching providers
31+
# LLM_PROVIDER (variable) - provider type for LLMAdapter (default: ollama)
32+
# LLM_BASE_URL (variable) - provider base URL (default: https://ollama.digitalmethods.net)
33+
# LLM_MODEL (variable) - model name (default: qwen2.5-coder:14b)
34+
# LLM_OUTPUT_MODE (variable) - structured or prompt (default: structured)
335

436
name: Sync Zeeschuimer map_item from 4CAT
537

638
on:
39+
push:
40+
branches: [master]
41+
paths:
42+
# Only datasource changes drive a push-triggered translation: the detect
43+
# job's plan-matrix diffs `datasources/**` and nothing else
44+
- 'datasources/**/search_*.py'
45+
- '.github/workflows/zeeschuimer_map_item_sync.yml'
746
workflow_dispatch:
47+
# NOTE: defaults are '' which falls through to Github settings (see Optional overrides above)
48+
inputs:
49+
bootstrap:
50+
description: 'Translate every Zeeschuimer datasource (initial sync, single PR). Ignored if "files" is set.'
51+
type: boolean
52+
default: false
53+
files:
54+
description: 'Space-separated list of datasource files to translate (e.g. "datasources/tiktok/search_tiktok.py"). Overrides bootstrap. One PR per module.'
55+
type: string
56+
default: ''
57+
llm_provider:
58+
description: 'LLM provider type for LLMAdapter. Leave blank to use LLM_PROVIDER variable or default (ollama).'
59+
type: string
60+
default: ''
61+
llm_base_url:
62+
description: 'LLM provider base URL. Leave blank to use LLM_BASE_URL variable or default (https://ollama.digitalmethods.net).'
63+
type: string
64+
default: ''
65+
llm_api_key:
66+
description: 'LLM API key. Leave blank to use LLM_PROVIDER_API_KEY secret or DMI_OLLAMA_KEY secret.'
67+
type: string
68+
default: ''
69+
model:
70+
description: 'LLM model name. Leave blank to use LLM_MODEL variable or default (qwen2.5-coder:14b). Examples: qwen2.5-coder:7b, deepseek-coder-v2:16b, gemma3:27b'
71+
type: string
72+
default: ''
73+
output_mode:
74+
description: 'LLM output mode (structured or prompt). Leave blank to use LLM_OUTPUT_MODE variable or default (structured). Use prompt for models that do not support structured output (e.g. gpt-oss-120b).'
75+
type: string
76+
default: ''
77+
78+
# Least privilege: this workflow's own GITHUB_TOKEN only needs to read the 4CAT
79+
permissions:
80+
contents: read
881

982
jobs:
10-
sync-map-item:
83+
detect:
84+
name: Detect modules to translate
1185
runs-on: ubuntu-latest
86+
outputs:
87+
mode: ${{ steps.plan.outputs.mode }}
88+
matrix: ${{ steps.plan.outputs.matrix }}
1289
steps:
13-
- name: Placeholder
14-
run: echo "Workflow scaffold is valid."
90+
- name: Checkout 4CAT
91+
uses: actions/checkout@v4
92+
with:
93+
# Full history: the push-event plan diffs `github.event.before`
94+
# against `github.sha`. A shallow clone may not contain `before` for
95+
# a multi-commit push, in which case the diff resolves to nothing and
96+
# the change is silently skipped.
97+
fetch-depth: 0
98+
99+
- name: Set up Python
100+
# `detect` runs map_item/ci.py (stdlib only — no LLM deps installed
101+
# here), but still needs a `python` on PATH; don't rely on the runner
102+
# image happening to provide one.
103+
uses: actions/setup-python@v5
104+
with:
105+
python-version: "3.11"
106+
107+
- name: Plan translation matrix
108+
id: plan
109+
env:
110+
EVENT_NAME: ${{ github.event_name }}
111+
INPUTS_FILES: ${{ inputs.files }}
112+
INPUTS_BOOTSTRAP: ${{ inputs.bootstrap }}
113+
BEFORE_SHA: ${{ github.event.before }}
114+
AFTER_SHA: ${{ github.sha }}
115+
# Validates paths against a strict datasource shape (dropping anything
116+
# else) and writes `mode` + `matrix` to $GITHUB_OUTPUT. See
117+
# helper-scripts/map_item/ci.py.
118+
run: python helper-scripts/map_item/ci.py plan-matrix
119+
120+
sync:
121+
name: Sync ${{ matrix.target.module }}
122+
needs: detect
123+
if: needs.detect.outputs.mode != 'none'
124+
runs-on: ubuntu-latest
125+
# Per-module concurrency: a newer push to master supersedes any in-flight
126+
# sync for the same module (LLM run gets cancelled, latest run wins).
127+
# Each matrix instance gets its own group, so different modules don't block.
128+
concurrency:
129+
group: zeeschuimer-sync-${{ matrix.target.module }}
130+
cancel-in-progress: true
131+
strategy:
132+
fail-fast: false
133+
matrix:
134+
target: ${{ fromJson(needs.detect.outputs.matrix) }}
135+
steps:
136+
- name: Checkout 4CAT
137+
uses: actions/checkout@v4
138+
with:
139+
# Full history so the PR-body builder can `git diff before..after`
140+
# for the changed Python file (see map_item/ci.py build-pr-body).
141+
fetch-depth: 0
142+
143+
- name: Set up Python
144+
uses: actions/setup-python@v5
145+
with:
146+
python-version: "3.11"
147+
148+
- name: Install LLM dependencies
149+
run: |
150+
# LLMAdapter (common/lib/llm.py) imports every provider's langchain
151+
# package at module load, so all of them are required even though we
152+
# only use the Ollama provider at runtime. Derive the exact list from
153+
# setup.py (single source of truth) so it can't drift from what 4CAT
154+
# declares; we install only this LLM subset, not all of 4CAT, to keep
155+
# the job light. Write the specs to a requirements file (one per line)
156+
# and install with `-r`, rather than an unquoted `pip install $VAR`:
157+
# that way a version specifier that contains a shell metacharacter
158+
# (e.g. a future `langchain_core>=0.3` pin — `>` is redirection) can't
159+
# be misparsed by the shell.
160+
python helper-scripts/map_item/ci.py llm-requirements > llm-requirements.txt
161+
echo "Installing from setup.py:"
162+
cat llm-requirements.txt
163+
pip install -r llm-requirements.txt
164+
165+
- name: Mint Zeeschuimer App token
166+
id: app_token
167+
uses: actions/create-github-app-token@v1
168+
with:
169+
app-id: ${{ secrets.ZEESCHUIMER_APP_ID }}
170+
private-key: ${{ secrets.ZEESCHUIMER_APP_PRIVATE_KEY }}
171+
owner: digitalmethodsinitiative
172+
repositories: zeeschuimer
173+
174+
- name: Checkout Zeeschuimer
175+
uses: actions/checkout@v4
176+
with:
177+
repository: digitalmethodsinitiative/zeeschuimer
178+
path: zeeschuimer-checkout
179+
token: ${{ steps.app_token.outputs.token }}
180+
181+
- name: Run translation
182+
env:
183+
PROVIDER_API_KEY: ${{ inputs.llm_api_key || secrets.LLM_PROVIDER_API_KEY || secrets.DMI_OLLAMA_KEY }}
184+
LLM_PROVIDER: ${{ inputs.llm_provider || vars.LLM_PROVIDER || 'ollama' }}
185+
LLM_BASE_URL: ${{ inputs.llm_base_url || vars.LLM_BASE_URL || 'https://ollama.digitalmethods.net' }}
186+
LLM_MODEL: ${{ inputs.model || vars.LLM_MODEL || 'qwen2.5-coder:14b' }}
187+
LLM_OUTPUT_MODE: ${{ inputs.output_mode || vars.LLM_OUTPUT_MODE || 'structured' }}
188+
# Pass matrix values through env rather than interpolating.
189+
# IS_BOOTSTRAP is always the literal true/false the detect job emitted
190+
# MODULE_FILES is a list of paths the detect job already validated against
191+
# `datasources/<module>/search_<name>.py` shape (no shell
192+
# metacharacters), so the unquoted `$MODULE_FILES` expansion is safe
193+
# and still word-splits into multiple --files arguments.
194+
IS_BOOTSTRAP: ${{ matrix.target.bootstrap }}
195+
MODULE_FILES: ${{ matrix.target.files }}
196+
run: |
197+
if [ "$IS_BOOTSTRAP" = "true" ]; then
198+
# Bootstrap translates every datasource in one run; --no-fail-fast
199+
# so one datasource failing doesn't abort the whole initial sync.
200+
python helper-scripts/map_item/converter.py \
201+
--bootstrap \
202+
--no-fail-fast \
203+
--zeeschuimer-checkout ./zeeschuimer-checkout \
204+
--output-manifest ./manifest.json
205+
else
206+
python helper-scripts/map_item/converter.py \
207+
--files $MODULE_FILES \
208+
--zeeschuimer-checkout ./zeeschuimer-checkout \
209+
--output-manifest ./manifest.json
210+
fi
211+
212+
- name: Build PR body
213+
id: pr_body
214+
env:
215+
MODULE: ${{ matrix.target.module }}
216+
BOOTSTRAP: ${{ matrix.target.bootstrap }}
217+
BEFORE_SHA: ${{ github.event.before }}
218+
AFTER_SHA: ${{ github.sha }}
219+
RUN_ID: ${{ github.run_id }}
220+
EVENT_NAME: ${{ github.event_name }}
221+
REPO: ${{ github.repository }}
222+
# Reads manifest.json, writes pr_body.md, and writes `title` to
223+
# $GITHUB_OUTPUT (delimiter form, injection-safe). See ci.py.
224+
run: python helper-scripts/map_item/ci.py build-pr-body --manifest manifest.json --out pr_body.md
225+
226+
- name: Check there are JS changes to PR
227+
id: have_changes
228+
working-directory: zeeschuimer-checkout
229+
run: |
230+
if [ -z "$(git status --porcelain)" ]; then
231+
echo "has_changes=false" >> "$GITHUB_OUTPUT"
232+
echo "No JS changes produced by translation; not opening a PR."
233+
else
234+
echo "has_changes=true" >> "$GITHUB_OUTPUT"
235+
fi
236+
237+
- name: Open or update Zeeschuimer PR
238+
if: steps.have_changes.outputs.has_changes == 'true'
239+
# Third-party action that operates with a write token to the Zeeschuimer
240+
# repo — pinned to a full commit SHA (the v6 release) rather than the
241+
# mutable `@v6` tag, so a tag move can't silently change what runs here.
242+
uses: peter-evans/create-pull-request@c5a7806660adbe173f04e3e038b0ccdcd758773c # v6
243+
with:
244+
path: zeeschuimer-checkout
245+
token: ${{ steps.app_token.outputs.token }}
246+
# Stable per-module branch: a fresh push that retranslates the same
247+
# module updates the same PR. Different modules never share a branch.
248+
branch: auto/4cat-map-item-sync-${{ matrix.target.module }}
249+
title: ${{ steps.pr_body.outputs.title }}
250+
commit-message: "chore: sync map_item for ${{ matrix.target.module }} from 4CAT ${{ github.sha }}"
251+
body-path: pr_body.md
252+
draft: true

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,9 @@ webtool/venv/
4646
*.ipynb
4747
venv/
4848
__pycache__/
49+
.claude/
50+
# ignore symlink at the repo root -> config/extensions.
51+
/extensions
4952

5053
# do not ignore interface images
5154
!webtool/static/img/*.png

‎common/lib/dataset.py‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -521,14 +521,18 @@ def iterate_items(
521521
:param max_unmappable: Skip at most this many unmappable items; if
522522
more are encountered, stop iterating. `None` to never stop.
523523
:param map_missing: Indicates what to do with mapped items for which
524-
some fields could not be mapped. Defaults to 'empty_str'. Must be one of:
524+
some fields could not be mapped. Defaults to 'default'. Must be one of:
525525
- 'default': fill missing fields with the default passed by map_item
526+
- 'keep': leave the MissingMappedField sentinel in place so the caller
527+
can tell which fields were missing (useful for JSON serialisation
528+
via MissingMappedFieldEncoder)
526529
- 'abort': raise a MappedItemIncompleteException if a field is missing
527530
- a callback: replace missing field with the return value of the
528531
callback. The MappedItem object is passed to the callback as the
529532
first argument and the name of the missing field as the second.
530533
- a dictionary with a key for each possible missing field: replace missing
531-
field with a strategy for that field ('default', 'abort', or a callback)
534+
field with a strategy for that field ('default', 'keep', 'abort', or
535+
a callback)
532536
:param get_annotations: Whether to also fetch annotations from the database.
533537
This can be disabled to help speed up iteration.
534538
:param offset: After how many rows we should yield items.
@@ -612,6 +616,10 @@ def iterate_items(
612616
mapped_item.data[missing_field] = strategy(
613617
mapped_item.data, missing_field
614618
)
619+
elif strategy == "keep":
620+
# leave the MissingMappedField in place so the
621+
# caller can distinguish missing from present
622+
continue
615623
elif strategy == "abort":
616624
# raise an exception to be handled at the processor level
617625
raise MappedItemIncompleteException(
@@ -624,7 +632,7 @@ def iterate_items(
624632
].value
625633
else:
626634
raise ValueError(
627-
"map_missing must be 'abort', 'default', or a callback."
635+
"map_missing must be 'abort', 'default', 'keep', or a callback."
628636
)
629637
else:
630638
mapped_item = original_item

‎common/lib/llm/adapter.py‎

Lines changed: 29 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,8 @@
1919
class LLMAdapter:
2020
def __init__(
2121
self,
22-
config,
2322
model: dict,
23+
server: dict = None,
2424
api_key: Optional[str] = None,
2525
temperature: float = 0.1,
2626
max_tokens: int = 1000,
@@ -29,17 +29,16 @@ def __init__(
2929
"""
3030
Instantiate an adapter to interface with an LLM model
3131
32-
:param config: 4CAT config reader
33-
:param dict model: Model metadata (as in `llm.available_models` 4CAT setting)
32+
:param dict model: Model metadata (as in `llm.available_models` 4CAT
33+
setting)
34+
:param dict server: Server metadata (as in `llm.servers` 4CAT setting)
3435
:param str api_key: API key, if needed
3536
:param float temperature: Temperature hyperparameter
3637
:param int max_tokens: Max tokens to generate
3738
:param dict client_kwargs: Optional parameters for the LLM adapter class
3839
"""
39-
known_servers = config.get("llm.servers", {})
40-
4140
self.model = model
42-
self.server = known_servers.get(model["server"])
41+
self.server = server
4342
self.api_key = api_key
4443
self.temperature = temperature
4544
self.structured_output = False
@@ -281,11 +280,24 @@ def _format_media_block(
281280
}}
282281
return {"type": "image_url", "image_url": {"url": data_uri}}
283282

284-
def set_structure(self, json_schema):
283+
def set_structure(self, json_schema, method=None, include_raw=False, strict=None):
285284
"""
286-
Set desired response JSON schema
287-
288-
:param json_schema:
285+
Bind a JSON schema so the model returns schema-validated structured output.
286+
287+
:param json_schema: JSON schema dict (or JSON string) describing the output.
288+
:param method: How structured output is enforced. None uses LangChain's
289+
per-provider default (usually "function_calling", which binds a tool).
290+
For reasoning models served over an OpenAI-compatible proxy, pass
291+
"json_schema" — constrained decoding forces the answer channel itself
292+
to match the schema, rather than relying on a clean tool call that the
293+
model may emit in the wrong channel (yielding empty, unparseable output).
294+
:param include_raw: When True, structured-output calls return a
295+
{"raw", "parsed", "parsing_error"} dict instead of raising on a parse
296+
failure, so callers can inspect the raw AIMessage (finish_reason,
297+
reasoning channel, token usage) to diagnose what went wrong.
298+
:param strict: Passed through to with_structured_output when not None.
299+
Use strict=False for schemas that don't satisfy OpenAI strict-mode
300+
requirements but are fine for a guided-decoding backend (e.g. vLLM).
289301
"""
290302
if not json_schema:
291303
raise ValueError("json_schema is None")
@@ -296,9 +308,14 @@ def set_structure(self, json_schema):
296308
json.dumps(json_schema) # To validate / raise an error
297309

298310
# LM Studio needs some more guidance
299-
if self.model["wrapper"] == "lmstudio":
311+
if self.model["wrapper"] == "openai-like":
300312
json_schema = {"type": "json_schema", "json_schema": {"schema": json_schema}}
301313
self.llm = self.llm.bind(response_format=json_schema)
302314
else:
303-
self.llm = self.llm.with_structured_output(json_schema)
315+
kwargs = {"include_raw": include_raw}
316+
if method:
317+
kwargs["method"] = method
318+
if strict is not None:
319+
kwargs["strict"] = strict
320+
self.llm = self.llm.with_structured_output(json_schema, **kwargs)
304321
self.structured_output = True

0 commit comments

Comments
 (0)