Bot account request: import Portuguese books from Biblioteca Nacional de Portugal (CC0)

[pagyn-app]

Hi @mekarpeles,

I'd like to request bot privileges (API usergroup access) for a new
account to import Portuguese-language books into Open Library.

Bot account: [pagyn_bot] (https://openlibrary.org/people/pagyn_bot)

Source: Biblioteca Nacional de Portugal (BNP) open data —
"Bibliografia Nacional Portuguesa" dataset, covering monographs
published in Portugal since 2002, ISBN included. License: CC0
(public domain). Source: https://opendata.bnportugal.gov.pt/downloads.htm

Why: Open Library's coverage of Portuguese-language books is
thin. This is a national library catalog, freely licensed, that
directly addresses recent/missing books that users are searching for.

Method: I'll transform the CC0 dataset into the Open Library
import schema (title, authors, publishers, publish_date,
source_records, isbn, languages: por) and submit via the batch
import endpoint (/import/batch/new), following the process described
in the Writing Bots docs. Source files and script will be committed
to openlibrary-bots per the guidelines, and I will not run bulk
modifications until the script has been reviewed.

Happy to start with a small sample batch first for review before
any larger import. Thanks!

[tfmorris]

I think it's great that you're willing to do this, but, in my opinion, OpenLibrary should own importing catalog records from national libraries. They are an excellent source of high quality, diverse metadata which have been created by professional librarians. This would help counteract the dramatic reduction of data quality caused by importing bookseller data, especially from BWB and 3rd party Amazon used book sellers.

[hornc]

@pagyn-app Thanks for the message and pointing the the BNP data. As @tfmorris points out, this open MARC data is a perfect candidate for the existing bulk-MARC ingestion process.

I've created a ol_data item for the raw MARC records and am in the process of converting them to the exact MARC format the current OL importer expects, it's a little more fiddly than I first expected since BNP is using UNIMARC, but it's progressing.

I can post an example import to confirm we are getting all the exisiting data import process that we'd expect.

[openlibrary-bot]

Thank you @pagyn-app for submitting this issue!

Issue Assessment

Open questions

• @hornc's comment (2026-07-31) indicates OL is already converting this same BNP dataset for the existing bulk-MARC ingestion pipeline, and @tfmorris's comment argues OL should own national-library imports directly rather than granting external bot accounts. Given that, is a separate pagyn_bot API-usergroup grant still needed, or does this request get superseded by the internal import once it lands?
• If the bot account is still wanted (e.g., as a stopgap or for a different subset of the data), what is the intended long-term ownership of the pagyn_bot account and its import script — will it be retired once the internal bulk-MARC path covers BNP, or run in parallel?
• Has a decision been made on /import/batch/new vs bulk-MARC-binary ingestion for this specific source, or is that still open pending @hornc's UNIMARC conversion work?

Action items

• Staff decision needed: confirm with @hornc whether the in-progress bulk-MARC conversion of BNP data makes the bot-account request unnecessary, or whether both approaches should proceed.

---

Context (research findings)

• Open Library has three existing import entry points that this proposal would need to go through one of: POST /api/import, POST /api/import/ia, and POST /import/batch/new (the endpoint mentioned in the issue body) — see docs/ai/imports/index.md.
• The bulk-MARC path (?bulk_marc=true) that @hornc references as already in progress is documented as "the primary path for IA scanning partners... and the main source of usable records" — docs/ai/imports/api.md#L34-L36.
• OL's MARC parsers (MarcBinary — marc_binary.py#L80, MarcXml — marc_xml.py#L65, and read_edition() — parse.py#L672) are built around MARC21 semantics. BNP publishes UNIMARC, a related but distinct standard — this is consistent with @hornc's comment that the conversion is "more fiddly than I first expected."
• If this proceeds as a community-run source adapter rather than an internal import, the documented pattern is a sources/<slug>/ adapter in the openlibrary-bots repo (record + provider classes) — docs/ai/imports/adding-sources.md#L5-L34.
• Related open "Bot Account Application" issues currently awaiting the same kind of decision: Bot Account Application: pillarsdotnetBot339 #13191 (pillarsdotnetBot339), Bot Account Application: AgenticCommonsBot #12887 (AgenticCommonsBot), and Bot Account Application: extend AgenticCommonsBot scope to remote_ids.wikidata backfill #13136 (AgenticCommonsBot scope extension) — all labeled Lead: @mekarpeles + Needs: Response, same category of request.
• Prior precedent for a lighter-weight community batch-import path: Community Batch Import Request: arabooks #9726 ("Community Batch Import Request: arabooks"), labeled Type: Question + Good First Issue.
• No duplicate or prior attempt at this specific BNP/Portuguese-language import was found in issue or PR search.

---

🔔 Stakeholders

@hornc — already active in this thread and the codebase owner for MARC/imports; flagging so the staff-decision question above reaches them directly alongside the Lead.

---

Full triage checklist

• Triage — correct labels applied (Affects: Data, Needs: Staff Decision added; Module: Import, Priority: 2, Lead: @mekarpeles already correctly set by staff)
• Problem / opportunity — clear: thin Portuguese-language coverage, audience is PT-language readers/searchers, source is a freely-licensed national library catalog
• Scope — well-scoped to a single dataset/source and a specific access grant
• Justification — reasoning given (CC0 national library data, addresses a known coverage gap); impact not quantified but reasonable for this issue type
• Success & testing criteria — author offers a "small sample batch first for review" but no explicit review/acceptance criteria are defined
• Proposal & breakdown — concrete: schema mapping (title/authors/publishers/publish_date/source_records/isbn/languages), target endpoint, and repo/review process are all specified
• If bug: environment & repro steps — not applicable, this is not a bug report
• Risks, concerns, open questions — surfaced by both prior commenters (data-quality ownership, national-library data governance) and by this comment (overlap with in-progress internal work)
• Richy context — codebase and related-issue research done; no duplicate found, but overlapping in-progress work identified
• References — source dataset URL and process description provided by author

---

Note

This comment was automatically generated by Richy, Open Library's issue enrichment assistant, on behalf of @mekarpeles. This is NOT a sign-off. This issue needs lead review before work begins.