[Feature] Unified lazy dataset loaders (Samanantar, IndicCorp, Wiki, Common Voice, etc.)

[soumendrak]

What

Unified, lazy loaders for popular Odia datasets, modeled on the HuggingFace datasets API but lightweight.

from openodia import datasets

ds = datasets.load("samanantar", split="train")          # English-Odia parallel
ds = datasets.load("indiccorp-or", split="train")        # Monolingual Odia
ds = datasets.load("naamapadam-or", split="train")       # NER
ds = datasets.load("common-voice-or", split="train")     # ASR audio
ds = datasets.load("wiki-or", split="train")             # Wikipedia dump

for row in ds:    # streaming — no full download/decompress
    print(row["text"])

Why

• Each Odia ML practitioner re-implements dataset download, decompression, and parsing.
• Datasets are scattered across AI4Bharat, HuggingFace, Common Voice, IndicNLP, individual mirrors.
• The roadmap calls for English-Odia Parallel corpus and Odia Monolingual corpus — this issue makes the package a one-stop loader.
• Lazy streaming avoids the install-size and disk-footprint problem of bundling data.

Impact

User

• Five lines of code to start training/finetuning on any major Odia dataset.
• Consistent return shape across datasets simplifies downstream code.

Developer

• Compositional: feed ds straight into [[ngram-utils]], [[stopwords-custom]], [[sentence-embeddings]].
• Pairs with [[ner-tagger]] (which needs Naamapadam).

Before / After

flowchart LR
    subgraph Before
      U1[Find dataset URL] --> U2[wget tar.gz]
      U2 --> U3[Decompress]
      U3 --> U4[Hand-parse format]
      U4 --> U5[Project-specific code]
    end
    subgraph After
      D2[datasets.load 'samanantar'] --> S2[Streaming iterator]
      S2 --> U6[Train/inference]
    end

Loading

Comparison

Loader	Bundled?	Streaming?	Audio?	Install size
HF datasets	Yes (heavy)	Yes	Yes	~500MB transitive
Custom per-script	No	No	Per-author	0
openodia.datasets	No (downloads on demand)	Yes	Yes	Tiny

Detailed explanation

Architecture:

1. A registry (openodia/datasets/_registry.py) of dataset descriptors:

   DatasetSpec(
     name="samanantar",
     splits={"train": ("https://.../odia-train.tar.gz", "sha256:...")},
     format="parquet",  # or "jsonl" / "tsv" / "audio"
     schema={"or": str, "en": str},
   )

2. A loader that:
   • Downloads to a cache dir (~/.cache/openodia/datasets/<name>),
   • Verifies SHA-256,
   • Returns a lazy iterator producing typed rows (dataclass or dict).
3. No mandatory deps — parquet/audio support gated behind [datasets] extra.
4. datasets.list() to enumerate registered datasets and their licenses.

Each row is a dataclass for type safety:

@dataclass
class ParallelRow:
    src: str   # English
    tgt: str   # Odia
    src_lang: str = "en"
    tgt_lang: str = "or"

Proposed solutions

Option A — Lightweight in-house loader (recommended)

• ~300 lines + a per-dataset adapter.
• Pros: tiny install, full control.
• Cons: must maintain dataset URLs and schemas.

Option B — Wrap HF datasets

• datasets.load_dataset("ai4bharat/samanantar", "or").
• Pros: zero work; community-maintained schemas.
• Cons: huge transitive dep (pyarrow, requests, multiprocessing, etc.) — kills the "tools that work" lightweight ethos.

Option C — Offer both

• Pure-Python by default; [hf] extra to delegate to datasets.
• Pros: best of both.
• Cons: dual code paths to maintain.

Recommended: A as v1 with 3–4 datasets, then C.

Acceptance criteria

• datasets.list() enumerates ≥ 4 datasets with name + license.
• datasets.load("<name>") streams rows without OOM on a 10GB dataset.
• Cache reuse: second call skips download.
• SHA-256 verification fails loudly on tampered files.

Backward compatibility

Pure addition. Datasets are downloaded on demand — no impact on install size.

Labels: enhancement, help wanted