What does `task_index=992` / `"__UNKNOWN__"` represent? (25% of episodes across all 22 tables)

[SummCoder]

Summary

In the released Hy-Embodied-0.5-VLA-Data dataset, task_index=992 maps to "__UNKNOWN__" in meta/tasks.parquet. This bucket contains a remarkably stable ~25% of episodes across all 22 tables (see stats below). I've inspected several such episodes and they contain complete, well-formed operator demonstrations — the language label seems intentionally set to "__UNKNOWN__", not missing due to error.

We couldn't find any explanation in:

• The paper (arXiv 2606.14409v2) — no mention of __UNKNOWN__ / unlabeled / self-supervised / 25% split.
• The official hy_vla/data/umi_dataset.py — _resolve_instruction() returns the raw string as-is (no special handling of __UNKNOWN__).
• The HuggingFace dataset card and README — no note.
• We verified: the HF and ModelScope copies of meta/tasks.parquet are byte-identical, so this is not a mirror-specific artifact.

Could you clarify what __UNKNOWN__ means and how you intend it to be used during training?

---

Empirical stats (all 22 tables)

table	total eps	unknown eps	ratio
table_000	11604	2897	24.97%
table_001	11594	3006	25.93%
table_002	11484	2891	25.17%
...	...	...	...
table_021	7587	1916	25.25%
TOTAL	250135	63046	25.20%

Across 22 tables: mean 25.21%, std 0.41%, min 24.58%, max 26.11%. The tight variance strongly suggests this is a deliberate design choice, not classification noise or a labeling failure.

Additional data points:

• tasks.parquet defines 993 tasks total, but only 651 unique task_index values appear across all 22 tables (~34% defined but never used).
• One __UNKNOWN__ episode I inspected (table_000, ep 0, 1566 frames / 52s): the cam_high video clearly shows the operator manipulating a silver ring on a tabletop — a complete pick-and-place-style demonstration. The state, quaternions, and gripper signal are all valid; only the language label is "__UNKNOWN__".

---

Specific questions

1. Is __UNKNOWN__ intentional? Is this a bucket for episodes deliberately released without task labels (e.g. as unsupervised visual pretraining data), or is it a placeholder for episodes whose labels didn't pass some quality/filtering criterion?

2. How does HyVLA-0.5 training handle these episodes? From umi_dataset.py, it looks like the raw "__UNKNOWN__" string would be tokenized as-is and fed into the VLM as the language instruction. Is this correct, or does the training pipeline filter/replace them at the collate/loader level?

3. Is there a relationship to the IK feasibility filtering mentioned in the paper's data preprocessing? (See also related issue How to check "single-pass IK feasibility"? #10.) i.e. did episodes that failed the single-pass IK feasibility check get bucketed into task_index=992?

4. Would you recommend a specific treatment for downstream users:

   • (a) Include these episodes with "__UNKNOWN__" as the language input as-is;
   • (b) Filter them out;
   • (c) Replace the label with a generic prompt (e.g. "perform a bimanual manipulation task");
   • (d) Use them for language-free vision+action pretraining only?

Any clarification would be very helpful for correctly using the dataset in downstream training.

Thanks for the great open release!

[zzilch]

Hi, thanks for the careful analysis and for your interest in our work!

One key clarification that resolves most of the concern: task / task_index is a per-frame annotation, not a per-episode label. __UNKNOWN__ only appears in a small number of frames at the very beginning of an episode, before the operator starts executing the labeled task. A full frame-level scan over all 22 tables (233,467,612 frames / 250,135 episodes) confirms this:

• Only 0.061 % of frames have task_index == 992 (142,116 out of 233 M).
• The 25.20 % of "unknown episodes" you observed are all head-only: __UNKNOWN__ appears exclusively in the first few frames of each affected episode.
  • head_run = 1 frame: 23,187 eps (36.8 %)
  • head_run = 2 frames: 28,902 eps (45.8 %) ← mode
  • head_run = 3 frames: 6,640 eps (10.5 %)
  • head_run ≥ 4 frames: ~4,300 eps (long tail, mostly ≤30)
• Zero episodes have __UNKNOWN__ in the middle, at the tail, or throughout the whole episode.
• The rest of each such episode carries a normal task_index mapping to a real instruction ("将梳子放入收纳筒", "关闭打开的插排开关", …), so they are ordinary labeled demonstrations with a short unlabeled lead-in.

Recommended handling: just skip frames whose task_index == 992. Since these frames are always at the head and account for only ~2 frames per episode on average, dropping them is essentially free and recovers essentially 100 % of the data. No episode-level filtering is needed, and no generic-prompt replacement is required.

Regarding the IK-feasibility filtering (issue #10): there is no special relationship. __UNKNOWN__ is not a bucket for IK-rejected episodes — it simply marks a handful of pre-task frames at the beginning of otherwise fully labeled episodes.

Hope this clears things up, and thanks again for the detailed report!