CI: Lemonade v10.7.0 breaks embedding-model jobs (nomic-embed-text-v2-moe / llama-server ≥ b6524)

[kovtcharov]

Summary

The Lemonade Server bump to v10.7.0 (#1571) broke every CI job that loads the embedding model. llama-server in v10.7.0 (build ≥ b6524) cannot load nomic-embed-text-v2-moe-GGUF, so all RAG/embedding CI jobs fail at server startup — before any test runs. This is a known upstream regression: lemonade-sdk/lemonade#612.

This is not flaky — it is deterministic and reproduces on every branch. The tell: jobs that load an LLM model (API Tests, Chat, Code, Unit Tests) pass; only jobs that load the embedding model fail.

Error

model_load_error: Failed to load model 'nomic-embed-text-v2-moe-GGUF': llama-server failed to start

(higher-level form in some jobs: [ERROR] Server health check failed after 60 seconds / Server failed to start)

Affected workflows / jobs

Job	Workflow
RAG Integration Tests	.github/workflows/test_rag.yml
Test Lemonade Embeddings API	.github/workflows/test_embeddings.yml
Lemonade Server Smoke Test (stx)	.github/workflows/test_lemonade_server.yml
Example Agents Integration Tests (stx)	.github/workflows/test_examples.yml (intermittent)

Evidence it is environment-wide

Test RAG / Test Lemonade Embeddings fail across multiple unrelated branches in the same window — e.g. claudia/task-8fa7ecef (#1455), feat/npu-flm-embedder, autofix/issue-1745, autofix/issue-1743. Onset (~2026-06-18) coincides with the v10.7.0 bump landing on main.

Root cause

LEMONADE_VERSION (in src/gaia/version.py) was bumped to v10.7.0 in #1571. v10.7.0 ships llama-server ≥ b6524, which per upstream lemonade-sdk/lemonade#612 does not work with nomic-embed-text-v2-moe. LLM GGUFs are unaffected, which is why only the embedding path breaks.

Fix

• In progress: fix(ci): pin Lemonade to 10.2.0 — v10.7.0 backend can't load nomic embedding model #1788 pins Lemonade back to 10.6.0 (rolls llama-server below the b6524 boundary).
• Revert condition: un-pin once upstream Known Issue: llama-server >= b6524 doesnt work with nomic-embed-text-v2-moe.Q8_0.gguf lemonade-sdk/lemonade#612 is resolved with a llama-server build that loads nomic-embed-text-v2-moe.
• Diagnostics follow-up: the RAG/embeddings/smoke workflows currently swallow the llama-server child stderr and only surface "Server failed to start" — they should print the child exit code/stderr so the next backend regression is debuggable from the CI log alone.

Related upstream issues

• Known Issue: llama-server >= b6524 doesnt work with nomic-embed-text-v2-moe.Q8_0.gguf lemonade-sdk/lemonade#612 — root cause (llama-server ≥ b6524 vs nomic-embed-text-v2-moe)
• Embedding model load failure lemonade-sdk/lemonade#941 — embedding model load failure
• Ollama-Compatible API endpoint /api/embed stopped to work with Open WebUI with lemonade version upper 10.0.0 lemonade-sdk/lemonade#1586 — /api/embed regressions after lemonade > 10.0.0

[github-actions]

I confirmed the diagnosis and checked the fix PR. Two things worth flagging for accuracy:

The pin target differs from what this issue says. The issue describes #1788 as pinning to 10.6.0, but the actual PR pins to 10.2.0 (the last-known-good that loaded the embedder and passed all six tests on the 06‑17 main run). Worth aligning the issue text with the PR so the revert-tracking is unambiguous.

The "diagnostics follow-up" is already inside #1788, not a separate task. The PR makes the embedding/RAG/smoke workflows print the llama-server child log on load failure, and it fixes a deeper root cause this issue doesn't mention: a version pin alone wasn't enough because the CI reconcile only swapped the MSI server exe and left the stale llama.cpp backend binary in place — so a downgraded server kept spawning the bad backend. That's the real reason the bump was hard to undo.

LEMONADE_VERSION on main is still 10.7.0 (version.py:12) since #1788 hasn't merged — so CI embedding jobs are still red until it lands.

Everything else in your root-cause is spot on: LLM GGUFs are unaffected, only the embedding path breaks, and the revert condition (un-pin once upstream lemonade#612 ships a working llama-server) is the right gate.

🔍 Technical details

• src/gaia/version.py:12 → LEMONADE_VERSION = "10.7.0" still on main; fix(ci): pin Lemonade to 10.2.0 — v10.7.0 backend can't load nomic embedding model #1788 (fix/lemonade-pin-10.2.0-embed-regression, OPEN) changes it to 10.2.0, not 10.6.0.
• PR fix(ci): pin Lemonade to 10.2.0 — v10.7.0 backend can't load nomic embedding model #1788 body: pins to 10.2.0, adds a path-agnostic backend wipe (searches all cache/install roots, logs each llama-server binary + version, removes only llamacpp backend dirs), and adds src/gaia/version.py to the embedding/RAG/lemonade-server workflow path filters so a future bump actually triggers the embedding tests.
• The diagnostics ask in this issue ("print child exit code/stderr") is implemented in that same PR.
• Upstream gate unchanged: Known Issue: llama-server >= b6524 doesnt work with nomic-embed-text-v2-moe.Q8_0.gguf lemonade-sdk/lemonade#612 (llama-server ≥ b6524 vs nomic-embed-text-v2-moe).

Next step: sync the issue's "10.6.0" → "10.2.0" to match #1788, then this becomes a clean "merge #1788, revert when upstream #612 resolves" tracker.