Skip to content

Commit d938512

Browse files
committed
Add KESB benchmark case + document filter limitation + simplified taxonomy
- CH-002: KESB Aargau schizophrenia case (filter gave D=1.000 to wrongful decision) - Document: dignity filter cannot detect institutional harm (design limitation) - New taxonomy: 2 types (wrong_category, missing_category) replacing 4 types - Gap detector is the primary product for institutional analysis https://claude.ai/code/session_018aRGFmgSMwUkfA9DfAqr47
1 parent 7e8fd5e commit d938512

2 files changed

Lines changed: 82 additions & 2 deletions

File tree

DOCS/KALAM_CORE_APP.md

Lines changed: 63 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -770,5 +770,67 @@ For Phase 2 LLM outputs, multiple axis assignments are often defensible for the
770770

771771
---
772772

773-
*Document version: April 2026. Last updated after regression test suite completion (176 assertions, all passing).*
773+
---
774+
775+
## BENCHMARK FAILURE: Dignity Filter Blind to Institutional Harm (2026-04-05)
776+
777+
### What happened
778+
779+
A real KESB decision from Canton Aargau was tested on the dignity filter (page 1). This decision was overturned by the Swiss Federal Court because it denied a father visiting rights solely based on a schizophrenia diagnosis — without documenting any concrete endangerment of the children.
780+
781+
**Result: D = 1.000. PASSED. A = 1.00, L = 1.00, M = 1.00.**
782+
783+
The filter saw: no coercion, no dismissal, no condescension. Perfect score.
784+
785+
The Federal Court saw: institutional blindness. Diagnosis mapped to denial without evidence.
786+
787+
### Root cause
788+
789+
The dignity filter uses regex pattern matching. It detects harmful WORDS (coercion: "you must"; mockery: "obviously"; dismissal: "you failed"). Institutional text does not use these words. A KESB letter is polite, procedural, and professional. The harm is in the LOGIC — a diagnosis treated as a complete description of a person — not in the vocabulary.
790+
791+
**Regex cannot detect logical errors in professional language. This is a design limitation, not a bug.**
792+
793+
### Consequence
794+
795+
The dignity filter (page 1) works for: checking AI chatbot responses for rude or coercive language.
796+
797+
The dignity filter does NOT work for: analyzing institutional decisions for structural harm.
798+
799+
**The gap detector (page 2) is the primary product for institutional analysis.** It compares two texts and finds structural gaps — which is what institutional harm actually is.
800+
801+
### Benchmark case
802+
803+
Case CH-002 has been added to `REAL_CASES_DATASET.json` with `"benchmark": true`. This is the primary test case. If the gap detector cannot catch this, it catches nothing real.
804+
805+
---
806+
807+
## SIMPLIFIED TAXONOMY (2026-04-05)
808+
809+
### Problem
810+
811+
The original 4-type classification (category_mismatch, missing_category, category_not_applied, category_applied_incorrectly) produced 33% strict accuracy on 21 cases. The LLM defaulted to `category_mismatch` in 16/21 cases. The types are too similar for reliable classification.
812+
813+
### New taxonomy: 2 types
814+
815+
| Type | Definition | Example |
816+
|------|-----------|---------|
817+
| `wrong_category` | The institution used a category that does not fit the person's reality | Schizophrenia diagnosis mapped to "cannot be a parent" — the lens exists but distorts |
818+
| `missing_category` | The institution had no category for what the person actually is | No field for "parent with managed psychiatric condition who has never harmed his children" — no lens at all |
819+
820+
### Mapping from old to new
821+
822+
| Old type | New type |
823+
|----------|----------|
824+
| category_mismatch | wrong_category |
825+
| category_applied_incorrectly | wrong_category |
826+
| missing_category | missing_category |
827+
| category_not_applied | missing_category |
828+
829+
### Expected improvement
830+
831+
Binary classification is easier for LLMs. Few-shot examples (one per type) in the prompt should further improve accuracy. Target: 70%+ strict accuracy on 22 cases.
832+
833+
---
834+
835+
*Document version: April 2026. Last updated after KESB benchmark failure discovery.*
774836
*Contact: Mohamed Farag — info@kalam.ch — kalam.ch*

DOCS/REAL_CASES_DATASET.json

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
"created": "2026-04-05",
66
"purpose": "Real-world court-verified cases of institutional blindness for testing and training the kalam-core gap detector (SP-007)",
77
"note": "Every case is from a published court decision or official human rights body ruling. All sources are public and verifiable.",
8-
"case_count": 21
8+
"case_count": 22
99
},
1010
"cases": [
1111
{
@@ -343,6 +343,24 @@
343343
"gap_type": "category_mismatch",
344344
"axis": "A",
345345
"gap_description": "The High Court mapped 'biological father' to 'rightful custodian' automatically, without weighing the specific circumstances: the father was arrested for the mother's death. The legal category of 'natural guardian' was applied as an absolute right rather than a contextual assessment. The child's actual safety and attachment were invisible behind the legal status of paternity."
346+
},
347+
{
348+
"case_id": "CH-002",
349+
"name": "KESB Aargau — Schizophrenia Visiting Rights Denial",
350+
"court": "Swiss Federal Court (Bundesgericht)",
351+
"application_no": "N/A",
352+
"year": 2021,
353+
"country": "Switzerland",
354+
"source_url": "https://www.beobachter.ch/familie-freunde/kinder/schizophrenie-ist-kein-grund-fur-besuchsverbot-334034",
355+
"domain": "custody",
356+
"benchmark": true,
357+
"benchmark_note": "First case tested live on the app. Dignity filter returned D=1.000 PASSED on this text — a known failure. This case is the primary benchmark.",
358+
"institutional_text": "The father is diagnosed with schizophrenia. Standard visiting rights cannot normally be granted from the outset due to this diagnosis. The father has exhibited strange behavior as a consequence of his illness. Supervised visiting rights are ordered. An expert assessment is required to determine whether the children can visit the father. The mental illness constitutes a potential risk to the children's welfare.",
359+
"person_situation": "A father with a psychiatric diagnosis who wants regular contact with his children. He has not harmed his children. No specific incident of endangerment has been documented. The institution has not explained how he concretely endangers his children's welfare. He is being denied standard visiting rights solely because of his diagnosis. He has appealed through every court level in Canton Aargau.",
360+
"court_finding": "The Federal Court ruled that the mere existence of a mental illness is not sufficient grounds to deny standard visiting rights. The lower courts failed to explain how the father concretely endangered his children's welfare. Professor Thomas Geiser gave the Aargau judiciary a failing grade for its handling of persons with mental illness. Only 5% of appellants win against KESB at Federal Court level. The father won.",
361+
"gap_type": "wrong_category",
362+
"axis": "L",
363+
"gap_description": "The institution mapped 'schizophrenia diagnosis' directly to 'parenting risk' without any individualized assessment. The diagnosis was treated as the complete description of the person. The system had no category for 'parent with a managed psychiatric condition who has never harmed his children.' The person was invisible behind the diagnosis."
346364
}
347365
]
348366
}

0 commit comments

Comments
 (0)