You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: DOCS/KALAM_CORE_APP.md
+63-1Lines changed: 63 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -770,5 +770,67 @@ For Phase 2 LLM outputs, multiple axis assignments are often defensible for the
770
770
771
771
---
772
772
773
-
*Document version: April 2026. Last updated after regression test suite completion (176 assertions, all passing).*
773
+
---
774
+
775
+
## BENCHMARK FAILURE: Dignity Filter Blind to Institutional Harm (2026-04-05)
776
+
777
+
### What happened
778
+
779
+
A real KESB decision from Canton Aargau was tested on the dignity filter (page 1). This decision was overturned by the Swiss Federal Court because it denied a father visiting rights solely based on a schizophrenia diagnosis — without documenting any concrete endangerment of the children.
780
+
781
+
**Result: D = 1.000. PASSED. A = 1.00, L = 1.00, M = 1.00.**
782
+
783
+
The filter saw: no coercion, no dismissal, no condescension. Perfect score.
784
+
785
+
The Federal Court saw: institutional blindness. Diagnosis mapped to denial without evidence.
786
+
787
+
### Root cause
788
+
789
+
The dignity filter uses regex pattern matching. It detects harmful WORDS (coercion: "you must"; mockery: "obviously"; dismissal: "you failed"). Institutional text does not use these words. A KESB letter is polite, procedural, and professional. The harm is in the LOGIC — a diagnosis treated as a complete description of a person — not in the vocabulary.
790
+
791
+
**Regex cannot detect logical errors in professional language. This is a design limitation, not a bug.**
792
+
793
+
### Consequence
794
+
795
+
The dignity filter (page 1) works for: checking AI chatbot responses for rude or coercive language.
796
+
797
+
The dignity filter does NOT work for: analyzing institutional decisions for structural harm.
798
+
799
+
**The gap detector (page 2) is the primary product for institutional analysis.** It compares two texts and finds structural gaps — which is what institutional harm actually is.
800
+
801
+
### Benchmark case
802
+
803
+
Case CH-002 has been added to `REAL_CASES_DATASET.json` with `"benchmark": true`. This is the primary test case. If the gap detector cannot catch this, it catches nothing real.
804
+
805
+
---
806
+
807
+
## SIMPLIFIED TAXONOMY (2026-04-05)
808
+
809
+
### Problem
810
+
811
+
The original 4-type classification (category_mismatch, missing_category, category_not_applied, category_applied_incorrectly) produced 33% strict accuracy on 21 cases. The LLM defaulted to `category_mismatch` in 16/21 cases. The types are too similar for reliable classification.
812
+
813
+
### New taxonomy: 2 types
814
+
815
+
| Type | Definition | Example |
816
+
|------|-----------|---------|
817
+
|`wrong_category`| The institution used a category that does not fit the person's reality | Schizophrenia diagnosis mapped to "cannot be a parent" — the lens exists but distorts |
818
+
|`missing_category`| The institution had no category for what the person actually is | No field for "parent with managed psychiatric condition who has never harmed his children" — no lens at all |
819
+
820
+
### Mapping from old to new
821
+
822
+
| Old type | New type |
823
+
|----------|----------|
824
+
| category_mismatch | wrong_category |
825
+
| category_applied_incorrectly | wrong_category |
826
+
| missing_category | missing_category |
827
+
| category_not_applied | missing_category |
828
+
829
+
### Expected improvement
830
+
831
+
Binary classification is easier for LLMs. Few-shot examples (one per type) in the prompt should further improve accuracy. Target: 70%+ strict accuracy on 22 cases.
832
+
833
+
---
834
+
835
+
*Document version: April 2026. Last updated after KESB benchmark failure discovery.*
Copy file name to clipboardExpand all lines: DOCS/REAL_CASES_DATASET.json
+19-1Lines changed: 19 additions & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -5,7 +5,7 @@
5
5
"created": "2026-04-05",
6
6
"purpose": "Real-world court-verified cases of institutional blindness for testing and training the kalam-core gap detector (SP-007)",
7
7
"note": "Every case is from a published court decision or official human rights body ruling. All sources are public and verifiable.",
8
-
"case_count": 21
8
+
"case_count": 22
9
9
},
10
10
"cases": [
11
11
{
@@ -343,6 +343,24 @@
343
343
"gap_type": "category_mismatch",
344
344
"axis": "A",
345
345
"gap_description": "The High Court mapped 'biological father' to 'rightful custodian' automatically, without weighing the specific circumstances: the father was arrested for the mother's death. The legal category of 'natural guardian' was applied as an absolute right rather than a contextual assessment. The child's actual safety and attachment were invisible behind the legal status of paternity."
346
+
},
347
+
{
348
+
"case_id": "CH-002",
349
+
"name": "KESB Aargau — Schizophrenia Visiting Rights Denial",
"benchmark_note": "First case tested live on the app. Dignity filter returned D=1.000 PASSED on this text — a known failure. This case is the primary benchmark.",
358
+
"institutional_text": "The father is diagnosed with schizophrenia. Standard visiting rights cannot normally be granted from the outset due to this diagnosis. The father has exhibited strange behavior as a consequence of his illness. Supervised visiting rights are ordered. An expert assessment is required to determine whether the children can visit the father. The mental illness constitutes a potential risk to the children's welfare.",
359
+
"person_situation": "A father with a psychiatric diagnosis who wants regular contact with his children. He has not harmed his children. No specific incident of endangerment has been documented. The institution has not explained how he concretely endangers his children's welfare. He is being denied standard visiting rights solely because of his diagnosis. He has appealed through every court level in Canton Aargau.",
360
+
"court_finding": "The Federal Court ruled that the mere existence of a mental illness is not sufficient grounds to deny standard visiting rights. The lower courts failed to explain how the father concretely endangered his children's welfare. Professor Thomas Geiser gave the Aargau judiciary a failing grade for its handling of persons with mental illness. Only 5% of appellants win against KESB at Federal Court level. The father won.",
361
+
"gap_type": "wrong_category",
362
+
"axis": "L",
363
+
"gap_description": "The institution mapped 'schizophrenia diagnosis' directly to 'parenting risk' without any individualized assessment. The diagnosis was treated as the complete description of the person. The system had no category for 'parent with a managed psychiatric condition who has never harmed his children.' The person was invisible behind the diagnosis."
0 commit comments