ai-benchmarking

Benchmarking the gap between AI agent hype and architecture. Three agent archetypes, 73-point performance spread, stress testing, network resilience, and ensemble coordination analysis with statistical validation.

FinMTM: A Multi-Turn Multimodal Benchmark for Financial Reasoning and Agent Evaluation

Eval framework. Define correct, test against it, get results.

LLM Benchmark

AI agent evaluation framework for multi-participant coordination tasks. Built with LangGraph, custom MCP tools, and LLM-as-a-Judge evaluation. MSc dissertation project (University of Edinburgh, 2025).

HiddenBench: Benchmark for evaluating collective reasoning in multi-agent LLM systems

ORBIT - Operational Research and Benchmarking Interface for Transformers

An evidence-hostile, container-isolated benchmark and behavior analysis platform for long-horizon AI software agents.

🛡️ Artemis: A high-fidelity OpenEnv benchmark for evaluating AI agents in realistic SOC triage and cybersecurity incident response scenarios.

Open-source, local-first evaluation infrastructure for applied AI systems, built for developer and agent workflows.

Uncertainty & Confidence Management (UCM): A healthcare AI benchmark suite for uncertainty recognition, justification boundaries, confidence calibration, proportionate action, and reassessment.

A comprehensive benchmarking platform for CPT, ICD-10, and HCPCS coding questions. Identifies the most reliable models for healthcare applications. Evaluates multiple AI models on medical coding expertise through iterative consensus-building.

Comprehensive multi-IDE AI model benchmarking framework supporting Cursor, Windsurf, VSCode, and other IDEs with automated testing and performance comparison capabilities

A benchmarking framework for designing and evaluating multi-agent AI systems. Implements structured task decomposition (Map-Reduce/Fan-out), containerized execution, and deterministic verification for complex AI orchestration

Which programming language is best for AI coding agents? Benchmarking 13 languages with Claude Code.

Implementation of SofIA project safety protocol

High-fidelity LLM evaluation framework focusing on expert-level domain knowledge and reasoning verification.

Standalone open-source verifier for MBX v2 — AiBenchLab's tamper-evident benchmark export format. Three dependencies, zero network access, reproduces the SHA-256 content hash to confirm an .mbx.json file hasn't been altered since export.

🔬 Research Project: An automated framework to generate, configure, and evaluate multi-agent AI crews for financial modeling using a Meta-Agent pipeline. This study evaluates the performance of dynamically synthesized MAS (Multi-Agent Systems) against manual expert-defined benchmarks in financial risk contexts.