llm-evaluation-toolkit

Here are 27 public repositories matching this topic...

PacificAI / langtest

Deliver safe & effective language models

nlp artificial-intelligence benchmarks benchmark-framework model-assessment ai-safety mlops responsible-ai ml-safety trustworthy-ai ethics-in-ai ml-testing large-language-models llm ai-testing llm-test llm-evaluation-toolkit llm-as-evaluator llm-testing

Updated Aug 13, 2026
Python

athina-ai / athina-evals

Star

Python SDK for running evaluations on LLM generated responses

evaluation evaluation-metrics evaluation-framework llmops llm-eval llm-ops llm-evaluation llm-evaluation-toolkit

Updated Jun 6, 2025
Python

Re-Align / just-eval

Star

A simple GPT-based evaluation tool for multi-aspect, interpretable assessment of LLMs.

evaluation gpt4 llm llm-eval llm-evaluation llm-evaluation-toolkit

Updated Jan 29, 2024
Python

parea-ai / parea-sdk-py

Star

Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

metrics good-first-issue llm prompt-engineering generative-ai llmops llm-eval llm-tools llm-evaluation llm-evaluation-toolkit llms-benchmarking llm-evaluation-framework

Updated Feb 13, 2025
Python

An easy python package to run quick basic QA evaluations. This package includes standardized QA evaluation metrics and semantic evaluation metrics: Black-box and Open-Source large language model prompting and evaluation, exact match, F1 Score, PEDANT semantic match, transformer match. Our package also supports prompting OPENAI and Anthropic API.

qa-automation-test rl-training llm exact-matching llm-evaluation llm-evaluation-toolkit llm-evaluation-framework reward-modeling

Updated Jul 18, 2025
Python

zhuohaoyu / KIEval

Star

[ACL'24] A Knowledge-grounded Interactive Evaluation Framework for Large Language Models

machine-learning explainable-ai llm llm-evaluation llm-evaluation-toolkit llm-evaluation-framework llm-evaluation-metrics acl2024

Updated Jul 19, 2024
Python

CodeEval-Pro / CodeEval-Pro

Star

[ACL'25 Findings] Official repo for "HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation Task"

code-generation llm llm-evaluation llm-evaluation-toolkit llm4code llm-reasoning

Updated Apr 7, 2025
Python

ARTPARK-SAHAI-ORG / calibrate

Star

Core engine behind Calibrate, a framework for evaluating AI agents: speech-to-text, text-to-speech, LLM evaluation, end-to-end simulations

Updated Aug 12, 2026
JavaScript

Evaliphy / evaliphy

Star

The E2E AI testing tool | No ML Overhead

ai test-automation testing-tools end-to-end-testing test-automation-framework rag ai-testing llm-evaluation rag-evaluation llm-evaluation-toolkit llm-evaluation-framework rag-pipeline llm-testing ai-testing-tool ai-test-automation

Updated May 7, 2026
TypeScript

scalexi / scalexi

Star

scalexi is a versatile open-source Python library, optimized for Python 3.11+, focuses on facilitating low-code development and fine-tuning of diverse Large Language Models (LLMs).

openai dataset-generation llms llm-evaluation-toolkit

Updated Feb 25, 2025
Python

Agenta-AI / job_extractor_template

Star

Template for an AI application that extracts the job information from a job description using openAI functions and langchain

template example extraction extract-information openai extract-data unstructured-text llm langchain llmops openai-function-example llm-evaluation llm-evaluation-toolkit

Updated Dec 21, 2023
Python

ronniross / confidence-scorer

Sponsor

Star

Measure of estimated confidence for non-hallucinative nature of outputs generated by Transformer-based Language Models.

dataset datasets llm llms llm-training llm-evaluation llms-reasoning llm-evaluation-toolkit llms-benchmarking llm-evaluation-framework llm-evaluation-metrics llms-efficency llms-evalution

Updated Feb 26, 2026
Python

nhsengland / evalsense

Star

Tools for systematic large language model evaluations

evaluation-metrics evaluation-framework llm llms llm-evaluation llm-evaluation-toolkit llm-evaluation-framework llm-evaluation-metrics llm-benchmarking

Updated Apr 12, 2026
Python

parea-ai / parea-sdk-ts

Star

TypeScript SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

llm prompt-engineering llms llm-eval llm-tools llm-evaluation llm-evaluation-toolkit llms-benchmarking llm-evaluation-framework

Updated Jan 17, 2025
TypeScript

dileepkpandiya / skilleval

Star

Objectively test Claude SKILL.md files with blind A/B testing and a randomized judge. Catch regressions before they ship. CI-ready.

open-source typescript claude cli-tool ai-testing anthropic claude-ai llm-evaluation llm-evaluation-toolkit

Updated Jul 13, 2026
TypeScript

ARTPARK-SAHAI-ORG / calibrate-frontend

Star

Frontend for Calibrate, a framework for evaluating AI agents: speech-to-text, text-to-speech, LLM evaluation, end-to-end simulations

Updated Aug 13, 2026
TypeScript

JDevShivamGarg / Aegis-LLM-Evaluator---Observability-Framework

Star

Aegis is a self-hosted evaluation and observability framework designed to continuously test prompt quality, track regressions, and gate CI/CD pipelines, completely local and air-gapped.

llm llm-evaluation llm-evaluation-toolkit llm-evaluation-framework llm-observability llm-observability-dashboard