AI-CARE

Carbon-Aware Reporting Evaluation Tool for AI Models

This repository provides the implementation of AI-CARE, a research framework for measuring predictive performance together with energy use, carbon impact, and billing-oriented efficiency metrics for AI systems.

The repository accompanies the paper:

• 📄 AI-CARE: Carbon-Aware Reporting Evaluation Metric for AI Models
• 🔗 https://arxiv.org/abs/2602.16042
• 📥 https://arxiv.org/pdf/2602.16042

AI-CARE treats energy and carbon as first-class evaluation quantities and now includes an LLM billing pipeline for studying token-based versus energy-based pricing under a reproducible experiment design.

---

Overview

AI-CARE contains two complementary workflows:

1. Core AI-CARE benchmark workflow

The original benchmark pipeline evaluates machine learning models under fixed experimental conditions, measures training and inference energy consumption, converts energy usage to carbon emissions, and reports performance–energy–carbon tradeoffs.

2. LLM billing experiment workflow

The LLM extension adds a research-ready pipeline for evaluating the mismatch between token-based billing and energy-based billing. The workflow includes:

• A fixed 30-prompt suite with three prompt categories:
  • 10 short factual QA prompts
  • 10 long summarization prompts
  • 10 reasoning / multi-step prompts
• A fixed experiment runner that evaluates four logical model profiles against a real NRP-hosted OpenAI-compatible LLM backend:
  • local-qa-llm-small
  • local-qa-llm-medium
  • local-qa-llm-large
  • alt-llm
• Cost and energy analysis modules that compute:
  • token usage
  • latency
  • approximate energy consumption
  • token-based cost
  • energy-based cost
  • efficiency ratios
• Statistical analysis for:
  • token–energy correlation
  • energy variance across token buckets
  • rank shifts between billing regimes
• Publication-quality plotting utilities for paper figures.

All LLM scripts preserve the same experiment matrix, logging, and CSV outputs, while now issuing real inference requests to the configured NRP endpoint.

---

Models

The broader repository includes representative model families such as:

• Multilayer Perceptron (MLP)
• Convolutional Neural Network (CNN)
• Transformer-based classifier
• MLP-Mixer
• MobileNetV2
• ResNet-18 (ImageNet-100 only)

The LLM billing pipeline now maps each logical experiment profile to a distinct NRP-hosted backend model while preserving the existing billing, evaluation, and reporting structure.

---

Datasets and Prompt Suite

Vision datasets used in AI-CARE

Experiments in the core benchmark are conducted on vision benchmarks of increasing complexity:

• MNIST
• Fashion-MNIST
• CIFAR-10
• CIFAR-100
• ImageNet-100

LLM prompt suite

The LLM billing pipeline uses experiments/llm/prompts.json, which contains a fixed set of 30 prompts split evenly across:

• Short factual QA
• Long summarization
• Reasoning / multi-step

This design supports controlled comparison of how prompt type affects tokens, latency, energy, and billing outcomes.

---

Project Structure

The most relevant directories for the LLM billing workflow are:

experiments/llm/   Prompt suite plus reusable execution pipeline modules
billing/           Cost analysis, table generation, statistical analysis, and plotting
results/           Generated CSV outputs, including results/llm/
paper/figures/     Publication-ready figures generated by the billing pipeline
paper/tables/      Automatically generated LaTeX tables for the paper workflow

Additional repository directories support the broader AI-CARE benchmark, paper artifacts, tables, model definitions, scripts, and utilities.

---

How to Run the LLM Billing Pipeline

Install dependencies first:

pip install -r requirements.txt

One-command paper asset pipeline (recommended)

Run the full publication pipeline (experiments + tables + plots + export zips):

make paper-assets

This command executes, in order:

1. python experiments/llm/run_llm_experiments.py
2. python billing/llm_cost_analysis.py
3. python billing/llm_stat_analysis.py
4. python billing/generate_tables.py
5. python billing/llm_plots.py
6. python billing/generate_plots.py
7. python billing/package_paper_assets.py

Exported zip bundles are written to:

• paper/exports/tables.zip
• paper/exports/figures.zip

Then run the LLM billing pipeline in the following order:

Step 1: Run the LLM experiments

python experiments/llm/run_llm_experiments.py

This executes the fixed prompt suite across four logical model profiles backed by multiple NRP-hosted models and writes per-run measurements to results/llm/llm_results.csv.

Step 2: Compute token and energy billing outputs

python billing/llm_cost_analysis.py

This reads the experiment results and computes token-based cost, energy-based cost, and efficiency metrics.

Step 3: Compute quantitative statistical analysis

python billing/llm_stat_analysis.py

This generates the correlation, token-bucket variance, and rank-shift outputs.

Step 4: Generate paper-ready LaTeX tables

python billing/generate_tables.py

This writes LaTeX tables to paper/tables/.

Step 5: Generate publication-quality plots

python billing/llm_plots.py
python billing/generate_plots.py

This saves the final paper figures to paper/figures/. All plotting scripts now export TikZ/PGFPlots figure sources to plots_tikz/ for direct LaTeX inclusion via \input{plots_tikz/<figure>.tex}. For LaTeX preamble support, include: \\usepackage{pgfplots} and \\pgfplotsset{compat=newest}.

Full pipeline command

make paper-assets

---

Output Files

1. Raw LLM experiment results

• results/llm/llm_results.csv

This file stores one row per model–prompt run and includes fields such as:

• model
• prompt_id
• prompt_type
• tokens_used
• latency_seconds
• approximate_energy_kwh
• carbon_emissions_gco2
• energy_backend (gpu_telemetry or cpu_fallback)
• accuracy
• response_model
• energy_per_token
• latency_per_token
• accuracy_per_energy

2. Cost analysis outputs

• results/llm/llm_cost_analysis.csv

This file adds billing-oriented metrics, including:

• token_cost
• energy_cost
• cost_gap
• carbon_per_token_gco2
• accuracy_per_carbon
• accuracy_per_token_cost
• accuracy_per_energy_cost

3. Statistical results

• results/llm/correlation.csv
• results/llm/energy_variance_by_token_bucket.csv
• results/llm/rank_shift.csv
• results/llm/regression_summary.csv
• results/llm/model_summary_with_ci.csv
• results/llm/prompt_summary_with_ci.csv
• results/llm/efficiency_summary.csv
• results/llm/scas_ranking.csv
• results/llm/key_findings.txt

These outputs provide quantitative evidence for:

• correlation between token usage and energy use
• variance in energy within comparable token ranges
• ranking differences under token-based vs energy-based billing
• ranking differences under token-cost vs energy-cost vs SCAS criteria

4. Generated tables

Tables are written to paper/tables/, including:

• llm_summary_table.tex
• llm_cost_table.tex
• llm_efficiency_table.tex
• llm_carbon_table.tex

5. Generated plots

TikZ/PGFPlots figure sources are written to plots_tikz/, including:

• llm_token_vs_energy_scatter.tex
• llm_pareto_frontier.tex
• llm_cost_mismatch_bar.tex
• llm_efficiency_per_cost.tex
• llm_rank_shift_comparison.tex
• llm_energy_variance_by_token_bucket.tex
• llm_prompt_type_comparison.tex
• energy_vs_tokens.tex
• carbon_vs_tokens.tex
• energy_by_model.tex
• carbon_by_model.tex
• accuracy_vs_energy.tex
• accuracy_vs_carbon.tex

These figures are intended to be suitable for manuscript and presentation use.

---

Key Metrics

The LLM billing workflow emphasizes the following metrics:

Token cost

token_cost = (tokens_used / 1000) * price_per_1k_tokens

Energy cost

energy_cost = approximate_energy_kwh * ALPHA

Efficiency ratios

• accuracy_per_token_cost
• accuracy_per_energy_cost

These metrics make it possible to compare whether billing methods agree on which models are most efficient and whether token counts accurately reflect underlying energy expenditure.

---

Research Contribution

The LLM billing extension contributes a framework for studying a central systems question in modern generative AI evaluation:

Do token-based prices faithfully reflect the real energy footprint of model inference?

By combining controlled prompts, real API-backed inference, per-run token and energy logging, billing calculations, variance analysis, correlation analysis, and model rank-shift comparisons, this repository enables journal-style investigation of token vs energy mismatch in LLM evaluation.

---

Using Real LLMs via NRP

Experiments now use real NRP-hosted LLMs through the NRP OpenAI-compatible API. The experiment matrix, CSV schema, energy estimation logic, and downstream billing/statistical analysis remain unchanged, which makes it easy to compare historical runs with newer live-inference runs.

Setup

Export your API key before running the experiment pipeline:

export OPENAI_API_KEY=your_api_key_here

The runner connects to the NRP OpenAI-compatible endpoint:

https://ellm.nrp-nautilus.io/v1

NRP model mapping

The LLM billing workflow now uses multiple real NRP models rather than a single shared backend:

• qwen3-small: lightweight backend for the local-qa-llm-small profile
• gpt-oss: default medium-profile backend for local-qa-llm-medium
• qwen3: higher-capacity backend for the local-qa-llm-large profile
• minimax-m2: alternate backend used for the alt-llm profile

The experiment runner also records the provider-reported response.model value in the raw LLM results CSV so you can confirm which backend actually served each request.

All logical experiment model names are preserved, but they now map to distinct NRP backend models so the original experiment structure stays intact while real model diversity is introduced:

• local-qa-llm-small -> qwen3-small
• local-qa-llm-medium -> gpt-oss
• local-qa-llm-large -> qwen3
• alt-llm -> minimax-m2

Example usage

from llm_client import NRPClient

client = NRPClient()
response = client.generate("Summarize the benefits of carbon-aware AI reporting.")
print(response["text"])
print(response["usage"])

Notes on reproducibility

Using a real hosted LLM improves realism for billing and latency measurements, but exact generations can vary across runs because the backend service is now live. The fixed prompt suite, preserved logical model profiles, and unchanged analysis pipeline still provide a consistent experiment structure for controlled comparisons.

High-Performance Experiment Pipeline

The LLM workflow now includes a high-performance execution path tailored for real NRP-hosted models:

• bounded parallel execution via a reusable pipeline module (1-5 workers, default 3) for rate-safe throughput
• per-model batching that reuses a single NRP client instance across prompt runs
• progress logging that reports model name, prompt type, run index, and overall completion percentage
• optional FAST_MODE (or --fast-mode) for debugging, which keeps two prompts per prompt type and shortens prompt text to reduce turnaround time
• reusable CLI subset selection via --models, --prompt-types, and --prompt-limit to support future dataset or prompt-suite extensions

Reproducibility notes

Parallel execution improves throughput, but live hosted-model behavior still means generations and latency can vary across runs. The fixed logical model profiles, prompt suite structure, CSV outputs, and downstream billing/statistics scripts remain stable for controlled comparisons.

Reproducibility

The LLM pipeline is designed for structured repeatability:

• fixed prompt suite
• preserved logical model profiles
• automatic directory creation for outputs
• repository-local billing/statistics/plotting pipeline
• live inference via the NRP OpenAI-compatible API

The broader AI-CARE benchmark likewise uses fixed reporting assumptions to make energy and carbon comparisons transparent and repeatable.

---

License

This project is licensed under the Apache License 2.0. See LICENSE for details.

Patent Notice

This project may be subject to patent protection. Certain components described in the associated publication are pending patent review.

---

Citation

If you use this work, please cite:

@misc{santosh2026aicarecarbonawarereportingevaluation,
  title={AI-CARE: Carbon-Aware Reporting Evaluation Metric for AI Models},
  author={KC Santosh and Srikanth Baride and Rodrigue Rizk},
  year={2026},
  eprint={2602.16042},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2602.16042}
}