AGENTS.md

AGENTS.md

This file provides guidance to AI coding assistants working on TulipaEnergyModel.jl. It applies to any coding agent (for example, Copilot, Claude, or similar tools).

For full developer documentation, see docs/src/90-contributing/91-developer.md.

Architecture

Julia package for modeling and optimization of electric energy systems. Uses DuckDB for data handling, JuMP for optimization modeling, and HiGHS as the default solver. Part of the Tulipa ecosystem (TulipaIO, TulipaBuilder, TulipaClustering).

Source Structure

• src/TulipaEnergyModel.jl — Main module; all using statements live here
• src/structures.jl — Core types: EnergyProblem, TulipaVariable, TulipaConstraint, TulipaExpression
• src/run-scenario.jl — High-level run_scenario entry point
• src/create-model.jl — create_model! / create_model
• src/solve-model.jl — solve_model! / solve_model / save_solution!
• src/objective.jl — Objective function construction
• src/model-preparation.jl — Data massage before model creation
• src/data-preparation.jl — populate_with_defaults!
• src/data-validation.jl — Input validation
• src/io.jl — create_internal_tables! / export_solution_to_csv_files
• src/input-schemas.jl + src/input-schemas.json — Table schema definitions
• src/solver-parameters.jl — Solver parameter handling
• src/utils.jl — Utility functions
• src/variables/ (7 files) — Variable creation (flows, investments, storage, etc.)
• src/constraints/ (16 files) — Constraint creation (capacity, energy, transport, etc.)
• src/expressions/ (3 files) — Expression creation (storage, intersection, multi-year)
• src/rolling-horizon/ (4 files) — Rolling horizon implementation
• src/sql/ (3 SQL files) — SQL templates for creating tables

Dynamic includes: The main module uses a loop to include all .jl files from variables/, constraints/, and expressions/ directories. New files added to these directories are automatically included.

Pipeline Flow

All-in-one: run_scenario(connection)

High-level (using EnergyProblem struct):

1. EnergyProblem(connection) — creates internal tables, computes variable/constraint indices
2. create_model!(energy_problem) — builds the JuMP model
3. solve_model!(energy_problem) — solves the optimization
4. save_solution!(energy_problem) — stores results back in DuckDB

Low-level (without EnergyProblem, for finer control):

1. create_internal_tables!(connection) — data preparation
2. compute_variables_indices(connection) / compute_constraints_indices(connection) — index computation
3. prepare_profiles_structure(connection) — profile setup
4. create_model(connection, variables, constraints, profiles) — builds model
5. solve_model(model) — solves the optimization
6. save_solution!(connection, model, variables, constraints) — stores results

See test/test-pipeline.jl for examples of both levels.

Performance Requirements

Apply these guidelines with judgement. Not every function is performance-critical. Focus optimization efforts on hot paths and frequently called code. If necessary, check the Julia Performance Tips. To investigate performance issues, check the "Investigating performance issues" section in docs/src/90-contributing/91-developer.md.

Anti-Patterns to Avoid

Type instability

Functions must return consistent concrete types. Check with @code_warntype.

• Bad: f(x) = x > 0 ? 1 : 1.0
• Good: f(x) = x > 0 ? 1.0 : 1.0

Abstract field types

Struct fields must have concrete types or be parameterized.

• Bad: struct Foo; data::AbstractVector; end
• Good: struct Foo{T<:AbstractVector}; data::T; end

Untyped containers

• Bad: Vector{Any}(), Vector{Real}()
• Good: Vector{Float64}(), Vector{Int}()

Non-const globals

• Bad: THRESHOLD = 0.5
• Good: const THRESHOLD = 0.5

Unnecessary allocations

• Use views instead of copies (@view, @views)
• Pre-allocate arrays instead of push! in loops
• Use in-place operations (functions ending with !)

Captured variables

When creating anonymous functions inside a local scope, don't use variables from that local scope.

Splatting penalty

Avoid splatting (...) in performance-critical code.

Abstract return types

Avoid returning Union types or abstract types.

Best Practices

• Use @inbounds when bounds are verified. Be sure when doing this because it might cause crashes.
• Use broadcasting (dot syntax) for element-wise operations
• Avoid try-catch in hot paths
• Use function barriers to isolate type instability

Code Conventions

Linting and formatting: Use pre-commit (which includes formatting checks) before committing.

CRITICAL: This list should be kept in sync with docs/src/90-contributing/91-developer.md.

Lightweight rules (see developer docs for full details):

• Naming: CamelCase for classes/modules, snake_case for functions/variables, kebab-case for file names and doc reference tags
• Imports: prefer using Package: A, B, C; avoid bare using Package; centralize using declarations in src/TulipaEnergyModel.jl
• Returns: explicitly state what a function returns; use explicit return
• Constructors: use function foo() not foo() = ...
• Globals: UPPER_CASE for constants
• Exports: define exports in the main module file (src/TulipaEnergyModel.jl)
• Comments: complete sentences, prefer why over how
• DuckDB / SQL: use src/sql/*.sql for large reusable query batches and inline SQL for short local queries; prefer CREATE OR REPLACE TEMP TABLE for intermediates reused across steps and CTEs for single-query decomposition; prefer set-based operations and bulk write patterns (DuckDB.Appender, DuckDB.register_table, UPDATE ... FROM); join on full keys, use explicit column lists and ARRAY_AGG(... ORDER BY ...), add ORDER BY only when row order matters, and prefer parameter binding for scalar values when possible
• Markdown docs: tables must satisfy MD060 column alignment

Documentation Practices and Requirements

Framework: Diataxis

Docstring requirements:

• Scope: all elements of public interface
• Include: function signatures and arguments list
• Automation: DocStringExtensions.TYPEDSIGNATURES (TYPEDFIELDS used sparingly)
• See also: add links for functions with same name (multiple dispatch)

Design Principles

• Elegance and concision in both interface and implementation
• Fail fast with actionable error messages rather than hiding problems
• Validate invariants explicitly in subtle cases
• Avoid over-adherence to backwards compatibility for internal helpers

Contribution Workflow

Branch naming: feature/description or fix/description

1. Create feature branch
2. Follow style guide and run formatter
3. Follow the style guide and run pre-commit when committing
4. Ensure tests pass
5. Submit pull request

Commit message style: Imperative, present tense subject line with an informative title (Add X, Fix Y, Remove Z), optionally prefixed with a bracket tag ([Fix], [docs], [Validation]). No trailing period. Add a body when the change needs more context. Put issue references (Closes #N, Fixes #N) in the body, not the subject. Avoid conventional commits style (feat:, fix:).

CRITICAL: When making commits, always add a co-authored line with the tool name, the agent model, and the relevant e-mail. For instance "Co-Authored-By: Claude Code (claude-sonnet-4-6) noreply@anthropic.com"

Data Schema Migrations

When a change requires modifying CSV data files (adding/removing columns, transforming values), create a Python + DuckDB migration script rather than editing files by hand. Save it in utils/scripts/. See migrate-investment-groups.py and migrate-group-asset-limits.py for examples. Run with:

uv run --with duckdb python utils/scripts/your-migration.py

Scripts must update all data folders: test/inputs/, benchmark/EU/, and docs/src/10-tutorials/my-awesome-energy-system/ (subfolders for each step).

When asked to do a data migration: make sure you are writing a script, make sure you have enough information or ask the user for more details, and suggest that the user reads docs/src/90-contributing/91-developer.md under "Using an AI Agent to Write the Script" for tips.

Rebase recovery: if data files conflict during a rebase, reset them with git restore and rerun the script.

Development Commands

CRITICAL: Always try Julia MCP first. It can also be used to run general Julia commands. Always pass the environment.

CRITICAL: Use the testing filters to avoid running too many tests at once.

Run tests (Julia MCP): Use julia_eval with env_path = "/full/path/to/test/" and @run_package_tests — see Julia MCP below. Run all tests (testitem CLI fallback): julia --project=test test/runtests.jl Docs: julia --project=docs -e "using LiveServer; servedocs()" Diff CSV files: git diff --word-diff-regex="[^[:space:],]+" <path> — highlights changed field values instead of whole lines.

Testing Strategy

Uses TestItemRunner.jl with @testitem, @testsnippet, @testmodule — not standard @testset. Test inputs are in test/inputs/.

Never run the full test suite unless you are explicitly asked to. Instead, run only the tests you created or modified.

Julia MCP

If julia_eval (from julia-mcp) is available, always use it to run Julia code and tests. It keeps a warm session alive between calls, avoiding Julia's startup and recompilation costs. Never use include("runtests.jl") or Pkg.test() through julia_eval — use @run_package_tests as shown below.

If julia_eval is not available, suggest installing julia-mcp. If it is installed but not working, try reconnecting (/mcp in Claude Code) or restarting the session (julia_restart). Only fall back to CLI commands (see below) if the user cannot or does not want to use julia-mcp.

Use env_path with the absolute path to the test/ directory (e.g., /full/path/to/TulipaEnergyModel.jl/test/) — relative paths resolve from the MCP server's directory, not the project root. The test env has its own Project.toml with the package as a path source. Always import TestItemRunner first, then call @run_package_tests — runtests.jl reads ARGS and won't work in a REPL:

using TestItemRunner

# Filter by file (most common)
@run_package_tests verbose=true filter = ti -> contains(ti.filename, "test-model")

# Filter by tags (AND logic)
@run_package_tests verbose=true filter = ti -> all(tag in ti.tags for tag in [:unit, :fast])

# Exclude slow tests
@run_package_tests verbose=true filter = ti -> !any(tag in ti.tags for tag in [:slow])

ti fields: .filename, .name, .tags — same semantics as the CLI flags below.

testitem CLI (fallback)

The file runtests.jl includes a CLI for running the test items directly.

CRITICAL: Use julia --project=<env> when running Julia code. NEVER use bare julia or julia --project without specifying the environment.

Run all tests: julia --project=test test/runtests.jl Run specific file: julia --project=test test/runtests.jl --file test-model Run fast tests only: julia --project=test test/runtests.jl --tags fast --exclude slow List available tags: julia --project=test test/runtests.jl --list-tags

Shared Setup (in test/utils.jl)

• @testsnippet CommonSetup — imports all standard libraries, defines INPUT_FOLDER, fixture helpers (_tiny_fixture, _storage_fixture, _multi_year_ fixture)
• @testmodule TestData — provides TestData.simplest_data dict for minimal test data

Available Tags (from TAGS_DATA in test/runtests.jl)

• Test types: :unit, :integration, :validation
• Complexity: :fast, :slow
• Feature areas: :case_study, :data_validation, :data_preparation, :io, :pipeline

CLI Filter Flags

Flag	Description
--tags t1,t2	Run tests that have ALL listed tags
--exclude t1,t2	Skip tests that have ANY of the listed tags
--file substr	Run tests from files whose path contains substr
--name substr	Run tests whose name contains substr
--pattern text	Run tests whose name OR filename contains text
--list-tags	Print all available tags and exit
--verbose	Show individual test results
--help	Show full usage help and exit

Multiple filters combine with AND logic. Use --list-tags to discover available tags when this file is stale.

Common workflows:

# Quick iteration — only fast tests
julia --project=test test/runtests.jl --tags fast --exclude slow

# Focus on a specific feature area
julia --project=test test/runtests.jl --file test-pipeline

# Run a single test by name
julia --project=test test/runtests.jl --name "My test description"

# CI-style: all integration tests, verbose
julia --project=test test/runtests.jl --tags integration --verbose

Writing New Tests

@testitem "Description" setup = [CommonSetup] tags = [:unit, :fast] begin
    @test result == expected
end

Use @testsnippet for per-test setup that should run fresh for each test item:

@testsnippet MySetup begin
    # Runs once per @testitem that lists it in setup=[...]
    data = Dict("key" => 1)
end

@testitem "Uses fresh data" setup = [MySetup] tags = [:unit, :fast] begin
    data["key"] = 99  # safe — each test gets its own copy
    @test data["key"] == 99
end

Use the auxiliary function _create_table_for_tests in test/utils.jl to create temporary tables in DuckDB for testing.

Use @testmodule for expensive one-time setup shared across many tests (loaded once, accessed via module name):

@testmodule SharedData begin
    # Runs once for the whole test suite
    const connection = open_connection("test/inputs/tiny")
end

@testitem "Reads shared connection" setup = [SharedData] tags = [:integration, :fast] begin
    @test SharedData.connection !== nothing
end

Prefer @testsnippet when tests mutate the data; prefer @testmodule when setup is slow and data is read-only.

New tags must be added to TAGS_DATA in test/runtests.jl. Target: 100% test coverage.

Adding a Data Validator

1. Write _validate_<description>!(error_messages, connection) in src/data-validation.jl. Push error strings into error_messages directly; return error_messages at the end.
2. Register it in the tuple in validate_data! with a log message and fail_fast = false (use true only if later validators would crash without this one passing).
3. Add tests in test/test-data-validation.jl tagged [:unit, :data_validation, :fast]. Call the validator as TEM._validate_foo!(String[], connection) and assert on the result.

MPS Regression Testing

MPS files in benchmark/model-mps-folder/ serve as regression tests for the optimization model. The CompareMPS.yml workflow runs automatically on PRs.

• If your change intentionally modifies the model, update MPS files: julia --project=. utils/scripts/model-mps-update.jl
• To check without updating: julia --project=. utils/scripts/model-mps-compare.jl
• If MPS comparison fails unexpectedly, investigate using the compare script log

Troubleshooting

Type instability in the code

• Symptom: Poor performance, many allocations
• Diagnosis: @code_warntype on suspect function
• Solution: See performance anti-patterns above

Pre-commit or formatter fails

• Symptom: pre-commit run -a returns one or more failures
• Solution: Apply the reported fixes, including formatter rules from .JuliaFormatter.toml, and run pre-commit run -a again

Test failures

• Symptom: Tests fail unexpectedly
• Solution: julia --project=test -e 'using Pkg; Pkg.instantiate()'