Skip to content

feat(cli): add gitt miner languages to rank file types by scoring weight and method - #1551

Closed
glorydavid03023 wants to merge 2 commits into
entrius:testfrom
glorydavid03023:feat/miner-languages-command
Closed

feat(cli): add gitt miner languages to rank file types by scoring weight and method#1551
glorydavid03023 wants to merge 2 commits into
entrius:testfrom
glorydavid03023:feat/miner-languages-command

Conversation

@glorydavid03023

Copy link
Copy Markdown

Summary

Language weight is only half of what decides a changed file's worth — the other half is how it is scored. Tree-sitter languages are scored on semantic AST tokens (high ceiling), plain extensions score per added line, and documentation/config extensions are additionally capped at MAX_LINES_SCORED_FOR_NON_CODE_EXT. Today a miner has no way to see this split, so it isn't obvious that (e.g.) editing Rust is worth far more per change than editing a capped Markdown file.

This adds gitt miner languages — a read-only command that loads the same programming_languages.json the validator scores against and prints extensions ranked by weight, each tagged with how it is scored and its tree-sitter language:

  • Scoring method per extension: token (tree-sitter AST), line-count, or line-count-capped (doc/config).
  • --code-only shows only the token-scored extensions (where AST/token score actually applies).
  • --top N, --search <substr> (case-insensitive), and --json-output, matching the existing gitt miner conventions.
  • Footer/JSON summary: counts by method and the non-code line cap. No network call or PAT required.

Example:

$ gitt miner languages --code-only --top 8
     Scoring Weights by File Extension
             (tree-sitter only)
┌───┬─────┬────────┬─────────┬─────────────┐
│ # │ Ext │ Weight │ Scoring │ Tree-sitter │
├───┼─────┼────────┼─────────┼─────────────┤
│ 1 │ c   │  2.000 │  token  │ c           │
│ 2 │ cc  │  2.000 │  token  │ cpp         │
│ 3 │ cpp │  2.000 │  token  │ cpp         │
│ … │ …   │  …     │  …      │ …           │
└───┴─────┴────────┴─────────┴─────────────┘

8 of 135 extensions shown | 101 token, 11 line-count, 23 capped doc/config

The scoring-method classification reuses the canonical NON_CODE_EXTENSIONS / MAX_LINES_SCORED_FOR_NON_CODE_EXT constants (the same ones tree_sitter_scoring uses) so it cannot drift from the scorer. Display/aggregation logic is split into small pure helpers (_scoring_method, _filter_languages, _rank_languages, _summarize_languages, _build_language_rows) that are independently unit-tested.

Related Issues

None — net-new miner UX. Complements the repository-weight view in a separate PR; this one is about file-type weight and scoring method.

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Documentation
  • Other (describe below)

Testing

  • Tests added/updated — tests/cli/test_miner_languages.py (method-classification unit tests incl. the non-code-overrides-tree-sitter edge case, plus end-to-end command tests: table, JSON, --code-only, --top, no-match, and empty-weights error path)
  • Manually tested — ran against the real programming_languages.json (135 extensions; 101 token / 11 line-count / 23 capped); verified ranking, method tags, and all flags.

Checklist

  • Code follows project style guidelines (ruff check + ruff format clean)
  • Self-review completed
  • Changes are documented (command help text + miner group docstring updated)

…eight and method

Language weight is only half of what decides a changed file's worth — the other
half is *how* it is scored. Tree-sitter languages are scored on semantic AST
tokens (high ceiling), plain extensions score per added line, and doc/config
extensions are additionally capped. Miners had no way to see this split, so it
was unclear that, say, editing Rust is worth far more per change than editing a
capped Markdown file.

Add `gitt miner languages`, which loads the same programming_languages.json the
validator scores against and prints extensions ranked by weight, each tagged
with its scoring method (token / line-count / capped) and tree-sitter language.
Supports `--code-only` (token-scored only), `--top N`, `--search`, and
`--json-output`. No network or PAT required.

The scoring-method classification reuses the canonical NON_CODE_EXTENSIONS /
MAX_LINES_SCORED_FOR_NON_CODE_EXT constants so it cannot drift from the scorer.
Pure helpers are unit-tested and the command is covered end-to-end (table, JSON,
--code-only, --top, no-match, empty-file error path) in
tests/cli/test_miner_languages.py.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@xiao-xiao-mao xiao-xiao-mao Bot added the feature Net-new functionality label Jun 26, 2026
@anderdc anderdc closed this Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature Net-new functionality

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants