feat(cli): add gitt miner languages to rank file types by scoring weight and method - #1551
Closed
glorydavid03023 wants to merge 2 commits into
Closed
feat(cli): add gitt miner languages to rank file types by scoring weight and method#1551glorydavid03023 wants to merge 2 commits into
gitt miner languages to rank file types by scoring weight and method#1551glorydavid03023 wants to merge 2 commits into
Conversation
…eight and method Language weight is only half of what decides a changed file's worth — the other half is *how* it is scored. Tree-sitter languages are scored on semantic AST tokens (high ceiling), plain extensions score per added line, and doc/config extensions are additionally capped. Miners had no way to see this split, so it was unclear that, say, editing Rust is worth far more per change than editing a capped Markdown file. Add `gitt miner languages`, which loads the same programming_languages.json the validator scores against and prints extensions ranked by weight, each tagged with its scoring method (token / line-count / capped) and tree-sitter language. Supports `--code-only` (token-scored only), `--top N`, `--search`, and `--json-output`. No network or PAT required. The scoring-method classification reuses the canonical NON_CODE_EXTENSIONS / MAX_LINES_SCORED_FOR_NON_CODE_EXT constants so it cannot drift from the scorer. Pure helpers are unit-tested and the command is covered end-to-end (table, JSON, --code-only, --top, no-match, empty-file error path) in tests/cli/test_miner_languages.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Language weight is only half of what decides a changed file's worth — the other half is how it is scored. Tree-sitter languages are scored on semantic AST tokens (high ceiling), plain extensions score per added line, and documentation/config extensions are additionally capped at
MAX_LINES_SCORED_FOR_NON_CODE_EXT. Today a miner has no way to see this split, so it isn't obvious that (e.g.) editing Rust is worth far more per change than editing a capped Markdown file.This adds
gitt miner languages— a read-only command that loads the sameprogramming_languages.jsonthe validator scores against and prints extensions ranked by weight, each tagged with how it is scored and its tree-sitter language:--code-onlyshows only the token-scored extensions (where AST/token score actually applies).--top N,--search <substr>(case-insensitive), and--json-output, matching the existinggitt minerconventions.Example:
The scoring-method classification reuses the canonical
NON_CODE_EXTENSIONS/MAX_LINES_SCORED_FOR_NON_CODE_EXTconstants (the same onestree_sitter_scoringuses) so it cannot drift from the scorer. Display/aggregation logic is split into small pure helpers (_scoring_method,_filter_languages,_rank_languages,_summarize_languages,_build_language_rows) that are independently unit-tested.Related Issues
None — net-new miner UX. Complements the repository-weight view in a separate PR; this one is about file-type weight and scoring method.
Type of Change
Testing
tests/cli/test_miner_languages.py(method-classification unit tests incl. the non-code-overrides-tree-sitter edge case, plus end-to-end command tests: table, JSON,--code-only,--top, no-match, and empty-weights error path)programming_languages.json(135 extensions; 101 token / 11 line-count / 23 capped); verified ranking, method tags, and all flags.Checklist
ruff check+ruff formatclean)