From 5f55f68667aecc9772be3ab15e0394ef8291f08f Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Wed, 17 Jun 2026 09:41:35 +0000 Subject: [PATCH] docs: update performance benchmarks (3.13.0) --- README.md | 50 +- generated/benchmarks/BUILD-BENCHMARKS.md | 2078 ++++++++++++++++- .../benchmarks/INCREMENTAL-BENCHMARKS.md | 130 +- generated/benchmarks/QUERY-BENCHMARKS.md | 80 +- 4 files changed, 2260 insertions(+), 78 deletions(-) diff --git a/README.md b/README.md index 2494a128..0bfd700a 100644 --- a/README.md +++ b/README.md @@ -643,30 +643,28 @@ Codegraph also extracts symbols from common callback patterns: Commander `.comma Self-measured on every release via CI ([build benchmarks](generated/benchmarks/BUILD-BENCHMARKS.md) | [embedding benchmarks](generated/benchmarks/EMBEDDING-BENCHMARKS.md) | [query benchmarks](generated/benchmarks/QUERY-BENCHMARKS.md) | [incremental benchmarks](generated/benchmarks/INCREMENTAL-BENCHMARKS.md) | [resolution precision/recall](tests/benchmarks/resolution/)): -*Last updated: v3.12.0 (2026-06-11)* +*Last updated: v3.13.0 (2026-06-17)* | Metric | Native | WASM | |---|---|---| -| Build speed | **4.4 ms/file** | **21.2 ms/file** | -| Query time | **38ms** | **48ms** | -| No-op rebuild | **30ms** | **27ms** | -| 1-file rebuild | **121ms** | **76ms** | -| Query: fn-deps | **2.7ms** | **2.6ms** | -| Query: path | **2.8ms** | **2.5ms** | -| ~50,000 files (est.) | **~220.0s build** | **~1060.0s build** | -| Resolution precision | **84.4%** | — | -| Resolution recall | **56.1%** | — | - -Metrics are normalized per file for cross-version comparability. Times above are for a full initial build — incremental rebuilds only re-parse changed files. v3.12.0 note: native build speed regressed ~22% (3.6→4.4 ms/file) and native 1-file incremental rebuild regressed ~41% (86→121 ms); tracked in [#1446](https://github.com/optave/ops-codegraph-tool/issues/1446). +| Build speed | **4.5 ms/file** | **27.4 ms/file** | +| Query time | **42ms** | **58ms** | +| No-op rebuild | **26ms** | **25ms** | +| 1-file rebuild | **108ms** | **78ms** | +| Query: fn-deps | **2.8ms** | **2.8ms** | +| Query: path | **2.9ms** | **2.5ms** | +| ~50,000 files (est.) | **~225.0s build** | **~1370.0s build** | +| Resolution precision | **93.1%** | — | +| Resolution recall | **61.0%** | — | + +Metrics are normalized per file for cross-version comparability. Times above are for a full initial build — incremental rebuilds only re-parse changed files.
Per-language resolution precision/recall -v3.12.0 note: global precision dropped 89.9%→84.4%, driven by new false positives in `elixir` (+17 FP), `julia` (+11 FP), and `objc` (+5 FP) — all three still have 0% recall; tracked in [#1447](https://github.com/optave/ops-codegraph-tool/issues/1447). Global recall improved substantially (42.3%→56.1%). - | Language | Precision | Recall | TP | FP | FN | Edges | Dynamic | |----------|----------:|-------:|---:|---:|---:|------:|--------:| -| javascript | 100.0% | 97.6% | 41 | 0 | 1 | 42 | 14/32 | -| typescript | 100.0% | 100.0% | 47 | 0 | 0 | 47 | — | +| javascript | 100.0% | 100.0% | 42 | 0 | 0 | 42 | 14/32 | +| typescript | 95.9% | 100.0% | 47 | 2 | 0 | 47 | — | | bash | 100.0% | 100.0% | 12 | 0 | 0 | 12 | 0/1 | | c | 100.0% | 100.0% | 9 | 0 | 0 | 9 | — | | clojure | 80.0% | 26.7% | 4 | 1 | 11 | 15 | — | @@ -674,19 +672,19 @@ v3.12.0 note: global precision dropped 89.9%→84.4%, driven by new false positi | csharp | 100.0% | 100.0% | 23 | 0 | 0 | 23 | — | | cuda | 50.0% | 33.3% | 4 | 4 | 8 | 12 | — | | dart | 0.0% | 0.0% | 0 | 0 | 18 | 18 | — | -| elixir | 0.0% | 0.0% | 0 | 17 | 21 | 21 | — | -| erlang | 100.0% | 100.0% | 12 | 0 | 0 | 12 | — | +| elixir | 100.0% | 81.0% | 17 | 0 | 4 | 21 | — | +| erlang | 0.0% | 0.0% | 0 | 0 | 12 | 12 | — | | fsharp | 0.0% | 0.0% | 0 | 11 | 12 | 12 | — | | gleam | 100.0% | 26.7% | 4 | 0 | 11 | 15 | — | | go | 100.0% | 69.2% | 9 | 0 | 4 | 13 | 13/14 | | groovy | 100.0% | 7.7% | 1 | 0 | 12 | 13 | — | | haskell | 100.0% | 33.3% | 4 | 0 | 8 | 12 | — | | hcl | 0.0% | 0.0% | 0 | 0 | 2 | 2 | — | -| java | 100.0% | 76.5% | 13 | 0 | 4 | 17 | — | -| julia | 0.0% | 0.0% | 0 | 11 | 15 | 15 | — | +| java | 100.0% | 80.0% | 16 | 0 | 4 | 20 | — | +| julia | 100.0% | 73.3% | 11 | 0 | 4 | 15 | — | | kotlin | 92.3% | 63.2% | 12 | 1 | 7 | 19 | — | | lua | 100.0% | 15.4% | 2 | 0 | 11 | 13 | — | -| objc | 0.0% | 0.0% | 0 | 6 | 12 | 12 | — | +| objc | 100.0% | 46.2% | 6 | 0 | 7 | 13 | — | | ocaml | 100.0% | 8.3% | 1 | 0 | 11 | 12 | — | | php | 100.0% | 57.9% | 11 | 0 | 8 | 19 | — | | pts-javascript | 100.0% | 100.0% | 13 | 0 | 0 | 13 | — | @@ -699,24 +697,24 @@ v3.12.0 note: global precision dropped 89.9%→84.4%, driven by new false positi | swift | 75.0% | 42.9% | 6 | 2 | 8 | 14 | 9/9 | | tsx | 100.0% | 100.0% | 13 | 0 | 0 | 13 | — | | verilog | 0.0% | 0.0% | 0 | 0 | 4 | 4 | — | -| zig | 0.0% | 0.0% | 0 | 0 | 15 | 15 | — | +| zig | 66.7% | 13.3% | 2 | 1 | 13 | 15 | — | **By resolution mode (all languages):** | Mode | Resolved | Expected | Recall | |------|--------:|---------:|-------:| | receiver-typed | 32 | 112 | 28.6% | -| module-function | 16 | 112 | 14.3% | -| static | 78 | 96 | 81.3% | +| module-function | 36 | 112 | 32.1% | +| static | 82 | 97 | 84.5% | | same-file | 66 | 90 | 73.3% | -| interface-dispatched | 19 | 19 | 100.0% | +| interface-dispatched | 22 | 22 | 100.0% | | class-inheritance | 8 | 12 | 66.7% | | callback | 7 | 7 | 100.0% | | pts-spread | 4 | 4 | 100.0% | | pts-define-property | 3 | 3 | 100.0% | | dynamic | 3 | 3 | 100.0% | | pts-create-prototype | 2 | 2 | 100.0% | -| points-to | 1 | 2 | 50.0% | +| points-to | 2 | 2 | 100.0% | | re-export | 2 | 2 | 100.0% | | pts-for-of | 2 | 2 | 100.0% | | pts-set | 2 | 2 | 100.0% | diff --git a/generated/benchmarks/BUILD-BENCHMARKS.md b/generated/benchmarks/BUILD-BENCHMARKS.md index d5937a74..4930fe56 100644 --- a/generated/benchmarks/BUILD-BENCHMARKS.md +++ b/generated/benchmarks/BUILD-BENCHMARKS.md @@ -5,6 +5,8 @@ Metrics are normalized per file for cross-version comparability. | Version | Engine | Date | Files | Build (ms/file) | Query (ms) | Nodes/file | Edges/file | DB (bytes/file) | |---------|--------|------|------:|----------------:|-----------:|-----------:|-----------:|----------------:| +| 3.13.0 | native | 2026-06-17 | 695 | 4.5 ↑2% | 42 ↑11% | 29.9 ~ | 61.7 ↓4% | 55564 ~ | +| 3.13.0 | wasm | 2026-06-17 | 695 | 27.4 ↑29% | 57.7 ↑20% | 29.9 ~ | 62.4 ~ | 55216 ~ | | 3.12.0 | native | 2026-06-11 | 670 | 4.4 ↑22% | 37.7 ↑10% | 30.3 ~ | 64 ~ | 55394 ↑3% | | 3.12.0 | wasm | 2026-06-11 | 670 | 21.2 ↑13% | 47.9 ↑9% | 30.3 ~ | 62.2 ↓3% | 54605 ~ | | 3.11.2 | native | 2026-06-01 | 628 | 3.6 ↓3% | 34.3 ~ | 30.9 ~ | 64 ~ | 53894 ~ | @@ -75,38 +77,38 @@ Metrics are normalized per file for cross-version comparability. | Metric | Value | |--------|-------| -| Build time | 2.9s | -| Query time | 38ms | -| Nodes | 20,318 | -| Edges | 42,851 | -| DB size | 35.4 MB | -| Files | 670 | +| Build time | 3.1s | +| Query time | 42ms | +| Nodes | 20,777 | +| Edges | 42,883 | +| DB size | 36.8 MB | +| Files | 695 | #### WASM | Metric | Value | |--------|-------| -| Build time | 14.2s | -| Query time | 48ms | -| Nodes | 20,284 | -| Edges | 41,671 | -| DB size | 34.9 MB | -| Files | 670 | +| Build time | 19.1s | +| Query time | 58ms | +| Nodes | 20,781 | +| Edges | 43,381 | +| DB size | 36.6 MB | +| Files | 695 | ### Build Phase Breakdown (latest) | Phase | Native (build) | WASM (build) | Native (1-file) | WASM (1-file) | |-------|---------------:|-------------:|----------------:|--------------:| -| Parse | 872.9 ms | 10029.5 ms | 0.4 ms | 2.7 ms | -| Insert nodes | 479.5 ms | 492.5 ms | 0.3 ms | 0.5 ms | -| Resolve imports | 5 ms | 20.2 ms | 0.4 ms | 0.5 ms | -| Build edges | 256.2 ms | 489.3 ms | 5.6 ms | 9 ms | -| Structure | 38.4 ms | 56.8 ms | 5.2 ms | 3.7 ms | -| Roles | 109.5 ms | 133 ms | 29.5 ms | 22.8 ms | -| AST nodes | 307.8 ms | 382.9 ms | 0.2 ms | 0.6 ms | -| Complexity | 24.8 ms | 1305.2 ms | 0 ms | 0.6 ms | -| CFG | 239.9 ms | 412.8 ms | 0 ms | 0.4 ms | -| Dataflow | 210.7 ms | 284.6 ms | 0.1 ms | 0.4 ms | +| Parse | 1083.1 ms | 10513.9 ms | 0.4 ms | 3.3 ms | +| Insert nodes | 497.2 ms | 511.9 ms | 0.3 ms | 0.6 ms | +| Resolve imports | 4.5 ms | 19.1 ms | 0.4 ms | 0.6 ms | +| Build edges | 273.6 ms | 4685.9 ms | 5.6 ms | 9.3 ms | +| Structure | 39.2 ms | 64.7 ms | 4.8 ms | 3.6 ms | +| Roles | 108.2 ms | 129.6 ms | 27.7 ms | 25.7 ms | +| AST nodes | 316.7 ms | 397.8 ms | 0.2 ms | 0.8 ms | +| Complexity | 24.5 ms | 1340 ms | 0 ms | 0.7 ms | +| CFG | 245.8 ms | 451.3 ms | 0 ms | 0.3 ms | +| Dataflow | 224.6 ms | 292.9 ms | 0 ms | 0.4 ms | ### Estimated performance at 50,000 files @@ -114,15 +116,17 @@ Extrapolated linearly from per-file metrics above. | Metric | Native (Rust) | WASM | |--------|---:|---:| -| Build time | 220.0s | 1060.0s | -| DB size | 2641.4 MB | 2603.8 MB | -| Nodes | 1,515,000 | 1,515,000 | -| Edges | 3,200,000 | 3,110,000 | +| Build time | 225.0s | 1370.0s | +| DB size | 2649.5 MB | 2632.9 MB | +| Nodes | 1,495,000 | 1,495,000 | +| Edges | 3,085,000 | 3,120,000 | ### Incremental Rebuilds | Version | Engine | No-op (ms) | 1-file (ms) | |---------|--------|----------:|-----------:| +| 3.13.0 | native | 26 ↓13% | 108 ↓11% | +| 3.13.0 | wasm | 25 ↓7% | 78 ↑3% | | 3.12.0 | native | 30 ↑20% | 121 ↑41% | | 3.12.0 | wasm | 27 ↑29% | 76 ↑27% | | 3.11.2 | native | 25 ↑4% | 86 ↑6% | @@ -185,6 +189,8 @@ Extrapolated linearly from per-file metrics above. | Version | Engine | fn-deps (ms) | fn-impact (ms) | path (ms) | roles (ms) | |---------|--------|------------:|--------------:|----------:|----------:| +| 3.13.0 | native | 2.8 ↑4% | 3 ↑11% | 2.9 ↑4% | 44 ↓8% | +| 3.13.0 | wasm | 2.8 ↑8% | 2.7 ↑4% | 2.5 ~ | 42 ↑3% | | 3.12.0 | native | 2.7 ~ | 2.7 ~ | 2.8 ↑4% | 47.8 ↑8% | | 3.12.0 | wasm | 2.6 ~ | 2.6 ↑4% | 2.5 ↑4% | 40.8 ↓4% | | 3.11.2 | native | 2.7 ↑8% | 2.7 ↑4% | 2.7 ↑8% | 44.1 ↑15% | @@ -312,6 +318,2026 @@ pre-parse that previously added ~388ms on native builds. **Note (3.9.5):** No build/rebuild metrics for this release (both engines null) — only import resolution data was collected. Both the WASM and native workers reached the 1-file rebuild phase and then hung past the benchmark's 10-minute per-engine timeout (see `scripts/lib/fork-engine.ts`), so each was killed (`SIGKILL`) before returning results. Import resolution is unaffected because it runs in the parent process and doesn't depend on the full build. 3.9.5 is consequently absent from the top-level version-history comparison table since there are no build-time figures to compare against prior releases. The workflow run is [here](https://github.com/optave/ops-codegraph-tool/actions/runs/24863501577); the root cause will be investigated and the numbers backfilled in a follow-up if possible. @@ -102,6 +151,65 @@ Import resolution: native batch vs JS fallback throughput.