Skip to content

perf(pm): read each config file once per command via an mtime-validated cache#137

Merged
colinhacks merged 2 commits into
nubjs:mainfrom
jdalton:read-config-once
Jun 25, 2026
Merged

perf(pm): read each config file once per command via an mtime-validated cache#137
colinhacks merged 2 commits into
nubjs:mainfrom
jdalton:read-config-once

Conversation

@jdalton

@jdalton jdalton commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

The pm_engine re-read and re-parsed the same config files several times per command — root package.json (3x via aube's parser, 3x via serde), .yarnrc.yml (once per key), and the .npmrc ancestor chain (re-walked per key). This caches each per-invocation so every file is read once.

A per-process cache in nub-core (config_cache::MtimeCache), path-keyed and validated on a (mtime, size) stamp. The serde package.json, the aube-parsed manifest, and the yarnrc/npmrc/bunfig readers route through it.

Output-preserving with no stale read: every lookup re-stats the file and serves the cached value only when the stamp is unchanged, so a mid-command rewrite re-reads; a missing/unparseable file is never cached, matching the prior .ok() paths exactly.

Verified: reads of each existing config file drop 3x to 1x on an install fixture; cached vs uncached install/list/why/outdated give byte-identical stdout, stderr, and exit codes. New config_cache unit tests cover hit, mtime invalidation, size invalidation, and not-cached-on-missing. clippy --all-targets --all-features -D warnings, fmt --check, and test -p nub-cli -p nub-core pass.

…ed cache

The pm_engine re-read and re-parsed the same config files several times per
command: the root package.json was parsed by aube's parser 3x (apply_config_scope,
the unsupported-config scan, the injected-deps default) and by serde 3x (the
declared_pm_raw calls in config-scoping, the lifecycle UA, and the install-signals
re-read); .yarnrc.yml was opened once per key (immutable, scripts, network,
hardened); and the .npmrc ancestor chain was re-walked per key.

Add a per-process, mtime+size-validated cache (nub-core config_cache::MtimeCache)
and route those readers through it: root_manifest (serde) in nub-core, a shared
cached_aube_manifest in pm_engine, and a config-text cache for the yarnrc/npmrc/
bunfig readers in unsupported_config. Each lookup re-stats the file and serves the
cached value only when its (mtime, size) stamp is unchanged, so a mid-command
rewrite re-reads — the no-stale-read property is structural, not call-ordering-
dependent. A missing/unparseable file is never cached, matching the prior
read_to_string().ok() behavior exactly.

Output-preserving: same parsed values, same command output and exit codes,
fewer reads. On a one-command install fixture the reads of each existing config
file drop from 3x to 1x.
@vercel

vercel Bot commented Jun 25, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
nub Ready Ready Preview, Comment Jun 25, 2026 4:04pm

Request Review

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes — a per-process, mtime+size-validated config-read cache that collapses the PM engine's repeated per-command reads/parses of the same config files to a single read each.

  • Add MtimeCache<V> primitive — new crates/nub-core/src/config_cache.rs: a path-keyed OnceLock<RwLock<HashMap<…, Entry<V>>>> storing Arc<V>, validated on a (mtime, size) Stamp re-read on every lookup; a None read (missing/unparseable) is never cached, and the value is re-stamped after the read.
  • Route the aube manifest parses through a cacheAUBE_MANIFEST_CACHE + cached_aube_manifest() in pm_engine/mod.rs, used by apply_config_scope, first_catalog_specifier, manifest_has_injected, and manifest_has_pnpm_overrides.
  • Route the unsupported-config readers through a cacheCONFIG_TEXT_CACHE + read_config_text() in unsupported_config.rs for the .npmrc/.yarnrc.yml/.yarnrc/bunfig readers (is_ok_andis_some_and on the classic-yarnrc path).
  • Cache the root manifest in pin resolutionROOT_MANIFEST_CACHE in pm/resolve.rs; root_manifest now returns Option<Arc<serde_json::Value>>, resolves the manifest path before keying the cache, and reuses the already-parsed project.manifest on the no-distinct-workspace-root miss path.

The no-stale-read property is structural — every lookup re-stats and serves the cached value only when the (mtime, size) stamp is unchanged, and the only manifest mutator (write_declared_pm) runs after all config reads. The cache key/content correspondence in root_manifest is sound: detect_project parses root/package.json into project.manifest, and the reuse happens only on the arm where the cache key is exactly that path. The shared statics are test-safe because the unit tests use unique per-test temp dirs, and the config reads run single-threaded before the install fan-out. Unit tests cover hit, mtime invalidation, size invalidation, and not-cached-on-missing.

Pullfrog  | View workflow run | Using Claude Opus𝕏

@colinhacks colinhacks merged commit 2fe966b into nubjs:main Jun 25, 2026
54 checks passed
@colinhacks

Copy link
Copy Markdown
Contributor

Shipped in v0.2.1: https://github.com/nubjs/nub/releases/tag/v0.2.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants