Consumption billing metrics walk the entire KV store every 10 minutes on the active node, with no way to disable it (2.0.x)

[vpotapepic]

Summary

After upgrading a 3 node HA cluster from 1.21.3 to 2.0.3, the active node started running out of
memory every few minutes. Standby nodes are unaffected. We traced it to the consumption billing
worker, which walks the entire KV store every 10 minutes just to count how many secrets exist.

On a small store this is invisible. On a large store it is fatal, and there is no configuration
option or environment variable to turn it off.

Environment

• Vault 2.0.3, Community Edition, official container image
• Storage backend: MySQL with ha_enabled = "true"
• Auto unseal via cloud KMS, 3 nodes, one active and two standby
• Running in Kubernetes with a memory limit on the container
• Previously running 1.21.3 with the same configuration and the same data

What happens

Memory on the active node stays flat for about 10 minutes after it acquires leadership, then grows
at roughly 70 MiB per minute until the container hits its limit and is OOM killed. Leadership moves
to another node and the same thing happens there, so the cluster ends up in a restart loop with a
period of about 15 minutes.

Timeline from one node, memory in MiB:

12:04 38 leader for 10 minutes, flat
12:05 131 growth starts
12:09 369
12:10 OOM, restart

For five days before the upgrade, on 1.21.3, the same nodes with the same data stayed between 47
and 61 MiB with zero restarts.

The last log line before every OOM is always the same:

core: updated replicated hwm role and managed key counts: prefix=replicated/

The next expected line, updated replicated max kv counts, never appears, so the process dies
inside the KV counting step that follows.

Where it comes from

• vault/billing/billing_counts.go: BillingWriteInterval = 10 * time.Minute
• vault/consumption_billing.go: consumptionBillingMetricsWorker is registered in
  postUnsealFuncs unconditionally, then updateBillingMetricsLocked calls
  UpdateReplicatedHWMMetrics and UpdateMaxKvCounts on the active node
• vault/consumption_billing_util.go: UpdateMaxKvCounts calls GetKvUsageMetricsByNamespace
• vault/core_metrics.go: GetKvUsageMetricsByNamespace calls walkKvMountSecrets for every KV
  mount, which walks every path in the mount and only increments a counter

So the whole store is enumerated every 10 minutes to produce a single number.

Evidence from a heap profile

Heap profile taken on the active node right after the task ran, top entries:

13.00MB 18.4% logical.(*StorageView).ExpandKey
10.00MB 14.1% vault.(*AESGCMBarrier).encrypt
6.60MB 9.3% golang-lru/simplelru.(*LRU).Add (cum 13.1MB)
6.50MB 9.2% container/list.(*List).insertValue
4.00MB 5.7% vault.(*AESGCMBarrier).putInternal (cum 19.0MB)
1.25MB 1.8% keysutil.(*encryptedKeyStorage).List (cum 14.0MB)
1.00MB 1.4% sdk/physical.(*Cache).Get

Differential profile across one run of the task shows the growth is in
keysutil.(*encryptedKeyStorage).List, mysql.(*MySQLBackend).List and simplelru.(*LRU).Add.

The walk itself does not retain secrets, but every entry it reads passes through the physical
storage read cache, whose default size is 131072 entries. Since the total cache size in bytes
depends on the size of the entries, a store with multi kilobyte entries ends up with a cache of
several hundred megabytes, built up in a few minutes by a task nobody asked for.

Goroutine count stays flat, leases stay at 1, and GC runs about once per minute, so this is not a
goroutine or lease leak. The objects are reachable, they are cache entries.

What we ruled out

• Client load. The active node had no client requests at all during the growth
  (vault_core_handle_request_count was not even present), only internal HA lock traffic
• disable_mlock, which is required on 2.0.2 and newer, since 1.21.3 with the same flag does not grow
• KMS auto unseal, vault_seal_encrypt_time_count stays at zero between unseals
• TLS to the database, reproduced with and without TLS with the same result
• Telemetry cardinality, under 200 metric lines
• Connection or file descriptor leaks, around 20 established connections, stable
• usage_gauge_period = "none", which is already set and reported as disabled in the log, and
  VAULT_DISABLE_KV_GAUGE, which applies to kvSecretGaugeCollector and not to the billing path

How to reproduce

1. Start MariaDB or MySQL and a single Vault 2.0.3 with the mysql storage backend and
   ha_enabled = "true", so the node becomes active
2. Write a large number of KV v2 secrets, for example 400000 entries with a few kilobytes each
3. Watch container memory and the log, and wait for the 10 minute billing tick

For comparison, the same steps with 1.21.3 never log the hwm lines and memory stays flat.

Why this matters

Our store is in the terabyte range, and the enumeration also means a continuous stream of list
queries against the database, roughly one list per 700 keys in our measurements, repeated every 10
minutes forever. The cost grows with the size of the store, so the more data an installation has,
the worse it behaves, and it appeared in a patch level upgrade without any note in the changelog.

Lowering cache_size keeps the memory bounded, but it does not stop the enumeration, and it makes
the read cache useless for real traffic, because every 10 minutes it is flushed with entries the
billing task touched.

What we would like

1. A way to disable this collection, either a configuration option or an environment variable
2. Or counting that does not enumerate the whole store, for example maintaining counters
   incrementally on write
3. Or at minimum, keeping this walk out of the shared read cache so it cannot evict live working
   set or blow the memory budget
4. Documentation for the behaviour, including the fixed 10 minute interval, since this is new in
   2.0.x and changes resource requirements for existing installations

[dosubot]

Hi @vpotapepic! I'm Dosu and I'm helping the HashiCorp, an IBM Company team.

This is an excellent and thorough bug report. I can confirm from the codebase what you've described: the consumptionBillingMetricsWorker runs unconditionally on active nodes every 10 minutes (BillingWriteInterval = 10 * time.Minute) [1], and calls updateBillingMetricsLocked which invokes CountMetricsSecretMounts(true, true) to walk all KV mounts [2]. The walkKvMountSecrets function performs recursive LIST operations across the entire KV directory structure [3], routing each through c.router.Route(ctx, listRequest) [4], which populates the physical storage cache with billing-related data — exactly the cache pollution mechanism you identified.

There is currently no configuration option or environment variable to disable consumption billing metrics collection. The VAULT_DISABLE_KV_GAUGE variable you already tried applies only to the separate kvSecretGaugeCollector, not the billing path [5]. There is a MetricsUpdateCadence field in the billing config [6], but it's marked as "For testing purposes" and not intended for production use.

A related PR #30466 refactored the KV enumeration walker in core_metrics.go [7], but it did not add a disable mechanism for the billing metrics collection itself.

Your proposed mitigations all seem reasonable — particularly option 1 (a disable flag) and option 3 (keeping the walk out of the shared read cache). As a workaround in the meantime, lowering cache_size bounds memory as you noted, but at the cost of cache effectiveness for real traffic.

Since this is a feature request / bug report for Community Edition, tagging @heatherezell to flag this for product and engineering review. The core issue — that the billing walk is unconditional, non-configurable, and interacts destructively with the storage cache for large KV stores — appears to be a genuine gap in the 2.0.x release.

Written by @dosu, approved by heatherezell

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.