crl-release-26.2: valsep: account for blob-file compression in split-decision size estimate - #6140
Merged
RaduBerinde merged 1 commit intoJul 31, 2026
Conversation
…mate During a compaction, the size of an output sstable is estimated as keys are written in order to decide when to split into a new table. When the compaction preserves existing blob references, the estimate added the sum of the *uncompressed* value sizes of the preserved references. Because blob files are compressed on disk, this overestimated the disk space the references contribute for compressible data, splitting output tables earlier than intended and producing smaller-than-target files. The final stored `TableMetadata` already accounted for compression via `BlobReference.EstimatedPhysicalSize`; only the in-flight estimate used uncompressed sizes. `ValueSeparator.EstimatedReferenceSize` now scales each preserved reference by its source blob file's compression ratio, using the same computation as `manifest.MakeBlobReference`, so the in-flight estimate converges to the size the table will ultimately report. The source `PhysicalBlobFile` is resolved once when a preserved reference is first created (rather than at `FinishOutput`) and carried on `pendingReference`; the `preserved` boolean is replaced by a `preserved()` method that reports `phys \!= nil`. The shared scaling computation is centralized in a new `PhysicalBlobFile.EstimatedReferencePhysicalSize` method that performs the multiplication in 128-bit precision (`bits.Mul64`/`bits.Div64`) to avoid overflow for large blob files. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
Member
|
This change is |
sumeerbhola
approved these changes
Jul 31, 2026
RaduBerinde
deleted the
valsep-account-for-blob-file-compression-in-split-26.2
branch
July 31, 2026 02:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
During a compaction, the size of an output sstable is estimated as keys
are written in order to decide when to split into a new table. When the
compaction preserves existing blob references, the estimate added the
sum of the uncompressed value sizes of the preserved references.
Because blob files are compressed on disk, this overestimated the disk
space the references contribute for compressible data, splitting output
tables earlier than intended and producing smaller-than-target files.
The final stored
TableMetadataalready accounted for compression viaBlobReference.EstimatedPhysicalSize; only the in-flight estimate useduncompressed sizes.
ValueSeparator.EstimatedReferenceSizenow scales each preservedreference by its source blob file's compression ratio, using the same
computation as
manifest.MakeBlobReference, so the in-flight estimateconverges to the size the table will ultimately report. The source
PhysicalBlobFileis resolved once when a preserved reference is firstcreated (rather than at
FinishOutput) and carried onpendingReference;the
preservedboolean is replaced by apreserved()method thatreports
phys \!= nil.The shared scaling computation is centralized in a new
PhysicalBlobFile.EstimatedReferencePhysicalSizemethod that performsthe multiplication in 128-bit precision (
bits.Mul64/bits.Div64) toavoid overflow for large blob files.
Co-Authored-By: roachdev-claude roachdev-claude-bot@cockroachlabs.com