[PureGo] Remove UC descriptor cache - #699
Merged
Merged
Conversation
Fetching a UC descriptor is now an explicit standalone call rather than a step inside stream creation, so callers can fetch once and reuse the bytes for every stream on a table. The TTL, entry limit, and credential-scoped cache key bought nothing in exchange for that complexity. Document the fetch-once-and-reuse contract in the README, package godoc, and FetchProtoDescriptorFromUC godoc so the cost of repeated calls is explicit. Signed-off-by: Zlata Stefanovic <zlata.stefanovic@databricks.com>
zlata-stefanovic-db
requested review from
a team,
andrijast-db and
teodordelibasic-db
August 7, 2026 14:02
The end-to-end public-flow test exercised the deprecated FetchProtoDescriptor, so the recommended entry point had no coverage and the test would break whenever the alias is removed. Point it at FetchProtoDescriptorFromUC and cover the alias with a delegation test. Signed-off-by: Zlata Stefanovic <zlata.stefanovic@databricks.com>
danilonajkov-db
approved these changes
Aug 7, 2026
danilonajkov-db
left a comment
Member
There was a problem hiding this comment.
LGTM, I like this approach its better than us internally managing it
Merged
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Removes the Unity Catalog descriptor cache from the PureGo SDK and documents the fetch-once-and-reuse contract in its place.
WHAT:
purego/zerobus/sdk.go: thedynamicSchemaCachefield,cachedDescriptorstruct, the 5-minute TTL and 128-entry constants,getDynamicDescriptorFromCache,storeDynamicDescriptor, the SHA-256 credential-scopeddynamicSchemaCacheKey, and the cache reset inClose.fetchProtoDescriptornow goes straight to UC, andSDK.muguards only the open-stream set and the closed flag.FetchProtoDescriptorFromUCgodoc (with a reuse-across-streams snippet and an explicit warning that per-stream calls put a UC round-trip on the ingestion path), a new "Proto descriptors from Unity Catalog" section in the package godoc, and a callout in the README's dynamic-proto section.examples/dynamic/singleout of itsopenStreamhelper and intomain, since bundling fetch into stream creation is exactly the pattern that now costs a round-trip per stream. Thebatchandprotoexamples already fetched once and only gain a comment.NEXT_CHANGELOG.mdentry to the current API name and state the fetch-once semantics.WHY:
Per @andrijast-db's review feedback: fetching the descriptor is now an explicit standalone call rather than a step inside stream creation, so a caller can fetch once and reuse the bytes for every stream on a table. That makes the cache's cost/benefit negative — it bought a TTL, an entry limit, a credential-scoped key, and a proposed
RefreshProtoDescriptorFromUCescape hatch, in exchange for saving a round-trip that correct callers do not make in the first place.Deleting it also resolves #676 by construction rather than by implementing the lifecycle machinery that issue proposed. There is no stale descriptor to invalidate because every call is fresh, no cache key to coalesce misses on, and no shared in-flight work whose cancellation needs isolating. OAuth token minting is still shared across streams via
auth.SharedTokenCache, which is unaffected by this change.This supersedes #678, which is now closed.
Note for reviewers:
FetchProtoDescriptorremains as aDeprecated:alias forFetchProtoDescriptorFromUC. Both names landed in #666 and v0.1.0 has not shipped, so it is deprecating a name no user has seen. I left it alone to keep this PR scoped to the cache removal, but I am happy to delete it here if you would rather not carry it into the first release.How is this tested?
cd purego && go test -race ./...— all packages pass.cd purego && go vet ./...andgofmt -l .— clean.cd purego/examples && go build ./... && go vet ./...— clean.Test changes:
TestSDKFetchProtoDescriptor_CacheHitbecameTestSDKFetchProtoDescriptor_FetchesOnEveryCall. It now asserts two UC schema requests for two fetches, pinning the no-caching behavior instead of the old "1 request" expectation, and still checks that repeated fetches return equivalent descriptors.TestSDKFetchProtoDescriptorFromUC_RejectsClosedSDK. The closed-SDK guard was previously only exercised through the deletedstoreDynamicDescriptortest, so this keeps it covered against the public API.Closes #676