Eviction backlog cleaning prevents bazel-remote from starting

[ofats]

We currently use grpc proxy with a single backend pod writing/reading to NFS (single as it cannot write there concurrently out of the box). Restarts to this single pod take tens of minutes for eviction (10TB disk) backlog cleaning:

bazel-remote/cache/disk/load.go

Lines 628 to 642 in 7ee9ad2

	if c.lru.queuedEvictionsSize.Load() > 0 {
	// We were either restarted with a lower cache size, or there is still
	// a backlog of blobs to be removed. Wait for them to be removed before
	// accepting connections, to avoid confusing error logs.
	log.Println("Waiting for blob eviction backlog to complete.")
	for c.lru.queuedEvictionsSize.Load() > 0 {
	time.Sleep(200 * time.Millisecond)
	}
	log.Println("Finished waiting for blob evictions.")
	}
	log.Println("Finished loading disk cache files.")

As the result, we get into the following situation:

• listeners are not up
• => backend is not accessible
• => frontend bazel-remote nodes are timing out on every request to backend
• => clients wait for frontends to Get/Put to backend, rather than failing fast
• => builds are slower with cache than if they were without it

Frontends have smaller local host dist, not able to hold the whole working set of cache, thus they need to go to backend for cold data.

There are 2 solutions to this:

1. (simpler) Only wait for backlog to clean up if MAX_SIZE_HARD_LIMIT is set - there is literally no point of waiting for this cleanup if hard limit is not set - user takes the risk of disk overflow in this case
2. (harder, but better) Start accepting connections and make disk cleanup in background (just like in normal non-startup flow). If MaxSize is close to MaxSizeHardLimit, Puts can reply "507 InsufficientStorage", just like they do already

I read comments in code and commit descriptions, but I don't see a good reason why this long blocking operation was put as a startup guard in the first place.

[ofats]

Yeah, I'd better fail fast with "confusing error logs", than timeout on all requests until cache has started

[mostynb]

cc @ulrfa who worked on this.

[ulrfa]

10 minutes to complete the eviction queue seems very long. I don't think NFS is suitable as storage for bazel-remote. I recommend using local SSD storage instead.

For me, it is more important that bazel-remote is in a good state before accepting connections than minimizing startup time. Examples:

• I use load balancing with redundant shards. I want to route traffic only to healthy alternatives, not to new shards that aren't yet in a good state.
• A typical user who restarts bazel-remote after reducing the configured size does not expect "507 InsufficientStorage" errors, nor that disk usage could initially grow before being reduced.

I don't recommend having multiple bazel-remote instances sharing the same storage, but you seem to be planning that in #896. If you do go that route, could you keep the old pod running until the new pod is ready to accept connections?

I don't have experience using the proxy functionality in bazel-remote. Instead, I run many bazel-remote instances as shards with a proprietary sharding implementation. There are also open-source gRPC cache sharding implementations, such as Buildbarn's bb-storage and bb-clientd, that should work well as frontends to bazel-remote.

builds are slower with cache than if they were without it

If you're OK with building without caches, could the timeouts be reduced so that builds proceed without caches rather than waiting for them?