Objects appear in listings but GetObject returns NoSuchKey

[dennisoderwald]

We migrated a few hundred thousand objects into the gateway using rclone (source was a Jottacloud crypt remote). Since then, 1,225 of these objects are listed correctly but cannot be downloaded: ListObjects and HeadObject return proper metadata (name, size, mtime), GetObject fails with NoSuchKey.

Example:

    $ rclone ls s3:opencloud/03feda23-.../3c/08/4d/6b/-c0e7-4fb1-b4c9-6656a058ef82
      422589712 -c0e7-4fb1-b4c9-6656a058ef82

    $ rclone cat s3:opencloud/03feda23-.../3c/08/4d/6b/-c0e7-4fb1-b4c9-6656a058ef82
    ERROR: operation error S3: GetObject, https response error StatusCode: 404,
    RequestID: E0KSB9FY0BUKSQVF, NoSuchKey

So the metadata exists but there is no data body behind the key. Objects in the same directory with the same naming scheme download fine, so it's not a general problem with these paths (the leading "-" in the names was my first suspect, but healthy objects have that too).

I thought I'd open an issue, since I'm not quite sure where this problem originated.

[kenchrcum]

Hi @dennisoderwald

do you have the logs of the migration? Did "something" occur during the migration of these files? Of how many files consisted the whole migration? Can you download the encrypted object blob through a direct connection to the S3 provider or is the file totally missing on the endpoint?

[dennisoderwald]

@kenchrcum Update: found the root cause, and it's reproducible. The missing objects were all multipart uploads, and the direct cause is the .mpu-manifest companion objects the gateway writes next to every encrypted MPU object.

What happened in our case: after the initial migration we ran a routine rclone sync <source> <gateway-bucket>. The manifests only exist on the gateway side, so rclone treated them as extraneous files and pruned all of them. From that point on, every multipart object in the bucket was listed with correct metadata but failed GetObject with NoSuchKey. The numbers match exactly: 1,225 broken objects then, 1,229 multipart objects in the freshly migrated bucket today.

Reproduction against 0.11.2:

# upload something over the multipart threshold
dd if=/dev/urandom of=test.bin bs=1M count=300
rclone copyto test.bin gw:bucket/test.bin      # works, creates test.bin.mpu-manifest

# delete the companion — this is what any mirror/sync tool will do,
# since the manifest doesn't exist on its source
rclone deletefile gw:bucket/test.bin.mpu-manifest

# read the object again
rclone cat gw:bucket/test.bin
# -> 404 NoSuchKey: The specified key does not exist: bucket/test.bin.mpu-manifest

Note the error detail: the 404 actually names the manifest key, not the requested key - that would have saved us a lot of debugging time if rclone surfaced it, but as a client you normally only see "NoSuchKey" for an object that is plainly there in the listing.

So the issue title still stands, but the actionable points are now:

1. .mpu-manifest companions are exposed through the client-facing API (listed and deletable). As long as that's the case, every deployment is one rclone sync away from silently breaking all of its multipart objects. Hiding them from listings, or refusing client deletes while the main object exists, would close the trap.
2. A GET that fails because the manifest is gone should return an honest error (e.g. InternalError with a hint) instead of passing the backend's NoSuchKey for the manifest through to a GET of the main object.
3. A consistency check ("list multipart objects without a readable manifest") would make the blast radius visible. We only recovered because the original source still existed and we could force a re-upload with --ignore-times.

Some thoughts on possible fixes, feel free to ignore:

1. Listing/deletion exposure: filtering the .mpu-manifest suffix out of ListObjects(V2) responses would be the cheap fix. Rejecting DeleteObject on a manifest while its main object still exists would close the other half of the trap (perhaps allow it with an explicit override header for admin cleanup). Longer term, moving the companions under a reserved prefix (e.g. .seg-meta/<key>) instead of <key>.mpu-manifest would keep them out of clients' way entirely — it would also avoid the ambiguity with a legitimate user object that happens to be named *.mpu-manifest.
2. If you do hide them, it should probably be configurable: tools that back up the raw bucket (including manifests, for disaster recovery) have a legitimate reason to see them.
3. Error mapping: in the MPU GET path the manifest fetch error is currently translated and returned as-is, which is how the backend's NoSuchKey for the manifest ends up as the response for the main object. A dedicated error ("encryption metadata missing for this object") plus a loud log line would make this diagnosable in minutes instead of days.

Workaround:

rclone sync gw:bucket --exclude "*.mpu-manifest"

[kenchrcum]

Thanks for investigating this and for providing the reproducible case. The root cause is clear now: encrypted multipart uploads use a .mpu-manifest companion object, and the subsequent rclone sync removed those gateway-owned manifests because they were not present on the source. The main objects then remained visible in listings and HEAD requests, but could no longer be decrypted.

A fix is on the way. The gateway will stop exposing .mpu-manifest objects through client-facing ListObjects responses, and missing MPU metadata will produce a clearer gateway error instead of forwarding the manifest's NoSuchKey response as if the main object were missing. We are also adding regression coverage for this exact scenario.

Funnily enough, exposing the manifests was a deliberate decision: they are real objects stored on the backend, and we wanted the gateway's listings to reflect the actual contents of the storage endpoint. In this case, however, showing the complete backend truth is not the best client-facing behavior. The manifests are implementation details required by the gateway, and ordinary synchronization tools should not treat them as user data.

Protection against direct deletion of manifests through the gateway is also important, but will be handled as a follow-up, as it is a bigger change. Until the listing change is available in the next version 0.11.3, excluding manifests remains necessary with your suggested workaround.

Thanks again for finding and reproducing this. The exact object counts and the direct deletion test made the cause unambiguous.

[kenchrcum]

We had to sacrifice release 0.11.3 to the release god, so please try out version 0.11.4, it contains the fix for MPU manifests to not show up in ListObject operations.

[dennisoderwald]

I will try it out :)

[dennisoderwald]

After an initial test, it looks like it's working.