Mappings Fail in ECS

[eyespies]

Describe the bug
I've deployed the ECS stack. I've configured an origin (HTTPS, cloudfront in front of replicated S3 buckets, so https://cdn.example.com). I've configured a transformation called "Mobile". I've added a mapping that uses the HTTPS origin and the transformation. I assigned the prefix "/mobile/*".

Based on the documentation which says:

https://<cloudfront-domain>/<mapping-pattern>/<image-path>

with the example

https://d1234567890.cloudfront.net/mobile/sample-image.jpg

I would expect that the flow would be:

https://dit.example.com/mobile/my-photo.jpg -> routes to DIT -> determines mapping / transform / origin -> requests https://cdn.example.com/my-photo.jpg -> transforms according to Mobile transformation -> returns transformed image

But that does not happen. Instead I get an error about INVALID_FORMAT and the log message "Origin does not serve image content. Content-Type: text/html" (it would be GREAT to know what the origin URL is here!)

My only guess, based on using curl, is that the DIT is using the unmodified path /mobile/my-photo.jpg when requesting from the origin, so I expected the origin request to be https://cdn.example.com/my-photo.jpg, but DIT is actually requesting https://cdn.example.com/mobile/my-photo.jpg.

There is an extreme lack of documentation on how to use DIT with ECS; there is a LOT of documentation for Lambda. I tried using generic patterns, e.g. the base64 encoded URLs, but those didn't work and there is no documentation for using them with mappings (are mappings even needed?)

To Reproduce

1. Deploy to ECS
2. Create a mapping with a path prefix where the prefix is NOT part of your origin URL path
3. Request the image from the DIT endpoint using the prefix + the full path to the image in the origin
4. boom error message

Expected behavior
The ability to setup mappings that strip the prefix and perform a transform. For example:

• /mobile/* -> transforms photos according to the MobileImages transform
• /profile/* -> transforms photos according to the UserProfile transform
• /large/* -> transforms photos according to the Large transform

also more documentation on how to use mappings (or not use mappings) with the other parameters..

Please complete the following information about the solution:

• [8.0.3] Version: [e.g. v1.0.0]
• [us-east-2] Region: [e.g. us-east-1]
• [no] Was the solution modified from the version published on this repository?
• [n/a] If the answer to the previous question was yes, are the changes available on GitHub?
• [no; but I doubt that is the issue] Have you checked your service quotas for the sevices this solution uses?
• [yes] Were there any errors in the CloudWatch Logs?

Screenshots

Additional context

[ikp4success]

Thank you for reporting this issue. We have added this to our backlog and are actively looking into it. We will provide an update once we have more information.

If you have any additional details or context that may help us investigate, please feel free to share.

[eyespies]

Hi @ikp4success - thanks for looking into this. I don't have any additional details. I'm hoping this is mostly just a "There is a mis-understanding, we'll add more details to the documentation" kind of fix.

Question for the team maintaining this - should the mapping prefix be stripped, meaning is my interpretation of the documentation correct? If that is the intended design, then I'm fairly certain there is a defect.

If that is not the intended design, and the mapping prefix is not supposed to be stripped, then how can I use query parameters to request custom (e.g. preconfigured in the web UI) or dynamic (e.g. via URL query parameters) transformations? I didn't see links in the ECS documentation, but admit I may have missed them.

[bryceaug]

Hey @eyespies, thanks for putting together this GH issue.

DIT does expose the full origin URL it attempted to resolve via logs, but only for certain error scenarios. I believe what is happening here is the incorrect URL is resolving to an HTML error page with an HTTP 200 status code. From DIT's perspective, the URL resolved successfully, and it proceeds to verifying the response's content type, which does not match any supported image content types. Unfortunately, we don't currently include the resolved origin URL for content-type mismatches. Seeing as this HTTP 200 + Content-Type: text/html pattern is common with CDN custom error responses, we will look into including the origin URL in these scenarios.

You are correct in your assessment that DIT does not currently strip the matched prefix from the incoming request path when constructing the origin URL. I can agree that this behavior does enforce similarities between configured mappings and the origins' content structure that might not always be present. We will take altering this behavior for consideration in DIT's future releases.

Since we currently provide the option to define an optional path attribute on origin entries, I could see a path forward where DIT consistently strips the matched prefix, and when configured, prepends the origin's path attribute before the resource. So in your scenario, as long as the origin was not configured with a path attribute, DIT would construct origin.com/<resource-path>. And had you wanted to, with an origin containing path: /mobile/, DIT would construct origin.com/mobile/<resource-path>. Would you see more utility in DIT with a mapping scheme like this implemented?

In the meantime, you can work around this by using a root-level mapping (e.g., /*) so that the request path passes through to the origin unmodified, or by ensuring your origin's content structure mirrors your mapping prefixes.

Lastly, I agree that our documentation on DIT's mapping implementation and common use cases is light. We will look at including more detail on how DIT evaluates and applies configured mappings when constructing origin requests.

[eyespies]

I have done what you suggested, using a root level mapping. But I then tried to use the Base64 encoded JSON to perform transformations and those did not work.

What is the mechanism, using these mappings, to perform multiple transformations on the static path. E.g., I've setup /* as a mapping, but I want to perform a mobile transform, desktop transform, profile photo transform, etc? I tried using a transform= parameter (suggested by AI) but that didn't work either. Trying to parse the Lambda documentation and test numerous scenarios is not an efficient mechanism to figure out the answer.

[eyespies]

Also @bryceaug - your explanation was my understanding as I traced the errors. I request /mobile/images/something.jpg and then DIT requests HEAD /mobile/images/something.jpg from the origin but the image exists in the origin as /images/something.jpg, so the HEAD request for an invalid URL returns a 200 but a content type of text/html and not the expected image type. I confirmed this behavior on the origin using curl (HEAD an invalid URL returns 200/Content-Type: text/html or whatever and it is not the required binary/image type).

[bryceaug]

@eyespies
The ECS and Lambda deployment options leverage different request patterns. With ECS, we no longer support the B64 encoded requests, or Thumbor style request patterns. Your options are to place transformations in URL parameters using the syntax described here or to capture the transformations in a policy. You can then either map to that policy by attaching a policy to the mapping object, or by marking the policy as default. So in your example, you can create mobile, desktop, and profile-photo policies, and attach each to their individual mapping entity.

[eyespies]

Ok, now that you said "URL parameters" the documentation URL you shared makes more sense. I was interpreting that as "Here are the options you can use when defining transforms in the Admin UI".

I don't know that there would be more utility in making the URL mappings strip the prefix, however it would make for a very clean setup. Instead of requesting /media/images/myimage.jpg?resize=0.5&greyscale=true&..., I could just have the prefix /greyscale map to the preset transform. So my URLs on my site are /greyscale/media/images/myimage.jpg and voila, I get the resized, greyscale image. Or /mobile or /desktop or whatever.

[eyespies]

@bryceaug - reviewing more documentation and I see that this page contains a link to signing URLs, but the signing URLs like is a 404. Updating the link would seem to be a top priority update given the security nature.

Update: added the missing links... Also, it seems that signature link is used in a few places, not just the landing page I mentioned.