error while downloading schema - received HTTP status 429 - for schema available locally

[dobesv]

Recently we have started to get failures running kubeconform in our automated builds. We get errors like:

error while downloading schema at https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.30.11-standalone-strict/sealedsecret-bitnami-v1alpha1.json - received HTTP status 429

Perhaps github added some new rate limiting logic, I'm not sure why this started failing after it worked for many months.

I tried turning on the cache feature using -cache but it seems like kubeconform might be hitting the server anyway even if we have a file in the cache (maybe validating the cache using ETag / Last-Modified ?).

We actually have all that schemas in a local folder so we don't even need or want to download them from github. It looks like I am able to workaround this by changing the order of the -schema-location arguments to pass default after the local repo.

I do wonder, though, if kubeconform should handle 429 gracefully. I suspect that it is handling status 404/403 for these files by falling back on the additional -schema-location we provide, but with 429 it is hard failing. It could try to use a cached version if found, or wait and retry (although I'm not sure how long it would have to wait for these not found rate limited responses).

Related - could/should the cache remember the 404 response from prior attempts and not try to download for a period of time? That might also help.

[phyzical]

It looks like I am able to workaround this by changing the order of the -schema-location arguments to pass default after the local repo

just to confirm are you saying if -cache is before any -schema-location it avoids the hard fail?

or you still havnt found a workaround for the hard fail yet?

[yannh]

You do need to pass the schema locations in order and pass the -schema-location with the local location first, so it starts looking up the local file first... I am a bit worried by the 429, this would be new to me, also we already use an HTTP client that retries several times... I hope the rate limit is only on cache misses and subsequent runs work fine? 👀

[phyzical]

bit of context (for my case), we currently dont cache (will look at changing that ofc)

But its intermittent for sure, we will run in parallel for 3 environments for example, and 1-2 will clear and a rerun usually lets the third through

wouldnt be surprised if they just have automated lists of highest pull traffic and your repo has made the list, similar to what happened with trivy's docker images back at the start of the year.

What i thought wouldn't matter is the fact these calls are being run from github runners to github servers

Maybe also a sleep is needed between the retries to help reduce the 429?

[vhbfernandes]

(just half-related to OP's problem but people could land here with the same issue)

What I did here for my actions was sparse-cloning master-standalone and master-standalone-strict from default (otherwise its a 20gb repo), and datreeio/CRDs-catalog, then just pointing to local folders and fallback to default

[wtait1-ff]

We're seeing the same issue - after some digging, Github did in fact make recent (May 8) updates to rate limits for "unauthenticated requests", which includes raw.githubusercontent.com.

The middle part of the update is interesting...

Why we’re updating
We’ve recently observed an increase in scraping activity targeting our API

Good to know about the ordering importance of the schema locations 💡 Just to clarify though, it sounds like to properly prioritize using the cache you have to assemble the flags like

kubeconform -cache <dir> -schema-location <dir> -schema-location default [-schema-location ...]

specifying the cache dir twice?

[dobesv]

I think the only ordering issue with the arguments is for the schema location you should put local paths first so it won't got the network for those ones.

…

On May 20, 2025, 8:58 AM -0700, Will Tait ***@***.***>, wrote: wtait1-ff left a comment (yannh/kubeconform#329) We're seeing the same issue - after some digging, Github did in fact make recent (May 8) updates to rate limits for "unauthenticated requests", which includes raw.githubusercontent.com. The middle part of the update is interesting... > Why we’re updating > We’ve recently observed an increase in scraping activity targeting our API Good to know about the ordering importance of the schema locations 💡 Just to clarify though, it sounds like to properly prioritize using the cache you have to assemble the flags like kubeconform -cache <dir> -schema-location <dir> -schema-location default [-schema-location ...] specifying the cache dir twice? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[dud225]

the problem with the current implementation is that it checks all registries in order and that the cache is implemented within the registry, which means that if you have default and then datreeio/CRDs-catalog it will still look up all CRD in default first
the consequence is that you still get those 429 errors even though caching is enabled

@vhbfernandes thanks for you workaround I confirm that it works great

[ErwinSteffens]

Could we not add a parameter to provide it with a GitHub token which is then used to fetch the schemas from GH?

They have applied stricter rate limits now on unauthenticated requests.

[yannh]

We could. I was confused since until recently this domain was using a CDN, and afaik was not rate limited nor supported auth 🤔 Looks like it is & does now, so I'll get to work soon :)

[sarasensible]

Wanted to share my working Github Actions code based on @vhbfernandes 's excellent answer. I am refreshing the Github Actions cache weekly since it is still heavy (17MB compressed and cached) and I don't need bleeding edge schemas. I am also not validating CRDs so I did not pull down the datree repo.

Feedback welcome of course.

jobs:
  validateKubeyaml:
    runs-on: ubuntu-latest
    environment: dev
    steps:
      - name: checkout code
        uses: actions/checkout@v4
      - name: Get date
        id: get-cache-key
        run: |
          echo "key=$(/bin/date -u "+%Y%U")" >> $GITHUB_OUTPUT
      - name: Restore Schemas
        id: kubeconform-schemas-restore
        uses: actions/cache/restore@v4
        with:
          path: kubeconform-schemas
          key: kubeconform-schemas-${{ steps.get-cache-key.outputs.key }}
      - name: Create schemas if do not exist
        if: steps.kubeconform-schemas-restore.outputs.cache-hit != 'true'
        run: |
          mkdir kubeconform-schemas
          cd kubeconform-schemas
          git clone -n --depth=1 --filter=tree:0 https://github.com/yannh/kubernetes-json-schema.git
          cd kubernetes-json-schema
          git sparse-checkout set --no-cone /master-standalone /master-standalone-strict
          git checkout
      - name: Cache Schemas
        if: steps.kubeconform-schemas-restore.outputs.cache-hit != 'true'
        id: cache-yaml
        uses: actions/cache/save@v4
        with:
          path: kubeconform-schemas
          key: kubeconform-schemas-${{ steps.get-cache-key.outputs.key }}
       - name: Check values-dev.yaml
          with:
            entrypoint: '/kubeconform'
            args: "-ignore-missing-schemas -schema-location /home/runner/work/infrastructure/infrastructure/kubeconform-schemas/kubernetes-json-schema -summary -n 10 destination/infrastructure/"

Hope it helps!