[Bug]: RetryPolicy.InternalServerErrorRetries is silently ignored in get_num_retries_from_retry_policy

[alekc]

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

RetryPolicy defines InternalServerErrorRetries as a field in litellm/types/router.py, but get_num_retries_from_retry_policy() in litellm/router_utils/get_retry_from_policy.py has no branch for InternalServerError.

The function handles five error types (AuthenticationError, Timeout, RateLimitError, ContentPolicyViolationError, BadRequestError) and omits InternalServerError. The value is accepted in config, validates against the pydantic model, and is never read at runtime.

This is the retry-side twin of #29283, which reports the identical omission for AllowedFailsPolicy.InternalServerErrorAllowedFails in get_allowed_fails_from_policy(). That issue has been open since 2026-05-29. Both policy resolvers are missing the same error type, in two different functions, so a fix for one will not fix the other.

Impact: upstream 5xx is the most common transient failure worth retrying, and it is the one error class the retry policy cannot express. Because router.py only overrides num_retries when the resolver returns non-None:

if _retry_policy_retries is not None:
    num_retries = _retry_policy_retries
    _retry_policy_applies = True

a configured InternalServerErrorRetries falls through to the global num_retries. Worse, _retry_policy_applies stays False, so the request additionally goes through should_retry_this_error(), which a caller who set an explicit per-error policy is not expecting to apply.

The silent-acceptance part is what makes this costly: an operator who configures InternalServerErrorRetries: 3 and nothing else gets zero 5xx retries, with no warning at startup and no error at request time.

User Flow

Configure a per-error retry policy on the proxy, scoped to a model group:

router_settings:
  model_group_retry_policy:
    my-model:
      TimeoutErrorRetries: 3
      InternalServerErrorRetries: 3
      RateLimitErrorRetries: 2

Expected: an upstream 500 is retried 3 times.

Actual: InternalServerErrorRetries is never read. The resolver returns None for InternalServerError, so the router falls back to the global num_retries (0 by default), and a 500 is not retried at all.

Proof the bug occurs

Resolving each error class through the public accessor. Timeout and RateLimit come back from the policy; InternalServerError does not:

import litellm
from litellm import Router

pol = {"TimeoutErrorRetries": 3, "InternalServerErrorRetries": 3,
       "RateLimitErrorRetries": 2, "BadRequestErrorRetries": 0,
       "AuthenticationErrorRetries": 0, "ContentPolicyViolationErrorRetries": 0}

rt = Router(model_list=[{"model_name": "m",
                         "litellm_params": {"model": "openai/gpt-4o-mini"}}],
            model_group_retry_policy={"m": pol})

for exc, label in ((litellm.Timeout("t", "openai", "m"), "Timeout"),
                   (litellm.InternalServerError("e", "m", "openai"), "InternalServerError"),
                   (litellm.RateLimitError("r", "m", "openai"), "RateLimit"),
                   (litellm.BadRequestError("b", "m", "openai"), "BadRequest")):
    print(label, rt.get_num_retries_from_retry_policy(exception=exc, model_group="m"))

Output:

Timeout 3
InternalServerError None
RateLimit 2
BadRequest 0

The field validates fine, which is why this is easy to miss:

>>> from litellm.types.router import RetryPolicy
>>> RetryPolicy(**pol).model_dump()
{'BadRequestErrorRetries': 0, 'AuthenticationErrorRetries': 0, 'TimeoutErrorRetries': 3,
 'RateLimitErrorRetries': 2, 'ContentPolicyViolationErrorRetries': 0,
 'InternalServerErrorRetries': 3}

Suggested fix, matching the shape of the existing branches in get_retry_from_policy.py:

if (
    isinstance(exception, InternalServerError)
    and retry_policy.InternalServerErrorRetries is not None
):
    return retry_policy.InternalServerErrorRetries

Two notes on ordering, since litellm.InternalServerError and the other mapped exceptions can overlap depending on how a provider maps status codes: the new branch should sit alongside the others rather than ahead of Timeout, and it is worth confirming no provider maps a 500 onto BadRequestError, which would shadow it.

What part of LiteLLM is this about?

Router / Proxy (retry and fallback logic)

What LiteLLM version are you on ?

1.97.0

Twitter / LinkedIn details

n/a

---

AI assistance used: Yes
Claude Code was used to trace the code path and draft this report; the reproduction above was run against a live 1.97.0 deployment.