[Bug]: Prisma query engine process dies after prolonged uptime, permanently breaking API key authentication

[sankalpmukim]

[Bug]: Prisma query engine process dies after prolonged uptime, permanently breaking API key authentication

What happened?

The Prisma client inside the LiteLLM proxy process loses its connection to the query engine after prolonged uptime (~5 days in our case). Once this happens, all non-master-key authentication requests fail permanently with a 401 error and the message:

Authentication Error, Client is not connected to the query engine, you must call `connect()` before attempting to query data.

The proxy continues to accept requests authenticated with the master key (since that's validated from env vars, not the DB), but any virtual key stored in the database becomes unusable. The proxy does not self-heal — the consecutive failure counter reached 1486+ with no recovery, and all subsequent requests continue to fail until a full service restart.

Expected behavior: The proxy should either:

• Automatically reconnect the Prisma client/query engine when the connection is lost, or
• Have a health check that detects this state and triggers a restart/reconnect, or
• At minimum, not silently degrade to a state where DB-backed auth is permanently broken with no path to recovery.

Steps to Reproduce

1. Deploy LiteLLM proxy with PostgreSQL database and database_url configured. Create virtual API keys via the proxy.
2. Run the proxy for an extended period (we observed failure after ~5 days of uptime).
3. At some point, the Prisma query engine process dies (PID becomes 0 in logs).
4. Send a request with a virtual key (non-master):

   curl -s http://localhost:8080/v1/chat/completions \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer sk-9xXw-abcde" \
     -d '{"model":"llm","messages":[{"role":"user","content":"test"}]}'

5. Observe 401 error. All subsequent requests with any virtual key also fail. Master key requests still succeed.
6. Only recovery is sudo systemctl restart litellm-proxy

Config

# config.yaml
general_settings:
  database_url: os.environ/DATABASE_URL
  master_key: os.environ/LITELLM_MASTER_KEY

router_settings:
  routing_strategy: least-busy
  redis_host: localhost
  redis_port: 6379

# .env
LITELLM_MASTER_KEY="ZSabcde="
DATABASE_URL="postgresql://litellm:<password>@localhost:5432/litellm"

Relevant log output

04:40:26 - LiteLLM Proxy:WARNING: utils.py:4176 - Escalating to heavy reconnect after 1485 consecutive failures. reason=auth_get_key_object_lookup_failure
04:40:26 - LiteLLM Proxy:WARNING: utils.py:4183 - Attempting Prisma DB reconnect. reason=auth_get_key_object_lookup_failure
04:40:26 - LiteLLM Proxy:WARNING: utils.py:4112 - prisma-query-engine PID 0 is dead; reconnecting.
04:40:26 - LiteLLM Proxy:ERROR: utils.py:4197 - Prisma DB reconnect failed (1486 consecutive). reason=auth_get_key_object_lookup_failure error=Client is not connected to the query engine, you must call `connect()` before attempting to query data.
04:40:26 - LiteLLM Proxy:ERROR: auth_exception_handler.py:95 - litellm.proxy.proxy_server.user_api_key_auth(): Exception occured - Client is not connected to the query engine, you must call `connect()` before attempting to query data.

Traceback (most recent call last):
  File ".../litellm/proxy/auth/user_api_key_auth.py", line 1194, in _user_api_key_auth_builder
    valid_token = await get_key_object(
  File ".../litellm/proxy/auth/auth_checks.py", line 2327, in get_key_object
    _valid_token: Optional[BaseModel] = await _fetch_key_object_from_db_with_reconnect(
  File ".../litellm/proxy/auth/auth_checks.py", line 2231, in _fetch_key_object_from_db_with_reconnect
    return await prisma_client.get_data(
  File ".../litellm/proxy/utils.py", line 3200, in get_data
    raise e
  File ".../litellm/proxy/utils.py", line 3112, in get_data
    response = await self._query_first_with_cached_plan_fallback(
  File ".../litellm/proxy/utils.py", line 2757, in _query_first_with_cached_plan_fallback
    return await self.db.query_first(sql_query, *args)
  File ".../prisma/client.py", line 524, in query_first
    results = await self.query_raw(query, *args)
  File ".../prisma/client.py", line 559, in query_raw
    resp = await self._execute(
  File ".../prisma/client.py", line 663, in _execute
    return await self._engine.query(builder.build(), tx_id=self._tx_id)
  File ".../prisma/client.py", line 701, in _engine
    raise errors.ClientNotConnectedError()
prisma.errors.ClientNotConnectedError: Client is not connected to the query engine, you must call `connect()` before attempting to query data.

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on?

v1.83.14

[danielrap-conifersai]

reproduced in 1.83.14

[jano-duvo]

reproduced in 1.83.7

[spflueger]

we also experienced this recently. in my case the root cause was that there were too many open file handles (that includes any open sockets). which caused all kinds of problems. In case you have similar issues you can at least increase the number of file handles. I'm a bit puzzled though, why litellm is not able to recover though. that would mean that the file handles never get released and the db connection cant aquire a new file handle and gets stuck on retrying

[sankalpmukim]

it seems juspay maintains a fork of litellm, will try shifting to that one

[lisuizhe]

We also had this issue recently.
Could liltellm verify this change and merge into master
https://github.com/juspay/litellm/pull/168/changes
https://github.com/juspay/litellm/pull/170/changes