Skip to content

up -d hangs indefinitely with depends_on: condition: service_healthy, even when the dependency becomes healthy #1541

Description

@lautou

Describe the bug

podman-compose -f compose.yaml up -d (no --wait) hangs indefinitely when a service uses depends_on: <service>: condition: service_healthy — confirmed stuck 40+ minutes across two separate CI runs before being cancelled. The hang starts immediately after the dependency's image finishes pulling ("Writing manifest to image destination"), before any container-creation message appears, with zero further output.

The dependency's own healthcheck is not the problem: a separate CI run with the condition: service_healthy clause removed entirely shows postgres reporting healthy within ~10s. So this isn't the dependency actually failing its healthcheck — podman-compose's own dependency-condition polling appears to hang rather than the underlying health status ever resolving.

Versions

  • podman-compose: 1.6.0 (installed via pip install podman-compose in CI)
  • podman: 4.9.3 (Ubuntu-bundled)
  • Environment: GitHub Actions ubuntu-latest runner

To Reproduce

compose.yaml:

services:
  postgres:
    image: docker.io/library/postgres:16-alpine@sha256:57c72fd2a128e416c7fcc499958864df5301e940bca0a56f58fddf30ffc07777
    environment:
      POSTGRES_USER: pie
      POSTGRES_PASSWORD: pie_password
      POSTGRES_DB: pie_db
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U pie -d pie_db"]
      interval: 5s
      timeout: 5s
      retries: 10

  backend:
    image: <any image with a long-running server process>
    depends_on:
      postgres:
        condition: service_healthy
podman-compose -f compose.yaml up -d

Expected: backend starts once postgres reports healthy (a few seconds).
Actual: the command hangs indefinitely with no output, well past the point postgres would already be healthy.

What we ruled out

  • Not a "2 services polling the same condition" issue. Our real compose file has two services depending on postgres's health (backend and a second worker service). A first fix attempt removed the condition from only the second service (leaving it depend on backend instead), keeping backend's own postgres: condition: service_healthy intact. The hang recurred at the identical point — proving a single service alone using the condition is sufficient to trigger it, not concurrent polling from multiple services.
  • Not Podman Compose 1.3.0 up -d command never returns/finishes/ends #1178 / podman-compose fails to enforce the service_healthy condition #1183. Those were fixed by Fix service_healthy condition enforcing #1184, merged into podman-compose 1.4.0 (2025-05-10). Our CI already runs 1.6.0, which includes that fix — this is a different, still-open issue.
  • Not the dependency's healthcheck config itself — see above, postgres becomes healthy quickly on its own once the condition is removed.

Workaround

Removing condition: service_healthy entirely (plain depends_on: [postgres], no condition) resolved the hang immediately — verified across multiple CI runs, an isolated local reproduction, and a real production-style release pipeline. We now rely on the app's own crash/restart resilience (restart: unless-stopped) plus an external, independent health check (a reverse proxy's active healthcheck) instead of podman-compose's own dependency-condition gate.

Full write-up, reproduction data, and the fix: lautou/pie-manager#71

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions