Increase compatibility for tty/no-tty options

[erikvanoosten]

Increase compatibility with docker compose for exec and run commands for the tty options:

• Add support for long option --no-tty besides -T.
• Add support for hidden options -t and --tty.
• Make default for tty auto-detected.
• When no-tty is specified (or detected), delegate to podman without -i (interactive).
• For the run command, do not add -i when detach is used, even when a tty is specified/detected.
• For the run command, forward the --rm option, even when detach is used.

The default for terminal/no-terminal is based on terminal detection, mimicking docker compose code for run and exec.

Hidden options -t and --tty are mimicking docker compose code for run and exec.

The added integration tests require the script command to emulate a TTY. This command is available in most unix environments, but might not be present in a minimal docker image. When script is not installed the test will be skipped.

Fixes #900.

[erikvanoosten]

When I run the integration tests from the main branch on my machine (actually, in a clean fedora pod), a lot of them fail. I failed to find the cause. Unfortunately, I can't verify if my work caused more failures. Any help with fixing main would be appreciated.

[erikvanoosten]

For the reviewers:
I realized that the run command should perhaps default to --no-tty and not add -i when --detach is used. For this I need some time to research what docker compose does.

Unfortunately, the Convert to draft button doesn't work at the moment!

[p12tic]

Overall looks good. I would have expected 4 commits for easier review and easier bisection if the PR breaks something.

[p12tic]

When I run the integration tests from the main branch on my machine (actually, in a clean fedora pod), a lot of them fail. I failed to find the cause. Unfortunately, I can't verify if my work caused more failures. Any help with fixing main would be appreciated.

What are the errors? You should be able to run all tests successfully on a Debian trixie container. See https://github.com/containers/podman-compose/blob/main/.github/workflows/test.yml for how it's setup -- you'll need --privileged --cgroupns=host

[erikvanoosten]

When I run the integration tests from the main branch on my machine (actually, in a clean fedora pod), a lot of them fail. I failed to find the cause. Unfortunately, I can't verify if my work caused more failures. Any help with fixing main would be appreciated.

What are the errors? You should be able to run all tests successfully on a Debian trixie container. See https://github.com/containers/podman-compose/blob/main/.github/workflows/test.yml for how it's setup -- you'll need --privileged --cgroupns=host

Thanks! Adding --cgroupns=host helped a lot. Now only 7 integration tests are failing. Here is my test script:

# I have docker/colima on the host.
# Run this from the repo root.
docker run --privileged --cgroupns=host --rm -ti -v $PWD:/dev/oss/podman-compose/ ubuntu:latest
# Then inside the container:
apt update
apt upgrade -y
apt install -y podman python3 python3.12-venv git
cd /dev/oss/podman-compose/
rm -rf .venv
python3 -m venv .venv
. .venv/bin/activate
# pip install '.[devel]'  <-- taken from docs, but it doesn't really seem to install everything
pip install -r test-requirements.txt.  # this does work

python3 -m unittest discover tests/unit
python3 -m unittest discover -v tests/integration

These are the failing tests:

======================================================================
FAIL: test_deps (deps.test_podman_compose_deps.TestComposeBaseDeps.test_deps)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/dev/oss/podman-compose/tests/integration/deps/test_podman_compose_deps.py", line 34, in test_deps
    self.assertIn(b"HTTP request sent, awaiting response... 200 OK", output)
AssertionError: b'HTTP request sent, awaiting response... 200 OK' not found in b'a540c8785b9fcfd52910f3e43569ccba1a9b2e5efab6c2fc75f36879216647f8\nee5791feb6a3aa06d316bcad41929e1b7422168771df8a7792e2853595c3ab05\ndeps_web_1\n127.0.0.1\tlocalhost\n::1\tlocalhost ip6-localhost ip6-loopback\nfe00::\tip6-localnet\nff00::\tip6-mcastprefix\nff02::1\tip6-allnodes\nff02::2\tip6-allrouters\n172.17.0.2\te1c4e45211f3\n10.89.2.1\thost.containers.internal host.docker.internal\n10.89.2.2\tee5791feb6a3 deps_web_1\n'

======================================================================
FAIL: test_deps_succeeds (deps.test_podman_compose_deps.TestComposeConditionalDeps.test_deps_succeeds)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/dev/oss/podman-compose/tests/integration/deps/test_podman_compose_deps.py", line 166, in test_deps_succeeds
    self.assertIn(b"HTTP request sent, awaiting response... 200 OK", output)
AssertionError: b'HTTP request sent, awaiting response... 200 OK' not found in b'7bbe8d4118d3c39b19ac7d5f639dccbfd5e34818dde2a3df83b4a068c557b664\n217b7417bf116e8b59ed3f4a128d3563fadd0c99327e9f4673298e8ed3b1cd3d\ndeps_web_1\n127.0.0.1\tlocalhost\n::1\tlocalhost ip6-localhost ip6-loopback\nfe00::\tip6-localnet\nff00::\tip6-mcastprefix\nff02::1\tip6-allnodes\nff02::2\tip6-allrouters\n172.17.0.2\te1c4e45211f3\n10.89.2.1\thost.containers.internal host.docker.internal\n10.89.2.9\t217b7417bf11 deps_web_1\n'

======================================================================
FAIL: test_taking_env_variables_from_env_files_from_different_directories (env_file_tests.test_podman_compose_env_file.TestComposeEnvFile.test_taking_env_variables_from_env_files_from_different_directories)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/dev/oss/podman-compose/tests/integration/env_file_tests/test_podman_compose_env_file.py", line 243, in test_taking_env_variables_from_env_files_from_different_directories
    self.assertEqual(
AssertionError: Lists differ: ['ZZV[45 chars]itten', 'ZZVAR2=This value is loaded from .env[38 chars], ''] != ['ZZV[45 chars]itten\r', 'ZZVAR2=This value is loaded from .e[44 chars], '']

First differing element 0:
'ZZVAR1=This value is loaded but should be overwritten'
'ZZVAR1=This value is loaded but should be overwritten\r'

- ['ZZVAR1=This value is loaded but should be overwritten',
+ ['ZZVAR1=This value is loaded but should be overwritten\r',
?                                                        ++

-  'ZZVAR2=This value is loaded from .env in project/ directory',
+  'ZZVAR2=This value is loaded from .env in project/ directory\r',
?                                                              ++

-  'ZZVAR3=TEST',
+  'ZZVAR3=TEST\r',
?              ++

   '']

======================================================================
FAIL: test_exec (exec_tty.test_podman_compose_exec.TestComposeExtraHosts.test_exec)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/dev/oss/podman-compose/tests/integration/exec_tty/test_podman_compose_exec.py", line 23, in test_exec
    self.run_subprocess_assert_returncode([
  File "/dev/oss/podman-compose/tests/integration/test_utils.py", line 94, in run_subprocess_assert_returncode
    self.assertEqual(  # type: ignore[attr-defined]
AssertionError: 1 != 0 : Invalid return code of process 1 != 0
stdout: 
stderr: CRITICAL:__main__:missing files: ['/dev/oss/podman-compose/tests/integration/exec/docker-compose.yml']



During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/dev/oss/podman-compose/tests/integration/exec_tty/test_podman_compose_exec.py", line 99, in test_exec
    self.run_subprocess_assert_returncode([
  File "/dev/oss/podman-compose/tests/integration/test_utils.py", line 94, in run_subprocess_assert_returncode
    self.assertEqual(  # type: ignore[attr-defined]
AssertionError: 1 != 0 : Invalid return code of process 1 != 0
stdout: 
stderr: CRITICAL:__main__:missing files: ['/dev/oss/podman-compose/tests/integration/exec/docker-compose.yml']



======================================================================
FAIL: test_networks (network.test_podman_compose_network.TestPodmanComposeNetwork.test_networks)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/dev/oss/podman-compose/tests/integration/network/test_podman_compose_network.py", line 96, in test_networks
    self.assertEqual(f"{expect}\r\n", out.decode('utf-8'))
AssertionError: 'test1\r\n' != 'test1\n'
- test1
?      -
+ test1


======================================================================
FAIL: test_interface_name (network_interface_name.test_podman_compose_network_interface_name.TestPodmanComposeNetworkInterfaceName.test_interface_name)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/dev/oss/podman-compose/tests/integration/network_interface_name/test_podman_compose_network_interface_name.py", line 56, in test_interface_name
    self.assertEqual("customName0  lo\r\n", out.decode())
AssertionError: 'customName0  lo\r\n' != 'customName0\nlo\n'
- customName0  lo
?            -----
+ customName0
+ lo


======================================================================
FAIL: test_exec (run_tty.test_podman_compose_run.TestComposeExtraHosts.test_exec)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/dev/oss/podman-compose/tests/integration/run_tty/test_podman_compose_run.py", line 22, in test_exec
    self.run_subprocess_assert_returncode([
  File "/dev/oss/podman-compose/tests/integration/test_utils.py", line 94, in run_subprocess_assert_returncode
    self.assertEqual(  # type: ignore[attr-defined]
AssertionError: 1 != 0 : Invalid return code of process 1 != 0
stdout: 
stderr: CRITICAL:__main__:missing files: ['/dev/oss/podman-compose/tests/integration/exec/docker-compose.yml']

These failing tests seem unrelated, however, they pass fine on main. So I'll have to dig a bit deeper...

[p12tic]

- ['ZZVAR1=This value is loaded but should be overwritten',
+ ['ZZVAR1=This value is loaded but should be overwritten\r',
?                                                        ++

-  'ZZVAR2=This value is loaded from .env in project/ directory',
+  'ZZVAR2=This value is loaded from .env in project/ directory\r',
?                                                              ++

-  'ZZVAR3=TEST',
+  'ZZVAR3=TEST\r',
?              ++

These are line ending issues. Seems related to the fact that you're running on MacOS. I would ignore these tests for now. While fixing them would allow to run tests on MacOS in CI, it does not make sense for me to ask this on your unrelated PR.

[erikvanoosten]

@p12tic from reading the Docker source code, I now believe that docker compose up simply forward the --no-tty/--tty option to docker even when detach is used.

I made a few more changes, so that podman-compose no longer passes -i to podman when no-tty is specified or detected, since interactivity is not possible when there is no tty.

Could you perhaps check what is going on in function compose_run? It calls container_to_args which adds the -i option when stdin_open is present. Then the option can be added again from compose_run (unless detach is used). Is this as expected?

Another thing I do not understand: in compose_run the code to add the --rm option is indented so that it is only added when not detached. Docker does allow --rm together with detach. Does podman not allow this? My change here makes it even more complicated. For sure a change is needed, but what?
IMHO this code:

    if args.tty and not args.detach:
        podman_args.insert(1, "-i")
        if args.rm:
            podman_args.insert(1, "--rm")

should either be changed to (podman allows --rm and detach): <---- chose solution in 6b272b4

    if args.tty and not args.detach:
        podman_args.insert(1, "-i")
    if args.rm:
        podman_args.insert(1, "--rm")

or (podman does not allow --rm and detach):

    if args.tty and not args.detach:
        podman_args.insert(1, "-i")
    if args.rm and not args.detach:
        podman_args.insert(1, "--rm")

WDYT?

[erikvanoosten]

These are line ending issues. Seems related to the fact that you're running on MacOS. I would ignore these tests for now. While fixing them would allow to run tests on MacOS in CI, it does not make sense for me to ask this on your unrelated PR.

The branch from this PR shows that these tests fail the same way on the github runner. 🤔

I don't understand. Both mac and unix use LF endings, Besides, I am also running the tests in a linux container.
Let's look at test test_taking_env_variables_from_env_files_from_different_directories, why does it assume output with CRLF line endings? All files around this test use LF line endings, well, at least in my git clone.

Could it be that how you clone the repository affects what newline character is used?

[erikvanoosten]

@p12tic This PR is ready for merging.

[erikvanoosten]

Re. line endings: I can only conclude that the failing test are written to be run under windows.

Could it be that how you clone the repository affects what newline character is used?

Answering myself: no, a fresh git clone from within a container running Ubuntu gives the same results as running the tests against a git clone taken from the Mac host.

[erikvanoosten]

@p12tic Is there anything you need so that this can get merged?