[Bug]: Failed binRefine task on local machine

[TrilobiteVenere]

Dear Francisco,

I am trying to run metaGEM on a set of 21 environmental samples. I followed the tutorial up to the three binning tools, which worked without problem. However, the binRefine task fails to run whatever I try. I found similar issues, but could not fix it even then.

From what I can tell, the error appears early in the process, likely around the renaming of the bin folders. I tried to fix it and to locate the error more precisely, to no avail.

Here is the line I used:
bash metaGEM.sh -t binRefine -j 1 -c 24 -l

Attached here is the complete snakemake log. Please let me know if you need anything else.
2025-11-18T150933.697803.snakemake.log

Thank you very much much in advance.

Sincerely,
Martin

[franciscozorrilla]

Hey Martin, could you please share one of the corresponding job logs? The Snakemake log you shared gives you an overview of the jobs and if they failed/ran properly, but inside the logs/ folder you should find some files with more specific info for each job, I suspect this file will shed light on the issue.

[TrilobiteVenere]

Hi Francisco, thank you for your quick answer. Are you talking about the /metaGEM/workflow/logs folder? If yes, it is completely empty, and I'm not sure why...
Sorry if I'm a bit lost. I'm primarily a geologist, so I'm kinda new to this :)

Thank you again,
Martin

[franciscozorrilla]

Hey Martin, no problem and welcome to the computational microbiology party 🥳

OK thanks for the info, actually that makes sense since in your case you are running the jobs locally; the log files are generated only when submitting jobs to the cluster as you can see in this config file:

metaGEM/config/cluster_config.json

Line 9 in 8609ad6

"output" : "logs/{wildcards}.%N.{rule}.out.log",

I actually never run metaGEM locally since it was designed for HPC usage, I only developed this feature based on user request so there may be some small issues like this (logs not getting written).

If you are feeling brace you can try modifying your Snakefile rules to add the log parameter as specified in the documentation here. However, in your current setup you should be getting the output/log of each job printing directly into your terminal, for example I would expect to see an output similar to this other user who was also running this task locally #143. Could you check your terminal to see if you find the output somewhere? Here I would expect to find some useful error message telling us what is going wrong.

[TrilobiteVenere]

Dear Francisco, I figured that was it, I got from the other issues that the workflow wasn't optimized for local use. Still works great, though!

I redirected the terminal towards a "manual" log file. From what I can see, the issue seems to fall somewhere around the "Renaming bin folders to avoid errors with metaWRAP" part (especially the concoct ones, see lines 202-221 of the file). I find that strange, because I had a look at the process using the 'watch' command on the 'scratch/refined_bins' folder, and I could see that the bins from all three binning tools got renamed just fine.
generated_log.txt

Hope this is enough for you to get a clear picture of all that.

Thanks again!

[franciscozorrilla]

Hey Martin, sorry for the delay I was busy with teaching responsibilities. Thanks for sharing the log, looks like you are correct there is some issue with renaming of the concoct bins folder

mv: cannot move '2NR1_L2.concoct-bins' to '2NR1_L2.concoct/2NR1_L2.concoct-bins': Directory not empty

This is strange behavior because the corresponding line of code

metaGEM/workflow/Snakefile

Line 772 in 8609ad6

mv $(basename {input.concoct}) $(echo $(basename {input.concoct})|sed 's/-bins//g')

is supposed to rename the folder, but this error message shows that you are trying to move a folder into another folder.
What most likely happened is that you had previously run these rules/jobs and there was a leftover folder called 2NR1_L2.concoct, so instead of renaming 2NR1_L2.concoct-bins it tries to move it into this old folder. Please make sure that all files/folders from any previous binRefine jobs are deleted before trying to re-run these jobs. Let me know if this fixes your issue!

Best,
Francisco

[TrilobiteVenere]

Dear Francisco,

I tried a bunch of things to fix my problem. I think I might know what my problem was.

When I try running the lines one by one in my terminal, I run into a problem at the metaWRAP command: 'metaWRAP command not found'. When I try again in the metawrap environment, it works, which makes sense.

From what I can get, you have to be in the metagem environment to launch metaGEM.sh. I'm guessing the environment should switch to metawrap at some point to be able to call the metaWRAP tool, but it doesn't. As you can see in the log file I sent previously, the first line of the shell script is the following: '# Activate metawrap environment'. However, this line is empty in the snakefile.

Do you have any idea what's causing this problem?

Thanks again for your help!

Cheers,
Martin