Allocate more time to dorado basecaller

[Sgornard]

Hello,

I am trying to use the pipeline to run on a folder containing 70 pod5 files. Therefore, dorado takes a lot of time. It is however terminated prematurely, because it exceeds 16 hours, and I get the following error in the .nextflow.log file :

Jun-11 05:46:03.655 [TaskFinalizer-1] ERROR nextflow.processor.TaskProcessor - Error executing process > 'NFCORE_METHYLONG:METHYLONG:ONT:DORADO_BASECALLER (zymo_bc01)'

Caused by:
  Process exceeded running time limit (16h)


Command executed:

  dorado basecaller \
       \
      sup \
      pod5 \
      --modified-bases 5mC_5hmC \
       \
      > zymo_bc01_calls.bam
  
  cat <<-END_VERSIONS > versions.yml
  
  "NFCORE_METHYLONG:METHYLONG:ONT:DORADO_BASECALLER":
      dorado: "$(dorado --version 2>&1 | head -n1)"
  END_VERSIONS

Command exit status:
  -

Command output:
  (empty)

I tried to allocate more time using a config file provide with the -c option of nextflow run nf-core/methylong but this hasn't proven successful. The config file I used looked like this :

process {
    resourceLimits = [
        cpus: 50,
        memory: '1200.GB',
        time: '350.h',
    ]
}

Is there a better way to tell the pipeline to allocate more time to dorado?
Thanks a lot!
Best,

[YiJin-Xiong]

Hi @Sgornard,

Thanks for reporting this.

In this case, resourceLimits will not increase the requested runtime for Dorado. It only defines the maximum allowed resources that tasks can request.

The 16-hour limit comes from the default process_high label in conf/base.config:

withLabel:process_high {
    cpus   = { 12    * task.attempt }
    memory = { 72.GB * task.attempt }
    time   = { 16.h  * task.attempt }
}

DORADO_BASECALLER uses this process_high label, so it inherits the default runtime limit of 16 hours. That is why the task was terminated.

If you are running a local copy of the pipeline, you can increase this limit by modifying the corresponding setting in conf/base.config, for example:

withLabel:process_high {
    cpus   = { 50     * task.attempt }
    memory = { 1200.GB  * task.attempt }
    time   = { 350.h  * task.attempt }
}

Alternatively, you can override only the Dorado process in a custom config file:

process {
    withName: 'NFCORE_METHYLONG:METHYLONG:ONT:DORADO_BASECALLER' {
        time = 350.h
        cpus = 50
        memory = 1200.GB
    }
}

and then run the pipeline with:

nextflow run nf-core/methylong \
    -c custom.config \
    ... \
    -resume

Since your input folder contains many POD5 files, Dorado may take a long time to finish. Depending on your dataset size, you may also need to increase the time limits for other labels in conf/base.config, such as process_low or process_medium.

Best,
YiJin