2.2.1

README updates

• added dag
• added run-time plot
• updated indexing storeDir image
• updated k+w guidance
• updated table to include accuracy metrics

Seed density for aDNA improves performance, particularly at 30bp

• updated w default to 3

Identity filter benchmark results

• We found no major benefit of applying the identity filter by default (it also does a bad job at replacing MAPQ for ultra-short reads, likely due to repeats. These can pass identity filters, but would fail MAPQ due to multi-mapping)
• Default set to false, and when turned on, 0.85. Should always be run alongside a suitable MAPQ filter.

Graph index storedir

• Existing storeDir target made rerunning with different k / w values prone to error (nextflow sees it as stored, but it should be re-run)
• Target name now includes k + w - respective indexes are properly stored for each iteration
• Compute snarls now stores to a snarls directory, no re-run required - but was previously alongside the indexes

2.2.0

GAM filtering defaults

• set default GAM filtering mapq to 30, which achieves a reasonable balance between yield, error, and microbial contamination removal
  • recommended settings would range from 20 to 30 depending on the level of sample contamination. Pure/high quality samples, use 20. Typical aDNA, use 30.
  • Below 20, mapping inaccuracy becomes too high. Above 30, you lose many good alignments for diminishing improvements to error rate
  • on the microbial side, 20 to 25 get quite a lot of mappings. 30 does a good job of removal.
• Thus 30 is the choice until we can implement effective & conservative prefiltering of microbial reads

Identity filter

• turned off the default identity filtering (since it's an untested filter). See issue #36

Current k + w default

• Full benchmarking showed 15 + 5 performed better for short reads (30-50), and had no cost above this. Until even more options have been tested to optimise the 30bp category, this will be the default

Grave now publishes library level BAMs

• previously we published: per-library filtered GAMs, per-library raw GAMs (if requested), and per-sample merged/deduplicated BAMs
• We now also publish the per-library BAMs (graph mode = surjected, RG tagged, mapped only, sorted. linear mode = RG tagged, mapped only, sorted)
• This adds utility for debugging library issues & also for benchmarking tasks that run only up to alignment

Stats on the raw GAM

• Now optional via param rawMapStats

Unmapped reads in graph modes & correcting surjection stats

• If users request the GAM to be filtered for mapped only, we previously filtered the library-level GAM only (i.e. remove unmapped relative to the graph)

• If that GAM goes on to be surjected to the linear backbone, some reads can become unmapped during that process (i.e. relative to linear coordinates)

• Now, if users ask to filter the GAM for mapped reads only, we also implicitly pass this through to the post-surjection BAM processing (samtools add readgroups, view --exclude-flags 4, sort. If they don't request GAM filtering, no filter is applied to the BAM either

• The other implication is that GAM alignment stats alone were misleading, since not all those filtered alignments survive surjection. We now also run flagstats on the surjected and processed BAM

Stats on the final merged BAMs

• Added stats for the final merged BAMs

Output publishing tweaks

• Due to all the additional stats reporting, reorganised the publishing directories for clarity
• Also cleaned up some of the stats file names to be more informative

nf-core tools

• Bumped to v4.0.2 in pixi env

Apptainer

• Added workflow.containerEngine in ['singularity', 'apptainer'] across all modules using containers, to allow use of the pre-built sif
• apptainer would previously evaluate to false on this check and pull/convert the docker image. No reason to do this & adds some overhead

Linear mode mapq

• Added two params to allow mapq filtering in linear mode (one to set mapq filtering off/on (default true), one for the value (default 20))

Graph snarls needs more HPC memory

• Bumped to 120GB + process_high

Profile to turn off scratch globally

• added no_scratch profile for testing/development purposes, included after the institutional profile in order to override any settings there
• should only be used on micro test datasets to avoid slow IO and disk quota issues

Logo

• added repo logo

Developer guidance

• added guidance to any module with resources specifically set outside of conf/base.config

Containers

• updated vg containers to 1.73

Pixi environment

• vg bumped to 1.73.0 & lock file updated

Giraffe aDNA scoring parameters

• Now implemented as parameters rather than hard-coded, though the settings are currently not changed

Giraffe resources

• testing shows Giraffe runtime scales well with cpu count. Increased default to 64 cores.

• Notes added to nextflow.config regarding successful mapping of ultra-short reads

• added README explainer on k and w choice in the context of aDNA alignment. Will need to be updated to benchmarked data, see #37

• Changed the minimiser and zipcode extensions throughout the codebase to conform with those currently used by vg developers. (i.e. minimiser = .withzip.min, and zipcodes = .zipcodes)

• updated schema with selected defaults

Prioritise filtered graphs

• Although unfiltered graphs + haplo mode are "best practices" in typical use cases, for grave the intended use case is aDNA. Samples will be highly contaminated & the target reads are both short & degraded. Thus we expect k-mer subsampling of the graph to perform worse than a depth filtered graph reference. Thus:

  • Updated basic tests to prioritise filtered graphs

  • Updated docs to reflect latest advice

  • Updated wiki to reflect latest advice

Reference stats reporting processes are now optional

• added the reference_stats parameter to make reference stats reporting processes optional. Prior to the change, running the index step in isolation still forced stats reporting, which can a) take a long time & b) is not what the user requested

vg stats resources

• changed resources label (low to medium) to avoid OOM errors for larger graphs

Read QC file naming

• Now renames input files to raw FASTQC processing. Before, the pre/post FASTP reports needed to be cross-referenced if input FASTQ files were not named by sample. Now, they will share exactly the same sample/read group prefix for quick navigation.

Added a giraffe alignment stats report

• added new stats report prior to alignment filtering, to quickly diagnose where issues might be occuring if low counts are observed, i.e., a) at entry into alignment, or b) at filtering after alignment

Added ovis test resources for internal usage

• requires local graphs/linear reference, not uploaded to the repo due to size

2.2.0-pre

GAM filtering defaults

• [Prelease]: running benchmarks

Logo

• added repo logo

Developer guidance

• added guidance to any module with resources specifically set outside of conf/base.config

Containers

• updated vg containers to 1.73

Pixi environment

• vg bumped to 1.73.0 & lock file updated

Giraffe aDNA scoring parameters

• Now implemented as parameters rather than hard-coded, though the settings are currently not changed

Giraffe resources

• testing shows Giraffe runtime scales well with cpu count. Increased default to 64 cores.

Graph indexing

• K-mer and window size default parameters should now work better for typical degraded aDNA reads (k 19, w 5 selected as a balanced default with excellent performance particularly at 50 & 70 bp, plus acceptable performance at 30)

• Notes added to nextflow.config regarding successful mapping of ultra-short reads

• added README explainer on k and w choice in the context of aDNA alignment, with examples of performance for target read lengths (e.g. k 15, w 5, if 30 bp performance needs to be improved)

• Changed the minimiser and zipcode extensions throughout the codebase to conform with those currently used by vg developers. (i.e. minimiser = .withzip.min, and zipcodes = .zipcodes)

• updated schema with selected defaults

GAM filtering defaults

• [Prelease]: running benchmarks, [release], set defaults

Prioritise filtered graphs

• Although unfiltered graphs + haplo mode are "best practices" in typical use cases, for grave the intended use case is aDNA. Samples will be highly contaminated & the target reads are both short & degraded. Thus we expect k-mer subsampling of the graph to perform worse than a depth filtered graph reference. Thus:

  • Updated basic tests to prioritise filtered graphs

  • Updated docs to reflect latest advice

  • Updated wiki to reflect latest advice

Reference stats reporting processes are now optional

• added the reference_stats parameter to make reference stats reporting processes optional. Prior to the change, running the index step in isolation still forced stats reporting, which can a) take a long time & b) is not what the user requested

vg stats resources

• changed resources label (low to medium) to avoid OOM errors for larger graphs

Read QC file naming

• Now renames input files to raw FASTQC processing. Before, the pre/post FASTP reports needed to be cross-referenced if input FASTQ files were not named by sample. Now, they will share exactly the same sample/read group prefix for quick navigation.

Added a giraffe alignment stats report

• added new stats report prior to alignment filtering, to quickly diagnose where issues might be occuring if low counts are observed, i.e., a) at entry into alignment, or b) at filtering after alignment

Added ovis test resources for internal usage

• requires local graphs/linear reference, not uploaded to the repo due to size

2.1.0

[2.1.0] - 20260309

Updated

HPC resource requests

• Significant improvements to the default resource requests when running on HPC. More scale tests will be needed to finalise these

GAM filtering

• Removed all hard coded GAM filtering choices to args (added switch params for each of these, with a setting param for fine-tuning)
• Unmapped GAM records are now discarded by default (mirrors linear workflow)

Versions

• pixi env nf-core version to 3.5.2

Reference + index map simplified

• Removed complex logic for assigning index elements to the map. Now use a slice operation which allows for variable file counts

Added

1. Parameter for FASTP minimum overlap for merge. Previously used program default of 30, which may have penalised our target fragment group. Set default to 11 to match AdapterRemoval (used by EAGER), + flash, used elsewhere.

2. New test profile & dataset for reads 20-40 bp

3. New logic to handle failed samples. Failed samples are those that have zero aligned reads after GAM filtering. Giraffe now exports the alignment count as an env variable, and the resulting channel is branched (0 = fail, >0 = pass). Previously these would pass to the next process and grave would crash.

The failed samples can now be seen in the results folder: 04_mapped_reads/failed_samples

Fixed

Freebayes memory issues

• The current regions calculation for Freebayes parallel uses extortionate memory and needs to be refactored. In the meantime, added a freebayes mode option, such that by default Freebayes will run in single-threaded mode. This has been tested and finishes using appropriate resource allocation. An issue has been raised to eventually address the parallel mode.

Linear indexing

• Fixed issue whereby accession versions were stripped from the linear reference index storeDir directory. This would have caused issues if users ran grave on two versions of the same reference

Indexing + haplotype subsampling params simplified

• We had 8 confusingly named parameters where 4 would suffice. This is implemented, and names are more clear.
• This additionally fixed a potential mapping issue if users changed the aDNAkmerHaplSubSam kmer value without mirroring the change in aDNAKmerMimizer. This scenario can now be safely handled by a single parameter change.

Confirmed zipcodes issue for aDNA samples

• Recent versions of vg create zipcodes files. If not found during giraffe runs, it auto creates new files with default params, which caused mapping failure for aDNA. This is fixed, with zipcode creation and provision now explicit

Potential mapping parameter issue for modern samples

• For modern DNA we previously allowed Giraffe to run auto-index construction (i.e. default kmer/window params), and controlled it for aDNA. This would cause problems if users adjusted these settings during .hapl construction
• Grave now controls index construction for both sample types for increased reliability

Linear workflow bug

• the BWA aln algorithm can output unmapped records with MAPQ > 0, which is incompatible with strict SAM specs enforced by picard. We now pipe from samse/mem into samtools view, and remove unmapped records prior to samtools sort

2.0.1

Added

• nf-test functionality for main run modes

Fixed

• multi-ref mode now functional again

2.0.0

Key version updates

• Nextflow to 25.10.0
• vg to 1.70.0
• apptainer to 1.4.5 (apptainer env only)

Added

• Linear mode to run mapping with traditional methods against a linear FASTA
• Workflow parameter validation & help message via nf-schema + custom function (removed existing custom validation & help)
• Docker support + other containerisation engines
• Future modules to be initiated with nf-core modules install, then patched
• All modules can now report versions (storeDir modules previously excluded, now require manual version update)

Changed

• Reorganised repository structure, especially data, modules, and subworkflows
• Improved SLURM helper script, now in bin
• General code cleanup
• Some process names changed for clarity

Refactored

Pixi environment

• Updated to version 2 environment
• Tasks overhauled for testing

Workflow

• replaced grave workflow with subworkflows
• main.nf now used to call distinct subworkflows, improving readability/maintainability/extensibility
• Added a linear mapping workflow option

Parameters

• Workflow can now run user-defined steps within the workflow (though some have step dependencies)

Containers

• Various version updates to process containers

Profiles

• Removed custom profiles, replaced with an nf-core style approach (base, resource, & institutional profiles) + test profile

1.4.1

[1.4.1] - 20251007

Bug fix

• Fixed missing args3 in pangenome map module

Parameters

• Tweaked default minimum allele support to allow leniency for aDNA (matches EAGER)

Others

• Small fixes to docs
• Added threads flag to samtools index
• Minor annotation changes
• Removed old --max-fragment-length argument in ancient DNA mapping script blocks - these will always be "single ended" at the mapping stage.

Wiki

• Wiki boilerplate pushed

1.4.0

[1.4.0] - 20251006

Reorganisation & documentation

• Params sections in nextflow.config clarified
• Corresponding help message sections have been clarified, and help messages improved for some parameters
• New dag
• Updated samplesheet info in README

FASTP

• exact hash deduplication now defaults to off, controlled with --fastpDedup
• new parameter to control whether FASTP performs duplicate removal or not (even if turned off we get a rough duplication rate calculation)
• json report now also published alongside html
• improved help message on FASTP duplicate rate/removal params
• keeping unmerged reads now outputs these alongside the merged, rather than into two extra files

FASTQC

• removed empty file check (unmerged reads used to cause this but are output into the merged readfile now)
• html output
• improved syntax for other file extensions

Deduplication overhaul

• Changed samplesheet metadata requirements (library_id and repeat_number: together with sample id, we can now generate a unique read group identifier)
• Removed all hash deduplication steps
• Mapping now done at the library level
• Surject now followed by readgroup annotation, sorting, BAM merging, and deduplication with Picard
  • New deduplication metrics file in results/statistics
  • --removeDuplicates (default true): If true, Picard MarkDuplicates will not write duplicate reads to the output. If false, they will be written, with duplicate flags set (true/false).
  • --duplicateTaggingPolicy (default DontTag): Policy for tagging duplicates in the DT optional SAM/BAM field (DontTag, OpticalOnly, All). Irrelevant if --removeDuplicates is true. See Picard docs for more info.
  • --dedupConsiderBothEnds (default true): If true, for appropriate samples Picard MarkDuplicates will consider both 5' and 3' ends of reads when identifying duplicates. If false, only the 5' end will be considered (true/false). Appropriate samples are "single ended reads" with both ends known, i.e. successfully merged aDNA reads.
• Picard settings allow for ancient DNA to use 3' and 5' information when identifying duplicates, mimicing dedup behaviour
• Since GAMs are split, implemented new merging logic for GAMs prior to vg call

Removed

• deconstructNestedSnarls - this was a bugged parameter, removed to avoid confusion for now.

Pangenome map filter

• Defraying of ambiguously aligned read ends is now turned off by default.
  • To turn it on, use --gamDefrayEnds
  • To control the maximum length of defray, use --defrayEndsLength (default 999)
• MAPQ filter is now on by default (default 30)
  • Control it with --gamFilterMapQ
  • Threshold set by --minimumMapQFilter

DeepVariant

• Updated module to use pangenome aware version, updated code and output processing accordingly

1.3.4

[1.3.4] - 20250912

Parameter updates

• Replaced --gamFilterMore with two parameters:

  • --gamDiscardUnmapped (default false): when true, unmapped reads are discarded when filtering GAM files
  • --gamFilterMapQ (default false): when true, applies MAPQ filtering when processing GAM files. Threshold value is set by --minimumMapQFilter

• Changed default for --minimumScorePrimaryAlign from 0.90 to 0.80, to be more inclusive of aDNA reads with potentially lower fraction identity

• Added --deconstructNestedSnarls parameter. When false, vg deconstruct outputs only top-level sites (non-nested sites). Setting to true provides --all-snarls to vg deconstruct, which will output a site for all snarls spanned by a reference path.

  • Note: incomplete feature! Currently setting the default to true, and raised GitHub issue for further development. To get this working, will need to move vcfbub to a new module. Setting to false results in no LV tag in the output VCF, causing a vcfbub error.

Documentation

• Started wiki on the repo for adding rationale to complex sections + explanation of behaviour

• Improvements to help message

Profile improvements

• Added options to go directly to nf-core profiles for rackham and dardel rather than the custom ones

• Re-enabled scratch space in rackhamSlurm profile, draft fix on the bind mount issue

1.3.3

[1.3.3] - 20250901

Bug fixes

• Fastq merge dedup had a bug whereby it expected library names as integers. Old approach accounted for this with an awk + read line operation

• Replaced by providing the library names within the metamap as a new flattened list, and using these instead

• Temporarily commented out sections applying scratch space in HPC profiles. Running into an issue I haven't yet diagnosed -> causes file not found errors. This is since the Nextflow version bump.