For converting image annotations datasets from one format into another. Filters can be supplied as well, e.g., for cleaning up the data.
Additional libraries further extend the available plugins with additional readers, filters and/or writers.
Via PyPI:
pip install image_dataset_converterThe latest code straight from the repository:
pip install git+https://github.com/waikato-datamining/image-dataset-converter.gitDocker images are available as well. Please see the following page por more information:
https://github.com/waikato-datamining/image-dataset-converter-all/tree/main/docker
The following dataset formats are supported:
| Domain | Format | Read | Write |
|---|---|---|---|
| Depth data | CSV | Y | Y |
| Depth data | Grayscale | Y | Y |
| Depth data | Numpy | Y | Y |
| Depth data | PFM | Y | Y |
| Image classification | ADAMS | Y | Y |
| Image classification | subdir | Y | Y |
| Image segmentation | Blue-channel | Y | Y |
| Image segmentation | Grayscale | Y | Y |
| Image segmentation | Indexed PNG | Y | Y |
| Image segmentation | Instance PNG | Y | |
| Image segmentation | Layer segments | Y | Y |
| Object detection | ADAMS | Y | Y |
| Object detection | COCO | Y | Y |
| Object detection | Instance PNG | Y | Y |
| Object detection | OPEX | Y | Y |
| Object detection | ROI CSV | Y | Y |
| Object detection | VOC | Y | Y |
| Object detection | YOLO | Y | Y |
Readers and writers typically have a suffix that denotes their data domain:
-dp- Depth data-ic- Image classification-is- Image segmentation-od- Object detection
usage: idc-convert [-h] [--help-all] [--help-plugin NAME] [-u INTERVAL]
[-l {DEBUG,INFO,WARNING,ERROR,CRITICAL}] [-b]
[--variables FILE] [--load_pipeline FILE]
[--dump_pipeline FILE]
Tool for converting between image annotation dataset formats.
readers (30):
cron, from-adams-ic, from-adams-od, from-blue-channel-is,
from-coco-od, from-csv-dp, from-data, from-grayscale-dp,
from-grayscale-is, from-indexed-png-is, from-instance-png-is,
from-instance-png-od, from-layer-segments-is, from-multi,
from-numpy-dp, from-opex-od, from-pfm-dp, from-pyfunc,
from-roicsv-od, from-storage, from-subdir-ic, from-text-file,
from-voc-od, from-yolo-od, get-email, list-files, poll-dir,
shell-exec, start, watch-dir
filters (79):
annotations-from-storage, annotations-to-storage, any-to-rgb,
apply-ext-mask, apply-label-mask, attach-metadata, block,
check-duplicate-filenames, coerce-box, coerce-mask,
convert-image-format, copy-files, count-data, count-specks,
delete-storage, depth-to-grayscale, depth-to-greyscale*,
dimension-discarder, dims-to-metadata, discard-blurry,
discard-by-name, discard-invalid-images, discard-negatives,
discard-positives, exif-autorotate, filter-labels, get-metadata,
grayscale-to-binary, greyscale-to-binary*, inspect, is-to-od,
label-from-name, label-present*, label-present-ic, label-present-is,
label-present-od, label-to-metadata, list-to-sequence, load-data,
log-data, log-placeholder*, log-variable, map-labels, max-records,
metadata, metadata-from-name, metadata-od, metadata-to-placeholder*,
metadata-to-variable, move-files, od-to-ic, od-to-is, passthrough,
polygon-discarder, polygon-simplifier, pyfunc-filter,
randomize-records, record-window, remove-alpha, remove-classes,
rename, rgb-to-grayscale, rgb-to-greyscale*, sample, sanitize-name,
set-metadata, set-placeholder*, set-storage, set-variable, sleep,
sort-pixels, split-records, stop, strip-annotations, sub-process,
tee, trigger, use-mask, write-labels
writers (27):
console, delete-files, send-email, to-adams-ic, to-adams-od,
to-blue-channel-is, to-coco-od, to-combined-csv-od, to-csv-dp,
to-data, to-grayscale-dp, to-grayscale-is, to-indexed-png-is,
to-instance-png-od, to-layer-segments-is, to-metadata, to-multi,
to-numpy-dp, to-opex-od, to-pfm-dp, to-pyfunc, to-roicsv-od,
to-storage, to-subdir-ic, to-text-file, to-voc-od, to-yolo-od
options:
-h, --help Show basic help message and exit.
--help-all Show basic help message plus help on all plugins and exit.
--help-plugin NAME Show help message for plugin NAME and exit.
-u, --update_interval INTERVAL
Outputs the progress every INTERVAL records (default: 1000).
-l {DEBUG,INFO,WARNING,ERROR,CRITICAL}, --logging_level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
The logging level to use (default: WARN).
-b, --force_batch Processes the data in batches.
--variables FILE The file with custom variables to load (format: key=value).
--load_pipeline FILE The file to load the pipeline command from.
--dump_pipeline FILE The file to dump the pipeline command in.
usage: idc-exec [-h] --exec_generator GENERATOR [--exec_dry_run]
[--exec_prefix PREFIX] [--exec_variables FILE]
[--exec_format {cmdline,file}]
[--exec_logging_level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
...
Tool for executing a pipeline multiple times, each time with a different set
of variables expanded. A variable is surrounded by curly quotes (e.g.,
variable 'i' gets referenced with '{i}'). When supplying multiple generators,
then these get treated as nested executions. Available generators: csv-file,
dirs, files, list, null, prompt, range, text-file
positional arguments:
pipeline The pipeline template with variables to expand and
then execute; see '--exec_format' option.
options:
-h, --help show this help message and exit
--exec_generator GENERATOR
The generator plugin(s) to use, incl. their options.
Flag needs to be specified for each generator.
(default: None)
--exec_dry_run Applies the generator to the pipeline template and
only outputs it on stdout. (default: False)
--exec_prefix PREFIX The string to prefix the pipeline with when in dry-run
mode. (default: None)
--exec_variables FILE
The file with custom variables to load (format:
key=value). (default: None)
--exec_format {cmdline,file}
The format that the pipeline is in. The format
'cmdline' interprets the remaining arguments as the
pipeline arguments to execute. The format 'file'
expects a file to load the pipeline arguments from.
This file format allows spreading the pipeline
arguments over multiple lines: it simply joins all
lines into a single command-line before splitting it
into individual arguments for execution; any line
starting with # is interpreted as commend and removed
before joining. (default: cmdline)
--exec_logging_level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
The logging level to use. (default: WARN)
Readers tend to support input via file lists. The idc-find tool can generate
these.
usage: idc-find [-h] -i DIR [DIR ...] [-r] -o FILE [-m [REGEXP ...]]
[-n [REGEXP ...]] [--split_ratios [SPLIT_RATIOS ...]]
[--split_names [SPLIT_NAMES ...]]
[--split_name_separator SPLIT_NAME_SEPARATOR]
[-l {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
Tool for locating files in directories that match certain patterns and store
them in files.
options:
-h, --help show this help message and exit
-i DIR [DIR ...], --input DIR [DIR ...]
The dir(s) to scan for files. (default: None)
-r, --recursive Whether to search the directories recursively
(default: False)
-o FILE, --output FILE
The file to store the located file names in (default:
None)
-m [REGEXP ...], --match [REGEXP ...]
The regular expression that the (full) file names must
match to be included (default: None)
-n [REGEXP ...], --not-match [REGEXP ...]
The regular expression that the (full) file names must
match to be excluded (default: None)
--split_ratios [SPLIT_RATIOS ...]
The split ratios to use for generating the splits
(int; must sum up to 100) (default: None)
--split_names [SPLIT_NAMES ...]
The split names to use as filename suffixes for the
generated splits (before .ext) (default: None)
--split_name_separator SPLIT_NAME_SEPARATOR
The separator to use between file name and split name
(default: -)
-l {DEBUG,INFO,WARNING,ERROR,CRITICAL}, --logging_level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
The logging level to use. (default: WARN)
usage: idc-help [-h] [-c [PACKAGE ...]] [-e EXCLUDED_CLASS_LISTERS]
[-T {pipeline,generator}] [-p NAME] [-f {text,markdown}]
[-L INT] [-o PATH] [-i FILE] [-t TITLE]
[-l {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
Tool for outputting help for plugins in various formats.
options:
-h, --help show this help message and exit
-c [PACKAGE ...], --custom_class_listers [PACKAGE ...]
The custom class listers to use, uses the default ones
if not provided. (default: None)
-e EXCLUDED_CLASS_LISTERS, --excluded_class_listers EXCLUDED_CLASS_LISTERS
The comma-separated list of class listers to exclude.
(default: None)
-T {pipeline,generator}, --plugin_type {pipeline,generator}
The types of plugins to generate the help for.
(default: pipeline)
-p NAME, --plugin_name NAME
The name of the plugin to generate the help for,
generates it for all if not specified (default: None)
-f {text,markdown}, --help_format {text,markdown}
The output format to generate (default: text)
-L INT, --heading_level INT
The level to use for the heading (default: 1)
-o PATH, --output PATH
The directory or file to store the help in; outputs it
to stdout if not supplied; if pointing to a directory,
automatically generates file name from plugin name and
help format (default: None)
-i FILE, --index_file FILE
The file in the output directory to generate with an
overview of all plugins, grouped by type (in markdown
format, links them to the other generated files)
(default: None)
-t TITLE, --index_title TITLE
The title to use in the index file (default: image-
dataset-converter plugins)
-l {DEBUG,INFO,WARNING,ERROR,CRITICAL}, --logging_level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
The logging level to use. (default: WARN)
usage: idc-registry [-h] [-c CUSTOM_CLASS_LISTERS] [-e EXCLUDED_CLASS_LISTERS]
[-l {plugins,pipeline,custom-class-listers,env-class-listers,readers,filters,writers,generators}]
For inspecting/querying the registry.
options:
-h, --help show this help message and exit
-c CUSTOM_CLASS_LISTERS, --custom_class_listers CUSTOM_CLASS_LISTERS
The comma-separated list of custom class listers to
use. (default: None)
-e EXCLUDED_CLASS_LISTERS, --excluded_class_listers EXCLUDED_CLASS_LISTERS
The comma-separated list of class listers to exclude.
(default: None)
-l {plugins,pipeline,custom-class-listers,env-class-listers,readers,filters,writers,generators}, --list {plugins,pipeline,custom-class-listers,env-class-listers,readers,filters,writers,generators}
For outputting various lists on stdout. (default:
None)
usage: idc-test-generator [-h] -g GENERATOR
[-l {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
Tool for testing generators by outputting the generated variables and their
associated values. Available generators: csv-file, dirs, list, null, prompt,
range, text-file
options:
-h, --help show this help message and exit
-g GENERATOR, --exec_generator GENERATOR
The generator plugin(s) to use, incl. their options.
Flag needs to be specified for each generator.
(default: None)
-l {DEBUG,INFO,WARNING,ERROR,CRITICAL}, --logging_level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
The logging level to use. (default: WARN)
usage: idc-layer-segments [-h] -i DIR [DIR ...] [-m REGEXP] [-g REGEXP]
[-a {none,grayscale-stretch}] -I DIR [DIR ...]
[-M REGEXP] [-G REGEXP] [-A {none,binary}]
[-S {provided,base,annotation}] [-s SUFFIX]
[--suffix_regexp REGEXP] [--suffix_groups REGEXP] -o
DIR [-n] [-l {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
Creates layer segments annotations from PNG files with individual annotations.
Each individual annotation will get a different suffix.
options:
-h, --help show this help message and exit
-i DIR [DIR ...], --base_input DIR [DIR ...]
The dir(s) to scan for image files that form the base
image for the annotations; glob syntax is supported.
(default: None)
-m REGEXP, --base_match REGEXP
The regular expression group(s) to use for extracting
the new base name for the image (extension is removed
before matching). (default: (.*))
-g REGEXP, --base_format REGEXP
The name format to use for constructing the new base
name from the groups (no extension; group placeholder:
{X} with X being group 1, 2, etc). (default: {1})
-a {none,grayscale-stretch}, --base_aug {none,grayscale-stretch}
The augmentation to apply to the base image (default:
none)
-I DIR [DIR ...], --annotations_input DIR [DIR ...]
The dirs(s) to scan for image files that make up the
annotations; glob syntax is supported. (default: None)
-M REGEXP, --annotations_match REGEXP
The regular expression group(s) to use for extracting
the new annotation name for the image (extension is
removed before matching). (default: (.*))
-G REGEXP, --annotations_format REGEXP
The name format to use for constructing the new
annotation name from the groups (no extension; group
placeholder: {X} with X being group 1, 2, etc).
(default: {1})
-A {none,binary}, --annotations_aug {none,binary}
The augmentation to apply to the annotation image
(default: none)
-S {provided,base,annotation}, --suffix_source {provided,base,annotation}
How to obtain the suffix (ie label). (default:
provided)
-s SUFFIX, --suffix SUFFIX
The suffix (ie label) to use for the layers, 1-based
indexes get automatically appended. (default:
-object-)
--suffix_regexp REGEXP
The regular expression to apply to either base or
annotation filename (no ext), ignored when using the
provided one. (default: (.*))
--suffix_groups REGEXP
The expression for assembling the suffix from the
detected groups; group placeholder: {X} with X being
group 1, 2, etc) (default: -{1})
-o DIR, --output DIR The directory to store the cleaned up base image and
annotations in. (default: None)
-n, --dry_run Whether to only simulate the generation. (default:
False)
-l {DEBUG,INFO,WARNING,ERROR,CRITICAL}, --logging_level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
The logging level to use. (default: WARN)
You can find help screens for the plugins here:
- Pipeline plugins (readers/filters/writers)
- Generator plugins (used by
idc-exec) - Data formatter plugins
Examples can be found on the image-dataset-converter-examples website.
The image-dataset-converter uses the class lister registry provided by the seppl library.
Each module defines a function, typically called list_classes that returns
a dictionary of names of superclasses associated with a list of modules that
should be scanned for derived classes. Here is an example:
from typing import List, Dict
def list_classes() -> Dict[str, List[str]]:
return {
"seppl.io.Reader": [
"mod.ule1",
"mod.ule2",
],
"seppl.io.Filter": [
"mod.ule3",
"mod.ule4",
],
"seppl.io.Writer": [
"mod.ule5",
],
}Such a class lister gets referenced in the entry_points section of the setup.py file:
entry_points={
"class_lister": [
"unique_string=module_name:function_name",
],
},:function_name can be omitted if :list_classes.
The following environment variables can be used to influence the class listers:
IDC_CLASS_LISTERSIDC_CLASS_LISTERS_EXCLIDC_CLASS_LISTERS_IGNORED- for class listers that provide ignored classes
Each variable is a comma-separated list of module_name:function_name, defining the class listers.
Whenever possible, images get copied rather than read and then rewritten, to avoid loss of quality by subsequent read/write operations. To minimize introduction of artifacts in JPEG images, a default quality of 90% is used, which is a good compromise between size and quality.
However, this quality parameter can be overridden globally with the following environment variable:
IDC_JPEG_QUALITY
For instance, the following setting would use a quality of 100%:
IDC_JPEG_QUALITY=100
In order to avoid strange effects from implicit EXIF rotations and having
to explicitly specify the exif-autorotate filter in the pipeline, you
can define the following environment variable to automatically apply
any EXIF rotation information when loading images using the
load_image_from_bytes and load_image_from_file functions
(module idc.api) as well as the image size determination of the
Image class (module idc.api):
IDC_EXIF_AUTOROTATE=true
The value of the variable is case-insensitive can be on or yes as well.
In order to speed up plugin discovery, they discovered plugins can be cached on disk after the initial discovery. Installing additional plugins after the cache has been initialized will not make them visible, the cache will require resetting first.
The cache can be managed through the following environment variable:
IDC_CLASS_CACHE
It supports the following options:
off: disables the cacheon: enables the cachereset: resets the cached plugins first and enables the cache