Out of memory error when running replace_identifiers on large number of files

[lbicknes-umn]

I am working with a team that is evaluating using this tool to de-identify DICOMs. The current script that we are testing with is essentially the following:

dicom_files = glob.glob(os.path.join(input_folder, "**", "*.dcm"), recursive=True)

items = get_identifiers(dicom_files)

# Clean the files
replace_identifiers(
        dicom_files=dicom_files,
        deid=recipe,
        strip_sequences=False, 
        ids=items,
        remove_private=True, 
        output_folder=output_folder, 
        overwrite=True,
        save=True,
)

Some of the operations that we will need have been removed in an attempt to get the script to complete (i.e., custom functions on the "items"). I have not included the recipe file, but using the default recipe results in the same behavior.

We have a large number of files in the input directory (e.g., 4-6 GB of total data), the script will consume 20+ GB of RAM. I also attempted to use the branch that skips the pixel data, and that did not change the behavior.

If a loop is created, and each DICOM file is cleaned individually with a manual delay it seems that the garbage collector will run and the total amount of memory is much lower. It seems that there is a memory leak.

I have attempted to profile the code and library without much luck.

Has this been observed by any other users? Is there a workaround? Possibility of a fix?

[vsoch]

I think the thing to do here is process your data in batches. It doesn't make sense to read in the entire massive amount of data into memory at once - logically that would lead to an error.

[lbicknes-umn]

@vsoch Is there a recommended batch approach?

I attempted to iterate over the each DICOM individually, but still ran into memory issues unless I added a delay between each file which seemed to allow garbage collection to run. It should be no problem batching the files, but I'm wondering what the best approach is. This is probably a more general Python question, but any help is appreciated.

[vsoch]

There are probably many ways to do it - typically I wold write a "chunked" function or similar. It looks like Python 3.12 added itertools.batched so maybe give that a shot first.

[lbicknes-umn]

I attempted the batched approach, but had the same issue.

I installed and profiled the memory using Fil: https://pythonspeed.com/fil/docs/index.html, and was able to trace to the following:

deid/deid/dicom/fields.py

Line 490 in 8d26fca

@cache

@cache
def _get_fields_inner(dicom, skip=None, expand_sequences=True, seen=None):

This method is using a memoization function that has an unbounded, in-memory cache. If the cache decorator is removed, the process runs with constant memory usage.

I wonder if it is possible to disable and/or bound the cache based on configuration. Does that sound like a feature that could be supported?

[vsoch]

Yes, that sounds reasonable. If you'd like to open a pull request that makes it optional (enabled by default) I'd be happy to review it.

[lbicknes-umn]

I've created a fork of the repository to make my changes: https://github.com/lbicknes-umn/deid/tree/configurable-caching. I'm not ready to create a PR, as I'd like to add some tests and documentation to support my changes.

I have a working PoC with cache configuration settings. Batching is still necessary for a large number of DICOMs.

However, in my testing it seems that the caching may not be working. I added debug output for the cache_info() of the decorated function, and there are no cache hits (there are incrementing misses, so I assume that I've done this correctly). If you add the following to the get_fields_with_lookup method on "fields.py" it demonstrates.

bot.warning(f"cache: {_get_fields_inner.cache_info()}")

I believe there should be cache hits with my test case, but it would be good to have someone else confirm the behavior.

[ahmetcakir]

I ran into the same issue and and found calling _get_fields_inner.cache_clear() every 100 files or so to have fixed it. In my example processing about 100 dicom files lead to a memory increase of about 400MB. See below

 === Memory growth at  200th file compared to 100th  ===
  pydicom/fileutil.py:342: size=408 MiB (+408 MiB), count=49 (+49)