diff --git a/docs/sphinx/features/file_output.rst b/docs/sphinx/features/file_output.rst index 34d9c7006..2ac4650f5 100644 --- a/docs/sphinx/features/file_output.rst +++ b/docs/sphinx/features/file_output.rst @@ -20,9 +20,14 @@ output is written: The values of these variables are used to construct unique filenames for output. The extension ``.log`` is used for logging output, and ``.stats`` for replay output. The filenames additionally contain the process ID and a unique -integer that is used to make multiple files with the same basename. -This ensures that multiple runs with the same IO configuration do not -overwrite files. +integer that is used to make multiple files with the same basename. This +ensures that multiple runs with the same IO configuration do not overwrite +files. + +Replay ``.stats`` files use the replay-v2 JSONL format: one header record +followed by ``make_allocator``, ``allocate``, and ``deallocate`` command +records, each with ``pending`` and ``committed`` lifecycle states when the +operation succeeds. The format of the filenames is: diff --git a/docs/sphinx/features/logging_and_replay.rst b/docs/sphinx/features/logging_and_replay.rst index 84ab59832..24152b471 100644 --- a/docs/sphinx/features/logging_and_replay.rst +++ b/docs/sphinx/features/logging_and_replay.rst @@ -1,39 +1,45 @@ .. _logging_and_replay: =================================== -Logging and Replay of Umpire Events +Logging and Replay of Umpire Events =================================== Logging ------- When debugging memory operation problems, it is sometimes helpful to enable -Umpire's logging facility. The logging functionality is enabled for default -builds unless ``-DUMPIRE_ENABLE_LOGGING=Off`` has been specified in which case it is -disabled. +Umpire's logging facility. The logging functionality is enabled for default +builds unless ``-DUMPIRE_ENABLE_LOGGING=Off`` has been specified, in which case +it is disabled. If Umpire logging is enabled, it may be controlled by setting the ``UMPIRE_LOG_LEVEL`` environment variable to ``Error``, ``Warning``, ``Info``, -or ``Debug``. The ``Debug`` value is the most verbose. +or ``Debug``. The ``Debug`` value is the most verbose. When ``UMPIRE_LOG_LEVEL`` has been set, events will be logged to the standard output. Replay -------- -Umpire provides a lightweight replay capability that can be used to investigate -performance of particular allocation patterns and reproduce bugs. By running an -executable that uses Umpire with the environment variable ``UMPIRE_REPLAY`` set -to ``On``, Umpire will emit information for the following Umpire events: - -- **version** - :func:`umpire::get_major_version`, - :func:`umpire::get_minor_version`, - and :func:`umpire::get_patch_version` -- **makeMemoryResource** - :func:`umpire::resource::MemoryResourceRegistry::makeMemoryResource` -- **makeAllocator** :func:`umpire::ResourceManager::makeAllocator` -- **allocate** :func:`umpire::Allocator::allocate` -- **deallocate** :func:`umpire::Allocator::deallocate` +------ +Umpire provides a replay capability that is focused on reproducibility. By +running an executable that uses Umpire with the environment variable +``UMPIRE_REPLAY`` set to ``On``, Umpire will emit a replay-v2 trace that can be +used later to recreate allocator construction, allocations, and deallocations +independent of the original application. + +Replay-v2 records attempted calls to: + +- built-in memory resource construction, serialized as ``make_allocator`` with + ``strategy`` set to ``MemoryResource`` +- :func:`umpire::ResourceManager::makeAllocator` +- :func:`umpire::Allocator::allocate` +- :func:`umpire::Allocator::deallocate` + +The output is newline-delimited JSON. The first line is a header that +identifies the schema version. Each later line is a replay command. Successful +operations are recorded as a lightweight two-line lifecycle: a ``pending`` +record written before execution and a ``committed`` record written after +success. If the process fails during an operation, the trace may end with a +trailing ``pending`` record. Running with Replay ------------------- @@ -43,96 +49,101 @@ To enable Umpire replay, one may execute as follows: UMPIRE_REPLAY="On" ./my_umpire_using_program -will write Umpire replay events to a file with a name like +This will write replay output to a file with a name like ``umpire...stats`` in the directory specified by -``UMPIRE_OUTPUT_DIR`` (or in the current directory if it is not set). The -resulting ``.stats`` file contains JSON-formatted lines with the following -kinds of information: +``UMPIRE_OUTPUT_DIR`` (or in the current directory if it is not set). -Interpretting Results - Version Event -------------------------------------- -The first event captured is the **version** event which shows the version -information. The example below is taken from the file -``examples/tutorial/tut_replay_log.json``, which contains the contents of a -replay ``.stats`` file in JSON format: +Interpreting Results - Header +----------------------------- +The first line in the trace is a replay header. The example below is taken from +``examples/tutorial/tut_replay_log.json``: .. literalinclude:: ../../../examples/tutorial/tut_replay_log.json - :start-after: _sphinx_tag_doc_version_start - :end-before: _sphinx_tag_doc_version_end + :start-after: _sphinx_tag_doc_header_start + :end-before: _sphinx_tag_doc_header_end :language: json -Each line contains the following set of common elements: +The header contains: **kind** - Always set to ``replay`` - -**uid** - This is the MPI rank of the process generating the event for - mpi programs or the PID for non-mpi. - -**timestamp** - Set to the time when the event occurred. + Always set to ``umpire_replay`` -**event** - Set to one of: ``version``, ``makeMemoryResource``, - ``makeAllocator``, ``allocate``, or ``deallocate`` +**schema** + The replay schema version. The current format is ``v2``. -**payload** - Optional and varies upon event type +**process** + Metadata about the process that generated the trace. This includes the OS + process ID and the MPI rank when MPI is enabled. -**result** - Optional and varies upon event type - -As can be seen, the *major*, *minor*, and *patch* version numbers are captured -within the *payload* for this event. +**umpire_version** + The Umpire version that produced the replay trace. makeMemoryResource Event ------------------------ -Next you will see events for the creation of the default memory resources -provided by Umpire with the **makeMemoryResource** event: +Built-in memory resources are recorded as ``make_allocator`` commands using the +``MemoryResource`` strategy: .. literalinclude:: ../../../examples/tutorial/tut_replay_log.json :start-after: _sphinx_tag_doc_makememoryresource_start :end-before: _sphinx_tag_doc_makememoryresource_end :language: json -The *payload* shows that a memory resource was created for *HOST*, *DEVICE*, -*PINNED*, *UM*, and *DEVICE_CONST* respectively. Note that this could also -be done with the *FILE* memory resource. The *result* is a reference -to the object that was created within Umpire for that resource. +The command includes: + +**seq** + A monotonically increasing sequence number used to preserve operation order. + +**allocator_id** + A stable allocator identity within the trace. + +**name** + The allocator or resource name that will be visible through the + :class:`umpire::Allocator`. + +**strategy** + ``MemoryResource`` for built-in memory resources. + +**tracking** + Whether the allocator was created with tracking enabled. + +**args** + Constructor arguments. For memory resources this includes the + ``resource_name`` and serialized :class:`umpire::MemoryResourceTraits`. + +**status** + ``pending`` before execution and ``committed`` after success. makeAllocator Event ------------------- -The **makeAllocator** event occurs whenever a new allocator instance is being -created. Each call to **makeAllocator** will generate a pair of JSON lines. The -first line will show the intent of the call and the second line will show both -the intent and the result. This is because the makeAllocator call can fail -and keeping both the intent and result allows us to reproduce this failure -later. - -:class:`umpire::Allocator`: +Each call to :func:`umpire::ResourceManager::makeAllocator` records a +``pending`` command before execution and, on success, a matching +``committed`` command. The example below shows a +:class:`umpire::strategy::QuickPool` construction: .. literalinclude:: ../../../examples/tutorial/tut_replay_log.json :start-after: _sphinx_tag_doc_makeallocator_start :end-before: _sphinx_tag_doc_makeallocator_end :language: json -The *payload* shows how the allocator was constructed. The *result* shows the -reference to the allocated object. +The ``args`` object contains the serialized constructor arguments for the +strategy. In this example, ``parent_allocator`` refers back to the previously +constructed ``HOST`` resource by allocator ID, and the pool parameters are +recorded explicitly so that replay can rebuild the same allocator. allocate Event -------------- -Like the **makeAllocator** event, the **allocate** event is captured as an -intention/result pair so that an error may be replayed in the event that -there is an allocation failure. +Each allocation attempt records a ``pending`` command and, on success, a +matching ``committed`` command with the allocator used, a stable allocation +ID, and the requested allocation size: .. literalinclude:: ../../../examples/tutorial/tut_replay_log.json :start-after: _sphinx_tag_doc_allocate_start :end-before: _sphinx_tag_doc_allocate_end :language: json -The *payload* shows the object reference of the allocator and the size of the -allocation request. The *result* shows the pointer to the memory allocated. +Replay uses ``allocation_id`` rather than the original process pointer value. +This keeps the trace stable across runs while still preserving allocation +lifetime. deallocate Event ---------------- @@ -141,15 +152,26 @@ deallocate Event :end-before: _sphinx_tag_doc_deallocate_end :language: json -The *payload* shows the reference to the allocator object and the pointer -to the allocated memory that is to be freed. +The ``deallocate`` command uses the same ``pending``/``committed`` lifecycle +and references the same ``allocator_id`` and ``allocation_id`` so that the +replay runtime can free the correct allocation. Replaying the session --------------------- -Loading the ``.stats`` file with the ``replay`` program will replay this -sequence of :class:`umpire::Allocator` creation, allocations, and -deallocations: +Loading the ``.stats`` file with the ``replay`` program will replay the +committed subset of this sequence of :class:`umpire::Allocator` creation, +allocations, and deallocations. Trailing ``pending`` records are preserved as +failure evidence but are not executed. .. code-block:: bash ./bin/replay -i umpire...stats + +To emit a ULTRA trace from the replayed allocator state, use: + +.. code-block:: bash + + ./bin/replay -d -i umpire...stats + +This writes ``replay.ult`` with allocator statistics sampled after each +replay command. That file can be visualized with ULTRA tooling. diff --git a/docs/sphinx/tutorial/replay.rst b/docs/sphinx/tutorial/replay.rst index 909081592..847ab487a 100644 --- a/docs/sphinx/tutorial/replay.rst +++ b/docs/sphinx/tutorial/replay.rst @@ -3,15 +3,17 @@ ====== Replay ====== -Umpire provides a lightweight replay capability that can be used to investigate -performance of particular allocation patterns and reproduce bugs. +Umpire provides a replay capability for reproducing allocator behavior and +debugging Umpire issues independent of the original application. Input Example ------------- -When replay is enabled, Umpire captures replay events and writes them as -JSON-formatted lines into a ``.stats`` file. This file can be used as input to -the ``replay`` application (available under the ``bin`` directory), which will -recreate the events that occurred as part of the run that generated the log. +When replay is enabled, Umpire writes replay-v2 JSONL into a ``.stats`` file. +The first line is a header and each later line is an operation record with a +``pending`` or ``committed`` lifecycle status. This file can be used as input +to the ``replay`` application (available under the ``bin`` directory), which +reconstructs committed allocator creation, allocation, and deallocation +activity from the recorded trace. The file ``tut_replay.cpp`` makes a :class:`umpire::strategy::QuickPool`: @@ -41,16 +43,29 @@ Running this program: UMPIRE_REPLAY="On" ./bin/examples/tutorial/tut_replay -will write Umpire replay events to a file with a name like +will write replay output to a file with a name like ``umpire...stats`` in the current directory (or in the directory -specified by ``UMPIRE_OUTPUT_DIR``). This file contains JSON formatted lines. +specified by ``UMPIRE_OUTPUT_DIR``). A minimal example looks like: + +.. literalinclude:: ../../../examples/tutorial/tut_replay_log.json + :start-after: _sphinx_tag_doc_header_start + :end-before: _sphinx_tag_doc_deallocate_end + :language: json Replaying the session --------------------- Loading this ``.stats`` file with the ``replay`` program will replay this -sequence of :class:`umpire::Allocator` creation, allocations, and +committed sequence of :class:`umpire::Allocator` creation, allocations, and deallocations: .. code-block:: bash ./bin/replay -i umpire...stats + +To also generate a ULTRA trace from the replayed allocator state: + +.. code-block:: bash + + ./bin/replay -d -i umpire...stats + +This writes ``replay.ult`` in the working directory. diff --git a/examples/tutorial/tut_replay_log.json b/examples/tutorial/tut_replay_log.json index 2121bc651..7c07f0729 100644 --- a/examples/tutorial/tut_replay_log.json +++ b/examples/tutorial/tut_replay_log.json @@ -4,213 +4,22 @@ # # SPDX-License-Identifier: (MIT) ############################################################################## -# _sphinx_tag_doc_version_start -{ "kind":"replay", "uid":27494, "timestamp":1558388052211435757, "event": "version", "payload": { "major":0, "minor":3, "patch":3 } } -# _sphinx_tag_doc_version_end +# _sphinx_tag_doc_header_start +{ "kind":"umpire_replay", "process": { "pid":59079, "rank":0 }, "schema":"v2", "umpire_version":"2025.12.0-bd2af093" } +# _sphinx_tag_doc_header_end # _sphinx_tag_doc_makememoryresource_start -{ "kind":"replay", "uid":27494, "timestamp":1558388052211477678, "event": "makeMemoryResource", "payload": { "name": "HOST" }, "result": "0x101626b0" } -{ "kind":"replay", "uid":27494, "timestamp":1558388052471684134, "event": "makeMemoryResource", "payload": { "name": "DEVICE" }, "result": "0x101d79a0" } -{ "kind":"replay", "uid":27494, "timestamp":1558388052471698804, "event": "makeMemoryResource", "payload": { "name": "PINNED" }, "result": "0x101d7a50" } -{ "kind":"replay", "uid":27494, "timestamp":1558388052472972935, "event": "makeMemoryResource", "payload": { "name": "UM" }, "result": "0x101d7b00" } -{ "kind":"replay", "uid":27494, "timestamp":1558388052595814979, "event": "makeMemoryResource", "payload": { "name": "DEVICE_CONST" }, "result": "0x101d7bb0" } +{ "allocator_id":"a1", "args": { "resource_name":"HOST", "traits": { "granularity":"unknown", "ipc":false, "kind":"unknown", "resource":"host", "scope":"unknown", "size":0, "tracking":true, "unified":false, "used_for":"any", "vendor":"unknown" } }, "name":"HOST", "op":"make_allocator", "seq":1, "status":"pending", "strategy":"MemoryResource", "tracking":true } +{ "allocator_id":"a1", "args": { "resource_name":"HOST", "traits": { "granularity":"unknown", "ipc":false, "kind":"unknown", "resource":"host", "scope":"unknown", "size":0, "tracking":true, "unified":false, "used_for":"any", "vendor":"unknown" } }, "name":"HOST", "op":"make_allocator", "seq":1, "status":"committed", "strategy":"MemoryResource", "tracking":true } # _sphinx_tag_doc_makememoryresource_end # _sphinx_tag_doc_makeallocator_start -{ "kind":"replay", "uid":27494, "timestamp":1558388052595864262, "event": "makeAllocator", "payload": { "type":"umpire::strategy::QuickPool", "with_introspection":true, "allocator_name":"pool", "args": [ "HOST" ] } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052595903505, "event": "makeAllocator", "payload": { "type":"umpire::strategy::QuickPool", "with_introspection":true, "allocator_name":"pool", "args": [ "HOST" ] }, "result": { "allocator_ref":"0x108a8730" } } +{ "allocator_id":"a2", "args": { "alignment":16, "heuristic": { "kind":"percent_releasable_hwm", "parameter":100 }, "initial_alloc_size":536870912, "min_alloc_size":1048576, "parent_allocator":"a1" }, "name":"pool", "op":"make_allocator", "seq":2, "status":"pending", "strategy":"QuickPool", "tracking":true } +{ "allocator_id":"a2", "args": { "alignment":16, "heuristic": { "kind":"percent_releasable_hwm", "parameter":100 }, "initial_alloc_size":536870912, "min_alloc_size":1048576, "parent_allocator":"a1" }, "name":"pool", "op":"make_allocator", "seq":2, "status":"committed", "strategy":"QuickPool", "tracking":true } # _sphinx_tag_doc_makeallocator_end # _sphinx_tag_doc_allocate_start -{ "kind":"replay", "uid":27494, "timestamp":1558388052595911583, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 0 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052595934822, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 0 }, "result": { "memory_ptr": "0x200040000010" } } +{ "allocation_id":"m1", "allocator_id":"a2", "op":"allocate", "seq":3, "size":32, "status":"pending" } +{ "allocation_id":"m1", "allocator_id":"a2", "op":"allocate", "seq":3, "size":32, "status":"committed" } # _sphinx_tag_doc_allocate_end -{ "kind":"replay", "uid":27494, "timestamp":1558388052595939623, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 134 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052595943793, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 134 }, "result": { "memory_ptr": "0x200040000010" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052595947408, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 774 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052595951548, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 774 }, "result": { "memory_ptr": "0x2000400000a0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052595954839, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 470 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052595958585, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 470 }, "result": { "memory_ptr": "0x2000400003b0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052595961448, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 546 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052595964866, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 546 }, "result": { "memory_ptr": "0x200040000590" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052595968229, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 224 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052595971700, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 224 }, "result": { "memory_ptr": "0x2000400007c0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052595974217, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 48 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052595977571, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 48 }, "result": { "memory_ptr": "0x2000400008a0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052595980115, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 695 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052595983332, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 695 }, "result": { "memory_ptr": "0x2000400008d0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052595985897, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 696 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052595989105, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 696 }, "result": { "memory_ptr": "0x200040000b90" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052595992121, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 958 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052595995576, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 958 }, "result": { "memory_ptr": "0x200040000e50" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052595998142, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 393 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596001380, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 393 }, "result": { "memory_ptr": "0x200040001210" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596003957, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 532 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596007251, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 532 }, "result": { "memory_ptr": "0x2000400013a0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596009753, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 851 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596013019, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 851 }, "result": { "memory_ptr": "0x2000400015c0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596015528, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 35 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596018772, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 35 }, "result": { "memory_ptr": "0x200040001920" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596021272, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 54 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596024513, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 54 }, "result": { "memory_ptr": "0x200040001950" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596026971, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 542 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596030176, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 542 }, "result": { "memory_ptr": "0x200040001990" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596032678, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 687 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596035862, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 687 }, "result": { "memory_ptr": "0x200040001bb0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596038953, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 7 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596042471, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 7 }, "result": { "memory_ptr": "0x200040001e60" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596045123, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 393 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596048644, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 393 }, "result": { "memory_ptr": "0x200040001e70" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596051129, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 68 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596054420, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 68 }, "result": { "memory_ptr": "0x200040002000" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596056892, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 427 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596088723, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 427 }, "result": { "memory_ptr": "0x200040002050" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596091516, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 703 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596094828, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 703 }, "result": { "memory_ptr": "0x200040002200" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596097358, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 603 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596100500, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 603 }, "result": { "memory_ptr": "0x2000400024c0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596102896, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 953 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596106176, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 953 }, "result": { "memory_ptr": "0x200040002720" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596108605, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 867 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596111800, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 867 }, "result": { "memory_ptr": "0x200040002ae0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596114191, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 540 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596117437, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 540 }, "result": { "memory_ptr": "0x200040002e50" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596119829, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 94 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596122989, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 94 }, "result": { "memory_ptr": "0x200040003070" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596125446, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 670 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596128589, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 670 }, "result": { "memory_ptr": "0x2000400030d0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596130974, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 426 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596134139, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 426 }, "result": { "memory_ptr": "0x200040003370" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596136522, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 718 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596139735, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 718 }, "result": { "memory_ptr": "0x200040003520" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596142133, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 933 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596145374, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 933 }, "result": { "memory_ptr": "0x2000400037f0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596147801, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 781 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596151133, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 781 }, "result": { "memory_ptr": "0x200040003ba0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596153537, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 269 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596156818, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 269 }, "result": { "memory_ptr": "0x200040003eb0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596159832, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 48 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596164931, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 48 }, "result": { "memory_ptr": "0x200040003fc0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596167453, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 754 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596170902, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 754 }, "result": { "memory_ptr": "0x200040003ff0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596173335, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 336 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596178062, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 336 }, "result": { "memory_ptr": "0x2000400042f0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596180524, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 648 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596183884, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 648 }, "result": { "memory_ptr": "0x200040004440" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596186294, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 775 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596190741, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 775 }, "result": { "memory_ptr": "0x2000400046d0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596193133, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 1015 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596196612, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 1015 }, "result": { "memory_ptr": "0x2000400049e0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596199032, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 374 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596202451, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 374 }, "result": { "memory_ptr": "0x200040004de0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596204828, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 253 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596209326, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 253 }, "result": { "memory_ptr": "0x200040004f60" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596211709, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 1007 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596215287, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 1007 }, "result": { "memory_ptr": "0x200040005060" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596217711, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 740 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596221090, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 740 }, "result": { "memory_ptr": "0x200040005450" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596223414, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 772 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596227831, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 772 }, "result": { "memory_ptr": "0x200040005740" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596230322, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 667 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596238989, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 667 }, "result": { "memory_ptr": "0x200040005a50" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596241491, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 74 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596244960, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 74 }, "result": { "memory_ptr": "0x200040005cf0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596247399, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 647 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596250794, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 647 }, "result": { "memory_ptr": "0x200040005d40" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596253208, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 906 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596257745, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 906 }, "result": { "memory_ptr": "0x200040005fd0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596260264, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 279 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596263920, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 279 }, "result": { "memory_ptr": "0x200040006360" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596266522, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 447 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596270064, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 447 }, "result": { "memory_ptr": "0x200040006480" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596272543, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 785 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596276015, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 785 }, "result": { "memory_ptr": "0x200040006640" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596278386, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 489 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596281878, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 489 }, "result": { "memory_ptr": "0x200040006960" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596284273, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 243 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596292646, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 243 }, "result": { "memory_ptr": "0x200040006b50" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596295173, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 281 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596298552, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 281 }, "result": { "memory_ptr": "0x200040006c50" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596301005, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 368 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596304407, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 368 }, "result": { "memory_ptr": "0x200040006d70" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596306870, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 170 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596310174, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 170 }, "result": { "memory_ptr": "0x200040006ee0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596312627, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 498 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596315891, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 498 }, "result": { "memory_ptr": "0x200040006f90" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596318262, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 920 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596321545, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 920 }, "result": { "memory_ptr": "0x200040007190" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596323957, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 931 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596327326, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 931 }, "result": { "memory_ptr": "0x200040007530" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596329810, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 62 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596333127, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 62 }, "result": { "memory_ptr": "0x2000400078e0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596335498, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 927 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596338824, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 927 }, "result": { "memory_ptr": "0x200040007920" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596341288, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 517 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596344527, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 517 }, "result": { "memory_ptr": "0x200040007cc0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596346898, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 529 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596350263, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 529 }, "result": { "memory_ptr": "0x200040007ed0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596352741, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 327 } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596356087, "event": "allocate", "payload": { "allocator_ref": "0x108a8730", "size": 327 }, "result": { "memory_ptr": "0x2000400080f0" } } # _sphinx_tag_doc_deallocate_start -{ "kind":"replay", "uid":27494, "timestamp":1558388052596358577, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040000010" } } +{ "allocation_id":"m1", "allocator_id":"a2", "op":"deallocate", "seq":4, "status":"pending" } +{ "allocation_id":"m1", "allocator_id":"a2", "op":"deallocate", "seq":4, "status":"committed" } # _sphinx_tag_doc_deallocate_end -{ "kind":"replay", "uid":27494, "timestamp":1558388052596364727, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040000010" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596369203, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x2000400000a0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596374324, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x2000400003b0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596379033, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040000590" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596383834, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x2000400007c0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596388492, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x2000400008a0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596392896, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x2000400008d0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596397388, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040000b90" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596401782, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040000e50" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596409895, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040001210" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596414264, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x2000400013a0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596418709, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x2000400015c0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596423008, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040001920" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596427403, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040001950" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596431787, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040001990" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596436148, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040001bb0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596440496, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040001e60" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596444820, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040001e70" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596449333, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040002000" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596453572, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040002050" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596457927, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040002200" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596462097, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x2000400024c0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596466313, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040002720" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596470559, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040002ae0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596474663, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040002e50" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596479030, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040003070" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596483201, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x2000400030d0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596487434, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040003370" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596491648, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040003520" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596495711, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x2000400037f0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596499738, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040003ba0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596503806, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040003eb0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596507990, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040003fc0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596511962, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040003ff0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596516183, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x2000400042f0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596520138, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040004440" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596524030, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x2000400046d0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596527932, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x2000400049e0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596531932, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040004de0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596535889, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040004f60" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596540092, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040005060" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596543998, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040005450" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596547855, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040005740" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596551670, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040005a50" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596555488, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040005cf0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596559216, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040005d40" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596562947, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040005fd0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596566857, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040006360" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596570669, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040006480" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596574478, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040006640" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596578141, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040006960" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596581794, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040006b50" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596585491, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040006c50" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596589110, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040006d70" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596592786, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040006ee0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596596539, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040006f90" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596600299, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040007190" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596603930, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040007530" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596611095, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x2000400078e0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596614796, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040007920" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596618390, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040007cc0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596622021, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x200040007ed0" } } -{ "kind":"replay", "uid":27494, "timestamp":1558388052596625724, "event": "deallocate", "payload": { "allocator_ref": "0x108a8730", "memory_ptr": "0x2000400080f0" } } diff --git a/src/umpire/Allocator.cpp b/src/umpire/Allocator.cpp index 2127138d6..d3cee60e3 100644 --- a/src/umpire/Allocator.cpp +++ b/src/umpire/Allocator.cpp @@ -87,6 +87,12 @@ strategy::AllocationStrategy* Allocator::getAllocationStrategy() noexcept return m_allocator; } +strategy::AllocationStrategy* Allocator::getAllocationStrategy() const noexcept +{ + UMPIRE_LOG(Debug, "() returning " << m_allocator); + return m_allocator; +} + Platform Allocator::getPlatform() noexcept { return m_allocator->getPlatform(); diff --git a/src/umpire/Allocator.hpp b/src/umpire/Allocator.hpp index cc53d440a..6824255dd 100644 --- a/src/umpire/Allocator.hpp +++ b/src/umpire/Allocator.hpp @@ -172,6 +172,7 @@ class Allocator : private strategy::mixins::Inspector, strategy::mixins::Allocat * \return Pointer to the AllocationStrategy. */ strategy::AllocationStrategy* getAllocationStrategy() noexcept; + strategy::AllocationStrategy* getAllocationStrategy() const noexcept; /*! * \brief Get the Platform object appropriate for this Allocator. diff --git a/src/umpire/Allocator.inl b/src/umpire/Allocator.inl index ee4f31f5c..bbff1bfcb 100644 --- a/src/umpire/Allocator.inl +++ b/src/umpire/Allocator.inl @@ -9,8 +9,7 @@ #include "umpire/Allocator.hpp" #include "umpire/config.hpp" -#include "umpire/event/event.hpp" -#include "umpire/event/recorder_factory.hpp" +#include "umpire/event/operation_recording.hpp" #include "umpire/strategy/ThreadSafeAllocator.hpp" #include "umpire/util/Macros.hpp" #include "umpire/util/error.hpp" @@ -19,32 +18,31 @@ namespace umpire { inline void* Allocator::do_allocate(std::size_t bytes) { - void* ret = nullptr; - UMPIRE_ASSERT(UMPIRE_VERSION_OK()); UMPIRE_LOG(Debug, "(" << bytes << ")"); - if (0 == bytes) { - ret = allocateNull(); - } else { - try { - ret = m_allocator->allocate(bytes); - } catch (umpire::out_of_memory_error& e) { - e.set_allocator_id(this->getId()); - e.set_requested_size(bytes); - throw; - } - } + return umpire::event::record_allocate(m_allocator, bytes, [&]() -> void* { + void* ret = nullptr; - if (m_tracking) { - registerAllocation(ret, bytes, m_allocator); - } + if (0 == bytes) { + ret = allocateNull(); + } else { + try { + ret = m_allocator->allocate(bytes); + } catch (umpire::out_of_memory_error& e) { + e.set_allocator_id(this->getId()); + e.set_requested_size(bytes); + throw; + } + } - umpire::event::record( - [&](auto& event) { event.size(bytes).ref((void*)m_allocator).ptr(ret); }); + if (m_tracking) { + registerAllocation(ret, bytes, m_allocator); + } - return ret; + return ret; + }); } inline void* Allocator::thread_safe_allocate(std::size_t bytes) @@ -79,95 +77,94 @@ inline void Allocator::thread_safe_resource_deallocate(void* ptr, camp::resource inline void* Allocator::do_named_allocate(const std::string& name, std::size_t bytes) { - void* ret = nullptr; - UMPIRE_ASSERT(UMPIRE_VERSION_OK()); UMPIRE_LOG(Debug, "(" << bytes << ")"); - if (0 == bytes) { - ret = allocateNull(); - } else { - ret = m_allocator->allocate_named(name, bytes); - } + return umpire::event::record_named_allocate(m_allocator, name, bytes, [&]() -> void* { + void* ret = nullptr; + + if (0 == bytes) { + ret = allocateNull(); + } else { + ret = m_allocator->allocate_named(name, bytes); + } - if (m_tracking) { - registerAllocation(ret, bytes, m_allocator, name); - } + if (m_tracking) { + registerAllocation(ret, bytes, m_allocator, name); + } - umpire::event::record( - [&](auto& event) { event.name(name).size(bytes).ref((void*)m_allocator).ptr(ret); }); - return ret; + return ret; + }); } inline void* Allocator::do_resource_allocate(std::size_t bytes, camp::resources::Resource const& r) { - void* ret = nullptr; - UMPIRE_ASSERT(UMPIRE_VERSION_OK()); UMPIRE_LOG(Debug, "(" << bytes << ")"); - if (0 == bytes) { - ret = allocateNull(); - } else { - ret = m_allocator->allocate_resource(bytes, r); - } + return umpire::event::record_resource_allocate(m_allocator, bytes, r, [&]() -> void* { + void* ret = nullptr; + + if (0 == bytes) { + ret = allocateNull(); + } else { + ret = m_allocator->allocate_resource(bytes, r); + } - if (m_tracking) { - registerAllocation(ret, bytes, m_allocator); - } + if (m_tracking) { + registerAllocation(ret, bytes, m_allocator); + } - umpire::event::record( - [&](auto& event) { event.size(bytes).ref((void*)m_allocator).ptr(ret).res(camp::resources::to_string(r)); }); - return ret; + return ret; + }); } inline void Allocator::do_deallocate(void* ptr) { - umpire::event::record([&](auto& event) { event.ref((void*)m_allocator).ptr(ptr); }); - UMPIRE_LOG(Debug, "(" << ptr << ")"); - if (!ptr) { - UMPIRE_LOG(Info, "Deallocating a null pointer (This behavior is intentionally allowed and ignored)"); - return; - } else { - if (m_tracking) { - auto record = deregisterAllocation(ptr, m_allocator); - if (!deallocateNull(ptr)) { - m_allocator->deallocate(ptr, record.size); - } + umpire::event::record_deallocate(m_allocator, ptr, [&]() { + if (!ptr) { + UMPIRE_LOG(Info, "Deallocating a null pointer (This behavior is intentionally allowed and ignored)"); + return; } else { - if (!deallocateNull(ptr)) { - m_allocator->deallocate(ptr); + if (m_tracking) { + auto record = deregisterAllocation(ptr, m_allocator); + if (!deallocateNull(ptr)) { + m_allocator->deallocate(ptr, record.size); + } + } else { + if (!deallocateNull(ptr)) { + m_allocator->deallocate(ptr); + } } } - } + }); } inline void Allocator::do_resource_deallocate(void* ptr, camp::resources::Resource const& r) { - umpire::event::record( - [&](auto& event) { event.ref((void*)m_allocator).ptr(ptr).res(camp::resources::to_string(r)); }); - UMPIRE_LOG(Debug, "(" << ptr << ")"); - if (!ptr) { - UMPIRE_LOG(Info, "Deallocating a null pointer (This behavior is intentionally allowed and ignored)"); - return; - } else { - if (m_tracking) { - auto record = deregisterAllocation(ptr, m_allocator); - if (!deallocateNull(ptr)) { - m_allocator->deallocate_resource(ptr, r, record.size); - } + umpire::event::record_resource_deallocate(m_allocator, ptr, r, [&]() { + if (!ptr) { + UMPIRE_LOG(Info, "Deallocating a null pointer (This behavior is intentionally allowed and ignored)"); + return; } else { - if (!deallocateNull(ptr)) { - m_allocator->deallocate_resource(ptr, r); + if (m_tracking) { + auto record = deregisterAllocation(ptr, m_allocator); + if (!deallocateNull(ptr)) { + m_allocator->deallocate_resource(ptr, r, record.size); + } + } else { + if (!deallocateNull(ptr)) { + m_allocator->deallocate_resource(ptr, r); + } } } - } + }); } inline void* Allocator::allocate(std::size_t bytes) diff --git a/src/umpire/CMakeLists.txt b/src/umpire/CMakeLists.txt index 276380396..77824d221 100644 --- a/src/umpire/CMakeLists.txt +++ b/src/umpire/CMakeLists.txt @@ -28,8 +28,12 @@ set (umpire_headers TypedAllocator.inl Umpire.hpp) +set (umpire_replay_headers + replay/Replay.hpp) + set (umpire_sources Allocator.cpp + Replay.cpp ResourceManager.cpp Umpire.cpp) @@ -134,7 +138,7 @@ endif () blt_add_library( NAME umpire - HEADERS ${umpire_headers} + HEADERS ${umpire_headers} ${umpire_replay_headers} SOURCES ${umpire_sources} DEPENDS_ON ${umpire_depends} DEFINES ${umpire_defines}) @@ -210,6 +214,10 @@ install(FILES ${umpire_headers} DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/umpire) +install(FILES + ${umpire_replay_headers} + DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/umpire/replay) + install(FILES ${CMAKE_BINARY_DIR}/include/umpire/config.hpp DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/umpire) diff --git a/src/umpire/Replay.cpp b/src/umpire/Replay.cpp new file mode 100644 index 000000000..7f4cc3e91 --- /dev/null +++ b/src/umpire/Replay.cpp @@ -0,0 +1,520 @@ +////////////////////////////////////////////////////////////////////////////// +// Copyright (c) 2016-26, Lawrence Livermore National Security, LLC and Umpire +// project contributors. See the COPYRIGHT file for details. +// +// SPDX-License-Identifier: (MIT) +////////////////////////////////////////////////////////////////////////////// + +#include "umpire/replay/Replay.hpp" + +#include +#include +#include +#include +#include +#include +#include + +#include "umpire/config.hpp" +#include "umpire/util/MPI.hpp" +#include "umpire/util/io.hpp" + +#if !defined(_MSC_VER) +#include +#else +#include +#define getpid _getpid +#endif + +namespace umpire { +namespace replay { + +namespace { + +thread_local int g_nested_resource_suppression_depth{0}; + +const char* statusString(ReplayCommandStatus status) noexcept +{ + switch (status) { + case ReplayCommandStatus::pending: + return "pending"; + case ReplayCommandStatus::committed: + return "committed"; + } + + return "committed"; +} + +struct AllocationInfo { + std::string allocation_id; + std::string allocator_id; +}; + +class ReplayRecorder { + public: + static ReplayRecorder& getInstance() + { + static ReplayRecorder recorder; + return recorder; + } + + std::string resolveAllocatorId(strategy::AllocationStrategy* allocator) + { + std::lock_guard lock(m_mutex); + auto found = m_allocator_ids.find(allocator); + if (found == m_allocator_ids.end()) { + UMPIRE_ERROR(runtime_error, + fmt::format("Replay allocator reference {} was used before it was recorded", fmt::ptr(allocator))); + } + return found->second; + } + + ReplayMakeAllocatorToken beginMakeAllocator(const std::string& name, bool tracking, const std::string& strategy_name, + const json& args) + { + if (g_nested_resource_suppression_depth > 0) { + return {}; + } + + std::lock_guard lock(m_mutex); + ReplayMakeAllocatorToken token; + token.active = true; + token.seq = nextSequence(); + token.allocator_id = nextAllocatorId(); + token.name = name; + token.strategy_name = strategy_name; + token.tracking = tracking; + token.args = args; + + writeCommand(json{{"op", "make_allocator"}, + {"seq", token.seq}, + {"allocator_id", token.allocator_id}, + {"name", token.name}, + {"strategy", token.strategy_name}, + {"tracking", token.tracking}, + {"args", token.args}, + {"status", statusString(ReplayCommandStatus::pending)}}); + + return token; + } + + void commitMakeAllocator(strategy::AllocationStrategy* allocator, const ReplayMakeAllocatorToken& token) + { + if (!token.active) { + return; + } + + std::lock_guard lock(m_mutex); + m_allocator_ids[allocator] = token.allocator_id; + + writeCommand(json{{"op", "make_allocator"}, + {"seq", token.seq}, + {"allocator_id", token.allocator_id}, + {"name", token.name}, + {"strategy", token.strategy_name}, + {"tracking", token.tracking}, + {"args", token.args}, + {"status", statusString(ReplayCommandStatus::committed)}}); + } + + ReplayAllocateToken beginAllocate(strategy::AllocationStrategy* allocator, std::size_t size) + { + std::lock_guard lock(m_mutex); + ReplayAllocateToken token; + token.active = true; + token.seq = nextSequence(); + token.allocator_id = resolveAllocatorIdUnlocked(allocator); + token.allocation_id = nextAllocationId(); + token.size = size; + + writeCommand(json{{"op", "allocate"}, + {"seq", token.seq}, + {"allocator_id", token.allocator_id}, + {"allocation_id", token.allocation_id}, + {"size", token.size}, + {"status", statusString(ReplayCommandStatus::pending)}}); + + return token; + } + + void commitAllocate(void* ptr, const ReplayAllocateToken& token) + { + if (!token.active) { + return; + } + + std::lock_guard lock(m_mutex); + m_allocations[ptr] = AllocationInfo{token.allocation_id, token.allocator_id}; + + writeCommand(json{{"op", "allocate"}, + {"seq", token.seq}, + {"allocator_id", token.allocator_id}, + {"allocation_id", token.allocation_id}, + {"size", token.size}, + {"status", statusString(ReplayCommandStatus::committed)}}); + } + + ReplayDeallocateToken beginDeallocate(strategy::AllocationStrategy* allocator, void* ptr) + { + if (!ptr) { + return {}; + } + + std::lock_guard lock(m_mutex); + const auto allocator_id = resolveAllocatorIdUnlocked(allocator); + auto found = m_allocations.find(ptr); + if (found == m_allocations.end()) { + UMPIRE_ERROR(runtime_error, + fmt::format("Replay allocation reference {} was deallocated before it was recorded", fmt::ptr(ptr))); + } + + if (found->second.allocator_id != allocator_id) { + UMPIRE_ERROR(runtime_error, + fmt::format("Replay deallocate allocator mismatch for allocation {}", found->second.allocation_id)); + } + + ReplayDeallocateToken token; + token.active = true; + token.seq = nextSequence(); + token.allocator_id = allocator_id; + token.allocation_id = found->second.allocation_id; + + writeCommand(json{{"op", "deallocate"}, + {"seq", token.seq}, + {"allocator_id", token.allocator_id}, + {"allocation_id", token.allocation_id}, + {"status", statusString(ReplayCommandStatus::pending)}}); + + return token; + } + + void commitDeallocate(const ReplayDeallocateToken& token) + { + if (!token.active) { + return; + } + + std::lock_guard lock(m_mutex); + auto found = + std::find_if(m_allocations.begin(), m_allocations.end(), [&](const auto& entry) -> bool { + return entry.second.allocation_id == token.allocation_id && entry.second.allocator_id == token.allocator_id; + }); + + if (found == m_allocations.end()) { + UMPIRE_ERROR(runtime_error, + fmt::format("Replay allocation {} was committed for deallocation before it was recorded", + token.allocation_id)); + } + + writeCommand(json{{"op", "deallocate"}, + {"seq", token.seq}, + {"allocator_id", token.allocator_id}, + {"allocation_id", token.allocation_id}, + {"status", statusString(ReplayCommandStatus::committed)}}); + + m_allocations.erase(found); + } + + private: + ReplayRecorder() = default; + + static std::string versionString() + { + std::ostringstream version; + version << UMPIRE_VERSION_MAJOR << "." << UMPIRE_VERSION_MINOR << "." << UMPIRE_VERSION_PATCH; + if (std::string{UMPIRE_VERSION_RC} != "") { + version << "-" << UMPIRE_VERSION_RC; + } + return version.str(); + } + + void ensureFile() + { + if (m_file.is_open()) { + return; + } + + m_filename = util::make_unique_filename(util::get_io_output_dir(), util::get_io_output_basename(), getpid(), "stats"); + m_file.open(m_filename); + if (!m_file) { + UMPIRE_ERROR(runtime_error, fmt::format("Failed to open replay output file {}", m_filename)); + } + } + + void ensureHeader() + { + if (m_header_written) { + return; + } + + ensureFile(); + const int rank = util::MPI::isInitialized() ? util::MPI::getRank() : 0; + json header{{"kind", "umpire_replay"}, + {"schema", "v2"}, + {"process", {{"pid", getpid()}, {"rank", rank}}}, + {"umpire_version", versionString()}}; + m_file << header.dump() << '\n'; + m_file.flush(); + m_header_written = true; + } + + std::size_t nextSequence() + { + return ++m_sequence; + } + + std::string nextAllocatorId() + { + return "a" + std::to_string(++m_next_allocator_id); + } + + std::string nextAllocationId() + { + return "m" + std::to_string(++m_next_allocation_id); + } + + std::string resolveAllocatorIdUnlocked(strategy::AllocationStrategy* allocator) + { + auto found = m_allocator_ids.find(allocator); + if (found == m_allocator_ids.end()) { + UMPIRE_ERROR(runtime_error, + fmt::format("Replay allocator reference {} was used before it was recorded", fmt::ptr(allocator))); + } + return found->second; + } + + void writeCommand(const json& command) + { + ensureHeader(); + m_file << command.dump() << '\n'; + m_file.flush(); + } + + std::mutex m_mutex{}; + std::ofstream m_file{}; + std::string m_filename{}; + bool m_header_written{false}; + std::size_t m_sequence{0}; + std::size_t m_next_allocator_id{0}; + std::size_t m_next_allocation_id{0}; + std::unordered_map m_allocator_ids{}; + std::unordered_map m_allocations{}; +}; + +MemoryResourceTraits::shared_scope sharedScopeFromString(const std::string& scope) +{ + if (scope == "node") { + return MemoryResourceTraits::shared_scope::node; + } + if (scope == "socket") { + return MemoryResourceTraits::shared_scope::socket; + } + return MemoryResourceTraits::shared_scope::unknown; +} + +MemoryResourceTraits::optimized_for optimizedForFromString(const std::string& value) +{ + if (value == "latency") { + return MemoryResourceTraits::optimized_for::latency; + } + if (value == "bandwidth") { + return MemoryResourceTraits::optimized_for::bandwidth; + } + if (value == "access") { + return MemoryResourceTraits::optimized_for::access; + } + return MemoryResourceTraits::optimized_for::any; +} + +MemoryResourceTraits::vendor_type vendorFromString(const std::string& vendor) +{ + if (vendor == "amd") { + return MemoryResourceTraits::vendor_type::amd; + } + if (vendor == "ibm") { + return MemoryResourceTraits::vendor_type::ibm; + } + if (vendor == "intel") { + return MemoryResourceTraits::vendor_type::intel; + } + if (vendor == "nvidia") { + return MemoryResourceTraits::vendor_type::nvidia; + } + return MemoryResourceTraits::vendor_type::unknown; +} + +MemoryResourceTraits::memory_type memoryTypeFromString(const std::string& type) +{ + if (type == "ddr") { + return MemoryResourceTraits::memory_type::ddr; + } + if (type == "gddr") { + return MemoryResourceTraits::memory_type::gddr; + } + if (type == "hbm") { + return MemoryResourceTraits::memory_type::hbm; + } + if (type == "nvme") { + return MemoryResourceTraits::memory_type::nvme; + } + return MemoryResourceTraits::memory_type::unknown; +} + +MemoryResourceTraits::resource_type resourceTypeFromString(const std::string& resource) +{ + if (resource == "host") { + return MemoryResourceTraits::resource_type::host; + } + if (resource == "device") { + return MemoryResourceTraits::resource_type::device; + } + if (resource == "device_const") { + return MemoryResourceTraits::resource_type::device_const; + } + if (resource == "pinned") { + return MemoryResourceTraits::resource_type::pinned; + } + if (resource == "um") { + return MemoryResourceTraits::resource_type::um; + } + if (resource == "file") { + return MemoryResourceTraits::resource_type::file; + } + if (resource == "shared") { + return MemoryResourceTraits::resource_type::shared; + } + return MemoryResourceTraits::resource_type::unknown; +} + +MemoryResourceTraits::granularity_type granularityFromString(const std::string& granularity) +{ + if (granularity == "fine_grained") { + return MemoryResourceTraits::granularity_type::fine_grained; + } + if (granularity == "coarse_grained") { + return MemoryResourceTraits::granularity_type::coarse_grained; + } + return MemoryResourceTraits::granularity_type::unknown; +} + +} // namespace + +ScopedNestedReplaySuppression::ScopedNestedReplaySuppression(bool enabled) : m_enabled(enabled) +{ + if (m_enabled) { + ++g_nested_resource_suppression_depth; + } +} + +ScopedNestedReplaySuppression::~ScopedNestedReplaySuppression() +{ + if (m_enabled) { + --g_nested_resource_suppression_depth; + } +} + +bool is_enabled() noexcept +{ + static const bool enabled = std::getenv("UMPIRE_REPLAY") != nullptr; + return enabled; +} + +std::string resolve_allocator_id(const Allocator& allocator) +{ + return resolve_allocator_id(allocator.getAllocationStrategy()); +} + +std::string resolve_allocator_id(strategy::AllocationStrategy* allocator) +{ + return ReplayRecorder::getInstance().resolveAllocatorId(allocator); +} + +ReplayMakeAllocatorToken begin_make_allocator(const std::string& name, bool tracking, const std::string& strategy_name, + const json& args) +{ + if (!is_enabled()) { + return {}; + } + + return ReplayRecorder::getInstance().beginMakeAllocator(name, tracking, strategy_name, args); +} + +void commit_make_allocator(strategy::AllocationStrategy* allocator, const ReplayMakeAllocatorToken& token) +{ + if (!is_enabled()) { + return; + } + + ReplayRecorder::getInstance().commitMakeAllocator(allocator, token); +} + +ReplayAllocateToken begin_allocate(strategy::AllocationStrategy* allocator, std::size_t size) +{ + if (!is_enabled()) { + return {}; + } + + return ReplayRecorder::getInstance().beginAllocate(allocator, size); +} + +void commit_allocate(void* ptr, const ReplayAllocateToken& token) +{ + if (!is_enabled()) { + return; + } + + ReplayRecorder::getInstance().commitAllocate(ptr, token); +} + +ReplayDeallocateToken begin_deallocate(strategy::AllocationStrategy* allocator, void* ptr) +{ + if (!is_enabled()) { + return {}; + } + + return ReplayRecorder::getInstance().beginDeallocate(allocator, ptr); +} + +void commit_deallocate(const ReplayDeallocateToken& token) +{ + if (!is_enabled()) { + return; + } + + ReplayRecorder::getInstance().commitDeallocate(token); +} + +json serialize_memory_resource_args(const std::string& resource_name, const MemoryResourceTraits& traits) +{ + return json{{"resource_name", resource_name}, + {"traits", + {{"unified", traits.unified}, + {"ipc", traits.ipc}, + {"size", traits.size}, + {"vendor", to_string(traits.vendor)}, + {"kind", to_string(traits.kind)}, + {"used_for", to_string(traits.used_for)}, + {"resource", to_string(traits.resource)}, + {"scope", to_string(traits.scope)}, + {"granularity", to_string(traits.granularity)}, + {"tracking", traits.tracking}}}}; +} + +MemoryResourceTraits deserialize_memory_resource_traits(const json& traits_json) +{ + MemoryResourceTraits traits; + traits.unified = traits_json.value("unified", false); + traits.ipc = traits_json.value("ipc", false); + traits.size = traits_json.value("size", std::size_t{0}); + traits.vendor = vendorFromString(traits_json.value("vendor", "unknown")); + traits.kind = memoryTypeFromString(traits_json.value("kind", "unknown")); + traits.used_for = optimizedForFromString(traits_json.value("used_for", "any")); + traits.resource = resourceTypeFromString(traits_json.value("resource", "unknown")); + traits.scope = sharedScopeFromString(traits_json.value("scope", "unknown")); + traits.granularity = granularityFromString(traits_json.value("granularity", "unknown")); + traits.tracking = traits_json.value("tracking", true); + return traits; +} + +} // namespace replay +} // namespace umpire diff --git a/src/umpire/ResourceManager.cpp b/src/umpire/ResourceManager.cpp index 8c31492a4..01c0bad15 100644 --- a/src/umpire/ResourceManager.cpp +++ b/src/umpire/ResourceManager.cpp @@ -12,8 +12,10 @@ #include "umpire/Umpire.hpp" #include "umpire/config.hpp" +#include "umpire/event/operation_recording.hpp" #include "umpire/op/MemoryOperation.hpp" #include "umpire/op/MemoryOperationRegistry.hpp" +#include "umpire/replay/Replay.hpp" #include "umpire/resource/MemoryResourceRegistry.hpp" #include "umpire/strategy/FixedPool.hpp" #if defined(UMPIRE_ENABLE_NUMA) @@ -159,17 +161,23 @@ Allocator ResourceManager::makeResource(const std::string& name, MemoryResourceT m_shared_allocator_names.push_back(name); } - std::unique_ptr allocator{registry.makeMemoryResource(name, getNextId(), traits)}; - allocator->setTracking(traits.tracking); - - umpire::event::record([&](auto& event) { - event.name("make_memory_resource") - .category(event::category::operation) - .arg("allocator_ref", (void*)allocator.get()) - .arg("introspection", traits.tracking) - .tag("allocator_name", name) - .tag("replay", "true"); - }); + auto allocator = umpire::event::record_make_resource( + name, traits.tracking, replay::serialize_memory_resource_args(name, traits), + [&]() -> std::unique_ptr { + auto created = std::unique_ptr{registry.makeMemoryResource(name, getNextId(), traits)}; + created->setTracking(traits.tracking); + return created; + }, + [&](strategy::AllocationStrategy* created) { + umpire::event::record([&](auto& event) { + event.name("make_memory_resource") + .category(event::category::operation) + .arg("allocator_ref", (void*)created) + .arg("introspection", traits.tracking) + .tag("allocator_name", name) + .tag("replay", "true"); + }); + }); int id{allocator->getId()}; m_allocators_by_name[name] = allocator.get(); diff --git a/src/umpire/ResourceManager.inl b/src/umpire/ResourceManager.inl index 2c96be943..820cd0528 100644 --- a/src/umpire/ResourceManager.inl +++ b/src/umpire/ResourceManager.inl @@ -11,7 +11,8 @@ #include "camp/list.hpp" #include "umpire/ResourceManager.hpp" -#include "umpire/event/event.hpp" +#include "umpire/event/operation_recording.hpp" +#include "umpire/replay/Replay.hpp" #include "umpire/util/Macros.hpp" #include "umpire/util/error.hpp" #include "umpire/util/make_unique.hpp" @@ -34,19 +35,52 @@ Allocator ResourceManager::makeAllocator(const std::string& name, Tracking track UMPIRE_ERROR(runtime_error, fmt::format("Allocator with name \"{}\" is already registered", name)); } - allocator = util::make_unique(name, getNextId(), std::forward(args)...); - allocator->setTracking(is_tracked); + replay::json replay_args{}; - umpire::event::record([&](auto& event) { - event.name("make_allocator") - .category(event::category::operation) - .arg("allocator_ref", (void*)allocator.get()) - .arg("type", typeid(Strategy).name()) - .arg("introspection", is_tracked) - .args(args...) - .tag("allocator_name", allocator->getName()) - .tag("replay", "true"); - }); +#if defined(UMPIRE_ENABLE_MPI) && defined(UMPIRE_ENABLE_IPC_SHARED_MEMORY) && \ + (defined(UMPIRE_ENABLE_CUDA) || defined(UMPIRE_ENABLE_HIP)) + constexpr bool suppress_nested_resources = std::is_same::value; +#else + constexpr bool suppress_nested_resources = false; +#endif + + if (replay::is_enabled()) { +#if defined(UMPIRE_ENABLE_MPI) && defined(UMPIRE_ENABLE_IPC_SHARED_MEMORY) && \ + (defined(UMPIRE_ENABLE_CUDA) || defined(UMPIRE_ENABLE_HIP)) + if constexpr (suppress_nested_resources) { + if constexpr (sizeof...(Args) == 0) { + auto device_allocator = getAllocator("DEVICE"); + replay_args = replay::serialize_allocator_args(); + replay_args["device_allocator"] = replay::resolve_allocator_id(device_allocator); + } else { + replay_args = replay::serialize_allocator_args(std::forward(args)...); + } + } else +#endif + { + replay_args = replay::serialize_allocator_args(std::forward(args)...); + } + } + + allocator = umpire::event::record_make_allocator( + name, is_tracked, replay::strategy_name(), replay_args, suppress_nested_resources, + [&]() -> std::unique_ptr { + auto created = util::make_unique(name, getNextId(), std::forward(args)...); + created->setTracking(is_tracked); + return created; + }, + [&](strategy::AllocationStrategy* created) { + umpire::event::record([&](auto& event) { + event.name("make_allocator") + .category(event::category::operation) + .arg("allocator_ref", (void*)created) + .arg("type", typeid(Strategy).name()) + .arg("introspection", is_tracked) + .args(args...) + .tag("allocator_name", created->getName()) + .tag("replay", "true"); + }); + }); m_allocators_by_name[name] = allocator.get(); m_allocators_by_id[allocator->getId()] = allocator.get(); diff --git a/src/umpire/event/event.hpp b/src/umpire/event/event.hpp index fb39bcc65..2883280c9 100644 --- a/src/umpire/event/event.hpp +++ b/src/umpire/event/event.hpp @@ -71,7 +71,7 @@ static const char* event_env{std::getenv("UMPIRE_EVENTS")}; #endif static const bool enable_replay{(replay_env != NULL)}; static const bool enable_event{(event_env != NULL)}; -static const bool event_build_enabled{enable_replay || enable_event}; +static const bool event_build_enabled{enable_event}; } // namespace enum class category { operation, statistic, metadata }; diff --git a/src/umpire/event/operation_recording.hpp b/src/umpire/event/operation_recording.hpp new file mode 100644 index 000000000..7827d86d9 --- /dev/null +++ b/src/umpire/event/operation_recording.hpp @@ -0,0 +1,156 @@ +////////////////////////////////////////////////////////////////////////////// +// Copyright (c) 2016-26, Lawrence Livermore National Security, LLC and Umpire +// project contributors. See the COPYRIGHT file for details. +// +// SPDX-License-Identifier: (MIT) +////////////////////////////////////////////////////////////////////////////// +#ifndef UMPIRE_event_operation_recording_HPP +#define UMPIRE_event_operation_recording_HPP + +#include +#include +#include +#include + +#include "camp/resource.hpp" +#include "umpire/event/event.hpp" +#include "umpire/replay/Replay.hpp" +#include "umpire/strategy/AllocationStrategy.hpp" + +namespace umpire { +namespace event { + +namespace detail { + +template +void* record_allocate_impl(strategy::AllocationStrategy* allocator, std::size_t bytes, AllocateFn&& allocate_fn, + EventFn&& event_fn) +{ + replay::ReplayAllocateToken replay_token{}; + if (replay::is_enabled()) { + replay_token = replay::begin_allocate(allocator, bytes); + } + + void* ptr = std::forward(allocate_fn)(); + + umpire::event::record([&](auto& event) { event_fn(event, ptr); }); + replay::commit_allocate(ptr, replay_token); + + return ptr; +} + +template +void record_deallocate_impl(strategy::AllocationStrategy* allocator, void* ptr, DeallocateFn&& deallocate_fn, + EventFn&& event_fn) +{ + if (!ptr) { + std::forward(deallocate_fn)(); + return; + } + + auto replay_token = replay::begin_deallocate(allocator, ptr); + + std::forward(deallocate_fn)(); + + umpire::event::record([&](auto& event) { event_fn(event); }); + replay::commit_deallocate(replay_token); +} + +template +std::unique_ptr record_make_allocator_impl( + const std::string& name, bool tracking, const std::string& replay_strategy, const replay::json& replay_args, + bool suppress_nested_replay, ConstructFn&& construct_fn, EventFn&& event_fn) +{ + const bool replay_enabled = replay::is_enabled(); + replay::ReplayMakeAllocatorToken replay_token{}; + + if (replay_enabled) { + replay_token = replay::begin_make_allocator(name, tracking, replay_strategy, replay_args); + } + + replay::ScopedNestedReplaySuppression suppress_nested(replay_enabled && suppress_nested_replay); + + auto allocator = std::forward(construct_fn)(); + + if (replay_enabled) { + replay::commit_make_allocator(allocator.get(), replay_token); + } + + event_fn(allocator.get()); + + return allocator; +} + +} // end of namespace detail + +template +void* record_allocate(strategy::AllocationStrategy* allocator, std::size_t bytes, AllocateFn&& allocate_fn) +{ + return detail::record_allocate_impl( + allocator, bytes, std::forward(allocate_fn), + [&](auto& event, void* ptr) { event.size(bytes).ref((void*)allocator).ptr(ptr); }); +} + +template +void* record_named_allocate(strategy::AllocationStrategy* allocator, const std::string& name, std::size_t bytes, + AllocateFn&& allocate_fn) +{ + return detail::record_allocate_impl( + allocator, bytes, std::forward(allocate_fn), + [&](auto& event, void* ptr) { event.name(name).size(bytes).ref((void*)allocator).ptr(ptr); }); +} + +template +void* record_resource_allocate(strategy::AllocationStrategy* allocator, std::size_t bytes, + camp::resources::Resource const& r, AllocateFn&& allocate_fn) +{ + const auto resource = camp::resources::to_string(r); + + return detail::record_allocate_impl( + allocator, bytes, std::forward(allocate_fn), + [&](auto& event, void* ptr) { event.size(bytes).ref((void*)allocator).ptr(ptr).res(resource); }); +} + +template +void record_deallocate(strategy::AllocationStrategy* allocator, void* ptr, DeallocateFn&& deallocate_fn) +{ + detail::record_deallocate_impl( + allocator, ptr, std::forward(deallocate_fn), + [&](auto& event) { event.ref((void*)allocator).ptr(ptr); }); +} + +template +void record_resource_deallocate(strategy::AllocationStrategy* allocator, void* ptr, camp::resources::Resource const& r, + DeallocateFn&& deallocate_fn) +{ + const auto resource = camp::resources::to_string(r); + + detail::record_deallocate_impl( + allocator, ptr, std::forward(deallocate_fn), + [&](auto& event) { event.ref((void*)allocator).ptr(ptr).res(resource); }); +} + +template +std::unique_ptr record_make_allocator( + const std::string& name, bool tracking, const std::string& replay_strategy, const replay::json& replay_args, + bool suppress_nested_replay, ConstructFn&& construct_fn, EventFn&& event_fn) +{ + return detail::record_make_allocator_impl(name, tracking, replay_strategy, replay_args, suppress_nested_replay, + std::forward(construct_fn), + std::forward(event_fn)); +} + +template +std::unique_ptr record_make_resource( + const std::string& name, bool tracking, const replay::json& replay_args, ConstructFn&& construct_fn, + EventFn&& event_fn) +{ + return detail::record_make_allocator_impl(name, tracking, "MemoryResource", replay_args, false, + std::forward(construct_fn), + std::forward(event_fn)); +} + +} // end of namespace event +} // end of namespace umpire + +#endif // UMPIRE_event_operation_recording_HPP diff --git a/src/umpire/replay/Replay.hpp b/src/umpire/replay/Replay.hpp new file mode 100644 index 000000000..cfe46546e --- /dev/null +++ b/src/umpire/replay/Replay.hpp @@ -0,0 +1,462 @@ +////////////////////////////////////////////////////////////////////////////// +// Copyright (c) 2016-26, Lawrence Livermore National Security, LLC and Umpire +// project contributors. See the COPYRIGHT file for details. +// +// SPDX-License-Identifier: (MIT) +////////////////////////////////////////////////////////////////////////////// +#ifndef UMPIRE_replay_Replay_HPP +#define UMPIRE_replay_Replay_HPP + +#include +#include +#include +#include + +#include "umpire/config.hpp" +#include "umpire/json/json.hpp" +#include "umpire/util/MemoryResourceTraits.hpp" +#include "umpire/util/error.hpp" + +namespace umpire { + +class Allocator; + +namespace strategy { +class AllocationStrategy; +class AllocationAdvisor; +class AllocationPrefetcher; +class AlignedAllocator; +class DynamicPoolList; +class FixedPool; +class MixedPool; +class MonotonicAllocationStrategy; +class NamedAllocationStrategy; +class NamingShim; +class QuickPool; +class ResourceAwarePool; +class SizeLimiter; +class SlotPool; +class ThreadSafeAllocator; +template +class PoolCoalesceHeuristic; + +#if defined(UMPIRE_ENABLE_NUMA) +class NumaPolicy; +#endif + +#if defined(UMPIRE_ENABLE_MPI) && defined(UMPIRE_ENABLE_IPC_SHARED_MEMORY) && \ + (defined(UMPIRE_ENABLE_CUDA) || defined(UMPIRE_ENABLE_HIP)) +class DeviceIpcAllocator; +#endif +} // namespace strategy + +namespace replay { + +using json = nlohmann::json; + +enum class ReplayCommandStatus { pending, committed }; + +struct ReplayMakeAllocatorToken { + bool active{false}; + std::size_t seq{0}; + std::string allocator_id{}; + std::string name{}; + std::string strategy_name{}; + bool tracking{true}; + json args{}; +}; + +struct ReplayAllocateToken { + bool active{false}; + std::size_t seq{0}; + std::string allocator_id{}; + std::string allocation_id{}; + std::size_t size{0}; +}; + +struct ReplayDeallocateToken { + bool active{false}; + std::size_t seq{0}; + std::string allocator_id{}; + std::string allocation_id{}; +}; + +class ScopedNestedReplaySuppression { + public: + explicit ScopedNestedReplaySuppression(bool enabled); + ~ScopedNestedReplaySuppression(); + + private: + bool m_enabled; +}; + +bool is_enabled() noexcept; + +std::string resolve_allocator_id(const Allocator& allocator); +std::string resolve_allocator_id(strategy::AllocationStrategy* allocator); + +ReplayMakeAllocatorToken begin_make_allocator(const std::string& name, bool tracking, const std::string& strategy_name, + const json& args); +void commit_make_allocator(strategy::AllocationStrategy* allocator, const ReplayMakeAllocatorToken& token); +ReplayAllocateToken begin_allocate(strategy::AllocationStrategy* allocator, std::size_t size); +void commit_allocate(void* ptr, const ReplayAllocateToken& token); +ReplayDeallocateToken begin_deallocate(strategy::AllocationStrategy* allocator, void* ptr); +void commit_deallocate(const ReplayDeallocateToken& token); + +json serialize_memory_resource_args(const std::string& resource_name, const MemoryResourceTraits& traits); +MemoryResourceTraits deserialize_memory_resource_traits(const json& traits_json); +template +json serialize_allocator_args(Args&&... args); +template +std::string strategy_name(); + +} // namespace replay +} // namespace umpire + +#include "umpire/Allocator.hpp" +#include "umpire/strategy/AlignedAllocator.hpp" +#include "umpire/strategy/AllocationAdvisor.hpp" +#include "umpire/strategy/AllocationPrefetcher.hpp" +#include "umpire/strategy/DynamicPoolList.hpp" +#include "umpire/strategy/FixedPool.hpp" +#include "umpire/strategy/MixedPool.hpp" +#include "umpire/strategy/MonotonicAllocationStrategy.hpp" +#include "umpire/strategy/NamedAllocationStrategy.hpp" +#include "umpire/strategy/NamingShim.hpp" +#include "umpire/strategy/PoolCoalesceHeuristic.hpp" +#include "umpire/strategy/QuickPool.hpp" +#include "umpire/strategy/SizeLimiter.hpp" +#include "umpire/strategy/SlotPool.hpp" +#include "umpire/strategy/ThreadSafeAllocator.hpp" + +#if defined(UMPIRE_ENABLE_NUMA) +#include "umpire/strategy/NumaPolicy.hpp" +#endif + +#if defined(UMPIRE_ENABLE_MPI) && defined(UMPIRE_ENABLE_IPC_SHARED_MEMORY) && \ + (defined(UMPIRE_ENABLE_CUDA) || defined(UMPIRE_ENABLE_HIP)) +#include "umpire/strategy/DeviceIpcAllocator.hpp" +#endif + +namespace umpire { +namespace replay { + +namespace detail { + +template +struct always_false : std::false_type {}; + +template +inline std::string replay_strategy_name(Strategy*) +{ + static_assert(always_false::value, "Replay strategy name is not implemented for this allocator type"); +} + +template +json serialize_heuristic(const strategy::PoolCoalesceHeuristic& heuristic) +{ + if (!heuristic.is_known()) { + UMPIRE_ERROR(runtime_error, + "UMPIRE_REPLAY only supports pool heuristics created with the built-in helper factories"); + } + + return json{{"kind", heuristic.kind_string()}, {"parameter", heuristic.parameter()}}; +} + +inline json serialize_parent_only(Allocator allocator) +{ + return json{{"parent_allocator", resolve_allocator_id(allocator)}}; +} + +inline json serialize_make_allocator_args(strategy::AllocationAdvisor*, Allocator allocator, + const std::string& advice_operation, int device_id = 0) +{ + return json{{"parent_allocator", resolve_allocator_id(allocator)}, + {"advice_operation", advice_operation}, + {"device_id", device_id}}; +} + +inline json serialize_make_allocator_args(strategy::AllocationAdvisor*, Allocator allocator, + const std::string& advice_operation, Allocator accessing_allocator, + int device_id = 0) +{ + return json{{"parent_allocator", resolve_allocator_id(allocator)}, + {"advice_operation", advice_operation}, + {"accessing_allocator", resolve_allocator_id(accessing_allocator)}, + {"device_id", device_id}}; +} + +inline json serialize_make_allocator_args(strategy::AllocationPrefetcher*, Allocator allocator, int device_id = 0) +{ + return json{{"parent_allocator", resolve_allocator_id(allocator)}, {"device_id", device_id}}; +} + +inline json serialize_make_allocator_args(strategy::AlignedAllocator*, Allocator allocator, std::size_t alignment = 16) +{ + return json{{"parent_allocator", resolve_allocator_id(allocator)}, {"alignment", alignment}}; +} + +inline json serialize_make_allocator_args( + strategy::DynamicPoolList*, Allocator allocator, + const std::size_t first_minimum_pool_allocation_size = 512 * 1024 * 1024, + const std::size_t next_minimum_pool_allocation_size = 1 * 1024 * 1024, + const std::size_t alignment = 16, + strategy::PoolCoalesceHeuristic should_coalesce = + strategy::DynamicPoolList::percent_releasable_hwm(100)) +{ + return json{{"parent_allocator", resolve_allocator_id(allocator)}, + {"initial_alloc_size", first_minimum_pool_allocation_size}, + {"min_alloc_size", next_minimum_pool_allocation_size}, + {"alignment", alignment}, + {"heuristic", serialize_heuristic(should_coalesce)}}; +} + +inline json serialize_make_allocator_args(strategy::FixedPool*, Allocator allocator, const std::size_t object_bytes, + const std::size_t objects_per_pool = 64 * sizeof(int) * 8) +{ + return json{{"parent_allocator", resolve_allocator_id(allocator)}, + {"object_bytes", object_bytes}, + {"objects_per_pool", objects_per_pool}}; +} + +inline json serialize_make_allocator_args( + strategy::MixedPool*, Allocator allocator, std::size_t smallest_fixed_obj_size = (1 << 8), + std::size_t largest_fixed_obj_size = (1 << 17), std::size_t max_initial_fixed_pool_size = 1024 * 1024 * 2, + std::size_t fixed_size_multiplier = 16, const std::size_t quick_pool_initial_alloc_size = (512 * 1024 * 1024), + const std::size_t quick_pool_min_alloc_size = (1 * 1024 * 1024), const std::size_t quick_pool_align_bytes = 16, + strategy::PoolCoalesceHeuristic should_coalesce = + strategy::QuickPool::percent_releasable(100)) +{ + return json{{"parent_allocator", resolve_allocator_id(allocator)}, + {"smallest_fixed_obj_size", smallest_fixed_obj_size}, + {"largest_fixed_obj_size", largest_fixed_obj_size}, + {"max_initial_fixed_pool_size", max_initial_fixed_pool_size}, + {"fixed_size_multiplier", fixed_size_multiplier}, + {"quick_pool_initial_alloc_size", quick_pool_initial_alloc_size}, + {"quick_pool_min_alloc_size", quick_pool_min_alloc_size}, + {"quick_pool_align_bytes", quick_pool_align_bytes}, + {"heuristic", serialize_heuristic(should_coalesce)}}; +} + +inline json serialize_make_allocator_args(strategy::MonotonicAllocationStrategy*, Allocator allocator, + std::size_t capacity) +{ + return json{{"parent_allocator", resolve_allocator_id(allocator)}, {"capacity", capacity}}; +} + +inline json serialize_make_allocator_args(strategy::NamedAllocationStrategy*, Allocator allocator) +{ + return serialize_parent_only(allocator); +} + +inline json serialize_make_allocator_args(strategy::NamingShim*, Allocator allocator) +{ + return serialize_parent_only(allocator); +} + +inline json serialize_make_allocator_args( + strategy::QuickPool*, Allocator allocator, + const std::size_t first_minimum_pool_allocation_size = 512 * 1024 * 1024, + const std::size_t next_minimum_pool_allocation_size = 1 * 1024 * 1024, + const std::size_t alignment = 16, + strategy::PoolCoalesceHeuristic should_coalesce = + strategy::QuickPool::percent_releasable_hwm(100)) +{ + return json{{"parent_allocator", resolve_allocator_id(allocator)}, + {"initial_alloc_size", first_minimum_pool_allocation_size}, + {"min_alloc_size", next_minimum_pool_allocation_size}, + {"alignment", alignment}, + {"heuristic", serialize_heuristic(should_coalesce)}}; +} + +inline json serialize_resource_aware_pool_args(Allocator allocator, const std::size_t first_minimum_pool_allocation_size, + const std::size_t next_minimum_pool_allocation_size, + const std::size_t alignment, const json& heuristic) +{ + return json{{"parent_allocator", resolve_allocator_id(allocator)}, + {"initial_alloc_size", first_minimum_pool_allocation_size}, + {"min_alloc_size", next_minimum_pool_allocation_size}, + {"alignment", alignment}, + {"heuristic", heuristic}}; +} + +inline json serialize_make_allocator_args(strategy::ResourceAwarePool*, Allocator allocator) +{ + return serialize_resource_aware_pool_args(allocator, 512 * 1024 * 1024, 1 * 1024 * 1024, 16, + json{{"kind", "percent_releasable_hwm"}, {"parameter", 100}}); +} + +inline json serialize_make_allocator_args(strategy::ResourceAwarePool*, Allocator allocator, + const std::size_t first_minimum_pool_allocation_size) +{ + return serialize_resource_aware_pool_args(allocator, first_minimum_pool_allocation_size, 1 * 1024 * 1024, 16, + json{{"kind", "percent_releasable_hwm"}, {"parameter", 100}}); +} + +inline json serialize_make_allocator_args(strategy::ResourceAwarePool*, Allocator allocator, + const std::size_t first_minimum_pool_allocation_size, + const std::size_t next_minimum_pool_allocation_size) +{ + return serialize_resource_aware_pool_args(allocator, first_minimum_pool_allocation_size, + next_minimum_pool_allocation_size, 16, + json{{"kind", "percent_releasable_hwm"}, {"parameter", 100}}); +} + +inline json serialize_make_allocator_args(strategy::ResourceAwarePool*, Allocator allocator, + const std::size_t first_minimum_pool_allocation_size, + const std::size_t next_minimum_pool_allocation_size, + const std::size_t alignment) +{ + return serialize_resource_aware_pool_args(allocator, first_minimum_pool_allocation_size, + next_minimum_pool_allocation_size, alignment, + json{{"kind", "percent_releasable_hwm"}, {"parameter", 100}}); +} + +inline json serialize_make_allocator_args(strategy::ResourceAwarePool*, Allocator allocator, + const std::size_t first_minimum_pool_allocation_size, + const std::size_t next_minimum_pool_allocation_size, + const std::size_t alignment, + strategy::PoolCoalesceHeuristic should_coalesce) +{ + return serialize_resource_aware_pool_args(allocator, first_minimum_pool_allocation_size, + next_minimum_pool_allocation_size, alignment, + serialize_heuristic(should_coalesce)); +} + +inline json serialize_make_allocator_args(strategy::SizeLimiter*, Allocator allocator, std::size_t size_limit) +{ + return json{{"parent_allocator", resolve_allocator_id(allocator)}, {"size_limit", size_limit}}; +} + +inline json serialize_make_allocator_args(strategy::SlotPool*, Allocator allocator, std::size_t slots) +{ + return json{{"parent_allocator", resolve_allocator_id(allocator)}, {"slots", slots}}; +} + +inline json serialize_make_allocator_args(strategy::ThreadSafeAllocator*, Allocator allocator) +{ + return serialize_parent_only(allocator); +} + +inline std::string replay_strategy_name(strategy::AllocationAdvisor*) +{ + return "AllocationAdvisor"; +} + +inline std::string replay_strategy_name(strategy::AllocationPrefetcher*) +{ + return "AllocationPrefetcher"; +} + +inline std::string replay_strategy_name(strategy::AlignedAllocator*) +{ + return "AlignedAllocator"; +} + +inline std::string replay_strategy_name(strategy::DynamicPoolList*) +{ + return "DynamicPoolList"; +} + +inline std::string replay_strategy_name(strategy::FixedPool*) +{ + return "FixedPool"; +} + +inline std::string replay_strategy_name(strategy::MixedPool*) +{ + return "MixedPool"; +} + +inline std::string replay_strategy_name(strategy::MonotonicAllocationStrategy*) +{ + return "MonotonicAllocationStrategy"; +} + +inline std::string replay_strategy_name(strategy::NamedAllocationStrategy*) +{ + return "NamedAllocationStrategy"; +} + +inline std::string replay_strategy_name(strategy::NamingShim*) +{ + return "NamingShim"; +} + +inline std::string replay_strategy_name(strategy::QuickPool*) +{ + return "QuickPool"; +} + +inline std::string replay_strategy_name(strategy::ResourceAwarePool*) +{ + return "ResourceAwarePool"; +} + +inline std::string replay_strategy_name(strategy::SizeLimiter*) +{ + return "SizeLimiter"; +} + +inline std::string replay_strategy_name(strategy::SlotPool*) +{ + return "SlotPool"; +} + +inline std::string replay_strategy_name(strategy::ThreadSafeAllocator*) +{ + return "ThreadSafeAllocator"; +} + +#if defined(UMPIRE_ENABLE_NUMA) +inline json serialize_make_allocator_args(strategy::NumaPolicy*, Allocator allocator, int numa_node) +{ + return json{{"parent_allocator", resolve_allocator_id(allocator)}, {"numa_node", numa_node}}; +} + +inline std::string replay_strategy_name(strategy::NumaPolicy*) +{ + return "NumaPolicy"; +} +#endif + +#if defined(UMPIRE_ENABLE_MPI) && defined(UMPIRE_ENABLE_IPC_SHARED_MEMORY) && \ + (defined(UMPIRE_ENABLE_CUDA) || defined(UMPIRE_ENABLE_HIP)) +inline json serialize_make_allocator_args(strategy::DeviceIpcAllocator*) +{ + return json{{"shared_scope", "socket"}, {"shared_memory_size", 1024 * 1024}}; +} + +inline json serialize_make_allocator_args(strategy::DeviceIpcAllocator*, Allocator device_allocator, + MemoryResourceTraits::shared_scope scope, + std::size_t shared_memory_size = 1024 * 1024) +{ + return json{{"device_allocator", resolve_allocator_id(device_allocator)}, + {"shared_scope", to_string(scope)}, + {"shared_memory_size", shared_memory_size}}; +} + +inline std::string replay_strategy_name(strategy::DeviceIpcAllocator*) +{ + return "DeviceIpcAllocator"; +} +#endif + +} // namespace detail + +template +json serialize_allocator_args(Args&&... args) +{ + return detail::serialize_make_allocator_args(static_cast(nullptr), std::forward(args)...); +} + +template +std::string strategy_name() +{ + return detail::replay_strategy_name(static_cast(nullptr)); +} + +} // namespace replay +} // namespace umpire + +#endif // UMPIRE_replay_Replay_HPP diff --git a/src/umpire/strategy/DynamicPoolList.cpp b/src/umpire/strategy/DynamicPoolList.cpp index 31cd42525..5113f5865 100644 --- a/src/umpire/strategy/DynamicPoolList.cpp +++ b/src/umpire/strategy/DynamicPoolList.cpp @@ -138,16 +138,22 @@ void DynamicPoolList::coalesce() noexcept PoolCoalesceHeuristic DynamicPoolList::blocks_releasable(std::size_t nblocks) { - return [=](const strategy::DynamicPoolList& pool) { - return pool.getReleasableBlocks() >= nblocks ? pool.getActualSize() : 0; - }; + return PoolCoalesceHeuristic::known( + [=](const strategy::DynamicPoolList& pool) { + return pool.getReleasableBlocks() >= nblocks ? pool.getActualSize() : 0; + }, + PoolCoalesceHeuristic::Kind::blocks_releasable, + nblocks); } PoolCoalesceHeuristic DynamicPoolList::blocks_releasable_hwm(std::size_t nblocks) { - return [=](const strategy::DynamicPoolList& pool) { - return pool.getReleasableBlocks() >= nblocks ? pool.getAlignedHighwaterMark() : 0; - }; + return PoolCoalesceHeuristic::known( + [=](const strategy::DynamicPoolList& pool) { + return pool.getReleasableBlocks() >= nblocks ? pool.getAlignedHighwaterMark() : 0; + }, + PoolCoalesceHeuristic::Kind::blocks_releasable_hwm, + nblocks); } PoolCoalesceHeuristic DynamicPoolList::percent_releasable(int percentage) @@ -158,16 +164,25 @@ PoolCoalesceHeuristic DynamicPoolList::percent_releasable(int p } if (percentage == 0) { - return [=](const DynamicPoolList& UMPIRE_UNUSED_ARG(pool)) { return 0; }; + return PoolCoalesceHeuristic::known( + [=](const DynamicPoolList& UMPIRE_UNUSED_ARG(pool)) { return 0; }, + PoolCoalesceHeuristic::Kind::percent_releasable, + percentage); } else if (percentage == 100) { - return [=](const strategy::DynamicPoolList& pool) { return pool.getCurrentSize() == 0 ? pool.getActualSize() : 0; }; + return PoolCoalesceHeuristic::known( + [=](const strategy::DynamicPoolList& pool) { return pool.getCurrentSize() == 0 ? pool.getActualSize() : 0; }, + PoolCoalesceHeuristic::Kind::percent_releasable, + percentage); } else { float f = (float)((float)percentage / (float)100.0); - return [=](const strategy::DynamicPoolList& pool) { - // Calculate threshold in bytes from the percentage - const std::size_t threshold = static_cast(f * pool.getActualSize()); - return pool.getReleasableSize() >= threshold ? pool.getActualSize() : 0; - }; + return PoolCoalesceHeuristic::known( + [=](const strategy::DynamicPoolList& pool) { + // Calculate threshold in bytes from the percentage + const std::size_t threshold = static_cast(f * pool.getActualSize()); + return pool.getReleasableSize() >= threshold ? pool.getActualSize() : 0; + }, + PoolCoalesceHeuristic::Kind::percent_releasable, + percentage); } } @@ -179,18 +194,27 @@ PoolCoalesceHeuristic DynamicPoolList::percent_releasable_hwm(i } if (percentage == 0) { - return [=](const DynamicPoolList& UMPIRE_UNUSED_ARG(pool)) { return 0; }; + return PoolCoalesceHeuristic::known( + [=](const DynamicPoolList& UMPIRE_UNUSED_ARG(pool)) { return 0; }, + PoolCoalesceHeuristic::Kind::percent_releasable_hwm, + percentage); } else if (percentage == 100) { - return [=](const strategy::DynamicPoolList& pool) { - return pool.getCurrentSize() == 0 ? pool.getAlignedHighwaterMark() : 0; - }; + return PoolCoalesceHeuristic::known( + [=](const strategy::DynamicPoolList& pool) { + return pool.getCurrentSize() == 0 ? pool.getAlignedHighwaterMark() : 0; + }, + PoolCoalesceHeuristic::Kind::percent_releasable_hwm, + percentage); } else { float f = (float)((float)percentage / (float)100.0); - return [=](const strategy::DynamicPoolList& pool) { - // Calculate threshold in bytes from the percentage - const std::size_t threshold = static_cast(f * pool.getActualSize()); - return pool.getReleasableSize() >= threshold ? pool.getAlignedHighwaterMark() : 0; - }; + return PoolCoalesceHeuristic::known( + [=](const strategy::DynamicPoolList& pool) { + // Calculate threshold in bytes from the percentage + const std::size_t threshold = static_cast(f * pool.getActualSize()); + return pool.getReleasableSize() >= threshold ? pool.getAlignedHighwaterMark() : 0; + }, + PoolCoalesceHeuristic::Kind::percent_releasable_hwm, + percentage); } } diff --git a/src/umpire/strategy/PoolCoalesceHeuristic.hpp b/src/umpire/strategy/PoolCoalesceHeuristic.hpp index a7149d9bc..b5fe110bd 100644 --- a/src/umpire/strategy/PoolCoalesceHeuristic.hpp +++ b/src/umpire/strategy/PoolCoalesceHeuristic.hpp @@ -8,13 +8,86 @@ #define UMPIRE_PoolCoalesceHeuristic_HPP #include +#include +#include +#include namespace umpire { namespace strategy { template -using PoolCoalesceHeuristic = std::function; +class PoolCoalesceHeuristic { + public: + using Function = std::function; + + enum class Kind { opaque, percent_releasable, percent_releasable_hwm, blocks_releasable, blocks_releasable_hwm }; + + PoolCoalesceHeuristic() = default; + + PoolCoalesceHeuristic(Function function) : m_function(std::move(function)) {} + + template , PoolCoalesceHeuristic>::value>> + PoolCoalesceHeuristic(Callable&& function) : m_function(std::forward(function)) + { + } + + static PoolCoalesceHeuristic known(Function function, Kind kind, std::size_t parameter) + { + PoolCoalesceHeuristic heuristic{std::move(function)}; + heuristic.m_kind = kind; + heuristic.m_parameter = parameter; + return heuristic; + } + + std::size_t operator()(const T& pool) const + { + return m_function ? m_function(pool) : 0; + } + + explicit operator bool() const noexcept + { + return static_cast(m_function); + } + + Kind kind() const noexcept + { + return m_kind; + } + + std::size_t parameter() const noexcept + { + return m_parameter; + } + + bool is_known() const noexcept + { + return m_kind != Kind::opaque; + } + + std::string kind_string() const + { + switch (m_kind) { + case Kind::percent_releasable: + return "percent_releasable"; + case Kind::percent_releasable_hwm: + return "percent_releasable_hwm"; + case Kind::blocks_releasable: + return "blocks_releasable"; + case Kind::blocks_releasable_hwm: + return "blocks_releasable_hwm"; + case Kind::opaque: + default: + return "opaque"; + } + } + + private: + Function m_function{}; + Kind m_kind{Kind::opaque}; + std::size_t m_parameter{0}; +}; } // end of namespace strategy } // end namespace umpire diff --git a/src/umpire/strategy/QuickPool.cpp b/src/umpire/strategy/QuickPool.cpp index 7603b2780..c5f5893ab 100644 --- a/src/umpire/strategy/QuickPool.cpp +++ b/src/umpire/strategy/QuickPool.cpp @@ -347,15 +347,20 @@ void QuickPool::do_coalesce(std::size_t suggested_size) noexcept PoolCoalesceHeuristic QuickPool::blocks_releasable(std::size_t nblocks) { - return - [=](const strategy::QuickPool& pool) { return pool.getReleasableBlocks() >= nblocks ? pool.getActualSize() : 0; }; + return PoolCoalesceHeuristic::known( + [=](const strategy::QuickPool& pool) { return pool.getReleasableBlocks() >= nblocks ? pool.getActualSize() : 0; }, + PoolCoalesceHeuristic::Kind::blocks_releasable, + nblocks); } PoolCoalesceHeuristic QuickPool::blocks_releasable_hwm(std::size_t nblocks) { - return [=](const strategy::QuickPool& pool) { - return pool.getReleasableBlocks() >= nblocks ? pool.getAlignedHighwaterMark() : 0; - }; + return PoolCoalesceHeuristic::known( + [=](const strategy::QuickPool& pool) { + return pool.getReleasableBlocks() >= nblocks ? pool.getAlignedHighwaterMark() : 0; + }, + PoolCoalesceHeuristic::Kind::blocks_releasable_hwm, + nblocks); } PoolCoalesceHeuristic QuickPool::percent_releasable(int percentage) @@ -365,18 +370,27 @@ PoolCoalesceHeuristic QuickPool::percent_releasable(int percentage) fmt::format("Invalid percentage: {}, percentage must be an integer between 0 and 100", percentage)); } if (percentage == 0) { - return [=](const QuickPool& UMPIRE_UNUSED_ARG(pool)) { return 0; }; + return PoolCoalesceHeuristic::known( + [=](const QuickPool& UMPIRE_UNUSED_ARG(pool)) { return 0; }, + PoolCoalesceHeuristic::Kind::percent_releasable, + percentage); } else if (percentage == 100) { - return [=](const strategy::QuickPool& pool) { - return pool.getActualSize() == pool.getReleasableSize() ? pool.getActualSize() : 0; - }; + return PoolCoalesceHeuristic::known( + [=](const strategy::QuickPool& pool) { + return pool.getActualSize() == pool.getReleasableSize() ? pool.getActualSize() : 0; + }, + PoolCoalesceHeuristic::Kind::percent_releasable, + percentage); } else { float f = (float)((float)percentage / (float)100.0); - return [=](const strategy::QuickPool& pool) { - // Calculate threshold in bytes from the percentage - const std::size_t threshold = static_cast(f * pool.getActualSize()); - return pool.getReleasableSize() >= threshold ? pool.getActualSize() : 0; - }; + return PoolCoalesceHeuristic::known( + [=](const strategy::QuickPool& pool) { + // Calculate threshold in bytes from the percentage + const std::size_t threshold = static_cast(f * pool.getActualSize()); + return pool.getReleasableSize() >= threshold ? pool.getActualSize() : 0; + }, + PoolCoalesceHeuristic::Kind::percent_releasable, + percentage); } } @@ -387,18 +401,27 @@ PoolCoalesceHeuristic QuickPool::percent_releasable_hwm(int percentag fmt::format("Invalid percentage: {}, percentage must be an integer between 0 and 100", percentage)); } if (percentage == 0) { - return [=](const QuickPool& UMPIRE_UNUSED_ARG(pool)) { return 0; }; + return PoolCoalesceHeuristic::known( + [=](const QuickPool& UMPIRE_UNUSED_ARG(pool)) { return 0; }, + PoolCoalesceHeuristic::Kind::percent_releasable_hwm, + percentage); } else if (percentage == 100) { - return [=](const strategy::QuickPool& pool) { - return pool.getActualSize() == pool.getReleasableSize() ? pool.getAlignedHighwaterMark() : 0; - }; + return PoolCoalesceHeuristic::known( + [=](const strategy::QuickPool& pool) { + return pool.getActualSize() == pool.getReleasableSize() ? pool.getAlignedHighwaterMark() : 0; + }, + PoolCoalesceHeuristic::Kind::percent_releasable_hwm, + percentage); } else { float f = (float)((float)percentage / (float)100.0); - return [=](const strategy::QuickPool& pool) { - // Calculate threshold in bytes from the percentage - const std::size_t threshold = static_cast(f * pool.getActualSize()); - return pool.getReleasableSize() >= threshold ? pool.getAlignedHighwaterMark() : 0; - }; + return PoolCoalesceHeuristic::known( + [=](const strategy::QuickPool& pool) { + // Calculate threshold in bytes from the percentage + const std::size_t threshold = static_cast(f * pool.getActualSize()); + return pool.getReleasableSize() >= threshold ? pool.getAlignedHighwaterMark() : 0; + }, + PoolCoalesceHeuristic::Kind::percent_releasable_hwm, + percentage); } } diff --git a/src/umpire/strategy/ResourceAwarePool.cpp b/src/umpire/strategy/ResourceAwarePool.cpp index 4c00f7d5b..37d85c072 100644 --- a/src/umpire/strategy/ResourceAwarePool.cpp +++ b/src/umpire/strategy/ResourceAwarePool.cpp @@ -504,16 +504,22 @@ void ResourceAwarePool::do_coalesce(std::size_t suggested_size) noexcept PoolCoalesceHeuristic ResourceAwarePool::blocks_releasable(std::size_t nblocks) { - return [=](const strategy::ResourceAwarePool& pool) { - return pool.getReleasableBlocks() >= nblocks ? pool.getActualSize() : 0; - }; + return PoolCoalesceHeuristic::known( + [=](const strategy::ResourceAwarePool& pool) { + return pool.getReleasableBlocks() >= nblocks ? pool.getActualSize() : 0; + }, + PoolCoalesceHeuristic::Kind::blocks_releasable, + nblocks); } PoolCoalesceHeuristic ResourceAwarePool::blocks_releasable_hwm(std::size_t nblocks) { - return [=](const strategy::ResourceAwarePool& pool) { - return pool.getReleasableBlocks() >= nblocks ? pool.getAlignedHighwaterMark() : 0; - }; + return PoolCoalesceHeuristic::known( + [=](const strategy::ResourceAwarePool& pool) { + return pool.getReleasableBlocks() >= nblocks ? pool.getAlignedHighwaterMark() : 0; + }, + PoolCoalesceHeuristic::Kind::blocks_releasable_hwm, + nblocks); } PoolCoalesceHeuristic ResourceAwarePool::percent_releasable(int percentage) @@ -523,18 +529,27 @@ PoolCoalesceHeuristic ResourceAwarePool::percent_releasable(i fmt::format("Invalid percentage: {}, percentage must be an integer between 0 and 100", percentage)); } if (percentage == 0) { - return [=](const ResourceAwarePool& UMPIRE_UNUSED_ARG(pool)) { return 0; }; + return PoolCoalesceHeuristic::known( + [=](const ResourceAwarePool& UMPIRE_UNUSED_ARG(pool)) { return 0; }, + PoolCoalesceHeuristic::Kind::percent_releasable, + percentage); } else if (percentage == 100) { - return [=](const strategy::ResourceAwarePool& pool) { - return pool.getActualSize() == pool.getReleasableSize() ? pool.getActualSize() : 0; - }; + return PoolCoalesceHeuristic::known( + [=](const strategy::ResourceAwarePool& pool) { + return pool.getActualSize() == pool.getReleasableSize() ? pool.getActualSize() : 0; + }, + PoolCoalesceHeuristic::Kind::percent_releasable, + percentage); } else { float f = (float)((float)percentage / (float)100.0); - return [=](const strategy::ResourceAwarePool& pool) { - // Calculate threshold in bytes from the percentage - const std::size_t threshold = static_cast(f * pool.getActualSize()); - return pool.getReleasableSize() >= threshold ? pool.getActualSize() : 0; - }; + return PoolCoalesceHeuristic::known( + [=](const strategy::ResourceAwarePool& pool) { + // Calculate threshold in bytes from the percentage + const std::size_t threshold = static_cast(f * pool.getActualSize()); + return pool.getReleasableSize() >= threshold ? pool.getActualSize() : 0; + }, + PoolCoalesceHeuristic::Kind::percent_releasable, + percentage); } } @@ -545,18 +560,27 @@ PoolCoalesceHeuristic ResourceAwarePool::percent_releasable_h fmt::format("Invalid percentage: {}, percentage must be an integer between 0 and 100", percentage)); } if (percentage == 0) { - return [=](const ResourceAwarePool& UMPIRE_UNUSED_ARG(pool)) { return 0; }; + return PoolCoalesceHeuristic::known( + [=](const ResourceAwarePool& UMPIRE_UNUSED_ARG(pool)) { return 0; }, + PoolCoalesceHeuristic::Kind::percent_releasable_hwm, + percentage); } else if (percentage == 100) { - return [=](const strategy::ResourceAwarePool& pool) { - return pool.getActualSize() == pool.getReleasableSize() ? pool.getAlignedHighwaterMark() : 0; - }; + return PoolCoalesceHeuristic::known( + [=](const strategy::ResourceAwarePool& pool) { + return pool.getActualSize() == pool.getReleasableSize() ? pool.getAlignedHighwaterMark() : 0; + }, + PoolCoalesceHeuristic::Kind::percent_releasable_hwm, + percentage); } else { float f = (float)((float)percentage / (float)100.0); - return [=](const strategy::ResourceAwarePool& pool) { - // Calculate threshold in bytes from the percentage - const std::size_t threshold = static_cast(f * pool.getActualSize()); - return pool.getReleasableSize() >= threshold ? pool.getAlignedHighwaterMark() : 0; - }; + return PoolCoalesceHeuristic::known( + [=](const strategy::ResourceAwarePool& pool) { + // Calculate threshold in bytes from the percentage + const std::size_t threshold = static_cast(f * pool.getActualSize()); + return pool.getReleasableSize() >= threshold ? pool.getAlignedHighwaterMark() : 0; + }, + PoolCoalesceHeuristic::Kind::percent_releasable_hwm, + percentage); } } diff --git a/tests/integration/replay/CMakeLists.txt b/tests/integration/replay/CMakeLists.txt index 35f0f9863..e824db688 100644 --- a/tests/integration/replay/CMakeLists.txt +++ b/tests/integration/replay/CMakeLists.txt @@ -15,69 +15,26 @@ if (UMPIRE_ENABLE_OPENMP_TARGET) openmp) endif() -set(raw_replay_file "output" ) -set(replay_config "") - -if (UMPIRE_ENABLE_NUMA) - set(replay_config "${replay_config}numa_" ) -endif() - -if (UMPIRE_ENABLE_CUDA OR UMPIRE_ENABLE_HIP) - set(replay_config "${replay_config}devices_" ) - - if (UMPIRE_ENABLE_DEVICE_CONST) - set(replay_config "${replay_config}const_") - endif () -endif () - -if (UMPIRE_ENABLE_OPENMP_TARGET) - set(replay_config "${replay_config}omp_") -endif () - -if (UMPIRE_ENABLE_SQLITE_EXPERIMENTAL) - set(replay_config "${replay_config}sqlite_") -endif () - -set(replay_config_stripped "") -string(REGEX REPLACE "_$" "" replay_config_stripped "${replay_config}") -set(raw_replay_file "${raw_replay_file}.${replay_config_stripped}.replay") -string(REGEX REPLACE "\\.+" "." replay_file "${raw_replay_file}") - blt_add_executable( NAME replay_tests SOURCES replay_tests.cpp DEPENDS_ON ${replay_integration_tests_depends}) -add_custom_target( - regen_replay_output - COMMAND ${CMAKE_COMMAND} -E env UMPIRE_REPLAY=On $ && mv *.stats ${CMAKE_CURRENT_SOURCE_DIR}/${replay_file} && rm -f *.stats) - -message(STATUS "Looking for ${replay_file} reference output...") -if (EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/${replay_file}) - if (BASH_PROGRAM) - message(STATUS "Using ${replay_file} as good replay test data") - configure_file( - "${CMAKE_CURRENT_SOURCE_DIR}/${replay_file}" - "${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/test_output.good" - COPYONLY - ) +blt_add_executable( + NAME replay_failure_tests + SOURCES replay_failure_tests.cpp + DEPENDS_ON ${replay_integration_tests_depends}) +if (BASH_PROGRAM) + if (NOT C_COMPILER_FAMILY_IS_PGI) add_test( - NAME replay_coverage_tests - COMMAND ${BASH_PROGRAM} ${CMAKE_CURRENT_SOURCE_DIR}/check_replay_coverage.bash - WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/src) + NAME replay_tests + COMMAND ${BASH_PROGRAM} ${CMAKE_CURRENT_SOURCE_DIR}/replay_tests.bash ${CMAKE_RUNTIME_OUTPUT_DIRECTORY} ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}) - if (NOT C_COMPILER_FAMILY_IS_PGI) - add_test( - NAME replay_tests - COMMAND ${BASH_PROGRAM} ${CMAKE_CURRENT_SOURCE_DIR}/replay_tests.bash ${CMAKE_RUNTIME_OUTPUT_DIRECTORY} ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}) - endif() - - else () - - message(STATUS "Bash not configured, replay tests disabled.") - - endif () + add_test( + NAME replay_failure_tests + COMMAND ${BASH_PROGRAM} ${CMAKE_CURRENT_SOURCE_DIR}/replay_failure_tests.bash ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}) + endif() else () - message(WARNING "No replay reference output available for current build configuration, replay test disabled.") + message(STATUS "Bash not configured, replay tests disabled.") endif () diff --git a/tests/integration/replay/replay_failure_tests.bash b/tests/integration/replay/replay_failure_tests.bash new file mode 100644 index 000000000..83f34a306 --- /dev/null +++ b/tests/integration/replay/replay_failure_tests.bash @@ -0,0 +1,59 @@ +#!/bin/bash +############################################################################## +# Copyright (c) 2016-26, Lawrence Livermore National Security, LLC and Umpire +# project contributors. See the COPYRIGHT file for details. +# +# SPDX-License-Identifier: (MIT) +############################################################################## +set -euo pipefail + +replay_tests_dir=$(cd "$1" && pwd) +testprogram=$replay_tests_dir/replay_failure_tests +topdir=$replay_tests_dir/.. + +cleanupandexit() { + local status=$1 + local mydir + mydir=$(pwd) + cd "$topdir" + find . -maxdepth 1 -name 'umpire.*.stats' | xargs -r rm -f + cd "$mydir" + exit "$status" +} + +trap 'cleanupandexit 1' ERR + +cd "$topdir" +find . -maxdepth 1 -name 'umpire.*.stats' | xargs -r rm -f + +echo "UMPIRE_REPLAY='On' $testprogram" +UMPIRE_REPLAY="On" "$testprogram" + +generated_trace=$(find . -maxdepth 1 -name 'umpire.*.stats' | head -n 1) +if [ -z "$generated_trace" ]; then + echo "Failed: replay failure test binary did not generate a .stats file" + cleanupandexit 1 +fi + +grep '"name":"bad_alignment_allocator".*"status":"pending"' "$generated_trace" >/dev/null +if grep '"name":"bad_alignment_allocator".*"status":"committed"' "$generated_trace" >/dev/null; then + echo "Failed: constructor failure was incorrectly committed in replay trace" + cleanupandexit 1 +fi + +limited_id=$(grep '"name":"size_limited_allocator".*"status":"committed"' "$generated_trace" | \ + sed -E 's/.*"allocator_id":"([^"]+)".*/\1/' | head -n 1) + +if [ -z "$limited_id" ]; then + echo "Failed: size-limited allocator creation was not committed" + cleanupandexit 1 +fi + +grep "\"allocator_id\":\"$limited_id\"" "$generated_trace" | grep '"op":"allocate"' | grep '"size":2' | grep '"status":"pending"' >/dev/null +if grep "\"allocator_id\":\"$limited_id\"" "$generated_trace" | grep '"op":"allocate"' | grep '"size":2' | grep '"status":"committed"' >/dev/null; then + echo "Failed: failed allocation was incorrectly committed in replay trace" + cleanupandexit 1 +fi + +trap - ERR +cleanupandexit 0 diff --git a/tests/integration/replay/replay_failure_tests.cpp b/tests/integration/replay/replay_failure_tests.cpp new file mode 100644 index 000000000..5abac2d75 --- /dev/null +++ b/tests/integration/replay/replay_failure_tests.cpp @@ -0,0 +1,51 @@ +////////////////////////////////////////////////////////////////////////////// +// Copyright (c) 2016-26, Lawrence Livermore National Security, LLC and Umpire +// project contributors. See the COPYRIGHT file for details. +// +// SPDX-License-Identifier: (MIT) +////////////////////////////////////////////////////////////////////////////// + +#include + +#include "umpire/Allocator.hpp" +#include "umpire/ResourceManager.hpp" +#include "umpire/Umpire.hpp" +#include "umpire/config.hpp" +#include "umpire/strategy/AlignedAllocator.hpp" +#include "umpire/strategy/SizeLimiter.hpp" + +int main(int, char**) +{ + auto& rm = umpire::ResourceManager::getInstance(); + auto host = rm.getAllocator("HOST"); + + bool constructor_failed{false}; + try { + auto invalid = rm.makeAllocator("bad_alignment_allocator", host, 8); + UMPIRE_USE_VAR(invalid); + } catch (...) { + constructor_failed = true; + } + + if (!constructor_failed) { + std::cerr << "Expected invalid aligned allocator construction to fail.\n"; + return 1; + } + + auto limited = rm.makeAllocator("size_limited_allocator", host, 1); + + bool allocation_failed{false}; + try { + auto* ptr = limited.allocate(2); + UMPIRE_USE_VAR(ptr); + } catch (...) { + allocation_failed = true; + } + + if (!allocation_failed) { + std::cerr << "Expected size-limited allocation to fail.\n"; + return 1; + } + + return 0; +} diff --git a/tests/integration/replay/replay_tests.bash b/tests/integration/replay/replay_tests.bash index 726a13d52..ba9d57c80 100644 --- a/tests/integration/replay/replay_tests.bash +++ b/tests/integration/replay/replay_tests.bash @@ -5,6 +5,8 @@ # # SPDX-License-Identifier: (MIT) ############################################################################## +set -euo pipefail + replay_tests_dir=$1 tools_dir=$2 testprogram=$replay_tests_dir/replay_tests @@ -12,49 +14,82 @@ diffprogram=$tools_dir/replaydiff replayprogram=$tools_dir/replay topdir=$tools_dir/.. -function cleanupandexit { - mydir=`pwd` - cd $topdir - for f in `find . -name '*.replay.bin' ; find -name '*.stats'`; - do - rm -f $f - done - cd $mydir - exit $1 +cleanupandexit() { + local status=$1 + local mydir + mydir=$(pwd) + cd "$topdir" + find . -name '*.stats' -o -name 'replay*.ult' | xargs -r rm -f + cd "$mydir" + exit "$status" } -# -# The following program will generate a file of Umpire activity that -# will be replayed. -# +trap 'cleanupandexit 1' ERR + +cd "$topdir" +find . -name '*.stats' -o -name 'replay*.ult' | xargs -r rm -f + echo "UMPIRE_REPLAY='On' $testprogram" -UMPIRE_REPLAY="On" $testprogram -if [ $? -ne 0 ]; then - echo "Failed: Unable to run $testprogram" - cleanupandexit 1 +UMPIRE_REPLAY="On" "$testprogram" + +generated_trace=$(find . -maxdepth 1 -name 'umpire.*.stats' | head -n 1) +if [ -z "$generated_trace" ]; then + echo "Failed: replay test binary did not generate a .stats file" + cleanupandexit 1 fi -echo $diffprogram -q --recompile $replay_tests_dir/test_output.good umpire.*.0.stats -$diffprogram -q --recompile $replay_tests_dir/test_output.good umpire.*.0.stats -if [ $? -ne 0 ]; then - echo "Diff failed on file generated by test" - cleanupandexit 1 +header=$(head -n 1 "$generated_trace") +echo "$header" | grep '"kind":"umpire_replay"' >/dev/null +echo "$header" | grep '"schema":"v2"' >/dev/null +grep '"status":"pending"' "$generated_trace" >/dev/null +grep '"status":"committed"' "$generated_trace" >/dev/null + +mv "$generated_trace" replay.original.stats + +echo "UMPIRE_REPLAY='On' $replayprogram -d -q -i replay.original.stats" +UMPIRE_REPLAY="On" "$replayprogram" -d -q -i replay.original.stats + +replayed_trace=$(find . -maxdepth 1 -name 'umpire.*.stats' | head -n 1) +if [ -z "$replayed_trace" ]; then + echo "Failed: replay tool did not generate a replayed .stats file" + cleanupandexit 1 fi -echo "/bin/mv umpire*stats replay.replay" -/bin/mv umpire*stats replay.replay -echo "UMPIRE_REPLAY='On' $replayprogram -q --recompile -i replay.replay" -UMPIRE_REPLAY="On" $replayprogram -i replay.replay -q --recompile -if [ $? -ne 0 ]; then - echo "$replayprogram Failed" - cleanupandexit 1 +if [ ! -s "$(find . -maxdepth 1 -name 'replay*.ult' | head -n 1)" ]; then + echo "Failed: replay tool did not emit a non-empty .ult file" + cleanupandexit 1 fi -echo "$diffprogram -q --recompile $replay_tests_dir/test_output.good umpire.*.0.stats" -$diffprogram -q --recompile $replay_tests_dir/test_output.good umpire.*.0.stats -if [ $? -ne 0 ]; then - echo "Diff failed on file generated by replay tool" - cleanupandexit 1 +echo "$diffprogram -q replay.original.stats $replayed_trace" +"$diffprogram" -q replay.original.stats "$replayed_trace" + +printf '%s\n' \ + '{"kind":"umpire_replay","schema":"v2","process":{"pid":1,"rank":0},"umpire_version":"test"}' \ + '{"op":"make_allocator","seq":1,"allocator_id":"a1","name":"HOST","strategy":"MemoryResource","tracking":true,"args":{"resource_name":"HOST","traits":{"unified":false,"ipc":false,"size":0,"vendor":"unknown","kind":"unknown","used_for":"any","resource":"host","scope":"unknown","granularity":"unknown","tracking":true}},"status":"pending"}' \ + '{"op":"make_allocator","seq":1,"allocator_id":"a1","name":"HOST","strategy":"MemoryResource","tracking":true,"args":{"resource_name":"HOST","traits":{"unified":false,"ipc":false,"size":0,"vendor":"unknown","kind":"unknown","used_for":"any","resource":"host","scope":"unknown","granularity":"unknown","tracking":true}},"status":"committed"}' \ + '{"op":"allocate","seq":2,"allocator_id":"a1","allocation_id":"m1","size":32,"status":"pending"}' \ + > replay.pending.stats + +"$replayprogram" -q -i replay.pending.stats + +printf '%s\n' \ + '{"kind":"umpire_replay","schema":"v2","process":{"pid":1,"rank":0},"umpire_version":"test"}' \ + '{"op":"make_allocator","seq":1,"allocator_id":"a1","name":"HOST","strategy":"MemoryResource","tracking":true,"args":{"resource_name":"HOST","traits":{"unified":false,"ipc":false,"size":0,"vendor":"unknown","kind":"unknown","used_for":"any","resource":"host","scope":"unknown","granularity":"unknown","tracking":true}},"status":"pending"}' \ + '{"op":"make_allocator","seq":1,"allocator_id":"a1","name":"HOST","strategy":"MemoryResource","tracking":true,"args":{"resource_name":"HOST","traits":{"unified":false,"ipc":false,"size":0,"vendor":"unknown","kind":"unknown","used_for":"any","resource":"host","scope":"unknown","granularity":"unknown","tracking":true}},"status":"committed"}' \ + '{"op":"allocate","seq":2,"allocator_id":"a1","allocation_id":"m1","size":32,"status":"committed"}' \ + > replay.invalid.stats + +trap - ERR +set +e +"$replayprogram" -q -i replay.invalid.stats >/dev/null 2>&1 +invalid_status=$? +set -e +trap 'cleanupandexit 1' ERR + +if [ "$invalid_status" -eq 0 ]; then + echo "Failed: replay accepted a committed command without a pending lifecycle record" + cleanupandexit 1 fi +trap - ERR cleanupandexit 0 diff --git a/tests/integration/replay/replay_tests.cpp b/tests/integration/replay/replay_tests.cpp index 436481906..ec6ddd643 100644 --- a/tests/integration/replay/replay_tests.cpp +++ b/tests/integration/replay/replay_tests.cpp @@ -12,7 +12,7 @@ #include "umpire/ResourceManager.hpp" #include "umpire/Umpire.hpp" #include "umpire/config.hpp" -#include "umpire/op/MemoryOperation.hpp" +#include "umpire/strategy/AlignedAllocator.hpp" #include "umpire/strategy/AllocationAdvisor.hpp" #include "umpire/strategy/AllocationPrefetcher.hpp" #include "umpire/strategy/AllocationStrategy.hpp" @@ -21,7 +21,10 @@ #include "umpire/strategy/MixedPool.hpp" #include "umpire/strategy/MonotonicAllocationStrategy.hpp" #include "umpire/strategy/NamedAllocationStrategy.hpp" +#include "umpire/strategy/NamingShim.hpp" #include "umpire/strategy/QuickPool.hpp" +#include "umpire/strategy/ResourceAwarePool.hpp" +#include "umpire/strategy/SizeLimiter.hpp" #include "umpire/strategy/SlotPool.hpp" #include "umpire/strategy/ThreadSafeAllocator.hpp" #include "umpire/util/wrap_allocator.hpp" @@ -36,100 +39,6 @@ namespace replay_test { const int TEST_ALLOCATIONS{3}; const std::size_t ALLOCATION_SIZE{32}; -void testCopy(std::string name) -{ - constexpr std::size_t MAX_ALLOCATION_SIZE = 128; - constexpr std::size_t OFFSET = 12; - constexpr std::size_t COPYAMOUNT = 64; - - auto& rm = umpire::ResourceManager::getInstance(); - auto dst_allocator = rm.getAllocator(name); - auto src_allocator = rm.getAllocator("HOST"); - - char* src_buffer = static_cast(src_allocator.allocate(MAX_ALLOCATION_SIZE)); - char* dst_buffer = static_cast(dst_allocator.allocate(MAX_ALLOCATION_SIZE)); - - rm.copy(dst_buffer, src_buffer + OFFSET, COPYAMOUNT); - - src_allocator.deallocate(src_buffer); - dst_allocator.deallocate(dst_buffer); -} - -void testMove(std::string name) -{ - constexpr std::size_t MAX_ALLOCATION_SIZE = 128; - - auto& rm = umpire::ResourceManager::getInstance(); - auto dst_allocator = rm.getAllocator(name); - auto src_allocator = rm.getAllocator("HOST"); - - char* src_buffer = static_cast(src_allocator.allocate(MAX_ALLOCATION_SIZE)); - void* dst_buffer = rm.move(src_buffer, dst_allocator); - - dst_allocator.deallocate(dst_buffer); -} - -void testReallocation(std::string name) -{ - constexpr std::size_t MAX_ALLOCATION_SIZE = 32; - auto& rm = umpire::ResourceManager::getInstance(); - auto alloc = rm.getAllocator(name); - - // - // Test by using 100% reallocate - // - for (std::size_t size = 0; size <= MAX_ALLOCATION_SIZE; size = size * 2 + 1) { - auto default_alloc = rm.getDefaultAllocator(); - - int buffer_size = size; - - rm.setDefaultAllocator(alloc); - int* buffer = static_cast(rm.reallocate(nullptr, buffer_size * sizeof(*buffer))); - - rm.setDefaultAllocator(default_alloc); - - buffer_size *= 3; // Reallocate to a larger size. - buffer = static_cast(rm.reallocate(buffer, buffer_size * sizeof(*buffer))); - - buffer_size /= 5; // Reallocate to a smaller size. - buffer = static_cast(rm.reallocate(buffer, buffer_size * sizeof(*buffer))); - - alloc.deallocate(buffer); - } - - // - // Test with first allocation being from normal allocate - // - for (std::size_t size = 0; size <= MAX_ALLOCATION_SIZE; size = size * 2 + 1) { - int buffer_size = size; - int* buffer = static_cast(alloc.allocate(buffer_size * sizeof(*buffer))); - - buffer_size *= 3; // Reallocate to a larger size. - buffer = static_cast(rm.reallocate(buffer, buffer_size * sizeof(*buffer))); - - buffer_size /= 5; // Reallocate to a smaller size. - buffer = static_cast(rm.reallocate(buffer, buffer_size * sizeof(*buffer))); - - alloc.deallocate(buffer); - } - - // - // Test using specific allocator for reallocate - // - for (std::size_t size = 0; size <= MAX_ALLOCATION_SIZE; size = size * 2 + 1) { - int buffer_size = size; - int* buffer = static_cast(alloc.allocate(buffer_size * sizeof(*buffer))); - - buffer_size *= 3; // Reallocate to a larger size. - buffer = static_cast(rm.reallocate(buffer, buffer_size * sizeof(*buffer), alloc)); - - buffer_size /= 5; // Reallocate to a smaller size. - buffer = static_cast(rm.reallocate(buffer, buffer_size * sizeof(*buffer), alloc)); - - alloc.deallocate(buffer); - } -} - void testAllocation(std::string name) { auto& rm = umpire::ResourceManager::getInstance(); @@ -176,10 +85,7 @@ static void runTest() #endif for (auto basename : allocators) { - testCopy(basename); - testMove(basename); testAllocation(basename); - testReallocation(basename); auto base_alloc = rm.getAllocator(basename); std::string name; @@ -254,6 +160,7 @@ static void runTest() auto pa2 = 1 * 1024; // min allocation size auto pa3 = 128; auto pa4 = umpire::strategy::QuickPool::percent_releasable(50); + auto rpa4 = umpire::strategy::ResourceAwarePool::percent_releasable(50); name = basename + "_Pool_spec_"; testAllocator(name + "0", base_alloc); testAllocator(name + "1", base_alloc, pa1); @@ -295,6 +202,21 @@ static void runTest() name = basename + "_NamedAllocationStrategy_no_instrospection_spec_"; testAllocator(name, base_alloc); + name = basename + "_NamingShim_spec_"; + testAllocator(name, base_alloc); + name = basename + "_NamingShim_no_instrospection_spec_"; + testAllocator(name, base_alloc); + + name = basename + "_AlignedAllocator_spec_"; + testAllocator(name, base_alloc, 64); + name = basename + "_AlignedAllocator_no_instrospection_spec_"; + testAllocator(name, base_alloc, 64); + + name = basename + "_SizeLimiter_spec_"; + testAllocator(name, base_alloc, 1024); + name = basename + "_SizeLimiter_no_instrospection_spec_"; + testAllocator(name, base_alloc, 1024); + auto sa1 = 64; // Slots name = basename + "_SlotPool_spec_"; testAllocator(name, base_alloc, sa1); @@ -306,6 +228,19 @@ static void runTest() name = basename + "_ThreadSafeAllocator_no_instrospection_spec_"; testAllocator(name, base_alloc); + name = basename + "_ResourceAwarePool_spec_"; + testAllocator(name + "0", base_alloc); + testAllocator(name + "1", base_alloc, pa1); + testAllocator(name + "2", base_alloc, pa1, pa2); + testAllocator(name + "3", base_alloc, pa1, pa2, pa3); + testAllocator(name + "4", base_alloc, pa1, pa2, pa3, rpa4); + name = basename + "_ResourceAwarePool_no_instrospection_spec_"; + testAllocator(name + "0", base_alloc); + testAllocator(name + "1", base_alloc, pa1); + testAllocator(name + "2", base_alloc, pa1, pa2); + testAllocator(name + "3", base_alloc, pa1, pa2, pa3); + testAllocator(name + "4", base_alloc, pa1, pa2, pa3, rpa4); + auto fpa1 = ALLOCATION_SIZE; // object_bytes auto fpa2 = 1024; // objects_per_pool name = basename + "_FixedPool_spec_"; @@ -316,14 +251,6 @@ static void runTest() testAllocator(name + "2", base_alloc, fpa1, fpa2); } - // test registering external pointers - { - int* data[10]; - umpire::register_external_allocation( - data, umpire::util::AllocationRecord(data, 10 * sizeof(int), rm.getAllocator("HOST").getAllocationStrategy(), - "external array")); - umpire::deregister_external_allocation(data); - } } } // namespace replay_test diff --git a/tools/replay/CMakeLists.txt b/tools/replay/CMakeLists.txt index 6ea131e17..420127a1f 100644 --- a/tools/replay/CMakeLists.txt +++ b/tools/replay/CMakeLists.txt @@ -12,17 +12,13 @@ if (UMPIRE_ENABLE_BACKTRACE_SYMBOLS) endif() set(replay_headers + ReplayConstructorRegistry.hpp ReplayInterpreter.hpp - ReplayInterpreter.inl - ReplayMacros.hpp - ReplayOperationManager.hpp - ReplayOptions.hpp - ReplayFile.hpp) + ReplayOptions.hpp) set(replay_sources - ReplayInterpreter.cpp - ReplayOperationManager.cpp - ReplayFile.cpp) + ReplayConstructorRegistry.cpp + ReplayInterpreter.cpp) blt_add_executable( NAME replay diff --git a/tools/replay/ReplayConstructorRegistry.cpp b/tools/replay/ReplayConstructorRegistry.cpp new file mode 100644 index 000000000..1858a8932 --- /dev/null +++ b/tools/replay/ReplayConstructorRegistry.cpp @@ -0,0 +1,276 @@ +////////////////////////////////////////////////////////////////////////////// +// Copyright (c) 2016-26, Lawrence Livermore National Security, LLC and Umpire +// project contributors. See the COPYRIGHT file for details. +// +// SPDX-License-Identifier: (MIT) +////////////////////////////////////////////////////////////////////////////// + +#if !defined(_MSC_VER) + +#include "ReplayConstructorRegistry.hpp" + +#include + +#include "umpire/Tracking.hpp" +#include "umpire/replay/Replay.hpp" +#include "umpire/strategy/AlignedAllocator.hpp" +#include "umpire/strategy/AllocationAdvisor.hpp" +#include "umpire/strategy/AllocationPrefetcher.hpp" +#include "umpire/strategy/DynamicPoolList.hpp" +#include "umpire/strategy/FixedPool.hpp" +#include "umpire/strategy/MixedPool.hpp" +#include "umpire/strategy/MonotonicAllocationStrategy.hpp" +#include "umpire/strategy/NamedAllocationStrategy.hpp" +#include "umpire/strategy/NamingShim.hpp" +#include "umpire/strategy/QuickPool.hpp" +#include "umpire/strategy/ResourceAwarePool.hpp" +#include "umpire/strategy/SizeLimiter.hpp" +#include "umpire/strategy/SlotPool.hpp" +#include "umpire/strategy/ThreadSafeAllocator.hpp" +#include "umpire/util/error.hpp" + +#if defined(UMPIRE_ENABLE_NUMA) +#include "umpire/strategy/NumaPolicy.hpp" +#endif + +#if defined(UMPIRE_ENABLE_MPI) && defined(UMPIRE_ENABLE_IPC_SHARED_MEMORY) && \ + (defined(UMPIRE_ENABLE_CUDA) || defined(UMPIRE_ENABLE_HIP)) +#include "umpire/strategy/DeviceIpcAllocator.hpp" +#endif + +namespace { + +umpire::Tracking trackingMode(const ReplayAllocatorSpec& spec) +{ + return spec.tracking ? umpire::Tracking::Tracked : umpire::Tracking::Untracked; +} + +umpire::Allocator allocatorArg(const ReplayAllocatorSpec& spec, ReplayContext& context, const char* key) +{ + const auto id = spec.args.at(key).get(); + auto found = context.allocators.find(id); + if (found == context.allocators.end()) { + UMPIRE_ERROR(umpire::runtime_error, + fmt::format("Replay allocator {} references unknown parent allocator {}", spec.name, id)); + } + return found->second; +} + +template +T jsonValue(const nlohmann::json& args, const char* key) +{ + return args.at(key).get(); +} + +template +T jsonValueOr(const nlohmann::json& args, const char* key, T fallback) +{ + auto found = args.find(key); + return found == args.end() ? fallback : found->get(); +} + +#if defined(UMPIRE_ENABLE_MPI) && defined(UMPIRE_ENABLE_IPC_SHARED_MEMORY) && \ + (defined(UMPIRE_ENABLE_CUDA) || defined(UMPIRE_ENABLE_HIP)) +umpire::MemoryResourceTraits::shared_scope sharedScopeFromString(const std::string& scope) +{ + if (scope == "node") { + return umpire::MemoryResourceTraits::shared_scope::node; + } + if (scope == "socket") { + return umpire::MemoryResourceTraits::shared_scope::socket; + } + return umpire::MemoryResourceTraits::shared_scope::unknown; +} +#endif + +template +umpire::strategy::PoolCoalesceHeuristic parseHeuristic(const nlohmann::json& heuristic_json) +{ + const auto kind = heuristic_json.at("kind").get(); + const auto parameter = heuristic_json.at("parameter").get(); + + if (kind == "percent_releasable") { + return Pool::percent_releasable(static_cast(parameter)); + } + if (kind == "percent_releasable_hwm") { + return Pool::percent_releasable_hwm(static_cast(parameter)); + } + if (kind == "blocks_releasable") { + return Pool::blocks_releasable(parameter); + } + if (kind == "blocks_releasable_hwm") { + return Pool::blocks_releasable_hwm(parameter); + } + + UMPIRE_ERROR(umpire::runtime_error, fmt::format("Unsupported replay heuristic kind {}", kind)); +} + +} // namespace + +ReplayConstructorRegistry::ReplayConstructorRegistry() +{ + registerFactory("MemoryResource", [](const ReplayAllocatorSpec& spec, ReplayContext& context) { + const auto resource_name = jsonValue(spec.args, "resource_name"); + const auto traits_it = spec.args.find("traits"); + + umpire::Allocator allocator = (traits_it == spec.args.end()) + ? context.resource_manager.makeResource(resource_name) + : context.resource_manager.makeResource( + resource_name, umpire::replay::deserialize_memory_resource_traits(*traits_it)); + + if (spec.name != resource_name) { + context.resource_manager.addAlias(spec.name, allocator); + } + + return allocator; + }); + + registerFactory("AllocationAdvisor", [](const ReplayAllocatorSpec& spec, ReplayContext& context) { + auto parent = allocatorArg(spec, context, "parent_allocator"); + const auto advice = jsonValue(spec.args, "advice_operation"); + const auto device_id = jsonValueOr(spec.args, "device_id", 0); + const auto accessing_it = spec.args.find("accessing_allocator"); + + if (accessing_it == spec.args.end()) { + return context.resource_manager.makeAllocator(spec.name, trackingMode(spec), + parent, advice, device_id); + } + + auto accessing = allocatorArg(spec, context, "accessing_allocator"); + return context.resource_manager.makeAllocator(spec.name, trackingMode(spec), + parent, advice, accessing, + device_id); + }); + + registerFactory("AllocationPrefetcher", [](const ReplayAllocatorSpec& spec, ReplayContext& context) { + return context.resource_manager.makeAllocator( + spec.name, trackingMode(spec), allocatorArg(spec, context, "parent_allocator"), + jsonValueOr(spec.args, "device_id", 0)); + }); + + registerFactory("AlignedAllocator", [](const ReplayAllocatorSpec& spec, ReplayContext& context) { + return context.resource_manager.makeAllocator( + spec.name, trackingMode(spec), allocatorArg(spec, context, "parent_allocator"), + jsonValueOr(spec.args, "alignment", 16)); + }); + + registerFactory("DynamicPoolList", [](const ReplayAllocatorSpec& spec, ReplayContext& context) { + return context.resource_manager.makeAllocator( + spec.name, trackingMode(spec), allocatorArg(spec, context, "parent_allocator"), + jsonValueOr(spec.args, "initial_alloc_size", + umpire::strategy::DynamicPoolList::s_default_first_block_size), + jsonValueOr(spec.args, "min_alloc_size", + umpire::strategy::DynamicPoolList::s_default_next_block_size), + jsonValueOr(spec.args, "alignment", umpire::strategy::DynamicPoolList::s_default_alignment), + parseHeuristic(spec.args.at("heuristic"))); + }); + + registerFactory("FixedPool", [](const ReplayAllocatorSpec& spec, ReplayContext& context) { + return context.resource_manager.makeAllocator( + spec.name, trackingMode(spec), allocatorArg(spec, context, "parent_allocator"), + jsonValue(spec.args, "object_bytes"), + jsonValueOr(spec.args, "objects_per_pool", 64 * sizeof(int) * 8)); + }); + + registerFactory("MixedPool", [](const ReplayAllocatorSpec& spec, ReplayContext& context) { + return context.resource_manager.makeAllocator( + spec.name, trackingMode(spec), allocatorArg(spec, context, "parent_allocator"), + jsonValueOr(spec.args, "smallest_fixed_obj_size", (1 << 8)), + jsonValueOr(spec.args, "largest_fixed_obj_size", (1 << 17)), + jsonValueOr(spec.args, "max_initial_fixed_pool_size", 1024 * 1024 * 2), + jsonValueOr(spec.args, "fixed_size_multiplier", 16), + jsonValueOr(spec.args, "quick_pool_initial_alloc_size", 512 * 1024 * 1024), + jsonValueOr(spec.args, "quick_pool_min_alloc_size", 1 * 1024 * 1024), + jsonValueOr(spec.args, "quick_pool_align_bytes", 16), + parseHeuristic(spec.args.at("heuristic"))); + }); + + registerFactory("MonotonicAllocationStrategy", [](const ReplayAllocatorSpec& spec, ReplayContext& context) { + return context.resource_manager.makeAllocator( + spec.name, trackingMode(spec), allocatorArg(spec, context, "parent_allocator"), + jsonValue(spec.args, "capacity")); + }); + + registerFactory("NamedAllocationStrategy", [](const ReplayAllocatorSpec& spec, ReplayContext& context) { + return context.resource_manager.makeAllocator( + spec.name, trackingMode(spec), allocatorArg(spec, context, "parent_allocator")); + }); + + registerFactory("NamingShim", [](const ReplayAllocatorSpec& spec, ReplayContext& context) { + return context.resource_manager.makeAllocator( + spec.name, trackingMode(spec), allocatorArg(spec, context, "parent_allocator")); + }); + + registerFactory("QuickPool", [](const ReplayAllocatorSpec& spec, ReplayContext& context) { + return context.resource_manager.makeAllocator( + spec.name, trackingMode(spec), allocatorArg(spec, context, "parent_allocator"), + jsonValueOr(spec.args, "initial_alloc_size", umpire::strategy::QuickPool::s_default_first_block_size), + jsonValueOr(spec.args, "min_alloc_size", umpire::strategy::QuickPool::s_default_next_block_size), + jsonValueOr(spec.args, "alignment", umpire::strategy::QuickPool::s_default_alignment), + parseHeuristic(spec.args.at("heuristic"))); + }); + + registerFactory("ResourceAwarePool", [](const ReplayAllocatorSpec& spec, ReplayContext& context) { + return context.resource_manager.makeAllocator( + spec.name, trackingMode(spec), allocatorArg(spec, context, "parent_allocator"), + jsonValueOr(spec.args, "initial_alloc_size", + umpire::strategy::ResourceAwarePool::s_default_first_block_size), + jsonValueOr(spec.args, "min_alloc_size", + umpire::strategy::ResourceAwarePool::s_default_next_block_size), + jsonValueOr(spec.args, "alignment", umpire::strategy::ResourceAwarePool::s_default_alignment), + parseHeuristic(spec.args.at("heuristic"))); + }); + + registerFactory("SizeLimiter", [](const ReplayAllocatorSpec& spec, ReplayContext& context) { + return context.resource_manager.makeAllocator( + spec.name, trackingMode(spec), allocatorArg(spec, context, "parent_allocator"), + jsonValue(spec.args, "size_limit")); + }); + + registerFactory("SlotPool", [](const ReplayAllocatorSpec& spec, ReplayContext& context) { + return context.resource_manager.makeAllocator( + spec.name, trackingMode(spec), allocatorArg(spec, context, "parent_allocator"), + jsonValue(spec.args, "slots")); + }); + + registerFactory("ThreadSafeAllocator", [](const ReplayAllocatorSpec& spec, ReplayContext& context) { + return context.resource_manager.makeAllocator( + spec.name, trackingMode(spec), allocatorArg(spec, context, "parent_allocator")); + }); + +#if defined(UMPIRE_ENABLE_NUMA) + registerFactory("NumaPolicy", [](const ReplayAllocatorSpec& spec, ReplayContext& context) { + return context.resource_manager.makeAllocator( + spec.name, trackingMode(spec), allocatorArg(spec, context, "parent_allocator"), + jsonValue(spec.args, "numa_node")); + }); +#endif + +#if defined(UMPIRE_ENABLE_MPI) && defined(UMPIRE_ENABLE_IPC_SHARED_MEMORY) && \ + (defined(UMPIRE_ENABLE_CUDA) || defined(UMPIRE_ENABLE_HIP)) + registerFactory("DeviceIpcAllocator", [](const ReplayAllocatorSpec& spec, ReplayContext& context) { + return context.resource_manager.makeAllocator( + spec.name, trackingMode(spec), allocatorArg(spec, context, "device_allocator"), + sharedScopeFromString(jsonValue(spec.args, "shared_scope")), + jsonValueOr(spec.args, "shared_memory_size", 1024 * 1024)); + }); +#endif +} + +umpire::Allocator ReplayConstructorRegistry::construct(const ReplayAllocatorSpec& spec, ReplayContext& context) const +{ + auto found = m_factories.find(spec.strategy); + if (found == m_factories.end()) { + UMPIRE_ERROR(umpire::runtime_error, + fmt::format("Replay strategy {} is not supported in this build", spec.strategy)); + } + + return found->second(spec, context); +} + +void ReplayConstructorRegistry::registerFactory(const std::string& strategy, Factory factory) +{ + m_factories[strategy] = std::move(factory); +} + +#endif // !defined(_MSC_VER) diff --git a/tools/replay/ReplayConstructorRegistry.hpp b/tools/replay/ReplayConstructorRegistry.hpp new file mode 100644 index 000000000..da282bb67 --- /dev/null +++ b/tools/replay/ReplayConstructorRegistry.hpp @@ -0,0 +1,57 @@ +////////////////////////////////////////////////////////////////////////////// +// Copyright (c) 2016-26, Lawrence Livermore National Security, LLC and Umpire +// project contributors. See the COPYRIGHT file for details. +// +// SPDX-License-Identifier: (MIT) +////////////////////////////////////////////////////////////////////////////// +#ifndef REPLAY_ReplayConstructorRegistry_HPP +#define REPLAY_ReplayConstructorRegistry_HPP + +#if !defined(_MSC_VER) + +#include +#include +#include +#include + +#include "umpire/Allocator.hpp" +#include "umpire/ResourceManager.hpp" +#include "umpire/json/json.hpp" + +struct ReplayAllocatorSpec { + std::string allocator_id; + std::string name; + std::string strategy; + bool tracking{true}; + nlohmann::json args{}; +}; + +struct ReplayAllocationState { + std::string allocator_id; + void* runtime_ptr{nullptr}; + std::size_t size{0}; +}; + +struct ReplayContext { + umpire::ResourceManager& resource_manager; + std::unordered_map allocators{}; + std::unordered_map allocations{}; + std::vector allocator_order{}; +}; + +class ReplayConstructorRegistry { + public: + using Factory = std::function; + + ReplayConstructorRegistry(); + + umpire::Allocator construct(const ReplayAllocatorSpec& spec, ReplayContext& context) const; + + private: + void registerFactory(const std::string& strategy, Factory factory); + + std::unordered_map m_factories{}; +}; + +#endif // !defined(_MSC_VER) +#endif // REPLAY_ReplayConstructorRegistry_HPP diff --git a/tools/replay/ReplayFile.cpp b/tools/replay/ReplayFile.cpp deleted file mode 100644 index a0313cdae..000000000 --- a/tools/replay/ReplayFile.cpp +++ /dev/null @@ -1,138 +0,0 @@ -////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-26, Lawrence Livermore National Security, LLC and Umpire -// project contributors. See the COPYRIGHT file for details. -// -// SPDX-License-Identifier: (MIT) -////////////////////////////////////////////////////////////////////////////// - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#if !defined(_MSC_VER) && !defined(_LIBCPP_VERSION) -#include "ReplayFile.hpp" -#include "ReplayMacros.hpp" -#include "ReplayOptions.hpp" - -ReplayFile::ReplayFile( const ReplayOptions& options ) : - m_options(options), m_binary_filename(m_options.input_file + ".bin") -{ - const int prot = PROT_READ|PROT_WRITE; - int flags; - - m_fd = open(m_binary_filename.c_str(), O_CREAT | O_RDWR, static_cast(0660)); - - if (m_fd < 0) - REPLAY_ERROR( "Unable to create: " << m_binary_filename ); - - checkHeader(); - - if ( compileNeeded() ) { - flags = MAP_SHARED; // Writes will make it to backing store - - if (lseek(m_fd, max_file_size-1, SEEK_SET) < 0) - REPLAY_ERROR("lseek failed on " << m_binary_filename); - - if (write(m_fd, "", 1) < 0) - REPLAY_ERROR("write failed to " << m_binary_filename); - } - else { - flags = MAP_PRIVATE; - } - - m_op_tables = static_cast(mmap(nullptr, max_file_size, prot, flags, m_fd, 0)); - if (m_op_tables == MAP_FAILED) - REPLAY_ERROR( "Unable to mmap to: " << m_binary_filename ); - - m_op_tables->m.magic = REPLAY_MAGIC; - m_op_tables->m.version = REPLAY_VERSION; -} - -std::string ReplayFile::getLine(std::size_t lineno) -{ - std::ifstream file{m_options.input_file}; - - if ( ! file.is_open() ) { - REPLAY_ERROR("Unable to open input file " << m_options.input_file); - } - - file.seekg(std::ios::beg); - for (std::size_t i=0; i < lineno - 1; ++i) { - file.ignore(std::numeric_limits::max(),'\n'); - } - - std::string line; - std::getline(file, line); - std::stringstream ss; - ss << m_options.input_file << " " << lineno << " " << line; - return ss.str(); -} - -ReplayFile::~ReplayFile() -{ - if (m_op_tables != nullptr && m_op_tables != MAP_FAILED ) { - if ( compileNeeded() ) { - off_t actual_size = sizeof(Header) + (m_op_tables->num_operations * sizeof(Operation)); - - if (ftruncate(m_fd, actual_size) < 0) - REPLAY_ERROR( "Failed to truncate file size for " << m_binary_filename); - } - - munmap(m_op_tables, max_file_size); - m_op_tables = nullptr; - } - - close(m_fd); -} - -ReplayFile::Header* ReplayFile::getOperationsTable() -{ - return m_op_tables; -} - -void ReplayFile::copyString(std::string source, char (&dest)[max_name_length]) -{ - if (source.length() >= max_name_length) { - REPLAY_ERROR( "String too large: " << source ); - } - strncpy( dest, source.c_str(), source.length() ); - dest[source.length()] = '\0'; -} - -void ReplayFile::checkHeader() -{ - struct stat sbuf; - Header::Magic m; - - if (! m_options.force_compile ) { - if (read(m_fd, &m, sizeof(m)) == sizeof(m)) { - if (m.magic == REPLAY_MAGIC) { - if (m.version == REPLAY_VERSION) { - m_compile_needed = false; - - if (stat(m_binary_filename.c_str(), &sbuf)) - REPLAY_ERROR( "Unable to open " << m_binary_filename ); - - max_file_size = sbuf.st_size; - return; - } - } - } - } - - m_compile_needed = true; - - if (stat(m_options.input_file.c_str(), &sbuf)) - REPLAY_ERROR( "Unable to open " << m_options.input_file ); - - max_file_size = sizeof(ReplayFile::Header) + sbuf.st_size; - -} -#endif diff --git a/tools/replay/ReplayFile.hpp b/tools/replay/ReplayFile.hpp deleted file mode 100644 index 92fce9b4b..000000000 --- a/tools/replay/ReplayFile.hpp +++ /dev/null @@ -1,151 +0,0 @@ -////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-26, Lawrence Livermore National Security, LLC and Umpire -// project contributors. See the COPYRIGHT file for details. -// -// SPDX-License-Identifier: (MIT) -////////////////////////////////////////////////////////////////////////////// -#ifndef REPLAY_ReplayFile_HPP -#define REPLAY_ReplayFile_HPP -#if !defined(_MSC_VER) && !defined(_LIBCPP_VERSION) - -#include -#include -#include "ReplayOptions.hpp" -#include "umpire/Allocator.hpp" - -class ReplayFile { -public: - enum rtype { - INIT = 0 - , MEMORY_RESOURCE = 1 - , ALLOCATION_ADVISOR = 2 - , DYNAMIC_POOL_LIST = 3 - , MONOTONIC = 4 - , NAMED = 5 - , SLOT_POOL = 6 - , SIZE_LIMITER = 7 - , THREADSAFE_ALLOCATOR = 8 - , FIXED_POOL = 9 - , MIXED_POOL = 10 - , ALLOCATION_PREFETCHER = 11 - , NUMA_POLICY = 12 - , QUICKPOOL = 13 - }; - - static const std::size_t max_allocators{256 * 1024}; - static const std::size_t max_name_length{512}; - - struct AllocatorTableEntry { - rtype type{rtype::INIT}; - std::size_t line_number{0}; // Causal line number of input file - bool introspection{0}; - char name[max_name_length]{0}; - char base_name[max_name_length]{0}; - int argc{0}; - union { - struct { - int device_id{0}; - char advice[max_name_length]{0}; - char accessing_allocator[max_name_length]{0}; - } advisor; - struct { - int node{0}; - } numa; - struct { - std::size_t initial_alloc_size{0}; - std::size_t min_alloc_size{0}; - int alignment{0}; - } pool; - struct { - std::size_t capacity{0}; - } monotonic_pool; - struct { - std::size_t slots{0}; - } slot_pool; - struct { - std::size_t size_limit{0}; - } size_limiter; - struct { - std::size_t object_bytes{0}; - std::size_t objects_per_pool{0}; - } fixed_pool; - struct { - std::size_t smallest_fixed_blocksize{0}; - std::size_t largest_fixed_blocksize{0}; - std::size_t max_fixed_blocksize{0}; - std::size_t size_multiplier{0}; - std::size_t dynamic_initial_alloc_bytes{0}; - std::size_t dynamic_min_alloc_bytes{0}; - std::size_t dynamic_align_bytes{0}; - } mixed_pool; - } argv; - umpire::Allocator* allocator{nullptr}; - }; - - enum otype { - ALLOCATOR_CREATION = 1 - , ALLOCATE - , COALESCE - , COPY - , MOVE - , DEALLOCATE - , REALLOCATE - , REALLOCATE_EX - , RELEASE - , SETDEFAULTALLOCATOR - }; - - struct Operation { - otype op_type; - std::size_t op_line_number; // Causal line number of input file - int op_allocator; - void* op_allocated_ptr; - std::size_t op_size; // Size of allocation/operation - std::size_t op_offsets[2]; // 0-src, 1-dst - std::size_t op_alloc_ops[2]; // 0-src, 1-dst/prev - }; - - const uint64_t REPLAY_MAGIC = - static_cast( - static_cast(0x7f) << 48 - | static_cast('R') << 40 - | static_cast('E') << 32 - | static_cast('P') << 24 - | static_cast('L') << 16 - | static_cast('A') << 8 - | static_cast('Y')); - - const uint64_t REPLAY_VERSION = 17; - - struct Header { - struct Magic { - uint64_t magic; - uint64_t version; - } m; - std::size_t num_allocators; - std::size_t num_operations; - AllocatorTableEntry allocators[max_allocators]; - Operation ops[1]; - }; - - ReplayFile( const ReplayOptions& options ); - ~ReplayFile( ); - ReplayFile::Header* getOperationsTable(); - - void copyString(std::string source, char (&dest)[max_name_length]); - bool compileNeeded() { return m_compile_needed; } - std::string getLine(std::size_t lineno); - std::string getInputFileName() { return m_options.input_file; } - -private: - ReplayOptions m_options; - Header* m_op_tables{nullptr}; - const std::string m_binary_filename; - int m_fd; - bool m_compile_needed{false}; - off_t max_file_size{0}; - - void checkHeader(); -}; -#endif // !defined(_MSC_VER) && !defined(_LIBCPP_VERSION) -#endif // REPLAY_ReplayFile_HPP diff --git a/tools/replay/ReplayInterpreter.cpp b/tools/replay/ReplayInterpreter.cpp index 06ff5b15b..99136659c 100644 --- a/tools/replay/ReplayInterpreter.cpp +++ b/tools/replay/ReplayInterpreter.cpp @@ -4,828 +4,353 @@ // // SPDX-License-Identifier: (MIT) ////////////////////////////////////////////////////////////////////////////// -#include + +#if !defined(_MSC_VER) + +#include "ReplayInterpreter.hpp" + +#include #include #include #include #include +#include -#if !defined(_MSC_VER) && !defined(_LIBCPP_VERSION) -#include // for __cxa_demangle - -#include "umpire/config.hpp" +#include "umpire/util/error.hpp" -#include "ReplayFile.hpp" -#include "ReplayInterpreter.hpp" -#include "ReplayMacros.hpp" -#include "ReplayOperationManager.hpp" -#include "ReplayOptions.hpp" -#include "umpire/event/event.hpp" -#include "umpire/json/json.hpp" - -#if defined(UMPIRE_ENABLE_SQLITE_EXPERIMENTAL) -#include "umpire/event/sqlite_database.hpp" +#if !defined(_MSC_VER) +#include #else -#include "umpire/event/json_file_store.hpp" +#include +#define getpid _getpid #endif -ReplayInterpreter::ReplayInterpreter( const ReplayOptions& options ) : m_options(options) +namespace { + +nlohmann::json readJsonLine(const std::string& line, const std::string& filename, std::size_t line_number) { - m_ops = new ReplayFile{m_options}; + try { + return nlohmann::json::parse(line); + } catch (const std::exception& ex) { + UMPIRE_ERROR(umpire::runtime_error, + fmt::format("Failed to parse replay JSON at {}:{} ({})", filename, line_number, ex.what())); + } } -ReplayInterpreter::~ReplayInterpreter() +std::string requiredString(const nlohmann::json& command, const char* key) { - if (m_ops != nullptr) { - delete m_ops; - m_ops = nullptr; + auto found = command.find(key); + if (found == command.end() || !found->is_string()) { + UMPIRE_ERROR(umpire::runtime_error, fmt::format("Replay command is missing required string field {}", key)); } + return found->get(); } -void ReplayInterpreter::runOperations() +std::size_t requiredSize(const nlohmann::json& command, const char* key) { - ReplayOperationManager m_operation_mgr{m_options, m_ops, m_ops->getOperationsTable()}; - - m_operation_mgr.runOperations(); + auto found = command.find(key); + if (found == command.end() || !found->is_number_unsigned()) { + UMPIRE_ERROR(umpire::runtime_error, fmt::format("Replay command is missing required numeric field {}", key)); + } + return found->get(); } -void ReplayInterpreter::buildOperations() +std::string commandStatus(const nlohmann::json& command) { - ReplayFile::Header* hdr{nullptr}; - ReplayFile::Operation* op{nullptr}; - - if ( ! m_ops->compileNeeded() ) { - return; + auto found = command.find("status"); + if (found == command.end()) { + return "committed"; } - hdr = m_ops->getOperationsTable(); - hdr->num_allocators = 0; - op = &hdr->ops[0]; - memset(op, 0, sizeof(*op)); - op->op_type = ReplayFile::otype::ALLOCATE; - op->op_line_number = m_line_number; - hdr->num_operations = 1; - -#if defined(UMPIRE_ENABLE_SQLITE_EXPERIMENTAL) -umpire::event::sqlite_database store{m_options.input_file}; -#else -umpire::event::json_file_store store{m_options.input_file, true}; -#endif - std::vector events; - events = store.get_events(); - - for (auto e : events) { - m_line_number++; - - m_event = e; - - if ( m_event.cat != umpire::event::category::operation && m_event.name == "version" ) { - m_version_ops++; - m_log_version_major = m_event.numeric_args["major"]; - m_log_version_minor = m_event.numeric_args["minor"]; - m_log_version_patch = m_event.numeric_args["patch"]; - - if ( m_log_version_major != UMPIRE_VERSION_MAJOR - || m_log_version_minor != UMPIRE_VERSION_MINOR - || m_log_version_patch != UMPIRE_VERSION_PATCH ) { - REPLAY_WARNING("Warning, version mismatch:\n" - << " Tool version: " << UMPIRE_VERSION_MAJOR << "." - << UMPIRE_VERSION_MINOR << "." << UMPIRE_VERSION_PATCH << std::endl - << " Log version: " - << m_log_version_major << "." - << m_log_version_minor << "." - << m_log_version_patch); - - if (m_log_version_major != UMPIRE_VERSION_MAJOR) { - REPLAY_WARNING("Warning, major version mismatch - attempting replay anyway...\n" - << " Tool version: " << UMPIRE_VERSION_MAJOR << "." - << UMPIRE_VERSION_MINOR << "." << UMPIRE_VERSION_PATCH << std::endl - << " Log version: " - << m_log_version_major << "." - << m_log_version_minor << "." - << m_log_version_patch); - } - } - continue; - } - - if ( m_event.cat != umpire::event::category::operation) - continue; - - try { - if ( m_event.name == "allocate" ) { - m_allocate_ops++; - compile_allocate(); - } - else if ( m_event.name == "deallocate" ) { - m_deallocate_ops++; - if (!compile_deallocate()) { - continue; - } - } - else if ( m_event.name == "make_allocator" ) { - m_make_allocator_ops++; - compile_make_allocator(); - } - else if ( m_event.name == "make_memory_resource" ) { - m_make_memory_resource_ops++; - compile_make_memory_resource(); - } - else if ( m_event.name == "copy" ) { - m_copy_ops++; - } - else if ( m_event.name == "move" ) { - m_move_ops++; - } - else if ( m_event.name == "reallocate" ) { - m_reallocate_ops++; - compile_reallocate(); - } - else if ( m_event.name == "set_default_allocator" ) { - m_set_default_allocator_ops++; - compile_set_default_allocator(); - } - else if ( m_event.name == "coalesce" ) { - m_coalesce_ops++; - compile_coalesce(); - } - else if ( m_event.name == "release" ) { - m_release_ops++; - compile_release(); - } - else if ( m_event.name == "register_external_allocation" ) { - m_register_external_pointer++; - } - else if ( m_event.name == "deregister_external_allocation" ) { - m_deregister_external_pointer++; - } - else { - REPLAY_ERROR("Unknown Replay Operation: " << m_ops->getLine(m_line_number)); - } - } - catch (...) { - REPLAY_ERROR("Failed to compile: " << m_ops->getLine(m_line_number)); - } + if (!found->is_string()) { + UMPIRE_ERROR(umpire::runtime_error, "Replay command has a non-string status field"); } - // - // Flush operations to compile file and read back in read-only (PRIVATE) mode - // - delete m_ops; - m_ops = new ReplayFile{m_options}; - - if ( ! m_options.quiet ) { - const std::size_t leaked_allocations{m_allocate_ops - m_deallocate_ops}; - - std::cout - << "Replay File Version: " << m_log_version_major << "." << m_log_version_minor << "." << m_log_version_patch << std::endl - << std::setw(12) << m_make_memory_resource_ops << " makeMemoryResource operations" << std::endl - << std::setw(12) << m_make_allocator_ops << " makeAllocator operations" << std::endl - << std::endl - << std::setw(12) << m_allocate_ops << " allocate operations" << std::endl - << std::setw(12) << m_deallocate_ops << " deallocate performed (" << leaked_allocations << " leaked)" << std::endl; - - if (m_deallocate_rogue_ignored) { - std::cout << " " << std::setw(12) << m_deallocate_rogue_ignored << " skipped due to being rogue" << std::endl; - } - - std::cout << std::endl - << std::setw(12) << m_register_external_pointer << " external registrations (not replayed)" << std::endl - << std::setw(12) << m_deregister_external_pointer << " external deregistrations (not replayed)" << std::endl - << std::endl - << std::setw(12) << m_copy_ops << " copy operations" << std::endl - << std::setw(12) << m_move_ops << " move operations" << std::endl - << std::setw(12) << m_reallocate_ops << " reallocate operations" << std::endl - << std::setw(12) << m_set_default_allocator_ops << " setDefaultAllocator operations" << std::endl - << std::setw(12) << m_coalesce_ops << " coalesce operations" << std::endl - << std::setw(12) << m_release_ops << " release operations" << std::endl - << std::setw(12) << m_version_ops << " version operations" - << std::endl; + const auto status = found->get(); + if (status != "pending" && status != "committed") { + UMPIRE_ERROR(umpire::runtime_error, fmt::format("Replay command has unsupported status {}", status)); } + + return status; } -std::string ReplayInterpreter::printAllocatorInfo(ReplayFile::AllocatorTableEntry* allocator) +nlohmann::json normalizedCommand(nlohmann::json command) { - std::stringstream ss; - - ss << "Line#: " << allocator->line_number - << ", argc: " << allocator->argc - << ", Name: " << allocator->name - << ", Basename: " << allocator->base_name - << ", Type: "; - - switch (allocator->type) { - default: ss << "??"; break; - case ReplayFile::MEMORY_RESOURCE: ss << " MEMORY_RESOURCE"; break; - case ReplayFile::ALLOCATION_ADVISOR: ss << " ALLOCATION_ADVISOR"; break; - case ReplayFile::DYNAMIC_POOL_LIST: ss << " DYNAMIC_POOL_LIST"; break; - case ReplayFile::QUICKPOOL: ss << " QUICKPOOL"; break; - case ReplayFile::MONOTONIC: ss << " MONOTONIC"; break; - case ReplayFile::SLOT_POOL: ss << " SLOT_POOL"; break; - case ReplayFile::SIZE_LIMITER: ss << " SIZE_LIMITER"; break; - case ReplayFile::THREADSAFE_ALLOCATOR: ss << " THREADSAFE_ALLOCATOR"; break; - case ReplayFile::FIXED_POOL: ss << " FIXED_POOL"; break; - case ReplayFile::MIXED_POOL: ss << " MIXED_POOL"; break; - case ReplayFile::ALLOCATION_PREFETCHER: ss << " ALLOCATION_PREFETCHER"; break; - case ReplayFile::NUMA_POLICY: ss << " NUMA_POLICY"; break; - } - return ss.str(); + command["status"] = commandStatus(command); + return command; } -void ReplayInterpreter::printAllocators(ReplayFile* rf) +nlohmann::json stripStatus(nlohmann::json command) { - auto optable = rf->getOperationsTable(); - std::cerr << rf->getInputFileName() << std::endl; - for (std::size_t i{0}; i < optable->num_allocators; ++i) { - std::cerr << printAllocatorInfo(&(optable->allocators[i])) << std::endl; - } + command.erase("status"); + return command; } -bool ReplayInterpreter::compareOperations(ReplayInterpreter& rh) +std::string commandKey(const nlohmann::json& command) { - bool rval = true; - - if (m_ops->getOperationsTable()->m.version != rh.m_ops->getOperationsTable()->m.version) { - std::cerr << "Number of version mismatch: " - << m_ops->getOperationsTable()->m.version - << " != " << rh.m_ops->getOperationsTable()->m.version - << std::endl; - rval = false; - } + const auto op = requiredString(command, "op"); - if (m_ops->getOperationsTable()->num_allocators != rh.m_ops->getOperationsTable()->num_allocators) { - std::cerr << "Number of allocators mismatch: " - << m_ops->getOperationsTable()->num_allocators - << " != " << rh.m_ops->getOperationsTable()->num_allocators - << std::endl; - printAllocators(m_ops); - printAllocators(rh.m_ops); - rval = false; + if (op == "make_allocator") { + return op + ":" + requiredString(command, "allocator_id") + ":" + std::to_string(requiredSize(command, "seq")); } - if (m_ops->getOperationsTable()->num_operations != rh.m_ops->getOperationsTable()->num_operations) { - std::cerr << "Number of operations mismatch: " - << m_ops->getOperationsTable()->num_operations - << " != " << rh.m_ops->getOperationsTable()->num_operations - << std::endl; + if (op == "allocate" || op == "deallocate") { + return op + ":" + requiredString(command, "allocation_id"); } - if ( rval != false ) { - for ( std::size_t i = 0; i < rh.m_ops->getOperationsTable()->num_allocators; ++i) { - if ( m_ops->getOperationsTable()->allocators[i].type != rh.m_ops->getOperationsTable()->allocators[i].type ) { - std::cerr << "AllocatorTable type data miscompare at type " << i << std::endl; - rval = false; - } + UMPIRE_ERROR(umpire::runtime_error, fmt::format("Unsupported replay operation {}", op)); +} - if ( m_ops->getOperationsTable()->allocators[i].introspection != rh.m_ops->getOperationsTable()->allocators[i].introspection ) { - std::cerr << "AllocatorTable introspection data miscompare at index " << i << std::endl; - rval = false; - } +} // namespace - if ( strcmp( m_ops->getOperationsTable()->allocators[i].name, rh.m_ops->getOperationsTable()->allocators[i].name ) ) { - std::cerr << "AllocatorTable name data miscompare at index " << i << std::endl; - rval = false; - } +ReplayInterpreter::ReplayInterpreter(const ReplayOptions& options) : m_options(options) {} - if ( strcmp( m_ops->getOperationsTable()->allocators[i].base_name, rh.m_ops->getOperationsTable()->allocators[i].base_name ) ) { - std::cerr << "AllocatorTable base_name data miscompare at index " << i << std::endl; - rval = false; - } +void ReplayInterpreter::buildOperations() +{ + std::ifstream input{m_options.input_file}; + if (!input) { + UMPIRE_ERROR(umpire::runtime_error, fmt::format("Unable to open replay file {}", m_options.input_file)); + } - if ( m_ops->getOperationsTable()->allocators[i].argc != rh.m_ops->getOperationsTable()->allocators[i].argc ) { - std::cerr << "AllocatorTable argc data miscompare at index " << i << std::endl - << " LHS: " << printAllocatorInfo(&m_ops->getOperationsTable()->allocators[i]) << std::endl - << " RHS: " << printAllocatorInfo(&rh.m_ops->getOperationsTable()->allocators[i]) << std::endl - << std::endl; - rval = false; - } + m_header = nlohmann::json{}; + m_commands.clear(); + std::string line; + std::size_t line_number{0}; - if (bcmp( &m_ops->getOperationsTable()->allocators[i].argv - , &rh.m_ops->getOperationsTable()->allocators[i].argv - , sizeof(ReplayFile::AllocatorTableEntry::argv))) - { - std::cerr << "AllocatorTable Union data miscompare at index " << i << std::endl; - rval = false; - } + while (std::getline(input, line)) { + ++line_number; + if (line.empty()) { + continue; } - bool mismatch{false}; - for (std::size_t i = 0; i < rh.m_ops->getOperationsTable()->num_operations; ++i) { - if ( m_ops->getOperationsTable()->ops[i].op_type != rh.m_ops->getOperationsTable()->ops[i].op_type ) { - std::cerr << "Operation Type mismatch" << std::endl; - mismatch = true; - } - if ( m_ops->getOperationsTable()->ops[i].op_allocator != rh.m_ops->getOperationsTable()->ops[i].op_allocator ) { - std::cerr << "Operation Allocator mismatch" << std::endl; - mismatch = true; - } - if ( m_ops->getOperationsTable()->ops[i].op_size != rh.m_ops->getOperationsTable()->ops[i].op_size ) { - std::cerr << "Operation Size mismatch" << std::endl; - mismatch = true; - } - if ( m_ops->getOperationsTable()->ops[i].op_offsets[0] != rh.m_ops->getOperationsTable()->ops[i].op_offsets[0] ) { - std::cerr << "Operation Offset[0] mismatch" << std::endl; - mismatch = true; - } - if ( m_ops->getOperationsTable()->ops[i].op_offsets[1] != rh.m_ops->getOperationsTable()->ops[i].op_offsets[1] ) { - std::cerr << "Operation Offset[1] mismatch" << std::endl; - mismatch = true; - } - if ( m_ops->getOperationsTable()->ops[i].op_alloc_ops[0] != rh.m_ops->getOperationsTable()->ops[i].op_alloc_ops[0] ) { - std::cerr << "Operation Offset[0] mismatch" << std::endl; - mismatch = true; - } - if ( m_ops->getOperationsTable()->ops[i].op_alloc_ops[1] != rh.m_ops->getOperationsTable()->ops[i].op_alloc_ops[1] ) { - std::cerr << "Operation Offset[1] mismatch" << std::endl; - mismatch = true; - } - if ( mismatch ) { - std::cerr << " LHS: " - << m_ops->getLine(m_ops->getOperationsTable()->ops[i].op_line_number) - << std::endl; - std::cerr << " RHS: " - << rh.m_ops->getLine(rh.m_ops->getOperationsTable()->ops[i].op_line_number) - << std::endl; - rval = false; - break; - } + auto json_line = readJsonLine(line, m_options.input_file, line_number); + if (line_number == 1) { + m_header = json_line; + } else { + m_commands.emplace_back(std::move(json_line)); } } - return rval; -} + if (m_header.empty()) { + UMPIRE_ERROR(umpire::runtime_error, fmt::format("Replay file {} is empty", m_options.input_file)); + } -template void get_from_string( const std::string& s, T& val ) -{ - std::istringstream ss(s); - ss >> val; + if (m_header.value("kind", "") != "umpire_replay" || m_header.value("schema", "") != "v2") { + UMPIRE_ERROR(umpire::runtime_error, + fmt::format("Replay file {} is not a supported umpire replay v2 trace", m_options.input_file)); + } } -void ReplayInterpreter::strip_off_base(std::string& s) +ReplayAllocatorSpec ReplayInterpreter::parseAllocatorSpec(const nlohmann::json& command) const { - const std::string base("_base"); - - if (s.length() > base.length()) { - if (s.compare(s.length() - base.length(), base.length(), base) == 0) { - s.erase(s.length() - base.length(), base.length()); - } - } + ReplayAllocatorSpec spec; + spec.allocator_id = requiredString(command, "allocator_id"); + spec.name = requiredString(command, "name"); + spec.strategy = requiredString(command, "strategy"); + spec.tracking = command.value("tracking", true); + spec.args = command.value("args", nlohmann::json::object()); + return spec; } -void ReplayInterpreter::compile_make_memory_resource() +nlohmann::json ReplayInterpreter::normalizeHeader(nlohmann::json header) const { - const std::string allocator_name{ m_event.tags["allocator_name"] }; - const uint64_t obj_p { getPointer( m_event.string_args["allocator_ref"] ) }; - ReplayFile::Header* hdr = m_ops->getOperationsTable(); - - m_allocator_indices[obj_p] = hdr->num_allocators; - - ReplayFile::AllocatorTableEntry* alloc = &(hdr->allocators[hdr->num_allocators]); - - alloc->type = ReplayFile::rtype::MEMORY_RESOURCE; - alloc->line_number = m_line_number; - alloc->introspection = true; - alloc->argc = 0; - m_ops->copyString(allocator_name, alloc->name); - - ReplayFile::Operation* op = &hdr->ops[hdr->num_operations]; - memset(op, 0, sizeof(*op)); - - op->op_type = ReplayFile::otype::ALLOCATOR_CREATION; - op->op_line_number = m_line_number; - op->op_allocator = hdr->num_allocators; - m_allocator_index[allocator_name] = hdr->num_allocators; - hdr->num_allocators++; - if (hdr->num_allocators >= ReplayFile::max_allocators) { - REPLAY_ERROR("Too many allocators for replay: " << hdr->num_allocators); + if (header.contains("process") && header["process"].is_object()) { + header["process"].erase("pid"); } - - hdr->num_operations++; + return header; } -void ReplayInterpreter::compile_make_allocator() +std::vector ReplayInterpreter::normalizeCommands() const { - const std::string allocator_name{m_event.tags["allocator_name"]}; - const bool introspection{m_event.numeric_args["introspection"] == 1}; - const std::string raw_mangled_type{m_event.string_args["type"]}; - - ReplayFile::Header* hdr{m_ops->getOperationsTable()}; - ReplayFile::AllocatorTableEntry* alloc{ &(m_ops->getOperationsTable()->allocators[hdr->num_allocators]) }; - - alloc->line_number = m_line_number; - - m_ops->copyString(allocator_name, alloc->name); - alloc->introspection = introspection; - - std::string type; - if (!m_options.do_not_demangle && m_log_version_major >= 2) { - const std::string type_prefix{raw_mangled_type.substr(0, 2)}; - - // Add _Z so that we can demangle the external symbol - const std::string mangled_type{ (type_prefix == "_Z") ? raw_mangled_type : std::string{"_Z"} + raw_mangled_type }; - - auto result = abi::__cxa_demangle( mangled_type.c_str(), nullptr, nullptr, nullptr); - if (!result) { - REPLAY_ERROR("Failed to demangle strategy type. Mangled type: " << mangled_type); - } - type = std::string{result}; - ::free(result); - } else { - type = raw_mangled_type; - } - - const std::string base_allocator_name{m_event.string_args["arg0"]}; - alloc->argc = 1; - - if ( type == "umpire::strategy::AllocationAdvisor" ) { - const std::string advice_operation{m_event.string_args["arg1"]}; - alloc->argc++; - - int device_id{-1}; // Use default argument if negative - - if ( m_event.string_args.find("arg2") != m_event.string_args.end() ) { // Accessing Allocator provided - alloc->argc++; - - if ( m_event.numeric_args.find("arg3") != m_event.numeric_args.end() ) { // ID provided as arg3 - alloc->argc++; - device_id = m_event.numeric_args["arg3"]; - } - } - else { // No accessing Allocator provided - if ( m_event.numeric_args.find("arg2") != m_event.numeric_args.end() ) { // ID provided as arg2 - alloc->argc++; - device_id = m_event.numeric_args["arg2"]; + std::vector normalized; + normalized.reserve(m_commands.size()); + + std::unordered_map pending_commands; + + for (const auto& raw_command : m_commands) { + const bool has_explicit_status = raw_command.contains("status"); + auto command = normalizedCommand(raw_command); + const auto status = requiredString(command, "status"); + const auto key = commandKey(command); + + if (status == "pending") { + auto inserted = pending_commands.emplace(key, command); + if (!inserted.second) { + UMPIRE_ERROR(umpire::runtime_error, + fmt::format("Replay command {} has duplicate pending lifecycle state", key)); + } + } else { + auto pending = pending_commands.find(key); + if (pending != pending_commands.end()) { + if (stripStatus(pending->second) != stripStatus(command)) { + UMPIRE_ERROR(umpire::runtime_error, + fmt::format("Replay command {} has mismatched pending and committed payloads", key)); + } + pending_commands.erase(pending); + } else if (has_explicit_status) { + UMPIRE_ERROR(umpire::runtime_error, + fmt::format("Replay command {} has committed lifecycle state without a pending record", key)); } } - alloc->type = ReplayFile::rtype::ALLOCATION_ADVISOR; - m_ops->copyString(base_allocator_name, alloc->base_name); - m_ops->copyString(advice_operation, alloc->argv.advisor.advice); - alloc->argv.advisor.device_id = device_id; - - if (device_id >= 0) { // Optional device ID specified - switch ( alloc->argc ) { - default: - REPLAY_ERROR("Invalid number of arguments (" << alloc->argc << " for " << type << " operation. Stopping"); - case 3: - break; - case 4: - const std::string accessing_allocator_name{m_event.string_args["arg2"]}; - m_ops->copyString(accessing_allocator_name, alloc->argv.advisor.accessing_allocator); - break; - } - } - else { // Use default device_id - switch ( alloc->argc ) { - default: - REPLAY_ERROR("Invalid number of arguments (" << alloc->argc << " for " << type << " operation. Stopping"); - case 2: - break; - case 3: - const std::string accessing_allocator_name{m_event.string_args["arg2"]}; - m_ops->copyString(accessing_allocator_name, alloc->argv.advisor.accessing_allocator); - break; - } - } + normalized.emplace_back(std::move(command)); } - else if ( type == "umpire::strategy::AllocationPrefetcher" ) { - alloc->type = ReplayFile::rtype::ALLOCATION_PREFETCHER; - m_ops->copyString(base_allocator_name, alloc->base_name); - } - else if ( type == "umpire::strategy::NumaPolicy" ) { - alloc->type = ReplayFile::rtype::NUMA_POLICY; - alloc->argv.numa.node = m_event.numeric_args["arg1"]; - alloc->argc++; - m_ops->copyString(base_allocator_name, alloc->base_name); - } - else if (type == "umpire::strategy::NamedAllocationStrategy") { - alloc->type = ReplayFile::rtype::NAMED; - m_ops->copyString(base_allocator_name, alloc->base_name); - } - else if ( type == "umpire::strategy::QuickPool" ) { - alloc->type = ReplayFile::rtype::QUICKPOOL; - m_ops->copyString(base_allocator_name, alloc->base_name); - - // Now grab the optional fields - if (m_event.numeric_args.find("arg3") != m_event.numeric_args.end()) { - alloc->argc = 4; // ignore potential heuristic parameters - alloc->argv.pool.initial_alloc_size = m_event.numeric_args["arg1"]; - alloc->argv.pool.min_alloc_size = m_event.numeric_args["arg2"]; - alloc->argv.pool.alignment = m_event.numeric_args["arg3"]; - } - else if (m_event.numeric_args.find("arg2") != m_event.numeric_args.end()) { - alloc->argc = 3; - alloc->argv.pool.initial_alloc_size = m_event.numeric_args["arg1"]; - alloc->argv.pool.min_alloc_size = m_event.numeric_args["arg2"]; - } - else if (m_event.numeric_args.find("arg1") != m_event.numeric_args.end()) { - alloc->argc = 2; - alloc->argv.pool.initial_alloc_size = m_event.numeric_args["arg1"]; - } - } - else if ( type == "umpire::strategy::DynamicPoolList" ) { - alloc->type = ReplayFile::rtype::DYNAMIC_POOL_LIST; - m_ops->copyString(base_allocator_name, alloc->base_name); - - // Now grab the optional fields - if (m_event.numeric_args.find("arg3") != m_event.numeric_args.end()) { - alloc->argc = 4; // ignore potential heuristic parameters - alloc->argv.pool.initial_alloc_size = m_event.numeric_args["arg1"]; - alloc->argv.pool.min_alloc_size = m_event.numeric_args["arg2"]; - alloc->argv.pool.alignment = m_event.numeric_args["arg3"]; - } - else if (m_event.numeric_args.find("arg2") != m_event.numeric_args.end()) { - alloc->argc = 3; - alloc->argv.pool.initial_alloc_size = m_event.numeric_args["arg1"]; - alloc->argv.pool.min_alloc_size = m_event.numeric_args["arg2"]; - } - else if (m_event.numeric_args.find("arg1") != m_event.numeric_args.end()) { - alloc->argc = 2; - alloc->argv.pool.initial_alloc_size = m_event.numeric_args["arg1"]; - } - } - else if ( type == "umpire::strategy::MonotonicAllocationStrategy" ) { - alloc->type = ReplayFile::rtype::MONOTONIC; - m_ops->copyString(base_allocator_name, alloc->base_name); - alloc->argv.monotonic_pool.capacity = m_event.numeric_args["arg1"]; - alloc->argc++; - } - else if ( type == "umpire::strategy::SlotPool" ) { - alloc->type = ReplayFile::rtype::SLOT_POOL; - m_ops->copyString(base_allocator_name, alloc->base_name); - alloc->argv.slot_pool.slots = m_event.numeric_args["arg1"]; - alloc->argc++; - } - else if ( type == "umpire::strategy::SizeLimiter" ) { - alloc->type = ReplayFile::rtype::SIZE_LIMITER; - m_ops->copyString(base_allocator_name, alloc->base_name); - alloc->argv.size_limiter.size_limit = m_event.numeric_args["arg1"]; - alloc->argc++; - } - else if ( type == "umpire::strategy::ThreadSafeAllocator" ) { - alloc->type = ReplayFile::rtype::THREADSAFE_ALLOCATOR; - m_ops->copyString(base_allocator_name, alloc->base_name); - } - else if ( type == "umpire::strategy::FixedPool" ) { - alloc->type = ReplayFile::rtype::FIXED_POOL; - m_ops->copyString(base_allocator_name, alloc->base_name); - alloc->argv.fixed_pool.object_bytes = m_event.numeric_args["arg1"]; - alloc->argc++; - - // Now grab the optional fields - if (m_event.numeric_args.find("arg2") != m_event.numeric_args.end()) { - alloc->argv.fixed_pool.objects_per_pool = m_event.numeric_args["arg2"]; - alloc->argc++; - } - } - else if ( type == "umpire::strategy::MixedPool" ) { - alloc->type = ReplayFile::rtype::MIXED_POOL; - m_ops->copyString(base_allocator_name, alloc->base_name); - - // Now grab the optional fields - if (m_event.numeric_args.find("arg7") != m_event.numeric_args.end()) { - alloc->argc = 8; - alloc->argv.mixed_pool.smallest_fixed_blocksize = m_event.numeric_args["arg1"]; - alloc->argv.mixed_pool.largest_fixed_blocksize = m_event.numeric_args["arg2"]; - alloc->argv.mixed_pool.max_fixed_blocksize = m_event.numeric_args["arg3"]; - alloc->argv.mixed_pool.size_multiplier = m_event.numeric_args["arg4"]; - alloc->argv.mixed_pool.dynamic_initial_alloc_bytes = m_event.numeric_args["arg5"]; - alloc->argv.mixed_pool.dynamic_min_alloc_bytes = m_event.numeric_args["arg6"]; - alloc->argv.mixed_pool.dynamic_align_bytes = m_event.numeric_args["arg7"]; - } - else if (m_event.numeric_args.find("arg6") != m_event.numeric_args.end()) { - alloc->argc = 7; - alloc->argv.mixed_pool.smallest_fixed_blocksize = m_event.numeric_args["arg1"]; - alloc->argv.mixed_pool.largest_fixed_blocksize = m_event.numeric_args["arg2"]; - alloc->argv.mixed_pool.max_fixed_blocksize = m_event.numeric_args["arg3"]; - alloc->argv.mixed_pool.size_multiplier = m_event.numeric_args["arg4"]; - alloc->argv.mixed_pool.dynamic_initial_alloc_bytes = m_event.numeric_args["arg5"]; - alloc->argv.mixed_pool.dynamic_min_alloc_bytes = m_event.numeric_args["arg6"]; - } - else if (m_event.numeric_args.find("arg5") != m_event.numeric_args.end()) { - alloc->argc = 6; - alloc->argv.mixed_pool.smallest_fixed_blocksize = m_event.numeric_args["arg1"]; - alloc->argv.mixed_pool.largest_fixed_blocksize = m_event.numeric_args["arg2"]; - alloc->argv.mixed_pool.max_fixed_blocksize = m_event.numeric_args["arg3"]; - alloc->argv.mixed_pool.size_multiplier = m_event.numeric_args["arg4"]; - alloc->argv.mixed_pool.dynamic_initial_alloc_bytes = m_event.numeric_args["arg5"]; - } - else if (m_event.numeric_args.find("arg4") != m_event.numeric_args.end()) { - alloc->argc = 5; - alloc->argv.mixed_pool.smallest_fixed_blocksize = m_event.numeric_args["arg1"]; - alloc->argv.mixed_pool.largest_fixed_blocksize = m_event.numeric_args["arg2"]; - alloc->argv.mixed_pool.max_fixed_blocksize = m_event.numeric_args["arg3"]; - alloc->argv.mixed_pool.size_multiplier = m_event.numeric_args["arg4"]; - } - else if (m_event.numeric_args.find("arg3") != m_event.numeric_args.end()) { - alloc->argc = 4; - alloc->argv.mixed_pool.smallest_fixed_blocksize = m_event.numeric_args["arg1"]; - alloc->argv.mixed_pool.largest_fixed_blocksize = m_event.numeric_args["arg2"]; - alloc->argv.mixed_pool.max_fixed_blocksize = m_event.numeric_args["arg3"]; - } - else if (m_event.numeric_args.find("arg2") != m_event.numeric_args.end()) { - alloc->argc = 3; - alloc->argv.mixed_pool.smallest_fixed_blocksize = m_event.numeric_args["arg1"]; - alloc->argv.mixed_pool.largest_fixed_blocksize = m_event.numeric_args["arg2"]; - } - else if (m_event.numeric_args.find("arg1") != m_event.numeric_args.end()) { - alloc->argc = 2; - alloc->argv.mixed_pool.smallest_fixed_blocksize = m_event.numeric_args["arg1"]; - } + return normalized; +} + +bool ReplayInterpreter::compareOperations(ReplayInterpreter& rh) +{ + if (m_header.empty()) { + buildOperations(); } - else { - REPLAY_ERROR("Unknown class (" << type << "), skipping."); + if (rh.m_header.empty()) { + rh.buildOperations(); } - const std::string allocator_ref_string{m_event.string_args["allocator_ref"]}; - const uint64_t obj_p{ getPointer(allocator_ref_string) }; - - m_allocator_indices[obj_p] = hdr->num_allocators; - - ReplayFile::Operation* op = &hdr->ops[hdr->num_operations]; - memset(op, 0, sizeof(*op)); - - op->op_type = ReplayFile::otype::ALLOCATOR_CREATION; - op->op_line_number = m_line_number; - op->op_allocator = hdr->num_allocators; - - m_allocator_index[allocator_name] = hdr->num_allocators; - - hdr->num_allocators++; - if (hdr->num_allocators >= ReplayFile::max_allocators) { - REPLAY_ERROR("Too many allocators for replay: " << hdr->num_allocators); + if (normalizeHeader(m_header) != normalizeHeader(rh.m_header)) { + return false; } - hdr->num_operations++; -} -void ReplayInterpreter::compile_set_default_allocator() -{ - const std::string allocator_ref_string{m_event.string_args["allocator_ref"]}; - ReplayFile::Header* hdr = m_ops->getOperationsTable(); - ReplayFile::Operation* op = &hdr->ops[hdr->num_operations]; - - memset(op, 0, sizeof(*op)); - op->op_type = ReplayFile::otype::SETDEFAULTALLOCATOR; - op->op_line_number = m_line_number; - op->op_allocator = getAllocatorIndex(allocator_ref_string); - hdr->num_operations++; + return normalizeCommands() == rh.normalizeCommands(); } -int ReplayInterpreter::getAllocatorIndex(const std::string& ref_s) +void ReplayInterpreter::appendStats(ReplayContext& context, std::size_t seq) { - const uint64_t ref_p{std::stoul(ref_s, nullptr, 0)}; - auto n_iter(m_allocator_indices.find(ref_p)); + for (const auto& allocator_id : context.allocator_order) { + auto found = context.allocators.find(allocator_id); + if (found == context.allocators.end()) { + continue; + } - if ( n_iter == m_allocator_indices.end() ) - REPLAY_ERROR("Unable to find allocator: " << ref_s); + auto allocator = found->second; + const auto& allocator_name = allocator.getName(); - return n_iter->second; + m_stat_series[allocator_name + " current_size"].push_back({seq, allocator.getCurrentSize()}); + m_stat_series[allocator_name + " actual_size"].push_back({seq, allocator.getActualSize()}); + m_stat_series[allocator_name + " high_watermark"].push_back({seq, allocator.getHighWatermark()}); + m_stat_series[allocator_name + " allocation_count"].push_back({seq, allocator.getAllocationCount()}); + } } -uint64_t ReplayInterpreter::getPointer(std::string ptr_name) +void ReplayInterpreter::dumpStats() const { - const uint64_t ptr{std::stoul(ptr_name, nullptr, 0)}; + std::ofstream file{"replay" + std::to_string(getpid()) + ".ult"}; + if (!file) { + UMPIRE_ERROR(umpire::runtime_error, "Unable to create replay ULT output file"); + } - return ptr; + for (const auto& stat_series : m_stat_series) { + file << "# " << stat_series.first << "\n"; + for (const auto& entry : stat_series.second) { + file << entry.first << " " << entry.second << "\n"; + } + } } -void ReplayInterpreter::compile_allocate() +void ReplayInterpreter::printStats(ReplayContext& context) const { - if (m_replaying_reallocate) - return; + const int name_width{40}; + const int num_width{16}; - ReplayFile::Header* hdr{m_ops->getOperationsTable()}; - ReplayFile::Operation* op{&hdr->ops[hdr->num_operations]}; - const std::string allocator_ref{m_event.string_args["allocator_ref"]}; - const std::size_t allocation_size{m_event.numeric_args["size"]}; - std::string pointer_string{m_event.string_args["pointer"]}; - const std::string pointer_key{allocator_ref + pointer_string}; + std::cout << std::setw(name_width) << std::left << "Filename" << std::setw(name_width) << std::left << "Allocator" + << std::setw(num_width) << std::left << "Current Size" << std::setw(num_width) << std::left << "Actual Size" + << std::setw(num_width) << std::left << "High Watermark" << "\n"; - memset(op, 0, sizeof(*op)); + for (const auto& allocator_id : context.allocator_order) { + auto found = context.allocators.find(allocator_id); + if (found == context.allocators.end()) { + continue; + } - op->op_type = ReplayFile::otype::ALLOCATE; - op->op_line_number = m_line_number; - op->op_allocator = getAllocatorIndex(allocator_ref); - op->op_size = allocation_size; + auto allocator = found->second; + if (allocator.getHighWatermark() == 0) { + continue; + } - if ( m_allocation_id.find(pointer_key) != m_allocation_id.end() ) { - REPLAY_ERROR("Pointer already allocated: " << m_ops->getLine(m_line_number) << std::endl); + std::cout << std::setw(name_width) << std::left << m_options.input_file << std::setw(name_width) << std::left + << allocator.getName() << std::setw(num_width) << std::left << allocator.getCurrentSize() + << std::setw(num_width) << std::left << allocator.getActualSize() << std::setw(num_width) << std::left + << allocator.getHighWatermark() << "\n"; } - - op->op_line_number = m_line_number; - m_allocation_id.insert({pointer_key, hdr->num_operations}); - hdr->num_operations++; } -void ReplayInterpreter::compile_reallocate() +void ReplayInterpreter::runOperations() { - const std::string allocator_ref{m_event.string_args["allocator_ref"]}; - ReplayFile::Header* hdr{m_ops->getOperationsTable()}; - ReplayFile::Operation* op{&hdr->ops[hdr->num_operations]}; - - if (m_event.string_args.find("new_ptr") == m_event.string_args.end() ) { - // - // First of two reallocate replays - // - const std::size_t allocation_size{m_event.numeric_args["size"]}; - const std::string current_ptr_string{m_event.string_args["current_ptr"]}; - const std::string current_ptr_key{allocator_ref + current_ptr_string}; - const uint64_t current_ptr{ getPointer(current_ptr_string) }; - - m_replaying_reallocate = true; - - if ( current_ptr != 0 && (m_allocation_id.find(current_ptr_key) == m_allocation_id.end()) ) { - REPLAY_ERROR("Rogue: " << m_ops->getLine(m_line_number) << std::endl); - } - - memset(op, 0, sizeof(*op)); - op->op_type = ReplayFile::otype::REALLOCATE_EX; - op->op_line_number = m_line_number; - op->op_alloc_ops[1] = (current_ptr == 0) ? 0 : m_allocation_id[current_ptr_key]; - op->op_size = allocation_size; - op->op_allocator = getAllocatorIndex(allocator_ref); - - if (current_ptr != 0) - m_allocation_id.erase(current_ptr_key); + if (m_header.empty()) { + buildOperations(); } - else { - const std::string new_ptr_string{m_event.string_args["new_ptr"]}; - const std::string new_ptr_key{allocator_ref + new_ptr_string}; - if ( m_allocation_id.find(new_ptr_key) != m_allocation_id.end() ) { - REPLAY_ERROR("Pointer already allocated: " << m_ops->getLine(m_line_number) << std::endl); - } - m_allocation_id.insert({new_ptr_key, hdr->num_operations}); - hdr->num_operations++; - m_replaying_reallocate = false; - } -} + ReplayContext context{umpire::ResourceManager::getInstance()}; + m_stat_series.clear(); + const auto normalized_commands = normalizeCommands(); -bool ReplayInterpreter::compile_deallocate() -{ - const std::string allocator_ref{m_event.string_args["allocator_ref"]}; - const std::string memory_ptr_string{m_event.string_args["pointer"]}; - const std::string memory_ptr_key{allocator_ref + memory_ptr_string}; - - if (m_replaying_reallocate) - return false; + for (const auto& command : normalized_commands) { + const auto op = requiredString(command, "op"); + const auto status = requiredString(command, "status"); + const auto seq = requiredSize(command, "seq"); - ReplayFile::Header* hdr = m_ops->getOperationsTable(); + if (status == "pending") { + continue; + } - if ( m_allocation_id.find(memory_ptr_key) == m_allocation_id.end() ) { - int id{getAllocatorIndex(allocator_ref)}; + if (op == "make_allocator") { + auto spec = parseAllocatorSpec(command); + auto allocator = m_registry.construct(spec, context); + context.allocators[spec.allocator_id] = allocator; + context.allocator_order.push_back(spec.allocator_id); + } else if (op == "allocate") { + const auto allocator_id = requiredString(command, "allocator_id"); + const auto allocation_id = requiredString(command, "allocation_id"); + const auto size = requiredSize(command, "size"); - std::cerr - << "[IGNORED] Rogue deallocate ptr= " << memory_ptr_string - << ", base_name=" << hdr->allocators[id].base_name - << ", name= " << hdr->allocators[id].name << " " - << m_ops->getLine(m_line_number) - << std::endl; + auto allocator_it = context.allocators.find(allocator_id); + if (allocator_it == context.allocators.end()) { + UMPIRE_ERROR(umpire::runtime_error, + fmt::format("Replay allocate references unknown allocator {}", allocator_id)); + } - m_deallocate_rogue_ignored++; - return false; - } + void* ptr = allocator_it->second.allocate(size); + context.allocations[allocation_id] = ReplayAllocationState{allocator_id, ptr, size}; + } else if (op == "deallocate") { + const auto allocator_id = requiredString(command, "allocator_id"); + const auto allocation_id = requiredString(command, "allocation_id"); - ReplayFile::Operation* op = &hdr->ops[hdr->num_operations]; + auto allocation_it = context.allocations.find(allocation_id); + if (allocation_it == context.allocations.end()) { + UMPIRE_ERROR(umpire::runtime_error, + fmt::format("Replay deallocate references unknown allocation {}", allocation_id)); + } - memset(op, 0, sizeof(*op)); + if (allocation_it->second.allocator_id != allocator_id) { + UMPIRE_ERROR(umpire::runtime_error, + fmt::format("Replay deallocate allocator mismatch for allocation {}", allocation_id)); + } - op->op_type = ReplayFile::otype::DEALLOCATE; - op->op_line_number = m_line_number; - op->op_allocator = getAllocatorIndex(allocator_ref); - op->op_alloc_ops[0] = m_allocation_id[memory_ptr_key]; - hdr->num_operations++; + auto allocator_it = context.allocators.find(allocator_id); + if (allocator_it == context.allocators.end()) { + UMPIRE_ERROR(umpire::runtime_error, + fmt::format("Replay deallocate references unknown allocator {}", allocator_id)); + } - m_allocation_id.erase(memory_ptr_key); - return true; -} + allocator_it->second.deallocate(allocation_it->second.runtime_ptr); + context.allocations.erase(allocation_it); + } else { + UMPIRE_ERROR(umpire::runtime_error, fmt::format("Unsupported replay operation {}", op)); + } -void ReplayInterpreter::compile_coalesce() -{ - std::string allocator_name{m_event.tags["allocator_name"]}; - strip_off_base(allocator_name); + if (m_options.dump_statistics || m_options.track_stats) { + appendStats(context, seq); + } + } - ReplayFile::Header* hdr = m_ops->getOperationsTable(); - ReplayFile::Operation* op = &hdr->ops[hdr->num_operations]; + if (m_options.dump_statistics) { + dumpStats(); + } - memset(op, 0, sizeof(*op)); - op->op_type = ReplayFile::otype::COALESCE; - op->op_line_number = m_line_number; - op->op_allocator = m_allocator_index[allocator_name]; - hdr->num_operations++; + if (m_options.track_stats && !m_options.quiet) { + printStats(context); + } } -void ReplayInterpreter::compile_release() -{ - const std::string allocator_ref_string{m_event.string_args["allocator_ref"]}; - ReplayFile::Header* hdr = m_ops->getOperationsTable(); - ReplayFile::Operation* op = &hdr->ops[hdr->num_operations]; - - memset(op, 0, sizeof(*op)); - op->op_type = ReplayFile::otype::RELEASE; - op->op_line_number = m_line_number; - op->op_allocator = getAllocatorIndex(allocator_ref_string); - hdr->num_operations++; -} -#endif // !defined(_MSC_VER) && !defined(_LIBCPP_VERSION) +#endif // !defined(_MSC_VER) diff --git a/tools/replay/ReplayInterpreter.hpp b/tools/replay/ReplayInterpreter.hpp index 59a55338b..de6d6684e 100644 --- a/tools/replay/ReplayInterpreter.hpp +++ b/tools/replay/ReplayInterpreter.hpp @@ -7,84 +7,41 @@ #ifndef REPLAY_ReplayInterpreter_HPP #define REPLAY_ReplayInterpreter_HPP -#if !defined(_MSC_VER) && !defined(_LIBCPP_VERSION) -#include +#if !defined(_MSC_VER) + +#include #include -#include -#include +#include +#include -#include "ReplayOperationManager.hpp" +#include "ReplayConstructorRegistry.hpp" #include "ReplayOptions.hpp" -#include "umpire/event/event.hpp" -#include "umpire/event/json_file_store.hpp" #include "umpire/json/json.hpp" class ReplayInterpreter { - public: - void buildOperations(); - void runOperations(); - bool compareOperations(ReplayInterpreter& rh); - - ReplayInterpreter( const ReplayOptions& options ); - ~ReplayInterpreter(); - - ReplayFile* m_ops{nullptr}; - - private: - using AllocatorIndex = int; - using AllocatorFromLog = uint64_t; - using AllocationFromLog = uint64_t; - using AllocatorIndexMap = std::unordered_map; - using AllocationAllocatorMap = std::unordered_map; - - ReplayOptions m_options; - std::unordered_map m_allocator_index; - std::string m_line; - umpire::event::event m_event; - std::vector m_row; - AllocatorIndexMap m_allocator_indices; - AllocationAllocatorMap m_allocation_id; - bool m_replaying_reallocate{false}; - std::size_t m_line_number{0}; - - int m_log_version_major; - int m_log_version_minor; - int m_log_version_patch; - std::size_t m_register_external_pointer{0}; - std::size_t m_deregister_external_pointer{0}; - - std::size_t m_make_allocator_ops{0}; - std::size_t m_make_memory_resource_ops{0}; - std::size_t m_copy_ops{0}; - std::size_t m_move_ops{0}; - std::size_t m_reallocate_ops{0}; - std::size_t m_set_default_allocator_ops{0}; - std::size_t m_allocate_ops{0}; - std::size_t m_deallocate_ops{0}; - std::size_t m_deallocate_rogue_ignored{0}; - - std::size_t m_coalesce_ops{0}; - std::size_t m_release_ops{0}; - std::size_t m_version_ops{0}; - - template void get_from_string( const std::string& s, T& val ); - - void strip_off_base(std::string& s); - void compile_make_memory_resource(); - void compile_set_default_allocator(); - void compile_make_allocator(); - void compile_reallocate(); - void compile_allocate(); - bool compile_deallocate(); - void compile_coalesce(); - void compile_release(); - int getAllocatorIndex(const std::string& ref_s); - uint64_t getPointer(std::string ptr_name); - void printAllocators(ReplayFile* optable); - std::string printAllocatorInfo(ReplayFile::AllocatorTableEntry* allocator); + public: + explicit ReplayInterpreter(const ReplayOptions& options); + + void buildOperations(); + void runOperations(); + bool compareOperations(ReplayInterpreter& rh); + + private: + using StatSeries = std::map>>; + + ReplayAllocatorSpec parseAllocatorSpec(const nlohmann::json& command) const; + nlohmann::json normalizeHeader(nlohmann::json header) const; + std::vector normalizeCommands() const; + void appendStats(ReplayContext& context, std::size_t seq); + void dumpStats() const; + void printStats(ReplayContext& context) const; + + ReplayOptions m_options; + nlohmann::json m_header{}; + std::vector m_commands{}; + ReplayConstructorRegistry m_registry{}; + StatSeries m_stat_series{}; }; -#include "ReplayInterpreter.inl" - -#endif //!defined(_MSC_VER) && !defined(_LIBCPP_VERSION) +#endif // !defined(_MSC_VER) #endif // REPLAY_ReplayInterpreter_HPP diff --git a/tools/replay/ReplayInterpreter.inl b/tools/replay/ReplayInterpreter.inl deleted file mode 100644 index 6f938c1b7..000000000 --- a/tools/replay/ReplayInterpreter.inl +++ /dev/null @@ -1,21 +0,0 @@ -////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-26, Lawrence Livermore National Security, LLC and Umpire -// project contributors. See the COPYRIGHT file for details. -// -// SPDX-License-Identifier: (MIT) -////////////////////////////////////////////////////////////////////////////// -#ifndef REPLAY_ReplayInterpreter_INL -#define REPLAY_ReplayInterpreter_INL - -#if !defined(_MSC_VER) && !defined(_LIBCPP_VERSION) -#include - -template void -ReplayInterpreter::get_from_string( const std::string& s, T& val ) -{ - std::istringstream ss(s); - ss >> val; -} -#endif // !defined(_MSC_VER) && !defined(_LIBCPP_VERSION) - -#endif // REPLAY_ReplayInterpreter_INL diff --git a/tools/replay/ReplayMacros.hpp b/tools/replay/ReplayMacros.hpp deleted file mode 100644 index bea61ad8c..000000000 --- a/tools/replay/ReplayMacros.hpp +++ /dev/null @@ -1,40 +0,0 @@ -////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-26, Lawrence Livermore National Security, LLC and Umpire -// project contributors. See the COPYRIGHT file for details. -// -// SPDX-License-Identifier: (MIT) -////////////////////////////////////////////////////////////////////////////// -#ifndef REPLAY_Macros_HPP -#define REPLAY_Macros_HPP -#if !defined(_MSC_VER) && !defined(_LIBCPP_VERSION) - -#include -#include - -// #define REPLAY_TRACE_ON - -#if defined(REPLAY_TRACE_ON) -#define REPLAY_TRACE( msg ) \ -{ \ - std::cout << std::string(__FILE__) << " " << __LINE__ << " " << __func__ \ - << " " << msg << std::endl; \ -} -#else -#define REPLAY_TRACE( msg ) -#endif // REPLAY_TRACE_ON - -#define REPLAY_WARNING( msg ) \ -{ \ - std::cout << std::string(__FILE__) << " " << __LINE__ << " " << __func__ \ - << " " << msg << std::endl; \ -} - -#define REPLAY_ERROR( msg ) \ -{ \ - std::cerr << std::string(__FILE__) << " " << __LINE__ << " " << __func__ \ - << " " << msg << std::endl; \ - exit(-1); \ -} - -#endif // !defined(_MSC_VER) && !defined(_LIBCPP_VERSION) -#endif // REPLAY_Macros_HPP diff --git a/tools/replay/ReplayOperationManager.cpp b/tools/replay/ReplayOperationManager.cpp deleted file mode 100644 index 18fd646e3..000000000 --- a/tools/replay/ReplayOperationManager.cpp +++ /dev/null @@ -1,1176 +0,0 @@ -////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-26, Lawrence Livermore National Security, LLC and Umpire -// project contributors. See the COPYRIGHT file for details. -// -// SPDX-License-Identifier: (MIT) -////////////////////////////////////////////////////////////////////////////// -#include -#include -#include -#include -#include - -#if !defined(_MSC_VER) && !defined(_LIBCPP_VERSION) -#include "umpire/Allocator.hpp" -#include "umpire/strategy/AllocationAdvisor.hpp" -#include "umpire/strategy/AllocationPrefetcher.hpp" -#include "umpire/strategy/PoolCoalesceHeuristic.hpp" -#include "umpire/strategy/SizeLimiter.hpp" -#include "umpire/strategy/NamedAllocationStrategy.hpp" -#include "umpire/strategy/QuickPool.hpp" -#include "umpire/util/AllocationRecord.hpp" -#include "umpire/util/wrap_allocator.hpp" -#include "umpire/ResourceManager.hpp" -#include "ReplayMacros.hpp" -#include "ReplayOperationManager.hpp" -#include "ReplayOptions.hpp" - -#include "ReplayOptions.hpp" -#if defined(UMPIRE_ENABLE_NUMA) -#include "umpire/strategy/NumaPolicy.hpp" -#include "umpire/util/numa.hpp" -#endif // defined(UMPIRE_ENABLE_NUMA) - -#if !defined(_MSC_VER) -#include // getpid() -#else -#include -#define getpid _getpid -#endif - -ReplayOperationManager::ReplayOperationManager( const ReplayOptions& options, - ReplayFile* rFile, ReplayFile::Header* Operations ) - : m_options(options), m_replay_file(rFile), m_ops_table(Operations) -{ -} - -ReplayOperationManager::~ReplayOperationManager() -{ - for (std::size_t i = 0; i < m_ops_table->num_allocators; i++) { - auto alloc = &m_ops_table->allocators[i]; - if (alloc->allocator != nullptr) - delete(alloc->allocator); - } -} - -namespace { - struct TrackedCounter { - void increment() { - current_count++; - if (current_count > high_watermark) - high_watermark = current_count; - }; - - void decrement() { - current_count--; - }; - - std::size_t current_count{0}; - std::size_t high_watermark{0}; - }; - - // - // https://stackoverflow.com/questions/11376288/fast-computing-of-log2-for-64-bit-integers - // - const int tab64[64] = { - 63, 0, 58, 1, 59, 47, 53, 2, - 60, 39, 48, 27, 54, 33, 42, 3, - 61, 51, 37, 40, 49, 18, 28, 20, - 55, 30, 34, 11, 43, 14, 22, 4, - 62, 57, 46, 52, 38, 26, 32, 41, - 50, 36, 17, 19, 29, 10, 13, 21, - 56, 45, 25, 31, 35, 16, 9, 12, - 44, 24, 15, 8, 23, 7, 6, 5}; - - int log2_64 (std::size_t size) - { - uint64_t value{static_cast(size)}; - value |= value >> 1; - value |= value >> 2; - value |= value >> 4; - value |= value >> 8; - value |= value >> 16; - value |= value >> 32; - uint64_t multiplier = UINT64_C(0x07EDD5E59A4E28C2); - uint64_t shifter = UINT64_C(58); - value = value - (value >> 1); - int index = (value * multiplier) >> shifter; - return tab64[index]; - } - - struct TrackedHistogram { - void increment(std::size_t size) { - int index{ log2_64(size) }; - - allocations++; - allocation_count++; - if ( size > largest_allocation ) - largest_allocation = size; - - log2_buckets[index].increment(); - }; - - void decrement(std::size_t size) { - int index{ log2_64(size) }; - deallocations++; - allocation_count--; - log2_buckets[index].decrement(); - }; - - void print(std::string name) const { - if (largest_allocation > 0) { - std::cout << std::endl << name << ":" << std::endl; - std::cout << " Total Deallocations: " << deallocations << std::endl; - std::cout << " Total Allocations: " << allocations << std::endl; - std::cout << " Largest Allocation: " << largest_allocation << std::endl; - std::cout << " Allocation Sizes Histogram (High watermark of allocations):" << std::endl; - for ( int i = 0; i < 64; i++ ) { - if (log2_buckets[i].high_watermark) { - std::cout << " [2^" << i << " - 2^" << i+1 << ") = "; - std::cout << log2_buckets[i].high_watermark << std::endl; - } - } - std::cout << std::endl; - } - }; - - TrackedCounter log2_buckets[64]{}; - std::size_t largest_allocation{0}; - std::size_t allocations{0}; - std::size_t deallocations{0}; - std::size_t allocation_count{0}; - }; -} - -void ReplayOperationManager::runOperations() -{ - std::map size_histogram; - std::size_t op_counter{0}; - auto& rm = umpire::ResourceManager::getInstance(); - - const int name_width{40}; - const int num_width{16}; - if (m_options.track_stats) { - std::cout - << std::setw(name_width) << std::left << "Filename" - << std::setw(name_width) << std::left << "Allocator" - << std::setw(num_width) << std::left << "Current Size" - << std::setw(num_width) << std::left << "Actual Size" - << std::setw(num_width) << std::left << "High Watermark" - << std::endl; - } - - for ( auto op = &m_ops_table->ops[1]; - op < &m_ops_table->ops[m_ops_table->num_operations]; - ++op) - { - try { - switch (op->op_type) { - case ReplayFile::otype::ALLOCATOR_CREATION: - if (m_options.track_stats) { - size_histogram[op->op_allocator] = TrackedHistogram{}; - } - makeAllocator(op); - break; - case ReplayFile::otype::SETDEFAULTALLOCATOR: - if (m_options.track_stats) { - size_histogram[op->op_allocator] = TrackedHistogram{}; - } - makeSetDefaultAllocator(op); - break; - case ReplayFile::otype::COPY: - if (m_options.skip_operations == false) { - makeCopy(op); - } - break; - case ReplayFile::otype::REALLOCATE: - makeReallocate(op); - break; - case ReplayFile::otype::REALLOCATE_EX: - makeReallocate_ex(op); - break; - case ReplayFile::otype::ALLOCATE: - if (m_options.track_stats || m_options.dump_statistics) { - size_histogram[op->op_allocator].increment(op->op_size); - } - makeAllocate(op); - break; - case ReplayFile::otype::DEALLOCATE: - if (m_options.track_stats || m_options.dump_statistics) { - auto alloc = &m_ops_table->allocators[op->op_allocator]; - auto ptr = m_ops_table->ops[op->op_alloc_ops[0]].op_allocated_ptr; - size_histogram[op->op_allocator].decrement(alloc->allocator->getSize(ptr)); - } - makeDeallocate(op); - break; - case ReplayFile::otype::COALESCE: - makeCoalesce(op); - break; - case ReplayFile::otype::RELEASE: - makeRelease(op); - break; - default: - REPLAY_ERROR("Unknown operation type: " << op->op_type); - break; - } - } - catch(...) { - std::cerr << std::endl << std::endl - << "Replay Failure Line Number: " << std::endl - << " Line: " << op->op_line_number << m_replay_file->getLine(op->op_line_number) - << std::endl << std::endl; - throw; - } - - if (m_options.dump_statistics) { - for (std::size_t i = 0; i < m_ops_table->num_allocators; i++) { - auto alloc = &m_ops_table->allocators[i]; - if (alloc->allocator == nullptr) - continue; - - auto alloc_name = alloc->allocator->getName(); - - std::string cur_stat_name{alloc_name + " current_size"}; - std::string actual_stat_name{alloc_name + " actual_size"}; - std::string hwm_stat_name{alloc_name + " hwm"}; - std::string allocs_stat_name{alloc_name + " allocation count"}; - - m_stat_series[cur_stat_name].push_back( - std::make_pair( - op_counter, - alloc->allocator->getCurrentSize())); - - m_stat_series[actual_stat_name].push_back( - std::make_pair( - op_counter, - alloc->allocator->getActualSize())); - - m_stat_series[hwm_stat_name].push_back( - std::make_pair( - op_counter, - alloc->allocator->getHighWatermark())); - - m_stat_series[allocs_stat_name].push_back( - std::make_pair( - op_counter, - size_histogram[i].allocation_count)); - - auto strategy = alloc->allocator->getAllocationStrategy(); - umpire::strategy::QuickPool* qp_strat{dynamic_cast(strategy)}; - - if (qp_strat != nullptr) { - std::string blocks_name{alloc_name + " total blocks"}; - std::string releasable_blocks_name{alloc_name + " releasable blocks"}; - - m_stat_series[releasable_blocks_name].push_back( - std::make_pair(op_counter, qp_strat->getReleasableBlocks())); - - m_stat_series[blocks_name].push_back( - std::make_pair(op_counter, qp_strat->getTotalBlocks())); - } - else { - umpire::strategy::DynamicPoolList* dpl_strat{dynamic_cast(strategy)}; - - if (dpl_strat != nullptr) { - std::string blocks_name{alloc_name + " total blocks"}; - std::string releasable_blocks_name{alloc_name + " releasable blocks"}; - - m_stat_series[releasable_blocks_name].push_back( - std::make_pair(op_counter, dpl_strat->getReleasableBlocks())); - - m_stat_series[blocks_name].push_back( - std::make_pair(op_counter, dpl_strat->getTotalBlocks())); - } - } - } - } - op_counter++; - } - - if (m_options.dump_statistics) { - dumpStats(); - } - - if (m_options.track_stats) { - for (const auto& alloc_name : rm.getAllocatorNames()) { - auto alloc = rm.getAllocator(alloc_name); - if (alloc.getHighWatermark()) { - std::cout - << std::setw(name_width) << std::left << m_replay_file->getInputFileName() - << std::setw(name_width) << std::left << alloc_name - << std::setw(num_width) << std::left << alloc.getCurrentSize() - << std::setw(num_width) << std::left << alloc.getActualSize() - << std::setw(num_width) << std::left << alloc.getHighWatermark() - << std::endl; - } - } - - for (auto const& x : size_histogram) - { - auto alloc = &m_ops_table->allocators[x.first]; - x.second.print(alloc->allocator->getName()); - } - } -} - -void ReplayOperationManager::makeAllocator(ReplayFile::Operation* op) -{ - auto alloc = &m_ops_table->allocators[op->op_allocator]; - auto& rm = umpire::ResourceManager::getInstance(); - - // - // Check to see if user requested that we switch to a different pool - // - if ( !m_options.pool_to_use.empty() ) { - if (alloc->type == ReplayFile::rtype::DYNAMIC_POOL_LIST || alloc->type == ReplayFile::rtype::QUICKPOOL) { - if (m_options.pool_to_use == "List") { - alloc->type = ReplayFile::rtype::DYNAMIC_POOL_LIST; - } - else if (m_options.pool_to_use == "Quick") { - alloc->type = ReplayFile::rtype::QUICKPOOL; - } - } - } - - switch (alloc->type) { - case ReplayFile::rtype::MEMORY_RESOURCE: - alloc->allocator = new umpire::Allocator(rm.getAllocator(alloc->name)); - break; - - case ReplayFile::rtype::ALLOCATION_ADVISOR: - if (alloc->argv.advisor.device_id >= 0) { // Optional device ID specified - switch ( alloc->argc ) { - default: - REPLAY_ERROR("Invalid number of arguments " << alloc->argc); - case 3: - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.advisor.advice - , alloc->argv.advisor.device_id - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.advisor.advice - , alloc->argv.advisor.device_id - ) - ); - } - break; - case 4: - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.advisor.advice - , rm.getAllocator(alloc->argv.advisor.accessing_allocator) - , alloc->argv.advisor.device_id - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.advisor.advice - , rm.getAllocator(alloc->argv.advisor.accessing_allocator) - , alloc->argv.advisor.device_id - ) - ); - } - break; - } - } - else { // Use default device_id - switch ( alloc->argc ) { - default: - REPLAY_ERROR("Invalid number of arguments " << alloc->argc); - case 2: - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.advisor.advice - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.advisor.advice - ) - ); - } - break; - case 3: - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.advisor.advice - , rm.getAllocator(alloc->argv.advisor.accessing_allocator) - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.advisor.advice - , rm.getAllocator(alloc->argv.advisor.accessing_allocator) - ) - ); - } - break; - } - } - break; - - case ReplayFile::rtype::ALLOCATION_PREFETCHER: - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - ) - ); - } - break; - - case ReplayFile::rtype::NUMA_POLICY: -#if defined(UMPIRE_ENABLE_NUMA) - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.numa.node - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.numa.node - ) - ); - } -#else - std::cerr - << "Warning, NUMA policy operation found and skipped, consider building" - << std::endl - << "version of replay with -DENABLE_NUMA=On." - << std::endl; -#endif // defined(UMPIRE_ENABLE_NUMA) - break; - - case ReplayFile::rtype::QUICKPOOL: - if (!m_options.heuristic_to_use.empty()) { - std::size_t init_alloc_size{ alloc->argv.pool.initial_alloc_size }; - std::size_t min_alloc_size{ alloc->argv.pool.min_alloc_size }; - std::size_t alignment{ static_cast(alloc->argv.pool.alignment) }; - umpire::strategy::PoolCoalesceHeuristic heuristic{umpire::strategy::QuickPool::percent_releasable(100)}; - - if (alloc->argc == 1) { - init_alloc_size = umpire::strategy::QuickPool::s_default_first_block_size; - min_alloc_size = umpire::strategy::QuickPool::s_default_next_block_size; - alignment = umpire::strategy::QuickPool::s_default_alignment; - } - else if (alloc->argc == 2) { - min_alloc_size = umpire::strategy::QuickPool::s_default_next_block_size; - alignment = umpire::strategy::QuickPool::s_default_alignment; - } - if (alloc->argc == 3) { - alignment = umpire::strategy::QuickPool::s_default_alignment; - } - - if (m_options.heuristic_to_use == "Block") { - heuristic = umpire::strategy::QuickPool::blocks_releasable(m_options.heuristic_parm); - } - else if (m_options.heuristic_to_use == "Block_hwm") { - heuristic = umpire::strategy::QuickPool::blocks_releasable_hwm(m_options.heuristic_parm); - } - else if (m_options.heuristic_to_use == "FreePercentage") { - heuristic = umpire::strategy::QuickPool::percent_releasable(m_options.heuristic_parm); - } - else if (m_options.heuristic_to_use == "FreePercentage_hwm") { - heuristic = umpire::strategy::QuickPool::percent_releasable_hwm(m_options.heuristic_parm); - } - - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , init_alloc_size - , min_alloc_size - , alignment - , heuristic)); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , init_alloc_size - , min_alloc_size - , alignment)); - } - } - else if (alloc->argc >= 4) { - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.pool.initial_alloc_size - , alloc->argv.pool.min_alloc_size - , alloc->argv.pool.alignment - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.pool.initial_alloc_size - , alloc->argv.pool.min_alloc_size - , alloc->argv.pool.alignment - ) - ); - } - } - else if (alloc->argc == 3) { - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.pool.initial_alloc_size - , alloc->argv.pool.min_alloc_size - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.pool.initial_alloc_size - , alloc->argv.pool.min_alloc_size - ) - ); - } - } - else if (alloc->argc == 2) { - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.pool.initial_alloc_size - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.pool.initial_alloc_size - ) - ); - } - } - else { - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - ) - ); - } - } - break; - - case ReplayFile::rtype::DYNAMIC_POOL_LIST: - if (!m_options.heuristic_to_use.empty()) { - std::size_t init_alloc_size{alloc->argv.pool.initial_alloc_size}; - std::size_t min_alloc_size{alloc->argv.pool.min_alloc_size}; - std::size_t alignment{static_cast(alloc->argv.pool.alignment)}; - umpire::strategy::PoolCoalesceHeuristic heuristic{ - umpire::strategy::DynamicPoolList::percent_releasable(100)}; - - if (alloc->argc == 1) { - init_alloc_size = umpire::strategy::DynamicPoolList::s_default_first_block_size; - min_alloc_size = umpire::strategy::DynamicPoolList::s_default_next_block_size; - alignment = umpire::strategy::DynamicPoolList::s_default_alignment; - } else if (alloc->argc == 2) { - min_alloc_size = umpire::strategy::DynamicPoolList::s_default_next_block_size; - alignment = umpire::strategy::DynamicPoolList::s_default_alignment; - } - if (alloc->argc == 3) { - alignment = umpire::strategy::DynamicPoolList::s_default_alignment; - } - - if (m_options.heuristic_to_use == "Block") { - heuristic = umpire::strategy::DynamicPoolList::blocks_releasable(m_options.heuristic_parm); - } - else if (m_options.heuristic_to_use == "Block_hwm") { - heuristic = umpire::strategy::DynamicPoolList::blocks_releasable_hwm(m_options.heuristic_parm); - } - else if (m_options.heuristic_to_use == "FreePercentage") { - heuristic = umpire::strategy::DynamicPoolList::percent_releasable(m_options.heuristic_parm); - } - else if (m_options.heuristic_to_use == "FreePercentage_hwm") { - heuristic = umpire::strategy::DynamicPoolList::percent_releasable_hwm(m_options.heuristic_parm); - } - - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator(rm.makeAllocator( - alloc->name, rm.getAllocator(alloc->base_name), init_alloc_size, min_alloc_size, alignment, heuristic)); - } else { - alloc->allocator = new umpire::Allocator(rm.makeAllocator( - alloc->name, rm.getAllocator(alloc->base_name), init_alloc_size, min_alloc_size, alignment)); - } - } else if (alloc->argc >= 4) { - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator(rm.makeAllocator( - alloc->name, rm.getAllocator(alloc->base_name), alloc->argv.pool.initial_alloc_size, - alloc->argv.pool.min_alloc_size, alloc->argv.pool.alignment)); - } else { - alloc->allocator = new umpire::Allocator(rm.makeAllocator( - alloc->name, rm.getAllocator(alloc->base_name), alloc->argv.pool.initial_alloc_size, - alloc->argv.pool.min_alloc_size, alloc->argv.pool.alignment)); - } - } else if (alloc->argc == 3) { - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator(rm.makeAllocator( - alloc->name, rm.getAllocator(alloc->base_name), alloc->argv.pool.initial_alloc_size, - alloc->argv.pool.min_alloc_size)); - } else { - alloc->allocator = new umpire::Allocator(rm.makeAllocator( - alloc->name, rm.getAllocator(alloc->base_name), alloc->argv.pool.initial_alloc_size, - alloc->argv.pool.min_alloc_size)); - } - } else if (alloc->argc == 2) { - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator(rm.makeAllocator( - alloc->name, rm.getAllocator(alloc->base_name), alloc->argv.pool.initial_alloc_size)); - } else { - alloc->allocator = new umpire::Allocator(rm.makeAllocator( - alloc->name, rm.getAllocator(alloc->base_name), alloc->argv.pool.initial_alloc_size)); - } - } else { - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator(rm.makeAllocator( - alloc->name, rm.getAllocator(alloc->base_name))); - } else { - alloc->allocator = new umpire::Allocator(rm.makeAllocator( - alloc->name, rm.getAllocator(alloc->base_name))); - } - } - break; - - case ReplayFile::rtype::MONOTONIC: - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.monotonic_pool.capacity - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.monotonic_pool.capacity - ) - ); - } - break; - - case ReplayFile::rtype::SLOT_POOL: - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.slot_pool.slots - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.slot_pool.slots - ) - ); - } - break; - - case ReplayFile::rtype::SIZE_LIMITER: - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.size_limiter.size_limit - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.size_limiter.size_limit - ) - ); - } - break; - - case ReplayFile::rtype::THREADSAFE_ALLOCATOR: - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - ) - ); - } - break; - - case ReplayFile::rtype::FIXED_POOL: - if (alloc->argc >= 3) { - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.fixed_pool.object_bytes - , alloc->argv.fixed_pool.objects_per_pool - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.fixed_pool.object_bytes - , alloc->argv.fixed_pool.objects_per_pool - ) - ); - } - } - else { - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.fixed_pool.object_bytes - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.fixed_pool.object_bytes - ) - ); - } - } - break; - - case ReplayFile::rtype::NAMED: - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - ) - ); - } else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - ) - ); - } - break; - - case ReplayFile::rtype::MIXED_POOL: - if (alloc->argc >= 8) { - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.mixed_pool.smallest_fixed_blocksize - , alloc->argv.mixed_pool.largest_fixed_blocksize - , alloc->argv.mixed_pool.max_fixed_blocksize - , alloc->argv.mixed_pool.size_multiplier - , alloc->argv.mixed_pool.dynamic_initial_alloc_bytes - , alloc->argv.mixed_pool.dynamic_min_alloc_bytes - , alloc->argv.mixed_pool.dynamic_align_bytes - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.mixed_pool.smallest_fixed_blocksize - , alloc->argv.mixed_pool.largest_fixed_blocksize - , alloc->argv.mixed_pool.max_fixed_blocksize - , alloc->argv.mixed_pool.size_multiplier - , alloc->argv.mixed_pool.dynamic_initial_alloc_bytes - , alloc->argv.mixed_pool.dynamic_min_alloc_bytes - , alloc->argv.mixed_pool.dynamic_align_bytes - ) - ); - } - } - else if (alloc->argc >= 7) { - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.mixed_pool.smallest_fixed_blocksize - , alloc->argv.mixed_pool.largest_fixed_blocksize - , alloc->argv.mixed_pool.max_fixed_blocksize - , alloc->argv.mixed_pool.size_multiplier - , alloc->argv.mixed_pool.dynamic_initial_alloc_bytes - , alloc->argv.mixed_pool.dynamic_min_alloc_bytes - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.mixed_pool.smallest_fixed_blocksize - , alloc->argv.mixed_pool.largest_fixed_blocksize - , alloc->argv.mixed_pool.max_fixed_blocksize - , alloc->argv.mixed_pool.size_multiplier - , alloc->argv.mixed_pool.dynamic_initial_alloc_bytes - , alloc->argv.mixed_pool.dynamic_min_alloc_bytes - ) - ); - } - } - else if (alloc->argc >= 6) { - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.mixed_pool.smallest_fixed_blocksize - , alloc->argv.mixed_pool.largest_fixed_blocksize - , alloc->argv.mixed_pool.max_fixed_blocksize - , alloc->argv.mixed_pool.size_multiplier - , alloc->argv.mixed_pool.dynamic_initial_alloc_bytes - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.mixed_pool.smallest_fixed_blocksize - , alloc->argv.mixed_pool.largest_fixed_blocksize - , alloc->argv.mixed_pool.max_fixed_blocksize - , alloc->argv.mixed_pool.size_multiplier - , alloc->argv.mixed_pool.dynamic_initial_alloc_bytes - ) - ); - } - } - else if (alloc->argc >= 5) { - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.mixed_pool.smallest_fixed_blocksize - , alloc->argv.mixed_pool.largest_fixed_blocksize - , alloc->argv.mixed_pool.max_fixed_blocksize - , alloc->argv.mixed_pool.size_multiplier - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.mixed_pool.smallest_fixed_blocksize - , alloc->argv.mixed_pool.largest_fixed_blocksize - , alloc->argv.mixed_pool.max_fixed_blocksize - , alloc->argv.mixed_pool.size_multiplier - ) - ); - } - } - else if (alloc->argc >= 4) { - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.mixed_pool.smallest_fixed_blocksize - , alloc->argv.mixed_pool.largest_fixed_blocksize - , alloc->argv.mixed_pool.max_fixed_blocksize - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.mixed_pool.smallest_fixed_blocksize - , alloc->argv.mixed_pool.largest_fixed_blocksize - , alloc->argv.mixed_pool.max_fixed_blocksize - ) - ); - } - } - else if (alloc->argc >= 3) { - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.mixed_pool.smallest_fixed_blocksize - , alloc->argv.mixed_pool.largest_fixed_blocksize - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.mixed_pool.smallest_fixed_blocksize - , alloc->argv.mixed_pool.largest_fixed_blocksize - ) - ); - } - } - else if (alloc->argc >= 2) { - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.mixed_pool.smallest_fixed_blocksize - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - , alloc->argv.mixed_pool.smallest_fixed_blocksize - ) - ); - } - } - else { - if (alloc->introspection) { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - ) - ); - } - else { - alloc->allocator = new umpire::Allocator( - rm.makeAllocator - ( alloc->name - , rm.getAllocator(alloc->base_name) - ) - ); - } - } - break; - - default: - REPLAY_ERROR("Unknown allocator type: " << alloc->type); - break; - } -} - -void ReplayOperationManager::makeAllocate(ReplayFile::Operation* op) -{ - auto alloc = &m_ops_table->allocators[op->op_allocator]; - - op->op_allocated_ptr = alloc->allocator->allocate(op->op_size); -} - -void ReplayOperationManager::makeSetDefaultAllocator(ReplayFile::Operation* op) -{ - auto alloc = &m_ops_table->allocators[op->op_allocator]; - auto& rm = umpire::ResourceManager::getInstance(); - rm.setDefaultAllocator(*(alloc->allocator)); -} - -void ReplayOperationManager::makeReallocate(ReplayFile::Operation* op) -{ - auto& rm = umpire::ResourceManager::getInstance(); - auto ptr = m_ops_table->ops[op->op_alloc_ops[1]].op_allocated_ptr; - op->op_allocated_ptr = rm.reallocate(ptr, op->op_size); -} - -void ReplayOperationManager::makeReallocate_ex(ReplayFile::Operation* op) -{ - auto alloc = &m_ops_table->allocators[op->op_allocator]; - auto& rm = umpire::ResourceManager::getInstance(); - auto ptr = m_ops_table->ops[op->op_alloc_ops[1]].op_allocated_ptr; - op->op_allocated_ptr = rm.reallocate(ptr, op->op_size, *(alloc->allocator)); -} - -void ReplayOperationManager::makeCopy(ReplayFile::Operation* op) -{ - auto& rm = umpire::ResourceManager::getInstance(); - char* src_ptr = static_cast(m_ops_table->ops[op->op_alloc_ops[0]].op_allocated_ptr); - char* dst_ptr = static_cast(m_ops_table->ops[op->op_alloc_ops[1]].op_allocated_ptr); - auto src_off = op->op_offsets[0]; - auto dst_off = op->op_offsets[1]; - auto size = op->op_size; - - rm.copy(dst_ptr+dst_off, src_ptr+src_off, size); -} - -void ReplayOperationManager::makeDeallocate(ReplayFile::Operation* op) -{ - try { - auto alloc = &m_ops_table->allocators[op->op_allocator]; - auto ptr = m_ops_table->ops[op->op_alloc_ops[0]].op_allocated_ptr; - alloc->allocator->deallocate(ptr); - } - catch (...) { - std::cerr << std::endl - << "Deallocation Failure Line Number: " << std::endl - << " Line: " << op->op_line_number << m_replay_file->getLine(op->op_line_number) << std::endl - << " for memory allocation at:" << std::endl - << " Line: " << m_ops_table->ops[op->op_alloc_ops[0]].op_line_number - << m_replay_file->getLine( - m_ops_table->ops[op->op_alloc_ops[0]].op_line_number) - << std::endl << std::endl; - throw; - } -} - -void ReplayOperationManager::makeCoalesce(ReplayFile::Operation* op) -{ - auto alloc = &m_ops_table->allocators[op->op_allocator]; - - switch (alloc->type) { - case ReplayFile::rtype::QUICKPOOL: - { - auto quick_pool = umpire::util::unwrap_allocator(*(alloc->allocator)); - quick_pool->coalesce(); - } - break; - case ReplayFile::rtype::DYNAMIC_POOL_LIST: - { - auto dynamic_pool = umpire::util::unwrap_allocator(*(alloc->allocator)); - dynamic_pool->coalesce(); - } - break; - default: - std::cerr << std::endl - << "Unknown pool type for coalesce: %d Skipped" << alloc->type << std::endl; - break; - } -} - -void ReplayOperationManager::makeRelease(ReplayFile::Operation* op) -{ - auto alloc = &m_ops_table->allocators[op->op_allocator]; - alloc->allocator->release(); -} - -void ReplayOperationManager::dumpStats() -{ - std::ofstream file; - const int pid{getpid()}; - - const std::string filename{"replay" + std::to_string(pid) + ".ult"}; - file.open(filename); - - for (const auto& stat_series : m_stat_series) { - file << "# " << stat_series.first << std::endl; - for (const auto& entry : stat_series.second) { - int t; - std::size_t val; - std::tie(t, val) = entry; - - file - << t - << " " << val << std::endl; - } - } -} - -#endif // !defined(_MSC_VER) && !defined(_LIBCPP_VERSION) diff --git a/tools/replay/ReplayOperationManager.hpp b/tools/replay/ReplayOperationManager.hpp deleted file mode 100644 index 686549c26..000000000 --- a/tools/replay/ReplayOperationManager.hpp +++ /dev/null @@ -1,56 +0,0 @@ -////////////////////////////////////////////////////////////////////////////// -// Copyright (c) 2016-26, Lawrence Livermore National Security, LLC and Umpire -// project contributors. See the COPYRIGHT file for details. -// -// SPDX-License-Identifier: (MIT) -////////////////////////////////////////////////////////////////////////////// -#ifndef REPLAY_ReplayOperationManager_HPP -#define REPLAY_ReplayOperationManager_HPP - -#if !defined(_MSC_VER) && !defined(_LIBCPP_VERSION) -#include -#include -#include - -#include "ReplayFile.hpp" -#include "ReplayOptions.hpp" -#include "umpire/Allocator.hpp" -#include "umpire/strategy/AllocationAdvisor.hpp" -#include "umpire/strategy/SizeLimiter.hpp" -#include "umpire/strategy/MixedPool.hpp" -#include "umpire/strategy/QuickPool.hpp" -#include "umpire/strategy/DynamicPoolList.hpp" -#include "umpire/strategy/MonotonicAllocationStrategy.hpp" -#include "umpire/strategy/SlotPool.hpp" -#include "umpire/strategy/ThreadSafeAllocator.hpp" -#include "umpire/util/AllocationRecord.hpp" -#include "umpire/ResourceManager.hpp" - -class ReplayOperationManager { -public: - ReplayOperationManager( const ReplayOptions& options, ReplayFile* rFile, - ReplayFile::Header* Operations ); - ~ReplayOperationManager(); - - void runOperations(); - -private: - std::map>> m_stat_series; - ReplayOptions m_options; - ReplayFile* m_replay_file; - ReplayFile::Header* m_ops_table; - - void makeAllocator(ReplayFile::Operation* op); - void makeAllocate(ReplayFile::Operation* op); - void makeDeallocate(ReplayFile::Operation* op); - void makeSetDefaultAllocator(ReplayFile::Operation* op); - void makeCopy(ReplayFile::Operation* op); - void makeReallocate(ReplayFile::Operation* op); - void makeReallocate_ex(ReplayFile::Operation* op); - void makeCoalesce(ReplayFile::Operation* op); - void makeRelease(ReplayFile::Operation* op); - void dumpStats(); -}; - -#endif // !defined(_MSC_VER) && !defined(_LIBCPP_VERSION) -#endif // REPLAY_ReplayOperationManager_HPP diff --git a/tools/replay/ReplayOptions.hpp b/tools/replay/ReplayOptions.hpp index 8a953712c..66f9ab222 100644 --- a/tools/replay/ReplayOptions.hpp +++ b/tools/replay/ReplayOptions.hpp @@ -8,31 +8,6 @@ #define REPLAY_ReplayOptions_HPP #include -#include "umpire/CLI11/CLI11.hpp" - -struct ReplayUsePoolValidator : public CLI::Validator { - ReplayUsePoolValidator() { - func_ = [](const std::string &str) { - if (str != "Quick" && str != "List") { - return std::string("Invalid pool name, must be Quick or List"); - } - else - return std::string(); - }; - } -}; - -struct ReplayUseHeuristicValidator : public CLI::Validator { - ReplayUseHeuristicValidator() { - func_ = [](const std::string &str) { - if (str != "Block" && str != "FreePercentage" && str != "Block_hwm" && str != "FreePercentage_hwm") { - return std::string("Invalid heuristic name, must be Block, Block_hwm, FreePercentage, or FreePercentage_hwm"); - } - else - return std::string(); - }; - } -}; struct ReplayOptions { ReplayOptions() {}; @@ -40,15 +15,9 @@ struct ReplayOptions { bool time_replay_parse{false}; // --time-parse bool dump_statistics{false}; // -d, --dump bool track_stats{false}; // -s, --stats - bool skip_operations{false}; // --skip-operations bool force_compile{false}; // -r,--recompile - bool do_not_demangle{false}; // --no-demangle bool quiet{false}; // -q,--quiet std::string input_file; // -i,-infile input_file - std::string pool_to_use; // -p,--use-pool - std::string heuristic_to_use{}; // --use-heuristic - int heuristic_parm{2}; // --heuristic-parm }; #endif // REPLAY_ReplayOptions_HPP - diff --git a/tools/replay/replay.cpp b/tools/replay/replay.cpp index bdd6b29ee..c82b82260 100644 --- a/tools/replay/replay.cpp +++ b/tools/replay/replay.cpp @@ -15,65 +15,58 @@ #include "umpire/util/Macros.hpp" -#if !defined(_MSC_VER) && !defined(_LIBCPP_VERSION) +#if !defined(_MSC_VER) #include "ReplayInterpreter.hpp" -#include "ReplayMacros.hpp" #include "ReplayOptions.hpp" #include "umpire/CLI11/CLI11.hpp" -const static ReplayUsePoolValidator ReplayValidPool; -const static ReplayUseHeuristicValidator ReplayValidHeuristic; - -#endif // !defined(_MSC_VER) && !defined(_LIBCPP_VERSION) +#endif // !defined(_MSC_VER) int main(int argc, char* argv[]) { -#if !defined(_MSC_VER) && !defined(_LIBCPP_VERSION) - ReplayOptions options; - CLI::App app{"Replay an umpire session from a file"}; - app.add_option("-i,--infile", options.input_file, "Input file")->required()->check(CLI::ExistingFile); - app.add_flag("-q,--quiet", options.quiet, "Only errors will be displayed."); - app.add_flag("-t,--time-run", options.time_replay_run, "Display time information for replay running operations"); - app.add_flag("-d,--dump", options.dump_statistics, "Dump ULTRA memory usage trace for each Allocator"); - app.add_flag("-s,--stats", options.track_stats, "Track/Display pool allocation size statistics"); - app.add_flag("--no-demangle" , options.do_not_demangle, "Disable demangling of replay file"); - app.add_flag("--skip-operations" , options.skip_operations, "Skip Umpire Operations during replays"); - app.add_flag("-r,--recompile" , options.force_compile, "Force recompile replay binary"); - app.add_option("-p,--use-pool", options.pool_to_use, "Specify pool to use: List or Quick")->check(ReplayValidPool); - app.add_option("--use-heuristic", options.heuristic_to_use, - "Heuristic: Block, Block_hwm, FreePercentage, or FreePercentage_hwm")->check(ReplayValidHeuristic); - app.add_option("--heuristic-parm", options.heuristic_parm, "Heuristic parameter to use")->check(CLI::Range(0,100)); - CLI11_PARSE(app, argc, argv); +#if !defined(_MSC_VER) + try { + ReplayOptions options; + CLI::App app{"Replay an umpire session from a file"}; + app.add_option("-i,--infile", options.input_file, "Input file")->required()->check(CLI::ExistingFile); + app.add_flag("-q,--quiet", options.quiet, "Only errors will be displayed."); + app.add_flag("-t,--time-run", options.time_replay_run, "Display time information for replay running operations"); + app.add_flag("-d,--dump", options.dump_statistics, "Dump ULTRA memory usage trace for each Allocator"); + app.add_flag("-s,--stats", options.track_stats, "Track/Display pool allocation size statistics"); + app.add_flag("-r,--recompile", options.force_compile, "Accepted for compatibility; ignored by replay v2"); + CLI11_PARSE(app, argc, argv); - std::chrono::high_resolution_clock::time_point t1; - std::chrono::high_resolution_clock::time_point t2; - std::chrono::duration time_span; + std::chrono::high_resolution_clock::time_point t1; + std::chrono::high_resolution_clock::time_point t2; + std::chrono::duration time_span; - t1 = std::chrono::high_resolution_clock::now(); - ReplayInterpreter replay(options); + t1 = std::chrono::high_resolution_clock::now(); + ReplayInterpreter replay(options); - replay.buildOperations(); + replay.buildOperations(); - if (options.time_replay_parse) { - t2 = std::chrono::high_resolution_clock::now(); - time_span = std::chrono::duration_cast>(t2 - t1); - std::cout << "Parsing replay log took " << time_span.count() << " seconds." << std::endl; - } + if (options.time_replay_parse) { + t2 = std::chrono::high_resolution_clock::now(); + time_span = std::chrono::duration_cast>(t2 - t1); + std::cout << "Parsing replay log took " << time_span.count() << " seconds." << std::endl; + } - t1 = std::chrono::high_resolution_clock::now(); - replay.runOperations(); + t1 = std::chrono::high_resolution_clock::now(); + replay.runOperations(); - if (options.time_replay_run) { - t2 = std::chrono::high_resolution_clock::now(); - time_span = std::chrono::duration_cast>(t2 - t1); - std::cout << "Running replay took " << time_span.count() << " seconds." << std::endl; + if (options.time_replay_run) { + t2 = std::chrono::high_resolution_clock::now(); + time_span = std::chrono::duration_cast>(t2 - t1); + std::cout << "Running replay took " << time_span.count() << " seconds." << std::endl; + } + } catch (const std::exception& ex) { + std::cerr << ex.what() << std::endl; + return 1; } #else UMPIRE_USE_VAR(argc); UMPIRE_USE_VAR(argv); - std::cerr << "This program requires the ability to demangle C++" << std::endl - << "However, this program was compiled with -stdlib=libc++ which does " << std::endl - << "not have this feature." << std::endl; -#endif // !defined(_MSC_VER) && !defined(_LIBCPP_VERSION) + std::cerr << "Replay tool is not supported on MSVC in this configuration." << std::endl; +#endif // !defined(_MSC_VER) return 0; } diff --git a/tools/replay/replaydiff.cpp b/tools/replay/replaydiff.cpp index 367d0d100..9fa73ce87 100644 --- a/tools/replay/replaydiff.cpp +++ b/tools/replay/replaydiff.cpp @@ -13,57 +13,60 @@ #include "umpire/util/Macros.hpp" -#if !defined(_MSC_VER) && !defined(_LIBCPP_VERSION) +#if !defined(_MSC_VER) #include "ReplayInterpreter.hpp" #include "ReplayOptions.hpp" #include "umpire/CLI11/CLI11.hpp" -#endif // !defined(_MSC_VER) && !defined(_LIBCPP_VERSION) +#endif // !defined(_MSC_VER) int main(int argc, char* argv[]) { -#if !defined(_MSC_VER) && !defined(_LIBCPP_VERSION) - ReplayOptions lhs_options; - ReplayOptions rhs_options; - CLI::App app{"Compare two replay result files created" - " by Umpire library with UMPIRE_REPLAY=On"}; +#if !defined(_MSC_VER) + try { + ReplayOptions lhs_options; + ReplayOptions rhs_options; + CLI::App app{"Compare two replay result files created" + " by Umpire library with UMPIRE_REPLAY=On"}; - std::vector positional_args; + std::vector positional_args; - app.add_flag("-r,--recompile" , lhs_options.force_compile, - "Force recompile replay binary"); + app.add_flag("-r,--recompile" , lhs_options.force_compile, + "Force recompile replay binary"); - app.add_flag("-q,--quiet", lhs_options.quiet, - "Only errors will be displayed."); + app.add_flag("-q,--quiet", lhs_options.quiet, + "Only errors will be displayed."); - app.add_option("files", positional_args, "replay_file_1 replay_file_2") - ->required() - ->expected(2) - ->check(CLI::ExistingFile); + app.add_option("files", positional_args, "replay_file_1 replay_file_2") + ->required() + ->expected(2) + ->check(CLI::ExistingFile); - CLI11_PARSE(app, argc, argv); + CLI11_PARSE(app, argc, argv); - rhs_options = lhs_options; + rhs_options = lhs_options; - lhs_options.input_file = positional_args[0]; - rhs_options.input_file = positional_args[1]; + lhs_options.input_file = positional_args[0]; + rhs_options.input_file = positional_args[1]; - ReplayInterpreter left{lhs_options}; - ReplayInterpreter right{rhs_options}; + ReplayInterpreter left{lhs_options}; + ReplayInterpreter right{rhs_options}; - left.buildOperations(); - right.buildOperations(); + left.buildOperations(); + right.buildOperations(); - if (left.compareOperations(right) == false) { - std::cerr << "Miscompare:" << std::endl; - return -2; + if (left.compareOperations(right) == false) { + std::cerr << "Miscompare:" << std::endl; + return -2; + } + } catch (const std::exception& ex) { + std::cerr << ex.what() << std::endl; + return 1; } #else UMPIRE_USE_VAR(argc); UMPIRE_USE_VAR(argv); - std::cerr << "This program requires the ability to demangle C++" << std::endl - << "However, this program was compiled with -stdlib=libc++ which does " << std::endl - << "not have this feature." << std::endl; -#endif // !defined(_MSC_VER) && !defined(_LIBCPP_VERSION) + std::cerr << "Replay diff tool is not supported on MSVC in this configuration." << std::endl; +#endif // !defined(_MSC_VER) return 0; }