Benchmarking MLPerf DLRM_v2 on CPU

DLRM_v2 model

The Deep Learning Recommendation Model (DLRM) v2 is a sophisticated neural network model specifically designed for recommendation systems. It combines both dense and sparse features to deliver highly accurate and personalized recommendations. The model architecture includes embedding tables for handling categorical data and multi-layer perceptrons (MLPs) for processing dense features. It also contains a Deep & Cross Network (DCN) for capturing complex feature interactions.
This package is put together to benchmark the performance of DLRM_v2 on CPU. The implementation of this DLRM_v2 is the MLPerf DLRM_V2 model for the Criteo 1TB click logs dataset. Follow the instructions below to run the benchmarking.

Download docker image

# Download the docker image
wget https://shared-dataset-bucket-2025.s3.us-east-2.amazonaws.com/dlrm_v2_benchmark_docker_img.tar

# load docker image
sudo docker load -i dlrm_v2_benchmark_docker_img.tar

Download benchmarking dataset

The testing dataset for DLRM_v2 is the Criteo 1TB Click Logs dataset. This dataset contains feature values and click feedback for millions of display ads. Its purpose is to benchmark algorithms for clickthrough rate (CTR) prediction. There are 13 features taking integer values (mostly count features) and 26 categorical features. The full dataset contains 24 days of click logs. For benchmarking DLRM_v2, only day 24 of the dataset is used. The preprcessing steps for the dataset can be found from the MLPerf inference reference implementation. The preprocessed dataset can be downloaded from the following link:

wget https://shared-dataset-bucket-2025.s3.us-east-2.amazonaws.com/criteo_multi_hot.tar
tar -xf criteo_multi_hot.tar

Download models

Following the the MLPerf inference reference implementation to download the DLRM_v2 model weights. Run python/dump_dlrm_dcn.py to dump the model weights in a pt file. Run python/quantize_models.py to generate quantized models of different precision (BF16 or INT8) using different quantization methods (dynamic or static) and different backend (see instructions below).
Already quantized models can be downloaded from the following link:

wget https://shared-dataset-bucket-2025.s3.us-east-2.amazonaws.com/dlrm_v2_models.tar
tar -xf dlrm_v2_models.tar

Generate quantized models

This package supports different backends (i.e., IPEX, zentorch, pytorch) for compiling the model and also supports different quantization approaches (i.e., dynamic and static).
This is a list of supported models:

• zentorch_compiled_bf16
• zentorch_compiled_fp32
• torch_dynamic_int8
• torch_static_int8
• torch_compiled_bf16
• ipex_compiled_bf16
• ipex_compiled_fp32
• ipex_static_int8
• ipex_dynamic_int8

Use exactly these model names when editing the batch run script. Those compiled models will be generated on the fly and not saved. The INT8 quantized models need to be generated using the python/quantize_models.py script before running benchmarking, if they are not already downloaded. Run the following command to generate INT8 quantized model:

python python/quantize_models.py --model-names ipex_static_int8 --dump --accuracy --max-samples 100000

arguments explained:

• --dump: save the model weights
• --accuracy: run an accuracy test
• --max-samples: number of samples used for accuracy test

Start container

Set correct directory paths for environment variables: MODEL_DIR, DATA_DIR, RESULTS_DIR, run the following command to start the container:

# set paths for models, data, and results
export MODEL_DIR=path/to/models
export DATA_DIR=path/to/criteo_multi_hot
export RESULTS_DIR=path/to/results/save
sudo docker run -it --rm --privileged \
	-v $RESULTS_DIR:/results \
	-v $MODEL_DIR:/models \
	-v $DATA_DIR:/data/criteo_multi_hot \
	dlrm_v2_benchmark

Run benchmarking in the container

Once inside the container, edit the environment variables in the setup_env.sh file to match the desired configuration. Set the CPU_NAME to match your CPU model.
Edit the script run_batch.sh to add desired tests and run the script to start the benchmarking:

./run_batch.sh

Here is an example of the run_batch.sh script. Edit this for your specific test.

test_name="06062025_dlrm_turinC"
models=("ipex_static_int8" "ipex_dynamic_int8" "torch_static_int8" "torch_dynamic_int8" "ipex_compiled_bf16" "zentorch_compiled_bf16" "torch_compiled_bf16")
model_precisions=("" "" "" "" "--use-bf16" "--use-bf16" "--use-bf16")
num_workergroups=1
n_models=${#model_precisions[@]}
max_batchsizes="800 1600 3200 6400"
num_instances="16 32 64"
counter=0

for rep in 1 2 3; do
for i in $(seq 0 $((n_models-1))); do
    model=${models[$i]}
    use_bf16=${model_precisions[$i]}
    extra_opts=$use_bf16
    counter=$((counter+1))
    echo "working on model No. ${counter}......"
    for num_instance in $num_instances; do
        if [ $num_instance -le 12 ]; then
            target_qps=1000
        else
            target_qps=2500
        fi
        for max_batchsize in $max_batchsizes; do
            echo "working ${model}_${num_instance}_${max_batchsize}"
            ./run_clean.sh
	        sleep 5
            folder_name="${model}_${num_instance}_${max_batchsize}_rep${rep}"
            echo "output directory: $folder_name"
            ./run_main.sh --test-name $test_name --folder-name $folder_name --scenario Offline \
                 --model $model --num-instances $num_instance --num-workergroups $num_workergroups \
                 --max-batchsize $max_batchsize --target-qps $target_qps --samples-per-query-offline 204800 \
                 $extra_opts

	    # exit 1
        done
    done
done

done

Key variables explained:

• models: an array of models to be tested. The name must be in the supported model list
• model_precisions: It is the corresponding precision flag for each models. It has the same lenght as models array. For BF16 models, use flag --use-bf16. For other models, use a blank string.
• num_workergoups: A worker group is a process that loads the model and spawns multiple workers/instances to run the inference in paralell. Inside a worker group, the model weights are shared among the workers. For single socket servers, the number of worker groups should be set to 1. For dual socket servers, the number of worker groups should be set to 2.
• max_batchsize: maximum number of records in each batch.
• num_instances: Number of model instances to run in parallel.
• target_qps: expected samples per second for the server under test. Set it bigger to increase the testing time.

Other arguments can be passed to run_main.sh script:

• --num-cores-to-use: Number of total cores to use for parallel inference. Default is to use all CPU cores
• --core-per-instance: Number of cores to use for each model instance. The total number of cores should be equal to num of instances times cores per instance. It is calculated automatically.
• --bind-to-cpu-ids: List of CPU IDs to bind the model to. The IDs should be separated by commas. For example "0,1,2,3" or "0-3". If not set, the model will be run on all available CPUs. Note: For some dual socket servers, the CPU IDs may be interleaved between the two sockets. In this case, the CPU IDs for socket 0 should be "0,2,4,6,..." and for socket 1 should be "1,3,5,7,...".
• The memory allocation libraries (e.g., tcmalloc, jemalloc) and the OpenMP libraries (e.g., libiomp5 and libgomp) can be set in the setup_env.sh file.

Note: use htop to monitor the CPU utilization and the number of cores used by the model. The CPU utilization should be close to 100% and the number of cores used should be equal to the number of cores assigned to the model.

Get benchmarking results

The following files are collected for each run a folder.

-rw-r--r-- 1 root root 2.8K Jan 12 21:05 args.json
-rw-r--r-- 1 root root 1.5K Jan 12 21:01 bios_info.txt
-rw-r--r-- 1 root root 3.3K Jan 12 21:01 cpu_info.txt
-rw-r--r-- 1 root root 7.5M Jan 12 21:01 dlrm_trace_of_aggregated_samples.txt
-rw-r--r-- 1 root root  22K Jan 12 21:01 dmidecode_memory.txt
-rw-r--r-- 1 root root 7.3K Jan 12 21:01 memory_info.txt
-rw-r--r-- 1 root root    4 Jan 12 21:04 mlperf_log_accuracy.json
-rw-r--r-- 1 root root 162K Jan 12 21:04 mlperf_log_detail.txt
-rw-r--r-- 1 root root 1.7K Jan 12 21:04 mlperf_log_summary.txt
-rw-r--r-- 1 root root 206M Jan 12 21:04 mlperf_log_trace.json
-rw-r--r-- 1 root root  520 Jan 12 21:01 numa_nodes.txt
-rw-r--r-- 1 root root 1.8K Jan 12 21:01 packages.txt
-rw-r--r-- 1 root root 2.5K Jan 12 21:01 printenv.txt
-rw-r--r-- 1 root root 2.2K Jan 12 21:05 results.json
-rw-r--r-- 1 root root  132 Jan 12 21:01 uname.txt

In each test folder, there is a file named mlperf_log_summary.txt. The throughput, in terms of samples per second, can be extracted from this file. Run the script python/extract_results.py to extract the throughput from all the test folders and save the results in a csv file:

python python/extract_results.py /results/cpu_name

To run the dlrm_int8 model

This section is only for running Intel optimized INT8 model: dlrm_int8.pt. The dlrm_int8 model is generated according to the Intel MLperf 4.1 published repository. This model contains Intel customerized operations (torch_ipex::mlperf_merged_emb_cat). Therefore, it should be run with Intel's docker image intel/intel-optimized-pytorch:mlperf-inference-4.1-dlrmv2.

Start a docker container with Intel's image

sudo docker run -it --rm --privileged \
	-v ~/workspace/DLRMv2/DLRM_v2_benchmarking:/myApp \
	-v $(pwd)/results:/results \
	-v $(pwd)/models:/models \
	-v $(pwd)/criteo_multi_hot:/data/criteo_multi_hot \
	intel/intel-optimized-pytorch:mlperf-inference-4.1-dlrmv2

Inside the container, install some required packages, edit the setup_env.sh file to match the desired configuration, edit run_batch.sh to add desired tests, and run the script to start the benchmarking:

# need to install numba
pip install numba

cd /myApp
vi setup_env.sh
vi run_batch.sh

# run the benchmarking
./run_batch.sh