Debugging

You are here: Home > Developer Documentation > Debugging

---

Add Additional Log Information

You can add additional information to your output by adding -DPIC_VERBOSE_LVL=<N> and -DPMACC_VERBOSE_LVL=<M> to your cmake options during compile time (or use ccmake . after $PICSRC/configure [...]).

To activate multiple levels, simply add them.

Example:

# PHYSICS (1) + CRITICAL(4) + SIMULATION_STATE(16)
$PICSRC/configure -c "-DPIC_VERBOSE_LVL=21" ../paramSets/lwfa

PIConGPU Log Levels

From src/picongpu/include/debug/PIConGPUVerbose.hpp:

DEFINE_LOGLVL(0,NOTHING);
DEFINE_LOGLVL(1,PHYSICS);
DEFINE_LOGLVL(2,DOMAINS);
DEFINE_LOGLVL(4,CRITICAL);
DEFINE_LOGLVL(8,MEMORY);
DEFINE_LOGLVL(16,SIMULATION_STATE);
DEFINE_LOGLVL(32,INPUT_OUTPUT);

libPMacc Log Levels

From src/libPMacc/include/debug/PMaccVerbose.hpp:

DEFINE_LOGLVL(0,NOTHING);
DEFINE_LOGLVL(1,MEMORY);
DEFINE_LOGLVL(2,INFO);
DEFINE_LOGLVL(4,CRITICAL);
DEFINE_LOGLVL(8,MPI);
DEFINE_LOGLVL(16,CUDA_RT);
DEFINE_LOGLVL(32,COMMUNICATION);

Add Debug Flags to the Code

The following tools will profit from additional information in your compiled binaries, such as code lines.

Consider activating at least the following cmake flags with ccmake . after the configure step:

• -DCUDA_SHOW_CODELINES=ON
• -DCUDA_BLOCKING_KERNEL=ON
• -DCUDA_NVCC_DEBUG_FLAGS="-g;-G"

On more information on the flags or how to use ccmake, see our documentation on available cmake flags.

A warning on debug flags: using -g/-G usually implies no code optimization or -O0. That might alter your code and can make it hard to track down race conditions.

---

Parallel Debugging

This page collects some useful hints about how to debug a hybrid (CUDA + device) parallel (MPI) application.

MPI + Valgrind

Use the OpenMPI supressions list

mpiexec <mpi flags> valgrind --suppressions=$MPI_ROOT/share/openmpi/openmpi-valgrind.supp picongpu ...

MPI + GDB

Multi-Node Host-Side

Login into an interactive shell/batch session with X-forwarding ssh -X. Launch PIConGPU with gdb and trigger start and back trace automatically:

mpiexec <mpi flags> xterm -e gdb -ex r -ex tb --args picongpu ...

MPI + CUDA-MEMCHECK

mpiexec <mpi flags> cuda-memcheck --tool <memcheck|racecheck> picongpu ...

CUDA-GDB

Single-Node device-side

Manual.

(!) Compile with nvcc -g -G <...> if you want to set device-side breakpoints.

cd <path>/simOutput
cuda-gdb --args <path2picongpu> -d 1 1 1 -g <...> -s 100 <...> 

in cuda-gdb:

b <FileName>:<LineNumber>
r