UCP/CUDA: Fix Valgrind failures with CUDA managed memory#11499
Conversation
| { | ||
| return (event->comp != NULL) && | ||
| (event->comp->func == uct_cuda_base_stream_flushed_cb); | ||
| return event->event == NULL; |
There was a problem hiding this comment.
Why is the previous check not working here?
There was a problem hiding this comment.
It dereferences comp->func, which can point into a UCP request. Valgrind may mark that request memory undefined/noaccess.
|
|
||
| status = ucs_memtype_cache_lookup(address, length, mem_info); | ||
| if (ucs_likely(status == UCS_ERR_NO_ELEM)) { | ||
| if (ucs_unlikely(RUNNING_ON_VALGRIND)) { |
There was a problem hiding this comment.
why needed with valgrind? is cuda/managed memory not present in memtype cache?
| ucs_queue_push(&iface->super.active_queue, &q_desc->queue); | ||
| } | ||
|
|
||
| VALGRIND_MAKE_MEM_DEFINED(&cuda_event->event, sizeof(cuda_event->event)); |
There was a problem hiding this comment.
i guess we need to do it after ucs_mpool_get, because the reason the event is valid is because it's cuda_event->event is coming from the mpool, and it should be probably before used in line 333
| ucs_queue_push(&iface->super.active_queue, &q_desc->queue); | ||
| } | ||
|
|
||
| VALGRIND_MAKE_MEM_DEFINED(&cuda_ipc_event->super.event, |
There was a problem hiding this comment.
same here -- maybe need inline wrapper to get object (derived from) uct_cuda_event_desc_t from a memory pool and make ->event defined?
3 participants
What?
Fix Valgrind failures with CUDA managed memory
Why?
Internal issue