Skip to content

fix(replication): safely shutdown transfer service - #105

Merged
kkkapu merged 1 commit into
mainfrom
fix4
May 5, 2026
Merged

fix(replication): safely shutdown transfer service#105
kkkapu merged 1 commit into
mainfrom
fix4

Conversation

@kkkapu

@kkkapu kkkapu commented May 5, 2026

Copy link
Copy Markdown
Collaborator

Clear self._transfer_service to avoid invalid C++ calls during shutdown.

- Clear `self._transfer_service` to avoid invalid C++ calls during shutdown.

Change-Id: Ifcd54ff898de560d52374c0e54b38ef4d74e6adf
@github-actions

github-actions Bot commented May 5, 2026

Copy link
Copy Markdown

Python Code Coverage Summary

Code Coverage

Package Line Rate Branch Rate Health
src.ml_flashpoint 100% 100%
src.ml_flashpoint.adapter 100% 100%
src.ml_flashpoint.adapter.megatron 97% 95%
src.ml_flashpoint.adapter.nemo 98% 93%
src.ml_flashpoint.adapter.pytorch 99% 92%
src.ml_flashpoint.checkpoint_object_manager 93% 93%
src.ml_flashpoint.core 95% 92%
src.ml_flashpoint.replication 83% 83%
Summary 95% (2397 / 2524) 92% (574 / 626)

Minimum allowed line rate is 90%

@github-actions

github-actions Bot commented May 5, 2026

Copy link
Copy Markdown

C++ Code Coverage Summary

Code Coverage

Package Line Rate Branch Rate Health
src.ml_flashpoint.checkpoint_object_manager.buffer_object 93% 54%
src.ml_flashpoint.replication.transfer_service 79% 40%
Summary 81% (893 / 1097) 43% (665 / 1539)

Minimum allowed line rate is 80%

@kkkapu
kkkapu merged commit 6e1c211 into main May 5, 2026
7 checks passed
@kkkapu
kkkapu deleted the fix4 branch May 5, 2026 14:20
_LOGGER.info("Shutting down ReplicationManager and TransferService...")
if self._transfer_service:
self._transfer_service.shutdown()
self._transfer_service = None

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

was discussing with @Leahlijuan last week - not sure if this truly fixes the problem, since there will still be ongoing requests, this just wont allow new requests.

We need to deterministically prevent any replication on train end, or wait for it all to finish, to fully fix this I think

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants