resubmission delay does not work in new code

[belforte]

refactored Retry/Resubmission code now allows for introducing additional, flexible, delays in the retry policy executed during automatic resubmissions

CRABServer/src/python/TaskWorker/Actions/RetryJob.py

Lines 23 to 26 in 1038e05

	EXIT_RETRY_POLICY = {
	1: {"type": "recoverable", "delay": 900, "msg": "Job failed to bootstrap CMSSW; likely a worker node issue."},
	50513: {"type": "recoverable", "delay": 900, "msg": "Job did not find functioning CMSSW on worker node."},

CRABServer/src/python/TaskWorker/Actions/PreJob.py

Lines 539 to 549 in 1038e05

	if os.path.exists(retry_info_file):
	try:
	with open(retry_info_file, "r", encoding="utf-8") as fd:
	retry_info = literal_eval(fd.read())
	key = str(self.dag_retry)
	if key in retry_info:
	retry_delay_until = retry_info[key].get("retry_delay_until")
	if retry_delay_until and currentTime < retry_delay_until:
	wait = int(retry_delay_until - currentTime)
	self.logger.info(f"Retry delay not elapsed yet. Deferring for {wait} seconds.")
	return True

but when testing the delay is not applied.

Initial guess is that

CRABServer/src/python/TaskWorker/Actions/PreJob.py

Line 543 in 1038e05

key = str(self.dag_retry)

points to current retry number while PostJob write the delay policy when processign the previous submission.

But an attempt change to

key = str(crab_retry-1)

did not work.

Deeper "digging" is needed.

[belforte]

@aspiringmind-code I will take over as agreed, please write down what you tried and which problems you found.

[aspiringmind-code]

See task for example.
The main problem is that the RRN remains the same in the next retry if the delay happens in the last retry (using inkey and str(crab_retry -1) concept). In the given task, RRN=2 for both prejob.1.1.txt and prejob.1.2.txt in which prejob.1.1 TaskManagerBootstrap ended with exit code 4 due to the delay. When prejob.1.2.txt picks it up again, we get Wed, 15 Jul 2026 10:28:49 CEST(+0200):INFO:PreJob RRN for job 1 reused as 2 (interrupted pre-job, not re-incrementing) and therefore RRN is at 2 in both. We run into prejob.1.3.txt for RRN=3.
Another problem because of this is that the retry in which delay happened (1 in this case) is missing from the all retries monitoring
To summarise, RRN concept needs to be aligned with the delay generated Exit Code 4. RRN originally without delay is reusing the last RRN and not re incrementing it to handle interrupted prejob.