Testing wavepeek on CVDP

[esynr3z]

I have wanted to study real wavepeek agent traces for a while. I get some from users, but not enough to see clear patterns. So I decided to generate a larger set from actual RTL debugging tasks and watch how the models used the tool.

I picked 18 tasks from CVDP and ran each in two modes. In the baseline, the agent had no access to wavepeek. In the other mode, the same agent had to use wavepeek before giving its final answer.

The experiment covered three GPT-5.6 profiles with three attempts per configuration, for 324 trials in total. Half of those trials used wavepeek, and the agents did invoke it in all 162 of them.

What happened

Forced wavepeek use did not produce a convincing improvement in quality on these tasks.

The pass rate went from 80.9% without wavepeek to 82.7% with it. That difference was not statistically conclusive, and the direction of the result varied between models. Costs went up much more clearly. The agents took more turns, made more tool calls, used more context and tokens, and finished a typical task about 80 seconds later.

In the end it looks like forcing an agent to use it on every RTL task is a bad policy.

CVDP was the wrong fit

Many of the CVDP prompts had already done most of the debugging. They named the problematic signals, described incorrect expressions, or even listed the required changes. The agent could usually work out a patch from the prompt, RTL, and tests. It did not need the waveform to discover anything new.

That changed the role of wavepeek. Instead of using it to find the root cause, agents often edited the RTL first and inspected the waveform afterward because the experiment required them to. wavepeek became a final check on a patch they had already decided to make.

The coverage numbers tell the same story. The baseline solved all 18 tasks at least once. The forced wavepeek branch solved 17 out of 18.

The binary was fast (of course), so its runtime was not the problem. The time went into finding the waveform, reading help, choosing signals, writing queries, processing the output, and then carrying all of that extra text through the remaining turns. About one third of wavepeek invocations were help or discovery calls.

What I got from the experiment

I started with two questions:

1. How do agents use wavepeek in practice?
2. Do they solve more tasks when they have it?

The first one got a useful answer. The experiment produced a large set of real agent traces and exposed specific friction in both the CLI and the instructions around it.

The second question needs a different eval. CVDP judges the final RTL patch: either it passes the verifier or it does not. That is useful, but it mostly measures whether the model can write working RTL. I care more about debugging: can the model inspect the available evidence, identify the root cause, and explain where the bug is?

What I want to test next

The first category is simple waveform QA. Give the agent a large waveform and ask a concrete question.

Then compare answer quality, agent steps, and token use with and without wavepeek.

The second category is closer to a real debugging incident. The agent gets RTL, a log, a description of the observed failure, and a large pre-recorded waveform.

That constraint matters. It cannot rerun the simulation after every $display added. In the real world such rerun can take hours or days. The report may contain only the logs and waves captured when the failure happened. The agent has to get as much as it can from those artifacts and locate the cause, ideally down to the relevant RTL file and lines.

In these tasks, the waveform should contain information that the agent needs for a reliable diagnosis. That is the setting where I expect wavepeek to be useful: helping the model find the answer more reliably, rather than creating some custom ad hoc waveform parser.

More to come.

---

Detailed results and data:

• final comparative report
• analysis of how agents used wavepeek in practice