The Practical Verification Debug Guide
| |

The Practical Verification Debug Guide: A Repeatable Method (+ AI-Assisted Debug)

DEBUG | UVM | SYSTEMVERILOG | AI-ASSISTED VERIFICATION
Updated November 2026 · By The Art of Verification

Quick answer: A repeatable verification debug method is a fixed order of five questions. 1) Was the stimulus legal? 2) Did it reach the DUT pins? 3) Did the design respond correctly? 4) Did the checker judge it correctly? 5) Do the log and waveform agree? Stop at the first “no”. That door holds your root cause. AI tools can now sort logs and suggest the door, but an engineer still confirms the cycle in the waveform before any fix.

Five-door debug method flowchart for verification engineers: legality, delivery, design, checking, consistency
The five-door debug method. Walk left to right and stop at the first “no”.

Key takeaways

  • Debug is the biggest single slice of a verification engineer’s week. Method beats talent.
  • Walk five doors in order: legality, delivery, design, checking, consistency.
  • Read the first UVM_ERROR, not the last. Later errors are usually echoes.
  • In the waveform, start at the symptom and walk backward.
  • Group regression failures by signature before you debug any single test.
  • AI is great at reading and sorting. You still own the verdict.

Why does debug skill decide your schedule?

Ask where a chip schedule really goes. The answer is rarely “writing tests”. It is figuring out why tests failed.

  • The Wilson Research Group functional verification study found verification engineers spend more time in debug than in any other activity, 44% of their time in the 2018 edition, up from 39% in 2016 (Siemens Verification Horizons).
  • The 2024 edition reported only 14% of projects hit first silicon success, the lowest in two decades (Verification Academy).

Put those two numbers together. Nearly half your week goes to debug, and most chips still need another spin. So the way you debug is not a side skill. It is the job.

Two engineers with the same tools can work at very different speeds:

  • The slow one opens the waveform first, drowns in thousands of signals, and loses an afternoon per failure.
  • The fast one narrows the problem to one layer before opening any viewer, and finds root cause in minutes.

There is a second cost too. A regression full of false failures trains the team to ignore red. Then a real bug walks through in the noise. Good debug keeps the regression honest.

What is a good analogy for debug in verification?

Pick whichever one sticks. They all say the same thing: check in order, from cheap to expensive.

1. The missing letter

Your letter never reached your friend. You do not blame the post office first. You check:

  • Did I write the right address? (door 1, legal stimulus)
  • Did I actually drop it in the box? (door 2, delivery)
  • Did the post office route it right? (door 3, design)
  • Did my friend check the right mailbox? (door 4, checker)
  • Am I sure I am remembering the right day? (door 5, consistency)

Blaming the design first is like blaming the post office before checking the address.

2. The doctor who checks vitals before the MRI

A good doctor takes temperature and blood pressure before ordering an MRI. The waveform is your MRI. It shows everything, it is slow, and it is useless if you do not know where to look. Logs and assertions are your vitals. Read them first.

And when the thermometer says 45°C but the patient looks fine, you check the thermometer. That is a monitor bug.

3. The match replay

When a volleyball team reviews a lost point, nobody rewinds to the warm-up. You start at the point that was lost and go back one touch at a time: bad set, late pass, wrong rotation. That is exactly how to read a waveform. Start at the mismatch. Walk back one cause at a time.

What is the five-door debug method?

Every failure lives behind one of five doors. Walk them in order and close each one until only the guilty one is open.

DoorQuestionWhere to lookTypical guilty party
1. LegalityWas the stimulus legal?Solved item print, constraints, sequenceMissing or wrong constraint
2. DeliveryDid it reach the DUT pins?Driver log, interface signalsVirtual interface or config_db
3. DesignDid the DUT respond correctly?First failing assertion, RTL waveformRTL bug
4. CheckingDid the checker judge correctly?Scoreboard, reference model, monitorTestbench bug
5. ConsistencyDo log and waveform agree?Seed, timestamps, tool versionsStale dump or wrong seed

Why the order matters: each door you clear removes a whole layer from the suspect list. Skip ahead to the RTL waveform and every layer stays a suspect. That is why unstructured debug feels endless.

The idea underneath: a failure has one first cause and many downstream effects. Logs and waves show effects everywhere. Your only job is to find the earliest cycle where correct became incorrect.

If all five doors say yes, your test expectation is wrong. Go back to the spec, not the RTL.

What comes before and after the five doors?

The five doors find the root cause. Two steps around them make the whole job repeatable: one before you start, one after you finish.

Verification debug loop: reproduce and shrink, walk the five doors in order, stop at the first no, find the root cause, then fix, lock it in and report
The full loop. The error IDs on the right are from the planted-bug runs further down this page.

Step 0: reproduce and shrink

  • Reproduce with the saved seed and the exact command line. A failure you cannot repeat can only be guessed at.
  • Shrink it: fewest transactions, shortest run, one agent if you can. A 2,000-cycle failure is a different job from a 2-million-cycle one.
  • Check the run got your settings. Print every plusarg and define the test reads in one banner line at time 0 (more in the mistakes below).
  • Analogy: a doctor asks you to show when it hurts before ordering any scan.

Think in hypotheses

  • Before you open a tool, write one line: “I think it is X, and if so I will see Y at time T.”
  • If Y is not there, X is wrong. Cross it off and write the next one.
  • This turns hours of scrolling into a short list of yes or no checks.

Step 6: fix, lock it in, report

  • Write the bug report so someone else can act on it: seed, command, first error line, the one signal and time that prove it, and which door.
  • Lock it in with an assertion, a check or a test that fails if the bug comes back.
  • Ask why the bench missed it earlier. A missing check is a second bug.

How do you build a UVM testbench that explains its own failures?

Debug speed is designed in, not bolted on. Your component boundaries are your debug boundaries.

UVM testbench debug map showing sequence, driver, virtual interface, DUT, monitor and scoreboard mapped to debug doors
Every UVM component boundary maps to a debug door.

What each component should do for you:

  • Sequence: print the solved item and reject illegal combinations. This moves most door 1 problems out of debug entirely.
  • Driver: log every item it drives, with the cycle. Driver logged it but pins stayed flat? Door 2, the interface is not connected.
  • Monitor: passive and complete. Rebuild every transaction and timestamp it. A monitor that misses one creates a fake mismatch.
  • Scoreboard: on a mismatch, print expected, actual, and the source transaction. Snapshot shared model state before you compare.
  • Everyone: use consistent message IDs so a simple grep finds the story.

New to the plumbing? Start with the typical UVM testbench architecture, the build a UVM environment series, sequencer and driver communication, and passing a virtual interface with uvm_config_db.

What SystemVerilog code makes debug faster?

Honesty note: the code below follows the SystemVerilog and UVM standards and common practice. It was not captured from a specific simulator run. Compile it in your own flow before relying on it.

Door 1: print what was really sent

Print the solved transaction, not the one you think you sent. Bad, good, best:

// BAD: you print your intent, not the solved values.
// An illegal randomized value stays invisible.
`uvm_info("SEQ", "sending a 4-beat burst to 0x1000", UVM_LOW)
`uvm_do(item)

// GOOD: print the solved item after randomization.
function void post_randomize();
  `uvm_info("SEQ", $sformatf("addr=%0h len=%0d size=%0d burst=%0d",
            addr, len, size, burst), UVM_LOW)
endfunction

// BEST: print it AND self-check legality, so an illegal
// stimulus fails loudly here, not later as a fake design bug.
function void post_randomize();
  `uvm_info("SEQ", $sformatf("addr=%0h len=%0d size=%0d burst=%0d",
            addr, len, size, burst), UVM_LOW)
  if (addr[11:0] + (len + 1) * (1 << size) > 13'h1000)
    `uvm_error("SEQ_ILLEGAL", "AXI INCR burst crosses a 4KB boundary")
endfunction

If randomize() itself returns 0, nothing legal gets sent at all. That is a different problem with its own clues in the log: see why SystemVerilog randomization fails for the six causes and a 5-minute triage.

  • Why it exists: an illegal burst makes the design misbehave correctly. Without this print you chase a ghost.
  • Next step: move the check into a shared protocol checker so every sequence gets it for free. The rules are in the AXI protocol verification guide.

Door 3: let an assertion mark the crime scene

// Fails at the exact cycle ack is late.
property p_req_then_ack;
  @(posedge clk) disable iff (!rst_n)
  $rose(req) |-> ##[1:5] ack;
endproperty
a_req_ack: assert property (p_req_then_ack)
  else `uvm_error("PROT_REQ_ACK", "ack did not follow req within 5 cycles")

// Prove the precondition actually happens.
c_req: cover property (@(posedge clk) disable iff (!rst_n) req);

Why $rose(req) and not plain req: while req stays high, a plain req |-> starts a new attempt every cycle, so one long request can raise errors that have nothing to do with the design. Trigger on the start of the request. This exact property is run on VCS in the planted-bug test below.

  • An assertion turns a long backward trace into one log line with a timestamp.
  • The cover matters. An assertion whose precondition never fires proves nothing. Formal tools hit the same vacuity trap; see formal vs simulation-based verification.
  • Bind assertions to the RTL module so they travel with the design into every test.

Door 4: a scoreboard a reviewer can trust

// Snapshot first, compare second, print everything on a miss.
function void check_item(axi_item act);
  axi_item exp = model.predict_snapshot(act);   // copy, not a live handle
  if (!act.compare(exp))
    `uvm_error("SB_MISMATCH", $sformatf("\nEXP: %s\nACT: %s\nSRC: %s",
               exp.convert2string(), act.convert2string(), act.src_txn_id))
endfunction

For a worked example where counters, not error lines, separate a design bug from a bench bug, see async FIFO verification part 1.

Does the method work? Three planted bugs, real VCS output

To show what each door looks like in a real log, here is one small UVM bench with a bug planted behind doors 1, 3 and 4. It is a 16-word memory with a req/ack handshake, a driver, a monitor, a scoreboard, and the checks from the section above. The plusarg +BUG=N picks the bug. All runs: Synopsys VCS X-2025.06-SP1, UVM 1.2.

RunPlanted bugDoorCaught byScoreboard
+BUG=0nonenonenothing to catchitems=16 errors=0
+BUG=14KB constraint switched off1, legality6 SEQ_ILLEGAL errors from post_randomize()0 errors
+BUG=3DUT acks after 6 cycles, spec says 53, design16 PROT_REQ_ACK assertion errors0 errors
+BUG=4monitor reuses one object4, checking16 SB_MISMATCH errors16 errors, design correct

The three planted bugs, as they appear in the code:

// +BUG=1 (sequence): the legality constraint is switched off
if (bug == 1) req.c_4k.constraint_mode(0);

// +BUG=3 (DUT): ack comes after 6 cycles instead of 2
initial lat = (planted_bug() == 3) ? 6 : 2;

// +BUG=4 (monitor): one object is reused for every transfer
if (!reuse || t == null) t = bus_item::type_id::create("t");

+BUG=1: door 1, illegal stimulus

UVM_INFO testbench.sv(181) @ 0: uvm_test_top [TEST] planted bug: 1
UVM_ERROR testbench.sv(65) @ 95: uvm_test_top.env.sqr@@s.req [SEQ_ILLEGAL] burst at ffc, 4 beats of 2 bytes crosses a 4KB boundary
UVM_ERROR testbench.sv(65) @ 175: uvm_test_top.env.sqr@@s.req [SEQ_ILLEGAL] burst at fd8, 13 beats of 4 bytes crosses a 4KB boundary
...
UVM_INFO testbench.sv(152) @ 655: uvm_test_top.env.sb [SB] items=16 errors=0
UVM_ERROR :    6

Six of 16 bursts were illegal, and the scoreboard said nothing, because this small DUT does not care about 4KB. A real AXI slave would, and the failure would look like a design bug. Only the check in post_randomize() names the true cause.

+BUG=3: door 3, a late design

UVM_INFO testbench.sv(181) @ 0: uvm_test_top [TEST] planted bug: 3
"testbench.sv", 44: tb.bif.a_req_ack: started at 35ns failed at 85ns
	Offending 'ack'
UVM_ERROR testbench.sv(45) @ 85: reporter [PROT_REQ_ACK] ack did not follow req within 5 cycles
...
UVM_INFO testbench.sv(152) @ 1295: uvm_test_top.env.sb [SB] items=16 errors=0
UVM_ERROR :   16

The assertion fired on every transfer, at the exact cycle the window closed. The scoreboard still passed: late data is still correct data. The only other hint was that the run took 1295 ns instead of 655.

+BUG=4: door 4, the design is innocent

UVM_INFO testbench.sv(181) @ 0: uvm_test_top [TEST] planted bug: 4
UVM_ERROR testbench.sv(148) @ 655: uvm_test_top.env.sb [SB_MISMATCH] item 0: RD word=0 data=1d57d3d8, expected xxxxxxxx
UVM_ERROR testbench.sv(148) @ 655: uvm_test_top.env.sb [SB_MISMATCH] item 1: RD word=0 data=1d57d3d8, expected xxxxxxxx
...
UVM_ERROR testbench.sv(148) @ 655: uvm_test_top.env.sb [SB_MISMATCH] item 15: RD word=0 data=1d57d3d8, expected xxxxxxxx
UVM_INFO testbench.sv(152) @ 655: uvm_test_top.env.sb [SB] items=16 errors=16

Sixteen errors, and the DUT is correct. The clue is in the errors themselves: all 16 print the same item. Sixteen different transfers cannot look identical, so the checker is holding one object sixteen times. The fix is one line: create a new object per transfer in the monitor, or clone before you store. More on this trap in UVM TLM explained.

What the three runs teach

  • The message ID points to the door. SEQ_ILLEGAL, PROT_REQ_ACK and SB_MISMATCH each sent us to a different place before we opened a waveform.
  • A scoreboard is one door, not the whole check. Two of the three bugs passed it.
  • When every error prints the same thing, suspect the checker before the design.

We later reused this bench as the baseline against an AI-written testbench on a design with four hidden bugs. It caught 2; the AI bench caught 4, and the reason was stimulus, not checking. See the experiment, logs and files.

Stuck on a scoreboard mismatch? See the 6 causes and a 5-minute triage, with five planted bugs and their VCS logs.

How do you read a waveform without getting lost?

Waveform debug example tracing a FIFO full flag bug backward from the scoreboard mismatch to the first bad cycle
Start at the mismatch and walk back to the first bad cycle.
  1. Jump to the failing time from the first error. Never scroll forward from time zero.
  2. Add only the signals of the failing interface plus the first failing assertion.
  3. Walk back one cause at a time. Ask “what made this value wrong?” and move one hop upstream.
  4. Drop a marker at each hop so the next engineer can follow your trail.
  5. Name the signal and the exact cycle in your bug report. “It looks wrong around the middle” helps nobody.

Door 2 quick read:

What you seeWhat it meansNext step
Driver logs a send, pins stay flatVirtual interface not connectedCheck uvm_config_db set and get paths
Pins toggle, no readyDUT not acceptingGo to door 3
Handshake completesDelivery is fineGo to door 3

How do you read UVM reports fast?

The report summary tells you the shape of the failure before you open anything. Read top down. The first UVM_ERROR is usually the cause.

Report clueLikely cause
Many errors at the same timestampOne root event, checks cascading
Error, long silence, then timeoutDesign hung after the first error
Only a UVM_FATAL about configEnvironment build problem, not a design bug
Objection drop warningTest ended before stimulus finished
Failure vanishes with full dump onRace or uninitialized value, timing shifted

Run the regression at low verbosity for speed. Rerun only the failing seed at high verbosity for detail.

UVM command-line switches worth knowing

SwitchShowsUse it when
+UVM_VERBOSITY=UVM_HIGHMore uvm_info messagesYou need door 1 prints, such as the solved item
+UVM_OBJECTION_TRACEEvery raise and dropThe test ends at 0 ns, or never ends
+UVM_CONFIG_DB_TRACEEvery set and getA get fails or returns the wrong value
+UVM_PHASE_TRACEPhase start and endThe run is stuck between phases
+UVM_MAX_QUIT_COUNT=1Stops at the first errorYou want the first error to be the last line
uvm_top.print_topology()The component treeA component is missing or extra
factory.print()Overrides in forceAn override did not take

Verbosity levels and IDs in detail: UVM verbosity and reporting.

What are the most common RTL and testbench bugs?

RTL bugs, by symptom

SymptomCommon RTL causeHow to confirm
Data right, one cycle lateMissing or extra pipeline stageCount register stages in the wave
Fails on some seeds onlyUninitialized register or raceLook for X right after reset
Fails after back-to-back requestsFIFO full or empty off by oneWatch occupancy at the boundary
Hang under heavy loadArbitration deadlockCheck grants when all agents request
Wrong only across clock domainsMissing synchronizerSee the CDC verification guide
Passes RTL, fails gatesX-propagation or timingSee the gate-level simulation guide

Testbench bugs that look like design bugs

  • Virtual interface never set, so the driver drives nothing.
  • Scoreboard compares before the design has produced the result.
  • Objection dropped too early. The test ends mid-traffic and looks like a hang.
  • Reference model misses a legal reordering and flags correct behavior.
  • Coverage sampled on the wrong edge, so a real hit looks like a hole.
  • Factory override forgotten. The old component runs and your fix “does nothing”.

Rule of thumb: design bugs tend to be narrow. Testbench bugs tend to be broad. When unrelated tests fail the same odd way, suspect shared testbench code first. Treating your own testbench as a suspect is a sign of a senior engineer. More on that mindset in the thought process behind great verification.

Coverage symptoms that send you the wrong way

Coverage symptomWhat it usually means
Hole in a bin you know you droveSample edge or condition is wrong
Hole in a truly rare binStimulus never got there. Likely bug hiding place.
Sudden drop after a mergeA covergroup got disconnected

Background: types of coverage metrics, functional coverage guidelines, and the coverage closure guide.

How do you triage a big regression in one morning?

Picture the morning after a nightly run on a subsystem with a CPU, a DMA engine and a memory controller. Two hundred red tests. Coffee is still hot.

Regression triage funnel collapsing 200 failures into 3 error signatures and 2 root causes
200 symptoms, 3 signatures, 2 root causes.
  1. Bucket first. Group by first UVM_ERROR text and first failing assertion, with numbers and timestamps stripped out.
  2. Read the buckets, not the tests. 40 share an objection warning from a shared base test. 90 hit the same AXI assertion. 70 are scattered.
  3. Debug one seed per bucket. The shortest, lowest-seed failure in each group.
  4. Check what changed. Diff RTL and testbench files since the last green run.
  5. Re-bucket after fixes land. Most scattered failures are cascades and vanish on their own.

Keep debug cheap while you do it:

SettingRuntime costWhen to use
Full dump, whole runSeveral times slowerRare, hard bugs only
Windowed dump around failureSmallDefault for reruns
High verbosity, whole regressionHuge logs, slowerAvoid. Rerun one seed instead.
Assertion-only runFastFirst-pass protocol triage

Always save the seed and command line for every job. A failure you cannot reproduce is a rumor, not a bug.

How can AI help with verification debug?

This part changed a lot in 2026. All three big EDA vendors now ship or preview AI agents aimed at debug:

  • Siemens announced the Questa One Agentic Toolkit in February 2026, including a Debug Agent that correlates waveforms, assertions, coverage and logs to point at suspicious transitions and suggest failure mechanisms for an engineer to review (Siemens).
  • Synopsys unveiled a Verification AgentEngineer in March 2026 that reads logs, groups errors, and checks the likely cause against waveforms. Synopsys also reported a 25% to 40% cut in debug cycle time in an early evaluation (Tom’s Hardware).
  • Cadence opened early access to its ChipStack agents in February 2026, aimed at front-end design and verification loops.

Two honest caveats. These numbers are vendor-reported and measured against different baselines. And every one of these tools keeps a human approving the final root cause. That tells you where the real line sits.

Here is the way I think about it: AI is a very fast junior engineer who has read every log but never seen the spec. Brilliant at reading and sorting. Not yet trusted to sign off. So give it the reading work, and keep the judgment.

AI-assisted debug flow for chip verification with a human verification gate before any fix
AI-assisted debug with a human gate before every fix.

For the wider picture beyond debug (test selection, coverage steering, drafting, and the guardrails that keep each honest), see AI in ASIC verification: a practical method, with limits.

The AI-assisted debug flow, step by step

  1. Make your logs machine-readable. One line per transaction, stable message IDs, seed and test name in the header. Good logs help humans and AI equally.
  2. Cluster failures automatically. Normalize the first error line and bucket. You do not even need an LLM for this step (script below).
  3. Ask the AI for a door, not a fix. Feed it the first error, about 200 lines of context, and the diff since the last green run. Ask which of the five doors is most likely guilty and why.
  4. Human gate. You open the waveform at the cycle it named and confirm or reject. No fix goes in without this.
  5. Fix and lock. Patch, add an assertion or check, rerun the exact seed, then the bucket.
  6. Feed it back. Store the signature, door, and root cause. That history makes the next suggestion better.

Step 2 script: bucket failures by first error

# bucket_failures.py : group regression failures by normalized first error
import re, sys, collections, pathlib

ERR = re.compile(r'^(UVM_ERROR|UVM_FATAL)\s.*@')
NORM = [(r'@\s*[\d.]+\s*[munpf]?s?', '@T'),        # timestamps
        (r"(0x|'h)[0-9a-fA-F_]+", 'HEX'),           # hex with a prefix
        (r'\b[0-9a-fA-F]{8,}\b', 'HEX'),             # bare %h words such as deadbeef
        (r'\b(?=[0-9a-fA-F]*\d)[0-9a-fA-F]{4,}\b', 'HEX'),  # shorter hex with a digit
        (r'\b\d+\b', 'N')]                           # decimal numbers

def signature(text):
    for line in text.splitlines():
        if ERR.match(line):
            for pat, rep in NORM:
                line = re.sub(pat, rep, line)
            return line.strip()[:160]
    return 'NO_UVM_ERROR (check timeout or crash)'

buckets = collections.defaultdict(list)
for log in pathlib.Path(sys.argv[1]).glob('*/sim.log'):
    buckets[signature(log.read_text(errors='ignore'))].append(log.parent.name)

for sig, tests in sorted(buckets.items(), key=lambda kv: -len(kv[1])):
    print(f'{len(tests):4d}  {sig}\n      e.g. {tests[0]}')

Tested on 7 sample logs. UVM prints hex with %h, with no 0x, so the two bare-hex patterns above matter: without them, one scoreboard failure printed as data=1a2b3c4d and data=deadbeef split into three buckets. With them, 7 failures group into 4 causes:

   3  UVM_ERROR testbench.sv(N) @T: uvm_test_top.env.sb [SB_MISMATCH] item N: RD word=N data=HEX, expected HEX
   2  UVM_ERROR testbench.sv(N) @T: reporter [PROT_REQ_ACK] ack did not follow req within N cycles
   1  NO_UVM_ERROR (check timeout or crash)
   1  UVM_FATAL testbench.sv(N) @T: uvm_test_top.env [NOVIF] no vif

Step 3 prompt: keep the AI inside the five doors

You are helping debug a UVM simulation failure.
Use the five-door order: 1 legality, 2 delivery, 3 design,
4 checker, 5 consistency.

Inputs: first error + 200 lines of log, solved item print,
git diff since last passing run.

Answer with:
- Most likely door and a confidence (low / medium / high)
- The exact log line numbers that support it
- One signal and one sim time I should check in the waveform
- What evidence would prove you wrong
Do not propose an RTL fix.

Why “what would prove you wrong”? Because AI tools sound just as sure when they are guessing. Asking for line numbers and a falsifying check turns a confident paragraph into something you can verify in two minutes.

Where AI saves time, door by door

AI is fastest where the work is reading and comparing many lines. It is slowest, and least trusted, where the work is knowing what the spec means.

DoorWhat AI does wellWhat you still checkAnalogy
1. LegalityScans thousands of solved-item prints and flags values outside the ranges you give itThe spec rule itselfA proofreader who catches every misspelt word but did not write the dictionary
2. DeliveryCompares the driver log with the monitor log and lists items sent but never seen; spots a missing setting in the banner lineThe pins in the waveformA parcel tracker that shows where the box stopped
3. DesignLines up the first assertion failure with nearby signal changes and names a time to look atThe cycle in the waveform, and what the spec really saysA smoke alarm: it tells you which room, you find the fire
4. CheckingNotices patterns across many errors: the same item every time, the same expected value, errors only at the end of the testThe monitor and scoreboard codeNoticing that every complaint letter is in the same handwriting
5. ConsistencyMatches log times to waveform timesThe waveform itselfA second pair of eyes on the timeline
RegressionGroups hundreds of failures by first errorOne seed per groupSorting the post before anyone opens a letter

Worked example: the +BUG=4 log

Here is the prompt above applied to the real +BUG=4 run from this page. The inputs are the first error with the log around it, the solved-item print, and the change since the last passing run (the monitor’s object handling).

A good answer looks like this. We wrote it as a model answer to show the shape to expect; it is not the output of an AI tool:

Door: 4 (checking). Confidence: high.
Evidence: lines 4 to 19 of the log are 16 SB_MISMATCH errors, and every one
prints the same item: "RD word=0 data=1d57d3d8". Sixteen different transfers
cannot be identical, so the scoreboard is likely holding one object 16 times.
All 16 errors come at 655 ns, in check_phase, after traffic stopped.
Check in the waveform: rdata at any two read acks. If it differs between
transfers, the DUT is returning different data and the checker is wrong.
What would prove me wrong: rdata is 1d57d3d8 on every read ack. Then the
design is at fault, not the checker.

Why this is quick for a tool and slow for a person: each of the 16 lines on its own looks like a design bug. The clue is only in the pattern across them, and comparing many lines is exactly the reading work a tool does in seconds. You still spend two minutes in the waveform to confirm it before touching any code.

How to measure the gain on your own team

Independent numbers on AI debug speed are thin. The figures above are vendor-reported, so measure your own. Three numbers are enough:

  • Time to first root cause. For each failure, from first look to the door you confirm. Compare the median before and after, over a few weeks.
  • Door hit rate. How often the first door the AI named was the right one. If it is wrong more often than right, change what you feed it before you blame the tool.
  • Failures per group. Failures in a regression divided by the groups they fall into. On the 7 sample logs on this page, the fixed script made 4 groups: 7 failures, 4 causes to read.

Write the three numbers down every week. If they do not move after a month, the tool is adding steps, not removing them.

Guardrails that matter

  • Never paste RTL, specs or logs into a public chatbot. Use your company’s approved, on-prem or enterprise setup.
  • Demand evidence. No line numbers, no trust.
  • Watch for the “clean story” trap. A tidy explanation that does not match the waveform is still wrong.
  • Measure it. Track time to first root cause and how often the suggested door was right. If it is not saving time after a month, change the inputs.

Typical debug patterns at IP, subsystem and SoC level

Three patterns that come up again and again. They are composites, not one team’s story, and one of them is the +BUG=4 run above.

IP level: a flag one cycle late

  • Symptom: a mismatch only when reads and writes land in the same cycle near full.
  • Door 3: the full flag rises one cycle late, so one write lands in a full FIFO.
  • Usual root cause: full decoded from the current count, not the next count.
  • Lock it in: an assertion that a write is never accepted when full.

Subsystem level: the design is innocent

  • Symptom: scoreboard errors on one seed in a few hundred, and the waveform looks fine.
  • Door 4: the checker compares a handle that another thread has already changed.
  • Lesson: clear the checker before a long RTL hunt. The +BUG=4 run shows exactly this.

SoC level: the hang that is not a hang

  • Symptom: a test never ends after a merge, with a long tail of errors.
  • Move: read the first error, then run with +UVM_OBJECTION_TRACE.
  • Usual root cause: a sequence drops its objection early, or a component raises one per packet and never drops the last. One forum user found 34 objections still raised by a scoreboard.

More SoC context in key areas of SoC verification and the SoC verification flow.

Common debug mistakes, from real forum threads

Retold in our own words, each with more than one fix.

  • Treating objections as an end-of-test switch only. A user asked how objections help debug at all (thread).
    • Fix A: know the two failure shapes. Nobody raises, so the test ends at 0 ns; or someone never drops, so it hangs.
    • Fix B: run with +UVM_OBJECTION_TRACE and read who raised last.
    • Fix C: raise and drop in the test, not in drivers, monitors or scoreboards.
  • A regression that never ends in CI (thread).
    • Fix A: a per-test timeout (set_timeout or +UVM_TIMEOUT) so CI always gets a log.
    • Fix B: a heartbeat or progress print, so the log shows where it stopped.
    • Fix C: then find the real cause. Replies in that thread stress that a timeout only hides it.
  • Objections raised per packet in the scoreboard. A test never ended with 34 objections still raised (thread).
    • Fix A: let the test own the objection and use drain_time for data still in flight.
    • Fix B: check scoreboard queues in check_phase instead of holding the phase open.
  • Stored transactions that change after you store them (thread).
    • Fix A: create a new object per transaction in the monitor.
    • Fix B: clone in the subscriber before you store it.
    • See it happen: the +BUG=4 run above.
  • Failures that change with the simulator or with dumping. Two blocks writing and reading the same variable at the same time give different results on different tools (thread).
    • Fix A: drive and sample through clocking blocks.
    • Fix B: keep testbench code in step with one clock edge (thread).
  • Debugging a run that never got your settings. While testing this page, three runs meant to plant bugs all ran clean: +BUG=N had gone into compile options, not run options.
    • Fix A: print every plusarg and define the test reads in one banner line at time 0, and read that line first.
    • Fix B: give each setting a compile-time fallback (+define+BUG=N) so either box works.
    • Fix C: when a “buggy” run passes, suspect the run settings (door 2) before you celebrate.

Debug interview questions for verification engineers

  1. A test fails. What do you check first?
    Whether the stimulus was legal, by printing the solved transaction.
  2. Design bug or testbench bug, how do you decide?
    Design bugs are narrow. Broad failures across unrelated tests point at shared testbench code.
  3. Thousands of signals. Where do you start?
    At the failing time, walking backward from the first failing assertion.
  4. Why read the first UVM error and not the last?
    The first is usually the root. Later ones are cascades.
  5. Lead-level: 100 tests fail the same way after a merge.
    Treat it as one cause. Read one representative’s first error and diff the changed files.
  6. How would you use AI in debug safely?
    For clustering and log reading on approved tools, with evidence required and a human confirming in the waveform.

FAQ

What is the first step in debugging a failing UVM test?

Reproduce it with the saved seed and command line, then print the solved transaction to check the stimulus was legal.

Should I always open the waveform first?

No. The waveform is door 3. Clear legality and delivery first using logs.

Why does my failure disappear when I turn on waveform dumping?

Dumping can shift timing. That is a strong hint of a race or an uninitialized value.

How do I tell a monitor bug from a design bug?

If the DUT pins show the correct value but the scoreboard disagrees, suspect the monitor or reference model.

How do I debug a simulation hang?

Find the last activity before silence, check for a dropped objection, and look at arbitration grants when every agent is requesting.

Can AI find root cause in chip verification on its own?

Not reliably yet. 2026 agents from Siemens, Synopsys and Cadence speed up log reading, clustering and waveform correlation, but all keep an engineer approving the final root cause.

What is the fastest way to triage a big regression?

Bucket failures by normalized first error, debug one seed per bucket, and diff changes since the last green run.

What is the five-door debug method?

Five questions in a fixed order: was the stimulus legal, did it reach the DUT, did the design respond correctly, did the checker judge correctly, and do the log and waveform agree. Stop at the first “no”; the root cause is behind that door.

How do I debug a UVM test that never ends?

Run with +UVM_OBJECTION_TRACE, find the component that raised last and never dropped, and keep objections in the test only. Add a timeout so the run always produces a log.

How do I group hundreds of regression failures?

Take the first error of each log, replace times, hex values and numbers with placeholders, and count identical signatures. Then debug one seed per group.

Printable debug checklists

Debug checklist (every failure)

  • Reproduce with the recorded seed and command line.
  • Door 1: print and judge the solved stimulus.
  • Door 2: driver log vs interface pins.
  • Door 3: first failing assertion, then trace backward.
  • Door 4: expected vs actual, decide which to trust.
  • Door 5: seed and timestamps match the log.

Testbench checklist

  • Sequences print solved items and reject illegal ones.
  • Driver and monitor log each transaction with a timestamp.
  • Scoreboard snapshots shared state before compare.
  • Message IDs are consistent and grep-friendly.
  • Assertion messages name the rule that broke.

RTL checklist

  • Counters and FIFO flags checked at boundaries.
  • Every register has a reset value. No X after reset.
  • Every clock crossing has a reviewed synchronizer.
  • Internal invariants carry designer assertions.

Regression and AI checklist

  • Seed and command line saved per job.
  • Windowed dump by default.
  • Failures bucketed by signature before triage.
  • AI suggestions come with log line numbers.
  • No fix without a human waveform check.

Signoff checklist

  • Every root cause added an assertion or check.
  • No open failure without a documented root cause.
  • Coverage drops after merges were investigated.

Summary

Debug is an order, not a gift. Legality, delivery, design, checking, consistency. Print before you blame. Read the first error. Walk the waveform backward. Group before you dig. Let AI do the reading, and keep the verdict for yourself.

The engineer who finds bugs fast is rarely the smartest one in the room. Just the most orderly one. 🔍

References

Related reading and learning path

  1. Typical UVM Testbench Architecture
  2. AXI Protocol Verification Guide
  3. CDC Verification Guide
  4. Gate-Level Simulation Guide
  5. Formal vs functional verification
  6. Coverage Closure Guide
  7. The Thought Process Behind Great Verification
  8. UVM Verbosity and Reporting
  9. UVM TLM: Ports, Exports, Analysis Ports and FIFOs
  10. Functional Coverage in SystemVerilog
  11. UVM Interview Questions by Level