Cranelift panic calculating timing total: why it crashes and how to fix it safely

This crash is a classic release-build invariant violation: Cranelift assumes compiler passes are recorded in a valid order, but when that ordering is broken, a debug_assert catches it only in debug builds. In production, the bad state survives long enough to trigger a panic while calculating the total timing tree.

In practice, this means telemetry may show a crash deep inside Cranelift even though the real bug happened earlier when a timing pass was started, ended, or nested incorrectly. The right fix is to make timing aggregation defensive in release mode and preserve the ordering check as a development-time assertion.

Understanding the Root Cause

Cranelift tracks compilation pass timings as a sequence of events or hierarchical spans. Later, it computes a total duration by walking that sequence and aggregating nested pass timings. That logic usually assumes:

• every started pass is ended correctly,
• nested passes are emitted in strict stack order,
• no pass closes a parent out of order,
• the event stream is internally consistent.

When those assumptions hold, total time calculation is straightforward. But if the event stream is malformed, the timing code may attempt to pop the wrong stack frame, subtract mismatched durations, or read a parent that does not exist. In debug builds, a debug_assert! may catch this earlier. In release builds, that assertion is disabled, so the invalid state propagates until a later panic.

A simplified version of the failure pattern looks like this:

// Expected nesting order: A -> B -> end B -> end A
start("A")
start("B")
end("A") // invalid: A closes before B
end("B")

If timing aggregation assumes a proper stack, the code handling end("A") may panic because "B" is actually on top. This is why the issue often appears as a crash in timing total calculation, even though the root bug is a corrupted pass ordering invariant.

There are two engineering lessons here:

1. Debug assertions are not enough when malformed state can occur in production telemetry.
2. Diagnostic subsystems must degrade gracefully; timing data should never bring down compilation.

Step-by-Step Solution

The safest fix is to convert the timing-total computation from a panic-prone invariant consumer into a best-effort validator. Keep assertions for developers, but in release mode return a safe result when ordering is invalid.

1. Reproduce the invariant failure locally

Before patching, add a regression test that simulates out-of-order timing events. The exact Cranelift API may differ, but the test should capture the malformed nesting pattern.

#[test]
fn malformed_pass_order_does_not_panic() {
    let mut timings = PassTimings::new();

    timings.start_pass("A");
    timings.start_pass("B");
    timings.end_pass("A"); // invalid ordering
    timings.end_pass("B");

    let total = timings.total_duration();
    assert!(total.is_some() || total.is_none());
}

The important part is not the exact assertion. The goal is to prove that malformed telemetry no longer crashes the compiler.

2. Replace panic-prone logic with checked state handling

If the existing code assumes the top of a stack must match the closing pass, change it to validate first. A typical defensive pattern is:

pub fn total_duration(&self) -> Option<Duration> {
    let mut stack = Vec::new();
    let mut total = Duration::ZERO;

    for event in &self.events {
        match event {
            TimingEvent::Start { name, at } => {
                stack.push((name, at));
            }
            TimingEvent::End { name, at } => {
                let Some((open_name, open_at)) = stack.pop() else {
                    log::warn!("timing end without matching start: {}", name);
                    return None;
                };

                if open_name != name {
                    log::warn!(
                        "timing pass order mismatch: expected end for {}, got {}",
                        open_name,
                        name
                    );
                    return None;
                }

                if *at < *open_at {
                    log::warn!("timing event has negative duration for pass: {}", name);
                    return None;
                }

                total += *at - *open_at;
            }
        }
    }

    if !stack.is_empty() {
        log::warn!("timing stack not empty after aggregation");
        return None;
    }

    Some(total)
}

This change does three things:

• prevents an unchecked pop or mismatched access,
• returns Option or Result instead of panicking,
• preserves visibility through logging or telemetry.

3. Keep the assertion, but do not rely on it

Assertions are still valuable during development. Keep them near the event-recording side, but ensure the aggregation side remains safe even when the invariant is violated.

pub fn end_pass(&mut self, name: &str) {
    debug_assert!(self.pass_order_is_valid_for_end(name));
    self.events.push(TimingEvent::End {
        name: name.to_string(),
        at: Instant::now(),
    });
}

This gives maintainers fast feedback in debug mode without exposing release users to compiler crashes.

4. Prefer Result if callers need diagnostics

If you need to distinguish malformed timing data from missing timing data, use Result with a dedicated error type.

#[derive(Debug)]
pub enum TimingError {
    EndWithoutStart(String),
    MismatchedEnd { expected: String, got: String },
    NegativeDuration(String),
    UnclosedPasses,
}

pub fn total_duration(&self) -> Result<Duration, TimingError> {
    // same validation flow, but return detailed errors
    # unimplemented!()
}

This is especially useful if the surrounding compiler pipeline wants to count, report, or suppress timing corruption separately.

5. Ensure callers tolerate missing totals

Once total calculation returns None or Err(...), update downstream code so timing is treated as optional metadata rather than a hard requirement.

match timings.total_duration() {
    Some(total) => report.total_compile_time = Some(total),
    None => {
        log::warn!("skipping invalid cranelift timing total");
        report.total_compile_time = None;
    }
}

This is the key production hardening step: invalid timing data must not fail compilation.

6. Add regression coverage for release behavior

Because debug_assert! disappears in release builds, add tests that specifically validate non-panicking behavior under optimized execution. If your project uses CI, run both debug and release test configurations.

cargo test
cargo test --release

If this bug was discovered through telemetry, also add a targeted unit test or fuzz-like test that randomizes start/end order and verifies the timing subsystem never panics.

7. If contributing upstream, shape the patch for maintainers

For an upstream Cranelift fix, keep the patch minimal and focused:

• one code path change in timing aggregation,
• one or two regression tests,
• a short commit message explaining that malformed pass ordering should not panic in release builds.

A good summary is: validate pass nesting during total calculation and return a non-fatal error on malformed timing streams.

Common Edge Cases

Even after fixing the main panic, several adjacent cases can still produce bad timing data if not handled carefully.

1. End event without a start event

This usually happens when instrumentation is conditionally compiled, partially skipped, or interrupted by an early return. Without a guard, stack-pop logic panics immediately.

2. Unclosed passes at function exit

If a pass starts but the function returns early on an error path, the stack may remain non-empty. Total calculation should detect this and reject the aggregate safely.

3. Re-entrant or nested instrumentation bugs

Compiler pipelines can invoke helper routines that also emit timings. If those nested spans are not consistently structured, you may see mismatched parent-child ordering even though individual functions seem correct.

4. Negative or nonsensical durations

While rarer, corrupted timestamps or mixed clock sources can produce negative durations or impossible intervals. Always validate timestamp ordering before subtraction.

5. Telemetry-only crashes masking the true source

The panic may appear only in user telemetry because production traffic exercises a broader set of code paths. The actual bug may be elsewhere in pass emission, not in timing math. Logging the first malformed event pair is critical for root-cause discovery.

6. Parallel compilation interactions

If timing state is shared accidentally across threads, events from different compilations can interleave and break stack assumptions. Each compilation unit should own isolated timing state, or synchronization must enforce correctness.

FAQ

Why does this panic show up only in release telemetry?

Because debug_assert! is disabled in release builds. The invalid pass order is still created, but instead of being caught immediately, it reaches the timing total calculation and panics later.

Should Cranelift panic at all for malformed timing data?

No. Timing is diagnostic metadata, not core compilation correctness. The compiler should log the problem, drop the invalid timing aggregate, and continue compiling.

Is removing the assertion enough to fix the bug?

No. Removing the assertion only hides the symptom in debug mode. The real fix is to make aggregation and reporting code handle malformed pass ordering safely with Option, Result, and validation checks.

The practical resolution is simple: treat pass timing order as an invariant for developers, but treat malformed timing streams as recoverable input in production. That preserves diagnostics, avoids crashes, and makes Cranelift significantly more robust under real-world telemetry.