Cranelift Function::is_leaf Is Wrong in Important Cases, and That Can Break ABI-Level Frame Pointer Decisions

When leaf-function detection is too optimistic, Cranelift can classify a function as a leaf even though later lowering introduces a call. Because is_leaf influences whether the ABI emits or omits a frame pointer, this mismatch can produce incorrect prologue or epilogue behavior, unstable unwinding assumptions, and architecture-specific bugs that only appear after register allocation or final lowering.

The core issue is simple: Function::is_leaf answers a question too early in the compilation pipeline. At the point where it runs, not every eventual call site is necessarily visible yet. Some calls are introduced indirectly by legalization, ABI expansion, stack checks, trap handling, register spills involving helper routines on some targets, or backend-specific lowering. If the ABI trusts that early answer, it may make a permanent frame-layout decision based on incomplete information.

Understanding the Root Cause

In compiler backends, a leaf function is usually defined as a function that does not call any other function. That sounds straightforward, but in Cranelift the observable call graph can change between IR construction and machine-code emission.

This creates the bug pattern:

1. The frontend or early middle-end builds a function that appears to contain no explicit call instructions.
2. Function::is_leaf returns true.
3. The ABI layer uses that result to decide whether a frame pointer is needed.
4. Later compilation stages introduce one or more calls, or otherwise require call-compatible frame semantics.
5. The function is no longer a true leaf, but the earlier ABI decision has already been baked in.

Why this is technically dangerous:

• Frame pointer omission can be valid for genuine leaf functions, but unsafe once outgoing calls, stack adjustments, or unwind-sensitive layouts are introduced.
• Backend lowering may transform abstract operations into helper calls that were not present in the original IR.
• Platform ABI rules often tie unwind info, stack walking, and debugger behavior to stable frame layout assumptions.
• Late machine passes know more than Function::is_leaf did when queried earlier.

So the actual root cause is not merely a wrong boolean. It is a phase-ordering bug: a property that should be decided late is being consumed early by code that needs a definitive answer.

A robust fix therefore does one of two things:

• Stops trusting early leaf detection for ABI-critical frame-pointer decisions, or
• Recomputes leaf-ness at a compilation stage where all effective calls are known.

In practice, the safest strategy is usually to make frame-pointer policy depend on a more conservative signal. If there is any possibility that later lowering can introduce calls, the function should be treated as non-leaf for ABI purposes unless proven otherwise at the final stage.

Step-by-Step Solution

The best fix is to move leaf-sensitive ABI decisions away from early Function::is_leaf deduction and toward a backend-final or conservative decision point. Below is a practical implementation strategy.

1. Find where is_leaf influences frame-pointer selection

Search the backend and ABI code paths for uses of Function::is_leaf, especially where it affects:

• frame-pointer preservation
• prologue generation
• callee-saved register setup
• unwind metadata generation

rg "is_leaf\(" cranelift/ winch/ wasmtime/

You are looking for logic similar to this:

let omit_frame_pointer = func.is_leaf() && other_conditions;

If that result is consumed before final lowering, it is the likely bug source.

2. Replace optimistic leaf detection with a conservative ABI predicate

Instead of using raw Function::is_leaf, introduce a dedicated ABI-facing predicate whose semantics reflect compilation reality.

pub fn may_omit_frame_pointer(func: &Function, flags: &Flags) -> bool {
    if flags.preserve_frame_pointers() {
        return false;
    }

    // Conservative: only omit if the backend can guarantee
    // that no late-introduced calls will appear.
    false
}

This first version intentionally disables omission in ambiguous cases. It is the safest correctness-first patch, and it is often appropriate for a bug fix.

3. If needed, compute final leaf-ness later in the pipeline

If performance or code size matters enough to preserve leaf optimization, compute leaf status after the backend has complete knowledge of effective calls.

pub struct FinalFrameInfo {
    pub has_calls: bool,
    pub needs_frame_pointer: bool,
}

pub fn compute_final_frame_info(mach_func: &MachineFunction, flags: &Flags) -> FinalFrameInfo {
    let has_calls = mach_func.blocks().iter().any(|block| {
        block.insts().iter().any(|inst| inst.is_call())
    });

    let needs_frame_pointer = flags.preserve_frame_pointers() || has_calls;

    FinalFrameInfo {
        has_calls,
        needs_frame_pointer,
    }
}

The exact data structure will differ across Cranelift internals, but the principle remains the same: use the machine-level representation, not just the original IR function object.

4. Decouple semantic leaf-ness from ABI frame policy

One common mistake is treating leaf-ness as the single source of truth for frame layout. Separate them explicitly:

pub enum FramePointerPolicy {
    Always,
    OmitWhenProvablySafe,
    NeverOmitForABI,
}

pub fn choose_frame_pointer_policy(flags: &Flags) -> FramePointerPolicy {
    if flags.preserve_frame_pointers() {
        FramePointerPolicy::Always
    } else {
        FramePointerPolicy::OmitWhenProvablySafe
    }
}

Then consume that policy only after the backend has enough information to prove safety.

5. Update the existing leaf helper to clarify its limitations

Even if you keep Function::is_leaf, document that it reflects only the currently visible IR and must not be used for ABI-critical final decisions unless no later pass can introduce calls.

/// Returns true if the current IR contains no direct call-like operations.
///
/// This is not a guaranteed final machine-level leaf determination.
/// Backend lowering or legalization may still introduce calls, so callers
/// must not rely on this alone for final ABI frame-pointer decisions.
pub fn is_leaf(&self) -> bool {
    // existing logic
}

6. Add regression tests for late-introduced calls

This is where the fix becomes durable. You want tests that fail under the old behavior and pass under the new one.

#[test]
fn frame_pointer_not_omitted_when_lowering_introduces_call() {
    let func = build_ir_that_looks_leaf_but_lowers_to_call();
    let compiled = compile(func);

    assert!(compiled.prologue().uses_frame_pointer());
}

Useful test categories include:

• IR that appears leaf but legalizes into helper calls
• target-specific lowering that emits call sequences
• stack probes or guard checks inserted late
• debug or unwind-sensitive compilation modes

7. Validate on architectures where frame-pointer policy matters most

Run targeted tests on backends where ABI conventions and unwind behavior are especially sensitive, such as x86_64 and AArch64.

cargo test -p cranelift-codegen frame_pointer
cargo test -p cranelift-codegen leaf
cargo test --workspace

If the repository uses filetests or ISA-specific backends, add those too:

cargo test -p cranelift-filetests
cargo test -p cranelift-codegen --features all-arch

8. Prefer correctness over micro-optimization in the patch

If you are preparing a contribution for review, a conservative fix is easier to merge than a clever one that still risks ABI regressions. A strong patch usually follows this progression:

1. Make frame-pointer omission safe by default.
2. Add tests proving the previous behavior was wrong.
3. Optionally restore optimization later using final machine-level analysis.

A minimal patch often looks like this conceptually:

// Before
let is_leaf = func.is_leaf();
let use_fp = flags.preserve_frame_pointers() || !is_leaf;

// After
let use_fp = true_if_not_provably_safe_to_omit_after_lowering();

Or, if you need a transitional implementation:

let use_fp = if flags.preserve_frame_pointers() {
    true
} else {
    // Conservative fallback until final leaf analysis is available.
    true
};

That may look pessimistic, but it fixes correctness immediately.

Common Edge Cases

Even after fixing the obvious is_leaf misuse, several edge cases can still cause incorrect behavior if not tested.

Late helper calls introduced by legalization

Some operations may lower into libcall-style helpers only on certain ISAs or only for specific operand widths. A function may be leaf on one target and non-leaf on another.

Stack probes and guard-page handling

Large stack allocations can trigger stack probing or target-specific probing sequences. If those rely on calls or frame-sensitive setup, early leaf deduction becomes invalid.

Exception handling and unwind metadata

Even if generated code happens to run, missing or incorrect unwind information can break profilers, debuggers, and panic propagation. Test both execution and metadata assumptions.

Tail calls versus normal calls

A backend may rewrite some calls as tail calls. Depending on ABI semantics, these may interact differently with frame-pointer requirements. Do not assume all call-like instructions have identical frame implications.

Inline assembly or backend pseudo-instructions

If a target backend uses pseudo-instructions that later expand to call sequences, a naive scan of early IR will miss them.

Conditional call insertion

Some late passes insert calls only under debugging, sanitization, trap handling, or special flag combinations. Regression tests should cover multiple compilation modes.

Leaf-ness cached too early

Even if you improve the analysis, cached booleans stored on the function object can go stale after mutation. If the pipeline mutates call behavior, either invalidate the cache or compute the answer from final state.

FAQ

1. Why not just fix Function::is_leaf itself?

Because the deeper problem is when the answer is used. If later passes can still introduce calls, no early IR-only implementation can be fully correct for ABI decisions. You either need a later analysis point or a conservative policy.

2. Does this bug always crash generated code?

No. In many cases the generated code may appear to work, but still violate ABI expectations for frame walking, debugging, profiling, or unwinding. That makes the issue subtle and platform-dependent.

3. What is the safest patch to submit first?

The safest first patch is to stop using early is_leaf to omit frame pointers when correctness is uncertain. Then add regression tests, and only after that consider reintroducing omission using final machine-level proof.

In short, the correct engineering fix is to treat early Function::is_leaf as advisory at most, and never as the final authority for ABI-level frame-pointer decisions unless the backend can prove no later stage will introduce calls. That change removes a fragile assumption, aligns frame policy with actual machine code, and prevents backend-specific correctness regressions that are otherwise very hard to diagnose.