Cranelift fuzzgen bug: fcmp returns the wrong result for NaN

A floating-point comparison bug in cranelift-fuzzgen can silently corrupt program semantics because NaN is not an ordinary numeric value. If generated test programs assume normal ordering rules for values that include NaN, the resulting fcmp expectations become invalid and fuzzing starts reporting mismatches that point to a deeper correctness issue in comparison lowering or oracle generation.

Understanding the Root Cause

The core of this issue is the special behavior of IEEE-754 NaN. Unlike regular floating-point numbers, NaN is unordered. That means comparisons such as ==, <, >, <=, and >= do not behave like standard numeric comparisons when either operand is NaN.

In practice:

• NaN == x is false for every x, including NaN.
• NaN < x and NaN > x are both false.
• unordered-aware predicates must explicitly account for NaN.

Cranelift exposes multiple floating-point comparison conditions, and each condition has precise semantics. The bug appears when fuzzgen generates a test case or expected result using logic that does not correctly model unordered comparisons. As a result, the generated oracle says one thing while the actual Cranelift IR fcmp semantics say another.

This usually happens in one of two places:

1. The test generator produces an expected boolean result using host-language comparisons that do not map cleanly to Cranelift condition codes.
2. The lowering or evaluation layer incorrectly treats NaN as comparable under ordered predicates.

For example, if a predicate is meant to represent an ordered comparison, then any operand containing NaN must usually force the result to false. If fuzzgen instead computes the expectation with incomplete NaN handling, the fuzz target reports a mismatch even when execution is otherwise stable.

The key takeaway is simple: NaN requires predicate-specific handling. You cannot derive correct fcmp results by reusing normal integer or total-order comparison intuition.

Step-by-Step Solution

The safest fix is to centralize floating-point comparison semantics in one helper used by fuzzgen when computing expected results. That helper should mirror Cranelift predicate semantics exactly.

1. Identify where fuzzgen computes expected fcmp outcomes

Search the fuzzgen codebase for floating-point comparison generation or interpretation logic.

git grep -n "fcmp
git grep -n "FloatCC
git grep -n "NaN"

You are looking for code that converts a generated FloatCC predicate and two float operands into a boolean expected result.

2. Replace ad-hoc comparison logic with a dedicated semantic helper

Create a helper that first detects unordered inputs, then applies the exact predicate mapping.

fn eval_fcmp(cond: FloatCC, a: f64, b: f64) -> bool {
    let unordered = a.is_nan() || b.is_nan();

    match cond {
        FloatCC::Ordered => !unordered,
        FloatCC::Unordered => unordered,
        FloatCC::Equal => !unordered && a == b,
        FloatCC::NotEqual => unordered || a != b,
        FloatCC::LessThan => !unordered && a < b,
        FloatCC::LessThanOrEqual => !unordered && a <= b,
        FloatCC::GreaterThan => !unordered && a > b,
        FloatCC::GreaterThanOrEqual => !unordered && a >= b,
        FloatCC::UnorderedOrEqual => unordered || a == b,
        FloatCC::UnorderedOrLessThan => unordered || a < b,
        FloatCC::UnorderedOrLessThanOrEqual => unordered || a <= b,
        FloatCC::UnorderedOrGreaterThan => unordered || a > b,
        FloatCC::UnorderedOrGreaterThanOrEqual => unordered || a >= b,
        _ => unimplemented!("handle all FloatCC variants explicitly"),
    }
}

If your local Cranelift version uses slightly different predicate names, map them according to the FloatCC enum in that revision.

3. Avoid relying on host comparison shortcuts

A common mistake is writing logic like this:

let expected = match cond {
    FloatCC::LessThan => a < b,
    FloatCC::Equal => a == b,
    FloatCC::NotEqual => a != b,
    _ => todo!(),
};

This is incomplete because it does not encode the full distinction between ordered and unordered predicates. For NaN-sensitive conditions, that shortcut breaks correctness.

4. Add a regression test using NaN-focused cases

Build tests that explicitly cover both signaling and quiet behavior at the semantic level, even if the host runtime only exposes standard is_nan() checks.

#[test]
fn fcmp_nan_regression() {
    use cranelift_codegen::ir::condcodes::FloatCC;

    let nan = f64::NAN;
    let one = 1.0f64;

    assert_eq!(eval_fcmp(FloatCC::Ordered, nan, one), false);
    assert_eq!(eval_fcmp(FloatCC::Unordered, nan, one), true);
    assert_eq!(eval_fcmp(FloatCC::Equal, nan, nan), false);
    assert_eq!(eval_fcmp(FloatCC::NotEqual, nan, nan), true);
    assert_eq!(eval_fcmp(FloatCC::LessThan, nan, one), false);
}

Also test normal numbers so the fix does not regress standard comparisons.

#[test]
fn fcmp_non_nan_still_works() {
    use cranelift_codegen::ir::condcodes::FloatCC;

    assert_eq!(eval_fcmp(FloatCC::Equal, 2.0, 2.0), true);
    assert_eq!(eval_fcmp(FloatCC::LessThan, 1.0, 2.0), true);
    assert_eq!(eval_fcmp(FloatCC::GreaterThan, 1.0, 2.0), false);
}

5. Re-run the fuzz target and regression suite

cargo test -p cranelift-fuzzgen
cargo fuzz run fuzzgen

If you have the original OSS-Fuzz reproducer, validate the fix directly against that testcase and confirm the mismatch no longer appears. When referencing the original report, link it using the OSS-Fuzz testcase page.

6. Keep the semantic mapping close to the source of truth

If possible, derive or share comparison semantics from a canonical Cranelift utility rather than duplicating the predicate matrix in multiple places. This reduces future drift between IR semantics, interpreter behavior, and fuzz oracle logic.

Common Edge Cases

• NaN vs NaN: Many engineers expect equality to be true here. It is false under IEEE-754 ordered equality.
• Unordered predicates: Conditions like unordered-or-equal must return true if either operand is NaN, even when numeric equality is meaningless.
• Negative zero: -0.0 == 0.0 is true, but some lower-level transformations may accidentally distinguish them if bitwise logic leaks into semantic comparison code.
• Infinity: +inf and -inf are ordered values, not NaN. They should follow normal ordering rules.
• f32 vs f64 mismatch: If fuzzgen promotes values inconsistently, expected results may differ from the actual instruction width being tested.
• Predicate coverage gaps: Handling only eq, lt, and gt while ignoring the rest leaves room for future fuzzbugs.
• Platform assumptions: Do not assume host compiler optimizations preserve the exact intended comparison semantics if helper logic is overly clever or incomplete.

FAQ

Why does NaN make fcmp so tricky?

Because NaN is unordered by IEEE-754 rules. Standard numeric expectations like equality or less-than simply do not apply unless the predicate explicitly models unordered behavior.

Is this a Cranelift code generation bug or a fuzzgen oracle bug?

It can be either, but issues like this often come from the expected-result generator in fuzzing infrastructure. The fix is to verify whether the runtime result or the oracle disagrees with documented FloatCC semantics.

What is the best long-term prevention strategy?

Centralize floating-point predicate evaluation, test every predicate against NaN-containing inputs, and keep regression cases derived from real fuzz reports such as this OSS-Fuzz reproducer.

In short, the issue is not that NaN is rare; it is that NaN invalidates ordinary comparison reasoning. Once fuzzgen computes expected fcmp results using exact ordered and unordered semantics, the false mismatches disappear and future floating-point regressions become much easier to catch.