How to Fix: There is another bug in “invoke_func” function

Wasmtime’s invoke_func parser is rejecting valid NaN payload literals like nan:0x200000 and -nan:0x200000, even though they follow the same float-text conventions expected by the CLI and test inputs. The visible symptom is the misleading error “invalid float literal”, but the real problem is narrower: the parser handles ordinary numeric floats and some special values, yet fails when a signed or unsigned NaN-with-payload form is passed through this specific code path.

Understanding the Root Cause

This bug is a parser consistency issue inside invoke_func. In WebAssembly tooling, floating-point values are not limited to decimal strings like 1.5 or special values like inf. They may also include NaN literals with explicit payload bits, such as nan:0x200000. These payload-bearing forms are especially important in spec tests and low-level runtime validation because they let developers verify exact IEEE-754 behavior.

The failure happens because the parsing logic in invoke_func does not fully recognize the grammar for NaN payload literals. In practice, the implementation likely takes one of these problematic approaches:

• It delegates directly to Rust’s standard float parsing, which does not understand strings like nan:0x200000.
• It special-cases nan and -nan but stops before handling the :0x... suffix.
• It validates the token too early and emits the generic invalid float literal error before the NaN-payload branch can run.

This explains why the bug is described as being similar to the earlier linked issue: one parser path was fixed, but another parsing entry point still contains incomplete support. The result is inconsistent behavior across commands that should accept the same literal format.

Technically, both nan:0x200000 and -nan:0x200000 should be interpreted as NaN bit patterns with a user-provided payload, while preserving sign handling where the surrounding format allows it. The parser therefore needs to do more than call str::parse<f32>() or str::parse<f64>(); it must explicitly parse the token structure, decode the hexadecimal payload, validate the payload range, and then construct the final float value from raw bits.

Step-by-Step Solution

The fix is to make invoke_func use the same float-literal grammar as the rest of the Wasmtime codebase. Instead of relying on generic float parsing alone, add a dedicated branch for NaN payload syntax.

1. Locate the argument parsing logic used by invoke_func.
2. Find where f32 and f64 arguments are converted from strings.
3. Add explicit handling for nan:0x... and -nan:0x....
4. Validate that the payload is hexadecimal and fits the mantissa constraints of the target float type.
5. Construct the value from raw bits instead of numeric parsing.
6. Reuse existing helper logic if another module already parses these literals correctly.

A practical implementation pattern looks like this:

fn parse_wasm_f32(s: &str) -> Result<f32, String> {
    if let Some(bits) = parse_nan_payload_f32(s)? {
        return Ok(f32::from_bits(bits));
    }

    match s {
        "inf" | "+inf" => Ok(f32::INFINITY),
        "-inf" => Ok(f32::NEG_INFINITY),
        "nan" | "+nan" => Ok(f32::NAN),
        "-nan" => Ok(-f32::NAN),
        _ => s.parse::<f32>().map_err(|_| format!("invalid float literal: {s}")),
    }
}

fn parse_nan_payload_f32(s: &str) -> Result<Option<u32>, String> {
    let (negative, rest) = if let Some(x) = s.strip_prefix("-") {
        (true, x)
    } else if let Some(x) = s.strip_prefix("+") {
        (false, x)
    } else {
        (false, s)
    };

    let payload_str = match rest.strip_prefix("nan:0x") {
        Some(x) => x,
        None => return Ok(None),
    };

    let payload = u32::from_str_radix(payload_str, 16)
        .map_err(|_| format!("invalid float literal: {s}"))?;

    if payload == 0 || payload > 0x7fffff {
        return Err(format!("invalid NaN payload for f32: {s}"));
    }

    let sign = if negative { 1u32 << 31 } else { 0 };
    let exponent = 0xffu32 << 23;
    let quiet_bit = 1u32 << 22;
    let mantissa = payload | quiet_bit;

    Ok(Some(sign | exponent | mantissa))
}

The same pattern applies to f64, except the payload width and bit layout differ:

fn parse_nan_payload_f64(s: &str) -> Result<Option<u64>, String> {
    let (negative, rest) = if let Some(x) = s.strip_prefix("-") {
        (true, x)
    } else if let Some(x) = s.strip_prefix("+") {
        (false, x)
    } else {
        (false, s)
    };

    let payload_str = match rest.strip_prefix("nan:0x") {
        Some(x) => x,
        None => return Ok(None),
    };

    let payload = u64::from_str_radix(payload_str, 16)
        .map_err(|_| format!("invalid float literal: {s}"))?;

    if payload == 0 || payload > 0x000f_ffff_ffff_ffff {
        return Err(format!("invalid NaN payload for f64: {s}"));
    }

    let sign = if negative { 1u64 << 63 } else { 0 };
    let exponent = 0x7ffu64 << 52;
    let quiet_bit = 1u64 << 51;
    let mantissa = payload | quiet_bit;

    Ok(Some(sign | exponent | mantissa))
}

If Wasmtime already contains a shared parser for spec-compliant float literals, the best fix is not duplicating logic but routing invoke_func through that existing helper. That avoids one parser accepting nan:0x200000 while another rejects it.

After applying the change, add targeted tests:

#[test]
fn parse_f32_nan_payload() {
    assert!(parse_wasm_f32("nan:0x200000").is_ok());
    assert!(parse_wasm_f32("-nan:0x200000").is_ok());
}

#[test]
fn parse_f64_nan_payload() {
    assert!(parse_wasm_f64("nan:0x200000").is_ok());
    assert!(parse_wasm_f64("-nan:0x200000").is_ok());
}

#[test]
fn reject_invalid_nan_payload() {
    assert!(parse_wasm_f32("nan:0x0").is_err());
    assert!(parse_wasm_f32("nan:xyz").is_err());
}

Then verify the CLI behavior manually in the reproduction path mentioned by the issue. The expected outcome is that valid payload-bearing NaN literals are accepted and only malformed ones produce errors.

Common Edge Cases

• Zero payloads: Depending on the parser rules you want to mirror, nan:0x0 may need to be rejected because it does not encode a meaningful payload distinct from plain nan.
• Payload overflow: f32 and f64 have different mantissa widths. A payload valid for f64 may be invalid for f32.
• Sign preservation: -nan:0x200000 should not be normalized away if the surrounding semantics permit a negative NaN bit pattern.
• Quiet vs signaling NaN: If your parser always forces the quiet bit, document that behavior clearly. Some runtimes canonicalize NaNs, while others preserve payload bits more faithfully.
• Case sensitivity: Inputs like NaN:0x200000 or NAN:0x200000 may currently fail if the grammar is intentionally lowercase-only.
• Error quality: Returning only invalid float literal makes debugging harder. A more specific message such as invalid NaN payload is better when the prefix is recognized but the payload is malformed.

FAQ

Why doesn’t standard Rust float parsing accept nan:0x200000?

Because Rust’s built-in float parser supports ordinary numeric strings and a small set of special literals, but not the extended WebAssembly-style NaN payload syntax. That syntax must be implemented separately.

Should the fix be made only in invoke_func?

No. The safest long-term fix is to centralize float literal parsing so every CLI path, test harness, and invocation entry point uses the same grammar and produces consistent results.

Why is -nan:0x200000 important if NaN is unordered anyway?

Because this issue is about bit-level correctness, not just arithmetic behavior. Spec tests, runtime conformance, and debugging tools often care about the exact encoded float representation, including sign and payload.

For maintainers, the key takeaway is simple: this is not a generic floating-point bug, but a missing parser branch for a valid WebAssembly float literal form. Once invoke_func is taught to parse payload-bearing NaNs the same way as the already-fixed path from the related issue, the misleading error disappears and CLI behavior becomes consistent.