Cranelift on riscv64: fixing an unexpected illegal instruction under QEMU

An illegal instruction trap on riscv64 usually means the generated machine code assumes CPU features that your emulated target does not actually expose. In this Cranelift case, the failure is typically caused by a mismatch between the target ISA flags, the enabled RISC-V extensions, and what qemu-riscv64 can execute.

Symptoms and reproduction

The issue appears when a .clif test case compiles successfully, but the produced binary crashes at runtime inside qemu-riscv64 with an illegal instruction. That usually indicates one of these conditions:

• Cranelift emitted an instruction from an extension such as M, A, F, D, C, or V that QEMU was not configured to support.
• The generated code targeted a more capable CPU model than the emulator actually provides.
• A backend lowering bug selected a legal instruction form for one ISA profile but not for the concrete runtime environment.

In practical terms, this is often reproducible when running the same generated code on a different RISC-V profile than the one assumed during code generation.

Understanding the Root Cause

Cranelift does not generate generic “any-riscv64” code by magic. It lowers IR to machine instructions based on a specific target configuration. On RISC-V, that configuration is highly sensitive to enabled ISA extensions. If codegen believes an extension is available, it may emit instructions that are perfectly valid for that profile but illegal on a narrower target.

This matters more on RISC-V than on some other architectures because the base ISA is intentionally small, and many common operations depend on optional extensions. For example:

• Integer multiply/divide may require the M extension.
• Atomic operations may require the A extension.
• Compressed encodings depend on the C extension.
• Floating-point operations depend on F and D.

When running under QEMU user-mode emulation, the advertised CPU behavior may differ from your assumptions. If your Cranelift pipeline enables an extension that QEMU does not implement for the selected CPU, execution reaches a machine instruction that the emulator rejects, causing the trap.

There is also a second class of root cause: a genuine backend legalization bug. In that scenario, Cranelift may lower a legal CLIF operation into an instruction sequence that is not valid for the active RISC-V feature set. The visible symptom is the same, but the fix is different: either constrain the target flags to avoid the bad lowering path, or patch the backend.

Step-by-Step Solution

The safest fix is to make the Cranelift target ISA and the QEMU CPU profile agree exactly, then verify whether the illegal instruction still occurs. If it disappears, the problem was feature mismatch. If it remains, you likely found a backend bug.

1. Inspect the actual target triple and ISA flags

Make sure your test is not silently compiling for a broader target than intended.

rustc -vV
qemu-riscv64 --version

If your Cranelift driver or test harness accepts ISA flags, print them explicitly and confirm whether extensions like m, a, f, d, or c are enabled.

2. Reproduce with a minimal and explicit RISC-V target

Start from a conservative baseline. If your setup allows selecting a target string, use something equivalent to rv64gc only if QEMU supports it. Otherwise reduce to a smaller profile such as rv64imac or whatever your environment guarantees.

# Example workflow conceptually
# 1. Compile the .clif file with explicit riscv64 flags
# 2. Emit disassembly or object code
# 3. Run under qemu-riscv64 with the intended CPU model

If your harness can dump generated instructions, do that before execution. The goal is to identify the exact opcode that traps.

3. Disassemble the generated code

Once you have the object or executable, disassemble it and inspect the faulting region.

riscv64-linux-gnu-objdump -d ./test-binary > disasm.txt

Then compare the crashing instruction with the enabled extensions. If the failing opcode belongs to an extension your emulator lacks, the fix is to align the target flags or CPU model.

4. Run QEMU with a compatible CPU configuration

Some environments default to a generic CPU that does not match the codegen assumptions. Use a CPU profile that explicitly supports the required extensions, if available in your setup.

qemu-riscv64 -cpu rv64 ./test-binary

If your environment uses a different valid CPU name, use that instead. The key point is consistency between generated instructions and emulated features.

5. Reduce Cranelift ISA features if QEMU is the limiting factor

If QEMU cannot support the extension set your code currently uses, disable those features in your Cranelift configuration. The exact API depends on your embedding, but the pattern is always the same: create the ISA builder, then set only the features your runtime guarantees.

// Pseudocode
let mut flag_builder = settings::builder();
// Set conservative codegen flags here

let isa_builder = cranelift_codegen::isa::lookup_by_name("riscv64")?;
let isa = isa_builder.finish(settings::Flags::new(flag_builder))?;

When available, configure the RISC-V feature subset explicitly rather than relying on defaults.

6. If the instruction is illegal even with matched features, treat it as a backend bug

At that point, the likely issue is incorrect lowering or legalization in Cranelift. Create a minimized reproducer from the original .clif file and inspect which IR operation maps to the bad instruction.

# Useful debugging flow
# - shrink the CLIF test
# - recompile after each reduction
# - keep the smallest case that still traps
# - attach CLIF, disassembly, and QEMU command line to the bug report

This is the most actionable path for upstream maintainers because it separates environment mismatch from a true codegen defect.

7. Validate the fix

After aligning features or patching the backend, rerun the same case under QEMU and confirm that:

• No illegal instruction occurs.
• The generated output is functionally correct.
• The disassembly no longer contains unsupported opcodes for the selected CPU profile.

Common Edge Cases

• Compressed instruction mismatch: Code may include C extension encodings while the emulator or runtime expects uncompressed instructions only.
• Floating-point lowering: A CLIF operation may lower to F or D instructions indirectly, even if your source logic looks integer-only after optimization.
• Atomic operations: Runtime helpers, stack maps, or synchronization code can introduce A extension requirements.
• Host/target confusion: Building on x86_64 and testing through emulation can hide the fact that the binary was emitted for a richer RISC-V profile than the emulator supports.
• Old QEMU version: Some instructions may be valid in the chosen ISA profile but not fully supported by the installed emulator version.
• Relocation or trampoline confusion: The trap may appear near helper stubs or veneers, making it look like a Cranelift instruction-selection bug when the actual issue is elsewhere in the generated binary.

FAQ

Why does the code compile successfully but crash only at runtime?

Compilation validates instruction selection against the configured target ISA, not against the exact emulator instance you later run. If the runtime environment exposes fewer extensions, the binary is still well-formed but not executable there.

How do I know whether this is a QEMU limitation or a Cranelift bug?

Match the Cranelift feature set to the QEMU CPU profile first. If the illegal instruction disappears, it was an environment mismatch. If it persists with a verified compatible feature set, the problem is likely in Cranelift lowering or legalization.

What should I include in a high-quality upstream bug report?

Provide the minimized .clif file, the exact Cranelift revision, the QEMU version, the command used to run the binary, the selected target flags, and a disassembly snippet around the faulting instruction. That combination makes the issue reproducible and much faster to diagnose.

The practical resolution is simple: keep Cranelift ISA settings and QEMU CPU capabilities in lockstep. On riscv64, even one incorrectly assumed extension is enough to turn valid generated code into an immediate illegal instruction trap.