Cranelift panic on vector icmp rules: fixing the missing ISLE type constraint

This panic is caused by a subtle mismatch between scalar-oriented ISLE lowering rules and vector-typed icmp instructions. When Cranelift tries to match comparison rules that do not explicitly constrain the operand or result types, vector inputs such as i32x4 can slip into a rule path that was only valid for scalar comparisons. The result is an invalid assumption during instruction selection and a compiler panic instead of a cleanly rejected match or correct lowering.

The failing pattern usually shows up with a test like this, where vector comparisons are optimized on x86_64:

test optimize
set opt_level=speed
target x86_64

function %trigger_bug(i32x4, i32x4) -> i32x4 {
block0(v0: i32x4, v1: i32x4):
    v2 = icmp sgt v0, v1
    v3 = icmp slt v0, v1
    ; ...
}

If the backend contains an ISLE rule for icmp that does not restrict the rule to scalar integer types, Cranelift may attempt to apply that rule to vector lanes as if they were ordinary scalar values.

Understanding the Root Cause

The bug exists because ISLE pattern matching is only as safe as the constraints attached to each rule. In Cranelift lowering, icmp is overloaded across multiple shapes:

• scalar integer compare
• vector compare
• target-specific compare forms

When a rule matches only on the opcode and condition code, but not on the expected value type, the rule becomes too broad. That broad rule can accidentally accept a vector type like i32x4.

Why does that lead to a panic instead of a normal compile-time miss?

• The rule body often calls helper constructors or extractors that assume a scalar machine value.
• Those helpers may inspect type bits, register classes, or immediate encodings that are invalid for vectors.
• Once the match proceeds far enough, an internal assertion or unreachable path is triggered.

In other words, the real problem is not the vector icmp itself. The problem is that the lowering rule lacks a type guard that tells ISLE, “this rule applies only to scalar integer comparisons.”

A typical problematic rule shape looks conceptually like this:

(rule (lower (icmp cc x y))
  ...scalar-specific lowering logic...)

That pattern is dangerous because x and y are unconstrained. A safer rule explicitly requires a scalar integer type or excludes vectors via a predicate.

Step-by-Step Solution

The fix is to add a missing type constraint to the affected ISLE icmp rules so that vector-typed comparisons do not match scalar lowering logic.

1. Locate the affected ISLE rules

Search the backend ISLE files for rules that lower icmp and route conditions such as sgt, slt, eq, or related signed and unsigned forms.

git grep "icmp" cranelift/codegen/src/isa

Focus on rules that:

• match generic icmp nodes
• do not mention a type predicate
• call scalar-specific helpers in the rule body

2. Identify the scalar assumption

Inspect the rule body and helper chain. If the rule ultimately builds a scalar compare, flag-setting operation, or general-purpose register sequence, it must not accept vector values.

A conceptually unsafe form:

(rule (lower (icmp cc a b))
  (some_scalar_cmp_sequence cc a b))

A safer form adds a type restriction before lowering:

(rule (lower (icmp cc a b))
  (if (scalar_int_ty (ty a)))
  (some_scalar_cmp_sequence cc a b))

The exact predicate name depends on the backend utilities already present in Cranelift. In practice, use the existing type helpers used elsewhere in the same ISA lowering files rather than inventing a new style.

3. Add an explicit type predicate

Update the ISLE rule so it matches only valid input kinds. The exact patch varies by file, but the structure should look like one of these patterns:

; Option A: constrain operand type
(rule (lower (icmp cc a b))
  (if-let ty (value_type a))
  (if (ty_is_scalar_int ty))
  (lower_scalar_icmp cc a b))

; Option B: use a dedicated extractor that already rejects vectors
(rule (lower (icmp cc a b))
  (if-let _ (as_scalar_int a))
  (lower_scalar_icmp cc a b))

If the target has separate vector lowering, keep that logic in a different rule:

(rule (lower (icmp cc a b))
  (if-let ty (value_type a))
  (if (ty_is_vector ty))
  (lower_vector_icmp cc a b))

The important part is not the exact helper name. The important part is that the scalar rule no longer matches vectors.

4. Preserve rule ordering carefully

ISLE rule selection can depend on specificity and order. After adding the type guard:

• put the more specific vector rule before broad fallback logic if required by the file’s conventions
• ensure generic fallback rules still handle legal scalar types
• avoid introducing unreachable duplicate rules

If a broad fallback remains unconstrained later in the file, it can reintroduce the same bug indirectly.

5. Add a regression test using the failing .clif

Create or extend a Cranelift test that reproduces the vector compare case. The goal is to guarantee that optimization and lowering no longer panic.

test optimize
set opt_level=speed
target x86_64

function %trigger_bug(i32x4, i32x4) -> i32x4 {
block0(v0: i32x4, v1: i32x4):
    v2 = icmp sgt v0, v1
    v3 = icmp slt v0, v1
    ; Use the compare results in a way that forces legalization/lowering.
    ; Example shape only; adapt to valid CLIF for the intended pipeline.
    return v0
}

Then run the relevant test suite:

cargo test -p cranelift-codegen
cargo test -p cranelift-filetests

If there is a targeted filetest runner in your workflow, run that as well so the exact lowering path is covered.

6. Validate both failure prevention and correctness

A good fix does two things:

• it prevents the panic
• it preserves correct lowering for scalar icmp cases

Test both categories:

; Scalar case should still match the scalar rule
function %scalar_ok(i32, i32) -> i8 {
block0(v0: i32, v1: i32):
    v2 = icmp sgt v0, v1
    return v2
}

; Vector case should no longer enter scalar-only lowering
function %vector_ok(i32x4, i32x4) -> i32x4 {
block0(v0: i32x4, v1: i32x4):
    v2 = icmp sgt v0, v1
    return v0
}

7. Write the patch message clearly

When opening the fix, describe the exact failure mode:

• an unconstrained ISLE icmp rule matched vector operands
• the rule body assumed scalar types
• adding a scalar type predicate prevented invalid matches and removed the panic

If possible, link the regression test and the issue in the commit message using the repository issue tracker.

Common Edge Cases

1. Only one operand is checked

If the rule constrains a but not b, and the compiler permits mixed forms in some intermediate stage, matching can still become inconsistent. In most compare rules, verify both operands are compatible or derive both from the same validated type source.

2. Result type versus operand type confusion

Some compare operations return a boolean-like lane mask or target-specific representation. Do not assume the result type alone is enough to determine legality. The dangerous assumption often lives in the operand type, not the result.

3. Backend-specific vector legalization paths

On one ISA, vectors may be lowered directly. On another, they may be legalized first. A fix that works for x86_64 should still be reviewed for other backends so that unconstrained rules are not duplicated elsewhere.

4. Signed and unsigned condition splits

It is common to fix sgt and slt but overlook ugt, ule, or equality-based rules that share the same helper chain. Audit the whole compare family, not just the first crashing condition code.

5. Test passes without covering instruction selection

A filetest that stops before the relevant lowering stage can give a false sense of safety. Make sure the test actually exercises the backend path where the panic originally happened.

6. Rule ordering masks the bug temporarily

Adding a more specific rule may hide the broad rule in one configuration while leaving it reachable in another optimization or legalization path. The safest fix is to constrain the broad rule itself.

FAQ

Why does Cranelift panic instead of just rejecting the vector icmp rule?

Because the unconstrained rule matches successfully at the pattern level, and the failure happens later inside scalar-only lowering logic. By that point, the backend has already committed to a path that assumes a valid scalar type.

Is this a bug in vector icmp support or in ISLE matching?

It is primarily a bug in how the lowering rule is written. ISLE is doing what the rule allows. The missing constraint makes the rule too permissive, so vector inputs reach code that was never meant to handle them.

What is the safest long-term fix pattern for similar issues?

Prefer narrow, explicit rules with strong type predicates. Any rule that builds scalar machine instructions should declare that requirement in the match itself. Then add a regression test with vector operands so future refactors cannot reopen the same bug.

The reliable fix is simple: add the missing scalar type constraint to the affected ISLE icmp rules, verify vectors no longer match that path, and lock it down with a regression test. That converts a backend panic into correct, predictable lowering behavior.