How to Fix: [GC] assertion failed: `types.is_subtype(actual_ty, expected_ty)`

A failing GC subtype assertion in Wasmtime usually means the validator or translator accepted a WebAssembly GC type relationship that later violates an internal invariant: types.is_subtype(actual_ty, expected_ty). In practice, the module contains a reference type, struct/array/function subtype, or recursive type usage that is legal in one phase of compilation but is checked more strictly in another, triggering a crash instead of a clean validation error.

Reproducing the crash

The issue is triggered by running Wasmtime with all proposals enabled against the provided test case:

./target/debug/wasmtime -W=all-proposals=y subtype...

If the input module exercises GC proposal subtype relationships aggressively, Wasmtime may hit an assertion while checking whether an actual heap type is a subtype of an expected heap type.

This is not a normal runtime trap. It is a compiler or validator invariant failure, which strongly suggests a bug in the engine rather than a user-level mistake in host code.

Understanding the Root Cause

The assertion types.is_subtype(actual_ty, expected_ty) fails when Wasmtime reaches a point where it assumes subtype correctness has already been established. That assumption can break for a few tightly related reasons:

• Inconsistent type canonicalization: recursive or nominal GC types may be represented differently across validation, translation, or optimization passes.
• Incorrect variance handling: fields, parameters, results, or aggregate members may be treated as subtype-compatible when they are not.
• Cross-recursion mismatches: a type inside a recursive group may reference another type index that looks structurally similar but is not the same canonical subtype.
• Proposal interaction bugs: enabling all proposals can expose combinations of GC, typed function references, and subtyping rules that do not appear in narrower configurations.

Technically, WebAssembly GC subtyping is subtle because it combines nominal type identity, recursive type groups, and heap-type relations. A module can pass an early parser stage, but later IR lowering may require a stronger guarantee: if an instruction expects one exact reference family, a merely similar but non-canonical type can trigger the assertion.

In short, the bug happens because Wasmtime internally trusts that subtype resolution is complete and correct by the time code generation or lowering runs. The supplied testcase exposes a path where that trust is invalid.

Step-by-Step Solution

Because this is an engine bug, the practical fix is to minimize the testcase, verify the failing type relationship, and patch Wasmtime so invalid subtype assumptions become proper validation failures or are handled correctly.

1. Reproduce with debug assertions enabled

cargo build --debug -p wasmtime-cli
./target/debug/wasmtime -W=all-proposals=y ./subtype.wasm

This confirms the assertion path and preserves line-level debug context.

2. Inspect the module’s type section

Use a Wasm inspection tool to review recursive groups, sub declarations, and reference-typed instructions:

wasm-tools print ./subtype.wasm > subtype.wat

Then inspect:

• type definitions using (sub ...)
• struct and array field types
• ref.cast, call_ref, struct.new, array.new, or other GC-sensitive instructions
• references across recursive type groups

3. Minimize the testcase

Reduce the module until the assertion still reproduces. This makes the real subtype bug obvious and easier to patch.

wasm-tools shrink ./subtype.wasm -o ./subtype-min.wasm
./target/debug/wasmtime -W=all-proposals=y ./subtype-min.wasm

If wasm-tools shrink does not preserve the exact failure, manually remove:

• unused functions
• unrelated exports
• extra recursive types
• non-GC instructions

4. Add logging around subtype checks in Wasmtime

Instrument the code path that performs or assumes subtyping. The exact file depends on the current Wasmtime revision, but the target is any location where heap types or translated module types are compared.

eprintln!("actual_ty = {:?}", actual_ty);
eprintln!("expected_ty = {:?}", expected_ty);
eprintln!("is_subtype = {}", types.is_subtype(actual_ty, expected_ty));

Also log:

• the instruction opcode being lowered
• the type index and recursive group
• whether types were canonicalized before comparison

5. Fix the validation or canonicalization gap

The correct engine-side fix usually falls into one of these categories:

• Validate earlier: reject invalid subtype relationships during module validation instead of asserting later.
• Canonicalize consistently: ensure both actual_ty and expected_ty are compared in the same type universe.
• Preserve recursive group identity: do not compare structurally similar types from different nominal contexts as if they were interchangeable.
• Use fallible errors instead of assertions: if malformed or unsupported combinations are possible, return a user-visible validation error.

A typical defensive patch looks like this:

if !types.is_subtype(actual_ty, expected_ty) {
    bail!("invalid GC subtype relation: actual type {:?} is not a subtype of expected type {:?}", actual_ty, expected_ty);
}

This does not replace the real fix, but it converts a crash into actionable diagnostics while the deeper subtype bug is resolved.

6. Add a regression test

Once fixed, add the minimized testcase to the Wasmtime test suite so the subtype relation is validated in CI.

cargo test gc -- --nocapture

Recommended regression coverage:

• the original crashing module
• a minimized equivalent
• nearby subtype variations that should pass
• invalid variants that should fail cleanly

7. Temporary workaround for users

If you only need to avoid the crash in production while waiting for an upstream fix:

• disable broad proposal enablement instead of using -W=all-proposals=y
• avoid the specific GC subtype pattern in generated Wasm
• test against a newer Wasmtime commit in case the bug is already fixed upstream

./target/debug/wasmtime ./module.wasm

Or selectively enable only required proposals rather than all experimental combinations.

Common Edge Cases

• Recursive type groups with reordered members: even if two groups look structurally identical, nominal identity can differ and break subtype assumptions.
• Nullable vs non-null references: a subtype check can fail if a lowering pass forgets nullability constraints.
• Function reference subtyping: parameter and result variance rules are easy to get wrong when mixed with typed function references.
• Cross-module assumptions: types imported from one module may not line up with locally defined canonical forms in another.
• Debug vs release behavior: debug builds assert loudly, while release builds may exhibit harder-to-diagnose miscompilation or a different error surface.
• Proposal skew: a testcase may fail only when GC, function references, and other proposals are enabled together.

FAQ

Is the WebAssembly module definitely invalid?

No. An assertion failure strongly suggests a Wasmtime bug. The module may be valid, invalid but poorly diagnosed, or valid under one interpretation and mishandled during translation. The engine should return a clean error instead of crashing.

Why does this happen only with -W=all-proposals=y?

That flag enables experimental and advanced feature combinations, including GC-related typing rules that exercise code paths not used by ordinary modules. It is a common way to uncover incomplete subtype handling.

What is the best long-term fix?

The best fix is to make subtype checking consistent across validation, canonicalization, and lowering, then add a regression test based on the minimized module. If the module is invalid, Wasmtime should emit a deterministic validation error. If the module is valid, the translator must preserve the correct nominal subtype relation all the way through compilation.

The key takeaway is simple: this crash is caused by a broken assumption about WebAssembly GC subtyping inside Wasmtime. Reproduce it in debug mode, minimize the testcase, inspect recursive and nominal type relationships, then patch the engine so subtype mismatches are either correctly accepted or cleanly rejected.