When wasmtime_instance_new succeeds but leaves trap untouched, your caller can read garbage and mis-handle success as failure.

This issue appears when native code expects out-parameters to be deterministic on every call path, but the success path in handle_instantiate does not explicitly clear the trap pointer. If the caller passes an uninitialized wasm_trap_t* trap; and instantiation succeeds, the function may return success while trap still contains whatever value happened to be in memory before the call.

The Problem

Consider the reported usage:

wasm_trap_t* trap;
auto err = wasmtime_instance_new(context, module, &import, 1, &instance, &trap);

If wasmtime_instance_new succeeds, many C and C++ callers reasonably expect:

• err == nullptr
• trap == nullptr
• instance to be initialized

The bug is that the success path can leave trap unchanged unless a trap actually occurred. That creates an API contract problem: successful instantiation does not fully initialize all output parameters.

In practice, this can lead to:

• false-positive trap handling in caller code
• undefined behavior when stale pointers are dereferenced
• non-deterministic test failures depending on stack state
• hard-to-reproduce crashes in FFI consumers

Understanding the Root Cause

The root cause is simple but important: the trap out-parameter is conditionally written instead of unconditionally initialized.

At the API boundary, wasmtime_instance_new ultimately relies on an internal instantiation handler such as handle_instantiate. On the failure or trap path, code typically assigns a trap object into the caller-provided location. But on the success path, if no explicit assignment like *trap = nullptr; happens first, the caller receives an indeterminate value.

This is especially dangerous in C-style APIs because:

• callers often declare output pointers without zero-initializing them
• FFI consumers assume null means “no result” or “no error”
• success and error are frequently represented through multiple outputs simultaneously

Technically, this is not just a caller mistake. A robust native API should ensure every documented out-parameter is initialized on all code paths where the pointer argument itself is non-null.

The intended semantic model is usually:

• err != nullptr: API-level error occurred
• trap != nullptr: WebAssembly trap occurred during instantiation or start
• both null: success

That model breaks if the implementation leaves trap uninitialized on success.

Step-by-Step Solution

The fix is to clear the trap output before any instantiation logic runs, or at minimum guarantee it is set to nullptr on every successful return path.

1. Initialize the trap out-parameter at function entry

If you maintain the runtime or are patching the affected code, update the implementation so the out-pointer is normalized immediately.

wasmtime_error_t* wasmtime_instance_new(..., wasm_trap_t** trap) {
    if (trap != nullptr) {
        *trap = nullptr;
    }

    // existing instantiation logic...
}

If the actual bug lives in handle_instantiate, the same principle applies there:

static wasmtime_error_t* handle_instantiate(..., wasm_trap_t** trap) {
    if (trap != nullptr) {
        *trap = nullptr;
    }

    // perform instantiation
    // if a trap occurs:
    //     *trap = created_trap;
    //     return nullptr;
    // if an error occurs:
    //     return error;
    // on success:
    //     return nullptr;
}

This is the safest pattern because it guarantees a known state before branching.

2. Preserve the semantic distinction between error and trap

Do not collapse traps into generic errors unless the API contract explicitly requires that. A correct implementation should still distinguish these cases:

if (api_error) {
    return error;
}

if (runtime_trap) {
    if (trap != nullptr) {
        *trap = runtime_trap;
    }
    return nullptr;
}

if (trap != nullptr) {
    *trap = nullptr;
}
return nullptr;

Even if you initialize at the top, writing the success semantics clearly makes maintenance easier.

3. Add a regression test for the success path

The issue exists because trap handling was likely tested only when a trap occurred. Add a regression test that proves success explicitly nulls the out-pointer.

TEST(case_trap_pointer_is_cleared_on_success) {
    wasm_trap_t* trap = reinterpret_cast<wasm_trap_t*>(0x1);
    wasmtime_instance_t instance;

    wasmtime_error_t* err = wasmtime_instance_new(
        context,
        module,
        imports,
        num_imports,
        &instance,
        &trap
    );

    ASSERT_EQ(err, nullptr);
    ASSERT_EQ(trap, nullptr);
}

Using a non-null sentinel value is important. If the test initializes trap to nullptr, it may pass even when the implementation is still broken.

4. Harden caller code as a defensive workaround

Even after patching the library, callers should still initialize output pointers defensively:

wasm_trap_t* trap = nullptr;
wasmtime_instance_t instance;
wasmtime_error_t* err = wasmtime_instance_new(
    context,
    module,
    &import,
    1,
    &instance,
    &trap
);

if (err != nullptr) {
    // handle API error
} else if (trap != nullptr) {
    // handle wasm trap
} else {
    // success
}

This does not replace the library fix, but it reduces exposure for downstream users.

5. Document the API contract explicitly

If you are contributing the fix upstream, update the API documentation or code comments to state that:

• if trap is provided, it will be set to nullptr on success
• if a trap occurs, it will point to a valid wasm_trap_t
• callers must still release trap resources according to the runtime’s ownership rules

That documentation matters for C, C++, and language bindings built on top of the native API.

Example patch pattern

static wasmtime_error_t* handle_instantiate(
    wasmtime_context_t* context,
    wasmtime_module_t* module,
    const wasmtime_extern_t* imports,
    size_t nimports,
    wasmtime_instance_t* instance,
    wasm_trap_t** trap
) {
    if (trap != nullptr) {
        *trap = nullptr;
    }

    // existing validation and instantiation code

    if (/* trapped during start or instantiation */) {
        if (trap != nullptr) {
            *trap = produced_trap;
        }
        return nullptr;
    }

    if (/* ordinary API error */) {
        return produced_error;
    }

    *instance = produced_instance;
    return nullptr;
}

The key idea is not the exact syntax. The key is enforcing deterministic output initialization.

Common Edge Cases

1. Caller passes an uninitialized local pointer

This is the exact reproducer. Without the fix, the stack value may look non-null and trigger bogus trap handling.

2. Caller passes nullptr for the trap out-parameter

The implementation must guard all assignments with if (trap != nullptr). Initializing the output is correct only when the output location exists.

3. Instantiation fails with an API error before trap-producing logic runs

Even in this case, it is best for trap to remain explicitly null if the caller supplied it. That avoids ambiguous mixed-state outputs.

4. Start function traps after instance creation begins

Some runtimes distinguish module instantiation from execution of the module’s start function. If the start function traps, the API should still reliably return the trap through trap and not through stale memory.

5. Bindings in Rust, Python, or other languages

FFI layers often assume native functions obey strict nullability rules. A non-deterministic trap out-parameter can break wrapper logic, produce incorrect exceptions, or leak resources due to confused ownership tracking.

6. Tests that only assert on err

A test suite may report success because err == nullptr while missing that trap is dirty. This bug survives when regression coverage checks only one output channel.

FAQ

Should callers still initialize trap to nullptr themselves?

Yes. Caller-side initialization is a good defensive habit, especially in C/C++. But the library should still set the trap out-parameter deterministically on success. Both sides can and should be safe.

Why return nullptr for both success and trap cases?

Because many Wasmtime-style APIs separate host/API errors from WebAssembly traps. An API error is returned as wasmtime_error_t*, while a trap is returned via wasm_trap_t**. That separation is useful, but only if both outputs are initialized consistently.

What is the minimal upstream fix for this issue?

The minimal safe fix is to add if (trap != nullptr) *trap = nullptr; at the beginning of the instantiation handler or wrapper function, before any early returns or trap-producing branches.

In short, the bug is not that trapping fails. The bug is that success does not clear the trap channel. Fix that by initializing the trap out-parameter on entry, keep error and trap semantics separate, and add a regression test using a non-null sentinel pointer so the issue never comes back.