Add the framework for BNIL emulator and implement LLIL emulator - #8314
Add the framework for BNIL emulator and implement LLIL emulator#8314xusheng6 wants to merge 44 commits into
Conversation
|
We also want to add at least one test for each LLIL instruction |
Addresses review feedback (erroneous formatting) on PR #8314. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…njaapi.h Addresses review feedback (erroneous formatting) on PR #8314. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…aders binaryninjaapi.h does not provide the emulator API, so the wide-integer type it pulls in for the plugin does not belong there. Move the (windows.h min/max-guarded) intx include into a shared emulator_intx.h included by the emulator's own core and API roots instead. Addresses review feedback on PR #8314. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…MIN/-1 DIVS/MODS/MULS_DP/CMP_S*/MINS/MAXS funneled operands through int64_t via SignExtend(v, sz, 8), which silently truncated any operand wider than 8 bytes to its low 64 bits, and native signed division of INT_MIN by -1 is undefined behavior (SIGFPE on x86). Add wide-signed helpers (IsNegative/SignedLess/SignedDivideOrModulo) that operate directly on the 512-bit value: comparisons use sign-aware unsigned compares, and division works on magnitudes then reapplies the sign, so INT_MIN / -1 wraps to INT_MIN (and % to 0) instead of trapping. Addresses bdash review comment on PR #8314. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The basic two-operand arithmetic cases masked operands only when recording the flag context, while evaluating the operation on the raw operands, and the double-precision cases mask to sz/2 for unrelated reasons. This was noted as confusing in review. Mask both operands to sz once at the top of each case and use those masked values for both the computation and m_lastArithmetic, and add a comment explaining why the double-precision ops mask differently. No behavior change. Addresses bdash review comment on PR #8314. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In the CALL/CALL_STACK_ADJUST/TAILCALL cases, a builtin stub or the call hook can request a stop (e.g. on an error) while returning false, but the code only checked the boolean return and fell through into the call/tailcall path, executing further despite the pending stop. Check m_stopReason after the stub and hook handling and return if a stop was requested. Addresses bdash review comment on PR #8314. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The plugin's LLILEmulator wrapper lived in namespace BinaryNinja, which is reserved for the core API. Move it to its own BinaryNinjaEmulatorAPI namespace, mirroring the debugger's BinaryNinjaDebuggerAPI. Addresses plafosse review comment on PR #8314. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…GetView The view member is reference counted; returning a raw BinaryView* from a public getter invites use-after-free if the emulator outlives the caller's assumptions. Return Ref<BinaryView> so callers hold their own reference. Addresses emesare review comment on PR #8314. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The emulator half-supported a null architecture with scattered guards that silently no-op'd or fell back to little-endian. Instead reject creation of an emulator whose view/IL has no architecture (BNCreateLLILEmulator* returns null with a logged error) and drop the now-unnecessary null-arch guards, so m_arch is a hard invariant everywhere it is used. Addresses emesare review comment on PR #8314. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… convention SetArgument always used the platform's default calling convention, contradicting the documented behavior of using the function's own convention. Derive the calling convention from the function being emulated (via its LLIL function), falling back to the platform default only when the function has none. Also note the address-size-as-stack-slot-size assumption for exotic ABIs. Addresses emesare review comments on PR #8314. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace ~130 raw operand accesses in EvalExpr/ExecuteCurrentInstruction with the typed LowLevelILInstruction accessors (GetLeftExpr/GetRightExpr/GetCarryExpr, GetSourceExpr, GetDestRegister, GetHighRegister/GetLowRegister, GetConstant, GetDestExpr, GetParameterExprs, GetTarget/GetTrueTarget/GetFalseTarget, etc.), which are self-documenting and avoid the operand-index mixups raw indexing invites. Each mapping was verified against the LowLevelILInstructionAccessor specializations to read the same operand index; LLIL_EXTERN_PTR is intentionally left raw (its constant+offset has no equivalent typed pair). No behavior change; the full emulator test suite passes. Addresses bdash review comment on PR #8314. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Per Peter's comment, we need to have at least one test for each LLIL instruciton
| @@ -0,0 +1,353 @@ | |||
| # BNIL Emulator — Python API Guide | |||
There was a problem hiding this comment.
Typically we have reserved this kind of content for the developer guide.
There was a problem hiding this comment.
I agree, but the only way to use the emulator is through its API (well except for using the BNIL debug adapter), so this should be fine
| if (!m_view) | ||
| return; | ||
|
|
||
| // Place arguments using the calling convention of the function being emulated, falling |
There was a problem hiding this comment.
Fixed in 8ccc939. Now it places the incoming arguments into the correct register/stack location when it can, and falls back to the default calling convention's location otherwise
| m_view(backingView), m_il(nullptr), m_arch(nullptr) | ||
| { | ||
| if (m_view) | ||
| m_arch = m_view->GetDefaultArchitecture(); |
There was a problem hiding this comment.
For binaries with multiple architecture (thumb) we should probably add some warning.
There was a problem hiding this comment.
Added. Rather than scanning every function up front, it now warns lazily: the architecture switches (SetEntryPoint/EnterFunction/tailcall/state-restore) go through a SetActiveArchitecture helper that logs a one-time warning when the arch being emulated differs from the binary's default. 609d0ac
Moving forward, we might be able to properly support arch switching code, though that would require more work and I will leave it as future work
|
We have also decided to mark the emulator as experimental (which we already did), and disable it in binja be default |
Add an experimental plugin under plugins/emulator that emulates Binary Ninja's Low Level IL with full register, flag, and memory state, structured like the debugger: a core engine (emulatorcore) exposing a C ABI, plus C++ (emulatorapi) and Python bindings. Supports cross-function emulation, breakpoints, arguments, built-in libc stubs, user hooks (call/syscall/memory/pre-instruction/intrinsic/ stdio), and JSON state serialization. Includes a self-contained Python unittest suite (plugins/emulator/test), a user guide (docs/guide/emulator.md) wired into the mkdocs nav, and the intx vendor library for wide register values. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses review feedback (erroneous formatting) on PR #8314. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MapMemory allocates a buffer of a caller-controlled length; a std::bad_alloc / length_error from a huge or garbage length would propagate out of the extern "C" map functions, which is undefined behavior. Catch allocation failures in EmulatorMemory::Map (the single point the four map FFI entry points funnel through) and turn them into a logged no-op. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In the CALL/CALL_STACK_ADJUST/TAILCALL cases, a builtin stub or the call hook can request a stop (e.g. on an error) while returning false, but the code only checked the boolean return and fell through into the call/tailcall path, executing further despite the pending stop. Check m_stopReason after the stub and hook handling and return if a stop was requested. Addresses bdash review comment on PR #8314. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Python memory-read hook did result.to_bytes(buf_len, ...), which raises OverflowError if the hook returns a value that does not fit in buf_len bytes; the bare except then swallowed it and silently returned False, dropping a valid hook result. Mask the value to the buffer width first (matching the C++ bridge's truncating behavior) so it is stored instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The plugin's LLILEmulator wrapper lived in namespace BinaryNinja, which is reserved for the core API. Move it to its own BinaryNinjaEmulatorAPI namespace, mirroring the debugger's BinaryNinjaDebuggerAPI. Addresses plafosse review comment on PR #8314. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…GetView The view member is reference counted; returning a raw BinaryView* from a public getter invites use-after-free if the emulator outlives the caller's assumptions. Return Ref<BinaryView> so callers hold their own reference. Addresses emesare review comment on PR #8314. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The emulator half-supported a null architecture with scattered guards that silently no-op'd or fell back to little-endian. Instead reject creation of an emulator whose view/IL has no architecture (BNCreateLLILEmulator* returns null with a logged error) and drop the now-unnecessary null-arch guards, so m_arch is a hard invariant everywhere it is used. Addresses emesare review comment on PR #8314. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… convention SetArgument always used the platform's default calling convention, contradicting the documented behavior of using the function's own convention. Derive the calling convention from the function being emulated (via its LLIL function), falling back to the platform default only when the function has none. Also note the address-size-as-stack-slot-size assumption for exotic ABIs. Addresses emesare review comments on PR #8314. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The README stated tests/python/test_emulator.py loads this module so the tests run as part of the main Binary Ninja test suite, but that file does not exist and nothing under tests/ references the emulator. Remove the inaccurate paragraph. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… path
- docs/guide/emulator.md: the stdout callback example used sys.stdout without
importing sys; use print(..., end='') so the snippet runs as shown.
- api/python/CMakeLists.txt: the non-BN_API_PATH include fell back to
${CMAKE_SOURCE_DIR}/api/cmake/..., which does not exist for an out-of-tree
build; point it at the api root via a correct relative path (matching warp).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ding The stdin callback's output buffer was a char* in the FFI, so the generated Python binding exposed it as c_char_p and ctypes handed the callback an immutable bytes copy, making the memmove into it write to the wrong memory. Type the buffer void* across the FFI/C++ bridge so the generated binding uses a writable c_void_p pointer; the Python callback now receives the real buffer address to write into. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace ~130 raw operand accesses in EvalExpr/ExecuteCurrentInstruction with the typed LowLevelILInstruction accessors (GetLeftExpr/GetRightExpr/GetCarryExpr, GetSourceExpr, GetDestRegister, GetHighRegister/GetLowRegister, GetConstant, GetDestExpr, GetParameterExprs, GetTarget/GetTrueTarget/GetFalseTarget, etc.), which are self-documenting and avoid the operand-index mixups raw indexing invites. Each mapping was verified against the LowLevelILInstructionAccessor specializations to read the same operand index; LLIL_EXTERN_PTR is intentionally left raw (its constant+offset has no equivalent typed pair). No behavior change; the full emulator test suite passes. Addresses bdash review comment on PR #8314. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…on/memory edges Cover the previously-untested behavior that recent fixes touch: flag computation (overflow/sign/carry/zero via setcc, signed compare via setl), sign/zero extension (movsx/movzx), shifts and rotate-through-carry (shl/shr/sar/rcl), signed division of INT_MIN by -1 (must wrap, not trap), division by zero (stops with Error), cross-segment memory reads, and rejection of overlapping maps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
LSL/LSR/ASR masked the shift count with & (sz*8-1), imposing x86-style masking on every architecture. But the architecture's own count masking is already encoded in the lifted IL (x86 emits e.g. `al u>> (cl & 0x1f)` and pre-masks immediates; aarch64 emits a raw `w0 u>> w1`), so re-masking was wrong: it made 8/16-bit x86 shifts by >= the operand width a no-op instead of 0, and imposed x86 semantics on other arches. Perform the literal shift instead — a count >= the operand width yields 0 for the logical shifts and a full sign-fill for the arithmetic shift. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to moving intx out of binaryninjaapi.h: the intermediate wrapper header was misplaced (first under the emulator plugin, which coupled the debugger to it). Drop it and include the shared api/vendor/intx/intx.hpp directly from the emulator's own headers, the same copy and style the debugger already uses. No windows.h min/max guard is needed — the codebase already relies on NOMINMAX in the adapters that include <windows.h>. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses review feedback: for binaries that mix architectures (most commonly ARM/Thumb) the single-architecture LLIL emulator cannot correctly emulate functions of a non-default architecture, so surface a warning. Rather than enumerate every analysis function up front to detect this, warn lazily: route the function-entry architecture switches (SetEntryPoint, EnterFunction, tailcall, state restore) through SetActiveArchitecture, which warns once when the architecture being emulated differs from the binary's default. This is O(1) per function entry and only fires when a non-default architecture is actually reached. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses review feedback on how arguments are placed for functions with non-default argument locations. SetArgument now consults the emulated function's parameter variables and places each argument at its actual location instead of always using the calling convention's defaults: - register-located parameters are placed in their assigned register - stack-located parameters are written to the stack slot the function reads from (the parameter's storage offset relative to the entry stack pointer) Parameters in any other location fall back to the calling convention's default placement as before. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses review feedback: MaskToSize, SignExtend, IsNegative, SignedLess and the signed divide/modulo helpers operate purely on intx::uint512 values and byte sizes with no LLIL dependency. Move them from LLILEmulator to the ILEmulator base class so future IL emulators can reuse them instead of duplicating the logic. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses review feedback on `m_flags[flag] = value ? 1 : 0;`. Every flag in the architecture model is a single boolean bit — ComputeFlagForRole, the only producer, always yields 0 or 1, and every consumer treats a flag as 0/1 — so represent flags as bool. m_flags stores bool, and GetFlag / SetFlag are bool in / bool out, assigning straight into the map without a ternary. The FFI boundary keeps its uint8_t representation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses review feedback: the LLIL_CALL and LLIL_CALL_STACK_ADJUST cases mixed break and return. Since nothing follows the dispatch switch, break and return are equivalent here; standardize on return to match the sibling control-flow cases (jumps, tailcalls) and avoid the confusion that the mixed style could cause. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ntion Addresses review feedback: SetArgument was updated to use the emulated function's calling convention, but ReadArgument and WriteReturnValue still used only the platform default. Extract a shared ResolveCallingConvention helper (function convention, falling back to the platform default) and use it in all three so argument and return-value handling stay consistent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses review feedback: Symbol::GetShortName already returns a std::string, so the explicit std::string(...) construction is redundant. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses review feedback: use std::optional<std::string> instead of an empty-string sentinel to signal that a call target name could not be resolved, and update the callers accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses review feedback that every LLIL instruction the emulator implements should have at least one unit test. Adds emulator_il_test.py, which builds LLIL directly with LowLevelILFunction (rather than lifting assembly) so each of the 88 operations the emulator handles is exercised in isolation: constants, register read/write and splits, the full arithmetic/logic/shift/rotate/compare/bit families, the double-precision multiply and divide/modulo ops, memory and stack, flags, control flow (goto/if/jump/call/ret/syscall/...), and the nop/trap/undef/unimpl/ intrinsic specials. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… function LowLevelILFunction::GetArchitecture() derives the architecture from the owning Function, so a standalone LowLevelILFunction -- one built directly through the API rather than lifted -- reports none. SetEntryPoint(il, index) passed that null straight into SetActiveArchitecture, clearing the valid architecture the constructor had taken from the binary view. Since a valid architecture is a hard emulator invariant (the null guards were removed deliberately), the very next register access dereferenced m_arch and segfaulted, which took out the entire emulator_il_test.py suite on its first test rather than failing it. Resolve the architecture through the view's default when the IL function does not supply one, and never install a null over a valid architecture. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
For the double-precision multiplies, LLIL's `size` is the width of each operand and the product is twice that: x86 `mul rbx` lifts to `rdx:rax = mulu.dp.q(rax, rbx)`, whose .q operands are 8 bytes feeding a 16-byte register pair. The handlers instead treated `size` as the width of the product, so they masked each operand to size/2 and then truncated the product to size. Both halves of that were wrong and compounded. Emulating `mul rbx` with rax = rbx = 2**32 masked each operand to 4 bytes, turning 2**32 into exactly zero, and the product mask discarded the high half unconditionally -- so the 128-bit result came back as 0:0 instead of 1:0. Any multiply whose result did not fit in a single register was silently wrong. Mask the operands to `size`, and the product to 2*size. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
LLIL_TEST_BIT takes a value and a bit index: x86 `bt rax, 2` lifts to `flag:c = test_bit(rax, 2)`, which must yield bit 2 of rax. The handler computed `left & right` instead, so it tested a mask. The two agree only when the index happens to equal a single-bit mask (index 1), so `bt` produced a wrong carry for nearly every bit position -- with rax = 4, testing bit 2 gave 4 & 2 == 0 instead of 1. Shift by the index and take the low bit. The index is applied literally, leaving any instruction-specific modulo to the lifter, with a bound check so an out-of-range index cannot shift past the emulator's value width. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
LLIL_JUMP_TO carries a map from candidate target address to IL instruction index, but the handler ignored it and resolved the destination with GetInstructionStart(). That looks an address up through the owning function's mapping, so it only works for IL lifted from a real function; for IL built directly through the API the lookup fails and the jump reports an unmapped address even though the instruction names the target index outright. Consult the instruction's target map first and keep GetInstructionStart() as the fallback, so JUMP_TO resolves in both cases. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Several per-instruction tests were themselves wrong, and only became visible once the suite stopped segfaulting on its first test. LLIL constants carry a 64-bit value, so il.const(16, 1 << 64) and il.const(16, -17) silently truncated: the first made SET_REG_SPLIT's source 2 rather than 2**64 + 2, and the second turned the DIVS_DP/MODS_DP dividend into a large positive 128-bit value, testing unsigned division under a signed name. Build those operands with a shift and a sign-extend instead. The ADC/SBB carry, SET_FLAG and IF conditions used il.const(0, 1) -- a zero-byte constant, which correctly masks to 0, so the tests asserted carry-in of 1 while passing 0. Real lifted IL uses a flag expression here (adc.q(rax, rbx, flag:c)), never a zero-size constant; use a 1-byte constant. test_load seeded memory with (0xefbeadde).to_bytes(4, 'little') and expected a little-endian load to return 0xdeadbeef, which is the byte-reversed value. The intrinsic hook took four parameters, but the documented and implemented contract is (emulator, intrinsic_id, params) returning a list of (register, value) pairs. The resulting TypeError was swallowed by the binding's bare except and surfaced only as an Unimplemented stop. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The existing suites test in the small: emulator_test.py assembles 3-13 byte x86-64 snippets and mostly exercises the API surface, and emulator_il_test.py checks one hand-built LLIL operation per assertion. Neither runs a long, arithmetic-dense trace, so a wrong bit in a carry, rotate or mask only shows up if someone thought to write that exact case. Add kat/kat.c with freestanding base64 and MD5 kernels, built for x86-64 and aarch64, and emulate them against the RFC 4648 and RFC 1321 vectors plus a differential comparison with hashlib across MD5's block boundaries. A published digest is a far better oracle than a hand-written assertion: one wrong bit anywhere changes it completely, and MD5's 64 rounds of rotate-and-wrapping-add exercise those paths unconditionally. The kernels are compiled -ffreestanding -nostdlib with builtins and autovectorization disabled, and take all buffers from the caller, so there is no process startup, no syscalls and no libc: what is under test is the emulator core rather than the libc stub layer. Compiling the same C per architecture also means one set of vectors validates each lifter/emulator pair -- which is how the aarch64 shift-count divergence documented in AArch64KATTests was found. The objects are checked in so the suite does not need a cross-compiler; rebuild them with kat/build.sh when kat.c changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

This PR adds the emulator framework and implements LLIL emulation. It closes #1212 and Vector35/debugger#320
The emulator is implemented as an open-source plugin in the API repo to enable 3rd party inspection and contribution. It may also later serve as a "spec" of the definition of our IL
The emulator does its only C FFI and Python/C++ API binding in the same way as BN core and the debugger.
The scope of this PR is to get the core LLIL emulation done, i.e., we should be able to emulate the LLIL instructions and functions properly. Full system emulation and API emulation is out of scope, though some of them are added to make it possible to verify the result of the emulation, e.g., things can be printed to stdout to validate the emulation
This PR is also accompanied by some core changes (https://github.com/Vector35/binaryninja/pull/1656) the a BNIL debug adapter (https://github.com/Vector35/debugger/tree/test_emulator) which enables the user to visually inspect the behavior of the emulator. The core changes would need to land with this PR. And I would also prefer to have the debugger changes land together because 1) it changes how the intx headers are shared between the emulator and the debugger, 2) it is the more straightforward way to inspect the emulator's status
The tests cover every LLIL instruction we have, plus a few small binaries that does computations (hash, encryption) (to be added)
The IL emulator will be marked as experimental and disabled by default.
Expected follow-up work:
Note: this PR also enables the boss plugin (https://github.com/xusheng6/boss) which uses emulation to detect obfuscated strings in the same way as flare-floss (https://github.com/mandiant/flare-floss)