Skip to contentSkip to conversation

x86 instruction emulator fuzzing

PATCH In review Sam Caccavale 4 participants 9 messages Jun 28, 2019 – Jul 3, 2019 KVM

lore
Share & tools
lore
#0 lore raw

v4 cover x86 instruction emulator fuzzing

Dear all,

This series aims to provide an entrypoint for, and fuzz KVM's x86 instruction emulator from userspace. It mirrors Xen's application of the AFL fuzzer to it's instruction emulator in the hopes of discovering vulnerabilities. Since this entrypoint also allows arbitrary execution of the emulators code from userspace, it may also be useful for testing.

The current 4 patches build the emulator and 2 harnesses: simple-harness is an example of unit testing; afl-harness is a frontend for the AFL fuzzer. The fifth patch contains useful scripts for development but is not intended for usptream consumption.

Patches
=======

- 01: Builds and links afl-harness with the required kernel objects.
- 02: Introduces the minimal set of emulator operations and supporting code to emulate simple instructions.
- 03: Demonstrates simple-harness as a unit test.
- 04: Adds scripts for install and building.
- 05: Useful scripts for development

Issues
=======

Currently, fuzzing results in a large amount of FPU related crashes. Xen's fuzzing efforts had this issue too. Their (temporary?) solution was to disable FPU exceptions after every instruction iteration? Some solution is desired for this project.

Changelog
=======

v1 -> v2:
- Moved -O0 to ifdef DEBUG
- Building with ASAN by default
- Removed a number of macros from emulator_ops.c and moved them as static inline functions in emulator_ops.h
- Accidentally changed the example in simple-harness (reverted in v3)
- Introduced patch 4 for scripts

v2 -> v3:
- Removed a workaround for printf smashing the stack when compiled with -mcmodel=kernel, and stopped compiling with -mcmodel=kernel
- Added a null check for malloc's return value
- Moved more macros from emulator_ops.c into emulator_ops.h as static inline functions
- Removed commented out code
- Moved changes to emulator_ops.h into the first patch
- Moved addition of afl-many script to the script patch
- Fixed spelling mistakes in documentation
- Reverted the simple-harness example back to the more useful original one
- Moved non-essential development scripts from patch 4 to new patch 5

v3 -> v4:
- Stubbed out all unimplemented emulator_ops with a unimplemented_op macro
- Setting FAIL_ON_UNIMPLEMENTED_OP on compile decides whether calling these is treated as a crash or ignored
- Moved setting up core dumps out of the default build/install path and detailed this change in the README
- Added a .sh extention to afl-many
- Added an optional timeout to afl-many.sh and made deploy_remote.sh use it
- Building no longer creates a new .config each time and does not force any config options
- Fixed a path bug in afl-many.sh

Any comments/suggestions are greatly appreciated.

Best,
Sam Caccavale

Sam Caccavale (5):
Build target for emulate.o as a userspace binary Emulate simple x86 instructions in userspace Demonstrating unit testing via simple-harness Added build and install scripts Development scripts for crash triage and deploy

 tools/Makefile                                |   9 +
 tools/fuzz/x86ie/.gitignore                   |   2 +
 tools/fuzz/x86ie/Makefile                     |  54 ++
 tools/fuzz/x86ie/README.md                    |  21 +
 tools/fuzz/x86ie/afl-harness.c                | 151 +++++
 tools/fuzz/x86ie/common.h                     |  87 +++
 tools/fuzz/x86ie/emulator_ops.c               | 590 ++++++++++++++++++
 tools/fuzz/x86ie/emulator_ops.h               | 134 ++++
 tools/fuzz/x86ie/scripts/afl-many.sh          |  31 +
 tools/fuzz/x86ie/scripts/bin.sh               |  49 ++
 tools/fuzz/x86ie/scripts/build.sh             |  34 +
 tools/fuzz/x86ie/scripts/coalesce.sh          |   5 +
 tools/fuzz/x86ie/scripts/deploy.sh            |   9 +
 tools/fuzz/x86ie/scripts/deploy_remote.sh     |  10 +
 tools/fuzz/x86ie/scripts/gen_output.sh        |  11 +
 tools/fuzz/x86ie/scripts/install_afl.sh       |  15 +
 .../fuzz/x86ie/scripts/install_deps_ubuntu.sh |   5 +
 tools/fuzz/x86ie/scripts/rebuild.sh           |   6 +
 tools/fuzz/x86ie/scripts/run.sh               |  10 +
 tools/fuzz/x86ie/scripts/summarize.sh         |   9 +
 tools/fuzz/x86ie/simple-harness.c             |  49 ++
 tools/fuzz/x86ie/stubs.c                      |  59 ++
 tools/fuzz/x86ie/stubs.h                      |  52 ++
 23 files changed, 1402 insertions(+)
 create mode 100644 tools/fuzz/x86ie/.gitignore
 create mode 100644 tools/fuzz/x86ie/Makefile
 create mode 100644 tools/fuzz/x86ie/README.md
 create mode 100644 tools/fuzz/x86ie/afl-harness.c
 create mode 100644 tools/fuzz/x86ie/common.h
 create mode 100644 tools/fuzz/x86ie/emulator_ops.c
 create mode 100644 tools/fuzz/x86ie/emulator_ops.h
 create mode 100755 tools/fuzz/x86ie/scripts/afl-many.sh
 create mode 100755 tools/fuzz/x86ie/scripts/bin.sh
 create mode 100755 tools/fuzz/x86ie/scripts/build.sh
 create mode 100755 tools/fuzz/x86ie/scripts/coalesce.sh
 create mode 100644 tools/fuzz/x86ie/scripts/deploy.sh
 create mode 100755 tools/fuzz/x86ie/scripts/deploy_remote.sh
 create mode 100755 tools/fuzz/x86ie/scripts/gen_output.sh
 create mode 100755 tools/fuzz/x86ie/scripts/install_afl.sh
 create mode 100755 tools/fuzz/x86ie/scripts/install_deps_ubuntu.sh
 create mode 100755 tools/fuzz/x86ie/scripts/rebuild.sh
 create mode 100755 tools/fuzz/x86ie/scripts/run.sh
 create mode 100755 tools/fuzz/x86ie/scripts/summarize.sh
 create mode 100644 tools/fuzz/x86ie/simple-harness.c
 create mode 100644 tools/fuzz/x86ie/stubs.c
 create mode 100644 tools/fuzz/x86ie/stubs.h

--
2.17.1

Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Ralf Herbrich Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879

#1 lore raw

PATCH v4 1/5 Build target for emulate.o as a userspace binary

This commit contains the minimal set of functionality to build afl-harness around arch/x86/emulate.c which allows exercising code in that source file, like x86_emulate_insn.

---

v1 -> v2:
- Moved -O0 to ifdef DEBUG
- Building with ASAN by default

v2 -> v3:
- Removed a workaround for printf smashing the stack when compiled with -mcmodel=kernel, and stopped compiling with -mcmodel=kernel
- Added a null check for malloc's return value
- Moved multiple macros from emulator_ops.c into emulator_ops.h as static inline functions.

v3 -> v4:
- Added unimplemented_op macro to stub out emulator ops
- envvar FAIL_ON_UNIMPLEMENTED_OP decides whether AFL treats this as a crash
- Updated README to include core dump setup

---
 tools/Makefile                  |   9 ++
 tools/fuzz/x86ie/.gitignore     |   2 +
 tools/fuzz/x86ie/Makefile       |  51 +++++++++++
 tools/fuzz/x86ie/README.md      |  21 +++++
 tools/fuzz/x86ie/afl-harness.c  | 151 ++++++++++++++++++++++++++++++++
 tools/fuzz/x86ie/common.h       |  87 ++++++++++++++++++
 tools/fuzz/x86ie/emulator_ops.c |  58 ++++++++++++
 tools/fuzz/x86ie/emulator_ops.h | 134 ++++++++++++++++++++++++++++
 tools/fuzz/x86ie/stubs.c        |  59 +++++++++++++
 tools/fuzz/x86ie/stubs.h        |  52 +++++++++++
 10 files changed, 624 insertions(+)
 create mode 100644 tools/fuzz/x86ie/.gitignore
 create mode 100644 tools/fuzz/x86ie/Makefile
 create mode 100644 tools/fuzz/x86ie/README.md
 create mode 100644 tools/fuzz/x86ie/afl-harness.c
 create mode 100644 tools/fuzz/x86ie/common.h
 create mode 100644 tools/fuzz/x86ie/emulator_ops.c
 create mode 100644 tools/fuzz/x86ie/emulator_ops.h
 create mode 100644 tools/fuzz/x86ie/stubs.c
 create mode 100644 tools/fuzz/x86ie/stubs.h
tools/Makefile +9 -0
index 3dfd72ae6c1a..b55990c19afd 100644
@@ -94,6 +94,12 @@ freefall: FORCE
 kvm_stat: FORCE
 	$(call descend,kvm/$@)
 
+fuzz: FORCE
+	$(call descend,fuzz/x86ie)
+
+fuzz_deps: FORCE
+	$(call descend,fuzz/x86ie,fuzz_deps)
+
 all: acpi cgroup cpupower gpio hv firewire liblockdep \
 		perf selftests spi turbostat usb \
 		virtio vm bpf x86_energy_perf_policy \
@@ -171,6 +177,9 @@ tmon_clean:
 freefall_clean:
 	$(call descend,laptop/freefall,clean)
 
+fuzz_clean:
+	$(call descend,fuzz/x86ie,clean)
+
 build_clean:
 	$(call descend,build,clean)
 
tools/fuzz/x86ie/.gitignore new file +2 -0
new file mode 100644
index 000000000000..7d44f7ce266e
@@ -0,0 +1,2 @@
+*.o
+*-harness
tools/fuzz/x86ie/Makefile new file +51 -0

51 lines not on this page · raw message

tools/fuzz/x86ie/README.md new file +21 -0
new file mode 100644
index 000000000000..d5971f184b18
@@ -0,0 +1,21 @@
+# Building
+
+Dependencies on Ubuntu are:
+make gcc wget screen build-essential libssh-dev flex bison libelf-dev bc
+
+From the root of linux, run:
+
+0. If intending to use AFL:
+   - Install, build, and export AFLPATH for AFL via:
+     `source ./tools/fuzz/x86ie/scripts/install_afl.sh`
+   - Set linux to create core dumps:
+     `sudo bash -c "echo core >/proc/sys/kernel/core_pattern"`
+1. Build the harness via `./tools/fuzz/x86ie/scripts/build.sh`
+   - Set the envvar CC with your desired compiler (afl-gcc if fuzzing with AFL) a la:
+     `CC=$AFLPATH/afl-gcc ./tools/fuzz/x86ie/scripts/build.sh`
+2. Run afl via `./tools/fuzz/x86ie/scripts/run.sh` which requires $AFLPATH set.
+   You must have built the harness with `afl-gcc` to fuzz with afl.
+   - Optionally, set `FUZZDIR` to specify a location other than the default
+    `./fuzz` to store fuzzer status.  Placing this in memory like `/dev/shm`
+    is advised as AFL can be IO bound.
+
tools/fuzz/x86ie/afl-harness.c new file +151 -0

151 lines not on this page · raw message

tools/fuzz/x86ie/common.h new file +87 -0

87 lines not on this page · raw message

tools/fuzz/x86ie/emulator_ops.c new file +58 -0

58 lines not on this page · raw message

tools/fuzz/x86ie/emulator_ops.h new file +134 -0

134 lines not on this page · raw message

tools/fuzz/x86ie/stubs.c new file +59 -0

59 lines not on this page · raw message

tools/fuzz/x86ie/stubs.h new file +52 -0

52 lines not on this page · raw message

Signature · 12 lines
2.17.1




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Ralf Herbrich
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
#2 lore raw

PATCH v4 2/5 Emulate simple x86 instructions in userspace

In reply to Sam Caccavale

Added the minimal subset of code to run afl-harness with a binary file as input. These bytes are used to populate the vcpu structure and then as an instruction stream for the emulator. It does not attempt to handle exceptions and only supports very simple ops.

---

v1 -> v2:
- Removed a number of macros and moved them as static inline functions in emulator_ops.h

v2 -> v3:
- Removed commented out code
- Moved changes to emulator_ops.h into the first patch
- Moved addition of afl-many script to a later patch
- Fixed a spelling mistake

v3 -> v4:
- Stubbed remaining emulator_ops with unimplemented_op macro

---
 tools/fuzz/x86ie/emulator_ops.c | 534 +++++++++++++++++++++++++++++++-
 1 file changed, 533 insertions(+), 1 deletion(-)
tools/fuzz/x86ie/emulator_ops.c +533 -1
index 55ae4e8fbd96..27ba486972e1 100644
@@ -29,17 +29,541 @@
 #include <asm/user_64.h>
 #include <asm/kvm.h>
 
+ulong emul_read_gpr(struct x86_emulate_ctxt *ctxt, unsigned int reg)
+{
+	assert(reg < number_of_gprs(ctxt));
+	return get_state(ctxt)->vcpu.regs[reg];
+}
+
+void emul_write_gpr(struct x86_emulate_ctxt *ctxt, unsigned int reg, ulong val)
+{
+	assert(reg < number_of_gprs(ctxt));
+	get_state(ctxt)->vcpu.regs[reg] = val;
+}
+
+int emul_read_std(struct x86_emulate_ctxt *ctxt,
+		unsigned long addr, void *val,
+		unsigned int bytes,
+		struct x86_exception *fault, bool system)
+{
+	unimplemented_op();
+}
+
+int emul_read_phys(struct x86_emulate_ctxt *ctxt, unsigned long addr,
+		void *val, unsigned int bytes)
+{
+	unimplemented_op();
+}
+
+int emul_write_std(struct x86_emulate_ctxt *ctxt,
+		unsigned long addr, void *val, unsigned int bytes,
+		struct x86_exception *fault, bool system)
+{
+	unimplemented_op();
+}
+
+/* All read ops: */
+
+static int _get_bytes(void *dst, struct state *state, unsigned int bytes,
+		      char *callee)
+{
+	if (state->bytes_consumed + bytes > state->data_available) {
+		fprintf(stderr, "Tried retrieving %d bytes\n", bytes);
+		fprintf(stderr, "%s failed to retrieve bytes for %s.\n",
+			__func__, callee);
+		return X86EMUL_UNHANDLEABLE;
+	}
+
+	memcpy(dst, &state->data[state->bytes_consumed], bytes);
+	return X86EMUL_CONTINUE;
+}
+
+/*
+ * The only function that any x86_emulate_ops should call to retrieve bytes.
+ * See comments in struct state definition for more information.
+ */
+static int get_bytes_and_increment(void *dst, struct state *state,
+				   unsigned int bytes, char *callee)
+{
+	int rc = _get_bytes(dst, state, bytes, callee);
+
+	if (rc == X86EMUL_CONTINUE)
+		state->bytes_consumed += bytes;
+
+	return rc;
+}
+
+/*
+ * This is called by x86_decode_insn to fetch bytes.
+ */
+int emul_fetch(struct x86_emulate_ctxt *ctxt, unsigned long addr, void *val,
+	       unsigned int bytes, struct x86_exception *fault)
+{
+	if (get_bytes_and_increment(val, get_state(ctxt), bytes,
+		"emul_fetch") != X86EMUL_CONTINUE) {
+		return X86EMUL_UNHANDLEABLE;
+	}
+
+	return X86EMUL_CONTINUE;
+}
+
+int emul_read_emulated(struct x86_emulate_ctxt *ctxt,
+		       unsigned long addr, void *val, unsigned int bytes,
+		       struct x86_exception *fault)
+{
+	if (get_bytes_and_increment(val, get_state(ctxt), bytes,
+		"emul_read_emulated") != X86EMUL_CONTINUE) {
+		return X86EMUL_UNHANDLEABLE;
+	}
+
+	return X86EMUL_CONTINUE;
+}
+
+int emul_write_emulated(struct x86_emulate_ctxt *ctxt,
+		   unsigned long addr, const void *val,
+		   unsigned int bytes,
+		   struct x86_exception *fault)
+{
+	return X86EMUL_CONTINUE;
+}
+
+
+int emul_cmpxchg_emulated(struct x86_emulate_ctxt *ctxt,
+			unsigned long addr,
+			const void *old,
+			const void *new,
+			unsigned int bytes,
+			struct x86_exception *fault)
+{
+	unimplemented_op();
+}
+
+void emul_invlpg(struct x86_emulate_ctxt *ctxt, ulong addr)
+{
+	unimplemented_op();
+}
+
+int emul_pio_in_emulated(struct x86_emulate_ctxt *ctxt,
+			int size, unsigned short port, void *val,
+			unsigned int count)
+{
+	unimplemented_op();
+}
+
+int emul_pio_out_emulated(struct x86_emulate_ctxt *ctxt,
+			int size, unsigned short port, const void *val,
+			unsigned int count)
+{
+	unimplemented_op();
+}
+
+bool emul_get_segment(struct x86_emulate_ctxt *ctxt, u16 *selector,
+			struct desc_struct *desc, u32 *base3, int seg)
+{
+	unimplemented_op();
+}
+
+void emul_set_segment(struct x86_emulate_ctxt *ctxt, u16 selector,
+			struct desc_struct *desc, u32 base3, int seg)
+{
+	unimplemented_op();
+}
+
+unsigned long emul_get_cached_segment_base(struct x86_emulate_ctxt *ctxt,
+						int seg)
+{
+	unimplemented_op();
+}
+
+void emul_get_gdt(struct x86_emulate_ctxt *ctxt, struct desc_ptr *dt)
+{
+	unimplemented_op();
+}
+
+void emul_get_idt(struct x86_emulate_ctxt *ctxt, struct desc_ptr *dt)
+{
+	unimplemented_op();
+}
+
+void emul_set_gdt(struct x86_emulate_ctxt *ctxt, struct desc_ptr *dt)
+{
+	unimplemented_op();
+}
+
+void emul_set_idt(struct x86_emulate_ctxt *ctxt, struct desc_ptr *dt)
+{
+	unimplemented_op();
+}
+
+ulong emul_get_cr(struct x86_emulate_ctxt *ctxt, int cr)
+{
+	return get_state(ctxt)->vcpu.cr[cr];
+}
+
+int emul_set_cr(struct x86_emulate_ctxt *ctxt, int cr, ulong val)
+{
+	get_state(ctxt)->vcpu.cr[cr] = val;
+	return 0;
+}
+
+int emul_cpl(struct x86_emulate_ctxt *ctxt)
+{
+	unimplemented_op();
+}
+
+int emul_get_dr(struct x86_emulate_ctxt *ctxt, int dr, ulong *dest)
+{
+	unimplemented_op();
+}
+
+int emul_set_dr(struct x86_emulate_ctxt *ctxt, int dr, ulong value)
+{
+	unimplemented_op();
+}
+
+u64 emul_get_smbase(struct x86_emulate_ctxt *ctxt)
+{
+	unimplemented_op();
+}
+
+void emul_set_smbase(struct x86_emulate_ctxt *ctxt, u64 smbase)
+{
+	unimplemented_op();
+}
+
+int emul_set_msr(struct x86_emulate_ctxt *ctxt, u32 msr_index, u64 data)
+{
+	unimplemented_op();
+}
+
+int emul_get_msr(struct x86_emulate_ctxt *ctxt, u32 msr_index, u64 *pdata)
+{
+	unimplemented_op();
+}
+
+int emul_check_pmc(struct x86_emulate_ctxt *ctxt, u32 pmc)
+{
+	unimplemented_op();
+}
+
+int emul_read_pmc(struct x86_emulate_ctxt *ctxt, u32 pmc, u64 *pdata)
+{
+	unimplemented_op();
+}
+
+void emul_halt(struct x86_emulate_ctxt *ctxt)
+{
+	unimplemented_op();
+}
+
+void emul_wbinvd(struct x86_emulate_ctxt *ctxt)
+{
+	unimplemented_op();
+}
+
+int emul_fix_hypercall(struct x86_emulate_ctxt *ctxt)
+{
+	unimplemented_op();
+}
+
+int emul_intercept(struct x86_emulate_ctxt *ctxt,
+			struct x86_instruction_info *info,
+			enum x86_intercept_stage stage)
+{
+	unimplemented_op();
+}
+
+bool emul_get_cpuid(struct x86_emulate_ctxt *ctxt, u32 *eax, u32 *ebx,
+			u32 *ecx, u32 *edx, bool check_limit)
+{
+	unimplemented_op();
+}
+
+void emul_set_nmi_mask(struct x86_emulate_ctxt *ctxt, bool masked)
+{
+	unimplemented_op();
+}
+
+unsigned int emul_get_hflags(struct x86_emulate_ctxt *ctxt)
+{
+	return get_state(ctxt)->vcpu.rflags;
+}
+
+void emul_set_hflags(struct x86_emulate_ctxt *ctxt, unsigned int hflags)
+{
+	get_state(ctxt)->vcpu.rflags = hflags;
+}
+
+int emul_pre_leave_smm(struct x86_emulate_ctxt *ctxt,
+			const char *smstate)
+{
+	unimplemented_op();
+}
+
+void emul_post_leave_smm(struct x86_emulate_ctxt *ctxt)
+{
+	unimplemented_op();
+}
+
+/* End of emulator ops */
+
+#define EMUL_OP(h) .h = emul_##h
+const struct x86_emulate_ops all_emulator_ops = {
+	EMUL_OP(read_gpr),
+	EMUL_OP(write_gpr),
+	EMUL_OP(read_std),
+	EMUL_OP(read_phys),
+	EMUL_OP(write_std),
+	EMUL_OP(fetch),
+	EMUL_OP(read_emulated),
+	EMUL_OP(write_emulated),
+	EMUL_OP(cmpxchg_emulated),
+	EMUL_OP(invlpg),
+	EMUL_OP(pio_in_emulated),
+	EMUL_OP(pio_out_emulated),
+	EMUL_OP(get_segment),
+	EMUL_OP(set_segment),
+	EMUL_OP(get_cached_segment_base),
+	EMUL_OP(get_gdt),
+	EMUL_OP(get_idt),
+	EMUL_OP(set_gdt),
+	EMUL_OP(set_idt),
+	EMUL_OP(get_cr),
+	EMUL_OP(set_cr),
+	EMUL_OP(cpl),
+	EMUL_OP(get_dr),
+	EMUL_OP(set_dr),
+	EMUL_OP(get_smbase),
+	EMUL_OP(set_smbase),
+	EMUL_OP(set_msr),
+	EMUL_OP(get_msr),
+	EMUL_OP(check_pmc),
+	EMUL_OP(read_pmc),
+	EMUL_OP(halt),
+	EMUL_OP(wbinvd),
+	EMUL_OP(fix_hypercall),
+	EMUL_OP(intercept),
+	EMUL_OP(get_cpuid),
+	EMUL_OP(set_nmi_mask),
+	EMUL_OP(get_hflags),
+	EMUL_OP(set_hflags),
+	EMUL_OP(pre_leave_smm),
+	EMUL_OP(post_leave_smm),
+};
+#undef EMUL_OP
+
+enum {
+	HOOK_read_gpr,
+	HOOK_write_gpr,
+	HOOK_fetch,
+	HOOK_read_emulated,
+	HOOK_write_emulated,
+	HOOK_get_cr,
+	HOOK_set_cr,
+	HOOK_get_hflags,
+	HOOK_set_hflags
+};
+
+/*
+ * Disable an x86_emulate_op if options << HOOK_op is set.
+ *
+ * Expects options to be defined.
+ */
+#define MAYBE_DISABLE_HOOK(h)						\
+	do {								\
+		if (options & (1 << HOOK_##h)) {			\
+			vcpu->ctxt.ops.h = NULL;			\
+			debug("Disabling hook " #h "\n");		\
+		}							\
+	} while (0)
+
+/*
+ * FROM XEN:
+ *
+ * Constrain input to architecturally-possible states where
+ * the emulator relies on these
+ *
+ * In general we want the emulator to be as absolutely robust as
+ * possible; which means that we want to minimize the number of things
+ * it assumes about the input state.  Tesing this means minimizing and
+ * removing as much of the input constraints as possible.
+ *
+ * So we only add constraints that (in general) have been proven to
+ * cause crashes in the emulator.
+ *
+ * For future reference: other constraints which might be necessary at
+ * some point:
+ *
+ * - EFER.LMA => !EFLAGS.NT
+ * - In VM86 mode, force segment...
+ *  - ...access rights to 0xf3
+ *  - ...limits to 0xffff
+ *  - ...bases to below 1Mb, 16-byte aligned
+ *  - ...selectors to (base >> 4)
+ */
+static void sanitize_input(struct state *s)
+{
+	/*
+	 * CR0.PG can't be set if CR0.PE isn't set.  Set is more interesting, so
+	 * set PE if PG is set.
+	 */
+	if (s->vcpu.cr[0] & X86_CR0_PG)
+		s->vcpu.cr[0] |= X86_CR0_PE;
+
+	/* EFLAGS.VM not available in long mode */
+	if (s->ctxt.mode == X86EMUL_MODE_PROT64)
+		s->vcpu.rflags &= ~X86_EFLAGS_VM;
+
+	/* EFLAGS.VM implies 16-bit mode */
+	if (s->vcpu.rflags & X86_EFLAGS_VM) {
+		s->vcpu.segments[x86_seg_cs].db = 0;
+		s->vcpu.segments[x86_seg_ss].db = 0;
+	}
+}
+
 void initialize_emulator(struct state *state)
 {
+	reset_emulator(state);
+	state->ctxt.ops = &all_emulator_ops;
+
+	/* See also sanitize_input, some hooks can't be disabled. */
+	// MAYBE_DISABLE_HOOK(read_gpr);
+
+	sanitize_input(state);
+}
+
+static const char *const x86emul_mode_string[] = {
+	[X86EMUL_MODE_REAL] = "X86EMUL_MODE_REAL",
+	[X86EMUL_MODE_VM86] = "X86EMUL_MODE_VM86",
+	[X86EMUL_MODE_PROT16] = "X86EMUL_MODE_PROT16",
+	[X86EMUL_MODE_PROT32] = "X86EMUL_MODE_PROT32",
+	[X86EMUL_MODE_PROT64] = "X86EMUL_MODE_PROT64",
+};
+
+static void dump_state_after(const char *desc, struct state *state)
+{
+	debug(" -- State after %s --\n", desc);
+	debug("mode: %s\n", x86emul_mode_string[state->ctxt.mode]);
+	debug(" cr0: %lx\n", state->vcpu.cr[0]);
+	debug(" cr3: %lx\n", state->vcpu.cr[3]);
+	debug(" cr4: %lx\n", state->vcpu.cr[4]);
+
+	debug("Decode _eip: %lu\n", state->ctxt._eip);
+	debug("Emulate eip: %lu\n", state->ctxt.eip);
+
+	debug("\n");
 }
 
+static void init_emulate_ctxt(struct state *state)
+{
+	struct x86_emulate_ctxt *ctxt = &state->ctxt;
+
+	ctxt->eflags = ctxt->ops->get_hflags(ctxt);
+	ctxt->tf = (ctxt->eflags & X86_EFLAGS_TF) != 0;
+
+	ctxt->mode = X86EMUL_MODE_PROT64; // TODO: eventually vary this
+
+	init_decode_cache(ctxt);
+}
+
+
 int step_emulator(struct state *state)
 {
-	return 0;
+	int rc;
+	unsigned long prev_eip = state->ctxt._eip;
+	unsigned long emul_offset;
+	int decode_size = state->data_available - state->bytes_consumed;
+
+	/*
+	 * This is annoing to have to explain the reasoning behind:
+	 * ._eip is incremented by x86_decode_insn.  It will be > .eip between
+	 * decoding and emulating.
+	 * .eip is incremented by x86_emulate_insn.  It may be incremented
+	 * beyond the length of instruction emulated E.G. if a jump is taken.
+	 *
+	 * If these are out of sync before emulating, then something is
+	 * horribly wrong with the harness.
+	 */
+	assert(state->ctxt.eip == state->ctxt._eip);
+
+	if (decode_size <= 0) {
+		debug("Out of instructions\n");
+		return X86EMUL_UNHANDLEABLE;
+	}
+
+	init_emulate_ctxt(state);
+	state->ctxt.interruptibility = 0;
+	state->ctxt.have_exception = false;
+	state->ctxt.exception.vector = -1;
+	state->ctxt.perm_ok = false;
+	state->ctxt.ud = 0; // (emulation_type(0) & EMULTYPE_TRAP_UD);
+
+	/*
+	 * When decoding with NULL, 0, the emulator will use the emul_fetch
+	 * op which will almost always try to grab 15 bytes which may be more
+	 * than are left in the stream.
+	 *
+	 * Calling x86_decode_insn from a buffer with a length causes it to
+	 * directly memcpy `insn_len` bytes into the ctxt structure's 15 byte
+	 * long buffer without any length check.
+	 *
+	 * We must specify a size to prevent the emulator from fetching more
+	 * bytes than are left, and limit the size to <= 15 so that memcpy
+	 * does not overflow.
+	 */
+	if (decode_size > 15)
+		decode_size = 15;
+
+	rc = x86_decode_insn(&state->ctxt,
+		&state->data[state->bytes_consumed], decode_size);
+	assert(state->ctxt._eip - prev_eip > 0); // Only move forward.
+	state->bytes_consumed += state->ctxt._eip - prev_eip;
+
+	debug("Decode result: %d\n", rc);
+	if (rc != X86EMUL_CONTINUE)
+		return rc;
+
+	emul_offset = state->ctxt._eip - state->ctxt.eip;
+	debug("Instruction: ");
+	print_n_bytes(&state->data[state->bytes_consumed - emul_offset],
+		      emul_offset);
+
+	state->ctxt.exception.address = state->vcpu.cr[2];
+
+	// This is extraneous but explicit due to the above assert
+	prev_eip = state->ctxt.eip;
+	rc = x86_emulate_insn(&state->ctxt);
+	debug("Emulation result: %d\n", rc);
+	dump_state_after("emulating", state);
+
+	if (rc == -1) {
+		return rc;
+	} else if (state->ctxt.have_exception) {
+		fprintf(stderr, "Emulator propagated exception: { ");
+		fprintf(stderr, "vector: %d, ", state->ctxt.exception.vector);
+		fprintf(stderr, "error code: %d }\n",
+			state->ctxt.exception.error_code);
+		rc = X86EMUL_UNHANDLEABLE;
+	} else if (prev_eip == state->ctxt.eip) {
+		fprintf(stderr, "ctxt.eip not advanced.\n");
+		rc = X86EMUL_UNHANDLEABLE;
+	}
+
+	if (state->bytes_consumed == state->data_available)
+		debug("emulator is done\n");
+
+	return rc;
 }
 
 int emulate_until_complete(struct state *state)
 {
+	int count = 0;
+
+	do {
+		count++;
+	} while (step_emulator(state) == X86EMUL_CONTINUE);
+
+	debug("Emulated %d instructions\n", count);
 	return 0;
 }
 
@@ -51,8 +575,16 @@ struct state *create_emulator(void)
 
 void reset_emulator(struct state *state)
 {
+	unsigned char *data = state->data;
+	size_t data_available = state->data_available;
+
+	memset(state, 0, sizeof(struct state));
+
+	state->data = data;
+	state->data_available = data_available;
 }
 
 void free_emulator(struct state *state)
 {
+	free(state);
 }
Signature · 12 lines
2.17.1




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Ralf Herbrich
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
#3 lore raw

PATCH v4 3/5 Demonstrating unit testing via simple-harness

In reply to Sam Caccavale

Simple-harness.c uses inline asm support to generate asm and then has the emulator emulate this code. This may be useful as a form of testing for the emulator.

---

v1 -> v2:
- Accidentally changed the example

V2 -> v3:
- Reverted the example back to the more useful original one

---
 tools/fuzz/x86ie/Makefile         |  7 +++--
 tools/fuzz/x86ie/simple-harness.c | 49 +++++++++++++++++++++++++++++++
 2 files changed, 54 insertions(+), 2 deletions(-)
 create mode 100644 tools/fuzz/x86ie/simple-harness.c
tools/fuzz/x86ie/Makefile +5 -2
index 07a784519100..2724b0cb387f 100644
@@ -44,8 +44,11 @@ LOCAL_OBJS := emulator_ops.o stubs.o
 afl-harness: afl-harness.o $(LOCAL_OBJS) $(KERNEL_OBJS)
 	@$(CC) -v $(KBUILD_CFLAGS) $(LOCAL_OBJS) $(KERNEL_OBJS) $< $(INCLUDES) -Istubs.h -o $@ -no-pie
 
-all: afl-harness
+simple-harness: simple-harness.o $(LOCAL_OBJS) $(KERNEL_OBJS)
+	@$(CC) -v $(KBUILD_CFLAGS) $(LOCAL_OBJS) $(KERNEL_OBJS) $< $(INCLUDES) -Istubs.h -o $@ -no-pie
+
+all: afl-harness simple-harness
 
 .PHONY: clean
 clean:
-	$(RM) -r *.o afl-harness
+	$(RM) -r *.o afl-harness simple-harness
tools/fuzz/x86ie/simple-harness.c new file +49 -0

49 lines not on this page · raw message

Signature · 12 lines
2.17.1




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Ralf Herbrich
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
#4 lore raw

PATCH v4 4/5 Added build and install scripts

In reply to Sam Caccavale

install_afl.sh installs AFL locally and emits AFLPATH, build.sh, and run.sh build and run respectively

---

v1 -> v2:
- Introduced this patch

v2 -> v3:
- Moved non-essential development scripts to a later patch

v3 -> v4:
- Building checks for existing .config and no longer overwrites it
- Removed extraneous forcing of some config options
- Renamed afl-many to afl-many.sh
- Added a timeout option to afl-many.sh
- Fixed an incorrect path in afl-many.sh

---
 tools/fuzz/x86ie/scripts/afl-many.sh    | 31 ++++++++++++++++++++++
 tools/fuzz/x86ie/scripts/build.sh       | 34 +++++++++++++++++++++++++
 tools/fuzz/x86ie/scripts/install_afl.sh | 17 +++++++++++++
 tools/fuzz/x86ie/scripts/run.sh         | 10 ++++++++
 4 files changed, 92 insertions(+)
 create mode 100755 tools/fuzz/x86ie/scripts/afl-many.sh
 create mode 100755 tools/fuzz/x86ie/scripts/build.sh
 create mode 100755 tools/fuzz/x86ie/scripts/install_afl.sh
 create mode 100755 tools/fuzz/x86ie/scripts/run.sh
tools/fuzz/x86ie/scripts/afl-many.sh new file +31 -0
new file mode 100755
index 000000000000..e56923ae16ff
@@ -0,0 +1,31 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+# This is for running AFL over NPROC or `nproc` cores with normal AFL options ex:
+# ./tools/fuzz/x86ie/scripts/afl-many.sh -m 22000000000 -i $FUZZDIR/in -o $FUZZDIR/out tools/fuzz/x86ie/afl-harness @@
+
+export AFL_NO_AFFINITY=1
+
+while [ -z "$sync_dir" ]; do
+  while getopts ":o:" opt; do
+    case "${opt}" in
+      o)
+        sync_dir="${OPTARG}"
+        ;;
+      *)
+        ;;
+    esac
+  done
+  ((OPTIND++))
+  [ $OPTIND -gt $# ] && break
+done
+
+# AFL/linux do some weird stuff with core affinity and will often run
+# N processes over < N virtual cores.  In order to avoid that, we taskset
+# each process to its own core.
+for i in $(seq 1 $(( ${NPROC:-$(nproc)} - 1)) ); do
+    taskset -c "$i" $AFLPATH/afl-fuzz -S "slave$i" $@ >/dev/null 2>&1 &
+done
+taskset -c 0 $AFLPATH/afl-fuzz -M master $@ >/dev/null 2>&1 &
+
+${TIMEOUT:+timeout -sKILL $TIMEOUT} watch -n1 "echo \"Executing '$AFLPATH/afl-fuzz $@' on ${NPROC:-$(nproc)} cores.\" && $AFLPATH/afl-whatsup -s ${sync_dir}"
+pkill afl-fuzz
tools/fuzz/x86ie/scripts/build.sh new file +34 -0
new file mode 100755
index 000000000000..5e0eab8ad721
@@ -0,0 +1,34 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+# Run from root of linux via `./tools/fuzz/x86ie/scripts/build.sh`
+
+kernel_objects="arch/x86/kvm/emulate.o arch/x86/lib/retpoline.o lib/find_bit.o"
+
+disable() { sed -i -r "/\b$1\b/c\# $1" .config; }
+enable() { sed -i -r "/\b$1\b/c\\$1=y" .config; }
+
+if [ ! -f .config ]; then
+  make ${CC:+ "CC=$CC"} defconfig
+fi
+
+# enable "CONFIG_DEBUG_INFO"
+
+yes ' ' | make ${CC:+ "CC=$CC"} ${DEBUG:+ "DEBUG=1"} $kernel_objects
+
+omit_arg () { args=$(echo "$args" | sed "s/ $1//g"); }
+add_arg () { args+=" $1"; }
+
+rebuild () {
+  args="$(head -1 $(dirname $1)/.$(basename $1).cmd | sed -e 's/.*:= //g')"
+  omit_arg "-mcmodel=kernel"
+  omit_arg "-mpreferred-stack-boundary=3"
+  add_arg "-fsanitize=address"
+  echo -e "Rebuilding $1 with \n$args"
+  eval "$args"
+}
+
+for object in $kernel_objects; do
+  rebuild $object
+done
+
+make ${CC:+ "CC=$CC"} ${DEBUG:+ "DEBUG=1"} tools/fuzz
tools/fuzz/x86ie/scripts/install_afl.sh new file +17 -0
new file mode 100755
index 000000000000..3bdbdf2a040b
@@ -0,0 +1,17 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+# Can be run where ever, but usually run from linux root:
+# `source ./tools/fuzz/x86ie/scripts/install_afl.sh`
+# (must be sourced to get the AFLPATH envvar, otherwise set manually)
+
+wget http://lcamtuf.coredump.cx/afl/releases/afl-latest.tgz
+mkdir -p afl
+tar xzf afl-latest.tgz -C afl --strip-components 1
+
+pushd afl
+set AFL_USE_ASAN
+make clean all
+export AFLPATH="$(pwd)"
+popd
+
+sudo bash -c "echo core >/proc/sys/kernel/core_pattern"
tools/fuzz/x86ie/scripts/run.sh new file +10 -0
new file mode 100755
index 000000000000..348c9c41021a
@@ -0,0 +1,10 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+
+FUZZDIR="${FUZZDIR:-$(pwd)/fuzz}"
+
+mkdir -p $FUZZDIR/in
+cp tools/fuzz/x86ie/rand_sample.bin $FUZZDIR/in
+mkdir -p $FUZZDIR/out
+
+screen bash -c "ulimit -Sv $[21999999999 << 10]; ${TIMEOUT:+TIMEOUT=$TIMEOUT} ./tools/fuzz/x86ie/scripts/afl-many.sh -m 22000000000 -i $FUZZDIR/in -o $FUZZDIR/out tools/fuzz/x86ie/afl-harness @@; exit \$?;"
Signature · 12 lines
2.17.1




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Ralf Herbrich
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
#5 lore raw

PATCH v4 5/5 Development scripts for crash triage and deploy

In reply to Sam Caccavale

Not meant for upstream consumption.

---

v2 -> v3:
- Introduced this patch as a place for non-essential dev scripts

v3 -> v4:
- Made deploy_remote.sh use a 1d timeout
- Removed core dump setup from install_afl.sh
- Added core dump setup into deploy_remote.sh

---
 tools/fuzz/x86ie/scripts/bin.sh               | 49 +++++++++++++++++++
 tools/fuzz/x86ie/scripts/coalesce.sh          |  5 ++
 tools/fuzz/x86ie/scripts/deploy.sh            |  9 ++++
 tools/fuzz/x86ie/scripts/deploy_remote.sh     | 10 ++++
 tools/fuzz/x86ie/scripts/gen_output.sh        | 11 +++++
 tools/fuzz/x86ie/scripts/install_afl.sh       |  2 -
 .../fuzz/x86ie/scripts/install_deps_ubuntu.sh |  5 ++
 tools/fuzz/x86ie/scripts/rebuild.sh           |  6 +++
 tools/fuzz/x86ie/scripts/summarize.sh         |  9 ++++
 9 files changed, 104 insertions(+), 2 deletions(-)
 create mode 100755 tools/fuzz/x86ie/scripts/bin.sh
 create mode 100755 tools/fuzz/x86ie/scripts/coalesce.sh
 create mode 100644 tools/fuzz/x86ie/scripts/deploy.sh
 create mode 100755 tools/fuzz/x86ie/scripts/deploy_remote.sh
 create mode 100755 tools/fuzz/x86ie/scripts/gen_output.sh
 create mode 100755 tools/fuzz/x86ie/scripts/install_deps_ubuntu.sh
 create mode 100755 tools/fuzz/x86ie/scripts/rebuild.sh
 create mode 100755 tools/fuzz/x86ie/scripts/summarize.sh
tools/fuzz/x86ie/scripts/bin.sh new file +49 -0
new file mode 100755
index 000000000000..6383a883ff33
@@ -0,0 +1,49 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+
+if [ "$#" -lt 3 ]; then
+  echo "Usage: './bin path/to/afl-harness path/to/afl_crash [path/to/linux/src/root]'"
+  exit
+fi
+
+export AFL_HARNESS="$1"
+export LINUX_SRC="$3"
+
+diagnose_segfault() {
+  SOURCE=$(gdb -batch -ex r -ex 'bt 2' --args $@ 2>&1 | grep -Po '#1.* \K([^ ]+:[0-9]+)');
+  IFS=: read FILE LINE <<< "$SOURCE"
+
+  OP="$(sed -n "${LINE}p" "$LINUX_SRC/$FILE" 2>/dev/null)"
+  if [ $? -ne 0 ]; then
+    OP="$(sed -n "${LINE}p" "$LINUX_SRC/tools/fuzz/x86_instruction_emulation/$FILE" 2>/dev/null)"
+  fi
+
+  OP="$(echo $OP | grep -Po 'ops->\K([^(]+)')"
+  if [ -z "$OP" ]; then
+    echo "SEGV: unknown, in $FILE:$LINE"
+  else
+    echo "Expected: segfaulting on emulator->$OP"
+  fi
+}
+export -f diagnose_segfault
+
+bin() {
+  OUTPUT=$(bash -c "timeout 1s $AFL_HARNESS $1 2>&1" 2>&1)
+  RETVAL=$?
+
+  echo "$OUTPUT"
+  if [ $RETVAL -eq 0 ]; then
+    echo "Terminated successfully"
+  elif [ $RETVAL -eq 124 ]; then
+    echo "Unknown: killed due to timeout.  Loop likely."
+  elif echo "$OUTPUT" | grep -q "SEGV"; then
+    echo "$(diagnose_segfault $AFL_HARNESS $1)"
+  elif echo "$OUTPUT" | grep -q "FPE"; then
+    echo "Expected: floating point exception."
+  else
+    echo "Unknown cause of crash."
+  fi
+}
+export -f bin
+
+echo "$(bin $2 2>&1)"
tools/fuzz/x86ie/scripts/coalesce.sh new file +5 -0
new file mode 100755
index 000000000000..b15d583c2c32
@@ -0,0 +1,5 @@
+#!/bin/bash
+
+mkdir -p all
+rm -rf all/*
+find . -type f -wholename '*crashes/id*' | parallel 'cp {} ./all/$(basename $(dirname {//})):{/}'
tools/fuzz/x86ie/scripts/deploy.sh new file +9 -0
new file mode 100644
index 000000000000..f95c3aa2b5b5
@@ -0,0 +1,9 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+
+REMOTE=$1
+DSTDIR=/dev/shm
+
+rsync -av $(pwd) $REMOTE:$DSTDIR
+
+ssh $REMOTE "cd $DSTDIR/$(basename $(pwd)); bash -s tools/fuzz/x86_instruction_emulation/scripts/deploy_remote.sh"
tools/fuzz/x86ie/scripts/deploy_remote.sh new file +10 -0
new file mode 100755
index 000000000000..a903294e145a
@@ -0,0 +1,10 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+
+SCRIPTDIR=$(pwd)/tools/fuzz/x86ie/scripts
+
+$SCRIPTDIR/install_deps_ubuntu.sh
+source $SCRIPTDIR/install_afl.sh
+CC=$AFLPATH/afl-gcc $SCRIPTDIR/build.sh
+sudo bash -c "echo core >/proc/sys/kernel/core_pattern"
+TIMEOUT="${TIMEOUT:-1d}" FUZZDIR="${FUZZDIR:-$(pwd)/fuzz}" $SCRIPTDIR/run.sh
tools/fuzz/x86ie/scripts/gen_output.sh new file +11 -0
new file mode 100755
index 000000000000..6c0707eb6d08
@@ -0,0 +1,11 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+
+if [ "$#" -lt 3 ]; then
+  echo "Usage: '$0 path/to/afl-harness path/to/afl_crash_dir path/to/linux/src/root'"
+  exit
+fi
+
+mkdir -p output
+rm -rf output/*
+find $2 -type f | parallel ./bin.sh $1 {} $3 '>' ./output/{/}.out
tools/fuzz/x86ie/scripts/install_afl.sh +0 -2
index 3bdbdf2a040b..e05e9942cc19 100755
@@ -13,5 +13,3 @@ set AFL_USE_ASAN
 make clean all
 export AFLPATH="$(pwd)"
 popd
-
-sudo bash -c "echo core >/proc/sys/kernel/core_pattern"
tools/fuzz/x86ie/scripts/install_deps_ubuntu.sh new file +5 -0
new file mode 100755
index 000000000000..5525bc8b659c
@@ -0,0 +1,5 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+
+sudo apt update
+sudo apt install -y make gcc wget screen build-essential libssh-dev flex bison libelf-dev bc
tools/fuzz/x86ie/scripts/rebuild.sh new file +6 -0
new file mode 100755
index 000000000000..809a4551cb0c
@@ -0,0 +1,6 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+
+make clean
+make tools/fuzz_clean
+FUZZDIR="./fuzz" ./tools/fuzz/x86ie/scripts/build.sh
tools/fuzz/x86ie/scripts/summarize.sh new file +9 -0
new file mode 100755
index 000000000000..27761f283ee3
@@ -0,0 +1,9 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+
+if [ "$#" -lt 1 ]; then
+  echo "Usage: '$0 path/to/output/dir'"
+  exit
+fi
+
+time bash -c "find $1 -type f -exec tail -n 1 {} \; | sort | uniq -c | sort -rn"
Signature · 12 lines
2.17.1




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Ralf Herbrich
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
#6 lore raw
In reply to Sam Caccavale
Quoted 96 lines · Sam Caccavale

Dear all,

This series aims to provide an entrypoint for, and fuzz KVM's x86 instruction emulator from userspace. It mirrors Xen's application of the AFL fuzzer to it's instruction emulator in the hopes of discovering vulnerabilities. Since this entrypoint also allows arbitrary execution of the emulators code from userspace, it may also be useful for testing.

The current 4 patches build the emulator and 2 harnesses: simple-harness is an example of unit testing; afl-harness is a frontend for the AFL fuzzer. The fifth patch contains useful scripts for development but is not intended for usptream consumption.

Patches
=======

- 01: Builds and links afl-harness with the required kernel objects.
- 02: Introduces the minimal set of emulator operations and supporting code to emulate simple instructions.
- 03: Demonstrates simple-harness as a unit test.
- 04: Adds scripts for install and building.
- 05: Useful scripts for development

Issues
=======

Currently, fuzzing results in a large amount of FPU related crashes. Xen's fuzzing efforts had this issue too. Their (temporary?) solution was to disable FPU exceptions after every instruction iteration? Some solution is desired for this project.

Changelog
=======

v1 -> v2:
- Moved -O0 to ifdef DEBUG
- Building with ASAN by default
- Removed a number of macros from emulator_ops.c and moved them as static inline functions in emulator_ops.h
- Accidentally changed the example in simple-harness (reverted in v3)
- Introduced patch 4 for scripts

v2 -> v3:
- Removed a workaround for printf smashing the stack when compiled with -mcmodel=kernel, and stopped compiling with -mcmodel=kernel
- Added a null check for malloc's return value
- Moved more macros from emulator_ops.c into emulator_ops.h as static inline functions
- Removed commented out code
- Moved changes to emulator_ops.h into the first patch
- Moved addition of afl-many script to the script patch
- Fixed spelling mistakes in documentation
- Reverted the simple-harness example back to the more useful original one
- Moved non-essential development scripts from patch 4 to new patch 5

v3 -> v4:
- Stubbed out all unimplemented emulator_ops with a unimplemented_op macro
- Setting FAIL_ON_UNIMPLEMENTED_OP on compile decides whether calling these is treated as a crash or ignored
- Moved setting up core dumps out of the default build/install path and detailed this change in the README
- Added a .sh extention to afl-many
- Added an optional timeout to afl-many.sh and made deploy_remote.sh use it
- Building no longer creates a new .config each time and does not force any config options
- Fixed a path bug in afl-many.sh

Any comments/suggestions are greatly appreciated.

Best,
Sam Caccavale

Sam Caccavale (5):
Build target for emulate.o as a userspace binary Emulate simple x86 instructions in userspace Demonstrating unit testing via simple-harness Added build and install scripts Development scripts for crash triage and deploy

  tools/Makefile                                |   9 +
  tools/fuzz/x86ie/.gitignore                   |   2 +
  tools/fuzz/x86ie/Makefile                     |  54 ++
  tools/fuzz/x86ie/README.md                    |  21 +
  tools/fuzz/x86ie/afl-harness.c                | 151 +++++
  tools/fuzz/x86ie/common.h                     |  87 +++
  tools/fuzz/x86ie/emulator_ops.c               | 590 ++++++++++++++++++
  tools/fuzz/x86ie/emulator_ops.h               | 134 ++++
  tools/fuzz/x86ie/scripts/afl-many.sh          |  31 +
  tools/fuzz/x86ie/scripts/bin.sh               |  49 ++
  tools/fuzz/x86ie/scripts/build.sh             |  34 +
  tools/fuzz/x86ie/scripts/coalesce.sh          |   5 +
  tools/fuzz/x86ie/scripts/deploy.sh            |   9 +
  tools/fuzz/x86ie/scripts/deploy_remote.sh     |  10 +
  tools/fuzz/x86ie/scripts/gen_output.sh        |  11 +
Quoted 8 lines
  tools/fuzz/x86ie/scripts/install_afl.sh       |  15 +
  .../fuzz/x86ie/scripts/install_deps_ubuntu.sh |   5 +
  tools/fuzz/x86ie/scripts/rebuild.sh           |   6 +
  tools/fuzz/x86ie/scripts/run.sh               |  10 +
  tools/fuzz/x86ie/scripts/summarize.sh         |   9 +
  tools/fuzz/x86ie/simple-harness.c             |  49 ++
  tools/fuzz/x86ie/stubs.c                      |  59 ++
  tools/fuzz/x86ie/stubs.h                      |  52 ++

Sorry I didn't realize it before. Isn't that missing a patch to the MAINTAINERS file?

Alex

maintainer
#7 lore raw
Quoted 120 lines · Alexander Graf
Quoted 108 lines · Sam Caccavale

Dear all,

This series aims to provide an entrypoint for, and fuzz KVM's x86 instruction
emulator from userspace.  It mirrors Xen's application of the AFL fuzzer to
it's instruction emulator in the hopes of discovering vulnerabilities. Since this entrypoint also allows arbitrary execution of the emulators code
from userspace, it may also be useful for testing.

The current 4 patches build the emulator and 2 harnesses:
simple-harness is
an example of unit testing; afl-harness is a frontend for the AFL fuzzer. The fifth patch contains useful scripts for development but is not intended
for usptream consumption.

Patches
=======

- 01: Builds and links afl-harness with the required kernel objects.
- 02: Introduces the minimal set of emulator operations and supporting code
to emulate simple instructions.
- 03: Demonstrates simple-harness as a unit test.
- 04: Adds scripts for install and building.
- 05: Useful scripts for development

Issues
=======

Currently, fuzzing results in a large amount of FPU related crashes. Xen's
fuzzing efforts had this issue too.  Their (temporary?) solution was to disable FPU exceptions after every instruction iteration?  Some solution is desired for this project.

Changelog
=======

v1 -> v2:
  - Moved -O0 to ifdef DEBUG
  - Building with ASAN by default
  - Removed a number of macros from emulator_ops.c and moved them as static inline functions in emulator_ops.h
  - Accidentally changed the example in simple-harness (reverted in v3)
  - Introduced patch 4 for scripts

v2 -> v3:
  - Removed a workaround for printf smashing the stack when compiled with -mcmodel=kernel, and stopped compiling with -mcmodel=kernel
  - Added a null check for malloc's return value
  - Moved more macros from emulator_ops.c into emulator_ops.h as static inline functions
  - Removed commented out code
  - Moved changes to emulator_ops.h into the first patch
  - Moved addition of afl-many script to the script patch
  - Fixed spelling mistakes in documentation
  - Reverted the simple-harness example back to the more useful original one
  - Moved non-essential development scripts from patch 4 to new patch 5

v3 -> v4:
  - Stubbed out all unimplemented emulator_ops with a unimplemented_op macro
  - Setting FAIL_ON_UNIMPLEMENTED_OP on compile decides whether calling these
    is treated as a crash or ignored
  - Moved setting up core dumps out of the default build/install path and detailed this change in the README
  - Added a .sh extention to afl-many
  - Added an optional timeout to afl-many.sh and made deploy_remote.sh use it
  - Building no longer creates a new .config each time and does not force any
    config options
  - Fixed a path bug in afl-many.sh

Any comments/suggestions are greatly appreciated.

Best,
Sam Caccavale

Sam Caccavale (5):
   Build target for emulate.o as a userspace binary Emulate simple x86 instructions in userspace Demonstrating unit testing via simple-harness Added build and install scripts Development scripts for crash triage and deploy

  tools/Makefile                                |   9 + tools/fuzz/x86ie/.gitignore                   |   2 + tools/fuzz/x86ie/Makefile                     |  54 ++ tools/fuzz/x86ie/README.md                    |  21 + tools/fuzz/x86ie/afl-harness.c                | 151 +++++ tools/fuzz/x86ie/common.h                     |  87 +++ tools/fuzz/x86ie/emulator_ops.c               | 590 ++++++++++++++++++ tools/fuzz/x86ie/emulator_ops.h               | 134 ++++ tools/fuzz/x86ie/scripts/afl-many.sh          |  31 + tools/fuzz/x86ie/scripts/bin.sh               |  49 ++ tools/fuzz/x86ie/scripts/build.sh             |  34 + tools/fuzz/x86ie/scripts/coalesce.sh          |   5 + tools/fuzz/x86ie/scripts/deploy.sh            |   9 + tools/fuzz/x86ie/scripts/deploy_remote.sh     |  10 + tools/fuzz/x86ie/scripts/gen_output.sh        |  11 +

Quoted 8 lines

  tools/fuzz/x86ie/scripts/install_afl.sh       |  15 + .../fuzz/x86ie/scripts/install_deps_ubuntu.sh |   5 + tools/fuzz/x86ie/scripts/rebuild.sh           |   6 + tools/fuzz/x86ie/scripts/run.sh               |  10 + tools/fuzz/x86ie/scripts/summarize.sh         |   9 + tools/fuzz/x86ie/simple-harness.c             |  49 ++ tools/fuzz/x86ie/stubs.c                      |  59 ++ tools/fuzz/x86ie/stubs.h                      |  52 ++

Quoted 2 lines

Sorry I didn't realize it before. Isn't that missing a patch to the MAINTAINERS file?

Yeah, and the directory should probably be tools/fuzz/kvm_emulate so as not to puzzle people. Also:

- let's limit the scripts to the minimum, i.e. only the run script which should be something like

#!/bin/bash
# SPDX-License-Identifier: GPL-2.0+

FUZZDIR="${FUZZDIR:-$(pwd)/fuzz}"

mkdir -p $FUZZDIR/in
cp tools/fuzz/kvm_emulate/rand_sample.bin $FUZZDIR/in mkdir -p $FUZZDIR/out

${TIMEOUT:+TIMEOUT=$TIMEOUT} ${AFL_FUZZ-afl-fuzz} "$@" \
-i $FUZZDIR/in -o $FUZZDIR/out tools/fuzz/kvm_emulate/afl-harness @@

where people can substitute afl-many.sh or add their own options using the AFL_FUZZ variable or the command line. Likewise for screen.

- the build should be just "make -C tools/fuzz/kvm_emulate" and it should just work. Feel free to steal the Makefile magic from other tools/ directories.

- finally, rand_sample.bin is missing.

Otherwise, it looks very nice.

Paolo

#8 lore raw
Quoted 124 lines · Paolo Bonzini
Quoted 120 lines · Alexander Graf
Quoted 116 lines · Sam Caccavale

Dear all,

This series aims to provide an entrypoint for, and fuzz KVM's x86 instruction
emulator from userspace. It mirrors Xen's application of the AFL fuzzer to
it's instruction emulator in the hopes of discovering vulnerabilities. Since this entrypoint also allows arbitrary execution of the emulators code
from userspace, it may also be useful for testing.

The current 4 patches build the emulator and 2 harnesses:
simple-harness is
an example of unit testing; afl-harness is a frontend for the AFL fuzzer. The fifth patch contains useful scripts for development but is not intended
for usptream consumption.

Patches
=======

- 01: Builds and links afl-harness with the required kernel objects.
- 02: Introduces the minimal set of emulator operations and supporting code
to emulate simple instructions.
- 03: Demonstrates simple-harness as a unit test.
- 04: Adds scripts for install and building.
- 05: Useful scripts for development

Issues
=======

Currently, fuzzing results in a large amount of FPU related crashes. Xen's
fuzzing efforts had this issue too. Their (temporary?) solution was to disable FPU exceptions after every instruction iteration? Some solution is desired for this project.

Changelog
=======

v1 -> v2:
- Moved -O0 to ifdef DEBUG
- Building with ASAN by default
- Removed a number of macros from emulator_ops.c and moved them as static inline functions in emulator_ops.h
- Accidentally changed the example in simple-harness (reverted in v3)
- Introduced patch 4 for scripts

v2 -> v3:
- Removed a workaround for printf smashing the stack when compiled with -mcmodel=kernel, and stopped compiling with -mcmodel=kernel
- Added a null check for malloc's return value
- Moved more macros from emulator_ops.c into emulator_ops.h as static inline functions
- Removed commented out code
- Moved changes to emulator_ops.h into the first patch
- Moved addition of afl-many script to the script patch
- Fixed spelling mistakes in documentation
- Reverted the simple-harness example back to the more useful
original one
- Moved non-essential development scripts from patch 4 to new patch 5

v3 -> v4:
- Stubbed out all unimplemented emulator_ops with a unimplemented_op
macro
- Setting FAIL_ON_UNIMPLEMENTED_OP on compile decides whether
calling these
is treated as a crash or ignored
- Moved setting up core dumps out of the default build/install path and detailed this change in the README
- Added a .sh extention to afl-many
- Added an optional timeout to afl-many.sh and made deploy_remote.sh
use it
- Building no longer creates a new .config each time and does not
force any
config options
- Fixed a path bug in afl-many.sh

Any comments/suggestions are greatly appreciated.

Best,
Sam Caccavale

Sam Caccavale (5):
Build target for emulate.o as a userspace binary Emulate simple x86 instructions in userspace Demonstrating unit testing via simple-harness Added build and install scripts Development scripts for crash triage and deploy

  tools/Makefile                                |   9 +
  tools/fuzz/x86ie/.gitignore                   |   2 +
  tools/fuzz/x86ie/Makefile                     |  54 ++
  tools/fuzz/x86ie/README.md                    |  21 +
  tools/fuzz/x86ie/afl-harness.c                | 151 +++++
  tools/fuzz/x86ie/common.h                     |  87 +++
  tools/fuzz/x86ie/emulator_ops.c               | 590 ++++++++++++++++++
  tools/fuzz/x86ie/emulator_ops.h               | 134 ++++
  tools/fuzz/x86ie/scripts/afl-many.sh          |  31 +
  tools/fuzz/x86ie/scripts/bin.sh               |  49 ++
  tools/fuzz/x86ie/scripts/build.sh             |  34 +
  tools/fuzz/x86ie/scripts/coalesce.sh          |   5 +
  tools/fuzz/x86ie/scripts/deploy.sh            |   9 +
  tools/fuzz/x86ie/scripts/deploy_remote.sh     |  10 +
  tools/fuzz/x86ie/scripts/gen_output.sh        |  11 +
  tools/fuzz/x86ie/scripts/install_afl.sh       |  15 +
  .../fuzz/x86ie/scripts/install_deps_ubuntu.sh |   5 +
  tools/fuzz/x86ie/scripts/rebuild.sh           |   6 +
  tools/fuzz/x86ie/scripts/run.sh               |  10 +
  tools/fuzz/x86ie/scripts/summarize.sh         |   9 +
  tools/fuzz/x86ie/simple-harness.c             |  49 ++
  tools/fuzz/x86ie/stubs.c                      |  59 ++
  tools/fuzz/x86ie/stubs.h                      |  52 ++
Quoted 2 lines
Quoted 2 lines

Sorry I didn't realize it before. Isn't that missing a patch to the MAINTAINERS file?

It is, I will add that.

Quoted 15 lines

Yeah, and the directory should probably be tools/fuzz/kvm_emulate so as not to puzzle people. Also:

- let's limit the scripts to the minimum, i.e. only the run script which should be something like

#!/bin/bash
# SPDX-License-Identifier: GPL-2.0+

FUZZDIR="${FUZZDIR:-$(pwd)/fuzz}"

mkdir -p $FUZZDIR/in
cp tools/fuzz/kvm_emulate/rand_sample.bin $FUZZDIR/in mkdir -p $FUZZDIR/out

Quoted 5 lines

${TIMEOUT:+TIMEOUT=$TIMEOUT} ${AFL_FUZZ-afl-fuzz} "$@" \
-i $FUZZDIR/in -o $FUZZDIR/out tools/fuzz/kvm_emulate/afl-harness @@

where people can substitute afl-many.sh or add their own options using the AFL_FUZZ variable or the command line. Likewise for screen.

Yep, both of those are sensible. I’ll update with next patch.

Quoted 3 lines

- the build should be just "make -C tools/fuzz/kvm_emulate" and it should just work. Feel free to steal the Makefile magic from other tools/ directories.

Yeah, the build is a bit of a sore point. I’ll reach out if I can’t get it to work.

Quoted 5 lines

- finally, rand_sample.bin is missing.

Otherwise, it looks very nice.

Paolo

Thanks for the feedback.

Per the email’s bouncing, I’ve removed my @amazon.de email and will be using this one going forward.

- Sam