Skip to contentSkip to conversation

hv: vmbus: add fuzz testing to hv devices

PATCH Superseded Branden Bonaby 2 participants 10 messages Aug 1, 2019 – Aug 2, 2019

Subsystems (2)

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

cover hv: vmbus: add fuzz testing to hv devices

This patchset introduces a testing framework for Hyper-V drivers. This framework allows us to introduce delays in the packet receive path on a per-device basis. While the current code only supports introducing arbitrary delays in the host/guest communication path, we intend to expand this to support error injection in the future.

Branden Bonaby (3):
drivers: hv: vmbus: Introduce latency testing
drivers: hv: vmbus: add fuzz test attributes to sysfs
tools: hv: add vmbus testing tool

 Documentation/ABI/stable/sysfs-bus-vmbus |  22 ++
 drivers/hv/connection.c                  |   5 +
 drivers/hv/ring_buffer.c                 |  10 +
 drivers/hv/vmbus_drv.c                   |  97 ++++++-
 include/linux/hyperv.h                   |  14 +
 tools/hv/vmbus_testing                   | 326 +++++++++++++++++++++++
 6 files changed, 473 insertions(+), 1 deletion(-)
 create mode 100644 tools/hv/vmbus_testing
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/3 drivers: hv: vmbus: Introduce latency testing

Introduce user specified latency in the packet reception path.

---
 drivers/hv/connection.c  |  5 +++++
 drivers/hv/ring_buffer.c | 10 ++++++++++
 include/linux/hyperv.h   | 14 ++++++++++++++
 3 files changed, 29 insertions(+)
drivers/hv/connection.c +5 -0
index 09829e15d4a0..2a2c22f5570e 100644
@@ -354,9 +354,14 @@ void vmbus_on_event(unsigned long data)
 {
 	struct vmbus_channel *channel = (void *) data;
 	unsigned long time_limit = jiffies + 2;
+	struct vmbus_channel *test_channel = !channel->primary_channel ?
+						channel :
+						channel->primary_channel;
 
 	trace_vmbus_on_event(channel);
 
+	if (unlikely(test_channel->fuzz_testing_buffer_delay > 0))
+		udelay(test_channel->fuzz_testing_buffer_delay);
 	do {
 		void (*callback_fn)(void *);
 
drivers/hv/ring_buffer.c +10 -0
index 9a03b163cbbd..d7627c9023d6 100644
@@ -395,7 +395,12 @@ struct vmpacket_descriptor *hv_pkt_iter_first(struct vmbus_channel *channel)
 {
 	struct hv_ring_buffer_info *rbi = &channel->inbound;
 	struct vmpacket_descriptor *desc;
+	struct vmbus_channel *test_channel = !channel->primary_channel ?
+						channel :
+						channel->primary_channel;
 
+	if (unlikely(test_channel->fuzz_testing_message_delay > 0))
+		udelay(test_channel->fuzz_testing_message_delay);
 	if (hv_pkt_iter_avail(rbi) < sizeof(struct vmpacket_descriptor))
 		return NULL;
 
@@ -420,7 +425,12 @@ __hv_pkt_iter_next(struct vmbus_channel *channel,
 	struct hv_ring_buffer_info *rbi = &channel->inbound;
 	u32 packetlen = desc->len8 << 3;
 	u32 dsize = rbi->ring_datasize;
+	struct vmbus_channel *test_channel = !channel->primary_channel ?
+						channel :
+						channel->primary_channel;
 
+	if (unlikely(test_channel->fuzz_testing_message_delay > 0))
+		udelay(test_channel->fuzz_testing_message_delay);
 	/* bump offset to next potential packet */
 	rbi->priv_read_index += packetlen + VMBUS_PKT_TRAILER;
 	if (rbi->priv_read_index >= dsize)
include/linux/hyperv.h +14 -0
index 6256cc34c4a6..8d068956dd67 100644
@@ -23,6 +23,7 @@
 #include <linux/mod_devicetable.h>
 #include <linux/interrupt.h>
 #include <linux/reciprocal_div.h>
+#include <linux/delay.h>
 
 #define MAX_PAGE_BUFFER_COUNT				32
 #define MAX_MULTIPAGE_BUFFER_COUNT			32 /* 128K */
@@ -926,6 +927,19 @@ struct vmbus_channel {
 	 * full outbound ring buffer.
 	 */
 	u64 out_full_first;
+
+	/* enabling/disabling fuzz testing on the channel (default is false)*/
+	bool fuzz_testing_state;
+
+	/* Buffer delay will delay the guest from emptying the ring buffer
+	 * for a specific amount of time. The delay is in microseconds and will
+	 * be between 1 to a maximum of 1000, its default is 0 (no delay).
+	 * The  Message delay will delay guest reading on a per message basis
+	 * in microseconds between 1 to 1000 with the default being 0
+	 * (no delay).
+	 */
+	u32 fuzz_testing_buffer_delay;
+	u32 fuzz_testing_message_delay;
 };
 
 static inline bool is_hvsock_channel(const struct vmbus_channel *c)
Signature · 1 line
2.17.1
#4 lore raw

PATCH 2/3 drivers: hv: vmbus: add fuzz test attributes to sysfs

In reply to Branden Bonaby

Expose the test parameters as part of the sysfs channel attributes. We will control the testing state via these attributes.

---
 Documentation/ABI/stable/sysfs-bus-vmbus | 22 ++++++
 drivers/hv/vmbus_drv.c                   | 97 +++++++++++++++++++++++-
 2 files changed, 118 insertions(+), 1 deletion(-)
Documentation/ABI/stable/sysfs-bus-vmbus +22 -0
index 8e8d167eca31..239fcb6fdc75 100644
@@ -185,3 +185,25 @@ Contact:        Michael Kelley <[email protected]>
 Description:    Total number of write operations that encountered an outbound
 		ring buffer full condition
 Users:          Debugging tools
+
+What:           /sys/bus/vmbus/devices/<UUID>/fuzz_test_state
+Date:           July 2019
+KernelVersion:  5.2
+Contact:        Branden Bonaby <[email protected]>
+Description:    Fuzz testing status of a vmbus device, whether its in an ON
+		 state or a OFF state
+Users:          Debugging tools
+
+What:           /sys/bus/vmbus/devices/<UUID>/fuzz_test_buffer_delay
+Date:           July 2019
+KernelVersion:  5.2
+Contact:        Branden Bonaby <[email protected]>
+Description:    Fuzz testing buffer delay value between 0 - 1000
+Users:          Debugging tools
+
+What:           /sys/bus/vmbus/devices/<UUID>/fuzz_test_message_delay
+Date:           July 2019
+KernelVersion:  5.2
+Contact:        Branden Bonaby <[email protected]>
+Description:    Fuzz testing message delay value between 0 - 1000
+Users:          Debugging tools
drivers/hv/vmbus_drv.c +96 -1

115 lines not on this page · raw message

Signature · 1 line
2.17.1
#7 lore raw

PATCH 3/3 tools: hv: add vmbus testing tool

In reply to Branden Bonaby

This is a userspace tool to drive the testing. Currently it supports introducing user specified delay in the host to guest communication path on a per-channel basis.

---
 tools/hv/vmbus_testing | 326 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 326 insertions(+)
 create mode 100644 tools/hv/vmbus_testing
tools/hv/vmbus_testing new file +326 -0
new file mode 100644
index 000000000000..8f5cd997952b
@@ -0,0 +1,326 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+#
+#Program to allow users to fuzz test Hyper-V drivers
+#by interfacing with Hyper-V sysfs directories
+#author: Branden Bonaby <[email protected]>
+
+import os
+import cmd
+import argparse
+from collections import defaultdict
+from argparse import RawDescriptionHelpFormatter
+
+#sysfs paths for vmbus must exist (same as in lsvmbus)
+vmbus_sys_path = '/sys/bus/vmbus/devices'
+if not os.path.isdir(vmbus_sys_path):
+            print("{} doesn't exist: exiting...".format(vmbus_sys_path))
+            exit(-1)
+
+#Do not change unless, you change the sysfs attributes
+#in "/sys/bus/vmbus/devices/<UUID>/". All fuzz testing
+#attributes will start with "fuzz_test".
+pathlen = len(vmbus_sys_path)
+fuzz_state_location = "fuzz_test_state"
+fuzz_states = {0 : "Disable", 1 : "Enable"}
+fuzz_methods ={1 : "Delay_testing"}
+fuzz_delay_types = {1 : "fuzz_test_buffer_delay", 2 :"fuzz_test_message_delay"}
+
+def parse_args():
+        parser = argparse.ArgumentParser(description = "vmbus_testing "\
+                "[-s] [0|1] [-q] [-p] <sysfs-path>\n""vmbus_testing [-s]"\
+                " [0|1] [-q] [-p] <sysfs-path> delay [-d] [val] [val] [-E|-D]\n"
+                "vmbus_testing [-q] disable-all\n"
+                "vmbus_testing [-q] view [-v|-V]\n"\
+                "vmbus_testing --version",
+                epilog = "Current testing options {}".format(fuzz_methods),
+                prog = 'vmbus_testing',
+                formatter_class = RawDescriptionHelpFormatter)
+        subparsers = parser.add_subparsers(dest="action")
+        parser.add_argument('--version', action='version',\
+                        version = '%(prog)s 1.0')
+        parser.add_argument("-q","--quiet",action = "store_true",\
+                        help = "silence none important test messages")
+        parser.add_argument("-s","--state",metavar = "", type = int,\
+                        choices = range(0,2),\
+                        help = "Turn testing ON or OFF for a single device."\
+                        " The value (1) will turn testing ON. The value"\
+                        " of (0) will turn testing OFF with the default set"\
+                        " to (0).")
+        parser.add_argument("-p","--path", metavar = "",\
+                        help = "Refers to the sysfs path to a vmbus device."
+                        " If the path is not a valid path to a sysfs device,"\
+                        " the program will exit. The path must be the"\
+                        " absolute path; use the lsvmbus command to find"\
+                        " the path.")
+        parser_delay = subparsers.add_parser("delay",\
+                        help = "Delay buffer/message reads in microseconds.",
+                        description = "vmbus_testing -s [0|1] [-q] -p "\
+                        "<sysfs-path> delay -d "\
+                        "[buffer-delay-value] [message-delay-value]\n"
+                        "vmbus_testing [-q] delay [buffer-delay-value] "\
+                                "[message-delay-value] -E\n"
+                        "vmbus_testing [-q] delay [buffer-delay-value] "\
+                                "[message-delay-value] -D",
+                        formatter_class = RawDescriptionHelpFormatter)
+        delay_group = parser_delay.add_mutually_exclusive_group()
+        delay_group.add_argument("-E","--en-all-delay", action = "store_true",\
+                        help = "Enable Buffer/Message Delay testing on ALL"\
+                        " devices. Use -d option with this to set the values"\
+                        " for both the buffer delay and the message delay. No"\
+                        " value can be (0) or less than (-1). If testing is"\
+                        " disabled on a device prior to running this command,"\
+                        " testing will be enabled on the device as a result"\
+                        " of this command.")
+        delay_group.add_argument("-D","--dis-all-delay", action="store_true",\
+                        help = "Disable Buffer/Message delay testing on ALL"\
+                        " devices. A  value equal to (-1) will keep the"\
+                        " current delay value, and a value equal to (0) will"\
+                        " remove delay testing for the specfied delay column."\
+                        " only values (-1) and (0) will be accepted but at"\
+                        " least one value must be a (0) or a (-1).")
+        parser_delay.add_argument("-d","--delay-time", metavar="", nargs=2,\
+                        type = check_range, default =[0,0], required = (True),\
+                        help = "Buffer/message delay time. A value of (0) will"\
+                        "disable delay testing on the specified delay column,"\
+                        " while a value of (-1) will ignore the specfied"\
+                        " delay column. The default values are [0] & [0]."\
+                        " The first column represents the buffer delay value"\
+                        " and the second represents the message delay value."\
+                        " Value constraints: -1 <= value <= 1000.")
+        parser_dis_all = subparsers.add_parser("disable-all",\
+                        help = "Disable ALL testing on all vmbus devices.",
+                        description = "vmbus_testing disable-all",
+                        formatter_class = RawDescriptionHelpFormatter)
+        parser_view = subparsers.add_parser("view",\
+                        help = "View testing on vmbus devices.",
+                        description = "vmbus_testing view -V\n"
+                        "vmbus_testing -p <sysfs-path> view -v",
+                        formatter_class = RawDescriptionHelpFormatter)
+        view_group = parser_view.add_mutually_exclusive_group()
+        view_group.add_argument("-V","--view-all-states",action = "store_true",\
+                        help = "View the test status for all vmbus devices.")
+        view_group.add_argument("-v","--view-single-device",\
+                        action = "store_true",help = "View test values for a"\
+                        " single vmbus device.")
+
+        return  parser.parse_args()
+
+#value checking for range checking input in parser
+def check_range(arg1):
+        try:
+                val = int(arg1)
+        except ValueError as err:
+                raise argparse.ArgumentTypeError(str(err))
+        if val < -1 or val > 1000:
+                message = ("\n\nError, Expected -1 <= value <= 1000, got value"\
+                        " {}\n").format(val)
+                raise argparse.ArgumentTypeError(message)
+        return val
+
+def main():
+        try:
+                dev_list = []
+                for dir in os.listdir(vmbus_sys_path):\
+                        dev_list.append(os.path.join(vmbus_sys_path,dir))
+                #key value, pairs
+                #key = sysfs device path
+                #value = list of fuzz testing attributes.
+                device_and_files = defaultdict(list)
+                for dev in dev_list:
+                        for f in os.listdir(dev):
+                                if (f.startswith("fuzz_test")):
+                                        device_and_files[dev].append(f)
+
+                device_and_files.default_factory = None
+                args = parse_args()
+                path = args.path
+                state = args.state
+                quiet = args.quiet
+                if (not quiet):
+                        print("*** Use lsvmbus to get vmbus device"\
+                                " information.*** ")
+                if (state is not None and validate_args_path(path,dev_list)):
+                        if (state is not get_test_state(path)):
+                                change_test_state(path,quiet)
+                        state = get_test_state(path)
+                if (state is 0 and path is not None):
+                        disable_testing_single_device(path,0,quiet)
+                        return
+                #Use subparsers as the key for different fuzz testing methods
+                if (args.action == "delay"):
+                        delay = args.delay_time
+                        if (validate_delay_values(args,delay)):
+                                delay_test_all_devices(dev_list,delay,quiet)
+                        elif (validate_args_path(path,dev_list)):
+                                if(get_test_state(path) is 1):
+                                        delay_test_store(path,delay,quiet)
+                                        return
+                                print("device testing OFF, use -s 1 to turn ON")
+                elif (args.action == "disable-all"):
+                        disable_all_testing(dev_list,quiet)
+                elif (args.action == "view"):
+                        if (args.view_all_states):
+                                all_devices_test_status(dev_list)
+                        elif (args.view_single_device):
+                                if (validate_args_path(path,dev_list)):
+                                        device_test_values(device_and_files,\
+                                                path)
+                                        return
+                                print("Error,(check path) usage: -p [path]"\
+                                            " view -v")
+        except AttributeError:
+                print("check usage, 1 or more elements not provided")
+                exit(-1)
+
+#Validate delay values to make sure they are acceptable to
+#to either enable all delays on a device or disable all
+#delays on a device
+def validate_delay_values(args,delay):
+        if (args.en_all_delay):
+                for i in delay:
+                        if (i < -1 or i == 0):
+                                print("\nError, Values must be"\
+                                        " equal to -1 or be > 0, use"\
+                                        " -d option")
+                                exit(-1)
+                return True
+        elif (args.dis_all_delay):
+                for i in delay:
+                        if (i < -1 or i > 0):
+                                print("\nError, at least 1 value"
+                                        " is not a (0) or a (-1)")
+                                exit(-1)
+                return True
+        else:
+                return False
+
+
+#Validate argument path
+def validate_args_path(path,dev_list):
+        if (path in dev_list):
+                return True
+        else:
+                return False
+
+#display Testing status of single device
+def device_test_values(device_and_files,path):
+        for test in  device_and_files.get(path):
+                print("{}".format(test), end = '')
+                print((" value =  {}")\
+                        .format(read_test_files(os.path.join(path,test))))
+
+#display Testing state of devices
+def all_devices_test_status(dev_list):
+    for device in dev_list:
+        if (get_test_state(device) is 1):
+                print("Testing = ON for: {}".format(device.split("/")[5]))
+        else:
+                print("Testing = OFF for: {}".format(device.split("/")[5]))
+
+#read the vmbus device files, path must be absolute path before calling
+def read_test_files(path):
+        try:
+                with open(path,"r") as f:
+                        state = f.readline().replace("\n","")
+                return int(state)
+
+        except IOError as e:
+                errno, strerror = e.args
+                print("I/O error({0}): {1} on file {2}"\
+                        .format(errno,strerror,path))
+                exit(-1)
+        except ValueError:
+                print ("Element to int conversion error in: \n{}".format(path))
+                exit(-1)
+
+#writing to vmbus device files, path must be absolute path before calling
+def write_test_files(path,value):
+        try:
+                with open(path,"w") as f:
+                        f.write("{}".format(value))
+        except IOError as e:
+                errno, strerror = e.args
+                print("I/O error({0}): {1} on file {2}"\
+                        .format(errno,strerror,path))
+                exit(-1)
+
+#change testing state of device
+def change_test_state(device,quiet):
+        state_path = os.path.join(device,fuzz_state_location)
+        if (get_test_state(device) is 0):
+                write_test_files(state_path,1)
+                if (not quiet):
+                            print("Testing = ON for device: {}"\
+                                    .format(state_path.split("/")[5]))
+        else:
+                write_test_files(state_path,0)
+                if (not quiet):
+                            print("Testing = OFF for device: {}"\
+                                    .format(state_path.split("/")[5]))
+
+#get testing state of device
+def get_test_state(device):
+        #state == 1 - test = ON
+        #state == 0 - test = OFF
+        return  read_test_files(os.path.join(device,fuzz_state_location))
+
+#Enter 1 - 1000 microseconds, into a single device using the
+#fuzz_test_buffer_delay and fuzz_test_message_delay
+#sysfs attributes
+def delay_test_store(device,delay_length,quiet):
+
+        try:
+                # delay[0]- buffer delay, delay[1]- message delay
+                buff_test = os.path.join(device,fuzz_delay_types.get(1))
+                mess_test = os.path.join(device,fuzz_delay_types.get(2))
+
+                if (delay_length[0] >= 0):
+                        write_test_files(buff_test,delay_length[0])
+                if (delay_length[1] >= 0):
+                        write_test_files(mess_test,delay_length[1])
+                if (not quiet):
+                        print("Buffer delay testing = {} for: {}"\
+                                .format(read_test_files(buff_test),\
+                                buff_test.split("/")[5]))
+                        print("Message delay testing = {} for: {}"\
+                                .format(read_test_files(mess_test),\
+                                mess_test.split("/")[5]))
+        except IOError as e:
+                errno, strerror = e.args
+                print("I/O error({0}): {1} on files {2}{3}"\
+                        .format(errno,strerror,buff_test,mess_test))
+                exit(-1)
+
+#enabling/disabling delay testing on all devices
+def delay_test_all_devices(dev_list,delay,quiet):
+
+        for device in (dev_list):
+                if (get_test_state(device) is 0):
+                        change_test_state(device,quiet)
+                delay_test_store(device,delay,quiet)
+
+#disabling testing on single device
+def disable_testing_single_device(device,test_type,quiet):
+
+        #test_type represents corresponding key
+        #delay method in delay_methods dict.
+        #special type 0 , used to disable all
+        #testing on SINGLE device.
+
+        if (test_type is 1 or test_type is 0):
+                #disable list [buffer,message]
+                disable_delay = [0,0]
+                if (get_test_state(device) is 1):
+                        change_test_state(device,quiet)
+                delay_test_store(device,disable_delay,quiet)
+
+#disabling testing on ALL devices
+def disable_all_testing(dev_list,quiet):
+
+        #delay disable list [buffer,message]
+        for device in dev_list:
+                disable_testing_single_device(device,0,quiet)
+
+if __name__ == "__main__":
+        main()
Signature · 1 line
2.17.1
#8 lore raw
In reply to Branden Bonaby
Quoted 11 lines · Branden Bonaby

This patchset introduces a testing framework for Hyper-V drivers. This framework allows us to introduce delays in the packet receive path on a per-device basis. While the current code only supports introducing arbitrary delays in the host/guest communication path, we intend to expand this to support error injection in the future.

Branden Bonaby (3):
drivers: hv: vmbus: Introduce latency testing
drivers: hv: vmbus: add fuzz test attributes to sysfs
tools: hv: add vmbus testing tool

Quoted 4 lines
 Documentation/ABI/stable/sysfs-bus-vmbus |  22 ++
 drivers/hv/connection.c                  |   5 +
 drivers/hv/ring_buffer.c                 |  10 +
 drivers/hv/vmbus_drv.c                   |  97 ++++++-

Can we have something like CONFIG_HYPERV_TESTING and put this new code under #ifdef?

Quoted 4 lines
 include/linux/hyperv.h                   |  14 +
 tools/hv/vmbus_testing                   | 326 +++++++++++++++++++++++
 6 files changed, 473 insertions(+), 1 deletion(-)
 create mode 100644 tools/hv/vmbus_testing
Signature · 1 line
Vitaly
#2 lore raw

Re: [PATCH 1/3] drivers: hv: vmbus: Introduce latency testing

In reply to Branden Bonaby
Quoted 46 lines · Branden Bonaby

Introduce user specified latency in the packet reception path.

---
 drivers/hv/connection.c  |  5 +++++
 drivers/hv/ring_buffer.c | 10 ++++++++++
 include/linux/hyperv.h   | 14 ++++++++++++++
 3 files changed, 29 insertions(+)
drivers/hv/connection.c +5 -0
index 09829e15d4a0..2a2c22f5570e 100644
@@ -354,9 +354,14 @@ void vmbus_on_event(unsigned long data)
 {
 	struct vmbus_channel *channel = (void *) data;
 	unsigned long time_limit = jiffies + 2;
+	struct vmbus_channel *test_channel = !channel->primary_channel ?
+						channel :
+						channel->primary_channel;
 
 	trace_vmbus_on_event(channel);
 
+	if (unlikely(test_channel->fuzz_testing_buffer_delay > 0))
+		udelay(test_channel->fuzz_testing_buffer_delay);
 	do {
 		void (*callback_fn)(void *);
 
drivers/hv/ring_buffer.c +5 -0
index 9a03b163cbbd..d7627c9023d6 100644
@@ -395,7 +395,12 @@ struct vmpacket_descriptor *hv_pkt_iter_first(struct vmbus_channel *channel)
 {
 	struct hv_ring_buffer_info *rbi = &channel->inbound;
 	struct vmpacket_descriptor *desc;
+	struct vmbus_channel *test_channel = !channel->primary_channel ?
+						channel :
+						channel->primary_channel;
 
+	if (unlikely(test_channel->fuzz_testing_message_delay > 0))
+		udelay(test_channel->fuzz_testing_message_delay);
 	if (hv_pkt_iter_avail(rbi) < sizeof(struct vmpacket_descriptor))
 		return NULL;
 

drivers/hv/ring_buffer.c __hv_pkt_iter_next(struct vmbus_channel *channel, · line 430

Quoted 6 lines
drivers/hv/ring_buffer.c +3 -0
@@ -420,7 +425,12 @@ __hv_pkt_iter_next(struct vmbus_channel *channel,
 	struct hv_ring_buffer_info *rbi = &channel->inbound;
 	u32 packetlen = desc->len8 << 3;
 	u32 dsize = rbi->ring_datasize;
+	struct vmbus_channel *test_channel = !channel->primary_channel ?
+						channel :
+						channel->primary_channel;

This pattern is repeated 3 times so a define is justified. I would also reversed the logic:

   test_channel = channel->primary_channel ? channel->primary_channel : channel;

drivers/hv/ring_buffer.c __hv_pkt_iter_next(struct vmbus_channel *channel, · line 432

Quoted 3 lines
drivers/hv/ring_buffer.c +2 -0
+	if (unlikely(test_channel->fuzz_testing_message_delay > 0))
+		udelay(test_channel->fuzz_testing_message_delay);

unlikely() is good but if it was under #ifdef it would've been even better.

Quoted 35 lines
drivers/hv/ring_buffer.c +0 -0
 	/* bump offset to next potential packet */
 	rbi->priv_read_index += packetlen + VMBUS_PKT_TRAILER;
 	if (rbi->priv_read_index >= dsize)
include/linux/hyperv.h +14 -0
index 6256cc34c4a6..8d068956dd67 100644
@@ -23,6 +23,7 @@
 #include <linux/mod_devicetable.h>
 #include <linux/interrupt.h>
 #include <linux/reciprocal_div.h>
+#include <linux/delay.h>
 
 #define MAX_PAGE_BUFFER_COUNT				32
 #define MAX_MULTIPAGE_BUFFER_COUNT			32 /* 128K */
@@ -926,6 +927,19 @@ struct vmbus_channel {
 	 * full outbound ring buffer.
 	 */
 	u64 out_full_first;
+
+	/* enabling/disabling fuzz testing on the channel (default is false)*/
+	bool fuzz_testing_state;
+
+	/* Buffer delay will delay the guest from emptying the ring buffer
+	 * for a specific amount of time. The delay is in microseconds and will
+	 * be between 1 to a maximum of 1000, its default is 0 (no delay).
+	 * The  Message delay will delay guest reading on a per message basis
+	 * in microseconds between 1 to 1000 with the default being 0
+	 * (no delay).
+	 */
+	u32 fuzz_testing_buffer_delay;
+	u32 fuzz_testing_message_delay;
 };
 
 static inline bool is_hvsock_channel(const struct vmbus_channel *c)
Signature · 1 line
Vitaly
#5 lore raw

Re: [PATCH 2/3] drivers: hv: vmbus: add fuzz test attributes to sysfs

In reply to Branden Bonaby
Quoted 14 lines · Branden Bonaby

Expose the test parameters as part of the sysfs channel attributes. We will control the testing state via these attributes.

---
 Documentation/ABI/stable/sysfs-bus-vmbus | 22 ++++++
 drivers/hv/vmbus_drv.c                   | 97 +++++++++++++++++++++++-
 2 files changed, 118 insertions(+), 1 deletion(-)
Documentation/ABI/stable/sysfs-bus-vmbus +0 -0
index 8e8d167eca31..239fcb6fdc75 100644

Documentation/ABI/stable/sysfs-bus-vmbus Contact: Michael Kelley <[email protected]> · line 189

Quoted 5 lines
Documentation/ABI/stable/sysfs-bus-vmbus +2 -0
@@ -185,3 +185,25 @@ Contact:        Michael Kelley <[email protected]>
 Description:    Total number of write operations that encountered an outbound
 		ring buffer full condition
 Users:          Debugging tools
+
+What:           /sys/bus/vmbus/devices/<UUID>/fuzz_test_state

I would prefer this to go under /sys/kernel/debug/ as this is clearly a debug/test feature.

Quoted 141 lines
Documentation/ABI/stable/sysfs-bus-vmbus +20 -0
+Date:           July 2019
+KernelVersion:  5.2
+Contact:        Branden Bonaby <[email protected]>
+Description:    Fuzz testing status of a vmbus device, whether its in an ON
+		 state or a OFF state
+Users:          Debugging tools
+
+What:           /sys/bus/vmbus/devices/<UUID>/fuzz_test_buffer_delay
+Date:           July 2019
+KernelVersion:  5.2
+Contact:        Branden Bonaby <[email protected]>
+Description:    Fuzz testing buffer delay value between 0 - 1000
+Users:          Debugging tools
+
+What:           /sys/bus/vmbus/devices/<UUID>/fuzz_test_message_delay
+Date:           July 2019
+KernelVersion:  5.2
+Contact:        Branden Bonaby <[email protected]>
+Description:    Fuzz testing message delay value between 0 - 1000
+Users:          Debugging tools
drivers/hv/vmbus_drv.c +96 -1
index 92b1874b3eb3..0c71fd66ef81 100644
@@ -22,7 +22,7 @@
 #include <linux/clockchips.h>
 #include <linux/cpu.h>
 #include <linux/sched/task_stack.h>
-
+#include <linux/kernel.h>
 #include <asm/mshyperv.h>
 #include <linux/notifier.h>
 #include <linux/ptrace.h>
@@ -584,6 +584,98 @@ static ssize_t driver_override_show(struct device *dev,
 }
 static DEVICE_ATTR_RW(driver_override);
 
+static ssize_t fuzz_test_state_store(struct device *dev,
+				     struct device_attribute *attr,
+				     const char *buf, size_t count)
+{
+	struct hv_device *hv_dev = device_to_hv_device(dev);
+	struct vmbus_channel *channel = hv_dev->channel;
+	int state;
+	int delay = kstrtoint(buf, 0, &state);
+
+	if (delay)
+		return count;
+	if (state)
+		channel->fuzz_testing_state = 1;
+	else
+		channel->fuzz_testing_state = 0;
+	return count;
+}
+
+static ssize_t fuzz_test_state_show(struct device *dev,
+				    struct device_attribute *dev_attr,
+				    char *buf)
+{
+	struct hv_device *hv_dev = device_to_hv_device(dev);
+	struct vmbus_channel *channel = hv_dev->channel;
+
+	return sprintf(buf, "%u\n", channel->fuzz_testing_state);
+}
+static DEVICE_ATTR_RW(fuzz_test_state);
+
+static ssize_t fuzz_test_buffer_delay_store(struct device *dev,
+					    struct device_attribute *attr,
+					    const char *buf, size_t count)
+{
+	struct hv_device *hv_dev = device_to_hv_device(dev);
+	struct vmbus_channel *channel = hv_dev->channel;
+	int val;
+	int delay = kstrtoint(buf, 0, &val);
+
+	if (delay)
+		return count;
+	if (val >= 1 && val <= 1000)
+		channel->fuzz_testing_buffer_delay = val;
+	/*Best to not use else statement here since we want
+	 *the buffer delay to remain the same if val > 1000
+	 */
+	else if (val <= 0)
+		channel->fuzz_testing_buffer_delay = 0;
+	return count;
+}
+
+static ssize_t fuzz_test_buffer_delay_show(struct device *dev,
+					   struct device_attribute *dev_attr,
+					   char *buf)
+{
+	struct hv_device *hv_dev = device_to_hv_device(dev);
+	struct vmbus_channel *channel = hv_dev->channel;
+
+	return sprintf(buf, "%u\n", channel->fuzz_testing_buffer_delay);
+}
+static DEVICE_ATTR_RW(fuzz_test_buffer_delay);
+
+static ssize_t fuzz_test_message_delay_store(struct device *dev,
+					     struct device_attribute *attr,
+					     const char *buf, size_t count)
+{
+	struct hv_device *hv_dev = device_to_hv_device(dev);
+	struct vmbus_channel *channel = hv_dev->channel;
+	int val;
+	int delay = kstrtoint(buf, 0, &val);
+
+	if (delay)
+		return count;
+	if (val >= 1 && val <= 1000)
+		channel->fuzz_testing_message_delay = val;
+	/*Best to not use else statement here since we want
+	 *the message delay to remain the same if val > 1000
+	 */
+	else if (val <= 0)
+		channel->fuzz_testing_message_delay = 0;
+	return count;
+}
+
+static ssize_t fuzz_test_message_delay_show(struct device *dev,
+					    struct device_attribute *dev_attr,
+					    char *buf)
+{
+	struct hv_device *hv_dev = device_to_hv_device(dev);
+	struct vmbus_channel *channel = hv_dev->channel;
+
+	return sprintf(buf, "%u\n", channel->fuzz_testing_message_delay);
+}
+static DEVICE_ATTR_RW(fuzz_test_message_delay);
 /* Set up per device attributes in /sys/bus/vmbus/devices/<bus device> */
 static struct attribute *vmbus_dev_attrs[] = {
 	&dev_attr_id.attr,
@@ -615,6 +707,9 @@ static struct attribute *vmbus_dev_attrs[] = {
 	&dev_attr_vendor.attr,
 	&dev_attr_device.attr,
 	&dev_attr_driver_override.attr,
+	&dev_attr_fuzz_test_state.attr,
+	&dev_attr_fuzz_test_buffer_delay.attr,
+	&dev_attr_fuzz_test_message_delay.attr,
 	NULL,
 };
Signature · 1 line
Vitaly
#9 lore raw
In reply to Vitaly Kuznetsov
Quoted 18 lines · Vitaly Kuznetsov
Quoted 11 lines · Branden Bonaby

This patchset introduces a testing framework for Hyper-V drivers. This framework allows us to introduce delays in the packet receive path on a per-device basis. While the current code only supports introducing arbitrary delays in the host/guest communication path, we intend to expand this to support error injection in the future.

Branden Bonaby (3):
drivers: hv: vmbus: Introduce latency testing
drivers: hv: vmbus: add fuzz test attributes to sysfs
tools: hv: add vmbus testing tool

Quoted 4 lines
 Documentation/ABI/stable/sysfs-bus-vmbus |  22 ++
 drivers/hv/connection.c                  |   5 +
 drivers/hv/ring_buffer.c                 |  10 +
 drivers/hv/vmbus_drv.c                   |  97 ++++++-
Quoted 10 lines

Can we have something like CONFIG_HYPERV_TESTING and put this new code under #ifdef?

Quoted 4 lines
 include/linux/hyperv.h                   |  14 +
 tools/hv/vmbus_testing                   | 326 +++++++++++++++++++++++
 6 files changed, 473 insertions(+), 1 deletion(-)
 create mode 100644 tools/hv/vmbus_testing
Signature · 1 line
Vitaly

You're right, it would be better to do it that way with ifdef's. Will edit my patches and resend.

Branden Bonaby

#3 lore raw

Re: [PATCH 1/3] drivers: hv: vmbus: Introduce latency testing

In reply to Vitaly Kuznetsov
Quoted 61 lines · Vitaly Kuznetsov
Quoted 46 lines · Branden Bonaby

Introduce user specified latency in the packet reception path.

---
 drivers/hv/connection.c  |  5 +++++
 drivers/hv/ring_buffer.c | 10 ++++++++++
 include/linux/hyperv.h   | 14 ++++++++++++++
 3 files changed, 29 insertions(+)
drivers/hv/connection.c +5 -0
index 09829e15d4a0..2a2c22f5570e 100644
@@ -354,9 +354,14 @@ void vmbus_on_event(unsigned long data)
 {
 	struct vmbus_channel *channel = (void *) data;
 	unsigned long time_limit = jiffies + 2;
+	struct vmbus_channel *test_channel = !channel->primary_channel ?
+						channel :
+						channel->primary_channel;
 
 	trace_vmbus_on_event(channel);
 
+	if (unlikely(test_channel->fuzz_testing_buffer_delay > 0))
+		udelay(test_channel->fuzz_testing_buffer_delay);
 	do {
 		void (*callback_fn)(void *);
 
drivers/hv/ring_buffer.c +5 -0
index 9a03b163cbbd..d7627c9023d6 100644
@@ -395,7 +395,12 @@ struct vmpacket_descriptor *hv_pkt_iter_first(struct vmbus_channel *channel)
 {
 	struct hv_ring_buffer_info *rbi = &channel->inbound;
 	struct vmpacket_descriptor *desc;
+	struct vmbus_channel *test_channel = !channel->primary_channel ?
+						channel :
+						channel->primary_channel;
 
+	if (unlikely(test_channel->fuzz_testing_message_delay > 0))
+		udelay(test_channel->fuzz_testing_message_delay);
 	if (hv_pkt_iter_avail(rbi) < sizeof(struct vmpacket_descriptor))
 		return NULL;
 

drivers/hv/ring_buffer.c __hv_pkt_iter_next(struct vmbus_channel *channel, · line 430

Quoted 6 lines
drivers/hv/ring_buffer.c +3 -0
@@ -420,7 +425,12 @@ __hv_pkt_iter_next(struct vmbus_channel *channel,
 	struct hv_ring_buffer_info *rbi = &channel->inbound;
 	u32 packetlen = desc->len8 << 3;
 	u32 dsize = rbi->ring_datasize;
+	struct vmbus_channel *test_channel = !channel->primary_channel ?
+						channel :
+						channel->primary_channel;

This pattern is repeated 3 times so a define is justified. I would also reversed the logic:

   test_channel = channel->primary_channel ? channel->primary_channel : channel;
Quoted 1 line
Quoted 8 lines

drivers/hv/ring_buffer.c

Quoted 2 lines
drivers/hv/ring_buffer.c +2 -0
+	if (unlikely(test_channel->fuzz_testing_message_delay > 0))
+		udelay(test_channel->fuzz_testing_message_delay);

unlikely() is good but if it was under #ifdef it would've been even better.

Quoted 1 line

/* bump offset to next potential packet */

Signature · 1 line
Vitaly

Makes sense, I'll address the repeated code and will change the way I handled that if statement. Using an ifdef CONFIG_HYPERV_TESTING seems like a good thing to add in here like you suggested.

#6 lore raw

Re: [PATCH 2/3] drivers: hv: vmbus: add fuzz test attributes to sysfs

In reply to Vitaly Kuznetsov
Quoted 11 lines · Vitaly Kuznetsov
Quoted 9 lines · Branden Bonaby

Expose the test parameters as part of the sysfs channel attributes. We will control the testing state via these attributes.

---
 Documentation/ABI/stable/sysfs-bus-vmbus | 22 ++++++
 drivers/hv/vmbus_drv.c                   | 97 +++++++++++++++++++++++-
 2 files changed, 118 insertions(+), 1 deletion(-)
Quoted 15 lines

Documentation/ABI/stable/sysfs-bus-vmbus Contact: Michael Kelley <[email protected]> · line 189

Quoted 10 lines
Documentation/ABI/stable/sysfs-bus-vmbus +2 -0
index 8e8d167eca31..239fcb6fdc75 100644
@@ -185,3 +185,25 @@ Contact:        Michael Kelley <[email protected]>
 Description:    Total number of write operations that encountered an outbound
 		ring buffer full condition
 Users:          Debugging tools
+
+What:           /sys/bus/vmbus/devices/<UUID>/fuzz_test_state

I would prefer this to go under /sys/kernel/debug/ as this is clearly a debug/test feature.

Signature · 1 line
Vitaly

Alright, it is testing so I see what you mean and why the code in this patch should go in debugfs. Will fix that and resend.

Branden Bonaby