hv: vmbus: add fuzz testing to hv device
Subsystems (2)
v3 cover hv: vmbus: add fuzz testing to hv device
This patchset introduces a testing framework for Hyper-V drivers. This framework allows us to introduce delays in the packet receive path on a per-device basis. While the current code only supports introducing arbitrary delays in the host/guest communication path, we intend to expand this to support error injection in the future.
changes in v3:
patch 2: change call to IS_ERR_OR_NULL, to IS_ERR.
patch 3: Align python tool to match Linux coding style.
Changes in v2:
Patch 1: As per Vitaly's suggestion, wrapped the test code under an
#ifdef and updated the Kconfig file, so that the test code will only be used when the config option is set to true. (default is false).
Updated hyperv_vmbus header to contain new #ifdef with new new functions for the test code.
Patch 2: Moved code from under sysfs to debugfs and wrapped it under
the new ifdef.
Updated MAINTAINERS file with new debugfs-hyperv file under the section for hyperv.
Patch 3: Updated testing tool with new debugfs location.
Branden Bonaby (3):
drivers: hv: vmbus: Introduce latency testing
drivers: hv: vmbus: add fuzz test attributes to debugfs
tools: hv: add vmbus testing tool
Documentation/ABI/testing/debugfs-hyperv | 21 ++ MAINTAINERS | 1 + drivers/hv/Kconfig | 7 + drivers/hv/connection.c | 3 + drivers/hv/hyperv_vmbus.h | 20 ++ drivers/hv/ring_buffer.c | 7 + drivers/hv/vmbus_drv.c | 167 +++++++++++ include/linux/hyperv.h | 21 ++ tools/hv/vmbus_testing | 342 +++++++++++++++++++++++ 9 files changed, 589 insertions(+) create mode 100644 Documentation/ABI/testing/debugfs-hyperv create mode 100644 tools/hv/vmbus_testing
Signature · 1 line
2.17.1
PATCH v3 1/3 drivers: hv: vmbus: Introduce latency testing
Introduce user specified latency in the packet reception path.
---
Changes in v2:
- Add #ifdef in Kconfig file so test code will not interfere with non-test code.
- Move test code functions for delay to hyperv_vmbus header file.
- Wrap test code under #ifdef statement.
drivers/hv/Kconfig | 7 +++++++ drivers/hv/connection.c | 3 +++ drivers/hv/hyperv_vmbus.h | 20 ++++++++++++++++++++ drivers/hv/ring_buffer.c | 7 +++++++ include/linux/hyperv.h | 21 +++++++++++++++++++++ 5 files changed, 58 insertions(+)
drivers/hv/Kconfig +7 -0
@@ -29,4 +29,11 @@ config HYPERV_BALLOON help Select this option to enable Hyper-V Balloon driver. +config HYPERV_TESTING + bool "Hyper-V testing" + default n + depends on HYPERV && DEBUG_FS + help + Select this option to enable Hyper-V vmbus testing. + endmenu
drivers/hv/connection.c +3 -0
@@ -357,6 +357,9 @@ void vmbus_on_event(unsigned long data)trace_vmbus_on_event (channel); +#ifdef CONFIG_HYPERV_TESTING +hv_debug_delay_test (channel,INTERRUPT_DELAY ); +#endif /* CONFIG_HYPERV_TESTING */ do { void (*callback_fn)(void *);
drivers/hv/hyperv_vmbus.h +20 -0
@@ -357,4 +357,24 @@ enum hvutil_device_state { HVUTIL_DEVICE_DYING, /* driver unload is in progress */ }; +#ifdef CONFIG_HYPERV_TESTING +#include <linux/debugfs.h> +#include <linux/delay.h> +#include <linux/err.h> +#define TESTING "hyperv" + +enum delay { +INTERRUPT_DELAY =0 , +MESSAGE_DELAY =1 , +}; + +int hv_debug_delay_files (struct hv_device *dev,struct dentry *root); +int hv_debug_add_dev_dir (struct hv_device *dev); +void hv_debug_rm_dev_dir (struct hv_device *dev); +void hv_debug_rm_all_dir (void ); +void hv_debug_set_dir_dentry (struct hv_device *dev,struct dentry *root); +void hv_debug_delay_test (struct vmbus_channel *channel,enum delay delay_type); + +#endif /* CONFIG_HYPERV_TESTING */ +#endif /* _HYPERV_VMBUS_H */
drivers/hv/ring_buffer.c +7 -0
@@ -396,6 +396,10 @@ struct vmpacket_descriptor *hv_pkt_iter_first(struct vmbus_channel *channel)struct hv_ring_buffer_info *rbi = &channel->inbound;struct vmpacket_descriptor *desc; +#ifdef CONFIG_HYPERV_TESTING +hv_debug_delay_test (channel,MESSAGE_DELAY ); +#endif /* CONFIG_HYPERV_TESTING */ +if (hv_pkt_iter_avail (rbi) <sizeof (struct vmpacket_descriptor ))return NULL ;
@@ -421,6 +425,9 @@ __hv_pkt_iter_next(struct vmbus_channel *channel,u32 packetlen = desc->len8 <<3 ;u32 dsize = rbi->ring_datasize; +#ifdef CONFIG_HYPERV_TESTING +hv_debug_delay_test (channel,MESSAGE_DELAY ); +#endif /* CONFIG_HYPERV_TESTING */ /* bump offset to next potential packet */ rbi->priv_read_index += packetlen +VMBUS_PKT_TRAILER ;if (rbi->priv_read_index >= dsize)
include/linux/hyperv.h +21 -0
@@ -926,6 +926,21 @@ struct vmbus_channel {* full outbound ring buffer. */ u64 out_full_first; + +#ifdef CONFIG_HYPERV_TESTING +/* enabling/disabling fuzz testing on the channel (default is false)*/ +bool fuzz_testing_state; + +/* Interrupt delay will delay the guest from emptying the ring buffer +* for a specific amount of time. The delay is in microseconds and will +* be between 1 to a maximum of 1000, its default is 0 (no delay). +* The Message delay will delay guest reading on a per message basis +* in microseconds between 1 to 1000 with the default being 0 +* (no delay). +*/ +u32 fuzz_testing_interrupt_delay; +u32 fuzz_testing_message_delay; +#endif /* CONFIG_HYPERV_TESTING */ };static inline bool is_hvsock_channel (const struct vmbus_channel *c)
@@ -1166,6 +1181,12 @@ struct hv_device {struct vmbus_channel *channel;struct kset *channels_kset; + +#ifdef CONFIG_HYPERV_TESTING +/* place holder to keep track of the dir for hv device in debugfs */ +struct dentry *debug_dir; +#endif /* CONFIG_HYPERV_TESTING */ + };
Signature · 1 line
2.17.1
PATCH v3 2/3 drivers: hv: vmbus: add test attributes to debugfs
In reply to Branden BonabyExpose the test parameters as part of the debugfs channel attributes. We will control the testing state via these attributes.
---
Changes in v3:
- Change call to IS_ERR_OR_NULL, to IS_ERR.
Changes in v2:
- Move test attributes to debugfs.
- Wrap test code under #ifdef statements.
- Add new documentation file under Documentation/ABI/testing.
- Make commit message reflect the change from from sysfs to debugfs.
Documentation/ABI/testing/debugfs-hyperv | 21 +++ MAINTAINERS | 1 + drivers/hv/vmbus_drv.c | 167 +++++++++++++++++++++++ 3 files changed, 189 insertions(+) create mode 100644 Documentation/ABI/testing/debugfs-hyperv
Documentation/ABI/testing/debugfs-hyperv new file +21 -0
@@ -0,0 +1,21 @@ +What: /sys/kernel/debug/hyperv/<UUID>/fuzz_test_state +Date: August 2019 +KernelVersion: 5.3 +Contact: Branden Bonaby <[email protected]> +Description: Fuzz testing status of a vmbus device, whether its in an ON + state or a OFF state +Users: Debugging tools + +What: /sys/kernel/debug/hyperv/<UUID>/delay/fuzz_test_buffer_interrupt_delay +Date: August 2019 +KernelVersion: 5.3 +Contact: Branden Bonaby <[email protected]> +Description: Fuzz testing buffer delay value between 0 - 1000 +Users: Debugging tools + +What: /sys/kernel/debug/hyperv/<UUID>/delay/fuzz_test_message_delay +Date: August 2019 +KernelVersion: 5.3 +Contact: Branden Bonaby <[email protected]> +Description: Fuzz testing message delay value between 0 - 1000 +Users: Debugging tools
MAINTAINERS +1 -0
@@ -7460,6 +7460,7 @@ F: include/uapi/linux/hyperv.h F: include/asm-generic/mshyperv.h F: tools/hv/ F: Documentation/ABI/stable/sysfs-bus-vmbus +F: Documentation/ABI/testing/debugfs-hyperv HYPERBUS SUPPORT M: Vignesh Raghavendra <[email protected]>
drivers/hv/vmbus_drv.c +167 -0
197 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v3 3/3 tools: hv: add vmbus testing tool
In reply to Branden BonabyThis is a userspace tool to drive the testing. Currently it supports introducing user specified delay in the host to guest communication path on a per-channel basis.
---
Changes in v3:
- Align python tool to match Linux coding style.
Changes in v2:
- Move testing location to new location in debugfs.
tools/hv/vmbus_testing | 342 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 342 insertions(+) create mode 100644 tools/hv/vmbus_testing
tools/hv/vmbus_testing new file +342 -0
@@ -0,0 +1,342 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# +# Program to allow users to fuzz test Hyper-V drivers +# by interfacing with Hyper-V debugfs directories +# author: Branden Bonaby + +import os +import cmd +import argparse +from collections import defaultdict +from argparse import RawDescriptionHelpFormatter + +# debugfs paths for vmbus must exist (same as in lsvmbus) +debugfs_sys_path = "/sys/kernel/debug/hyperv" +if not os.path.isdir(debugfs_sys_path): + print("{} doesn't exist/check permissions".format(debugfs_sys_path)) + exit(-1) +# Do not change unless, you change the debugfs attributes +# in "/sys/kernel/debug/hyperv/<UUID>/". All fuzz testing +# attributes will start with "fuzz_test". +pathlen = len(debugfs_sys_path) +fuzz_state_location = "fuzz_test_state" +fuzz_states = { + 0 : "Disable", + 1 : "Enable" +} + +fuzz_methods = { + 1 : "Delay_testing" +} + +fuzz_delay_types = { + 1 : "fuzz_test_buffer_interrupt_delay", + 2 : "fuzz_test_message_delay" +} + +def parse_args(): + parser = argparse.ArgumentParser(description = "vmbus_testing " + "[-s] [0|1] [-q] [-p] <debugfs-path>\n""vmbus_testing [-s]" + " [0|1] [-q][-p] <debugfs-path> delay [-d] [val][val] [-E|-D]\n" + "vmbus_testing [-q] disable-all\n" + "vmbus_testing [-q] view [-v|-V]\n" + "vmbus_testing --version", + epilog = "Current testing options {}".format(fuzz_methods), + prog = 'vmbus_testing', + formatter_class = RawDescriptionHelpFormatter) + subparsers = parser.add_subparsers(dest = "action") + parser.add_argument("--version", action = "version", + version = '%(prog)s 1.0') + parser.add_argument("-q","--quiet", action = "store_true", + help = "silence none important test messages") + parser.add_argument("-s","--state", metavar = "", type = int, + choices = range(0, 2), + help = "Turn testing ON or OFF for a single device." + " The value (1) will turn testing ON. The value" + " of (0) will turn testing OFF with the default set" + " to (0).") + parser.add_argument("-p","--path", metavar = "", + help = "Refers to the debugfs path to a vmbus device." + " If the path is not a valid path to a vmbus device," + " the program will exit. The path must be the" + " absolute path; use the lsvmbus command to find" + " the path.") + parser_delay = subparsers.add_parser("delay", + help = "Delay buffer/message reads in microseconds.", + description = "vmbus_testing -s [0|1] [-q] -p " + "<debugfs-path> delay -d " + "[buffer-delay-value] [message-delay-value]\n" + "vmbus_testing [-q] delay [buffer-delay-value] " + "[message-delay-value] -E\n" + "vmbus_testing [-q] delay [buffer-delay-value] " + "[message-delay-value] -D", + formatter_class = RawDescriptionHelpFormatter) + delay_group = parser_delay.add_mutually_exclusive_group() + delay_group.add_argument("-E", "--en_all", action = "store_true", + help = "Enable Buffer/Message Delay testing on ALL" + " devices. Use -d option with this to set the values" + " for both the buffer delay and the message delay. No" + " value can be (0) or less than (-1). If testing is" + " disabled on a device prior to running this command," + " testing will be enabled on the device as a result" + " of this command.") + delay_group.add_argument("-D", "--dis_all", action = "store_true", + help = "Disable Buffer/Message delay testing on ALL" + " devices. A value equal to (-1) will keep the" + " current delay value, and a value equal to (0) will" + " remove delay testing for the specfied delay column." + " only values (-1) and (0) will be accepted but at" + " least one value must be a (0) or a (-1).") + parser_delay.add_argument("-d", "--delay_time", metavar = "", nargs = 2, + type = check_range, default = [0, 0], required = (True), + help = "Buffer/message delay time. A value of (0) will" + "disable delay testing on the specified delay column," + " while a value of (-1) will ignore the specified" + " delay column. The default values are [0] & [0]." + " The first column represents the buffer delay value" + " and the second represents the message delay value." + " Value constraints: -1 <= value <= 1000.") + parser_dis_all = subparsers.add_parser("disable-all", + help = "Disable ALL testing on all vmbus devices.", + description = "vmbus_testing disable-all", + formatter_class = RawDescriptionHelpFormatter) + parser_view = subparsers.add_parser("view", + help = "View testing on vmbus devices.", + description = "vmbus_testing view -V\n" + "vmbus_testing -p <debugfs-path> view -v", + formatter_class = RawDescriptionHelpFormatter) + view_group = parser_view.add_mutually_exclusive_group() + view_group.add_argument("-V", "--view_all", action = "store_true", + help = "View the test status for all vmbus devices.") + view_group.add_argument("-v", "--view_single", action = "store_true", + help = "View test values for a single vmbus device.") + + return parser.parse_args() + +# value checking for range checking input in parser +def check_range(arg1): + try: + val = int(arg1) + except ValueError as err: + raise argparse.ArgumentTypeError(str(err)) + if val < -1 or val > 1000: + message = ("\n\nExpected -1 <= value <= 1000, got value" + " {}\n").format(val) + raise argparse.ArgumentTypeError(message) + return val + +def main(): + try: + dev_list = [] + for dir in os.listdir(debugfs_sys_path): + dev_list.append(os.path.join(debugfs_sys_path, dir)) + #key value, pairs + #key = debugfs device path + #value = list of fuzz testing attributes. + dev_files = defaultdict(list) + for dev in dev_list: + path = os.path.join(dev, "delay") + for f in os.listdir(path): + if (f.startswith("fuzz_test")): + dev_files[path].append(f) + + dev_files.default_factory = None + args = parse_args() + path = args.path + state = args.state + quiet = args.quiet + if (not quiet): + print("*** Use lsvmbus to get vmbus device type" + " information.*** ") + if (state is not None and validate_args_path(path, dev_list)): + if (state is not get_test_state(path)): + change_test_state(path, quiet) + state = get_test_state(path) + if (state == 0 and path is not None): + disable_testing_single_device(path, 0, quiet) + return + #Use subparsers as the key for different fuzz testing methods + if (args.action == "delay"): + delay = args.delay_time + if (validate_delay_values(args, delay)): + delay_test_all_devices(dev_list, delay, quiet) + elif (validate_args_path(path, dev_list)): + if(get_test_state(path) == 1): + delay_test_store(path, delay, quiet) + return + print("device testing OFF, use -s 1 to turn ON") + elif (args.action == "disable-all"): + disable_all_testing(dev_list, quiet) + elif (args.action == "view"): + if (args.view_all): + all_devices_test_status(dev_list) + elif (args.view_single): + if (validate_args_path(path, dev_list)): + device_test_values(dev_files, path) + return + print("Error,(check path) usage: -p"\ + " <debugfs device path> view -v") + except AttributeError: + print("check usage, 1 or more elements not provided") + exit(-1) + +# Validate delay values to make sure they are acceptable to +# to either enable all delays on a device or disable all +# delays on a device +def validate_delay_values(args, delay): + if (args.en_all): + for i in delay: + if (i < -1 or i == 0): + print("\nError, Values must be" + " equal to -1 or be > 0, use" + " -d option") + exit(-1) + return True + elif (args.dis_all): + for i in delay: + if (i < -1 or i > 0): + print("\nError, at least 1 value" + " is not a (0) or a (-1)") + exit(-1) + return True + else: + return False + + +# Validate argument path +def validate_args_path(path, dev_list): + if (path in dev_list): + return True + else: + return False + +# display Testing status of single device +def device_test_values(dev_files, path): + + delay_path = os.path.join(path, 'delay') + for test in dev_files.get(delay_path): + print("{}".format(test), end = '') + print((" value = {}")\ + .format(read_test_files(os.path.join(delay_path, test)))) + +# display Testing state of devices +def all_devices_test_status(dev_list): + for device in dev_list: + if (get_test_state(device) is 1): + print("Testing = ON for: {}".format(device.split("/")[5])) + else: + print("Testing = OFF for: {}".format(device.split("/")[5])) + +# read the vmbus device files, path must be absolute path before calling +def read_test_files(path): + try: + with open(path,"r") as f: + state = f.readline().strip() + if (state == 'N'): + state = 0 + elif (state == 'Y'): + state = 1 + return int(state) + + except IOError as e: + errno, strerror = e.args + print("I/O error({0}): {1} on file {2}" + .format(errno, strerror, path)) + exit(-1) + except ValueError: + print ("Element to int conversion error in: \n{}".format(path)) + exit(-1) + +# writing to vmbus device files, path must be absolute path before calling +def write_test_files(path, value): + try: + with open(path,"w") as f: + f.write("{}".format(value)) + except IOError as e: + errno, strerror = e.args + print("I/O error({0}): {1} on file {2}" + .format(errno, strerror, path)) + exit(-1) + +# change testing state of device +def change_test_state(device, quiet): + state_path = os.path.join(device, fuzz_state_location) + if (get_test_state(device) is 0): + write_test_files(state_path, 1) + if (not quiet): + print("Testing = ON for device: {}" + .format(state_path.split("/")[5])) + else: + write_test_files(state_path, 0) + if (not quiet): + print("Testing = OFF for device: {}" + .format(state_path.split("/")[5])) + +# get testing state of device +def get_test_state(device): + #state == 1 - test = ON + #state == 0 - test = OFF + return read_test_files(os.path.join(device, fuzz_state_location)) + +# Enter 1 - 1000 microseconds, into a single device using the +# fuzz_test_buffer_interrupt_delay and fuzz_test_message_delay +# debugfs attributes +def delay_test_store(device,delay_length, quiet): + + try: + # delay[0]- buffer delay, delay[1]- message delay + buff_test = os.path.join(os.path.sep,device, 'delay', + fuzz_delay_types.get(1)) + mess_test = os.path.join(os.path.sep,device, 'delay', + fuzz_delay_types.get(2)) + + if (delay_length[0] >= 0): + write_test_files(buff_test, delay_length[0]) + if (delay_length[1] >= 0): + write_test_files(mess_test, delay_length[1]) + if (not quiet): + print("Buffer delay testing = {} for: {}" + .format(read_test_files(buff_test), + buff_test.split("/")[5])) + print("Message delay testing = {} for: {}" + .format(read_test_files(mess_test), + mess_test.split("/")[5])) + except IOError as e: + errno, strerror = e.args + print("I/O error({0}): {1} on files {2}{3}" + .format(errno, strerror, buff_test, mess_test)) + exit(-1) + +#enabling/disabling delay testing on all devices +def delay_test_all_devices(dev_list,delay,quiet): + + for device in (dev_list): + if (get_test_state(device) is 0): + change_test_state(device,quiet) + delay_test_store(device, delay, quiet) + +#disabling testing on single device +def disable_testing_single_device(device,test_type,quiet): + + #test_type represents corresponding key + #delay method in delay_methods dict. + #special type 0 , used to disable all + #testing on SINGLE device. + + if (test_type is 1 or test_type is 0): + #disable list [buffer,message] + disable_delay = [0, 0] + if (get_test_state(device) is 1): + change_test_state(device, quiet) + delay_test_store(device, disable_delay, quiet) + +#disabling testing on ALL devices +def disable_all_testing(dev_list,quiet): + + #delay disable list [buffer,message] + for device in dev_list: + disable_testing_single_device(device, 0, quiet) + +if __name__ == "__main__": + main()
Signature · 1 line
2.17.1
RE: [PATCH v3 2/3] drivers: hv: vmbus: add test attributes to debugfs
In reply to Branden BonabyFrom: Branden Bonaby <[email protected]> Sent: Tuesday, August 20, 2019 4:39 PM
Quoted 28 lines
Expose the test parameters as part of the debugfs channel attributes. We will control the testing state via these attributes.
Signed-off-by Branden Bonaby <[email protected]>---
Changes in v3:
- Change call toIS_ERR_OR_NULL, toIS_ERR.Changes in v2:
- Move test attributes to debugfs.
- Wrap test code under #ifdef statements.
- Add new documentation file underDocumentation/ABI/testing.
- Make commit message reflect the change from from sysfs to debugfs.Documentation/ABI/testing/debugfs-hyperv | 21 +++ MAINTAINERS | 1 + drivers/hv/vmbus_drv.c | 167 +++++++++++++++++++++++ 3 files changed, 189 insertions(+) create mode 100644 Documentation/ABI/testing/debugfs-hypervDocumentation/ABI/testing/debugfs-hyperv new file +0
-0
Documentation/ABI/testing/debugfs-hyperv line 6
Quoted 6 lines
Documentation/ABI/testing/debugfs-hyperv new file +6
-0@@ -0,0 +1,21 @@ +What: /sys/kernel/debug/hyperv/<UUID>/fuzz_test_state +Date: August 2019 +KernelVersion: 5.3 +Contact: Branden Bonaby <[email protected]> +Description: Fuzz testing status of a vmbus device, whether its in an ON + state or a OFF state
Document what values are actually returned?
Documentation/ABI/testing/debugfs-hyperv line 13
Quoted 7 lines
Documentation/ABI/testing/debugfs-hyperv new file +7
-0+Users: Debugging tools + +What: /sys/kernel/debug/hyperv/<UUID>/delay/fuzz_test_buffer_interrupt_delay +Date: August 2019 +KernelVersion: 5.3 +Contact: Branden Bonaby <[email protected]> +Description: Fuzz testing buffer delay value between 0 - 1000
It would be helpful to document the units -- I think this is 0 to 1000 microseconds.
Documentation/ABI/testing/debugfs-hyperv line 20
Quoted 7 lines
Documentation/ABI/testing/debugfs-hyperv new file +7
-0+Users: Debugging tools + +What: /sys/kernel/debug/hyperv/<UUID>/delay/fuzz_test_message_delay +Date: August 2019 +KernelVersion: 5.3 +Contact: Branden Bonaby <[email protected]> +Description: Fuzz testing message delay value between 0 - 1000
Same here.
Quoted 18 lines
Documentation/ABI/testing/debugfs-hyperv new file +1
-0+Users: Debugging toolsMAINTAINERS +1
-0@@ -7460,6 +7460,7 @@ F: include/uapi/linux/hyperv.h F: include/asm-generic/mshyperv.h F: tools/hv/ F: Documentation/ABI/stable/sysfs-bus-vmbus +F: Documentation/ABI/testing/debugfs-hyperv HYPERBUS SUPPORT M: Vignesh Raghavendra <[email protected]>drivers/hv/vmbus_drv.c +0
-0
drivers/hv/vmbus_drv.c static void vmbus_device_release(struct device *device) · line 924
Quoted 6 lines
drivers/hv/vmbus_drv.c +3
-0@@ -919,6 +919,10 @@ static void vmbus_device_release(struct device *device)struct hv_device *hv_dev =device_to_hv_device (device);struct vmbus_channel *channel = hv_dev->channel; +#ifdef CONFIG_HYPERV_TESTING +hv_debug_rm_dev_dir (hv_dev); +#endif /* CONFIG_HYPERV_TESTING */
Same comment in as previous patch about #ifdef inline in the code, and similarly for other occurrences in this patch.
Quoted 25 lines
drivers/hv/vmbus_drv.c +11
-0+mutex_lock (&vmbus_connection.channel_mutex);hv_process_channel_removal (channel);mutex_unlock (&vmbus_connection.channel_mutex);@@ -1727,6 +1731,9 @@ int vmbus_device_register(struct hv_device *child_device_obj)pr_err ("Unable to register primary channeln" ); goto err_kset_unregister; } +#ifdef CONFIG_HYPERV_TESTING +hv_debug_add_dev_dir (child_device_obj); +#endif /* CONFIG_HYPERV_TESTING */ return 0 ;@@ -2086,6 +2093,159 @@ static void hv_crash_handler(struct pt_regs *regs)hyperv_cleanup (); }; +#ifdef CONFIG_HYPERV_TESTING + +struct dentry *hv_root; + +static int hv_debugfs_delay_get (void *data,u64 *val) +{ + *val = *(u32 *)data;
drivers/hv/vmbus_drv.c static void hv_crash_handler(struct pt_regs *regs) · line 2112
Quoted 10 lines
drivers/hv/vmbus_drv.c +10
-0+return 0 ; +} + +static int hv_debugfs_delay_set (void *data,u64 val) +{ +if (val >=1 && val <=1000 ) + *(u32 *)data = val; +/*Best to not use else statement here since we want +* the delay to remain the same if val > 1000 +*/
The standard multi-line comment style would be:
/* * Best to not use else statement here since we want * the delay to remain the same if val > 1000 */
drivers/hv/vmbus_drv.c static void hv_crash_handler(struct pt_regs *regs) · line 2114
Quoted 2 lines
drivers/hv/vmbus_drv.c +2
-0+else if (val <=0 ) + *(u32 *)data =0 ;
You could consider returning an error for an invalid value (< 0, or > 1000).
drivers/hv/vmbus_drv.c static void hv_crash_handler(struct pt_regs *regs) · line 2120
Quoted 6 lines
drivers/hv/vmbus_drv.c +6
-0+return 0 ; +} + +DEFINE_DEBUGFS_ATTRIBUTE (hv_debugfs_delay_fops, hv_debugfs_delay_get, + hv_debugfs_delay_set,"%llu\n" ); +
Michael
RE: [PATCH v3 3/3] tools: hv: add vmbus testing tool
In reply to Branden BonabyTool function issues: Please validate args errors for '-p' and '--path', in or following validate_args_path().
Comments of functionality:
- it's confusing when fuzz_testing are all OFF, then user run ' python3 /home/lisa/vmbus_testing -p /sys/kernel/debug/hyperv/000d3a6e-4548-000d-3a6e-4548000d3a6e delay -d 0 0 -D ' which will enable all delay testing state ('Y' in state files). even I used "-D", "--dis_all" param.
- if we have subparsers of "disable-all" for the testing tool, then probably we don't need the mutually_exclusive_group under subparsers of "delay"
- the path argument (-p) could be an argument for subparsers of "delay" and "view" only.
Regards,
Harry
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Branden Bonaby
Sent: Tuesday, August 20, 2019 4:40 PM
To: KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; Stephen Hemminger <[email protected]>; [email protected]
Cc: brandonbonaby94 <[email protected]>; [email protected]; [email protected]
Subject: [PATCH v3 3/3] tools: hv: add vmbus testing tool
This is a userspace tool to drive the testing. Currently it supports introducing user specified delay in the host to guest communication path on a per-channel basis.
---
Changes in v3:
- Align python tool to match Linux coding style.
Changes in v2:
- Move testing location to new location in debugfs.
tools/hv/vmbus_testing | 342 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 342 insertions(+) create mode 100644 tools/hv/vmbus_testing
tools/hv/vmbus_testing new file +342 -0
@@ -0,0 +1,342 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# +# Program to allow users to fuzz test Hyper-V drivers # by interfacing +with Hyper-V debugfs directories # author: Branden Bonaby + +import os +import cmd +import argparse +from collections import defaultdict +from argparse import RawDescriptionHelpFormatter + +# debugfs paths for vmbus must exist (same as in lsvmbus) +debugfs_sys_path = "/sys/kernel/debug/hyperv" +if not os.path.isdir(debugfs_sys_path): + print("{} doesn't exist/check permissions".format(debugfs_sys_path)) + exit(-1) +# Do not change unless, you change the debugfs attributes # in +"/sys/kernel/debug/hyperv/<UUID>/". All fuzz testing # attributes will +start with "fuzz_test". +pathlen = len(debugfs_sys_path) +fuzz_state_location = "fuzz_test_state" +fuzz_states = { + 0 : "Disable", + 1 : "Enable" +} + +fuzz_methods = { + 1 : "Delay_testing" +} + +fuzz_delay_types = { + 1 : "fuzz_test_buffer_interrupt_delay", + 2 : "fuzz_test_message_delay" +} + +def parse_args(): + parser = argparse.ArgumentParser(description = "vmbus_testing " + "[-s] [0|1] [-q] [-p] <debugfs-path>\n""vmbus_testing [-s]" + " [0|1] [-q][-p] <debugfs-path> delay [-d] [val][val] [-E|-D]\n" + "vmbus_testing [-q] disable-all\n" + "vmbus_testing [-q] view [-v|-V]\n" + "vmbus_testing --version", + epilog = "Current testing options {}".format(fuzz_methods), + prog = 'vmbus_testing', + formatter_class = RawDescriptionHelpFormatter) + subparsers = parser.add_subparsers(dest = "action") + parser.add_argument("--version", action = "version", + version = '%(prog)s 1.0') + parser.add_argument("-q","--quiet", action = "store_true", + help = "silence none important test messages") + parser.add_argument("-s","--state", metavar = "", type = int, + choices = range(0, 2), + help = "Turn testing ON or OFF for a single device." + " The value (1) will turn testing ON. The value" + " of (0) will turn testing OFF with the default set" + " to (0).") + parser.add_argument("-p","--path", metavar = "", + help = "Refers to the debugfs path to a vmbus device." + " If the path is not a valid path to a vmbus device," + " the program will exit. The path must be the" + " absolute path; use the lsvmbus command to find" + " the path.") + parser_delay = subparsers.add_parser("delay", + help = "Delay buffer/message reads in microseconds.", + description = "vmbus_testing -s [0|1] [-q] -p " + "<debugfs-path> delay -d " + "[buffer-delay-value] [message-delay-value]\n" + "vmbus_testing [-q] delay [buffer-delay-value] " + "[message-delay-value] -E\n" + "vmbus_testing [-q] delay [buffer-delay-value] " + "[message-delay-value] -D", + formatter_class = RawDescriptionHelpFormatter) + delay_group = parser_delay.add_mutually_exclusive_group() + delay_group.add_argument("-E", "--en_all", action = "store_true", + help = "Enable Buffer/Message Delay testing on ALL" + " devices. Use -d option with this to set the values" + " for both the buffer delay and the message delay. No" + " value can be (0) or less than (-1). If testing is" + " disabled on a device prior to running this command," + " testing will be enabled on the device as a result" + " of this command.") + delay_group.add_argument("-D", "--dis_all", action = "store_true", + help = "Disable Buffer/Message delay testing on ALL" + " devices. A value equal to (-1) will keep the" + " current delay value, and a value equal to (0) will" + " remove delay testing for the specfied delay column." + " only values (-1) and (0) will be accepted but at" + " least one value must be a (0) or a (-1).") + parser_delay.add_argument("-d", "--delay_time", metavar = "", nargs = 2, + type = check_range, default = [0, 0], required = (True), + help = "Buffer/message delay time. A value of (0) will" + "disable delay testing on the specified delay column," + " while a value of (-1) will ignore the specified" + " delay column. The default values are [0] & [0]." + " The first column represents the buffer delay value" + " and the second represents the message delay value." + " Value constraints: -1 <= value <= 1000.") + parser_dis_all = subparsers.add_parser("disable-all", + help = "Disable ALL testing on all vmbus devices.", + description = "vmbus_testing disable-all", + formatter_class = RawDescriptionHelpFormatter) + parser_view = subparsers.add_parser("view", + help = "View testing on vmbus devices.", + description = "vmbus_testing view -V\n" + "vmbus_testing -p <debugfs-path> view -v", + formatter_class = RawDescriptionHelpFormatter) + view_group = parser_view.add_mutually_exclusive_group() + view_group.add_argument("-V", "--view_all", action = "store_true", + help = "View the test status for all vmbus devices.") + view_group.add_argument("-v", "--view_single", action = "store_true", + help = "View test values for a single vmbus +device.") + + return parser.parse_args() + +# value checking for range checking input in parser def +check_range(arg1): + try: + val = int(arg1) + except ValueError as err: + raise argparse.ArgumentTypeError(str(err)) + if val < -1 or val > 1000: + message = ("\n\nExpected -1 <= value <= 1000, got value" + " {}\n").format(val) + raise argparse.ArgumentTypeError(message) + return val + +def main(): + try: + dev_list = [] + for dir in os.listdir(debugfs_sys_path): + dev_list.append(os.path.join(debugfs_sys_path, dir)) + #key value, pairs + #key = debugfs device path + #value = list of fuzz testing attributes. + dev_files = defaultdict(list) + for dev in dev_list: + path = os.path.join(dev, "delay") + for f in os.listdir(path): + if (f.startswith("fuzz_test")): + dev_files[path].append(f) + + dev_files.default_factory = None + args = parse_args() + path = args.path + state = args.state + quiet = args.quiet + if (not quiet): + print("*** Use lsvmbus to get vmbus device type" + " information.*** ") + if (state is not None and validate_args_path(path, dev_list)): + if (state is not get_test_state(path)): + change_test_state(path, quiet) + state = get_test_state(path) + if (state == 0 and path is not None): + disable_testing_single_device(path, 0, quiet) + return + #Use subparsers as the key for different fuzz testing methods + if (args.action == "delay"): + delay = args.delay_time + if (validate_delay_values(args, delay)): + delay_test_all_devices(dev_list, delay, quiet) + elif (validate_args_path(path, dev_list)): + if(get_test_state(path) == 1): + delay_test_store(path, delay, quiet) + return + print("device testing OFF, use -s 1 to turn ON") + elif (args.action == "disable-all"): + disable_all_testing(dev_list, quiet) + elif (args.action == "view"): + if (args.view_all): + all_devices_test_status(dev_list) + elif (args.view_single): + if (validate_args_path(path, dev_list)): + device_test_values(dev_files, path) + return + print("Error,(check path) usage: -p"\ + " <debugfs device path> view -v") + except AttributeError: + print("check usage, 1 or more elements not provided") + exit(-1) + +# Validate delay values to make sure they are acceptable to # to either +enable all delays on a device or disable all # delays on a device def +validate_delay_values(args, delay): + if (args.en_all): + for i in delay: + if (i < -1 or i == 0): + print("\nError, Values must be" + " equal to -1 or be > 0, use" + " -d option") + exit(-1) + return True + elif (args.dis_all): + for i in delay: + if (i < -1 or i > 0): + print("\nError, at least 1 value" + " is not a (0) or a (-1)") + exit(-1) + return True + else: + return False + + +# Validate argument path +def validate_args_path(path, dev_list): + if (path in dev_list): + return True + else: + return False + +# display Testing status of single device def +device_test_values(dev_files, path): + + delay_path = os.path.join(path, 'delay') + for test in dev_files.get(delay_path): + print("{}".format(test), end = '') + print((" value = {}")\ + + .format(read_test_files(os.path.join(delay_path, test)))) + +# display Testing state of devices +def all_devices_test_status(dev_list): + for device in dev_list: + if (get_test_state(device) is 1): + print("Testing = ON for: {}".format(device.split("/")[5])) + else: + print("Testing = OFF for: +{}".format(device.split("/")[5])) + +# read the vmbus device files, path must be absolute path before +calling def read_test_files(path): + try: + with open(path,"r") as f: + state = f.readline().strip() + if (state == 'N'): + state = 0 + elif (state == 'Y'): + state = 1 + return int(state) + + except IOError as e: + errno, strerror = e.args + print("I/O error({0}): {1} on file {2}" + .format(errno, strerror, path)) + exit(-1) + except ValueError: + print ("Element to int conversion error in: \n{}".format(path)) + exit(-1) + +# writing to vmbus device files, path must be absolute path before +calling def write_test_files(path, value): + try: + with open(path,"w") as f: + f.write("{}".format(value)) + except IOError as e: + errno, strerror = e.args + print("I/O error({0}): {1} on file {2}" + .format(errno, strerror, path)) + exit(-1) + +# change testing state of device +def change_test_state(device, quiet): + state_path = os.path.join(device, fuzz_state_location) + if (get_test_state(device) is 0): + write_test_files(state_path, 1) + if (not quiet): + print("Testing = ON for device: {}" + .format(state_path.split("/")[5])) + else: + write_test_files(state_path, 0) + if (not quiet): + print("Testing = OFF for device: {}" + .format(state_path.split("/")[5])) + +# get testing state of device +def get_test_state(device): + #state == 1 - test = ON + #state == 0 - test = OFF + return read_test_files(os.path.join(device, +fuzz_state_location)) + +# Enter 1 - 1000 microseconds, into a single device using the # +fuzz_test_buffer_interrupt_delay and fuzz_test_message_delay # debugfs +attributes def delay_test_store(device,delay_length, quiet): + + try: + # delay[0]- buffer delay, delay[1]- message delay + buff_test = os.path.join(os.path.sep,device, 'delay', + fuzz_delay_types.get(1)) + mess_test = os.path.join(os.path.sep,device, 'delay', + fuzz_delay_types.get(2)) + + if (delay_length[0] >= 0): + write_test_files(buff_test, delay_length[0]) + if (delay_length[1] >= 0): + write_test_files(mess_test, delay_length[1]) + if (not quiet): + print("Buffer delay testing = {} for: {}" + .format(read_test_files(buff_test), + buff_test.split("/")[5])) + print("Message delay testing = {} for: {}" + .format(read_test_files(mess_test), + mess_test.split("/")[5])) + except IOError as e: + errno, strerror = e.args + print("I/O error({0}): {1} on files {2}{3}" + .format(errno, strerror, buff_test, mess_test)) + exit(-1) + +#enabling/disabling delay testing on all devices def +delay_test_all_devices(dev_list,delay,quiet): + + for device in (dev_list): + if (get_test_state(device) is 0): + change_test_state(device,quiet) + delay_test_store(device, delay, quiet) + +#disabling testing on single device +def disable_testing_single_device(device,test_type,quiet): + + #test_type represents corresponding key + #delay method in delay_methods dict. + #special type 0 , used to disable all + #testing on SINGLE device. + + if (test_type is 1 or test_type is 0): + #disable list [buffer,message] + disable_delay = [0, 0] + if (get_test_state(device) is 1): + change_test_state(device, quiet) + delay_test_store(device, disable_delay, quiet) + +#disabling testing on ALL devices +def disable_all_testing(dev_list,quiet): + + #delay disable list [buffer,message] + for device in dev_list: + disable_testing_single_device(device, 0, quiet) + +if __name__ == "__main__":
+ main()
--
2.17.1
Re: [PATCH v3 3/3] tools: hv: add vmbus testing tool
Quoted 10 lines · Harry Zhang
Tool function issues: Please validate args errors for '-p' and '--path', in or following
validate_args_path().Comments of functionality:
(quoted diff) +0
-3- it's confusing when fuzz_testing are all OFF, then user run ' python3 /home/lisa/vmbus_testing -p /sys/kernel/debug/hyperv/000d3a6e-4548-000d-3a6e-4548000d3a6e delay -d 0 0 -D ' which will enable all delay testing state ('Y' in state files). even I used "-D", "--dis_all" param. - if we have subparsers of "disable-all" for the testing tool, then probably we don't need the mutually_exclusive_group under subparsers of "delay" - the path argument (-p) could be an argument for subparsers of "delay" and "view" only.Regards,
Harry
So I made the choice to keep disabling the state and disabling delay testing seperate, because once we start adding other testing options you wouldn't want to inadvertently disable all testing especially if you were doing more than one type of test at a time. So with your configuration
'python3 /home/lisa/vmbus_testing -p /sys/kernel/debug/hyperv/000d3a6e-4548-000d-3a6e-4548000d3a6e delay -d 0 0 -D '
this would stop all delay testing on all the devices but wouldn't change their test state to OFF 'N'.So thats why I have the option -s --state to change the state to Off with a -s 0. Then to disable all types of testing and change the state to OFF thats where the 'disable-all' subparser comes in. with:
'python3 /home/lisa/vmbus_testing disable-all
For that last point I don't understand what you mean, are you saying it would be better to have something like this using delay as an example?
'python3 /home/lisa/vmbus_testing delay -p /sys/kernel/debug/hyperv/000d3a6e-4548-000d-3a6e-4548000d3a6e'
If thats what you mean I figured it was better to make the -p accessible to all test type so I made it apart of the main parser. This would allow us to just have it there once instead of having to make a -p for every subparser.
Also maybe I need to change the examples and the help text because with the -D option for delay you wouldnt actually need to put in the path. As
'python3 /home/lisa/vmbus_testing delay -d 0 0 -D '
would suffice to stop delay testing on all devices; -E would enable it for all devices and change the state to On 'Y' if it wasn't already.
let me know your thoughts
branden bonaby
Re: [PATCH v3 2/3] drivers: hv: vmbus: add test attributes to debugfs
In reply to Michael KelleyQuoted 9 lines
Documentation/ABI/testing/debugfs-hyperv line 6
Quoted 6 lines
Documentation/ABI/testing/debugfs-hyperv new file +6
-0+What: /sys/kernel/debug/hyperv/<UUID>/fuzz_test_state +Date: August 2019 +KernelVersion: 5.3 +Contact: Branden Bonaby <[email protected]> +Description: Fuzz testing status of a vmbus device, whether its in an ON + state or a OFF stateDocument what values are actually returned?
Quoted 10 lines
Documentation/ABI/testing/debugfs-hyperv line 13
Quoted 7 lines
Documentation/ABI/testing/debugfs-hyperv new file +7
-0+Users: Debugging tools + +What: /sys/kernel/debug/hyperv/<UUID>/delay/fuzz_test_buffer_interrupt_delay +Date: August 2019 +KernelVersion: 5.3 +Contact: Branden Bonaby <[email protected]> +Description: Fuzz testing buffer delay value between 0 - 1000It would be helpful to document the units -- I think this is 0 to 1000 microseconds.
you're right, that makes sense I'll add that information in. Also to confirm, it is microseconds like you said.
Quoted 10 lines
drivers/hv/vmbus_drv.c static void hv_crash_handler(struct pt_regs *regs) · line 2112
Quoted 7 lines
drivers/hv/vmbus_drv.c +7
-0+static int hv_debugfs_delay_set (void *data,u64 val) +{ +if (val >=1 && val <=1000 ) + *(u32 *)data = val; +/*Best to not use else statement here since we want +* the delay to remain the same if val > 1000 +*/ The standard multi-line comment style would be:
Quoted 5 lines
/* * Best to not use else statement here since we want * the delay to remain the same if val > 1000 */
will change
Quoted 6 lines
drivers/hv/vmbus_drv.c static void hv_crash_handler(struct pt_regs *regs) · line 2114
Quoted 2 lines
drivers/hv/vmbus_drv.c +2
-0+else if (val <=0 ) + *(u32 *)data =0 ;You could consider returning an error for an invalid value (< 0, or > 1000).
its subtle but it does make sense and shows anyone reading that the only acceptable values in the function are 0 <= 1000 at a glance. I'll add that in.
Re: [PATCH v3 1/3] drivers: hv: vmbus: Introduce latency testing
In reply to Branden BonabyQuoted 6 lines · On Tue, 20 Aug 2019 16:39:02 -0700 "Branden Bonaby"…
drivers/hv/Kconfig +0
-0@@ -29,4 +29,11 @@ config HYPERV_BALLOON help
drivers/hv/Kconfig config HYPERV_BALLOON · line 39
Quoted 10 lines
drivers/hv/Kconfig +7
-0Select this option to enable Hyper-V Balloon driver. +config HYPERV_TESTING + bool "Hyper-V testing" + default n + depends on HYPERV && DEBUG_FS + help + Select this option to enable Hyper-V vmbus testing. + endmenu
Maybe this should go under the Kernel hacking
section in lib/Kconfig.debug?
Re: [PATCH v3 1/3] drivers: hv: vmbus: Introduce latency testing
Quoted 11 lines · Stephen Hemminger
Quoted 8 lines · On Tue, 20 Aug 2019 16:39:02 -0700 "Branden Bonaby"…
drivers/hv/Kconfig +0
-0@@ -29,4 +29,11 @@ config HYPERV_BALLOON help Select this option to enable Hyper-V Balloon driver.
Quoted 12 lines
drivers/hv/Kconfig config HYPERV_BALLOON · line 39
Quoted 8 lines
drivers/hv/Kconfig +7
-0+config HYPERV_TESTING + bool "Hyper-V testing" + default n + depends on HYPERV && DEBUG_FS + help + Select this option to enable Hyper-V vmbus testing. + endmenuMaybe this should go under the Kernel hacking
section inlib/Kconfig.debug?
These lines are the same in the v4 patch I sent a bit after this so in my v5 version I'll update this.
Thanks