kcov: collect coverage from usb and vhost
Subsystems (10)
v2 cover kcov: collect coverage from usb and vhost
This patchset extends kcov to allow collecting coverage from the USB subsystem and vhost workers. See the first patch description for details about the kcov extension. The other two patches apply this kcov extension to USB and vhost.
These patches have been used to enable coverage-guided USB fuzzing with syzkaller for the last few years, see the details here:
https://github.com/google/syzkaller/blob/master/docs/linux/external_fuzzing_usb.md
This patchset has been pushed to the public Linux kernel Gerrit instance:
https://linux-review.googlesource.com/c/linux/kernel/git/torvalds/linux/+/1524
Changes v1 -> v2:
- Changed common_handle type back to u64 (to allow extending it in the future).
- Reworked kcov_remote_handle() helpers.
- Fixed vhost annotations when CONFIG_KCOV is not enabled.
- Use kcov_disable() instead of kcov_remote_reset() when KCOV_REMOTE_ENABLE fails.
Changes RFC v1 -> v1:
- Remove unnecessary #ifdef's from drivers/vhost/vhost.c.
- Reset t->kcov when area allocation fails in kcov_remote_start().
- Use struct_size to calculate array size in kcov_ioctl().
- Add a limit on area_size in kcov_remote_arg.
- Added kcov_disable() helper.
- Changed encoding of kcov remote handle ids, see the documentation.
- Added a comment reference for kcov_sequence task_struct field.
- Change common_handle type to u32.
- Add checks for handle validity into kcov_ioctl_locked() and kcov_remote_start().
- Updated documentation to reflect the changes.
Andrey Konovalov (3):
kcov: remote coverage support
usb, kcov: collect coverage from hub_event
vhost, kcov: collect coverage from vhost_worker
Documentation/dev-tools/kcov.rst | 128 +++++++++ drivers/usb/core/hub.c | 5 + drivers/vhost/vhost.c | 6 + drivers/vhost/vhost.h | 1 + include/linux/kcov.h | 23 ++ include/linux/sched.h | 6 + include/uapi/linux/kcov.h | 25 ++ kernel/kcov.c | 480 ++++++++++++++++++++++++++++--- 8 files changed, 639 insertions(+), 35 deletions(-)
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
PATCH v2 1/3 kcov: remote coverage support
This patch adds background thread coverage collection ability to kcov.
With KCOV_ENABLE coverage is collected only for syscalls that are issued from the current process. With KCOV_REMOTE_ENABLE it's possible to collect coverage for arbitrary parts of the kernel code, provided that those parts are annotated with kcov_remote_start()/kcov_remote_stop().
This allows to collect coverage from two types of kernel background
threads: the global ones, that are spawned during kernel boot and are always running (e.g. USB hub_event()); and the local ones, that are spawned when a user interacts with some kernel interface (e.g. vhost workers).
To enable collecting coverage from a global background thread, a unique global handle must be assigned and passed to the corresponding kcov_remote_start() call. Then a userspace process can pass a list of such handles to the KCOV_REMOTE_ENABLE ioctl in the handles array field of the kcov_remote_arg struct. This will attach the used kcov device to the code sections, that are referenced by those handles.
Since there might be many local background threads spawned from different userspace processes, we can't use a single global handle per annotation. Instead, the userspace process passes a non-zero handle through the common_handle field of the kcov_remote_arg struct. This common handle gets saved to the kcov_handle field in the current task_struct and needs to be passed to the newly spawned threads via custom annotations. Those threads should in turn be annotated with kcov_remote_start()/kcov_remote_stop().
Internally kcov stores handles as u64 integers. The top byte of a handle is used to denote the id of a subsystem that this handle belongs to, and the lower 4 bytes are used to denote a handle id within that subsystem. A reserved value 0 is used as a subsystem id for common handles as they don't belong to a particular subsystem. The bytes 4-7 are currently reserved and must be zero. In the future the number of bytes used for the subsystem or handle ids might be increased.
When a particular userspace proccess collects coverage by via a common handle, kcov will collect coverage for each code section that is annotated to use the common handle obtained as kcov_handle from the current task_struct. However non common handles allow to collect coverage selectively from different subsystems.
--- Documentation/dev-tools/kcov.rst | 128 +++++++++ include/linux/kcov.h | 23 ++ include/linux/sched.h | 6 + include/uapi/linux/kcov.h | 25 ++ kernel/kcov.c | 480 ++++++++++++++++++++++++++++--- 5 files changed, 627 insertions(+), 35 deletions(-)
Documentation/dev-tools/kcov.rst +128 -0
@@ -34,6 +34,7 @@ Profiling data will only become accessible once debugfs has been mounted:: Coverage collection ------------------- + The following program demonstrates coverage collection from within a test program using kcov:
@@ -128,6 +129,7 @@ only need to enable coverage (disable happens automatically on thread end). Comparison operands collection ------------------------------ + Comparison operands collection is similar to coverage collection: .. code-block:: c
@@ -202,3 +204,129 @@ Comparison operands collection is similar to coverage collection: Note that the kcov modes (coverage collection or comparison operands) are mutually exclusive. + +Remote coverage collection +-------------------------- + +With KCOV_ENABLE coverage is collected only for syscalls that are issued from +the current process. With KCOV_REMOTE_ENABLE it's possible to collect coverage +for arbitrary parts of the kernel code, provided that those parts are annotated +with kcov_remote_start()/kcov_remote_stop(). + +This allows to collect coverage from two types of kernel background threads: +the global ones, that are spawned during kernel boot and are always running +(e.g. USB hub_event()); and the local ones, that are spawned when a user +interacts with some kernel interface (e.g. vhost workers). + +To enable collecting coverage from a global background thread, a unique global +handle must be assigned and passed to the corresponding kcov_remote_start() +call. Then a userspace process can pass a list of such handles to the +KCOV_REMOTE_ENABLE ioctl in the handles array field of the kcov_remote_arg +struct. This will attach the used kcov device to the code sections, that are +referenced by those handles. + +Since there might be many local background threads spawned from different +userspace processes, we can't use a single global handle per annotation. +Instead, the userspace process passes a non-zero handle through the +common_handle field of the kcov_remote_arg struct. This common handle gets +saved to the kcov_handle field in the current task_struct and needs to be +passed to the newly spawned threads via custom annotations. Those threads +should in turn be annotated with kcov_remote_start()/kcov_remote_stop(). + +Internally kcov stores handles as u64 integers. The top byte of a handle is +used to denote the id of a subsystem that this handle belongs to, and the lower +4 bytes are used to denote a handle id within that subsystem. A reserved value +0 is used as a subsystem id for common handles as they don't belong to a +particular subsystem. The bytes 4-7 are currently reserved and must be zero. +In the future the number of bytes used for the subsystem or handle ids might +be increased. + +When a particular userspace proccess collects coverage by via a common handle, +kcov will collect coverage for each code section that is annotated to use the +common handle obtained as kcov_handle from the current task_struct. However +non common handles allow to collect coverage selectively from different +subsystems. + +.. code-block:: c + + struct kcov_remote_arg { + unsigned trace_mode; + unsigned area_size; + unsigned num_handles; + uint64_t common_handle; + uint64_t handles[0]; + }; + + #define KCOV_INIT_TRACE _IOR('c', 1, unsigned long) + #define KCOV_DISABLE _IO('c', 101) + #define KCOV_REMOTE_ENABLE _IOW('c', 102, struct kcov_remote_arg) + + #define COVER_SIZE (64 << 10) + + #define KCOV_TRACE_PC 0 + + #define KCOV_SUBSYSTEM_COMMON (0x00ull << 56) + #define KCOV_SUBSYSTEM_USB (0x01ull << 56) + + #define KCOV_SUBSYSTEM_MASK (0xffull << 56) + #define KCOV_HANDLE_ID_MASK (0xffffffffull) + + static inline __u64 kcov_remote_handle(__u64 subsys, __u64 id) + { + if (subsys & ~KCOV_SUBSYSTEM_MASK || id & ~KCOV_HANDLE_ID_MASK) + return 0; + return subsys | id; + } + + #define KCOV_COMMON_ID 0x42 + #define KCOV_USB_BUS_NUM 1 + + int main(int argc, char **argv) + { + int fd; + unsigned long *cover, n, i; + struct kcov_remote_arg *arg; + + fd = open("/sys/kernel/debug/kcov", O_RDWR); + if (fd == -1) + perror("open"), exit(1); + if (ioctl(fd, KCOV_INIT_TRACE, COVER_SIZE)) + perror("ioctl"), exit(1); + cover = (unsigned long*)mmap(NULL, COVER_SIZE * sizeof(unsigned long), + PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); + if ((void*)cover == MAP_FAILED) + perror("mmap"), exit(1); + + /* Enable coverage collection via common handle and from USB bus #1. */ + arg = calloc(1, sizeof(*arg) + sizeof(uint64_t)); + if (!arg) + perror("calloc"), exit(1); + arg->trace_mode = KCOV_TRACE_PC; + arg->area_size = COVER_SIZE; + arg->num_handles = 1; + arg->common_handle = kcov_remote_handle(KCOV_SUBSYSTEM_COMMON, + KCOV_COMMON_ID); + arg->handles[0] = kcov_remote_handle(KCOV_SUBSYSTEM_USB, + KCOV_USB_BUS_NUM); + if (ioctl(fd, KCOV_REMOTE_ENABLE, arg)) + perror("ioctl"), free(arg), exit(1); + free(arg); + + /* + * Here the user needs to trigger execution of a kernel code section + * that is either annotated with the common handle, or to trigger some + * activity on USB bus #1. + */ + sleep(2); + + n = __atomic_load_n(&cover[0], __ATOMIC_RELAXED); + for (i = 0; i < n; i++) + printf("0x%lx\n", cover[i + 1]); + if (ioctl(fd, KCOV_DISABLE, 0)) + perror("ioctl"), exit(1); + if (munmap(cover, COVER_SIZE * sizeof(unsigned long))) + perror("munmap"), exit(1); + if (close(fd)) + perror("close"), exit(1); + return 0; + }
include/linux/kcov.h +23 -0
@@ -37,12 +37,35 @@ do { \ (t)->kcov_mode &= ~KCOV_IN_CTXSW ; \ } while (0) +/* See Documentation/dev-tools/kcov.rst for usage details. */ +void kcov_remote_start(u64 handle); +void kcov_remote_stop (void ); +u64 kcov_common_handle (void ); + +static inline void kcov_remote_start_common (u64 id) +{ +kcov_remote_start (kcov_remote_handle (KCOV_SUBSYSTEM_COMMON , id)); +} + +static inline void kcov_remote_start_usb (u64 id) +{ +kcov_remote_start (kcov_remote_handle (KCOV_SUBSYSTEM_USB , id)); +} +#else static inline void kcov_task_init (struct task_struct *t) {}static inline void kcov_task_exit (struct task_struct *t) {}static inline void kcov_prepare_switch (struct task_struct *t) {}static inline void kcov_finish_switch (struct task_struct *t) {} +static inline void kcov_remote_start (u64 handle) {} +static inline void kcov_remote_stop (void ) {} +static inline u64 kcov_common_handle (void ) +{ +return 0 ; +} +static inline void kcov_remote_start_common (u64 id) {} +static inline void kcov_remote_start_usb (u64 id) {}#endif /* CONFIG_KCOV */ #endif /* _LINUX_KCOV_H */
include/linux/sched.h +6 -0
@@ -1214,6 +1214,12 @@ struct task_struct {/* KCOV descriptor wired with this task or NULL: */ struct kcov *kcov; + +/* KCOV common handle for remote coverage collection: */ +u64 kcov_handle; + +/* KCOV sequence number, see kernel/kcov.c for details: */ +int kcov_sequence;#endif #ifdef CONFIG_MEMCG
include/uapi/linux/kcov.h +25 -0
@@ -4,9 +4,21 @@#include <linux/types.h> +/* See Documentation/dev-tools/kcov.rst for usage details. */ +struct kcov_remote_arg { +unsigned int trace_mode;/* KCOV_TRACE_PC or KCOV_TRACE_CMP */ +unsigned int area_size;/* Length of coverage buffer in words */ +unsigned int num_handles;/* Size of handles array */ +__u64 common_handle; +__u64 handles[0 ]; +}; + +#define KCOV_REMOTE_MAX_HANDLES 0x10000 +#define KCOV_INIT_TRACE _IOR('c', 1, unsigned long)#define KCOV_ENABLE _IO('c', 100)#define KCOV_DISABLE _IO('c', 101) +#define KCOV_REMOTE_ENABLE _IOW('c', 102, struct kcov_remote_arg)enum {/*
@@ -32,4 +44,17 @@ enum {#define KCOV_CMP_SIZE (n) ((n) << 1)#define KCOV_CMP_MASK KCOV_CMP_SIZE(3) +#define KCOV_SUBSYSTEM_COMMON (0x00ull << 56) +#define KCOV_SUBSYSTEM_USB (0x01ull << 56) + +#define KCOV_SUBSYSTEM_MASK (0xffull << 56) +#define KCOV_HANDLE_ID_MASK (0xffffffffull) + +static inline __u64 kcov_remote_handle (__u64 subsys,__u64 id) +{ +if (subsys & ~KCOV_SUBSYSTEM_MASK || id & ~KCOV_HANDLE_ID_MASK ) +return 0 ; +return subsys | id; +} +#endif /* _LINUX_KCOV_IOCTLS_H */
kernel/kcov.c +445 -35
613 lines not on this page · raw message
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
PATCH v2 2/3 usb, kcov: collect coverage from hub_event
In reply to Andrey KonovalovThis patch adds kcov_remote_start()/kcov_remote_stop() annotations to the hub_event() function, which is responsible for processing events on USB buses, in particular events that happen during USB device enumeration. Since hub_event() is run in a global background kernel thread (see Documentation/dev-tools/kcov.rst for details), each USB bus gets a unique global handle id from the USB subsystem kcov handle id range. As the result kcov can now be used to collect coverage from events that happen on a particular USB bus.
--- drivers/usb/core/hub.c | 5 +++++ 1 file changed, 5 insertions(+)
drivers/usb/core/hub.c +5 -0
@@ -29,6 +29,7 @@#include <linux/random.h> #include <linux/pm_qos.h> #include <linux/kobject.h> +#include <linux/kcov.h> #include <linux/uaccess.h> #include <asm/byteorder.h>
@@ -5374,6 +5375,8 @@ static void hub_event(struct work_struct *work) hub_dev = hub->intfdev; intf =to_usb_interface (hub_dev); +kcov_remote_start_usb ((u64 )hdev->bus->busnum); + dev_dbg(hub_dev, "state %d ports %d chg %04x evt %04x\n", hdev->state, hdev->maxchild,/* NOTE: expects max 15 ports... */
@@ -5480,6 +5483,8 @@ static void hub_event(struct work_struct *work)/* Balance the stuff in kick_hub_wq() and allow autosuspend */ usb_autopm_put_interface (intf);kref_put (&hub->kref, hub_release); + +kcov_remote_stop (); }static const struct usb_device_id hub_id_table[] = {
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
PATCH v2 3/3 vhost, kcov: collect coverage from vhost_worker
In reply to Andrey KonovalovThis patch adds kcov_remote_start()/kcov_remote_stop() annotations to the vhost_worker() function, which is responsible for processing vhost works. Since vhost_worker() threads are spawned per vhost device instance the common kcov handle is used for kcov_remote_start()/stop() annotations (see Documentation/dev-tools/kcov.rst for details). As the result kcov can now be used to collect coverage from vhost worker threads.
--- drivers/vhost/vhost.c | 6 ++++++ drivers/vhost/vhost.h | 1 + 2 files changed, 7 insertions(+)
drivers/vhost/vhost.c +6 -0
@@ -30,6 +30,7 @@#include <linux/sched/signal.h> #include <linux/interval_tree_generic.h> #include <linux/nospec.h> +#include <linux/kcov.h> #include "vhost.h"
@@ -357,7 +358,9 @@ static int vhost_worker(void *data)llist_for_each_entry_safe (work, work_next, node, node) {clear_bit (VHOST_WORK_QUEUED , &work->flags);__set_current_state (TASK_RUNNING ); +kcov_remote_start_common (dev->kcov_handle); work->fn (work); +kcov_remote_stop ();if (need_resched ())schedule (); }
@@ -546,6 +549,7 @@ long vhost_dev_set_owner(struct vhost_dev *dev)/* No owner, become one */ dev->mm =get_task_mm (current); + dev->kcov_handle =kcov_common_handle (); worker =kthread_create (vhost_worker, dev,"vhost-%d" , current->pid);if (IS_ERR (worker)) { err =PTR_ERR (worker);
@@ -571,6 +575,7 @@ long vhost_dev_set_owner(struct vhost_dev *dev)if (dev->mm)mmput (dev->mm); dev->mm =NULL ; + dev->kcov_handle =0 ; err_mm:return err; }
@@ -682,6 +687,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev)if (dev->worker) {kthread_stop (dev->worker); dev->worker =NULL ; + dev->kcov_handle =0 ; }if (dev->mm)mmput (dev->mm);
drivers/vhost/vhost.h +1 -0
@@ -173,6 +173,7 @@ struct vhost_dev {int iov_limit;int weight;int byte_weight; +u64 kcov_handle; };bool vhost_exceeds_weight (struct vhost_virtqueue *vq,int pkts,int total_len);
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Quoted 7 lines · On Wed, 23 Oct 2019 17:24:28 +0200 Andrey Konovalov <[email protected]>…
This patchset extends kcov to allow collecting coverage from the USB subsystem and vhost workers. See the first patch description for details about the kcov extension. The other two patches apply this kcov extension to USB and vhost.
These patches have been used to enable coverage-guided USB fuzzing with syzkaller for the last few years
I find it surprising that this material is so focused on USB. Is there something unique about USB that gave rise to this situation, or is it expected that the new kcov feature will be used elsewhere in the kernel?
If the latter, which are the expected subsystems?
Quoted 7 lines · On Wed, 23 Oct 2019 17:24:28 +0200 Andrey Konovalov <[email protected]>…
This patchset extends kcov to allow collecting coverage from the USB subsystem and vhost workers. See the first patch description for details about the kcov extension. The other two patches apply this kcov extension to USB and vhost.
These patches have been used to enable coverage-guided USB fuzzing with syzkaller for the last few years
I find it surprising that this material is so focused on USB. Is there something unique about USB that gave rise to this situation, or is it expected that the new kcov feature will be used elsewhere in the kernel?
If the latter, which are the expected subsystems?
Re: [PATCH v2 1/3] kcov: remote coverage support
In reply to Andrey KonovalovQuoted 68 lines · On Wed, 23 Oct 2019 17:24:29 +0200 Andrey Konovalov <[email protected]>…
This patch adds background thread coverage collection ability to kcov.
With
KCOV_ENABLEcoverage is collected only for syscalls that are issued from the current process. WithKCOV_REMOTE_ENABLEit's possible to collect coverage for arbitrary parts of the kernel code, provided that those parts are annotated with kcov_remote_start()/kcov_remote_stop().This allows to collect coverage from two types of kernel background
threads: the global ones, that are spawned during kernel boot and are always running (e.g. USBhub_event()); and the local ones, that are spawned when a user interacts with some kernel interface (e.g. vhost workers).To enable collecting coverage from a global background thread, a unique global handle must be assigned and passed to the corresponding
kcov_remote_start()call. Then a userspace process can pass a list of such handles to theKCOV_REMOTE_ENABLEioctl in the handles array field of thekcov_remote_argstruct. This will attach the used kcov device to the code sections, that are referenced by those handles.Since there might be many local background threads spawned from different userspace processes, we can't use a single global handle per annotation. Instead, the userspace process passes a non-zero handle through the
common_handlefield of thekcov_remote_argstruct. This common handle gets saved to thekcov_handlefield in the currenttask_structand needs to be passed to the newly spawned threads via custom annotations. Those threads should in turn be annotated with kcov_remote_start()/kcov_remote_stop().Internally kcov stores handles as u64 integers. The top byte of a handle is used to denote the id of a subsystem that this handle belongs to, and the lower 4 bytes are used to denote a handle id within that subsystem. A reserved value 0 is used as a subsystem id for common handles as they don't belong to a particular subsystem. The bytes 4-7 are currently reserved and must be zero. In the future the number of bytes used for the subsystem or handle ids might be increased.
When a particular userspace proccess collects coverage by via a common handle, kcov will collect coverage for each code section that is annotated to use the common handle obtained as
kcov_handlefrom the currenttask_struct. However non common handles allow to collect coverage selectively from different subsystems....
kernel/kcov.c +21
-0+static struct kcov_remote *kcov_remote_add (struct kcov *kcov,u64 handle) +{ +struct kcov_remote *remote; + +if (kcov_remote_find (handle)) +return ERR_PTR (-EEXIST ); + remote =kmalloc (sizeof (*remote),GFP_ATOMIC ); +if (!remote) +return ERR_PTR (-ENOMEM ); + remote->handle = handle; + remote->kcov = kcov; +hash_add (kcov_remote_map, &remote->hnode, handle); +return remote; +} + +spin_lock (&kcov_remote_lock); +for (i =0 ; i < remote_arg->num_handles; i++) { +kcov_debug ("handle %llx\n" , remote_arg->handles[i]); +if (!kcov_check_handle (remote_arg->handles[i], + false, true, false)) { +spin_unlock (&kcov_remote_lock);
kernel/kcov.c struct kcov { · line 130
Quoted 10 lines
kernel/kcov.c +10
-0+kcov_disable (t, kcov); +return -EINVAL ; + } + remote =kcov_remote_add (kcov, remote_arg->handles[i]); +if (IS_ERR (remote)) { +spin_unlock (&kcov_remote_lock); +kcov_disable (t, kcov); +return PTR_ERR (remote); + } + }
It's worrisome that this code can perform up to 65536 GFP_ATOMIC allocations without coming up for air. The possibility of ENOMEM or of causing collateral problems is significant. It doesn't look too hard to change this to use GFP_KERNEL?
kernel/kcov.c static const struct file_operations kcov_fops = { · line 703
Quoted 4 lines
kernel/kcov.c +4
-0+u64 kcov_common_handle (void ) +{ +return current->kcov_handle; +}
I don't immediately understand what this "common handle" thing is all about. Code is rather lacking in this sort of high-level commentary?
Re: [PATCH v2 1/3] kcov: remote coverage support
In reply to Andrey KonovalovQuoted 2 lines · Andrey Konovalov
This patch adds background thread coverage collection ability to kcov.
...
kernel/kcov.c struct kcov { · line 110
Quoted 12 lines
kernel/kcov.c +12
-0+static struct kcov_remote *kcov_remote_add (struct kcov *kcov,u64 handle) +{ +struct kcov_remote *remote; + +if (kcov_remote_find (handle)) +return ERR_PTR (-EEXIST ); + remote =kmalloc (sizeof (*remote),GFP_ATOMIC ); +if (!remote) +return ERR_PTR (-ENOMEM ); + remote->handle = handle; + remote->kcov = kcov; +hash_add (kcov_remote_map, &remote->hnode, handle);
I think it will make sense to check that there is no existing kcov with the same handle registered. Such condition will be extremely hard to debug based on episodically missing coverage.
...
Quoted 5 lines
void
kcov_task_exit(struct task_struct *t){ struct kcov *kcov;kernel/kcov.c +0
-0@@ -256,15 +401,23 @@ void kcov_task_exit(struct task_struct *t) kcov = t->kcov;
kernel/kcov.c void kcov_task_exit(struct task_struct *t) · line 411
Quoted 10 lines
kernel/kcov.c +7
-0if (kcov ==NULL )return ; +spin_lock (&kcov->lock); +kcov_debug ("t = %px, kcov->t = %px\n" , t, kcov->t); +/* +* If !kcov->remote, this checks that t->kcov->t == t. +* If kcov->remote == true then the exiting task is either: +* 1. a remote task between kcov_remote_start() and kcov_remote_stop(), +* in this case t != kcov->t and we'll print a warning; or
Why? Is kcov->t == NULL for remote kcov's? May be worth mentioning in the comment b/c it's a very condensed form to check lots of different things at once.
Otherwise the series look good to me:
But Andrew's comments stand. It's possible I understand all of this only because I already know how it works and why it works this way.
Re: [PATCH v2 1/3] kcov: remote coverage support
In reply to Andrey KonovalovQuoted 2 lines · Andrey Konovalov
This patch adds background thread coverage collection ability to kcov.
...
kernel/kcov.c struct kcov { · line 110
Quoted 12 lines
kernel/kcov.c +12
-0+static struct kcov_remote *kcov_remote_add (struct kcov *kcov,u64 handle) +{ +struct kcov_remote *remote; + +if (kcov_remote_find (handle)) +return ERR_PTR (-EEXIST ); + remote =kmalloc (sizeof (*remote),GFP_ATOMIC ); +if (!remote) +return ERR_PTR (-ENOMEM ); + remote->handle = handle; + remote->kcov = kcov; +hash_add (kcov_remote_map, &remote->hnode, handle);
I think it will make sense to check that there is no existing kcov with the same handle registered. Such condition will be extremely hard to debug based on episodically missing coverage.
...
Quoted 5 lines
void
kcov_task_exit(struct task_struct *t){ struct kcov *kcov;kernel/kcov.c +0
-0@@ -256,15 +401,23 @@ void kcov_task_exit(struct task_struct *t) kcov = t->kcov;
kernel/kcov.c void kcov_task_exit(struct task_struct *t) · line 411
Quoted 10 lines
kernel/kcov.c +7
-0if (kcov ==NULL )return ; +spin_lock (&kcov->lock); +kcov_debug ("t = %px, kcov->t = %px\n" , t, kcov->t); +/* +* If !kcov->remote, this checks that t->kcov->t == t. +* If kcov->remote == true then the exiting task is either: +* 1. a remote task between kcov_remote_start() and kcov_remote_stop(), +* in this case t != kcov->t and we'll print a warning; or
Why? Is kcov->t == NULL for remote kcov's? May be worth mentioning in the comment b/c it's a very condensed form to check lots of different things at once.
Otherwise the series look good to me:
But Andrew's comments stand. It's possible I understand all of this only because I already know how it works and why it works this way.
Quoted 8 lines · Andrew Morton
Quoted 5 lines · On Wed, 23 Oct 2019 17:24:28 +0200 Andrey Konovalov <[email protected]>…
This patchset extends kcov to allow collecting coverage from the USB subsystem and vhost workers. See the first patch description for details about the kcov extension. The other two patches apply this kcov extension to USB and vhost.
Quoted 5 lines
Quoted 2 lines
These patches have been used to enable coverage-guided USB fuzzing with syzkaller for the last few years
I find it surprising that this material is so focused on USB. Is there something unique about USB that gave rise to this situation, or
USB fuzzing is the thing that I've been working on that requires this functionality. But the idea is to make the interface generic enough to make it useful for other subsystems as well as long as the annotations are added.
Quoted 4 lines
is it expected that the new kcov feature will be used elsewhere in the kernel?
If the latter, which are the expected subsystems?
Currently we encountered two cases where this is useful: USB and vhost workers. Most probably there are more subsystems that will benefit from this kcov extension to get better fuzzing coverage. I don't have a list of them, but the provided interface should be easy to use when more of such cases are encountered.
Re: [PATCH v2 1/3] kcov: remote coverage support
In reply to Andrew MortonQuoted 78 lines · Andrew Morton
Quoted 75 lines · On Wed, 23 Oct 2019 17:24:29 +0200 Andrey Konovalov <[email protected]>…
This patch adds background thread coverage collection ability to kcov.
With
KCOV_ENABLEcoverage is collected only for syscalls that are issued from the current process. WithKCOV_REMOTE_ENABLEit's possible to collect coverage for arbitrary parts of the kernel code, provided that those parts are annotated with kcov_remote_start()/kcov_remote_stop().This allows to collect coverage from two types of kernel background
threads: the global ones, that are spawned during kernel boot and are always running (e.g. USBhub_event()); and the local ones, that are spawned when a user interacts with some kernel interface (e.g. vhost workers).To enable collecting coverage from a global background thread, a unique global handle must be assigned and passed to the corresponding
kcov_remote_start()call. Then a userspace process can pass a list of such handles to theKCOV_REMOTE_ENABLEioctl in the handles array field of thekcov_remote_argstruct. This will attach the used kcov device to the code sections, that are referenced by those handles.Since there might be many local background threads spawned from different userspace processes, we can't use a single global handle per annotation. Instead, the userspace process passes a non-zero handle through the
common_handlefield of thekcov_remote_argstruct. This common handle gets saved to thekcov_handlefield in the currenttask_structand needs to be passed to the newly spawned threads via custom annotations. Those threads should in turn be annotated with kcov_remote_start()/kcov_remote_stop().Internally kcov stores handles as u64 integers. The top byte of a handle is used to denote the id of a subsystem that this handle belongs to, and the lower 4 bytes are used to denote a handle id within that subsystem. A reserved value 0 is used as a subsystem id for common handles as they don't belong to a particular subsystem. The bytes 4-7 are currently reserved and must be zero. In the future the number of bytes used for the subsystem or handle ids might be increased.
When a particular userspace proccess collects coverage by via a common handle, kcov will collect coverage for each code section that is annotated to use the common handle obtained as
kcov_handlefrom the currenttask_struct. However non common handles allow to collect coverage selectively from different subsystems....
kernel/kcov.c +28
-0+static struct kcov_remote *kcov_remote_add (struct kcov *kcov,u64 handle) +{ +struct kcov_remote *remote; + +if (kcov_remote_find (handle)) +return ERR_PTR (-EEXIST ); + remote =kmalloc (sizeof (*remote),GFP_ATOMIC ); +if (!remote) +return ERR_PTR (-ENOMEM ); + remote->handle = handle; + remote->kcov = kcov; +hash_add (kcov_remote_map, &remote->hnode, handle); +return remote; +} + +spin_lock (&kcov_remote_lock); +for (i =0 ; i < remote_arg->num_handles; i++) { +kcov_debug ("handle %llx\n" , remote_arg->handles[i]); +if (!kcov_check_handle (remote_arg->handles[i], + false, true, false)) { +spin_unlock (&kcov_remote_lock); +kcov_disable (t, kcov); +return -EINVAL ; + } + remote =kcov_remote_add (kcov, remote_arg->handles[i]); +if (IS_ERR (remote)) { +spin_unlock (&kcov_remote_lock); +kcov_disable (t, kcov);
Quoted 8 lines
kernel/kcov.c static int kcov_ioctl_locked(struct kcov *kcov, unsigned int cmd, · line 628
Quoted 3 lines
kernel/kcov.c +3
-0+return PTR_ERR (remote); + } + }It's worrisome that this code can perform up to 65536
GFP_ATOMICallocations without coming up for air. The possibility of ENOMEM or of causing collateral problems is significant. It doesn't look too hard to change this to useGFP_KERNEL?
Sure, I'll do that in v3. I can also change the limit on the number of handles to something lower (0x100?).
Quoted 8 lines
kernel/kcov.c static const struct file_operations kcov_fops = { · line 703
Quoted 5 lines
kernel/kcov.c +4
-0+u64 kcov_common_handle (void ) +{ +return current->kcov_handle; +}I don't immediately understand what this "common handle" thing is all about. Code is rather lacking in this sort of high-level commentary?
It's described in the documentation, but I'll add comments to the exported functions to explain how they work and how they should be used.
Thanks!
Re: [PATCH v2 1/3] kcov: remote coverage support
In reply to Dmitry VyukovQuoted 5 lines · Dmitry Vyukov
Quoted 2 lines · Andrey Konovalov
This patch adds background thread coverage collection ability to kcov.
...
Quoted 16 lines
kernel/kcov.c struct kcov { · line 110
Quoted 12 lines
kernel/kcov.c +12
-0+static struct kcov_remote *kcov_remote_add (struct kcov *kcov,u64 handle) +{ +struct kcov_remote *remote; + +if (kcov_remote_find (handle)) +return ERR_PTR (-EEXIST ); + remote =kmalloc (sizeof (*remote),GFP_ATOMIC ); +if (!remote) +return ERR_PTR (-ENOMEM ); + remote->handle = handle; + remote->kcov = kcov; +hash_add (kcov_remote_map, &remote->hnode, handle);I think it will make sense to check that there is no existing kcov with the same handle registered. Such condition will be extremely hard to debug based on episodically missing coverage.
Will do in v3.
Quoted 13 lines
...
Quoted 11 lines
void
kcov_task_exit(struct task_struct *t){ struct kcov *kcov;kernel/kcov.c +3
-0@@ -256,15 +401,23 @@ void kcov_task_exit(struct task_struct *t) kcov = t->kcov;if (kcov ==NULL )return ; +spin_lock (&kcov->lock); +kcov_debug ("t = %px, kcov->t = %px\n" , t, kcov->t); +/*
Quoted 8 lines
kernel/kcov.c void kcov_task_exit(struct task_struct *t) · line 411
Quoted 4 lines
kernel/kcov.c +4
-0+* If !kcov->remote, this checks that t->kcov->t == t. +* If kcov->remote == true then the exiting task is either: +* 1. a remote task between kcov_remote_start() and kcov_remote_stop(), +* in this case t != kcov->t and we'll print a warning; or Why? Is
kcov->t== NULL for remote kcov's? May be worth mentioning in the comment b/c it's a very condensed form to check lots of different things at once.
For remote kcov instances kcov->t points to the thread that created the kcov device (I'll update the comment in struct kcov). When a task is between kcov_remote_start() and kcov_remote_stop(), it's t->kcov point to the remote kcov. So t is current for this task, and t->kcov->t is the task that created the kcov instance. I'll expand the comment to explain this better.
Quoted 3 lines
Otherwise the series look good to me:
Reviewed-by Dmitry Vyukov <[email protected]>
Great, thanks!
Quoted 3 lines
But Andrew's comments stand. It's possible I understand all of this only because I already know how it works and why it works this way.
Quoted 10 lines · On Thu, 24 Oct 2019 14:47:31 +0200 Andrey Konovalov <[email protected]>…
Quoted 4 lines
is it expected that the new kcov feature will be used elsewhere in the kernel?
If the latter, which are the expected subsystems?
Currently we encountered two cases where this is useful: USB and vhost workers. Most probably there are more subsystems that will benefit from this kcov extension to get better fuzzing coverage. I don't have a list of them, but the provided interface should be easy to use when more of such cases are encountered.
It would be helpful to add such a list to the changelog. Best-effort and approximate is OK - just to help people understand the eventual usefulness of the proposal.
Quoted 10 lines · On Thu, 24 Oct 2019 14:47:31 +0200 Andrey Konovalov <[email protected]>…
Quoted 4 lines
is it expected that the new kcov feature will be used elsewhere in the kernel?
If the latter, which are the expected subsystems?
Currently we encountered two cases where this is useful: USB and vhost workers. Most probably there are more subsystems that will benefit from this kcov extension to get better fuzzing coverage. I don't have a list of them, but the provided interface should be easy to use when more of such cases are encountered.
It would be helpful to add such a list to the changelog. Best-effort and approximate is OK - just to help people understand the eventual usefulness of the proposal.
Quoted 14 lines · Andrew Morton
Quoted 10 lines · On Thu, 24 Oct 2019 14:47:31 +0200 Andrey Konovalov <[email protected]>…
Quoted 4 lines
is it expected that the new kcov feature will be used elsewhere in the kernel?
If the latter, which are the expected subsystems?
Currently we encountered two cases where this is useful: USB and vhost workers. Most probably there are more subsystems that will benefit from this kcov extension to get better fuzzing coverage. I don't have a list of them, but the provided interface should be easy to use when more of such cases are encountered.
Quoted 3 lines
It would be helpful to add such a list to the changelog. Best-effort and approximate is OK - just to help people understand the eventual usefulness of the proposal.
OK, I'll add it to the cover letter in v3, thanks!
kbuild test robot
build botRe: [PATCH v2 3/3] vhost, kcov: collect coverage from vhost_worker
In reply to Andrey KonovalovThank you for the patch! Yet something to improve:
Hi Andrey,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on linus/master]
[cannot apply to v5.4-rc4 next-20191025]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Andrey-Konovalov/kcov-collect-coverage-from-usb-and-vhost/20191025-145734 base: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 39a38bcba4ab6e5285b07675b0e42c96eec35e67 config: s390-allmodconfig (attached as .config) compiler: s390-linux-gcc (GCC) 7.4.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=7.4.0 make.cross ARCH=s390
If you fix the issue, kindly add following tag
All errors (new ones prefixed by >>):
>> ERROR: "kcov_common_handle" [drivers/vhost/vhost.ko] undefined!
ERROR: "kcov_remote_stop" [drivers/vhost/vhost.ko] undefined!
ERROR: "kcov_remote_start" [drivers/vhost/vhost.ko] undefined!
ERROR: "kcov_remote_stop" [drivers/usb/core/usbcore.ko] undefined!
ERROR: "kcov_remote_start" [drivers/usb/core/usbcore.ko] undefined!
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
kbuild test robot
build botRe: [PATCH v2 3/3] vhost, kcov: collect coverage from vhost_worker
In reply to Andrey KonovalovThank you for the patch! Yet something to improve:
Hi Andrey,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on linus/master]
[cannot apply to v5.4-rc4 next-20191025]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Andrey-Konovalov/kcov-collect-coverage-from-usb-and-vhost/20191025-145734 base: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 39a38bcba4ab6e5285b07675b0e42c96eec35e67 config: s390-allmodconfig (attached as .config) compiler: s390-linux-gcc (GCC) 7.4.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=7.4.0 make.cross ARCH=s390
If you fix the issue, kindly add following tag
All errors (new ones prefixed by >>):
>> ERROR: "kcov_common_handle" [drivers/vhost/vhost.ko] undefined!
ERROR: "kcov_remote_stop" [drivers/vhost/vhost.ko] undefined!
ERROR: "kcov_remote_start" [drivers/vhost/vhost.ko] undefined!
ERROR: "kcov_remote_stop" [drivers/usb/core/usbcore.ko] undefined!
ERROR: "kcov_remote_start" [drivers/usb/core/usbcore.ko] undefined!
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v2 1/3] kcov: remote coverage support
In reply to Dmitry VyukovQuoted 5 lines · Dmitry Vyukov
Quoted 2 lines · Andrey Konovalov
This patch adds background thread coverage collection ability to kcov.
...
Quoted 16 lines
kernel/kcov.c struct kcov { · line 110
Quoted 12 lines
kernel/kcov.c +12
-0+static struct kcov_remote *kcov_remote_add (struct kcov *kcov,u64 handle) +{ +struct kcov_remote *remote; + +if (kcov_remote_find (handle)) +return ERR_PTR (-EEXIST ); + remote =kmalloc (sizeof (*remote),GFP_ATOMIC ); +if (!remote) +return ERR_PTR (-ENOMEM ); + remote->handle = handle; + remote->kcov = kcov; +hash_add (kcov_remote_map, &remote->hnode, handle);I think it will make sense to check that there is no existing kcov with the same handle registered. Such condition will be extremely hard to debug based on episodically missing coverage.
Although looking at this again: we already check that by calling kcov_remote_find().
Quoted 28 lines
...
Quoted 5 lines
void
kcov_task_exit(struct task_struct *t){ struct kcov *kcov;kernel/kcov.c +0
-0@@ -256,15 +401,23 @@ void kcov_task_exit(struct task_struct *t) kcov = t->kcov;kernel/kcov.c void kcov_task_exit(struct task_struct *t) · line 411
Quoted 10 lines
kernel/kcov.c +7
-0if (kcov ==NULL )return ; +spin_lock (&kcov->lock); +kcov_debug ("t = %px, kcov->t = %px\n" , t, kcov->t); +/* +* If !kcov->remote, this checks that t->kcov->t == t. +* If kcov->remote == true then the exiting task is either: +* 1. a remote task between kcov_remote_start() and kcov_remote_stop(), +* in this case t != kcov->t and we'll print a warning; or Why? Is
kcov->t== NULL for remote kcov's? May be worth mentioning in the comment b/c it's a very condensed form to check lots of different things at once.Otherwise the series look good to me:
Reviewed-by Dmitry Vyukov <[email protected]>But Andrew's comments stand. It's possible I understand all of this only because I already know how it works and why it works this way.
Re: [PATCH v2 1/3] kcov: remote coverage support
In reply to Andrew MortonQuoted 78 lines · Andrew Morton
Quoted 75 lines · On Wed, 23 Oct 2019 17:24:29 +0200 Andrey Konovalov <[email protected]>…
This patch adds background thread coverage collection ability to kcov.
With
KCOV_ENABLEcoverage is collected only for syscalls that are issued from the current process. WithKCOV_REMOTE_ENABLEit's possible to collect coverage for arbitrary parts of the kernel code, provided that those parts are annotated with kcov_remote_start()/kcov_remote_stop().This allows to collect coverage from two types of kernel background
threads: the global ones, that are spawned during kernel boot and are always running (e.g. USBhub_event()); and the local ones, that are spawned when a user interacts with some kernel interface (e.g. vhost workers).To enable collecting coverage from a global background thread, a unique global handle must be assigned and passed to the corresponding
kcov_remote_start()call. Then a userspace process can pass a list of such handles to theKCOV_REMOTE_ENABLEioctl in the handles array field of thekcov_remote_argstruct. This will attach the used kcov device to the code sections, that are referenced by those handles.Since there might be many local background threads spawned from different userspace processes, we can't use a single global handle per annotation. Instead, the userspace process passes a non-zero handle through the
common_handlefield of thekcov_remote_argstruct. This common handle gets saved to thekcov_handlefield in the currenttask_structand needs to be passed to the newly spawned threads via custom annotations. Those threads should in turn be annotated with kcov_remote_start()/kcov_remote_stop().Internally kcov stores handles as u64 integers. The top byte of a handle is used to denote the id of a subsystem that this handle belongs to, and the lower 4 bytes are used to denote a handle id within that subsystem. A reserved value 0 is used as a subsystem id for common handles as they don't belong to a particular subsystem. The bytes 4-7 are currently reserved and must be zero. In the future the number of bytes used for the subsystem or handle ids might be increased.
When a particular userspace proccess collects coverage by via a common handle, kcov will collect coverage for each code section that is annotated to use the common handle obtained as
kcov_handlefrom the currenttask_struct. However non common handles allow to collect coverage selectively from different subsystems....
kernel/kcov.c +28
-0+static struct kcov_remote *kcov_remote_add (struct kcov *kcov,u64 handle) +{ +struct kcov_remote *remote; + +if (kcov_remote_find (handle)) +return ERR_PTR (-EEXIST ); + remote =kmalloc (sizeof (*remote),GFP_ATOMIC ); +if (!remote) +return ERR_PTR (-ENOMEM ); + remote->handle = handle; + remote->kcov = kcov; +hash_add (kcov_remote_map, &remote->hnode, handle); +return remote; +} + +spin_lock (&kcov_remote_lock); +for (i =0 ; i < remote_arg->num_handles; i++) { +kcov_debug ("handle %llx\n" , remote_arg->handles[i]); +if (!kcov_check_handle (remote_arg->handles[i], + false, true, false)) { +spin_unlock (&kcov_remote_lock); +kcov_disable (t, kcov); +return -EINVAL ; + } + remote =kcov_remote_add (kcov, remote_arg->handles[i]); +if (IS_ERR (remote)) { +spin_unlock (&kcov_remote_lock); +kcov_disable (t, kcov);
Quoted 8 lines
kernel/kcov.c static int kcov_ioctl_locked(struct kcov *kcov, unsigned int cmd, · line 628
Quoted 3 lines
kernel/kcov.c +3
-0+return PTR_ERR (remote); + } + }It's worrisome that this code can perform up to 65536
GFP_ATOMICallocations without coming up for air. The possibility of ENOMEM or of causing collateral problems is significant. It doesn't look too hard to change this to useGFP_KERNEL?
Looking at this again: it seems easy to get rid of locking kcov_remote_lock when doing kmalloc, but a bit harder to get rid of kcov->lock. Andrew, would it be OK to just change the max number of GFP_ATOMIC allocations to 256?
Quoted 10 lines
kernel/kcov.c static const struct file_operations kcov_fops = { · line 703
Quoted 5 lines
kernel/kcov.c +4
-0+u64 kcov_common_handle (void ) +{ +return current->kcov_handle; +}I don't immediately understand what this "common handle" thing is all about. Code is rather lacking in this sort of high-level commentary?