Hardware Latency Detector (formerly SMI detector)
Subsystems (2)
cover Hardware Latency Detector (formerly SMI detector)
Hi folks,
Please find attached my second re-write of the SMI detector, now called the Hardware Latency Detector. This is a loadable module that grabs the CPU for configurable periods of time (all under stop_machine()) and samples the TSC looking for discontinuity. If observed latencies exceed a threshold (for example caused by an System Management Interrupt or similar) then the event is recorded in a global ring_buffer, readable via debugfs.
The previous version was too x86-centric, since there is no reason one could not also use this newly renamed hwlat_detector on any supported platform with some kind of underlying firmware/virtualization issues effecting observable system latency measurements.
Thanks to akpm and others for feedback comments so far.
Changes since the previous version:
- Renamed to hwlat_detector
- Followed Andrew's cleanup advice
- Rewritten the documentation
Jon.
PATCH 1/1 hwlat_detector: A system hardware latency detector
This patch introduces a new hardware latency detector module that can be used to detect high hardware-induced latencies within the system. It was originally written for use in the RT kernel, but has wider applications.
Documentation/hwlat_detector.txt new file +64 -0
@@ -0,0 +1,64 @@ +Introduction: +------------- + +The module hwlat_detector is a special purpose kernel module that is used to +detect large system latencies induced by the behavior of certain underlying +hardware or firmware, independent of Linux itself. The code was developed +originally to detect SMIs (System Management Interrupts) on x86 systems, +however there is nothing x86 specific about this patchset. It was +originally written for use by the "RT" patch since the Real Time +kernel is highly latency sensitive. + +SMIs are usually not serviced by the Linux kernel, which typically does not +even know that they are occuring. SMIs are instead are set up by BIOS code +and are serviced by BIOS code, usually for "critical" events such as +management of thermal sensors and fans. Sometimes though, SMIs are used for +other tasks and those tasks can spend an inordinate amount of time in the +handler (sometimes measured in milliseconds). Obviously this is a problem if +you are trying to keep event service latencies down in the microsecond range. + +The hardware latency detector works by hogging all of the cpus for configurable +amounts of time (by calling stop_machine()), polling the CPU Time Stamp Counter +for some period, then looking for gaps in the TSC data. Any gap indicates a +time when the polling was interrupted and since the machine is stopped and +interrupts turned off the only thing that could do that would be an SMI. + +Note that the SMI detector should *NEVER* be used in a production environment. +It is intended to be run manually to determine if the hardware platform has a +problem with long system firmware service routines. + +Usage: +------ + +Loading the module hwlat_detector passing the parameter "enabled=1" (or by +setting the "enable" entry in "hwlat_detector" debugfs toggled on) is the only +step required to start the hwlat_detector. It is possible to redefine the +threshold in microseconds (us) above which latency spikes will be taken +into account (parameter "threshold="). + +Example: + + # modprobe hwlat_detector enabled=1 threshold=100 + +After the module is loaded, it creates a directory named "hwlat_detector" under +the debugfs mountpoint, "/debug/hwlat_detector" for this text. It is necessary +to have debugfs mounted, which might be on /sys/debug on your system. + +The /debug/hwlat_detector interface contains the following files: + +count - number of latency spikes observed since last reset +enable - a global enable/disable toggle (0/1), resets count +max - maximum hardware latency actually observed (usecs) +sample - a pipe from which to read current raw sample data + in the format <timestamp> <latency observed usecs> + (can be opened O_NONBLOCK for a single sample) +threshold - minimum latency value to be considered (usecs) +width - time period to sample with CPUs held (usecs) + must be less than the total window size (enforced) +window - total period of sampling, width being inside (usecs) + +By default we will set width to 500,000 and window to 1,000,000, meaning that +we will sample every 1,000,000 usecs (1s) for 500,000 usecs (0.5s). If we +observe any latencies that exceed the threshold (initially 100 usecs), +then we write to a global sample ring buffer of 8K samples, which is +consumed by reading from the "sample" (pipe) debugfs file interface.
drivers/misc/Kconfig +28 -0
@@ -76,6 +76,34 @@ config IBM_ASM information on the specific driver level and support statement for your IBM server. +config HWLAT_DETECTOR + tristate "Testing module to detect hardware-induced latencies" + depends on DEBUG_FS + default m + ---help--- + A simple hardware latency detector. Use this module to detect + large latencies introduced by the behavior of the underlying + system firmware external to Linux. We do this using periodic + use of stop_machine to grab all available CPUs and measure + for unexplainable gaps in the CPU timestamp counter(s). By + default, the module is not enabled until the "enable" file + within the "hwlat_detector" debugfs directory is toggled. + + This module is often used to detect SMI (System Management + Interrupts) on x86 systems, though is not x86 specific. To + this end, we default to using a sample window of 1 second, + during which we will sample for 0.5 seconds. If an SMI or + similar event occurs during that time, it is recorded + into an 8K samples global ring buffer until retreived. + + WARNING: This software should never be enabled (it can be built + but should not be turned on after it is loaded) in a production + environment where high latencies are a concern since the + sampling mechanism actually introduces latencies for + regular tasks while the CPU(s) are being held. + + If unsure, say N + config PHANTOM tristate "Sensable PHANToM (PCI)" depends on PCI
drivers/misc/Makefile +1 -0
@@ -21,3 +21,4 @@ obj-$(CONFIG_HP_ILO) += hpilo.o obj-$(CONFIG_ISL29003) += isl29003.o obj-$(CONFIG_C2PORT) += c2port/ obj-y += eeprom/ +obj-$(CONFIG_HWLAT_DETECTOR) += hwlat_detector.o
drivers/misc/hwlat_detector.c new file +1207 -0
1,207 lines not on this page · raw message
MAINTAINERS +9 -0
@@ -2457,6 +2457,15 @@ W: http://www.kernel.org/pub/linux/kerne S: Maintained F: drivers/hwmon/hdaps.c +HARDWARE LATENCY DETECTOR +P: Jon Masters +M: [email protected] +W: http://www.kernel.org/pub/linux/kernel/people/jcm/hwlat_detector/ +S: Supported +L: [email protected] +F: Documentation/hwlat_detector.txt +F: drivers/misc/hwlat_detector.c + HYPERVISOR VIRTUAL CONSOLE DRIVER L: [email protected] L: [email protected]
Re: [PATCH 1/1] hwlat_detector: A system hardware latency detector
Quoted 16 lines · On Thu, 11 Jun 2009 00:58:30 -0400 Jon Masters <[email protected]> wrote:
This patch introduces a new hardware latency detector module that can be used to detect high hardware-induced latencies within the system. It was originally written for use in the RT kernel, but has wider applications.
...
drivers/misc/Kconfig +9
-0+config HWLAT_DETECTOR + tristate "Testing module to detect hardware-induced latencies" + depends on DEBUG_FS + default m + ---help--- + A simple hardware latency detector. Use this module to detect + large latencies introduced by the behavior of the underlying + system firmware external to Linux. We do this using periodic + use of stop_machine to grab all available CPUs and measure
drivers/misc/Kconfig config IBM_ASM · line 97
Quoted 10 lines
drivers/misc/Kconfig +10
-0+ for unexplainable gaps in the CPU timestamp counter(s). By + default, the module is not enabled until the "enable" file + within the "hwlat_detector" debugfs directory is toggled. + + This module is often used to detect SMI (System Management + Interrupts) on x86 systems, though is not x86 specific. To + this end, we default to using a sample window of 1 second, + during which we will sample for 0.5 seconds. If an SMI or + similar event occurs during that time, it is recorded + into an 8K samples global ring buffer until retreived.
"i before e except after c"!
Quoted 14 lines
(quoted diff) +11
-0+ WARNING: This software should never be enabled (it can be built + but should not be turned on after it is loaded) in a production + environment where high latencies are a concern since the + sampling mechanism actually introduces latencies for + regular tasks while the CPU(s) are being held. + + If unsure, say N + +#include <linux/module.h> +#include <linux/init.h> +#include <linux/ring_buffer.h>
Review
Quoted 10 lines
(quoted diff) +10
-0+#include <linux/stop_machine.h> +#include <linux/time.h> +#include <linux/hrtimer.h> +#include <linux/kthread.h> +#include <linux/debugfs.h> +#include <linux/seq_file.h> +#include <linux/uaccess.h> +#include <linux/version.h> + +#define BUF_SIZE_DEFAULT 262144UL /* 8K*(sizeof(entry)) */
I still don't understand this.
What is an "entry"? Seems to be a u64? If so, the value is 4x too large.
It would be clearer to code this as "8192 * sizeof(something)".
Later, we do this:
Quoted 1 line
+static unsigned long
buf_size=BUF_SIZE_DEFAULT;
But there is no provision for altering `buf_size', so we might as well have just used the constant directly.
Quoted 51 lines
drivers/misc/hwlat_detector.c new file +51
-0+#define BUF_FLAGS (RB_FL_OVERWRITE)/* no block on full */ +#define U64STR_SIZE 22/* 20 digits max */ + +#define VERSION "1.0.0" +#define BANNER "hwlat_detector: " +#define DRVNAME "hwlat_detector" +#define DEFAULT_SAMPLE_WINDOW 1000000/* 1s */ +#define DEFAULT_SAMPLE_WIDTH 500000/* 0.5s */ +#define DEFAULT_LAT_THRESHOLD 10/* 10us */ + +/* Module metadata */ + +MODULE_LICENSE ("GPL" ); +MODULE_AUTHOR ("Jon Masters <[email protected]>" ); +MODULE_DESCRIPTION ("A simple hardware latency detector" ); +MODULE_VERSION (VERSION ); + +/* Module parameters */ + +static int debug; +static int enabled; +static int threshold; + +module_param (debug, int,0 );/* enable debug */ +module_param (enabled, int,0 );/* enable detector */ +module_param (threshold, int,0 );/* latency threshold */ + +/* Buffering and sampling */ + +static struct ring_buffer *ring_buffer;/* sample buffer */ +static DEFINE_MUTEX (ring_buffer_mutex);/* lock changes */ +static unsigned long buf_size =BUF_SIZE_DEFAULT ; +static struct task_struct *kthread;/* sampling thread */ + +/* DebugFS filesystem entries */ + +static struct dentry *debug_dir;/* debugfs directory */ +static struct dentry *debug_max;/* maximum TSC delta */ +static struct dentry *debug_count;/* total detect count */ +static struct dentry *debug_sample_width;/* sample width us */ +static struct dentry *debug_sample_window;/* sample window us */ +static struct dentry *debug_sample;/* raw samples us */ +static struct dentry *debug_threshold;/* threshold us */ +static struct dentry *debug_enable;/* enable/disable */ + +/* Individual samples and global state */ + +struct sample ;/* latency sample */ +struct data ;/* Global state */ + +/* Sampling functions */
drivers/misc/hwlat_detector.c line 114
Quoted 10 lines
drivers/misc/hwlat_detector.c new file +10
-0+static int __buffer_add_sample (struct sample *sample); +static struct sample *buffer_get_sample (struct sample *sample); +static int get_sample (void *unused); + +/* Threading and state */ +static int kthread_fn (void *unused); +static int start_kthread (void ); +static int stop_kthread (void ); +static void __reset_stats (void ); +static int init_stats (void );
We seem to be forward-declaring functions which didn't need forward declarations. This adds duplication and noise - personally I think it's better to just get the functions in the correct order and only use forward-decls where circularities are present.
A couple of struct are needlessly forward-decalred too. etc.
Quoted 34 lines
drivers/misc/hwlat_detector.c new file +31
-0+/* Debugfs interface */ +static ssize_t simple_data_read (struct file *filp,char __user *ubuf, +size_t cnt,loff_t *ppos,const u64 *entry); +static ssize_t simple_data_write (struct file *filp,const char __user *ubuf, +size_t cnt,loff_t *ppos,u64 *entry); +static int debug_sample_fopen (struct inode *inode,struct file *filp); +static ssize_t debug_sample_fread (struct file *filp,char __user *ubuf, +size_t cnt,loff_t *ppos); +static int debug_sample_release (struct inode *inode,struct file *filp); +static int debug_enable_fopen (struct inode *inode,struct file *filp); +static ssize_t debug_enable_fread (struct file *filp,char __user *ubuf, +size_t cnt,loff_t *ppos); +static ssize_t debug_enable_fwrite (struct file *file, +const char __user *user_buffer, +size_t user_size,loff_t *offset); + +static int kthread_fn (void *unused) +{ +int err =0 ; +u64 interval =0 ; + +while (!kthread_should_stop ()) { + +mutex_lock (&data.lock); + + err =stop_machine (get_sample, unused,0 ); +if (err) { +/* Houston, we have a problem */ +mutex_unlock (&data.lock); + goto err_out; + }
drivers/misc/hwlat_detector.c
Quoted 10 lines
drivers/misc/hwlat_detector.c new file +10
-0+ + interval = data.sample_window - data.sample_width; +do_div (interval,USEC_PER_MSEC );/* modifies interval value */ + +mutex_unlock (&data.lock); + +if (msleep_interruptible (interval)) + goto out; + } + goto out;
whitespace broke.
Quoted 8 lines
drivers/misc/hwlat_detector.c new file +8
-0+err_out: + printk(KERN_ERR BANNER "could not call stop_machine, disabling\n"); + enabled =0 ; +out: +return err; + +} +
drivers/misc/hwlat_detector.c line 329
Quoted 10 lines
drivers/misc/hwlat_detector.c new file +10
-0+/** +* start_kthread - Kick off the hardware latency sampling/detector kthread +* +* This starts a kernel thread that will sit and sample the CPU timestamp +* counter (TSC or similar) and look for potential hardware latencies. +*/ +static int start_kthread (void ) +{ + kthread =kthread_run (kthread_fn,NULL , +DRVNAME );
unneeded newline.
Quoted 9 lines
drivers/misc/hwlat_detector.c new file +9
-0+if (IS_ERR (kthread)) { + printk(KERN_ERR BANNER "could not start sampling thread\n"); + enabled =0 ; +return -ENOMEM ; + } + +return 0 ; +} +
drivers/misc/hwlat_detector.c line 382
Quoted 10 lines
drivers/misc/hwlat_detector.c new file +7
-0+static int init_stats (void ) +{ +int ret = -ENOMEM ; + +mutex_init (&data.lock); +init_waitqueue_head (&data.wq); +atomic_set (&data.sample_open,0 );
Some (and probably all) of these initialisations can be performed at compile-time.
drivers/misc/hwlat_detector.c line 393
Quoted 10 lines
drivers/misc/hwlat_detector.c new file +10
-0+ ring_buffer =ring_buffer_alloc (buf_size,BUF_FLAGS ); + + if (WARN(!ring_buffer, KERN_ERR BANNER +"failed to allocate ring buffer!\n" )) + goto out; + +__reset_stats (); + data.threshold =DEFAULT_LAT_THRESHOLD ;/* threshold us */ + data.sample_window =DEFAULT_SAMPLE_WINDOW ;/* window us */ + data.sample_width =DEFAULT_SAMPLE_WIDTH ;/* width us */
Ditto.
Quoted 31 lines
drivers/misc/hwlat_detector.c new file +31
-0+ ret =0 ; + +out: +return ret; + +} + +/* +* simple_data_read - Wrapper read function for global state debugfs entries +* @filp: The active open file structure for the debugfs "file" +* @ubuf: The userspace provided buffer to read value into +* @cnt: The maximum number of bytes to read +* @ppos: The current "file" position +* @entry: The entry to read from +* +* This function provides a generic read implementation for the global state +* "data" structure debugfs filesystem entries. It would be nice to use +* simple_attr_read directly, but we need to make sure that the data.lock +* spinlock is held during the actual read (even though we likely won't ever +* actually race here as the updater runs under a stop_machine context). +*/ +static ssize_t simple_data_read (struct file *filp,char __user *ubuf, +size_t cnt,loff_t *ppos,const u64 *entry) +{ +char buf[U64STR_SIZE ]; +u64 val =0 ; +int len =0 ; + +memset (buf,0 ,sizeof (buf)); + +if (!entry)
drivers/misc/hwlat_detector.c line 435
Quoted 10 lines
drivers/misc/hwlat_detector.c new file +10
-0+return -EFAULT ; + +mutex_lock (&data.lock); + val = *entry; +mutex_unlock (&data.lock); + + len =snprintf (buf,sizeof (buf),"%llu\n" , (unsigned long long )val); + +return simple_read_from_buffer (ubuf, cnt, ppos, buf, len); +
extra newline.
Quoted 1 line
+}
Perhaps we should have a simple_read_u64_from_buffer() (etc). It seems to be a fairly common pattern.
Quoted 13 lines
drivers/misc/hwlat_detector.c new file +13
-0+/* +* simple_data_write - Wrapper write function for global state debugfs entries +* @filp: The active open file structure for the debugfs "file" +* @ubuf: The userspace provided buffer to write value from +* @cnt: The maximum number of bytes to write +* @ppos: The current "file" position +* @entry: The entry to write to +* +* This function provides a generic write implementation for the global state +* "data" structure debugfs filesystem entries. It would be nice to use +* simple_attr_write directly, but we need to make sure that the data.lock +* spinlock is held during the actual write (even though we likely won't ever +* actually race here as the updater runs under a stop_machine context).
drivers/misc/hwlat_detector.c line 460
Quoted 10 lines
drivers/misc/hwlat_detector.c new file +10
-0+*/ +static ssize_t simple_data_write (struct file *filp,const char __user *ubuf, +size_t cnt,loff_t *ppos,u64 *entry) +{ +char buf[U64STR_SIZE ]; +int csize =min (cnt,sizeof (buf)); +u64 val =0 ; +int err =0 ; + +memset (buf,'\0' ,sizeof (buf));
This is unneeded.
drivers/misc/hwlat_detector.c line 467
Quoted 7 lines
drivers/misc/hwlat_detector.c new file +7
-0+if (copy_from_user (buf, ubuf, csize)) +return -EFAULT ; + + buf[U64STR_SIZE -1 ] ='\0' ;/* just in case */ + err =strict_strtoull (buf,10 , &val); +if (err) +return -EINVAL ;
Again, all the above looks terribly generic. Isn't there already a helper function for this? If not, there should be one. (I should know, but I'm asleep).
Quoted 12 lines
drivers/misc/hwlat_detector.c new file +9
-0+mutex_lock (&data.lock); + *entry = val; +mutex_unlock (&data.lock); + +return csize; +} + +static ssize_t debug_enable_fread (struct file *filp,char __user *ubuf, +size_t cnt,loff_t *ppos)
drivers/misc/hwlat_detector.c line 561
Quoted 10 lines
drivers/misc/hwlat_detector.c new file +10
-0+{ +char buf[4 ]; + +if ((cnt <sizeof (buf)) || (*ppos)) +return 0 ; + + buf[0 ] = enabled ?'1' :'0' ; + buf[1 ] ='\n' ; + buf[2 ] ='\0' ; +if (copy_to_user (ubuf, buf,strlen (buf)))
"3" :)
Quoted 18 lines
drivers/misc/hwlat_detector.c new file +18
-0+return -EFAULT ; +return *ppos =strlen (buf); +} + +/** +* debug_enable_fwrite - Write function for "enable" debugfs interface +* @filp: The active open file structure for the debugfs "file" +* @ubuf: The user buffer that contains the value to write +* @cnt: The maximum number of bytes to write to "file" +* @ppos: The current position in the debugfs "file" +* +* This function provides a write implementation for the "enable" debugfs +* interface to the hardware latency detector. Can be used to enable or +* disable the detector, which will have the side-effect of possibly +* also resetting the global stats and kicking off the measuring +* kthread (on an enable) or the converse (upon a disable). +*/ +static ssize_t debug_enable_fwrite(struct file *filp,
drivers/misc/hwlat_detector.c line 589
Quoted 10 lines
drivers/misc/hwlat_detector.c new file +10
-0+const char __user *ubuf, + size_t cnt, + loff_t *ppos) +{ + char buf[4]; +int csize =min (cnt,sizeof (buf)); +long val =0 ; +int err =0 ; + +memset (buf,'\0' ,sizeof (buf));
Unneeded
Quoted 5 lines
drivers/misc/hwlat_detector.c new file +5
-0+if (copy_from_user (buf, ubuf, csize)) +return -EFAULT ; + + buf[sizeof (buf)-1 ] ='\0' ;/* just in case */ + err =strict_strtoul (buf,10 , &val);
drivers/misc/hwlat_detector.c line 604
Quoted 10 lines
drivers/misc/hwlat_detector.c new file +10
-0+if (0 != err) +return -EINVAL ; + +if (val) { +if (enabled) + goto unlock; + enabled =1 ; +__reset_stats (); +if (start_kthread ()) +return -EFAULT ;
-EFAULT seems inappropriate.
(I have a feeling I've mentioned several of these things before?)
Quoted 5 lines
drivers/misc/hwlat_detector.c new file +5
-0+ }else { +if (!enabled) + goto unlock; + enabled =0 ; +stop_kthread ();
drivers/misc/hwlat_detector.c
Quoted 10 lines
drivers/misc/hwlat_detector.c new file +7
-0+wake_up (&data.wq);/* reader(s) should return */ + } +unlock: +return csize; +} + +static ssize_t debug_width_fwrite(struct file *filp,
extra space
drivers/misc/hwlat_detector.c line 889
Quoted 10 lines
drivers/misc/hwlat_detector.c new file +10
-0+const char __user *ubuf, + size_t cnt, + loff_t *ppos) +{ + char buf[U64STR_SIZE]; +int csize =min (cnt,sizeof (buf)); +u64 val =0 ; +int err =0 ; + +memset (buf,'\0' ,sizeof (buf));
unneeded
Quoted 17 lines
drivers/misc/hwlat_detector.c new file +17
-0+if (copy_from_user (buf, ubuf, csize)) +return -EFAULT ; + + buf[U64STR_SIZE -1 ] ='\0' ;/* just in case */ + err =strict_strtoull (buf,10 , &val); +if (0 != err) +return -EINVAL ; + +mutex_lock (&data.lock); +if (val < data.sample_window) + data.sample_width = val; +else { +mutex_unlock (&data.lock); +return -EINVAL ; + } +mutex_unlock (&data.lock); +
drivers/misc/hwlat_detector.c line 963
Quoted 10 lines
drivers/misc/hwlat_detector.c new file +7
-0+if (enabled) +wake_up_process (kthread); + +return csize; +} + +static ssize_t debug_window_fwrite(struct file *filp,
more extra space
drivers/misc/hwlat_detector.c line 889
Quoted 10 lines
drivers/misc/hwlat_detector.c new file +10
-0+const char __user *ubuf, + size_t cnt, + loff_t *ppos) +{ + char buf[U64STR_SIZE]; +int csize =min (cnt,sizeof (buf)); +u64 val =0 ; +int err =0 ; + +memset (buf,'\0' ,sizeof (buf));
unneeded.
Again, the amount of generic-looking code here is a bit distressing.
Quoted 20 lines
drivers/misc/hwlat_detector.c new file +20
-0+if (copy_from_user (buf, ubuf, csize)) +return -EFAULT ; + + buf[U64STR_SIZE -1 ] ='\0' ;/* just in case */ + err =strict_strtoull (buf,10 , &val); +if (0 != err) +return -EINVAL ; + +mutex_lock (&data.lock); +if (data.sample_width < val) + data.sample_window = val; +else { +mutex_unlock (&data.lock); +return -EINVAL ; + } +mutex_unlock (&data.lock); + +return csize; +} +
drivers/misc/hwlat_detector.c line 1078
Quoted 10 lines
drivers/misc/hwlat_detector.c new file +7
-0+static int init_debugfs (void ) +{ +int ret = -ENOMEM ; + + debug_dir =debugfs_create_dir (DRVNAME ,NULL ); +if (!debug_dir) + goto err_debug_dir;
The debugfs functions should return an errno, not a NULL on error. Thwap whoever did that.
Quoted 67 lines
drivers/misc/hwlat_detector.c new file +66
-0+ debug_sample =debugfs_create_file ("sample" ,0444 , + debug_dir,NULL , + &sample_fops); +if (!debug_sample) + goto err_sample; + + debug_count =debugfs_create_file ("count" ,0444 , + debug_dir,NULL , + &count_fops); +if (!debug_count) + goto err_count; + + debug_max =debugfs_create_file ("max" ,0444 , + debug_dir,NULL , + &max_fops); +if (!debug_max) + goto err_max; + + debug_sample_window =debugfs_create_file ("window" ,0644 , + debug_dir,NULL , + &window_fops); +if (!debug_sample_window) + goto err_window; + + debug_sample_width =debugfs_create_file ("width" ,0644 , + debug_dir,NULL , + &width_fops); +if (!debug_sample_width) + goto err_width; + + debug_threshold =debugfs_create_file ("threshold" ,0644 , + debug_dir,NULL , + &threshold_fops); +if (!debug_threshold) + goto err_threshold; + + debug_enable =debugfs_create_file ("enable" ,0644 , + debug_dir, &enabled, + &enable_fops); +if (!debug_enable) + goto err_enable; + +else { + ret =0 ; + goto out; + } + +err_enable: +debugfs_remove (debug_threshold); +err_threshold: +debugfs_remove (debug_sample_width); +err_width: +debugfs_remove (debug_sample_window); +err_window: +debugfs_remove (debug_max); +err_max: +debugfs_remove (debug_count); +err_count: +debugfs_remove (debug_sample); +err_sample: +debugfs_remove (debug_dir); +err_debug_dir: +out: +return ret; +} +
drivers/misc/hwlat_detector.c line 1172
Quoted 10 lines
drivers/misc/hwlat_detector.c new file +8
-0+static int detector_init (void ) +{ +int ret = -ENOMEM ; + + printk(KERN_INFO BANNER "version %s\n", VERSION); + + ret =init_stats (); +if (0 != ret)
Didn't I already whine about the dyslexic comparisons?
Quoted 21 lines
drivers/misc/hwlat_detector.c new file +18
-0+ goto out; + + ret =init_debugfs (); +if (0 != ret) + goto err_stats; + +if (enabled) + ret =start_kthread (); + + goto out; + +err_stats: +ring_buffer_free (ring_buffer); +out: +return ret; + +} +
Re: [PATCH 1/1] hwlat_detector: A system hardware latency detector
Hey Andrew,
Thanks for the feedback. It's appreciated, as always.
Quoted 6 lines · Andrew Morton
drivers/misc/Kconfig config IBM_ASM · line 97
Quoted 1 line · On Thu, 11 Jun 2009 00:58:30 -0400 Jon Masters <[email protected]> wrote:
drivers/misc/Kconfig +1
-0+ into an 8K samples global ring buffer until retreived."i before e except after c"!
Good point! All that American miss-spelling of English words has finally driven me to this, and now there can surely be little hope left for me.
Quoted 3 lines
Quoted 1 line
+#define
BUF_SIZE_DEFAULT262144UL /* 8K*(sizeof(entry)) */I still don't understand this.
Steven's ring_buffer stores whatever "object" you want it to, and in this case I store sample structs in there, which makes it correct. I guess I'm happy to change the allocation, I'm just doing it the way other users do in-kernel already (I won't get started on the number of places where the ring_buffer could be used but is re-invented again) ;)
Quoted 6 lines
Later, we do this:
Quoted 1 line
+static unsigned long
buf_size=BUF_SIZE_DEFAULT;But there is no provision for altering `buf_size', so we might as well have just used the constant directly.
I intend to add that - seems reasonable to design with that in mind. Steven's ring_buffer code supports resizing, so I stuck that in. I don't happen to handle CPU hotplug very well either (shouldn't crash though, just might lose readings) but then, if you're trying to benchmark and you hotadd/remove a CPU while you're doing it...that's quite insane.
Quoted 6 lines
We seem to be forward-declaring functions which didn't need forward declarations. This adds duplication and noise - personally I think it's better to just get the functions in the correct order and only use forward-decls where circularities are present.
A couple of struct are needlessly forward-decalred too. etc.
Well, call me pedantic but I was always taught to do it this way for completeness. If that's not how we want to do it in-kernel, I'm happy but I'd like to know where this is spelled out (if it is) so I can go add that to my to-read list and get this sorted out in my head. And conversely, if it's not documented then I should also be fixing it.
Quoted 8 lines
drivers/misc/hwlat_detector.c line 460
Quoted 6 lines
drivers/misc/hwlat_detector.c new file +6
-0+char buf[U64STR_SIZE ]; +int csize =min (cnt,sizeof (buf)); +u64 val =0 ; +int err =0 ; + +memset (buf,'\0' ,sizeof (buf));This is unneeded.
I disagree. I always like to explicitly initialize everything to known values, even if it might be done for me. In this case, I want to ensure there is no way the copy_from_user will give me a non-terminated buffer (I add an explicit NULL later too) and that it is zeroed before use. If this were some hot-path that mattered, it would be different, but it's a trivial function that's already dealing with a copy_from_user. I'm not fussy about changing, but just so you understand my pedantic logic.
Quoted 11 lines
drivers/misc/hwlat_detector.c line 467
Quoted 7 lines
drivers/misc/hwlat_detector.c new file +7
-0+if (copy_from_user (buf, ubuf, csize)) +return -EFAULT ; + + buf[U64STR_SIZE -1 ] ='\0' ;/* just in case */ + err =strict_strtoull (buf,10 , &val); +if (err) +return -EINVAL ;Again, all the above looks terribly generic. Isn't there already a helper function for this? If not, there should be one. (I should know, but I'm asleep).
Nope. There is to go the other way, but not this way (I did look). I guess I could write one - I was trying to keep this fairly self contained to get it in upstream sooner (so people will stop whining about "Linux sucking" on systems with huge SMI braindeadness, and self contained selfishly because I have a vendor kernel backport here too) but I'm happy to look for where to add generic helpers in the next iteration. I should really work on being more helpful like that.
Quoted 6 lines
drivers/misc/hwlat_detector.c line 561
Quoted 4 lines
drivers/misc/hwlat_detector.c new file +4
-0+ buf[0 ] = enabled ?'1' :'0' ; + buf[1 ] ='\n' ; + buf[2 ] ='\0' ; +if (copy_to_user (ubuf, buf,strlen (buf)))"3" :)
I dislike hard-coding constants of any kind. Again, it's a style thing and I guess we might just have different approaches, from your previous comments on my style choices. I can change to suit the norm... ;)
Quoted 9 lines
drivers/misc/hwlat_detector.c line 604
Quoted 7 lines
drivers/misc/hwlat_detector.c new file +7
-0+if (val) { +if (enabled) + goto unlock; + enabled =1 ; +__reset_stats (); +if (start_kthread ()) +return -EFAULT ;
-EFAULTseems inappropriate.
If you're unable to kick off the kernel thread, what should you return? I thought of ENOMEM or something (because it's normally the case that we couldn't make the kthread due to that), but that doesn't feel right either because there are other ways in which we might be failing.
Quoted 1 line
(I have a feeling I've mentioned several of these things before?)
Nah, I listened and fixed those.
Quoted 10 lines
drivers/misc/hwlat_detector.c line 1078
Quoted 7 lines
drivers/misc/hwlat_detector.c new file +7
-0+static int init_debugfs (void ) +{ +int ret = -ENOMEM ; + + debug_dir =debugfs_create_dir (DRVNAME ,NULL ); +if (!debug_dir) + goto err_debug_dir;The debugfs functions should return an errno, not a NULL on error. Thwap whoever did that.
Well, at least they do internal locking of a fashion on debugfs created simple file entries (that is mostly useless in many cases) - how many people in-kernel are using these incorrectly I have no idea (yet). I wind up being really pedantic and making sure I'm locking everything explicitly for myself to ensure there's no debugfs races later.
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ ret = init_stats(); + if (0 != ret)Didn't I already whine about the dyslexic comparisons?
Argh. You did, and I tried to listen. Just like I'm going to try to listen about not explicitly nullifying things, forward declaring, or other things I really seem to enjoy doing ;)
So...I'll post another update (thanks again). Am I certainly too late for this merge window? That's ok too.
Jon.
Re: [PATCH 1/1] hwlat_detector: A system hardware latency detector
Quoted 6 lines · On Mon, 22 Jun 2009 16:58:03 -0400 Jon Masters <[email protected]> wrote:
Quoted 6 lines
We seem to be forward-declaring functions which didn't need forward declarations. This adds duplication and noise - personally I think it's better to just get the functions in the correct order and only use forward-decls where circularities are present.
Quoted 7 lines
Quoted 1 line
A couple of struct are needlessly forward-decalred too. etc.
Well, call me pedantic but I was always taught to do it this way for completeness. If that's not how we want to do it in-kernel, I'm happy but I'd like to know where this is spelled out (if it is) so I can go add that to my to-read list and get this sorted out in my head. And conversely, if it's not documented then I should also be fixing it.
It's not a major issue by any means. It's just a little unusual for kernel code.
Quoted 9 lines
Quoted 8 lines
drivers/misc/hwlat_detector.c line 460
Quoted 6 lines
drivers/misc/hwlat_detector.c new file +6
-0+char buf[U64STR_SIZE ]; +int csize =min (cnt,sizeof (buf)); +u64 val =0 ; +int err =0 ; + +memset (buf,'\0' ,sizeof (buf));This is unneeded.
Quoted 7 lines
I disagree. I always like to explicitly initialize everything to known values, even if it might be done for me. In this case, I want to ensure there is no way the
copy_from_userwill give me a non-terminated buffer (I add an explicit NULL later too) and that it is zeroed before use. If this were some hot-path that mattered, it would be different, but it's a trivial function that's already dealing with acopy_from_user. I'm not fussy about changing, but just so you understand my pedantic logic.
Well. The kernel prefers "fast, risky and well tested" over "super-defensive".
copy_from_user(buf, ubuf, csize)
will reliably write to buf[0] .. buf[csize-1].
But what you want here is to toss all this code out and call the not-yet-written u64_from_user(). I think I hinted at someone about this a few days ago..
Quoted 9 lines
Quoted 7 lines
drivers/misc/hwlat_detector.c line 604
Quoted 5 lines
drivers/misc/hwlat_detector.c new file +5
-0+ goto unlock; + enabled =1 ; +__reset_stats (); +if (start_kthread ()) +return -EFAULT ;
-EFAULTseems inappropriate.If you're unable to kick off the kernel thread, what should you return?
Propagate the kthread_run() return error code back up.
Quoted 3 lines
So...I'll post another update (thanks again). Am I certainly too late for this merge window? That's ok too.
We're generally pretty relaxed about merging updates to just-added subsystems. There's not much point in releasing them in a known-to-be-unfinished form. And if a late fix _does_ break the driver, that still isn't strictly a regression against the previous kernel release.
Re: [PATCH 1/1] hwlat_detector: A system hardware latency detector
Quoted 11 lines · Andrew Morton
Quoted 2 lines · On Mon, 22 Jun 2009 16:58:03 -0400 Jon Masters <[email protected]> wrote:
So...I'll post another update (thanks again). Am I certainly too late for this merge window? That's ok too.
We're generally pretty relaxed about merging updates to just-added subsystems. There's not much point in releasing them in a known-to-be-unfinished form. And if a late fix _does_ break the driver, that still isn't strictly a regression against the previous kernel release.
Ok, so there's my carrot...I'll go find the stick and beat myself until it's done. Thanks again, Andrew.
Jon.