Add blockconsole version 1.1 (try 3)
Subsystems (9)
cover Add blockconsole version 1.1 (try 3)
Blockconsole is a console driver very roughly similar to netconsole. Instead of sending messages out via UDP, they are written to a block device. Typically a USB stick is chosen, although in principle any block device will do.
In most cases blockconsole is useful where netconsole is not, i.e. single machines without network access or without an accessable netconsole capture server. When using both blockconsole and netconsole, I have found netconsole to sometimes create a mess under high message load (sysrq-t, etc.) while blockconsole does not.
Most importantly, a number of bugs were identified and fixed that would have been unexplained machine reboots without blockconsole.
More highlights:
* reasonably small and self-contained code,
* some 100+ machine years of runtime,
* nice tutorial with a 30-sec guide for the impatient.
Special thanks to Borislav Petkov for many improvements and kicking my behind to provide a proper git tree and resend patches.
Git tree is on kernel.org and I intend to keep it stable, as people seem to be using it already. It has been in -next since Mar 7.
git://git.kernel.org/pub/scm/linux/kernel/git/joern/bcon2.git
Joern Engel (10):
do_mounts: constify name_to_dev_t parameter
add blockconsole version 1.1
printk: add CON_ALLDATA console flag
netconsole: use CON_ALLDATA
blockconsole: use CON_ALLDATA
bcon: add a release work struct
bcon: check for hdparm in bcon_tail
bcon: remove version 1.0 support
bcon: Fix wrap-around behaviour
netconsole: s/syslogd/cancd/ in documentation
Takashi Iwai (4):
blockconsole: Allow to pass a device file path to bcon_tail
blockconsole: Fix undefined MAX_RT_PRIO
blockconsole: Rename device_lock with bc_device_lock
blockconsole: Mark a local work struct static
Documentation/block/blockconsole.txt | 94 ++++ Documentation/block/blockconsole/bcon_tail | 82 +++ Documentation/block/blockconsole/mkblockconsole | 29 ++ Documentation/networking/netconsole.txt | 16 +- block/partitions/Makefile | 1 + block/partitions/blockconsole.c | 22 + block/partitions/check.c | 3 + block/partitions/check.h | 3 + drivers/block/Kconfig | 6 + drivers/block/Makefile | 1 + drivers/block/blockconsole.c | 618 +++++++++++++++++++++++ drivers/net/netconsole.c | 2 +- include/linux/blockconsole.h | 7 + include/linux/console.h | 1 + include/linux/mount.h | 2 +- init/do_mounts.c | 2 +- kernel/printk.c | 5 +- 17 files changed, 885 insertions(+), 9 deletions(-) create mode 100644 Documentation/block/blockconsole.txt create mode 100755 Documentation/block/blockconsole/bcon_tail create mode 100755 Documentation/block/blockconsole/mkblockconsole create mode 100644 block/partitions/blockconsole.c create mode 100644 drivers/block/blockconsole.c create mode 100644 include/linux/blockconsole.h
Signature · 1 line
1.7.10.4
PATCH 1/14 do_mounts: constify name_to_dev_t parameter
--- include/linux/mount.h | 2 +- init/do_mounts.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
include/linux/mount.h +1 -1
@@ -74,6 +74,6 @@ extern struct vfsmount *vfs_kern_mount(struct file_system_type *type,extern void mnt_set_expiry (struct vfsmount *mnt,struct list_head *expiry_list);extern void mark_mounts_for_expiry (struct list_head *mounts); -extern dev_t name_to_dev_t (char *name); +extern dev_t name_to_dev_t (const char *name);#endif /* _LINUX_MOUNT_H */
init/do_mounts.c +1 -1
@@ -202,7 +202,7 @@ done:* bangs. */ -dev_t name_to_dev_t (char *name) +dev_t name_to_dev_t (const char *name) {char s[32 ];char *p;
Signature · 1 line
1.7.10.4
Re: [PATCH 01/14] do_mounts: constify name_to_dev_t parameter
Quoted 15 lines · On Thu, 9 May 2013 16:42:57 -0400 Joern Engel <[email protected]> wrote:
include/linux/mount.h +1
-1@@ -74,6 +74,6 @@ extern struct vfsmount *vfs_kern_mount(struct file_system_type *type,extern void mnt_set_expiry (struct vfsmount *mnt,struct list_head *expiry_list);extern void mark_mounts_for_expiry (struct list_head *mounts); -extern dev_t name_to_dev_t (char *name); +extern dev_t name_to_dev_t (const char *name);#endif /* _LINUX_MOUNT_H */ init/do_mounts.c +0
-0
init/do_mounts.c done: · line 208
Quoted 8 lines
init/do_mounts.c +1
-1@@ -202,7 +202,7 @@ done:* bangs. */ -dev_t name_to_dev_t (char *name) +dev_t name_to_dev_t (const char *name) {char s[32 ];char *p;
A changelog would be nice. There are 1.1 billion places in the kernel where we could make such a change - why was this one made?
Re: [PATCH 01/14] do_mounts: constify name_to_dev_t parameter
Quoted 20 lines · Andrew Morton
Quoted 18 lines · On Thu, 9 May 2013 16:42:57 -0400 Joern Engel <[email protected]> wrote:
include/linux/mount.h +1
-1@@ -74,6 +74,6 @@ extern struct vfsmount *vfs_kern_mount(struct file_system_type *type,extern void mnt_set_expiry (struct vfsmount *mnt,struct list_head *expiry_list);extern void mark_mounts_for_expiry (struct list_head *mounts); -extern dev_t name_to_dev_t (char *name); +extern dev_t name_to_dev_t (const char *name);#endif /* _LINUX_MOUNT_H */ init/do_mounts.c +0
-0@@ -202,7 +202,7 @@ done:* bangs. */
Quoted 8 lines
init/do_mounts.c done: · line 208
Quoted 5 lines
init/do_mounts.c +1
-1-dev_t name_to_dev_t (char *name) +dev_t name_to_dev_t (const char *name) {char s[32 ];char *p;A changelog would be nice. There are 1.1 billion places in the kernel where we could make such a change - why was this one made?
I happened to hit this, got a compiler warning and decided that the proper fix was the patch above. The alternative would have been to cast away the const in users of name_to_dev_t(). I can add some comment, sure.
Jörn
--
It is better to die of hunger having lived without grief and fear, than to live with a troubled spirit amid abundance.
-- Epictetus
PATCH 2/14 add blockconsole version 1.1
In reply to Joern EngelConsole driver similar to netconsole, except it writes to a block device. Can be useful in a setup where netconsole, for whatever reasons, is impractical.
Changes since version 1.0:
- Header format overhaul, addressing several annoyances when actually using blockconsole for production.
- Steve Hodgson added a panic notifier.
- Added improvements and cleanups from Borislav Petkov.
--- Documentation/block/blockconsole.txt | 94 ++++ Documentation/block/blockconsole/bcon_tail | 62 +++ Documentation/block/blockconsole/mkblockconsole | 29 ++ block/partitions/Makefile | 1 + block/partitions/blockconsole.c | 22 + block/partitions/check.c | 3 + block/partitions/check.h | 3 + drivers/block/Kconfig | 6 + drivers/block/Makefile | 1 + drivers/block/blockconsole.c | 621 +++++++++++++++++++++++ include/linux/blockconsole.h | 7 + 11 files changed, 849 insertions(+) create mode 100644 Documentation/block/blockconsole.txt create mode 100755 Documentation/block/blockconsole/bcon_tail create mode 100755 Documentation/block/blockconsole/mkblockconsole create mode 100644 block/partitions/blockconsole.c create mode 100644 drivers/block/blockconsole.c create mode 100644 include/linux/blockconsole.h
Documentation/block/blockconsole.txt new file +94 -0
@@ -0,0 +1,94 @@ +started by Jörn Engel <[email protected]> 2012.03.17 + +Blocksonsole for the impatient +============================== + +1. Find an unused USB stick and prepare it for blockconsole by writing + the blockconsole signature to it: + $ ./mkblockconsole /dev/<usb_stick> + +2. USB stick is ready for use, replug it so that the kernel can start + logging to it. + +3. After you've done logging, read out the logs from it like this: + $ ./bcon_tail + + This creates a file called /var/log/bcon.<UUID> which contains the + last 16M of the logs. Open it with a sane editor like vim which + can display zeroed gaps as a single line and start staring at the + logs. + For the really impatient, use: + $ vi `./bcon_tail` + +Introduction: +============= + +This module logs kernel printk messages to block devices, e.g. usb +sticks. It allows after-the-fact debugging when the main +disk/filesystem fails and serial consoles and netconsole are +impractical. + +It can currently only be used built-in. Blockconsole hooks into the +partition scanning code and will bring up configured block devices as +soon as possible. While this doesn't allow capture of early kernel +panics, it does capture most of the boot process. + +Block device configuration: +================================== + +Blockconsole has no configuration parameter. In order to use a block +device for logging, the blockconsole header has to be written to the +device in question. Logging to partitions is not supported. + +The example program mkblockconsole can be used to generate such a +header on a device. + +Header format: +============== + +A legal header looks like this: + +Linux blockconsole version 1.1 +818cf322 +00000000 +00000000 + +It consists of a newline, the "Linux blockconsole version 1.1" string +plus three numbers on separate lines each. Numbers are all 32bit, +represented as 8-byte hex strings, with letters in lowercase. The +first number is a uuid for this particular console device. Just pick +a random number when generating the device. The second number is a +wrap counter and unlikely to ever increment. The third is a tile +counter, with a tile being one megabyte in size. + +Miscellaneous notes: +==================== + +Blockconsole will write a new header for every tile or once every +megabyte. The header starts with a newline in order to ensure the +"Linux blockconsole...' string always ends up at the beginning of a +line if you read the blockconsole in a text editor. + +The blockconsole header is constructed such that opening the log +device in a text editor, ignoring memory constraints due to large +devices, should just work and be reasonably non-confusing to readers. +However, the example program bcon_tail can be used to copy the last 16 +tiles of the log device to /var/log/bcon.<uuid>, which should be much +easier to handle. + +The wrap counter is used by blockconsole to determine where to +continue logging after a reboot. New logs will be written to the +first tile that wasn't written to by the last instance of +blockconsole. Similarly bcon_tail is doing a binary search to find +the end of the log. + +Writing to the log device is strictly circular. This should give +optimal performance and reliability on cheap devices, like usb sticks. + +Writing to block devices has to happen in sector granularity, while +kernel logging happens in byte granularity. In order not to lose +messages in important cases like kernel crashes, a timer will write +out partial sectors if no new messages appear for a while. The +unwritten part of the sector will be filled with spaces and a single +newline. In a quiet system, these empty lines can make up the bulk of +the log.
Documentation/block/blockconsole/bcon_tail new file +62 -0
62 lines not on this page · raw message
Documentation/block/blockconsole/mkblockconsole new file +29 -0
@@ -0,0 +1,29 @@ +#!/bin/bash + +# handle the case when using files for logging instead of +# real devices (with kvm, for example) +DD_OPTS="conv=notrunc" + +if [ ! $# -eq 1 ]; then + echo "Usage: $0 <dev>" + exit 1 +elif mount|fgrep -q $1; then + echo Device appears to be mounted - aborting + exit 1 +else + dd if=/dev/zero of=$1 bs=1M count=1 $DD_OPTS + # The funky formatting is actually needed! + UUID=`head -c4 /dev/urandom |hexdump -e '/4 "%08x"'` + echo > /tmp/$UUID + echo 'Linux blockconsole version 1.1' >> /tmp/$UUID + echo "$UUID" >> /tmp/$UUID + echo 00000000 >> /tmp/$UUID + echo 00000000 >> /tmp/$UUID + for i in `seq 452`; do echo -n " " >> /tmp/$UUID; done + echo >> /tmp/$UUID + + dd if=/tmp/$UUID of=$1 $DD_OPTS + rm /tmp/$UUID + sync + exit 0 +fi
block/partitions/Makefile +1 -0
@@ -18,3 +18,4 @@ obj-$(CONFIG_IBM_PARTITION) += ibm.o obj-$(CONFIG_EFI_PARTITION) += efi.o obj-$(CONFIG_KARMA_PARTITION) += karma.o obj-$(CONFIG_SYSV68_PARTITION) += sysv68.o +obj-$(CONFIG_BLOCKCONSOLE) += blockconsole.o
block/partitions/blockconsole.c new file +22 -0
@@ -0,0 +1,22 @@ +#include <linux/blockconsole.h> + +#include "check.h" + +int blockconsole_partition (struct parsed_partitions *state) +{ +Sector sect; +void *data; +int err =0 ; + + data =read_part_sector (state,0 , §); +if (!data) +return -EIO ; +if (!bcon_magic_present (data)) + goto out; + +bcon_add (state->name); + err =1 ; +out: +put_dev_sector (sect); +return err; +}
block/partitions/check.c +3 -0
@@ -41,6 +41,9 @@ static int (*check_part[])(struct parsed_partitions *) = {* Probe partition formats with tables at disk address 0 * that also have an ADFS boot block at 0xdc0. */ +#ifdef CONFIG_BLOCKCONSOLE + blockconsole_partition, +#endif #ifdef CONFIG_ACORN_PARTITION_ICS adfspart_check_ICS,#endif
block/partitions/check.h +3 -0
@@ -50,3 +50,6 @@ put_partition(struct parsed_partitions *p, int n, sector_t from, sector_t size)extern int warn_no_part; +#ifdef CONFIG_BLOCKCONSOLE +int blockconsole_partition (struct parsed_partitions *state); +#endif
drivers/block/Kconfig +6 -0
@@ -544,4 +544,10 @@ config BLK_DEV_RBD If unsure, say N. +config BLOCKCONSOLE + bool "Block device console logging support" + help + This enables logging to block devices. + See <file:Documentation/block/blockconsole.txt> for details. + endif # BLK_DEV
drivers/block/Makefile +1 -0
@@ -40,5 +40,6 @@ obj-$(CONFIG_XEN_BLKDEV_BACKEND) += xen-blkback/ obj-$(CONFIG_BLK_DEV_DRBD) += drbd/ obj-$(CONFIG_BLK_DEV_RBD) += rbd.o obj-$(CONFIG_BLK_DEV_PCIESSD_MTIP32XX) += mtip32xx/ +obj-$(CONFIG_BLOCKCONSOLE) += blockconsole.o swim_mod-y := swim.o swim_asm.o
drivers/block/blockconsole.c new file +621 -0
621 lines not on this page · raw message
include/linux/blockconsole.h new file +7 -0
@@ -0,0 +1,7 @@ +#ifndef LINUX_BLOCKCONSOLE_H +#define LINUX_BLOCKCONSOLE_H + +int bcon_magic_present (const void *data); +void bcon_add (const char *name); + +#endif
Signature · 1 line
1.7.10.4
Re: [PATCH 02/14] add blockconsole version 1.1
Quoted 3 lines · On Thu, 9 May 2013 16:43:00 -0400 Joern Engel <[email protected]> wrote:
Console driver similar to netconsole, except it writes to a block device. Can be useful in a setup where netconsole, for whatever reasons, is impractical.
It would be useful to provide a description of how the code works. How it avoids sleeping memory allocations on the disk-writing path. What the kernel thread does. General overview of data flow. How we avoid recursion when it's called from within block/driver/etc code paths. What's all that special-case handling of panic() for.
Quoted 6 lines
...
Documentation/block/blockconsole.txt | 94 ++++ Documentation/block/blockconsole/bcon_tail | 62 +++ Documentation/block/blockconsole/mkblockconsole | 29 ++
We really need somewhere better to put userspace tools.
Quoted 10 lines
block/partitions/Makefile | 1 + block/partitions/blockconsole.c | 22 + block/partitions/check.c | 3 + block/partitions/check.h | 3 + drivers/block/Kconfig | 6 + drivers/block/Makefile | 1 + drivers/block/blockconsole.c | 621 +++++++++++++++++++++++...
drivers/block/blockconsole.c line 41
Quoted 8 lines
drivers/block/blockconsole.c new file +8
-0+#define CACHE_MASK (CACHE_SIZE - 1) +#define SECTOR_SHIFT (9) +#define SECTOR_SIZE (1 << SECTOR_SHIFT) +#define SECTOR_MASK (~(SECTOR_SIZE-1)) +#define PG_SECTOR_MASK ((PAGE_SIZE >> 9) - 1) + +#undef pr_fmt +#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt
This normally goes before the #includes, making the undef unnecessary.
Quoted 23 lines
(quoted diff) +20
-0+struct bcon_bio { + struct bio bio; + struct bio_vec bvec; + void *sector; + int in_flight; +}; +static int sync_read(struct blockconsole *bc, u64 ofs) +{ + struct bio bio; + struct bio_vec bio_vec; + struct completion complete; + + bio_init(&bio); + bio.bi_io_vec = &bio_vec; + bio_vec.bv_page = bc->pages; + bio_vec.bv_len = SECTOR_SIZE; + bio_vec.bv_offset = 0; + bio.bi_vcnt = 1; + bio.bi_idx = 0; + bio.bi_size = SECTOR_SIZE;
Review
Quoted 10 lines
(quoted diff) +10
-0+ bio.bi_bdev = bc->bdev; + bio.bi_sector = ofs >> SECTOR_SHIFT; + init_completion(&complete); + bio.bi_private = &complete; + bio.bi_end_io = request_complete; + + submit_bio(READ, &bio); + wait_for_completion(&complete); + return test_bit(BIO_UPTODATE, &bio.bi_flags) ? 0 : -EIO; +}
It wouldn't hurt to have a few explanatory comments over these functions.
Quoted 4 lines
...
drivers/block/blockconsole.c new file +1
-0+static int bcon_convert_old_format (struct blockconsole *bc)
drivers/block/blockconsole.c line 237
Quoted 10 lines
drivers/block/blockconsole.c new file +10
-0+{ + bc->uuid =get_random_int (); + bc->round =0 ; + bc->console_bytes = bc->write_bytes =0 ; +bcon_advance_console_bytes (bc,0 );/* To skip the header */ +bcon_advance_write_bytes (bc,0 );/* To wrap around, if necessary */ +bcon_erase_segment (bc); +pr_info ("converted %s from old format\n" , bc->devname); +return 0 ; +}
It's strange to have an upgrade-from-old-format feature in something which hasn't even been merged yet. Can we just ditch all this stuff?
Quoted 11 lines
...
drivers/block/blockconsole.c new file +8
-0+#define BCON_MAX_ERRORS 10 +static void bcon_end_io (struct bio *bio,int err) +{ + struct bcon_bio *bcon_bio = container_of(bio, struct bcon_bio, bio); +struct blockconsole *bc = bio->bi_private; +unsigned long flags; + +/*
drivers/block/blockconsole.c line 310
Quoted 10 lines
drivers/block/blockconsole.c new file +10
-0+* We want to assume the device broken and free this console if +* we accumulate too many errors. But if errors are transient, +* we also want to forget about them once writes succeed again. +* Oh, and we only want to reset the counter if it hasn't reached +* the limit yet, so we don't bcon_put() twice from here. +*/ +spin_lock_irqsave (&bc->end_io_lock, flags); +if (err) { +if (bc->error_count++ ==BCON_MAX_ERRORS ) { +pr_info ("no longer logging to %s\n" , bc->devname);
pr_info() may be too low a severity level?
How does the code handle the obvious recursion concern here?
Quoted 32 lines
drivers/block/blockconsole.c new file +32
-0+schedule_work (&bc->unregister_work); + } + }else { +if (bc->error_count && bc->error_count <BCON_MAX_ERRORS ) + bc->error_count =0 ; + } +/* +* Add padding (a bunch of spaces and a newline) early so bcon_pad +* only has to advance a pointer. +*/ +clear_sector (bcon_bio->sector); + bcon_bio->in_flight =0 ; +spin_unlock_irqrestore (&bc->end_io_lock, flags); +bcon_put (bc); +} + +static void bcon_writesector (struct blockconsole *bc,int index) +{ +struct bcon_bio *bcon_bio = bc->bio_array + index; +struct bio *bio = &bcon_bio->bio; + +rmb (); +if (bcon_bio->in_flight) +return ; +bcon_get (bc); + +bio_init (bio); + bio->bi_io_vec = &bcon_bio->bvec; + bio->bi_vcnt =1 ; + bio->bi_size =SECTOR_SIZE ; + bio->bi_bdev = bc->bdev; + bio->bi_private = bc;
drivers/block/blockconsole.c line 352
Quoted 10 lines
drivers/block/blockconsole.c new file +10
-0+ bio->bi_end_io = bcon_end_io; + + bio->bi_idx =0 ; + bio->bi_sector = bc->write_bytes >>9 ; + bcon_bio->in_flight =1 ; +wmb (); +submit_bio (WRITE , bio); +} + +static int bcon_writeback (void *_bc)
So this is actually a kernel thread service loop. The poorly-chosen name and absence of code comments make this unobvious.
drivers/block/blockconsole.c line 363
Quoted 11 lines
drivers/block/blockconsole.c new file +11
-0+{ +struct blockconsole *bc = _bc; +struct sched_param (sp); + + sp.sched_priority =MAX_RT_PRIO -1 ;/* Highest realtime prio */ +sched_setscheduler_nocheck (current,SCHED_FIFO , &sp); +for (;;) { +set_current_state (TASK_INTERRUPTIBLE ); +schedule (); +if (kthread_should_stop ()) +break ;
This looks wrong. The sequence should be
set_current_state(TASK_INTERRUPTIBLE); if (!kthread_should_stop()) schedule();
to avoid missed-wakeup races.
Quoted 19 lines
drivers/block/blockconsole.c new file +19
-0+while (bcon_write_sector (bc) !=bcon_console_sector (bc)) { +bcon_writesector (bc,bcon_write_sector (bc)); +bcon_advance_write_bytes (bc,SECTOR_SIZE ); +if (bcon_write_sector (bc) ==0 ) +bcon_erase_segment (bc); + } + } +return 0 ; +} + +static void bcon_pad (unsigned long data) +{ +struct blockconsole *bc = (void *)data; +unsigned int n; + +/* +* We deliberately race against bcon_write here. If we lose the race, +* our padding is no longer where we expected it to be, i.e. it is +* no longer a bunch of spaces with a newline at the end. There could
drivers/block/blockconsole.c line 392
Quoted 10 lines
drivers/block/blockconsole.c new file +10
-0+* not be a newline at all or it could be somewhere in the middle. +* Either way, the log corruption is fairly obvious to spot and ignore +* for human readers. +*/ + n =SECTOR_SIZE -bcon_console_ofs (bc); +if (n !=SECTOR_SIZE ) { +bcon_advance_console_bytes (bc, n); +wake_up_process (bc->writeback_thread); + } +}
wake_up_process() from within printk is problematic - it'll deadlock if the caller is holding certain scheduler locks. That's why wake_up_klogd() does weird things.
drivers/block/blockconsole.c line 398
Quoted 5 lines
drivers/block/blockconsole.c new file +5
-0+static void bcon_write (struct console *console,const char *msg, +unsigned int len) +{ + struct blockconsole *bc = container_of(console, struct blockconsole, + console);
struct blockconsole *bc; bc = container_of(console, struct blockconsole, console);
drivers/block/blockconsole.c line 410
Quoted 12 lines
drivers/block/blockconsole.c new file +12
-0+unsigned int n; +u64 console_bytes; +int i; + +while (len) { + console_bytes = bc->console_bytes; + i =__bcon_console_sector (console_bytes); +rmb (); +if (bc->bio_array[i].in_flight) +break ; + n =min_t (int, len,SECTOR_SIZE - +__bcon_console_ofs (console_bytes));
Yes, the types are a bit random in all this code. A mix of int, unsigned, u64 in various places, often where one would expect a size_t.
drivers/block/blockconsole.c line 421
Quoted 11 lines
drivers/block/blockconsole.c new file +11
-0+memcpy (bc->bio_array[i].sector + +__bcon_console_ofs (console_bytes), msg, n); + len -= n; + msg += n; +bcon_advance_console_bytes (bc, n); + } +wake_up_process (bc->writeback_thread); +mod_timer (&bc->pad_timer, jiffies +HZ ); +} + +static void bcon_init_bios (struct blockconsole *bc)
This code really needs some comments :(
Oh I see. It's BIOs, not BIOS. Had me fooled there.
Quoted 16 lines
drivers/block/blockconsole.c new file +16
-0+{ +int i; + +for (i =0 ; i <SECTOR_COUNT ; i++) { +int page_index = i >> (PAGE_SHIFT -SECTOR_SHIFT ); +struct page *page = bc->pages + page_index; +struct bcon_bio *bcon_bio = bc->bio_array + i; +struct bio_vec *bvec = &bcon_bio->bvec; + + bcon_bio->in_flight =0 ; + bcon_bio->sector =page_address (bc->pages + page_index) + +SECTOR_SIZE * (i &PG_SECTOR_MASK ); +clear_sector (bcon_bio->sector); + bvec->bv_page = page; + bvec->bv_len =SECTOR_SIZE ; + bvec->bv_offset =SECTOR_SIZE * (i &PG_SECTOR_MASK );
drivers/block/blockconsole.c line 461
Quoted 10 lines
drivers/block/blockconsole.c new file +7
-0+ } +} +static int blockconsole_panic (struct notifier_block *this,unsigned long event, +void *ptr) +{ + struct blockconsole *bc = container_of(this, struct blockconsole, + panic_block);
ditto
Quoted 42 lines
drivers/block/blockconsole.c new file +42
-0+unsigned int n; + + n =SECTOR_SIZE -bcon_console_ofs (bc); +if (n !=SECTOR_SIZE ) +bcon_advance_console_bytes (bc, n); +bcon_writeback (bc); +return NOTIFY_DONE ; +} + +static int bcon_create (const char *devname) +{ +const fmode_t mode =FMODE_READ |FMODE_WRITE ; +struct blockconsole *bc; +int err; + + bc =kzalloc (sizeof (*bc),GFP_KERNEL ); +if (!bc) +return -ENOMEM ; +memset (bc->devname,' ' ,sizeof (bc->devname)); +strlcpy (bc->devname, devname,sizeof (bc->devname)); +spin_lock_init (&bc->end_io_lock); +strcpy (bc->console.name,"bcon" ); + bc->console.flags =CON_PRINTBUFFER |CON_ENABLED ; + bc->console.write = bcon_write; + bc->bdev =blkdev_get_by_path (devname, mode,NULL ); +#ifndef MODULE +if (IS_ERR (bc->bdev)) { +dev_t devt =name_to_dev_t (devname); +if (devt) + bc->bdev =blkdev_get_by_dev (devt, mode,NULL ); + } +#endif +if (IS_ERR (bc->bdev)) + goto out; + bc->pages =alloc_pages (GFP_KERNEL ,8 ); +if (!bc->pages) + goto out; + bc->zero_page =alloc_pages (GFP_KERNEL ,0 ); +if (!bc->zero_page) + goto out1; +bcon_init_bios (bc); +bcon_init_zero_bio (bc);
drivers/block/blockconsole.c line 513
Quoted 10 lines
drivers/block/blockconsole.c new file +10
-0+setup_timer (&bc->pad_timer, bcon_pad, (unsigned long )bc); + bc->max_bytes = bc->bdev->bd_inode->i_size & ~CACHE_MASK ; + err =bcon_find_end_of_log (bc); +if (err) + goto out2; +kref_init (&bc->kref);/* This reference gets freed on errors */ + bc->writeback_thread =kthread_run (bcon_writeback, bc,"bcon_%s" , + devname); +if (IS_ERR (bc->writeback_thread)) + goto out2;
Should propagate the error, not assume ENOMEM. Minor.
Quoted 10 lines
drivers/block/blockconsole.c new file +10
-0+INIT_WORK (&bc->unregister_work, bcon_unregister); +register_console (&bc->console); + bc->panic_block.notifier_call = blockconsole_panic; + bc->panic_block.priority =INT_MAX ; +atomic_notifier_chain_register (&panic_notifier_list, &bc->panic_block); +pr_info ("now logging to %s at %llx\n" , devname, bc->console_bytes >>20 ); + +return 0 ; + +out2:
drivers/block/blockconsole.c line 533
Quoted 10 lines
drivers/block/blockconsole.c new file +10
-0+__free_pages (bc->zero_page,0 ); +out1: +__free_pages (bc->pages,8 ); +out: +kfree (bc); +/* Not strictly correct, be the caller doesn't care */ +return -ENOMEM ; +} + +static void bcon_create_fuzzy (const char *name)
comments, comments, comments. What's this do and why does it do it and why does it make assumptions about userspace namespace configuration.
drivers/block/blockconsole.c line 544
Quoted 11 lines
drivers/block/blockconsole.c new file +11
-0+{ +char *longname; +int err; + + err =bcon_create (name); +if (err) { + longname =kzalloc (strlen (name) +6 ,GFP_KERNEL ); +if (!longname) +return ; +strcpy (longname,"/dev/" ); +strcat (longname, name);
kasprintf()?
Quoted 29 lines
drivers/block/blockconsole.c new file +29
-0+bcon_create (longname); +kfree (longname); + } +} + +static DEFINE_SPINLOCK (device_lock); +static char scanned_devices[80 ]; + +static void bcon_do_add (struct work_struct *work) +{ +char local_devices[80 ], *name, *remainder = local_devices; + +spin_lock (&device_lock); +memcpy (local_devices, scanned_devices,sizeof (local_devices)); +memset (scanned_devices,0 ,sizeof (scanned_devices)); +spin_unlock (&device_lock); + +while (remainder && remainder[0 ]) { + name =strsep (&remainder,"," ); +bcon_create_fuzzy (name); + } +} + +DECLARE_WORK (bcon_add_work, bcon_do_add); + +void bcon_add (const char *name) +{ +/* +* We add each name to a small static buffer and ask for a workqueue
drivers/block/blockconsole.c line 583
Quoted 10 lines
drivers/block/blockconsole.c new file +10
-0+* to go pick it up asap. Once it is picked up, the buffer is empty +* again, so hopefully it will suffice for all sane users. +*/ +spin_lock (&device_lock); +if (scanned_devices[0 ]) +strncat (scanned_devices,"," ,sizeof (scanned_devices)); +strncat (scanned_devices, name,sizeof (scanned_devices)); +spin_unlock (&device_lock); +schedule_work (&bcon_add_work); +}
What's all this do?
Quoted 3 lines
...
Re: [PATCH 02/14] add blockconsole version 1.1
Quoted 11 lines · Andrew Morton
Quoted 3 lines · On Thu, 9 May 2013 16:43:00 -0400 Joern Engel <[email protected]> wrote:
Console driver similar to netconsole, except it writes to a block device. Can be useful in a setup where netconsole, for whatever reasons, is impractical.
It would be useful to provide a description of how the code works. How it avoids sleeping memory allocations on the disk-writing path. What the kernel thread does. General overview of data flow. How we avoid recursion when it's called from within
block/driver/etccode paths. What's all that special-case handling ofpanic()for.
Documentation/block/blockconsole.txt has some of this, but not all. Will add more inline documentation.
Quoted 5 lines
Quoted 3 lines
Documentation/block/blockconsole.txt | 94 ++++ Documentation/block/blockconsole/bcon_tail | 62 +++ Documentation/block/blockconsole/mkblockconsole | 29 ++We really need somewhere better to put userspace tools.
Agreed. It started as "use this horrible code as a bad example" and turned into a repository for userspace code, which it should have been.
Do you have an opinion on tools/ vs. a standalong git tree for the tools?
Quoted 10 lines
drivers/block/blockconsole.c line 41
Quoted 8 lines
drivers/block/blockconsole.c new file +8
-0+#define CACHE_MASK (CACHE_SIZE - 1) +#define SECTOR_SHIFT (9) +#define SECTOR_SIZE (1 << SECTOR_SHIFT) +#define SECTOR_MASK (~(SECTOR_SIZE-1)) +#define PG_SECTOR_MASK ((PAGE_SIZE >> 9) - 1) + +#undef pr_fmt +#define pr_fmt (fmt) KBUILD_MODNAME ": " fmtThis normally goes before the #includes, making the undef unnecessary.
Will do.
Quoted 1 line
It wouldn't hurt to have a few explanatory comments over these functions.
Sure.
Quoted 6 lines
Quoted 6 lines
drivers/block/blockconsole.c new file +6
-0+static int bcon_convert_old_format (struct blockconsole *bc) +{ + bc->uuid =get_random_int (); + bc->round =0 ; + bc->console_bytes = bc->write_bytes =0 ; +bcon_advance_console_bytes (bc,0 );/* To skip the header */
Quoted 8 lines
drivers/block/blockconsole.c line 237
Quoted 5 lines
drivers/block/blockconsole.c new file +5
-0+bcon_advance_write_bytes (bc,0 );/* To wrap around, if necessary */ +bcon_erase_segment (bc); +pr_info ("converted %s from old format\n" , bc->devname); +return 0 ; +}It's strange to have an upgrade-from-old-format feature in something which hasn't even been merged yet. Can we just ditch all this stuff?
Done in a later patch, as you noticed.
Quoted 12 lines
Quoted 12 lines
drivers/block/blockconsole.c new file +12
-0+#define BCON_MAX_ERRORS 10 +static void bcon_end_io (struct bio *bio,int err) +{ + struct bcon_bio *bcon_bio = container_of(bio, struct bcon_bio, bio); +struct blockconsole *bc = bio->bi_private; +unsigned long flags; + +/* +* We want to assume the device broken and free this console if +* we accumulate too many errors. But if errors are transient, +* we also want to forget about them once writes succeed again. +* Oh, and we only want to reset the counter if it hasn't reached
Quoted 8 lines
drivers/block/blockconsole.c line 310
Quoted 6 lines
drivers/block/blockconsole.c new file +6
-0+* the limit yet, so we don't bcon_put() twice from here. +*/ +spin_lock_irqsave (&bc->end_io_lock, flags); +if (err) { +if (bc->error_count++ ==BCON_MAX_ERRORS ) { +pr_info ("no longer logging to %s\n" , bc->devname);
pr_info()may be too low a severity level?
There is no distinction between "someone pulled the usb device" and "broken cable". Blockconsole will continue writing and retire the device if errors accumulate. The code is deliberately stupid, because stupid code tends to be more robust and devoid of strange corner-cases.
Quoted 1 line
How does the code handle the obvious recursion concern here?
I think your next comment answers that.
Quoted 19 lines
Quoted 1 line
+static int
bcon_writeback(void *_bc)So this is actually a kernel thread service loop. The poorly-chosen name and absence of code comments make this unobvious.
drivers/block/blockconsole.c line 363
Quoted 11 lines
drivers/block/blockconsole.c new file +11
-0+{ +struct blockconsole *bc = _bc; +struct sched_param (sp); + + sp.sched_priority =MAX_RT_PRIO -1 ;/* Highest realtime prio */ +sched_setscheduler_nocheck (current,SCHED_FIFO , &sp); +for (;;) { +set_current_state (TASK_INTERRUPTIBLE ); +schedule (); +if (kthread_should_stop ()) +break ;This looks wrong. The sequence should be
Quoted 5 lines
set_current_state(TASK_INTERRUPTIBLE); if (!kthread_should_stop()) schedule();to avoid missed-wakeup races.
Ack. I may have copied that from somewhere else. Will see if there are any other bad examples in the kernel.
Quoted 15 lines
Quoted 15 lines
drivers/block/blockconsole.c new file +15
-0+static void bcon_pad (unsigned long data) +{ +struct blockconsole *bc = (void *)data; +unsigned int n; + +/* +* We deliberately race against bcon_write here. If we lose the race, +* our padding is no longer where we expected it to be, i.e. it is +* no longer a bunch of spaces with a newline at the end. There could +* not be a newline at all or it could be somewhere in the middle. +* Either way, the log corruption is fairly obvious to spot and ignore +* for human readers. +*/ + n =SECTOR_SIZE -bcon_console_ofs (bc); +if (n !=SECTOR_SIZE ) {
Quoted 8 lines
drivers/block/blockconsole.c line 392
Quoted 4 lines
drivers/block/blockconsole.c new file +4
-0+bcon_advance_console_bytes (bc, n); +wake_up_process (bc->writeback_thread); + } +}
wake_up_process()from within printk is problematic - it'll deadlock if the caller is holding certain scheduler locks. That's whywake_up_klogd()does weird things.
Will take a look. My lame answer is "but I am running this on X machines and it seems to work." Would be nice if deadlocks were not just rare in practice, but impossible - to the extend one can prove anything in a codebase the size of the kernel.
Quoted 9 lines
drivers/block/blockconsole.c line 398
Quoted 5 lines
drivers/block/blockconsole.c new file +5
-0+static void bcon_write (struct console *console,const char *msg, +unsigned int len) +{ + struct blockconsole *bc = container_of(console, struct blockconsole, + console);struct blockconsole *bc; bc = container_of(console, struct blockconsole, console);
Ack.
Quoted 7 lines
Quoted 7 lines
drivers/block/blockconsole.c new file +7
-0+unsigned int n; +u64 console_bytes; +int i; + +while (len) { + console_bytes = bc->console_bytes; + i =__bcon_console_sector (console_bytes);
Quoted 8 lines
drivers/block/blockconsole.c line 410
Quoted 5 lines
drivers/block/blockconsole.c new file +5
-0+rmb (); +if (bc->bio_array[i].in_flight) +break ; + n =min_t (int, len,SECTOR_SIZE - +__bcon_console_ofs (console_bytes));Yes, the types are a bit random in all this code. A mix of int, unsigned, u64 in various places, often where one would expect a
size_t.
Will take a look.
Quoted 14 lines
drivers/block/blockconsole.c line 421
Quoted 11 lines
drivers/block/blockconsole.c new file +11
-0+memcpy (bc->bio_array[i].sector + +__bcon_console_ofs (console_bytes), msg, n); + len -= n; + msg += n; +bcon_advance_console_bytes (bc, n); + } +wake_up_process (bc->writeback_thread); +mod_timer (&bc->pad_timer, jiffies +HZ ); +} + +static void bcon_init_bios (struct blockconsole *bc)This code really needs some comments :(
Quoted 1 line
Oh I see. It's BIOs, not BIOS. Had me fooled there.
But, but, but isn't this obvious to someone who has just written said function? It has been a year now, so my own definition of obvious may have changed as well.
Quoted 4 lines
drivers/block/blockconsole.c line 513
Quoted 2 lines
drivers/block/blockconsole.c new file +2
-0+if (IS_ERR (bc->writeback_thread)) + goto out2;Should propagate the error, not assume ENOMEM. Minor.
Agreed. The error to propagate is actually ENOMEM - until someone changes the code in one place and fails to notice the other.
Quoted 4 lines
Quoted 1 line
+static void
bcon_create_fuzzy(const char *name)comments, comments, comments. What's this do and why does it do it and why does it make assumptions about userspace namespace configuration.
Ack.
Quoted 5 lines
Quoted 5 lines
drivers/block/blockconsole.c new file +5
-0+{ +char *longname; +int err; + + err =bcon_create (name);
Quoted 8 lines
drivers/block/blockconsole.c line 544
Quoted 6 lines
drivers/block/blockconsole.c new file +6
-0+if (err) { + longname =kzalloc (strlen (name) +6 ,GFP_KERNEL ); +if (!longname) +return ; +strcpy (longname,"/dev/" ); +strcat (longname, name);
kasprintf()?
Much nicer! Will do.
Quoted 8 lines
Quoted 8 lines
drivers/block/blockconsole.c new file +8
-0+void bcon_add (const char *name) +{ +/* +* We add each name to a small static buffer and ask for a workqueue +* to go pick it up asap. Once it is picked up, the buffer is empty +* again, so hopefully it will suffice for all sane users. +*/ +spin_lock (&device_lock);
Quoted 8 lines
drivers/block/blockconsole.c line 583
Quoted 6 lines
drivers/block/blockconsole.c new file +6
-0+if (scanned_devices[0 ]) +strncat (scanned_devices,"," ,sizeof (scanned_devices)); +strncat (scanned_devices, name,sizeof (scanned_devices)); +spin_unlock (&device_lock); +schedule_work (&bcon_add_work); +}What's all this do?
Work around init ordering problems. Quite likely there is a much nicer way to this that I should know about and don't.
Jörn
--
Optimizations always bust things, because all optimizations are, in the long haul, a form of cheating, and cheaters eventually get caught. -- Larry Wall
Re: [PATCH 02/14] add blockconsole version 1.1
Quoted 4 lines · On Wed, 22 May 2013 17:04:16 -0400 J__rn Engel <[email protected]> wrote:
Quoted 4 lines
Quoted 3 lines
Documentation/block/blockconsole.txt | 94 ++++ Documentation/block/blockconsole/bcon_tail | 62 +++ Documentation/block/blockconsole/mkblockconsole | 29 ++
Quoted 8 lines
Quoted 1 line
We really need somewhere better to put userspace tools.
Agreed. It started as "use this horrible code as a bad example" and turned into a repository for userspace code, which it should have been.
Do you have an opinion on
tools/vs. a standalong git tree for the tools?
./tools/blockconsole/ sounds nice. I like maintaining simple userspace utils in the main kernel tree - it's very easy and efficient for us.
Quoted 21 lines
Quoted 12 lines
Quoted 12 lines
drivers/block/blockconsole.c new file +12
-0+#define BCON_MAX_ERRORS 10 +static void bcon_end_io (struct bio *bio,int err) +{ + struct bcon_bio *bcon_bio = container_of(bio, struct bcon_bio, bio); +struct blockconsole *bc = bio->bi_private; +unsigned long flags; + +/* +* We want to assume the device broken and free this console if +* we accumulate too many errors. But if errors are transient, +* we also want to forget about them once writes succeed again. +* Oh, and we only want to reset the counter if it hasn't reached Quoted 8 lines
drivers/block/blockconsole.c line 310
Quoted 6 lines
drivers/block/blockconsole.c new file +6
-0+* the limit yet, so we don't bcon_put() twice from here. +*/ +spin_lock_irqsave (&bc->end_io_lock, flags); +if (err) { +if (bc->error_count++ ==BCON_MAX_ERRORS ) { +pr_info ("no longer logging to %s\n" , bc->devname);
pr_info()may be too low a severity level?
Quoted 5 lines
There is no distinction between "someone pulled the usb device" and "broken cable". Blockconsole will continue writing and retire the device if errors accumulate. The code is deliberately stupid, because stupid code tends to be more robust and devoid of strange corner-cases.
I meant should we use pr_err() or pr_emerg() rather than pr_info(). Chances are that pr_info() is being hidden.
Quoted 3 lines
Quoted 1 line
How does the code handle the obvious recursion concern here?
I think your next comment answers that.
hm, that was clever of me. I meant more generally, if we do a printk() from within this code how do we prevent arbitrarily deep recursion?
Quoted 14 lines
Quoted 14 lines
drivers/block/blockconsole.c line 421
Quoted 11 lines
drivers/block/blockconsole.c new file +11
-0+memcpy (bc->bio_array[i].sector + +__bcon_console_ofs (console_bytes), msg, n); + len -= n; + msg += n; +bcon_advance_console_bytes (bc, n); + } +wake_up_process (bc->writeback_thread); +mod_timer (&bc->pad_timer, jiffies +HZ ); +} + +static void bcon_init_bios (struct blockconsole *bc)This code really needs some comments :(
Quoted 5 lines
Quoted 1 line
Oh I see. It's BIOs, not BIOS. Had me fooled there.
But, but, but isn't this obvious to someone who has just written said function? It has been a year now, so my own definition of obvious may have changed as well.
heh. I blame Jens for calling it a "bio".
Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
drivers/block/blockconsole.c new file +8
-0+void bcon_add (const char *name) +{ +/* +* We add each name to a small static buffer and ask for a workqueue +* to go pick it up asap. Once it is picked up, the buffer is empty +* again, so hopefully it will suffice for all sane users. +*/ +spin_lock (&device_lock);Quoted 8 lines
drivers/block/blockconsole.c line 583
Quoted 6 lines
drivers/block/blockconsole.c new file +6
-0+if (scanned_devices[0 ]) +strncat (scanned_devices,"," ,sizeof (scanned_devices)); +strncat (scanned_devices, name,sizeof (scanned_devices)); +spin_unlock (&device_lock); +schedule_work (&bcon_add_work); +}What's all this do?
Quoted 2 lines
Work around init ordering problems. Quite likely there is a much nicer way to this that I should know about and don't.
Well, without adequate description of the problem, nobody can help solve it!
Re: [PATCH 02/14] add blockconsole version 1.1
Quoted 12 lines · Andrew Morton
Quoted 10 lines · On Wed, 22 May 2013 17:04:16 -0400 J__rn Engel <[email protected]> wrote:
Quoted 5 lines
Quoted 3 lines
Documentation/block/blockconsole.txt | 94 ++++ Documentation/block/blockconsole/bcon_tail | 62 +++ Documentation/block/blockconsole/mkblockconsole | 29 ++We really need somewhere better to put userspace tools.
Agreed. It started as "use this horrible code as a bad example" and turned into a repository for userspace code, which it should have been.
Quoted 5 lines
Quoted 2 lines
Do you have an opinion on
tools/vs. a standalong git tree for the tools?
./tools/blockconsole/sounds nice. I like maintaining simple userspace utils in the main kernel tree - it's very easy and efficient for us.
tools/blockconsole/ is shall be!
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
Quoted 8 lines
drivers/block/blockconsole.c new file +8
-0+#define BCON_MAX_ERRORS 10 +static void bcon_end_io (struct bio *bio,int err) +{ + struct bcon_bio *bcon_bio = container_of(bio, struct bcon_bio, bio); +struct blockconsole *bc = bio->bi_private; +unsigned long flags; + +/* drivers/block/blockconsole.c line 310
Quoted 10 lines
drivers/block/blockconsole.c new file +10
-0+* We want to assume the device broken and free this console if +* we accumulate too many errors. But if errors are transient, +* we also want to forget about them once writes succeed again. +* Oh, and we only want to reset the counter if it hasn't reached +* the limit yet, so we don't bcon_put() twice from here. +*/ +spin_lock_irqsave (&bc->end_io_lock, flags); +if (err) { +if (bc->error_count++ ==BCON_MAX_ERRORS ) { +pr_info ("no longer logging to %s\n" , bc->devname);
pr_info()may be too low a severity level?
Quoted 8 lines
Quoted 5 lines
There is no distinction between "someone pulled the usb device" and "broken cable". Blockconsole will continue writing and retire the device if errors accumulate. The code is deliberately stupid, because stupid code tends to be more robust and devoid of strange corner-cases.
I meant should we use
pr_err()orpr_emerg()rather thanpr_info(). Chances are thatpr_info()is being hidden.
Agreed. Trouble is that pr_info() is about right for "someone pulled the usb device", which pr_err() or pr_emerg() is about right for "broken hardware". So either choice will be wrong some of the time.
I guess in the end very few people care about debugging blockconsole failures. Those that run blockconsole tend to care about debugging something else already. So if you lean one way or the other, you have a stronger opinion than me.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
How does the code handle the obvious recursion concern here?
I think your next comment answers that.
hm, that was clever of me. I meant more generally, if we do a
printk()from within this code how do we prevent arbitrarily deep recursion?
I think in general you have to be careful and think twice before adding a printk. Which is why there are only two instances so far. The more interesting case is in functions called from blockconsole (or netconsole or serial console or...). And here the small set of functions when writing to the buffer should be safe and the much larger and complex set of functions for writeback is in the context of a different thread.
I have some fuzzy memory of this coming up in the past for netconsole. Wasn't hard to fix, which explains the lack of detailed memory.
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
drivers/block/blockconsole.c line 421
Quoted 11 lines
drivers/block/blockconsole.c new file +11
-0+memcpy (bc->bio_array[i].sector + +__bcon_console_ofs (console_bytes), msg, n); + len -= n; + msg += n; +bcon_advance_console_bytes (bc, n); + } +wake_up_process (bc->writeback_thread); +mod_timer (&bc->pad_timer, jiffies +HZ ); +} + +static void bcon_init_bios (struct blockconsole *bc)This code really needs some comments :(
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Oh I see. It's BIOs, not BIOS. Had me fooled there.
But, but, but isn't this obvious to someone who has just written said function? It has been a year now, so my own definition of obvious may have changed as well.
heh. I blame Jens for calling it a "bio".
I wonder what his response is when a recruiter asks him for a recent bio.
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 4 lines
drivers/block/blockconsole.c new file +4
-0+void bcon_add (const char *name) +{ +/* +* We add each name to a small static buffer and ask for a workqueue drivers/block/blockconsole.c line 583
Quoted 10 lines
drivers/block/blockconsole.c new file +10
-0+* to go pick it up asap. Once it is picked up, the buffer is empty +* again, so hopefully it will suffice for all sane users. +*/ +spin_lock (&device_lock); +if (scanned_devices[0 ]) +strncat (scanned_devices,"," ,sizeof (scanned_devices)); +strncat (scanned_devices, name,sizeof (scanned_devices)); +spin_unlock (&device_lock); +schedule_work (&bcon_add_work); +}What's all this do?
Quoted 5 lines
Quoted 2 lines
Work around init ordering problems. Quite likely there is a much nicer way to this that I should know about and don't.
Well, without adequate description of the problem, nobody can help solve it!
Will dig through the dark corners of my memory or reproduce it, whichever is easier.
Jörn
--
When in doubt, use brute force.
-- Ken Thompson
PATCH 1/1 mpt2sas/mpt3sas: prevent double free on error path
In reply to Joern EngelI noticed this one when list_del was called with poisoned list pointers, but the real problem is a double-free (and a use-after-free just before that).
Both _scsih_probe_boot_devices() and _scsih_sas_device_add() put the sas_device onto a list, thereby giving up control. Next they call mpt2sas_transport_port_add() and will list_del and free the object on errors. If some other function already did the list_del and free, it will happen again.
This patch adds reference counting to prevent the double free. One reference count goes to the caller of mpt2sas_transport_port_add(), the second to the list. Whoever removes the object from the list gets to drop one reference count. _scsih_probe_boot_devices() and _scsih_sas_device_add() get a second reference count to ensure the object is not freed while they are still accessing it.
To prevent the double list_del(), I changed the code to list_del_init() and added a list_empty() check before that. Since the list_empty/list_del_init is always called under a lock, this should be safe.
I hate the complexity this patch adds, but see no alternative to it.
mpt2sas0: failure at drivers/scsi/mpt2sas/mpt2sas_transport.c:708/mpt2sas_transport_port_add()!
general protection fault: 0000 [#1] SMP CPU 9 Pid: 3097, comm: kworker/u:11 Tainted: G W O 3.6.10+ #31392.trunk /0JP31P RIP: 0010:[<ffffffffa0309744>] [<ffffffffa0309744>] _scsih_sas_device_remove+0x54/0x90 [mpt2sas] RSP: 0018:ffff881fed4d7ab0 EFLAGS: 00010046 RAX: dead000000200200 RBX: ffff881ff6a5cd88 RCX: 00000000000010e8 RDX: ffff881ff7dab800 RSI: ffff881ff7daba00 RDI: dead000000100100 RBP: ffff881fed4d7ad0 R08: dead000000200200 R09: ffff880fff802200 R10: ffffffffa0317980 R11: 0000000000000000 R12: ffff881ff7daba00 R13: 0000000000000286 R14: 500605ba006c9d09 R15: ffff881ff7daba00 FS: 0000000000000000(0000) GS:ffff88203fc80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f8ac89ec458 CR3: 0000001ff4c5c000 CR4: 00000000000407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kworker/u:11 (pid: 3097, threadinfo ffff881fed4d6000, task ffff881402f3d9c0) Stack: 0000000000000401 ffff881ff6a5c6b0 0000000000000401 0000000000000016 ffff881fed4d7bb0 ffffffffa030f93e 0000000000000000 ffff881ff6a5cd88 0012000e0f000008 006c9d090002000b 00180009500605ba 0000040100000016 Call Trace: [<ffffffffa030f93e>] _scsih_add_device.clone.32+0x2fe/0x420 [mpt2sas] [<ffffffffa03126e5>] _scsih_sas_topology_change_event.clone.38+0x285/0x620 [mpt2sas] [<ffffffff81078c90>] ? load_balance+0x100/0x7a0 [<ffffffffa0312a80>] ? _scsih_sas_topology_change_event.clone.38+0x620/0x620 [mpt2sas] [<ffffffffa0312d8a>] _firmware_event_work+0x30a/0xfc0 [mpt2sas] [<ffffffff810015cc>] ? __switch_to+0x14c/0x410 [<ffffffff8106dfdb>] ? finish_task_switch+0x4b/0xf0 [<ffffffffa0312a80>] ? _scsih_sas_topology_change_event.clone.38+0x620/0x620 [mpt2sas] [<ffffffff8105bf40>] process_one_work+0x140/0x500 [<ffffffff8105d354>] worker_thread+0x194/0x510 [<ffffffff8106dfdb>] ? finish_task_switch+0x4b/0xf0 [<ffffffff8105d1c0>] ? manage_workers+0x320/0x320 [<ffffffff8106282e>] kthread+0x9e/0xb0 [<ffffffff815bef44>] kernel_thread_helper+0x4/0x10 [<ffffffff815b5e5d>] ? retint_restore_args+0x13/0x13 [<ffffffff81062790>] ? kthread_freezable_should_stop+0x70/0x70 [<ffffffff815bef40>] ? gs_change+0x13/0x13
--- drivers/scsi/mpt2sas/mpt2sas_base.h | 1 + drivers/scsi/mpt2sas/mpt2sas_scsih.c | 57 ++++++++++++++++++++++++++++------ drivers/scsi/mpt3sas/mpt3sas_base.h | 1 + drivers/scsi/mpt3sas/mpt3sas_scsih.c | 57 ++++++++++++++++++++++++++++------ 4 files changed, 98 insertions(+), 18 deletions(-)
drivers/scsi/mpt2sas/mpt2sas_base.h +1 -0
@@ -367,6 +367,7 @@ struct _sas_device {u16 slot;u8 phy;u8 responding; +struct kref kref; };/**
drivers/scsi/mpt2sas/mpt2sas_scsih.c +48 -9
137 lines not on this page · raw message
drivers/scsi/mpt3sas/mpt3sas_base.h +1 -0
@@ -293,6 +293,7 @@ struct _sas_device {u8 phy;u8 responding;u8 fast_path; +struct kref kref; };/**
drivers/scsi/mpt3sas/mpt3sas_scsih.c +48 -9
137 lines not on this page · raw message
Signature · 1 line
1.7.10.4
Re: [PATCH] mpt2sas/mpt3sas: prevent double free on error path
Quoted 1 line · Joern Engel
Subject: [PATCH] mpt2sas/mpt3sas: prevent double free on error path
The two mtp[23]sas patches shouldn't have been in here. Sorry about the noise!
Jörn
--
Das Aufregende am Schreiben ist es, eine Ordnung zu schaffen, wo vorher keine existiert hat.
-- Doris Lessing
Re: [PATCH] mpt2sas/mpt3sas: prevent double free on error path
In reply to Joern EngelQuoted 17 lines · On Thu, 9 May 2013 16:42:58 -0400 Joern Engel <[email protected]> wrote:
I noticed this one when
list_delwas called with poisoned list pointers, but the real problem is a double-free (and a use-after-free just before that).Both
_scsih_probe_boot_devices()and_scsih_sas_device_add()put thesas_deviceonto a list, thereby giving up control. Next they callmpt2sas_transport_port_add()and willlist_deland free the object on errors. If some other function already did thelist_deland free, it will happen again.This patch adds reference counting to prevent the double free. One reference count goes to the caller of
mpt2sas_transport_port_add(), the second to the list. Whoever removes the object from the list gets to drop one reference count._scsih_probe_boot_devices()and_scsih_sas_device_add()get a second reference count to ensure the object is not freed while they are still accessing it.
Quoted 6 lines
To prevent the double
list_del(), I changed the code tolist_del_init()and added alist_empty()check before that. Since the list_empty/list_del_init is always called under a lock, this should be safe.I hate the complexity this patch adds, but see no alternative to it.
It's regrettable that this bugfix is joined at the hip with a new kernel feature.
Quoted 16 lines
index 543d8d6..ceb7d41 100644
drivers/scsi/mpt2sas/mpt2sas_base.h +1
-0@@ -367,6 +367,7 @@ struct _sas_device {u16 slot;u8 phy;u8 responding; +struct kref kref; };/** drivers/scsi/mpt2sas/mpt2sas_scsih.c +0
-0
drivers/scsi/mpt2sas/mpt2sas_scsih.c _scsih_sas_device_find_by_handle(struct MPT2SAS_ADAPTER *ioc, u16 handle) · line 576
Quoted 7 lines
drivers/scsi/mpt2sas/mpt2sas_scsih.c +4
-0@@ -570,6 +570,19 @@ _scsih_sas_device_find_by_handle(struct MPT2SAS_ADAPTER *ioc, u16 handle)return NULL ; } +static void free_sas_device (struct kref *kref) +{ + struct _sas_device *sas_device = container_of(kref, struct _sas_device, + kref);
yuk. Easily fixed:
struct _sas_device *sas_device; sas_device = container_of(kref, struct _sas_device, kref);
Quoted 21 lines
drivers/scsi/mpt2sas/mpt2sas_scsih.c +9
-0+kfree (sas_device); +} + +static void put_sas_device (struct _sas_device *sas_device) +{ +kref_put (&sas_device->kref, free_sas_device); +} +/** * _scsih_sas_device_remove - remove sas_device from list. * @ioc: per adapter object @@ -583,14 +596,19 @@ _scsih_sas_device_remove(struct MPT2SAS_ADAPTER *ioc, struct _sas_device *sas_device) {unsigned long flags; +int was_on_list =0 ;if (!sas_device)return ;spin_lock_irqsave (&ioc->sas_device_lock, flags);
drivers/scsi/mpt2sas/mpt2sas_scsih.c _scsih_sas_device_remove(struct MPT2SAS_ADAPTER *ioc, · line 612
Quoted 10 lines
drivers/scsi/mpt2sas/mpt2sas_scsih.c +6
-2-list_del (&sas_device->list); -kfree (sas_device); +if (!list_empty (&sas_device->list)) { +list_del_init (&sas_device->list); + was_on_list =1 ; + }spin_unlock_irqrestore (&ioc->sas_device_lock, flags); +if (was_on_list) +put_sas_device (sas_device); }
This looks a bit awkward.
- list_del() on an empty list_head is acceptable, although inefficient when used in a high-frequency codepath (whcih this isn't). So the code could be made simpler-looking.
- afaict put_sas_device() can be called from under that lock, so why bother with the was_on_list deferral thing?
- it's strange that this function can ever be called when the device is not on the list. Perhaps somethnig else is screwy?
Quoted 13 lines
drivers/scsi/mpt2sas/mpt2sas_scsih.c +2
-0@@ -612,6 +630,8 @@ _scsih_sas_device_add(struct MPT2SAS_ADAPTER *ioc,"(0x%04x), sas_addr(0x%016llx)\n" , ioc->name, __func__, sas_device->handle, (unsigned long long)sas_device->sas_address)); +/* Get an extra refcount... */ From the annals of unuseful comments ;) +kref_get (&sas_device->kref);drivers/scsi/mpt2sas/mpt2sas_scsih.c +0
-0spin_lock_irqsave (&ioc->sas_device_lock, flags);list_add_tail (&sas_device->list, &ioc->sas_device_list);spin_unlock_irqrestore (&ioc->sas_device_lock, flags);
drivers/scsi/mpt2sas/mpt2sas_scsih.c _scsih_sas_device_add(struct MPT2SAS_ADAPTER *ioc, · line 654
Quoted 5 lines
drivers/scsi/mpt2sas/mpt2sas_scsih.c +2
-0@@ -630,6 +650,12 @@ _scsih_sas_device_add(struct MPT2SAS_ADAPTER *ioc, sas_device->sas_address_parent);_scsih_sas_device_remove (ioc, sas_device); } +/* +* ...and drop it again. If an error already happend, this is the
"happened"
drivers/scsi/mpt2sas/mpt2sas_scsih.c _scsih_sas_device_add(struct MPT2SAS_ADAPTER *ioc, · line 659
Quoted 5 lines
drivers/scsi/mpt2sas/mpt2sas_scsih.c +4
-0+* final put and we free the object now. Otherwise whoever removes +* the object from the list will do the final put and free. +*/ +put_sas_device (sas_device); }
But if this kref_get/put_sas_device pair were not here, this function's _scsih_sas_device_remove() could free the device for us, if that was_on_list oddity wasn't there.
Perhaps it would help to write down the refcounting rules. "one for the object, one for the presence of a list"?
Quoted 32 lines
drivers/scsi/mpt2sas/mpt2sas_scsih.c +4
-3/** @@ -5270,6 +5296,7 @@ _scsih_add_device(struct MPT2SAS_ADAPTER *ioc, u16 handle, u8 phy_num, u8 is_pd)return -1 ; } +kref_init (&sas_device->kref); sas_device->handle = handle; if (_scsih_get_sas_address(ioc, le16_to_cpu (sas_device_pg0.ParentDevHandle),@@ -5341,7 +5368,7 @@ _scsih_remove_device(struct MPT2SAS_ADAPTER *ioc,"handle(0x%04x), sas_addr(0x%016llx)\n" , ioc->name, __func__, sas_device->handle, (unsigned long long ) sas_device->sas_address)); -kfree (sas_device); +put_sas_device (sas_device); }/** * _scsih_device_remove_by_handle - removing device object by handle @@ -5355,16 +5382,21 @@ _scsih_device_remove_by_handle(struct MPT2SAS_ADAPTER *ioc, u16 handle) {struct _sas_device *sas_device;unsigned long flags; +int was_on_list =0 ;if (ioc->shost_recovery)return ;spin_lock_irqsave (&ioc->sas_device_lock, flags); sas_device =_scsih_sas_device_find_by_handle (ioc, handle); -if (sas_device) -list_del (&sas_device->list); +if (sas_device) {
drivers/scsi/mpt2sas/mpt2sas_scsih.c _scsih_device_remove_by_handle(struct MPT2SAS_ADAPTER *ioc, u16 handle) · line 5401
Quoted 10 lines
drivers/scsi/mpt2sas/mpt2sas_scsih.c +6
-1+if (!list_empty (&sas_device->list)) { +list_del_init (&sas_device->list); + was_on_list =1 ; + } + }spin_unlock_irqrestore (&ioc->sas_device_lock, flags); -if (sas_device) +if (was_on_list)_scsih_remove_device (ioc, sas_device); }
hm, duplication. Should this call _scsih_sas_device_remove()?
Quoted 15 lines
drivers/scsi/mpt2sas/mpt2sas_scsih.c +2
-2@@ -5381,6 +5413,7 @@ mpt2sas_device_remove_by_sas_address(struct MPT2SAS_ADAPTER *ioc, {struct _sas_device *sas_device;unsigned long flags; +int was_on_list =0 ;if (ioc->shost_recovery)return ;@@ -5388,10 +5421,14 @@ mpt2sas_device_remove_by_sas_address(struct MPT2SAS_ADAPTER *ioc,spin_lock_irqsave (&ioc->sas_device_lock, flags); sas_device =mpt2sas_scsih_sas_device_find_by_sas_address (ioc, sas_address); -if (sas_device) -list_del (&sas_device->list); +if (sas_device) {
drivers/scsi/mpt2sas/mpt2sas_scsih.c mpt2sas_device_remove_by_sas_address(struct MPT2SAS_ADAPTER *ioc, · line 5433
Quoted 10 lines
drivers/scsi/mpt2sas/mpt2sas_scsih.c +6
-1+if (!list_empty (&sas_device->list)) { +list_del_init (&sas_device->list); + was_on_list =1 ; + } + }spin_unlock_irqrestore (&ioc->sas_device_lock, flags); -if (sas_device) +if (was_on_list)_scsih_remove_device (ioc, sas_device); }
eek, there it is again!
drivers/scsi/mpt3sas/mpt3sas_scsih.c _scsih_sas_device_find_by_handle(struct MPT3SAS_ADAPTER *ioc, u16 handle) · line 575
Quoted 12 lines
...
drivers/scsi/mpt3sas/mpt3sas_scsih.c +4
-0@@ -569,6 +569,19 @@ _scsih_sas_device_find_by_handle(struct MPT3SAS_ADAPTER *ioc, u16 handle)return NULL ; } +static void free_sas_device (struct kref *kref) +{ + struct _sas_device *sas_device = container_of(kref, struct _sas_device, + kref);
A ditto here.
Quoted 21 lines
drivers/scsi/mpt3sas/mpt3sas_scsih.c +9
-0+kfree (sas_device); +} + +static void put_sas_device (struct _sas_device *sas_device) +{ +kref_put (&sas_device->kref, free_sas_device); +} +/** * _scsih_sas_device_remove - remove sas_device from list. * @ioc: per adapter object @@ -582,14 +595,19 @@ _scsih_sas_device_remove(struct MPT3SAS_ADAPTER *ioc, struct _sas_device *sas_device) {unsigned long flags; +int was_on_list =0 ;if (!sas_device)return ;spin_lock_irqsave (&ioc->sas_device_lock, flags);
drivers/scsi/mpt3sas/mpt3sas_scsih.c _scsih_sas_device_remove(struct MPT3SAS_ADAPTER *ioc, · line 611
Quoted 10 lines
drivers/scsi/mpt3sas/mpt3sas_scsih.c +6
-2-list_del (&sas_device->list); -kfree (sas_device); +if (!list_empty (&sas_device->list)) { +list_del_init (&sas_device->list); + was_on_list =1 ; + }spin_unlock_irqrestore (&ioc->sas_device_lock, flags); +if (was_on_list) +put_sas_device (sas_device); }
etcetera.
I dunno, it all seems quite duplicative and awkward. Is there something exceptional about the problem space here, or is it that the code is just screwed up and this fix doesn't really unscrew it?
Re: [PATCH] mpt2sas/mpt3sas: prevent double free on error path
Quoted 3 lines · Andrew Morton
It's regrettable that this bugfix is joined at the hip with a new kernel feature.
Not merged at the hip, I merely ended up reusing an old temp directory for the new patch series (hips for brains?). Ignore this patch.
Jörn
--
No art, however minor, demands less than total dedication if you want to excel in it.
-- Leon Battista Alberti
PATCH 3/14 printk: add CON_ALLDATA console flag
In reply to Joern EngelFor consoles like netconsole and blockconsole the loglevel filtering really doesn't make any sense. If a line gets printed at all, please send it down to that console, no questions asked.
For vga_con, it is a completely different matter, as the user sitting in front of his console could get spammed by messages while trying to login or similar. So ignore_loglevel doesn't work as a one-size-fits-all approach. Add a per-console flag instead so that netconsole and blockconsole can opt-in.
--- include/linux/console.h | 1 + kernel/printk.c | 5 +++-- 2 files changed, 4 insertions(+), 2 deletions(-)
include/linux/console.h +1 -0
@@ -116,6 +116,7 @@ static inline int con_debug_leave(void)#define CON_BOOT (8)#define CON_ANYTIME (16)/* Safe to call when cpu is offline */ #define CON_BRL (32)/* Used for a braille device */ +#define CON_ALLDATA (64)/* per-console ignore_loglevel */ struct console {char name[16 ];
kernel/printk.c +3 -2
@@ -1261,8 +1261,6 @@ static void call_console_drivers(int level, const char *text, size_t len)trace_console (text,0 , len, len); -if (level >= console_loglevel && !ignore_loglevel) -return ;if (!console_drivers)return ;
@@ -1276,6 +1274,9 @@ static void call_console_drivers(int level, const char *text, size_t len)if (!cpu_online (smp_processor_id ()) && !(con->flags &CON_ANYTIME ))continue ; +if (level >= console_loglevel && !ignore_loglevel && + !(con->flags &CON_ALLDATA )) +continue ; con->write (con, text, len); } }
Signature · 1 line
1.7.10.4
PATCH 1/1 mpt2sas: prevent double free on error path
In reply to Joern EngelI noticed this one when list_del was called with poisoned list pointers, but the real problem is a double-free (and a use-after-free just before that).
Both _scsih_probe_boot_devices() and _scsih_sas_device_add() put the sas_device onto a list, thereby giving up control. Next they call mpt2sas_transport_port_add() and will list_del and free the object on errors. If some other function already did the list_del and free, it will happen again.
This patch adds reference counting to prevent the double free. One reference count goes to the caller of mpt2sas_transport_port_add(), the second to the list. Whoever removes the object from the list gets to drop one reference count. _scsih_probe_boot_devices() and _scsih_sas_device_add() get a second reference count to ensure the object is not freed while they are still accessing it.
To prevent the double list_del(), I changed the code to list_del_init() and added a list_empty() check before that. Since the list_empty/list_del_init is always called under a lock, this should be safe.
I hate the complexity this patch adds, but see no alternative to it.
mpt2sas0: failure at drivers/scsi/mpt2sas/mpt2sas_transport.c:708/mpt2sas_transport_port_add()!
general protection fault: 0000 [#1] SMP CPU 9 Pid: 3097, comm: kworker/u:11 Tainted: G W O 3.6.10+ #31392.trunk /0JP31P RIP: 0010:[<ffffffffa0309744>] [<ffffffffa0309744>] _scsih_sas_device_remove+0x54/0x90 [mpt2sas] RSP: 0018:ffff881fed4d7ab0 EFLAGS: 00010046 RAX: dead000000200200 RBX: ffff881ff6a5cd88 RCX: 00000000000010e8 RDX: ffff881ff7dab800 RSI: ffff881ff7daba00 RDI: dead000000100100 RBP: ffff881fed4d7ad0 R08: dead000000200200 R09: ffff880fff802200 R10: ffffffffa0317980 R11: 0000000000000000 R12: ffff881ff7daba00 R13: 0000000000000286 R14: 500605ba006c9d09 R15: ffff881ff7daba00 FS: 0000000000000000(0000) GS:ffff88203fc80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f8ac89ec458 CR3: 0000001ff4c5c000 CR4: 00000000000407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kworker/u:11 (pid: 3097, threadinfo ffff881fed4d6000, task ffff881402f3d9c0) Stack: 0000000000000401 ffff881ff6a5c6b0 0000000000000401 0000000000000016 ffff881fed4d7bb0 ffffffffa030f93e 0000000000000000 ffff881ff6a5cd88 0012000e0f000008 006c9d090002000b 00180009500605ba 0000040100000016 Call Trace: [<ffffffffa030f93e>] _scsih_add_device.clone.32+0x2fe/0x420 [mpt2sas] [<ffffffffa03126e5>] _scsih_sas_topology_change_event.clone.38+0x285/0x620 [mpt2sas] [<ffffffff81078c90>] ? load_balance+0x100/0x7a0 [<ffffffffa0312a80>] ? _scsih_sas_topology_change_event.clone.38+0x620/0x620 [mpt2sas] [<ffffffffa0312d8a>] _firmware_event_work+0x30a/0xfc0 [mpt2sas] [<ffffffff810015cc>] ? __switch_to+0x14c/0x410 [<ffffffff8106dfdb>] ? finish_task_switch+0x4b/0xf0 [<ffffffffa0312a80>] ? _scsih_sas_topology_change_event.clone.38+0x620/0x620 [mpt2sas] [<ffffffff8105bf40>] process_one_work+0x140/0x500 [<ffffffff8105d354>] worker_thread+0x194/0x510 [<ffffffff8106dfdb>] ? finish_task_switch+0x4b/0xf0 [<ffffffff8105d1c0>] ? manage_workers+0x320/0x320 [<ffffffff8106282e>] kthread+0x9e/0xb0 [<ffffffff815bef44>] kernel_thread_helper+0x4/0x10 [<ffffffff815b5e5d>] ? retint_restore_args+0x13/0x13 [<ffffffff81062790>] ? kthread_freezable_should_stop+0x70/0x70 [<ffffffff815bef40>] ? gs_change+0x13/0x13
--- drivers/scsi/mpt2sas/mpt2sas_base.h | 1 + drivers/scsi/mpt2sas/mpt2sas_scsih.c | 55 ++++++++++++++++++++++++++++------ 2 files changed, 47 insertions(+), 9 deletions(-)
drivers/scsi/mpt2sas/mpt2sas_base.h +1 -0
@@ -367,6 +367,7 @@ struct _sas_device {u16 slot;u8 phy;u8 responding; +struct kref kref; };/**
drivers/scsi/mpt2sas/mpt2sas_scsih.c +46 -9
129 lines not on this page · raw message
Signature · 1 line
1.7.10.4
PATCH 4/14 netconsole: use CON_ALLDATA
In reply to Joern EngelNetconsole should really see every message ever printed. The alternative is to try debugging with information like this:
[166135.633974] Stack: [166135.634016] Call Trace: [166135.634029] <IRQ> [166135.634156] <EOI> [166135.634177] Code: 00 00 55 48 89 e5 0f 1f 44 00 00 ff 15 31 49 80 00 c9 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 1f 44 00 00 [166135.634384] 48 8b 14 25 98 24 01 00 48 8d 14 92 48 8d 04 bd 00 00 00 00
--- drivers/net/netconsole.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/netconsole.c +1 -1
@@ -718,7 +718,7 @@ static void write_msg(struct console *con, const char *msg, unsigned int len)static struct console netconsole = { .name ="netcon" , - .flags =CON_ENABLED , + .flags =CON_ENABLED |CON_ALLDATA , .write = write_msg, };
Signature · 1 line
1.7.10.4
PATCH 5/14 blockconsole: use CON_ALLDATA
In reply to Joern EngelBlockconsole should really see every message ever printed. The alternative is to try debugging with information like this:
[166135.633974] Stack: [166135.634016] Call Trace: [166135.634029] <IRQ> [166135.634156] <EOI> [166135.634177] Code: 00 00 55 48 89 e5 0f 1f 44 00 00 ff 15 31 49 80 00 c9 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 1f 44 00 00 [166135.634384] 48 8b 14 25 98 24 01 00 48 8d 14 92 48 8d 04 bd 00 00 00 00
--- drivers/block/blockconsole.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/block/blockconsole.c +1 -1
@@ -481,7 +481,7 @@ static int bcon_create(const char *devname)strlcpy (bc->devname, devname,sizeof (bc->devname));spin_lock_init (&bc->end_io_lock);strcpy (bc->console.name,"bcon" ); - bc->console.flags =CON_PRINTBUFFER |CON_ENABLED ; + bc->console.flags =CON_PRINTBUFFER |CON_ENABLED |CON_ALLDATA ; bc->console.write = bcon_write; bc->bdev =blkdev_get_by_path (devname, mode,NULL );#ifndef MODULE
Signature · 1 line
1.7.10.4
PATCH 6/14 bcon: add a release work struct
In reply to Joern EngelThe final bcon_put() can be called from atomic context, by way of bio_endio(). In that case we would sleep in invalidate_mapping_pages(), with the usual unhappy results.
In nearly a year of production use, I have only seen a matching backtrace once. There was a second known issue that could be reproduced by "yes h > /proc/sysrq-trigger" and concurrently pulling and replugging the blockconsole device. It took be somewhere around 30 pulls and sore thumbs to reproduce and I never found the time to get to the bottom of it. Quite likely the two issues are identical.
--- drivers/block/blockconsole.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-)
drivers/block/blockconsole.c +13 -2
@@ -65,6 +65,7 @@ struct blockconsole {struct block_device *bdev;struct console console;struct work_struct unregister_work; +struct work_struct release_work;struct task_struct *writeback_thread;struct notifier_block panic_block; };
@@ -74,9 +75,10 @@ static void bcon_get(struct blockconsole *bc)kref_get (&bc->kref); } -static void bcon_release (struct kref *kref) +static void __bcon_release (struct work_struct *work) { - struct blockconsole *bc = container_of(kref, struct blockconsole, kref); + struct blockconsole *bc = container_of(work, struct blockconsole, + release_work);__free_pages (bc->zero_page,0 );__free_pages (bc->pages,8 );
@@ -85,6 +87,14 @@ static void bcon_release(struct kref *kref)kfree (bc); } +static void bcon_release (struct kref *kref) +{ + struct blockconsole *bc = container_of(kref, struct blockconsole, kref); + +/* bcon_release can be called from atomic context */ +schedule_work (&bc->release_work); +} +static void bcon_put (struct blockconsole *bc) {kref_put (&bc->kref, bcon_release);
@@ -512,6 +522,7 @@ static int bcon_create(const char *devname)if (IS_ERR (bc->writeback_thread)) goto out2;INIT_WORK (&bc->unregister_work, bcon_unregister); +INIT_WORK (&bc->release_work, __bcon_release);register_console (&bc->console); bc->panic_block.notifier_call = blockconsole_panic; bc->panic_block.priority =INT_MAX ;
Signature · 1 line
1.7.10.4
Re: [PATCH 06/14] bcon: add a release work struct
Quoted 10 lines · On Thu, 9 May 2013 16:43:04 -0400 Joern Engel <[email protected]> wrote:
The final
bcon_put()can be called from atomic context, by way ofbio_endio(). In that case we would sleep ininvalidate_mapping_pages(), with the usual unhappy results.In nearly a year of production use, I have only seen a matching backtrace once. There was a second known issue that could be reproduced by "yes h >
/proc/sysrq-trigger" and concurrently pulling and replugging the blockconsole device. It took be somewhere around 30 pulls and sore thumbs to reproduce and I never found the time to get to the bottom of it. Quite likely the two issues are identical.
This fix should be folded into the base patch?
PATCH 7/14 bcon: check for hdparm in bcon_tail
In reply to Joern EngelFrom: Borislav Petkov <[email protected]>
--- Documentation/block/blockconsole/bcon_tail | 5 +++++ 1 file changed, 5 insertions(+)
Documentation/block/blockconsole/bcon_tail +5 -0
@@ -14,6 +14,11 @@ if [ -z "$(which lsscsi)" ]; then exit 1 fi +if [ -z "$(which hdparm)" ]; then + echo "You need to install the hdparm package on your distro." + exit 1 +fi + end_of_log() { DEV=$1 UUID=`head -c40 $DEV|tail -c8`
Signature · 1 line
1.7.10.4
Re: [PATCH 07/14] bcon: check for hdparm in bcon_tail
Quoted 6 lines · On Thu, 9 May 2013 16:43:05 -0400 Joern Engel <[email protected]> wrote:
...
Documentation/block/blockconsole/bcon_tail +0
-0@@ -14,6 +14,11 @@ if [ -z "$(which lsscsi)" ]; then exit 1
Documentation/block/blockconsole/bcon_tail if [ -z "$(which lsscsi)" ]; then · line 24
Quoted 10 lines
Documentation/block/blockconsole/bcon_tail +5
-0fi +if [ -z "$(which hdparm)" ]; then + echo "You need to install the hdparm package on your distro." + exit 1 +fi + end_of_log() { DEV=$1 UUID=`head -c40 $DEV|tail -c8`
Fold this one as well?
PATCH 8/14 blockconsole: Allow to pass a device file path to bcon_tail
In reply to Joern EngelFrom: Takashi Iwai <[email protected]>
... instead of always looking through all devices.
Minor tweak: Moved the "CANDIDATES=" line below Takashi's new code.
--- Documentation/block/blockconsole/bcon_tail | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
Documentation/block/blockconsole/bcon_tail +15 -0
@@ -58,6 +58,21 @@ end_of_log() { # HEADER contains a newline, so the funny quoting is necessary HEADER=' Linux blockconsole version 1.1' + +DEV="$1" +if [ -n "$DEV" ]; then + if [ ! -b "$DEV" ]; then + echo "bcon_tail: No block device file $DEV" + exit 1 + fi + if [ "`head -c32 $DEV`" != "$HEADER" ]; then + echo "bcon_tail: Invalid device file $DEV" + exit 1 + fi + end_of_log $DEV + exit 0 +fi + CANDIDATES=`lsscsi |sed 's|.*/dev|/dev|'` for DEV in $CANDIDATES; do
Signature · 1 line
1.7.10.4
PATCH 9/14 bcon: remove version 1.0 support
In reply to Joern EngelVery few machines ever ran with 1.0 format and by now I doubt whether a single one still does. No need to carry that code along.
--- drivers/block/blockconsole.c | 19 ++----------------- 1 file changed, 2 insertions(+), 17 deletions(-)
drivers/block/blockconsole.c +2 -17
@@ -18,7 +18,6 @@#include <linux/sched.h> #include <linux/ctype.h> -#define BLOCKCONSOLE_MAGIC_OLD "\nLinux blockconsole version 1.0\n"#define BLOCKCONSOLE_MAGIC "\nLinux blockconsole version 1.1\n"#define BCON_UUID_OFS (32)#define BCON_ROUND_OFS (41)
@@ -234,18 +233,6 @@ static void bcon_advance_write_bytes(struct blockconsole *bc, int bytes) } } -static int bcon_convert_old_format (struct blockconsole *bc) -{ - bc->uuid =get_random_int (); - bc->round =0 ; - bc->console_bytes = bc->write_bytes =0 ; -bcon_advance_console_bytes (bc,0 );/* To skip the header */ -bcon_advance_write_bytes (bc,0 );/* To wrap around, if necessary */ -bcon_erase_segment (bc); -pr_info ("converted %s from old format\n" , bc->devname); -return 0 ; -} -static int bcon_find_end_of_log (struct blockconsole *bc) {u64 start =0 , end = bc->max_bytes, middle;
@@ -258,8 +245,8 @@ static int bcon_find_end_of_log(struct blockconsole *bc)return err;/* Second sanity check, out of sheer paranoia */ version =bcon_magic_present (sec0); -if (version ==10 ) -return bcon_convert_old_format (bc); +if (!version) +return -EINVAL ; bc->uuid =simple_strtoull (sec0 +BCON_UUID_OFS ,NULL ,16 ); bc->round =simple_strtoull (sec0 +BCON_ROUND_OFS ,NULL ,16 );
@@ -618,8 +605,6 @@ int bcon_magic_present(const void *data) {size_t len =strlen (BLOCKCONSOLE_MAGIC ); -if (!memcmp (data,BLOCKCONSOLE_MAGIC_OLD , len)) -return 10 ;if (memcmp (data,BLOCKCONSOLE_MAGIC , len))return 0 ;if (!is_four_byte_hex (data +BCON_UUID_OFS ))
Signature · 1 line
1.7.10.4
Re: [PATCH 09/14] bcon: remove version 1.0 support
Quoted 2 lines · On Thu, 9 May 2013 16:43:07 -0400 Joern Engel <[email protected]> wrote:
Very few machines ever ran with 1.0 format and by now I doubt whether a single one still does. No need to carry that code along.
aw geeze. Can we please clean up the patch series?
Re: [PATCH 09/14] bcon: remove version 1.0 support
Quoted 6 lines · Andrew Morton
Quoted 2 lines · On Thu, 9 May 2013 16:43:07 -0400 Joern Engel <[email protected]> wrote:
Very few machines ever ran with 1.0 format and by now I doubt whether a single one still does. No need to carry that code along.
aw geeze. Can we please clean up the patch series?
People started using my git tree, so I decided to keep it stable and not fold fixes into the main patch. That admittedly sucks for review.
Would you like me to create a single patch for review? And after that, do you want to carry a single patch in your series or should I keep my tree and try to get that merged? Either way works for me.
Jörn
--
Ninety percent of everything is crap.
-- Sturgeon's Law
Re: [PATCH 09/14] bcon: remove version 1.0 support
Quoted 6 lines · On Wed, 22 May 2013 15:51:09 -0400 J__rn Engel <[email protected]> wrote:
Quoted 5 lines · Andrew Morton
Quoted 2 lines · On Thu, 9 May 2013 16:43:07 -0400 Joern Engel <[email protected]> wrote:
Very few machines ever ran with 1.0 format and by now I doubt whether a single one still does. No need to carry that code along.
Quoted 9 lines
Quoted 1 line
aw geeze. Can we please clean up the patch series?
People started using my git tree, so I decided to keep it stable and not fold fixes into the main patch. That admittedly sucks for review.
Would you like me to create a single patch for review? And after that, do you want to carry a single patch in your series or should I keep my tree and try to get that merged? Either way works for me.
Well I do think a significant amount of change is needed, in particular some more use of the /* operator.
Can you set up a new tree for that and for linux-next and leave the existing one in place?
Re: [PATCH 09/14] bcon: remove version 1.0 support
Quoted 3 lines · Andrew Morton
Well I do think a significant amount of change is needed, in particular some more use of the /* operator.
Ack.
Quoted 2 lines
Can you set up a new tree for that and for linux-next and leave the existing one in place?
Will do. May take a few days, this is a busy time.
Jörn
--
Fancy algorithms are slow when n is small, and n is usually small. Fancy algorithms have big constants. Until you know that n is frequently going to be big, don't get fancy.
-- Rob Pike
PATCH 10/14 blockconsole: Fix undefined MAX_RT_PRIO
In reply to Joern EngelFrom: Takashi Iwai <[email protected]>
--- drivers/block/blockconsole.c | 1 + 1 file changed, 1 insertion(+)
drivers/block/blockconsole.c +1 -0
@@ -16,6 +16,7 @@#include <linux/string.h> #include <linux/workqueue.h> #include <linux/sched.h> +#include <linux/sched/rt.h> #include <linux/ctype.h> #define BLOCKCONSOLE_MAGIC "\nLinux blockconsole version 1.1\n"
Signature · 1 line
1.7.10.4
PATCH 11/14 blockconsole: Rename device_lock with bc_device_lock
In reply to Joern EngelFrom: Takashi Iwai <[email protected]>
Avoid the name conflict with device_lock() defined in linux/device.h.
--- drivers/block/blockconsole.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
drivers/block/blockconsole.c +5 -5
@@ -546,17 +546,17 @@ static void bcon_create_fuzzy(const char *name) } } -static DEFINE_SPINLOCK (device_lock); +static DEFINE_SPINLOCK (bcon_device_lock);static char scanned_devices[80 ];static void bcon_do_add (struct work_struct *work) {char local_devices[80 ], *name, *remainder = local_devices; -spin_lock (&device_lock); +spin_lock (&bcon_device_lock);memcpy (local_devices, scanned_devices,sizeof (local_devices));memset (scanned_devices,0 ,sizeof (scanned_devices)); -spin_unlock (&device_lock); +spin_unlock (&bcon_device_lock);while (remainder && remainder[0 ]) { name =strsep (&remainder,"," );
@@ -573,11 +573,11 @@ void bcon_add(const char *name)* to go pick it up asap. Once it is picked up, the buffer is empty * again, so hopefully it will suffice for all sane users. */ -spin_lock (&device_lock); +spin_lock (&bcon_device_lock);if (scanned_devices[0 ])strncat (scanned_devices,"," ,sizeof (scanned_devices));strncat (scanned_devices, name,sizeof (scanned_devices)); -spin_unlock (&device_lock); +spin_unlock (&bcon_device_lock);schedule_work (&bcon_add_work); }
Signature · 1 line
1.7.10.4
PATCH 12/14 blockconsole: Mark a local work struct static
In reply to Joern EngelFrom: Takashi Iwai <[email protected]>
--- drivers/block/blockconsole.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/block/blockconsole.c +1 -1
@@ -564,7 +564,7 @@ static void bcon_do_add(struct work_struct *work) } } -DECLARE_WORK (bcon_add_work, bcon_do_add); +static DECLARE_WORK(bcon_add_work, bcon_do_add);void bcon_add (const char *name) {
Signature · 1 line
1.7.10.4
PATCH 13/14 bcon: Fix wrap-around behaviour
In reply to Joern EngelThis seems to have broken around the introduction of format 1.1. When wrapping around, we should increment the wrap counter before writing it out, not after.
--- drivers/block/blockconsole.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/block/blockconsole.c +1 -1
@@ -229,8 +229,8 @@ static void bcon_advance_write_bytes(struct blockconsole *bc, int bytes) bc->write_bytes += bytes;if (bc->write_bytes >= bc->max_bytes) { bc->write_bytes =0 ; -bcon_init_first_page (bc); bc->round++; +bcon_init_first_page (bc); } }
Signature · 1 line
1.7.10.4
PATCH 14/14 netconsole: s/syslogd/cancd/ in documentation
In reply to Joern EngelUsing syslogd to capture netconsole is known to be broken, see for example https://bugzilla.redhat.com/show_bug.cgi?id=432160 or any of the many other bug reports. We should not advertise it, much less as a first choice. The fact that syslogd tends to initially work makes it worse, as that creates false hope.
Cancd is a syslog-for-netconsole of sorts and in my experience works better than any alternative for non-trivial setups, i.e. more than a single machine sending netconsole traffic.
Since my hacked-up version of cancd is no longer compatible with Oracle's original, I linked to both.
--- Documentation/networking/netconsole.txt | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-)
Documentation/networking/netconsole.txt +12 -4
@@ -54,9 +54,7 @@ address. The remote host has several options to receive the kernel messages, for example: -1) syslogd - -2) netcat +1) netcat On distributions using a BSD-based netcat version (e.g. Fedora, openSUSE and Ubuntu) the listening port must be specified without
@@ -65,10 +63,20 @@ for example: 'nc -u -l -p <port>' / 'nc -u -l <port>' or 'netcat -u -l -p <port>' / 'netcat -u -l <port>' -3) socat +2) socat 'socat udp-recv:<port> -' +3) cancd + + A daemon written specifically for netconsole that is good at capturing + output from many machines. Using netcat for several machines either + interleaves output from all machines or requires the use of per-machine + ports. + + https://git.kernel.org/cgit/linux/kernel/git/joern/cancd.git/ + https://oss.oracle.com/projects/cancd/ + Dynamic reconfiguration: ========================
Signature · 1 line
1.7.10.4
cover Add blockconsole version 1.1 (try 2)
In reply to Joern EngelBlockconsole is a console driver very roughly similar to netconsole. Instead of sending messages out via UDP, they are written to a block device. Typically a USB stick is chosen, although in principle any block device will do.
In most cases blockconsole is useful where netconsole is not, i.e. single machines without network access or without an accessable netconsole capture server. When using both blockconsole and netconsole, I have found netconsole to sometimes create a mess under high message load (sysrq-t, etc.) while blockconsole does not.
Most importantly, a number of bugs were identified and fixed that would have been unexplained machine reboots without blockconsole.
More highlights:
* reasonably small and self-contained code,
* some 100+ machine years of runtime,
* nice tutorial with a 30-sec guide for the impatient.
Special thanks to Borislav Petkov for many improvements and kicking my behind to provide a proper git tree and resend patches.
A number of cleanup patches could be folded into the main patch, but I decided not to mess with git history and leave any further mistakes for the world to laugh at:git://git.kernel.org/pub/scm/linux/kernel/git/joern/bcon2.git
Joern Engel (8):
do_mounts: constify name_to_dev_t parameter
add blockconsole version 1.1
printk: add CON_ALLDATA console flag
netconsole: use CON_ALLDATA
blockconsole: use CON_ALLDATA
bcon: add a release work struct
bcon: check for hdparm in bcon_tail
bcon: remove version 1.0 support
Takashi Iwai (1):
blockconsole: Allow to pass a device file path to bcon_tail
Documentation/block/blockconsole.txt | 94 ++++ Documentation/block/blockconsole/bcon_tail | 82 +++ Documentation/block/blockconsole/mkblockconsole | 29 ++ block/partitions/Makefile | 1 + block/partitions/blockconsole.c | 22 + block/partitions/check.c | 3 + block/partitions/check.h | 3 + drivers/block/Kconfig | 6 + drivers/block/Makefile | 1 + drivers/block/blockconsole.c | 617 +++++++++++++++++++++++ drivers/net/netconsole.c | 2 +- include/linux/blockconsole.h | 7 + include/linux/console.h | 1 + include/linux/mount.h | 2 +- init/do_mounts.c | 2 +- kernel/printk.c | 5 +- 16 files changed, 872 insertions(+), 5 deletions(-) create mode 100644 Documentation/block/blockconsole.txt create mode 100755 Documentation/block/blockconsole/bcon_tail create mode 100755 Documentation/block/blockconsole/mkblockconsole create mode 100644 block/partitions/blockconsole.c create mode 100644 drivers/block/blockconsole.c create mode 100644 include/linux/blockconsole.h
Signature · 1 line
1.7.10.4