GFS2: Add blktrace support to glocks
Subsystems (3)
Hi,
Since I last posted this pair of patches, I've done some extensive updating of the kernel patch, so it should now be happy to compile under all possible Kconfigs (fingers crossed) and also its a fair bit cleaner too.
I'm adding the linux-btrace list, since I didn't know about that list when I made the initial posting.
Since there is probably more GFS2 changes than blktrace changes, I could push this through the GFS2 tree. Let me know if you'd prefer it to go via the blktrace tree. I'd like to be able to push this in at the next merge window if possible. This patch is against the head of the GFS2 -nmw git tree (obviously that makes no difference to the blktrace side of the patch).
An updated blktrace userland patch follows in the next email, although the changes from the last version are fairly minor,
Steve.
From 68ebfea9d79d9b5123e17a3345b7ebe39c8575f9 Mon Sep 17 00:00:00 2001
From: Steven Whitehouse <[email protected]>
Date: Thu, 19 Feb 2009 15:38:29 +0000
Subject: [PATCH] GFS2: Add blktrace support to glocks
Glocks are the cache control subsystem for GFS2. It is very useful to be able to trace the state changes of the glocks using the same set of sequence numbers as the I/O requests, since this allows us to see if there are any ordering errors.
The interface uses DLM lock modes so that hopefully it is generic enough that other cluster filesystems might be able to use the same interface.
Glocks are identified by two numbers: the type number and the glock number. The latter is (for most glocks) based upon the disk block number of the object (inode, resource group, etc) which it protects and for the other glocks, its a small integer. I think it makes sense to use the existing sector field for this. The type number is reported as part of the struct blk_trace_io_glock. I've also added a flags field to that structure (currently unused) in case of future need.
block/blktrace.c +60 -5
@@ -38,8 +38,8 @@ static void blk_unregister_tracepoints(void);/* * Send out a notify message. */ -static void trace_note (struct blk_trace *bt,pid_t pid,int action, -const void *data,size_t len) +static void trace_note (struct blk_trace *bt,sector_t sector,pid_t pid, +int action,const void *data,size_t len,int seq) {struct blk_io_trace *t;
@@ -48,11 +48,17 @@ static void trace_note(struct blk_trace *bt, pid_t pid, int action,const int cpu =smp_processor_id (); t->magic =BLK_IO_TRACE_MAGIC |BLK_IO_TRACE_VERSION ; +if (seq) { +unsigned long *sequence =per_cpu_ptr (bt->sequence, cpu); + t->sequence = ++(*sequence); + } t->time =ktime_to_ns (ktime_get ()); + t->sector = sector; t->device = bt->dev; t->action = action; t->pid = pid; t->cpu = cpu; + t->error =0 ; t->pdu_len = len;memcpy ((void *) t +sizeof (*t), data, len); }
@@ -65,7 +71,7 @@ static void trace_note(struct blk_trace *bt, pid_t pid, int action,static void trace_note_tsk (struct blk_trace *bt,struct task_struct *tsk) { tsk->btrace_seq = blktrace_seq; -trace_note (bt, tsk->pid,BLK_TN_PROCESS , tsk->comm,sizeof (tsk->comm)); +trace_note (bt,0 , tsk->pid,BLK_TN_PROCESS , tsk->comm,sizeof (tsk->comm),0 ); }static void trace_note_time (struct blk_trace *bt)
@@ -79,7 +85,7 @@ static void trace_note_time(struct blk_trace *bt) words[1 ] = now.tv_nsec;local_irq_save (flags); -trace_note (bt,0 ,BLK_TN_TIMESTAMP , words,sizeof (words)); +trace_note (bt,0 ,0 ,BLK_TN_TIMESTAMP , words,sizeof (words),0 );local_irq_restore (flags); }
@@ -96,7 +102,7 @@ void __trace_note_message(struct blk_trace *bt, const char *fmt, ...) n =vscnprintf (buf,BLK_TN_MAX_MSG , fmt, args);va_end (args); -trace_note (bt,0 ,BLK_TN_MESSAGE , buf, n); +trace_note (bt,0 ,0 ,BLK_TN_MESSAGE , buf, n,0 );local_irq_restore (flags); }EXPORT_SYMBOL_GPL (__trace_note_message);
@@ -797,6 +803,55 @@ static void blk_add_trace_remap(struct request_queue *q, struct bio *bio, }/** +* blk_add_trace_glock - Add a trace message about a glock +* @bdev: The block device in question +* @gltype: The type of the glock +* @glnum: The glock number +* @cur_state: The current state of the glock (always sent) +* @new_state: If the state has changed, this is the new state (maybe 0) +* @tgt_state: The target lock state (maybe 0) +* @dmt_state: The current requested demote state (maybe 0) +* +* The states are DLM lock states in order to make this a generic +* interface, even though its initial use is restricted to glocks. +* +*/ + +void blk_add_trace_glock (struct block_device *bdev,u32 gltype,sector_t glnum, +u8 cur_state,u8 new_state,u8 tgt_state,u8 dmt_state) +{ +struct task_struct *tsk = current; +struct request_queue *rq =bdev_get_queue (bdev); +struct blk_trace *bt = rq->blk_trace; +struct blk_io_trace_glock g; +unsigned long flags; +pid_t pid; + +if (likely (!bt)) +return ; +if (unlikely (bt->trace_state != Blktrace_running)) +return ; + + pid = tsk->pid; +if (unlikely (act_log_check (bt,BLK_TN_GLOCK , glnum, pid))) +return ; + + g.type =cpu_to_be32 (gltype); + g.flags =0 ; + g.cur_state = cur_state; + g.new_state = new_state; + g.dmt_state = dmt_state; + g.tgt_state = tgt_state; + +local_irq_save (flags); +if (unlikely (tsk->btrace_seq != blktrace_seq)) +trace_note_tsk (bt, tsk); +trace_note (bt, glnum, pid,BLK_TN_GLOCK , &g,sizeof (g),1 ); +local_irq_restore (flags); +}; +EXPORT_SYMBOL_GPL (blk_add_trace_glock); + +/** * blk_add_driver_data - Add binary message with driver-specific data * @q: queue the io is for * @rq: io request
fs/gfs2/glock.c +100 -1
153 lines not on this page · raw message
fs/gfs2/incore.h +1 -0
@@ -189,6 +189,7 @@ enum {GLF_REPLY_PENDING =9 ,GLF_INITIAL =10 ,GLF_FROZEN =11 , + GLF_TRACE_INITIAL = 12, }; struct gfs2_glock {
include/linux/blktrace_api.h +29 -2
@@ -62,6 +62,7 @@ enum blktrace_notify { __BLK_TN_PROCESS =0 ,/* establish pid/name mapping */ __BLK_TN_TIMESTAMP,/* include system clock */ __BLK_TN_MESSAGE,/* Character string message */ + __BLK_TN_GLOCK, /* Glock data */ };
@@ -89,6 +90,7 @@ enum blktrace_notify {#define BLK_TN_PROCESS (__BLK_TN_PROCESS | BLK_TC_ACT(BLK_TC_NOTIFY))#define BLK_TN_TIMESTAMP (__BLK_TN_TIMESTAMP | BLK_TC_ACT(BLK_TC_NOTIFY))#define BLK_TN_MESSAGE (__BLK_TN_MESSAGE | BLK_TC_ACT(BLK_TC_NOTIFY)) +#define BLK_TN_GLOCK (__BLK_TN_GLOCK | BLK_TC_ACT(BLK_TC_NOTIFY))#define BLK_IO_TRACE_MAGIC 0x65617400#define BLK_IO_TRACE_VERSION 0x07
@@ -119,6 +121,29 @@ struct blk_io_trace_remap {__be64 sector; }; +/* Glock lock states, so we don't need to add any header deps */ +enum { +BLK_GLS_NONE =1 ,/* i.e. invalid */ +BLK_GLS_NULL ,/* Null lock (preserves LVB content) */ +BLK_GLS_CREAD ,/* Concurrent read */ +BLK_GLS_CWRITE ,/* Concurrent write */ +BLK_GLS_PREAD ,/* Protected read */ +BLK_GLS_PWRITE ,/* Protected write */ +BLK_GLS_EXCLUSIVE ,/* Exclusive */ +}; + +/* +* Glock info +*/ +struct blk_io_trace_glock { +__be32 type;/* Glock type, as per gl_name.ln_type */ +__be32 flags;/* Flags, currently unused */ +u8 cur_state;/* Current state */ +u8 new_state;/* New state */ +u8 dmt_state;/* Requested demote state */ +u8 tgt_state;/* Target state */ +}; +enum { Blktrace_setup =1 , Blktrace_running,
@@ -191,7 +216,9 @@ extern int blk_trace_setup(struct request_queue *q, char *name, dev_t dev, char __user *arg);extern int blk_trace_startstop (struct request_queue *q,int start);extern int blk_trace_remove (struct request_queue *q); - +extern void blk_add_trace_glock (struct block_device *bdev,u32 gltype, +sector_t glnum,u8 cur_state,u8 new_state, +u8 tgt_state,u8 dmt_state);#else /* !CONFIG_BLK_DEV_IO_TRACE */ #define blk_trace_ioctl (bdev, cmd, arg) (-ENOTTY)#define blk_trace_shutdown (q) do { } while (0)
@@ -201,7 +228,7 @@ extern int blk_trace_remove(struct request_queue *q);#define blk_trace_startstop (q, start) (-ENOTTY)#define blk_trace_remove (q) (-ENOTTY)#define blk_add_trace_msg (q, fmt, ...) do { } while (0) - +#define blk_add_trace_glock (bdev, gltype, glnum, c, n, t, d) do { } while (0)#endif /* CONFIG_BLK_DEV_IO_TRACE */ #endif /* __KERNEL__ */ #endif
include/trace/gfs2.h new file +20 -0
@@ -0,0 +1,20 @@ +#ifndef _TRACE_GFS2_H +#define _TRACE_GFS2_H + +#include <linux/tracepoint.h> + +struct gfs2_glock ; + +DECLARE_TRACE (gfs2_glock_state_change, + TPPROTO(struct gfs2_glock *gl, unsigned int new_state), +TPARGS (gl, new_state)); + +DECLARE_TRACE (gfs2_glock_put, + TPPROTO(struct gfs2_glock *gl), +TPARGS (gl)); + +DECLARE_TRACE (gfs2_demote_rq, + TPPROTO(struct gfs2_glock *gl), +TPARGS (gl)); + +#endif
Signature · 1 line
1.6.0.3
This patch is against the head of the blktrace git tree. It adds support for the display of glock messages from blktrace.
blkparse.c +47 -3
@@ -570,6 +570,25 @@ static struct process_pid_map *add_ppm_hash(pid_t pid, const char *name)return ppm; } +/* Matches the enum in linux/blktrace_api.h */ +const char *glock_states[] = { +"??" , +"IV" , +"NL" , +"CR" , +"CW" , +"PR" , +"PW" , +"EX" +}; + +static const char *glstate2str (u8 state) +{ +if (state > (sizeof (glock_states)/sizeof (const char *))) +return glock_states[0 ]; +return glock_states[state]; +} +static void handle_notify (struct blk_io_trace *bit) {void *payload = (caddr_t ) bit +sizeof (*bit);
@@ -614,6 +633,27 @@ static void handle_notify(struct blk_io_trace *bit) }break ; +case BLK_TN_GLOCK : +if (bit->pdu_len ==sizeof (struct blk_io_trace_glock )) { +struct blk_io_trace_glock *g = (struct blk_io_trace_glock *)payload; +fprintf (ofp, +"%3d,%-3d %2d %8lu %5d.%09lu %5u %2s %3s %u:%llu cur:%s" , +MAJOR (bit->device),MINOR (bit->device), + bit->cpu, (unsigned long )bit->sequence, + (int )SECONDS (bit->time), + (unsigned long )NANO_SECONDS (bit->time), + bit->pid,"m" ,"G" ,be32_to_cpu (g->type), + (unsigned long long )bit->sector, +glstate2str (g->cur_state)); +if (g->new_state) +fprintf (ofp,",new:%s" ,glstate2str (g->new_state)); +if (g->tgt_state) +fprintf (ofp,",tgt:%s" ,glstate2str (g->tgt_state)); +if (g->dmt_state) +fprintf (ofp,",dmt:%s" ,glstate2str (g->dmt_state)); +fprintf (ofp," [%s]\n" ,find_process_name (bit->pid)); + + }default :/* Ignore unknown notify events */ ;
@@ -1605,7 +1645,7 @@ static void dump_trace(struct blk_io_trace *t, struct per_cpu_info *pci, struct per_dev_info *pdi) {if (text_output) { -if (t->action ==BLK_TN_MESSAGE ) +if (t->action ==BLK_TN_MESSAGE || t->action ==BLK_TN_GLOCK )handle_notify (t);else if (t->action &BLK_TC_ACT (BLK_TC_PC ))dump_trace_pc (t, pdi, pci);
@@ -2209,7 +2249,9 @@ static int read_events(int fd, int always_block, int *fdblock)/* * not a real trace, so grab and handle it here */ -if (bit->action &BLK_TC_ACT (BLK_TC_NOTIFY ) && bit->action !=BLK_TN_MESSAGE ) { +if (bit->action &BLK_TC_ACT (BLK_TC_NOTIFY ) && + bit->action !=BLK_TN_MESSAGE && + bit->action !=BLK_TN_GLOCK ) {handle_notify (bit);output_binary (bit,sizeof (*bit) + bit->pdu_len);continue ;
@@ -2352,7 +2394,9 @@ static int ms_prime(struct ms_stream *msp)if (verify_trace (bit)) goto err; -if (bit->action &BLK_TC_ACT (BLK_TC_NOTIFY ) && bit->action !=BLK_TN_MESSAGE ) { +if (bit->action &BLK_TC_ACT (BLK_TC_NOTIFY ) && + bit->action !=BLK_TN_MESSAGE && + bit->action !=BLK_TN_GLOCK ) {handle_notify (bit);output_binary (bit,sizeof (*bit) + bit->pdu_len);bit_free (bit);
blktrace_api.h +25 -0
@@ -59,6 +59,7 @@ enum blktrace_notify { __BLK_TN_PROCESS =0 ,/* establish pid/name mapping */ __BLK_TN_TIMESTAMP,/* include system clock */ __BLK_TN_MESSAGE,/* Character string message */ + __BLK_TN_GLOCK, /* Glock data */ };/*
@@ -85,6 +86,7 @@ enum blktrace_notify {#define BLK_TN_PROCESS (__BLK_TN_PROCESS | BLK_TC_ACT(BLK_TC_NOTIFY))#define BLK_TN_TIMESTAMP (__BLK_TN_TIMESTAMP | BLK_TC_ACT(BLK_TC_NOTIFY))#define BLK_TN_MESSAGE (__BLK_TN_MESSAGE | BLK_TC_ACT(BLK_TC_NOTIFY)) +#define BLK_TN_GLOCK (__BLK_TN_GLOCK | BLK_TC_ACT(BLK_TC_NOTIFY))#define BLK_IO_TRACE_MAGIC 0x65617400#define BLK_IO_TRACE_VERSION 0x07
@@ -115,6 +117,29 @@ struct blk_io_trace_remap {__u64 sector; }; +/* Glock lock states, so we don't need to add any header deps */ +enum { +BLK_GLS_NONE =1 ,/* i.e. invalid */ +BLK_GLS_NULL ,/* Null lock (preserves LVB content) */ +BLK_GLS_CREAD ,/* Concurrent read */ +BLK_GLS_CWRITE ,/* Concurrent write */ +BLK_GLS_PREAD ,/* Protected read */ +BLK_GLS_PWRITE ,/* Protected write */ +BLK_GLS_EXCLUSIVE ,/* Exclusive */ +}; + +/* +* Glock info +*/ +struct blk_io_trace_glock { +__u32 type;/* Glock type, as per gl_name.ln_type */ +__u32 flags;/* Unused at the moment */ +__u8 cur_state;/* Current state */ +__u8 new_state;/* New state */ +__u8 dmt_state;/* Requested demote state */ +__u8 tgt_state;/* Target state */ +}; +/* * User setup structure passed with BLKSTARTTRACE */
Quoted 17 lines · Steven Whitehouse
Hi,
Since I last posted this pair of patches, I've done some extensive updating of the kernel patch, so it should now be happy to compile under all possible Kconfigs (fingers crossed) and also its a fair bit cleaner too.
I'm adding the linux-btrace list, since I didn't know about that list when I made the initial posting.
Since there is probably more GFS2 changes than blktrace changes, I could push this through the GFS2 tree. Let me know if you'd prefer it to go via the blktrace tree. I'd like to be able to push this in at the next merge window if possible. This patch is against the head of the GFS2 -nmw git tree (obviously that makes no difference to the blktrace side of the patch).
Quoted 2 lines
An updated blktrace userland patch follows in the next email, although the changes from the last version are fairly minor,
Umm, why would you put fs internal stuff into blktrace? Just use the generic ringbuffer directly and trace into that one.
Quoted 18 lines · Christoph Hellwig
Quoted 17 lines · Steven Whitehouse
Hi,
Since I last posted this pair of patches, I've done some extensive updating of the kernel patch, so it should now be happy to compile under all possible Kconfigs (fingers crossed) and also its a fair bit cleaner too.
I'm adding the linux-btrace list, since I didn't know about that list when I made the initial posting.
Since there is probably more GFS2 changes than blktrace changes, I could push this through the GFS2 tree. Let me know if you'd prefer it to go via the blktrace tree. I'd like to be able to push this in at the next merge window if possible. This patch is against the head of the GFS2 -nmw git tree (obviously that makes no difference to the blktrace side of the patch).
Quoted 5 lines
Quoted 2 lines
An updated blktrace userland patch follows in the next email, although the changes from the last version are fairly minor,
Umm, why would you put fs internal stuff into blktrace? Just use the generic ringbuffer directly and trace into that one.
Agree, some months ago this would have made more sense, since it was easy to just piggy back on top of blktrace. But now tracing is easily available, so going that route makes more sense now. Especially since this lock tracking doesn't need much of what blkparse offers anyway.
Signature · 1 line
Jens Axboe
Hi,
Quoted 18 lines · Christoph Hellwig
Quoted 17 lines · Steven Whitehouse
Hi,
Since I last posted this pair of patches, I've done some extensive updating of the kernel patch, so it should now be happy to compile under all possible Kconfigs (fingers crossed) and also its a fair bit cleaner too.
I'm adding the linux-btrace list, since I didn't know about that list when I made the initial posting.
Since there is probably more GFS2 changes than blktrace changes, I could push this through the GFS2 tree. Let me know if you'd prefer it to go via the blktrace tree. I'd like to be able to push this in at the next merge window if possible. This patch is against the head of the GFS2 -nmw git tree (obviously that makes no difference to the blktrace side of the patch).
Quoted 6 lines
Quoted 2 lines
An updated blktrace userland patch follows in the next email, although the changes from the last version are fairly minor,
Umm, why would you put fs internal stuff into blktrace? Just use the generic ringbuffer directly and trace into that one.
From the patch description:
Glocks are the cache control subsystem for GFS2. It is very useful to be able to trace the state changes of the glocks using the same set of sequence numbers as the I/O requests, since this allows us to see if there are any ordering errors.
Maybe there is a way to keep the sequencing between I/O and glocks some other way, but it looked to me like I would lose that if I were to have a separate logging system for glocks. So really what I'm saying is that logging glocks on their own isn't a lot of use, its only useful when combined with I/O logging,
Steve.