high resolution timers, scheduling & sleep granularity
Hi Thomas & Ingo,
Josef has been working on some patches to try and get ext3/4 to dynamically detect the latency of a storage device and use that base latency to tune the amount of time we sleep waiting for others to join in a transaction. The logic in question lives in jbd/transaction.c (transaction_stop).
The code was originally developed to try and allow multiple threads to join in a big, slow transaction. For example, transacations that write to a slow ATA or S-ATA drive take in the neighborhood of 10 to 20 ms.
Faster devices, for example a disk array, can complete the transaction in 1.3 ms. Even higher speed SSD devices boast of a latency of 0.1ms, not to mention RAM disks ;-)
The current logic makes us wait way too long, especially with a 250HZ kernel since we sleep many times longer than it takes to complete the IO ;-)
Do either of you have any thoughts on how to get a better, fine grained sleep capability that we could use that would allow us to sleep in sub-jiffie chunks?
Regards,
Ric
Quoted 16 lines · Ric Wheeler
Hi Thomas & Ingo,
Josef has been working on some patches to try and get ext3/4 to dynamically detect the latency of a storage device and use that base latency to tune the amount of time we sleep waiting for others to join in a transaction. The logic in question lives in
jbd/transaction.c(transaction_stop).The code was originally developed to try and allow multiple threads to join in a big, slow transaction. For example, transacations that write to a slow ATA or S-ATA drive take in the neighborhood of 10 to 20 ms.
Faster devices, for example a disk array, can complete the transaction in 1.3 ms. Even higher speed SSD devices boast of a latency of 0.1ms, not to mention RAM disks ;-)
Quoted 8 lines
The current logic makes us wait way too long, especially with a 250HZ kernel since we sleep many times longer than it takes to complete the IO ;-)
Do either of you have any thoughts on how to get a better, fine grained sleep capability that we could use that would allow us to sleep in sub-jiffie chunks?
Hello,
This is the most recent iteration of my patch using hrtimers. It works really well for ramdisks, so anything with low latency writes is going to be really fast, but I'm still trying to come up with a smart way to sleep long enough to not hurt SATA performance. As it stands now I'm getting a 5% decrease in speed on SATA. So I think I've got the sleep as little as possible part down right, just can't quite get it to sleep long enough if the disk is slow. Thanks,
Josef
fs/jbd/commit.c +15 -0
@@ -288,6 +288,7 @@ void journal_commit_transaction(journal_int flags;int err;unsigned long blocknr; +unsigned long long commit_time, start_time;char *tagp =NULL ;journal_header_t *header;journal_block_tag_t *tag =NULL ;
@@ -400,6 +401,7 @@ void journal_commit_transaction(journal_ commit_transaction->t_state =T_FLUSH ; journal->j_committing_transaction = commit_transaction; journal->j_running_transaction =NULL ; + start_time = jiffies; commit_transaction->t_log_start = journal->j_head;wake_up (&journal->j_wait_transaction_locked);spin_unlock (&journal->j_state_lock);
@@ -873,6 +875,19 @@ restart_loop:J_ASSERT (commit_transaction == journal->j_committing_transaction); journal->j_commit_sequence = commit_transaction->t_tid; journal->j_committing_transaction =NULL ; + commit_time =elapsed_jiffies (start_time, jiffies); +if (unlikely (!journal->j_average_commit_time)) + journal->j_average_commit_time = commit_time; +else + journal->j_average_commit_time = (commit_time + + journal->j_average_commit_time) /2 ; + +if (journal->print_count <100 ) { + journal->print_count++; +printk (KERN_ERR "avg commit time = %lu\n" , + journal->j_average_commit_time); + } +spin_unlock (&journal->j_state_lock); if (commit_transaction->t_checkpoint_list == NULL &&
fs/jbd/transaction.c +53 -5
89 lines not on this page · raw message
include/linux/jbd.h +19 -0
@@ -31,6 +31,7 @@#include <linux/mutex.h> #include <linux/timer.h> #include <linux/lockdep.h> +#include <linux/jiffies.h> #define journal_oom_retry 1
@@ -79,6 +80,15 @@ static inline void jbd_free(void *ptr, sfree_pages ((unsigned long )ptr,get_order (size)); }; +static inline unsigned long elapsed_jiffies (unsigned long start, +unsigned long end) +{ +if (end >= start) +return end - start; + +return end + (MAX_JIFFY_OFFSET - start) +1 ; +} +#define JFS_MIN_JOURNAL_BLOCKS 1024
@@ -543,6 +553,11 @@ struct transaction_sunsigned long t_expires;/* +* When this transaction started, in jiffies [no locking] +*/ +unsigned long t_start_time; + +/* * How many handles used this transaction? [t_handle_lock] */ int t_handle_count;
@@ -800,6 +815,10 @@ struct journal_spid_t j_last_sync_writer; +unsigned long j_average_commit_time; + +int print_count; +/* * An opaque pointer to fs-private information. ext3 puts its * superblock pointer here
kernel/hrtimer.c +1 -0
@@ -1458,6 +1458,7 @@ void hrtimer_init_sleeper(struct hrtimer sl->timer.cb_mode =HRTIMER_CB_IRQSAFE_NO_SOFTIRQ ;#endif } +EXPORT_SYMBOL_GPL (hrtimer_init_sleeper); static int __sched do_nanosleep(struct hrtimer_sleeper *t, enum hrtimer_mode mode) {
Quoted 18 lines · Josef Bacik
Quoted 4 lines · Ric Wheeler
Hi Thomas & Ingo,
Josef has been working on some patches to try and get ext3/4 to dynamically detect the latency of a storage device and use that base latency to tune the amount of time we sleep waiting for others to join in a transaction. The logic in question lives in
jbd/transaction.c(transaction_stop).Quoted 9 lines
The code was originally developed to try and allow multiple threads to join in a big, slow transaction. For example, transacations that write to a slow ATA or S-ATA drive take in the neighborhood of 10 to 20 ms.
Faster devices, for example a disk array, can complete the transaction in 1.3 ms. Even higher speed SSD devices boast of a latency of 0.1ms, not to mention RAM disks ;-)
The current logic makes us wait way too long, especially with a 250HZ kernel since we sleep many times longer than it takes to complete the IO ;-)
Do either of you have any thoughts on how to get a better, fine grained sleep capability that we could use that would allow us to sleep in sub-jiffie chunks?
Hello,
Quoted 9 lines
This is the most recent iteration of my patch using hrtimers. It works really well for ramdisks, so anything with low latency writes is going to be really fast, but I'm still trying to come up with a smart way to sleep long enough to not hurt SATA performance. As it stands now I'm getting a 5% decrease in speed on SATA. So I think I've got the sleep as little as possible part down right, just can't quite get it to sleep long enough if the disk is slow. Thanks,
Josef
I think that this (or similar) kind of precision_sleep() should be generically useful.
One question on the code, would it be better to measure the average transaction time in the same units as your precision sleep uses - aren't jiffies are still too coarse?
It also might be interesting to export the measured average commit time via some non-printk method (/sys/fs ??)
ric
Quoted 208 lines
Index:
linux-2.6/fs/jbd/commit.c
===================================================================fs/jbd/commit.c +15
-0@@ -288,6 +288,7 @@ void journal_commit_transaction(journal_int flags;int err;unsigned long blocknr; +unsigned long long commit_time, start_time;char *tagp =NULL ;journal_header_t *header;journal_block_tag_t *tag =NULL ;@@ -400,6 +401,7 @@ void journal_commit_transaction(journal_ commit_transaction->t_state =T_FLUSH ; journal->j_committing_transaction = commit_transaction; journal->j_running_transaction =NULL ; + start_time = jiffies; commit_transaction->t_log_start = journal->j_head;wake_up (&journal->j_wait_transaction_locked);spin_unlock (&journal->j_state_lock);@@ -873,6 +875,19 @@ restart_loop:J_ASSERT (commit_transaction == journal->j_committing_transaction); journal->j_commit_sequence = commit_transaction->t_tid; journal->j_committing_transaction =NULL ; + commit_time =elapsed_jiffies (start_time, jiffies); +if (unlikely (!journal->j_average_commit_time)) + journal->j_average_commit_time = commit_time; +else + journal->j_average_commit_time = (commit_time + + journal->j_average_commit_time) /2 ; + +if (journal->print_count <100 ) { + journal->print_count++; +printk (KERN_ERR "avg commit time = %lu\n" , + journal->j_average_commit_time); + } +spin_unlock (&journal->j_state_lock); if (commit_transaction->t_checkpoint_list == NULL &&Index:
linux-2.6/fs/jbd/transaction.c
===================================================================fs/jbd/transaction.c +53
-5@@ -25,6 +25,7 @@#include <linux/timer.h> #include <linux/mm.h> #include <linux/highmem.h> +#include <linux/hrtimer.h> static void __journal_temp_unlink_buffer (struct journal_head *jh);@@ -49,6 +50,7 @@ get_transaction(journal_t *journal, tran { transaction->t_journal = journal; transaction->t_state =T_RUNNING ; + transaction->t_start_time = jiffies; transaction->t_tid = journal->j_transaction_sequence++; transaction->t_expires = jiffies + journal->j_commit_interval;spin_lock_init (&transaction->t_handle_lock);@@ -63,6 +65,32 @@ get_transaction(journal_t *journal, tranreturn transaction; } +static void precision_sleep (unsigned long time) +{ +struct hrtimer_sleeper t; + +hrtimer_init_on_stack (&t.timer,CLOCK_REALTIME ,HRTIMER_MODE_ABS ); +hrtimer_init_sleeper (&t, current); + t.timer.expires =ktime_add_ns (ktime_get_real (), time); + +do { +set_current_state (TASK_UNINTERRUPTIBLE ); + +hrtimer_start (&t.timer, t.timer.expires,HRTIMER_MODE_ABS ); +if (!hrtimer_active (&t.timer)) + t.task =NULL ; + +if (likely (t.task)) +schedule (); + +hrtimer_cancel (&t.timer); + }while (t.task); + +set_current_state (TASK_RUNNING ); + +destroy_hrtimer_on_stack (&t.timer); +} +/* * Handle management. * @@ -1361,7 +1389,7 @@ int journal_stop(handle_t *handle) {transaction_t *transaction = handle->h_transaction;journal_t *journal = transaction->t_journal; -int old_handle_count, err; +int err;pid_t pid;J_ASSERT (journal_current_handle () == handle);@@ -1397,11 +1425,31 @@ int journal_stop(handle_t *handle)*/ pid = current->pid;if (handle->h_sync && journal->j_last_sync_writer != pid) { +unsigned long commit_time; +int print_count; +unsigned long sleep_time; + journal->j_last_sync_writer = pid; -do { - old_handle_count = transaction->t_handle_count; -schedule_timeout_uninterruptible (1 ); - }while (old_handle_count != transaction->t_handle_count); + +spin_lock (&journal->j_state_lock); + commit_time = journal->j_average_commit_time; +spin_unlock (&journal->j_state_lock); + + sleep_time =elapsed_jiffies (transaction->t_start_time, jiffies); +if (!sleep_time) + sleep_time =1 ; + sleep_time = (commit_time / sleep_time) * commit_time; + +/* +if (print_count >= 100 && print_count < 200) { +journal->print_count++; +printk(KERN_ERR "sleep_time=%lu\n", (unsigned long)sleep_time); +} + +sleep_time = (long)(commit_time - sleep_time); +*/ + +precision_sleep ((unsigned long )sleep_time); } current->journal_info =NULL ;Index:
linux-2.6/include/linux/jbd.h
===================================================================include/linux/jbd.h +19
-0@@ -31,6 +31,7 @@#include <linux/mutex.h> #include <linux/timer.h> #include <linux/lockdep.h> +#include <linux/jiffies.h> #define journal_oom_retry 1@@ -79,6 +80,15 @@ static inline void jbd_free(void *ptr, sfree_pages ((unsigned long )ptr,get_order (size)); }; +static inline unsigned long elapsed_jiffies (unsigned long start, +unsigned long end) +{ +if (end >= start) +return end - start; + +return end + (MAX_JIFFY_OFFSET - start) +1 ; +} +#define JFS_MIN_JOURNAL_BLOCKS 1024@@ -543,6 +553,11 @@ struct transaction_sunsigned long t_expires;/* +* When this transaction started, in jiffies [no locking] +*/ +unsigned long t_start_time; + +/* * How many handles used this transaction? [t_handle_lock] */ int t_handle_count;@@ -800,6 +815,10 @@ struct journal_spid_t j_last_sync_writer; +unsigned long j_average_commit_time; + +int print_count; +/* * An opaque pointer to fs-private information. ext3 puts its * superblock pointer here Index:
linux-2.6/kernel/hrtimer.c
===================================================================kernel/hrtimer.c +1
-1@@ -1458,6 +1458,7 @@ void hrtimer_init_sleeper(struct hrtimer sl->timer.cb_mode =HRTIMER_CB_IRQSAFE_NO_SOFTIRQ ;#endif } +EXPORT_SYMBOL_GPL (hrtimer_init_sleeper); static int __sched do_nanosleep(struct hrtimer_sleeper *t, enum hrtimer_mode mode) { --To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 41 lines · Ric Wheeler
Quoted 30 lines · Josef Bacik
Quoted 16 lines · Ric Wheeler
Hi Thomas & Ingo,
Josef has been working on some patches to try and get ext3/4 to dynamically detect the latency of a storage device and use that base latency to tune the amount of time we sleep waiting for others to join in a transaction. The logic in question lives in
jbd/transaction.c(transaction_stop).The code was originally developed to try and allow multiple threads to join in a big, slow transaction. For example, transacations that write to a slow ATA or S-ATA drive take in the neighborhood of 10 to 20 ms.
Faster devices, for example a disk array, can complete the transaction in 1.3 ms. Even higher speed SSD devices boast of a latency of 0.1ms, not to mention RAM disks ;-)
Quoted 9 lines
The current logic makes us wait way too long, especially with a 250HZ kernel since we sleep many times longer than it takes to complete the IO ;-)
Do either of you have any thoughts on how to get a better, fine grained sleep capability that we could use that would allow us to sleep in sub-jiffie chunks?
Hello,
Quoted 9 lines
This is the most recent iteration of my patch using hrtimers. It works really well for ramdisks, so anything with low latency writes is going to be really fast, but I'm still trying to come up with a smart way to sleep long enough to not hurt SATA performance. As it stands now I'm getting a 5% decrease in speed on SATA. So I think I've got the sleep as little as possible part down right, just can't quite get it to sleep long enough if the disk is slow. Thanks,
Josef
Quoted 7 lines
I think that this (or similar) kind of
precision_sleep()should be generically useful.One question on the code, would it be better to measure the average transaction time in the same units as your precision sleep uses - aren't jiffies are still too coarse?
Oh well crap I guess thats why I'm not sleeping long enough, i'm only sleeping jiffies number of nanoseconds, and jiffies is much higher than nanoseconds...
/me writes jiffies * HZ = seconds backwards on his forehead
Josef
Quoted 28 lines · Josef Bacik
Index:
linux-2.6/fs/jbd/transaction.c
===================================================================fs/jbd/transaction.c +19
-0@@ -63,6 +65,32 @@ get_transaction(journal_t *journal, tranreturn transaction; } +static void precision_sleep (unsigned long time) +{ +struct hrtimer_sleeper t; + +hrtimer_init_on_stack (&t.timer,CLOCK_REALTIME ,HRTIMER_MODE_ABS ); +hrtimer_init_sleeper (&t, current); + t.timer.expires =ktime_add_ns (ktime_get_real (), time); + + do { + set_current_state(TASK_UNINTERRUPTIBLE); + + hrtimer_start(&t.timer, t.timer.expires, HRTIMER_MODE_ABS); + if (!hrtimer_active(&t.timer)) + t.task = NULL; + + if (likely(t.task)) + schedule(); + + hrtimer_cancel(&t.timer);
fs/jbd/transaction.c get_transaction(journal_t *journal, tran · line 96
Quoted 10 lines
fs/jbd/transaction.c +7
-0+ }while (t.task); + +set_current_state (TASK_RUNNING ); + +destroy_hrtimer_on_stack (&t.timer); +} +/* * Handle management. *
I was convinced we already had such a creature,. but I guess I was wrong as I can't find it ;-)
Anyway, I'm thinking this function ought to live in kernel/hrtimer.c and possibly get renamed to something like hrtimer_sleep_ns() or some such (means you can also reuse the do_nanosleep helper in there).
Also, have you considered the impact on platforms that do not support hrtimers, or don't have high resolution clock events available?
kernel/hrtimer.c void hrtimer_init_sleeper(struct hrtimer · line 1464
Quoted 12 lines
Index:
linux-2.6/kernel/hrtimer.c
===================================================================kernel/hrtimer.c +1
-0@@ -1458,6 +1458,7 @@ void hrtimer_init_sleeper(struct hrtimer sl->timer.cb_mode =HRTIMER_CB_IRQSAFE_NO_SOFTIRQ ;#endif } +EXPORT_SYMBOL_GPL (hrtimer_init_sleeper); static int __sched do_nanosleep(struct hrtimer_sleeper *t, enum hrtimer_mode mode) {
That also gets rid of this export..
Quoted 44 lines · Peter Zijlstra
Quoted 28 lines · Josef Bacik
Index:
linux-2.6/fs/jbd/transaction.c
===================================================================fs/jbd/transaction.c +19
-0@@ -63,6 +65,32 @@ get_transaction(journal_t *journal, tranreturn transaction; } +static void precision_sleep (unsigned long time) +{ +struct hrtimer_sleeper t; + +hrtimer_init_on_stack (&t.timer,CLOCK_REALTIME ,HRTIMER_MODE_ABS ); +hrtimer_init_sleeper (&t, current); + t.timer.expires =ktime_add_ns (ktime_get_real (), time); + + do { + set_current_state(TASK_UNINTERRUPTIBLE); + + hrtimer_start(&t.timer, t.timer.expires, HRTIMER_MODE_ABS); + if (!hrtimer_active(&t.timer)) + t.task = NULL; + + if (likely(t.task)) + schedule(); + + hrtimer_cancel(&t.timer);fs/jbd/transaction.c get_transaction(journal_t *journal, tran · line 96
Quoted 10 lines
fs/jbd/transaction.c +7
-0+ }while (t.task); + +set_current_state (TASK_RUNNING ); + +destroy_hrtimer_on_stack (&t.timer); +} +/* * Handle management. * I was convinced we already had such a creature,. but I guess I was wrong as I can't find it ;-)
Quoted 7 lines
Anyway, I'm thinking this function ought to live in
kernel/hrtimer.cand possibly get renamed to something likehrtimer_sleep_ns()or some such (means you can also reuse thedo_nanosleephelper in there).Also, have you considered the impact on platforms that do not support hrtimers, or don't have high resolution clock events available?
Not yet :), this is just a proof of concept atm, eventually if I do use hrtimers I would just fallback to using jiffies if the arch doesn't support hrtimers as it still gets significantly better performance than what we currently have. Thanks,
Josef
Quoted 19 lines
kernel/hrtimer.c void hrtimer_init_sleeper(struct hrtimer · line 1464
Quoted 12 lines
Index:
linux-2.6/kernel/hrtimer.c
===================================================================kernel/hrtimer.c +1
-0@@ -1458,6 +1458,7 @@ void hrtimer_init_sleeper(struct hrtimer sl->timer.cb_mode =HRTIMER_CB_IRQSAFE_NO_SOFTIRQ ;#endif } +EXPORT_SYMBOL_GPL (hrtimer_init_sleeper); static int __sched do_nanosleep(struct hrtimer_sleeper *t, enum hrtimer_mode mode) {That also gets rid of this export..
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 48 lines · Peter Zijlstra
Quoted 31 lines · Josef Bacik
Index:
linux-2.6/fs/jbd/transaction.c
===================================================================fs/jbd/transaction.c +19
-0@@ -63,6 +65,32 @@ get_transaction(journal_t *journal, tranreturn transaction; } +static void precision_sleep (unsigned long time) +{ +struct hrtimer_sleeper t; + +hrtimer_init_on_stack (&t.timer,CLOCK_REALTIME ,HRTIMER_MODE_ABS ); +hrtimer_init_sleeper (&t, current); + t.timer.expires =ktime_add_ns (ktime_get_real (), time); + + do { + set_current_state(TASK_UNINTERRUPTIBLE); + + hrtimer_start(&t.timer, t.timer.expires, HRTIMER_MODE_ABS); + if (!hrtimer_active(&t.timer)) + t.task = NULL; + + if (likely(t.task)) + schedule(); + + hrtimer_cancel(&t.timer);fs/jbd/transaction.c get_transaction(journal_t *journal, tran · line 96
Quoted 10 lines
fs/jbd/transaction.c +7
-0+ }while (t.task); + +set_current_state (TASK_RUNNING ); + +destroy_hrtimer_on_stack (&t.timer); +} +/* * Handle management. * I was convinced we already had such a creature,. but I guess I was wrong as I can't find it ;-)
Quoted 7 lines
Anyway, I'm thinking this function ought to live in
kernel/hrtimer.cand possibly get renamed to something likehrtimer_sleep_ns()or some such (means you can also reuse thedo_nanosleephelper in there).Also, have you considered the impact on platforms that do not support hrtimers, or don't have high resolution clock events available?
I agree that this is definitely a much more generically interesting function.
With high res timers, falling back to the jiffie's granularity (or not sleeping at all in the case of the file system transaction batching) seems to be a reasonable fall back,
ric
Quoted 18 lines
Quoted 7 lines
Index:
linux-2.6/kernel/hrtimer.c
===================================================================kernel/hrtimer.c void hrtimer_init_sleeper(struct hrtimer · line 1464
Quoted 7 lines
kernel/hrtimer.c +1
-0@@ -1458,6 +1458,7 @@ void hrtimer_init_sleeper(struct hrtimer sl->timer.cb_mode =HRTIMER_CB_IRQSAFE_NO_SOFTIRQ ;#endif } +EXPORT_SYMBOL_GPL (hrtimer_init_sleeper); static int __sched do_nanosleep(struct hrtimer_sleeper *t, enum hrtimer_mode mode) {That also gets rid of this export..
fs/jbd/commit.c restart_loop: · line 883
Quoted 5 lines · Josef Bacik
fs/jbd/commit.c +5
-0+if (unlikely (!journal->j_average_commit_time)) + journal->j_average_commit_time = commit_time; +else + journal->j_average_commit_time = (commit_time + + journal->j_average_commit_time) /2 ;
You may also consider making this a decaying average, so that minor changes in the workload are smoothed out. Also, it is probably easier to read likely(foo) instead of unlikely(!foo)...
if (likely(journal->j_average_commit_time != 0)) journal->j_average_commit_time = (commit_time * 3 + journal->j_average_commit_time) / 4; else journal->j_average_commit_time = commit_time;
fs/jbd/commit.c restart_loop: · line 889
Quoted 5 lines
fs/jbd/commit.c +5
-0+if (journal->print_count <100 ) { + journal->print_count++; +printk (KERN_ERR "avg commit time = %lu\n" , + journal->j_average_commit_time); + }
There is already the jbd stats patch in jbd2 that is reporting this information for the previous transactions.
fs/jbd/transaction.c int journal_stop(handle_t *handle) · line 1441
Quoted 9 lines
fs/jbd/transaction.c +8
-0+spin_lock (&journal->j_state_lock); + commit_time = journal->j_average_commit_time; +spin_unlock (&journal->j_state_lock); + + sleep_time =elapsed_jiffies (transaction->t_start_time, jiffies); +if (!sleep_time) + sleep_time =1 ; + sleep_time = (commit_time / sleep_time) * commit_time;
I was also going to comment on the use of jiffies here, but Ric beat me to it.
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.