Skip to contentSkip to conversation

very low performance on SCSI disks if device node is in tmpfs

DISCUSSION Olaf Hering 6 participants 13 messages May 25, 2004 – May 26, 2004 VFS

lore
Share & tools
lore
#0 lore raw

Any ideas why the location of the device node makes such a big difference? I always wondered why my firewire is so dog slow with 2.6. Now I know the reason: /dev is in tmpfs.
I dont see that with IDE disks, only with SCSI.

(none):/# /sbin/hdparm -tT /dev/sdb

/dev/sdb:
Timing buffer-cache reads: 2532 MB in 2.00 seconds = 1264.93 MB/sec
Timing buffered disk reads: 48 MB in 3.12 seconds = 15.41 MB/sec

(none):/# /sbin/hdparm -tT /tmp/sdb

/tmp/sdb:
Timing buffer-cache reads: 2328 MB in 2.00 seconds = 1163.01 MB/sec
Timing buffered disk reads: 82 MB in 3.03 seconds = 27.03 MB/sec

This happens also with 2.6.1-mm kernels. I attach the profile with devnode in tmpfs and on disk.

Signature · 3 lines
USB is for mice, FireWire is for men!

sUse lINUX ag, nÜRNBERG
     1 .save_remaining_regs                       0.0063
     1 .DoSyscall                                 0.0093
     8 .handle_irq_event                          0.0278
 36214 .__iommu_free                             79.4167
     1 .pSeries_flush_hash_range                  0.0013
     1 .openpic2_get_irq                          0.0147
     1 .do_page_fault                             0.0007
     3 .__wake_up                                 0.0208
     8 .schedule                                  0.0036
     1 .__tasklet_schedule                        0.0039
     5 .r_start                                   0.0240
     1 .alloc_uid                                 0.0021
     3 .__lock_page_wq                            0.0068
     5 .wait_on_page_bit_wq                       0.0118
     3 .unlock_page                               0.0208
    76 .find_lock_page                            0.1131
     9 .add_to_page_cache_lru                     0.0833
     6 .read_cache_page                           0.0032
   142 .do_generic_mapping_read                   0.0625
     1 .grab_cache_page_nowait                    0.0013
     2 .__generic_file_aio_write_nolock           0.0006
     3 .mempool_resize                            0.0045
     4 .mempool_create                            0.0092
     3 .__free_pages_ok                           0.0066
     1 .__alloc_pages                             0.0009
     4 .write_one_page                            0.0049
     1 .page_cache_readahead                      0.0014
     6 .force_page_cache_readahead                0.0091
     3 .read_cache_pages                          0.0030
     1 .free_block                                0.0022
     1 .do_drain                                  0.0086
     2 .__cache_shrink                            0.0048
     1 .kstrdup_vec                               0.0035
     2 .slabinfo_write                            0.0037
     4 .__pagevec_lru_add_active                  0.0094
     1 .__pagevec_lru_add                         0.0026
     5 .lru_add_drain                             0.0298
     5 .__pagevec_release                         0.0694
     6 .invalidate_complete_page                  0.0117
    10 .truncate_complete_page                    0.0243
     3 .truncate_inode_pages                      0.0025
     1 .unmap_vmas                                0.0010
     1 .page_remove_rmap                          0.0026
     1 .shmem_truncate                            0.0008
     2 .shmem_getpage                             0.0005
     2 .generic_file_llseek                       0.0068
     1 .remove_inode_buffers                      0.0040
     2 .end_bio_bh_io_sync                        0.0119
     1 .nobh_truncate_page                        0.0017
     1 .buffer_io_error                           0.0114
     2 .drop_buffers                              0.0026
     2 .__find_get_block_slow                     0.0033
     1 .invalidate_bh_lru                         0.0068
     1 .__find_get_block                          0.0013
     3 .__bforget                                 0.0142
     1 .__block_commit_write                      0.0026
    10 .mark_buffer_dirty_inode                   0.0490
     4 .block_truncate_page                       0.0042
     6 .__block_prepare_write                     0.0044
     3 .block_invalidatepage                      0.0063
     2 .__bread                                   0.0060
     2 .end_buffer_async_write                    0.0042
     1 .bio_split                                 0.0027
     6 .bio_check_pages_dirty                     0.0101
     1 .bio_dirty_fn                              0.0021
     1 .sync_filesystems                          0.0018
     1 .bd_release                                0.0066
     1 .bd_claim                                  0.0045
     1 .pipe_release                              0.0028
     1 .dput                                      0.0013
     2 .ilookup                                   0.0069
     2 .destroy_inode                             0.0079
     1 .get_filesystem_list                       0.0045
     3 .generic_osync_inode                       0.0066
     1 .reiserfs_read_locked_inode                0.0004
     2 .reiserfs_iget                             0.0082
     1 .reiserfs_fill_super                       0.0003
    26 .search_by_key                             0.0034
     3 .reiserfs_delete_solid_item                0.0016
     4 .flush_commit_list                         0.0017
     1 .reiserfs_add_ordered_list                 0.0034
     4 .do_journal_end                            0.0008
     2 .do_journal_release                        0.0038
     1 .reiserfs_in_journal                       0.0016
     1 .reiserfs_resize                           0.0005
     1 .memmove                                   0.0833
     4 .__strncpy_from_user                       0.0667
     3 .bitreverse                                0.0174
  1482 .crc32_le                                  4.1167
    49 .qsort                                     0.0407
     6 .zlib_inflate_blocks                       0.0020
     1 .n_tty_ioctl                               0.0003
     1 .dev_set_rdonly                            0.0053
     2 .blk_queue_find_tag                        0.0417
     1 .blk_queue_init_tags                       0.0047
     1 .blk_queue_invalidate_tags                 0.0035
     2 .__make_request                            0.0010
     1 .blk_init_queue                            0.0019
     9 .blkdev_ioctl                              0.0036
     1 .bcm5201_suspend                           0.0033
    11 .scsi_host_lookup                          0.0372
     8 .__scsi_mode_sense                         0.0112
     1 .scsi_mode_sense                           0.0050
     1 .scsi_free_host_dev                        0.0038
     1 .scsi_alloc_sdev                           0.0008
     1 .scsi_probe_and_add_lun                    0.0003
     1 .ata_scsi_slave_config                     0.0045
     1 .bit_func                                  0.0417
     2 .i2c_stop                                  0.0030
     1 .dev_ethtool                               0.0003
     1 .tcp_sendmsg                               0.0002
     1 .inetdev_init                              0.0016
 38268 total                                      0.0102
     4 .save_remaining_regs                       0.0250
     1 .DoSyscall                                 0.0093
    35 .handle_irq_event                          0.1215
     1 .sched_clock                               0.0312
     1 .__down_interruptible                      0.0027
     5 .power4_idle                               0.0347
 37021 .__iommu_free                             81.1864
     1 .openpic2_get_irq                          0.0147
     1 .flush_dcache_page                         0.0179
     5 .__wake_up                                 0.0347
    27 .schedule                                  0.0122
     1 .io_schedule                               0.0028
     1 .finish_wait                               0.0053
     2 .prepare_to_wait                           0.0106
     1 .remove_wait_queue                         0.0069
     1 .copy_process                              0.0002
     2 .__tasklet_hi_schedule                     0.0078
     3 .__tasklet_schedule                        0.0117
    25 .r_start                                   0.1202
     1 .alloc_uid                                 0.0021
     3 .__lock_page_wq                            0.0068
     2 .wait_on_page_bit_wq                       0.0047
     1 .unlock_page                               0.0069
    75 .find_lock_page                            0.1116
     1 .add_to_page_cache                         0.0014
     6 .add_to_page_cache_lru                     0.0556
     1 .read_cache_page                           0.0005
   147 .do_generic_mapping_read                   0.0647
     1 .wait_on_page_writeback_range_wq           0.0015
     3 .__generic_file_aio_write_nolock           0.0009
     3 .mempool_resize                            0.0045
     3 .mempool_create                            0.0069
     1 .setup_per_zone_pages_min                  0.0022
     1 .__free_pages_ok                           0.0022
     1 .__alloc_pages                             0.0009
     1 .write_one_page                            0.0012
     1 .do_page_cache_readahead                   0.0016
     6 .read_cache_pages                          0.0059
     1 .free_block                                0.0022
     1 .do_drain                                  0.0086
     3 .__cache_shrink                            0.0071
     1 .__kmalloc                                 0.0044
     1 .__pagevec_lru_add_active                  0.0024
     2 .lru_add_drain                             0.0119
     4 .__pagevec_release                         0.0556
     1 .invalidate_complete_page                  0.0020
     1 .truncate_complete_page                    0.0024
     2 .truncate_inode_pages                      0.0017
     1 .wakeup_kswapd                             0.0100
     1 .anon_vma_prepare                          0.0038
     1 .page_remove_rmap                          0.0026
     1 .unmap_pte_page                            0.0009
     2 .try_to_unmap_one                          0.0014
     2 .sys_readv                                 0.0063
     1 .invalidate_inode_buffers                  0.0048
     1 .nobh_commit_write                         0.0069
     1 .generic_cont_expand                       0.0014
     2 .submit_bh                                 0.0032
     1 .__find_get_block_slow                     0.0016
     2 .mark_buffer_dirty                         0.0116
     2 .block_invalidatepage                      0.0042
     1 .end_buffer_async_write                    0.0021
     1 .match_token                               0.0014
     2 .__strncpy_from_user                       0.0333
     4 .bitreverse                                0.0233
  1542 .crc32_le                                  4.2833
    42 .qsort                                     0.0349
     4 .zlib_inflate_blocks                       0.0013
     2 .tty_read                                  0.0044
     1 .start_tty                                 0.0032
     1 .blk_rq_map_sg                             0.0021
     1 .blk_queue_init_tags                       0.0047
     1 .blk_queue_invalidate_tags                 0.0035
     4 .__make_request                            0.0020
    11 .blkdev_ioctl                              0.0043
     1 .rd_open                                   0.0042
     1 .gem_interrupt                             0.0001
     1 .scsi_unregister                           0.0208
    50 .scsi_host_lookup                          0.1689
     1 .scsi_add_host                             0.0023
    18 .__scsi_mode_sense                         0.0253
     3 .scsi_mode_sense                           0.0150
    13 .scsi_free_host_dev                        0.0492
     2 .scsi_alloc_sdev                           0.0017
     1 .scsi_forget_host                          0.0046
     1 .scsi_probe_and_add_lun                    0.0003
     1 .keywest_timeout                           0.0040
     2 .i2c_stop                                  0.0030
     1 .netdev_run_todo                           0.0009
     1 .qdisc_kill_estimator                      0.0030
 39145 total                                      0.0105
#1 lore raw
Quoted 17 lines · Olaf Hering

Any ideas why the location of the device node makes such a big difference? I always wondered why my firewire is so dog slow with 2.6. Now I know the reason: /dev is in tmpfs.
I dont see that with IDE disks, only with SCSI.

(none):/# /sbin/hdparm -tT /dev/sdb

/dev/sdb:
Timing buffer-cache reads: 2532 MB in 2.00 seconds = 1264.93 MB/sec
Timing buffered disk reads: 48 MB in 3.12 seconds = 15.41 MB/sec

(none):/# /sbin/hdparm -tT /tmp/sdb

/tmp/sdb:
Timing buffer-cache reads: 2328 MB in 2.00 seconds = 1163.01 MB/sec
Timing buffered disk reads: 82 MB in 3.03 seconds = 27.03 MB/sec

Quoted 6 lines

This happens also with 2.6.1-mm kernels. I attach the profile with devnode in tmpfs and on disk.

--
USB is for mice, FireWire is for men!

You are not actually reading or writing from the 'device'-files. They are just a trick to associate a major/minor number with a device. After that initial open, the fd is the handle by which the device is accessed, not the device-files. Given this, it doesn't make any difference if the device-file is generated dynamically as in devfs or statically with mknod. Unless you have a bench mark program that does:

             do {
		open();
		write();
		close();
		open();
		read();
		close();
                } while(forever);

If that's what the program does, it's probably not a good example to be used as a bench-mark. If the program does that, and you want to optimize for that, then the way a read is handled (for the open in devfs), would need to be optimized to use non-paged memory because it is possible, in principle, to have to page in the contents of devfs for each access. Right now, kmalloc(GFP_KERNEL) is being used and that's pagable.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5570.56 BogoMips).
Note 96.31% of all statistics are fiction.

#2 lore raw

On Tue, May 25, Richard B. Johnson wrote unrelated stuff:

did you actually try it?

(none):~ # /usr/bin/time dd if=/dev/sdb bs=1M count=123 of=/dev/null 123+0 records in
123+0 records out
0.00user 0.75system 0:17.13elapsed 4%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+426minor)pagefaults 0swaps
(none):~ # /usr/bin/time dd if=/dev/sdb bs=1M count=123 of=/dev/null 123+0 records in
123+0 records out
0.00user 0.74system 0:15.66elapsed 4%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+426minor)pagefaults 0swaps
(none):~ # /usr/bin/time dd if=/dev/sdb bs=1M count=123 of=/dev/null 123+0 records in
123+0 records out
0.00user 0.78system 0:15.67elapsed 5%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+426minor)pagefaults 0swaps
(none):~ # /usr/bin/time dd if=/tmp/sdb bs=1M count=123 of=/dev/null 123+0 records in
123+0 records out
0.00user 0.47system 0:06.44elapsed 7%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+426minor)pagefaults 0swaps
(none):~ # /usr/bin/time dd if=/tmp/sdb bs=1M count=123 of=/dev/null 123+0 records in
123+0 records out
0.00user 0.47system 0:06.44elapsed 7%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+426minor)pagefaults 0swaps
(none):~ # /usr/bin/time dd if=/tmp/sdb bs=1M count=123 of=/dev/null 123+0 records in
123+0 records out
0.00user 0.50system 0:06.44elapsed 7%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+426minor)pagefaults 0swaps

Signature · 3 lines
USB is for mice, FireWire is for men!

sUse lINUX ag, nÜRNBERG
#3 lore raw
Quoted 2 lines · Olaf Hering

On Tue, May 25, Richard B. Johnson wrote unrelated stuff:

It is not unrelated stuff. If you think it is, you will come to an incorrect conclusion with your experiments. As I stated, you are not actually communicating with a device-file except for the initial open().

Quoted 2 lines

did you actually try it?

[SNIPPED... unrelated access time information]

Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5570.56 BogoMips).
Note 96.31% of all statistics are fiction.

#4 lore raw

Am Dienstag, 25. Mai 2004 21:44 schrieb Richard B. Johnson:

Quoted 9 lines · Olaf Hering
Quoted 3 lines

On Tue, May 25, Richard B. Johnson wrote unrelated stuff:

It is not unrelated stuff. If you think it is, you will come to an incorrect conclusion with your experiments. As I stated, you are not actually communicating with a device-file except for the initial open().

Exactly therefore the results are so extremely odd. Either looking up a device node is very expensive, or we have an unknown phenomenon.

Regards
Oliver

#5 lore raw
In reply to Olaf Hering
Quoted 5 lines · Olaf Hering

Any ideas why the location of the device node makes such a big difference? I always wondered why my firewire is so dog slow with 2.6. Now I know the reason: /dev is in tmpfs.
I dont see that with IDE disks, only with SCSI.

This is truly bizarre. Reading /dev/sda I get 24MB/sec at 700 context switches/sec. Reading /mnt/tmpfs/sda it's 14MB/sec, 7000 switches/sec. /mnt/ramfs/sda is slow too. /mnt/hda5/sda is fast.

I'd assumed that the kernel got the backing_dev_info's screwed up and the tmpfs node isn't doing readahead but that appears to not be the case (/dev/sda is still fast with zero readahead).

You really, really get the weird-bug-of-the-month award for this one. I'll poke at it some more later on.

#6 lore raw
Quoted 9 lines · Andrew Morton
Quoted 5 lines · Olaf Hering

Any ideas why the location of the device node makes such a big difference? I always wondered why my firewire is so dog slow with 2.6. Now I know the reason: /dev is in tmpfs.
I dont see that with IDE disks, only with SCSI.

This is truly bizarre.

It affects IDE too - the context switch and interrupt rates are through the roof. IDE disks have always seemed better at doing readahead, which is why the transfer bandwidth is much the same.

Everything there is consistent with "not doing readahead". The fickle finger points in the general direction of fs/block_dev.c:do_open(). Later.

#7 lore raw
Quoted 3 lines · Andrew Morton

Everything there is consistent with "not doing readahead".

We need to set file->f_ra _after_ calling blkdev_open(), when inode->i_mapping points at the right thing. And we need to get it from inode->i_mapping->host->i_mapping too, which represents the underlying device.

fs/open.c renamed from fs/open.c~blockdev-readahead-fix +2 -1
@@ -790,7 +790,6 @@ struct file *dentry_open(struct dentry *
 	}
 
 	f->f_mapping = inode->i_mapping;
-	file_ra_state_init(&f->f_ra, f->f_mapping);
 	f->f_dentry = dentry;
 	f->f_vfsmnt = mnt;
 	f->f_pos = 0;
@@ -804,6 +803,8 @@ struct file *dentry_open(struct dentry *
 	}
 	f->f_flags &= ~(O_CREAT | O_EXCL | O_NOCTTY | O_TRUNC);
 
+	file_ra_state_init(&f->f_ra, f->f_mapping->host->i_mapping);
+
 	/* NB: we're sure to have correct a_ops only after f_op->open */
 	if (f->f_flags & O_DIRECT) {
 		if (!f->f_mapping || !f->f_mapping->a_ops ||

_

#8 lore raw
Quoted 4 lines · Andrew Morton

We need to set file->f_ra _after_ calling blkdev_open(), when inode->i_mapping points at the right thing. And we need to get it from inode->i_mapping->host->i_mapping too, which represents the underlying device.

Hmm.. Is f_mapping is guaranteed to be non-NULL? At least for the O_DIRECT case, we explicitly test for f_mapping being non-NULL, although that test is quite possibly bogus. Maybe we should fix that too?

Linus

#9 lore raw
Quoted 11 lines · Linus Torvalds
Quoted 4 lines · Andrew Morton

We need to set file->f_ra _after_ calling blkdev_open(), when inode->i_mapping points at the right thing. And we need to get it from inode->i_mapping->host->i_mapping too, which represents the underlying device.

Hmm.. Is f_mapping is guaranteed to be non-NULL? At least for the O_DIRECT case, we explicitly test for f_mapping being non-NULL, although that test is quite possibly bogus. Maybe we should fix that too?

->f_mapping should never be NULL after successive open().

#10 lore raw
In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Andrew Morton

We need to set file->f_ra _after_ calling blkdev_open(), when inode->i_mapping points at the right thing. And we need to get it from inode->i_mapping->host->i_mapping too, which represents the underlying device.

Hmm.. Is f_mapping is guaranteed to be non-NULL?

Yes, we just did

        f->f_mapping = inode->i_mapping;
Quoted 3 lines

At least for the O_DIRECT
case, we explicitly test for f_mapping being non-NULL, although that test is quite possibly bogus. Maybe we should fix that too?

yup.

#11 lore raw
In reply to Andrew Morton
Quoted 10 lines · Andrew Morton
Quoted 3 lines · Andrew Morton

Everything there is consistent with "not doing readahead".

We need to set file->f_ra _after_ calling blkdev_open(), when inode->i_mapping points at the right thing. And we need to get it from inode->i_mapping->host->i_mapping too, which represents the underlying device.

That fixed it, thanks for the patch and the award!
I will take the patch and pass the award to the guy who found the bug.

Signature · 3 lines
USB is for mice, FireWire is for men!

sUse lINUX ag, nÜRNBERG
#12 lore raw
In reply to Andrew Morton
Quoted 12 lines · Andrew Morton
Quoted 5 lines · Olaf Hering

Any ideas why the location of the device node makes such a big difference? I always wondered why my firewire is so dog slow with 2.6. Now I know the reason: /dev is in tmpfs.
I dont see that with IDE disks, only with SCSI.

This is truly bizarre. Reading /dev/sda I get 24MB/sec at 700 context switches/sec. Reading /mnt/tmpfs/sda it's 14MB/sec, 7000 switches/sec. /mnt/ramfs/sda is slow too. /mnt/hda5/sda is fast.

Quoted 8 lines

I'd assumed that the kernel got the backing_dev_info's screwed up and the tmpfs node isn't doing readahead but that appears to not be the case (/dev/sda is still fast with zero readahead).

You really, really get the weird-bug-of-the-month award for this one. I'll poke at it some more later on.
-

Try the attached utility to compare the open time on devfs and on the regular fs.

I note that opening a file on a RAM disk seems to take a lot longer than on a physical device. This might be paging time.

Anyway, this might give some hints of what's causing the problem.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5570.56 BogoMips).
Note 96.31% of all statistics are fiction.