Linux-2.5.14..

There's a lot of stuff that has happened in the 2.5.x series lately, and you can see the gory details in the ChangeLog files that accompany releases these days, but I thought I'd point out 2.5.14, since it has some interesting fundamental changes to how dirty state is maintained in the VM.

(The big changes were actually in 2.5.12, but 2.5.13 contained various minor fixes and tweaks, and 2.5.14 contains a number of fixes especially wrt truncate, so hopefully it's fairly _stable_ as of 2.5.14.)

Credit goes to Andrew Morton, and not only does it clean up the code a lot, it also seems to perform a lot better in many circumstances.

There's a lot of other stuff in the 2.5.x tree too, but few things are so fundamental. Please test (but also, please be careful - backups are always a good idea).

Linus

bugs with ext3, 2k block file systems and 2.5 have been fixed. Should I expect this release to address the problems I was seeing?

Daniel

I keep my head above the surface, trying to breath, looking for land.
I keep an eye at the distant horizon waiting for help, clutching the sky.
        -- Covenant, _Phoenix_

"Expect" is too strong a word. I'd say "hope" - a number of truncate bugs were fixed, but whether that was what bit you, nobody knows.

I suspect the real answer is that we'd love for you to test things out, but that if it ends up being too painful to recover if the problems happen again, you probably shouldn't..

Linus

Well, hope seems justified...

I did, and I failed to reproduce it working on a scratch disk. This was a period of playing and I /hope/ that it's conclusive. I couldn't get .12 to reliably fail, though, which is less inspiring.

I should be able to find some time in the next day or so to test it a bit more on the scratch disk and then, if that works, I will update my backups. :)

Still, it seems good so far.
Daniel

Television is the ideal propaganda medium, a mendacious monster, not primarily
out of malice but from its amoral nature.
        -- Paul Johnson

Right. I got brave enough to test it on a real, live system after extensive fake testing. It seems to work well, at least so far as running the same workload that cause massive file corruption correctly.

So, I believe that 2.5.14 is working correctly with 2k ext3 filesystems, at least for minimal use. I didn't do any sort of extreme load testing or anything like that, being cautious about it.

On reboot, I got an assertion in ext3, though, and the following BUG trace. So, something still isn't well, but it seems to be getting it much more right. :)

Daniel

ksymoops 2.4.5 on i686 2.5.6. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.5.6/ (default)
-m /boot/System.map-2.5.14 (specified)

Error (regular_file): read_ksyms stat /proc/ksyms failed No modules in ksyms, skipping objects
No ksyms, skipping lsmod
Invalid Operand: 0000

CPU: 0

EIP: 0010:[<c015cf45>] Not Tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010286
EAX: 00000061 EBX: dc883900 ECX: c14ee080 EDX: df954ca0
ESI: dd36d200 EDI: dfd53600 EBP: d0d805c0 ESP: c14f1e6c
DS: 0018 ES: 0018 SS: 0018

Stack: c02c0060 c02c04e1 c02c0040 00000460 c02c0537 d2821380 d0d805c0 00000000
       c14f1f04 c01557fd d0d805c0 d2821380 d2821380 00000800 d2821380 00000800
       00000800 000000c0 c015555c d0d805c0 d2821380 0005f700 00000000 bfd71c00
Call Trace: [<c01557fd>] [<c015555c0>] [<c0155909>] [<c01557e4>] [<c0126bab>]
            [<c01535fa>] [<c0132576>] [<c0106c97>]
Code: 0f 0b 60 04 40 00 2c c0 83 c4 14 6a 03 8b 45 00 50 53 e8 1c

Trace; c01557fd <commit_write_fn+19/5c>
Trace; c015555c0 <END_OF_CODE+b411aa04c/????>
Trace; c0155909 <ext3_commit_write+c9/1e4>
Trace; c01557e4 <commit_write_fn+0/5c>
Trace; c0126bab <generic_file_write+4c3/6e4>
Trace; c01535fa <ext3_file_write+46/4c>
Trace; c0132576 <sys_write+96/f0>
Trace; c0106c97 <syscall_call+7/b>

Code; c015cf45 <journal_dirty_metadata+13d/174>
00000000 <_EIP>:
Code; c015cf45 <journal_dirty_metadata+13d/174> <=====
0: 0f 0b ud2a <=====
Code; c015cf47 <journal_dirty_metadata+13f/174>
2: 60 pusha
Code; c015cf48 <journal_dirty_metadata+140/174>
3: 04 40 add $0x40,%al
Code; c015cf4a <journal_dirty_metadata+142/174>
5: 00 2c c0 add %ch,(%eax,%eax,8)
Code; c015cf4d <journal_dirty_metadata+145/174>
8: 83 c4 14 add $0x14,%esp
Code; c015cf50 <journal_dirty_metadata+148/174>
b: 6a 03 push $0x3
Code; c015cf52 <journal_dirty_metadata+14a/174>
d: 8b 45 00 mov 0x0(%ebp),%eax
Code; c015cf55 <journal_dirty_metadata+14d/174>
10: 50 push %eax
Code; c015cf56 <journal_dirty_metadata+14e/174>
11: 53 push %ebx
Code; c015cf57 <journal_dirty_metadata+14f/174>
12: e8 1c 00 00 00 call 33 <_EIP+0x33> c015cf78 <journal_dirty_metadata+170/174>

1 error issued. Results may not be reliable.

The artistic temperment is a disease which afflicts amateurs.
        -- G. K. Chesterton, _Heretics_, 1905

hmm.

I've been testing 2.5.14 pretty hard for a couple of days.

ext2, ext3-ordered, ext3-writeback (all with small blocks) are solid.

reiserfs and vfat are solid.

JFS deadlocks (see the BUGBUG comment in jfs_txnmgr.c - it happens). I've asked the JFS guys for help on this; possibly the code can just be removed:
the buffer-based writeout which I replaced wouldn't have written the pages anyway...

ext3-journalled is not happy.

There's a locking bug between try_to_free_buffers and buffer_insert_inode_queue which never seems to trigger. I've got that fixed.

There's a known race between unmount and writeback which is probably impossible to trigger. (see the FIXME at __sync_list). Testing the fix for that at present.

The "sync" functions aren't right. Pages which are both dirty and under writeback are not correctly waited upon. This is a minor correctness thing which nobody would notice. Still thinking about the best way to close this.

So unless you're a JFS or ext3-journalled user, 2.5.14 is OK.

04 60 -> line 1120. Yup, I get that one too. I assume you were testing with data=journal.

Thanks again...

-

Not conclusive, I know, but getting a panic after a brief test stopped it at that point. :)

[...]

Which probably explains my error, then, as I have not stepped back from data journaled mode on my system.

[...]

Well, unmount wouldn't have happened for quite a long time in the shutdown process, given it was at the initial 'send SIGTERM' stage...

[...]

The latter. Ah, well, at least you know.

[...]

Confirmed.
Daniel

No, no, you're not thinking, you're just being logical.
        -- Niels Bohr

I parsed this 'dirty state' sentence all wrong at first :-) Andrew, Linus - where does the current VM lie in between rmap-vm and aa-vm?

Regards,

bert hubert

http://www.PowerDNS.com          Versatile DNS Software & Services
http://www.tk                              the dot in .tk
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

I don't have an explanation for the ext3 problem which you saw. It's conceivable that 2.5.13 was leaving dirty buffers around after they were "deleted", and that fsync grabbed them via the i_dirty_buffers back door, and wrote them where they shouldn't have been written.

But they wouldn't have been mapped anywhere...

So I still need to try to reproduce that one. If you could have another shot, it would be appreciated. But if it _does_ work OK, I can't say it's fixed until I know what caused the 2.4.13 failure.

ext3 is very sensitive to what is going on in buffer.c. There's a lot of tension in there between the desire to share code and the desire to not be damaged by changes in the code which we share.

Generally, ext3 with data=journal is not happy at present.

Partly because it contains assertions of things which aren't true any more.

Partly because of a known problem in ext3: assertion failure at transaction.c:606. Stephen has a fix for this which we need to wiggle into 2.4. For some reason, the 2.5 changes are triggering it much more easily.

I'll spend a few hours this week trying to resurrect data=journal, but if that doesn't work out I think I'll just turn it off for the while, make it emit a warning and use data=ordered.

-

"VM" is a broad term. The memory allocator, page replacement, swap and all that stuff is unaltered - it is the same as 2.4.current. ie: Andrea's VM from when his changes stopped going into the mainline kernel.

I made minimal changes in there to teach the page allocator that all dirty memory is written back via pages and not sometimes-pages, sometimes-buffers. Also to add support for the new `clustering writeback' which address_spaces can perform.

It's probably not as well tuned as it could be at present, but I don't see a lot of point in fiddling with it. As long as the VM doesn't actually impede 2.5 development and evaulation of 2.5 performance, best to leave it alone until a VM developer steps up to do the 2.6 VM.

The change to which Linus refers is:

In 2.4, dirty data from the write(2) system call is encapsulated in buffer_heads and is placed on a global buffer list for writeout. And dirty data from shared mappings is attached to its inode.

In 2.5, the buffer list went away, and dirty data from write(2) is now managed in the same way as dirty data from mmap().

And because the kupdate and bdflush functions used to work against the buffer LRU, replacements were introduced which do the same thing against the inodes, instead of against the buffers.

So it's all page-oriented now.

-

Nice, this will make it possible to have much cleaner page replacement code.

regards,

Rik

Bravely reimplemented by the knights who say "NIH".

http://www.surriel.com/		http://distro.conectiva.com/

Sun May 5 16:34:59 CEST 2002 ide-clean-55

Resync with 2.5.14.

- Update HPT374 driver carried over from 2.4.xx series by Andrew Morton. Resync it with the recent host chip driver changes, or better the introduction of an API at all.

- Consolidate the handling of device ID byte order in one place. This was spotted and patched by Bartomiej onierkiewicz.

- Eliminate CONFIG_BLK_DEV_IDEPCI - it's duplicating the functionality of the already present and fine CONFIG_PCI flag and if we are a PCI host, we are indeed very likely to need host chip support anyway.

- Remove some redundant info about the model and channel number from /proc/ide. Remove the binary entries not helpful to the user, and not used by any program and redundant to corresponding ioctls.

- Properly return udma_read and udma_write values in taskfile.

- Only initialize XXX_udma to the default handlers if it has not been initialized by the host chip initialization.

I have enabled spin lock debugging and can see that on device flush the spin locks get wrong counts... no problems elsewher ethus far. I will re check them next time around.

Martin, how do you know it isn't used by any program?

changing /proc stuff 'becouse it's availabe elsewhere' is not a good idea, people writing shell scripts do not nessasarily have access to ioctls.

you must have missed the proc wars of past kernels, but in general it's not a good idea to tinker with anything in /proc unless it is a significant improvement and 'redundant to corresponding ioctls' is not a significant improvement.

David Lang

Hi,

Please don't do this! There are configurations possible with pci enabled but without a pci ide adapter.

Could you please try to compile without CONFIG_BLK_DEV_IDEDMA_PCI enabled?

bye, Roman

Uz.ytkownik Roman Zippel napisa?:

So please just don't configure any PCI host chip support there.

Hi,

Then ide-pci.c is still compiled into the kernel. Why?

bye, Roman

Uz.ytkownik Roman Zippel napisa?:

Becouse the big tables there are subject to go.

Uz.ytkownik Martin Dalecki napisa?:

And at some point in time it will check whatever there is request for any host chip support.

Hi,

Could you please then do the above change _after_ you have done this?

bye, Roman

Uz.ytkownik Roman Zippel napisa?:

Well one question renames: Please name me one PCI based architecture which contains IDE support and does not contain any special host chip attached to the very same PCI bus as well.

Hi,

Architectures which are not directly PCI based, but which can have have PCI bridges. On the other hand even on PCI based archs you don't necessarily want to compile in ide support automatically, please leave that to Eric's autoconf.

bye, Roman

SiS 496/497 PCI chipset for i486's. It has integrated IDE controller, but this controller is not connected to PCI bus.

Andrey Panin            | Embedded systems software engineer
[email protected]        | PGP key: wwwkeys.eu.pgp.net

OK - That actually is an argument I follow. Thank you I will adjust the code. (Not quite right jet maybe but I will do it.)

If by 'attached' you mean integrated into chipset, I have several such machines that have no IDE without an extra add-on card.

My quad ppro has two PCI buses, but no IDE controller. My PCI 486 has no onboard IDE.

| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Hi,

Another thing: where is the equivalilent part of this removed code?

-static __inline__ void ide_fix_driveid(struct hd_driveid *id)
-{
-#if defined(CONFIG_AMIGA) || defined (CONFIG_MAC) || defined(M68K_IDE_SWAPW)

-   u_char *p = (u_char *)id;
-   int i, j, cnt;
-   u_char t;

-
- if (!MACH_IS_AMIGA && !MACH_IS_MAC && !MACH_IS_Q40 && !MACH_IS_ATARI)
- return;
-#ifdef M68K_IDE_SWAPW
- if (M68K_IDE_SWAPW) /* fix bus byteorder first */
- for (i=0; i < 512; i+=2) {
- t = p[i]; p[i] = p[i+1]; p[i+1] = t;
- }
-#endif

bye, Roman

Uz.ytkownik Roman Zippel napisa?:

Look closer it's there in ide-probe.c.

Hi,

Does it still take the correct byte swapping into account? Did you consider using a table for the fixup? It's nothing perfomance critical and this might generate more compact code.

bye, Roman

Uz.ytkownik Roman Zippel napisa?:

Well right now we have two different reimplementation of get device id code, so this areas is subject to change anyway. BTW.> It should indeed take both in to account as far as I can see.(Despite the fact that I could affort an ATARI I hardly can find one...)

Hi,

That's not necessary, but I'm only afraid that functionality gets lost, which isn't needed on the latest hardware.

bye, Roman

Mon May 6 13:29:44 CEST 2002 ide-clean-56

- Push poll_timeout down from the hwgroup to the channel. We are resetting the channel and not a whole hwgroup. This way using multiple pdc202xxx cards should magically start to work with multiple performance and resets will no longer lock the system.

- Updates for PDC4030 by Peter Denison <[email protected]>.

- Make ide_raw_taskfile don't care about request buffers. They where always NULL.

- Port set multi mode count over from the special setting interface to ide_raw_taskfile. Fix errors in the corresponding interrupt handler in one go as well. It turned out that this is precisely the same code as in task_no_data_intr, so we can nuke it altogether. And finally we have found some problems with the set_pio_mode() command which can fail with -EBUSY - this is in esp. probably *very* common during boot hdparm usage those days! (OK it was masked by reportig too early that it finished... Crap Crap utter crap it was!!!) Right now hdparm should just be extendid to properly
sync and retry on -EBUSY and everything should be fine.

And now the 1 Milion EUR question for everybody who loves to put driver settings in to /proc:

How the hell could echo > /proc/ide/ide0/settings blah blah blah blah handle properly cases like -EIO, -EBUSY and so on??? Having the possibility o do it does not mean that it is a good idea to use it.

OK. After realizing the simple fact that quite a lot of low level hardware manipulating ioctls may require assistance in usage from proper logic which is *very* unlikely to be implemented in a bash (for me preferable still ksh) I have made my mind up.

/proc/ide will be nuked.

- Execute the recalibration for error recovery on precisely the same request as the one which failed.

- Remove set geometry. It's crap by means of standard specification. Because:

1. We relay on the existence of the identify command anyway.

2. This command was obsoleted *before* the identify command existed as far as I can see.

2. I'm able to have a look at what other ATA/ATAPI drivers in the wild do:
They don't do it.

- Just call tuneproc in set_pio_mode() directly - we are already behind the rq queue there.

- After we have uncovered the broken logics behind the whole ioctl handling we now just have made ide_spin_wait_hwgroup() waiting for a proper somehow longer timeout before giving up. This was previously just hiding the broken concept of setting ioctl values through /proc/ide/ideX/settings - now it just really helps hdparm to not to give up too early. (It shouldn't probably play wreck havock on the global driver spin lock as well. I will look in to this later.)

- Scrap the non necessary, to say the least, disabling of interrupts for 3, read it again please, 3 seconds, on the local CPU inside ide_spin_wait_hwgroup(). Spin lock handling needs checking there badly as I see now as well...

Hey apparently any "special" requests are gone. We now have only to deal with REQ_DEVICE_ACB and REQ_DEVICE_CMD. One of them is still too much and will be killed.

diff -urN linux-2.5.14/drivers/ide/ide.c linux/drivers/ide/ide.c
@@ -193,7 +193,7 @@
 #ifdef CONFIG_BLK_DEV_HD
 	if (ch->io_ports[IDE_DATA_OFFSET] == HD_DATA)
 		ch->noprobe = 1; /* may be overridden by ide_setup() */
-#endif /* CONFIG_BLK_DEV_HD */
+#endif
 	ch->major = ide_major[index];
 	sprintf(ch->name, "ide%d", index);
 	ch->bus_state = BUSSTATE_ON;

@@ -201,15 +201,14 @@
 	for (unit = 0; unit < MAX_DRIVES; ++unit) {
 		struct ata_device *drive = &ch->drives[unit];
 
-		drive->type			= ATA_DISK;
-		drive->select.all		= (unit<<4)|0xa0;
-		drive->channel			= ch;
-		drive->ctl			= 0x08;
-		drive->ready_stat		= READY_STAT;
-		drive->bad_wstat		= BAD_W_STAT;
-		drive->special_cmd = (ATA_SPECIAL_RECALIBRATE | ATA_SPECIAL_GEOMETRY);
+		drive->type		= ATA_DISK;
+		drive->select.all	= (unit<<4)|0xa0;
+		drive->channel		= ch;
+		drive->ctl		= 0x08;
+		drive->ready_stat	= READY_STAT;
+		drive->bad_wstat	= BAD_W_STAT;
 		sprintf(drive->name, "hd%c", 'a' + (index * MAX_DRIVES) + unit);
-		drive->max_failures		= IDE_DEFAULT_MAX_FAILURES;
+		drive->max_failures	= IDE_DEFAULT_MAX_FAILURES;
 
 		init_waitqueue_head(&drive->wqueue);
 	}

@@ -354,11 +353,8 @@
 	spin_unlock_irqrestore(&ide_lock, flags);
 }
 
-static void ata_pre_reset(struct ata_device *drive)
+static void check_crc_errors(struct ata_device *drive)
 {
-	if (ata_ops(drive) && ata_ops(drive)->pre_reset)
-		ata_ops(drive)->pre_reset(drive);
-
 	if (!drive->using_dma)
 	    return;
 

@@ -392,38 +388,6 @@
 	return ~0UL;
 }
 
-/*
- * This is used to issue WIN_SPECIFY, WIN_RESTORE, and WIN_SETMULT commands to
- * a drive.
- */
-static ide_startstop_t ata_special(struct ata_device *drive)
-{
-	unsigned char special_cmd = drive->special_cmd;
-
-#ifdef DEBUG
-	printk("%s: ata_special: 0x%02x\n", drive->name, special_cmd);
-#endif
-	if (special_cmd & ATA_SPECIAL_TUNE) {
-		drive->special_cmd &= ~ATA_SPECIAL_TUNE;
-		if (drive->channel->tuneproc != NULL)
-			drive->channel->tuneproc(drive, drive->tune_req);
-	} else if (drive->driver != NULL) {
-		if (ata_ops(drive)->special)
-			return ata_ops(drive)->special(drive);
-		else {
-			drive->special_cmd = 0;
-			drive->mult_req = 0;
-
-			return ide_stopped;
-		}
-	} else if (special_cmd) {
-		printk("%s: bad special flag: 0x%02x\n", drive->name, special_cmd);
-		drive->special_cmd = 0;
-	}
-
-	return ide_stopped;
-}
-
 extern struct block_device_operations ide_fops[];
 
 /*

@@ -460,24 +424,24 @@
  */
 static ide_startstop_t atapi_reset_pollfunc(struct ata_device *drive, struct request *__rq)
 {
-	ide_hwgroup_t *hwgroup = HWGROUP(drive);
-	byte stat;
+	struct ata_channel *ch = drive->channel;
+	u8 stat;
 
-	SELECT_DRIVE(drive->channel,drive);
+	SELECT_DRIVE(ch,drive);
 	udelay (10);
 
 	if (OK_STAT(stat=GET_STAT(), 0, BUSY_STAT)) {
 		printk("%s: ATAPI reset complete\n", drive->name);
 	} else {
-		if (time_before(jiffies, hwgroup->poll_timeout)) {
+		if (time_before(jiffies, ch->poll_timeout)) {
 			ide_set_handler (drive, atapi_reset_pollfunc, HZ/20, NULL);
 			return ide_started;	/* continue polling */
 		}
-		hwgroup->poll_timeout = 0;	/* end of polling */
+		ch->poll_timeout = 0;	/* end of polling */
 		printk("%s: ATAPI reset timed-out, status=0x%02x\n", drive->name, stat);
 		return do_reset1 (drive, 1);	/* do it the old fashioned way */
 	}
-	hwgroup->poll_timeout = 0;	/* done polling */
+	ch->poll_timeout = 0;	/* done polling */
 
 	return ide_stopped;
 }

@@ -489,19 +453,18 @@
  */
 static ide_startstop_t reset_pollfunc(struct ata_device *drive, struct request *__rq)
 {
-	ide_hwgroup_t *hwgroup = HWGROUP(drive);
-	struct ata_channel *hwif = drive->channel;
+	struct ata_channel *ch = drive->channel;
 	u8 stat;
 
 	if (!OK_STAT(stat=GET_STAT(), 0, BUSY_STAT)) {
-		if (time_before(jiffies, hwgroup->poll_timeout)) {
+		if (time_before(jiffies, ch->poll_timeout)) {
 			ide_set_handler(drive, reset_pollfunc, HZ/20, NULL);
 			return ide_started;	/* continue polling */
 		}
-		printk("%s: reset timed-out, status=0x%02x\n", hwif->name, stat);
+		printk("%s: reset timed-out, status=0x%02x\n", ch->name, stat);
 		drive->failures++;
 	} else  {
-		printk("%s: reset: ", hwif->name);
+		printk("%s: reset: ", ch->name);
 		if ((stat = GET_ERR()) == 1) {
 			printk("success\n");
 			drive->failures = 0;

@@ -531,7 +494,8 @@
 			drive->failures++;
 		}
 	}
-	hwgroup->poll_timeout = 0;	/* done polling */
+	ch->poll_timeout = 0;	/* done polling */
+
 	return ide_stopped;
 }
 

@@ -555,21 +519,21 @@
 {
 	unsigned int unit;
 	unsigned long flags;
-	struct ata_channel *hwif = drive->channel;
-	ide_hwgroup_t *hwgroup = HWGROUP(drive);
+	struct ata_channel *ch = drive->channel;
 
 	__save_flags(flags);	/* local CPU only */
 	__cli();		/* local CPU only */
 
 	/* For an ATAPI device, first try an ATAPI SRST. */
 	if (drive->type != ATA_DISK && !do_not_try_atapi) {
-		ata_pre_reset(drive);
-		SELECT_DRIVE(hwif,drive);
+		check_crc_errors(drive);
+		SELECT_DRIVE(ch, drive);
 		udelay (20);
 		OUT_BYTE(WIN_SRST, IDE_COMMAND_REG);
-		hwgroup->poll_timeout = jiffies + WAIT_WORSTCASE;
+		ch->poll_timeout = jiffies + WAIT_WORSTCASE;
 		ide_set_handler(drive, atapi_reset_pollfunc, HZ/20, NULL);
 		__restore_flags(flags);	/* local CPU only */
+
 		return ide_started;
 	}
 

@@ -578,11 +542,12 @@
 	 * for any of the drives on this interface.
 	 */
 	for (unit = 0; unit < MAX_DRIVES; ++unit)
-		ata_pre_reset(&hwif->drives[unit]);
+		check_crc_errors(&ch->drives[unit]);
 
 #if OK_TO_RESET_CONTROLLER
 	if (!IDE_CONTROL_REG) {
 		__restore_flags(flags);
+
 		return ide_stopped;
 	}
 	/*

@@ -601,7 +566,7 @@
 		OUT_BYTE(drive->ctl|2,IDE_CONTROL_REG);	/* clear SRST, leave nIEN */
 	}
 	udelay(10);			/* more than enough time */
-	hwgroup->poll_timeout = jiffies + WAIT_WORSTCASE;
+	ch->poll_timeout = jiffies + WAIT_WORSTCASE;
 	ide_set_handler(drive, reset_pollfunc, HZ/20, NULL);
 
 	/*

@@ -609,9 +574,10 @@
 	 * state when the disks are reset this way. At least, the Winbond
 	 * 553 documentation says that
 	 */
-	if (hwif->resetproc != NULL)
-		hwif->resetproc(drive);
+	if (ch->resetproc != NULL)
+		ch->resetproc(drive);
 
+	/* FIXME: we should handle mulit mode setting here as well ! */
 #endif
 
 	__restore_flags (flags);	/* local CPU only */

@@ -789,6 +755,36 @@
 	}
 }
 
+#ifdef CONFIG_BLK_DEV_PDC4030
+# define IS_PDC4030_DRIVE (drive->channel->chipset == ide_pdc4030)
+#else
+# define IS_PDC4030_DRIVE (0)	/* auto-NULLs out pdc4030 code */
+#endif
+
+/*
+ * We are still on the old request path here so issuing the recalibrate command
+ * directly should just work.
+ */
+static int do_recalibrate(struct ata_device *drive)
+{
+	printk(KERN_INFO "%s: recalibrating!\n", drive->name);
+
+	if (drive->type != ATA_DISK)
+		return ide_stopped;
+
+	if (!IS_PDC4030_DRIVE) {
+		struct ata_taskfile args;
+
+		memset(&args, 0, sizeof(args));
+		args.taskfile.sector_count = drive->sect;
+		args.taskfile.command = WIN_RESTORE;
+		args.handler = recal_intr;
+		ata_taskfile(drive, &args, NULL);
+	}
+
+	return IS_PDC4030_DRIVE ? ide_stopped : ide_started;
+}
+
 /*
  * Take action based on the error returned by the drive.
  */

@@ -835,13 +831,11 @@
 		else
 			ide_end_request(drive, rq, 0);
 	} else {
-		if ((rq->errors & ERROR_RESET) == ERROR_RESET) {
-			++rq->errors;
+		++rq->errors;
+		if ((rq->errors & ERROR_RESET) == ERROR_RESET)
 			return do_reset1(drive, 0);
-		}
 		if ((rq->errors & ERROR_RECAL) == ERROR_RECAL)
-			drive->special_cmd |= ATA_SPECIAL_RECALIBRATE;
-		++rq->errors;
+			return do_recalibrate(drive);
 	}
 	return ide_stopped;
 }

@@ -960,7 +954,7 @@
 		goto kill_rq;
 	}
 
-	block    = rq->sector;
+	block = rq->sector;
 
 	/* Strange disk manager remap.
 	 */

@@ -991,14 +985,6 @@
 		}
 	}
 
-	/* FIXME: We can see nicely here that all commands should be submitted
-	 * through the request queue and that the special field in drive should
-	 * go as soon as possible!
-	 */
-
-	if (drive->special_cmd)
-		return ata_special(drive);
-
 	/* This issues a special drive command, usually initiated by ioctl()
 	 * from the external hdparm program.
 	 */

@@ -1011,7 +997,7 @@
 		ata_taskfile(drive, args, NULL);
 
 		if (((args->command_type == IDE_DRIVE_TASK_RAW_WRITE) ||
-					(args->command_type == IDE_DRIVE_TASK_OUT)) &&
+		     (args->command_type == IDE_DRIVE_TASK_OUT)) &&
 				args->prehandler && args->handler)
 			return args->prehandler(drive, rq);
 

@@ -1053,6 +1039,7 @@
 			return ata_ops(drive)->do_request(drive, rq, block);
 		else {
 			ide_end_request(drive, rq, 0);
+
 			return ide_stopped;
 		}
 	}

@@ -1499,7 +1486,7 @@
 			disable_irq(ch->irq);	/* disable_irq_nosync ?? */
 #endif
 			__cli();	/* local CPU only, as if we were handling an interrupt */
-			if (hwgroup->poll_timeout != 0) {
+			if (ch->poll_timeout != 0) {
 				startstop = handler(drive, ch->hwgroup->rq);
 			} else if (drive_is_ready(drive)) {
 				if (drive->waiting_for_dma)

@@ -1598,7 +1585,7 @@
 	if (!ide_ack_intr(ch))
 		goto out_lock;
 
-	if (handler == NULL || hwgroup->poll_timeout != 0) {
+	if (handler == NULL || ch->poll_timeout != 0) {
 #if 0
 		printk(KERN_INFO "ide: unexpected interrupt %d %d\n", ch->unit, irq);
 #endif

@@ -2327,23 +2314,24 @@
 int ide_spin_wait_hwgroup(struct ata_device *drive)
 {
 	ide_hwgroup_t *hwgroup = HWGROUP(drive);
-	unsigned long timeout = jiffies + (3 * HZ);
+
+	/* FIXME: Wait on a proper timer. Instead of playing games on the
+	 * spin_lock().
+	 */
+
+	unsigned long timeout = jiffies + (10 * HZ);
 
 	spin_lock_irq(&ide_lock);
 
 	while (test_bit(IDE_BUSY, &hwgroup->flags)) {
-		unsigned long lflags;
 		spin_unlock_irq(&ide_lock);
-		__save_flags(lflags);	/* local CPU only */
-		__sti();		/* local CPU only; needed for jiffies */
-		if (0 < (signed long)(jiffies - timeout)) {
-			__restore_flags(lflags);	/* local CPU only */
+		if (time_after(jiffies, timeout)) {
 			printk("%s: channel busy\n", drive->name);
 			return -EBUSY;
 		}
-		__restore_flags(lflags);	/* local CPU only */
 		spin_lock_irq(&ide_lock);
 	}
+
 	return 0;
 }
 

@@ -2411,18 +2399,17 @@
 
 static int set_pio_mode(struct ata_device *drive, int arg)
 {
-	struct request rq;
-
 	if (!drive->channel->tuneproc)
 		return -ENOSYS;
 
-	if (drive->special_cmd & ATA_SPECIAL_TUNE)
+	/* FIXME: This is very much the same kind of problem as we have with
+	 * set_mutlmode() see for a edscription there.
+	 */
+	if (HWGROUP(drive)->handler)
 		return -EBUSY;
 
-	ide_init_drive_cmd(&rq);
-	drive->tune_req = (u8) arg;
-	drive->special_cmd |= ATA_SPECIAL_TUNE;
-	ide_do_drive_cmd(drive, &rq, ide_wait);
+	if (drive->channel->tuneproc != NULL)
+		drive->channel->tuneproc(drive, (u8) arg);
 
 	return 0;
 }

diff -urN linux-2.5.14/drivers/ide/ide-tape.c linux/drivers/ide/ide-tape.c
@@ -4273,16 +4273,6 @@
 }
 
 /*
- *	idetape_pre_reset is called before an ATAPI/ATA software reset.
- */
-static void idetape_pre_reset (ide_drive_t *drive)
-{
-	idetape_tape_t *tape = drive->driver_data;
-	if (tape != NULL)
-		set_bit (IDETAPE_IGNORE_DSC, &tape->flags);
-}
-
-/*
  *	Character device interface functions
  */
 static ide_drive_t *get_drive_ptr (kdev_t i_rdev)

@@ -6164,8 +6154,6 @@
 	release:		idetape_blkdev_release,
 	check_media_change:	NULL,
 	revalidate:		idetape_revalidate,
-	pre_reset:		idetape_pre_reset,
-	capacity:		NULL,
 	proc:			idetape_proc
 };
 

diff -urN linux-2.5.14/drivers/ide/tcq.c linux/drivers/ide/tcq.c
@@ -418,7 +418,7 @@
 	 * pass NOP with sub-code 0x01 to device, so the command will not
 	 * fail there
 	 */
-	ide_raw_taskfile(drive, &args, NULL);
+	ide_raw_taskfile(drive, &args);
 	if (args.taskfile.feature & ABRT_ERR)
 		return 1;
 

@@ -448,7 +448,7 @@
 	args.taskfile.command = WIN_SETFEATURES;
 	ide_cmd_type_parser(&args);
 
-	if (ide_raw_taskfile(drive, &args, NULL)) {
+	if (ide_raw_taskfile(drive, &args)) {
 		printk("%s: failed to enable write cache\n", drive->name);
 		return 1;
 	}

@@ -462,7 +462,7 @@
 	args.taskfile.command = WIN_SETFEATURES;
 	ide_cmd_type_parser(&args);
 
-	if (ide_raw_taskfile(drive, &args, NULL)) {
+	if (ide_raw_taskfile(drive, &args)) {
 		printk("%s: disabling release interrupt fail\n", drive->name);
 		return 1;
 	}

@@ -476,7 +476,7 @@
 	args.taskfile.command = WIN_SETFEATURES;
 	ide_cmd_type_parser(&args);
 
-	if (ide_raw_taskfile(drive, &args, NULL)) {
+	if (ide_raw_taskfile(drive, &args)) {
 		printk("%s: enabling service interrupt fail\n", drive->name);
 		return 1;
 	}

diff -urN linux-2.5.14/include/linux/ide.h linux/include/linux/ide.h
@@ -47,7 +47,7 @@
 # define DISK_RECOVERY_TIME	0	/*  for hardware that needs it */
 #endif
 #ifndef OK_TO_RESET_CONTROLLER		/* 1 needed for good error recovery */
-# define OK_TO_RESET_CONTROLLER	1	/* 0 for use with AH2372A/B interface */
+# define OK_TO_RESET_CONTROLLER	0	/* 0 for use with AH2372A/B interface */
 #endif
 #ifndef FANCY_STATUS_DUMPS		/* 1 for human-readable drive errors */
 # define FANCY_STATUS_DUMPS	1	/* 0 to reduce kernel size */

@@ -327,19 +327,9 @@
 	 */
 	request_queue_t	queue;	/* per device request queue */
 
-
 	unsigned long sleep;	/* sleep until this time */
 
-	/* Flags requesting/indicating one of the following special commands
-	 * executed on the request queue.
-	 */
-#define ATA_SPECIAL_GEOMETRY		0x01
-#define ATA_SPECIAL_RECALIBRATE		0x02
-#define ATA_SPECIAL_MMODE		0x04
-#define ATA_SPECIAL_TUNE		0x08
-	unsigned char special_cmd;
-	u8 mult_req;			/* requested multiple sector setting */
-	u8 tune_req;			/* requested drive tuning setting */
+	u8 XXX_tune_req;			/* requested drive tuning setting */
 
 	byte     using_dma;		/* disk is using dma for read/write */
 	byte	 using_tcq;		/* disk is using queueing */

@@ -409,6 +399,7 @@
 	unsigned int	failures;	/* current failure count */
 	unsigned int	max_failures;	/* maximum allowed failure count */
 	struct device	device;		/* global device tree handle */
+
 	/*
 	 * tcq statistics
 	 */

@@ -517,6 +508,8 @@
 	/* driver soft-power interface */
 	int (*busproc)(struct ata_device *, int);
 	byte		bus_state;	/* power state of the IDE bus */
+
+	unsigned long poll_timeout; /* timeout value during polled operations */
 };
 
 /*

@@ -565,17 +558,19 @@
 	return 1;
 }
 #else
-#define ata_pending_commands(drive)	(0)
-#define ata_can_queue(drive)		(1)
+# define ata_pending_commands(drive)	(0)
+# define ata_can_queue(drive)		(1)
 #endif
 
 typedef struct hwgroup_s {
+	/* FIXME: We should look for busy request queues instead of looking at
+	 * the !NULL state of this field.
+	 */
 	ide_startstop_t (*handler)(struct ata_device *, struct request *);	/* irq handler, if active */
 	unsigned long flags;		/* BUSY, SLEEPING */
 	struct ata_device *XXX_drive;	/* current drive */
 	struct request *rq;		/* current request */
 	struct timer_list timer;	/* failsafe timer */
-	unsigned long poll_timeout;	/* timeout value during long polls */
 	int (*expiry)(struct ata_device *, struct request *);	/* irq handler, if active */
 } ide_hwgroup_t;
 

@@ -675,9 +670,7 @@
 	int (*check_media_change)(struct ata_device *);
 	void (*revalidate)(struct ata_device *);
 
-	void (*pre_reset)(struct ata_device *);
 	sector_t (*capacity)(struct ata_device *);
-	ide_startstop_t	(*special)(struct ata_device *);
 
 	ide_proc_entry_t *proc;
 };

@@ -827,15 +820,13 @@
  */
 
 extern ide_startstop_t recal_intr(struct ata_device *, struct request *);
-extern ide_startstop_t set_geometry_intr(struct ata_device *, struct request *);
-extern ide_startstop_t set_multmode_intr(struct ata_device *, struct request *);
 extern ide_startstop_t task_no_data_intr(struct ata_device *, struct request *);
 
 
 /* This is setting up all fields in args, which depend upon the command type.
  */
 extern void ide_cmd_type_parser(struct ata_taskfile *args);
-extern int ide_raw_taskfile(struct ata_device *drive, struct ata_taskfile *cmd, byte *buf);
+extern int ide_raw_taskfile(struct ata_device *, struct ata_taskfile *);
 extern int ide_cmd_ioctl(struct ata_device *drive, unsigned long arg);
 
 void ide_delay_50ms(void);

Tue May 7 02:37:49 CEST 2002 ide-clean-57

Nuke /proc/ide. For explanations why, please see the frustrated comments in the previous change log. If one still don't see why it wasn't a good thing, well please just take a look at the following:

Kernel size before:

/usr/src/linux# size vmlinux
text data bss dec hex filename
1716049 403968 470252 2590269 27863d vmlinux
/usr/src/linux#

Kernel size after:

/usr/src/linux# size vmlinux
text data bss dec hex filename
1680993 403488 470124 2554605 26faed vmlinux
/usr/src/linux#

2% of overall size! And this is not exactly an minimalistic setup. Wow! What a waste of space!!!! Not even counting the runtime size of this crap! And then let's take a look at the following self flattery:

-/*
- * Copyright (C) 1997-1998 Mark Lord
- *
- * This is the /proc/ide/ filesystem implementation.
- *
- * The major reason this exists is to provide sufficient access
- * to driver and config data, such that user-mode programs can
- * be developed to handle chipset tuning for most PCI interfaces.
- * This should provide better utilities, and less kernel bloat.

                                              ^^^^^^^^^^^^^^^^^^

Well there could only be an answer to this which would be universally understandable in every Slavic language... but since it's mothers day...

EOD.

diff -urN linux-2.5.14/drivers/ide/aec62xx.c linux/drivers/ide/aec62xx.c
@@ -26,7 +26,7 @@
 
 #include "ata-timing.h"
 
-#define DISPLAY_AEC62XX_TIMINGS
+#undef DISPLAY_AEC62XX_TIMINGS
 
 #ifndef HIGH_4
 #define HIGH_4(H)		((H)=(H>>4))

@@ -503,7 +503,7 @@
 		bmide_dev = dev;
 		aec62xx_display_info = &aec62xx_get_info;
 	}
-#endif /* DISPLAY_AEC62XX_TIMINGS && CONFIG_PROC_FS */
+#endif
 
 	return dev->irq;
 }

diff -urN linux-2.5.14/drivers/ide/alim15x3.c linux/drivers/ide/alim15x3.c
@@ -28,7 +28,7 @@
 
 #include "ata-timing.h"
 
-#define DISPLAY_ALI_TIMINGS
+#undef DISPLAY_ALI_TIMINGS
 
 #if defined(DISPLAY_ALI_TIMINGS) && defined(CONFIG_PROC_FS)
 #include <linux/stat.h>

diff -urN linux-2.5.14/drivers/ide/amd74xx.c linux/drivers/ide/amd74xx.c
@@ -94,7 +94,7 @@
  * AMD /proc entry.
  */
 
-#ifdef CONFIG_PROC_FS
+#if 0 && defined(CONFIG_PROC_FS)
 
 #include <linux/stat.h>
 #include <linux/proc_fs.h>

@@ -384,7 +384,7 @@
  * Register /proc/ide/amd74xx entry
  */
 
-#ifdef CONFIG_PROC_FS
+#if 0 && defined(CONFIG_PROC_FS)
 	if (!amd74xx_proc) {
 		amd_base = pci_resource_start(dev, 4);
 		bmide_dev = dev;

diff -urN linux-2.5.14/drivers/ide/cmd64x.c linux/drivers/ide/cmd64x.c
@@ -79,7 +79,7 @@
 #define UDIDETCR1	0x7B
 #define DTPR1		0x7C
 
-#define DISPLAY_CMD64X_TIMINGS
+#undef DISPLAY_CMD64X_TIMINGS
 
 #if defined(DISPLAY_CMD64X_TIMINGS) && defined(CONFIG_PROC_FS)
 #include <linux/stat.h>

diff -urN linux-2.5.14/drivers/ide/cs5530.c linux/drivers/ide/cs5530.c
@@ -29,7 +29,7 @@
 
 #include "ata-timing.h"
 
-#define DISPLAY_CS5530_TIMINGS
+#undef DISPLAY_CS5530_TIMINGS
 
 #if defined(DISPLAY_CS5530_TIMINGS) && defined(CONFIG_PROC_FS)
 #include <linux/stat.h>

diff -urN linux-2.5.14/drivers/ide/hpt366.c linux/drivers/ide/hpt366.c
@@ -65,7 +65,7 @@
 
 #include "ata-timing.h"
 
-#define DISPLAY_HPT366_TIMINGS
+#undef DISPLAY_HPT366_TIMINGS
 
 /* various tuning parameters */
 #define HPT_RESET_STATE_ENGINE

diff -urN linux-2.5.14/drivers/ide/ide.c linux/drivers/ide/ide.c
@@ -1921,12 +1921,6 @@
 	return 0;
 }
 
-#ifdef CONFIG_PROC_FS
-ide_proc_entry_t generic_subdriver_entries[] = {
-	{ NULL, 0, NULL, NULL }
-};
-#endif
-
 void ide_unregister(struct ata_channel *ch)
 {
 	struct gendisk *gd;

@@ -1983,9 +1977,6 @@
 			}
 		}
 	}
-#ifdef CONFIG_PROC_FS
-	destroy_proc_ide_drives(ch);
-#endif
 	spin_lock_irqsave(&ide_lock, flags);
 
 	/*

@@ -2208,9 +2199,6 @@
 	if (!initializing) {
 		ideprobe_init();
 		revalidate_drives();
-#ifdef CONFIG_PROC_FS
-		create_proc_ide_interfaces();
-#endif
 		/* FIXME: Do we really have to call it second time here?! */
 		ide_driver_module();
 	}

@@ -3198,11 +3186,7 @@
 	}
 	drive->revalidate = 1;
 	drive->suspend_reset = 0;
-#ifdef CONFIG_PROC_FS
-	ide_add_proc_entries(drive->proc, generic_subdriver_entries, drive);
-	if (ata_ops(drive))
-		ide_add_proc_entries(drive->proc, ata_ops(drive)->proc, drive);
-#endif
+
 	return 0;
 }
 

@@ -3233,11 +3217,6 @@
 #if defined(CONFIG_BLK_DEV_ISAPNP) && defined(CONFIG_ISAPNP) && defined(MODULE)
 	pnpide_init(0);
 #endif
-#ifdef CONFIG_PROC_FS
-	if (ata_ops(drive))
-		ide_remove_proc_entries(drive->proc, ata_ops(drive)->proc);
-	ide_remove_proc_entries(drive->proc, generic_subdriver_entries);
-#endif
 	auto_remove_settings(drive);
 	drive->driver = NULL;
 	drive->present = 0;

@@ -3315,10 +3294,6 @@
 EXPORT_SYMBOL(ide_cmd);
 EXPORT_SYMBOL(ide_delay_50ms);
 EXPORT_SYMBOL(ide_stall_queue);
-#ifdef CONFIG_PROC_FS
-EXPORT_SYMBOL(ide_add_proc_entries);
-EXPORT_SYMBOL(ide_remove_proc_entries);
-#endif
 EXPORT_SYMBOL(ide_add_setting);
 EXPORT_SYMBOL(ide_remove_setting);
 

@@ -3484,10 +3459,6 @@
 # endif
 #endif
 
-#ifdef CONFIG_PROC_FS
-	proc_ide_create();
-#endif
-
 	/*
 	 * Initialize all device type driver modules.
 	 */

@@ -3553,9 +3524,6 @@
 		ide_unregister(&ide_hwifs[h]);
 	}
 
-# ifdef CONFIG_PROC_FS
-	proc_ide_destroy();
-# endif
 	devfs_unregister(ide_devfs_handle);
 }
 

diff -urN linux-2.5.14/drivers/ide/ide-cd.c linux/drivers/ide/ide-cd.c
@@ -2906,7 +2906,6 @@
 	check_media_change:	ide_cdrom_check_media_change,
 	revalidate:		ide_cdrom_revalidate,
 	capacity:		ide_cdrom_capacity,
-	proc:			NULL
 };
 
 /* options */

diff -urN linux-2.5.14/drivers/ide/ide-tape.c linux/drivers/ide/ide-tape.c
@@ -6108,31 +6108,6 @@
 	return 0;
 }
 
-#ifdef CONFIG_PROC_FS
-
-static int proc_idetape_read_name
-	(char *page, char **start, off_t off, int count, int *eof, void *data)
-{
-	ide_drive_t	*drive = (ide_drive_t *) data;
-	idetape_tape_t	*tape = drive->driver_data;
-	char		*out = page;
-	int		len;
-
-	len = sprintf(out, "%s\n", tape->name);
-	PROC_IDE_READ_RETURN(page, start, off, count, eof, len);
-}
-
-static ide_proc_entry_t idetape_proc[] = {
-	{ "name",	S_IFREG|S_IRUGO,	proc_idetape_read_name,	NULL },
-	{ NULL, 0, NULL, NULL }
-};
-
-#else
-
-#define	idetape_proc	NULL
-
-#endif
-
 static void idetape_revalidate(ide_drive_t *_dummy)
 {
 	/* We don't have to handle any partition information here, which is the

@@ -6154,7 +6129,6 @@
 	release:		idetape_blkdev_release,
 	check_media_change:	NULL,
 	revalidate:		idetape_revalidate,
-	proc:			idetape_proc
 };
 
 /*

diff -urN linux-2.5.14/drivers/ide/Makefile linux/drivers/ide/Makefile
@@ -72,8 +72,6 @@
 obj-$(CONFIG_BLK_DEV_ATARAID_PDC)	+= pdcraid.o
 obj-$(CONFIG_BLK_DEV_ATARAID_HPT)	+= hptraid.o
 
-ide-obj-$(CONFIG_PROC_FS)		+= ide-proc.o
-
 ide-mod-objs		:= ide-taskfile.o ide.o ide-probe.o ide-geometry.o ide-features.o ata-timing.o $(ide-obj-y)
 
 include $(TOPDIR)/Rules.make

diff -urN linux-2.5.14/drivers/ide/pdc202xx.c linux/drivers/ide/pdc202xx.c
@@ -51,7 +51,7 @@
 #define PDC202XX_DEBUG_DRIVE_INFO		0
 #define PDC202XX_DECODE_REGISTER_INFO		0
 
-#define DISPLAY_PDC202XX_TIMINGS
+#undef DISPLAY_PDC202XX_TIMINGS
 
 #ifndef SPLIT_BYTE
 #define SPLIT_BYTE(B,H,L)	((H)=(B>>4), (L)=(B-((B>>4)<<4)))

diff -urN linux-2.5.14/drivers/ide/piix.c linux/drivers/ide/piix.c
@@ -110,7 +110,7 @@
  * PIIX/ICH /proc entry.
  */
 
-#ifdef CONFIG_PROC_FS
+#if 0 && defined(CONFIG_PROC_FS)
 
 #include <linux/stat.h>
 #include <linux/proc_fs.h>

@@ -520,7 +520,7 @@
  * Register /proc/ide/piix entry
  */
 
-#ifdef CONFIG_PROC_FS
+#if 0 && defined(CONFIG_PROC_FS)
 	if (!piix_proc) {
 		piix_base = pci_resource_start(dev, 4);
 		bmide_dev = dev;

diff -urN linux-2.5.14/drivers/ide/serverworks.c linux/drivers/ide/serverworks.c
@@ -93,15 +93,16 @@
 
 #include "ata-timing.h"
 
-#define DISPLAY_SVWKS_TIMINGS	1
+#undef DISPLAY_SVWKS_TIMINGS
 #undef SVWKS_DEBUG_DRIVE_INFO
 
+static u8 svwks_revision = 0;
+
 #if defined(DISPLAY_SVWKS_TIMINGS) && defined(CONFIG_PROC_FS)
 #include <linux/stat.h>
 #include <linux/proc_fs.h>
 
 static struct pci_dev *bmide_dev;
-static byte svwks_revision = 0;
 
 static int svwks_get_info(char *, char **, off_t, int);
 extern int (*svwks_display_info)(char *, char **, off_t, int); /* ide-proc.c */

diff -urN linux-2.5.14/drivers/ide/sis5513.c linux/drivers/ide/sis5513.c
@@ -58,7 +58,7 @@
 /* When BROKEN_LEVEL is defined it limits the DMA mode
    at boot time to its value */
 // #define BROKEN_LEVEL XFER_SW_DMA_0
-#define DISPLAY_SIS_TIMINGS
+#undef DISPLAY_SIS_TIMINGS
 
 /* Miscellaneaous flags */
 #define SIS5513_LATENCY		0x01

diff -urN linux-2.5.14/drivers/ide/via82cxxx.c linux/drivers/ide/via82cxxx.c
@@ -136,7 +136,7 @@
  * VIA /proc entry.
  */
 
-#ifdef CONFIG_PROC_FS
+#if 0 && defined(CONFIG_PROC_FS)
 
 #include <linux/stat.h>
 #include <linux/proc_fs.h>

@@ -497,7 +497,7 @@
  * Setup /proc/ide/via entry.
  */
 
-#ifdef CONFIG_PROC_FS
+#if 0 && defined(CONFIG_PROC_FS)
 	if (!via_proc) {
 		via_base = pci_resource_start(dev, 4);
 		bmide_dev = dev;

diff -urN linux-2.5.14/drivers/scsi/ide-scsi.c linux/drivers/scsi/ide-scsi.c
@@ -557,7 +557,6 @@
 	check_media_change:	NULL,
 	revalidate:		idescsi_revalidate,
 	capacity:		NULL,
-	proc:			NULL
 };
 
 /*

This is a big mistake IMO.

Nuking the ability to change settings, fair enough, but only if alternative interface is provided for userspace to tweak everything, otherwise provide the interface before you remove the existing one. (There may be already another interface, I don't know...I am sure someone will tell me if there is!)

Removing the information provided by /proc/ide is very bad! It is very useful to diagnose one's ide setup, to see what the host is configured as, what all settings are set to, etc. This is the first place I look to check whether the interfaces are configured as I expect them to be and in case of problems, this is again the first place I look.

What alternatives are you going to present to give all the information that /proc/ide gives? If the answer is none IMHO your patch is not acceptable...

Best regards,

Anton

  "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Uz.ytkownik Anton Altaparmakov napisa?:

Ehmm... There *is* one interface there. hdparm will help you. Note: the upcomming release of hdparm should contain the following patch which incearses it's usability vastly to the average user. Just for convenience I'm attaching it here.

If you don't like hdparm - well please shoot the people who wrote init, ifconfig, eject and so on...

diff -ur hdparm-4.9/hdparm.8 hdparm-4.9-new/hdparm.8
@@ -368,14 +368,16 @@
 Tristate device for hotswap (DANGEROUS).
 .TP
 .I -X 
-Set the IDE transfer mode for newer (E)IDE/ATA2 drives.
+Set the IDE transfer mode for newer (E)IDE/ATA-7 drives.
 This is typically used in combination with
 .I -d1
 when enabling DMA to/from a drive on a supported interface chipset, where
-.I -X34
-is used to select multiword DMA mode2 transfers.
+.I -X mdma2
+is used to select multiword DMA mode2 transfers and
+.I -X sdma1 
+is used to select simple mode 1 DMA transfers.
 With systems which support UltraDMA burst timings,
-.I -X66
+.I -X udma2
 is used to select UltraDMA mode2 transfers (you'll need to prepare
 the chipset for UltraDMA beforehand).
 Apart from that, use of this flag is

hdparm -i requires root privs. cat /proc/ide/${file} does not. hdparm is NOT an acceptable substitute for /proc/ide/.

/Mikael

How do I get this information with hdparm please?

[aia21@drop ide]$ cat via
----------VIA BusMastering IDE Configuration----------------

Driver Version:                     3.34
South Bridge:                       VIA vt82c686b
Revision:                           ISA 0x40 IDE 0x6
Highest DMA rate:                   UDMA100
BM-DMA base:                        0xd000
PCI clock:                          33.3MHz
Master Read  Cycle IRDY:            0ws
Master Write Cycle IRDY:            0ws
BM IDE Status Register Read Retry:  yes
Max DRDY Pulse Width:               No limit
-----------------------Primary IDE-------Secondary IDE------
Read DMA FIFO flush:          yes                 yes
End Sector FIFO flush:         no                  no
Prefetch Buffer:              yes                  no
Post Write Buffer:            yes                  no
Enabled:                      yes                 yes
Simplex only:                  no                  no
Cable Type:                   80w                 40w
-------------------drive0----drive1----drive2----drive3-----
Transfer Mode:       UDMA       PIO       DMA      UDMA
Address Setup:       30ns     120ns      30ns      30ns
Cmd Active:          90ns      90ns      90ns      90ns
Cmd Recovery:        30ns      30ns      30ns      30ns
Data Active:         90ns     330ns      90ns      90ns
Data Recovery:       30ns     270ns      30ns      30ns
Cycle Time:          20ns     600ns     120ns      60ns
Transfer Rate:   99.9MB/s   3.3MB/s  16.6MB/s  33.3MB/s

hdparm is a tool to query a device and how the controller is programmed to talk to the device. But it is not designed nor capable of giving information about the host itself. I just read the man page for hdparm and there are no options in sight to show any of the things I have shown above.

Also the below work as normal user but hdparm requires super user... It is debateable whether a normal user should be allowed access but still you are taking away existing functionality...

[aia21@drop hda]$ cat cache
1916
[aia21@drop hda]$ cat capacity
80418240
[aia21@drop hda]$ cat geometry
physical 79780/16/63
logical 5005/255/63

And hdparm never gives you the physical geometry AFAICS.

Either I am missing something or you are removing a lot of functionality and replacing it with nothingness...

And as I said, I can understand removing the ability to write values into /proc/ide/*, what I disagree with is the removal of the information provided by read-only access to /proc/ide/*. And that is because I am not aware of any other way to get the same information.

Best regards,

Anton

  "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Uz.ytkownik Anton Altaparmakov napisa?:

http://www.j2.ru/frozenfido/ru.unix.bsd/1329707b3e3f8.html

Porting it should be fairly tirvial. Basically lspci + the parsing crap.

Feel free to do it yourself - in user space where it belongs.

I don't want to port anything. I don't know ide and I don't want to know ide. I want to be able to use it. I am an ide USER. You are the ide DEVELOPER. If you take away functionality YOU have to provide a replacement. NOT tell me, the USER to write it.

I don't want to do it myself. I want YOU to do it because YOU are taking away the functionality that already exists.

Anton

  "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Can't find where I got it from, and it seems to have fallen off google. I put up the last version I had (which I hacked up a bit) at
http://www.codemonkey.org.uk/cruft/ide-info-0.0.5-dj.tar.gz

must.. resist.. /proc ascii/bin... holywar.. (besides, sysctl interface gives you ascii in /proc/sys/)

That's likely exactly the reason we ended up with the dungheap we have now. Rewriting the parser when we already have a usable sysctl interface seems to have no gain over the existing mess to me.

Dave.

| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Ok, will get that. Someone else emailed me a url and I tried that earlier on (ages ago it seems) it said version 0.0.4 and it displayed a lot of crap on a 2.5.14 running kernel. Certainly it bears no resemblance to what /proc/ide/via has to say and it certainly bears no resemblance to reality... )-: i hope...

It does indeed (if implemented). Agreed if Martin were to change to sysctl with /proc interface great, it would just mean /proc/ide becomes /proc/sys/ide, nothing against that....

Probably... I agree sysctl is great. I use it in ntfs myself. (-: And i think the /proc/sys is very nice... And for people who don't like it or who don;'t compile /proc fs they can use _sysctl...

Cheers,

Anton

  "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

[snip]

Please consider this carefully, especially the read only bits. One particular thing I use a lot is: /proc/ide/hda/capacity Will there be another interface easily usable by scripts to get this information?

Am I going to have to parse hdparm output?
....
geometry = 2434/255/63, sectors = 39102336, start = 0

Am I going to need hdparm on my embedded system?

Padraig.

hdparm itself doesn't, but you must be able to read /dev/hd* Some distros have this owned by group 'disk' for eg. Adding yourself as a member of this group should allow you to use hdparm without becoming root.

| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Bartlomiej Zolnierkiewicz moved all this stuff to userspace a long time ago in 'ideinfo'.

Why would a normal user ever need to know this info?

The parsing gunk we have for /proc/ide is fugly, and should have been done with sysctls from day one imo.

| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

[aia21@drop hda]$ ideinfo
bash: ideinfo: command not found

Obviously distros haven't caught up with this development. )-:

Care to give me a URL? A quick google for "ideinfo Linux download" didn't bring up anything looking relevant.

I want to know this info. (-: Admittedly normal users don't need it... It is useful for diagnosing problems with NTFS and MD setups for example (in conjunction with fdisk -l shown in sectors).

I like text parsing... It is not performance critical and makes info human readable... Whether existing text parsers are any good or not, I don't care, write a better one if you don't like the existing one or go beat up the people who wrote the bad ones... That seems to be Martin's standard reply, so I thought I would use it, too. (-;

Best regards,

Anton

  "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

I don't think 0.0.5 actually hit the streets, I just named it that as this one contained something or other I did (can't remember what exactly diff will tell you) that I was intended to send to the author for 0.0.5

Likely it needs an update for the newer VIA chipsets, as this code is ~2 years old. What it does do however, is proove that this doesn't need to be done in kernel space.

Dave.

| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

vaio:~ # hdparm -i /dev/hda

/dev/hda:

Model=FUJITSU MHJ2181AT, FwRev=D034, SerialNo=01001697 Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs } RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=4 BuffType=unknown, BuffSize=512kB, MaxMultSect=16, MultSect=16 CurCHS=17475/15/63, CurSects=16513875, LBA=yes, LBAsects=35433216 IORDY=yes, tPIO={min:240,w/IORDY:120}, tDMA={min:120,rec:120} PIO modes: pio0 pio1 pio2 pio3 pio4
DMA modes: mdma0 mdma1 mdma2 udma0 udma1 *udma2 udma3 udma4 Drive Supports : ATA-2 ATA-3 ATA-4 ATA-5
Kernel Drive Geometry LogicalCHS=2205/255/63 PhysicalCHS=37495/15/63

BS Dave it does parse the difference nicely

Andre Hedrick
LAD Storage Consulting Group

Uz.ytkownik Dave Jones napisa?:

Amen. For where it turn outs to be really really worth it I indeed plan to move to sysctl. For example currently we have on ioctl level still the problem that many of them are attached to the device but act on the channel.

hdparm -xxx /dev/hda & hdparm -xxx /dev/hdc - BANG race condition. (At least on the level of logics).

Uz.ytkownik Padraig Brady napisa?:

Yes. Or just fsck hardcode the defaults.

This is stupid! And if that isn't obvious to you, you should think a bit more carefully...

Linux's power is exactly that it can be used on anything from a wristwatch to a huge server and that it is flexible about everything. You are breaking this flexibility for no apparent reason. (I don't accept "I can't cope with this so I remove it." as a reason, sorry).

As the new IDE maintainer so far we have only seen you removing one feature after the other in the name of cleanup, without adequate or even any at all(!) replacements, renaming all functions to hell and back, and breaking the ide core here there and everywhere. All critical bug fixes seem to have been contributed by other people looking at your code which doesn't inspire a lot of confidence in you... Even Alan Cox said a while ago that you have his vote of no confidence (probably slightly rephrased here) because of changes you were introducing and I tend to trust bearded kernel hackers from Whales. (-;

Aren't you noticing that something is wrong here???

Best regards,

Anton

  "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

[ First off: any IDE-only thing that doesn't work for SCSI or other disks
doesn't solve a generic problem, so the complaint that some generic tools might use it is totally invalid. ]

Run the 57 patch, and complain if something doesn't work.

Linux's power is that we FIX stuff. That we make it the best system possible, and that we don't just whine and argue about things.

Who cares? Have you found _anything_ that Martin removed that was at all worthwhile? I sure haven't.

Guys, you have to realize that the IDE layer has eight YEARS of absolute crap in it. Seriously. It's _never_ been cleaned up before. It has stuff so distasteful that t's scary.

Take it from me: it's a _lot_ easier to add cruft and crap on top of clean code. You can do it yourself if you want to. You don't need a maintainer to add barnacles.

All the information that /proc/ide gave you is basically available in hdparm, and for your dear embedded system it apparently takes up less space by being in user space. So what is the problem?

My vote is to remove as much as humanly possible.

"Everything should be made as simple as possible, but not
simpler" - Albert Einstein

Think about it, and really _understand_ it.

Linus

Well my "dear" embedded system doesn't have libc :-(
So 35664 saved in kernel (less on disk), requires 25212 extra for hdparm + more for static linked uclibc (hope it works ;-)). As a side note if this happens hdparm would be a requirement for busybox IMHO, anyway getting back on topic...

All the info I've ever needed is /proc/ide/hdx/capacity which I could get from /proc/partitions with more a bit more effort, so I vote for removing /proc/ide.

I think everyone realises Martin is doing great and much needed work on IDE (btw I'll have those flash support patches soon Martin ;-)), but I did think this change needed debate. In general I know it's a hard decision what to export in proc, especially if there are existing dependencies, a few already mentioned possibles in RH7.1:

/sbin/mkinitrd
/sbin/fdisk
/sbin/sfdisk
/sbin/sndconfig
/usr/sbin/mouseconfig
/usr/sbin/kudzu
/usr/sbin/module_upgrade
/usr/sbin/updfstab
/usr/sbin/glidelink
/usr/sbin/sndconfig
/usr/lib/python1.5/site-packages/_kudzumodule.so
/usr/bin/X11/Xconfigurator

For e.g. could the same arguments could be made for lspci only interface to pci info rather than /proc/bus/pci? The following references are made to /proc/bus/pci on my system:

/sbin/lspci
/sbin/setpci
/sbin/sndconfig
/usr/sbin/mouseconfig
/usr/sbin/kudzu
/usr/sbin/module_upgrade
/usr/sbin/updfstab
/usr/sbin/glidelink
/usr/sbin/sndconfig
/usr/sbin/adsl-config
/usr/sbin/internet-config
/usr/sbin/isdn-config
/usr/lib/python1.5/site-packages/_kudzumodule.so
/usr/bin/X11/XFree86
/usr/bin/X11/pcitweak
/usr/bin/X11/scanpci
/usr/bin/X11/Xconfigurator

cheers,
Padraig.

/proc/ide has useful information in it that you can't get easily by other means at the moment - which controller is driving the disks, what devices are present etc.

lspci relies on /proc/bus/pci - its the only part of the universe that actually knows how to handle PCI and virtualised PCI devices. Unlike the older /proc/pci interface it keeps all the complex gunk out of the kernel

I'd love for somebody to add the devices to the real device tree, at which point this kind of information would be very much visible..

Right now devicefs isn't even mounted by default, but it's the only _really_ generic way of showing things like this that we have. For people who haven't seen it before, do a

mount -t driverfs /devfs /devfs

and go look in there.. In particular, if you have a PCI system with a USB device tree (or _multiple_ such trees), notice how you can look at things like

	/driverfs/root/pci0/00:1f.4/usb_bus/000/

and it wouldn't be impossible (or even necessarily very hard) to make an IDE controller export the "IDE device tree" the same way a USB controller now exports the "USB device tree".

For things like hotplug etc, I think driverfs is eventually the only way to go, simply because it gives you the full (and unambiguous) path to _any_ device, and is completely bus-agnostic.

But there is definitely a potential backwards-compatibility-issue.

Linus

One interesting thing here would be to have some optional link between the bus-oriented device tree and the function-oriented tree (ie. devfs or simply /dev). For example, an IDE node in driverfs could eventually hold symlinks to the entries it provides in /dev when using devfs (or just provide major/minor when not using devfs).

What do you think ?

One problem I've been faced with on ppc is to be able to match a linux device with what the firmware (Open Firmware) thinks that device is. The firmware view is bus-centered and it would be pretty easy to provide some additional entries in driverfs that give the OF fullpath of a given device. But then, the link between the actual driver in driverfs and the "device" as used by, for example, the filesystem isn't trivial.

Ben.

There isn't any 1:1 thing - the device/bus-oriented one should _not_ show virtual things like partitions etc that have no relevance for a driver, while /dev (and thus devfs) obviously think that that is the important part, much more important than how we actually got to the device.

I think we need to have some way of getting a mapping from /dev -> devicefs, but I don't think that has to be a filesystem thing (it might even be as simple as just one ioctl or new system call: 'get the "path" of this device').

There aren't that many people who actually care, I suspect.

Linus

Sure, It's obviously not 1:1, what I had in mind was for the controller to show what devices it exports in the sense of raw devices, but I agree the other way makes a lot more sense. My problem was how to be devfs agnostic, but you answered with "ioctl or syscall" and that would indeed be ok. The ioctl things make it appliable to network interfaces as well, which is good.

The need to do this link from a /dev to the driverfs, I suspect, will exist only for case like setting up the firmware, though I can imagine one may want to tweak some IDE settings (available via driverfs in your proposed scheme) knowing only the /dev node.

Ben.

Actually, I've always said that I think devfs should care about both views. And that's why I think putting the driver tree (ala driverfs) in devfs, and making the device-oriented part of the tree be symlinks into the bus-oriented tree, is a good idea.

Fugly. What's wrong with readlink(2) as this "magic syscall"?

				Regards,

					Richard....
Permanent: [email protected]
Current:   [email protected]

And I think you're completely wrong.

The fact is, they are two completely different and orthogonal things, and they have _nothing_ in common except for a very weak linkage of actual "physical device" (which does not always exist).

The set of people that cares about one view is almost 100% different from the set of people that care about the other view.

Ehh - like the fact that it doesn't work on device files?

Linus

I can't find anything in Posix/SuS that says it isnt allowed to however 8)

We can certainly do it, it just doesn't buy us much of anything, since none of the standard tools (ie "ls") will actually do the readlink() for anything but a symlink.

So at that point it's just another magic syscall, except we've overloaded an old one.

Which may certainly be acceptable, of course.

Linus

I wasn't suggesting a magic readlink(2). I was suggesting a *real* one. Device nodes get stored in the physical tree (what you call driverfs), and the entries in the logical tree are symlinks. Such as:

/dev/scsi/host0 symlink to /dev/bus/pci0/slot1/function2

or something like that. Easy to implement, easy to understand, easy to manage.

				Regards,

					Richard....
Permanent: [email protected]
Current:   [email protected]

NO.

This is one backwards compatibility thing that I'm _not_ removing.

We have tons of existign /dev trees, and I'm not making them into symlinks.

Also, you obviously haven't thought it through AT ALL. Hint: partitions.

If you have /dev/hda1, that _cannot_ be a symlink to the physical tree, because on a physical level that partition DOES NOT EXIST. It's purely a virtual mapping.

Yet clearly there _is_ a mapping from /dev/hda1 onto the physical device in question, and clearly it _is_ a meaninful operation to operate on the physical device underlying /dev/hda1.

So if you want to have a sane interface, you need to have a way to look up the physical device that underlies /dev/hda1.

Yet it clearly cannot be a symlink.

QED.

So stop mixing up physical devices and /dev. They should NOT be handled by the same mechanism.

Linus

Hi,

Somehow I expect Al to step in with something like:

mount -t partfs /devfs/bus/... /dev/hda

:-)

bye, Roman

[email protected] (Linus Torvalds) wrote on 07.05.02 in <[email protected]>:

Well ... one *could* argue that there's justification for showing those partitions by the exact same argument that there's reason to show devices on a SCSI or USB bus. It's just going further down the tree.

Say something like

        /driverfs/root/pci0/00:1f.4/scsi_bus/003/pc_partition/2

Sure, it's software, not hardware. OTOH, it's one of the things that change with hotplug. (And incidentally, fdisk changing partitions *might* be handled somewhat like a hotplug event ...)

As to linking to /dev, I see no reason why you couldn't have that tree include information (not in the tree *structure*, obviously) of what the relevant device numbers are. That's more expensive than a lookup with a pointer gotten from /dev, but it's certainly possible.

MfG Kai

I agree. I think it'd be great to have something like the above.

Hi!

I don't see why partitions in devicefs are bad idea.

Some physical chips show as multiple devices in devicefs, so I'd guess it would be okay for partitions to be there, too.
Pavel

Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.

Now you take the last step toward solaris and realize why I was always against your solution (no personal offence)
to the device management problem - they do it all in user space by precisely the above symlink system....

Yes, when will we get something that associates the physical device with network ethX name?

Ben, what you are proposing is fairly similar to what Solaris does today. There is a /devices directory that contains the real path while /dev contains the legacy stuff. Seems to work fine and given the proper docs, you can decipher what the /devices path points to fairly easily. So I certainly wouldnt mind seeing this happen for Linux eventually.

--Jauder

Uz.ytkownik Jauder Ho napisa?:

Amen, We would only have to add a device special file to some of the /devices Stuff and /dev/ could be a symlink tree pointing there...

I have *intentionally* named the standard mounting point of the devicefs /devices the time I added the description how to mount it to the driver-model.txt. The following words are from *me*:

This can be done permanently by providing the following entry into the /dev/fstab (under the provision that the mount point does exist, of course):

none /devices driverfs defaults 0 0

Or by hand on the command line:

~: mount -t driverfs none /devices

- phone: +49 214 8656 283
- job:   eVision-Ventures AG, LEV .de (MY OPINIONS ARE MY OWN!)
- langs: de_DE.ISO8859-1, en_US, pl_PL.ISO8859-2, last ressort: ru_RU.KOI8-R

I agree with such a concept, but as Linus said, it should go the other way, from the functional interface to physical interface. There are many details involved in doing such a thing, but it should work something like this:

The logical subystems (ide disks, networking, etc) would register with the device model core and get a directory in driverfs:

/driverfs/class/ide/

Devices would be discovered and get a driverfs directory representing the physical location of the device:

/driverfs/root/pci0/07.2/

Note that no drivers have been bound to the device. When the driver is bound, it registers the device with the subsystem, passing in a subsystem-specific structure. These can be made to point in some way to the generic struct device of the device (from which the physical path can be inferred).

When this happens, the subsystem creates a directory underneath its driverfs directory, so you get:

/driverfs/class/ide/0/

And, a symlink is created to point to the directory in the physical path. As the driver discovers partitions on the device, it can create special nodes in its class directory.

At this point, userspace can be notified (via /sbin/hotplug). That can create symlinks in /dev to the nodes that were just created, emulating current /dev behavior.

So, what does this do? To an extent, it reengineers the funtionality of devfs. I'll be the first to admit it. However, it centers less around the filesystem, and more on the device model core.

Most devices already register with their subsystems, so having the subsystesm pass device info onto the core is relatively easy.

As partitions are discovered, you get paths like:

/driverfs/class/ide/0/2

Which gives you a default name for the device. With /sbin/hotplug, simple userspace policy, and symlinks in /dev, you can emulate the current device hierarchy. So, you get a device naming solution that gives you only the device names for the devices you have.

This approach also de-emphasizes the dependency on major and minor numbers. If device nodes are created in kernel space initially, userspace doesn't need to know what the major/minor is for a particular device. The symlink to the device node is all that's need to operate on the device.

Without the need to coordinate between kernel and userspace, at least some majors/minors can be dynamically allocated as the subsystems and devices are registered with the core. (These can then be exported via files in driverfs). (This is similar to the dynamic allocation of minor numbers in the USB subsystem that showed up recently...)

Oh, and it's with a modern, clean filesystem, 1/5 the size of devfs.

Thoughts? Comments? Flames?

-pat

The size argument is not an issue. I've already said that devfs will shrink a lot once I move tree management from my own code to the VFS. At that point devfs will mostly be:
- an API
- a way fo supporting the devfsd protocol.

				Regards,

					Richard....
Permanent: [email protected]
Current:   [email protected]

I agree 100%. However, I think that move will be very painful. I tried to do it a couple of months ago, and there were so many interdependencies and oddities that I gave up after about 6 hours.

I argue that you shouldn't need a separate daemon. We already have the /sbin/hotplug interface. It's simple and sweet. We shouldn't need to rely on an entirely separate daemon.

-pat

Oh, it's certainly more that 6 hours of work. But it *will* get done.

The devfsd protocol is more lightweight. Plus it doesn't require fork(2)+execve(2) overheads. And more importantly, you can capture lookup() events.

				Regards,

					Richard....
Permanent: [email protected]
Current:   [email protected]

Even the mtrr driver was a good 8 hours to clean up, make readable and more object-oriented. I wish you luck, as well as anyone that has to attempt to decipher it.

These events are not performance critical, so the overhead is less important. Besides, almost all systems have /sbin/hotplug, since it can be anything - a shell script, a perl script, a tiny C executable.

The hotplug interface doesn't rely on any particular implementation. It only relies on something on the other side implementing a particular interface. The implementation can be replaced, as well as the format of the policy, based on the constratints of the system or the whims of the distro.

It also doesn't rely on a process running to capture events. What happens if the devfsd process is killed?

-pat

Hi,

Why not fixing devfs for that? My root directory is messed up enough. We have dev, proc, tmp, ...
We might have /dev/driver or such, which doesn't make the root directory any fuller. (And also not to disturb the newbies any further. It's hard a lot to explain to a windows user why he can't remove /proc and /dev, and what this is supposed to be.)
This is just my opinion...

Regards,
Thunder

if (errno == ENOTAVAIL)
    fprintf(stderr, "Error: Talking to Microsoft server!\n");

For one, I am of the camp that believes devfs is unfixable.

For two, where driverfs is mounted is irrelevant: /driverfs, /sys, /proc/bus are all valid places.

Besides, who cares what's in /? You have /home, which is all that really matters, no?

So don't give them root access. Or, explain to them that they're magic, like the pagefile.sys file. :)

-pat

But it's not actually broken, now that the locking is fixed.

				Regards,

					Richard....
Permanent: [email protected]
Current:   [email protected]

Really? What about the case of the missing BKL for device opens that you haven't really commented on?

Seems like devfs _still_ has locking problems.

Russell King ([email protected])                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

I did comment to you, privately, saying I was waiting to see what the consensus was on the issue of whether to move the BKL or not. I'll be sending a patch later this week to fix it.

A pretty minor one, given the comment I was responding to: "devfs is unfixable". I've noticed that even Al has gone quiet on the "devfs races" issue, now that the new code is in place :-)

				Regards,

					Richard....
Permanent: [email protected]
Current:   [email protected]

Yes, and hey, we still have the problem a week layer, even after the discussion went dead.

Never the less, your comment about "no locking problems" is inaccurate. devfs is calling at least one part of the kernel without obeying the existing locking rules. That's definitely a devfs bug.

Russell King ([email protected])                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Uz.ytkownik Patrick Mochel napisa?:

Just a side note...
Please please name it /devices/ Some old boys like me (age 30) can gain from similarities with some quite common "legacy" systems. We don't have to "invent" for the sake of it.

And /devices/ is the way I have named it in the corresponding documentation.

And is exactly why this showed up in the USB subsystem :)

And it removes the dependency of devfsd and its interface, replacing it with the existing /sbin/hotplug interface. This allows different people to implement different naming schemes if they so desire, moving naming policy out of the kernel into userspace, where it belongs.

Yes, there will probably be a "default" naming scheme, matching what we have today, but the ability to replace it with another one is _so_ much easier than having to try to tie into devfsd (like the devreg implementation does: http://www-124.ibm.com/devreg/ )

greg k-h

Uz.ytkownik Linus Torvalds napisa?:

Linus - there are no backward compatibility issues here. No single application from my system does mess with /proc/ide. They showed you a list of programs which use /proc and not a list of programs which use anything out of /proc/ide... RedHat even disables all this chip set specific reporting in theyr public kernels. OK kudzu is using it, but it does not *rely on it*. Heck kudzu is running all the time I rebooted my system during developement and nothing ugly did happen.

for fdisk on my notebook, well it runs just fine:

[root@kozaczek root]# fdisk /dev/hda

The number of cylinders for this disk is set to 2584. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/hda: 240 heads, 63 sectors, 2584 cylinders Units = cylinders of 15120 * 512 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/hda1   *         1         7     52888+  83  Linux
/dev/hda2             8      2556  19270440    5  Extended
/dev/hda4          2557      2584    211680   a0  IBM Thinkpad hibernation
/dev/hda5             8      2166  16322008+  83  Linux
/dev/hda6          2167      2219    400648+  82  Linux swap

Neither the programmer who wrote fdisk or cdrecord or anything else was stiupid enough to use anything out there, since using a simple ioctl is easier anyway. I *did* check them. (Admittedly I don't care about kudzu, but fdisk an friends I was fully aware of.)

BTW. If one needs the size of the disk well we could attach it as a file size to the device file in /dev IMHO. Why not?

The boot kernel has a lot of it disabled not the main ones.

I can't speak directly for the Kudzu maintainer but I can say that having a sane way to obtain the list of ide devices (all of them not just non pcmcia) and the device bindings/type has been a long standing request.

If 2.6 breaks a 2.4 installer and nothing else I don't think its a big disaster and the cleanup may well be justified

Uz.ytkownik Alan Cox napisa?:

Well personally I would just love if there where a "go ahead and don't care about "compatibility" for the following:

Make hdX gone and use the scsi device major/minor number stuff instead.

And then just making the ATA driver looking like if it where some incapable SCSI would actually reduce tons of code from kudzu and friends without the need for any adjustment there.

The SCSI layer is significant overhead even in 2.5. Right now for example it appears to be the primary bottleneck for the aacraid drivers. ATA6 is also more capable than SCSI in several areas regardless of the notional market positioning.

Linus talked about having a /dev/disc/... which once you have 32bit dev_t makes complete sense. What you don't do however is throw IDE through the SCSI midlayer, you merely make the /dev/disc/ point call into the right drivers - be they raid, scsi or ide. That also lets the scsi emulation crap get ripped out of the megaraid and aacraid drivers which will up performance.

Alan

Uz.ytkownik Alan Cox napisa?:

Alan... you have taken me wrong. What I mean is just the following. Take away some minors from use by SCSI (or more propably a common repository) and use the same ioctl numbers where possible. Perhaps implement some ioctl here and there... not more!

Not the whole: "we are just another SCSI device on the driver level". That would not make sense indeed. Since in esp. the SCSI mid-layer isn't taht pritty too...

i did some benchmarking on a high-end dual P3 Xeon (Serverworks chipset ) with QLogic 2300 (2gbit/s) 64/66 Fibre Channel controllers.

using the '/dev/sgX' interface to issue scsi reads/writes allowed me to hit the magical limit of 200mbyte/sec throughput. (basically just about linerate). (simultaneous "sg_read if=/dev/sgX mmap=1 bs=512 count=35M"; sg_read from the sg-tools package)

doing the same test thru the block-layer was basically capped at around 135mbyte/sec. (simultaneous "dd if=/dev/sdX of=/dev/null bs=512 count=35M").

whether the bottleneck was copy-from-kernel-to-userspace (ie. exhaustion of Front-Side-Bus / memory bandwidth) or related to block-layer overhead and scsi layer overheads, i haven't yet validated, but at a ~35% performance difference is relatively significant nontheless.

cpu utlization on the sg interface was under 10%. using 'dd' on the sd interface, both gigahertz P3 Xeons had 0% idle time.

cheers,

lincoln.

Lincoln,

You are right on the money!
There is about 35-40% throughput killjoy "copy-from-kernel-to-userspace". It is easy to demo if you have a bus analizer and can do accounting on the data io less the command block overhead.

CR3's are your friend, not ...

Andre Hedrick
LAD Storage Consulting Group

You should be able to verify that by using large O_DIRECT I/O's. The block layer itself may well be part of the overhead. With the scsi layer, the block->scsi handling code is definitely a bottleneck to performance

Tweak your dd to use O_DIRECT and use an O_DIRECT capable fs - that tells you if its copy overhead or disk side stuff.

You need to be careful with this stuff. Cache effects dominate.

I believe the /dev/sgX driver uses a fixed kernel-side buffer for the transfer. So the source of the copy_to_user() will always come out of cache if the CPU is snooping the busmastering. But not if the CPU is performing cache invalidates in response to that busmastering.

But for `dd', which has to copy the data out of pagecache, the copy_from_user() will get 100% misses on the source, guaranteed.

Also, the `sg_read' command reads everything into the same (small)
chunk of userspace memory. So the destination of copy_to_user()
is always in cache. Probably, the same is true with `dd bs=512', but one would have to go read the dd source to verify.

This is also why the scsi_debug driver runs so much faster than normal
devices: it copies everything out of a fixed in-kernel buffer. ie:
out of L1 cache. Fast.

Similarly, sg_dd' against scsi_debug is copying a fixed kernel buffer into a fixed userspace buffer But when dd' tries to do the same thing it incurs an additional copy into the pagecache. If the pagecache readahead window exceeds your L1 cache size (it does) then it will appear to be a lot slower.

Summary: the block layer ain't slow - it's memory which is slow ;)

-

...

i've validated that the performance difference is due to copy_to_user().
i created a hack in the tree where a read() on a file opened with the option O_NOCOPY causes no copy_to_user() to occur. (diff at the bottom of this email).

on this test machine (dual P3 Xeon / 256K L2 cache, 2G PC133 SDRAM, QLogic 2300 FC HBA, 8 x 15K RPM disks). maximum theoretical performance is 2gbit/s (~200mbyte/sec). kernel is 2.4.18.

i get the following performance numbers with 256K reads syncronously from the disks:

        /dev/md0 raid-0 with O_DIRECT:          91847kbyte/sec (2781usec 
avg latency/read)
        /dev/md0 raid-0:                                129455kbyte/sec 
(1978usec avg latency/read)
        /dev/md0 raid-0 with O_NOCOPY:  195868kbyte/sec (1297usec avg 

latency/read)

requests split evenly across /dev/sd[e-l] w/ O_DIRECT:
78279kbyte/sec (3276usec
avg latency/read)
requests split evenly across /dev/sd[e-l]: 105130kbyte/sec
(2437usec avg latency/read)
requests split evenly across /dev/sd[e-l] w/ O_NOCOPY:
123050kb/sec (2088usec avg
latency/read)

there's some interesting numbers here.
- given the performance difference between O_NOCOPY and pristine on /dev/md0 one can definitely point at that being the copy_to_user()
overhead.
- however, when requests are split across multiple block-devices
(/dev/sd[e-l])
the difference are significantly decreased -- and something wierdo is
going on:
- readahead?
- scsi priorities?
perhaps some form of async i/o is required to get the performance back.

i'll do some experiments with a 2.5.xx and see if the block-layer changes cause any significant changes.

--- pristine/linux/include/asm-i386/fcntl.h Tue Sep 18
06:16:30 2001

        +++ linux/include/asm-i386/fcntl.h      Thu May  9 18:56:46 2002
        @@ -20,6 +20,7 @@
         #define O_LARGEFILE    0100000
         #define O_DIRECTORY    0200000 /* must be a directory */
         #define O_NOFOLLOW     0400000 /* don't follow links */
        +#define O_NOCOPY        04 /* LTD HACK: dont do copy_to_user */

         #define F_DUPFD                0       /* dup */
         #define F_GETFD                1       /* get close_on_exec */
        --- pristine/linux/mm/filemap.c Tue Feb 26 06:38:13 2002
        +++ linux/mm/filemap.c  Thu May  9 18:56:48 2002
        @@ -1544,6 +1544,19 @@
                return retval;
         }

        +int file_read_nocopy_actor(read_descriptor_t * desc, struct page 

*page, unsigned long offset, unsigned long size)
+{
+ unsigned long count = desc->count;
+
+ if (size > count)
+ size = count;
+
+ desc->count = count - size;
+ desc->written += size;
+ desc->buf += size;
+ return size;
+}
+
int file_read_actor(read_descriptor_t * desc, struct page *page,
unsigned long offset, unsigned long size)

         {
                char *kaddr;
@@ -1591,7 +1604,7 @@
                        desc.count = count;
                        desc.buf = buf;
                        desc.error = 0;

- do_generic_file_read(filp, ppos, &desc, file_read_actor);
+ do_generic_file_read(filp, ppos, &desc, ((filp->f_flags & O_NOCOPY) ? file_read_nocopy_actor : file_read_actor));

                        retval = desc.written;
                        if (!retval)

cheers,

lincoln.

For bulk read() and write() I/O the best sized buffer is 8 kbytes. 4k is pretty good, too. Anything larger blows the user-side buffer out of L1. This is for x86.

This is a pretty important point, so let's repeat it:

Userspace programmers who are writing bulk-transfer read/write loops should use an 8 kbyte transfer buffer.

hmm. Why is O_DIRECT always the slowest? (and it would presumably do even worse with an 8k transfer size).

-

i just reproduced the test to validate the data. i'm using 8kbyte blocks here. on kernel is 2.4.18, O_DIRECT is still the slowest.

this machine has 2GB RAM, so it has 1.1GB RAM in HighMem.

booting a kernel with 'profile=2' set, the numbers were as follows:

- Base performance, /dev/md0 raid-0 8-disk array:

        [root@mel-stglab-host1 src]# readprofile -r; 
./test_disk_performance bs=8k blocks=4M /dev/md0
        Completed writing 31250 mbytes in 214. 94761 seconds (153.05 

Mbytes/sec), 53usec mean latency

- using /dev/md0 raid-0 8-disk array with O_DIRECT:

        [root@mel-stglab-host1 src]# readprofile -r; 
./test_disk_performance bs=8k blocks=4M direct /dev/md0
        Completed reading 31250 mbytes in 1229.830726 seconds (26.64 

Mbytes/sec), 306usec mean latency

- using /dev/md0 raid-0 8-disk array with O_NOCOPY hack:

        [root@mel-stglab-host1 src]# readprofile -r; 
./test_disk_performance bs=8k blocks=4M nocopy /dev/md0
        Completed writing 31250 mbytes in 163.602116 seconds (200.29 

Mbytes/sec), 39usec mean latency

so O_DIRECT in 2.4.18 still shows up as a 55% performance hit versus no O_DIRECT. anyone have any clues?

from the profile of the O_DIRECT kernel, we have:
[root@mel-stglab-host1 src]# cat /tmp/profile2.txt | sort -n -k3 |
tail -20

        c01ceb90 submit_bh                                   270   2.4107
        c01fc8c0 scsi_init_io_vc                             286   0.7772
        c0136ec0 create_bounce                               323   0.9908
        c0139d80 unlock_buffer                               353   4.4125
        c012f7d0 kmem_cache_alloc                            465   1.6146
        c0115a40 __wake_up                                   470   2.4479
        c01fa720 __scsi_end_request                          509   1.7674
        c01fae00 scsi_request_fn                             605   0.7002
        c013cab0 end_buffer_io_kiobuf                        675  10.5469
        c01154e0 schedule                                    849   0.6170
        c0131a40 rmqueue                                     868   1.5069
        c025ede0 raid0_make_request                          871   2.5923
        c0225ee0 qla2x00_done                                973   1.6436
        c013cb60 brw_kiovec                                 1053   1.0446
        c01ce400 __make_request                             1831   1.1110
        c01f30e0 scsi_dispatch_cmd                          1854   2.0692
        c011d010 do_softirq                                 2183   9.7455
        c0136c30 bounce_end_io_read                        13947  39.6222
        c0105230 default_idle                             231472 3616.7500
        00000000 total                                    266665   0.1425

contrast this to the profile where we're not using O_DIRECT:
[root@mel-stglab-host1 src]# cat /tmp/profile3_base.txt | sort -n
-k3 | tail -20

        c012fdc0 kmem_cache_reap                             369   0.4707
        c013b330 set_bh_page                                 397   4.9625
        c011d010 do_softirq                                  419   1.8705
        c0131a40 rmqueue                                     466   0.8090
        c01fa720 __scsi_end_request                          484   1.6806
        c012fa60 kmem_cache_free                             496   3.8750
        c013bd00 block_read_full_page                        523   0.7783
        c012f7d0 kmem_cache_alloc                            571   1.9826
        c013db39 _text_lock_buffer                           729   0.9812
        c0130ca0 shrink_cache                                747   0.7781
        c01cea70 generic_make_request                        833   2.8924
        c025ede0 raid0_make_request                          930   2.7679
        c013b280 get_unused_buffer_head                      975   5.5398
        c01fc8c0 scsi_init_io_vc                            1003   2.7255
        c013d490 try_to_free_buffers                        1757   4.7745
        c013a9d0 end_buffer_io_async                        2482  14.1023
        c01ce400 __make_request                             2687   1.6305
        c012a6e0 file_read_actor                            6951  27.1523
        c0105230 default_idle                              15227 237.9219
        00000000 total                                     45048   0.0241

the biggest difference here is bounce_end_io_read in O_DIRECT.
given there's still lots of idle-time, i'll file up lockmeter on here and see if theres any gremlins there.

cheers,

lincoln.

Yes.

O_DIRECT isn't doing any read-ahead.

For O_DIRECT to be a win, you need to make it asynchronous.

Linus

In message <[email protected]>, > : Li

O_DIRECT is especially useful for applications which maintain their own cache, e.g. a database. And adding Async to it is an even bigger bonus (another Oracleism we did in PTX). No read ahead, no attempt to keep the buffer in memory until memory pressure kicks in. Just a good tool for doing random IO (like an OLTP database would do).

gerrit

The thing that has always disturbed me about O_DIRECT is that the whole interface is just stupid, and was probably designed by a deranged monkey on some serious mind-controlling substances [*].

It's simply not very pretty, and it doesn't perform very well either because of the bad interfaces (where synchronocity of read/write is part of it, but the inherent page-table-walking is another issue).

I bet you could get _better_ performance more cleanly by splitting up the actual IO generation and the "user-space mapping" thing sanely. For example, if you want to do an O_DIRECT read into a buffer, there is no reason why it shouldn't be done in two phases:

(1) readahead: allocate pages, and start the IO asynchronously
(2) mmap the file with a MAP_UNCACHED flag, which causes read-faults to "steal" the page from the page cache and make it private to the mapping on page faults.

If you split it up like that, you can do much more interesting things than O_DIRECT can do (ie the above is inherently asynchronous - we'll wait only for IO to complete when the page is actually faulted in).

For O_DIRECT writes, you split it the other way around:

(1) mmwrite() takes the pages in the memory area, and moves them into the page cache, removing the page from the page table (and only copies if existing pages already exist)
(2) fdatasync_area(fd, offset, len)

Again, the above is likely to be a lot more efficient _and_ can do things that O_DIRECT only dreams on.

With my suggested _sane_ interface, I can do a noncached file copy that should be "perfect" even in the face of memory pressure by simply doing

	addr = mmap( ..  MAP_UNCACHED ..  src .. )
	mwrite(dst, addr, len);

which does true zero-copy (and, since mwrite removes it from the page table anyway, you can actually avoid even the TLB overhead trivially: if mwrite notices that the page isn't mapped, it will just take it directly from the page cache).

Sadly, database people don't seem to have any understanding of good taste, and various OS people end up usually just saying "Yes, Mr Oracle, I'll open up any orifice I have for your pleasure".

Linus

[*] In other words, it's an Oracleism.

Used with aio its extremely nice. Without the aio patches its a bit lacking whenever readahead is useful

Alan

But the point is that AIO is needed just to cover up the fundamental idiocy in the interface. If the interface had been properly designed, it would have been useful _without_ AIO.

Linus

This works for your load, but it does not work for the load it is designed for. Sequentially reading and througput is not the way to measure it. You need random reading and lattency to see it's merrits.

Greetings
Bernd

You're only halfway right. You want to avoid the mmap altogether. To see why, postulate that you have infinitely fast I/O devices (I know that's not true but it's close enough if you get enough DMA channels going at once, it doesn't take very many to saturate memory). For any server application, now all your time is in the mmap(). And there is no need for it in general, it's just there because the upper layer of the system is too lame to handle real page frames.

Go read the splice notes, ftp://bitmover.com/pub/splice.ps because those were written after we had tuned things enough in IRIX that it was the VM manipulations that became the bottleneck.

Another way to think of it is this: figure out how fast the hardware could move the data. Now make it go that fast. Unless you can hide all the VM crud somehow, you won't achieve 100% of the hardware's capability.

I know I've done a bad job explaining the splice crud, but there is some pretty cool stuff in there, if you really got it, you'd see how the server stuff, the database stuff, the aio stuff, all I/O of any kind can be done in terms of the splice:pull() and splice:push() interfaces and that it is the absolute lowest cost way to have a generic I/O layer.

---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

See my details on doing the perfect zero-copy copy thing.

The mmap doesn't actually touch the page tables - it ends up being nothing but a "placeholder".

So if you do

addr = mmap( .. MAP_UNCACHED .. src .. ) mwrite(dst, addr, len);

then you can think of the mmap as just a "cookie" or the "hose" between the source and the destination.

Does it have to be an mmap? No. But the advantage of the mmap is that you can use the mmap to modify the stream if you want to, quite transparently. And it gives the whole thing a whole lot more flexibility, in that if you generate the data yourself, you'd just do the mwrite() - again with zero copy overhead.

And I personally believe that "generate the data yourself" is actually a very common case. A pure pipe between two places is not what a computer is good at, or what a computer should be used for.

Linus

Hello Linus ,

On Sat, 11 May 2002, Linus Torvalds wrote:
... Large snip ...

Hmmm , (This may not apply here But ...)
What about linux as a router (ip/ipx/...) or a bridge device ?
Tia , JimL

+------------------------------------------------------------------+
| James W. Laferriere | System Techniques | Give me VMS |
| Network Engineer | P.O. Box 854 | Give me Linux |
| [email protected] | Coudersport PA 16915 | only on AXP |
+------------------------------------------------------------------+

as the person who started this whole thread and made the assertion that copying from A to B is common:

i think you'd be surprised. if we include "pipe from disk to network" then a large number of 'server' applications do exactly this. webservers do. fileservers do. http caches do. streaming-media servers do.

sure, they may add additional headers on the front and still generate dynamic content in some cases, but the "common case" is 'pipe from disk to network' or 'pipe from network to disk'. 'network' is typically TCP but can be UDP (with rate-limiting) in some cases.

its very good to see this being discussed. thats a large step forward from many people believing the problem was nonexistent.

i'm skeptical that continuing to use the page-cache is the correct way to go -- many of these kinds of applications are doing their own form of memory-management and hot-content 'caching' so are happy to manage a few-to-several hundred megabytes of "page cache equivalent" data themselves. at least on many of the 2.3.xx linux releases, that was one of the big attractions of 'raw' devices -- they didn't get the box into an OOM situation.
if 2.5.xx and recent 2.4.xx has the issues of page-cache-doesn't-shrink-fast-enough solved, then its forseeable it will fly.

cheers,

lincoln.

Huh, I must have missed something, does the mmap() not create any page tables at all?

---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

It can. But go down to the end in my first explanation to see why it doesn't have to.

I'll write up the implementation notes and you'll see what I'm talking about:

- readahead(fd, offset, size)

Obvious (except the readahead is free to ignore the size, it's just a hint)

- mmap( MAP_UNCACHED )

This only sets up the "vma" descriptor (like all other MMAP's). It's exactly like a regular private mapping, except instead of just incrementing the page count on a page-in, it will look at whether the page can just be removed from the page cache and inserted as a private page into the mapping ("stealing" the page).

- fdatasync_area( fd, offset, len)

Obvious. It's fdatasync, except it only guarantees the specific range.

- mwrite(fd, addr, len)

This is really does the "reverse" of mmap(MAP_UNCACHED) (and like a mapping, addr/len have to be page-aligned).

This walks the page tables, and does the _smart_ thing:

- if no mapping exists, it looks at the backing store of the vma, and gets the page directly from the backing store instead of bothering to populate the page tables.

- if the mapped page exists, it removes it from the page table

- in either case, it moves the page it got into the page cache of the destination file descriptor.

NOTE on zero-copy / no-page-fault behaviour:
- mwrite has to walk the page tables _anyway_ (the same as O_DIRECT), since that's the only way to do zero-copy.
- since mwrite has to do that part, it's trivial to notice that the page tables don't exist. In fact, it's a very natural result of the whole algorithm.
- if user space doesn't touch the mapping itself in any way (other than point mwrite() at it), you never build up any page tables at all, and you never even need to touch the TLB (ie no flushes, no nothing).
- note how even "mmap( MAP_UNCACHED )" doesn't actually touch the TLB or the page tables (unless it uses MAP_FIXED and you use it to unmap a previous area, of course - that's all in the normal mmap code already)

See?

I will _guarantee_ that this is more efficient than any O_DIRECT ever was, and it will get very close to your "optimal" thing (it does need to look at some page tables, but since the page tables haven't ever really needed to be built up for the pure copy case, it will be able to decide that the page isn't there from the top-level page table if you align the virtual area properly - ie at 4MB boundaries on an x86).

I suspect that this is about a few hundred lines of code (and a lot of testing). And you can emulate O_DIRECT behaviour with it, along with splice (only for page-cache entities, though), and a lot of other off-by-one uses.

Linus

In message <[email protected]>, > : Lin

[...snip... lots of good ideas...]

I'm not sure this is quite the same problem that Oracle (and others) typically used O_DIRECT for (not trying to be an apologist here, just making sure the right problem gets solved)...

Most of what Oracle was managing with O_DIRECT was its "Shared Global Area", which is usually a region of all possible memory that the OS and other applications aren't using. It uses that space like a giant buffer cache. Most of the IO's for OLTP applications were little bitty random 2K IOs. So, their ideal goal was to have the ability to say here's a list of 10,000 random 2K IOs I want you to do really quickly and spread them out at these spots within the SGS. Those IOs can be read asynchronously, but there needs to be some way to know when the bits make it from disk to memory. Think of it as something like a big async readv, ideally with the buffer cache and as much of the OS out of the way as possible.

When the SGA is "full" (memory pressure) they do big async, no buffer cache, non-deferred writev's (by non deferred, I mean that the write is actually scheduled for disk, not buffered in memory indefinitely - they really believe they are done with those buffers).

Now the mmap( MAP_UNCACHED ) thing might work, except that this isn't really a private mapping - it's a shared mapping. So something like tmpfs might be the answer, where the tmpfs had a property of being uncached (in fact, Oracle would love it if that space were pinned into memory/non-pageable/non-swappable). That way the clients don't block taking page faults and the server schedules activities to get the greatest throughput (e.g. schedule clients who wouldn't block).

Unfortunately, tmpfs takes away the niceness of the VM optimizations, I fear.

Oh, and Database DSS workloads (Decision Support, scan all disks looking for needles in a big haystack) has different tradeoffs, mostly needing to focus on lots of sequential IO where pre-fetching and reading, and discard buffers immediately after use are the primary focus and write performance is not critical.

gerrit

We did some i/o profiling about 6 years ago on a big scientific
app that had started in fortran and had been rewritten in c++
the fortran code used r/w on files and used temp files
the c++ did memmaps and had big data structures - taking advantage of memory management.
one thing I thought was interesting is that it was easy to see how a smart algorithm, not even such a smart one, could adapt i/o to the patterns of i/o in the fortran code, but the c++ i/o patters were really complex.

when everything goes into the page cache, it seems like you will loose information.

---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

That is certainly the case. If the application is seekily writing to a file then we currently lay the file out on-disk in the order in which the application seeked. So reading the file back linearly is very slow.

Now this is not necessarily a bad thing - if the file was created seekily then it will probably be _used_ seekily so no big loss probably.

This problem is pretty unsolvable for filesystems which map blocks to their disk address at write(2) time. It can be solved for allocate-on-flush filesystems via a sort of the dirty page list, or by maintaining ->dirty_pages in a tree or whatever.

There is one "file" where this problem really does matter - the blockdev mapping "/dev/hda1". It is both highly fragmented and poorly sorted on the dirty_pages list.

It's pretty trivial to perform a sillysort at writeout time:
if we just wrote page N and the next page isn't N+1 then do a
pagecache probe for "N+1". That's probably sufficient. If
not, there's a simple little sort routine over at
http://www.chiark.greenend.org.uk/~sgtatham/algorithms/listsort.html
which is appropriate to our lists.

I'll be taking a look at the sillysort option once I've cleared away some other I/O scheduling glitches.

-

I've never liked mmap although that may just be my advanced age ("we never had mmap, we copied files by cutting cuneiform in fresh
clay tablets, the way the gods intended ")

	struct kio k;
	k.count = RECORDSIZE;
	fd1 = open("inputfile",KIO_READ);
	fd1a = dup(fd1); //dup creates a non KIO descript for the samefile
	fd2 = open("outputfile",KIO_WRITE);
	while( (n=read(fd1,&k,sizeof struct kio) 
            {
		write(fd2,&k,sizof struct kio);
		if(k.seekposition%10000){
			write(fd1a,"Another record sent,Mr  E.\n",GROVELSIZE);
		}
            }

When you drive by that campus in redwood city you start to understand how insignificant you are.

---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Will the use of O_DIRECT affect disk elevatoring?

Sorry if this is OT - I just need to know

Please cc: to me as I'm not on the list

roy

i believe the elevator is based on the 'block' layer and anything that goes thru it. so the answer is that the requests would use the elevator.

for the test in question, i was doing sequential reads from the first block of each disk until some block later on in the disk. (ie. a 2gbyte read or 18gbyte read). given that was the case and the only i/o ops were 'read' operations, elevator would make no difference here.

cheers,

lincoln.

No, the I/O scheduler can't even tell whether it's being handed O_DIRECT buffers or not.

Jens

We tried disabling the elevator while doing Raw IO with DB2 a couple of weeks ago. The database performance degraded much more than expected. Disks were FC connected Tritons or SCSI connected ServerRaid (or both?). Oracle often asks for a patch to disable the elevator since they believe they can schedule IO better. We didn't try with Oracle in this case, but DB2 and RAW IO without and elevator was not a good choice.

gerrit

I'm curious how you did this -- did you disable sorting and merging, or just sorting? Merging is pretty essential to getting decent I/O speeds in current kernels.

Due to excessive queue scan times, lock contention, or just slight waste of cycles?

Jens Axboe

I believe sorting AND merging were turned off. BTW, this was 2.4 only, our primary focus is getting product into people's hands this year. We are hoping to play with the 2.5 IO scheduler, possibly in a few months.

A lot more interrupts on the RAID device, indicating a lot more IOs, probably a direct result of disabling merging. Overall IO throughput dropped pretty dramatically, reducing database throughput.

A good indication to gen a patch with just sorting turned off and see where that gets us...

gerrit

8k would rather disadvantage O_DIRECT. It can't do readahead and it can't to write-behind. It'll be forced to do tons of tiny I/Os.

Well hopefully that will be gone in 2.4.20, for many popular controllers.

Not sure why it bit you here. It could be that the page allocator just happened to give you highmem pages for the O_DIRECT test. Turning off highmem may make the test more repeatable.

lockmeter will go off the dial. All those copies happen at interrupt time, inside the global io_request_lock. It's horrid.

Try it with the block-highmem patch:
http://www.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.19pre1aa1/00_block-highmem-all-18b-4.gz

-

given i had to recompile the kernel to add lockmeter, i'd already cheated and changed PAGE_OFFSET from 0xc0000000 to 0x80000000, obviating the requirement for highmem altogether.

being fair to O_DIRECT and giving it 1mbyte disk-reads to work with and giving normal i/o 8kbyte reads to work with.
still using 2.4.18 with profile=2 enabled and lockmeter in the kernel but not turned on. still using the same disk spindles (just 6 this time), each a 18G 15K RPM disk spindle. i got tired of scanning the entire available space on an 18G disk so just dropped the test down to the first 2G of each disk.

O_DIRECT is still a ~30% performance hit versus just talking to the /dev/sdX device directly. profile traces at bottom.

normal block-device disks sd[m-r] without O_DIRECT, 64K x 8k reads:

        [root@mel-stglab-host1 src]# readprofile -r; 
./test_disk_performance blocks=64K bs=8k /dev/sd[m-r]
        Completed reading 12000 mbytes in 125.028612 seconds (95.98 

Mbytes/sec), 76usec mean

normal block-device disks sd[m-r] with O_DIRECT, 5K x 1 megabyte reads:

        [root@mel-stglab-host1 src]# readprofile -r; 
./test_disk_performance blocks=5K bs=1m direct /dev/sd[m-r]
        Completed reading 12000 mbytes in 182.492975 seconds (65.76 

Mbytes/sec), 15416usec mean

for interests-sake, compare this to using the 'raw' versions of the same disks:

        [root@mel-stglab-host1 src]# readprofile -r; 
./test_disk_performance blocks=5K bs=1m /dev/raw/raw[2-7]
        Completed reading 12000 mbytes in 206.346371 seconds (58.15 

Mbytes/sec), 16860usec mean

of course, these are all ~25% worse than if a mechanism of performing the i/o avoiding the copy_to_user() altogether:

        [root@mel-stglab-host1 src]# readprofile -r; 
./test_disk_performance blocks=64K bs=8k nocopy /dev/sd[m-r]
        Completed reading 12000 mbytes in 97.846938 seconds (122.64 

Mbytes/sec), 59usec mean

anyone want to see any other benchmarks performed? would a comparison to 2.5.x be useful?

comparative profile=2 traces:
- no O_DIRECT:

        [root@mel-stglab-host1 src]# readprofile -v | sort -n -k3 | tail -10
        80125060 _spin_lock_                                 718   6.4107
        8013bfc0 brw_kiovec                                  798   0.9591
        801cbb40 generic_make_request                        830   2.8819
        801f9400 scsi_init_io_vc                             831   2.2582
        8013c840 try_to_free_buffers                        1198   3.4034
        8013a190 end_buffer_io_async                        2453  12.7760
        8012b100 file_read_actor                            3459  36.0312
        801cb4e0 __make_request                             7532   4.6152
        80105220 default_idle                             106468 1663.5625
        00000000 total                                    134102   0.0726

- O_DIRECT, disks /dev/sd[m-r]:

        [root@mel-stglab-host1 src]# readprofile -v | sort -n -k3 | tail -10
        801cbb40 generic_make_request                         72   0.2500
        8013ab00 set_bh_page                                  73   1.1406
        801cbc60 submit_bh                                   116   1.0357
        801f72a0 __scsi_end_request                          133   0.4618
        80139540 unlock_buffer                               139   1.7375
        8013bf10 end_buffer_io_kiobuf                        302   4.7188
        8013bfc0 brw_kiovec                                  357   0.4291
        801cb4e0 __make_request                              995   0.6097
        80105220 default_idle                              34243 535.0469
        00000000 total                                     37101   0.0201

- /dev/raw/raw[2-7]:

        [root@mel-stglab-host1 src]# readprofile -v | sort -n -k3 | tail -10
        8013bf50 wait_kio                                    349   3.1161
        801cbb40 generic_make_request                        461   1.6007
        801cbc60 submit_bh                                   526   4.6964
        80139540 unlock_buffer                               666   8.3250
        801f72a0 __scsi_end_request                          699   2.4271
        8013bf10 end_buffer_io_kiobuf                       1672  26.1250
        8013bfc0 brw_kiovec                                 1906   2.2909
        801cb4e0 __make_request                            10495   6.4308
        80105220 default_idle                              84418 1319.0312
        00000000 total                                    103516   0.0560

- O_NOCOPY hack: (userspace doesn't actually get the read data)

        801f9400 scsi_init_io_vc                             785   2.1332
        8013c840 try_to_free_buffers                         950   2.6989
        801f72a0 __scsi_end_request                          966   3.3542
        801cbb40 generic_make_request                       1017   3.5312
        8013bf10 end_buffer_io_kiobuf                       1672  26.1250
        8013a190 end_buffer_io_async                        1693   8.8177
        8013bfc0 brw_kiovec                                 1906   2.2909
        801cb4e0 __make_request                            13682   8.3836
        80105220 default_idle                             112345 1755.3906
        00000000 total                                    144891   0.0784

if the program is doing only what shown in the main loop, then you're reading 10 times more data with O_DIRECT, that was my point in saying 64k*8k == 5k * 1M / 10, but I assume you took it into account (otherwise it means O_DIRECT is just 5 times faster than buffered-I/O for you)

Also I would suggest to measure the time taken by the whole workload, not only the time for read/write syscalls.

it is set to 512 bytes.

yep, as said that's because O_DIRECT uses the softblocksize (1k) large b_size, while raw uses the hardblocksize that is 512bytes, so raw wastes 2 times more memory and cpu in handling those list of smaller bh. That is fair comparison with the buffered-IO, also the buffered-IO uses 1k b_size.

Yes, I only wanted to make clear the no-copy hack will always be faster then anything that ends putting the data in user memory (or providing information where the data in userspace is) somehow.

You'd at least need to reserve only a contigous part of the physical pages for that purposes, or you would run out of virtual address space on a 32bit arch, that just is a problem with fragmentation. Secondly you should use such mmapped /dev/mem area as your backing store for the application cache or it's again a copy-user. It seems very messy. I think writing a software TLB for a certain special VMA allowing the resolution of a virtual address in the VMA to a struct page with a very efficient lookup would be better if something to skip the overhead of the pagetable management. And it doesn't need special userspace hacks with horrible API.

Sure just search the .h file for something like this:

#define QLOGICISP {							   \
	detect:			isp1020_detect,				   \
	release:		isp1020_release,			   \
	info:			isp1020_info,				   \
	queuecommand:		isp1020_queuecommand,			   \
	abort:			isp1020_abort,				   \
	reset:			isp1020_reset,				   \
	bios_param:		isp1020_biosparam,			   \
	can_queue:		QLOGICISP_REQ_QUEUE_LEN,		   \
	this_id:		-1,					   \
	sg_tablesize:		QLOGICISP_MAX_SG(QLOGICISP_REQ_QUEUE_LEN), \
	cmd_per_lun:		1,					   \
	present:		0,					   \
	unchecked_isa_dma:	0,					   \
	use_clustering:		DISABLE_CLUSTERING,			   \
	can_do_varyio:		1,					   \
}

and add can_do_varyio: 1 like in the above file for qlogicisp.h.

(btw, then the vary-io will allow more efficient I/O handling than the buffered-IO, so it would get unfair with the so underpowered buffered-IO, but still it would be interesting to see the effect of varyio in numbers)

You also mentioned the md device. If you do I/O to a raid0 array with 5 disks attached to an MD device then your buffersize with O_DIRECT must be 5*512k at least or you cannot send bit large scsi commands to each scsi disk and performance would be very bad compared to reading 1M from each /dev/sd separately.

One thing I would also recommend is to write a threaded version of the program, that reads or writes to all the /dev/sd disks simultaneously, first w/ O_DIRECT then w/o O_DIRECT. The reason is that currently you aren't using all the disks at once with O_DIRECT due the lack of async-io, while for example async-writeback w/o O_DIRECT allows a better scaling over the disks, not to tell the fact you only benchmark the duration of the syscall sounds not accurate if async I/O happens over userspace (userspace can get stalled by completion interrupts etc.. and you're not measuring such overhead).

If you instead make a single raid0 array and you use a buffer size of nr_PV*512k (or nr_PV*1M even better) then also O_DIRECT without threading should perform similar to the buffered IO.

In my measurements the lack of async-io with O_DIRECT (with a single disk) wasn't significant in the bandwith numbers, let's say a few percent slower than the buffered-IO, but the CPU utilization and mem bandwith was so much optimized that I thought it definitely pays off even now without a kernel side async-io (note that I wasn't doing simultaneous I/O to multiple devices, futhmore the bandwith of the membus was not shared with any other workload).

Since you "stripe" by hand in all disks you do a different workload than my previous benchs and you definitely want to keep all the harddisk running at the same time. I would also suggest to benchmark a single disk, to see if there is still such a big performance difference (again: including the cost outside the syscalls too).

Thanks for the interesting big-iron number-feedback :)

Andrea

no- count them up above.
without O_DIRECT i was doing: 64000 x 8192 = 524288000 bytes with O_DIRECT i was doing: 5000 x 1048576 = 524288000 bytes.

ie. same amount of data.

regardless, the "mbyte/sec" is calculated at the very end.

the time taken for the whole workload _is_ calculated at the very end of the workload. that way and "readahead" doesn't have an unfair advantage. i also have gettimeofday() calls to measure latency on a per-read or per-write basis

...

i've thought about doing that - shame that there isn't an async version of read(). the hard part is that i want to keep the disks at roughly the same "block" at the same time, so there will still need to be some syncronization of threads. otherwise, basic SCSI id priority means that it won't be fair across all disks. (remeber that i'm also attempting to measure latency)

...

a single-disk means we hit the performance limits of a single disk spindle. ie. around 45mbyte/sec sustained throughput. if you don't think there's any real overhead in the MD driver, i'll just use that instead for now. (raid-0)

kinda fun. :-)

cheers,

lincoln.

btw, since you've 8 disks it probably would be a bit faster with 8meg, so you submit at least 2 scsi commands for each disk at the same time, not only 1.

120 vs 127 is pretty good, also considering an 8meg buffer may be enough to give you such 7 mbyte/sec back too.

yes, a performance hit in absolute disk performance is expected due the additional synchronization after each read/write syscall returns (the synchronous beahviour), if the mem bandwith is very high and the mem bandwith and cpu is otherwise unused, so if the machine is completly dedicated to I/O and the mem bandwith is high. But look at the profiling...

here you see the machine was basically idle for the whole time.

and here the machine was almost completly loaded in the file_read_actor, it was unusable for anything other than I/O.

To make a similar example my first pentium had a very slow harddisk that in DMA mode was running 10/20% slower than in PIO mode, but the cpu utilization of the DMA mode was so much lower that DMA was an huge global win during kernel compiles etc...

In any real usage where I/O is combined with CPU and mem bus utilization for computations, not only to move data from disk to userspace memory O_DIRECT is an _huge_ win as you found out. You have 100000 more usable ticks for computations over 150000 total ticks. I didn't do exact math but a rough measure is that with "buffered I/O" (i.e. base) only around 20% of the cpu was available for doing useful things, with O_DIRECT the 90% of the cpu is available for doing useful things, not to tell the difference in membus utilization. So I think the improvement of O_DIRECT is just huge and pays off as I also mentioned in an erarlier email in this thread, even if the absolute I/O performance may decrease of a few point percent due the synchronous behaviour (and one of the items for 2.5 is to optionally make the synchronous behaviour to go away, plus providing a generic_direct_IO that synchronizes with the pagecache via anchors and so allows O_DIRECT to be used with data journaling too).

And again, if you use larger buffers you may be able to fix also the last 8mbyte/sec difference :).

Andrea

At 05:51 PM 21/05/2002 +0200, Andrea Arcangeli wrote:
...

...

no disagreement there.

however, there is still an enormous difference between access-/dev/sdX-via-block-layer and access-dev/sdX-via-mmap (127mbyte/sec versus 175+mbyte/sec) that for many "disk intensive" applications is a huge difference.

if we _could_ produce superior performance with something like:
(a) additional improvements to O_DIRECT (say, a mmap() hack for O_DIRECT) whereby i/o can go direct-to-userspace or
userspace-can-mmap(readonly)-into-
kernel-space
or
(b) O_DIRECT with async-i/o
or
(c) /dev/rawN-like interface (eg. /dev/directN) within a fixed disk
buffer in kernel
(eg. physical memory allocated at bootup) that is populated readonly into user-space)
would such a hack be accepted in the 2.5 tree?

it sure isn't your "average desktop" setup where you hit these things as performance limitations, but many folk (myself included) have very specific applications which are essentially bottlenecked on PC hardware limitations. these limitations are a side-effect of how the linux kernel works. that doesn't mean that other OSes don't suffer from the same thing -- since no-other mainstream OS does the right thing, that doesn't mean that we cannot make linux even better.

obviously, many of the above would only be suitable for disk-read operations and less-so for disk-write -- but its probably easier to tackle these one by one.

cheers,

lincoln.

hmm wait a moment, in your last email 175MB/sec was the nocopy hack, not mmap("/dev/sda"). mmap has the same overhead of O_DIRECT in dealing with the user pagetables (actually O_DIRECT has the potential to be a bit lighter because map_user_kiobuf solves the page faults before triggering a real CPU page fault that has to pass through do_page_fault for every single page, and we prefault all of them before starting the I/O), so if 175MB/sec was the nocopy hack, 175MB/sec definitely cannot be the mmap too.

Infact a nice addition to your proggy would be to also benchmark the I/O bandwith using mmap driven I/O, instead of buffered I/O, raw, O_DIRECT and the nocopy hack. Like raw and O_DIRECT also the mmap driven I/O is zerocopy I/O, so only the buffered-IO would show the total waste of cpu and mem bandwith. I guess mmap will be on the same performance lines with O_DIRECT (potentially a bit faster thanks to readahead), at least during reads, writes are a bit different, you've to flush by hand with msync region by region like if they would be write syscalls, or to wait vm pressure.

175MB/sec means you only read to kernel space and you don't make such information visible to userspace, so it is unbeatable no matter what :). Only in kernel service like sendfile and/or tux can take advantage of it by taking the radix tree lock and looking up the radix tree to find the physical page.

even if you get a direct physical view of some of the address space (as said in a earlier email on 32bit the lack of address space would reduce the place you can map to a little zone), you still need the information on the physical memory where the I/O gone, so that's still an API from kernel to user, so again an overhead compared to the nocopy hack. And that information is exactly the radix tree/hashtable (and btw, I still I'm not a believer in the radix tree and I wonder if anybody did any real benchmark on the big irons with a properly tuned hashtable that takes into account the highmem too like with Davem's patch, the last benchmarks I seen where fake IIRC, the bonus of the radix tree is the per-inode locking and that it reduces the amount of normal zone needed to be allocated statically for numa-q, of course until somebody notices they can exploit the radix tree with a few liner proggy and hogs the machine with unfreable radix tree metadata by allocating only 1 page of highmem cache for each radix tree leaf entry and that will need further instrumentation to the vm so that it will release highmem memory related to the lowmem radix tree metadata, at least it's only a security problem, not a pratical problem with useful users). Not to tell if there's a bug in your proggy you'd destabilize the kernel. The truth is, this is no different at all than writing a proper kernel service, if you need that kind of performance you should write kernel code, not user code, and to work with the pagecache directly. That's what tux does of course, that's what sendfile does too, sendfile is like a part of an userspace fileserver that is been moved in kernel space to send stuff over the wire in zerocopy with knowledge about kernel data structures incidentally like pagecache/raidx tree. You should build a sendfile like operation and anyways to do the stuff in kernel. So you also take all the advantages of tlb 4Mpages etc..etc.. that kernel code has. Also take in mind sendfile can just now work to copy data in the fs too, infact before 2.4.3 it was useful only for doing that fs copies avoiding passing through userspace before the pskb layer was introduced in the network code. Now thanks to the blkdev-in-pagecache introduced in 2.4.10 sendfile will even work with blkdevices, it couldn't work with blockdevices previously. Unfortunately /bin/cp never started using it, I made a patch to change cp to use sendfile it and to fallback to read/write if it returned -ENOSYS, but it is never been integrated, it's still available in my ftp area at suse.com/pub/people/andrea if anybody is interested to actually copy data in the filesystem without wasting mem bandwith (I don't copy heavily stuff around [I only create directory with -l] so it is never been a real need for me). The worst part of my /bin/cp sendfile patch is that sendfile isn't capable of being interrupted by a signal, so copying an huge file would hang the machine, but the same is just for huge read and write, so I consider that a kernel issue and I think userspace should just write the whole thing at once (however until the kernel remains uncapable of interrupting sendfile, userspace can as well sendfile in chunks of a few mbyte each so there's no risk of an accidental DoS at least, as far as the chunks are of the order of the few megs there should be no loss in performance). btw, now that sendfile64 is available in 2.5 the cp hack is doable again with LFS support too.

Returning to our topic :), the pagetables can be see as a representation of the same information encoded in the radix tree. They're the data structure meant to provide the information of where is the physical page where the I/O happened to userspace, and they provide a security layer between kernel and userspace unlike /dev/mem or a kernel service accessing the radix tree directly. instead of mapping physical contigous ram via /dev/mem, and telling userspace where's the ram it needs to use, the kenrel sets the pagetables that lead userspace to reach the right ram without having to deal with its physical position, and this provides protection and the app cannot crash the kernel (unless you find a bug in map_user_kiobuf :).

that's a different problem, this only increases the I/O pipeline, so it will only give you such 7mbyte/sec back, it won't reduce cpu utilization further or it won't make the I/O significantly faster (compared to using large buffers for synchronous read/writes). It makes sure that no disk is idle for no good reason, but it doesn't change at all the overhead in the VM side to provide the information from disk to userspace (and the other way around for writes).

again, if you need to skip the overhead of passing the info to the usersapce you should write kernel code. If you want to take advantage of the protection of the userspace, you also need to pay for the pagetable settings or pagetable walking instead. Anything that knows physical pages is kernel code, writing userspace mapping /dev/mem is an illusion of writing userspace code, that's kernel code anyways. The framebuffer thing of X is different, X really only needs to talk to a device, it doesn't really care about the kernel, it doesn't need to communicate with the kernel, infact if it needs to communicate with the kernel to do that it needs proper kernel modules like the pci/drm/agpgart/mtrr kernel drivers. What the X server does with /dev/mem is no different than mmap("/dev/fb0") and using the so mapped framebuffer. For read and writes via pagecache or userspace memory, you instead need to enterely deal with kernel internals data structures instead that only the kernel knows, and that's why it's kernel code as soon as you want to bypass the protection layer of the pagetables.

One way to speedup overwrites would be to make faster the lookup from [virtual address,vma] to [physical page] using a data structure that would act as a software tlb, that could be attached to a vma, and to en abled with an mmap flag so you enable it only on the SGA, so an overwrite of disk data would be faster and you could skip the pagetable walking the second time, but if you seek an huge lot and the number of pages is huge, at least if they're not 4M pages, the number of entries to remeber would be quite huge and it would waste some ram, plus it wouldn't be a critical optimization, you would skip a few levels of indirection in the lookup, like a radix tree with only one level vs a radix tree with 3 levels of indirection and such data structure would need some sychronization during the vma modifications too.

the point is, why are you reading from disk, what are you going to do with such data? For example, assume you read from disk to find the string "xxx" in a file the fastest possible, then the right way to exploit the max bandwith is to write a kernel module that triggers the readpage asynchronously (trivial in kernel, just don't wait_on_page until you need the data), lookup the radix tree, kmap the page and search the string, kunmap, start a new async readpage, lookup the next page, wait_on_page the next page and so on. That's the only way to run at the core 175MB/sec speed, anything running in userspace would be slower (even kernel code will be slower than 175MB if kmap actually does something with the pte if you've more than 1G and CONFIG_3G selected etc..)

Now I'm not suggesting to move a database to kernel space for the sake of performance, the layer of protection is foundamental for the reliability of the system, but if you have special needs for certain special things, then a kernel module accellerator is the only way to go faster than buffered-IO/read/mmap/O_DIRECT/raw or whatever that makes that disk data visible from disk to userspace and that in turn needs to deal with the virtual mappings too and not only with the physical pages mapped in the kernel direct mapping.

Long email sorry.

Andrea

g'day Andrea,

..

i've had some spare time to enhance my benchmark program. it is now multithreaded and can do mmap() i/o also.

interestingly enough, it shows that mmap() actually sucks. i didn't take a "readprofile" to see why, but it used far more cpu time than anything else.

with the same hardware as before (Dual P3 Xeon 733MHz, 133MHz FSB, 2G PC133 SDRAM, QLogic 2300 HBA in 64/66 PCI slot) and same kernel (linux 2.4.19pre8 with vary-io, lockmeter & profile=2), the aggregate throughput across 8 x 15K RPM FC disks (2gbit/s FC switched network) are:

now that i've multithreaded the test app. the results change somewhat given we now have multiple threads issuing i/o operations:
raw i/o is slightly better than O_DIRECT again. mmap()'ed i/o is the worst.

[summary]
'base' Completed reading 81920 mbytes in 524.289107 seconds
(156.25 Mbytes/sec), 386usec mean/block
'nocopy' Completed reading 81920 mbytes in 435.653909 seconds
(188.04 Mbytes/sec), 302usec mean/block
'o_direct' Completed reading 81920 mbytes in 429.757513 seconds
(190.62 Mbytes/sec), 157655usec mean/block
'raw' Completed reading 81920 mbytes in 420.940382 seconds
(194.61 Mbytes/sec), 164368usec mean/block
'mmap' Completed reading 81920 mbytes in 1049.446053 seconds (78.06
Mbytes/sec), 401124usec mean/block

given i'm not hitting the peak performance of a single 2gbit/s fibre channel connection, i'll now migrate to dual FC HBAs and see what happens ... :-)

[detail]

(1)

# performance of a single disk-spindle using normal linux block i/o layer.  use 10G worth of 8kbyte blocks.
# this will show the peak performance of a single 15K RPM disk without any bus contention.
[root@mel-stglab-host1 root]#  ./test_disk_performance2 bs=8k blocks=1280k mode=basic /dev/sde
Completed reading 10240 mbytes across 1 devices using 1310720 blocks of 8192 in 190.028450 seconds (53.89 Mbytes/sec), 151usec mean

(2)

# performance of multiple disk-spindles (8 disks, sde-sdl) using normal linux block i/o layer.
# read 10G sequentially from each disk using 8kbyte blocks parallel across all disks.
[root@mel-stglab-host1 root]#  ./test_disk_performance2 bs=8k blocks=1280k mode=basic /dev/sd[e-l]
Completed reading 81920 mbytes across 8 devices using 10485760 blocks of 8192 in 524.289107 seconds (156.25 Mbytes/sec), 386usec mean
  device #0 (/dev/sde) 10240 megabytes in 520.588206 seconds using 1310720 
reads of 8192 bytes (19.67 Mbytes/sec), 320usec
  device #1 (/dev/sdf) 10240 megabytes in 509.305642 seconds using 1310720 
reads of 8192 bytes (20.11 Mbytes/sec), 352usec
  device #2 (/dev/sdg) 10240 megabytes in 516.744421 seconds using 1310720 
reads of 8192 bytes (19.82 Mbytes/sec), 351usec
  device #3 (/dev/sdh) 10240 megabytes in 519.215781 seconds using 1310720 
reads of 8192 bytes (19.72 Mbytes/sec), 382usec
  device #4 (/dev/sdi) 10240 megabytes in 517.312922 seconds using 1310720 
reads of 8192 bytes (19.79 Mbytes/sec), 339usec
  device #5 (/dev/sdj) 10240 megabytes in 524.289183 seconds using 1310720 
reads of 8192 bytes (19.53 Mbytes/sec), 315usec
  device #6 (/dev/sdk) 10240 megabytes in 521.006994 seconds using 1310720 
reads of 8192 bytes (19.65 Mbytes/sec), 322usec
  device #7 (/dev/sdl) 10240 megabytes in 518.434472 seconds using 1310720 
reads of 8192 bytes (19.75 Mbytes/sec), 345usec

(3)

# same test as above, but using 'nocopy hack' so PC architecture isn't the bottleneck.
# look at 8 disks (sde-sdl) using 8kbyte/block, 10G on each disk: (80G total) in parallel.
# using 'nocopy' means we can measure peak performance without PC front-side-bus becoming
# the bottleneck, but we're still using the normal read-file case.
[root@mel-stglab-host1 root]#  ./test_disk_performance2 bs=8k blocks=1280k mode=nocopy /dev/sd[e-l]
Completed reading 81920 mbytes across 8 devices using 10485760 blocks of 8192 in 435.653909 seconds (188.04 Mbytes/sec), 302usec mean
  device #0 (/dev/sde) 10240 megabytes in 408.566056 seconds using 1310720 
reads of 8192 bytes (25.06 Mbytes/sec), 305usec
  device #1 (/dev/sdf) 10240 megabytes in 411.353237 seconds using 1310720 
reads of 8192 bytes (24.89 Mbytes/sec), 309usec
  device #2 (/dev/sdg) 10240 megabytes in 414.244129 seconds using 1310720 
reads of 8192 bytes (24.72 Mbytes/sec), 295usec
  device #3 (/dev/sdh) 10240 megabytes in 416.225989 seconds using 1310720 
reads of 8192 bytes (24.60 Mbytes/sec), 293usec
  device #4 (/dev/sdi) 10240 megabytes in 417.593914 seconds using 1310720 
reads of 8192 bytes (24.52 Mbytes/sec), 268usec
  device #5 (/dev/sdj) 10240 megabytes in 422.646633 seconds using 1310720 
reads of 8192 bytes (24.23 Mbytes/sec), 259usec
  device #6 (/dev/sdk) 10240 megabytes in 431.343133 seconds using 1310720 
reads of 8192 bytes (23.74 Mbytes/sec), 209usec
  device #7 (/dev/sdl) 10240 megabytes in 435.654122 seconds using 1310720 
reads of 8192 bytes (23.50 Mbytes/sec), 216usec

(4)

# use O_DIRECT to bypass readahead and most of block-layer memory-copies
# since there isn't any readahead anymore, use a large buffer to achieve peak performance
[root@mel-stglab-host1 root]#  ./test_disk_performance2 bs=4m blocks=2560 mode=direct /dev/sd[e-l]
Completed reading 81920 mbytes across 8 devices using 20480 blocks of 4194304 in 429.757513 seconds (190.62 Mbytes/sec), 157655usec mean
  device #0 (/dev/sde) 10240 megabytes in 412.931460 seconds using 2560 
reads of 4194304 bytes (24.80 Mbytes/sec), 160898usec
  device #1 (/dev/sdf) 10240 megabytes in 414.531502 seconds using 2560 
reads of 4194304 bytes (24.70 Mbytes/sec), 159234usec
  device #2 (/dev/sdg) 10240 megabytes in 414.585683 seconds using 2560 
reads of 4194304 bytes (24.70 Mbytes/sec), 158187usec
  device #3 (/dev/sdh) 10240 megabytes in 415.257838 seconds using 2560 
reads of 4194304 bytes (24.66 Mbytes/sec), 154983usec
  device #4 (/dev/sdi) 10240 megabytes in 416.209758 seconds using 2560 
reads of 4194304 bytes (24.60 Mbytes/sec), 138563usec
  device #5 (/dev/sdj) 10240 megabytes in 419.144473 seconds using 2560 
reads of 4194304 bytes (24.43 Mbytes/sec), 133593usec
  device #6 (/dev/sdk) 10240 megabytes in 424.031534 seconds using 2560 
reads of 4194304 bytes (24.15 Mbytes/sec), 102836usec
  device #7 (/dev/sdl) 10240 megabytes in 429.773878 seconds using 2560 
reads of 4194304 bytes (23.83 Mbytes/sec), 98300usec

(5)

# same as (4) but using /dev/raw/rawX instead of O_DIRECT
[root@mel-stglab-host1 root]#  ./test_disk_performance2 bs=4m blocks=2560 mode=basic /dev/raw/raw[1-8]
Completed reading 81920 mbytes across 8 devices using 20480 blocks of 4194304 in 420.940382 seconds (194.61 Mbytes/sec), 164368usec mean
  device #0 (raw1) 10240 megabytes in 420.869676 seconds using 2560 reads 
of 4194304 bytes (24.33 Mbytes/sec), 164385usec
  device #1 (raw2) 10240 megabytes in 420.859291 seconds using 2560 reads 
of 4194304 bytes (24.33 Mbytes/sec), 164386usec
  device #2 (raw3) 10240 megabytes in 420.834023 seconds using 2560 reads 
of 4194304 bytes (24.33 Mbytes/sec), 164385usec
  device #3 (raw4) 10240 megabytes in 420.896964 seconds using 2560 reads 
of 4194304 bytes (24.33 Mbytes/sec), 163879usec
  device #4 (raw5) 10240 megabytes in 420.838551 seconds using 2560 reads 
of 4194304 bytes (24.33 Mbytes/sec), 164385usec
  device #5 (raw6) 10240 megabytes in 420.867115 seconds using 2560 reads 
of 4194304 bytes (24.33 Mbytes/sec), 164060usec
  device #6 (raw7) 10240 megabytes in 420.906691 seconds using 2560 reads 
of 4194304 bytes (24.33 Mbytes/sec), 164386usec
  device #7 (raw8) 10240 megabytes in 420.932318 seconds using 2560 reads 
of 4194304 bytes (24.33 Mbytes/sec), 164357usec

(6)

# use memory-mapped i/o, more for interests-sake looking at the efficiency of the virtual-mapping of the linux kernel.
# use a large blocksize as it was shown that a 8kbyte blocksize is about 40% of the performance with a 4m blocksize
[root@mel-stglab-host1 root]#  ./test_disk_performance2 bs=4m blocks=2560 mode=mmap /dev/sd[e-l]
Completed reading 81920 mbytes across 8 devices using 20480 blocks of 4194304 in 1049.446053 seconds (78.06 Mbytes/sec), 401124usec mean
  device #0 (/dev/sde) 10240 megabytes in 1045.814369 seconds using 2560 
reads of 4194304 bytes (9.79 Mbytes/sec), 342912usec
  device #1 (/dev/sdf) 10240 megabytes in 1034.783187 seconds using 2560 
reads of 4194304 bytes (9.90 Mbytes/sec), 430320usec
  device #2 (/dev/sdg) 10240 megabytes in 1046.222514 seconds using 2560 
reads of 4194304 bytes (9.79 Mbytes/sec), 340732usec
  device #3 (/dev/sdh) 10240 megabytes in 1049.445858 seconds using 2560 
reads of 4194304 bytes (9.76 Mbytes/sec), 353895usec
  device #4 (/dev/sdi) 10240 megabytes in 1047.926118 seconds using 2560 
reads of 4194304 bytes (9.77 Mbytes/sec), 362176usec
  device #5 (/dev/sdj) 10240 megabytes in 1047.454030 seconds using 2560 
reads of 4194304 bytes (9.78 Mbytes/sec), 377308usec
  device #6 (/dev/sdk) 10240 megabytes in 1043.274980 seconds using 2560 
reads of 4194304 bytes (9.82 Mbytes/sec), 387720usec
  device #7 (/dev/sdl) 10240 megabytes in 1048.326277 seconds using 2560 
reads of 4194304 bytes (9.77 Mbytes/sec), 347532usec

[ ignoring the fact that O_DIRECT and /dev/rawN are faster than 'nocopy' for a moment ... :-) ]

around 18 months ago, i had it working.

basically, there was a character device-driver which allowed user-space to mmap() kernel space.
a seperate 'zone' existed that was reserved exclusively for data to/from disk and network.
this zone was sized to 75% of RAM -- and (on the hardware being used) with 2G RAM, meant that there was 1.5GB available in this space.

the kernel would indicate the physical address of a "skb" on network sockets and each packet put on the skbuff receive_q for a socket marked for 'zerocopy' would result in a zerocopy_buf_t being queued to user-space. user-space could poll to get a list of the current zerocopy_buf_t's. (this list wasn't copied to userspace using copy_to_user; user-space picked this list up via the mmap() into kernel space too)

userspace could control what to do with buffers by various ioctl() calls on the character-device. things like marking "i'm not interested in that buffer anymore" would result in atomic_dec_and_test() on the refcount and a subsequent skb_free() if there wasn't any more interest in the buffer. there were also ioctl()s for queueing the buffer on a different socket.

yes -- rather radical -- but it did bypass the bottleneck of memory-bandwidth becoming the bottleneck in an application which predominantly did:
- read-from-disk, send on network socket
- read from network socket, write to disk
- read from network socket, write to alternate network socket, write to disk

given user-space could crash at any point, there was some housekeeping that ensured that buffers tied to a process that had failed didn't stay around - they were cleaned up.

there was also some rudimentary checks to ensure that user-space passing buffer pointers to kernel-space was passing buffers that kernel-space already knew about. ie. you couldn't crash the kernel by passing a bad pointer.

i haven't resurrected this work since the VM underwent some radical changes -- but it wouldn't be too hard to do. it'd still be a bit ugly in the tie-ins to the tcp stack and buffer-cache but ....

i guess given O_DIRECT and /dev/rawN performance for to-disk/from-disk being what it currently is (very good), there's less importance on having it, if we could find a 'fast' way to queue to/from disk from/to network. perhaps Ben's async i/o is the way to go for those apps.

i was travelling and it gave me a chance to test out the 8 hour battery life in my new laptop on a 14 hour flight. :-)

cheers,

lincoln.

follow-page and get_user_pages is the actual cpu cost of walking the pagetables. that could be trimmed down by wasting some memory for an efficient software-kernel-side tlb.

the cpu cost is much smaller than base as you can see and most of it will be optimized away with vary-io that should lead to follow_page and get_user_pages to move down in the above profiling.

as expected the biggest cost is file_read_actor.

Both profiling looks fine, as expected.

expected again, as you can see the cost goes quite up for __make_request compared to O_DIRECT due the 512 b_size (raw compared to base is unfair because base uses 1k b_size and raw uses 512 b_size and that's why it's wasting so much cpu time there, o_direct vs base is instead fair because they both uses 1k of b_size, once o_direct will take advantage of varyio in your upgraded driver, the comparison between o_direct vs base will become unfair too, o_direct will be more advantaged by a virtual-common 4k b_size)

very similar to o_direct, notice the overhead in _spin_lock_ here (and also in "base") it's certainly the page_cache lock that won't go away in 2.5 with radix tree because you're working on the same file, higher bandwith because the disks runs at the same time and possibly because read/write returns faster and part of the cost of the I/O happens outside your time measurements (for a more fair comparison you can benchmark the whole workload, not only how fast read/write returns).

Andrea

Depends. For IDE, that is so. For SCSI, actually no io_request_lock is not held while doing the bounce copy. The write bounce copy never happens with io_request_lock held for either, the copy-back on reads only does if the caller holds io_request_lock while entering end_that_request_first() (or its own replacement, __scsi_end_request() for instance).

The read copy-back is nasty for most users regardless of io_request_lock status, because it happens with interrupts disabled.

That's good advise :-)

Jens Axboe

yes, that's the main factor. Suggested read/write buffer size with O_DIRECT are of the order of 512k at least (that's the high limit for a single scsi dma request in 2.4), and 1M even better so you I/O pipeline some more in the ll_rw_block layer.

Andrea

is any of the disks mounted?

can you post your test_disk_performance program so I in particular we can see the semantics of blocks and bs? 64k*8k == 5k * 1M / 10.

O_DIRECT has to do some more work to check for the coherency with the pagecache and it has some more overhead with the address space operations, but O_DIRECT by default uses the blocksize of the blkdev, that is set to 1k by default (if you never mounted it) versus the hardblocksize of 512bytes used by the raw device (assuming the sd[m-r] aren't mounted).

This is most probably why O_DIRECT is faster than raw.c, otherwise they would run almost at the same rate, the pagecache coherency fast paths and the address space ops overhead of O_DIRECT shouldn't be noticeable.