Skip to contentSkip to conversation

o_sync in vfat driver

DISCUSSION [email protected] 14 participants 33 messages Feb 26, 2006 – Mar 2, 2006 8 messages not in the archive

lore
Share & tools
lore
#8 lore raw
In reply to a message not in the archive

Hi,

OMG what do I have to do to post here? 10th attempt. {part2}

Here is a non-exhaustive list of typical devices types requiring fat vfat support:

fd ide-hd scsi-hd usb-hd cdrom usb-hd usb-handheld (iPod, iRiver etc) usb-flash (usbsticks, cameras, some music devices.)

IIRC the sync mount option for vfat is ignored for file systems >2G, this effectively (and probably intentionally) excludes nearly all hd partitions and iPod type devices.

sync does not have any meaning for CD DVD media.

#9 lore raw
Quoted 8 lines · On Sun, Feb 26, 2006 at 11:50:40PM +0100, [email protected] wrote:

Hi,

OMG what do I have to do to post here? 10th attempt. {part2}

Here is a non-exhaustive list of typical devices types requiring fat vfat support:

Quoted 6 lines

fd ide-hd scsi-hd usb-hd cdrom usb-hd usb-handheld (iPod, iRiver etc) usb-flash (usbsticks, cameras, some music devices.)

IIRC the sync mount option for vfat is ignored for file systems >2G, this effectively (and probably intentionally) excludes nearly all hd partitions and iPod type devices.

I think many people wish it was ignored on smaller devices too given what it does to write performance. And if your device is flash based and is one of the ones that doesn't have proper wear leveling the card won't last long with sync enabled (even with wear leveling rewriting the fat that often as sync seems to do can't be good for the lifespan of the flash).

I suspect either vfat should ignore sync all the time, or it should at least warn about its use so distributions don't think enabling it on all removeable media is a good idea in general. Or perhaps the vfat driver could be made to wait for a file to be closed or at least have some timeout before updating the fat table again. Not sure.

Len Sorensen

#10 lore raw
Quoted 12 lines · Lennart Sorensen
Quoted 11 lines · On Sun, Feb 26, 2006 at 11:50:40PM +0100, [email protected] wrote:

Hi,

OMG what do I have to do to post here? 10th attempt. {part2}

Here is a non-exhaustive list of typical devices types requiring fat vfat support:

fd ide-hd scsi-hd usb-hd cdrom usb-hd usb-handheld (iPod, iRiver etc) usb-flash (usbsticks, cameras, some music devices.)

Quoted 6 lines
Quoted 3 lines

IIRC the sync mount option for vfat is ignored for file systems >2G, this effectively (and probably intentionally) excludes nearly all hd partitions and iPod type devices.

I think many people wish it was ignored on smaller devices too given what it does to write performance.

well. If you don't want it *DO NOT USE IT AT THE MOUNT COMMAND LINE* !!!

Quoted 5 lines

And if your device is flash based
and is one of the ones that doesn't have proper wear leveling the card won't last long with sync enabled (even with wear leveling rewriting the fat that often as sync seems to do can't be good for the lifespan of the flash).

patient> doctor doctor it hurts when I do this
doctor> Then don't do that

#11 lore raw
Quoted 13 lines · Arjan van de Ven
Quoted 12 lines · Lennart Sorensen
Quoted 11 lines · On Sun, Feb 26, 2006 at 11:50:40PM +0100, [email protected] wrote:

Hi,

OMG what do I have to do to post here? 10th attempt. {part2}

Here is a non-exhaustive list of typical devices types requiring fat vfat support:

fd ide-hd scsi-hd usb-hd cdrom usb-hd usb-handheld (iPod, iRiver etc) usb-flash (usbsticks, cameras, some music devices.)

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

IIRC the sync mount option for vfat is ignored for file systems >2G, this effectively (and probably intentionally) excludes nearly all hd partitions and iPod type devices.

I think many people wish it was ignored on smaller devices too given what it does to write performance.

well. If you don't want it *DO NOT USE IT AT THE MOUNT COMMAND LINE* !!!

That is easy to say when you are using the command line... Modern distros (as you know I am sure) mount all hot-plug devices like usb keys, usb hard disks, etc automatically at plug-in time and at least some distros use "-o sync" for everything so you don't get (too much) data loss when the user unplugs a device and so a umount to unplug the device does not take ages...

Being someone who maintains a distribution based on one of the big distributions I can tell you that figuring out how to change that default behaviour is not always pretty. Usually involves hacking files deep in the bowels of the hotplug framework on the system.

Best regards,

Anton

Signature · 4 lines
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/
#39 lore raw
In reply to Lennart Sorensen
Quoted 23 lines · Lennart Sorensen
Quoted 8 lines · On Sun, Feb 26, 2006 at 11:50:40PM +0100, [email protected] wrote:

Hi,

OMG what do I have to do to post here? 10th attempt. {part2}

Here is a non-exhaustive list of typical devices types requiring fat vfat support:

Quoted 6 lines

fd ide-hd scsi-hd usb-hd cdrom usb-hd usb-handheld (iPod, iRiver etc) usb-flash (usbsticks, cameras, some music devices.)

IIRC the sync mount option for vfat is ignored for file systems >2G, this effectively (and probably intentionally) excludes nearly all hd partitions and iPod type devices.

I think many people wish it was ignored on smaller devices too given what it does to write performance. And if your device is flash based and is one of the ones that doesn't have proper wear leveling the card won't last long with sync enabled (even with wear leveling rewriting the fat that often as sync seems to do can't be good for the lifespan of the flash).

Quoted 7 lines

I suspect either vfat should ignore sync all the time, or it should at least warn about its use so distributions don't think enabling it on all removeable media is a good idea in general. Or perhaps the vfat driver could be made to wait for a file to be closed or at least have some timeout before updating the fat table again. Not sure.

Len Sorensen

I really don't think one needs to worry about this! The flash-file- system designers know how to minimize wear and spread the wear throughout the device. It's not up to the file-systems to be concerned whatsoever! The filesystems need to concern themselves with the proper implementation of their structural details, nothing else. Any special device considerations do not belong in the file-system code. If there are any special device considerations, they need to be in the device driver, nowhere else.

BYW, even the drivers can't effectively compensate for any potential wear because they don't know where the physical write will occur. The physical sectors (pages) of many of these devices are 64k. All of the access, both read and write, is buffered in read/write static RAM. It's only when the disk emulator of the FlashRAM decides that the static RAM needs to be flushed to flash, that the write actually occurs. Typically, a LRU 64k page is erased and re-written. Then a table is updated to reference the new correct block. This is all transparent, and it needs to be, because erasing a 64k block takes nearly a second! Without the buffering, write performance would be unacceptable.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.15.4 on an i686 machine (5589.53 BogoMips). Warning : 98.36% of all statistics are fiction, book release in April. _


****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

#12 lore raw
In reply to Anton Altaparmakov
Quoted 21 lines · Anton Altaparmakov
Quoted 20 lines · Arjan van de Ven
Quoted 18 lines · Lennart Sorensen
Quoted 8 lines · On Sun, Feb 26, 2006 at 11:50:40PM +0100, [email protected] wrote:

Hi,

OMG what do I have to do to post here? 10th attempt. {part2}

Here is a non-exhaustive list of typical devices types requiring fat vfat support:

Quoted 6 lines

fd ide-hd scsi-hd usb-hd cdrom usb-hd usb-handheld (iPod, iRiver etc) usb-flash (usbsticks, cameras, some music devices.)

IIRC the sync mount option for vfat is ignored for file systems >2G, this effectively (and probably intentionally) excludes nearly all hd partitions and iPod type devices.

I think many people wish it was ignored on smaller devices too given what it does to write performance.

Quoted 6 lines
Quoted 1 line

well. If you don't want it *DO NOT USE IT AT THE MOUNT COMMAND LINE* !!!

That is easy to say when you are using the command line... Modern distros (as you know I am sure) mount all hot-plug devices like usb keys, usb hard disks, etc automatically at plug-in time and at least some distros use "-o sync"

that is a bad misdesign of that distro or at least the tool the distro uses for this (I don't know which it is so I can say that without sounding partial :)

the tool that decides to use "sync", or at least the author thereof, should be aware of what flash is, and that it has a limited lifespan etc etc, and that you thus want maximum caching etc.

#13 lore raw
Quoted 29 lines · Arjan van de Ven
Quoted 21 lines · Anton Altaparmakov
Quoted 20 lines · Arjan van de Ven
Quoted 18 lines · Lennart Sorensen
Quoted 8 lines · On Sun, Feb 26, 2006 at 11:50:40PM +0100, [email protected] wrote:

Hi,

OMG what do I have to do to post here? 10th attempt. {part2}

Here is a non-exhaustive list of typical devices types requiring fat vfat support:

Quoted 6 lines

fd ide-hd scsi-hd usb-hd cdrom usb-hd usb-handheld (iPod, iRiver etc) usb-flash (usbsticks, cameras, some music devices.)

IIRC the sync mount option for vfat is ignored for file systems >2G, this effectively (and probably intentionally) excludes nearly all hd partitions and iPod type devices.

I think many people wish it was ignored on smaller devices too given what it does to write performance.

Quoted 6 lines
Quoted 1 line

well. If you don't want it *DO NOT USE IT AT THE MOUNT COMMAND LINE* !!!

That is easy to say when you are using the command line... Modern distros (as you know I am sure) mount all hot-plug devices like usb keys, usb hard disks, etc automatically at plug-in time and at least some distros use "-o sync"

Quoted 7 lines

that is a bad misdesign of that distro or at least the tool the distro uses for this (I don't know which it is so I can say that without sounding partial :)

the tool that decides to use "sync", or at least the author thereof, should be aware of what flash is, and that it has a limited lifespan etc etc, and that you thus want maximum caching etc.

I agree completely which is why we hack the system to remove the o_sync on our distro derivative. (-:

But my point was that your solution of "don't do that then" is not much use to your average user who sits in front of such distro in graphical desktop as they are not technical enough to find and hack their hotplug system to work properly...

Best regards,

Anton

Signature · 4 lines
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/
#40 lore raw
In reply to linux-os (Dick Johnson)
Quoted 5 lines

I really don't think one needs to worry about this! The flash-file- system designers know how to minimize wear and spread the wear throughout the device. It's not up to the file-systems to be concerned whatsoever!

Yes, the filesystem designers, JFFS and such. But most people unfortunately have to use something not-optimized-for-flash called VFAT to be able to read it on Win32 too. I would like to use UDF instead, but Windows seems to have a nogo with UDF on non-CDROMs.

Jan Engelhardt

#14 lore raw
In reply to Anton Altaparmakov
Quoted 48 lines · Anton Altaparmakov
Quoted 36 lines · Arjan van de Ven
Quoted 3 lines · Anton Altaparmakov
Quoted 2 lines · Arjan van de Ven
Quoted 1 line · Lennart Sorensen

On Sun, Feb 26, 2006 at 11:50:40PM +0100, [email protected]

Quoted 6 lines · wrote:
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Hi,

OMG what do I have to do to post here? 10th attempt. {part2}

Here is a non-exhaustive list of typical devices types

requiring fat vfat

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

support:

fd ide-hd scsi-hd usb-hd cdrom usb-hd usb-handheld (iPod,

iRiver etc)

Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

usb-flash (usbsticks, cameras, some music devices.)

IIRC the sync mount option for vfat is ignored for file systems

2G, this

Quoted 1 line
Quoted 1 line
Quoted 1 line

effectively (and probably intentionally) excludes nearly all hd

partitions

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

and iPod type devices.

I think many people wish it was ignored on smaller devices too

given

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

what it does to write performance.

well. If you don't want it *DO NOT USE IT AT THE MOUNT COMMAND

LINE* !!!

Quoted 5 lines

That is easy to say when you are using the command line... Modern distros (as you know I am sure) mount all hot-plug devices like usb keys, usb hard disks, etc automatically at plug-in time and at least some distros use "-o sync"

Quoted 7 lines

that is a bad misdesign of that distro or at least the tool the distro uses for this (I don't know which it is so I can say that without sounding partial :)

the tool that decides to use "sync", or at least the author thereof, should be aware of what flash is, and that it has a limited lifespan etc etc, and that you thus want maximum caching etc.

I agree completely which is why we hack the system to remove the o_sync on our distro derivative. (-:

Quoted 10 lines

But my point was that your solution of "don't do that then" is not much use to your average user who sits in front of such distro in graphical desktop as they are not technical enough to find and hack their hotplug system to work properly...

Best regards,

Anton

Quoted 1 line

If you don't want it *DO NOT USE IT AT THE MOUNT COMMAND LINE* !!!

Yeah, cleaver.
That is not really a constructive responce. I dont use , I do use command line mount all the time. I never was in danger of damaging my drive with this new "feature".

Telling a user who has just burnt out a brand new 1GB usb device he should have RTFM and modified that HAL configuration to insure it did not use sync it not likely to win much confidence in the linux kernel.

The point of raising this is that the vast majority of linux users have no awareness of this. If there is a danger of this sync implementation damaging hardware it should be done differently.

More importantly this sync strategy is very likely _increasing_ the danger of data loss that is the core reason for using sync in the first place.

To quote from my earlier post:

The new model attempts to be more rigourous by updating the FAT every time a block of data is written. Thus the "hammering" of the physical memory hosting the FAT record.

In view of the nature of flash memory this may actually be drastically increasing the chance that the whole FAT gets erased.

If a pullout occurs during write , there is now a near 50% chance that this takes out the entire FAT.

Now if that analysis is inaccurate I'd like be corrected. But flash has to be zeroed to be written. If every second write is zeroing the FAT this would seem much more likely to destroy the whole fs than to provide better protection from a untimely pull-out.

[Note: I am not subscribed to LKML, if you wish me to recieve any follow ups please BCC: col-pepper at piments point com . thx]

#15 lore raw
Quoted 4 lines

Telling a user who has just burnt out a brand new 1GB usb device he should have RTFM and modified that HAL configuration to insure it did not use sync it not likely to win much confidence in the linux kernel.

or in HAL. really.

there was a very long discussion abuot kernel stability. The problem is that once depending on the absence of a feature becomes ABI ... there is a big problem.

#17 lore raw
In reply to [email protected]
Quoted 75 lines · On Mon, 27 Feb 2006 [email protected] wrote:
Quoted 48 lines · Anton Altaparmakov
Quoted 36 lines · Arjan van de Ven
Quoted 3 lines · Anton Altaparmakov
Quoted 2 lines · Arjan van de Ven
Quoted 1 line · Lennart Sorensen

On Sun, Feb 26, 2006 at 11:50:40PM +0100, [email protected]

Quoted 6 lines · wrote:
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Hi,

OMG what do I have to do to post here? 10th attempt. {part2}

Here is a non-exhaustive list of typical devices types

requiring fat vfat

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

support:

fd ide-hd scsi-hd usb-hd cdrom usb-hd usb-handheld (iPod,

iRiver etc)

Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

usb-flash (usbsticks, cameras, some music devices.)

IIRC the sync mount option for vfat is ignored for file systems

2G, this

Quoted 1 line
Quoted 1 line
Quoted 1 line

effectively (and probably intentionally) excludes nearly all hd

partitions

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

and iPod type devices.

I think many people wish it was ignored on smaller devices too

given

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

what it does to write performance.

well. If you don't want it *DO NOT USE IT AT THE MOUNT COMMAND

LINE* !!!

Quoted 5 lines

That is easy to say when you are using the command line... Modern distros (as you know I am sure) mount all hot-plug devices like usb keys, usb hard disks, etc automatically at plug-in time and at least some distros use "-o sync"

Quoted 7 lines

that is a bad misdesign of that distro or at least the tool the distro uses for this (I don't know which it is so I can say that without sounding partial :)

the tool that decides to use "sync", or at least the author thereof, should be aware of what flash is, and that it has a limited lifespan etc etc, and that you thus want maximum caching etc.

I agree completely which is why we hack the system to remove the o_sync on our distro derivative. (-:

Quoted 10 lines

But my point was that your solution of "don't do that then" is not much use to your average user who sits in front of such distro in graphical desktop as they are not technical enough to find and hack their hotplug system to work properly...

Best regards,

Anton

Quoted 1 line

If you don't want it *DO NOT USE IT AT THE MOUNT COMMAND LINE* !!!

Yeah, cleaver.
That is not really a constructive responce. I dont use , I do use command line mount all the time. I never was in danger of damaging my drive with this new "feature".

Telling a user who has just burnt out a brand new 1GB usb device he should have RTFM and modified that HAL configuration to insure it did not use sync it not likely to win much confidence in the linux kernel.

The point of raising this is that the vast majority of linux users have no awareness of this. If there is a danger of this sync implementation damaging hardware it should be done differently.

Quoted 8 lines

More importantly this sync strategy is very likely _increasing_ the danger of data loss that is the core reason for using sync in the first place.

To quote from my earlier post:

The new model attempts to be more rigourous by updating the FAT every time a block of data is written. Thus the "hammering" of the physical memory hosting the FAT record.

Nobody should care.

Quoted 4 lines

In view of the nature of flash memory this may actually be drastically increasing the chance that the whole FAT gets erased.

Will not happen because that's not how they work.

Quoted 3 lines

If a pullout occurs during write , there is now a near 50% chance that this takes out the entire FAT.

If a pullout or a power-failure occurs, you just have an incomplete write, an old FAT entry just like ejecting a floppy during a write.

Quoted 5 lines

Now if that analysis is inaccurate I'd like be corrected. But flash has to be zeroed to be written. If every second write is zeroing the FAT this would seem much more likely to destroy the whole fs than to provide better protection from a untimely pull-out.

Flash does not get zeroed to be written! It gets erased, which sets all the bits to '1', i.e., all bytes to 0xff. Further, the designers of flash disks are not stupid as you assume. The direct access occurs to static RAM (read/write stuff). After a few milliseconds of it becoming dirty, and/or when a new page needs to be accessed, the chip erases some page that was not used yet, or was used a long time ago and is not on the active list. Then, it becomes buzy, writes the current sector to the newly erased sector, and (after that write occurs) replaces the entry in the table that tells the disk implimentation the logical to physical translation of that page. In the case where a page will be changed, the new page's data is read from the device into static RAM before access. In any case, the chip then becomes non-buzy. The power can fail at any time and you just have the previous data instead of the new data, just like a real disk drive, except that the sectors are large (64 k).

You see, these are not just flash-RAM chips. They are disc drive emulators that contain an ASIC for the bus interface and control logic, some static RAM, and the flash RAM.

The IDE emulators, like CompaqFlash, as tiny as they are, actually have the same pin-outs as an IDE drive!!

Quoted 9 lines

[Note: I am not subscribed to LKML, if you wish me to recieve any follow ups please BCC: col-pepper at piments point com . thx]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Cheers,
Dick Johnson
Penguin : Linux version 2.6.15.4 on an i686 machine (5589.53 BogoMips). Warning : 98.36% of all statistics are fiction, book release in April. _


****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

#16 lore raw
In reply to Arjan van de Ven
Quoted 7 lines · Arjan van de Ven
Quoted 5 lines

Telling a user who has just burnt out a brand new 1GB usb device he should
have RTFM and modified that HAL configuration to insure it did not use sync it not likely to win much confidence in the linux kernel.

or in HAL. really.

It may unfairly reflect on HAL in the users' mind but hal still does exactly what it is set up to do.

Quoted 8 lines

there was a very long discussion abuot kernel stability. The problem is that once depending on the absence of a feature becomes ABI ... there is a big problem.

It was not totally absent. If it was absent no-one would configure anything to use it anyway. It seems that big problem was that it functionality was fundamentlly changed but it was passed on like a minor mod that no-one needed to worry about and the doc was not updated at the time.

#18 lore raw
In reply to linux-os (Dick Johnson)
Quoted 2 lines · linux-os (Dick Johnson)

Flash does not get zeroed to be written! It gets erased, which sets all the bits to '1', i.e., all bytes to 0xff.

Thanks for the correction, but that does not change the discussion.

Quoted 3 lines

Further, the designers of
flash disks are not stupid as you assume. The direct access occurs to static RAM (read/write stuff).

I'm not assuming anything . Some hardware has been killed by this issue.
http://lkml.org/lkml/2005/5/13/144

It seems that it's you making the assumption that all of these devices are manufactured the same way.

The constant dirtying of the buffer will still cause excessive use of the flash block hosting the FAT. Clearly not all devices use a load spreading mechanism and this can lead to premature failure.

#19 lore raw
Quoted 6 lines · On Mon, 27 Feb 2006 [email protected] wrote:
Quoted 2 lines · linux-os (Dick Johnson)

Flash does not get zeroed to be written! It gets erased, which sets all the bits to '1', i.e., all bytes to 0xff.

Quoted 8 lines

Thanks for the correction, but that does not change the discussion.

Quoted 3 lines

Further, the designers of
flash disks are not stupid as you assume. The direct access occurs to static RAM (read/write stuff).

I'm not assuming anything . Some hardware has been killed by this issue.
http://lkml.org/lkml/2005/5/13/144

No. That hardware was not killed by that issue. The writer, or another who had encountered the same issue, eventually repartitioned and reformatted the device. The partition table had gotten corrupted by some experiments and the writer assumed that the device was broken. It wasn't.

Also, if you read other elements in this thread, you would have learned about something that has become somewhat of a red herring.

It takes about a second to erase a 64k physical sector. This is a required operation before it is written. Since the projected life of these new devices is about 5 to 10 million such cycles, (older NAND flash used in modems was only 100-200k) the writer would have to be running that "brand new device" for at least 5 million seconds. Let's see:

60 seconds per minute
3600 seconds per hour
86400 seconds per day.

5,000,000 / 86400 = 57 days of continuous writes to the same sector. The writer surely would have a strange file because he states that even a single large file can destroy the drive if it is mounted with the "sync" option.

Also, the failure mode of NAND flash is not that it becomes "destroyed". The failure mode is a slow loss of data. The devices no longer retain data for a zillion years, only a few hundred, eventually, only a year or so. So when somebody claims that the flash has gotten destroyed, they need to have written it for a few months, then waited for a few years before reporting the event.

Clearly the writer is wrong.

Quoted 8 lines

It seems that it's you making the assumption that all of these devices are manufactured the same way.

The constant dirtying of the buffer will still cause excessive use of the flash block hosting the FAT. Clearly not all devices use a load spreading mechanism and this can lead to premature failure.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.15.4 on an i686 machine (5589.53 BogoMips). Warning : 98.36% of all statistics are fiction, book release in April. _


****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

#20 lore raw
Quoted 5 lines · linux-os \(Dick Johnson\)
Quoted 3 lines · On Mon, 27 Feb 2006 [email protected] wrote:

On Mon, 27 Feb 2006 22:32:07 +0100, linux-os (Dick Johnson) <[email protected]> wrote:

[...]

Quoted 6 lines

It takes about a second to erase a 64k physical sector. This is a required operation before it is written.
Since the projected life of these new devices is about 5 to 10 million such cycles, (older NAND flash used in modems was only 100-200k) the writer would have to be running that "brand new device" for at least 5 million seconds. Let's see:

What FLASH are you talking about? I work with NAND FLASH chips directly in embedded projects, and for both Toshiba and Samsung NAND FLASH the erase time of 128Kb (64K words) block is 2 milliseconds typical. Page program time is 0.3 milliseconds typical, so, having 64 pages per block, total erase-write block cycle is about 22ms.

Those chips indeed support about 100K program/erase cycles. Well, maybe there are some new NAND FLASH chips that support more program/erase cycles (just checked Samsung but found none), but I doubt they are 1000 times slower for block erase anyway.

-- Sergei.

#21 lore raw
In reply to linux-os (Dick Johnson)
Quoted 27 lines · linux-os (Dick Johnson)
Quoted 6 lines · On Mon, 27 Feb 2006 [email protected] wrote:
Quoted 2 lines · linux-os (Dick Johnson)

Flash does not get zeroed to be written! It gets erased, which sets all the bits to '1', i.e., all bytes to 0xff.

Quoted 8 lines

Thanks for the correction, but that does not change the discussion.

Quoted 3 lines

Further, the designers of
flash disks are not stupid as you assume. The direct access occurs to static RAM (read/write stuff).

I'm not assuming anything . Some hardware has been killed by this issue.
http://lkml.org/lkml/2005/5/13/144

No. That hardware was not killed by that issue. The writer, or another who had encountered the same issue, eventually repartitioned and reformatted the device. The partition table had gotten corrupted by some experiments and the writer assumed that the device was broken. It wasn't.

Also, if you read other elements in this thread, you would have learned about something that has become somewhat of a red herring.

Quoted 6 lines

It takes about a second to erase a 64k physical sector. This is a required operation before it is written. Since the projected life of these new devices is about 5 to 10 million such cycles, (older NAND flash used in modems was only 100-200k) the writer would have to be running that "brand new device" for at least 5 million seconds. Let's see:

How come I can write to my compact flash at about 2M/s if you claim it takes 1s to erase a 64k sector? Somehow I think your number is much too high. Or it can do multiple erases at the same time.

Also the 5 to 10 million is a lot higher than the numbers the makers of the compact flash cards I use claim.

Quoted 9 lines

60 seconds per minute
3600 seconds per hour
86400 seconds per day.

5,000,000 / 86400 = 57 days of continuous writes to the same sector. The writer surely would have a strange file because he states that even a single large file can destroy the drive if it is mounted with the "sync" option.

Quoted 7 lines

Also, the failure mode of NAND flash is not that it becomes "destroyed". The failure mode is a slow loss of data. The devices no longer retain data for a zillion years, only a few hundred, eventually, only a year or so. So when somebody claims that the flash has gotten destroyed, they need to have written it for a few months, then waited for a few years before reporting the event.

Some flash devices can be "destroyed" by loosing power in the middle of a write, since this causes them to corrupt their table of blocks and only the manufacturer has the ability to reset that. Fortunately this doesn't seem like too common a design.

Len Sorensen

#37 lore raw
In reply to [email protected]
Quoted 62 lines · [email protected] wrote:
Quoted 58 lines · Anton Altaparmakov
Quoted 36 lines · Arjan van de Ven
Quoted 3 lines · Anton Altaparmakov
Quoted 2 lines · Arjan van de Ven
Quoted 1 line · Lennart Sorensen

On Sun, Feb 26, 2006 at 11:50:40PM +0100,

Quoted 6 lines · [email protected] wrote:
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Hi,

OMG what do I have to do to post here? 10th attempt. {part2}

Here is a non-exhaustive list of typical devices types

requiring fat vfat

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

support:

fd ide-hd scsi-hd usb-hd cdrom usb-hd usb-handheld (iPod,

iRiver etc)

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

usb-flash (usbsticks, cameras, some music devices.)

IIRC the sync mount option for vfat is ignored for file

systems >2G, this

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

effectively (and probably intentionally) excludes nearly all

hd partitions

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

and iPod type devices.

I think many people wish it was ignored on smaller devices

too given

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

what it does to write performance.

well. If you don't want it *DO NOT USE IT AT THE MOUNT COMMAND

LINE* !!!

Quoted 5 lines

That is easy to say when you are using the command line... Modern distros (as you know I am sure) mount all hot-plug devices like usb keys, usb hard disks, etc automatically at plug-in time and at least some distros use "-o sync"

Quoted 7 lines

that is a bad misdesign of that distro or at least the tool the distro uses for this (I don't know which it is so I can say that without sounding partial :)

the tool that decides to use "sync", or at least the author thereof, should be aware of what flash is, and that it has a limited lifespan etc etc, and that you thus want maximum caching etc.

I agree completely which is why we hack the system to remove the o_sync on our distro derivative. (-:

But my point was that your solution of "don't do that then" is not much use to your average user who sits in front of such distro in graphical desktop as they are not technical enough to find and hack their hotplug system to work properly...

Best regards,

Anton

Quoted 7 lines
Quoted 2 lines
Quoted 1 line

If you don't want it *DO NOT USE IT AT THE MOUNT COMMAND LINE* !!!

Yeah, cleaver.
That is not really a constructive responce. I dont use , I do use command line mount all the time. I never was in danger of damaging my drive with this new "feature".

Telling a user who has just burnt out a brand new 1GB usb device he should have RTFM and modified that HAL configuration to insure it did not use sync it not likely to win much confidence in the linux kernel.

No problem in the kernel. The system is set up wrong. A simple user may not be able to figure out his distro's hotplug setup to fix this - but then this problem is the fault of _the distro_, not the kernel. Complain to distributors instead.

There is no need for the kernel to treat o_sync VFAT in any special way. The users, or more likely the distros, can skip that o_sync part.

Not all distros have such problems either. On debian, I had to set up /etc/fstab myself - where not specifying sync is easy enough.

Quoted 2 lines

The point of raising this is that the vast majority of linux users have no awareness of this. If there is a danger of this sync implementation damaging hardware it should be done differently.

Which is why people is working on the "sync on close" alternative.

Quoted 9 lines

More importantly this sync strategy is very likely _increasing_ the danger of data loss that is the core reason for using sync in the first place.

To quote from my earlier post:

The new model attempts to be more rigourous by updating the FAT every time a block of data is written. Thus the "hammering" of the physical memory hosting the FAT record.

Quoted 5 lines

In view of the nature of flash memory this may actually be drastically increasing the chance that the whole FAT gets erased.

If a pullout occurs during write , there is now a near 50% chance that this takes out the entire FAT.

No, only one FAT entry. And the users who pull out during writes _really_ get what they deserve anyway. You don't need deep linux knowledge for that. In the day of the floppy, people respected the activity light regardless of OS.

Helge Hafting

#22 lore raw
In reply to Lennart Sorensen
Quoted 35 lines · Lennart Sorensen
Quoted 27 lines · linux-os (Dick Johnson)
Quoted 6 lines · On Mon, 27 Feb 2006 [email protected] wrote:
Quoted 2 lines · linux-os (Dick Johnson)

Flash does not get zeroed to be written! It gets erased, which sets all the bits to '1', i.e., all bytes to 0xff.

Quoted 8 lines

Thanks for the correction, but that does not change the discussion.

Quoted 3 lines

Further, the designers of
flash disks are not stupid as you assume. The direct access occurs to static RAM (read/write stuff).

I'm not assuming anything . Some hardware has been killed by this issue.
http://lkml.org/lkml/2005/5/13/144

No. That hardware was not killed by that issue. The writer, or another who had encountered the same issue, eventually repartitioned and reformatted the device. The partition table had gotten corrupted by some experiments and the writer assumed that the device was broken. It wasn't.

Also, if you read other elements in this thread, you would have learned about something that has become somewhat of a red herring.

Quoted 6 lines

It takes about a second to erase a 64k physical sector. This is a required operation before it is written. Since the projected life of these new devices is about 5 to 10 million such cycles, (older NAND flash used in modems was only 100-200k) the writer would have to be running that "brand new device" for at least 5 million seconds. Let's see:

Quoted 7 lines

How come I can write to my compact flash at about 2M/s if you claim it takes 1s to erase a 64k sector? Somehow I think your number is much too high. Or it can do multiple erases at the same time.

Also the 5 to 10 million is a lot higher than the numbers the makers of the compact flash cards I use claim.

Here is an instrumented erase function on a driver that rewrites the first sector of a BIOS ROM. Unlike the Flash DISKS, the BIOS ROM has no buffering in static RAM so you can gustimate the actual time to erase............

//-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= //
// This erases a page and waits for the erasure to complete. It
// returns false if it failed.
//
static int erase(void *bios, int page)

{
     int era;
     flags_t flags;
     jiffie_t ticks, start;
     spin_lock_irqsave(&info->lock, flags);
     erase_page(bios, page);
     spin_unlock_irqrestore(&info->lock, flags);

     start = jiffies;

     ticks = jiffies + (ERA_TIME * HZ);
     era = 0x00;
     while(time_before(jiffies, ticks))
     {
         if((era = check_erase(bios, page)))
             break;
         if(signal_pending(current))
             break;
         set_current_state(TASK_INTERRUPTIBLE);
         schedule_timeout(1);
     }
     set_current_state(TASK_RUNNING);


     printk("They don't believe... %d\n", (int) (jiffies - start));

     return era;
}

[SNIPPED...]

On the system I rewrite a BIOS sector on, jiffies is 1024 ticks/second.

parport: PnPBIOS parport detected.
parport0: PC-style at 0x378, irq 7 [PCSPP,TRISTATE]
lp0: using parport0 (interrupt-driven).
lp0: console ready
device eth0 entered promiscuous mode
device eth0 left promiscuous mode
device eth0 entered promiscuous mode
device eth0 left promiscuous mode Analogic-BiosDev : Initialization complete They don't believe... 1004

Now, the wait for erase always sleeps for at least a timer-tick (about a milisecond) so this may take longer than the physical erase, but not much longer.

The erase function is:

#-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
#

#   This erases the NVRAM page (block). It doesn't wait for completion.
#   Each block is 64k in length.
#   M29W040B chip
#
.section        .text
erase_page:
         pushl   %ebx
         movl    BUF(%esp), %ebx         # Address of the chip
         movl    DAT(%esp), %ecx         # The page
         andl    $0x07, %ecx             # Max pages
         shll    $0x10, %ecx             # Times 64k
         movb    $0xf0, (%ebx)           # Reset
         movb    $0xaa, 0x555(%ebx)
         movb    $0x55, 0x2aa(%ebx)
         movb    $0x80, 0x555(%ebx)
         movb    $0xaa, 0x555(%ebx)
         movb    $0x55, 0x2aa(%ebx)
         movb    $0x30, (%ecx,%ebx)
         popl    %ebx

ret
.size erase_page,.-erase_page
.type erase_page,@function
.global erase_page

And the check-erase function is this:

#-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
#
# This reads the whole M29W040B page, looking for all 0xffff words.
# It returns non-zero if it has been erased and zero otherwise.
#
check_erase:

         pushl   %edi
         movl    BUF(%esp), %edi         # Point to buffer
         movl    DAT(%esp), %eax         # 64k page
         andl    $0x07, %eax             # Max pages possible
         shll    $0x10, %eax             # Times 64k
         addl    %eax, %edi              # Offset to start

cld
movl $0x8000, %ecx # Number of words to check
movl $-1, %eax # What to look for
repz scasw # Look for all 0xffff
jz 1f # All erased
incl %eax # -1 becomes zero
1: popl %edi
ret
.size check_erase,.-check_erase
.type check_erase,@function
.global check_erase

Quoted 17 lines
Quoted 9 lines

60 seconds per minute
3600 seconds per hour
86400 seconds per day.

5,000,000 / 86400 = 57 days of continuous writes to the same sector. The writer surely would have a strange file because he states that even a single large file can destroy the drive if it is mounted with the "sync" option.

Quoted 7 lines

Also, the failure mode of NAND flash is not that it becomes "destroyed". The failure mode is a slow loss of data. The devices no longer retain data for a zillion years, only a few hundred, eventually, only a year or so. So when somebody claims that the flash has gotten destroyed, they need to have written it for a few months, then waited for a few years before reporting the event.

Quoted 5 lines

Some flash devices can be "destroyed" by loosing power in the middle of a write, since this causes them to corrupt their table of blocks and only the manufacturer has the ability to reset that. Fortunately this doesn't seem like too common a design.

# dd if=/dev/zero of=/dev/whatever bs=1M count=128

Fixes a 128 megabyte flash disk, plug in other values for other sizes.

Quoted 1 line

Len Sorensen

Cheers,
Dick Johnson
Penguin : Linux version 2.6.15.4 on an i686 machine (5589.54 BogoMips). Warning : 98.36% of all statistics are fiction, book release in April. _


****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

#23 lore raw
Quoted 41 lines · linux-os \(Dick Johnson\)
Quoted 39 lines · Lennart Sorensen
Quoted 27 lines · linux-os (Dick Johnson)
Quoted 6 lines · On Mon, 27 Feb 2006 [email protected] wrote:
Quoted 2 lines · linux-os (Dick Johnson)

Flash does not get zeroed to be written! It gets erased, which sets all the bits to '1', i.e., all bytes to 0xff.

Quoted 8 lines

Thanks for the correction, but that does not change the discussion.

Quoted 3 lines

Further, the designers of
flash disks are not stupid as you assume. The direct access occurs to static RAM (read/write stuff).

I'm not assuming anything . Some hardware has been killed by this issue.
http://lkml.org/lkml/2005/5/13/144

No. That hardware was not killed by that issue. The writer, or another who had encountered the same issue, eventually repartitioned and reformatted the device. The partition table had gotten corrupted by some experiments and the writer assumed that the device was broken. It wasn't.

Also, if you read other elements in this thread, you would have learned about something that has become somewhat of a red herring.

Quoted 6 lines

It takes about a second to erase a 64k physical sector. This is a required operation before it is written. Since the projected life of these new devices is about 5 to 10 million such cycles, (older NAND flash used in modems was only 100-200k) the writer would have to be running that "brand new device" for at least 5 million seconds. Let's see:

How come I can write to my compact flash at about 2M/s if you claim it takes 1s to erase a 64k sector? Somehow I think your number is much too high. Or it can do multiple erases at the same time.

Quoted 8 lines
Quoted 3 lines

Also the 5 to 10 million is a lot higher than the numbers the makers of the compact flash cards I use claim.

Here is an instrumented erase function on a driver that rewrites the first sector of a BIOS ROM. Unlike the Flash DISKS, the BIOS ROM has no buffering in static RAM so you can gustimate the actual time to erase............

BIOS ROM is never NAND FLASH, it's most probably NOR FLASH, and FLASH DISKS are most probably NAND FLASH. NOR and NAND are very different technologies. You compare apples and oranges, -- static RAM has nothing to do with that.

-- Sergei.

#24 lore raw
In reply to linux-os (Dick Johnson)
Quoted 4 lines · linux-os \(Dick Johnson\)

Here is an instrumented erase function on a driver that rewrites the first sector of a BIOS ROM. Unlike the Flash DISKS, the BIOS ROM has no buffering in static RAM so you can gustimate the actual time to erase............

The NOR flash is different but Samsung manual for K9F5608U0A-YCB0, K9F5608U0A-YIB0 32M x 8 Bit NAND Flash Memory says:

FEATURES GENERAL DESCRIPTION
- Page Program : (512 + 16)Byte
- Block Erase : (16K + 512)Byte
- Program time : 200us(Typ.)
- Block Erase Time : 2ms(Typ.)
- Endurance : 100K Program/Erase Cycles
- Data Retention : 10 Years

Signature · 1 line
Krzysztof Halasa
#25 lore raw
In reply to linux-os (Dick Johnson)
Quoted 5 lines · linux-os (Dick Johnson)

No. That hardware was not killed by that issue. The writer, or another who had encountered the same issue, eventually repartitioned and reformatted the device. The partition table had gotten corrupted by some experiments and the writer assumed that the device was broken. It wasn't.

I did not get the info you posted from that thread so maybe I missed something you saw. Or indeed it was someone else.

Many thanks for your comments. If this is a false alert all the better.

Quoted 4 lines

Also, the failure mode of NAND flash is not that it becomes "destroyed". The failure mode is a slow loss of data. The devices no longer retain data for a zillion years, only a few hundred, eventually, only a year or so.

There was a comment about the failure mode, no time scale was given. I see no reason why the degradation would stop at a year though.

Quoted 2 lines

Since the projected life of these new devices is about 5 to 10million such cycles,(older NAND flash used in modems was only 100-200k)

Maybe some of the cheap devices are not using the new flash memory in which case it would come down to between 24 and 48hrs of constant use. This would be a realistic problem.

Alan Cox refered to some devices that could be damaged as "crap", so it seems he is aware of some hardware differences.

In conclusion it seems from Andrew Morton's posts that the way this is handled is under review so I am confident that a robust and stable solution will result.

Thanks again for your thoughts on this.

#38 lore raw
In reply to [email protected]

Hi!

Quoted 14 lines
Quoted 13 lines

I agree completely which is why we hack the system to remove the o_sync on our distro derivative. (-:

But my point was that your solution of "don't do that then" is not much use to your average user who sits in front of such distro in graphical desktop as they are not technical enough to find and hack their hotplug system to work properly...

Best regards,

Anton

Quoted 1 line

If you don't want it *DO NOT USE IT AT THE MOUNT COMMAND LINE* !!!

Quoted 8 lines

Yeah, cleaver.
That is not really a constructive responce. I dont use , I do use command line mount all the time. I never was in danger of damaging my drive with this new "feature".

Telling a user who has just burnt out a brand new 1GB usb device he should have RTFM and modified that HAL configuration to insure it did not use sync it not likely to win much confidence in the linux kernel.

Return that 1GB usb device to manufacturer, it was broken.

Quoted 3 lines

The point of raising this is that the vast majority of linux users have no awareness of this. If there is a danger of this sync implementation damaging hardware it should be done differently.

Fix the distribution, then.
Pavel

Signature · 1 line
Web maintainer for suspend.sf.net (www.sf.net/projects/suspend) wanted...
#26 lore raw
In reply to [email protected]
Quoted 6 lines · On Út 28-02-06 00:21:53, [email protected] wrote:
Quoted 2 lines · linux-os (Dick Johnson)

Flash does not get zeroed to be written! It gets erased, which sets all the bits to '1', i.e., all bytes to 0xff.

Quoted 8 lines

Thanks for the correction, but that does not change the discussion.

Quoted 3 lines

Further, the designers of
flash disks are not stupid as you assume. The direct access occurs to static RAM (read/write stuff).

I'm not assuming anything . Some hardware has been killed by this issue.
http://lkml.org/lkml/2005/5/13/144

I have seen flash disk dead in 5 minutes, even without o-sync. Those devices are often crap. (I copied tar file to flash by cat foo.tar > /dev/sda. That was apparently enough to kill that flash. Label "Yahoo" should have warned me).
Pavel

Signature · 1 line
Web maintainer for suspend.sf.net (www.sf.net/projects/suspend) wanted...
#27 lore raw

why VM_SHM has been removed from mm.h?

Hello all,

VM_SHM is used by DIPC to quickly recognise when we are dealing with a System V IPC segment. It has been "removed" from recent kernels (set to 0). Is there an easy way to find out if a segment is a Sys V shm? if not, I suggest we re-activate it.

-Kamran

#28 lore raw

Re: why VM_SHM has been removed from mm.h?

Is there a reason that you posted this as a reply to the "o_sync in vfat" thread?

Quoted 6 lines · Kamran Karimi

Hello all,

VM_SHM is used by DIPC to quickly recognise when we are dealing with a System V IPC segment. It has been "removed" from recent kernels (set to 0). Is there an easy way to find out if a segment is a Sys V shm? if not, I suggest we re-activate it.

-Kamran

#34 lore raw
In reply to Pavel Machek
Quoted 1 line · Pavel Machek

I have seen flash disk dead in 5 minutes, even without o-sync. Those devices are often crap. (I copied tar file to flash by cat foo.tar > /dev/sda. That was apparently enough to kill that flash. Label "Yahoo" should have warned me).

Sometimes a flash device can have a temporary error condition that is solved by rewriting the data. (I've seen it triggered by buggy USB hubs that don't provide the rated power). It seems that a number of flash drives have internal checks, and when those trigger it reports a bad sector (even if it isn't permanently bad). My 1GB flashdrive failed in that way, and I was able to fix the error by erasing with "dd if=/dev/full of=/dev/usbkey" and reformatting. After the error occurred I started md5summing every file I put on the drive, but I've been using it for a month now and not a single checksum has miscomputed.

Cheers,
Kyle Moffett

#29 lore raw

Re: why VM_SHM has been removed from mm.h?

In reply to Kamran Karimi
Quoted 3 lines · Kamran Karimi

VM_SHM is used by DIPC to quickly recognise when we are dealing with a System V IPC segment. It has been "removed" from recent kernels (set to 0).

Curious: VM_SHM wasn't set on a System V IPC shm vma in any 2.4 or 2.6 kernel that I know of; but was set on the vmas of a random collection of drivers. Perhaps you've been using your own patch to set it on SysV IPC shm vmas, and clear it from drivers' vmas?

(We'll remove VM_SHM entirely once I've trawled through those drivers.)

Quoted 1 line

Is there an easy way to find out if a segment is a Sys V shm?

Nothing easy and reliable springs immediately to mind - from a VM point of view, they're treated much the same as tmpfs files; but there probably is some hacky way if we think about it long enough.

Quoted 1 line

if not, I suggest we re-activate it.

It seems that either you've been doing the wrong thing up to now, and never noticed it; or that you've been using your own flag in your own patch, and can continue to do so. No need for vanilla kernel to reinstate VM_SHM.

Are you sure you need to recognize them?

Hugh

#30 lore raw

Re: why VM_SHM has been removed from mm.h?

Thank you Hugh for the reply. Last time I used VM_SHM was in 2.2.x kernels. I have a programme called DIPC which makes System V shared memory segments (and also messages and semaphores) work over a network.

In the arch/xyz/mm/fault.c file, it checks the VM_SHM flag and then calls its logic. As a substitute I've been trying this ad-hoc code to see if a vma represents a Sys V shm:

     file = vma->vm_file;
     if(file && (file->f_dentry) && (file->f_dentry->d_inode) &&
        (id = file->f_dentry->d_inode->i_ino)) {
             shp = shm_lock(id);
             if(shp == NULL)
                     return 0; // not a Sys V shm
     }
     else return 0; // not a Sys V shm

But the kernel hangs with an invalid-pointer error message. Any suggestions?

-Kamran

Quoted 29 lines · Kamran Karimi
Quoted 2 lines

VM_SHM is used by DIPC to quickly recognise when we are dealing with a

System

Quoted 1 line

V IPC segment. It has been "removed" from recent kernels (set to 0).

Curious: VM_SHM wasn't set on a System V IPC shm vma in any 2.4 or 2.6 kernel that I know of; but was set on the vmas of a random collection of drivers. Perhaps you've been using your own patch to set it on SysV IPC shm vmas, and clear it from drivers' vmas?

(We'll remove VM_SHM entirely once I've trawled through those drivers.)

Quoted 1 line

Is there an easy way to find out if a segment is a Sys V shm?

Nothing easy and reliable springs immediately to mind - from a VM point of view, they're treated much the same as tmpfs files; but there probably is some hacky way if we think about it long enough.

Quoted 1 line

if not, I suggest we re-activate it.

It seems that either you've been doing the wrong thing up to now, and never noticed it; or that you've been using your own flag in your own patch, and can continue to do so. No need for vanilla kernel to reinstate VM_SHM.

Are you sure you need to recognize them?

Hugh

#31 lore raw

Re: why VM_SHM has been removed from mm.h?

Quoted 18 lines · Kamran Karimi

Thank you Hugh for the reply. Last time I used VM_SHM was in 2.2.x kernels. I have a programme called DIPC which makes System V shared memory segments (and also messages and semaphores) work over a network.

In the arch/xyz/mm/fault.c file, it checks the VM_SHM flag and then calls its logic. As a substitute I've been trying this ad-hoc code to see if a vma represents a Sys V shm:

      file = vma->vm_file;
      if(file && (file->f_dentry) && (file->f_dentry->d_inode) &&
         (id = file->f_dentry->d_inode->i_ino)) {
              shp = shm_lock(id);
              if(shp == NULL)
                      return 0; // not a Sys V shm
      }
      else return 0; // not a Sys V shm
Quoted 1 line

But the kernel hangs with an invalid-pointer error message. Any suggestions?

It's not obvious to me why the kernel would hang with an invalid pointer error message there: ipc_lock appears to have good safety against being passed a random id. Perhaps the invalid pointer message comes from other code you've not shown (for example, I hope you shm_unlock(shp) and return 1 when shm_lock succeeds), or perhaps I'm misreading.

But what you're doing there looks entirely weird and meaningless to me:
if shm_lock happens to succeed or fail on the inode number of some file on some filesystem, that tells you nothing about whether that file is SysV shm or not. Ah, I see ipc/shm.c saves id in i_ino: so if you're dealing with a SysV shm file, then indeed that ought to tell whether you're dealing with a SysV shm file - but that hasn't helped much!

Since you're already patching base kernel source (you mention arch/xyz/mm/fault.c), why don't you just patch your own VM_SYSVSHM into include/linux/mm.h, and set it on the vma in ipc/shm.c?

Hugh

#32 lore raw

Re: why VM_SHM has been removed from mm.h?

Quoted 5 lines

It's not obvious to me why the kernel would hang with an invalid pointer error message there: ipc_lock appears to have good safety against being passed a random id. Perhaps the invalid pointer message comes from other code you've not shown (for example, I hope you shm_unlock(shp) and return 1 when shm_lock succeeds), or perhaps I'm misreading.

I have put printk() statements all over the place. The hang (which is during boot time) occurs within the block of code that I sent you. There is a shm_unlock() statement after the code, but it is never reached.

Quoted 3 lines

Since you're already patching base kernel source (you mention arch/xyz/mm/fault.c), why don't you just patch your own VM_SYSVSHM into include/linux/mm.h, and set it on the vma in ipc/shm.c?

Yes this looks like a good solution. I have changed VM_SHM in mm.h to be 0x0800000 and am looking for a good place to include it in the vma->vm_flags. shmat() looks like a good place. How can I find the vma of a SysV shm in that routine?

-Kamran

#33 lore raw

Re: why VM_SHM has been removed from mm.h?

Quoted 7 lines · Kamran Karimi
Quoted 4 lines

Since you're already patching base kernel source (you mention arch/xyz/mm/fault.c), why don't you just patch your own VM_SYSVSHM into include/linux/mm.h, and set it on the vma in ipc/shm.c?

Yes this looks like a good solution. I have changed VM_SHM in mm.h to be 0x0800000 and am looking for a good place to include it in the vma->vm_flags.

I already pointed out that several drivers are setting VM_SHM; and that we shall remove it in due course. Your DIPC patch should use its own flag.

Quoted 2 lines

shmat() looks like a good place. How can I find the vma of a SysV shm in that routine?

shm_mmap would be the right place: shmat's do_mmap will call it.

Hugh

#35 lore raw
In reply to Pavel Machek
Quoted 14 lines · Pavel Machek
Quoted 13 lines · On Út 28-02-06 00:21:53, [email protected] wrote:
Quoted 1 line · linux-os (Dick Johnson)

Flash does not get zeroed to be written! It gets erased, which sets

all

Quoted 1 line

the bits to '1', i.e., all bytes to 0xff.

Thanks for the correction, but that does not change the discussion.

Quoted 3 lines

Further, the designers of
flash disks are not stupid as you assume. The direct access occurs to static RAM (read/write stuff).

Quoted 8 lines
Quoted 2 lines

I'm not assuming anything . Some hardware has been killed by this issue.
http://lkml.org/lkml/2005/5/13/144

I have seen flash disk dead in 5 minutes, even without o-sync. Those devices are often crap. (I copied tar file to flash by cat foo.tar > /dev/sda. That was apparently enough to kill that flash. Label "Yahoo" should have warned me).
Pavel

If I'm not mistaken, writing to the device with cat will output that file byte by byte. This would probably be even harder on the device than using a formatted device with o_sync, since it would dirty a 64k block 64k times!

It seems some of the less elaborate devices dont support this type of use.

I suspect if you had tried using dd with a suitable bs you may still own a crap Yahoo usb device.

Just because the linux kernel lets us use the abstract /dev devices freely does not mean everything you can do with a /dev is a good idea for all h/w that gets a device name.

I think that is the heart of the problem. Manufacturers are designing these devices for the windows market. They are specifically designed and supplied, preformatted with a fat fs, to be used in that way.

If linux distros, MacOS or anybody else wants to claim to support these devices the default setup should probably handle the devices in a _similar_ way to the native windows drivers.

#36 lore raw
Quoted 24 lines · On Čt 02-03-06 09:23:02, [email protected] wrote:
Quoted 14 lines · Pavel Machek
Quoted 13 lines · On Út 28-02-06 00:21:53, [email protected] wrote:
Quoted 1 line · linux-os (Dick Johnson)

Flash does not get zeroed to be written! It gets erased, which sets

all

Quoted 1 line

the bits to '1', i.e., all bytes to 0xff.

Thanks for the correction, but that does not change the discussion.

Quoted 3 lines

Further, the designers of
flash disks are not stupid as you assume. The direct access occurs to static RAM (read/write stuff).

Quoted 7 lines
Quoted 2 lines

I'm not assuming anything . Some hardware has been killed by this issue.
http://lkml.org/lkml/2005/5/13/144

I have seen flash disk dead in 5 minutes, even without o-sync. Those devices are often crap. (I copied tar file to flash by cat foo.tar > /dev/sda. That was apparently enough to kill that flash. Label "Yahoo" should have warned me).

Quoted 4 lines

If I'm not mistaken, writing to the device with cat will output that file byte by byte. This would probably be even harder on the device than using a formatted device with o_sync, since it would dirty a 64k block 64k times!

No.

Quoted 5 lines

It seems some of the less elaborate devices dont support this type of use.

I suspect if you had tried using dd with a suitable bs you may still own a crap Yahoo usb device.

Quoted 7 lines

Just because the linux kernel lets us use the abstract /dev devices freely does not mean everything you can do with a /dev is a good idea for all h/w that gets a device name.

I think that is the heart of the problem. Manufacturers are designing these devices for the windows market. They are specifically designed and supplied, preformatted with a fat fs, to be used in that way.

There's USB mass storage specification, that says nothing about FAT, or expected use of the device... if your device is broken FAT thing that will break if used any other way, do not advertise it as USB mass storage.
Pavel

Signature · 1 line
Web maintainer for suspend.sf.net (www.sf.net/projects/suspend) wanted...