SEEK_HOLE and SEEK_DATA support?
All,
Has there been any thought about adding SEEK_HOLE and SEEK_DATA (*) support to Linux?
I ask primarily because of the interplay between 64-bit systems and things like /var/log/lastlog (which appears as a 1.2TiB file due to the nfsnobody UID of 4294967294).
(I'm realize that adding support for these additional seek() flags wouldn't solve the problem ... archiving tools would still have to implement it. And I can also hear the argument that Red Hat and other distributions should re-implement lastlog handling to use a more modern and efficient hashing/index format and perhaps that they should set nfsnobody to "-1" ... I'd be curious if those details are driven by some published standard or if they are artifacts of porting. I'd also be curious what's happened with other 64-bit UNIX ports and whether this issue ever came up in Linux ports to the Alpha or other 64-bit processors).
As a stray data point I just did a quick experiment and just doing a time cat /var/log/lastlog > /dev/null took about:
36.33user 2453.99system 41:35.90elapsed 99%CPU
(0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (133major+15minor)pagefaults 0swaps
On an otherwise idle 2GHz dual Opteron (yes, of course the extra CPU is wasted for this job), reading SCSI disk hanging off a Fusion MPT controller.
From what I hear our Networker processes pore over these NULs for about two hours any time someone fails to exclude /var/log/lastlog from their backup list.
* (see http://blogs.sun.com/roller/page/bonwick?entry=seek_hole_and_seek_data
for details)
(Please feel free to cc me on any responses, or I'll pick them up via the archives and KT ... my account dropped off LKML years ago and I don't want to punish my poor old IDSL line with the traffic now)
Signature · 1 line
Jim Dennis
Quoted 11 lines · Jim Dennis
I ask primarily because of the interplay between 64-bit systems and things like
/var/log/lastlog(which appears as a 1.2TiB file due to the nfsnobody UID of 4294967294).(I'm realize that adding support for these additional
seek()flags wouldn't solve the problem ... archiving tools would still have to implement it. And I can also hear the argument that Red Hat and other distributions should re-implement lastlog handling to use a more modern and efficient hashing/index format and perhaps that they should set nfsnobody to "-1" ...
So the presence of very high UIDs causes lastlog to be huge? That just sounds like a RedHat bug.
Lee
Quoted 13 lines · Lee Revell
Quoted 11 lines · Jim Dennis
I ask primarily because of the interplay between 64-bit systems and things like
/var/log/lastlog(which appears as a 1.2TiB file due to the nfsnobody UID of 4294967294).(I'm realize that adding support for these additional
seek()flags wouldn't solve the problem ... archiving tools would still have to implement it. And I can also hear the argument that Red Hat and other distributions should re-implement lastlog handling to use a more modern and efficient hashing/index format and perhaps that they should set nfsnobody to "-1" ...
Quoted 2 lines
So the presence of very high UIDs causes lastlog to be huge? That just sounds like a RedHat bug.
it causes it to be a sparse file
lastlog is an array based file format ;)
but sparse
Quoted 30 lines · [email protected] (Jim Dennis) wrote:
All,
Has there been any thought about adding
SEEK_HOLEandSEEK_DATA(*) support to Linux?I ask primarily because of the interplay between 64-bit systems and things like
/var/log/lastlog(which appears as a 1.2TiB file due to the nfsnobody UID of 4294967294).(I'm realize that adding support for these additional
seek()flags wouldn't solve the problem ... archiving tools would still have to implement it. And I can also hear the argument that Red Hat and other distributions should re-implement lastlog handling to use a more modern and efficient hashing/index format and perhaps that they should set nfsnobody to "-1" ... I'd be curious if those details are driven by some published standard or if they are artifacts of porting. I'd also be curious what's happened with other 64-bit UNIX ports and whether this issue ever came up in Linux ports to the Alpha or other 64-bit processors).As a stray data point I just did a quick experiment and just doing a time cat
/var/log/lastlog>/dev/nulltook about:36.33user 2453.99system 41:35.90elapsed 99%CPU
(0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (133major+15minor)pagefaults 0swaps
Quoted 7 lines
On an otherwise idle 2GHz dual Opteron (yes, of course the extra CPU is wasted for this job), reading SCSI disk hanging off a Fusion MPT controller.
From what I hear our Networker processes pore over these NULs for about two hours any time someone fails to exclude
/var/log/lastlogfrom their backup list.
This can already be solved in userspace (and I'm sure it already has been in some backup programs). Use the FIBMAP ioctl() against the fd to find out whether a particular block in the file is actually instantiated.
Not all filesystems necessarily implement FIBMAP, so one would need to fall back to sucky mode if FIBMAP failed.
Quoted 39 lines · Andrew Morton
Quoted 30 lines · [email protected] (Jim Dennis) wrote:
All,
Has there been any thought about adding
SEEK_HOLEandSEEK_DATA(*) support to Linux?I ask primarily because of the interplay between 64-bit systems and things like
/var/log/lastlog(which appears as a 1.2TiB file due to the nfsnobody UID of 4294967294).(I'm realize that adding support for these additional
seek()flags wouldn't solve the problem ... archiving tools would still have to implement it. And I can also hear the argument that Red Hat and other distributions should re-implement lastlog handling to use a more modern and efficient hashing/index format and perhaps that they should set nfsnobody to "-1" ... I'd be curious if those details are driven by some published standard or if they are artifacts of porting. I'd also be curious what's happened with other 64-bit UNIX ports and whether this issue ever came up in Linux ports to the Alpha or other 64-bit processors).As a stray data point I just did a quick experiment and just doing a time cat
/var/log/lastlog>/dev/nulltook about:36.33user 2453.99system 41:35.90elapsed 99%CPU
(0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (133major+15minor)pagefaults 0swapsQuoted 7 lines
On an otherwise idle 2GHz dual Opteron (yes, of course the extra CPU is wasted for this job), reading SCSI disk hanging off a Fusion MPT controller.
From what I hear our Networker processes pore over these NULs for about two hours any time someone fails to exclude
/var/log/lastlogfrom their backup list.
Quoted 7 lines
This can already be solved in userspace (and I'm sure it already has been in some backup programs). Use the FIBMAP
ioctl()against the fd to find out whether a particular block in the file is actually instantiated.Not all filesystems necessarily implement FIBMAP, so one would need to fall back to sucky mode if FIBMAP failed.
FIBMAP is a privileged operation.
Signature · 1 line
Nicholas Miell <[email protected]>