Split some metadata onto separate device?

Hi list,

back when I was using XFS, there was an option to put one particular piece of metadata -- the journal -- onto separate physical device. The speed-up was very significant, especially for operations on lots of files. For example, with external journal removing whole kernel source tree on XFS would take less time than on ext2 or ext3; with internal it would be much longer than ext2 or 3.

The journal was needed only in case of unclean unmount, as only then it contained any information not applied yet; if the unmount was clean, the journal contained no important information.

XFS journals only metadata changes, btw.

My question is, would it be feasible to move some part of NILFS2 metadata onto separate device for R/W/GC speed-up?

Perhaps some of metadata could even be considered `recoverable' -- that is, if the other device gets broken or lost, the metadata on it could be re- constructed from content of the main device.

Regards,

Signature · 14 lines

dexen deVries

[[[↓][→]]]

For example, if the first thing in the file is:
   <?kzy irefvba="1.0" rapbqvat="ebg13"?>
an XML parser will recognize that the document is stored in the traditional 
ROT13 encoding.

(( Joe English, http://www.flightlab.com/~joe/sgml/faq-not.txt ))
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

(...)
NILFS2 is a log-structured filesystem and that means the filesystem itself is a big journal. My understanding is using separate journal device has no merit of performance.

On the other hand, We may be able to speed up the filesystem by putting DAT (disk address translation) metadata to a separate device. I mean putting a full copy of the DAT metadata on the sperate device instead of journal of it.

But, I think we have other approaches to think about before that with regard to performance improvement. For example, applying "extent" to DAT seems much more effective.

Hi Ryusuke,

Quoted 9 lines · On Friday 03 June 2011 19:26:21 you wrote:

(...)
NILFS2 is a log-structured filesystem and that means the filesystem itself is a big journal. My understanding is using separate journal device has no merit of performance.

On the other hand, We may be able to speed up the filesystem by putting DAT (disk address translation) metadata to a separate device. I mean putting a full copy of the DAT metadata on the sperate device instead of journal of it.

*nods* sounds interesting. Of course changing on-disk format would hurt ;-)

Quoted 3 lines

But, I think we have other approaches to think about before that with regard to performance improvement. For example, applying "extent" to DAT seems much more effective.

Currently every block of file gets own entry in DAT, right?

By the way, it seems to me that the in-kernel GC does not attempt to de- fragment files. Is that the case? If so, would probably mean extents would not help very much for fragmented files. Also, on magnetic media, a lot of seeking happens in case of fragmented files.

Regards,

Signature · 7 lines

dexen deVries

``One can't proceed from the informal to the formal by formal means.''
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 5 lines · dexen deVries

By the way, it seems to me that the in-kernel GC does not attempt to de- fragment files. Is that the case? If so, would probably mean extents would not help very much for fragmented files. Also, on magnetic media, a lot of seeking happens in case of fragmented files.

Yes, but extents will decrease the amount of metadata, and this has potential for speed up.

NILFS2 uses 32 bytes metadata per disk block at present. I guess you know that the number of DAT blocks are actually indispensable through analysis using the dumpseg tool.

Quoted 10 lines · Ryusuke Konishi

Quoted 5 lines · dexen deVries

By the way, it seems to me that the in-kernel GC does not attempt to de- fragment files. Is that the case? If so, would probably mean extents would not help very much for fragmented files. Also, on magnetic media, a lot of seeking happens in case of fragmented files.

Yes, but extents will decrease the amount of metadata, and this has potential for speed up.

Quoted 3 lines

NILFS2 uses 32 bytes metadata per disk block at present. I guess you know that the number of DAT blocks are actually indispensable through analysis using the dumpseg tool.

Oh, looks confusing. I meant the amount of DAT blocks is not negligible.

Hi again,

Quoted 15 lines · On Wednesday 08 of June 2011 14:58:03 you wrote:

Quoted 13 lines · Ryusuke Konishi

Quoted 5 lines · dexen deVries

By the way, it seems to me that the in-kernel GC does not attempt to de- fragment files. Is that the case? If so, would probably mean extents would not help very much for fragmented files. Also, on magnetic media, a lot of seeking happens in case of fragmented files.

Yes, but extents will decrease the amount of metadata, and this has potential for speed up.

NILFS2 uses 32 bytes metadata per disk block at present. I guess you know that the number of DAT blocks are actually indispensable through analysis using the dumpseg tool.

Quoted 2 lines

Oh, looks confusing. I meant the amount of DAT blocks is not negligible.

Understood ;-)

Anyway, isn't some fragmentation avoidance necessary to profit from extents (that we hope we'll use at some point)?

AFAIK, an extent is only good to describe file blocks if they are laid out continuous on the block device.

Cheers,

Signature · 14 lines

dexen deVries

[[[↓][→]]]

For example, if the first thing in the file is:
   <?kzy irefvba="1.0" rapbqvat="ebg13"?>
an XML parser will recognize that the document is stored in the traditional 
ROT13 encoding.

(( Joe English, http://www.flightlab.com/~joe/sgml/faq-not.txt ))
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 7 lines

Quoted 7 lines

Quoted 6 lines

Yes, but extents will decrease the amount of metadata, and this has potential for speed up.

NILFS2 uses 32 bytes metadata per disk block at present. I guess you know that the number of DAT blocks are actually indispensable through analysis using the dumpseg tool.

Quoted 7 lines

Quoted 2 lines

Oh, looks confusing. I meant the amount of DAT blocks is not negligible.

Understood ;-)

Anyway, isn't some fragmentation avoidance necessary to profit from extents (that we hope we'll use at some point)?

I guess it doesn't become a big issue unless we do many small random writes like database.

Quoted 2 lines

AFAIK, an extent is only good to describe file blocks if they are laid out continuous on the block device.

Yep. And, we know files are enough continuous in most cases.

We use at least 16 bytes per 4 kiro-bytes disk block to point the block (16KB for 8MB segment); btrees of nilfs2 use an 8 bytes key and an 8 bytes pointer for a disk block.

In typical case, this can be reduced to 16 bytes per file (16 bytes for 8MB segment) with the extents.

IMO, 64-bit filesystems have a reasonable reason to adopt extents though nilfs2 did not apply it.

I guess applying extents to the DAT file is especially effective.

Hi Dexen, hi Ryusuke,

been a while since i was active on the list or active with NiLFS in general but the last vacation did remind me why i started working on NiLFS for NetBSD again! Using FFS, even with logging, UDF or msdosfs just doesn't work well on Flash media ;)

Quoted 27 lines · Ryusuke Konishi

Quoted 7 lines

Quoted 7 lines

Quoted 6 lines

Yes, but extents will decrease the amount of metadata, and this has potential for speed up.

NILFS2 uses 32 bytes metadata per disk block at present. I guess you know that the number of DAT blocks are actually indispensable through analysis using the dumpseg tool.

Quoted 7 lines

Quoted 2 lines

Oh, looks confusing. I meant the amount of DAT blocks is not negligible.

Understood ;-)

Anyway, isn't some fragmentation avoidance necessary to profit from extents (that we hope we'll use at some point)?

I guess it doesn't become a big issue unless we do many small random writes like database.

Quoted 2 lines

AFAIK, an extent is only good to describe file blocks if they are laid out continuous on the block device.

Yep. And, we know files are enough continuous in most cases.

We use at least 16 bytes per 4 kiro-bytes disk block to point the block (16KB for 8MB segment); btrees of nilfs2 use an 8 bytes key and an 8 bytes pointer for a disk block.

Quoted 7 lines

In typical case, this can be reduced to 16 bytes per file (16 bytes for 8MB segment) with the extents.

IMO, 64-bit filesystems have a reasonable reason to adopt extents though nilfs2 did not apply it.

I guess applying extents to the DAT file is especially effective.

I've used extents a lot in UDF since its all-extent based, and yes, it CAN be a lot smaller indeed :) BUT only when files are NOT sparse and NOT fragmented.

Anyway, my sugestion is an intermediate one since it doesn't change the disc format: on garbage collection, make an extra pass to seek (very) fragmented extents in the DAT, how much might be a parameter, read it all in and then relocate/write out a whole (or partial) segment with it. This could also be done with the CP and SU files although those files are relatively tiny. This will leave the DAT file in just a few segments when the filesystem has been garbage collected and (hopefully) most references only within the segment also easing cachability.

Other files could also be un-sparsified this way but i think the DAT will have the most benefit and it's relatively easy to do this way.