Inode metadata and file data syncing
I am in the process of writing a file system in Linux. This file system has a separate mechanism by which we manage metadata so I do not want to write the file inode metadata to disk without explicitly requesting an update. I do need the file data pages to be written to disk as per the normal writeback process.
If I use the common mechanism of creating an inode and inserting it into the hash via insert_inode_locked(), the inode will be in the I_NEW state and when the inode is marked dirty it will be put on the dirty list and eventually flushed out to disk. One way I thought I could get around this is by initializing the inode to i_state = I_DIRTY, skipping I_NEW, and using insert_inode_hash() instead, so that if mark_inode_dirty() is called it won't get put on the dirty list. The issue with this approach is that it looks like this inode's pages will not get flushed to disk either since it won't ever get on the dirty list. I need the pages written just not the inode itself.
I am handling directory inodes differently. Looking at shmem I see that the backing_dev_info is set to:
struct backing_dev_info brnl_backing_dev_info = {
.ra_pages = 0,
.capabilities = BDI_CAP_NO_ACCT_AND_WRITEBACK | BDI_CAP_SWAP_BACKED,
};
I have done the same in my code to prevent directory inodes from being written to disk.
Can I manage the inode->i_state with the I_DIRTY flag and then somehow mark the inode pages dirty and add them to the dirty page list independently? What I am worried about is what affect doing this will have on the processing of anything in page cache or inode cache related to this inode.
Thank you for your help,
Sarah Jelinek
Quoted 6 lines · Sarah
I am in the process of writing a file system in Linux. This file system has a separate mechanism by which we manage metadata so I do not want to write the file inode metadata to disk without explicitly requesting an update. I do need the file data pages to be written to disk as per the normal writeback process.
Quoted 11 lines
If I use the common mechanism of creating an inode and inserting it into the hash via
insert_inode_locked(), the inode will be in theI_NEWstate and when the inode is marked dirty it will be put on the dirty list and eventually flushed out to disk. One way I thought I could get around this is by initializing the inode toi_state=I_DIRTY, skippingI_NEW, and usinginsert_inode_hash()instead, so that ifmark_inode_dirty()is called it won't get put on the dirty list. The issue with this approach is that it looks like this inode's pages will not get flushed to disk either since it won't ever get on the dirty list. I need the pages written just not the inode itself.
All the major file systems do this currently. If you just don't implement ->write_inode() you can control when the metadata is written out, the data will be flushed normally and you can write out the metadata whenever you feel like it, like the rest of us do without doing horrible things to the i_state. Thanks,
Josef
Hi Josef,
Thank you! This helps a lot. I hadn't seen the use of write_inode(), or not using it as a way to do this.
Regards,
sarah
Quoted 32 lines · Josef Bacik
Quoted 6 lines · Sarah
I am in the process of writing a file system in Linux. This file system has a separate mechanism by which we manage metadata so I do not want to write the file inode metadata to disk without explicitly requesting an update. I do need the file data pages to be written to disk as per the normal writeback process.
Quoted 15 lines
If I use the common mechanism of creating an inode and inserting it into the hash via
insert_inode_locked(), the inode will be in theI_NEWstate and when the inode is marked dirty it will be put on the dirty list and eventually flushed out to disk. One way I thought I could get around this
is by initializing the inode toi_state=I_DIRTY, skippingI_NEW, and usinginsert_inode_hash()instead, so that ifmark_inode_dirty()is called
it won't get put on the dirty list. The issue with this approach is that it looks like this inode's pages will not get flushed to disk either since
it won't ever get on the dirty list. I need the pages written just not the
inode itself.All the major file systems do this currently. If you just don't implement ->write_inode() you can control when the metadata is written out, the data will
be flushed normally and you can write out the metadata whenever you feel like
it, like the rest of us do without doing horrible things to thei_state. Thanks,Josef
Quoted 5 lines · Sarah
I am in the process of writing a file system in Linux. This file system has a separate mechanism by which we manage metadata so I do not want to write the file inode metadata to disk without explicitly requesting an update. I do need the file data pages to be written to disk as per the normal writeback process.
The first, most important, question is why are you writing a new filesystem for Linux? There are lots of filesystems already, and the amount of effort to write a complete filesystem (instead of a simple filesystem with only basic functionality) is fairly high.
Unless there is an overwhelmingly good reason to implement a new filesystem, it is better to improve some other existing filesystem to have the feature(s) that you are missing, instead of creating a new one. That helps you avoid a lot of effort, and adds value to everyone else that is using the existing filesystem, instead of making a niche filesystem only useful to yourself and needing ongoing maintenance.
Quoted 36 lines
If I use the common mechanism of creating an inode and inserting it into the hash via
insert_inode_locked(), the inode will be in theI_NEWstate and when the inode is marked dirty it will be put on the dirty list and eventually flushed out to disk. One way I thought I could get around this is by initializing the inode toi_state=I_DIRTY, skippingI_NEW, and usinginsert_inode_hash()instead, so that ifmark_inode_dirty()is called it won't get put on the dirty list. The issue with this approach is that it looks like this inode's pages will not get flushed to disk either since it won't ever get on the dirty list. I need the pages written just not the inode itself.I am handling directory inodes differently. Looking at shmem I see that the
backing_dev_infois set to:struct
backing_dev_infobrnl_backing_dev_info= {
.ra_pages = 0,
.capabilities =BDI_CAP_NO_ACCT_AND_WRITEBACK|BDI_CAP_SWAP_BACKED,
};I have done the same in my code to prevent directory inodes from being written to disk.
Can I manage the
inode->i_statewith theI_DIRTYflag and then somehow mark the inode pages dirty and add them to the dirty page list independently? What I am worried about is what affect doing this will have on the processing of anything in page cache or inode cache related to this inode.Thank you for your help,
Sarah Jelinek--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
I am doing a project for my company.
Quoted 63 lines · Andreas Dilger
Quoted 5 lines · Sarah
I am in the process of writing a file system in Linux. This file system has a separate mechanism by which we manage metadata so I do not want to write the file inode metadata to disk without explicitly requesting an update. I do need the file data pages to be written to disk as per the normal writeback process.
The first, most important, question is why are you writing a new filesystem for Linux? There are lots of filesystems already, and the amount of effort to write a complete filesystem (instead of a simple filesystem with only basic functionality) is fairly high.
Unless there is an overwhelmingly good reason to implement a new filesystem, it is better to improve some other existing filesystem to have the feature(s) that you are missing, instead of creating a new one. That helps you avoid a lot of effort, and adds value to everyone else that is using the existing filesystem, instead of making a niche filesystem only useful to yourself and needing ongoing maintenance.
Quoted 44 lines
If I use the common mechanism of creating an inode and inserting it into the hash via
insert_inode_locked(), the inode will be in theI_NEWstate and when the inode is marked dirty it will be put on the dirty list and eventually flushed out to disk. One way I thought I could get around this
is by initializing the inode toi_state=I_DIRTY, skippingI_NEW, and usinginsert_inode_hash()instead, so that ifmark_inode_dirty()is called
it won't get put on the dirty list. The issue with this approach is that it looks like this inode's pages will not get flushed to disk either since
it won't ever get on the dirty list. I need the pages written just not the
inode itself.I am handling directory inodes differently. Looking at shmem I see that the
backing_dev_infois set to:struct
backing_dev_infobrnl_backing_dev_info= {
.ra_pages = 0,
.capabilities =BDI_CAP_NO_ACCT_AND_WRITEBACK|BDI_CAP_SWAP_BACKED,
};I have done the same in my code to prevent directory inodes from being written to disk.
Can I manage the
inode->i_statewith theI_DIRTYflag and then somehow mark the inode pages dirty and add them to the dirty page list independently? What I am worried about is what affect doing this will have
on the processing of anything in page cache or inode cache related to this
inode.Thank you for your help,
Sarah Jelinek--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Involving in writing a new file system as a job requirement? There should be some value or special features that the file system has, maybe? Could you tell what new FS features this introduces?
/Sorin
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jelinek, Sarah
Sent: Thursday, July 19, 2012 8:45 AM
To: Andreas Dilger
Cc: [email protected]
Subject: Re: Inode metadata and file data syncing
I am doing a project for my company.
Quoted 12 lines · Andreas Dilger
Quoted 5 lines · Sarah
I am in the process of writing a file system in Linux. This file system has a separate mechanism by which we manage metadata so I do not want to write the file inode metadata to disk without explicitly requesting an update. I do need the file data pages to be written to disk as per the normal writeback process.
The first, most important, question is why are you writing a new filesystem for Linux? There are lots of filesystems already, and the amount of effort to write a complete filesystem (instead of a simple filesystem with only basic functionality) is fairly high.
Quoted 51 lines
Unless there is an overwhelmingly good reason to implement a new filesystem, it is better to improve some other existing filesystem to have the feature(s) that you are missing, instead of creating a new one. That helps you avoid a lot of effort, and adds value to everyone else that is using the existing filesystem, instead of making a niche filesystem only useful to yourself and needing ongoing maintenance.
Quoted 44 lines
If I use the common mechanism of creating an inode and inserting it into the hash via
insert_inode_locked(), the inode will be in theI_NEWstate and when the inode is marked dirty it will be put on the dirty list and eventually flushed out to disk. One way I thought I could get around this
is by initializing the inode toi_state=I_DIRTY, skippingI_NEW, and usinginsert_inode_hash()instead, so that ifmark_inode_dirty()is called
it won't get put on the dirty list. The issue with this approach is that it looks like this inode's pages will not get flushed to disk either since
it won't ever get on the dirty list. I need the pages written just not the
inode itself.I am handling directory inodes differently. Looking at shmem I see that the
backing_dev_infois set to:struct
backing_dev_infobrnl_backing_dev_info= {
.ra_pages = 0,
.capabilities =BDI_CAP_NO_ACCT_AND_WRITEBACK|BDI_CAP_SWAP_BACKED,
};I have done the same in my code to prevent directory inodes from being written to disk.
Can I manage the
inode->i_statewith theI_DIRTYflag and then somehow mark the inode pages dirty and add them to the dirty page list independently? What I am worried about is what affect doing this will have
on the processing of anything in page cache or inode cache related to this
inode.Thank you for your help,
Sarah Jelinek--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
No, I am sorry I can't give you any details at this time. Its still under development.
Quoted 92 lines · On 7/19/12 7:12 AM, "[email protected]" <[email protected]> wrote:
Involving in writing a new file system as a job requirement? There should be some value or special features that the file system has, maybe? Could you tell what new FS features this introduces?
/Sorin
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jelinek, Sarah
Sent: Thursday, July 19, 2012 8:45 AM
To: Andreas Dilger
Cc: [email protected]
Subject: Re: Inode metadata and file data syncingI am doing a project for my company.
Quoted 13 lines · Andreas Dilger
Quoted 6 lines · Sarah
I am in the process of writing a file system in Linux. This file system has a separate mechanism by which we manage metadata so I do not want to
write the file inode metadata to disk without explicitly requesting an update. I do need the file data pages to be written to disk as per the normal writeback process.The first, most important, question is why are you writing a new filesystem for Linux? There are lots of filesystems already, and the amount of effort to write a complete filesystem (instead of a simple filesystem with only basic functionality) is fairly high.
Quoted 54 lines
Unless there is an overwhelmingly good reason to implement a new filesystem, it is better to improve some other existing filesystem to have the feature(s) that you are missing, instead of creating a new one. That helps you avoid a lot of effort, and adds value to everyone else that is using the existing filesystem, instead of making a niche filesystem only useful to yourself and needing ongoing maintenance.
Quoted 47 lines
If I use the common mechanism of creating an inode and inserting it into
the hash viainsert_inode_locked(), the inode will be in theI_NEWstate
and when the inode is marked dirty it will be put on the dirty list and eventually flushed out to disk. One way I thought I could get around this
is by initializing the inode toi_state=I_DIRTY, skippingI_NEW, and usinginsert_inode_hash()instead, so that ifmark_inode_dirty()is called
it won't get put on the dirty list. The issue with this approach is that
it looks like this inode's pages will not get flushed to disk either since
it won't ever get on the dirty list. I need the pages written just not the
inode itself.I am handling directory inodes differently. Looking at shmem I see that the
backing_dev_infois set to:struct
backing_dev_infobrnl_backing_dev_info= {
.ra_pages = 0,
.capabilities =BDI_CAP_NO_ACCT_AND_WRITEBACK|BDI_CAP_SWAP_BACKED,
};I have done the same in my code to prevent directory inodes from being written to disk.
Can I manage the
inode->i_statewith theI_DIRTYflag and then somehow mark the inode pages dirty and add them to the dirty page list independently? What I am worried about is what affect doing this will have
on the processing of anything in page cache or inode cache related to this
inode.Thank you for your help,
Sarah Jelinek--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
I am simply curious what all the current file systems features we lack today as maybe others can contribute. Hope this makes sense
/Sorin
-----Original Message-----
From: Jelinek, Sarah [mailto:[email protected]]
Sent: Thursday, July 19, 2012 9:18 AM
To: faibish, sorin; [email protected]
Cc: [email protected]
Subject: Re: Inode metadata and file data syncing
No, I am sorry I can't give you any details at this time. Its still under development.
Quoted 92 lines · On 7/19/12 7:12 AM, "[email protected]" <[email protected]> wrote:
Involving in writing a new file system as a job requirement? There should be some value or special features that the file system has, maybe? Could you tell what new FS features this introduces?
/Sorin
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jelinek, Sarah
Sent: Thursday, July 19, 2012 8:45 AM
To: Andreas Dilger
Cc: [email protected]
Subject: Re: Inode metadata and file data syncingI am doing a project for my company.
Quoted 13 lines · Andreas Dilger
Quoted 6 lines · Sarah
I am in the process of writing a file system in Linux. This file system has a separate mechanism by which we manage metadata so I do not want to
write the file inode metadata to disk without explicitly requesting an update. I do need the file data pages to be written to disk as per the normal writeback process.The first, most important, question is why are you writing a new filesystem for Linux? There are lots of filesystems already, and the amount of effort to write a complete filesystem (instead of a simple filesystem with only basic functionality) is fairly high.
Quoted 54 lines
Unless there is an overwhelmingly good reason to implement a new filesystem, it is better to improve some other existing filesystem to have the feature(s) that you are missing, instead of creating a new one. That helps you avoid a lot of effort, and adds value to everyone else that is using the existing filesystem, instead of making a niche filesystem only useful to yourself and needing ongoing maintenance.
Quoted 47 lines
If I use the common mechanism of creating an inode and inserting it into
the hash viainsert_inode_locked(), the inode will be in theI_NEWstate
and when the inode is marked dirty it will be put on the dirty list and eventually flushed out to disk. One way I thought I could get around this
is by initializing the inode toi_state=I_DIRTY, skippingI_NEW, and usinginsert_inode_hash()instead, so that ifmark_inode_dirty()is called
it won't get put on the dirty list. The issue with this approach is that
it looks like this inode's pages will not get flushed to disk either since
it won't ever get on the dirty list. I need the pages written just not the
inode itself.I am handling directory inodes differently. Looking at shmem I see that the
backing_dev_infois set to:struct
backing_dev_infobrnl_backing_dev_info= {
.ra_pages = 0,
.capabilities =BDI_CAP_NO_ACCT_AND_WRITEBACK|BDI_CAP_SWAP_BACKED,
};I have done the same in my code to prevent directory inodes from being written to disk.
Can I manage the
inode->i_statewith theI_DIRTYflag and then somehow mark the inode pages dirty and add them to the dirty page list independently? What I am worried about is what affect doing this will have
on the processing of anything in page cache or inode cache related to this
inode.Thank you for your help,
Sarah Jelinek--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 2 lines · On Thu, Jul 19, 2012 at 09:50:59AM -0400, [email protected] wrote:
I am simply curious what all the current file systems features we lack today as maybe others can contribute. Hope this makes sense
It doesn't take much effort to imagine features which are tied to hardware that is still under wraps and which make current designs look completely silly. I'm willing to give them the benefit of the doubt.
I'm happy that they're engaging with fsdevel early, even if its clumsy because they can't really talk about details. I think it's worth spending a small amount of time to help them understand the (very confusing!) fs paths. It has a decent chance of reducing the number of problems in the code we'll no doubt see some day and have to review.
- z
Quoted 7 lines · Zach Brown
Quoted 2 lines · On Thu, Jul 19, 2012 at 09:50:59AM -0400, [email protected] wrote:
I am simply curious what all the current file systems features we lack today as maybe others can contribute. Hope this makes sense
It doesn't take much effort to imagine features which are tied to hardware that is still under wraps and which make current designs look completely silly. I'm willing to give them the benefit of the doubt.
Sure, I totally agree. I just wanted to head off any efforts that are just duplicating btrfs or cachefs or whatever for very little total benefit.
It's also worthwhile to ask that question early on, before there is a large vested interest in the new filesystem, rather than months/years later when the time and effort is already sunk into the other project and not some existing project.
Quoted 5 lines
I'm happy that they're engaging with fsdevel early, even if its clumsy because they can't really talk about details. I think it's worth spending a small amount of time to help them understand the (very confusing!) fs paths. It has a decent chance of reducing the number of problems in the code we'll no doubt see some day and have to review.
Agreed.
Cheers, Andreas
Quoted 1 line · Andreas Dilger
Sure, I totally agree. I just wanted to head off any efforts that are just duplicating btrfs or cachefs or whatever for very little total benefit.
I can assure you there's nothing out there like this :-)
Quoted 1 line
It's also worthwhile to ask that question early on, before there is a large vested interest in the new filesystem, rather than months/years later when the time and effort is already sunk into the other project and not some existing project.
My initial commit to this project was July 2009. It takes a long time to get from "wouldn't it be crazy if ..." to production quality hardware.
And, of course, there are never false turns along that path!
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."