Is it time of refreshing interest to NILFS2 file system?
Subsystems (2)
Hello,
QLC NAND flash makes really tough requirements for file systems to be really flash friendly. NILFS2 is log-structured file system supporting continuous snapshotting. It could be considered like a really good basis of meeting: (1) QLC NAND flash requirements, (2) reliability requirements, (3) cold storage requirements. AI/ML workloads require to store and access huge volume of data during training phase. And NILFS2 is completely efficient as cold storage solution for AI/ML workloads (especially, because GC could be excluded from the equation for the case of cold storage). NILFS2 suggests really good model of operations for the case of long stream of sensors’ data that needs to be continuously stored, analyzed or used for training/inference, and, finally, archived or discarded/disposed. Potentially, NILFS2 is capable decrease TCO cost (and even power consumption).
NILFS2 represents one of the unique piece of technology in the family of Linux file systems. However, it unfairly lost attention of the open-source community. Currently, we have multiple xfstests failures (around 50 can be reproduced in stable manner), multiple features are still not implemented yet (atime, xattrs, fsck, etc). NILFS2 is low-hanging fruit for ZNS and FDP SSD support. It makes sense to implement unit-tests to cover the file system’s code base. Potentially, it is possible to consider of adding compression, encryption, erasure coding features, multiple drives support and so on (you are welcome to suggest and implement your favorite feature). There are plenty of room for performance optimizations. NILFS2 is really good basis for exploring and experimenting with ML approaches for better GC logic and FSCK functionality.
Fresh Linux kernel guys always ask how they can contribute to Linux kernel and many guys are considering the file system direction. NILFS2 is viable direction with plenty opportunities for optimizations and new features implementation. I would like to deliver this talk with the goals of: (1) encouraging fresh Linux kernel developers of joining to contribution into NILFS2, and (2) convincing open-source community to revive the interest to NILFS2. I believe that NILFS2 deserves the second life in the world of QLC NAND flash and AI/ML workloads. NILFS2 is part of Linux ecosystem with unique set of features and it makes sense to make it more efficient, secure, and reliable.
Thanks,
Slava.
Quoted 10 lines · Viacheslav Dubeyko
Fresh Linux kernel guys always ask how they can contribute to Linux kernel and many guys are considering the file system direction. NILFS2 is viable direction with plenty opportunities for optimizations and new features implementation. I would like to deliver this talk with the goals of: (1) encouraging fresh Linux kernel developers of joining to contribution into NILFS2, and (2) convincing open-source community to revive the interest to NILFS2. I believe that NILFS2 deserves the second life in the world of QLC NAND flash and AI/ML workloads. NILFS2 is part of Linux ecosystem with unique set of features and it makes sense to make it more efficient, secure, and reliable.
I wonder if this might be better fit for the Linux Plumbers Conference. The LSF/MM/BPF is workshop is a invite-only workshop which is focused on discussions, not talks. If the target of your talk includes "fresh Linux kernel developers", it is unlikely that there will be many at the LSF/MM. They are more likely to be at Plumbers, which will have roughly an order of magnitude ore attendees.
- Ted
Quoted 12 lines · Theodore Tso
Quoted 10 lines · Viacheslav Dubeyko
Fresh Linux kernel guys always ask how they can contribute to Linux kernel and many guys are considering the file system direction. NILFS2 is viable direction with plenty opportunities for optimizations and new features implementation. I would like to deliver this talk with the goals of: (1) encouraging fresh Linux kernel developers of joining to contribution into NILFS2, and (2) convincing open-source community to revive the interest to NILFS2. I believe that NILFS2 deserves the second life in the world of QLC NAND flash and AI/ML workloads. NILFS2 is part of Linux ecosystem with unique set of features and it makes sense to make it more efficient, secure, and reliable.
Quoted 7 lines
I wonder if this might be better fit for the Linux Plumbers Conference. The LSF/MM/BPF is workshop is a invite-only workshop which is focused on discussions, not talks. If the target of your talk includes "fresh Linux kernel developers", it is unlikely that there will be many at the LSF/MM. They are more likely to be at Plumbers, which will have roughly an order of magnitude ore attendees.
I think there are three target audiences: (1) fresh Linux kernel developers, (2) file system maintainers, (3) potential customers. If we consider only fresh Linux kernel developers, then, yes, LPC could be better place. However, we cannot consider this audience in isolation. Because, these guys prefer to ask guidance from the top kernel maintainers. So, the more important audience is the second one. Multiple other Linux kernel file systems are in better shape because they have more attention. So, the main point of this topic is sharing the current status of NILFS2 (issues and TODOs) and to have discussion how we can move the NILFS2 to better shape. I believe, we need to consider Linux as ecosystem and all pieces of this puzzle should be reliable, efficient, and secure. And this is why I would like to attract attention to NILFS2. LFS/MM/BPF is really good stage for this. And also it is the way to send a signal to third audience (potential customers) that NILFS2 is not dead. NILFS2 still has good potential but it simply has not enough attention, from my point of view. It was clear from the beginning that NILFS2 can be easily modified to support ZNS SSD, for example. Now, even XFS has ZNS SSD support, but NILFS2 still hasn't. We need to rebuild the NILFS2 community, but it requires of creating some noise. But, of course, I can go to LPC or/and OSS and I am already creating the noise. :)
Thanks,
Slava.
Quoted 4 lines · Viacheslav Dubeyko
Multiple other Linux kernel file systems are in better shape because they have more attention. So, the main point of this topic is sharing the current status of NILFS2 (issues and TODOs) and to have discussion how we can move the NILFS2 to better shape.
I'd gently suggest to you that if you want to give a status of NILFS2, a better place would be either writing an LWN article, or a talk or BOF at Linux Plumbers Conference.
In terms of how to move NILFS2 to a better shape, perhaps we can start a discussion on this e-mail thread?
1) You need fsck.nilfs2. I've looked at nilfs-tools, and you have a mkfs.nilfs2, but you don't have a utility to (a) verify that the file system is self-consistent, and (b) repair the file systgem if it isn't. Without such a tool, I can pretty much guarantee that no one will take the file system seriously, and using it in production would be professional malpractice.
2) You need to test nilfs2 using fstests, which means you need to add support for nilfs2 to fstests. Having an fsck program is super useful for fstests, because if a kernel bug corrupts the file system, you need an fsck program to detect that fact, and then you need to fix said bug so that customers don't have their data get scrambled.
I would consider (1) and (2) table stakes. If you don't have these, potential customers are not going to take the file system seriously, and no talk/presentation at any conference, whether it's LSF/MM or LPC or OSS is going to change that.
Quoted 4 lines
And this is why I would like to attract attention to NILFS2. LFS/MM/BPF is really good stage for this. And also it is the way to send a signal to third audience (potential customers) that NILFS2 is not dead.... We need to rebuild the NILFS2 community, but it requires of creating some noise.
So I'm going to disagree with you here. Noise/attention is not what you need. What you need is a business case for why a company should assign engineers to work on nilfs2. I can't think of a single major file system in Linux is which has all of its development done by volunteer hobbyists.
So it's not noise that you need; you need to convince managers and executives at various companies why they should invest in nilfs2. And for the most part, the people who make the final decisions on things like Fall Plan (in the case of IBM), or OKR's (in the case of companies like Intel, Google, and Facebook) tend not be well
represented at LSF/MM. :-)
Cheers,
- Ted