XFS NVMe RDMA?
Dear XFS Experts,
as you may or may not know, XFS on NVMe was used as part of the #1 entry to the IO500 benchmarks, as announced at ISC21. That entry swept away the other competition (albeit on large custom hardware), including systems using Intel’s DAOS using Octane/PMem, WekaIO, Lustre, GekkoFS and others.
There’s no publication associated with it, however there’s a video presenting how they did it:
https://www.youtube.com/watch?v=BJpkpA6hsDc
In it they describe how they used XFS for storing data chunks, and RocksDB for storing metadata. I’m trying to dig deeper on how they could have used XFS, and in particular how they could have used RDMA to access XFS data. The XFS DAX mode as far as I’m aware requires PMem rather than NVMe?
Do you have any ideas how they could have been able to utilise RDMA so that node A can directly access data chunks stored on XFS on node B? Is the only approach to mmap the chunk on node B and then RDMA it to/from node A?
Kind regards,
Dan
Quoted 1 line · Dan Greenfield
Do you have any ideas how they could have been able to utilise RDMA so that node A can directly access data chunks stored on XFS on node B? Is the only approach to mmap the chunk on node B and then RDMA it to/from node A?
I'm not going to watch a video, but with the pNFS code other nodes can access data on an XFS node directly using any SCSI transport. For RMDA that would be SRP or iSCSI/iSER.
Note that I also have an unfinished draft to support NVMe, which has an RDMA transports as well and someone else could trivially reimplement that as well.
Quoted 10 lines · Christoph Hellwig
Quoted 1 line · Dan Greenfield
Do you have any ideas how they could have been able to utilise RDMA so that node A can directly access data chunks stored on XFS on node B? Is the only approach to mmap the chunk on node B and then RDMA it to/from node A?
I'm not going to watch a video, but with the pNFS code other nodes can access data on an XFS node directly using any SCSI transport. For RMDA that would be SRP or iSCSI/iSER.
Note that I also have an unfinished draft to support NVMe, which has an RDMA transports as well and someone else could trivially reimplement that as well.
Oh, and just FYI here are my slides on the pNFS support:
https://events.static.linuxfound.org/sites/events/files/slides/pnfs.pdf
Thank you Christoph! That’s very cool and makes sense.
Quick Question: would I be correct in assuming that if I mmap()ed each 8-64MB (chunk) file from XFS, and then did RDMA from the mmap region, that it would first be copied from NVMe into DRAM (does this bypass CPU?) and *then* be copied across RDMA, rather than directly be copied from NVMe by RDMA? Or does O_DIRECT properly allow bypass straight to NVMe for RDMA?
For what this #1 entry are doing though, each of the 512 nodes have their own separate XFS FS as well as their own separate RocksDB, both backed by NVMe. They are doing filesystem ops almost entirely in user-mode (no kernel, no FUSE) by intercepting application binaries and rewriting syscall instructions into jumps into their user-mode library code and doing message passing to RDMA transfers to/from application memory from/to remote node’s NVMe. I don’t believe they’ve modified XFS, nor using pNFS. I don’t know if there’s any other mechanism though other than mmap() and then RDMA on that region?
- Dan
Quoted 17 lines · Christoph Hellwig
Quoted 10 lines · Christoph Hellwig
Quoted 1 line · Dan Greenfield
Do you have any ideas how they could have been able to utilise RDMA so that node A can directly access data chunks stored on XFS on node B? Is the only approach to mmap the chunk on node B and then RDMA it to/from node A?
I'm not going to watch a video, but with the pNFS code other nodes can access data on an XFS node directly using any SCSI transport. For RMDA that would be SRP or iSCSI/iSER.
Note that I also have an unfinished draft to support NVMe, which has an RDMA transports as well and someone else could trivially reimplement that as well.
Oh, and just FYI here are my slides on the pNFS support:
https://events.static.linuxfound.org/sites/events/files/slides/pnfs.pdf
Thank you again Christoph for your earlier answer.
Quick Question: would I be correct in assuming that if I mmap()ed each 8-64MB (chunk) file from XFS, and then did RDMA from the mmap region, that it would first be copied from NVMe into DRAM (does this bypass CPU?) and *then* be copied across RDMA, rather than directly be copied from NVMe by RDMA? Or does O_DIRECT properly allow bypass straight to NVMe for RDMA?
For what this #1 entry are doing though, each of the 512 nodes have their own separate XFS FS as well as their own separate RocksDB, both backed by NVMe. They are doing filesystem ops almost entirely in user-mode (no kernel, no FUSE) by intercepting application binaries and rewriting syscall instructions into jumps into their user-mode library code and doing message passing to RDMA transfers to/from application memory from/to remote node’s NVMe. I don’t believe they’ve modified XFS, nor using pNFS. I don’t know if there’s any other mechanism though other than mmap() and then RDMA on that region?
- Dan
Quoted 17 lines · Christoph Hellwig
Quoted 10 lines · Christoph Hellwig
Quoted 1 line · Dan Greenfield
Do you have any ideas how they could have been able to utilise RDMA so that node A can directly access data chunks stored on XFS on node B? Is the only approach to mmap the chunk on node B and then RDMA it to/from node A?
I'm not going to watch a video, but with the pNFS code other nodes can access data on an XFS node directly using any SCSI transport. For RMDA that would be SRP or iSCSI/iSER.
Note that I also have an unfinished draft to support NVMe, which has an RDMA transports as well and someone else could trivially reimplement that as well.
Oh, and just FYI here are my slides on the pNFS support:
https://events.static.linuxfound.org/sites/events/files/slides/pnfs.pdf
Hi Dan,
sorry for the late reply. This sat in my outbox for a while almost fully written.
Quoted 1 line · Dan Greenfield
Quick Question: would I be correct in assuming that if I mmap()ed each 8-64MB (chunk) file from XFS, and then did RDMA from the mmap region, that it would first be copied from NVMe into DRAM (does this bypass CPU?) and *then* be copied across RDMA, rather than directly be copied from NVMe by RDMA? Or does
O_DIRECTproperly allow bypass straight to NVMe for RDMA?
The answer is: it depends. mmap on a non-DAX file system always copied into DRAM. mmap on a DAX file system (that is using pmem) can map the "storage" directly into memory, in which case some RDMA setups can DMA without a copy.
But with a plain old SSD there is no path available to userspace to transfer without copying to DRAM. If OTOH you use in-kernel NVMe over fabics target, it can directly transfers from the SSDs in some circumstances. Currently that does require using the SSD directly without a file system, but with a little more work it could also work using a file system.