Skip to contentSkip to conversation

Rust PuzleFS filesystem driver

RFC PATCH Superseded Ariel Miculas 4 participants 20 messages Jul 26, 2023 – Jul 27, 2023

Subsystems (2)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover Rust PuzleFS filesystem driver

Hello,

This is a proof of concept driver written for the PuzzleFS next-generation container filesystem [1]. This patch series is based on top of the puzzlefs_dependencies [4] branch, which contain the backported filesystem abstractions of Wedson Almeida Filho [2][3] and the additional third-party crates capnp [5] and hex [6]. The code is also available on github [11]. I've adapted the user space puzzlefs code
[1] so that the puzzlefs kernel module could present the directory hierarchy and implement the basic read functionality. For some additional context, puzzlefs was started by Tycho Andersen and it's the successor of atomfs. This FOSDEM presentation from 2019 [10] covers the rationale for a new oci image format and presents a higher level overview of our goals with puzzlefs.

Changes since v1:
* the main change is that I've replaced serde_cbor with Cap'n proto at Kent Overstreet's suggestion; this simplifies the dependency tree of puzzlefs and results in cleaner code
* I've also modified the filesystem abstractions to follow the latest "Rust for Linux" best practices (in the puzzlefs_dependencies_v2 branch [4])

I've split the rest of the cover letter in following sections (using a markdown style):
* Example: it describes a practical example of what was achieved
* Limitations: it presents the existing limitations of this POC
* Setup: it shows how to setup the necessary environment for testing the puzzlefs driver

# Example
An example is provided below:

```
~ # cat /proc/filesystems | grep puzzlefs
nodev   puzzlefs
~ # mount -t puzzlefs -o oci_root_dir="/home/puzzlefs_oci" -o image_manifest="c4
3e5ab9d0cee1dcfbf442d18023b34410de3deb0f6dbffcec72732b6830db09" none /mnt
~ # ls -lR /mnt/
/mnt/:
total 0
drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-1
drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-2
drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-3
drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-4
-rw-r--r--    1 0        0                0 Jun  8 12:26 file1
-rw-r--r--    1 0        0                0 Jun  8 12:26 file2

/mnt/dir-1:
total 0

/mnt/dir-2:
total 0

/mnt/dir-3:
total 0

/mnt/dir-4:
total 0
~ # cat /mnt/file2
ana are mere bla bla bla
~ # wc /mnt/file1
      202       202      5454 /mnt/file1
```

In this example, /home/puzzlefs_oci is the puzzlefs oci directory:

```
~ # ls -lR /home/puzzlefs_oci/
/home/puzzlefs_oci/:
total 8
drwxr-xr-x    3 1000     1000             0 Jul 26 14:37 blobs
-rw-r--r--    1 1000     1000           267 Jul 26 14:37 index.json
-rw-r--r--    1 1000     1000            37 Jul 26 14:37 oci-layout

/home/puzzlefs_oci/blobs:
total 0
drwxr-xr-x    2 1000     1000             0 Jul 26 14:37 sha256

/home/puzzlefs_oci/blobs/sha256:
total 16
-rw-------    1 1000     1000           272 Jul 26 14:37 c43e5ab9d0cee1dcfbf442d18023b34410de3deb0f6dbffcec72732b6830db09
-rw-------    1 1000     1000          5479 Jul 26 14:37 d86a87b19bd9a2fec0d31687c1d669cdb59e8fb579b0b2bfe354473699732bfb
-rw-------    1 1000     1000           752 Jul 26 14:37 fe6ee75ca93ded15da5f6e0abf335e6a4eb24277c59acd47282d8ed2475f3384
```

c43e5ab9d0cee1dcfbf442d18023b34410de3deb0f6dbffcec72732b6830db09 is the puzzlefs image manifest hash for the first_try tag:

```
❯ cat index.json | jq .
{
  "schemaVersion": -1,
  "manifests": [
    {
      "digest": "sha256:c43e5ab9d0cee1dcfbf442d18023b34410de3deb0f6dbffcec72732b6830db09",
      "size": 272,
      "media_type": "application/vnd.puzzlefs.image.rootfs.v1",
      "annotations": {
        "org.opencontainers.image.ref.name": "first_try"
      }
    }
  ],
  "annotations": {}
}
```

I will describe how to build a puzzlefs image in step 5 of the Setup section.

# Limitations
One limitation is that the puzzlefs driver doesn't implement any lookup functionality and instead it inserts every directory entry into the dcache during init (see the DCACHE_BASED constant). This is similar to how the sample rust_fs driver works, but the goal is to implement proper lookup functions. However, more filesystem abstractions need to be implemented before this can be achieved.

# Setup
My setup is based on Wedson's tutorial [8]. Next, I will describe the necessary steps to build an initrd and run a custom kernel under qemu.

1. Get the code from the puzzlefs_rfc_v2 branch on github [11]

2. Follow the rust quickstart guide [9]

3. configure and build the kernel

```
$ make LLVM=1 allnoconfig qemu-busybox-min.config rust.config
$ make LLVM=1 -j$(nproc)
```

4. setup busybox

```
git clone git://git.busybox.net/busybox
cd busybox
make menuconfig # enable 'Build static binary' config
make
make install
```

This will create the _install directory with the rootfs inside it.

5. create a home directory in the rootfs and copy a puzzlefs oci directory in home/puzzlefs_oci
To create a puzzlefs oci directory:
* clone the puzzlefs repository [1]

* run make release

* create a simple filesystem structure with a few directories and files (I've created one at ../test-puzzlefs/simple_rootfs)

* build a puzzlefs oci image at ~/work/busybox/_install/home/puzzlefs_oci (replace this path with your busybox path) with the tag first_try:

```
$ target/release/puzzlefs build ../test-puzzlefs/simple_rootfs \
 ~/work/busybox/_install/home/puzzlefs_oci first_try
```

* get first_try's image manifest from index.json (inside puzzlefs_oci)

```
$ cat index.json | jq . | grep digest
      "digest": "sha256:c43e5ab9d0cee1dcfbf442d18023b34410de3deb0f6dbffcec72732b6830db09",
```

6. add the following 'init' script in the busybox rootfs (rootfs path defaults to `./_install'):

```
#!/bin/sh
mount -t devtmpfs none /dev
mkdir -p /proc
mount -t proc none /proc

ifconfig lo up
udhcpc -i eth0

mkdir /mnt
mount -t puzzlefs -o oci_root_dir="/home/puzzlefs_oci" -o \
image_manifest="c43e5ab9d0cee1dcfbf442d18023b34410de3deb0f6dbffcec72732b6830db09" \
none /mnt

setsid sh -c 'exec sh -l </dev/ttyS0 >/dev/ttyS0 2>&1'
```

Make sure to replace the image_manifest with your own digest. This init script will be passed to rdinit in the kernel command line.

7. generate the initramfs

Assuming busybox is in ~/work/busybox:

```
cd ~/work/busybox/_install && find . | cpio -H newc -o | gzip > ../ramdisk.img
```

This will generate a compressed ramdisk image in
~/work/busybox/ramdisk.img.

8. run with qemu (this assumes the linux tree is at '../linux' and busybox is at '../busybox'):

```
qemu-system-x86_64 \
    -accel kvm \
    -cpu host \
    -m 4G \
    -initrd ../busybox/ramdisk.img \
    -kernel ../linux/arch/x86/boot/bzImage \
    -nographic \
    -append 'console=ttyS0 nokaslr debug rdinit=/init' \
    -nic user,model=rtl8139 \
    -no-reboot
```

9. Check whether puzzlefs has been successfully mounted

```
~ # mount | grep puzzlefs
none on /mnt type puzzlefs (rw,relatime)
~ # ls /mnt/
dir-1  dir-2  dir-3  dir-4  file1  file2
```

[1] https://github.com/project-machine/puzzlefs
[2] https://github.com/wedsonaf/linux/tree/fs
[3] https://github.com/Rust-for-Linux/linux/issues/1004
[4] https://github.com/ariel-miculas/linux/tree/puzzlefs_dependencies_v2
[5] https://docs.rs/capnp/latest/capnp/
[6] https://docs.rs/hex/0.4.3/hex/
[7] https://hackmd.io/@cyphar/ociv2-brainstorm
[8] https://www.youtube.com/watch?v=tPs1uRqOnlk
[9] https://docs.kernel.org/rust/quick-start.html
[10] https://archive.fosdem.org/2019/schedule/event/containers_atomfs/
[11] https://github.com/ariel-miculas/linux/tree/puzzlefs_rfc_v2

Ariel Miculas (10):
samples: puzzlefs: add initial puzzlefs sample, copied from rust_fs.rs
kernel: configs: enable rust samples in rust.config
rust: kernel: add an abstraction over vfsmount to allow cloning a new
private mount
rust: file: Add a new RegularFile newtype useful for reading files
samples: puzzlefs: add basic deserializing support for the puzzlefs
metadata
rust: file: pass the filesystem context to the open function
samples: puzzlefs: populate the directory entries with the inodes from
the puzzlefs metadata file
rust: puzzlefs: read the puzzlefs image manifest instead of an
individual metadata layer
rust: puzzlefs: add support for reading files
rust: puzzlefs: add oci_root_dir and image_manifest filesystem
parameters

 kernel/configs/rust.config                  |   10 +
 rust/alloc/vec/mod.rs                       |   17 +
 rust/bindings/bindings_helper.h             |    1 +
 rust/helpers.c                              |    6 +
 rust/kernel/error.rs                        |    6 +-
 rust/kernel/file.rs                         |  146 +-
 rust/kernel/fs.rs                           |    4 +-
 rust/kernel/lib.rs                          |    1 +
 rust/kernel/mount.rs                        |   71 +
 samples/rust/Kconfig                        |   10 +
 samples/rust/Makefile                       |    1 +
 samples/rust/puzzle.rs                      |    5 +
 samples/rust/puzzle/error.rs                |  106 +
 samples/rust/puzzle/inode.rs                |  101 +
 samples/rust/puzzle/oci.rs                  |   71 +
 samples/rust/puzzle/types.rs                |  340 ++
 samples/rust/puzzle/types/manifest.capnp    |   15 +
 samples/rust/puzzle/types/manifest_capnp.rs |  773 ++++
 samples/rust/puzzle/types/metadata.capnp    |   69 +
 samples/rust/puzzle/types/metadata_capnp.rs | 4458 +++++++++++++++++++
 samples/rust/puzzlefs.rs                    |  220 +
 samples/rust/rust_fs.rs                     |    3 +-
 22 files changed, 6425 insertions(+), 9 deletions(-)
 create mode 100644 rust/kernel/mount.rs
 create mode 100644 samples/rust/puzzle.rs
 create mode 100644 samples/rust/puzzle/error.rs
 create mode 100644 samples/rust/puzzle/inode.rs
 create mode 100644 samples/rust/puzzle/oci.rs
 create mode 100644 samples/rust/puzzle/types.rs
 create mode 100644 samples/rust/puzzle/types/manifest.capnp
 create mode 100644 samples/rust/puzzle/types/manifest_capnp.rs
 create mode 100644 samples/rust/puzzle/types/metadata.capnp
 create mode 100644 samples/rust/puzzle/types/metadata_capnp.rs
 create mode 100644 samples/rust/puzzlefs.rs
Signature · 1 line
2.41.0
#1 lore raw

RFC PATCH v2 1/10 samples: puzzlefs: add initial puzzlefs sample, copied from rust_fs.rs

---
 samples/rust/Kconfig     |  10 ++++
 samples/rust/Makefile    |   1 +
 samples/rust/puzzlefs.rs | 105 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 116 insertions(+)
 create mode 100644 samples/rust/puzzlefs.rs
samples/rust/Kconfig +10 -0
index 2bd736f99189..05ca21fbba06 100644
@@ -40,6 +40,16 @@ config SAMPLE_RUST_FS
 
 	  If unsure, say N.
 
+config SAMPLE_PUZZLEFS
+	tristate "Puzzlefs file system"
+	help
+	  This option builds the Rust puzzlefs file system sample.
+
+	  To compile this as a module, choose M here:
+	  the module will be called puzzlefs.
+
+	  If unsure, say N.
+
 config SAMPLE_RUST_HOSTPROGS
 	bool "Host programs"
 	help
samples/rust/Makefile +1 -0
index e5941037e673..364a38dbf90b 100644
@@ -3,5 +3,6 @@
 obj-$(CONFIG_SAMPLE_RUST_MINIMAL)		+= rust_minimal.o
 obj-$(CONFIG_SAMPLE_RUST_PRINT)			+= rust_print.o
 obj-$(CONFIG_SAMPLE_RUST_FS)			+= rust_fs.o
+obj-$(CONFIG_SAMPLE_PUZZLEFS)			+= puzzlefs.o
 
 subdir-$(CONFIG_SAMPLE_RUST_HOSTPROGS)		+= hostprogs
samples/rust/puzzlefs.rs new file +105 -0

105 lines not on this page · raw message

Signature · 1 line
2.41.0
#2 lore raw

RFC PATCH v2 2/10 kernel: configs: enable rust samples in rust.config

In reply to Ariel Miculas
---
 kernel/configs/rust.config | 10 ++++++++++
 1 file changed, 10 insertions(+)
kernel/configs/rust.config +10 -0
index 38a7c5362c9c..63798ae5f3a5 100644
@@ -1 +1,11 @@
 CONFIG_RUST=y
+
+CONFIG_MODULES=y
+CONFIG_SAMPLES=y
+CONFIG_SAMPLES_RUST=y
+CONFIG_SAMPLE_RUST_MINIMAL=m
+CONFIG_SAMPLE_RUST_PRINT=m
+CONFIG_SAMPLE_RUST_FS=m
+CONFIG_SAMPLE_PUZZLEFS=y
+CONFIG_SAMPLE_RUST_HOSTPROGS=y
+
Signature · 1 line
2.41.0
#3 lore raw

RFC PATCH v2 3/10 rust: kernel: add an abstraction over vfsmount to allow cloning a new private mount

In reply to Ariel Miculas
---
 rust/bindings/bindings_helper.h |  1 +
 rust/kernel/lib.rs              |  1 +
 rust/kernel/mount.rs            | 71 +++++++++++++++++++++++++++++++++
 3 files changed, 73 insertions(+)
 create mode 100644 rust/kernel/mount.rs
rust/bindings/bindings_helper.h +1 -0
index e5fc3b1d408d..205ef50dfd4c 100644
@@ -18,6 +18,7 @@
 #include <linux/uio.h>
 #include <linux/uaccess.h>
 #include <linux/delay.h>
+#include <linux/namei.h>
 
 /* `bindgen` gets confused at certain things. */
 const gfp_t BINDINGS_GFP_KERNEL = GFP_KERNEL;
rust/kernel/lib.rs +1 -0
index 08f67833afcf..7dc07bdb5d55 100644
@@ -46,6 +46,7 @@
 pub mod ioctl;
 pub mod iov_iter;
 pub mod mm;
+pub mod mount;
 pub mod pages;
 pub mod prelude;
 pub mod print;
rust/kernel/mount.rs new file +71 -0

71 lines not on this page · raw message

Signature · 1 line
2.41.0
reviewer
#4 lore raw

Re: [RFC PATCH v2 03/10] rust: kernel: add an abstraction over vfsmount to allow cloning a new private mount

Quoted 7 lines · Ariel Miculas

[...]
+++ b/rust/kernel/mount.rs

rust/kernel/mount.rs new file +3 -0
@@ -0,0 +1,71 @@
[...]
+impl Vfsmount {
+    /// Create a new private mount clone based on a path name
+    pub fn new_private_mount(path_name: &CStr) -> Result<Self> {

rust/kernel/mount.rs line 41

Quoted 10 lines
rust/kernel/mount.rs new file +10 -0
+        let path: Path = Path(Opaque::uninit());
+        // SAFETY: path_name is a &CStr, so it's a valid string pointer; path is an uninitialized
+        // struct stored on the stack and it's ok because kern_path expects an out parameter
+        let err = unsafe {
+            bindings::kern_path(
+                path_name.as_ptr() as *const i8,
+                bindings::LOOKUP_FOLLOW | bindings::LOOKUP_DIRECTORY,
+                path.0.get(),
+            )
+        };

Nit: I believe that .cast::<i8>() is preferred over as *const T because it makes it harder to
accidentally change constness or indirection level. This isn't always followed in kernel, but
good practice.

Quoted 5 lines
rust/kernel/mount.rs new file +5 -0
+        if err != 0 {
+            pr_err!("failed to resolve '{}': {}\n", path_name, err);
+            return Err(EINVAL);
+        }
+

rust/kernel/mount.rs line 56

Quoted 10 lines
rust/kernel/mount.rs new file +10 -0
+        // SAFETY: path is a struct stored on the stack and it is  initialized because the call to
+        // kern_path succeeded
+        let vfsmount = unsafe { from_err_ptr(bindings::clone_private_mount(path.0.get()))? };
+
+        // Don't inherit atime flags
+        // SAFETY: we called from_err_ptr so it's safe to dereference this pointer
+        unsafe {
+            (*vfsmount).mnt_flags &=
+                !(bindings::MNT_NOATIME | bindings::MNT_NODIRATIME | bindings::MNT_RELATIME) as i32;
+        }

Nit: it looks a bit cleaner to OR the flags outside of the unsafe block, which is also nice because then
the unsafe block only contains the unsafe ops (could do this above too). Also - I think maybe style
might encourage // CAST: where as is used?

let new_flags =
!(bindings::MNT_NOATIME | bindings::MNT_NODIRATIME |
bindings::MNT_RELATIME) as i32;

    // SAFETY: we called from_err_ptr so it's safe to dereference this pointer
    unsafe { (*vfsmount).mnt_flags &= new_flags }
Quoted 19 lines
rust/kernel/mount.rs new file +15 -1
+        Ok(Self { vfsmount })
+    }
+
+    /// Returns a raw pointer to vfsmount
+    pub fn get(&self) -> *mut bindings::vfsmount {
+        self.vfsmount
+    }
+}
+
+impl Drop for Vfsmount {
+    fn drop(&mut self) {
+        // SAFETY new_private_mount makes sure to return a valid pointer
+        unsafe { bindings::kern_unmount(self.vfsmount) };
+    }
+}
--

2.41.0

#6 lore raw

RFC PATCH v2 4/10 rust: file: Add a new RegularFile newtype useful for reading files

In reply to Ariel Miculas

Implement from_path, from_path_in_root_mnt, read_with_offset, read_to_end and get_file_size methods for a RegularFile newtype.

---
 rust/helpers.c       |   6 ++
 rust/kernel/error.rs |   4 +-
 rust/kernel/file.rs  | 129 ++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 135 insertions(+), 4 deletions(-)
rust/helpers.c +6 -0
index eed8ace52fb5..9e860a554cda 100644
@@ -213,6 +213,12 @@ void *rust_helper_alloc_inode_sb(struct super_block *sb,
 }
 EXPORT_SYMBOL_GPL(rust_helper_alloc_inode_sb);
 
+loff_t rust_helper_i_size_read(const struct inode *inode)
+{
+	return i_size_read(inode);
+}
+EXPORT_SYMBOL_GPL(rust_helper_i_size_read);
+
 /*
  * We use `bindgen`'s `--size_t-is-usize` option to bind the C `size_t` type
  * as the Rust `usize` type, so we can use it in contexts where Rust
rust/kernel/error.rs +1 -3
index 05fcab6abfe6..e061c83f806a 100644
@@ -273,9 +273,7 @@ pub fn to_result(err: core::ffi::c_int) -> Result {
 ///     }
 /// }
 /// ```
-// TODO: Remove `dead_code` marker once an in-kernel client is available.
-#[allow(dead_code)]
-pub(crate) fn from_err_ptr<T>(ptr: *mut T) -> Result<*mut T> {
+pub fn from_err_ptr<T>(ptr: *mut T) -> Result<*mut T> {
     // CAST: Casting a pointer to `*const core::ffi::c_void` is always valid.
     let const_ptr: *const core::ffi::c_void = ptr.cast();
     // SAFETY: The FFI function does not deref the pointer.
rust/kernel/file.rs +128 -1

151 lines not on this page · raw message

Signature · 1 line
2.41.0
reviewer
#7 lore raw

Re: [RFC PATCH v2 04/10] rust: file: Add a new RegularFile newtype useful for reading files

Quoted 75 lines · Ariel Miculas

Implement from_path, from_path_in_root_mnt, read_with_offset, read_to_end and get_file_size methods for a RegularFile newtype.

---
 rust/helpers.c       |   6 ++
 rust/kernel/error.rs |   4 +-
 rust/kernel/file.rs  | 129 ++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 135 insertions(+), 4 deletions(-)
rust/helpers.c +6 -0
index eed8ace52fb5..9e860a554cda 100644
@@ -213,6 +213,12 @@ void *rust_helper_alloc_inode_sb(struct super_block *sb,
 }
 EXPORT_SYMBOL_GPL(rust_helper_alloc_inode_sb);
 
+loff_t rust_helper_i_size_read(const struct inode *inode)
+{
+       return i_size_read(inode);
+}
+EXPORT_SYMBOL_GPL(rust_helper_i_size_read);
+
 /*
  * We use `bindgen`'s `--size_t-is-usize` option to bind the C `size_t` type
  * as the Rust `usize` type, so we can use it in contexts where Rust
rust/kernel/error.rs +1 -3
index 05fcab6abfe6..e061c83f806a 100644
@@ -273,9 +273,7 @@ pub fn to_result(err: core::ffi::c_int) -> Result {
 ///     }
 /// }
 /// ```
-// TODO: Remove `dead_code` marker once an in-kernel client is available.
-#[allow(dead_code)]
-pub(crate) fn from_err_ptr<T>(ptr: *mut T) -> Result<*mut T> {
+pub fn from_err_ptr<T>(ptr: *mut T) -> Result<*mut T> {
     // CAST: Casting a pointer to `*const core::ffi::c_void` is always valid.
     let const_ptr: *const core::ffi::c_void = ptr.cast();
     // SAFETY: The FFI function does not deref the pointer.
rust/kernel/file.rs +5 -1
index 494939ba74df..a3002c416dbb 100644
@@ -8,11 +8,13 @@
 use crate::{
     bindings,
     cred::Credential,
-    error::{code::*, from_result, Error, Result},
+    error::{code::*, from_err_ptr, from_result, Error, Result},
     fs,
     io_buffer::{IoBufferReader, IoBufferWriter},
     iov_iter::IovIter,
     mm,
+    mount::Vfsmount,
+    str::CStr,
     sync::CondVar,
     types::ARef,
     types::AlwaysRefCounted,
@@ -20,6 +22,7 @@
     types::Opaque,
     user_ptr::{UserSlicePtr, UserSlicePtrReader, UserSlicePtrWriter},
 };
+use alloc::vec::Vec;
 use core::convert::{TryFrom, TryInto};
 use core::{marker, mem, ptr};
 use macros::vtable;
@@ -201,6 +204,130 @@ unsafe fn dec_ref(obj: ptr::NonNull<Self>) {
     }
 }
 
+/// A newtype over file, specific to regular files

rust/kernel/file.rs unsafe fn dec_ref(obj: ptr::NonNull<Self>) { · line 217

Quoted 10 lines
rust/kernel/file.rs +10 -0
+pub struct RegularFile(ARef<File>);
+impl RegularFile {
+    /// Creates a new instance of Self if the file is a regular file
+    ///
+    /// # Safety
+    ///
+    /// The caller must ensure file_ptr.f_inode is initialized to a valid pointer (e.g. file_ptr is
+    /// a pointer returned by path_openat); It must also ensure that file_ptr's reference count was
+    /// incremented at least once
+    fn create_if_regular(file_ptr: ptr::NonNull<bindings::file>) -> Result<RegularFile> {

This function should be unsafe, correct? You instead take a &bindings::file instead of NonNull but still keep it unsafe, so the "valid pointer" invariant is always reached.

Or, could this take &kernel::file::File instead to reduce some duplication?

rust/kernel/file.rs unsafe fn dec_ref(obj: ptr::NonNull<Self>) { · line 225

Quoted 8 lines
rust/kernel/file.rs +8 -0
+        // SAFETY: file_ptr is a NonNull pointer
+        let inode = unsafe { core::ptr::addr_of!((*file_ptr.as_ptr()).f_inode).read() };
+        // SAFETY: the caller must ensure f_inode is initialized to a valid pointer
+        unsafe {
+            if bindings::S_IFMT & ((*inode).i_mode) as u32 != bindings::S_IFREG {
+                return Err(EINVAL);
+            }
+        }

Nit: factor unsafe { ((*inode).i_mode) } out so it doesn't look like the whole statement is unsafe

Quoted 4 lines
rust/kernel/file.rs +4 -0
+        // SAFETY: the safety requirements state that file_ptr's reference count was incremented at
+        // least once
+        Ok(RegularFile(unsafe { ARef::from_raw(file_ptr.cast()) }))
+    }

rust/kernel/file.rs unsafe fn dec_ref(obj: ptr::NonNull<Self>) { · line 239

Quoted 10 lines
rust/kernel/file.rs +10 -0
+    /// Constructs a new [`struct file`] wrapper from a path.
+    pub fn from_path(filename: &CStr, flags: i32, mode: u16) -> Result<Self> {
+        let file_ptr = unsafe {
+            // SAFETY: filename is a reference, so it's a valid pointer
+            from_err_ptr(bindings::filp_open(
+                filename.as_ptr() as *const i8,
+                flags,
+                mode,
+            ))?
+        };

I mentioned in another email that .cast::<i8>() can be more idiomatic and a bit safer than as, it's a stylistic choice but there are a few places it could be changed here if desired

Also, the // SAFETY comments need to go outside the unsafe block

rust/kernel/file.rs unsafe fn dec_ref(obj: ptr::NonNull<Self>) { · line 244

Quoted 5 lines
rust/kernel/file.rs +5 -0
+        let file_ptr = ptr::NonNull::new(file_ptr).ok_or(ENOENT)?;
+
+        // SAFETY: `filp_open` initializes the refcount with 1
+        Self::create_if_regular(file_ptr)
+    }

Will need unsafe block if create_if_regular becomes unsafe

Quoted 16 lines
rust/kernel/file.rs +16 -0
+
+    /// Constructs a new [`struct file`] wrapper from a path and a vfsmount.
+    pub fn from_path_in_root_mnt(
+        mount: &Vfsmount,
+        filename: &CStr,
+        flags: i32,
+        mode: u16,
+    ) -> Result<Self> {
+        let file_ptr = {
+            let mnt = mount.get();
+            // construct a path from vfsmount, see file_open_root_mnt
+            let raw_path = bindings::path {
+                mnt,
+                // SAFETY: Vfsmount structure stores a valid vfsmount object
+                dentry: unsafe { (*mnt).mnt_root },
+            };

rust/kernel/file.rs unsafe fn dec_ref(obj: ptr::NonNull<Self>) { · line 270

Quoted 10 lines
rust/kernel/file.rs +10 -0
+            unsafe {
+                // SAFETY: raw_path and filename are both references
+                from_err_ptr(bindings::file_open_root(
+                    &raw_path,
+                    filename.as_ptr() as *const i8,
+                    flags,
+                    mode,
+                ))?
+            }
+        };

Is there a reason to use the larger scope block, rather than moving mnt and raw_path out and doing let file_ptr = unsafe { ... }? If so, a comment would be good.

Quoted 12 lines
rust/kernel/file.rs +12 -0
+        let file_ptr = ptr::NonNull::new(file_ptr).ok_or(ENOENT)?;
+
+        // SAFETY: `file_open_root` initializes the refcount with 1
+        Self::create_if_regular(file_ptr)
+    }
+
+    /// Read from the file into the specified buffer
+    pub fn read_with_offset(&self, buf: &mut [u8], offset: u64) -> Result<usize> {
+        Ok({
+            // kernel_read_file expects a pointer to a "void *" buffer
+            let mut ptr_to_buf = buf.as_mut_ptr() as *mut core::ffi::c_void;
+            // Unless we give a non-null pointer to the file size:

rust/kernel/file.rs unsafe fn dec_ref(obj: ptr::NonNull<Self>) { · line 292

Quoted 10 lines
rust/kernel/file.rs +10 -0
+            // 1. we cannot give a non-zero value for the offset
+            // 2. we cannot have offset 0 and buffer_size > file_size
+            let mut file_size = 0;
+
+            // SAFETY: 'file' is valid because it's taken from Self, 'buf' and 'file_size` are
+            // references to the stack variables 'ptr_to_buf' and 'file_size'; ptr_to_buf is also
+            // a pointer to a valid buffer that was obtained from a reference
+            let result = unsafe {
+                bindings::kernel_read_file(
+                    self.0 .0.get(),

Is this spacing intentional? If so, (self.0).0.get() may be cleaner

Quoted 6 lines
rust/kernel/file.rs +6 -0
+                    offset.try_into()?,
+                    &mut ptr_to_buf,
+                    buf.len(),
+                    &mut file_size,
+                    bindings::kernel_read_file_id_READING_UNKNOWN,
+                )

rust/kernel/file.rs unsafe fn dec_ref(obj: ptr::NonNull<Self>) { · line 308

Quoted 10 lines
rust/kernel/file.rs +10 -0
+            };
+
+            // kernel_read_file returns the number of bytes read on success or negative on error.
+            if result < 0 {
+                return Err(Error::from_errno(result.try_into()?));
+            }
+
+            result.try_into()?
+        })
+    }

I think you could remove the block here and just return Ok(result.try_into()?)

Quoted 29 lines
rust/kernel/file.rs +22 -1
+
+    /// Allocate and return a vector containing the contents of the entire file
+    pub fn read_to_end(&self) -> Result<Vec<u8>> {
+        let file_size = self.get_file_size()?;
+        let mut buffer = Vec::try_with_capacity(file_size)?;
+        buffer.try_resize(file_size, 0)?;
+        self.read_with_offset(&mut buffer, 0)?;
+        Ok(buffer)
+    }
+
+    fn get_file_size(&self) -> Result<usize> {
+        // SAFETY: 'file' is valid because it's taken from Self
+        let file_size = unsafe { bindings::i_size_read((*self.0 .0.get()).f_inode) };
+
+        if file_size < 0 {
+            return Err(EINVAL);
+        }
+
+        Ok(file_size.try_into()?)
+    }
+}
+
 /// A file descriptor reservation.
 ///
 /// This allows the creation of a file descriptor in two steps: first, we reserve a slot for it,
--

2.41.0

#8 lore raw

Re: [RFC PATCH v2 04/10] rust: file: Add a new RegularFile newtype useful for reading files

Quoted 92 lines · Trevor Gross
Quoted 75 lines · Ariel Miculas

Implement from_path, from_path_in_root_mnt, read_with_offset, read_to_end and get_file_size methods for a RegularFile newtype.

---
 rust/helpers.c       |   6 ++
 rust/kernel/error.rs |   4 +-
 rust/kernel/file.rs  | 129 ++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 135 insertions(+), 4 deletions(-)
rust/helpers.c +6 -0
index eed8ace52fb5..9e860a554cda 100644
@@ -213,6 +213,12 @@ void *rust_helper_alloc_inode_sb(struct super_block *sb,
 }
 EXPORT_SYMBOL_GPL(rust_helper_alloc_inode_sb);
 
+loff_t rust_helper_i_size_read(const struct inode *inode)
+{
+       return i_size_read(inode);
+}
+EXPORT_SYMBOL_GPL(rust_helper_i_size_read);
+
 /*
  * We use `bindgen`'s `--size_t-is-usize` option to bind the C `size_t` type
  * as the Rust `usize` type, so we can use it in contexts where Rust
rust/kernel/error.rs +1 -3
index 05fcab6abfe6..e061c83f806a 100644
@@ -273,9 +273,7 @@ pub fn to_result(err: core::ffi::c_int) -> Result {
 ///     }
 /// }
 /// ```
-// TODO: Remove `dead_code` marker once an in-kernel client is available.
-#[allow(dead_code)]
-pub(crate) fn from_err_ptr<T>(ptr: *mut T) -> Result<*mut T> {
+pub fn from_err_ptr<T>(ptr: *mut T) -> Result<*mut T> {
     // CAST: Casting a pointer to `*const core::ffi::c_void` is always valid.
     let const_ptr: *const core::ffi::c_void = ptr.cast();
     // SAFETY: The FFI function does not deref the pointer.
rust/kernel/file.rs +5 -1
index 494939ba74df..a3002c416dbb 100644
@@ -8,11 +8,13 @@
 use crate::{
     bindings,
     cred::Credential,
-    error::{code::*, from_result, Error, Result},
+    error::{code::*, from_err_ptr, from_result, Error, Result},
     fs,
     io_buffer::{IoBufferReader, IoBufferWriter},
     iov_iter::IovIter,
     mm,
+    mount::Vfsmount,
+    str::CStr,
     sync::CondVar,
     types::ARef,
     types::AlwaysRefCounted,
@@ -20,6 +22,7 @@
     types::Opaque,
     user_ptr::{UserSlicePtr, UserSlicePtrReader, UserSlicePtrWriter},
 };
+use alloc::vec::Vec;
 use core::convert::{TryFrom, TryInto};
 use core::{marker, mem, ptr};
 use macros::vtable;
@@ -201,6 +204,130 @@ unsafe fn dec_ref(obj: ptr::NonNull<Self>) {
     }
 }
 
+/// A newtype over file, specific to regular files

rust/kernel/file.rs unsafe fn dec_ref(obj: ptr::NonNull<Self>) { · line 217

Quoted 10 lines
rust/kernel/file.rs +10 -0
+pub struct RegularFile(ARef<File>);
+impl RegularFile {
+    /// Creates a new instance of Self if the file is a regular file
+    ///
+    /// # Safety
+    ///
+    /// The caller must ensure file_ptr.f_inode is initialized to a valid pointer (e.g. file_ptr is
+    /// a pointer returned by path_openat); It must also ensure that file_ptr's reference count was
+    /// incremented at least once
+    fn create_if_regular(file_ptr: ptr::NonNull<bindings::file>) -> Result<RegularFile> {

This function should be unsafe, correct? You instead take a &bindings::file instead of NonNull but still keep it unsafe, so the "valid pointer" invariant is always reached.

Quoted 2 lines

Or, could this take &kernel::file::File instead to reduce some duplication?

Yes, this should have been unsafe, I've also changed it to receive a *mut bindings::file instead of NonNull.

Quoted 27 lines

rust/kernel/file.rs unsafe fn dec_ref(obj: ptr::NonNull<Self>) { · line 225

Quoted 8 lines
rust/kernel/file.rs +8 -0
+        // SAFETY: file_ptr is a NonNull pointer
+        let inode = unsafe { core::ptr::addr_of!((*file_ptr.as_ptr()).f_inode).read() };
+        // SAFETY: the caller must ensure f_inode is initialized to a valid pointer
+        unsafe {
+            if bindings::S_IFMT & ((*inode).i_mode) as u32 != bindings::S_IFREG {
+                return Err(EINVAL);
+            }
+        }

Nit: factor unsafe { ((*inode).i_mode) } out so it doesn't look like the whole statement is unsafe

Quoted 4 lines
rust/kernel/file.rs +4 -0
+        // SAFETY: the safety requirements state that file_ptr's reference count was incremented at
+        // least once
+        Ok(RegularFile(unsafe { ARef::from_raw(file_ptr.cast()) }))
+    }

rust/kernel/file.rs unsafe fn dec_ref(obj: ptr::NonNull<Self>) { · line 239

Quoted 10 lines
rust/kernel/file.rs +10 -0
+    /// Constructs a new [`struct file`] wrapper from a path.
+    pub fn from_path(filename: &CStr, flags: i32, mode: u16) -> Result<Self> {
+        let file_ptr = unsafe {
+            // SAFETY: filename is a reference, so it's a valid pointer
+            from_err_ptr(bindings::filp_open(
+                filename.as_ptr() as *const i8,
+                flags,
+                mode,
+            ))?
+        };
Quoted 3 lines

I mentioned in another email that .cast::<i8>() can be more idiomatic and a bit safer than as, it's a stylistic choice but there are a few places it could be changed here if desired

Thanks, didn't know about this.

Quoted 38 lines

Also, the // SAFETY comments need to go outside the unsafe block

rust/kernel/file.rs unsafe fn dec_ref(obj: ptr::NonNull<Self>) { · line 244

Quoted 5 lines
rust/kernel/file.rs +5 -0
+        let file_ptr = ptr::NonNull::new(file_ptr).ok_or(ENOENT)?;
+
+        // SAFETY: `filp_open` initializes the refcount with 1
+        Self::create_if_regular(file_ptr)
+    }

Will need unsafe block if create_if_regular becomes unsafe

Quoted 16 lines
rust/kernel/file.rs +16 -0
+
+    /// Constructs a new [`struct file`] wrapper from a path and a vfsmount.
+    pub fn from_path_in_root_mnt(
+        mount: &Vfsmount,
+        filename: &CStr,
+        flags: i32,
+        mode: u16,
+    ) -> Result<Self> {
+        let file_ptr = {
+            let mnt = mount.get();
+            // construct a path from vfsmount, see file_open_root_mnt
+            let raw_path = bindings::path {
+                mnt,
+                // SAFETY: Vfsmount structure stores a valid vfsmount object
+                dentry: unsafe { (*mnt).mnt_root },
+            };

rust/kernel/file.rs unsafe fn dec_ref(obj: ptr::NonNull<Self>) { · line 270

Quoted 10 lines
rust/kernel/file.rs +10 -0
+            unsafe {
+                // SAFETY: raw_path and filename are both references
+                from_err_ptr(bindings::file_open_root(
+                    &raw_path,
+                    filename.as_ptr() as *const i8,
+                    flags,
+                    mode,
+                ))?
+            }
+        };
Quoted 3 lines

Is there a reason to use the larger scope block, rather than moving mnt and raw_path out and doing let file_ptr = unsafe { ... }? If so, a comment would be good.

No, that's just how the code has evolved from splitting unsafe blocks, I've changed this in
https://github.com/ariel-miculas/linux/tree/puzzlefs_rfc

Quoted 17 lines
Quoted 17 lines
rust/kernel/file.rs +16 -0
+        let file_ptr = ptr::NonNull::new(file_ptr).ok_or(ENOENT)?;
+
+        // SAFETY: `file_open_root` initializes the refcount with 1
+        Self::create_if_regular(file_ptr)
+    }
+
+    /// Read from the file into the specified buffer
+    pub fn read_with_offset(&self, buf: &mut [u8], offset: u64) -> Result<usize> {
+        Ok({
+            // kernel_read_file expects a pointer to a "void *" buffer
+            let mut ptr_to_buf = buf.as_mut_ptr() as *mut core::ffi::c_void;
+            // Unless we give a non-null pointer to the file size:
+            // 1. we cannot give a non-zero value for the offset
+            // 2. we cannot have offset 0 and buffer_size > file_size
+            let mut file_size = 0;
+
Quoted 8 lines

rust/kernel/file.rs unsafe fn dec_ref(obj: ptr::NonNull<Self>) { · line 292

Quoted 6 lines
rust/kernel/file.rs +6 -0
+            // SAFETY: 'file' is valid because it's taken from Self, 'buf' and 'file_size` are
+            // references to the stack variables 'ptr_to_buf' and 'file_size'; ptr_to_buf is also
+            // a pointer to a valid buffer that was obtained from a reference
+            let result = unsafe {
+                bindings::kernel_read_file(
+                    self.0 .0.get(),

Is this spacing intentional? If so, (self.0).0.get() may be cleaner

No, it's not intentional, this is rustfmt`s creation.

Quoted 11 lines
Quoted 11 lines
rust/kernel/file.rs +10 -0
+                    offset.try_into()?,
+                    &mut ptr_to_buf,
+                    buf.len(),
+                    &mut file_size,
+                    bindings::kernel_read_file_id_READING_UNKNOWN,
+                )
+            };
+
+            // kernel_read_file returns the number of bytes read on success or negative on error.
+            if result < 0 {
Quoted 8 lines

rust/kernel/file.rs unsafe fn dec_ref(obj: ptr::NonNull<Self>) { · line 308

Quoted 6 lines
rust/kernel/file.rs +6 -0
+                return Err(Error::from_errno(result.try_into()?));
+            }
+
+            result.try_into()?
+        })
+    }

I think you could remove the block here and just return Ok(result.try_into()?)

Good idea.

Quoted 31 lines
Quoted 30 lines
rust/kernel/file.rs +22 -1
+
+    /// Allocate and return a vector containing the contents of the entire file
+    pub fn read_to_end(&self) -> Result<Vec<u8>> {
+        let file_size = self.get_file_size()?;
+        let mut buffer = Vec::try_with_capacity(file_size)?;
+        buffer.try_resize(file_size, 0)?;
+        self.read_with_offset(&mut buffer, 0)?;
+        Ok(buffer)
+    }
+
+    fn get_file_size(&self) -> Result<usize> {
+        // SAFETY: 'file' is valid because it's taken from Self
+        let file_size = unsafe { bindings::i_size_read((*self.0 .0.get()).f_inode) };
+
+        if file_size < 0 {
+            return Err(EINVAL);
+        }
+
+        Ok(file_size.try_into()?)
+    }
+}
+
 /// A file descriptor reservation.
 ///
 /// This allows the creation of a file descriptor in two steps: first, we reserve a slot for it,
--

2.41.0

#9 lore raw

RFC PATCH v2 5/10 samples: puzzlefs: add basic deserializing support for the puzzlefs metadata

In reply to Ariel Miculas

Puzzlefs uses capnproto for storing the filesystem metadata, defined in manifest.capnp and metadata.capnp. The files manifest_capnp.rs and metadata_capnp.rs were generated with (Cap'n Proto version 0.10.4):

```
capnp compile -orust manifest.capnp
capnp compile -orust metadata.capnp
```
, formatted with rustfmt and I have also added the following lines:
```
```

to avoid warnings.

Ideally, manifest_capnp.rs and metadata_capnp.rs should be automatically generated at build time. However, this means the capnp binary becomes a dependency for the build and I didn't want this. Besides, the metadata schemas are not expected to change frequently, so it's acceptable to manually regenerate the two rust files when this happens.

The code is adapted from the puzzlefs FUSE driver [1]. The data structures required for the filesystem metadata are defined in types.rs. Each structure has a from_capnp method used to convert between the capnp memory layout and the native Rust structures. This leaves room for improvement [2].

The deserializing code also uses NoAllocBufferSegments which is not yet merged in capnproto-rust upstream [3]. When a new release is published, then we'll be able to use capnproto-rust in the kernel with only a few patches to it.

inode.rs implements the PuzzleFS structure which contains a list of metadata layers.

---
 rust/kernel/error.rs                        |    2 +-
 samples/rust/puzzle.rs                      |    4 +
 samples/rust/puzzle/error.rs                |  106 +
 samples/rust/puzzle/inode.rs                |   36 +
 samples/rust/puzzle/types.rs                |  340 ++
 samples/rust/puzzle/types/manifest.capnp    |   15 +
 samples/rust/puzzle/types/manifest_capnp.rs |  773 ++++
 samples/rust/puzzle/types/metadata.capnp    |   69 +
 samples/rust/puzzle/types/metadata_capnp.rs | 4458 +++++++++++++++++++
 samples/rust/puzzlefs.rs                    |    4 +
 10 files changed, 5806 insertions(+), 1 deletion(-)
 create mode 100644 samples/rust/puzzle.rs
 create mode 100644 samples/rust/puzzle/error.rs
 create mode 100644 samples/rust/puzzle/inode.rs
 create mode 100644 samples/rust/puzzle/types.rs
 create mode 100644 samples/rust/puzzle/types/manifest.capnp
 create mode 100644 samples/rust/puzzle/types/manifest_capnp.rs
 create mode 100644 samples/rust/puzzle/types/metadata.capnp
 create mode 100644 samples/rust/puzzle/types/metadata_capnp.rs
rust/kernel/error.rs +1 -1
index e061c83f806a..7d25046a61b8 100644
@@ -99,7 +99,7 @@ impl Error {
     ///
     /// It is a bug to pass an out-of-range `errno`. `EINVAL` would
     /// be returned in such a case.
-    pub(crate) fn from_errno(errno: core::ffi::c_int) -> Error {
+    pub fn from_errno(errno: core::ffi::c_int) -> Error {
         if errno < -(bindings::MAX_ERRNO as i32) || errno >= 0 {
             // TODO: Make it a `WARN_ONCE` once available.
             crate::pr_warn!(
samples/rust/puzzle.rs new file +4 -0
new file mode 100644
index 000000000000..a809cddfb817
@@ -0,0 +1,4 @@
+pub(crate) mod error;
+pub(crate) mod types;
+pub(crate) use types::{manifest_capnp, metadata_capnp};
+pub(crate) mod inode;
samples/rust/puzzle/error.rs new file +106 -0

106 lines not on this page · raw message

samples/rust/puzzle/inode.rs new file +36 -0
new file mode 100644
index 000000000000..d792f661aa00
@@ -0,0 +1,36 @@
+// This contents of this file is taken from puzzlefs.rs (the userspace implementation)
+// It is named inode.rs instead puzzlefs.rs since the root of this kernel module already has that name
+
+use crate::puzzle::error::Result;
+use crate::puzzle::error::WireFormatError;
+use crate::puzzle::types as format;
+use crate::puzzle::types::{Inode, InodeMode, MetadataBlob};
+use alloc::vec::Vec;
+use kernel::prelude::ENOENT;
+
+pub(crate) struct PuzzleFS {
+    layers: Vec<format::MetadataBlob>,
+}
+
+impl PuzzleFS {
+    pub(crate) fn new(md: MetadataBlob) -> Result<Self> {
+        let mut v = Vec::new();
+        v.try_push(md)?;
+        Ok(PuzzleFS { layers: v })
+    }
+
+    pub(crate) fn find_inode(&self, ino: u64) -> Result<Inode> {
+        for layer in self.layers.iter() {
+            if let Some(inode) = layer.find_inode(ino)? {
+                let inode = Inode::from_capnp(inode)?;
+                if let InodeMode::Wht = inode.mode {
+                    // TODO: seems like this should really be an Option.
+                    return Err(WireFormatError::from_errno(ENOENT));
+                }
+                return Ok(inode);
+            }
+        }
+
+        Err(WireFormatError::from_errno(ENOENT))
+    }
+}
samples/rust/puzzle/types.rs new file +340 -0

340 lines not on this page · raw message

samples/rust/puzzle/types/manifest.capnp new file +15 -0
new file mode 100644
index 000000000000..24d8a6f1fe19
@@ -0,0 +1,15 @@
+@0xe0b6d460b1f22bb5;
+
+using Metadata = import "metadata.capnp";
+
+struct VerityData {
+        digest@0: Data;
+        verity@1: Data;
+}
+
+struct Rootfs {
+        metadatas@0: List(Metadata.BlobRef);
+        fsVerityData@1: List(VerityData);
+        manifestVersion@2: UInt64;
+}
+
samples/rust/puzzle/types/manifest_capnp.rs new file +773 -0

773 lines not on this page · raw message

samples/rust/puzzle/types/metadata.capnp new file +69 -0

69 lines not on this page · raw message

samples/rust/puzzle/types/metadata_capnp.rs new file +4458 -0

1,649 lines not on this page · raw message

samples/rust/puzzlefs.rs +4 -0
index 0cf42762e81a..9afd82745b64 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.41.0

Part of the diff is not shown here. View the raw message.

#10 lore raw

RFC PATCH v2 6/10 rust: file: pass the filesystem context to the open function

In reply to Ariel Miculas

This allows us to create a Vfsmount structure and pass it to the read callback.

---
 rust/kernel/file.rs      | 17 +++++++++++++++--
 samples/rust/puzzlefs.rs | 40 +++++++++++++++++++++++++++++++++++-----
 samples/rust/rust_fs.rs  |  3 ++-
 3 files changed, 52 insertions(+), 8 deletions(-)
rust/kernel/file.rs +15 -2
index a3002c416dbb..af1eb1ee9267 100644
@@ -457,9 +457,15 @@ impl<A: OpenAdapter<T::OpenData>, T: Operations> OperationsVtable<A, T> {
             // `fileref` never outlives this function, so it is guaranteed to be
             // valid.
             let fileref = unsafe { File::from_ptr(file) };
+
+            // SAFETY: into_foreign was called in fs::NewSuperBlock<..., NeedsInit>::init and
+            // it is valid until from_foreign will be called in fs::Tables::free_callback
+            let fs_info =
+                unsafe { <T::Filesystem as fs::Type>::Data::borrow((*(*inode).i_sb).s_fs_info) };
+
             // SAFETY: `arg` was previously returned by `A::convert` and must
             // be a valid non-null pointer.
-            let ptr = T::open(unsafe { &*arg }, fileref)?.into_foreign();
+            let ptr = T::open(fs_info, unsafe { &*arg }, fileref)?.into_foreign();
             // SAFETY: The C contract guarantees that `private_data` is available
             // for implementers of the file operations (no other C code accesses
             // it), so we know that there are no concurrent threads/CPUs accessing
@@ -930,10 +936,17 @@ pub trait Operations {
     /// The type of the context data passed to [`Operations::open`].
     type OpenData: Sync = ();
 
+    /// Data associated with each file system instance.
+    type Filesystem: fs::Type;
+
     /// Creates a new instance of this file.
     ///
     /// Corresponds to the `open` function pointer in `struct file_operations`.
-    fn open(context: &Self::OpenData, file: &File) -> Result<Self::Data>;
+    fn open(
+        fs_info: <<Self::Filesystem as fs::Type>::Data as ForeignOwnable>::Borrowed<'_>,
+        context: &Self::OpenData,
+        file: &File,
+    ) -> Result<Self::Data>;
 
     /// Cleans up after the last reference to the file goes away.
     ///
samples/rust/puzzlefs.rs +35 -5
index 9afd82745b64..8a64e0bd437d 100644
@@ -3,8 +3,14 @@
 //! Rust file system sample.
 
 use kernel::module_fs;
+use kernel::mount::Vfsmount;
 use kernel::prelude::*;
-use kernel::{c_str, file, fs, io_buffer::IoBufferWriter};
+use kernel::{
+    c_str, file, fmt, fs,
+    io_buffer::IoBufferWriter,
+    str::CString,
+    sync::{Arc, ArcBorrow},
+};
 
 mod puzzle;
 // Required by the autogenerated '_capnp.rs' files
@@ -19,6 +25,12 @@
 
 struct PuzzleFsModule;
 
+#[derive(Debug)]
+struct PuzzlefsInfo {
+    base_path: CString,
+    vfs_mount: Arc<Vfsmount>,
+}
+
 #[vtable]
 impl fs::Context<Self> for PuzzleFsModule {
     type Data = ();
@@ -46,14 +58,23 @@ fn try_new() -> Result {
 impl fs::Type for PuzzleFsModule {
     type Context = Self;
     type INodeData = &'static [u8];
+    type Data = Box<PuzzlefsInfo>;
     const SUPER_TYPE: fs::Super = fs::Super::Independent;
     const NAME: &'static CStr = c_str!("puzzlefs");
     const FLAGS: i32 = fs::flags::USERNS_MOUNT;
     const DCACHE_BASED: bool = true;
 
     fn fill_super(_data: (), sb: fs::NewSuperBlock<'_, Self>) -> Result<&fs::SuperBlock<Self>> {
+        let base_path = CString::try_from_fmt(fmt!("hello world"))?;
+        pr_info!("base_path {:?}\n", base_path);
+        let vfs_mount = Vfsmount::new_private_mount(c_str!("/home/puzzlefs_oci"))?;
+        pr_info!("vfs_mount {:?}\n", vfs_mount);
+
         let sb = sb.init(
-            (),
+            Box::try_new(PuzzlefsInfo {
+                base_path,
+                vfs_mount: Arc::try_new(vfs_mount)?,
+            })?,
             &fs::SuperParams {
                 magic: 0x72757374,
                 ..fs::SuperParams::DEFAULT
@@ -88,14 +109,23 @@ fn fill_super(_data: (), sb: fs::NewSuperBlock<'_, Self>) -> Result<&fs::SuperBl
 
 #[vtable]
 impl file::Operations for FsFile {
+    // must be the same as INodeData
     type OpenData = &'static [u8];
+    type Filesystem = PuzzleFsModule;
+    // this is an Arc because Data must be ForeignOwnable and the only implementors of it are Box,
+    // Arc and (); we cannot pass a reference to read, so we share Vfsmount using and Arc
+    type Data = Arc<Vfsmount>;
 
-    fn open(_context: &Self::OpenData, _file: &file::File) -> Result<Self::Data> {
-        Ok(())
+    fn open(
+        fs_info: &PuzzlefsInfo,
+        _context: &Self::OpenData,
+        _file: &file::File,
+    ) -> Result<Self::Data> {
+        Ok(fs_info.vfs_mount.clone())
     }
 
     fn read(
-        _data: (),
+        data: ArcBorrow<'_, Vfsmount>,
         file: &file::File,
         writer: &mut impl IoBufferWriter,
         offset: u64,
samples/rust/rust_fs.rs +2 -1
index 7527681ee024..c58ed1560e06 100644
@@ -85,8 +85,9 @@ fn fill_super(_data: (), sb: fs::NewSuperBlock<'_, Self>) -> Result<&fs::SuperBl
 #[vtable]
 impl file::Operations for FsFile {
     type OpenData = &'static [u8];
+    type Filesystem = RustFs;
 
-    fn open(_context: &Self::OpenData, _file: &file::File) -> Result<Self::Data> {
+    fn open(_fs_info: (), _context: &Self::OpenData, _file: &file::File) -> Result<Self::Data> {
         Ok(())
     }
 
Signature · 1 line
2.41.0
#11 lore raw

Re: [RFC PATCH v2 06/10] rust: file: pass the filesystem context to the open function

Quoted 152 lines · Ariel Miculas

This allows us to create a Vfsmount structure and pass it to the read callback.

---
 rust/kernel/file.rs      | 17 +++++++++++++++--
 samples/rust/puzzlefs.rs | 40 +++++++++++++++++++++++++++++++++++-----
 samples/rust/rust_fs.rs  |  3 ++-
 3 files changed, 52 insertions(+), 8 deletions(-)
rust/kernel/file.rs +15 -2
index a3002c416dbb..af1eb1ee9267 100644
@@ -457,9 +457,15 @@ impl<A: OpenAdapter<T::OpenData>, T: Operations> OperationsVtable<A, T> {
             // `fileref` never outlives this function, so it is guaranteed to be
             // valid.
             let fileref = unsafe { File::from_ptr(file) };
+
+            // SAFETY: into_foreign was called in fs::NewSuperBlock<..., NeedsInit>::init and
+            // it is valid until from_foreign will be called in fs::Tables::free_callback
+            let fs_info =
+                unsafe { <T::Filesystem as fs::Type>::Data::borrow((*(*inode).i_sb).s_fs_info) };
+
             // SAFETY: `arg` was previously returned by `A::convert` and must
             // be a valid non-null pointer.
-            let ptr = T::open(unsafe { &*arg }, fileref)?.into_foreign();
+            let ptr = T::open(fs_info, unsafe { &*arg }, fileref)?.into_foreign();
             // SAFETY: The C contract guarantees that `private_data` is available
             // for implementers of the file operations (no other C code accesses
             // it), so we know that there are no concurrent threads/CPUs accessing
@@ -930,10 +936,17 @@ pub trait Operations {
     /// The type of the context data passed to [`Operations::open`].
     type OpenData: Sync = ();
 
+    /// Data associated with each file system instance.
+    type Filesystem: fs::Type;
+
     /// Creates a new instance of this file.
     ///
     /// Corresponds to the `open` function pointer in `struct file_operations`.
-    fn open(context: &Self::OpenData, file: &File) -> Result<Self::Data>;
+    fn open(
+        fs_info: <<Self::Filesystem as fs::Type>::Data as ForeignOwnable>::Borrowed<'_>,
+        context: &Self::OpenData,
+        file: &File,
+    ) -> Result<Self::Data>;
 
     /// Cleans up after the last reference to the file goes away.
     ///
samples/rust/puzzlefs.rs +35 -5
index 9afd82745b64..8a64e0bd437d 100644
@@ -3,8 +3,14 @@
 //! Rust file system sample.
 
 use kernel::module_fs;
+use kernel::mount::Vfsmount;
 use kernel::prelude::*;
-use kernel::{c_str, file, fs, io_buffer::IoBufferWriter};
+use kernel::{
+    c_str, file, fmt, fs,
+    io_buffer::IoBufferWriter,
+    str::CString,
+    sync::{Arc, ArcBorrow},
+};
 
 mod puzzle;
 // Required by the autogenerated '_capnp.rs' files
@@ -19,6 +25,12 @@
 
 struct PuzzleFsModule;
 
+#[derive(Debug)]
+struct PuzzlefsInfo {
+    base_path: CString,
+    vfs_mount: Arc<Vfsmount>,
+}
+
 #[vtable]
 impl fs::Context<Self> for PuzzleFsModule {
     type Data = ();
@@ -46,14 +58,23 @@ fn try_new() -> Result {
 impl fs::Type for PuzzleFsModule {
     type Context = Self;
     type INodeData = &'static [u8];
+    type Data = Box<PuzzlefsInfo>;
     const SUPER_TYPE: fs::Super = fs::Super::Independent;
     const NAME: &'static CStr = c_str!("puzzlefs");
     const FLAGS: i32 = fs::flags::USERNS_MOUNT;
     const DCACHE_BASED: bool = true;
 
     fn fill_super(_data: (), sb: fs::NewSuperBlock<'_, Self>) -> Result<&fs::SuperBlock<Self>> {
+        let base_path = CString::try_from_fmt(fmt!("hello world"))?;
+        pr_info!("base_path {:?}\n", base_path);
+        let vfs_mount = Vfsmount::new_private_mount(c_str!("/home/puzzlefs_oci"))?;
+        pr_info!("vfs_mount {:?}\n", vfs_mount);
+
         let sb = sb.init(
-            (),
+            Box::try_new(PuzzlefsInfo {
+                base_path,
+                vfs_mount: Arc::try_new(vfs_mount)?,
+            })?,
             &fs::SuperParams {
                 magic: 0x72757374,
                 ..fs::SuperParams::DEFAULT
@@ -88,14 +109,23 @@ fn fill_super(_data: (), sb: fs::NewSuperBlock<'_, Self>) -> Result<&fs::SuperBl
 
 #[vtable]
 impl file::Operations for FsFile {
+    // must be the same as INodeData
     type OpenData = &'static [u8];
+    type Filesystem = PuzzleFsModule;
+    // this is an Arc because Data must be ForeignOwnable and the only implementors of it are Box,
+    // Arc and (); we cannot pass a reference to read, so we share Vfsmount using and Arc
+    type Data = Arc<Vfsmount>;
 
-    fn open(_context: &Self::OpenData, _file: &file::File) -> Result<Self::Data> {
-        Ok(())
+    fn open(
+        fs_info: &PuzzlefsInfo,
+        _context: &Self::OpenData,
+        _file: &file::File,
+    ) -> Result<Self::Data> {
+        Ok(fs_info.vfs_mount.clone())
     }
 
     fn read(
-        _data: (),
+        data: ArcBorrow<'_, Vfsmount>,
         file: &file::File,
         writer: &mut impl IoBufferWriter,
         offset: u64,
samples/rust/rust_fs.rs +2 -2
index 7527681ee024..c58ed1560e06 100644
@@ -85,8 +85,9 @@ fn fill_super(_data: (), sb: fs::NewSuperBlock<'_, Self>) -> Result<&fs::SuperBl
 #[vtable]
 impl file::Operations for FsFile {
     type OpenData = &'static [u8];
+    type Filesystem = RustFs;
 
-    fn open(_context: &Self::OpenData, _file: &file::File) -> Result<Self::Data> {
+    fn open(_fs_info: (), _context: &Self::OpenData, _file: &file::File) -> Result<Self::Data> {
         Ok(())
     }
 
--
Quoted 3 lines

2.41.0

Hey Wedson,

Is it ok to couple file::Operations with fs::Type? I didn't find a better way to implement this.
I'm asking because I've seen you've gone to great lengths to decouple them.

Cheers,
Ariel

#12 lore raw

RFC PATCH v2 7/10 samples: puzzlefs: populate the directory entries with the inodes from the puzzlefs metadata file

In reply to Ariel Miculas

I've used the same strategy as in try_new_populated_root_dentry from rust/kernel/fs.rs, that is, recursively traverse the filesystem starting from the root and create the necessary dentries. This is not ideal because it creates all the dentries up front, but there are no rust filesystem abstractions for implementing lookups yet.

---
 rust/alloc/vec/mod.rs    |  17 +++++
 rust/kernel/fs.rs        |   4 +-
 samples/rust/puzzlefs.rs | 141 +++++++++++++++++++++++++++++----------
 3 files changed, 127 insertions(+), 35 deletions(-)
rust/alloc/vec/mod.rs +17 -0
index e8344c77939e..b755cf0b936c 100644
@@ -2499,6 +2499,15 @@ unsafe fn split_at_spare_mut_with_len(
     }
 }
 
+impl<T: Clone> Vec<T> {
+    /// Try to clone the vector using the global allocator
+    #[inline]
+    #[stable(feature = "kernel", since = "1.0.0")]
+    pub fn try_clone(&self) -> Result<Self, TryReserveError> {
+        self.try_clone_in(Global)
+    }
+}
+
 impl<T: Clone, A: Allocator> Vec<T, A> {
     /// Resizes the `Vec` in-place so that `len` is equal to `new_len`.
     ///
@@ -2623,6 +2632,14 @@ pub fn try_extend_from_slice(&mut self, other: &[T]) -> Result<(), TryReserveErr
         self.try_spec_extend(other.iter())
     }
 
+    /// Tries to clone the vector using the given allocator
+    #[stable(feature = "kernel", since = "1.0.0")]
+    pub fn try_clone_in(&self, allocator: A) -> Result<Self, TryReserveError> {
+        let mut new_vec = Vec::try_with_capacity_in(self.len(), allocator)?;
+        new_vec.try_extend_from_slice(&self)?;
+        Ok(new_vec)
+    }
+
     /// Copies elements from `src` range to the end of the vector.
     ///
     /// # Panics
rust/kernel/fs.rs +3 -1
index 10f90f78fc54..709b79fa0030 100644
@@ -781,7 +781,9 @@ impl SuperParams {
 ///
 /// The superblock is a newly-created one and this is the only active pointer to it.
 pub struct NewSuperBlock<'a, T: Type + ?Sized, S = NeedsInit> {
-    sb: &'a mut SuperBlock<T>,
+    /// Pointer to the superblock; this fields is public so puzzlefs can call
+    /// try_new_dcache_dir_inode when populating the directory hierarchy
+    pub sb: &'a mut SuperBlock<T>,
 
     // This also forces `'a` to be invariant.
     _p: PhantomData<&'a mut &'a S>,
samples/rust/puzzlefs.rs +107 -34

201 lines not on this page · raw message

Signature · 1 line
2.41.0
#13 lore raw

RFC PATCH v2 8/10 rust: puzzlefs: read the puzzlefs image manifest instead of an individual metadata layer

In reply to Ariel Miculas

The puzzlefs image manifest is the file referenced by index.json and it contains the array of metadata layers that describe the puzzlefs image. This file represents the root of the puzzlefs filesystem since we can't parse json files. When this filesystem is mounted, usermode will need to read the tag from index.json, find the corresponding puzzlefs image manifest and pass it to the kernel module.

Due to the lack of BTreeMap in the kernel, only image manifests without fs verity information are supported for now.

---
 samples/rust/puzzle.rs       |  1 +
 samples/rust/puzzle/inode.rs | 22 ++++++++++++----
 samples/rust/puzzle/oci.rs   | 51 ++++++++++++++++++++++++++++++++++++
 samples/rust/puzzlefs.rs     | 23 ++++++++--------
 4 files changed, 80 insertions(+), 17 deletions(-)
 create mode 100644 samples/rust/puzzle/oci.rs
samples/rust/puzzle.rs +1 -0
index a809cddfb817..e74a248c39ff 100644
@@ -2,3 +2,4 @@
 pub(crate) mod types;
 pub(crate) use types::{manifest_capnp, metadata_capnp};
 pub(crate) mod inode;
+pub(crate) mod oci;
samples/rust/puzzle/inode.rs +17 -5
index d792f661aa00..f63cdbc1eac4 100644
@@ -3,20 +3,32 @@
 
 use crate::puzzle::error::Result;
 use crate::puzzle::error::WireFormatError;
+use crate::puzzle::oci::Image;
 use crate::puzzle::types as format;
-use crate::puzzle::types::{Inode, InodeMode, MetadataBlob};
+use crate::puzzle::types::{Digest, Inode, InodeMode};
 use alloc::vec::Vec;
+use kernel::mount::Vfsmount;
 use kernel::prelude::ENOENT;
+use kernel::str::CStr;
+use kernel::sync::Arc;
 
 pub(crate) struct PuzzleFS {
+    pub(crate) oci: Image,
     layers: Vec<format::MetadataBlob>,
 }
 
 impl PuzzleFS {
-    pub(crate) fn new(md: MetadataBlob) -> Result<Self> {
-        let mut v = Vec::new();
-        v.try_push(md)?;
-        Ok(PuzzleFS { layers: v })
+    pub(crate) fn open(vfsmount: Arc<Vfsmount>, rootfs_path: &CStr) -> Result<PuzzleFS> {
+        let oci = Image::open(vfsmount)?;
+        let rootfs = oci.open_rootfs_blob(rootfs_path)?;
+
+        let mut layers = Vec::new();
+        for md in rootfs.metadatas.iter() {
+            let digest = Digest::try_from(md)?;
+            layers.try_push(oci.open_metadata_blob(&digest)?)?;
+        }
+
+        Ok(PuzzleFS { oci, layers })
     }
 
     pub(crate) fn find_inode(&self, ino: u64) -> Result<Inode> {
samples/rust/puzzle/oci.rs new file +51 -0

51 lines not on this page · raw message

samples/rust/puzzlefs.rs +11 -12
index 1f0073716d91..76dc59403db3 100644
@@ -14,7 +14,7 @@
 mod puzzle;
 // Required by the autogenerated '_capnp.rs' files
 use puzzle::inode::PuzzleFS;
-use puzzle::types::{Inode, InodeMode, MetadataBlob};
+use puzzle::types::{Inode, InodeMode};
 use puzzle::{manifest_capnp, metadata_capnp};
 
 use kernel::fs::{DEntry, INodeParams, NeedsRoot, NewSuperBlock, RootDEntry};
@@ -69,7 +69,7 @@ fn puzzlefs_populate_dir(
         return Err(E2BIG);
     }
 
-    let inode = Arc::try_new(pfs.find_inode(ino).map_err(|_| EINVAL)?)?;
+    let inode = Arc::try_new(pfs.find_inode(ino)?)?;
     match &inode.mode {
         InodeMode::File { chunks: _ } => {
             let params = INodeParams {
@@ -154,18 +154,17 @@ fn fill_super(_data: (), sb: fs::NewSuperBlock<'_, Self>) -> Result<&fs::SuperBl
             },
         )?;
 
-        let file = file::RegularFile::from_path_in_root_mnt(
-            &arc_vfs_mount,
-            c_str!("997eed138af30d187e87d682dd2ae9f240fae78f668907a0519460b397c82467"),
-            file::flags::O_RDONLY.try_into().unwrap(),
-            0,
-        )?;
+        let puzzlefs = PuzzleFS::open(
+            arc_vfs_mount,
+            c_str!("2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b"),
+        );
 
-        // TODO: figure out how to go from WireFormatError to kernel::error::Error
-        let metadata = MetadataBlob::new(file).map_err(|_| EINVAL)?;
+        if let Err(ref e) = puzzlefs {
+            pr_info!("error opening puzzlefs {e}\n");
+        }
 
-        let mut puzzlefs = PuzzleFS::new(metadata).map_err(|_| EINVAL)?;
-        let root_inode = Arc::try_new(puzzlefs.find_inode(1).map_err(|_| EINVAL)?)?;
+        let mut puzzlefs = puzzlefs?;
+        let root_inode = Arc::try_new(puzzlefs.find_inode(1)?)?;
 
         let root = try_new_populated_root_puzzlefs_dentry(&sb, &mut puzzlefs, root_inode)?;
         let sb = sb.init_root(root)?;
Signature · 1 line
2.41.0
#14 lore raw

RFC PATCH v2 9/10 rust: puzzlefs: add support for reading files

In reply to Ariel Miculas

Each file has an associated list of file chunks, which are identified using a content addressable blob and an offset. These were generated by the puzzlefs build command, which uses FastCDC to split a filesystem into chunks.

---
 samples/rust/puzzle/inode.rs | 61 +++++++++++++++++++++++++++++++++---
 samples/rust/puzzle/oci.rs   | 32 +++++++++++++++----
 samples/rust/puzzlefs.rs     | 54 +++++++++++++------------------
 3 files changed, 105 insertions(+), 42 deletions(-)
samples/rust/puzzle/inode.rs +57 -4
index f63cdbc1eac4..03c9f6bee75f 100644
@@ -7,10 +7,10 @@
 use crate::puzzle::types as format;
 use crate::puzzle::types::{Digest, Inode, InodeMode};
 use alloc::vec::Vec;
+use core::cmp::min;
 use kernel::mount::Vfsmount;
-use kernel::prelude::ENOENT;
+use kernel::prelude::{ENOENT, ENOTDIR};
 use kernel::str::CStr;
-use kernel::sync::Arc;
 
 pub(crate) struct PuzzleFS {
     pub(crate) oci: Image,
@@ -18,8 +18,9 @@ pub(crate) struct PuzzleFS {
 }
 
 impl PuzzleFS {
-    pub(crate) fn open(vfsmount: Arc<Vfsmount>, rootfs_path: &CStr) -> Result<PuzzleFS> {
-        let oci = Image::open(vfsmount)?;
+    pub(crate) fn open(oci_root_dir: &CStr, rootfs_path: &CStr) -> Result<PuzzleFS> {
+        let vfs_mount = Vfsmount::new_private_mount(oci_root_dir)?;
+        let oci = Image::open(vfs_mount)?;
         let rootfs = oci.open_rootfs_blob(rootfs_path)?;
 
         let mut layers = Vec::new();
@@ -46,3 +47,55 @@ pub(crate) fn find_inode(&self, ino: u64) -> Result<Inode> {
         Err(WireFormatError::from_errno(ENOENT))
     }
 }
+
+pub(crate) fn file_read(
+    oci: &Image,
+    inode: &Inode,
+    offset: usize,
+    data: &mut [u8],
+) -> Result<usize> {
+    let chunks = match &inode.mode {
+        InodeMode::File { chunks } => chunks,
+        _ => return Err(WireFormatError::from_errno(ENOTDIR)),
+    };
+
+    // TODO: fix all this casting...
+    let end = offset + data.len();
+
+    let mut file_offset = 0;
+    let mut buf_offset = 0;
+    for chunk in chunks {
+        // have we read enough?
+        if file_offset > end {
+            break;
+        }
+
+        // should we skip this chunk?
+        if file_offset + (chunk.len as usize) < offset {
+            file_offset += chunk.len as usize;
+            continue;
+        }
+
+        let addl_offset = if offset > file_offset {
+            offset - file_offset
+        } else {
+            0
+        };
+
+        // ok, need to read this chunk; how much?
+        let left_in_buf = data.len() - buf_offset;
+        let to_read = min(left_in_buf, chunk.len as usize - addl_offset);
+
+        let start = buf_offset;
+        let finish = start + to_read;
+        file_offset += addl_offset;
+
+        // how many did we actually read?
+        let n = oci.fill_from_chunk(chunk.blob, addl_offset as u64, &mut data[start..finish])?;
+        file_offset += n;
+        buf_offset += n;
+    }
+
+    // discard any extra if we hit EOF
+    Ok(buf_offset)
+}
samples/rust/puzzle/oci.rs +26 -6
index becb2b868450..5aa60ded8419 100644
@@ -1,19 +1,21 @@
-use crate::puzzle::error::Result;
+use crate::puzzle::error::{Result, WireFormatError};
+use crate::puzzle::types as format;
 use crate::puzzle::types::{Digest, MetadataBlob, Rootfs};
 use kernel::c_str;
 use kernel::file;
 use kernel::file::RegularFile;
 use kernel::mount::Vfsmount;
-use kernel::pr_info;
+use kernel::pr_debug;
+use kernel::prelude::ENOTSUPP;
 use kernel::str::{CStr, CString};
-use kernel::sync::Arc;
 
+#[derive(Debug)]
 pub(crate) struct Image {
-    vfs_mount: Arc<Vfsmount>,
+    pub(crate) vfs_mount: Vfsmount,
 }
 
 impl Image {
-    pub(crate) fn open(vfsmount: Arc<Vfsmount>) -> Result<Self> {
+    pub(crate) fn open(vfsmount: Vfsmount) -> Result<Self> {
         Ok(Image {
             vfs_mount: vfsmount,
         })
@@ -26,7 +28,7 @@ pub(crate) fn blob_path_relative(&self) -> &CStr {
     fn open_raw_blob(&self, digest: &Digest) -> Result<RegularFile> {
         let filename =
             CString::try_from_fmt(format_args!("{}/{digest}", self.blob_path_relative()))?;
-        pr_info!("trying to open {:?}\n", &filename);
+        pr_debug!("trying to open {:?}\n", &*filename);
 
         let file = RegularFile::from_path_in_root_mnt(
             &self.vfs_mount,
@@ -48,4 +50,22 @@ pub(crate) fn open_rootfs_blob(&self, path: &CStr) -> Result<Rootfs> {
         let rootfs = Rootfs::open(self.open_raw_blob(&digest)?)?;
         Ok(rootfs)
     }
+
+    pub(crate) fn fill_from_chunk(
+        &self,
+        chunk: format::BlobRef,
+        addl_offset: u64,
+        buf: &mut [u8],
+    ) -> Result<usize> {
+        let digest = &<Digest>::try_from(chunk)?;
+
+        let blob = if chunk.compressed {
+            return Err(WireFormatError::KernelError(ENOTSUPP));
+        } else {
+            self.open_raw_blob(digest)?
+        };
+
+        let n = blob.read_with_offset(buf, chunk.offset + addl_offset)?;
+        Ok(n)
+    }
 }
samples/rust/puzzlefs.rs +22 -32

110 lines not on this page · raw message

Signature · 1 line
2.41.0
#15 lore raw

RFC PATCH v2 10/10 rust: puzzlefs: add oci_root_dir and image_manifest filesystem parameters

In reply to Ariel Miculas

These parameters are passed when mounting puzzlefs using '-o' option of mount:
-o oci_root_dir="/path/to/oci/dir"
-o image_manifest="root_hash_of_image_manifest"

For a particular manifest in the manifests array in index.json (located in the oci_root_dir), the root hash of the image manifest is found in the digest field.

It would be nicer if we could pass the tag, but we don't support json deserialization.

Example of mount:
mount -t puzzlefs -o oci_root_dir="/home/puzzlefs_oci" -o \
image_manifest="2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b" \
none /mnt

---
 samples/rust/puzzlefs.rs | 63 ++++++++++++++++++++++++++--------------
 1 file changed, 41 insertions(+), 22 deletions(-)
samples/rust/puzzlefs.rs +41 -22
index dad7ecc76eca..4e9a8aedf0c1 100644
@@ -7,6 +7,7 @@
 use kernel::{
     c_str, file, fs,
     io_buffer::IoBufferWriter,
+    str::CString,
     sync::{Arc, ArcBorrow},
 };
 
@@ -31,27 +32,29 @@ struct PuzzlefsInfo {
     puzzlefs: Arc<PuzzleFS>,
 }
 
+#[derive(Default)]
+struct PuzzleFsParams {
+    oci_root_dir: Option<CString>,
+    image_manifest: Option<CString>,
+}
+
 #[vtable]
 impl fs::Context<Self> for PuzzleFsModule {
-    type Data = ();
-
-    kernel::define_fs_params! {(),
-        {flag, "flag", |_, v| { pr_info!("flag passed-in: {v}\n"); Ok(()) } },
-        {flag_no, "flagno", |_, v| { pr_info!("flagno passed-in: {v}\n"); Ok(()) } },
-        {bool, "bool", |_, v| { pr_info!("bool passed-in: {v}\n"); Ok(()) } },
-        {u32, "u32", |_, v| { pr_info!("u32 passed-in: {v}\n"); Ok(()) } },
-        {u32oct, "u32oct", |_, v| { pr_info!("u32oct passed-in: {v}\n"); Ok(()) } },
-        {u32hex, "u32hex", |_, v| { pr_info!("u32hex passed-in: {v}\n"); Ok(()) } },
-        {s32, "s32", |_, v| { pr_info!("s32 passed-in: {v}\n"); Ok(()) } },
-        {u64, "u64", |_, v| { pr_info!("u64 passed-in: {v}\n"); Ok(()) } },
-        {string, "string", |_, v| { pr_info!("string passed-in: {v}\n"); Ok(()) } },
-        {enum, "enum", [("first", 10), ("second", 20)], |_, v| {
-            pr_info!("enum passed-in: {v}\n"); Ok(()) }
-        },
+    type Data = Box<PuzzleFsParams>;
+
+    kernel::define_fs_params! {Box<PuzzleFsParams>,
+        {string, "oci_root_dir", |s, v| {
+                                      s.oci_root_dir = Some(CString::try_from_fmt(format_args!("{v}"))?);
+                                      Ok(())
+                                  }},
+        {string, "image_manifest", |s, v| {
+                                      s.image_manifest = Some(CString::try_from_fmt(format_args!("{v}"))?);
+                                      Ok(())
+                                  }},
     }
 
-    fn try_new() -> Result {
-        Ok(())
+    fn try_new() -> Result<Self::Data> {
+        Ok(Box::try_new(PuzzleFsParams::default())?)
     }
 }
 
@@ -136,11 +139,27 @@ impl fs::Type for PuzzleFsModule {
     const FLAGS: i32 = fs::flags::USERNS_MOUNT;
     const DCACHE_BASED: bool = true;
 
-    fn fill_super(_data: (), sb: fs::NewSuperBlock<'_, Self>) -> Result<&fs::SuperBlock<Self>> {
-        let puzzlefs = PuzzleFS::open(
-            c_str!("/home/puzzlefs_oci"),
-            c_str!("2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b"),
-        );
+    fn fill_super(
+        data: Box<PuzzleFsParams>,
+        sb: fs::NewSuperBlock<'_, Self>,
+    ) -> Result<&fs::SuperBlock<Self>> {
+        let oci_root_dir = match data.oci_root_dir {
+            Some(val) => val,
+            None => {
+                pr_err!("missing oci_root_dir parameter!\n");
+                return Err(ENOTSUPP);
+            }
+        };
+
+        let image_manifest = match data.image_manifest {
+            Some(val) => val,
+            None => {
+                pr_err!("missing image_manifest parameter!\n");
+                return Err(ENOTSUPP);
+            }
+        };
+
+        let puzzlefs = PuzzleFS::open(&oci_root_dir, &image_manifest);
 
         if let Err(ref e) = puzzlefs {
             pr_info!("error opening puzzlefs {e}\n");
Signature · 1 line
2.41.0
reviewer
#16 lore raw

Re: [RFC PATCH v2 10/10] rust: puzzlefs: add oci_root_dir and image_manifest filesystem parameters

Quoted 101 lines · Ariel Miculas

These parameters are passed when mounting puzzlefs using '-o' option of mount:

(quoted diff) +0 -2
-o oci_root_dir="/path/to/oci/dir"
-o image_manifest="root_hash_of_image_manifest"

For a particular manifest in the manifests array in index.json (located in the oci_root_dir), the root hash of the image manifest is found in the digest field.

It would be nicer if we could pass the tag, but we don't support json deserialization.

Example of mount:
mount -t puzzlefs -o oci_root_dir="/home/puzzlefs_oci" -o \
image_manifest="2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b" \
none /mnt

---
 samples/rust/puzzlefs.rs | 63 ++++++++++++++++++++++++++--------------
 1 file changed, 41 insertions(+), 22 deletions(-)
samples/rust/puzzlefs.rs +30 -22
index dad7ecc76eca..4e9a8aedf0c1 100644
@@ -7,6 +7,7 @@
 use kernel::{
     c_str, file, fs,
     io_buffer::IoBufferWriter,
+    str::CString,
     sync::{Arc, ArcBorrow},
 };
 
@@ -31,27 +32,29 @@ struct PuzzlefsInfo {
     puzzlefs: Arc<PuzzleFS>,
 }
 
+#[derive(Default)]
+struct PuzzleFsParams {
+    oci_root_dir: Option<CString>,
+    image_manifest: Option<CString>,
+}
+
 #[vtable]
 impl fs::Context<Self> for PuzzleFsModule {
-    type Data = ();
-
-    kernel::define_fs_params! {(),
-        {flag, "flag", |_, v| { pr_info!("flag passed-in: {v}\n"); Ok(()) } },
-        {flag_no, "flagno", |_, v| { pr_info!("flagno passed-in: {v}\n"); Ok(()) } },
-        {bool, "bool", |_, v| { pr_info!("bool passed-in: {v}\n"); Ok(()) } },
-        {u32, "u32", |_, v| { pr_info!("u32 passed-in: {v}\n"); Ok(()) } },
-        {u32oct, "u32oct", |_, v| { pr_info!("u32oct passed-in: {v}\n"); Ok(()) } },
-        {u32hex, "u32hex", |_, v| { pr_info!("u32hex passed-in: {v}\n"); Ok(()) } },
-        {s32, "s32", |_, v| { pr_info!("s32 passed-in: {v}\n"); Ok(()) } },
-        {u64, "u64", |_, v| { pr_info!("u64 passed-in: {v}\n"); Ok(()) } },
-        {string, "string", |_, v| { pr_info!("string passed-in: {v}\n"); Ok(()) } },
-        {enum, "enum", [("first", 10), ("second", 20)], |_, v| {
-            pr_info!("enum passed-in: {v}\n"); Ok(()) }
-        },
+    type Data = Box<PuzzleFsParams>;
+
+    kernel::define_fs_params! {Box<PuzzleFsParams>,
+        {string, "oci_root_dir", |s, v| {
+                                      s.oci_root_dir = Some(CString::try_from_fmt(format_args!("{v}"))?);
+                                      Ok(())
+                                  }},
+        {string, "image_manifest", |s, v| {
+                                      s.image_manifest = Some(CString::try_from_fmt(format_args!("{v}"))?);
+                                      Ok(())
+                                  }},
     }
 
-    fn try_new() -> Result {
-        Ok(())
+    fn try_new() -> Result<Self::Data> {
+        Ok(Box::try_new(PuzzleFsParams::default())?)
     }
 }
 
@@ -136,11 +139,27 @@ impl fs::Type for PuzzleFsModule {
     const FLAGS: i32 = fs::flags::USERNS_MOUNT;
     const DCACHE_BASED: bool = true;
 
-    fn fill_super(_data: (), sb: fs::NewSuperBlock<'_, Self>) -> Result<&fs::SuperBlock<Self>> {
-        let puzzlefs = PuzzleFS::open(
-            c_str!("/home/puzzlefs_oci"),
-            c_str!("2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b"),
-        );
+    fn fill_super(
+        data: Box<PuzzleFsParams>,
+        sb: fs::NewSuperBlock<'_, Self>,
+    ) -> Result<&fs::SuperBlock<Self>> {
+        let oci_root_dir = match data.oci_root_dir {
+            Some(val) => val,
+            None => {
+                pr_err!("missing oci_root_dir parameter!\n");
+                return Err(ENOTSUPP);
+            }

samples/rust/puzzlefs.rs impl fs::Type for PuzzleFsModule { · line 161

Quoted 10 lines
samples/rust/puzzlefs.rs +10 -0
+        };
+
+        let image_manifest = match data.image_manifest {
+            Some(val) => val,
+            None => {
+                pr_err!("missing image_manifest parameter!\n");
+                return Err(ENOTSUPP);
+            }
+        };
+

The guard syntax (available since 1.65) can make these kinds of match statements cleaner:

    let Some(oci_root_dir) = data.oci_root_dir else {
        pr_err!("missing oci_root_dir parameter!\n");
        return Err(ENOTSUPP);
    }

    let Some(image_manifest) ...
Quoted 8 lines
samples/rust/puzzlefs.rs +1 -1
+        let puzzlefs = PuzzleFS::open(&oci_root_dir, &image_manifest);
 
         if let Err(ref e) = puzzlefs {
             pr_info!("error opening puzzlefs {e}\n");
--

2.41.0

#17 lore raw

Re: [RFC PATCH v2 10/10] rust: puzzlefs: add oci_root_dir and image_manifest filesystem parameters

Quoted 108 lines · Trevor Gross
Quoted 106 lines · Ariel Miculas

These parameters are passed when mounting puzzlefs using '-o' option of mount:

(quoted diff) +0 -2
-o oci_root_dir="/path/to/oci/dir"
-o image_manifest="root_hash_of_image_manifest"

For a particular manifest in the manifests array in index.json (located in the oci_root_dir), the root hash of the image manifest is found in the digest field.

It would be nicer if we could pass the tag, but we don't support json deserialization.

Example of mount:
mount -t puzzlefs -o oci_root_dir="/home/puzzlefs_oci" -o \
image_manifest="2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b" \
none /mnt

---
 samples/rust/puzzlefs.rs | 63 ++++++++++++++++++++++++++--------------
 1 file changed, 41 insertions(+), 22 deletions(-)
samples/rust/puzzlefs.rs +35 -22
index dad7ecc76eca..4e9a8aedf0c1 100644
@@ -7,6 +7,7 @@
 use kernel::{
     c_str, file, fs,
     io_buffer::IoBufferWriter,
+    str::CString,
     sync::{Arc, ArcBorrow},
 };
 
@@ -31,27 +32,29 @@ struct PuzzlefsInfo {
     puzzlefs: Arc<PuzzleFS>,
 }
 
+#[derive(Default)]
+struct PuzzleFsParams {
+    oci_root_dir: Option<CString>,
+    image_manifest: Option<CString>,
+}
+
 #[vtable]
 impl fs::Context<Self> for PuzzleFsModule {
-    type Data = ();
-
-    kernel::define_fs_params! {(),
-        {flag, "flag", |_, v| { pr_info!("flag passed-in: {v}\n"); Ok(()) } },
-        {flag_no, "flagno", |_, v| { pr_info!("flagno passed-in: {v}\n"); Ok(()) } },
-        {bool, "bool", |_, v| { pr_info!("bool passed-in: {v}\n"); Ok(()) } },
-        {u32, "u32", |_, v| { pr_info!("u32 passed-in: {v}\n"); Ok(()) } },
-        {u32oct, "u32oct", |_, v| { pr_info!("u32oct passed-in: {v}\n"); Ok(()) } },
-        {u32hex, "u32hex", |_, v| { pr_info!("u32hex passed-in: {v}\n"); Ok(()) } },
-        {s32, "s32", |_, v| { pr_info!("s32 passed-in: {v}\n"); Ok(()) } },
-        {u64, "u64", |_, v| { pr_info!("u64 passed-in: {v}\n"); Ok(()) } },
-        {string, "string", |_, v| { pr_info!("string passed-in: {v}\n"); Ok(()) } },
-        {enum, "enum", [("first", 10), ("second", 20)], |_, v| {
-            pr_info!("enum passed-in: {v}\n"); Ok(()) }
-        },
+    type Data = Box<PuzzleFsParams>;
+
+    kernel::define_fs_params! {Box<PuzzleFsParams>,
+        {string, "oci_root_dir", |s, v| {
+                                      s.oci_root_dir = Some(CString::try_from_fmt(format_args!("{v}"))?);
+                                      Ok(())
+                                  }},
+        {string, "image_manifest", |s, v| {
+                                      s.image_manifest = Some(CString::try_from_fmt(format_args!("{v}"))?);
+                                      Ok(())
+                                  }},
     }
 
-    fn try_new() -> Result {
-        Ok(())
+    fn try_new() -> Result<Self::Data> {
+        Ok(Box::try_new(PuzzleFsParams::default())?)
     }
 }
 
@@ -136,11 +139,27 @@ impl fs::Type for PuzzleFsModule {
     const FLAGS: i32 = fs::flags::USERNS_MOUNT;
     const DCACHE_BASED: bool = true;
 
-    fn fill_super(_data: (), sb: fs::NewSuperBlock<'_, Self>) -> Result<&fs::SuperBlock<Self>> {
-        let puzzlefs = PuzzleFS::open(
-            c_str!("/home/puzzlefs_oci"),
-            c_str!("2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b"),
-        );
+    fn fill_super(
+        data: Box<PuzzleFsParams>,
+        sb: fs::NewSuperBlock<'_, Self>,
+    ) -> Result<&fs::SuperBlock<Self>> {
+        let oci_root_dir = match data.oci_root_dir {
+            Some(val) => val,
+            None => {
+                pr_err!("missing oci_root_dir parameter!\n");
+                return Err(ENOTSUPP);
+            }
+        };
+
+        let image_manifest = match data.image_manifest {
+            Some(val) => val,
+            None => {
Quoted 8 lines

samples/rust/puzzlefs.rs impl fs::Type for PuzzleFsModule { · line 161

Quoted 5 lines
samples/rust/puzzlefs.rs +5 -0
+                pr_err!("missing image_manifest parameter!\n");
+                return Err(ENOTSUPP);
+            }
+        };
+

The guard syntax (available since 1.65) can make these kinds of match statements cleaner:

This is unstable though.

We try to stay away from unstable features unless we really need them, which I feel is not the case here.

Quoted 15 lines
    let Some(oci_root_dir) = data.oci_root_dir else {
        pr_err!("missing oci_root_dir parameter!\n");
        return Err(ENOTSUPP);
    }

    let Some(image_manifest) ...
Quoted 8 lines
samples/rust/puzzlefs.rs +1 -1
+        let puzzlefs = PuzzleFS::open(&oci_root_dir, &image_manifest);
 
         if let Err(ref e) = puzzlefs {
             pr_info!("error opening puzzlefs {e}\n");
--

2.41.0

reviewer
#18 lore raw

Re: [RFC PATCH v2 10/10] rust: puzzlefs: add oci_root_dir and image_manifest filesystem parameters

Quoted 12 lines · Wedson Almeida Filho
Quoted 1 line · Trevor Gross

[...]

Quoted 2 lines

The guard syntax (available since 1.65) can make these kinds of match statements cleaner:

This is unstable though.

We try to stay away from unstable features unless we really need them, which I feel is not the case here.

Let/else has been stable since 1.65 and is in pretty heavy use, the kernel is on 1.68.2 correct? Could you be thinking of let chains (multiple matches joined with &&/||)?

Quoted 5 lines
    let Some(oci_root_dir) = data.oci_root_dir else {
        pr_err!("missing oci_root_dir parameter!\n");
        return Err(ENOTSUPP);
    }

[...]

#19 lore raw

Re: [RFC PATCH v2 10/10] rust: puzzlefs: add oci_root_dir and image_manifest filesystem parameters

Quoted 11 lines · Trevor Gross
Quoted 9 lines · Wedson Almeida Filho
Quoted 1 line · Trevor Gross

[...]

Quoted 2 lines

The guard syntax (available since 1.65) can make these kinds of match statements cleaner:

This is unstable though.

Quoted 7 lines
Quoted 3 lines

We try to stay away from unstable features unless we really need them, which I feel is not the case here.

Let/else has been stable since 1.65 and is in pretty heavy use, the kernel is on 1.68.2 correct? Could you be thinking of let chains (multiple matches joined with &&/||)?

Oh, my bad. I got it mixed with "if let guard" (issue 51114).

And yeah, we're on 1.68.2, so this code can benefit from this simplification.

Thanks!

Quoted 6 lines
Quoted 6 lines
    let Some(oci_root_dir) = data.oci_root_dir else {
        pr_err!("missing oci_root_dir parameter!\n");
        return Err(ENOTSUPP);
    }

[...]