Rust PuzzleFS filesystem driver
Subsystems (9)
cover Rust PuzzleFS filesystem driver
Hi all!
This is a proof of concept driver written for the PuzzleFS next-generation container filesystem [1]. I've included a short abstract about puzzlefs further below. This driver is based on the rust-next branch, on top of which I've backported the filesystem abstractions from Wedson Almeida Filho [2][3] and Miguel Ojeda's third-party crates
support: proc-macro2, quote, syn, serde and serde_derive [4]. I've added the additional third-party crates serde_cbor[5] and hex [6]. Then I've adapted the user space puzzlefs code [1] so that the puzzlefs kernel module could present the directory hierarchy and implement the basic read functionality.
For some additional context, puzzlefs was started by Tycho Andersen and it's the successor of atomfs. This FOSDEM presentation from 2019 [12] covers the rationale for a new oci image format and presents a higher level overview of our goals with puzzlefs.
I've split the rest of the cover letter in following sections (using a markdown style):
* Example: it describes a practical example of what was achieved
* Limitations: it presents the existing limitations of this POC
* Upstreaming steps: it describes the steps needed for upstreaming this driver
* Setup: it shows how to setup the necessary environment for testing the puzzlefs driver
* Puzzlefs abstract: it provides a short overview of puzzlefs
# Example
An example is provided below:
```
~ # cat /proc/filesystems | grep puzzlefs
nodev puzzlefs
~ # mount -t puzzlefs -o oci_root_dir="/home/puzzlefs_oci" -o image_manifest="2d
6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b" none /mnt
~ # ls -lR /mnt/
/mnt/:
total 0
drwxr-xr-x 2 0 0 0 Jun 8 12:26 dir-1
drwxr-xr-x 2 0 0 0 Jun 8 12:26 dir-2
drwxr-xr-x 2 0 0 0 Jun 8 12:26 dir-3
drwxr-xr-x 2 0 0 0 Jun 8 12:26 dir-4
-rw-r--r-- 1 0 0 0 Jun 8 12:26 file1
-rw-r--r-- 1 0 0 0 Jun 8 12:26 file2
/mnt/dir-1:
total 0
/mnt/dir-2:
total 0
/mnt/dir-3:
total 0
/mnt/dir-4:
total 0
~ # cat /mnt/file2
ana are mere bla bla bla
~ # wc /mnt/file1
202 202 5454 /mnt/file1
```In this example, /home/puzzlefs_oci is the puzzlefs oci directory:
``` ~ # ls -lR /home/puzzlefs_oci/ /home/puzzlefs_oci/: total 8 drwxr-xr-x 3 1000 1000 0 Jun 8 14:33 blobs -rw-r--r-- 1 1000 1000 266 Jun 8 14:33 index.json -rw-r--r-- 1 1000 1000 37 Jun 8 14:33 oci-layout /home/puzzlefs_oci/blobs: total 0 drwxr-xr-x 2 1000 1000 0 Jun 8 14:33 sha256 /home/puzzlefs_oci/blobs/sha256: total 16 -rw------- 1 1000 1000 89 Jun 8 14:33 2d6602d678140540dc7e96de652a76a8b16eb -rw------- 1 1000 1000 925 Jun 8 14:33 4df03518eea406343dbb55046720f6a478881 -rw------- 1 1000 1000 5479 Jun 8 14:33 d86a87b19bd9a2fec0d31687c1d669cdb59eb ```
2d6602d678140540dc7e96de652a76a8b16eb is the puzzlefs image manifest hash for the first_try tag:
```
$ cat /tmp/oci-simple/index.json | jq .
{
"schemaVersion": -1,
"manifests": [
{
"digest": "sha256:2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b",
"size": 89,
"media_type": "application/vnd.puzzlefs.image.rootfs.v1",
"annotations": {
"org.opencontainers.image.ref.name": "first_try"
}
}
],
"annotations": {}
}
```I will describe how to build a puzzlefs image in the Setup section, at step 5.
# Limitations
One limitation is that the puzzlefs driver doesn't implement any lookup functionality and instead it inserts every directory entry into the dcache during init (see the DCACHE_BASED constant). This is similar to how the sample rust_fs driver works, but the goal is to implement proper lookup functions. However, more filesystem abstractions need to be implemented before this can be achieved.
# Upstreaming steps
Before the puzzlefs driver can be upstreamed, the following need to be merged:
* Wedson's filesystem abstractions [3]
* the necessary third-party crates [4] (with the preliminary discussion about whether this is desirable)
# Setup
My setup is based on Wedson's tutorial [8]. Next, I will describe the necessary steps to build an initrd and run a custom kernel under qemu.
1. Get the linux rust-next branch [9] and apply this patchset
2. Follow the rust quickstart guide [10]
3. configure and build the kernel
``` $ make LLVM=1 allnoconfig qemu-busybox-min.config rust.config $ make LLVM=1 -j$(nproc) ```
4. setup busybox
``` git clone git://git.busybox.net/busybox cd busybox make menuconfig # enable 'Build static binary' config make make install ```
This will create the _install directory with the rootfs inside it.
5. create a home directory in the rootfs and copy a puzzlefs oci directory in home/puzzlefs_oci
To create a puzzlefs oci directory:
* download this custom puzzlefs repository [11] (it's custom because we want to build an image without verity data)
* run make release
* create a simple filesystem structure with a few directories and files (I've created one at ../test-puzzlefs/simple_rootfs)
* build a puzzlefs oci image at ~/work/busybox/_install/home/puzzlefs_oci (replace this path with your busybox path) with the tag first_try:
```
$ target/release/puzzlefs build --omit-verity \
../test-puzzlefs/simple_rootfs ~/work/busybox/_install/home/puzzlefs_oci \
first_try
```
* get first_try's image manifest from index.json (inside `puzzlefs_oci`)
```
$ cat index.json | jq . | grep digest
"digest": "sha256:2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b",
```6. add the following 'init' script in the busybox rootfs (rootfs path defaults to `./_install'):
``` #!/bin/sh mount -t devtmpfs none /dev mkdir -p /proc mount -t proc none /proc ifconfig lo up udhcpc -i eth0 mkdir /mnt mount -t puzzlefs -o oci_root_dir="/home/puzzlefs_oci" -o \ image_manifest="2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b" \ none /mnt setsid sh -c 'exec sh -l </dev/ttyS0 >/dev/ttyS0 2>&1' ```
Make sure to replace the image_manifest with your own digest. This init script will be passed to rdinit in the kernel command line.
7. generate the initramfs
Assuming busybox is in ~/work/busybox:
``` cd ~/work/busybox/_install && find . | cpio -H newc -o | gzip > ../ramdisk.img ```
This will generate a compressed ramdisk image in~/work/busybox/ramdisk.img.
8. run with qemu (this assumes the linux tree is at '../linux' and busybox is at '../busybox'):
```
qemu-system-x86_64 \
-accel kvm \
-cpu host \
-m 4G \
-initrd ../busybox/ramdisk.img \
-kernel ../linux/arch/x86/boot/bzImage \
-nographic \
-append 'console=ttyS0 nokaslr debug rdinit=/init' \
-nic user,model=rtl8139 \
-no-reboot
```9. Check whether puzzlefs has been successfully mounted
``` ~ # mount | grep puzzlefs none on /mnt type puzzlefs (rw,relatime) ~ # ls /mnt/ dir-1 dir-2 dir-3 dir-4 file1 file2 ```
# Puzzlefs abstract
Puzzlefs [1] is a container filesystem designed to address the limitations of the existing OCI format. The main goals of the project are reduced duplication, reproducible image builds, direct mounting support and memory safety guarantees, some inspired by the OCIv2 design document [7].
Reduced duplication is achieved using the content defined chunking algorithm FastCDC. This implementation allows chunks to be shared among layers. Building a new layer starting from an existing one allows reusing most of the chunks.
Another goal of the project is reproducible image builds, which is achieved by defining a canonical representation of the image format.
Direct mounting support is a key feature of puzzlefs and, together with fs-verity, it provides data integrity. Currently, puzzlefs is implemented as a userspace filesystem (FUSE). A read-only kernel filesystem driver is underway.
Lastly, memory safety is critical to puzzlefs, leading to the decision to implement it in Rust. Another goal is to share the same code between user space and kernel space in order to provide one secure implementation.
[1] https://github.com/anuvu/puzzlefs
[2] https://github.com/wedsonaf/linux/tree/fs
[3] https://github.com/Rust-for-Linux/linux/issues/1004
[4] https://github.com/Rust-for-Linux/linux/pull/1007
[5] https://docs.rs/serde_cbor/latest/serde_cbor/
[6] https://docs.rs/hex/0.4.3/hex/
[7] https://hackmd.io/@cyphar/ociv2-brainstorm
[8] https://www.youtube.com/watch?v=tPs1uRqOnlk
[9] https://github.com/Rust-for-Linux/linux/tree/rust-next
[10] https://docs.kernel.org/rust/quick-start.html
[11] https://github.com/ariel-miculas/puzzlefs/tree/support-no-verity-data
[12] https://archive.fosdem.org/2019/schedule/event/containers_atomfs/
Ariel Miculas (58):
rust: kernel: add libraries required by the filesystem abstractions
rust: kernel: backport the delay module from the rust branch
rust: kernel: add container_of macro
rust: kernel: add offset_of macro
drop: Add crate::pr_warn declaration
rust: kernel: rename from_kernel_errno to from_errno
rust: kernel: Rename from_pointer to from_foreing and into_pointer to
into_foreign
rust: kernel: add count_paren_items macro, needed by define_fs_params
macro
rust: helpers: add missing rust helper 'alloc_pages'
kernel: configs: add qemu-busybox-min.config
rust: kernel: format the rust code
samples: puzzlefs: add initial puzzlefs sample, copied from rust_fs.rs
kernel: configs: enable rust samples in rust.config
Add SAMPLE_RUST_SERDE in rust.config
rust: kernel: fix compile errors after rebase to rust-next
rust: serde_cbor: import crate
rust: serde_cbor: add SPDX License Identifiers
rust: serde_cbor: add no_fp_fmt_parse support
rust: Kbuild: enable serde_cbor
samples: rust: add cbor serialize/deserialize example
rust: serde_cbor: add support for serde_cbor's from_slice method by
using a custom alloc_kernel feature
rust: serde: add support for deserializing Vec with kernel_alloc
feature
rust: file: Replace UnsafeCell with Opaque for File
rust: kernel: implement fmt::Debug for CString
samples: puzzlefs: rename RustFs to PuzzleFs
samples: puzzlefs: add basic deserializing support for the puzzlefs
metadata
rust: file: present the filesystem context to the open function
rust: kernel: add an abstraction over vfsmount to allow cloning a new
private mount
rust: file: add from_path, from_path_in_root_mnt and read_with_offset
methods to File
samples: puzzlefs: pass the Vfsmount structure from open to read and
return the contents of the data file inside /home/puzzlefs_oci
rust: file: move from_path, from_path_in_root_mnt and read_with_offset
methods to a RegularFile newtype
rust: file: ensure RegularFile can only create regular files
rust: file: add get_pos method to RegularFile
rust: file: add methods read_to_end, get_file_size and update_pos to
RegularFile
rust: file: define a minimal Read trait and implement it for
RegularFile
samples: puzzlefs: add cbor_get_array_size method
samples: puzzlefs: add KernelError to WireFormatError and implement
From conversion
samples: puzzlefs: implement new for MetadataBlob
samples: puzzlefs: build puzzlefs into the kernel, thus avoiding the
need to export rust symbols
rust: alloc: add try_clone for Vec<T>
rust: alloc: add from_iter_fallible for Vec<T>
samples: puzzlefs: implement to_errno and from_errno for
WireFormatError
samples: puzzlefs: add TryReserveError (and from conversion) to
WireFormatError
samples: puzzlefs: add higher level inode related functionality
samples: puzzlefs: populate the directory entries with the inodes from
the puzzlefs metadata file
rust: hex: import crate
rust: hex: add SPDX license identifiers
rust: Kbuild: enable hex
rust: hex: implement FromHex trait and hex::decode using a custom
kernel_alloc feature
rust: hex: add encode_hex_iter and encode_hex_upper_iter methods
rust: puzzlefs: add HexError to WireFormatError and implement the From
conversion
rust: puzzlefs: display the error value for
WireFormatError::KernelError
samples: puzzlefs: add Rootfs and Digest structs to types.rs
samples: puzzlefs: implement the conversion from WireFormatError to
kernel::error::Error
rust: puzzlefs: read the puzzlefs image manifest instead of an
individual metadata layer
rust: puzzlefs: rename PuzzleFs to PuzzleFsModule to avoid confusion
with the PuzzleFS struct
rust: puzzlefs: add support for reading files
rust: puzzlefs: add oci_root_dir and image_manifest filesystem
parameters
Miguel Ojeda (15):
rust: proc-macro2: import crate
rust: proc-macro2: add SPDX License Identifiers
rust: proc-macro2: remove unicode_ident dependency
rust: quote: import crate
rust: quote: add SPDX License Identifiers
rust: syn: import crate
rust: syn: add SPDX License Identifiers
rust: syn: remove unicode-ident dependency
rust: serde: import crate
rust: serde: add no_fp_fmt_parse support
rust: serde: add SPDX License Identifiers
rust: serde_derive: import crate
rust: serde_derive: add SPDX License Identifiers
rust: Kbuild: enable proc-macro2, quote, syn, serde and
serde_derive
rust: test serde support
Wedson Almeida Filho (7):
rust: add definitions for ref-counted inodes and dentries
rust: add ability to register a file system
rust: define fs context
rust: add support for file system parameters
rust: allow fs driver to initialise new superblocks
rust: add module_fs macro
WIP: rust: allow fs to be populated
Makefile | 14 +- arch/x86/configs/qemu-busybox-min.config | 11 + kernel/configs/qemu-busybox-min.config | 56 + kernel/configs/rust.config | 11 + rust/.gitignore | 1 + rust/Makefile | 232 +- rust/alloc/vec/mod.rs | 48 + rust/bindings/bindings_helper.h | 14 + rust/bindings/lib.rs | 5 + rust/helpers.c | 76 + rust/hex/error.rs | 78 + rust/hex/lib.rs | 506 +++ rust/hex/serde.rs | 104 + rust/kernel/cred.rs | 46 + rust/kernel/delay.rs | 104 + rust/kernel/driver.rs | 28 + rust/kernel/error.rs | 52 +- rust/kernel/file.rs | 1117 ++++++ rust/kernel/fs.rs | 1478 ++++++++ rust/kernel/fs/param.rs | 558 +++ rust/kernel/io_buffer.rs | 153 + rust/kernel/iov_iter.rs | 81 + rust/kernel/lib.rs | 83 + rust/kernel/mm.rs | 149 + rust/kernel/mount.rs | 66 + rust/kernel/pages.rs | 144 + rust/kernel/str.rs | 6 + rust/kernel/test_serde.rs | 26 + rust/kernel/test_serde/de.rs | 439 +++ rust/kernel/test_serde/error.rs | 73 + rust/kernel/test_serde/ser.rs | 466 +++ rust/kernel/user_ptr.rs | 175 + rust/proc-macro2/detection.rs | 77 + rust/proc-macro2/fallback.rs | 1004 ++++++ rust/proc-macro2/lib.rs | 1341 ++++++++ rust/proc-macro2/marker.rs | 20 + rust/proc-macro2/parse.rs | 874 +++++ rust/proc-macro2/rcvec.rs | 144 + rust/proc-macro2/wrapper.rs | 996 ++++++ rust/quote/ext.rs | 112 + rust/quote/format.rs | 170 + rust/quote/ident_fragment.rs | 88 + rust/quote/lib.rs | 1436 ++++++++ rust/quote/runtime.rs | 440 +++ rust/quote/spanned.rs | 45 + rust/quote/to_tokens.rs | 211 ++ rust/serde/de/format.rs | 32 + rust/serde/de/ignored_any.rs | 246 ++ rust/serde/de/impls.rs | 2755 +++++++++++++++ rust/serde/de/mod.rs | 2313 +++++++++++++ rust/serde/de/seed.rs | 21 + rust/serde/de/utf8.rs | 48 + rust/serde/de/value.rs | 1718 ++++++++++ rust/serde/integer128.rs | 84 + rust/serde/lib.rs | 351 ++ rust/serde/macros.rs | 238 ++ rust/serde/private/de.rs | 2997 ++++++++++++++++ rust/serde/private/doc.rs | 161 + rust/serde/private/mod.rs | 52 + rust/serde/private/ser.rs | 1316 +++++++ rust/serde/private/size_hint.rs | 23 + rust/serde/ser/fmt.rs | 180 + rust/serde/ser/impls.rs | 987 ++++++ rust/serde/ser/impossible.rs | 218 ++ rust/serde/ser/mod.rs | 1992 +++++++++++ rust/serde/std_error.rs | 50 + rust/serde_cbor/de.rs | 1370 ++++++++ rust/serde_cbor/error.rs | 320 ++ rust/serde_cbor/lib.rs | 371 ++ rust/serde_cbor/read.rs | 647 ++++ rust/serde_cbor/ser.rs | 748 ++++ rust/serde_cbor/tags.rs | 224 ++ rust/serde_cbor/value/de.rs | 168 + rust/serde_cbor/value/mod.rs | 158 + rust/serde_cbor/value/ser.rs | 447 +++ rust/serde_cbor/write.rs | 177 + rust/serde_derive/bound.rs | 408 +++ rust/serde_derive/de.rs | 3148 +++++++++++++++++ rust/serde_derive/dummy.rs | 46 + rust/serde_derive/fragment.rs | 76 + rust/serde_derive/internals/ast.rs | 204 ++ rust/serde_derive/internals/attr.rs | 1908 +++++++++++ rust/serde_derive/internals/case.rs | 199 ++ rust/serde_derive/internals/check.rs | 445 +++ rust/serde_derive/internals/ctxt.rs | 64 + rust/serde_derive/internals/mod.rs | 30 + rust/serde_derive/internals/receiver.rs | 287 ++ rust/serde_derive/internals/respan.rs | 18 + rust/serde_derive/internals/symbol.rs | 71 + rust/serde_derive/lib.rs | 112 + rust/serde_derive/pretend.rs | 203 ++ rust/serde_derive/ser.rs | 1342 ++++++++ rust/serde_derive/this.rs | 34 + rust/serde_derive/try.rs | 26 + rust/syn/attr.rs | 664 ++++ rust/syn/await.rs | 4 + rust/syn/bigint.rs | 68 + rust/syn/buffer.rs | 400 +++ rust/syn/custom_keyword.rs | 255 ++ rust/syn/custom_punctuation.rs | 302 ++ rust/syn/data.rs | 495 +++ rust/syn/derive.rs | 276 ++ rust/syn/discouraged.rs | 196 ++ rust/syn/error.rs | 430 +++ rust/syn/export.rs | 41 + rust/syn/expr.rs | 3560 +++++++++++++++++++ rust/syn/ext.rs | 141 + rust/syn/file.rs | 127 + rust/syn/gen/clone.rs | 2243 ++++++++++++ rust/syn/gen/debug.rs | 3044 +++++++++++++++++ rust/syn/gen/eq.rs | 2197 ++++++++++++ rust/syn/gen/fold.rs | 3343 ++++++++++++++++++ rust/syn/gen/hash.rs | 2871 ++++++++++++++++ rust/syn/gen/visit.rs | 3788 +++++++++++++++++++++ rust/syn/gen/visit_mut.rs | 3788 +++++++++++++++++++++ rust/syn/gen_helper.rs | 156 + rust/syn/generics.rs | 1339 ++++++++ rust/syn/group.rs | 284 ++ rust/syn/ident.rs | 103 + rust/syn/item.rs | 3315 ++++++++++++++++++ rust/syn/lib.rs | 985 ++++++ rust/syn/lifetime.rs | 156 + rust/syn/lit.rs | 1602 +++++++++ rust/syn/lookahead.rs | 171 + rust/syn/mac.rs | 221 ++ rust/syn/macros.rs | 179 + rust/syn/op.rs | 236 ++ rust/syn/parse.rs | 1316 +++++++ rust/syn/parse_macro_input.rs | 181 + rust/syn/parse_quote.rs | 169 + rust/syn/pat.rs | 929 +++++ rust/syn/path.rs | 856 +++++ rust/syn/print.rs | 18 + rust/syn/punctuated.rs | 1070 ++++++ rust/syn/reserved.rs | 46 + rust/syn/sealed.rs | 6 + rust/syn/span.rs | 69 + rust/syn/spanned.rs | 116 + rust/syn/stmt.rs | 351 ++ rust/syn/thread.rs | 43 + rust/syn/token.rs | 1015 ++++++ rust/syn/tt.rs | 109 + rust/syn/ty.rs | 1288 +++++++ rust/syn/verbatim.rs | 35 + rust/syn/whitespace.rs | 67 + samples/rust/Kconfig | 28 + samples/rust/Makefile | 3 + samples/rust/local_data_format/de.rs | 422 +++ samples/rust/local_data_format/error.rs | 73 + samples/rust/local_data_format/ser.rs | 443 +++ samples/rust/puzzle.rs | 4 + samples/rust/puzzle/error.rs | 91 + samples/rust/puzzle/inode.rs | 150 + samples/rust/puzzle/oci.rs | 71 + samples/rust/puzzle/types.rs | 389 +++ samples/rust/puzzle/types/cbor_helpers.rs | 50 + samples/rust/puzzlefs.rs | 220 ++ samples/rust/rust_fs.rs | 105 + samples/rust/rust_serde.rs | 125 + scripts/Makefile.build | 4 +- 160 files changed, 89204 insertions(+), 29 deletions(-) create mode 100644 arch/x86/configs/qemu-busybox-min.config create mode 100644 kernel/configs/qemu-busybox-min.config create mode 100644 rust/hex/error.rs create mode 100644 rust/hex/lib.rs create mode 100644 rust/hex/serde.rs create mode 100644 rust/kernel/cred.rs create mode 100644 rust/kernel/delay.rs create mode 100644 rust/kernel/driver.rs create mode 100644 rust/kernel/file.rs create mode 100644 rust/kernel/fs.rs create mode 100644 rust/kernel/fs/param.rs create mode 100644 rust/kernel/io_buffer.rs create mode 100644 rust/kernel/iov_iter.rs create mode 100644 rust/kernel/mm.rs create mode 100644 rust/kernel/mount.rs create mode 100644 rust/kernel/pages.rs create mode 100644 rust/kernel/test_serde.rs create mode 100644 rust/kernel/test_serde/de.rs create mode 100644 rust/kernel/test_serde/error.rs create mode 100644 rust/kernel/test_serde/ser.rs create mode 100644 rust/kernel/user_ptr.rs create mode 100644 rust/proc-macro2/detection.rs create mode 100644 rust/proc-macro2/fallback.rs create mode 100644 rust/proc-macro2/lib.rs create mode 100644 rust/proc-macro2/marker.rs create mode 100644 rust/proc-macro2/parse.rs create mode 100644 rust/proc-macro2/rcvec.rs create mode 100644 rust/proc-macro2/wrapper.rs create mode 100644 rust/quote/ext.rs create mode 100644 rust/quote/format.rs create mode 100644 rust/quote/ident_fragment.rs create mode 100644 rust/quote/lib.rs create mode 100644 rust/quote/runtime.rs create mode 100644 rust/quote/spanned.rs create mode 100644 rust/quote/to_tokens.rs create mode 100644 rust/serde/de/format.rs create mode 100644 rust/serde/de/ignored_any.rs create mode 100644 rust/serde/de/impls.rs create mode 100644 rust/serde/de/mod.rs create mode 100644 rust/serde/de/seed.rs create mode 100644 rust/serde/de/utf8.rs create mode 100644 rust/serde/de/value.rs create mode 100644 rust/serde/integer128.rs create mode 100644 rust/serde/lib.rs create mode 100644 rust/serde/macros.rs create mode 100644 rust/serde/private/de.rs create mode 100644 rust/serde/private/doc.rs create mode 100644 rust/serde/private/mod.rs create mode 100644 rust/serde/private/ser.rs create mode 100644 rust/serde/private/size_hint.rs create mode 100644 rust/serde/ser/fmt.rs create mode 100644 rust/serde/ser/impls.rs create mode 100644 rust/serde/ser/impossible.rs create mode 100644 rust/serde/ser/mod.rs create mode 100644 rust/serde/std_error.rs create mode 100644 rust/serde_cbor/de.rs create mode 100644 rust/serde_cbor/error.rs create mode 100644 rust/serde_cbor/lib.rs create mode 100644 rust/serde_cbor/read.rs create mode 100644 rust/serde_cbor/ser.rs create mode 100644 rust/serde_cbor/tags.rs create mode 100644 rust/serde_cbor/value/de.rs create mode 100644 rust/serde_cbor/value/mod.rs create mode 100644 rust/serde_cbor/value/ser.rs create mode 100644 rust/serde_cbor/write.rs create mode 100644 rust/serde_derive/bound.rs create mode 100644 rust/serde_derive/de.rs create mode 100644 rust/serde_derive/dummy.rs create mode 100644 rust/serde_derive/fragment.rs create mode 100644 rust/serde_derive/internals/ast.rs create mode 100644 rust/serde_derive/internals/attr.rs create mode 100644 rust/serde_derive/internals/case.rs create mode 100644 rust/serde_derive/internals/check.rs create mode 100644 rust/serde_derive/internals/ctxt.rs create mode 100644 rust/serde_derive/internals/mod.rs create mode 100644 rust/serde_derive/internals/receiver.rs create mode 100644 rust/serde_derive/internals/respan.rs create mode 100644 rust/serde_derive/internals/symbol.rs create mode 100644 rust/serde_derive/lib.rs create mode 100644 rust/serde_derive/pretend.rs create mode 100644 rust/serde_derive/ser.rs create mode 100644 rust/serde_derive/this.rs create mode 100644 rust/serde_derive/try.rs create mode 100644 rust/syn/attr.rs create mode 100644 rust/syn/await.rs create mode 100644 rust/syn/bigint.rs create mode 100644 rust/syn/buffer.rs create mode 100644 rust/syn/custom_keyword.rs create mode 100644 rust/syn/custom_punctuation.rs create mode 100644 rust/syn/data.rs create mode 100644 rust/syn/derive.rs create mode 100644 rust/syn/discouraged.rs create mode 100644 rust/syn/error.rs create mode 100644 rust/syn/export.rs create mode 100644 rust/syn/expr.rs create mode 100644 rust/syn/ext.rs create mode 100644 rust/syn/file.rs create mode 100644 rust/syn/gen/clone.rs create mode 100644 rust/syn/gen/debug.rs create mode 100644 rust/syn/gen/eq.rs create mode 100644 rust/syn/gen/fold.rs create mode 100644 rust/syn/gen/hash.rs create mode 100644 rust/syn/gen/visit.rs create mode 100644 rust/syn/gen/visit_mut.rs create mode 100644 rust/syn/gen_helper.rs create mode 100644 rust/syn/generics.rs create mode 100644 rust/syn/group.rs create mode 100644 rust/syn/ident.rs create mode 100644 rust/syn/item.rs create mode 100644 rust/syn/lib.rs create mode 100644 rust/syn/lifetime.rs create mode 100644 rust/syn/lit.rs create mode 100644 rust/syn/lookahead.rs create mode 100644 rust/syn/mac.rs create mode 100644 rust/syn/macros.rs create mode 100644 rust/syn/op.rs create mode 100644 rust/syn/parse.rs create mode 100644 rust/syn/parse_macro_input.rs create mode 100644 rust/syn/parse_quote.rs create mode 100644 rust/syn/pat.rs create mode 100644 rust/syn/path.rs create mode 100644 rust/syn/print.rs create mode 100644 rust/syn/punctuated.rs create mode 100644 rust/syn/reserved.rs create mode 100644 rust/syn/sealed.rs create mode 100644 rust/syn/span.rs create mode 100644 rust/syn/spanned.rs create mode 100644 rust/syn/stmt.rs create mode 100644 rust/syn/thread.rs create mode 100644 rust/syn/token.rs create mode 100644 rust/syn/tt.rs create mode 100644 rust/syn/ty.rs create mode 100644 rust/syn/verbatim.rs create mode 100644 rust/syn/whitespace.rs create mode 100644 samples/rust/local_data_format/de.rs create mode 100644 samples/rust/local_data_format/error.rs create mode 100644 samples/rust/local_data_format/ser.rs create mode 100644 samples/rust/puzzle.rs create mode 100644 samples/rust/puzzle/error.rs create mode 100644 samples/rust/puzzle/inode.rs create mode 100644 samples/rust/puzzle/oci.rs create mode 100644 samples/rust/puzzle/types.rs create mode 100644 samples/rust/puzzle/types/cbor_helpers.rs create mode 100644 samples/rust/puzzlefs.rs create mode 100644 samples/rust/rust_fs.rs create mode 100644 samples/rust/rust_serde.rs
Signature · 1 line
2.40.1
PATCH 1/80 rust: add definitions for ref-counted inodes and dentries
From: Wedson Almeida Filho <[email protected]>
This is in preparation for adding support for the implementation of file systems in Rust.
--- rust/bindings/bindings_helper.h | 1 + rust/helpers.c | 7 +++++ rust/kernel/fs.rs | 53 +++++++++++++++++++++++++++++++++ rust/kernel/lib.rs | 1 + 4 files changed, 62 insertions(+) create mode 100644 rust/kernel/fs.rs
rust/bindings/bindings_helper.h +1 -0
@@ -10,6 +10,7 @@#include <linux/refcount.h> #include <linux/wait.h> #include <linux/sched.h> +#include <linux/fs.h> /* `bindgen` gets confused at certain things. */ const gfp_t BINDINGS_GFP_KERNEL =GFP_KERNEL ;
rust/helpers.c +7 -0
@@ -26,6 +26,7 @@#include <linux/spinlock.h> #include <linux/sched/signal.h> #include <linux/wait.h> +#include <linux/fs_parser.h> __noreturn void rust_helper_BUG (void ) {
@@ -128,6 +129,12 @@ void rust_helper_put_task_struct(struct task_struct *t) }EXPORT_SYMBOL_GPL (rust_helper_put_task_struct); +struct dentry *rust_helper_dget (struct dentry *dentry) +{ +return dget (dentry); +} +EXPORT_SYMBOL_GPL (rust_helper_dget); +/* * We use `bindgen`'s `--size_t-is-usize` option to bind the C `size_t` type * as the Rust `usize` type, so we can use it in contexts where Rust
rust/kernel/fs.rs new file +53 -0
53 lines not on this page · raw message
rust/kernel/lib.rs +1 -0
@@ -30,6 +30,7 @@#[cfg(not(test))] #[cfg(not(testlib))] mod allocator; +pub mod fs;mod build_assert;pub mod error;pub mod init;
Signature · 1 line
2.40.1
PATCH 2/80 rust: add ability to register a file system
In reply to Ariel MiculasFrom: Wedson Almeida Filho <[email protected]>
Also add sample that uses the new public API. The registered file system cannot be mounted yet, but can be seen in /proc/filesystems.
More functionality will be added in subsequent commits.
--- rust/kernel/fs.rs | 116 +++++++++++++++++++++++++++++++++++++++- samples/rust/Kconfig | 10 ++++ samples/rust/Makefile | 1 + samples/rust/rust_fs.rs | 31 +++++++++++ scripts/Makefile.build | 2 +- 5 files changed, 157 insertions(+), 3 deletions(-) create mode 100644 samples/rust/rust_fs.rs
rust/kernel/fs.rs +114 -2
@@ -4,9 +4,121 @@//! //! C headers: [`include/linux/fs.h`](../../../../include/linux/fs.h) -use crate ::{bindings}; +use crate ::{bindings, error::code::*, str::CStr ,ThisModule }; +use crate ::error::{to_result,Result };use crate ::types::AlwaysRefCounted ; -use core::{cell::UnsafeCell , ptr}; +use core::{cell::UnsafeCell , marker::PhantomPinned , pin::Pin , ptr}; + +/// A file system type. +pub trait Type { +/// The name of the file system type. +const NAME : &'staticCStr ; + +/// The flags of this file system type. +/// +/// It is a combination of the flags in the [`flags`] module. +const FLAGS :i32 ; +} + +/// File system flags. +pub mod flags { +use crate ::bindings; + +/// The file system requires a device. +pub const REQUIRES_DEV :i32 = bindings::FS_REQUIRES_DEV as _ ; + +/// The options provided when mounting are in binary form. +pub const BINARY_MOUNTDATA :i32 = bindings::FS_BINARY_MOUNTDATA as _ ; + +/// The file system has a subtype. It is extracted from the name and passed in as a parameter. +pub const HAS_SUBTYPE :i32 = bindings::FS_HAS_SUBTYPE as _ ; + +/// The file system can be mounted by userns root. +pub const USERNS_MOUNT :i32 = bindings::FS_USERNS_MOUNT as _ ; + +/// Disables fanotify permission events. +pub const DISALLOW_NOTIFY_PERM :i32 = bindings::FS_DISALLOW_NOTIFY_PERM as _ ; + +/// The file system has been updated to handle vfs idmappings. +pub const ALLOW_IDMAP :i32 = bindings::FS_ALLOW_IDMAP as _ ; + +/// The file systen will handle `d_move` during `rename` internally. +pub const RENAME_DOES_D_MOVE :i32 = bindings::FS_RENAME_DOES_D_MOVE as _ ; +} + +/// A file system registration. +#[derive( Default )] +pub struct Registration { + is_registered:bool , + fs:UnsafeCell <bindings::file_system_type >, + _pin:PhantomPinned , +} + +// SAFETY: `Registration` doesn't really provide any `&self` methods, so it is safe to pass +// references to it around. +unsafe impl Sync for Registration {} + +// SAFETY: Both registration and unregistration are implemented in C and safe to be performed from +// any thread, so `Registration` is `Send`. +unsafe impl Send for Registration {} + +impl Registration { +/// Creates a new file system registration. +/// +/// It is not visible or accessible yet. A successful call to [`Registration::register`] needs +/// to be made before users can mount it. +pub fn new () ->Self { +Self { + is_registered:false , + fs:UnsafeCell ::new (bindings::file_system_type::default ()), + _pin:PhantomPinned , + } + } + +/// Registers a file system so that it can be mounted by users. +/// +/// The file system is described by the [`Type`] argument. +/// +/// It is automatically unregistered when the registration is dropped. +pub fn register <T :Type >(self:Pin <&mut Self >, module: &'staticThisModule ) ->Result { +// SAFETY: We never move out of `this`. +let this =unsafe { self.get_unchecked_mut () }; + +if this.is_registered { +return Err (EINVAL ); + } + +let mut fs = this.fs.get_mut (); + fs.owner = module.0 ; + fs.name =T ::NAME .as_char_ptr (); + fs.fs_flags =T ::FLAGS ; + fs.init_fs_context =Some (Self ::init_fs_context_callback ::<T >); + fs.kill_sb =Some (Self ::kill_sb_callback ::<T >); +// SAFETY: Pointers stored in `fs` are either static so will live for as long as the +// registration is active (it is undone in `drop`). +to_result (unsafe { bindings::register_filesystem (this.fs.get ()) })?; + this.is_registered =true ; +Ok (()) + } + +unsafe extern "C" fn init_fs_context_callback <T :Type >( + _fc_ptr: *mut bindings::fs_context , + ) -> core::ffi::c_int { +EINVAL .to_errno () + } + +unsafe extern "C" fn kill_sb_callback <T :Type >(_sb_ptr: *mut bindings::super_block ) {} +} + +impl Drop for Registration { +fn drop (&mut self) { +if self.is_registered { +// SAFETY: When `is_registered` is `true`, a previous call to `register_filesystem` has +// succeeded, so it is safe to unregister here. +unsafe { bindings::unregister_filesystem (self.fs.get ()) }; + } + } +}/// Wraps the kernel's `struct inode`. ///
samples/rust/Kconfig +10 -0
@@ -30,6 +30,16 @@ config SAMPLE_RUST_PRINT If unsure, say N. +config SAMPLE_RUST_FS + tristate "File system" + help + This option builds the Rust file system sample. + + To compile this as a module, choose M here: + the module will be called rust_fs. + + If unsure, say N. + config SAMPLE_RUST_HOSTPROGS bool "Host programs" help
samples/rust/Makefile +1 -0
@@ -2,5 +2,6 @@ obj-$(CONFIG_SAMPLE_RUST_MINIMAL) += rust_minimal.o obj-$(CONFIG_SAMPLE_RUST_PRINT) += rust_print.o +obj-$(CONFIG_SAMPLE_RUST_FS) += rust_fs.o subdir-$(CONFIG_SAMPLE_RUST_HOSTPROGS) += hostprogs
samples/rust/rust_fs.rs new file +31 -0
@@ -0,0 +1,31 @@ +// SPDX-License-Identifier: GPL-2.0 + +//! Rust file system sample. + +use kernel::prelude::*; +use kernel::{c_str, fs}; + +module! { +type :FsModule , + name:"rust_fs" , + author:"Rust for Linux Contributors" , + license:"GPL" , +} + +struct RustFs ; +impl fs::Type for RustFs { +const NAME : &'staticCStr =c_str! ("rustfs" ); +const FLAGS :i32 = fs::flags::USERNS_MOUNT ; +} + +struct FsModule { + _fs:Pin <Box <fs::Registration >>, +} + +impl kernel::Module for FsModule { +fn init (module: &'staticThisModule ) ->Result <Self > { +let mut reg =Pin ::from (Box ::try_new (fs::Registration ::new ())?); + reg.as_mut ().register ::<RustFs >(module)?; +Ok (Self { _fs: reg }) + } +}
scripts/Makefile.build +1 -1
@@ -277,7 +277,7 @@ $(obj)/%.lst: $(src)/%.c FORCE # Compile Rust sources (.rs) # --------------------------------------------------------------------------- -rust_allowed_features := new_uninit +rust_allowed_features := allocator_api,new_uninit rust_common_cmd = \ RUST_MODFILE=$(modfile) $(RUSTC_OR_CLIPPY) $(rust_flags) \
Signature · 1 line
2.40.1
Re: [PATCH 02/80] rust: add ability to register a file system
Quoted 9 lines · Ariel Miculas
From: Wedson Almeida Filho <[email protected]>
Also add sample that uses the new public API. The registered file system cannot be mounted yet, but can be seen in
/proc/filesystems.More functionality will be added in subsequent commits.
Signed-off-by Wedson Almeida Filho <[email protected]>
Note that you need to add your SoB too in cases where you forward a patch, even if you are just handling the patch; see
https://docs.kernel.org/process/submitting-patches.html#sign-your-work-the-developer-s-certificate-of-origin.
Cheers,
Miguel
PATCH 3/80 rust: define fs context
In reply to Ariel MiculasFrom: Wedson Almeida Filho <[email protected]>
Also make fs mountable, but empty for now.
--- rust/bindings/bindings_helper.h | 3 + rust/bindings/lib.rs | 2 + rust/helpers.c | 12 ++ rust/kernel/error.rs | 46 +++++++ rust/kernel/fs.rs | 224 ++++++++++++++++++++++++++++++-- rust/kernel/lib.rs | 5 + samples/rust/rust_fs.rs | 13 ++ 7 files changed, 297 insertions(+), 8 deletions(-)
rust/bindings/bindings_helper.h +3 -0
@@ -11,7 +11,10 @@#include <linux/wait.h> #include <linux/sched.h> #include <linux/fs.h> +#include <linux/fs_parser.h> /* `bindgen` gets confused at certain things. */ const gfp_t BINDINGS_GFP_KERNEL =GFP_KERNEL ;const gfp_t BINDINGS___GFP_ZERO = __GFP_ZERO; + +const loff_t BINDINGS_MAX_LFS_FILESIZE =MAX_LFS_FILESIZE ;
rust/bindings/lib.rs +2 -0
@@ -51,3 +51,5 @@ mod bindings_helper {pub const GFP_KERNEL :gfp_t =BINDINGS_GFP_KERNEL ;pub const __GFP_ZERO:gfp_t =BINDINGS___GFP_ZERO ; + +pub const MAX_LFS_FILESIZE :loff_t =BINDINGS_MAX_LFS_FILESIZE ;
rust/helpers.c +12 -0
@@ -135,6 +135,18 @@ struct dentry *rust_helper_dget(struct dentry *dentry) }EXPORT_SYMBOL_GPL (rust_helper_dget); +void rust_helper_lockdep_register_key (struct lock_class_key *key) +{ +lockdep_register_key (key); +} +EXPORT_SYMBOL_GPL (rust_helper_lockdep_register_key); + +void rust_helper_lockdep_unregister_key (struct lock_class_key *key) +{ +lockdep_unregister_key (key); +} +EXPORT_SYMBOL_GPL (rust_helper_lockdep_unregister_key); +/* * We use `bindgen`'s `--size_t-is-usize` option to bind the C `size_t` type * as the Rust `usize` type, so we can use it in contexts where Rust
rust/kernel/error.rs +46 -0
49 lines not on this page · raw message
rust/kernel/fs.rs +216 -8
258 lines not on this page · raw message
rust/kernel/lib.rs +5 -0
@@ -98,3 +98,8 @@ fn panic(info: &core::panic::PanicInfo<'_>) -> ! {// instead of `!`. See <https://github.com/rust-lang/rust-bindgen/issues/2094>. loop {} } + +/// Page size defined in terms of the `PAGE_SHIFT` macro from C. +/// +/// [`PAGE_SHIFT`]: ../../../include/asm-generic/page.h +pub const PAGE_SIZE :usize =1 << bindings::PAGE_SHIFT ;
samples/rust/rust_fs.rs +13 -0
@@ -13,9 +13,22 @@ }struct RustFs ; + +#[vtable] +impl fs::Context <Self >for RustFs { +type Data = (); + +fn try_new () ->Result { +pr_info! ("context created!\n" ); +Ok (()) + } +} +impl fs::Type for RustFs { +type Context =Self ;const NAME : &'staticCStr =c_str! ("rustfs" );const FLAGS :i32 = fs::flags::USERNS_MOUNT ; +const MAGIC :u32 =0x72757374 ; }struct FsModule {
Signature · 1 line
2.40.1
PATCH 4/80 rust: add support for file system parameters
In reply to Ariel MiculasFrom: Wedson Almeida Filho <[email protected]>
This allows file system contexts to be further initialised with parameters from userspace before a fs is mounted or reconfigured.
--- rust/helpers.c | 9 + rust/kernel/fs.rs | 142 ++++++++++- rust/kernel/fs/param.rs | 537 ++++++++++++++++++++++++++++++++++++++++ rust/kernel/lib.rs | 1 + samples/rust/rust_fs.rs | 15 ++ 5 files changed, 701 insertions(+), 3 deletions(-) create mode 100644 rust/kernel/fs/param.rs
rust/helpers.c +9 -0
@@ -147,6 +147,15 @@ void rust_helper_lockdep_unregister_key(struct lock_class_key *key) }EXPORT_SYMBOL_GPL (rust_helper_lockdep_unregister_key); +int rust_helper_fs_parse (struct fs_context *fc, +const struct fs_parameter_spec *desc, +struct fs_parameter *param, +struct fs_parse_result *result) +{ +return fs_parse (fc, desc, param, result); +} +EXPORT_SYMBOL_GPL (rust_helper_fs_parse); +/* * We use `bindgen`'s `--size_t-is-usize` option to bind the C `size_t` type * as the Rust `usize` type, so we can use it in contexts where Rust
rust/kernel/fs.rs +139 -3
195 lines not on this page · raw message
rust/kernel/fs/param.rs new file +537 -0
537 lines not on this page · raw message
rust/kernel/lib.rs +1 -0
@@ -18,6 +18,7 @@#![feature(new_uninit)] #![feature(receiver_trait)] #![feature(unsize)] +#![feature(const_mut_refs)] // Ensure conditional compilation based on the kernel configuration works; // otherwise we may silently break things like initcall handling.
samples/rust/rust_fs.rs +15 -0
@@ -18,6 +18,21 @@impl fs::Context <Self >for RustFs {type Data = (); + kernel::define_fs_params! {(), + {flag,"flag" , |_, v| {pr_info! ("flag passed-in: {v}\n" );Ok (()) } }, + {flag_no,"flagno" , |_, v| {pr_info! ("flagno passed-in: {v}\n" );Ok (()) } }, + {bool ,"bool" , |_, v| {pr_info! ("bool passed-in: {v}\n" );Ok (()) } }, + {u32 ,"u32" , |_, v| {pr_info! ("u32 passed-in: {v}\n" );Ok (()) } }, + {u32oct,"u32oct" , |_, v| {pr_info! ("u32oct passed-in: {v}\n" );Ok (()) } }, + {u32hex,"u32hex" , |_, v| {pr_info! ("u32hex passed-in: {v}\n" );Ok (()) } }, + {s32,"s32" , |_, v| {pr_info! ("s32 passed-in: {v}\n" );Ok (()) } }, + {u64 ,"u64" , |_, v| {pr_info! ("u64 passed-in: {v}\n" );Ok (()) } }, + {string,"string" , |_, v| {pr_info! ("string passed-in: {v}\n" );Ok (()) } }, + {enum ,"enum" , [("first" ,10 ), ("second" ,20 )], |_, v| { +pr_info! ("enum passed-in: {v}\n" );Ok (()) } + }, + } +fn try_new () ->Result {pr_info! ("context created!\n" );Ok (())
Signature · 1 line
2.40.1
PATCH 5/80 rust: kernel: add libraries required by the filesystem abstractions
In reply to Ariel MiculasAdd cred.rs, file.rs, io_buffer.rs, iov_iter.rs, mm.rs, pages.rs and user_ptr.rs from the rust branch.
--- rust/bindings/bindings_helper.h | 4 + rust/helpers.c | 35 ++ rust/kernel/cred.rs | 46 ++ rust/kernel/file.rs | 913 ++++++++++++++++++++++++++++++++ rust/kernel/io_buffer.rs | 153 ++++++ rust/kernel/iov_iter.rs | 81 +++ rust/kernel/lib.rs | 11 +- rust/kernel/mm.rs | 149 ++++++ rust/kernel/pages.rs | 144 +++++ rust/kernel/user_ptr.rs | 175 ++++++ 10 files changed, 1710 insertions(+), 1 deletion(-) create mode 100644 rust/kernel/cred.rs create mode 100644 rust/kernel/file.rs create mode 100644 rust/kernel/io_buffer.rs create mode 100644 rust/kernel/iov_iter.rs create mode 100644 rust/kernel/mm.rs create mode 100644 rust/kernel/pages.rs create mode 100644 rust/kernel/user_ptr.rs
rust/bindings/bindings_helper.h +4 -0
@@ -12,6 +12,10 @@#include <linux/sched.h> #include <linux/fs.h> #include <linux/fs_parser.h> +#include <linux/cred.h> +#include <linux/poll.h> +#include <linux/uio.h> +#include <linux/uaccess.h> /* `bindgen` gets confused at certain things. */ const gfp_t BINDINGS_GFP_KERNEL =GFP_KERNEL ;
rust/helpers.c +35 -0
@@ -27,6 +27,7 @@#include <linux/sched/signal.h> #include <linux/wait.h> #include <linux/fs_parser.h> +#include <linux/highmem.h> __noreturn void rust_helper_BUG (void ) {
@@ -156,6 +157,40 @@ int rust_helper_fs_parse(struct fs_context *fc, }EXPORT_SYMBOL_GPL (rust_helper_fs_parse); +const struct cred *rust_helper_get_cred (const struct cred *cred) +{ +return get_cred (cred); +} +EXPORT_SYMBOL_GPL (rust_helper_get_cred); + +void rust_helper_put_cred (const struct cred *cred) { +put_cred (cred); +} +EXPORT_SYMBOL_GPL (rust_helper_put_cred); + +struct file *rust_helper_get_file (struct file *f) +{ +return get_file (f); +} +EXPORT_SYMBOL_GPL (rust_helper_get_file); + +unsigned long rust_helper_clear_user (void __user *to,unsigned long n) +{ +return clear_user (to, n); +} + +void *rust_helper_kmap (struct page *page) +{ +return kmap (page); +} +EXPORT_SYMBOL_GPL (rust_helper_kmap); + +void rust_helper_kunmap (struct page *page) +{ +return kunmap (page); +} +EXPORT_SYMBOL_GPL (rust_helper_kunmap); +/* * We use `bindgen`'s `--size_t-is-usize` option to bind the C `size_t` type * as the Rust `usize` type, so we can use it in contexts where Rust
rust/kernel/cred.rs new file +46 -0
46 lines not on this page · raw message
rust/kernel/file.rs new file +913 -0
913 lines not on this page · raw message
rust/kernel/io_buffer.rs new file +153 -0
153 lines not on this page · raw message
rust/kernel/iov_iter.rs new file +81 -0
81 lines not on this page · raw message
rust/kernel/lib.rs +10 -1
@@ -13,7 +13,9 @@#![no_std] #![feature(allocator_api)] +#![feature(associated_type_defaults)] #![feature(coerce_unsized)] +#![feature(c_size_t)] #![feature(dispatch_from_dyn)] #![feature(new_uninit)] #![feature(receiver_trait)]
@@ -31,11 +33,17 @@#[cfg(not(test))] #[cfg(not(testlib))] mod allocator; -pub mod fs;mod build_assert; +pub mod cred;pub mod error; +pub mod file; +pub mod fs;pub mod init; +pub mod io_buffer;pub mod ioctl; +pub mod iov_iter; +pub mod mm; +pub mod pages;pub mod prelude;pub mod print;mod static_assert;
@@ -45,6 +53,7 @@pub mod sync;pub mod task;pub mod types; +pub mod user_ptr;#[doc(hidden)] pub use bindings;
rust/kernel/mm.rs new file +149 -0
149 lines not on this page · raw message
rust/kernel/pages.rs new file +144 -0
144 lines not on this page · raw message
rust/kernel/user_ptr.rs new file +175 -0
175 lines not on this page · raw message
Signature · 1 line
2.40.1
Re: [PATCH 05/80] rust: kernel: add libraries required by the filesystem abstractions
Quoted 5 lines · Ariel Miculas
Add
cred.rs,file.rs,io_buffer.rs,iov_iter.rs,mm.rs,pages.rsanduser_ptr.rsfrom the rust branch.Signed-off-by Ariel Miculas <[email protected]>
I guess you added these in bulk for RFC purposes, so that is fine, but eventually these will need to be submitted individually etc.
An alternative (for RFCs), if you want people to focus on another part of the series, is to create a branch somewhere with the dependencies applied and then base the patches on top (being clear that the dependencies would be needed, of course). Some extra notes at [1] for others that may be interested in that approach.
[1] https://rust-for-linux.com/contributing#submitting-new-abstractions-and-modules
Cheers,
Miguel
PATCH 6/80 rust: allow fs driver to initialise new superblocks
In reply to Ariel MiculasFrom: Wedson Almeida Filho <[email protected]>
They are also allowed to specify the type of superblock keying.
With this change, drivers get a new callback, fill_super, that they need to implement to initialise new superblocks.
In subsequent commits, they'll be able to populate the root with entries, but for now it's empty.
--- rust/kernel/fs.rs | 406 ++++++++++++++++++++++++++++++++-------- rust/kernel/fs/param.rs | 17 +- samples/rust/rust_fs.rs | 14 +- 3 files changed, 357 insertions(+), 80 deletions(-)
rust/kernel/fs.rs +328 -78
@@ -7,11 +7,31 @@use crate ::{bindings, error::code::*, str::CStr ,ThisModule };use crate ::error::{to_result, from_kernel_result,Error ,Result };use crate ::types::{AlwaysRefCounted ,ForeignOwnable ,ScopeGuard }; -use core::{cell::UnsafeCell , marker::PhantomPinned , pin::Pin , ptr}; +use core::{cell::UnsafeCell , marker::PhantomData , marker::PhantomPinned , pin::Pin , ptr};use macros::vtable;pub mod param; +/// Type of superblock keying. +/// +/// It determines how C's `fs_context_operations::get_tree` is implemented. +pub enum Super { +/// Only one such superblock may exist. +Single , + +/// As [`Super::Single`], but reconfigure if it exists. +SingleReconf , + +/// Superblocks with different data pointers may exist. +Keyed , + +/// Multiple independent superblocks may exist. +Independent , + +/// Uses a block device. +BlockDev , +} +/// A file system context. /// /// It is used to gather configuration to then mount or reconfigure a file system.
@@ -48,6 +68,16 @@ fn parse_monolithic<'a>( ) -> Result<Option<&'a mut [u8]>> { Ok(None) } + +/// Returns the superblock data to be used by this file system context. +/// +/// This is only needed when [`Type::SUPER_TYPE`] is [`Super::Keyed`], otherwise it is never +/// called. In the former case, when the fs is being mounted, an existing superblock is reused +/// if one can be found with the same data as the returned value; otherwise a new superblock is +/// created. +fn tree_key (_data: &mut Self ::Data ) ->Result <T ::Data > { +Err (ENOTSUPP ) + } }struct Tables <T :Type + ?Sized >(T );
@@ -67,12 +97,22 @@ impl<T: Type + ?Sized> Tables<T> {unsafe extern "C" fn free_callback (fc: *mut bindings::fs_context ) {// SAFETY: The callback contract guarantees that `fc` is valid. -let ptr =unsafe { (*fc).fs_private }; +let fc =unsafe { &*fc }; + +let ptr = fc.fs_private;if !ptr.is_null () {// SAFETY: `fs_private` was initialised with the result of a `to_pointer` call in // `init_fs_context_callback`, so it's ok to call `from_foreign` here. unsafe { <T ::Context as Context <T >>::Data ::from_foreign (ptr) }; } + +let ptr = fc.s_fs_info; +if !ptr.is_null () { +// SAFETY: `s_fs_info` may be initialised with the result of a `to_pointer` call in +// `get_tree_callback` when keyed superblocks are used (`get_tree_keyed` sets it), so +// it's ok to call `from_foreign` here. +unsafe {T ::Data ::from_foreign (ptr) }; + } } unsafe extern "C" fn parse_param_callback(
@@ -84,8 +124,8 @@ impl<T: Type + ?Sized> Tables<T> {let ptr =unsafe { (*fc).fs_private };// SAFETY: The value of `ptr` (coming from `fs_private` was initialised in -// `init_fs_context_callback` to the result of an `into_pointer` call. Since the -// context is valid, `from_pointer` wasn't called yet, so `ptr` is valid. Additionally, +// `init_fs_context_callback` to the result of an `into_foreign` call. Since the +// context is valid, `from_foreign` wasn't called yet, so `ptr` is valid. Additionally, // the callback contract guarantees that callbacks are serialised, so it is ok to // mutably reference it. let mut data =
@@ -126,73 +166,91 @@ impl<T: Type + ?Sized> Tables<T> {unsafe extern "C" fn fill_super_callback ( sb_ptr: *mut bindings::super_block , - _fc: *mut bindings::fs_context , + fc: *mut bindings::fs_context , ) -> core::ffi::c_int {from_kernel_result! { -// The following is temporary code to create the root inode and dentry. It will be -// replaced with calls to Rust code. - -// SAFETY: The callback contract guarantees that `sb_ptr` is the only pointer to a -// newly-allocated superblock, so it is safe to mutably reference it. -let sb =unsafe { &mut *sb_ptr }; - - sb.s_maxbytes = bindings::MAX_LFS_FILESIZE ; - sb.s_blocksize =crate ::PAGE_SIZE as _ ; - sb.s_blocksize_bits = bindings::PAGE_SHIFT as _ ; - sb.s_magic =T ::MAGIC as _ ; - sb.s_op = &Tables ::<T >::SUPER_BLOCK ; - sb.s_time_gran =1 ; - -// Create and initialise the root inode. - -// SAFETY: `sb` was just created initialised, so it is safe pass it to `new_inode`. -let inode =unsafe { bindings::new_inode (sb) }; -if inode.is_null () { -return Err (ENOMEM ); - } - - { -// SAFETY: This is a newly-created inode. No other references to it exist, so it is -// safe to mutably dereference it. -let inode =unsafe { &mut *inode }; - -// SAFETY: `current_time` requires that `inode.sb` be valid, which is the case here -// since we allocated the inode through the superblock. -let time =unsafe { bindings::current_time (inode) }; - inode.i_ino =1 ; - inode.i_mode = (bindings::S_IFDIR |0o755 )as _ ; - inode.i_mtime = time; - inode.i_atime = time; - inode.i_ctime = time; - -// SAFETY: `simple_dir_operations` never changes, it's safe to reference it. - inode.__bindgen_anon_3.i_fop =unsafe { &bindings::simple_dir_operations }; - -// SAFETY: `simple_dir_inode_operations` never changes, it's safe to reference it. - inode.i_op =unsafe { &bindings::simple_dir_inode_operations }; - -// SAFETY: `inode` is valid for write. -unsafe { bindings::set_nlink (inode,2 ) }; - } - -// SAFETY: `d_make_root` requires that `inode` be valid and referenced, which is the -// case for this call. -// -// It takes over the inode, even on failure, so we don't need to clean it up. -let dentry =unsafe { bindings::d_make_root (inode) }; -if dentry.is_null () { -return Err (ENOMEM ); - } +// SAFETY: The callback contract guarantees that `fc` is valid. It also guarantees that +// the callbacks are serialised for a given `fc`, so it is safe to mutably dereference +// it. +let fc =unsafe { &mut *fc }; +let ptr = core::mem::replace (&mut fc.fs_private, ptr::null_mut ()); - sb.s_root = dentry; +// SAFETY: The value of `ptr` (coming from `fs_private` was initialised in +// `init_fs_context_callback` to the result of an `into_foreign` call. The context is +// being used to initialise a superblock, so we took over `ptr` (`fs_private` is set to +// null now) and call `from_foreign` below. +let data = +unsafe { <<T ::Context as Context <T >>::Data as ForeignOwnable >::from_foreign (ptr) }; + +// SAFETY: The callback contract guarantees that `sb_ptr` is a unique pointer to a +// newly-created superblock. +let newsb =unsafe {NewSuperBlock ::new (sb_ptr) }; +T ::fill_super (data, newsb)?;Ok (0 ) } }unsafe extern "C" fn get_tree_callback (fc: *mut bindings::fs_context ) -> core::ffi::c_int { -// SAFETY: `fc` is valid per the callback contract. `fill_super_callback` also has the -// right type and is a valid callback. -unsafe { bindings::get_tree_nodev (fc,Some (Self ::fill_super_callback)) } +// N.B. When new types are added below, we may need to update `kill_sb_callback` to ensure +// that we're cleaning up properly. +match T ::SUPER_TYPE { +Super ::Single =>unsafe { +// SAFETY: `fc` is valid per the callback contract. `fill_super_callback` also has +// the right type and is a valid callback. + bindings::get_tree_single (fc,Some (Self ::fill_super_callback)) + }, +Super ::SingleReconf =>unsafe { +// SAFETY: `fc` is valid per the callback contract. `fill_super_callback` also has +// the right type and is a valid callback. + bindings::get_tree_single_reconf (fc,Some (Self ::fill_super_callback)) + }, +Super ::Independent =>unsafe { +// SAFETY: `fc` is valid per the callback contract. `fill_super_callback` also has +// the right type and is a valid callback. + bindings::get_tree_nodev (fc,Some (Self ::fill_super_callback)) + }, +Super ::BlockDev =>unsafe { +// SAFETY: `fc` is valid per the callback contract. `fill_super_callback` also has +// the right type and is a valid callback. + bindings::get_tree_bdev (fc,Some (Self ::fill_super_callback)) + }, +Super ::Keyed => { +from_kernel_result! { +// SAFETY: `fc` is valid per the callback contract. +let ctx =unsafe { &*fc }; +let ptr = ctx.fs_private; + +// SAFETY: The value of `ptr` (coming from `fs_private` was initialised in +// `init_fs_context_callback` to the result of an `into_foreign` call. Since +// the context is valid, `from_foreign` wasn't called yet, so `ptr` is valid. +// Additionally, the callback contract guarantees that callbacks are +// serialised, so it is ok to mutably reference it. +let mut data =unsafe { + <<T ::Context as Context <T >>::Data as ForeignOwnable >::borrow_mut (ptr) + }; +let fs_data =T ::Context ::tree_key (&mut data)?; +let fs_data_ptr = fs_data.into_foreign (); + +// `get_tree_keyed` reassigns `ctx.s_fs_info`, which should be ok because +// nowhere else is it assigned a non-null value. However, we add the assert +// below to ensure that there are no unexpected paths on the C side that may do +// this. +assert_eq! (ctx.s_fs_info, core::ptr::null_mut ()); + +// SAFETY: `fc` is valid per the callback contract. `fill_super_callback` also +// has the right type and is a valid callback. Lastly, we just called +// `into_foreign` above, so `fs_data_ptr` is also valid. +to_result (unsafe { + bindings::get_tree_keyed ( + fc, +Some (Self ::fill_super_callback), + fs_data_ptras _ , + ) + })?; +Ok (0 ) + } + } + } }unsafe extern "C" fn reconfigure_callback (_fc: *mut bindings::fs_context ) -> core::ffi::c_int {
@@ -208,8 +266,8 @@ impl<T: Type + ?Sized> Tables<T> {let ptr =unsafe { (*fc).fs_private };// SAFETY: The value of `ptr` (coming from `fs_private` was initialised in -// `init_fs_context_callback` to the result of an `into_pointer` call. Since the -// context is valid, `from_pointer` wasn't called yet, so `ptr` is valid. Additionally, +// `init_fs_context_callback` to the result of an `into_foreign` call. Since the +// context is valid, `from_foreign` wasn't called yet, so `ptr` is valid. Additionally, // the callback contract guarantees that callbacks are serialised, so it is ok to // mutably reference it. let mut data =
@@ -270,18 +328,25 @@ pub trait Type {/// The context used to build fs configuration before it is mounted or reconfigured. type Context :Context <Self > + ?Sized ; +/// Data associated with each file system instance. + type Data: ForeignOwnable + Send + Sync = (); + +/// Determines how superblocks for this file system type are keyed. +const SUPER_TYPE :Super ; +/// The name of the file system type. const NAME : &'staticCStr ; -/// The magic number associated with the file system. -/// -/// This is normally one of the values in `include/uapi/linux/magic.h`. -const MAGIC :u32 ; -/// The flags of this file system type. /// /// It is a combination of the flags in the [`flags`] module. const FLAGS :i32 ; + +/// Initialises a super block for this file system type. +fn fill_super ( + data: <Self ::Context as Context <Self >>::Data , + sb:NewSuperBlock <'_,Self >, + ) ->Result <&SuperBlock <Self >>; }/// File system flags.
@@ -425,14 +490,33 @@ unsafe fn unregister_keys(fs: *mut bindings::file_system_type) { }unsafe extern "C" fn kill_sb_callback <T :Type + ?Sized >(sb_ptr: *mut bindings::super_block ) { -// SAFETY: We always call `get_tree_nodev` from `get_tree_callback`, so we never have a -// device, so it is ok to call the function below. Additionally, the callback contract -// guarantees that `sb_ptr` is valid. -unsafe { bindings::kill_anon_super (sb_ptr) } - -// SAFETY: The callback contract guarantees that `sb_ptr` is valid, and the `kill_sb` -// callback being called implies that the `s_type` is also valid. -unsafe {Self ::unregister_keys ((*sb_ptr).s_type) }; +if let Super ::BlockDev =T ::SUPER_TYPE { +// SAFETY: When the superblock type is `BlockDev`, we have a block device so it's safe +// to call `kill_block_super`. Additionally, the callback contract guarantees that +// `sb_ptr` is valid. +unsafe { bindings::kill_block_super (sb_ptr) } + }else { +// SAFETY: We always call a `get_tree_nodev` variant from `get_tree_callback` without a +// device when `T::SUPER_TYPE` is not `BlockDev`, so we never have a device in such +// cases, therefore it is ok to call the function below. Additionally, the callback +// contract guarantees that `sb_ptr` is valid. +unsafe { bindings::kill_anon_super (sb_ptr) } + } + +// SAFETY: The callback contract guarantees that `sb_ptr` is valid. +let sb =unsafe { &*sb_ptr }; + +// SAFETY: The `kill_sb` callback being called implies that the `s_type` field is valid. +unsafe {Self ::unregister_keys (sb.s_type) }; + +let ptr = sb.s_fs_info; +if !ptr.is_null () { +// SAFETY: The only place where `s_fs_info` is assigned is `NewSuperBlock::init`, where +// it's initialised with the result of a `to_pointer` call. We checked above that ptr +// is non-null because it would be null if we never reached the point where we init the +// field. +unsafe {T ::Data ::from_foreign (ptr) }; + } } }
@@ -446,6 +530,172 @@ fn drop(&mut self) { } } +/// State of [`NewSuperBlock`] that indicates that [`NewSuperBlock::init`] needs to be called +/// eventually. +pub struct NeedsInit ; + +/// State of [`NewSuperBlock`] that indicates that [`NewSuperBlock::init_root`] needs to be called +/// eventually. +pub struct NeedsRoot ; + +/// Required superblock parameters. +/// +/// This is used in [`NewSuperBlock::init`]. +pub struct SuperParams { +/// The magic number of the superblock. +pub magic:u32 , + +/// The size of a block in powers of 2 (i.e., for a value of `n`, the size is `2^n`. +pub blocksize_bits:u8 , + +/// Maximum size of a file. +pub maxbytes:i64 , + +/// Granularity of c/m/atime in ns (cannot be worse than a second). +pub time_gran:u32 , +} + +impl SuperParams { +/// Default value for instances of [`SuperParams`]. +pub const DEFAULT :Self =Self { + magic:0 , + blocksize_bits:crate ::PAGE_SIZE as _ , + maxbytes: bindings::MAX_LFS_FILESIZE , + time_gran:1 , + }; +} + +/// A superblock that is still being initialised. +/// +/// It uses type states to ensure that callers use the right sequence of calls. +/// +/// # Invariants +/// +/// The superblock is a newly-created one and this is the only active pointer to it. +pub struct NewSuperBlock <'a,T :Type + ?Sized ,S =NeedsInit > { + sb: *mut bindings::super_block , + _p:PhantomData <(&'aT ,S )>, +} + +impl <'a,T :Type + ?Sized >NewSuperBlock <'a,T ,NeedsInit > { +/// Creates a new instance of [`NewSuperBlock`]. +/// +/// # Safety +/// +/// `sb` must point to a newly-created superblock and it must be the only active pointer to it. +unsafe fn new (sb: *mut bindings::super_block ) ->Self { +// INVARIANT: The invariants are satisfied by the safety requirements of this function. +Self { + sb, + _p:PhantomData , + } + } + +/// Initialises the superblock so that it transitions to the [`NeedsRoot`] type state. +pub fn init ( + self, + data:T ::Data , + params: &SuperParams , + ) ->Result <NewSuperBlock <'a,T ,NeedsRoot >> { +// SAFETY: The type invariant guarantees that `self.sb` is the only pointer to a +// newly-allocated superblock, so it is safe to mutably reference it. +let sb =unsafe { &mut *self.sb }; + + sb.s_magic = params.magicas _ ; + sb.s_op = &Tables ::<T >::SUPER_BLOCK ; + sb.s_maxbytes = params.maxbytes; + sb.s_time_gran = params.time_gran; + sb.s_blocksize_bits = params.blocksize_bits; + sb.s_blocksize =1 ; +if sb.s_blocksize.leading_zeros () < params.blocksize_bits.into () { +return Err (EINVAL ); + } + sb.s_blocksize =1 << sb.s_blocksize_bits; + +// Keyed file systems already have `s_fs_info` initialised. +let info = data.into_foreign ()as *mut _ ; +if let Super ::Keyed =T ::SUPER_TYPE { +// SAFETY: We just called `into_foreign` above. +unsafe {T ::Data ::from_foreign (info) }; + +if sb.s_fs_info != info { +return Err (EINVAL ); + } + }else { + sb.s_fs_info = info; + } + +Ok (NewSuperBlock { + sb: self.sb, + _p:PhantomData , + }) + } +} + +impl <'a,T :Type + ?Sized >NewSuperBlock <'a,T ,NeedsRoot > { +/// Initialises the root of the superblock. +pub fn init_root (self) ->Result <&'aSuperBlock <T >> { +// The following is temporary code to create the root inode and dentry. It will be replaced +// once we allow inodes and dentries to be created directly from Rust code. + +// SAFETY: `sb` is initialised (`NeedsRoot` typestate implies it), so it is safe to pass it +// to `new_inode`. +let inode =unsafe { bindings::new_inode (self.sb) }; +if inode.is_null () { +return Err (ENOMEM ); + } + + { +// SAFETY: This is a newly-created inode. No other references to it exist, so it is +// safe to mutably dereference it. +let inode =unsafe { &mut *inode }; + +// SAFETY: `current_time` requires that `inode.sb` be valid, which is the case here +// since we allocated the inode through the superblock. +let time =unsafe { bindings::current_time (inode) }; + inode.i_ino =1 ; + inode.i_mode = (bindings::S_IFDIR |0o755 )as _ ; + inode.i_mtime = time; + inode.i_atime = time; + inode.i_ctime = time; + +// SAFETY: `simple_dir_operations` never changes, it's safe to reference it. + inode.__bindgen_anon_3.i_fop =unsafe { &bindings::simple_dir_operations }; + +// SAFETY: `simple_dir_inode_operations` never changes, it's safe to reference it. + inode.i_op =unsafe { &bindings::simple_dir_inode_operations }; + +// SAFETY: `inode` is valid for write. +unsafe { bindings::set_nlink (inode,2 ) }; + } + +// SAFETY: `d_make_root` requires that `inode` be valid and referenced, which is the +// case for this call. +// +// It takes over the inode, even on failure, so we don't need to clean it up. +let dentry =unsafe { bindings::d_make_root (inode) }; +if dentry.is_null () { +return Err (ENOMEM ); + } + +// SAFETY: The typestate guarantees that `self.sb` is valid. +unsafe { (*self.sb).s_root = dentry }; + +// SAFETY: The typestate guarantees that `self.sb` is initialised and we just finished +// setting its root, so it's a fully ready superblock. +Ok (unsafe { &mut *self.sb.cast () }) + } +} + +/// A file system super block. +/// +/// Wraps the kernel's `struct super_block`. +#[repr(transparent)] +pub struct SuperBlock <T :Type + ?Sized >( +pub (crate )UnsafeCell <bindings::super_block >, +PhantomData <T >, +); +/// Wraps the kernel's `struct inode`. /// /// # Invariants
rust/kernel/fs/param.rs +16 -1
@@ -487,9 +487,24 @@ macro_rules! count_brace_items {/// /// # impl fs::Type for Example { /// # type Context = Self; +/// # const SUPER_TYPE: fs::Super = fs::Super::Independent; /// # const NAME: &'static CStr = c_str!("example"); /// # const FLAGS: i32 = 0; -/// # const MAGIC: u32 = 0x6578616d; +/// # +/// # fn fill_super<'a>( +/// # _data: Box<State>, +/// # sb: fs::NewSuperBlock<'_, Self>, +/// # ) -> Result<&fs::SuperBlock<Self>> { +/// # let sb = sb.init( +/// # (), +/// # &fs::SuperParams { +/// # magic: 0x6578616d, +/// # ..fs::SuperParams::DEFAULT +/// # }, +/// # )?; +/// # let sb = sb.init_root()?; +/// # Ok(sb) +/// # } /// # } /// ``` #[macro_export]
samples/rust/rust_fs.rs +13 -1
@@ -41,9 +41,21 @@ fn try_new() -> Result {impl fs::Type for RustFs {type Context =Self ; +const SUPER_TYPE : fs::Super = fs::Super ::Independent ;const NAME : &'staticCStr =c_str! ("rustfs" );const FLAGS :i32 = fs::flags::USERNS_MOUNT ; -const MAGIC :u32 =0x72757374 ; + +fn fill_super (_data: (), sb: fs::NewSuperBlock <'_,Self >) ->Result <&fs::SuperBlock <Self >> { +let sb = sb.init ( + (), + &fs::SuperParams { + magic:0x72757374 , + ..fs::SuperParams ::DEFAULT + }, + )?; +let sb = sb.init_root ()?; +Ok (sb) + } }struct FsModule {
Signature · 1 line
2.40.1
PATCH 7/80 rust: add `module_fs` macro
In reply to Ariel MiculasFrom: Wedson Almeida Filho <[email protected]>
This reduces the amount of code needed to implement a module that only implements a file system.
--- rust/kernel/fs.rs | 77 +++++++++++++++++++++++++++++++++++++++++ rust/kernel/prelude.rs | 1 + samples/rust/rust_fs.rs | 16 ++------- 3 files changed, 80 insertions(+), 14 deletions(-)
rust/kernel/fs.rs +77 -0
@@ -4,6 +4,7 @@//! //! C headers: [`include/linux/fs.h`](../../../../include/linux/fs.h) +use alloc::boxed::Box ;use crate ::{bindings, error::code::*, str::CStr ,ThisModule };use crate ::error::{to_result, from_kernel_result,Error ,Result };use crate ::types::{AlwaysRefCounted ,ForeignOwnable ,ScopeGuard };
@@ -757,3 +758,79 @@ pub(crate) unsafe fn from_ptr<'a>(ptr: *const bindings::filename) -> &'a Filenamunsafe { &*ptr.cast () } } } + +/// Kernel module that exposes a single file system implemented by `T`. +pub struct Module <T :Type > { + _fs:Pin <Box <Registration >>, + _p:PhantomData <T >, +} + +impl <T :Type +Sync >crate ::Module for Module <T > { +fn init (_name: &'staticCStr , module: &'staticThisModule ) ->Result <Self > { +let mut reg =Pin ::from (Box ::try_new (Registration ::new ())?); + reg.as_mut ().register ::<T >(module)?; +Ok (Self { + _fs: reg, + _p:PhantomData , + }) + } +} + +/// Declares a kernel module that exposes a single file system. +/// +/// The `type` argument must be a type which implements the [`Type`] trait. Also accepts various +/// forms of kernel metadata. +/// +/// # Examples +/// +/// ```ignore +/// use kernel::prelude::*; +/// use kernel::{c_str, fs}; +/// +/// module_fs! { +/// type: MyFs, +/// name: b"my_fs_kernel_module", +/// author: b"Rust for Linux Contributors", +/// description: b"My very own file system kernel module!", +/// license: b"GPL", +/// } +/// +/// struct MyFs; +/// +/// #[vtable] +/// impl fs::Context<Self> for MyFs { +/// type Data = (); +/// fn try_new() -> Result { +/// Ok(()) +/// } +/// } +/// +/// impl fs::Type for MyFs { +/// type Context = Self; +/// const SUPER_TYPE: fs::Super = fs::Super::Independent; +/// const NAME: &'static CStr = c_str!("example"); +/// const FLAGS: i32 = 0; +/// +/// fn fill_super(_data: (), sb: fs::NewSuperBlock<'_, Self>) -> Result<&fs::SuperBlock<Self>> { +/// let sb = sb.init( +/// (), +/// &fs::SuperParams { +/// magic: 0x6578616d, +/// ..fs::SuperParams::DEFAULT +/// }, +/// )?; +/// let sb = sb.init_root()?; +/// Ok(sb) +/// } +/// } +/// ``` +#[macro_export] +macro_rules! module_fs { + (type : $type:ty, $($f:tt)*) => { +type ModuleType = $crate::fs::Module <$type>; + $crate::macros::module! { +type :ModuleType , + $($f)* + } + } +}
rust/kernel/prelude.rs +1 -0
@@ -28,6 +28,7 @@pub use super ::{pr_alert, pr_crit, pr_debug, pr_emerg, pr_err, pr_info, pr_notice, pr_warn};pub use super ::{init, pin_init, try_init, try_pin_init}; +pub use super ::{module_fs, module_misc_device};pub use super ::static_assert;
samples/rust/rust_fs.rs +2 -14
@@ -5,8 +5,8 @@use kernel::prelude::*;use kernel::{c_str, fs}; -module! { -type :FsModule , +module_fs! { +type :RustFs , name:"rust_fs" , author:"Rust for Linux Contributors" , license:"GPL" ,
@@ -57,15 +57,3 @@ fn fill_super(_data: (), sb: fs::NewSuperBlock<'_, Self>) -> Result<&fs::SuperBlOk (sb) } } - -struct FsModule { - _fs:Pin <Box <fs::Registration >>, -} - -impl kernel::Module for FsModule { -fn init (module: &'staticThisModule ) ->Result <Self > { -let mut reg =Pin ::from (Box ::try_new (fs::Registration ::new ())?); - reg.as_mut ().register ::<RustFs >(module)?; -Ok (Self { _fs: reg }) - } -}
Signature · 1 line
2.40.1
PATCH 8/80 WIP: rust: allow fs to be populated
In reply to Ariel MiculasFrom: Wedson Almeida Filho <[email protected]>
--- rust/bindings/bindings_helper.h | 4 + rust/bindings/lib.rs | 3 + rust/helpers.c | 7 + rust/kernel/fs.rs | 777 +++++++++++++++++++++++++++++--- rust/kernel/fs/param.rs | 8 +- rust/kernel/prelude.rs | 1 - samples/rust/rust_fs.rs | 51 ++- 7 files changed, 778 insertions(+), 73 deletions(-)
rust/bindings/bindings_helper.h +4 -0
@@ -22,3 +22,7 @@ const gfp_t BINDINGS_GFP_KERNEL = GFP_KERNEL;const gfp_t BINDINGS___GFP_ZERO = __GFP_ZERO;const loff_t BINDINGS_MAX_LFS_FILESIZE =MAX_LFS_FILESIZE ; + +const slab_flags_t BINDINGS_SLAB_RECLAIM_ACCOUNT =SLAB_RECLAIM_ACCOUNT ; +const slab_flags_t BINDINGS_SLAB_MEM_SPREAD =SLAB_MEM_SPREAD ; +const slab_flags_t BINDINGS_SLAB_ACCOUNT =SLAB_ACCOUNT ;
rust/bindings/lib.rs +3 -0
@@ -53,3 +53,6 @@ mod bindings_helper {pub const __GFP_ZERO:gfp_t =BINDINGS___GFP_ZERO ;pub const MAX_LFS_FILESIZE :loff_t =BINDINGS_MAX_LFS_FILESIZE ; + +pub const SLAB_RECLAIM_ACCOUNT :slab_flags_t =BINDINGS_SLAB_RECLAIM_ACCOUNT ; +pub const SLAB_MEM_SPREAD :slab_flags_t =BINDINGS_SLAB_MEM_SPREAD ;
rust/helpers.c +7 -0
@@ -191,6 +191,13 @@ void rust_helper_kunmap(struct page *page) }EXPORT_SYMBOL_GPL (rust_helper_kunmap); +void *rust_helper_alloc_inode_sb (struct super_block *sb, +struct kmem_cache *cache,gfp_t gfp) +{ +return alloc_inode_sb (sb, cache, gfp); +} +EXPORT_SYMBOL_GPL (rust_helper_alloc_inode_sb); +/* * We use `bindgen`'s `--size_t-is-usize` option to bind the C `size_t` type * as the Rust `usize` type, so we can use it in contexts where Rust
rust/kernel/fs.rs +709 -68
941 lines not on this page · raw message
rust/kernel/fs/param.rs +7 -1
@@ -502,7 +502,13 @@ macro_rules! count_brace_items {/// # ..fs::SuperParams::DEFAULT /// # }, /// # )?; -/// # let sb = sb.init_root()?; +/// # let root_inode = sb.try_new_dcache_dir_inode(fs::INodeParams { +/// # mode: 0o755, +/// # ino: 1, +/// # value: (), +/// # })?; +/// # let root = sb.try_new_root_dentry(root_inode)?; +/// # let sb = sb.init_root(root)?; /// # Ok(sb) /// # } /// # }
rust/kernel/prelude.rs +0 -1
@@ -28,7 +28,6 @@pub use super ::{pr_alert, pr_crit, pr_debug, pr_emerg, pr_err, pr_info, pr_notice, pr_warn};pub use super ::{init, pin_init, try_init, try_pin_init}; -pub use super ::{module_fs, module_misc_device};pub use super ::static_assert;
samples/rust/rust_fs.rs +48 -3
78 lines not on this page · raw message
Signature · 1 line
2.40.1
PATCH 9/80 rust: kernel: backport the delay module from the rust branch
In reply to Ariel Miculas--- rust/kernel/delay.rs | 104 +++++++++++++++++++++++++++++++++++++++++++ rust/kernel/lib.rs | 1 + 2 files changed, 105 insertions(+) create mode 100644 rust/kernel/delay.rs
rust/kernel/delay.rs new file +104 -0
@@ -0,0 +1,104 @@ +// SPDX-License-Identifier: GPL-2.0 + +//! Delay functions for operations like sleeping. +//! +//! C header: [`include/linux/delay.h`](../../../../include/linux/delay.h) + +use crate ::bindings; +use core::{cmp::min, time::Duration }; + +const MILLIS_PER_SEC :u64 =1_000 ; + +fn coarse_sleep_conversion (duration:Duration ) -> core::ffi::c_uint { +let milli_as_nanos =Duration ::MILLISECOND .subsec_nanos (); + +// Rounds the nanosecond component of `duration` up to the nearest millisecond. +let nanos_as_millis = duration.subsec_nanos ().wrapping_add (milli_as_nanos -1 ) / milli_as_nanos; + +// Saturates the second component of `duration` to `c_uint::MAX`. +let seconds_as_millis =min ( + duration.as_secs ().saturating_mul (MILLIS_PER_SEC ), + u64::from (core::ffi::c_uint::MAX ), + )as core::ffi::c_uint ; + + seconds_as_millis.saturating_add (nanos_as_millis) +} + +/// Sleeps safely even with waitqueue interruptions. +/// +/// This function forwards the call to the C side `msleep` function. As a result, +/// `duration` will be rounded up to the nearest millisecond if granularity less +/// than a millisecond is provided. Any [`Duration`] that exceeds +/// [`c_uint::MAX`][core::ffi::c_uint::MAX] in milliseconds is saturated. +/// +/// # Examples +/// +// Keep these in sync with `test_coarse_sleep_examples`. +/// ``` +/// # use core::time::Duration; +/// # use kernel::delay::coarse_sleep; +/// coarse_sleep(Duration::ZERO); // Equivalent to `msleep(0)`. +/// coarse_sleep(Duration::from_nanos(1)); // Equivalent to `msleep(1)`. +/// +/// coarse_sleep(Duration::from_nanos(1_000_000)); // Equivalent to `msleep(1)`. +/// coarse_sleep(Duration::from_nanos(1_000_001)); // Equivalent to `msleep(2)`. +/// coarse_sleep(Duration::from_nanos(1_999_999)); // Equivalent to `msleep(2)`. +/// +/// coarse_sleep(Duration::from_millis(1)); // Equivalent to `msleep(1)`. +/// coarse_sleep(Duration::from_millis(2)); // Equivalent to `msleep(2)`. +/// +/// coarse_sleep(Duration::from_secs(1)); // Equivalent to `msleep(1000)`. +/// coarse_sleep(Duration::new(1, 1)); // Equivalent to `msleep(1001)`. +/// coarse_sleep(Duration::new(1, 2)); // Equivalent to `msleep(1001)`. +/// ``` +pub fn coarse_sleep (duration:Duration ) { +// SAFETY: msleep is safe for all values of an `unsigned int`. +unsafe { bindings::msleep (coarse_sleep_conversion (duration)) } +} + +#[cfg(test)] +mod tests { +use super ::{coarse_sleep_conversion,MILLIS_PER_SEC }; +use core::time::Duration ; + +#[test] +fn test_coarse_sleep_examples () { +// Keep these in sync with `coarse_sleep`'s `# Examples` section. + +assert_eq! (coarse_sleep_conversion (Duration ::ZERO ),0 ); +assert_eq! (coarse_sleep_conversion (Duration ::from_nanos (1 )),1 ); + +assert_eq! (coarse_sleep_conversion (Duration ::from_nanos (1_000_000 )),1 ); +assert_eq! (coarse_sleep_conversion (Duration ::from_nanos (1_000_001 )),2 ); +assert_eq! (coarse_sleep_conversion (Duration ::from_nanos (1_999_999 )),2 ); + +assert_eq! (coarse_sleep_conversion (Duration ::from_millis (1 )),1 ); +assert_eq! (coarse_sleep_conversion (Duration ::from_millis (2 )),2 ); + +assert_eq! (coarse_sleep_conversion (Duration ::from_secs (1 )),1000 ); +assert_eq! (coarse_sleep_conversion (Duration ::new (1 ,1 )),1001 ); +assert_eq! (coarse_sleep_conversion (Duration ::new (1 ,2 )),1001 ); + } + +#[test] +fn test_coarse_sleep_saturation () { +assert! ( +coarse_sleep_conversion (Duration ::new ( + core::ffi::c_uint::MAX as u64 /MILLIS_PER_SEC , +0 + )) < core::ffi::c_uint::MAX + ); +assert_eq! ( +coarse_sleep_conversion (Duration ::new ( + core::ffi::c_uint::MAX as u64 /MILLIS_PER_SEC , +999_999_999 + )), + core::ffi::c_uint::MAX + ); + +assert_eq! ( +coarse_sleep_conversion (Duration ::MAX ), + core::ffi::c_uint::MAX + ); + } +}
rust/kernel/lib.rs +1 -0
@@ -35,6 +35,7 @@mod allocator;mod build_assert;pub mod cred; +pub mod delay;pub mod error;pub mod file;pub mod fs;
Signature · 1 line
2.40.1
Re: [PATCH 09/80] rust: kernel: backport the delay module from the rust branch
Quoted 2 lines · Ariel Miculas
Signed-off-by Ariel Miculas <[email protected]>
If we end up adding it as it was in the rust branch, then this would need proper attribution (Cc'ing John).
We are also trying to have the original authors from the rust branch to submit the code they wrote, if possible / they have the time, so that they are involved in the discussion.
Cheers,
Miguel
PATCH 10/80 rust: kernel: add container_of macro
In reply to Ariel Miculas--- rust/kernel/lib.rs | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
rust/kernel/lib.rs +31 -0
@@ -99,6 +99,37 @@ impl ThisModule { } } +/// Produces a pointer to an object from a pointer to one of its fields. +/// +/// # Safety +/// +/// Callers must ensure that the pointer to the field is in fact a pointer to the specified field, +/// as opposed to a pointer to another object of the same type. If this condition is not met, +/// any dereference of the resulting pointer is UB. +/// +/// # Example +/// +/// ``` +/// # use kernel::container_of; +/// struct Test { +/// a: u64, +/// b: u32, +/// } +/// +/// let test = Test { a: 10, b: 20 }; +/// let b_ptr = &test.b; +/// let test_alias = container_of!(b_ptr, Test, b); +/// assert!(core::ptr::eq(&test, test_alias)); +/// ``` +#[macro_export] +macro_rules! container_of { + ($ptr:expr, $type:ty, $($f:tt)*) => {{ +let ptr = $ptras *const _ as *const u8 ; +let offset = $crate::offset_of!($type , $($f)*); + ptr.wrapping_offset (-offset)as *const $type + }} +} +#[cfg(not(any(testlib, test)))] #[panic_handler] fn panic (info: &core::panic::PanicInfo <'_>) -> ! {
Signature · 1 line
2.40.1
PATCH 11/80 rust: kernel: add offset_of macro
In reply to Ariel Miculas--- rust/kernel/lib.rs | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+)
rust/kernel/lib.rs +32 -0
@@ -97,6 +97,38 @@ impl ThisModule {pub const unsafe fn from_ptr (ptr: *mut bindings::module ) ->ThisModule {ThisModule (ptr) } + +} +/// Calculates the offset of a field from the beginning of the struct it belongs to. +/// +/// # Example +/// +/// ``` +/// # use kernel::prelude::*; +/// # use kernel::offset_of; +/// struct Test { +/// a: u64, +/// b: u32, +/// } +/// +/// assert_eq!(offset_of!(Test, b), 8); +/// ``` +#[macro_export] +macro_rules! offset_of { + ($type:ty, $($f:tt)*) => {{ +let tmp = core::mem::MaybeUninit ::<$type>::uninit (); +let outer = tmp.as_ptr (); +// To avoid warnings when nesting `unsafe` blocks. +#[allow(unused_unsafe)] +// SAFETY: The pointer is valid and aligned, just not initialised; `addr_of` ensures that +// we don't actually read from `outer` (which would be UB) nor create an intermediate +// reference. +let inner =unsafe { core::ptr::addr_of!((*outer).$($f)*) }as *const u8 ; +// To avoid warnings when nesting `unsafe` blocks. +#[allow(unused_unsafe)] +// SAFETY: The two pointers are within the same allocation block. +unsafe { inner.offset_from (outeras *const u8 ) } + }} }/// Produces a pointer to an object from a pointer to one of its fields.
Signature · 1 line
2.40.1
PATCH 12/80 drop: Add crate::pr_warn declaration
In reply to Ariel Miculas--- rust/kernel/fs.rs | 1 - 1 file changed, 1 deletion(-)
rust/kernel/fs.rs +0 -1
@@ -17,7 +17,6 @@use core::{ cell::UnsafeCell , marker::PhantomData , marker::PhantomPinned , ops::Deref , pin::Pin , ptr, }; -use macros::vtable;pub mod param;
Signature · 1 line
2.40.1
Re: [PATCH 12/80] drop: Add crate::pr_warn declaration
Quoted 4 lines · Ariel Miculas
--- rust/kernel/fs.rs | 1 - 1 file changed, 1 deletion(-)
Empty patch, probably due to rebasing fun? :)
Cheers,
Miguel
Re: [PATCH 12/80] drop: Add crate::pr_warn declaration
Yes, I should have dropped this commit, that's why it says "drop" in the message.
Regards,
Ariel
From: Miguel Ojeda <[email protected]>
Sent: Friday, June 9, 2023 12:29 PM
To: Ariel Miculas (amiculas) <[email protected]>
Cc: [email protected] <[email protected]>
Subject: Re: [PATCH 12/80] drop: Add crate::pr_warn declaration
Quoted 4 lines · Ariel Miculas
---
rust/kernel/fs.rs| 1 -1 file changed, 1 deletion(-)
Empty patch, probably due to rebasing fun? :)
Cheers,
Miguel
PATCH 13/80 rust: kernel: rename from_kernel_errno to from_errno
In reply to Ariel Miculas--- rust/kernel/file.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
rust/kernel/file.rs +1 -1
@@ -214,7 +214,7 @@ pub fn new(flags: u32) -> Result<Self> {// SAFETY: FFI call, there are no safety requirements on `flags`. let fd =unsafe { bindings::get_unused_fd_flags (flags) };if fd <0 { -return Err (Error ::from_kernel_errno (fd)); +return Err (Error ::from_errno (fd)); }Ok (Self { fd: fdas _ }) }
Signature · 1 line
2.40.1
Re: [PATCH 13/80] rust: kernel: rename from_kernel_errno to from_errno
Quoted 2 lines · Ariel Miculas
Signed-off-by Ariel Miculas <[email protected]>
This one should also be applied to the original patch.
Cheers,
Miguel
PATCH 14/80 rust: kernel: Rename from_pointer to from_foreing and into_pointer to into_foreign
In reply to Ariel Miculas--- rust/kernel/file.rs | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-)
rust/kernel/file.rs +12 -12
@@ -330,7 +330,7 @@ impl<A: OpenAdapter<T::OpenData>, T: Operations> OperationsVtable<A, T> {let fileref =unsafe {File ::from_ptr (file) };// SAFETY: `arg` was previously returned by `A::convert` and must // be a valid non-null pointer. -let ptr =T ::open (unsafe { &*arg }, fileref)?.into_pointer (); +let ptr =T ::open (unsafe { &*arg }, fileref)?.into_foreign ();// SAFETY: The C contract guarantees that `private_data` is available // for implementers of the file operations (no other C code accesses // it), so we know that there are no concurrent threads/CPUs accessing
@@ -349,7 +349,7 @@ impl<A: OpenAdapter<T::OpenData>, T: Operations> OperationsVtable<A, T> {from_kernel_result! {let mut data =unsafe {UserSlicePtr ::new (bufas *mut core::ffi::c_void , len).writer () };// SAFETY: `private_data` was initialised by `open_callback` with a value returned by -// `T::Data::into_pointer`. `T::Data::from_pointer` is only called by the +// `T::Data::into_foreign`. `T::Data::from_foreign` is only called by the // `release` callback, which the C API guarantees that will be called only when all // references to `file` have been released, so we know it can't be called while this // function is running.
@@ -376,7 +376,7 @@ impl<A: OpenAdapter<T::OpenData>, T: Operations> OperationsVtable<A, T> {let file =unsafe { (*iocb).ki_filp };let offset =unsafe { (*iocb).ki_pos };// SAFETY: `private_data` was initialised by `open_callback` with a value returned by -// `T::Data::into_pointer`. `T::Data::from_pointer` is only called by the +// `T::Data::into_foreign`. `T::Data::from_foreign` is only called by the // `release` callback, which the C API guarantees that will be called only when all // references to `file` have been released, so we know it can't be called while this // function is running.
@@ -397,7 +397,7 @@ impl<A: OpenAdapter<T::OpenData>, T: Operations> OperationsVtable<A, T> {from_kernel_result! {let mut data =unsafe {UserSlicePtr ::new (bufas *mut core::ffi::c_void , len).reader () };// SAFETY: `private_data` was initialised by `open_callback` with a value returned by -// `T::Data::into_pointer`. `T::Data::from_pointer` is only called by the +// `T::Data::into_foreign`. `T::Data::from_foreign` is only called by the // `release` callback, which the C API guarantees that will be called only when all // references to `file` have been released, so we know it can't be called while this // function is running.
@@ -424,7 +424,7 @@ impl<A: OpenAdapter<T::OpenData>, T: Operations> OperationsVtable<A, T> {let file =unsafe { (*iocb).ki_filp };let offset =unsafe { (*iocb).ki_pos };// SAFETY: `private_data` was initialised by `open_callback` with a value returned by -// `T::Data::into_pointer`. `T::Data::from_pointer` is only called by the +// `T::Data::into_foreign`. `T::Data::from_foreign` is only called by the // `release` callback, which the C API guarantees that will be called only when all // references to `file` have been released, so we know it can't be called while this // function is running.
@@ -441,7 +441,7 @@ impl<A: OpenAdapter<T::OpenData>, T: Operations> OperationsVtable<A, T> { file: *mut bindings::file, ) -> core::ffi::c_int {let ptr = mem::replace (unsafe { &mut (*file).private_data }, ptr::null_mut ()); -T ::release (unsafe {T ::Data ::from_pointer (ptras _ ) },unsafe { +T ::release (unsafe {T ::Data ::from_foreign (ptras _ ) },unsafe {File ::from_ptr (file) });0
@@ -460,7 +460,7 @@ impl<A: OpenAdapter<T::OpenData>, T: Operations> OperationsVtable<A, T> { _ => return Err(EINVAL), };// SAFETY: `private_data` was initialised by `open_callback` with a value returned by -// `T::Data::into_pointer`. `T::Data::from_pointer` is only called by the +// `T::Data::into_foreign`. `T::Data::from_foreign` is only called by the // `release` callback, which the C API guarantees that will be called only when all // references to `file` have been released, so we know it can't be called while this // function is running.
@@ -477,7 +477,7 @@ impl<A: OpenAdapter<T::OpenData>, T: Operations> OperationsVtable<A, T> { ) -> core::ffi::c_long {from_kernel_result! {// SAFETY: `private_data` was initialised by `open_callback` with a value returned by -// `T::Data::into_pointer`. `T::Data::from_pointer` is only called by the +// `T::Data::into_foreign`. `T::Data::from_foreign` is only called by the // `release` callback, which the C API guarantees that will be called only when all // references to `file` have been released, so we know it can't be called while this // function is running.
@@ -495,7 +495,7 @@ impl<A: OpenAdapter<T::OpenData>, T: Operations> OperationsVtable<A, T> { ) -> core::ffi::c_long {from_kernel_result! {// SAFETY: `private_data` was initialised by `open_callback` with a value returned by -// `T::Data::into_pointer`. `T::Data::from_pointer` is only called by the +// `T::Data::into_foreign`. `T::Data::from_foreign` is only called by the // `release` callback, which the C API guarantees that will be called only when all // references to `file` have been released, so we know it can't be called while this // function is running.
@@ -512,7 +512,7 @@ impl<A: OpenAdapter<T::OpenData>, T: Operations> OperationsVtable<A, T> { ) -> core::ffi::c_int {from_kernel_result! {// SAFETY: `private_data` was initialised by `open_callback` with a value returned by -// `T::Data::into_pointer`. `T::Data::from_pointer` is only called by the +// `T::Data::into_foreign`. `T::Data::from_foreign` is only called by the // `release` callback, which the C API guarantees that will be called only when all // references to `file` have been released, so we know it can't be called while this // function is running.
@@ -540,7 +540,7 @@ impl<A: OpenAdapter<T::OpenData>, T: Operations> OperationsVtable<A, T> {let end = end.try_into ()?;let datasync = datasync !=0 ;// SAFETY: `private_data` was initialised by `open_callback` with a value returned by -// `T::Data::into_pointer`. `T::Data::from_pointer` is only called by the +// `T::Data::into_foreign`. `T::Data::from_foreign` is only called by the // `release` callback, which the C API guarantees that will be called only when all // references to `file` have been released, so we know it can't be called while this // function is running.
@@ -555,7 +555,7 @@ impl<A: OpenAdapter<T::OpenData>, T: Operations> OperationsVtable<A, T> { wait: *mut bindings::poll_table_struct, ) -> bindings::__poll_t {// SAFETY: `private_data` was initialised by `open_callback` with a value returned by -// `T::Data::into_pointer`. `T::Data::from_pointer` is only called by the `release` +// `T::Data::into_foreign`. `T::Data::from_foreign` is only called by the `release` // callback, which the C API guarantees that will be called only when all references to // `file` have been released, so we know it can't be called while this function is running. let f =unsafe {T ::Data ::borrow ((*file).private_data) };
Signature · 1 line
2.40.1
Re: [PATCH 14/80] rust: kernel: Rename from_pointer to from_foreing and into_pointer to into_foreign
Quoted 2 lines · Ariel Miculas
Signed-off-by Ariel Miculas <[email protected]>
This one should be squashed into the patch that adds the code too.
Cheers,
Miguel
PATCH 15/80 rust: kernel: add count_paren_items macro, needed by define_fs_params macro
In reply to Ariel MiculasChanges:
* add count_paren_items in rust/kernel/driver.rs, used by define_fs_params macro in rust/kernel/fs/param.rs
* add missing uring_cmd_iopoll in file_operations VTABLE
* fix accesses to the vm_flags field in vm_area_struct
* fix init's signature in rust/kernel/fs.rs
* import module_fs in samples/rust/rust_fs.rs
--- rust/bindings/bindings_helper.h | 1 + rust/kernel/driver.rs | 28 ++++++++++++++++++++++++++++ rust/kernel/file.rs | 1 + rust/kernel/fs.rs | 2 +- rust/kernel/lib.rs | 3 ++- rust/kernel/mm.rs | 4 ++-- samples/rust/rust_fs.rs | 1 + 7 files changed, 36 insertions(+), 4 deletions(-) create mode 100644 rust/kernel/driver.rs
rust/bindings/bindings_helper.h +1 -0
@@ -16,6 +16,7 @@#include <linux/poll.h> #include <linux/uio.h> #include <linux/uaccess.h> +#include <linux/delay.h> /* `bindgen` gets confused at certain things. */ const gfp_t BINDINGS_GFP_KERNEL =GFP_KERNEL ;
rust/kernel/driver.rs new file +28 -0
@@ -0,0 +1,28 @@ +// SPDX-License-Identifier: GPL-2.0 + +//! Generic support for drivers of different buses (e.g., PCI, Platform, Amba, etc.). +//! +//! Each bus/subsystem is expected to implement [`DriverOps`], which allows drivers to register +//! using the [`Registration`] class. + +/// Counts the number of parenthesis-delimited, comma-separated items. +/// +/// # Examples +/// +/// ``` +/// # use kernel::count_paren_items; +/// +/// assert_eq!(0, count_paren_items!()); +/// assert_eq!(1, count_paren_items!((A))); +/// assert_eq!(1, count_paren_items!((A),)); +/// assert_eq!(2, count_paren_items!((A), (B))); +/// assert_eq!(2, count_paren_items!((A), (B),)); +/// assert_eq!(3, count_paren_items!((A), (B), (C))); +/// assert_eq!(3, count_paren_items!((A), (B), (C),)); +/// ``` +#[macro_export] +macro_rules! count_paren_items { + (($($item:tt)*), $($remaining:tt)*) => {1 + $crate::count_paren_items!($($remaining)*) }; + (($($item:tt)*)) => {1 }; + () => {0 }; +}
rust/kernel/file.rs +1 -0
@@ -637,6 +637,7 @@ impl<A: OpenAdapter<T::OpenData>, T: Operations> OperationsVtable<A, T> {None }, uring_cmd: None, + uring_cmd_iopoll: None, write_iter: if T::HAS_WRITE {Some (Self ::write_iter_callback) }else {
rust/kernel/fs.rs +1 -1
@@ -1263,7 +1263,7 @@ pub struct Module<T: Type> { }impl <T :Type +Sync >crate ::Module for Module <T > { -fn init (_name: &'staticCStr , module: &'staticThisModule ) ->Result <Self > { +fn init (module: &'staticThisModule ) ->Result <Self > {let mut reg =Pin ::from (Box ::try_new (Registration ::new ())?); reg.as_mut ().register ::<T >(module)?;Ok (Self {
rust/kernel/lib.rs +2 -1
@@ -21,6 +21,7 @@#![feature(receiver_trait)] #![feature(unsize)] #![feature(const_mut_refs)] +#![feature(duration_constants)] // Ensure conditional compilation based on the kernel configuration works; // otherwise we may silently break things like initcall handling.
@@ -36,6 +37,7 @@mod build_assert;pub mod cred;pub mod delay; +pub mod driver;pub mod error;pub mod file;pub mod fs;
@@ -97,7 +99,6 @@ impl ThisModule {pub const unsafe fn from_ptr (ptr: *mut bindings::module ) ->ThisModule {ThisModule (ptr) } - }/// Calculates the offset of a field from the beginning of the struct it belongs to. ///
rust/kernel/mm.rs +2 -2
@@ -38,7 +38,7 @@ pub(crate) unsafe fn from_ptr(vma: *mut bindings::vm_area_struct) -> Self {/// The possible flags are a combination of the constants in [`flags`]. pub fn flags (&self) ->usize {// SAFETY: `self.vma` is valid by the type invariants. -unsafe { (*self.vma).vm_flagsas _ } +unsafe { (*self.vma).__bindgen_anon_1.vm_flagsas _ } }/// Sets the flags associated with the virtual memory area.
@@ -46,7 +46,7 @@ pub fn flags(&self) -> usize {/// The possible flags are a combination of the constants in [`flags`]. pub fn set_flags (&mut self, flags:usize ) {// SAFETY: `self.vma` is valid by the type invariants. -unsafe { (*self.vma).vm_flags = flagsas _ }; +unsafe { (*self.vma).__bindgen_anon_1.vm_flags = flagsas _ }; }/// Returns the start address of the virtual memory area.
samples/rust/rust_fs.rs +1 -0
@@ -2,6 +2,7 @@//! Rust file system sample. +use kernel::module_fs;use kernel::prelude::*;use kernel::{c_str, file, fs, io_buffer::IoBufferWriter };
Signature · 1 line
2.40.1
PATCH 16/80 rust: helpers: add missing rust helper 'alloc_pages'
In reply to Ariel Miculas--- rust/helpers.c | 6 ++++++ 1 file changed, 6 insertions(+)
rust/helpers.c +6 -0
@@ -198,6 +198,12 @@ void *rust_helper_alloc_inode_sb(struct super_block *sb, }EXPORT_SYMBOL_GPL (rust_helper_alloc_inode_sb); +struct page *rust_helper_alloc_pages (gfp_t gfp_mask,unsigned int order) +{ +return alloc_pages (gfp_mask, order); +} +EXPORT_SYMBOL_GPL (rust_helper_alloc_pages); +/* * We use `bindgen`'s `--size_t-is-usize` option to bind the C `size_t` type * as the Rust `usize` type, so we can use it in contexts where Rust
Signature · 1 line
2.40.1
Re: [PATCH 16/80] rust: helpers: add missing rust helper 'alloc_pages'
Quoted 2 lines · Ariel Miculas
Signed-off-by Ariel Miculas <[email protected]>
I think this could be added to the patch where the user/caller is. Otherwise, the commit message should explain the need for it.
Cheers,
Miguel
PATCH 17/80 kernel: configs: add qemu-busybox-min.config
In reply to Ariel Miculas--- arch/x86/configs/qemu-busybox-min.config | 11 +++++ kernel/configs/qemu-busybox-min.config | 56 ++++++++++++++++++++++++ 2 files changed, 67 insertions(+) create mode 100644 arch/x86/configs/qemu-busybox-min.config create mode 100644 kernel/configs/qemu-busybox-min.config
arch/x86/configs/qemu-busybox-min.config new file +11 -0
@@ -0,0 +1,11 @@ +CONFIG_64BIT=y +CONFIG_ACPI=y + +CONFIG_SERIAL_8250=y +CONFIG_SERIAL_8250_CONSOLE=y + +CONFIG_HYPERVISOR_GUEST=y +CONFIG_PVH=y + +CONFIG_CMDLINE_BOOL=y +CONFIG_CMDLINE="console=ttyS0 nokaslr rdinit=/sbin/init"
kernel/configs/qemu-busybox-min.config new file +56 -0
56 lines not on this page · raw message
Signature · 1 line
2.40.1
Re: [PATCH 17/80] kernel: configs: add qemu-busybox-min.config
Quoted 2 lines · Ariel Miculas
Signed-off-by Ariel Miculas <[email protected]>
These originally came from Wedson (Cc'd).
Cheers,
Miguel
PATCH 18/80 rust: kernel: format the rust code
In reply to Ariel Miculas--- rust/kernel/file.rs | 3 ++- rust/kernel/fs/param.rs | 2 +- rust/kernel/iov_iter.rs | 2 +- rust/kernel/mm.rs | 2 +- rust/kernel/pages.rs | 4 ++-- rust/kernel/user_ptr.rs | 2 +- 6 files changed, 8 insertions(+), 7 deletions(-)
rust/kernel/file.rs +2 -1
@@ -14,9 +14,10 @@ iov_iter::IovIter, mm, sync::CondVar, + types::ARef, + types::AlwaysRefCounted, types::ForeignOwnable, user_ptr::{UserSlicePtr, UserSlicePtrReader, UserSlicePtrWriter}, - types::ARef, types::AlwaysRefCounted, };use core::convert::{TryFrom ,TryInto };use core::{cell::UnsafeCell , marker, mem, ptr};
rust/kernel/fs/param.rs +1 -1
@@ -4,7 +4,7 @@//! //! C headers: [`include/linux/fs_parser.h`](../../../../../include/linux/fs_parser.h) -use crate ::{bindings, file, fs, str::CStr , error::Result }; +use crate ::{bindings, error::Result , file, fs, str::CStr };use core::{marker::PhantomData , ptr};/// The value of a file system parameter.
rust/kernel/iov_iter.rs +1 -1
@@ -7,8 +7,8 @@use crate ::{ bindings, error::code::*, - io_buffer::{IoBufferReader ,IoBufferWriter }, error::Result , + io_buffer::{IoBufferReader ,IoBufferWriter }, };/// Wraps the kernel's `struct iov_iter`.
rust/kernel/mm.rs +1 -1
@@ -4,7 +4,7 @@//! //! C header: [`include/linux/mm.h`](../../../../include/linux/mm.h) -use crate ::{bindings, pages, error::to_result, error::Result }; +use crate ::{bindings, error::to_result, error::Result , pages};/// Virtual memory. pub mod virt {
rust/kernel/pages.rs +2 -2
@@ -5,8 +5,8 @@//! TODO: This module is a work in progress. use crate ::{ - bindings, error::code::*, io_buffer::IoBufferReader , user_ptr::UserSlicePtrReader , error::Result , -PAGE_SIZE , + bindings, error::code::*, error::Result , io_buffer::IoBufferReader , + user_ptr::UserSlicePtrReader ,PAGE_SIZE , };use core::{marker::PhantomData , ptr};
rust/kernel/user_ptr.rs +1 -1
@@ -7,8 +7,8 @@use crate ::{ bindings, error::code::*, - io_buffer::{IoBufferReader ,IoBufferWriter }, error::Result , + io_buffer::{IoBufferReader ,IoBufferWriter }, };use alloc::vec::Vec ;
Signature · 1 line
2.40.1
Re: [PATCH 18/80] rust: kernel: format the rust code
Quoted 2 lines · Ariel Miculas
Signed-off-by Ariel Miculas <[email protected]>
This should be done by the commits that change the respective code.
Cheers,
Miguel
PATCH 19/80 samples: puzzlefs: add initial puzzlefs sample, copied from rust_fs.rs
In reply to Ariel Miculas--- samples/rust/Kconfig | 10 ++++ samples/rust/Makefile | 1 + samples/rust/puzzlefs.rs | 105 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 116 insertions(+) create mode 100644 samples/rust/puzzlefs.rs
samples/rust/Kconfig +10 -0
@@ -40,6 +40,16 @@ config SAMPLE_RUST_FS If unsure, say N. +config SAMPLE_PUZZLEFS + tristate "Puzzlefs file system" + help + This option builds the Rust puzzlefs file system sample. + + To compile this as a module, choose M here: + the module will be called puzzlefs. + + If unsure, say N. + config SAMPLE_RUST_HOSTPROGS bool "Host programs" help
samples/rust/Makefile +1 -0
@@ -3,5 +3,6 @@ obj-$(CONFIG_SAMPLE_RUST_MINIMAL) += rust_minimal.o obj-$(CONFIG_SAMPLE_RUST_PRINT) += rust_print.o obj-$(CONFIG_SAMPLE_RUST_FS) += rust_fs.o +obj-$(CONFIG_SAMPLE_PUZZLEFS) += puzzlefs.o subdir-$(CONFIG_SAMPLE_RUST_HOSTPROGS) += hostprogs
samples/rust/puzzlefs.rs new file +105 -0
105 lines not on this page · raw message
Signature · 1 line
2.40.1
PATCH 20/80 kernel: configs: enable rust samples in rust.config
In reply to Ariel Miculas--- kernel/configs/rust.config | 10 ++++++++++ 1 file changed, 10 insertions(+)
kernel/configs/rust.config +10 -0
@@ -1 +1,11 @@ CONFIG_RUST=y + +CONFIG_MODULES=y +CONFIG_SAMPLES=y +CONFIG_SAMPLES_RUST=y +CONFIG_SAMPLE_RUST_MINIMAL=m +CONFIG_SAMPLE_RUST_PRINT=m +CONFIG_SAMPLE_RUST_FS=m +CONFIG_SAMPLE_PUZZLEFS=m +CONFIG_SAMPLE_RUST_HOSTPROGS=y +
Signature · 1 line
2.40.1
Re: [PATCH 20/80] kernel: configs: enable rust samples in rust.config
Quoted 2 lines · Ariel Miculas
Signed-off-by Ariel Miculas <[email protected]>
I think the intention of rust.config is to enable the Rust support, not everything related to Rust.
So it would be better to have another file to enable samples, or one to enable as much as possible (e.g. for testing/CI), etc.
Cheers,
Miguel
PATCH 22/80 rust: proc-macro2: add SPDX License Identifiers
In reply to Ariel MiculasFrom: Miguel Ojeda <[email protected]>
--- rust/proc-macro2/detection.rs | 2 ++ rust/proc-macro2/fallback.rs | 2 ++ rust/proc-macro2/lib.rs | 2 ++ rust/proc-macro2/marker.rs | 2 ++ rust/proc-macro2/parse.rs | 2 ++ rust/proc-macro2/rcvec.rs | 2 ++ rust/proc-macro2/wrapper.rs | 2 ++ 7 files changed, 14 insertions(+)
rust/proc-macro2/detection.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use core::sync::atomic::{AtomicUsize ,Ordering };use std::sync::Once ;
rust/proc-macro2/fallback.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use crate ::parse::{self,Cursor };use crate ::rcvec::{RcVec ,RcVecBuilder ,RcVecIntoIter ,RcVecMut };use crate ::{Delimiter ,Spacing ,TokenTree };
rust/proc-macro2/lib.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! [![github]](https://github.com/dtolnay/proc-macro2) [![crates-io]](https://crates.io/crates/proc-macro2) [![docs-rs]](crate) //! //! [github]: https://img.shields.io/badge/github-8da0cb?style=for-the-badge&labelColor=555555&logo=github
rust/proc-macro2/marker.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use core::marker::PhantomData ;use std::panic::{RefUnwindSafe ,UnwindSafe };use std::rc::Rc ;
rust/proc-macro2/parse.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use crate ::fallback::{ is_ident_continue, is_ident_start,Group ,LexError ,Literal ,Span ,TokenStream ,TokenStreamBuilder ,
rust/proc-macro2/rcvec.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use core::mem;use core::slice;use std::rc::Rc ;
rust/proc-macro2/wrapper.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use crate ::detection::inside_proc_macro;use crate ::{fallback,Delimiter ,Punct ,Spacing ,TokenTree };use core::fmt::{self,Debug ,Display };
Signature · 1 line
2.40.1
PATCH 23/80 rust: proc-macro2: remove `unicode_ident` dependency
In reply to Ariel MiculasFrom: Miguel Ojeda <[email protected]>
The proc-macro2 crate depends on the unicode-ident crate to determine whether characters have the XID_Start or XID_Continue properties according to Unicode Standard Annex #31.
However, we only need ASCII identifiers in the kernel, thus we can simplify the check and remove completely that dependency.
--- rust/proc-macro2/fallback.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
rust/proc-macro2/fallback.rs +2 -2
@@ -724,11 +724,11 @@ pub fn set_span(&mut self, span: Span) { }pub (crate )fn is_ident_start (c:char ) ->bool { - c =='_' || unicode_ident::is_xid_start (c) + c =='_' || c.is_ascii_alphabetic () }pub (crate )fn is_ident_continue (c:char ) ->bool { - unicode_ident::is_xid_continue (c) + c =='_' || c.is_ascii_alphanumeric () }fn validate_ident (string: &str , raw:bool ) {
Signature · 1 line
2.40.1
PATCH 24/80 rust: quote: import crate
In reply to Ariel MiculasFrom: Miguel Ojeda <[email protected]>
This is a subset of the Rust quote crate, version 1.0.21, licensed under "Apache-2.0 OR MIT", from:
https://github.com/dtolnay/quote/raw/1.0.21/src
The files are copied as-is, with no modifications whatsoever (not even adding the SPDX identifiers).
For copyright details, please see:
https://github.com/dtolnay/quote/blob/1.0.21/README.md#license https://github.com/dtolnay/quote/blob/1.0.21/LICENSE-APACHE https://github.com/dtolnay/quote/blob/1.0.21/LICENSE-MIT
The next patch modifies these files as needed for use within the kernel. This patch split allows reviewers to double-check the import and to clearly see the differences introduced.
The following script may be used to verify the contents:
for path in $(cd rust/quote/ && find . -type f -name '*.rs'); do
curl --silent --show-error --location \
https://github.com/dtolnay/quote/raw/1.0.21/src/$path \
| diff --unified rust/quote/$path - && echo $path: OK
done--- rust/quote/ext.rs | 110 +++ rust/quote/format.rs | 168 ++++ rust/quote/ident_fragment.rs | 86 ++ rust/quote/lib.rs | 1434 ++++++++++++++++++++++++++++++++++ rust/quote/runtime.rs | 438 +++++++++++ rust/quote/spanned.rs | 43 + rust/quote/to_tokens.rs | 209 +++++ 7 files changed, 2488 insertions(+) create mode 100644 rust/quote/ext.rs create mode 100644 rust/quote/format.rs create mode 100644 rust/quote/ident_fragment.rs create mode 100644 rust/quote/lib.rs create mode 100644 rust/quote/runtime.rs create mode 100644 rust/quote/spanned.rs create mode 100644 rust/quote/to_tokens.rs
rust/quote/ext.rs new file +110 -0
@@ -0,0 +1,110 @@ +use super ::ToTokens ; +use core::iter; +use proc_macro2::{TokenStream ,TokenTree }; + +/// TokenStream extension trait with methods for appending tokens. +/// +/// This trait is sealed and cannot be implemented outside of the `quote` crate. +pub trait TokenStreamExt : private::Sealed { +/// For use by `ToTokens` implementations. +/// +/// Appends the token specified to this list of tokens. +fn append <U >(&mut self, token:U ) +where +U :Into <TokenTree >; + +/// For use by `ToTokens` implementations. +/// +/// ``` +/// # use quote::{quote, TokenStreamExt, ToTokens}; +/// # use proc_macro2::TokenStream; +/// # +/// struct X; +/// +/// impl ToTokens for X { +/// fn to_tokens(&self, tokens: &mut TokenStream) { +/// tokens.append_all(&[true, false]); +/// } +/// } +/// +/// let tokens = quote!(#X); +/// assert_eq!(tokens.to_string(), "true false"); +/// ``` +fn append_all <I >(&mut self, iter:I ) +where +I :IntoIterator , +I ::Item :ToTokens ; + +/// For use by `ToTokens` implementations. +/// +/// Appends all of the items in the iterator `I`, separated by the tokens +/// `U`. +fn append_separated <I ,U >(&mut self, iter:I , op:U ) +where +I :IntoIterator , +I ::Item :ToTokens , +U :ToTokens ; + +/// For use by `ToTokens` implementations. +/// +/// Appends all tokens in the iterator `I`, appending `U` after each +/// element, including after the last element of the iterator. +fn append_terminated <I ,U >(&mut self, iter:I , term:U ) +where +I :IntoIterator , +I ::Item :ToTokens , +U :ToTokens ; +} + +impl TokenStreamExt for TokenStream { +fn append <U >(&mut self, token:U ) +where +U :Into <TokenTree >, + { + self.extend (iter::once (token.into ())); + } + +fn append_all <I >(&mut self, iter:I ) +where +I :IntoIterator , +I ::Item :ToTokens , + { +for tokenin iter { + token.to_tokens (self); + } + } + +fn append_separated <I ,U >(&mut self, iter:I , op:U ) +where +I :IntoIterator , +I ::Item :ToTokens , +U :ToTokens , + { +for (i, token)in iter.into_iter ().enumerate () { +if i >0 { + op.to_tokens (self); + } + token.to_tokens (self); + } + } + +fn append_terminated <I ,U >(&mut self, iter:I , term:U ) +where +I :IntoIterator , +I ::Item :ToTokens , +U :ToTokens , + { +for tokenin iter { + token.to_tokens (self); + term.to_tokens (self); + } + } +} + +mod private { +use proc_macro2::TokenStream ; + +pub trait Sealed {} + +impl Sealed for TokenStream {} +}
rust/quote/format.rs new file +168 -0
168 lines not on this page · raw message
rust/quote/ident_fragment.rs new file +86 -0
86 lines not on this page · raw message
rust/quote/lib.rs new file +1434 -0
1,434 lines not on this page · raw message
rust/quote/runtime.rs new file +438 -0
438 lines not on this page · raw message
rust/quote/spanned.rs new file +43 -0
43 lines not on this page · raw message
rust/quote/to_tokens.rs new file +209 -0
209 lines not on this page · raw message
Signature · 1 line
2.40.1
PATCH 25/80 rust: quote: add SPDX License Identifiers
In reply to Ariel MiculasFrom: Miguel Ojeda <[email protected]>
--- rust/quote/ext.rs | 2 ++ rust/quote/format.rs | 2 ++ rust/quote/ident_fragment.rs | 2 ++ rust/quote/lib.rs | 2 ++ rust/quote/runtime.rs | 2 ++ rust/quote/spanned.rs | 2 ++ rust/quote/to_tokens.rs | 2 ++ 7 files changed, 14 insertions(+)
rust/quote/ext.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use super ::ToTokens ;use core::iter;use proc_macro2::{TokenStream ,TokenTree };
rust/quote/format.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +/// Formatting macro for constructing `Ident`s. /// /// <br>
rust/quote/ident_fragment.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use core::fmt;use proc_macro2::{Ident ,Span };use std::borrow::Cow ;
rust/quote/lib.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! [![github]](https://github.com/dtolnay/quote) [![crates-io]](https://crates.io/crates/quote) [![docs-rs]](https://docs.rs/quote) //! //! [github]: https://img.shields.io/badge/github-8da0cb?style=for-the-badge&labelColor=555555&logo=github
rust/quote/runtime.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use crate ::{IdentFragment ,ToTokens ,TokenStreamExt };use core::fmt;use core::iter;
rust/quote/spanned.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use crate ::ToTokens ;use proc_macro2::{Span ,TokenStream };
rust/quote/to_tokens.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use super ::TokenStreamExt ;use core::iter;use proc_macro2::{Group ,Ident ,Literal ,Punct ,Span ,TokenStream ,TokenTree };
Signature · 1 line
2.40.1
PATCH 27/80 rust: syn: add SPDX License Identifiers
In reply to Ariel MiculasFrom: Miguel Ojeda <[email protected]>
--- rust/syn/attr.rs | 2 ++ rust/syn/await.rs | 2 ++ rust/syn/bigint.rs | 2 ++ rust/syn/buffer.rs | 2 ++ rust/syn/custom_keyword.rs | 2 ++ rust/syn/custom_punctuation.rs | 2 ++ rust/syn/data.rs | 2 ++ rust/syn/derive.rs | 2 ++ rust/syn/discouraged.rs | 2 ++ rust/syn/error.rs | 2 ++ rust/syn/export.rs | 2 ++ rust/syn/expr.rs | 2 ++ rust/syn/ext.rs | 2 ++ rust/syn/file.rs | 2 ++ rust/syn/gen/clone.rs | 2 ++ rust/syn/gen/debug.rs | 2 ++ rust/syn/gen/eq.rs | 2 ++ rust/syn/gen/fold.rs | 2 ++ rust/syn/gen/hash.rs | 2 ++ rust/syn/gen/visit.rs | 2 ++ rust/syn/gen/visit_mut.rs | 2 ++ rust/syn/gen_helper.rs | 2 ++ rust/syn/generics.rs | 2 ++ rust/syn/group.rs | 2 ++ rust/syn/ident.rs | 2 ++ rust/syn/item.rs | 2 ++ rust/syn/lib.rs | 2 ++ rust/syn/lifetime.rs | 2 ++ rust/syn/lit.rs | 2 ++ rust/syn/lookahead.rs | 2 ++ rust/syn/mac.rs | 2 ++ rust/syn/macros.rs | 2 ++ rust/syn/op.rs | 2 ++ rust/syn/parse.rs | 2 ++ rust/syn/parse_macro_input.rs | 2 ++ rust/syn/parse_quote.rs | 2 ++ rust/syn/pat.rs | 2 ++ rust/syn/path.rs | 2 ++ rust/syn/print.rs | 2 ++ rust/syn/punctuated.rs | 2 ++ rust/syn/reserved.rs | 2 ++ rust/syn/sealed.rs | 2 ++ rust/syn/span.rs | 2 ++ rust/syn/spanned.rs | 2 ++ rust/syn/stmt.rs | 2 ++ rust/syn/thread.rs | 2 ++ rust/syn/token.rs | 2 ++ rust/syn/tt.rs | 2 ++ rust/syn/ty.rs | 2 ++ rust/syn/verbatim.rs | 2 ++ rust/syn/whitespace.rs | 2 ++ 51 files changed, 102 insertions(+)
rust/syn/attr.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use super ::*;use crate ::punctuated::Punctuated ;use proc_macro2::TokenStream ;
rust/syn/await.rs +2 -0
@@ -1,2 +1,4 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +// See include!("await.rs") in token.rs. export_token_macro! {[await ]}
rust/syn/bigint.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use std::ops::{AddAssign ,MulAssign };// For implementing base10_digits() accessor on LitInt.
rust/syn/buffer.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! A stably addressed token buffer supporting efficient traversal based on a //! cheaply copyable cursor. //!
rust/syn/custom_keyword.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +/// Define a type that supports parsing and printing a given identifier as if it /// were a keyword. ///
rust/syn/custom_punctuation.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +/// Define a type that supports parsing and printing a multi-character symbol /// as if it were a punctuation token. ///
rust/syn/data.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use super ::*;use crate ::punctuated::Punctuated ;
rust/syn/derive.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use super ::*;use crate ::punctuated::Punctuated ;
rust/syn/discouraged.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! Extensions to the parsing API with niche applicability. use super ::*;
rust/syn/error.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +#[cfg(feature = "parsing" )] use crate ::buffer::Cursor ;use crate ::thread::ThreadBound ;
rust/syn/export.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +pub use std::clone::Clone ;pub use std::cmp::{Eq ,PartialEq };pub use std::default::Default ;
rust/syn/expr.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use super ::*;use crate ::punctuated::Punctuated ;#[cfg(feature = "full" )]
rust/syn/ext.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! Extension traits to provide parsing methods on foreign types. //! //! *This module is available only if Syn is built with the `"parsing"` feature.*
rust/syn/file.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use super ::*;ast_struct! {
rust/syn/gen/clone.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +// This file is @generated by syn-internal-codegen. // It is not intended for manual editing.
rust/syn/gen/debug.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +// This file is @generated by syn-internal-codegen. // It is not intended for manual editing.
rust/syn/gen/eq.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +// This file is @generated by syn-internal-codegen. // It is not intended for manual editing.
rust/syn/gen/fold.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +// This file is @generated by syn-internal-codegen. // It is not intended for manual editing.
rust/syn/gen/hash.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +// This file is @generated by syn-internal-codegen. // It is not intended for manual editing.
rust/syn/gen/visit.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +// This file is @generated by syn-internal-codegen. // It is not intended for manual editing.
rust/syn/gen/visit_mut.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +// This file is @generated by syn-internal-codegen. // It is not intended for manual editing.
rust/syn/gen_helper.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +#[cfg(feature = "fold" )] pub mod fold {use crate ::fold::Fold ;
rust/syn/generics.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use super ::*;use crate ::punctuated::{Iter ,IterMut ,Punctuated };#[cfg(all(feature = "printing" , feature = "extra-traits" ))]
rust/syn/group.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use crate ::error::Result ;use crate ::parse::ParseBuffer ;use crate ::token;
rust/syn/ident.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +#[cfg(feature = "parsing" )] use crate ::buffer::Cursor ;#[cfg(feature = "parsing" )]
rust/syn/item.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use super ::*;use crate ::derive::{Data ,DataEnum ,DataStruct ,DataUnion ,DeriveInput };use crate ::punctuated::Punctuated ;
rust/syn/lib.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! [![github]](https://github.com/dtolnay/syn) [![crates-io]](https://crates.io/crates/syn) [![docs-rs]](crate) //! //! [github]: https://img.shields.io/badge/github-8da0cb?style=for-the-badge&labelColor=555555&logo=github
rust/syn/lifetime.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use proc_macro2::{Ident ,Span };use std::cmp::Ordering ;use std::fmt::{self,Display };
rust/syn/lit.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +#[cfg(feature = "parsing" )] use crate ::lookahead;#[cfg(feature = "parsing" )]
rust/syn/lookahead.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use crate ::buffer::Cursor ;use crate ::error::{self,Error };use crate ::sealed::lookahead::Sealed ;
rust/syn/mac.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use super ::*;use crate ::token::{Brace ,Bracket ,Paren };use proc_macro2::TokenStream ;
rust/syn/macros.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT + #[cfg_attr( not(any(feature = "full", feature = "derive")), allow(unknown_lints, unused_macro_rules)
rust/syn/op.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +ast_enum! {/// A binary operator: `+`, `+=`, `&`. ///
rust/syn/parse.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! Parsing interface for parsing a token stream into a syntax tree node. //! //! Parsing in Syn is built on parser functions that take in a [`ParseStream`]
rust/syn/parse_macro_input.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +/// Parse the input TokenStream of a macro, triggering a compile error if the /// tokens fail to parse. ///
rust/syn/parse_quote.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +/// Quasi-quotation macro that accepts input like the [`quote!`] macro but uses /// type inference to figure out a return type for those tokens. ///
rust/syn/pat.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use super ::*;use crate ::punctuated::Punctuated ;use proc_macro2::TokenStream ;
rust/syn/path.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use super ::*;use crate ::punctuated::Punctuated ;
rust/syn/print.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use proc_macro2::TokenStream ;use quote::ToTokens ;
rust/syn/punctuated.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! A punctuated sequence of syntax tree nodes separated by punctuation. //! //! Lots of things in Rust are punctuated sequences.
rust/syn/reserved.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +// Type for a syntax tree node that is reserved for future use. // // For example ExprReference contains a field `raw` of type Reserved. If `&raw
rust/syn/sealed.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +#[cfg(feature = "parsing" )] pub mod lookahead {pub trait Sealed :Copy {}
rust/syn/span.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use proc_macro2::Span ;pub trait IntoSpans <S > {
rust/syn/spanned.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! A trait that can provide the `Span` of the complete contents of a syntax //! tree node. //!
rust/syn/stmt.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use super ::*;ast_struct! {
rust/syn/thread.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use std::fmt::{self,Debug };use std::thread::{self,ThreadId };
rust/syn/token.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! Tokens representing Rust punctuation, keywords, and delimiters. //! //! The type names in this module can be difficult to keep straight, so we
rust/syn/tt.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use proc_macro2::{Delimiter ,TokenStream ,TokenTree };use std::hash::{Hash ,Hasher };
rust/syn/ty.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use super ::*;use crate ::punctuated::Punctuated ;use proc_macro2::TokenStream ;
rust/syn/verbatim.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use crate ::parse::{ParseBuffer ,ParseStream };use proc_macro2::{Delimiter ,TokenStream };use std::cmp::Ordering ;
rust/syn/whitespace.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +pub fn skip (mut s: &str ) -> &str { 'skip:while !s.is_empty () {let byte = s.as_bytes ()[0 ];
Signature · 1 line
2.40.1
PATCH 28/80 rust: syn: remove `unicode-ident` dependency
In reply to Ariel MiculasFrom: Miguel Ojeda <[email protected]>
The syn crate depends on the unicode-ident crate
to determine whether characters have the XID_Start or XID_Continue properties according to Unicode Standard Annex #31.
However, we only need ASCII identifiers in the kernel, thus we can simplify the check and remove completely that dependency.
--- rust/syn/ident.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
rust/syn/ident.rs +2 -2
@@ -91,11 +91,11 @@ fn from(token: Token![_]) -> Ident {pub fn xid_ok (symbol: &str ) ->bool {let mut chars = symbol.chars ();let first = chars.next ().unwrap (); -if !(first =='_' || unicode_ident::is_xid_start (first)) { +if !(first =='_' || first.is_ascii_alphabetic ()) {return false ; }for chin chars { -if !unicode_ident::is_xid_continue (ch) { +if !(ch =='_' || ch.is_ascii_alphanumeric ()) {return false ; } }
Signature · 1 line
2.40.1
PATCH 30/80 rust: serde: add `no_fp_fmt_parse` support
In reply to Ariel MiculasFrom: Miguel Ojeda <[email protected]>
We do not have formatting for floating point in the kernel, thus simple compile out all that.
Probably we should name it differently, more similar to integer128.
--- rust/serde/de/ignored_any.rs | 1 + rust/serde/de/impls.rs | 2 ++ rust/serde/de/mod.rs | 16 ++++++++++++++-- rust/serde/de/value.rs | 2 ++ rust/serde/private/ser.rs | 4 ++++ rust/serde/ser/fmt.rs | 8 ++++++-- rust/serde/ser/impls.rs | 2 ++ rust/serde/ser/mod.rs | 2 ++ 8 files changed, 33 insertions(+), 4 deletions(-)
rust/serde/de/ignored_any.rs +1 -0
@@ -152,6 +152,7 @@ fn visit_u128<E>(self, x: u128) -> Result<Self::Value, E> { } } +#[cfg(not(no_fp_fmt_parse))] #[inline] fn visit_f64 <E >(self, x:f64 ) ->Result <Self ::Value ,E > {let _ = x;
rust/serde/de/impls.rs +2 -0
@@ -350,6 +350,7 @@ fn $visit<E>(self, v: $ty) -> Result<Self::Value, E>uint_to_self! (u32 :visit_u32u64 :visit_u64); } +#[cfg(not(no_fp_fmt_parse))] impl_deserialize_num! {f32 , deserialize_f32num_self! (f32 :visit_f32);
@@ -358,6 +359,7 @@ fn $visit<E>(self, v: $ty) -> Result<Self::Value, E>num_as_self! (u8 :visit_u8u16 :visit_u16u32 :visit_u32u64 :visit_u64); } +#[cfg(not(no_fp_fmt_parse))] impl_deserialize_num! {f64 , deserialize_f64num_self! (f64 :visit_f64);
rust/serde/de/mod.rs +14 -2
@@ -344,6 +344,7 @@ pub enum Unexpected<'a> {/// The input contained a floating point `f32` or `f64` that was not /// expected. +#[cfg(not(no_fp_fmt_parse))] Float(f64),/// The input contained a `char` that was not expected.
@@ -400,6 +401,7 @@ fn fmt(&self, formatter: &mut fmt::Formatter) -> fmt::Result { Bool(b) => write!(formatter, "boolean `{}`", b), Unsigned(i) => write!(formatter, "integer `{}`", i), Signed(i) => write!(formatter, "integer `{}`", i), +#[cfg(not(no_fp_fmt_parse))] Float(f) => write!(formatter, "floating point `{}`", f), Char(c) => write!(formatter, "character `{}`", c), Str(s) => write!(formatter, "string {:?}", s),
@@ -996,12 +998,20 @@ fn deserialize_u128<V>(self, visitor: V) -> Result<V::Value, Self::Error>/// Hint that the `Deserialize` type is expecting a `f32` value. fn deserialize_f32 <V >(self, visitor:V ) ->Result <V ::Value ,Self ::Error >where -V :Visitor <'de>; +V :Visitor <'de> + { +let _ = visitor; +Err (Error ::custom ("f32 is not supported" )) + }/// Hint that the `Deserialize` type is expecting a `f64` value. fn deserialize_f64 <V >(self, visitor:V ) ->Result <V ::Value ,Self ::Error >where -V :Visitor <'de>; +V :Visitor <'de> + { +let _ = visitor; +Err (Error ::custom ("f64 is not supported" )) + }/// Hint that the `Deserialize` type is expecting a `char` value. fn deserialize_char <V >(self, visitor:V ) ->Result <V ::Value ,Self ::Error >
@@ -1446,6 +1456,7 @@ fn visit_u128<E>(self, v: u128) -> Result<Self::Value, E>/// The default implementation forwards to [`visit_f64`]. /// /// [`visit_f64`]: #method.visit_f64 +#[cfg(not(no_fp_fmt_parse))] fn visit_f32 <E >(self, v:f32 ) ->Result <Self ::Value ,E >where E :Error ,
@@ -1456,6 +1467,7 @@ fn visit_f32<E>(self, v: f32) -> Result<Self::Value, E>/// The input contains an `f64`. /// /// The default implementation fails with a type error. +#[cfg(not(no_fp_fmt_parse))] fn visit_f64 <E >(self, v:f64 ) ->Result <Self ::Value ,E >where E :Error ,
rust/serde/de/value.rs +2 -0
@@ -298,7 +298,9 @@ fn fmt(&self, formatter: &mut fmt::Formatter) -> fmt::Result {primitive_deserializer! (u16 ,"a `u16`." ,U16Deserializer , visit_u16);primitive_deserializer! (u64 ,"a `u64`." ,U64Deserializer , visit_u64);primitive_deserializer! (usize ,"a `usize`." ,UsizeDeserializer , visit_u64as u64 ); +#[cfg(not(no_fp_fmt_parse))] primitive_deserializer! (f32 ,"an `f32`." ,F32Deserializer , visit_f32); +#[cfg(not(no_fp_fmt_parse))] primitive_deserializer! (f64 ,"an `f64`." ,F64Deserializer , visit_f64);primitive_deserializer! (char ,"a `char`." ,CharDeserializer , visit_char);
rust/serde/private/ser.rs +4 -0
@@ -46,6 +46,7 @@ struct TaggedSerializer<S> {enum Unsupported {Boolean ,Integer , +#[cfg(not(no_fp_fmt_parse))] Float ,Char ,String ,
@@ -64,6 +65,7 @@ fn fmt(&self, formatter: &mut fmt::Formatter) -> fmt::Result {match *self {Unsupported ::Boolean => formatter.write_str ("a boolean" ),Unsupported ::Integer => formatter.write_str ("an integer" ), +#[cfg(not(no_fp_fmt_parse))] Unsupported ::Float => formatter.write_str ("a float" ),Unsupported ::Char => formatter.write_str ("a char" ),Unsupported ::String => formatter.write_str ("a string" ),
@@ -150,10 +152,12 @@ fn serialize_u64(self, _: u64) -> Result<Self::Ok, Self::Error> {Err (self.bad_type (Unsupported ::Integer )) } +#[cfg(not(no_fp_fmt_parse))] fn serialize_f32 (self, _:f32 ) ->Result <Self ::Ok ,Self ::Error > {Err (self.bad_type (Unsupported ::Float )) } +#[cfg(not(no_fp_fmt_parse))] fn serialize_f64 (self, _:f64 ) ->Result <Self ::Ok ,Self ::Error > {Err (self.bad_type (Unsupported ::Float )) }
rust/serde/ser/fmt.rs +6 -2
@@ -55,13 +55,17 @@ impl<'a, 'b> Serializer for &'a mut fmt::Formatter<'b> { serialize_u16: u16, serialize_u32: u32, serialize_u64: u64, - serialize_f32: f32, - serialize_f64: f64, serialize_char: char, serialize_str: &str, serialize_unit_struct: &'static str, } +#[cfg(not(no_fp_fmt_parse))] +fmt_primitives! { + serialize_f32:f32 , + serialize_f64:f64 , + } +serde_if_integer128! {fmt_primitives! { serialize_i128:i128 ,
rust/serde/ser/impls.rs +2 -0
@@ -29,7 +29,9 @@ fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>primitive_impl! (u16 , serialize_u16);primitive_impl! (u32 , serialize_u32);primitive_impl! (u64 , serialize_u64); +#[cfg(not(no_fp_fmt_parse))] primitive_impl! (f32 , serialize_f32); +#[cfg(not(no_fp_fmt_parse))] primitive_impl! (f64 , serialize_f64);primitive_impl! (char , serialize_char);
rust/serde/ser/mod.rs +2 -0
@@ -644,6 +644,7 @@ fn serialize_u128(self, v: u128) -> Result<Self::Ok, Self::Error> {/// } /// } /// ``` +#[cfg(not(no_fp_fmt_parse))] fn serialize_f32 (self, v:f32 ) ->Result <Self ::Ok ,Self ::Error >;/// Serialize an `f64` value.
@@ -662,6 +663,7 @@ fn serialize_u128(self, v: u128) -> Result<Self::Ok, Self::Error> {/// } /// } /// ``` +#[cfg(not(no_fp_fmt_parse))] fn serialize_f64 (self, v:f64 ) ->Result <Self ::Ok ,Self ::Error >;/// Serialize a character.
Signature · 1 line
2.40.1
PATCH 31/80 rust: serde: add SPDX License Identifiers
In reply to Ariel MiculasFrom: Miguel Ojeda <[email protected]>
--- rust/serde/de/format.rs | 2 ++ rust/serde/de/ignored_any.rs | 2 ++ rust/serde/de/impls.rs | 2 ++ rust/serde/de/mod.rs | 2 ++ rust/serde/de/seed.rs | 2 ++ rust/serde/de/utf8.rs | 2 ++ rust/serde/de/value.rs | 2 ++ rust/serde/integer128.rs | 2 ++ rust/serde/lib.rs | 2 ++ rust/serde/macros.rs | 2 ++ rust/serde/private/de.rs | 2 ++ rust/serde/private/doc.rs | 2 ++ rust/serde/private/mod.rs | 2 ++ rust/serde/private/ser.rs | 2 ++ rust/serde/private/size_hint.rs | 2 ++ rust/serde/ser/fmt.rs | 2 ++ rust/serde/ser/impls.rs | 2 ++ rust/serde/ser/impossible.rs | 2 ++ rust/serde/ser/mod.rs | 2 ++ rust/serde/std_error.rs | 2 ++ 20 files changed, 40 insertions(+)
rust/serde/de/format.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use lib::fmt::{self,Write };use lib::str;
rust/serde/de/ignored_any.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use lib::*;use de::{
rust/serde/de/impls.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use lib::*;use de::{
rust/serde/de/mod.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! Generic data structure deserialization framework. //! //! The two most important traits in this module are [`Deserialize`] and
rust/serde/de/seed.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use de::{Deserialize ,DeserializeSeed ,Deserializer };/// A DeserializeSeed helper for implementing deserialize_in_place Visitors.
rust/serde/de/utf8.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use lib::*;const TAG_CONT :u8 =0b1000_0000 ;
rust/serde/de/value.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! Building blocks for deserializing basic values using the `IntoDeserializer` //! trait. //!
rust/serde/integer128.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +/// Conditional compilation depending on whether Serde is built with support for /// 128-bit integers. ///
rust/serde/lib.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! # Serde //! //! Serde is a framework for ***ser***ializing and ***de***serializing Rust data
rust/serde/macros.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +// Super explicit first paragraph because this shows up at the top level and // trips up people who are just looking for basic Serialize / Deserialize // documentation.
rust/serde/private/de.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use lib::*;use de::value::{BorrowedBytesDeserializer ,BytesDeserializer };
rust/serde/private/doc.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +// Used only by Serde doc tests. Not public API. use lib::*;
rust/serde/private/mod.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +#[cfg(not(no_serde_derive))] pub mod de;#[cfg(not(no_serde_derive))]
rust/serde/private/ser.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use lib::*;use ser::{self,Impossible ,Serialize ,SerializeMap ,SerializeStruct ,Serializer };
rust/serde/private/size_hint.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use lib::*;pub fn from_bounds <I >(iter: &I ) ->Option <usize >
rust/serde/ser/fmt.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use lib::*;use ser::{Error ,Impossible ,Serialize ,Serializer };
rust/serde/ser/impls.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use lib::*;use ser::{Error ,Serialize ,SerializeTuple ,Serializer };
rust/serde/ser/impossible.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! This module contains `Impossible` serializer and its implementations. use lib::*;
rust/serde/ser/mod.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! Generic data structure serialization framework. //! //! The two most important traits in this module are [`Serialize`] and
rust/serde/std_error.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use lib::{Debug ,Display };/// Either a re-export of std::error::Error or a new identical trait, depending
Signature · 1 line
2.40.1
Re: [PATCH 31/80] rust: serde: add SPDX License Identifiers
Quoted 31 lines · Ariel Miculas
From: Miguel Ojeda <[email protected]>
Signed-off-by Miguel Ojeda <[email protected]>--- rust/serde/de/format.rs | 2 ++ rust/serde/de/ignored_any.rs | 2 ++ rust/serde/de/impls.rs | 2 ++ rust/serde/de/mod.rs | 2 ++ rust/serde/de/seed.rs | 2 ++ rust/serde/de/utf8.rs | 2 ++ rust/serde/de/value.rs | 2 ++ rust/serde/integer128.rs | 2 ++ rust/serde/lib.rs | 2 ++ rust/serde/macros.rs | 2 ++ rust/serde/private/de.rs | 2 ++ rust/serde/private/doc.rs | 2 ++ rust/serde/private/mod.rs | 2 ++ rust/serde/private/ser.rs | 2 ++ rust/serde/private/size_hint.rs | 2 ++ rust/serde/ser/fmt.rs | 2 ++ rust/serde/ser/impls.rs | 2 ++ rust/serde/ser/impossible.rs | 2 ++ rust/serde/ser/mod.rs | 2 ++ rust/serde/std_error.rs | 2 ++ 20 files changed, 40 insertions(+)rust/serde/de/format.rs +0
-0
rust/serde/de/format.rs line 4
Quoted 4 lines
rust/serde/de/format.rs +2
-0@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use lib::fmt::{self,Write };use lib::str;
Hang on, are we applying our own local modifications to code vendored from an outside repository?
We gotta stop doing this - this turns our version into an unsupported fork. What do we then do when we need to merge in security updates from the upstream repository?
The only sane thing to do is keep the vendored code in lockstep with the upstream repository - i.e. always a direct copy of a specific commit. Any changes for the kernel need to be applied to the upstream repository (which could be an intermediate repository, not necessarly _the_ upstream repository; the point is, that way there's a proper revision history for _that_ repo, and changes can get merged upstream).
Otherwise, we're setting ourselves up for a maintainence nightmare...
Greg KH
maintainerRe: [PATCH 31/80] rust: serde: add SPDX License Identifiers
Quoted 2 lines · Kent Overstreet
Hang on, are we applying our own local modifications to code vendored from an outside repository?
For adding a single license line, I think it's going to be ok, those are trivial to handle :)
Quoted 3 lines
We gotta stop doing this - this turns our version into an unsupported fork. What do we then do when we need to merge in security updates from the upstream repository?
Same as we do for other parts of the kernel that are copies of other codebases, we merge them in as needed.
thanks,
greg k-h
Re: [PATCH 31/80] rust: serde: add SPDX License Identifiers
Quoted 6 lines · Greg KH
Quoted 2 lines · Kent Overstreet
Hang on, are we applying our own local modifications to code vendored from an outside repository?
For adding a single license line, I think it's going to be ok, those are trivial to handle :)
They'll still be lost when the package is updated from upstream. Just do a proper upstream patch - I see no reason why SPDX identifiers wouldn't be accepted by the host project.
Quoted 7 lines
Quoted 4 lines
We gotta stop doing this - this turns our version into an unsupported fork. What do we then do when we need to merge in security updates from the upstream repository?
Same as we do for other parts of the kernel that are copies of other codebases, we merge them in as needed.
Yeah, and that workflow still leaves something to be desired...
Greg KH
maintainerRe: [PATCH 31/80] rust: serde: add SPDX License Identifiers
Quoted 11 lines · Kent Overstreet
Quoted 6 lines · Greg KH
Quoted 2 lines · Kent Overstreet
Hang on, are we applying our own local modifications to code vendored from an outside repository?
For adding a single license line, I think it's going to be ok, those are trivial to handle :)
They'll still be lost when the package is updated from upstream. Just do a proper upstream patch - I see no reason why SPDX identifiers wouldn't be accepted by the host project.
Great, then submit them upstream also, but for now, when added to the kernel tree, they need to be present.
thanks,
greg k-h
Re: [PATCH 31/80] rust: serde: add SPDX License Identifiers
In reply to Ariel MiculasQuoted 17 lines · Ariel Miculas
From: Miguel Ojeda <[email protected]>
Signed-off-by Miguel Ojeda <[email protected]>--- rust/serde/de/format.rs | 2 ++ rust/serde/de/ignored_any.rs | 2 ++ rust/serde/de/impls.rs | 2 ++ rust/serde/de/mod.rs | 2 ++ rust/serde/de/seed.rs | 2 ++ rust/serde/de/utf8.rs | 2 ++ rust/serde/de/value.rs | 2 ++ rust/serde/integer128.rs | 2 ++ rust/serde/lib.rs | 2 ++ rust/serde/macros.rs | 2 ++ rust/serde/private/de.rs | 2 ++ rust/serde/private/doc.rs | 2 ++ rust/serde/private/mod.rs | 2 ++
Quoted 8 lines
rust/serde/private/ser.rs | 2 ++ rust/serde/private/size_hint.rs | 2 ++ rust/serde/ser/fmt.rs | 2 ++ rust/serde/ser/impls.rs | 2 ++ rust/serde/ser/impossible.rs | 2 ++ rust/serde/ser/mod.rs | 2 ++ rust/serde/std_error.rs | 2 ++ 20 files changed, 40 insertions(+)
Separately from that - are we sure we want to be pulling Serde into the kernel?
Don't get me wrong, Serde is amazing - it's highly ergonomic, and makes a lot of compatibility issues just completely go away. It's impresssively well done.
But it's a full json parser (are you using it in json mode here? it supports multiple formats), and object size is going to be a consideration. In particular, the way it works is by generating (via procedural macros) encode/decode functions for every type you use #[derive] an encoder/decoder for, so depending on how that was implemented that's going to add up.
I would need to see text size numbers - the bloaty tool might be useful here: https://github.com/google/bloaty
I'd give serious consideration to Cap'n proto instead, it's not as ergonomic as Serde (the get/set interface is not ideal, ask me for more details on how this could be fixed) but it's lower overhead than Serde.
Re: [PATCH 31/80] rust: serde: add SPDX License Identifiers
Quoted 27 lines · Kent Overstreet
Quoted 17 lines · Ariel Miculas
From: Miguel Ojeda <[email protected]>
Signed-off-by Miguel Ojeda <[email protected]>--- rust/serde/de/format.rs | 2 ++ rust/serde/de/ignored_any.rs | 2 ++ rust/serde/de/impls.rs | 2 ++ rust/serde/de/mod.rs | 2 ++ rust/serde/de/seed.rs | 2 ++ rust/serde/de/utf8.rs | 2 ++ rust/serde/de/value.rs | 2 ++ rust/serde/integer128.rs | 2 ++ rust/serde/lib.rs | 2 ++ rust/serde/macros.rs | 2 ++ rust/serde/private/de.rs | 2 ++ rust/serde/private/doc.rs | 2 ++ rust/serde/private/mod.rs | 2 ++Quoted 8 lines
rust/serde/private/ser.rs | 2 ++ rust/serde/private/size_hint.rs | 2 ++ rust/serde/ser/fmt.rs | 2 ++ rust/serde/ser/impls.rs | 2 ++ rust/serde/ser/impossible.rs | 2 ++ rust/serde/ser/mod.rs | 2 ++ rust/serde/std_error.rs | 2 ++ 20 files changed, 40 insertions(+)
Quoted 8 lines
Separately from that - are we sure we want to be pulling Serde into the kernel?
Don't get me wrong, Serde is amazing - it's highly ergonomic, and makes a lot of compatibility issues just completely go away. It's impresssively well done.
But it's a full json parser (are you using it in json mode here? it
I think the json capability lives in serde_json, so this is actually not a full json parser.
BR Andreas
Quoted 14 lines
supports multiple formats), and object size is going to be a consideration. In particular, the way it works is by generating (via procedural macros) encode/decode functions for every type you use #[derive] an encoder/decoder for, so depending on how that was implemented that's going to add up.
I would need to see text size numbers - the bloaty tool might be useful here: https://github.com/google/bloaty
I'd give serious consideration to Cap'n proto instead, it's not as ergonomic as Serde (the get/set interface is not ideal, ask me for more details on how this could be fixed) but it's lower overhead than Serde.
Re: [PATCH 31/80] rust: serde: add SPDX License Identifiers
Quoted 33 lines · Andreas Hindborg (Samsung)
Quoted 30 lines · Kent Overstreet
Quoted 17 lines · Ariel Miculas
From: Miguel Ojeda <[email protected]>
Signed-off-by Miguel Ojeda <[email protected]>--- rust/serde/de/format.rs | 2 ++ rust/serde/de/ignored_any.rs | 2 ++ rust/serde/de/impls.rs | 2 ++ rust/serde/de/mod.rs | 2 ++ rust/serde/de/seed.rs | 2 ++ rust/serde/de/utf8.rs | 2 ++ rust/serde/de/value.rs | 2 ++ rust/serde/integer128.rs | 2 ++ rust/serde/lib.rs | 2 ++ rust/serde/macros.rs | 2 ++ rust/serde/private/de.rs | 2 ++ rust/serde/private/doc.rs | 2 ++ rust/serde/private/mod.rs | 2 ++Quoted 8 lines
rust/serde/private/ser.rs | 2 ++ rust/serde/private/size_hint.rs | 2 ++ rust/serde/ser/fmt.rs | 2 ++ rust/serde/ser/impls.rs | 2 ++ rust/serde/ser/impossible.rs | 2 ++ rust/serde/ser/mod.rs | 2 ++ rust/serde/std_error.rs | 2 ++ 20 files changed, 40 insertions(+)Separately from that - are we sure we want to be pulling Serde into the kernel?
Quoted 8 lines
Quoted 5 lines
Don't get me wrong, Serde is amazing - it's highly ergonomic, and makes a lot of compatibility issues just completely go away. It's impresssively well done.
But it's a full json parser (are you using it in json mode here? it
I think the json capability lives in
serde_json, so this is actually not a full json parser.
Good to know
I don't think I'm the only one who's curious though - what's serde being used for here? On disk data structures of some kind?
I'm probably not the only filesystem author eagerly waiting for the day when I can stop using carefully packed C structs for this, with all the sharp edges that entails...
Re: [PATCH 31/80] rust: serde: add SPDX License Identifiers
Quoted 44 lines · Kent Overstreet
Quoted 33 lines · Andreas Hindborg (Samsung)
Quoted 30 lines · Kent Overstreet
Quoted 17 lines · Ariel Miculas
From: Miguel Ojeda <[email protected]>
Signed-off-by Miguel Ojeda <[email protected]>--- rust/serde/de/format.rs | 2 ++ rust/serde/de/ignored_any.rs | 2 ++ rust/serde/de/impls.rs | 2 ++ rust/serde/de/mod.rs | 2 ++ rust/serde/de/seed.rs | 2 ++ rust/serde/de/utf8.rs | 2 ++ rust/serde/de/value.rs | 2 ++ rust/serde/integer128.rs | 2 ++ rust/serde/lib.rs | 2 ++ rust/serde/macros.rs | 2 ++ rust/serde/private/de.rs | 2 ++ rust/serde/private/doc.rs | 2 ++ rust/serde/private/mod.rs | 2 ++Quoted 8 lines
rust/serde/private/ser.rs | 2 ++ rust/serde/private/size_hint.rs | 2 ++ rust/serde/ser/fmt.rs | 2 ++ rust/serde/ser/impls.rs | 2 ++ rust/serde/ser/impossible.rs | 2 ++ rust/serde/ser/mod.rs | 2 ++ rust/serde/std_error.rs | 2 ++ 20 files changed, 40 insertions(+)Separately from that - are we sure we want to be pulling Serde into the kernel?
Quoted 8 lines
Quoted 5 lines
Don't get me wrong, Serde is amazing - it's highly ergonomic, and makes a lot of compatibility issues just completely go away. It's impresssively well done.
But it's a full json parser (are you using it in json mode here? it
I think the json capability lives in
serde_json, so this is actually not a full json parser.
Quoted 8 lines
Good to know
I don't think I'm the only one who's curious though - what's serde being used for here? On disk data structures of some kind?
I'm probably not the only filesystem author eagerly waiting for the day when I can stop using carefully packed C structs for this, with all the sharp edges that entails...
serde, together with serde_cbor (added in another commit [3]), is used to decode the puzzlefs metadata (check the "format" section for the initial design [1]).
Sorry for the github links, but the patch series is incomplete because I didn't send them to [email protected], the entire "pull request" is available here [2].
https://github.com/anuvu/puzzlefs/pull/6#issue-801680988 [1]
https://github.com/Rust-for-Linux/linux/compare/rust-next...ariel-miculas:puzzlefs_rfc_v1
[2]
https://github.com/Rust-for-Linux/linux/commit/3fe9d58fbd89b0ed488d3150620eb07be17f1f16
[3]
Re: [PATCH 31/80] rust: serde: add SPDX License Identifiers
In reply to Kent OverstreetQuoted 3 lines · Kent Overstreet
Separately from that - are we sure we want to be pulling Serde into the kernel?
No, we are not -- see https://rust-for-linux.com/third-party-crates
(these have my SoBs because I provide an experiment branch at
https://github.com/Rust-for-Linux/linux/pull/1007, since some people wanted to use them, but it is not decided which ones to have).
Cheers,
Miguel
Re: [PATCH 31/80] rust: serde: add SPDX License Identifiers
In reply to Kent OverstreetQuoted 42 lines · Kent Overstreet
Quoted 17 lines · Ariel Miculas
From: Miguel Ojeda <[email protected]>
Signed-off-by Miguel Ojeda <[email protected]>--- rust/serde/de/format.rs | 2 ++ rust/serde/de/ignored_any.rs | 2 ++ rust/serde/de/impls.rs | 2 ++ rust/serde/de/mod.rs | 2 ++ rust/serde/de/seed.rs | 2 ++ rust/serde/de/utf8.rs | 2 ++ rust/serde/de/value.rs | 2 ++ rust/serde/integer128.rs | 2 ++ rust/serde/lib.rs | 2 ++ rust/serde/macros.rs | 2 ++ rust/serde/private/de.rs | 2 ++ rust/serde/private/doc.rs | 2 ++ rust/serde/private/mod.rs | 2 ++Quoted 8 lines
rust/serde/private/ser.rs | 2 ++ rust/serde/private/size_hint.rs | 2 ++ rust/serde/ser/fmt.rs | 2 ++ rust/serde/ser/impls.rs | 2 ++ rust/serde/ser/impossible.rs | 2 ++ rust/serde/ser/mod.rs | 2 ++ rust/serde/std_error.rs | 2 ++ 20 files changed, 40 insertions(+)Separately from that - are we sure we want to be pulling Serde into the kernel?
Don't get me wrong, Serde is amazing - it's highly ergonomic, and makes a lot of compatibility issues just completely go away. It's impresssively well done.
But it's a full json parser (are you using it in json mode here? it supports multiple formats), and object size is going to be a consideration. In particular, the way it works is by generating (via procedural macros) encode/decode functions for every type you use #[derive] an encoder/decoder for, so depending on how that was implemented that's going to add up.
Quoted 8 lines
I would need to see text size numbers - the bloaty tool might be useful here: https://github.com/google/bloaty
I'd give serious consideration to Cap'n proto instead, it's not as ergonomic as Serde (the get/set interface is not ideal, ask me for more details on how this could be fixed) but it's lower overhead than Serde.
Unfortunately bloaty seems broken [1], but here's the output:
```
$ bloaty puzzlefs_vmlinux -- vmlinux
FILE SIZE VM SIZE
-------------- --------------
+102% +921Ki [ = ] 0 .debug_info
+139% +849Ki [ = ] 0 .debug_str
+251% +574Ki [ = ] 0 .debug_loc
+116% +246Ki [ = ] 0 .strtab
+292% +226Ki [ = ] 0 .debug_pubtypes
+297% +200Ki [ = ] 0 .debug_pubnames
+115% +176Ki +115% +176Ki .rodata
+103% +160Ki [ = ] 0 .debug_line
+287% +152Ki [ = ] 0 .debug_ranges
+104% +44.9Ki [ = ] 0 .debug_frame
+103% +43.2Ki [ = ] 0 .symtab
+112% +24.0Ki +112% +24.0Ki __ksymtab_strings
+100% +5.31Ki [ = ] 0 .debug_abbrev
+100% +4.20Ki [ = ] 0 .debug_str_offsets
+108% +2.89Ki +108% +2.89Ki __ksymtab_gpl
+100% +2.13Ki [ = ] 0 .debug_addr
+100% +452 [ = ] 0 .debug_loclists
+129% +416 [ = ] 0 .debug_aranges
+0.0% +60 +0.0% +68 [8 Others]
+99% -64 +99% -64 [LOAD #3 [RWX]]
+96% -202Ki [ = ] 0 [Unmapped]
+104% +90.2Mi +101% +13.8Mi TOTAL
```
where puzzlefs_vmlinux is the kernel built with my puzzlefs changes
and vmlinux is the kernel built on the rust-next branch.
```
$ ls -lah puzzlefs_vmlinux
-rwxr-xr-x 1 amiculas amiculas 91M Jun 12 14:38 puzzlefs_vmlinux
~/work/linux rust-next*
$ ls -lah vmlinux
-rwxr-xr-x 1 amiculas amiculas 87M Jun 12 14:38 vmlinux
```It's a +4M difference between the two versions (3.35 M to be exact), but they are both debug builds (with CONFIG_DEBUG_KERNEL and CONFIG_KALLSYMS set).
If I should do additional comparisons, please let me know.
Re: [PATCH 31/80] rust: serde: add SPDX License Identifiers
In reply to Kent OverstreetQuoted 42 lines · Kent Overstreet
Quoted 17 lines · Ariel Miculas
From: Miguel Ojeda <[email protected]>
Signed-off-by Miguel Ojeda <[email protected]>--- rust/serde/de/format.rs | 2 ++ rust/serde/de/ignored_any.rs | 2 ++ rust/serde/de/impls.rs | 2 ++ rust/serde/de/mod.rs | 2 ++ rust/serde/de/seed.rs | 2 ++ rust/serde/de/utf8.rs | 2 ++ rust/serde/de/value.rs | 2 ++ rust/serde/integer128.rs | 2 ++ rust/serde/lib.rs | 2 ++ rust/serde/macros.rs | 2 ++ rust/serde/private/de.rs | 2 ++ rust/serde/private/doc.rs | 2 ++ rust/serde/private/mod.rs | 2 ++Quoted 8 lines
rust/serde/private/ser.rs | 2 ++ rust/serde/private/size_hint.rs | 2 ++ rust/serde/ser/fmt.rs | 2 ++ rust/serde/ser/impls.rs | 2 ++ rust/serde/ser/impossible.rs | 2 ++ rust/serde/ser/mod.rs | 2 ++ rust/serde/std_error.rs | 2 ++ 20 files changed, 40 insertions(+)Separately from that - are we sure we want to be pulling Serde into the kernel?
Don't get me wrong, Serde is amazing - it's highly ergonomic, and makes a lot of compatibility issues just completely go away. It's impresssively well done.
But it's a full json parser (are you using it in json mode here? it supports multiple formats), and object size is going to be a consideration. In particular, the way it works is by generating (via procedural macros) encode/decode functions for every type you use #[derive] an encoder/decoder for, so depending on how that was implemented that's going to add up.
Quoted 6 lines
I would need to see text size numbers - the bloaty tool might be useful here: https://github.com/google/bloaty
I'd give serious consideration to Cap'n proto instead, it's not as ergonomic as Serde (the get/set interface is not ideal, ask me for more details on how this could be fixed) but it's lower overhead than Serde.
I've been checking out Cap'n proto and indeed that get/set interface seems cumbersome, so how do you fix it?
Quoted 2 lines
Regards,
Ariel
Re: [PATCH 31/80] rust: serde: add SPDX License Identifiers
Quoted 44 lines · Ariel Miculas
Quoted 42 lines · Kent Overstreet
Quoted 17 lines · Ariel Miculas
From: Miguel Ojeda <[email protected]>
Signed-off-by Miguel Ojeda <[email protected]>--- rust/serde/de/format.rs | 2 ++ rust/serde/de/ignored_any.rs | 2 ++ rust/serde/de/impls.rs | 2 ++ rust/serde/de/mod.rs | 2 ++ rust/serde/de/seed.rs | 2 ++ rust/serde/de/utf8.rs | 2 ++ rust/serde/de/value.rs | 2 ++ rust/serde/integer128.rs | 2 ++ rust/serde/lib.rs | 2 ++ rust/serde/macros.rs | 2 ++ rust/serde/private/de.rs | 2 ++ rust/serde/private/doc.rs | 2 ++ rust/serde/private/mod.rs | 2 ++Quoted 8 lines
rust/serde/private/ser.rs | 2 ++ rust/serde/private/size_hint.rs | 2 ++ rust/serde/ser/fmt.rs | 2 ++ rust/serde/ser/impls.rs | 2 ++ rust/serde/ser/impossible.rs | 2 ++ rust/serde/ser/mod.rs | 2 ++ rust/serde/std_error.rs | 2 ++ 20 files changed, 40 insertions(+)Separately from that - are we sure we want to be pulling Serde into the kernel?
Don't get me wrong, Serde is amazing - it's highly ergonomic, and makes a lot of compatibility issues just completely go away. It's impresssively well done.
But it's a full json parser (are you using it in json mode here? it supports multiple formats), and object size is going to be a consideration. In particular, the way it works is by generating (via procedural macros) encode/decode functions for every type you use #[derive] an encoder/decoder for, so depending on how that was implemented that's going to add up.
Quoted 8 lines
Quoted 6 lines
I would need to see text size numbers - the bloaty tool might be useful here: https://github.com/google/bloaty
I'd give serious consideration to Cap'n proto instead, it's not as ergonomic as Serde (the get/set interface is not ideal, ask me for more details on how this could be fixed) but it's lower overhead than Serde.
I've been checking out Cap'n proto and indeed that get/set interface seems cumbersome, so how do you fix it?
One thing that makes it cumbersome is that any get can return an error:
a straightforward change would be to add a method that does all validation/bounds checking up front, and returns a new type with methods that can't return errors.
Kenton did it that way to avoid any O(n) parsing overhead up front; with messages that can be arbitrarily sized it's definitely an option that you want to have, especially when you won't necessarily be reading the entire thing. But in my experience most messages are short, and there's a reading-off-the-disk step or reading-from-the-network step, and you probably want to do other validation, so doing all validation up front simplifies error handling.
For the next step, what we really what we'd want is for cap'n proto to generate normal Rust types but with a #[repr] that matches the cap'n proto type layout.
Having the cap'n proto compiler emit #[repr(c)] types with the appropriate padding isn't enough because:
- integers are stored little endian, and need byte swabbing to access
- booleans are stored as bitfields, and Rust doesn't have native bitfields
Doing this right would require importing computed properties from Swift:
a computed property is something that looks like a struct field but requires get/set calls to access it, the compiler just knows how to emit them.
Miguel might be able to point us towards rustc people to talk to about this :) Having native language-level types which are directly serializable is amazing, as we've already seen with Serde. If we could skip the encode/decode step and just mmap a file and start walking objects - this could really change how we write and design filesystems, databases - even compilers. Debug info (e.g. dwarf) is all about serializing compiler internals, and a lot of the problems compiler people are tackling right now basically require doing this in a more principled way.
PATCH 33/80 rust: serde_derive: add SPDX License Identifiers
In reply to Ariel MiculasFrom: Miguel Ojeda <[email protected]>
--- rust/serde_derive/bound.rs | 2 ++ rust/serde_derive/de.rs | 2 ++ rust/serde_derive/dummy.rs | 2 ++ rust/serde_derive/fragment.rs | 2 ++ rust/serde_derive/internals/ast.rs | 2 ++ rust/serde_derive/internals/attr.rs | 2 ++ rust/serde_derive/internals/case.rs | 2 ++ rust/serde_derive/internals/check.rs | 2 ++ rust/serde_derive/internals/ctxt.rs | 2 ++ rust/serde_derive/internals/mod.rs | 2 ++ rust/serde_derive/internals/receiver.rs | 2 ++ rust/serde_derive/internals/respan.rs | 2 ++ rust/serde_derive/internals/symbol.rs | 2 ++ rust/serde_derive/lib.rs | 2 ++ rust/serde_derive/pretend.rs | 2 ++ rust/serde_derive/ser.rs | 2 ++ rust/serde_derive/this.rs | 2 ++ rust/serde_derive/try.rs | 2 ++ 18 files changed, 36 insertions(+)
rust/serde_derive/bound.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use std::collections::HashSet ;use syn;
rust/serde_derive/de.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use proc_macro2::{Literal ,Span ,TokenStream };use quote::ToTokens ;use syn::punctuated::Punctuated ;
rust/serde_derive/dummy.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use proc_macro2::{Ident ,TokenStream };use quote::format_ident;
rust/serde_derive/fragment.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use proc_macro2::TokenStream ;use quote::ToTokens ;use syn::token;
rust/serde_derive/internals/ast.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! A Serde ast, parsed from the Syn ast and ready to generate Rust code. use internals::attr;
rust/serde_derive/internals/attr.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use internals::symbol::*;use internals::{ungroup,Ctxt };use proc_macro2::{Spacing ,Span ,TokenStream ,TokenTree };
rust/serde_derive/internals/case.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! Code to convert the Rust-styled field/variant (e.g. `my_field`, `MyType`) to the //! case of the source (e.g. `my-field`, `MY_FIELD`).
rust/serde_derive/internals/check.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use internals::ast::{Container ,Data ,Field ,Style };use internals::attr::{Identifier ,TagType };use internals::{ungroup,Ctxt ,Derive };
rust/serde_derive/internals/ctxt.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use quote::ToTokens ;use std::cell::RefCell ;use std::fmt::Display ;
rust/serde_derive/internals/mod.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +pub mod ast;pub mod attr;
rust/serde_derive/internals/receiver.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use internals::respan::respan;use proc_macro2::Span ;use quote::ToTokens ;
rust/serde_derive/internals/respan.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use proc_macro2::{Group ,Span ,TokenStream ,TokenTree };pub (crate )fn respan (stream:TokenStream , span:Span ) ->TokenStream {
rust/serde_derive/internals/symbol.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use std::fmt::{self,Display };use syn::{Ident ,Path };
rust/serde_derive/lib.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! This crate provides Serde's two derive macros. //! //! ```edition2018
rust/serde_derive/pretend.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use proc_macro2::TokenStream ;use quote::format_ident;
rust/serde_derive/ser.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use proc_macro2::{Span ,TokenStream };use syn::spanned::Spanned ;use syn::{self,Ident ,Index ,Member };
rust/serde_derive/this.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use internals::ast::Container ;use syn::{Path ,PathArguments ,Token };
rust/serde_derive/try.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use proc_macro2::{Punct ,Spacing ,TokenStream };// None of our generated code requires the `From::from` error conversion
Signature · 1 line
2.40.1
PATCH 34/80 rust: Kbuild: enable `proc-macro2`, `quote`, `syn`, `serde` and `serde_derive`
In reply to Ariel MiculasFrom: Miguel Ojeda <[email protected]>
With all the new files in place from the new crates, this patch adds support for them in the build system.
Note that, if we decide to add third-party crates support to upstream, we would probably want to do it in a different way. But this is useful for playing around with these crates, experimenting, etc.
--- Makefile | 12 ++- rust/.gitignore | 1 + rust/Makefile | 194 ++++++++++++++++++++++++++++++++++++----- scripts/Makefile.build | 2 +- 4 files changed, 185 insertions(+), 24 deletions(-)
Makefile +11 -1
@@ -1606,7 +1606,7 @@ MRPROPER_FILES += include/config include/generated \ certs/x509.genkey \ vmlinux-gdb.py \ *.spec rpmbuild \ - rust/libmacros.so + rust/libmacros.so rust/libserde_derive.so # clean - Delete most, but leave enough to build external modules #
@@ -1848,8 +1848,18 @@ PHONY += rustfmt rustfmtcheck rustfmt: $(Q)find $(abs_srctree) -type f -name '*.rs' \ -o -path $(abs_srctree)/rust/alloc -prune \ + -o -path $(abs_srctree)/rust/proc-macro2 -prune \ + -o -path $(abs_srctree)/rust/quote -prune \ + -o -path $(abs_srctree)/rust/syn -prune \ + -o -path $(abs_srctree)/rust/serde -prune \ + -o -path $(abs_srctree)/rust/serde_derive -prune \ -o -path $(abs_objtree)/rust/test -prune \ | grep -Fv $(abs_srctree)/rust/alloc \ + | grep -Fv $(abs_srctree)/rust/proc-macro2 \ + | grep -Fv $(abs_srctree)/rust/quote \ + | grep -Fv $(abs_srctree)/rust/syn \ + | grep -Fv $(abs_srctree)/rust/serde \ + | grep -Fv $(abs_srctree)/rust/serde_derive \ | grep -Fv $(abs_objtree)/rust/test \ | grep -Fv generated \ | xargs $(RUSTFMT) $(rustfmt_flags)
rust/.gitignore +1 -0
@@ -1,5 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 +*.rlib bindings_generated.rs bindings_helpers_generated.rs uapi_generated.rs
rust/Makefile +172 -22
277 lines not on this page · raw message
scripts/Makefile.build +1 -1
@@ -284,7 +284,7 @@ rust_common_cmd = \ -Zallow-features=$(rust_allowed_features) \ -Zcrate-attr=no_std \ -Zcrate-attr='feature($(rust_allowed_features))' \ - --extern alloc --extern kernel \ + --extern alloc --extern kernel --extern serde --extern serde_derive \ --crate-type rlib -L $(objtree)/rust/ \ --crate-name $(basename $(notdir $@)) \ --emit=dep-info=$(depfile)
Signature · 1 line
2.40.1
PATCH 35/80 rust: test `serde` support
In reply to Ariel MiculasFrom: Miguel Ojeda <[email protected]>
A trivial example based on serde's `example-format' [1].
It contains both a in-kernel data format later used by the kernel module, as well as a local data format in the module.
The kernel module gives an output such as:
[ 0.801425] rust_serde: Rust serde sample (init)
[ 0.801634] rust_serde: original = S { a: (), b: false, c: true, d: () }
[ 0.802079] rust_serde: serialized = [2, 0, 1, 0, 1, 1, 0, 3]
[ 0.802506] rust_serde: deserialized = S { a: (), b: false, c: true, d: () }
[ 0.802718] rust_serde: serialized (local) = [2, 0, 1, 42, 1, 43, 0, 3]
[ 0.802895] rust_serde: deserialized (local) = S { a: (), b: false, c: true, d: () }
[ 0.808954] rust_serde: Rust serde sample (exit)Note that this is just a quick draft/hack to check the previous commits work. It is not intended to be merged at all.
--- rust/kernel/lib.rs | 1 + rust/kernel/test_serde.rs | 26 ++ rust/kernel/test_serde/de.rs | 442 ++++++++++++++++++++++ rust/kernel/test_serde/error.rs | 73 ++++ rust/kernel/test_serde/ser.rs | 469 ++++++++++++++++++++++++ samples/rust/Kconfig | 8 + samples/rust/Makefile | 1 + samples/rust/local_data_format/de.rs | 422 +++++++++++++++++++++ samples/rust/local_data_format/error.rs | 73 ++++ samples/rust/local_data_format/ser.rs | 443 ++++++++++++++++++++++ samples/rust/rust_serde.rs | 73 ++++ 11 files changed, 2031 insertions(+) create mode 100644 rust/kernel/test_serde.rs create mode 100644 rust/kernel/test_serde/de.rs create mode 100644 rust/kernel/test_serde/error.rs create mode 100644 rust/kernel/test_serde/ser.rs create mode 100644 samples/rust/local_data_format/de.rs create mode 100644 samples/rust/local_data_format/error.rs create mode 100644 samples/rust/local_data_format/ser.rs create mode 100644 samples/rust/rust_serde.rs
rust/kernel/lib.rs +1 -0
@@ -178,3 +178,4 @@ fn panic(info: &core::panic::PanicInfo<'_>) -> ! {/// /// [`PAGE_SHIFT`]: ../../../include/asm-generic/page.h pub const PAGE_SIZE :usize =1 << bindings::PAGE_SHIFT ; +pub mod test_serde;
rust/kernel/test_serde.rs new file +26 -0
@@ -0,0 +1,26 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT + +//! Test `serde`. +//! +//! It contains a data format used by the `rust_serde` sample, as well +//! as a quick check that `serde_derive` works in the `kernel` crate too. + +#![allow(missing_docs)] + +mod de; +mod error; +mod ser; + +pub use de::{from_bytes,Deserializer }; +pub use error::{Error ,Result }; +pub use ser::{to_vec,Serializer }; + +use serde_derive::{Deserialize ,Serialize }; + +#[derive( Serialize , Deserialize , Debug )] +pub struct S { + a: (), + b:bool , + c:bool , + d: (), +}
rust/kernel/test_serde/de.rs new file +442 -0
443 lines not on this page · raw message
rust/kernel/test_serde/error.rs new file +73 -0
73 lines not on this page · raw message
rust/kernel/test_serde/ser.rs new file +469 -0
469 lines not on this page · raw message
samples/rust/Kconfig +8 -0
@@ -48,6 +48,14 @@ config SAMPLE_PUZZLEFS To compile this as a module, choose M here: the module will be called puzzlefs. +config SAMPLE_RUST_SERDE + tristate "Serde" + help + This option builds the Rust `serde` sample. + + To compile this as a module, choose M here: + the module will be called rust_serde. + If unsure, say N. config SAMPLE_RUST_HOSTPROGS
samples/rust/Makefile +1 -0
@@ -4,5 +4,6 @@ obj-$(CONFIG_SAMPLE_RUST_MINIMAL) += rust_minimal.o obj-$(CONFIG_SAMPLE_RUST_PRINT) += rust_print.o obj-$(CONFIG_SAMPLE_RUST_FS) += rust_fs.o obj-$(CONFIG_SAMPLE_PUZZLEFS) += puzzlefs.o +obj-$(CONFIG_SAMPLE_RUST_SERDE) += rust_serde.o subdir-$(CONFIG_SAMPLE_RUST_HOSTPROGS) += hostprogs
samples/rust/local_data_format/de.rs new file +422 -0
422 lines not on this page · raw message
samples/rust/local_data_format/error.rs new file +73 -0
73 lines not on this page · raw message
samples/rust/local_data_format/ser.rs new file +443 -0
443 lines not on this page · raw message
samples/rust/rust_serde.rs new file +73 -0
73 lines not on this page · raw message
Signature · 1 line
2.40.1
PATCH 36/80 Add SAMPLE_RUST_SERDE in rust.config
In reply to Ariel Miculas--- kernel/configs/rust.config | 1 + 1 file changed, 1 insertion(+)
kernel/configs/rust.config +1 -0
@@ -8,4 +8,5 @@ CONFIG_SAMPLE_RUST_PRINT=m CONFIG_SAMPLE_RUST_FS=m CONFIG_SAMPLE_PUZZLEFS=m CONFIG_SAMPLE_RUST_HOSTPROGS=y +CONFIG_SAMPLE_RUST_SERDE=y
Signature · 1 line
2.40.1
PATCH 37/80 rust: kernel: fix compile errors after rebase to rust-next
In reply to Ariel Miculas--- rust/kernel/iov_iter.rs | 2 +- rust/kernel/mm.rs | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-)
rust/kernel/iov_iter.rs +1 -1
@@ -23,7 +23,7 @@ pub struct IovIter {impl IovIter {fn common_len (&self) ->usize {// SAFETY: `IovIter::ptr` is guaranteed to be valid by the type invariants. -unsafe { (*self.ptr).count } +unsafe { (*self.ptr).__bindgen_anon_2.__bindgen_anon_1.count } }/// Constructs a new [`struct iov_iter`] wrapper.
rust/kernel/mm.rs +4 -4
@@ -38,7 +38,7 @@ pub(crate) unsafe fn from_ptr(vma: *mut bindings::vm_area_struct) -> Self {/// The possible flags are a combination of the constants in [`flags`]. pub fn flags (&self) ->usize {// SAFETY: `self.vma` is valid by the type invariants. -unsafe { (*self.vma).__bindgen_anon_1.vm_flagsas _ } +unsafe { (*self.vma).__bindgen_anon_2.vm_flagsas _ } }/// Sets the flags associated with the virtual memory area.
@@ -46,19 +46,19 @@ pub fn flags(&self) -> usize {/// The possible flags are a combination of the constants in [`flags`]. pub fn set_flags (&mut self, flags:usize ) {// SAFETY: `self.vma` is valid by the type invariants. -unsafe { (*self.vma).__bindgen_anon_1.vm_flags = flagsas _ }; +unsafe { (*self.vma).__bindgen_anon_2.vm_flags = flagsas _ }; }/// Returns the start address of the virtual memory area. pub fn start (&self) ->usize {// SAFETY: `self.vma` is valid by the type invariants. -unsafe { (*self.vma).vm_startas _ } +unsafe { (*self.vma).__bindgen_anon_1.__bindgen_anon_1.vm_startas _ } }/// Returns the end address of the virtual memory area. pub fn end (&self) ->usize {// SAFETY: `self.vma` is valid by the type invariants. -unsafe { (*self.vma).vm_endas _ } +unsafe { (*self.vma).__bindgen_anon_1.__bindgen_anon_1.vm_endas _ } }/// Maps a single page at the given address within the virtual memory area.
Signature · 1 line
2.40.1
Re: [PATCH 37/80] rust: kernel: fix compile errors after rebase to rust-next
Quoted 2 lines · Ariel Miculas
Signed-off-by Ariel Miculas <[email protected]>
Not a big deal since it is an RFC, but please note that we want to keep all commits building (and passing docs, tests etc.).
If the patch is from somebody else (which may be the reason why you
split it), then ideally the original author would sign off on your
changes (or, even better, they can send the updated patch themselves),
potentially having you as Co-developed-by if they are bigger
changes, or you can otherwise use the [ ... ] notation in the
message to indicate what you changed (e.g. for small notes I use this
style: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f431c5c581fa).
Cheers,
Miguel
PATCH 39/80 rust: serde_cbor: add SPDX License Identifiers
In reply to Ariel Miculas--- rust/serde_cbor/de.rs | 2 ++ rust/serde_cbor/error.rs | 2 ++ rust/serde_cbor/lib.rs | 2 ++ rust/serde_cbor/read.rs | 2 ++ rust/serde_cbor/ser.rs | 2 ++ rust/serde_cbor/tags.rs | 2 ++ rust/serde_cbor/value/de.rs | 2 ++ rust/serde_cbor/value/mod.rs | 2 ++ rust/serde_cbor/value/ser.rs | 2 ++ rust/serde_cbor/write.rs | 2 ++ 10 files changed, 20 insertions(+)
rust/serde_cbor/de.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! Deserialization. use core::f32;
rust/serde_cbor/error.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! When serializing or deserializing CBOR goes wrong. use core::fmt;use core::result;
rust/serde_cbor/lib.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! CBOR and serialization. //! //! # Usage
rust/serde_cbor/read.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +#[cfg(feature = "alloc" )] use alloc::{vec, vec::Vec };#[cfg(feature = "std" )]
rust/serde_cbor/ser.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! Serialize a Rust data structure to CBOR data. #[cfg(feature = "alloc" )]
rust/serde_cbor/tags.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! Support for cbor tags use core::fmt;use core::marker::PhantomData ;
rust/serde_cbor/value/de.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use std::collections::BTreeMap ;use std::fmt;
rust/serde_cbor/value/mod.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! CBOR values, keys and serialization routines. mod de;
rust/serde_cbor/value/ser.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +// Copyright 2017 Serde Developers // // Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
rust/serde_cbor/write.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +#[cfg(feature = "alloc" )] use alloc::vec::Vec ;#[cfg(not(feature = "std" ))]
Signature · 1 line
2.40.1
PATCH 40/80 rust: serde_cbor: add no_fp_fmt_parse support
In reply to Ariel MiculasWe do not have formatting for floating point in the kernel, thus simple compile out all that.
--- rust/serde_cbor/de.rs | 8 ++++++++ rust/serde_cbor/ser.rs | 3 +++ rust/serde_cbor/tags.rs | 2 ++ rust/serde_cbor/value/ser.rs | 2 ++ 4 files changed, 15 insertions(+)
rust/serde_cbor/de.rs +8 -0
@@ -2,10 +2,12 @@//! Deserialization. +#[cfg(not(no_fp_fmt_parse))] use core::f32;use core::marker::PhantomData ;use core::result;use core::str; +#[cfg(not(no_fp_fmt_parse))] use half::f16;use serde::de;#[cfg(feature = "std" )]
@@ -566,14 +568,17 @@ fn parse_indefinite_enum<V>(&mut self, visitor: V) -> Result<V::Value> }) } +#[cfg(not(no_fp_fmt_parse))] fn parse_f16 (&mut self) ->Result <f32 > {Ok (f32::from (f16::from_bits (self.parse_u16 ()?))) } +#[cfg(not(no_fp_fmt_parse))] fn parse_f32 (&mut self) ->Result <f32 > { self.parse_u32 ().map (|i| f32::from_bits (i)) } +#[cfg(not(no_fp_fmt_parse))] fn parse_f64 (&mut self) ->Result <f64 > { self.parse_u64 ().map (|i| f64::from_bits (i)) }
@@ -756,14 +761,17 @@ fn parse_value<V>(&mut self, visitor: V) -> Result<V::Value> 0xf6 => visitor.visit_unit(), 0xf7 => visitor.visit_unit(), 0xf8 => Err(self.error(ErrorCode::UnassignedCode)), +#[cfg(not(no_fp_fmt_parse))] 0xf9 => {let value = self.parse_f16 ()?; visitor.visit_f32 (value) } + #[cfg(not(no_fp_fmt_parse))] 0xfa => { let value = self.parse_f32()?; visitor.visit_f32 (value) } +#[cfg(not(no_fp_fmt_parse))] 0xfb => { let value = self.parse_f64()?; visitor.visit_f64(value)
rust/serde_cbor/ser.rs +3 -0
@@ -10,6 +10,7 @@pub use crate ::write::{SliceWrite ,Write };use crate ::error::{Error ,Result }; +#[cfg(not(no_fp_fmt_parse))] use half::f16;use serde::ser::{self,Serialize };#[cfg(feature = "std" )]
@@ -311,6 +312,7 @@ fn serialize_u128(self, value: u128) -> Result<()> {#[inline] #[allow(clippy::float_cmp)] +#[cfg(not(no_fp_fmt_parse))] fn serialize_f32 (self, value:f32 ) ->Result <()> {if value.is_infinite () {if value.is_sign_positive () {
@@ -334,6 +336,7 @@ fn serialize_f32(self, value: f32) -> Result<()> {#[inline] #[allow(clippy::float_cmp)] +#[cfg(not(no_fp_fmt_parse))] fn serialize_f64 (self, value:f64 ) ->Result <()> {if !value.is_finite () || f64::from (valueas f32 ) == value { self.serialize_f32 (valueas f32 )
rust/serde_cbor/tags.rs +2 -0
@@ -129,7 +129,9 @@ fn expecting(&self, fmt: &mut fmt::Formatter<'_>) -> fmt::Result {delegate! (visit_u32,u32 );delegate! (visit_u64,u64 ); +#[cfg(not(no_fp_fmt_parse))] delegate! (visit_f32,f32 ); +#[cfg(not(no_fp_fmt_parse))] delegate! (visit_f64,f64 );delegate! (visit_char,char );
rust/serde_cbor/value/ser.rs +2 -0
@@ -101,11 +101,13 @@ fn serialize_u64(self, value: u64) -> Result<Value, Error> { }#[inline] +#[cfg(not(no_fp_fmt_parse))] fn serialize_f32 (self, value:f32 ) ->Result <Value ,Error > { self.serialize_f64 (f64::from (value)) }#[inline] +#[cfg(not(no_fp_fmt_parse))] fn serialize_f64 (self, value:f64 ) ->Result <Value ,Error > {Ok (Value ::Float (value)) }
Signature · 1 line
2.40.1
PATCH 41/80 rust: Kbuild: enable serde_cbor
In reply to Ariel Miculas--- Makefile | 2 ++ rust/Makefile | 23 ++++++++++++++++++++--- scripts/Makefile.build | 2 +- 3 files changed, 23 insertions(+), 4 deletions(-)
Makefile +2 -0
@@ -1853,6 +1853,7 @@ rustfmt: -o -path $(abs_srctree)/rust/syn -prune \ -o -path $(abs_srctree)/rust/serde -prune \ -o -path $(abs_srctree)/rust/serde_derive -prune \ + -o -path $(abs_srctree)/rust/serde_cbor -prune \ -o -path $(abs_objtree)/rust/test -prune \ | grep -Fv $(abs_srctree)/rust/alloc \ | grep -Fv $(abs_srctree)/rust/proc-macro2 \
@@ -1860,6 +1861,7 @@ rustfmt: | grep -Fv $(abs_srctree)/rust/syn \ | grep -Fv $(abs_srctree)/rust/serde \ | grep -Fv $(abs_srctree)/rust/serde_derive \ + | grep -Fv $(abs_srctree)/rust/serde_cbor \ | grep -Fv $(abs_objtree)/rust/test \ | grep -Fv generated \ | xargs $(RUSTFMT) $(rustfmt_flags)
rust/Makefile +20 -3
@@ -13,7 +13,7 @@ always-$(CONFIG_RUST) += libserde_derive.so libmacros.so no-clean-files += libserde_derive.so libmacros.so always-$(CONFIG_RUST) += bindings/bindings_generated.rs bindings/bindings_helpers_generated.rs -obj-$(CONFIG_RUST) += alloc.o bindings.o serde.o kernel.o +obj-$(CONFIG_RUST) += alloc.o bindings.o serde.o serde_cbor.o kernel.o always-$(CONFIG_RUST) += exports_alloc_generated.h exports_bindings_generated.h \ exports_kernel_generated.h
@@ -122,6 +122,17 @@ serde-flags := \ -Amissing_docs \ --cfg no_fp_fmt_parse +serde_cbor-skip_flags := \ + --edition=2021 \ + -Drust_2018_idioms \ + -Dunreachable_pub + +serde_cbor-flags := \ + --edition=2018 \ + -Amissing_docs \ + --cfg no_fp_fmt_parse \ + --extern serde + quiet_cmd_rustdoc = RUSTDOC $(if $(rustdoc_host),H, ) $< cmd_rustdoc = \ OBJTREE=$(abspath $(objtree)) \
@@ -568,12 +579,18 @@ $(obj)/serde.o: private rustc_target_flags = $(serde-flags) $(obj)/serde.o: $(src)/serde/lib.rs $(obj)/compiler_builtins.o FORCE $(call if_changed_dep,rustc_library) +$(obj)/serde_cbor.o: private skip_clippy = 1 +$(obj)/serde_cbor.o: private skip_flags = $(serde_cbor-skip_flags) +$(obj)/serde_cbor.o: private rustc_target_flags = $(serde_cbor-flags) +$(obj)/serde_cbor.o: $(src)/serde_cbor/lib.rs $(obj)/compiler_builtins.o FORCE + $(call if_changed_dep,rustc_library) + $(obj)/kernel.o: private rustc_target_flags = --extern alloc \ --extern build_error --extern macros --extern bindings --extern uapi \ - --extern serde --extern serde_derive + --extern serde --extern serde_derive --extern serde_cbor $(obj)/kernel.o: $(src)/kernel/lib.rs $(obj)/alloc.o $(obj)/build_error.o \ $(obj)/libmacros.so $(obj)/bindings.o $(obj)/uapi.o \ - $(obj)/serde.o $(obj)/libserde_derive.so FORCE + $(obj)/serde.o $(obj)/libserde_derive.so $(obj)/serde_cbor.o FORCE $(call if_changed_dep,rustc_library) endif # CONFIG_RUST
scripts/Makefile.build +1 -1
@@ -284,7 +284,7 @@ rust_common_cmd = \ -Zallow-features=$(rust_allowed_features) \ -Zcrate-attr=no_std \ -Zcrate-attr='feature($(rust_allowed_features))' \ - --extern alloc --extern kernel --extern serde --extern serde_derive \ + --extern alloc --extern kernel --extern serde --extern serde_derive --extern serde_cbor \ --crate-type rlib -L $(objtree)/rust/ \ --crate-name $(basename $(notdir $@)) \ --emit=dep-info=$(depfile)
Signature · 1 line
2.40.1
Re: [PATCH 41/80] rust: Kbuild: enable serde_cbor
Quoted 2 lines · Ariel Miculas
Signed-off-by Ariel Miculas <[email protected]>
Regarding serde_cbor: the repository upstream says it was archived almost two years ago.
While it says it is widely used, it also says it is unmaintained and recommends other two crates for new users.
Since this is an RFC, this is fine for the moment to have something that works, but if eventually the kernel decides to merge puzzlefs and its dependencies, we will need to consider who will maintain this, or whether another crate would be a better fit (or a kernel-made one), etc.
Cheers,
Miguel
PATCH 42/80 samples: rust: add cbor serialize/deserialize example
In reply to Ariel Miculas--- samples/rust/rust_serde.rs | 56 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+)
samples/rust/rust_serde.rs +56 -0
@@ -7,6 +7,8 @@use kernel::prelude::*;use serde_derive::{Deserialize ,Serialize }; +use serde_cbor::ser::SliceWrite ; +use serde::Serialize ;module! {type :RustSerde ,
@@ -38,6 +40,57 @@ struct S { d: (), } +#[derive( Debug , PartialEq , Serialize , Deserialize )] +struct User { + user_id:u32 , + password_hash: [u8 ;4 ], +} + +fn cbor_serialize () ->Result <(), serde_cbor::Error > { +let mut buf = [0u8 ;100 ]; +let writer =SliceWrite ::new (&mut buf[..]); +let mut ser = serde_cbor::Serializer ::new (writer); +let user =User { + user_id:42 , + password_hash: [1 ,2 ,3 ,4 ], + }; + user.serialize (&mut ser)?; +let writer = ser.into_inner (); +let size = writer.bytes_written (); +let expected = [ +0xa2 ,0x67 ,0x75 ,0x73 ,0x65 ,0x72 ,0x5f ,0x69 ,0x64 ,0x18 ,0x2a ,0x6d , +0x70 ,0x61 ,0x73 ,0x73 ,0x77 ,0x6f ,0x72 ,0x64 ,0x5f ,0x68 ,0x61 ,0x73 , +0x68 ,0x84 ,0x1 ,0x2 ,0x3 ,0x4 + ]; +assert_eq! (&buf[..size], expected); + +crate ::pr_info!("cbor serialized = {:?}" , buf); + +Ok (()) +} + +fn cbor_deserialize () ->Result <(), serde_cbor::Error > { +let value = [ +0xa2 ,0x67 ,0x75 ,0x73 ,0x65 ,0x72 ,0x5f ,0x69 ,0x64 ,0x18 ,0x2a ,0x6d , +0x70 ,0x61 ,0x73 ,0x73 ,0x77 ,0x6f ,0x72 ,0x64 ,0x5f ,0x68 ,0x61 ,0x73 , +0x68 ,0x84 ,0x1 ,0x2 ,0x3 ,0x4 + ]; + +// from_slice_with_scratch will not alter input data, use it whenever you +// borrow from somewhere else. +// You will have to size your scratch according to the input data you +// expect. +let mut scratch = [0u8 ;32 ]; +let user:User = serde_cbor::de::from_slice_with_scratch (&value[..], &mut scratch)?; +assert_eq! (user,User { + user_id:42 , + password_hash: [1 ,2 ,3 ,4 ], + }); + +crate ::pr_info!("cbor deserialized = {:?}" , user); +Ok (()) +} +impl kernel::Module for RustSerde {fn init (_module: &'staticThisModule ) ->Result <Self > {pr_info! ("Rust serde sample (init)\n" );
@@ -62,6 +115,9 @@ fn init(_module: &'static ThisModule) -> Result<Self> {let deserialized:S = local_data_format::from_bytes (&serialized).unwrap ();crate ::pr_info!("deserialized (local) = {:?}" , deserialized); +cbor_serialize ().unwrap (); +cbor_deserialize ().unwrap (); +Ok (RustSerde ) } }
Signature · 1 line
2.40.1
PATCH 43/80 rust: serde_cbor: add support for serde_cbor's from_slice method by using a custom alloc_kernel feature
In reply to Ariel Miculas--- rust/Makefile | 2 ++ rust/kernel/test_serde/de.rs | 7 ++----- rust/kernel/test_serde/ser.rs | 5 +---- rust/serde_cbor/de.rs | 6 +++--- rust/serde_cbor/lib.rs | 2 +- rust/serde_cbor/read.rs | 18 +++++++++++++----- samples/rust/rust_serde.rs | 32 ++++++++++++++------------------ 7 files changed, 36 insertions(+), 36 deletions(-)
rust/Makefile +2 -0
@@ -131,6 +131,8 @@ serde_cbor-flags := \ --edition=2018 \ -Amissing_docs \ --cfg no_fp_fmt_parse \ + --cfg 'feature="kernel_alloc"' \ + --extern alloc \ --extern serde quiet_cmd_rustdoc = RUSTDOC $(if $(rustdoc_host),H, ) $<
rust/kernel/test_serde/de.rs +2 -5
@@ -434,9 +434,6 @@ struct Test { }let j = &[2 ,0 ,1 ,0 ,3 ]; -let expected =Test { - a: (), - b:false , - }; +let expected =Test { a: (), b:false };assert_eq! (expected,from_bytes (j).unwrap ()); -} +}
rust/kernel/test_serde/ser.rs +1 -4
@@ -454,10 +454,7 @@ struct Test { b: bool, } -let test =Test { - a: (), - b:false , - }; +let test =Test { a: (), b:false };let mut expected =Vec ::new (); expected.try_push (2 ).unwrap ();
rust/serde_cbor/de.rs +3 -3
@@ -21,7 +21,7 @@#[cfg(feature = "std" )] pub use crate ::read::IoRead ;use crate ::read::Offset ; -#[cfg(any(feature = "std" , feature = "alloc" ))] +#[cfg(any(feature = "std" , feature = "alloc" , feature = "kernel_alloc" ))] pub use crate ::read::SliceRead ;pub use crate ::read::{MutSliceRead ,Read ,SliceReadFixed };#[cfg(feature = "tags" )]
@@ -47,7 +47,7 @@/// let value: &str = de::from_slice(&v[..]).unwrap(); /// assert_eq!(value, "foobar"); /// ``` -#[cfg(any(feature = "std" , feature = "alloc" ))] +#[cfg(any(feature = "std" , feature = "alloc" , feature = "kernel_alloc" ))] pub fn from_slice <'a,T >(slice: &'a [u8 ]) ->Result <T >where T : de::Deserialize <'a>,
@@ -150,7 +150,7 @@ pub fn from_reader(reader: R) -> Deserializer<IoRead<R>> { } } -#[cfg(any(feature = "std" , feature = "alloc" ))] +#[cfg(any(feature = "std" , feature = "alloc" , feature = "kernel_alloc" ))] impl <'a>Deserializer <SliceRead <'a>> {/// Constructs a `Deserializer` which reads from a slice. ///
rust/serde_cbor/lib.rs +1 -1
@@ -324,7 +324,7 @@#[cfg(all(not(feature = "std" ), test))] extern crate std; -#[cfg(feature = "alloc" )] +#[cfg(any(feature = "alloc" , feature = "kernel_alloc" ))] extern crate alloc;pub mod de;
rust/serde_cbor/read.rs +13 -5
@@ -2,6 +2,8 @@#[cfg(feature = "alloc" )] use alloc::{vec, vec::Vec }; +#[cfg(feature = "kernel_alloc" )] +use alloc::{vec::Vec };#[cfg(feature = "std" )] use core::cmp;use core::mem;
@@ -298,7 +300,7 @@ fn read(&mut self, buf: &mut [u8]) -> io::Result<usize> { }/// A CBOR input source that reads from a slice of bytes. -#[cfg(any(feature = "std" , feature = "alloc" ))] +#[cfg(any(feature = "std" , feature = "alloc" , feature = "kernel_alloc" ))] #[derive( Debug )] pub struct SliceRead <'a> { slice: &'a [u8 ],
@@ -306,13 +308,16 @@ pub struct SliceRead<'a> { index: usize, } -#[cfg(any(feature = "std" , feature = "alloc" ))] +#[cfg(any(feature = "std" , feature = "alloc" , feature = "kernel_alloc" ))] impl <'a>SliceRead <'a> {/// Creates a CBOR input source to read from a slice of bytes. pub fn new (slice: &'a [u8 ]) ->SliceRead <'a> {SliceRead { slice, +#[cfg(not(feature = "kernel_alloc" ))] scratch:vec! [], +#[cfg(feature = "kernel_alloc" )] + scratch:Vec ::new (), index:0 , } }
@@ -328,7 +333,7 @@ fn end(&self, n: usize) -> Result<usize> { } } -#[cfg(any(feature = "std" , feature = "alloc" ))] +#[cfg(any(feature = "std" , feature = "alloc" , feature = "kernel_alloc" ))] impl <'a>Offset for SliceRead <'a> {#[inline] fn byte_offset (&self) ->usize {
@@ -337,12 +342,12 @@ fn byte_offset(&self) -> usize { }#[cfg(all( -any(feature = "std" , feature = "alloc" ), +any(feature = "std" , feature = "alloc" , feature = "kernel_alloc" ), not(feature = "unsealed_read_write" ) ))] impl <'a> private::Sealed for SliceRead <'a> {} -#[cfg(any(feature = "std" , feature = "alloc" ))] +#[cfg(any(feature = "std" , feature = "alloc" , feature = "kernel_alloc" ))] impl <'a>Read <'a>for SliceRead <'a> {#[inline] fn next (&mut self) ->Result <Option <u8 >> {
@@ -371,7 +376,10 @@ fn clear_buffer(&mut self) {fn read_to_buffer (&mut self, n:usize ) ->Result <()> {let end = self.end (n)?;let slice = &self.slice[self.index..end]; +#[cfg(not(feature = "kernel_alloc" ))] self.scratch.extend_from_slice (slice); +#[cfg(feature = "kernel_alloc" )] + self.scratch.try_extend_from_slice (slice).unwrap (); self.index = end;Ok (())
samples/rust/rust_serde.rs +14 -18
@@ -6,9 +6,9 @@//! one here ("local"). Then it uses both on a type that uses `serve_derive`. use kernel::prelude::*; -use serde_derive::{Deserialize ,Serialize }; -use serde_cbor::ser::SliceWrite ;use serde::Serialize ; +use serde_cbor::ser::SliceWrite ; +use serde_derive::{Deserialize ,Serialize };module! {type :RustSerde ,
@@ -58,9 +58,8 @@ fn cbor_serialize() -> Result<(), serde_cbor::Error> {let writer = ser.into_inner ();let size = writer.bytes_written ();let expected = [ -0xa2 ,0x67 ,0x75 ,0x73 ,0x65 ,0x72 ,0x5f ,0x69 ,0x64 ,0x18 ,0x2a ,0x6d , -0x70 ,0x61 ,0x73 ,0x73 ,0x77 ,0x6f ,0x72 ,0x64 ,0x5f ,0x68 ,0x61 ,0x73 , -0x68 ,0x84 ,0x1 ,0x2 ,0x3 ,0x4 +0xa2 ,0x67 ,0x75 ,0x73 ,0x65 ,0x72 ,0x5f ,0x69 ,0x64 ,0x18 ,0x2a ,0x6d ,0x70 ,0x61 ,0x73 , +0x73 ,0x77 ,0x6f ,0x72 ,0x64 ,0x5f ,0x68 ,0x61 ,0x73 ,0x68 ,0x84 ,0x1 ,0x2 ,0x3 ,0x4 , ];assert_eq! (&buf[..size], expected);
@@ -71,21 +70,18 @@ fn cbor_serialize() -> Result<(), serde_cbor::Error> {fn cbor_deserialize () ->Result <(), serde_cbor::Error > {let value = [ -0xa2 ,0x67 ,0x75 ,0x73 ,0x65 ,0x72 ,0x5f ,0x69 ,0x64 ,0x18 ,0x2a ,0x6d , -0x70 ,0x61 ,0x73 ,0x73 ,0x77 ,0x6f ,0x72 ,0x64 ,0x5f ,0x68 ,0x61 ,0x73 , -0x68 ,0x84 ,0x1 ,0x2 ,0x3 ,0x4 +0xa2 ,0x67 ,0x75 ,0x73 ,0x65 ,0x72 ,0x5f ,0x69 ,0x64 ,0x18 ,0x2a ,0x6d ,0x70 ,0x61 ,0x73 , +0x73 ,0x77 ,0x6f ,0x72 ,0x64 ,0x5f ,0x68 ,0x61 ,0x73 ,0x68 ,0x84 ,0x1 ,0x2 ,0x3 ,0x4 , ]; -// from_slice_with_scratch will not alter input data, use it whenever you -// borrow from somewhere else. -// You will have to size your scratch according to the input data you -// expect. -let mut scratch = [0u8 ;32 ]; -let user:User = serde_cbor::de::from_slice_with_scratch (&value[..], &mut scratch)?; -assert_eq! (user,User { - user_id:42 , - password_hash: [1 ,2 ,3 ,4 ], - }); +let user:User = serde_cbor::de::from_slice (&value[..])?; +assert_eq! ( + user, +User { + user_id:42 , + password_hash: [1 ,2 ,3 ,4 ], + } + );crate ::pr_info!("cbor deserialized = {:?}" , user);Ok (())
Signature · 1 line
2.40.1
Re: [PATCH 43/80] rust: serde_cbor: add support for serde_cbor's from_slice method by using a custom alloc_kernel feature
rust/kernel/test_serde/de.rs struct Test { · line 437
Quoted 7 lines · Ariel Miculas
let j = &[2, 0, 1, 0, 3];rust/kernel/test_serde/de.rs +1
-4-let expected =Test { - a: (), - b:false , - }; +let expected =Test { a: (), b:false };
It seems rustfmt was run within this commit, which makes it harder to spot the changes. Please apply it on the commits that add the code itself instead.
rust/serde_cbor/read.rs pub struct SliceRead<'a> { · line 320
Quoted 9 lines
impl<'a> SliceRead<'a> { /// Creates a CBOR input source to read from a slice of bytes. pub fn new(slice: &'a [u8]) -> SliceRead<'a> { SliceRead { slice,rust/serde_cbor/read.rs +3
-0+#[cfg(not(feature = "kernel_alloc" ))] scratch: vec![], +#[cfg(feature = "kernel_alloc" )] + scratch:Vec ::new (),
Can this be simply replaced, i.e. without the gate, or is there a reason for it?
Either that, or we go ahead and make vec! work, but originally I thought it would be more confusing, see
https://github.com/rust-lang/rust/commit/8cec88ba76e1d6edc98f30101c40f9247c754898.
rust/serde_cbor/read.rs fn clear_buffer(&mut self) { · line 382
Quoted 8 lines
rust/serde_cbor/read.rs +3
-0@@ -371,7 +376,10 @@ fn clear_buffer(&mut self) {fn read_to_buffer (&mut self, n:usize ) ->Result <()> {let end = self.end (n)?;let slice = &self.slice[self.index..end]; +#[cfg(not(feature = "kernel_alloc" ))] self.scratch.extend_from_slice (slice); +#[cfg(feature = "kernel_alloc" )] + self.scratch.try_extend_from_slice (slice).unwrap ();
Do we know this allocation would not fail? i.e. we need to be extra careful on unwrap()s within the kernel. Especially for facilities that may have more than one caller/user like libraries.
Cheers,
Miguel
PATCH 44/80 rust: serde: add support for deserializing Vec with kernel_alloc feature
In reply to Ariel Miculas--- rust/Makefile | 4 +++- rust/serde/de/impls.rs | 19 +++++++++++++++++-- rust/serde/lib.rs | 2 +- rust/serde/private/size_hint.rs | 2 +- 4 files changed, 22 insertions(+), 5 deletions(-)
rust/Makefile +3 -1
@@ -120,7 +120,9 @@ serde-skip_flags := \ serde-flags := \ -Amissing_docs \ - --cfg no_fp_fmt_parse + --cfg no_fp_fmt_parse \ + --cfg 'feature="kernel_alloc"' \ + --extern alloc serde_cbor-skip_flags := \ --edition=2021 \
rust/serde/de/impls.rs +17 -2
@@ -11,9 +11,12 @@use seed::InPlaceSeed ; -#[cfg(any(feature = "std" , feature = "alloc" ))] +#[cfg(any(feature = "std" , feature = "alloc" , feature = "kernel_alloc" ))] use __private::size_hint; +#[cfg(feature = "kernel_alloc" )] +use alloc::vec::Vec ; +//////////////////////////////////////////////////////////////////////////////// struct UnitVisitor ;
@@ -1012,7 +1015,7 @@ fn nop_reserve<T>(_seq: T, _n: usize) {}//////////////////////////////////////////////////////////////////////////////// -#[cfg(any(feature = "std" , feature = "alloc" ))] +#[cfg(any(feature = "std" , feature = "alloc" , feature = "kernel_alloc" ))] impl <'de,T >Deserialize <'de>for Vec <T >where T :Deserialize <'de>,
@@ -1039,10 +1042,16 @@ fn visit_seq<A>(self, mut seq: A) -> Result<Self::Value, A::Error> where A: SeqAccess<'de>, { +#[cfg(not(feature = "kernel_alloc" ))] let mut values =Vec ::with_capacity (size_hint::cautious (seq.size_hint ())); +#[cfg(feature = "kernel_alloc" )] +let mut values =Vec ::try_with_capacity (size_hint::cautious (seq.size_hint ())).unwrap (); while let Some(value) = try!(seq.next_element()) { +#[cfg(not(feature = "kernel_alloc" ))] values.push (value); +#[cfg(feature = "kernel_alloc" )] + values.try_push (value).unwrap (); }Ok (values)
@@ -1077,7 +1086,10 @@ fn visit_seq<A>(self, mut seq: A) -> Result<Self::Value, A::Error> {let hint = size_hint::cautious (seq.size_hint ());if let Some (additional) = hint.checked_sub (self.0 .len ()) { +#[cfg(not(feature = "kernel_alloc" ))] self.0 .reserve (additional); +#[cfg(feature = "kernel_alloc" )] + self.0 .try_reserve (additional).unwrap (); }for iin 0 ..self.0 .len () {
@@ -1092,7 +1104,10 @@ fn visit_seq<A>(self, mut seq: A) -> Result<Self::Value, A::Error> } while let Some(value) = try!(seq.next_element()) { +#[cfg(not(feature = "kernel_alloc" ))] self.0 .push (value); +#[cfg(feature = "kernel_alloc" )] + self.0 .try_push (value).unwrap (); }Ok (())
rust/serde/lib.rs +1 -1
@@ -156,7 +156,7 @@//////////////////////////////////////////////////////////////////////////////// -#[cfg(feature = "alloc" )] +#[cfg(any(feature = "alloc" , feature = "kernel_alloc" ))] extern crate alloc;/// A facade around all the types we need from the `std`, `core`, and `alloc`
rust/serde/private/size_hint.rs +1 -1
@@ -9,7 +9,7 @@ pub fn from_bounds<I>(iter: &I) -> Option<usize>helper (iter.size_hint ()) } -#[cfg(any(feature = "std" , feature = "alloc" ))] +#[cfg(any(feature = "std" , feature = "alloc" , feature = "kernel_alloc" ))] #[inline] pub fn cautious (hint:Option <usize >) ->usize { cmp::min (hint.unwrap_or (0 ),4096 )
Signature · 1 line
2.40.1
Re: [PATCH 44/80] rust: serde: add support for deserializing Vec with kernel_alloc feature
rust/serde/de/impls.rs fn visit_seq<A>(self, mut seq: A) -> Result<Self::Value, A::Error> · line 1048
Quoted 5 lines · Ariel Miculas
rust/serde/de/impls.rs +3
-0+#[cfg(not(feature = "kernel_alloc" ))] let mut values =Vec ::with_capacity (size_hint::cautious (seq.size_hint ())); +#[cfg(feature = "kernel_alloc" )] +let mut values =Vec ::try_with_capacity (size_hint::cautious (seq.size_hint ())).unwrap ();
Same note on these unwrap()s as in the other patch.
Cheers,
Miguel
PATCH 45/80 rust: file: Replace UnsafeCell with Opaque for File
In reply to Ariel Miculas--- rust/kernel/file.rs | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
rust/kernel/file.rs +3 -2
@@ -17,10 +17,11 @@ types::ARef, types::AlwaysRefCounted, types::ForeignOwnable, + types::Opaque, user_ptr::{UserSlicePtr, UserSlicePtrReader, UserSlicePtrWriter}, };use core::convert::{TryFrom ,TryInto }; -use core::{cell::UnsafeCell , marker, mem, ptr}; +use core::{marker, mem, ptr};use macros::vtable;/// Flags associated with a [`File`].
@@ -113,7 +114,7 @@ pub mod flags {/// Instances of this type are always ref-counted, that is, a call to `get_file` ensures that the /// allocation remains valid at least until the matching call to `fput`. #[repr(transparent)] -pub struct File (pub (crate )UnsafeCell <bindings::file >); +pub struct File (pub (crate )Opaque <bindings::file >);// TODO: Accessing fields of `struct file` through the pointer is UB because other threads may be // writing to them. However, this is how the C code currently operates: naked reads and writes to
Signature · 1 line
2.40.1
PATCH 46/80 rust: kernel: implement fmt::Debug for CString
In reply to Ariel Miculas--- rust/kernel/str.rs | 6 ++++++ 1 file changed, 6 insertions(+)
rust/kernel/str.rs +6 -0
@@ -236,6 +236,12 @@ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { } } +impl fmt::Debug for CString { +fn fmt (&self, f: &mut fmt::Formatter <'_>) -> fmt::Result { + fmt::Debug ::fmt (&**self, f) + } +} +impl fmt::Debug for CStr {/// Formats printable ASCII characters with a double quote on either end, escaping the rest. ///
Signature · 1 line
2.40.1
PATCH 47/80 samples: puzzlefs: rename RustFs to PuzzleFs
In reply to Ariel Miculas--- samples/rust/puzzlefs.rs | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
samples/rust/puzzlefs.rs +5 -5
@@ -7,16 +7,16 @@use kernel::{c_str, file, fs, io_buffer::IoBufferWriter };module_fs! { -type :RustFs , +type :PuzzleFs , name:"puzzlefs" , author:"Ariel Miculas" , license:"GPL" , } -struct RustFs ; +struct PuzzleFs ;#[vtable] -impl fs::Context <Self >for RustFs { +impl fs::Context <Self >for PuzzleFs {type Data = (); kernel::define_fs_params! {(),
@@ -39,7 +39,7 @@ fn try_new() -> Result { } } -impl fs::Type for RustFs { +impl fs::Type for PuzzleFs {type Context =Self ;type INodeData = &'static [u8 ];const SUPER_TYPE : fs::Super = fs::Super ::Independent ;
@@ -97,7 +97,7 @@ fn read( offset: u64, ) -> Result<usize> { file::read_from_slice ( - file.inode ::<RustFs >().ok_or (EINVAL )?.fs_data (), + file.inode ::<PuzzleFs >().ok_or (EINVAL )?.fs_data (), writer, offset, )
Signature · 1 line
2.40.1
PATCH 48/80 samples: puzzlefs: add basic deserializing support for the puzzlefs metadata
In reply to Ariel Miculas--- samples/rust/puzzle.rs | 2 + samples/rust/puzzle/error.rs | 32 +++ samples/rust/puzzle/types.rs | 304 ++++++++++++++++++++++ samples/rust/puzzle/types/cbor_helpers.rs | 9 + samples/rust/puzzlefs.rs | 2 + 5 files changed, 349 insertions(+) create mode 100644 samples/rust/puzzle.rs create mode 100644 samples/rust/puzzle/error.rs create mode 100644 samples/rust/puzzle/types.rs create mode 100644 samples/rust/puzzle/types/cbor_helpers.rs
samples/rust/puzzle.rs new file +2 -0
@@ -0,0 +1,2 @@ +pub (crate )mod error; +mod types;
samples/rust/puzzle/error.rs new file +32 -0
@@ -0,0 +1,32 @@ +use core::fmt::{self,Display }; + +// TODO use String in error types (when it's available from the kernel) + +pub (crate )enum WireFormatError { +LocalRefError , +SeekOtherError , +ValueMissing , +CBORError (serde_cbor::Error ), +} + +impl Display for WireFormatError { +fn fmt (&self, f: &mut fmt::Formatter <'_>) -> fmt::Result { +match self { +WireFormatError ::LocalRefError => f.write_str ("cannot turn local ref into a digest" ), +WireFormatError ::SeekOtherError => f.write_str ("cannot seek to other blob" ), +WireFormatError ::ValueMissing => f.write_str ("no value present" ), +WireFormatError ::CBORError (_) => f.write_str ("CBOR error" ), + } + } +} + +pub (crate )type Result <T > = kernel::error::Result <T ,WireFormatError >; + +// TODO figure out how to use thiserror +#[allow(unused_qualifications)] +impl core::convert::From <serde_cbor::Error >for WireFormatError { +#[allow(deprecated)] +fn from (source: serde_cbor::Error ) ->Self { +WireFormatError ::CBORError (source) + } +}
samples/rust/puzzle/types.rs new file +304 -0
304 lines not on this page · raw message
samples/rust/puzzle/types/cbor_helpers.rs new file +9 -0
@@ -0,0 +1,9 @@ +pub (crate )const fn cbor_size_of_list_header (size:usize ) ->usize { +match size { +0 ..=23 =>1 , +24 ..=255 =>2 , +256 ..=65535 =>3 , +65536 ..=4294967295 =>4 , + _ =>8 , + } +}
samples/rust/puzzlefs.rs +2 -0
@@ -6,6 +6,8 @@use kernel::prelude::*;use kernel::{c_str, file, fs, io_buffer::IoBufferWriter }; +mod puzzle; +module_fs! {type :PuzzleFs , name:"puzzlefs" ,
Signature · 1 line
2.40.1
PATCH 49/80 rust: file: present the filesystem context to the open function
In reply to Ariel MiculasThis is not a good implementation because it does not ensure that the type Operations::FSData in file.rs is the same as Type::Data in fs.rs. This means that we could store a Box in the field s_fs_info and retrieve an Arc from there, which is obviously wrong and it leads to UB.
--- rust/kernel/file.rs | 18 ++++++++++++++++-- samples/rust/puzzlefs.rs | 21 ++++++++++++++++++--- samples/rust/rust_fs.rs | 2 +- 3 files changed, 35 insertions(+), 6 deletions(-)
rust/kernel/file.rs +16 -2
@@ -330,9 +330,14 @@ impl<A: OpenAdapter<T::OpenData>, T: Operations> OperationsVtable<A, T> {// `fileref` never outlives this function, so it is guaranteed to be // valid. let fileref =unsafe {File ::from_ptr (file) }; + +// SAFETY: into_foreign was called in fs::NewSuperBlock<..., NeedsInit>::init and +// it is valid until from_foreign will be called in fs::Tables::free_callback +let fs_info =unsafe {T ::FSData ::borrow ((*(*inode).i_sb).s_fs_info) }; +// SAFETY: `arg` was previously returned by `A::convert` and must // be a valid non-null pointer. -let ptr =T ::open (unsafe { &*arg }, fileref)?.into_foreign (); +let ptr =T ::open (fs_info,unsafe { &*arg }, fileref)?.into_foreign ();// SAFETY: The C contract guarantees that `private_data` is available // for implementers of the file operations (no other C code accesses // it), so we know that there are no concurrent threads/CPUs accessing
@@ -793,10 +798,19 @@ pub trait Operations {/// The type of the context data passed to [`Operations::open`]. type OpenData: Sync = (); +/// Data associated with each file system instance. +// SAFETY: this is not safe because we don't enforce the same type as fs::Type::Data, so it's +// possible to store a Box in s_fs_info and retrieve an Arc when open is called, leading to UB + type FSData: ForeignOwnable + Send + Sync = (); +/// Creates a new instance of this file. /// /// Corresponds to the `open` function pointer in `struct file_operations`. -fn open (context: &Self ::OpenData , file: &File ) ->Result <Self ::Data >; +fn open ( + fs_info: <Self ::FSData as ForeignOwnable >::Borrowed <'_>, + context: &Self ::OpenData , + file: &File , + ) ->Result <Self ::Data >;/// Cleans up after the last reference to the file goes away. ///
samples/rust/puzzlefs.rs +18 -3
@@ -4,7 +4,7 @@use kernel::module_fs;use kernel::prelude::*; -use kernel::{c_str, file, fs, io_buffer::IoBufferWriter }; +use kernel::{c_str, file, fs, io_buffer::IoBufferWriter , fmt, str::CString };mod puzzle;
@@ -17,6 +17,11 @@struct PuzzleFs ; +#[derive( Debug )] +struct PuzzlefsInfo { + base_path:CString , +} +#[vtable] impl fs::Context <Self >for PuzzleFs {type Data = ();
@@ -44,14 +49,17 @@ fn try_new() -> Result { impl fs::Type for PuzzleFs { type Context = Self; type INodeData = &'static [u8]; + type Data = Box<PuzzlefsInfo>; const SUPER_TYPE: fs::Super = fs::Super::Independent; const NAME: &'static CStr = c_str!("puzzlefs"); const FLAGS: i32 = fs::flags::USERNS_MOUNT; const DCACHE_BASED: bool = true; fn fill_super(_data: (), sb: fs::NewSuperBlock<'_, Self>) -> Result<&fs::SuperBlock<Self>> { + let base_path = CString::try_from_fmt(fmt!("hello world"))?; + pr_info!("base_path {:?}\n", base_path); let sb = sb.init( - (), + Box::try_new(PuzzlefsInfo { base_path })?, &fs::SuperParams { magic: 0x72757374, ..fs::SuperParams::DEFAULT
@@ -87,8 +95,15 @@ fn fill_super(_data: (), sb: fs::NewSuperBlock<'_, Self>) -> Result<&fs::SuperBl#[vtable] impl file::Operations for FsFile {type OpenData = &'static [u8 ]; +type FSData =Box <PuzzlefsInfo >; + +fn open ( + fs_info: &PuzzlefsInfo , + _context: &Self ::OpenData , + _file: &file::File , + ) ->Result <Self ::Data > { +pr_info! ("got {:?}\n" , fs_info); -fn open (_context: &Self ::OpenData , _file: &file::File ) ->Result <Self ::Data > {Ok (()) }
samples/rust/rust_fs.rs +1 -1
@@ -86,7 +86,7 @@ fn fill_super(_data: (), sb: fs::NewSuperBlock<'_, Self>) -> Result<&fs::SuperBlimpl file::Operations for FsFile {type OpenData = &'static [u8 ]; -fn open (_context: &Self ::OpenData , _file: &file::File ) ->Result <Self ::Data > { +fn open (_fs_info: (), _context: &Self ::OpenData , _file: &file::File ) ->Result <Self ::Data > {Ok (()) }
Signature · 1 line
2.40.1
PATCH 50/80 rust: kernel: add an abstraction over vfsmount to allow cloning a new private mount
In reply to Ariel Miculas--- rust/bindings/bindings_helper.h | 1 + rust/kernel/lib.rs | 1 + rust/kernel/mount.rs | 66 +++++++++++++++++++++++++++++++++ 3 files changed, 68 insertions(+) create mode 100644 rust/kernel/mount.rs
rust/bindings/bindings_helper.h +1 -0
@@ -17,6 +17,7 @@#include <linux/uio.h> #include <linux/uaccess.h> #include <linux/delay.h> +#include <linux/namei.h> /* `bindgen` gets confused at certain things. */ const gfp_t BINDINGS_GFP_KERNEL =GFP_KERNEL ;
rust/kernel/lib.rs +1 -0
@@ -46,6 +46,7 @@pub mod ioctl;pub mod iov_iter;pub mod mm; +pub mod mount;pub mod pages;pub mod prelude;pub mod print;
rust/kernel/mount.rs new file +66 -0
66 lines not on this page · raw message
Signature · 1 line
2.40.1
PATCH 51/80 rust: file: add from_path, from_path_in_root_mnt and read_with_offset methods to File
In reply to Ariel Miculas--- rust/kernel/error.rs | 4 +-- rust/kernel/file.rs | 74 +++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 74 insertions(+), 4 deletions(-)
rust/kernel/error.rs +1 -3
@@ -234,9 +234,7 @@ pub fn to_result(err: core::ffi::c_int) -> Result {/// } /// } /// ``` -// TODO: Remove `dead_code` marker once an in-kernel client is available. -#[allow(dead_code)] -pub (crate )fn from_err_ptr <T >(ptr: *mut T ) ->Result <*mut T > { +pub fn from_err_ptr <T >(ptr: *mut T ) ->Result <*mut T > {// CAST: Casting a pointer to `*const core::ffi::c_void` is always valid. let const_ptr: *const core::ffi::c_void = ptr.cast ();// SAFETY: The FFI function does not deref the pointer.
rust/kernel/file.rs +73 -1
90 lines not on this page · raw message
Signature · 1 line
2.40.1
PATCH 52/80 samples: puzzlefs: pass the Vfsmount structure from open to read and return the contents of the data file inside /home/puzzlefs_oci
In reply to Ariel Miculas--- samples/rust/puzzlefs.rs | 44 +++++++++++++++++++++++++++++----------- 1 file changed, 32 insertions(+), 12 deletions(-)
samples/rust/puzzlefs.rs +32 -12
@@ -3,8 +3,14 @@//! Rust file system sample. use kernel::module_fs; +use kernel::mount::Vfsmount ;use kernel::prelude::*; -use kernel::{c_str, file, fs, io_buffer::IoBufferWriter , fmt, str::CString }; +use kernel::{ + c_str, file, fmt, fs, + io_buffer::IoBufferWriter , + str::CString , + sync::{Arc ,ArcBorrow }, +};mod puzzle;
@@ -20,6 +26,7 @@#[derive( Debug )] struct PuzzlefsInfo { base_path:CString , + vfs_mount:Arc <Vfsmount >, }#[vtable]
@@ -58,8 +65,14 @@ impl fs::Type for PuzzleFs {fn fill_super (_data: (), sb: fs::NewSuperBlock <'_,Self >) ->Result <&fs::SuperBlock <Self >> {let base_path =CString ::try_from_fmt (fmt! ("hello world" ))?;pr_info! ("base_path {:?}\n" , base_path); +let vfs_mount =Vfsmount ::new_private_mount (c_str! ("/home/puzzlefs_oci" ))?; +pr_info! ("vfs_mount {:?}\n" , vfs_mount); + let sb = sb.init( - Box::try_new(PuzzlefsInfo { base_path })?, + Box::try_new(PuzzlefsInfo { + base_path, + vfs_mount: Arc::try_new(vfs_mount)?, + })?, &fs::SuperParams { magic: 0x72757374, ..fs::SuperParams::DEFAULT
@@ -94,29 +107,36 @@ fn fill_super(_data: (), sb: fs::NewSuperBlock<'_, Self>) -> Result<&fs::SuperBl#[vtable] impl file::Operations for FsFile { +// must be the same as INodeData type OpenData = &'static [u8 ];type FSData =Box <PuzzlefsInfo >; +// this is an Arc because Data must be ForeignOwnable and the only implementors of it are Box, +// Arc and (); we cannot pass a reference to read, so we share Vfsmount using and Arc +type Data =Arc <Vfsmount >;fn open ( fs_info: &PuzzlefsInfo , _context: &Self ::OpenData , _file: &file::File , ) ->Result <Self ::Data > { -pr_info! ("got {:?}\n" , fs_info); - -Ok (()) +Ok (fs_info.vfs_mount.clone ()) }fn read ( - _data: (), - file: &file::File , + data:ArcBorrow <'_,Vfsmount >, + _file: &file::File , writer: &mut impl IoBufferWriter , offset:u64 , ) ->Result <usize > { - file::read_from_slice ( - file.inode ::<PuzzleFs >().ok_or (EINVAL )?.fs_data (), - writer, - offset, - ) +let mut buf =Vec ::try_with_capacity (writer.len ())?; + buf.try_resize (writer.len (),0 )?; +let file = file::File ::from_path_in_root_mnt ( + &data, +c_str! ("data" ), + file::flags::O_RDONLY .try_into ().unwrap (), +0 , + )?; +let nr_bytes_read = file.read_with_offset (&mut buf[..], offset)?; + file::read_from_slice (&buf[..nr_bytes_read], writer,0 ) } }
Signature · 1 line
2.40.1
PATCH 53/80 rust: file: move from_path, from_path_in_root_mnt and read_with_offset methods to a RegularFile newtype
In reply to Ariel Miculas--- rust/kernel/file.rs | 144 ++++++++++++++++++++------------------- samples/rust/puzzlefs.rs | 2 +- 2 files changed, 75 insertions(+), 71 deletions(-)
rust/kernel/file.rs +74 -70
@@ -133,76 +133,6 @@ pub fn from_fd(fd: u32) -> Result<ARef<Self>> {Ok (unsafe {ARef ::from_raw (ptr.cast ()) }) } -/// Constructs a new [`struct file`] wrapper from a path. -pub fn from_path (filename: &CStr , flags:i32 , mode:u16 ) ->Result <ARef <Self >> { -let file_ptr =unsafe { -from_err_ptr (bindings::filp_open ( - filename.as_ptr ()as *const i8 , - flags, - mode, - ))? - }; -let file_ptr = ptr::NonNull ::new (file_ptr).ok_or (ENOENT )?; - -// SAFETY: `filp_open` initializes the refcount with 1 -Ok (unsafe {ARef ::from_raw (file_ptr.cast ()) }) - } - -/// Constructs a new [`struct file`] wrapper from a path and a vfsmount. -pub fn from_path_in_root_mnt ( - mount: &Vfsmount , - filename: &CStr , - flags:i32 , - mode:u16 , - ) ->Result <ARef <Self >> { -let file_ptr =unsafe { -let mnt = mount.get (); -let raw_path = bindings::path { - mnt, - dentry: (*mnt).mnt_root, - }; -from_err_ptr (bindings::file_open_root ( - &raw_path, - filename.as_ptr ()as *const i8 , - flags, - mode, - ))? - }; -let file_ptr = ptr::NonNull ::new (file_ptr).ok_or (ENOENT )?; - -// SAFETY: `file_open_root` increments the refcount before returning. (TODO does it?) -Ok (unsafe {ARef ::from_raw (file_ptr.cast ()) }) - } - -/// Read from the file into the specified buffer -pub fn read_with_offset (&self, buf: &mut [u8 ], offset:u64 ) ->Result <usize > { -Ok (unsafe { -// kernel_read_file expects a pointer to a "void *" buffer -let mut ptr_to_buf = buf.as_mut_ptr ()as *mut core::ffi::c_void ; -// Unless we give a non-null pointer to the file size: -// 1. we cannot give a non-zero value for the offset -// 2. we cannot have offset 0 and buffer_size > file_size -let mut file_size =0 ; - -// SAFETY: all the pointers to kernel_read_file are valid -let result = bindings::kernel_read_file ( - self.0 .get (), - offset.try_into ()?, - &mut ptr_to_buf, - buf.len (), - &mut file_size, - bindings::kernel_read_file_id_READING_UNKNOWN, - ); - -// kernel_read_file returns the number of bytes read on success or negative on error. -if result <0 { -return Err (Error ::from_errno (result.try_into ()?)); - } - - result.try_into ()? - }) - } -/// Creates a reference to a [`File`] from a valid pointer. /// /// # Safety
@@ -273,6 +203,80 @@ unsafe fn dec_ref(obj: ptr::NonNull<Self>) { } } +/// A newtype over file, specific to regular files +pub struct RegularFile (ARef <File >); +impl RegularFile { +/// Constructs a new [`struct file`] wrapper from a path. +pub fn from_path (filename: &CStr , flags:i32 , mode:u16 ) ->Result <Self > { +let file_ptr =unsafe { +from_err_ptr (bindings::filp_open ( + filename.as_ptr ()as *const i8 , + flags, + mode, + ))? + }; +let file_ptr = ptr::NonNull ::new (file_ptr).ok_or (ENOENT )?; + +// SAFETY: `filp_open` initializes the refcount with 1 +Ok (RegularFile (unsafe {ARef ::from_raw (file_ptr.cast ()) })) + } + +/// Constructs a new [`struct file`] wrapper from a path and a vfsmount. +pub fn from_path_in_root_mnt ( + mount: &Vfsmount , + filename: &CStr , + flags:i32 , + mode:u16 , + ) ->Result <Self > { +let file_ptr =unsafe { +let mnt = mount.get (); +let raw_path = bindings::path { + mnt, + dentry: (*mnt).mnt_root, + }; +from_err_ptr (bindings::file_open_root ( + &raw_path, + filename.as_ptr ()as *const i8 , + flags, + mode, + ))? + }; +let file_ptr = ptr::NonNull ::new (file_ptr).ok_or (ENOENT )?; + +// SAFETY: `file_open_root` increments the refcount before returning. (TODO does it?) +Ok (RegularFile (unsafe {ARef ::from_raw (file_ptr.cast ()) })) + } + +/// Read from the file into the specified buffer +pub fn read_with_offset (&self, buf: &mut [u8 ], offset:u64 ) ->Result <usize > { +Ok (unsafe { +// kernel_read_file expects a pointer to a "void *" buffer +let mut ptr_to_buf = buf.as_mut_ptr ()as *mut core::ffi::c_void ; +// Unless we give a non-null pointer to the file size: +// 1. we cannot give a non-zero value for the offset +// 2. we cannot have offset 0 and buffer_size > file_size +let mut file_size =0 ; + +// SAFETY: all the pointers to kernel_read_file are valid +let result = bindings::kernel_read_file ( + self.0 .0 .get (), + offset.try_into ()?, + &mut ptr_to_buf, + buf.len (), + &mut file_size, + bindings::kernel_read_file_id_READING_UNKNOWN, + ); + +// kernel_read_file returns the number of bytes read on success or negative on error. +if result <0 { +return Err (Error ::from_errno (result.try_into ()?)); + } + + result.try_into ()? + }) + } +} +/// A file descriptor reservation. /// /// This allows the creation of a file descriptor in two steps: first, we reserve a slot for it,
samples/rust/puzzlefs.rs +1 -1
@@ -130,7 +130,7 @@ fn read( ) -> Result<usize> {let mut buf =Vec ::try_with_capacity (writer.len ())?; buf.try_resize (writer.len (),0 )?; -let file = file::File ::from_path_in_root_mnt ( +let file = file::RegularFile ::from_path_in_root_mnt ( &data,c_str! ("data" ), file::flags::O_RDONLY.try_into().unwrap(),
Signature · 1 line
2.40.1
PATCH 54/80 rust: file: ensure RegularFile can only create regular files
In reply to Ariel Miculas--- rust/kernel/file.rs | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)
rust/kernel/file.rs +16 -4
@@ -206,8 +206,21 @@ unsafe fn dec_ref(obj: ptr::NonNull<Self>) { /// A newtype over file, specific to regular files pub struct RegularFile(ARef<File>); impl RegularFile { + fn create_if_regular(file_ptr: ptr::NonNull<bindings::file>) -> Result<RegularFile> { + // make sure the file is a regular file + // TODO: create a helper function + // SAFETY: we have a NonNull pointer + unsafe { + let inode = core::ptr::addr_of!((*file_ptr.as_ptr()).f_inode).read(); + if bindings::S_IFMT & ((*inode).i_mode) as u32 != bindings::S_IFREG { + return Err(EINVAL); + } + } + Ok(RegularFile(unsafe { ARef::from_raw(file_ptr.cast()) })) + } /// Constructs a new [`struct file`] wrapper from a path. pub fn from_path(filename: &CStr, flags: i32, mode: u16) -> Result<Self> { + // SAFETY: `filp_open` initializes the refcount with 1 let file_ptr = unsafe { from_err_ptr(bindings::filp_open( filename.as_ptr() as *const i8,
@@ -217,8 +230,7 @@ pub fn from_path(filename: &CStr, flags: i32, mode: u16) -> Result<Self> { };let file_ptr = ptr::NonNull ::new (file_ptr).ok_or (ENOENT )?; -// SAFETY: `filp_open` initializes the refcount with 1 -Ok (RegularFile (unsafe {ARef ::from_raw (file_ptr.cast ()) })) +Self ::create_if_regular (file_ptr) }/// Constructs a new [`struct file`] wrapper from a path and a vfsmount.
@@ -228,6 +240,7 @@ pub fn from_path_in_root_mnt( flags: i32, mode: u16, ) -> Result<Self> { + // SAFETY: `file_open_root` increments the refcount before returning. (TODO does it?) let file_ptr = unsafe { let mnt = mount.get(); let raw_path = bindings::path {
@@ -243,8 +256,7 @@ pub fn from_path_in_root_mnt( };let file_ptr = ptr::NonNull ::new (file_ptr).ok_or (ENOENT )?; -// SAFETY: `file_open_root` increments the refcount before returning. (TODO does it?) -Ok (RegularFile (unsafe {ARef ::from_raw (file_ptr.cast ()) })) +Self ::create_if_regular (file_ptr) }/// Read from the file into the specified buffer
Signature · 1 line
2.40.1
PATCH 55/80 rust: file: add get_pos method to RegularFile
In reply to Ariel Miculas--- rust/kernel/file.rs | 5 +++++ 1 file changed, 5 insertions(+)
rust/kernel/file.rs +5 -0
@@ -287,6 +287,11 @@ pub fn read_with_offset(&self, buf: &mut [u8], offset: u64) -> Result<usize> { result.try_into ()? }) } + +/// Get the current file position +pub fn get_pos (&self) ->u64 { + self.0 .pos () + } }/// A file descriptor reservation.
Signature · 1 line
2.40.1
PATCH 56/80 rust: file: add methods read_to_end, get_file_size and update_pos to RegularFile
In reply to Ariel Miculas--- rust/kernel/file.rs | 52 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+)
rust/kernel/file.rs +52 -0
@@ -22,6 +22,7 @@ types::Opaque, user_ptr::{UserSlicePtr, UserSlicePtrReader, UserSlicePtrWriter}, }; +use alloc::vec::Vec ;use core::convert::{TryFrom ,TryInto };use core::{marker, mem, ptr};use macros::vtable;
@@ -288,10 +289,61 @@ pub fn read_with_offset(&self, buf: &mut [u8], offset: u64) -> Result<usize> { }) } +/// Allocate and return a vector containing the contents of the entire file +pub fn read_to_end (&self) ->Result <Vec <u8 >> { +let file_size = self.get_file_size ()?; +let mut buffer =Vec ::try_with_capacity (file_size)?; + buffer.try_resize (file_size,0 )?; + self.read_with_offset (&mut buffer,0 )?; +Ok (buffer) + } +/// Get the current file position pub fn get_pos (&self) ->u64 { self.0 .pos () } + +fn get_file_size (&self) ->Result <usize > { +let mut buf =Vec ::new (); +unsafe { +// kernel_read_file expects a pointer to a "void *" buffer +let mut ptr_to_buf = buf.as_mut_ptr ()as *mut core::ffi::c_void ; +let mut file_size =0 ; + +// SAFETY: all the pointers to kernel_read_file are valid +let result = bindings::kernel_read_file ( + self.0 .0 .get (), +0 , + &mut ptr_to_buf, + buf.len (), + &mut file_size, + bindings::kernel_read_file_id_READING_UNKNOWN, + ); + +// kernel_read_file returns the number of bytes read on success or negative on error. +if result <0 { +return Err (Error ::from_errno (result.try_into ()?)); + } +Ok (file_size) + } + } + +fn update_pos (&self, where_to:SeekFrom ) ->Result <u64 > { +let (offset, whence): (i64 ,_ ) =match where_to { +SeekFrom ::Start (off) => (off.try_into ()?, bindings::SEEK_SET ), +SeekFrom ::Current (off) => (off, bindings::SEEK_CUR ), +SeekFrom ::End (off) => (off, bindings::SEEK_END ), + }; + +let offset = +unsafe { bindings::generic_file_llseek (self.0 .0 .get (), offset, whence.try_into ()?) }; +if offset <0 { +return Err (Error ::from_errno (offset.try_into ()?)); + } + +// we just checked that offset is non negative +Ok (offsetas u64 ) + } }/// A file descriptor reservation.
Signature · 1 line
2.40.1
PATCH 57/80 rust: file: define a minimal Read trait and implement it for RegularFile
In reply to Ariel Miculas--- rust/kernel/file.rs | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)
rust/kernel/file.rs +42 -0
@@ -204,6 +204,40 @@ unsafe fn dec_ref(obj: ptr::NonNull<Self>) { } } +/// adapted from https://doc.rust-lang.org/std/io/trait.Read.html +pub trait Read { +// Required method +/// Pull some bytes from this source into the specified buffer, returning how many bytes were read. +/// See https://doc.rust-lang.org/std/io/trait.Read.html#tymethod.read for details +/// This is the equivalent of read from std::io::Read::read, but in kernel space instead of user space +/// The buffer is a kernel-allocated buffer +fn read (&mut self, buf: &mut [u8 ]) ->Result <usize >; + +// Provided methods + +// Implementation adapted from https://github.com/rust-lang/rust/blob/80917360d350fb55aebf383e7ff99efea41f63fd/library/std/src/io/mod.rs#L465 +/// Read the exact number of bytes required to fill buf. +/// See https://doc.rust-lang.org/std/io/trait.Read.html#method.read_exact for details +fn read_exact (&mut self,mut buf: &mut [u8 ]) ->Result <()> { +while !buf.is_empty () { +match self.read (buf) { +Ok (0 ) =>break , +Ok (n) => { +let tmp = buf; + buf = &mut tmp[n..]; + } +Err (e) =>return Err (e), + } + } + +if !buf.is_empty () { +Err (EINVAL ) + }else { +Ok (()) + } + } +} +/// A newtype over file, specific to regular files pub struct RegularFile (ARef <File >);impl RegularFile {
@@ -346,6 +380,14 @@ fn update_pos(&self, where_to: SeekFrom) -> Result<u64> { } } +impl Read for &mut RegularFile { +fn read (&mut self, buf: &mut [u8 ]) ->Result <usize > { +let nr_bytes_read = self.read_with_offset (buf, self.get_pos ())?; + self.update_pos (SeekFrom ::Current (nr_bytes_read.try_into ()?))?; +Ok (nr_bytes_read) + } +} +/// A file descriptor reservation. /// /// This allows the creation of a file descriptor in two steps: first, we reserve a slot for it,
Signature · 1 line
2.40.1
PATCH 58/80 samples: puzzlefs: add cbor_get_array_size method
In reply to Ariel Miculas--- samples/rust/puzzle/types/cbor_helpers.rs | 41 +++++++++++++++++++++++ 1 file changed, 41 insertions(+)
samples/rust/puzzle/types/cbor_helpers.rs +41 -0
@@ -1,3 +1,6 @@ +use crate ::puzzle::types::{Result ,WireFormatError }; +use kernel::file::Read ; +pub (crate )const fn cbor_size_of_list_header (size:usize ) ->usize {match size {0 ..=23 =>1 ,
@@ -7,3 +10,41 @@ pub(crate) const fn cbor_size_of_list_header(size: usize) -> usize { _ => 8, } } + +fn parse_u8 (mut reader:impl Read ) ->Result <u8 > { +let mut buf = [0 ;1 ]; + reader.read_exact (&mut buf)?; +Ok (u8::from_be_bytes (buf)) +} + +fn parse_u16 (mut reader:impl Read ) ->Result <u16 > { +let mut buf = [0 ;2 ]; + reader.read_exact (&mut buf)?; +Ok (u16::from_be_bytes (buf)) +} + +fn parse_u32 (mut reader:impl Read ) ->Result <u32 > { +let mut buf = [0 ;4 ]; + reader.read_exact (&mut buf)?; +Ok (u32::from_be_bytes (buf)) +} + +fn parse_u64 (mut reader:impl Read ) ->Result <u64 > { +let mut buf = [0 ;8 ]; + reader.read_exact (&mut buf)?; +Ok (u64::from_be_bytes (buf)) +} + +pub (crate )fn cbor_get_array_size <R :Read >(mut reader:R ) ->Result <u64 > { +let mut buf = [0 ;1 ]; + reader.read_exact (&mut buf)?; + +match buf[0 ] { +0x80 ..=0x97 =>Ok ((buf[0 ] -0x80 )as u64 ), +0x98 =>parse_u8 (reader).map (u64::from), +0x99 =>parse_u16 (reader).map (u64::from), +0x9a =>parse_u32 (reader).map (u64::from), +0x9b =>parse_u64 (reader).map (u64::from), + _ =>Err (WireFormatError ::ValueMissing ), + } +}
Signature · 1 line
2.40.1
PATCH 59/80 samples: puzzlefs: add KernelError to WireFormatError and implement From conversion
In reply to Ariel Miculas--- samples/rust/puzzle/error.rs | 10 ++++++++++ 1 file changed, 10 insertions(+)
samples/rust/puzzle/error.rs +10 -0
@@ -7,6 +7,7 @@ pub(crate) enum WireFormatError { SeekOtherError, ValueMissing, CBORError(serde_cbor::Error), + KernelError(kernel::error::Error), }impl Display for WireFormatError {
@@ -16,6 +17,7 @@ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { WireFormatError::SeekOtherError => f.write_str("cannot seek to other blob"), WireFormatError::ValueMissing => f.write_str("no value present"), WireFormatError::CBORError(_) => f.write_str("CBOR error"), + WireFormatError::KernelError(_) => f.write_str("Kernel error"), } } }
@@ -30,3 +32,11 @@ fn from(source: serde_cbor::Error) -> Self {WireFormatError ::CBORError (source) } } + +#[allow(unused_qualifications)] +impl core::convert::From <kernel::error::Error >for WireFormatError { +#[allow(deprecated)] +fn from (source: kernel::error::Error ) ->Self { +WireFormatError ::KernelError (source) + } +}
Signature · 1 line
2.40.1
PATCH 60/80 samples: puzzlefs: implement new for MetadataBlob
In reply to Ariel Miculas--- samples/rust/puzzle/types.rs | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-)
samples/rust/puzzle/types.rs +14 -4
@@ -1,5 +1,4 @@use crate ::puzzle::error::WireFormatError ; -use alloc::boxed::Box ;use alloc::vec::Vec ;use core::mem::size_of;use serde::de::Error as SerdeError ;
@@ -8,7 +7,8 @@use serde_derive::Deserialize ;mod cbor_helpers;use crate ::puzzle::error::Result ; -pub (crate )use cbor_helpers::cbor_size_of_list_header; +pub (crate )use cbor_helpers::{cbor_get_array_size, cbor_size_of_list_header}; +use kernel::file;#[derive( Deserialize , Debug )] pub (crate )struct InodeAdditional {
@@ -26,9 +26,10 @@ pub(crate) struct Xattr { pub(crate) val: Vec<u8>, } +#[derive( Debug )] pub (crate )struct MetadataBlob { - mmapped_region:Box <[u8 ]>, - inode_count:usize , +pub (crate ) mmapped_region:Vec <u8 >, +pub (crate ) inode_count:usize , }fn read_one_from_slice <'a,T :Deserialize <'a>>(bytes: &'a [u8 ]) ->Result <T > {
@@ -116,6 +117,15 @@ fn visit_bytes<E>(self, v: &[u8]) -> kernel::error::Result<BlobRef, E> }impl MetadataBlob { +pub (crate )fn new (mut f: file::RegularFile ) ->Result <MetadataBlob > { +let inodes_count =cbor_get_array_size (&mut f)?as usize ; +let mmapped_region = f.read_to_end ()?; +Ok (MetadataBlob { + mmapped_region, + inode_count: inodes_count, + }) + } +pub (crate )fn seek_ref (&mut self, r: &BlobRef ) ->Result <u64 > {match r.kind {BlobRefKind ::Other { .. } =>Err (WireFormatError ::SeekOtherError ),
Signature · 1 line
2.40.1
PATCH 61/80 samples: puzzlefs: build puzzlefs into the kernel, thus avoiding the need to export rust symbols
In reply to Ariel Miculas--- kernel/configs/rust.config | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/configs/rust.config +1 -1
@@ -6,7 +6,7 @@ CONFIG_SAMPLES_RUST=y CONFIG_SAMPLE_RUST_MINIMAL=m CONFIG_SAMPLE_RUST_PRINT=m CONFIG_SAMPLE_RUST_FS=m -CONFIG_SAMPLE_PUZZLEFS=m +CONFIG_SAMPLE_PUZZLEFS=y CONFIG_SAMPLE_RUST_HOSTPROGS=y CONFIG_SAMPLE_RUST_SERDE=y
Signature · 1 line
2.40.1
PATCH 62/80 rust: alloc: add try_clone for Vec<T>
In reply to Ariel Miculas--- rust/alloc/vec/mod.rs | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
rust/alloc/vec/mod.rs +17 -0
@@ -2489,6 +2489,15 @@ unsafe fn split_at_spare_mut_with_len( } } +impl <T :Clone >Vec <T > { +/// Try to clone the vector using the global allocator +#[inline] +#[stable(feature = "kernel" , since = "1.0.0" )] +pub fn try_clone (&self) ->Result <Self ,TryReserveError > { + self.try_clone_in (Global ) + } +} +impl <T :Clone ,A :Allocator >Vec <T ,A > {/// Resizes the `Vec` in-place so that `len` is equal to `new_len`. ///
@@ -2613,6 +2622,14 @@ pub fn try_extend_from_slice(&mut self, other: &[T]) -> Result<(), TryReserveErr self.try_spec_extend (other.iter ()) } +/// Tries to clone the vector using the given allocator +#[stable(feature = "kernel" , since = "1.0.0" )] +pub fn try_clone_in (&self, allocator:A ) ->Result <Self ,TryReserveError > { +let mut new_vec =Vec ::try_with_capacity_in (self.len (), allocator)?; + new_vec.try_extend_from_slice (&self)?; +Ok (new_vec) + } +/// Copies elements from `src` range to the end of the vector. /// /// # Panics
Signature · 1 line
2.40.1
PATCH 63/80 rust: alloc: add from_iter_fallible for Vec<T>
In reply to Ariel Miculas--- rust/alloc/vec/mod.rs | 10 ++++++++++ 1 file changed, 10 insertions(+)
rust/alloc/vec/mod.rs +10 -0
@@ -649,6 +649,16 @@ pub fn try_with_capacity(capacity: usize) -> Result<Self, TryReserveError> {pub unsafe fn from_raw_parts (ptr: *mut T , length:usize , capacity:usize ) ->Self {unsafe {Self ::from_raw_parts_in (ptr, length, capacity,Global ) } } + +/// See https://doc.rust-lang.org/std/vec/struct.Vec.html#examples-31 +#[stable(feature = "rust1" , since = "1.0.0" )] +pub fn from_iter_fallible (iter:impl Iterator <Item =T >) ->Result <Vec <T >,TryReserveError > { +let mut vec =Vec ::new (); +for valuein iter { + vec.try_push (value)?; + } +Ok (vec) + } }impl <T ,A :Allocator >Vec <T ,A > {
Signature · 1 line
2.40.1
Re: [PATCH 63/80] rust: alloc: add from_iter_fallible for Vec<T>
rust/alloc/vec/mod.rs pub fn try_with_capacity(capacity: usize) -> Result<Self, TryReserveError> { · line 654
Quoted 3 lines · Ariel Miculas
rust/alloc/vec/mod.rs +2
-0+/// See https://doc.rust-lang.org/std/vec/struct.Vec.html#examples-31 +#[stable(feature = "rust1" , since = "1.0.0" )]
For additions to the alloc crate, please use kernel as the feature name. In addition, we write (or rather, copy, if possible, from the infallible ones :) docs.
Cheers,
Miguel
PATCH 64/80 samples: puzzlefs: implement to_errno and from_errno for WireFormatError
In reply to Ariel Miculas--- samples/rust/puzzle/error.rs | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
samples/rust/puzzle/error.rs +19 -0
@@ -1,4 +1,6 @@ +use core::ffi::c_int;use core::fmt::{self,Display }; +use kernel::prelude::{EINVAL ,ENOENT ,ESPIPE };// TODO use String in error types (when it's available from the kernel)
@@ -10,6 +12,23 @@ pub(crate) enum WireFormatError { KernelError(kernel::error::Error), } +impl WireFormatError { +pub (crate )fn to_errno (&self) ->c_int { +match self { +WireFormatError ::LocalRefError => kernel::error::Error ::to_errno (EINVAL ), +WireFormatError ::SeekOtherError => kernel::error::Error ::to_errno (ESPIPE ), +WireFormatError ::ValueMissing => kernel::error::Error ::to_errno (ENOENT ), +WireFormatError ::CBORError (..) => kernel::error::Error ::to_errno (EINVAL ), +WireFormatError ::KernelError (e) => kernel::error::Error ::to_errno (*e), +WireFormatError ::TryReserveError (e) => kernel::error::Error ::to_errno (EINVAL ), + } + } + +pub (crate )fn from_errno (errno: kernel::error::Error ) ->Self { +Self ::KernelError (errno) + } +} +impl Display for WireFormatError {fn fmt (&self, f: &mut fmt::Formatter <'_>) -> fmt::Result {match self {
Signature · 1 line
2.40.1
PATCH 65/80 samples: puzzlefs: add TryReserveError (and from conversion) to WireFormatError
In reply to Ariel Miculas--- samples/rust/puzzle/error.rs | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
samples/rust/puzzle/error.rs +12 -1
@@ -1,3 +1,4 @@ +use alloc::collections::TryReserveError ;use core::ffi::c_int;use core::fmt::{self,Display };use kernel::prelude::{EINVAL ,ENOENT ,ESPIPE };
@@ -10,6 +11,7 @@ pub(crate) enum WireFormatError { ValueMissing, CBORError(serde_cbor::Error), KernelError(kernel::error::Error), + TryReserveError(TryReserveError), }impl WireFormatError {
@@ -20,7 +22,7 @@ pub(crate) fn to_errno(&self) -> c_int { WireFormatError::ValueMissing => kernel::error::Error::to_errno(ENOENT), WireFormatError::CBORError(..) => kernel::error::Error::to_errno(EINVAL), WireFormatError::KernelError(e) => kernel::error::Error::to_errno(*e), - WireFormatError::TryReserveError(e) => kernel::error::Error::to_errno(EINVAL), + WireFormatError::TryReserveError(_) => kernel::error::Error::to_errno(EINVAL), } }
@@ -37,6 +39,7 @@ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { WireFormatError::ValueMissing => f.write_str("no value present"), WireFormatError::CBORError(_) => f.write_str("CBOR error"), WireFormatError::KernelError(_) => f.write_str("Kernel error"), + WireFormatError::TryReserveError(_) => f.write_str("TryReserveError"), } } }
@@ -59,3 +62,11 @@ fn from(source: kernel::error::Error) -> Self {WireFormatError ::KernelError (source) } } + +#[allow(unused_qualifications)] +impl core::convert::From <TryReserveError >for WireFormatError { +#[allow(deprecated)] +fn from (source:TryReserveError ) ->Self { +WireFormatError ::TryReserveError (source) + } +}
Signature · 1 line
2.40.1
PATCH 66/80 samples: puzzlefs: add higher level inode related functionality
In reply to Ariel Miculas--- samples/rust/puzzle.rs | 3 +- samples/rust/puzzle/inode.rs | 83 ++++++++++++++++++++++++++++++++++++ 2 files changed, 85 insertions(+), 1 deletion(-) create mode 100644 samples/rust/puzzle/inode.rs
samples/rust/puzzle.rs +2 -1
@@ -1,2 +1,3 @@pub (crate )mod error; -mod types; +pub (crate )mod inode; +pub (crate )mod types;
samples/rust/puzzle/inode.rs new file +83 -0
83 lines not on this page · raw message
Signature · 1 line
2.40.1
PATCH 67/80 samples: puzzlefs: populate the directory entries with the inodes from the puzzlefs metadata file
In reply to Ariel Miculas--- rust/kernel/fs.rs | 4 +- samples/rust/puzzlefs.rs | 125 ++++++++++++++++++++++++++++++--------- 2 files changed, 100 insertions(+), 29 deletions(-)
rust/kernel/fs.rs +3 -1
@@ -775,7 +775,9 @@ impl SuperParams {/// /// The superblock is a newly-created one and this is the only active pointer to it. pub struct NewSuperBlock <'a,T :Type + ?Sized ,S =NeedsInit > { - sb: &'amut SuperBlock <T >, +/// Pointer to the superblock; this fields is public so puzzlefs can call +/// try_new_dcache_dir_inode when populating the directory hierarchy +pub sb: &'amut SuperBlock <T >,// This also forces `'a` to be invariant. _p:PhantomData <&'amut &'aS >,
samples/rust/puzzlefs.rs +97 -28
174 lines not on this page · raw message
Signature · 1 line
2.40.1
PATCH 68/80 rust: hex: import crate
In reply to Ariel MiculasThis is a subset of the Rust hex crate, version v0.4.3, licensed under "Apache-2.0 OR MIT", from:
https://github.com/KokaKiwi/rust-hex/tree/v0.4.3/src
The files are copied as-is, with no modifications whatsoever (not even adding the SPDX identifiers).
For copyright details, please see:
https://github.com/KokaKiwi/rust-hex/blob/v0.4.3/README.md#license
https://github.com/KokaKiwi/rust-hex/blob/v0.4.3/LICENSE-APACHE
https://github.com/KokaKiwi/rust-hex/blob/v0.4.3/LICENSE-MIT
The next three patches modify these files as needed for use within the kernel. This patch split allows reviewers to double-check the import and to clearly see the differences introduced.
The following script may be used to verify the contents:
for path in $(cd rust/hex/ && find . -type f -name '*.rs'); do
curl --silent --show-error --location \
https://github.com/KokaKiwi/rust-hex/raw/v0.4.3/src/$path \
| diff --unified rust/hex/$path - && echo $path: OKdone
--- rust/hex/error.rs | 59 ++++++ rust/hex/lib.rs | 525 ++++++++++++++++++++++++++++++++++++++++++++++ rust/hex/serde.rs | 102 +++++++++ 3 files changed, 686 insertions(+) create mode 100644 rust/hex/error.rs create mode 100644 rust/hex/lib.rs create mode 100644 rust/hex/serde.rs
rust/hex/error.rs new file +59 -0
@@ -0,0 +1,59 @@ +use core::fmt; + +/// The error type for decoding a hex string into `Vec<u8>` or `[u8; N]`. +#[derive( Debug , Clone , Copy , PartialEq )] +pub enum FromHexError { +/// An invalid character was found. Valid ones are: `0...9`, `a...f` +/// or `A...F`. +InvalidHexCharacter { c:char , index:usize }, + +/// A hex string's length needs to be even, as two digits correspond to +/// one byte. +OddLength , + +/// If the hex string is decoded into a fixed sized container, such as an +/// array, the hex string's length * 2 has to match the container's +/// length. +InvalidStringLength , +} + +#[cfg(feature = "std" )] +impl std::error::Error for FromHexError {} + +impl fmt::Display for FromHexError { +fn fmt (&self, f: &mut fmt::Formatter ) -> fmt::Result { +match *self { +FromHexError ::InvalidHexCharacter { c, index } => { +write! (f,"Invalid character {:?} at position {}" , c, index) + } +FromHexError ::OddLength =>write! (f,"Odd number of digits" ), +FromHexError ::InvalidStringLength =>write! (f,"Invalid string length" ), + } + } +} + +#[cfg(test)] +// this feature flag is here to suppress unused +// warnings of `super::*` and `pretty_assertions::assert_eq` +#[cfg(feature = "alloc" )] +mod tests { +use super ::*; +#[cfg(feature = "alloc" )] +use alloc::string::ToString ; +use pretty_assertions::assert_eq; + +#[test] +#[cfg(feature = "alloc" )] +fn test_display () { +assert_eq! ( +FromHexError ::InvalidHexCharacter { c:'\n' , index:5 }.to_string (), +"Invalid character '\\n' at position 5" + ); + +assert_eq! (FromHexError ::OddLength .to_string (),"Odd number of digits" ); +assert_eq! ( +FromHexError ::InvalidStringLength .to_string (), +"Invalid string length" + ); + } +}
rust/hex/lib.rs new file +525 -0
525 lines not on this page · raw message
rust/hex/serde.rs new file +102 -0
102 lines not on this page · raw message
Signature · 1 line
2.40.1
PATCH 69/80 rust: hex: add SPDX license identifiers
In reply to Ariel Miculas--- rust/hex/error.rs | 2 ++ rust/hex/lib.rs | 2 ++ rust/hex/serde.rs | 2 ++ 3 files changed, 6 insertions(+)
rust/hex/error.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +use core::fmt;/// The error type for decoding a hex string into `Vec<u8>` or `[u8; N]`.
rust/hex/lib.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +// Copyright (c) 2013-2014 The Rust Project Developers. // Copyright (c) 2015-2020 The rust-hex Developers. //
rust/hex/serde.rs +2 -0
@@ -1,3 +1,5 @@ +// SPDX-License-Identifier: Apache-2.0 OR MIT +//! Hex encoding with `serde`. #[cfg_attr( all(feature = "alloc", feature = "serde"),
Signature · 1 line
2.40.1
PATCH 70/80 rust: Kbuild: enable `hex`
In reply to Ariel MiculasWith all the new files in place from the hex crate, this patch adds support for them in the build system.
--- rust/Makefile | 21 ++++++++++++++++++--- scripts/Makefile.build | 2 +- 2 files changed, 19 insertions(+), 4 deletions(-)
rust/Makefile +18 -3
@@ -13,7 +13,7 @@ always-$(CONFIG_RUST) += libserde_derive.so libmacros.so no-clean-files += libserde_derive.so libmacros.so always-$(CONFIG_RUST) += bindings/bindings_generated.rs bindings/bindings_helpers_generated.rs -obj-$(CONFIG_RUST) += alloc.o bindings.o serde.o serde_cbor.o kernel.o +obj-$(CONFIG_RUST) += alloc.o bindings.o serde.o serde_cbor.o hex.o kernel.o always-$(CONFIG_RUST) += exports_alloc_generated.h exports_bindings_generated.h \ exports_kernel_generated.h
@@ -137,6 +137,15 @@ serde_cbor-flags := \ --extern alloc \ --extern serde +hex-skip_flags := \ + --edition=2021 \ + -Drust_2018_idioms \ + -Dunreachable_pub + +hex-flags := \ + --edition=2018 \ + -Amissing_docs + quiet_cmd_rustdoc = RUSTDOC $(if $(rustdoc_host),H, ) $< cmd_rustdoc = \ OBJTREE=$(abspath $(objtree)) \
@@ -589,12 +598,18 @@ $(obj)/serde_cbor.o: private rustc_target_flags = $(serde_cbor-flags) $(obj)/serde_cbor.o: $(src)/serde_cbor/lib.rs $(obj)/compiler_builtins.o FORCE $(call if_changed_dep,rustc_library) +$(obj)/hex.o: private skip_clippy = 1 +$(obj)/hex.o: private skip_flags = $(hex-skip_flags) +$(obj)/hex.o: private rustc_target_flags = $(hex-flags) +$(obj)/hex.o: $(src)/hex/lib.rs $(obj)/compiler_builtins.o FORCE + $(call if_changed_dep,rustc_library) + $(obj)/kernel.o: private rustc_target_flags = --extern alloc \ --extern build_error --extern macros --extern bindings --extern uapi \ - --extern serde --extern serde_derive --extern serde_cbor + --extern serde --extern serde_derive --extern serde_cbor --extern hex $(obj)/kernel.o: $(src)/kernel/lib.rs $(obj)/alloc.o $(obj)/build_error.o \ $(obj)/libmacros.so $(obj)/bindings.o $(obj)/uapi.o \ - $(obj)/serde.o $(obj)/libserde_derive.so $(obj)/serde_cbor.o FORCE + $(obj)/serde.o $(obj)/libserde_derive.so $(obj)/serde_cbor.o $(obj)/hex.o FORCE $(call if_changed_dep,rustc_library) endif # CONFIG_RUST
scripts/Makefile.build +1 -1
@@ -284,7 +284,7 @@ rust_common_cmd = \ -Zallow-features=$(rust_allowed_features) \ -Zcrate-attr=no_std \ -Zcrate-attr='feature($(rust_allowed_features))' \ - --extern alloc --extern kernel --extern serde --extern serde_derive --extern serde_cbor \ + --extern alloc --extern kernel --extern serde --extern serde_derive --extern serde_cbor --extern hex \ --crate-type rlib -L $(objtree)/rust/ \ --crate-name $(basename $(notdir $@)) \ --emit=dep-info=$(depfile)
Signature · 1 line
2.40.1
PATCH 71/80 rust: hex: implement FromHex trait and hex::decode using a custom kernel_alloc feature
In reply to Ariel MiculasThis commit also adds a process_results function for Vec<Result<T,E>> which transforms it into a Result<Vec<T,E>>, collecting all the values inside the Result elements into a new vector which directly stores the elements.
For this to work, the TryReserveError was added to the FromHexError enum.
This is probably not the ideal design, but since collect cannot be used directly, it's a good substitute.
--- rust/Makefile | 4 +++- rust/alloc/vec/mod.rs | 21 +++++++++++++++++++++ rust/hex/error.rs | 17 +++++++++++++++++ rust/hex/lib.rs | 19 +++++++++++++++---- 4 files changed, 56 insertions(+), 5 deletions(-)
rust/Makefile +3 -1
@@ -144,7 +144,9 @@ hex-skip_flags := \ hex-flags := \ --edition=2018 \ - -Amissing_docs + -Amissing_docs \ + --cfg 'feature="kernel_alloc"' \ + --extern alloc quiet_cmd_rustdoc = RUSTDOC $(if $(rustdoc_host),H, ) $< cmd_rustdoc = \
rust/alloc/vec/mod.rs +21 -0
@@ -661,6 +661,27 @@ pub fn from_iter_fallible(iter: impl Iterator<Item=T>) -> Result<Vec<T>, TryRese } } +impl <T ,E : core::convert::From <TryReserveError >>Vec <Result <T ,E >> { +/// Iterates through the vector, consuming self, unwrapping the individual values in each +/// Result and creating a new vector with them. If there's any error value, it returns it +/// immediately +#[stable(feature = "rust1" , since = "1.0.0" )] +pub fn process_results (self) ->Result <Vec <T >,E > { +let mut vec =Vec ::new (); +for valuein self { +match value { +Ok (v) => { + vec.try_push (v)?; + }, +Err (e) => { +return Err (e); + } + } + } +Ok (vec) + } +} +impl <T ,A :Allocator >Vec <T ,A > {/// Constructs a new, empty `Vec<T, A>`. ///
rust/hex/error.rs +17 -0
@@ -1,5 +1,7 @@// SPDX-License-Identifier: Apache-2.0 OR MIT +#[cfg(feature = "kernel_alloc" )] +use alloc::collections::TryReserveError ;use core::fmt;/// The error type for decoding a hex string into `Vec<u8>` or `[u8; N]`.
@@ -17,6 +19,10 @@ pub enum FromHexError {/// array, the hex string's length * 2 has to match the container's /// length. InvalidStringLength, + +#[cfg(feature = "kernel_alloc" )] +/// If an allocation fails, useful in the kernel context + TryReserveError, }#[cfg(feature = "std" )]
@@ -30,10 +36,21 @@ fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { } FromHexError::OddLength => write!(f, "Odd number of digits"), FromHexError::InvalidStringLength => write!(f, "Invalid string length"), +#[cfg(feature = "kernel_alloc" )] + FromHexError::TryReserveError => write!(f, "TryReserveError"), } } } +#[cfg(feature = "kernel_alloc" )] +#[allow(unused_qualifications)] +impl core::convert::From <TryReserveError >for FromHexError { +#[allow(deprecated)] +fn from (_source:TryReserveError ) ->Self { +FromHexError ::TryReserveError + } +} +#[cfg(test)] // this feature flag is here to suppress unused // warnings of `super::*` and `pretty_assertions::assert_eq`
rust/hex/lib.rs +15 -4
@@ -41,6 +41,8 @@#[cfg(feature = "alloc" )] extern crate alloc; +#[cfg(feature = "kernel_alloc" )] +use alloc::vec::Vec ;#[cfg(feature = "alloc" )] use alloc::{string::String , vec::Vec };
@@ -186,7 +188,7 @@ fn val(c: u8, idx: usize) -> Result<u8, FromHexError> { } } -#[cfg(feature = "alloc" )] +#[cfg(any(feature = "alloc" , feature = "kernel_alloc" ))] impl FromHex for Vec <u8 > {type Error =FromHexError ;
@@ -196,10 +198,19 @@ fn from_hex<T: AsRef<[u8]>>(hex: T) -> Result<Self, Self::Error> {return Err (FromHexError ::OddLength ); } - hex.chunks (2 ) +#[cfg(not(feature = "kernel_alloc" ))] +return hex + .chunks (2 ) .enumerate () .map (|(i, pair)|Ok (val (pair[0 ],2 * i)? <<4 |val (pair[1 ],2 * i +1 )?)) - .collect () + .collect (); +#[cfg(feature = "kernel_alloc" )] +return Vec ::from_iter_fallible ( + hex.chunks (2 ) + .enumerate () + .map (|(i, pair)|Ok (val (pair[0 ],2 * i)? <<4 |val (pair[1 ],2 * i +1 )?)), + )? + .process_results (); } }
@@ -294,7 +305,7 @@ pub fn encode_upper<T: AsRef<[u8]>>(data: T) -> String {/// assert_eq!(hex::decode("123"), Err(hex::FromHexError::OddLength)); /// assert!(hex::decode("foo").is_err()); /// ``` -#[cfg(feature = "alloc" )] +#[cfg(any(feature = "alloc" , feature = "kernel_alloc" ))] pub fn decode <T :AsRef <[u8 ]>>(data:T ) ->Result <Vec <u8 >,FromHexError > {FromHex ::from_hex (data) }
Signature · 1 line
2.40.1
PATCH 72/80 rust: hex: add encode_hex_iter and encode_hex_upper_iter methods
In reply to Ariel MiculasAdd encode_hex_iter and encode_hex_upper_iter which return an iterator, so it can be used in conjunction with the Vec::from_iter_fallible function to (try to) create a hex string.
Replace BytesToHexChars with BytesToHexByteSequence because String is not available. We need to collect the values in a Vec<u8>, append a NUL character and the create a CStr on top of it.
Also remove the ToHex trait because it uses the FromIterator trait which cannot be implemented in the kernel due to the prohibition of infallible allocations.
--- rust/hex/lib.rs | 60 ++++++++++++------------------------------------- 1 file changed, 14 insertions(+), 46 deletions(-)
rust/hex/lib.rs +14 -46
@@ -59,43 +59,18 @@#[cfg(all(feature = "alloc" , feature = "serde" ))] pub use crate ::serde::{serialize, serialize_upper}; -/// Encoding values as hex string. -/// -/// This trait is implemented for all `T` which implement `AsRef<[u8]>`. This -/// includes `String`, `str`, `Vec<u8>` and `[u8]`. -/// -/// # Example -/// -/// ``` -/// use hex::ToHex; -/// -/// println!("{}", "Hello world!".encode_hex::<String>()); -/// # assert_eq!("Hello world!".encode_hex::<String>(), "48656c6c6f20776f726c6421".to_string()); -/// ``` -/// -/// *Note*: instead of using this trait, you might want to use [`encode()`]. -pub trait ToHex { -/// Encode the hex strict representing `self` into the result. Lower case -/// letters are used (e.g. `f9b4ca`) -fn encode_hex <T : iter::FromIterator <char >>(&self) ->T ; - -/// Encode the hex strict representing `self` into the result. Upper case -/// letters are used (e.g. `F9B4CA`) -fn encode_hex_upper <T : iter::FromIterator <char >>(&self) ->T ; -} -const HEX_CHARS_LOWER : &[u8 ;16 ] =b"0123456789abcdef" ;const HEX_CHARS_UPPER : &[u8 ;16 ] =b"0123456789ABCDEF" ; -struct BytesToHexChars <'a> { +struct BytesToHexByteSequence <'a> { inner: ::core::slice::Iter <'a,u8 >, table: &'static [u8 ;16 ], - next:Option <char >, + next:Option <u8 >, } -impl <'a>BytesToHexChars <'a> { -fn new (inner: &'a [u8 ], table: &'static [u8 ;16 ]) ->BytesToHexChars <'a> { -BytesToHexChars { +impl <'a>BytesToHexByteSequence <'a> { +fn new (inner: &'a [u8 ], table: &'static [u8 ;16 ]) ->BytesToHexByteSequence <'a> { +BytesToHexByteSequence { inner: inner.iter (), table, next:None ,
@@ -103,15 +78,15 @@ impl<'a> BytesToHexChars<'a> { } } -impl <'a>Iterator for BytesToHexChars <'a> { -type Item =char ; +impl <'a>Iterator for BytesToHexByteSequence <'a> { +type Item =u8 ;fn next (&mut self) ->Option <Self ::Item > {match self.next.take () {Some (current) =>Some (current),None => self.inner.next ().map (|byte| { -let current = self.table[(byte >>4 )as usize ]as char ; - self.next =Some (self.table[(byte &0x0F )as usize ]as char ); +let current = self.table[(byte >>4 )as usize ]; + self.next =Some (self.table[(byte &0x0F )as usize ]); current }), }
@@ -123,7 +98,7 @@ fn size_hint(&self) -> (usize, Option<usize>) { } } -impl <'a> iter::ExactSizeIterator for BytesToHexChars <'a> { +impl <'a> iter::ExactSizeIterator for BytesToHexByteSequence <'a> {fn len (&self) ->usize {let mut length = self.inner.len () *2 ;if self.next.is_some () {
@@ -133,19 +108,12 @@ fn len(&self) -> usize { } } -#[inline] -fn encode_to_iter <T : iter::FromIterator <char >>(table: &'static [u8 ;16 ], source: &[u8 ]) ->T { -BytesToHexChars ::new (source, table).collect () +pub fn encode_hex_iter <'a>(source: &'a [u8 ]) ->impl iter::Iterator <Item =u8 > + 'a { +BytesToHexByteSequence ::new (source,HEX_CHARS_LOWER ).into_iter () } -impl <T :AsRef <[u8 ]>>ToHex for T { -fn encode_hex <U : iter::FromIterator <char >>(&self) ->U { -encode_to_iter (HEX_CHARS_LOWER , self.as_ref ()) - } - -fn encode_hex_upper <U : iter::FromIterator <char >>(&self) ->U { -encode_to_iter (HEX_CHARS_UPPER , self.as_ref ()) - } +pub fn encode_hex_upper_iter <'a>(source: &'a [u8 ]) ->impl iter::Iterator <Item =u8 > + 'a { +BytesToHexByteSequence ::new (source,HEX_CHARS_UPPER ).into_iter () }/// Types that can be decoded from a hex string.
Signature · 1 line
2.40.1
PATCH 73/80 rust: puzzlefs: add HexError to WireFormatError and implement the From conversion
In reply to Ariel Miculas--- samples/rust/puzzle/error.rs | 11 +++++++++++ 1 file changed, 11 insertions(+)
samples/rust/puzzle/error.rs +11 -0
@@ -12,6 +12,7 @@ pub(crate) enum WireFormatError { CBORError(serde_cbor::Error), KernelError(kernel::error::Error), TryReserveError(TryReserveError), + HexError(hex::FromHexError), }impl WireFormatError {
@@ -23,6 +24,7 @@ pub(crate) fn to_errno(&self) -> c_int { WireFormatError::CBORError(..) => kernel::error::Error::to_errno(EINVAL), WireFormatError::KernelError(e) => kernel::error::Error::to_errno(*e), WireFormatError::TryReserveError(_) => kernel::error::Error::to_errno(EINVAL), + WireFormatError::HexError(_) => kernel::error::Error::to_errno(EINVAL), } }
@@ -40,6 +42,7 @@ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { WireFormatError::CBORError(_) => f.write_str("CBOR error"), WireFormatError::KernelError(_) => f.write_str("Kernel error"), WireFormatError::TryReserveError(_) => f.write_str("TryReserveError"), + WireFormatError::HexError(_) => f.write_str("HexError"), } } }
@@ -70,3 +73,11 @@ fn from(source: TryReserveError) -> Self {WireFormatError ::TryReserveError (source) } } + +#[allow(unused_qualifications)] +impl core::convert::From <hex::FromHexError >for WireFormatError { +#[allow(deprecated)] +fn from (source: hex::FromHexError ) ->Self { +WireFormatError ::HexError (source) + } +}
Signature · 1 line
2.40.1
PATCH 74/80 rust: puzzlefs: display the error value for WireFormatError::KernelError
In reply to Ariel Miculas--- samples/rust/puzzle/error.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
samples/rust/puzzle/error.rs +1 -1
@@ -40,7 +40,7 @@ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { WireFormatError::SeekOtherError => f.write_str("cannot seek to other blob"), WireFormatError::ValueMissing => f.write_str("no value present"), WireFormatError::CBORError(_) => f.write_str("CBOR error"), - WireFormatError::KernelError(_) => f.write_str("Kernel error"), + WireFormatError::KernelError(e) => write!(f, "Kernel error {:?}", e.to_errno()), WireFormatError::TryReserveError(_) => f.write_str("TryReserveError"), WireFormatError::HexError(_) => f.write_str("HexError"), }
Signature · 1 line
2.40.1
PATCH 75/80 samples: puzzlefs: add Rootfs and Digest structs to types.rs
In reply to Ariel Miculas--- samples/rust/puzzle/types.rs | 75 ++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+)
samples/rust/puzzle/types.rs +75 -0
@@ -8,7 +8,10 @@mod cbor_helpers;use crate ::puzzle::error::Result ;pub (crate )use cbor_helpers::{cbor_get_array_size, cbor_size_of_list_header}; +use core::fmt; +use hex::{encode_hex_iter,FromHexError };use kernel::file; +use kernel::str::CStr ;#[derive( Deserialize , Debug )] pub (crate )struct InodeAdditional {
@@ -32,6 +35,8 @@ pub(crate) struct MetadataBlob { pub(crate) inode_count: usize, } +pub (crate )const SHA256_BLOCK_SIZE :usize =32 ; +fn read_one_from_slice <'a,T :Deserialize <'a>>(bytes: &'a [u8 ]) ->Result <T > {// serde complains when we leave extra bytes on the wire, which we often want to do. as a // hack, we create a streaming deserializer for the type we're about to read, and then only
@@ -41,6 +46,23 @@ fn read_one_from_slice<'a, T: Deserialize<'a>>(bytes: &'a [u8]) -> Result<T> { v.ok_or (WireFormatError ::ValueMissing ) } +#[derive( Deserialize , Debug )] +pub (crate )struct Rootfs { +pub (crate ) metadatas:Vec <BlobRef >, +// TODO: deserialize fs_verity_data, for the moment BTreeMap is not supported +#[allow(dead_code)] +pub (crate ) fs_verity_data: (), +#[allow(dead_code)] +pub (crate ) manifest_version:u64 , +} + +impl Rootfs { +pub (crate )fn open (file: file::RegularFile ) ->Result <Rootfs > { +let buffer = file.read_to_end ()?; +read_one_from_slice (&buffer) + } +} +#[derive( Debug , Clone , Copy , PartialEq , Eq )] pub (crate )enum BlobRefKind {Local ,
@@ -176,6 +198,59 @@ pub(crate) fn find_inode(&mut self, ino: Ino) -> Result<Option<Inode>> { } } +#[derive( Debug , Clone , PartialEq , Eq )] +pub (crate )struct Digest ([u8 ;SHA256_BLOCK_SIZE ]); + +impl Digest { +pub (crate )fn underlying (&self) -> [u8 ;SHA256_BLOCK_SIZE ] { +let mut dest = [0_u8 ;SHA256_BLOCK_SIZE ]; + dest.copy_from_slice (&self.0 ); + dest + } +} + +impl fmt::Display for Digest { +fn fmt (&self, f: &mut fmt::Formatter <'_>) -> fmt::Result { +let mut hex_string = +Vec ::from_iter_fallible (encode_hex_iter (&self.underlying ())).map_err (|_| fmt::Error )?; +// append NUL character + hex_string.try_push (0 ).map_err (|_| fmt::Error )?; +let hex_string =CStr ::from_bytes_with_nul (&hex_string).map_err (|_| fmt::Error )?; +write! (f,"{}" , hex_string) + } +} + +impl TryFrom <&CStr >for Digest { +type Error =FromHexError ; +fn try_from (s: &CStr ) -> kernel::error::Result <Self ,Self ::Error > { +let digest = hex::decode (s)?; +let digest: [u8 ;SHA256_BLOCK_SIZE ] = digest + .try_into () + .map_err (|_|FromHexError ::InvalidStringLength )?; +Ok (Digest (digest)) + } +} + +impl TryFrom <BlobRef >for Digest { +type Error =WireFormatError ; +fn try_from (v:BlobRef ) -> kernel::error::Result <Self ,Self ::Error > { +match v.kind { +BlobRefKind ::Other { digest } =>Ok (Digest (digest)), +BlobRefKind ::Local =>Err (WireFormatError ::LocalRefError ), + } + } +} + +impl TryFrom <&BlobRef >for Digest { +type Error =WireFormatError ; +fn try_from (v: &BlobRef ) -> kernel::error::Result <Self ,Self ::Error > { +match v.kind { +BlobRefKind ::Other { digest } =>Ok (Digest (digest)), +BlobRefKind ::Local =>Err (WireFormatError ::LocalRefError ), + } + } +} +#[derive( Deserialize , Debug )] pub (crate )struct DirEnt {pub (crate ) ino:Ino ,
Signature · 1 line
2.40.1
PATCH 76/80 samples: puzzlefs: implement the conversion from WireFormatError to kernel::error::Error
In reply to Ariel Miculas--- rust/kernel/error.rs | 2 +- samples/rust/puzzle/error.rs | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-)
rust/kernel/error.rs +1 -1
@@ -96,7 +96,7 @@ impl Error {/// /// It is a bug to pass an out-of-range `errno`. `EINVAL` would /// be returned in such a case. -pub (crate )fn from_errno (errno: core::ffi::c_int ) ->Error { +pub fn from_errno (errno: core::ffi::c_int ) ->Error {if errno < -(bindings::MAX_ERRNO as i32 ) || errno >=0 {// TODO: Make it a `WARN_ONCE` once available. crate ::pr_warn!(
samples/rust/puzzle/error.rs +8 -0
@@ -81,3 +81,11 @@ fn from(source: hex::FromHexError) -> Self {WireFormatError ::HexError (source) } } + +#[allow(unused_qualifications)] +impl core::convert::From <WireFormatError >for kernel::error::Error { +#[allow(deprecated)] +fn from (source:WireFormatError ) ->Self { + kernel::error::Error ::from_errno (source.to_errno ()) + } +}
Signature · 1 line
2.40.1
PATCH 77/80 rust: puzzlefs: read the puzzlefs image manifest instead of an individual metadata layer
In reply to Ariel MiculasThe puzzlefs image manifest is the file referenced by index.json and it contains the array of metadata layers that describe the puzzlefs image. This file represents the root of the puzzlefs filesystem since we can't parse json files. When this filesystem is mounted, usermode will need to read the tag from index.json, find the corresponding puzzlefs image manifest and pass it to the kernel module.
Due to the lack of BTreeMap in the kernel, only image manifests without fs verity information are supported for now.
--- samples/rust/puzzle.rs | 1 + samples/rust/puzzle/inode.rs | 23 ++++++++++++---- samples/rust/puzzle/oci.rs | 51 ++++++++++++++++++++++++++++++++++++ samples/rust/puzzlefs.rs | 23 +++++++--------- 4 files changed, 80 insertions(+), 18 deletions(-) create mode 100644 samples/rust/puzzle/oci.rs
samples/rust/puzzle.rs +1 -0
@@ -1,3 +1,4 @@pub (crate )mod error;pub (crate )mod inode; +pub (crate )mod oci;pub (crate )mod types;
samples/rust/puzzle/inode.rs +18 -5
@@ -3,10 +3,15 @@use crate ::puzzle::error::Result ;use crate ::puzzle::error::WireFormatError ; +use crate ::puzzle::oci::Image ;use crate ::puzzle::typesas format; +use crate ::puzzle::types::Digest ;use crate ::puzzle::types::{FileChunk ,Ino ,InodeAdditional ,MetadataBlob };use alloc::vec::Vec ; +use kernel::mount::Vfsmount ;use kernel::prelude::{ENOENT ,ENOTDIR }; +use kernel::str::CStr ; +use kernel::sync::Arc ;#[derive( Debug )] pub (crate )struct Inode {
@@ -17,17 +22,25 @@ pub(crate) struct Inode { }pub (crate )struct PuzzleFS { +pub (crate ) oci:Image , layers:Vec <format::MetadataBlob >, }impl PuzzleFS { -pub (crate )fn new (md:MetadataBlob ) ->Result <Self > { -let mut v =Vec ::new (); - v.try_push (md)?; -Ok (PuzzleFS { layers: v }) +pub (crate )fn open (vfsmount:Arc <Vfsmount >, rootfs_path: &CStr ) ->Result <PuzzleFS > { +let oci =Image ::open (vfsmount)?; +let rootfs = oci.open_rootfs_blob (rootfs_path)?; + +let layers = +Vec ::from_iter_fallible (rootfs.metadatas.iter ().map (|md| ->Result <MetadataBlob > { +let digest =Digest ::try_from (md)?; + oci.open_metadata_blob (&digest) + }))? + .process_results ()?; + +Ok (PuzzleFS { oci, layers }) } -// Temporary helper function used until PuzzleFs is integrated pub (crate )fn find_inode (&mut self, ino:u64 ) ->Result <Inode > {for layerin self.layers.iter_mut () {if let Some (inode) = layer.find_inode (ino)? {
samples/rust/puzzle/oci.rs new file +51 -0
51 lines not on this page · raw message
samples/rust/puzzlefs.rs +10 -13
@@ -13,7 +13,6 @@mod puzzle;use puzzle::inode::{Inode ,InodeMode ,PuzzleFS }; -use puzzle::types::MetadataBlob ;use kernel::fs::{DEntry ,INodeParams ,NeedsRoot ,NewSuperBlock ,RootDEntry };
@@ -67,7 +66,7 @@ fn puzzlefs_populate_dir(return Err (E2BIG ); } -let inode =Arc ::try_new (pfs.find_inode (ino).map_err (|_|EINVAL )?)?; +let inode =Arc ::try_new (pfs.find_inode (ino)?)?;match &inode.mode {InodeMode ::File { chunks: _ } => { let params = INodeParams {
@@ -152,19 +151,17 @@ fn fill_super(_data: (), sb: fs::NewSuperBlock<'_, Self>) -> Result<&fs::SuperBl }, )?; -let file = file::RegularFile ::from_path_in_root_mnt ( - &arc_vfs_mount, -c_str! ("997eed138af30d187e87d682dd2ae9f240fae78f668907a0519460b397c82467" ), - file::flags::O_RDONLY .try_into ().unwrap (), -0 , - )?; +let puzzlefs =PuzzleFS ::open ( + arc_vfs_mount, +c_str! ("2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b" ), + ); -// TODO: figure out how to go from WireFormatError to kernel::error::Error -let metadata =MetadataBlob ::new (file).map_err (|_|EINVAL )?; -pr_info! ("number of inodes {:?}\n" , metadata.inode_count); +if let Err (ref e) = puzzlefs { +pr_info! ("error opening puzzlefs {e}\n" ); + } -let mut puzzlefs =PuzzleFS ::new (metadata).map_err (|_|EINVAL )?; -let root_inode =Arc ::try_new (puzzlefs.find_inode (1 ).map_err (|_|EINVAL )?)?; +let mut puzzlefs = puzzlefs?; +let root_inode =Arc ::try_new (puzzlefs.find_inode (1 )?)?;let root =try_new_populated_root_puzzlefs_dentry (&sb, &mut puzzlefs, root_inode)?;let sb = sb.init_root (root)?;
Signature · 1 line
2.40.1
PATCH 78/80 rust: puzzlefs: rename PuzzleFs to PuzzleFsModule to avoid confusion with the PuzzleFS struct
In reply to Ariel Miculas--- samples/rust/puzzlefs.rs | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
samples/rust/puzzlefs.rs +9 -9
@@ -17,13 +17,13 @@use kernel::fs::{DEntry ,INodeParams ,NeedsRoot ,NewSuperBlock ,RootDEntry };module_fs! { -type :PuzzleFs , +type :PuzzleFsModule , name:"puzzlefs" , author:"Ariel Miculas" , license:"GPL" , } -struct PuzzleFs ; +struct PuzzleFsModule ;#[derive( Debug )] struct PuzzlefsInfo {
@@ -31,7 +31,7 @@ struct PuzzlefsInfo { }#[vtable] -impl fs::Context <Self >for PuzzleFs { +impl fs::Context <Self >for PuzzleFsModule {type Data = (); kernel::define_fs_params! {(),
@@ -55,9 +55,9 @@ fn try_new() -> Result { }fn puzzlefs_populate_dir ( - sb: &NewSuperBlock <'_,PuzzleFs ,NeedsRoot >, + sb: &NewSuperBlock <'_,PuzzleFsModule ,NeedsRoot >, pfs: &mut PuzzleFS , - parent: &DEntry <PuzzleFs >, + parent: &DEntry <PuzzleFsModule >, ino:u64 , name: &CStr , recursion: usize,
@@ -110,10 +110,10 @@ fn puzzlefs_populate_dir(/// Creates a new root dentry populated with the given entries. fn try_new_populated_root_puzzlefs_dentry ( - sb: &NewSuperBlock <'_,PuzzleFs ,NeedsRoot >, + sb: &NewSuperBlock <'_,PuzzleFsModule ,NeedsRoot >, pfs: &mut PuzzleFS , - root_value: <PuzzleFs as fs::Type >::INodeData , -) ->Result <RootDEntry <PuzzleFs >> { + root_value: <PuzzleFsModule as fs::Type >::INodeData , +) ->Result <RootDEntry <PuzzleFsModule >> { let root_inode = sb.sb.try_new_dcache_dir_inode(INodeParams { mode: 0o755, ino: root_value.inode.ino,
@@ -125,7 +125,7 @@ fn try_new_populated_root_puzzlefs_dentry(Ok (root) } -impl fs::Type for PuzzleFs { +impl fs::Type for PuzzleFsModule {type Context =Self ;// this is Arc so it can be cloned in puzzlefs_populate_dir type INodeData =Arc <Inode >;
Signature · 1 line
2.40.1
PATCH 79/80 rust: puzzlefs: add support for reading files
In reply to Ariel Miculas--- samples/rust/puzzle/inode.rs | 68 ++++++++++++++++++++++++++++++++---- samples/rust/puzzle/oci.rs | 32 +++++++++++++---- samples/rust/puzzle/types.rs | 10 +++--- samples/rust/puzzlefs.rs | 59 ++++++++++++++----------------- 4 files changed, 118 insertions(+), 51 deletions(-)
samples/rust/puzzle/inode.rs +61 -7
@@ -8,10 +8,10 @@use crate ::puzzle::types::Digest ;use crate ::puzzle::types::{FileChunk ,Ino ,InodeAdditional ,MetadataBlob };use alloc::vec::Vec ; +use core::cmp::min;use kernel::mount::Vfsmount ;use kernel::prelude::{ENOENT ,ENOTDIR };use kernel::str::CStr ; -use kernel::sync::Arc ;#[derive( Debug )] pub (crate )struct Inode {
@@ -21,14 +21,16 @@ pub(crate) struct Inode { pub(crate) additional: Option<InodeAdditional>, } +#[derive( Debug )] pub (crate )struct PuzzleFS {pub (crate ) oci:Image , layers:Vec <format::MetadataBlob >, }impl PuzzleFS { -pub (crate )fn open (vfsmount:Arc <Vfsmount >, rootfs_path: &CStr ) ->Result <PuzzleFS > { -let oci =Image ::open (vfsmount)?; +pub (crate )fn open (oci_root_dir: &CStr , rootfs_path: &CStr ) ->Result <PuzzleFS > { +let vfs_mount =Vfsmount ::new_private_mount (oci_root_dir)?; +let oci =Image ::open (vfs_mount)?;let rootfs = oci.open_rootfs_blob (rootfs_path)?; let layers =
@@ -41,8 +43,8 @@ pub(crate) fn open(vfsmount: Arc<Vfsmount>, rootfs_path: &CStr) -> Result<PuzzleOk (PuzzleFS { oci, layers }) } -pub (crate )fn find_inode (&mut self, ino:u64 ) ->Result <Inode > { -for layerin self.layers.iter_mut () { +pub (crate )fn find_inode (&self, ino:u64 ) ->Result <Inode > { +for layerin self.layers.iter () {if let Some (inode) = layer.find_inode (ino)? {return Inode ::new (layer, inode); }
@@ -52,7 +54,7 @@ pub(crate) fn find_inode(&mut self, ino: u64) -> Result<Inode> { }impl Inode { -fn new (layer: &mut MetadataBlob , inode: format::Inode ) ->Result <Inode > { +fn new (layer: &MetadataBlob , inode: format::Inode ) ->Result <Inode > {let mode =match inode.mode { format::InodeMode ::Reg { offset } => {let chunks = layer.read_file_chunks (offset)?;
@@ -64,7 +66,7 @@ fn new(layer: &mut MetadataBlob, inode: format::Inode) -> Result<Inode> { layer .read_dir_list (offset)? .entries - .iter_mut () + .iter () .map(|de| (de.name.try_clone().unwrap(), de.ino)), )?;// Unstable sort is used because it avoids memory allocation
@@ -94,3 +96,55 @@ pub(crate) enum InodeMode { Dir { entries: Vec<(Vec<u8>, Ino)> },Other , } + +pub (crate )fn file_read ( + oci: &Image , + inode: &Inode , + offset:usize , + data: &mut [u8 ], +) ->Result <usize > { +let chunks =match &inode.mode { +InodeMode ::File { chunks } => chunks, + _ =>return Err (WireFormatError ::from_errno (ENOTDIR )), + }; + +// TODO: fix all this casting... +let end = offset + data.len (); + +let mut file_offset =0 ; +let mut buf_offset =0 ; +for chunkin chunks { +// have we read enough? +if file_offset > end { +break ; + } + +// should we skip this chunk? +if file_offset + (chunk.lenas usize ) < offset { + file_offset += chunk.lenas usize ; +continue ; + } + +let addl_offset =if offset > file_offset { + offset - file_offset + }else { +0 + }; + +// ok, need to read this chunk; how much? +let left_in_buf = data.len () - buf_offset; +let to_read =min (left_in_buf, chunk.lenas usize - addl_offset); + +let start = buf_offset; +let finish = start + to_read; + file_offset += addl_offset; + +// how many did we actually read? +let n = oci.fill_from_chunk (chunk.blob, addl_offsetas u64 , &mut data[start..finish])?; + file_offset += n; + buf_offset += n; + } + +// discard any extra if we hit EOF +Ok (buf_offset) +}
samples/rust/puzzle/oci.rs +26 -6
@@ -1,19 +1,21 @@ -use crate ::puzzle::error::Result ; +use crate ::puzzle::error::{Result ,WireFormatError }; +use crate ::puzzle::typesas format;use crate ::puzzle::types::{Digest ,MetadataBlob ,Rootfs };use kernel::c_str;use kernel::file;use kernel::file::RegularFile ;use kernel::mount::Vfsmount ; -use kernel::pr_info; +use kernel::pr_debug; +use kernel::prelude::ENOTSUPP ;use kernel::str::{CStr ,CString }; -use kernel::sync::Arc ; +#[derive( Debug )] pub (crate )struct Image { - vfs_mount:Arc <Vfsmount >, +pub (crate ) vfs_mount:Vfsmount , }impl Image { -pub (crate )fn open (vfsmount:Arc <Vfsmount >) ->Result <Self > { +pub (crate )fn open (vfsmount:Vfsmount ) ->Result <Self > {Ok (Image { vfs_mount: vfsmount, })
@@ -26,7 +28,7 @@ pub(crate) fn blob_path_relative(&self) -> &CStr {fn open_raw_blob (&self, digest: &Digest ) ->Result <RegularFile > {let filename =CString ::try_from_fmt (format_args! ("{}/{digest}" , self.blob_path_relative ()))?; -pr_info! ("trying to open {:?}\n" , &filename); +pr_debug! ("trying to open {:?}\n" , &filename);let file =RegularFile ::from_path_in_root_mnt ( &self.vfs_mount,
@@ -48,4 +50,22 @@ pub(crate) fn open_rootfs_blob(&self, path: &CStr) -> Result<Rootfs> {let rootfs =Rootfs ::open (self.open_raw_blob (&digest)?)?;Ok (rootfs) } + +pub (crate )fn fill_from_chunk ( + &self, + chunk: format::BlobRef , + addl_offset:u64 , + buf: &mut [u8 ], + ) ->Result <usize > { +let digest = &<Digest >::try_from (chunk)?; + +let blob =if chunk.compressed { +return Err (WireFormatError ::KernelError (ENOTSUPP )); + }else { + self.open_raw_blob (digest)? + }; + +let n = blob.read_with_offset (buf, chunk.offset + addl_offset)?; +Ok (n) + } }
samples/rust/puzzle/types.rs +5 -5
@@ -148,28 +148,28 @@ pub(crate) fn new(mut f: file::RegularFile) -> Result<MetadataBlob> { }) } -pub (crate )fn seek_ref (&mut self, r: &BlobRef ) ->Result <u64 > { +pub (crate )fn seek_ref (&self, r: &BlobRef ) ->Result <u64 > {match r.kind {BlobRefKind ::Other { .. } =>Err (WireFormatError ::SeekOtherError ),BlobRefKind ::Local =>Ok (r.offset), } } -pub (crate )fn read_file_chunks (&mut self, offset:u64 ) ->Result <Vec <FileChunk >> { +pub (crate )fn read_file_chunks (&self, offset:u64 ) ->Result <Vec <FileChunk >> {read_one_from_slice ::<FileChunkList >(&self.mmapped_region[offsetas usize ..]) .map (|cl| cl.chunks) } -pub (crate )fn read_dir_list (&mut self, offset:u64 ) ->Result <DirList > { +pub (crate )fn read_dir_list (&self, offset:u64 ) ->Result <DirList > {read_one_from_slice (&self.mmapped_region[offsetas usize ..]) } -pub (crate )fn read_inode_additional (&mut self, r: &BlobRef ) ->Result <InodeAdditional > { +pub (crate )fn read_inode_additional (&self, r: &BlobRef ) ->Result <InodeAdditional > {let offset = self.seek_ref (r)?as usize ;read_one_from_slice (&self.mmapped_region[offset..]) } -pub (crate )fn find_inode (&mut self, ino:Ino ) ->Result <Option <Inode >> { +pub (crate )fn find_inode (&self, ino:Ino ) ->Result <Option <Inode >> {let mut left =0 ;let mut right = self.inode_count;
samples/rust/puzzlefs.rs +26 -33
128 lines not on this page · raw message
Signature · 1 line
2.40.1
PATCH 80/80 rust: puzzlefs: add oci_root_dir and image_manifest filesystem parameters
In reply to Ariel MiculasThese parameters are passed when mounting puzzlefs using '-o' option of mount:
-o oci_root_dir="/path/to/oci/dir"
-o image_manifest="root_hash_of_image_manifest"
For a particular manifest in the manifests array in index.json (located in the oci_root_dir), the root hash of the image manifest is found in the digest field.
It would be nicer if we could pass the tag, but we don't support json deserialization.
Example of mount:
mount -t puzzlefs -o oci_root_dir="/home/puzzlefs_oci" -o \
image_manifest="2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b" \
none /mnt
--- samples/rust/puzzlefs.rs | 63 ++++++++++++++++++++++++++-------------- 1 file changed, 41 insertions(+), 22 deletions(-)
samples/rust/puzzlefs.rs +41 -22
@@ -7,6 +7,7 @@use kernel::{ c_str, file, fs, io_buffer::IoBufferWriter , + str::CString , sync::{Arc ,ArcBorrow }, };
@@ -29,27 +30,29 @@ struct PuzzlefsInfo { puzzlefs: Arc<PuzzleFS>, } +#[derive( Default )] +struct PuzzleFsParams { + oci_root_dir:Option <CString >, + image_manifest:Option <CString >, +} +#[vtable] impl fs::Context <Self >for PuzzleFsModule { -type Data = (); - - kernel::define_fs_params! {(), - {flag,"flag" , |_, v| {pr_info! ("flag passed-in: {v}\n" );Ok (()) } }, - {flag_no,"flagno" , |_, v| {pr_info! ("flagno passed-in: {v}\n" );Ok (()) } }, - {bool ,"bool" , |_, v| {pr_info! ("bool passed-in: {v}\n" );Ok (()) } }, - {u32 ,"u32" , |_, v| {pr_info! ("u32 passed-in: {v}\n" );Ok (()) } }, - {u32oct,"u32oct" , |_, v| {pr_info! ("u32oct passed-in: {v}\n" );Ok (()) } }, - {u32hex,"u32hex" , |_, v| {pr_info! ("u32hex passed-in: {v}\n" );Ok (()) } }, - {s32,"s32" , |_, v| {pr_info! ("s32 passed-in: {v}\n" );Ok (()) } }, - {u64 ,"u64" , |_, v| {pr_info! ("u64 passed-in: {v}\n" );Ok (()) } }, - {string,"string" , |_, v| {pr_info! ("string passed-in: {v}\n" );Ok (()) } }, - {enum ,"enum" , [("first" ,10 ), ("second" ,20 )], |_, v| { -pr_info! ("enum passed-in: {v}\n" );Ok (()) } - }, +type Data =Box <PuzzleFsParams >; + + kernel::define_fs_params! {Box <PuzzleFsParams >, + {string,"oci_root_dir" , |s, v| { + s.oci_root_dir =Some (CString ::try_from_fmt (format_args! ("{v}" ))?); +Ok (()) + }}, + {string,"image_manifest" , |s, v| { + s.image_manifest =Some (CString ::try_from_fmt (format_args! ("{v}" ))?); +Ok (()) + }}, } -fn try_new () ->Result { -Ok (()) +fn try_new () ->Result <Self ::Data > { +Ok (Box ::try_new (PuzzleFsParams ::default ())?) } }
@@ -134,11 +137,27 @@ impl fs::Type for PuzzleFsModule {const FLAGS :i32 = fs::flags::USERNS_MOUNT ;const DCACHE_BASED :bool =true ; -fn fill_super (_data: (), sb: fs::NewSuperBlock <'_,Self >) ->Result <&fs::SuperBlock <Self >> { -let puzzlefs =PuzzleFS ::open ( -c_str! ("/home/puzzlefs_oci" ), -c_str! ("2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b" ), - ); +fn fill_super ( + data:Box <PuzzleFsParams >, + sb: fs::NewSuperBlock <'_,Self >, + ) ->Result <&fs::SuperBlock <Self >> { +let oci_root_dir =match data.oci_root_dir { +Some (val) => val, +None => { +pr_err! ("missing oci_root_dir parameter!\n" ); +return Err (ENOTSUPP ); + } + }; + +let image_manifest =match data.image_manifest { +Some (val) => val, +None => { +pr_err! ("missing image_manifest parameter!\n" ); +return Err (ENOTSUPP ); + } + }; + +let puzzlefs =PuzzleFS ::open (&oci_root_dir, &image_manifest);if let Err (ref e) = puzzlefs {pr_info! ("error opening puzzlefs {e}\n" );
Signature · 1 line
2.40.1
Quoted 4 lines · Ariel Miculas
This is a proof of concept driver written for the PuzzleFS next-generation container filesystem [1]. I've included a short abstract about puzzlefs further below. This driver is based on the rust-next
Thanks for preparing this RFC Ariel!
I have left some procedural notes here and there.
Cheers,
Miguel
Christian Brauner
maintainerQuoted 3 lines · Ariel Miculas
Hi all!
This is a proof of concept driver written for the PuzzleFS
Uhm, the PuzzleFS filesystem isn't actually sent to fsdevel? Yet I see tons of patches in there that add wrappers to our core fs data structures. I even see a ton of new files that all fall clearly into fsdevel territory:
create mode 100644 rust/kernel/cred.rs
create mode 100644 rust/kernel/delay.rs
create mode 100644 rust/kernel/driver.rs
create mode 100644 rust/kernel/file.rs
create mode 100644 rust/kernel/fs.rs
create mode 100644 rust/kernel/fs/param.rs
create mode 100644 rust/kernel/io_buffer.rs
create mode 100644 rust/kernel/iov_iter.rs
create mode 100644 rust/kernel/mm.rs
create mode 100644 rust/kernel/mount.rs
create mode 100644 rust/kernel/pages.rs
There's also quite a lot of new mm/ in there, no?
Any wrappers and code for core fs should be maintained as part of fs. Rust shouldn't become a way to avoid our reviews once you have a few wrappers added somewhere.
Quoted 677 lines
next-generation container filesystem [1]. I've included a short abstract about puzzlefs further below. This driver is based on the rust-next branch, on top of which I've backported the filesystem abstractions from Wedson Almeida Filho [2][3] and Miguel Ojeda's third-party crates
support: proc-macro2, quote, syn, serde andserde_derive[4]. I've added the additional third-party crates serde_cbor[5] and hex [6]. Then I've adapted the user space puzzlefs code [1] so that the puzzlefs kernel module could present the directory hierarchy and implement the basic read functionality.
For some additional context, puzzlefs was started by Tycho Andersen and it's the successor of atomfs. This FOSDEM presentation from 2019 [12] covers the rationale for a new oci image format and presents a higher level overview of our goals with puzzlefs.
I've split the rest of the cover letter in following sections (using a markdown style):
* Example: it describes a practical example of what was achieved
* Limitations: it presents the existing limitations of this POC
* Upstreaming steps: it describes the steps needed for upstreaming this driver
* Setup: it shows how to setup the necessary environment for testing the puzzlefs driver
* Puzzlefs abstract: it provides a short overview of puzzlefs# Example
An example is provided below:```
~ # cat/proc/filesystems| grep puzzlefs
nodev puzzlefs
~ # mount -t puzzlefs -o oci_root_dir="/home/puzzlefs_oci" -o image_manifest="2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b" none /mnt ~ # ls -lR/mnt//mnt/:
total 0drwxr-xr-x 2 0 0 0 Jun 8 12:26 dir-1 drwxr-xr-x 2 0 0 0 Jun 8 12:26 dir-2 drwxr-xr-x 2 0 0 0 Jun 8 12:26 dir-3 drwxr-xr-x 2 0 0 0 Jun 8 12:26 dir-4(quoted diff) +0
-2-rw-r--r-- 1 0 0 0 Jun 8 12:26 file1 -rw-r--r-- 1 0 0 0 Jun 8 12:26 file2
/mnt/dir-1:
total 0
/mnt/dir-2:
total 0
/mnt/dir-3:
total 0
/mnt/dir-4:
total 0
~ # cat/mnt/file2
ana are mere bla bla bla
~ # wc/mnt/file1202 202 5454 /mnt/file1 ``` In this example, /home/puzzlefs_oci is the puzzlefs oci directory: ```~ # ls -lR
/home/puzzlefs_oci//home/puzzlefs_oci/:
total 8
drwxr-xr-x 3 1000 1000 0 Jun 8 14:33 blobs(quoted diff) +0
-2-rw-r--r-- 1 1000 1000 266 Jun 8 14:33 index.json -rw-r--r-- 1 1000 1000 37 Jun 8 14:33 oci-layout
/home/puzzlefs_oci/blobs:
total 0
drwxr-xr-x 2 1000 1000 0 Jun 8 14:33 sha256
/home/puzzlefs_oci/blobs/sha256:
total 16(quoted diff) +0
-3-rw------- 1 1000 1000 89 Jun 8 14:33 2d6602d678140540dc7e96de652a76a8b16eb -rw------- 1 1000 1000 925 Jun 8 14:33 4df03518eea406343dbb55046720f6a478881 -rw------- 1 1000 1000 5479 Jun 8 14:33 d86a87b19bd9a2fec0d31687c1d669cdb59eb``` `2d6602d678140540dc7e96de652a76a8b16eb` is the puzzlefs image manifest hash for the first_try tag: ``` $ cat /tmp/oci-simple/index.json | jq . { "schemaVersion": -1, "manifests": [ { "digest": "sha256:2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b", "size": 89, "media_type": "application/vnd.puzzlefs.image.rootfs.v1", "annotations": { "org.opencontainers.image.ref.name": "first_try" } } ], "annotations": {} } ```I will describe how to build a puzzlefs image in the
Setupsection, at step 5.# Limitations
One limitation is that the puzzlefs driver doesn't implement any lookup functionality and instead it inserts every directory entry into the dcache during init (see theDCACHE_BASEDconstant). This is similar to how the samplerust_fsdriver works, but the goal is to implement proper lookup functions. However, more filesystem abstractions need to be implemented before this can be achieved.# Upstreaming steps
Before the puzzlefs driver can be upstreamed, the following need to be merged:
* Wedson's filesystem abstractions [3]
* the necessary third-party crates [4] (with the preliminary discussion about whether this is desirable)# Setup
My setup is based on Wedson's tutorial [8]. Next, I will describe the necessary steps to build an initrd and run a custom kernel under qemu.1. Get the linux rust-next branch [9] and apply this patchset
2. Follow the rust quickstart guide [10]
3. configure and build the kernel
``` $ make LLVM=1 allnoconfig qemu-busybox-min.config rust.config $ make LLVM=1 -j$(nproc) ```4. setup busybox
``` git clone git://git.busybox.net/busybox cd busybox make menuconfig # enable 'Build static binary' config make make install ```This will create the
_installdirectory with the rootfs inside it.5. create a home directory in the rootfs and copy a puzzlefs oci directory in home/puzzlefs_oci
To create a puzzlefs oci directory:
* download this custom puzzlefs repository [11] (it's custom because we want to build an image without verity data)
* runmake release
* create a simple filesystem structure with a few directories and files (I've created one at ../test-puzzlefs/simple_rootfs)
* build a puzzlefs oci image at~/work/busybox/_install/home/puzzlefs_oci(replace this path with your busybox path) with the tagfirst_try:``` $ target/release/puzzlefs build --omit-verity \ ../test-puzzlefs/simple_rootfs ~/work/busybox/_install/home/puzzlefs_oci \ first_try ``` * get first_try's image manifest from index.json (inside `puzzlefs_oci`) ``` $ cat index.json | jq . | grep digest "digest": "sha256:2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b", ```6. add the following 'init' script in the busybox rootfs (rootfs path defaults to `./_install'):
``` #!/bin/sh mount -t devtmpfs none /dev mkdir -p /proc mount -t proc none /proc ifconfig lo up udhcpc -i eth0 mkdir /mnt mount -t puzzlefs -o oci_root_dir="/home/puzzlefs_oci" -o \ image_manifest="2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b" \ none /mnt setsid sh -c 'exec sh -l </dev/ttyS0 >/dev/ttyS0 2>&1' ```Make sure to replace the
image_manifestwith your own digest. This init script will be passed to rdinit in the kernel command line.7. generate the initramfs
Assuming busybox is in
~/work/busybox:``` cd ~/work/busybox/_install && find . | cpio -H newc -o | gzip > ../ramdisk.img ```This will generate a compressed ramdisk image in
~/work/busybox/ramdisk.img.8. run with qemu (this assumes the linux tree is at '../linux' and busybox is at '../busybox'):
``` qemu-system-x86_64 \ -accel kvm \ -cpu host \ -m 4G \ -initrd ../busybox/ramdisk.img \ -kernel ../linux/arch/x86/boot/bzImage \ -nographic \ -append 'console=ttyS0 nokaslr debug rdinit=/init' \ -nic user,model=rtl8139 \ -no-reboot ```9. Check whether puzzlefs has been successfully mounted
``` ~ # mount | grep puzzlefs none on /mnt type puzzlefs (rw,relatime) ~ # ls /mnt/ dir-1 dir-2 dir-3 dir-4 file1 file2 ```# Puzzlefs abstract
Puzzlefs [1] is a container filesystem designed to address the limitations of the existing OCI format. The main goals of the project are reduced duplication, reproducible image builds, direct mounting support and memory safety guarantees, some inspired by the OCIv2 design document [7].Reduced duplication is achieved using the content defined chunking algorithm FastCDC. This implementation allows chunks to be shared among layers. Building a new layer starting from an existing one allows reusing most of the chunks.
Another goal of the project is reproducible image builds, which is achieved by defining a canonical representation of the image format.
Direct mounting support is a key feature of puzzlefs and, together with fs-verity, it provides data integrity. Currently, puzzlefs is implemented as a userspace filesystem (FUSE). A read-only kernel filesystem driver is underway.
Lastly, memory safety is critical to puzzlefs, leading to the decision to implement it in Rust. Another goal is to share the same code between user space and kernel space in order to provide one secure implementation.
[1] https://github.com/anuvu/puzzlefs
[2] https://github.com/wedsonaf/linux/tree/fs
[3] https://github.com/Rust-for-Linux/linux/issues/1004
[4] https://github.com/Rust-for-Linux/linux/pull/1007
[5] https://docs.rs/serde_cbor/latest/serde_cbor/
[6] https://docs.rs/hex/0.4.3/hex/
[7] https://hackmd.io/@cyphar/ociv2-brainstorm
[8] https://www.youtube.com/watch?v=tPs1uRqOnlk
[9] https://github.com/Rust-for-Linux/linux/tree/rust-next
[10] https://docs.kernel.org/rust/quick-start.html
[11] https://github.com/ariel-miculas/puzzlefs/tree/support-no-verity-data
[12] https://archive.fosdem.org/2019/schedule/event/containers_atomfs/Ariel Miculas (58):
rust: kernel: add libraries required by the filesystem abstractions
rust: kernel: backport the delay module from the rust branch
rust: kernel: addcontainer_ofmacro
rust: kernel: addoffset_ofmacro
drop: Addcrate::pr_warndeclaration
rust: kernel: renamefrom_kernel_errnotofrom_errno
rust: kernel: Renamefrom_pointertofrom_foreingandinto_pointerto
into_foreign
rust: kernel: addcount_paren_itemsmacro, needed bydefine_fs_params
macro
rust: helpers: add missing rust helper 'alloc_pages'
kernel: configs: add qemu-busybox-min.config
rust: kernel: format the rust code
samples: puzzlefs: add initial puzzlefs sample, copied fromrust_fs.rs
kernel: configs: enable rust samples in rust.config
AddSAMPLE_RUST_SERDEin rust.config
rust: kernel: fix compile errors after rebase to rust-next
rust:serde_cbor: import crate
rust:serde_cbor: add SPDX License Identifiers
rust:serde_cbor: addno_fp_fmt_parsesupport
rust:Kbuild: enableserde_cbor
samples: rust: add cbor serialize/deserialize example
rust:serde_cbor: add support for serde_cbor'sfrom_slicemethod by
using a customalloc_kernelfeature
rust: serde: add support for deserializing Vec withkernel_alloc
feature
rust: file: Replace UnsafeCell with Opaque for File
rust: kernel: implementfmt::Debugfor CString
samples: puzzlefs: rename RustFs to PuzzleFs
samples: puzzlefs: add basic deserializing support for the puzzlefs
metadata
rust: file: present the filesystem context to the open function
rust: kernel: add an abstraction over vfsmount to allow cloning a new
private mount
rust: file: addfrom_path,from_path_in_root_mntandread_with_offset
methods to File
samples: puzzlefs: pass the Vfsmount structure from open to read and
return the contents of the data file inside/home/puzzlefs_oci
rust: file: movefrom_path,from_path_in_root_mntandread_with_offset
methods to a RegularFile newtype
rust: file: ensure RegularFile can only create regular files
rust: file: addget_posmethod to RegularFile
rust: file: add methodsread_to_end,get_file_sizeandupdate_posto
RegularFile
rust: file: define a minimal Read trait and implement it for
RegularFile
samples: puzzlefs: addcbor_get_array_sizemethod
samples: puzzlefs: add KernelError to WireFormatError and implement
From conversion
samples: puzzlefs: implement new for MetadataBlob
samples: puzzlefs: build puzzlefs into the kernel, thus avoiding the
need to export rust symbols
rust: alloc: addtry_clonefor Vec<T>
rust: alloc: addfrom_iter_falliblefor Vec<T>
samples: puzzlefs: implementto_errnoandfrom_errnofor
WireFormatError
samples: puzzlefs: add TryReserveError (and from conversion) to
WireFormatError
samples: puzzlefs: add higher level inode related functionality
samples: puzzlefs: populate the directory entries with the inodes from
the puzzlefs metadata file
rust: hex: import crate
rust: hex: add SPDX license identifiers
rust:Kbuild: enablehex
rust: hex: implement FromHex trait andhex::decodeusing a custom
kernel_allocfeature
rust: hex: addencode_hex_iterandencode_hex_upper_itermethods
rust: puzzlefs: add HexError to WireFormatError and implement the From
conversion
rust: puzzlefs: display the error value for
WireFormatError::KernelError
samples: puzzlefs: add Rootfs and Digest structs totypes.rs
samples: puzzlefs: implement the conversion from WireFormatError to
kernel::error::Error
rust: puzzlefs: read the puzzlefs image manifest instead of an
individual metadata layer
rust: puzzlefs: rename PuzzleFs to PuzzleFsModule to avoid confusion
with the PuzzleFS struct
rust: puzzlefs: add support for reading files
rust: puzzlefs: addoci_root_dirandimage_manifestfilesystem
parametersMiguel Ojeda (15):
rust: proc-macro2: import crate
rust: proc-macro2: add SPDX License Identifiers
rust: proc-macro2: removeunicode_identdependency
rust: quote: import crate
rust: quote: add SPDX License Identifiers
rust: syn: import crate
rust: syn: add SPDX License Identifiers
rust: syn: removeunicode-identdependency
rust: serde: import crate
rust: serde: addno_fp_fmt_parsesupport
rust: serde: add SPDX License Identifiers
rust:serde_derive: import crate
rust:serde_derive: add SPDX License Identifiers
rust:Kbuild: enableproc-macro2,quote,syn,serdeand
serde_derive
rust: testserdesupportWedson Almeida Filho (7):
rust: add definitions for ref-counted inodes and dentries
rust: add ability to register a file system
rust: define fs context
rust: add support for file system parameters
rust: allow fs driver to initialise new superblocks
rust: addmodule_fsmacro
WIP: rust: allow fs to be populatedMakefile | 14 +- arch/x86/configs/qemu-busybox-min.config | 11 + kernel/configs/qemu-busybox-min.config | 56 + kernel/configs/rust.config | 11 + rust/.gitignore | 1 + rust/Makefile | 232 +- rust/alloc/vec/mod.rs | 48 + rust/bindings/bindings_helper.h | 14 + rust/bindings/lib.rs | 5 + rust/helpers.c | 76 + rust/hex/error.rs | 78 + rust/hex/lib.rs | 506 +++ rust/hex/serde.rs | 104 + rust/kernel/cred.rs | 46 + rust/kernel/delay.rs | 104 + rust/kernel/driver.rs | 28 + rust/kernel/error.rs | 52 +- rust/kernel/file.rs | 1117 ++++++ rust/kernel/fs.rs | 1478 ++++++++ rust/kernel/fs/param.rs | 558 +++ rust/kernel/io_buffer.rs | 153 + rust/kernel/iov_iter.rs | 81 + rust/kernel/lib.rs | 83 + rust/kernel/mm.rs | 149 + rust/kernel/mount.rs | 66 + rust/kernel/pages.rs | 144 + rust/kernel/str.rs | 6 + rust/kernel/test_serde.rs | 26 + rust/kernel/test_serde/de.rs | 439 +++ rust/kernel/test_serde/error.rs | 73 + rust/kernel/test_serde/ser.rs | 466 +++ rust/kernel/user_ptr.rs | 175 + rust/proc-macro2/detection.rs | 77 + rust/proc-macro2/fallback.rs | 1004 ++++++ rust/proc-macro2/lib.rs | 1341 ++++++++ rust/proc-macro2/marker.rs | 20 + rust/proc-macro2/parse.rs | 874 +++++ rust/proc-macro2/rcvec.rs | 144 + rust/proc-macro2/wrapper.rs | 996 ++++++ rust/quote/ext.rs | 112 + rust/quote/format.rs | 170 + rust/quote/ident_fragment.rs | 88 + rust/quote/lib.rs | 1436 ++++++++ rust/quote/runtime.rs | 440 +++ rust/quote/spanned.rs | 45 + rust/quote/to_tokens.rs | 211 ++ rust/serde/de/format.rs | 32 + rust/serde/de/ignored_any.rs | 246 ++ rust/serde/de/impls.rs | 2755 +++++++++++++++ rust/serde/de/mod.rs | 2313 +++++++++++++ rust/serde/de/seed.rs | 21 + rust/serde/de/utf8.rs | 48 + rust/serde/de/value.rs | 1718 ++++++++++ rust/serde/integer128.rs | 84 + rust/serde/lib.rs | 351 ++ rust/serde/macros.rs | 238 ++ rust/serde/private/de.rs | 2997 ++++++++++++++++ rust/serde/private/doc.rs | 161 + rust/serde/private/mod.rs | 52 + rust/serde/private/ser.rs | 1316 +++++++ rust/serde/private/size_hint.rs | 23 + rust/serde/ser/fmt.rs | 180 + rust/serde/ser/impls.rs | 987 ++++++ rust/serde/ser/impossible.rs | 218 ++ rust/serde/ser/mod.rs | 1992 +++++++++++ rust/serde/std_error.rs | 50 + rust/serde_cbor/de.rs | 1370 ++++++++ rust/serde_cbor/error.rs | 320 ++ rust/serde_cbor/lib.rs | 371 ++ rust/serde_cbor/read.rs | 647 ++++ rust/serde_cbor/ser.rs | 748 ++++ rust/serde_cbor/tags.rs | 224 ++ rust/serde_cbor/value/de.rs | 168 + rust/serde_cbor/value/mod.rs | 158 + rust/serde_cbor/value/ser.rs | 447 +++ rust/serde_cbor/write.rs | 177 + rust/serde_derive/bound.rs | 408 +++ rust/serde_derive/de.rs | 3148 +++++++++++++++++ rust/serde_derive/dummy.rs | 46 + rust/serde_derive/fragment.rs | 76 + rust/serde_derive/internals/ast.rs | 204 ++ rust/serde_derive/internals/attr.rs | 1908 +++++++++++ rust/serde_derive/internals/case.rs | 199 ++ rust/serde_derive/internals/check.rs | 445 +++ rust/serde_derive/internals/ctxt.rs | 64 + rust/serde_derive/internals/mod.rs | 30 + rust/serde_derive/internals/receiver.rs | 287 ++ rust/serde_derive/internals/respan.rs | 18 + rust/serde_derive/internals/symbol.rs | 71 + rust/serde_derive/lib.rs | 112 + rust/serde_derive/pretend.rs | 203 ++ rust/serde_derive/ser.rs | 1342 ++++++++ rust/serde_derive/this.rs | 34 + rust/serde_derive/try.rs | 26 + rust/syn/attr.rs | 664 ++++ rust/syn/await.rs | 4 + rust/syn/bigint.rs | 68 + rust/syn/buffer.rs | 400 +++ rust/syn/custom_keyword.rs | 255 ++ rust/syn/custom_punctuation.rs | 302 ++ rust/syn/data.rs | 495 +++ rust/syn/derive.rs | 276 ++ rust/syn/discouraged.rs | 196 ++ rust/syn/error.rs | 430 +++ rust/syn/export.rs | 41 + rust/syn/expr.rs | 3560 +++++++++++++++++++ rust/syn/ext.rs | 141 + rust/syn/file.rs | 127 + rust/syn/gen/clone.rs | 2243 ++++++++++++ rust/syn/gen/debug.rs | 3044 +++++++++++++++++ rust/syn/gen/eq.rs | 2197 ++++++++++++ rust/syn/gen/fold.rs | 3343 ++++++++++++++++++ rust/syn/gen/hash.rs | 2871 ++++++++++++++++ rust/syn/gen/visit.rs | 3788 +++++++++++++++++++++ rust/syn/gen/visit_mut.rs | 3788 +++++++++++++++++++++ rust/syn/gen_helper.rs | 156 + rust/syn/generics.rs | 1339 ++++++++ rust/syn/group.rs | 284 ++ rust/syn/ident.rs | 103 + rust/syn/item.rs | 3315 ++++++++++++++++++ rust/syn/lib.rs | 985 ++++++ rust/syn/lifetime.rs | 156 + rust/syn/lit.rs | 1602 +++++++++ rust/syn/lookahead.rs | 171 + rust/syn/mac.rs | 221 ++ rust/syn/macros.rs | 179 + rust/syn/op.rs | 236 ++ rust/syn/parse.rs | 1316 +++++++ rust/syn/parse_macro_input.rs | 181 + rust/syn/parse_quote.rs | 169 + rust/syn/pat.rs | 929 +++++ rust/syn/path.rs | 856 +++++ rust/syn/print.rs | 18 + rust/syn/punctuated.rs | 1070 ++++++ rust/syn/reserved.rs | 46 + rust/syn/sealed.rs | 6 + rust/syn/span.rs | 69 + rust/syn/spanned.rs | 116 + rust/syn/stmt.rs | 351 ++ rust/syn/thread.rs | 43 + rust/syn/token.rs | 1015 ++++++ rust/syn/tt.rs | 109 + rust/syn/ty.rs | 1288 +++++++ rust/syn/verbatim.rs | 35 + rust/syn/whitespace.rs | 67 + samples/rust/Kconfig | 28 + samples/rust/Makefile | 3 + samples/rust/local_data_format/de.rs | 422 +++ samples/rust/local_data_format/error.rs | 73 + samples/rust/local_data_format/ser.rs | 443 +++ samples/rust/puzzle.rs | 4 + samples/rust/puzzle/error.rs | 91 + samples/rust/puzzle/inode.rs | 150 + samples/rust/puzzle/oci.rs | 71 + samples/rust/puzzle/types.rs | 389 +++ samples/rust/puzzle/types/cbor_helpers.rs | 50 + samples/rust/puzzlefs.rs | 220 ++ samples/rust/rust_fs.rs | 105 + samples/rust/rust_serde.rs | 125 + scripts/Makefile.build | 4 +- 160 files changed, 89204 insertions(+), 29 deletions(-) create mode 100644 arch/x86/configs/qemu-busybox-min.config create mode 100644 kernel/configs/qemu-busybox-min.config create mode 100644 rust/hex/error.rs create mode 100644 rust/hex/lib.rs create mode 100644 rust/hex/serde.rs create mode 100644 rust/kernel/cred.rs create mode 100644 rust/kernel/delay.rs create mode 100644 rust/kernel/driver.rs create mode 100644 rust/kernel/file.rs create mode 100644 rust/kernel/fs.rs create mode 100644 rust/kernel/fs/param.rs create mode 100644 rust/kernel/io_buffer.rs create mode 100644 rust/kernel/iov_iter.rs create mode 100644 rust/kernel/mm.rs create mode 100644 rust/kernel/mount.rs create mode 100644 rust/kernel/pages.rs create mode 100644 rust/kernel/test_serde.rs create mode 100644 rust/kernel/test_serde/de.rs create mode 100644 rust/kernel/test_serde/error.rs create mode 100644 rust/kernel/test_serde/ser.rs create mode 100644 rust/kernel/user_ptr.rs create mode 100644 rust/proc-macro2/detection.rs create mode 100644 rust/proc-macro2/fallback.rs create mode 100644 rust/proc-macro2/lib.rs create mode 100644 rust/proc-macro2/marker.rs create mode 100644 rust/proc-macro2/parse.rs create mode 100644 rust/proc-macro2/rcvec.rs create mode 100644 rust/proc-macro2/wrapper.rs create mode 100644 rust/quote/ext.rs create mode 100644 rust/quote/format.rs create mode 100644 rust/quote/ident_fragment.rs create mode 100644 rust/quote/lib.rs create mode 100644 rust/quote/runtime.rs create mode 100644 rust/quote/spanned.rs create mode 100644 rust/quote/to_tokens.rs create mode 100644 rust/serde/de/format.rs create mode 100644 rust/serde/de/ignored_any.rs create mode 100644 rust/serde/de/impls.rs create mode 100644 rust/serde/de/mod.rs create mode 100644 rust/serde/de/seed.rs create mode 100644 rust/serde/de/utf8.rs create mode 100644 rust/serde/de/value.rs create mode 100644 rust/serde/integer128.rs create mode 100644 rust/serde/lib.rs create mode 100644 rust/serde/macros.rs create mode 100644 rust/serde/private/de.rs create mode 100644 rust/serde/private/doc.rs create mode 100644 rust/serde/private/mod.rs create mode 100644 rust/serde/private/ser.rs create mode 100644 rust/serde/private/size_hint.rs create mode 100644 rust/serde/ser/fmt.rs create mode 100644 rust/serde/ser/impls.rs create mode 100644 rust/serde/ser/impossible.rs create mode 100644 rust/serde/ser/mod.rs create mode 100644 rust/serde/std_error.rs create mode 100644 rust/serde_cbor/de.rs create mode 100644 rust/serde_cbor/error.rs create mode 100644 rust/serde_cbor/lib.rs create mode 100644 rust/serde_cbor/read.rs create mode 100644 rust/serde_cbor/ser.rs create mode 100644 rust/serde_cbor/tags.rs create mode 100644 rust/serde_cbor/value/de.rs create mode 100644 rust/serde_cbor/value/mod.rs create mode 100644 rust/serde_cbor/value/ser.rs create mode 100644 rust/serde_cbor/write.rs create mode 100644 rust/serde_derive/bound.rs create mode 100644 rust/serde_derive/de.rs create mode 100644 rust/serde_derive/dummy.rs create mode 100644 rust/serde_derive/fragment.rs create mode 100644 rust/serde_derive/internals/ast.rs create mode 100644 rust/serde_derive/internals/attr.rs create mode 100644 rust/serde_derive/internals/case.rs create mode 100644 rust/serde_derive/internals/check.rs create mode 100644 rust/serde_derive/internals/ctxt.rs create mode 100644 rust/serde_derive/internals/mod.rs create mode 100644 rust/serde_derive/internals/receiver.rs create mode 100644 rust/serde_derive/internals/respan.rs create mode 100644 rust/serde_derive/internals/symbol.rs create mode 100644 rust/serde_derive/lib.rs create mode 100644 rust/serde_derive/pretend.rs create mode 100644 rust/serde_derive/ser.rs create mode 100644 rust/serde_derive/this.rs create mode 100644 rust/serde_derive/try.rs create mode 100644 rust/syn/attr.rs create mode 100644 rust/syn/await.rs create mode 100644 rust/syn/bigint.rs create mode 100644 rust/syn/buffer.rs create mode 100644 rust/syn/custom_keyword.rs create mode 100644 rust/syn/custom_punctuation.rs create mode 100644 rust/syn/data.rs create mode 100644 rust/syn/derive.rs create mode 100644 rust/syn/discouraged.rs create mode 100644 rust/syn/error.rs create mode 100644 rust/syn/export.rs create mode 100644 rust/syn/expr.rs create mode 100644 rust/syn/ext.rs create mode 100644 rust/syn/file.rs create mode 100644 rust/syn/gen/clone.rs create mode 100644 rust/syn/gen/debug.rs create mode 100644 rust/syn/gen/eq.rs create mode 100644 rust/syn/gen/fold.rs create mode 100644 rust/syn/gen/hash.rs create mode 100644 rust/syn/gen/visit.rs create mode 100644 rust/syn/gen/visit_mut.rs create mode 100644 rust/syn/gen_helper.rs create mode 100644 rust/syn/generics.rs create mode 100644 rust/syn/group.rs create mode 100644 rust/syn/ident.rs create mode 100644 rust/syn/item.rs create mode 100644 rust/syn/lib.rs create mode 100644 rust/syn/lifetime.rs create mode 100644 rust/syn/lit.rs create mode 100644 rust/syn/lookahead.rs create mode 100644 rust/syn/mac.rs create mode 100644 rust/syn/macros.rs create mode 100644 rust/syn/op.rs create mode 100644 rust/syn/parse.rs create mode 100644 rust/syn/parse_macro_input.rs create mode 100644 rust/syn/parse_quote.rs create mode 100644 rust/syn/pat.rs create mode 100644 rust/syn/path.rs create mode 100644 rust/syn/print.rs create mode 100644 rust/syn/punctuated.rs create mode 100644 rust/syn/reserved.rs create mode 100644 rust/syn/sealed.rs create mode 100644 rust/syn/span.rs create mode 100644 rust/syn/spanned.rs create mode 100644 rust/syn/stmt.rs create mode 100644 rust/syn/thread.rs create mode 100644 rust/syn/token.rs create mode 100644 rust/syn/tt.rs create mode 100644 rust/syn/ty.rs create mode 100644 rust/syn/verbatim.rs create mode 100644 rust/syn/whitespace.rs create mode 100644 samples/rust/local_data_format/de.rs create mode 100644 samples/rust/local_data_format/error.rs create mode 100644 samples/rust/local_data_format/ser.rs create mode 100644 samples/rust/puzzle.rs create mode 100644 samples/rust/puzzle/error.rs create mode 100644 samples/rust/puzzle/inode.rs create mode 100644 samples/rust/puzzle/oci.rs create mode 100644 samples/rust/puzzle/types.rs create mode 100644 samples/rust/puzzle/types/cbor_helpers.rs create mode 100644 samples/rust/puzzlefs.rs create mode 100644 samples/rust/rust_fs.rs create mode 100644 samples/rust/rust_serde.rsSignature · 1 line
2.40.1
Hello Christian,
I didn't send these patches to a wider audience because this is an initial prototype of the PuzzleFS driver, and it has a few prerequisites before it could be even considered for merging. First of all, the rust filesystem abstractions and their dependencies need to be upstreamed, then there needs to be a discussion regarding the inclusion of third-party crates in the linux kernel.
My plan was to send these patches to the rust-for-linux mailing list and then start a discussion with Miguel Ojeda regarding the upstreaming approach.
There are a lot of new files added in this patch series because I've included all the dependencies required so that my patches could be applied to the rust-next branch, but these dependencies will most likely need to be upstreamed separately.
It was never my intention to avoid your reviews, should I also send subsequent patches to linux-fsdevel, even if they're in the early stages of development?
Regards,
Ariel
________________________________
From: Christian Brauner <[email protected]>
Sent: Friday, June 9, 2023 1:36 PM
To: Ariel Miculas (amiculas) <[email protected]>; [email protected] <[email protected]>
Cc: [email protected] <[email protected]>; [email protected] <[email protected]>
Subject: Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver
Quoted 3 lines · Ariel Miculas
Hi all!
This is a proof of concept driver written for the PuzzleFS
Uhm, the PuzzleFS filesystem isn't actually sent to fsdevel? Yet I see tons of patches in there that add wrappers to our core fs data structures. I even see a ton of new files that all fall clearly into fsdevel territory:
create mode 100644 rust/kernel/cred.rs
create mode 100644 rust/kernel/delay.rs
create mode 100644 rust/kernel/driver.rs
create mode 100644 rust/kernel/file.rs
create mode 100644 rust/kernel/fs.rs
create mode 100644 rust/kernel/fs/param.rs
create mode 100644 rust/kernel/io_buffer.rs
create mode 100644 rust/kernel/iov_iter.rs
create mode 100644 rust/kernel/mm.rs
create mode 100644 rust/kernel/mount.rs
create mode 100644 rust/kernel/pages.rs
There's also quite a lot of new mm/ in there, no?
Any wrappers and code for core fs should be maintained as part of fs. Rust shouldn't become a way to avoid our reviews once you have a few wrappers added somewhere.
Quoted 677 lines
next-generation container filesystem [1]. I've included a short abstract about puzzlefs further below. This driver is based on the rust-next branch, on top of which I've backported the filesystem abstractions from Wedson Almeida Filho [2][3] and Miguel Ojeda's third-party crates
support: proc-macro2, quote, syn, serde andserde_derive[4]. I've added the additional third-party crates serde_cbor[5] and hex [6]. Then I've adapted the user space puzzlefs code [1] so that the puzzlefs kernel module could present the directory hierarchy and implement the basic read functionality.
For some additional context, puzzlefs was started by Tycho Andersen and it's the successor of atomfs. This FOSDEM presentation from 2019 [12] covers the rationale for a new oci image format and presents a higher level overview of our goals with puzzlefs.
I've split the rest of the cover letter in following sections (using a markdown style):
* Example: it describes a practical example of what was achieved
* Limitations: it presents the existing limitations of this POC
* Upstreaming steps: it describes the steps needed for upstreaming this driver
* Setup: it shows how to setup the necessary environment for testing the puzzlefs driver
* Puzzlefs abstract: it provides a short overview of puzzlefs# Example
An example is provided below:```
~ # cat/proc/filesystems| grep puzzlefs
nodev puzzlefs
~ # mount -t puzzlefs -o oci_root_dir="/home/puzzlefs_oci" -o image_manifest="2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b" none /mnt ~ # ls -lR/mnt//mnt/:
total 0drwxr-xr-x 2 0 0 0 Jun 8 12:26 dir-1 drwxr-xr-x 2 0 0 0 Jun 8 12:26 dir-2 drwxr-xr-x 2 0 0 0 Jun 8 12:26 dir-3 drwxr-xr-x 2 0 0 0 Jun 8 12:26 dir-4(quoted diff) +0
-2-rw-r--r-- 1 0 0 0 Jun 8 12:26 file1 -rw-r--r-- 1 0 0 0 Jun 8 12:26 file2
/mnt/dir-1:
total 0
/mnt/dir-2:
total 0
/mnt/dir-3:
total 0
/mnt/dir-4:
total 0
~ # cat/mnt/file2
ana are mere bla bla bla
~ # wc/mnt/file1202 202 5454 /mnt/file1 ``` In this example, /home/puzzlefs_oci is the puzzlefs oci directory: ```~ # ls -lR
/home/puzzlefs_oci//home/puzzlefs_oci/:
total 8
drwxr-xr-x 3 1000 1000 0 Jun 8 14:33 blobs(quoted diff) +0
-2-rw-r--r-- 1 1000 1000 266 Jun 8 14:33 index.json -rw-r--r-- 1 1000 1000 37 Jun 8 14:33 oci-layout
/home/puzzlefs_oci/blobs:
total 0
drwxr-xr-x 2 1000 1000 0 Jun 8 14:33 sha256
/home/puzzlefs_oci/blobs/sha256:
total 16(quoted diff) +0
-3-rw------- 1 1000 1000 89 Jun 8 14:33 2d6602d678140540dc7e96de652a76a8b16eb -rw------- 1 1000 1000 925 Jun 8 14:33 4df03518eea406343dbb55046720f6a478881 -rw------- 1 1000 1000 5479 Jun 8 14:33 d86a87b19bd9a2fec0d31687c1d669cdb59eb``` `2d6602d678140540dc7e96de652a76a8b16eb` is the puzzlefs image manifest hash for the first_try tag: ``` $ cat /tmp/oci-simple/index.json | jq . { "schemaVersion": -1, "manifests": [ { "digest": "sha256:2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b", "size": 89, "media_type": "application/vnd.puzzlefs.image.rootfs.v1", "annotations": { "org.opencontainers.image.ref.name": "first_try" } } ], "annotations": {} } ```I will describe how to build a puzzlefs image in the
Setupsection, at step 5.# Limitations
One limitation is that the puzzlefs driver doesn't implement any lookup functionality and instead it inserts every directory entry into the dcache during init (see theDCACHE_BASEDconstant). This is similar to how the samplerust_fsdriver works, but the goal is to implement proper lookup functions. However, more filesystem abstractions need to be implemented before this can be achieved.# Upstreaming steps
Before the puzzlefs driver can be upstreamed, the following need to be merged:
* Wedson's filesystem abstractions [3]
* the necessary third-party crates [4] (with the preliminary discussion about whether this is desirable)# Setup
My setup is based on Wedson's tutorial [8]. Next, I will describe the necessary steps to build an initrd and run a custom kernel under qemu.1. Get the linux rust-next branch [9] and apply this patchset
2. Follow the rust quickstart guide [10]
3. configure and build the kernel
``` $ make LLVM=1 allnoconfig qemu-busybox-min.config rust.config $ make LLVM=1 -j$(nproc) ```4. setup busybox
``` git clone git://git.busybox.net/busybox cd busybox make menuconfig # enable 'Build static binary' config make make install ```This will create the
_installdirectory with the rootfs inside it.5. create a home directory in the rootfs and copy a puzzlefs oci directory in home/puzzlefs_oci
To create a puzzlefs oci directory:
* download this custom puzzlefs repository [11] (it's custom because we want to build an image without verity data)
* runmake release
* create a simple filesystem structure with a few directories and files (I've created one at ../test-puzzlefs/simple_rootfs)
* build a puzzlefs oci image at~/work/busybox/_install/home/puzzlefs_oci(replace this path with your busybox path) with the tagfirst_try:``` $ target/release/puzzlefs build --omit-verity \ ../test-puzzlefs/simple_rootfs ~/work/busybox/_install/home/puzzlefs_oci \ first_try ``` * get first_try's image manifest from index.json (inside `puzzlefs_oci`) ``` $ cat index.json | jq . | grep digest "digest": "sha256:2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b", ```6. add the following 'init' script in the busybox rootfs (rootfs path defaults to `./_install'):
``` #!/bin/sh mount -t devtmpfs none /dev mkdir -p /proc mount -t proc none /proc ifconfig lo up udhcpc -i eth0 mkdir /mnt mount -t puzzlefs -o oci_root_dir="/home/puzzlefs_oci" -o \ image_manifest="2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b" \ none /mnt setsid sh -c 'exec sh -l </dev/ttyS0 >/dev/ttyS0 2>&1' ```Make sure to replace the
image_manifestwith your own digest. This init script will be passed to rdinit in the kernel command line.7. generate the initramfs
Assuming busybox is in
~/work/busybox:``` cd ~/work/busybox/_install && find . | cpio -H newc -o | gzip > ../ramdisk.img ```This will generate a compressed ramdisk image in
~/work/busybox/ramdisk.img.8. run with qemu (this assumes the linux tree is at '../linux' and busybox is at '../busybox'):
``` qemu-system-x86_64 \ -accel kvm \ -cpu host \ -m 4G \ -initrd ../busybox/ramdisk.img \ -kernel ../linux/arch/x86/boot/bzImage \ -nographic \ -append 'console=ttyS0 nokaslr debug rdinit=/init' \ -nic user,model=rtl8139 \ -no-reboot ```9. Check whether puzzlefs has been successfully mounted
``` ~ # mount | grep puzzlefs none on /mnt type puzzlefs (rw,relatime) ~ # ls /mnt/ dir-1 dir-2 dir-3 dir-4 file1 file2 ```# Puzzlefs abstract
Puzzlefs [1] is a container filesystem designed to address the limitations of the existing OCI format. The main goals of the project are reduced duplication, reproducible image builds, direct mounting support and memory safety guarantees, some inspired by the OCIv2 design document [7].Reduced duplication is achieved using the content defined chunking algorithm FastCDC. This implementation allows chunks to be shared among layers. Building a new layer starting from an existing one allows reusing most of the chunks.
Another goal of the project is reproducible image builds, which is achieved by defining a canonical representation of the image format.
Direct mounting support is a key feature of puzzlefs and, together with fs-verity, it provides data integrity. Currently, puzzlefs is implemented as a userspace filesystem (FUSE). A read-only kernel filesystem driver is underway.
Lastly, memory safety is critical to puzzlefs, leading to the decision to implement it in Rust. Another goal is to share the same code between user space and kernel space in order to provide one secure implementation.
[1] https://github.com/anuvu/puzzlefs
[2] https://github.com/wedsonaf/linux/tree/fs
[3] https://github.com/Rust-for-Linux/linux/issues/1004
[4] https://github.com/Rust-for-Linux/linux/pull/1007
[5] https://docs.rs/serde_cbor/latest/serde_cbor/
[6] https://docs.rs/hex/0.4.3/hex/
[7] https://hackmd.io/@cyphar/ociv2-brainstorm
[8] https://www.youtube.com/watch?v=tPs1uRqOnlk
[9] https://github.com/Rust-for-Linux/linux/tree/rust-next
[10] https://docs.kernel.org/rust/quick-start.html
[11] https://github.com/ariel-miculas/puzzlefs/tree/support-no-verity-data
[12] https://archive.fosdem.org/2019/schedule/event/containers_atomfs/Ariel Miculas (58):
rust: kernel: add libraries required by the filesystem abstractions
rust: kernel: backport the delay module from the rust branch
rust: kernel: addcontainer_ofmacro
rust: kernel: addoffset_ofmacro
drop: Addcrate::pr_warndeclaration
rust: kernel: renamefrom_kernel_errnotofrom_errno
rust: kernel: Renamefrom_pointertofrom_foreingandinto_pointerto
into_foreign
rust: kernel: addcount_paren_itemsmacro, needed bydefine_fs_params
macro
rust: helpers: add missing rust helper 'alloc_pages'
kernel: configs: add qemu-busybox-min.config
rust: kernel: format the rust code
samples: puzzlefs: add initial puzzlefs sample, copied fromrust_fs.rs
kernel: configs: enable rust samples in rust.config
AddSAMPLE_RUST_SERDEin rust.config
rust: kernel: fix compile errors after rebase to rust-next
rust:serde_cbor: import crate
rust:serde_cbor: add SPDX License Identifiers
rust:serde_cbor: addno_fp_fmt_parsesupport
rust:Kbuild: enableserde_cbor
samples: rust: add cbor serialize/deserialize example
rust:serde_cbor: add support for serde_cbor'sfrom_slicemethod by
using a customalloc_kernelfeature
rust: serde: add support for deserializing Vec withkernel_alloc
feature
rust: file: Replace UnsafeCell with Opaque for File
rust: kernel: implementfmt::Debugfor CString
samples: puzzlefs: rename RustFs to PuzzleFs
samples: puzzlefs: add basic deserializing support for the puzzlefs
metadata
rust: file: present the filesystem context to the open function
rust: kernel: add an abstraction over vfsmount to allow cloning a new
private mount
rust: file: addfrom_path,from_path_in_root_mntandread_with_offset
methods to File
samples: puzzlefs: pass the Vfsmount structure from open to read and
return the contents of the data file inside/home/puzzlefs_oci
rust: file: movefrom_path,from_path_in_root_mntandread_with_offset
methods to a RegularFile newtype
rust: file: ensure RegularFile can only create regular files
rust: file: addget_posmethod to RegularFile
rust: file: add methodsread_to_end,get_file_sizeandupdate_posto
RegularFile
rust: file: define a minimal Read trait and implement it for
RegularFile
samples: puzzlefs: addcbor_get_array_sizemethod
samples: puzzlefs: add KernelError to WireFormatError and implement
From conversion
samples: puzzlefs: implement new for MetadataBlob
samples: puzzlefs: build puzzlefs into the kernel, thus avoiding the
need to export rust symbols
rust: alloc: addtry_clonefor Vec<T>
rust: alloc: addfrom_iter_falliblefor Vec<T>
samples: puzzlefs: implementto_errnoandfrom_errnofor
WireFormatError
samples: puzzlefs: add TryReserveError (and from conversion) to
WireFormatError
samples: puzzlefs: add higher level inode related functionality
samples: puzzlefs: populate the directory entries with the inodes from
the puzzlefs metadata file
rust: hex: import crate
rust: hex: add SPDX license identifiers
rust:Kbuild: enablehex
rust: hex: implement FromHex trait andhex::decodeusing a custom
kernel_allocfeature
rust: hex: addencode_hex_iterandencode_hex_upper_itermethods
rust: puzzlefs: add HexError to WireFormatError and implement the From
conversion
rust: puzzlefs: display the error value for
WireFormatError::KernelError
samples: puzzlefs: add Rootfs and Digest structs totypes.rs
samples: puzzlefs: implement the conversion from WireFormatError to
kernel::error::Error
rust: puzzlefs: read the puzzlefs image manifest instead of an
individual metadata layer
rust: puzzlefs: rename PuzzleFs to PuzzleFsModule to avoid confusion
with the PuzzleFS struct
rust: puzzlefs: add support for reading files
rust: puzzlefs: addoci_root_dirandimage_manifestfilesystem
parametersMiguel Ojeda (15):
rust: proc-macro2: import crate
rust: proc-macro2: add SPDX License Identifiers
rust: proc-macro2: removeunicode_identdependency
rust: quote: import crate
rust: quote: add SPDX License Identifiers
rust: syn: import crate
rust: syn: add SPDX License Identifiers
rust: syn: removeunicode-identdependency
rust: serde: import crate
rust: serde: addno_fp_fmt_parsesupport
rust: serde: add SPDX License Identifiers
rust:serde_derive: import crate
rust:serde_derive: add SPDX License Identifiers
rust:Kbuild: enableproc-macro2,quote,syn,serdeand
serde_derive
rust: testserdesupportWedson Almeida Filho (7):
rust: add definitions for ref-counted inodes and dentries
rust: add ability to register a file system
rust: define fs context
rust: add support for file system parameters
rust: allow fs driver to initialise new superblocks
rust: addmodule_fsmacro
WIP: rust: allow fs to be populatedMakefile | 14 +- arch/x86/configs/qemu-busybox-min.config | 11 + kernel/configs/qemu-busybox-min.config | 56 + kernel/configs/rust.config | 11 + rust/.gitignore | 1 + rust/Makefile | 232 +- rust/alloc/vec/mod.rs | 48 + rust/bindings/bindings_helper.h | 14 + rust/bindings/lib.rs | 5 + rust/helpers.c | 76 + rust/hex/error.rs | 78 + rust/hex/lib.rs | 506 +++ rust/hex/serde.rs | 104 + rust/kernel/cred.rs | 46 + rust/kernel/delay.rs | 104 + rust/kernel/driver.rs | 28 + rust/kernel/error.rs | 52 +- rust/kernel/file.rs | 1117 ++++++ rust/kernel/fs.rs | 1478 ++++++++ rust/kernel/fs/param.rs | 558 +++ rust/kernel/io_buffer.rs | 153 + rust/kernel/iov_iter.rs | 81 + rust/kernel/lib.rs | 83 + rust/kernel/mm.rs | 149 + rust/kernel/mount.rs | 66 + rust/kernel/pages.rs | 144 + rust/kernel/str.rs | 6 + rust/kernel/test_serde.rs | 26 + rust/kernel/test_serde/de.rs | 439 +++ rust/kernel/test_serde/error.rs | 73 + rust/kernel/test_serde/ser.rs | 466 +++ rust/kernel/user_ptr.rs | 175 + rust/proc-macro2/detection.rs | 77 + rust/proc-macro2/fallback.rs | 1004 ++++++ rust/proc-macro2/lib.rs | 1341 ++++++++ rust/proc-macro2/marker.rs | 20 + rust/proc-macro2/parse.rs | 874 +++++ rust/proc-macro2/rcvec.rs | 144 + rust/proc-macro2/wrapper.rs | 996 ++++++ rust/quote/ext.rs | 112 + rust/quote/format.rs | 170 + rust/quote/ident_fragment.rs | 88 + rust/quote/lib.rs | 1436 ++++++++ rust/quote/runtime.rs | 440 +++ rust/quote/spanned.rs | 45 + rust/quote/to_tokens.rs | 211 ++ rust/serde/de/format.rs | 32 + rust/serde/de/ignored_any.rs | 246 ++ rust/serde/de/impls.rs | 2755 +++++++++++++++ rust/serde/de/mod.rs | 2313 +++++++++++++ rust/serde/de/seed.rs | 21 + rust/serde/de/utf8.rs | 48 + rust/serde/de/value.rs | 1718 ++++++++++ rust/serde/integer128.rs | 84 + rust/serde/lib.rs | 351 ++ rust/serde/macros.rs | 238 ++ rust/serde/private/de.rs | 2997 ++++++++++++++++ rust/serde/private/doc.rs | 161 + rust/serde/private/mod.rs | 52 + rust/serde/private/ser.rs | 1316 +++++++ rust/serde/private/size_hint.rs | 23 + rust/serde/ser/fmt.rs | 180 + rust/serde/ser/impls.rs | 987 ++++++ rust/serde/ser/impossible.rs | 218 ++ rust/serde/ser/mod.rs | 1992 +++++++++++ rust/serde/std_error.rs | 50 + rust/serde_cbor/de.rs | 1370 ++++++++ rust/serde_cbor/error.rs | 320 ++ rust/serde_cbor/lib.rs | 371 ++ rust/serde_cbor/read.rs | 647 ++++ rust/serde_cbor/ser.rs | 748 ++++ rust/serde_cbor/tags.rs | 224 ++ rust/serde_cbor/value/de.rs | 168 + rust/serde_cbor/value/mod.rs | 158 + rust/serde_cbor/value/ser.rs | 447 +++ rust/serde_cbor/write.rs | 177 + rust/serde_derive/bound.rs | 408 +++ rust/serde_derive/de.rs | 3148 +++++++++++++++++ rust/serde_derive/dummy.rs | 46 + rust/serde_derive/fragment.rs | 76 + rust/serde_derive/internals/ast.rs | 204 ++ rust/serde_derive/internals/attr.rs | 1908 +++++++++++ rust/serde_derive/internals/case.rs | 199 ++ rust/serde_derive/internals/check.rs | 445 +++ rust/serde_derive/internals/ctxt.rs | 64 + rust/serde_derive/internals/mod.rs | 30 + rust/serde_derive/internals/receiver.rs | 287 ++ rust/serde_derive/internals/respan.rs | 18 + rust/serde_derive/internals/symbol.rs | 71 + rust/serde_derive/lib.rs | 112 + rust/serde_derive/pretend.rs | 203 ++ rust/serde_derive/ser.rs | 1342 ++++++++ rust/serde_derive/this.rs | 34 + rust/serde_derive/try.rs | 26 + rust/syn/attr.rs | 664 ++++ rust/syn/await.rs | 4 + rust/syn/bigint.rs | 68 + rust/syn/buffer.rs | 400 +++ rust/syn/custom_keyword.rs | 255 ++ rust/syn/custom_punctuation.rs | 302 ++ rust/syn/data.rs | 495 +++ rust/syn/derive.rs | 276 ++ rust/syn/discouraged.rs | 196 ++ rust/syn/error.rs | 430 +++ rust/syn/export.rs | 41 + rust/syn/expr.rs | 3560 +++++++++++++++++++ rust/syn/ext.rs | 141 + rust/syn/file.rs | 127 + rust/syn/gen/clone.rs | 2243 ++++++++++++ rust/syn/gen/debug.rs | 3044 +++++++++++++++++ rust/syn/gen/eq.rs | 2197 ++++++++++++ rust/syn/gen/fold.rs | 3343 ++++++++++++++++++ rust/syn/gen/hash.rs | 2871 ++++++++++++++++ rust/syn/gen/visit.rs | 3788 +++++++++++++++++++++ rust/syn/gen/visit_mut.rs | 3788 +++++++++++++++++++++ rust/syn/gen_helper.rs | 156 + rust/syn/generics.rs | 1339 ++++++++ rust/syn/group.rs | 284 ++ rust/syn/ident.rs | 103 + rust/syn/item.rs | 3315 ++++++++++++++++++ rust/syn/lib.rs | 985 ++++++ rust/syn/lifetime.rs | 156 + rust/syn/lit.rs | 1602 +++++++++ rust/syn/lookahead.rs | 171 + rust/syn/mac.rs | 221 ++ rust/syn/macros.rs | 179 + rust/syn/op.rs | 236 ++ rust/syn/parse.rs | 1316 +++++++ rust/syn/parse_macro_input.rs | 181 + rust/syn/parse_quote.rs | 169 + rust/syn/pat.rs | 929 +++++ rust/syn/path.rs | 856 +++++ rust/syn/print.rs | 18 + rust/syn/punctuated.rs | 1070 ++++++ rust/syn/reserved.rs | 46 + rust/syn/sealed.rs | 6 + rust/syn/span.rs | 69 + rust/syn/spanned.rs | 116 + rust/syn/stmt.rs | 351 ++ rust/syn/thread.rs | 43 + rust/syn/token.rs | 1015 ++++++ rust/syn/tt.rs | 109 + rust/syn/ty.rs | 1288 +++++++ rust/syn/verbatim.rs | 35 + rust/syn/whitespace.rs | 67 + samples/rust/Kconfig | 28 + samples/rust/Makefile | 3 + samples/rust/local_data_format/de.rs | 422 +++ samples/rust/local_data_format/error.rs | 73 + samples/rust/local_data_format/ser.rs | 443 +++ samples/rust/puzzle.rs | 4 + samples/rust/puzzle/error.rs | 91 + samples/rust/puzzle/inode.rs | 150 + samples/rust/puzzle/oci.rs | 71 + samples/rust/puzzle/types.rs | 389 +++ samples/rust/puzzle/types/cbor_helpers.rs | 50 + samples/rust/puzzlefs.rs | 220 ++ samples/rust/rust_fs.rs | 105 + samples/rust/rust_serde.rs | 125 + scripts/Makefile.build | 4 +- 160 files changed, 89204 insertions(+), 29 deletions(-) create mode 100644 arch/x86/configs/qemu-busybox-min.config create mode 100644 kernel/configs/qemu-busybox-min.config create mode 100644 rust/hex/error.rs create mode 100644 rust/hex/lib.rs create mode 100644 rust/hex/serde.rs create mode 100644 rust/kernel/cred.rs create mode 100644 rust/kernel/delay.rs create mode 100644 rust/kernel/driver.rs create mode 100644 rust/kernel/file.rs create mode 100644 rust/kernel/fs.rs create mode 100644 rust/kernel/fs/param.rs create mode 100644 rust/kernel/io_buffer.rs create mode 100644 rust/kernel/iov_iter.rs create mode 100644 rust/kernel/mm.rs create mode 100644 rust/kernel/mount.rs create mode 100644 rust/kernel/pages.rs create mode 100644 rust/kernel/test_serde.rs create mode 100644 rust/kernel/test_serde/de.rs create mode 100644 rust/kernel/test_serde/error.rs create mode 100644 rust/kernel/test_serde/ser.rs create mode 100644 rust/kernel/user_ptr.rs create mode 100644 rust/proc-macro2/detection.rs create mode 100644 rust/proc-macro2/fallback.rs create mode 100644 rust/proc-macro2/lib.rs create mode 100644 rust/proc-macro2/marker.rs create mode 100644 rust/proc-macro2/parse.rs create mode 100644 rust/proc-macro2/rcvec.rs create mode 100644 rust/proc-macro2/wrapper.rs create mode 100644 rust/quote/ext.rs create mode 100644 rust/quote/format.rs create mode 100644 rust/quote/ident_fragment.rs create mode 100644 rust/quote/lib.rs create mode 100644 rust/quote/runtime.rs create mode 100644 rust/quote/spanned.rs create mode 100644 rust/quote/to_tokens.rs create mode 100644 rust/serde/de/format.rs create mode 100644 rust/serde/de/ignored_any.rs create mode 100644 rust/serde/de/impls.rs create mode 100644 rust/serde/de/mod.rs create mode 100644 rust/serde/de/seed.rs create mode 100644 rust/serde/de/utf8.rs create mode 100644 rust/serde/de/value.rs create mode 100644 rust/serde/integer128.rs create mode 100644 rust/serde/lib.rs create mode 100644 rust/serde/macros.rs create mode 100644 rust/serde/private/de.rs create mode 100644 rust/serde/private/doc.rs create mode 100644 rust/serde/private/mod.rs create mode 100644 rust/serde/private/ser.rs create mode 100644 rust/serde/private/size_hint.rs create mode 100644 rust/serde/ser/fmt.rs create mode 100644 rust/serde/ser/impls.rs create mode 100644 rust/serde/ser/impossible.rs create mode 100644 rust/serde/ser/mod.rs create mode 100644 rust/serde/std_error.rs create mode 100644 rust/serde_cbor/de.rs create mode 100644 rust/serde_cbor/error.rs create mode 100644 rust/serde_cbor/lib.rs create mode 100644 rust/serde_cbor/read.rs create mode 100644 rust/serde_cbor/ser.rs create mode 100644 rust/serde_cbor/tags.rs create mode 100644 rust/serde_cbor/value/de.rs create mode 100644 rust/serde_cbor/value/mod.rs create mode 100644 rust/serde_cbor/value/ser.rs create mode 100644 rust/serde_cbor/write.rs create mode 100644 rust/serde_derive/bound.rs create mode 100644 rust/serde_derive/de.rs create mode 100644 rust/serde_derive/dummy.rs create mode 100644 rust/serde_derive/fragment.rs create mode 100644 rust/serde_derive/internals/ast.rs create mode 100644 rust/serde_derive/internals/attr.rs create mode 100644 rust/serde_derive/internals/case.rs create mode 100644 rust/serde_derive/internals/check.rs create mode 100644 rust/serde_derive/internals/ctxt.rs create mode 100644 rust/serde_derive/internals/mod.rs create mode 100644 rust/serde_derive/internals/receiver.rs create mode 100644 rust/serde_derive/internals/respan.rs create mode 100644 rust/serde_derive/internals/symbol.rs create mode 100644 rust/serde_derive/lib.rs create mode 100644 rust/serde_derive/pretend.rs create mode 100644 rust/serde_derive/ser.rs create mode 100644 rust/serde_derive/this.rs create mode 100644 rust/serde_derive/try.rs create mode 100644 rust/syn/attr.rs create mode 100644 rust/syn/await.rs create mode 100644 rust/syn/bigint.rs create mode 100644 rust/syn/buffer.rs create mode 100644 rust/syn/custom_keyword.rs create mode 100644 rust/syn/custom_punctuation.rs create mode 100644 rust/syn/data.rs create mode 100644 rust/syn/derive.rs create mode 100644 rust/syn/discouraged.rs create mode 100644 rust/syn/error.rs create mode 100644 rust/syn/export.rs create mode 100644 rust/syn/expr.rs create mode 100644 rust/syn/ext.rs create mode 100644 rust/syn/file.rs create mode 100644 rust/syn/gen/clone.rs create mode 100644 rust/syn/gen/debug.rs create mode 100644 rust/syn/gen/eq.rs create mode 100644 rust/syn/gen/fold.rs create mode 100644 rust/syn/gen/hash.rs create mode 100644 rust/syn/gen/visit.rs create mode 100644 rust/syn/gen/visit_mut.rs create mode 100644 rust/syn/gen_helper.rs create mode 100644 rust/syn/generics.rs create mode 100644 rust/syn/group.rs create mode 100644 rust/syn/ident.rs create mode 100644 rust/syn/item.rs create mode 100644 rust/syn/lib.rs create mode 100644 rust/syn/lifetime.rs create mode 100644 rust/syn/lit.rs create mode 100644 rust/syn/lookahead.rs create mode 100644 rust/syn/mac.rs create mode 100644 rust/syn/macros.rs create mode 100644 rust/syn/op.rs create mode 100644 rust/syn/parse.rs create mode 100644 rust/syn/parse_macro_input.rs create mode 100644 rust/syn/parse_quote.rs create mode 100644 rust/syn/pat.rs create mode 100644 rust/syn/path.rs create mode 100644 rust/syn/print.rs create mode 100644 rust/syn/punctuated.rs create mode 100644 rust/syn/reserved.rs create mode 100644 rust/syn/sealed.rs create mode 100644 rust/syn/span.rs create mode 100644 rust/syn/spanned.rs create mode 100644 rust/syn/stmt.rs create mode 100644 rust/syn/thread.rs create mode 100644 rust/syn/token.rs create mode 100644 rust/syn/tt.rs create mode 100644 rust/syn/ty.rs create mode 100644 rust/syn/verbatim.rs create mode 100644 rust/syn/whitespace.rs create mode 100644 samples/rust/local_data_format/de.rs create mode 100644 samples/rust/local_data_format/error.rs create mode 100644 samples/rust/local_data_format/ser.rs create mode 100644 samples/rust/puzzle.rs create mode 100644 samples/rust/puzzle/error.rs create mode 100644 samples/rust/puzzle/inode.rs create mode 100644 samples/rust/puzzle/oci.rs create mode 100644 samples/rust/puzzle/types.rs create mode 100644 samples/rust/puzzle/types/cbor_helpers.rs create mode 100644 samples/rust/puzzlefs.rs create mode 100644 samples/rust/rust_fs.rs create mode 100644 samples/rust/rust_serde.rs--
2.40.1
Christian Brauner
maintainerQuoted 7 lines · Ariel Miculas (amiculas)
Hello Christian,
I didn't send these patches to a wider audience because this is an initial prototype of the PuzzleFS driver, and it has a few prerequisites before it could be even considered for merging. First of all, the rust filesystem abstractions and their dependencies need to be upstreamed, then there needs to be a discussion regarding the
Yes.
Quoted 9 lines
inclusion of third-party crates in the linux kernel.
My plan was to send these patches to the rust-for-linux mailing list and then start a discussion with Miguel Ojeda regarding the upstreaming approach.
There are a lot of new files added in this patch series because I've included all the dependencies required so that my patches could be applied to the rust-next branch, but these dependencies will most likely need to be upstreamed separately.It was never my intention to avoid your reviews, should I also send subsequent patches to linux-fsdevel, even if they're in the early stages of development?
Yeah, I think that would be great.
Because the series you sent here touches on a lot of things in terms of infrastructure alone. That work could very well be rather interesting independent of PuzzleFS. We might just want to get enough infrastructure to start porting a tiny existing fs (binderfs or something similar small) to Rust to see how feasible this is and to wet our appetite for bigger changes such as accepting a new filesystem driver completely written in Rust.
But aside from the infrastructure discussion:
This is yet another filesystem for solving the container image problem in the kernel with the addition of yet another filesystem. We just went through this excercise with another filesystem. So I'd expect some reluctance here. Tbh, the container world keeps sending us filesystems at an alarming rate. That's two within a few months and that leaves a rather disorganized impression.
Adding Serge Hallyn for visibility.
Regards,
Ariel
From: Christian Brauner <[email protected]>
Sent: Friday, June 9, 2023 2:45 PM
To: Ariel Miculas (amiculas) <[email protected]>
Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Subject: Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver
Quoted 7 lines · Ariel Miculas (amiculas)
Hello Christian,
I didn't send these patches to a wider audience because this is an initial prototype of the PuzzleFS driver, and it has a few prerequisites before it could be even considered for merging. First of all, the rust filesystem abstractions and their dependencies need to be upstreamed, then there needs to be a discussion regarding the
Yes.
Quoted 9 lines
inclusion of third-party crates in the linux kernel.
My plan was to send these patches to the rust-for-linux mailing list and then start a discussion with Miguel Ojeda regarding the upstreaming approach.
There are a lot of new files added in this patch series because I've included all the dependencies required so that my patches could be applied to the rust-next branch, but these dependencies will most likely need to be upstreamed separately.It was never my intention to avoid your reviews, should I also send subsequent patches to linux-fsdevel, even if they're in the early stages of development?
Yeah, I think that would be great.
Because the series you sent here touches on a lot of things in terms of infrastructure alone. That work could very well be rather interesting independent of PuzzleFS. We might just want to get enough infrastructure to start porting a tiny existing fs (binderfs or something similar small) to Rust to see how feasible this is and to wet our appetite for bigger changes such as accepting a new filesystem driver completely written in Rust.
But aside from the infrastructure discussion:
This is yet another filesystem for solving the container image problem in the kernel with the addition of yet another filesystem. We just went through this excercise with another filesystem. So I'd expect some reluctance here. Tbh, the container world keeps sending us filesystems at an alarming rate. That's two within a few months and that leaves a rather disorganized impression.
On 2023/6/9 20:03, Ariel Miculas (amiculas) wrote:
...
Quoted 9 lines
But aside from the infrastructure discussion:
This is yet another filesystem for solving the container image problem in the kernel with the addition of yet another filesystem. We just went through this excercise with another filesystem. So I'd expect some reluctance here. Tbh, the container world keeps sending us filesystems at an alarming rate. That's two within a few months and that leaves a rather disorganized impression.
Just a head up. Since Rust kernel infrastructure is too premature, it's impossible to handle page cache / iomap and many useful stuffs. In the long term, at least (someday) after Rust infrastructure is mature, I will implement EROFS ino Rust as a try as well.
As for chunk CDC, I don't see it's hard (since we already have some CDC approach since Linux v6.1) but as an effective disk filesystem for performance, EROFS on-disk data is all block-aligned to match storage and page cache alignment. If it's really needed, I could update a more complete (but ineffective and slow) index version to implement unaligned extents (both for decoded and encoded sides). Yet I really think the main purpose of a kernel filesystem is to make full use of kernel infrastructure for performance (like page cache handling) otherwise a FUSE approach is enough.
Finally, as for OCI container image stuffs, I'd like to avoid saying this topic anymore on the list (too tired about this). I've seen _three_ in-kernel approaches already before this one and I tend to avoid listing the complete names (including FUSE alternatives) here. I really suggest if you guys could sit down and plan at least a complete OCI standard for the next image format (even you don't want to reuse any exist filesystem for whatever reasons).
Thanks,
Gao Xiang
Quoted 8 lines · Christian Brauner
Because the series you sent here touches on a lot of things in terms of infrastructure alone. That work could very well be rather interesting independent of PuzzleFS. We might just want to get enough infrastructure to start porting a tiny existing fs (binderfs or something similar small) to Rust to see how feasible this is and to wet our appetite for bigger changes such as accepting a new filesystem driver completely written in Rust.
That would be great, thanks Christian! (Cc'ing Alice for binderfs -- I think Rust Binder is keeping binderfs in C for the moment, but if you are willing to try things, they are probably interested :)
Ariel: sorry, we crossed messages; I didn't receive your message at [1], the rust-for-linux list probably dropped it due to the included HTML.
Cheers,
Miguel
Sorry about that, it seems like I need to switch to plain text mode for every reply in outlook, which is annoying.
Regards,
Ariel
From: Miguel Ojeda <[email protected]>
Sent: Friday, June 9, 2023 3:07 PM
To: Christian Brauner <[email protected]>
Cc: Ariel Miculas (amiculas) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Alice Ryhl <[email protected]>
Subject: Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver
Quoted 8 lines · Christian Brauner
Because the series you sent here touches on a lot of things in terms of infrastructure alone. That work could very well be rather interesting independent of PuzzleFS. We might just want to get enough infrastructure to start porting a tiny existing fs (binderfs or something similar small) to Rust to see how feasible this is and to wet our appetite for bigger changes such as accepting a new filesystem driver completely written in Rust.
That would be great, thanks Christian! (Cc'ing Alice for binderfs -- I think Rust Binder is keeping binderfs in C for the moment, but if you are willing to try things, they are probably interested :)
Ariel: sorry, we crossed messages; I didn't receive your message at [1], the rust-for-linux list probably dropped it due to the included HTML.
Cheers,
Miguel
Greg KH
maintainerQuoted 1 line · Ariel Miculas (amiculas)
Sorry about that, it seems like I need to switch to plain text mode for every reply in outlook, which is annoying.
You should also turn off the top-posting mode, as that's not good etiquette on the kernel mailing lists for obvious reasons :)
thanks,
greg k-h
Alice Ryhl
maintainerQuoted 3 lines · Miguel Ojeda
That would be great, thanks Christian! (Cc'ing Alice for binderfs -- I think Rust Binder is keeping binderfs in C for the moment, but if you are willing to try things, they are probably interested :)
Yeah, Rust binder already needs bindings to a lot of other parts of the kernel, so I decided to not rewrite the binderfs part for now to cut down on the number of subsystems I would need to upstream bindings for. Upstreaming bindings has proved to be a lot of work.
If someone else wants to upstream the filesystem bindings that binderfs would need, then we can certainly look into using them in Rust binder.
Alice
Quoted 8 lines · Christian Brauner
Because the series you sent here touches on a lot of things in terms of infrastructure alone. That work could very well be rather interesting independent of PuzzleFS. We might just want to get enough infrastructure to start porting a tiny existing fs (binderfs or something similar small) to Rust to see how feasible this is and to wet our appetite for bigger changes such as accepting a new filesystem driver completely written in Rust.
(Not a kernel developer, but this argument makes sense to me)
Quoted 8 lines
But aside from the infrastructure discussion:
This is yet another filesystem for solving the container image problem in the kernel with the addition of yet another filesystem. We just went through this excercise with another filesystem. So I'd expect some reluctance here. Tbh, the container world keeps sending us filesystems at an alarming rate. That's two within a few months and that leaves a rather disorganized impression.
I am sure you are aware there's not some "container world" monoculture, there are many organizations, people and companies here with some healthy co-opetition but also some duplication inherent from that.
That said at a practical level, Ariel in the https://github.com/containers GH organization we're kind of a "big tent" place. A subset of the organization is very heavily Rust oriented now (certainly the parts I touch) and briefly skimming the puzzlefs code, there are definitely some bits of code we could consider sharing in userspace. Actually though since this isn't releated to the in-kernel discussion I'll file an issue on Github and we can discuss there.
But there is definitely a subset of the discussion that Christian is referring to here that is about the intersections/overlap with the composefs approach that is relevant for this list. Maybe we could try to collaborate on an unbiased "puzzlefs vs composefs" document? (What's in https://github.com/anuvu/puzzlefs/tree/master/doc is a bit sparse right now)
Christian Brauner
maintainerQuoted 23 lines · Colin Walters
Quoted 8 lines · Christian Brauner
Because the series you sent here touches on a lot of things in terms of infrastructure alone. That work could very well be rather interesting independent of PuzzleFS. We might just want to get enough infrastructure to start porting a tiny existing fs (binderfs or something similar small) to Rust to see how feasible this is and to wet our appetite for bigger changes such as accepting a new filesystem driver completely written in Rust.
(Not a kernel developer, but this argument makes sense to me)
Quoted 8 lines
But aside from the infrastructure discussion:
This is yet another filesystem for solving the container image problem in the kernel with the addition of yet another filesystem. We just went through this excercise with another filesystem. So I'd expect some reluctance here. Tbh, the container world keeps sending us filesystems at an alarming rate. That's two within a few months and that leaves a rather disorganized impression.
Quoted 2 lines
I am sure you are aware there's not some "container world" monoculture, there are many organizations, people and companies here
That submission here explicitly references OCI v2. Composefs explicitly advertises 100% compatibility with OCI. So, there's a set of OCI specs including runtime and image. As far as I'm concerned you're all one container world under the OCI umbrella.
We're not going to push multiple filesystems into the kernel that all do slightly different things but all serve the OCI container world and use some spec as an argument to move stuff into the kernel.
The OCI initiative is hailed as unifying the container ecosystem. Hence, we can expect coordination.
Quoted 24 lines · Christian Brauner
Quoted 23 lines · Colin Walters
Quoted 8 lines · Christian Brauner
Because the series you sent here touches on a lot of things in terms of infrastructure alone. That work could very well be rather interesting independent of PuzzleFS. We might just want to get enough infrastructure to start porting a tiny existing fs (binderfs or something similar small) to Rust to see how feasible this is and to wet our appetite for bigger changes such as accepting a new filesystem driver completely written in Rust.
(Not a kernel developer, but this argument makes sense to me)
Quoted 8 lines
But aside from the infrastructure discussion:
This is yet another filesystem for solving the container image problem in the kernel with the addition of yet another filesystem. We just went through this excercise with another filesystem. So I'd expect some reluctance here. Tbh, the container world keeps sending us filesystems at an alarming rate. That's two within a few months and that leaves a rather disorganized impression.
Quoted 5 lines
Quoted 2 lines
I am sure you are aware there's not some "container world" monoculture, there are many organizations, people and companies here
That submission here explicitly references OCI v2. Composefs explicitly advertises 100% compatibility with OCI. So, there's a set of OCI specs
OCI v2 doesn't currently exist :) There were many design goals for it, and near as I can tell, after everyone discussed those together, everyone went off to work on implementing the bits the needed - which is a right and proper step before coming back and comparing notes about what went well, etc.
The two main things where puzzlefs is experimenting are the use of content defined chunking (CDC), and being written in rust (and, especially, to have the same rust code base be used for the in kernel driver, the fuse mounter, the builder, and the userspace extractor). (Well, and reproducible images through a canonical representation, but...)
It requires a POC in order to really determine whether the CDC will be worth it, or will have pitfalls. So far, it looks very promising. Adding that functionality to composefs one day could be cool.
Likewise, it requires a user in order to push on the infrastructure required to support a full filesystem in rust in the kernel. But that really isn't something we can "add to composefs". :)
The main goal of this posting, then, was to show the infrastructure pieces and work on that with the community. We're definitely not (currently :) asking for puzzlefs to be included. However, it is our (business and personal) justification for the rest of the work.
Quoted 2 lines
including runtime and image. As far as I'm concerned you're all one container world under the OCI umbrella.
One big happy family!
Quoted 6 lines
We're not going to push multiple filesystems into the kernel that all do slightly different things but all serve the OCI container world and use some spec as an argument to move stuff into the kernel.
The OCI initiative is hailed as unifying the container ecosystem. Hence, we can expect coordination.
A "puzzlefs vs composefs" document sounds like a good idea. The documentation in puzzlefs is a little outdated and could be improved. Feel free to create a github issue and tag me in there.
PS: as soon as I figure out how to turn off the top-posting mode, I'll do it.
Regards,
Ariel
________________________________________
From: Colin Walters <[email protected]>
Sent: Friday, June 9, 2023 3:20 PM
To: Christian Brauner; Ariel Miculas (amiculas)
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver
Quoted 8 lines · Christian Brauner
Because the series you sent here touches on a lot of things in terms of infrastructure alone. That work could very well be rather interesting independent of PuzzleFS. We might just want to get enough infrastructure to start porting a tiny existing fs (binderfs or something similar small) to Rust to see how feasible this is and to wet our appetite for bigger changes such as accepting a new filesystem driver completely written in Rust.
(Not a kernel developer, but this argument makes sense to me)
Quoted 8 lines
But aside from the infrastructure discussion:
This is yet another filesystem for solving the container image problem in the kernel with the addition of yet another filesystem. We just went through this excercise with another filesystem. So I'd expect some reluctance here. Tbh, the container world keeps sending us filesystems at an alarming rate. That's two within a few months and that leaves a rather disorganized impression.
I am sure you are aware there's not some "container world" monoculture, there are many organizations, people and companies here with some healthy co-opetition but also some duplication inherent from that.
That said at a practical level, Ariel in the https://github.com/containers GH organization we're kind of a "big tent" place. A subset of the organization is very heavily Rust oriented now (certainly the parts I touch) and briefly skimming the puzzlefs code, there are definitely some bits of code we could consider sharing in userspace. Actually though since this isn't releated to the in-kernel discussion I'll file an issue on Github and we can discuss there.
But there is definitely a subset of the discussion that Christian is referring to here that is about the intersections/overlap with the composefs approach that is relevant for this list. Maybe we could try to collaborate on an unbiased "puzzlefs vs composefs" document? (What's in https://github.com/anuvu/puzzlefs/tree/master/doc is a bit sparse right now)
Quoted 5 lines · Ariel Miculas (amiculas)
A "puzzlefs vs composefs" document sounds like a good idea. The documentation in puzzlefs is a little outdated and could be improved. Feel free to create a github issue and tag me in there.
PS: as soon as I figure out how to turn off the top-posting mode, I'll do it.
Let me know as well if you could do w/ Outlook :). Switch to other email clients if possible.
---Trilok Soni
I could switch to my personal gmail, but last time Miguel Ojeda asked me to use my cisco email when I send commits signed off by [email protected]. If this is not a hard requirement, then I could switch.
Regards,
Ariel
________________________________________
From: Trilok Soni <[email protected]>
Sent: Friday, June 9, 2023 8:10 PM
To: Ariel Miculas (amiculas); Colin Walters; Christian Brauner
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver
Quoted 5 lines · Ariel Miculas (amiculas)
A "puzzlefs vs composefs" document sounds like a good idea. The documentation in puzzlefs is a little outdated and could be improved. Feel free to create a github issue and tag me in there.
PS: as soon as I figure out how to turn off the top-posting mode, I'll do it.
Let me know as well if you could do w/ Outlook :). Switch to other email clients if possible.
---Trilok Soni
Quoted 3 lines · Ariel Miculas (amiculas)
I could switch to my personal gmail, but last time Miguel Ojeda asked me to use my cisco email when I send commits signed off by [email protected]. If this is not a hard requirement, then I could switch.
For patches, yeah, that is ideal, so that it matches the Git author / From:.
But for the other emails, you could use your personal address, if that makes things easier.
Hope that helps!
Cheers,
Miguel
Quoted 8 lines · Miguel Ojeda
Quoted 5 lines · Ariel Miculas (amiculas)
I could switch to my personal gmail, but last time Miguel Ojeda asked me to use my cisco email when I send commits signed off by [email protected]. If this is not a hard requirement, then I could switch.
Quoted 5 lines
For patches, yeah, that is ideal, so that it matches the Git author /
From:.But for the other emails, you could use your personal address, if that makes things easier.
It's still not a requirement, though. You can send from your gmail account and still have
From: Ariel Miculas <[email protected]>
As the first line (separated from the commit message by a blank line), which git am (or b4) will pick up as the author email. This behaviour is specifically for people who want the author to be their corporate email address, but have failed to persuade corporate IT to make it possible.
Regards,
James
Quoted 7 lines · James Bottomley
Quoted 2 lines · Miguel Ojeda
For patches, yeah, that is ideal, so that it matches the Git author /
From:.It's still not a requirement, though. You can send from your gmail account and still have
Yeah, that is what I said "ideal". When Ariel sent the first patch, he didn't use the From: tag within the email body, but attributed it to Cisco, and I had not seen any commit/email from his Cisco address yet, so I asked him if he could use his corporate address.
Cheers,
Miguel
Yes, but then how can you be sure that [email protected] is the real author of the commit? I think that's why Miguel Ojeda asked me to send them from my business email, otherwise some random gmail account could claim that he is "Ariel Miculas", so he's entitled to sign-off as [email protected].
Regards,
Ariel
Quoted 31 lines · James Bottomley
Quoted 8 lines · Miguel Ojeda
Quoted 5 lines · Ariel Miculas (amiculas)
I could switch to my personal gmail, but last time Miguel Ojeda asked me to use my cisco email when I send commits signed off by [email protected]. If this is not a hard requirement, then I could switch.
Quoted 5 lines
For patches, yeah, that is ideal, so that it matches the Git author /
From:.But for the other emails, you could use your personal address, if that makes things easier.
It's still not a requirement, though. You can send from your gmail account and still have
From: Ariel Miculas <[email protected]>
As the first line (separated from the commit message by a blank line), which git am (or b4) will pick up as the author email. This behaviour is specifically for people who want the author to be their corporate email address, but have failed to persuade corporate IT to make it possible.
Regards,
James
Quoted 5 lines · Ariel Miculas
Yes, but then how can you be sure that [email protected] is the real author of the commit? I think that's why Miguel Ojeda asked me to send them from my business email, otherwise some random gmail account could claim that he is "Ariel Miculas", so he's entitled to sign-off as [email protected].
Because it's a public list and the real one would observe and repudiate
...
The DCO is an attestation which we basically believe absent any proof to the contrary. It's also exhausting and an incredible amount of pointless admin not to believe anything until it's proved, which is why we don't.
James
Quoted 6 lines · Ariel Miculas
Yes, but then how can you be sure that [email protected] is the real author of the commit? I think that's why Miguel Ojeda asked me to send them from my business email, otherwise some random gmail account could claim that he is "Ariel Miculas", so he's entitled to sign-off as [email protected].
Yeah, that was the main thing; and by asking that, if you managed to send that original patch from the corporate one, then there was no need for the From: to begin with :)
Cheers,
Miguel
Quoted 4 lines · Ariel Miculas (amiculas)
I could switch to my personal gmail, but last time Miguel Ojeda asked me to use my cisco email when I send commits signed off by [email protected].
If this is not a hard requirement, then I could switch.
For sending patches, you can simply use git-send-email. All you need to point it at is the outgoing email server (which should be a config setting in whatever tool you are using now). We have a (reasonably) up to date document with some recommendations:
https://www.kernel.org/doc/html/latest/process/email-clients.html
I've successfully used evolution with an exchange server for many years, but the interface isn't to everyone's taste and Mozilla Thunderbird is also known to connect to it. Basic outlook has proven impossible to configure correctly (which is why it doesn't have an entry).
Regards,
James
I did use git send-email for sending this patch series, but I cannot find any setting in the Outlook web client for disabling "top posting" when replying to emails:
https://answers.microsoft.com/en-us/outlook_com/forum/all/eliminate-top-posting/5e1e5729-30f8-41e9-84cb-fb5e81229c7c
Regards,
Ariel
________________________________________
From: James Bottomley <[email protected]>
Sent: Friday, June 9, 2023 9:43 PM
To: Ariel Miculas (amiculas); Trilok Soni; Colin Walters; Christian Brauner
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver
Quoted 4 lines · Ariel Miculas (amiculas)
I could switch to my personal gmail, but last time Miguel Ojeda asked me to use my cisco email when I send commits signed off by [email protected].
If this is not a hard requirement, then I could switch.
For sending patches, you can simply use git-send-email. All you need to point it at is the outgoing email server (which should be a config setting in whatever tool you are using now). We have a (reasonably) up to date document with some recommendations:
https://www.kernel.org/doc/html/latest/process/email-clients.html
I've successfully used evolution with an exchange server for many years, but the interface isn't to everyone's taste and Mozilla Thunderbird is also known to connect to it. Basic outlook has proven impossible to configure correctly (which is why it doesn't have an entry).
Regards,
James
Now if I could figure out how to disable "top posting" in gmail...
Regards,
Ariel
Quoted 37 lines · Ariel Miculas (amiculas)
I did use git send-email for sending this patch series, but I cannot find any setting in the Outlook web client for disabling "top posting" when replying to emails:
https://answers.microsoft.com/en-us/outlook_com/forum/all/eliminate-top-posting/5e1e5729-30f8-41e9-84cb-fb5e81229c7cRegards,
Ariel________________________________________
From: James Bottomley <[email protected]>
Sent: Friday, June 9, 2023 9:43 PM
To: Ariel Miculas (amiculas); Trilok Soni; Colin Walters; Christian Brauner
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driverQuoted 4 lines · Ariel Miculas (amiculas)
I could switch to my personal gmail, but last time Miguel Ojeda asked me to use my cisco email when I send commits signed off by [email protected].
If this is not a hard requirement, then I could switch.For sending patches, you can simply use git-send-email. All you need to point it at is the outgoing email server (which should be a config setting in whatever tool you are using now). We have a (reasonably) up to date document with some recommendations:
https://www.kernel.org/doc/html/latest/process/email-clients.html
I've successfully used evolution with an exchange server for many years, but the interface isn't to everyone's taste and Mozilla Thunderbird is also known to connect to it. Basic outlook has proven impossible to configure correctly (which is why it doesn't have an entry).
Regards,
James
Quoted 39 lines · Ariel Miculas
Now if I could figure out how to disable "top posting" in gmail...
Regards,
ArielQuoted 32 lines · Ariel Miculas (amiculas)
I did use git send-email for sending this patch series, but I cannot find any setting in the Outlook web client for disabling "top posting" when replying to emails:
https://answers.microsoft.com/en-us/outlook_com/forum/all/eliminate-top-posting/5e1e5729-30f8-41e9-84cb-fb5e81229c7cRegards,
Ariel________________________________________
From: James Bottomley <[email protected]>
Sent: Friday, June 9, 2023 9:43 PM
To: Ariel Miculas (amiculas); Trilok Soni; Colin Walters; Christian Brauner
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driverQuoted 4 lines · Ariel Miculas (amiculas)
I could switch to my personal gmail, but last time Miguel Ojeda asked me to use my cisco email when I send commits signed off by [email protected].
If this is not a hard requirement, then I could switch.For sending patches, you can simply use git-send-email. All you need to point it at is the outgoing email server (which should be a config setting in whatever tool you are using now). We have a (reasonably) up to date document with some recommendations:
https://www.kernel.org/doc/html/latest/process/email-clients.html
I've successfully used evolution with an exchange server for many years, but the interface isn't to everyone's taste and Mozilla Thunderbird is also known to connect to it. Basic outlook has proven impossible to configure correctly (which is why it doesn't have an entry).
One of the big reasons why I have quic_tsoni at quicinc dot com (Thunderbird/Mutt/whateveryoulike except Outlook) on top of tsoni at quicinc dot com (Outlook) to make sure that we comply w/ upstream guidelines. Two email IDs are hard to manage but it gives
the good separation and freedom.
---Trilok Soni
Quoted 5 lines · Ariel Miculas (amiculas)
I did use git send-email for sending this patch series, but I cannot find any setting in the Outlook web client for disabling "top posting" when replying to emails:
https://answers.microsoft.com/en-us/outlook_com/forum/all/eliminate-top-posting/5e1e5729-30f8-41e9-84cb-fb5e81229c7cRegards,
Ariel
You can also use git-send-email for sending replies. Just make a .txt file with this format:
Subject: <subject goes here>
<your reply after an empty line>
The lore.kernel.org site generates the appropriate git-send-email command for sending the reply, so you can copy-paste that to send it.
I send most of my replies like that.
Quoted 29 lines
From: James Bottomley <[email protected]>
Sent: Friday, June 9, 2023 9:43 PM
To: Ariel Miculas (amiculas); Trilok Soni; Colin Walters; Christian Brauner
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driverQuoted 4 lines · Ariel Miculas (amiculas)
I could switch to my personal gmail, but last time Miguel Ojeda asked me to use my cisco email when I send commits signed off by [email protected].
If this is not a hard requirement, then I could switch.For sending patches, you can simply use git-send-email. All you need to point it at is the outgoing email server (which should be a config setting in whatever tool you are using now). We have a (reasonably) up to date document with some recommendations:
https://www.kernel.org/doc/html/latest/process/email-clients.html
I've successfully used evolution with an exchange server for many years, but the interface isn't to everyone's taste and Mozilla Thunderbird is also known to connect to it. Basic outlook has proven impossible to configure correctly (which is why it doesn't have an entry).
Regards,
James
Quoted 4 lines · Christian Brauner
Any wrappers and code for core fs should be maintained as part of fs. Rust shouldn't become a way to avoid our reviews once you have a few wrappers added somewhere.
Definitely and, to be clear, we are strict about it (e.g.
https://rust-for-linux.com/contributing#the-rust-subsystem).
In fact, we appreciate maintainers that are willing to take patches through their tree and take ownership of code too (e.g. KUnit and DRM are doing so already).
I imagine Ariel sent the RFC as a way to announce his work early on, especially given how some patches were split, the lack of commit messages and tags, etc.
In other cases, we have reviewed patches privately first to iron out this sort of thing, but I wasn't aware of this series coming (I knew Ariel was working on puzzlefs and that he wanted to submit it eventually, but not that an RFC was ready).
Also, some of the code here comes from the rust branch, but some of it may not be ready yet for upstreaming.
Cheers,
Miguel
Quoted 22 lines · Christian Brauner
Quoted 3 lines · Ariel Miculas
Hi all!
This is a proof of concept driver written for the PuzzleFS
Uhm, the PuzzleFS filesystem isn't actually sent to fsdevel? Yet I see tons of patches in there that add wrappers to our core fs data structures. I even see a ton of new files that all fall clearly into fsdevel territory:
create mode 100644
rust/kernel/cred.rs
create mode 100644rust/kernel/delay.rs
create mode 100644rust/kernel/driver.rs
create mode 100644rust/kernel/file.rs
create mode 100644rust/kernel/fs.rs
create mode 100644rust/kernel/fs/param.rs
create mode 100644rust/kernel/io_buffer.rs
create mode 100644rust/kernel/iov_iter.rs
create mode 100644rust/kernel/mm.rs
create mode 100644rust/kernel/mount.rs
create mode 100644rust/kernel/pages.rs
Quoted 5 lines
There's also quite a lot of new
mm/in there, no?Any wrappers and code for core fs should be maintained as part of fs. Rust shouldn't become a way to avoid our reviews once you have a few wrappers added somewhere.
I'm of the opinion that Rust wrappers should generally live in the same directories as the code they're wrapping. The Rust wrappers are unsafe code that is intimately coupled with the C code they're creating safe interfaces for, and the organizational structure should reflect that.
This'll be particularly important when mixing Rust and C in the same module, as I intend to do in bcachefs in the not-too-distant future.
Quoted 5 lines · Kent Overstreet
I'm of the opinion that Rust wrappers should generally live in the same directories as the code they're wrapping. The Rust wrappers are unsafe code that is intimately coupled with the C code they're creating safe interfaces for, and the organizational structure should reflect that.
+1, that is the plan, which also means MAINTAINERS does not need extra F fields etc.
Cheers,
Miguel
Quoted 4 lines · Ariel Miculas
Hi all!
This is a proof of concept driver written for the PuzzleFS next-generation container filesystem [1].
I can't find the link to the git tree that corresponds to the patch series you just posted. Could you make sure that is front and center when submitting large patch series? I find it a lot easier to get my head around a large body of code by fetching into my local repository.