btrfs-progs: introduce kernelsrc directory to update it
cover btrfs-progs: introduce kernelsrc directory to update it
Hello,
This is a proposal on a way to keep btrfs-progs in sync with kernel code.
https://btrfs.wiki.kernel.org/index.php/Cleanup_ideas#Use_the_kernel_code_in_user_mode
For now, the patch introduces a method to import and update the kernel source files into the btrfs-progs tree. Further work will be needed to update the kernel source code so to make it generic (i.e. add #define's and #ifdef's) so that it can be compiled in both kernel and user space.
The Python script is just a draft, it's already functional but error handling (and reporting) could definitely be improved, the code could be cleaned up, it could cache the kernel source directory and allow a --force parameter to allow moving to an tag that's not a descendant, but it's already fully functional and it's probably a good start...
To see it in action, try this after applying the commits:
$ kernelsrc/update_kernelsrc.py /path/to/linux v3.6 $ git status $ git commit -m "btrfs-progs: initial checkout of kernelsrc v3.6" $ vim kernelsrc/update_kernelsrc.filelist (add/delete a line) $ kernelsrc/update_kernelsrc.py ../linux-btrfs/ v3.7 $ git status $ git commit -m "btrfs-progs: upgrade kernelsrc from v3.6 to v3.7"
Let me know what you think and if you think it's worth to continue on this path.
Cheers,
Filipe
Filipe Brandenburger (2):
btrfs-progs: anchor gitignore patterns on root
btrfs-progs: introduce kernelsrc directory and script to update it
.gitignore | 32 +++++--- kernelsrc/README.kernelsrc | 108 ++++++++++++++++++++++++ kernelsrc/update_kernelsrc.filelist | 23 ++++++ kernelsrc/update_kernelsrc.py | 158 ++++++++++++++++++++++++++++++++++++ 4 files changed, 308 insertions(+), 13 deletions(-) create mode 100644 kernelsrc/README.kernelsrc create mode 100644 kernelsrc/update_kernelsrc.filelist create mode 100755 kernelsrc/update_kernelsrc.py
Signature · 1 line
1.8.1.2
PATCH 1/2 btrfs-progs: anchor gitignore patterns on the root of the tree
This is needed in order to allow checking out fs/btrfs/, otherwise the ignore rule for the btrfs binary would conflict with it.
--- .gitignore | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-)
.gitignore +19 -13
@@ -1,15 +1,21 @@ *.o .*.o.d -version.h -man/*.gz -btrfs -btrfs-debug-tree -btrfs-map-logical -btrfs-show -btrfs-vol -btrfsck -btrfsctl -find-root -mkfs.btrfs -repair -restore +/version.h +/man/*.gz +/btrfs +/btrfsck +/btrfs-convert +/btrfsctl +/btrfs-debug-tree +/btrfs-find-root +/btrfs-image +/btrfs-map-logical +/btrfs-restore +/btrfs-show +/btrfstune +/btrfs-vol +/btrfs-zero-log +/find-root +/mkfs.btrfs +/repair +/restore
Signature · 1 line
1.8.1.2
PATCH 2/2 btrfs-progs: introduce kernelsrc directory and script to update it
In reply to Filipe BrandenburgerThe kernelsrc directory will contain a copy of the Btrfs-related files from the Linux kernel source tree (mainly the files in fs/btrfs/ directory.)
This commit introduces an initial filelist (generated by comparing the list of files in both directories) and a script to fetch the files from a local clone of the kernel git directory into the kernelsrc directory. When updating the directory, a git tag needs to be specified, which is recorded in a local file, in a way that it is possible to track down to which kernel release the kernelsrc files originally belong.
A README file was also added to explain the idea behind it and to document the procedure to update the kernelsrc directory.
--- kernelsrc/README.kernelsrc | 108 ++++++++++++++++++++++++ kernelsrc/update_kernelsrc.filelist | 23 ++++++ kernelsrc/update_kernelsrc.py | 158 ++++++++++++++++++++++++++++++++++++ 3 files changed, 289 insertions(+) create mode 100644 kernelsrc/README.kernelsrc create mode 100644 kernelsrc/update_kernelsrc.filelist create mode 100755 kernelsrc/update_kernelsrc.py
kernelsrc/README.kernelsrc new file +108 -0
@@ -0,0 +1,108 @@ + +The kernelsrc directory tree +============================ + +The `kernelsrc' directory in the btrfs-progs tree contains a copy of some of the +files in the Linux kernel tree, mainly the files in fs/btrfs/, that are shared +between the implementation of the filesystem kernel module and some of the +command-line utilities. + +These files should never be edited on the btrfs-progs tree. Changes to these +files should be committed and pushed to the Linux kernel git tree (optionally +using #define's and #ifdef's to make code compatible with both kernel and user +spaces) and then it should be updated here from the kernel tree. + +In order to update the `kernelsrc' directory, you need to have a checkout of +both the btrfs-progs git tree and the Linux kernel git tree. + +Additionally to the kernel source files, the `kernelsrc' directory tree also +contains the scripts and control files used to update this same tree. + + +Updating the kernelsrc tree with update_kernelsrc.py +==================================================== + +In order to update the `kernelsrc' tree to include the source code files from a +newer version of the Linux kernel tree, these are the steps that should be +followed: + + 1. Make sure that you are in the root of the btrfs-progs git tree and that + the tree is clean (i.e., run `git status' and check its output.) + + 2. Make sure you have a clone of the Linux kernel git tree. Let's assume + for this example that your kernel git tree is in directory ../linux/ + + 3. Make sure you have know to which tag of the Linux kernel you want to + upgrade the `kernelsrc' tree to (e.g. v3.7 or v3.8-rc7) + + 4. (Optional:) You may change the list of files that will be copied from + the Linux kernel tree. In order to do that, edit the file + `kernelsrc/update_kernelsrc.filelist' and either add or remove entries + from that file. + + 5. Run the following command from the root of the btrfs-progs git tree: + + $ kernelsrc/update_kernelsrc.py ../linux/ v3.7 + + (assuming Linux kernel git tree at ../linux/ and upgrading to tag v3.7) + + 6. Check whether the update matches your expectations. In particular, the + commands `git status' and `git diff --cached' may be helpful. + + 7. Build btrfs-progs (using `make') and test them to make sure they are + still compatible with the new kernel source tree. Any needed adjustments + may be done by editing files *OUTSIDE* of the `kernelsrc' tree. Once + everything seems to be in place and working correctly, you may add those + files to the same git commit using the `git add' command. + + 8. Commit the change, using the `git commit' command. You may use the + suggested commit command from the script output, with a standard commit + message. The editor will open anyways, so you can add more details + (particularly about files outside the `kernelsrc' tree that needed + changes.) + + +Migration to kernelsrc +====================== + +Using the `kernelsrc' tree allows for a gradual migration from the current +situation, where copies of the original source files from the kernel tree were +made in btrfs-progs, adjustments were made to allow them to compile in user +space, and they diverged further over time. + +First, it allows checking out the files from the kernel tree and adding them to +the btrfs-progs without actually using them initially. Then, it allows gradually +migrating the files one by one. One way to do it would be replacing the file in +the btrfs-progs root directory with a symbolic link to the same file in the +`kernelsrc' tree. Another way would be integrating the build of the `kernelsrc' +files into the Makefile (assuming all of them compile in userspace without +errors) so that they get built and can be linked from there. The integration may +be switched from the former to the latter at any point in a commit that +basically only touches the Makefiles and symlinks. + + +Alternatives to using a script to update kernelsrc +================================================== + +When coming up with the idea for `update_kernelsrc.py', my first idea was to use +`git submodule' instead. It seems to perfectly capture the idea of having a +repository with source code files that are used by a project in another +repository. So, in theory, it would be possible to have the Linux kernel git +tree as a submodule of the btrfs-progs source tree (it could even be placed +under a `kernelsrc' subdirectory.) Git would then track the revision of the +Linux git tree, and it would be possible to upgrade the tree by doing a checkout +on the Linux tree and then committing on the btrfs-progs tree, which would +simply record the git revision where the Linux tree was pointing to at the time +of the commit. + +The main problem with this approach is the size of the Linux git tree. A clone +of the btrfs-progs tree takes about 2 or 3 MB of diskspace, while the Linux tree +takes about 1 GB. I tried to look at options of doing a `sparse checkout' in +git, but it doesn't seem to be possible to do it for a submodule... Even if it +was, I'm not fully convinced that it would reduce the diskspace requirements, as +it would probably still need to fetch the objects into the .git tree anyways. + +So, the idea of a script that would update the tree by copying the files and +keeping the revision in control files seemed like a close enough solution that +would implement a similar concept without the diskspace problems. +
kernelsrc/update_kernelsrc.filelist new file +23 -0
@@ -0,0 +1,23 @@ +fs/btrfs/ctree.c +fs/btrfs/ctree.h +fs/btrfs/dir-item.c +fs/btrfs/disk-io.c +fs/btrfs/disk-io.h +fs/btrfs/extent_io.c +fs/btrfs/extent_io.h +fs/btrfs/extent-tree.c +fs/btrfs/file-item.c +fs/btrfs/hash.h +fs/btrfs/inode-item.c +fs/btrfs/inode-map.c +fs/btrfs/ioctl.h +fs/btrfs/print-tree.c +fs/btrfs/print-tree.h +fs/btrfs/qgroup.c +fs/btrfs/root-tree.c +fs/btrfs/send.h +fs/btrfs/transaction.h +fs/btrfs/version.h +fs/btrfs/volumes.c +fs/btrfs/volumes.h +include/linux/list.h
kernelsrc/update_kernelsrc.py new file +158 -0
158 lines not on this page · raw message
Signature · 1 line
1.8.1.2