Skip to contentSkip to conversation

btrfs-progs: introduce kernelsrc directory to update it

RFC PATCH New Filipe Brandenburger 1 participant 3 messages Feb 12, 2013 Btrfs file system

lore
Share & tools
lore
#0 lore raw

cover btrfs-progs: introduce kernelsrc directory to update it

Hello,

This is a proposal on a way to keep btrfs-progs in sync with kernel code.
https://btrfs.wiki.kernel.org/index.php/Cleanup_ideas#Use_the_kernel_code_in_user_mode

For now, the patch introduces a method to import and update the kernel source files into the btrfs-progs tree. Further work will be needed to update the kernel source code so to make it generic (i.e. add #define's and #ifdef's) so that it can be compiled in both kernel and user space.

The Python script is just a draft, it's already functional but error handling (and reporting) could definitely be improved, the code could be cleaned up, it could cache the kernel source directory and allow a --force parameter to allow moving to an tag that's not a descendant, but it's already fully functional and it's probably a good start...

To see it in action, try this after applying the commits:

  $ kernelsrc/update_kernelsrc.py /path/to/linux v3.6
  $ git status
  $ git commit -m "btrfs-progs: initial checkout of kernelsrc v3.6"
  $ vim kernelsrc/update_kernelsrc.filelist (add/delete a line)
  $ kernelsrc/update_kernelsrc.py ../linux-btrfs/ v3.7
  $ git status
  $ git commit -m "btrfs-progs: upgrade kernelsrc from v3.6 to v3.7"

Let me know what you think and if you think it's worth to continue on this path.

Cheers,
Filipe

Filipe Brandenburger (2):
btrfs-progs: anchor gitignore patterns on root
btrfs-progs: introduce kernelsrc directory and script to update it

 .gitignore                          |  32 +++++---
 kernelsrc/README.kernelsrc          | 108 ++++++++++++++++++++++++
 kernelsrc/update_kernelsrc.filelist |  23 ++++++
 kernelsrc/update_kernelsrc.py       | 158 ++++++++++++++++++++++++++++++++++++
 4 files changed, 308 insertions(+), 13 deletions(-)
 create mode 100644 kernelsrc/README.kernelsrc
 create mode 100644 kernelsrc/update_kernelsrc.filelist
 create mode 100755 kernelsrc/update_kernelsrc.py
Signature · 1 line
1.8.1.2
#1 lore raw

PATCH 1/2 btrfs-progs: anchor gitignore patterns on the root of the tree

This is needed in order to allow checking out fs/btrfs/, otherwise the ignore rule for the btrfs binary would conflict with it.

---
 .gitignore | 32 +++++++++++++++++++-------------
 1 file changed, 19 insertions(+), 13 deletions(-)
.gitignore +19 -13
index 0e560d5..3d360b6 100644
@@ -1,15 +1,21 @@
 *.o
 .*.o.d
-version.h
-man/*.gz
-btrfs
-btrfs-debug-tree
-btrfs-map-logical
-btrfs-show
-btrfs-vol
-btrfsck
-btrfsctl
-find-root
-mkfs.btrfs
-repair
-restore
+/version.h
+/man/*.gz
+/btrfs
+/btrfsck
+/btrfs-convert
+/btrfsctl
+/btrfs-debug-tree
+/btrfs-find-root
+/btrfs-image
+/btrfs-map-logical
+/btrfs-restore
+/btrfs-show
+/btrfstune
+/btrfs-vol
+/btrfs-zero-log
+/find-root
+/mkfs.btrfs
+/repair
+/restore
Signature · 1 line
1.8.1.2
#2 lore raw

PATCH 2/2 btrfs-progs: introduce kernelsrc directory and script to update it

In reply to Filipe Brandenburger

The kernelsrc directory will contain a copy of the Btrfs-related files from the Linux kernel source tree (mainly the files in fs/btrfs/ directory.)

This commit introduces an initial filelist (generated by comparing the list of files in both directories) and a script to fetch the files from a local clone of the kernel git directory into the kernelsrc directory. When updating the directory, a git tag needs to be specified, which is recorded in a local file, in a way that it is possible to track down to which kernel release the kernelsrc files originally belong.

A README file was also added to explain the idea behind it and to document the procedure to update the kernelsrc directory.

---
 kernelsrc/README.kernelsrc          | 108 ++++++++++++++++++++++++
 kernelsrc/update_kernelsrc.filelist |  23 ++++++
 kernelsrc/update_kernelsrc.py       | 158 ++++++++++++++++++++++++++++++++++++
 3 files changed, 289 insertions(+)
 create mode 100644 kernelsrc/README.kernelsrc
 create mode 100644 kernelsrc/update_kernelsrc.filelist
 create mode 100755 kernelsrc/update_kernelsrc.py
kernelsrc/README.kernelsrc new file +108 -0
new file mode 100644
index 0000000..0aed4bb
@@ -0,0 +1,108 @@
+
+The kernelsrc directory tree
+============================
+
+The `kernelsrc' directory in the btrfs-progs tree contains a copy of some of the
+files in the Linux kernel tree, mainly the files in fs/btrfs/, that are shared
+between the implementation of the filesystem kernel module and some of the
+command-line utilities.
+
+These files should never be edited on the btrfs-progs tree. Changes to these
+files should be committed and pushed to the Linux kernel git tree (optionally
+using #define's and #ifdef's to make code compatible with both kernel and user
+spaces) and then it should be updated here from the kernel tree.
+
+In order to update the `kernelsrc' directory, you need to have a checkout of
+both the btrfs-progs git tree and the Linux kernel git tree.
+
+Additionally to the kernel source files, the `kernelsrc' directory tree also
+contains the scripts and control files used to update this same tree.
+
+
+Updating the kernelsrc tree with update_kernelsrc.py
+====================================================
+
+In order to update the `kernelsrc' tree to include the source code files from a
+newer version of the Linux kernel tree, these are the steps that should be
+followed:
+
+    1.	Make sure that you are in the root of the btrfs-progs git tree and that
+	the tree is clean (i.e., run `git status' and check its output.)
+
+    2.	Make sure you have a clone of the Linux kernel git tree. Let's assume
+	for this example that your kernel git tree is in directory ../linux/
+
+    3.	Make sure you have know to which tag of the Linux kernel you want to
+	upgrade the `kernelsrc' tree to (e.g. v3.7 or v3.8-rc7)
+
+    4.	(Optional:) You may change the list of files that will be copied from
+	the Linux kernel tree. In order to do that, edit the file
+	`kernelsrc/update_kernelsrc.filelist' and either add or remove entries
+	from that file.
+
+    5.	Run the following command from the root of the btrfs-progs git tree:
+
+	$ kernelsrc/update_kernelsrc.py ../linux/ v3.7
+
+	(assuming Linux kernel git tree at ../linux/ and upgrading to tag v3.7)
+
+    6.	Check whether the update matches your expectations. In particular, the
+	commands `git status' and `git diff --cached' may be helpful.
+
+    7.	Build btrfs-progs (using `make') and test them to make sure they are
+	still compatible with the new kernel source tree. Any needed adjustments
+	may be done by editing files *OUTSIDE* of the `kernelsrc' tree. Once
+	everything seems to be in place and working correctly, you may add those
+	files to the same git commit using the `git add' command.
+
+    8.	Commit the change, using the `git commit' command. You may use the
+	suggested commit command from the script output, with a standard commit
+	message. The editor will open anyways, so you can add more details
+	(particularly about files outside the `kernelsrc' tree that needed
+	changes.)
+
+
+Migration to kernelsrc
+======================
+
+Using the `kernelsrc' tree allows for a gradual migration from the current
+situation, where copies of the original source files from the kernel tree were
+made in btrfs-progs, adjustments were made to allow them to compile in user
+space, and they diverged further over time.
+
+First, it allows checking out the files from the kernel tree and adding them to
+the btrfs-progs without actually using them initially. Then, it allows gradually
+migrating the files one by one. One way to do it would be replacing the file in
+the btrfs-progs root directory with a symbolic link to the same file in the
+`kernelsrc' tree. Another way would be integrating the build of the `kernelsrc'
+files into the Makefile (assuming all of them compile in userspace without
+errors) so that they get built and can be linked from there. The integration may
+be switched from the former to the latter at any point in a commit that
+basically only touches the Makefiles and symlinks.
+
+
+Alternatives to using a script to update kernelsrc
+==================================================
+
+When coming up with the idea for `update_kernelsrc.py', my first idea was to use
+`git submodule' instead. It seems to perfectly capture the idea of having a
+repository with source code files that are used by a project in another
+repository. So, in theory, it would be possible to have the Linux kernel git
+tree as a submodule of the btrfs-progs source tree (it could even be placed
+under a `kernelsrc' subdirectory.) Git would then track the revision of the
+Linux git tree, and it would be possible to upgrade the tree by doing a checkout
+on the Linux tree and then committing on the btrfs-progs tree, which would
+simply record the git revision where the Linux tree was pointing to at the time
+of the commit.
+
+The main problem with this approach is the size of the Linux git tree. A clone
+of the btrfs-progs tree takes about 2 or 3 MB of diskspace, while the Linux tree
+takes about 1 GB. I tried to look at options of doing a `sparse checkout' in
+git, but it doesn't seem to be possible to do it for a submodule... Even if it
+was, I'm not fully convinced that it would reduce the diskspace requirements, as
+it would probably still need to fetch the objects into the .git tree anyways.
+
+So, the idea of a script that would update the tree by copying the files and
+keeping the revision in control files seemed like a close enough solution that
+would implement a similar concept without the diskspace problems.
+
kernelsrc/update_kernelsrc.filelist new file +23 -0
new file mode 100644
index 0000000..50cdf95
@@ -0,0 +1,23 @@
+fs/btrfs/ctree.c
+fs/btrfs/ctree.h
+fs/btrfs/dir-item.c
+fs/btrfs/disk-io.c
+fs/btrfs/disk-io.h
+fs/btrfs/extent_io.c
+fs/btrfs/extent_io.h
+fs/btrfs/extent-tree.c
+fs/btrfs/file-item.c
+fs/btrfs/hash.h
+fs/btrfs/inode-item.c
+fs/btrfs/inode-map.c
+fs/btrfs/ioctl.h
+fs/btrfs/print-tree.c
+fs/btrfs/print-tree.h
+fs/btrfs/qgroup.c
+fs/btrfs/root-tree.c
+fs/btrfs/send.h
+fs/btrfs/transaction.h
+fs/btrfs/version.h
+fs/btrfs/volumes.c
+fs/btrfs/volumes.h
+include/linux/list.h
kernelsrc/update_kernelsrc.py new file +158 -0

158 lines not on this page · raw message

Signature · 1 line
1.8.1.2