Skip to contentSkip to conversation

Kselftest update for Linux 5.4-rc1

GIT PULL In review Shuah Khan 11 participants 27 messages Sep 17, 2019 – Oct 3, 2019 1 message not in the archive

Subsystems (13)
lore
Share & tools
lore

Shuah Khan

author maintainer
#0 lore raw

Hi Linus,

Please pull the following Kselftest update for Linux 5.4-rc1.

This Kselftest update for Linux 5.4-rc1 consists of several fixes to existing tests and adds KUnit, a lightweight unit testing and mocking framework for the Linux kernel from Brendan Higgins.

KUnit is not an end-to-end testing framework. It is currently supported on UML and sub-systems can write unit tests and run them in UML env. KUnit documentation is included in this update.

In the future KUnit will be linked to Kselftest framework to provide a way to trigger KUnit tests from user-space.

diff is attached.

Please note that there is conflict between

tools/testing/selftests/tpm2/Makefile

between commit:

3fb2179b0f3553a ("selftests/tpm2: Add the missing TEST_FILES assignment")

from the tpmdd tree and commit:

d04e26067d13f01 ("selftests: tpm2: install python files")

Please take the fix from kselftest tree which is the correct version for this change.

d04e26067d13f01 ("selftests: tpm2: install python files")

Please let me know if you have any questions and/or concerns.

thanks,
-- Shuah

----------------------------------------------------------------
The following changes since commit d1abaeb3be7b5fa6d7a1fbbd2e14e3310005c4c1:

Linux 5.3-rc5 (2019-08-18 14:31:08 -0700)

are available in the Git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest
tags/linux-kselftest-5.4-rc1

for you to fetch changes up to ce54eab71e210f36eebe291e632ed85913c5617e:

kunit: fix failure to build without printk (2019-09-06 09:56:08 -0600)

----------------------------------------------------------------
linux-kselftest-5.4-rc1

This Kselftest update for Linux 5.4-rc1 consists of several fixes to existing tests and adds KUnit, a lightweight unit testing and mocking framework for the Linux kernel from Brendan Higgins.

KUnit is not an end-to-end testing framework. It is currently supported on UML and sub-systems can write unit tests and run them in UML env. KUnit documentation is included in this update.

In the future KUnit will be linked to Kselftest framework to provide a way to trigger KUnit tests from user-space.

----------------------------------------------------------------
Anders Roxell (2):
selftests: livepatch: add missing fragments to config
selftests: tpm2: install python files

Avinash Kondareddy (1):
kunit: test: add tests for KUnit managed resources

Brendan Higgins (16):
kunit: test: add KUnit test runner core
kunit: test: add test resource management API
kunit: test: add string_stream a std::stream like string builder
kunit: test: add assertion printing library
kunit: test: add the concept of expectations
kbuild: enable building KUnit
kunit: test: add initial tests
objtool: add kunit_try_catch_throw to the noreturn list
kunit: test: add support for test abort
kunit: test: add tests for kunit test abort
kunit: test: add the concept of assertions
kunit: defconfig: add defconfigs for building KUnit tests
Documentation: kunit: add documentation for KUnit
MAINTAINERS: add entry for KUnit the unit testing framework
MAINTAINERS: add proc sysctl KUnit test to PROC SYSCTL section
kunit: fix failure to build without printk

Felix Guo (1):
kunit: tool: add Python wrappers for running KUnit tests

George G. Davis (2):
selftests: watchdog: Add optional file argument
selftests: watchdog: cleanup whitespace in usage options

Ilya Leoshkevich (1):

      selftests: use "$(MAKE)" instead of "make"

Iurii Zaikin (1):
kernel/sysctl-test: Add null pointer test for
sysctl.c:proc_dointvec()

Masanari Iida (1):
selftest/ftrace: Fix typo in trigger-snapshot.tc

Tycho Andersen (1):
selftests/seccomp: fix build on older kernels

 Documentation/dev-tools/index.rst                  |    1 +
 Documentation/dev-tools/kunit/api/index.rst        |   16 +
 Documentation/dev-tools/kunit/api/test.rst         |   11 +
 Documentation/dev-tools/kunit/faq.rst              |   62 +
 Documentation/dev-tools/kunit/index.rst            |   79 ++
 Documentation/dev-tools/kunit/start.rst            |  180 +++
 Documentation/dev-tools/kunit/usage.rst            |  576 ++++++++
 Kconfig                                            |    2 +
 MAINTAINERS                                        |   13 +
 Makefile                                           |    2 +
 arch/um/configs/kunit_defconfig                    |    3 +
 include/kunit/assert.h                             |  356 +++++
 include/kunit/string-stream.h                      |   51 +
 include/kunit/test.h                               | 1490 
++++++++++++++++++++
 include/kunit/try-catch.h                          |   75 +
 kernel/Makefile                                    |    2 +
 kernel/sysctl-test.c                               |  392 +++++
 kunit/Kconfig                                      |   38 +
 kunit/Makefile                                     |    9 +
 kunit/assert.c                                     |  141 ++
 kunit/example-test.c                               |   88 ++
 kunit/string-stream-test.c                         |   52 +
 kunit/string-stream.c                              |  217 +++
 kunit/test-test.c                                  |  331 +++++
 kunit/test.c                                       |  478 +++++++
 kunit/try-catch.c                                  |  118 ++
 lib/Kconfig.debug                                  |   11 +
 tools/objtool/check.c                              |    1 +
 tools/testing/kunit/.gitignore                     |    3 +
 tools/testing/kunit/configs/all_tests.config       |    3 +
 tools/testing/kunit/kunit.py                       |  136 ++
 tools/testing/kunit/kunit_config.py                |   66 +
 tools/testing/kunit/kunit_kernel.py                |  149 ++
 tools/testing/kunit/kunit_parser.py                |  310 ++++
 tools/testing/kunit/kunit_tool_test.py             |  206 +++
 .../test_data/test_is_test_passed-all_passed.log   |   32 +
 .../kunit/test_data/test_is_test_passed-crash.log  |   69 +
 .../test_data/test_is_test_passed-failure.log      |   36 +
 .../test_data/test_is_test_passed-no_tests_run.log |   75 +
 .../test_data/test_output_isolated_correctly.log   |  106 ++
 .../kunit/test_data/test_read_from_file.kconfig    |   17 +
 tools/testing/selftests/Makefile                   |   22 +-
 .../ftrace/test.d/trigger/trigger-snapshot.tc      |    2 +-
 tools/testing/selftests/livepatch/config           |    2 +
 tools/testing/selftests/seccomp/seccomp_bpf.c      |    5 +
 tools/testing/selftests/tpm2/Makefile              |    1 +
 tools/testing/selftests/watchdog/watchdog-test.c   |   41 +-
47 files changed, 6051 insertions(+), 25 deletions(-)
 create mode 100644 Documentation/dev-tools/kunit/api/index.rst
 create mode 100644 Documentation/dev-tools/kunit/api/test.rst
 create mode 100644 Documentation/dev-tools/kunit/faq.rst
 create mode 100644 Documentation/dev-tools/kunit/index.rst
 create mode 100644 Documentation/dev-tools/kunit/start.rst
 create mode 100644 Documentation/dev-tools/kunit/usage.rst
 create mode 100644 arch/um/configs/kunit_defconfig
 create mode 100644 include/kunit/assert.h
 create mode 100644 include/kunit/string-stream.h
 create mode 100644 include/kunit/test.h
 create mode 100644 include/kunit/try-catch.h
 create mode 100644 kernel/sysctl-test.c
 create mode 100644 kunit/Kconfig
 create mode 100644 kunit/Makefile
 create mode 100644 kunit/assert.c
 create mode 100644 kunit/example-test.c
 create mode 100644 kunit/string-stream-test.c
 create mode 100644 kunit/string-stream.c
 create mode 100644 kunit/test-test.c
 create mode 100644 kunit/test.c
 create mode 100644 kunit/try-catch.c
 create mode 100644 tools/testing/kunit/.gitignore
 create mode 100644 tools/testing/kunit/configs/all_tests.config
 create mode 100755 tools/testing/kunit/kunit.py
 create mode 100644 tools/testing/kunit/kunit_config.py
 create mode 100644 tools/testing/kunit/kunit_kernel.py
 create mode 100644 tools/testing/kunit/kunit_parser.py
 create mode 100755 tools/testing/kunit/kunit_tool_test.py
 create mode 100644 
tools/testing/kunit/test_data/test_is_test_passed-all_passed.log
 create mode 100644 
tools/testing/kunit/test_data/test_is_test_passed-crash.log
 create mode 100644 
tools/testing/kunit/test_data/test_is_test_passed-failure.log
 create mode 100644 
tools/testing/kunit/test_data/test_is_test_passed-no_tests_run.log
 create mode 100644 
tools/testing/kunit/test_data/test_output_isolated_correctly.log
 create mode 100644 

tools/testing/kunit/test_data/test_read_from_file.kconfig
----------------------------------------------------------------

Documentation/dev-tools/index.rst +1 -0
index b0522a4dd107..09dee10d2592 100644
@@ -24,6 +24,7 @@ whole; patches welcome!
    gdb-kernel-debugging
    kgdb
    kselftest
+   kunit/index
 
 
 .. only::  subproject and html
Documentation/dev-tools/kunit/api/index.rst new file +16 -0
new file mode 100644
index 000000000000..9b9bffe5d41a
@@ -0,0 +1,16 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=============
+API Reference
+=============
+.. toctree::
+
+	test
+
+This section documents the KUnit kernel testing API. It is divided into the
+following sections:
+
+================================= ==============================================
+:doc:`test`                       documents all of the standard testing API
+                                  excluding mocking or mocking related features.
+================================= ==============================================
Documentation/dev-tools/kunit/api/test.rst new file +11 -0
new file mode 100644
index 000000000000..aaa97f17e5b3
@@ -0,0 +1,11 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+========
+Test API
+========
+
+This file documents all of the standard testing API excluding mocking or mocking
+related features.
+
+.. kernel-doc:: include/kunit/test.h
+   :internal:
Documentation/dev-tools/kunit/faq.rst new file +62 -0

62 lines not on this page · raw message

Documentation/dev-tools/kunit/index.rst new file +79 -0

79 lines not on this page · raw message

Documentation/dev-tools/kunit/start.rst new file +180 -0

180 lines not on this page · raw message

Documentation/dev-tools/kunit/usage.rst new file +576 -0

576 lines not on this page · raw message

Kconfig +2 -0
index e10b3ee084d4..47886dbd6c2a 100644
@@ -32,3 +32,5 @@ source "lib/Kconfig"
 source "lib/Kconfig.debug"
 
 source "Documentation/Kconfig"
+
+source "kunit/Kconfig"
MAINTAINERS +13 -0
index 43604d6ab96c..8c1fbcc0d58c 100644
@@ -8807,6 +8807,17 @@ S:	Maintained
 F:	tools/testing/selftests/
 F:	Documentation/dev-tools/kselftest*
 
+KERNEL UNIT TESTING FRAMEWORK (KUnit)
+M:	Brendan Higgins <[email protected]>
+L:	[email protected]
+L:	[email protected]
+W:	https://google.github.io/kunit-docs/third_party/kernel/docs/
+S:	Maintained
+F:	Documentation/dev-tools/kunit/
+F:	include/kunit/
+F:	kunit/
+F:	tools/testing/kunit/
+
 KERNEL USERMODE HELPER
 M:	Luis Chamberlain <[email protected]>
 L:	[email protected]
@@ -12964,12 +12975,14 @@ F:	Documentation/filesystems/proc.txt
 PROC SYSCTL
 M:	Luis Chamberlain <[email protected]>
 M:	Kees Cook <[email protected]>
+M:	Iurii Zaikin <[email protected]>
 L:	[email protected]
 L:	[email protected]
 S:	Maintained
 F:	fs/proc/proc_sysctl.c
 F:	include/linux/sysctl.h
 F:	kernel/sysctl.c
+F:	kernel/sysctl-test.c
 F:	tools/testing/selftests/sysctl/
 
 PS3 NETWORK SUPPORT
Makefile +2 -0
index 9fa18613566f..3db2e47473e2 100644
@@ -1008,6 +1008,8 @@ export MODORDER := $(if $(KBUILD_EXTMOD),$(KBUILD_EXTMOD)/)modules.order
 ifeq ($(KBUILD_EXTMOD),)
 core-y		+= kernel/ certs/ mm/ fs/ ipc/ security/ crypto/ block/
 
+core-$(CONFIG_KUNIT) += kunit/
+
 vmlinux-dirs	:= $(patsubst %/,%,$(filter %/, $(init-y) $(init-m) \
 		     $(core-y) $(core-m) $(drivers-y) $(drivers-m) \
 		     $(net-y) $(net-m) $(libs-y) $(libs-m) $(virt-y)))
arch/um/configs/kunit_defconfig new file +3 -0
new file mode 100644
index 000000000000..9235b7d42d38
@@ -0,0 +1,3 @@
+CONFIG_KUNIT=y
+CONFIG_KUNIT_TEST=y
+CONFIG_KUNIT_EXAMPLE_TEST=y
include/kunit/assert.h new file +356 -0

356 lines not on this page · raw message

include/kunit/string-stream.h new file +51 -0

51 lines not on this page · raw message

include/kunit/test.h new file +1490 -0

1,490 lines not on this page · raw message

include/kunit/try-catch.h new file +75 -0

75 lines not on this page · raw message

kernel/Makefile +2 -0
index ef0d95a190b4..63e9ea6122c2 100644
@@ -113,6 +113,8 @@ obj-$(CONFIG_TORTURE_TEST) += torture.o
 obj-$(CONFIG_HAS_IOMEM) += iomem.o
 obj-$(CONFIG_RSEQ) += rseq.o
 
+obj-$(CONFIG_SYSCTL_KUNIT_TEST) += sysctl-test.o
+
 obj-$(CONFIG_GCC_PLUGIN_STACKLEAK) += stackleak.o
 KASAN_SANITIZE_stackleak.o := n
 KCOV_INSTRUMENT_stackleak.o := n
kernel/sysctl-test.c new file +392 -0

42 lines not on this page · raw message

kunit/Kconfig new file +38 -0
new file mode 100644
index 000000000000..8541ef95b65a

Lines of this file are not shown. View the raw message.

kunit/Makefile new file +9 -0
new file mode 100644
index 000000000000..769d9402b5d3

Lines of this file are not shown. View the raw message.

kunit/assert.c new file +141 -0

141 lines not on this page · raw message

kunit/example-test.c new file +88 -0

88 lines not on this page · raw message

kunit/string-stream-test.c new file +52 -0

52 lines not on this page · raw message

kunit/string-stream.c new file +217 -0

217 lines not on this page · raw message

kunit/test-test.c new file +331 -0

331 lines not on this page · raw message

kunit/test.c new file +478 -0

478 lines not on this page · raw message

kunit/try-catch.c new file +118 -0

118 lines not on this page · raw message

lib/Kconfig.debug +11 -0
index 5960e2980a8a..a425741907b0 100644

Lines of this file are not shown. View the raw message.

tools/objtool/check.c +1 -0
index 176f2f084060..0c8e17f946cd 100644

Lines of this file are not shown. View the raw message.

tools/testing/kunit/.gitignore new file +3 -0
new file mode 100644
index 000000000000..c791ff59a37a

Lines of this file are not shown. View the raw message.

tools/testing/kunit/configs/all_tests.config new file +3 -0
new file mode 100644
index 000000000000..9235b7d42d38

Lines of this file are not shown. View the raw message.

tools/testing/kunit/kunit.py new file +136 -0

136 lines not on this page · raw message

tools/testing/kunit/kunit_config.py new file +66 -0

66 lines not on this page · raw message

tools/testing/kunit/kunit_kernel.py new file +149 -0

149 lines not on this page · raw message

tools/testing/kunit/kunit_parser.py new file +310 -0

310 lines not on this page · raw message

tools/testing/kunit/kunit_tool_test.py new file +206 -0

206 lines not on this page · raw message

tools/testing/kunit/test_data/test_is_test_passed-all_passed.log new file +32 -0
new file mode 100644
index 000000000000..62ebc0288355

Lines of this file are not shown. View the raw message.

tools/testing/kunit/test_data/test_is_test_passed-crash.log new file +69 -0

69 lines not on this page · raw message

tools/testing/kunit/test_data/test_is_test_passed-failure.log new file +36 -0
new file mode 100644
index 000000000000..2ed870b6de72

Lines of this file are not shown. View the raw message.

tools/testing/kunit/test_data/test_is_test_passed-no_tests_run.log new file +75 -0

75 lines not on this page · raw message

tools/testing/kunit/test_data/test_output_isolated_correctly.log new file +106 -0

106 lines not on this page · raw message

tools/testing/kunit/test_data/test_read_from_file.kconfig new file +17 -0
new file mode 100644
index 000000000000..d2a4928ac773

Lines of this file are not shown. View the raw message.

tools/testing/selftests/Makefile +11 -11
index 25b43a8c2b15..c3feccb99ff5 100644

Lines of this file are not shown. View the raw message.

tools/testing/selftests/ftrace/test.d/trigger/trigger-snapshot.tc +1 -1
index 7717c0a09686..ac738500d17f 100644

Lines of this file are not shown. View the raw message.

tools/testing/selftests/livepatch/config +2 -0
index 0dd7700464a8..ad23100cb27c 100644

Lines of this file are not shown. View the raw message.

tools/testing/selftests/seccomp/seccomp_bpf.c +5 -0
index 6ef7f16c4cf5..7f8b5c8982e3 100644

Lines of this file are not shown. View the raw message.

tools/testing/selftests/tpm2/Makefile +1 -0
index 9dd848427a7b..1a5db1eb8ed5 100644

Lines of this file are not shown. View the raw message.

tools/testing/selftests/watchdog/watchdog-test.c +28 -13

41 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Part of the diff is not shown here. View the raw message.

#1 lore raw
Quoted 4 lines · Shuah Khan

This Kselftest update for Linux 5.4-rc1 consists of several fixes to existing tests and adds KUnit, a lightweight unit testing and mocking framework for the Linux kernel from Brendan Higgins.

So I pulled this, but then I almost immediately unpulled it.

My reason for doing that may be odd, but it's because of the top-level 'kunit' directory. This shouldn't be on the top level.

The reason I react so strongly is that it actually breaks my finger memory. I don't type out filenames - I auto-compete them. So "kernel/" is "k<tab>", "drivers/" is "d<tab>" etc.

It already doesn't work for everything ("mm/" is actually "mm<tab>" not because we have files in the git tree, but because the build creates various "module" files), but this breaks a common pattern for me.

Quoted 2 lines

In the future KUnit will be linked to Kselftest framework to provide a way to trigger KUnit tests from user-space.

Can the kernel parts please move to lib/kunit/ or something like that?

Linus

reviewer
#2 lore raw
Quoted 20 lines · Linus Torvalds
Quoted 4 lines · Shuah Khan

This Kselftest update for Linux 5.4-rc1 consists of several fixes to existing tests and adds KUnit, a lightweight unit testing and mocking framework for the Linux kernel from Brendan Higgins.

So I pulled this, but then I almost immediately unpulled it.

My reason for doing that may be odd, but it's because of the top-level 'kunit' directory. This shouldn't be on the top level.

The reason I react so strongly is that it actually breaks my finger memory. I don't type out filenames - I auto-compete them. So "kernel/" is "k<tab>", "drivers/" is "d<tab>" etc.

It already doesn't work for everything ("mm/" is actually "mm<tab>" not because we have files in the git tree, but because the build creates various "module" files), but this breaks a common pattern for me.

Quoted 4 lines
Quoted 2 lines

In the future KUnit will be linked to Kselftest framework to provide a way to trigger KUnit tests from user-space.

Can the kernel parts please move to lib/kunit/ or something like that?

Please also move the top-level Kconfig menu item "KUnit support" to somewhere that is not top-level. Maybe also in the lib/ menu. Maybe in lib/Kconfig.debug.

Signature · 1 line
~Randy

A message that is not in the archive (sent off-list or not yet received): CAKRRn-edxk9Du70A27V=d3Na73fh=fVvGEVsQRGROrQm05YRrA@mail.gmail.com

#4 lore raw
Quoted 24 lines · Shuah Khan

Hi Linus,

Quoted 20 lines · Linus Torvalds
Quoted 4 lines · Shuah Khan

This Kselftest update for Linux 5.4-rc1 consists of several fixes to existing tests and adds KUnit, a lightweight unit testing and mocking framework for the Linux kernel from Brendan Higgins.

So I pulled this, but then I almost immediately unpulled it.

My reason for doing that may be odd, but it's because of the top-level 'kunit' directory. This shouldn't be on the top level.

The reason I react so strongly is that it actually breaks my finger memory. I don't type out filenames - I auto-compete them. So "kernel/" is "k<tab>", "drivers/" is "d<tab>" etc.

It already doesn't work for everything ("mm/" is actually "mm<tab>" not because we have files in the git tree, but because the build creates various "module" files), but this breaks a common pattern for me.

Sorry about that. I am surprised that none of the other reviewers brought this up.

Quoted 6 lines

On hindsight, I probably should have run this by you to get your feedback.

Quoted 4 lines
Quoted 2 lines

In the future KUnit will be linked to Kselftest framework to provide a way to trigger KUnit tests from user-space.

Can the kernel parts please move to lib/kunit/ or something like that.

I'm fine with lib/kunit/.

Quoted 1 line

I will work with Brendan and come up with a plan and send another request early next week.

Cheers

#5 lore raw
Quoted 3 lines · Brendan Higgins

Sorry about that. I am surprised that none of the other reviewers brought this up.

I think I'm "special".

There was some other similar change a few years ago, which I absolutely hated because of how it broke autocomplete for me. Very few other people seemed to react to it.

Part of it may be that the kernel is almost the _only_ project I work with, so unlike a lot of other developers, I end up having muscle memory for kernel-specific issues.

Auto-completion was also one of the (many) reasons why I hated CVS - having that annoying "CVS" directory there just always annoyed me. There's a reason why git uses a dot-file.

So I just have issues that perhaps other people don't react to as much. And aggressive tab-completion happens to be a thing for me.

Linus

#6 lore raw
Quoted 8 lines · Linus Torvalds
Quoted 3 lines · Brendan Higgins

Sorry about that. I am surprised that none of the other reviewers brought this up.

I think I'm "special".

Heh.

Quoted 3 lines

There was some other similar change a few years ago, which I absolutely hated because of how it broke autocomplete for me. Very few other people seemed to react to it.

Well, it's good to know I'm not the first. :-)

Quoted 7 lines

Part of it may be that the kernel is almost the _only_ project I work with, so unlike a lot of other developers, I end up having muscle memory for kernel-specific issues.

Auto-completion was also one of the (many) reasons why I hated CVS - having that annoying "CVS" directory there just always annoyed me. There's a reason why git uses a dot-file.

Yuck. I have never used CVS myself, but the dot-file approach seems much more natural to me. Then again, I have been using git pretty much since I first started programming, so it's hard to say that I am not biased.

Quoted 2 lines

So I just have issues that perhaps other people don't react to as much. And aggressive tab-completion happens to be a thing for me.

Fair enough. On that note, are you okay with the include/kunit/ directory, or do you want me to move it to include/linux/kunit?

#7 lore raw
Quoted 3 lines · Brendan Higgins

Fair enough. On that note, are you okay with the include/kunit/ directory, or do you want me to move it to include/linux/kunit?

"include/kunit" should work just fine for me. At least I didn't react to it immediately when I had done my test-pull, and it doesn't change any auto-completion patterns for me either.

[ We already have two 'k' names under include, but even if that wasn't true, I don't type those names anyway so I wouldn't have had muscle-memory for those two directories in the first place.

Under include, it's "linux" (and to a smaller extent "asm-generic")
that I autocomplete. ]

Linus

#8 lore raw
Quoted 10 lines · Linus Torvalds
Quoted 3 lines · Brendan Higgins

Fair enough. On that note, are you okay with the include/kunit/ directory, or do you want me to move it to include/linux/kunit?

"include/kunit" should work just fine for me. At least I didn't react to it immediately when I had done my test-pull, and it doesn't change any auto-completion patterns for me either.

Sounds good to me.

Thanks!

author maintainer
#9 lore raw
In reply to Linus Torvalds
Quoted 16 lines · Linus Torvalds
Quoted 3 lines · Brendan Higgins

Sorry about that. I am surprised that none of the other reviewers brought this up.

I think I'm "special".

There was some other similar change a few years ago, which I absolutely hated because of how it broke autocomplete for me. Very few other people seemed to react to it.

Part of it may be that the kernel is almost the _only_ project I work with, so unlike a lot of other developers, I end up having muscle memory for kernel-specific issues.

Quoted 7 lines

Auto-completion was also one of the (many) reasons why I hated CVS - having that annoying "CVS" directory there just always annoyed me. There's a reason why git uses a dot-file.

So I just have issues that perhaps other people don't react to as much. And aggressive tab-completion happens to be a thing for me.

Thanks for explaining. Brendan and I will get this sorted out.

Looks like my previous response didn't make it to the kselftest and kernel lists.

thanks,
-- Shuah

#10 lore raw
In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Brendan Higgins

Sorry about that. I am surprised that none of the other reviewers brought this up.

I think I'm "special".

There was some other similar change a few years ago, which I absolutely hated because of how it broke autocomplete for me. Very few other people seemed to react to it.

FWIW, I am obsessively sensitive to autocomplete and overall source code file hieararchy and nomenclature details as well, so it's not just you.

Beyond the muscle memory aspect, nonsensical naming and inanely flat file hierarchies annoy kernel developers and makes it harder for newbies to understand the kernel source as well.

The less clutter, the more organization, the better - and there's very few valid technical reasons to add any new files or directories to the top level directory - we should probably *remove* quite a few.

For example 'firmware/' was recently moved to drivers/firmware/, and in a similar fashion about a third of the remaining 22 directories should probably be moved too:

  drwxr-xr-x    arch
  drwxr-xr-x    block
  drwxr-xr-x    certs           # move to build/certs/ dir
  drwxr-xr-x    crypto          # move to kernel/crypto/ or security/crypto/
  drwxr-xr-x    Documentation
  drwxr-xr-x    drivers
  drwxr-xr-x    fs
  drwxr-xr-x    include
  drwxr-xr-x    init
  drwxr-xr-x    ipc             # move to kernel/ipc/
  drwxr-xr-x    kernel
  drwxr-xr-x    lib
  drwxr-xr-x    LICENSES
  drwxr-xr-x    mm
  drwxr-xr-x    net
  drwxr-xr-x    samples         # move to Documentation/samples/
  drwxr-xr-x    scripts         # move to build/scripts/
  drwxr-xr-x    security
  drwxr-xr-x    sound           # move to drivers/sound/
  drwxr-xr-x    tools
  drwxr-xr-x    usr             # move to build/usr/
  drwxr-xr-x    virt            # move to the already existing drivers/virt/
  -rw-r--r--    COPYING
  -rw-r--r--    CREDITS
  -rw-r--r--    Kbuild
  -rw-r--r--    Kconfig
  -rw-r--r--    MAINTAINERS
  -rw-r--r--    Makefile
  -rw-r--r--    README

There's a few borderline ones:

- 'block' could in principle move to drivers/block/core/ but it's fine at the top level too I think.

- 'init' could in principle be moved to kernel/init/ - but it's not wrong at the top level either.

The remaining top level hierarchy would look pretty sweet and short:

  drwxr-xr-x    arch
  drwxr-xr-x    block
  drwxr-xr-x    build             # new
  drwxr-xr-x    Documentation
  drwxr-xr-x    drivers
  drwxr-xr-x    fs
  drwxr-xr-x    include
  drwxr-xr-x    init
  drwxr-xr-x    kernel
  drwxr-xr-x    lib
  drwxr-xr-x    LICENSES
  drwxr-xr-x    mm
  drwxr-xr-x    net
  drwxr-xr-x    security
  drwxr-xr-x    tools
  -rw-r--r--    COPYING
  -rw-r--r--    CREDITS
  -rw-r--r--    Kbuild
  -rw-r--r--    Kconfig
  -rw-r--r--    MAINTAINERS
  -rw-r--r--    Makefile
  -rw-r--r--    README

I'm volunteering to do this (in a scripted, repeatable, reviewable, tweakable and "easy to execute in a quiet moment" fashion), although I also expect you to balk at the churn. :-)

Thanks,

Ingo

#11 lore raw
Quoted 95 lines · Ingo Molnar
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Brendan Higgins

Sorry about that. I am surprised that none of the other reviewers brought this up.

I think I'm "special".

There was some other similar change a few years ago, which I absolutely hated because of how it broke autocomplete for me. Very few other people seemed to react to it.

FWIW, I am obsessively sensitive to autocomplete and overall source code file hieararchy and nomenclature details as well, so it's not just you.

Beyond the muscle memory aspect, nonsensical naming and inanely flat file hierarchies annoy kernel developers and makes it harder for newbies to understand the kernel source as well.

The less clutter, the more organization, the better - and there's very few valid technical reasons to add any new files or directories to the top level directory - we should probably *remove* quite a few.

For example 'firmware/' was recently moved to drivers/firmware/, and in a similar fashion about a third of the remaining 22 directories should probably be moved too:

  drwxr-xr-x    arch
  drwxr-xr-x    block
  drwxr-xr-x    certs           # move to build/certs/ dir
  drwxr-xr-x    crypto          # move to kernel/crypto/ or security/crypto/
  drwxr-xr-x    Documentation
  drwxr-xr-x    drivers
  drwxr-xr-x    fs
  drwxr-xr-x    include
  drwxr-xr-x    init
  drwxr-xr-x    ipc             # move to kernel/ipc/
  drwxr-xr-x    kernel
  drwxr-xr-x    lib
  drwxr-xr-x    LICENSES
  drwxr-xr-x    mm
  drwxr-xr-x    net
  drwxr-xr-x    samples         # move to Documentation/samples/
  drwxr-xr-x    scripts         # move to build/scripts/
  drwxr-xr-x    security
  drwxr-xr-x    sound           # move to drivers/sound/
  drwxr-xr-x    tools
  drwxr-xr-x    usr             # move to build/usr/
  drwxr-xr-x    virt            # move to the already existing drivers/virt/
  -rw-r--r--    COPYING
  -rw-r--r--    CREDITS
  -rw-r--r--    Kbuild
  -rw-r--r--    Kconfig
  -rw-r--r--    MAINTAINERS
  -rw-r--r--    Makefile
  -rw-r--r--    README

There's a few borderline ones:

- 'block' could in principle move to drivers/block/core/ but it's fine at the top level too I think.

- 'init' could in principle be moved to kernel/init/ - but it's not wrong at the top level either.

The remaining top level hierarchy would look pretty sweet and short:

  drwxr-xr-x    arch
  drwxr-xr-x    block
  drwxr-xr-x    build             # new
  drwxr-xr-x    Documentation
  drwxr-xr-x    drivers
  drwxr-xr-x    fs
  drwxr-xr-x    include
  drwxr-xr-x    init
  drwxr-xr-x    kernel
  drwxr-xr-x    lib
  drwxr-xr-x    LICENSES
  drwxr-xr-x    mm
  drwxr-xr-x    net
  drwxr-xr-x    security
  drwxr-xr-x    tools
  -rw-r--r--    COPYING
  -rw-r--r--    CREDITS
  -rw-r--r--    Kbuild
  -rw-r--r--    Kconfig
  -rw-r--r--    MAINTAINERS
  -rw-r--r--    Makefile
  -rw-r--r--    README
Quoted 3 lines

I'm volunteering to do this (in a scripted, repeatable, reviewable, tweakable and "easy to execute in a quiet moment" fashion), although I also expect you to balk at the churn. :-)

I for one would love the above changes. And I'm the one that has to deal with all of the backporting issues that arise with stable backports :)

thanks,

greg k-h

author maintainer
#12 lore raw
Quoted 96 lines · Greg KH
Quoted 95 lines · Ingo Molnar
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Brendan Higgins

Sorry about that. I am surprised that none of the other reviewers brought this up.

I think I'm "special".

There was some other similar change a few years ago, which I absolutely hated because of how it broke autocomplete for me. Very few other people seemed to react to it.

FWIW, I am obsessively sensitive to autocomplete and overall source code file hieararchy and nomenclature details as well, so it's not just you.

Beyond the muscle memory aspect, nonsensical naming and inanely flat file hierarchies annoy kernel developers and makes it harder for newbies to understand the kernel source as well.

The less clutter, the more organization, the better - and there's very few valid technical reasons to add any new files or directories to the top level directory - we should probably *remove* quite a few.

For example 'firmware/' was recently moved to drivers/firmware/, and in a similar fashion about a third of the remaining 22 directories should probably be moved too:

   drwxr-xr-x    arch
   drwxr-xr-x    block
   drwxr-xr-x    certs           # move to build/certs/ dir
   drwxr-xr-x    crypto          # move to kernel/crypto/ or security/crypto/
   drwxr-xr-x    Documentation
   drwxr-xr-x    drivers
   drwxr-xr-x    fs
   drwxr-xr-x    include
   drwxr-xr-x    init
   drwxr-xr-x    ipc             # move to kernel/ipc/
   drwxr-xr-x    kernel
   drwxr-xr-x    lib
   drwxr-xr-x    LICENSES
   drwxr-xr-x    mm
   drwxr-xr-x    net
   drwxr-xr-x    samples         # move to Documentation/samples/
   drwxr-xr-x    scripts         # move to build/scripts/
   drwxr-xr-x    security
   drwxr-xr-x    sound           # move to drivers/sound/
   drwxr-xr-x    tools
   drwxr-xr-x    usr             # move to build/usr/
   drwxr-xr-x    virt            # move to the already existing drivers/virt/
   -rw-r--r--    COPYING
   -rw-r--r--    CREDITS
   -rw-r--r--    Kbuild
   -rw-r--r--    Kconfig
   -rw-r--r--    MAINTAINERS
   -rw-r--r--    Makefile
   -rw-r--r--    README

There's a few borderline ones:

- 'block' could in principle move to drivers/block/core/ but it's fine at the top level too I think.

- 'init' could in principle be moved to kernel/init/ - but it's not wrong at the top level either.

The remaining top level hierarchy would look pretty sweet and short:

   drwxr-xr-x    arch
   drwxr-xr-x    block
   drwxr-xr-x    build             # new
   drwxr-xr-x    Documentation
   drwxr-xr-x    drivers
   drwxr-xr-x    fs
   drwxr-xr-x    include
   drwxr-xr-x    init
   drwxr-xr-x    kernel
   drwxr-xr-x    lib
   drwxr-xr-x    LICENSES
   drwxr-xr-x    mm
   drwxr-xr-x    net
   drwxr-xr-x    security
   drwxr-xr-x    tools
   -rw-r--r--    COPYING
   -rw-r--r--    CREDITS
   -rw-r--r--    Kbuild
   -rw-r--r--    Kconfig
   -rw-r--r--    MAINTAINERS
   -rw-r--r--    Makefile
   -rw-r--r--    README
Quoted 7 lines
Quoted 3 lines

I'm volunteering to do this (in a scripted, repeatable, reviewable, tweakable and "easy to execute in a quiet moment" fashion), although I also expect you to balk at the churn. :-)

I for one would love the above changes. And I'm the one that has to deal with all of the backporting issues that arise with stable backports :)

I am exploring the possibility to move selftests to a better location or add a git alias so it can be found easily. With the addition of KUnit and future work that is planned to connect kselftest and KUnit, it would make sense have selftests to be in a location that is better suited than where it currently resides.

I have been getting feedback from some developers that they would like to see selftests more visible and easier to find.

There are some dependencies (unintended, shouldn't exist) between some tests and content under tools that might pose some logistical problems, in addition to the churn of backporting.

I haven't explored "git alias" yet though. Since this topic of moving came up, I would liek to get feedback on selftests location in general and where would be a good place for it.

thanks,
-- Shuah

#13 lore raw
Quoted 9 lines · Shuah Khan

I am exploring the possibility to move selftests to a better location or add a git alias so it can be found easily. With the addition of KUnit and future work that is planned to connect kselftest and KUnit, it would make sense have selftests to be in a location that is better suited than where it currently resides.

I have been getting feedback from some developers that they would like to see selftests more visible and easier to find.

Quoted 7 lines

There are some dependencies (unintended, shouldn't exist) between some tests and content under tools that might pose some logistical problems, in addition to the churn of backporting.

I haven't explored "git alias" yet though. Since this topic of moving came up, I would liek to get feedback on selftests location in general and where would be a good place for it.

I'm not sure about the Git alias thing - but I do agree that tools/testing/selftests is a pretty obscure location given the importance of kernel unit tests - and I think it could be moved one level higher, to tools/selftests? The "selftest" name already implies the "test" aspect after all.

Thanks,

Ingo

reviewer
#14 lore raw
Quoted 20 lines · Ingo Molnar
Quoted 9 lines · Shuah Khan

I am exploring the possibility to move selftests to a better location or add a git alias so it can be found easily. With the addition of KUnit and future work that is planned to connect kselftest and KUnit, it would make sense have selftests to be in a location that is better suited than where it currently resides.

I have been getting feedback from some developers that they would like to see selftests more visible and easier to find.

Quoted 7 lines

There are some dependencies (unintended, shouldn't exist) between some tests and content under tools that might pose some logistical problems, in addition to the churn of backporting.

I haven't explored "git alias" yet though. Since this topic of moving came up, I would liek to get feedback on selftests location in general and where would be a good place for it.

Quoted 5 lines

I'm not sure about the Git alias thing - but I do agree that tools/testing/selftests is a pretty obscure location given the importance of kernel unit tests - and I think it could be moved one level higher, to tools/selftests? The "selftest" name already implies the "test" aspect after all.

Without trying to use too much paint, I would move testing/ to a top-level dir, outside of tools/, and leave selftest/ under testing/.

Signature · 1 line
~Randy
author maintainer
#15 lore raw
Quoted 21 lines · Randy Dunlap
Quoted 20 lines · Ingo Molnar
Quoted 16 lines · Shuah Khan

I am exploring the possibility to move selftests to a better location or add a git alias so it can be found easily. With the addition of KUnit and future work that is planned to connect kselftest and KUnit, it would make sense have selftests to be in a location that is better suited than where it currently resides.

I have been getting feedback from some developers that they would like to see selftests more visible and easier to find.

There are some dependencies (unintended, shouldn't exist) between some tests and content under tools that might pose some logistical problems, in addition to the churn of backporting.

I haven't explored "git alias" yet though. Since this topic of moving came up, I would liek to get feedback on selftests location in general and where would be a good place for it.

Quoted 6 lines
Quoted 5 lines

I'm not sure about the Git alias thing - but I do agree that tools/testing/selftests is a pretty obscure location given the importance of kernel unit tests - and I think it could be moved one level higher, to tools/selftests? The "selftest" name already implies the "test" aspect after all.

Right. Obscure location given the importance is the problem.

Quoted 4 lines

Without trying to use too much paint, I would move testing/ to a top-level dir, outside of tools/, and leave selftest/ under testing/.

Right. What you suggesting is very similar to and more complete than what I have been thinking about and proposed at the KS kselftest track.

i.e move tools/testing/selftests to kselftest at the root level. I like your idea of moving tools/testing up to root and keep selftests under it.

If we are good with this kind of change, I would like to get this done sooner than later. There is some back-porting churn to worry about.

thanks,
-- Shuah

#16 lore raw
Quoted 9 lines · Shuah Khan

Right. What you suggesting is very similar to and more complete than what I have been thinking about and proposed at the KS kselftest track.

i.e move tools/testing/selftests to kselftest at the root level. I like your idea of moving tools/testing up to root and keep selftests under it.

If we are good with this kind of change, I would like to get this done sooner than later. There is some back-porting churn to worry about.

I think the movement I suggested would be sufficient:

tools/testing/selftests/ => tools/selftests/

I.e. let's not clutter up the top level directory.

Thanks,

Ingo

author maintainer
#17 lore raw
Quoted 10 lines · Ingo Molnar
Quoted 7 lines · Shuah Khan

Right. What you suggesting is very similar to and more complete than what I have been thinking about and proposed at the KS kselftest track.

i.e move tools/testing/selftests to kselftest at the root level. I like your idea of moving tools/testing up to root and keep selftests under it.

Quoted 9 lines
Quoted 2 lines

If we are good with this kind of change, I would like to get this done sooner than later. There is some back-porting churn to worry about.

I think the movement I suggested would be sufficient:

tools/testing/selftests/ => tools/selftests/

I.e. let's not clutter up the top level directory.

Yeah good point.

-- Shuah

#18 lore raw

[RFC, Tree] De-clutter the top level directory, move ipc/ => kernel/ipc/, samples/ => Documentation/samples/ and sound/ => drivers/sound/

In reply to Greg KH
Quoted 96 lines · Greg KH
Quoted 95 lines · Ingo Molnar
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Brendan Higgins

Sorry about that. I am surprised that none of the other reviewers brought this up.

I think I'm "special".

There was some other similar change a few years ago, which I absolutely hated because of how it broke autocomplete for me. Very few other people seemed to react to it.

FWIW, I am obsessively sensitive to autocomplete and overall source code file hieararchy and nomenclature details as well, so it's not just you.

Beyond the muscle memory aspect, nonsensical naming and inanely flat file hierarchies annoy kernel developers and makes it harder for newbies to understand the kernel source as well.

The less clutter, the more organization, the better - and there's very few valid technical reasons to add any new files or directories to the top level directory - we should probably *remove* quite a few.

For example 'firmware/' was recently moved to drivers/firmware/, and in a similar fashion about a third of the remaining 22 directories should probably be moved too:

  drwxr-xr-x    arch
  drwxr-xr-x    block
  drwxr-xr-x    certs           # move to build/certs/ dir
  drwxr-xr-x    crypto          # move to kernel/crypto/ or security/crypto/
  drwxr-xr-x    Documentation
  drwxr-xr-x    drivers
  drwxr-xr-x    fs
  drwxr-xr-x    include
  drwxr-xr-x    init
  drwxr-xr-x    ipc             # move to kernel/ipc/
  drwxr-xr-x    kernel
  drwxr-xr-x    lib
  drwxr-xr-x    LICENSES
  drwxr-xr-x    mm
  drwxr-xr-x    net
  drwxr-xr-x    samples         # move to Documentation/samples/
  drwxr-xr-x    scripts         # move to build/scripts/
  drwxr-xr-x    security
  drwxr-xr-x    sound           # move to drivers/sound/
  drwxr-xr-x    tools
  drwxr-xr-x    usr             # move to build/usr/
  drwxr-xr-x    virt            # move to the already existing drivers/virt/
  -rw-r--r--    COPYING
  -rw-r--r--    CREDITS
  -rw-r--r--    Kbuild
  -rw-r--r--    Kconfig
  -rw-r--r--    MAINTAINERS
  -rw-r--r--    Makefile
  -rw-r--r--    README

There's a few borderline ones:

- 'block' could in principle move to drivers/block/core/ but it's fine at the top level too I think.

- 'init' could in principle be moved to kernel/init/ - but it's not wrong at the top level either.

The remaining top level hierarchy would look pretty sweet and short:

  drwxr-xr-x    arch
  drwxr-xr-x    block
  drwxr-xr-x    build             # new
  drwxr-xr-x    Documentation
  drwxr-xr-x    drivers
  drwxr-xr-x    fs
  drwxr-xr-x    include
  drwxr-xr-x    init
  drwxr-xr-x    kernel
  drwxr-xr-x    lib
  drwxr-xr-x    LICENSES
  drwxr-xr-x    mm
  drwxr-xr-x    net
  drwxr-xr-x    security
  drwxr-xr-x    tools
  -rw-r--r--    COPYING
  -rw-r--r--    CREDITS
  -rw-r--r--    Kbuild
  -rw-r--r--    Kconfig
  -rw-r--r--    MAINTAINERS
  -rw-r--r--    Makefile
  -rw-r--r--    README
Quoted 6 lines
Quoted 3 lines

I'm volunteering to do this (in a scripted, repeatable, reviewable, tweakable and "easy to execute in a quiet moment" fashion), although I also expect you to balk at the churn. :-)

I for one would love the above changes. And I'm the one that has to deal with all of the backporting issues that arise with stable backports :)

Oh, that's a pleasant surprise, I didn't expect _100%_ support! :-)

So I started working on this today and whipped up three of these movements, in a 100% scripted fashion.

You can have a sneak preview at the result in this tree:

git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git WIP.core/toplevel

...

   2515 files changed, 42476 insertions(+), 42476 deletions(-)

I decided not to post these first patches to lkml, because many of the patches are unreviewably large even with smart rename detection. Even the diffstat was 5,000+ lines so I abbreviated it ...

The tree has 6 commits for now:

6576382877c0: toplevel: Move ipc/ to kernel/ipc/ 8f7ace98971a: toplevel: Fix up kernel/ipc/ movement effects
cc703bc54b00: toplevel: Move sound/ to drivers/sound/
a63bdbd797da: toplevel: Fix up drivers/sound/ movement effects 28adaae5ae8f: toplevel: Move samples/ to Documentation/samples/ 4e9f67e3bc48: toplevel: Fix up samples/ movement effects

Due to the scripting all the commits are auto-generated for the time being - without SOB lines and proper changelogs, etc.

Note that each movement action consists of two commits:

cc703bc54b00: toplevel: Move sound/ to drivers/sound/
a63bdbd797da: toplevel: Fix up drivers/sound/ movement effects

The first one is just the result of the 'git mv', which gets most of the noise.

The second one fixes up the build system, Kconfig system and any references to the old file names. Forward and backward porting of patches across the rename boundary is in most cases a simple matter of fixing up the file names in the patches - I'll put both forward and backward patch-consersion one-liner scripts for that into the changelog itself.

I believe this is easier to review, but it has the disadvantage that bisection will trivially break if it hits any of the base commits. Can combine the commits as well, or can add a suggestion to use "git bisect next" to the changelog itself. Either approach is fine to me.

Anyway, I will fix the aforementioned details, but wanted to show the gist of these changes and give you an opportunity to request changes, before I go down this path too much.

The tree is functional here after some light testing - will put it through more rigorous testing before posting future iterations.

The advantage of the scripting is that:

- I can re-run this on any base tree and with any granularity, and my scripts also have some sanity checks to see whether any old filename patterns escaped the conversion, etc.

- The order of the movement is discretionary as well, i.e. if you prefer the sound/ movement in a single tree and at a specific time, that all can be phased arbitrarily.

- The scripts can be re-run with less than a minute of runtime, so it can all be timed for maximum convenience to you, Linus and affected maintainers.

Is this close to what you had in mind?

Thanks,

Ingo

------------------>
Ingo Molnar (6):
toplevel: Move ipc/ to kernel/ipc/
toplevel: Fix up kernel/ipc/ movement effects
toplevel: Move sound/ to drivers/sound/
toplevel: Fix up drivers/sound/ movement effects
toplevel: Move samples/ to Documentation/samples/
toplevel: Fix up samples/ movement effects

 2515 files changed, 42476 insertions(+), 42476 deletions(-)
#19 lore raw

PATCH v2 1/1 De-clutter the top level directory, move ipc/ => kernel/ipc/, samples/ => Documentation/samples/ and sound/ => drivers/sound/

Quoted 5 lines · Ingo Molnar

Oh, that's a pleasant surprise, I didn't expect _100%_ support! :-)

So I started working on this today and whipped up three of these movements, in a 100% scripted fashion.

Quoted 7 lines

You can have a sneak preview at the result in this tree:

   git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git WIP.core/toplevel

   ...

   2515 files changed, 42476 insertions(+), 42476 deletions(-)

I have pushed out a -v2 version:

git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git WIP.core/toplevel

   2523 files changed, 41304 insertions(+), 41302 deletions(-)

Changes relative to -v1:

- Fixed a bunch of bugs that light testing and light review missed:
missed rename patterns and some build bugs as well. This tree has passed much wider testing, including cross-platform build testing, a distro kernel package build and it also got some light boot testing, just in case.

- Split it into finer grained steps (3 instead of 2 patches per movement), for easier review and bisection testing:

toplevel: Move ipc/ to kernel/ipc/: move the files
toplevel: Move ipc/ to kernel/ipc/: adjust the build system
toplevel: Move ipc/ to kernel/ipc/: adjust comments and documentation

toplevel: Move sound/ to drivers/sound/: move the files
toplevel: Move sound/ to drivers/sound/: adjust the build system
toplevel: Move sound/ to drivers/sound/: adjust comments and documentation

toplevel: Move samples/ to Documentation/samples/: move the files
toplevel: Move samples/ to Documentation/samples/: adjust the build system
toplevel: Move samples/ to Documentation/samples/: adjust comments and documentation

- The changes are now bisection safe if the #1 ('move') and #2 ('build system') patches are squashed. The final #3 'adjust comments and documentation' patch is non-functional in the normal kernel build. I still kept the three steps separate, for reviewability: for many of the changes the build system changes are lost in the noise of the file movement diff itself. (See patch-splitting notes further below.)

- Made some of the build system changes less ad-hoc - but it's still all 100% scripted and automated. Added a SOB to the changelogs, but the changelogs are still barebones. It's on top of Linus's latestest.

- The longer term plan outlined in my first mail is still in flux - the 'scripts/' movement is probaly a bad idea due to its widespread use.

I'm still torn about whether to do a 3-part or 2-part approach for each directory movement:

- The problem with the 2-part approach that merges the 'pure file move' and 'build system' patches so that the latter gets lost in the first one in an almost unreviewable fashion. Someone would have to re-do the git mv step and generate a diff by hand to see the build system changes in isolation...

- The problem with the 3-part approach is that it breaks bisection between the first two patches, although 'git bisect next' would always step to a working commit, because the bisection build-breakage window is only one commit wide.

So I'm slightly leaning toward the 3-part approach for the documentation and review value - but no strong feelings either way.

Anyway, I know everyone is super busy with the merge window, will keep posting new versions every now and then until you or Linus tells me not to bother :-)

Thanks,

Ingo

#20 lore raw

Re: [Tree, v2] De-clutter the top level directory, move ipc/ => kernel/ipc/, samples/ => Documentation/samples/ and sound/ => drivers/sound/

Quoted 6 lines · Ingo Molnar

- Split it into finer grained steps (3 instead of 2 patches per movement), for easier review and bisection testing:

toplevel: Move ipc/ to kernel/ipc/: move the files
toplevel: Move ipc/ to kernel/ipc/: adjust the build system
toplevel: Move ipc/ to kernel/ipc/: adjust comments and documentation

Can we not mess with ipc/ please.

I know that will mess with my muscle memory and I don't see the point. Especially as long as it is named ipc and not sysvipc.

A half cleanup really looks worse than a real cleanup.

SysV IPC really is a side car on the kernel and on unix in general and having the directory structure reflect that seems completely sensible.

Eric

#21 lore raw

Re: [Tree, v2] De-clutter the top level directory, move ipc/ => kernel/ipc/, samples/ => Documentation/samples/ and sound/ => drivers/sound/

Quoted 11 lines · Eric W. Biederman
Quoted 6 lines · Ingo Molnar

- Split it into finer grained steps (3 instead of 2 patches per movement), for easier review and bisection testing:

toplevel: Move ipc/ to kernel/ipc/: move the files
toplevel: Move ipc/ to kernel/ipc/: adjust the build system
toplevel: Move ipc/ to kernel/ipc/: adjust comments and documentation

Can we not mess with ipc/ please.

Quoted 7 lines

I know that will mess with my muscle memory and I don't see the point. Especially as long as it is named ipc and not sysvipc.

A half cleanup really looks worse than a real cleanup.

SysV IPC really is a side car on the kernel and on unix in general and having the directory structure reflect that seems completely sensible.

While such objections are I think valid for scripts/, the situation is very different for ipc/:

- ipc/ is a tiny subsystem of just ~9k lines of code, and it's in the top level directory for historical reasons.

- ipc/ is an established subsystem of old interfaces with comparatively very few changes these days: there were just about 16 commits added in the last 12 months that changed the code and had 'ipc' in the title. Somewhat ironically, the biggest commit of all was a "system call renaming" commit:

275f22148e87: ipc: rename old-style shmctl/semctl/msgctl syscalls 14 files changed, 137 insertions(+), 62 deletions(-)

Many of the remaining 15 commits were simple in nature - and there were 9 different authors, i.e. the per author frequency of changes for ipc/ is even lower: around one per year for those 9 developers who are interested in ipc/ changes. I doubt there's much muscle memory even for them as a result.

- The 'muscle memory' argument has to be weighted by probability of interest (linecount), probability of change (frequency of commits) and number of authors. These factors add up to a very low change frequency for ipc/. We've moved and consolidated much, much bigger and higher frequency subsystems in the recent past, such as kernel/sched/ or kernel/locking/.

- The ipc/ location is arguably random and idiosynchratic - it's a leftover from old times nobody really bothered to clean up - but that fact shouldn't be a permanent barrier IMO.

- While uncluttering the top level directory for aesthethic and documentation reasons is one technical factor, there's another reason for my ipc/ movement: I have the secret hope to be able to move init/ to kernel/init/ as well, in which case there's a big muscle memory advantage for the future: 'i<tab>' would expand to include/ in a single step! :-) Now *that* is perhaps the highest frequency muscle memory location in the kernel. ;-)

Or looking at it from another angle: if we applied your ipc/ benchmark then basically almost *nothing* could be moved from the toplevel directory or any other location, pretty much *ever*.

Thanks,

Ingo

#22 lore raw

Re: [Tree, v2] De-clutter the top level directory, move ipc/ => kernel/ipc/, samples/ => Documentation/samples/ and sound/ => drivers/sound/

Quoted 33 lines · Ingo Molnar
Quoted 11 lines · Eric W. Biederman
Quoted 6 lines · Ingo Molnar

- Split it into finer grained steps (3 instead of 2 patches per movement), for easier review and bisection testing:

toplevel: Move ipc/ to kernel/ipc/: move the files
toplevel: Move ipc/ to kernel/ipc/: adjust the build system
toplevel: Move ipc/ to kernel/ipc/: adjust comments and documentation

Can we not mess with ipc/ please.

Quoted 7 lines

I know that will mess with my muscle memory and I don't see the point. Especially as long as it is named ipc and not sysvipc.

A half cleanup really looks worse than a real cleanup.

SysV IPC really is a side car on the kernel and on unix in general and having the directory structure reflect that seems completely sensible.

While such objections are I think valid for scripts/, the situation is very different for ipc/:

- ipc/ is a tiny subsystem of just ~9k lines of code, and it's in the top level directory for historical reasons.

- ipc/ is an established subsystem of old interfaces with comparatively very few changes these days: there were just about 16 commits added in the last 12 months that changed the code and had 'ipc' in the title. Somewhat ironically, the biggest commit of all was a "system call renaming" commit:

Quoted 9 lines

275f22148e87: ipc: rename old-style shmctl/semctl/msgctl syscalls 14 files changed, 137 insertions(+), 62 deletions(-)

Many of the remaining 15 commits were simple in nature - and there were 9 different authors, i.e. the per author frequency of changes for ipc/ is even lower: around one per year for those 9 developers who are interested in ipc/ changes. I doubt there's much muscle memory even for them as a result.

*snort* Given how long some of those changes took to review. No they
were not all simple. Maybe the final result was but the amount of work
was not simple.

Quoted 6 lines

- The 'muscle memory' argument has to be weighted by probability of interest (linecount), probability of change (frequency of commits) and number of authors. These factors add up to a very low change frequency for ipc/. We've moved and consolidated much, much bigger and higher frequency subsystems in the recent past, such as kernel/sched/ or kernel/locking/.

With I presume with the consent and the input of the developers of those subsystems. Quite frankly digging into kernel/sched to understand what is happening right now is a real mess. It has not made the code any easier to understand (from my outside perspective). But if the developers of the subsystem are fine with it that is their lookout.

Quoted 3 lines

- The ipc/ location is arguably random and idiosynchratic - it's a leftover from old times nobody really bothered to clean up - but that fact shouldn't be a permanent barrier IMO.

Most of the kernel interfaces are random and idiosyncratic. Most of the files and most of their contents as well. That doesn't make them wrong. It does mean that changing them for the sake of change just increases the cognitive load on people who have to look after things like that for no reason.

I figure I have some say in the matter because I am one of the developers doing some of that looking after.

Quoted 9 lines

- While uncluttering the top level directory for aesthethic and documentation reasons is one technical factor, there's another reason for my ipc/ movement: I have the secret hope to be able to move init/ to kernel/init/ as well, in which case there's a big muscle memory advantage for the future: 'i<tab>' would expand to include/ in a single step! :-) Now *that* is perhaps the highest frequency muscle memory location in the kernel. ;-)

Quoted 3 lines

Or looking at it from another angle: if we applied your ipc/ benchmark then basically almost *nothing* could be moved from the toplevel directory or any other location, pretty much *ever*.

Please if it ain't broke don't fix it.

On the other hand we still have a terrible mess from the introduction of threads into the kernel in the 2.5 era. The cleanup code for tasks is in terrible shape. By terrible shape I mean we have data that should be in signal_struct sill in task_struct, and we have code in release_task (called when we reap (aka wait on a zombie)) that can very reasonably be run much sooner, and free up resources in a timely manner. The worst part are zombie thread group leaders that are essentially untested in practice but are a recurring source of errors in kernel code.

Want to help me clean up that code and get rid of zombie thread group leaders?

Especially if you aren't interested in helping please stay away from the code I feel responsible for so I can focus on picking up the messes like that.

Eric

#23 lore raw
In reply to Ingo Molnar
Quoted 41 lines

-----Original Message-----
From: Ingo Molnar on Sunday, September 22, 2019 1:26 AM

Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Brendan Higgins

Sorry about that. I am surprised that none of the other reviewers brought this up.

I think I'm "special".

There was some other similar change a few years ago, which I absolutely hated because of how it broke autocomplete for me. Very few other people seemed to react to it.

FWIW, I am obsessively sensitive to autocomplete and overall source code file hieararchy and nomenclature details as well, so it's not just you.

Beyond the muscle memory aspect, nonsensical naming and inanely flat file hierarchies annoy kernel developers and makes it harder for newbies to understand the kernel source as well.

The less clutter, the more organization, the better - and there's very few valid technical reasons to add any new files or directories to the top level directory - we should probably *remove* quite a few.

For example 'firmware/' was recently moved to drivers/firmware/, and in a similar fashion about a third of the remaining 22 directories should probably be moved too:

  drwxr-xr-x    arch
  drwxr-xr-x    block
  drwxr-xr-x    certs           # move to build/certs/ dir
  drwxr-xr-x    crypto          # move to kernel/crypto/ or security/crypto/
  drwxr-xr-x    Documentation
  drwxr-xr-x    drivers
  drwxr-xr-x    fs
  drwxr-xr-x    include
  drwxr-xr-x    init
Quoted 8 lines
  drwxr-xr-x    ipc             # move to kernel/ipc/
  drwxr-xr-x    kernel
  drwxr-xr-x    lib
  drwxr-xr-x    LICENSES
  drwxr-xr-x    mm
  drwxr-xr-x    net
  drwxr-xr-x    samples         # move to Documentation/samples/
  drwxr-xr-x    scripts         # move to build/scripts/

This one seems like it would break a lot of workflows, and contributor muscle memory and scripts. get_maintainer.pl and checkpatch.pl are probably in quite a few people's scripts.

Also, I'm not sure '/build' is the right destination for this. There are a lot more things in there than just build scripts. If you really want to remove the top level 'scripts', it might be best to put the scripts from top-level '/scripts' into '/tools/scripts', which is mostly empty now.
-- Tim

... rest snipped ...

#24 lore raw
Quoted 54 lines · * [email protected] <[email protected]> wrote:
Quoted 41 lines

-----Original Message-----
From: Ingo Molnar on Sunday, September 22, 2019 1:26 AM

Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Brendan Higgins

Sorry about that. I am surprised that none of the other reviewers brought this up.

I think I'm "special".

There was some other similar change a few years ago, which I absolutely hated because of how it broke autocomplete for me. Very few other people seemed to react to it.

FWIW, I am obsessively sensitive to autocomplete and overall source code file hieararchy and nomenclature details as well, so it's not just you.

Beyond the muscle memory aspect, nonsensical naming and inanely flat file hierarchies annoy kernel developers and makes it harder for newbies to understand the kernel source as well.

The less clutter, the more organization, the better - and there's very few valid technical reasons to add any new files or directories to the top level directory - we should probably *remove* quite a few.

For example 'firmware/' was recently moved to drivers/firmware/, and in a similar fashion about a third of the remaining 22 directories should probably be moved too:

  drwxr-xr-x    arch
  drwxr-xr-x    block
  drwxr-xr-x    certs           # move to build/certs/ dir
  drwxr-xr-x    crypto          # move to kernel/crypto/ or security/crypto/
  drwxr-xr-x    Documentation
  drwxr-xr-x    drivers
  drwxr-xr-x    fs
  drwxr-xr-x    include
  drwxr-xr-x    init
Quoted 8 lines
  drwxr-xr-x    ipc             # move to kernel/ipc/
  drwxr-xr-x    kernel
  drwxr-xr-x    lib
  drwxr-xr-x    LICENSES
  drwxr-xr-x    mm
  drwxr-xr-x    net
  drwxr-xr-x    samples         # move to Documentation/samples/
  drwxr-xr-x    scripts         # move to build/scripts/

This one seems like it would break a lot of workflows, and contributor muscle memory and scripts. get_maintainer.pl and checkpatch.pl are probably in quite a few people's scripts.

Quoted 5 lines

Also, I'm not sure '/build' is the right destination for this. There are a lot more things in there than just build scripts. If you really want to remove the top level 'scripts', it might be best to put the scripts from top-level '/scripts' into '/tools/scripts', which is mostly empty now.

Agreed - I'll leave it alone for now, because you are right that it's widely used.

Thanks,

Ingo

#25 lore raw
In reply to Ingo Molnar
Quoted 26 lines · Ingo Molnar
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Brendan Higgins

Sorry about that. I am surprised that none of the other reviewers brought this up.

I think I'm "special".

There was some other similar change a few years ago, which I absolutely hated because of how it broke autocomplete for me. Very few other people seemed to react to it.

FWIW, I am obsessively sensitive to autocomplete and overall source code file hieararchy and nomenclature details as well, so it's not just you.

Beyond the muscle memory aspect, nonsensical naming and inanely flat file hierarchies annoy kernel developers and makes it harder for newbies to understand the kernel source as well.

The less clutter, the more organization, the better - and there's very few valid technical reasons to add any new files or directories to the top level directory - we should probably *remove* quite a few.

Quoted 8 lines

For example 'firmware/' was recently moved to drivers/firmware/, and in a similar fashion about a third of the remaining 22 directories should probably be moved too:

  drwxr-xr-x    arch
  drwxr-xr-x    block
  drwxr-xr-x    certs           # move to build/certs/ dir
  drwxr-xr-x    crypto          # move to kernel/crypto/ or security/crypto/

For code with lots of history and active development, moving is quite counterproductive as it makes tracking a change tedious. Git can follow the path changes, but that's exactly the step I'd like not to do. That's similar to pure whitespace cleanup patches that are noise.

The decision for move should be IMO up to the maintainers of the code, that apparently worked for firmware/ -> drivers/firmware that has been mentioned. That's fine.

The muscle memory argument sounds quite weak to me, each of us has some habits, editor settings and coding style preferences, we will never agree. That's fine too.

The reason I'd find valid for moving is to reduce confusion when working with the files, not to promote a "formally correct classification" and hierarchy of directories that will stand in the way in the daily work.

Though I'm not directly affected by most of the proposed changes, I feel I should speak up before the file maneuvers reach code I care about.

Quoted 2 lines

- 'block' could in principle move to drivers/block/core/ but it's fine at the top level too I think.

Following that principle, we can move mm/ -> drivers/char/memory/ right? :)

#26 lore raw
In reply to Ingo Molnar

Hi!

Quoted 32 lines
Quoted 5 lines

I think I'm "special".

There was some other similar change a few years ago, which I absolutely hated because of how it broke autocomplete for me. Very few other people seemed to react to it.

FWIW, I am obsessively sensitive to autocomplete and overall source code file hieararchy and nomenclature details as well, so it's not just you.

Beyond the muscle memory aspect, nonsensical naming and inanely flat file hierarchies annoy kernel developers and makes it harder for newbies to understand the kernel source as well.

The less clutter, the more organization, the better - and there's very few valid technical reasons to add any new files or directories to the top level directory - we should probably *remove* quite a few.

For example 'firmware/' was recently moved to drivers/firmware/, and in a similar fashion about a third of the remaining 22 directories should probably be moved too:

  drwxr-xr-x    arch
  drwxr-xr-x    block
  drwxr-xr-x    certs           # move to build/certs/ dir
  drwxr-xr-x    crypto          # move to kernel/crypto/ or security/crypto/
  drwxr-xr-x    Documentation
  drwxr-xr-x    drivers
  drwxr-xr-x    fs
  drwxr-xr-x    include
  drwxr-xr-x    init
  drwxr-xr-x    ipc             # move to kernel/ipc/
  drwxr-xr-x    kernel
Quoted 8 lines
  drwxr-xr-x    lib
  drwxr-xr-x    LICENSES
  drwxr-xr-x    mm
  drwxr-xr-x    net
  drwxr-xr-x    samples         # move to Documentation/samples/
  drwxr-xr-x    scripts         # move to build/scripts/
  drwxr-xr-x    security
  drwxr-xr-x    sound           # move to drivers/sound/

Heh, I was always surprised that sound/ made it into top level... and no, I'd not mind it being moved away.

Quoted 7 lines

There's a few borderline ones:

- 'block' could in principle move to drivers/block/core/ but it's fine at the top level too I think.

- 'init' could in principle be moved to kernel/init/ - but it's not wrong at the top level either.

net would also make sense as drivers/net/core... That is what inspired sound/ afaict.

Quoted 3 lines

I'm volunteering to do this (in a scripted, repeatable, reviewable, tweakable and "easy to execute in a quiet moment" fashion), although I also expect you to balk at the churn. :-)

I'd like to see that happen...

Best regards,
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#27 lore raw
In reply to Ingo Molnar
Quoted 67 lines · Ingo Molnar
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Brendan Higgins

Sorry about that. I am surprised that none of the other reviewers brought this up.

I think I'm "special".

There was some other similar change a few years ago, which I absolutely hated because of how it broke autocomplete for me. Very few other people seemed to react to it.

FWIW, I am obsessively sensitive to autocomplete and overall source code file hieararchy and nomenclature details as well, so it's not just you.

Beyond the muscle memory aspect, nonsensical naming and inanely flat file hierarchies annoy kernel developers and makes it harder for newbies to understand the kernel source as well.

The less clutter, the more organization, the better - and there's very few valid technical reasons to add any new files or directories to the top level directory - we should probably *remove* quite a few.

For example 'firmware/' was recently moved to drivers/firmware/, and in a similar fashion about a third of the remaining 22 directories should probably be moved too:

  drwxr-xr-x    arch
  drwxr-xr-x    block
  drwxr-xr-x    certs           # move to build/certs/ dir
  drwxr-xr-x    crypto          # move to kernel/crypto/ or security/crypto/
  drwxr-xr-x    Documentation
  drwxr-xr-x    drivers
  drwxr-xr-x    fs
  drwxr-xr-x    include
  drwxr-xr-x    init
  drwxr-xr-x    ipc             # move to kernel/ipc/
  drwxr-xr-x    kernel
  drwxr-xr-x    lib
  drwxr-xr-x    LICENSES
  drwxr-xr-x    mm
  drwxr-xr-x    net
  drwxr-xr-x    samples         # move to Documentation/samples/
  drwxr-xr-x    scripts         # move to build/scripts/
  drwxr-xr-x    security
  drwxr-xr-x    sound           # move to drivers/sound/
  drwxr-xr-x    tools
  drwxr-xr-x    usr             # move to build/usr/
  drwxr-xr-x    virt            # move to the already existing drivers/virt/
  -rw-r--r--    COPYING
  -rw-r--r--    CREDITS
  -rw-r--r--    Kbuild
  -rw-r--r--    Kconfig
  -rw-r--r--    MAINTAINERS
  -rw-r--r--    Makefile
  -rw-r--r--    README

There's a few borderline ones:

- 'block' could in principle move to drivers/block/core/ but it's fine at the top level too I think.

Quoted 8 lines

- 'init' could in principle be moved to kernel/init/ - but it's not wrong at the top level either.

The remaining top level hierarchy would look pretty sweet and short:

  drwxr-xr-x    arch
  drwxr-xr-x    block
  drwxr-xr-x    build             # new

I am opposed to the "build".

The build tools will go too deep, like build/scripts/kconfig ?

I often use checkpatch.pl, get_maintainer.pl etc.

Do I have to type build/scripts/checkpatch.pl ?

Quoted 28 lines
  drwxr-xr-x    Documentation
  drwxr-xr-x    drivers
  drwxr-xr-x    fs
  drwxr-xr-x    include
  drwxr-xr-x    init
  drwxr-xr-x    kernel
  drwxr-xr-x    lib
  drwxr-xr-x    LICENSES
  drwxr-xr-x    mm
  drwxr-xr-x    net
  drwxr-xr-x    security
  drwxr-xr-x    tools
  -rw-r--r--    COPYING
  -rw-r--r--    CREDITS
  -rw-r--r--    Kbuild
  -rw-r--r--    Kconfig
  -rw-r--r--    MAINTAINERS
  -rw-r--r--    Makefile
  -rw-r--r--    README

I'm volunteering to do this (in a scripted, repeatable, reviewable, tweakable and "easy to execute in a quiet moment" fashion), although I also expect you to balk at the churn. :-)

Thanks,

Ingo

Signature · 2 lines
Best Regards
Masahiro Yamada