Skip to contentSkip to conversation

Unable to load modules from 9p filesystem with kmod 16

DISCUSSION Wakko Warner 3 participants 5 messages Jan 10, 2014 – Jan 11, 2014

lore
Share & tools
lore
#0 lore raw

Kernel 3.12.7 from kernel.org
With kmod-16, I'm unable to load any modules on my guest kvm machines. The vm is booted via direct kernel boot. The modules are located on the host and is passed to the guest via the fsdev.

I have a mountpoint on the guest filesystem located at /kernel. It is mounted like this:
kernel /kernel 9p rw,sync,dirsync,relatime,trans=virtio,version=9p2000.L 0 0

/boot, /lib/modules, and /lib/firmware are tmpfs filesystems like this:
kboot /boot tmpfs rw,relatime,size=0k,nr_inodes=8 0 0 kfirmware /lib/firmware tmpfs rw,relatime,size=0k,nr_inodes=8 0 0 kmodules /lib/modules tmpfs rw,relatime,size=0k,nr_inodes=8 0 0

/lib/modules/$(uname -r) is a symlink to /kernel/lib/modules/$(uname -r)

When trying to load any module, I get
Error: could not insert module /lib/modules/3.12.7/kernel/crypto/af_alg.ko:
Invalid module format

This module was just one I picked at random, all modules fail the same way.
Strace shows this:
open("/lib/modules/3.12.7/kernel/crypto/af_alg.ko", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=13822, ...}) = 0
mmap(NULL, 13822, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f199aebd000
syscall_313(0x3, 0x7f199aaa2de0, 0, 0x3, 0, 0x7f199b7b2010, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) = -1 (errno 8)
munmap(0x7f199aebd000, 13822) = 0
close(3) = 0

If I use kmod-9, it works w/o problems. An strace of kmod-9 shows:
open("/lib/modules/3.12.7/kernel/crypto/af_alg.ko", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=13822, ...}) = 0 mmap(NULL, 13822, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f658ba41000

init_module(0x7f658ba41000, 13822, "")  = 0
munmap(0x7f658ba41000, 13822)           = 0
close(3)                                = 0

If I copy the module to /tmp, kmod-16 loads it just fine. It appears to be related to the 9p filesystem.

I won't rule out a bug in kmod-16, but I was unable to figure out what the cause of the error was. I tried to search through kmod's source and the kernel source, but I gave up because I don't understand what's going on.

Signature · 2 lines
 Microsoft has beaten Volkswagen's world record.  Volkswagen only created 22
 million bugs.
#1 lore raw
Quoted 37 lines · Wakko Warner

Kernel 3.12.7 from kernel.org
With kmod-16, I'm unable to load any modules on my guest kvm machines. The vm is booted via direct kernel boot. The modules are located on the host and is passed to the guest via the fsdev.

I have a mountpoint on the guest filesystem located at /kernel. It is mounted like this:
kernel /kernel 9p rw,sync,dirsync,relatime,trans=virtio,version=9p2000.L 0 0

/boot, /lib/modules, and /lib/firmware are tmpfs filesystems like this:
kboot /boot tmpfs rw,relatime,size=0k,nr_inodes=8 0 0 kfirmware /lib/firmware tmpfs rw,relatime,size=0k,nr_inodes=8 0 0 kmodules /lib/modules tmpfs rw,relatime,size=0k,nr_inodes=8 0 0

/lib/modules/$(uname -r) is a symlink to /kernel/lib/modules/$(uname -r)

When trying to load any module, I get
Error: could not insert module /lib/modules/3.12.7/kernel/crypto/af_alg.ko:
Invalid module format

This module was just one I picked at random, all modules fail the same way.
Strace shows this:
open("/lib/modules/3.12.7/kernel/crypto/af_alg.ko", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=13822, ...}) = 0
mmap(NULL, 13822, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f199aebd000
syscall_313(0x3, 0x7f199aaa2de0, 0, 0x3, 0, 0x7f199b7b2010, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) = -1 (errno 8)
munmap(0x7f199aebd000, 13822) = 0
close(3) = 0

If I use kmod-9, it works w/o problems. An strace of kmod-9 shows:
open("/lib/modules/3.12.7/kernel/crypto/af_alg.ko", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=13822, ...}) = 0 mmap(NULL, 13822, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f658ba41000

init_module(0x7f658ba41000, 13822, "")  = 0
munmap(0x7f658ba41000, 13822)           = 0
close(3)                                = 0
Quoted 6 lines

If I copy the module to /tmp, kmod-16 loads it just fine. It appears to be related to the 9p filesystem.

I won't rule out a bug in kmod-16, but I was unable to figure out what the cause of the error was. I tried to search through kmod's source and the kernel source, but I gave up because I don't understand what's going on.

I did some testing with the various kmod versions between kmod-9 and 16. The problem was introduced in 13. The change that broke this was the addition of finit_module(). Appears that the kernel cannot handle modules that is on the 9p filesystem. On the line that checked err == 0 (around line libkmod-module.c:823), I added a test for ENOEXEC. The finit_module still fails,but it uses the old method and works.

I would say from what I've seen, the problem is in the kernel for finit_module(). I don't know enough about the kernel to go any further.

#2 lore raw

[adding Cc:s]

Quoted 55 lines · Wakko Warner
Quoted 37 lines · Wakko Warner

Kernel 3.12.7 from kernel.org
With kmod-16, I'm unable to load any modules on my guest kvm machines. The vm is booted via direct kernel boot. The modules are located on the host and is passed to the guest via the fsdev.

I have a mountpoint on the guest filesystem located at /kernel. It is mounted like this:
kernel /kernel 9p rw,sync,dirsync,relatime,trans=virtio,version=9p2000.L 0 0

/boot, /lib/modules, and /lib/firmware are tmpfs filesystems like this:
kboot /boot tmpfs rw,relatime,size=0k,nr_inodes=8 0 0 kfirmware /lib/firmware tmpfs rw,relatime,size=0k,nr_inodes=8 0 0 kmodules /lib/modules tmpfs rw,relatime,size=0k,nr_inodes=8 0 0

/lib/modules/$(uname -r) is a symlink to /kernel/lib/modules/$(uname -r)

When trying to load any module, I get
Error: could not insert module /lib/modules/3.12.7/kernel/crypto/af_alg.ko:
Invalid module format

This module was just one I picked at random, all modules fail the same way.
Strace shows this:
open("/lib/modules/3.12.7/kernel/crypto/af_alg.ko", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=13822, ...}) = 0
mmap(NULL, 13822, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f199aebd000
syscall_313(0x3, 0x7f199aaa2de0, 0, 0x3, 0, 0x7f199b7b2010, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) = -1 (errno 8)
munmap(0x7f199aebd000, 13822) = 0
close(3) = 0

If I use kmod-9, it works w/o problems. An strace of kmod-9 shows:
open("/lib/modules/3.12.7/kernel/crypto/af_alg.ko", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=13822, ...}) = 0 mmap(NULL, 13822, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f658ba41000

init_module(0x7f658ba41000, 13822, "")  = 0
munmap(0x7f658ba41000, 13822)           = 0
close(3)                                = 0
Quoted 6 lines

If I copy the module to /tmp, kmod-16 loads it just fine. It appears to be related to the 9p filesystem.

I won't rule out a bug in kmod-16, but I was unable to figure out what the cause of the error was. I tried to search through kmod's source and the kernel source, but I gave up because I don't understand what's going on.

I did some testing with the various kmod versions between kmod-9 and 16. The problem was introduced in 13. The change that broke this was the addition of finit_module(). Appears that the kernel cannot handle modules that is on the 9p filesystem. On the line that checked err == 0 (around line libkmod-module.c:823), I added a test for ENOEXEC. The finit_module still fails,but it uses the old method and works.

I would say from what I've seen, the problem is in the kernel for finit_module(). I don't know enough about the kernel to go any further. --

Signature · 1 line
~Randy
#3 lore raw

Hi,

Randy Dunlap wrote on Fri, Jan 10, 2014 :

Quoted 1 line

[adding Cc:s]

Thanks for the extra Cc.

Quoted 22 lines · Wakko Warner
Quoted 21 lines
Quoted 20 lines · Wakko Warner

Kernel 3.12.7 from kernel.org
With kmod-16, I'm unable to load any modules on my guest kvm machines. The vm is booted via direct kernel boot. The modules are located on the host and is passed to the guest via the fsdev.

I have a mountpoint on the guest filesystem located at /kernel. It is mounted like this:
kernel /kernel 9p rw,sync,dirsync,relatime,trans=virtio,version=9p2000.L 0 0

/boot, /lib/modules, and /lib/firmware are tmpfs filesystems like this:
kboot /boot tmpfs rw,relatime,size=0k,nr_inodes=8 0 0 kfirmware /lib/firmware tmpfs rw,relatime,size=0k,nr_inodes=8 0 0 kmodules /lib/modules tmpfs rw,relatime,size=0k,nr_inodes=8 0 0

/lib/modules/$(uname -r) is a symlink to /kernel/lib/modules/$(uname -r)

When trying to load any module, I get
Error: could not insert module /lib/modules/3.12.7/kernel/crypto/af_alg.ko:
Invalid module format

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

This module was just one I picked at random, all modules fail the same way.
Strace shows this:
open("/lib/modules/3.12.7/kernel/crypto/af_alg.ko", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=13822, ...}) = 0
mmap(NULL, 13822, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f199aebd000
syscall_313(0x3, 0x7f199aaa2de0, 0, 0x3, 0, 0x7f199b7b2010, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) = -1 (errno 8)
munmap(0x7f199aebd000, 13822) = 0
close(3) = 0

I didn't look at the patch very closely because the mail apparently never reached v9fs-developer (although it looks in copy), but I got pointed out this patch recently:
http://www.spinics.net/lists/linux-virtualization/msg21716.html

Which quotes "This patch enables 9p-virtio to correctly handle this case. This not only enables us to load Linux kernel modules off virtfs,"

Perhaps would it be what you need to support this syscall_313?

Good luck,

Signature · 1 line
Dominique Martinet
#4 lore raw
Quoted 1 line · Dominique Martinet

Hi,

Hi.

The following was trimmed.

Quoted 12 lines
Quoted 11 lines · Wakko Warner
Quoted 10 lines
Quoted 9 lines · Wakko Warner

Kernel 3.12.7 from kernel.org
With kmod-16, I'm unable to load any modules on my guest kvm machines.

open("/lib/modules/3.12.7/kernel/crypto/af_alg.ko", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=13822, ...}) = 0
mmap(NULL, 13822, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f199aebd000
syscall_313(0x3, 0x7f199aaa2de0, 0, 0x3, 0, 0x7f199b7b2010, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) = -1 (errno 8)
munmap(0x7f199aebd000, 13822) = 0
close(3) = 0

Quoted 4 lines

I didn't look at the patch very closely because the mail apparently never reached v9fs-developer (although it looks in copy), but I got pointed out this patch recently:
http://www.spinics.net/lists/linux-virtualization/msg21716.html

I applied the patch on that page. Here's my result:

# df -T /lib/modules/3.12.7/kernel              
Filesystem     Type 1K-blocks   Used Available Use% Mounted on
kernel         9p     2015056 721356   1172460  39% /kernel
# uname -r
3.12.7
# modprobe -V
kmod version 16
# lsmod
Module                  Size  Used by
nfsd                  217278  2 
auth_rpcgss            41307  1 nfsd
oid_registry            2179  1 auth_rpcgss
exportfs                3499  1 nfsd
nfs                   111109  0 
...
Quoted 4 lines

Which quotes "This patch enables 9p-virtio to correctly handle this case. This not only enables us to load Linux kernel modules off virtfs,"

Perhaps would it be what you need to support this syscall_313?

Looks like that's the fix.

Signature · 2 lines
 Microsoft has beaten Volkswagen's world record.  Volkswagen only created 22
 million bugs.