modules, split MODULE_STATE_UNFORMED into separate states
Subsystems (3)
PATCH 1/1 modules, split MODULE_STATE_UNFORMED into separate states
Just a heads up, the changes here are really to the core module code. The changes to kgdb and the tracing code are small simple changes.
P.
---8<---
A panic was seen in the following sitation.
There are two threads running on the system. The first thread is a system monitoring thread that is reading /proc/modules. The second thread is loading and unloading a module (in this example I'm using my simple dummy-module.ko). Note, in the "real world" this occurred with the qlogic driver module.
When doing this, the following panic occurred:
------------[ cut here ]------------ kernel BUG at kernel/module.c:3739! invalid opcode: 0000 [#1] SMP Modules linked in: binfmt_misc sg nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel lrw igb gf128mul glue_helper iTCO_wdt iTCO_vendor_support ablk_helper ptp sb_edac cryptd pps_core edac_core shpchp i2c_i801 pcspkr wmi lpc_ich ioatdma mfd_core dca ipmi_si nfsd ipmi_msghandler auth_rpcgss nfs_acl lockd sunrpc xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_common mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm isci drm libsas ahci libahci scsi_transport_sas libata i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: dummy_module] CPU: 37 PID: 186343 Comm: cat Tainted: GF O-------------- 3.10.0+ #7 Hardware name: Intel Corporation S2600CP/S2600CP, BIOS RMLSDP.86I.00.29.D696.1311111329 11/11/2013 task: ffff8807fd2d8000 ti: ffff88080fa7c000 task.ti: ffff88080fa7c000 RIP: 0010:[<ffffffff810d64c5>] [<ffffffff810d64c5>] module_flags+0xb5/0xc0 RSP: 0018:ffff88080fa7fe18 EFLAGS: 00010246 RAX: 0000000000000003 RBX: ffffffffa03b5200 RCX: 0000000000000000 RDX: 0000000000001000 RSI: ffff88080fa7fe38 RDI: ffffffffa03b5000 RBP: ffff88080fa7fe28 R08: 0000000000000010 R09: 0000000000000000 R10: 0000000000000000 R11: 000000000000000f R12: ffffffffa03b5000 R13: ffffffffa03b5008 R14: ffffffffa03b5200 R15: ffffffffa03b5000 FS: 00007f6ae57ef740(0000) GS:ffff88101e7a0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000404f70 CR3: 0000000ffed48000 CR4: 00000000001407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Stack: ffffffffa03b5200 ffff8810101e4800 ffff88080fa7fe70 ffffffff810d666c ffff88081e807300 000000002e0f2fbf 0000000000000000 ffff88100f257b00 ffffffffa03b5008 ffff88080fa7ff48 ffff8810101e4800 ffff88080fa7fee0 Call Trace: [<ffffffff810d666c>] m_show+0x19c/0x1e0 [<ffffffff811e4d7e>] seq_read+0x16e/0x3b0 [<ffffffff812281ed>] proc_reg_read+0x3d/0x80 [<ffffffff811c0f2c>] vfs_read+0x9c/0x170 [<ffffffff811c1a58>] SyS_read+0x58/0xb0 [<ffffffff81605829>] system_call_fastpath+0x16/0x1b Code: 48 63 c2 83 c2 01 c6 04 03 29 48 63 d2 eb d9 0f 1f 80 00 00 00 00 48 63 d2 c6 04 13 2d 41 8b 0c 24 8d 50 02 83 f9 01 75 b2 eb cb <0f> 0b 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41 RIP [<ffffffff810d64c5>] module_flags+0xb5/0xc0 RSP <ffff88080fa7fe18>
Consider the two processes running on the system.
CPU 0 (/proc/modules reader)
CPU 1 (loading/unloading module)
CPU 0 opens /proc/modules, and starts displaying data for each module by traversing the modules list via fs/seq_file.c:seq_open() and fs/seq_file.c:seq_read(). For each module in the modules list, seq_read does
op->start() <-- this is a pointer to m_start()
op->show() <- this is a pointer to m_show()
op->stop() <-- this is a pointer to m_stop()
The m_start(), m_show(), and m_stop() module functions are defined in kernel/module.c. The m_start() and m_stop() functions acquire and release the module_mutex respectively.
ie) When reading /proc/modules, the module_mutex is acquired and released for each module.
m_show() is called with the module_mutex held. It accesses the module struct data and attempts to write out module data. It is in this code path that the above BUG_ON() warning is encountered, specifically m_show() calls
static char *module_flags(struct module *mod, char *buf) {
int bx = 0;
BUG_ON(mod->state == MODULE_STATE_UNFORMED);
...
The other thread, CPU 1, in unloading the module calls the syscall delete_module() defined in kernel/module.c. The module_mutex is acquired for a short time, and then released. free_module() is called without the module_mutex. free_module() then sets mod->state = MODULE_STATE_UNFORMED, also without the module_mutex. Some additional code is called and then the module_mutex is reacquired to remove the module from the modules list:
/* Now we can delete it from the lists */
mutex_lock(&module_mutex);
stop_machine(__unlink_module, mod, NULL);
mutex_unlock(&module_mutex);
This is the sequence of events that leads to the panic.
CPU 1 is removing dummy_module via delete_module(). It acquires the module_mutex, and then releases it. CPU 1 has NOT set dummy_module->state to MODULE_STATE_UNFORMED yet.
CPU 0, which is reading the /proc/modules, acquires the module_mutex and acquires a pointer to the dummy_module which is still in the modules list. CPU 0 calls m_show for dummy_module. The check in m_show() for MODULE_STATE_UNFORMED passed for dummy_module even though it is being torn down.
Meanwhile CPU 1, which has been continuing to remove dummy_module without holding the module_mutex, now calls free_module() and sets dummy_module->state to MODULE_STATE_UNFORMED.
CPU 0 now calls module_flags() with dummy_module and ...
static char *module_flags(struct module *mod, char *buf)
{
int bx = 0;
BUG_ON(mod->state == MODULE_STATE_UNFORMED);and BOOM.
Inspection of the code calls into question whether-or-not module_flags() should have that BUG_ON(). It seems to me that the BUG_ON() only has purpose during a module load, as the data is in a questionable state when module_flags() is called and a BUG_ON() is appropriate. On a module delete the data is valid until the module is removed from the modules list, after which module_flags() won't be called on the module.
MODULE_STATE_UNFORMED needs to be separated into two states; one for the module load (MODULE_STATE_LOAD), and one for the module delete (MODULE_STATE_DELETE). This fixes the "false" BUG_ON(), and allows the output of module_flags to be consistent and correct for panic output and /proc/modules output.
To make the code a bit cleaner I've added a helper function, module_state_is_unformed(), which reports if the module is in either new state.
In addition to this I've updated the module state descriptions with slightly more detail.
Testing: In the unpatched kernel I can panic the system within 1 minute by doing
while (true) do insmod dummy_module.ko; rmmod dummy_module.ko; done
and
while (true) do cat /proc/modules; done
in separate terminals.
In the patched kernel I was able to run just over one hour without seeing any issues. I also verified the output of panic via sysrq-c and the output of /proc/modules looks correct for all three states for the dummy_module.
dummy_module 12661 0 - Unloading 0xffffffffa03a5000 (OE-)
dummy_module 12661 0 - Live 0xffffffffa03bb000 (OE)
dummy_module 14015 1 - Loading 0xffffffffa03a5000 (OE+)--- include/linux/module.h | 13 ++++++-- kernel/debug/kdb/kdb_main.c | 2 +- kernel/module.c | 74 ++++++++++++++++++++++++++----------------- kernel/tracepoint.c | 3 +- 4 files changed, 59 insertions(+), 33 deletions(-)
include/linux/module.h +11 -2
@@ -206,8 +206,9 @@ struct module_use {enum module_state {MODULE_STATE_LIVE ,/* Normal state. */ MODULE_STATE_COMING ,/* Full formed, running module_init. */ -MODULE_STATE_GOING ,/* Going away. */ -MODULE_STATE_UNFORMED ,/* Still setting it up. */ +MODULE_STATE_GOING ,/* Early delete. */ +MODULE_STATE_LOAD ,/* Early init. */ +MODULE_STATE_DELETE ,/* Late delete. */ };/**
@@ -390,6 +391,14 @@ static inline int module_is_live(struct module *mod)return mod->state !=MODULE_STATE_GOING ; } +static inline bool module_state_is_unformed (struct module *module) +{ +if ((module->state ==MODULE_STATE_LOAD ) || + (module->state ==MODULE_STATE_DELETE )) +return true; +return false; +} +struct module *__module_text_address (unsigned long addr);struct module *__module_address (unsigned long addr);bool is_module_address (unsigned long addr);
kernel/debug/kdb/kdb_main.c +1 -1
@@ -1973,7 +1973,7 @@ static int kdb_lsmod(int argc, const char **argv)kdb_printf ("Module Size modstruct Used by\n" );list_for_each_entry (mod, kdb_modules, list) { -if (mod->state ==MODULE_STATE_UNFORMED ) +if (module_state_is_unformed (mod))continue ; kdb_printf("%-20s%8u 0x%p ", mod->name,
kernel/module.c +45 -29
210 lines not on this page · raw message
kernel/tracepoint.c +2 -1
@@ -435,7 +435,8 @@ static int tracepoint_module_notify(struct notifier_block *self,case MODULE_STATE_GOING :tracepoint_module_going (mod);break ; -case MODULE_STATE_UNFORMED : +case MODULE_STATE_LOAD : +case MODULE_STATE_DELETE :break ; }return ret;
Signature · 1 line
1.7.9.3
Quoted 4 lines · Prarit Bhargava
MODULE_STATE_UNFORMEDneeds to be separated into two states; one for the module load (MODULE_STATE_LOAD), and one for the module delete (MODULE_STATE_DELETE).
And personally I think this makes sense in any case, but I can't really comment the changes in this area.
kernel/module.c static int m_show(struct seq_file *m, void *p) · line 3655
Quoted 13 lines
kernel/module.c +6
-2@@ -3647,18 +3646,29 @@ static int m_show(struct seq_file *m, void *p) struct module *mod = list_entry(p, struct module, list);char buf[8 ]; -/* We always ignore unformed modules. */ -if (mod->state ==MODULE_STATE_UNFORMED ) +/* +* If the state is MODULE_STATE_LOAD then the module is in +* the early stages of loading. No information should be printed +* for this module as the data could be in an uninitialized state. +*/ +if (mod->state ==MODULE_STATE_LOAD )return 0 ;
So this assumes that _UNFORMED state is fine...
Not sure, but I can be easily wrong. For example, print_unload_info() -> module_refcount() plays with mod->refptr, while free_module() does module_unload_free() -> free_percpu(mod->refptr). No?
Perhaps it makes sense to start with the simple patch for stable,
+ // sync with m_show() + mutex_lock(module_mutex); mod->state = MODULE_STATE_UNFORMED; + mutex_unlock(module_mutex);
then do a more sophisticated fix?
Oleg.
argh, sorry for noise,
Quoted 14 lines · Oleg Nesterov
Quoted 4 lines · Prarit Bhargava
MODULE_STATE_UNFORMEDneeds to be separated into two states; one for the module load (MODULE_STATE_LOAD), and one for the module delete (MODULE_STATE_DELETE).And personally I think this makes sense in any case, but I can't really comment the changes in this area.
Quoted 4 lines
kernel/module.c +0
-0@@ -3647,18 +3646,29 @@ static int m_show(struct seq_file *m, void *p) struct module *mod = list_entry(p, struct module, list);char buf[8 ];
Quoted 11 lines
kernel/module.c static int m_show(struct seq_file *m, void *p) · line 3655
Quoted 9 lines
kernel/module.c +6
-2-/* We always ignore unformed modules. */ -if (mod->state ==MODULE_STATE_UNFORMED ) +/* +* If the state is MODULE_STATE_LOAD then the module is in +* the early stages of loading. No information should be printed +* for this module as the data could be in an uninitialized state. +*/ +if (mod->state ==MODULE_STATE_LOAD )return 0 ;So this assumes that
_UNFORMEDstate is fine...
^^^^^^^^^
I meant MODULE_STATE_DELETE of course...
Quoted 14 lines
Not sure, but I can be easily wrong. For example,
print_unload_info()->module_refcount()plays withmod->refptr, whilefree_module()doesmodule_unload_free()->free_percpu(mod->refptr). No?Perhaps it makes sense to start with the simple patch for stable,
+ // sync with m_show() + mutex_lock(module_mutex); mod->state = MODULE_STATE_UNFORMED; + mutex_unlock(module_mutex);then do a more sophisticated fix?
Oleg.
Quoted 25 lines · Oleg Nesterov
Quoted 4 lines · Prarit Bhargava
MODULE_STATE_UNFORMEDneeds to be separated into two states; one for the module load (MODULE_STATE_LOAD), and one for the module delete (MODULE_STATE_DELETE).And personally I think this makes sense in any case, but I can't really comment the changes in this area.
kernel/module.c static int m_show(struct seq_file *m, void *p) · line 3655
Quoted 13 lines
kernel/module.c +6
-2@@ -3647,18 +3646,29 @@ static int m_show(struct seq_file *m, void *p) struct module *mod = list_entry(p, struct module, list);char buf[8 ]; -/* We always ignore unformed modules. */ -if (mod->state ==MODULE_STATE_UNFORMED ) +/* +* If the state is MODULE_STATE_LOAD then the module is in +* the early stages of loading. No information should be printed +* for this module as the data could be in an uninitialized state. +*/ +if (mod->state ==MODULE_STATE_LOAD )return 0 ;So this assumes that
_UNFORMEDstate is fine...
Quoted 3 lines
Not sure, but I can be easily wrong. For example,
print_unload_info()->module_refcount()plays withmod->refptr, whilefree_module()doesmodule_unload_free()->free_percpu(mod->refptr). No?
Oh geez -- I didn't see that in module_unload_free(). I had assumed that all the percpu data was free'd in free_module() call to percpu_modfree(mod) ...
You're right though, the _DELETE state is not okay in this path, and if that's the case then I'm not sure we have to distinguish the two cases.
Quoted 9 lines
Perhaps it makes sense to start with the simple patch for stable,
+ // sync with m_show() + mutex_lock(module_mutex); mod->state = MODULE_STATE_UNFORMED; + mutex_unlock(module_mutex);then do a more sophisticated fix?
I actually toyed around with this but thought that was too "hacky" for a fix. But if Rusty is okay with it, I'd be okay with it too.
P.
Quoted 3 lines
Oleg.