Migrating net/sched to new module interface
Subsystems (2)
RFC 1/1 Migrating net/sched to new module interface
Hi,
I'm working on net/sched to convert it to new module interface. I'm going to add a new field 'owner' to struct Qdisc_ops. The init function of each module registers the callback functions and a reference to the module itself. When a qdisc is created (qdisc_create) I check try_module_get() to see if I can use the registered callback functions and if I can't I return -ENOSYS. When a qdisc is delete (qdisc_destroy) I use module_put() to decrement the refcount. The attacched patch shows this preliminar work on sch_api.c sch_generic.c.
The next step is to update the other schedulers removing
MOD_{INC,DEC}_USE_COUNT and adding the new 'owner' field.
This work is based on my (maybe poor) understanding on the new module system, so I'm waiting some feedback before going on.
include/net/pkt_sched.h +3 -0
@@ -10,6 +10,7 @@#include <linux/config.h> #include <linux/types.h> #include <linux/pkt_sched.h> +#include <linux/module.h> #include <net/pkt_cls.h> #ifdef CONFIG_X86_TSC
@@ -67,6 +68,8 @@int (*change)(struct Qdisc *,struct rtattr *arg);int (*dump)(struct Qdisc *,struct sk_buff *); +/* Protects callbacks */ +struct module *owner; };extern rwlock_t qdisc_tree_lock;
net/sched/sch_api.c +6 -6
@@ -409,6 +409,10 @@if (ops ==NULL ) goto err_out; + err = -ENOSYS ; +if (try_module_get (ops->owner) ==0 ) + goto err_module; + size =sizeof (*sch) + ops->priv_size; sch =kmalloc (size,GFP_KERNEL );
@@ -416,12 +420,6 @@if (!sch) goto err_out; -/* Grrr... Resolve race condition with module unload */ - - err = -EINVAL ; -if (ops !=qdisc_lookup_ops (kind)) - goto err_out; -memset (sch,0 , size);skb_queue_head_init (&sch->q);
@@ -460,6 +458,8 @@ } err_out: +module_put (ops->owner); +err_module: *errp = err;if (sch)kfree (sch);
net/sched/sch_generic.c +4 -0
@@ -29,6 +29,7 @@#include <linux/skbuff.h> #include <linux/rtnetlink.h> #include <linux/init.h> +#include <linux/module.h> #include <net/sock.h> #include <net/pkt_sched.h>
@@ -356,6 +357,8 @@ .init = pfifo_fast_init, .reset = pfifo_fast_reset, + + .owner = THIS_MODULE, };struct Qdisc *qdisc_create_dflt (struct net_device *dev,struct Qdisc_ops *ops)
@@ -422,6 +425,7 @@ ops->reset (qdisc);if (ops->destroy) ops->destroy (qdisc); +module_put (ops->owner);if (!(qdisc->flags&TCQ_F_BUILTIN ))kfree (qdisc); }
ciao,
Luca
Signature · 5 lines
Reply-To: [email protected] Home: http://kronoz.cjb.net "Chi parla in tono cortese, ma continua a prepararsi, potra` andare avanti; chi parla in tono bellicoso e avanza rapidamente dovra` ritirarsi" Sun Tzu -- L'arte della guerra
In message <[email protected]> you write:
Quoted 4 lines
Hi,
I'm working on net/sched to convert it to new module interface. I'm going to add a new field 'owner' to struct
Qdisc_ops.
Hmm, I thought the sched stuff all runs under the network brlock? If so, it doesn't need to be held in, since it's not preemptible.
I haven't checked, though...
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
Quoted 2 lines · Rusty Russell
Hmm, I thought the sched stuff all runs under the network brlock? If so, it doesn't need to be held in, since it's not preemptible.
The packet schedulers transmit, not receive. They have their own queue locking, along with the device xmit lock.
In message <[email protected]> you write:
Quoted 7 lines · Rusty Russell
Quoted 2 lines
Hmm, I thought the sched stuff all runs under the network brlock? If so, it doesn't need to be held in, since it's not preemptible.
The packet schedulers transmit, not receive. They have their own queue locking, along with the device xmit lock.
Then the patch to mark the "owner" should be straightforward. I haven't looked at kronos's patch in detail, though.
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
Il Sat, Jan 04, 2003 at 05:09:41PM +1100, Rusty Russell ha scritto:
Quoted 1 line
Then the patch to mark the "owner" should be straightforward.
Yes, it seems so :)
This is the patch:
net/sched/cls_api.c +9 -0
@@ -216,6 +216,13 @@kfree (tp); goto errout; } + +if (try_module_get (tp_ops->owner) ==0 ) { + err = -ENOSYS ; +kfree (tp); + goto errout; + } +memset (tp,0 ,sizeof (*tp)); tp->ops = tp_ops; tp->protocol = protocol;
@@ -225,6 +232,7 @@ tp->classid = parent; err = tp_ops->init (tp);if (err) { +module_put (tp_ops->owner);kfree (tp); goto errout; }
@@ -248,6 +256,7 @@write_unlock (&qdisc_tree_lock); tp->ops->destroy (tp); +module_put (tp->ops->owner);kfree (tp); err =0 ; goto errout;
net/sched/cls_fw.c +15 -15
@@ -117,7 +117,6 @@static int fw_init (struct tcf_proto *tp) { -MOD_INC_USE_COUNT ;return 0 ; }
@@ -128,7 +127,6 @@int h;if (head ==NULL ) { -MOD_DEC_USE_COUNT ;return ; }
@@ -146,7 +144,6 @@ } }kfree (head); -MOD_DEC_USE_COUNT ; }static int fw_delete (struct tcf_proto *tp,unsigned long arg)
@@ -351,18 +348,21 @@ }struct tcf_proto_ops cls_fw_ops = { -NULL , -"fw" , - fw_classify, - fw_init, - fw_destroy, - - fw_get, - fw_put, - fw_change, - fw_delete, - fw_walk, - fw_dump + .next =NULL , + .kind ="fw" , + .classify = fw_classify, + .init = fw_init, + .destroy = fw_destroy, + + .get = fw_get, + .put = fw_put, + .change = fw_change, + .delete = fw_delete, + .walk = fw_walk, + + .dump = fw_dump, + + .owner =THIS_MODULE , };#ifdef MODULE
net/sched/cls_route.c +15 -15
@@ -272,7 +272,6 @@static int route4_init (struct tcf_proto *tp) { -MOD_INC_USE_COUNT ;return 0 ; }
@@ -282,7 +281,6 @@int h1, h2;if (head ==NULL ) { -MOD_DEC_USE_COUNT ;return ; }
@@ -309,7 +307,6 @@ } }kfree (head); -MOD_DEC_USE_COUNT ; }static int route4_delete (struct tcf_proto *tp,unsigned long arg)
@@ -607,18 +604,21 @@ }struct tcf_proto_ops cls_route4_ops = { -NULL , -"route" , - route4_classify, - route4_init, - route4_destroy, - - route4_get, - route4_put, - route4_change, - route4_delete, - route4_walk, - route4_dump + .next =NULL , + .kind ="route" , + .classify = route4_classify, + .init = route4_init, + .destroy = route4_destroy, + + .get = route4_get, + .put = route4_put, + .change = route4_change, + .delete = route4_delete, + .walk = route4_walk, + + .dump = route4_dump, + + .owner =THIS_MODULE , };#ifdef MODULE
net/sched/cls_rsvp.h +15 -15
@@ -247,14 +247,12 @@ {struct rsvp_head *data; -MOD_INC_USE_COUNT ; data =kmalloc (sizeof (struct rsvp_head ),GFP_KERNEL );if (data) {memset (data,0 ,sizeof (struct rsvp_head )); tp->root = data;return 0 ; } -MOD_DEC_USE_COUNT ;return -ENOBUFS ; }
@@ -294,7 +292,6 @@ } }kfree (data); -MOD_DEC_USE_COUNT ; }static int rsvp_delete (struct tcf_proto *tp,unsigned long arg)
@@ -673,18 +670,21 @@ }struct tcf_proto_ops RSVP_OPS = { -NULL , -RSVP_ID , - rsvp_classify, - rsvp_init, - rsvp_destroy, - - rsvp_get, - rsvp_put, - rsvp_change, - rsvp_delete, - rsvp_walk, - rsvp_dump + .next =NULL , + .kind =RSVP_ID , + .classify = rsvp_classify, + .init = rsvp_init, + .destroy = rsvp_destroy, + + .get = rsvp_get, + .put = rsvp_put, + .change = rsvp_change, + .delete = rsvp_delete, + .walk = rsvp_walk, + + .dump = rsvp_dump, + + .owner =THIS_MODULE , };#ifdef MODULE
net/sched/cls_tcindex.c +15 -16
@@ -144,10 +144,8 @@struct tcindex_data *p;DPRINTK ("tcindex_init(tp %p)\n" ,tp); -MOD_INC_USE_COUNT ; p =kmalloc (sizeof (struct tcindex_data ),GFP_KERNEL );if (!p) { -MOD_DEC_USE_COUNT ;return -ENOMEM ; } tp->root = p;
@@ -417,7 +415,6 @@kfree (p->h);kfree (p); tp->root =NULL ; -MOD_DEC_USE_COUNT ; }
@@ -480,20 +477,22 @@ }struct tcf_proto_ops cls_tcindex_ops = { -NULL , -"tcindex" , - tcindex_classify, - tcindex_init, - tcindex_destroy, - - tcindex_get, - tcindex_put, - tcindex_change, - tcindex_delete, - tcindex_walk, - tcindex_dump -}; + .next =NULL , + .kind ="tcindex" , + .classify = tcindex_classify, + .init = tcindex_init, + .destroy = tcindex_destroy, + + .get = tcindex_get, + .put = tcindex_put, + .change = tcindex_change, + .delete = tcindex_delete, + .walk = tcindex_walk, + + .dump = tcindex_dump, + .owner =THIS_MODULE , +};#ifdef MODULE int init_module (void )
net/sched/cls_u32.c +15 -17
@@ -262,15 +262,12 @@struct tc_u_hnode *root_ht;struct tc_u_common *tp_c; -MOD_INC_USE_COUNT ; -for (tp_c = u32_list; tp_c; tp_c = tp_c->next)if (tp_c->q == tp->q)break ; root_ht =kmalloc (sizeof (*root_ht),GFP_KERNEL );if (root_ht ==NULL ) { -MOD_DEC_USE_COUNT ;return -ENOBUFS ; }memset (root_ht,0 ,sizeof (*root_ht));
@@ -282,7 +279,6 @@ tp_c =kmalloc (sizeof (*tp_c),GFP_KERNEL );if (tp_c ==NULL ) {kfree (root_ht); -MOD_DEC_USE_COUNT ;return -ENOBUFS ; }memset (tp_c,0 ,sizeof (*tp_c));
@@ -407,7 +403,6 @@kfree (tp_c); } -MOD_DEC_USE_COUNT ; tp->data =NULL ; }
@@ -695,18 +690,21 @@ }struct tcf_proto_ops cls_u32_ops = { -NULL , -"u32" , - u32_classify, - u32_init, - u32_destroy, - - u32_get, - u32_put, - u32_change, - u32_delete, - u32_walk, - u32_dump + .next =NULL , + .kind ="u32" , + .classify = u32_classify, + .init = u32_init, + .destroy = u32_destroy, + + .get = u32_get, + .put = u32_put, + .change = u32_change, + .delete = u32_delete, + .walk = u32_walk, + + .dump = u32_dump, + + .owner =THIS_MODULE , };#ifdef MODULE
net/sched/sch_api.c +6 -6
@@ -409,6 +409,10 @@if (ops ==NULL ) goto err_out; + err = -ENOSYS ; +if (try_module_get (ops->owner) ==0 ) + goto err_module; + size =sizeof (*sch) + ops->priv_size; sch =kmalloc (size,GFP_KERNEL );
@@ -416,12 +420,6 @@if (!sch) goto err_out; -/* Grrr... Resolve race condition with module unload */ - - err = -EINVAL ; -if (ops !=qdisc_lookup_ops (kind)) - goto err_out; -memset (sch,0 , size);skb_queue_head_init (&sch->q);
@@ -460,6 +458,8 @@ } err_out: +module_put (ops->owner); +err_module: *errp = err;if (sch)kfree (sch);
net/sched/sch_atm.c +5 -5
@@ -575,7 +575,6 @@ p->link.ref =1 ; p->link.next =NULL ;tasklet_init (&p->task,sch_atm_dequeue,(unsigned long ) sch); -MOD_INC_USE_COUNT ;return 0 ; }
@@ -612,7 +611,6 @@ } }tasklet_kill (&p->task); -MOD_DEC_USE_COUNT ; }
@@ -697,9 +695,10 @@ .destroy = atm_tc_destroy, .change = NULL, - .dump = atm_tc_dump -}; + .dump = atm_tc_dump, + .owner = THIS_MODULE, +};#ifdef MODULE int init_module (void )
@@ -707,9 +706,10 @@ return register_qdisc(&atm_qdisc_ops); } -void cleanup_module (void ) {unregister_qdisc (&atm_qdisc_ops); }#endif + +MODULE_LICENSE ("GPL" );
net/sched/sch_cbq.c +2 -3
@@ -1411,9 +1411,7 @@ r =RTA_DATA (tb[TCA_CBQ_RATE -1 ]); -MOD_INC_USE_COUNT ;if ((q->link.R_tab =qdisc_get_rtab (r, tb[TCA_CBQ_RTAB -1 ])) ==NULL ) { -MOD_DEC_USE_COUNT ;return -EINVAL ; }
@@ -1749,7 +1747,6 @@ }qdisc_put_rtab (q->link.R_tab); -MOD_DEC_USE_COUNT ; }static void cbq_put (struct Qdisc *sch,unsigned long arg)
@@ -2099,6 +2096,8 @@ .change = NULL, .dump = cbq_dump, + + .owner = THIS_MODULE, };#ifdef MODULE
net/sched/sch_csz.c +3 -3
@@ -749,7 +749,7 @@static void csz_destroy (struct Qdisc * sch) { -MOD_DEC_USE_COUNT ; +/* nop */ }static int csz_init (struct Qdisc *sch,struct rtattr *opt)
@@ -791,7 +791,6 @@ q->wd_timer.data = (unsigned long )sch; q->wd_timer.function = csz_watchdog;#endif -MOD_INC_USE_COUNT ;return 0 ; }
@@ -1046,8 +1045,9 @@ .change = NULL, .dump = csz_dump, -}; + .owner = THIS_MODULE, +};#ifdef MODULE int init_module (void )
net/sched/sch_dsmark.c +3 -4
@@ -352,7 +352,6 @@if (!(p->q =qdisc_create_dflt (sch->dev, &pfifo_qdisc_ops))) p->q = &noop_qdisc;DPRINTK ("dsmark_init: qdisc %p\n" ,&p->q); -MOD_INC_USE_COUNT ;return 0 ; }
@@ -381,7 +380,6 @@qdisc_destroy (p->q); p->q = &noop_qdisc;kfree (p->mask); -MOD_DEC_USE_COUNT ; }
@@ -466,7 +464,9 @@ .destroy = dsmark_destroy, .change = NULL, - .dump = dsmark_dump + .dump = dsmark_dump, + + .owner = THIS_MODULE, };#ifdef MODULE
@@ -474,7 +474,6 @@ {return register_qdisc (&dsmark_qdisc_ops); } -void cleanup_module (void ) {
net/sched/sch_fifo.c +2 -0
@@ -206,4 +206,6 @@ .change = fifo_init, .dump = fifo_dump, + + .owner = THIS_MODULE, };
net/sched/sch_generic.c +6 -0
@@ -29,6 +29,7 @@#include <linux/skbuff.h> #include <linux/rtnetlink.h> #include <linux/init.h> +#include <linux/module.h> #include <net/sock.h> #include <net/pkt_sched.h>
@@ -253,6 +254,8 @@ .enqueue = noop_enqueue, .dequeue = noop_dequeue, .requeue = noop_requeue, + + .owner = THIS_MODULE, }; struct Qdisc noqueue_qdisc =
@@ -356,6 +359,8 @@ .init = pfifo_fast_init, .reset = pfifo_fast_reset, + + .owner = THIS_MODULE, };struct Qdisc *qdisc_create_dflt (struct net_device *dev,struct Qdisc_ops *ops)
@@ -422,6 +427,7 @@ ops->reset (qdisc);if (ops->destroy) ops->destroy (qdisc); +module_put (ops->owner);if (!(qdisc->flags&TCQ_F_BUILTIN ))kfree (qdisc); }
net/sched/sch_gred.c +2 -4
@@ -348,7 +348,6 @@ table->grio=sopt->grio; table->initd=0 ;/* probably need to clear all the table DP entries as well */ -MOD_INC_USE_COUNT ;return 0 ; }
@@ -490,7 +489,6 @@ table->def=sopt->def_DP; table->grio=sopt->grio; table->initd=0 ; -MOD_INC_USE_COUNT ;return 0 ; }
@@ -602,7 +600,6 @@if (table->tab[i])kfree (table->tab[i]); } -MOD_DEC_USE_COUNT ; } struct Qdisc_ops gred_qdisc_ops =
@@ -623,8 +620,9 @@ .change = gred_change, .dump = gred_dump, + + .owner = THIS_MODULE, }; -#ifdef MODULE int init_module (void )
net/sched/sch_htb.c +29 -29
82 lines not on this page · raw message
net/sched/sch_ingress.c +4 -9
@@ -257,7 +257,6 @@memset (p,0 ,sizeof (*p)); p->filter_list =NULL ; p->q = &noop_qdisc; -MOD_INC_USE_COUNT ;return 0 ; error:return -EINVAL ;
@@ -304,8 +303,6 @@qdisc_destroy (p->q);#endif -MOD_DEC_USE_COUNT ; - }
@@ -359,22 +356,20 @@ .change = NULL, .dump = ingress_dump, -}; + .owner = THIS_MODULE, +};#ifdef MODULE int init_module (void ) {int ret =0 ; -if ((ret =register_qdisc (&ingress_qdisc_ops)) <0 ) { -printk ("Unable to register Ingress qdisc\n" ); -return ret; - } +if ((ret =register_qdisc (&ingress_qdisc_ops)) <0 ) +printk (KERN_ERR "Unable to register Ingress qdisc\n" );return ret; } -void cleanup_module (void ) {
net/sched/sch_prio.c +2 -4
@@ -163,7 +163,6 @@qdisc_destroy (q->queues[prio]); q->queues[prio] = &noop_qdisc; } -MOD_DEC_USE_COUNT ; }static int prio_tune (struct Qdisc *sch,struct rtattr *opt)
@@ -227,7 +226,6 @@if ((err=prio_tune (sch, opt)) !=0 )return err; } -MOD_INC_USE_COUNT ;return 0 ; }
@@ -399,10 +397,11 @@ .change = prio_tune, .dump = prio_dump, + + .owner = THIS_MODULE, };#ifdef MODULE -int init_module (void ) {return register_qdisc (&prio_qdisc_ops);
@@ -412,6 +411,5 @@ {unregister_qdisc (&prio_qdisc_ops); } -#endif MODULE_LICENSE ("GPL" );
net/sched/sch_red.c +4 -10
@@ -407,14 +407,7 @@static int red_init (struct Qdisc * sch,struct rtattr *opt) { -int err; - -MOD_INC_USE_COUNT ; - -if ((err =red_change (sch, opt)) !=0 ) { -MOD_DEC_USE_COUNT ; - } -return err; +return red_change (sch, opt); }
@@ -458,7 +451,7 @@static void red_destroy (struct Qdisc *sch) { -MOD_DEC_USE_COUNT ; +/* nop */ } struct Qdisc_ops red_qdisc_ops =
@@ -479,8 +472,9 @@ .change = red_change, .dump = red_dump, -}; + .owner = THIS_MODULE, +};#ifdef MODULE int init_module (void )
net/sched/sch_sfq.c +2 -2
@@ -435,7 +435,6 @@ }for (i=0 ; i<SFQ_DEPTH ; i++)sfq_link (q, i); -MOD_INC_USE_COUNT ;return 0 ; }
@@ -443,7 +442,6 @@ {struct sfq_sched_data *q = (struct sfq_sched_data *)sch->data;del_timer (&q->perturb_timer); -MOD_DEC_USE_COUNT ; }static int sfq_dump (struct Qdisc *sch,struct sk_buff *skb)
@@ -486,6 +484,8 @@ .change = NULL, .dump = sfq_dump, + + .owner = THIS_MODULE, };#ifdef MODULE
net/sched/sch_tbf.c +3 -10
@@ -330,23 +330,17 @@static int tbf_init (struct Qdisc * sch,struct rtattr *opt) { -int err;struct tbf_sched_data *q = (struct tbf_sched_data *)sch->data;if (opt ==NULL )return -EINVAL ; -MOD_INC_USE_COUNT ; -PSCHED_GET_TIME (q->t_c);init_timer (&q->wd_timer); q->wd_timer.function = tbf_watchdog; q->wd_timer.data = (unsigned long )sch; -if ((err =tbf_change (sch, opt)) !=0 ) { -MOD_DEC_USE_COUNT ; - } -return err; +return tbf_change (sch, opt); }static void tbf_destroy (struct Qdisc *sch)
@@ -359,8 +353,6 @@qdisc_put_rtab (q->P_tab);if (q->R_tab)qdisc_put_rtab (q->R_tab); - -MOD_DEC_USE_COUNT ; }static int tbf_dump (struct Qdisc *sch,struct sk_buff *skb)
@@ -409,8 +401,9 @@ .change = tbf_change, .dump = tbf_dump, -}; + .owner = THIS_MODULE, +};#ifdef MODULE int init_module (void )
net/sched/sch_teql.c +2 -3
@@ -178,7 +178,6 @@ }while ((prev = q) != master->slaves); } -MOD_DEC_USE_COUNT ; }static int teql_qdisc_init (struct Qdisc *sch,struct rtattr *opt)
@@ -223,7 +222,6 @@ m->dev.flags = (m->dev.flags&~FMASK )|(dev->flags&FMASK ); } -MOD_INC_USE_COUNT ;return 0 ; }
@@ -454,8 +452,9 @@ .reset = teql_reset, .destroy = teql_destroy, .dump = NULL, -},}; + .owner = THIS_MODULE, +},};#ifdef MODULE int init_module (void )
include/net/pkt_cls.h +3 -0
@@ -3,6 +3,7 @@#include <linux/pkt_cls.h> +#include <linux/module.h> struct rtattr ;struct tcmsg ;
@@ -56,6 +57,8 @@/* rtnetlink specific */ int (*dump)(struct tcf_proto *,unsigned long ,struct sk_buff *skb,struct tcmsg *); + +struct module *owner; };/* Main classifier routine: scans classifier chain attached
include/net/pkt_sched.h +3 -0
@@ -10,6 +10,7 @@#include <linux/config.h> #include <linux/types.h> #include <linux/pkt_sched.h> +#include <linux/module.h> #include <net/pkt_cls.h> #ifdef CONFIG_X86_TSC
@@ -67,6 +68,8 @@int (*change)(struct Qdisc *,struct rtattr *arg);int (*dump)(struct Qdisc *,struct sk_buff *); +/* Protects callbacks */ +struct module *owner; };extern rwlock_t qdisc_tree_lock;
I've tested it a bit and seems to work ok. TEQL queue still
uses MOD_{INC,DEC}_USE_COUNT because it register a device andregister_netdevice doesn't use the new interface yet.
ciao,
Luca
Signature · 5 lines
Reply-To: [email protected] Home: http://kronoz.cjb.net The trouble with computers is that they do what you tell them, not what you want. D. Cohen
Hello!
Quoted 4 lines
Hmm, I thought the sched stuff all runs under the network brlock? If so, it doesn't need to be held in, since it's not preemptible.
I haven't checked, though...
It runs under semaphore.
Which does not matter at all, because the hole
void cleanup_module(void)
{
<an instance is cloned here>
unregister_qdisc(&cbq_qdisc_ops); }
remained in any case, be it under some preemptive, nonprepemtive lock or not under a lock at all.
BTW, Rusty, a question... I do not understand, what is purpose of this "new" module stuff at all? If we still need to query something in module.c to create each instanse of something it smells exactly "old" broken approach. I just do not see differences. It is not essential in this particular case, but it would be funny to ask it each time when creating a tcp socket.
Alexey
Quoted 2 lines · At 01:32 AM 1/14/2003 +0300, [email protected] wrote:
It is not essential in this particular case,
but it would be funny to ask it each time when creating a tcp socket.
That's what we're going to do btw. I'm cooking up new patch. See "New module refcounting for net_proto_family" thread.
Max
Il Tue, Jan 14, 2003 at 01:32:56AM +0300, Alexey N. Kuznetsov ha scritto:
Quoted 10 lines
Which does not matter at all, because the hole
void
cleanup_module(void)
{
<an instance is cloned here>unregister_qdisc(&cbq_qdisc_ops); }remained in any case, be it under some preemptive, nonprepemtive lock or not under a lock at all.
Hmm, this can't happen now. The try_module_get() will fail if the module is going away and so qdisc_create won't create a new qdisc.
Quoted 2 lines
BTW, Rusty, a question... I do not understand, what is purpose of this "new" module stuff at all?
The idea is to use try_module_get() before using any interface
registered by the module. Every module has a state associated to it: if the module has been loaded but it is still in its init section its state will be MODULE_STATE_COMING (and try_module_get() will fail); if the module is in process of being unloaded its state will be MODULE_STATE_LIVE (and try_module_get() will fail); otherwise the state will be MODULE_STATE_LIVE (and try_module_get() will success).
If we use try_module_get() before using the module and module_put() when we are done everything will work:
int create_foo(foo **new) {
if (try_module_get(foo->owner) == 0)
/* Module is being unloaded, it's not safe to use it */
return -ENOSYS;
*new = foo->new_foo();
}void destroy_foo(foo *foo) {
foo->finalize();
module_put(foo->owner);
}This is what I've done for the packet scheduler: try_module_get() is called *before* cloning the qdisc and if it fails the qdisc won't be cloned. module_put() is called in qdisc_destroy(), after ->destroy(), when we are sure that nobody will ever use a reference to the qdisc.
Luca
Signature · 4 lines
Reply-To: [email protected] Home: http://kronoz.cjb.net Non capisco tutta questa eccitazione per il Multitasking: io sono anni che leggo in bagno.
Hi,
Quoted 7 lines · Kronos
The idea is to use
try_module_get()before using any interface
registered by the module. Every module has a state associated to it: if the module has been loaded but it is still in its init section its state will beMODULE_STATE_COMING(andtry_module_get()will fail); if the module is in process of being unloaded its state will beMODULE_STATE_LIVE(andtry_module_get()will fail); otherwise the state will beMODULE_STATE_LIVE(andtry_module_get()will success).
Rusty had to revert this to the old scheme, as e.g. scsi relied on it. This means the kernel will oops, if someone gets a reference to a module and its init function fails.
Above scheme is not without problems either, a failing try_module_get() suddenly gets a double meaning. Now it also means that the module might be ready soon, please try again later, what increases the complexity of the whole module business.
BTW what makes it currently even more complex is that there is also no synchronization by the module-init-tools done anymore, a try_module_get()/request_module()/try_module_get() sequence does give no definitive answer anymore, whether a module is available or not. The results will be spurious kmod failures.
Quoted 10 lines
If we use
try_module_get()before using the module andmodule_put()when we are done everything will work:int create_foo(foo **new) { if (try_module_get(foo->owner) == 0) /* Module is being unloaded, it's not safe to use it */ return -ENOSYS; *new = foo->new_foo(); }
It's not that simple, try_module_get() does not everything. It's the responsibility of the caller to protect the owner pointer, so try_module_get() itself is only usable within some locking. This means your patch is still not safe, as the test is done too late.
bye, Roman
Hello!
Quoted 3 lines
Above scheme is not without problems either, a failing
try_module_get()suddenly gets a double meaning. Now it also means that the module might be ready soon, please try again later,
... and btw completely useless thing, because each module user already has its own means to do serialization of this kind even when used _NOT_ as module.
Somewhat overdone.
Alexey
Quoted 1 line · [email protected] wrote:
Somewhat overdone.
I think it would be nice to introduce in 2.7 a shutdowncall (*) function class for modules that works like exitcall, but with the following differences:
- does not return before the module has really de-registered itself everywhere, including synchronization with any callbacks, etc.
- has a return code, and can fail if it would have to sleep for a possibly long time
Before calling the shutdown function, all symbols exported by the module are hidden, and after the shutdown functions returns, the module can be unloaded.
That way, the module reference count becomes merely advisory information, and the real locking, synchronization, etc. is inside the module.
The module unload mechanism could then check whether the module is old (uses exitcall) or new (uses shutdowncall), and act accordingly. Modules and the services they use could then be gradually fixed.
(*) Or maybe use a nicer name :-)
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
In message <[email protected]> you write:
Quoted 13 lines
Quoted 1 line · [email protected] wrote:
Somewhat overdone.
I think it would be nice to introduce in 2.7 a shutdowncall (*) function class for modules that works like exitcall, but with the following differences:
- does not return before the module has really de-registered itself everywhere, including synchronization with any callbacks, etc.
- has a return code, and can fail if it would have to sleep for a possibly long time
Quoted 3 lines
Before calling the shutdown function, all symbols exported by the module are hidden, and after the shutdown functions returns, the module can be unloaded.
This already happens. This is why all accesses to the module are protected by try_module_get().
I've analyzed dozens of "here's my implementation idea" mails over the last two years. Here's the executive summary:
1) It's simply not a good idea to force 1600 modules to change, no matter what timescale. And changing it in a way that is *more*, not *less* complex is even worse.
2) It's bad enough to force the interfaces to change: at least the primitive they are to use is one many of them are already using, and is very simple to understand.
Rusty.
PS. The *implementation* flaw in your scheme: someone starts using a
module as you try to deregister it. Either you re-register the module (ie. you can never unload security modules), or you leave it half unloaded (even worse).
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
Quoted 2 lines · Rusty Russell
1) It's simply not a good idea to force 1600 modules to change, no matter what timescale.
That's the part that I don't believe. The kernel interfaces aren't static. Locking rules have changed several times, the wait/sleep interface has changed, the concept of a module owner has been added, various other interfaces have changed.
So a module will eventually have to be maintained. And the more important a module, the more timely this will be done. (Well, or so everybody wishes :-) Unmaintained modules will eventually just fail to work or even compile for other reasons, at which point they cease to be a problem as far as the loading/unloading mechanism is concerned.
I agree that it's a bad idea to force changes. That's why there should be a reasonably long period where the current hack and a clean mechanism are available in parallel.
Quoted 2 lines
And changing it in a way that is *more*,
not *less* complex is even worse.
The implementation may be more complex, but the change I'm suggesting would greatly simplify the rules: no endless set of voodoo rites, but one simple rule: "the shutdowncall function either does nothing, and returns failure, or it returns success, and completely de-registers everything it has previously registered".
Quoted 2 lines
PS. The *implementation* flaw in your scheme: someone starts using a
module as you try to deregister it.
How could they ? The symbols are hidden, so that way is blocked. If another module has registered callbacks using symbols obtained from that module, that other module needs to be unloaded first anyway, so these references are gone too.
If the static kernel accesses a module by resolving symbols (except for the well-controlled operations of the module loader itself), yes, then that module would become un-unloadable. I'm not sure this is much of an issue.
If a callback comes in at the wrong moment, the module can choose to de-register and wait until the subsystem has finished any callbacks, or detect that it's about to shut itself down, and fail the callback. The point is that all the locking is now under control of the module, and not scattered all over kernel+module(s).
Quoted 1 line
(ie. you can never unload security modules),
Hmm, what makes security modules (what exactly do you mean by that ?) special ? In any case, a module that's truly unloadable would simply return failure from its shutdowncall.
Quoted 1 line
or you leave it half unloaded (even worse).
There's always the choice of just sleeping through any inconvenient callbacks. In some cases, this is perfectly acceptable, because the callback won't keep the module busy for a long time anyway (interrupts, timers, tasklets, etc.). In other cases (e.g. "open" functions), a callback could keep it busy forever.
In that case, the module needs to maintain its own usage count and have some flag that indicates that it's shutting down. So the callback would look like this:
static int foo_open(...)
{
atomic_inc(&busy);
if (shutting_down) {
atomic_dec(&busy);
return -EGONEFISHING;
}
...
}static int foo_shutdown(...)
{
shutting_down = 1;
if (atomic_read(&busy)) {
shutting_down = 0;
return -EBUSY;
}
deregister_whatever(&myself);
/* deregister and synchronize */
...
}"busy" could of course just be the module use count.
There is the race condition that the module could briefly disappear (i.e. "foo_open" fails), and then come back, because it turned out to be or appear to be busy. But I think, considering that we're actually trying to make it go away, this is acceptable behaviour. You have the race conditions "is busy" vs. "can unload" and "can use" vs. "has been unloaded" anyway.
This can be greatly simplified if we have a try_deregister_whatever that just returns an error if it would have to sleep, i.e. if the synchronization is pushed into the service.
Of course, if the "whatever" subsystem will make callbacks after returning from deregister_whatever, this doesn't work, and the module has to use the old mechanisms. But that's really a bug in the "whatever" subsystem.
The problem I see with the current module interface is that it just tries to hack the current mess into a less buggy state, but doesn't provide much of an incentive for actually cleaning up the interfaces.
Avoiding the bugs is good, but we should also work towards a clean long-term solution.
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
In message <[email protected]> you write:
Quoted 8 lines
Quoted 2 lines · Rusty Russell
1) It's simply not a good idea to force 1600 modules to change, no matter what timescale.
That's the part that I don't believe. The kernel interfaces aren't static. Locking rules have changed several times, the wait/sleep interface has changed, the concept of a module owner has been added, various other interfaces have changed.
Deprecating every module, and rewriting their initialization routines is ambitious beyond the scale of anything you have mentioned. Not that 90% of the kernel code couldn't use a damn good spring cleaning, but I'm not prepared to make such a change personally.
And remember why we're doing it: for a fairly obscure race condition.
Quoted 9 lines
Quoted 2 lines
And changing it in a way that is *more*,
not *less* complex is even worse.The implementation may be more complex, but the change I'm suggesting would greatly simplify the rules: no endless set of voodoo rites, but one simple rule: "the shutdowncall function either does nothing, and returns failure, or it returns success, and completely de-registers everything it has previously registered".
So we go from:
int init(void)
{
if (!register_foo(&foo))
return -err;
if (!register_bar(&bar)) {
unregister_foo(&foo);
return -err;
}
return 0;
}void fini(void)
{
unregister_foo(&foo);
unregister_bar(&bar);
}to:
int fini(void)
{
if (!unregister_foo(&foo))
return -err;
if (!unregister_bar(&bar)) {
if (!register_foo(&foo))
????
return -err;
}
return 0;
}Quoted 9 lines
Quoted 2 lines
PS. The *implementation* flaw in your scheme: someone starts using a
module as you try to deregister it.If a callback comes in at the wrong moment, the module can choose to de-register and wait until the subsystem has finished any callbacks, or detect that it's about to shut itself down, and fail the callback. The point is that all the locking is now under control of the module, and not scattered all over kernel+module(s).
Something like this?
static int i_am_live;
static spinlock_t my_lock = SPIN_LOCK_UNLOCKED;
/* This is our registered function. */
static int foo_function(void *somedata)
{
int live;
spin_lock(&my_lock);
live = i_am_live;
spin_unlock(&my_lock);
if (!live)
return -EIGNOREME???;
...
}int fini(void)
{
spin_lock(&my_lock);
i_am_live = 0;
spin_unlock(&my_lock);
if (!unregister_foo(&foo)) {
spin_lock(&my_lock);
i_am_live = 1;
spin_unlock(&my_lock);
return -err;
}
if (!unregister_bar(&bar)) {
if (!register_foo(&foo))
????
spin_lock(&my_lock);
i_am_live = 1;
spin_unlock(&my_lock);
return -err;
}
return 0;
}Now, if someone tries to remove a module, but it's busy, you get a window of spurious failure, even though the module isn't actually removed. Secondly, there is often no way of returning a value which says "I'm going away, act as if I'm not here": only the level above can sanely know what it would do if this were not found.
Quoted 6 lines
Quoted 1 line
(ie. you can never unload security modules),
Hmm, what makes security modules (what exactly do you mean by that ?) special ? In any case, a module that's truly unloadable would simply return failure from its shutdowncall.
On a busy system, they're never not being used. Your unload routine would always fail. Same with netfilter modules.
Quoted 8 lines
Quoted 1 line
or you leave it half unloaded (even worse).
There's always the choice of just sleeping through any inconvenient callbacks. In some cases, this is perfectly acceptable, because the callback won't keep the module busy for a long time anyway (interrupts, timers, tasklets, etc.). In other cases (e.g. "open" functions), a callback could keep it busy forever.
Exactly. It's kept there forever, while the module is useless.
Quoted 7 lines
The problem I see with the current module interface is that it just tries to hack the current mess into a less buggy state, but doesn't provide much of an incentive for actually cleaning up the interfaces.
Avoiding the bugs is good, but we should also work towards a clean long-term solution.
The current scheme is clean: it's two-stage delete with a nice helper function "try_module_get()" which tells you when it's going away, and no requirement that modules actually implement two-stage delete themselves. The patch to mirror this in two-stage init was posted yesterday, as well.
It also puts the (minimal) burden in the right place: in the interface coder's lap, not the interface user's lap.
Unfortunately, I don't have the patience to explain this once for every kernel developer.
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
Quoted 2 lines · Rusty Russell
Deprecating every module, and rewriting their initialization routines is ambitious beyond the scale of anything you have mentioned.
Well, it has happened before, e.g. sleep_on is now deprecated, cli() doesn't give the comprehensive protection it used to, holding spinlocks while sleeping used to be frowned upon, but it's only recently that it moved to being forbidden, etc.
And the people making the new rules didn't go and fix every module in existence. You may get this kind of responsibility only if you actually break the old interface, but that's not what I'm suggesting to do.
Quoted 1 line
And remember why we're doing it: for a fairly obscure race condition.
No, I want to do this to fix the reason for the fix for the obscure race condition :-)
Also, all this smells of a fundamental design problem: modules aren't the only things that can become unavailable. So why construct a special mechanism that only applies to modules ?
Quoted 1 line
So we go from:
Yes. Note that the problem case only occurs if you have more than one registration that could block you for a long time. Otherwise, you just take that one first, and sleep on all the others.
Otherwise, things get messy, I admit. But that's not a new problem either, e.g.
Quoted 10 lines
int init(void)
{ if (!register_foo(&foo)) return -err; if (!register_bar(&bar)) { unregister_foo(&foo); return -err; } return 0; }
What if unregister_foo fails, because somebody already started to use the foo interface ? If "block until it works" is a valid answer, you could probably use this also for the unload case.
This would actually work even with the current interface: just make the module exit function sleep until all references are gone. This means than rmmod might take a looong time, but is this really a problem ? If a module wishes to accelerate its demise, it can always add errors exits. If the interface doesn't provide error exits, this is again a general problem, and actually one of the interface.
If there's a really nasty case, where you absolutely can't afford to sleep, you need to change the service to split "deregister" into:
- prepare_deregister (like "deregister", but reversible)
- commit_deregister
- undo_deregister
But there probably aren't many cases where you'd really need this.
Quoted 1 line
Something like this?
Yes, for instance. Or use atomic operations.
Quoted 3 lines
Now, if someone tries to remove a module, but it's busy, you get a window of spurious failure, even though the module isn't actually removed.
Correct. But does this matter ? After all, we were prepared to get failures if the removal succeeds anyway. So this is not a new race condition.
Quoted 3 lines
Secondly, there is often no way of returning a value which says "I'm going away, act as if I'm not here": only the level above can sanely know what it would do if this were not found.
Okay, my approach usually puts the responsibility of finally giving up in user space. If user space just ignores the error, and keeps the thing open anyway, you're in for a long wait. But that's not different from opening some device and never releasing it. In either case, your module becomes un-unloadable, unless you kill the user-space hog. Again, not a new problem.
Quoted 2 lines
On a busy system, they're never not being used. Your unload routine would always fail. Same with netfilter modules.
But they're only active for a short time, so the deregistration could just sleep.
Quoted 2 lines
It also puts the (minimal) burden in the right place: in the interface coder's lap, not the interface user's lap.
Well, both need to cooperate. And I don't see why the interface coder should have to know whether its users are modules or not, while the interface user, who is implicitly very aware if he's a module or not, shouldn't.
Also note that this isn't just a module problem. Instead of tripping over code that all of a sudden isn't there, one may well trip over a data structure that has just been removed. In either case, you need to get the synchronization right. What I'm proposing is simply to make those two cases more similar, such that handling one case correctly also handles the other one.
Example:
struct foo *my_data;
int foo_callback()
{
do_something(my_data->blah);
...
}void foo_exit()
{
deregister(...);
kfree(my_data);
...
}So if "deregister" allows callbacks after it returns, you'll still end up getting your oops every once in a while.
Now make this a non-module, and if foo_exit or some equivalent can be called for some other reason (e.g. power-down, hotplugging, etc.), you a) still have the same problem, and b) all the miracle protection of try_module_get has vanished. So you have to do the synchronization twice, i.e. on both sides of the interface.
Quoted 2 lines
Unfortunately, I don't have the patience to explain this once for every kernel developer.
Sorry for being so persistent. But I really think the module situation is rapidly approaching the point where just fixing the next bug isn't good enough, but where we need to get back to the drawing board, look at the larger picture, and then work towards a cleaner solution.
Also, although the task may seem daunting, don't forget that even the BLK wasn't removed in a day :-)
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
In message <[email protected]> you write:
Quoted 4 lines
Quoted 1 line
And remember why we're doing it: for a fairly obscure race condition.
No, I want to do this to fix the reason for the fix for the obscure race condition :-)
Semantics.
Quoted 3 lines
Also, all this smells of a fundamental design problem: modules aren't the only things that can become unavailable. So why construct a special mechanism that only applies to modules ?
NO NO NO. Listen *carefully*.
The ONLY time that FUNCTIONS vanish is when MODULES get UNLOADED (or fail to LOAD).
So you're suggesting we should lock ALL functions the way we lock all other datastructures. I look forward to your compiler patch.
I've explained this multiple times. If you're not convinced, fine.
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
Quoted 2 lines · Rusty Russell
The ONLY time that FUNCTIONS vanish is when MODULES get UNLOADED (or fail to LOAD).
I totally agree with Rusty. If you don't understand this fundamental difference between module unloading vs. arbitrary kernel objects going away, then you really need to apply some gray matter to it before you continue in this conversation :)
Quoted 3 lines · David S. Miller
I totally agree with Rusty. If you don't understand this fundamental difference between module unloading vs. arbitrary kernel objects going away,
I think I understand that part. What I'm saying is that any interface that will still call you after deregistration will also cause problems with normal data accesses, even if no modules are involved.
So, if interfaces with this kind of bug are fixed, all of a sudden the second (1) synchronization mechanism for module unloads - returning from the cleanup function (2) - becomes a viable alternative to try_module_get.
(1) The first mechanism being the use count.
(2) Or, in the case of initialization failure, returning from the init function.
And that's what I think we should build upon. Rusty doesn't see things this way, and I'd like to find out where exactly we disagree.
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Hi,
Quoted 8 lines · David S. Miller
Quoted 2 lines · Rusty Russell
The ONLY time that FUNCTIONS vanish is when MODULES get UNLOADED (or fail to LOAD).
I totally agree with Rusty. If you don't understand this fundamental difference between module unloading vs. arbitrary kernel objects going away, then you really need to apply some gray matter to it before you continue in this conversation :)
Above is not really not wrong, but there is still no fundamental difference to other kernel objects. We will get this wrong as long as we see functions as some kind of very special case, this is simply not true.
Module functions are nothing else than readonly data structures, which are allocated and initialized in module.c, by calling init_module() the ownership is transferred to the module. How the ownership is transferred back is now (hopefully) the point of the discussion. The only thing special here is that the data possibly needs to be flushed out of the data cache, before it can be used as a function. Besides of this there is not a single technical difference to other reference counted data structures, just the implementations differ.
Functions are not the only thing that vanish when modules get unloaded. It's a lot more common to register functions via data structures and it would be very foolish to think that only the functions need protecting. In the first place one has to get access to the data structure, only then the functions can be safely used. Again, how these data structures are allocated doesn't matter, but at any time we have to know, who owns these data structures, so that we can safely remove and deallocate them.
bye, Roman
Quoted 2 lines · Rusty Russell
The ONLY time that FUNCTIONS vanish is when MODULES get UNLOADED (or fail to LOAD).
Yes, of course. That's not what I mean. (See below.)
Quoted 2 lines
So you're suggesting we should lock ALL functions the way we lock all other datastructures. I look forward to your compiler patch.
No, it's not locking for concurrent access, but locking for access vs. removal.
If a function pointer gets disseminated without any way for the caller to revoke it, then try_module_get is indeed the only possible solution. But I think this is the rare exception.
The usual way for disseminating function pointers seems to be through registration at some service interface, or through a callback. Do you agree with this ?
Service interfaces normally also have a deregistration function, and callbacks are either synchronous, so you have to wait for them to complete, but you're busy anyway, or asynchronous, usually with a way to cancel them.
Now, if the service insists on calling back even after deregistration or cancellation, with no means for making sure that at some point no further callback will happen, I'd consider this a bug of the service - a bug that should be fixed. (If there's a way to deregister with and without synchronization, that's fine, of course.) Do you agree with this ?
If you agree so far, then I hope you'll also agree that a module can always be safely unloaded, without try_module_get, if its cleanup function just deregisters/cancels and synchronizes all the service registrations/callbacks the module made. Plain and simple.
Of course, this means that module can spend an unbounded amount of time in its cleanup function.
Do you agree so far ?
Now, if we accept that there may be the requirement that we can't always wait for deregistration/cancellation to complete, this leads to the more complex scenario we've discussed earlier in this thread.
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Hi,
Quoted 7 lines · Werner Almesberger
If there's a really nasty case, where you absolutely can't afford to sleep, you need to change the service to split "deregister" into:
-
prepare_deregister(like "deregister", but reversible)
-commit_deregister
-undo_deregister
You can simplify this. All you need are the following simple functions:
- void register();
- void unregister();
- int is_registered();
- void inc_usecount();
- void dec_usecount();
- int get_usecount();
It's important to understand that the registered state and the usecount are completely independent. As soon as the object is unregistered and the usecount is zero, the object can be freed, but it doesn't matter in which order it happens.
The problem is now that we are very limited how we can use these functions. We can only unregister an object after the usecount became zero, although it's also possible to first unregister the object and then wait for the usecount. Only when we can do the latter is it possible to safely force the removal of the object.
bye, Roman
Quoted 4 lines · Roman Zippel
Quoted 3 lines
-
prepare_deregister(like "deregister", but reversible)
-commit_deregister
-undo_deregister
Quoted 8 lines
You can simplify this. All you need are the following simple functions:
- void
register();
- voidunregister();
- intis_registered();
- voidinc_usecount();
- voiddec_usecount();
- intget_usecount();
I'm not sure if you, you're not changing the semantics. What I was describing was a non-blocking interface, e.g.
if (!prepare_deregister(foo))
return -E...;
if (!prepare_deregister(bar)) {
undo_deregister(foo);
return -E...;
}
commit_deregister(foo);
commit_deregister(bar);
return 0;With your interface, you're not guaranteed that you can re-register. Well, you could externalize this, e.g. with
int error = 0;
inc_usecount(foo);
inc_usecount(bar);
unregister(foo); /* does nothing irrevokable since use count > 0 */
unregister(bar);
if (get_usecount(foo) > 1 || get_usecount(bar) > 1) {
register(foo); /* re-registers foo */
register(bar); /* re-registers bar */
error = E...;
}
dec_usecount(foo);
dec_usecount(bar);
return error;Is that what you had in mind ?
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Hi,
Quoted 10 lines · Werner Almesberger
Quoted 8 lines
You can simplify this. All you need are the following simple functions:
- void
register();
- voidunregister();
- intis_registered();
- voidinc_usecount();
- voiddec_usecount();
- intget_usecount();I'm not sure if you, you're not changing the semantics.
See above functions as primitives, which one can use to build any other resource management you want.
The module code does the get_usecount() test for every driver and if it was zero it disables inc_usecount() and only then the driver is allowed to unregister(). The module code has to add another is_active state for this and is so actually adding more complexity to it.
Quoted 12 lines
What I was
describing was a non-blocking interface, e.g.if (!prepare_deregister(foo)) return -E...; if (!prepare_deregister(bar)) { undo_deregister(foo); return -E...; } commit_deregister(foo); commit_deregister(bar); return 0;
You are making it too complex, as you probably need an is_active state as well, it would be just per interface instead of global. If you really want to reduce complexity, all you can do, is to get rid of the is_active state. For the majority of drivers such state isn't needed at all, but it's currently forced on all drivers.
So to keep things simple, we should just finish shutting down the driver, as soon as we started with it and don't restart or undo anything. To avoid the sleeping we can also simply return -EBUSY and continue later, so the cleanup could look like this:
if (!ptr) return 0; if (is_registered()) unregister(); if (get_usecount()) return -EBUSY; release(); return 0;
This way the caller just needs to do the following, which can be called as often as necessary:
if (shutdown(ptr)) return -EBUSY; ptr = NULL;
There is really no need to introduce extra states to make things more complex.
bye, Roman
Quoted 13 lines · Roman Zippel
Hi,
Quoted 9 lines · Werner Almesberger
Quoted 8 lines
You can simplify this. All you need are the following simple functions:
- void
register();
- voidunregister();
- intis_registered();
- voidinc_usecount();
- voiddec_usecount();
- intget_usecount();
Quoted 4 lines
Quoted 1 line
I'm not sure if you, you're not changing the semantics.
See above functions as primitives, which one can use to build any other resource management you want.
That same functionality is already present in a kobj for you to use, instead of rolling your own resource management code. :)
(sorry, I couldn't help myself...)
greg k-h
In message <[email protected]> you write:
Quoted 6 lines
Quoted 2 lines · Rusty Russell
Deprecating every module, and rewriting their initialization routines is ambitious beyond the scale of anything you have mentioned.
Well, it has happened before, e.g.
sleep_onis now deprecated,cli()doesn't give the comprehensive protection it used to,
They gave us SMP. What do we gain for your change?
Quoted 2 lines
holding spinlocks while sleeping used to be frowned upon, but it's only recently that it moved to being forbidden, etc.
No, it's always been forbidden, it's only recently we detect it.
But apologies for the tone of my previous mail: it seems I'm oversensitive to criticism of the module stuff now 8(
To go someway towards an explanation, at least, I humbly submit a fairly complete description of the approach used (assuming the module init race fix patch gets merged).
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
================
Rusty's Unreliable Guide To The 2.5 Module Locking Implementation (draft) aka. All Locking and No Play Make Rusty Go Crazy
Any object in the kernel which might disappear, needs to have some form of locking; we're very familiar with locking data structures in the kernel. Modules are no exception, except that in this case it's the functions themselves need locking.
The main difference is that we are not prepared to pay the price for locking every function we call: the vast majority of speed-critical functions are not in modules at all, and for many others, the call is internal to the same module. So any locking solution must be lightweight, and (or course) vanish if CONFIG_MODULES is not set.
Now, we don't need to lock unless we are going across a module boundary (presumbably if you are already in the module, the locking is sorted out already), and it's easy to ensure that you don't need to lock for a normal function call, since the module loader can make the caller statically depend on the callee: if module B calls "foo", which is in module A, the loader knows that B refers to "foo" when it loads B, so simply enforces that B uses A, and A cannot be unloaded before B.
The cases which remain, however, are function calls through pointers:
such strategy functions are used widely, and naturally need some form of protection against going away. Since the advent of SMP, and preemption, it is extremely difficult for such functions to protect themselves.
Two Stage Delete
================
So let's apply standard locking techniques. A standard approach to this for data (eg networking packets) involves two-stage delete: we keep a reference count in the object, which is usually 1 + number of users, and to delete it we mark it deleted so noone new can use it, and drop the reference count. The last user, who drops the reference count to zero, actually does the free.
This is called two-stage delete: Alexey Kuznetsov's reason why IPv4 was not a module was the lack of two-stage delete support for modules.
The usual and logical place for the deleted flag and reference count is in the structure being protected: in this case, that's the module itself. You can, instead, have a flag and/or reference count in every object used by the module, but it's not quite the same thing, and deactivating them all atomically is impossible, which introduces its own set of problems.
The main problem advantage of a single flag which says whether the module is active or not, is that every module would need to add a function which deactivated it. There are 1600 modules in the kernel, and they work fine when built into the kernel: such a change merely for the module case seems overly disruptive. Also, there is the other case to consider: modules can disappear because they failed to initialize. The same "deleted" flag can be used to isolate modules during initialization: otherwise each module would need to implement two-stage initialization as well.
Finally, the try_module_get() primitive (which combines the deleted flag test with the reference count increment in one convenient form) was already fairly widely used by the filesystem code and others, where it was called "try_mod_inc_use_count()". Most people seemed to have little problem using the primitive correctly.
Corner Cases And Optimizations
==============================
Some interfaces need to do more than simply flip a flag on activation:
they might want to fire off a hotplug event, for example, or scan partition tables. So a notifier chain is supplied for them to do exactly this (in the "Proposed Module Init Race Fix" patch). Modules are free to "roll their own" two-stage init using module_make_live() if they want, too.
One of the critical things when designing this, was that getting references to modules had to be fast, and it didn't matter if removing modules was relatively slow. The logical implementation of try_module_get() looks like:
static int try_module_get(struct module *mod)
{
int ret;
unsigned long flags;
read_lock_irqsave(&module_lock, flags);
if (mod->deleted)
ret = 0;
else {
ret = 1;
atomic_inc(&mod->use);
}
read_unlock_irqrestore(&module_lock, flags);
return ret;
}
static int module_put(struct module *mod)
{
if (atomic_dec_and_test(&mod->use))
if (mod->deleted)
wake_up(mod->whoever_is_waiting);
}And module unloading would look like:
...
write_lock_irq(&module_lock);
if (atomic_read(&mod->use) != 0)
ret = -EBUSY;
else {
mod->deleted = 1;
ret = 0;
}
write_unlock_irq(&module_lock);But we can do better than this. The try_module_get contains interrupt saving code (even though it's often unneccessary), a spin lock and an atomic operation. Worse, on SMP the spinlock and atomic variable will have to bounce from one CPU to another. The answer is to use an atomic counter per CPU, and a "bogolock", which looks (conceptually) like this:
void bogo_read_lock(void)
{
preempt_disable();
}
void bogo_read_unlock(void)
{
preempt_enable();
}
void bogo_write_lock(void)
{
run thread on every other CPU
tell them all to stop interrupts
stop interrupts locally
}
void bogo_write_unlock(void)
{
tell threads to unblock interrupts and exit
restore interrupts locally
}So the real try_module_get and module_put look like:
static inline int try_module_get(struct module *module)
{
int ret = 1;
if (module) {
unsigned int cpu = get_cpu(); /* Disables preemption */
if (likely(module_is_live(module)))
local_inc(&module->ref[cpu].count);
else
ret = 0;
put_cpu();
}
return ret;
}
static inline void module_put(struct module *module)
{
if (module) {
unsigned int cpu = get_cpu();
local_dec(&module->ref[cpu].count);
/* Maybe they're waiting for us to drop reference? */
if (unlikely(!module_is_live(module)))
wake_up_process(module->waiter);
put_cpu();
}
}And the remove code looks like:
ret = stop_refcounts(); /* bogo_write_lock */
if (ret != 0)
goto out;
/* If it's not unused, quit unless we are told to block. */
if ((flags & O_NONBLOCK) && module_refcount(mod) != 0) {
forced = try_force(flags);
if (!forced)
ret = -EWOULDBLOCK;
} else {
mod->waiter = current;
mod->state = MODULE_STATE_GOING;
}
restart_refcounts(); /* bogo)write_unlock */Cheers,
Rusty.
Hi,
Quoted 9 lines · Rusty Russell
Two Stage Delete
================So let's apply standard locking techniques. A standard approach to this for data (eg networking packets) involves two-stage delete: we keep a reference count in the object, which is usually 1 + number of users, and to delete it we mark it deleted so noone new can use it, and drop the reference count. The last user, who drops the reference count to zero, actually does the free.
Rusty, could you please show me where you found the deleted flag in network packets? Where did you find that it's required to test the deleted flag before you can get a reference to the object? Actually it's far more common to just delete the object when all references are gone without any need for a deleted flag. Why does everyone else get away with a "single stage delete"? Where is the deleted flag (or active flag) for modules coming from?
(BTW I would be really happy to get any response to this, I already explained this all before, so I'd really like to know whether someone understands what I'm talking about or what is so difficult to understand.)
As already mentioned every object can only be deleted, when it's nowhere registered anymore and all users are gone, the same is true for modules:
if (!is_registered(obj) && !get_usecount(obj)) delete(obj);
The is_registered() check can be omitted, if we unregister the object ourselves, but this requires some locking to prevent new users until the object is unregistered:
lock();
if (get_usecount(obj)) {
unlock();
return -EBUSY;
}
unregister(obj);
unlock();
delete(obj);New users have to do this:
lock(); obj = lookup(); if (obj) inc_usecount(obj); unlock();
This is the standard mechanism we can find all over the kernel. The problem with modules now is that the usecount check and the unregister happen at two different places and the lock is not held until the object is unregistered, so we have to add a new flag:
mod_lock();
if (get_usecount(obj)) {
mod_unlock();
return -EBUSY;
}
deactivate(obj);
mod_unlock();
...
obj_lock();
unregister(obj);
obj_unlock();
delete(obj);New users have to do this now:
obj_lock();
obj = lookup();
if (obj) {
mod_lock();
if (is_active(obj))
inc_usecount(obj);
mod_unlock();
}
obj_unlock();Rusty now worked very hard to make the read path as cheap as possible, but the general complexity is still the same as always. The only way to make this even cheaper is to reduce the complexity and this is only possible by getting rid of the is_active() check. The only consequence would be that we had no global activate switch anymore, but is it really needed? (Insert compelling reason here, why such a switch is absolutely required, as I don't know any.)
To understand the possible alternatives I have to rename Rustys "Two Stage Delete" into "Three stage delete":
1. deactivate
2. unregister
3. delete
(In case anyone was wondering where the single reference in Rustys example is coming from - it's the registered state from the second stage).
As said above the delete stage can only be entered if the object is not registered and not used anymore, this is the only absolute requirement, but currently we can't even get past the first stage as long as there are users (otherwise we risk a deadlock).
When Rusty is now talking about exposing a two stage api to the modules, he actually means the first stage in order to leave it to the module how to deactivate the module. It makes indeed little sense to expose this stage, because this is the stage, which is causing the extra complexity and which we should rather get rid of.
On the other hand it would make quite a lot more sense to expose the second stage to the modules. This stage is independent of the usecount, this means we can easily and safely prevent new users from using the module. The standard mechanism above to remove an object can easily be changed into:
unregister(force):
lock();
if (get_usecount(obj) && !force) {
unlock();
return -EBUSY;
}
unregister(obj);
unlock();
delete:
if (get_usecount(obj))
return -EBUSY;
delete(obj);Doing these stages with one or two functions doesn't really matter, the work has to be done anyway and causes no real extra complexity.
bye, Roman
m.au on Fri, Jan 17, 2003 at 01:27:56PM +1100
Sorry for the long pause ...
Quoted 1 line · Rusty Russell
They gave us SMP. What do we gain for your change?
Mainly a simpler interface - one that doesn't treat module unloading as such a special case, but just as yet another fairly regular synchronization problem.
This should also have a performance impact: the current approach puts "code locking" outside of the module, and "data locking" inside of it. Unifying this eliminates one layer of locking mechanisms.
Independent of this, we should fix the interfaces that give us unstoppable callbacks. These are just disasters waiting to happen, modules or no.
Of course, since this may imply interface changes (not necessarily in terms of changing an existing interface, but perhaps in terms of adding a properly synchronized version, and discovering bugs in modules using the not-synchronized one), it would be good if the module cleanup simplification could be done in parallel.
I think, once we know exactly what semantics to aim for, the change could be relatively straightforward. (And, I wholeheartedly agree, there must be no "flag day".)
Quoted 2 lines
But apologies for the tone of my previous mail: it seems I'm oversensitive to criticism of the module stuff now 8(
No problem. I actually admire your thick skin, given all the unjustified and nasty stuff you get thrown at you :-)
Quoted 3 lines
To go someway towards an explanation, at least, I humbly submit a fairly complete description of the approach used (assuming the module init race fix patch gets merged).
Thanks ! That part looks fine. But, of course, it's not how you do it that I don't like, but what you're doing :-)
Anyway, my plan is to first get my simulation infrastructure working, and then make a few test cases that show callbacks after deregistration causing trouble. After that, hopefully other people can pick up the cleanup work.
Do you see any obvious technical problems with the approach of using return from module initialization/cleanup as "ready to unload" indicator ?
- Werner
--
_________________________________________________________________________
/ Werner Almesberger, Buenos Aires, Argentina [email protected] /
/_http://www.almesberger.net/____________________________________________/
In message <[email protected]> you write:
Quoted 7 lines
m.au on Fri, Jan 17, 2003 at 01:27:56PM +1100
Sorry for the long pause ...
Quoted 1 line · Rusty Russell
They gave us SMP. What do we gain for your change?
Quoted 7 lines
Mainly a simpler interface - one that doesn't treat module unloading as such a special case, but just as yet another fairly regular synchronization problem.
This should also have a performance impact: the current approach puts "code locking" outside of the module, and "data locking" inside of it. Unifying this eliminates one layer of locking mechanisms.
Um, no. You're special case "optimizing" it.
When you have an object which may vanish, the linux kernel idiom runs something like this:
write_lock()
find available object in list
inc object refcount
write_unlock()
The writelock protects the infrastructure, the refcount protects the object. Deleting is done in two stages (mark deleted and drop refcount, then whoever drops the refcount to 0 actually does the free). Usually "mark deleted" means simply remove from the list, but not always.
The current module code uses exactly the same idiom for the code
itself: we use a heavily read-optimized lock, but that's an implementation detail.
Now, could you instead lock all the (arbitrary, widespread, unrelated) accesses to the object, instead of the object itself? Sure. It'd be unique in the kernel, and involve changing the way every interface works, and probably expose some of the complexity to every module author (every workable implementation I've seen has this problem), but you could do it. See below for why I don't think it's worth it.
Quoted 3 lines
Independent of this, we should fix the interfaces that give us unstoppable callbacks. These are just disasters waiting to happen, modules or no.
I assume you're referring to the many places where we assume that the structure being added was not dynamically allocated, so don't bother to protect against its deletion?
That is, of course, orthogonal.
And in general, I agree: not including a refcount is asking for trouble. But these reference counts are *not* free. The module reference counts, by comparison, can be made extremely cheap (see implementation), because we can allocate a cacheline per cpu, and we can bias "read" speed in preference of awful "write" speed.
In summary: the most obvious implementation is to lock to module as an object separately from any objects it might create. The current implementation is extremely fast, requires neither module changes nor (many) interface changes, and in effect canonicalizes a single existing method of locking, which coders seem quite comfortable with.
Given these reasons, you can see why I no longer discuss new implementation ideas with people 8(
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
Quoted 4 lines · Rusty Russell
Um, no. You're special case "optimizing" it.
When you have an object which may vanish, the linux kernel idiom runs something like this:
Yes, that's when you view it as a locking problem, as for data objects. What I'm saying is that, if you manage the data structures properly, plus fix a few interfaces that currently don't have to manage data structures properly, you're already perfectly synchronized, so no further locking is needed.
Quoted 3 lines
I assume you're referring to the many places where we assume that the structure being added was not dynamically allocated, so don't bother to protect against its deletion?
Yes.
Quoted 2 lines
And in general, I agree: not including a refcount is asking for trouble. But these reference counts are *not* free.
Alas, no. But we how long can we afford not to fix them, at least the "public" interfaces ? Even if they're unsafe, people will use them, particularly if they're given no other choice.
Quoted 4 lines
object separately from any objects it might create. The current implementation is extremely fast, requires neither module changes nor (many) interface changes, and in effect canonicalizes a single existing method of locking, which coders seem quite comfortable with.
It's more the changes behind the interfaces I'm worried about. It may not be bad today, but every function pointer is a potential problem.
Anyway, without fixing a good number of the "ghost from the past" interfaces first, my point is moot. So I won't trouble you again with module locking before there is some progress in this area :-)
Quoted 2 lines
Given these reasons, you can see why I no longer discuss new implementation ideas with people 8(
Nah, don't give up ! :-)
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Hi,
Quoted 14 lines · Rusty Russell
When you have an object which may vanish, the linux kernel idiom runs something like this:
write_lock()
find available object in list
inc object refcount
write_unlock()The writelock protects the infrastructure, the refcount protects the object. Deleting is done in two stages (mark deleted and drop refcount, then whoever drops the refcount to 0 actually does the free). Usually "mark deleted" means simply remove from the list, but not always.
Quoted 3 lines
The current module code uses exactly the same idiom for the code
itself: we use a heavily read-optimized lock, but that's an implementation detail.
It's not the same, please see:
http://marc.theaimsgroup.com/?l=linux-kernel&m=104284223130775&w=2
I explained why the current module locking is more complex and why it's actually a three stage delete.
Let's take an example: procfs entries. That's a more interesting example, because these can be dynamically created, but they also include pointers to the module.
To keep it simple we create/remove an entry from a nonmodular kernel (so module functions are dummies):
foo_entry = create_proc_entry(); foo_entry->data = kmalloc(); foo_entry->read_proc = foo_read; foo_entry->write_proc = foo_write;
Problem 1: If the user opens that new proc entry, he must be prepared that the new entry is not fully initialized yet.
Problem 2: There are no memory barriers, that means it's undefined in which order the data, read_proc, write_proc members arrive at another cpu, so it's possible that foo_read/foo_write can be called with a NULL data pointer.
Now let's remove it again:
data = foo_entry->data; remove_proc_entry(); kfree(data);
Problem: Should the entry still be busy, procfs just prints a warning, delays cleanup and returns immediately, so the data can be accessed after it was freed.
Now we do the same from a module but not from module_init/module_exit, that means we dynamically want to create/remove entries during the module life time. When we create the entry we also initialize the owner member, but this doesn't help with any of the above problems. The only thing module locking changes is that we know that it's safe to free the data at the time module_exit is called, but this might happen in the distant future or even never.
Rusty, above are real problems, the module locking fixes these problems during module_init/module_exit, but how can these problems fixed in the other cases and how does the module locking help?
bye, Roman
In message <Pine.LNX.4.44.0302141035270.1336-100000@serv> you write:
Quoted 4 lines
It's not the same, please see:
http://marc.theaimsgroup.com/?l=linux-kernel&m=104284223130775&w=2
I explained why the current module locking is more complex and why it's actually a three stage delete.
No, here is where you show *your* ignorance of kernel locking idioms, and that your axiom is that "the new system is more complex".
I suggest you read the kernel locking guide: it's in the kernel sources in Documentation/DocBook/kernel-locking.*, try "make psdocs".
Quoted 3 lines
Rusty, above are real problems, the module locking fixes these problems during module_init/module_exit, but how can these problems fixed in the other cases and how does the module locking help?
This isn't even a sensible question: "This is not a module problem. How does module locking help?"
You're wasting your own valuable time, too.
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
Hi,
Quoted 7 lines · Rusty Russell
Quoted 4 lines
It's not the same, please see:
http://marc.theaimsgroup.com/?l=linux-kernel&m=104284223130775&w=2
I explained why the current module locking is more complex and why it's actually a three stage delete.No, here is where you show *your* ignorance of kernel locking idioms, and that your axiom is that "the new system is more complex".
Well, it should be simple then to point out the problem, would you _please_ do it?
Quoted 8 lines
Quoted 3 lines
Rusty, above are real problems, the module locking fixes these problems during module_init/module_exit, but how can these problems fixed in the other cases and how does the module locking help?
This isn't even a sensible question: "This is not a module problem. How does module locking help?"
You're wasting your own valuable time, too.
I hoped we could discuss locking problems, as I said these problems are real, so it should be worth to describe possible solutions. Then we can compare the different locking mechanism and maybe we find one, which not only solves a part of the problem.
Rusty, I'm asking all these questions on purpose, I want to help you to understand the complete problem and how limited your solutions are. So either please answer my questions or point out the mistakes in my argumentation.
bye, Roman
In message <Pine.LNX.4.44.0302141334560.1336-100000@serv> you write:
Quoted 4 lines
Rusty, I'm asking all these questions on purpose, I want to help you to understand the complete problem and how limited your solutions are. So either please answer my questions or point out the mistakes in my argumentation.
There were two major changes in the module code. The first was to simplify the userland interface, from:
sys_create_module(name, size) sys_query_module(...) (a 5-way multiplexed syscall) sys_init_module(name, ptr)
to:
sys_init_module(ptr, len, userargs)
To argue against this change is a demonstration of lack of understanding, or a complete lack of taste.
The second change was the speed up one system of module locking in the kernel which wasn't racy, and deprecate the other system which was racy in 99% of its uses. That is all.
Did it solve all the races in the kernel? Of course not. But it's simple to use, already well understood in the kernel, and avoids massive changes. It also allows connection tracking to be properly modularized, which was my long-lost original purpose.
I've repeated this enough now, I think.
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
Hi,
Quoted 7 lines · Rusty Russell
There were two major changes in the module code. The first was to simplify the userland interface, from:
sys_create_module(name, size)sys_query_module(...)(a 5-way multiplexed syscall)sys_init_module(name, ptr)
Quoted 5 lines
to:
sys_init_module(ptr, len, userargs)To argue against this change is a demonstration of lack of understanding, or a complete lack of taste.
Maybe you could share a bit of your wisdom?
1. Doing the linking in userspace requires two steps, but I still don't know what's so bad about it.
2. This still doesn't explain, why everything has to be moved into kernel, why can't we move more into userspace?
3. You simply moved part of the query syscall functionality to /proc/modules (which btw is still not enough to fix ksymoops).
Quoted 3 lines
The second change was the speed up one system of module locking in the kernel which wasn't racy, and deprecate the other system which was racy in 99% of its uses. That is all.
Well, I'm not against optimizing the module locking (*), as we won't get rid of it in the near feature, but it still has problems.
1. It's adding complexity (however you implement it), I explained it in detail and you still haven't told me, where I'm wrong.
2. The module interface is incompatible with other kernel interfaces, I tried to explain that in the mail from saturday, if you think I'm wrong, your input is very welcome, but _please_ answer to that mail.
Quoted 4 lines
Did it solve all the races in the kernel? Of course not. But it's simple to use, already well understood in the kernel, and avoids massive changes. It also allows connection tracking to be properly modularized, which was my long-lost original purpose.
It's too much fun to quote Al here:
"And no, I don't buy arguments about poor interface-writers. You do some infrastructure intended to be used by driver-writers - you are supposed to have a clue. 'Cause having rabid monkeys on crack on *both* sides of an interface is a recipe for disaster and on the driver side you are guaranteed to have a bunch of them.
We need to have interfaces cleaned up. No silver bullets there. There's maybe a dozen of interfaces that cover 99% of all drivers out there. Remaining 1% can and should fend for itself - you do something really tricky, you are responsible for getting it right."
Quoted 1 line
I've repeated this enough now, I think.
Yes, you repeated your "executive summaries" now often enough, maybe you can impress some manager with it, but it's highly offensive to kernel hackers. Could you _please_ come up with some arguments now?
bye, Roman
(*) BTW I have patch that would make the unload path usable again, it would only be required to add a single smp_mb() to the fast path.
In message <Pine.LNX.4.44.0302171112390.1336-100000@serv> you write:
Quoted 7 lines
Maybe you could share a bit of your wisdom?
1. Doing the linking in userspace requires two steps, but I still don't know what's so bad about it.
2. This still doesn't explain, why everything has to be moved into kernel, why can't we move more into userspace?
3. You simply moved part of the query syscall functionality to/proc/modules(which btw is still not enough to fix ksymoops).
I think you'd do far better to implement it yourself for half a dozen architectures. It's not my job to teach you things which can be gained by reading the code and thinking a little.
We're going in a circle again.
Quoted 9 lines
Quoted 3 lines
The second change was the speed up one system of module locking in the kernel which wasn't racy, and deprecate the other system which was racy in 99% of its uses. That is all.
Well, I'm not against optimizing the module locking (*), as we won't get rid of it in the near feature, but it still has problems.
1. It's adding complexity (however you implement it), I explained it in detail and you still haven't told me, where I'm wrong.
No, it's exactly the same as before. You can't see that, and I've given up explaining it.
Quoted 3 lines
2. The module interface is incompatible with other kernel interfaces, I tried to explain that in the mail from saturday, if you think I'm wrong, your input is very welcome, but _please_ answer to that mail.
This problem is in your mind Roman.
Deal with it.
Quoted 6 lines
Quoted 4 lines
Did it solve all the races in the kernel? Of course not. But it's simple to use, already well understood in the kernel, and avoids massive changes. It also allows connection tracking to be properly modularized, which was my long-lost original purpose.
It's too much fun to quote Al here:
Quoting Al's rant isn't an argument. It wasn't very coherent when he wrote it, and it doesn't gain with repetition.
The code exists. It's simple to use.
I give up. You're killfiled again 8(
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
Hi,
Quoted 11 lines · Rusty Russell
Quoted 7 lines
Maybe you could share a bit of your wisdom?
1. Doing the linking in userspace requires two steps, but I still don't know what's so bad about it.
2. This still doesn't explain, why everything has to be moved into kernel, why can't we move more into userspace?
3. You simply moved part of the query syscall functionality to/proc/modules(which btw is still not enough to fix ksymoops).I think you'd do far better to implement it yourself for half a dozen architectures. It's not my job to teach you things which can be gained by reading the code and thinking a little.
As usual you explain nothing, so I still don't know why a complete rewrite was necessary. The old implementation did work fine within limits and already has support for all architectures, so why should I just throw it away? Why was it not possible to first fix the problems of the old system?
Quoted 8 lines
Quoted 5 lines
Well, I'm not against optimizing the module locking (*), as we won't get rid of it in the near feature, but it still has problems.
1. It's adding complexity (however you implement it), I explained it in detail and you still haven't told me, where I'm wrong.
No, it's exactly the same as before. You can't see that, and I've given up explaining it.
So far you explained nothing and if you would just read and try to understand that damned mail(*), you would know, that I already said that the complexity is "exactly the same as before". I'm comparing it to other solutions, which you obviously haven't understood.
(*) http://marc.theaimsgroup.com/?l=linux-kernel&m=104284223130775&w=2
Quoted 5 lines
Quoted 3 lines
2. The module interface is incompatible with other kernel interfaces, I tried to explain that in the mail from saturday, if you think I'm wrong, your input is very welcome, but _please_ answer to that mail.
This problem is in your mind Roman.
Thanks for another detailed explaination. :(
Quoted 4 lines
Quoted 1 line
It's too much fun to quote Al here:
Quoting Al's rant isn't an argument. It wasn't very coherent when he wrote it, and it doesn't gain with repetition.
Well, if you don't even try to understand, what Al is trying to tell you, I'm afraid I can't help you either.
Quoted 3 lines
The code exists. It's simple to use.
I give up. You're killfiled again 8(
I seriously consider to take over modules maintainership, but I have neither the energy nor the time to do this alone, so I can only wish everyone much fun with modules during 2.6.
bye, Roman
Hi,
Quoted 7 lines · Rusty Russell
Quoted 4 lines
It's not the same, please see:
http://marc.theaimsgroup.com/?l=linux-kernel&m=104284223130775&w=2
I explained why the current module locking is more complex and why it's actually a three stage delete.No, here is where you show *your* ignorance of kernel locking idioms, and that your axiom is that "the new system is more complex".
Another point you probably misunderstood: I said that the complexity of the new and the old system is exactly the same, please read more carefully before flaming, it might backfire.
Quoted 6 lines
Quoted 3 lines
Rusty, above are real problems, the module locking fixes these problems during module_init/module_exit, but how can these problems fixed in the other cases and how does the module locking help?
This isn't even a sensible question: "This is not a module problem. How does module locking help?"
I hate to drag people into a discussion, but maybe you're more inclined to believe Al:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103761331525509&w=2
http://marc.theaimsgroup.com/?l=linux-kernel&m=103769023500378&w=2
Please read this very careful and think about it.
bye, Roman
Quoted 3 lines · Roman Zippel
Quoted 2 lines · Rusty Russell
This isn't even a sensible question: "This is not a module problem. How does module locking help?"
I have to side with Rusty here - it's really not a module problem.
The module changes only make this a little worse, because they follow the philosophy that this can't be fixed, so try_module_get and friends try to implement the right kind of locking for unload races (but for little else) without tripping over those "unfixable" bugs.
The good news is that you can start fixing all those bugs right now, and even without Rusty's consent :-)
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Hi,
Quoted 5 lines · Werner Almesberger
The module changes only make this a little worse, because they follow the philosophy that this can't be fixed, so
try_module_getand friends try to implement the right kind of locking for unload races (but for little else) without tripping over those "unfixable" bugs.
If you see these bugs as "unfixable", you already gave up and you end up putting one band-aid over another, each only solving part of the problem. Please try work with me here and we might find a more general solution. I could just post possible solutions, but as long nobody understands the problem, they will be rejected anyway.
bye, Roman
Quoted 2 lines · Roman Zippel
If you see these bugs as "unfixable", you already gave up and you end up putting one band-aid over another, each only solving part of the problem.
Yup, that's what I don't like either.
Quoted 3 lines
Please try work with me here and we might find a more general solution. I could just post possible solutions, but as long nobody understands the problem, they will be rejected anyway.
Step one is to fix those "unfixable" problems. That's independent of modules, and I'm convinced that it needs to be done.
I don't want to have to go through lots of subsystems and try to figure our how to do their locking right, so my plan is to use a simulator to exercise such obscure race conditions, and once the Oops has been shown live and in color, leave the actual fixes to the maintainers of the respective code. (And eventually, also leave the bug hunt to others. I'm lousy at maintenance-type of work.)
I expect to have that simulator ready for useful work on UP in about one week. SMP will come later. (For a sneak preview, you can look at umlsim.sourceforge.net, which has the (old) kernel side, and at netbb/dumpbb/README.UMLSIM in netbb from www.almesberger.net/netbb, which describes some of the user-space side (basically a script-driven debugger).)
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Hi,
Quoted 6 lines · Werner Almesberger
Quoted 3 lines
Please try work with me here and we might find a more general solution. I could just post possible solutions, but as long nobody understands the problem, they will be rejected anyway.
Step one is to fix those "unfixable" problems. That's independent of modules, and I'm convinced that it needs to be done.
Step one is still to understand the problem. I described a very real problem, once this problem is solved (which can be done in different ways), I can explain how this can be applied to modules. It's not really independent of modules.
bye, Roman
Quoted 3 lines · Roman Zippel
Step one is still to understand the problem. I described a very real problem, once this problem is solved (which can be done in different ways),
I though we were talking about
static data_used_by_callbacks; ... register_foo(&stuff_with_callbacks); ... unregister_foo(&stuff_with_callbacks); make_unusable(&data_used_by_callbacks) ... /* oops, we just got a callback */
("data_used_by_callbacks" could be a pointer to kmalloc'ed memory, etc.)
This kind of problem seems to be understood well enough. But maybe you mean something else ?
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Hi,
Quoted 11 lines · Werner Almesberger
I though we were talking about
static data_used_by_callbacks; ... register_foo(&stuff_with_callbacks); ... unregister_foo(&stuff_with_callbacks); make_unusable(&data_used_by_callbacks) ... /* oops, we just got a callback */
Quoted 4 lines
("
data_used_by_callbacks" could be a pointer to kmalloc'ed memory, etc.)This kind of problem seems to be understood well enough.
Yes, and now compare how the solutions differ when the data is static and when it's allocated.
bye, Roman
Quoted 2 lines · Roman Zippel
Yes, and now compare how the solutions differ when the data is static and when it's allocated.
Do they ? Even if the data is static, it can become invalid (in the sense that accessing it from a callback would lead to some kind of undesirable behaviour, even though the access itself would work), so I don't quite see why the difference would matter.
Example:
static ... common_callback(...)
{
switch (my_state) {
...
}
}
...
my_state = A;
register_fancy_timer_A(&me_A,common_callback);
...
unregister_fancy_timer_A(&me_A);
my_state = B;
/* stray fancy_timer_A call to common_callback would
trigger action for state B */
...
register_fancy_timer_B(&me_B,common_callback);
...Depending on "my_state", the callback would perform different actions. (The "fancy timers" would be some timer-like service that doesn't del_timer_sync.)
This is getting to abstract. Why don't you just say where you see the difference ? :-)
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Hi,
Quoted 8 lines · Werner Almesberger
Quoted 2 lines
Yes, and now compare how the solutions differ when the data is static and when it's allocated.
Do they ? Even if the data is static, it can become invalid (in the sense that accessing it from a callback would lead to some kind of undesirable behaviour, even though the access itself would work), so I don't quite see why the difference would matter.
Let's stay at the main problem, we have find out when it's safe to delete an object. For dynamic objects you have the following options:
- callbacks: when the refcount becomes zero, we call a function to remove the object.
- failure: we just return -EBUSY and try again later.
- wait: we simply wait until the refcount becomes zero
Static objects and functions are freed by the module code and usually we want to unregister them at module unload time, so there are basically two ways:
- we use the module count via try_module_get/module_put
- we use your own refcount and must wait in module_exit until all users are gone
If we exclude the possibly-wait-forever-option, do you see the problem for dynamic objects which also contain references to static data/ functions? Procfs entries are such objects, there is a count field for the dynamic part and an owner field for the static part and proc_get_inode always has to get two references. The interesting question is now, can we get rid of one of them?
If the answer is no, it would mean we need two procfs APIs, one which can be used from module_exit and another which can be used to remove dynamic entries. OTOH if we want to avoid this it would mean we have to make one or more of the above options generically usable.
I stop here, because before we can discuss possible solutions, we have to agree, that this problem is real.
Rusty, did I make any mistake so far? I'd really like to have your opinion too. You are free to flame me, if I made any mistake, but please do it with a bit more substance than just accusing me of ignorance.
bye, Roman
Quoted 2 lines · Roman Zippel
Let's stay at the main problem, we have find out when it's safe to delete an object. For dynamic objects you have the following options:
[...]
Quoted 1 line
Static objects and functions are freed by the module code and usually we
[...]
Okay so far.
Quoted 3 lines
If we exclude the possibly-wait-forever-option, do you see the problem for dynamic objects which also contain references to static data/ functions?
You mean that two locking mechanisms are used, where one of them shouldn't be doing all that much ? Well, yes.
Now, is this a problem, or just a symptom ? I'd say it's a symptom:
we already have a perfectly good locking/synchronization method, and that's through the register/unregister interface, so the module-specific part is unnecessary.
That much about the theory. Of course, in real life, we have to face a few more problems:
- if callbacks can happen after apparently successful "unregister", we die
- if accesses to other static data owned by a module can happen after apparently successful "unregister", we may die
- if a module doesn't "unregister" at all, we die too
But all these problems equally affect code that does other things that can break a callback/access, e.g. if we destroy *de->data immediately after remove_proc_entry returns.
So this is not a module-specific problem.
Agreed ?
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
[RFC] Is an alternative module interface needed/possible?
Hi,
(Subject changed to hopefully get a bit more attention.)
Quoted 11 lines · Werner Almesberger
Quoted 3 lines
If we exclude the possibly-wait-forever-option, do you see the problem for dynamic objects which also contain references to static data/ functions?
You mean that two locking mechanisms are used, where one of them shouldn't be doing all that much ? Well, yes.
Now, is this a problem, or just a symptom ? I'd say it's a symptom:
we already have a perfectly good locking/synchronization method, and that's through the register/unregister interface, so the module-specific part is unnecessary.
If it was perfectly good, we hadn't a problem. :)
Quoted 9 lines
That much about the theory. Of course, in real life, we have to face a few more problems:
- if callbacks can happen after apparently successful "unregister", we die
- if accesses to other static data owned by a module can happen after apparently successful "unregister", we may die
- if a module doesn't "unregister" at all, we die too
Quoted 5 lines
But all these problems equally affect code that does other things that can break a callback/access, e.g. if we destroy
*de->dataimmediately afterremove_proc_entryreturns.So this is not a module-specific problem.
You're skipping ahead. You haven't solved the problem yet, but you're already jumping to conclusions. :-)
Remember, that we want to savely remove a proc entry and as added bonus, we only want a single reference count. Let's look first at the possible solutions:
module count: by design this only works for entries, which are removed during module exit, but not for dynamic entries.
failure: if the object is still busy, we just return -EBUSY. This is simple, but this doesn't work for modules, since during module exit you can't fail anymore.
callbacks: the callback function itself had to be protected somehow, so just to unregister a proc entry, you have to register a callback. To unregister that callback, it would be silly to use another callback and failure doesn't work with modules, so that only leaves the module count.
The last solution sounds complicated, but exactly this is done for filesystems and we didn't really get rid of the second reference count, we just moved it somewhere else, where it hurts least.
Without interface changes this is also the only generic option to export dynamic data - the drivers have to get a filesystem like interface (or just become filesystem themselves).
The very basic reason which prevents another solution is that static data (which includes functions) is controlled by the generic module code and dynamic data is controlled by the driver itself. It's obvious that we can't give the module code control over dynamic data, on the other hand would it be possible to give the driver control over the static data? This way it suddenly it becomes a module-specific problem - how can we give drivers more control over its data?
bye, Roman
Re: [RFC] Is an alternative module interface needed/possible?
Quoted 1 line · Roman Zippel
If it was perfectly good, we hadn't a problem. :)
I said we he have the method. Now we need to use it properly :-)
Quoted 2 lines
You're skipping ahead. You haven't solved the problem yet, but you're already jumping to conclusions. :-)
The solution is another issue. I simply stated that the problem happens with or without modules.
Quoted 2 lines
module count: by design this only works for entries, which are removed during module exit, but not for dynamic entries.
Works only for modules, not good.
Quoted 3 lines
failure: if the object is still busy, we just return
-EBUSY. This is simple, but this doesn't work for modules, since during module exit you can't fail anymore.
That's a modules API problem. And yes, I think modules should eventually be able to say that they're busy.
Quoted 3 lines
callbacks: the callback function itself had to be protected somehow, so just to unregister a proc entry, you have to register a callback. To unregister that callback, it would be silly to use another callback and
If all you want to do is to decrement the module count, you could have a global handler for this that is guaranteed not to reside in a module.
By the way, a loong time ago, in the modules thread, I suggested a "decrement_module_count_and_return" function [1]. Such a construct would be useful in this specific case.
[1] http://www.uwsg.iu.edu/hypermail/linux/kernel/0207.0/0147.html
Quoted 1 line
failure doesn't work with modules, so that only leaves the module count.
And how would you ensure correct access to static data in the absence of modules ? Any solution that _requires_ a module count looks highly suspicious to me.
Likewise, possibly dynamically allocated data that is synchronized by the caller, e.g. "user" in "struct proc_dir_entry".
Quoted 3 lines
The last solution sounds complicated, but exactly this is done for filesystems and we didn't really get rid of the second reference count, we just moved it somewhere else, where it hurts least.
Hmm, I'm confused. With "filesystem", do you mean the file system driver per se (e.g. "ext3"), or a specific instance of such a file system (e.g. /dev/hda1 mounted on /) ?
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Re: [RFC] Is an alternative module interface needed/possible?
In message <[email protected]> you write:
Quoted 3 lines
Quoted 3 lines
failure: if the object is still busy, we just return
-EBUSY. This is simple, but this doesn't work for modules, since during module exit you can't fail anymore.
Of course, they never could fail module exit. You could use can_unload() which was only protected by the BKL, hence so rarely
used: grep 2.4.20 to see all four users.
Having a separate can_unload function means a window between asking can_unload() and doing the actual unload. Doing it in the function itself (ie. cleanup returns an int, or a separate "prepare_to_unload" kind of function) means you have to deal with the "I started deregistering my stuff, but then had to re-register them" which leaves a race where the module is partially unavailable (ie. the spurious failure problem).
Both these problems are soluble, but they're not trivial.
Quoted 2 lines
That's a modules API problem. And yes, I think modules should eventually be able to say that they're busy.
Take a look at the current ip_conntrack code. It keeps its own reference count and spins until it hits zero in unload, because otherwise it would never be unloadable (it attaches a callback to each packet, so logically it would bump its refcount there). In 2.5 it can use try_module_get() and be unloaded with rmmod --wait in this worst case (I'm slack, patching is on my todo list).
Security modules have the same issues, and Greg Kroah-Hartmann was telling me USB has it as well.
You can see why I considered deprecating module unloading: it's a hard problem, and fixing it in general probably means massive changes. Which I'm happy to leave to someone else.
Quoted 3 lines
By the way, a loong time ago, in the modules thread, I suggested a "
decrement_module_count_and_return" function [1]. Such a construct would be useful in this specific case.
Stephen and I even implemented one, for x86 (see below). But implementing the matching "enter and inc" case remains problematic, so I dropped that idea.
Anyway, good luck!
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
Name: module_put_return primitive for x86
Author: Rusty Russell
Status: Experimental
Depends: Module/forceunload.patch.gz
D: This patch implements module_put_return() for x86, which allows a
D: module to control its own reference counts in some cases. A module
D: must never use module_put() on itself, since this may result in the
D: module being removable immediately: this is the alternative, which
D: atomically decrements the count and returns.
D:
D: Each architecture will need to implement this for preempt.
arch/i386/kernel/module.c +3 -0
@@ -28,6 +28,9 @@#define DEBUGP (fmt , ...)#endif +/* For the magic module return. */ +struct module_percpu module_percpu[NR_CPUS ]; +static void *alloc_and_zero (unsigned long size) {void *ret;
arch/i386/lib/module.S +33 -0
@@ -0,0 +1,33 @@ +/* Icky, icky, return trampoline for dying modules. Entered with + interrupts off. (C) 2002 Stephen Rothwell, IBM Corporation. */ + +#include <asm/thread_info.h> + +.text +.align 4 +.globl __magic_module_return + +#define MODULE_PERCPU_SIZE_ORDER 3 + +__magic_module_return: + /* Save one working variable */ + pushl %eax + + /* Get CPU number from current. */ + GET_THREAD_INFO(%eax) + movl TI_CPU(%eax), %eax + + /* Push module_percpu[cpu].flags on the stack */ + shll $MODULE_PERCPU_SIZE_ORDER, %eax + pushl module_percpu(%eax) + + /* Put module_percpu[cpu].returnaddr into %eax */ + movl module_percpu+4(%eax), %eax + + /* Push returnaddr and restore eax */ + xchgl %eax, 4(%esp) + + /* Restore interrupts */ + popf + /* Go home */ + ret
include/asm-i386/module.h +29 -0
@@ -8,4 +8,33 @@ struct mod_arch_specific#define Elf_Shdr Elf32_Shdr#define Elf_Sym Elf32_Sym#define Elf_Ehdr Elf32_Ehdr + +/* Non-preemtible decrement the refcount and return. */ +#define module_put_return (firstarg , ...) \ +do { \ + unsigned int cpu; \ + unsigned long flags; \ + \ + local_irq_save(flags); \ + if (unlikely(module_put(THIS_MODULE)) { \ + module_percpu[cpu].flags = flags; \ + module_percpu[cpu].returnaddr = ((void **)&(firstarg))[-1]; \ + ((void **)&(firstarg))[-1] = __magic_module_return; \ + } else \ + local_irq_restore(flags); \ + return __VA_ARGS__; \ +} while(0) + +/* FIXME: Use per-cpu vars --RR */ +struct module_percpu +{ +unsigned long flags; +void *returnaddr; +}; + +extern struct module_percpu module_percpu[NR_CPUS ]; + +/* Restore flags and return to caller. */ +extern void __magic_module_return (void ); +#endif /* _ASM_I386_MODULE_H */
Re: [RFC] Is an alternative module interface needed/possible?
Grr, I hate this. I just typed a long reply to your posting, and wanted to finish it with a reminder that the issue is more general than just modules, so that we shouldn't look at modules yet. Then I realized of course that everything above was about modules :-(
I don't think we'll make much progress if we keep on mixing issues of interface correctness, current module constraints, and possible module interface changes, all that with performance considerations and minimum invasive migration plans thrown in. So I'd suggest the following sequence:
1) do we agree that the current registration/deregistration interfaces are potential hazards for their users, be they modules or not ?
2) one we agree with this, we can look for mechanisms that solve this, again for general users, which may or may not be modules
3) last but not least, we can look at what this means for modules (and that's where beautiful tools like "module_put_return" (thanks !), or also ideas about module_exit redesign have their place)
4) "the root of all evil ...". Okay, and now to which level of hell would all this shoot our performance ? (And back we go to step 2.)
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Re: [RFC] Is an alternative module interface needed/possible?
Hi,
Quoted 6 lines · Werner Almesberger
I don't think we'll make much progress if we keep on mixing issues of interface correctness, current module constraints, and possible module interface changes, all that with performance considerations and minimum invasive migration plans thrown in. So I'd suggest the following sequence:
Quoted 13 lines
1) do we agree that the current registration/deregistration interfaces are potential hazards for their users, be they modules or not ?
2) one we agree with this, we can look for mechanisms that solve this, again for general users, which may or may not be modules
3) last but not least, we can look at what this means for modules (and that's where beautiful tools like "module_put_return" (thanks !), or also ideas aboutmodule_exitredesign have their place)
4) "the root of all evil ...". Okay, and now to which level of hell would all this shoot our performance ? (And back we go to step 2.)
Basically I can agree with this, although I'd like to avoid that we iterate too much over these steps, as it would too easily divert the discussion to other things, so I'd rather take smaller steps and keep the scope a bit broader.
Another point is the perfomance, which is not that important right now. I'm more interested in the general complexity, it's simply easier to optimize a simple design than a very complex design.
bye, Roman
Re: [RFC] Is an alternative module interface needed/possible?
(I think we should limit the Cc:s to Roman, Rusty, the list, and me, and leave the others in peace. Please yell if you really want that extra copy :-)
Quoted 4 lines · Roman Zippel
Basically I can agree with this, although I'd like to avoid that we iterate too much over these steps, as it would too easily divert the discussion to other things, so I'd rather take smaller steps and keep the scope a bit broader.
Good :-) I want to avoid modules as much as possible, because they've extensively been tackled in the past (which didn't help much making the interfaces better), and also because they're just a bit too political an issue.
Okay, this brings us to the issue of broken interfaces. Do we have agreement that there are cases where interfaces like remove_proc_entry, in their current state, cannot be used correctly ?
Example:
static ...callback...(... struct file *file ...)
{
void *user = PDE(file->f_dentry->d_inode)->data;
...
do something with "*user"
...
}...
struct proc_dir_entry *de = create_proc_entry(...); void *my_data; de->data = my_data = kmalloc(...); ... remove_proc_entry(...); /* what happens with "my_data", formerly known as "de->data" ? */
a) kfree it. May cause an oops or other problems if we're in the middle of the callback.
b) don't kfree it. So we now have a (hopefully small) memory leak.
Problem: my_data may point to a lot more than just some tiny allocation.
Okay so far ?
(Possible solutions on the next, slid^H^H^H^Hposting :-)
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Re: [RFC] Is an alternative module interface needed/possible?
Next round: possible remedies and their side-effects. As usual, if you disagree with something, please holler.
If yes, let's look at possible (and not overly insane) solutions, using remove_proc_entry as a case study:
1) still don't kfree, and leave it to the user to somehow minimize the damage. (Good luck :-)
2) add a callback that is invoked when the proc entry gets deleted. (This callback may be called before remove_proc_entry completes.) Problem: unload/return race for modules.
3) change remove_proc_entry or add remove_proc_entry_wait that works like remove_proc_entry, but blocks until the entry is deleted. Problem: may sleep "forever".
4) make remove_proc_entry return an indication of whether the entry was successfully removed or not. Problem: if it wasn't, what can we do then ?
5) like above, but don't remove the entry if we can't do it immediately. Problem: there's no notification for when we should try again, so we'd have to poll.
6) export the lookup mechanism, and let the caller poll for removal. Problem: races with creation of a new entry with the same name.
7) transfer ownership of de->data to procfs, and set some (possibly configurable) destruction policy (e.g. always kfree, or such). Similar to 2), but less flexible.
Any more ?
I think that most programmers would intuitively expect an interface of type 3). In cases where we can't sleep, either type 2) or type 5) would be common choices.
Does that sound reasonable so far ?
I'll wait a little until I continue with ways for dealing with the side-effects.
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Re: [RFC] Is an alternative module interface needed/possible?
In message <[email protected]> you write:
Quoted 6 lines
Next round: possible remedies and their side-effects. As usual, if you disagree with something, please holler.
If yes, let's look at possible (and not overly insane) solutions, using
remove_proc_entryas a case study:
Quoted 6 lines
1) still don't kfree, and leave it to the user to somehow minimize the damage. (Good luck :-)
2) add a callback that is invoked when the proc entry gets deleted. (This callback may be called before
remove_proc_entrycompletes.) Problem: unload/return race for modules.
OK. For reference, the "state of 2.4" solution (which is also the "state of 2.5" solution) looks like:
Quoted 4 lines
struct proc_dir_entry *de = create_proc_entry(...); void *my_data; de->data = my_data = kmalloc(...);
=====> de->owner = THIS_MODULE;
Quoted 3 lines
... remove_proc_entry(...); /* what happens with "my_data", formerly known as "de->data" ? */
And have proc_file_operations do the standard owner get and release:
open: proc_open,
release: proc_release,
static int proc_open(struct inode *inode, struct file *filp)
{
struct proc_dir_entry *dp = PDE(inode);
if (!try_module_get(dp->owner))
return -ENOENT;
return 0;
}static int proc_release(struct inode *inode, struct file *filp)
{
struct proc_dir_entry *dp = PDE(inode);
module_put(dp->owner);
return 0;
}Now, if remove_proc_entry() is called from module_exit(), the kfree() works fine, since (1) we wouldn't be in module_exit() if the proc entry was in used, and (2) the try_module_get() prevents any new users.
Of course, if you wanted to remove the entry at any other time (eg. hotplug), this doesn't help you one damn bit (which is kind of your point).
Quoted 3 lines
3) change
remove_proc_entryor addremove_proc_entry_waitthat works likeremove_proc_entry, but blocks until the entry is deleted. Problem: may sleep "forever".
This is what network devices do, and what the sockopt registration code does, too, so this is already in the kernel, too. It's not great, but it becomes a noop for the module deregistration stuff.
Thanks!
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
Re: [RFC] Is an alternative module interface needed/possible?
Quoted 3 lines · Rusty Russell
Of course, if you wanted to remove the entry at any other time (eg. hotplug), this doesn't help you one damn bit (which is kind of your point).
Yep, try_module_get solves the general synchronization problem for the special but interesting case of modules, but not for the general case.
Quoted 3 lines
This is what network devices do, and what the sockopt registration code does, too, so this is already in the kernel, too. It's not great, but it becomes a noop for the module deregistration stuff.
Yes, I think just sleeping isn't so bad at all. First of all, we already have the module use count as a kind of "don't unload now" advice (not sure if you plan to phase out MOD_INC_USE_COUNT ?), and second, it's not exactly without precedent anyway. E.g. umount will have little qualms of letting you sleep "forever". (And, naturally, every once in a while, people hate it for this :-)
Anyway, I'll write more about this tomorrow. For tonight, I have my advanced insanity 101 to finish, topic "ptracing more than one UML/TT at the same time".
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Re: [RFC] Is an alternative module interface needed/possible?
In message <[email protected]> you write:
Quoted 8 lines
Quoted 3 lines · Rusty Russell
Of course, if you wanted to remove the entry at any other time (eg. hotplug), this doesn't help you one damn bit (which is kind of your point).
Yep,
try_module_getsolves the general synchronization problem for the special but interesting case of modules, but not for the general case.
Absolutely. And I admire your (and Roman's) bravery for trying the general case.
Quoted 2 lines
will have little qualms of letting you sleep "forever". (And, naturally, every once in a while, people hate it for this :-)
(Well, MOD_INC_USE_COUNT is already deprecated, mainly because it's used to try to control module counts from within, which preempt makes really hard).
Yes, the addition of "umount -l" is congruent to "rmmod --wait". The assumption is "I don't want any new users, and I'll handle any current ones". You can get yourself in trouble with both of them.
Keep up the good work,
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
Re: [RFC] Is an alternative module interface needed/possible?
Hi,
Quoted 3 lines · Rusty Russell
Yes, the addition of "umount -l" is congruent to "rmmod --wait". The assumption is "I don't want any new users, and I'll handle any current ones". You can get yourself in trouble with both of them.
With the small difference that "umount -l" won't deadlock.
bye, Roman
Re: [RFC] Is an alternative module interface needed/possible?
In reply to Werner AlmesbergerHi,
Quoted 8 lines · Werner Almesberger
Next round: possible remedies and their side-effects. As usual, if you disagree with something, please holler.
If yes, let's look at possible (and not overly insane) solutions, using
remove_proc_entryas a case study:1) still don't kfree, and leave it to the user to somehow minimize the damage. (Good luck :-)
Obviously not a good idea.
Quoted 3 lines
3) change
remove_proc_entryor addremove_proc_entry_waitthat works likeremove_proc_entry, but blocks until the entry is deleted. Problem: may sleep "forever".
Sleeping is not a good general solution, as you have to be very careful not to hold any important locks, otherwise it's easy to abuse.
Quoted 3 lines
6) export the lookup mechanism, and let the caller poll for removal. Problem: races with creation of a new entry with the same name.
I don't really understand this one.
Quoted 3 lines
7) transfer ownership of
de->datato procfs, and set some (possibly configurable) destruction policy (e.g. always kfree, or such). Similar to 2), but less flexible.
de->data might contain references to other data and de->data might not the only dynamic data, just the most visible one, the read/write functions can access other dynamic data as well.
Quoted 8 lines
2) add a callback that is invoked when the proc entry gets deleted. (This callback may be called before
remove_proc_entrycompletes.) Problem: unload/return race for modules.4) make
remove_proc_entryreturn an indication of whether the entry was successfully removed or not. Problem: if it wasn't, what can we do then ?
Quoted 5 lines
5) like above, but don't remove the entry if we can't do it immediately. Problem: there's no notification for when we should try again, so we'd have to poll.
Any more ?
If you want to make 4) and 5) separate options, you could the same with 2), you can unlink the entry during remove_proc_entry or after the last user is gone (before the callback is called).
It would not be difficult to separate an unlink_proc_entry from remove_proc_entry, so we can force an unlink if needed and we can reduce this again to two basic options in two variations. :)
The question is now whether we return an error value or use a callback. When a device is removed, we usually also want to remove all its data structures, on the other hand we can only remove a module when there are no users, so here we return an error value.
Now I need a bigger example to put this into a context, a nice example is scsi_unregister. It removes among other things procfs entries and these entries have a reference to struct Scsi_Host. As long as scsi_unregister is called from module_exit everything works fine, but a bit searching reveals drivers/usb/storage/usb.c, which create/removes a scsi host when you plug/unplug a storage device (let's ignore other problems here, like it's still mounted). In this case nothing protects the removal of the proc entries anymore, this means if someone accesses /proc/scsi/usb-storage/... while the device is unplugged he will access freed data. Here would be now the patch from yesterday useful to implement a callback, but now we also can't simply free the Scsi_Host structure, so we have to add a reference count to it and only if all references to it are gone, it can be removed as well.
At this point I can also respond to Rusty's main argument against giving more control to the modules, that they suddenly had to deal with "I started deregistering my stuff, but then...". Unless interfaces are only used during module_init/module_exit, they have to deal with this anyway. I can understand that the module interface is tempting - just wait until all user are gone and then disable the module. The only problem is that this breaks horribly, when the driver interface is used in a different context (as demonstrated with the usb/scsi example). So what speaks against forcing driver/interface writers to get it right from the beginning?
(I at least could think of a few more reason that speak for this, but I'll continue this later.)
bye, Roman
Re: [RFC] Is an alternative module interface needed/possible?
Quoted 2 lines · Roman Zippel
Sleeping is not a good general solution, as you have to be very careful not to hold any important locks, otherwise it's easy to abuse.
Sleeping has many problems, but it also has one great advantage:
it's relatively easy to understand and doesn't need much extra code (if any) in the user.
Also, I wouldn't expect lock interdependencies to be a major problem at the level of something removing all traces of itself (e.g. a driver module unregistering itself). Or would you happen to know some example where such problems are happening ?
Quoted 5 lines
Quoted 3 lines
6) export the lookup mechanism, and let the caller poll for removal. Problem: races with creation of a new entry with the same name.
I don't really understand this one.
Something like:
foo_schedule_destruction(&whatever); while (foo_lookup(whatever.name)) yield();
Usually too ugly to consider for anything but desperate cases.
Quoted 3 lines
It would not be difficult to separate an
unlink_proc_entryfromremove_proc_entry, so we can force an unlink if needed and we can reduce this again to two basic options in two variations. :)
Yes, separating unlink (quick, non-blocking, always succeeds) and destruction (slow, may run nethack, fragile) clearly has some appeal.
Quoted 1 line
The question is now whether we return an error value or use a callback.
There's also the difference that the version using a callback would schedule the removal, while the version returning an error would do nothing (unless the caller tries again).
You could also do both, but that gets pretty difficult to use.
Quoted 2 lines
Now I need a bigger example to put this into a context, a nice example is
scsi_unregister.
Nicely illustrates the problem of the "can look around one corner, but not two" property of things like try_module_get, yes.
I'd also expect many cases of multiple devices which are serviced by the same driver to have the same sort of problems. E.g. I'm pretty sure I committed a few such sins in the ATM code, too.
Quoted 2 lines
So what speaks against forcing driver/interface writers to get it right from the beginning?
Raising the general awareness of such problems is exactly what I'm trying to accomplish here :-) By the way, also the argument "it may be broken but it has never failed so far" is getting weaker with the emergence of technologies that change relative code path lengths, like hyperthreading.
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Re: [RFC] Is an alternative module interface needed/possible?
Hi,
Quoted 7 lines · Werner Almesberger
Quoted 2 lines · Roman Zippel
Sleeping is not a good general solution, as you have to be very careful not to hold any important locks, otherwise it's easy to abuse.
Sleeping has many problems, but it also has one great advantage:
it's relatively easy to understand and doesn't need much extra code (if any) in the user.
Uninterruptable sleep should generally be avoided at kernel level, either it should be possible to interrupt it or it should at least include a timeout, but then you have to deal with failure again.
OTOH if you have callbacks, you can easily implement sleeping yourself.
Quoted 4 lines
Also, I wouldn't expect lock interdependencies to be a major problem at the level of something removing all traces of itself (e.g. a driver module unregistering itself). Or would you happen to know some example where such problems are happening ?
Imagine you want to remove a device structure, if you manage to block this somehow, you might completely block the creation and removeal of other devices.
Quoted 5 lines
Quoted 2 lines
Now I need a bigger example to put this into a context, a nice example is
scsi_unregister.Nicely illustrates the problem of the "can look around one corner, but not two" property of things like
try_module_get, yes.
Thanks, it seems I didn't cause too much confusion yet, so I can go on with the next part. I think we can agree that the module locking only protects a special case and as hotpluggable devices become more and more important, we should rather get it right in the general case.
Anyway, this alone would be not reason enough to change the module interface, but another module interface would give us more flexibility and reduce the locking complexity. To make it easy for Rusty I take an example he is familiar with - netfilter.
Let's assume we had an interface which allows exit to fail, how would that help? First, struct nf_hook_ops gets a reference count (struct module does the job), now with a small change to nf_unregister_hook:
void nf_unregister_hook(struct nf_hook_ops *reg)
{
br_write_lock_bh(BR_NETPROTO_LOCK);
- list_del(®->list);
+ list_del_init(®->list);
br_write_unlock_bh(BR_NETPROTO_LOCK);
}
we can do the following from module_exit:
int foo_exit()
{
nf_unregister_hook(&foo_ops);
return module_refcount(THIS_MODULE) ? -EBUSY : 0;
}to get a reference to the module this would be enough:
static inline void __module_get(struct module *module)
{
local_inc(&module->ref[smp_processor_id()].count);
}releasing the reference is as simple:
static inline void __module_put(struct module *module)
{
local_dec(&module->ref[smp_processor_id()].count);
}This probably needs a bit of explanation:
1. more flexibility: The driver has better the knowledge about how the module can be stopped and especially only the driver knows which events should prevent the shutdown of the module.
There is an important difference between stopping the module and actually removing the module. Any reference to the module must prevent the module removal, but not everything needs to stop the module from shutting down. For example an incoming network packet doesn't need to prevent the shutdown, so we can just remove the hook above independent of the reference count and just wait for the reference to become zero. On the other hand this means the module could be in a disabled state, but this can rather be compared with lazy unmount (via "umount -l") - after the last user is gone the module can be removed without problems, but the module could also be reenabled.
2. reduced locking complexity: Above demonstrates how it could look like, if we can get rid of the live state (at least outside of the generic module code). I explained the theory behind this before and posted the link already a few times:
http://marc.theaimsgroup.com/?l=linux-kernel&m=104284223130775&w=2
This live state isn't needed most of the time and if some module should really need this, it can easily implement that itself.
To help understanding this it maybe helps to clear up a common
misconception: modules are no special objects and they are not different to other objects which can be removed while the kernel is running. Even Rusty confuses this issue (http://lwn.net/Articles/15404/). The question should rather be: how can I safely access an object which I don't own? There are exactly two rules:
- The user has to lock out the owner.
- When the user wants to access the object outside the lock, he has to tell that the owner, usually done via a reference count.
One of these conditions must be true (no exception), otherwise we have a problem.
Applied to netfilter this means as long as we hold the netproto brlock we can safely access any netfilter hook and only if we need a reference to the hook outside of this lock, we had to call try_module_get(). This also means that we don't have to try to get a reference for every packet, instead e.g. it would be enough to do this once per connection. (Why now an atomic counter wasn't sufficient for this and why Rusty had to mess up the module code for this, will probably stay one of the big mysteries...)
bye, Roman
Re: [RFC] Is an alternative module interface needed/possible?
Quoted 3 lines · Roman Zippel
Anyway, this alone would be not reason enough to change the module interface, but another module interface would give us more flexibility and reduce the locking complexity.
Wait, wait ! :-) There's one step you've left out: what we actually expect the module interface to do. We have:
- what it currently does, or what it did in the past
- what users think it does
- what users want it to do
- what we think the users should want
- what we think is a comfortable compromise
With "users", I mean primarily the guy who invokes "rmmod", or such.
Anyway, I'm afraid I can't offer much wisdom from experience for this part, for I'm not much of a module user myself. I'll have more to say on service interfaces, though.
Sorry for slowing down, but I'm currently quite busy absorbing all the cool stuff that's recently been happening with UML. (So, blame Jeff ;-))
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Re: [RFC] Is an alternative module interface needed/possible?
Hi,
Quoted 7 lines · Werner Almesberger
Quoted 3 lines · Roman Zippel
Anyway, this alone would be not reason enough to change the module interface, but another module interface would give us more flexibility and reduce the locking complexity.
Wait, wait ! :-) There's one step you've left out: what we actually expect the module interface to do. We have:
There are several module interfaces:
- module user interface
- module load interface
- module driver interface
These are quite independent and so far we were talking about the last one, so I'm a bit confused about your request to talk about the first. <rant>
BTW Why Rusty had to completely break the first two interfaces to "improve" the last one, is probably another mystery, I'll never understand.
</rant>
bye, Roman
Re: [RFC] Is an alternative module interface needed/possible?
Quoted 4 lines · Roman Zippel
There are several module interfaces:
- module user interface
- module load interface
- module driver interface
Hmm, the "load interface" would be the system calls, while the "driver interface" would be initialization and cleanup functions in the module ?
We've already established that most things called from the latter isn't actually module specific.
Quoted 2 lines
These are quite independent and so far we were talking about the last one, so I'm a bit confused about your request to talk about the first.
I'm not so sure about them being independent. E.g. if we just had a blocking single-phase cleanup, users would always see success, but resources may be tied up indefinitely, which would break uses like
rmmod foo
insmod foo.o
On the other hand, with a non-blocking two-phase cleanup, users would still always see success, but only "anonymous" resources (memory, etc.) would be tied up.
Last but not least, a non-blocking single-phase cleanup would expose all failures to the user, and require some retry strategy.
Furthermore, use counts can have several meanings:
- indicate when it's convenient for the module to be removed
- indicate when it's possible for the module to be removed
- indicate what consequences the user may experience if trying to remove now (e.g. blocking)
The "old" module count was a bit of a mixture of the first two. The "new" count is clearer.
Oh, lest I be misunderstood: I'm not saying that we should redesign everything, and screw compatibility. I just want to bring the hidden assumptions that have piled up over the life of the module system out in the open.
Then, of course, there are still plenty of opportunities to plot any massive breakage ;-)
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Re: [RFC] Is an alternative module interface needed/possible?
Hi,
Quoted 11 lines · Werner Almesberger
Quoted 4 lines
There are several module interfaces:
- module user interface
- module load interface
- module driver interfaceHmm, the "load interface" would be the system calls, while the "driver interface" would be initialization and cleanup functions in the module ?
We've already established that most things called from the latter isn't actually module specific.
But the module driver interface has an impact on the other driver interfaces and that's what I tried to describe in the previous mail.
Quoted 10 lines
Quoted 2 lines
These are quite independent and so far we were talking about the last one, so I'm a bit confused about your request to talk about the first.
I'm not so sure about them being independent. E.g. if we just had a blocking single-phase cleanup, users would always see success, but resources may be tied up indefinitely, which would break uses like
rmmod foo
insmod foo.o
rmmod could already fail before, because the module is busy, so I'm not sure, what should break here.
Quoted 6 lines
On the other hand, with a non-blocking two-phase cleanup, users would still always see success, but only "anonymous" resources (memory, etc.) would be tied up.
Last but not least, a non-blocking single-phase cleanup would expose all failures to the user, and require some retry strategy.
Now you're skipping ahead. You are already looking at the possible consequences for the module user interface, before we actually made clear in which ways the module driver interface could be changed. There are of course dependencies between the interfaces, but you can change a lot in one before you have to modify another.
Quoted 8 lines
Furthermore, use counts can have several meanings:
- indicate when it's convenient for the module to be removed
- indicate when it's possible for the module to be removed
- indicate what consequences the user may experience if trying to remove now (e.g. blocking)The "old" module count was a bit of a mixture of the first two. The "new" count is clearer.
There is/was no count for the first?!
Which "new" count?
bye, Roman
Re: [RFC] Is an alternative module interface needed/possible?
In reply to Werner AlmesbergerQuoted 5 lines · Werner Almesberger
Next round: possible remedies and their side-effects. As usual, if you disagree with something, please holler.
If yes, let's look at possible (and not overly insane) solutions, using
remove_proc_entryas a case study:
Hi Werner,
Thanks for putting together this list; I've been following this thread for a while and it seems that the discussion always deteriorates into semantics. Anyway, I think there is an eighth possible solution to the list you proposed.
Just to be specific, the requirements for the proc entry stuff are:
a) a mechanism needs to be defined to indicate that the entry is no longer needed (something that starts the delete process -- ie, remove_proc_entry),
b) the code must conform to a system that ensures de->data will not be used after the "release" code is executed, and
c) the "release" code must be eventually executed.
Assuming these requirements are really requirements (your options 1 and 4 don't seem to meet these) then an "eighth" solution is:
8) Have the unregister code (remove_proc_entry) set an external flag (eg, de->data_is_there), and update all users of de->data to check the flag before following the pointer.
Option 8 may not qualify as "sane", but I think it is important to add it because it is what the module code is currently using. Thus, one need not look at the module stuff as a "special case", but as a general (if complicated) resource management solution.
Finally, one could probably apply any of the "options" for any dynamically allocated memory. It is interesting that Linus seems to prefer option 2/7 (from the recent kobject work and CodingStyle).
If I've missed something, please let me know.
-Kevin
Quoted 26 lines
1) still don't kfree, and leave it to the user to somehow minimize the damage. (Good luck :-)
2) add a callback that is invoked when the proc entry gets deleted. (This callback may be called before
remove_proc_entrycompletes.) Problem: unload/return race for modules.3) change
remove_proc_entryor addremove_proc_entry_waitthat works likeremove_proc_entry, but blocks until the entry is deleted. Problem: may sleep "forever".4) make
remove_proc_entryreturn an indication of whether the entry was successfully removed or not. Problem: if it wasn't, what can we do then ?5) like above, but don't remove the entry if we can't do it immediately. Problem: there's no notification for when we should try again, so we'd have to poll.
6) export the lookup mechanism, and let the caller poll for removal. Problem: races with creation of a new entry with the same name.
7) transfer ownership of
de->datato procfs, and set some (possibly configurable) destruction policy (e.g. always kfree, or such). Similar to 2), but less flexible.
Signature · 4 lines
------------------------------------------------------------------------ | Kevin O'Connor "BTW, IMHO we need a FAQ for | | [email protected] 'IMHO', 'FAQ', 'BTW', etc. !" | ------------------------------------------------------------------------
Re: [RFC] Is an alternative module interface needed/possible?
Hi,
Quoted 8 lines · Kevin O'Connor
8) Have the unregister code (
remove_proc_entry) set an external flag (eg,de->data_is_there), and update all users ofde->datato check the flag before following the pointer.Option 8 may not qualify as "sane", but I think it is important to add it because it is what the module code is currently using. Thus, one need not look at the module stuff as a "special case", but as a general (if complicated) resource management solution.
Yes, it's another possible solution, but it has the same problem as the current module locking - increased locking complexity.
Such flag actually exists already ("deleted"), but no user can use it currently, because the read/write functions don't have the proc entry argument. Even if they could use it, switching this flag isn't enough remove_proc_entry also had to synchronize with active users, so users had to take some lock just to read the data, where a simple reference was sufficient before.
bye, Roman
Re: [RFC] Is an alternative module interface needed/possible?
In reply to Werner AlmesbergerQuoted 9 lines · Werner Almesberger
Good :-) I want to avoid modules as much as possible, because they've extensively been tackled in the past (which didn't help much making the interfaces better), and also because they're just a bit too political an issue.
Okay, this brings us to the issue of broken interfaces. Do we have agreement that there are cases where interfaces like
remove_proc_entry, in their current state, cannot be used correctly ?
I hope this discussion is taking place in the context of looking forward towards something to implement in 2.7. IMHO we are much too late in the 2.5 cycle to implement this now.
Re: [RFC] Is an alternative module interface needed/possible?
In reply to Werner AlmesbergerHi,
Quoted 5 lines · Werner Almesberger
I don't think we'll make much progress if we keep on mixing issues of interface correctness, current module constraints, and possible module interface changes, all that with performance considerations and minimum invasive migration plans thrown in. So I'd suggest the following sequence:
Um, another point, let's ignore "minimum invasive migration plans", if we found a good solution, we can still figure out how to get there smoothly, so this shouldn't be a primary concern.
bye, Roman
Re: [RFC] Is an alternative module interface needed/possible?
Quoted 3 lines · Roman Zippel
Um, another point, let's ignore "minimum invasive migration plans", if we found a good solution, we can still figure out how to get there smoothly, so this shouldn't be a primary concern.
In my next posting, I just wrote (this very minute)
"For brainstorming's sake, let's not worry about backward-compatibility
for now:"
So I guess I don't need this disclaimer then ;-)
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Re: [RFC] Is an alternative module interface needed/possible?
In reply to Werner AlmesbergerHi,
Quoted 6 lines · Werner Almesberger
Quoted 3 lines
failure: if the object is still busy, we just return
-EBUSY. This is simple, but this doesn't work for modules, since during module exit you can't fail anymore.That's a modules API problem. And yes, I think modules should eventually be able to say that they're busy.
As Rusty already correctly said, this is currently not possible with the current modules API. On the other hand he also jumps very quickly to conclusions, so let's look at all the options first and then decide how trivial they are.
Anyway, for now just keep this option in mind and let's look at the other options, which don't require a module API change.
Quoted 3 lines
By the way, a loong time ago, in the modules thread, I suggested a "
decrement_module_count_and_return" function [1]. Such a construct would be useful in this specific case.
This would just be an optimization(?) for the module count, it doesn't change the general problem and is not useful as generic solution, so I'd rather put it back for now.
Quoted 5 lines
Quoted 1 line
failure doesn't work with modules, so that only leaves the module count.
And how would you ensure correct access to static data in the absence of modules ? Any solution that _requires_ a module count looks highly suspicious to me.
In that case it would be kernel memory, which cannot be freed, so it will not go away and requires no module count.
Quoted 2 lines
Likewise, possibly dynamically allocated data that is synchronized by the caller, e.g. "user" in "struct
proc_dir_entry".
user?
Quoted 7 lines
Quoted 3 lines
The last solution sounds complicated, but exactly this is done for filesystems and we didn't really get rid of the second reference count, we just moved it somewhere else, where it hurts least.
Hmm, I'm confused. With "filesystem", do you mean the file system driver per se (e.g. "ext3"), or a specific instance of such a file system (e.g.
/dev/hda1mounted on /) ?
A generic file system as it's registered via register_filesystem. A file system exports lots of dynamic and static data and the only (used) module pointer you will find is in struct file_system_type (the owner pointer in struct file_operations is not used by any standard fs). It's interesting to see how file systems keep the module business at a minimum (no try_module_get() is still cheaper than the cheapest try_module_get()).
Um, it's getting late and I just played too much with procfs to find a sensible solution. Below is an experimental patch to add callback which would allow it to safely remove user data. Very lightly tested, so no guarantees.
bye, Roman
Index: include/linux/proc_fs.h
===================================================================
RCS file: /usr/src/cvsroot/linux-2.5/include/linux/proc_fs.h,v retrieving revision 1.1.1.9
linux/proc_fs.h +1 -0
@@ -69,6 +69,7 @@ struct proc_dir_entry {void *data;read_proc_t *read_proc;write_proc_t *write_proc; +void (*release_proc)(struct proc_dir_entry *);atomic_t count;/* use count */ int deleted;/* delete flag */ kdev_t rdev;
Index: fs/proc/generic.c
===================================================================
RCS file: /usr/src/cvsroot/linux-2.5/fs/proc/generic.c,v retrieving revision 1.1.1.12
proc/generic.c +4 -0
@@ -639,6 +639,8 @@ void free_proc_entry(struct proc_dir_entif (ino <PROC_DYNAMIC_FIRST || ino >=PROC_DYNAMIC_FIRST +PROC_NDYNAMIC )return ; +if (de->release_proc) + de->release_proc (de);if (S_ISLNK (de->mode) && de->data)kfree (de->data);kfree (de);
@@ -655,6 +657,7 @@ void remove_proc_entry(const char *name,const char *fn = name;int len; +lock_kernel ();if (!parent &&xlate_proc_name (name, &parent, &fn) !=0 ) goto out; len =strlen (fn);
@@ -680,5 +683,6 @@ void remove_proc_entry(const char *name,break ; } out: +unlock_kernel ();return ; }
Index: fs/proc/inode.c
===================================================================
RCS file: /usr/src/cvsroot/linux-2.5/fs/proc/inode.c,v retrieving revision 1.1.1.13
proc/inode.c +22 -38
101 lines not on this page · raw message
Re: [RFC] Is an alternative module interface needed/possible?
Quoted 2 lines · Roman Zippel
Anyway, for now just keep this option in mind and let's look at the other options, which don't require a module API change.
Yes, we can look at the modules case at the end.
Quoted 2 lines
In that case it would be kernel memory, which cannot be freed, so it will not go away and requires no module count.
kfree isn't the only way to make data unusable. Remember the "my_state" example.
Quoted 4 lines
Quoted 2 lines
Likewise, possibly dynamically allocated data that is synchronized by the caller, e.g. "user" in "struct
proc_dir_entry".user?
"data", sorry. I always call this kind of argument something like "user" in my code ...
Quoted 1 line
A generic file system as it's registered via
register_filesystem.
Okay, I'll have a look at that.
Quoted 4 lines
Um, it's getting late and I just played too much with procfs to find a sensible solution. Below is an experimental patch to add callback which would allow it to safely remove user data. Very lightly tested, so no guarantees.
Yep, that's the kind of callback I had in mind.
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Hi,
Quoted 4 lines · Rusty Russell
Deprecating every module, and rewriting their initialization routines is ambitious beyond the scale of anything you have mentioned. Not that 90% of the kernel code couldn't use a damn good spring cleaning, but I'm not prepared to make such a change personally.
The transition would be rather painless:
if (mod->new_exit) {
res = -EBUSY;
if (force || !mod_use_count(mod))
res = mod->new_exit();
} else {
res = mod_try_exit();
if (!res && mod->old_exit) {
mod->old_exit();
}Yes, there is a small race, but at least it's nonfatal, as the module has to repeat the module count test or is protected otherwise. Now we can start fixing interfaces and exit functions can still call mod_try_exit() before calling old style interfaces.
Quoted 13 lines
So we go from:
int init(void)
{ if (!register_foo(&foo)) return -err; if (!register_bar(&bar)) { unregister_foo(&foo); return -err; } return 0; }
Quoted 7 lines
void fini(void)
{ unregister_foo(&foo); unregister_bar(&bar); }to:
int init(void)
{
if (register_foo(&foo))
return -err;
if (register_bar(&bar))
return -err;
return 0;
}void fini(void)
{
if (unregister_bar(&bar))
return -err;
if (unregister_foo(&foo))
return -err;
return 0;
}Hmm, looks as simple if not simpler to me.
Quoted 7 lines
Something like this?
static int
i_am_live;
staticspinlock_tmy_lock=SPIN_LOCK_UNLOCKED;/* This is our registered function. */
static intfoo_function(void *somedata)
Quoted 10 lines
{
int live;
spin_lock(&my_lock);
live = i_am_live;
spin_unlock(&my_lock);
if (!live)
return -EIGNOREME???;
...
}If the exit function starts the shutdown, there is no going back anymore. Let's take loop.c as example. It can either deconfigure the devices itself (via loop_clr_fd()) or it let's the user do it and only removes unused devices and prevents any new loop_set_fd() to succeed. In any case there will be no deadlock.
Quoted 5 lines
Quoted 2 lines
Hmm, what makes security modules (what exactly do you mean by that ?) special ?
On a busy system, they're never not being used. Your unload routine would always fail. Same with netfilter modules.
int unregister(foo)
{
lock();
list_del_init(&foo->list);
unlock();
return foo->usecount ? -EBUSY : 0;
}This makes sure no new user will start using it and at some point it has to return zero.
Quoted 5 lines
The current scheme is clean: it's two-stage delete with a nice helper function "
try_module_get()" which tells you when it's going away, and no requirement that modules actually implement two-stage delete themselves. The patch to mirror this in two-stage init was posted yesterday, as well.
The current scheme is the reserve/activate scheme in disguise and it sucks as much.
Quoted 2 lines
Unfortunately, I don't have the patience to explain this once for every kernel developer.
Well, I'd be happy to see an implementation that not only works for the most simplest cases.
bye, Roman
Hi,
Quoted 3 lines · Rusty Russell
2) It's bad enough to force the interfaces to change: at least the primitive they are to use is one many of them are already using, and is very simple to understand.
They are indeed simple, but only because it's impossible to implement anything more complex.
An example: A "rmmod -w loop" will currently deadlock on a busy loop module. Could you please explain, how it will be possible to force a safe removal of the loop module?
Quoted 4 lines
PS. The *implementation* flaw in your scheme: someone starts using a
module as you try to deregister it. Either you re-register the module (ie. you can never unload security modules), or you leave it half unloaded (even worse).
What is the problem with a half unloaded module? Only the module knows which interfaces it can safely remove to prevent new users, afterwards it only has to wait for remaining user to leave to complete the cleanup. BTW this also solves nicely the module init race.
bye, Roman