Skip to contentSkip to conversation

priv_data 0/2

PATCH In review Massimiliano Hofer 5 participants 16 messages Jun 26, 2006 – Aug 14, 2006 netfilter

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 priv_data 0/2

Hi,
this is a version of my priv_data patch that updates targets and renames functions as suggested by Patrick.

Since xt_init_match() and xt_init_target() (formerly xt_check_match() and xt_check_target()) no longer just check and they needed some argument changes anyway, I included some more common code previously replicated in ip_tables.c, ip6_tables.c and arp_tables.c.
Similarly I introduced xt_destroy_match() and xt_destroy_taget().

The resulting patches are larger than I anticipated, but most of the space is taken by function ranames and argument adjustments.

My previous example with xt_condition still applies (just rename checkentry to init in struct xt_match).

I tested several combination with iptables, ip6tables and arptables. It can't make it fail, but I didn't try it with a real world network load. Right now I don't have a 64 bit machine available for testing (I should be able to use one in a few days), so I didn't test compat at all.

Testing and comments, as always, are appreciated.

Signature · 2 lines
Saluti,
   Massimiliano Hofer
#1 lore raw
Quoted 16 lines · Massimiliano Hofer

Hi,
this is a version of my priv_data patch that updates targets and renames functions as suggested by Patrick.

Since xt_init_match() and xt_init_target() (formerly xt_check_match() and xt_check_target()) no longer just check and they needed some argument changes anyway, I included some more common code previously replicated in ip_tables.c, ip6_tables.c and arp_tables.c.
Similarly I introduced xt_destroy_match() and xt_destroy_taget().

The resulting patches are larger than I anticipated, but most of the space is taken by function ranames and argument adjustments.

My previous example with xt_condition still applies (just rename checkentry to init in struct xt_match).

Quoted 6 lines

I tested several combination with iptables, ip6tables and arptables. It can't make it fail, but I didn't try it with a real world network load. Right now I don't have a 64 bit machine available for testing (I should be able to use one in a few days), so I didn't test compat at all.

Testing and comments, as always, are appreciated.

Nice work, thanks Massimiliano. I'll have a closer look later and will probably apply it tommorrow. I hope we can still get this in 2.6.18.

#2 lore raw
Quoted 2 lines · Patrick McHardy

Nice work, thanks Massimiliano. I'll have a closer look later and will probably apply it tommorrow. I hope we can still get this in 2.6.18.

Thank you, this would be great.
If priv_data is merged I plan on having condition in good shape before 2.6.18 is released and hope it would be considered fit for merging in 2.6.19.

Signature · 2 lines
Bye,
   Massimiliano Hofer
#3 lore raw
Quoted 4 lines · Massimiliano Hofer
Quoted 2 lines · Patrick McHardy

Nice work, thanks Massimiliano. I'll have a closer look later and will probably apply it tommorrow. I hope we can still get this in 2.6.18.

If this is merged, I would like to see a negated limit... I need some advice to preserve compatibility here.

Regards
Sven

Signature · 9 lines
 Sven Anders <[email protected]>                 () Ascii Ribbon Campaign
                                                 /\ Support plain text e-mail
 ANDURAS service solutions AG
 Innstraße 71 - 94036 Passau - Germany
 Web: www.anduras.de - Tel: +49 (0)851-4 90 50-0 - Fax: +49 (0)851-4 90 50-55

Rechtsform: Aktiengesellschaft - Sitz: Passau - Amtsgericht Passau HRB 6032
Mitglieder des Vorstands: Sven Anders, Marcus Junker, Michael Schön
Vorsitzender des Aufsichtsrats: Dipl. Kfm. Thomas Träger
#4 lore raw
Quoted 2 lines · Sven Anders

If this is merged, I would like to see a negated limit... I need some advice to preserve compatibility here.

Once this is merged we can convert matches and targets to keep their private state outside of the structure shared with userspace and can start reusing the fields, since they are ignored by userspace anyway. I'm not sure, but I think currently userspace initializes them to 0, which should make this easy.

#5 lore raw
2006-07-04 02:20:40+0200, Patrick McHardy <[email protected]> ->
Quoted 10 lines
Quoted 2 lines · Sven Anders

If this is merged, I would like to see a negated limit... I need some advice to preserve compatibility here.

Once this is merged we can convert matches and targets to keep their private state outside of the structure shared with userspace and can start reusing the fields, since they are ignored by userspace anyway. I'm not sure, but I think currently userspace initializes them to 0, which should make this easy.

I wonder where all this priv_data changes ended? I have a few modules (that I havn't submited, because i've been way to lazy :-/) that will benefit from this. Among those a new limiter matcher simply called "lim". This match will solve Sven Anders problem with negating match. It's also much more aqurate and using a better limiter algoritm. Including being able to limit on packet size etc etc. It has been working stable in large scale routers (several 100Mbit/s) for a year now I think.

So where did this end up. The reason I am asking is because I was working on sending them in as patches. But they could really use this priv_data as they are using alot of ugly things to work this problem around. Same goes for a few target matches I have. They also need priv_data. Among them a "simple" -j COUNT that presents counters in /proc in an easy and fancy way (for scripts/rrdtool to read). I know hfpac wants this things they have droped the counters built in for each rule.

#6 lore raw
Quoted 15 lines · Joakim Axelsson
2006-07-04 02:20:40+0200, Patrick McHardy <[email protected]> ->
Quoted 11 lines
Quoted 2 lines · Sven Anders

If this is merged, I would like to see a negated limit... I need some advice to preserve compatibility here.

Once this is merged we can convert matches and targets to keep their private state outside of the structure shared with userspace and can start reusing the fields, since they are ignored by userspace anyway. I'm not sure, but I think currently userspace initializes them to 0, which should make this easy.

Quoted 1 line

I wonder where all this priv_data changes ended?

We missed the 2.6.18 merge window and I'm currently busy with a few other things, so I delayed it a bit. It will go in 2.6.19, I'll post the patches once I've merged them in my 2.6.19 tree.

Quoted 7 lines

I have a few modules (that
I havn't submited, because i've been way to lazy :-/) that will benefit from this. Among those a new limiter matcher simply called "lim". This match will solve Sven Anders problem with negating match. It's also much more aqurate and using a better limiter algoritm. Including being able to limit on packet size etc etc. It has been working stable in large scale routers (several 100Mbit/s) for a year now I think.

Please just post what you've got, I'd be interested in seeing why you chose to introduce a new match instead of enhancing the existing limit match.

#7 lore raw
2006-07-21 01:25:19+0200, Patrick McHardy <[email protected]> ->
Quoted 16 lines
Quoted 15 lines · Joakim Axelsson
2006-07-04 02:20:40+0200, Patrick McHardy <[email protected]> ->
Quoted 11 lines
Quoted 2 lines · Sven Anders

If this is merged, I would like to see a negated limit... I need some advice to preserve compatibility here.

Once this is merged we can convert matches and targets to keep their private state outside of the structure shared with userspace and can start reusing the fields, since they are ignored by userspace anyway. I'm not sure, but I think currently userspace initializes them to 0, which should make this easy.

Quoted 6 lines
Quoted 1 line

I wonder where all this priv_data changes ended?

We missed the 2.6.18 merge window and I'm currently busy with a few other things, so I delayed it a bit. It will go in 2.6.19, I'll post the patches once I've merged them in my 2.6.19 tree.

Okie. Guess i'll wait with updating it.

Quoted 11 lines
Quoted 7 lines

I have a few modules (that
I havn't submited, because i've been way to lazy :-/) that will benefit from this. Among those a new limiter matcher simply called "lim". This match will solve Sven Anders problem with negating match. It's also much more aqurate and using a better limiter algoritm. Including being able to limit on packet size etc etc. It has been working stable in large scale routers (several 100Mbit/s) for a year now I think.

Please just post what you've got, I'd be interested in seeing why you chose to introduce a new match instead of enhancing the existing limit match.

Because i wrote it before revisions existed and its takes totally different userspace parameters. You can find it here:
http://www.gozem.se/~gozem/netfilter/kernel/lim/

Userspace here:
http://www.gozem.se/~gozem/netfilter/userspace/

I'll try to submit them in pom-ng -format as soon as possible. That is if netfilter-dev wants them. I know there are supposed to be external svn-repositories nowdays, but i havn't really figured out how that is supposed to work yet. So please point me in the right direction.

Ihave a patch for kernel but it includes 3 more modules described below:
http://www.gozem.se/~gozem/netfilter/kernel/lim-norm-COUNT-SPEED.patch

Please read the comments on the algoritm in the code. Its more accurate as it never useds any division which will have rounding errors. However, its uses a few 64 bits integers. But i trade memory for a more accurate alogritm. Specially when dealing with a few 1000/s. The existing limit doesn't do this correctly at all. It starts limiting at a lower level. Also it has fetuares as:
- Negation
- bytes-limit instead of packets
- Blackout. Limit down to zero if limited.
- Much more flexible way ot taking the limit amount. The following is valid for example: --lim-packets 10k/2sec. Meaning 10 000 for each two seconds.
- Also, the --lim-start makes more sense than --limit-burst. --limit-burst is a multiplier of --limit. I belivie many thinks of it as an absolut figure then to start the limiting rather than being related as a multiplier for the base figure of --limit.
- lim is shorter to write :-) It a very commonly used match in my firewall setup.

The 3 other modules:
COUNT - As i've said before. A counting target that represents its figures in /proc

SPEED - A speed-caclulating target, other the same as COUNT.

norm - A normalizer of traffic/packet flow. Uses the algoritm of my 'lim' and my 'speed' combined. It has not been tested that well had the code needs cleaning up.

A few more coming is:
-j LOG new revision. It will take a longer --log-prefix. 29 chars are way to short for me. I have a large scale firewall setup and need to log the chain name and a reason. This simply doesn't fit in 29 chars for most of the time. Also a --log-limit and --drop. Its very common that you want to log the packets you DROP but do not want to flood yourself with messages in case of attacks. This solution will otherwise need a new chain and 2 or 3 rules in that new chain. Unnessasary waste of memory, CPU and cachelines.

-m log, Same as -J LOG but (wrongly) using a match so not an extra rule or chain is needed to actually do something with the packets later.

-m unclean new revision. With _alot_ of options so you can specifty exactly which tests you want. Also with a --log that logs the packet with a reason and an option prefix, rather than using the printk it does now. I know the general option is that unclean is dangerous and possibly wrong. However i use it very succesfully for several years now. A bit patched though to correct a few things.

If anything above sounds intressting (or wrong). Please let me know.

#8 lore raw
In reply to Patrick McHardy

* Patrick McHardy wrote, On 04/07/06 01:20:

Quoted 9 lines
Quoted 2 lines · Sven Anders

If this is merged, I would like to see a negated limit... I need some advice to preserve compatibility here.

Once this is merged we can convert matches and targets to keep their private state outside of the structure shared with userspace and can start reusing the fields, since they are ignored by userspace anyway. I'm not sure, but I think currently userspace initializes them to 0, which should make this easy.

Relying on userspace to initialize kernel structures to 0 smells like danger, and a possible exploit point.

Sam

#9 lore raw
Quoted 14 lines · Amin Azez

* Patrick McHardy wrote, On 04/07/06 01:20:

Quoted 10 lines
Quoted 2 lines · Sven Anders

If this is merged, I would like to see a negated limit... I need some advice to preserve compatibility here.

Once this is merged we can convert matches and targets to keep their private state outside of the structure shared with userspace and can start reusing the fields, since they are ignored by userspace anyway. I'm not sure, but I think currently userspace initializes them to 0, which should make this easy.

Quoted 2 lines

Relying on userspace to initialize kernel structures to 0 smells like danger, and a possible exploit point.

There is no difference here to a million other spots in the kernel. If something can create trouble, it needs to be verified. In the case of negation it just means "0 - don't negate", "1 - negate", and the fact that it used to be 0 can be used to provide backwards compatibility without introducing a new revision.

BTW, can you please fix your mailer so every mail doesn't go to the list twice?

#10 lore raw
In reply to Massimiliano Hofer
2006-06-26 16:41:46+0200, Massimiliano Hofer <[email protected]> ->
Quoted 16 lines

Hi,
this is a version of my priv_data patch that updates targets and renames functions as suggested by Patrick.

Since xt_init_match() and xt_init_target() (formerly xt_check_match() and xt_check_target()) no longer just check and they needed some argument changes anyway, I included some more common code previously replicated in ip_tables.c, ip6_tables.c and arp_tables.c.
Similarly I introduced xt_destroy_match() and xt_destroy_taget().

The resulting patches are larger than I anticipated, but most of the space is taken by function ranames and argument adjustments.

My previous example with xt_condition still applies (just rename checkentry to init in struct xt_match).

Quoted 10 lines

I tested several combination with iptables, ip6tables and arptables. It can't make it fail, but I didn't try it with a real world network load. Right now I don't have a 64 bit machine available for testing (I should be able to use one in a few days), so I didn't test compat at all.

Testing and comments, as always, are appreciated.

Signature · 2 lines
Saluti,
   Massimiliano Hofer

I'm currently porting some of my modules to use this API with priv_data. However, I ran into some troubles. For example i'm writing a revision 1 for the quota module which allows the count to go negative, and in the same time I'm porting it to use this API (as it's using the same stupid thing as -m limit with q->master = q;). However, having the quota counter in the priv_data struct makes it impossible to report the counter to userspace when the user issues iptables -L.

"Hidden", kernel data only is good with priv_data, but sometimes it will be too hidden.

Do we have any good solution for this?

--
Joakim Axelsson

#11 lore raw
Quoted 8 lines · Joakim Axelsson

I'm currently porting some of my modules to use this API with priv_data. However, I ran into some troubles. For example i'm writing a revision 1 for the quota module which allows the count to go negative, and in the same time I'm porting it to use this API (as it's using the same stupid thing as -m limit with q->master = q;). However, having the quota counter in the priv_data struct makes it impossible to report the counter to userspace when the user issues iptables -L.

That is actually a bug in the current module IMO (I looked into fixing this yesterday as well). iptables-save/restore won't work properly since they will save the _remaining_ quota, not the one the rule was created with. This might be useful as well, but it diverges from what other modules do.

Quoted 4 lines

"Hidden", kernel data only is good with priv_data, but sometimes it will be too hidden.

Do we have any good solution for this?

Not really, see the mail I just wrote.

#12 lore raw
2006-08-14 16:22:31+0200, Patrick McHardy <[email protected]> ->
Quoted 10 lines
Quoted 8 lines · Joakim Axelsson

I'm currently porting some of my modules to use this API with priv_data. However, I ran into some troubles. For example i'm writing a revision 1 for the quota module which allows the count to go negative, and in the same time I'm porting it to use this API (as it's using the same stupid thing as -m limit with q->master = q;). However, having the quota counter in the priv_data struct makes it impossible to report the counter to userspace when the user issues iptables -L.

Quoted 6 lines

That is actually a bug in the current module IMO (I looked into fixing this yesterday as well). iptables-save/restore won't work properly since they will save the _remaining_ quota, not the one the rule was created with. This might be useful as well, but it diverges from what other modules do.

Well, the priv_data patch will then solve this problem. As the userspace struct will always have the initial quota unaltered. The priv_data quota will decrease. However i guess you want to be able to see both. And in my case, saving the remaining quota with iptables-save is what i want. (I case of the router for some reason craches, the last state must be saved and as accurate as possible.)

Solution for this might be using two parameters for the userspace struct. One for initial quota and one for remaining, saving both. Possible even an option saying with figure should be set when iptables-restore is used.

Still, we can't nicly access the remaining quota with priv_data unless the modules each time it "matches" writes in both priv_data and userspace info, which is kinda ugly.

There is also the way of using a /proc-entry to list the current remaining quota. Also restoring it by echoing a new figure to the entry. Keep the iptables-save/restore only with the initial quota figure.

I can write this if this is what we want?

--
Joakim Axelsson

#13 lore raw
Quoted 18 lines · Joakim Axelsson
2006-08-14 16:22:31+0200, Patrick McHardy <[email protected]> ->
Quoted 10 lines
Quoted 7 lines · Joakim Axelsson

I'm currently porting some of my modules to use this API with priv_data. However, I ran into some troubles. For example i'm writing a revision 1 for the quota module which allows the count to go negative, and in the same time I'm porting it to use this API (as it's using the same stupid thing as -m limit with q->master = q;). However, having the quota counter in the priv_data struct makes it impossible to report the counter to userspace when the user issues iptables -L.

Quoted 5 lines

That is actually a bug in the current module IMO (I looked into fixing this yesterday as well). iptables-save/restore won't work properly since they will save the _remaining_ quota, not the one the rule was created with. This might be useful as well, but it diverges from what other modules do.

Quoted 3 lines

Well, the priv_data patch will then solve this problem. As the userspace struct will always have the initial quota unaltered. The priv_data quota will decrease.

That was the initial idea, forgetting about the atomic table exchange.

Quoted 1 line

However i guess you want to be able to see both.

Not really "want", its necessary, otherwise the quota will be refilled on each ruleset update.

Quoted 4 lines

And in my case,
saving the remaining quota with iptables-save is what i want. (I case of the router for some reason craches, the last state must be saved and as accurate as possible.)

I can see that this would also be useful, however none of the other stateful matches does this, so the more important point IMO is to get quota to do the expected thing. If we can do both, also fine, but it would have to be optional and I can't see how to cleanly do this.

Quoted 3 lines

Solution for this might be using two parameters for the userspace struct. One for initial quota and one for remaining, saving both. Possible even an option saying with figure should be set when iptables-restore is used.

Yes. Without breaking compatibility, we could make quota fixed, use the master pointer to store the current quota and put the new master pointer somewhere outside of the struct. But thats really not very pretty.

Quoted 3 lines

Still, we can't nicly access the remaining quota with priv_data unless the modules each time it "matches" writes in both priv_data and userspace info, which is kinda ugly.

As I said, I don't see saving current state as necessary (we don't do it anywhere and it solves an entirely different problem). The unfortunate fact though is that we need to pass it to userspace anyway, because of the limitations of the userspace interface.

Quoted 5 lines

There is also the way of using a /proc-entry to list the current remaining quota. Also restoring it by echoing a new figure to the entry. Keep the iptables-save/restore only with the initial quota figure.

I can write this if this is what we want?

Definitely not :) We want a better userspace interface that doesn't require us to put in hacks like this.

#14 lore raw

Re: xt_quota (Was: [PATCH] priv_data 0/2)

2006-08-14 17:43:29+0200, Patrick McHardy <[email protected]> ->
Quoted 6 lines

As I said, I don't see saving current state as necessary (we don't do it anywhere and it solves an entirely different problem). The unfortunate fact though is that we need to pass it to userspace anyway, because of the limitations of the userspace interface.

That might be correct. However the other modules doesn't normally suffer that much as they will farily quickly work thier way to the same state again. I guess it's very uncommon that you are using so low limits like a few packets each hour.

Quoted 9 lines
Quoted 5 lines

There is also the way of using a /proc-entry to list the current remaining quota. Also restoring it by echoing a new figure to the entry. Keep the iptables-save/restore only with the initial quota figure.

I can write this if this is what we want?

Definitely not :) We want a better userspace interface that doesn't require us to put in hacks like this.

I however really needs some way of figuring out how much of the quota that remains. This is to be able to report this to our users (that receives a certain number of gigabytes each day). So they can see how much they have left (using som scripted interface to iptables). Also saving this holy figure (as it has become :-)) for the user if the router for some reason craches. This is also the reason i need negative quota figures. The users are allowed to "borrow" from their future quota. Doing so only under a byte limiting match (-m lim --limit-bytes 20k/s).

In my opinion its more important to save the remaining quota, rather than the original. And most important to in some way be able to see how much is left of the quota.

Perhaps this wil satisfy both of us:

Somehting put out by iptables-save
iptables -m quota --init-quota 1000 --remain-quota 123 --use-quota remain

Somthing you write with iptables to create a new rule:
iptables -m quota --init-quota 1000 (using --use-quota init explicity)
iptables -m quota --init-quota 1000 --use-quota remain

But this sure is ugly.

--
Joakim Axelsson

#15 lore raw

Re: xt_quota (Was: [PATCH] priv_data 0/2)

Quoted 15 lines · Joakim Axelsson

I however really needs some way of figuring out how much of the quota that remains. This is to be able to report this to our users (that receives a certain number of gigabytes each day). So they can see how much they have left (using som scripted interface to iptables). Also saving this holy figure (as it has become :-)) for the user if the router for some reason craches. This is also the reason i need negative quota figures. The users are allowed to "borrow" from their future quota. Doing so only under a byte limiting match (-m lim --limit-bytes 20k/s).

In my opinion its more important to save the remaining quota, rather than the original. And most important to in some way be able to see how much is left of the quota.

Perhaps this wil satisfy both of us:

Quoted 8 lines

Somehting put out by iptables-save
iptables -m quota --init-quota 1000 --remain-quota 123 --use-quota remain

Somthing you write with iptables to create a new rule:
iptables -m quota --init-quota 1000 (using --use-quota init explicity)
iptables -m quota --init-quota 1000 --use-quota remain

But this sure is ugly.

It should be an explicit flag to iptable-save/restore to save the current state, because we don't do it anywhere else and therefore it is unexpected. The limit match for example does neither show nor save the current amount of tokens, last refill time, ... And I'm not too much of a fan of adding such a flag because it can only be done for a subset of all modules, hashlimit, recent etc. all can't do it. The most extreme case would be the state match :)

With a netlink API we could actually dump all internal state (including things like recently seen IP addresses) and accept changes from userspace. This would allow us to get rid of the ugly proc interfaces and covers your need as well.