modifying initializers with spatch?
Hi,
I've been playing with C99 initializers a bit, what I'm trying to do is replace them with explicit memset()/assignments.
This works as expected:
@@
identifier x;
type t;
@@
?t x =
-{0};
+{};
This actually removes the braces, but not only if they're empty, and it leaves the inner content there without braces, which won't even compile afterwards:
@@
identifier x;
type t;
@@
?t x
- ={}
?;
e.g.:
- const u8 q[] = { 1, (const u8) '*', 0 };
+ const u8 q[] 1, (const u8) '*', 0;
I've also not managed to add any memset(&x, 0, sizeof(x)); in the same patch, especially since it should go after all other declarations.
Do you think this is possible at all?
johannes
Quoted 20 lines · Johannes Berg
Hi,
I've been playing with C99 initializers a bit, what I'm trying to do is replace them with explicit memset()/assignments.
This works as expected:
@@
identifier x;
type t;
@@
?t x =(quoted diff) +1
-1-{0}; +{};This actually removes the braces, but not only if they're empty, and it leaves the inner content there without braces, which won't even compile afterwards:
Quoted 7 lines
@@
identifier x;
type t;
@@
?t x
- ={}
?;
I'm not sure what you are trying to do here. Remove only the empty ones?
Quoted 9 lines
e.g.:
(quoted diff) +1
-1- const u8 q[] = { 1, (const u8) '*', 0 }; + const u8 q[] 1, (const u8) '*', 0;I've also not managed to add any
memset(&x, 0, sizeof(x)); in the same patch, especially since it should go after all other declarations.
Here you would want something like:
f(...) {
when != S
+ memset(...);
S1
...
}
S and S1 should be statements. If there could be multiple memsets at the beginning of a single function, use ++ instead of +.
julia
Quoted 9 lines
Do you think this is possible at all?
johannes
_______________________________________________
Cocci mailing list
Cocci at systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci
Quoted 11 lines
Quoted 8 lines
@@
identifier x;
type t;
@@
?t x
- ={}
?;I'm not sure what you are trying to do here.??Remove only the empty ones?
I actually tried something more like
@@
@@
identifier x;
type t;
@@
t x
- ={}
;
+memset(&x, 0, sizeof(x));
but that didn't really work either :)
Quoted 13 lines
Quoted 3 lines
I've also not managed to add any
memset(&x, 0, sizeof(x)); in the same
patch, especially since it should go after all other declarations.Here you would want something like:
f(...) {
? when != S
+ memset(...);
? S1
? ...
}
Quoted 2 lines
S and S1 should be statements.??If there could be multiple memsets at the beginning of a single function, use ++ instead of +.
Ok, so I should combine with the previous one before - do I need the f(...) or are just the braces enough? This could be in a nested block, after all.
johannes
Quoted 39 lines · Johannes Berg
Quoted 11 lines
Quoted 8 lines
@@
identifier x;
type t;
@@
?t x
- ={}
?;I'm not sure what you are trying to do here.??Remove only the empty ones?
I actually tried something more like
@@
@@
identifier x;
type t;
@@
t x
- ={}
;
+memset(&x, 0, sizeof(x));but that didn't really work either :)
Quoted 13 lines
Quoted 3 lines
I've also not managed to add any
memset(&x, 0, sizeof(x)); in the same
patch, especially since it should go after all other declarations.Here you would want something like:
f(...) {
? when != S
+ memset(...);
? S1
? ...
}
Quoted 6 lines
Quoted 2 lines
S and S1 should be statements.??If there could be multiple memsets at the beginning of a single function, use ++ instead of +.
Ok, so I should combine with the previous one before - do I need the f(...) or are just the braces enough? This could be in a nested block, after all.
Try for the braces. Maybe something like
{
... when != { ... }
T x
- = {0}
;
... when != S
when any
++ memset(...)
S1
... when any
}
Quoted 13 lines
Try for the braces.??Maybe something like
{
? ... when != { ... }
? T x
-?????= {0}
? ;
? ... when != S
??????when any
++ memset(...)
???S1
???... when any
}
Yep, thanks! (I'm on the phone, and didn't get a chance to try yet)
That works for anything that's actually "= {0}" today, but for I can't even match "= {}" properly, it seems?
johannes
Quoted 14 lines · Johannes Berg
Quoted 13 lines
Try for the braces.??Maybe something like
{
? ... when != { ... }
? T x
-?????= {0}
? ;
? ... when != S
??????when any
++ memset(...)
???S1
???... when any
}
Quoted 4 lines
Yep, thanks! (I'm on the phone, and didn't get a chance to try yet)
That works for anything that's actually "= {0}" today, but for I can't even match "= {}" properly, it seems?
The problem is that Coccinelle lets you put fewer field initializers than are actually available. The reasoning is that they are optional for fields, and the user doesn't really know how many will be present in each case. {} is the degenerate case where none are mentioned. But that should not apply to your case, 1) because you are removing the braces, and
2) because you aren't doing field initializations anyway. I'll have to look into it.
julia
Quoted 11 lines
The problem is that Coccinelle lets you put fewer field initializers than
are actually available.??The reasoning is that they are optional for fields, and the user doesn't really know how many will be present in each
case.??{} is the degenerate case where none are mentioned.??But that should not apply to your case, 1) because you are removing the braces, and
2) because you aren't doing field initializations anyway.??I'll have to look into it.
Ok, thanks.
I actually had a separate thought, that I could perhaps do something like
{
... when != { ... }
T x = {
- .field = E,
};
... when != S
when any
++ x.field = E;
To get rid of all the initializers, but (as I'm still on the phone) I haven't tried it yet :)
Btw, how was the ++ supposed to work? I had to run this more than once to actually make it catch multiple instances of = {0}. Also, in the case above, even if it works, the order of initializations might be inverted if I run the spatch multiple times, so not sure I should do that.
johannes
Quoted 28 lines · Johannes Berg
Quoted 11 lines
The problem is that Coccinelle lets you put fewer field initializers than
are actually available.??The reasoning is that they are optional for fields, and the user doesn't really know how many will be present in each
case.??{} is the degenerate case where none are mentioned.??But that should not apply to your case, 1) because you are removing the braces, and
2) because you aren't doing field initializations anyway.??I'll have to look into it.Ok, thanks.
I actually had a separate thought, that I could perhaps do something like
{
... when != { ... }
T x = {
- .field = E,
};
... when != S
when any
++ x.field = E;
Quoted 8 lines
To get rid of all the initializers, but (as I'm still on the phone) I haven't tried it yet :)
Btw, how was the ++ supposed to work? I had to run this more than once to actually make it catch multiple instances of = {0}. Also, in the case above, even if it works, the order of initializations might be inverted if I run the spatch multiple times, so not sure I should do that.
Put a when any above the T x = line.
On the other hand, ++ makes no guarantee about the order in which things are generated.
To ensure the order, put the when any above the T x line, and drop the one below and run multiple times. Then each run will work on the last one and put it first, so they will come out in order.
julia
Quoted 3 lines
johannes
Partial progress: The github version will no longer match {} against { 0, 1, 2 }, but it persists in matching {} against { .a = 1, .b = 2 }, and will still remove the outer braces without removing the rest.
julia
Quoted 4 lines · Julia Lawall
Partial progress:??The github version will no longer match {} against { 0, 1, 2 }, but it persists in matching {} against { .a = 1, .b = 2 },
and will still remove the outer braces without removing the rest.
Out of curiosity: will it require some form of "..." to match something inside now?
We have a few things like this:
@@
identifier x;
identifer fn;
@@my_time x = {
+#ifdef XYZ
.this_method_needs_xyz = fn,
+#endif
...
};
in backports etc. and I've always wondered why it didn't need (or even accept, IIRC) ... before the +#ifdef line.
johannes
Quoted 8 lines · Johannes Berg
Quoted 4 lines · Julia Lawall
Partial progress:??The github version will no longer match {} against { 0, 1, 2 }, but it persists in matching {} against { .a = 1, .b = 2 },
and will still remove the outer braces without removing the rest.Out of curiosity: will it require some form of "..." to match something inside now?
No.
Quoted 14 lines
We have a few things like this:
@@
identifier x;
identifer fn;
@@my_timex = {(quoted diff) +2
-0+#ifdef XYZ .this_method_needs_xyz = fn, +#endif };
Quoted 2 lines
in backports etc. and I've always wondered why it didn't need (or even accept, IIRC) ... before the +#ifdef line.
It considers that the rule writer has no knowledge or control of the field names that are provided or the order in which they appear, so you can specify any subset of them. You only need ... when you want to add a new one at the beginning or at the end of the structure. The ... lets you specify beginning or end.
I want to allow you to remove the {} only when all of the field initializations are also removed. Some work though is required.
julia
Quoted 4 lines · Julia Lawall
Quoted 2 lines
Out of curiosity: will it require some form of "..." to match something inside now?
No.
Ok.
Quoted 5 lines
It considers that the rule writer has no knowledge or control of the field names that are provided or the order in which they appear, so you can specify any subset of them.??You only need ... when you want to add a new one at the beginning or at the end of the structure.??The ... lets you specify beginning or end.
Interesting. I thought I needed the ... at the end, but that may have been a case of "let's see if I need both - doesn't work - remove one - ok works now" :)
Quoted 2 lines
I want to allow you to remove the {} only when all of the field initializations are also removed.??Some work though is required.
That'd make sense.
I'm very grateful for this. I'm dealing with an awful compiler that emits stupid code for C99 initializers, so I'm thinking of using what we were discussing here to remove them all at build time. I'm going to be travelling (again) next week, but will give it a try later.
Thanks a lot!
johannes
Quoted 5 lines
Quoted 4 lines
To get rid of all the initializers, but (as I'm still on the phone) I haven't tried it yet :)
Quoted 8 lines
Put a when any above the T x = line.
On the other hand, ++ makes no guarantee about the order in which things are generated.
To ensure the order, put the when any above the T x line, and drop the one below and run multiple times.??Then each run will work on the last one and put it first, so they will come out in order.
Finally got a chance to try this.
I can't seem to get the ordering right. I'm trying this:
@@
type T;
expression E;
statement S1, S2;
identifier x;
identifier f;
@@
?{
... when != { ... }
????when any
?T x = {
-??.f = E,
?};
... when != S1
+ x.f = E;
?S2
?...
?}
on a very simple file:
int main()
{
struct foo bar = {
.y = 8,
.x = 7,
};
printf("%d\n", bar.x);
printf("%d\n", bar.y);
}but it doesn't end well:
$ spatch??--sp-file /tmp/init.spatch /tmp/test.c init_defs_builtins: /usr/lib/coccinelle/standard.h HANDLING: /tmp/test.c ????? previous modification:
? <<< x.f = E;
CONTEXT
According to environment 3:
???rule starting on line 1.E -> 8
???rule starting on line 1.f -> id y
???rule starting on line 1.x -> id bar
current modification:
? <<< x.f = E;
CONTEXT
According to environment 3:
???rule starting on line 1.E -> 7
???rule starting on line 1.f -> id x
???rule starting on line 1.x -> id bar
Fatal error: exception Failure("rule starting on line 1: already tagged token:\nC code context\nFile \"/tmp/test.c\", line 8, column 1,??charpos = 59\n????around = 'printf', whole content = \tprintf(\"%d\\n\", bar.x);")
If I use ++ instead of +, it still replaces all of them, so I can't run multiple times (usefully anyway)
OTOH, I haven't found any place where the order actually matters (in the code base I'm interested in), and although it seems that it theoretically could matter, I'm not sure I really need to solve that problem - although I'm thinking I might want to run this not just once but before each compile, to keep the code neater.
(Note that I also haven't upgraded to the git version yet, TBD)
johannes
Quoted 74 lines · Johannes Berg
Quoted 5 lines
Quoted 4 lines
To get rid of all the initializers, but (as I'm still on the phone) I haven't tried it yet :)
Quoted 8 lines
Put a when any above the T x = line.
On the other hand, ++ makes no guarantee about the order in which things are generated.
To ensure the order, put the when any above the T x line, and drop the one below and run multiple times.??Then each run will work on the last one and put it first, so they will come out in order.
Finally got a chance to try this.
I can't seem to get the ordering right. I'm trying this:
@@
type T;
expression E;
statement S1, S2;
identifier x;
identifier f;
@@
?{
... when != { ... }
????when any
?T x = {
-??.f = E,
?};
... when != S1
+ x.f = E;
?S2
?...
?}on a very simple file:
int
main(){ struct foo bar = { .y = 8, .x = 7, }; printf("%d\n", bar.x); printf("%d\n", bar.y); }but it doesn't end well:
$ spatch??--sp-file /tmp/init.spatch /tmp/test.c init_defs_builtins: /usr/lib/coccinelle/standard.h HANDLING: /tmp/test.c ????? previous modification:? <<< x.f = E;
CONTEXT
According to environment 3:
???rule starting on line 1.E -> 8
???rule starting on line 1.f -> id y
???rule starting on line 1.x -> id barcurrent modification:
? <<< x.f = E;
CONTEXT
According to environment 3:
???rule starting on line 1.E -> 7
???rule starting on line 1.f -> id x
???rule starting on line 1.x -> id bar
Quoted 4 lines
Fatal error: exception Failure("rule starting on line 1: already tagged token:\nC code context\nFile \"/tmp/test.c\", line 8, column 1,??charpos = 59\n????around = 'printf', whole content = \tprintf(\"%d\\n\", bar.x);")
Yeah, it doesn't work. There is no way to get the fields out of the structure one at a time. So ++ seems like the only option. The suggested organization would work for when there is more than one structure definition in the function, but doesn't help for multiple fields within a structure.
julia
Quoted 15 lines
If I use ++ instead of +, it still replaces all of them, so I can't run multiple times (usefully anyway)
OTOH, I haven't found any place where the order actually matters (in the code base I'm interested in), and although it seems that it theoretically could matter, I'm not sure I really need to solve that problem - although I'm thinking I might want to run this not just once but before each compile, to keep the code neater.
(Note that I also haven't upgraded to the git version yet, TBD)
johannes
Quoted 19 lines · Julia Lawall
Quoted 10 lines
@@
type T;
expression E;
statement S1, S2;
identifier x;
identifier f;
@@
?{
... when != { ... }
????when anyQuoted 8 lines
?T x = {
-??.f = E,
?};
... when != S1
+ x.f = E;
?S2
?...
?}
Quoted 2 lines
Yeah, it doesn't work.??There is no way to get the fields out of the structure one at a time.??So ++ seems like the only option.??
Right. Like I said, I don't think it matters much for me right now. I also found that most of these are actually in the part of the code that I can change and don't really need to touch at build time, although it'd be nice to be able to write the code in the more canonical C99 style and still do it at build time.
Quoted 5 lines
The suggested
organization would work for when there is more than one structure definition in the function, but doesn't help for multiple fields within a
structure.
Right, it easily works for multiple structures, and it does even seem to keep the order of variables, even if it doesn't keep the order of the fields within each variable (but sorts them alphabetically instead, apparently)
The git version should now actually be sufficient for me, since you said it will no longer remove braces from "= { 1, 2, 3}", and I'm moving all the initializers anyway like indicated above, and translating "={0}" to "={}" as well. I'll need to build & try it, but if it fails on some obscure cases and removes the braces anyway the compile will fail so I'm not really worried :)
johannes
Quoted 41 lines · Johannes Berg
Quoted 19 lines · Julia Lawall
Quoted 10 lines
@@
type T;
expression E;
statement S1, S2;
identifier x;
identifier f;
@@
?{
... when != { ... }
????when anyQuoted 8 lines
?T x = {
-??.f = E,
?};
... when != S1
+ x.f = E;
?S2
?...
?}Quoted 2 lines
Yeah, it doesn't work.??There is no way to get the fields out of the structure one at a time.??So ++ seems like the only option.??
Right. Like I said, I don't think it matters much for me right now. I also found that most of these are actually in the part of the code that I can change and don't really need to touch at build time, although it'd be nice to be able to write the code in the more canonical C99 style and still do it at build time.
Quoted 5 lines
The suggested
organization would work for when there is more than one structure definition in the function, but doesn't help for multiple fields within a
structure.Right, it easily works for multiple structures, and it does even seem to keep the order of variables, even if it doesn't keep the order of the fields within each variable (but sorts them alphabetically instead, apparently)
Quoted 6 lines
The git version should now actually be sufficient for me, since you said it will no longer remove braces from "= { 1, 2, 3}", and I'm moving all the initializers anyway like indicated above, and translating "={0}" to "={}" as well. I'll need to build & try it, but if it fails on some obscure cases and removes the braces anyway the compile will fail so I'm not really worried :)
Ok, sounds good. Let me know if you encounter further problems.
julia
Quoted 2 lines · Julia Lawall
Ok, sounds good.??Let me know if you encounter further problems.
You jinxed it ;-)
Now with the git version.
struct foo v = {
.nested = {
.x = 7,
.y = 8,
},
};still broke, because of course my initializer replacement rule didn't actually find it, and then the bug hit and removed the {} anyway even though they weren't empty :)
Similarly,
char * foo[] = {
[1] = "name1",
[2] = "name2",
};was butchered quite a bit :)
The latter case - though not with strings - actually does seem to be present in the code I care about, but I can probably just replace it manually first.
Interestingly, this whole exercise is obviously not even a good idea for static variables - need to see if I can exclude those somehow, first naive attempts didn't work :)
johannes
Quoted 2 lines · Julia Lawall
Ok, sounds good.??Let me know if you encounter further problems.
I've now ended up with the following overall:
https://p.sipsolutions.net/15e6265ea961d9a6.txt
Interestingly, that takes minutes to run on this file:
https://w1.fi/cgit/hostap/tree/wpa_supplicant/dbus/dbus_new_handlers_p2p.c
Not that it matters much to me, but I can't really explain it, all other files I found finish quickly.
johannes
Quoted 10 lines · Johannes Berg
Quoted 2 lines · Julia Lawall
Ok, sounds good.??Let me know if you encounter further problems.
I've now ended up with the following overall:
https://p.sipsolutions.net/15e6265ea961d9a6.txtInterestingly, that takes minutes to run on this file:
https://w1.fi/cgit/hostap/tree/wpa_supplicant/dbus/dbus_new_handlers_p2p.c
Well, it's been 20 minutes - I'm giving up :)
johannes
Quoted 5 lines · Johannes Berg
Quoted 4 lines · Johannes Berg
Quoted 2 lines · Julia Lawall
Ok, sounds good.??Let me know if you encounter further problems.
Quoted 8 lines
Quoted 6 lines
I've now ended up with the following overall:
https://p.sipsolutions.net/15e6265ea961d9a6.txtInterestingly, that takes minutes to run on this file:
https://w1.fi/cgit/hostap/tree/wpa_supplicant/dbus/dbus_new_handlers_p2p.cWell, it's been 20 minutes - I'm giving up :)
It seems that the control flow was quite complete in one function, and the when any at the end of the rule was taking a long time. But actually you don't need the outer ... and braces at all. Furthermore, from the initial declaration to the first following statement, there should be no branches, so it is safe to put exists on each rule, ie trace through each possible control-flow path individually. The revised semantic patch is attached. It terminates quite quickly.
You can add an option --timeout 120, or some other number of seconds. It will tell you if there are any timeouts, so you can see where there may be problems.
julia
-------------- next part --------------
@nonconst1@
type T;
identifier x;
position p;
@@
const T x at p = { ... };
@exists@
type T;
expression E;
statement S1, S2;
identifier x;
identifier f;
position p != nonconst1.p;
@@
T x at p = {
- .f = E,
};
... when != S1
++ x.f = E;
S2
@nonconst2@
type T;
identifier x;
position p;
@@
const T x at p = { ... };
@exists@
type T;
statement S1, S2;
identifier x;
position p != nonconst2.p;
@@
T x at p
- = {0}
;
... when != S1
when != x
++ _memclr(&x, sizeof(x));
S2
@nonconst3@
type T;
identifier x;
position p;
@@
const T x at p = { ... };
@exists@
type T;
statement S1, S2;
identifier x;
position p != nonconst3.p;
@@
T x at p
- = {}
;
... when != S1
when != x
++ _memclr(&x, sizeof(x));
S2
Quoted 3 lines
It seems that the control flow was quite complete in one function, and the when any at the end of the rule was taking a long time.
Oh, interesting, but that makes sense.
Quoted 6 lines
But actually you don't need the outer ... and braces at all.??Furthermore, from the initial declaration to the first following statement, there should be no branches,
so it is safe to put exists on each rule, ie trace through each possible control-flow path individually.??The revised semantic patch is attached. It terminates quite quickly.
Good point, thanks!
Quoted 3 lines
You can add an option --timeout 120, or some other number of seconds.??It will tell you if there are any timeouts, so you can see where there may be problems.
That's also useful, didn't know about that. :)
johannes