Skip to contentSkip to conversation

modifying initializers with spatch?

Cocci Johannes Berg 2 participants 21 messages Jan 26, 2017 – Jan 28, 2017

lore
Share & tools
lore
#0 lore raw

Hi,

I've been playing with C99 initializers a bit, what I'm trying to do is replace them with explicit memset()/assignments.

This works as expected:

@@
identifier x;
type t;
@@
?t x =
-{0};
+{};

This actually removes the braces, but not only if they're empty, and it leaves the inner content there without braces, which won't even compile afterwards:

@@
identifier x;
type t;
@@
?t x
- ={}
?;

e.g.:

- const u8 q[] = { 1, (const u8) '*', 0 };
+ const u8 q[] 1, (const u8) '*', 0;

I've also not managed to add any memset(&x, 0, sizeof(x)); in the same patch, especially since it should go after all other declarations.

Do you think this is possible at all?

johannes

#1 lore raw
Quoted 20 lines · Johannes Berg

Hi,

I've been playing with C99 initializers a bit, what I'm trying to do is replace them with explicit memset()/assignments.

This works as expected:

@@
identifier x;
type t;
@@
?t x =

(quoted diff) +1 -1
-{0};
+{};

This actually removes the braces, but not only if they're empty, and it leaves the inner content there without braces, which won't even compile afterwards:

Quoted 7 lines

@@
identifier x;
type t;
@@
?t x
- ={}
?;

I'm not sure what you are trying to do here. Remove only the empty ones?

Quoted 9 lines

e.g.:

(quoted diff) +1 -1
-	const u8 q[] = { 1, (const u8) '*', 0 };
+	const u8 q[] 1, (const u8) '*', 0;

I've also not managed to add any memset(&x, 0, sizeof(x)); in the same patch, especially since it should go after all other declarations.

Here you would want something like:

f(...) {
when != S
+ memset(...);
S1
...
}

S and S1 should be statements. If there could be multiple memsets at the beginning of a single function, use ++ instead of +.

julia

Quoted 9 lines

Do you think this is possible at all?

johannes
_______________________________________________
Cocci mailing list
Cocci at systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

#2 lore raw
Quoted 11 lines
Quoted 8 lines

@@
identifier x;
type t;
@@
?t x
- ={}
?;

I'm not sure what you are trying to do here.??Remove only the empty ones?

I actually tried something more like

@@
@@
identifier x;
type t;
@@
t x
- ={}
;
+memset(&x, 0, sizeof(x));

but that didn't really work either :)

Quoted 13 lines
Quoted 3 lines

I've also not managed to add any memset(&x, 0, sizeof(x)); in the same
patch, especially since it should go after all other declarations.

Here you would want something like:

f(...) {
? when != S
+ memset(...);
? S1
? ...
}

Quoted 2 lines

S and S1 should be statements.??If there could be multiple memsets at the beginning of a single function, use ++ instead of +.

Ok, so I should combine with the previous one before - do I need the f(...) or are just the braces enough? This could be in a nested block, after all.

johannes

#3 lore raw
Quoted 39 lines · Johannes Berg
Quoted 11 lines
Quoted 8 lines

@@
identifier x;
type t;
@@
?t x
- ={}
?;

I'm not sure what you are trying to do here.??Remove only the empty ones?

I actually tried something more like

@@
@@
identifier x;
type t;
@@
t x
- ={}
;
+memset(&x, 0, sizeof(x));

but that didn't really work either :)

Quoted 13 lines
Quoted 3 lines

I've also not managed to add any memset(&x, 0, sizeof(x)); in the same
patch, especially since it should go after all other declarations.

Here you would want something like:

f(...) {
? when != S
+ memset(...);
? S1
? ...
}

Quoted 6 lines
Quoted 2 lines

S and S1 should be statements.??If there could be multiple memsets at the beginning of a single function, use ++ instead of +.

Ok, so I should combine with the previous one before - do I need the f(...) or are just the braces enough? This could be in a nested block, after all.

Try for the braces. Maybe something like

{
... when != { ... }
T x
- = {0}
;
... when != S
when any
++ memset(...)
S1
... when any
}

#4 lore raw
Quoted 13 lines

Try for the braces.??Maybe something like

{
? ... when != { ... }
? T x
-?????= {0}
? ;
? ... when != S
??????when any
++ memset(...)
???S1
???... when any
}

Yep, thanks! (I'm on the phone, and didn't get a chance to try yet)

That works for anything that's actually "= {0}" today, but for I can't even match "= {}" properly, it seems?

johannes

#5 lore raw
Quoted 14 lines · Johannes Berg
Quoted 13 lines

Try for the braces.??Maybe something like

{
? ... when != { ... }
? T x
-?????= {0}
? ;
? ... when != S
??????when any
++ memset(...)
???S1
???... when any
}

Quoted 4 lines

Yep, thanks! (I'm on the phone, and didn't get a chance to try yet)

That works for anything that's actually "= {0}" today, but for I can't even match "= {}" properly, it seems?

The problem is that Coccinelle lets you put fewer field initializers than are actually available. The reasoning is that they are optional for fields, and the user doesn't really know how many will be present in each case. {} is the degenerate case where none are mentioned. But that should not apply to your case, 1) because you are removing the braces, and
2) because you aren't doing field initializations anyway. I'll have to look into it.

julia

#6 lore raw
Quoted 11 lines

The problem is that Coccinelle lets you put fewer field initializers than
are actually available.??The reasoning is that they are optional for fields, and the user doesn't really know how many will be present in each
case.??{} is the degenerate case where none are mentioned.??But that should not apply to your case, 1) because you are removing the braces, and
2) because you aren't doing field initializations anyway.??I'll have to look into it.

Ok, thanks.

I actually had a separate thought, that I could perhaps do something like

{
... when != { ... }
T x = {
- .field = E,
};
... when != S
when any
++ x.field = E;

To get rid of all the initializers, but (as I'm still on the phone) I haven't tried it yet :)

Btw, how was the ++ supposed to work? I had to run this more than once to actually make it catch multiple instances of = {0}. Also, in the case above, even if it works, the order of initializations might be inverted if I run the spatch multiple times, so not sure I should do that.

johannes

#7 lore raw
Quoted 28 lines · Johannes Berg
Quoted 11 lines

The problem is that Coccinelle lets you put fewer field initializers than
are actually available.??The reasoning is that they are optional for fields, and the user doesn't really know how many will be present in each
case.??{} is the degenerate case where none are mentioned.??But that should not apply to your case, 1) because you are removing the braces, and
2) because you aren't doing field initializations anyway.??I'll have to look into it.

Ok, thanks.

I actually had a separate thought, that I could perhaps do something like

{
... when != { ... }
T x = {
- .field = E,
};
... when != S
when any
++ x.field = E;

Quoted 8 lines

To get rid of all the initializers, but (as I'm still on the phone) I haven't tried it yet :)

Btw, how was the ++ supposed to work? I had to run this more than once to actually make it catch multiple instances of = {0}. Also, in the case above, even if it works, the order of initializations might be inverted if I run the spatch multiple times, so not sure I should do that.

Put a when any above the T x = line.

On the other hand, ++ makes no guarantee about the order in which things are generated.

To ensure the order, put the when any above the T x line, and drop the one below and run multiple times. Then each run will work on the last one and put it first, so they will come out in order.

julia

Quoted 3 lines

johannes

#17 lore raw
In reply to Johannes Berg

Partial progress: The github version will no longer match {} against { 0, 1, 2 }, but it persists in matching {} against { .a = 1, .b = 2 }, and will still remove the outer braces without removing the rest.

julia

#18 lore raw
Quoted 4 lines · Julia Lawall

Partial progress:??The github version will no longer match {} against { 0, 1, 2 }, but it persists in matching {} against { .a = 1, .b = 2 },
and will still remove the outer braces without removing the rest.

Out of curiosity: will it require some form of "..." to match something inside now?

We have a few things like this:

@@
identifier x;
identifer fn;
@@
my_time x = {
+#ifdef XYZ
.this_method_needs_xyz = fn,
+#endif
...
};

in backports etc. and I've always wondered why it didn't need (or even accept, IIRC) ... before the +#ifdef line.

johannes

#19 lore raw
Quoted 8 lines · Johannes Berg
Quoted 4 lines · Julia Lawall

Partial progress:??The github version will no longer match {} against { 0, 1, 2 }, but it persists in matching {} against { .a = 1, .b = 2 },
and will still remove the outer braces without removing the rest.

Out of curiosity: will it require some form of "..." to match something inside now?

No.

Quoted 14 lines

We have a few things like this:

@@
identifier x;
identifer fn;
@@
my_time x = {

(quoted diff) +2 -0
+#ifdef XYZ
 .this_method_needs_xyz = fn,
+#endif
...
 };
Quoted 2 lines

in backports etc. and I've always wondered why it didn't need (or even accept, IIRC) ... before the +#ifdef line.

It considers that the rule writer has no knowledge or control of the field names that are provided or the order in which they appear, so you can specify any subset of them. You only need ... when you want to add a new one at the beginning or at the end of the structure. The ... lets you specify beginning or end.

I want to allow you to remove the {} only when all of the field initializations are also removed. Some work though is required.

julia

#20 lore raw
Quoted 4 lines · Julia Lawall
Quoted 2 lines

Out of curiosity: will it require some form of "..." to match something inside now?

No.

Ok.

Quoted 5 lines

It considers that the rule writer has no knowledge or control of the field names that are provided or the order in which they appear, so you can specify any subset of them.??You only need ... when you want to add a new one at the beginning or at the end of the structure.??The ... lets you specify beginning or end.

Interesting. I thought I needed the ... at the end, but that may have been a case of "let's see if I need both - doesn't work - remove one - ok works now" :)

Quoted 2 lines

I want to allow you to remove the {} only when all of the field initializations are also removed.??Some work though is required.

That'd make sense.

I'm very grateful for this. I'm dealing with an awful compiler that emits stupid code for C99 initializers, so I'm thinking of using what we were discussing here to remove them all at build time. I'm going to be travelling (again) next week, but will give it a try later.

Thanks a lot!

johannes

#8 lore raw
In reply to Julia Lawall
Quoted 5 lines
Quoted 4 lines

To get rid of all the initializers, but (as I'm still on the phone) I haven't tried it yet :)

Quoted 8 lines

Put a when any above the T x = line.

On the other hand, ++ makes no guarantee about the order in which things are generated.

To ensure the order, put the when any above the T x line, and drop the one below and run multiple times.??Then each run will work on the last one and put it first, so they will come out in order.

Finally got a chance to try this.

I can't seem to get the ordering right. I'm trying this:

@@
type T;
expression E;
statement S1, S2;
identifier x;
identifier f;
@@
?{
... when != { ... }
????when any
?T x = {
-??.f = E,
?};
... when != S1
+ x.f = E;
?S2
?...
?}

on a very simple file:

int main()

{
	struct foo bar = {
		.y = 8,
		.x = 7,
	};

	printf("%d\n", bar.x);
	printf("%d\n", bar.y);
}

but it doesn't end well:

$ spatch??--sp-file /tmp/init.spatch /tmp/test.c
init_defs_builtins: /usr/lib/coccinelle/standard.h
HANDLING: /tmp/test.c
?????
previous modification:

? <<< x.f = E;
CONTEXT
According to environment 3:
???rule starting on line 1.E -> 8
???rule starting on line 1.f -> id y
???rule starting on line 1.x -> id bar

current modification:

? <<< x.f = E;
CONTEXT
According to environment 3:
???rule starting on line 1.E -> 7
???rule starting on line 1.f -> id x
???rule starting on line 1.x -> id bar

Fatal error: exception Failure("rule starting on line 1: already tagged token:\nC code context\nFile \"/tmp/test.c\", line 8, column 1,??charpos = 59\n????around = 'printf', whole content = \tprintf(\"%d\\n\", bar.x);")

If I use ++ instead of +, it still replaces all of them, so I can't run multiple times (usefully anyway)

OTOH, I haven't found any place where the order actually matters (in the code base I'm interested in), and although it seems that it theoretically could matter, I'm not sure I really need to solve that problem - although I'm thinking I might want to run this not just once but before each compile, to keep the code neater.

(Note that I also haven't upgraded to the git version yet, TBD)

johannes

#9 lore raw
Quoted 74 lines · Johannes Berg
Quoted 5 lines
Quoted 4 lines

To get rid of all the initializers, but (as I'm still on the phone) I haven't tried it yet :)

Quoted 8 lines

Put a when any above the T x = line.

On the other hand, ++ makes no guarantee about the order in which things are generated.

To ensure the order, put the when any above the T x line, and drop the one below and run multiple times.??Then each run will work on the last one and put it first, so they will come out in order.

Finally got a chance to try this.

I can't seem to get the ordering right. I'm trying this:

@@
type T;
expression E;
statement S1, S2;
identifier x;
identifier f;
@@
?{
... when != { ... }
????when any
?T x = {
-??.f = E,
?};
... when != S1
+ x.f = E;
?S2
?...
?}

on a very simple file:

int main()

{
	struct foo bar = {
		.y = 8,
		.x = 7,
	};

	printf("%d\n", bar.x);
	printf("%d\n", bar.y);
}

but it doesn't end well:

$ spatch??--sp-file /tmp/init.spatch /tmp/test.c
init_defs_builtins: /usr/lib/coccinelle/standard.h
HANDLING: /tmp/test.c
?????
previous modification:

? <<< x.f = E;
CONTEXT
According to environment 3:
???rule starting on line 1.E -> 8
???rule starting on line 1.f -> id y
???rule starting on line 1.x -> id bar

current modification:

? <<< x.f = E;
CONTEXT
According to environment 3:
???rule starting on line 1.E -> 7
???rule starting on line 1.f -> id x
???rule starting on line 1.x -> id bar

Quoted 4 lines

Fatal error: exception Failure("rule starting on line 1: already tagged token:\nC code context\nFile \"/tmp/test.c\", line 8, column 1,??charpos = 59\n????around = 'printf', whole content = \tprintf(\"%d\\n\", bar.x);")

Yeah, it doesn't work. There is no way to get the fields out of the structure one at a time. So ++ seems like the only option. The suggested organization would work for when there is more than one structure definition in the function, but doesn't help for multiple fields within a structure.

julia

Quoted 15 lines

If I use ++ instead of +, it still replaces all of them, so I can't run multiple times (usefully anyway)

OTOH, I haven't found any place where the order actually matters (in the code base I'm interested in), and although it seems that it theoretically could matter, I'm not sure I really need to solve that problem - although I'm thinking I might want to run this not just once but before each compile, to keep the code neater.

(Note that I also haven't upgraded to the git version yet, TBD)

johannes

#10 lore raw
Quoted 19 lines · Julia Lawall
Quoted 10 lines

@@
type T;
expression E;
statement S1, S2;
identifier x;
identifier f;
@@
?{
... when != { ... }
????when any

Quoted 8 lines

?T x = {
-??.f = E,
?};
... when != S1
+ x.f = E;
?S2
?...
?}

Quoted 2 lines

Yeah, it doesn't work.??There is no way to get the fields out of the structure one at a time.??So ++ seems like the only option.??

Right. Like I said, I don't think it matters much for me right now. I also found that most of these are actually in the part of the code that I can change and don't really need to touch at build time, although it'd be nice to be able to write the code in the more canonical C99 style and still do it at build time.

Quoted 5 lines

The suggested
organization would work for when there is more than one structure definition in the function, but doesn't help for multiple fields within a
structure.

Right, it easily works for multiple structures, and it does even seem to keep the order of variables, even if it doesn't keep the order of the fields within each variable (but sorts them alphabetically instead, apparently)

The git version should now actually be sufficient for me, since you said it will no longer remove braces from "= { 1, 2, 3}", and I'm moving all the initializers anyway like indicated above, and translating "={0}" to "={}" as well. I'll need to build & try it, but if it fails on some obscure cases and removes the braces anyway the compile will fail so I'm not really worried :)

johannes

#11 lore raw
Quoted 41 lines · Johannes Berg
Quoted 19 lines · Julia Lawall
Quoted 10 lines

@@
type T;
expression E;
statement S1, S2;
identifier x;
identifier f;
@@
?{
... when != { ... }
????when any

Quoted 8 lines

?T x = {
-??.f = E,
?};
... when != S1
+ x.f = E;
?S2
?...
?}

Quoted 2 lines

Yeah, it doesn't work.??There is no way to get the fields out of the structure one at a time.??So ++ seems like the only option.??

Right. Like I said, I don't think it matters much for me right now. I also found that most of these are actually in the part of the code that I can change and don't really need to touch at build time, although it'd be nice to be able to write the code in the more canonical C99 style and still do it at build time.

Quoted 5 lines

The suggested
organization would work for when there is more than one structure definition in the function, but doesn't help for multiple fields within a
structure.

Right, it easily works for multiple structures, and it does even seem to keep the order of variables, even if it doesn't keep the order of the fields within each variable (but sorts them alphabetically instead, apparently)

Quoted 6 lines

The git version should now actually be sufficient for me, since you said it will no longer remove braces from "= { 1, 2, 3}", and I'm moving all the initializers anyway like indicated above, and translating "={0}" to "={}" as well. I'll need to build & try it, but if it fails on some obscure cases and removes the braces anyway the compile will fail so I'm not really worried :)

Ok, sounds good. Let me know if you encounter further problems.

julia

#12 lore raw
Quoted 2 lines · Julia Lawall

Ok, sounds good.??Let me know if you encounter further problems.

You jinxed it ;-)
Now with the git version.

struct foo v = {
	.nested = {
		.x = 7,
		.y = 8,
	},
};

still broke, because of course my initializer replacement rule didn't actually find it, and then the bug hit and removed the {} anyway even though they weren't empty :)

Similarly,

char * foo[] = {
	[1] = "name1",
	[2] = "name2",
};

was butchered quite a bit :)

The latter case - though not with strings - actually does seem to be present in the code I care about, but I can probably just replace it manually first.

Interestingly, this whole exercise is obviously not even a good idea for static variables - need to see if I can exclude those somehow, first naive attempts didn't work :)

johannes

#13 lore raw
In reply to Julia Lawall
Quoted 2 lines · Julia Lawall

Ok, sounds good.??Let me know if you encounter further problems.

I've now ended up with the following overall:
https://p.sipsolutions.net/15e6265ea961d9a6.txt

Interestingly, that takes minutes to run on this file:
https://w1.fi/cgit/hostap/tree/wpa_supplicant/dbus/dbus_new_handlers_p2p.c

Not that it matters much to me, but I can't really explain it, all other files I found finish quickly.

johannes

#15 lore raw
Quoted 5 lines · Johannes Berg
Quoted 4 lines · Johannes Berg
Quoted 2 lines · Julia Lawall

Ok, sounds good.??Let me know if you encounter further problems.

Quoted 8 lines
Quoted 6 lines

I've now ended up with the following overall:
https://p.sipsolutions.net/15e6265ea961d9a6.txt

Interestingly, that takes minutes to run on this file:
https://w1.fi/cgit/hostap/tree/wpa_supplicant/dbus/dbus_new_handlers_
p2p.c

Well, it's been 20 minutes - I'm giving up :)

It seems that the control flow was quite complete in one function, and the when any at the end of the rule was taking a long time. But actually you don't need the outer ... and braces at all. Furthermore, from the initial declaration to the first following statement, there should be no branches, so it is safe to put exists on each rule, ie trace through each possible control-flow path individually. The revised semantic patch is attached. It terminates quite quickly.

You can add an option --timeout 120, or some other number of seconds. It will tell you if there are any timeouts, so you can see where there may be problems.

julia
-------------- next part --------------
@nonconst1@
type T;
identifier x;
position p;
@@
const T x at p = { ... };

@exists@
type T;
expression E;
statement S1, S2;
identifier x;
identifier f;
position p != nonconst1.p;
@@

T x at p = {
- .f = E,
};
... when != S1
++ x.f = E;
S2

@nonconst2@
type T;
identifier x;
position p;
@@
const T x at p = { ... };

@exists@
type T;
statement S1, S2;
identifier x;
position p != nonconst2.p;
@@

T x at p
- = {0}
;
... when != S1

    when != x

++ _memclr(&x, sizeof(x));
S2

@nonconst3@
type T;
identifier x;
position p;
@@
const T x at p = { ... };

@exists@
type T;
statement S1, S2;
identifier x;
position p != nonconst3.p;
@@

T x at p
- = {}
;
... when != S1

    when != x

++ _memclr(&x, sizeof(x));
S2

#16 lore raw
Quoted 3 lines

It seems that the control flow was quite complete in one function, and the when any at the end of the rule was taking a long time.

Oh, interesting, but that makes sense.

Quoted 6 lines

But actually you don't need the outer ... and braces at all.??Furthermore, from the initial declaration to the first following statement, there should be no branches,
so it is safe to put exists on each rule, ie trace through each possible control-flow path individually.??The revised semantic patch is attached. It terminates quite quickly.

Good point, thanks!

Quoted 3 lines

You can add an option --timeout 120, or some other number of seconds.??It will tell you if there are any timeouts, so you can see where there may be problems.

That's also useful, didn't know about that. :)

johannes