Using or for structural #ifdefs
An examle or SmPL patch (from upstream Coccinelle tests/orexp.cocci):
@@
expression E, F;
@@
(
- foo(E)
+ 4
|
- bar(F)
+ 4
)
This works nicely for code within functions, for backporting we at times need to use or clauses for code outside of functions. We can avoid the or statements when backporting but if we could use them it would provide the ability to produce unified diffs which are much more readible and would resemble more what humans would try to write. I'll explain with an example.
So this works, one can for example apply this to the file on linux-next:
drivers/media/v4l2-core/videobuf2-dma-contig.c:
@ vb2_mem_ops @
identifier s, get_dma, map_dma, unmap_dma, attach_dma, detach_dma; @@
struct vb2_mem_ops s = {
.attach_dmabuf = attach_dma,
.detach_dmabuf = detach_dma,
.map_dmabuf = map_dma,
.unmap_dmabuf = unmap_dma,
.get_dmabuf = get_dma,
};
@ mod_get_dma depends on vb2_mem_ops @ identifier vb2_mem_ops.get_dma, vb2_mem_ops.attach_dma; @@
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,5,0)
get_dma(...)
{
...
}
+#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(3,5,0) */
@ mod_op_get_dma depends on vb2_mem_ops @
identifier s, vb2_mem_ops.get_dma;
@@
struct vb2_mem_ops s = {
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,5,0)
.get_dmabuf = get_dma,
+#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(3,5,0) */ };
This SmPL patch uses the needle-in-the-haystack strategy to first localize the backport we want to apply, but also uses this to the subsequent rules to #ifdef them out. The needle-in-the-haystack solution works for me if and only if all structural definitions are delcared, grammatically fortunately the order does not matter here though.
Can we use or clauses to make some of these optional? I tried but failed.
The same would apply to the other rules, right now I'd have to define a rule in order to be able for example to #idef out all routines as with rule mod_get_dma. It'd be for example convenient to just use or clauses in one rule. There's two things that could be done to help here, one is for Coccinelle to see the additional contigious #idefs and if they match assume it can skip an #ifdef and just join the two statements. That is to make something like this:
#ifdef FOO
static int foo(void)
{
return 1;
}#endif /* FOO */
#ifdef FOO
static int bar(void)
{
return 2;
}#endif /* FOO */
just be:
#ifdef FOO
static int foo(void)
{
return 1;
}static int bar(void)
{
return 2; }
#endif /* FOO */
This can be accomplished by 3 methods I think:
a) Having Coccinelle infer when it can do this based on code it is generating / modifying. This might be rather complex to support I think.
b) Having Coccinelle allow SmPL rules to let you write structural or's on on both struct declarations or function delcarations. This seems generally useful however I am not sure of its feasability.
c) Having Coccinelle let you declare variables that Coccinelle can use to let you group metavariables so that you can shorten the number of rules you have two write to #idef out if the #idef is generic across the metavariables, and perhaps only if this feature is used allow you to do the or'ing on the group variables. For example:
@ mod_all_dma_ops depends on vb2_mem_ops @
identifier vb2_mem_ops.get_dma, vb2_mem_ops.map_dma, vb2_mem_ops.unmap_dma, vb2_mem_ops.attach_dma, vb2_mem_ops.detach_dma; group identifier any_vb2_mem_ops = { get_dma | map_dma | unmap_dma | attach_dma | detach_dma };
@@
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,5,0)
any_vb2_mem_ops(...)
{
...
}
+#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(3,5,0) */
c) might be more complex than I believe, but it sure would be nice to have (TM).
Luis
Quoted 15 lines · Luis R. Rodriguez
An examle or SmPL patch (from upstream Coccinelle
tests/orexp.cocci):@@
expression E, F;
@@(
- foo(E)
+ 4
|
- bar(F)
+ 4
)
Quoted 6 lines
This works nicely for code within functions, for backporting we at times need to use or clauses for code outside of functions. We can avoid the or statements when backporting but if we could use them it would provide the ability to produce unified diffs which are much more readible and would resemble more what humans would try to write. I'll explain with an example.
I don't think that or is the problem. If you have:
+ foo();
(
- x();
+ y();
|
- m();
+ n();
)
+ foo2();
the disjunction will still only match one statement. Of course, you could put between the calls to foo and foo2 something that would match two statements, and then that would match two statements. Or something that would match one or two statements.
The problem for structure field initializations would seem to be that they are unordered. That is, if you put an initialization for x and then an initialization for y, the C code might have x initialized before y or vice versa, and with arbitrary things between them. So the #ifdef/#endif would very likely come out in undesirable places.
The problem for functions is that Coccinelle only sees one at a time. It has no idea whether one function is before or after another, and whether they are contiguous or there is something between them.
Maybe this would be better done with some tool that is separate from Coccinelle. There could be a danger of modifying #ifdefs that were in the original code. If that would be a bad thing, then the tool could take the old and new code as inputs, and only eliminate ifdefs that are unique to the new code.
julia
Quoted 118 lines
So this works, one can for example apply this to the file on linux-next:
drivers/media/v4l2-core/videobuf2-dma-contig.c:@
vb2_mem_ops@
identifier s,get_dma,map_dma,unmap_dma,attach_dma,detach_dma; @@struct
vb2_mem_opss = {
.attach_dmabuf =attach_dma,
.detach_dmabuf =detach_dma,
.map_dmabuf =map_dma,
.unmap_dmabuf =unmap_dma,
.get_dmabuf =get_dma,
};@
mod_get_dmadepends onvb2_mem_ops@ identifier vb2_mem_ops.get_dma, vb2_mem_ops.attach_dma; @@(quoted diff) +2
-0+#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,5,0) get_dma(...) { } +#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(3,5,0) */@
mod_op_get_dmadepends onvb2_mem_ops@
identifier s, vb2_mem_ops.get_dma;
@@struct
vb2_mem_opss = {(quoted diff) +2
-0+#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,5,0) .get_dmabuf = get_dma, +#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(3,5,0) */};
This SmPL patch uses the needle-in-the-haystack strategy to first localize the backport we want to apply, but also uses this to the subsequent rules to #ifdef them out. The needle-in-the-haystack solution works for me if and only if all structural definitions are delcared, grammatically fortunately the order does not matter here though.
Can we use or clauses to make some of these optional? I tried but failed.
The same would apply to the other rules, right now I'd have to define a rule in order to be able for example to #idef out all routines as with rule
mod_get_dma. It'd be for example convenient to just use or clauses in one rule. There's two things that could be done to help here, one is for Coccinelle to see the additional contigious #idefs and if they match assume it can skip an #ifdef and just join the two statements. That is to make something like this:#ifdef FOO
static int foo(void){ return 1; }#endif /* FOO */
#ifdef FOO
static int bar(void){ return 2; }#endif /* FOO */
just be:
#ifdef FOO
static int foo(void){ return 1; }static int bar(void)
{return 2; }#endif /* FOO */
This can be accomplished by 3 methods I think:
a) Having Coccinelle infer when it can do this based on code it is generating / modifying. This might be rather complex to support I think.
b) Having Coccinelle allow SmPL rules to let you write structural or's on on both struct declarations or function delcarations. This seems generally useful however I am not sure of its feasability.
c) Having Coccinelle let you declare variables that Coccinelle can use to let you group metavariables so that you can shorten the number of rules you have two write to #idef out if the #idef is generic across the metavariables, and perhaps only if this feature is used allow you to do the or'ing on the group variables. For example:
@
mod_all_dma_opsdepends onvb2_mem_ops@
identifier vb2_mem_ops.get_dma, vb2_mem_ops.map_dma, vb2_mem_ops.unmap_dma, vb2_mem_ops.attach_dma, vb2_mem_ops.detach_dma; group identifierany_vb2_mem_ops= {get_dma|map_dma|unmap_dma|attach_dma|detach_dma};
@@(quoted diff) +2
-0+#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,5,0) any_vb2_mem_ops(...) { } +#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(3,5,0) */c) might be more complex than I believe, but it sure would be nice to have (TM).
Luis
_______________________________________________
Cocci mailing list
Cocci at systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci
Quoted 41 lines · Julia Lawall
Quoted 15 lines · Luis R. Rodriguez
An examle or SmPL patch (from upstream Coccinelle
tests/orexp.cocci):@@
expression E, F;
@@(
- foo(E)
+ 4
|
- bar(F)
+ 4
)Quoted 6 lines
This works nicely for code within functions, for backporting we at times need to use or clauses for code outside of functions. We can avoid the or statements when backporting but if we could use them it would provide the ability to produce unified diffs which are much more readible and would resemble more what humans would try to write. I'll explain with an example.
I don't think that or is the problem. If you have:
+
foo();
(
-x();
+y();
|
-m();
+n();
)
+foo2();the disjunction will still only match one statement. Of course, you could put between the calls to foo and foo2 something that would match two statements, and then that would match two statements. Or something that would match one or two statements.
Quoted 5 lines
The problem for structure field initializations would seem to be that they are unordered. That is, if you put an initialization for x and then an initialization for y, the C code might have x initialized before y or vice versa, and with arbitrary things between them. So the #ifdef/#endif would very likely come out in undesirable places.
Doesn't the original developer have the responsibility over the order? So long as we proceed on the code in order and perform the operations asked atomically its not clear to me how this would be an issue.
Quoted 3 lines
The problem for functions is that Coccinelle only sees one at a time. It has no idea whether one function is before or after another, and whether they are contiguous or there is something between them.
Understood, similarly -- it would seem to me fair for Coccinelle to trust the order already in place and just address disjunctions atomically as they are found top - down.
I may have misunderstood the issue though.
Quoted 5 lines
Maybe this would be better done with some tool that is separate from Coccinelle. There could be a danger of modifying #ifdefs that were in the original code. If that would be a bad thing, then the tool could take the old and new code as inputs, and only eliminate ifdefs that are unique to the new code.
A tool to optimize patches seems like a super useful tool. Grammatically however I also think it'd be useful to be able to use disjunctions on functions / structur declarations. The last item (c) on grouping metavariables would also be extremey useful as it would shorten the length of the rules and I think also enable more readible SmPL patches.
Luis
Quoted 3 lines
Doesn't the original developer have the responsibility over the order? So long as we proceed on the code in order and perform the operations asked atomically its not clear to me how this would be an issue.
In the case where you add new things, you control the order, and I think there is actually no problem. You just have to check that both things you considered adding were actually added. For example if you have one rule that might add an initializer A at the end of a structure, and then another rule that might add an initialier B in this position, then you now A is before B. If you now that both rules matched then you can just add an #ifdef before A and an #endif after B. Coccinelle doesn't even care if they are added in an unbalanced way, ie one rule could add the #ifdef and another could add the #endif.
Similarly for function definitions.
There problem comes if you eg just a replace an existing field, rather than explicitly adding a new one at the end of the structure. Or put the new variant next to an existing one. Or, in terms of functions, if you add a definition after one function and before another one. In that case, Coccinelle doesn't make available any information about the relative position of the new things. You could assume that certain things were always defined together in the original code, but it seems risky.
Quoted 16 lines
Quoted 3 lines
The problem for functions is that Coccinelle only sees one at a time. It has no idea whether one function is before or after another, and whether they are contiguous or there is something between them.
Understood, similarly -- it would seem to me fair for Coccinelle to trust the order already in place and just address disjunctions atomically as they are found top - down.
I may have misunderstood the issue though.
Quoted 5 lines
Maybe this would be better done with some tool that is separate from Coccinelle. There could be a danger of modifying #ifdefs that were in the original code. If that would be a bad thing, then the tool could take the old and new code as inputs, and only eliminate ifdefs that are unique to the new code.
Quoted 5 lines
A tool to optimize patches seems like a super useful tool. Grammatically however I also think it'd be useful to be able to use disjunctions on functions / structur declarations. The last item (c) on grouping metavariables would also be extremey useful as it would shorten the length of the rules and I think also enable more readible SmPL patches.
It would certainly be useful to have disjnctions on more kinds of things. This is just a development time/priority issue.
I'm not sure to understand the grouping metavariable idea. People who like regular expressions would just write a regular expression for what you propose. Since I don't particularly like regular expressions, and since I know that Coccinelle doesn't use them for indexing, I would rather write:
\(get_dma \| map_dma \| unmap_dma \| attach_dma \| detach_dma\)
That may not work at the moment in the context in which it is wanted. But that is just another or problem, not a metavariable problem.
julia
Quoted 14 lines · Julia Lawall
Quoted 3 lines
Doesn't the original developer have the responsibility over the order? So long as we proceed on the code in order and perform the operations asked atomically its not clear to me how this would be an issue.
In the case where you add new things, you control the order, and I think there is actually no problem. You just have to check that both things you considered adding were actually added. For example if you have one rule that might add an initializer A at the end of a structure, and then another rule that might add an initialier B in this position, then you now A is before B. If you now that both rules matched then you can just add an #ifdef before A and an #endif after B. Coccinelle doesn't even care if they are added in an unbalanced way, ie one rule could add the #ifdef and another could add the #endif.
Quoted 1 line
Similarly for function definitions.
OK one typical backport case we tend to have is multiple, maybe optional, initializers on the struct and prior to this multiple, depending on the struct initializers present, function definitions. The ability to ifdef grouped items then would only work if the entire set of struct initializers were present and I think contiguous but the hard thing about this lies in that the functions we would want to #idef out are group'd and optional and go first prior to the struct initializer.
Quoted 3 lines
There problem comes if you eg just a replace an existing field, rather than explicitly adding a new one at the end of the structure. Or put the new variant next to an existing one.
I believe I had used
struct foo {
...+ int a;
};
before to hint to add at the top while using the ... at the end to the bottom, I typically to add to the top to avoid issues with possible uses in code of the end variable being used for a flexible array which requires it to be declared at the end so adding something new at the end would break that flexible array.
I haven't yet had the need, at leaest for backports, to care where exactly to add *new* fields.
Quoted 5 lines
Or, in terms of functions, if you add a
definition after one function and before another one. In that case, Coccinelle doesn't make available any information about the relative position of the new things. You could assume that certain things were always defined together in the original code, but it seems risky.
I see, I'd expect the rule writer to be aware of this risk, but do understand the possible issue that could arise.
Quoted 16 lines
Quoted 16 lines
Quoted 3 lines
The problem for functions is that Coccinelle only sees one at a time. It has no idea whether one function is before or after another, and whether they are contiguous or there is something between them.
Understood, similarly -- it would seem to me fair for Coccinelle to trust the order already in place and just address disjunctions atomically as they are found top - down.
I may have misunderstood the issue though.
Quoted 5 lines
Maybe this would be better done with some tool that is separate from Coccinelle. There could be a danger of modifying #ifdefs that were in the original code. If that would be a bad thing, then the tool could take the old and new code as inputs, and only eliminate ifdefs that are unique to the new code.
Quoted 8 lines
Quoted 5 lines
A tool to optimize patches seems like a super useful tool. Grammatically however I also think it'd be useful to be able to use disjunctions on functions / structur declarations. The last item (c) on grouping metavariables would also be extremey useful as it would shorten the length of the rules and I think also enable more readible SmPL patches.
It would certainly be useful to have disjnctions on more kinds of things. This is just a development time/priority issue.
Got it.
Quoted 1 line
I'm not sure to understand the grouping metavariable idea.
The grouping idea would enable the disjuction expressed as you note below within the rule declaration of metavariables instead of on the rule itself. That would enable avoiding regexp or escaping disjuctions as below by just making the associatin of one metavariable to a series of metavariables delcared under one name.
Quoted 10 lines
People who
like regular expressions would just write a regular expression for what you propose. Since I don't particularly like regular expressions, and since I know that Coccinelle doesn't use them for indexing, I would rather write:\(get_dma \|
map_dma\|unmap_dma\|attach_dma\| detach_dma\)That may not work at the moment in the context in which it is wanted. But that is just another or problem, not a metavariable problem.
I see, thanks!
Luis