Parse error with genl-const.cocci
Hello,
I'd like to say thank you for this very useful project. I am relatively new to kernel development, so it would be impossible for me to sort out the "collateral evolutions" and history of the linux internals without backports.
I am trying to backport the rtl8192cu driver from linux v4.14-rc4 to an Android 5.11 system running linux kernel v3.10. My development system is Linux Mint 17.1. I chose v4.14-rc4 because that's the latest tag in backports. My backports repo and linux-next repo are checked out to v4.14-rc4-1 and v4.14-rc4, respectively.
I am getting an error when I try to create a backport package:
backports $ ./gentree.py --clean --verbose --git-revision v4.14-rc4 ../linux-next/ ../linux-backport/ Get original source files from git ...
Applying patches from patches to ../linux-backport/ ...
Applying patch 0000-upstream-backport-changes/0005-uapi-compromise/mwifiex.patch [...snip lots of good patches here...]
Applying SmPL patch 0027-genl-const/genl-const.cocci
Failed to process SmPL patch 0027-genl-const/genl-const.cocci with 1
Getting the output of pycocci:
backports $ ./devel/pycocci -j 1 -v ./patches/0027-genl-const/genl-const.cocci ../linux-backport/
spatch --sp-file ./patches/0027-genl-const/genl-const.cocci --in-place --recursive-includes --relax-include-path --use-coccigrep --timeout 120 --dir ../linux-backport/ Failed to apply changes from genl-const.cocci
Specific log output from change that failed using genl-const.cocciinit_defs_builtins: /usr/local/bin/../lib/coccinelle/standard.h
meta: parse error:
File "./patches/0027-genl-const/genl-const.cocci", line 2, column 10, charpos = 13
around = '__genl_const',
whole content = attribute __genl_const;
Full log using genl-const.cocciinit_defs_builtins: /usr/local/bin/../lib/coccinelle/standard.h
meta: parse error:
File "./patches/0027-genl-const/genl-const.cocci", line 2, column 10, charpos = 13 around = '__genl_const',
whole content = attribute __genl_const;
[...snip callstack...]
This is very similar to this problem:
<https://www.spinics.net/lists/backports/msg04216.html>
The conclusion of that thread is that the --integrate workflow is no longer supported.
However, I am not using --integrate, and I am still getting errors about parsing '__genl_const'. Is that a bug in the patch or possibly a regression in spatch? I've tried v4.14-rc2 also, but nothing older.
This page has a lot of great example info, and suggested that it could be a problem with my version of spatch. Specifically, that I was missing the menhir and libmenhir-ocaml-dev packages when building Coccinelle and a rebuild should help.
<http://stonyslp.blogspot.com/2017/08/backport-driver.html>
Unfortunately, I still have the same errors after running make distclean and rebuilding.
I have:
coccinelle $ git describe --tags
1.0.6-365-gd73c6e6
coccinelle $ spatch --version
spatch version 1.0.6-00365-gd73c6e6 compiled with OCaml version 4.01.0 Flags passed to the configure script: [none]
OCaml scripting support: yes
Python scripting support: yes
Syntax of regular expresssions: PCRE
I found this patch which describes how older versions can't handle the __genl_const changes (20150209):
<https://www.spinics.net/lists/backports/msg03223.html>
But, I also see commit 748116c8 from shortly after (20150507) which introduced genl-const.cocci and says only spatch 1.0.0-rc23 is needed.
I feel like I'm close to figuring this out, but would appreciate any help or pointers.
Thank you,
Remington Furman
--
To unsubscribe from this list: send the line "unsubscribe backports" in
+cocci list
Hi,
Looks like you made a really deep investigation here - awesome, thanks for doing that!
Quoted 9 lines · Remington Furman
Full log using
genl-const.cocciinit_defs_builtins:/usr/local/bin/../lib/coccinelle/standard.h
meta: parse error:File "./patches/0027-genl-const/genl-const.cocci", line 2, column 10,charpos = 13
around = '__genl_const',
whole content = attribute__genl_const;
[...snip callstack...]
FWIW, the patch is pretty simple:
@@
attribute __genl_const;
@@
(
-const struct genl_multicast_group
+__genl_const struct genl_multicast_group
|
-const struct genl_ops
+__genl_const struct genl_ops
)
(yes, that's the full patch)
Quoted 7 lines
This is very similar to this problem:
<https://www.spinics.net/lists/backports/msg04216.html>
The conclusion of that thread is that the --integrate workflow is no longer supported.However, I am not using --integrate, and I am still getting errors about parsing '
__genl_const'.
Yeah this --integrate is unrelated to the problem - the problem is that spatch can't parse this patch (any more).
Quoted 2 lines
Is that a bug in the patch
or possibly a regression in spatch?
I think it's a regression in spatch.
Quoted 2 lines
I've tried v4.14-rc2 also,
but nothing older.
That's a kernel version so not really relevant - spatch doesn't even attempt to apply the changes since it can't parse them.
Quoted 5 lines
This page has a lot of great example info, and suggested that it could be a problem with my version of spatch. Specifically, that I was missing the menhir and libmenhir-ocaml-dev packages when building Coccinelle and a rebuild should help.
<http://stonyslp.blogspot.com/2017/08/backport-driver.html>
That I don't know - I've never built spatch myself.
Quoted 12 lines
Unfortunately, I still have the same errors after running make distclean and rebuilding.
I have:
coccinelle $ git describe --tags
1.0.6-365-gd73c6e6
coccinelle $ spatch --version
spatch version 1.0.6-00365-gd73c6e6 compiled with OCaml version 4.01.0 Flags passed to the configure script: [none]
OCaml scripting support: yes
Python scripting support: yes
Syntax of regular expresssions: PCRE
That seems reasonable as far as configuration is concerned.
Quoted 7 lines
I found this patch which describes how older versions can't handle the
__genl_constchanges (20150209):
<https://www.spinics.net/lists/backports/msg03223.html>But, I also see commit
748116c8from shortly after (20150507) which introducedgenl-const.cocciand says only spatch 1.0.0-rc23 is needed.
Right.
Quoted 2 lines
I feel like I'm close to figuring this out, but would appreciate any help or pointers.
Yeah I think you're pretty close.
Can you try obtaining/building spatch 1.0.4? That definitely works for me, so I think there's probably a regression in spatch somewhere between 1.0.4 and the version that you're running.
johannes
Quoted 6 lines · Johannes Berg
+cocci list
Hi,
Looks like you made a really deep investigation here - awesome, thanks for doing that!
Hello,
The attribute metavariable declaration has become attribute name, to be more like eg declarer name, iterator name, etc.
The handling of attributes has also been extended so that you can actually match against them in some cases, such as on function declarations. On the oher hand, nothing was done to allow attributes in front of types that are alone.
I can try to fix this. In the short term, perhaps using 1.0.4 is acceptable.
julia
Quoted 99 lines · Remington Furman
Quoted 9 lines
Full log using
genl-const.cocciinit_defs_builtins:/usr/local/bin/../lib/coccinelle/standard.h
meta: parse error:File "./patches/0027-genl-const/genl-const.cocci", line 2, column 10,charpos = 13
around = '__genl_const',
whole content = attribute__genl_const;
[...snip callstack...]FWIW, the patch is pretty simple:
@@
attribute__genl_const;
@@
((quoted diff) +1
-1-const struct genl_multicast_group +__genl_const struct genl_multicast_group|
(quoted diff) +1
-1-const struct genl_ops +__genl_const struct genl_ops)
(yes, that's the full patch)
Quoted 7 lines
This is very similar to this problem:
<https://www.spinics.net/lists/backports/msg04216.html>
The conclusion of that thread is that the --integrate workflow is no longer supported.However, I am not using --integrate, and I am still getting errors about parsing '
__genl_const'.Yeah this --integrate is unrelated to the problem - the problem is that spatch can't parse this patch (any more).
Quoted 2 lines
Is that a bug in the patch
or possibly a regression in spatch?I think it's a regression in spatch.
Quoted 2 lines
I've tried v4.14-rc2 also,
but nothing older.That's a kernel version so not really relevant - spatch doesn't even attempt to apply the changes since it can't parse them.
Quoted 5 lines
This page has a lot of great example info, and suggested that it could be a problem with my version of spatch. Specifically, that I was missing the menhir and libmenhir-ocaml-dev packages when building Coccinelle and a rebuild should help.
<http://stonyslp.blogspot.com/2017/08/backport-driver.html>That I don't know - I've never built spatch myself.
Quoted 12 lines
Unfortunately, I still have the same errors after running make distclean and rebuilding.
I have:
coccinelle $ git describe --tags
1.0.6-365-gd73c6e6
coccinelle $ spatch --version
spatch version 1.0.6-00365-gd73c6e6 compiled with OCaml version 4.01.0 Flags passed to the configure script: [none]
OCaml scripting support: yes
Python scripting support: yes
Syntax of regular expresssions: PCREThat seems reasonable as far as configuration is concerned.
Quoted 7 lines
I found this patch which describes how older versions can't handle the
__genl_constchanges (20150209):
<https://www.spinics.net/lists/backports/msg03223.html>But, I also see commit
748116c8from shortly after (20150507) which introducedgenl-const.cocciand says only spatch 1.0.0-rc23 is needed.Right.
Quoted 2 lines
I feel like I'm close to figuring this out, but would appreciate any help or pointers.
Yeah I think you're pretty close.
Can you try obtaining/building spatch 1.0.4? That definitely works for me, so I think there's probably a regression in spatch somewhere between 1.0.4 and the version that you're running.
johannes
_______________________________________________
Cocci mailing list
Cocci at systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci
Hi Julia,
Quoted 3 lines · Julia Lawall
The attribute metavariable declaration has become attribute name, to be more like eg declarer name, iterator name, etc.
So we have to write
attribute name __genl_const;
right?
Quoted 4 lines
The handling of attributes has also been extended so that you can actually match against them in some cases, such as on function declarations. On the oher hand, nothing was done to allow attributes in front of types that are alone.
Very cool!
Quoted 2 lines
I can try to fix this. In the short term, perhaps using 1.0.4 is acceptable.
To me that's OK, I'll try to keep it in mind.
Can we do some "#ifdef" type syntax in spatch on the version of it? :-)
I guess I could also have two different versions of the patch and write some code to pick up which one to use, but that's a bit awkward ...
But let me know if you can't actually fix this (easily), then I can do that instead.
johannes
Quoted 4 lines · Johannes Berg
Hi Julia,
Quoted 1 line · Julia Lawall
Quoted 8 lines
Quoted 2 lines
The attribute metavariable declaration has become attribute name, to be more like eg declarer name, iterator name, etc.
So we have to write
attribute name
__genl_const;right?
Yes.
Quoted 5 lines
Quoted 4 lines
The handling of attributes has also been extended so that you can actually match against them in some cases, such as on function declarations. On the oher hand, nothing was done to allow attributes in front of types that are alone.
Quoted 8 lines
Very cool!
Quoted 2 lines
I can try to fix this. In the short term, perhaps using 1.0.4 is acceptable.
To me that's OK, I'll try to keep it in mind.
Can we do some "#ifdef" type syntax in spatch on the version of it? :-)
There's nothing for that available at the moment.
Quoted 5 lines
I guess I could also have two different versions of the patch and write some code to pick up which one to use, but that's a bit awkward ...
But let me know if you can't actually fix this (easily), then I can do that instead.
OK, I'll try to look at it soon.
julia
Quoted 15 lines · Julia Lawall
Quoted 7 lines · Johannes Berg
Hi Julia,
Quoted 4 lines · Julia Lawall
The handling of attributes has also been extended so that you can actually match against them in some cases, such as on function declarations. On the oher hand, nothing was done to allow attributes in front of types that are alone.
Quoted 6 lines
Very cool!
Quoted 2 lines
I can try to fix this. In the short term, perhaps using 1.0.4 is acceptable.
To me that's OK, I'll try to keep it in mind.
Can we do some "#ifdef" type syntax in spatch on the version of it? :-)
There's nothing for that available at the moment.
Quoted 8 lines
Quoted 5 lines
I guess I could also have two different versions of the patch and write some code to pick up which one to use, but that's a bit awkward ...
But let me know if you can't actually fix this (easily), then I can do that instead.
OK, I'll try to look at it soon.
julia
I thought I'd point out the layers of coevolution that result from the changing spatch behavior.? :) It's a bit fractal when every layer in the dependency graph has the possibility to change.? I suppose a #ifdef type solution might remove the potential need to backport the .cocci patches themselves going forward.
But, I think it might be sufficient to just document what version of the tools, in this case spatch, were used for a particular backports commit/release.? It's easy enough to build any version of spatch from git.? Or perhaps each .cocci patch could have a comment listing the spatch version it was developed with.? Future backports releases could then update the patch syntax as needed, document the spatch version, and leave it at that.
-Remington
Quoted 32 lines · Remington Furman
Quoted 17 lines · Julia Lawall
Quoted 9 lines · Johannes Berg
Hi Julia,
Quoted 6 lines · Julia Lawall
The handling of attributes has also been extended so that you can actually
match against them in some cases, such as on function declarations. On the oher hand, nothing was done to allow attributes in front of types that
are alone.Quoted 6 lines
Very cool!
Quoted 2 lines
I can try to fix this. In the short term, perhaps using 1.0.4 is acceptable.
To me that's OK, I'll try to keep it in mind.
Can we do some "#ifdef" type syntax in spatch on the version of it? :-)
There's nothing for that available at the moment.
Quoted 8 lines
Quoted 5 lines
I guess I could also have two different versions of the patch and write some code to pick up which one to use, but that's a bit awkward ...
But let me know if you can't actually fix this (easily), then I can do that instead.
OK, I'll try to look at it soon.
julia
I thought I'd point out the layers of coevolution that result from the changing spatch behavior.? :) It's a bit fractal when every layer in the dependency graph has the possibility to change.? I suppose a #ifdef type solution might remove the potential need to backport the .cocci patches themselves going forward.
Quoted 6 lines
But, I think it might be sufficient to just document what version of the tools, in this case spatch, were used for a particular backports commit/release.? It's easy enough to build any version of spatch from git.? Or perhaps each .cocci patch could have a comment listing the spatch version it was developed with.? Future backports releases could then update the patch syntax as needed, document the spatch version, and leave it at that.
Hello,
I updated the github version of Coccinelle such that patches/0027-genl-const/genl-const.cocci
will almost parse correctly. The only thing that remains is to change attribute into attribute name.
julia
Quoted 4 lines · Julia Lawall
I updated the github version of Coccinelle such that
patches/0027-genl-const/genl-const.cocci
will almost parse correctly. The only thing that remains is to change attribute into attribute name.
Ok, thanks, I'll keep that in mind when I update the version. It's a b it unfortunate you couldn't get it to be completely backwards compatible, but we'll deal with it :)
johannes
Julia,
Quoted 4 lines
I updated the github version of Coccinelle such that
patches/0027-genl-const/genl-const.cocci
will almost parse correctly. The only thing that remains is to change attribute into attribute name.
I'm not sure why, but for some reason I thought back then (https://git.k
ernel.org/pub/scm/linux/kernel/git/backports/backports.git/commit/?id=98 272f479c2126a135dfcb12484e93d5888164ab) that the following would work with 1.0.6:
@@
attribute name __genl_const;
@@
(
-const struct genl_multicast_group
+__genl_const struct genl_multicast_group
|
-const struct genl_ops
+__genl_const struct genl_ops
)
However, Hauke pointed out it doesn't, and I can confirm. On 1.0.6, I can't get this to work at all, it refuses to parse "attribute name" and it refuses to parse the patch part with just "attribute".
Is there any way to get this to work again?
johannes
Hi Johannes and Julia,
Wow, you two are great.? Thank you for the quick response.? More notes below, including another spatch error after downgrading to spatch 1.0.4.
Quoted 6 lines · Johannes Berg
+cocci list
Hi,
Looks like you made a really deep investigation here - awesome, thanks for doing that!
It's the least I could do, considering the work you've put in before me.
Quoted 6 lines · Remington Furman
Quoted 5 lines
Full log using
genl-const.cocciinit_defs_builtins:/usr/local/bin/../lib/coccinelle/standard.h
meta: parse error:
??? File "./patches/0027-genl-const/genl-const.cocci", line 2, column
10,
Quoted 18 lines
Quoted 4 lines
charpos = 13
??? around = '__genl_const',
??? whole content = attribute__genl_const;
[...snip callstack...]FWIW, the patch is pretty simple:
@@
attribute__genl_const;
@@
((quoted diff) +1
-1-const struct genl_multicast_group +__genl_const struct genl_multicast_group|
(quoted diff) +1
-1-const struct genl_ops +__genl_const struct genl_ops)
Quoted 1 line
(yes, that's the full patch)
I had considered trying to manually apply the patch, but didn't know what files it would touch, or if spatch would fail on other patches later on.
Quoted 8 lines
Quoted 7 lines
This is very similar to this problem:
<https://www.spinics.net/lists/backports/msg04216.html>
The conclusion of that thread is that the --integrate workflow is no longer supported.However, I am not using --integrate, and I am still getting errors about parsing '
__genl_const'.
Quoted 7 lines
Yeah this --integrate is unrelated to the problem - the problem is that spatch can't parse this patch (any more).
Quoted 2 lines
Is that a bug in the patch
or possibly a regression in spatch?I think it's a regression in spatch.
Ok, I'm glad Julia confirmed that the behavior changed between releases.
Quoted 6 lines
Quoted 2 lines
I've tried v4.14-rc2 also,
but nothing older.That's a kernel version so not really relevant - spatch doesn't even attempt to apply the changes since it can't parse them.
Quoted 7 lines
Quoted 5 lines
This page has a lot of great example info, and suggested that it could be a problem with my version of spatch. Specifically, that I was missing the menhir and libmenhir-ocaml-dev packages when building Coccinelle and a rebuild should help.
<http://stonyslp.blogspot.com/2017/08/backport-driver.html>That I don't know - I've never built spatch myself.
If I remember correctly, I had to because the version in my package manager was only 1.0.0-rc19, and that was too old for the backports tag I checked out.? So, I chose the latest release in git.
Quoted 28 lines
Quoted 12 lines
Unfortunately, I still have the same errors after running make distclean and rebuilding.
I have:
coccinelle $ git describe --tags
1.0.6-365-gd73c6e6
coccinelle $ spatch --version
spatch version 1.0.6-00365-gd73c6e6 compiled with OCaml version 4.01.0 Flags passed to the configure script: [none]
OCaml scripting support: yes
Python scripting support: yes
Syntax of regular expresssions: PCREThat seems reasonable as far as configuration is concerned.
Quoted 7 lines
I found this patch which describes how older versions can't handle the
__genl_constchanges (20150209):
<https://www.spinics.net/lists/backports/msg03223.html>But, I also see commit
748116c8from shortly after (20150507) which introducedgenl-const.cocciand says only spatch 1.0.0-rc23 is needed.Right.
Quoted 2 lines
I feel like I'm close to figuring this out, but would appreciate any help or pointers.
Quoted 7 lines
Yeah I think you're pretty close.
Can you try obtaining/building spatch 1.0.4? That definitely works for me, so I think there's probably a regression in spatch somewhere between 1.0.4 and the version that you're running.
johannes
Ok, I've rebuilt with 1.0.4, and it applied genl-const.cocci just fine.? But, later on my CPU was held at 100% for 29 minutes:
Applying SmPL patch 0079-netdev-destructor.cocci
[...waited 29 minutes...]
^CTraceback (most recent call last):
? File "./gentree.py", line 1107, in <module>
??? ret = _main()
? File "./gentree.py", line 724, in _main
??? logwrite=logwrite)
? File "./gentree.py", line 906, in process
??? failure = apply_patches(args, "backport", source_dir, 'patches',
bpid.target_dir, logwrite)
? File "./gentree.py", line 601, in apply_patches
??? output = sprocess.communicate()[0]
? File "/usr/lib/python2.7/subprocess.py", line 791, in communicate ??? stdout = _eintr_retry_call(self.stdout.read)
? File "/usr/lib/python2.7/subprocess.py", line 476, in _eintr_retry_call ??? return func(*args)
KeyboardInterrupt
Any ideas there?? Perhaps I will try another release version of spatch and report back.? Or I will update the genl-const.cocci patch and try the latest spatch release again.
Thank you,
Remington
Quoted 3 lines · Remington Furman
Ok, I've rebuilt with 1.0.4, and it applied
genl-const.coccijust fine. But, later on my CPU was held at 100% for 29 minutes:
Yeah, known issue. Give it more time - that particular patch is *really* slow. Yes, more than 29 minutes, on my system as well.
What I can recommend is that you remove things from copy-list that you don't care about, e.g. bluetooth drivers if you don't need them, certain wifi drivers you don't need, etc. (before running gentree.py)
johannes
Quoted 11 lines · Johannes Berg
Quoted 2 lines · Remington Furman
Ok, I've rebuilt with 1.0.4, and it applied
genl-const.coccijust fine. But, later on my CPU was held at 100% for 29 minutes:Yeah, known issue. Give it more time - that particular patch is *really* slow. Yes, more than 29 minutes, on my system as well.
What I can recommend is that you remove things from copy-list that you don't care about, e.g. bluetooth drivers if you don't need them, certain wifi drivers you don't need, etc. (before running
gentree.py)johannes
Ok, that's great to know.? I will restart it.? I didn't know about the copy list yet, that will be helpful.
Thank you again for your help.
-Remington
Quoted 11 lines · Remington Furman
Quoted 10 lines · Johannes Berg
Quoted 2 lines · Remington Furman
Ok, I've rebuilt with 1.0.4, and it applied
genl-const.coccijust fine.? But, later on my CPU was held at 100% for 29 minutes:Yeah, known issue. Give it more time - that particular patch is *really* slow. Yes, more than 29 minutes, on my system as well.
What I can recommend is that you remove things from copy-list that you don't care about, e.g. bluetooth drivers if you don't need them, certain wifi drivers you don't need, etc. (before running
gentree.py)
Quoted 8 lines
Quoted 1 line
johannes
Ok, that's great to know.? I will restart it.? I didn't know about the copy list yet, that will be helpful.
Thank you again for your help.
-Remington
Following up, running gentree.py again (without modifying the copy list) worked fine.
-Remington
Quoted 3 lines · Remington Furman
Following up, running
gentree.pyagain (without modifying the copy list) worked fine.
Good to hear :-)
Out of curiosity - how long did it take?
johannes
Quoted 11 lines · Johannes Berg
Quoted 3 lines · Remington Furman
Ok, I've rebuilt with 1.0.4, and it applied
genl-const.coccijust fine. But, later on my CPU was held at 100% for 29 minutes:Yeah, known issue. Give it more time - that particular patch is *really* slow. Yes, more than 29 minutes, on my system as well.
What I can recommend is that you remove things from copy-list that you don't care about, e.g. bluetooth drivers if you don't need them, certain wifi drivers you don't need, etc. (before running
gentree.py)
You can also set a timeout, for example --timeout 120 (120 seconds). It will report on the files that timed out and you can check on them manually. Some other useful options are:
--profile: see how long it spends in each rule or various parts of the OCaml code
--show-trying: see what function it is working on
--debug see what rule and with what inputs it is working on
It may work better to put --debug before the others.
julia
Quoted 7 lines
johannes
_______________________________________________
Cocci mailing list
Cocci at systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci
Quoted 3 lines · Julia Lawall
You can also set a timeout, for example --timeout 120 (120 seconds). It will report on the files that timed out and you can check on them manually. Some other useful options are:
Right, but the result won't compile if you do that and it doesn't modify some files :-)
Quoted 6 lines
--profile: see how long it spends in each rule or various parts of the OCaml code
--show-trying: see what function it is working on
--debug see what rule and with what inputs it is working on
I guess I should play with these and figure out why this particular spatch is so slow, thanks for the pointers!
johannes