Skip to contentSkip to conversation

Dependencies between spatch hunks

Cocci Kieran Bingham 2 participants 8 messages Apr 20, 2016

lore
Share & tools
lore
#0 lore raw

Hi all,

I am finding that a rule which matches a part of code, sometimes causes hunks which depend on it to act, and sometimes not to act

My spatch is successfully removing and converting hunks that I desire changed, however the requirement became apparent *to not process the file at all* if the variable is used in the probe function, so I added in a dependency on probe_id_unused (established in my previous mail thread).

Therefore, I would expect to be able to set the 'depends on' to be on 'probe_id_unused' for each of the actions, and have actions only taken if the full dependency chain (C4->C1 below) is met.

However, I get a non-consistent application of this, where some hunks operate when (I believe) they shouldn't:

My full spatch for reference, is at:

https://gist.github.com/kbingham/96477177dd20a72b1c2f

In essence, it does the following {C}hecks:
C1 - of_dev_id_present : Check for a struct of_device_id
C2 - dev_id : Check for a struct i2c_device_id
C3 - driver : Check and identify the probefunc in the driver structure C4 - probe_id_unused : Establish if the id is used in the probe function

Where C4 depends on C3 depends on C2 depends on C1

The aim is that if all of the above checks/identifiers are met, it will take the following actions:
A rewrite the probe function declaration
B re-point the function pointer in the driver structure
C remove the i2c_device_id reference
D remove the i2c_device_id array
E remove the MODULE_DEVICE_TABLE macro

For this example, I'll take three files from the kernel source, all of which meet conditions C1 - > C3 (but only F3 meets C4):

F1: drivers/gpio/gpio-pca953x.c : probe id used ( ! C4 )
F2: drivers/staging/iio/light/isl29018.c : probe id used ( ! C4 )
F3: sound/soc/codecs/wm8737.c : probe id UNUSED ( C4 )

If all the actions (A->E) start with @ depends on driver @ (to depend on C3) all the actions complete on these files. (Shown in sequence in the c file)

F1 : D E A B C
F2 : A D E B C
F3 : A D E B C

However, if all the actions depend on @ depends on probe_id_unused @ (depends on C4), Some actions complete, and some do not!

F1 : D E C (Unexpected behaviour - I expect F1 : <no change>)
F2 : D E C (Unexpected behaviour - I expect F2 : <no change>)
F3 : A D E B C (Expected behaviour)

So of course, I want actions D E and C to *not* complete on F1 and F2, but I can't understand why they do not comply with their 'depends' chain. Am I looking at a bug in Coccinelle here or a bug with my interpretation of the depends keyword?

Sorry for the long mail, and look forward to any ideas!

--
Regards

Kieran

#1 lore raw
Quoted 9 lines · Kieran Bingham

Hi all,

I am finding that a rule which matches a part of code, sometimes causes hunks which depend on it to act, and sometimes not to act

My spatch is successfully removing and converting hunks that I desire changed, however the requirement became apparent *to not process the file at all* if the variable is used in the probe function, so I added in a dependency on probe_id_unused (established in my previous mail thread).

I'll check on the rest shortly, but if you really want to not process the file at all, just look for the condition in which that is the case, and then write a python rule that depends on it, and run Cocci.exit() (not sure about the exact syntax - writing an ocaml rule and putting Coccilib.exit() will also work).

julia

Quoted 65 lines

Therefore, I would expect to be able to set the 'depends on' to be on 'probe_id_unused' for each of the actions, and have actions only taken if the full dependency chain (C4->C1 below) is met.

However, I get a non-consistent application of this, where some hunks operate when (I believe) they shouldn't:

My full spatch for reference, is at:

https://gist.github.com/kbingham/96477177dd20a72b1c2f

In essence, it does the following {C}hecks:
C1 - of_dev_id_present : Check for a struct of_device_id
C2 - dev_id : Check for a struct i2c_device_id
C3 - driver : Check and identify the probefunc in the driver structure C4 - probe_id_unused : Establish if the id is used in the probe function

Where C4 depends on C3 depends on C2 depends on C1

The aim is that if all of the above checks/identifiers are met, it will take the following actions:
A rewrite the probe function declaration
B re-point the function pointer in the driver structure
C remove the i2c_device_id reference
D remove the i2c_device_id array
E remove the MODULE_DEVICE_TABLE macro

For this example, I'll take three files from the kernel source, all of which meet conditions C1 - > C3 (but only F3 meets C4):

F1: drivers/gpio/gpio-pca953x.c : probe id used ( ! C4 )
F2: drivers/staging/iio/light/isl29018.c : probe id used ( ! C4 )
F3: sound/soc/codecs/wm8737.c : probe id UNUSED ( C4 )

If all the actions (A->E) start with @ depends on driver @ (to depend on C3) all the actions complete on these files. (Shown in sequence in the c file)

F1 : D E A B C
F2 : A D E B C
F3 : A D E B C

However, if all the actions depend on @ depends on probe_id_unused @ (depends on C4), Some actions complete, and some do not!

F1 : D E C (Unexpected behaviour - I expect F1 : <no change>)
F2 : D E C (Unexpected behaviour - I expect F2 : <no change>)
F3 : A D E B C (Expected behaviour)

So of course, I want actions D E and C to *not* complete on F1 and F2, but I can't understand why they do not comply with their 'depends' chain. Am I looking at a bug in Coccinelle here or a bug with my interpretation of the depends keyword?

Sorry for the long mail, and look forward to any ideas!

--
Regards

Kieran
_______________________________________________
Cocci mailing list
Cocci at systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

#2 lore raw
Quoted 13 lines · Julia Lawall
Quoted 9 lines · Kieran Bingham

Hi all,

I am finding that a rule which matches a part of code, sometimes causes hunks which depend on it to act, and sometimes not to act

My spatch is successfully removing and converting hunks that I desire changed, however the requirement became apparent *to not process the file at all* if the variable is used in the probe function, so I added in a dependency on probe_id_unused (established in my previous mail thread).

Quoted 5 lines

I'll check on the rest shortly, but if you really want to not process the file at all, just look for the condition in which that is the case, and then write a python rule that depends on it, and run Cocci.exit() (not sure about the exact syntax - writing an ocaml rule and putting Coccilib.exit() will also work).

Thanks Julia,

The following worked for my needs:

@ script:python depends on driver && !probe_id_unused @ @@
print("Probe function uses the ID parameter") cocci.exit()

However, interestingly - it's not useful if I specify a minimal set of files on the command line:

Using the command:
spatch --linux-spacing --sp-file patches/i2c-dt.cocci $A $B $C $D
HANDLING: sound/soc/codecs/wm8737.c drivers/staging/iio/light/isl29018.c drivers/gpio/gpio-pca953x.c ./drivers/rtc/rtc-ds1307.c
Probe function uses the ID parameter
kbingham at CookieMonster:/opt/projects/linux/kbuild-bbb/sources/linux$

appears to stop at the first file (which is expected to stop) without processing the next files.

However, simply using the '.' target - does appear to iterate through the entire sources successfully, parsing files it can, and stopping on files that it should stop on.

Thanks again,

--
Kieran

Quoted 67 lines

julia

Quoted 65 lines

Therefore, I would expect to be able to set the 'depends on' to be on 'probe_id_unused' for each of the actions, and have actions only taken if the full dependency chain (C4->C1 below) is met.

However, I get a non-consistent application of this, where some hunks operate when (I believe) they shouldn't:

My full spatch for reference, is at:

https://gist.github.com/kbingham/96477177dd20a72b1c2f

In essence, it does the following {C}hecks:
C1 - of_dev_id_present : Check for a struct of_device_id
C2 - dev_id : Check for a struct i2c_device_id
C3 - driver : Check and identify the probefunc in the driver structure C4 - probe_id_unused : Establish if the id is used in the probe function

Where C4 depends on C3 depends on C2 depends on C1

The aim is that if all of the above checks/identifiers are met, it will take the following actions:
A rewrite the probe function declaration
B re-point the function pointer in the driver structure
C remove the i2c_device_id reference
D remove the i2c_device_id array
E remove the MODULE_DEVICE_TABLE macro

For this example, I'll take three files from the kernel source, all of which meet conditions C1 - > C3 (but only F3 meets C4):

F1: drivers/gpio/gpio-pca953x.c : probe id used ( ! C4 )
F2: drivers/staging/iio/light/isl29018.c : probe id used ( ! C4 )
F3: sound/soc/codecs/wm8737.c : probe id UNUSED ( C4 )

If all the actions (A->E) start with @ depends on driver @ (to depend on C3) all the actions complete on these files. (Shown in sequence in the c file)

F1 : D E A B C
F2 : A D E B C
F3 : A D E B C

However, if all the actions depend on @ depends on probe_id_unused @ (depends on C4), Some actions complete, and some do not!

F1 : D E C (Unexpected behaviour - I expect F1 : <no change>)
F2 : D E C (Unexpected behaviour - I expect F2 : <no change>)
F3 : A D E B C (Expected behaviour)

So of course, I want actions D E and C to *not* complete on F1 and F2, but I can't understand why they do not comply with their 'depends' chain. Am I looking at a bug in Coccinelle here or a bug with my interpretation of the depends keyword?

Sorry for the long mail, and look forward to any ideas!

--
Regards

Kieran
_______________________________________________
Cocci mailing list
Cocci at systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

#3 lore raw
Quoted 40 lines · Kieran Bingham
Quoted 13 lines · Julia Lawall
Quoted 9 lines · Kieran Bingham

Hi all,

I am finding that a rule which matches a part of code, sometimes causes hunks which depend on it to act, and sometimes not to act

My spatch is successfully removing and converting hunks that I desire changed, however the requirement became apparent *to not process the file at all* if the variable is used in the probe function, so I added in a dependency on probe_id_unused (established in my previous mail thread).

Quoted 5 lines

I'll check on the rest shortly, but if you really want to not process the file at all, just look for the condition in which that is the case, and then write a python rule that depends on it, and run Cocci.exit() (not sure about the exact syntax - writing an ocaml rule and putting Coccilib.exit() will also work).

Thanks Julia,

The following worked for my needs:

@ script:python depends on driver && !probe_id_unused @ @@
print("Probe function uses the ID parameter") cocci.exit()

However, interestingly - it's not useful if I specify a minimal set of files on the command line:

Using the command:
spatch --linux-spacing --sp-file patches/i2c-dt.cocci $A $B $C $D
HANDLING: sound/soc/codecs/wm8737.c drivers/staging/iio/light/isl29018.c drivers/gpio/gpio-pca953x.c ./drivers/rtc/rtc-ds1307.c
Probe function uses the ID parameter
kbingham at CookieMonster:/opt/projects/linux/kbuild-bbb/sources/linux$

Quoted 6 lines

appears to stop at the first file (which is expected to stop) without processing the next files.

However, simply using the '.' target - does appear to iterate through the entire sources successfully, parsing files it can, and stopping on files that it should stop on.

Putting multiple files on the command line is not the right way to process multiple files. What that means is to process those files all at once. Normally Coccinelle processes only one file at a time, which means that when there is a call from a function in one file to a function in another, it won't find the definition of the called function. If you put multiple files on the command line, it will see them all at once. The downside though is that Coccinelle will ultimately run more slowly, because it will be using a lot more memory at a given time.

If you have a directory where you want to process only some of the files, you can use the --file-groups <file> option. <file> could be eg

path/foo.c
path/bar.c

path/xyz.c

Then foo.c and bar.c will be processed at once, and xyz.c will be processed separately.

But it could be easier to just work on the whole directory and let Coccinelle to fail to do anything with most of the files.

julia

#6 lore raw
In reply to Kieran Bingham
Quoted 59 lines · Kieran Bingham

Hi all,

I am finding that a rule which matches a part of code, sometimes causes hunks which depend on it to act, and sometimes not to act

My spatch is successfully removing and converting hunks that I desire changed, however the requirement became apparent *to not process the file at all* if the variable is used in the probe function, so I added in a dependency on probe_id_unused (established in my previous mail thread).

Therefore, I would expect to be able to set the 'depends on' to be on 'probe_id_unused' for each of the actions, and have actions only taken if the full dependency chain (C4->C1 below) is met.

However, I get a non-consistent application of this, where some hunks operate when (I believe) they shouldn't:

My full spatch for reference, is at:

https://gist.github.com/kbingham/96477177dd20a72b1c2f

In essence, it does the following {C}hecks:
C1 - of_dev_id_present : Check for a struct of_device_id
C2 - dev_id : Check for a struct i2c_device_id
C3 - driver : Check and identify the probefunc in the driver structure C4 - probe_id_unused : Establish if the id is used in the probe function

Where C4 depends on C3 depends on C2 depends on C1

The aim is that if all of the above checks/identifiers are met, it will take the following actions:
A rewrite the probe function declaration
B re-point the function pointer in the driver structure
C remove the i2c_device_id reference
D remove the i2c_device_id array
E remove the MODULE_DEVICE_TABLE macro

For this example, I'll take three files from the kernel source, all of which meet conditions C1 - > C3 (but only F3 meets C4):

F1: drivers/gpio/gpio-pca953x.c : probe id used ( ! C4 )
F2: drivers/staging/iio/light/isl29018.c : probe id used ( ! C4 )
F3: sound/soc/codecs/wm8737.c : probe id UNUSED ( C4 )

If all the actions (A->E) start with @ depends on driver @ (to depend on C3) all the actions complete on these files. (Shown in sequence in the c file)

F1 : D E A B C
F2 : A D E B C
F3 : A D E B C

However, if all the actions depend on @ depends on probe_id_unused @ (depends on C4), Some actions complete, and some do not!

F1 : D E C (Unexpected behaviour - I expect F1 : <no change>)
F2 : D E C (Unexpected behaviour - I expect F2 : <no change>)
F3 : A D E B C (Expected behaviour)

Quoted 6 lines

So of course, I want actions D E and C to *not* complete on F1 and F2, but I can't understand why they do not comply with their 'depends' chain. Am I looking at a bug in Coccinelle here or a bug with my interpretation of the depends keyword?

Sorry for the long mail, and look forward to any ideas!

What version of Coccinelle do you have? I get the expected behavior. No change on F1 and F2, and a change on F3.

julia

#4 lore raw
In reply to Julia Lawall
Quoted 5 lines · Julia Lawall

On Wed, 20 Apr 2016, Kieran Bingham wrote:

Quoted 7 lines
Quoted 7 lines · Julia Lawall
Quoted 6 lines · Kieran Bingham
Quoted 4 lines

Hi all,

I am finding that a rule which matches a part of code, sometimes

causes

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

hunks which depend on it to act, and sometimes not to act

My spatch is successfully removing and converting hunks that I desire changed, however the requirement became apparent *to not process the file at all* if the variable is used in the probe function, so I

added

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

in a dependency on probe_id_unused (established in my previous mail

thread).

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I'll check on the rest shortly, but if you really want to not process

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

file at all, just look for the condition in which that is the case,

and

Quoted 46 lines
Quoted 24 lines
Quoted 3 lines

then write a python rule that depends on it, and run Cocci.exit() (not sure about the exact syntax - writing an ocaml rule and putting Coccilib.exit() will also work).

Thanks Julia,

The following worked for my needs:

@ script:python depends on driver && !probe_id_unused @ @@
print("Probe function uses the ID parameter") cocci.exit()

However, interestingly - it's not useful if I specify a minimal set of files on the command line:

Using the command:
spatch --linux-spacing --sp-file patches/i2c-dt.cocci $A $B $C $D
HANDLING: sound/soc/codecs/wm8737.c drivers/staging/iio/light/isl29018.c drivers/gpio/gpio-pca953x.c ./drivers/rtc/rtc-ds1307.c
Probe function uses the ID parameter
kbingham at CookieMonster:/opt/projects/linux/kbuild-bbb/sources/linux$

Quoted 6 lines

appears to stop at the first file (which is expected to stop) without processing the next files.

However, simply using the '.' target - does appear to iterate through the entire sources successfully, parsing files it can, and stopping on files that it should stop on.

Putting multiple files on the command line is not the right way to process multiple files. What that means is to process those files all at once. Normally Coccinelle processes only one file at a time, which means that when there is a call from a function in one file to a function in another, it won't find the definition of the called function. If you put multiple files on the command line, it will see them all at once. The downside though is that Coccinelle will ultimately run more slowly, because it will be using a lot more memory at a given time.

If you have a directory where you want to process only some of the files, you can use the --file-groups <file> option. <file> could be eg

path/foo.c
path/bar.c

Quoted 7 lines

path/xyz.c

Then foo.c and bar.c will be processed at once, and xyz.c will be processed separately.

But it could be easier to just work on the whole directory and let Coccinelle to fail to do anything with most of the files.

Ah I see. I bet that explains my other dependency issue as well. It was probably matching state from other processed files in the same processing group.

I had been specifying a group of files to make sure the right thing was being done in the processing. In the whole tree about 250 files match so I was trying to reduce the set to verify.

It looks like the act of me trying to reduce the set changed the processing though ;-)

I guess I could create a single folder with the reduced set in, but it looks like its doing the right thing now with the exit ()

Thanks

Kieran

Quoted 2 lines

julia

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://systeme.lip6.fr/pipermail/cocci/attachments/20160420/81814080/attachment.html>

#7 lore raw
In reply to Julia Lawall
Quoted 7 lines · Julia Lawall
Quoted 2 lines · Kieran Bingham

Hi all,

Quoted 7 lines
Quoted 7 lines

I am finding that a rule which matches a part of code, sometimes causes hunks which depend on it to act, and sometimes not to act

My spatch is successfully removing and converting hunks that I desire changed, however the requirement became apparent *to not process the file at all* if the variable is used in the probe function, so I added in a dependency on probe_id_unused (established in my previous mail

thread).

Quoted 55 lines
Quoted 55 lines

Therefore, I would expect to be able to set the 'depends on' to be on 'probe_id_unused' for each of the actions, and have actions only taken if the full dependency chain (C4->C1 below) is met.

However, I get a non-consistent application of this, where some hunks operate when (I believe) they shouldn't:

My full spatch for reference, is at:

https://gist.github.com/kbingham/96477177dd20a72b1c2f

In essence, it does the following {C}hecks:
C1 - of_dev_id_present : Check for a struct of_device_id
C2 - dev_id : Check for a struct i2c_device_id
C3 - driver : Check and identify the probefunc in the driver structure C4 - probe_id_unused : Establish if the id is used in the probe function

Where C4 depends on C3 depends on C2 depends on C1

The aim is that if all of the above checks/identifiers are met, it will take the following actions:
A rewrite the probe function declaration
B re-point the function pointer in the driver structure
C remove the i2c_device_id reference
D remove the i2c_device_id array
E remove the MODULE_DEVICE_TABLE macro

For this example, I'll take three files from the kernel source, all of which meet conditions C1 - > C3 (but only F3 meets C4):

F1: drivers/gpio/gpio-pca953x.c : probe id used ( ! C4 )
F2: drivers/staging/iio/light/isl29018.c : probe id used ( ! C4 )
F3: sound/soc/codecs/wm8737.c : probe id UNUSED ( C4 )

If all the actions (A->E) start with @ depends on driver @ (to depend on C3) all the actions complete on these files. (Shown in sequence in the c file)

F1 : D E A B C
F2 : A D E B C
F3 : A D E B C

However, if all the actions depend on @ depends on probe_id_unused @ (depends on C4), Some actions complete, and some do not!

F1 : D E C (Unexpected behaviour - I expect F1 : <no change>)
F2 : D E C (Unexpected behaviour - I expect F2 : <no change>)
F3 : A D E B C (Expected behaviour)

So of course, I want actions D E and C to *not* complete on F1 and F2, but I can't understand why they do not comply with their 'depends' chain. Am I looking at a bug in Coccinelle here or a bug with my interpretation of the depends keyword?

Quoted 4 lines
Quoted 1 line

Sorry for the long mail, and look forward to any ideas!

What version of Coccinelle do you have? I get the expected behavior. No change on F1 and F2, and a change on F3.

I think the fault was somewhere between the keyboard and chair :-)

I'm away from my keyboard now, so I can't check but I'll assume it was because I was specifying the files in one command line.

Regards

Kieran

Quoted 2 lines

julia

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://systeme.lip6.fr/pipermail/cocci/attachments/20160420/dfcb6499/attachment-0001.html>

#5 lore raw
In reply to Kieran Bingham
Quoted 91 lines · Kieran Bingham
Quoted 5 lines · Julia Lawall

On Wed, 20 Apr 2016, Kieran Bingham wrote:

Quoted 7 lines
Quoted 7 lines · Julia Lawall
Quoted 6 lines · Kieran Bingham
Quoted 4 lines

Hi all,

I am finding that a rule which matches a part of code, sometimes

causes

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

hunks which depend on it to act, and sometimes not to act

My spatch is successfully removing and converting hunks that I desire changed, however the requirement became apparent *to not process the file at all* if the variable is used in the probe function, so I

added

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

in a dependency on probe_id_unused (established in my previous mail

thread).

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I'll check on the rest shortly, but if you really want to not process

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

file at all, just look for the condition in which that is the case,

and

Quoted 46 lines
Quoted 24 lines
Quoted 3 lines

then write a python rule that depends on it, and run Cocci.exit() (not sure about the exact syntax - writing an ocaml rule and putting Coccilib.exit() will also work).

Thanks Julia,

The following worked for my needs:

@ script:python depends on driver && !probe_id_unused @ @@
print("Probe function uses the ID parameter") cocci.exit()

However, interestingly - it's not useful if I specify a minimal set of files on the command line:

Using the command:
spatch --linux-spacing --sp-file patches/i2c-dt.cocci $A $B $C $D
HANDLING: sound/soc/codecs/wm8737.c drivers/staging/iio/light/isl29018.c drivers/gpio/gpio-pca953x.c ./drivers/rtc/rtc-ds1307.c
Probe function uses the ID parameter
kbingham at CookieMonster:/opt/projects/linux/kbuild-bbb/sources/linux$

Quoted 6 lines

appears to stop at the first file (which is expected to stop) without processing the next files.

However, simply using the '.' target - does appear to iterate through the entire sources successfully, parsing files it can, and stopping on files that it should stop on.

Putting multiple files on the command line is not the right way to process multiple files.? What that means is to process those files all at once. Normally Coccinelle processes only one file at a time, which means that when there is a call from a function in one file to a function in another, it won't find the definition of the called function.? If you put multiple files on the command line, it will see them all at once.? The downside though is that Coccinelle will ultimately run more slowly, because it will be using a lot more memory at a given time.

If you have a directory where you want to process only some of the files, you can use the --file-groups <file> option.? <file> could be eg

path/foo.c
path/bar.c

Quoted 7 lines

path/xyz.c

Then foo.c and bar.c will be processed at once, and xyz.c will be processed separately.

But it could be easier to just work on the whole directory and let Coccinelle to fail to do anything with most of the files.

Ah I see.? I bet that explains my other dependency issue as well.? It was probably matching state from other processed files in the same processing group.

I had been specifying a group of files to make sure the right thing was being done in the processing. In the whole tree about 250 files match so I was trying to reduce the set to verify.?

Quoted 5 lines

It looks like the act of me trying to reduce the set changed the processing though ;-)

I guess I could create a single folder with the reduced set in, but it looks like its doing the right thing now with the exit ()

You may find it useful to use glimpse or id-utils to index your files. In coccinelle/scripts there are idutils_index.sh and glimpseindex_cocci.sh, respectively. The you can use --use-glimpse or --use-idutils, and it will go directly to the files that contain the important keywords of your semantic patch. Not parsing the other files will be a big savings. The downside of course is that you have to redo the index every time your code base changes.

julia