Skip to contentSkip to conversation

SmPL script construction with included files

Cocci SF Markus Elfring 2 participants 23 messages Nov 15, 2016 – Nov 19, 2016

lore
Share & tools
lore
#1 lore raw
Quoted 8 lines · SF Markus Elfring

Hello,

The semantic patch language supports also the inclusion of external files.
http://coccinelle.lip6.fr/docs/main_grammar001.html

* Does syntax matter for this SmPL statement?

* Can SmPL rules be constructed from smaller parts by it?

I don't understand what it would mean for syntax to or not to matter. The main issue seems to be that the includes can only appear before the normal SmPL rules.

julia

#2 lore raw
Quoted 3 lines
Quoted 1 line

* Can SmPL rules be constructed from smaller parts by it?

I don't understand what it would mean for syntax to or not to matter.

I am curious if the included data need to be usable also alone in the SmPL sense.

Quoted 2 lines

The main issue seems to be that the includes can only appear before the normal SmPL rules.

I would appreciate if the SmPL include statement could be used at more script places.

Regards,
Markus

#3 lore raw
Quoted 6 lines · SF Markus Elfring
Quoted 3 lines
Quoted 1 line

* Can SmPL rules be constructed from smaller parts by it?

I don't understand what it would mean for syntax to or not to matter.

I am curious if the included data need to be usable also alone in the SmPL sense.

I think that if you have multiple includes, later includes can refer to rules in earlier includes.

julia

#4 lore raw
In reply to Julia Lawall
Quoted 1 line

I don't understand what it would mean for syntax to or not to matter.

I am checking if the mentioned SmPL statement could be used just for the inclusion of a bit of code for a metavariable constraint within a script template.

Quoted 2 lines

The main issue seems to be that the includes can only appear before the normal SmPL rules.

I guess that this information means that I need to concatenate pieces by tools which are external to the semantic patch language in my use case.

Regards,
Markus

#5 lore raw
Quoted 5 lines · SF Markus Elfring
Quoted 1 line

I don't understand what it would mean for syntax to or not to matter.

I am checking if the mentioned SmPL statement could be used just for the inclusion of a bit of code for a metavariable constraint within a script template.

In a script rule, the metavariable declarations are interpreted by Coccinelle, not the scripting language, so an include would not be allowed there.

Script rules also don't have constraints on metavariables anyway, so I'm not sure what you are asking about.

julia

#6 lore raw
Quoted 2 lines

In a script rule, the metavariable declarations are interpreted by Coccinelle, not the scripting language, so an include would not be allowed there.

This detail was clear for me in principle.

Quoted 1 line

Script rules also don't have constraints on metavariables anyway,

Can other SmPL rule types specify constraints in the code form of a regular expression or a programming language script?

Quoted 1 line

so I'm not sure what you are asking about.

I am looking for ways to avoid source code duplication by inclusion of other files.
Would it make sense to allow the use of an include statement within the SmPL variable definition block of a rule?

Regards,
Markus

#7 lore raw
Quoted 10 lines · SF Markus Elfring
Quoted 2 lines

In a script rule, the metavariable declarations are interpreted by Coccinelle, not the scripting language, so an include would not be allowed there.

This detail was clear for me in principle.

Quoted 1 line

Script rules also don't have constraints on metavariables anyway,

Can other SmPL rule types specify constraints in the code form of a regular expression or a programming language script?

At the moment, the specification of constraints is not very uniform across the different metavariable types. We are working on improving this.

Quoted 6 lines
Quoted 1 line

so I'm not sure what you are asking about.

I am looking for ways to avoid source code duplication by inclusion of other files.
Would it make sense to allow the use of an include statement within the SmPL variable definition block of a rule?

I have the impression that what you want would be adequately implemented by a virtual identifier that could be tested in the code or by a virtual rule. For example, the semantic patches in the Linux kernel provide alternate implementations depending on whether one uses the arguments -D patch, -D context, etc.

julia

#8 lore raw

[Cocci] Further developments around SmPL constraints?

Quoted 4 lines
Quoted 2 lines

Can other SmPL rule types specify constraints in the code form of a regular expression or a programming language script?

At the moment, the specification of constraints is not very uniform across the different metavariable types. We are working on improving this.

* Have you got any time frame in mind for this software evolution? How many people are involved already?

* Is this the reason that the functionality ?Script constraints on meta-variables? is not explained in the SmPL manual so far?

https://github.com/coccinelle/coccinelle/blob/cbc751b30d9e02390d60ebed643c8e4a3fa0bb2b/tests/idcon_python.cocci#L1

Regards,
Markus

#9 lore raw
In reply to Julia Lawall
Quoted 7 lines
Quoted 5 lines

I am looking for ways to avoid source code duplication by inclusion of other files.
Would it make sense to allow the use of an include statement within the SmPL variable definition block of a rule?

I have the impression that what you want would be adequately implemented by a virtual identifier that could be tested in the code or by a virtual rule.

These syntax elements will fit also to some use cases.

Quoted 2 lines

For example, the semantic patches in the Linux kernel provide alternate implementations depending on whether one uses the arguments -D patch, -D context, etc.

These parameters have got an influence on rule dependencies.

I am thinking about an other use case a bit more where I have got a few SmPL script templates which are working to some degree. If I would like to use also the functionality ?Script constraints on meta-variables? (instead of a big regular expression) for a function name selection on demand there, I am going to recombine existing pieces by a corresponding make script as a kind of preprocessing step.

Regards,
Markus

#10 lore raw
Quoted 6 lines

I am thinking about an other use case a bit more where I have got a few SmPL script templates which are working to some degree. If I would like to use also the functionality ?Script constraints on meta-variables? (instead of a big regular expression) for a function name selection on demand there, I am going to recombine existing pieces by a corresponding make script as a kind of preprocessing step.

The idea with the script constraints on metavariables feature is that the script code there should only contain a function call. The actual function definition should be elsewhere; probably in an initialize rule.

julia

#11 lore raw
Quoted 4 lines

The idea with the script constraints on metavariables feature is that the script code there should only contain a function call. The actual function definition should be elsewhere; probably in an initialize rule.

This information can be reasonable to some degree.

How will the software situation evolve if each script constraint will need another bit of initialisation code?
Can it become an interesting software development challenge then if the constraint specification approach should be switchable for a few of them by a corresponding make script?

Regards,
Markus

#12 lore raw

[Cocci] Further developments around function callbacks for SmPL metavariables?

In reply to Julia Lawall
Quoted 2 lines

The idea with the script constraints on metavariables feature is that the script code there should only contain a function call.

I would like to add another view to this software evolution for further considerations.
The purpose of such a function call can be interpreted in different ways, can't it?

1. I get the impression from the word ?constraint? that something should be restricted. Does this aspect mean that the functionality is finally mapped to a predicate?

2. Can it be also used as a data generator or content provider for any other non-boolean value?

Regards,
Markus

#13 lore raw

[Cocci] Further developments around function callbacks for SmPL metavariables?

Quoted 8 lines · SF Markus Elfring
Quoted 2 lines

The idea with the script constraints on metavariables feature is that the script code there should only contain a function call.

I would like to add another view to this software evolution for further considerations.
The purpose of such a function call can be interpreted in different ways, can't it?

Quoted 6 lines

1. I get the impression from the word ?constraint? that something should be restricted. Does this aspect mean that the functionality is finally mapped to a predicate?

2. Can it be also used as a data generator or content provider for any other non-boolean value?

No. The return value of the function call must be a boolean. The metavariable matches if and only if the function call returns true.

julia

#14 lore raw

[Cocci] Inclusion of script constraints in SmPL

In reply to Julia Lawall
Quoted 3 lines

The idea with the script constraints on metavariables feature is that the script code there should only contain a function call. The actual function definition should be elsewhere; probably in an initialize rule.

* Did you consider the use case already when such a constraint specification would be included from an external file as a component or together with a rule selection for the semantic patch language?

* Would such an inclusion approach trigger special requirements on the position for corresponding initialisation code?

Regards,
Markus

#15 lore raw

[Cocci] Inclusion of script constraints in SmPL

Quoted 10 lines · SF Markus Elfring
Quoted 3 lines

The idea with the script constraints on metavariables feature is that the script code there should only contain a function call. The actual function definition should be elsewhere; probably in an initialize rule.

* Did you consider the use case already when such a constraint specification would be included from an external file as a component or together with a rule selection for the semantic patch language?

* Would such an inclusion approach trigger special requirements on the position for corresponding initialisation code?

The constraint specification should be a single function call. You can set up your @initialize:python@ code to import whatever you want.

julia

#16 lore raw

[Cocci] Inclusion of script constraints in SmPL

Quoted 4 lines
Quoted 2 lines

* Would such an inclusion approach trigger special requirements on the position for corresponding initialisation code?

The constraint specification should be a single function call.

Does the script constraint syntax allow more programming language code there?

Quoted 1 line

You can set up your @initialize:python@ code to import whatever you want.

I would prefer to keep initialisation code for the implementation of the predicate function closely together with the call of the included metavariable.

Regards,
Markus

#17 lore raw

[Cocci] Inclusion of script constraints in SmPL

Quoted 6 lines · SF Markus Elfring
Quoted 4 lines
Quoted 2 lines

* Would such an inclusion approach trigger special requirements on the position for corresponding initialisation code?

The constraint specification should be a single function call.

Does the script constraint syntax allow more programming language code there?

For all practical purposes, no. Concretely, the script constraint code is initially parsed using a C parser, not a parser for the script language. Function calls, of the form f(a,b,c) are in the intersection of C and the supported scripting languages.

Quoted 5 lines
Quoted 1 line

You can set up your @initialize:python@ code to import whatever you want.

I would prefer to keep initialisation code for the implementation of the predicate function closely together with the call of the included metavariable.

You are most welcome to implement in ocaml the lexer and parser for the scripting language of your choice.

julia

#18 lore raw

[Cocci] Inclusion of script constraints in SmPL

Quoted 3 lines
Quoted 1 line

Does the script constraint syntax allow more programming language code there?

For all practical purposes, no.

If you would like to be so strict, I wonder then about the curly brackets in the published test example.
https://github.com/coccinelle/coccinelle/blob/cbc751b30d9e02390d60ebed643c8e4a3fa0bb2b/tests/idcon_python.cocci#L1

Quoted 1 line

Concretely, the script constraint code is initially parsed using a C parser,

I find such an information interesting to some degree.

Why do you pass data to a software component for ?C? at this place?

Quoted 1 line

not a parser for the script language.

I do not see that would be needed in the restricted use case.

Quoted 2 lines

Function calls, of the form f(a,b,c) are in the intersection of C and the supported scripting languages.

Have you got more control about the construction of a single function call?

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

You can set up your @initialize:python@ code to import whatever you want.

I would prefer to keep initialisation code for the implementation of the predicate function closely together with the call of the included metavariable.

You are most welcome to implement in ocaml the lexer and parser for the scripting language of your choice.

It seems that I stumble once again on special cases which become relevant for code composability in bigger usage scenarios.
I am trying to clarify the existing programming interfaces for the semantic patch language a bit more as usual.

Regards,
Markus

#19 lore raw

[Cocci] Inclusion of script constraints in SmPL

Quoted 7 lines · SF Markus Elfring
Quoted 3 lines
Quoted 1 line

Does the script constraint syntax allow more programming language code there?

For all practical purposes, no.

If you would like to be so strict, I wonder then about the curly brackets in the published test example.
https://github.com/coccinelle/coccinelle/blob/cbc751b30d9e02390d60ebed643c8e4a3fa0bb2b/tests/idcon_python.cocci#L1

The syntax is curly brackets around a C expression.

Quoted 5 lines
Quoted 1 line

Concretely, the script constraint code is initially parsed using a C parser,

I find such an information interesting to some degree.

Why do you pass data to a software component for ?C? at this place?

Seemed convenient. Perhaps it could be possible to parse to a closing }, but your goal seems to be to put more complex terms in the script code that might then include }, making things more complicated.

Quoted 9 lines
Quoted 1 line

not a parser for the script language.

I do not see that would be needed in the restricted use case.

Quoted 2 lines

Function calls, of the form f(a,b,c) are in the intersection of C and the supported scripting languages.

Have you got more control about the construction of a single function call?

No.

julia

#20 lore raw

[Cocci] Inclusion of script constraints in SmPL

Quoted 3 lines
Quoted 1 line

https://github.com/coccinelle/coccinelle/blob/cbc751b30d9e02390d60ebed643c8e4a3fa0bb2b/tests/idcon_python.cocci#L1

The syntax is curly brackets around a C expression.

I do not understand this information completely at the moment.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Concretely, the script constraint code is initially parsed using a C parser,

I find such an information interesting to some degree.

Why do you pass data to a software component for ?C? at this place?

Seemed convenient.

I can not follow your views for software convenience with this detail so far.

Quoted 1 line

Perhaps it could be possible to parse to a closing },

I have got another idea in the meantime.

I propose to distinguish these delimiters better for different use cases.

Quoted 2 lines

but your goal seems to be to put more complex terms in the script code that might then include }, making things more complicated.

I hope that it will be possible to pass growing programming scripts there.

Quoted 3 lines
Quoted 1 line

Have you got more control about the construction of a single function call?

No.

I find this answer strange.

How do you think about to support cases like named and unnamed (ad hoc) predicates there?

Examples:

@ad_hoc_replacement@
identifier ID: script:python() { return ID in ['a', 'c'] }; @@
-ID
+36

@initialize:python@
@@
def is_watched(id):

    return id in ['a', 'c']

@structured_replacement@
identifier ID: script:python() ? is_watched;
@@
-ID
+18

Regards,
Markus

#21 lore raw

[Cocci] Inclusion of script constraints in SmPL

Quoted 54 lines · SF Markus Elfring
Quoted 3 lines
Quoted 1 line

https://github.com/coccinelle/coccinelle/blob/cbc751b30d9e02390d60ebed643c8e4a3fa0bb2b/tests/idcon_python.cocci#L1

The syntax is curly brackets around a C expression.

I do not understand this information completely at the moment.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Concretely, the script constraint code is initially parsed using a C parser,

I find such an information interesting to some degree.

Why do you pass data to a software component for ?C? at this place?

Seemed convenient.

I can not follow your views for software convenience with this detail so far.

Quoted 1 line

Perhaps it could be possible to parse to a closing },

I have got another idea in the meantime.

I propose to distinguish these delimiters better for different use cases.

Quoted 2 lines

but your goal seems to be to put more complex terms in the script code that might then include }, making things more complicated.

I hope that it will be possible to pass growing programming scripts there.

Quoted 3 lines
Quoted 1 line

Have you got more control about the construction of a single function call?

No.

I find this answer strange.

How do you think about to support cases like named and unnamed (ad hoc) predicates there?

Examples:

@ad_hoc_replacement@
identifier ID: script:python() { return ID in ['a', 'c'] }; @@
-ID
+36

@initialize:python@
@@
def is_watched(id):

Quoted 7 lines
    return id in ['a', 'c']

@structured_replacement@
identifier ID: script:python() ? is_watched;
@@
-ID
+18

No interest in supporting either. How are these so vastly better than:

@initialize:python@
@@
def is_watched(id):

    return id in ['a', 'c']

@structured_replacement@
identifier ID: script:python() { is_watched(ID) }; @@
-ID
+18

julia

#22 lore raw

[Cocci] Better syntax for SmPL script constraints

Quoted 1 line

No interest in supporting either.

This is a pity at the moment.

Are there any other software development concerns involved which result into such a rejection?

Quoted 5 lines

How are these so vastly better than:

@initialize:python@
@@
def is_watched(id):

Quoted 7 lines
    return id in ['a', 'c']

@structured_replacement@
identifier ID: script:python()

{ is_watched(ID) };

I find this notation improvable. Can a few characters be omitted for the data processing of a named predicate to make it a bit more succinct?

I find that curly brackets would only be useful there if a function body would be directly passed instead of a function name.
Is the suggested usage of ad hoc (unnamed) predicates occasionally more convenient?

Regards,
Markus