1.0.2
Some of the main changes:
* OCaml 3.10 is no longer supported.
* --use-gitgrep falls back to --use-coccigrep for a non-git repository. Thus, it is generally safe to use --use-gitgrep, except when processing an unmanaged subdirectory of a git repository.
* It is no longer possible to create an identifier in OCaml code by saying x := "identifier_name". Instead some constructors have been added for constructing various kinds of code:
val make_ident : string -> Ast_c.metavar_binding_kind
(* string is the identifier name *)
val make_expr : string -> Ast_c.metavar_binding_kind
(* string is the expression *)
val make_expr_with_env : string -> string -> Ast_c.metavar_binding_kind
(* The first argument is a series of variable declarations, designed to
provide type information for the second argument, which is an expression. For example, make_expr_with_env "int x;" "x+3". *)
val make_stmt : string -> Ast_c.metavar_binding_kind
(* The string argument is a statement *)
val make_stmt_with_env : string -> string -> Ast_c.metavar_binding_kind
(* The first argument is a series of declarations and the second
argument is the statement to create *)
val make_type : string -> Ast_c.metavar_binding_kind
(* The string is the type to create *)
val make_listlen : int -> Ast_c.metavar_binding_kind
(* The integer is the list length to create, eg for a parameter list *)
val make_position :
Common.filename ->
string -> int -> int -> int -> int -> Ast_c.metavar_binding_kind
(* The arguments are: file name (string), function name, start line,
start column, end line, end column. Lines start with 1, columns start with 0. *)
* exp at S will bind statement metavariable to the smallest statement enclosing the expression exp. For example, for the C code, return x;, x at S would match S to return x;
* If an identifier metavariable is constrained to match a set of possible names, eg identifier fn = {kmalloc, kfree};, then these names will be taken into account when selecting files. On the other hand, constraints expressed as regular expressions are still not taken into account.
* Autogen creates a version number that corresponds to the most recent commit.
* -j should now work correctly.
julia
[Cocci] release: 1.0.2: Better file selection
Quoted 3 lines
* If an identifier metavariable is constrained to match a set of possible names, eg identifier fn = {kmalloc, kfree};, then these names will be taken into account when selecting files.
Will any more information about this file selection algorithm appear also in the manual?
Regards,
Markus
[Cocci] release: 1.0.2: Software evolution around parallelisation support
In reply to Julia LawallQuoted 1 line
* -j should now work correctly.
An incompatibility for the SmPL rules "initialize" and "finalize" is mentioned in the manual.
https://github.com/coccinelle/coccinelle/blob/a46bef70162d17cec6b0fc6101d737989f735ee4/docs/manual/spatch_options.tex#L673
Can this software limitation be explained better there?
Does the parallelisation support need any more fine-tuning?
* How often is task preparation required before further data processing can be performed in parallel?
* Which programming interfaces do you prefer for the storage of collected data in the fork-join work flow?
Regards,
Markus
[Cocci] release: 1.0.2: Software evolution around parallelisation support
Quoted 7 lines · SF Markus Elfring
Quoted 1 line
* -j should now work correctly.
An incompatibility for the SmPL rules "initialize" and "finalize" is mentioned in the manual.
https://github.com/coccinelle/coccinelle/blob/a46bef70162d17cec6b0fc6101d737989f735ee4/docs/manual/spatch_options.tex#L673Can this software limitation be explained better there?
There is no shared memory in ocaml. initialize and finalize naturally suggest that locations created in initialize can all be collected in a single execution of finalize. Since this is not possible, initialize and finalize are not allowed. If you are OK with disjoint memory spaces per thread, just use the existing script for parallelism.
Quoted 8 lines
Does the parallelisation support need any more fine-tuning?
* How often is task preparation required before further data processing can be performed in parallel?* Which programming interfaces do you prefer for the storage of collected data in the fork-join work flow?
No idea what either of these questions mean, but the whole point is that there is no join.
julia
[Cocci] release: 1.0.2: Software evolution around parallelisation support
Quoted 1 line
There is no shared memory in ocaml.
Specific services (like databases) can provide some shared data storage.
Quoted 2 lines
initialize and finalize naturally suggest that locations created in initialize can all be collected in a single execution of finalize. Since this is not possible,
I imagine that these SmPL rules could still be supported for their original purpose.
Quoted 1 line
initialize and finalize are not allowed.
It might be easier for a moment to exclude these components under the constraints of limited software development resources. Such limitations will result in some software development challenges, won't it?
Quoted 2 lines
If you are OK with disjoint memory spaces per thread, just use the existing script for parallelism.
How do you think about the potential for extensions around task-specific preparation and clean-up?
Quoted 8 lines
Quoted 6 lines
Does the parallelisation support need any more fine-tuning?
* How often is task preparation required before further data processing can be performed in parallel?* Which programming interfaces do you prefer for the storage of collected data in the fork-join work flow?
No idea what either of these questions mean, but the whole point is that there is no join.
This technical restriction can lead to data processing difficulties, can't it?
Are there any situations where SmPL scripts will need more control about parallel work?
Regards,
Markus
[Cocci] release: 1.0.2: Software evolution around parallelisation support
In reply to Julia LawallQuoted 9 lines
Quoted 6 lines
Does the parallelisation support need any more fine-tuning?
* How often is task preparation required before further data processing can be performed in parallel?* Which programming interfaces do you prefer for the storage of collected data in the fork-join work flow?
No idea what either of these questions mean, but the whole point is that there is no join.
This view needs an adjustment.
A fork-join work flow is implemented by the Coccinelle software for its main programs with the help of the application programming interface "parmap", isn't it?
https://rdicosmo.github.io/parmap/
The manual contains the following information for this approach.
* Preparation for spawning subprocesses:
"This option furthermore creates a temporary directory in the directory from which spatch is executed that has the name of the semantic patch (without its extension) and that contains stdout and stderr files generated by the various processes."
* Actions for joining:
"When the semantic patch completes, the contents of these files are printed to standard output and standard error, respectively, and the directory is removed."
I propose to reactivate the support for the SmPL rules "initialize" and "finalize" for all uses cases without unwanted software limitations.
I imagine that such rules can still be evaluated and executed at the appropriate data processing phases after a bit more software development even if the multi-programming technique is reused.
Regards,
Markus
[Cocci] release: 1.0.2: Software evolution around parallelisation support
Quoted 10 lines · SF Markus Elfring
Quoted 9 lines
Quoted 6 lines
Does the parallelisation support need any more fine-tuning?
* How often is task preparation required before further data processing can be performed in parallel?* Which programming interfaces do you prefer for the storage of collected data in the fork-join work flow?
No idea what either of these questions mean, but the whole point is that there is no join.
Quoted 6 lines
This view needs an adjustment.
A fork-join work flow is implemented by the Coccinelle software for its main programs with the help of the application programming interface "parmap", isn't it?
https://rdicosmo.github.io/parmap/
Parmap only forks. It can join what is returned explicitly. The problem is that the scripting language environment is separate from the environment of Coccinelle itself. So there is no easy way to get the right values into the right variables. Coccinelle doesn't know anything about the variables of the ocaml or python scripts.
Quoted 11 lines
The manual contains the following information for this approach.
* Preparation for spawning subprocesses:
"This option furthermore creates a temporary directory in the directory from which spatch is executed that has the name of the semantic patch (without its extension) and that contains stdout and stderr files generated by the various processes."* Actions for joining:
"When the semantic patch completes, the contents of these files are printed to standard output and standard error, respectively, and the directory is removed."
Yes but stndard output is a global thing.
Quoted 3 lines
I propose to reactivate the support for the SmPL rules "initialize" and "finalize" for all uses cases without unwanted software limitations.
There is no way that Coccinelle can tell what is a "use case with an unwanted software limitation. Coccinelle doesn't know what the scripting code does. I would prefer not to promise a behavior that is not implemented. As I said before, if an unshared initialize and finalize are acceptable to you, then just use the preexisting parallelism solution, which is still supported.
julia
[Cocci] release: 1.0.2: Software evolution around parallelisation support
Quoted 3 lines
The problem is that the scripting language environment is separate from the environment of Coccinelle itself. So there is no easy way to get the right values into the right variables.
I can also understand corresponding software development challennges to some degree.
Quoted 1 line
Coccinelle doesn't know anything about the variables of the ocaml or python scripts.
How do you think about possibilities to extend its knowledge? Can an improved software design help to achieve the needed data exchange?
Quoted 5 lines
Quoted 2 lines
I propose to reactivate the support for the SmPL rules "initialize" and "finalize" for all uses cases without unwanted software limitations.
There is no way that Coccinelle can tell what is a "use case with an unwanted software limitation.
Can the exclusion of these rules be seen as a temporary software restriction when the parameter "--jobs" to the program "spatch"?
Quoted 1 line
Coccinelle doesn't know what the scripting code does.
But you have still got control on the data which are transferred between the involved subprocesses.
Quoted 1 line
I would prefer not to promise a behavior that is not implemented.
I hope that corresponding technical concerns can be improved.
Quoted 2 lines
As I said before, if an unshared initialize and finalize are acceptable to you, then just use the preexisting parallelism solution, which is still supported.
I am using this approach with a make script for a while.
How do you think about variants for these actions at the "unshared" stage and the task/subprocess level?
Regards,
Markus
[Cocci] release: 1.0.2: Software evolution around parallelisation support
In reply to Julia LawallQuoted 1 line
There is no shared memory in ocaml.
How do you think about to reuse any other application programming interfaces for inter-process communication?
https://ocaml.github.io/ocamlunix/sockets.html
Can it be determined from the content of a semantic patch language script if the spawned subprocesses need any more data exchange (with the parent process) besides the usual files like standard output and standard error?
Regards,
Markus
[Cocci] release: 1.0.2: Software evolution around parallelisation support
Quoted 10 lines · SF Markus Elfring
Quoted 1 line
There is no shared memory in ocaml.
How do you think about to reuse any other application programming interfaces for inter-process communication?
https://ocaml.github.io/ocamlunix/sockets.htmlCan it be determined from the content of a semantic patch language script if the spawned subprocesses need any more data exchange (with the parent process) besides the usual files like standard output and standard error?
In the short term, there is no improvement planned in this direction. It is important that Coccinelle does not know anything about the script code
- we don't want to include parsers for other languages, and it would further raise the barrier for adding other scripting languages. If you need initialize and finalize, just use the current parallelism solution. You will get n results instead of 1 result, but it is better than getting 15 000 results, as would happen with parmap.
julia
[Cocci] release: 1.0.2: Software evolution around parallelisation support
Quoted 6 lines
Quoted 3 lines
Can it be determined from the content of a semantic patch language script if the spawned subprocesses need any more data exchange (with the parent process) besides the usual files like standard output and standard error?
In the short term, there is no improvement planned in this direction. It is important that Coccinelle does not know anything about the script code
Thanks for your feedback.
Quoted 1 line
- we don't want to include parsers for other languages,
I assume that might mean that no such extensions will be added in the near future. I guess that the scripting interfaces are generally extensible so that other programming languages can be integrated in principle if someone would be keen enough to invest further software development efforts.
Quoted 1 line
and it would further raise the barrier for adding other scripting languages.
The inherent system complexity was already increased just by the added parallelisation support with the parmap interface.
Quoted 1 line
If you need initialize and finalize, just use the current parallelism solution.
I am using this approach around the parameters "--index" and "--max" for a while. I came similar challenges along during the development of my make scripts.
I imagine that these SmPL rules can still be executed by the main process while the execution of the other rules will be delegated to subprocesses (or even threads). The scripting environments in the subprocesses would additionally check then which variables accesses should be transformed into data processing requests for the main process.
When would you like to pick corresponding software development challenges up around inter-process communication?
Quoted 2 lines
You will get n results instead of 1 result, but it is better than getting 15 000 results, as would happen with parmap.
There are some trade-offs involved. I find it nicer when the Coccinelle software can be directly reused for more use cases.
Regards,
Markus