exists do not work if file group is too big (>49)
When operating on too many files the exists keyword no longer work. For instance:
@exists@
expression E1;
@@
set_page_dirty(
+NULL,
E1)
Using linux kernel as playground and looking at at fs/f2fs/gc.c where there is set_page_dirty() calls in nested blocks.
Works:
spatch --no-includes --sp-file test.spatch arch/powerpc/kvm/book3s_64_mmu_radix.c arch/powerpc/kvm/e500_mmu.c arch/s390/kvm/interrupt.c arch/x86/kvm/svm.c block/bio.c drivers/block/zram/zram_drv.c drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c drivers/gpu/drm/drm_gem.c drivers/gpu/drm/exynos/exynos_drm_g2d.c drivers/gpu/drm/i915/i915_gem.c drivers/gpu/drm/i915/i915_gem_fence_reg.c drivers/gpu/drm/i915/i915_gem_userptr.c drivers/gpu/drm/radeon/radeon_ttm.c drivers/gpu/drm/ttm/ttm_tt.c drivers/infiniband/core/umem.c drivers/infiniband/core/umem_odp.c drivers/infiniband/hw/hfi1/user_pages.c drivers/infiniband/hw/qib/qib_user_pages.c drivers/infiniband/hw/usnic/usnic_uiom.c drivers/media/common/videobuf2/videobuf2-dma-contig.c drivers/media/common/videobuf2/videobuf2-dma-sg.c drivers/media/common/videobuf2/videobuf2-vmalloc.c drivers/misc/genwqe/card_utils.c drivers/misc/vmw_vmci/vmci_queue_pair.c drivers/mtd/devices/block2mtd.c drivers/platform/goldfish/goldfish_pipe.c drivers/sbus/char/oradax.c drivers/staging/lustre/lustre/llite/rw26.c drivers/staging/lustre/lustre/llite/vvp_io.c drivers/staging/vc04_services/interface/vchiq_arm/vchiq_2835_arm.c drivers/vhost/vhost.c drivers/video/fbdev/core/fb_defio.c fs/9p/vfs_addr.c fs/afs/dir.c fs/afs/file.c --include fs/afs/internal.h fs/afs/write.c fs/aio.c fs/block_dev.c fs/btrfs/disk-io.c fs/btrfs/extent_io.c fs/btrfs/file.c fs/btrfs/inode.c fs/btrfs/ioctl.c fs/btrfs/relocation.c fs/buffer.c fs/ceph/addr.c fs/f2fs/gc.c > toto
Don't works:
spatch --no-includes --sp-file test.spatch arch/powerpc/kvm/book3s_64_mmu_radix.c arch/powerpc/kvm/e500_mmu.c arch/s390/kvm/interrupt.c arch/x86/kvm/svm.c block/bio.c drivers/block/zram/zram_drv.c drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c drivers/gpu/drm/drm_gem.c drivers/gpu/drm/exynos/exynos_drm_g2d.c drivers/gpu/drm/i915/i915_gem.c drivers/gpu/drm/i915/i915_gem_fence_reg.c drivers/gpu/drm/i915/i915_gem_userptr.c drivers/gpu/drm/radeon/radeon_ttm.c drivers/gpu/drm/ttm/ttm_tt.c drivers/infiniband/core/umem.c drivers/infiniband/core/umem_odp.c drivers/infiniband/hw/hfi1/user_pages.c drivers/infiniband/hw/qib/qib_user_pages.c drivers/infiniband/hw/usnic/usnic_uiom.c drivers/media/common/videobuf2/videobuf2-dma-contig.c drivers/media/common/videobuf2/videobuf2-dma-sg.c drivers/media/common/videobuf2/videobuf2-vmalloc.c drivers/misc/genwqe/card_utils.c drivers/misc/vmw_vmci/vmci_queue_pair.c drivers/mtd/devices/block2mtd.c drivers/platform/goldfish/goldfish_pipe.c drivers/sbus/char/oradax.c drivers/staging/lustre/lustre/llite/rw26.c drivers/staging/lustre/lustre/llite/vvp_io.c drivers/staging/vc04_services/interface/vchiq_arm/vchiq_2835_arm.c drivers/vhost/vhost.c drivers/video/fbdev/core/fb_defio.c fs/9p/vfs_addr.c fs/afs/dir.c fs/afs/file.c --include fs/afs/internal.h fs/afs/write.c fs/aio.c fs/block_dev.c fs/btrfs/disk-io.c fs/btrfs/extent_io.c fs/btrfs/file.c fs/btrfs/inode.c fs/btrfs/ioctl.c fs/btrfs/relocation.c fs/buffer.c fs/ceph/addr.c fs/cifs/cifssmb.c fs/f2fs/gc.c > toto
Exact same command line except the latter have 49 files before fs/f2fs/gc.c while the former has 48 (and it works then). Note that it seems that running the script multiple times even when they are too many files will eventualy work.
Cheers,
J?r?me
Quoted 10 lines · Jerome Glisse
When operating on too many files the exists keyword no longer work. For instance:
@exists@
expression E1;
@@
set_page_dirty(
+NULL,
E1)
Quoted 6 lines
Using linux kernel as playground and looking at at
fs/f2fs/gc.cwhere there isset_page_dirty()calls in nested blocks.Works:
spatch --no-includes --sp-file test.spatch
arch/powerpc/kvm/book3s_64_mmu_radix.carch/powerpc/kvm/e500_mmu.carch/s390/kvm/interrupt.carch/x86/kvm/svm.cblock/bio.cdrivers/block/zram/zram_drv.cdrivers/gpu/drm/amd/amdgpu/amdgpu_ttm.cdrivers/gpu/drm/drm_gem.cdrivers/gpu/drm/exynos/exynos_drm_g2d.cdrivers/gpu/drm/i915/i915_gem.cdrivers/gpu/drm/i915/i915_gem_fence_reg.cdrivers/gpu/drm/i915/i915_gem_userptr.cdrivers/gpu/drm/radeon/radeon_ttm.cdrivers/gpu/drm/ttm/ttm_tt.cdrivers/infiniband/core/umem.cdrivers/infiniband/core/umem_odp.cdrivers/infiniband/hw/hfi1/user_pages.cdrivers/infiniband/hw/qib/qib_user_pages.cdrivers/infiniband/hw/usnic/usnic_uiom.cdrivers/media/common/videobuf2/videobuf2-dma-contig.cdrivers/media/common/videobuf2/videobuf2-dma-sg.cdrivers/media/common/videobuf2/videobuf2-vmalloc.cdrivers/misc/genwqe/card_utils.cdrivers/misc/vmw_vmci/vmci_queue_pair.cdrivers/mtd/devices/block2mtd.cdrivers/platform/goldfish/goldfish_pipe.cdrivers/sbus/!
char/oradax.c drivers/staging/lustre/lustre/llite/rw26.c drivers/staging/lustre/lustre/llite/vvp_io.c drivers/staging/vc04_services/interface/vchiq_arm/vchiq_2835_arm.c drivers/vhost/vhost.c drivers/video/fbdev/core/fb_defio.c fs/9p/vfs_addr.c fs/afs/dir.c fs/afs/file.c --include fs/afs/internal.h fs/afs/write.c fs/aio.c fs/block_dev.c fs/btrfs/disk-io.c fs/btrfs/extent_io.c fs/btrfs/file.c fs/btrfs/inode.c fs/btrfs/ioctl.c fs/btrfs/relocation.c fs/buffer.c fs/ceph/addr.c fs/f2fs/gc.c > toto
Quoted 5 lines
Don't works:
spatch --no-includes --sp-file test.spatch
arch/powerpc/kvm/book3s_64_mmu_radix.carch/powerpc/kvm/e500_mmu.carch/s390/kvm/interrupt.carch/x86/kvm/svm.cblock/bio.cdrivers/block/zram/zram_drv.cdrivers/gpu/drm/amd/amdgpu/amdgpu_ttm.cdrivers/gpu/drm/drm_gem.cdrivers/gpu/drm/exynos/exynos_drm_g2d.cdrivers/gpu/drm/i915/i915_gem.cdrivers/gpu/drm/i915/i915_gem_fence_reg.cdrivers/gpu/drm/i915/i915_gem_userptr.cdrivers/gpu/drm/radeon/radeon_ttm.cdrivers/gpu/drm/ttm/ttm_tt.cdrivers/infiniband/core/umem.cdrivers/infiniband/core/umem_odp.cdrivers/infiniband/hw/hfi1/user_pages.cdrivers/infiniband/hw/qib/qib_user_pages.cdrivers/infiniband/hw/usnic/usnic_uiom.cdrivers/media/common/videobuf2/videobuf2-dma-contig.cdrivers/media/common/videobuf2/videobuf2-dma-sg.cdrivers/media/common/videobuf2/videobuf2-vmalloc.cdrivers/misc/genwqe/card_utils.cdrivers/misc/vmw_vmci/vmci_queue_pair.cdrivers/mtd/devices/block2mtd.cdrivers/platform/goldfish/goldfish_pipe.cdrivers/sbus/!
char/oradax.c drivers/staging/lustre/lustre/llite/rw26.c drivers/staging/lustre/lustre/llite/vvp_io.c drivers/staging/vc04_services/interface/vchiq_arm/vchiq_2835_arm.c drivers/vhost/vhost.c drivers/video/fbdev/core/fb_defio.c fs/9p/vfs_addr.c fs/afs/dir.c fs/afs/file.c --include fs/afs/internal.h fs/afs/write.c fs/aio.c fs/block_dev.c fs/btrfs/disk-io.c fs/btrfs/extent_io.c fs/btrfs/file.c fs/btrfs/inode.c fs/btrfs/ioctl.c fs/btrfs/relocation.c fs/buffer.c fs/ceph/addr.c fs/cifs/cifssmb.c fs/f2fs/gc.c > toto
Quoted 6 lines
Exact same command line except the latter have 49 files before
fs/f2fs/gc.cwhile the former has 48 (and it works then). Note that it seems that running the script multiple times even when they are too many files will eventualy work.
I'm surprised that the number of files would have an impact, unless by not working you mean that it runs out of memory or doesn't terminate.
In general, you should only give one file name or one directory name to Coccinelle. If you give multiple files, then they are all treated at once, ie their CFGs are all in memory at the same time. If you give the name of a directory containing the files, then it will work on one file at a time, and you can give the option -j XXX to have it run XXX threads in parallel. Giving multiple file names on the command line is only for the case where information collected from one file is needed for the treatment of another file.
If the set of files you want to work on doesn't correspond to the directory structure, you can make a file with the names of the files to treat one by one each separated by a blank line, and then give the command line argument --file-groups filename.
Actually, exists is not relevant to your rule, because it has no ... Exists means that there exists a path through the ... that matches the pattern. Otherwise, all paths have to match the pattern.
julia
Quoted 8 lines
Cheers,
J?r?me
_______________________________________________
Cocci mailing list
Cocci at systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci
Quoted 29 lines · Julia Lawall
Quoted 10 lines · Jerome Glisse
When operating on too many files the exists keyword no longer work. For instance:
@exists@
expression E1;
@@
set_page_dirty(
+NULL,
E1)Quoted 6 lines
Using linux kernel as playground and looking at at
fs/f2fs/gc.cwhere there isset_page_dirty()calls in nested blocks.Works:
spatch --no-includes --sp-file test.spatch
arch/powerpc/kvm/book3s_64_mmu_radix.carch/powerpc/kvm/e500_mmu.carch/s390/kvm/interrupt.carch/x86/kvm/svm.cblock/bio.cdrivers/block/zram/zram_drv.cdrivers/gpu/drm/amd/amdgpu/amdgpu_ttm.cdrivers/gpu/drm/drm_gem.cdrivers/gpu/drm/exynos/exynos_drm_g2d.cdrivers/gpu/drm/i915/i915_gem.cdrivers/gpu/drm/i915/i915_gem_fence_reg.cdrivers/gpu/drm/i915/i915_gem_userptr.cdrivers/gpu/drm/radeon/radeon_ttm.cdrivers/gpu/drm/ttm/ttm_tt.cdrivers/infiniband/core/umem.cdrivers/infiniband/core/umem_odp.cdrivers/infiniband/hw/hfi1/user_pages.cdrivers/infiniband/hw/qib/qib_user_pages.cdrivers/infiniband/hw/usnic/usnic_uiom.cdrivers/media/common/videobuf2/videobuf2-dma-contig.cdrivers/media/common/videobuf2/videobuf2-dma-sg.cdrivers/media/common/videobuf2/videobuf2-vmalloc.cdrivers/misc/genwqe/card_utils.cdrivers/misc/vmw_vmci/vmci_queue_pair.cdrivers/mtd/devices/block2mtd.cdrivers/platform/goldfish/goldfish_pipe.cdrivers/sbus/char/orad
ax.cdrivers/staging/lustre/lustre/llite/rw26.cdrivers/staging/lustre/lustre/llite/vvp_io.cdrivers/staging/vc04_services/interface/vchiq_arm/vchiq_2835_arm.cdrivers/vhost/vhost.cdrivers/video/fbdev/core/fb_defio.cfs/9p/vfs_addr.cfs/afs/dir.cfs/afs/file.c--includefs/afs/internal.hfs/afs/write.cfs/aio.cfs/block_dev.cfs/btrfs/disk-io.cfs/btrfs/extent_io.cfs/btrfs/file.cfs/btrfs/inode.cfs/btrfs/ioctl.cfs/btrfs/relocation.cfs/buffer.cfs/ceph/addr.cfs/f2fs/gc.c> totoQuoted 5 lines
Don't works:
spatch --no-includes --sp-file test.spatch
arch/powerpc/kvm/book3s_64_mmu_radix.carch/powerpc/kvm/e500_mmu.carch/s390/kvm/interrupt.carch/x86/kvm/svm.cblock/bio.cdrivers/block/zram/zram_drv.cdrivers/gpu/drm/amd/amdgpu/amdgpu_ttm.cdrivers/gpu/drm/drm_gem.cdrivers/gpu/drm/exynos/exynos_drm_g2d.cdrivers/gpu/drm/i915/i915_gem.cdrivers/gpu/drm/i915/i915_gem_fence_reg.cdrivers/gpu/drm/i915/i915_gem_userptr.cdrivers/gpu/drm/radeon/radeon_ttm.cdrivers/gpu/drm/ttm/ttm_tt.cdrivers/infiniband/core/umem.cdrivers/infiniband/core/umem_odp.cdrivers/infiniband/hw/hfi1/user_pages.cdrivers/infiniband/hw/qib/qib_user_pages.cdrivers/infiniband/hw/usnic/usnic_uiom.cdrivers/media/common/videobuf2/videobuf2-dma-contig.cdrivers/media/common/videobuf2/videobuf2-dma-sg.cdrivers/media/common/videobuf2/videobuf2-vmalloc.cdrivers/misc/genwqe/card_utils.cdrivers/misc/vmw_vmci/vmci_queue_pair.cdrivers/mtd/devices/block2mtd.cdrivers/platform/goldfish/goldfish_pipe.cdrivers/sbus/char/orad
ax.cdrivers/staging/lustre/lustre/llite/rw26.cdrivers/staging/lustre/lustre/llite/vvp_io.cdrivers/staging/vc04_services/interface/vchiq_arm/vchiq_2835_arm.cdrivers/vhost/vhost.cdrivers/video/fbdev/core/fb_defio.cfs/9p/vfs_addr.cfs/afs/dir.cfs/afs/file.c--includefs/afs/internal.hfs/afs/write.cfs/aio.cfs/block_dev.cfs/btrfs/disk-io.cfs/btrfs/extent_io.cfs/btrfs/file.cfs/btrfs/inode.cfs/btrfs/ioctl.cfs/btrfs/relocation.cfs/buffer.cfs/ceph/addr.cfs/cifs/cifssmb.cfs/f2fs/gc.c> totoQuoted 2 lines
Quoted 7 lines
Quoted 4 lines
Exact same command line except the latter have 49 files before
fs/f2fs/gc.cwhile the former has 48 (and it works then). Note that it seems that running the script multiple times even when they are too many files will eventualy work.I'm surprised that the number of files would have an impact, unless by not working you mean that it runs out of memory or doesn't terminate.
Don't work == only set_page_dirty in non nested block are modified, set_page_dirty in nested block are not. So it runs and terminate and all files are processed (ie it is not a command line size limitation in the shell).
Quoted 9 lines
In general, you should only give one file name or one directory name to Coccinelle. If you give multiple files, then they are all treated at once, ie their CFGs are all in memory at the same time. If you give the name of a directory containing the files, then it will work on one file at a time, and you can give the option -j XXX to have it run XXX threads in parallel. Giving multiple file names on the command line is only for the case where information collected from one file is needed for the treatment of another file.
Saddly for modification i am doing i need to treat many files as one as otherwise it does not work and even sadder for me is that this groups can get pretty big.
Quoted 5 lines
If the set of files you want to work on doesn't correspond to the directory structure, you can make a file with the names of the files to treat one by one each separated by a blank line, and then give the command line argument --file-groups filename.
This is _slower_ (by an order of magnitude) and also spit out a lot of error messages (about trying to cd to directory and failing) than passing all the files on the command line.
Quoted 4 lines
Actually, exists is not relevant to your rule, because it has no ... Exists means that there exists a path through the ... that matches the pattern. Otherwise, all paths have to match the pattern.
Oh i thought i would need it, nonetheless when there is too many files the above snipet stop at first level blocks and never modify block with deeper nesting.
Thank you for all your feedback :)
Cheers,
J?r?me
Quoted 15 lines · Jerome Glisse
Quoted 14 lines · Julia Lawall
Quoted 10 lines · Jerome Glisse
When operating on too many files the exists keyword no longer work. For instance:
@exists@
expression E1;
@@
set_page_dirty(
+NULL,
E1)
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Using linux kernel as playground and looking at at
fs/f2fs/gc.cwhere there isset_page_dirty()calls in nested blocks.Works:
spatch --no-includes --sp-file test.spatch
arch/powerpc/kvm/book3s_64_mmu_radix.carch/powerpc/kvm/e500_mmu.carch/s390/kvm/interrupt.carch/x86/kvm/svm.cblock/bio.cdrivers/block/zram/zram_drv.cdrivers/gpu/drm/amd/amdgpu/amdgpu_ttm.cdrivers/gpu/drm/drm_gem.cdrivers/gpu/drm/exynos/exynos_drm_g2d.cdrivers/gpu/drm/i915/i915_gem.cdrivers/gpu/drm/i915/i915_gem_fence_reg.cdrivers/gpu/drm/i915/i915_gem_userptr.cdrivers/gpu/drm/radeon/radeon_ttm.cdrivers/gpu/drm/ttm/ttm_tt.cdrivers/infiniband/core/umem.cdrivers/infiniband/core/umem_odp.cdrivers/infiniband/hw/hfi1/user_pages.cdrivers/infiniband/hw/qib/qib_user_pages.cdrivers/infiniband/hw/usnic/usnic_uiom.cdrivers/media/common/videobuf2/videobuf2-dma-contig.cdrivers/media/common/videobuf2/videobuf2-dma-sg.cdrivers/media/common/videobuf2/videobuf2-vmalloc.cdrivers/misc/genwqe/card_utils.cdrivers/misc/vmw_vmci/vmci_queue_pair.cdrivers/mtd/devices/block2mtd.cdrivers/platform/goldfish/goldfish_pipe.cdrivers/s!
bus/char/orad
Quoted 6 lines
Quoted 6 lines
ax.cdrivers/staging/lustre/lustre/llite/rw26.cdrivers/staging/lustre/lustre/llite/vvp_io.cdrivers/staging/vc04_services/interface/vchiq_arm/vchiq_2835_arm.cdrivers/vhost/vhost.cdrivers/video/fbdev/core/fb_defio.cfs/9p/vfs_addr.cfs/afs/dir.cfs/afs/file.c--includefs/afs/internal.hfs/afs/write.cfs/aio.cfs/block_dev.cfs/btrfs/disk-io.cfs/btrfs/extent_io.cfs/btrfs/file.cfs/btrfs/inode.cfs/btrfs/ioctl.cfs/btrfs/relocation.cfs/buffer.cfs/ceph/addr.cfs/f2fs/gc.c> totoQuoted 5 lines
Don't works:
spatch --no-includes --sp-file test.spatch
arch/powerpc/kvm/book3s_64_mmu_radix.carch/powerpc/kvm/e500_mmu.carch/s390/kvm/interrupt.carch/x86/kvm/svm.cblock/bio.cdrivers/block/zram/zram_drv.cdrivers/gpu/drm/amd/amdgpu/amdgpu_ttm.cdrivers/gpu/drm/drm_gem.cdrivers/gpu/drm/exynos/exynos_drm_g2d.cdrivers/gpu/drm/i915/i915_gem.cdrivers/gpu/drm/i915/i915_gem_fence_reg.cdrivers/gpu/drm/i915/i915_gem_userptr.cdrivers/gpu/drm/radeon/radeon_ttm.cdrivers/gpu/drm/ttm/ttm_tt.cdrivers/infiniband/core/umem.cdrivers/infiniband/core/umem_odp.cdrivers/infiniband/hw/hfi1/user_pages.cdrivers/infiniband/hw/qib/qib_user_pages.cdrivers/infiniband/hw/usnic/usnic_uiom.cdrivers/media/common/videobuf2/videobuf2-dma-contig.cdrivers/media/common/videobuf2/videobuf2-dma-sg.cdrivers/media/common/videobuf2/videobuf2-vmalloc.cdrivers/misc/genwqe/card_utils.cdrivers/misc/vmw_vmci/vmci_queue_pair.cdrivers/mtd/devices/block2mtd.cdrivers/platform/goldfish/goldfish_pipe.cdrivers/s!
bus/char/orad
Quoted 45 lines
Quoted 10 lines
ax.cdrivers/staging/lustre/lustre/llite/rw26.cdrivers/staging/lustre/lustre/llite/vvp_io.cdrivers/staging/vc04_services/interface/vchiq_arm/vchiq_2835_arm.cdrivers/vhost/vhost.cdrivers/video/fbdev/core/fb_defio.cfs/9p/vfs_addr.cfs/afs/dir.cfs/afs/file.c--includefs/afs/internal.hfs/afs/write.cfs/aio.cfs/block_dev.cfs/btrfs/disk-io.cfs/btrfs/extent_io.cfs/btrfs/file.cfs/btrfs/inode.cfs/btrfs/ioctl.cfs/btrfs/relocation.cfs/buffer.cfs/ceph/addr.cfs/cifs/cifssmb.cfs/f2fs/gc.c> totoQuoted 6 lines
Exact same command line except the latter have 49 files before
fs/f2fs/gc.cwhile the former has 48 (and it works then). Note that it seems that running the script multiple times even when they are too many files will eventualy work.I'm surprised that the number of files would have an impact, unless by not working you mean that it runs out of memory or doesn't terminate.
Don't work == only
set_page_dirtyin non nested block are modified,set_page_dirtyin nested block are not. So it runs and terminate and all files are processed (ie it is not a command line size limitation in the shell).Quoted 9 lines
In general, you should only give one file name or one directory name to Coccinelle. If you give multiple files, then they are all treated at once, ie their CFGs are all in memory at the same time. If you give the name of a directory containing the files, then it will work on one file at a time, and you can give the option -j XXX to have it run XXX threads in parallel. Giving multiple file names on the command line is only for the case where information collected from one file is needed for the treatment of another file.
Saddly for modification i am doing i need to treat many files as one as otherwise it does not work and even sadder for me is that this groups can get pretty big.
Quoted 5 lines
If the set of files you want to work on doesn't correspond to the directory structure, you can make a file with the names of the files to treat one by one each separated by a blank line, and then give the command line argument --file-groups filename.
This is _slower_ (by an order of magnitude) and also spit out a lot of error messages (about trying to cd to directory and failing) than passing all the files on the command line.
Quoted 4 lines
Actually, exists is not relevant to your rule, because it has no ... Exists means that there exists a path through the ... that matches the pattern. Otherwise, all paths have to match the pattern.
Quoted 5 lines
Oh i thought i would need it, nonetheless when there is too many files the above snipet stop at first level blocks and never modify block with deeper nesting.
Thank you for all your feedback :)
Sorry, but none of what you are saying makes any sense to me. Coccinele works one function at a time. There is no reason why its behavior on nested calls would be affected by the number of files. And no reason why working on one file at a time would be slower than working on all of the files at once. I'll have to try your command lines, but I probably can't do that until tomorrow afternoon...
julia
Quoted 82 lines · Julia Lawall
Quoted 15 lines · Jerome Glisse
Quoted 14 lines · Julia Lawall
Quoted 10 lines · Jerome Glisse
When operating on too many files the exists keyword no longer work. For instance:
@exists@
expression E1;
@@
set_page_dirty(
+NULL,
E1)Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Using linux kernel as playground and looking at at
fs/f2fs/gc.cwhere there isset_page_dirty()calls in nested blocks.Works:
spatch --no-includes --sp-file test.spatch
arch/powerpc/kvm/book3s_64_mmu_radix.carch/powerpc/kvm/e500_mmu.carch/s390/kvm/interrupt.carch/x86/kvm/svm.cblock/bio.cdrivers/block/zram/zram_drv.cdrivers/gpu/drm/amd/amdgpu/amdgpu_ttm.cdrivers/gpu/drm/drm_gem.cdrivers/gpu/drm/exynos/exynos_drm_g2d.cdrivers/gpu/drm/i915/i915_gem.cdrivers/gpu/drm/i915/i915_gem_fence_reg.cdrivers/gpu/drm/i915/i915_gem_userptr.cdrivers/gpu/drm/radeon/radeon_ttm.cdrivers/gpu/drm/ttm/ttm_tt.cdrivers/infiniband/core/umem.cdrivers/infiniband/core/umem_odp.cdrivers/infiniband/hw/hfi1/user_pages.cdrivers/infiniband/hw/qib/qib_user_pages.cdrivers/infiniband/hw/usnic/usnic_uiom.cdrivers/media/common/videobuf2/videobuf2-dma-contig.cdrivers/media/common/videobuf2/videobuf2-dma-sg.cdrivers/media/common/videobuf2/videobuf2-vmalloc.cdrivers/misc/genwqe/card_utils.cdrivers/misc/vmw_vmci/vmci_queue_pair.cdrivers/mtd/devices/block2mtd.cdrivers/platform/goldfish/goldfish_pipe.cdrivers/sbus/char/orad
Quoted 6 lines
Quoted 6 lines
ax.cdrivers/staging/lustre/lustre/llite/rw26.cdrivers/staging/lustre/lustre/llite/vvp_io.cdrivers/staging/vc04_services/interface/vchiq_arm/vchiq_2835_arm.cdrivers/vhost/vhost.cdrivers/video/fbdev/core/fb_defio.cfs/9p/vfs_addr.cfs/afs/dir.cfs/afs/file.c--includefs/afs/internal.hfs/afs/write.cfs/aio.cfs/block_dev.cfs/btrfs/disk-io.cfs/btrfs/extent_io.cfs/btrfs/file.cfs/btrfs/inode.cfs/btrfs/ioctl.cfs/btrfs/relocation.cfs/buffer.cfs/ceph/addr.cfs/f2fs/gc.c> totoQuoted 5 lines
Don't works:
spatch --no-includes --sp-file test.spatch
arch/powerpc/kvm/book3s_64_mmu_radix.carch/powerpc/kvm/e500_mmu.carch/s390/kvm/interrupt.carch/x86/kvm/svm.cblock/bio.cdrivers/block/zram/zram_drv.cdrivers/gpu/drm/amd/amdgpu/amdgpu_ttm.cdrivers/gpu/drm/drm_gem.cdrivers/gpu/drm/exynos/exynos_drm_g2d.cdrivers/gpu/drm/i915/i915_gem.cdrivers/gpu/drm/i915/i915_gem_fence_reg.cdrivers/gpu/drm/i915/i915_gem_userptr.cdrivers/gpu/drm/radeon/radeon_ttm.cdrivers/gpu/drm/ttm/ttm_tt.cdrivers/infiniband/core/umem.cdrivers/infiniband/core/umem_odp.cdrivers/infiniband/hw/hfi1/user_pages.cdrivers/infiniband/hw/qib/qib_user_pages.cdrivers/infiniband/hw/usnic/usnic_uiom.cdrivers/media/common/videobuf2/videobuf2-dma-contig.cdrivers/media/common/videobuf2/videobuf2-dma-sg.cdrivers/media/common/videobuf2/videobuf2-vmalloc.cdrivers/misc/genwqe/card_utils.cdrivers/misc/vmw_vmci/vmci_queue_pair.cdrivers/mtd/devices/block2mtd.cdrivers/platform/goldfish/goldfish_pipe.cdrivers/sbus/char/orad
Quoted 49 lines
Quoted 10 lines
ax.cdrivers/staging/lustre/lustre/llite/rw26.cdrivers/staging/lustre/lustre/llite/vvp_io.cdrivers/staging/vc04_services/interface/vchiq_arm/vchiq_2835_arm.cdrivers/vhost/vhost.cdrivers/video/fbdev/core/fb_defio.cfs/9p/vfs_addr.cfs/afs/dir.cfs/afs/file.c--includefs/afs/internal.hfs/afs/write.cfs/aio.cfs/block_dev.cfs/btrfs/disk-io.cfs/btrfs/extent_io.cfs/btrfs/file.cfs/btrfs/inode.cfs/btrfs/ioctl.cfs/btrfs/relocation.cfs/buffer.cfs/ceph/addr.cfs/cifs/cifssmb.cfs/f2fs/gc.c> totoQuoted 6 lines
Exact same command line except the latter have 49 files before
fs/f2fs/gc.cwhile the former has 48 (and it works then). Note that it seems that running the script multiple times even when they are too many files will eventualy work.I'm surprised that the number of files would have an impact, unless by not working you mean that it runs out of memory or doesn't terminate.
Don't work == only
set_page_dirtyin non nested block are modified,set_page_dirtyin nested block are not. So it runs and terminate and all files are processed (ie it is not a command line size limitation in the shell).Quoted 9 lines
In general, you should only give one file name or one directory name to Coccinelle. If you give multiple files, then they are all treated at once, ie their CFGs are all in memory at the same time. If you give the name of a directory containing the files, then it will work on one file at a time, and you can give the option -j XXX to have it run XXX threads in parallel. Giving multiple file names on the command line is only for the case where information collected from one file is needed for the treatment of another file.
Saddly for modification i am doing i need to treat many files as one as otherwise it does not work and even sadder for me is that this groups can get pretty big.
Quoted 5 lines
If the set of files you want to work on doesn't correspond to the directory structure, you can make a file with the names of the files to treat one by one each separated by a blank line, and then give the command line argument --file-groups filename.
This is _slower_ (by an order of magnitude) and also spit out a lot of error messages (about trying to cd to directory and failing) than passing all the files on the command line.
Quoted 4 lines
Actually, exists is not relevant to your rule, because it has no ... Exists means that there exists a path through the ... that matches the pattern. Otherwise, all paths have to match the pattern.
Oh i thought i would need it, nonetheless when there is too many files the above snipet stop at first level blocks and never modify block with deeper nesting.
Quoted 8 lines
Quoted 1 line
Thank you for all your feedback :)
Sorry, but none of what you are saying makes any sense to me. Coccinele works one function at a time. There is no reason why its behavior on nested calls would be affected by the number of files. And no reason why working on one file at a time would be slower than working on all of the files at once. I'll have to try your command lines, but I probably can't do that until tomorrow afternoon...
Yes this is weird bug i actually tested again and it only shows if exists is use. If i remove exists like you adviced it seems to work no matter how many files i pass as argument.
When the rules has the exists modifier then it only modify inside nested block if i pass 48 files or less. If i pass 49 files or more then nested block are not modified but first level block is.
My buest guess is some memory corruption somewhere or some variable that is not reset properly.
I am on version 1.0.6 (fedora 27).
Note i am in no rush, i just wanted to report this as it is likely a bug somewhere moreover i have a work around.
Cheers,
J?r?me
Quoted 20 lines · Jerome Glisse
Quoted 19 lines · Julia Lawall
Quoted 15 lines · Jerome Glisse
Quoted 14 lines · Julia Lawall
Quoted 10 lines · Jerome Glisse
When operating on too many files the exists keyword no longer work. For instance:
@exists@
expression E1;
@@
set_page_dirty(
+NULL,
E1)
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Using linux kernel as playground and looking at at
fs/f2fs/gc.cwhere there isset_page_dirty()calls in nested blocks.Works:
spatch --no-includes --sp-file test.spatch
arch/powerpc/kvm/book3s_64_mmu_radix.carch/powerpc/kvm/e500_mmu.carch/s390/kvm/interrupt.carch/x86/kvm/svm.cblock/bio.cdrivers/block/zram/zram_drv.cdrivers/gpu/drm/amd/amdgpu/amdgpu_ttm.cdrivers/gpu/drm/drm_gem.cdrivers/gpu/drm/exynos/exynos_drm_g2d.cdrivers/gpu/drm/i915/i915_gem.cdrivers/gpu/drm/i915/i915_gem_fence_reg.cdrivers/gpu/drm/i915/i915_gem_userptr.cdrivers/gpu/drm/radeon/radeon_ttm.cdrivers/gpu/drm/ttm/ttm_tt.cdrivers/infiniband/core/umem.cdrivers/infiniband/core/umem_odp.cdrivers/infiniband/hw/hfi1/user_pages.cdrivers/infiniband/hw/qib/qib_user_pages.cdrivers/infiniband/hw/usnic/usnic_uiom.cdrivers/media/common/videobuf2/videobuf2-dma-contig.cdrivers/media/common/videobuf2/videobuf2-dma-sg.cdrivers/media/common/videobuf2/videobuf2-vmalloc.cdrivers/misc/genwqe/card_utils.cdrivers/misc/vmw_vmci/vmci_queue_pair.cdrivers/mtd/devices/block2mtd.cdrivers/platform/goldfish/goldfish_pipe.cdrive!
rs/sbus/char/
Quoted 7 lines
Quoted 7 lines
orad
Quoted 6 lines
Quoted 6 lines
ax.cdrivers/staging/lustre/lustre/llite/rw26.cdrivers/staging/lustre/lustre/llite/vvp_io.cdrivers/staging/vc04_services/interface/vchiq_arm/vchiq_2835_arm.cdrivers/vhost/vhost.cdrivers/video/fbdev/core/fb_defio.cfs/9p/vfs_addr.cfs/afs/dir.cfs/afs/file.c--includefs/afs/internal.hfs/afs/write.cfs/aio.cfs/block_dev.cfs/btrfs/disk-io.cfs/btrfs/extent_io.cfs/btrfs/file.cfs/btrfs/inode.cfs/btrfs/ioctl.cfs/btrfs/relocation.cfs/buffer.cfs/ceph/addr.cfs/f2fs/gc.c> totoQuoted 5 lines
Don't works:
spatch --no-includes --sp-file test.spatch
arch/powerpc/kvm/book3s_64_mmu_radix.carch/powerpc/kvm/e500_mmu.carch/s390/kvm/interrupt.carch/x86/kvm/svm.cblock/bio.cdrivers/block/zram/zram_drv.cdrivers/gpu/drm/amd/amdgpu/amdgpu_ttm.cdrivers/gpu/drm/drm_gem.cdrivers/gpu/drm/exynos/exynos_drm_g2d.cdrivers/gpu/drm/i915/i915_gem.cdrivers/gpu/drm/i915/i915_gem_fence_reg.cdrivers/gpu/drm/i915/i915_gem_userptr.cdrivers/gpu/drm/radeon/radeon_ttm.cdrivers/gpu/drm/ttm/ttm_tt.cdrivers/infiniband/core/umem.cdrivers/infiniband/core/umem_odp.cdrivers/infiniband/hw/hfi1/user_pages.cdrivers/infiniband/hw/qib/qib_user_pages.cdrivers/infiniband/hw/usnic/usnic_uiom.cdrivers/media/common/videobuf2/videobuf2-dma-contig.cdrivers/media/common/videobuf2/videobuf2-dma-sg.cdrivers/media/common/videobuf2/videobuf2-vmalloc.cdrivers/misc/genwqe/card_utils.cdrivers/misc/vmw_vmci/vmci_queue_pair.cdrivers/mtd/devices/block2mtd.cdrivers/platform/goldfish/goldfish_pipe.cdrive!
rs/sbus/char/
Quoted 63 lines
Quoted 50 lines
orad
Quoted 49 lines
Quoted 10 lines
ax.cdrivers/staging/lustre/lustre/llite/rw26.cdrivers/staging/lustre/lustre/llite/vvp_io.cdrivers/staging/vc04_services/interface/vchiq_arm/vchiq_2835_arm.cdrivers/vhost/vhost.cdrivers/video/fbdev/core/fb_defio.cfs/9p/vfs_addr.cfs/afs/dir.cfs/afs/file.c--includefs/afs/internal.hfs/afs/write.cfs/aio.cfs/block_dev.cfs/btrfs/disk-io.cfs/btrfs/extent_io.cfs/btrfs/file.cfs/btrfs/inode.cfs/btrfs/ioctl.cfs/btrfs/relocation.cfs/buffer.cfs/ceph/addr.cfs/cifs/cifssmb.cfs/f2fs/gc.c> totoQuoted 6 lines
Exact same command line except the latter have 49 files before
fs/f2fs/gc.cwhile the former has 48 (and it works then). Note that it seems that running the script multiple times even when they are too many files will eventualy work.I'm surprised that the number of files would have an impact, unless by not working you mean that it runs out of memory or doesn't terminate.
Don't work == only
set_page_dirtyin non nested block are modified,set_page_dirtyin nested block are not. So it runs and terminate and all files are processed (ie it is not a command line size limitation in the shell).Quoted 9 lines
In general, you should only give one file name or one directory name to Coccinelle. If you give multiple files, then they are all treated at once, ie their CFGs are all in memory at the same time. If you give the name of a directory containing the files, then it will work on one file at a time, and you can give the option -j XXX to have it run XXX threads in parallel. Giving multiple file names on the command line is only for the case where information collected from one file is needed for the treatment of another file.
Saddly for modification i am doing i need to treat many files as one as otherwise it does not work and even sadder for me is that this groups can get pretty big.
Quoted 5 lines
If the set of files you want to work on doesn't correspond to the directory structure, you can make a file with the names of the files to treat one by one each separated by a blank line, and then give the command line argument --file-groups filename.
This is _slower_ (by an order of magnitude) and also spit out a lot of error messages (about trying to cd to directory and failing) than passing all the files on the command line.
Quoted 4 lines
Actually, exists is not relevant to your rule, because it has no ... Exists means that there exists a path through the ... that matches the pattern. Otherwise, all paths have to match the pattern.
Oh i thought i would need it, nonetheless when there is too many files the above snipet stop at first level blocks and never modify block with deeper nesting.
Quoted 8 lines
Quoted 1 line
Thank you for all your feedback :)
Sorry, but none of what you are saying makes any sense to me. Coccinele works one function at a time. There is no reason why its behavior on nested calls would be affected by the number of files. And no reason why working on one file at a time would be slower than working on all of the files at once. I'll have to try your command lines, but I probably can't do that until tomorrow afternoon...
Yes this is weird bug i actually tested again and it only shows if exists is use. If i remove exists like you adviced it seems to work no matter how many files i pass as argument.
Quoted 6 lines
When the rules has the exists modifier then it only modify inside nested block if i pass 48 files or less. If i pass 49 files or more then nested block are not modified but first level block is.
My buest guess is some memory corruption somewhere or some variable that is not reset properly.
Are you using the bytecode or the native code version of Coccinelle?
I think you can say make byte-only to get the bytecode version, from the sources on github.
julia
Quoted 9 lines
I am on version 1.0.6 (fedora 27).
Note i am in no rush, i just wanted to report this as it is likely a bug somewhere moreover i have a work around.
Cheers,
J?r?me
Quoted 106 lines · Julia Lawall
Quoted 20 lines · Jerome Glisse
Quoted 19 lines · Julia Lawall
Quoted 15 lines · Jerome Glisse
Quoted 14 lines · Julia Lawall
Quoted 10 lines · Jerome Glisse
When operating on too many files the exists keyword no longer work. For instance:
@exists@
expression E1;
@@
set_page_dirty(
+NULL,
E1)Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Using linux kernel as playground and looking at at
fs/f2fs/gc.cwhere there isset_page_dirty()calls in nested blocks.Works:
spatch --no-includes --sp-file test.spatch
arch/powerpc/kvm/book3s_64_mmu_radix.carch/powerpc/kvm/e500_mmu.carch/s390/kvm/interrupt.carch/x86/kvm/svm.cblock/bio.cdrivers/block/zram/zram_drv.cdrivers/gpu/drm/amd/amdgpu/amdgpu_ttm.cdrivers/gpu/drm/drm_gem.cdrivers/gpu/drm/exynos/exynos_drm_g2d.cdrivers/gpu/drm/i915/i915_gem.cdrivers/gpu/drm/i915/i915_gem_fence_reg.cdrivers/gpu/drm/i915/i915_gem_userptr.cdrivers/gpu/drm/radeon/radeon_ttm.cdrivers/gpu/drm/ttm/ttm_tt.cdrivers/infiniband/core/umem.cdrivers/infiniband/core/umem_odp.cdrivers/infiniband/hw/hfi1/user_pages.cdrivers/infiniband/hw/qib/qib_user_pages.cdrivers/infiniband/hw/usnic/usnic_uiom.cdrivers/media/common/videobuf2/videobuf2-dma-contig.cdrivers/media/common/videobuf2/videobuf2-dma-sg.cdrivers/media/common/videobuf2/videobuf2-vmalloc.cdrivers/misc/genwqe/card_utils.cdrivers/misc/vmw_vmci/vmci_queue_pair.cdrivers/mtd/devices/block2mtd.cdrivers/platform/goldfish/goldfish_pipe.cdrivers/sbus/char/
Quoted 7 lines
Quoted 7 lines
orad
Quoted 6 lines
Quoted 6 lines
ax.cdrivers/staging/lustre/lustre/llite/rw26.cdrivers/staging/lustre/lustre/llite/vvp_io.cdrivers/staging/vc04_services/interface/vchiq_arm/vchiq_2835_arm.cdrivers/vhost/vhost.cdrivers/video/fbdev/core/fb_defio.cfs/9p/vfs_addr.cfs/afs/dir.cfs/afs/file.c--includefs/afs/internal.hfs/afs/write.cfs/aio.cfs/block_dev.cfs/btrfs/disk-io.cfs/btrfs/extent_io.cfs/btrfs/file.cfs/btrfs/inode.cfs/btrfs/ioctl.cfs/btrfs/relocation.cfs/buffer.cfs/ceph/addr.cfs/f2fs/gc.c> totoQuoted 5 lines
Don't works:
spatch --no-includes --sp-file test.spatch
arch/powerpc/kvm/book3s_64_mmu_radix.carch/powerpc/kvm/e500_mmu.carch/s390/kvm/interrupt.carch/x86/kvm/svm.cblock/bio.cdrivers/block/zram/zram_drv.cdrivers/gpu/drm/amd/amdgpu/amdgpu_ttm.cdrivers/gpu/drm/drm_gem.cdrivers/gpu/drm/exynos/exynos_drm_g2d.cdrivers/gpu/drm/i915/i915_gem.cdrivers/gpu/drm/i915/i915_gem_fence_reg.cdrivers/gpu/drm/i915/i915_gem_userptr.cdrivers/gpu/drm/radeon/radeon_ttm.cdrivers/gpu/drm/ttm/ttm_tt.cdrivers/infiniband/core/umem.cdrivers/infiniband/core/umem_odp.cdrivers/infiniband/hw/hfi1/user_pages.cdrivers/infiniband/hw/qib/qib_user_pages.cdrivers/infiniband/hw/usnic/usnic_uiom.cdrivers/media/common/videobuf2/videobuf2-dma-contig.cdrivers/media/common/videobuf2/videobuf2-dma-sg.cdrivers/media/common/videobuf2/videobuf2-vmalloc.cdrivers/misc/genwqe/card_utils.cdrivers/misc/vmw_vmci/vmci_queue_pair.cdrivers/mtd/devices/block2mtd.cdrivers/platform/goldfish/goldfish_pipe.cdrivers/sbus/char/
Quoted 67 lines
Quoted 50 lines
orad
Quoted 49 lines
Quoted 10 lines
ax.cdrivers/staging/lustre/lustre/llite/rw26.cdrivers/staging/lustre/lustre/llite/vvp_io.cdrivers/staging/vc04_services/interface/vchiq_arm/vchiq_2835_arm.cdrivers/vhost/vhost.cdrivers/video/fbdev/core/fb_defio.cfs/9p/vfs_addr.cfs/afs/dir.cfs/afs/file.c--includefs/afs/internal.hfs/afs/write.cfs/aio.cfs/block_dev.cfs/btrfs/disk-io.cfs/btrfs/extent_io.cfs/btrfs/file.cfs/btrfs/inode.cfs/btrfs/ioctl.cfs/btrfs/relocation.cfs/buffer.cfs/ceph/addr.cfs/cifs/cifssmb.cfs/f2fs/gc.c> totoQuoted 6 lines
Exact same command line except the latter have 49 files before
fs/f2fs/gc.cwhile the former has 48 (and it works then). Note that it seems that running the script multiple times even when they are too many files will eventualy work.I'm surprised that the number of files would have an impact, unless by not working you mean that it runs out of memory or doesn't terminate.
Don't work == only
set_page_dirtyin non nested block are modified,set_page_dirtyin nested block are not. So it runs and terminate and all files are processed (ie it is not a command line size limitation in the shell).Quoted 9 lines
In general, you should only give one file name or one directory name to Coccinelle. If you give multiple files, then they are all treated at once, ie their CFGs are all in memory at the same time. If you give the name of a directory containing the files, then it will work on one file at a time, and you can give the option -j XXX to have it run XXX threads in parallel. Giving multiple file names on the command line is only for the case where information collected from one file is needed for the treatment of another file.
Saddly for modification i am doing i need to treat many files as one as otherwise it does not work and even sadder for me is that this groups can get pretty big.
Quoted 5 lines
If the set of files you want to work on doesn't correspond to the directory structure, you can make a file with the names of the files to treat one by one each separated by a blank line, and then give the command line argument --file-groups filename.
This is _slower_ (by an order of magnitude) and also spit out a lot of error messages (about trying to cd to directory and failing) than passing all the files on the command line.
Quoted 4 lines
Actually, exists is not relevant to your rule, because it has no ... Exists means that there exists a path through the ... that matches the pattern. Otherwise, all paths have to match the pattern.
Oh i thought i would need it, nonetheless when there is too many files the above snipet stop at first level blocks and never modify block with deeper nesting.
Quoted 8 lines
Quoted 1 line
Thank you for all your feedback :)
Sorry, but none of what you are saying makes any sense to me. Coccinele works one function at a time. There is no reason why its behavior on nested calls would be affected by the number of files. And no reason why working on one file at a time would be slower than working on all of the files at once. I'll have to try your command lines, but I probably can't do that until tomorrow afternoon...
Yes this is weird bug i actually tested again and it only shows if exists is use. If i remove exists like you adviced it seems to work no matter how many files i pass as argument.
When the rules has the exists modifier then it only modify inside nested block if i pass 48 files or less. If i pass 49 files or more then nested block are not modified but first level block is.
Quoted 7 lines
Quoted 2 lines
My buest guess is some memory corruption somewhere or some variable that is not reset properly.
Are you using the bytecode or the native code version of Coccinelle?
I think you can say make byte-only to get the bytecode version, from the sources on github.
Not sure what that is here is the output of spatch --version on fedora 27:
[glisse at localhost ~]$ spatch --version
spatch version 1.0.6 compiled with OCaml version 4.05.0
Flags passed to the configure script: --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --enable-release=yes --with-python=/usr/bin/python3 --with-menhir=/usr/bin/menhir Python scripting support: yes
Syntax of regular expresssions: PCRE
Cheers,
J?r?me
Quoted 5 lines
[glisse at localhost ~]$ spatch --version
spatch version 1.0.6 compiled with OCaml version 4.05.0
Flags passed to the configure script: --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --enable-release=yes --with-python=/usr/bin/python3 --with-menhir=/usr/bin/menhir Python scripting support: yes
Syntax of regular expresssions: PCRE
I don't think this indicates whether it is bytecode or native code, and I don't know what the fedora people offer.
My bytecode executable is 41M and my native code executable is 13M.
julia
Quoted 10 lines · Julia Lawall
Quoted 5 lines
[glisse at localhost ~]$ spatch --version
spatch version 1.0.6 compiled with OCaml version 4.05.0
Flags passed to the configure script: --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --enable-release=yes --with-python=/usr/bin/python3 --with-menhir=/usr/bin/menhir Python scripting support: yes
Syntax of regular expresssions: PCREI don't think this indicates whether it is bytecode or native code, and I don't know what the fedora people offer.
My bytecode executable is 41M and my native code executable is 13M.
I am guessing native code:
-rwxr-xr-x. 1 root root 1.3K May 10 13:51 /usr/bin/spatch
J?r?me
A message that is not in the archive (sent off-list or not yet received): [email protected]
[Cocci] Checking consequences from “exists” usage with big file name selection
Quoted 5 lines · SF Markus Elfring
Quoted 1 line
Note i am in no rush, i just wanted to report this as it is likely a bug somewhere
Thanks for your description of a strange software behaviour.
How often do you work with the specification ?exists? in other SmPL scripts?
I need to review all of them but i think i can remove most of them now that Julia explained that i do not need them if there is no ... between first match and second match in my rules.
Quoted 5 lines
Quoted 3 lines
moreover i have a work around.
Is the other transformation approach the solution which is really desired?
My work around was to add function with nested block to an extra group on which i run the same semantic patch again to take care of nested block when the original group as two big.
This is a minor inconvenience now that i have found why my semantic patch was not modifying nested block (took me a while to converge on the number of files as root of the issue).
Cheers,
J?r?me
A message that is not in the archive (sent off-list or not yet received): [email protected]
[Cocci] exists do not work if file group is too big
Quoted 5 lines · SF Markus Elfring
Quoted 1 line
When operating on too many files the exists keyword no longer work.
How do you think about to show a more precise error message for one approach to add the preprocessor symbol ?NULL? as the first parameter in a call of the function ?set_page_dirty??
There is no error message in the issue i reported. Nested block are just silently no longer modified just because there is too many files.
Quoted 5 lines
Quoted 3 lines
spatch ? --include
fs/afs/internal.h?* I find the position of this command parameter unusual.
I am using git grep to find files which reference the symbols i am modifying and i am doing a bit of sed regular expression on its out- put to prepend header files with --include
In the end i probably will have 40-50 semantics patches and i do not wish to create by hand a list of files to modify especialy as if i do so then i loose the big selling point of coccinelle ie not having to worry about rebasing patches if things move around.
Quoted 3 lines
* Are you sure that you would like to adjust the selected source files by a single command?
So as i am modifying a lot of functions which are close in sources files, i am getting error with --in-place with headers files, i could spit a patch and use patch -r - -f ... to force thing but this not only complexify the whole set of commands needed to run, it also fails in more sublte way (when doing twice the same modification which can lead to missing chunk modification).
This is why i am gretting groups of files to modify one function at a time and i also pass the function name to coccinelle using -D fn=name
I have try many ways to do this and after coupls weeks of iterations this is the cleanest simplest, fastest solution i converged on.
Quoted 2 lines
* Would you like to pass any file name to a command variant in a loop?
I would not see the benefit in that, i use --no-includes in my command line and modification are pretty fast so i am roughly happy as it is :)
Hopes this feedback helps. My biggest issue with coccinelle is the lack of support for function pointer typedef. Otherwise it is a pretty smooth experience. I have also an issue with python execution but this might be a fedora packaging issue, i have a local patch that make it works in fedora 27 but i need to check coccinelle git first to see if it is still an issue in lastest version. I need to inspect fedora spec file too.
Big kudos to everyone working on this tools.
Cheers,
J?r?me
[Cocci] exists do not work if file group is too big
In terms of the running time, I get a running time of 11 seconds with the command line of 48 files and I get a running time of 22 seconds if I just run spatch on the entire kernel. That seems slower, but if I use the command line of 48 files, I get changes in 27 files, while if I run it on the entire kernel, I get changes in 75 files. If I run it on the entire kernel using 40 cores (-j 40), I get changes in 75 files in 1.7 seconds.
julia
[Cocci] exists do not work if file group is too big
Quoted 6 lines · Julia Lawall
In terms of the running time, I get a running time of 11 seconds with the command line of 48 files and I get a running time of 22 seconds if I just run spatch on the entire kernel. That seems slower, but if I use the command line of 48 files, I get changes in 27 files, while if I run it on the entire kernel, I get changes in 75 files. If I run it on the entire kernel using 40 cores (-j 40), I get changes in 75 files in 1.7 seconds.
In this simple test case this would work but in my real spatch i have to change one function at a time as otherwise there is conflict on header update. Moreover i also have to use --recursive-headers if i don't do the git grep to provide file list which is also one of the reasons why it was much slower.
Ideally if header files could be updated --in-place even when there is conflict i might first add proper #include as a first step to avoid the recursive flag.
As an example if i want to add a new argument to both:
int set_page_dirty(struct page *page);
int set_page_dirty_lock(struct page *page);
then because inside include/linux/mm.h they are declare one after the other --in-place will fails. This case is not the worse, i can split this in 2 spatches and it would work. I can't do that when updating callback which have same pattern ie 2 callback prototype declare one after the other.
Cheers,
J?r?me
[Cocci] exists do not work if file group is too big
Quoted 14 lines · Jerome Glisse
Quoted 6 lines · Julia Lawall
In terms of the running time, I get a running time of 11 seconds with the command line of 48 files and I get a running time of 22 seconds if I just run spatch on the entire kernel. That seems slower, but if I use the command line of 48 files, I get changes in 27 files, while if I run it on the entire kernel, I get changes in 75 files. If I run it on the entire kernel using 40 cores (-j 40), I get changes in 75 files in 1.7 seconds.
In this simple test case this would work but in my real spatch i have to change one function at a time as otherwise there is conflict on header update. Moreover i also have to use --recursive-headers if i don't do the git grep to provide file list which is also one of the reasons why it was much slower.
Quoted 3 lines
Ideally if header files could be updated --in-place even when there is conflict i might first add proper #include as a first step to avoid the recursive flag.
I'm not sure to understand any of the above. At the moment you have --no-includes, so no headers are consulted. What do you need from header files? Do you need type information? Do you need to change the files? Do the changes in the header files interact with the changes in the .c file?
If it just happens that there are also calls to set_page_dirty in the header files that you want to update, then you can give the options --no-includes --include-headers.
If the only thing you want from the header files is type information, then you can say eg --recursive-includes --include-headers-for-types. That would be much faster, because the transformation rules wouldn't be applied over and over to the header files.
If you want to do both, the above options can be combined.
Quoted 4 lines
As an example if i want to add a new argument to both:
Quoted 8 lines
int
set_page_dirty(struct page *page);
intset_page_dirty_lock(struct page *page);then because inside
include/linux/mm.hthey are declare one after the other --in-place will fails. This case is not the worse, i can split this in 2 spatches and it would work. I can't do that when updating callback which have same pattern ie 2 callback prototype declare one after the other.
Do you actually need to automate this part?
I think I could be more helpful if I had an example that more competely shows what you want to do.
julia
[Cocci] exists do not work if file group is too big
Quoted 32 lines · Julia Lawall
Quoted 14 lines · Jerome Glisse
Quoted 6 lines · Julia Lawall
In terms of the running time, I get a running time of 11 seconds with the command line of 48 files and I get a running time of 22 seconds if I just run spatch on the entire kernel. That seems slower, but if I use the command line of 48 files, I get changes in 27 files, while if I run it on the entire kernel, I get changes in 75 files. If I run it on the entire kernel using 40 cores (-j 40), I get changes in 75 files in 1.7 seconds.
In this simple test case this would work but in my real spatch i have to change one function at a time as otherwise there is conflict on header update. Moreover i also have to use --recursive-headers if i don't do the git grep to provide file list which is also one of the reasons why it was much slower.
Quoted 3 lines
Ideally if header files could be updated --in-place even when there is conflict i might first add proper #include as a first step to avoid the recursive flag.
I'm not sure to understand any of the above. At the moment you have --no-includes, so no headers are consulted. What do you need from header files? Do you need type information? Do you need to change the files? Do the changes in the header files interact with the changes in the .c file?
If it just happens that there are also calls to
set_page_dirtyin the header files that you want to update, then you can give the options --no-includes --include-headers.
Quoted 6 lines
If the only thing you want from the header files is type information, then you can say eg --recursive-includes --include-headers-for-types. That would be much faster, because the transformation rules wouldn't be applied over and over to the header files.
If you want to do both, the above options can be combined.
I need to update function prototype and one semantic patch might update multiples function and those function might have function prototype in same header files just couple lines of each others in which case this does not work. This is an issue mostly when updating callback for which i do not know the function name at time of writing the semantic patch.
The recursive is needed because in many intance the function prototype is define in a header file that is not directly included by the c file but is included through another include or a complex include chains.
Quoted 13 lines
Quoted 4 lines
As an example if i want to add a new argument to both:
Quoted 8 lines
int
set_page_dirty(struct page *page);
intset_page_dirty_lock(struct page *page);then because inside
include/linux/mm.hthey are declare one after the other --in-place will fails. This case is not the worse, i can split this in 2 spatches and it would work. I can't do that when updating callback which have same pattern ie 2 callback prototype declare one after the other.
Quoted 4 lines
Do you actually need to automate this part?
I think I could be more helpful if I had an example that more competely shows what you want to do.
Below is an actual example, i run it with:
spatch --dir . --include-headers -D grep --sp-file spatch
cat /tmp/unicorn-functions | while read line ; do git grep -l \
$line -- '*.[ch]' | sed -e "s/\(.\+\)\.h/--include \1.h/g" \
Quoted 3 lines
/tmp/unicorn-group; echo "--includeinclude/linux/fs.h" \Quoted 2 lines
/tmp/unicorn-group; echo "--includeinclude/linux/mm.h" \/tmp/unicorn-group; cat/tmp/unicorn-group| tr '\n' ' ' ; \
echo ; done > /tmp/unicorn-groups
cat /tmp/unicorn-groups | while read line ; do spatch --in-place \
--no-includes --sp-file spatch $line ; done
spatch:
// First part of this file is only executed when -D grep argument is // provided to spatch. It use to grep all function name that needs to // be modified. Collected function name are written into a temporary // file (/tmp/unicorn-functions) and git grep is use to find all the // files that will be modified using second part of this semantic // patch
virtual grep
// initialize file where we collect all function name (erase it) @initialize:python depends on grep@
@@
file=open('/tmp/unicorn-functions', 'w')file.close()
// match function name use as a callback
@G1 depends on grep@
identifier I1, FN;
@@
struct address_space_operations I1 = {..., .set_page_dirty = FN, ...};
@script:python depends on G1@
funcname << G1.FN;
@@
if funcname != "NULL":
file=open('/tmp/unicorn-functions', 'a')
file.write(funcname + '\n')
file.close()
// Also collect all function name that use a_ops->set_page_dirty() @G2 depends on grep exists@
expression E1, E2;
identifier FN;
type T1;
@@
T1 FN(...) { ...
(E1.a_ops->set_page_dirty(E2)
|E1->a_ops->set_page_dirty(E2)
)
... }
@script:python depends on G2@
funcname << G2.FN;
@@
file=open('/tmp/unicorn-functions', 'a')file.write(funcname + '\n')
print(funcname)file.close()
// -------------------------------------------------------------------
// Below is the actual modification. We use the fn argument (provided // by passing -D fn=value to spatch). To identify which function is // being modified in this run. Semantic patch is run once per function // because of conflict when updating multiple functions in one run.
// Update the address_space_operations structure (should happens only // once with the first group of files)
@depends on !grep@
@@
struct address_space_operations { ... int (*set_page_dirty)( +struct address_space *,
struct page *page); ... };
// Update caller of address_space_operation.set_page_dirty (pass NULL) @depends on !grep@
expression E1, E2;
@@
E1.a_ops->set_page_dirty(
+MAPPING_NULL,
E2)
@depends on !grep@
expression E1, E2;
@@
E1->a_ops->set_page_dirty(
+MAPPING_NULL,
E2)
// Find function name use for the set_page_dirty callback (does not change // anything, this just find the name)
@R1 depends on !grep exists@
identifier I1, virtual.fn;
@@
struct address_space_operations I1 = {..., .set_page_dirty = fn ,...};
// Add address_space argument to the function (set_page_dirty callback one) @depends on R1@
type T1;
identifier I1;
identifier virtual.fn;
@@
int fn(
+struct address_space *__mapping,
T1 I1) { ... }
// Modify every place the function (use for set_page_dirty callback) is call // to pass NULL as for the new address_space argument.
@depends on R1@
identifier virtual.fn;
expression E1;
@@
fn(
+MAPPING_NULL,
E1)
// Modify local variable use to store pointer to callback @R2 depends on !grep exists@
expression E1;
identifier I1;
@@
int (*I1)(
+struct address_space *,
struct page *) = E1->a_ops->set_page_dirty;
@depends on R2 exists@
expression E1;
identifier R2.I1;
@@
-(*I1)(E1)
+I1(MAPPING_NULL, E1)
A message that is not in the archive (sent off-list or not yet received): [email protected]
[Cocci] Checking run times for transformation of Linux source code with SmPL
Quoted 5 lines · SF Markus Elfring
Quoted 3 lines
In terms of the running time, I get a running time of 11 seconds with the command line of 48 files and I get a running time of 22 seconds if I just run spatch on the entire kernel.
* Can such a comparison result be amazing?
No, I don't see anything suprising about it. More work takes more time.
Quoted 9 lines
* How do you think about share any more information about the concrete test environment?
Quoted 2 lines
That seems slower, but if I use the command line of 48 files, I get changes in 27 files, while if I run it on the entire kernel, I get changes in 75 files.
Did this try work without the command parameter ?--file-groups??
Yes. --file-groups is not needed here.
Quoted 6 lines
Quoted 4 lines
If I run it on the entire kernel using 40 cores (-j 40), I get changes in 75 files in 1.7 seconds.
Have you got access to computation resources which are so powerful?
Yes.
julia
A message that is not in the archive (sent off-list or not yet received): [email protected]
[Cocci] Checking run times for transformation of Linux source code with SmPL
Quoted 6 lines · SF Markus Elfring
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
In terms of the running time, I get a running time of 11 seconds with the command line of 48 files and I get a running time of 22 seconds if I just run spatch on the entire kernel.
* Can such a comparison result be amazing?
Quoted 7 lines
Quoted 1 line
No, I don't see anything suprising about it.
I suggest to take another look at these performance numbers from your powerful test environment. All Linux source files were checked to some degree according to another specific search pattern.
The half of this execution time was taken by only a tiny fraction of update candidates.
The 48 files are all parsed. Coccinelle doesn't parse every file in the kernel. It only parses the files that contain the name of the function being modified.
julia
Quoted 12 lines
Quoted 3 lines
More work takes more time.
This usual in principle. It seems that I got a bit of understanding difficulties for this simple test report.
How reasonable is it that a lot of more work could be performed somehow in the double time range?Regards,
Markus
A message that is not in the archive (sent off-list or not yet received): [email protected]
[Cocci] Checking run times for transformation of Linux source code with SmPL
Quoted 8 lines · SF Markus Elfring
Quoted 1 line
Coccinelle doesn't parse every file in the kernel.
I find this information strange.
Quoted 1 line
It only parses the files that contain the name of the function being modified.
Do you interpret the required source code parsing efforts in a special way?
Coccinelle searches the semantic patch for the important words. Then it implements something like grep to find files that contain those imprtant words. Or maybe it just uses git grep by default.
julia
Quoted 2 lines · Jerome Glisse
When operating on too many files the exists keyword no longer work. For instance:
It has nothing to do with the number of files. The problem is that some property of one file causes a parse error in another file, due to the various heuristics. You can see this with the --verbose-parsing option. I'm not yet sure where the bad interaction comes from, but I get the same bad behavior with the command line:
spatch --verbose-parsing --no-includes --sp-file test.cocci fs/ceph/addr.c fs/cifs/cifssmb.c fs/f2fs/gc.c
julia
Quoted 8 lines
@exists@
expression E1;
@@
set_page_dirty(
+NULL,
E1)
Quoted 6 lines
Using linux kernel as playground and looking at at
fs/f2fs/gc.cwhere there isset_page_dirty()calls in nested blocks.Works:
spatch --no-includes --sp-file test.spatch
arch/powerpc/kvm/book3s_64_mmu_radix.carch/powerpc/kvm/e500_mmu.carch/s390/kvm/interrupt.carch/x86/kvm/svm.cblock/bio.cdrivers/block/zram/zram_drv.cdrivers/gpu/drm/amd/amdgpu/amdgpu_ttm.cdrivers/gpu/drm/drm_gem.cdrivers/gpu/drm/exynos/exynos_drm_g2d.cdrivers/gpu/drm/i915/i915_gem.cdrivers/gpu/drm/i915/i915_gem_fence_reg.cdrivers/gpu/drm/i915/i915_gem_userptr.cdrivers/gpu/drm/radeon/radeon_ttm.cdrivers/gpu/drm/ttm/ttm_tt.cdrivers/infiniband/core/umem.cdrivers/infiniband/core/umem_odp.cdrivers/infiniband/hw/hfi1/user_pages.cdrivers/infiniband/hw/qib/qib_user_pages.cdrivers/infiniband/hw/usnic/usnic_uiom.cdrivers/media/common/videobuf2/videobuf2-dma-contig.cdrivers/media/common/videobuf2/videobuf2-dma-sg.cdrivers/media/common/videobuf2/videobuf2-vmalloc.cdrivers/misc/genwqe/card_utils.cdrivers/misc/vmw_vmci/vmci_queue_pair.cdrivers/mtd/devices/block2mtd.cdrivers/platform/goldfish/goldfish_pipe.cdrivers/sbus/!
char/oradax.c drivers/staging/lustre/lustre/llite/rw26.c drivers/staging/lustre/lustre/llite/vvp_io.c drivers/staging/vc04_services/interface/vchiq_arm/vchiq_2835_arm.c drivers/vhost/vhost.c drivers/video/fbdev/core/fb_defio.c fs/9p/vfs_addr.c fs/afs/dir.c fs/afs/file.c --include fs/afs/internal.h fs/afs/write.c fs/aio.c fs/block_dev.c fs/btrfs/disk-io.c fs/btrfs/extent_io.c fs/btrfs/file.c fs/btrfs/inode.c fs/btrfs/ioctl.c fs/btrfs/relocation.c fs/buffer.c fs/ceph/addr.c fs/f2fs/gc.c > toto
Quoted 5 lines
Don't works:
spatch --no-includes --sp-file test.spatch
arch/powerpc/kvm/book3s_64_mmu_radix.carch/powerpc/kvm/e500_mmu.carch/s390/kvm/interrupt.carch/x86/kvm/svm.cblock/bio.cdrivers/block/zram/zram_drv.cdrivers/gpu/drm/amd/amdgpu/amdgpu_ttm.cdrivers/gpu/drm/drm_gem.cdrivers/gpu/drm/exynos/exynos_drm_g2d.cdrivers/gpu/drm/i915/i915_gem.cdrivers/gpu/drm/i915/i915_gem_fence_reg.cdrivers/gpu/drm/i915/i915_gem_userptr.cdrivers/gpu/drm/radeon/radeon_ttm.cdrivers/gpu/drm/ttm/ttm_tt.cdrivers/infiniband/core/umem.cdrivers/infiniband/core/umem_odp.cdrivers/infiniband/hw/hfi1/user_pages.cdrivers/infiniband/hw/qib/qib_user_pages.cdrivers/infiniband/hw/usnic/usnic_uiom.cdrivers/media/common/videobuf2/videobuf2-dma-contig.cdrivers/media/common/videobuf2/videobuf2-dma-sg.cdrivers/media/common/videobuf2/videobuf2-vmalloc.cdrivers/misc/genwqe/card_utils.cdrivers/misc/vmw_vmci/vmci_queue_pair.cdrivers/mtd/devices/block2mtd.cdrivers/platform/goldfish/goldfish_pipe.cdrivers/sbus/!
char/oradax.c drivers/staging/lustre/lustre/llite/rw26.c drivers/staging/lustre/lustre/llite/vvp_io.c drivers/staging/vc04_services/interface/vchiq_arm/vchiq_2835_arm.c drivers/vhost/vhost.c drivers/video/fbdev/core/fb_defio.c fs/9p/vfs_addr.c fs/afs/dir.c fs/afs/file.c --include fs/afs/internal.h fs/afs/write.c fs/aio.c fs/block_dev.c fs/btrfs/disk-io.c fs/btrfs/extent_io.c fs/btrfs/file.c fs/btrfs/inode.c fs/btrfs/ioctl.c fs/btrfs/relocation.c fs/buffer.c fs/ceph/addr.c fs/cifs/cifssmb.c fs/f2fs/gc.c > toto
Quoted 14 lines
Exact same command line except the latter have 49 files before
fs/f2fs/gc.cwhile the former has 48 (and it works then). Note that it seems that running the script multiple times even when they are too many files will eventualy work.Cheers,
J?r?me
_______________________________________________
Cocci mailing list
Cocci at systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci
Concretely, the problem is with LOCK_REQ, which is used as a typedef name in cifs and as a enum constant in f2fs.
Would there be any chance of getting rid of the LOCK_REQ typedef? Typedefs for structure types are discouraged by the Linux kernel coding style.
julia
Quoted 8 lines · Julia Lawall
Concretely, the problem is with
LOCK_REQ, which is used as a typedef name in cifs and as a enum constant in f2fs.Would there be any chance of getting rid of the
LOCK_REQtypedef? Typedefs for structure types are discouraged by the Linux kernel coding style.julia
I will try to make patches for that, thought not anytime soon i am going on PTO so not before i get back.
Thank you for debugging this further.
Cheers,
J?r?me