Skip to contentSkip to conversation

xfstests results over NFS

DISCUSSION Jeff Layton 4 participants 16 messages Aug 17, 2023 – Aug 22, 2023

Subsystems (2)
lore
Share & tools
lore

Jeff Layton

author maintainer
#0 lore raw

I finally got my kdevops (https://github.com/linux-kdevops/kdevops) test rig working well enough to get some publishable results. To run fstests, kdevops will spin up a server and (in this case) 2 clients to run xfstests' auto group. One client mounts with default options, and the other uses NFSv3.

I tested 3 kernels:

v6.4.0 (stock release)
6.5.0-rc6-g4853c74bd7ab (Linus' tree as of a couple of days ago) 6.5.0-rc6-next-20230816-gef66bf8aeb91 (linux-next as of yesterday morning)

Here are the results summary of all 3:

KERNEL:    6.4.0
CPUS:      8

nfs_v3: 727 tests, 12 failures, 569 skipped, 14863 seconds
Failures: generic/053 generic/099 generic/105 generic/124

    generic/193 generic/258 generic/294 generic/318 generic/319 
    generic/444 generic/528 generic/529 
nfs_default: 727 tests, 18 failures, 452 skipped, 21899 seconds
  Failures: generic/053 generic/099 generic/105 generic/186 
    generic/187 generic/193 generic/294 generic/318 generic/319 
    generic/357 generic/444 generic/486 generic/513 generic/528 
    generic/529 generic/578 generic/675 generic/688 

Totals: 1454 tests, 1021 skipped, 30 failures, 0 errors, 35096s

KERNEL:    6.5.0-rc6-g4853c74bd7ab
CPUS:      8

nfs_v3: 727 tests, 9 failures, 570 skipped, 14775 seconds
Failures: generic/053 generic/099 generic/105 generic/258

    generic/294 generic/318 generic/319 generic/444 generic/529 
nfs_default: 727 tests, 16 failures, 453 skipped, 22326 seconds
  Failures: generic/053 generic/099 generic/105 generic/186 
    generic/187 generic/294 generic/318 generic/319 generic/357 
    generic/444 generic/486 generic/513 generic/529 generic/578 
    generic/675 generic/688 

Totals: 1454 tests, 1023 skipped, 25 failures, 0 errors, 35396s

KERNEL:    6.5.0-rc6-next-20230816-gef66bf8aeb91
CPUS:      8

nfs_v3: 727 tests, 9 failures, 570 skipped, 14657 seconds
Failures: generic/053 generic/099 generic/105 generic/258

    generic/294 generic/318 generic/319 generic/444 generic/529 
nfs_default: 727 tests, 18 failures, 453 skipped, 21757 seconds
  Failures: generic/053 generic/099 generic/105 generic/186 
    generic/187 generic/294 generic/318 generic/319 generic/357 
    generic/444 generic/486 generic/513 generic/529 generic/578 
    generic/675 generic/683 generic/684 generic/688 

Totals: 1454 tests, 1023 skipped, 27 failures, 0 errors, 34870s

With NFSv4.2, v6.4.0 has 2 extra failures that the current mainline kernel doesn't:

generic/193 (some sort of setattr problem)
generic/528 (known problem with btime handling in client that has been fixed)

While I haven't investigated, I'm assuming the 193 bug is also something that has been fixed in recent kernels. There are also 3 other NFSv3 tests that started passing since v6.4.0. I haven't looked into those.

With the linux-next kernel there are 2 new regressions:

    generic/683
    generic/684

Both of these look like problems with setuid/setgid stripping, and still need to be investigated. I have more verbose result info on the test failures if anyone is interested.

Cheers,

Signature · 1 line
Jeff Layton <[email protected]>
#1 lore raw
Quoted 68 lines · Jeff Layton

I finally got my kdevops (https://github.com/linux-kdevops/kdevops) test rig working well enough to get some publishable results. To run fstests, kdevops will spin up a server and (in this case) 2 clients to run xfstests' auto group. One client mounts with default options, and the other uses NFSv3.

I tested 3 kernels:

v6.4.0 (stock release)
6.5.0-rc6-g4853c74bd7ab (Linus' tree as of a couple of days ago) 6.5.0-rc6-next-20230816-gef66bf8aeb91 (linux-next as of yesterday morning)

Here are the results summary of all 3:

KERNEL:    6.4.0
CPUS:      8

nfs_v3: 727 tests, 12 failures, 569 skipped, 14863 seconds
Failures: generic/053 generic/099 generic/105 generic/124
generic/193 generic/258 generic/294 generic/318 generic/319 generic/444 generic/528 generic/529
nfs_default: 727 tests, 18 failures, 452 skipped, 21899 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/193 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/528 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1021 skipped, 30 failures, 0 errors, 35096s

KERNEL:    6.5.0-rc6-g4853c74bd7ab
CPUS:      8

nfs_v3: 727 tests, 9 failures, 570 skipped, 14775 seconds
Failures: generic/053 generic/099 generic/105 generic/258
generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 16 failures, 453 skipped, 22326 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1023 skipped, 25 failures, 0 errors, 35396s

KERNEL:    6.5.0-rc6-next-20230816-gef66bf8aeb91
CPUS:      8

nfs_v3: 727 tests, 9 failures, 570 skipped, 14657 seconds
Failures: generic/053 generic/099 generic/105 generic/258
generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 18 failures, 453 skipped, 21757 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/683 generic/684 generic/688
Totals: 1454 tests, 1023 skipped, 27 failures, 0 errors, 34870s

With NFSv4.2, v6.4.0 has 2 extra failures that the current mainline kernel doesn't:

generic/193 (some sort of setattr problem)
generic/528 (known problem with btime handling in client that has been fixed)

While I haven't investigated, I'm assuming the 193 bug is also something that has been fixed in recent kernels. There are also 3 other NFSv3 tests that started passing since v6.4.0. I haven't looked into those.

Quoted 8 lines

With the linux-next kernel there are 2 new regressions:

   generic/683
   generic/684

Both of these look like problems with setuid/setgid stripping, and still need to be investigated. I have more verbose result info on the test failures if anyone is interested.

100% awesome sauce. Out of curiosity:

Does kdevops have a way of publishing (via an autonomous web site) and archiving these results?

Does the "auto" group include tests that require a SCRATCH_DEV?

--
Chuck Lever

author maintainer
#2 lore raw
Quoted 72 lines · Chuck Lever III
Quoted 72 lines · Jeff Layton

I finally got my kdevops (https://github.com/linux-kdevops/kdevops) test rig working well enough to get some publishable results. To run fstests, kdevops will spin up a server and (in this case) 2 clients to run xfstests' auto group. One client mounts with default options, and the other uses NFSv3.

I tested 3 kernels:

v6.4.0 (stock release)
6.5.0-rc6-g4853c74bd7ab (Linus' tree as of a couple of days ago) 6.5.0-rc6-next-20230816-gef66bf8aeb91 (linux-next as of yesterday morning)

Here are the results summary of all 3:

KERNEL:    6.4.0
CPUS:      8

nfs_v3: 727 tests, 12 failures, 569 skipped, 14863 seconds
Failures: generic/053 generic/099 generic/105 generic/124
generic/193 generic/258 generic/294 generic/318 generic/319 generic/444 generic/528 generic/529
nfs_default: 727 tests, 18 failures, 452 skipped, 21899 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/193 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/528 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1021 skipped, 30 failures, 0 errors, 35096s

KERNEL:    6.5.0-rc6-g4853c74bd7ab
CPUS:      8

nfs_v3: 727 tests, 9 failures, 570 skipped, 14775 seconds
Failures: generic/053 generic/099 generic/105 generic/258
generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 16 failures, 453 skipped, 22326 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1023 skipped, 25 failures, 0 errors, 35396s

KERNEL:    6.5.0-rc6-next-20230816-gef66bf8aeb91
CPUS:      8

nfs_v3: 727 tests, 9 failures, 570 skipped, 14657 seconds
Failures: generic/053 generic/099 generic/105 generic/258
generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 18 failures, 453 skipped, 21757 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/683 generic/684 generic/688
Totals: 1454 tests, 1023 skipped, 27 failures, 0 errors, 34870s

With NFSv4.2, v6.4.0 has 2 extra failures that the current mainline kernel doesn't:

generic/193 (some sort of setattr problem)
generic/528 (known problem with btime handling in client that has been fixed)

While I haven't investigated, I'm assuming the 193 bug is also something that has been fixed in recent kernels. There are also 3 other NFSv3 tests that started passing since v6.4.0. I haven't looked into those.

With the linux-next kernel there are 2 new regressions:

   generic/683
   generic/684
Quoted 9 lines
Quoted 3 lines

Both of these look like problems with setuid/setgid stripping, and still need to be investigated. I have more verbose result info on the test failures if anyone is interested.

100% awesome sauce. Out of curiosity:

Does kdevops have a way of publishing (via an autonomous web site) and archiving these results?

There's nothing much prewritten for this. There is some support for sending emails when you run a "ci" loop. I need to do more investigation here.

Note that there has been some parallel effort toward CI in the SMB space using buildbot. It may worthwhile to consider combining efforts somehow.

Quoted 2 lines

Does the "auto" group include tests that require a SCRATCH_DEV?

Yes. The nfs server is configured with 2 exported fs', so I have it mounting a directory under one as "test" and the other as "scratch".

Signature · 1 line
Jeff Layton <[email protected]>
maintainer
#3 lore raw
Quoted 49 lines · Jeff Layton
Quoted 47 lines · Chuck Lever III
Quoted 47 lines · Jeff Layton

I finally got my kdevops (https://github.com/linux-kdevops/kdevops) test rig working well enough to get some publishable results. To run fstests, kdevops will spin up a server and (in this case) 2 clients to run xfstests' auto group. One client mounts with default options, and the other uses NFSv3.

I tested 3 kernels:

v6.4.0 (stock release)
6.5.0-rc6-g4853c74bd7ab (Linus' tree as of a couple of days ago) 6.5.0-rc6-next-20230816-gef66bf8aeb91 (linux-next as of yesterday morning)

Here are the results summary of all 3:

KERNEL:    6.4.0
CPUS:      8

nfs_v3: 727 tests, 12 failures, 569 skipped, 14863 seconds
Failures: generic/053 generic/099 generic/105 generic/124
generic/193 generic/258 generic/294 generic/318 generic/319 generic/444 generic/528 generic/529
nfs_default: 727 tests, 18 failures, 452 skipped, 21899 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/193 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/528 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1021 skipped, 30 failures, 0 errors, 35096s

KERNEL:    6.5.0-rc6-g4853c74bd7ab
CPUS:      8

nfs_v3: 727 tests, 9 failures, 570 skipped, 14775 seconds
Failures: generic/053 generic/099 generic/105 generic/258
generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 16 failures, 453 skipped, 22326 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1023 skipped, 25 failures, 0 errors, 35396s

KERNEL:    6.5.0-rc6-next-20230816-gef66bf8aeb91
CPUS:      8
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

nfs_v3: 727 tests, 9 failures, 570 skipped, 14657 seconds
Failures: generic/053 generic/099 generic/105 generic/258
generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 18 failures, 453 skipped, 21757 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/683 generic/684 generic/688
Totals: 1454 tests, 1023 skipped, 27 failures, 0 errors, 34870s

As long as we're sharing results ... here is what I'm seeing with a 6.5-rc6 client & server:

anna@gouda ~ % xfstestsdb xunit list --results --runid 1741 --color=none

+------+----------------------+---------+----------+------+------+------+-------+
|  run | device               | xunit   | hostname | pass | fail |
skip |  time |
+------+----------------------+---------+----------+------+------+------+-------+
| 1741 | server:/srv/xfs/test | tcp-3   | client   |  125 |    4 |
464 | 447 s |
| 1741 | server:/srv/xfs/test | tcp-4.0 | client   |  117 |   11 |
465 | 478 s |
| 1741 | server:/srv/xfs/test | tcp-4.1 | client   |  119 |   12 |
462 | 404 s |
| 1741 | server:/srv/xfs/test | tcp-4.2 | client   |  212 |   18 |
363 | 564 s |
+------+----------------------+---------+----------+------+------+------+-------+

anna@gouda ~ % xfstestsdb show --failure 1741 --color=none

+-------------+---------+---------+---------+---------+
|    testcase | tcp-3   | tcp-4.0 | tcp-4.1 | tcp-4.2 |
+-------------+---------+---------+---------+---------+
| generic/053 | passed  | failure | failure | failure |
| generic/099 | passed  | failure | failure | failure |
| generic/105 | passed  | failure | failure | failure |
| generic/140 | skipped | skipped | skipped | failure |
| generic/188 | skipped | skipped | skipped | failure |
| generic/258 | failure | passed  | passed  | failure |
| generic/294 | failure | failure | failure | failure |
| generic/318 | passed  | failure | failure | failure |
| generic/319 | passed  | failure | failure | failure |
| generic/357 | skipped | skipped | skipped | failure |
| generic/444 | failure | failure | failure | failure |
| generic/465 | passed  | failure | failure | failure |
| generic/513 | skipped | skipped | skipped | failure |
| generic/529 | passed  | failure | failure | failure |
| generic/604 | passed  | passed  | failure | passed  |
| generic/675 | skipped | skipped | skipped | failure |
| generic/688 | skipped | skipped | skipped | failure |
| generic/697 | passed  | failure | failure | failure |
|     nfs/002 | failure | failure | failure | failure |
+-------------+---------+---------+---------+---------+
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

With NFSv4.2, v6.4.0 has 2 extra failures that the current mainline kernel doesn't:

generic/193 (some sort of setattr problem)
generic/528 (known problem with btime handling in client that has been fixed)

While I haven't investigated, I'm assuming the 193 bug is also something that has been fixed in recent kernels. There are also 3 other NFSv3 tests that started passing since v6.4.0. I haven't looked into those.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

With the linux-next kernel there are 2 new regressions:

   generic/683
   generic/684

Both of these look like problems with setuid/setgid stripping, and still need to be investigated. I have more verbose result info on the test failures if anyone is interested.

Interesting that I'm not seeing the 683 & 684 failures. What type of filesystem is your server exporting?

Quoted 10 lines
Quoted 6 lines

100% awesome sauce. Out of curiosity:

Does kdevops have a way of publishing (via an autonomous web site) and archiving these results?

There's nothing much prewritten for this. There is some support for sending emails when you run a "ci" loop. I need to do more investigation here.

xfstests has an option to generate an xunit file, which can help here. I use with my own archiving tool to stick everything into a sqlite database (https://git.nowheycreamery.com/anna/xfstestsdb).

Quoted 3 lines

Note that there has been some parallel effort toward CI in the SMB space using buildbot. It may worthwhile to consider combining efforts somehow.

It might be nice to at least see what they're doing. If they have something that works well, then setting up something similar might be a good idea.

Anna

Quoted 8 lines
Quoted 3 lines

Does the "auto" group include tests that require a SCRATCH_DEV?

Yes. The nfs server is configured with 2 exported fs', so I have it mounting a directory under one as "test" and the other as "scratch". --
Jeff Layton <[email protected]>

author maintainer
#4 lore raw
Quoted 120 lines · Anna Schumaker
Quoted 49 lines · Jeff Layton
Quoted 47 lines · Chuck Lever III
Quoted 47 lines · Jeff Layton

I finally got my kdevops (https://github.com/linux-kdevops/kdevops) test rig working well enough to get some publishable results. To run fstests, kdevops will spin up a server and (in this case) 2 clients to run xfstests' auto group. One client mounts with default options, and the other uses NFSv3.

I tested 3 kernels:

v6.4.0 (stock release)
6.5.0-rc6-g4853c74bd7ab (Linus' tree as of a couple of days ago) 6.5.0-rc6-next-20230816-gef66bf8aeb91 (linux-next as of yesterday morning)

Here are the results summary of all 3:

KERNEL:    6.4.0
CPUS:      8

nfs_v3: 727 tests, 12 failures, 569 skipped, 14863 seconds
Failures: generic/053 generic/099 generic/105 generic/124
generic/193 generic/258 generic/294 generic/318 generic/319 generic/444 generic/528 generic/529
nfs_default: 727 tests, 18 failures, 452 skipped, 21899 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/193 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/528 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1021 skipped, 30 failures, 0 errors, 35096s

KERNEL:    6.5.0-rc6-g4853c74bd7ab
CPUS:      8

nfs_v3: 727 tests, 9 failures, 570 skipped, 14775 seconds
Failures: generic/053 generic/099 generic/105 generic/258
generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 16 failures, 453 skipped, 22326 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1023 skipped, 25 failures, 0 errors, 35396s

KERNEL:    6.5.0-rc6-next-20230816-gef66bf8aeb91
CPUS:      8
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

nfs_v3: 727 tests, 9 failures, 570 skipped, 14657 seconds
Failures: generic/053 generic/099 generic/105 generic/258
generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 18 failures, 453 skipped, 21757 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/683 generic/684 generic/688
Totals: 1454 tests, 1023 skipped, 27 failures, 0 errors, 34870s

As long as we're sharing results ... here is what I'm seeing with a 6.5-rc6 client & server:

anna@gouda ~ % xfstestsdb xunit list --results --runid 1741 --color=none

+------+----------------------+---------+----------+------+------+------+-------+
>  run | device               | xunit   | hostname | pass | fail |
skip |  time |
+------+----------------------+---------+----------+------+------+------+-------+
> 1741 | server:/srv/xfs/test | tcp-3   | client   |  125 |    4 |
464 | 447 s |
> 1741 | server:/srv/xfs/test | tcp-4.0 | client   |  117 |   11 |
465 | 478 s |
> 1741 | server:/srv/xfs/test | tcp-4.1 | client   |  119 |   12 |
462 | 404 s |
> 1741 | server:/srv/xfs/test | tcp-4.2 | client   |  212 |   18 |
363 | 564 s |
+------+----------------------+---------+----------+------+------+------+-------+

anna@gouda ~ % xfstestsdb show --failure 1741 --color=none

+-------------+---------+---------+---------+---------+
>    testcase | tcp-3   | tcp-4.0 | tcp-4.1 | tcp-4.2 |
+-------------+---------+---------+---------+---------+
> generic/053 | passed  | failure | failure | failure |
> generic/099 | passed  | failure | failure | failure |
> generic/105 | passed  | failure | failure | failure |
> generic/140 | skipped | skipped | skipped | failure |
> generic/188 | skipped | skipped | skipped | failure |
> generic/258 | failure | passed  | passed  | failure |
> generic/294 | failure | failure | failure | failure |
> generic/318 | passed  | failure | failure | failure |
> generic/319 | passed  | failure | failure | failure |
> generic/357 | skipped | skipped | skipped | failure |
> generic/444 | failure | failure | failure | failure |
> generic/465 | passed  | failure | failure | failure |
> generic/513 | skipped | skipped | skipped | failure |
> generic/529 | passed  | failure | failure | failure |
> generic/604 | passed  | passed  | failure | passed  |
> generic/675 | skipped | skipped | skipped | failure |
> generic/688 | skipped | skipped | skipped | failure |
> generic/697 | passed  | failure | failure | failure |
>     nfs/002 | failure | failure | failure | failure |
+-------------+---------+---------+---------+---------+
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines

With NFSv4.2, v6.4.0 has 2 extra failures that the current mainline kernel doesn't:

generic/193 (some sort of setattr problem)
generic/528 (known problem with btime handling in client that has been fixed)

While I haven't investigated, I'm assuming the 193 bug is also something that has been fixed in recent kernels. There are also 3 other NFSv3 tests that started passing since v6.4.0. I haven't looked into those.

With the linux-next kernel there are 2 new regressions:

   generic/683
   generic/684
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Both of these look like problems with setuid/setgid stripping, and still need to be investigated. I have more verbose result info on the test failures if anyone is interested.

Interesting that I'm not seeing the 683 & 684 failures. What type of filesystem is your server exporting?

btrfs

You are testing linux-next? I need to go back and confirm these results too.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

100% awesome sauce. Out of curiosity:

Does kdevops have a way of publishing (via an autonomous web site) and archiving these results?

Quoted 8 lines
Quoted 3 lines

There's nothing much prewritten for this. There is some support for sending emails when you run a "ci" loop. I need to do more investigation here.

xfstests has an option to generate an xunit file, which can help here. I use with my own archiving tool to stick everything into a sqlite database (https://git.nowheycreamery.com/anna/xfstestsdb).

Yeah, kdevops uses the xunit file to generate its results, AFAIU. TBH, a lot of the automation surrounding how to collate and evaluate test results is still something I need to look at more closely. It's not well documented and is still under pretty heavy development.

Quoted 8 lines
Quoted 3 lines

Note that there has been some parallel effort toward CI in the SMB space using buildbot. It may worthwhile to consider combining efforts somehow.

It might be nice to at least see what they're doing. If they have something that works well, then setting up something similar might be a good idea.

Just my gut feel is that kdevops seems to be more geared toward "maintainer wants to see a set of results vs. particular kernels", whereas buildbot seems to be more geared toward automation, and CI type workloads. There are some CI-ish automation bits in kdevops, but doesn't seem to be as straightforward as what buildbot has.

Signature · 1 line
Jeff Layton <[email protected]>
#5 lore raw
Quoted 129 lines · Jeff Layton
Quoted 124 lines · Anna Schumaker
Quoted 49 lines · Jeff Layton
Quoted 47 lines · Chuck Lever III
Quoted 47 lines · Jeff Layton

I finally got my kdevops (https://github.com/linux-kdevops/kdevops) test rig working well enough to get some publishable results. To run fstests, kdevops will spin up a server and (in this case) 2 clients to run xfstests' auto group. One client mounts with default options, and the other uses NFSv3.

I tested 3 kernels:

v6.4.0 (stock release)
6.5.0-rc6-g4853c74bd7ab (Linus' tree as of a couple of days ago) 6.5.0-rc6-next-20230816-gef66bf8aeb91 (linux-next as of yesterday morning)

Here are the results summary of all 3:

KERNEL:    6.4.0
CPUS:      8

nfs_v3: 727 tests, 12 failures, 569 skipped, 14863 seconds
Failures: generic/053 generic/099 generic/105 generic/124
generic/193 generic/258 generic/294 generic/318 generic/319 generic/444 generic/528 generic/529
nfs_default: 727 tests, 18 failures, 452 skipped, 21899 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/193 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/528 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1021 skipped, 30 failures, 0 errors, 35096s

KERNEL:    6.5.0-rc6-g4853c74bd7ab
CPUS:      8

nfs_v3: 727 tests, 9 failures, 570 skipped, 14775 seconds
Failures: generic/053 generic/099 generic/105 generic/258
generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 16 failures, 453 skipped, 22326 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1023 skipped, 25 failures, 0 errors, 35396s

KERNEL:    6.5.0-rc6-next-20230816-gef66bf8aeb91
CPUS:      8
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

nfs_v3: 727 tests, 9 failures, 570 skipped, 14657 seconds
Failures: generic/053 generic/099 generic/105 generic/258
generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 18 failures, 453 skipped, 21757 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/683 generic/684 generic/688
Totals: 1454 tests, 1023 skipped, 27 failures, 0 errors, 34870s

As long as we're sharing results ... here is what I'm seeing with a 6.5-rc6 client & server:

anna@gouda ~ % xfstestsdb xunit list --results --runid 1741 --color=none

+------+----------------------+---------+----------+------+------+------+-------+
> run | device               | xunit   | hostname | pass | fail |
skip |  time |
+------+----------------------+---------+----------+------+------+------+-------+
> 1741 | server:/srv/xfs/test | tcp-3   | client   |  125 |    4 |
464 | 447 s |
> 1741 | server:/srv/xfs/test | tcp-4.0 | client   |  117 |   11 |
465 | 478 s |
> 1741 | server:/srv/xfs/test | tcp-4.1 | client   |  119 |   12 |
462 | 404 s |
> 1741 | server:/srv/xfs/test | tcp-4.2 | client   |  212 |   18 |
363 | 564 s |
+------+----------------------+---------+----------+------+------+------+-------+

anna@gouda ~ % xfstestsdb show --failure 1741 --color=none

+-------------+---------+---------+---------+---------+
>   testcase | tcp-3   | tcp-4.0 | tcp-4.1 | tcp-4.2 |
+-------------+---------+---------+---------+---------+
> generic/053 | passed  | failure | failure | failure |
> generic/099 | passed  | failure | failure | failure |
> generic/105 | passed  | failure | failure | failure |
> generic/140 | skipped | skipped | skipped | failure |
> generic/188 | skipped | skipped | skipped | failure |
> generic/258 | failure | passed  | passed  | failure |
> generic/294 | failure | failure | failure | failure |
> generic/318 | passed  | failure | failure | failure |
> generic/319 | passed  | failure | failure | failure |
> generic/357 | skipped | skipped | skipped | failure |
> generic/444 | failure | failure | failure | failure |
> generic/465 | passed  | failure | failure | failure |
> generic/513 | skipped | skipped | skipped | failure |
> generic/529 | passed  | failure | failure | failure |
> generic/604 | passed  | passed  | failure | passed  |
> generic/675 | skipped | skipped | skipped | failure |
> generic/688 | skipped | skipped | skipped | failure |
> generic/697 | passed  | failure | failure | failure |
>    nfs/002 | failure | failure | failure | failure |
+-------------+---------+---------+---------+---------+
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines

With NFSv4.2, v6.4.0 has 2 extra failures that the current mainline kernel doesn't:

generic/193 (some sort of setattr problem)
generic/528 (known problem with btime handling in client that has been fixed)

While I haven't investigated, I'm assuming the 193 bug is also something that has been fixed in recent kernels. There are also 3 other NFSv3 tests that started passing since v6.4.0. I haven't looked into those.

With the linux-next kernel there are 2 new regressions:

generic/683
generic/684

Both of these look like problems with setuid/setgid stripping, and still need to be investigated. I have more verbose result info on the test failures if anyone is interested.

Quoted 8 lines
Quoted 3 lines

Interesting that I'm not seeing the 683 & 684 failures. What type of filesystem is your server exporting?

btrfs

You are testing linux-next? I need to go back and confirm these results too.

IMO linux-next is quite important : we keep hitting bugs that appear only after integration -- block and network changes in other trees especially can impact the NFS drivers.

--
Chuck Lever

author maintainer
#6 lore raw
Quoted 132 lines · Chuck Lever III
Quoted 132 lines · Jeff Layton
Quoted 127 lines · Anna Schumaker
Quoted 49 lines · Jeff Layton
Quoted 47 lines · Chuck Lever III
Quoted 47 lines · Jeff Layton

I finally got my kdevops (https://github.com/linux-kdevops/kdevops) test rig working well enough to get some publishable results. To run fstests, kdevops will spin up a server and (in this case) 2 clients to run xfstests' auto group. One client mounts with default options, and the other uses NFSv3.

I tested 3 kernels:

v6.4.0 (stock release)
6.5.0-rc6-g4853c74bd7ab (Linus' tree as of a couple of days ago) 6.5.0-rc6-next-20230816-gef66bf8aeb91 (linux-next as of yesterday morning)

Here are the results summary of all 3:

KERNEL:    6.4.0
CPUS:      8

nfs_v3: 727 tests, 12 failures, 569 skipped, 14863 seconds
Failures: generic/053 generic/099 generic/105 generic/124
generic/193 generic/258 generic/294 generic/318 generic/319 generic/444 generic/528 generic/529
nfs_default: 727 tests, 18 failures, 452 skipped, 21899 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/193 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/528 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1021 skipped, 30 failures, 0 errors, 35096s

KERNEL:    6.5.0-rc6-g4853c74bd7ab
CPUS:      8

nfs_v3: 727 tests, 9 failures, 570 skipped, 14775 seconds
Failures: generic/053 generic/099 generic/105 generic/258
generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 16 failures, 453 skipped, 22326 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1023 skipped, 25 failures, 0 errors, 35396s

KERNEL:    6.5.0-rc6-next-20230816-gef66bf8aeb91
CPUS:      8
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

nfs_v3: 727 tests, 9 failures, 570 skipped, 14657 seconds
Failures: generic/053 generic/099 generic/105 generic/258
generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 18 failures, 453 skipped, 21757 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/683 generic/684 generic/688
Totals: 1454 tests, 1023 skipped, 27 failures, 0 errors, 34870s

As long as we're sharing results ... here is what I'm seeing with a 6.5-rc6 client & server:

anna@gouda ~ % xfstestsdb xunit list --results --runid 1741 --color=none

+------+----------------------+---------+----------+------+------+------+-------+
> run | device               | xunit   | hostname | pass | fail |
skip |  time |
+------+----------------------+---------+----------+------+------+------+-------+
> 1741 | server:/srv/xfs/test | tcp-3   | client   |  125 |    4 |
464 | 447 s |
> 1741 | server:/srv/xfs/test | tcp-4.0 | client   |  117 |   11 |
465 | 478 s |
> 1741 | server:/srv/xfs/test | tcp-4.1 | client   |  119 |   12 |
462 | 404 s |
> 1741 | server:/srv/xfs/test | tcp-4.2 | client   |  212 |   18 |
363 | 564 s |
+------+----------------------+---------+----------+------+------+------+-------+

anna@gouda ~ % xfstestsdb show --failure 1741 --color=none

+-------------+---------+---------+---------+---------+
>   testcase | tcp-3   | tcp-4.0 | tcp-4.1 | tcp-4.2 |
+-------------+---------+---------+---------+---------+
> generic/053 | passed  | failure | failure | failure |
> generic/099 | passed  | failure | failure | failure |
> generic/105 | passed  | failure | failure | failure |
> generic/140 | skipped | skipped | skipped | failure |
> generic/188 | skipped | skipped | skipped | failure |
> generic/258 | failure | passed  | passed  | failure |
> generic/294 | failure | failure | failure | failure |
> generic/318 | passed  | failure | failure | failure |
> generic/319 | passed  | failure | failure | failure |
> generic/357 | skipped | skipped | skipped | failure |
> generic/444 | failure | failure | failure | failure |
> generic/465 | passed  | failure | failure | failure |
> generic/513 | skipped | skipped | skipped | failure |
> generic/529 | passed  | failure | failure | failure |
> generic/604 | passed  | passed  | failure | passed  |
> generic/675 | skipped | skipped | skipped | failure |
> generic/688 | skipped | skipped | skipped | failure |
> generic/697 | passed  | failure | failure | failure |
>    nfs/002 | failure | failure | failure | failure |
+-------------+---------+---------+---------+---------+
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

With NFSv4.2, v6.4.0 has 2 extra failures that the current mainline kernel doesn't:

generic/193 (some sort of setattr problem)
generic/528 (known problem with btime handling in client that has been fixed)

While I haven't investigated, I'm assuming the 193 bug is also something that has been fixed in recent kernels. There are also 3 other NFSv3 tests that started passing since v6.4.0. I haven't looked into those.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

With the linux-next kernel there are 2 new regressions:

generic/683
generic/684

Both of these look like problems with setuid/setgid stripping, and still need to be investigated. I have more verbose result info on the test failures if anyone is interested.

Interesting that I'm not seeing the 683 & 684 failures. What type of filesystem is your server exporting?

Quoted 9 lines
Quoted 4 lines

btrfs

You are testing linux-next? I need to go back and confirm these results too.

IMO linux-next is quite important : we keep hitting bugs that appear only after integration -- block and network changes in other trees especially can impact the NFS drivers.

Indeed, I suspect this is probably something from the vfs tree (though we definitely need to confirm that). Today I'm testing:

    6.5.0-rc6-next-20230817-g47762f086974

[vagrant@kdevops-nfs-default xfstests]$ sudo ./check generic/683 generic/684

SECTION       -- default
FSTYP         -- nfs
PLATFORM      -- Linux/x86_64 kdevops-nfs-default 6.5.0-rc6-next-20230817-g47762f086974 #37 SMP PREEMPT_DYNAMIC Thu Aug 17 10:17:27 EDT 2023
MKFS_OPTIONS  -- kdevops-nfsd:/export/1/fstests/kdevops-nfs-default

MOUNT_OPTIONS -- kdevops-nfsd:/export/1/fstests/kdevops-nfs-default /media/scratch

generic/683 - output mismatch (see /data/fstests-install/xfstests/results/kdevops-nfs-default/6.5.0-rc6-next-20230817-g47762f086974/default/generic/683.out.bad)

    --- tests/generic/683.out   2023-08-17 15:50:52.428385413 +0000
    +++ /data/fstests-install/xfstests/results/kdevops-nfs-default/6.5.0-rc6-next-20230817-g47762f086974/default/generic/683.out.bad    2023-08-17 17:10:05.017250750 +0000
    @@ -1,19 +1,19 @@
     QA output created by 683
     Test 1 - qa_user, non-exec file falloc
     6666 -rwSrwSrw- TEST_DIR/683/a
    -666 -rw-rw-rw- TEST_DIR/683/a
    +6666 -rwSrwSrw- TEST_DIR/683/a
     
     Test 2 - qa_user, group-exec file falloc
    ...
    (Run 'diff -u /data/fstests-install/xfstests/tests/generic/683.out /data/fstests-install/xfstests/results/kdevops-nfs-default/6.5.0-rc6-next-20230817-g47762f086974/default/generic/683.out.bad'  to see the entire diff)
generic/684       - output mismatch (see /data/fstests-install/xfstests/results/kdevops-nfs-default/6.5.0-rc6-next-20230817-g47762f086974/default/generic/684.out.bad)
    --- tests/generic/684.out   2023-08-17 15:50:52.456385413 +0000
    +++ /data/fstests-install/xfstests/results/kdevops-nfs-default/6.5.0-rc6-next-20230817-g47762f086974/default/generic/684.out.bad    2023-08-17 17:10:11.409250750 +0000
    @@ -1,19 +1,19 @@
     QA output created by 684
     Test 1 - qa_user, non-exec file fpunch
     6666 -rwSrwSrw- TEST_DIR/684/a
    -666 -rw-rw-rw- TEST_DIR/684/a
    +6666 -rwSrwSrw- TEST_DIR/684/a
     
     Test 2 - qa_user, group-exec file fpunch
    ...
    (Run 'diff -u /data/fstests-install/xfstests/tests/generic/684.out /data/fstests-install/xfstests/results/kdevops-nfs-default/6.5.0-rc6-next-20230817-g47762f086974/default/generic/684.out.bad'  to see the entire diff)

Ran: generic/683 generic/684
Failures: generic/683 generic/684
Failed 2 of 2 tests

SECTION -- default
=========================
Ran: generic/683 generic/684
Failures: generic/683 generic/684
Failed 2 of 2 tests

author maintainer
#7 lore raw
Quoted 143 lines · Jeff Layton
Quoted 132 lines · Chuck Lever III
Quoted 132 lines · Jeff Layton
Quoted 127 lines · Anna Schumaker
Quoted 49 lines · Jeff Layton
Quoted 47 lines · Chuck Lever III
Quoted 47 lines · Jeff Layton

I finally got my kdevops (https://github.com/linux-kdevops/kdevops) test rig working well enough to get some publishable results. To run fstests, kdevops will spin up a server and (in this case) 2 clients to run xfstests' auto group. One client mounts with default options, and the other uses NFSv3.

I tested 3 kernels:

v6.4.0 (stock release)
6.5.0-rc6-g4853c74bd7ab (Linus' tree as of a couple of days ago) 6.5.0-rc6-next-20230816-gef66bf8aeb91 (linux-next as of yesterday morning)

Here are the results summary of all 3:

KERNEL:    6.4.0
CPUS:      8

nfs_v3: 727 tests, 12 failures, 569 skipped, 14863 seconds
Failures: generic/053 generic/099 generic/105 generic/124
generic/193 generic/258 generic/294 generic/318 generic/319 generic/444 generic/528 generic/529
nfs_default: 727 tests, 18 failures, 452 skipped, 21899 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/193 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/528 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1021 skipped, 30 failures, 0 errors, 35096s

KERNEL:    6.5.0-rc6-g4853c74bd7ab
CPUS:      8

nfs_v3: 727 tests, 9 failures, 570 skipped, 14775 seconds
Failures: generic/053 generic/099 generic/105 generic/258
generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 16 failures, 453 skipped, 22326 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1023 skipped, 25 failures, 0 errors, 35396s

KERNEL:    6.5.0-rc6-next-20230816-gef66bf8aeb91
CPUS:      8
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

nfs_v3: 727 tests, 9 failures, 570 skipped, 14657 seconds
Failures: generic/053 generic/099 generic/105 generic/258
generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 18 failures, 453 skipped, 21757 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/683 generic/684 generic/688
Totals: 1454 tests, 1023 skipped, 27 failures, 0 errors, 34870s

As long as we're sharing results ... here is what I'm seeing with a 6.5-rc6 client & server:

anna@gouda ~ % xfstestsdb xunit list --results --runid 1741 --color=none

+------+----------------------+---------+----------+------+------+------+-------+
> run | device               | xunit   | hostname | pass | fail |
skip |  time |
+------+----------------------+---------+----------+------+------+------+-------+
> 1741 | server:/srv/xfs/test | tcp-3   | client   |  125 |    4 |
464 | 447 s |
> 1741 | server:/srv/xfs/test | tcp-4.0 | client   |  117 |   11 |
465 | 478 s |
> 1741 | server:/srv/xfs/test | tcp-4.1 | client   |  119 |   12 |
462 | 404 s |
> 1741 | server:/srv/xfs/test | tcp-4.2 | client   |  212 |   18 |
363 | 564 s |
+------+----------------------+---------+----------+------+------+------+-------+

anna@gouda ~ % xfstestsdb show --failure 1741 --color=none

+-------------+---------+---------+---------+---------+
>   testcase | tcp-3   | tcp-4.0 | tcp-4.1 | tcp-4.2 |
+-------------+---------+---------+---------+---------+
> generic/053 | passed  | failure | failure | failure |
> generic/099 | passed  | failure | failure | failure |
> generic/105 | passed  | failure | failure | failure |
> generic/140 | skipped | skipped | skipped | failure |
> generic/188 | skipped | skipped | skipped | failure |
> generic/258 | failure | passed  | passed  | failure |
> generic/294 | failure | failure | failure | failure |
> generic/318 | passed  | failure | failure | failure |
> generic/319 | passed  | failure | failure | failure |
> generic/357 | skipped | skipped | skipped | failure |
> generic/444 | failure | failure | failure | failure |
> generic/465 | passed  | failure | failure | failure |
> generic/513 | skipped | skipped | skipped | failure |
> generic/529 | passed  | failure | failure | failure |
> generic/604 | passed  | passed  | failure | passed  |
> generic/675 | skipped | skipped | skipped | failure |
> generic/688 | skipped | skipped | skipped | failure |
> generic/697 | passed  | failure | failure | failure |
>    nfs/002 | failure | failure | failure | failure |
+-------------+---------+---------+---------+---------+
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

With NFSv4.2, v6.4.0 has 2 extra failures that the current mainline kernel doesn't:

generic/193 (some sort of setattr problem)
generic/528 (known problem with btime handling in client that has been fixed)

While I haven't investigated, I'm assuming the 193 bug is also something that has been fixed in recent kernels. There are also 3 other NFSv3 tests that started passing since v6.4.0. I haven't looked into those.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

With the linux-next kernel there are 2 new regressions:

generic/683
generic/684

Both of these look like problems with setuid/setgid stripping, and still need to be investigated. I have more verbose result info on the test failures if anyone is interested.

Interesting that I'm not seeing the 683 & 684 failures. What type of filesystem is your server exporting?

Quoted 9 lines
Quoted 4 lines

btrfs

You are testing linux-next? I need to go back and confirm these results too.

IMO linux-next is quite important : we keep hitting bugs that appear only after integration -- block and network changes in other trees especially can impact the NFS drivers.

Quoted 5 lines

Indeed, I suspect this is probably something from the vfs tree (though we definitely need to confirm that). Today I'm testing:

    6.5.0-rc6-next-20230817-g47762f086974

Nope, I was wrong. I ran a bisect and it landed here. I confirmed it by turning off leases on the nfs server and the test started passing. I probably won't have the cycles to chase this down further.

The capture looks something like this:

OPEN (get a write delegation
WRITE
CLOSE
SETATTR (mode 06666)

...then presumably a task on the client opens the file again, but the setuid bits don't get stripped.

I think either the client will need to strip these bits on a delegated open, or we'll need to recall write delegations from the client when it tries to do a SETATTR with a mode that could later end up needing to be stripped on a subsequent open:

66ce3e3b98a7a9e970ea463a7f7dc0575c0a244b is the first bad commit commit 66ce3e3b98a7a9e970ea463a7f7dc0575c0a244b

Author: Dai Ngo <[email protected]>
Date:   Thu Jun 29 18:52:40 2023 -0700

NFSD: Enable write delegation support

#8 lore raw
Quoted 159 lines · Jeff Layton
Quoted 130 lines · Jeff Layton
Quoted 129 lines · Chuck Lever III
Quoted 129 lines · Jeff Layton
Quoted 113 lines · Anna Schumaker
Quoted 47 lines · Jeff Layton
Quoted 46 lines · Chuck Lever III
Quoted 46 lines · Jeff Layton

I finally got my kdevops (https://github.com/linux-kdevops/kdevops) test rig working well enough to get some publishable results. To run fstests, kdevops will spin up a server and (in this case) 2 clients to run xfstests' auto group. One client mounts with default options, and the other uses NFSv3.

I tested 3 kernels:

v6.4.0 (stock release)
6.5.0-rc6-g4853c74bd7ab (Linus' tree as of a couple of days ago) 6.5.0-rc6-next-20230816-gef66bf8aeb91 (linux-next as of yesterday morning)

Here are the results summary of all 3:

KERNEL:    6.4.0
CPUS:      8

nfs_v3: 727 tests, 12 failures, 569 skipped, 14863 seconds
Failures: generic/053 generic/099 generic/105 generic/124
generic/193 generic/258 generic/294 generic/318 generic/319 generic/444 generic/528 generic/529
nfs_default: 727 tests, 18 failures, 452 skipped, 21899 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/193 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/528 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1021 skipped, 30 failures, 0 errors, 35096s

KERNEL:    6.5.0-rc6-g4853c74bd7ab
CPUS:      8

nfs_v3: 727 tests, 9 failures, 570 skipped, 14775 seconds
Failures: generic/053 generic/099 generic/105 generic/258
generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 16 failures, 453 skipped, 22326 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1023 skipped, 25 failures, 0 errors, 35396s

KERNEL:    6.5.0-rc6-next-20230816-gef66bf8aeb91
CPUS:      8
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

nfs_v3: 727 tests, 9 failures, 570 skipped, 14657 seconds
Failures: generic/053 generic/099 generic/105 generic/258
generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 18 failures, 453 skipped, 21757 seconds
Failures: generic/053 generic/099 generic/105 generic/186
generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/683 generic/684 generic/688
Totals: 1454 tests, 1023 skipped, 27 failures, 0 errors, 34870s

As long as we're sharing results ... here is what I'm seeing with a 6.5-rc6 client & server:

anna@gouda ~ % xfstestsdb xunit list --results --runid 1741 --color=none

+------+----------------------+---------+----------+------+------+------+-------+
> run | device               | xunit   | hostname | pass | fail |
skip |  time |
+------+----------------------+---------+----------+------+------+------+-------+
> 1741 | server:/srv/xfs/test | tcp-3   | client   |  125 |    4 |
464 | 447 s |
> 1741 | server:/srv/xfs/test | tcp-4.0 | client   |  117 |   11 |
465 | 478 s |
> 1741 | server:/srv/xfs/test | tcp-4.1 | client   |  119 |   12 |
462 | 404 s |
> 1741 | server:/srv/xfs/test | tcp-4.2 | client   |  212 |   18 |
363 | 564 s |
+------+----------------------+---------+----------+------+------+------+-------+

anna@gouda ~ % xfstestsdb show --failure 1741 --color=none

+-------------+---------+---------+---------+---------+
>    testcase | tcp-3   | tcp-4.0 | tcp-4.1 | tcp-4.2 |
+-------------+---------+---------+---------+---------+
> generic/053 | passed  | failure | failure | failure |
> generic/099 | passed  | failure | failure | failure |
> generic/105 | passed  | failure | failure | failure |
> generic/140 | skipped | skipped | skipped | failure |
> generic/188 | skipped | skipped | skipped | failure |
> generic/258 | failure | passed  | passed  | failure |
> generic/294 | failure | failure | failure | failure |
> generic/318 | passed  | failure | failure | failure |
> generic/319 | passed  | failure | failure | failure |
> generic/357 | skipped | skipped | skipped | failure |
> generic/444 | failure | failure | failure | failure |
> generic/465 | passed  | failure | failure | failure |
> generic/513 | skipped | skipped | skipped | failure |
> generic/529 | passed  | failure | failure | failure |
> generic/604 | passed  | passed  | failure | passed  |
> generic/675 | skipped | skipped | skipped | failure |
> generic/688 | skipped | skipped | skipped | failure |
> generic/697 | passed  | failure | failure | failure |
>     nfs/002 | failure | failure | failure | failure |
+-------------+---------+---------+---------+---------+
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines

With NFSv4.2, v6.4.0 has 2 extra failures that the current mainline kernel doesn't:

generic/193 (some sort of setattr problem)
generic/528 (known problem with btime handling in client that has been fixed)

While I haven't investigated, I'm assuming the 193 bug is also something that has been fixed in recent kernels. There are also 3 other NFSv3 tests that started passing since v6.4.0. I haven't looked into those.

With the linux-next kernel there are 2 new regressions:

Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
   generic/683
   generic/684

Both of these look like problems with setuid/setgid stripping, and still need to be investigated. I have more verbose result info on the test failures if anyone is interested.

Interesting that I'm not seeing the 683 & 684 failures. What type of filesystem is your server exporting?

btrfs

You are testing linux-next? I need to go back and confirm these results too.

Quoted 9 lines
Quoted 4 lines

IMO linux-next is quite important : we keep hitting bugs that appear only after integration -- block and network changes in other trees especially can impact the NFS drivers.

Indeed, I suspect this is probably something from the vfs tree (though we definitely need to confirm that). Today I'm testing:

     6.5.0-rc6-next-20230817-g47762f086974

Nope, I was wrong. I ran a bisect and it landed here. I confirmed it by turning off leases on the nfs server and the test started passing. I probably won't have the cycles to chase this down further.

The capture looks something like this:

OPEN (get a write delegation
WRITE
CLOSE
SETATTR (mode 06666)

...then presumably a task on the client opens the file again, but the setuid bits don't get stripped.

I think either the client will need to strip these bits on a delegated open, or we'll need to recall write delegations from the client when it tries to do a SETATTR with a mode that could later end up needing to be stripped on a subsequent open:

Quoted 6 lines

66ce3e3b98a7a9e970ea463a7f7dc0575c0a244b is the first bad commit commit 66ce3e3b98a7a9e970ea463a7f7dc0575c0a244b

Author: Dai Ngo <[email protected]>
Date:   Thu Jun 29 18:52:40 2023 -0700

NFSD: Enable write delegation support

The SETATTR should cause the delegation to be recalled. However, I think there is an optimization on server that skips the recall if the SETATTR comes from the same client that has the delegation.

I'll take a look.

Thanks,
-Dai

Quoted 1 line
#9 lore raw
Quoted 170 lines · On 8/17/23 3:23 PM, [email protected] wrote:
Quoted 168 lines · Jeff Layton
Quoted 134 lines · Jeff Layton
Quoted 133 lines · Chuck Lever III
Quoted 133 lines · Jeff Layton
Quoted 117 lines · Anna Schumaker
Quoted 47 lines · Jeff Layton
Quoted 46 lines · Chuck Lever III
Quoted 46 lines · Jeff Layton

I finally got my kdevops (https://github.com/linux-kdevops/kdevops) test rig working well enough to get some publishable results. To run fstests, kdevops will spin up a server and (in this case) 2 clients to run xfstests' auto group. One client mounts with default options, and the other uses NFSv3.

I tested 3 kernels:

v6.4.0 (stock release)
6.5.0-rc6-g4853c74bd7ab (Linus' tree as of a couple of days ago) 6.5.0-rc6-next-20230816-gef66bf8aeb91 (linux-next as of yesterday morning)

Here are the results summary of all 3:

KERNEL:    6.4.0
CPUS:      8

nfs_v3: 727 tests, 12 failures, 569 skipped, 14863 seconds
Failures: generic/053 generic/099 generic/105 generic/124 generic/193 generic/258 generic/294 generic/318 generic/319 generic/444 generic/528 generic/529
nfs_default: 727 tests, 18 failures, 452 skipped, 21899 seconds
Failures: generic/053 generic/099 generic/105 generic/186 generic/187 generic/193 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/528 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1021 skipped, 30 failures, 0 errors, 35096s

KERNEL:    6.5.0-rc6-g4853c74bd7ab
CPUS:      8

nfs_v3: 727 tests, 9 failures, 570 skipped, 14775 seconds
Failures: generic/053 generic/099 generic/105 generic/258 generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 16 failures, 453 skipped, 22326 seconds
Failures: generic/053 generic/099 generic/105 generic/186 generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1023 skipped, 25 failures, 0 errors, 35396s

KERNEL:    6.5.0-rc6-next-20230816-gef66bf8aeb91
CPUS:      8

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

nfs_v3: 727 tests, 9 failures, 570 skipped, 14657 seconds
Failures: generic/053 generic/099 generic/105 generic/258 generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 18 failures, 453 skipped, 21757 seconds
Failures: generic/053 generic/099 generic/105 generic/186 generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/683 generic/684 generic/688
Totals: 1454 tests, 1023 skipped, 27 failures, 0 errors, 34870s

As long as we're sharing results ... here is what I'm seeing with a 6.5-rc6 client & server:

anna@gouda ~ % xfstestsdb xunit list --results --runid 1741 --color=none
+------+----------------------+---------+----------+------+------+------+-------+

Quoted 1 line

run | device               | xunit   | hostname | pass | fail |

skip |  time |
+------+----------------------+---------+----------+------+------+------+-------+

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-3   | client   |  125 |    4 |

464 | 447 s |

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-4.0 | client   |  117 |   11 |

465 | 478 s |

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-4.1 | client   |  119 |   12 |

462 | 404 s |

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-4.2 | client   |  212 |   18 |

363 | 564 s |
+------+----------------------+---------+----------+------+------+------+-------+

anna@gouda ~ % xfstestsdb show --failure 1741 --color=none

+-------------+---------+---------+---------+---------+
>    testcase | tcp-3   | tcp-4.0 | tcp-4.1 | tcp-4.2 |
+-------------+---------+---------+---------+---------+
> generic/053 | passed  | failure | failure | failure |
> generic/099 | passed  | failure | failure | failure |
> generic/105 | passed  | failure | failure | failure |
> generic/140 | skipped | skipped | skipped | failure |
> generic/188 | skipped | skipped | skipped | failure |
> generic/258 | failure | passed  | passed  | failure |
> generic/294 | failure | failure | failure | failure |
> generic/318 | passed  | failure | failure | failure |
> generic/319 | passed  | failure | failure | failure |
> generic/357 | skipped | skipped | skipped | failure |
> generic/444 | failure | failure | failure | failure |
> generic/465 | passed  | failure | failure | failure |
> generic/513 | skipped | skipped | skipped | failure |
> generic/529 | passed  | failure | failure | failure |
> generic/604 | passed  | passed  | failure | passed  |
> generic/675 | skipped | skipped | skipped | failure |
> generic/688 | skipped | skipped | skipped | failure |
> generic/697 | passed  | failure | failure | failure |
>     nfs/002 | failure | failure | failure | failure |
+-------------+---------+---------+---------+---------+
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines

With NFSv4.2, v6.4.0 has 2 extra failures that the current mainline kernel doesn't:

   generic/193 (some sort of setattr problem)
   generic/528 (known problem with btime handling in client that has been fixed)

While I haven't investigated, I'm assuming the 193 bug is also something that has been fixed in recent kernels. There are also 3 other NFSv3 tests that started passing since v6.4.0. I haven't looked into those.

With the linux-next kernel there are 2 new regressions:

Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

   generic/683
   generic/684

Both of these look like problems with setuid/setgid stripping, and still need to be investigated. I have more verbose result info on the test failures if anyone is interested.

Interesting that I'm not seeing the 683 & 684 failures. What type of filesystem is your server exporting?

btrfs

You are testing linux-next? I need to go back and confirm these results too.

Quoted 9 lines
Quoted 4 lines

IMO linux-next is quite important : we keep hitting bugs that appear only after integration -- block and network changes in other trees especially can impact the NFS drivers.

Indeed, I suspect this is probably something from the vfs tree (though we definitely need to confirm that). Today I'm testing:

     6.5.0-rc6-next-20230817-g47762f086974

Nope, I was wrong. I ran a bisect and it landed here. I confirmed it by turning off leases on the nfs server and the test started passing. I probably won't have the cycles to chase this down further.

The capture looks something like this:

OPEN (get a write delegation
WRITE
CLOSE
SETATTR (mode 06666)

...then presumably a task on the client opens the file again, but the setuid bits don't get stripped.

I think either the client will need to strip these bits on a delegated open, or we'll need to recall write delegations from the client when it tries to do a SETATTR with a mode that could later end up needing to be stripped on a subsequent open:

66ce3e3b98a7a9e970ea463a7f7dc0575c0a244b is the first bad commit commit 66ce3e3b98a7a9e970ea463a7f7dc0575c0a244b
Author: Dai Ngo <[email protected]>
Date:   Thu Jun 29 18:52:40 2023 -0700

Quoted 5 lines
Quoted 1 line

     NFSD: Enable write delegation support

The SETATTR should cause the delegation to be recalled. However, I think there is an optimization on server that skips the recall if the SETATTR comes from the same client that has the delegation.

The optimization on the server was done by this commit:

28df3d1539de nfsd: clients don't need to break their own delegations

Perhaps we should allow this optimization for read delegation only?

Or should the NFS client be responsible for handling the SETATTR and and local OPEN on the file that has write delegation granted?

-Dai

Quoted 7 lines

I'll take a look.

Thanks,
-Dai

Quoted 1 line
author maintainer
#10 lore raw
Quoted 191 lines · On Thu, 2023-08-17 at 15:59 -0700, [email protected] wrote:
Quoted 184 lines · On 8/17/23 3:23 PM, [email protected] wrote:
Quoted 182 lines · Jeff Layton
Quoted 148 lines · Jeff Layton
Quoted 147 lines · Chuck Lever III
Quoted 147 lines · Jeff Layton
Quoted 131 lines · Anna Schumaker
Quoted 52 lines · Jeff Layton
Quoted 51 lines · Chuck Lever III
Quoted 51 lines

On Aug 17, 2023, at 7:21 AM, Jeff Layton <[email protected]> wrote:

I finally got my kdevops
(https://github.com/linux-kdevops/kdevops) test
rig working well enough to get some publishable results. To run fstests,
kdevops will spin up a server and (in this case) 2 clients to run xfstests' auto group. One client mounts with default options, and the
other uses NFSv3.

I tested 3 kernels:

v6.4.0 (stock release)
6.5.0-rc6-g4853c74bd7ab (Linus' tree as of a couple of days ago) 6.5.0-rc6-next-20230816-gef66bf8aeb91 (linux-next as of yesterday morning)

Here are the results summary of all 3:

KERNEL:    6.4.0
CPUS:      8

nfs_v3: 727 tests, 12 failures, 569 skipped, 14863 seconds
Failures: generic/053 generic/099 generic/105 generic/124 generic/193 generic/258 generic/294 generic/318 generic/319 generic/444 generic/528 generic/529
nfs_default: 727 tests, 18 failures, 452 skipped, 21899 seconds
Failures: generic/053 generic/099 generic/105 generic/186 generic/187 generic/193 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/528 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1021 skipped, 30 failures, 0 errors, 35096s

KERNEL:    6.5.0-rc6-g4853c74bd7ab
CPUS:      8

nfs_v3: 727 tests, 9 failures, 570 skipped, 14775 seconds
Failures: generic/053 generic/099 generic/105 generic/258 generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 16 failures, 453 skipped, 22326 seconds
Failures: generic/053 generic/099 generic/105 generic/186 generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1023 skipped, 25 failures, 0 errors, 35396s

KERNEL:    6.5.0-rc6-next-20230816-gef66bf8aeb91
CPUS:      8

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

nfs_v3: 727 tests, 9 failures, 570 skipped, 14657 seconds
Failures: generic/053 generic/099 generic/105 generic/258 generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 18 failures, 453 skipped, 21757 seconds
Failures: generic/053 generic/099 generic/105 generic/186 generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/683 generic/684 generic/688
Totals: 1454 tests, 1023 skipped, 27 failures, 0 errors, 34870s

As long as we're sharing results ... here is what I'm seeing with a 6.5-rc6 client & server:

anna@gouda ~ % xfstestsdb xunit list --results --runid 1741 --color=none
+------+----------------------+---------+----------+------+------+------+-------+

Quoted 1 line

run | device               | xunit   | hostname | pass | fail |

skip |  time |
+------+----------------------+---------+----------+------+------+------+-------+

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-3   | client   |  125 |    4 |

464 | 447 s |

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-4.0 | client   |  117 |   11 |

465 | 478 s |

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-4.1 | client   |  119 |   12 |

462 | 404 s |

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-4.2 | client   |  212 |   18 |

363 | 564 s |
+------+----------------------+---------+----------+------+------+------+-------+

anna@gouda ~ % xfstestsdb show --failure 1741 --color=none

+-------------+---------+---------+---------+---------+
>    testcase | tcp-3   | tcp-4.0 | tcp-4.1 | tcp-4.2 |
+-------------+---------+---------+---------+---------+
> generic/053 | passed  | failure | failure | failure |
> generic/099 | passed  | failure | failure | failure |
> generic/105 | passed  | failure | failure | failure |
> generic/140 | skipped | skipped | skipped | failure |
> generic/188 | skipped | skipped | skipped | failure |
> generic/258 | failure | passed  | passed  | failure |
> generic/294 | failure | failure | failure | failure |
> generic/318 | passed  | failure | failure | failure |
> generic/319 | passed  | failure | failure | failure |
> generic/357 | skipped | skipped | skipped | failure |
> generic/444 | failure | failure | failure | failure |
> generic/465 | passed  | failure | failure | failure |
> generic/513 | skipped | skipped | skipped | failure |
> generic/529 | passed  | failure | failure | failure |
> generic/604 | passed  | passed  | failure | passed  |
> generic/675 | skipped | skipped | skipped | failure |
> generic/688 | skipped | skipped | skipped | failure |
> generic/697 | passed  | failure | failure | failure |
>     nfs/002 | failure | failure | failure | failure |
+-------------+---------+---------+---------+---------+
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines

With NFSv4.2, v6.4.0 has 2 extra failures that the current mainline
kernel doesn't:

   generic/193 (some sort of setattr problem) generic/528 (known problem with btime handling in client that has been fixed)

While I haven't investigated, I'm assuming the 193 bug is also something
that has been fixed in recent kernels. There are also 3 other NFSv3
tests that started passing since v6.4.0. I haven't looked into those.

With the linux-next kernel there are 2 new regressions:

   generic/683
   generic/684

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Both of these look like problems with setuid/setgid stripping, and still
need to be investigated. I have more verbose result info on the test
failures if anyone is interested.

Interesting that I'm not seeing the 683 & 684 failures. What type of filesystem is your server exporting?

btrfs

You are testing linux-next? I need to go back and confirm these results
too.

Quoted 9 lines
Quoted 4 lines

IMO linux-next is quite important : we keep hitting bugs that appear only after integration -- block and network changes in other trees especially can impact the NFS drivers.

Indeed, I suspect this is probably something from the vfs tree (though we definitely need to confirm that). Today I'm testing:

     6.5.0-rc6-next-20230817-g47762f086974

Nope, I was wrong. I ran a bisect and it landed here. I confirmed it by turning off leases on the nfs server and the test started passing. I probably won't have the cycles to chase this down further.

The capture looks something like this:

OPEN (get a write delegation
WRITE
CLOSE
SETATTR (mode 06666)

...then presumably a task on the client opens the file again, but the setuid bits don't get stripped.

I think either the client will need to strip these bits on a delegated open, or we'll need to recall write delegations from the client when it tries to do a SETATTR with a mode that could later end up needing to be stripped on a subsequent open:

66ce3e3b98a7a9e970ea463a7f7dc0575c0a244b is the first bad commit commit 66ce3e3b98a7a9e970ea463a7f7dc0575c0a244b
Author: Dai Ngo <[email protected]>
Date:   Thu Jun 29 18:52:40 2023 -0700

Quoted 5 lines
Quoted 1 line

     NFSD: Enable write delegation support

The SETATTR should cause the delegation to be recalled. However, I think there is an optimization on server that skips the recall if the SETATTR comes from the same client that has the delegation.

Quoted 9 lines

The optimization on the server was done by this commit:

28df3d1539de nfsd: clients don't need to break their own delegations

Perhaps we should allow this optimization for read delegation only?

Or should the NFS client be responsible for handling the SETATTR and and local OPEN on the file that has write delegation granted?

I think that setuid/setgid files are really a special case.

We already avoid giving out delegations on setuid/gid files. What we're not doing currently is revoking the write delegation if the holder tries to set a mode that involves a setuid/gid bit. If we add that, then that should close the hole, I think.

Signature · 1 line
Jeff Layton <[email protected]>
#11 lore raw
Quoted 198 lines · Jeff Layton
Quoted 188 lines · On Thu, 2023-08-17 at 15:59 -0700, [email protected] wrote:
Quoted 183 lines · On 8/17/23 3:23 PM, [email protected] wrote:
Quoted 182 lines · Jeff Layton
Quoted 148 lines · Jeff Layton
Quoted 147 lines · Chuck Lever III
Quoted 147 lines · Jeff Layton
Quoted 131 lines · Anna Schumaker
Quoted 52 lines · Jeff Layton
Quoted 51 lines · Chuck Lever III
Quoted 51 lines

On Aug 17, 2023, at 7:21 AM, Jeff Layton <[email protected]> wrote:

I finally got my kdevops
(https://github.com/linux-kdevops/kdevops) test
rig working well enough to get some publishable results. To run fstests,
kdevops will spin up a server and (in this case) 2 clients to run xfstests' auto group. One client mounts with default options, and the
other uses NFSv3.

I tested 3 kernels:

v6.4.0 (stock release)
6.5.0-rc6-g4853c74bd7ab (Linus' tree as of a couple of days ago) 6.5.0-rc6-next-20230816-gef66bf8aeb91 (linux-next as of yesterday morning)

Here are the results summary of all 3:

KERNEL:    6.4.0
CPUS:      8

nfs_v3: 727 tests, 12 failures, 569 skipped, 14863 seconds
Failures: generic/053 generic/099 generic/105 generic/124
   generic/193 generic/258 generic/294 generic/318 generic/319 generic/444 generic/528 generic/529
nfs_default: 727 tests, 18 failures, 452 skipped, 21899 seconds
Failures: generic/053 generic/099 generic/105 generic/186
   generic/187 generic/193 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/528 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1021 skipped, 30 failures, 0 errors, 35096s

KERNEL:    6.5.0-rc6-g4853c74bd7ab
CPUS:      8

nfs_v3: 727 tests, 9 failures, 570 skipped, 14775 seconds
Failures: generic/053 generic/099 generic/105 generic/258
   generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 16 failures, 453 skipped, 22326 seconds
Failures: generic/053 generic/099 generic/105 generic/186
   generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1023 skipped, 25 failures, 0 errors, 35396s

KERNEL:    6.5.0-rc6-next-20230816-gef66bf8aeb91
CPUS:      8

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

nfs_v3: 727 tests, 9 failures, 570 skipped, 14657 seconds
Failures: generic/053 generic/099 generic/105 generic/258
   generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 18 failures, 453 skipped, 21757 seconds
Failures: generic/053 generic/099 generic/105 generic/186
   generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/683 generic/684 generic/688
Totals: 1454 tests, 1023 skipped, 27 failures, 0 errors, 34870s

As long as we're sharing results ... here is what I'm seeing with a 6.5-rc6 client & server:

anna@gouda ~ % xfstestsdb xunit list --results --runid 1741 --color=none
+------+----------------------+---------+----------+------+------+------+-------+

Quoted 1 line

run | device               | xunit   | hostname | pass | fail |

skip |  time |
+------+----------------------+---------+----------+------+------+------+-------+

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-3   | client   |  125 |    4 |

464 | 447 s |

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-4.0 | client   |  117 |   11 |

465 | 478 s |

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-4.1 | client   |  119 |   12 |

462 | 404 s |

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-4.2 | client   |  212 |   18 |

363 | 564 s |
+------+----------------------+---------+----------+------+------+------+-------+

anna@gouda ~ % xfstestsdb show --failure 1741 --color=none

+-------------+---------+---------+---------+---------+
>     testcase | tcp-3   | tcp-4.0 | tcp-4.1 | tcp-4.2 |
+-------------+---------+---------+---------+---------+
> generic/053 | passed  | failure | failure | failure |
> generic/099 | passed  | failure | failure | failure |
> generic/105 | passed  | failure | failure | failure |
> generic/140 | skipped | skipped | skipped | failure |
> generic/188 | skipped | skipped | skipped | failure |
> generic/258 | failure | passed  | passed  | failure |
> generic/294 | failure | failure | failure | failure |
> generic/318 | passed  | failure | failure | failure |
> generic/319 | passed  | failure | failure | failure |
> generic/357 | skipped | skipped | skipped | failure |
> generic/444 | failure | failure | failure | failure |
> generic/465 | passed  | failure | failure | failure |
> generic/513 | skipped | skipped | skipped | failure |
> generic/529 | passed  | failure | failure | failure |
> generic/604 | passed  | passed  | failure | passed  |
> generic/675 | skipped | skipped | skipped | failure |
> generic/688 | skipped | skipped | skipped | failure |
> generic/697 | passed  | failure | failure | failure |
>      nfs/002 | failure | failure | failure | failure |
+-------------+---------+---------+---------+---------+
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines

With NFSv4.2, v6.4.0 has 2 extra failures that the current mainline
kernel doesn't:

   generic/193 (some sort of setattr problem) generic/528 (known problem with btime handling in client
that has been fixed)

While I haven't investigated, I'm assuming the 193 bug is also something
that has been fixed in recent kernels. There are also 3 other NFSv3
tests that started passing since v6.4.0. I haven't looked into those.

With the linux-next kernel there are 2 new regressions:

   generic/683
   generic/684

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Both of these look like problems with setuid/setgid stripping, and still
need to be investigated. I have more verbose result info on the test
failures if anyone is interested.

Interesting that I'm not seeing the 683 & 684 failures. What type of filesystem is your server exporting?

btrfs

You are testing linux-next? I need to go back and confirm these results
too.

Quoted 9 lines
Quoted 4 lines

IMO linux-next is quite important : we keep hitting bugs that appear only after integration -- block and network changes in other trees especially can impact the NFS drivers.

Indeed, I suspect this is probably something from the vfs tree (though we definitely need to confirm that). Today I'm testing:

     6.5.0-rc6-next-20230817-g47762f086974

Nope, I was wrong. I ran a bisect and it landed here. I confirmed it by turning off leases on the nfs server and the test started passing. I probably won't have the cycles to chase this down further.

The capture looks something like this:

OPEN (get a write delegation
WRITE
CLOSE
SETATTR (mode 06666)

...then presumably a task on the client opens the file again, but the setuid bits don't get stripped.

I think either the client will need to strip these bits on a delegated open, or we'll need to recall write delegations from the client when it tries to do a SETATTR with a mode that could later end up needing to be stripped on a subsequent open:

66ce3e3b98a7a9e970ea463a7f7dc0575c0a244b is the first bad commit commit 66ce3e3b98a7a9e970ea463a7f7dc0575c0a244b
Author: Dai Ngo <[email protected]>
Date:   Thu Jun 29 18:52:40 2023 -0700

Quoted 4 lines
Quoted 1 line

     NFSD: Enable write delegation support

The SETATTR should cause the delegation to be recalled. However, I think there is an optimization on server that skips the recall if the SETATTR comes from the same client that has the delegation.

Quoted 9 lines

The optimization on the server was done by this commit:

28df3d1539de nfsd: clients don't need to break their own delegations

Perhaps we should allow this optimization for read delegation only?

Or should the NFS client be responsible for handling the SETATTR and and local OPEN on the file that has write delegation granted?

Quoted 6 lines

I think that setuid/setgid files are really a special case.

We already avoid giving out delegations on setuid/gid files. What we're not doing currently is revoking the write delegation if the holder tries to set a mode that involves a setuid/gid bit. If we add that, then that should close the hole, I think.

This approach seems reasonable, I'll work the patch to take care of this condition.

Thanks Jeff,
-Dai

Quoted 1 line
#12 lore raw
In reply to Jeff Layton
Quoted 167 lines · Jeff Layton
Quoted 166 lines · On Thu, 2023-08-17 at 15:59 -0700, [email protected] wrote:
Quoted 165 lines · On 8/17/23 3:23 PM, [email protected] wrote:
Quoted 164 lines · Jeff Layton
Quoted 148 lines · Jeff Layton
Quoted 147 lines · Chuck Lever III
Quoted 147 lines · Jeff Layton
Quoted 131 lines · Anna Schumaker
Quoted 52 lines · Jeff Layton
Quoted 51 lines · Chuck Lever III
Quoted 51 lines

On Aug 17, 2023, at 7:21 AM, Jeff Layton <[email protected]> wrote:

I finally got my kdevops
(https://github.com/linux-kdevops/kdevops) test
rig working well enough to get some publishable results. To run fstests,
kdevops will spin up a server and (in this case) 2 clients to run xfstests' auto group. One client mounts with default options, and the
other uses NFSv3.

I tested 3 kernels:

v6.4.0 (stock release)
6.5.0-rc6-g4853c74bd7ab (Linus' tree as of a couple of days ago) 6.5.0-rc6-next-20230816-gef66bf8aeb91 (linux-next as of yesterday morning)

Here are the results summary of all 3:

KERNEL:    6.4.0
CPUS:      8

nfs_v3: 727 tests, 12 failures, 569 skipped, 14863 seconds
Failures: generic/053 generic/099 generic/105 generic/124
   generic/193 generic/258 generic/294 generic/318 generic/319 generic/444 generic/528 generic/529
nfs_default: 727 tests, 18 failures, 452 skipped, 21899 seconds
Failures: generic/053 generic/099 generic/105 generic/186
   generic/187 generic/193 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/528 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1021 skipped, 30 failures, 0 errors, 35096s

KERNEL:    6.5.0-rc6-g4853c74bd7ab
CPUS:      8

nfs_v3: 727 tests, 9 failures, 570 skipped, 14775 seconds
Failures: generic/053 generic/099 generic/105 generic/258
   generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 16 failures, 453 skipped, 22326 seconds
Failures: generic/053 generic/099 generic/105 generic/186
   generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1023 skipped, 25 failures, 0 errors, 35396s

KERNEL:    6.5.0-rc6-next-20230816-gef66bf8aeb91
CPUS:      8

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

nfs_v3: 727 tests, 9 failures, 570 skipped, 14657 seconds
Failures: generic/053 generic/099 generic/105 generic/258
   generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 18 failures, 453 skipped, 21757 seconds
Failures: generic/053 generic/099 generic/105 generic/186
   generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/683 generic/684 generic/688
Totals: 1454 tests, 1023 skipped, 27 failures, 0 errors, 34870s

As long as we're sharing results ... here is what I'm seeing with a 6.5-rc6 client & server:

anna@gouda ~ % xfstestsdb xunit list --results --runid 1741 --color=none
+------+----------------------+---------+----------+------+------+------+-------+

Quoted 1 line

run | device               | xunit   | hostname | pass | fail |

skip |  time |
+------+----------------------+---------+----------+------+------+------+-------+

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-3   | client   |  125 |    4 |

464 | 447 s |

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-4.0 | client   |  117 |   11 |

465 | 478 s |

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-4.1 | client   |  119 |   12 |

462 | 404 s |

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-4.2 | client   |  212 |   18 |

363 | 564 s |
+------+----------------------+---------+----------+------+------+------+-------+

anna@gouda ~ % xfstestsdb show --failure 1741 --color=none

+-------------+---------+---------+---------+---------+
>     testcase | tcp-3   | tcp-4.0 | tcp-4.1 | tcp-4.2 |
+-------------+---------+---------+---------+---------+
> generic/053 | passed  | failure | failure | failure |
> generic/099 | passed  | failure | failure | failure |
> generic/105 | passed  | failure | failure | failure |
> generic/140 | skipped | skipped | skipped | failure |
> generic/188 | skipped | skipped | skipped | failure |
> generic/258 | failure | passed  | passed  | failure |
> generic/294 | failure | failure | failure | failure |
> generic/318 | passed  | failure | failure | failure |
> generic/319 | passed  | failure | failure | failure |
> generic/357 | skipped | skipped | skipped | failure |
> generic/444 | failure | failure | failure | failure |
> generic/465 | passed  | failure | failure | failure |
> generic/513 | skipped | skipped | skipped | failure |
> generic/529 | passed  | failure | failure | failure |
> generic/604 | passed  | passed  | failure | passed  |
> generic/675 | skipped | skipped | skipped | failure |
> generic/688 | skipped | skipped | skipped | failure |
> generic/697 | passed  | failure | failure | failure |
>      nfs/002 | failure | failure | failure | failure |
+-------------+---------+---------+---------+---------+
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines

With NFSv4.2, v6.4.0 has 2 extra failures that the current mainline
kernel doesn't:

   generic/193 (some sort of setattr problem) generic/528 (known problem with btime handling in client
that has been fixed)

While I haven't investigated, I'm assuming the 193 bug is also something
that has been fixed in recent kernels. There are also 3 other NFSv3
tests that started passing since v6.4.0. I haven't looked into those.

With the linux-next kernel there are 2 new regressions:

   generic/683
   generic/684

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Both of these look like problems with setuid/setgid stripping, and still
need to be investigated. I have more verbose result info on the test
failures if anyone is interested.

Interesting that I'm not seeing the 683 & 684 failures. What type of filesystem is your server exporting?

btrfs

You are testing linux-next? I need to go back and confirm these results
too.

Quoted 9 lines
Quoted 4 lines

IMO linux-next is quite important : we keep hitting bugs that appear only after integration -- block and network changes in other trees especially can impact the NFS drivers.

Indeed, I suspect this is probably something from the vfs tree (though we definitely need to confirm that). Today I'm testing:

     6.5.0-rc6-next-20230817-g47762f086974

Nope, I was wrong. I ran a bisect and it landed here. I confirmed it by turning off leases on the nfs server and the test started passing. I probably won't have the cycles to chase this down further.

The capture looks something like this:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

OPEN (get a write delegation
WRITE
CLOSE
SETATTR (mode 06666)

...then presumably a task on the client opens the file again, but the setuid bits don't get stripped.

OPEN (get a write delegation
WRITE
CLOSE
SETATTR (mode 06666)

The client continues with:

(ALLOCATE,GETATTR)  <<===  this is when the server stripped the SUID and SGID bit
READDIR             ====>  file mode shows 0666  (SUID & SGID were stripped)
READDIR             ====>  file mode shows 0666  (SUID & SGID were stripped)

DELERETURN

Here is stack trace of ALLOCATE when the SUID & SGID were stripped:

**** start of notify_change, notice the i_mode bits, SUID & SGID were set:
[notify_change]: d_iname[a] ia_valid[0x1a00] ia_mode[0x0] i_mode[0x8db6] [nfsd:2409:Mon Aug 21 23:05:31 2023]

                        KILL[0] KILL_SUID[1] KILL_SGID[1]

**** end of notify_change, notice the i_mode bits, SUID & SGID were stripped:
[notify_change]: RET[0] d_iname[a] ia_valid[0x1a01] ia_mode[0x81b6] i_mode[0x81b6] [nfsd:2409:Mon Aug 21 23:05:31 2023]

**** stack trace of notify_change comes from ALLOCATE:

Returning from:  0xffffffffb726e764 : notify_change+0x4/0x500 [kernel]
Returning to  :  0xffffffffb726bf99 : __file_remove_privs+0x119/0x170 [kernel]

0xffffffffb726cfad : file_modified_flags+0x4d/0x110 [kernel] 0xffffffffc0a2330b : xfs_file_fallocate+0xfb/0x490 [xfs] 0xffffffffb723e7d8 : vfs_fallocate+0x158/0x380 [kernel] 0xffffffffc0ddc30a : nfsd4_vfs_fallocate+0x4a/0x70 [nfsd] 0xffffffffc0def7f2 : nfsd4_allocate+0x72/0xc0 [nfsd] 0xffffffffc0df2663 : nfsd4_proc_compound+0x3d3/0x730 [nfsd] 0xffffffffc0dd633b : nfsd_dispatch+0xab/0x1d0 [nfsd] 0xffffffffc0bda476 : svc_process_common+0x306/0x6e0 [sunrpc] 0xffffffffc0bdb081 : svc_process+0x131/0x180 [sunrpc] 0xffffffffc0dd4864 : nfsd+0x84/0xd0 [nfsd] 0xffffffffb6f0bfd6 : kthread+0xe6/0x120 [kernel] 0xffffffffb6e587d4 : ret_from_fork+0x34/0x50 [kernel] 0xffffffffb6e03a3b : ret_from_fork_asm+0x1b/0x30 [kernel]

I think the problem here is that the client does not update the file attribute after ALLOCATE. The GETATTR in the ALLOCATE compound does not include the mode bits.

The READDIR's reply show the test file's mode has the SUID & SGID bit stripped (0666) but apparently these were not used o update the file attribute.

The test passes when server does not grant write delegation because:

OPEN
WRITE
CLOSE
SETATTR (06666)
OPEN (CLAIM_FH, NOCREATE)
ALLOCATE <<=== server clear SUID & SGID
GETATTR, CLOSE <<=== GETATTR has mode bit as 0666, client updates file attribute READDIR
READDIR

As expected, if the server recalls the write delegation when SETATTR with SUID/SGID set then the test passes. This is because it forces the client to send the 2nd OPEN with CLAIM_FH, NOCREATE and then the (GETATTR, CLOSE) which cause the client to update the file attribute.

-Dai

P.S I have the pcaps of the pass, fail and fixed case if anyone want to see. I do the tests with generic/683.

Quoted 31 lines
Quoted 15 lines
Quoted 11 lines
Quoted 11 lines

I think either the client will need to strip these bits on a delegated open, or we'll need to recall write delegations from the client when it tries to do a SETATTR with a mode that could later end up needing to be stripped on a subsequent open:

66ce3e3b98a7a9e970ea463a7f7dc0575c0a244b is the first bad commit commit 66ce3e3b98a7a9e970ea463a7f7dc0575c0a244b
Author: Dai Ngo <[email protected]>
Date:   Thu Jun 29 18:52:40 2023 -0700

Quoted 4 lines
Quoted 1 line

     NFSD: Enable write delegation support

The SETATTR should cause the delegation to be recalled. However, I think there is an optimization on server that skips the recall if the SETATTR comes from the same client that has the delegation.

Quoted 9 lines

The optimization on the server was done by this commit:

28df3d1539de nfsd: clients don't need to break their own delegations

Perhaps we should allow this optimization for read delegation only?

Or should the NFS client be responsible for handling the SETATTR and and local OPEN on the file that has write delegation granted?

I think that setuid/setgid files are really a special case.

We already avoid giving out delegations on setuid/gid files. What we're not doing currently is revoking the write delegation if the holder tries to set a mode that involves a setuid/gid bit. If we add that, then that should close the hole, I think.

author maintainer
#13 lore raw
Quoted 214 lines · On Tue, 2023-08-22 at 09:07 -0700, [email protected] wrote:
Quoted 167 lines · Jeff Layton
Quoted 166 lines · On Thu, 2023-08-17 at 15:59 -0700, [email protected] wrote:
Quoted 165 lines · On 8/17/23 3:23 PM, [email protected] wrote:
Quoted 164 lines · Jeff Layton
Quoted 148 lines · Jeff Layton
Quoted 147 lines · Chuck Lever III
Quoted 147 lines · Jeff Layton
Quoted 131 lines · Anna Schumaker
Quoted 52 lines · Jeff Layton
Quoted 51 lines · Chuck Lever III
Quoted 51 lines

On Aug 17, 2023, at 7:21 AM, Jeff Layton <[email protected]> wrote:

I finally got my kdevops
(https://github.com/linux-kdevops/kdevops) test
rig working well enough to get some publishable results. To run fstests,
kdevops will spin up a server and (in this case) 2 clients to run xfstests' auto group. One client mounts with default options, and the
other uses NFSv3.

I tested 3 kernels:

v6.4.0 (stock release)
6.5.0-rc6-g4853c74bd7ab (Linus' tree as of a couple of days ago) 6.5.0-rc6-next-20230816-gef66bf8aeb91 (linux-next as of yesterday morning)

Here are the results summary of all 3:

KERNEL:    6.4.0
CPUS:      8

nfs_v3: 727 tests, 12 failures, 569 skipped, 14863 seconds
Failures: generic/053 generic/099 generic/105 generic/124
   generic/193 generic/258 generic/294 generic/318 generic/319 generic/444 generic/528 generic/529
nfs_default: 727 tests, 18 failures, 452 skipped, 21899 seconds
Failures: generic/053 generic/099 generic/105 generic/186
   generic/187 generic/193 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/528 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1021 skipped, 30 failures, 0 errors, 35096s

KERNEL:    6.5.0-rc6-g4853c74bd7ab
CPUS:      8

nfs_v3: 727 tests, 9 failures, 570 skipped, 14775 seconds
Failures: generic/053 generic/099 generic/105 generic/258
   generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 16 failures, 453 skipped, 22326 seconds
Failures: generic/053 generic/099 generic/105 generic/186
   generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1023 skipped, 25 failures, 0 errors, 35396s

KERNEL:    6.5.0-rc6-next-20230816-gef66bf8aeb91
CPUS:      8

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

nfs_v3: 727 tests, 9 failures, 570 skipped, 14657 seconds
Failures: generic/053 generic/099 generic/105 generic/258
   generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 18 failures, 453 skipped, 21757 seconds
Failures: generic/053 generic/099 generic/105 generic/186
   generic/187 generic/294 generic/318 generic/319 generic/357 generic/444 generic/486 generic/513 generic/529 generic/578 generic/675 generic/683 generic/684 generic/688
Totals: 1454 tests, 1023 skipped, 27 failures, 0 errors, 34870s

As long as we're sharing results ... here is what I'm seeing with a 6.5-rc6 client & server:

anna@gouda ~ % xfstestsdb xunit list --results --runid 1741 --color=none
+------+----------------------+---------+----------+------+------+------+-------+

Quoted 1 line

run | device               | xunit   | hostname | pass | fail |

skip |  time |
+------+----------------------+---------+----------+------+------+------+-------+

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-3   | client   |  125 |    4 |

464 | 447 s |

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-4.0 | client   |  117 |   11 |

465 | 478 s |

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-4.1 | client   |  119 |   12 |

462 | 404 s |

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-4.2 | client   |  212 |   18 |

363 | 564 s |
+------+----------------------+---------+----------+------+------+------+-------+

anna@gouda ~ % xfstestsdb show --failure 1741 --color=none

+-------------+---------+---------+---------+---------+
>     testcase | tcp-3   | tcp-4.0 | tcp-4.1 | tcp-4.2 |
+-------------+---------+---------+---------+---------+
> generic/053 | passed  | failure | failure | failure |
> generic/099 | passed  | failure | failure | failure |
> generic/105 | passed  | failure | failure | failure |
> generic/140 | skipped | skipped | skipped | failure |
> generic/188 | skipped | skipped | skipped | failure |
> generic/258 | failure | passed  | passed  | failure |
> generic/294 | failure | failure | failure | failure |
> generic/318 | passed  | failure | failure | failure |
> generic/319 | passed  | failure | failure | failure |
> generic/357 | skipped | skipped | skipped | failure |
> generic/444 | failure | failure | failure | failure |
> generic/465 | passed  | failure | failure | failure |
> generic/513 | skipped | skipped | skipped | failure |
> generic/529 | passed  | failure | failure | failure |
> generic/604 | passed  | passed  | failure | passed  |
> generic/675 | skipped | skipped | skipped | failure |
> generic/688 | skipped | skipped | skipped | failure |
> generic/697 | passed  | failure | failure | failure |
>      nfs/002 | failure | failure | failure | failure |
+-------------+---------+---------+---------+---------+
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines

With NFSv4.2, v6.4.0 has 2 extra failures that the current mainline
kernel doesn't:

   generic/193 (some sort of setattr problem) generic/528 (known problem with btime handling in client
that has been fixed)

While I haven't investigated, I'm assuming the 193 bug is also something
that has been fixed in recent kernels. There are also 3 other NFSv3
tests that started passing since v6.4.0. I haven't looked into those.

With the linux-next kernel there are 2 new regressions:

   generic/683
   generic/684

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Both of these look like problems with setuid/setgid stripping, and still
need to be investigated. I have more verbose result info on the test
failures if anyone is interested.

Interesting that I'm not seeing the 683 & 684 failures. What type of filesystem is your server exporting?

btrfs

You are testing linux-next? I need to go back and confirm these results
too.

Quoted 9 lines
Quoted 4 lines

IMO linux-next is quite important : we keep hitting bugs that appear only after integration -- block and network changes in other trees especially can impact the NFS drivers.

Indeed, I suspect this is probably something from the vfs tree (though we definitely need to confirm that). Today I'm testing:

     6.5.0-rc6-next-20230817-g47762f086974

Nope, I was wrong. I ran a bisect and it landed here. I confirmed it by turning off leases on the nfs server and the test started passing. I probably won't have the cycles to chase this down further.

The capture looks something like this:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

OPEN (get a write delegation
WRITE
CLOSE
SETATTR (mode 06666)

...then presumably a task on the client opens the file again, but the setuid bits don't get stripped.

OPEN (get a write delegation
WRITE
CLOSE
SETATTR (mode 06666)

The client continues with:

(ALLOCATE,GETATTR)  <<===  this is when the server stripped the SUID and SGID bit
READDIR             ====>  file mode shows 0666  (SUID & SGID were stripped)
READDIR             ====>  file mode shows 0666  (SUID & SGID were stripped)

DELERETURN

Here is stack trace of ALLOCATE when the SUID & SGID were stripped:

**** start of notify_change, notice the i_mode bits, SUID & SGID were set:
[notify_change]: d_iname[a] ia_valid[0x1a00] ia_mode[0x0] i_mode[0x8db6] [nfsd:2409:Mon Aug 21 23:05:31 2023]

                         KILL[0] KILL_SUID[1] KILL_SGID[1]

**** end of notify_change, notice the i_mode bits, SUID & SGID were stripped:
[notify_change]: RET[0] d_iname[a] ia_valid[0x1a01] ia_mode[0x81b6] i_mode[0x81b6] [nfsd:2409:Mon Aug 21 23:05:31 2023]

**** stack trace of notify_change comes from ALLOCATE:

Returning from:  0xffffffffb726e764 : notify_change+0x4/0x500 [kernel]
Returning to  :  0xffffffffb726bf99 : __file_remove_privs+0x119/0x170 [kernel]

0xffffffffb726cfad : file_modified_flags+0x4d/0x110 [kernel] 0xffffffffc0a2330b : xfs_file_fallocate+0xfb/0x490 [xfs] 0xffffffffb723e7d8 : vfs_fallocate+0x158/0x380 [kernel] 0xffffffffc0ddc30a : nfsd4_vfs_fallocate+0x4a/0x70 [nfsd] 0xffffffffc0def7f2 : nfsd4_allocate+0x72/0xc0 [nfsd] 0xffffffffc0df2663 : nfsd4_proc_compound+0x3d3/0x730 [nfsd] 0xffffffffc0dd633b : nfsd_dispatch+0xab/0x1d0 [nfsd] 0xffffffffc0bda476 : svc_process_common+0x306/0x6e0 [sunrpc] 0xffffffffc0bdb081 : svc_process+0x131/0x180 [sunrpc] 0xffffffffc0dd4864 : nfsd+0x84/0xd0 [nfsd] 0xffffffffb6f0bfd6 : kthread+0xe6/0x120 [kernel] 0xffffffffb6e587d4 : ret_from_fork+0x34/0x50 [kernel] 0xffffffffb6e03a3b : ret_from_fork_asm+0x1b/0x30 [kernel]

Quoted 4 lines

I think the problem here is that the client does not update the file attribute after ALLOCATE. The GETATTR in the ALLOCATE compound does not include the mode bits.

Oh, interesting! Have you tried adding the FATTR4_MODE to that GETATTR call on the client? Does it also fix this?

Quoted 16 lines

The READDIR's reply show the test file's mode has the SUID & SGID bit stripped (0666) but apparently these were not used o update the file attribute.

The test passes when server does not grant write delegation because:

OPEN
WRITE
CLOSE
SETATTR (06666)
OPEN (CLAIM_FH, NOCREATE)
ALLOCATE <<=== server clear SUID & SGID
GETATTR, CLOSE <<=== GETATTR has mode bit as 0666, client updates file attribute READDIR
READDIR

Quoted 5 lines

As expected, if the server recalls the write delegation when SETATTR with SUID/SGID set then the test passes. This is because it forces the client to send the 2nd OPEN with CLAIM_FH, NOCREATE and then the (GETATTR, CLOSE) which cause the client to update the file attribute.

What's your sense of the best way to fix this? The stripping of mode bits isn't covered by the NFSv4 spec, so this will ultimately come down to a judgment call.

Signature · 1 line
Jeff Layton <[email protected]>
#14 lore raw
Quoted 214 lines · Jeff Layton
Quoted 213 lines · On Tue, 2023-08-22 at 09:07 -0700, [email protected] wrote:
Quoted 167 lines · Jeff Layton
Quoted 166 lines · On Thu, 2023-08-17 at 15:59 -0700, [email protected] wrote:
Quoted 165 lines · On 8/17/23 3:23 PM, [email protected] wrote:
Quoted 164 lines · Jeff Layton
Quoted 148 lines · Jeff Layton
Quoted 147 lines · Chuck Lever III
Quoted 147 lines · Jeff Layton
Quoted 131 lines · Anna Schumaker
Quoted 52 lines · Jeff Layton
Quoted 51 lines · Chuck Lever III
Quoted 51 lines

On Aug 17, 2023, at 7:21 AM, Jeff Layton <[email protected]>
wrote:

I finally got my kdevops
(https://github.com/linux-kdevops/kdevops) test
rig working well enough to get some publishable results. To
run fstests,
kdevops will spin up a server and (in this case) 2 clients to run
xfstests' auto group. One client mounts with default options,
and the
other uses NFSv3.

I tested 3 kernels:

v6.4.0 (stock release)
6.5.0-rc6-g4853c74bd7ab (Linus' tree as of a couple of days ago)
6.5.0-rc6-next-20230816-gef66bf8aeb91 (linux-next as of
yesterday morning)

Here are the results summary of all 3:

KERNEL:    6.4.0
CPUS:      8

nfs_v3: 727 tests, 12 failures, 569 skipped, 14863 seconds
Failures: generic/053 generic/099 generic/105 generic/124
   generic/193 generic/258 generic/294 generic/318 generic/319
   generic/444 generic/528 generic/529
nfs_default: 727 tests, 18 failures, 452 skipped, 21899 seconds
Failures: generic/053 generic/099 generic/105 generic/186
   generic/187 generic/193 generic/294 generic/318 generic/319
   generic/357 generic/444 generic/486 generic/513 generic/528
   generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1021 skipped, 30 failures, 0 errors, 35096s

KERNEL:    6.5.0-rc6-g4853c74bd7ab
CPUS:      8

nfs_v3: 727 tests, 9 failures, 570 skipped, 14775 seconds
Failures: generic/053 generic/099 generic/105 generic/258
   generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 16 failures, 453 skipped, 22326 seconds
Failures: generic/053 generic/099 generic/105 generic/186
   generic/187 generic/294 generic/318 generic/319 generic/357
   generic/444 generic/486 generic/513 generic/529 generic/578
   generic/675 generic/688
Totals: 1454 tests, 1023 skipped, 25 failures, 0 errors, 35396s

KERNEL:    6.5.0-rc6-next-20230816-gef66bf8aeb91
CPUS:      8

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

nfs_v3: 727 tests, 9 failures, 570 skipped, 14657 seconds
Failures: generic/053 generic/099 generic/105 generic/258
   generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 18 failures, 453 skipped, 21757 seconds
Failures: generic/053 generic/099 generic/105 generic/186
   generic/187 generic/294 generic/318 generic/319 generic/357
   generic/444 generic/486 generic/513 generic/529 generic/578
   generic/675 generic/683 generic/684 generic/688
Totals: 1454 tests, 1023 skipped, 27 failures, 0 errors, 34870s

As long as we're sharing results ... here is what I'm seeing with a 6.5-rc6 client & server:

anna@gouda ~ % xfstestsdb xunit list --results --runid 1741 --color=none
+------+----------------------+---------+----------+------+------+------+-------+

Quoted 1 line

run | device               | xunit   | hostname | pass | fail |

skip |  time |
+------+----------------------+---------+----------+------+------+------+-------+

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-3   | client   |  125 |    4 |

464 | 447 s |

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-4.0 | client   |  117 |   11 |

465 | 478 s |

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-4.1 | client   |  119 |   12 |

462 | 404 s |

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-4.2 | client   |  212 |   18 |

363 | 564 s |
+------+----------------------+---------+----------+------+------+------+-------+

anna@gouda ~ % xfstestsdb show --failure 1741 --color=none

+-------------+---------+---------+---------+---------+
>      testcase | tcp-3   | tcp-4.0 | tcp-4.1 | tcp-4.2 |
+-------------+---------+---------+---------+---------+
> generic/053 | passed  | failure | failure | failure |
> generic/099 | passed  | failure | failure | failure |
> generic/105 | passed  | failure | failure | failure |
> generic/140 | skipped | skipped | skipped | failure |
> generic/188 | skipped | skipped | skipped | failure |
> generic/258 | failure | passed  | passed  | failure |
> generic/294 | failure | failure | failure | failure |
> generic/318 | passed  | failure | failure | failure |
> generic/319 | passed  | failure | failure | failure |
> generic/357 | skipped | skipped | skipped | failure |
> generic/444 | failure | failure | failure | failure |
> generic/465 | passed  | failure | failure | failure |
> generic/513 | skipped | skipped | skipped | failure |
> generic/529 | passed  | failure | failure | failure |
> generic/604 | passed  | passed  | failure | passed  |
> generic/675 | skipped | skipped | skipped | failure |
> generic/688 | skipped | skipped | skipped | failure |
> generic/697 | passed  | failure | failure | failure |
>       nfs/002 | failure | failure | failure | failure |
+-------------+---------+---------+---------+---------+
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines

With NFSv4.2, v6.4.0 has 2 extra failures that the current
mainline
kernel doesn't:

   generic/193 (some sort of setattr problem)
   generic/528 (known problem with btime handling in client
that has been fixed)

While I haven't investigated, I'm assuming the 193 bug is also
something
that has been fixed in recent kernels. There are also 3 other
NFSv3
tests that started passing since v6.4.0. I haven't looked into
those.

With the linux-next kernel there are 2 new regressions:

   generic/683
   generic/684

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Both of these look like problems with setuid/setgid stripping,
and still
need to be investigated. I have more verbose result info on
the test
failures if anyone is interested.

Interesting that I'm not seeing the 683 & 684 failures. What type of filesystem is your server exporting?

btrfs

You are testing linux-next? I need to go back and confirm these results
too.

Quoted 9 lines
Quoted 4 lines

IMO linux-next is quite important : we keep hitting bugs that appear only after integration -- block and network changes in other trees especially can impact the NFS drivers.

Indeed, I suspect this is probably something from the vfs tree (though we definitely need to confirm that). Today I'm testing:

     6.5.0-rc6-next-20230817-g47762f086974

Nope, I was wrong. I ran a bisect and it landed here. I confirmed it by turning off leases on the nfs server and the test started passing. I probably won't have the cycles to chase this down further.

The capture looks something like this:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

OPEN (get a write delegation
WRITE
CLOSE
SETATTR (mode 06666)

...then presumably a task on the client opens the file again, but the setuid bits don't get stripped.

OPEN (get a write delegation
WRITE
CLOSE
SETATTR (mode 06666)

The client continues with:

(ALLOCATE,GETATTR)  <<===  this is when the server stripped the SUID and SGID bit
READDIR             ====>  file mode shows 0666  (SUID & SGID were stripped)
READDIR             ====>  file mode shows 0666  (SUID & SGID were stripped)

DELERETURN

Here is stack trace of ALLOCATE when the SUID & SGID were stripped:

**** start of notify_change, notice the i_mode bits, SUID & SGID were set:
[notify_change]: d_iname[a] ia_valid[0x1a00] ia_mode[0x0] i_mode[0x8db6] [nfsd:2409:Mon Aug 21 23:05:31 2023]

                          KILL[0] KILL_SUID[1] KILL_SGID[1]

**** end of notify_change, notice the i_mode bits, SUID & SGID were stripped:
[notify_change]: RET[0] d_iname[a] ia_valid[0x1a01] ia_mode[0x81b6] i_mode[0x81b6] [nfsd:2409:Mon Aug 21 23:05:31 2023]

**** stack trace of notify_change comes from ALLOCATE:

Returning from:  0xffffffffb726e764 : notify_change+0x4/0x500 [kernel]
Returning to  :  0xffffffffb726bf99 : __file_remove_privs+0x119/0x170 [kernel]

0xffffffffb726cfad : file_modified_flags+0x4d/0x110 [kernel] 0xffffffffc0a2330b : xfs_file_fallocate+0xfb/0x490 [xfs] 0xffffffffb723e7d8 : vfs_fallocate+0x158/0x380 [kernel] 0xffffffffc0ddc30a : nfsd4_vfs_fallocate+0x4a/0x70 [nfsd] 0xffffffffc0def7f2 : nfsd4_allocate+0x72/0xc0 [nfsd] 0xffffffffc0df2663 : nfsd4_proc_compound+0x3d3/0x730 [nfsd] 0xffffffffc0dd633b : nfsd_dispatch+0xab/0x1d0 [nfsd] 0xffffffffc0bda476 : svc_process_common+0x306/0x6e0 [sunrpc] 0xffffffffc0bdb081 : svc_process+0x131/0x180 [sunrpc] 0xffffffffc0dd4864 : nfsd+0x84/0xd0 [nfsd] 0xffffffffb6f0bfd6 : kthread+0xe6/0x120 [kernel] 0xffffffffb6e587d4 : ret_from_fork+0x34/0x50 [kernel] 0xffffffffb6e03a3b : ret_from_fork_asm+0x1b/0x30 [kernel]

Quoted 6 lines
Quoted 4 lines

I think the problem here is that the client does not update the file attribute after ALLOCATE. The GETATTR in the ALLOCATE compound does not include the mode bits.

Oh, interesting! Have you tried adding the FATTR4_MODE to that GETATTR call on the client? Does it also fix this?

Yes, this is what I'm going to try next.

Quoted 17 lines
Quoted 17 lines

The READDIR's reply show the test file's mode has the SUID & SGID bit stripped (0666) but apparently these were not used o update the file attribute.

The test passes when server does not grant write delegation because:

OPEN
WRITE
CLOSE
SETATTR (06666)
OPEN (CLAIM_FH, NOCREATE)
ALLOCATE <<=== server clear SUID & SGID
GETATTR, CLOSE <<=== GETATTR has mode bit as 0666, client updates file attribute READDIR
READDIR

Quoted 8 lines
Quoted 5 lines

As expected, if the server recalls the write delegation when SETATTR with SUID/SGID set then the test passes. This is because it forces the client to send the 2nd OPEN with CLAIM_FH, NOCREATE and then the (GETATTR, CLOSE) which cause the client to update the file attribute.

What's your sense of the best way to fix this? The stripping of mode bits isn't covered by the NFSv4 spec, so this will ultimately come down to a judgment call.

Yes, I did not find anything regarding stripping of SUID/SGID in the NFS4.2 specs. It's done by the 'fs' layer and it has been there since 4/2005 in the big merge to Linux-2.6.12-rc2 done by Linus. So I think we should leave it there.

The stripping makes some sense to me since if the file is being expanded (to be written to) then it should not an executable therefor its SUID/SGID should be stripped.

-Dai

author maintainer
#15 lore raw
Quoted 215 lines · On Tue, 2023-08-22 at 12:51 -0700, [email protected] wrote:
Quoted 214 lines · Jeff Layton
Quoted 213 lines · On Tue, 2023-08-22 at 09:07 -0700, [email protected] wrote:
Quoted 167 lines · Jeff Layton
Quoted 166 lines · On Thu, 2023-08-17 at 15:59 -0700, [email protected] wrote:
Quoted 165 lines · On 8/17/23 3:23 PM, [email protected] wrote:
Quoted 164 lines · Jeff Layton
Quoted 148 lines · Jeff Layton
Quoted 147 lines · Chuck Lever III
Quoted 147 lines · Jeff Layton
Quoted 131 lines · Anna Schumaker
Quoted 52 lines · Jeff Layton
Quoted 51 lines · Chuck Lever III

On Aug 17, 2023, at 7:21 AM, Jeff Layton <[email protected]>
wrote:

I finally got my kdevops
(https://github.com/linux-kdevops/kdevops) test
rig working well enough to get some publishable results. To
run fstests,
kdevops will spin up a server and (in this case) 2 clients to run
xfstests' auto group. One client mounts with default options,
and the
other uses NFSv3.

I tested 3 kernels:

v6.4.0 (stock release)
6.5.0-rc6-g4853c74bd7ab (Linus' tree as of a couple of days ago)
6.5.0-rc6-next-20230816-gef66bf8aeb91 (linux-next as of
yesterday morning)

Here are the results summary of all 3:

KERNEL:    6.4.0
CPUS:      8

nfs_v3: 727 tests, 12 failures, 569 skipped, 14863 seconds
Failures: generic/053 generic/099 generic/105 generic/124
   generic/193 generic/258 generic/294 generic/318 generic/319
   generic/444 generic/528 generic/529
nfs_default: 727 tests, 18 failures, 452 skipped, 21899 seconds
Failures: generic/053 generic/099 generic/105 generic/186
   generic/187 generic/193 generic/294 generic/318 generic/319
   generic/357 generic/444 generic/486 generic/513 generic/528
   generic/529 generic/578 generic/675 generic/688
Totals: 1454 tests, 1021 skipped, 30 failures, 0 errors, 35096s

KERNEL:    6.5.0-rc6-g4853c74bd7ab
CPUS:      8

nfs_v3: 727 tests, 9 failures, 570 skipped, 14775 seconds
Failures: generic/053 generic/099 generic/105 generic/258
   generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 16 failures, 453 skipped, 22326 seconds
Failures: generic/053 generic/099 generic/105 generic/186
   generic/187 generic/294 generic/318 generic/319 generic/357
   generic/444 generic/486 generic/513 generic/529 generic/578
   generic/675 generic/688
Totals: 1454 tests, 1023 skipped, 25 failures, 0 errors, 35396s

KERNEL:    6.5.0-rc6-next-20230816-gef66bf8aeb91
CPUS:      8

Quoted 9 lines
Quoted 9 lines

nfs_v3: 727 tests, 9 failures, 570 skipped, 14657 seconds
Failures: generic/053 generic/099 generic/105 generic/258
   generic/294 generic/318 generic/319 generic/444 generic/529
nfs_default: 727 tests, 18 failures, 453 skipped, 21757 seconds
Failures: generic/053 generic/099 generic/105 generic/186
   generic/187 generic/294 generic/318 generic/319 generic/357
   generic/444 generic/486 generic/513 generic/529 generic/578
   generic/675 generic/683 generic/684 generic/688
Totals: 1454 tests, 1023 skipped, 27 failures, 0 errors, 34870s

As long as we're sharing results ... here is what I'm seeing with a 6.5-rc6 client & server:

anna@gouda ~ % xfstestsdb xunit list --results --runid 1741 --color=none
+------+----------------------+---------+----------+------+------+------+-------+

Quoted 1 line

run | device               | xunit   | hostname | pass | fail |

skip |  time |
+------+----------------------+---------+----------+------+------+------+-------+

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-3   | client   |  125 |    4 |

464 | 447 s |

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-4.0 | client   |  117 |   11 |

465 | 478 s |

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-4.1 | client   |  119 |   12 |

462 | 404 s |

Quoted 1 line

1741 | server:/srv/xfs/test | tcp-4.2 | client   |  212 |   18 |

363 | 564 s |
+------+----------------------+---------+----------+------+------+------+-------+

anna@gouda ~ % xfstestsdb show --failure 1741 --color=none

+-------------+---------+---------+---------+---------+
>      testcase | tcp-3   | tcp-4.0 | tcp-4.1 | tcp-4.2 |
+-------------+---------+---------+---------+---------+
> generic/053 | passed  | failure | failure | failure |
> generic/099 | passed  | failure | failure | failure |
> generic/105 | passed  | failure | failure | failure |
> generic/140 | skipped | skipped | skipped | failure |
> generic/188 | skipped | skipped | skipped | failure |
> generic/258 | failure | passed  | passed  | failure |
> generic/294 | failure | failure | failure | failure |
> generic/318 | passed  | failure | failure | failure |
> generic/319 | passed  | failure | failure | failure |
> generic/357 | skipped | skipped | skipped | failure |
> generic/444 | failure | failure | failure | failure |
> generic/465 | passed  | failure | failure | failure |
> generic/513 | skipped | skipped | skipped | failure |
> generic/529 | passed  | failure | failure | failure |
> generic/604 | passed  | passed  | failure | passed  |
> generic/675 | skipped | skipped | skipped | failure |
> generic/688 | skipped | skipped | skipped | failure |
> generic/697 | passed  | failure | failure | failure |
>       nfs/002 | failure | failure | failure | failure |
+-------------+---------+---------+---------+---------+
Quoted 20 lines
Quoted 20 lines

With NFSv4.2, v6.4.0 has 2 extra failures that the current
mainline
kernel doesn't:

   generic/193 (some sort of setattr problem)
   generic/528 (known problem with btime handling in client
that has been fixed)

While I haven't investigated, I'm assuming the 193 bug is also
something
that has been fixed in recent kernels. There are also 3 other
NFSv3
tests that started passing since v6.4.0. I haven't looked into
those.

With the linux-next kernel there are 2 new regressions:

   generic/683
   generic/684

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

Both of these look like problems with setuid/setgid stripping,
and still
need to be investigated. I have more verbose result info on
the test
failures if anyone is interested.

Interesting that I'm not seeing the 683 & 684 failures. What type of filesystem is your server exporting?

btrfs

You are testing linux-next? I need to go back and confirm these results
too.

Quoted 9 lines
Quoted 4 lines

IMO linux-next is quite important : we keep hitting bugs that appear only after integration -- block and network changes in other trees especially can impact the NFS drivers.

Indeed, I suspect this is probably something from the vfs tree (though we definitely need to confirm that). Today I'm testing:

     6.5.0-rc6-next-20230817-g47762f086974

Nope, I was wrong. I ran a bisect and it landed here. I confirmed it by turning off leases on the nfs server and the test started passing. I probably won't have the cycles to chase this down further.

The capture looks something like this:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

OPEN (get a write delegation
WRITE
CLOSE
SETATTR (mode 06666)

...then presumably a task on the client opens the file again, but the setuid bits don't get stripped.

OPEN (get a write delegation
WRITE
CLOSE
SETATTR (mode 06666)

The client continues with:

(ALLOCATE,GETATTR)  <<===  this is when the server stripped the SUID and SGID bit
READDIR             ====>  file mode shows 0666  (SUID & SGID were stripped)
READDIR             ====>  file mode shows 0666  (SUID & SGID were stripped)

DELERETURN

Here is stack trace of ALLOCATE when the SUID & SGID were stripped:

**** start of notify_change, notice the i_mode bits, SUID & SGID were set:
[notify_change]: d_iname[a] ia_valid[0x1a00] ia_mode[0x0] i_mode[0x8db6] [nfsd:2409:Mon Aug 21 23:05:31 2023]

                          KILL[0] KILL_SUID[1] KILL_SGID[1]

**** end of notify_change, notice the i_mode bits, SUID & SGID were stripped:
[notify_change]: RET[0] d_iname[a] ia_valid[0x1a01] ia_mode[0x81b6] i_mode[0x81b6] [nfsd:2409:Mon Aug 21 23:05:31 2023]

**** stack trace of notify_change comes from ALLOCATE:

Returning from:  0xffffffffb726e764 : notify_change+0x4/0x500 [kernel]
Returning to  :  0xffffffffb726bf99 : __file_remove_privs+0x119/0x170 [kernel]

0xffffffffb726cfad : file_modified_flags+0x4d/0x110 [kernel] 0xffffffffc0a2330b : xfs_file_fallocate+0xfb/0x490 [xfs] 0xffffffffb723e7d8 : vfs_fallocate+0x158/0x380 [kernel] 0xffffffffc0ddc30a : nfsd4_vfs_fallocate+0x4a/0x70 [nfsd] 0xffffffffc0def7f2 : nfsd4_allocate+0x72/0xc0 [nfsd] 0xffffffffc0df2663 : nfsd4_proc_compound+0x3d3/0x730 [nfsd] 0xffffffffc0dd633b : nfsd_dispatch+0xab/0x1d0 [nfsd] 0xffffffffc0bda476 : svc_process_common+0x306/0x6e0 [sunrpc] 0xffffffffc0bdb081 : svc_process+0x131/0x180 [sunrpc] 0xffffffffc0dd4864 : nfsd+0x84/0xd0 [nfsd] 0xffffffffb6f0bfd6 : kthread+0xe6/0x120 [kernel] 0xffffffffb6e587d4 : ret_from_fork+0x34/0x50 [kernel] 0xffffffffb6e03a3b : ret_from_fork_asm+0x1b/0x30 [kernel]

Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

I think the problem here is that the client does not update the file attribute after ALLOCATE. The GETATTR in the ALLOCATE compound does not include the mode bits.

Oh, interesting! Have you tried adding the FATTR4_MODE to that GETATTR call on the client? Does it also fix this?

Yes, this is what I'm going to try next.

Great. Keep us posted.

Quoted 26 lines
Quoted 17 lines
Quoted 17 lines

The READDIR's reply show the test file's mode has the SUID & SGID bit stripped (0666) but apparently these were not used o update the file attribute.

The test passes when server does not grant write delegation because:

OPEN
WRITE
CLOSE
SETATTR (06666)
OPEN (CLAIM_FH, NOCREATE)
ALLOCATE <<=== server clear SUID & SGID
GETATTR, CLOSE <<=== GETATTR has mode bit as 0666, client updates file attribute READDIR
READDIR

Quoted 8 lines
Quoted 5 lines

As expected, if the server recalls the write delegation when SETATTR with SUID/SGID set then the test passes. This is because it forces the client to send the 2nd OPEN with CLAIM_FH, NOCREATE and then the (GETATTR, CLOSE) which cause the client to update the file attribute.

What's your sense of the best way to fix this? The stripping of mode bits isn't covered by the NFSv4 spec, so this will ultimately come down to a judgment call.

Quoted 9 lines

Yes, I did not find anything regarding stripping of SUID/SGID in the NFS4.2 specs. It's done by the 'fs' layer and it has been there since 4/2005 in the big merge to Linux-2.6.12-rc2 done by Linus. So I think we should leave it there.

The stripping makes some sense to me since if the file is being expanded (to be written to) then it should not an executable therefor its SUID/SGID should be stripped.

Right. The point is that POSIX requires setuid clearing, but the NFSv4 spec doesn't say anything about it. Ultimately, it's the server's responsibility to actually clear the bits.

Having the client also fetch the mode does sound like the right thing to do here. It should be cheap for most servers to provide anyway, given that they will have the inode in-core.

Signature · 1 line
Jeff Layton <[email protected]>