Skip to contentSkip to conversation

unionmount-testsuite: master branch updated to 9c60a9c

ANNOUNCE Amir Goldstein 2 participants 5 messages May 29, 2020 – Jun 19, 2020

lore
Share & tools
lore
#0 lore raw

Hi All,

The master branch on the unionmount-testsuite tree [1] has been updated.

Changes in this update:
- Support user configurable underlying filesystem

So far, unionmount-testsuite used hardcoded paths for layers and mount point. Using underlying filesystem other than tmpfs was possible, but not very easy to setup.

This update brings the ability for user to configure custom paths with a custom filesystem for the underlying layers. This is intended to be used for integration with xfstests [2].

Here is an excerpt from the README:
---
The following environment variables are supported:

UNIONMOUNT_BASEDIR - parent dir of all samefs layers (default: /base) UNIONMOUNT_LOWERDIR - lower layer path for non samefs (default: /lower) UNIONMOUNT_MNTPOINT - mount point for executing tests (default: /mnt)

When user provides UNIONMOUNT_LOWERDIR:

1) Path should be an existing directory whose content will be deleted.
2) Path is assumed to be on a different filesystem than base dir, so --samefs setup is not supported.

When user provides UNIONMOUNT_BASEDIR:

1) Path should be an existing directory whose content will be deleted.
2) Upper layer and middle layers will be created under base dir.
3) If UNIONMOUNT_MNTPOINT is not provided, the overlay mount point will be created under base dir.
4) If UNIONMOUNT_LOWERDIR is not provided, the lower layer dir will be created under base dir.
5) If UNIONMOUNT_LOWERDIR is not provided, the test setup defaults to --samefs (i.e. lower and upper layers are on the same base fs). However, if --maxfs=<M> is specified, a tmpfs instance will be mounted on the lower layer dir that was created under base dir.
---

Many thanks to Vivek for review and testing.

Thanks,
Amir.

[1] https://github.com/amir73il/unionmount-testsuite
[2] https://github.com/amir73il/xfstests/commits/unionmount

The head of the master branch is commit:

9c60a9c Configure custom layers via environment variables

New commits:

Amir Goldstein (3):
Add command run --clean-up to cleanup old test mounts Stop using bind mounts for --samefs Configure custom layers via environment variables

 README           |  26 ++++++++++
 mount_union.py   |  13 +----
 run              |  32 +++++-------
 set_up.py        | 127 +++++++++++++++++++++++++----------------------
 settings.py      |  85 +++++++++++++++++++++++++------
 unmount_union.py |  15 +++---
 6 files changed, 187 insertions(+), 111 deletions(-)
Signature · 1 line
2.17.1
#1 lore raw
Quoted 15 lines · Amir Goldstein

Hi All,

The master branch on the unionmount-testsuite tree [1] has been updated.

Changes in this update:
- Support user configurable underlying filesystem

So far, unionmount-testsuite used hardcoded paths for layers and mount point. Using underlying filesystem other than tmpfs was possible, but not very easy to setup.

This update brings the ability for user to configure custom paths with a custom filesystem for the underlying layers. This is intended to be used for integration with xfstests [2].

Quoted 5 lines

Here is an excerpt from the README:
---
The following environment variables are supported:

UNIONMOUNT_BASEDIR - parent dir of all samefs layers (default: /base)

Hi Amir,

I am running these tests with.

UNIONMOUNT_BASEDIR="/mnt/foo/"

- ./run --ov runs fine. But when I try to run it again it complains that.

rm: cannot remove '/mnt/overlayfs//m': Device or resource busy

So I have to first unmount /mnt/overlayfs/m/ and then run tests again.

I think it will be nice if it can clear the environment by itself.

- I am running one the recent kernel (5.7.0+) and following errors out.

# ./run --ov --verify
Environment variables:
UNIONMOUNT_BASEDIR=/mnt/overlayfs/

***
*** ./run --ov --samefs --ts=0 open-plain
***
TEST open-plain.py:10: Open O_RDONLY
/mnt/overlayfs/m/a/foo100: not on union mount

Will spend more time to figure out what happened.

- I am planning to use these environment variables and run overlay over virtiofs tests. Can I do the same thing with xfstests overlay tests. In README.overlay I see that I need to specify two separate devices. Can I specify to directories (and not devices) to be used as TEST and SCRATCH and run overlay test.

Thanks
Vivek

Quoted 49 lines

1) Path should be an existing directory whose content will be deleted.
2) Path is assumed to be on a different filesystem than base dir, so --samefs setup is not supported.

When user provides UNIONMOUNT_BASEDIR:

1) Path should be an existing directory whose content will be deleted.
2) Upper layer and middle layers will be created under base dir.
3) If UNIONMOUNT_MNTPOINT is not provided, the overlay mount point will be created under base dir.
4) If UNIONMOUNT_LOWERDIR is not provided, the lower layer dir will be created under base dir.
5) If UNIONMOUNT_LOWERDIR is not provided, the test setup defaults to --samefs (i.e. lower and upper layers are on the same base fs). However, if --maxfs=<M> is specified, a tmpfs instance will be mounted on the lower layer dir that was created under base dir.
---

Many thanks to Vivek for review and testing.

Thanks,
Amir.

[1] https://github.com/amir73il/unionmount-testsuite
[2] https://github.com/amir73il/xfstests/commits/unionmount

The head of the master branch is commit:

9c60a9c Configure custom layers via environment variables

New commits:

Amir Goldstein (3):
Add command run --clean-up to cleanup old test mounts Stop using bind mounts for --samefs Configure custom layers via environment variables

 README           |  26 ++++++++++
 mount_union.py   |  13 +----
 run              |  32 +++++-------
 set_up.py        | 127 +++++++++++++++++++++++++----------------------
 settings.py      |  85 +++++++++++++++++++++++++------
 unmount_union.py |  15 +++---
 6 files changed, 187 insertions(+), 111 deletions(-)
Signature · 1 line
2.17.1
#2 lore raw

On Thu, Jun 18, 2020 at 05:38:31PM -0400, Vivek Goyal wrote:

[..]

Quoted 5 lines
# ./run --ov --verify
Environment variables:
UNIONMOUNT_BASEDIR=/mnt/overlayfs/
Quoted 7 lines

***
*** ./run --ov --samefs --ts=0 open-plain
***
TEST open-plain.py:10: Open O_RDONLY
/mnt/overlayfs/m/a/foo100: not on union mount

Will spend more time to figure out what happened.

Ok, this is an issue only if I specify UNIONMOUNT_BASEDIR. If I don't, then it passes.

Thanks
Vivek

#3 lore raw
In reply to Vivek Goyal
Quoted 21 lines · Vivek Goyal
Quoted 19 lines · Amir Goldstein

Hi All,

The master branch on the unionmount-testsuite tree [1] has been updated.

Changes in this update:
- Support user configurable underlying filesystem

So far, unionmount-testsuite used hardcoded paths for layers and mount point. Using underlying filesystem other than tmpfs was possible, but not very easy to setup.

This update brings the ability for user to configure custom paths with a custom filesystem for the underlying layers. This is intended to be used for integration with xfstests [2].

Here is an excerpt from the README:
---
The following environment variables are supported:

Quoted 7 lines
Quoted 1 line

UNIONMOUNT_BASEDIR - parent dir of all samefs layers (default: /base)

Hi Amir,

I am running these tests with.

UNIONMOUNT_BASEDIR="/mnt/foo/"

I suspect there is a bug with trailing / in UNIONMOUNT_BASEDIR I did not test with trailing /
Can you try without it?

Quoted 11 lines

- ./run --ov runs fine. But when I try to run it again it complains that.

rm: cannot remove '/mnt/overlayfs//m': Device or resource busy

So I have to first unmount /mnt/overlayfs/m/ and then run tests again.

I think it will be nice if it can clear the environment by itself.

It should.
All the tests I tried cleaned up previous test env automatically. The only case where explicit cleanup is needed is before changing context and modifying envvars.

Quoted 13 lines

- I am running one the recent kernel (5.7.0+) and following errors out.

# ./run --ov --verify
Environment variables:
UNIONMOUNT_BASEDIR=/mnt/overlayfs/

***
*** ./run --ov --samefs --ts=0 open-plain
***
TEST open-plain.py:10: Open O_RDONLY
/mnt/overlayfs/m/a/foo100: not on union mount

Quoted 7 lines

Will spend more time to figure out what happened.

- I am planning to use these environment variables and run overlay over virtiofs tests. Can I do the same thing with xfstests overlay tests. In README.overlay I see that I need to specify two separate devices. Can I specify to directories (and not devices) to be used as TEST and SCRATCH and run overlay test.

As far as I can tell, SCRATCH_DEV and TEST_DEV are directories in all those non-block filesystems including virtiofs:
_scratch_mkfs()
...
case $FSTYP in
nfs*|cifs|ceph|overlay|glusterfs|pvfs2|9p|virtiofs)
# unable to re-create this fstyp, just remove all files in # $SCRATCH_MNT to avoid EEXIST caused by the leftover files # created in previous runs _scratch_cleanup_files

So I think you just need to set SCRATCH_DEV and TEST_DEV to two different directories and you are good to go for running check or check -overlay.

Thanks,
Amir.

#4 lore raw

On Fri, Jun 19, 2020 at 06:29:08AM +0300, Amir Goldstein wrote:

[..]

Quoted 9 lines
Quoted 5 lines

Hi Amir,

I am running these tests with.

UNIONMOUNT_BASEDIR="/mnt/foo/"

I suspect there is a bug with trailing / in UNIONMOUNT_BASEDIR I did not test with trailing /
Can you try without it?

Hi Amir,

This indeed is trailing / issue. I removed it and things are working fine.

Quoted 10 lines
Quoted 10 lines

- ./run --ov runs fine. But when I try to run it again it complains that.

rm: cannot remove '/mnt/overlayfs//m': Device or resource busy

So I have to first unmount /mnt/overlayfs/m/ and then run tests again.

Quoted 7 lines
Quoted 2 lines

I think it will be nice if it can clear the environment by itself.

It should.
All the tests I tried cleaned up previous test env automatically. The only case where explicit cleanup is needed is before changing context and modifying envvars.

After removing trailing /, cleanup was fine too.

Quoted 33 lines
Quoted 14 lines

- I am running one the recent kernel (5.7.0+) and following errors out.

# ./run --ov --verify
Environment variables:
UNIONMOUNT_BASEDIR=/mnt/overlayfs/

***
*** ./run --ov --samefs --ts=0 open-plain
***
TEST open-plain.py:10: Open O_RDONLY
/mnt/overlayfs/m/a/foo100: not on union mount

Quoted 7 lines

Will spend more time to figure out what happened.

- I am planning to use these environment variables and run overlay over virtiofs tests. Can I do the same thing with xfstests overlay tests. In README.overlay I see that I need to specify two separate devices. Can I specify to directories (and not devices) to be used as TEST and SCRATCH and run overlay test.

As far as I can tell, SCRATCH_DEV and TEST_DEV are directories in all those non-block filesystems including virtiofs:
_scratch_mkfs()
...
case $FSTYP in
nfs*|cifs|ceph|overlay|glusterfs|pvfs2|9p|virtiofs)
# unable to re-create this fstyp, just remove all files in # $SCRATCH_MNT to avoid EEXIST caused by the leftover files # created in previous runs _scratch_cleanup_files

Quoted 3 lines

So I think you just need to set SCRATCH_DEV and TEST_DEV to two different directories and you are good to go for running check or check -overlay.

It seems to work. I am seeing bunch of failures but that probably are issues with virtiofs and overlayfs interaction. Will look into these one by one.

Thanks
Vivek