Skip to contentSkip to conversation

Enlarging w/ xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Inappropriate ioctl for device

RFC PATCH In review Nick Bowler 5 participants 26 messages Dec 10, 2018 – Jan 15, 2019 XFS filesystem

lore
Share & tools
lore
#0 lore raw

Hello,

I'm a bit new to using XFS and I ran into some errors trying to enlarge a filesystem. This setup uses dmcrypt on top of md raid and I just reshaped the array to add additional storage. The underlying block device reflects the new size, but the filesystem hasn't been enlarged yet:

  # blockdev --report /dev/mapper/data             
  RO    RA   SSZ   BSZ   StartSec            Size   Device
  rw  4096   512  4096          0  20001386921984   /dev/mapper/data
  # findmnt /dev/mapper/data
  TARGET    SOURCE           FSTYPE OPTIONS
  /mnt/data /dev/mapper/data xfs    rw,relatime,attr2,inode64,sunit=1024,swidth=2048,noquota
  # df -h /mnt/data
  Filesystem        Size  Used Avail Use% Mounted on
  /dev/mapper/data  9.1T  8.5T  649G  94% /mnt/data

So I read the manpage and it seems all I should need to do is run xfs_growfs on the mounted filesystem but...

  # xfs_growfs /mnt/data
  meta-data=/dev/mapper/data       isize=512    agcount=32, agsize=76299136 blks
           =                       sectsz=4096  attr=2, projid32bit=1
           =                       crc=1        finobt=1, sparse=1, rmapbt=0
           =                       reflink=0
  data     =                       bsize=4096   blocks=2441572352, imaxpct=5
           =                       sunit=128    swidth=256 blks
  naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
  log      =internal log           bsize=4096   blocks=521728, version=2
           =                       sectsz=4096  sunit=1 blks, lazy-count=1
  realtime =none                   extsz=4096   blocks=0, rtextents=0
  xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Inappropriate ioctl for device
  xfs_growfs: XFS_IOC_FSGEOMETRY xfsctl failed: Inappropriate ioctl for device

... and the filesystem is not enlarged. Looking at strace output, the failing ioctls seem to be:

openat(AT_FDCWD, "/mnt/data", O_RDONLY) = 3
[...]
ioctl(3, _IOC(_IOC_WRITE, 0x58, 0x6e, 0x10), 0xffcc9a80) = -1 ENOTTY (Inappropriate ioctl for device) [...]
ioctl(3, _IOC(_IOC_READ, 0x58, 0x64, 0x70), 0xffcc9ba0) = -1 ENOTTY (Inappropriate ioctl for device)

Kernel version is 4.14.82 with xfsprogs 4.17.0, although I tried also with xfsprogs 4.19.0 and received the same errors.

Am I missing something obvious here? What further steps should I take to help solve this?

Thanks,
Nick

#1 lore raw
Quoted 46 lines · Nick Bowler

Hello,

I'm a bit new to using XFS and I ran into some errors trying to enlarge a filesystem. This setup uses dmcrypt on top of md raid and I just reshaped the array to add additional storage. The underlying block device reflects the new size, but the filesystem hasn't been enlarged yet:

  # blockdev --report /dev/mapper/data             
  RO    RA   SSZ   BSZ   StartSec            Size   Device
  rw  4096   512  4096          0  20001386921984   /dev/mapper/data
  # findmnt /dev/mapper/data
  TARGET    SOURCE           FSTYPE OPTIONS
  /mnt/data /dev/mapper/data xfs    rw,relatime,attr2,inode64,sunit=1024,swidth=2048,noquota
  # df -h /mnt/data
  Filesystem        Size  Used Avail Use% Mounted on
  /dev/mapper/data  9.1T  8.5T  649G  94% /mnt/data

So I read the manpage and it seems all I should need to do is run xfs_growfs on the mounted filesystem but...

  # xfs_growfs /mnt/data
  meta-data=/dev/mapper/data       isize=512    agcount=32, agsize=76299136 blks
           =                       sectsz=4096  attr=2, projid32bit=1
           =                       crc=1        finobt=1, sparse=1, rmapbt=0
           =                       reflink=0
  data     =                       bsize=4096   blocks=2441572352, imaxpct=5
           =                       sunit=128    swidth=256 blks
  naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
  log      =internal log           bsize=4096   blocks=521728, version=2
           =                       sectsz=4096  sunit=1 blks, lazy-count=1
  realtime =none                   extsz=4096   blocks=0, rtextents=0
  xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Inappropriate ioctl for device
  xfs_growfs: XFS_IOC_FSGEOMETRY xfsctl failed: Inappropriate ioctl for device

... and the filesystem is not enlarged. Looking at strace output, the failing ioctls seem to be:

openat(AT_FDCWD, "/mnt/data", O_RDONLY) = 3
[...]
ioctl(3, _IOC(_IOC_WRITE, 0x58, 0x6e, 0x10), 0xffcc9a80) = -1 ENOTTY (Inappropriate ioctl for device) [...]
ioctl(3, _IOC(_IOC_READ, 0x58, 0x64, 0x70), 0xffcc9ba0) = -1 ENOTTY (Inappropriate ioctl for device)

Quoted 6 lines

Kernel version is 4.14.82 with xfsprogs 4.17.0, although I tried also with xfsprogs 4.19.0 and received the same errors.

Am I missing something obvious here? What further steps should I take to help solve this?

The only thing that comes to mind while poking through the code is perhaps xfsprogs is sending the traditional XFS_IOC_GROWFSDATA command into the compat_ioctl() path somehow or another (assuming BROKEN_X86_ALIGNMENT is set).

What arch is your kernel/xfsprogs? What does 'cat /sys/kernel/debug/trace/trace' show if you run 'trace-cmd start -e xfs:xfs_file*ioctl*' and then attempt the growfs?

Brian

Quoted 2 lines

Thanks,
Nick

#2 lore raw

Hi Brian,

Quoted 6 lines · Brian Foster

The only thing that comes to mind while poking through the code is perhaps xfsprogs is sending the traditional XFS_IOC_GROWFSDATA command into the compat_ioctl() path somehow or another (assuming BROKEN_X86_ALIGNMENT is set).

What arch is your kernel/xfsprogs?

This system is running an amd64 kernel with x32 userspace (including xfsprogs).

Quoted 2 lines

What does 'cat /sys/kernel/debug/trace/trace' show if you run 'trace-cmd start -e xfs:xfs_file*ioctl*' and then attempt the growfs?

Looks like I don't have the required tracing enabled in my kernel configuration, but I can build a new one if needed. Is CONFIG_FTRACE sufficient for this?

Thanks,
Nick

#3 lore raw
Quoted 8 lines · Nick Bowler

Hi Brian,

Quoted 5 lines · Brian Foster

The only thing that comes to mind while poking through the code is perhaps xfsprogs is sending the traditional XFS_IOC_GROWFSDATA command into the compat_ioctl() path somehow or another (assuming BROKEN_X86_ALIGNMENT is set).

Quoted 5 lines
Quoted 1 line

What arch is your kernel/xfsprogs?

This system is running an amd64 kernel with x32 userspace (including xfsprogs).

Ok, so I think that means BROKEN_X86_ALIGNMENT should be set since XFS defines it as:

#if defined(CONFIG_IA64) || defined(CONFIG_X86_64) #define BROKEN_X86_ALIGNMENT
...

Quoted 7 lines
Quoted 2 lines

What does 'cat /sys/kernel/debug/trace/trace' show if you run 'trace-cmd start -e xfs:xfs_file*ioctl*' and then attempt the growfs?

Looks like I don't have the required tracing enabled in my kernel configuration, but I can build a new one if needed. Is CONFIG_FTRACE sufficient for this?

Not sure. I think you need to have CONFIG_TRACING enabled, which may require FTRACE and/or some other options. Hmm, perhaps you'd be covered if you make sure you have CONFIG_DYNAMIC_FTRACE enabled.

Quoted 1 line

From your strace output:

ioctl(3, _IOC(_IOC_WRITE, 0x58, 0x6e, 0x10), 0xffcc9a80) = -1 ENOTTY (Inappropriate ioctl for device)

0x6e corresponds to the GROWFSDATA[_32] cmd and I think 0x10 is the size, which is 16 bytes as opposed to the 12 bytes expected for GROWFSDATA_32 for struct compat_xfs_growfs_data:

typedef struct compat_xfs_growfs_data {

        __u64           newblocks;      /* new data subvol size, fsblocks */
        __u32           imaxpct;        /* new inode space percentage limit */

} __attribute__((packed)) compat_xfs_growfs_data_t;

On a 64-bit kernel, that packed attribute is the difference between expecting a padded 16 byte struct vs. a 12 byte version presumably from a 32-bit application. So if you are calling into the ->compat_ioctl() path I think the question is why is your xfsprogs sending the 16 byte structure?

Brian

Quoted 2 lines

Thanks,
Nick

#4 lore raw
Quoted 45 lines · Brian Foster
Quoted 8 lines · Nick Bowler

Hi Brian,

Quoted 5 lines · Brian Foster

The only thing that comes to mind while poking through the code is perhaps xfsprogs is sending the traditional XFS_IOC_GROWFSDATA command into the compat_ioctl() path somehow or another (assuming BROKEN_X86_ALIGNMENT is set).

Quoted 5 lines
Quoted 1 line

What arch is your kernel/xfsprogs?

This system is running an amd64 kernel with x32 userspace (including xfsprogs).

Ok, so I think that means BROKEN_X86_ALIGNMENT should be set since XFS defines it as:

#if defined(CONFIG_IA64) || defined(CONFIG_X86_64) #define BROKEN_X86_ALIGNMENT
...

Quoted 7 lines
Quoted 2 lines

What does 'cat /sys/kernel/debug/trace/trace' show if you run 'trace-cmd start -e xfs:xfs_file*ioctl*' and then attempt the growfs?

Looks like I don't have the required tracing enabled in my kernel configuration, but I can build a new one if needed. Is CONFIG_FTRACE sufficient for this?

Not sure. I think you need to have CONFIG_TRACING enabled, which may require FTRACE and/or some other options. Hmm, perhaps you'd be covered if you make sure you have CONFIG_DYNAMIC_FTRACE enabled.

From your strace output:

ioctl(3, _IOC(_IOC_WRITE, 0x58, 0x6e, 0x10), 0xffcc9a80) = -1 ENOTTY (Inappropriate ioctl for device)

0x6e corresponds to the GROWFSDATA[_32] cmd and I think 0x10 is the size, which is 16 bytes as opposed to the 12 bytes expected for GROWFSDATA_32 for struct compat_xfs_growfs_data:

typedef struct compat_xfs_growfs_data {

        __u64           newblocks;      /* new data subvol size, fsblocks */
        __u32           imaxpct;        /* new inode space percentage limit */
Quoted 7 lines

} __attribute__((packed)) compat_xfs_growfs_data_t;

On a 64-bit kernel, that packed attribute is the difference between expecting a padded 16 byte struct vs. a 12 byte version presumably from a 32-bit application. So if you are calling into the ->compat_ioctl() path I think the question is why is your xfsprogs sending the 16 byte structure?

...because the x32 ABI is weird in that pointers are 4 bytes like on x86, but the registers are 64 bits wide like on x64, and (except for pointers being 4 bytes wide) the structure alignment rules follow x64.

Normally xfs structures are explicitly padded to 8-byte boundaries and pointers forced into u64 fields to avoid all of this compatibility headache, but this wasn't done with struct xfs_growfs_data, so it needs a compatibility shim for every ABI supported by Linux.

As you can tell, we never really bothered to check in XFS. The creators of the x32 ABI even call out XFS ioctl32.c[1] specifically on their list of things that needed fixing, but they never got around to it.

https://sites.google.com/site/x32abi/

$ cat b.c
struct moo {
        unsigned long long a;
        unsigned int b;
};

int bork(struct moo *m)

{
        return m->b;
}
$ gcc -Wall -g -m64 -c -o b.o b.c	# x86_64
$ gdb b.o
Reading symbols from b.o...done.
(gdb) p sizeof(struct moo)
$1 = 12
(gdb) quit
$ gcc -Wall -g -m32 -c -o b.o b.c	# i386
$ gdb b.o
Reading symbols from b.o...done.
(gdb) p sizeof(struct moo)
$1 = 12
(gdb) quit
$ gcc -Wall -g -mx32 -c -o b.o b.c	# x32
$ gdb b.o
Reading symbols from b.o...done.
(gdb) p sizeof(struct moo)
$1 = 16

So I guess someone needs to fix the headers to detect x32 and point it at the x64 definitions ... or something. Personally, I thought x32 was basically dead at this point, but clearly not. :/

--D

Quoted 4 lines

Brian

Quoted 2 lines

Thanks,
Nick

#5 lore raw
Quoted 76 lines · Darrick J. Wong
Quoted 45 lines · Brian Foster
Quoted 8 lines · Nick Bowler

Hi Brian,

Quoted 5 lines · Brian Foster

The only thing that comes to mind while poking through the code is perhaps xfsprogs is sending the traditional XFS_IOC_GROWFSDATA command into the compat_ioctl() path somehow or another (assuming BROKEN_X86_ALIGNMENT is set).

Quoted 5 lines
Quoted 1 line

What arch is your kernel/xfsprogs?

This system is running an amd64 kernel with x32 userspace (including xfsprogs).

Ok, so I think that means BROKEN_X86_ALIGNMENT should be set since XFS defines it as:

#if defined(CONFIG_IA64) || defined(CONFIG_X86_64) #define BROKEN_X86_ALIGNMENT
...

Quoted 7 lines
Quoted 2 lines

What does 'cat /sys/kernel/debug/trace/trace' show if you run 'trace-cmd start -e xfs:xfs_file*ioctl*' and then attempt the growfs?

Looks like I don't have the required tracing enabled in my kernel configuration, but I can build a new one if needed. Is CONFIG_FTRACE sufficient for this?

Not sure. I think you need to have CONFIG_TRACING enabled, which may require FTRACE and/or some other options. Hmm, perhaps you'd be covered if you make sure you have CONFIG_DYNAMIC_FTRACE enabled.

From your strace output:

ioctl(3, _IOC(_IOC_WRITE, 0x58, 0x6e, 0x10), 0xffcc9a80) = -1 ENOTTY (Inappropriate ioctl for device)

0x6e corresponds to the GROWFSDATA[_32] cmd and I think 0x10 is the size, which is 16 bytes as opposed to the 12 bytes expected for GROWFSDATA_32 for struct compat_xfs_growfs_data:

typedef struct compat_xfs_growfs_data {

        __u64           newblocks;      /* new data subvol size, fsblocks */
        __u32           imaxpct;        /* new inode space percentage limit */
Quoted 7 lines

} __attribute__((packed)) compat_xfs_growfs_data_t;

On a 64-bit kernel, that packed attribute is the difference between expecting a padded 16 byte struct vs. a 12 byte version presumably from a 32-bit application. So if you are calling into the ->compat_ioctl() path I think the question is why is your xfsprogs sending the 16 byte structure?

...because the x32 ABI is weird in that pointers are 4 bytes like on x86, but the registers are 64 bits wide like on x64, and (except for pointers being 4 bytes wide) the structure alignment rules follow x64.

Normally xfs structures are explicitly padded to 8-byte boundaries and pointers forced into u64 fields to avoid all of this compatibility headache, but this wasn't done with struct xfs_growfs_data, so it needs a compatibility shim for every ABI supported by Linux.

As you can tell, we never really bothered to check in XFS. The creators of the x32 ABI even call out XFS ioctl32.c[1] specifically on their list of things that needed fixing, but they never got around to it.

https://sites.google.com/site/x32abi/

$ cat b.c
struct moo {
        unsigned long long a;
        unsigned int b;
};

int bork(struct moo *m)

Quoted 8 lines
{
        return m->b;
}
$ gcc -Wall -g -m64 -c -o b.o b.c	# x86_64
$ gdb b.o
Reading symbols from b.o...done.
(gdb) p sizeof(struct moo)
$1 = 12

<sigh> paste error, that's supposed to be 16.

--D

Quoted 23 lines

(gdb) quit

$ gcc -Wall -g -m32 -c -o b.o b.c	# i386
$ gdb b.o
Reading symbols from b.o...done.
(gdb) p sizeof(struct moo)
$1 = 12
(gdb) quit
$ gcc -Wall -g -mx32 -c -o b.o b.c	# x32
$ gdb b.o
Reading symbols from b.o...done.
(gdb) p sizeof(struct moo)
$1 = 16

So I guess someone needs to fix the headers to detect x32 and point it at the x64 definitions ... or something. Personally, I thought x32 was basically dead at this point, but clearly not. :/

--D

Quoted 4 lines

Brian

Quoted 2 lines

Thanks,
Nick

#6 lore raw
In reply to Darrick J. Wong
Quoted 54 lines · Darrick J. Wong
Quoted 45 lines · Brian Foster
Quoted 8 lines · Nick Bowler

Hi Brian,

Quoted 5 lines · Brian Foster

The only thing that comes to mind while poking through the code is perhaps xfsprogs is sending the traditional XFS_IOC_GROWFSDATA command into the compat_ioctl() path somehow or another (assuming BROKEN_X86_ALIGNMENT is set).

Quoted 5 lines
Quoted 1 line

What arch is your kernel/xfsprogs?

This system is running an amd64 kernel with x32 userspace (including xfsprogs).

Ok, so I think that means BROKEN_X86_ALIGNMENT should be set since XFS defines it as:

#if defined(CONFIG_IA64) || defined(CONFIG_X86_64) #define BROKEN_X86_ALIGNMENT
...

Quoted 7 lines
Quoted 2 lines

What does 'cat /sys/kernel/debug/trace/trace' show if you run 'trace-cmd start -e xfs:xfs_file*ioctl*' and then attempt the growfs?

Looks like I don't have the required tracing enabled in my kernel configuration, but I can build a new one if needed. Is CONFIG_FTRACE sufficient for this?

Not sure. I think you need to have CONFIG_TRACING enabled, which may require FTRACE and/or some other options. Hmm, perhaps you'd be covered if you make sure you have CONFIG_DYNAMIC_FTRACE enabled.

From your strace output:

ioctl(3, _IOC(_IOC_WRITE, 0x58, 0x6e, 0x10), 0xffcc9a80) = -1 ENOTTY (Inappropriate ioctl for device)

0x6e corresponds to the GROWFSDATA[_32] cmd and I think 0x10 is the size, which is 16 bytes as opposed to the 12 bytes expected for GROWFSDATA_32 for struct compat_xfs_growfs_data:

typedef struct compat_xfs_growfs_data {

        __u64           newblocks;      /* new data subvol size, fsblocks */
        __u32           imaxpct;        /* new inode space percentage limit */
Quoted 7 lines

} __attribute__((packed)) compat_xfs_growfs_data_t;

On a 64-bit kernel, that packed attribute is the difference between expecting a padded 16 byte struct vs. a 12 byte version presumably from a 32-bit application. So if you are calling into the ->compat_ioctl() path I think the question is why is your xfsprogs sending the 16 byte structure?

Quoted 4 lines

...because the x32 ABI is weird in that pointers are 4 bytes like on x86, but the registers are 64 bits wide like on x64, and (except for pointers being 4 bytes wide) the structure alignment rules follow x64.

Thanks, I wasn't aware of that. I just read x32 as x86.

Quoted 11 lines

Normally xfs structures are explicitly padded to 8-byte boundaries and pointers forced into u64 fields to avoid all of this compatibility headache, but this wasn't done with struct xfs_growfs_data, so it needs a compatibility shim for every ABI supported by Linux.

As you can tell, we never really bothered to check in XFS. The creators of the x32 ABI even call out XFS ioctl32.c[1] specifically on their list of things that needed fixing, but they never got around to it.

https://sites.google.com/site/x32abi/

...

Quoted 5 lines

So I guess someone needs to fix the headers to detect x32 and point it at the x64 definitions ... or something. Personally, I thought x32 was basically dead at this point, but clearly not. :/

Yeah, it seems to me that fundamentally conflicts with the whole BROKEN_X86_ALIGNMENT thing we have now. IIUC, compat_ioctl() on an x86_64 kernel needs to account for x86 userspace via all of the associated _32 ioctl commands as it already does, but at the same time x32 means we could get any of the traditional x86_64 commands through that path as well.

I guess the short answer in the meantime is that XFS apparently doesn't support this architecture.

Brian

Quoted 6 lines

--D

Quoted 4 lines

Brian

Quoted 2 lines

Thanks,
Nick

#7 lore raw
Quoted 22 lines · Brian Foster
Quoted 21 lines · Darrick J. Wong
Quoted 11 lines · Brian Foster

0x6e corresponds to the GROWFSDATA[_32] cmd and I think 0x10 is the size, which is 16 bytes as opposed to the 12 bytes expected for GROWFSDATA_32 for struct compat_xfs_growfs_data:

typedef struct compat_xfs_growfs_data {
__u64 newblocks; /* new data subvol size,
fsblocks */
__u32 imaxpct; /* new inode space percentage
limit */
} __attribute__((packed)) compat_xfs_growfs_data_t;

Quoted 5 lines

On a 64-bit kernel, that packed attribute is the difference between expecting a padded 16 byte struct vs. a 12 byte version presumably from a 32-bit application. So if you are calling into the ->compat_ioctl() path I think the question is why is your xfsprogs sending the 16 byte structure?

...because the x32 ABI is weird in that pointers are 4 bytes like on x86, but the registers are 64 bits wide like on x64, and (except for pointers being 4 bytes wide) the structure alignment rules follow x64.

[...]

Quoted 6 lines

Yeah, it seems to me that fundamentally conflicts with the whole BROKEN_X86_ALIGNMENT thing we have now. IIUC, compat_ioctl() on an x86_64 kernel needs to account for x86 userspace via all of the associated _32 ioctl commands as it already does, but at the same time x32 means we could get any of the traditional x86_64 commands through that path as well.

In the specific case of this one ioctl on this one architecture since the only problem is unused padding at the end of the structure, the fix might be simple: just accept both ioctl numbers in the compat path. The packed compat struct layout looks like it should match what x32 userspace sends just fine. (I didn't realize x32 syscalls would go through compat_ioctl).

i.e., perhaps we just do something like this? (TOTALLY UNTESTED)

fs/xfs/xfs_ioctl32.c +3 -0
index fba115f4103a..b5a02f36d568 100644
@@ -581,6 +581,9 @@ xfs_file_compat_ioctl(
        }
        case XFS_IOC_FSGEOMETRY_V1_32:
                return xfs_compat_ioc_fsgeometry_v1(mp, arg);
+#ifdef CONFIG_X86_X32
+       case XFS_IOC_FSGROWFSDATA:
+#endif
        case XFS_IOC_FSGROWFSDATA_32: {
                struct xfs_growfs_data  in;
 

I can have a go at fixing the FSGEOMETRY ioctl too (and submit it properly) if this approach seems reasonable. Possibly other things may be broken too but I haven't hit any other issues yet in my XFS adventure.

Cheers,
Nick

#8 lore raw
Quoted 44 lines · Nick Bowler
Quoted 22 lines · Brian Foster
Quoted 21 lines · Darrick J. Wong
Quoted 11 lines · Brian Foster

0x6e corresponds to the GROWFSDATA[_32] cmd and I think 0x10 is the size, which is 16 bytes as opposed to the 12 bytes expected for GROWFSDATA_32 for struct compat_xfs_growfs_data:

typedef struct compat_xfs_growfs_data {
__u64 newblocks; /* new data subvol size,
fsblocks */
__u32 imaxpct; /* new inode space percentage
limit */
} __attribute__((packed)) compat_xfs_growfs_data_t;

Quoted 5 lines

On a 64-bit kernel, that packed attribute is the difference between expecting a padded 16 byte struct vs. a 12 byte version presumably from a 32-bit application. So if you are calling into the ->compat_ioctl() path I think the question is why is your xfsprogs sending the 16 byte structure?

...because the x32 ABI is weird in that pointers are 4 bytes like on x86, but the registers are 64 bits wide like on x64, and (except for pointers being 4 bytes wide) the structure alignment rules follow x64.

[...]

Quoted 6 lines

Yeah, it seems to me that fundamentally conflicts with the whole BROKEN_X86_ALIGNMENT thing we have now. IIUC, compat_ioctl() on an x86_64 kernel needs to account for x86 userspace via all of the associated _32 ioctl commands as it already does, but at the same time x32 means we could get any of the traditional x86_64 commands through that path as well.

In the specific case of this one ioctl on this one architecture since the only problem is unused padding at the end of the structure, the fix might be simple: just accept both ioctl numbers in the compat path. The packed compat struct layout looks like it should match what x32 userspace sends just fine. (I didn't realize x32 syscalls would go through compat_ioctl).

i.e., perhaps we just do something like this? (TOTALLY UNTESTED)

fs/xfs/xfs_ioctl32.c +0 -0
index fba115f4103a..b5a02f36d568 100644

fs/xfs/xfs_ioctl32.c xfs_file_compat_ioctl( · line 588

Quoted 8 lines
fs/xfs/xfs_ioctl32.c +3 -0
@@ -581,6 +581,9 @@ xfs_file_compat_ioctl(
        }
        case XFS_IOC_FSGEOMETRY_V1_32:
                return xfs_compat_ioc_fsgeometry_v1(mp, arg);
+#ifdef CONFIG_X86_X32
+       case XFS_IOC_FSGROWFSDATA:
+#endif
        case XFS_IOC_FSGROWFSDATA_32: {
                struct xfs_growfs_data  in;

Ugly, but something like that may be our only option here.

Quoted 5 lines

I can have a go at fixing the FSGEOMETRY ioctl too (and submit it properly) if this approach seems reasonable. Possibly other things may be broken too but I haven't hit any other issues yet in my XFS adventure.

We really need to audit all the compat ioctls for this same problem and fix all of them in one go, not just slap a bandaid on the messenger and ignore the rest....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#9 lore raw

Hi Dave,

Quoted 9 lines · Dave Chinner
Quoted 4 lines · Nick Bowler

I can have a go at fixing the FSGEOMETRY ioctl too (and submit it properly) if this approach seems reasonable. Possibly other things may be broken too but I haven't hit any other issues yet in my XFS adventure.

We really need to audit all the compat ioctls for this same problem and fix all of them in one go, not just slap a bandaid on the messenger and ignore the rest....

OK then. This patch should cover all of them. However, I wouldn't know where to start with verification of a change like this, since I don't know what these ioctls actually do, but xfs_growfs does seem to work for me now on a test filesystem with this applied.

I didn't look at the set of ioctls marked 'No size or alignment issue on any arch' since those should presumably all be fine.

I manually inspected the structures used by everything else. On all the cases under BROKEN_X86_ALIGNMENT, the structures differ due to 64-bit integer alignment reasons alone. In this case, x32 will match amd64 exactly, including the calculated ioctl number (which is different from the ia32 ioctl number since the structure sizes change), so we just need to emit BOTH sets of cases to handle this correctly.

The odd one out is XFS_IOC_SWAPEXT. This is a rather big structure but it appears to consist only of integers of various sizes, with some time_t in there. That should still all exactly match between x32 and amd64, and the structure size is different from ia32 so we can add a case that calls the native version.

All remaining ioctls use structures that consist exclusively of 32-bit integers and one or more pointers (or, recursively, structures containing only such members). That means x32 should match ia32 exactly for these, including the calculated ioctl number, so the compat versions should be fine as-is for these and no change is required.

fs/xfs/xfs_ioctl32.c +6 -2
index fba115f4103a..dd77f20f42ec 100644
@@ -547,7 +547,7 @@ xfs_file_compat_ioctl(
 	case FS_IOC_GETFSMAP:
 	case XFS_IOC_SCRUB_METADATA:
 		return xfs_file_ioctl(filp, cmd, p);
-#ifndef BROKEN_X86_ALIGNMENT
+#if !defined(BROKEN_X86_ALIGNMENT) || defined(CONFIG_X86_X32)
 	/* These are handled fine if no alignment issues */
 	case XFS_IOC_ALLOCSP:
 	case XFS_IOC_FREESP:
@@ -561,8 +561,12 @@ xfs_file_compat_ioctl(
 	case XFS_IOC_FSGROWFSDATA:
 	case XFS_IOC_FSGROWFSRT:
 	case XFS_IOC_ZERO_RANGE:
+#ifdef CONFIG_X86_X32
+	case XFS_IOC_SWAPEXT:
+#endif
 		return xfs_file_ioctl(filp, cmd, p);
-#else
+#endif
+#if defined(BROKEN_X86_ALIGNMENT)
 	case XFS_IOC_ALLOCSP_32:
 	case XFS_IOC_FREESP_32:
 	case XFS_IOC_ALLOCSP64_32:
#10 lore raw
Quoted 9 lines · Nick Bowler

Hi Dave,

Quoted 6 lines · Dave Chinner
Quoted 4 lines · Nick Bowler

I can have a go at fixing the FSGEOMETRY ioctl too (and submit it properly) if this approach seems reasonable. Possibly other things may be broken too but I haven't hit any other issues yet in my XFS adventure.

Quoted 9 lines
Quoted 3 lines

We really need to audit all the compat ioctls for this same problem and fix all of them in one go, not just slap a bandaid on the messenger and ignore the rest....

OK then. This patch should cover all of them. However, I wouldn't know where to start with verification of a change like this, since I don't know what these ioctls actually do, but xfs_growfs does seem to work for me now on a test filesystem with this applied.

Given that the structure size essentially changes the command value, I'm kind of curious why we have this ifdeffery in the first place. That aside, the patch seems reasonable to me at a glance (though some brief comments around the ifdefs would be nice).

Note that verification doesn't just mean checking whether your particular configuration works, but also that nothing has broken on other potentially affected configurations. For example, I'd suggest that this kind of change at least warrants an additional regression test of 32-bit (x86) userspace on a 64-bit kernel that has CONFIG_X86_X32 enabled so we can make sure that we haven't disrupted anything in the compat_ioctl() path for the non-x32 case.

As for test mechanism, look at the fstests suite:

https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git

It contains a suite of tests that all fs devs run against most changes before they are posted/committed to the upstream kernel. Within that tool, see the xfstests-dev/ltp/fsstress and xfstests-dev/ltp/fsx test utilities. Each of those utils basically runs a stress test that consists of all supported operations against a target directory. For example, the set of operations noted by fsstress is:

the valid operation names are:
afsync allocsp aread attr_remove attr_set awrite bulkstat bulkstat1 chown clonerange copyrange creat deduperange dread dwrite fallocate fdatasync fiemap freesp fsync getattr getdents link mkdir mknod mread mwrite punch zero collapse insert read readlink readv rename resvsp rmdir setattr setxattr stat symlink sync truncate unlink unresvsp write writev

... which covers several of these ioctls (block allocation, bulkstat, etc.). The broader fstests suite has tests to cover things like growfs, etc., and will also run the aforementioned tools, but you can run them directly for quicker verification during development.

So if you'd like to propose this change for upstream inclusion, I'd suggest to take advantage of those tools to test your change and then run the xfstests suite (re: the auto test group) against the targeted configuration along with one or more affected configurations.

Brian

Quoted 49 lines

I didn't look at the set of ioctls marked 'No size or alignment issue on any arch' since those should presumably all be fine.

I manually inspected the structures used by everything else. On all the cases under BROKEN_X86_ALIGNMENT, the structures differ due to 64-bit integer alignment reasons alone. In this case, x32 will match amd64 exactly, including the calculated ioctl number (which is different from the ia32 ioctl number since the structure sizes change), so we just need to emit BOTH sets of cases to handle this correctly.

The odd one out is XFS_IOC_SWAPEXT. This is a rather big structure but it appears to consist only of integers of various sizes, with some time_t in there. That should still all exactly match between x32 and amd64, and the structure size is different from ia32 so we can add a case that calls the native version.

All remaining ioctls use structures that consist exclusively of 32-bit integers and one or more pointers (or, recursively, structures containing only such members). That means x32 should match ia32 exactly for these, including the calculated ioctl number, so the compat versions should be fine as-is for these and no change is required.

fs/xfs/xfs_ioctl32.c +6 -2
index fba115f4103a..dd77f20f42ec 100644
@@ -547,7 +547,7 @@ xfs_file_compat_ioctl(
 	case FS_IOC_GETFSMAP:
 	case XFS_IOC_SCRUB_METADATA:
 		return xfs_file_ioctl(filp, cmd, p);
-#ifndef BROKEN_X86_ALIGNMENT
+#if !defined(BROKEN_X86_ALIGNMENT) || defined(CONFIG_X86_X32)
 	/* These are handled fine if no alignment issues */
 	case XFS_IOC_ALLOCSP:
 	case XFS_IOC_FREESP:
@@ -561,8 +561,12 @@ xfs_file_compat_ioctl(
 	case XFS_IOC_FSGROWFSDATA:
 	case XFS_IOC_FSGROWFSRT:
 	case XFS_IOC_ZERO_RANGE:
+#ifdef CONFIG_X86_X32
+	case XFS_IOC_SWAPEXT:
+#endif
 		return xfs_file_ioctl(filp, cmd, p);
-#else
+#endif
+#if defined(BROKEN_X86_ALIGNMENT)
 	case XFS_IOC_ALLOCSP_32:
 	case XFS_IOC_FREESP_32:
 	case XFS_IOC_ALLOCSP64_32:
#11 lore raw
Quoted 17 lines · Brian Foster
Quoted 7 lines · Nick Bowler
Quoted 6 lines · Dave Chinner
Quoted 4 lines · Nick Bowler

I can have a go at fixing the FSGEOMETRY ioctl too (and submit it properly) if this approach seems reasonable. Possibly other things may be broken too but I haven't hit any other issues yet in my XFS adventure.

Quoted 8 lines
Quoted 3 lines

We really need to audit all the compat ioctls for this same problem and fix all of them in one go, not just slap a bandaid on the messenger and ignore the rest....

OK then. This patch should cover all of them. However, I wouldn't know where to start with verification of a change like this, since I don't know what these ioctls actually do, but xfs_growfs does seem to work for me now on a test filesystem with this applied.

Quoted 4 lines

Given that the structure size essentially changes the command value, I'm kind of curious why we have this ifdeffery in the first place. That aside, the patch seems reasonable to me at a glance (though some brief comments around the ifdefs would be nice).

OK. The ifdeffery will mean the "native" ioctls number only get accepted in the compat case on select architectures. In principle it could be removed and both "native" and "compat" ioctl numbers would be accepted all the time in the compat case (so the possible impact would mainly be on non-x86 and hopefully would be limited to code size and/or "some ioctl numbers gave errors before and now don't")

I think it's a separate discussion for whether that's prudent or not. Current code has the ifdeffery. Also since it's a syntax error to have multiple case labels with the same value it'd be essential to validate that all supported architectures architectures end up with different values for each XFS_IOC_xyz and the corresponding XFS_IOC_xyz_32.

Quoted 7 lines

Note that verification doesn't just mean checking whether your particular configuration works, but also that nothing has broken on other potentially affected configurations. For example, I'd suggest that this kind of change at least warrants an additional regression test of 32-bit (x86) userspace on a 64-bit kernel that has CONFIG_X86_X32 enabled so we can make sure that we haven't disrupted anything in the compat_ioctl() path for the non-x32 case.

Indeed. I will run xfstests in a vm (thanks for the instructions) and report before/after results for x32-on-x86_64 and i686-on-x86_64.

Cheers,
Nick

#12 lore raw
Quoted 5 lines · Nick Bowler
Quoted 4 lines · Brian Foster

Given that the structure size essentially changes the command value, I'm kind of curious why we have this ifdeffery in the first place. That aside, the patch seems reasonable to me at a glance (though some brief comments around the ifdefs would be nice).

[...]

Quoted 4 lines

Current code has the ifdeffery. Also since it's a syntax error to have multiple case labels with the same value it'd be essential to validate that all supported architectures architectures end up with different values for each XFS_IOC_xyz and the corresponding XFS_IOC_xyz_32.

Right after I write this, I realize that it's almost certainly the case that architectures which _don't_ define BROKEN_X86_ALIGNMENT will have matching ioctl numbers between e.g., XFS_IOC_ALLOCSP and XFS_IOC_ALLOCSP_32. Thus the ifdeffery is essential for the above syntactic reason.

Cheers,
Nick

#13 lore raw
Quoted 12 lines · Nick Bowler
Quoted 5 lines · Nick Bowler
Quoted 4 lines · Brian Foster

Given that the structure size essentially changes the command value, I'm kind of curious why we have this ifdeffery in the first place. That aside, the patch seems reasonable to me at a glance (though some brief comments around the ifdefs would be nice).

[...]

Quoted 4 lines

Current code has the ifdeffery. Also since it's a syntax error to have multiple case labels with the same value it'd be essential to validate that all supported architectures architectures end up with different values for each XFS_IOC_xyz and the corresponding XFS_IOC_xyz_32.

Quoted 6 lines

Right after I write this, I realize that it's almost certainly the case that architectures which _don't_ define BROKEN_X86_ALIGNMENT will have matching ioctl numbers between e.g., XFS_IOC_ALLOCSP and XFS_IOC_ALLOCSP_32. Thus the ifdeffery is essential for the above syntactic reason.

So the two struct [compat_]xfs_flock64 variants are essentially the same with the exception of internal alignment padding that leaves a hole after the first 4 bytes in the 64-bit variant. So on x86_64, these are essentially two different (sized) structures and ioctl commands. On some non-broken alignment arch, the packing may be implied and thus these definitions evaluate down to the same ioctl command. If so, then having those separate but equivalent value commands in the same switch statement results in a compilation error. Am I following that correctly?

If so, then it does seem we need an ifdef to exlude those definitions so long as we follow the one huge switch statement implementation. But is there anything that fundamentally prevents a multiple switch statement implementation to avoid such syntax errors?

Note that in the end I don't care too much about whether we have an ifdef or not. I'm more interested in the most simple and elegant implementation and it just seemed that trying to dilute the existing ifdef may push us in the opposite direction (as opposed to something that for example called into the "native" ioctl and fell back to the _32 variants on failure before returning an error). There may be technical complications to that or using some form of the ifdef may just end up being the more simple approach after all.

Brian

Quoted 2 lines

Cheers,
Nick

#14 lore raw
Quoted 5 lines · Brian Foster
Quoted 4 lines · Nick Bowler

Right after I write this, I realize that it's almost certainly the case that architectures which _don't_ define BROKEN_X86_ALIGNMENT will have matching ioctl numbers between e.g., XFS_IOC_ALLOCSP and XFS_IOC_ALLOCSP_32.

[...]

Quoted 9 lines

So the two struct [compat_]xfs_flock64 variants are essentially the same with the exception of internal alignment padding that leaves a hole after the first 4 bytes in the 64-bit variant. So on x86_64, these are essentially two different (sized) structures and ioctl commands. On some non-broken alignment arch, the packing may be implied and thus these definitions evaluate down to the same ioctl command. If so, then having those separate but equivalent value commands in the same switch statement results in a compilation error. Am I following that correctly?

Quoted 4 lines

If so, then it does seem we need an ifdef to exlude those definitions so long as we follow the one huge switch statement implementation. But is there anything that fundamentally prevents a multiple switch statement implementation to avoid such syntax errors?

I think nothing fundamentally prevents this approach and it is an option.

The only point I see is that, without the configuration ifdefs, the impact of the change is expanded; there are (presumably benign) functional changes on more architectures, most of which I don't have the ability to test. Making it dependent on the arch configuration means nothing should change anywhere except on amd64 kernels that have x32 binary support turned on.

Quoted 8 lines

Note that in the end I don't care too much about whether we have an ifdef or not. I'm more interested in the most simple and elegant implementation and it just seemed that trying to dilute the existing ifdef may push us in the opposite direction (as opposed to something that for example called into the "native" ioctl and fell back to the _32 variants on failure before returning an error). There may be technical complications to that or using some form of the ifdef may just end up being the more simple approach after all.

OK, since I have some test results I'll submit the current one for RFC; afterwards I'll try to follow up with a less ifdeffy variant for comparison purposes, then we can decide which version is better. :)

Cheers,
Nick

#15 lore raw
In reply to Brian Foster
Quoted 20 lines · Brian Foster
Quoted 9 lines · Nick Bowler

Hi Dave,

Quoted 6 lines · Dave Chinner
Quoted 4 lines · Nick Bowler

I can have a go at fixing the FSGEOMETRY ioctl too (and submit it properly) if this approach seems reasonable. Possibly other things may be broken too but I haven't hit any other issues yet in my XFS adventure.

Quoted 9 lines
Quoted 3 lines

We really need to audit all the compat ioctls for this same problem and fix all of them in one go, not just slap a bandaid on the messenger and ignore the rest....

OK then. This patch should cover all of them. However, I wouldn't know where to start with verification of a change like this, since I don't know what these ioctls actually do, but xfs_growfs does seem to work for me now on a test filesystem with this applied.

Quoted 4 lines

Given that the structure size essentially changes the command value, I'm kind of curious why we have this ifdeffery in the first place. That aside, the patch seems reasonable to me at a glance (though some brief comments around the ifdefs would be nice).

OK, xfstests has revealed some trouble with the three "bulkstat" ioctls, since while the xfs_bulkstat structure itself is fine, one of its members is used as a pointer to various structures which are not fine. This wasn't too hard to fix though.

I'll keep testing and hope to get to submitting these properly.

fs/xfs/xfs_ioctl32.c +19 -4
index 001b30605d45..eb6e80c7f2bd 100644
@@ -241,6 +241,21 @@ xfs_compat_ioc_bulkstat(
 	int			done;
 	int			error;
 
+	/* These functions and size are used later to handle individual
+	 * entries; x32 is annoying and needs different functions. */
+	inumbers_fmt_pf inumbers_func = xfs_inumbers_fmt_compat;
+	bulkstat_one_pf	bs_one_func = xfs_bulkstat_one_compat;
+	size_t bs_one_size = sizeof (compat_xfs_bstat_t);
+
+#ifdef CONFIG_X86_X32
+	if (in_x32_syscall()) {
+		/* x32 matches native amd64 bstat and inogrp layout */
+		inumbers_func = xfs_inumbers_fmt;
+		bs_one_func = xfs_bulkstat_one;
+		bs_one_size = sizeof (xfs_bstat_t);
+	}
+#endif
+
 	/* done = 1 if there are more stats to get and if bulkstat */
 	/* should be called again (unused here, but used in dmapi) */
 
@@ -272,15 +287,15 @@ xfs_compat_ioc_bulkstat(
 
 	if (cmd == XFS_IOC_FSINUMBERS_32) {
 		error = xfs_inumbers(mp, &inlast, &count,
-				bulkreq.ubuffer, xfs_inumbers_fmt_compat);
+				bulkreq.ubuffer, inumbers_func);
 	} else if (cmd == XFS_IOC_FSBULKSTAT_SINGLE_32) {
 		int res;
 
-		error = xfs_bulkstat_one_compat(mp, inlast, bulkreq.ubuffer,
-				sizeof(compat_xfs_bstat_t), NULL, &res);
+		error = bs_one_func(mp, inlast, bulkreq.ubuffer,
+		                    bs_one_size, NULL, &res);
 	} else if (cmd == XFS_IOC_FSBULKSTAT_32) {
 		error = xfs_bulkstat(mp, &inlast, &count,
-			xfs_bulkstat_one_compat, sizeof(compat_xfs_bstat_t),
+			bs_one_func, bs_one_size,
 			bulkreq.ubuffer, &done);
 	} else
 		error = -EINVAL;
#16 lore raw
Quoted 26 lines · Nick Bowler
Quoted 20 lines · Brian Foster
Quoted 9 lines · Nick Bowler

Hi Dave,

Quoted 6 lines · Dave Chinner
Quoted 4 lines · Nick Bowler

I can have a go at fixing the FSGEOMETRY ioctl too (and submit it properly) if this approach seems reasonable. Possibly other things may be broken too but I haven't hit any other issues yet in my XFS adventure.

Quoted 9 lines
Quoted 3 lines

We really need to audit all the compat ioctls for this same problem and fix all of them in one go, not just slap a bandaid on the messenger and ignore the rest....

OK then. This patch should cover all of them. However, I wouldn't know where to start with verification of a change like this, since I don't know what these ioctls actually do, but xfs_growfs does seem to work for me now on a test filesystem with this applied.

Quoted 4 lines

Given that the structure size essentially changes the command value, I'm kind of curious why we have this ifdeffery in the first place. That aside, the patch seems reasonable to me at a glance (though some brief comments around the ifdefs would be nice).

Quoted 4 lines

OK, xfstests has revealed some trouble with the three "bulkstat" ioctls, since while the xfs_bulkstat structure itself is fine, one of its members is used as a pointer to various structures which are not fine. This wasn't too hard to fix though.

IIRC, there's bigger problems than you realise here - the bulkstat structure has embedded timestamps in them and on x32 struct timeval doesn't match either ia32 or x86-64. i.e. on ia32, struct timeval is 8 bytes, on x86-64 it is 16 bytes, and in x32 it is 12 bytes.

IOWs, using the x86-64 handlers for bulkstat is wrong, as is using the compat handlers. That's one of the reasons why x32 is such a Charlie Foxtrot when it comes to compat handlers - we basically have to audit ioctl structures one by one with pahole to determine which arch version they *may* be compatible with.

And then there is testing that we get identical output from all three versions for each ioctl.

Right now, I'd much prefer we simply put this at the start of xfs_fs_fill_super():

#ifdef CONFIG_X86_X32
	xfs_warn("XFS not supported on x32 architectures")
	return -ENOSYS;

#endif

Or, alternatively, tag it as EXPERIMENTAL and "use at your own risk".

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#17 lore raw
Quoted 10 lines · Dave Chinner
Quoted 4 lines · Nick Bowler

OK, xfstests has revealed some trouble with the three "bulkstat" ioctls, since while the xfs_bulkstat structure itself is fine, one of its members is used as a pointer to various structures which are not fine. This wasn't too hard to fix though.

IIRC, there's bigger problems than you realise here - the bulkstat structure has embedded timestamps in them and on x32 struct timeval doesn't match either ia32 or x86-64. i.e. on ia32, struct timeval is 8 bytes, on x86-64 it is 16 bytes, and in x32 it is 12 bytes.

This is not the case: struct timeval is 16 bytes on x32:

sizeof (struct timeval): 16

  tv_sec          size:   8 offset:   0
  tv_usec         size:   8 offset:   8

This is the same as what I get on native 64-bit compilations; but anyway the xfs_bstat structure has xfs_bstime members, with the following characteristics on x32:

sizeof (struct xfs_bstime): 16

  tv_sec          size:   8 offset:   0
  tv_nsec         size:   4 offset:   8

which is also the same as native 64-bit (time_t is the same on x32 and
native: 8 bytes with 8 byte alignment).

I manually verified every member of the xfs_bstat structure with sizeof and offsetof on -mx32 and -m64 compilations to ensure that this structure matches precisely between the x32 and native 64-bit cases.

#18 lore raw
Quoted 27 lines · Nick Bowler
Quoted 11 lines · Dave Chinner
Quoted 5 lines · Nick Bowler

OK, xfstests has revealed some trouble with the three "bulkstat" ioctls, since while the xfs_bulkstat structure itself is fine, one of its members
is used as a pointer to various structures which are not fine. This wasn't too hard to fix though.

IIRC, there's bigger problems than you realise here - the bulkstat structure has embedded timestamps in them and on x32 struct timeval doesn't match either ia32 or x86-64. i.e. on ia32, struct timeval is 8 bytes, on x86-64 it is 16 bytes, and in x32 it is 12 bytes.

This is not the case: struct timeval is 16 bytes on x32:

sizeof (struct timeval): 16

  tv_sec          size:   8 offset:   0
  tv_usec         size:   8 offset:   8

This is the same as what I get on native 64-bit compilations; but anyway the xfs_bstat structure has xfs_bstime members, with the following characteristics on x32:

sizeof (struct xfs_bstime): 16

  tv_sec          size:   8 offset:   0
  tv_nsec         size:   4 offset:   8
Quoted 6 lines

which is also the same as native 64-bit (time_t is the same on x32 and
native: 8 bytes with 8 byte alignment).

I manually verified every member of the xfs_bstat structure with sizeof and offsetof on -mx32 and -m64 compilations to ensure that this structure matches precisely between the x32 and native 64-bit cases.

To expand on this, for each structure which my RFC patchset feeds up to the native handler, I first checked them by manual inspection and then double checked using the following program; we can compile with both -mx32 and -m64 and check that the output is identical.

  #include <stdio.h>
  #include <stddef.h>
  #include <xfs/xfs.h>
  #include <time.h>

  #define SHOWSIZE(type) do { \
  	printf("sizeof (%s): %zu\n", #type, sizeof (type)); \
  } while(0)

  #define SHOWMEMBER(type, member) do { \
  	printf("  %-15s size: %3zu offset: %3zu\n", #member, \
  	       sizeof (type){0}.member, offsetof(type, member)); \
  } while(0)

  int main(void)
  {
  	SHOWSIZE(struct xfs_flock64);
  	SHOWMEMBER(struct xfs_flock64, l_type);
  	SHOWMEMBER(struct xfs_flock64, l_whence);
  	SHOWMEMBER(struct xfs_flock64, l_start);
  	SHOWMEMBER(struct xfs_flock64, l_len);
  	SHOWMEMBER(struct xfs_flock64, l_sysid);
  	SHOWMEMBER(struct xfs_flock64, l_pid);
  	SHOWMEMBER(struct xfs_flock64, l_pad);

  	SHOWSIZE(struct xfs_fsop_geom_v1);
  	SHOWMEMBER(struct xfs_fsop_geom_v1, blocksize);
  	SHOWMEMBER(struct xfs_fsop_geom_v1, rtextsize);
  	SHOWMEMBER(struct xfs_fsop_geom_v1, agblocks);
  	SHOWMEMBER(struct xfs_fsop_geom_v1, agcount);
  	SHOWMEMBER(struct xfs_fsop_geom_v1, logblocks);
  	SHOWMEMBER(struct xfs_fsop_geom_v1, sectsize);
  	SHOWMEMBER(struct xfs_fsop_geom_v1, inodesize);
  	SHOWMEMBER(struct xfs_fsop_geom_v1, imaxpct);
  	SHOWMEMBER(struct xfs_fsop_geom_v1, datablocks);
  	SHOWMEMBER(struct xfs_fsop_geom_v1, rtblocks);
  	SHOWMEMBER(struct xfs_fsop_geom_v1, rtextents);
  	SHOWMEMBER(struct xfs_fsop_geom_v1, logstart);
  	SHOWMEMBER(struct xfs_fsop_geom_v1, uuid);
  	SHOWMEMBER(struct xfs_fsop_geom_v1, sunit);
  	SHOWMEMBER(struct xfs_fsop_geom_v1, swidth);
  	SHOWMEMBER(struct xfs_fsop_geom_v1, version);
  	SHOWMEMBER(struct xfs_fsop_geom_v1, flags);
  	SHOWMEMBER(struct xfs_fsop_geom_v1, logsectsize);
  	SHOWMEMBER(struct xfs_fsop_geom_v1, rtsectsize);
  	SHOWMEMBER(struct xfs_fsop_geom_v1, dirblocksize);

  	SHOWSIZE(struct xfs_growfs_data);
  	SHOWMEMBER(struct xfs_growfs_data, newblocks);
  	SHOWMEMBER(struct xfs_growfs_data, imaxpct);

  	SHOWSIZE(struct xfs_growfs_rt);
  	SHOWMEMBER(struct xfs_growfs_rt, newblocks);
  	SHOWMEMBER(struct xfs_growfs_rt, extsize);

  	SHOWSIZE(struct xfs_bstime);
  	SHOWMEMBER(struct xfs_bstime, tv_sec);
  	SHOWMEMBER(struct xfs_bstime, tv_nsec);

  	SHOWSIZE(struct xfs_bstat);
  	SHOWMEMBER(struct xfs_bstat, bs_ino);
  	SHOWMEMBER(struct xfs_bstat, bs_mode);
  	SHOWMEMBER(struct xfs_bstat, bs_nlink);
  	SHOWMEMBER(struct xfs_bstat, bs_uid);
  	SHOWMEMBER(struct xfs_bstat, bs_gid);
  	SHOWMEMBER(struct xfs_bstat, bs_rdev);
  	SHOWMEMBER(struct xfs_bstat, bs_blksize);
  	SHOWMEMBER(struct xfs_bstat, bs_size);
  	SHOWMEMBER(struct xfs_bstat, bs_atime);
  	SHOWMEMBER(struct xfs_bstat, bs_mtime);
  	SHOWMEMBER(struct xfs_bstat, bs_ctime);
  	SHOWMEMBER(struct xfs_bstat, bs_blocks);
  	SHOWMEMBER(struct xfs_bstat, bs_xflags);
  	SHOWMEMBER(struct xfs_bstat, bs_extsize);
  	SHOWMEMBER(struct xfs_bstat, bs_extents);
  	SHOWMEMBER(struct xfs_bstat, bs_gen);
  	SHOWMEMBER(struct xfs_bstat, bs_projid_lo);
  	SHOWMEMBER(struct xfs_bstat, bs_forkoff);
  	SHOWMEMBER(struct xfs_bstat, bs_projid_hi);
  	SHOWMEMBER(struct xfs_bstat, bs_pad);
  	SHOWMEMBER(struct xfs_bstat, bs_cowextsize);
  	SHOWMEMBER(struct xfs_bstat, bs_dmevmask);
  	SHOWMEMBER(struct xfs_bstat, bs_dmstate);
  	SHOWMEMBER(struct xfs_bstat, bs_aextents);

  	SHOWSIZE(struct xfs_inogrp);
  	SHOWMEMBER(struct xfs_inogrp, xi_startino);
  	SHOWMEMBER(struct xfs_inogrp, xi_alloccount);
  	SHOWMEMBER(struct xfs_inogrp, xi_allocmask);

  	SHOWSIZE(struct xfs_swapext);
  	SHOWMEMBER(struct xfs_swapext, sx_version);
  	SHOWMEMBER(struct xfs_swapext, sx_fdtarget);
  	SHOWMEMBER(struct xfs_swapext, sx_fdtmp);
  	SHOWMEMBER(struct xfs_swapext, sx_offset);
  	SHOWMEMBER(struct xfs_swapext, sx_length);
  	SHOWMEMBER(struct xfs_swapext, sx_pad);
  	SHOWMEMBER(struct xfs_swapext, sx_stat);
  }
#19 lore raw
Quoted 14 lines · Nick Bowler
Quoted 13 lines · Nick Bowler
Quoted 11 lines · Dave Chinner
Quoted 5 lines · Nick Bowler

OK, xfstests has revealed some trouble with the three "bulkstat" ioctls, since while the xfs_bulkstat structure itself is fine, one of its members
is used as a pointer to various structures which are not fine. This wasn't too hard to fix though.

IIRC, there's bigger problems than you realise here - the bulkstat structure has embedded timestamps in them and on x32 struct timeval doesn't match either ia32 or x86-64. i.e. on ia32, struct timeval is 8 bytes, on x86-64 it is 16 bytes, and in x32 it is 12 bytes.

Quoted 5 lines
Quoted 5 lines

This is not the case: struct timeval is 16 bytes on x32:

sizeof (struct timeval): 16

  tv_sec          size:   8 offset:   0
  tv_usec         size:   8 offset:   8

I was just quoting the kernel time subsystem maintainer who said that these structures had problems with size and packing.

Quoted 12 lines
Quoted 12 lines

This is the same as what I get on native 64-bit compilations; but anyway the xfs_bstat structure has xfs_bstime members, with the following characteristics on x32:

sizeof (struct xfs_bstime): 16

  tv_sec          size:   8 offset:   0
  tv_nsec         size:   4 offset:   8

which is also the same as native 64-bit (time_t is the same on x32 and
native: 8 bytes with 8 byte alignment).

Quoted 8 lines
Quoted 3 lines

I manually verified every member of the xfs_bstat structure with sizeof and offsetof on -mx32 and -m64 compilations to ensure that this structure matches precisely between the x32 and native 64-bit cases.

To expand on this, for each structure which my RFC patchset feeds up to the native handler, I first checked them by manual inspection and then double checked using the following program; we can compile with both -mx32 and -m64 and check that the output is identical.

So, turn that into an xfstest so that it is always run, diffs the output between compat/native depending on which one is used complete with guards that break the test when we add a new ioctl. We already we have a test that is for explicitly checking that structures on disk are the same for 32/64 bit architectures: tests/xfs/122

That test automates the generation of the test code and output, and if it changes from the golden output, then the test fails. I'd suggest that a similar thing is done here for /all/ the structures we expose in ioctls.

FWIW, pahole can make this easy. e.g you can harvest every ioctl structure from xfs_fs.h, write them into a file like so:

$ cat t.c
#include <xfs/xfs.h>

/* these should be the same for x32/x86-64, but not i386 */ int main(int argc, char *argv[])

{
	struct xfs_bstat bs = {0};
	struct xfs_fsop_geom_v1 geo1 = {0};
	struct xfs_fsop_geom geo = {0};
	struct xfs_growfs_data gd = {0};
	struct xfs_growfs_rt gr = {0};
	struct xfs_flock64 fl = {0};
	struct xfs_inogrp ig = {0};
	struct xfs_swapext se = {0};

	return 0;
}

And then compile and dump the structure layouts like so:

$ gcc -m64 -gdwarf-2 t.c -c; pahole t.o > t.x86-64
$ gcc -mx32 -gdwarf-2 t.c -c; pahole t.o > t.x32
$ gcc -m32 -gdwarf-2 t.c -c; pahole t.o > t.i386

Then we'll have tests that will fail if we ever change an ioctl or add a new one and don't add it to the test. That guarantees we won't ever forget about this....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#20 lore raw
Quoted 11 lines · Dave Chinner
Quoted 4 lines · Nick Bowler

To expand on this, for each structure which my RFC patchset feeds up to the native handler, I first checked them by manual inspection and then double checked using the following program; we can compile with both -mx32 and -m64 and check that the output is identical.

So, turn that into an xfstest so that it is always run, diffs the output between compat/native depending on which one is used complete with guards that break the test when we add a new ioctl. We already we have a test that is for explicitly checking that structures on disk are the same for 32/64 bit architectures: tests/xfs/122

[...]

Quoted 3 lines

Then we'll have tests that will fail if we ever change an ioctl or add a new one and don't add it to the test. That guarantees we won't ever forget about this....

OK, I will give it a shot to implement such a test. A possible issue is that developers might not have a working x32 build or runtime environment so the test might not get run a lot. But hopefully people adding brand new ioctls don't introduce brand new compat problems; one can dream, right?

Thanks,
Nick

#21 lore raw
Quoted 12 lines · Nick Bowler
Quoted 11 lines · Dave Chinner
Quoted 4 lines · Nick Bowler

To expand on this, for each structure which my RFC patchset feeds up to the native handler, I first checked them by manual inspection and then double checked using the following program; we can compile with both -mx32 and -m64 and check that the output is identical.

So, turn that into an xfstest so that it is always run, diffs the output between compat/native depending on which one is used complete with guards that break the test when we add a new ioctl. We already we have a test that is for explicitly checking that structures on disk are the same for 32/64 bit architectures: tests/xfs/122

Quoted 8 lines

[...]

Quoted 3 lines

Then we'll have tests that will fail if we ever change an ioctl or add a new one and don't add it to the test. That guarantees we won't ever forget about this....

OK, I will give it a shot to implement such a test. A possible issue is that developers might not have a working x32 build or runtime environment so the test might not get run a lot.

We generally test for the <thing> being supported first and only run the test if it is supported. for x86-64 systems, we should at least already support -m32/-m64, so we'll at least get coverage on that, and the reminder that...

Quoted 2 lines

But hopefully people adding brand
new ioctls don't introduce brand new compat problems; one can dream, right?

... introducing new ioctls need thought to ensure we don't introduce new compat problems :)

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#22 lore raw
In reply to Dave Chinner
Quoted 7 lines · Dave Chinner

That test automates the generation of the test code and output, and if it changes from the golden output, then the test fails. I'd suggest that a similar thing is done here for /all/ the structures we expose in ioctls.

FWIW, pahole can make this easy. e.g you can harvest every ioctl structure from xfs_fs.h,

OK, I did just that, by snarfing up the preprocessor output from #including "xfs/xfs_fs.h" and searching for 'struct'.

The pahole tool was previously unknown to me. I like it.

One thing I'll note is that it does not appear to be possible to use pahole to get information about structs that don't have a tag. The xfs_fsid_t typedef is such a type. Fortunately that one does not look like a problem!

Cheers,
Nick

#23 lore raw

RFC PATCH 1/1 xfs: add tests to validate ioctl structure layout.

Introduce two new test cases. The first one greps xfs_fs.h to find any structure types possibly relevant to ioctls, then compares the structure layouts between 64-bit and 32-bit objects. The goal is to ensure that any _new_ structure has the same characteristics on 32-bit versus 64-bit mode, ensuring a sane compat handler. This requires a toolchain that can build both 64-bit and 32-bit objects, and hopefully works on non-x86 platforms with compat woes.

The whitelist was constructed by manually inspecting the current compat ioctl implementation. The test validates that everything in the whitelist actually got found by the grepping as a sort of self-sanity check, but if older versions lacked some of these types then this might need tweaking a bit.

The second test validates that a well-known set of structures on x32 crrectly match the ia32 or amd64 layouts as expected. This requires a toolchain that can build for all three ABIs.

The idea is that new additions to the header where the structures differ in 32 and 64-bit x86 will be caught by the first test case and that will be sufficient to detect potential x32 problems too.

---
 .gitignore           |   1 +
 common/config        |   1 +
 configure.ac         |   2 +-
 include/build-env.in |   6 +++
 tests/xfs/499        | 102 +++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/499.out    |  16 ++++++++
 tests/xfs/500        | 105 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/500.out    |   2 +
 tests/xfs/group      |   2 +
 9 files changed, 236 insertions(+), 1 deletion(-)
 create mode 100644 include/build-env.in
 create mode 100755 tests/xfs/499
 create mode 100644 tests/xfs/499.out
 create mode 100755 tests/xfs/500
 create mode 100644 tests/xfs/500.out
.gitignore +1 -0
index ea1aac8a..7bfd20fe 100644
@@ -28,6 +28,7 @@
 /ltmain.sh
 
 # build system
+/include/build-env
 /include/builddefs
 /include/config.h
 /include/config.h.in
common/config +1 -0
index a87cb4a2..707c5b2f 100644
@@ -194,6 +194,7 @@ export GETCAP_PROG="$(type -P getcap)"
 export CHECKBASHISMS_PROG="$(type -P checkbashisms)"
 export XFS_INFO_PROG="$(type -P xfs_info)"
 export DUPEREMOVE_PROG="$(type -P duperemove)"
+export PAHOLE_PROG="$(type -P pahole)"
 
 # use 'udevadm settle' or 'udevsettle' to wait for lv to be settled.
 # newer systems have udevadm command but older systems like RHEL5 don't.
configure.ac +1 -1
index 19798824..69a4e396 100644
@@ -73,5 +73,5 @@ AC_HAVE_COPY_FILE_RANGE
 AC_CHECK_FUNCS([renameat2])
 
 AC_CONFIG_HEADER(include/config.h)
-AC_CONFIG_FILES([include/builddefs])
+AC_CONFIG_FILES([include/builddefs include/build-env])
 AC_OUTPUT
include/build-env.in new file +6 -0
new file mode 100644
index 00000000..9a9a6270
@@ -0,0 +1,6 @@
+# Generated by config.status to export build toolchain information
+# for use in testcases.
+
+: "${CC_PROG=`command -v @CC@`}"
+: "${CPPFLAGS=@CPPFLAGS@}"
+: "${CFLAGS=@CFLAGS@}"
tests/xfs/499 new file +102 -0

102 lines not on this page · raw message

tests/xfs/499.out new file +16 -0
new file mode 100644
index 00000000..80e26046
@@ -0,0 +1,16 @@
+QA output created by 499
+whitelist struct xfs_attr_multiop
+whitelist struct xfs_bstat
+whitelist struct xfs_bstime
+whitelist struct xfs_flock64
+whitelist struct xfs_fsop_attrlist_handlereq
+whitelist struct xfs_fsop_attrmulti_handlereq
+whitelist struct xfs_fsop_bulkreq
+whitelist struct xfs_fsop_geom_v1
+whitelist struct xfs_fsop_handlereq
+whitelist struct xfs_fsop_setdm_handlereq
+whitelist struct xfs_growfs_data
+whitelist struct xfs_growfs_rt
+whitelist struct xfs_inogrp
+whitelist struct xfs_swapext
+Silence is golden
tests/xfs/500 new file +105 -0

105 lines not on this page · raw message

tests/xfs/500.out new file +2 -0
new file mode 100644
index 00000000..883b2caf
@@ -0,0 +1,2 @@
+QA output created by 500
+Silence is golden
tests/xfs/group +2 -0
index dfaae2bc..cb69d28a 100644
@@ -496,3 +496,5 @@
 496 dangerous_fuzzers dangerous_scrub dangerous_repair
 497 dangerous_fuzzers dangerous_scrub dangerous_online_repair
 498 dangerous_fuzzers dangerous_norepair
+499 auto quick
+500 auto quick
Signature · 1 line
2.16.1
#24 lore raw

Re: [RFC PATCH xfstests] xfs: add tests to validate ioctl structure layout.

Quoted 18 lines · Nick Bowler

Introduce two new test cases. The first one greps xfs_fs.h to find any structure types possibly relevant to ioctls, then compares the structure layouts between 64-bit and 32-bit objects. The goal is to ensure that any _new_ structure has the same characteristics on 32-bit versus 64-bit mode, ensuring a sane compat handler. This requires a toolchain that can build both 64-bit and 32-bit objects, and hopefully works on non-x86 platforms with compat woes.

The whitelist was constructed by manually inspecting the current compat ioctl implementation. The test validates that everything in the whitelist actually got found by the grepping as a sort of self-sanity check, but if older versions lacked some of these types then this might need tweaking a bit.

The second test validates that a well-known set of structures on x32 crrectly match the ia32 or amd64 layouts as expected. This requires a toolchain that can build for all three ABIs.

Quoted 5 lines

The idea is that new additions to the header where the structures differ in 32 and 64-bit x86 will be caught by the first test case and that will be sufficient to detect potential x32 problems too.

I believe this effort was dropped. Respin?

Luis

#25 lore raw
In reply to Dave Chinner
Quoted 49 lines · Dave Chinner
Quoted 26 lines · Nick Bowler
Quoted 20 lines · Brian Foster
Quoted 9 lines · Nick Bowler

Hi Dave,

Quoted 6 lines · Dave Chinner
Quoted 4 lines · Nick Bowler

I can have a go at fixing the FSGEOMETRY ioctl too (and submit it properly) if this approach seems reasonable. Possibly other things may be broken too but I haven't hit any other issues yet in my XFS adventure.

Quoted 9 lines
Quoted 3 lines

We really need to audit all the compat ioctls for this same problem and fix all of them in one go, not just slap a bandaid on the messenger and ignore the rest....

OK then. This patch should cover all of them. However, I wouldn't know where to start with verification of a change like this, since I don't know what these ioctls actually do, but xfs_growfs does seem to work for me now on a test filesystem with this applied.

Quoted 4 lines

Given that the structure size essentially changes the command value, I'm kind of curious why we have this ifdeffery in the first place. That aside, the patch seems reasonable to me at a glance (though some brief comments around the ifdefs would be nice).

Quoted 4 lines

OK, xfstests has revealed some trouble with the three "bulkstat" ioctls, since while the xfs_bulkstat structure itself is fine, one of its members is used as a pointer to various structures which are not fine. This wasn't too hard to fix though.

IIRC, there's bigger problems than you realise here - the bulkstat structure has embedded timestamps in them and on x32 struct timeval doesn't match either ia32 or x86-64. i.e. on ia32, struct timeval is 8 bytes, on x86-64 it is 16 bytes, and in x32 it is 12 bytes.

IOWs, using the x86-64 handlers for bulkstat is wrong, as is using the compat handlers. That's one of the reasons why x32 is such a Charlie Foxtrot when it comes to compat handlers - we basically have to audit ioctl structures one by one with pahole to determine which arch version they *may* be compatible with.

And then there is testing that we get identical output from all three versions for each ioctl.

Right now, I'd much prefer we simply put this at the start of xfs_fs_fill_super():

Quoted 7 lines
#ifdef CONFIG_X86_X32
	xfs_warn("XFS not supported on x32 architectures")
	return -ENOSYS;

#endif

Or, alternatively, tag it as EXPERIMENTAL and "use at your own risk".

You(r distro) can enable X32 in the x86_64 kernel even if you never use it, so this as proposed would break XFS on amd64. I'd rather just have something like this in xfs_file_ioctl, and gated on is_x32_syscall().

--D

Quoted 6 lines

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]