Fix some 4-byte vs. 8-byte alignment issues in atomic bit operations
Subsystems (6)
v2 cover Fix some 4-byte vs. 8-byte alignment issues in atomic bit operations
A split lock is an unaligned atomic operation that spans two cache lines. The split lock access needs to hold bus lock and will degrade performance.
Accessing one split lock long can take over one thousand extra cycles than atomically accessing one unaligned long within one cache line. And while the split lock access holds the bus lock, any other memory accesses are not allowed and the overall memory access performance is degraded.
Because badly performance impact by split lock, this patch series solve the split lock issues instead of other alignment issues.
These parts are all simple fixes which are a necessary pre-cursor before we can enable #AC traps for split lock access. But they are also worthwhile performance fixes in their own right. So no sense in holding them back while we discuss the merits of the rest of the series.
The alignment issues may be fixed by changing the atomic bit operations APIs e.g. new set_bit_32() for 4-byte alignment. But the fixes will be complex because they touch a lot of call sites and architectures.
Change Log:
v2:
- Remove patch 1 and 3 in v1 because they are in the tip tree already.
- Add new patches 2-4 per David Laight's comments:
https://lore.kernel.org/lkml/[email protected]/
Running "grep -r --include '*.[ch]' '_bit([^(]*, *([^)]*\*)' ." returns about 200 results. Most of them don't have split lock issues.
Fenghua Yu (3):
xen-pcifront: Align address of flags to size of unsigned long
mtd: rawnand: fsmc: Change to non-atomic bit operations x86/mce: Force alignment for atomic bit operations
Peter Zijlstra (1):
drivers/net/b44: Change to non-atomic bit operations
arch/x86/include/asm/mce.h | 3 ++- drivers/mtd/nand/raw/fsmc_nand.c | 4 ++-- drivers/net/ethernet/broadcom/b44.c | 4 ++-- include/xen/interface/io/pciif.h | 7 +++++-- 4 files changed, 11 insertions(+), 7 deletions(-)
Signature · 1 line
2.19.1
PATCH v2 1/4 drivers/net/b44: Change to non-atomic bit operations
From: Peter Zijlstra <[email protected]>
Since "pwol_mask" is local and never exposed to concurrency, there is no need to set bit in pwol_mask by costly atomic operations.
--- drivers/net/ethernet/broadcom/b44.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/ethernet/broadcom/b44.c +2 -2
@@ -1520,7 +1520,7 @@ static int b44_magic_pattern(u8 *macaddr, u8 *ppattern, u8 *pmask, int offset)memset (ppattern + offset,0xff , magicsync);for (j =0 ; j < magicsync; j++) -set_bit (len++, (unsigned long *) pmask); +__set_bit (len++, (unsigned long *)pmask);for (j =0 ; j <B44_MAX_PATTERNS ; j++) {if ((B44_PATTERN_SIZE - len) >=ETH_ALEN )
@@ -1532,7 +1532,7 @@ static int b44_magic_pattern(u8 *macaddr, u8 *ppattern, u8 *pmask, int offset)for (k =0 ; k< ethaddr_bytes; k++) { ppattern[offset + magicsync + (j *ETH_ALEN ) + k] = macaddr[k]; -set_bit (len++, (unsigned long *) pmask); +__set_bit (len++, (unsigned long *)pmask); } }return len -1 ;
Signature · 1 line
2.19.1
RE: [PATCH v2 1/4] drivers/net/b44: Change to non-atomic bit operations
From: Fenghua Yu
Quoted 17 lines
Sent: 25 November 2019 19:43
From: Peter Zijlstra <[email protected]>Since "
pwol_mask" is local and never exposed to concurrency, there is no need to set bit inpwol_maskby costly atomic operations.Signed-off-by Peter Zijlstra <[email protected]>Signed-off-by Fenghua Yu <[email protected]>--- drivers/net/ethernet/broadcom/b44.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/net/ethernet/broadcom/b44.c +0
-0
drivers/net/ethernet/broadcom/b44.c static int b44_magic_pattern(u8 *macaddr, u8 *ppattern, u8 *pmask, int offset) · line 1523
Quoted 5 lines
drivers/net/ethernet/broadcom/b44.c +1
-1@@ -1520,7 +1520,7 @@ static int b44_magic_pattern(u8 *macaddr, u8 *ppattern, u8 *pmask, int offset)memset (ppattern + offset,0xff , magicsync);for (j =0 ; j < magicsync; j++) -set_bit (len++, (unsigned long *) pmask); +__set_bit (len++, (unsigned long *)pmask);
While this stops the misaligned locks, the code is still horribly borked on BE systems. The way pmask is used definitely wanst a u32[] not a u64[] one.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Re: [PATCH v2 1/4] drivers/net/b44: Change to non-atomic bit operations
Quoted 6 lines · David Laight
drivers/net/ethernet/broadcom/b44.c static int b44_magic_pattern(u8 *macaddr, u8 *ppattern, u8 *pmask, int offset) · line 1523
Quoted 4 lines
drivers/net/ethernet/broadcom/b44.c +1
-1memset (ppattern + offset,0xff , magicsync);for (j =0 ; j < magicsync; j++) -set_bit (len++, (unsigned long *) pmask); +__set_bit (len++, (unsigned long *)pmask);While this stops the misaligned locks, the code is still horribly borked on BE systems.
Quite so.
Quoted 1 line
The way pmask is used definitely wanst a u32[] not a u64[] one.
Not sure, the code seems fairly consistent in using u8[].
And I suppose we can write the above like:
pmask[len >> 3] |= BIT(len & 7); len++;
instead.
PATCH v2 2/4 xen-pcifront: Align address of flags to size of unsigned long
In reply to Fenghua YuThe address of "flags" is passed to atomic bitops which require the address is aligned to size of unsigned long.
--- include/xen/interface/io/pciif.h | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
include/xen/interface/io/pciif.h +5 -2
@@ -103,8 +103,11 @@ struct xen_pcie_aer_op {uint32_t devfn; };struct xen_pci_sharedinfo { -/* flags - XEN_PCIF_* */ -uint32_t flags; +/* flags - XEN_PCIF_*. Force alignment for atomic bit operations. */ +union { +uint32_t flags; +unsigned long flags_alignment; + };struct xen_pci_op op;struct xen_pcie_aer_op aer_op; };
Signature · 1 line
2.19.1
RE: [PATCH v2 2/4] xen-pcifront: Align address of flags to size of unsigned long
From: Fenghua Yu <[email protected]>
Quoted 14 lines
Sent: 25 November 2019 19:43
The address of "flags" is passed to atomic bitops which require the address is aligned to size of unsigned long.Signed-off-by Fenghua Yu <[email protected]>--- include/xen/interface/io/pciif.h | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)include/xen/interface/io/pciif.h +0
-0
include/xen/interface/io/pciif.h struct xen_pcie_aer_op { · line 110
Quoted 10 lines
include/xen/interface/io/pciif.h +5
-2@@ -103,8 +103,11 @@ struct xen_pcie_aer_op {uint32_t devfn; };struct xen_pci_sharedinfo { -/* flags - XEN_PCIF_* */ -uint32_t flags; +/* flags - XEN_PCIF_*. Force alignment for atomic bit operations. */ +union { +uint32_t flags; +unsigned long flags_alignment; + };
This is papering over the cracks.....
Changing flags to be 'unsigned long' and removing the casts is correct. Either that or change to code to use simple assignments on the single 'flags' variable instead of the relatively expensive function calls that are designed for large arrays of bits.
David.
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Re: [PATCH v2 2/4] xen-pcifront: Align address of flags to size of unsigned long
In reply to Fenghua YuQuoted 17 lines · Fenghua Yu
The address of "flags" is passed to atomic bitops which require the address is aligned to size of unsigned long.
Signed-off-by Fenghua Yu <[email protected]>--- include/xen/interface/io/pciif.h | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)include/xen/interface/io/pciif.h +0
-0@@ -103,8 +103,11 @@ struct xen_pcie_aer_op {uint32_t devfn; };struct xen_pci_sharedinfo {
include/xen/interface/io/pciif.h struct xen_pcie_aer_op { · line 113
Quoted 10 lines
include/xen/interface/io/pciif.h +5
-2-/* flags - XEN_PCIF_* */ -uint32_t flags; +/* flags - XEN_PCIF_*. Force alignment for atomic bit operations. */ +union { +uint32_t flags; +unsigned long flags_alignment; + };struct xen_pci_op op;struct xen_pcie_aer_op aer_op; };
NAK.
This is an interface definition for communication between Xen dom0 and guests via shared memory. It can't be changed.
BTW: you should Cc: the maintainers for the files you are modifying.
Juergen
Re: [PATCH v2 2/4] xen-pcifront: Align address of flags to size of unsigned long
Quoted 31 lines · Jürgen Groß
Quoted 17 lines · Fenghua Yu
The address of "flags" is passed to atomic bitops which require the address is aligned to size of unsigned long.
Signed-off-by Fenghua Yu <[email protected]>--- include/xen/interface/io/pciif.h | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)include/xen/interface/io/pciif.h +0
-0@@ -103,8 +103,11 @@ struct xen_pcie_aer_op {uint32_t devfn; };struct xen_pci_sharedinfo {include/xen/interface/io/pciif.h struct xen_pcie_aer_op { · line 113
Quoted 10 lines
include/xen/interface/io/pciif.h +5
-2-/* flags - XEN_PCIF_* */ -uint32_t flags; +/* flags - XEN_PCIF_*. Force alignment for atomic bit operations. */ +union { +uint32_t flags; +unsigned long flags_alignment; + };struct xen_pci_op op;struct xen_pcie_aer_op aer_op; };NAK.
Quoted 2 lines
This is an interface definition for communication between Xen dom0 and guests via shared memory. It can't be changed.
Bother. There doesn't seem to be anything else in the xen_pci_sharedinfo structure (or any of its sub-components) that would force the whole thing to be any more strictly aligned than uint32_t
Oh ... but it looks like this is the only allocation:
pdev->sh_info =
(struct xen_pci_sharedinfo *)__get_free_page(GFP_KERNEL);so "flags" is page aligned.
Fenghua: drop this patch, it isn't needed.
-Tony
Re: [PATCH v2 2/4] xen-pcifront: Align address of flags to size of unsigned long
In reply to Jürgen GroßQuoted 25 lines · Jürgen Groß
Quoted 8 lines · Fenghua Yu
The address of "flags" is passed to atomic bitops which require the address is aligned to size of unsigned long.
Signed-off-by Fenghua Yu <[email protected]> ---
include/xen/interface/io/pciif.h| 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)(quoted diff) +0
-0Quoted 13 lines
b/include/xen/interface/io/pciif.hindexd9922ae36eb5..639d5fb484a3100644 ---a/include/xen/interface/io/pciif.h+++b/include/xen/interface/io/pciif.h@@ -103,8 +103,11 @@ structxen_pcie_aer_op{
uint32_tdevfn;
};
structxen_pci_sharedinfo{ - /* flags - XEN_PCIF_* */ -uint32_t
flags; + /* flags - XEN_PCIF_*. Force alignment for atomic bit operations. */ + union { +uint32_tflags; + unsigned longflags_alignment; + };
structxen_pci_opop; structxen_pcie_aer_opaer_op;
};NAK.
Quoted 7 lines
This is an interface definition for communication between Xen dom0 and guests via shared memory. It can't be changed.
BTW: you should Cc: the maintainers for the files you are modifying.
Juergen
After spending more time on the patch, actually I find this patch is not needed because there is no split lock issue on the "flags" field:
The bit offsets in all atomic bitops on the "flags" field are always constant values. On x86, an atomic bitop is optimized to use a locked byte instruciton which only operates the byte containing the bit. So the byte won't across cache line boundaries and there won't be split lock issue in the atomic bitops
I will remove this patch in the next version of the patch set.
Thanks.
-Fenghua
PATCH v2 3/4 mtd: rawnand: fsmc: Change to non-atomic bit operations
In reply to Fenghua YuNo need for expensive atomic change_bit() on the local variable err_idx[].
--- drivers/mtd/nand/raw/fsmc_nand.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/mtd/nand/raw/fsmc_nand.c +2 -2
@@ -809,8 +809,8 @@ static int fsmc_bch8_correct_data(struct nand_chip *chip, u8 *dat, i =0 ;while (num_err--) { -change_bit (0 , (unsigned long *)&err_idx[i]); -change_bit (1 , (unsigned long *)&err_idx[i]); +__change_bit (0 , (unsigned long *)&err_idx[i]); +__change_bit (1 , (unsigned long *)&err_idx[i]);if (err_idx[i] < chip->ecc.size *8 ) {change_bit (err_idx[i], (unsigned long *)dat);
Signature · 1 line
2.19.1
Re: [PATCH v2 3/4] mtd: rawnand: fsmc: Change to non-atomic bit operations
Quoted 12 lines · Fenghua Yu
No need for expensive atomic
change_bit()on the local variable err_idx[].Signed-off-by Fenghua Yu <[email protected]>--- drivers/mtd/nand/raw/fsmc_nand.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/mtd/nand/raw/fsmc_nand.c +0
-0
drivers/mtd/nand/raw/fsmc_nand.c static int fsmc_bch8_correct_data(struct nand_chip *chip, u8 *dat, · line 816
Quoted 10 lines
drivers/mtd/nand/raw/fsmc_nand.c +2
-2@@ -809,8 +809,8 @@ static int fsmc_bch8_correct_data(struct nand_chip *chip, u8 *dat, i =0 ;while (num_err--) { -change_bit (0 , (unsigned long *)&err_idx[i]); -change_bit (1 , (unsigned long *)&err_idx[i]); +__change_bit (0 , (unsigned long *)&err_idx[i]); +__change_bit (1 , (unsigned long *)&err_idx[i]);if (err_idx[i] < chip->ecc.size *8 ) {change_bit (err_idx[i], (unsigned long *)dat);
I'm thinking this all can be written like:
err_idx[i] ^= 3;
if (err_idx[i] < chip->ecc.size * 8) {
err = err_idx[i];
dat[err >> 3] ^= BIT(err & 7);
}It seems unlikely that the @dat we're correcting has concurrency issues.
PATCH v2 4/4 x86/mce: Force alignment for atomic bit operations
In reply to Fenghua YuThe "flags" field is passed to atomic bit operations which require unsigned long alignment.
--- arch/x86/include/asm/mce.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/include/asm/mce.h +2 -1
@@ -134,10 +134,11 @@* is set. */ struct mce_log_buffer { +/* Force alignment for atomic bit operations. */ +unsigned flags;char signature[12 ];/* "MACHINECHECK" */ unsigned len;/* = MCE_LOG_LEN */ unsigned next; -unsigned flags;unsigned recordlen;/* length of struct mce */ struct mce entry[MCE_LOG_LEN ]; };
Signature · 1 line
2.19.1
From: Fenghua Yu
Quoted 1 line
Sent: 25 November 2019 19:43
...
Quoted 7 lines
Change Log:
v2:
- Remove patch 1 and 3 in v1 because they are in the tip tree already.
- Add new patches 2-4 per David Laight's comments:
https://lore.kernel.org/lkml/[email protected]/
Running "grep -r --include '*.[ch]' '_bit([^(]*, *([^)]*\*)' ." returns about 200 results. Most of them don't have split lock issues.
Except that any code that has to cast the long[] array argument to any of the bitops functions is almost certainly broken in some way or other. At best it is relying on accidental alignment of whatever address it is passing to avoid alignment faults (for general misaligned accesses).
In other cases attempting to run the code on a BE system will update the wrong bits - potentially in some other member of the structure.
Now, you might say that 'this code is LE only', but at some point it might get used on a BE system.
So all of those 200 results need fixing to remove the casts. Just using the unblocked __bit_xxx() functions doesn't fix all the problems.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)