Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot)

Hi,

My raid 10 array (5 disk with one spare) was doing a resync after an upgrade to 4.4.0 from 4.1.15. The resync progress was steady and at the end the /proc/mdstat was apparently complete but when rebooting, it started resycing over and over. I noticed my dmesg was totally filled with raid10 conf printout message so it was impossible to trace anything else.

Did a resync test with 3.14.58 (because I knew it had worked for resync before and was still available as a boot option) and the array was correctly rebuild. Runs fine with 4.1.16 now.

Please CC me as I'm not subscribed.

-- eric

Could you please provide more info, like mdadm -D /dev/md0 in v4.4? If you run a stop/reassemble, does the resync start?

Thanks,
Shaohua

I'm not going to retry kernel 4.4.0 on this device as I'm no more confident about raid10 support with this 4.4 version.

So on 4.1.16 :

mdadm -D /dev/md0
/dev/md0:

        Version : 1.2

Creation Time : Wed Jun 20 23:56:59 2012
Raid Level : raid10
Array Size : 5860268032 (5588.79 GiB 6000.91 GB)
Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB)
Raid Devices : 4
Total Devices : 5
Persistence : Superblock is persistent

Update Time : Sun Jan 24 17:15:58 2016
State : clean
Active Devices : 4
Working Devices : 5
Failed Devices : 0
Spare Devices : 1

Layout : near=2
Chunk Size : 512K

Name : nas2:0 (local to host nas2) UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928
Events : 480

Number Major Minor RaidDevice State

0 8 17 0 active sync set-A /dev/sdb1

1 8 33 1 active sync set-B /dev/sdc1

2 8 49 2 active sync set-A /dev/sdd1

3 8 65 3 active sync set-B /dev/sde1

4 8 81 - spare /dev/sdf1

uname -a
Linux nas2 4.1.16 #1 SMP Sat Jan 23 19:29:59 CET 2016 x86_64 GNU/Linux

And if I stop the array manually and reboot, its no more resynced with this 4.1.16 kernel.

Thanks! I know switching to a kernel which is not working well is risky, but the info from 4.1 doesn't have enough info for debuging. I also tried to reproduce the issue locally, but no success. Did you have any other info which could help debugging, for example special config?

Thanks,
Shaohua

If really needed can rebuild a 4.4 kernel and boot it. What do you mean by special config? I can send you my kernel .config once rebuild (oldconfig from 4.1.16)

But I doubt the problem will show up now that the array is correctly rebuild. From memory I used 4.4 kernel without problem for a few days, until it decided to resync the array for an unknown reason (standard periodic rebuild, disk block read/write error detected, other?).

NB: I have a disk on the array with 56 sectors in error reported by smart and noticed read error/SATA reset sequence during first array rebuild sequence, but not in any later resync afterward. It did not even decided to use the spare disk...

Let me know how I can help debugging further...

-- eric

Linux nas2 4.4.1 #1 SMP Sun Jan 31 23:46:26 CET 2016 x86_64 GNU/Linux

mdadm -D /dev/md0
/dev/md0:

        Version : 1.2

Creation Time : Wed Jun 20 23:56:59 2012
Raid Level : raid10
Array Size : 5860268032 (5588.79 GiB 6000.91 GB)
Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB)
Raid Devices : 4
Total Devices : 5
Persistence : Superblock is persistent

Update Time : Sun Jan 31 23:51:44 2016
State : active, resyncing
Active Devices : 4
Working Devices : 5
Failed Devices : 0
Spare Devices : 1

Layout : near=2
Chunk Size : 512K

Resync Status : 0% complete

Name : nas2:0 (local to host nas2) UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928
Events : 488

Number Major Minor RaidDevice State
0 8 17 0 active sync set-A /dev/sdb1
1 8 33 1 active sync set-B /dev/sdc1
2 8 49 2 active sync set-A /dev/sdd1
3 8 65 3 active sync set-B /dev/sde1

4 8 81 - spare /dev/sdf1

So it starts again rechecking the array

mdadm -E /dev/sd[b-f]1
/dev/sdb1:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928
           Name : nas2:0  (local to host nas2)

Creation Time : Wed Jun 20 23:56:59 2012
Raid Level : raid10
Raid Devices : 4

Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB)

     Array Size : 5860268032 (5588.79 GiB 6000.91 GB)

Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=911 sectors
State : active
Device UUID : ce9d878a:37a4f3a3:936bd905:c4ed9970

Update Time : Sun Jan 31 23:56:38 2016
Checksum : c92206ec - correct
Events : 489

Layout : near=2
Chunk Size : 512K

Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928
           Name : nas2:0  (local to host nas2)

Creation Time : Wed Jun 20 23:56:59 2012
Raid Level : raid10
Raid Devices : 4

Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB)

     Array Size : 5860268032 (5588.79 GiB 6000.91 GB)
  Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=911 sectors
          State : active
    Device UUID : 8c89bdf8:4f3f8ace:c15b5634:7a874071

    Update Time : Sun Jan 31 23:56:38 2016
       Checksum : 911217a8 - correct
         Events : 489

         Layout : near=2
     Chunk Size : 512K

Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928
           Name : nas2:0  (local to host nas2)

Creation Time : Wed Jun 20 23:56:59 2012
Raid Level : raid10
Raid Devices : 4

Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB)

     Array Size : 5860268032 (5588.79 GiB 6000.91 GB)

Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=911 sectors
State : active
Device UUID : 9c0655c3:dca7722e:8ecc1a17:fb4d9089

Update Time : Sun Jan 31 23:56:38 2016
Checksum : b6a6bcd3 - correct
Events : 489

Layout : near=2
Chunk Size : 512K

Device Role : Active device 2
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928
           Name : nas2:0  (local to host nas2)

Creation Time : Wed Jun 20 23:56:59 2012
Raid Level : raid10
Raid Devices : 4

Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB)

     Array Size : 5860268032 (5588.79 GiB 6000.91 GB)
  Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=911 sectors
          State : active
    Device UUID : c97b767a:84d2e7e2:52557d30:51c39784

    Update Time : Sun Jan 31 23:56:38 2016
       Checksum : 36a6d284 - correct
         Events : 489

         Layout : near=2
     Chunk Size : 512K

Device Role : Active device 3
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928
           Name : nas2:0  (local to host nas2)

Creation Time : Wed Jun 20 23:56:59 2012
Raid Level : raid10
Raid Devices : 4

Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB)

     Array Size : 5860268032 (5588.79 GiB 6000.91 GB)

Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=911 sectors
State : active
Device UUID : 82667e81:a6158319:85e0282e:845eec1c

Update Time : Sun Jan 31 23:51:44 2016
Checksum : a4a259e - correct
Events : 488

Layout : near=2
Chunk Size : 512K

Device Role : spare
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
root@nas2:~#

-- eric

Nothing useful. Did you try stop the array and reassemble it in 4.4 and check if resync runs? I can only imagine the reboot script didn't clean shutdown the array and trigger the issue. A stop/reassemble circle will tell us if this is the issue.

Thanks,
Shaohua

If the issue was in the script it would happen with all the kernel version no? Si far 3.14, 3.18, 4.1 works correctly (when upgrading). Only 4.4 fails.

4.4 in fact seems to fails to boot the first time I boot it. As I have no console normally, I have to shut down via power button and then it resync. I can understand this behavior but not
1) that the resync fails each time and
2) that it fails to boot the first time I change for a 4.4 kernel.

-- eric