nvme nvme0: I/O 0 (I/O Cmd) QID 1 timeout, aborting, source drive corruption observed
I have done an fsck check on the /dev/nvme0n1p3 file system after the rsync invocation referenced earlier. In the first run I found errors which fsck should have repaired if I understand correctly. In repeating the fsck invocation immediately afterwards, I found errors again each time. This was done using the replacement Samsung nvme ssd (Samsung 970 EVO Plus 500G).
Here is the output :
-bash-3.2# fsck -f -C /dev/nvme0n1p3
fsck from util-linux 2.34
e2fsck 1.42.12 (29-Aug-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences: -87163220 -87163384 -87187960 -122079572 -122079736 Fix<y>? yes
/dev/nvme0n1p3: ***** FILE SYSTEM WAS MODIFIED *****/dev/nvme0n1p3: 699563/30523392 files (0.1% non-contiguous), 13841458/122079744 blocks -bash-3.2# fsck -f -C /dev/nvme0n1p3
fsck from util-linux 2.34
e2fsck 1.42.12 (29-Aug-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences: -87163220 -87163384 -122079736
Fix<y>? yes
/dev/nvme0n1p3: ***** FILE SYSTEM WAS MODIFIED *****/dev/nvme0n1p3: 699563/30523392 files (0.1% non-contiguous), 13841458/122079744 blocks -bash-3.2# fsck -f -C /dev/nvme0n1p3
fsck from util-linux 2.34
e2fsck 1.42.12 (29-Aug-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences: -87163220 -87163384 -122079572 -122079736
Fix<y>? yes
/dev/nvme0n1p3: ***** FILE SYSTEM WAS MODIFIED *****/dev/nvme0n1p3: 699563/30523392 files (0.1% non-contiguous), 13841458/122079744 blocks
I also tried loading a smaller partition on the Samsung card at /dev/nvme0n1p3. The copy stopped with a "no space left on device" error, which should not have been possible as the source device is a 32MB partition, and the destination partition on the nvme ssd is a 64MB partition. The two files to be transferred were very small and could not have accounted for this as they totaled less than 5MB. I found file system damage on the nvme destination partition in this case as well. It also occurred repeatedly. I am still investigating this last case.
In no instance did I note any otherwise unusual log messages or errors from the nvme driver.
I do not yet know if there has been any damage to any other filesystems, but I will check.
J. Hart