NVM and swap device

DISCUSSION Stephen Hemminger 5 participants 11 messages Jan 13, 2016 – Jan 20, 2016 NVMe

Share & tools

Stephen Hemminger

author

2016-01-13 03:40 UTC #0 lore raw

I have a nice shiny new Intel NVM PCI card; decided to use it for a filesystem and swap. The filesystem (btrfs) is doing fine, but the swap device was throwing occasional random errors. Suspect a driver problem rather than hardware.

I am using 4.4 kernel without patches.

kern.log:Jan 12 08:11:57 xeon-e3 kernel: [159474.037390] Read-error on swap-device (259:0:17597808)

kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87938.855526] Read-error on swap-device (259:0:11355648)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87938.855530] Read-error on swap-device (259:0:11355656)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87939.855467] Read-error on swap-device (259:0:16180824)
kern.log.1:Jan  8 08:24:07 xeon-e3 kernel: [63670.777981] Read-error on swap-device (259:0:32690768)
kern.log.1:Jan  9 09:25:02 xeon-e3 kernel: [153720.919325] Read-error on swap-device (259:0:220488)
kern.log.1:Jan  9 16:40:05 xeon-e3 kernel: [179820.957675] Read-error on swap-device (259:0:24476232)
kern.log.1:Jan  9 16:40:05 xeon-e3 kernel: [179820.962673] Read-error on swap-device (259:0:33292816)

The swap device was being added via /etc/fstab by UUID.

I gave up and went back to spinning rust for swap device for stabilty.

Device partitions are:

Disk /dev/nvme0n1: 781422768 sectors, 372.6 GiB Logical sector size: 512 bytes
Disk identifier (GUID): 304117A4-18EF-4B51-92F4-8015758B5CB0 Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 781422734 Partitions will be aligned on 8-sector boundaries Total free space is 0 sectors (0 bytes)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              34            2047   1007.0 KiB  EF02  BIOS boot partition
   2            2048        33556479   16.0 GiB    8200  Linux swap
   3        33556480       781422734   356.6 GiB   8300  Linux filesystem

Hannes Reinecke

2016-01-13 08:26 UTC #1 lore raw

Quoted 29 lines · Stephen Hemminger

I have a nice shiny new Intel NVM PCI card; decided to use it for a filesystem and swap. The filesystem (btrfs) is doing fine, but the swap device was throwing occasional random errors. Suspect a driver problem rather than hardware.
I am using 4.4 kernel without patches.
kern.log:Jan 12 08:11:57 xeon-e3 kernel: [159474.037390] Read-error on swap-device (259:0:17597808)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87938.855526] Read-error on swap-device (259:0:11355648)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87938.855530] Read-error on swap-device (259:0:11355656)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87939.855467] Read-error on swap-device (259:0:16180824)
kern.log.1:Jan  8 08:24:07 xeon-e3 kernel: [63670.777981] Read-error on swap-device (259:0:32690768)
kern.log.1:Jan  9 09:25:02 xeon-e3 kernel: [153720.919325] Read-error on swap-device (259:0:220488)
kern.log.1:Jan  9 16:40:05 xeon-e3 kernel: [179820.957675] Read-error on swap-device (259:0:24476232)
kern.log.1:Jan  9 16:40:05 xeon-e3 kernel: [179820.962673] Read-error on swap-device (259:0:33292816)
The swap device was being added via /etc/fstab by UUID.
I gave up and went back to spinning rust for swap device for stabilty.
Device partitions are:
Disk /dev/nvme0n1: 781422768 sectors, 372.6 GiB Logical sector size: 512 bytes
Disk identifier (GUID): 304117A4-18EF-4B51-92F4-8015758B5CB0 Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 781422734 Partitions will be aligned on 8-sector boundaries Total free space is 0 sectors (0 bytes)

Quoted 5 lines

Number  Start (sector)    End (sector)  Size       Code  Name
    1              34            2047   1007.0 KiB  EF02  BIOS boot partition
    2            2048        33556479   16.0 GiB    8200  Linux swap
    3        33556480       781422734   356.6 GiB   8300  Linux filesystem

Ouch.

34 sectors is aligned to basically nothing, and is guaranteed to trip any alignment issues there are.

Please repartition the device and use some sane value like 2M alignment.

Cheers,

Hannes

Signature · 5 lines

Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)

Matthew Wilcox

2016-01-13 15:13 UTC #2 lore raw

Quoted 35 lines · Hannes Reinecke

Quoted 29 lines · Stephen Hemminger
I have a nice shiny new Intel NVM PCI card; decided to use it for a filesystem and swap. The filesystem (btrfs) is doing fine, but the swap device was throwing occasional random errors. Suspect a driver problem rather than hardware.
I am using 4.4 kernel without patches.
kern.log:Jan 12 08:11:57 xeon-e3 kernel: [159474.037390] Read-error on swap-device (259:0:17597808)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87938.855526] Read-error on swap-device (259:0:11355648)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87938.855530] Read-error on swap-device (259:0:11355656)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87939.855467] Read-error on swap-device (259:0:16180824)
kern.log.1:Jan  8 08:24:07 xeon-e3 kernel: [63670.777981] Read-error on swap-device (259:0:32690768)
kern.log.1:Jan  9 09:25:02 xeon-e3 kernel: [153720.919325] Read-error on swap-device (259:0:220488)
kern.log.1:Jan  9 16:40:05 xeon-e3 kernel: [179820.957675] Read-error on swap-device (259:0:24476232)
kern.log.1:Jan  9 16:40:05 xeon-e3 kernel: [179820.962673] Read-error on swap-device (259:0:33292816)
The swap device was being added via /etc/fstab by UUID.
I gave up and went back to spinning rust for swap device for stabilty.
Device partitions are:
Disk /dev/nvme0n1: 781422768 sectors, 372.6 GiB Logical sector size: 512 bytes
Disk identifier (GUID): 304117A4-18EF-4B51-92F4-8015758B5CB0 Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 781422734 Partitions will be aligned on 8-sector boundaries Total free space is 0 sectors (0 bytes)
Quoted 5 lines
Number  Start (sector)    End (sector)  Size       Code  Name
   1              34            2047   1007.0 KiB  EF02  BIOS boot partition
   2            2048        33556479   16.0 GiB    8200  Linux swap
   3        33556480       781422734   356.6 GiB   8300  Linux filesystem

Quoted 4 lines

Ouch.
34 sectors is aligned to basically nothing, and is guaranteed to trip any alignment issues there are.

Maybe, but that's the BIOS boot partition; who cares? The swap partition is aligned to 2048 sectors which should be good enough for anyone (it's 1MB).

Can you run a straight 'dd' from that partition, to see if you get errors?

Jens Axboe

2016-01-13 17:47 UTC #3 lore raw

In reply to Stephen Hemminger

Quoted 15 lines · Stephen Hemminger

I have a nice shiny new Intel NVM PCI card; decided to use it for a filesystem and swap. The filesystem (btrfs) is doing fine, but the swap device was throwing occasional random errors. Suspect a driver problem rather than hardware.
I am using 4.4 kernel without patches.
kern.log:Jan 12 08:11:57 xeon-e3 kernel: [159474.037390] Read-error on swap-device (259:0:17597808)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87938.855526] Read-error on swap-device (259:0:11355648)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87938.855530] Read-error on swap-device (259:0:11355656)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87939.855467] Read-error on swap-device (259:0:16180824)
kern.log.1:Jan  8 08:24:07 xeon-e3 kernel: [63670.777981] Read-error on swap-device (259:0:32690768)
kern.log.1:Jan  9 09:25:02 xeon-e3 kernel: [153720.919325] Read-error on swap-device (259:0:220488)
kern.log.1:Jan  9 16:40:05 xeon-e3 kernel: [179820.957675] Read-error on swap-device (259:0:24476232)
kern.log.1:Jan  9 16:40:05 xeon-e3 kernel: [179820.962673] Read-error on swap-device (259:0:33292816)

Quoted 3 lines

The swap device was being added via /etc/fstab by UUID.
I gave up and went back to spinning rust for swap device for stabilty.

That's very odd. Why are you suspecting a driver problem? Have you tried to thoroughly beat the device up with normal IO?

Signature · 1 line

Jens Axboe

Stephen Hemminger

author

2016-01-13 18:51 UTC #4 lore raw

Quoted 16 lines · On Wed, 13 Jan 2016 10:47:40 -0700 Jens Axboe <[email protected]> wrote:

Quoted 15 lines · Stephen Hemminger
I have a nice shiny new Intel NVM PCI card; decided to use it for a filesystem and swap. The filesystem (btrfs) is doing fine, but the swap device was throwing occasional random errors. Suspect a driver problem rather than hardware.
I am using 4.4 kernel without patches.
kern.log:Jan 12 08:11:57 xeon-e3 kernel: [159474.037390] Read-error on swap-device (259:0:17597808)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87938.855526] Read-error on swap-device (259:0:11355648)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87938.855530] Read-error on swap-device (259:0:11355656)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87939.855467] Read-error on swap-device (259:0:16180824)
kern.log.1:Jan  8 08:24:07 xeon-e3 kernel: [63670.777981] Read-error on swap-device (259:0:32690768)
kern.log.1:Jan  9 09:25:02 xeon-e3 kernel: [153720.919325] Read-error on swap-device (259:0:220488)
kern.log.1:Jan  9 16:40:05 xeon-e3 kernel: [179820.957675] Read-error on swap-device (259:0:24476232)
kern.log.1:Jan  9 16:40:05 xeon-e3 kernel: [179820.962673] Read-error on swap-device (259:0:33292816)

Quoted 7 lines

Quoted 3 lines
The swap device was being added via /etc/fstab by UUID.
I gave up and went back to spinning rust for swap device for stabilty.
That's very odd. Why are you suspecting a driver problem? Have you tried to thoroughly beat the device up with normal IO?

I will try it tonight. Do you have a favorite test?

Jens Axboe

2016-01-13 18:55 UTC #5 lore raw

Quoted 21 lines · Stephen Hemminger

Quoted 18 lines · On Wed, 13 Jan 2016 10:47:40 -0700 Jens Axboe <[email protected]> wrote:
Quoted 17 lines · Stephen Hemminger
I have a nice shiny new Intel NVM PCI card; decided to use it for a filesystem and swap. The filesystem (btrfs) is doing fine, but the swap device was throwing occasional random errors. Suspect a driver problem rather than hardware.
I am using 4.4 kernel without patches.
kern.log:Jan 12 08:11:57 xeon-e3 kernel: [159474.037390] Read-error on swap-device (259:0:17597808)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87938.855526] Read-error on swap-device (259:0:11355648)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87938.855530] Read-error on swap-device (259:0:11355656)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87939.855467] Read-error on swap-device (259:0:16180824)
kern.log.1:Jan  8 08:24:07 xeon-e3 kernel: [63670.777981] Read-error on swap-device (259:0:32690768)
kern.log.1:Jan  9 09:25:02 xeon-e3 kernel: [153720.919325] Read-error on swap-device (259:0:220488)
kern.log.1:Jan  9 16:40:05 xeon-e3 kernel: [179820.957675] Read-error on swap-device (259:0:24476232)
kern.log.1:Jan  9 16:40:05 xeon-e3 kernel: [179820.962673] Read-error on swap-device (259:0:33292816)
The swap device was being added via /etc/fstab by UUID.

Quoted 7 lines

Quoted 5 lines
Quoted 1 line
I gave up and went back to spinning rust for swap device for stabilty.
That's very odd. Why are you suspecting a driver problem? Have you tried to thoroughly beat the device up with normal IO?
I will try it tonight. Do you have a favorite test?

I'd run something that just beats up on it, reads and writes. If you have fio installed, something ala:

fio --ioengine=libaio --iodepth=8 --direct=1 --bs=4k --filename=/dev/nvme0n1 --numjobs=4 --norandommap --runtime=1h --time_based=1 --name=reads --rw=randread --name=writes --rw=randwrite

This will run 4 processes that randomly read from the device, and 4 that randomly write. Replace /dev/nvme0n1 with your swap partition. The test will run for 1 hour.

Signature · 1 line

Jens Axboe

Stephen Hemminger

author

2016-01-20 21:30 UTC #6 lore raw

Quoted 10 lines · On Wed, 13 Jan 2016 11:55:41 -0700 Jens Axboe <[email protected]> wrote:

I'd run something that just beats up on it, reads and writes. If you have fio installed, something ala:
fio --ioengine=libaio --iodepth=8 --direct=1 --bs=4k --filename=/dev/nvme0n1 --numjobs=4 --norandommap --runtime=1h --time_based=1 --name=reads --rw=randread --name=writes --rw=randwrite
This will run 4 processes that randomly read from the device, and 4 that randomly write. Replace /dev/nvme0n1 with your swap partition. The test will run for 1 hour.

The fio test ran fine without errors. I am beginning to think it something unique to how I/O in swap path gets done.

Will turn it back on for swap and see what happens.

Keith Busch

2016-01-15 17:42 UTC #7 lore raw

In reply to Stephen Hemminger

Quoted 15 lines · Stephen Hemminger

I have a nice shiny new Intel NVM PCI card; decided to use it for a filesystem and swap. The filesystem (btrfs) is doing fine, but the swap device was throwing occasional random errors. Suspect a driver problem rather than hardware.
I am using 4.4 kernel without patches.
kern.log:Jan 12 08:11:57 xeon-e3 kernel: [159474.037390] Read-error on swap-device (259:0:17597808)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87938.855526] Read-error on swap-device (259:0:11355648)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87938.855530] Read-error on swap-device (259:0:11355656)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87939.855467] Read-error on swap-device (259:0:16180824)
kern.log.1:Jan  8 08:24:07 xeon-e3 kernel: [63670.777981] Read-error on swap-device (259:0:32690768)
kern.log.1:Jan  9 09:25:02 xeon-e3 kernel: [153720.919325] Read-error on swap-device (259:0:220488)
kern.log.1:Jan  9 16:40:05 xeon-e3 kernel: [179820.957675] Read-error on swap-device (259:0:24476232)
kern.log.1:Jan  9 16:40:05 xeon-e3 kernel: [179820.962673] Read-error on swap-device (259:0:33292816)

Quoted 1 line

The swap device was being added via /etc/fstab by UUID.

If you haven't any further insights into the issue, could you check the device's health? I'd be surprised if there is a problem with that since you mentioned it was new card, but would like to rule that out if this has hit a dead end on the other testing.

For that, we need smart logs. There are various tools available that can read those logs. Here's an open source version:

https://github.com/linux-nvme/nvme-cli

Here's example output from one of my drives with the above tool:

  # nvme smart-log /dev/nvme0
  Smart Log for NVME device:/dev/nvme0 namespace-id:ffffffff
  critical_warning                    : 0
  temperature                         : 29 C
  available_spare                     : 100%
  available_spare_threshold           : 10%
  percentage_used                     : 0%
  data_units_read                     : 577,600
  data_units_written                  : 3,182,404
  host_read_commands                  : 4,537,801
  host_write_commands                 : 18,713,235
  controller_busy_time                : 17
  power_cycles                        : 1
  power_on_hours                      : 163
  unsafe_shutdowns                    : 1
  media_errors                        : 0
  num_err_log_entries                 : 0

Stephen Hemminger

author

2016-01-15 18:18 UTC #8 lore raw

Quoted 39 lines · On Fri, 15 Jan 2016 17:42:36 +0000 Keith Busch <[email protected]> wrote:

Quoted 15 lines · Stephen Hemminger
I have a nice shiny new Intel NVM PCI card; decided to use it for a filesystem and swap. The filesystem (btrfs) is doing fine, but the swap device was throwing occasional random errors. Suspect a driver problem rather than hardware.
I am using 4.4 kernel without patches.
kern.log:Jan 12 08:11:57 xeon-e3 kernel: [159474.037390] Read-error on swap-device (259:0:17597808)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87938.855526] Read-error on swap-device (259:0:11355648)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87938.855530] Read-error on swap-device (259:0:11355656)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87939.855467] Read-error on swap-device (259:0:16180824)
kern.log.1:Jan  8 08:24:07 xeon-e3 kernel: [63670.777981] Read-error on swap-device (259:0:32690768)
kern.log.1:Jan  9 09:25:02 xeon-e3 kernel: [153720.919325] Read-error on swap-device (259:0:220488)
kern.log.1:Jan  9 16:40:05 xeon-e3 kernel: [179820.957675] Read-error on swap-device (259:0:24476232)
kern.log.1:Jan  9 16:40:05 xeon-e3 kernel: [179820.962673] Read-error on swap-device (259:0:33292816)
Quoted 1 line
The swap device was being added via /etc/fstab by UUID.
If you haven't any further insights into the issue, could you check the device's health? I'd be surprised if there is a problem with that since you mentioned it was new card, but would like to rule that out if this has hit a dead end on the other testing.
For that, we need smart logs. There are various tools available that can read those logs. Here's an open source version:
https://github.com/linux-nvme/nvme-cli
Here's example output from one of my drives with the above tool:
  # nvme smart-log /dev/nvme0
  Smart Log for NVME device:/dev/nvme0 namespace-id:ffffffff
  critical_warning                    : 0
  temperature                         : 29 C
  available_spare                     : 100%
  available_spare_threshold           : 10%
  percentage_used                     : 0%
  data_units_read                     : 577,600
  data_units_written                  : 3,182,404

Quoted 8 lines

  host_read_commands                  : 4,537,801
  host_write_commands                 : 18,713,235
  controller_busy_time                : 17
  power_cycles                        : 1
  power_on_hours                      : 163
  unsafe_shutdowns                    : 1
  media_errors                        : 0
  num_err_log_entries                 : 0

I wanted to run more stress tests before reporting back.

Stephen Hemminger

author

2016-01-15 18:20 UTC #9 lore raw

In reply to Keith Busch

Quoted 39 lines · On Fri, 15 Jan 2016 17:42:36 +0000 Keith Busch <[email protected]> wrote:

Quoted 15 lines · Stephen Hemminger
I have a nice shiny new Intel NVM PCI card; decided to use it for a filesystem and swap. The filesystem (btrfs) is doing fine, but the swap device was throwing occasional random errors. Suspect a driver problem rather than hardware.
I am using 4.4 kernel without patches.
kern.log:Jan 12 08:11:57 xeon-e3 kernel: [159474.037390] Read-error on swap-device (259:0:17597808)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87938.855526] Read-error on swap-device (259:0:11355648)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87938.855530] Read-error on swap-device (259:0:11355656)
kern.log.1:Jan  7 08:32:10 xeon-e3 kernel: [87939.855467] Read-error on swap-device (259:0:16180824)
kern.log.1:Jan  8 08:24:07 xeon-e3 kernel: [63670.777981] Read-error on swap-device (259:0:32690768)
kern.log.1:Jan  9 09:25:02 xeon-e3 kernel: [153720.919325] Read-error on swap-device (259:0:220488)
kern.log.1:Jan  9 16:40:05 xeon-e3 kernel: [179820.957675] Read-error on swap-device (259:0:24476232)
kern.log.1:Jan  9 16:40:05 xeon-e3 kernel: [179820.962673] Read-error on swap-device (259:0:33292816)
Quoted 1 line
The swap device was being added via /etc/fstab by UUID.
If you haven't any further insights into the issue, could you check the device's health? I'd be surprised if there is a problem with that since you mentioned it was new card, but would like to rule that out if this has hit a dead end on the other testing.
For that, we need smart logs. There are various tools available that can read those logs. Here's an open source version:
https://github.com/linux-nvme/nvme-cli
Here's example output from one of my drives with the above tool:
  # nvme smart-log /dev/nvme0
  Smart Log for NVME device:/dev/nvme0 namespace-id:ffffffff
  critical_warning                    : 0
  temperature                         : 29 C
  available_spare                     : 100%
  available_spare_threshold           : 10%
  percentage_used                     : 0%
  data_units_read                     : 577,600
  data_units_written                  : 3,182,404

Quoted 8 lines

  host_read_commands                  : 4,537,801
  host_write_commands                 : 18,713,235
  controller_busy_time                : 17
  power_cycles                        : 1
  power_on_hours                      : 163
  unsafe_shutdowns                    : 1
  media_errors                        : 0
  num_err_log_entries                 : 0

Are these logs persistant?

# ./nvme smart-log /dev/nvme0
Smart Log for NVME device:/dev/nvme0 namespace-id:ffffffff
critical_warning                    : 0
temperature                         : 29 C
available_spare                     : 100%
available_spare_threshold           : 10%
percentage_used                     : 0%
data_units_read                     : 2,768,267
data_units_written                  : 4,085,497
host_read_commands                  : 74,458,451
host_write_commands                 : 68,420,945
controller_busy_time                : 12
power_cycles                        : 33
power_on_hours                      : 227
unsafe_shutdowns                    : 6
media_errors                        : 0
num_err_log_entries                 : 0
Critical Composite Temperature Time : 0
Temperature Sensor 1                : 0 C
Temperature Sensor 2                : 0 C
Temperature Sensor 3                : 0 C
Temperature Sensor 4                : 0 C
Temperature Sensor 5                : 0 C
Temperature Sensor 6                : 0 C
Temperature Sensor 7                : 0 C
Temperature Sensor 8                : 0 C

Keith Busch

2016-01-15 21:19 UTC #10 lore raw

Quoted 1 line · Stephen Hemminger

Are these logs persistant?

Yep, these are persistent logs.

Nothing unusual here. Sorry for the digression.

Quoted 15 lines

critical_warning                    : 0
temperature                         : 29 C
available_spare                     : 100%
available_spare_threshold           : 10%
percentage_used                     : 0%
data_units_read                     : 2,768,267
data_units_written                  : 4,085,497
host_read_commands                  : 74,458,451
host_write_commands                 : 68,420,945
controller_busy_time                : 12
power_cycles                        : 33
power_on_hours                      : 227
unsafe_shutdowns                    : 6
media_errors                        : 0
num_err_log_entries                 : 0