Skip to contentSkip to conversation

NVMe scalability issue

DISCUSSION Ming Lin 6 participants 13 messages Jun 1, 2015 – Jun 2, 2015 NVMe

lore
Share & tools
lore

Ming Lin

author
#0 lore raw

Hi list,

I'm playing with 8 high performance NVMe devices on a 4 sockets server. Each device can get 730K 4k read IOPS.

Kernel: 4.1-rc3
fio test shows it doesn't scale well with 4 or more devices. I wonder any possible direction to improve it.

devices theory actual

		IOPS(K)		IOPS(K)

------- ------- -------
1 733 733
2 1466 1446.8
3 2199 2174.5
4 2932 2354.9
5 3665 3024.5
6 4398 3818.9
7 5131 4526.3
8 5864 4621.2

And a graph here:
http://minggr.net/pub/20150601/nvme-scalability.jpg

With 8 devices, CPU is still 43% idle, so CPU is not the bottleneck.

"top" data

Tasks: 565 total, 30 running, 535 sleeping, 0 stopped, 0 zombie
%Cpu(s): 17.5 us, 39.2 sy, 0.0 ni, 43.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

KiB Mem:  52833033+total,  3103032 used, 52522732+free,    18472 buffers
KiB Swap:  7999484 total,        0 used,  7999484 free.  1506732 cached Mem

"perf top" data

PerfTop: 124581 irqs/sec kernel:78.6% exact: 0.0% [4000Hz cycles], (all, 48 CPUs)
-----------------------------------------------------------------------------------------

     3.30%  [kernel]       [k] do_blockdev_direct_IO      
     2.99%  fio            [.] get_io_u                   
     2.79%  fio            [.] axmap_isset                
     2.40%  [kernel]       [k] irq_entries_start          
     1.91%  [kernel]       [k] _raw_spin_lock             
     1.77%  [kernel]       [k] nvme_process_cq            
     1.73%  [kernel]       [k] _raw_spin_lock_irqsave     
     1.71%  fio            [.] fio_gettime                
     1.33%  [kernel]       [k] blk_account_io_start       
     1.24%  [kernel]       [k] blk_account_io_done        
     1.23%  [kernel]       [k] kmem_cache_alloc           
     1.23%  [kernel]       [k] nvme_queue_rq              
     1.22%  fio            [.] io_u_queued_complete       
     1.14%  [kernel]       [k] native_read_tsc            
     1.11%  [kernel]       [k] kmem_cache_free            
     1.05%  [kernel]       [k] __acct_update_integrals    
     1.01%  [kernel]       [k] context_tracking_exit      
     0.94%  [kernel]       [k] _raw_spin_unlock_irqrestore
     0.91%  [kernel]       [k] rcu_eqs_enter_common       
     0.86%  [kernel]       [k] cpuacct_account_field      
     0.84%  fio            [.] td_io_queue

fio script

[global]
rw=randread
bs=4k
direct=1
ioengine=libaio
iodepth=64
time_based
runtime=60
group_reporting
numjobs=4

[job0]
filename=/dev/nvme0n1

[job1]
filename=/dev/nvme1n1

[job2]
filename=/dev/nvme2n1

[job3]
filename=/dev/nvme3n1

[job4]
filename=/dev/nvme4n1

[job5]
filename=/dev/nvme5n1

[job6]
filename=/dev/nvme6n1

[job7]
filename=/dev/nvme7n1

#1 lore raw
Quoted 8 lines · Ming Lin

Hi list,

I'm playing with 8 high performance NVMe devices on a 4 sockets server. Each device can get 730K 4k read IOPS.

Kernel: 4.1-rc3
fio test shows it doesn't scale well with 4 or more devices. I wonder any possible direction to improve it.

There was a demo at SC'14 with a heck of a lot more NVMe drives than that, and performance scaled quite linearly. Are your devices sharing PCI-e lanes?

You could try setting "cpus_allowed" on each job to the CPU's on the socket local to the nvme device. That should get a measurable improvement, and if your irq's are appropriately affinitized.

author
#2 lore raw
Quoted 7 lines · Keith Busch
Quoted 6 lines · Ming Lin

Hi list,

I'm playing with 8 high performance NVMe devices on a 4 sockets server. Each device can get 730K 4k read IOPS.

Quoted 7 lines
Quoted 3 lines

Kernel: 4.1-rc3
fio test shows it doesn't scale well with 4 or more devices. I wonder any possible direction to improve it.

There was a demo at SC'14 with a heck of a lot more NVMe drives than that, and performance scaled quite linearly. Are your devices sharing PCI-e lanes?

Is there a way to check it via, for example, /sys?

Quoted 3 lines

You could try setting "cpus_allowed" on each job to the CPU's on the socket local to the nvme device. That should get a measurable improvement, and if your irq's are appropriately affinitized.

How to know which socket is local to which nvme device?

I did a quick test with:
node0: nvme0 and nvme1
node1: nvme2 and nvme3
node2: nvme4 and nvme5
node3: nvme6 and nvme7

[job0]
filename=/dev/nvme0n1
cpus_allowed=0,4,8,12,16,20,24,28,32,36,40,44

[job1]
filename=/dev/nvme1n1
cpus_allowed=0,4,8,12,16,20,24,28,32,36,40,44

[job2]
filename=/dev/nvme2n1
cpus_allowed=1,5,9,13,17,21,25,29,33,37,41,45

[job3]
filename=/dev/nvme3n1
cpus_allowed=1,5,9,13,17,21,25,29,33,37,41,45

[job4]
filename=/dev/nvme4n1
cpus_allowed=2,6,10,14,18,22,26,30,34,38,42,46

[job5]
filename=/dev/nvme5n1
cpus_allowed=2,6,10,14,18,22,26,30,34,38,42,46

[job6]
filename=/dev/nvme6n1
cpus_allowed=3,7,11,15,19,23,27,31,35,39,43,47

[job7]
filename=/dev/nvme7n1
cpus_allowed=3,7,11,15,19,23,27,31,35,39,43,47

But it doesn't make much difference.

devices         theory          actual          actual
                IOPS(K)         IOPS(K)         IOPS(K) "cpus_allowed"
-------         -------            -------
-------------------------------
1               733             733                   733
2               1466            1446.8             1467.7
3               2199            2174.5             2213.9
4               2932            2354.9             2354.8
5               3665            3024.5             3085.2
6               4398            3818.9             3822.6
7               5131            4526.3             4517.8
8               5864            4621.2             4722.4
#3 lore raw
Quoted 5 lines · Ming Lin
Quoted 2 lines · Keith Busch

There was a demo at SC'14 with a heck of a lot more NVMe drives than that, and performance scaled quite linearly. Are your devices sharing PCI-e lanes?

Is there a way to check it via, for example, /sys?

   # lspci -tv
Quoted 5 lines
Quoted 3 lines

You could try setting "cpus_allowed" on each job to the CPU's on the socket local to the nvme device. That should get a measurable improvement, and if your irq's are appropriately affinitized.

How to know which socket is local to which nvme device?

   # cat /sys/class/nvme/nvme<#>/device/numa_node
author
#4 lore raw
Quoted 10 lines · Keith Busch
Quoted 9 lines · Ming Lin
Quoted 5 lines · Keith Busch

There was a demo at SC'14 with a heck of a lot more NVMe drives than that,
and performance scaled quite linearly. Are your devices sharing PCI-e lanes?

Quoted 4 lines
Quoted 1 line

Is there a way to check it via, for example, /sys?

  # lspci -tv

Each 4 drives share x16 lane.

Quoted 11 lines
Quoted 8 lines
Quoted 5 lines

You could try setting "cpus_allowed" on each job to the CPU's on the socket local to the nvme device. That should get a measurable improvement,
and if your irq's are appropriately affinitized.

How to know which socket is local to which nvme device?

  # cat /sys/class/nvme/nvme<#>/device/numa_node
# grep . /sys/class/nvme/nvme*/device/numa_node
/sys/class/nvme/nvme0/device/numa_node:1
/sys/class/nvme/nvme1/device/numa_node:1
/sys/class/nvme/nvme2/device/numa_node:1
/sys/class/nvme/nvme3/device/numa_node:1
/sys/class/nvme/nvme4/device/numa_node:2
/sys/class/nvme/nvme5/device/numa_node:2
/sys/class/nvme/nvme6/device/numa_node:2
/sys/class/nvme/nvme7/device/numa_node:2

With correct numa_node binding, now I can get 5010K IOPS with 8 drives. It's better now, but still not linear scaled to 5864K

I'll check if irq's are appropriately affinitized.

Thanks.

#5 lore raw
Quoted 40 lines · Ming Lin
Quoted 10 lines · Keith Busch
Quoted 9 lines · Ming Lin
Quoted 5 lines · Keith Busch

There was a demo at SC'14 with a heck of a lot more NVMe drives than that,
and performance scaled quite linearly. Are your devices sharing PCI-e lanes?

Quoted 4 lines
Quoted 1 line

Is there a way to check it via, for example, /sys?

   # lspci -tv

Each 4 drives share x16 lane.

Quoted 11 lines
Quoted 8 lines
Quoted 5 lines

You could try setting "cpus_allowed" on each job to the CPU's on the socket local to the nvme device. That should get a measurable improvement,
and if your irq's are appropriately affinitized.

How to know which socket is local to which nvme device?

   # cat /sys/class/nvme/nvme<#>/device/numa_node
# grep . /sys/class/nvme/nvme*/device/numa_node
/sys/class/nvme/nvme0/device/numa_node:1
/sys/class/nvme/nvme1/device/numa_node:1
/sys/class/nvme/nvme2/device/numa_node:1
/sys/class/nvme/nvme3/device/numa_node:1
/sys/class/nvme/nvme4/device/numa_node:2
/sys/class/nvme/nvme5/device/numa_node:2
/sys/class/nvme/nvme6/device/numa_node:2
/sys/class/nvme/nvme7/device/numa_node:2
Quoted 4 lines

With correct numa_node binding, now I can get 5010K IOPS with 8 drives. It's better now, but still not linear scaled to 5864K

I'll check if irq's are appropriately affinitized.

Just a thought, but one thing that fio is pretty intensive on is time keeping. Depending on the platform, there's some shared state between the fio IO threads. Does the picture change if you add gtod_reduce=0? In general, I'd also turn off strict random tracking. Either add 'norandommap' as an option, or use random_generator=lfsr instead.

Signature · 1 line
Jens Axboe
author
#6 lore raw
Quoted 55 lines · Jens Axboe
Quoted 54 lines · Ming Lin
Quoted 15 lines · Keith Busch
Quoted 13 lines · Ming Lin
Quoted 6 lines · Keith Busch

There was a demo at SC'14 with a heck of a lot more NVMe drives than that,
and performance scaled quite linearly. Are your devices sharing PCI-e lanes?

Quoted 5 lines
Quoted 1 line

Is there a way to check it via, for example, /sys?

   # lspci -tv

Each 4 drives share x16 lane.

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

You could try setting "cpus_allowed" on each job to the CPU's on the socket local to the nvme device. That should get a measurable improvement,
and if your irq's are appropriately affinitized.

Quoted 5 lines
Quoted 1 line

How to know which socket is local to which nvme device?

   # cat /sys/class/nvme/nvme<#>/device/numa_node
# grep . /sys/class/nvme/nvme*/device/numa_node
/sys/class/nvme/nvme0/device/numa_node:1
/sys/class/nvme/nvme1/device/numa_node:1
/sys/class/nvme/nvme2/device/numa_node:1
/sys/class/nvme/nvme3/device/numa_node:1
/sys/class/nvme/nvme4/device/numa_node:2
/sys/class/nvme/nvme5/device/numa_node:2
/sys/class/nvme/nvme6/device/numa_node:2
/sys/class/nvme/nvme7/device/numa_node:2

With correct numa_node binding, now I can get 5010K IOPS with 8 drives. It's better now, but still not linear scaled to 5864K

Quoted 8 lines
Quoted 1 line

I'll check if irq's are appropriately affinitized.

Just a thought, but one thing that fio is pretty intensive on is time keeping. Depending on the platform, there's some shared state between the fio IO threads. Does the picture change if you add gtod_reduce=0? In general, I'd also turn off strict random tracking. Either add 'norandommap' as an option, or use random_generator=lfsr instead.

I'll try it once the server is free.

It's 4 NUMA nodes with 8 NVMe drives.
With current installation, each 4 drives are local to one node and share one PCIE 3.0 x16 lane.
I'll re-install it, so each 2 drives are local to one node and share one x16 lane.

That will probably also help.

Quoted 4 lines

--
Jens Axboe

#7 lore raw
In reply to Keith Busch

I ran some tests last year before SC using 8 drives in a SuperMicro server. Please see attached. OS was CentOS 6.5 I think.

-Azher

Quoted 20 lines · Keith Busch
Quoted 8 lines · Ming Lin

Hi list,

I'm playing with 8 high performance NVMe devices on a 4 sockets server. Each device can get 730K 4k read IOPS.

Kernel: 4.1-rc3
fio test shows it doesn't scale well with 4 or more devices. I wonder any possible direction to improve it.

There was a demo at SC'14 with a heck of a lot more NVMe drives than that,
and performance scaled quite linearly. Are your devices sharing PCI-e lanes?

You could try setting "cpus_allowed" on each job to the CPU's on the socket local to the nvme device. That should get a measurable improvement,
and if your irq's are appropriately affinitized.

Quoted 5 lines

_______________________________________________
Linux-nvme mailing list
Linux-nvme at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 8drives-dd-SC9.PNG
Type: image/png
Size: 83980 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-nvme/attachments/20150601/814c0441/attachment-0001.png>

#8 lore raw
In reply to Ming Lin
Quoted 28 lines · Ming Lin

Hi list,

[global]
rw=randread
bs=4k
direct=1
ioengine=libaio
iodepth=64
time_based
runtime=60
group_reporting
numjobs=4

[job0]
filename=/dev/nvme0n1

[job1]
filename=/dev/nvme1n1

[job2]
filename=/dev/nvme2n1

[job3]
filename=/dev/nvme3n1

[job4]
filename=/dev/nvme4n1

Quoted 9 lines

[job5]
filename=/dev/nvme5n1

[job6]
filename=/dev/nvme6n1

[job7]
filename=/dev/nvme7n1

A wild guess, the jobs might run on a remote CPU compared to the device. Try to affinitize the jobs so they run on the CPU with the attached device.

#9 lore raw
In reply to Ming Lin
Quoted 34 lines · Ming Lin

Hi list,

I'm playing with 8 high performance NVMe devices on a 4 sockets server. Each device can get 730K 4k read IOPS.

Kernel: 4.1-rc3
fio test shows it doesn't scale well with 4 or more devices. I wonder any possible direction to improve it.

devices         theory          actual
                IOPS(K)         IOPS(K)
-------         -------         -------
1               733             733
2               1466            1446.8
3               2199            2174.5
4               2932            2354.9
5               3665            3024.5
6               4398            3818.9
7               5131            4526.3
8               5864            4621.2

And a graph here:
http://minggr.net/pub/20150601/nvme-scalability.jpg

With 8 devices, CPU is still 43% idle, so CPU is not the bottleneck.

"top" data

Tasks: 565 total, 30 running, 535 sleeping, 0 stopped, 0 zombie
%Cpu(s): 17.5 us, 39.2 sy, 0.0 ni, 43.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

KiB Mem:  52833033+total,  3103032 used, 52522732+free,    18472 buffers
KiB Swap:  7999484 total,        0 used,  7999484 free.  1506732 cached Mem
Quoted 8 lines

"perf top" data

   PerfTop:  124581 irqs/sec  kernel:78.6%  exact:  0.0% [4000Hz cycles],  (all, 48 CPUs)

-----------------------------------------------------------------------------------------

     3.30%  [kernel]       [k] do_blockdev_direct_IO
     2.99%  fio            [.] get_io_u
     2.79%  fio            [.] axmap_isset

Just a thought as well, but axmap_isset cpu usage is suspiciously high, given a read-only workload where it's essentially a noop.

Regards,
Andrey

Quoted 63 lines
     2.40%  [kernel]       [k] irq_entries_start
     1.91%  [kernel]       [k] _raw_spin_lock
     1.77%  [kernel]       [k] nvme_process_cq
     1.73%  [kernel]       [k] _raw_spin_lock_irqsave
     1.71%  fio            [.] fio_gettime
     1.33%  [kernel]       [k] blk_account_io_start
     1.24%  [kernel]       [k] blk_account_io_done
     1.23%  [kernel]       [k] kmem_cache_alloc
     1.23%  [kernel]       [k] nvme_queue_rq
     1.22%  fio            [.] io_u_queued_complete
     1.14%  [kernel]       [k] native_read_tsc
     1.11%  [kernel]       [k] kmem_cache_free
     1.05%  [kernel]       [k] __acct_update_integrals
     1.01%  [kernel]       [k] context_tracking_exit
     0.94%  [kernel]       [k] _raw_spin_unlock_irqrestore
     0.91%  [kernel]       [k] rcu_eqs_enter_common
     0.86%  [kernel]       [k] cpuacct_account_field
     0.84%  fio            [.] td_io_queue

fio script

[global]
rw=randread
bs=4k
direct=1
ioengine=libaio
iodepth=64
time_based
runtime=60
group_reporting
numjobs=4

[job0]
filename=/dev/nvme0n1

[job1]
filename=/dev/nvme1n1

[job2]
filename=/dev/nvme2n1

[job3]
filename=/dev/nvme3n1

[job4]
filename=/dev/nvme4n1

[job5]
filename=/dev/nvme5n1

[job6]
filename=/dev/nvme6n1

[job7]
filename=/dev/nvme7n1

_______________________________________________
Linux-nvme mailing list
Linux-nvme at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

#10 lore raw
Quoted 40 lines · Andrey Kuzmin
Quoted 39 lines · Ming Lin

Hi list,

I'm playing with 8 high performance NVMe devices on a 4 sockets server. Each device can get 730K 4k read IOPS.

Kernel: 4.1-rc3
fio test shows it doesn't scale well with 4 or more devices. I wonder any possible direction to improve it.

devices theory actual
IOPS(K) IOPS(K)

-------         -------         -------
1               733             733
2               1466            1446.8
3               2199            2174.5
4               2932            2354.9
5               3665            3024.5
6               4398            3818.9
7               5131            4526.3
8               5864            4621.2

And a graph here:
http://minggr.net/pub/20150601/nvme-scalability.jpg

With 8 devices, CPU is still 43% idle, so CPU is not the bottleneck.

"top" data

Tasks: 565 total, 30 running, 535 sleeping, 0 stopped, 0 zombie
%Cpu(s): 17.5 us, 39.2 sy, 0.0 ni, 43.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

KiB Mem:  52833033+total,  3103032 used, 52522732+free,    18472 buffers
KiB Swap:  7999484 total,        0 used,  7999484 free.  1506732 cached Mem

"perf top" data

    PerfTop:  124581 irqs/sec  kernel:78.6%  exact:  0.0% [4000Hz cycles],  (all, 48 CPUs)

-----------------------------------------------------------------------------------------

Quoted 6 lines
Quoted 3 lines
      3.30%  [kernel]       [k] do_blockdev_direct_IO
      2.99%  fio            [.] get_io_u
      2.79%  fio            [.] axmap_isset

Just a thought as well, but axmap_isset cpu usage is suspiciously high, given a read-only workload where it's essentially a noop.

Read or write doesn't matter, it's still marked in the random map. Both of them will maintain that state.

Signature · 1 line
Jens Axboe
#11 lore raw
Quoted 52 lines · Jens Axboe
Quoted 51 lines · Andrey Kuzmin
Quoted 47 lines · Ming Lin

Hi list,

I'm playing with 8 high performance NVMe devices on a 4 sockets server. Each device can get 730K 4k read IOPS.

Kernel: 4.1-rc3
fio test shows it doesn't scale well with 4 or more devices. I wonder any possible direction to improve it.

devices theory actual
IOPS(K) IOPS(K)

-------         -------         -------
1               733             733
2               1466            1446.8
3               2199            2174.5
4               2932            2354.9
5               3665            3024.5
6               4398            3818.9
7               5131            4526.3
8               5864            4621.2

And a graph here:
http://minggr.net/pub/20150601/nvme-scalability.jpg

With 8 devices, CPU is still 43% idle, so CPU is not the bottleneck.

"top" data

Tasks: 565 total, 30 running, 535 sleeping, 0 stopped, 0 zombie
%Cpu(s): 17.5 us, 39.2 sy, 0.0 ni, 43.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

KiB Mem:  52833033+total,  3103032 used, 52522732+free,    18472 buffers
KiB Swap:  7999484 total,        0 used,  7999484 free.  1506732 cached

Mem

"perf top" data

    PerfTop:  124581 irqs/sec  kernel:78.6%  exact:  0.0% [4000Hz

cycles], (all, 48 CPUs)

-----------------------------------------------------------------------------------------

      3.30%  [kernel]       [k] do_blockdev_direct_IO
      2.99%  fio            [.] get_io_u
      2.79%  fio            [.] axmap_isset
Quoted 7 lines
Quoted 2 lines

Just a thought as well, but axmap_isset cpu usage is suspiciously high, given a read-only workload where it's essentially a noop.

Read or write doesn't matter, it's still marked in the random map. Both of them will maintain that state.

Not sure keeping track of blocks read was the intention in the test, so it's worth rerunning with norandommap=1.

Regards,
Andrey

Quoted 3 lines

--
Jens Axboe

#12 lore raw
Quoted 57 lines · Andrey Kuzmin
Quoted 56 lines · Jens Axboe
Quoted 53 lines · Andrey Kuzmin
Quoted 39 lines · Ming Lin

Hi list,

I'm playing with 8 high performance NVMe devices on a 4 sockets server. Each device can get 730K 4k read IOPS.

Kernel: 4.1-rc3
fio test shows it doesn't scale well with 4 or more devices. I wonder any possible direction to improve it.

devices theory actual
IOPS(K) IOPS(K)

-------         -------         -------
1               733             733
2               1466            1446.8
3               2199            2174.5
4               2932            2354.9
5               3665            3024.5
6               4398            3818.9
7               5131            4526.3
8               5864            4621.2

And a graph here:
http://minggr.net/pub/20150601/nvme-scalability.jpg

With 8 devices, CPU is still 43% idle, so CPU is not the bottleneck.

"top" data

Tasks: 565 total, 30 running, 535 sleeping, 0 stopped, 0 zombie
%Cpu(s): 17.5 us, 39.2 sy, 0.0 ni, 43.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

KiB Mem:  52833033+total,  3103032 used, 52522732+free,    18472 buffers
KiB Swap:  7999484 total,        0 used,  7999484 free.  1506732 cached

Mem

"perf top" data

Quoted 8 lines
     PerfTop:  124581 irqs/sec  kernel:78.6%  exact:  0.0% [4000Hz

cycles], (all, 48 CPUs)

-----------------------------------------------------------------------------------------

       3.30%  [kernel]       [k] do_blockdev_direct_IO
       2.99%  fio            [.] get_io_u
       2.79%  fio            [.] axmap_isset

Just a thought as well, but axmap_isset cpu usage is suspiciously high, given a read-only workload where it's essentially a noop.

Quoted 6 lines
Quoted 3 lines

Read or write doesn't matter, it's still marked in the random map. Both of them will maintain that state.

Not sure keeping track of blocks read was the intention in the test, so it's worth rerunning with norandommap=1.

Right, it doesn't matter for this test. But it's only a few percent of CPU, and should not impact scaling. I suspect the time keeping would be a bigger offender.

Signature · 1 line
Jens Axboe