Skip to contentSkip to conversation

ublk-nbd: ublk-nbd is avaialbe

DISCUSSION Ming Lei 5 participants 10 messages Jan 19, 2023 – Mar 12, 2023

Subsystems (2)
Block layer io_uring
lore
Share & tools
lore

Ming Lei

author
#0 lore raw

Hi,

ublk-nbd[1] is available now.

Basically it is one nbd client, but totally implemented in userspace, and wrt. current nbd-client in [2], the transmission phase is done by linux block nbd driver.

The handshake implementation is borrowed from nbd project[2], so basically ublk-nbd just adds new code for implementing transmission phase, and it can be thought as moving linux block nbd driver into userspace.

The added new code is basically in nbd/tgt_nbd.cpp, and io handling is based on liburing[3], and implemented by c++20 coroutine, so everything is done in single pthread totally lockless, meantime turns out it is pretty easy to design & implement, attributed to ublk framework, c++20 coroutine and liburing.

ublk-nbd supports both tcp and unix socket, and allows to enable io_uring send zero copy via command line '--send_zc', see details in README[4].

No regression is found in xfstests by using ublk-nbd as both test device and scratch device, and builtin test(make test T=nbd) runs well.

Fio test("make test T=nbd") shows that ublk-nbd performance is basically same with nbd-client/nbd driver when running fio on real ethernet link(1g, 10+g), but ublk-nbd IOPS is higher by ~40% than nbd-client(nbd driver) with 512K BS, which is because linux nbd driver sets max_sectors_kb as 64KB at default.

But when running fio over local tcp socket, it is observed in my test machine that ublk-nbd performs better than nbd-client/nbd driver, especially with 2 queue/2 jobs, and the gap could be 10% ~ 30% according to different block size.

Any comments are welcome!

[1] https://github.com/ming1/ubdsrv/blob/master/nbd
[2] https://github.com/NetworkBlockDevice/nbd
[3] https://github.com/axboe/liburing
[4] https://github.com/ming1/ubdsrv/blob/master/nbd/README.rst

Thanks,
Ming

maintainer
#1 lore raw
Quoted 31 lines · Ming Lei

Hi,

ublk-nbd[1] is available now.

Basically it is one nbd client, but totally implemented in userspace, and wrt. current nbd-client in [2], the transmission phase is done by linux block nbd driver.

The handshake implementation is borrowed from nbd project[2], so basically ublk-nbd just adds new code for implementing transmission phase, and it can be thought as moving linux block nbd driver into userspace.

The added new code is basically in nbd/tgt_nbd.cpp, and io handling is based on liburing[3], and implemented by c++20 coroutine, so everything is done in single pthread totally lockless, meantime turns out it is pretty easy to design & implement, attributed to ublk framework, c++20 coroutine and liburing.

ublk-nbd supports both tcp and unix socket, and allows to enable io_uring send zero copy via command line '--send_zc', see details in README[4].

No regression is found in xfstests by using ublk-nbd as both test device and scratch device, and builtin test(make test T=nbd) runs well.

Fio test("make test T=nbd") shows that ublk-nbd performance is basically same with nbd-client/nbd driver when running fio on real ethernet link(1g, 10+g), but ublk-nbd IOPS is higher by ~40% than nbd-client(nbd driver) with 512K BS, which is because linux nbd driver sets max_sectors_kb as 64KB at default.

Quoted 4 lines

But when running fio over local tcp socket, it is observed in my test machine that ublk-nbd performs better than nbd-client/nbd driver, especially with 2 queue/2 jobs, and the gap could be 10% ~ 30% according to different block size.

This is pretty nice! Just curious, have you tried setting up your ring with

p.flags |= IORING_SETUP_SINGLE_ISSUER | IORING_SETUP_DEFER_TASKRUN;

and see if that yields any extra performance improvements for you? Depending on how you do processing, you should not need to do any further changes there.

A "lighter" version is just setting IORING_SETUP_COOP_TASKRUN.

Signature · 1 line
Jens Axboe
author
#2 lore raw

Hi Jens,

Quoted 40 lines · Jens Axboe
Quoted 31 lines · Ming Lei

Hi,

ublk-nbd[1] is available now.

Basically it is one nbd client, but totally implemented in userspace, and wrt. current nbd-client in [2], the transmission phase is done by linux block nbd driver.

The handshake implementation is borrowed from nbd project[2], so basically ublk-nbd just adds new code for implementing transmission phase, and it can be thought as moving linux block nbd driver into userspace.

The added new code is basically in nbd/tgt_nbd.cpp, and io handling is based on liburing[3], and implemented by c++20 coroutine, so everything is done in single pthread totally lockless, meantime turns out it is pretty easy to design & implement, attributed to ublk framework, c++20 coroutine and liburing.

ublk-nbd supports both tcp and unix socket, and allows to enable io_uring send zero copy via command line '--send_zc', see details in README[4].

No regression is found in xfstests by using ublk-nbd as both test device and scratch device, and builtin test(make test T=nbd) runs well.

Fio test("make test T=nbd") shows that ublk-nbd performance is basically same with nbd-client/nbd driver when running fio on real ethernet link(1g, 10+g), but ublk-nbd IOPS is higher by ~40% than nbd-client(nbd driver) with 512K BS, which is because linux nbd driver sets max_sectors_kb as 64KB at default.

Quoted 4 lines

But when running fio over local tcp socket, it is observed in my test machine that ublk-nbd performs better than nbd-client/nbd driver, especially with 2 queue/2 jobs, and the gap could be 10% ~ 30% according to different block size.

This is pretty nice! Just curious, have you tried setting up your ring with

Quoted 7 lines

p.flags |= IORING_SETUP_SINGLE_ISSUER | IORING_SETUP_DEFER_TASKRUN;

and see if that yields any extra performance improvements for you? Depending on how you do processing, you should not need to do any further changes there.

A "lighter" version is just setting IORING_SETUP_COOP_TASKRUN.

IORING_SETUP_COOP_TASKRUN is enabled in current ublksrv.

After disabling COOP_TASKRUN and enabling SINGLE_ISSUER & DEFER_TASKRUN, not see obvious improvement, meantime regression is observed on 64k rw.

Thanks,
Ming

#3 lore raw

Hi,

Quoted 49 lines · Ming Lei

Hi Jens,

Quoted 46 lines · Jens Axboe
Quoted 31 lines · Ming Lei

Hi,

ublk-nbd[1] is available now.

Basically it is one nbd client, but totally implemented in userspace, and wrt. current nbd-client in [2], the transmission phase is done by linux block nbd driver.

The handshake implementation is borrowed from nbd project[2], so basically ublk-nbd just adds new code for implementing transmission phase, and it can be thought as moving linux block nbd driver into userspace.

The added new code is basically in nbd/tgt_nbd.cpp, and io handling is based on liburing[3], and implemented by c++20 coroutine, so everything is done in single pthread totally lockless, meantime turns out it is pretty easy to design & implement, attributed to ublk framework, c++20 coroutine and liburing.

ublk-nbd supports both tcp and unix socket, and allows to enable io_uring send zero copy via command line '--send_zc', see details in README[4].

No regression is found in xfstests by using ublk-nbd as both test device and scratch device, and builtin test(make test T=nbd) runs well.

Fio test("make test T=nbd") shows that ublk-nbd performance is basically same with nbd-client/nbd driver when running fio on real ethernet link(1g, 10+g), but ublk-nbd IOPS is higher by ~40% than nbd-client(nbd driver) with 512K BS, which is because linux nbd driver sets max_sectors_kb as 64KB at default.

Quoted 4 lines

But when running fio over local tcp socket, it is observed in my test machine that ublk-nbd performs better than nbd-client/nbd driver, especially with 2 queue/2 jobs, and the gap could be 10% ~ 30% according to different block size.

This is pretty nice! Just curious, have you tried setting up your ring with

p.flags |= IORING_SETUP_SINGLE_ISSUER | IORING_SETUP_DEFER_TASKRUN;

and see if that yields any extra performance improvements for you? Depending on how you do processing, you should not need to do any further changes there.

Quoted 7 lines
Quoted 1 line

A "lighter" version is just setting IORING_SETUP_COOP_TASKRUN.

IORING_SETUP_COOP_TASKRUN is enabled in current ublksrv.

After disabling COOP_TASKRUN and enabling SINGLE_ISSUER & DEFER_TASKRUN, not see obvious improvement, meantime regression is observed on 64k rw.

Does it handle network errors better than the default nbd client, i.e. is it able to seamlessly reconnect after while keeping the same device or do you end up with multiple devices ? That's one big trouble I faced with the original nbd client, forcing you to unmount and remount everything after a network outage for example.

Thanks!
Willy

author
#4 lore raw
Quoted 60 lines · Willy Tarreau

Hi,

Quoted 49 lines · Ming Lei

Hi Jens,

Quoted 46 lines · Jens Axboe
Quoted 31 lines · Ming Lei

Hi,

ublk-nbd[1] is available now.

Basically it is one nbd client, but totally implemented in userspace, and wrt. current nbd-client in [2], the transmission phase is done by linux block nbd driver.

The handshake implementation is borrowed from nbd project[2], so basically ublk-nbd just adds new code for implementing transmission phase, and it can be thought as moving linux block nbd driver into userspace.

The added new code is basically in nbd/tgt_nbd.cpp, and io handling is based on liburing[3], and implemented by c++20 coroutine, so everything is done in single pthread totally lockless, meantime turns out it is pretty easy to design & implement, attributed to ublk framework, c++20 coroutine and liburing.

ublk-nbd supports both tcp and unix socket, and allows to enable io_uring send zero copy via command line '--send_zc', see details in README[4].

No regression is found in xfstests by using ublk-nbd as both test device and scratch device, and builtin test(make test T=nbd) runs well.

Fio test("make test T=nbd") shows that ublk-nbd performance is basically same with nbd-client/nbd driver when running fio on real ethernet link(1g, 10+g), but ublk-nbd IOPS is higher by ~40% than nbd-client(nbd driver) with 512K BS, which is because linux nbd driver sets max_sectors_kb as 64KB at default.

Quoted 4 lines

But when running fio over local tcp socket, it is observed in my test machine that ublk-nbd performs better than nbd-client/nbd driver, especially with 2 queue/2 jobs, and the gap could be 10% ~ 30% according to different block size.

This is pretty nice! Just curious, have you tried setting up your ring with

p.flags |= IORING_SETUP_SINGLE_ISSUER | IORING_SETUP_DEFER_TASKRUN;

and see if that yields any extra performance improvements for you? Depending on how you do processing, you should not need to do any further changes there.

Quoted 7 lines
Quoted 1 line

A "lighter" version is just setting IORING_SETUP_COOP_TASKRUN.

IORING_SETUP_COOP_TASKRUN is enabled in current ublksrv.

After disabling COOP_TASKRUN and enabling SINGLE_ISSUER & DEFER_TASKRUN, not see obvious improvement, meantime regression is observed on 64k rw.

Quoted 5 lines

Does it handle network errors better than the default nbd client, i.e. is it able to seamlessly reconnect after while keeping the same device or do you end up with multiple devices ? That's one big trouble I faced with the original nbd client, forcing you to unmount and remount everything after a network outage for example.

All kinds of ublk disk supports such seamlessly recovery which is provided by UBLK_CMD_START_USER_RECOVERY/UBLK_CMD_END_USER_RECOVERY. During user recovery, the bdev and gendisk instance won't be gone, and will become fully functional after the recovery(such as reconnect) is successful.

So yes for this seamlessly reconnect error handling.

Thanks,
Ming

#5 lore raw
Quoted 67 lines · Ming Lei
Quoted 60 lines · Willy Tarreau

Hi,

Quoted 49 lines · Ming Lei

Hi Jens,

Quoted 46 lines · Jens Axboe
Quoted 31 lines · Ming Lei

Hi,

ublk-nbd[1] is available now.

Basically it is one nbd client, but totally implemented in userspace, and wrt. current nbd-client in [2], the transmission phase is done by linux block nbd driver.

The handshake implementation is borrowed from nbd project[2], so basically ublk-nbd just adds new code for implementing transmission phase, and it can be thought as moving linux block nbd driver into userspace.

The added new code is basically in nbd/tgt_nbd.cpp, and io handling is based on liburing[3], and implemented by c++20 coroutine, so everything is done in single pthread totally lockless, meantime turns out it is pretty easy to design & implement, attributed to ublk framework, c++20 coroutine and liburing.

ublk-nbd supports both tcp and unix socket, and allows to enable io_uring send zero copy via command line '--send_zc', see details in README[4].

No regression is found in xfstests by using ublk-nbd as both test device and scratch device, and builtin test(make test T=nbd) runs well.

Fio test("make test T=nbd") shows that ublk-nbd performance is basically same with nbd-client/nbd driver when running fio on real ethernet link(1g, 10+g), but ublk-nbd IOPS is higher by ~40% than nbd-client(nbd driver) with 512K BS, which is because linux nbd driver sets max_sectors_kb as 64KB at default.

Quoted 4 lines

But when running fio over local tcp socket, it is observed in my test machine that ublk-nbd performs better than nbd-client/nbd driver, especially with 2 queue/2 jobs, and the gap could be 10% ~ 30% according to different block size.

This is pretty nice! Just curious, have you tried setting up your ring with

p.flags |= IORING_SETUP_SINGLE_ISSUER | IORING_SETUP_DEFER_TASKRUN;

and see if that yields any extra performance improvements for you? Depending on how you do processing, you should not need to do any further changes there.

Quoted 7 lines
Quoted 1 line

A "lighter" version is just setting IORING_SETUP_COOP_TASKRUN.

IORING_SETUP_COOP_TASKRUN is enabled in current ublksrv.

After disabling COOP_TASKRUN and enabling SINGLE_ISSUER & DEFER_TASKRUN, not see obvious improvement, meantime regression is observed on 64k rw.

Quoted 5 lines

Does it handle network errors better than the default nbd client, i.e. is it able to seamlessly reconnect after while keeping the same device or do you end up with multiple devices ? That's one big trouble I faced with the original nbd client, forcing you to unmount and remount everything after a network outage for example.

Quoted 7 lines

All kinds of ublk disk supports such seamlessly recovery which is provided by UBLK_CMD_START_USER_RECOVERY/UBLK_CMD_END_USER_RECOVERY. During user recovery, the bdev and gendisk instance won't be gone, and will become fully functional after the recovery(such as reconnect) is successful.

So yes for this seamlessly reconnect error handling.

Nice, it's tempting to give it a try then ;-)

Willy

#6 lore raw
In reply to Ming Lei

Hi!

Quoted 5 lines

ublk-nbd[1] is available now.

Basically it is one nbd client, but totally implemented in userspace, and wrt. current nbd-client in [2], the transmission phase is done by linux block nbd driver.

There is reason nbd-client needs to be in kernel, and the reason is deadlocks during low memory situations.

Best regards,
Pavel

Signature · 1 line
People of Russia, stop Putin before his war on Ukraine escalates.
author
#7 lore raw
Quoted 10 lines · Pavel Machek

Hi!

Quoted 5 lines

ublk-nbd[1] is available now.

Basically it is one nbd client, but totally implemented in userspace, and wrt. current nbd-client in [2], the transmission phase is done by linux block nbd driver.

There is reason nbd-client needs to be in kernel, and the reason is deadlocks during low memory situations.

Last time, the nbd memory deadlock is solved by the following approach [1], which is used for ublk too.

Actually ublk can be thought as replacing nbd socket communication with (much more lightweight & generic) uring_cmd, and move nbd socket communication into userspace for ublk-nbd. Not see such way may cause memory deadlock.

Also, ublk has built-in user recovery mechanism, killing deadlock user daemon and recovering it can be the last straw, and the disk node won't be gone away during the recovery.

So please provide some analysis or reproductions, otherwise I may have to ignore your comments.

[1] https://lore.kernel.org/linux-fsdevel/[email protected]/

Thanks,
Ming

#8 lore raw
In reply to Ming Lei

Hi,

Quoted 4 lines · Ming Lei

The handshake implementation is borrowed from nbd project[2], so basically ublk-nbd just adds new code for implementing transmission phase, and it can be thought as moving linux block nbd driver into userspace.

[...]

Quoted 1 line

Any comments are welcome!

I see you copied nbd-client.c and modified it, but removed all the author information from it (including mine).

Please don't do that. nbd-client is not public domain, it is GPLv2, which means you need to keep copyright statements around somewhere. You can move them into an AUTHORS file or some such if you prefer, but you can't just remove them blindly.

Thanks.

Signature · 4 lines
     w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

I will have a Tin-Actinium-Potassium mixture, thanks.
author
#9 lore raw
Quoted 11 lines · Wouter Verhelst

Hi,

Quoted 4 lines · Ming Lei

The handshake implementation is borrowed from nbd project[2], so basically ublk-nbd just adds new code for implementing transmission phase, and it can be thought as moving linux block nbd driver into userspace.

[...]

Quoted 1 line

Any comments are welcome!

Quoted 7 lines

I see you copied nbd-client.c and modified it, but removed all the author information from it (including mine).

Please don't do that. nbd-client is not public domain, it is GPLv2, which means you need to keep copyright statements around somewhere. You can move them into an AUTHORS file or some such if you prefer, but you can't just remove them blindly.

Thanks for finding it, and it must be one accident, and I will add the author info
back soon.

thanks,