Let's design the next-gen libiio!
Hello beautiful people,
First of all, apologies for sending such a long email. Double apologies if you're caught in the email chain and you don't care about the topic. Just don't click "reply all" when you send me an angry response, as the thread will be publicly logged.
The reason for you receiving this, is that you've been involved with libiio
(https://github.com/analogdevicesinc/libiio) either directly or indirectly, or subscribed to the IIO kernel mailing list, or have shown interest in the project, or have an homonym who does.
About five years ago libiio was born, as an easy-to-use library for interfacing to Linux' Industrial Input/Output (IIO) kernel subsystem, either with devices preset on the host platform, or on a remote target platform. I think we did a good job, as the API was never broken since its first release, and the praise/hate ratio we received was very encouraging. We've seen people using it in all kinds of applications, from car infotainment to space research, from drones to software radio.
It's not perfect, though, and its current design has room for improvements, both inside and outside the library. However, we've hit a point where such improvements would require a redesign of the library and kernel bits, and as a consequence, a breakage of the API. It's time for the next-gen libiio, and a overhaul of the whole IIO stack, really.
Your mission, should you choose to accept it, is to feed me with ideas, concerns, comments about what you think libiio was lacking or just not doing right. We want to open the possibility for all interested parties to help sketch the future library.
I have compiled below a list of changes that we think should be done, comments are very welcome. Note that emails in HTML form will probably be bounced back by the IIO mailing list, so please write responses in plain text.
Thank you for your time.
Kind regards,
Paul Cercueil
------
Kernel
------
* Support for buffer metadata
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Support tagging the buffers with any relevant information. Either
from the IIO
core for general accounting (e.g. timestamp) or from the driver for hardware-facing code (e.g. buffer underflow).
Metadata would be attached to a specific sample in the buffer
(metadata that
applies to the whole buffer can be attached to the first sample).
Several
entries could be attached to the same sample. The format would be a
simple
key="value", I don't think we need anything more complex than that. For DACs, it should be possible to set metadata from userspace. The
drivers
would then interpret the metadata tags if they need to.
* Facility to detect overflow/underflow
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
There should be a way to detect overflows (for ADCs) and underflows
(for
DACs), if the hardware supports it, and report them to userspace
(through a
IIO event, I suppose). This is something that could be done by the
drivers,
but maybe it would make sense to have this functionality in the IIO
common
code?
* Delayed attribute write / Command queues
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
We need a mechanism that allows us to change the value of an
attribute at a
very specific time (or sample number) after a capture or upload is
started;
some kind of script mechanism, or 'command list' analog to the
display lists
used in old GPUs. This would be used for instance in software
applications
doing time-divison duplexing (TDD).
* Better high-speed buffer mechanism
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The current buffer API is fine for low-speed devices, but we're
dealing with
ADCs and DACs of the GB/s class. ADI already contributed an improved
(faster)
buffer mechanism, but which is incomplete (no support for DACs),
doesn't
integrate very well in the ecosystem, and is somewhat redundant with
the old
one.
The idea would be to deprecate this API and propose an alternative
that
makes use of current technology, like dmabuf. The concept would stay
the
same, each IIO device has a pool of DMA buffers, and userspace can
queue
and dequeue buffers.
The rationale behind this change, is that with the current two APIs
it is not
possible to move data between IIO devices and a network or USB card
without
having the CPU copy the data. This is an important problem, as
high-speed
ADCs and DACs are generally connected to FPGAs running Linux on a
softcore,
which stream their data to a workstation for further processing. With
the
new API, the userspace software would simply obtain a pointer to a
dmabuf
from the IIO interface, and simply pass it to the network card (this
means
the network stack and USB stack would also need to support dmabuf).
An alternative would be to keep the current file-based buffer and buffer-queue-based APIs, but enhance the former one with support for splicing (with splice()/vmsplice()). This might arguably be easier to
do,
since the network stack already supports it. The problem with
splicing is
that the kernel swaps each data page with a fresh zeroed page in
order to
avoid leaking kernel memory. To be truely zero-copy, this requires
the page
cleaning mechanism to be offloaded to e.g. DMA, otherwise the process
isn't
entirely CPU-free.
* Parsable attribute names
~~~~~~~~~~~~~~~~~~~~~~~~
IIO attributes follow a certain formatting that does not make it
possible to
reconstruct the devices/channels/attributes tree in userspace. The
channel
attributes are formatted as:
<direction>_<type><id>_<modifier>_<extended-name>_<attribute-name> The problem is that <extended-name>, <modifier> and <attribute-name>
can
contain underscores, which makes it impossible to parse. For instance, with an attribute named "in_voltage0_high_impedence_line_enable":
Is the extended name "high_impedence_line" and the attribute name
"enable",
or is the extended name "high_impedence" and the attribute name
"line_enable"?
Since the sysfs attributes is ABI, the solution could be to have a
separate
sysfs file, for instance <direction>_<type><id>_extended_name, that
would
contain the extended name of the channel. This should be enough for
the IIO
sysfs interface to be machine-readable.
-------
Libiio2
-------
* stdio redirection
~~~~~~~~~~~~~~~~~
Right now libiio prints its debug information unconditionally to stdout/stderr, it should be modified to be able to output its log to
any
valid file descriptor. While that looks simple on paper, it must be
set up
before a library context is created, since the context creation
itself may
print debug information. This require the context allocation to be
separated
from its initialization, so a break of API.
* Separate allocation from initialization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
As stated above, functions to alloc/destroy and init/deinit the
various IIO
structures should be separate; this would allow e.g. to set some
parameters
to a iio_context before the context creation happens, for instance
where to
redirect the debug output.
* Modular backends
~~~~~~~~~~~~~~~~
Right now the backends of libiio are tightly coupled to the core. It
would
be great if the users could install only the backends they're
interested in.
Of course, backends could still be backed into the core library, and
that
would stay the default behaviour for non-Linux platforms.
This requires two things:
- The libiio backends must be able to be updated independently of the
main
library, therefore they should only depend on the top-level API of libiio. Apart from a few exceptions here and there, this is already mostly the case.
- There needs to be a facility to load external backends based on the
backend name. Thanksfully with URIs this becomes easy: creating a context from the URI "foo:" would result in the backend module "foo" loaded from the disk, if not already baked in.
* Event support
~~~~~~~~~~~~~
The IIO subsystem has support for events. This allows the userspace
to get
notified for instance when a temperature gets over or under a
threshold.
Right now, libiio cannot easily support events, as the client/server protocol of the network backend doesn't really allow it. Therefore,
the new
libiio2 library should be designed from the ground up with support for events, and the API should offer a way to register a callback that
would be
called when a IIO event occurs.
* Context change detection
~~~~~~~~~~~~~~~~~~~~~~~~
When a device is added, removed or the context becomes unavailable
(e.g. the
USB cable was unplugged), there should be a built-in mechanism to
notify the
applications using libiio2, maybe using the same mechanism as for IIO
events.
The same would apply on the local backend, if for instance a new
device
appears, the library should be able to pick it up and report the new
device
to the application.
* Asynchronous network communication
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The network communication between libiio and IIO is sub-par, because
it uses
a synchronous request-response protocol. This causes the throughput
to max
out way below the theorical maximum throughput of Gigabit Ethernet. The communication protocol should be modified in a way that fully
maximizes
the throughput.
The old communication protocol should still be available and used by
default
unless the two parties agree to use the new protocol. This ensures
that
TinyIIOD (the microcontroller variant of IIOD, the server that
communicates
with the network backend of libiio) can still work with the network
backend of
libiio2. Alternatively, TinyIIOD could be updated to the new network
protocol.
That way, the old network protocol wouldn't have to stick around.
To implement the asynchronous network communication, we could have a
look at
ZeroMQ (https://zeromq.org) which seems to be designed for that
particular
task, and is available under a LGPLv3 license. It is however
uncertain how
that would work on the IIOD side (because of the zero-copy
requirement - see
below).
Additionally, the network backend should part ways with its current ASCII-based protocol. This made sense when libiio was first created,
but it
requires quite a complex parser on the server side that could go away
by
resorting to a more classic protocol.
* Zero-copy
~~~~~~~~~
Apart from supporting the new network protocol, libiio should provide
a way
to stream data between IIO devices and standard file descriptors or
sockets
without having the CPU copy the data.
If a dmabuf-based IIO kernel interface is used, we need a way to send
a
dmabuf to a socket, and I'm not sure that this is currently possible. But as dmabuf are internally just scatterlists, and the sockets
support
splice(), that should be somewhat doable.
To complicate things, IIOD (the network server) must be able to
stream the
same data to more than one client at a time, unless we decide that
this is
no more a requirement. This means that one single dmabuf would be
submitted
to more than one socket.
Streaming data to multiple clients using zero-copy techniques in IIOD
also
implies that the data cannot be modified, which means that the server
cannot
do the demuxing, and this task is devoted to the clients; so the
(currently
optional) server-side demuxing option would be removed.
* USB3
~~~~
Just like the network, the USB backend should be updated to be able
to reach
out the maximum throughput offered by USB3.
The principal problem with USB is the short number of endpoints,
which limits
the functionality, as it may not be possible to stream data from
different
devices at the same time. Right now, a pair of endpoints is reserved
for
generic commands (read attributes, open/close IIO device, etc.), and
one pair
is reserved per IIO device streaming data.
There are several possibilities:
- the USB backend could multiplex accesses into one single pair of
endpoints, which basically do in software what USB controllers do in hardware;
- Rework the communication protocol so that only one endpoint is
needed per
streaming device instead of two;
- Negate the problem, and consider it okay that the number of USB
endpoints
is a limiting factor.
* PCIe backend
~~~~~~~~~~~~
There is a need for a backend to support the PCIe cards populated with various chips and a FPGA with a softcore running Linux. The
communication
between the remote OS and the host OS would be something similar to
Xillybus.
* Command queues
~~~~~~~~~~~~~~
The libiio2 API bits to use the "delayed attribute write / command
queue"
feature of the kernel. A command queue would contain several commands
(e.g.
submit buffer, change a parameter, submit another buffer), that would
be
built in the upper layer of the library using a specific API, and then performed atomically on the remote device.
* Buffer overhaul
~~~~~~~~~~~~~~~
Instead of having one iio_buffer, that is either pushed or refilled,
the idea
would be to provide an API that allows the client application to
request,
enqueue or dequeue the buffers itself. This would offer much greater
control
on the buffer management to the application.
* Backwards-compatibility
~~~~~~~~~~~~~~~~~~~~~~~
Backwards compatibility is not a hard requirement, but it should
still be
possible to be done by implementing the libiio API on top of libiio2.
Things
like buffer refill/push of libiio can be implemented on top of the
new buffer
queue system. New features, like buffer metadata support wouldn't be backported.
This should be considered low-priority - we're not yet at the point
where
libiio1 is deprecated.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi Alexandru,
I wanted to delay my response to answer in a big email all the feature requests and concerns that came my way, but you've been the only one to answer so far and it's been more than one week now, so I'll answer your comments below.
Quoted 5 lines · Alexandru
Quoted 3 lines · Paul Cercueil
Hello beautiful people,
Quoted 8 lines
Hey,
Before going forward, looking at the current backlog [on Github] would be an idea. There are some PRs there which are unresolved and have some real value.
Like the plugin-support for libiio, and the Android build support. Not saying we should resolve them now, but they should get resolved in time [either implement or drop].
The plugin support for libiio was in my list, as "modular backends". The PRs which are unresolved generally mean that more work or testing is needed.
Quoted 11 lines
There's also some static-code-checker that's running, and one of them [Codacy] seems to yield some interesting stuff for C# bindings.
https://app.codacy.com/manual/commodo/libiio/dashboard
So, going forward, a few things from my side that have cropped up in some time.
I did go quickly through your list and eliminated a few [1 or 2] that were already there. If am repeating something, it is possible.
Quoted 10 lines
Bindings:
- would be nice for libiio to be present on pypi (for the Python bindings);
I am not sure if this was ever a request, but the idea popped up here-n- there
- Java-bindings for Android; personally, I'm not a fan of Java, but Android
seems to be; and there was someone trying to integrate a heart-rate sensor with libiio on Android; which makes sense, given all the sensors people are putting into phones these day
About pypi: indeed, that would be a good idea. I didn't add these "small features" to my list as they don't depend on a redesign of the library to be brought to life, which is the focus here.
Kotlin is all the rage these days in the Android world, maybe it would make more sense than Java? Or both?
Quoted 7 lines
Deployment-wise:
- there's been talk of trying to deploy libiio in Launchpad [for Ubuntu]
and something [similar to Launchpad] for CentOS/RedHat RPMs; the main benefit here is ease-of-deployment with the latest libiio; people seem to try to get the latest, and have trouble with it
I think it's important to always have the latest version in the software managers of the distributions. It's the case for Debian/Ubuntu, but I don't think it's being packaged elsewhere. Besides, if releases are made more often (e.g. with patch-level revisions, 1.0.x versions), then there shouldn't be a need for an off-the-track solution to get updates.
Quoted 6 lines
Configuration-wise:
- an alternative to XML seemed to be interesting at at time; JSON was preferred, but now I'm wondering if YAML would also be preferred [these- days]; in any case, the jump from JSON to YAML may be not as big, so JSON would be great as a first step; one argument against XML was libxml2's size
XML is only used internally, as a software representation of an IIO context. It's never there to see or modify, so I don't really see the point of switching to another markup language, besides the libxml2 problem you mention (but then again, there are other XML libraries we could switch to). Additionally, that would mean breaking the compatibility with the old libiio/IIOD, and I want to avoid that as much as possible.
Quoted 13 lines
Communication-wise:
- I started getting this idea to maybe use websockets for data-stream communication [using libwebsockets]; the whole idea would be to maybe convert/add a web-server part to IIOD. This would allow people to write web-pages [HTML + JS] that talk directly to libiio. This would simplify [and maybe accelerate] some app-development; web-browsers have really great capabilities to display animated data, and given that there are more JS devs [than C++/desktop-app devs] in this world, getting things displayed is much faster. Also, creating simple mobile-apps is really easy with JS. While it sounded that I am a JS fan, I'm actually not, but it's big enough that it's hard to avoid it, and I do feel that it has become a gateway language for web & mobile [as is C for OS development].
I don't really like the idea of having IIOD exposed to the outside world. This has ramifications that I'm not comfortable with, as it means the server software must be fully secure, provide cryptographic functionalities, etc.
Support for server-side JavaScript (via NodeJS) or PHP would probably be a good alternative, I think. The server software would communicate the IIO data to the client page the way it wants to.
Quoted 13 lines
Misc:
- throughput & loss-rate measurement: not sure if this should be part of
the library, or an example app [or both], but people have been asking this same question many times: how do i measure my throughput and how do i check for losses? at least, i was never able to give a good answer here
- profile support: this is something from Lars; but for some more complex
devices, configuration is challenging; so, something to load all configuration params, and perhaps save them back, would be interesting; this has been pushed to apps, but who knows? maybe libiio? i am not sure how valid this is
The throughput and loss-rate measurement would be easy to obtain if there was a mechanism built into IIO to report overflows/underflows. This is definitely something that we want and need, yes.
About profile support, I didn't list it here, because I consider it a side project - it should be its own library, "libiioprofile" if you want, which would use libiio, and not a mechanism built into libiio.
Quoted 4 lines
I guess for now, this is mostly it.
Not sure if I forgot anything, but these were collected from various parts of memory, notes and some issue trackers.
Thank you very much for taking the time to answer.
Cheers,
-Paul
Quoted 497 lines
Thanks
AlexQuoted 493 lines
First of all, apologies for sending such a long email. Double
apologies
if
you're caught in the email chain and you don't care about the topic. Just
don't click "reply all" when you send me an angry response, as the thread will
be publicly logged.The reason for you receiving this, is that you've been involved with libiio
(https://github.com/analogdevicesinc/libiio) either directly or indirectly,
or subscribed to the IIO kernel mailing list, or have shown
interest in
the
project, or have an homonym who does.About five years ago libiio was born, as an easy-to-use library for interfacing to Linux' Industrial Input/Output (IIO) kernel
subsystem,
either
with devices preset on the host platform, or on a remote target platform.
I think we did a good job, as the API was never broken since its
first
release,
and the praise/hate ratio we received was very encouraging. We've
seen
people
using it in all kinds of applications, from car infotainment to
space
research,
from drones to software radio.It's not perfect, though, and its current design has room for improvements,
both inside and outside the library. However, we've hit a point
where
such
improvements would require a redesign of the library and kernel
bits,
and as a
consequence, a breakage of the API. It's time for the next-gen
libiio,
and a
overhaul of the whole IIO stack, really.Your mission, should you choose to accept it, is to feed me with
ideas,
concerns, comments about what you think libiio was lacking or just
not
doing
right. We want to open the possibility for all interested parties to help
sketch the future library.I have compiled below a list of changes that we think should be
done,
comments
are very welcome. Note that emails in HTML form will probably be bounced back
by the IIO mailing list, so please write responses in plain text.Thank you for your time.
Kind regards,
Paul Cercueil------
Kernel
------* Support for buffer metadata
~~~~~~~~~~~~~~~~~~~~~~~~~~~Support tagging the buffers with any relevant information. Either
from the IIO
core for general accounting (e.g. timestamp) or from the driver
for
hardware-facing code (e.g. buffer underflow).
Metadata would be attached to a specific sample in the buffer
(metadata that
applies to the whole buffer can be attached to the first sample).
Several
entries could be attached to the same sample. The format would be
a
simple
key="value", I don't think we need anything more complex than
that.
For DACs, it should be possible to set metadata from userspace.
The
drivers
would then interpret the metadata tags if they need to.* Facility to detect overflow/underflow
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~There should be a way to detect overflows (for ADCs) and
underflows
(for
DACs), if the hardware supports it, and report them to userspace
(through a
IIO event, I suppose). This is something that could be done by the
drivers,
but maybe it would make sense to have this functionality in the
IIO
common
code?* Delayed attribute write / Command queues
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~We need a mechanism that allows us to change the value of an
attribute at a
very specific time (or sample number) after a capture or upload is
started;
some kind of script mechanism, or 'command list' analog to the
display lists
used in old GPUs. This would be used for instance in software
applications
doing time-divison duplexing (TDD).* Better high-speed buffer mechanism
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~The current buffer API is fine for low-speed devices, but we're
dealing with
ADCs and DACs of the GB/s class. ADI already contributed an
improved
(faster)
buffer mechanism, but which is incomplete (no support for DACs),
doesn't
integrate very well in the ecosystem, and is somewhat redundant
with
the old
one.The idea would be to deprecate this API and propose an alternative
that
makes use of current technology, like dmabuf. The concept would
stay
the
same, each IIO device has a pool of DMA buffers, and userspace can
queue
and dequeue buffers.The rationale behind this change, is that with the current two
APIs
it is not
possible to move data between IIO devices and a network or USB
card
without
having the CPU copy the data. This is an important problem, as
high-speed
ADCs and DACs are generally connected to FPGAs running Linux on a
softcore,
which stream their data to a workstation for further processing.
With
the
new API, the userspace software would simply obtain a pointer to a
dmabuf
from the IIO interface, and simply pass it to the network card
(this
means
the network stack and USB stack would also need to support
dmabuf).An alternative would be to keep the current file-based buffer and buffer-queue-based APIs, but enhance the former one with support
for
splicing (with splice()/vmsplice()). This might arguably be
easier to
do,
since the network stack already supports it. The problem with
splicing is
that the kernel swaps each data page with a fresh zeroed page in
order to
avoid leaking kernel memory. To be truely zero-copy, this requires
the page
cleaning mechanism to be offloaded to e.g. DMA, otherwise the
process
isn't
entirely CPU-free.* Parsable attribute names
~~~~~~~~~~~~~~~~~~~~~~~~IIO attributes follow a certain formatting that does not make it
possible to
reconstruct the devices/channels/attributes tree in userspace. The
channel
attributes are formatted as:
<direction>_<type><id>_<modifier>_<extended-name>_<attribute-name> The problem is that <extended-name>, <modifier> and
<attribute-name>
can
contain underscores, which makes it impossible to parse. For
instance,
with an attribute named "in_voltage0_high_impedence_line_enable":
Is the extended name "high_impedence_line" and the attribute name
"enable",
or is the extended name "high_impedence" and the attribute name
"line_enable"?Since the sysfs attributes is ABI, the solution could be to have a
separate
sysfs file, for instance <direction>_<type><id>_extended_name,
that
would
contain the extended name of the channel. This should be enough
for
the IIO
sysfs interface to be machine-readable.-------
Libiio2
-------* stdio redirection
~~~~~~~~~~~~~~~~~Right now libiio prints its debug information unconditionally to stdout/stderr, it should be modified to be able to output its log
to
any
valid file descriptor. While that looks simple on paper, it must
be
set up
before a library context is created, since the context creation
itself may
print debug information. This require the context allocation to be
separated
from its initialization, so a break of API.* Separate allocation from initialization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~As stated above, functions to alloc/destroy and init/deinit the
various IIO
structures should be separate; this would allow e.g. to set some
parameters
to aiio_contextbefore the context creation happens, for instance
where to
redirect the debug output.* Modular backends
~~~~~~~~~~~~~~~~Right now the backends of libiio are tightly coupled to the core.
It
would
be great if the users could install only the backends they're
interested in.
Of course, backends could still be backed into the core library,
and
that
would stay the default behaviour for non-Linux platforms.This requires two things:
- The libiio backends must be able to be updated independently of
the
main
library, therefore they should only depend on the top-level API of
libiio.
Apart from a few exceptions here and there, this is already mostly
the
case.
- There needs to be a facility to load external backends based on
the
backend name. Thanksfully with URIs this becomes easy: creating a
context
from the URI "foo:" would result in the backend module "foo" loaded
from
the disk, if not already baked in.* Event support
~~~~~~~~~~~~~The IIO subsystem has support for events. This allows the
userspace
to get
notified for instance when a temperature gets over or under a
threshold.
Right now, libiio cannot easily support events, as the
client/server
protocol of the network backend doesn't really allow it.
Therefore,
the new
libiio2 library should be designed from the ground up with
support for
events, and the API should offer a way to register a callback that
would be
called when a IIO event occurs.* Context change detection
~~~~~~~~~~~~~~~~~~~~~~~~When a device is added, removed or the context becomes unavailable
(e.g. the
USB cable was unplugged), there should be a built-in mechanism to
notify the
applications using libiio2, maybe using the same mechanism as for
IIO
events.The same would apply on the local backend, if for instance a new
device
appears, the library should be able to pick it up and report the
new
device
to the application.* Asynchronous network communication
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~The network communication between libiio and IIO is sub-par,
because
it uses
a synchronous request-response protocol. This causes the
throughput
to max
out way below the theorical maximum throughput of Gigabit
Ethernet.
The communication protocol should be modified in a way that fully
maximizes
the throughput.The old communication protocol should still be available and used
by
default
unless the two parties agree to use the new protocol. This ensures
that
TinyIIOD (the microcontroller variant of IIOD, the server that
communicates
with the network backend of libiio) can still work with the
network
backend of
libiio2. Alternatively, TinyIIOD could be updated to the new
network
protocol.
That way, the old network protocol wouldn't have to stick around.To implement the asynchronous network communication, we could
have a
look at
ZeroMQ (https://zeromq.org) which seems to be designed for that
particular
task, and is available under a LGPLv3 license. It is however
uncertain how
that would work on the IIOD side (because of the zero-copy
requirement - see
below).Additionally, the network backend should part ways with its
current
ASCII-based protocol. This made sense when libiio was first
created,
but it
requires quite a complex parser on the server side that could go
away
by
resorting to a more classic protocol.* Zero-copy
~~~~~~~~~Apart from supporting the new network protocol, libiio should
provide
a way
to stream data between IIO devices and standard file descriptors
or
sockets
without having the CPU copy the data.If a dmabuf-based IIO kernel interface is used, we need a way to
send
a
dmabuf to a socket, and I'm not sure that this is currently
possible.
But as dmabuf are internally just scatterlists, and the sockets
support
splice(), that should be somewhat doable.To complicate things, IIOD (the network server) must be able to
stream the
same data to more than one client at a time, unless we decide that
this is
no more a requirement. This means that one single dmabuf would be
submitted
to more than one socket.Streaming data to multiple clients using zero-copy techniques in
IIOD
also
implies that the data cannot be modified, which means that the
server
cannot
do the demuxing, and this task is devoted to the clients; so the
(currently
optional) server-side demuxing option would be removed.* USB3
~~~~Just like the network, the USB backend should be updated to be
able
to reach
out the maximum throughput offered by USB3.The principal problem with USB is the short number of endpoints,
which limits
the functionality, as it may not be possible to stream data from
different
devices at the same time. Right now, a pair of endpoints is
reserved
for
generic commands (read attributes, open/close IIO device, etc.),
and
one pair
is reserved per IIO device streaming data.There are several possibilities:
- the USB backend could multiplex accesses into one single pair of
endpoints, which basically do in software what USB controllers do
in
hardware;
- Rework the communication protocol so that only one endpoint is
needed per
streaming device instead of two;
- Negate the problem, and consider it okay that the number of USB
endpoints
is a limiting factor.* PCIe backend
~~~~~~~~~~~~There is a need for a backend to support the PCIe cards populated
with
various chips and a FPGA with a softcore running Linux. The
communication
between the remote OS and the host OS would be something similar
to
Xillybus.* Command queues
~~~~~~~~~~~~~~The libiio2 API bits to use the "delayed attribute write / command
queue"
feature of the kernel. A command queue would contain several
commands
(e.g.
submit buffer, change a parameter, submit another buffer), that
would
be
built in the upper layer of the library using a specific API, and
then
performed atomically on the remote device.* Buffer overhaul
~~~~~~~~~~~~~~~Instead of having one
iio_buffer, that is either pushed or
refilled,
the idea
would be to provide an API that allows the client application to
request,
enqueue or dequeue the buffers itself. This would offer much
greater
control
on the buffer management to the application.* Backwards-compatibility
~~~~~~~~~~~~~~~~~~~~~~~Backwards compatibility is not a hard requirement, but it should
still be
possible to be done by implementing the libiio API on top of
libiio2.
Things
like buffer refill/push of libiio can be implemented on top of the
new buffer
queue system. New features, like buffer metadata support wouldn't
be
backported.This should be considered low-priority - we're not yet at the
point
where
libiio1 is deprecated.
Quoted 147 lines · On Fri, 20 Dec 2019 16:28:27 +0100 Paul Cercueil <[email protected]> wrote:
Hi Alexandru,
I wanted to delay my response to answer in a big email all the feature requests and concerns that came my way, but you've been the only one to answer so far and it's been more than one week now, so I'll answer your comments below.
Quoted 7 lines · Alexandru
Quoted 3 lines · Paul Cercueil
Hello beautiful people,
Hey,
Quoted 9 lines
Before going forward, looking at the current backlog [on Github] would be
an idea. There are some PRs there which are unresolved and have some real
value.
Like the plugin-support for libiio, and the Android build support. Not saying we should resolve them now, but they should get resolved in time
[either implement or drop].The plugin support for libiio was in my list, as "modular backends". The PRs which are unresolved generally mean that more work or testing is needed.
Quoted 13 lines
There's also some static-code-checker that's running, and one of them [Codacy] seems to yield some interesting stuff for C# bindings.
https://app.codacy.com/manual/commodo/libiio/dashboard
So, going forward, a few things from my side that have cropped up in some
time.
I did go quickly through your list and eliminated a few [1 or 2] that were
already there. If am repeating something, it is possible.Quoted 13 lines
Bindings:
- would be nice for libiio to be present on pypi (for the Python bindings);
I am not sure if this was ever a request, but the idea popped up here-n-
there
- Java-bindings for Android; personally, I'm not a fan of Java, but Android
seems to be; and there was someone trying to integrate a heart-rate sensor
with libiio on Android; which makes sense, given all the sensors people are
putting into phones these dayAbout pypi: indeed, that would be a good idea. I didn't add these "small features" to my list as they don't depend on a redesign of the library to be brought to life, which is the focus here.
Kotlin is all the rage these days in the Android world, maybe it would make more sense than Java? Or both?
Quoted 8 lines
Deployment-wise:
- there's been talk of trying to deploy libiio in Launchpad [for Ubuntu]
and something [similar to Launchpad] for CentOS/RedHat RPMs; the main benefit here is ease-of-deployment with the latest libiio; people seem to
try to get the latest, and have trouble with itI think it's important to always have the latest version in the software managers of the distributions. It's the case for Debian/Ubuntu, but I don't think it's being packaged elsewhere. Besides, if releases are made more often (e.g. with patch-level revisions, 1.0.x versions), then there shouldn't be a need for an off-the-track solution to get updates.
Quoted 9 lines
Configuration-wise:
- an alternative to XML seemed to be interesting at at time; JSON was preferred, but now I'm wondering if YAML would also be preferred [these-
days]; in any case, the jump from JSON to YAML may be not as big, so JSON
would be great as a first step; one argument against XML was libxml2's sizeXML is only used internally, as a software representation of an IIO context. It's never there to see or modify, so I don't really see the point of switching to another markup language, besides the libxml2 problem you mention (but then again, there are other XML libraries we could switch to). Additionally, that would mean breaking the compatibility with the old libiio/IIOD, and I want to avoid that as much as possible.
Quoted 11 lines
Communication-wise:
- I started getting this idea to maybe use websockets for data-stream communication [using libwebsockets]; the whole idea would be to maybe convert/add a web-server part to IIOD. This would allow people to write
web-pages [HTML + JS] that talk directly to libiio. This would simplify
[and maybe accelerate] some app-development; web-browsers have really great
capabilities to display animated data, and given that there are moreQuoted 8 lines
JS
devs [than C++/desktop-app devs] in this world, getting things displayed is
much faster. Also, creating simple mobile-apps is really easy with JS. While it sounded that I am a JS fan, I'm actually not, but it's big enough
that it's hard to avoid it, and I do feel that it has become a gateway language for web & mobile [as is C for OS development].I don't really like the idea of having IIOD exposed to the outside world. This has ramifications that I'm not comfortable with, as it means the server software must be fully secure, provide cryptographic functionalities, etc.
Support for server-side JavaScript (via NodeJS) or PHP would probably be a good alternative, I think. The server software would communicate the IIO data to the client page the way it wants to.
Quoted 17 lines
Misc:
- throughput & loss-rate measurement: not sure if this should be part of
the library, or an example app [or both], but people have been asking this
same question many times: how do i measure my throughput and how do i check
for losses? at least, i was never able to give a good answer here
- profile support: this is something from Lars; but for some more complex
devices, configuration is challenging; so, something to load all configuration params, and perhaps save them back, would be interesting;
this has been pushed to apps, but who knows? maybe libiio? i am not sure
how valid this is
Quoted 3 lines
The throughput and loss-rate measurement would be easy to obtain if there was a mechanism built into IIO to report overflows/underflows. This is definitely something that we want and need, yes.
This would change how overflow/underflow are reported though. If you want stats you need a 'how much did we drop' counter. If you just care about did overflow happen you would be better off with a single event.
If we are doing overflow for DMA buffers then we can't just push one event for each scan dropped - that could be an insane number of events. Hence something cleverer would be needed.
Jonathan
Quoted 516 lines
About profile support, I didn't list it here, because I consider it a side project - it should be its own library, "libiioprofile" if you want, which would use libiio, and not a mechanism built into libiio.
Quoted 5 lines
I guess for now, this is mostly it.
Not sure if I forgot anything, but these were collected from various parts
of memory, notes and some issue trackers.Thank you very much for taking the time to answer.
Cheers,
-PaulQuoted 497 lines
Thanks
AlexQuoted 493 lines
First of all, apologies for sending such a long email. Double
apologies
if
you're caught in the email chain and you don't care about the topic. Just
don't click "reply all" when you send me an angry response, as the thread will
be publicly logged.The reason for you receiving this, is that you've been involved with libiio
(https://github.com/analogdevicesinc/libiio) either directly or indirectly,
or subscribed to the IIO kernel mailing list, or have shown
interest in
the
project, or have an homonym who does.About five years ago libiio was born, as an easy-to-use library for interfacing to Linux' Industrial Input/Output (IIO) kernel
subsystem,
either
with devices preset on the host platform, or on a remote target platform.
I think we did a good job, as the API was never broken since its
first
release,
and the praise/hate ratio we received was very encouraging. We've
seen
people
using it in all kinds of applications, from car infotainment to
space
research,
from drones to software radio.It's not perfect, though, and its current design has room for improvements,
both inside and outside the library. However, we've hit a point
where
such
improvements would require a redesign of the library and kernel
bits,
and as a
consequence, a breakage of the API. It's time for the next-gen
libiio,
and a
overhaul of the whole IIO stack, really.Your mission, should you choose to accept it, is to feed me with
ideas,
concerns, comments about what you think libiio was lacking or just
not
doing
right. We want to open the possibility for all interested parties to help
sketch the future library.I have compiled below a list of changes that we think should be
done,
comments
are very welcome. Note that emails in HTML form will probably be bounced back
by the IIO mailing list, so please write responses in plain text.Thank you for your time.
Kind regards,
Paul Cercueil------
Kernel
------* Support for buffer metadata
~~~~~~~~~~~~~~~~~~~~~~~~~~~Support tagging the buffers with any relevant information. Either
from the IIO
core for general accounting (e.g. timestamp) or from the driver
for
hardware-facing code (e.g. buffer underflow).
Metadata would be attached to a specific sample in the buffer
(metadata that
applies to the whole buffer can be attached to the first sample).
Several
entries could be attached to the same sample. The format would be
a
simple
key="value", I don't think we need anything more complex than
that.
For DACs, it should be possible to set metadata from userspace.
The
drivers
would then interpret the metadata tags if they need to.* Facility to detect overflow/underflow
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~There should be a way to detect overflows (for ADCs) and
underflows
(for
DACs), if the hardware supports it, and report them to userspace
(through a
IIO event, I suppose). This is something that could be done by the
drivers,
but maybe it would make sense to have this functionality in the
IIO
common
code?* Delayed attribute write / Command queues
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~We need a mechanism that allows us to change the value of an
attribute at a
very specific time (or sample number) after a capture or upload is
started;
some kind of script mechanism, or 'command list' analog to the
display lists
used in old GPUs. This would be used for instance in software
applications
doing time-divison duplexing (TDD).* Better high-speed buffer mechanism
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~The current buffer API is fine for low-speed devices, but we're
dealing with
ADCs and DACs of the GB/s class. ADI already contributed an
improved
(faster)
buffer mechanism, but which is incomplete (no support for DACs),
doesn't
integrate very well in the ecosystem, and is somewhat redundant
with
the old
one.The idea would be to deprecate this API and propose an alternative
that
makes use of current technology, like dmabuf. The concept would
stay
the
same, each IIO device has a pool of DMA buffers, and userspace can
queue
and dequeue buffers.The rationale behind this change, is that with the current two
APIs
it is not
possible to move data between IIO devices and a network or USB
card
without
having the CPU copy the data. This is an important problem, as
high-speed
ADCs and DACs are generally connected to FPGAs running Linux on a
softcore,
which stream their data to a workstation for further processing.
With
the
new API, the userspace software would simply obtain a pointer to a
dmabuf
from the IIO interface, and simply pass it to the network card
(this
means
the network stack and USB stack would also need to support
dmabuf).An alternative would be to keep the current file-based buffer and buffer-queue-based APIs, but enhance the former one with support
for
splicing (with splice()/vmsplice()). This might arguably be
easier to
do,
since the network stack already supports it. The problem with
splicing is
that the kernel swaps each data page with a fresh zeroed page in
order to
avoid leaking kernel memory. To be truely zero-copy, this requires
the page
cleaning mechanism to be offloaded to e.g. DMA, otherwise the
process
isn't
entirely CPU-free.* Parsable attribute names
~~~~~~~~~~~~~~~~~~~~~~~~IIO attributes follow a certain formatting that does not make it
possible to
reconstruct the devices/channels/attributes tree in userspace. The
channel
attributes are formatted as:
<direction>_<type><id>_<modifier>_<extended-name>_<attribute-name> The problem is that <extended-name>, <modifier> and
<attribute-name>
can
contain underscores, which makes it impossible to parse. For
instance,
with an attribute named "in_voltage0_high_impedence_line_enable":
Is the extended name "high_impedence_line" and the attribute name
"enable",
or is the extended name "high_impedence" and the attribute name
"line_enable"?Since the sysfs attributes is ABI, the solution could be to have a
separate
sysfs file, for instance <direction>_<type><id>_extended_name,
that
would
contain the extended name of the channel. This should be enough
for
the IIO
sysfs interface to be machine-readable.-------
Libiio2
-------* stdio redirection
~~~~~~~~~~~~~~~~~Right now libiio prints its debug information unconditionally to stdout/stderr, it should be modified to be able to output its log
to
any
valid file descriptor. While that looks simple on paper, it must
be
set up
before a library context is created, since the context creation
itself may
print debug information. This require the context allocation to be
separated
from its initialization, so a break of API.* Separate allocation from initialization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~As stated above, functions to alloc/destroy and init/deinit the
various IIO
structures should be separate; this would allow e.g. to set some
parameters
to aiio_contextbefore the context creation happens, for instance
where to
redirect the debug output.* Modular backends
~~~~~~~~~~~~~~~~Right now the backends of libiio are tightly coupled to the core.
It
would
be great if the users could install only the backends they're
interested in.
Of course, backends could still be backed into the core library,
and
that
would stay the default behaviour for non-Linux platforms.This requires two things:
- The libiio backends must be able to be updated independently of
the
main
library, therefore they should only depend on the top-level API of
libiio.
Apart from a few exceptions here and there, this is already mostly
the
case.
- There needs to be a facility to load external backends based on
the
backend name. Thanksfully with URIs this becomes easy: creating a
context
from the URI "foo:" would result in the backend module "foo" loaded
from
the disk, if not already baked in.* Event support
~~~~~~~~~~~~~The IIO subsystem has support for events. This allows the
userspace
to get
notified for instance when a temperature gets over or under a
threshold.
Right now, libiio cannot easily support events, as the
client/server
protocol of the network backend doesn't really allow it.
Therefore,
the new
libiio2 library should be designed from the ground up with
support for
events, and the API should offer a way to register a callback that
would be
called when a IIO event occurs.* Context change detection
~~~~~~~~~~~~~~~~~~~~~~~~When a device is added, removed or the context becomes unavailable
(e.g. the
USB cable was unplugged), there should be a built-in mechanism to
notify the
applications using libiio2, maybe using the same mechanism as for
IIO
events.The same would apply on the local backend, if for instance a new
device
appears, the library should be able to pick it up and report the
new
device
to the application.* Asynchronous network communication
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~The network communication between libiio and IIO is sub-par,
because
it uses
a synchronous request-response protocol. This causes the
throughput
to max
out way below the theorical maximum throughput of Gigabit
Ethernet.
The communication protocol should be modified in a way that fully
maximizes
the throughput.The old communication protocol should still be available and used
by
default
unless the two parties agree to use the new protocol. This ensures
that
TinyIIOD (the microcontroller variant of IIOD, the server that
communicates
with the network backend of libiio) can still work with the
network
backend of
libiio2. Alternatively, TinyIIOD could be updated to the new
network
protocol.
That way, the old network protocol wouldn't have to stick around.To implement the asynchronous network communication, we could
have a
look at
ZeroMQ (https://zeromq.org) which seems to be designed for that
particular
task, and is available under a LGPLv3 license. It is however
uncertain how
that would work on the IIOD side (because of the zero-copy
requirement - see
below).Additionally, the network backend should part ways with its
current
ASCII-based protocol. This made sense when libiio was first
created,
but it
requires quite a complex parser on the server side that could go
away
by
resorting to a more classic protocol.* Zero-copy
~~~~~~~~~Apart from supporting the new network protocol, libiio should
provide
a way
to stream data between IIO devices and standard file descriptors
or
sockets
without having the CPU copy the data.If a dmabuf-based IIO kernel interface is used, we need a way to
send
a
dmabuf to a socket, and I'm not sure that this is currently
possible.
But as dmabuf are internally just scatterlists, and the sockets
support
splice(), that should be somewhat doable.To complicate things, IIOD (the network server) must be able to
stream the
same data to more than one client at a time, unless we decide that
this is
no more a requirement. This means that one single dmabuf would be
submitted
to more than one socket.Streaming data to multiple clients using zero-copy techniques in
IIOD
also
implies that the data cannot be modified, which means that the
server
cannot
do the demuxing, and this task is devoted to the clients; so the
(currently
optional) server-side demuxing option would be removed.* USB3
~~~~Just like the network, the USB backend should be updated to be
able
to reach
out the maximum throughput offered by USB3.The principal problem with USB is the short number of endpoints,
which limits
the functionality, as it may not be possible to stream data from
different
devices at the same time. Right now, a pair of endpoints is
reserved
for
generic commands (read attributes, open/close IIO device, etc.),
and
one pair
is reserved per IIO device streaming data.There are several possibilities:
- the USB backend could multiplex accesses into one single pair of
endpoints, which basically do in software what USB controllers do
in
hardware;
- Rework the communication protocol so that only one endpoint is
needed per
streaming device instead of two;
- Negate the problem, and consider it okay that the number of USB
endpoints
is a limiting factor.* PCIe backend
~~~~~~~~~~~~There is a need for a backend to support the PCIe cards populated
with
various chips and a FPGA with a softcore running Linux. The
communication
between the remote OS and the host OS would be something similar
to
Xillybus.* Command queues
~~~~~~~~~~~~~~The libiio2 API bits to use the "delayed attribute write / command
queue"
feature of the kernel. A command queue would contain several
commands
(e.g.
submit buffer, change a parameter, submit another buffer), that
would
be
built in the upper layer of the library using a specific API, and
then
performed atomically on the remote device.* Buffer overhaul
~~~~~~~~~~~~~~~Instead of having one
iio_buffer, that is either pushed or
refilled,
the idea
would be to provide an API that allows the client application to
request,
enqueue or dequeue the buffers itself. This would offer much
greater
control
on the buffer management to the application.* Backwards-compatibility
~~~~~~~~~~~~~~~~~~~~~~~Backwards compatibility is not a hard requirement, but it should
still be
possible to be done by implementing the libiio API on top of
libiio2.
Things
like buffer refill/push of libiio can be implemented on top of the
new buffer
queue system. New features, like buffer metadata support wouldn't
be
backported.This should be considered low-priority - we're not yet at the
point
where
libiio1 is deprecated.
Quoted 1 line · On Mon, 09 Dec 2019 16:26:03 +0100 Paul Cercueil <[email protected]> wrote:
Hello beautiful people,
:) Good opening
Sorry for slow response on this. I read the intro and thought this will take a while and then had work snowball somewhat in the run up to xmas.
Whilst I've kept a vague eye on libiio and appreciate it certainly reduced the number of questions we got on the linux-iio@vger about the userspace interface, I'll confess I have never actually used it. As such my inputs are going to be very much centered around the kernel interface.
Quoted 65 lines
First of all, apologies for sending such a long email. Double apologies if
you're caught in the email chain and you don't care about the topic. Just
don't click "reply all" when you send me an angry response, as the thread will
be publicly logged.The reason for you receiving this, is that you've been involved with libiio
(https://github.com/analogdevicesinc/libiio) either directly or indirectly,
or subscribed to the IIO kernel mailing list, or have shown interest in the
project, or have an homonym who does.About five years ago libiio was born, as an easy-to-use library for interfacing to Linux' Industrial Input/Output (IIO) kernel subsystem, either
with devices preset on the host platform, or on a remote target platform.
I think we did a good job, as the API was never broken since its first release,
and the praise/hate ratio we received was very encouraging. We've seen people
using it in all kinds of applications, from car infotainment to space research,
from drones to software radio.It's not perfect, though, and its current design has room for improvements,
both inside and outside the library. However, we've hit a point where such
improvements would require a redesign of the library and kernel bits, and as a
consequence, a breakage of the API. It's time for the next-gen libiio, and a
overhaul of the whole IIO stack, really.Your mission, should you choose to accept it, is to feed me with ideas, concerns, comments about what you think libiio was lacking or just not doing
right. We want to open the possibility for all interested parties to help
sketch the future library.I have compiled below a list of changes that we think should be done, comments
are very welcome. Note that emails in HTML form will probably be bounced back
by the IIO mailing list, so please write responses in plain text.Thank you for your time.
Kind regards,
Paul Cercueil------
Kernel
------* Support for buffer metadata
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Quoted 14 lines
Support tagging the buffers with any relevant information. Either
from the IIO
core for general accounting (e.g. timestamp) or from the driver for hardware-facing code (e.g. buffer underflow).
Metadata would be attached to a specific sample in the buffer
(metadata that
applies to the whole buffer can be attached to the first sample).
Several
entries could be attached to the same sample. The format would be a
simple
key="value", I don't think we need anything more complex than that. For DACs, it should be possible to set metadata from userspace. The
drivers
would then interpret the metadata tags if they need to.
At the kernel level, this one is a hard problem to solve cleanly and I would imagine some of the problems apply even if we somehow insert this data 'up the stack'. What we don't want to break is random access into the data stream. So the spacing in the raw data being read back has to be predictable...
We've had a few discussions on how to do this in the kernel layer. One option was a magic 'meta data index' channel. It's been a while so I may miss some points from earlier discussions of this concept.
Basically if that channel contains 0 there is no meta data associated with a particular sample, if there is meta data then a magic 'tag' value references and entry in a separate meta data store that is read via an out of band means (ioctl or possibly a separate file handle similar to the event one).
This separates somewhat the main data flow from the metadata and allows a nice key/value type format such as you describe (rather than out of band description in sysfs). However metadata lookups become expensive... I'm not sure if that is a problem though.
For DAC meta data, we'd be parsing in kernel which is a fair bit nastier as potentially we have a large attack surface.
Quoted 4 lines
* Facility to detect overflow/underflow
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Quoted 9 lines
There should be a way to detect overflows (for ADCs) and underflows
(for
DACs), if the hardware supports it, and report them to userspace
(through a
IIO event, I suppose). This is something that could be done by the
drivers,
but maybe it would make sense to have this functionality in the IIO
common
code?
Agreed, this would be useful. So taking ADCs there are 3 cases I think.
1) Overflow of software buffer (that needs to be in core)
2) Overflow of hardware buffer (that needs to be at least partly driver specific)
3) DMA overflow (descriptor not provided in time).
It's made more fun by the fact these may be per device, or per channel (for hardware buffers or dma)
The IIO event namespace doesn't have that much room, but we can probably work out some magic coding to squeeze this in.
Every now and then there is a general kernel discussion about how to report 'unexpected / error cases' in a generic fashion. However it's never gotten far as it's a very hard problem to get right.
Quoted 4 lines
* Delayed attribute write / Command queues
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Quoted 9 lines
We need a mechanism that allows us to change the value of an
attribute at a
very specific time (or sample number) after a capture or upload is
started;
some kind of script mechanism, or 'command list' analog to the
display lists
used in old GPUs. This would be used for instance in software
applications
doing time-divison duplexing (TDD).
In kernel or in userspace?
It may be hard to put together a generic way of doing this in kernel. GPUs and similar get way with this stuff because they have very rich APIs.
If it's a rigid multiplex case, then we can do something using consumer drivers - I'm not sure how how to handle this in kernel in general.
Quoted 4 lines
* Better high-speed buffer mechanism
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Quoted 9 lines
The current buffer API is fine for low-speed devices, but we're
dealing with
ADCs and DACs of the GB/s class. ADI already contributed an improved
(faster)
buffer mechanism, but which is incomplete (no support for DACs),
doesn't
integrate very well in the ecosystem, and is somewhat redundant with
the old
one.
Agreed. That area has moved on somewhat since the IIO code was written!
Quoted 9 lines
The idea would be to deprecate this API and propose an alternative
that
makes use of current technology, like dmabuf. The concept would stay
the
same, each IIO device has a pool of DMA buffers, and userspace can
queue
and dequeue buffers.
Quoted 15 lines
The rationale behind this change, is that with the current two APIs
it is not
possible to move data between IIO devices and a network or USB card
without
having the CPU copy the data. This is an important problem, as
high-speed
ADCs and DACs are generally connected to FPGAs running Linux on a
softcore,
which stream their data to a workstation for further processing. With
the
new API, the userspace software would simply obtain a pointer to a
dmabuf
from the IIO interface, and simply pass it to the network card (this
means
the network stack and USB stack would also need to support dmabuf).
I'm curious if there are network or USB devices using dmabuf?
Quoted 14 lines
An alternative would be to keep the current file-based buffer and buffer-queue-based APIs, but enhance the former one with support for splicing (with splice()/vmsplice()). This might arguably be easier to
do,
since the network stack already supports it. The problem with
splicing is
that the kernel swaps each data page with a fresh zeroed page in
order to
avoid leaking kernel memory. To be truely zero-copy, this requires
the page
cleaning mechanism to be offloaded to e.g. DMA, otherwise the process
isn't
entirely CPU-free.
Splice also tends to put annoying limits on size of blocks. Not totally sure that is an issue here but it's a pain with crypto ;)
I hate to say it but... 'show us your code'. I'd love to get this whole area cleaned up, but I'm not sure what the right solution is.
If we can get dma_buff working well I think that approach is a more long term solution. However, there are other more experienced people in these sorts of interfaces so I'm open to alternatives!
Quoted 19 lines
* Parsable attribute names
~~~~~~~~~~~~~~~~~~~~~~~~IIO attributes follow a certain formatting that does not make it
possible to
reconstruct the devices/channels/attributes tree in userspace. The
channel
attributes are formatted as:
<direction>_<type><id>_<modifier>_<extended-name>_<attribute-name> The problem is that <extended-name>, <modifier> and <attribute-name>
can
contain underscores, which makes it impossible to parse. For instance, with an attribute named "in_voltage0_high_impedence_line_enable":
Is the extended name "high_impedence_line" and the attribute name
"enable",
or is the extended name "high_impedence" and the attribute name
"line_enable"?
Quoted 7 lines
Since the sysfs attributes is ABI, the solution could be to have a
separate
sysfs file, for instance <direction>_<type><id>_extended_name, that
would
contain the extended name of the channel. This should be enough for
the IIO
sysfs interface to be machine-readable.
Almost agreed :) Call it <direction>_<type><id>_label and I'm happy. Label tends to come from DT bindings though, so do we need that and extend_name?
extended_name was a mistake but we are stick with that as it currently is. Thankfully there aren't that many users but we can't touch the ones that do use it.
Quoted 136 lines
-------
Libiio2
-------* stdio redirection
~~~~~~~~~~~~~~~~~Right now libiio prints its debug information unconditionally to stdout/stderr, it should be modified to be able to output its log to
any
valid file descriptor. While that looks simple on paper, it must be
set up
before a library context is created, since the context creation
itself may
print debug information. This require the context allocation to be
separated
from its initialization, so a break of API.* Separate allocation from initialization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~As stated above, functions to alloc/destroy and init/deinit the
various IIO
structures should be separate; this would allow e.g. to set some
parameters
to aiio_contextbefore the context creation happens, for instance
where to
redirect the debug output.* Modular backends
~~~~~~~~~~~~~~~~Right now the backends of libiio are tightly coupled to the core. It
would
be great if the users could install only the backends they're
interested in.
Of course, backends could still be backed into the core library, and
that
would stay the default behaviour for non-Linux platforms.This requires two things:
- The libiio backends must be able to be updated independently of the
main
library, therefore they should only depend on the top-level API of
libiio.
Apart from a few exceptions here and there, this is already mostly the case.
- There needs to be a facility to load external backends based on the
backend name. Thanksfully with URIs this becomes easy: creating a
context
from the URI "foo:" would result in the backend module "foo" loaded
from
the disk, if not already baked in.* Event support
~~~~~~~~~~~~~The IIO subsystem has support for events. This allows the userspace
to get
notified for instance when a temperature gets over or under a
threshold.
Right now, libiio cannot easily support events, as the client/server protocol of the network backend doesn't really allow it. Therefore,
the new
libiio2 library should be designed from the ground up with support for events, and the API should offer a way to register a callback that
would be
called when a IIO event occurs.* Context change detection
~~~~~~~~~~~~~~~~~~~~~~~~When a device is added, removed or the context becomes unavailable
(e.g. the
USB cable was unplugged), there should be a built-in mechanism to
notify the
applications using libiio2, maybe using the same mechanism as for IIO
events.The same would apply on the local backend, if for instance a new
device
appears, the library should be able to pick it up and report the new
device
to the application.* Asynchronous network communication
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~The network communication between libiio and IIO is sub-par, because
it uses
a synchronous request-response protocol. This causes the throughput
to max
out way below the theorical maximum throughput of Gigabit Ethernet. The communication protocol should be modified in a way that fully
maximizes
the throughput.The old communication protocol should still be available and used by
default
unless the two parties agree to use the new protocol. This ensures
that
TinyIIOD (the microcontroller variant of IIOD, the server that
communicates
with the network backend of libiio) can still work with the network
backend of
libiio2. Alternatively, TinyIIOD could be updated to the new network
protocol.
That way, the old network protocol wouldn't have to stick around.To implement the asynchronous network communication, we could have a
look at
ZeroMQ (https://zeromq.org) which seems to be designed for that
particular
task, and is available under a LGPLv3 license. It is however
uncertain how
that would work on the IIOD side (because of the zero-copy
requirement - see
below).Additionally, the network backend should part ways with its current ASCII-based protocol. This made sense when libiio was first created,
but it
requires quite a complex parser on the server side that could go away
by
resorting to a more classic protocol.* Zero-copy
~~~~~~~~~Apart from supporting the new network protocol, libiio should provide
a way
to stream data between IIO devices and standard file descriptors or
sockets
without having the CPU copy the data.
Quoted 6 lines
If a dmabuf-based IIO kernel interface is used, we need a way to send
a
dmabuf to a socket, and I'm not sure that this is currently possible. But as dmabuf are internally just scatterlists, and the sockets
support
splice(), that should be somewhat doable.
I'd have a chat to the people working a lot with dma_buf (Sumit Semwal and John Stultz). Seems likely they would have at least thought about how to do network streaming.
Quoted 8 lines
To complicate things, IIOD (the network server) must be able to
stream the
same data to more than one client at a time, unless we decide that
this is
no more a requirement. This means that one single dmabuf would be
submitted
to more than one socket.
Hmm. That may be 'interesting'.
I'd definitely do some benchmarking of this. People spend a lot of time doing zero copy tricks and sometimes it's just quicker to make a copy and get rid of complexity. Hard to know until you have a prototype though.
Quoted 49 lines
Streaming data to multiple clients using zero-copy techniques in IIOD
also
implies that the data cannot be modified, which means that the server
cannot
do the demuxing, and this task is devoted to the clients; so the
(currently
optional) server-side demuxing option would be removed.* USB3
~~~~Just like the network, the USB backend should be updated to be able
to reach
out the maximum throughput offered by USB3.The principal problem with USB is the short number of endpoints,
which limits
the functionality, as it may not be possible to stream data from
different
devices at the same time. Right now, a pair of endpoints is reserved
for
generic commands (read attributes, open/close IIO device, etc.), and
one pair
is reserved per IIO device streaming data.There are several possibilities:
- the USB backend could multiplex accesses into one single pair of
endpoints, which basically do in software what USB controllers do in hardware;
- Rework the communication protocol so that only one endpoint is
needed per
streaming device instead of two;
- Negate the problem, and consider it okay that the number of USB
endpoints
is a limiting factor.* PCIe backend
~~~~~~~~~~~~There is a need for a backend to support the PCIe cards populated with various chips and a FPGA with a softcore running Linux. The
communication
between the remote OS and the host OS would be something similar to
Xillybus.* Command queues
~~~~~~~~~~~~~~
Quoted 8 lines
The libiio2 API bits to use the "delayed attribute write / command
queue"
feature of the kernel. A command queue would contain several commands
(e.g.
submit buffer, change a parameter, submit another buffer), that would
be
built in the upper layer of the library using a specific API, and then performed atomically on the remote device.
So this does indeed suggest you are thinking to put the write queueing
into the kernel. I'm not sure how we can actually do this without
ending up with really sophisticated handling in kernel (and this stuff really needs to be realtime). If I were designing a platform to do this stuff I'd push the actual synchronization out to an fpga.
Quoted 24 lines
* Buffer overhaul
~~~~~~~~~~~~~~~Instead of having one
iio_buffer, that is either pushed or refilled,
the idea
would be to provide an API that allows the client application to
request,
enqueue or dequeue the buffers itself. This would offer much greater
control
on the buffer management to the application.* Backwards-compatibility
~~~~~~~~~~~~~~~~~~~~~~~Backwards compatibility is not a hard requirement, but it should
still be
possible to be done by implementing the libiio API on top of libiio2.
Things
like buffer refill/push of libiio can be implemented on top of the
new buffer
queue system. New features, like buffer metadata support wouldn't be backported.
Quoted 3 lines
This should be considered low-priority - we're not yet at the point
where
libiio1 is deprecated.
Just a side note to this with my IIO maintainer hat on. Compatibility at the kernel interfaces is a must, but we do have a little bit of flexibility when we have good reason to think no one is using a given interface...
Some interesting ideas. Look forward to seeing how it goes!
Jonathan
Quoted 2 lines
Hi Jonathan,
Quoted 10 lines · Jonathan Cameron
Quoted 1 line · On Mon, 09 Dec 2019 16:26:03 +0100 Paul Cercueil <[email protected]> wrote:
Hello beautiful people,
:) Good opening
Sorry for slow response on this. I read the intro and thought this will take a while and then had work snowball somewhat in the run up to xmas.
No problem, I think everybody was in xmas holidays anyway.
Quoted 5 lines
Whilst I've kept a vague eye on libiio and appreciate it certainly reduced the number of questions we got on the linux-iio@vger about the userspace interface, I'll confess I have never actually used it. As such my inputs are going to be very much centered around the kernel interface.
Which is perfectly fine, that's the kind of input I was expecting from you :)
Quoted 112 lines
Quoted 87 lines
First of all, apologies for sending such a long email. Double
apologies
if
you're caught in the email chain and you don't care about the topic. Just
don't click "reply all" when you send me an angry response, as the thread will
be publicly logged.The reason for you receiving this, is that you've been involved with libiio
(https://github.com/analogdevicesinc/libiio) either directly or indirectly,
or subscribed to the IIO kernel mailing list, or have shown
interest in
the
project, or have an homonym who does.About five years ago libiio was born, as an easy-to-use library for interfacing to Linux' Industrial Input/Output (IIO) kernel
subsystem,
either
with devices preset on the host platform, or on a remote target platform.
I think we did a good job, as the API was never broken since its
first
release,
and the praise/hate ratio we received was very encouraging. We've
seen
people
using it in all kinds of applications, from car infotainment to
space
research,
from drones to software radio.It's not perfect, though, and its current design has room for improvements,
both inside and outside the library. However, we've hit a point
where
such
improvements would require a redesign of the library and kernel
bits,
and as a
consequence, a breakage of the API. It's time for the next-gen
libiio,
and a
overhaul of the whole IIO stack, really.Your mission, should you choose to accept it, is to feed me with
ideas,
concerns, comments about what you think libiio was lacking or just
not
doing
right. We want to open the possibility for all interested parties to help
sketch the future library.I have compiled below a list of changes that we think should be
done,
comments
are very welcome. Note that emails in HTML form will probably be bounced back
by the IIO mailing list, so please write responses in plain text.Thank you for your time.
Kind regards,
Paul Cercueil------
Kernel
------* Support for buffer metadata
~~~~~~~~~~~~~~~~~~~~~~~~~~~Support tagging the buffers with any relevant information. Either
from the IIO
core for general accounting (e.g. timestamp) or from the driver
for
hardware-facing code (e.g. buffer underflow).
Metadata would be attached to a specific sample in the buffer
(metadata that
applies to the whole buffer can be attached to the first sample).
Several
entries could be attached to the same sample. The format would beQuoted 8 lines
a
simple
key="value", I don't think we need anything more complex than
that.
For DACs, it should be possible to set metadata from userspace.
The
drivers
would then interpret the metadata tags if they need to.At the kernel level, this one is a hard problem to solve cleanly and I would imagine some of the problems apply even if we somehow insert this data 'up the stack'. What we don't want to break is random access into the data stream. So the spacing in the raw data being read back has to be predictable...
We've had a few discussions on how to do this in the kernel layer. One option was a magic 'meta data index' channel. It's been a while so I may miss some points from earlier discussions of this concept.
Basically if that channel contains 0 there is no meta data associated with a particular sample, if there is meta data then a magic 'tag' value references and entry in a separate meta data store that is read via an out of band means (ioctl or possibly a separate file handle similar to the event one).
Quoted 4 lines
This separates somewhat the main data flow from the metadata and allows a nice key/value type format such as you describe (rather than out of band description in sysfs). However metadata lookups become expensive... I'm not sure if that is a problem though.
That's more or less what I had in mind, yes.
Quoted 2 lines
For DAC meta data, we'd be parsing in kernel which is a fair bit nastier as potentially we have a large attack surface.
I would think that the default handler could be in the IIO core and just handle "iio_filename=value", as if a sysfs file was written. For now I don't think any driver needs to handle any custom metadata tags.
Quoted 16 lines
Quoted 4 lines
* Facility to detect overflow/underflow
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Quoted 11 lines
There should be a way to detect overflows (for ADCs) and
underflows
(for
DACs), if the hardware supports it, and report them to userspace
(through a
IIO event, I suppose). This is something that could be done by the
drivers,
but maybe it would make sense to have this functionality in the
IIO
common
code?
Quoted 8 lines
Agreed, this would be useful. So taking ADCs there are 3 cases I think.
1) Overflow of software buffer (that needs to be in core)
2) Overflow of hardware buffer (that needs to be at least partly driver specific)
3) DMA overflow (descriptor not provided in time).It's made more fun by the fact these may be per device, or per channel (for hardware buffers or dma)
Do userspace needs to know which one of the three cases happens? I believe (I may be wrong) it's enough to know that an overflow happened, which means that the data in the buffer is non-contiguous; that's all the apps probably care about.
Quoted 21 lines
The IIO event namespace doesn't have that much room, but we can probably work out some magic coding to squeeze this in.
Every now and then there is a general kernel discussion about how to report 'unexpected / error cases' in a generic fashion. However it's never gotten far as it's a very hard problem to get right.
Quoted 4 lines
* Delayed attribute write / Command queues
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Quoted 9 lines
We need a mechanism that allows us to change the value of an
attribute at a
very specific time (or sample number) after a capture or upload is
started;
some kind of script mechanism, or 'command list' analog to the
display lists
used in old GPUs. This would be used for instance in software
applications
doing time-divison duplexing (TDD).
Quoted 7 lines
In kernel or in userspace?
It may be hard to put together a generic way of doing this in kernel. GPUs and similar get way with this stuff because they have very rich APIs.
If it's a rigid multiplex case, then we can do something using consumer drivers - I'm not sure how how to handle this in kernel in general.
This could be done with "iio_filename=value" metadata tags linked to specific samples, I believe.
Quoted 49 lines
Quoted 4 lines
* Better high-speed buffer mechanism
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Quoted 11 lines
The current buffer API is fine for low-speed devices, but we're
dealing with
ADCs and DACs of the GB/s class. ADI already contributed an
improved
(faster)
buffer mechanism, but which is incomplete (no support for DACs),
doesn't
integrate very well in the ecosystem, and is somewhat redundant
with
the old
one.Agreed. That area has moved on somewhat since the IIO code was written!
Quoted 22 lines
The idea would be to deprecate this API and propose an alternative
that
makes use of current technology, like dmabuf. The concept would
stay
the
same, each IIO device has a pool of DMA buffers, and userspace can
queue
and dequeue buffers.The rationale behind this change, is that with the current two
APIs
it is not
possible to move data between IIO devices and a network or USB
card
without
having the CPU copy the data. This is an important problem, as
high-speed
ADCs and DACs are generally connected to FPGAs running Linux on a
softcore,
which stream their data to a workstation for further processing.
WithQuoted 8 lines
the
new API, the userspace software would simply obtain a pointer to a
dmabuf
from the IIO interface, and simply pass it to the network card
(this
means
the network stack and USB stack would also need to support
dmabuf).
Quoted 1 line
I'm curious if there are network or USB devices using dmabuf?
Not that I'm aware of, but I may have overlooked it. We're fishing in exotic seas here.
Quoted 18 lines
Quoted 17 lines
An alternative would be to keep the current file-based buffer and buffer-queue-based APIs, but enhance the former one with support
for
splicing (with splice()/vmsplice()). This might arguably be
easier to
do,
since the network stack already supports it. The problem with
splicing is
that the kernel swaps each data page with a fresh zeroed page in
order to
avoid leaking kernel memory. To be truely zero-copy, this requires
the page
cleaning mechanism to be offloaded to e.g. DMA, otherwise the
process
isn't
entirely CPU-free.
Quoted 2 lines
Splice also tends to put annoying limits on size of blocks. Not totally sure that is an issue here but it's a pain with crypto ;)
Splice also works with memory pages, which puts more constraints on the hardware since the DMA must support scatter-gather.
Quoted 2 lines
I hate to say it but... 'show us your code'. I'd love to get this whole area cleaned up, but I'm not sure what the right solution is.
I'll show the code once I know I'm starting with the right design ;)
Quoted 35 lines
If we can get
dma_buffworking well I think that approach is a more long term solution. However, there are other more experienced people in these sorts of interfaces so I'm open to alternatives!Quoted 21 lines
* Parsable attribute names
~~~~~~~~~~~~~~~~~~~~~~~~IIO attributes follow a certain formatting that does not make it
possible to
reconstruct the devices/channels/attributes tree in userspace. The
channel
attributes are formatted as:
<direction>_<type><id>_<modifier>_<extended-name>_<attribute-name> The problem is that <extended-name>, <modifier> and
<attribute-name>
can
contain underscores, which makes it impossible to parse. For
instance,
with an attribute named "in_voltage0_high_impedence_line_enable":
Is the extended name "high_impedence_line" and the attribute name
"enable",
or is the extended name "high_impedence" and the attribute name
"line_enable"?Quoted 9 lines
Since the sysfs attributes is ABI, the solution could be to have a
separate
sysfs file, for instance <direction>_<type><id>_extended_name,
that
would
contain the extended name of the channel. This should be enough
for
the IIO
sysfs interface to be machine-readable.
Quoted 3 lines
Almost agreed :) Call it <direction>_<type><id>_label and I'm happy. Label tends to come from DT bindings though, so do we need that and
extend_name?
Great that we agree on this. We can work out the details when I send the patchset ;)
Quoted 174 lines
extended_namewas a mistake but we are stick with that as it currently is. Thankfully there aren't that many users but we can't touch the ones that do use it.Quoted 161 lines
-------
Libiio2
-------* stdio redirection
~~~~~~~~~~~~~~~~~Right now libiio prints its debug information unconditionally to stdout/stderr, it should be modified to be able to output its log
to
any
valid file descriptor. While that looks simple on paper, it must
be
set up
before a library context is created, since the context creation
itself may
print debug information. This require the context allocation to be
separated
from its initialization, so a break of API.* Separate allocation from initialization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~As stated above, functions to alloc/destroy and init/deinit the
various IIO
structures should be separate; this would allow e.g. to set some
parameters
to aiio_contextbefore the context creation happens, for instance
where to
redirect the debug output.* Modular backends
~~~~~~~~~~~~~~~~Right now the backends of libiio are tightly coupled to the core.
It
would
be great if the users could install only the backends they're
interested in.
Of course, backends could still be backed into the core library,
and
that
would stay the default behaviour for non-Linux platforms.This requires two things:
- The libiio backends must be able to be updated independently of
the
main
library, therefore they should only depend on the top-level API of
libiio.
Apart from a few exceptions here and there, this is already mostly
the
case.
- There needs to be a facility to load external backends based on
the
backend name. Thanksfully with URIs this becomes easy: creating a
context
from the URI "foo:" would result in the backend module "foo" loaded
from
the disk, if not already baked in.* Event support
~~~~~~~~~~~~~The IIO subsystem has support for events. This allows the
userspace
to get
notified for instance when a temperature gets over or under a
threshold.
Right now, libiio cannot easily support events, as the
client/server
protocol of the network backend doesn't really allow it.
Therefore,
the new
libiio2 library should be designed from the ground up with
support for
events, and the API should offer a way to register a callback that
would be
called when a IIO event occurs.* Context change detection
~~~~~~~~~~~~~~~~~~~~~~~~When a device is added, removed or the context becomes unavailable
(e.g. the
USB cable was unplugged), there should be a built-in mechanism to
notify the
applications using libiio2, maybe using the same mechanism as for
IIO
events.The same would apply on the local backend, if for instance a new
device
appears, the library should be able to pick it up and report the
new
device
to the application.* Asynchronous network communication
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~The network communication between libiio and IIO is sub-par,
because
it uses
a synchronous request-response protocol. This causes the
throughput
to max
out way below the theorical maximum throughput of Gigabit
Ethernet.
The communication protocol should be modified in a way that fully
maximizes
the throughput.The old communication protocol should still be available and used
by
default
unless the two parties agree to use the new protocol. This ensures
that
TinyIIOD (the microcontroller variant of IIOD, the server that
communicates
with the network backend of libiio) can still work with the
network
backend of
libiio2. Alternatively, TinyIIOD could be updated to the new
network
protocol.
That way, the old network protocol wouldn't have to stick around.To implement the asynchronous network communication, we could
have a
look at
ZeroMQ (https://zeromq.org) which seems to be designed for that
particular
task, and is available under a LGPLv3 license. It is however
uncertain how
that would work on the IIOD side (because of the zero-copy
requirement - see
below).Additionally, the network backend should part ways with its
current
ASCII-based protocol. This made sense when libiio was first
created,
but it
requires quite a complex parser on the server side that could go
away
by
resorting to a more classic protocol.* Zero-copy
~~~~~~~~~Apart from supporting the new network protocol, libiio should
provide
a way
to stream data between IIO devices and standard file descriptors
or
sockets
without having the CPU copy the data.Quoted 8 lines
If a dmabuf-based IIO kernel interface is used, we need a way to
send
a
dmabuf to a socket, and I'm not sure that this is currently
possible.
But as dmabuf are internally just scatterlists, and the sockets
support
splice(), that should be somewhat doable.
Quoted 3 lines
I'd have a chat to the people working a lot with
dma_buf(Sumit Semwal and John Stultz). Seems likely they would have at least thought about how to do network streaming.
Thanks, I'll drop them an email.
Quoted 9 lines
Quoted 8 lines
To complicate things, IIOD (the network server) must be able to
stream the
same data to more than one client at a time, unless we decide that
this is
no more a requirement. This means that one single dmabuf would be
submitted
to more than one socket.
Quoted 6 lines
Hmm. That may be 'interesting'.
I'd definitely do some benchmarking of this. People spend a lot of time doing zero copy tricks and sometimes it's just quicker to make a copy and get rid of complexity. Hard to know until you have a prototype though.
Definitely, there will be benchmarks with small test apps before I implement anything in libiio.
About zerocopy vs. copy: another aspect is that copying from a mmapped DMA buffer means copying from uncached memory, which quickly becomes the bottleneck at high speeds.
Quoted 70 lines
Quoted 58 lines
Streaming data to multiple clients using zero-copy techniques in
IIOD
also
implies that the data cannot be modified, which means that the
server
cannot
do the demuxing, and this task is devoted to the clients; so the
(currently
optional) server-side demuxing option would be removed.* USB3
~~~~Just like the network, the USB backend should be updated to be
able
to reach
out the maximum throughput offered by USB3.The principal problem with USB is the short number of endpoints,
which limits
the functionality, as it may not be possible to stream data from
different
devices at the same time. Right now, a pair of endpoints is
reserved
for
generic commands (read attributes, open/close IIO device, etc.),
and
one pair
is reserved per IIO device streaming data.There are several possibilities:
- the USB backend could multiplex accesses into one single pair of
endpoints, which basically do in software what USB controllers do
in
hardware;
- Rework the communication protocol so that only one endpoint is
needed per
streaming device instead of two;
- Negate the problem, and consider it okay that the number of USB
endpoints
is a limiting factor.* PCIe backend
~~~~~~~~~~~~There is a need for a backend to support the PCIe cards populated
with
various chips and a FPGA with a softcore running Linux. The
communication
between the remote OS and the host OS would be something similar
to
Xillybus.* Command queues
~~~~~~~~~~~~~~Quoted 11 lines
The libiio2 API bits to use the "delayed attribute write / command
queue"
feature of the kernel. A command queue would contain several
commands
(e.g.
submit buffer, change a parameter, submit another buffer), that
would
be
built in the upper layer of the library using a specific API, and
then
performed atomically on the remote device.
Quoted 5 lines
So this does indeed suggest you are thinking to put the write queueing
into the kernel. I'm not sure how we can actually do this without
ending up with really sophisticated handling in kernel (and this stuff really needs to be realtime). If I were designing a platform to do this stuff I'd push the actual synchronization out to an fpga.
Honestly I'm not sure where in the stack it belongs. It would make sense to have it in the FPGA, but then that's a non-standard feature and there is no point in having that supported in libiio, which is meant to be generic.
If the kernel supports "iio_filename=value" metadata for DACs, it would be possible to change a parameter at a very specific moment in time, but I don't know if that's enough to cover all applications. That's a question for @analog.com guys I believe.
Quoted 33 lines
Quoted 28 lines
* Buffer overhaul
~~~~~~~~~~~~~~~Instead of having one
iio_buffer, that is either pushed or
refilled,
the idea
would be to provide an API that allows the client application to
request,
enqueue or dequeue the buffers itself. This would offer much
greater
control
on the buffer management to the application.* Backwards-compatibility
~~~~~~~~~~~~~~~~~~~~~~~Backwards compatibility is not a hard requirement, but it should
still be
possible to be done by implementing the libiio API on top of
libiio2.
Things
like buffer refill/push of libiio can be implemented on top of the
new buffer
queue system. New features, like buffer metadata support wouldn't
be
backported.Quoted 4 lines
This should be considered low-priority - we're not yet at the
point
where
libiio1 is deprecated.
Quoted 8 lines
Just a side note to this with my IIO maintainer hat on. Compatibility at the kernel interfaces is a must, but we do have a little bit of flexibility when we have good reason to think no one is using a given interface...
Some interesting ideas. Look forward to seeing how it goes!
Jonathan
Great, thank you for the feedback!
-Paul
Quoted 148 lines · On Fri, 3 Jan 2020 03:10:19 +0100 Paul Cercueil <[email protected]> wrote:
Hi Jonathan,
Quoted 10 lines · Jonathan Cameron
Quoted 1 line · On Mon, 09 Dec 2019 16:26:03 +0100 Paul Cercueil <[email protected]> wrote:
Hello beautiful people,
:) Good opening
Sorry for slow response on this. I read the intro and thought this will take a while and then had work snowball somewhat in the run up to xmas.
No problem, I think everybody was in xmas holidays anyway.
Quoted 5 lines
Whilst I've kept a vague eye on libiio and appreciate it certainly reduced the number of questions we got on the linux-iio@vger about the userspace interface, I'll confess I have never actually used it. As such my inputs are going to be very much centered around the kernel interface.
Which is perfectly fine, that's the kind of input I was expecting from you :)
Quoted 112 lines
Quoted 87 lines
First of all, apologies for sending such a long email. Double
apologies
if
you're caught in the email chain and you don't care about the topic. Just
don't click "reply all" when you send me an angry response, as the thread will
be publicly logged.The reason for you receiving this, is that you've been involved with libiio
(https://github.com/analogdevicesinc/libiio) either directly or indirectly,
or subscribed to the IIO kernel mailing list, or have shown
interest in
the
project, or have an homonym who does.About five years ago libiio was born, as an easy-to-use library for interfacing to Linux' Industrial Input/Output (IIO) kernel
subsystem,
either
with devices preset on the host platform, or on a remote target platform.
I think we did a good job, as the API was never broken since its
first
release,
and the praise/hate ratio we received was very encouraging. We've
seen
people
using it in all kinds of applications, from car infotainment to
space
research,
from drones to software radio.It's not perfect, though, and its current design has room for improvements,
both inside and outside the library. However, we've hit a point
where
such
improvements would require a redesign of the library and kernel
bits,
and as a
consequence, a breakage of the API. It's time for the next-gen
libiio,
and a
overhaul of the whole IIO stack, really.Your mission, should you choose to accept it, is to feed me with
ideas,
concerns, comments about what you think libiio was lacking or just
not
doing
right. We want to open the possibility for all interested parties to help
sketch the future library.I have compiled below a list of changes that we think should be
done,
comments
are very welcome. Note that emails in HTML form will probably be bounced back
by the IIO mailing list, so please write responses in plain text.Thank you for your time.
Kind regards,
Paul Cercueil------
Kernel
------* Support for buffer metadata
~~~~~~~~~~~~~~~~~~~~~~~~~~~Support tagging the buffers with any relevant information. Either
from the IIO
core for general accounting (e.g. timestamp) or from the driver
for
hardware-facing code (e.g. buffer underflow).
Metadata would be attached to a specific sample in the buffer
(metadata that
applies to the whole buffer can be attached to the first sample).
Several
entries could be attached to the same sample. The format would beQuoted 8 lines
a
simple
key="value", I don't think we need anything more complex than
that.
For DACs, it should be possible to set metadata from userspace.
The
drivers
would then interpret the metadata tags if they need to.At the kernel level, this one is a hard problem to solve cleanly and I would imagine some of the problems apply even if we somehow insert this data 'up the stack'. What we don't want to break is random access into the data stream. So the spacing in the raw data being read back has to be predictable...
We've had a few discussions on how to do this in the kernel layer. One option was a magic 'meta data index' channel. It's been a while so I may miss some points from earlier discussions of this concept.
Basically if that channel contains 0 there is no meta data associated with a particular sample, if there is meta data then a magic 'tag' value references and entry in a separate meta data store that is read via an out of band means (ioctl or possibly a separate file handle similar to the event one).
Quoted 5 lines
This separates somewhat the main data flow from the metadata and allows
a nice key/value type format such as you describe (rather than out of band description in sysfs). However metadata lookups become expensive... I'm not sure if that is a problem though.That's more or less what I had in mind, yes.
Quoted 6 lines
Quoted 2 lines
For DAC meta data, we'd be parsing in kernel which is a fair bit nastier as potentially we have a large attack surface.
I would think that the default handler could be in the IIO core and just handle "iio_filename=value", as if a sysfs file was written. For now I don't think any driver needs to handle any custom metadata tags.
OK. Let's see what that looks like. (or at least a partial prototype looks like :)
Quoted 25 lines
Quoted 25 lines
Quoted 5 lines
* Facility to detect overflow/underflow
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Quoted 11 lines
There should be a way to detect overflows (for ADCs) and
underflows
(for
DACs), if the hardware supports it, and report them to userspace
(through a
IIO event, I suppose). This is something that could be done by the
drivers,
but maybe it would make sense to have this functionality in the
IIO
common
code?Agreed, this would be useful. So taking ADCs there are 3 cases I think.
1) Overflow of software buffer (that needs to be in core)
2) Overflow of hardware buffer (that needs to be at least partly driver specific)
3) DMA overflow (descriptor not provided in time).
Quoted 8 lines
Quoted 3 lines
It's made more fun by the fact these may be per device, or per channel (for
hardware buffers or dma)Do userspace needs to know which one of the three cases happens? I believe (I may be wrong) it's enough to know that an overflow happened, which means that the data in the buffer is non-contiguous; that's all the apps probably care about.
Agreed. They may all map to a particular signal to userspace, but the handling ends up in different places (all calling the same function in their handlers). Probably doesn't matter from userspace point of view.
Quoted 30 lines
Quoted 30 lines
The IIO event namespace doesn't have that much room, but we can probably
work out some magic coding to squeeze this in.Every now and then there is a general kernel discussion about how to report 'unexpected / error cases' in a generic fashion. However it's never
gotten far as it's a very hard problem to get right.Quoted 4 lines
* Delayed attribute write / Command queues
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Quoted 9 lines
We need a mechanism that allows us to change the value of an
attribute at a
very specific time (or sample number) after a capture or upload is
started;
some kind of script mechanism, or 'command list' analog to the
display lists
used in old GPUs. This would be used for instance in software
applications
doing time-divison duplexing (TDD).In kernel or in userspace?
It may be hard to put together a generic way of doing this in kernel. GPUs
and similar get way with this stuff because they have very rich APIs.
Quoted 6 lines
Quoted 3 lines
If it's a rigid multiplex case, then we can do something using consumer
drivers - I'm not sure how how to handle this in kernel in general.This could be done with "iio_filename=value" metadata tags linked to specific samples, I believe.
For output devices, that would work. For input devices do we propose to have a kind of parallel buffer for this stuff?
Either way, to get something efficient may be 'interesting' and require some caching etc so the fairly heavy weight meta data processing isn't happening in the fast path of sampling.
Quoted 52 lines
Quoted 52 lines
Quoted 6 lines
* Better high-speed buffer mechanism
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Quoted 11 lines
The current buffer API is fine for low-speed devices, but we're
dealing with
ADCs and DACs of the GB/s class. ADI already contributed an
improved
(faster)
buffer mechanism, but which is incomplete (no support for DACs),
doesn't
integrate very well in the ecosystem, and is somewhat redundant
with
the old
one.Agreed. That area has moved on somewhat since the IIO code was written!
Quoted 30 lines
The idea would be to deprecate this API and propose an alternative
that
makes use of current technology, like dmabuf. The concept would
stay
the
same, each IIO device has a pool of DMA buffers, and userspace can
queue
and dequeue buffers.The rationale behind this change, is that with the current two
APIs
it is not
possible to move data between IIO devices and a network or USB
card
without
having the CPU copy the data. This is an important problem, as
high-speed
ADCs and DACs are generally connected to FPGAs running Linux on a
softcore,
which stream their data to a workstation for further processing.
With
the
new API, the userspace software would simply obtain a pointer to a
dmabuf
from the IIO interface, and simply pass it to the network card
(this
means
the network stack and USB stack would also need to support
dmabuf).
Quoted 4 lines
Quoted 1 line
I'm curious if there are network or USB devices using dmabuf?
Not that I'm aware of, but I may have overlooked it. We're fishing in exotic seas here.
Definitely. Needs some exploration. We don't want to pick a route for IIO on the assumption that other subsystems will move in a fashion that helps us to find they go a different way.
Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
An alternative would be to keep the current file-based buffer and buffer-queue-based APIs, but enhance the former one with support
for
splicing (with splice()/vmsplice()). This might arguably be
easier to
do,
since the network stack already supports it. The problem with
splicing is
that the kernel swaps each data page with a fresh zeroed page in
order to
avoid leaking kernel memory. To be truely zero-copy, this requires
the page
cleaning mechanism to be offloaded to e.g. DMA, otherwise the
process
isn't
entirely CPU-free.
Quoted 6 lines
Quoted 3 lines
Splice also tends to put annoying limits on size of blocks. Not totally
sure that is an issue here but it's a pain with crypto ;)Splice also works with memory pages, which puts more constraints on the hardware since the DMA must support scatter-gather.
True. Though for any non trivial DMA that's pretty much a requirement anyway - or CMA pain occurs.
Quoted 5 lines
Quoted 3 lines
I hate to say it but... 'show us your code'. I'd love to get this whole area cleaned up, but I'm not sure what the right solution is.
I'll show the code once I know I'm starting with the right design ;)
:) You know you'll never be sure until prototypes are done!
Quoted 239 lines
Quoted 36 lines
If we can get
dma_buffworking well I think that approach is a more long term solution. However, there are other more experienced people in these sorts of interfaces so I'm open to alternatives!Quoted 21 lines
* Parsable attribute names
~~~~~~~~~~~~~~~~~~~~~~~~IIO attributes follow a certain formatting that does not make it
possible to
reconstruct the devices/channels/attributes tree in userspace. The
channel
attributes are formatted as:
<direction>_<type><id>_<modifier>_<extended-name>_<attribute-name> The problem is that <extended-name>, <modifier> and
<attribute-name>
can
contain underscores, which makes it impossible to parse. For
instance,
with an attribute named "in_voltage0_high_impedence_line_enable":
Is the extended name "high_impedence_line" and the attribute name
"enable",
or is the extended name "high_impedence" and the attribute name
"line_enable"?Quoted 9 lines
Since the sysfs attributes is ABI, the solution could be to have a
separate
sysfs file, for instance <direction>_<type><id>_extended_name,
that
would
contain the extended name of the channel. This should be enough
for
the IIO
sysfs interface to be machine-readable.Quoted 3 lines
Almost agreed :) Call it <direction>_<type><id>_label and I'm happy. Label tends to come from DT bindings though, so do we need that and
extend_name?Great that we agree on this. We can work out the details when I send the patchset ;)
Quoted 174 lines
extended_namewas a mistake but we are stick with that as it currently is. Thankfully there aren't that many users but we can't touch the ones that do use it.Quoted 161 lines
-------
Libiio2
-------* stdio redirection
~~~~~~~~~~~~~~~~~Right now libiio prints its debug information unconditionally to stdout/stderr, it should be modified to be able to output its log
to
any
valid file descriptor. While that looks simple on paper, it must
be
set up
before a library context is created, since the context creation
itself may
print debug information. This require the context allocation to be
separated
from its initialization, so a break of API.* Separate allocation from initialization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~As stated above, functions to alloc/destroy and init/deinit the
various IIO
structures should be separate; this would allow e.g. to set some
parameters
to aiio_contextbefore the context creation happens, for instance
where to
redirect the debug output.* Modular backends
~~~~~~~~~~~~~~~~Right now the backends of libiio are tightly coupled to the core.
It
would
be great if the users could install only the backends they're
interested in.
Of course, backends could still be backed into the core library,
and
that
would stay the default behaviour for non-Linux platforms.This requires two things:
- The libiio backends must be able to be updated independently of
the
main
library, therefore they should only depend on the top-level API of
libiio.
Apart from a few exceptions here and there, this is already mostly
the
case.
- There needs to be a facility to load external backends based on
the
backend name. Thanksfully with URIs this becomes easy: creating a
context
from the URI "foo:" would result in the backend module "foo" loaded
from
the disk, if not already baked in.* Event support
~~~~~~~~~~~~~The IIO subsystem has support for events. This allows the
userspace
to get
notified for instance when a temperature gets over or under a
threshold.
Right now, libiio cannot easily support events, as the
client/server
protocol of the network backend doesn't really allow it.
Therefore,
the new
libiio2 library should be designed from the ground up with
support for
events, and the API should offer a way to register a callback that
would be
called when a IIO event occurs.* Context change detection
~~~~~~~~~~~~~~~~~~~~~~~~When a device is added, removed or the context becomes unavailable
(e.g. the
USB cable was unplugged), there should be a built-in mechanism to
notify the
applications using libiio2, maybe using the same mechanism as for
IIO
events.The same would apply on the local backend, if for instance a new
device
appears, the library should be able to pick it up and report the
new
device
to the application.* Asynchronous network communication
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~The network communication between libiio and IIO is sub-par,
because
it uses
a synchronous request-response protocol. This causes the
throughput
to max
out way below the theorical maximum throughput of Gigabit
Ethernet.
The communication protocol should be modified in a way that fully
maximizes
the throughput.The old communication protocol should still be available and used
by
default
unless the two parties agree to use the new protocol. This ensures
that
TinyIIOD (the microcontroller variant of IIOD, the server that
communicates
with the network backend of libiio) can still work with the
network
backend of
libiio2. Alternatively, TinyIIOD could be updated to the new
network
protocol.
That way, the old network protocol wouldn't have to stick around.To implement the asynchronous network communication, we could
have a
look at
ZeroMQ (https://zeromq.org) which seems to be designed for that
particular
task, and is available under a LGPLv3 license. It is however
uncertain how
that would work on the IIOD side (because of the zero-copy
requirement - see
below).Additionally, the network backend should part ways with its
current
ASCII-based protocol. This made sense when libiio was first
created,
but it
requires quite a complex parser on the server side that could go
away
by
resorting to a more classic protocol.* Zero-copy
~~~~~~~~~Apart from supporting the new network protocol, libiio should
provide
a way
to stream data between IIO devices and standard file descriptors
or
sockets
without having the CPU copy the data.Quoted 8 lines
If a dmabuf-based IIO kernel interface is used, we need a way to
send
a
dmabuf to a socket, and I'm not sure that this is currently
possible.
But as dmabuf are internally just scatterlists, and the sockets
support
splice(), that should be somewhat doable.Quoted 3 lines
I'd have a chat to the people working a lot with
dma_buf(Sumit Semwal and John Stultz). Seems likely they would have at least thought about how to do network streaming.Thanks, I'll drop them an email.
Quoted 9 lines
Quoted 8 lines
To complicate things, IIOD (the network server) must be able to
stream the
same data to more than one client at a time, unless we decide that
this is
no more a requirement. This means that one single dmabuf would be
submitted
to more than one socket.Quoted 6 lines
Hmm. That may be 'interesting'.
I'd definitely do some benchmarking of this. People spend a lot of time doing zero copy tricks and sometimes it's just quicker to make a copy and get rid of complexity. Hard to know until you have a prototype though.
Quoted 6 lines
Definitely, there will be benchmarks with small test apps before I implement anything in libiio.
About zerocopy vs. copy: another aspect is that copying from a mmapped DMA buffer means copying from uncached memory, which quickly becomes the bottleneck at high speeds.
True enough. You should get some hardware that will do full shared virtual addressing / page faulting etc. Gets around these annoying limitations ;) Can't see that happening on embedded devices for a while yet though *sigh*
Quoted 78 lines
Quoted 71 lines
Quoted 59 lines
Streaming data to multiple clients using zero-copy techniques in
IIOD
also
implies that the data cannot be modified, which means that the
server
cannot
do the demuxing, and this task is devoted to the clients; so the
(currently
optional) server-side demuxing option would be removed.* USB3
~~~~Just like the network, the USB backend should be updated to be
able
to reach
out the maximum throughput offered by USB3.The principal problem with USB is the short number of endpoints,
which limits
the functionality, as it may not be possible to stream data from
different
devices at the same time. Right now, a pair of endpoints is
reserved
for
generic commands (read attributes, open/close IIO device, etc.),
and
one pair
is reserved per IIO device streaming data.There are several possibilities:
- the USB backend could multiplex accesses into one single pair of
endpoints, which basically do in software what USB controllers do
in
hardware;
- Rework the communication protocol so that only one endpoint is
needed per
streaming device instead of two;
- Negate the problem, and consider it okay that the number of USB
endpoints
is a limiting factor.* PCIe backend
~~~~~~~~~~~~There is a need for a backend to support the PCIe cards populated
with
various chips and a FPGA with a softcore running Linux. The
communication
between the remote OS and the host OS would be something similar
to
Xillybus.* Command queues
~~~~~~~~~~~~~~Quoted 11 lines
The libiio2 API bits to use the "delayed attribute write / command
queue"
feature of the kernel. A command queue would contain several
commands
(e.g.
submit buffer, change a parameter, submit another buffer), that
would
be
built in the upper layer of the library using a specific API, and
then
performed atomically on the remote device.Quoted 6 lines
So this does indeed suggest you are thinking to put the write queueing
into the kernel. I'm not sure how we can actually do this without
ending up with really sophisticated handling in kernel (and this stuff really needs to be realtime). If I were designing a platform to do this
stuff I'd push the actual synchronization out to an fpga.
Quoted 4 lines
Honestly I'm not sure where in the stack it belongs. It would make sense to have it in the FPGA, but then that's a non-standard feature and there is no point in having that supported in libiio, which is meant to be generic.
It might be possible to have optional offload, as long as there is a less performant fallback.
Quoted 6 lines
If the kernel supports "iio_filename=value" metadata for DACs, it would be possible to change a parameter at a very specific moment in time, but I don't know if that's enough to cover all applications. That's a question for @analog.com guys I believe.
Agreed. This stuff is always cool to think about, but I have no idea what the real usecases are driving it.
Quoted 40 lines
Quoted 40 lines
Quoted 29 lines
* Buffer overhaul
~~~~~~~~~~~~~~~Instead of having one
iio_buffer, that is either pushed or
refilled,
the idea
would be to provide an API that allows the client application to
request,
enqueue or dequeue the buffers itself. This would offer much
greater
control
on the buffer management to the application.* Backwards-compatibility
~~~~~~~~~~~~~~~~~~~~~~~Backwards compatibility is not a hard requirement, but it should
still be
possible to be done by implementing the libiio API on top of
libiio2.
Things
like buffer refill/push of libiio can be implemented on top of the
new buffer
queue system. New features, like buffer metadata support wouldn't
be
backported.Quoted 4 lines
This should be considered low-priority - we're not yet at the
point
where
libiio1 is deprecated.Just a side note to this with my IIO maintainer hat on. Compatibility at the kernel interfaces is a must, but we do have a little
bit of flexibility when we have good reason to think no one is using a given interface...
Quoted 8 lines
Quoted 3 lines
Some interesting ideas. Look forward to seeing how it goes!
Jonathan
Great, thank you for the feedback!
-Paul
You are welcome!
I'm particularly looking forward to seeing how some of these high performance devices actually fit into a wider framework. Who doesn't like insanely quick ADCs and DACs? :)
Thanks,
Jonathan