Skip to contentSkip to conversation

EVT_NUM_COMP_PKTS and LE, GATT and SM

DISCUSSION Brian Gix 3 participants 5 messages Dec 7, 2010 Bluetooth drivers

lore
Share & tools
lore

Brian Gix

author
#0 lore raw

Hi All,

I've have encountered a problem while using the gatttool, where my write commands get clobbered by the LE ACL being disconnected prior to the ATT (fixed channel 4) WRITE_CMD being sent over the LE based ACL link.

I believe this is fundamentally due to there being no dependency on the EVT_NUM_COMP_PKTS event when gatttool decides that the WRITE_CMD has been successfully sent. There is multiple parts to this.

1. In User space, the WRITE_CMD pkt is written to the socket. Gatttool erroneously considers a successful socket write as completion, disconnects the socket and exits.

2. In Kernel space, the ACL packet is added to an ACL queue, which is separate from the CMD queue, which can allow either the Disconnect request, or the ACL packet to be sent over the H4 link to the baseband.

3. In the baseband, due to LE clocking (and possible other baseband activity) the ACL packet could be received first, and the Disconnect CMD second, and still result in the connection being detached prior to Tx of the ACL packet containing the ATT WRITE_CMD.

This is not an issue with any of the ATT READ/FIND/MTU or WRITE_REQ transactions, because they require a response from the server. I believe for ATT, this problem is restricted to the WRITE_CMD only, due to it's unacknowledged nature.

However, this will also be an issue with the LE Security Manager, because as stated in the Core Spec v4.0, Vol 3, in the last paragraph of 3.6.1 Key Distribution on page 630 (of 656):

> Key distribution is complete in the device sending the final key
when it receives
> the baseband acknowledgement for that key and is complete in the
receiving
> device when it receives the final key being distributed.

This is intended to prevent exactly the kind of problem I am experiencing with the ATT WRITE_CMD, and the acknowledgement from the baseband can only be the EVT_NUM_COMP_PKTS event.

While talking to my colleagues here, we were thinking that the cleanest method to get this accomplished would be by using the "select" method with the ATT socket, where the socket could be marked as non-writable by the kernel driver until the EVT_NUM_COMP_PKTS is received. The User space code could then wait for the socket to become writeable before issuing the socket disconnect.

If the Security manager is totally within the kernel, it probably does not have to do as much work, however it does still need to wait on the EVT_NUM_COMP_PKTS before disconnecting the remote device.

Has anybody else observed this issue with ATT WRITE_CMD? It could be getting exacerbated by slow (115Kbps) H4 links that I am using, however the hcidump tool confirms that the disconnect happens prior to the EVT_NUM_COMP_PKTS.

Brian Gix
[email protected]
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum

author
#1 lore raw

Additional point--

Quoted 62 lines

-----Original Message-----
From: [email protected] [mailto:linux-bluetooth- [email protected]] On Behalf Of Brian Gix
Sent: 06 December, 2010 4:36 PM
To: [email protected]
Subject: EVT_NUM_COMP_PKTS and LE, GATT and SM

Hi All,

I've have encountered a problem while using the gatttool, where my write
commands get clobbered by the LE ACL being disconnected prior to the ATT
(fixed channel 4) WRITE_CMD being sent over the LE based ACL link.

I believe this is fundamentally due to there being no dependency on the EVT_NUM_COMP_PKTS event when gatttool decides that the WRITE_CMD has been
successfully sent. There is multiple parts to this.

1. In User space, the WRITE_CMD pkt is written to the socket. Gatttool erroneously considers a successful socket write as completion, disconnects
the socket and exits.

2. In Kernel space, the ACL packet is added to an ACL queue, which is separate from the CMD queue, which can allow either the Disconnect request,
or the ACL packet to be sent over the H4 link to the baseband.

3. In the baseband, due to LE clocking (and possible other baseband activity) the ACL packet could be received first, and the Disconnect CMD
second, and still result in the connection being detached prior to Tx of the
ACL packet containing the ATT WRITE_CMD.

This is not an issue with any of the ATT READ/FIND/MTU or WRITE_REQ transactions, because they require a response from the server. I believe
for ATT, this problem is restricted to the WRITE_CMD only, due to it's unacknowledged nature.

However, this will also be an issue with the LE Security Manager, because as
stated in the Core Spec v4.0, Vol 3, in the last paragraph of 3.6.1 Key Distribution on page 630 (of 656):

> Key distribution is complete in the device sending the final
key
when it receives
> the baseband acknowledgement for that key and is complete in
the
receiving
> device when it receives the final key being distributed.

This is intended to prevent exactly the kind of problem I am experiencing
with the ATT WRITE_CMD, and the acknowledgement from the baseband can only
be the EVT_NUM_COMP_PKTS event.

Quoted 6 lines

While talking to my colleagues here, we were thinking that the cleanest method to get this accomplished would be by using the "select" method with the ATT socket, where the socket could be marked as non-writable by the kernel driver until the EVT_NUM_COMP_PKTS is received. The User space code could then wait for the socket to become writeable before issuing the socket disconnect.

This solution could be isolated to LE links (and perhaps even fixed channel 4) because of the strict cmd-resp nature of ATT transactions. This would cause the inability to queue multiple WRITE_CMDS before disconnecting however.

Quoted 26 lines

If the Security manager is totally within the kernel, it probably does not
have to do as much work, however it does still need to wait on the EVT_NUM_COMP_PKTS before disconnecting the remote device.

Has anybody else observed this issue with ATT WRITE_CMD? It could be getting
exacerbated by slow (115Kbps) H4 links that I am using, however the hcidump
tool confirms that the disconnect happens prior to the EVT_NUM_COMP_PKTS.

Brian Gix
[email protected]
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum

--
To unsubscribe from this list: send the line "unsubscribe linux- bluetooth" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
In reply to Brian Gix

Hi Brian,

Quoted 10 lines · ext Brian Gix

Hi All,

I've have encountered a problem while using the gatttool, where my write commands get clobbered by the LE ACL being disconnected prior to the ATT (fixed channel 4) WRITE_CMD being sent over the LE based ACL link.

I believe this is fundamentally due to there being no dependency on the EVT_NUM_COMP_PKTS event when gatttool decides that the WRITE_CMD has been successfully sent. There is multiple parts to this.

Quoted 8 lines

1. In User space, the WRITE_CMD pkt is written to the socket. Gatttool erroneously considers a successful socket write as completion, disconnects the socket and exits.

2. In Kernel space, the ACL packet is added to an ACL queue, which is separate from the CMD queue, which can allow either the Disconnect request, or the ACL packet to be sent over the H4 link to the baseband.

And in usb case CMD and ACL are even using different endpoints.

Quoted 9 lines

3. In the baseband, due to LE clocking (and possible other baseband activity) the ACL packet could be received first, and the Disconnect CMD second, and still result in the connection being detached prior to Tx of the ACL packet containing the ATT WRITE_CMD.

This is not an issue with any of the ATT READ/FIND/MTU or WRITE_REQ transactions, because they require a response from the server. I believe for ATT, this problem is restricted to the WRITE_CMD only, due to it's unacknowledged nature.

True.

Quoted 14 lines

However, this will also be an issue with the LE Security Manager, because as stated in the Core Spec v4.0, Vol 3, in the last paragraph of 3.6.1 Key Distribution on page 630 (of 656):

> Key distribution is complete in the device sending the final key
when it receives
> the baseband acknowledgement for that key and is complete in the
receiving
> device when it receives the final key being distributed.

This is intended to prevent exactly the kind of problem I am experiencing with the ATT WRITE_CMD, and the acknowledgement from the baseband can only be the EVT_NUM_COMP_PKTS event.

Quoted 7 lines

While talking to my colleagues here, we were thinking that the cleanest method to get this accomplished would be by using the "select" method with the ATT socket, where the socket could be marked as non-writable by the kernel driver until the EVT_NUM_COMP_PKTS is received. The User space code could then wait for the socket to become writeable before issuing the socket disconnect.

How about implement waiting on l2cap_sock_shutdown() like for ERTM. User space could then call shutdown() before closing the socket so make sure the data is sent or timeout occurred.

Would this be enough to solve the problem?

Quoted 7 lines

If the Security manager is totally within the kernel, it probably does not have to do as much work, however it does still need to wait on the EVT_NUM_COMP_PKTS before disconnecting the remote device.

Has anybody else observed this issue with ATT WRITE_CMD? It could be getting exacerbated by slow (115Kbps) H4 links that I am using, however the hcidump tool confirms that the disconnect happens prior to the EVT_NUM_COMP_PKTS.

I have seen this problem but didn't dig it deeper.

Signature · 1 line
Ville
author
#3 lore raw

Hi Ville, Vinicius,

Quoted 13 lines · Ville Tervo

Hi Brian,

Quoted 3 lines · ext Brian Gix

Hi All,

I've have encountered a problem while using the gatttool, where my

write

Quoted 1 line

commands get clobbered by the LE ACL being disconnected prior to the

ATT

Quoted 3 lines

(fixed channel 4) WRITE_CMD being sent over the LE based ACL link.

I believe this is fundamentally due to there being no dependency on

the

Quoted 9 lines
Quoted 1 line

EVT_NUM_COMP_PKTS event when gatttool decides that the WRITE_CMD has

been

Quoted 3 lines

successfully sent. There is multiple parts to this.

1. In User space, the WRITE_CMD pkt is written to the socket.

Gatttool

Quoted 1 line

erroneously considers a successful socket write as completion,

disconnects

Quoted 1 line

the socket and exits.

I would like to try Vinicius' patch that (may) address this, but again I do not have access to infradead, although I think gitorious, which you used before, was not a problem.

Quoted 55 lines
Quoted 3 lines

2. In Kernel space, the ACL packet is added to an ACL queue, which is separate from the CMD queue, which can allow either the Disconnect

request,

Quoted 2 lines

or the ACL packet to be sent over the H4 link to the baseband.

And in usb case CMD and ACL are even using different endpoints.

Quoted 2 lines

3. In the baseband, due to LE clocking (and possible other baseband activity) the ACL packet could be received first, and the Disconnect

CMD

Quoted 1 line

second, and still result in the connection being detached prior to Tx

of the

Quoted 4 lines

ACL packet containing the ATT WRITE_CMD.

This is not an issue with any of the ATT READ/FIND/MTU or WRITE_REQ transactions, because they require a response from the server. I

believe

Quoted 1 line

for ATT, this problem is restricted to the WRITE_CMD only, due to

it's

Quoted 1 line

unacknowledged nature.

True.

Quoted 1 line

However, this will also be an issue with the LE Security Manager,

because as

Quoted 1 line

stated in the Core Spec v4.0, Vol 3, in the last paragraph of 3.6.1

Key

Quoted 3 lines

Distribution on page 630 (of 656):

> Key distribution is complete in the device sending the final

key

Quoted 2 lines

when it receives
> the baseband acknowledgement for that key and is complete in

the

Quoted 4 lines

receiving
> device when it receives the final key being distributed.

This is intended to prevent exactly the kind of problem I am

experiencing

Quoted 1 line

with the ATT WRITE_CMD, and the acknowledgement from the baseband can

only

Quoted 3 lines

be the EVT_NUM_COMP_PKTS event.

While talking to my colleagues here, we were thinking that the

cleanest

Quoted 1 line

method to get this accomplished would be by using the "select" method

with

Quoted 1 line

the ATT socket, where the socket could be marked as non-writable by

the

Quoted 1 line

kernel driver until the EVT_NUM_COMP_PKTS is received. The User space

code

Quoted 1 line

could then wait for the socket to become writeable before issuing the

socket

Quoted 8 lines
Quoted 2 lines

disconnect.

How about implement waiting on l2cap_sock_shutdown() like for ERTM. User space
could then call shutdown() before closing the socket so make sure the data is
sent or timeout occurred.

If I understand l2cap_sock_shutdown() correctly, then I think this is the correct solution. The socket mode would be BASIC, and the channel (dcid) would be fixed-4 (or perhaps fixed-5 or fixed-6 as well), but instead of waiting for an ack, it would be waiting on the EVT_NUM_COMP_PKTS, that indicates the number of outstanding ACL packets on the connection is Zero, with the rest of the logic unchanged.

Quoted 19 lines

Would this be enough to solve the problem?

Quoted 1 line

If the Security manager is totally within the kernel, it probably

does not

Quoted 4 lines

have to do as much work, however it does still need to wait on the EVT_NUM_COMP_PKTS before disconnecting the remote device.

Has anybody else observed this issue with ATT WRITE_CMD? It could be

getting

Quoted 1 line

exacerbated by slow (115Kbps) H4 links that I am using, however the

hcidump

Quoted 1 line

tool confirms that the disconnect happens prior to the

EVT_NUM_COMP_PKTS.

I have seen this problem but didn't dig it deeper.

--
Ville

#4 lore raw

Hi Brian,

Quoted 21 lines · Brian Gix

Hi Ville, Vinicius,

Quoted 18 lines · Ville Tervo

Hi Brian,

Quoted 3 lines · ext Brian Gix

Hi All,

I've have encountered a problem while using the gatttool, where my

write

Quoted 1 line

commands get clobbered by the LE ACL being disconnected prior to the

ATT

Quoted 3 lines

(fixed channel 4) WRITE_CMD being sent over the LE based ACL link.

I believe this is fundamentally due to there being no dependency on

the

Quoted 1 line

EVT_NUM_COMP_PKTS event when gatttool decides that the WRITE_CMD has

been

Quoted 3 lines

successfully sent. There is multiple parts to this.

1. In User space, the WRITE_CMD pkt is written to the socket.

Quoted 9 lines
Quoted 4 lines

Gatttool

Quoted 1 line

erroneously considers a successful socket write as completion,

disconnects

Quoted 1 line

the socket and exits.

I would like to try Vinicius' patch that (may) address this, but again I do not have access to infradead, although I think gitorious, which you used before, was not a problem.

Sorry, I forgot about this problem, please take a look here:

http://gitorious.org/~vcgomes/bluez/vcgomes-bluez

Quoted 89 lines
Quoted 55 lines
Quoted 3 lines

2. In Kernel space, the ACL packet is added to an ACL queue, which is separate from the CMD queue, which can allow either the Disconnect

request,

Quoted 2 lines

or the ACL packet to be sent over the H4 link to the baseband.

And in usb case CMD and ACL are even using different endpoints.

Quoted 2 lines

3. In the baseband, due to LE clocking (and possible other baseband activity) the ACL packet could be received first, and the Disconnect

CMD

Quoted 1 line

second, and still result in the connection being detached prior to Tx

of the

Quoted 4 lines

ACL packet containing the ATT WRITE_CMD.

This is not an issue with any of the ATT READ/FIND/MTU or WRITE_REQ transactions, because they require a response from the server. I

believe

Quoted 1 line

for ATT, this problem is restricted to the WRITE_CMD only, due to

it's

Quoted 1 line

unacknowledged nature.

True.

Quoted 1 line

However, this will also be an issue with the LE Security Manager,

because as

Quoted 1 line

stated in the Core Spec v4.0, Vol 3, in the last paragraph of 3.6.1

Key

Quoted 3 lines

Distribution on page 630 (of 656):

> Key distribution is complete in the device sending the final

key

Quoted 2 lines

when it receives
> the baseband acknowledgement for that key and is complete in

the

Quoted 4 lines

receiving
> device when it receives the final key being distributed.

This is intended to prevent exactly the kind of problem I am

experiencing

Quoted 1 line

with the ATT WRITE_CMD, and the acknowledgement from the baseband can

only

Quoted 3 lines

be the EVT_NUM_COMP_PKTS event.

While talking to my colleagues here, we were thinking that the

cleanest

Quoted 1 line

method to get this accomplished would be by using the "select" method

with

Quoted 1 line

the ATT socket, where the socket could be marked as non-writable by

the

Quoted 1 line

kernel driver until the EVT_NUM_COMP_PKTS is received. The User space

code

Quoted 1 line

could then wait for the socket to become writeable before issuing the

socket

Quoted 8 lines
Quoted 2 lines

disconnect.

How about implement waiting on l2cap_sock_shutdown() like for ERTM. User space
could then call shutdown() before closing the socket so make sure the data is
sent or timeout occurred.

If I understand l2cap_sock_shutdown() correctly, then I think this is the correct solution. The socket mode would be BASIC, and the channel (dcid) would be fixed-4 (or perhaps fixed-5 or fixed-6 as well), but instead of waiting for an ack, it would be waiting on the EVT_NUM_COMP_PKTS, that indicates the number of outstanding ACL packets on the connection is Zero, with the rest of the logic unchanged.

Quoted 17 lines

Would this be enough to solve the problem?

Quoted 1 line

If the Security manager is totally within the kernel, it probably

does not

Quoted 4 lines

have to do as much work, however it does still need to wait on the EVT_NUM_COMP_PKTS before disconnecting the remote device.

Has anybody else observed this issue with ATT WRITE_CMD? It could be

getting

Quoted 1 line

exacerbated by slow (115Kbps) H4 links that I am using, however the

hcidump

Quoted 1 line

tool confirms that the disconnect happens prior to the

EVT_NUM_COMP_PKTS.

I have seen this problem but didn't dig it deeper.

Quoted 7 lines
Quoted 2 lines

--
Ville

--
To unsubscribe from this list: send the line "unsubscribe linux-bluetooth" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Cheers,

Signature · 1 line
Vinicius