Skip to contentSkip to conversation

sctp discarding received data chunks

DISCUSSION David Laight 3 participants 9 messages Oct 8, 2020 – Oct 11, 2020 SCTP protocol

lore
Share & tools
lore
#0 lore raw

One of my local tests (that I run quite often) failed in a 'new and exciting way'. I sent 510 messages through an M3UA-MTP2-M3UA link and only got ~350 at the far end. (This is part of my SS7+TCAP regression test - gets run a lot.) Usually I can find 'lost' messages logged as discarded due to my own flow control. In this case there is no sign of any error traces.
I expect to have each message traced 6 times (on each send and receive) but the missing messages are only traced 5 times.

Now /proc/net/sctp/snmp has SctpInDataChunksDiscards set to 163,
this matches the number of messages I'm missing.
Any idea how I can find out why one (or more) the SCTP connections (which are still connected - unless there is a power cut) has discarded a lot of receive packets?

Each M3UA link is actually 4 SCTP connections (TCP style 1-1).
All are loopback connected to 127.0.0.1 or ::1 (the assocs print is 'interesting'). The local port is 'random' the listening port 2905 or 2906.
I'd expect the data to loadshare evenly between them but I've not checked the actual distribution.
The packet data contains a sequence number, I'm missing all the x1, x2, x9, xa and half the x6 and xe packets - so I think at least one of the sctp connections is just discarding the receive chunks.
(I'll sort out which one tomorrow.)

This is a 5.6.0-rc7 kernel.
I've not seen anything like this before - I've run this same test for over 10 years, probably going back to at least 2.6.28.

Data chunks will have gone though all the connections when they were initialised.

Is there anything anywhere that indicates why a data chunk was dropped?

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#1 lore raw

Can you see this issue with the 5.4 kernel too?

I did yesterday some testing by upgrading kernel from 5.4 to 5.7 and I run into all sorts of links going off after a while so I had to revert back. 5.4 is stable for me. 5.7 is not. And I have lots of M2PA and M3UA connections like you

Quoted 40 lines · David Laight

One of my local tests (that I run quite often) failed in a 'new and exciting way'. I sent 510 messages through an M3UA-MTP2-M3UA link and only got ~350 at the far end. (This is part of my SS7+TCAP regression test - gets run a lot.) Usually I can find 'lost' messages logged as discarded due to my own flow control. In this case there is no sign of any error traces.
I expect to have each message traced 6 times (on each send and receive) but the missing messages are only traced 5 times.

Now /proc/net/sctp/snmp has SctpInDataChunksDiscards set to 163,
this matches the number of messages I'm missing.
Any idea how I can find out why one (or more) the SCTP connections (which are still connected - unless there is a power cut) has discarded a lot of receive packets?

Each M3UA link is actually 4 SCTP connections (TCP style 1-1).
All are loopback connected to 127.0.0.1 or ::1 (the assocs print is 'interesting'). The local port is 'random' the listening port 2905 or 2906.
I'd expect the data to loadshare evenly between them but I've not checked the actual distribution.
The packet data contains a sequence number, I'm missing all the x1, x2, x9, xa and half the x6 and xe packets - so I think at least one of the sctp connections is just discarding the receive chunks.
(I'll sort out which one tomorrow.)

This is a 5.6.0-rc7 kernel.
I've not seen anything like this before - I've run this same test for over 10 years, probably going back to at least 2.6.28.

Data chunks will have gone though all the connections when they were initialised.

Is there anything anywhere that indicates why a data chunk was dropped?

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#2 lore raw
In reply to David Laight

From: Andreas Fink

Quoted 3 lines

Sent: 09 October 2020 08:25

Can you see this issue with the 5.4 kernel too?

I've never seen anything like it before.
I've probably run the same tests quite a few times on a post 5.4 kernel. I'll have run them on the same 5.6.0-rc7 (from April 2020) a few times as well.

Quoted 4 lines

I did yesterday some testing by upgrading kernel from 5.4 to 5.7 and I run into all sorts of links going off after a while so I had to revert back. 5.4 is stable for me. 5.7 is not.
And I have lots of M2PA and M3UA connections like you

Glad it's just not me...

I've only had it go horribly wrong once.
I've kept in it the failed state to do further tests. Regressing is hard, my SCCP and MTP3 code is in a big driver and I'd need to find the correct kernel headers to rebuild it.

If it is still broken now I'll push some single packets through while running ftrace.

Actual test is TCAP test app connected to a 'local' dual pair of ss7 protocol stacks (MTP3 has a MTPA-like link to its peer). The traffic from SCCP is directed to M3UA (not MTP3) into a pair of M3UA connections (from each half the dual) to a second MTP3 dual, each of which has an MTP2 signalling link to the other pointcode. The other system is the same.
So I have 4 SS7 protocol stacks on each of 2 pointcodes. Eight M3UA connections - 16 SCTP sockets.
Normally this would require 8 computers in 4 geographically separated locations - but they are all piled up on a single system! Oh - and the tcap app connects by TCP - so that could be on yet another system!

David

Quoted 33 lines
Quoted 33 lines · David Laight

One of my local tests (that I run quite often) failed in a 'new and exciting way'. I sent 510 messages through an M3UA-MTP2-M3UA link and only got ~350 at the far end. (This is part of my SS7+TCAP regression test - gets run a lot.) Usually I can find 'lost' messages logged as discarded due to my own flow control. In this case there is no sign of any error traces.
I expect to have each message traced 6 times (on each send and receive) but the missing messages are only traced 5 times.

Now /proc/net/sctp/snmp has SctpInDataChunksDiscards set to 163,
this matches the number of messages I'm missing.
Any idea how I can find out why one (or more) the SCTP connections (which are still connected - unless there is a power cut) has discarded a lot of receive packets?

Each M3UA link is actually 4 SCTP connections (TCP style 1-1).
All are loopback connected to 127.0.0.1 or ::1 (the assocs print is 'interesting'). The local port is 'random' the listening port 2905 or 2906.
I'd expect the data to loadshare evenly between them but I've not checked the actual distribution.
The packet data contains a sequence number, I'm missing all the x1, x2, x9, xa and half the x6 and xe packets - so I think at least one of the sctp connections is just discarding the receive chunks.
(I'll sort out which one tomorrow.)

This is a 5.6.0-rc7 kernel.
I've not seen anything like this before - I've run this same test for over 10 years, probably going back to at least 2.6.28.

Data chunks will have gone though all the connections when they were initialised.

Quoted 9 lines
Quoted 8 lines

Is there anything anywhere that indicates why a data chunk was dropped?

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#3 lore raw
In reply to David Laight

From: Andreas Fink

Quoted 7 lines

Sent: 09 October 2020 08:25

Can you see this issue with the 5.4 kernel too?

I did yesterday some testing by upgrading kernel from 5.4 to 5.7 and I run into all sorts of links going off after a while so I had to revert back.
5.4 is stable for me. 5.7 is not. And I have lots of M2PA and M3UA connections like you

I've just spent hours digging through my traces.
It is only some messages through the connection that get lost!

Now SCTP_MIN_IN_DATA_CHUNK_DISCARDS is only incremented in two adjacent places in sm_statefuncs.c.

Either for bad TSN (unlikely when everything is using "lo") and bad STREAM.
I suspect it is the 'bad stream' case.
I've not double-checked but I bet the discarded packets all have a large stream number.

So it is likely that the addition of 'sctp streams' broke the negotiation of the maximum stream number, and the reporting of that value back to the application in getsockopt().

I've probably recently changed my test to request 17 streams (not 5). Since the default number of streams is 10 that may be why it worked on this kernel before.

There was a similar bug that got fixed very recently. Ah yes, I wrote this on 14th August:

At some point the negotiation of the number of SCTP streams seems to have got broken.
I've definitely tested it in the past (probably 10 years ago!) but on a 5.8.0 kernel getsockopt(SCTP_INFO) seems to be returning the 'num_ostreams' set by setsockopt(SCTP_INIT) rather than the smaller of that value and that configured at the other end of the connection.

Although I though that stopped packets being set, not received.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#4 lore raw
In reply to David Laight

From: David Laight

Quoted 14 lines

Sent: 09 October 2020 12:14

From: Andreas Fink

Quoted 7 lines

Sent: 09 October 2020 08:25

Can you see this issue with the 5.4 kernel too?

I did yesterday some testing by upgrading kernel from 5.4 to 5.7 and I run into all sorts of links going off after a while so I had to revert back.
5.4 is stable for me. 5.7 is not. And I have lots of M2PA and M3UA connections like you

I've just spent hours digging through my traces.
It is only some messages through the connection that get lost!

Quoted 8 lines

Now SCTP_MIN_IN_DATA_CHUNK_DISCARDS is only incremented in two adjacent places in sm_statefuncs.c.

Either for bad TSN (unlikely when everything is using "lo") and bad STREAM.
I suspect it is the 'bad stream' case.
I've not double-checked but I bet the discarded packets all have a large stream number.

...

If I dump out /proc/net/sctp/assocs and look way over to the right (on the next monitor but 1) there are two columns INS and OUTS. I've just realised that these are the number of streams. Now all my connections are loopback - so I see both sockets for each. So I'd expect the INS to match the OUTS of the peer.
This isn't true.
When the value should be negotiated down the OUTS value is unchanged. So the kernel is sending packets with illegal stream numbers. These are acked and then silently discarded.

I've not check a HEAD kernel.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

maintainer
#5 lore raw
In reply to David Laight
Quoted 26 lines · David Laight

From: David Laight

Quoted 22 lines

Sent: 09 October 2020 12:14

From: Andreas Fink

Quoted 7 lines

Sent: 09 October 2020 08:25

Can you see this issue with the 5.4 kernel too?

I did yesterday some testing by upgrading kernel from 5.4 to 5.7 and I run into all sorts of links going off after a while so I had to revert back.
5.4 is stable for me. 5.7 is not. And I have lots of M2PA and M3UA connections like you

I've just spent hours digging through my traces.
It is only some messages through the connection that get lost!

Now SCTP_MIN_IN_DATA_CHUNK_DISCARDS is only incremented in two adjacent places in sm_statefuncs.c.

Either for bad TSN (unlikely when everything is using "lo") and bad STREAM.
I suspect it is the 'bad stream' case.
I've not double-checked but I bet the discarded packets all have a large stream number.

...

Quoted 9 lines

If I dump out /proc/net/sctp/assocs and look way over to the right (on the next monitor but 1) there are two columns INS and OUTS. I've just realised that these are the number of streams. Now all my connections are loopback - so I see both sockets for each. So I'd expect the INS to match the OUTS of the peer.
This isn't true.
When the value should be negotiated down the OUTS value is unchanged. So the kernel is sending packets with illegal stream numbers. These are acked and then silently discarded.

did it do addstream reconfig or receive any duplicate COOKIE-ECHO in your case?

#6 lore raw
In reply to David Laight

RnJvbTogWGluIExvbmcNCj4gU2VudDogMTAgT2N0b2JlciAyMDIwIDAzOjM1DQo+IE9uIEZyaSwg T2N0IDksIDIwMjAgYXQgOTowMyBQTSBEYXZpZCBMYWlnaHQgPERhdmlkLkxhaWdodEBhY3VsYWIu Y29tPiB3cm90ZToNCj4gPg0KPiA+IEZyb206IERhdmlkIExhaWdodA0KPiA+ID4gU2VudDogMDkg T2N0b2JlciAyMDIwIDEyOjE0DQo+ID4gPg0KPiA+ID4gRnJvbTogQW5kcmVhcyBGaW5rDQo+ID4g PiA+IFNlbnQ6IDA5IE9jdG9iZXIgMjAyMCAwODoyNQ0KPiA+ID4gPg0KPiA+ID4gPiBDYW4geW91 IHNlZSB0aGlzIGlzc3VlIHdpdGggdGhlIDUuNCBrZXJuZWwgdG9vPw0KPiA+ID4gPg0KPiA+ID4g PiBJIGRpZCB5ZXN0ZXJkYXkgc29tZSB0ZXN0aW5nIGJ5IHVwZ3JhZGluZyBrZXJuZWwgZnJvbSA1 LjQgdG8gNS43IGFuZCBJIHJ1biBpbnRvIGFsbCBzb3J0cyBvZg0KPiBsaW5rcw0KPiA+ID4gPiBn b2luZyBvZmYgYWZ0ZXIgYSB3aGlsZSBzbyBJIGhhZCB0byByZXZlcnQgYmFjay4NCj4gPiA+ID4g NS40IGlzIHN0YWJsZSBmb3IgbWUuIDUuNyBpcyBub3QuIEFuZCBJIGhhdmUgbG90cyBvZiBNMlBB IGFuZCBNM1VBIGNvbm5lY3Rpb25zIGxpa2UgeW91DQo+ID4gPg0KPiA+ID4gSSd2ZSBqdXN0IHNw ZW50IGhvdXJzIGRpZ2dpbmcgdGhyb3VnaCBteSB0cmFjZXMuDQo+ID4gPiBJdCBpcyBvbmx5IHNv bWUgbWVzc2FnZXMgdGhyb3VnaCB0aGUgY29ubmVjdGlvbiB0aGF0IGdldCBsb3N0IQ0KPiA+ID4N Cj4gPiA+IE5vdyBTQ1RQX01JTl9JTl9EQVRBX0NIVU5LX0RJU0NBUkRTIGlzIG9ubHkgaW5jcmVt ZW50ZWQgaW4gdHdvDQo+ID4gPiBhZGphY2VudCBwbGFjZXMgaW4gc21fc3RhdGVmdW5jcy5jLg0K PiA+ID4NCj4gPiA+IEVpdGhlciBmb3IgYmFkIFRTTiAodW5saWtlbHkgd2hlbiBldmVyeXRoaW5n IGlzIHVzaW5nICJsbyIpDQo+ID4gPiBhbmQgYmFkIFNUUkVBTS4NCj4gPiA+IEkgc3VzcGVjdCBp dCBpcyB0aGUgJ2JhZCBzdHJlYW0nIGNhc2UuDQo+ID4gPiBJJ3ZlIG5vdCBkb3VibGUtY2hlY2tl ZCBidXQgSSBiZXQgdGhlIGRpc2NhcmRlZCBwYWNrZXRzDQo+ID4gPiBhbGwgaGF2ZSBhIGxhcmdl IHN0cmVhbSBudW1iZXIuDQo+ID4gLi4uDQo+ID4NCj4gPiBJZiBJIGR1bXAgb3V0IC9wcm9jL25l dC9zY3RwL2Fzc29jcyBhbmQgbG9vayB3YXkgb3ZlciB0byB0aGUgcmlnaHQNCj4gPiAob24gdGhl IG5leHQgbW9uaXRvciBidXQgMSkgdGhlcmUgYXJlIHR3byBjb2x1bW5zIElOUyBhbmQgT1VUUy4N Cj4gPiBJJ3ZlIGp1c3QgcmVhbGlzZWQgdGhhdCB0aGVzZSBhcmUgdGhlIG51bWJlciBvZiBzdHJl YW1zLg0KPiA+IE5vdyBhbGwgbXkgY29ubmVjdGlvbnMgYXJlIGxvb3BiYWNrIC0gc28gSSBzZWUg Ym90aCBzb2NrZXRzIGZvciBlYWNoLg0KPiA+IFNvIEknZCBleHBlY3QgdGhlIElOUyB0byBtYXRj aCB0aGUgT1VUUyBvZiB0aGUgcGVlci4NCj4gPiBUaGlzIGlzbid0IHRydWUuDQo+ID4gV2hlbiB0 aGUgdmFsdWUgc2hvdWxkIGJlIG5lZ290aWF0ZWQgZG93biB0aGUgT1VUUyB2YWx1ZSBpcyB1bmNo YW5nZWQuDQo+ID4gU28gdGhlIGtlcm5lbCBpcyBzZW5kaW5nIHBhY2tldHMgd2l0aCBpbGxlZ2Fs IHN0cmVhbSBudW1iZXJzLg0KPiA+IFRoZXNlIGFyZSBhY2tlZCBhbmQgdGhlbiBzaWxlbnRseSBk aXNjYXJkZWQuDQoNCj4gZGlkIGl0IGRvIGFkZHN0cmVhbSByZWNvbmZpZyBvciByZWNlaXZlIGFu eSBkdXBsaWNhdGUgQ09PS0lFLUVDSE8gaW4geW91ciBjYXNlPw0KDQpFeHRyZW1lbHkgdW5saWtl bHkuDQoNCkxvb2tpbmcgYXQgdGhlIGxhdGVzdCB2ZXJzaW9uIG9mIG15IGRyaXZlciBjb2RlDQoo d2hpY2ggSSB3YXNuJ3QgdXNpbmcpIEkgd3JvdGUgdGhlIGZvbGxvd2luZzoNCg0KICogU2luY2Ug dGhlIGNvZGUgdGhhdCBuZWdvdGlhdGVzIHRoZSBudW1iZXIgb2Ygc3RyZWFtcyBnb3QgYnJva2Vu DQogKiBpbiB2ZXJzaW9uIDUuMSB3ZSBuZWVkIHRvIGV4dHJhY3QgdGhlIGNvcnJlY3QgdmFsdWUg ZnJvbSB0aGUNCiAqIGludGVybmFsIHN0cnVjdHVyZXMgdG8gYXZvaWQgU0NUUCBzZW5kaW5nIG1l c3NhZ2VzIHRoZSByZW1vdGUNCiAqIHN5c3RlbSB3aWxsIGRpc2NhcmQuDQoNCiAgICAvKiBzdHJl YW0ub3V0Y250IGlzIHRoZSB2YWx1ZSB3ZSBzaG91bGQgYmUgdXNpbmcuDQogICAgICogQnV0IGtl cm5lbHMgNS4xIHRvIDUuOCBmYWlsIHRvIHJlZHVjZSBpdCBiYXNlZCBvbiB0aGUgbnVtYmVyDQog ICAgICogcmVjZWl2ZWQgZnJvbSB0aGUgcmVtb3RlIHN5c3RlbS4NCiAgICAgKiBTbyBib3VuZCBo ZXJlIHNvIHRoYXQgdHJhbnNtaXR0ZWQgbWVzc2FnZXMgZG9uJ3QgZ2V0IGRpc2NhcmRlZC4gKi8N CiAgICBvdXRjbnQgPSBhc29jLT5zdHJlYW0ub3V0Y250Ow0KICAgIG51bV9vc3RyZWFtcyA9IGFz b2MtPmMuc2luaXRfbnVtX29zdHJlYW1zOw0KDQpJIHRoaW5rIHRoZXJlIHdhcyBhIHBhdGNoIGRv bmUgZm9yIDUuOS4NCkl0IG5lZWRzIGJhY2stcG9ydGluZy4NCg0KQWx0aG91Z2ggQW5kcmVhcyBz YWlkIDUuNCB3b3JrZWQgZm9yIGhpbS4NClNvIG1heWJlIGhlIGhhcyBhIGRpZmZlcmVudCBwcm9i bGVtLg0KDQoJRGF2aWQNCg0KLQ0KUmVnaXN0ZXJlZCBBZGRyZXNzIExha2VzaWRlLCBCcmFtbGV5 IFJvYWQsIE1vdW50IEZhcm0sIE1pbHRvbiBLZXluZXMsIE1LMSAxUFQsIFVLDQpSZWdpc3RyYXRp b24gTm86IDEzOTczODYgKFdhbGVzKQ0K

#7 lore raw
In reply to David Laight
Quoted 16 lines

Extremely unlikely.

Looking at the latest version of my driver code
(which I wasn't using) I wrote the following:

* Since the code that negotiates the number of streams got broken
* in version 5.1 we need to extract the correct value from the
* internal structures to avoid SCTP sending messages the remote
* system will discard.

/* stream.outcnt is the value we should be using.
* But kernels 5.1 to 5.8 fail to reduce it based on the number
* received from the remote system.
* So bound here so that transmitted messages don't get discarded. */

Quoted 9 lines
   outcnt = asoc->stream.outcnt;
   num_ostreams = asoc->c.sinit_num_ostreams;

I think there was a patch done for 5.9.
It needs back-porting.

Although Andreas said 5.4 worked for him.
So maybe he has a different problem.

In my application code, I never use anything else than stream 0 and 1 So I see some other issue in kernel 5.8 which makes it go haywire.

#8 lore raw
In reply to David Laight

Li4uDQo+ID4gPiBJZiBJIGR1bXAgb3V0IC9wcm9jL25ldC9zY3RwL2Fzc29jcyBhbmQgbG9vayB3 YXkgb3ZlciB0byB0aGUgcmlnaHQNCj4gPiA+IChvbiB0aGUgbmV4dCBtb25pdG9yIGJ1dCAxKSB0 aGVyZSBhcmUgdHdvIGNvbHVtbnMgSU5TIGFuZCBPVVRTLg0KPiA+ID4gSSd2ZSBqdXN0IHJlYWxp c2VkIHRoYXQgdGhlc2UgYXJlIHRoZSBudW1iZXIgb2Ygc3RyZWFtcy4NCj4gPiA+IE5vdyBhbGwg bXkgY29ubmVjdGlvbnMgYXJlIGxvb3BiYWNrIC0gc28gSSBzZWUgYm90aCBzb2NrZXRzIGZvciBl YWNoLg0KPiA+ID4gU28gSSdkIGV4cGVjdCB0aGUgSU5TIHRvIG1hdGNoIHRoZSBPVVRTIG9mIHRo ZSBwZWVyLg0KPiA+ID4gVGhpcyBpc24ndCB0cnVlLg0KPiA+ID4gV2hlbiB0aGUgdmFsdWUgc2hv dWxkIGJlIG5lZ290aWF0ZWQgZG93biB0aGUgT1VUUyB2YWx1ZSBpcyB1bmNoYW5nZWQuDQo+ID4g PiBTbyB0aGUga2VybmVsIGlzIHNlbmRpbmcgcGFja2V0cyB3aXRoIGlsbGVnYWwgc3RyZWFtIG51 bWJlcnMuDQo+ID4gPiBUaGVzZSBhcmUgYWNrZWQgYW5kIHRoZW4gc2lsZW50bHkgZGlzY2FyZGVk Lg0KPiANCj4gPiBkaWQgaXQgZG8gYWRkc3RyZWFtIHJlY29uZmlnIG9yIHJlY2VpdmUgYW55IGR1 cGxpY2F0ZSBDT09LSUUtRUNITyBpbiB5b3VyIGNhc2U/DQo+IA0KPiBFeHRyZW1lbHkgdW5saWtl bHkuDQo+IA0KPiBMb29raW5nIGF0IHRoZSBsYXRlc3QgdmVyc2lvbiBvZiBteSBkcml2ZXIgY29k ZQ0KPiAod2hpY2ggSSB3YXNuJ3QgdXNpbmcpIEkgd3JvdGUgdGhlIGZvbGxvd2luZzoNCj4gDQo+ ICAqIFNpbmNlIHRoZSBjb2RlIHRoYXQgbmVnb3RpYXRlcyB0aGUgbnVtYmVyIG9mIHN0cmVhbXMg Z290IGJyb2tlbg0KPiAgKiBpbiB2ZXJzaW9uIDUuMSB3ZSBuZWVkIHRvIGV4dHJhY3QgdGhlIGNv cnJlY3QgdmFsdWUgZnJvbSB0aGUNCj4gICogaW50ZXJuYWwgc3RydWN0dXJlcyB0byBhdm9pZCBT Q1RQIHNlbmRpbmcgbWVzc2FnZXMgdGhlIHJlbW90ZQ0KPiAgKiBzeXN0ZW0gd2lsbCBkaXNjYXJk Lg0KPiANCj4gICAgIC8qIHN0cmVhbS5vdXRjbnQgaXMgdGhlIHZhbHVlIHdlIHNob3VsZCBiZSB1 c2luZy4NCj4gICAgICAqIEJ1dCBrZXJuZWxzIDUuMSB0byA1LjggZmFpbCB0byByZWR1Y2UgaXQg YmFzZWQgb24gdGhlIG51bWJlcg0KPiAgICAgICogcmVjZWl2ZWQgZnJvbSB0aGUgcmVtb3RlIHN5 c3RlbS4NCj4gICAgICAqIFNvIGJvdW5kIGhlcmUgc28gdGhhdCB0cmFuc21pdHRlZCBtZXNzYWdl cyBkb24ndCBnZXQgZGlzY2FyZGVkLiAqLw0KPiAgICAgb3V0Y250ID0gYXNvYy0+c3RyZWFtLm91 dGNudDsNCj4gICAgIG51bV9vc3RyZWFtcyA9IGFzb2MtPmMuc2luaXRfbnVtX29zdHJlYW1zOw0K PiANCj4gSSB0aGluayB0aGVyZSB3YXMgYSBwYXRjaCBkb25lIGZvciA1LjkuDQo+IEl0IG5lZWRz IGJhY2stcG9ydGluZy4NCg0KWWVzLCBJIHdyb3RlIHRoZSBwYXRjaC4NCkFwcGxpZWQgdG8gbmV0 LW5leHQuDQoNCmh0dHBzOi8vZ2l0Lmtlcm5lbC5vcmcvcHViL3NjbS9saW51eC9rZXJuZWwvZ2l0 L25leHQvbGludXgtbmV4dC5naXQvY29tbWl0L25ldC9zY3RwL3N0cmVhbS5jP2lkPWFiOTIxZjNj ZGJlYzAxYzY4NzA1YTdhZGU4YmVjNjI4ZDU0MWZjMmINCg0KCURhdmlkDQoNCi0NClJlZ2lzdGVy ZWQgQWRkcmVzcyBMYWtlc2lkZSwgQnJhbWxleSBSb2FkLCBNb3VudCBGYXJtLCBNaWx0b24gS2V5 bmVzLCBNSzEgMVBULCBVSw0KUmVnaXN0cmF0aW9uIE5vOiAxMzk3Mzg2IChXYWxlcykNCg=