wifi: rtw88: preparations for RTL8723B/RTL8723BS

From: Luka Gejak <[email protected]>

This is the first of two series adding support for the Realtek RTL8723B 802.11n chipset and its RTL8723BS SDIO variant to rtw88. It contains only the changes to the shared rtw88 core that the chip driver depends on. The chip itself, the build glue and the MAINTAINERS entry are a second series.

v1 had 19 patches, v2 had 11, v3 had 5, v4 had 7, v5 had 6, v6 had 6, v7 had 6, v8 had 6, v9 had 6, v10 had 6, and this has 7.

There is no rtw88-style firmware for this chip and no documentation for the vendor blob it does use, so while bringing it up I deliberately kept the driver as close to the vendor driver's behaviour as I could. With that many unknowns at once, matching the vendor exactly was the only way to tell which difference actually mattered when something did not work, rather than guessing. A number of the v1 patches came from that: they reproduced vendor behaviour that was useful to hold fixed during bring up, not behaviour the chip turns out to require.

Bitterblue Smith's review of v1 prompted me to go back and remove each of those on hardware to find which were actually load bearing; eight were not. Ping-Ke Shih's review of v2 did the same for a further six, and the changelog below gives the measurement for each. In four of those cases I had told the list, or claimed in a commit message, that the chip needed something it does not, on the strength of bring-up notes I had not re-measured. All four corrections are in the changelog.

What is left is there because removing it made something measurably worse, not because the vendor driver does it.

What is left falls into these groups:

- patches 1 and 2: the chip type and helper plus a one-line receive

    fix;
  - patch 3: a longer TX-report purge timeout, needed on slower SDIO
    hosts (see below);

- patch 4: a fix to the shared SDIO transmit path, see below;
- SDIO (patches 5, 6, 7): software free-page and output-queue accounting, RX aggregation and interrupt setup, and TX back-pressure with a retry on page starvation.

This series no longer touches coex.c, mac80211.c or fw.c at all.

Scope
=====

Patch 4 is a deliberate exception to what follows, and it is new in this revision. rtw_sdio_write_port() rounds the transfer up with sdio_align_size() and hands that length to sdio_memcpy_toio() while the skb still holds only skb->len bytes, so between one and 511 bytes from past the end of the frame are transmitted. Whether that stays inside the skb's allocation depends on the tailroom it happens to have. That is a pre-existing bug and it affects every SDIO part, so the fix is not gated on anything and it does change behaviour for the other five: the padding they send is now zeroed, and a transmit can now fail with -ENOMEM if the skb has to be reallocated. Ping-Ke asked for it to come before the RTL8723BS work rather than after the series, so that it backports on its own and so that it is clear it is an existing problem.

Everything else is gated on rtw_is_8723bs(), which is false for every chip currently supported, so behaviour for existing devices is otherwise unchanged. Patch 2 extends an existing RTL8703B zero-length-packet check to this chip. Earlier revisions of this cover letter said that check was gated on the SDIO interface rather than the chip id. That was true of an old version and I did not update the text; the code you have adds rtw_is_8723bs() alongside the existing RTL8703B test, so it is gated on the chip id as well and the RTL8703B behaviour is untouched. Ping-Ke has suggested the check may not need a chip test at all, but I would rather measure that on RTL8703B hardware than assume it, and I do not have that part, so it is unchanged here.

Patch 6 changes how the SDIO interrupt status is acknowledged. In v1 that applied to all SDIO parts; it is scoped to this chip only, and the others keep writing the status word back unchanged.

Since the point of this section is to make that checkable rather than asserted, here is the whole of what is not gated, and what the other parts run as a result.

New code that is not chip gated. Patch 4 above is the one behavioural item; the rest is structural:

sdio.h struct delayed_work work
rtw_sdio_init_tx() INIT_DELAYED_WORK(), mutex_init(), and
mutex_destroy() on its error path
rtw_sdio_tx_kick_off() mod_delayed_work(..., 0) in place of
queue_work()
rtw_sdio_deinit_tx() cancel_delayed_work_sync(), mutex_destroy()

Only this chip ever arms a delay, so on the other parts the work is still queued immediately, and only this chip ever takes the mutex, so there it is one init and one destroy and nothing else.

Two existing functions were also restructured, and every chip runs through them. rtw_sdio_process_tx_queue() returns a value instead of void, and rtw_sdio_tx_handler() uses it. The behaviour is meant to be unchanged for the other parts: on an empty queue the new "if (ret > 0) break" takes the place of the old skb_queue_empty() break, and on a failed write the frame is still requeued and the loop still retries, because rtw_sdio_8723bs_reschedule_tx() returns false for anything that is not this chip.

Two things worth stating because they can be checked against the base commit rather than taken on trust. rtw_sdio_check_free_txpg() differs by exactly one inserted branch ahead of the existing 8051 test, which becomes an else if, with everything else unchanged. And the generic transmit path is the same sequence it was, once patch 4's padding is set
aside: the same address from skb->len, the same sdio_align_size(), the same free page check, the same transfer and the same failure message. What patch 5 does to it is split it up, not change it. rtw_sdio_write_port() works out the address and the aligned size, rtw_sdio_write_port_generic() does the free page check, and rtw_sdio_write_to_port() does the transfer.

So there are exactly two differences another part can observe, and both are named above: the padding it sends is now zeroed and a transmit can fail with -ENOMEM, from patch 4; and the unaligned SKB warning prints rtw_sdio_write_to_port as its function name, since that is where the check now lives.

rtw88 supports five other SDIO parts that share sdio.c: the RTL8723CS, RTL8723DS, RTL8821CS, RTL8822BS and RTL8822CS. I have none of them, so nothing here has been run on one. If anyone on the list does, a boot and some traffic with this series applied would be worth more than the argument above.

Chip specific code in the common path
=====================================

Ping-Ke asked whether this needs a set of ops rather than a chip test sprinkled through the core. Working through that question is what removed most of this series: once the association and coexistence patches turned out not to be needed, the chip tests they carried went with them.

What remains is one test in rx.c, one in tx.c and eleven in sdio.c. There are none in fw.c, mac80211.c or coex.c. The SDIO ones are in the bus driver rather than the chip driver, and rtw_sdio has no per-chip ops table today; Ping-Ke confirmed an inline chip id check there is fine.

Fixed in the second series
==========================

Two of Bitterblue's review points are addressed in the chip driver rather than here, because that is the right place for both:

- the hardware capability is filled in from a chip-specific read_efuse, instead of special casing rtw_dump_hw_feature() in the core;
- the receive FCS handling is a chip configuration fix. v1 stopped the core advertising RX_INCLUDES_FCS for this chip. The real cause was that the chip cleared BIT_APP_FCS by assigning WLAN_RCR_CFG over hal.rcr wholesale, where the 8723x siblings only write the register and leave hal.rcr alone. Setting that bit in the chip's WLAN_RCR_CFG makes the descriptor carry the FCS like every other rtw88 chip, so the core needs no special case and both hunks are gone from this series.

Neither change is visible here. I planned to send that series once this one is applied to rtw-next, but I can send it now alongside if you would rather review them together, or fold the two into one series if that is easier.

Testing
=======

Tested on RTL8723BS hardware together with the second series: repeated scan, authentication, association, WPA2-PSK/CCMP handshake, DHCP, bidirectional traffic, reconnects and module reloads. This revision was run on the tree posted here on an ordinary kernel. The lockdep results below were taken on v10, and they carry over because the locking is untouched here: the same single guard(mutex), the same two guard(spinlock_irqsave) sites and the same two lockdep_assert_held() calls, and the back-pressure patch is code identical to v9.

The full driver, this preparation series plus the chip support that follows in a second series, is available for testing at:

https://github.com/MocLG/rtw/tree/8723bs-v11

Throughput is measured against an iperf3 server one hop behind the AP, with the wlan0 byte counters as ground truth because the test machine's wired interface shares the same subnet. On one AP at strong signal, 2.4 GHz HT40, TCP runs at roughly 35 Mbit/s down and 20 Mbit/s up, and UDP reaches 40 Mbit/s down with no loss. The exact figures move by several Mbit/s from one day to the next with the radio environment, so they are a range rather than a benchmark.

Uplink on this band drifts by several Mbit/s from day to day, enough to swamp the differences I was trying to measure, so where the changelog below compares two builds the comparison is a paired test: the two builds alternate within one session, rebuilt and reinstalled each time, rather than being measured on different days.

The locking change in patch 7 was retested that way for v8, against a build carrying the v7 barrier version of the same patch, alternating between the two within one session and reassociating each time. The two came out the same within the usual run to run spread, uplink and downlink alike, and the log was empty after every run.

Suspend and resume are not covered: this machine only offers s2idle and does not reliably come back from it.

On this revision: 5/5 scans while connected with the target AP seen every time and the link still up afterwards, 5/5 cold module reloads reassociated, 3/3 link up/down cycles, 5/5 reconnects, 200/200 pings at 0% loss, and an idle period in power save that woke with no loss. No tx-report timeouts, no leave-LPS failures, no failed H2C commands, no SDIO errors, and nothing logged by the driver at all.

A longer soak on v10, whose transmit path differs from this one only by patch 4's padding and the split in patch 5, gave 20/20 cold reloads, 10/10 scans and 2000/2000 pings at 0% loss.

That lockdep kernel matters for patch 5 and patch 7 in particular, since the lockdep_assert_held() calls in patch 5 compile to nothing without CONFIG_PROVE_LOCKING, and patch 7 calls ieee80211_stop_queue() and ieee80211_wake_queue() with interrupts disabled under the TX queue lock. Over the same tests there was no lockdep report of any kind, no "sleeping function called from invalid context", and debug_locks was still 1 at the end, so lockdep had not switched itself off part way through and every check was live for the whole run. Throughput on that kernel is roughly half, which is the instrumentation, so the figures above are from the ordinary one.

Every patch builds with W=1 with no warnings on its own, sparse is clean at the tip, and smatch reports nothing in any file this series touches. checkpatch --strict is clean across this series. The other rtw88 bus and chip modules continue to build.

Known limitation
================

After a long idle period the firmware intermittently fails to leave LPS, which rtw88 reports as "firmware failed to leave lps state". That check is the REG_TCR poll in __rtw_fw_leave_lps_check_reg(). On one occurrence here the warning was followed fourteen seconds later by a full reauthentication, so it is not only log noise.

Power save is enabled on my test machine and it reproduces there, roughly once per day of mostly idle uptime, and the occurrences cluster. It was first reported on ARM SDIO boards, but it is not specific to slow hosts.

Instrumenting the poll settles what a failure is, and it is not a budget that is slightly too small. A local debug patch logged REG_TCR on every poll of __rtw_fw_leave_lps_check_reg() and counted how many polls each wake needed. Over 110755 wakes:

poll0=158 poll1=110557 poll2=35 poll3=1 poll4=0 fails=4

pollN is the number of wakes that finished on the Nth read, so poll1 needed one 20 ms sleep and poll0 was already clear on entry; fails is the number still set after all five. Nothing ever finished on the last poll, yet four gave up completely, and in all four the patch kept reading for a further 500 ms with the bit still set. The distribution does not taper into the failures, it is bimodal: the wake is either quick or the firmware never completes it, about once in every 28000 wakes. Raising the poll count would not have caught any of the four.

In v7 I described a failure as the far tail of the ordinary distribution. That was wrong, and the measurement above is what corrects it.

Ping-Ke suggests the firmware may be unable to leave LPS while packets are still sitting in a hardware queue, which fits that shape better than anything I had. I am extending the instrumentation to dump the per-queue lengths and the free page and output queue counters at the failure.

It is not caused by anything in this series, and it needs the chip driver from the second series to be reachable at all, but it is open and I am chasing it.

Changes in v11:
- a new patch 4 fixes the shared transmit path: rtw_sdio_write_port() hands sdio_memcpy_toio() an aligned length while the skb still holds only skb->len bytes, so bytes from past the end of the frame go out. Ping-Ke asked for it ahead of the RTL8723BS work rather than after the series, so it backports on its own and so it is clear it is an existing problem. It is not gated on anything and it does change behaviour for the other five SDIO parts, which is called out in the Scope section above.
- sdio: rtw_sdio_write_port() now works out the transfer address and the aligned transfer size and passes both down, since the two paths began the same way. The padding from patch 4 stays there, so it still runs once for both paths and outside the credit mutex, and the transfer moves into rtw_sdio_write_to_port(), which both call.
- sdio: the RTL8723BS path keeps its own length for the page accounting. Ping-Ke suggested it was unnecessary, but the chip charges pages by frame length rather than by the padded transfer, matching the vendor driver, and the two differ just above a block
boundary: a 1025 byte frame is nine pages by length and twelve by the padded size.
- sdio: -ENOMEM is retried again in the back-pressure patch. It became reachable once more when the padding moved into the series, and without the retry a stopped access category has nothing left to kick it. That patch is now code identical to v9; only its comments differ.
- patches 1, 2, 3 and the RX aggregation patch are unchanged.

Changes in v10:
- sdio: patch 4 no longer pads the transfer up to the SDIO block size. Ping-Ke asked why the generic path does not need the same thing, and the answer is that it does: rtw_sdio_write_port() rounds the transfer up and hands that length to sdio_memcpy_toio() while the skb still holds only skb->len bytes, so bytes from past the end of the frame are transmitted. That is pre-existing and applies to every SDIO part, so it is a separate patch and not something this series carries for one chip. It will follow once this series is applied, against rtw_sdio_write_to_port(), which both paths call and which only exists after this series, so one patch covers every part including this one. Measured on hardware, this chip does not need the padding for its own
sake: transferring unpadded, exactly as the generic path does today, measured no worse than padded in an interleaved comparison.
- sdio: patch 4 takes the transfer size from sdio_align_size(), as Ping-Ke suggested. On this card, which reports multi block support and a 512 byte block size, it is identical to the rounding it replaces, checked at the block boundaries and then against every frame of a session. The equivalence rests on those two card properties rather than being unconditional.
- sdio: patch 6 no longer retries -ENOMEM. With the padding gone from patch 4 nothing on that path can return it, so the case was dead. If the separate padding fix is applied it becomes reachable again, for every chip, and this has to come back with it, otherwise a stopped access category can be left with nothing to kick it.
- the comments are cut back across the whole series, as Ping-Ke asked. This revision adds 62 comment lines where v9 added 103. What is left records things the code cannot show: which page pool a queue draws from and how that was measured, why the counters are atomic_t, why only the defined interrupt status bits are acknowledged, and the two places where an skb has already been handed on and must not be touched.
- patches 1, 2, 3 and 5 are unchanged.

Changes in v9:
- sdio: rtw_sdio_write_port() is a dispatcher now, with the two paths in rtw_sdio_write_port_8723bs() and rtw_sdio_write_port_generic(), as Ping-Ke asked. The chip specific one takes the mutex with guard() directly, so the extra rtw_sdio_8723bs_write_port() layer, which only existed to give the guard a scope while the generic path shared the outer function, is gone.
- sdio: the unaligned SKB warning sits in rtw_sdio_write_to_port() now, once, next to the transfer whose pointer it checks and after any padding. It was duplicated in both callers in v8 to keep __func__ reading rtw_sdio_write_port on the other parts; Ping-Ke had no preference there, and one copy in the right place seemed better than two in the wrong one.
- the Scope section above is corrected and extended. It described patch 2 as gated on the SDIO interface rather than the chip id, which stopped being true several revisions ago, and it did not mention that rtw_sdio_process_tx_queue() and rtw_sdio_tx_handler() are restructured for every chip.
- patches 1, 2, 3 and 5 are unchanged, and patch 6 is unchanged apart from Ping-Ke's Acked-by.

Changes in v8:
- sdio: patch 4 accounted RTW_TX_QUEUE_VO against the high page pool, but rtw_sdio_get_tx_addr() writes that queue to the normal transmit FIFO, and the FIFO written to is the pool the chip charges. Polling REG_SDIO_FREE_TXPG through a saturating VO flood, the high pool never lost a page while the public pool drained, so the check was reading a pool the traffic does not use and overstating the free space by the whole dedicated high allocation. It is accounted against the normal pool now, which is also what the other two rtw88 SDIO parts do. RTW_TX_QUEUE_MGMT stays on the high pool: it goes to the extra FIFO, which the register has no counter for and this chip gives no pages to, and there is nowhere better to put it.
- sdio: patch 4 claimed an output queue entry before the transfer and did not hand it back when the transfer failed, leaving the cached count low until the next resync from the chip.
- sdio: patch 4 called mutex_init() without a matching mutex_destroy() on teardown or on the init error path, unlike rtw_core_deinit().
- sdio: patch 6 no longer stops a queue and then undoes it. Ping-Ke pointed out that pci.c has no such back and forth, and he is right:
it holds irq_lock across the descriptor check and the stop in rtw_pci_tx_write(), and the wake in rtw_pci_tx_isr() runs with that same lock already held by its caller, which is why there is not one barrier in the whole file. Both SDIO sides now take the TX queue lock across the length check and the flag update, so a queue can only be stopped while it really is above the watermark and there is nothing left to undo. The smp_mb() pair, the READ_ONCE and WRITE_ONCE accessors and the re-check all go with it. It matches pci.c in shape too: the enqueue is its own critical section and the check and the stop are a second one.
- sdio: patch 4 moves the locked region into rtw_sdio_8723bs_write_port() so the mutex can be taken with guard(mutex). guard() expands to a declaration and so cannot be made conditional, and the lock is only wanted for this chip, so the generic path is a short branch that never reaches it. The transfer itself is shared as rtw_sdio_write_to_port(); the unaligned SKB warning stays in both callers so that the message other SDIO chips log is unchanged.
- sdio: lockdep_assert_held() added to the three helpers that must run under tx_credit_lock, and checked on a kernel built with CONFIG_PROVE_LOCKING over bidirectional traffic and three module
reloads: nothing fired, and lockdep reported nothing against the TX queue lock the back-pressure path now takes either.
- sdio: patch 4's padding uses a pad_size local, and its comment is cut to the one point that matters, that __skb_pad() must not free the skb. The comment above the lock is gone, since the one at the declaration of tx_credit_lock already covers it.
- the LPS note under Known limitation is corrected against a much longer instrumentation run; see there.
- patches 1, 2, 3 and 5 are unchanged.

Changes in v7:
- sdio: patch 6 read the mac80211 queue index from the skb after skb_queue_tail() had published it to the TX worker, which can dequeue, transmit and free it first. That is a use after free; the index is now read into a local before the enqueue.
- sdio: patch 4 passed the padded size to rtw_sdio_get_tx_addr(), where upstream passes skb->len. That value is encoded into the CMD53 address as the transfer length, so the encoded length changed for every other SDIO chip whenever sdio_align_size() padded. For the RTL8723BS both expressions are the same value, so the change bought this chip nothing. The padding itself also ran on the generic path, giving the other parts an allocation and an -ENOMEM path they did not have. skb->len is restored and the padding is gated on the chip.
- sdio: the free page check, the output queue wait and the accounting after the transfer are serialised. The TX worker and the H2C path both reach rtw_sdio_write_port(), and two writers could each pass the checks and claim the same pages and output queue entry, after which the chip discards one of the transfers silently. The vendor driver funnels all transmission through one thread and so never had to handle this.
- sdio: the padding called skb_put_zero() after checking only the tailroom, so a cloned skb with room to spare had its shared buffer written. It uses __skb_pad() now, which reallocates in that case and does not move skb->len, so the trim on the way out is gone too.
- sdio: the back-pressure stop could race the drain. The worker could empty the queue after the length check but before the stopped flag became visible, see the flag still clear in its wake check, and never wake the queue, leaving an access category stopped with nothing left to wake it. The stop path now re-checks after setting the flag and undoes the stop if the queue drained.
- patches 1, 2, 3 and 5 are unchanged.

Most of these were found by Sashiko's automated review of v6. Each was verified against the code before acting on it.

Changes in v6:
- Each patch now carries its own changelog after the --- delimiter, as you asked, so the notes sit next to the code they describe.
- sdio: dropped rtw_sdio_reschedule_tx_work(). You did not ask for it in v4 and it was a thin wrapper that hid queue_delayed_work() for no gain. I had read your comment as asking for one, which was my mistake.
- sdio: the reschedule conditions moved into rtw_sdio_8723bs_reschedule_tx(). rtw_sdio_tx_handler() no longer explains any chip specific condition in the common flow, which is what you were actually asking for. No functional change.
- sdio: dropped an unconditional break on a failed transfer that v4 and v5 carried. It changed the other SDIO parts without meaning to:
upstream requeues the frame and the loop retries, and breaking gave up after the first failure. The two errors this chip needs to retry are handled in the helper above, so the rest keep the existing behaviour and the other parts really are untouched again, as the Scope section claims.

Changes in v5:
- Dropped "fw: handle the RTL8723BS management TX reports", which v4 had restored. It cannot do what its commit message claimed. This firmware does not advertise FW_FEATURE_LPS_C2H, so rtw_fw_leave_lps_check() takes the REG_TCR polling path and nothing ever waits on the C2H completion the patch rerouted; and with the driver instrumented, 93 TX reports over three scans all arrive as C2H id 0x03, with no event ever arriving as a top level 0x12 or 0x32. Bitterblue Smith made exactly this point on v1 and was right. A tester on a Rockchip RK3288 board confirmed the timeout patch in this series is enough on its own. This is the fourth correction of something I had claimed on the list; restoring it in v4 was based on a tester reporting that a branch containing it cleared the warnings, which showed the branch helped, not that this patch in it did. With it gone the series no longer touches fw.c.
- Power save is now enabled on the test machine, so LPS is exercised there. The v2 entry below dropped a patch that gated LPS entry on smoothed throughput, partly on the grounds that LPS never engaged on this setup at all, which was never a sound reason to drop it. It is measured now, paired against this series in one session with power save on: 60 idle pings average 5.1 ms without the gating and 6.0 ms with it, and TCP is 30.0 down and 19.1 up against 32.3 and 18.9. It makes no difference, because the gate tests the smoothed throughput and interactive traffic rounds to zero there, so it never fires in the case it was meant to help. It stays dropped, now on a measurement rather than for want of one.
- sdio: fixed a transmit stall in the back-pressure patch, found while auditing a tester report of transmit stopping under bidirectional load with receive unaffected and nothing in the log. The TX work item was only re-armed when a transfer failed with a page shortage. Once the mac80211 queue can be stopped, that is not enough: a stopped queue is handed no further frames, so on any other failure nothing would kick the worker again and the affected access category would stay stopped for good, with the link still up. It now also re-arms on a failed skb expansion, which together with the page shortage covers both of the failures that produce no log message. The remaining errors are each logged where they happen, so unlike those two they are visible rather than an unexplained hang, and they keep the existing behaviour rather than being retried forever.
- rx: the zero length test now evaluates pkt_stat->pkt_len first, so the unlikely case short circuits before the chip test.
- sdio: the commit message now says what OQT is, as far as the vendor driver reveals it: the vendor calls the register the OQT free space and never expands the acronym, and it holds the number of further transfers the SDIO output queue will accept.
- sdio: rtw_sdio_8723bs_sync_free_txpg() now returns whether the chip reported anything and is the only caller of rtw_sdio_8723bs_store_free_txpg(), and rtw_sdio_8723bs_init_free_txpg() returns an error rather than nothing. Working through that suggestion found a real bug: the public pool size was computed as acq_pg_num minus the reserved queues with no check, so a chip that came up with no transmit page allocation at all would underflow a u16 and leave the driver believing it had about 65000 free pages. That calculation is now rtw_sdio_8723bs_pubq_num(), shared with the queue page allocation repair path, and it fails cleanly instead.
- sdio: the output queue wait is now bounded by a jiffies deadline rather than a loop count, so RTW_SDIO_OQT_TIMEOUT_MS really is a timeout in milliseconds. It was 1000 iterations of a 1 to 2 ms sleep before, so the name was only approximately true.
- sdio: rtw_sdio_8723bs_check_rqpn() returns an error instead of silently doing nothing when the transmit page pool cannot cover the reserved queues, and rtw_sdio_start() propagates it. Both early returns are now explained: a non-zero free page count means the allocation latched during power on and must be left alone.
- sdio: fixed the interrupt acknowledgment. It masked the status word with both irq_mask and RTW_SDIO_HISR_CLEAR_MASK, but for this chip irq_mask is REG_SDIO_HIMR_RX_REQUEST alone and that bit is not in the clear mask, so the two had no bits in common and the driver acknowledged nothing at all. It now masks with the defined bits only, which is what the comment described. Thanks to Ping-Ke for catching this. Retested on hardware.
- sdio: the back-pressure and wake conditions moved into rtw_sdio_8723bs_stop_tx_queue() and _wake_tx_queue(), and the two requeue sites into rtw_sdio_reschedule_tx_work(), instead of long conditions inline.
- sdio: rtw_sdio_process_tx_queue() now returns 0 on success and 1 for the empty queue case, rather than the other way round.
- sdio: queue_stopped[] renamed to tx_queue_stopped[].
- RTW_SDIO_TX_RETRY_DELAY is left as msecs_to_jiffies(1). Ping-Ke asked whether it becomes 0 when HZ is below 1000; it does not, since msecs_to_jiffies() rounds up for HZ < 1000 and returns one jiffy.

Changes in v4:
- Restored "tx: extend the TX report purge timeout to RTL8723BS", dropped in v2 because I could not reproduce the failure on my test machine, a fast x86 host, so the path never ran for me. Testers on slower ARM SDIO hosts, Peter Robinson on the RFC and Zhang Ning on a Rockchip RK3288 board, both hit "failed to get tx report from firmware". It extends the exact fix from commit c80788f7c5ae ("wifi:
rtw88: increase TX report timeout to fix race condition"), already applied to the RTL8723DU, to the RTL8723BS, since they run the same firmware and hit the same off-channel-scan race.

Changes in v3:
- Dropped "fw: fix the reserved page upload on RTL8723BS". Its commit message said the BIT_BCN_VALID handshake fails on 8723BS SDIO without it. That does not reproduce. The failure path is an explicit rtw_err("error beacon valid") returning -EBUSY, reached from BSS_CHANGED_ASSOC on every association; over three associations on each of five builds with the patch reverted it never fired, associations were 3/3, and there was no packet loss after 30 s idle in power save, which is the path the reserved page's null data and PS-poll frames serve. As a control, rtw_info messages appear in the same captures and rtw_err outranks them, so a failure would have been logged. AP mode and WoWLAN also download the reserved page and are not tested here, so if it turns out to matter there it can come back with evidence behind it.
- Kept "sdio: add TX back-pressure and retry on page starvation", now with a measurement rather than an assertion, since you asked whether the remaining chip tests are necessary. Counting rtw_sdio_tx_handler invocations under an identical 60 Mbit/s UDP flood, because the retry re-arms that same work item: 1798 and 3095 on two baseline runs against 69 with the patch reverted. Achievable transmit rate falls by about a third without it, 25.5 Mbit/s against 38.3 and 40.2. tx_dropped and tx_errors stay at zero throughout, so the failure is not dropping frames, it is failing to push them.
- Kept "sdio: set up RX aggregation and interrupts". The aggregation half makes no measurable difference to throughput here, and on that evidence alone I would have dropped it. The rest of the patch does not show up in a throughput test at all: the chip keeps raising the interrupt after resume if undefined status bits are written back when acknowledging.

I have not been able to validate the suspend and resume path: the test machine only offers s2idle and does not reliably resume from it, so I cannot exercise the interrupt acknowledgment change. It is kept because the failure it addresses was observed during bring up, not because it has been re-measured. If someone with hardware that suspends cleanly finds it unnecessary, it should go.
- Dropped "run the RTL8723BS association register sequence", 440 lines, which was the vendor join sequence, six sites in rtw_ops_bss_info_changed() and both sites in rtw_ops_set_key().

Ping-Ke asked for this to run from rtw_chip_prepare_tx() rather than a chip test in mgd_prepare_tx(), and I had done that with a new chip_ops::prepare_tx. Since the whole patch is gone, so is the callback; the core is left as it was rather than gaining an op with no user.

I had told the list that association fails without these registers. That was wrong, and it was bring-up-era data I repeated without re-measuring. With the whole patch reverted: 18/18 scans saw the AP, 6/6 associations, and the five minute soak above, whose 15 reconnects and 24 rekeys exercise the exact path the sequence ran on, since it ran from mgd_prepare_tx() on every authentication. Uplink measured paired against the full series, 19.0 against 19.2 Mbit/s over ten runs each.

The default key search handling in that patch was also simply wrong:
rtw_sec_enable_sec_engine() sets sec->default_key_search itself and programs all four USE_DK bits on that basis, so the enable path was a no-op and the disable path was clearing bits the core had deliberately set.
- Dropped "coex: add the RTL8723BS scan antenna workaround" and "coex:
reassert the antenna path when associating", and with them "fw: add the GNT_BT firmware command", whose only caller they were.

The commit message on the first claimed the site survey does not hear the AP reliably without it. That is also wrong: 18/18 scans without it. Both were gated on the chip and on coex bt_disabled, so that gate is their entire scope rather than a subset of it, and the measurements above cover it.
- Dropped "record beacons from the target BSSID before authenticating". Ping-Ke doubted the beacon wait affected the connection and he was right: with the wait removed, association succeeded 12/12 from a cold module reload and 12/12 from a warm reconnect, against 10/10 and 10/10 with it, at the same latency. The pre-auth deauth already sleeps 100 ms with the recording window open, so on a 100 ms beacon interval the wait was almost always already satisfied when it was reached. struct rtw_auth_sync and the RX-path hook go with it.
- Dropped the BT_MP version queries and the BT_INFO query at scan start, and the BT_INFO query plus two of three repeated PS-TDMA H2Cs in the pre-auth replay. Measured over cold module reloads with three scans each, 8 reloads with the queries and 10 without: 29/30 scans saw the target AP and 10/10 associations succeeded without them, against 24/24 and 8/8 with them.
- The cached SDIO page counters are clamped at zero. A lost update between the free page check and the accounting could previously drive the public counter negative, and the sum was assigned to an unsigned, which made the check pass unconditionally from then on and suppressed the resync that was supposed to recover it.
- REG_SDIO_FREE_TXPG now has field masks and is decoded with u32_get_bits(); the RX DMA burst count uses BIT_DMA_BURST_CNT; open-coded BIT(0)|BIT(1) on REG_SYS_FUNC_EN uses the existing names.
- The RTL8723BS-specific SDIO helpers are named _8723bs_ and the chip test moved to their callers.
- rtw_sdio_process_tx_queue() returns a value instead of taking a "processed" out-parameter, which makes the retry and back-pressure logic in the TX handler readable.
- mod_delayed_work() in rtw_sdio_tx_kick_off() carries a comment saying why it is not queue_delayed_work(): a page-starvation retry may already be armed with a delay, and the newly queued frame should not wait for it.
- An argument that had a single value at every call site is gone, along with its dead branch.
- Block comments use the general kernel style, per commit 82b8000c28b5 ("net: drop special comment style").

Changes in v2:
- Dropped "tx: extend the TX report purge timeout". Instrumenting the report path shows payload[6] & 0xfc equals the enqueued sequence number, 32 out of 32 times, with rtw88's existing decode, and no report is missed. The longer timeout was covering for the next patch, not for the hardware.
- Dropped "fw: handle the RTL8723BS management TX reports". 0x12 and 0x32 are the first payload byte of C2H_CCX_TX_RPT, not C2H IDs; every C2H event this chip sends carries a known id, so the existing handler already covers them.
- Dropped "fw: send rate adaptation and RSSI info in the vendor layout". The vendor byte layout is equivalent to what the existing macros produce; uplink and the negotiated rate are unchanged without it (16.4/16.5/17.0 against 15.4/16.7/16.6 Mbit/s, MCS7 both ways).
- Dropped "fw: send the media status report in the vendor layout". The role field the vendor sets makes no difference here, as with the other chips.
- Dropped "sdio: handle the RTL8723BS management TX path". Sequence numbers do work for management frames on this chip, so reporting completion at DMA completion is unnecessary.
- Dropped "calibrate and tune the PHY" entirely. All three parts were vendor-matching scaffolding and none survived measurement: the scan-time initial gain override is worse than rtw_phy_dig_set_max_coverage() (five scans found 41 BSSes without it against 34 with it); the IQ calibration works from the normal phy_calibration path; and the hardcoded per-rate TX AGC table overrode the efuse calibration by up to 22 index units and bypassed the regulatory limit the by-rate path applies, which is not something a driver should do.
- Dropped "match the RTL8723BS firmware connect and power save behaviour". Deferring the connect report changes nothing, and the LPS gating cannot be justified from measurement: with RTW_DBG_PS enabled, LPS never engages on this setup at all, so the gating never takes effect here.
- Dropped "advertise the correct receive capabilities". Both halves move into the chip driver, as described above.
- "fw: add the vendor firmware commands" now adds only GNT_BT. rtw88 already implements MACID_CFG and the WL channel info report, and the coexistence antenna select reserve became unused once the PHY patch went.
- The zero-length packet check is gated on SDIO rather than the chip

    id.

- The SDIO interrupt acknowledgment change is scoped to this chip.
- The association sequence no longer re-applies the BSS capability at association time; BSS_CHANGED_ERP_PREAMBLE and BSS_CHANGED_ERP_SLOT are handled later in the same callback.
- Register accesses that had raw addresses now use the existing REG_GNT_BT and REG_BT_COEX_ENH_INTR_CTRL, plus a named define for the BB antenna select register.

The implementation is based on the initial RTL8723B work by Michael Straube <[email protected]>:

https://github.com/mistraube/rtw88/tree/rtl8723bs

Based on the rtw-next branch from pkshih/rtw.
Luka Gejak (7):
wifi: rtw88: add the RTL8723B chip type and SDIO helper
wifi: rtw88: rx: mark zero length packets on RTL8723BS
wifi: rtw88: tx: extend the TX report purge timeout to RTL8723BS
wifi: rtw88: sdio: zero the padding added to a TX transfer
wifi: rtw88: sdio: track free TX pages and OQT credits for RTL8723BS
wifi: rtw88: sdio: set up RX aggregation and interrupts for RTL8723BS
wifi: rtw88: sdio: add TX back-pressure and retry on page starvation

 drivers/net/wireless/realtek/rtw88/main.h |   7 +
 drivers/net/wireless/realtek/rtw88/rx.c   |   8 +-
 drivers/net/wireless/realtek/rtw88/sdio.c | 504 ++++++++++++++++++++--
 drivers/net/wireless/realtek/rtw88/sdio.h |  24 +-
 drivers/net/wireless/realtek/rtw88/tx.c   |   5 +-
 5 files changed, 515 insertions(+), 33 deletions(-)

base-commit: 81510c3d6f2199889a4a936d8cf8b9e05911b10e

2.55.0

From: Luka Gejak <[email protected]>

The RTL8723B is a 802.11n combo chip whose SDIO variant, RTL8723BS, runs a Realtek vendor firmware that differs from the firmware used by the other rtw88 8723 family devices. Supporting it needs a number of small adjustments spread across the shared core, all of which have to be restricted to this one chip and bus combination.

Add the chip type and a helper that tests for it, so the changes that follow can be gated without repeating the chip and bus comparison.

---

Notes:
Changes in v11: none.
Changes in v10: none.
Changes in v9: none.
Changes in v8: none.
Changes in v7: none.
Changes in v6: none.
Changes in v5: none.

 drivers/net/wireless/realtek/rtw88/main.h | 7 +++++++
 1 file changed, 7 insertions(+)

index 6883fbd9f768..d59f6e323adf 100644
@@ -194,6 +194,7 @@ enum rtw_chip_type {
 	RTW_CHIP_TYPE_8723D,
 	RTW_CHIP_TYPE_8821C,
 	RTW_CHIP_TYPE_8703B,
+	RTW_CHIP_TYPE_8723B,
 	RTW_CHIP_TYPE_8821A,
 	RTW_CHIP_TYPE_8812A,
 	RTW_CHIP_TYPE_8814A,

@@ -2195,6 +2196,12 @@ static inline bool rtw_chip_has_tx_stbc(struct rtw_dev *rtwdev)
 	return rtwdev->chip->tx_stbc;
 }
 
+static inline bool rtw_is_8723bs(struct rtw_dev *rtwdev)
+{
+	return rtwdev->chip->id == RTW_CHIP_TYPE_8723B &&
+	       rtwdev->hci.type == RTW_HCI_TYPE_SDIO;
+}
+
 static inline u8 rtw_acquire_macid(struct rtw_dev *rtwdev)
 {
 	unsigned long mac_id;

2.55.0

7 patch(es) applied to rtw-next branch of rtw.git, thanks.

673662a0411e wifi: rtw88: add the RTL8723B chip type and SDIO helper 5f92fb993456 wifi: rtw88: rx: mark zero length packets on RTL8723BS 4d688f22ebbf wifi: rtw88: tx: extend the TX report purge timeout to RTL8723BS b81ead28aeb1 wifi: rtw88: sdio: zero the padding added to a TX transfer 5a95bd460175 wifi: rtw88: sdio: track free TX pages and OQT credits for RTL8723BS 8c17eb82a0fb wifi: rtw88: sdio: set up RX aggregation and interrupts for RTL8723BS 28eaf7971a20 wifi: rtw88: sdio: add TX back-pressure and retry on page starvation

---
https://github.com/pkshih/rtw.git

Apologies for the late reply but if it's useful:

Though I don't want to touch git repository, it is worth to have this for the record. :)

From: Luka Gejak <[email protected]>

Like the RTL8703B, the RTL8723BS reports receive descriptors with a zero packet length, which the vendor driver drops outright. rtw88 already flags these as having no PSDU for the RTL8703B, so extend the same handling rather than passing an empty frame up.

This is gated on the SDIO interface rather than the chip id, because it has only been observed there; the USB variant is not known to do it.

---

Notes:
Changes in v11: none.
Changes in v10: none.
Changes in v9: none.
Changes in v8: none.
Changes in v7: none.
Changes in v6: none.
Changes in v5: the zero length test now evaluates pkt_stat->pkt_len first, so the unlikely case short circuits before the chip test.

 drivers/net/wireless/realtek/rtw88/rx.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

index 01fd299abb7f..fde21c840ac0 100644
@@ -253,10 +253,12 @@ static void rtw_rx_fill_rx_status(struct rtw_dev *rtwdev,
 
 	rtw_rx_addr_match(rtwdev, pkt_stat, hdr);
 
-	/* Rtl8723cs driver checks for size < 14 or size > 8192 and
-	 * simply drops the packet.
+	/*
+	 * Rtl8723cs and rtl8723bs drivers check for size < 14 or size > 8192
+	 * and simply drop the packet.
 	 */
-	if (rtwdev->chip->id == RTW_CHIP_TYPE_8703B && pkt_stat->pkt_len == 0) {
+	if (pkt_stat->pkt_len == 0 &&
+	    (rtwdev->chip->id == RTW_CHIP_TYPE_8703B || rtw_is_8723bs(rtwdev))) {
 		rx_status->flag |= RX_FLAG_NO_PSDU;
 		rtw_dbg(rtwdev, RTW_DBG_RX, "zero length packet");
 	}

2.55.0

From: Luka Gejak <[email protected]>

Commit c80788f7c5ae ("wifi: rtw88: increase TX report timeout to fix race condition") raised the purge timeout to 2500 ms for the RTL8723DU, because the firmware can stay off channel during background scans for longer than the 500 ms default, which delays the TX reports and lets the purge timer drop the tracking skbs. The host stack then reads the missing status as loss and collapses TCP throughput.

The RTL8723BS runs the same vendor firmware over a slower SDIO host and hits the same race. Testers on ARM SDIO boards see "failed to get tx report from firmware" under load, with the same throughput collapse. Extend the 2500 ms timeout to the RTL8723BS.

---

Notes:
Changes in v11: none.
Changes in v10: none.
Changes in v9: none.
Changes in v8: none.
Changes in v7: none.
Changes in v6: none.
Changes in v5: none. Restored in v4 after being dropped in v2, because testers on slower ARM SDIO hosts hit the warning that commit c80788f7c5ae already fixes for the RTL8723DU.

 drivers/net/wireless/realtek/rtw88/tx.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

index 8786bbb421c2..4e110a457e71 100644
@@ -208,8 +208,9 @@ void rtw_tx_report_enqueue(struct rtw_dev *rtwdev, struct sk_buff *skb, u8 sn)
 	__skb_queue_tail(&tx_report->queue, skb);
 	spin_unlock_irqrestore(&tx_report->q_lock, flags);
 
-	if (rtwdev->chip->id == RTW_CHIP_TYPE_8723D &&
-	    rtwdev->hci.type == RTW_HCI_TYPE_USB)
+	if ((rtwdev->chip->id == RTW_CHIP_TYPE_8723D &&
+	     rtwdev->hci.type == RTW_HCI_TYPE_USB) ||
+	    rtw_is_8723bs(rtwdev))
 		timeout = msecs_to_jiffies(2500);
 
 	mod_timer(&tx_report->purge_timer, jiffies + timeout);

2.55.0

From: Luka Gejak <[email protected]>

rtw_sdio_write_port() rounds the transfer up with sdio_align_size() and then hands that length to sdio_memcpy_toio() while the skb still only holds skb->len bytes. The difference, between one and 511 bytes, is read from beyond the end of the frame and transmitted. Whether it stays inside the skb's allocation depends on how much tailroom the skb happens to have, so this is at best sending uninitialised memory over the air.

Pad the skb up to the transfer size first. __skb_pad() zeroes the added bytes, reallocates a cloned skb rather than writing into a buffer a clone still shares, and leaves skb->len alone, so nothing else in the transmit path has to change.

It must not free the skb on failure: rtw_sdio_write_data() frees the skb itself and rtw_sdio_process_tx_queue() requeues it, so both callers still own it and would double free.

Found while reworking this path for the RTL8723BS. Measured on RTL8723BS hardware, padding the transfer costs nothing observable: uplink is 19.5 to 19.8 Mbit/s padded against 20.9 to 21.1 Mbit/s unpadded in an interleaved A/B, with scans, reconnection and a UDP flood clean in both. The other SDIO parts sharing this path are untested; I have only the RTL8723BS.

---

Notes:
New in v11.

Ping-Ke asked for this to come before the RTL8723BS accounting patch rather than after the series, so that it backports on its own and so that it is clear it is an existing problem rather than something the RTL8723BS work introduced.

The pad_size local is the shape he asked for on v9: declared at the top, computed unconditionally, and tested with if (pad_size > 0).

 drivers/net/wireless/realtek/rtw88/sdio.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

index 5b40d74b16ee..8466abad972a 100644
@@ -636,6 +636,7 @@ static int rtw_sdio_write_port(struct rtw_dev *rtwdev, struct sk_buff *skb,
 			       enum rtw_tx_queue_type queue)
 {
 	struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv;
+	size_t pad_size;
 	bool bus_claim;
 	size_t txsize;
 	u32 txaddr;

@@ -646,6 +647,17 @@ static int rtw_sdio_write_port(struct rtw_dev *rtwdev, struct sk_buff *skb,
 		return -EINVAL;
 
 	txsize = sdio_align_size(rtwsdio->sdio_func, skb->len);
+	pad_size = txsize - skb->len;
+
+	if (pad_size > 0) {
+		/*
+		 * __skb_pad() must not free the skb on failure: both callers
+		 * still own it, one requeues it and the other frees it.
+		 */
+		ret = __skb_pad(skb, pad_size, false);
+		if (ret)
+			return ret;
+	}
 
 	ret = rtw_sdio_check_free_txpg(rtwdev, queue, txsize);
 	if (ret)

2.55.0

With __skb_pad(), it might increase CPU usage.
Could you roughly measure that?

Measured with the ftrace function profiler over a 30 second saturating uplink transfer, 18.6 Mbit/s, four cores:

  function              hits      time(us)   share of one core
  rtw_sdio_write_port  48358      26839023        89.2%
  sdio_memcpy_toio     47855      11783994        39.2%
  __skb_pad            48344        616489         2.1%
  pskb_expand_head     47869        466036         1.6%

__skb_pad() is about 2% of one core with the link saturated. The whole transfer takes 14.8% of the four cores against 0.77% idle, so the padding is roughly 3.5% of the CPU the transfer already uses and 2.3% of the driver's write path. On throughput it is about 6%, which is the A/B in the commit message.

pskb_expand_head() runs on 99% of the calls, so nearly every frame takes the reallocating path rather than the memset. That is three quarters of the cost; without it the padding would be around 0.5% of one core. I have not worked out yet whether the skb is cloned or short of tailroom. If it is tailroom it is probably avoidable, and I can chase it as a follow up.

I do not think this is a blocker. A few percent seems a fair price for not putting uninitialised memory on the air.

One thing to decide, though. The commit message says the padding "costs nothing observable", which is too strong given the numbers in that same paragraph. Do you think it needs rewording? If so, would you be willing to amend it yourself when applying? That seems better than a resend of seven patches for one line.

Thanks for the Acked-by tag, one both patches, I will send 2nd series for the actual driver code after this prep series is merged to rtw-next.

Best regards,
Luka Gejak

Following up on the padding cost, since I said I would find out where it goes.

It is tailroom, not cloning. Over 70000 padded frames, none were cloned:
frames average 1575 bytes and need about 471 bytes of padding, but arrive with about 106 bytes of tailroom, so __skb_pad() reallocates on 98% of them. mac80211 reserves IEEE80211_ENCRYPT_TAILROOM, 18 bytes, and offers no way for a driver to ask for more on TX; extra_tx_headroom is headroom and extra_beacon_tailroom is beacons only.

So I tried the other way round, a preallocated per-device buffer: copy the frame plus the zeros into it and transfer from there, which avoids the allocation. Interleaved A/B, 20 second uplink runs:

  mode         throughput   CPU (4 cores)   __skb_pad
  __skb_pad    18.6, 18.3      14.3, 15.0    444, 449 ms
  bounce       19.0, 19.8      13.8, 14.3        0 ms

That recovers most of it, roughly 2% of one core and 3 to 5% of throughput.

I am not proposing it for this series. It is a workaround for the missing tailroom knob, it costs a copy, and it is only clean where a lock already serialises the transfer, which is true for this chip but not for the generic path. If you would rather have it as a separate patch later, or think the tailroom knob is worth raising with Johannes, I am happy either way.

Best regards,
Luka Gejak

[...]

You can modify mac80211 to reserve larger ndev->needed_tailroom to see if it can really resolve the symptom. If so, you can propose to add fields for tailroom like headroom:

struct ieee80211_hw:: extra_tx_tailroom
struct ieee80211_local:: tx_headroom

Ping-Ke

Note that the ndev->needed_tailroom isn't guaranteed by comments, so doing some experiments by normal use case would be helpful to know if it's worth.

* @needed_tailroom: Extra tailroom the hardware may need, but not in all
* cases can this be guaranteed. Some cases also use
* LL_MAX_HEADER instead to allocate the skb

Ping-Ke

I ran it: one line in net/mac80211/iface.c, IEEE80211_ENCRYPT_TAILROOM
+ 512, nothing else. It does not help. Over 60000 frames the average tailroom went from 106 to 139 bytes against an average pad of 469, and 96% still reallocate.

Your caveat is why, and it is stronger than the comment suggests:
mac80211 never reads ndev->needed_tailroom as the field appears once in net/mac80211, which is the assignment itself. The only code that acts on it is skb_ensure_writable_head_tail(), whose one caller is net/dsa/user.c. And the protocols that do honour it read it when they allocate, which TCP never does, which is why only the few non-TCP frames moved the average.

So that tested the wrong knob rather than the idea.

Your fields do work. I built them and measured it. ieee80211_skb_resize() is the only place on the TX path that grows tailroom, and it derives tail_need from IEEE80211_ENCRYPT_TAILROOM alone, gated on the frame needing software crypto tailroom, which is false for data frames under hardware CCMP. The headroom side is already what you describe:
hw.extra_tx_headroom folds into local->tx_headroom and both callers add it into head_need. Adding hw.extra_tx_tailroom, folding it into a new local->tx_tailroom and adding that to tail_need outside the crypto gate delivers the tailroom. One further piece is needed: ieee80211_build_hdr() skips the resize entirely when it wants no headroom and the skb is not cloned, so that condition has to widen as well.

With the driver asking for one SDIO block, frames arriving short of tailroom go from 29056 in 30000 to 1 in 30000, and uplink is about 6% faster in an interleaved A/B.

One caveat though: this does not remove the reallocation, it moves it. pskb_expand_head() is called just as often, from ieee80211_skb_resize() now instead of from __skb_pad() in the driver. The driver side becomes much cheaper, 445 ms of __skb_pad per 20 s run against 82 ms, and total time in pskb_expand_head falls by about a third, but whole system CPU does not drop. The gain looks like moving the work off the SDIO critical path rather than doing less of it.

I can write that up for Johannes as a separate mac80211 patch, with the rtw88 side as a follow up. It is independent of this series, but considering your question was about CPU usage and this change doesn't overall reduce it, it is probably not worth it.

Best regards,
Luka Gejak

Thanks for the experiments. I think it is not worth...

I agree, just let me know when the series is merged so I can send driver series. Best regards,
Luka Gejak

I'll merge this series in coming days. Based on review experience of this series, you can review them yourself again before sending driver series.

Ping-Ke

Luka Gejak <[email protected]>

I can amend it myself as

"... padding the transfer costs about 2% observable over transfer."

Or you can help to reword the paragraph. I can copy & paste it.

Ping-Ke

Please amend it yourself, that is much better than a resend. Here is the change:

"padding the transfer costs nothing observable:" to "padding the transfer costs about 2% observable over transfer:"

Leave the rest as is, after semicolons there are numbers to explain that 2%.

Best regards,
Luka Gejak

From: Luka Gejak <[email protected]>

The RTL8723BS reports free TX page counts that the generic 8051 path reads back from the chip on every transfer, which is both slow over SDIO and unreliable on this part: the register frequently reads back zero while pages are in fact available. It also gates transmission on a free count in the SDIO output queue, REG_SDIO_OQT_FREE_PG, which rtw88 does not track at all. The vendor driver calls this the OQT free space and never expands the acronym; the register holds the number of further transfers the SDIO output queue can accept, and the chip discards writes that arrive when it has run out.

Mirror the vendor driver and keep the per-queue and public page counts in software, seeded at start and resynchronised from the chip only when the cached counts say there is not enough room. Wait for a free output queue entry before writing, and account for the pages consumed after a successful transfer.

rtw_sdio_write_port() becomes a dispatcher. It works out the transfer address and the aligned transfer size, which both paths need, and hands them to rtw_sdio_write_port_8723bs() or rtw_sdio_write_port_generic(). The transfer itself moves into rtw_sdio_write_to_port(), which both call, so the generic path is step for step what it was. The padding added by the previous patch stays in rtw_sdio_write_port(), so it still runs once for both paths and stays outside the credit mutex.

The RTL8723BS path keeps a separate length for the accounting. The chip charges pages by the frame length rather than by the padded transfer, as the vendor driver does, and the two differ just above a block boundary:
a 1025 byte frame is nine pages by length and twelve by the padded size.

The check, the output queue wait and the accounting are serialised by a mutex. The TX worker and the H2C path reach this function concurrently, and two writers that both pass the checks can otherwise claim the same pages and output queue entry, after which the chip silently discards whichever transfer arrives second. The vendor driver avoids the same race by funnelling all transmission through one thread.

Measured on RTL8723BS hardware against an iperf3 server one hop behind the AP, with the wlan0 byte counters as ground truth. On the generic path the association completes but no data passes at all: TCP and UDP both measure 0 bit/s in either direction. With this patch TCP is 25.3 Mbit/s up and 37.3 Mbit/s down, and UDP is 25.0 Mbit/s up at 0% loss.

---

Notes:
Changes in v11:
- rtw_sdio_write_port() works out the transfer address and the aligned transfer size and passes both down, since the two paths began the same way, as you asked.
- the padding added by the previous patch moves into rtw_sdio_write_to_port() with the transfer, so both paths keep it.
- the RTL8723BS path keeps its own length for the accounting. The chip charges pages by frame length rather than by the padded transfer, matching the vendor driver, and the two differ just above a block
boundary: a 1025 byte frame is nine pages by length and twelve by the padded size, so using the aligned size would over charge.
Changes in v10:
- the transfer size comes from sdio_align_size() now. On this card, which reports multi block support and a 512 byte block size, it is identical to the open coded rounding: checked at the block boundaries and then against every frame of a session, 80000 frames with no difference.
- the block padding is gone from this patch. Measured on hardware it is not something the chip needs. Transferring unpadded, exactly as the generic path does, gives 20.9 to 21.1 Mbit/s uplink against 19.5 to 19.8 padded, with scans, reconnection and a UDP flood clean either way. What the padding really fixes is that the aligned length is read from past the end of the frame, which every SDIO part does today, so it is a separate fix and not part of this patch.
- dropped the comment above rtw_sdio_write_port_8723bs() and trimmed the ones on the page counters.
Changes in v9:
- rtw_sdio_write_port() is now only a dispatcher, with the two paths in rtw_sdio_write_port_8723bs() and rtw_sdio_write_port_generic(), as you asked. The chip specific one takes the mutex with guard() directly, so the extra rtw_sdio_8723bs_write_port() layer, which only existed to give the guard a scope while the generic path shared the outer function, is gone.
- the unaligned SKB warning is in rtw_sdio_write_to_port() now, once, next to the transfer whose pointer it checks and after any padding. __func__ therefore reads rtw_sdio_write_to_port on every SDIO part.
Changes in v8:
- RTW_TX_QUEUE_VO is accounted against the normal page pool rather than the high one. rtw_sdio_get_tx_addr() writes it to the normal transmit FIFO, and that is the pool the chip charges: over a saturating VO flood the high pool never lost a single page while the public pool drained. The vendor maps VO to the high queue, but it also writes VO to the high FIFO, which rtw88 does not.
- the output queue entry claimed before the transfer is handed back when the transfer fails, rather than staying claimed until the cache next resyncs from the chip.
- mutex_destroy() for tx_credit_lock on teardown and on the init error path, matching rtw_core_deinit().
- the comment on the cached page counters no longer explains the atomic_t by the several contexts that write them. Since v7 the transmit paths all update them under tx_credit_lock; what is left outside the lock is rtw_sdio_start() seeding them, and that is the reason they stay atomic_t.
- the free page check, the output queue wait, the transfer and the accounting moved into rtw_sdio_8723bs_write_port(), which takes the lock with guard(mutex) as you suggested. The generic path is a short branch in rtw_sdio_write_port() that never touches the lock, and the transfer itself is shared as rtw_sdio_write_to_port().
- lockdep_assert_held() added to rtw_sdio_8723bs_write_port(), rtw_sdio_8723bs_wait_tx_oqt() and rtw_sdio_8723bs_consume_txpg(), and checked on a kernel built with CONFIG_PROVE_LOCKING over bidirectional traffic and three module reloads: nothing fired.
- the padding uses a pad_size local and the comment is down to the one point that matters, that __skb_pad() must not free the skb.
- the comment above the lock is gone; the one at the declaration of tx_credit_lock covers it.
Changes in v7:
- the free page check, the output queue wait and the accounting after the transfer are serialised by a mutex. The TX worker and the H2C path run concurrently, and two writers that both passed the checks could claim the same pages and output queue entry; the OQT refill could also read the register while the other writer was between its claim and its transfer. The chip silently discards the overcommitted transfer, which for H2C means a lost firmware command.
- the CMD53 address is computed from skb->len again, as upstream does. Passing the aligned size changed the encoded transfer length for every other SDIO chip whenever sdio_align_size() padded; for the RTL8723BS the two encodings are the same value.
- the padding is applied with __skb_pad() and only on the RTL8723BS path. The open coded version wrote into a cloned skb's shared buffer when the tailroom happened to be large enough, and it also ran on the generic path, giving other chips a new allocation and failure mode. __skb_pad() does not move skb->len, so the trim on the way out is gone too. It must not free the skb on failure, since one caller requeues it and the other frees it.

Changes in v6: none.
Changes in v5:
- the commit message now says what OQT is, as far as the vendor driver reveals it.
- rtw_sdio_8723bs_sync_free_txpg() returns whether the chip reported anything and is the only caller of _store_free_txpg(), and _init_free_txpg() returns an error rather than nothing. That found a real bug: the public pool size was acq_pg_num minus the reserved queues with no check, so a chip coming up with no transmit page allocation would underflow a u16 and leave about 65000 free pages. It is now rtw_sdio_8723bs_pubq_num(), shared with the queue page allocation repair path, and it fails cleanly.
- the output queue wait is bounded by a jiffies deadline rather than a loop count, so RTW_SDIO_OQT_TIMEOUT_MS is really milliseconds.
- rtw_sdio_8723bs_check_rqpn() returns an error instead of silently doing nothing when the pool cannot cover the reserved queues, and rtw_sdio_start() propagates it. Both early returns are explained.

 drivers/net/wireless/realtek/rtw88/sdio.c | 349 ++++++++++++++++++++--
 drivers/net/wireless/realtek/rtw88/sdio.h |  12 +
 2 files changed, 334 insertions(+), 27 deletions(-)

index 8466abad972a..0dc1277619f6 100644
@@ -20,6 +20,7 @@
 #include "tx.h"
 
 #define RTW_SDIO_INDIRECT_RW_RETRIES			50
+#define RTW_SDIO_OQT_TIMEOUT_MS				1000
 
 static bool rtw_sdio_is_bus_addr(u32 addr)
 {

@@ -548,12 +549,149 @@ static int rtw_sdio_read_port(struct rtw_dev *rtwdev, u8 *buf, size_t count)
 	return ret;
 }
 
+/*
+ * The counters are atomic_t because rtw_sdio_start() seeds them outside
+ * tx_credit_lock, which the transmit paths hold while using them.
+ */
+static void rtw_sdio_8723bs_store_free_txpg(struct rtw_dev *rtwdev,
+					    u32 free_txpg)
+{
+	struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv;
+
+	atomic_set(&rtwsdio->free_pg_high,
+		   u32_get_bits(free_txpg, BIT_FREE_TXPG_HIGH));
+	atomic_set(&rtwsdio->free_pg_normal,
+		   u32_get_bits(free_txpg, BIT_FREE_TXPG_NORMAL));
+	atomic_set(&rtwsdio->free_pg_low,
+		   u32_get_bits(free_txpg, BIT_FREE_TXPG_LOW));
+	atomic_set(&rtwsdio->free_pg_pub,
+		   u32_get_bits(free_txpg, BIT_FREE_TXPG_PUB));
+}
+
+static bool rtw_sdio_8723bs_sync_free_txpg(struct rtw_dev *rtwdev)
+{
+	u32 free_txpg = rtw_read32(rtwdev, REG_SDIO_FREE_TXPG);
+
+	if (!free_txpg)
+		return false;
+
+	rtw_sdio_8723bs_store_free_txpg(rtwdev, free_txpg);
+
+	return true;
+}
+
+static int rtw_sdio_8723bs_pubq_num(struct rtw_dev *rtwdev, u16 *pubq_num)
+{
+	const struct rtw_page_table *pg_tbl = &rtwdev->chip->page_table[0];
+	u16 acq_pg_num = rtwdev->fifo.acq_pg_num;
+	u16 reserved_num;
+
+	reserved_num = pg_tbl->hq_num + pg_tbl->lq_num + pg_tbl->nq_num +
+		       pg_tbl->exq_num + pg_tbl->gapq_num;
+	if (acq_pg_num <= reserved_num) {
+		rtw_err(rtwdev,
+			"no transmit pages left for the public queue: %u of %u reserved\n",
+			reserved_num, acq_pg_num);
+		return -EINVAL;
+	}
+
+	*pubq_num = acq_pg_num - reserved_num;
+
+	return 0;
+}
+
+static int rtw_sdio_8723bs_init_free_txpg(struct rtw_dev *rtwdev)
+{
+	struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv;
+	const struct rtw_page_table *pg_tbl;
+	u16 pubq_num;
+	int ret;
+
+	/* Seed from the page table when the chip has nothing to report yet. */
+	if (!rtw_sdio_8723bs_sync_free_txpg(rtwdev)) {
+		ret = rtw_sdio_8723bs_pubq_num(rtwdev, &pubq_num);
+		if (ret)
+			return ret;
+
+		pg_tbl = &rtwdev->chip->page_table[0];
+		atomic_set(&rtwsdio->free_pg_high, pg_tbl->hq_num);
+		atomic_set(&rtwsdio->free_pg_normal, pg_tbl->nq_num);
+		atomic_set(&rtwsdio->free_pg_low, pg_tbl->lq_num);
+		atomic_set(&rtwsdio->free_pg_pub, pubq_num);
+	}
+
+	atomic_set(&rtwsdio->tx_oqt_free,
+		   rtw_read8(rtwdev, REG_SDIO_OQT_FREE_PG));
+
+	return 0;
+}
+
+/*
+ * Clamped at zero: a lost update against rtw_sdio_8723bs_consume_txpg() can
+ * briefly drive a counter negative, and wrapping would hide the shortage
+ * instead of triggering a resync from the chip.
+ */
+static unsigned int rtw_sdio_8723bs_pages_free(struct rtw_dev *rtwdev,
+					       atomic_t *dedicated)
+{
+	struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv;
+	int free;
+
+	free = atomic_read(dedicated) + atomic_read(&rtwsdio->free_pg_pub);
+
+	return free > 0 ? free : 0;
+}
+
+/*
+ * The pool a queue draws from follows the transmit FIFO that
+ * rtw_sdio_get_tx_addr() writes into, since that is the one the chip charges.
+ * RTW_TX_QUEUE_MGMT is the exception: it goes to the extra FIFO, which
+ * REG_SDIO_FREE_TXPG has no counter for and this chip allocates no pages to,
+ * so it is accounted against the high pool and served from the public one.
+ */
+static atomic_t *rtw_sdio_8723bs_free_txpg(struct rtw_dev *rtwdev, u8 queue)
+{
+	struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv;
+
+	switch (queue) {
+	case RTW_TX_QUEUE_VI:
+	case RTW_TX_QUEUE_VO:
+		return &rtwsdio->free_pg_normal;
+	case RTW_TX_QUEUE_BE:
+	case RTW_TX_QUEUE_BK:
+		return &rtwsdio->free_pg_low;
+	case RTW_TX_QUEUE_BCN:
+	case RTW_TX_QUEUE_H2C:
+	case RTW_TX_QUEUE_HI0:
+	case RTW_TX_QUEUE_MGMT:
+		return &rtwsdio->free_pg_high;
+	default:
+		return NULL;
+	}
+}
+
 static int rtw_sdio_check_free_txpg(struct rtw_dev *rtwdev, u8 queue,
 				    size_t count)
 {
 	unsigned int pages_free, pages_needed;
 
-	if (rtw_chip_wcpu_8051(rtwdev)) {
+	if (rtw_is_8723bs(rtwdev)) {
+		atomic_t *dedicated;
+
+		dedicated = rtw_sdio_8723bs_free_txpg(rtwdev, queue);
+		if (!dedicated) {
+			rtw_warn(rtwdev, "Unknown mapping for queue %u\n", queue);
+			return -EINVAL;
+		}
+
+		pages_free = rtw_sdio_8723bs_pages_free(rtwdev, dedicated);
+		pages_needed = DIV_ROUND_UP(count, rtwdev->chip->page_size);
+		if (pages_needed <= pages_free)
+			return 0;
+
+		rtw_sdio_8723bs_sync_free_txpg(rtwdev);
+		pages_free = rtw_sdio_8723bs_pages_free(rtwdev, dedicated);
+	} else if (rtw_chip_wcpu_8051(rtwdev)) {
 		u32 free_txpg;
 
 		free_txpg = rtw_sdio_read32(rtwdev, REG_SDIO_FREE_TXPG);

@@ -632,36 +770,60 @@ static int rtw_sdio_check_free_txpg(struct rtw_dev *rtwdev, u8 queue,
 	return 0;
 }
 
-static int rtw_sdio_write_port(struct rtw_dev *rtwdev, struct sk_buff *skb,
-			       enum rtw_tx_queue_type queue)
+static int rtw_sdio_8723bs_wait_tx_oqt(struct rtw_dev *rtwdev)
 {
 	struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv;
-	size_t pad_size;
-	bool bus_claim;
-	size_t txsize;
-	u32 txaddr;
-	int ret;
+	unsigned long deadline;
+	u8 free;
 
-	txaddr = rtw_sdio_get_tx_addr(rtwdev, skb->len, queue);
-	if (!txaddr)
-		return -EINVAL;
+	lockdep_assert_held(&rtwsdio->tx_credit_lock);
 
-	txsize = sdio_align_size(rtwsdio->sdio_func, skb->len);
-	pad_size = txsize - skb->len;
+	if (atomic_add_unless(&rtwsdio->tx_oqt_free, -1, 0))
+		return 0;
 
-	if (pad_size > 0) {
-		/*
-		 * __skb_pad() must not free the skb on failure: both callers
-		 * still own it, one requeues it and the other frees it.
-		 */
-		ret = __skb_pad(skb, pad_size, false);
-		if (ret)
-			return ret;
-	}
+	deadline = jiffies + msecs_to_jiffies(RTW_SDIO_OQT_TIMEOUT_MS);
+	do {
+		free = rtw_read8(rtwdev, REG_SDIO_OQT_FREE_PG);
+		if (free) {
+			atomic_set(&rtwsdio->tx_oqt_free, free - 1);
+			return 0;
+		}
+		usleep_range(1000, 2000);
+	} while (time_before(jiffies, deadline));
 
-	ret = rtw_sdio_check_free_txpg(rtwdev, queue, txsize);
-	if (ret)
-		return ret;
+	return -EBUSY;
+}
+
+static void rtw_sdio_8723bs_consume_txpg(struct rtw_dev *rtwdev, u8 queue,
+					 unsigned int pages)
+{
+	struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv;
+	atomic_t *dedicated;
+	unsigned int taken;
+	int free;
+
+	lockdep_assert_held(&rtwsdio->tx_credit_lock);
+
+	dedicated = rtw_sdio_8723bs_free_txpg(rtwdev, queue);
+	if (!dedicated)
+		return;
+
+	free = atomic_read(dedicated);
+	taken = min_t(unsigned int, pages, free > 0 ? free : 0);
+	atomic_sub(taken, dedicated);
+
+	pages -= taken;
+	if (pages && atomic_sub_return(pages, &rtwsdio->free_pg_pub) < 0)
+		atomic_set(&rtwsdio->free_pg_pub, 0);
+}
+
+static int rtw_sdio_write_to_port(struct rtw_dev *rtwdev, struct sk_buff *skb,
+				  enum rtw_tx_queue_type queue, u32 txaddr,
+				  size_t write_size)
+{
+	struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv;
+	bool bus_claim;
+	int ret;
 
 	if (!IS_ALIGNED((unsigned long)skb->data, RTW_SDIO_DATA_PTR_ALIGN))
 		rtw_warn(rtwdev, "Got unaligned SKB in %s() for queue %u\n",

@@ -672,7 +834,8 @@ static int rtw_sdio_write_port(struct rtw_dev *rtwdev, struct sk_buff *skb,
 	if (bus_claim)
 		sdio_claim_host(rtwsdio->sdio_func);
 
-	ret = sdio_memcpy_toio(rtwsdio->sdio_func, txaddr, skb->data, txsize);
+	ret = sdio_memcpy_toio(rtwsdio->sdio_func, txaddr, skb->data,
+			       write_size);
 
 	if (bus_claim)
 		sdio_release_host(rtwsdio->sdio_func);

@@ -680,11 +843,95 @@ static int rtw_sdio_write_port(struct rtw_dev *rtwdev, struct sk_buff *skb,
 	if (ret)
 		rtw_warn(rtwdev,
 			 "Failed to write %zu byte(s) to SDIO port 0x%08x",
-			 txsize, txaddr);
+			 write_size, txaddr);
 
 	return ret;
 }
 
+static int rtw_sdio_write_port_8723bs(struct rtw_dev *rtwdev,
+				      struct sk_buff *skb,
+				      enum rtw_tx_queue_type queue, u32 txaddr,
+				      size_t write_size)
+{
+	struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv;
+	unsigned int pages;
+	size_t txsize;
+	int ret;
+
+	/* the chip charges pages by frame length, not by the padded transfer */
+	txsize = round_up(skb->len, 4);
+
+	guard(mutex)(&rtwsdio->tx_credit_lock);
+
+	ret = rtw_sdio_check_free_txpg(rtwdev, queue, txsize);
+	if (ret)
+		return ret;
+
+	ret = rtw_sdio_8723bs_wait_tx_oqt(rtwdev);
+	if (ret)
+		return ret;
+
+	ret = rtw_sdio_write_to_port(rtwdev, skb, queue, txaddr, write_size);
+	if (ret) {
+		/* nothing was queued, so hand the output queue entry back */
+		atomic_inc(&rtwsdio->tx_oqt_free);
+		return ret;
+	}
+
+	pages = DIV_ROUND_UP(txsize, rtwdev->chip->page_size);
+	rtw_sdio_8723bs_consume_txpg(rtwdev, queue, pages);
+
+	return 0;
+}
+
+static int rtw_sdio_write_port_generic(struct rtw_dev *rtwdev,
+				       struct sk_buff *skb,
+				       enum rtw_tx_queue_type queue, u32 txaddr,
+				       size_t write_size)
+{
+	int ret;
+
+	ret = rtw_sdio_check_free_txpg(rtwdev, queue, write_size);
+	if (ret)
+		return ret;
+
+	return rtw_sdio_write_to_port(rtwdev, skb, queue, txaddr, write_size);
+}
+
+static int rtw_sdio_write_port(struct rtw_dev *rtwdev, struct sk_buff *skb,
+			       enum rtw_tx_queue_type queue)
+{
+	struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv;
+	size_t write_size;
+	size_t pad_size;
+	u32 txaddr;
+	int ret;
+
+	txaddr = rtw_sdio_get_tx_addr(rtwdev, skb->len, queue);
+	if (!txaddr)
+		return -EINVAL;
+
+	write_size = sdio_align_size(rtwsdio->sdio_func, skb->len);
+	pad_size = write_size - skb->len;
+
+	if (pad_size > 0) {
+		/*
+		 * __skb_pad() must not free the skb on failure: both callers
+		 * still own it, one requeues it and the other frees it.
+		 */
+		ret = __skb_pad(skb, pad_size, false);
+		if (ret)
+			return ret;
+	}
+
+	if (rtw_is_8723bs(rtwdev))
+		return rtw_sdio_write_port_8723bs(rtwdev, skb, queue, txaddr,
+						  write_size);
+
+	return rtw_sdio_write_port_generic(rtwdev, skb, queue, txaddr,
+					   write_size);
+}
+
 static void rtw_sdio_init(struct rtw_dev *rtwdev)
 {
 	struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv;

@@ -761,8 +1008,51 @@ static int rtw_sdio_setup(struct rtw_dev *rtwdev)
 	return 0;
 }
 
+/*
+ * Reprogram the queue page allocation if the chip came up with none. This is
+ * a repair path, not part of the normal start sequence: a non-zero free page
+ * count means the allocation latched during power on and must be left alone,
+ * and without a transmit page pool there is nothing to divide up either.
+ */
+static int rtw_sdio_8723bs_check_rqpn(struct rtw_dev *rtwdev)
+{
+	const struct rtw_chip_info *chip = rtwdev->chip;
+	struct rtw_fifo_conf *fifo = &rtwdev->fifo;
+	const struct rtw_page_table *pg_tbl;
+	u32 free_txpg;
+	u16 pubq_num;
+	int ret;
+
+	free_txpg = rtw_read32(rtwdev, REG_SDIO_FREE_TXPG);
+	if (free_txpg || !fifo->acq_pg_num)
+		return 0;
+
+	ret = rtw_sdio_8723bs_pubq_num(rtwdev, &pubq_num);
+	if (ret)
+		return ret;
+
+	pg_tbl = &chip->page_table[0];
+	rtw_write32(rtwdev, REG_RQPN_NPQ,
+		    BIT_RQPN_NE(pg_tbl->nq_num, pg_tbl->exq_num));
+	rtw_write32(rtwdev, REG_RQPN,
+		    BIT_RQPN_HLP(pg_tbl->hq_num, pg_tbl->lq_num, pubq_num));
+
+	return 0;
+}
+
 static int rtw_sdio_start(struct rtw_dev *rtwdev)
 {
+	if (rtw_is_8723bs(rtwdev)) {
+		int ret = rtw_sdio_8723bs_check_rqpn(rtwdev);
+
+		if (ret)
+			return ret;
+
+		ret = rtw_sdio_8723bs_init_free_txpg(rtwdev);
+		if (ret)
+			return ret;
+	}
+
 	rtw_sdio_enable_rx_aggregation(rtwdev);
 	rtw_sdio_enable_interrupt(rtwdev);
 

@@ -1306,6 +1596,8 @@ static int rtw_sdio_init_tx(struct rtw_dev *rtwdev)
 		return -ENOMEM;
 	}
 
+	mutex_init(&rtwsdio->tx_credit_lock);
+
 	for (i = 0; i < RTK_MAX_TX_QUEUE_NUM; i++)
 		skb_queue_head_init(&rtwsdio->tx_queue[i]);
 	rtwsdio->tx_handler_data = kmalloc_obj(*rtwsdio->tx_handler_data);

@@ -1318,6 +1610,7 @@ static int rtw_sdio_init_tx(struct rtw_dev *rtwdev)
 	return 0;
 
 err_destroy_wq:
+	mutex_destroy(&rtwsdio->tx_credit_lock);
 	destroy_workqueue(rtwsdio->txwq);
 	return -ENOMEM;
 }

@@ -1332,6 +1625,8 @@ static void rtw_sdio_deinit_tx(struct rtw_dev *rtwdev)
 
 	for (i = 0; i < RTK_MAX_TX_QUEUE_NUM; i++)
 		ieee80211_purge_tx_queue(rtwdev->hw, &rtwsdio->tx_queue[i]);
+
+	mutex_destroy(&rtwsdio->tx_credit_lock);
 }
 
 int rtw_sdio_probe(struct sdio_func *sdio_func,

index 457e8b02380e..f43f1c6309b7 100644
@@ -86,6 +86,10 @@
 #define REG_SDIO_OQT_FREE_PG			(SDIO_LOCAL_OFFSET + 0x001E)
 /* Free Tx Buffer Page */
 #define REG_SDIO_FREE_TXPG			(SDIO_LOCAL_OFFSET + 0x0020)
+#define BIT_FREE_TXPG_HIGH			GENMASK(7, 0)
+#define BIT_FREE_TXPG_NORMAL			GENMASK(15, 8)
+#define BIT_FREE_TXPG_LOW			GENMASK(23, 16)
+#define BIT_FREE_TXPG_PUB			GENMASK(31, 24)
 /* HCI Current Power Mode 1 */
 #define REG_SDIO_HCPWM1				(SDIO_LOCAL_OFFSET + 0x0024)
 /* HCI Current Power Mode 2 */

@@ -159,6 +163,14 @@ struct rtw_sdio {
 	struct workqueue_struct *txwq;
 	struct rtw_sdio_work_data *tx_handler_data;
 	struct sk_buff_head tx_queue[RTK_MAX_TX_QUEUE_NUM];
+
+	atomic_t free_pg_high;
+	atomic_t free_pg_normal;
+	atomic_t free_pg_low;
+	atomic_t free_pg_pub;
+	atomic_t tx_oqt_free;
+	/* one writer at a time between the credit check and the accounting */
+	struct mutex tx_credit_lock;
 };
 
 extern const struct dev_pm_ops rtw_sdio_pm_ops;

2.55.0

From: Luka Gejak <[email protected]>

Enable the existing RX aggregation setup for this chip and select the larger DMA burst count it needs. The RTL8723BS does not raise CPWM1, so leave that source out of its interrupt mask, and set the SDIO TX control bit the vendor driver uses to have transfers always recognised.

The chip was also seen to keep raising the interrupt after resume when undefined status bits were written back on acknowledgment, so acknowledge only the bits this driver defines. That observation dates from bring up and I cannot re-measure it: the test machine only offers s2idle and does not reliably return from it, so the resume path is not exercised here. The change is scoped to this chip; the other SDIO parts keep writing the status word back unchanged.

---

Notes:
Changes in v11: none.
Changes in v10: none.
Changes in v9: none.
Changes in v8: none.
Changes in v7: none.
Changes in v6: none.
Changes in v5: fixed the interrupt acknowledgment. It masked the status word with both irq_mask and RTW_SDIO_HISR_CLEAR_MASK, but for this chip irq_mask is REG_SDIO_HIMR_RX_REQUEST alone and that bit is not in the clear mask, so the two had no bits in common and the driver acknowledged nothing at all. It now masks with the defined bits only, which is what the comment described. The commit message also no longer presents the resume behaviour as re-measured, because it is not.

 drivers/net/wireless/realtek/rtw88/sdio.c | 28 ++++++++++++++++++++++-
 drivers/net/wireless/realtek/rtw88/sdio.h |  9 ++++++++
 2 files changed, 36 insertions(+), 1 deletion(-)

index 0dc1277619f6..696cd203919e 100644
@@ -936,7 +936,11 @@ static void rtw_sdio_init(struct rtw_dev *rtwdev)
 {
 	struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv;
 
-	rtwsdio->irq_mask = REG_SDIO_HIMR_RX_REQUEST | REG_SDIO_HIMR_CPWM1;
+	if (rtw_is_8723bs(rtwdev))
+		rtwsdio->irq_mask = REG_SDIO_HIMR_RX_REQUEST;
+	else
+		rtwsdio->irq_mask = REG_SDIO_HIMR_RX_REQUEST |
+				    REG_SDIO_HIMR_CPWM1;
 }
 
 static void rtw_sdio_enable_rx_aggregation(struct rtw_dev *rtwdev)

@@ -944,6 +948,7 @@ static void rtw_sdio_enable_rx_aggregation(struct rtw_dev *rtwdev)
 	u8 size, timeout;
 
 	switch (rtwdev->chip->id) {
+	case RTW_CHIP_TYPE_8723B:
 	case RTW_CHIP_TYPE_8703B:
 	case RTW_CHIP_TYPE_8821A:
 	case RTW_CHIP_TYPE_8812A:

@@ -971,6 +976,8 @@ static void rtw_sdio_enable_rx_aggregation(struct rtw_dev *rtwdev)
 		    FIELD_PREP(BIT_DMA_AGG_TO_V1, timeout));
 
 	rtw_write8_set(rtwdev, REG_RXDMA_MODE, BIT_DMA_MODE);
+	if (rtw_is_8723bs(rtwdev))
+		rtw_write8_mask(rtwdev, REG_RXDMA_MODE, BIT_DMA_BURST_CNT, 0x3);
 }
 
 static void rtw_sdio_enable_interrupt(struct rtw_dev *rtwdev)

@@ -1042,6 +1049,8 @@ static int rtw_sdio_8723bs_check_rqpn(struct rtw_dev *rtwdev)
 
 static int rtw_sdio_start(struct rtw_dev *rtwdev)
 {
+	u32 clear;
+
 	if (rtw_is_8723bs(rtwdev)) {
 		int ret = rtw_sdio_8723bs_check_rqpn(rtwdev);
 

@@ -1054,6 +1063,13 @@ static int rtw_sdio_start(struct rtw_dev *rtwdev)
 	}
 
 	rtw_sdio_enable_rx_aggregation(rtwdev);
+
+	if (rtw_is_8723bs(rtwdev)) {
+		clear = rtw_read32(rtwdev, REG_SDIO_HISR) & RTW_SDIO_HISR_CLEAR_MASK;
+		if (clear)
+			rtw_write32(rtwdev, REG_SDIO_HISR, clear);
+	}
+
 	rtw_sdio_enable_interrupt(rtwdev);
 
 	return 0;

@@ -1133,6 +1149,8 @@ static void rtw_sdio_interface_cfg(struct rtw_dev *rtwdev)
 
 	val = rtw_read32(rtwdev, REG_SDIO_TX_CTRL);
 	val &= 0xfff8;
+	if (rtw_is_8723bs(rtwdev))
+		val |= BIT_SDIO_TX_CTRL_ALWAYS_RECOGNIZE;
 	rtw_write32(rtwdev, REG_SDIO_TX_CTRL, val);
 }
 

@@ -1397,6 +1415,14 @@ static void rtw_sdio_handle_interrupt(struct sdio_func *sdio_func)
 		rtw_sdio_rx_isr(rtwdev);
 	}
 
+	/*
+	 * RTL8723BS keeps raising the interrupt after resume if undefined
+	 * status bits are written back, so acknowledge only the bits this
+	 * driver defines. Other chips keep the existing behaviour.
+	 */
+	if (rtw_is_8723bs(rtwdev))
+		hisr &= RTW_SDIO_HISR_CLEAR_MASK;
+
 	rtw_write32(rtwdev, REG_SDIO_HISR, hisr);
 
 	rtwsdio->irq_thread = NULL;

index f43f1c6309b7..634c0b1339bb 100644
@@ -22,6 +22,7 @@
 
 /* SDIO Tx Control */
 #define REG_SDIO_TX_CTRL			(SDIO_LOCAL_OFFSET + 0x0000)
+#define BIT_SDIO_TX_CTRL_ALWAYS_RECOGNIZE	BIT(4)
 
 /*SDIO status timeout*/
 #define REG_SDIO_TIMEOUT			(SDIO_LOCAL_OFFSET + 0x0002)

@@ -77,6 +78,14 @@
 /* the following two are RTL8188 SDIO Specific */
 #define REG_SDIO_HISR_MCU_ERR			BIT(28)
 #define REG_SDIO_HISR_TSF_BIT32_TOGGLE		BIT(29)
+#define RTW_SDIO_HISR_CLEAR_MASK		\
+	(REG_SDIO_HISR_TXERR | REG_SDIO_HISR_RXERR | \
+	 REG_SDIO_HISR_TXFOVW | REG_SDIO_HISR_RXFOVW | \
+	 REG_SDIO_HISR_TXBCNOK | REG_SDIO_HISR_TXBCNERR | \
+	 REG_SDIO_HISR_C2HCMD | REG_SDIO_HISR_CPWM1 | \
+	 REG_SDIO_HISR_CPWM2 | REG_SDIO_HISR_HSISR_IND | \
+	 REG_SDIO_HISR_GTINT3_IND | REG_SDIO_HISR_GTINT4_IND | \
+	 REG_SDIO_HISR_PSTIMEOUT | REG_SDIO_HISR_OCPINT)
 
 /* HCI Current Power Mode */
 #define REG_SDIO_HCPWM				(SDIO_LOCAL_OFFSET + 0x0019)

2.55.0

From: Luka Gejak <[email protected]>

Two problems show up on RTL8723BS uplink. The per-AC software FIFO is unbounded, so mac80211 keeps handing frames down until latency collapses under load. And when a transfer cannot be completed the queue is simply abandoned for that pass, which stalls the AC until something else kicks the worker.

Stop the mac80211 queue once a data AC fills past a high watermark and wake it from the drain path when it falls back to a low one. Both sides take the TX queue lock across the length check and the flag update, the way rtw_pci_tx_write() and rtw_pci_tx_isr() use irq_lock. Without it the producer could stop a queue on a length the drain path had already emptied, having seen the flag still clear and so skipped the wake, and the AC would have stayed stopped with nothing left to wake it.

Convert the TX work item to a delayed work and re-arm it when a transfer fails for a reason that can clear on its own, so it is retried rather than the AC abandoned, and cancel the work on teardown.

Retrying matters once the queue can be stopped. A stopped queue is handed no further frames, so nothing else would kick the worker, and the AC would stay stopped for good with the link still up and receive unaffected. The two retried cases, a transmit page or output queue shortage and a failed skb expansion, are also the two that fail silently; the rest are logged where they happen, so they are visible rather than an unexplained hang, and they keep the existing behaviour rather than being retried indefinitely.

Measured on RTL8723BS hardware, uplink goes from 11.9 Mbit/s with 204 TCP retransmits to 20.1 Mbit/s with 2.

---

Notes:
Changes in v11:
- -ENOMEM is retried again. It became reachable once more when the padding moved into the series, since __skb_pad() can return it, and without the retry a stopped access category has nothing left to kick it. This is the consequence noted in the v10 changelog.
- one comment shortened to fit in 80 columns.
Changes in v10:
- -ENOMEM is no longer retried. With the padding moved out of patch 4 nothing on this path can return it, so the case was dead. If the padding fix lands in rtw_sdio_write_port() it becomes reachable again, for every chip, and this has to come back with it.
- trimmed the comments here too.
Changes in v9: none, beyond recording your Acked-by.
Changes in v8:
- the stop and the wake take the TX queue lock with guard(spinlock_irqsave) across the length check and the flag update, following pci.c, which holds irq_lock on both sides and needs no barrier because of it. The smp_mb() pair, the READ_ONCE and WRITE_ONCE accessors and the re-check that undid a stop all went with it.
- q_map is read later, just before the call that uses it. It cannot move past rtw_sdio_indicate_tx_status(), which consumes the skb, so that is as far down as it goes.
Changes in v7:
- the mac80211 queue index is read before skb_queue_tail() publishes the skb to the TX worker, which may process and free it immediately; reading it afterwards was a use after free.
- the stop path re-checks the FIFO after setting the stopped flag and undoes the stop if the worker drained it meanwhile. In that window the drain side still saw the flag clear, so neither side would have woken the queue and the AC stayed stopped for good. The flag is accessed with READ_ONCE/WRITE_ONCE and the re-check is ordered against the drain path with a barrier pair.

Changes in v6:
- dropped rtw_sdio_reschedule_tx_work(). It was a thin wrapper around queue_delayed_work() and hid the kernel API for no gain.
- the reschedule conditions moved into rtw_sdio_8723bs_reschedule_tx(), so rtw_sdio_tx_handler() no longer explains any chip specific condition in the common flow.
- dropped the unconditional break on a failed transfer. v4 and v5 had it, and it quietly changed the other SDIO parts: upstream requeues the frame and the loop retries, and breaking gave up after the first failure. The two errors this chip needs to retry are handled in the helper above, so the rest can keep the existing behaviour and the other parts are untouched again.
Changes in v5:
- the back-pressure and wake conditions moved into rtw_sdio_8723bs_stop_tx_queue() and _wake_tx_queue().
- rtw_sdio_process_tx_queue() returns 0 on success and 1 for the empty queue case, rather than the other way round.
- queue_stopped[] renamed to tx_queue_stopped[].
- fixed a transmit stall: the work item was only re-armed on a page shortage, so on any other failure a stopped access category would stay stopped for good. It now also re-arms on a failed skb expansion, which with the page shortage covers both failures that produce no log message.
- RTW_SDIO_TX_RETRY_DELAY is left as msecs_to_jiffies(1): it does not become 0 for HZ < 1000, since msecs_to_jiffies() rounds up.

 drivers/net/wireless/realtek/rtw88/sdio.c | 137 ++++++++++++++++++++--
 drivers/net/wireless/realtek/rtw88/sdio.h |   3 +-
 2 files changed, 129 insertions(+), 11 deletions(-)

index 696cd203919e..dc2fd0f8f9ff 100644
@@ -22,6 +22,11 @@
 #define RTW_SDIO_INDIRECT_RW_RETRIES			50
 #define RTW_SDIO_OQT_TIMEOUT_MS				1000
 
+/* Bounds the queueing latency of the unbounded per-AC software FIFO. */
+#define RTW_SDIO_TX_FIFO_HIWATER			16
+#define RTW_SDIO_TX_FIFO_LOWATER			8
+#define RTW_SDIO_TX_RETRY_DELAY			msecs_to_jiffies(1)
+
 static bool rtw_sdio_is_bus_addr(u32 addr)
 {
 	return !!(addr & RTW_SDIO_BUS_MSK);

@@ -1133,7 +1138,11 @@ static void rtw_sdio_tx_kick_off(struct rtw_dev *rtwdev)
 {
 	struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv;
 
-	queue_work(rtwsdio->txwq, &rtwsdio->tx_handler_data->work);
+	/*
+	 * A retry may already be pending with a delay; re-arm it so a newly
+	 * queued frame is not held back by it.
+	 */
+	mod_delayed_work(rtwsdio->txwq, &rtwsdio->tx_handler_data->work, 0);
 }
 
 static void rtw_sdio_link_ps(struct rtw_dev *rtwdev, bool enter)

@@ -1242,12 +1251,61 @@ static int rtw_sdio_write_data_h2c(struct rtw_dev *rtwdev, u8 *buf, u32 size)
 	return rtw_sdio_write_data(rtwdev, &pkt_info, skb, RTW_TX_QUEUE_H2C);
 }
 
+/*
+ * Both this and the wake below hold the TX queue lock across the length check
+ * and the flag update, so a queue is only ever stopped while it really is
+ * above the watermark, and the drain path always sees the flag the producer
+ * set.
+ */
+static void rtw_sdio_8723bs_stop_tx_queue(struct rtw_dev *rtwdev,
+					  enum rtw_tx_queue_type queue,
+					  u16 q_map)
+{
+	struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv;
+
+	if (!rtw_is_8723bs(rtwdev) || queue >= RTW_TX_QUEUE_BCN)
+		return;
+
+	guard(spinlock_irqsave)(&rtwsdio->tx_queue[queue].lock);
+
+	if (rtwsdio->tx_queue_stopped[queue])
+		return;
+
+	if (skb_queue_len(&rtwsdio->tx_queue[queue]) < RTW_SDIO_TX_FIFO_HIWATER)
+		return;
+
+	rtwsdio->tx_queue_stopped[queue] = true;
+	ieee80211_stop_queue(rtwdev->hw, q_map);
+}
+
+static void rtw_sdio_8723bs_wake_tx_queue(struct rtw_dev *rtwdev,
+					  enum rtw_tx_queue_type queue,
+					  u16 q_map)
+{
+	struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv;
+
+	if (!rtw_is_8723bs(rtwdev) || queue >= RTW_TX_QUEUE_BCN)
+		return;
+
+	guard(spinlock_irqsave)(&rtwsdio->tx_queue[queue].lock);
+
+	if (!rtwsdio->tx_queue_stopped[queue])
+		return;
+
+	if (skb_queue_len(&rtwsdio->tx_queue[queue]) > RTW_SDIO_TX_FIFO_LOWATER)
+		return;
+
+	rtwsdio->tx_queue_stopped[queue] = false;
+	ieee80211_wake_queue(rtwdev->hw, q_map);
+}
+
 static int rtw_sdio_tx_write(struct rtw_dev *rtwdev,
 			     struct rtw_tx_pkt_info *pkt_info,
 			     struct sk_buff *skb)
 {
 	struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv;
 	enum rtw_tx_queue_type queue = rtw_tx_queue_mapping(skb);
+	u16 q_map = skb_get_queue_mapping(skb);
 	struct rtw_sdio_tx_data *tx_data;
 
 	rtw_sdio_tx_skb_prepare(rtwdev, pkt_info, skb, queue);

@@ -1255,8 +1313,11 @@ static int rtw_sdio_tx_write(struct rtw_dev *rtwdev,
 	tx_data = rtw_sdio_get_tx_data(skb);
 	tx_data->sn = pkt_info->sn;
 
+	/* The TX worker may already have freed the skb, so do not touch it. */
 	skb_queue_tail(&rtwsdio->tx_queue[queue], skb);
 
+	rtw_sdio_8723bs_stop_tx_queue(rtwdev, queue, q_map);
+
 	return 0;
 }
 

@@ -1559,33 +1620,80 @@ static void rtw_sdio_indicate_tx_status(struct rtw_dev *rtwdev,
 	ieee80211_tx_status_irqsafe(hw, skb);
 }
 
-static void rtw_sdio_process_tx_queue(struct rtw_dev *rtwdev,
-				      enum rtw_tx_queue_type queue)
+/*
+ * Send one frame from @queue. Returns 0 when a frame was written, 1 when the
+ * queue was empty and a negative errno when the write failed, in which case
+ * the frame is put back at the head of the queue.
+ */
+static int rtw_sdio_process_tx_queue(struct rtw_dev *rtwdev,
+				     enum rtw_tx_queue_type queue)
 {
 	struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv;
 	struct sk_buff *skb;
+	u16 q_map;
 	int ret;
 
 	skb = skb_dequeue(&rtwsdio->tx_queue[queue]);
 	if (!skb)
-		return;
+		return 1;
 
 	ret = rtw_sdio_write_port(rtwdev, skb, queue);
 	if (ret) {
 		skb_queue_head(&rtwsdio->tx_queue[queue], skb);
-		return;
+		return ret;
 	}
 
+	/* rtw_sdio_indicate_tx_status() consumes the skb, so read this first */
+	q_map = skb_get_queue_mapping(skb);
+
 	rtw_sdio_indicate_tx_status(rtwdev, skb);
+
+	rtw_sdio_8723bs_wake_tx_queue(rtwdev, queue, q_map);
+
+	return 0;
+}
+
+/*
+ * A page or output queue shortage, and a failed skb expansion, are transient
+ * and leave the frame queued, so come back for it shortly. That matters once
+ * the mac80211 queue can be stopped: a stopped queue is handed no further
+ * frames, so nothing else would kick this work item and the access category
+ * would stay stopped for good. The remaining errors are logged where they
+ * happen and are not retried.
+ *
+ * After a management frame, restart from the highest priority queue so the
+ * join sequence is not held up behind a data backlog.
+ */
+static bool rtw_sdio_8723bs_reschedule_tx(struct rtw_dev *rtwdev,
+					  struct rtw_sdio_work_data *work_data,
+					  enum rtw_tx_queue_type queue, int ret)
+{
+	struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv;
+	unsigned long delay;
+
+	if (!rtw_is_8723bs(rtwdev))
+		return false;
+
+	if (ret == -EBUSY || ret == -ENOMEM)
+		delay = RTW_SDIO_TX_RETRY_DELAY;
+	else if (ret == 0 && queue == RTW_TX_QUEUE_MGMT)
+		delay = 0;
+	else
+		return false;
+
+	queue_delayed_work(rtwsdio->txwq, &work_data->work, delay);
+
+	return true;
 }
 
 static void rtw_sdio_tx_handler(struct work_struct *work)
 {
 	struct rtw_sdio_work_data *work_data =
-		container_of(work, struct rtw_sdio_work_data, work);
+		container_of(to_delayed_work(work), struct rtw_sdio_work_data,
+			     work);
 	struct rtw_sdio *rtwsdio;
 	struct rtw_dev *rtwdev;
-	int limit, queue;
+	int limit, queue, ret;
 
 	rtwdev = work_data->rtwdev;
 	rtwsdio = (struct rtw_sdio *)rtwdev->priv;

@@ -1595,7 +1703,13 @@ static void rtw_sdio_tx_handler(struct work_struct *work)
 
 	for (queue = RTK_MAX_TX_QUEUE_NUM - 1; queue >= 0; queue--) {
 		for (limit = 0; limit < 1000; limit++) {
-			rtw_sdio_process_tx_queue(rtwdev, queue);
+			ret = rtw_sdio_process_tx_queue(rtwdev, queue);
+			if (ret > 0)
+				break;
+
+			if (rtw_sdio_8723bs_reschedule_tx(rtwdev, work_data,
+							  queue, ret))
+				return;
 
 			if (skb_queue_empty(&rtwsdio->tx_queue[queue]))
 				break;

@@ -1624,14 +1738,16 @@ static int rtw_sdio_init_tx(struct rtw_dev *rtwdev)
 
 	mutex_init(&rtwsdio->tx_credit_lock);
 
-	for (i = 0; i < RTK_MAX_TX_QUEUE_NUM; i++)
+	for (i = 0; i < RTK_MAX_TX_QUEUE_NUM; i++) {
 		skb_queue_head_init(&rtwsdio->tx_queue[i]);
+		rtwsdio->tx_queue_stopped[i] = false;
+	}
 	rtwsdio->tx_handler_data = kmalloc_obj(*rtwsdio->tx_handler_data);
 	if (!rtwsdio->tx_handler_data)
 		goto err_destroy_wq;
 
 	rtwsdio->tx_handler_data->rtwdev = rtwdev;
-	INIT_WORK(&rtwsdio->tx_handler_data->work, rtw_sdio_tx_handler);
+	INIT_DELAYED_WORK(&rtwsdio->tx_handler_data->work, rtw_sdio_tx_handler);
 
 	return 0;
 

@@ -1646,6 +1762,7 @@ static void rtw_sdio_deinit_tx(struct rtw_dev *rtwdev)
 	struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv;
 	int i;
 
+	cancel_delayed_work_sync(&rtwsdio->tx_handler_data->work);
 	destroy_workqueue(rtwsdio->txwq);
 	kfree(rtwsdio->tx_handler_data);
 

index 634c0b1339bb..6e7e6009744b 100644
@@ -156,7 +156,7 @@ struct rtw_sdio_tx_data {
 };
 
 struct rtw_sdio_work_data {
-	struct work_struct work;
+	struct delayed_work work;
 	struct rtw_dev *rtwdev;
 };
 

@@ -172,6 +172,7 @@ struct rtw_sdio {
 	struct workqueue_struct *txwq;
 	struct rtw_sdio_work_data *tx_handler_data;
 	struct sk_buff_head tx_queue[RTK_MAX_TX_QUEUE_NUM];
+	bool tx_queue_stopped[RTK_MAX_TX_QUEUE_NUM];
 
 	atomic_t free_pg_high;
 	atomic_t free_pg_normal;

2.55.0