r8168 is needed to enter P-state: Package State 6 (pc6) on Haswell hardware
With in-kernel r8169 module, only P-state package C3 (pc3) can be reached when enabling ASPM.
Only after installing r8168 driver (and enabling ASPM), a Haswell Celeron G1820/G1840 processer will enter package C6 state (pc6).
Tested with Ubuntu kernel 3.13 x86_64
and Ubuntu mainline
Linux ubuntu14 3.17.0-999-generic #201409240305 SMP Wed Sep 24 02:07:04 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
I hope this gets fixed: Haswell processor entering pc6 with ASPM enabled and r8169 in-kernel module.
R8111GR hardware:
# lspci -nnkvv -s 03:00.0 03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 11) Subsystem: ASRock Incorporation Motherboard (one of many) [1849:8168] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 27 Region 0: I/O ports at e000 [size=256] Region 2: Memory at f0404000 (64-bit, non-prefetchable) [size=4K] Region 4: Memory at f0400000 (64-bit, prefetchable) [size=16K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee0100c Data: 41e1 Capabilities: [70] Express (v2) Endpoint, MSI 01 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 4096 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via message/WAKE# DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [b0] MSI-X: Enable- Count=4 Masked- Vector table: BAR=4 offset=00000000 PBA: BAR=4 offset=00000800 Capabilities: [d0] Vital Product Data Unknown small resource type 00, will not decode more. Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP+ BadDLLP- Rollover- Timeout+ NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [140 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- Capabilities: [160 v1] Device Serial Number <cut> Capabilities: [170 v1] Latency Tolerance Reporting Max snoop latency: 71680ns Max no snoop latency: 71680ns
Ceriel Jacobs <[email protected]> :
Quoted 5 lines
With in-kernel r8169 module, only P-state package C3 (pc3) can be reached when enabling ASPM.
Only after installing r8168 driver (and enabling ASPM), a Haswell Celeron G1820/G1840 processer will enter package C6 state (pc6).
Vanilla kernel r8169 driver logs a message that contains an "XID" string. Please grep for it in your dmesg so that I can narrow the search for meaningful differences in Realtek's r8168 driver.
Thanks.
Signature · 1 line
Ueimor
This is the requested "XID" dmesg line is:
[ 1.479298] r8169 0000:03:00.0 eth0: RTL8168g/8111g at 0xffffc90004752000, d0:50:99:0b:09:90, XID 0c000800 IRQ 27
Francois Romieu schreef op 05-10-14 om 18:59:
Quoted 13 lines
Ceriel Jacobs <[email protected]> :
Quoted 5 lines
With in-kernel r8169 module, only P-state package C3 (pc3) can be reached when enabling ASPM.
Only after installing r8168 driver (and enabling ASPM), a Haswell Celeron G1820/G1840 processer will enter package C6 state (pc6).
Vanilla kernel r8169 driver logs a message that contains an "XID" string. Please grep for it in your dmesg so that I can narrow the search for meaningful differences in Realtek's r8168 driver.
Thanks.
RE: r8168 is needed to enter P-state: Package State 6 (pc6)onHaswell hardware
In reply to Francois RomieuFrancois Romieu [mailto:[email protected]]
Quoted 1 line
Sent: Monday, October 06, 2014 12:59 AM
[...]
Quoted 4 lines
Quoted 4 lines
With in-kernel r8169 module, only P-state package C3 (pc3) can be reached
when enabling ASPM.
Quoted 8 lines
Quoted 3 lines
Only after installing r8168 driver (and enabling ASPM), a Haswell Celeron
G1820/G1840 processer will enter package C6 state (pc6).Vanilla kernel r8169 driver logs a message that contains an "XID" string.
Please grep for it in your dmesg so that I can narrow the search for meaningful differences in Realtek's r8168 driver.
I don't sure if the following information is helpful. Besides, I remember the rtl_init_one() would disable it.
Best Regards,
Hayes
Re: r8168 is needed to enter P-state: Package State 6 (pc6)onHaswell hardware
Hayes Wang <[email protected]> :
Quoted 1 line
Francois Romieu [mailto:[email protected]]
[...]
Quoted 6 lines
I don't sure if the following information is helpful. Besides, I remember the
rtl_init_one()would disable it.
Yes, I did not expect this stuff to stay in geostationary orbit for long :o/
Realtek's r8168 driver defaults to CONFIG_ASPM=1 but I guess some users need to disable it and there's no known pattern / blacklist, right ?
Ceriel, does the patch below against current kernel make a difference ?
drivers/net/ethernet/realtek/r8169.c +4 -1
@@ -468,6 +468,7 @@ enum rtl8168_registers {#define PWM_EN (1 << 22)#define RXDV_GATED_EN (1 << 19)#define EARLY_TALLY_EN (1 << 16) +#define FORCE_CLK (1 << 15)/* force clock request */ };enum rtl_register_content {
@@ -5279,8 +5280,10 @@ static void rtl_hw_start_8168g_1(struct rtl8169_private *tp)rtl_eri_write (tp,0x2f8 ,ERIAR_MASK_0011 ,0x1d8f ,ERIAR_EXGMAC );RTL_W8 (ChipCmd, CmdTxEnb | CmdRxEnb); -RTL_W32 (MISC ,RTL_R32 (MISC ) & ~RXDV_GATED_EN ); +RTL_W32 (MISC , (RTL_R32 (MISC ) |FORCE_CLK ) & ~RXDV_GATED_EN );RTL_W8 (MaxTxPacketSize, EarlySize); +RTL_W8 (Config5,RTL_R8 (Config5) | ASPM_en); +RTL_W8 (Config2,RTL_R8 (Config2) | ClkReqEn);rtl_eri_write (tp,0xc0 ,ERIAR_MASK_0011 ,0x0000 ,ERIAR_EXGMAC );rtl_eri_write (tp,0xb8 ,ERIAR_MASK_0011 ,0x0000 ,ERIAR_EXGMAC );
RE: r8168 is needed to enter P-state: Package State 6(pc6)onHaswellhardware
Francois Romieu [mailto:[email protected]]
Quoted 1 line
Sent: Tuesday, October 07, 2014 6:13 AM
[...]
Quoted 3 lines
Realtek's r8168 driver defaults to
CONFIG_ASPM=1but I guess some users
need to disable it and there's no known pattern / blacklist, right ?
When enabling the ASPM, it would influence the thrpughput. It is hard to choose performance or power saving. Therefore, we reserve the config to let the user determines it.
Best Regards,
Hayes
Re: r8168 is needed to enter P-state: Package State 6(pc6)onHaswellhardware
Hayes Wang <[email protected]> :
[...]
Quoted 3 lines
When enabling the ASPM, it would influence the thrpughput. It is hard to choose performance or power saving. Therefore, we reserve the config to let the user determines it.
Mmmm...
How do Realtek's devices behave if Config{2, 3} + MISC registers are configured for ASPM whereas PCI config space registers aren't ?
Signature · 1 line
Ueimor
RE: r8168 is needed to enter P-state: Package State6(pc6)onHaswellhardware
Francois Romieu [mailto:[email protected]]
Quoted 1 line
Sent: Wednesday, October 08, 2014 4:17 AM
[...]
Quoted 9 lines
Quoted 4 lines
When enabling the ASPM, it would influence the thrpughput. It is hard to
choose performance or power saving. Therefore, we reserve the config to let the user determines it.Mmmm...
How do Realtek's devices behave if Config{2, 3} + MISC registers are configured for ASPM whereas PCI config space registers aren't ?
The feature of the ASPM for the nic would be disabled. ASPM is enabled when both the internal settings and the PCI config space registers are enabled.
Best Regards,
Hayes
Re: r8168 is needed to enter P-state: Package State6(pc6)onHaswellhardware
Hayes Wang schreef op 08-10-14 om 04:35:
Quoted 8 lines
Francois Romieu [mailto:[email protected]]
Quoted 1 line
Sent: Wednesday, October 08, 2014 4:17 AM
[...]
Quoted 5 lines
Quoted 4 lines
When enabling the ASPM, it would influence the thrpughput. It is hard to
choose performance or power saving. Therefore, we reserve the config to let the user determines it.
Quoted 8 lines
Quoted 4 lines
Mmmm...
How do Realtek's devices behave if Config{2, 3} + MISC registers are configured for ASPM whereas PCI config space registers aren't ?
The feature of the ASPM for the nic would be disabled. ASPM is enabled when both the internal settings and the PCI config space registers are enabled.
Have in mind a situation where the PCI config space registers are set manually (after user login).
At which point in time will be determined whether ASPM for the nic will be disabled or enabled? Once, when loading the kernel module? Or more often?
Re: r8168 is needed to enter P-state: Package State 6 (pc6)onHaswell hardware
In reply to Francois RomieuFrancois Romieu schreef op 07-10-14 om 00:13:
Quoted 6 lines
Hayes Wang <[email protected]> :
Quoted 1 line
Francois Romieu [mailto:[email protected]]
[...]
Quoted 3 lines
I don't sure if the following information is helpful. Besides, I remember the
rtl_init_one()would disable it.
Quoted 7 lines
Quoted 3 lines
Yes, I did not expect this stuff to stay in geostationary orbit for long :o/
Realtek's r8168 driver defaults to
CONFIG_ASPM=1
# modinfo r8168 suggests the opposite (ASPM is disabled by default): version: 8.039.00-NAPI parm: aspm:Enable ASPM. (int)
When ASPM would be enabled by default, one would need a boot parameter like:
parm: aspm:Disable ASPM. (int)
but I guess some users
Quoted 1 line
need to disable it and there's no known pattern / blacklist, right ?
I don't want to disable ASPM. In fact the r8168 module I am even running with poot params like:
# cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-3.17.0-999-generic root=/dev/sda1 ro biosdevname=0 intel_pstate=enable ipv6.disabled=1 debug ignore_loglevel panic=10 pcie_aspm.policy=powersave pcie_aspm=force r8168.aspm=1 r8168.eee_enable=1 oops=panic
Quoted 2 lines
Ceriel, does the patch below against current kernel make a difference ?
Francois, what do you mean with "current kernel", the latest Ubuntu mainline kernel or something different?
Quoted 26 lines
drivers/net/ethernet/realtek/r8169.c +4
-1@@ -468,6 +468,7 @@ enum rtl8168_registers {#define PWM_EN (1 << 22)#define RXDV_GATED_EN (1 << 19)#define EARLY_TALLY_EN (1 << 16) +#define FORCE_CLK (1 << 15)/* force clock request */ };enum rtl_register_content {@@ -5279,8 +5280,10 @@ static void rtl_hw_start_8168g_1(struct rtl8169_private *tp)rtl_eri_write (tp,0x2f8 ,ERIAR_MASK_0011 ,0x1d8f ,ERIAR_EXGMAC );RTL_W8 (ChipCmd, CmdTxEnb | CmdRxEnb); -RTL_W32 (MISC ,RTL_R32 (MISC ) & ~RXDV_GATED_EN ); +RTL_W32 (MISC , (RTL_R32 (MISC ) |FORCE_CLK ) & ~RXDV_GATED_EN );RTL_W8 (MaxTxPacketSize, EarlySize); +RTL_W8 (Config5,RTL_R8 (Config5) | ASPM_en); +RTL_W8 (Config2,RTL_R8 (Config2) | ClkReqEn);rtl_eri_write (tp,0xc0 ,ERIAR_MASK_0011 ,0x0000 ,ERIAR_EXGMAC );rtl_eri_write (tp,0xb8 ,ERIAR_MASK_0011 ,0x0000 ,ERIAR_EXGMAC );
Re: r8168 is needed to enter P-state: Package State 6 (pc6)onHaswell hardware
Ceriel Jacobs <[email protected]> :
Quoted 1 line
Francois Romieu schreef op 07-10-14 om 00:13:
[...]
Quoted 5 lines
Quoted 1 line
Realtek's r8168 driver defaults to
CONFIG_ASPM=1# modinfo r8168 suggests the opposite (ASPM is disabled by default): version: 8.039.00-NAPI parm: aspm:Enable ASPM. (int)
$ grep CONFIG_ASPM src/Makefile CONFIG_ASPM = y
$ less src/r8168_n.c [...] #ifdef CONFIG_ASPM static int aspm = 1; #else static int aspm = 0; #endif
[...]
Quoted 2 lines
Francois, what do you mean with "current kernel", the latest Ubuntu mainline kernel or something different?
Sorry. I meant current mainline as seen from git but anything close like v3.15.x, v3.16.y or v3.17.k should be fine (patch applies).
Signature · 1 line
Ueimor
Re: r8168 is needed to enter P-state: Package State 6 (pc6)onHaswell hardware: does the patch below against current kernel make a difference?
In reply to Francois RomieuSorry for the delay. As newbie I first needed to learn about depmod to restore loading the r8169 module instead of r8168.
Now, I am struggling with applying the patch. Even with fuzz 99 applying this patch fails for hunk #2:
~/linux-3.13.0# patch -Np1 -F99 -r - --dry-run <<'END'
Quoted 1 line
index 0921302..b4a3881 100644
...
Quoted 2 lines
rtl_eri_write(tp, 0xb8, ERIAR_MASK_0011, 0x0000, ERIAR_EXGMAC);
END
checking file drivers/net/ethernet/realtek/r8169.c Hunk #1 succeeded at 468 with fuzz 3.
Hunk #2 FAILED at 5280.
1 out of 2 hunks FAILED
~/linux-3.13.0# grep -nA8 'rtl_eri_write(tp, 0x2f8, ERIAR_MASK_0011, 0x1d8f, ERIAR_EXGMAC)' drivers/net/ethernet/realtek/r8169.c
5274: rtl_eri_write(tp, 0x2f8, ERIAR_MASK_0011, 0x1d8f, ERIAR_EXGMAC);
5275-
5276- RTL_W8(ChipCmd, CmdTxEnb | CmdRxEnb);
5277- RTL_W32(MISC, RTL_R32(MISC) & ~RXDV_GATED_EN);
5278- RTL_W8(MaxTxPacketSize, EarlySize);
5279-
5280- rtl_eri_write(tp, 0xc0, ERIAR_MASK_0011, 0x0000, ERIAR_EXGMAC);
5281- rtl_eri_write(tp, 0xb8, ERIAR_MASK_0011, 0x0000, ERIAR_EXGMAC);
5282-
# ls -l $(locate drivers/net/ethernet/realtek/r8169.c) -rw-r--r-- 1 root root 177933 Aug 12 23:42 /home/user/src/linux-3.13.0/drivers/net/ethernet/realtek/r8169.c -rw-r--r-- 1 root root 177933 Aug 24 22:41 /root/linux-3.13.0/drivers/net/ethernet/realtek/r8169.c
# md5sum /root/linux-3.13.0/drivers/net/ethernet/realtek/r8169.c 95056b56932b375f8b65a6379009f704 /root/linux-3.13.0/drivers/net/ethernet/realtek/r8169.c
Francois, could you help me apply your patch?
Francois Romieu schreef op 07-10-14 om 00:13:
Quoted 42 lines
Hayes Wang <[email protected]> :
Quoted 1 line
Francois Romieu [mailto:[email protected]]
[...]
Quoted 6 lines
I don't sure if the following information is helpful. Besides, I remember the
rtl_init_one()would disable it.Yes, I did not expect this stuff to stay in geostationary orbit for long :o/
Realtek's r8168 driver defaults to
CONFIG_ASPM=1but I guess some users need to disable it and there's no known pattern / blacklist, right ?Ceriel, does the patch below against current kernel make a difference ?
drivers/net/ethernet/realtek/r8169.c +4
-1@@ -468,6 +468,7 @@ enum rtl8168_registers {#define PWM_EN (1 << 22)#define RXDV_GATED_EN (1 << 19)#define EARLY_TALLY_EN (1 << 16) +#define FORCE_CLK (1 << 15)/* force clock request */ };enum rtl_register_content {@@ -5279,8 +5280,10 @@ static void rtl_hw_start_8168g_1(struct rtl8169_private *tp)rtl_eri_write (tp,0x2f8 ,ERIAR_MASK_0011 ,0x1d8f ,ERIAR_EXGMAC );RTL_W8 (ChipCmd, CmdTxEnb | CmdRxEnb); -RTL_W32 (MISC ,RTL_R32 (MISC ) & ~RXDV_GATED_EN ); +RTL_W32 (MISC , (RTL_R32 (MISC ) |FORCE_CLK ) & ~RXDV_GATED_EN );RTL_W8 (MaxTxPacketSize, EarlySize); +RTL_W8 (Config5,RTL_R8 (Config5) | ASPM_en); +RTL_W8 (Config2,RTL_R8 (Config2) | ClkReqEn);rtl_eri_write (tp,0xc0 ,ERIAR_MASK_0011 ,0x0000 ,ERIAR_EXGMAC );rtl_eri_write (tp,0xb8 ,ERIAR_MASK_0011 ,0x0000 ,ERIAR_EXGMAC );
Re: r8168 is needed to enter P-state: Package State 6 (pc6)onHaswell hardware: does the patch below against current kernel make a difference?
Ceriel Jacobs <[email protected]> :
[...]
Quoted 1 line
Francois, could you help me apply your patch?
$ git describe v3.13 $ patch -p1 --dry-run < /tmp/plop checking file drivers/net/ethernet/realtek/r8169.c Hunk #1 succeeded at 467 (offset -1 lines). Hunk #2 succeeded at 5275 (offset -5 lines).
$ git rev-list v3.16 -- drivers/net/ethernet/realtek/r8169.c | while read x; do echo $x:$(git cat-file -p $x:drivers/net/ethernet/realtek/r8169.c | md5sum); done | grep 95056b56932b375f8b65a6379009f704
-> oualou
Where does your 3.13 source tree come from ?
Signature · 1 line
Ueimor
Re: r8168 is needed to enter P-state: Package State 6 (pc6)onHaswell hardware: does the patch below against current kernel make a difference?
The 3.13 source was installed using "apt-get source linux-source-3.13.0"
The patch hunk#2 failing issue is something in the "heredoc" notation (me trying to apply the patch without storing the patch lines as temporary input file). I am now using the temp file without any issue.
The next newly blocking issue is this:
# make -C /lib/modules/$(uname -r)/build M=$(pwd) modules make: Entering directory `/usr/src/linux-headers-3.17.0-031700rc7-generic' CC [M] /root/linux-3.13.0/drivers/net/ethernet/realtek/8139cp.o CC [M] /root/linux-3.13.0/drivers/net/ethernet/realtek/8139too.o CC [M] /root/linux-3.13.0/drivers/net/ethernet/realtek/atp.o CC [M] /root/linux-3.13.0/drivers/net/ethernet/realtek/r8169.o /root/linux-3.13.0/drivers/net/ethernet/realtek/r8169.c: In function 'rtl_init_one': /root/linux-3.13.0/drivers/net/ethernet/realtek/r8169.c:7127:2: error: implicit declaration of function 'SET_ETHTOOL_OPS' [-Werror=implicit-function-declaration] SET_ETHTOOL_OPS(dev, &rtl8169_ethtool_ops); ^ cc1: some warnings being treated as errors make[1]: *** [/root/linux-3.13.0/drivers/net/ethernet/realtek/r8169.o] Error 1 make: *** [_module_/root/linux-3.13.0/drivers/net/ethernet/realtek] Error 2 make: Leaving directory `/usr/src/linux-headers-3.17.0-031700rc7-generic'
Any new suggestion how to honour your patch/test request?
Francois Romieu schreef op 09-10-14 om 00:17:
Quoted 17 lines
Ceriel Jacobs <[email protected]> :
[...]Quoted 1 line
Francois, could you help me apply your patch?
$ git describe v3.13 $ patch -p1 --dry-run < /tmp/plop checking file drivers/net/ethernet/realtek/r8169.c Hunk #1 succeeded at 467 (offset -1 lines). Hunk #2 succeeded at 5275 (offset -5 lines).$ git rev-list v3.16 -- drivers/net/ethernet/realtek/r8169.c | while read x; do echo $x:$(git cat-file -p $x:drivers/net/ethernet/realtek/r8169.c | md5sum); done | grep 95056b56932b375f8b65a6379009f704-> oualou
Where does your 3.13 source tree come from ?
Re: r8168 is needed to enter P-state: Package State 6 (pc6)onHaswell hardware: does the patch below against current kernel make a difference?
Ceriel Jacobs <[email protected]> :
Quoted 10 lines
The 3.13 source was installed using "apt-get source linux-source-3.13.0"
The patch hunk#2 failing issue is something in the "heredoc" notation (me trying to apply the patch without storing the patch lines as temporary input file). I am now using the temp file without any issue.
The next newly blocking issue is this:
# make -C /lib/modules/$(uname -r)/build M=$(pwd) modules make: Entering directory `/usr/src/linux-headers-3.17.0-031700rc7-generic'
You are mixing 3.13 and 3.17. Try 'uname -r' alone. Got it ?
Please get some pristine 3.17 sources - they're available on www.kernel.org - and try again.
Signature · 1 line
Ueimor
Re: r8168 is needed to enter P-state: Package State 6 (pc6)onHaswell hardware: does the patch below against current kernel make a difference?
Francois Romieu schreef op 09-10-14 om 01:26:
Quoted 1 line
You are mixing 3.13 and 3.17. Try 'uname -r' alone. Got it ?
I think I have got it.
Francois Romieu schreef op 07-10-14 om 00:13:
Quoted 1 line
Ceriel, does the patch below against current kernel make a difference?
Old r8169 "powertop --auto-tune && powertop" result:
C2 (pc2) 26.1% | | C3 (pc3) 73.6% | C3 (cc3) 1.6% | C3-HSW 1.6% 108.0 ms C6 (pc6) 0.0% | C6 (cc6) 98.3% | C6-HSW 98.3% 194.9 ms --- C2 (pc2) 0.0% | | C3 (pc3) 99.4% | C3 (cc3) 0.0% | C3-HSW 0.0% 0.0 ms C6 (pc6) 0.0% | C6 (cc6) 99.6% | C6-HSW 99.6% 237.4 ms
---
New r8169 "powertop" result (even without --auto-tune):
C2 (pc2) 0.0% | | C3 (pc3) 9.9% | C3 (cc3) 0.7% | C3-HSW 0.7% 16.4 ms C6 (pc6) 89.9% | C6 (cc6) 99.2% | C6-HSW 99.2% 223.2 ms
---
Re: r8168 is needed to enter P-state: Package State 6 (pc6)onHaswell hardware: does the patch below against current kernel make a difference?
Ceriel Jacobs <[email protected]> :
Quoted 2 lines
Francois Romieu schreef op 07-10-14 om 00:13:
Quoted 1 line
Ceriel, does the patch below against current kernel make a difference?
[...]
Quoted 5 lines
New r8169 "powertop" result (even without --auto-tune):
C2 (pc2) 0.0% | | C3 (pc3) 9.9% | C3 (cc3) 0.7% | C3-HSW 0.7% 16.4 ms C6 (pc6) 89.9% | C6 (cc6) 99.2% | C6-HSW 99.2% 223.2 ms---
Fine (almost: I hope that ASPM was enabled from bios or during boot behind your back).
Remember your "lspci -nnkvv -s 03:00.0" ?
03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 11) [...]
Capabilities: [70] Express (v2) Endpoint, MSI 01
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
ClockPM+ Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
It should now look like:
[...]
LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
Let's temporarily disable it and see if powertop notices a difference.
<full disclosure>
"Capabilities: [70]" above gives you the offset of the relevant registers:
# lspci -xxx -s 03:00.0 [...] 70: .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ^^ -> "[70]"
You are interested in the Link Control register, aka PCI_EXP_LNKCTL in /usr/include/pci/header.h (devel part of pciutils) or kernel's include/uapi/linux/pci_regs.h. It's 16 bytes further, thus [...]
70: .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
80: 42 .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
^^
0x42 matches "LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+ ExtSynch-" built from above. There may be differences but the 3 lower weight binary digits in 0x42 encode ASPM control (0=nada, 1=L0, 2=L1, see PCI_EXP_LNKCTL_ASPxyz in include/uapi/linux/pci_regs.h). Mask these out (0x42 & ~0x03) and feed the resulting value back into the Link Control register:
# setpci -s 03:00.0 CAP_EXP+10.b=0x40
(CAP_EXP is pciutils's alias for the PCI Express capability block, see PCI_CAP_ID_EXP in kernel's include/uapi/linux/pci_regs.h)
If you are not too sure about the 0x40 value, you can retrieve it with lspci and an unpatched r8169 driver.
</full disclosure>
If I have understood Hayes correctly and he got my question right, lspci should now tell that ASPM is disabled. C6 should not be reached anymore.
ASPM could thus be enabled unconditionally at the driver level, then controled through the PCI config registers. Kernel r8169 driver would thus protect polar bears as Realtek's own r8168 driver already does.
I can't exclude that it will fail miserably in a firework of smelly smoke though.
Signature · 1 line
Ueimor
Re: r8168 is needed to enter P-state: Package State 6 (pc6)onHaswell hardware: does the patch below against current kernel make a difference? Yes, it does
ASPM is enabled in the BIOS as far as possible.
ASPM is also enabled using kernel parameters:
1. pcie_aspm.policy=powersave
2. pcie_aspm=force
Despite the result for 03:00.0 (and 2 other PCIe devices) is:
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
I have filed a bug report for that at:
https://bugzilla.kernel.org/show_bug.cgi?id=85621
In my testing before, I did manually enable L0s L1 ASPM after login prompt using:
# setpci -s 03:00.0 0x80.B=0x43 (also for 3 other ASPM PCIe devices)
=============================
Now back to your Realtek test request
=============================
For your information: I am running all testing remote via SSH over the Realtek NIc.
=======
L0s and L1 = PC6
=======
# setpci -s 03:00.0 0x80.B=0x43 # lspci -vvvv -s 03:00.0 | grep LnkCtl: LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- ...
PC6 reached in 1 powertop screen update (5 second interval)
C3 (pc3) 21.0% | C3 (cc3) 0.0% | C3-HSW 0.0% 0.0 ms C6 (pc6) 76.0% | C6 (cc6) 99.9% | C6-HSW 99.9% 198.1 ms
=======
L1 only = no PC6
=======
# setpci -s 03:00.0 0x80.B=0x42 LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
C3 (pc3) 99.8% | C3 (cc3) 0.0% | C3-HSW 0.0% 0.0 ms C6 (pc6) 0.0% | C6 (cc6) 99.9% | C6-HSW 100.0% 206.4 ms
=======
L0s only = PC6 !!
=======
# setpci -s 03:00.0 0x80.B=0x41 LnkCtl: ASPM L0s Enabled; RCB 64 bytes Disabled- ...
C3 (pc3) 10.6% | C3 (cc3) 0.0% | C3-HSW 0.0% 0.0 ms C6 (pc6) 89.1% | C6 (cc6) 99.9% | C6-HSW 99.9% 210.7 ms
=======
Disabled = no PC6
=======
# setpci -s 03:00.0 0x80.B=0x40 LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
C3 (pc3) 99.8% | C3 (cc3) 0.0% | C3-HSW 0.0% 0.0 ms C6 (pc6) 0.0% | C6 (cc6) 99.9% | C6-HSW 99.9% 230.1 ms
Now I am wondering whether it is by design or a bug that PC6 is not reached when only L1 ASPM is activated
An applause for:
1. saving "polar bears",
2. letting me test a little more,
3. realtime behavior changing (no module unloading and loading to active ASPM, just set the config space registers).
Francois Romieu schreef op 10-10-14 om 00:14:
Quoted 78 lines
Ceriel Jacobs <[email protected]> :
Quoted 2 lines
Francois Romieu schreef op 07-10-14 om 00:13:
Quoted 1 line
Ceriel, does the patch below against current kernel make a difference?
[...]
Quoted 5 lines
New r8169 "powertop" result (even without --auto-tune):
C2 (pc2) 0.0% | | C3 (pc3) 9.9% | C3 (cc3) 0.7% | C3-HSW 0.7% 16.4 ms C6 (pc6) 89.9% | C6 (cc6) 99.2% | C6-HSW 99.2% 223.2 ms---
Fine (almost: I hope that ASPM was enabled from bios or during boot behind your back).
Remember your "lspci -nnkvv -s 03:00.0" ?
03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 11) [...]
Capabilities: [70] Express (v2) Endpoint, MSI 01
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
ClockPM+ Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-It should now look like:
[...]
LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+Let's temporarily disable it and see if powertop notices a difference.
<full disclosure>
"Capabilities: [70]" above gives you the offset of the relevant registers:
# lspci -xxx -s 03:00.0 [...] 70: .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ^^ -> "[70]"You are interested in the Link Control register, aka
PCI_EXP_LNKCTLin/usr/include/pci/header.h(devel part of pciutils) or kernel'sinclude/uapi/linux/pci_regs.h. It's 16 bytes further, thus [...]
70: .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
80: 42 .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..^^
0x42matches "LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+ ExtSynch-" built from above. There may be differences but the 3 lower weight binary digits in0x42encode ASPM control (0=nada, 1=L0, 2=L1, seePCI_EXP_LNKCTL_ASPxyzininclude/uapi/linux/pci_regs.h). Mask these out (0x42& ~0x03) and feed the resulting value back into the Link Control register:# setpci -s 03:00.0 CAP_EXP+10.b=0x40(
CAP_EXPis pciutils's alias for the PCI Express capability block, seePCI_CAP_ID_EXPin kernel'sinclude/uapi/linux/pci_regs.h)If you are not too sure about the
0x40value, you can retrieve it with lspci and an unpatched r8169 driver.</full disclosure>
If I have understood Hayes correctly and he got my question right, lspci should now tell that ASPM is disabled. C6 should not be reached anymore.
ASPM could thus be enabled unconditionally at the driver level, then controled through the PCI config registers. Kernel r8169 driver would thus protect polar bears as Realtek's own r8168 driver already does.
I can't exclude that it will fail miserably in a firework of smelly smoke though.