Skip to contentSkip to conversation

Implement full clockgating for Kepler1 and 2

RFC Superseded Lyude Paul 4 participants 9 messages Jan 15, 2018 – Jan 21, 2018 2 messages not in the archive

Subsystems (3)

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Lyude Paul

author maintainer
#0 lore raw

cover Implement full clockgating for Kepler1 and 2

It's here! After a lot of investigation, rewrites, and traces, I present the patch series to implement all known levels of clockgating for Kepler1 and Kepler2 GPUs.

Starting with Fermi GPUs (this is probably present on earlier GPUs as well, but with a far less easy to manage interface), nvidia added two clockgating levels that are handled mostly in firmware (with the exception of course, of the driver initially programming all of the register values containing engine delays and that stuff):
- CG_CTRL - Main register for enabling/disabling clockgating for engines and hw blocks
- BLCG - "Block-level clockgating", a deeper level of clockgating
Starting with kepler2 as well, nvidia also introduced:
- SLCG - "??? clockgating" even deeper level of clockgating

Originally this patchset was going to include work for making this work on Fermi, however on closer investigation it seems that Fermi has one pretty big difference in it's requirements that we don't entirely understand yet. On Fermi the CG_CTRL register for the gr needs to be adjusted during reclocking, while kepler and later generations need no such adjustments. Since this requires more research; the current plan is to leave fermi out of this patchset, and then just rework the code at a later point in time to support Fermi once we understand more.

For the time being however, we put as much code we're sure Fermi will be sharing into gf100 files for the future.

For the time being; this patchset supports every GPU in the kepler1 and kepler2 family. The main kernel config option to enable this is
config=NvPmEnableGating=<level>
Where <level> is the deepest level of powersaving to enable. The levels being:
0. NOCG
1. CG (just CG_CTRL)
2. BLCG
3. SLCG

Additionally, we leave a couple of small TODO comments to mark spots we'll eventually need to add code for the final level of power saving we have yet to implement or understand: ELPG (engine-level powergating).

Here's some very lazily done benchmarks on how much power this saves as well. It should be noted that chances are, these patches save a lot more power then you see here since clockgating works best when the GPU is under load. I have a feeling this will be especially true with SLCG.

GK104 stats:
Idle (1 4k display @ fbcon):

    NOCG:
      07: 20.53W
      0a: 44.12W
      0e: 61.23W
      0f: 62.23W
    CG:
      07: 18.95W-19.02W
      0a: 34.95W-37.48W
      0e: 52.70W-54.32W
      0f: 53.53W-55.68W
    BLCG:
      07: 18.64W-19.02W
      0a: 32.68W-34.78W
      0e: 49.14W-50.51W
      0f: 49.75W-51.73W
GK110 stats:
  Idle (1 4k display @ fbcon):
    NOCG:
      07: 25.68W
      0a: 36.16W
      0d: 64.71W
      0f: 64.99W
    CG:
      07: 25.68W
      0a: 35.58W
      0d: 61.54W-61.74W
      0f: 61.93W-62.12W
    BLCG:
      07: 25W-25.2W
      0a: 33.85W-34.04W
      0d: 59.25W-59.44W
      0f: 59.35W-59.54W
    SLCG:
      07: 25W-25.2W
      0a: 33.85W-33.95W (spikes to 34.04W briefly every now and then)
      0d: 59.15W-59.35W
      0f: 59.35W-59.54W

For the time being, this will be disabled by default in the kernel as we wait to get more widespread testing with this patchset. If you're willing to put up with the potential of instability, please feel free to try this patchset and let us know how well it works for you!

Lyude Paul (4):
drm/nouveau: Add support for basic clockgating on Kepler1 drm/nouveau: Add support for BLCG clockgating for Kepler1 drm/nouveau: Add BLCG clockgating for Kepler2 drm/nouveau: Add SLCG clockgating for Kepler2

 drivers/gpu/drm/nouveau/include/nvkm/subdev/fb.h   |   1 +
 .../gpu/drm/nouveau/include/nvkm/subdev/therm.h    |  25 +++
 drivers/gpu/drm/nouveau/nvkm/engine/device/base.c  |  25 +--
 drivers/gpu/drm/nouveau/nvkm/engine/gr/gf100.h     |   1 +
 drivers/gpu/drm/nouveau/nvkm/engine/gr/gk104.c     | 215 ++++++++++++++++++++-
 drivers/gpu/drm/nouveau/nvkm/engine/gr/gk104.h     |  55 ++++++
 drivers/gpu/drm/nouveau/nvkm/engine/gr/gk110.c     | 176 +++++++++++++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/Kbuild      |   1 +
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gf100.c     |   6 +
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk104.c     |  53 +++++
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk104.h     |  35 ++++
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk110.c     |  77 ++++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/priv.h      |   3 +
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/Kbuild   |   2 +
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/base.c   |  84 +++++++-
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf100.c  |  77 ++++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf100.h  |  35 ++++
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf119.c  |   8 +-
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.c  | 136 +++++++++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.h  |  56 ++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gt215.c  |   2 +-
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/priv.h   |  23 ++-
 22 files changed, 1069 insertions(+), 27 deletions(-)
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/engine/gr/gk104.h
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk104.h
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk110.c
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf100.c
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf100.h
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.c
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.h
Signature · 6 lines
2.14.3

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
author maintainer
#1 lore raw

RFC 1/4 drm/nouveau: Add support for basic clockgating on Kepler1

This adds support for enabling automatic clockgating on nvidia GPUs for Kepler1, referred to as "CG" throughout the driver. This is one of two powersaving levels that Kepler1 supports.

This introduces two therm helpers for controlling basic clockgating:
nvkm_therm_clkgate_enable() - enables clockgating through CG_CTRL, done after initializing the GPU fully nvkm_therm_clkgate_fini() - prepares clockgating for suspend or driver unload

As well, we add the nouveau kernel config parameter NvPmEnableGating, which can be set to the highest level of clockgating (in this case, we only have CG) the user desires to enable. Since we've only had limited testing on this thus far, we disable this by default.

A lot of this code was originally going to be based off of fermi; however it turns out that while Fermi's the first line of GPUs that introduced this kind of power saving, Fermi requires more fine tuned control of the CG_CTRL registers from the driver while reclocking that we don't entirely understand yet.

For the simple parts we will be sharing with Fermi for certain however, we at least add those into a new subdev/therm/gf100.h header.

---
 .../gpu/drm/nouveau/include/nvkm/subdev/therm.h    |  10 ++
 drivers/gpu/drm/nouveau/nvkm/engine/device/base.c  |  17 +--
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/Kbuild   |   1 +
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/base.c   |  72 +++++++++--
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf100.h  |  35 ++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf119.c  |   8 +-
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.c  | 135 +++++++++++++++++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.h  |  56 +++++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/priv.h   |  15 ++-
 9 files changed, 328 insertions(+), 21 deletions(-)
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf100.h
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.c
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.h
drivers/gpu/drm/nouveau/include/nvkm/subdev/therm.h +10 -0
index b1ac47eb786e..a9204c09975b 100644
@@ -46,6 +46,11 @@ enum nvkm_therm_attr_type {
 	NVKM_THERM_ATTR_THRS_SHUTDOWN_HYST = 17,
 };
 
+enum nvkm_therm_clkgate_level {
+	NVKM_THERM_CLKGATE_NONE = 0,
+	NVKM_THERM_CLKGATE_CG, /* basic clockgating */
+};
+
 struct nvkm_therm {
 	const struct nvkm_therm_func *func;
 	struct nvkm_subdev subdev;
@@ -85,17 +90,22 @@ struct nvkm_therm {
 
 	int (*attr_get)(struct nvkm_therm *, enum nvkm_therm_attr_type);
 	int (*attr_set)(struct nvkm_therm *, enum nvkm_therm_attr_type, int);
+
+	enum nvkm_therm_clkgate_level clkgate_level;
 };
 
 int nvkm_therm_temp_get(struct nvkm_therm *);
 int nvkm_therm_fan_sense(struct nvkm_therm *);
 int nvkm_therm_cstate(struct nvkm_therm *, int, int);
+void nvkm_therm_clkgate_enable(struct nvkm_therm *);
+void nvkm_therm_clkgate_fini(struct nvkm_therm *, bool);
 
 int nv40_therm_new(struct nvkm_device *, int, struct nvkm_therm **);
 int nv50_therm_new(struct nvkm_device *, int, struct nvkm_therm **);
 int g84_therm_new(struct nvkm_device *, int, struct nvkm_therm **);
 int gt215_therm_new(struct nvkm_device *, int, struct nvkm_therm **);
 int gf119_therm_new(struct nvkm_device *, int, struct nvkm_therm **);
+int gk104_therm_new(struct nvkm_device *, int, struct nvkm_therm **);
 int gm107_therm_new(struct nvkm_device *, int, struct nvkm_therm **);
 int gm200_therm_new(struct nvkm_device *, int, struct nvkm_therm **);
 int gp100_therm_new(struct nvkm_device *, int, struct nvkm_therm **);
drivers/gpu/drm/nouveau/nvkm/engine/device/base.c +10 -7
index 00eeaaffeae5..6c5f966c66ad 100644
@@ -28,6 +28,7 @@
 #include <core/option.h>
 
 #include <subdev/bios.h>
+#include <subdev/therm.h>
 
 static DEFINE_MUTEX(nv_devices_mutex);
 static LIST_HEAD(nv_devices);
@@ -1682,7 +1683,7 @@ nve4_chipset = {
 	.mxm = nv50_mxm_new,
 	.pci = gk104_pci_new,
 	.pmu = gk104_pmu_new,
-	.therm = gf119_therm_new,
+	.therm = gk104_therm_new,
 	.timer = nv41_timer_new,
 	.top = gk104_top_new,
 	.volt = gk104_volt_new,
@@ -1721,7 +1722,7 @@ nve6_chipset = {
 	.mxm = nv50_mxm_new,
 	.pci = gk104_pci_new,
 	.pmu = gk104_pmu_new,
-	.therm = gf119_therm_new,
+	.therm = gk104_therm_new,
 	.timer = nv41_timer_new,
 	.top = gk104_top_new,
 	.volt = gk104_volt_new,
@@ -1760,7 +1761,7 @@ nve7_chipset = {
 	.mxm = nv50_mxm_new,
 	.pci = gk104_pci_new,
 	.pmu = gk104_pmu_new,
-	.therm = gf119_therm_new,
+	.therm = gk104_therm_new,
 	.timer = nv41_timer_new,
 	.top = gk104_top_new,
 	.volt = gk104_volt_new,
@@ -1824,7 +1825,7 @@ nvf0_chipset = {
 	.mxm = nv50_mxm_new,
 	.pci = gk104_pci_new,
 	.pmu = gk110_pmu_new,
-	.therm = gf119_therm_new,
+	.therm = gk104_therm_new,
 	.timer = nv41_timer_new,
 	.top = gk104_top_new,
 	.volt = gk104_volt_new,
@@ -1862,7 +1863,7 @@ nvf1_chipset = {
 	.mxm = nv50_mxm_new,
 	.pci = gk104_pci_new,
 	.pmu = gk110_pmu_new,
-	.therm = gf119_therm_new,
+	.therm = gk104_therm_new,
 	.timer = nv41_timer_new,
 	.top = gk104_top_new,
 	.volt = gk104_volt_new,
@@ -1900,7 +1901,7 @@ nv106_chipset = {
 	.mxm = nv50_mxm_new,
 	.pci = gk104_pci_new,
 	.pmu = gk208_pmu_new,
-	.therm = gf119_therm_new,
+	.therm = gk104_therm_new,
 	.timer = nv41_timer_new,
 	.top = gk104_top_new,
 	.volt = gk104_volt_new,
@@ -1938,7 +1939,7 @@ nv108_chipset = {
 	.mxm = nv50_mxm_new,
 	.pci = gk104_pci_new,
 	.pmu = gk208_pmu_new,
-	.therm = gf119_therm_new,
+	.therm = gk104_therm_new,
 	.timer = nv41_timer_new,
 	.top = gk104_top_new,
 	.volt = gk104_volt_new,
@@ -2508,6 +2509,7 @@ nvkm_device_fini(struct nvkm_device *device, bool suspend)
 		}
 	}
 
+	nvkm_therm_clkgate_fini(device->therm, suspend);
 
 	if (device->func->fini)
 		device->func->fini(device, suspend);
@@ -2597,6 +2599,7 @@ nvkm_device_init(struct nvkm_device *device)
 	}
 
 	nvkm_acpi_init(device);
+	nvkm_therm_clkgate_enable(device->therm);
 
 	time = ktime_to_us(ktime_get()) - time;
 	nvdev_trace(device, "init completed in %lldus\n", time);
drivers/gpu/drm/nouveau/nvkm/subdev/therm/Kbuild +1 -0
index 7ba56b12badd..4bac4772d8ed 100644
@@ -10,6 +10,7 @@ nvkm-y += nvkm/subdev/therm/nv50.o
 nvkm-y += nvkm/subdev/therm/g84.o
 nvkm-y += nvkm/subdev/therm/gt215.o
 nvkm-y += nvkm/subdev/therm/gf119.o
+nvkm-y += nvkm/subdev/therm/gk104.o
 nvkm-y += nvkm/subdev/therm/gm107.o
 nvkm-y += nvkm/subdev/therm/gm200.o
 nvkm-y += nvkm/subdev/therm/gp100.o
drivers/gpu/drm/nouveau/nvkm/subdev/therm/base.c +64 -8

102 lines not on this page · raw message

drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf100.h new file +35 -0
new file mode 100644
index 000000000000..cfb25af77c60
@@ -0,0 +1,35 @@
+/*
+ * Copyright 2018 Red Hat Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Authors: Lyude Paul
+ */
+
+#ifndef __GF100_THERM_H__
+#define __GF100_THERM_H__
+
+#include <core/device.h>
+
+struct gf100_idle_filter {
+	u32 fecs;
+	u32 hubmmu;
+};
+
+#endif
drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf119.c +4 -4
index 06dcfd6ee966..0981b02790e2 100644
@@ -49,7 +49,7 @@ pwm_info(struct nvkm_therm *therm, int line)
 	return -ENODEV;
 }
 
-static int
+int
 gf119_fan_pwm_ctrl(struct nvkm_therm *therm, int line, bool enable)
 {
 	struct nvkm_device *device = therm->subdev.device;
@@ -63,7 +63,7 @@ gf119_fan_pwm_ctrl(struct nvkm_therm *therm, int line, bool enable)
 	return 0;
 }
 
-static int
+int
 gf119_fan_pwm_get(struct nvkm_therm *therm, int line, u32 *divs, u32 *duty)
 {
 	struct nvkm_device *device = therm->subdev.device;
@@ -85,7 +85,7 @@ gf119_fan_pwm_get(struct nvkm_therm *therm, int line, u32 *divs, u32 *duty)
 	return -EINVAL;
 }
 
-static int
+int
 gf119_fan_pwm_set(struct nvkm_therm *therm, int line, u32 divs, u32 duty)
 {
 	struct nvkm_device *device = therm->subdev.device;
@@ -102,7 +102,7 @@ gf119_fan_pwm_set(struct nvkm_therm *therm, int line, u32 divs, u32 duty)
 	return 0;
 }
 
-static int
+int
 gf119_fan_pwm_clock(struct nvkm_therm *therm, int line)
 {
 	struct nvkm_device *device = therm->subdev.device;
drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.c new file +135 -0

135 lines not on this page · raw message

drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.h new file +56 -0

56 lines not on this page · raw message

drivers/gpu/drm/nouveau/nvkm/subdev/therm/priv.h +13 -2
index 1f46e371d7c4..f30202dd88e7 100644
@@ -32,6 +32,8 @@
 
 int nvkm_therm_new_(const struct nvkm_therm_func *, struct nvkm_device *,
 		    int index, struct nvkm_therm **);
+void nvkm_therm_ctor(struct nvkm_therm *therm, struct nvkm_device *device,
+		     int index, const struct nvkm_therm_func *func);
 
 struct nvkm_fan {
 	struct nvkm_therm *parent;
@@ -66,8 +68,6 @@ int nvkm_therm_fan_set(struct nvkm_therm *, bool now, int percent);
 int nvkm_therm_fan_user_get(struct nvkm_therm *);
 int nvkm_therm_fan_user_set(struct nvkm_therm *, int percent);
 
-int nvkm_therm_preinit(struct nvkm_therm *);
-
 int  nvkm_therm_sensor_init(struct nvkm_therm *);
 int  nvkm_therm_sensor_fini(struct nvkm_therm *, bool suspend);
 void nvkm_therm_sensor_preinit(struct nvkm_therm *);
@@ -96,6 +96,9 @@ struct nvkm_therm_func {
 	int (*fan_sense)(struct nvkm_therm *);
 
 	void (*program_alarms)(struct nvkm_therm *);
+
+	void (*clkgate_enable)(struct nvkm_therm *);
+	void (*clkgate_fini)(struct nvkm_therm *, bool);
 };
 
 void nv40_therm_intr(struct nvkm_therm *);
@@ -112,8 +115,16 @@ void g84_therm_fini(struct nvkm_therm *);
 int gt215_therm_fan_sense(struct nvkm_therm *);
 
 void g84_therm_init(struct nvkm_therm *);
+
+int gf119_fan_pwm_ctrl(struct nvkm_therm *, int, bool);
+int gf119_fan_pwm_get(struct nvkm_therm *, int, u32 *, u32 *);
+int gf119_fan_pwm_set(struct nvkm_therm *, int, u32, u32);
+int gf119_fan_pwm_clock(struct nvkm_therm *, int);
 void gf119_therm_init(struct nvkm_therm *);
 
+void gk104_clkgate_enable(struct nvkm_therm *);
+void gk104_clkgate_fini(struct nvkm_therm *, bool);
+
 int nvkm_fanpwm_create(struct nvkm_therm *, struct dcb_gpio_func *);
 int nvkm_fantog_create(struct nvkm_therm *, struct dcb_gpio_func *);
 int nvkm_fannil_create(struct nvkm_therm *);
Signature · 6 lines
2.14.3

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
#2 lore raw

Re: [RFC 1/4] drm/nouveau: Add support for basic clockgating on Kepler1

Quoted 3 lines · Lyude Paul

This adds support for enabling automatic clockgating on nvidia GPUs for Kepler1, referred to as "CG" throughout the driver. This is one of two powersaving levels that Kepler1 supports.

Thanks a lot for all this work! It was long overdue and it is nice to see the project finally getting to an end, after passing into so many hands!

Quoted 11 lines

This introduces two therm helpers for controlling basic clockgating:
nvkm_therm_clkgate_enable() - enables clockgating through CG_CTRL, done after initializing the GPU fully nvkm_therm_clkgate_fini() - prepares clockgating for suspend or driver unload

As well, we add the nouveau kernel config parameter NvPmEnableGating, which can be set to the highest level of clockgating (in this case, we only have CG) the user desires to enable. Since we've only had limited testing on this thus far, we disable this by default.

I am not sure I understand the purpose of this level here. As far as I understand, you only have per-engine control whether you want to enable CG or not. What you call BLCG and SLCG levels just mean "don't use the boot values, but rather use our values (taken from nvidia)".

Now, here comes the nasty part: NVIDIA only ever validated the boot values (I guess they are extremely safe ones), and the optimised values (the ones coming from your patch 2, 3, and 4 along with the level 3.

I think introducing a single parameter that controls both CG, PG, and automatic reclocking would be safer. For CG and PG, it would be a all-or-nothing (either boot values, or everything like nvidia).

Quoted 213 lines

A lot of this code was originally going to be based off of fermi; however it turns out that while Fermi's the first line of GPUs that introduced this kind of power saving, Fermi requires more fine tuned control of the CG_CTRL registers from the driver while reclocking that we don't entirely understand yet.

For the simple parts we will be sharing with Fermi for certain however, we at least add those into a new subdev/therm/gf100.h header.

---
 .../gpu/drm/nouveau/include/nvkm/subdev/therm.h    |  10 ++
 drivers/gpu/drm/nouveau/nvkm/engine/device/base.c  |  17 +--
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/Kbuild   |   1 +
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/base.c   |  72 +++++++++--
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf100.h  |  35 ++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf119.c  |   8 +-
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.c  | 135 +++++++++++++++++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.h  |  56 +++++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/priv.h   |  15 ++-
 9 files changed, 328 insertions(+), 21 deletions(-)
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf100.h
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.c
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.h
drivers/gpu/drm/nouveau/include/nvkm/subdev/therm.h +10 -0
index b1ac47eb786e..a9204c09975b 100644
@@ -46,6 +46,11 @@ enum nvkm_therm_attr_type {
 	NVKM_THERM_ATTR_THRS_SHUTDOWN_HYST = 17,
 };
 
+enum nvkm_therm_clkgate_level {
+	NVKM_THERM_CLKGATE_NONE = 0,
+	NVKM_THERM_CLKGATE_CG, /* basic clockgating */
+};
+
 struct nvkm_therm {
 	const struct nvkm_therm_func *func;
 	struct nvkm_subdev subdev;
@@ -85,17 +90,22 @@ struct nvkm_therm {
 
 	int (*attr_get)(struct nvkm_therm *, enum nvkm_therm_attr_type);
 	int (*attr_set)(struct nvkm_therm *, enum nvkm_therm_attr_type, int);
+
+	enum nvkm_therm_clkgate_level clkgate_level;
 };
 
 int nvkm_therm_temp_get(struct nvkm_therm *);
 int nvkm_therm_fan_sense(struct nvkm_therm *);
 int nvkm_therm_cstate(struct nvkm_therm *, int, int);
+void nvkm_therm_clkgate_enable(struct nvkm_therm *);
+void nvkm_therm_clkgate_fini(struct nvkm_therm *, bool);
 
 int nv40_therm_new(struct nvkm_device *, int, struct nvkm_therm **);
 int nv50_therm_new(struct nvkm_device *, int, struct nvkm_therm **);
 int g84_therm_new(struct nvkm_device *, int, struct nvkm_therm **);
 int gt215_therm_new(struct nvkm_device *, int, struct nvkm_therm **);
 int gf119_therm_new(struct nvkm_device *, int, struct nvkm_therm **);
+int gk104_therm_new(struct nvkm_device *, int, struct nvkm_therm **);
 int gm107_therm_new(struct nvkm_device *, int, struct nvkm_therm **);
 int gm200_therm_new(struct nvkm_device *, int, struct nvkm_therm **);
 int gp100_therm_new(struct nvkm_device *, int, struct nvkm_therm **);
drivers/gpu/drm/nouveau/nvkm/engine/device/base.c +10 -7
index 00eeaaffeae5..6c5f966c66ad 100644
@@ -28,6 +28,7 @@
 #include <core/option.h>
 
 #include <subdev/bios.h>
+#include <subdev/therm.h>
 
 static DEFINE_MUTEX(nv_devices_mutex);
 static LIST_HEAD(nv_devices);
@@ -1682,7 +1683,7 @@ nve4_chipset = {
 	.mxm = nv50_mxm_new,
 	.pci = gk104_pci_new,
 	.pmu = gk104_pmu_new,
-	.therm = gf119_therm_new,
+	.therm = gk104_therm_new,
 	.timer = nv41_timer_new,
 	.top = gk104_top_new,
 	.volt = gk104_volt_new,
@@ -1721,7 +1722,7 @@ nve6_chipset = {
 	.mxm = nv50_mxm_new,
 	.pci = gk104_pci_new,
 	.pmu = gk104_pmu_new,
-	.therm = gf119_therm_new,
+	.therm = gk104_therm_new,
 	.timer = nv41_timer_new,
 	.top = gk104_top_new,
 	.volt = gk104_volt_new,
@@ -1760,7 +1761,7 @@ nve7_chipset = {
 	.mxm = nv50_mxm_new,
 	.pci = gk104_pci_new,
 	.pmu = gk104_pmu_new,
-	.therm = gf119_therm_new,
+	.therm = gk104_therm_new,
 	.timer = nv41_timer_new,
 	.top = gk104_top_new,
 	.volt = gk104_volt_new,
@@ -1824,7 +1825,7 @@ nvf0_chipset = {
 	.mxm = nv50_mxm_new,
 	.pci = gk104_pci_new,
 	.pmu = gk110_pmu_new,
-	.therm = gf119_therm_new,
+	.therm = gk104_therm_new,
 	.timer = nv41_timer_new,
 	.top = gk104_top_new,
 	.volt = gk104_volt_new,
@@ -1862,7 +1863,7 @@ nvf1_chipset = {
 	.mxm = nv50_mxm_new,
 	.pci = gk104_pci_new,
 	.pmu = gk110_pmu_new,
-	.therm = gf119_therm_new,
+	.therm = gk104_therm_new,
 	.timer = nv41_timer_new,
 	.top = gk104_top_new,
 	.volt = gk104_volt_new,
@@ -1900,7 +1901,7 @@ nv106_chipset = {
 	.mxm = nv50_mxm_new,
 	.pci = gk104_pci_new,
 	.pmu = gk208_pmu_new,
-	.therm = gf119_therm_new,
+	.therm = gk104_therm_new,
 	.timer = nv41_timer_new,
 	.top = gk104_top_new,
 	.volt = gk104_volt_new,
@@ -1938,7 +1939,7 @@ nv108_chipset = {
 	.mxm = nv50_mxm_new,
 	.pci = gk104_pci_new,
 	.pmu = gk208_pmu_new,
-	.therm = gf119_therm_new,
+	.therm = gk104_therm_new,
 	.timer = nv41_timer_new,
 	.top = gk104_top_new,
 	.volt = gk104_volt_new,
@@ -2508,6 +2509,7 @@ nvkm_device_fini(struct nvkm_device *device, bool suspend)
 		}
 	}
 
+	nvkm_therm_clkgate_fini(device->therm, suspend);
 
 	if (device->func->fini)
 		device->func->fini(device, suspend);
@@ -2597,6 +2599,7 @@ nvkm_device_init(struct nvkm_device *device)
 	}
 
 	nvkm_acpi_init(device);
+	nvkm_therm_clkgate_enable(device->therm);
 
 	time = ktime_to_us(ktime_get()) - time;
 	nvdev_trace(device, "init completed in %lldus\n", time);
drivers/gpu/drm/nouveau/nvkm/subdev/therm/Kbuild +1 -0
index 7ba56b12badd..4bac4772d8ed 100644
@@ -10,6 +10,7 @@ nvkm-y += nvkm/subdev/therm/nv50.o
 nvkm-y += nvkm/subdev/therm/g84.o
 nvkm-y += nvkm/subdev/therm/gt215.o
 nvkm-y += nvkm/subdev/therm/gf119.o
+nvkm-y += nvkm/subdev/therm/gk104.o
 nvkm-y += nvkm/subdev/therm/gm107.o
 nvkm-y += nvkm/subdev/therm/gm200.o
 nvkm-y += nvkm/subdev/therm/gp100.o
drivers/gpu/drm/nouveau/nvkm/subdev/therm/base.c +30 -0
index f27fc6d0d4c6..ba1a3aabb559 100644
@@ -21,6 +21,7 @@
  *
  * Authors: Martin Peres
  */
+#include <nvkm/core/option.h>
 #include "priv.h"
 
 int
@@ -297,6 +298,47 @@ nvkm_therm_attr_set(struct nvkm_therm *therm,
 	return -EINVAL;
 }
 
+void
+nvkm_therm_clkgate_enable(struct nvkm_therm *therm)
+{
+	if (!therm->func->clkgate_enable || !therm->clkgate_level)
+		return;
+
+	nvkm_debug(&therm->subdev,
+		   "Enabling clock/powergating\n");
+	therm->func->clkgate_enable(therm);
+}
+
+void
+nvkm_therm_clkgate_fini(struct nvkm_therm *therm, bool suspend)
+{
+	if (!therm->func->clkgate_fini || !therm->clkgate_level)
+		return;
+
+	nvkm_debug(&therm->subdev,
+		   "Preparing clock/powergating for %s\n",
+		   suspend ? "suspend" : "fini");
+	therm->func->clkgate_fini(therm, suspend);
+}
+
+static void
+nvkm_therm_clkgate_oneinit(struct nvkm_therm *therm)
+{
+	const char *clkgate_str;
+
+	if (!therm->func->clkgate_enable || !therm->clkgate_level)

drivers/gpu/drm/nouveau/nvkm/subdev/therm/base.c nvkm_therm_attr_set(struct nvkm_therm *therm, · line 339

Quoted 10 lines
drivers/gpu/drm/nouveau/nvkm/subdev/therm/base.c +10 -0
+		return;
+
+	switch (therm->clkgate_level) {
+	case NVKM_THERM_CLKGATE_CG: clkgate_str = "CG"; break;
+	default: BUG();
+	}
+
+	nvkm_info(&therm->subdev,
+		  "Clock/powergating enabled up to level %d (%s)\n",
+		  therm->clkgate_level, clkgate_str);

This message is a bit odd, whether we keep the notion of levels or not.

Can you get rid of the mention of powergating given that this is not part of this patchset?

If you agree with having a single enable bit for CG, then a simple:
"Clockgating status: (boot|optimized)" would work perfectly.

Quoted 180 lines
drivers/gpu/drm/nouveau/nvkm/subdev/therm/base.c +24 -8
+}
+
 static void
 nvkm_therm_intr(struct nvkm_subdev *subdev)
 {
@@ -333,6 +375,7 @@ nvkm_therm_oneinit(struct nvkm_subdev *subdev)
 	nvkm_therm_fan_ctor(therm);
 	nvkm_therm_fan_mode(therm, NVKM_THERM_CTRL_AUTO);
 	nvkm_therm_sensor_preinit(therm);
+	nvkm_therm_clkgate_oneinit(therm);
 	return 0;
 }
 
@@ -374,15 +417,10 @@ nvkm_therm = {
 	.intr = nvkm_therm_intr,
 };
 
-int
-nvkm_therm_new_(const struct nvkm_therm_func *func, struct nvkm_device *device,
-		int index, struct nvkm_therm **ptherm)
+void
+nvkm_therm_ctor(struct nvkm_therm *therm, struct nvkm_device *device,
+		int index, const struct nvkm_therm_func *func)
 {
-	struct nvkm_therm *therm;
-
-	if (!(therm = *ptherm = kzalloc(sizeof(*therm), GFP_KERNEL)))
-		return -ENOMEM;
-
 	nvkm_subdev_ctor(&nvkm_therm, device, index, &therm->subdev);
 	therm->func = func;
 
@@ -395,5 +433,23 @@ nvkm_therm_new_(const struct nvkm_therm_func *func, struct nvkm_device *device,
 	therm->attr_get = nvkm_therm_attr_get;
 	therm->attr_set = nvkm_therm_attr_set;
 	therm->mode = therm->suspend = -1; /* undefined */
+
+	therm->clkgate_level =
+		clamp((int)nvkm_longopt(device->cfgopt,
+					"NvPmEnableGating",
+					NVKM_THERM_CLKGATE_NONE),
+		      NVKM_THERM_CLKGATE_NONE, NVKM_THERM_CLKGATE_CG);
+}
+
+int
+nvkm_therm_new_(const struct nvkm_therm_func *func, struct nvkm_device *device,
+		int index, struct nvkm_therm **ptherm)
+{
+	struct nvkm_therm *therm;
+
+	if (!(therm = *ptherm = kzalloc(sizeof(*therm), GFP_KERNEL)))
+		return -ENOMEM;
+
+	nvkm_therm_ctor(therm, device, index, func);
 	return 0;
 }
drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf100.h new file +35 -0
new file mode 100644
index 000000000000..cfb25af77c60
@@ -0,0 +1,35 @@
+/*
+ * Copyright 2018 Red Hat Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Authors: Lyude Paul
+ */
+
+#ifndef __GF100_THERM_H__
+#define __GF100_THERM_H__
+
+#include <core/device.h>
+
+struct gf100_idle_filter {
+	u32 fecs;
+	u32 hubmmu;
+};
+
+#endif
drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf119.c +4 -4
index 06dcfd6ee966..0981b02790e2 100644
@@ -49,7 +49,7 @@ pwm_info(struct nvkm_therm *therm, int line)
 	return -ENODEV;
 }
 
-static int
+int
 gf119_fan_pwm_ctrl(struct nvkm_therm *therm, int line, bool enable)
 {
 	struct nvkm_device *device = therm->subdev.device;
@@ -63,7 +63,7 @@ gf119_fan_pwm_ctrl(struct nvkm_therm *therm, int line, bool enable)
 	return 0;
 }
 
-static int
+int
 gf119_fan_pwm_get(struct nvkm_therm *therm, int line, u32 *divs, u32 *duty)
 {
 	struct nvkm_device *device = therm->subdev.device;
@@ -85,7 +85,7 @@ gf119_fan_pwm_get(struct nvkm_therm *therm, int line, u32 *divs, u32 *duty)
 	return -EINVAL;
 }
 
-static int
+int
 gf119_fan_pwm_set(struct nvkm_therm *therm, int line, u32 divs, u32 duty)
 {
 	struct nvkm_device *device = therm->subdev.device;
@@ -102,7 +102,7 @@ gf119_fan_pwm_set(struct nvkm_therm *therm, int line, u32 divs, u32 duty)
 	return 0;
 }
 
-static int
+int
 gf119_fan_pwm_clock(struct nvkm_therm *therm, int line)
 {
 	struct nvkm_device *device = therm->subdev.device;
drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.c new file +37 -0
new file mode 100644
index 000000000000..81bc73a6d143
@@ -0,0 +1,135 @@
+/*
+ * Copyright 2018 Red Hat Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Authors: Lyude Paul
+ */
+#include <core/device.h>
+
+#include "priv.h"
+#include "gk104.h"
+
+void
+gk104_clkgate_enable(struct nvkm_therm *base)
+{
+	struct gk104_therm *therm = gk104_therm(base);
+	struct nvkm_device *dev = therm->base.subdev.device;
+	const struct gk104_clkgate_engine_info *order = therm->clkgate_order;
+	int i;
+
+	/* Program ENG_MANT, ENG_FILTER */

drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.c line 47

Quoted 10 lines
drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.c new file +10 -0
+	for (i = 0; order[i].engine != NVKM_SUBDEV_NR; i++) {
+		if (!nvkm_device_subdev(dev, order[i].engine))
+			continue;
+
+		nvkm_mask(dev, 0x20200 + order[i].offset, 0xff00, 0x4500);
+	}
+
+	/* magic */
+	nvkm_wr32(dev, 0x020288, therm->idle_filter->fecs);
+	nvkm_wr32(dev, 0x02028c, therm->idle_filter->hubmmu);

All this time, I thought these parameters were for power gating... I also did not expect that clock gating had to be disabled before we could program them.

Great find!

Quoted 69 lines
drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.c new file +69 -0
+
+	/* Enable clockgating (ENG_CLK = RUN->AUTO) */
+	for (i = 0; order[i].engine != NVKM_SUBDEV_NR; i++) {
+		if (!nvkm_device_subdev(dev, order[i].engine))
+			continue;
+
+		nvkm_mask(dev, 0x20200 + order[i].offset, 0x00ff, 0x0045);
+	}
+}
+
+void
+gk104_clkgate_fini(struct nvkm_therm *base, bool suspend)
+{
+	struct gk104_therm *therm = gk104_therm(base);
+	struct nvkm_device *dev = therm->base.subdev.device;
+	const struct gk104_clkgate_engine_info *order = therm->clkgate_order;
+	int i;
+
+	/* ENG_CLK = AUTO->RUN, ENG_PWR = RUN->AUTO */
+	for (i = 0; order[i].engine != NVKM_SUBDEV_NR; i++) {
+		if (!nvkm_device_subdev(dev, order[i].engine))
+			continue;
+
+		nvkm_mask(dev, 0x20200 + order[i].offset, 0xff, 0x54);
+	}
+}
+
+const struct gk104_clkgate_engine_info gk104_clkgate_engine_info[] = {
+	{ NVKM_ENGINE_GR,     0x00 },
+	{ NVKM_ENGINE_MSPDEC, 0x04 },
+	{ NVKM_ENGINE_MSPPP,  0x08 },
+	{ NVKM_ENGINE_MSVLD,  0x0c },
+	{ NVKM_ENGINE_CE0,    0x10 },
+	{ NVKM_ENGINE_CE1,    0x14 },
+	{ NVKM_ENGINE_MSENC,  0x18 },
+	{ NVKM_ENGINE_CE2,    0x1c },
+	{ NVKM_SUBDEV_NR, 0 },
+};
+
+const struct gf100_idle_filter gk104_idle_filter = {
+	.fecs = 0x00001000,
+	.hubmmu = 0x00001000,
+};
+
+static const struct nvkm_therm_func
+gk104_therm_func = {
+	.init = gf119_therm_init,
+	.fini = g84_therm_fini,
+	.pwm_ctrl = gf119_fan_pwm_ctrl,
+	.pwm_get = gf119_fan_pwm_get,
+	.pwm_set = gf119_fan_pwm_set,
+	.pwm_clock = gf119_fan_pwm_clock,
+	.temp_get = g84_temp_get,
+	.fan_sense = gt215_therm_fan_sense,
+	.program_alarms = nvkm_therm_program_alarms_polling,
+	.clkgate_enable = gk104_clkgate_enable,
+	.clkgate_fini = gk104_clkgate_fini,
+};
+
+int
+gk104_therm_new_(const struct nvkm_therm_func *func,
+		 struct nvkm_device *device,
+		 int index,
+		 const struct gk104_clkgate_engine_info *clkgate_order,
+		 const struct gf100_idle_filter *idle_filter,
+		 struct nvkm_therm **ptherm)
+{
+	struct gk104_therm *therm = kzalloc(sizeof(*therm), GFP_KERNEL);
+

drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.c line 126

Quoted 10 lines
drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.c new file +10 -0
+	if (!therm)
+		return -ENOMEM;
+
+	nvkm_therm_ctor(&therm->base, device, index, func);
+	*ptherm = &therm->base;
+	therm->clkgate_order = clkgate_order;
+	therm->idle_filter = idle_filter;
+
+	return 0;
+}

Why introduce gk104_therm_new_? I can't find references to it in this patch (outside of the function below) or in the following patches.

As you even export this function, it looks like you used to use this function in an earlier revision of this series.

Aside from all these nitpicks, the approach is quite self contained and I like the following patches. Well done!

Once we settle on the configuration parameter, I can give you my R-b :)

Martin

Quoted 121 lines
drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.c new file +9 -0
+
+int
+gk104_therm_new(struct nvkm_device *device,
+		int index, struct nvkm_therm **ptherm)
+{
+	return gk104_therm_new_(&gk104_therm_func, device, index,
+				gk104_clkgate_engine_info, &gk104_idle_filter,
+				ptherm);
+}
drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.h new file +56 -0
new file mode 100644
index 000000000000..a2bb5304469f
@@ -0,0 +1,56 @@
+/*
+ * Copyright 2018 Red Hat Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Authors: Lyude Paul
+ */
+
+#ifndef __GK104_THERM_H__
+#define __GK104_THERM_H__
+#define gk104_therm(p) (container_of((p), struct gk104_therm, base))
+
+#include <subdev/therm.h>
+#include "priv.h"
+#include "gf100.h"
+
+struct gk104_clkgate_engine_info {
+	enum nvkm_devidx engine;
+	u8 offset;
+};
+
+struct gk104_therm {
+	struct nvkm_therm base;
+
+	const struct gk104_clkgate_engine_info *clkgate_order;
+	const struct gf100_idle_filter *idle_filter;
+};
+
+int
+gk104_therm_new_(const struct nvkm_therm_func *func,
+		 struct nvkm_device *device,
+		 int,
+		 const struct gk104_clkgate_engine_info *,
+		 const struct gf100_idle_filter *,
+		 struct nvkm_therm **);
+
+extern const struct gk104_clkgate_engine_info gk104_clkgate_engine_info[];
+extern const struct gf100_idle_filter gk104_idle_filter;
+
+#endif
drivers/gpu/drm/nouveau/nvkm/subdev/therm/priv.h +13 -2
index 1f46e371d7c4..f30202dd88e7 100644
@@ -32,6 +32,8 @@
 
 int nvkm_therm_new_(const struct nvkm_therm_func *, struct nvkm_device *,
 		    int index, struct nvkm_therm **);
+void nvkm_therm_ctor(struct nvkm_therm *therm, struct nvkm_device *device,
+		     int index, const struct nvkm_therm_func *func);
 
 struct nvkm_fan {
 	struct nvkm_therm *parent;
@@ -66,8 +68,6 @@ int nvkm_therm_fan_set(struct nvkm_therm *, bool now, int percent);
 int nvkm_therm_fan_user_get(struct nvkm_therm *);
 int nvkm_therm_fan_user_set(struct nvkm_therm *, int percent);
 
-int nvkm_therm_preinit(struct nvkm_therm *);
-
 int  nvkm_therm_sensor_init(struct nvkm_therm *);
 int  nvkm_therm_sensor_fini(struct nvkm_therm *, bool suspend);
 void nvkm_therm_sensor_preinit(struct nvkm_therm *);
@@ -96,6 +96,9 @@ struct nvkm_therm_func {
 	int (*fan_sense)(struct nvkm_therm *);
 
 	void (*program_alarms)(struct nvkm_therm *);
+
+	void (*clkgate_enable)(struct nvkm_therm *);
+	void (*clkgate_fini)(struct nvkm_therm *, bool);
 };
 
 void nv40_therm_intr(struct nvkm_therm *);
@@ -112,8 +115,16 @@ void g84_therm_fini(struct nvkm_therm *);
 int gt215_therm_fan_sense(struct nvkm_therm *);
 
 void g84_therm_init(struct nvkm_therm *);
+
+int gf119_fan_pwm_ctrl(struct nvkm_therm *, int, bool);
+int gf119_fan_pwm_get(struct nvkm_therm *, int, u32 *, u32 *);
+int gf119_fan_pwm_set(struct nvkm_therm *, int, u32, u32);
+int gf119_fan_pwm_clock(struct nvkm_therm *, int);
 void gf119_therm_init(struct nvkm_therm *);
 
+void gk104_clkgate_enable(struct nvkm_therm *);
+void gk104_clkgate_fini(struct nvkm_therm *, bool);
+
 int nvkm_fanpwm_create(struct nvkm_therm *, struct dcb_gpio_func *);
 int nvkm_fantog_create(struct nvkm_therm *, struct dcb_gpio_func *);
 int nvkm_fannil_create(struct nvkm_therm *);
author maintainer
#3 lore raw

RFC 2/4 drm/nouveau: Add support for BLCG clockgating for Kepler1

In reply to Lyude Paul

This enables the second level of clockgating for Kepler1: BLCG (presumably, this stands for block-level clockgating). This is a deeper level of power savings then just CG_CTRL. Setting it up is as simple as loading some mmio packs, and turning it on and off is still controlled through CG_CTRL.

We also introduce the nvkm_therm_clkgate_init() helper, which handles programming various BLCG register values while initializing the GPU, but before CG_CTRL is enabled.

Enabling BLCG is done with the nouveau config option NvPmEnableGating=2.

As well, this commit shares a lot more code with Fermi since BLCG is mostly the same there as far as we can tell. In the future, it's likely we'll reformat the clkgate_packs for kepler1 so that they share a list of mmio packs with Fermi.

---
 .../gpu/drm/nouveau/include/nvkm/subdev/therm.h    |  14 ++
 drivers/gpu/drm/nouveau/nvkm/engine/gr/gf100.h     |   1 +
 drivers/gpu/drm/nouveau/nvkm/engine/gr/gk104.c     | 215 ++++++++++++++++++++-
 drivers/gpu/drm/nouveau/nvkm/engine/gr/gk104.h     |  55 ++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gf100.c     |   6 +
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk104.c     |  53 +++++
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk104.h     |  35 ++++
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/priv.h      |   3 +
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/Kbuild   |   1 +
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/base.c   |  15 +-
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf100.c  |  77 ++++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.c  |   1 +
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gt215.c  |   2 +-
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/priv.h   |   8 +
 14 files changed, 482 insertions(+), 4 deletions(-)
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/engine/gr/gk104.h
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk104.h
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf100.c
drivers/gpu/drm/nouveau/include/nvkm/subdev/therm.h +14 -0
index a9204c09975b..4b49561415ef 100644
@@ -49,6 +49,18 @@ enum nvkm_therm_attr_type {
 enum nvkm_therm_clkgate_level {
 	NVKM_THERM_CLKGATE_NONE = 0,
 	NVKM_THERM_CLKGATE_CG, /* basic clockgating */
+	NVKM_THERM_CLKGATE_BLCG,
+};
+
+struct nvkm_therm_clkgate_init {
+	u32 addr;
+	u8  count;
+	u32 data;
+};
+
+struct nvkm_therm_clkgate_pack {
+	enum nvkm_therm_clkgate_level level;
+	const struct nvkm_therm_clkgate_init **init;
 };
 
 struct nvkm_therm {
@@ -97,6 +109,8 @@ struct nvkm_therm {
 int nvkm_therm_temp_get(struct nvkm_therm *);
 int nvkm_therm_fan_sense(struct nvkm_therm *);
 int nvkm_therm_cstate(struct nvkm_therm *, int, int);
+void nvkm_therm_clkgate_init(struct nvkm_therm *,
+			     const struct nvkm_therm_clkgate_pack *);
 void nvkm_therm_clkgate_enable(struct nvkm_therm *);
 void nvkm_therm_clkgate_fini(struct nvkm_therm *, bool);
 
drivers/gpu/drm/nouveau/nvkm/engine/gr/gf100.h +1 -0
index d7c2adb9b543..c8ec3fd97155 100644
@@ -137,6 +137,7 @@ struct gf100_gr_func {
 	int (*rops)(struct gf100_gr *);
 	int ppc_nr;
 	const struct gf100_grctx_func *grctx;
+	const struct nvkm_therm_clkgate_pack *clkgate_pack;
 	struct nvkm_sclass sclass[];
 };
 
drivers/gpu/drm/nouveau/nvkm/engine/gr/gk104.c +214 -1

245 lines not on this page · raw message

drivers/gpu/drm/nouveau/nvkm/engine/gr/gk104.h new file +55 -0

55 lines not on this page · raw message

drivers/gpu/drm/nouveau/nvkm/subdev/fb/gf100.c +6 -0
index 47d28c279707..cdc4e0a2cc6b 100644
@@ -26,6 +26,7 @@
 
 #include <core/memory.h>
 #include <core/option.h>
+#include <subdev/therm.h>
 
 void
 gf100_fb_intr(struct nvkm_fb *base)
@@ -92,6 +93,11 @@ gf100_fb_init(struct nvkm_fb *base)
 
 	if (fb->r100c10_page)
 		nvkm_wr32(device, 0x100c10, fb->r100c10 >> 8);
+
+	if (base->func->clkgate_pack) {
+		nvkm_therm_clkgate_init(device->therm,
+					base->func->clkgate_pack);
+	}
 }
 
 void *
drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk104.c +53 -0

69 lines not on this page · raw message

drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk104.h new file +35 -0
new file mode 100644
index 000000000000..b3c78e4ff706
@@ -0,0 +1,35 @@
+/*
+ * Copyright 2018 Red Hat Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Authors: Lyude Paul
+ */
+
+#ifndef __GK104_FB_H__
+#define __GK104_FB_H__
+
+#include <subdev/therm.h>
+
+extern const struct nvkm_therm_clkgate_init gk104_fb_clkgate_blcg_init_unk_0[];
+extern const struct nvkm_therm_clkgate_init gk104_fb_clkgate_blcg_init_vm_0[];
+extern const struct nvkm_therm_clkgate_init gk104_fb_clkgate_blcg_init_main_0[];
+extern const struct nvkm_therm_clkgate_init gk104_fb_clkgate_blcg_init_bcast_0[];
+
+#endif
drivers/gpu/drm/nouveau/nvkm/subdev/fb/priv.h +3 -0
index 9351188d5d76..cc6e0cbf761b 100644
@@ -3,6 +3,7 @@
 #define __NVKM_FB_PRIV_H__
 #define nvkm_fb(p) container_of((p), struct nvkm_fb, subdev)
 #include <subdev/fb.h>
+#include <subdev/therm.h>
 struct nvkm_bios;
 
 struct nvkm_fb_func {
@@ -27,6 +28,7 @@ struct nvkm_fb_func {
 	int (*ram_new)(struct nvkm_fb *, struct nvkm_ram **);
 
 	u8 default_bigpage;
+	const struct nvkm_therm_clkgate_pack *clkgate_pack;
 };
 
 void nvkm_fb_ctor(const struct nvkm_fb_func *, struct nvkm_device *device,
@@ -66,4 +68,5 @@ int gf100_fb_oneinit(struct nvkm_fb *);
 int gf100_fb_init_page(struct nvkm_fb *);
 
 int gm200_fb_init_page(struct nvkm_fb *);
+
 #endif
drivers/gpu/drm/nouveau/nvkm/subdev/therm/Kbuild +1 -0
index 4bac4772d8ed..550702eab0b1 100644
@@ -9,6 +9,7 @@ nvkm-y += nvkm/subdev/therm/nv40.o
 nvkm-y += nvkm/subdev/therm/nv50.o
 nvkm-y += nvkm/subdev/therm/g84.o
 nvkm-y += nvkm/subdev/therm/gt215.o
+nvkm-y += nvkm/subdev/therm/gf100.o
 nvkm-y += nvkm/subdev/therm/gf119.o
 nvkm-y += nvkm/subdev/therm/gk104.o
 nvkm-y += nvkm/subdev/therm/gm107.o
drivers/gpu/drm/nouveau/nvkm/subdev/therm/base.c +13 -2
index ba1a3aabb559..ee028d099f6a 100644
@@ -330,7 +330,8 @@ nvkm_therm_clkgate_oneinit(struct nvkm_therm *therm)
 		return;
 
 	switch (therm->clkgate_level) {
-	case NVKM_THERM_CLKGATE_CG: clkgate_str = "CG"; break;
+	case NVKM_THERM_CLKGATE_CG:   clkgate_str = "CG"; break;
+	case NVKM_THERM_CLKGATE_BLCG: clkgate_str = "BLCG"; break;
 	default: BUG();
 	}
 
@@ -400,6 +401,16 @@ nvkm_therm_init(struct nvkm_subdev *subdev)
 	return 0;
 }
 
+void
+nvkm_therm_clkgate_init(struct nvkm_therm *therm,
+			const struct nvkm_therm_clkgate_pack *p)
+{
+	if (!therm->clkgate_level || !therm->func->clkgate_init)
+		return;
+
+	therm->func->clkgate_init(therm, p);
+}
+
 static void *
 nvkm_therm_dtor(struct nvkm_subdev *subdev)
 {
@@ -438,7 +449,7 @@ nvkm_therm_ctor(struct nvkm_therm *therm, struct nvkm_device *device,
 		clamp((int)nvkm_longopt(device->cfgopt,
 					"NvPmEnableGating",
 					NVKM_THERM_CLKGATE_NONE),
-		      NVKM_THERM_CLKGATE_NONE, NVKM_THERM_CLKGATE_CG);
+		      NVKM_THERM_CLKGATE_NONE, NVKM_THERM_CLKGATE_BLCG);
 }
 
 int
drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf100.c new file +77 -0

77 lines not on this page · raw message

drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.c +1 -0
index 81bc73a6d143..efc8a23b55f8 100644
@@ -100,6 +100,7 @@ gk104_therm_func = {
 	.temp_get = g84_temp_get,
 	.fan_sense = gt215_therm_fan_sense,
 	.program_alarms = nvkm_therm_program_alarms_polling,
+	.clkgate_init = gf100_clkgate_init,
 	.clkgate_enable = gk104_clkgate_enable,
 	.clkgate_fini = gk104_clkgate_fini,
 };
drivers/gpu/drm/nouveau/nvkm/subdev/therm/gt215.c +1 -1
index c08097f2aff5..4caf401d001a 100644
@@ -36,7 +36,7 @@ gt215_therm_fan_sense(struct nvkm_therm *therm)
 	return -ENODEV;
 }
 
-static void
+void
 gt215_therm_init(struct nvkm_therm *therm)
 {
 	struct nvkm_device *device = therm->subdev.device;
drivers/gpu/drm/nouveau/nvkm/subdev/therm/priv.h +8 -0
index f30202dd88e7..a737e9b8a584 100644
@@ -97,6 +97,8 @@ struct nvkm_therm_func {
 
 	void (*program_alarms)(struct nvkm_therm *);
 
+	void (*clkgate_init)(struct nvkm_therm *,
+			     const struct nvkm_therm_clkgate_pack *);
 	void (*clkgate_enable)(struct nvkm_therm *);
 	void (*clkgate_fini)(struct nvkm_therm *, bool);
 };
@@ -114,6 +116,9 @@ void g84_therm_fini(struct nvkm_therm *);
 
 int gt215_therm_fan_sense(struct nvkm_therm *);
 
+void gf100_clkgate_init(struct nvkm_therm *,
+			const struct nvkm_therm_clkgate_pack *);
+
 void g84_therm_init(struct nvkm_therm *);
 
 int gf119_fan_pwm_ctrl(struct nvkm_therm *, int, bool);
@@ -122,6 +127,9 @@ int gf119_fan_pwm_set(struct nvkm_therm *, int, u32, u32);
 int gf119_fan_pwm_clock(struct nvkm_therm *, int);
 void gf119_therm_init(struct nvkm_therm *);
 
+void gt215_therm_init(struct nvkm_therm *therm);
+
+void gk104_therm_init(struct nvkm_therm *);
 void gk104_clkgate_enable(struct nvkm_therm *);
 void gk104_clkgate_fini(struct nvkm_therm *, bool);
 
Signature · 6 lines
2.14.3

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
author maintainer
#4 lore raw

RFC 3/4 drm/nouveau: Add BLCG clockgating for Kepler2

In reply to Lyude Paul

This is mostly the same as Kepler1, but we save even more power!

---
 drivers/gpu/drm/nouveau/include/nvkm/subdev/fb.h  |  1 +
 drivers/gpu/drm/nouveau/nvkm/engine/device/base.c |  8 +--
 drivers/gpu/drm/nouveau/nvkm/engine/gr/gk110.c    | 68 ++++++++++++++++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/Kbuild     |  1 +
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk110.c    | 77 +++++++++++++++++++++++
 5 files changed, 151 insertions(+), 4 deletions(-)
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk110.c
drivers/gpu/drm/nouveau/include/nvkm/subdev/fb.h +1 -0
index adb78f7d083a..92be0e5269c6 100644
@@ -75,6 +75,7 @@ int mcp89_fb_new(struct nvkm_device *, int, struct nvkm_fb **);
 int gf100_fb_new(struct nvkm_device *, int, struct nvkm_fb **);
 int gf108_fb_new(struct nvkm_device *, int, struct nvkm_fb **);
 int gk104_fb_new(struct nvkm_device *, int, struct nvkm_fb **);
+int gk110_fb_new(struct nvkm_device *, int, struct nvkm_fb **);
 int gk20a_fb_new(struct nvkm_device *, int, struct nvkm_fb **);
 int gm107_fb_new(struct nvkm_device *, int, struct nvkm_fb **);
 int gm200_fb_new(struct nvkm_device *, int, struct nvkm_fb **);
drivers/gpu/drm/nouveau/nvkm/engine/device/base.c +4 -4
index 6c5f966c66ad..332586de9a7b 100644
@@ -1812,7 +1812,7 @@ nvf0_chipset = {
 	.bus = gf100_bus_new,
 	.clk = gk104_clk_new,
 	.devinit = gf100_devinit_new,
-	.fb = gk104_fb_new,
+	.fb = gk110_fb_new,
 	.fuse = gf100_fuse_new,
 	.gpio = gk104_gpio_new,
 	.i2c = gk104_i2c_new,
@@ -1850,7 +1850,7 @@ nvf1_chipset = {
 	.bus = gf100_bus_new,
 	.clk = gk104_clk_new,
 	.devinit = gf100_devinit_new,
-	.fb = gk104_fb_new,
+	.fb = gk110_fb_new,
 	.fuse = gf100_fuse_new,
 	.gpio = gk104_gpio_new,
 	.i2c = gk104_i2c_new,
@@ -1888,7 +1888,7 @@ nv106_chipset = {
 	.bus = gf100_bus_new,
 	.clk = gk104_clk_new,
 	.devinit = gf100_devinit_new,
-	.fb = gk104_fb_new,
+	.fb = gk110_fb_new,
 	.fuse = gf100_fuse_new,
 	.gpio = gk104_gpio_new,
 	.i2c = gk104_i2c_new,
@@ -1926,7 +1926,7 @@ nv108_chipset = {
 	.bus = gf100_bus_new,
 	.clk = gk104_clk_new,
 	.devinit = gf100_devinit_new,
-	.fb = gk104_fb_new,
+	.fb = gk110_fb_new,
 	.fuse = gf100_fuse_new,
 	.gpio = gk104_gpio_new,
 	.i2c = gk104_i2c_new,
drivers/gpu/drm/nouveau/nvkm/engine/gr/gk110.c +68 -0

86 lines not on this page · raw message

drivers/gpu/drm/nouveau/nvkm/subdev/fb/Kbuild +1 -0
index 2571530e82f1..b4f22cce5d43 100644
@@ -22,6 +22,7 @@ nvkm-y += nvkm/subdev/fb/mcp89.o
 nvkm-y += nvkm/subdev/fb/gf100.o
 nvkm-y += nvkm/subdev/fb/gf108.o
 nvkm-y += nvkm/subdev/fb/gk104.o
+nvkm-y += nvkm/subdev/fb/gk110.o
 nvkm-y += nvkm/subdev/fb/gk20a.o
 nvkm-y += nvkm/subdev/fb/gm107.o
 nvkm-y += nvkm/subdev/fb/gm200.o
drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk110.c new file +77 -0

77 lines not on this page · raw message

Signature · 6 lines
2.14.3

_______________________________________________
Nouveau mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/nouveau
author maintainer
#5 lore raw

RFC 4/4 drm/nouveau: Add SLCG clockgating for Kepler2

In reply to Lyude Paul

That's right, there's still more power saving to go! This enables the third level of clockgating, SLCG (this stands for... we don't actually know what this stands for yet :\). While the register values look a little different, programming them is exactly the same as BLCG. Additionally, it should be noted that SLCG was added starting with kepler2, previous generations have no SLCG.

SLCG can be enabled with the nouveau config option, NvPmEnableGating=3

---
 .../gpu/drm/nouveau/include/nvkm/subdev/therm.h    |   1 +
 drivers/gpu/drm/nouveau/nvkm/engine/gr/gk110.c     | 108 +++++++++++++++++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/base.c   |   3 +-
 3 files changed, 111 insertions(+), 1 deletion(-)
drivers/gpu/drm/nouveau/include/nvkm/subdev/therm.h +1 -0
index 4b49561415ef..ba50207791e0 100644
@@ -50,6 +50,7 @@ enum nvkm_therm_clkgate_level {
 	NVKM_THERM_CLKGATE_NONE = 0,
 	NVKM_THERM_CLKGATE_CG, /* basic clockgating */
 	NVKM_THERM_CLKGATE_BLCG,
+	NVKM_THERM_CLKGATE_SLCG,
 };
 
 struct nvkm_therm_clkgate_init {
drivers/gpu/drm/nouveau/nvkm/engine/gr/gk110.c +108 -0

120 lines not on this page · raw message

drivers/gpu/drm/nouveau/nvkm/subdev/therm/base.c +2 -1
index ee028d099f6a..587dcc7444a7 100644
@@ -332,6 +332,7 @@ nvkm_therm_clkgate_oneinit(struct nvkm_therm *therm)
 	switch (therm->clkgate_level) {
 	case NVKM_THERM_CLKGATE_CG:   clkgate_str = "CG"; break;
 	case NVKM_THERM_CLKGATE_BLCG: clkgate_str = "BLCG"; break;
+	case NVKM_THERM_CLKGATE_SLCG: clkgate_str = "SLCG"; break;
 	default: BUG();
 	}
 
@@ -449,7 +450,7 @@ nvkm_therm_ctor(struct nvkm_therm *therm, struct nvkm_device *device,
 		clamp((int)nvkm_longopt(device->cfgopt,
 					"NvPmEnableGating",
 					NVKM_THERM_CLKGATE_NONE),
-		      NVKM_THERM_CLKGATE_NONE, NVKM_THERM_CLKGATE_BLCG);
+		      NVKM_THERM_CLKGATE_NONE, NVKM_THERM_CLKGATE_SLCG);
 }
 
 int
Signature · 6 lines
2.14.3

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

A message that is not in the archive (sent off-list or not yet received): CACY=MfW3b0qXALO1Y7bt=F7jMCfrdgi3qmmzKOXehJK5Xc5MmQ@mail.gmail.com

A message that is not in the archive (sent off-list or not yet received): CACY=MfVkNPsvNWGiy9o9A=YAZciV5k=jCJ4hrNUBocThgA56sQ@mail.gmail.com

#8 lore raw

Hello Paul

I have a GTX 480 (GF100 I believe) at home in an old machine. Is the rest of the Fermi patch set available somewhere for me to test?

Thank you
Regards Brock

On 16 Jan. 2018 9:08 am, "Lyude Paul" <[email protected]> wrote:

It's here! After a lot of investigation, rewrites, and traces, I present the patch series to implement all known )levels of clockgating for Kepler1 and Kepler2 GPUs.

Starting with Fermi GPUs (this is probably present on earlier GPUs as well, but with a far less easy to manage interface), nvidia added two clockgating levels that are handled mostly in firmware (with the exception of course, of the driver initially programming all of the register values containing engine delays and that stuff):
- CG_CTRL - Main register for enabling/disabling clockgating for engines and hw blocks
- BLCG - "Block-level clockgating", a deeper level of clockgating
Starting with kepler2 as well, nvidia also introduced:
- SLCG - "??? clockgating" even deeper level of clockgating

Originally this patchset was going to include work for making this work on Fermi, however on closer investigation it seems that Fermi has one pretty big difference in it's requirements that we don't entirely understand yet. On Fermi the CG_CTRL register for the gr needs to be adjusted during reclocking, while kepler and later generations need no such adjustments. Since this requires more research; the current plan is to leave fermi out of this patchset, and then just rework the code at a later point in time to support Fermi once we understand more.

For the time being however, we put as much code we're sure Fermi will be sharing into gf100 files for the future.

For the time being; this patchset supports every GPU in the kepler1 and kepler2 family. The main kernel config option to enable this is
config=NvPmEnableGating=<level>
Where <level> is the deepest level of powersaving to enable. The levels being:
0. NOCG
1. CG (just CG_CTRL)
2. BLCG
3. SLCG

Additionally, we leave a couple of small TODO comments to mark spots we'll eventually need to add code for the final level of power saving we have yet to implement or understand: ELPG (engine-level powergating).

Here's some very lazily done benchmarks on how much power this saves as well. It should be noted that chances are, these patches save a lot more power then you see here since clockgating works best when the GPU is under load. I have a feeling this will be especially true with SLCG.

GK104 stats:
Idle (1 4k display @ fbcon):

    NOCG:
      07: 20.53W
      0a: 44.12W
      0e: 61.23W
      0f: 62.23W
    CG:
      07: 18.95W-19.02W
      0a: 34.95W-37.48W
      0e: 52.70W-54.32W
      0f: 53.53W-55.68W
    BLCG:
      07: 18.64W-19.02W
      0a: 32.68W-34.78W
      0e: 49.14W-50.51W
      0f: 49.75W-51.73W
GK110 stats:
  Idle (1 4k display @ fbcon):
    NOCG:
      07: 25.68W
      0a: 36.16W
      0d: 64.71W
      0f: 64.99W
    CG:
      07: 25.68W
      0a: 35.58W
      0d: 61.54W-61.74W
      0f: 61.93W-62.12W
    BLCG:
      07: 25W-25.2W
      0a: 33.85W-34.04W
      0d: 59.25W-59.44W
      0f: 59.35W-59.54W
    SLCG:
      07: 25W-25.2W
      0a: 33.85W-33.95W (spikes to 34.04W briefly every now and then)
      0d: 59.15W-59.35W
      0f: 59.35W-59.54W

For the time being, this will be disabled by default in the kernel as we wait to get more widespread testing with this patchset. If you're willing to put up with the potential of instability, please feel free to try this patchset and let us know how well it works for you!

Lyude Paul (4):
drm/nouveau: Add support for basic clockgating on Kepler1 drm/nouveau: Add support for BLCG clockgating for Kepler1 drm/nouveau: Add BLCG clockgating for Kepler2 drm/nouveau: Add SLCG clockgating for Kepler2

 drivers/gpu/drm/nouveau/include/nvkm/subdev/fb.h   |   1 +
 .../gpu/drm/nouveau/include/nvkm/subdev/therm.h    |  25 +++
 drivers/gpu/drm/nouveau/nvkm/engine/device/base.c  |  25 +--
 drivers/gpu/drm/nouveau/nvkm/engine/gr/gf100.h     |   1 +
 drivers/gpu/drm/nouveau/nvkm/engine/gr/gk104.c     | 215
++++++++++++++++++++-
 drivers/gpu/drm/nouveau/nvkm/engine/gr/gk104.h     |  55 ++++++
 drivers/gpu/drm/nouveau/nvkm/engine/gr/gk110.c     | 176 +++++++++++++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/Kbuild      |   1 +
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gf100.c     |   6 +
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk104.c     |  53 +++++
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk104.h     |  35 ++++
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk110.c     |  77 ++++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/priv.h      |   3 +
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/Kbuild   |   2 +
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/base.c   |  84 +++++++-
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf100.c  |  77 ++++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf100.h  |  35 ++++
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf119.c  |   8 +-
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.c  | 136 +++++++++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.h  |  56 ++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gt215.c  |   2 +-
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/priv.h   |  23 ++-
 22 files changed, 1069 insertions(+), 27 deletions(-)
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/engine/gr/gk104.h
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk104.h
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk110.c
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf100.c
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf100.h
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.c
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.h

--
2.14.3

_______________________________________________
Nouveau mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/nouveau

_______________________________________________
Nouveau mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/nouveau

author maintainer
#9 lore raw

Unfortunately no. in order to support reclocking on Fermi with powergating (which is important; since you can't test clockgating very well without it) we are going to need to do some more investigation. Unlike Kepler and later generations, Fermi seems to require some extra maintanence of it's clockgating registers (CG_CTRL) during reclocking operations that we don't quite understand yet. We'll figure it out, but I'd like to get out initial support for Kepler first since I'm confident this series is pretty solid and it would be good to get early feedback on this.
Plus, Fermi doesn't have working reclocking in nouveau just yet. I am planning on eventually adding Fermi clockgating support when reclocking gets fixed!

Quoted 148 lines · Brock York

Hello Paul

I have a GTX 480 (GF100 I believe) at home in an old machine. Is the rest of the Fermi patch set available somewhere for me to test?

Thank you
Regards Brock

Quoted 137 lines · Lyude Paul

It's here! After a lot of investigation, rewrites, and traces, I present the patch series to implement all known )levels of clockgating for Kepler1 and Kepler2 GPUs.

Starting with Fermi GPUs (this is probably present on earlier GPUs as well, but with a far less easy to manage interface), nvidia added two clockgating levels that are handled mostly in firmware (with the exception of course, of the driver initially programming all of the register values containing engine delays and that stuff):
- CG_CTRL - Main register for enabling/disabling clockgating for engines and hw blocks
- BLCG - "Block-level clockgating", a deeper level of clockgating
Starting with kepler2 as well, nvidia also introduced:
- SLCG - "??? clockgating" even deeper level of clockgating

Originally this patchset was going to include work for making this work on Fermi, however on closer investigation it seems that Fermi has one pretty big difference in it's requirements that we don't entirely understand yet. On Fermi the CG_CTRL register for the gr needs to be adjusted during reclocking, while kepler and later generations need no such adjustments. Since this requires more research; the current plan is to leave fermi out of this patchset, and then just rework the code at a later point in time to support Fermi once we understand more.

For the time being however, we put as much code we're sure Fermi will be sharing into gf100 files for the future.

For the time being; this patchset supports every GPU in the kepler1 and kepler2 family. The main kernel config option to enable this is
config=NvPmEnableGating=<level>
Where <level> is the deepest level of powersaving to enable. The levels being:
0. NOCG
1. CG (just CG_CTRL)
2. BLCG
3. SLCG

Additionally, we leave a couple of small TODO comments to mark spots we'll eventually need to add code for the final level of power saving we have yet to implement or understand: ELPG (engine-level powergating).

Here's some very lazily done benchmarks on how much power this saves as well. It should be noted that chances are, these patches save a lot more power then you see here since clockgating works best when the GPU is under load. I have a feeling this will be especially true with SLCG.

GK104 stats:
Idle (1 4k display @ fbcon):

    NOCG:
      07: 20.53W
      0a: 44.12W
      0e: 61.23W
      0f: 62.23W
    CG:
      07: 18.95W-19.02W
      0a: 34.95W-37.48W
      0e: 52.70W-54.32W
      0f: 53.53W-55.68W
    BLCG:
      07: 18.64W-19.02W
      0a: 32.68W-34.78W
      0e: 49.14W-50.51W
      0f: 49.75W-51.73W
GK110 stats:
  Idle (1 4k display @ fbcon):
    NOCG:
      07: 25.68W
      0a: 36.16W
      0d: 64.71W
      0f: 64.99W
    CG:
      07: 25.68W
      0a: 35.58W
      0d: 61.54W-61.74W
      0f: 61.93W-62.12W
    BLCG:
      07: 25W-25.2W
      0a: 33.85W-34.04W
      0d: 59.25W-59.44W
      0f: 59.35W-59.54W
    SLCG:
      07: 25W-25.2W
      0a: 33.85W-33.95W (spikes to 34.04W briefly every now and then)
      0d: 59.15W-59.35W
      0f: 59.35W-59.54W

For the time being, this will be disabled by default in the kernel as we wait to get more widespread testing with this patchset. If you're willing to put up with the potential of instability, please feel free to try this patchset and let us know how well it works for you!

Lyude Paul (4):
drm/nouveau: Add support for basic clockgating on Kepler1 drm/nouveau: Add support for BLCG clockgating for Kepler1 drm/nouveau: Add BLCG clockgating for Kepler2 drm/nouveau: Add SLCG clockgating for Kepler2

 drivers/gpu/drm/nouveau/include/nvkm/subdev/fb.h   |   1 +
 .../gpu/drm/nouveau/include/nvkm/subdev/therm.h    |  25 +++
 drivers/gpu/drm/nouveau/nvkm/engine/device/base.c  |  25 +--
 drivers/gpu/drm/nouveau/nvkm/engine/gr/gf100.h     |   1 +
 drivers/gpu/drm/nouveau/nvkm/engine/gr/gk104.c     | 215
++++++++++++++++++++-
 drivers/gpu/drm/nouveau/nvkm/engine/gr/gk104.h     |  55 ++++++
 drivers/gpu/drm/nouveau/nvkm/engine/gr/gk110.c     | 176
+++++++++++++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/Kbuild      |   1 +
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gf100.c     |   6 +
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk104.c     |  53 +++++
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk104.h     |  35 ++++
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk110.c     |  77 ++++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/fb/priv.h      |   3 +
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/Kbuild   |   2 +
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/base.c   |  84 +++++++-
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf100.c  |  77 ++++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf100.h  |  35 ++++
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf119.c  |   8 +-
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.c  | 136 +++++++++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.h  |  56 ++++++
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gt215.c  |   2 +-
 drivers/gpu/drm/nouveau/nvkm/subdev/therm/priv.h   |  23 ++-
 22 files changed, 1069 insertions(+), 27 deletions(-)
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/engine/gr/gk104.h
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk104.h
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk110.c
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf100.c
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf100.h
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.c
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.h

--
2.14.3

_______________________________________________
Nouveau mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/nouveau

Signature · 6 lines
Cheers,
	Lyude Paul
_______________________________________________
Nouveau mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/nouveau
#10 lore raw
In reply to Lyude Paul
Quoted 4 lines · Lyude Paul

It's here! After a lot of investigation, rewrites, and traces, I present the patch series to implement all known levels of clockgating for Kepler1 and Kepler2 GPUs.

Quoted 10 lines

Starting with Fermi GPUs (this is probably present on earlier GPUs as well, but with a far less easy to manage interface), nvidia added two clockgating levels that are handled mostly in firmware (with the exception of course, of the driver initially programming all of the register values containing engine delays and that stuff):
- CG_CTRL - Main register for enabling/disabling clockgating for engines and hw blocks
- BLCG - "Block-level clockgating", a deeper level of clockgating
Starting with kepler2 as well, nvidia also introduced:
- SLCG - "??? clockgating" even deeper level of clockgating

FWIW, SLCG stands for "second level clock gating".

Cheers,
Mikko