memory: tegra: Introduce Tegra30 EMC driver

A proper External Memory Controller clock rounding and parent selection functionality is required by the EMC drivers, it is not available using the generic clock implementation because only the Memory Controller driver is aware of what clock rates are actually available for a particular device. EMC drivers will have to register a Tegra-specific CLK-API callback which will perform rounding of a requested rate. EMC clock users won't be able to request EMC clock by getting -EPROBE_DEFER until EMC driver is probed and the callback is set up.

drivers/clk/tegra/Makefile +0 -0

index 4812e45c2214..df966ca06788 100644

Quoted 8 lines

A proper External Memory Controller clock rounding and parent selection functionality is required by the EMC drivers, it is not available using the generic clock implementation because only the Memory Controller driver is aware of what clock rates are actually available for a particular device. EMC drivers will have to register a Tegra-specific CLK-API callback which will perform rounding of a requested rate. EMC clock users won't be able to request EMC clock by getting -EPROBE_DEFER until EMC driver is probed and the callback is set up.

[...]

Quoted 9 lines

drivers/clk/tegra/Makefile +1 -0

index 4812e45c2214..df966ca06788 100644
@@ -17,7 +17,9 @@ obj-y					+= clk-tegra-fixed.o
 obj-y					+= clk-tegra-super-gen4.o
 obj-$(CONFIG_TEGRA_CLK_EMC)		+= clk-emc.o
 obj-$(CONFIG_ARCH_TEGRA_2x_SOC)         += clk-tegra20.o
+obj-$(CONFIG_ARCH_TEGRA_2x_SOC)		+= clk-tegra20-emc.o

drivers/clk/tegra/Makefile obj-y += clk-tegra-fixed.o · line 25

Quoted 5 lines

drivers/clk/tegra/Makefile +1 -0

 obj-$(CONFIG_ARCH_TEGRA_3x_SOC)         += clk-tegra30.o
+obj-$(CONFIG_ARCH_TEGRA_3x_SOC)		+= clk-tegra20-emc.o
 obj-$(CONFIG_ARCH_TEGRA_114_SOC)	+= clk-tegra114.o
 obj-$(CONFIG_ARCH_TEGRA_124_SOC)	+= clk-tegra124.o
 obj-$(CONFIG_TEGRA_CLK_DFLL)		+= clk-tegra124-dfll-fcpu.o

Doesn't it complain when both CONFIG_ARCH_TEGRA_2x_SOC and CONFIG_ARCH_TEGRA_3x_SOC are enabled at the same time?

13.08.2019 2:12, Michał Mirosław пишет:

Quoted 19 lines · Dmitry Osipenko

Quoted 8 lines

A proper External Memory Controller clock rounding and parent selection functionality is required by the EMC drivers, it is not available using the generic clock implementation because only the Memory Controller driver is aware of what clock rates are actually available for a particular device. EMC drivers will have to register a Tegra-specific CLK-API callback which will perform rounding of a requested rate. EMC clock users won't be able to request EMC clock by getting -EPROBE_DEFER until EMC driver is probed and the callback is set up.

[...]

Quoted 9 lines

drivers/clk/tegra/Makefile +1 -0

index 4812e45c2214..df966ca06788 100644
@@ -17,7 +17,9 @@ obj-y					+= clk-tegra-fixed.o
 obj-y					+= clk-tegra-super-gen4.o
 obj-$(CONFIG_TEGRA_CLK_EMC)		+= clk-emc.o
 obj-$(CONFIG_ARCH_TEGRA_2x_SOC)         += clk-tegra20.o
+obj-$(CONFIG_ARCH_TEGRA_2x_SOC)		+= clk-tegra20-emc.o

Quoted 8 lines

drivers/clk/tegra/Makefile obj-y += clk-tegra-fixed.o · line 25

Quoted 5 lines

drivers/clk/tegra/Makefile +1 -0

 obj-$(CONFIG_ARCH_TEGRA_3x_SOC)         += clk-tegra30.o
+obj-$(CONFIG_ARCH_TEGRA_3x_SOC)		+= clk-tegra20-emc.o
 obj-$(CONFIG_ARCH_TEGRA_114_SOC)	+= clk-tegra114.o
 obj-$(CONFIG_ARCH_TEGRA_124_SOC)	+= clk-tegra124.o
 obj-$(CONFIG_TEGRA_CLK_DFLL)		+= clk-tegra124-dfll-fcpu.o

Doesn't it complain when both CONFIG_ARCH_TEGRA_2x_SOC and CONFIG_ARCH_TEGRA_3x_SOC are enabled at the same time?

No, at least not with my toolchain setup. Are you getting some warning?

A proper External Memory Controller clock rounding and parent selection functionality is required by the EMC drivers, it is not available using the generic clock implementation because only the Memory Controller driver is aware of what clock rates are actually available for a particular device. EMC drivers will have to register a Tegra-specific CLK-API callback which will perform rounding of a requested rate. EMC clock users won't be able to request EMC clock by getting -EPROBE_DEFER until EMC driver is probed and the callback is set up.

The functionality is somewhat similar to the clk-emc.c which serves Tegra124+ SoCs. The later HW generations support more parent clock sources and the HW configuration / integration with the EMC drivers differs a tad from the older gens, hence it's not really worth to try to squash everything into a single source file.

Acked-by Peter De Schrijver <[email protected]>

Signed-off-by Dmitry Osipenko <[email protected]>

---

The Tegra30 binding will actually differ from the Tegra124 a tad, in particular the EMEM configuration description. Hence rename the binding to Tegra124 during of the conversion to YAML.

Reviewed-by Rob Herring <[email protected]>

Signed-off-by Dmitry Osipenko <[email protected]>

---
 .../nvidia,tegra124-mc.yaml                   | 152 ++++++++++++++++++
 .../memory-controllers/nvidia,tegra30-mc.txt  | 123 --------------
 2 files changed, 152 insertions(+), 123 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/memory-controllers/nvidia,tegra124-mc.yaml
 delete mode 100644 Documentation/devicetree/bindings/memory-controllers/nvidia,tegra30-mc.txt

Quoted 5 lines · Dmitry Osipenko

The Tegra30 binding will actually differ from the Tegra124 a tad, in particular the EMEM configuration description. Hence rename the binding to Tegra124 during of the conversion to YAML.

Quoted 8 lines

Reviewed-by Rob Herring <[email protected]>

Signed-off-by Dmitry Osipenko <[email protected]>

---
 .../nvidia,tegra124-mc.yaml                   | 152 ++++++++++++++++++
 .../memory-controllers/nvidia,tegra30-mc.txt  | 123 --------------
 2 files changed, 152 insertions(+), 123 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/memory-controllers/nvidia,tegra124-mc.yaml
 delete mode 100644 Documentation/devicetree/bindings/memory-controllers/nvidia,tegra30-mc.txt

Reviewed-by Rob Herring <[email protected]>

Quoted 16 lines · Dmitry Osipenko

Quoted 13 lines

The Tegra30 binding will actually differ from the Tegra124 a tad, in particular the EMEM configuration description. Hence rename the binding to Tegra124 during of the conversion to YAML.

Reviewed-by Rob Herring <[email protected]>

Signed-off-by Dmitry Osipenko <[email protected]>

---
 .../nvidia,tegra124-mc.yaml                   | 152 ++++++++++++++++++
 .../memory-controllers/nvidia,tegra30-mc.txt  | 123 --------------
 2 files changed, 152 insertions(+), 123 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/memory-controllers/nvidia,tegra124-mc.yaml
 delete mode 100644 Documentation/devicetree/bindings/memory-controllers/nvidia,tegra30-mc.txt

Quoted 1 line

Reviewed-by Rob Herring <[email protected]>

Wrong patch, but still R-by... :)

Add binding for the NVIDIA Tegra30 SoC Memory Controller.

Signed-off-by Dmitry Osipenko <[email protected]>

---
 .../memory-controllers/nvidia,tegra30-mc.yaml | 167 ++++++++++++++++++
 1 file changed, 167 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/memory-controllers/nvidia,tegra30-mc.yaml

Add device-tree binding for NVIDIA Tegra30 External Memory Controller. The binding is based on the Tegra124 EMC binding since hardware is similar, although there are couple significant differences.

Note that the memory timing description is given in a platform-specific form because there is no detailed information on how to convert a typical-common DDR timing into the register values. The timing format is borrowed from downstream kernel, hence there is no hurdle in regards to upstreaming of memory timings for the boards.

Acked-by Peter De Schrijver <[email protected]>

Signed-off-by Dmitry Osipenko <[email protected]>

---
 .../nvidia,tegra30-emc.yaml                   | 336 ++++++++++++++++++
 1 file changed, 336 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/memory-controllers/nvidia,tegra30-emc.yaml

Introduce driver for the External Memory Controller (EMC) found on Tegra30 chips, it controls the external DRAM on the board. The purpose of this driver is to program memory timing for external memory on the EMC clock rate change.

Acked-by Peter De Schrijver <[email protected]>

Signed-off-by Dmitry Osipenko <[email protected]>

---
 drivers/memory/tegra/Kconfig       |   10 +
 drivers/memory/tegra/Makefile      |    1 +
 drivers/memory/tegra/mc.c          |    9 +-
 drivers/memory/tegra/mc.h          |   30 +-
 drivers/memory/tegra/tegra30-emc.c | 1232 ++++++++++++++++++++++++++++
 drivers/memory/tegra/tegra30.c     |   42 +
 include/soc/tegra/mc.h             |    2 +-
 7 files changed, 1311 insertions(+), 15 deletions(-)
 create mode 100644 drivers/memory/tegra/tegra30-emc.c

drivers/memory/tegra/Kconfig +10 -0

index 4680124ddcab..fbfbaada61a2 100644
@@ -17,6 +17,16 @@ config TEGRA20_EMC
          This driver is required to change memory timings / clock rate for
          external memory.
 
+config TEGRA30_EMC
+       bool "NVIDIA Tegra30 External Memory Controller driver"
+       default y
+       depends on TEGRA_MC && ARCH_TEGRA_3x_SOC
+       help
+         This driver is for the External Memory Controller (EMC) found on
+         Tegra30 chips. The EMC controls the external DRAM on the board.
+         This driver is required to change memory timings / clock rate for
+         external memory.
+
 config TEGRA124_EMC
        bool "NVIDIA Tegra124 External Memory Controller driver"
        default y

drivers/memory/tegra/Makefile +1 -0

index 3971a6b7c487..3d23c4261104 100644
@@ -11,5 +11,6 @@ tegra-mc-$(CONFIG_ARCH_TEGRA_210_SOC) += tegra210.o
 obj-$(CONFIG_TEGRA_MC) += tegra-mc.o
 
 obj-$(CONFIG_TEGRA20_EMC)  += tegra20-emc.o
+obj-$(CONFIG_TEGRA30_EMC)  += tegra30-emc.o
 obj-$(CONFIG_TEGRA124_EMC) += tegra124-emc.o
 obj-$(CONFIG_ARCH_TEGRA_186_SOC) += tegra186.o

drivers/memory/tegra/mc.c +4 -5

index 3d8d322511c5..43819e8df95c 100644
@@ -48,9 +48,6 @@
 #define MC_EMEM_ADR_CFG 0x54
 #define MC_EMEM_ADR_CFG_EMEM_NUMDEV BIT(0)
 
-#define MC_TIMING_CONTROL              0xfc
-#define MC_TIMING_UPDATE               BIT(0)
-
 static const struct of_device_id tegra_mc_of_match[] = {
 #ifdef CONFIG_ARCH_TEGRA_2x_SOC
        { .compatible = "nvidia,tegra20-mc-gart", .data = &tegra20_mc_soc },

@@ -307,7 +304,7 @@ static int tegra_mc_setup_latency_allowance(struct tegra_mc *mc)
        return 0;
 }
 
-void tegra_mc_write_emem_configuration(struct tegra_mc *mc, unsigned long rate)
+int tegra_mc_write_emem_configuration(struct tegra_mc *mc, unsigned long rate)
 {
        unsigned int i;
        struct tegra_mc_timing *timing = NULL;

@@ -322,11 +319,13 @@ void tegra_mc_write_emem_configuration(struct tegra_mc *mc, unsigned long rate)
        if (!timing) {
                dev_err(mc->dev, "no memory timing registered for rate %lu\n",
                        rate);
-               return;
+               return -EINVAL;
        }
 
        for (i = 0; i < mc->soc->num_emem_regs; ++i)
                mc_writel(mc, timing->emem_data[i], mc->soc->emem_regs[i]);
+
+       return 0;
 }
 
 unsigned int tegra_mc_get_emem_device_count(struct tegra_mc *mc)

drivers/memory/tegra/mc.h +21 -9

index f9353494b708..410efc4d7e7b 100644
@@ -6,20 +6,32 @@
 #ifndef MEMORY_TEGRA_MC_H
 #define MEMORY_TEGRA_MC_H
 
+#include <linux/bits.h>
 #include <linux/io.h>
 #include <linux/types.h>
 
 #include <soc/tegra/mc.h>
 
-#define MC_INT_DECERR_MTS (1 << 16)
-#define MC_INT_SECERR_SEC (1 << 13)
-#define MC_INT_DECERR_VPR (1 << 12)
-#define MC_INT_INVALID_APB_ASID_UPDATE (1 << 11)
-#define MC_INT_INVALID_SMMU_PAGE (1 << 10)
-#define MC_INT_ARBITRATION_EMEM (1 << 9)
-#define MC_INT_SECURITY_VIOLATION (1 << 8)
-#define MC_INT_INVALID_GART_PAGE (1 << 7)
-#define MC_INT_DECERR_EMEM (1 << 6)
+#define MC_INT_DECERR_MTS                              BIT(16)
+#define MC_INT_SECERR_SEC                              BIT(13)
+#define MC_INT_DECERR_VPR                              BIT(12)
+#define MC_INT_INVALID_APB_ASID_UPDATE                 BIT(11)
+#define MC_INT_INVALID_SMMU_PAGE                       BIT(10)
+#define MC_INT_ARBITRATION_EMEM                                BIT(9)
+#define MC_INT_SECURITY_VIOLATION                      BIT(8)
+#define MC_INT_INVALID_GART_PAGE                       BIT(7)
+#define MC_INT_DECERR_EMEM                             BIT(6)
+
+#define MC_EMEM_ARB_OUTSTANDING_REQ                    0x94
+#define MC_EMEM_ARB_OUTSTANDING_REQ_MAX_MASK           0x1ff
+#define MC_EMEM_ARB_OUTSTANDING_REQ_HOLDOFF_OVERRIDE   BIT(30)
+#define MC_EMEM_ARB_OUTSTANDING_REQ_LIMIT_ENABLE       BIT(31)
+
+#define MC_EMEM_ARB_OVERRIDE                           0xe8
+#define MC_EMEM_ARB_OVERRIDE_EACK_MASK                 0x3
+
+#define MC_TIMING_CONTROL                              0xfc
+#define MC_TIMING_UPDATE                               BIT(0)
 
 static inline u32 mc_readl(struct tegra_mc *mc, unsigned long offset)
 {

drivers/memory/tegra/tegra30-emc.c new file +1232 -0

new file mode 100644
index 000000000000..6929980bf907
@@ -0,0 +1,1232 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Tegra30 External Memory Controller driver
+ *
+ * Based on downstream driver from NVIDIA and tegra124-emc.c
+ * Copyright (C) 2011-2014 NVIDIA Corporation
+ *
+ * Author: Dmitry Osipenko <[email protected]>
+ * Copyright (C) 2019 GRATE-DRIVER project
+ */
+
+#include <linux/clk.h>
+#include <linux/clk/tegra.h>
+#include <linux/completion.h>
+#include <linux/delay.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/iopoll.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/of_platform.h>
+#include <linux/platform_device.h>
+#include <linux/sort.h>
+#include <linux/types.h>
+
+#include <soc/tegra/fuse.h>
+
+#include "mc.h"
+
+#define EMC_INTSTATUS                          0x000
+#define EMC_INTMASK                            0x004
+#define EMC_DBG                                        0x008
+#define EMC_CFG                                        0x00c
+#define EMC_REFCTRL                            0x020
+#define EMC_TIMING_CONTROL                     0x028
+#define EMC_RC                                 0x02c
+#define EMC_RFC                                        0x030
+#define EMC_RAS                                        0x034
+#define EMC_RP                                 0x038
+#define EMC_R2W                                        0x03c
+#define EMC_W2R                                        0x040
+#define EMC_R2P                                        0x044
+#define EMC_W2P                                        0x048
+#define EMC_RD_RCD                             0x04c
+#define EMC_WR_RCD                             0x050
+#define EMC_RRD                                        0x054
+#define EMC_REXT                               0x058
+#define EMC_WDV                                        0x05c
+#define EMC_QUSE                               0x060
+#define EMC_QRST                               0x064
+#define EMC_QSAFE                              0x068
+#define EMC_RDV                                        0x06c
+#define EMC_REFRESH                            0x070
+#define EMC_BURST_REFRESH_NUM                  0x074
+#define EMC_PDEX2WR                            0x078
+#define EMC_PDEX2RD                            0x07c
+#define EMC_PCHG2PDEN                          0x080
+#define EMC_ACT2PDEN                           0x084
+#define EMC_AR2PDEN                            0x088
+#define EMC_RW2PDEN                            0x08c
+#define EMC_TXSR                               0x090
+#define EMC_TCKE                               0x094
+#define EMC_TFAW                               0x098
+#define EMC_TRPAB                              0x09c
+#define EMC_TCLKSTABLE                         0x0a0
+#define EMC_TCLKSTOP                           0x0a4
+#define EMC_TREFBW                             0x0a8
+#define EMC_QUSE_EXTRA                         0x0ac
+#define EMC_ODT_WRITE                          0x0b0
+#define EMC_ODT_READ                           0x0b4
+#define EMC_WEXT                               0x0b8
+#define EMC_CTT                                        0x0bc
+#define EMC_MRS_WAIT_CNT                       0x0c8
+#define EMC_MRS                                        0x0cc
+#define EMC_EMRS                               0x0d0
+#define EMC_SELF_REF                           0x0e0
+#define EMC_MRW                                        0x0e8
+#define EMC_XM2DQSPADCTRL3                     0x0f8
+#define EMC_FBIO_SPARE                         0x100
+#define EMC_FBIO_CFG5                          0x104
+#define EMC_FBIO_CFG6                          0x114
+#define EMC_CFG_RSV                            0x120
+#define EMC_AUTO_CAL_CONFIG                    0x2a4
+#define EMC_AUTO_CAL_INTERVAL                  0x2a8
+#define EMC_AUTO_CAL_STATUS                    0x2ac
+#define EMC_STATUS                             0x2b4
+#define EMC_CFG_2                              0x2b8
+#define EMC_CFG_DIG_DLL                                0x2bc
+#define EMC_CFG_DIG_DLL_PERIOD                 0x2c0
+#define EMC_CTT_DURATION                       0x2d8
+#define EMC_CTT_TERM_CTRL                      0x2dc
+#define EMC_ZCAL_INTERVAL                      0x2e0
+#define EMC_ZCAL_WAIT_CNT                      0x2e4
+#define EMC_ZQ_CAL                             0x2ec
+#define EMC_XM2CMDPADCTRL                      0x2f0
+#define EMC_XM2DQSPADCTRL2                     0x2fc
+#define EMC_XM2DQPADCTRL2                      0x304
+#define EMC_XM2CLKPADCTRL                      0x308
+#define EMC_XM2COMPPADCTRL                     0x30c
+#define EMC_XM2VTTGENPADCTRL                   0x310
+#define EMC_XM2VTTGENPADCTRL2                  0x314
+#define EMC_XM2QUSEPADCTRL                     0x318
+#define EMC_DLL_XFORM_DQS0                     0x328
+#define EMC_DLL_XFORM_DQS1                     0x32c
+#define EMC_DLL_XFORM_DQS2                     0x330
+#define EMC_DLL_XFORM_DQS3                     0x334
+#define EMC_DLL_XFORM_DQS4                     0x338
+#define EMC_DLL_XFORM_DQS5                     0x33c
+#define EMC_DLL_XFORM_DQS6                     0x340
+#define EMC_DLL_XFORM_DQS7                     0x344
+#define EMC_DLL_XFORM_QUSE0                    0x348
+#define EMC_DLL_XFORM_QUSE1                    0x34c
+#define EMC_DLL_XFORM_QUSE2                    0x350
+#define EMC_DLL_XFORM_QUSE3                    0x354
+#define EMC_DLL_XFORM_QUSE4                    0x358
+#define EMC_DLL_XFORM_QUSE5                    0x35c
+#define EMC_DLL_XFORM_QUSE6                    0x360
+#define EMC_DLL_XFORM_QUSE7                    0x364
+#define EMC_DLL_XFORM_DQ0                      0x368
+#define EMC_DLL_XFORM_DQ1                      0x36c
+#define EMC_DLL_XFORM_DQ2                      0x370
+#define EMC_DLL_XFORM_DQ3                      0x374
+#define EMC_DLI_TRIM_TXDQS0                    0x3a8
+#define EMC_DLI_TRIM_TXDQS1                    0x3ac
+#define EMC_DLI_TRIM_TXDQS2                    0x3b0
+#define EMC_DLI_TRIM_TXDQS3                    0x3b4
+#define EMC_DLI_TRIM_TXDQS4                    0x3b8
+#define EMC_DLI_TRIM_TXDQS5                    0x3bc
+#define EMC_DLI_TRIM_TXDQS6                    0x3c0
+#define EMC_DLI_TRIM_TXDQS7                    0x3c4
+#define EMC_STALL_THEN_EXE_BEFORE_CLKCHANGE    0x3c8
+#define EMC_STALL_THEN_EXE_AFTER_CLKCHANGE     0x3cc
+#define EMC_UNSTALL_RW_AFTER_CLKCHANGE         0x3d0
+#define EMC_SEL_DPD_CTRL                       0x3d8
+#define EMC_PRE_REFRESH_REQ_CNT                        0x3dc
+#define EMC_DYN_SELF_REF_CONTROL               0x3e0
+#define EMC_TXSRDLL                            0x3e4
+
+#define EMC_STATUS_TIMING_UPDATE_STALLED       BIT(23)
+
+#define EMC_MODE_SET_DLL_RESET                 BIT(8)
+#define EMC_MODE_SET_LONG_CNT                  BIT(26)
+
+#define EMC_SELF_REF_CMD_ENABLED               BIT(0)
+
+#define DRAM_DEV_SEL_ALL                       (0 << 30)
+#define DRAM_DEV_SEL_0                         (2 << 30)
+#define DRAM_DEV_SEL_1                         (1 << 30)
+#define DRAM_BROADCAST(num) \
+       ((num) > 1 ? DRAM_DEV_SEL_ALL : DRAM_DEV_SEL_0)
+
+#define EMC_ZQ_CAL_CMD                         BIT(0)
+#define EMC_ZQ_CAL_LONG                                BIT(4)
+#define EMC_ZQ_CAL_LONG_CMD_DEV0 \
+       (DRAM_DEV_SEL_0 | EMC_ZQ_CAL_LONG | EMC_ZQ_CAL_CMD)
+#define EMC_ZQ_CAL_LONG_CMD_DEV1 \
+       (DRAM_DEV_SEL_1 | EMC_ZQ_CAL_LONG | EMC_ZQ_CAL_CMD)
+
+#define EMC_DBG_READ_MUX_ASSEMBLY              BIT(0)
+#define EMC_DBG_WRITE_MUX_ACTIVE               BIT(1)
+#define EMC_DBG_FORCE_UPDATE                   BIT(2)
+#define EMC_DBG_CFG_PRIORITY                   BIT(24)
+
+#define EMC_CFG5_QUSE_MODE_SHIFT               13
+#define EMC_CFG5_QUSE_MODE_MASK                        (7 << EMC_CFG5_QUSE_MODE_SHIFT)
+
+#define EMC_CFG5_QUSE_MODE_INTERNAL_LPBK       2
+#define EMC_CFG5_QUSE_MODE_PULSE_INTERN                3
+
+#define EMC_SEL_DPD_CTRL_QUSE_DPD_ENABLE       BIT(9)
+
+#define EMC_XM2COMPPADCTRL_VREF_CAL_ENABLE     BIT(10)
+
+#define EMC_XM2QUSEPADCTRL_IVREF_ENABLE                BIT(4)
+
+#define EMC_XM2DQSPADCTRL2_VREF_ENABLE         BIT(5)
+#define EMC_XM2DQSPADCTRL3_VREF_ENABLE         BIT(5)
+
+#define EMC_AUTO_CAL_STATUS_ACTIVE             BIT(31)
+
+#define        EMC_FBIO_CFG5_DRAM_TYPE_MASK            0x3
+
+#define EMC_MRS_WAIT_CNT_SHORT_WAIT_MASK       0x3ff
+#define EMC_MRS_WAIT_CNT_LONG_WAIT_SHIFT       16
+#define EMC_MRS_WAIT_CNT_LONG_WAIT_MASK \
+       (0x3ff << EMC_MRS_WAIT_CNT_LONG_WAIT_SHIFT)
+
+#define EMC_REFCTRL_DEV_SEL_MASK               0x3
+#define EMC_REFCTRL_ENABLE                     BIT(31)
+#define EMC_REFCTRL_ENABLE_ALL(num) \
+       (((num) > 1 ? 0 : 2) | EMC_REFCTRL_ENABLE)
+#define EMC_REFCTRL_DISABLE_ALL(num)           ((num) > 1 ? 0 : 2)
+
+#define EMC_CFG_PERIODIC_QRST                  BIT(21)
+#define EMC_CFG_DYN_SREF_ENABLE                        BIT(28)
+
+#define EMC_CLKCHANGE_REQ_ENABLE               BIT(0)
+#define EMC_CLKCHANGE_PD_ENABLE                        BIT(1)
+#define EMC_CLKCHANGE_SR_ENABLE                        BIT(2)
+
+#define EMC_TIMING_UPDATE                      BIT(0)
+
+#define EMC_REFRESH_OVERFLOW_INT               BIT(3)
+#define EMC_CLKCHANGE_COMPLETE_INT             BIT(4)
+
+enum emc_dram_type {
+       DRAM_TYPE_DDR3,
+       DRAM_TYPE_DDR1,
+       DRAM_TYPE_LPDDR2,
+       DRAM_TYPE_DDR2,
+};
+
+enum emc_dll_change {
+       DLL_CHANGE_NONE,
+       DLL_CHANGE_ON,
+       DLL_CHANGE_OFF
+};
+
+static const u16 emc_timing_registers[] = {
+       [0] = EMC_RC,
+       [1] = EMC_RFC,
+       [2] = EMC_RAS,
+       [3] = EMC_RP,
+       [4] = EMC_R2W,
+       [5] = EMC_W2R,
+       [6] = EMC_R2P,
+       [7] = EMC_W2P,
+       [8] = EMC_RD_RCD,
+       [9] = EMC_WR_RCD,
+       [10] = EMC_RRD,
+       [11] = EMC_REXT,
+       [12] = EMC_WEXT,
+       [13] = EMC_WDV,
+       [14] = EMC_QUSE,
+       [15] = EMC_QRST,
+       [16] = EMC_QSAFE,
+       [17] = EMC_RDV,
+       [18] = EMC_REFRESH,
+       [19] = EMC_BURST_REFRESH_NUM,
+       [20] = EMC_PRE_REFRESH_REQ_CNT,
+       [21] = EMC_PDEX2WR,
+       [22] = EMC_PDEX2RD,
+       [23] = EMC_PCHG2PDEN,
+       [24] = EMC_ACT2PDEN,
+       [25] = EMC_AR2PDEN,
+       [26] = EMC_RW2PDEN,
+       [27] = EMC_TXSR,
+       [28] = EMC_TXSRDLL,
+       [29] = EMC_TCKE,
+       [30] = EMC_TFAW,
+       [31] = EMC_TRPAB,
+       [32] = EMC_TCLKSTABLE,
+       [33] = EMC_TCLKSTOP,
+       [34] = EMC_TREFBW,
+       [35] = EMC_QUSE_EXTRA,
+       [36] = EMC_FBIO_CFG6,
+       [37] = EMC_ODT_WRITE,
+       [38] = EMC_ODT_READ,
+       [39] = EMC_FBIO_CFG5,
+       [40] = EMC_CFG_DIG_DLL,
+       [41] = EMC_CFG_DIG_DLL_PERIOD,
+       [42] = EMC_DLL_XFORM_DQS0,
+       [43] = EMC_DLL_XFORM_DQS1,
+       [44] = EMC_DLL_XFORM_DQS2,
+       [45] = EMC_DLL_XFORM_DQS3,
+       [46] = EMC_DLL_XFORM_DQS4,
+       [47] = EMC_DLL_XFORM_DQS5,
+       [48] = EMC_DLL_XFORM_DQS6,
+       [49] = EMC_DLL_XFORM_DQS7,
+       [50] = EMC_DLL_XFORM_QUSE0,
+       [51] = EMC_DLL_XFORM_QUSE1,
+       [52] = EMC_DLL_XFORM_QUSE2,
+       [53] = EMC_DLL_XFORM_QUSE3,
+       [54] = EMC_DLL_XFORM_QUSE4,
+       [55] = EMC_DLL_XFORM_QUSE5,
+       [56] = EMC_DLL_XFORM_QUSE6,
+       [57] = EMC_DLL_XFORM_QUSE7,
+       [58] = EMC_DLI_TRIM_TXDQS0,
+       [59] = EMC_DLI_TRIM_TXDQS1,
+       [60] = EMC_DLI_TRIM_TXDQS2,
+       [61] = EMC_DLI_TRIM_TXDQS3,
+       [62] = EMC_DLI_TRIM_TXDQS4,
+       [63] = EMC_DLI_TRIM_TXDQS5,
+       [64] = EMC_DLI_TRIM_TXDQS6,
+       [65] = EMC_DLI_TRIM_TXDQS7,
+       [66] = EMC_DLL_XFORM_DQ0,
+       [67] = EMC_DLL_XFORM_DQ1,
+       [68] = EMC_DLL_XFORM_DQ2,
+       [69] = EMC_DLL_XFORM_DQ3,
+       [70] = EMC_XM2CMDPADCTRL,
+       [71] = EMC_XM2DQSPADCTRL2,
+       [72] = EMC_XM2DQPADCTRL2,
+       [73] = EMC_XM2CLKPADCTRL,
+       [74] = EMC_XM2COMPPADCTRL,
+       [75] = EMC_XM2VTTGENPADCTRL,
+       [76] = EMC_XM2VTTGENPADCTRL2,
+       [77] = EMC_XM2QUSEPADCTRL,
+       [78] = EMC_XM2DQSPADCTRL3,
+       [79] = EMC_CTT_TERM_CTRL,
+       [80] = EMC_ZCAL_INTERVAL,
+       [81] = EMC_ZCAL_WAIT_CNT,
+       [82] = EMC_MRS_WAIT_CNT,
+       [83] = EMC_AUTO_CAL_CONFIG,
+       [84] = EMC_CTT,
+       [85] = EMC_CTT_DURATION,
+       [86] = EMC_DYN_SELF_REF_CONTROL,
+       [87] = EMC_FBIO_SPARE,
+       [88] = EMC_CFG_RSV,
+};
+
+struct emc_timing {
+       unsigned long rate;
+
+       u32 data[ARRAY_SIZE(emc_timing_registers)];
+
+       u32 emc_auto_cal_interval;
+       u32 emc_mode_1;
+       u32 emc_mode_2;
+       u32 emc_mode_reset;
+       u32 emc_zcal_cnt_long;
+       bool emc_cfg_periodic_qrst;
+       bool emc_cfg_dyn_self_ref;
+};
+
+struct tegra_emc {
+       struct device *dev;
+       struct tegra_mc *mc;
+       struct completion clk_handshake_complete;
+       struct notifier_block clk_nb;
+       struct clk *clk;
+       void __iomem *regs;
+       unsigned int irq;
+
+       struct emc_timing *timings;
+       unsigned int num_timings;
+
+       u32 mc_override;
+       u32 emc_cfg;
+
+       u32 emc_mode_1;
+       u32 emc_mode_2;
+       u32 emc_mode_reset;
+
+       bool vref_cal_toggle : 1;
+       bool zcal_long : 1;
+       bool dll_on : 1;
+       bool prepared : 1;
+       bool bad_state : 1;
+};
+
+static irqreturn_t tegra_emc_isr(int irq, void *data)
+{
+       struct tegra_emc *emc = data;
+       u32 intmask = EMC_REFRESH_OVERFLOW_INT | EMC_CLKCHANGE_COMPLETE_INT;
+       u32 status;
+
+       status = readl_relaxed(emc->regs + EMC_INTSTATUS) & intmask;
+       if (!status)
+               return IRQ_NONE;
+
+       /* notify about EMC-CAR handshake completion */
+       if (status & EMC_CLKCHANGE_COMPLETE_INT)
+               complete(&emc->clk_handshake_complete);
+
+       /* notify about HW problem */
+       if (status & EMC_REFRESH_OVERFLOW_INT)
+               dev_err_ratelimited(emc->dev,
+                                   "refresh request overflow timeout\n");
+
+       /* clear interrupts */
+       writel_relaxed(status, emc->regs + EMC_INTSTATUS);
+
+       return IRQ_HANDLED;
+}
+
+static struct emc_timing *emc_find_timing(struct tegra_emc *emc,
+                                         unsigned long rate)
+{
+       struct emc_timing *timing = NULL;
+       unsigned int i;
+
+       for (i = 0; i < emc->num_timings; i++) {
+               if (emc->timings[i].rate >= rate) {
+                       timing = &emc->timings[i];
+                       break;
+               }
+       }
+
+       if (!timing) {
+               dev_err(emc->dev, "no timing for rate %lu\n", rate);
+               return NULL;
+       }
+
+       return timing;
+}
+
+static bool emc_dqs_preset(struct tegra_emc *emc, struct emc_timing *timing,
+                          bool *schmitt_to_vref)
+{
+       bool preset = false;
+       u32 val;
+
+       if (timing->data[71] & EMC_XM2DQSPADCTRL2_VREF_ENABLE) {
+               val = readl_relaxed(emc->regs + EMC_XM2DQSPADCTRL2);
+
+               if (!(val & EMC_XM2DQSPADCTRL2_VREF_ENABLE)) {
+                       val |= EMC_XM2DQSPADCTRL2_VREF_ENABLE;
+                       writel_relaxed(val, emc->regs + EMC_XM2DQSPADCTRL2);
+
+                       preset = true;
+               }
+       }
+
+       if (timing->data[78] & EMC_XM2DQSPADCTRL3_VREF_ENABLE) {
+               val = readl_relaxed(emc->regs + EMC_XM2DQSPADCTRL3);
+
+               if (!(val & EMC_XM2DQSPADCTRL3_VREF_ENABLE)) {
+                       val |= EMC_XM2DQSPADCTRL3_VREF_ENABLE;
+                       writel_relaxed(val, emc->regs + EMC_XM2DQSPADCTRL3);
+
+                       preset = true;
+               }
+       }
+
+       if (timing->data[77] & EMC_XM2QUSEPADCTRL_IVREF_ENABLE) {
+               val = readl_relaxed(emc->regs + EMC_XM2QUSEPADCTRL);
+
+               if (!(val & EMC_XM2QUSEPADCTRL_IVREF_ENABLE)) {
+                       val |= EMC_XM2QUSEPADCTRL_IVREF_ENABLE;
+                       writel_relaxed(val, emc->regs + EMC_XM2QUSEPADCTRL);
+
+                       *schmitt_to_vref = true;
+                       preset = true;
+               }
+       }
+
+       return preset;
+}
+
+static int emc_seq_update_timing(struct tegra_emc *emc)
+{
+       u32 val;
+       int err;
+
+       writel_relaxed(EMC_TIMING_UPDATE, emc->regs + EMC_TIMING_CONTROL);
+
+       err = readl_relaxed_poll_timeout_atomic(emc->regs + EMC_STATUS, val,
+                               !(val & EMC_STATUS_TIMING_UPDATE_STALLED),
+                               1, 200);
+       if (err) {
+               dev_err(emc->dev, "failed to update timing: %d\n", err);
+               return err;
+       }
+
+       return 0;
+}
+
+static int emc_prepare_mc_clk_cfg(struct tegra_emc *emc, unsigned long rate)
+{
+       struct tegra_mc *mc = emc->mc;
+       unsigned int misc0_index = 16;
+       unsigned int i;
+       bool same;
+
+       for (i = 0; i < mc->num_timings; i++) {
+               if (mc->timings[i].rate != rate)
+                       continue;
+
+               if (mc->timings[i].emem_data[misc0_index] & BIT(27))
+                       same = true;
+               else
+                       same = false;
+
+               return tegra20_clk_prepare_emc_mc_same_freq(emc->clk, same);
+       }
+
+       return -EINVAL;
+}
+
+static int emc_prepare_timing_change(struct tegra_emc *emc, unsigned long rate)
+{
+       struct emc_timing *timing = emc_find_timing(emc, rate);
+       enum emc_dll_change dll_change;
+       enum emc_dram_type dram_type;
+       bool schmitt_to_vref = false;
+       unsigned int pre_wait = 0;
+       bool qrst_used = false;
+       unsigned int dram_num;
+       unsigned int i;
+       u32 fbio_cfg5;
+       u32 emc_dbg;
+       u32 val;
+       int err;
+
+       if (!timing || emc->bad_state)
+               return -EINVAL;
+
+       dev_dbg(emc->dev, "%s: using timing rate %lu for requested rate %lu\n",
+               __func__, timing->rate, rate);
+
+       emc->bad_state = true;
+
+       err = emc_prepare_mc_clk_cfg(emc, rate);
+       if (err) {
+               dev_err(emc->dev, "mc clock preparation failed: %d\n", err);
+               return err;
+       }
+
+       emc->vref_cal_toggle = false;
+       emc->mc_override = mc_readl(emc->mc, MC_EMEM_ARB_OVERRIDE);
+       emc->emc_cfg = readl_relaxed(emc->regs + EMC_CFG);
+       emc_dbg = readl_relaxed(emc->regs + EMC_DBG);
+
+       if (emc->dll_on == !!(timing->emc_mode_1 & 0x1))
+               dll_change = DLL_CHANGE_NONE;
+       else if (timing->emc_mode_1 & 0x1)
+               dll_change = DLL_CHANGE_ON;
+       else
+               dll_change = DLL_CHANGE_OFF;
+
+       emc->dll_on = !!(timing->emc_mode_1 & 0x1);
+
+       if (timing->data[80] && !readl_relaxed(emc->regs + EMC_ZCAL_INTERVAL))
+               emc->zcal_long = true;
+       else
+               emc->zcal_long = false;
+
+       fbio_cfg5 = readl_relaxed(emc->regs + EMC_FBIO_CFG5);
+       dram_type = fbio_cfg5 & EMC_FBIO_CFG5_DRAM_TYPE_MASK;
+
+       dram_num = tegra_mc_get_emem_device_count(emc->mc);
+
+       /* disable dynamic self-refresh */
+       if (emc->emc_cfg & EMC_CFG_DYN_SREF_ENABLE) {
+               emc->emc_cfg &= ~EMC_CFG_DYN_SREF_ENABLE;
+               writel_relaxed(emc->emc_cfg, emc->regs + EMC_CFG);
+
+               pre_wait = 5;
+       }
+
+       /* update MC arbiter settings */
+       val = mc_readl(emc->mc, MC_EMEM_ARB_OUTSTANDING_REQ);
+       if (!(val & MC_EMEM_ARB_OUTSTANDING_REQ_HOLDOFF_OVERRIDE) ||
+           ((val & MC_EMEM_ARB_OUTSTANDING_REQ_MAX_MASK) > 0x50)) {
+
+               val = MC_EMEM_ARB_OUTSTANDING_REQ_LIMIT_ENABLE |
+                     MC_EMEM_ARB_OUTSTANDING_REQ_HOLDOFF_OVERRIDE | 0x50;
+               mc_writel(emc->mc, val, MC_EMEM_ARB_OUTSTANDING_REQ);
+               mc_writel(emc->mc, MC_TIMING_UPDATE, MC_TIMING_CONTROL);
+       }
+
+       if (emc->mc_override & MC_EMEM_ARB_OVERRIDE_EACK_MASK)
+               mc_writel(emc->mc,
+                         emc->mc_override & ~MC_EMEM_ARB_OVERRIDE_EACK_MASK,
+                         MC_EMEM_ARB_OVERRIDE);
+
+       /* check DQ/DQS VREF delay */
+       if (emc_dqs_preset(emc, timing, &schmitt_to_vref)) {
+               if (pre_wait < 3)
+                       pre_wait = 3;
+       }
+
+       if (pre_wait) {
+               err = emc_seq_update_timing(emc);
+               if (err)
+                       return err;
+
+               udelay(pre_wait);
+       }
+
+       /* disable auto-calibration if VREF mode is switching */
+       if (timing->emc_auto_cal_interval) {
+               val = readl_relaxed(emc->regs + EMC_XM2COMPPADCTRL);
+               val ^= timing->data[74];
+
+               if (val & EMC_XM2COMPPADCTRL_VREF_CAL_ENABLE) {
+                       writel_relaxed(0, emc->regs + EMC_AUTO_CAL_INTERVAL);
+
+                       err = readl_relaxed_poll_timeout_atomic(
+                               emc->regs + EMC_AUTO_CAL_STATUS, val,
+                               !(val & EMC_AUTO_CAL_STATUS_ACTIVE), 1, 300);
+                       if (err) {
+                               dev_err(emc->dev,
+                                       "failed to disable auto-cal: %d\n",
+                                       err);
+                               return err;
+                       }
+
+                       emc->vref_cal_toggle = true;
+               }
+       }
+
+       /* program shadow registers */
+       for (i = 0; i < ARRAY_SIZE(timing->data); i++) {
+               /* EMC_XM2CLKPADCTRL should be programmed separately */
+               if (i != 73)
+                       writel_relaxed(timing->data[i],
+                                      emc->regs + emc_timing_registers[i]);
+       }
+
+       err = tegra_mc_write_emem_configuration(emc->mc, timing->rate);
+       if (err)
+               return err;
+
+       /* DDR3: predict MRS long wait count */
+       if (dram_type == DRAM_TYPE_DDR3 && dll_change == DLL_CHANGE_ON) {
+               u32 cnt = 512;
+
+               if (emc->zcal_long)
+                       cnt -= dram_num * 256;
+
+               val = timing->data[82] & EMC_MRS_WAIT_CNT_SHORT_WAIT_MASK;
+               if (cnt < val)
+                       cnt = val;
+
+               val = timing->data[82] & ~EMC_MRS_WAIT_CNT_LONG_WAIT_MASK;
+               val |= (cnt << EMC_MRS_WAIT_CNT_LONG_WAIT_SHIFT) &
+                       EMC_MRS_WAIT_CNT_LONG_WAIT_MASK;
+
+               writel_relaxed(val, emc->regs + EMC_MRS_WAIT_CNT);
+       }
+
+       /* disable interrupt since read access is prohibited after stalling */
+       disable_irq(emc->irq);
+
+       /* this read also completes the writes */
+       val = readl_relaxed(emc->regs + EMC_SEL_DPD_CTRL);
+
+       if (!(val & EMC_SEL_DPD_CTRL_QUSE_DPD_ENABLE) && schmitt_to_vref) {
+               u32 cur_mode, new_mode;
+
+               cur_mode = fbio_cfg5 & EMC_CFG5_QUSE_MODE_MASK;
+               cur_mode >>= EMC_CFG5_QUSE_MODE_SHIFT;
+
+               new_mode = timing->data[39] & EMC_CFG5_QUSE_MODE_MASK;
+               new_mode >>= EMC_CFG5_QUSE_MODE_SHIFT;
+
+               if ((cur_mode != EMC_CFG5_QUSE_MODE_PULSE_INTERN &&
+                    cur_mode != EMC_CFG5_QUSE_MODE_INTERNAL_LPBK) ||
+                   (new_mode != EMC_CFG5_QUSE_MODE_PULSE_INTERN &&
+                    new_mode != EMC_CFG5_QUSE_MODE_INTERNAL_LPBK))
+                       qrst_used = true;
+       }
+
+       /* flow control marker 1 */
+       writel_relaxed(0x1, emc->regs + EMC_STALL_THEN_EXE_BEFORE_CLKCHANGE);
+
+       /* enable periodic reset */
+       if (qrst_used) {
+               writel_relaxed(emc_dbg | EMC_DBG_WRITE_MUX_ACTIVE,
+                              emc->regs + EMC_DBG);
+               writel_relaxed(emc->emc_cfg | EMC_CFG_PERIODIC_QRST,
+                              emc->regs + EMC_CFG);
+               writel_relaxed(emc_dbg, emc->regs + EMC_DBG);
+       }
+
+       /* disable auto-refresh to save time after clock change */
+       writel_relaxed(EMC_REFCTRL_DISABLE_ALL(dram_num),
+                      emc->regs + EMC_REFCTRL);
+
+       /* turn off DLL and enter self-refresh on DDR3 */
+       if (dram_type == DRAM_TYPE_DDR3) {
+               if (dll_change == DLL_CHANGE_OFF)
+                       writel_relaxed(timing->emc_mode_1,
+                                      emc->regs + EMC_EMRS);
+
+               writel_relaxed(DRAM_BROADCAST(dram_num) |
+                              EMC_SELF_REF_CMD_ENABLED,
+                              emc->regs + EMC_SELF_REF);
+       }
+
+       /* flow control marker 2 */
+       writel_relaxed(0x1, emc->regs + EMC_STALL_THEN_EXE_AFTER_CLKCHANGE);
+
+       /* enable write-active MUX, update unshadowed pad control */
+       writel_relaxed(emc_dbg | EMC_DBG_WRITE_MUX_ACTIVE, emc->regs + EMC_DBG);
+       writel_relaxed(timing->data[73], emc->regs + EMC_XM2CLKPADCTRL);
+
+       /* restore periodic QRST and disable write-active MUX */
+       val = !!(emc->emc_cfg & EMC_CFG_PERIODIC_QRST);
+       if (qrst_used || timing->emc_cfg_periodic_qrst != val) {
+               if (timing->emc_cfg_periodic_qrst)
+                       emc->emc_cfg |= EMC_CFG_PERIODIC_QRST;
+               else
+                       emc->emc_cfg &= ~EMC_CFG_PERIODIC_QRST;
+
+               writel_relaxed(emc->emc_cfg, emc->regs + EMC_CFG);
+       }
+       writel_relaxed(emc_dbg, emc->regs + EMC_DBG);
+
+       /* exit self-refresh on DDR3 */
+       if (dram_type == DRAM_TYPE_DDR3)
+               writel_relaxed(DRAM_BROADCAST(dram_num),
+                              emc->regs + EMC_SELF_REF);
+
+       /* set DRAM-mode registers */
+       if (dram_type == DRAM_TYPE_DDR3) {
+               if (timing->emc_mode_1 != emc->emc_mode_1)
+                       writel_relaxed(timing->emc_mode_1,
+                                      emc->regs + EMC_EMRS);
+
+               if (timing->emc_mode_2 != emc->emc_mode_2)
+                       writel_relaxed(timing->emc_mode_2,
+                                      emc->regs + EMC_EMRS);
+
+               if (timing->emc_mode_reset != emc->emc_mode_reset ||
+                   dll_change == DLL_CHANGE_ON) {
+                       val = timing->emc_mode_reset;
+                       if (dll_change == DLL_CHANGE_ON) {
+                               val |= EMC_MODE_SET_DLL_RESET;
+                               val |= EMC_MODE_SET_LONG_CNT;
+                       } else {
+                               val &= ~EMC_MODE_SET_DLL_RESET;
+                       }
+                       writel_relaxed(val, emc->regs + EMC_MRS);
+               }
+       } else {
+               if (timing->emc_mode_2 != emc->emc_mode_2)
+                       writel_relaxed(timing->emc_mode_2,
+                                      emc->regs + EMC_MRW);
+
+               if (timing->emc_mode_1 != emc->emc_mode_1)
+                       writel_relaxed(timing->emc_mode_1,
+                                      emc->regs + EMC_MRW);
+       }
+
+       emc->emc_mode_1 = timing->emc_mode_1;
+       emc->emc_mode_2 = timing->emc_mode_2;
+       emc->emc_mode_reset = timing->emc_mode_reset;
+
+       /* issue ZCAL command if turning ZCAL on */
+       if (emc->zcal_long) {
+               writel_relaxed(EMC_ZQ_CAL_LONG_CMD_DEV0,
+                              emc->regs + EMC_ZQ_CAL);
+
+               if (dram_num > 1)
+                       writel_relaxed(EMC_ZQ_CAL_LONG_CMD_DEV1,
+                                      emc->regs + EMC_ZQ_CAL);
+       }
+
+       /* re-enable auto-refresh */
+       writel_relaxed(EMC_REFCTRL_ENABLE_ALL(dram_num),
+                      emc->regs + EMC_REFCTRL);
+
+       /* flow control marker 3 */
+       writel_relaxed(0x1, emc->regs + EMC_UNSTALL_RW_AFTER_CLKCHANGE);
+
+       reinit_completion(&emc->clk_handshake_complete);
+
+       /* interrupt can be re-enabled now */
+       enable_irq(emc->irq);
+
+       emc->bad_state = false;
+       emc->prepared = true;
+
+       return 0;
+}
+
+static int emc_complete_timing_change(struct tegra_emc *emc,
+                                     unsigned long rate)
+{
+       struct emc_timing *timing = emc_find_timing(emc, rate);
+       unsigned long timeout;
+       int ret;
+
+       timeout = wait_for_completion_timeout(&emc->clk_handshake_complete,
+                                             msecs_to_jiffies(100));
+       if (timeout == 0) {
+               dev_err(emc->dev, "emc-car handshake failed\n");
+               emc->bad_state = true;
+               return -EIO;
+       }
+
+       /* restore auto-calibration */
+       if (emc->vref_cal_toggle)
+               writel_relaxed(timing->emc_auto_cal_interval,
+                              emc->regs + EMC_AUTO_CAL_INTERVAL);
+
+       /* restore dynamic self-refresh */
+       if (timing->emc_cfg_dyn_self_ref) {
+               emc->emc_cfg |= EMC_CFG_DYN_SREF_ENABLE;
+               writel_relaxed(emc->emc_cfg, emc->regs + EMC_CFG);
+       }
+
+       /* set number of clocks to wait after each ZQ command */
+       if (emc->zcal_long)
+               writel_relaxed(timing->emc_zcal_cnt_long,
+                              emc->regs + EMC_ZCAL_WAIT_CNT);
+
+       udelay(2);
+       /* update restored timing */
+       ret = emc_seq_update_timing(emc);
+       if (ret)
+               emc->bad_state = true;
+
+       /* restore early ACK */
+       mc_writel(emc->mc, emc->mc_override, MC_EMEM_ARB_OVERRIDE);
+
+       emc->prepared = false;
+
+       return ret;
+}
+
+static int emc_unprepare_timing_change(struct tegra_emc *emc,
+                                      unsigned long rate)
+{
+       if (emc->prepared && !emc->bad_state) {
+               /* shouldn't ever happen in practice */
+               dev_err(emc->dev, "timing configuration can't be reverted\n");
+               emc->bad_state = true;
+       }
+
+       return 0;
+}
+
+static int emc_clk_change_notify(struct notifier_block *nb,
+                                unsigned long msg, void *data)
+{
+       struct tegra_emc *emc = container_of(nb, struct tegra_emc, clk_nb);
+       struct clk_notifier_data *cnd = data;
+       int err;
+
+       switch (msg) {
+       case PRE_RATE_CHANGE:
+               err = emc_prepare_timing_change(emc, cnd->new_rate);
+               break;
+
+       case ABORT_RATE_CHANGE:
+               err = emc_unprepare_timing_change(emc, cnd->old_rate);
+               break;
+
+       case POST_RATE_CHANGE:
+               err = emc_complete_timing_change(emc, cnd->new_rate);
+               break;
+
+       default:
+               return NOTIFY_DONE;
+       }
+
+       return notifier_from_errno(err);
+}
+
+static int load_one_timing_from_dt(struct tegra_emc *emc,
+                                  struct emc_timing *timing,
+                                  struct device_node *node)
+{
+       u32 value;
+       int err;
+
+       err = of_property_read_u32(node, "clock-frequency", &value);
+       if (err) {
+               dev_err(emc->dev, "timing %pOF: failed to read rate: %d\n",
+                       node, err);
+               return err;
+       }
+
+       timing->rate = value;
+
+       err = of_property_read_u32_array(node, "nvidia,emc-configuration",
+                                        timing->data,
+                                        ARRAY_SIZE(emc_timing_registers));
+       if (err) {
+               dev_err(emc->dev,
+                       "timing %pOF: failed to read emc timing data: %d\n",
+                       node, err);
+               return err;
+       }
+
+#define EMC_READ_BOOL(prop, dtprop) \
+       timing->prop = of_property_read_bool(node, dtprop);
+
+#define EMC_READ_U32(prop, dtprop) \
+       err = of_property_read_u32(node, dtprop, &timing->prop); \
+       if (err) { \
+               dev_err(emc->dev, \
+                       "timing %pOFn: failed to read " #prop ": %d\n", \
+                       node, err); \
+               return err; \
+       }
+
+       EMC_READ_U32(emc_auto_cal_interval, "nvidia,emc-auto-cal-interval")
+       EMC_READ_U32(emc_mode_1, "nvidia,emc-mode-1")
+       EMC_READ_U32(emc_mode_2, "nvidia,emc-mode-2")
+       EMC_READ_U32(emc_mode_reset, "nvidia,emc-mode-reset")
+       EMC_READ_U32(emc_zcal_cnt_long, "nvidia,emc-zcal-cnt-long")
+       EMC_READ_BOOL(emc_cfg_dyn_self_ref, "nvidia,emc-cfg-dyn-self-ref")
+       EMC_READ_BOOL(emc_cfg_periodic_qrst, "nvidia,emc-cfg-periodic-qrst")
+
+#undef EMC_READ_U32
+#undef EMC_READ_BOOL
+
+       dev_dbg(emc->dev, "%s: %pOF: rate %lu\n", __func__, node, timing->rate);
+
+       return 0;
+}
+
+static int cmp_timings(const void *_a, const void *_b)
+{
+       const struct emc_timing *a = _a;
+       const struct emc_timing *b = _b;
+
+       if (a->rate < b->rate)
+               return -1;
+
+       if (a->rate > b->rate)
+               return 1;
+
+       return 0;
+}
+
+static int emc_check_mc_timings(struct tegra_emc *emc)
+{
+       struct tegra_mc *mc = emc->mc;
+       unsigned int i;
+
+       if (emc->num_timings != mc->num_timings) {
+               dev_err(emc->dev, "emc/mc timings number mismatch: %u %u\n",
+                       emc->num_timings, mc->num_timings);
+               return -EINVAL;
+       }
+
+       for (i = 0; i < mc->num_timings; i++) {
+               if (emc->timings[i].rate != mc->timings[i].rate) {
+                       dev_err(emc->dev,
+                               "emc/mc timing rate mismatch: %lu %lu\n",
+                               emc->timings[i].rate, mc->timings[i].rate);
+                       return -EINVAL;
+               }
+       }
+
+       return 0;
+}
+
+static int emc_load_timings_from_dt(struct tegra_emc *emc,
+                                   struct device_node *node)
+{
+       struct device_node *child;
+       struct emc_timing *timing;
+       int child_count;
+       int err;
+
+       child_count = of_get_child_count(node);
+       if (!child_count) {
+               dev_err(emc->dev, "no memory timings in: %pOF\n", node);
+               return -EINVAL;
+       }
+
+       emc->timings = devm_kcalloc(emc->dev, child_count, sizeof(*timing),
+                                   GFP_KERNEL);
+       if (!emc->timings)
+               return -ENOMEM;
+
+       emc->num_timings = child_count;
+       timing = emc->timings;
+
+       for_each_child_of_node(node, child) {
+               err = load_one_timing_from_dt(emc, timing++, child);
+               if (err) {
+                       of_node_put(child);
+                       return err;
+               }
+       }
+
+       sort(emc->timings, emc->num_timings, sizeof(*timing), cmp_timings,
+            NULL);
+
+       err = emc_check_mc_timings(emc);
+       if (err)
+               return err;
+
+       dev_info(emc->dev,
+                "got %u timings for RAM code %u (min %luMHz max %luMHz)\n",
+                emc->num_timings,
+                tegra_read_ram_code(),
+                emc->timings[0].rate / 1000000,
+                emc->timings[emc->num_timings - 1].rate / 1000000);
+
+       return 0;
+}
+
+static struct device_node *emc_find_node_by_ram_code(struct device *dev)
+{
+       struct device_node *np;
+       u32 value, ram_code;
+       int err;
+
+       ram_code = tegra_read_ram_code();
+
+       for_each_child_of_node(dev->of_node, np) {
+               err = of_property_read_u32(np, "nvidia,ram-code", &value);
+               if (err || value != ram_code)
+                       continue;
+
+               return np;
+       }
+
+       dev_err(dev, "no memory timings for RAM code %u found in device-tree\n",
+               ram_code);
+
+       return NULL;
+}
+
+static int emc_setup_hw(struct tegra_emc *emc)
+{
+       u32 intmask = EMC_REFRESH_OVERFLOW_INT | EMC_CLKCHANGE_COMPLETE_INT;
+       u32 fbio_cfg5, emc_cfg, emc_dbg;
+       enum emc_dram_type dram_type;
+
+       fbio_cfg5 = readl_relaxed(emc->regs + EMC_FBIO_CFG5);
+       dram_type = fbio_cfg5 & EMC_FBIO_CFG5_DRAM_TYPE_MASK;
+
+       emc_cfg = readl_relaxed(emc->regs + EMC_CFG_2);
+
+       /* enable EMC and CAR to handshake on PLL divider/source changes */
+       emc_cfg |= EMC_CLKCHANGE_REQ_ENABLE;
+
+       /* configure clock change mode accordingly to DRAM type */
+       switch (dram_type) {
+       case DRAM_TYPE_LPDDR2:
+               emc_cfg |= EMC_CLKCHANGE_PD_ENABLE;
+               emc_cfg &= ~EMC_CLKCHANGE_SR_ENABLE;
+               break;
+
+       default:
+               emc_cfg &= ~EMC_CLKCHANGE_SR_ENABLE;
+               emc_cfg &= ~EMC_CLKCHANGE_PD_ENABLE;
+               break;
+       }
+
+       writel_relaxed(emc_cfg, emc->regs + EMC_CFG_2);
+
+       /* initialize interrupt */
+       writel_relaxed(intmask, emc->regs + EMC_INTMASK);
+       writel_relaxed(0xffffffff, emc->regs + EMC_INTSTATUS);
+
+       /* ensure that unwanted debug features are disabled */
+       emc_dbg = readl_relaxed(emc->regs + EMC_DBG);
+       emc_dbg |= EMC_DBG_CFG_PRIORITY;
+       emc_dbg &= ~EMC_DBG_READ_MUX_ASSEMBLY;
+       emc_dbg &= ~EMC_DBG_WRITE_MUX_ACTIVE;
+       emc_dbg &= ~EMC_DBG_FORCE_UPDATE;
+       writel_relaxed(emc_dbg, emc->regs + EMC_DBG);
+
+       return 0;
+}
+
+static long emc_round_rate(unsigned long rate,
+                          unsigned long min_rate,
+                          unsigned long max_rate,
+                          void *arg)
+{
+       struct emc_timing *timing = NULL;
+       struct tegra_emc *emc = arg;
+       unsigned int i;
+
+       min_rate = min(min_rate, emc->timings[emc->num_timings - 1].rate);
+
+       for (i = 0; i < emc->num_timings; i++) {
+               if (emc->timings[i].rate < rate && i != emc->num_timings - 1)
+                       continue;
+
+               if (emc->timings[i].rate > max_rate) {
+                       i = max(i, 1u) - 1;
+
+                       if (emc->timings[i].rate < min_rate)
+                               break;
+               }
+
+               if (emc->timings[i].rate < min_rate)
+                       continue;
+
+               timing = &emc->timings[i];
+               break;
+       }
+
+       if (!timing) {
+               dev_err(emc->dev, "no timing for rate %lu min %lu max %lu\n",
+                       rate, min_rate, max_rate);
+               return -EINVAL;
+       }
+
+       return timing->rate;
+}
+
+static int tegra_emc_probe(struct platform_device *pdev)
+{
+       struct platform_device *mc;
+       struct device_node *np;
+       struct tegra_emc *emc;
+       int err;
+
+       if (of_get_child_count(pdev->dev.of_node) == 0) {
+               dev_info(&pdev->dev,
+                        "device-tree node doesn't have memory timings\n");
+               return 0;
+       }
+
+       np = of_parse_phandle(pdev->dev.of_node, "nvidia,memory-controller", 0);
+       if (!np) {
+               dev_err(&pdev->dev, "could not get memory controller node\n");
+               return -ENOENT;
+       }
+
+       mc = of_find_device_by_node(np);
+       of_node_put(np);
+       if (!mc)
+               return -ENOENT;
+
+       np = emc_find_node_by_ram_code(&pdev->dev);
+       if (!np)
+               return -EINVAL;
+
+       emc = devm_kzalloc(&pdev->dev, sizeof(*emc), GFP_KERNEL);
+       if (!emc) {
+               of_node_put(np);
+               return -ENOMEM;
+       }
+
+       emc->mc = platform_get_drvdata(mc);
+       if (!emc->mc)
+               return -EPROBE_DEFER;
+
+       init_completion(&emc->clk_handshake_complete);
+       emc->clk_nb.notifier_call = emc_clk_change_notify;
+       emc->dev = &pdev->dev;
+
+       err = emc_load_timings_from_dt(emc, np);
+       of_node_put(np);
+       if (err)
+               return err;
+
+       emc->regs = devm_platform_ioremap_resource(pdev, 0);
+       if (IS_ERR(emc->regs))
+               return PTR_ERR(emc->regs);
+
+       err = emc_setup_hw(emc);
+       if (err)
+               return err;
+
+       err = platform_get_irq(pdev, 0);
+       if (err < 0) {
+               dev_err(&pdev->dev, "interrupt not specified: %d\n", err);
+               return err;
+       }
+       emc->irq = err;
+
+       err = devm_request_irq(&pdev->dev, emc->irq, tegra_emc_isr, 0,
+                              dev_name(&pdev->dev), emc);
+       if (err) {
+               dev_err(&pdev->dev, "failed to request irq: %d\n", err);
+               return err;
+       }
+
+       tegra20_clk_set_emc_round_callback(emc_round_rate, emc);
+
+       emc->clk = devm_clk_get(&pdev->dev, "emc");
+       if (IS_ERR(emc->clk)) {
+               err = PTR_ERR(emc->clk);
+               dev_err(&pdev->dev, "failed to get emc clock: %d\n", err);
+               goto unset_cb;
+       }
+
+       err = clk_notifier_register(emc->clk, &emc->clk_nb);
+       if (err) {
+               dev_err(&pdev->dev, "failed to register clk notifier: %d\n",
+                       err);
+               goto unset_cb;
+       }
+
+       platform_set_drvdata(pdev, emc);
+
+       return 0;
+
+unset_cb:
+       tegra20_clk_set_emc_round_callback(NULL, NULL);
+
+       return err;
+}
+
+static int tegra_emc_suspend(struct device *dev)
+{
+       struct tegra_emc *emc = dev_get_drvdata(dev);
+
+       /*
+        * Suspending in a bad state will hang machine. The "prepared" var
+        * shall be always false here unless it's a kernel bug that caused
+        * suspending in a wrong order.
+        */
+       if (WARN_ON(emc->prepared) || emc->bad_state)
+               return -EINVAL;
+
+       emc->bad_state = true;
+
+       return 0;
+}
+
+static int tegra_emc_resume(struct device *dev)
+{
+       struct tegra_emc *emc = dev_get_drvdata(dev);
+
+       emc_setup_hw(emc);
+       emc->bad_state = false;
+
+       return 0;
+}
+
+static const struct dev_pm_ops tegra_emc_pm_ops = {
+       .suspend = tegra_emc_suspend,
+       .resume = tegra_emc_resume,
+};
+
+static const struct of_device_id tegra_emc_of_match[] = {
+       { .compatible = "nvidia,tegra30-emc", },
+       {},
+};
+
+static struct platform_driver tegra_emc_driver = {
+       .probe = tegra_emc_probe,
+       .driver = {
+               .name = "tegra30-emc",
+               .of_match_table = tegra_emc_of_match,
+               .pm = &tegra_emc_pm_ops,
+               .suppress_bind_attrs = true,
+       },
+};
+
+static int __init tegra_emc_init(void)
+{
+       return platform_driver_register(&tegra_emc_driver);
+}
+subsys_initcall(tegra_emc_init);

drivers/memory/tegra/tegra30.c +42 -0

index 14788fc2f9e8..b1226d3f067f 100644
@@ -10,6 +10,46 @@
 
 #include "mc.h"
 
+#define MC_EMEM_ARB_CFG                                0x90
+#define MC_EMEM_ARB_OUTSTANDING_REQ            0x94
+#define MC_EMEM_ARB_TIMING_RCD                 0x98
+#define MC_EMEM_ARB_TIMING_RP                  0x9c
+#define MC_EMEM_ARB_TIMING_RC                  0xa0
+#define MC_EMEM_ARB_TIMING_RAS                 0xa4
+#define MC_EMEM_ARB_TIMING_FAW                 0xa8
+#define MC_EMEM_ARB_TIMING_RRD                 0xac
+#define MC_EMEM_ARB_TIMING_RAP2PRE             0xb0
+#define MC_EMEM_ARB_TIMING_WAP2PRE             0xb4
+#define MC_EMEM_ARB_TIMING_R2R                 0xb8
+#define MC_EMEM_ARB_TIMING_W2W                 0xbc
+#define MC_EMEM_ARB_TIMING_R2W                 0xc0
+#define MC_EMEM_ARB_TIMING_W2R                 0xc4
+#define MC_EMEM_ARB_DA_TURNS                   0xd0
+#define MC_EMEM_ARB_DA_COVERS                  0xd4
+#define MC_EMEM_ARB_MISC0                      0xd8
+#define MC_EMEM_ARB_RING1_THROTTLE             0xe0
+
+static const unsigned long tegra30_mc_emem_regs[] = {
+       MC_EMEM_ARB_CFG,
+       MC_EMEM_ARB_OUTSTANDING_REQ,
+       MC_EMEM_ARB_TIMING_RCD,
+       MC_EMEM_ARB_TIMING_RP,
+       MC_EMEM_ARB_TIMING_RC,
+       MC_EMEM_ARB_TIMING_RAS,
+       MC_EMEM_ARB_TIMING_FAW,
+       MC_EMEM_ARB_TIMING_RRD,
+       MC_EMEM_ARB_TIMING_RAP2PRE,
+       MC_EMEM_ARB_TIMING_WAP2PRE,
+       MC_EMEM_ARB_TIMING_R2R,
+       MC_EMEM_ARB_TIMING_W2W,
+       MC_EMEM_ARB_TIMING_R2W,
+       MC_EMEM_ARB_TIMING_W2R,
+       MC_EMEM_ARB_DA_TURNS,
+       MC_EMEM_ARB_DA_COVERS,
+       MC_EMEM_ARB_MISC0,
+       MC_EMEM_ARB_RING1_THROTTLE,
+};
+
 static const struct tegra_mc_client tegra30_mc_clients[] = {
        {
                .id = 0x00,

@@ -994,6 +1034,8 @@ const struct tegra_mc_soc tegra30_mc_soc = {
        .atom_size = 16,
        .client_id_mask = 0x7f,
        .smmu = &tegra30_smmu_soc,
+       .emem_regs = tegra30_mc_emem_regs,
+       .num_emem_regs = ARRAY_SIZE(tegra30_mc_emem_regs),
        .intmask = MC_INT_INVALID_SMMU_PAGE | MC_INT_SECURITY_VIOLATION |
                   MC_INT_DECERR_EMEM,
        .reset_ops = &tegra_mc_reset_ops_common,

include/soc/tegra/mc.h +1 -2

index 16e2c2fb5f6c..1238e35653d1 100644
@@ -181,7 +181,7 @@ struct tegra_mc {
        spinlock_t lock;
 };
 
-void tegra_mc_write_emem_configuration(struct tegra_mc *mc, unsigned long rate);
+int tegra_mc_write_emem_configuration(struct tegra_mc *mc, unsigned long rate);
 unsigned int tegra_mc_get_emem_device_count(struct tegra_mc *mc);
 
 #endif /* __SOC_TEGRA_MC_H__ */
--

2.22.0

Tested on the Ouya (tegra30).

Tested-by Peter Geis <[email protected]>

Quoted 9 lines · Dmitry Osipenko

Introduce driver for the External Memory Controller (EMC) found on Tegra30 chips, it controls the external DRAM on the board. The purpose of this driver is to program memory timing for external memory on the EMC clock rate change.

Acked-by Peter De Schrijver <[email protected]>

Signed-off-by Dmitry Osipenko <[email protected]>

---

Introduce driver for the External Memory Controller (EMC) found on Tegra30 chips, it controls the external DRAM on the board. The purpose of this driver is to program memory timing for external memory on the EMC clock rate change.

Acked-by Peter De Schrijver <[email protected]>

Signed-off-by Dmitry Osipenko <[email protected]>

---
 drivers/memory/tegra/Kconfig       |   10 +
 drivers/memory/tegra/Makefile      |    1 +
 drivers/memory/tegra/mc.c          |    9 +-
 drivers/memory/tegra/mc.h          |   30 +-
 drivers/memory/tegra/tegra30-emc.c | 1232 ++++++++++++++++++++++++++++
 drivers/memory/tegra/tegra30.c     |   42 +
 include/soc/tegra/mc.h             |    2 +-
 7 files changed, 1311 insertions(+), 15 deletions(-)
 create mode 100644 drivers/memory/tegra/tegra30-emc.c

Quoted 7 lines

Introduce driver for the External Memory Controller (EMC) found on Tegra30 chips, it controls the external DRAM on the board. The purpose of this driver is to program memory timing for external memory on the EMC clock rate change.

Acked-by Peter De Schrijver <[email protected]>

Signed-off-by Dmitry Osipenko <[email protected]>

Quoted 10 lines

---
 drivers/memory/tegra/Kconfig       |   10 +
 drivers/memory/tegra/Makefile      |    1 +
 drivers/memory/tegra/mc.c          |    9 +-
 drivers/memory/tegra/mc.h          |   30 +-
 drivers/memory/tegra/tegra30-emc.c | 1232 ++++++++++++++++++++++++++++
 drivers/memory/tegra/tegra30.c     |   42 +
 include/soc/tegra/mc.h             |    2 +-
 7 files changed, 1311 insertions(+), 15 deletions(-)
 create mode 100644 drivers/memory/tegra/tegra30-emc.c

This patch is causing the following crash on Tegra30-cardhu-a04 on entering suspend ...

[   58.320034] 8<--- cut here ---
[   58.323166] Unable to handle kernel NULL pointer dereference at virtual address 0000004c
[   58.331262] pgd = 62bca252
[   58.334028] [0000004c] *pgd=00000000
[   58.337615] Internal error: Oops: 5 [#1] PREEMPT SMP ARM
[   58.342927] Modules linked in: brcmfmac brcmutil
[   58.347559] CPU: 1 PID: 689 Comm: rtcwake Tainted: G        W         5.4.0-rc6-next-20191108 #1
[   58.356343] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[   58.362620] PC is at tegra_emc_suspend+0x4/0x50
[   58.367161] LR is at dpm_run_callback+0x38/0x1d4
[   58.371778] pc : [<c07a9ce0>]    lr : [<c05c2a2c>]    psr: a0000113
[   58.378042] sp : ee3afd98  ip : 00000000  fp : 00000000
[   58.383265] r10: c1077e1c  r9 : ef254c54  r8 : c1077de4
[   58.388488] r7 : c0d1d3e4  r6 : ef254c10  r5 : 00000002  r4 : c05b5e28
[   58.395013] r3 : c07a9cdc  r2 : 00000000  r1 : c0bbadc4  r0 : ef254c10
[   58.401542] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[   58.408677] Control: 10c5387d  Table: adcf804a  DAC: 00000051
[   58.414423] Process rtcwake (pid: 689, stack limit = 0x8afb9f6f)
[   58.420429] Stack: (0xee3afd98 to 0xee3b0000)
[   58.424787] fd80:                                                       ef254c10 c110c438
[   58.432969] fda0: 00000000 00000002 c1077de4 ef254c54 c1077e1c c05c30fc 00000000 ef254ca8
[   58.441150] fdc0: ef254c10 c1077de4 c110c438 c1077de4 c05c3560 c1077e1c 00000000 c05c5730
[   58.449330] fde0: 00000002 c10c3648 c10d3df0 00000000 00000002 00000003 c1004e48 00000004
[   58.457511] fe00: ede68100 edf50210 00475228 c05c6030 c10d3dbc 00000000 00000003 c017d16c
[   58.465693] fe20: c10c889c 00000004 ede68100 edf50210 00475228 c0181404 c0cbbba4 ee3afe5c
[   58.473873] fe40: c10c889c e8e19516 ee3afe5c c1004e48 00000000 00000003 c10c889c 00000004
[   58.482054] fe60: ede68100 edf50210 00475228 c017dd18 00000008 e8e19516 00000cc0 00000003
[   58.490234] fe80: c0cbeff4 00000003 c10d3dd0 c017c088 00000004 edf50200 00000000 00000000
[   58.498415] fea0: ede68100 ee3aff78 edf50210 c02fcfe4 00000000 00000000 c1004e48 c02fceec
[   58.506602] fec0: 00476438 ee3aff78 00000000 00476438 00000004 c027a3f0 00000055 00000cc0
[   58.514791] fee0: 00000477 00477000 edcf8010 edcf8010 00000000 00000000 00000000 00000000
[   58.522981] ff00: ffefe1dc eff2b394 00000000 e8e19516 00000001 ee3affb0 ef2bb700 edf2a200
[   58.531166] ff20: 0047743c 00000817 eddd5ad0 e8e19516 edf2a240 00000004 edc53e40 00476438
[   58.539352] ff40: ee3aff78 00000000 00476438 00000004 00475228 c027b99c 00000000 00000000
[   58.547542] ff60: c1004e48 edc53e40 00000000 00000000 edc53e40 c027bc3c 00000000 00000000

[   58.555732] ff80: 00001008 e8e19516 0000006c 00476438 00475228 00000004 c0101204 ee3ae000
[   58.563922] ffa0: 00000004 c0101000 0000006c 00476438 00000004 00476438 00000004 00000000
[   58.572107] ffc0: 0000006c 00476438 00475228 00000004 00000004 00000004 0046278c 00475228
[   58.580294] ffe0: 00000004 be9db9b8 b6eb3c0b b6e3e206 600f0030 00000004 00000000 00000000
[   58.588491] [<c07a9ce0>] (tegra_emc_suspend) from [<c110c438>] (0xc110c438)
[   58.595466] Code: e7c4321f e5c4304c e8bd8010 e5902040 (e5d2304c) 
[   58.601656] ---[ end trace 8d7d1a7fb898a1d0 ]---
[   88.142888] ------------[ cut here ]------------

Hello,

This series introduces driver for the External Memory Controller (EMC) found on Tegra30 chips, it controls the external DRAM on the board. The purpose of this driver is to program memory timing for external memory on the EMC clock rate change. The driver was tested using the ACTMON devfreq driver that performs memory frequency scaling based on memory-usage load.

Changelog:

v10: - Addressed review comments that were made by Rob Herring to v9 by
dropping unnecessary reg descriptions, specifying valid ranges and using boolean type where appropriate in the device-tree patches.

v9: - Fixed memory corruption bug that was uncovered after introducing
some extra optimizations to the devfreq driver that allows CPU to stay longer in the LP2 cpuidle state. The corruption is caused by a very late AUTO-REFRESH re-enabling due to a possible schedule on waiting for clk-change completion, the re-enabling is now a part of "EMC exec-after-clkchange" hardware sequence.

- Added "type: object" to T124 MC YAML, that was missed in v8 by accident.

v8: - Added two new patches:

memory: tegra20-emc: Increase handshake timeout
memory: tegra20-emc: wait_for_completion_timeout() doesn't return error

Turned out that memory-clk handshake may take much more time under some circumstances. The second patch is a minor cleanup. The same changes are also applied to the Terga30 EMC driver addition-patch.

The pattern-properties of YAML bindings gained "type: object", for consistency.

v7: - Addressed review comments that were made by Rob Herring to v6 by
removing old Terga30 Memory Controller binding once it's converted to YAML, by using explicit patterns for the sub-nodes and specifying min/max clock rates in the YAML.

- Two patches that were added in v6 are removed from the series:

clk: tegra20: emc: Add tegra20_clk_emc_on_pllp()
ARM: tegra30: cpuidle: Don't enter LP2 on CPU0 when EMC runs off PLLP

Because the problem with the PLLP is resolved now, turned out it was a bug in the CPU-suspend code.

- The "Introduce Tegra30 EMC driver" patch got a fix for the "Same Freq" bit typo, it's a bit 27 and not 16.

v6: - Tegra124 Memory Controller binding factored out into standalone
binding because it requires to specify MC_EMEM_ARB_MISC1 for EMEM programming, which is not required for Tegra30. This makes the upstream MC registers specification to match downstream exactly, easing porting of boards memory timings configuration to upstream.

- Tegra30/124 Memory Controller binding converted to YAML.

- Tegra30 External Memory Controller binding now is in YAML format.

- Added workaround for hanging during LP2 when EMC runs off PLLP on Tegra30 in this new patches:

clk: tegra20: emc: Add tegra20_clk_emc_on_pllp()
ARM: tegra30: cpuidle: Don't enter LP2 on CPU0 when EMC runs off PLLP

- Added info message to the Tegra20/30 EMC drivers, telling about RAM code and a number of available timings:

memory: tegra20-emc: Print a brief info message about the timings

v5: - Addressed review comments that were made by Thierry Reding to v4 by
adding appropriate copyrights to the source code headers and making Tegra30 EMC driver to use common Tegra20 CLK API directly instead of having a dummy-proxy functions specifically for Tegra30.

- Addressed review comments that were made by Stephen Boyd to v4 by rewording commit message of the "Add custom EMC clock implementation" patch and adding clarifying comment (to that patch as well) which tells why EMC is a critical clock.

- Added suspend-resume to Tegra30 EMC driver to error out if EMC driver is in a "bad state" as it will likely cause a hang on entering suspend.

- Dropped patch "tegra20-emc: Replace clk_get_sys with devm_clk_get" because the replaced clocks are actually should be removed altogether in the "Drop setting EMC rate to max on probe" patch and that was missed by an accident.

- Added "tegra20-emc: Pre-configure debug register" patch which ensures that inappropriate HW debug features are disabled at a probe time. The same change is also made in the "Introduce Tegra30 EMC driver" patch.

- Added ACKs to the patches from Peter De Schrijver that he gave to v4 since all of the v5 changes are actually very minor.

v4: - Addressed review comments that were made by Peter De Schrijver to v3
by adding fence_udelay() after writes in the "Add custom EMC clock implementation" patch.

- Added two new minor patches:

memory: tegra: Ensure timing control debug features are disabled
memory: tegra: Consolidate registers definition into one place

The first one is needed to ensure that EMC driver will work properly regardless of hardware configuration left after boot. The second patch is just a minor code cleanup.

- The "Introduce Tegra30 EMC driver" got also few very minor changes. Now every possible error case is handled, nothing is ignored. The EMC_DBG register is explicitly initialized during probe to be on the safe side.

v3: - Addressed review comments that were made by Stephen Boyd to v2 by
adding explicit typing for the callback variable, by including "clk-provider.h" directly in the code and by dropping __clk_lookup usage where possible.

- Added more patches into this series:

memory: tegra20-emc: Drop setting EMC rate to max on probe
memory: tegra20-emc: Adapt for clock driver changes
memory: tegra20-emc: Include io.h instead of iopoll.h
memory: tegra20-emc: Replace clk_get_sys with devm_clk_get

Initially I was going to include these patches into other patchset, but changed my mind after rearranging things a tad. The "Adapt for clock driver changes" patch is directly related to the clock changes done in the first patch of this series, the rest are minor cleanups that are fine to include here as well.

- Added some more words to the commit message of "Add binding for NVIDIA Tegra30 External Memory Controller" patch, clarifying why common DDR timing device-tree form isn't suitable for Tegra30.

- The Tegra30 EMC driver now explicitly selects the registers access mode (EMC_DBG mux), not relying on the setting left from bootloader.

v2: - Added support for changing MC clock diver configuration based on
Memory Controller (MC) configuration which is part of the memory timing.

- Merged the "Add custom EMC clock implementation" patch into this series because the "Introduce Tegra30 EMC driver" patch directly depends on it. Please note that Tegra20 EMC driver will need to be adapted for the clock changes as well, I'll send out the Tegra20 patches after this series will be applied because of some other dependencies (devfreq) and because the temporary breakage won't be critical (driver will just error out on probe).

- EMC driver now performs MC configuration validation by checking that the number of MC / EMC timings matches and that the timings rate is the same.

- EMC driver now supports timings that want to change the MC clock configuration.

- Other minor prettifying changes of the code.

Dmitry Osipenko (15):
clk: tegra20/30: Add custom EMC clock implementation
memory: tegra20-emc: Drop setting EMC rate to max on probe
memory: tegra20-emc: Adapt for clock driver changes
memory: tegra20-emc: Include io.h instead of iopoll.h
memory: tegra20-emc: Pre-configure debug register
memory: tegra20-emc: Print a brief info message about the timings
memory: tegra20-emc: Increase handshake timeout
memory: tegra20-emc: wait_for_completion_timeout() doesn't return
error

dt-bindings: memory: tegra30: Convert to Tegra124 YAML
dt-bindings: memory: Add binding for NVIDIA Tegra30 Memory Controller
dt-bindings: memory: Add binding for NVIDIA Tegra30 External Memory
Controller
memory: tegra: Introduce Tegra30 EMC driver
memory: tegra: Ensure timing control debug features are disabled
memory: tegra: Consolidate registers definition into common header
ARM: dts: tegra30: Add External Memory Controller node