Conform L3 Cache Index Disable to Linux Standards
PATCH 1/1 Conform L3 Cache Index Disable to Linux Standards
Add ABI Documentation entry and fix some /sys directory formating issues with the L3 Cache Index Disable feature for future AMD processors. Add a check to disable it for family 0x10 models that do not support it properly. Correct the disable algorithm to reflect erratum 388.
Documentation/ABI/testing/sysfs-devices-cache_disable new file +18 -0
@@ -0,0 +1,18 @@ +What: /sys/devices/system/cpu/cpu*/cache/index*/cache_disable_X +Date: August 2008 +KernelVersion: 2.6.27 +Contact: [email protected] +Description: These files exist in every cpu's cache index directories. + There are currently 2 cache_disable_# files in each + directory. Reading from these files on a supported + processor will return that cache disable index value + for that processor and node. Writing to one of these + files will cause the specificed cache index to be disabled. + + Currently, only AMD Family 10h Processors support cache index + disable, and only for their L3 caches. See the BIOS and + Kernel Developer's Guide at + http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/31116-Public-GH-BKDG_3.20_2-4-09.pdf + for formatting information and other details on the + cache index disable. +Users: [email protected]
arch/x86/include/asm/k8.h +4 -0
@@ -6,7 +6,11 @@extern struct pci_device_id k8_nb_ids[];extern int early_is_k8_nb (u32 value); +#ifdef CONFIG_K8_NB extern struct pci_dev **k8_northbridges; +#else +struct pci_dev **k8_northbridges; +#endif extern int num_k8_northbridges;extern int cache_k8_northbridges (void );extern void k8_flush_garts (void );
arch/x86/kernel/cpu/intel_cacheinfo.c +78 -102
222 lines not on this page · raw message
Quoted 7 lines · Mark Langsdorf
Add ABI Documentation entry and fix some /sys directory formating issues with the L3 Cache Index Disable feature for future AMD processors. Add a check to disable it for family
0x10models that do not support it properly. Correct the disable algorithm to reflect erratum 388.Signed-off-by Mark Langsdorf <[email protected]>
i'm still getting this build failure:
arch/x86/mm/built-in.o:(.bss+0x3640): multiple definition of `k8_northbridges'
i've reported this build failure before. I've excluded this commit from tip:master for now.
Ingo
PATCH 1/1 Conform L3 Cache Index Disable to Linux Standards
In reply to Mark Langsdorfcommit eb40831ca29a89f056f8c6128c8b36d3691f6698
Author: Mark Langsdorf <[email protected]> Date: Wed Apr 8 15:48:45 2009 -0500
Add ABI Documentation entry and fix some /sys directory formating issues with the L3 Cache Index Disable feature for future AMD processors. Add a check to disable it for family 0x10 models that do not support it yet.
Documentation/ABI/testing/sysfs-devices-cache_disable new file +18 -0
@@ -0,0 +1,18 @@ +What: /sys/devices/system/cpu/cpu*/cache/index*/cache_disable_X +Date: August 2008 +KernelVersion: 2.6.27 +Contact: [email protected] +Description: These files exist in every cpu's cache index directories. + There are currently 2 cache_disable_# files in each + directory. Reading from these files on a supported + processor will return that cache disable index value + for that processor and node. Writing to one of these + files will cause the specificed cache index to be disabled. + + Currently, only AMD Family 10h Processors support cache index + disable, and only for their L3 caches. See the BIOS and + Kernel Developer's Guide at + http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/31116-Public-GH-BKDG_3.20_2-4-09.pdf + for formatting information and other details on the + cache index disable. +Users: [email protected]
arch/x86/include/asm/k8.h +9 -0
@@ -12,4 +12,13 @@ extern int cache_k8_northbridges(void);extern void k8_flush_garts (void );extern int k8_scan_nodes (unsigned long start,unsigned long end); +static inline struct pci_dev *get_k8_northbridge (int node) +{ +#ifdef CONFIG_K8_NB +return k8_northbridges[node]; +#else +return NULL ; +#endif +} +#endif /* _ASM_X86_K8_H */
arch/x86/kernel/cpu/intel_cacheinfo.c +80 -102
224 lines not on this page · raw message
Quoted 10 lines · Mark Langsdorf
commit
eb40831ca29a89f056f8c6128c8b36d3691f6698Author: Mark Langsdorf <[email protected]> Date: Wed Apr 8 15:48:45 2009 -0500Add ABI Documentation entry and fix some /sys directory formating issues with the L3 Cache Index Disable feature for future AMD processors. Add a check to disable it for family
0x10models that do not support it yet.Signed-off-by Mark Langsdorf <[email protected]>
Could you please send a delta patch against the tip:x86/cpu branch, which currently contains an earlier (broken) patch of yours:
45ca863: x86, cpu: conform L3 Cache Index Disable to Linux standards
You can get that branch by doing:
git checkout tip/x86/cpu
After having picked up -tip as a remote repository as per:
http://people.redhat.com/mingo/tip.git/README
Thanks,
Ingo
Quoted 10 lines · Mark Langsdorf
commit
eb40831ca29a89f056f8c6128c8b36d3691f6698Author: Mark Langsdorf <[email protected]> Date: Wed Apr 8 15:48:45 2009 -0500Add ABI Documentation entry and fix some /sys directory formating issues with the L3 Cache Index Disable feature for future AMD processors. Add a check to disable it for family
0x10models that do not support it yet.Signed-off-by Mark Langsdorf <[email protected]>
I feel uncomfortable with this patch (and the previous ones). The patch addresses issues with the current Linux code but introduces new ones.
I suggest to revert last two patches on tip/x86/cpu and to split Mark's last patch into several pieces. (BTW, i'll do this in any case asap).
This would really help to understand the code changes and to ease review and further adaptions of this code.
Quoted 6 lines
arch/x86/include/asm/k8.h +0
-0@@ -12,4 +12,13 @@ extern int cache_k8_northbridges(void);extern void k8_flush_garts (void );
arch/x86/include/asm/k8.h extern int cache_k8_northbridges(void); · line 22
Quoted 10 lines
arch/x86/include/asm/k8.h +8
-0extern int k8_scan_nodes (unsigned long start,unsigned long end); +static inline struct pci_dev *get_k8_northbridge (int node) +{ +#ifdef CONFIG_K8_NB +return k8_northbridges[node]; +#else +return NULL ; +#endif +}
IMHO a macro would suffice to return either k8_northbridges[node] or NULL. And num_k8_northbridges should be checked, no?
Quoted 11 lines
arch/x86/include/asm/k8.h +0
-0#endif /* _ASM_X86_K8_H */ arch/x86/kernel/cpu/intel_cacheinfo.c +2
-0@@ -639,6 +640,70 @@ static ssize_t show_##file_name \return sprintf (buf,"%lu\n" , (unsigned long )this_leaf->object + val); \ } +static ssize_t show_cache_disable (struct _cpuid4_info *this_leaf,char *buf, +unsigned int index)
arch/x86/kernel/cpu/intel_cacheinfo.c static ssize_t show_##file_name \ · line 654
Quoted 10 lines
arch/x86/kernel/cpu/intel_cacheinfo.c +10
-0+{ +int cpu =cpumask_first (to_cpumask (this_leaf->shared_cpu_map)); +int node =cpu_to_node (cpu); +struct pci_dev *dev =get_k8_northbridge (node); +unsigned int reg =0 ; + +if (!this_leaf->can_disable) +return -EINVAL ; + +pci_read_config_dword (dev,0x1BC + index *4 , ®);
Do I miss something, or why is there no check for dev==NULL? IMHO this will cause an Oops when dereferencing dev->bus in pci_read_config_dword in case of "ifndef CONFIG_K8_NB".
Quoted 21 lines
arch/x86/kernel/cpu/intel_cacheinfo.c +21
-0+return sprintf (buf,"%x\n" , reg); +} + +#define SHOW_CACHE_DISABLE (index) \ +static ssize_t \ +show_cache_disable_##index(struct _cpuid4_info *this_leaf, char *buf) \ +{ \ + return show_cache_disable(this_leaf, buf, index); \ +} + +static ssize_t +store_cache_disable (struct _cpuid4_info *this_leaf,const char *buf, +size_t count,unsigned int index) +{ +int cpu =cpumask_first (to_cpumask (this_leaf->shared_cpu_map)); +int node =cpu_to_node (cpu); +struct pci_dev *dev =get_k8_northbridge (node); +unsigned long val =0 ; +unsigned int scrubber =0 ; + +if (!this_leaf->can_disable)
arch/x86/kernel/cpu/intel_cacheinfo.c static ssize_t show_##file_name \ · line 685
Quoted 10 lines
arch/x86/kernel/cpu/intel_cacheinfo.c +10
-0+return -EINVAL ; + +if (!capable (CAP_SYS_ADMIN )) +return -EPERM ; + +if (strict_strtoul (buf,10 , &val) <0 ) +return -EINVAL ; + + val |=0xc0000000 ; +pci_read_config_dword (dev,0x58 , &scrubber);
Missing check for dev==NULL and potential Oops, no?
arch/x86/kernel/cpu/intel_cacheinfo.c static ssize_t show_##file_name \ · line 687
Quoted 2 lines
arch/x86/kernel/cpu/intel_cacheinfo.c +2
-0+ scrubber &= ~0x0f800000 ; +pci_write_config_dword (dev,0x58 , scrubber);
This adapts L3 scrub rate but accesses reserved bits. Perhaps you meant to zero out bits 24-28 which would mean to use
scrubber &= ~0x1f000000;
(... but maybe I need some more coffee to do the bit fiddling ... ;-)
Regards,
Andreas
PATCH 1/1 Enable GART-IOMMU only after setting up protection methods
In reply to Mark LangsdorfThe current code to set up the GART as an IOMMU enables GART translations before it removes the aperture from the kernel memory map, sets the GART PTEs to UC, sets up the guard and scratch pages, or does a wbinvd(). This leaves the possibility of cache aliasing open and can cause system crashes.
Re-order the code and add tlbflush so as to enable the GART translations only after all safeguards are in place and the tlb has been flushed.
AMD has tested this patch and seen no adverse effects.
arch/x86/kernel/pci-gart_64.c +8 -2
@@ -38,6 +38,7 @@#include <asm/swiotlb.h> #include <asm/dma.h> #include <asm/k8.h> +#include <asm/tlbflush.h> static unsigned long iommu_bus_base;/* GART remapping area (physical) */ static unsigned long iommu_size;/* size of remapping area bytes */
@@ -682,9 +683,9 @@if (set_memory_uc ((unsigned long )gatt, gatt_size >>PAGE_SHIFT ))panic ("Could not set GART PTEs to uncacheable pages" ); +wbinvd (); + agp_gatt_table = gatt; - -enable_gart_translations (); error =sysdev_class_register (&gart_sysdev_class);if (!error)
@@ -855,6 +856,11 @@for (i =EMERGENCY_PAGES ; i < iommu_pages; i++) iommu_gatt_base[i] = gart_unmapped_entry; +wbinvd (); +flush_tlb_all (); + +enable_gart_translations (); +flush_gart (); dma_ops = &gart_dma_ops; }
Re: [PATCH] Enable GART-IOMMU only after setting up protection methods
Quoted 11 lines · Mark Langsdorf
The current code to set up the GART as an IOMMU enables GART translations before it removes the aperture from the kernel memory map, sets the GART PTEs to UC, sets up the guard and scratch pages, or does a
wbinvd(). This leaves the possibility of cache aliasing open and can cause system crashes.Re-order the code and add tlbflush so as to enable the GART translations only after all safeguards are in place and the tlb has been flushed.
AMD has tested this patch and seen no adverse effects.
how about system with AGP card and more than 4g system RAM?
also it seems flush_gart() in init_k8_gatt() and gart_iommu_init() could be removed?
YH
RE: [PATCH] Enable GART-IOMMU only after setting up protection methods
Quoted 3 lines
Quoted 1 line
AMD has tested this patch and seen no adverse effects.
how about system with AGP card and more than 4g system RAM?
Tested on an Iwill DK8X 2P system with 8 GB of DRAM.
Quoted 2 lines
also it seems
flush_gart()ininit_k8_gatt()andgart_iommu_init()could be removed?
Possibly, but I'd rather be cautious than not.
-Mark Langsdorf
Operating System Research Center
AMD
Re: [PATCH] Enable GART-IOMMU only after setting up protection methods
In reply to Mark LangsdorfQuoted 5 lines · Mark Langsdorf
The current code to set up the GART as an IOMMU enables GART translations before it removes the aperture from the kernel memory map, sets the GART PTEs to UC, sets up the guard and scratch pages, or does a
wbinvd(). This leaves the possibility of cache aliasing open and can cause system crashes.
This patch looks broken. I don't think that there are any aliasing problems in the existing code. Can you elaborate were do you see these problems? Is there any existing bug report for this?
Quoted 20 lines
Re-order the code and add tlbflush so as to enable the GART translations only after all safeguards are in place and the tlb has been flushed.
AMD has tested this patch and seen no adverse effects.
Signed-off-by Mark Langsdorf <[email protected]>diff -r
0d1744c7acc7arch/x86/kernel/pci-gart_64.carch/x86/kernel/pci-gart_64.c +1
-0@@ -38,6 +38,7 @@#include <asm/swiotlb.h> #include <asm/dma.h> #include <asm/k8.h> +#include <asm/tlbflush.h> static unsigned long iommu_bus_base;/* GART remapping area (physical) */ static unsigned long iommu_size;/* size of remapping area bytes */
arch/x86/kernel/pci-gart_64.c line 687
Quoted 8 lines
arch/x86/kernel/pci-gart_64.c +2
-2@@ -682,9 +683,9 @@if (set_memory_uc ((unsigned long )gatt, gatt_size >>PAGE_SHIFT ))panic ("Could not set GART PTEs to uncacheable pages" ); +wbinvd (); + agp_gatt_table = gatt; - -enable_gart_translations ();
To enable the GART here has the advantage that all other CPUs are not yet up. Since the GATT has only non-present entries there is no cache aliasing problem here. If you enable it later you need to flush the cache on _all_ CPUs and not just the CPU where the GART initialization code runs.
arch/x86/kernel/pci-gart_64.c line 859
Quoted 8 lines
arch/x86/kernel/pci-gart_64.c +1
-0error =sysdev_class_register (&gart_sysdev_class);if (!error)@@ -855,6 +856,11 @@for (i =EMERGENCY_PAGES ; i < iommu_pages; i++) iommu_gatt_base[i] = gart_unmapped_entry; +wbinvd ();
Why a wbinvd() here? A few lines above is already one.
arch/x86/kernel/pci-gart_64.c line 860
Quoted 1 line
arch/x86/kernel/pci-gart_64.c +1
-0+flush_tlb_all ();
Same with TLB flush. The set_memory_np() function does that for us.
arch/x86/kernel/pci-gart_64.c line 863
Quoted 3 lines
arch/x86/kernel/pci-gart_64.c +3
-0+ +enable_gart_translations (); +
If we change the enablement logic like done in this patch you need to invalidate the caches on _all_ cpus. I don't think this is necessary if we just enable all GARTs when only the boot cpu is up.
Quoted 3 lines
arch/x86/kernel/pci-gart_64.c +0
-0flush_gart (); dma_ops = &gart_dma_ops; }