Page tables on machines with >2GB RAM
I think we can end up truncating a PMD or PGD entry (I get confused easily about levels of the page tables; bear with me)
/* NOTE: even on 64 bits, these entries are __u32 because we allocate
* the pmd and pgd in ZONE_DMA (i.e. under 4GB) */
typedef struct { __u32 pgd; } pgd_t;
...
typedef struct { __u32 pmd; } pmd_t;
...
pgd_t *pgd = (pgd_t *)__get_free_pages(GFP_KERNEL,
PGD_ALLOC_ORDER);
...
return (pmd_t *)__get_free_pages(GFP_PGTABLE_KERNEL, PMD_ORDER);so if we have more than 2GB of RAM, we can allocate a page with the top bit set, which we interpret to mean PAGE_PRESENT in the TLB miss handler and mask it off, causing us to load the wrong page for the next level of the page table walk.
Have I missed something?
Oh and I think this bug was introduced in 2004 ...
Quoted 16 lines · Matthew Wilcox
I think we can end up truncating a PMD or PGD entry (I get confused easily about levels of the page tables; bear with me)
/* NOTE: even on 64 bits, these entries are
__u32because we allocate
* the pmd and pgd inZONE_DMA(i.e. under 4GB) */
typedef struct {__u32pgd; }pgd_t;
...
typedef struct {__u32pmd; }pmd_t;...
pgd_t *pgd = (pgd_t *)__get_free_pages(GFP_KERNEL, PGD_ALLOC_ORDER); ... return (pmd_t *)__get_free_pages(GFP_PGTABLE_KERNEL, PMD_ORDER);
Quoted 6 lines
so if we have more than 2GB of RAM, we can allocate a page with the top bit set, which we interpret to mean
PAGE_PRESENTin the TLB miss handler and mask it off, causing us to load the wrong page for the next level of the page table walk.Have I missed something?
Yes, yes I have.
We store the PFN, not the physical address. So we have 28 bits for storing the PFN and 4 bits for the PxD bits, supporting 28 + 12 = 40 bits (1TB) of physical address space.
Quoted 22 lines · Matthew Wilcox
Quoted 21 lines · Matthew Wilcox
I think we can end up truncating a PMD or PGD entry (I get confused easily about levels of the page tables; bear with me)
/* NOTE: even on 64 bits, these entries are
__u32because we allocate
* the pmd and pgd inZONE_DMA(i.e. under 4GB) */
typedef struct {__u32pgd; }pgd_t;
...
typedef struct {__u32pmd; }pmd_t;...
pgd_t *pgd = (pgd_t *)__get_free_pages(GFP_KERNEL, PGD_ALLOC_ORDER); ... return (pmd_t *)__get_free_pages(GFP_PGTABLE_KERNEL, PMD_ORDER);so if we have more than 2GB of RAM, we can allocate a page with the top bit set, which we interpret to mean
PAGE_PRESENTin the TLB miss handler and mask it off, causing us to load the wrong page for the next level of the page table walk.
Quoted 6 lines
Quoted 1 line
Have I missed something?
Yes, yes I have.
We store the PFN, not the physical address. So we have 28 bits for storing the PFN and 4 bits for the PxD bits, supporting 28 + 12 = 40 bits (1TB) of physical address space.
The comment in pgalloc.h says 8TB? I think improving the description as to how this works would be welcome.
Signature · 1 line
John David Anglin [email protected]
Quoted 23 lines · John David Anglin
Quoted 22 lines · Matthew Wilcox
Quoted 21 lines · Matthew Wilcox
I think we can end up truncating a PMD or PGD entry (I get confused easily about levels of the page tables; bear with me)
/* NOTE: even on 64 bits, these entries are
__u32because we allocate
* the pmd and pgd inZONE_DMA(i.e. under 4GB) */
typedef struct {__u32pgd; }pgd_t;
...
typedef struct {__u32pmd; }pmd_t;...
pgd_t *pgd = (pgd_t *)__get_free_pages(GFP_KERNEL, PGD_ALLOC_ORDER); ... return (pmd_t *)__get_free_pages(GFP_PGTABLE_KERNEL, PMD_ORDER);so if we have more than 2GB of RAM, we can allocate a page with the top bit set, which we interpret to mean
PAGE_PRESENTin the TLB miss handler and mask it off, causing us to load the wrong page for the next level of the page table walk.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Have I missed something?
Yes, yes I have.
We store the PFN, not the physical address. So we have 28 bits for storing the PFN and 4 bits for the PxD bits, supporting 28 + 12 = 40 bits (1TB) of physical address space.
The comment in
pgalloc.hsays 8TB? I think improving the description as to how this works would be welcome.
It's talking about 8TB of virtual address space. But I think it's wrong. On 64-bit,
Each PTE defines a 4kB region of address space (ie one page). Each PMD is a 4kB allocation with 8-byte entries, so covers 512 * 4kB = 2MB Each PGD is an 8kB allocation with 4-byte entries, so covers 2048 * 2M = 4GB The top-level allocation is a 32kB allocation, but the first 8kB is used for the first PGD, so it covers 24kB / 4 bytes * 4GB = 24TB.
I think the top level allocation was supposed to be an order-2 allocation, which would be an 8TB address space, but it's order-3.
There's a lot of commentary which disagrees with the code. For example,
#define PMD_ORDER 1 /* Number of pages per pmd */
That's just not true; an order-1 allocation is 2 pages, not 1.
Quoted 30 lines · Matthew Wilcox
Quoted 29 lines · John David Anglin
Quoted 28 lines · Matthew Wilcox
Quoted 16 lines · Matthew Wilcox
I think we can end up truncating a PMD or PGD entry (I get confused easily about levels of the page tables; bear with me)
/* NOTE: even on 64 bits, these entries are
__u32because we allocate
* the pmd and pgd inZONE_DMA(i.e. under 4GB) */
typedef struct {__u32pgd; }pgd_t;
...
typedef struct {__u32pmd; }pmd_t;...
pgd_t *pgd = (pgd_t *)__get_free_pages(GFP_KERNEL, PGD_ALLOC_ORDER); ... return (pmd_t *)__get_free_pages(GFP_PGTABLE_KERNEL, PMD_ORDER);Quoted 6 lines
so if we have more than 2GB of RAM, we can allocate a page with the top bit set, which we interpret to mean
PAGE_PRESENTin the TLB miss handler and mask it off, causing us to load the wrong page for the next level of the page table walk.Have I missed something?
Yes, yes I have.
We store the PFN, not the physical address. So we have 28 bits for storing the PFN and 4 bits for the PxD bits, supporting 28 + 12 = 40 bits (1TB) of physical address space.
Quoted 8 lines
Quoted 2 lines
The comment in
pgalloc.hsays 8TB? I think improving the description as to how this works would be welcome.It's talking about 8TB of virtual address space. But I think it's wrong. On 64-bit,
Each PTE defines a 4kB region of address space (ie one page). Each PMD is a 4kB allocation with 8-byte entries, so covers 512 * 4kB = 2MB
No, PMD is 4kb allocation with 4-byte entries, so covers 1024 * 4kb = 4MB We always us 4-byte entries, for 32- and 64-bit kernels.
Quoted 1 line
Each PGD is an 8kB allocation with 4-byte entries, so covers 2048 * 2M = 4GB
No. each PGD is a 4kb allocation with 4-byte entries. so covers 1024 * 4MB = 4GB Still, my calculation ends up with 4GB, like yours.
Quoted 2 lines
The top-level allocation is a 32kB allocation, but the first 8kB is used for the first PGD, so it covers 24kB / 4 bytes * 4GB = 24TB.
size of PGD (swapper_pg_dir) is 8k, so we have 8k / 4 bytes * 4GB = 8 TB virtual address space.
At boot we want to map (1 << KERNEL_INITIAL_ORDER) pages (=64MB on 64bit kernel) and for this pmd0 gets pre-allocated with 8k size, and pg0 with 132k to simplify the filling the initial page tables - but that's not relevant for the calculations above.
Quoted 7 lines
I think the top level allocation was supposed to be an order-2 allocation, which would be an 8TB address space, but it's order-3.
There's a lot of commentary which disagrees with the code. For example,
#define
PMD_ORDER1 /* Number of pages per pmd */
That's just not true; an order-1 allocation is 2 pages, not 1.
Yes, that should be fixed up.
Helge
Quoted 9 lines · Helge Deller
Quoted 5 lines · Matthew Wilcox
It's talking about 8TB of virtual address space. But I think it's wrong. On 64-bit,
Each PTE defines a 4kB region of address space (ie one page). Each PMD is a 4kB allocation with 8-byte entries, so covers 512 * 4kB = 2MB
No, PMD is 4kb allocation with 4-byte entries, so covers 1024 * 4kb = 4MB We always us 4-byte entries, for 32- and 64-bit kernels.
#if CONFIG_PGTABLE_LEVELS == 3
#define PGD_ORDER 1 /* Number of pages per pgd */
#define PMD_ORDER 1 /* Number of pages per pmd */
#define PGD_ALLOC_ORDER (2 + 1) /* first pgd contains pmd */
...
#if CONFIG_PGTABLE_LEVELS == 3
...
static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long address)
{
return (pmd_t *)__get_free_pages(GFP_PGTABLE_KERNEL, PMD_ORDER);
}We're definitely doing an 8kB allocation. If we should be allocating 4kB, then PMD_ORDER should be 0.
The 32-bit entries, even on 64-bit are a nice hack. I think that just means we're over-allocating memory for the page tables.
Quoted 4 lines
Quoted 1 line
Each PGD is an 8kB allocation with 4-byte entries, so covers 2048 * 2M = 4GB
No. each PGD is a 4kb allocation with 4-byte entries. so covers 1024 * 4MB = 4GB Still, my calculation ends up with 4GB, like yours.
Again, I think there's an order vs count confusion here.
Quoted 10 lines
Quoted 2 lines
The top-level allocation is a 32kB allocation, but the first 8kB is used for the first PGD, so it covers 24kB / 4 bytes * 4GB = 24TB.
size of PGD (
swapper_pg_dir) is 8k, so we have 8k / 4 bytes * 4GB = 8 TB virtual address space.At boot we want to map (1 <<
KERNEL_INITIAL_ORDER) pages (=64MB on 64bit kernel) and for this pmd0 gets pre-allocated with 8k size, and pg0 with 132k to simplify the filling the initial page tables - but that's not relevant for the calculations above.
I was talking about pgd_alloc():
pgd_t *pgd = (pgd_t *)__get_free_pages(GFP_KERNEL,
PGD_ALLOC_ORDER);where we allocate 8 * 4kB pages
Quoted 11 lines
Quoted 7 lines
I think the top level allocation was supposed to be an order-2 allocation, which would be an 8TB address space, but it's order-3.
There's a lot of commentary which disagrees with the code. For example,
#define
PMD_ORDER1 /* Number of pages per pmd */
That's just not true; an order-1 allocation is 2 pages, not 1.Yes, that should be fixed up.
Helge