mm/mprotect: avoid unnecessary TLB flushes
Subsystems (11)
v3 cover mm/mprotect: avoid unnecessary TLB flushes
From: Nadav Amit <[email protected]>
This patch-set is intended to remove unnecessary TLB flushes during mprotect() syscalls. Once this patch-set make it through, similar and further optimizations for MADV_COLD and userfaultfd would be possible.
Sorry for the time between it took me to get to v3.
Basically, there are 3 optimizations in this patch-set:
1. Use TLB batching infrastructure to batch flushes across VMAs and do better/fewer flushes. This would also be handy for later userfaultfd enhancements.
2. Avoid TLB flushes on permission demotion. This optimization is the one that provides most of the performance benefits. Note that the previous batching infrastructure changes are needed for that to happen.
3. Avoiding TLB flushes on change_huge_pmd() that are only needed to prevent the A/D bits from changing.
Andrew asked for some benchmark numbers. I do not have an easy determinate macrobenchmark in which it is easy to show benefit. I therre ran a microbenchmark: a loop that does the following on anonymous memory, just as a sanity check to see that time is saved by avoiding TLB flushes. The loop goes:
mprotect(p, PAGE_SIZE, PROT_READ) mprotect(p, PAGE_SIZE, PROT_READ|PROT_WRITE) *p = 0; // make the page writable
The test was run in KVM guest with 1 or 2 threads (the second thread was busy-looping). I measured the time (cycles) of each operation:
1 thread 2 threads
mmots +patch mmots +patchPROT_READ 3494 2725 (-22%) 8630 7788 (-10%)
PROT_READ|WRITE 3952 2724 (-31%) 9075 2865 (-68%)
[ mmots = v5.17-rc6-mmots-2022-03-06-20-38 ]
The exact numbers are really meaningless, but the benefit is clear. There are 2 interesting results though.
(1) PROT_READ is cheaper, while one can expect it not to be affected. This is presumably due to TLB miss that is saved
(2) Without memory access (*p = 0), the speedup of the patch is even greater. In that scenario mprotect(PROT_READ) also avoids the TLB flush. As a result both operations on the patched kernel take roughly ~1500 cycles (with either 1 or 2 threads), whereas on mmotm their cost is as high as presented in the table.
--
v2 -> v3:
* Fix orders of patches (order could lead to breakage)
* Better comments
* Clearer KNL detection [Dave]
* Assertion on PF error-code [Dave]
* Comments, code, function names improvements [PeterZ]
* Flush on access-bit clearing on PMD changes to follow the way flushing on x86 is done today in the kernel.
v1 -> v2:
* Wrong detection of permission demotion [Andrea]
* Better comments [Andrea]
* Handle THP [Andrea]
* Batching across VMAs [Peter Xu]
* Avoid open-coding PTE analysis
* Fix wrong use of the mmu_gather()
Nadav Amit (5):
x86: Detection of Knights Landing A/D leak x86/mm: check exec permissions on fault mm/mprotect: use mmu_gather
mm/mprotect: do not flush on permission promotion
mm: avoid unnecessary flush on change_huge_pmd()
arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/pgtable.h | 5 ++ arch/x86/include/asm/pgtable_types.h | 2 + arch/x86/include/asm/tlbflush.h | 82 ++++++++++++++++++++++++ arch/x86/kernel/cpu/intel.c | 5 ++ arch/x86/mm/fault.c | 22 ++++++- arch/x86/mm/pgtable.c | 10 +++ fs/exec.c | 6 +- include/asm-generic/tlb.h | 14 +++++ include/linux/huge_mm.h | 5 +- include/linux/mm.h | 5 +- include/linux/pgtable.h | 20 ++++++ mm/huge_memory.c | 19 ++++-- mm/mprotect.c | 94 +++++++++++++++------------- mm/pgtable-generic.c | 8 +++ mm/userfaultfd.c | 6 +- 16 files changed, 248 insertions(+), 56 deletions(-)
Signature · 1 line
2.25.1
PATCH v3 1/5 x86: Detection of Knights Landing A/D leak
From: Nadav Amit <[email protected]>
Knights Landing has a issue that a thread setting A or D bits may not do so atomically against checking the present bit. A thread which is going to page fault may still set those bits, even though the present bit was already atomically cleared.
This implies that when the kernel clears present atomically, some time later the supposed to be zero entry could be corrupted with stray A or D bits.
Since the PTE could be already used for storing a swap index, or a NUMA migration index, this cannot be tolerated. Most of the time the kernel detects the problem, but in some rare cases it may not.
This patch adds an interface to detect the bug, which will be used in a following patch.
--- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/kernel/cpu/intel.c | 5 +++++ 2 files changed, 6 insertions(+)
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -442,5 +442,6 @@#define X86_BUG_TAA X86_BUG(22)/* CPU is affected by TSX Async Abort(TAA) */ #define X86_BUG_ITLB_MULTIHIT X86_BUG(23)/* CPU may incur MCE during certain page attribute changes */ #define X86_BUG_SRBDS X86_BUG(24)/* CPU may leak RNG bits if not mitigated */ +#define X86_BUG_PTE_LEAK X86_BUG(25)/* PTE may leak A/D bits after clear */ #endif /* _ASM_X86_CPUFEATURES_H */
arch/x86/kernel/cpu/intel.c +5 -0
@@ -296,6 +296,11 @@ static void early_init_intel(struct cpuinfo_x86 *c) } } +if (c->x86_model ==INTEL_FAM6_XEON_PHI_KNL ) { +pr_info_once ("Enabling PTE leaking workaround\n" ); +set_cpu_bug (c,X86_BUG_PTE_LEAK ); + } +/* * Intel Quark Core DevMan_001.pdf section 6.4.11 * "The operating system also is required to invalidate (i.e., flush)
Signature · 1 line
2.25.1
PATCH v3 2/5 x86/mm: check exec permissions on fault
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
access_error() currently does not check for execution permission violation. As a result, spurious page-faults due to execution permission violation cause SIGSEGV.
It appears not to be an issue so far, but the next patches avoid TLB flushes on permission promotion, which can lead to this scenario. nodejs for instance crashes when TLB flush is avoided on permission promotion.
Add a check to prevent access_error() from returning mistakenly that spurious page-faults due to instruction fetch are a reason for an access error.
It is assumed that error code bits of "instruction fetch" and "write" in the hardware error code are mutual exclusive, and the change assumes so. However, to be on the safe side, especially if hypervisors misbehave, assert this is the case and warn otherwise.
--- arch/x86/mm/fault.c | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-)
arch/x86/mm/fault.c +20 -2
@@ -1107,10 +1107,28 @@ access_error(unsigned long error_code, struct vm_area_struct *vma) (error_code & X86_PF_INSTR), foreign))return 1 ; -if (error_code &X86_PF_WRITE ) { +if (error_code & (X86_PF_WRITE |X86_PF_INSTR )) { +/* +* CPUs are not expected to set the two error code bits +* together, but to ensure that hypervisors do not misbehave, +* run an additional sanity check. +*/ +if ((error_code & (X86_PF_WRITE |X86_PF_INSTR )) == + (X86_PF_WRITE |X86_PF_INSTR )) { +WARN_ON_ONCE (1 ); +return 1 ; + } +/* write, present and write, not present: */ -if (unlikely (!(vma->vm_flags &VM_WRITE ))) +if ((error_code &X86_PF_WRITE ) && +unlikely (!(vma->vm_flags &VM_WRITE ))) +return 1 ; + +/* exec, present and exec, not present: */ +if ((error_code &X86_PF_INSTR ) && +unlikely (!(vma->vm_flags &VM_EXEC )))return 1 ; +return 0 ; }
Signature · 1 line
2.25.1
PATCH v3 3/5 mm/mprotect: use mmu_gather
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
change_pXX_range() currently does not use mmu_gather, but instead implements its own deferred TLB flushes scheme. This both complicates the code, as developers need to be aware of different invalidation schemes, and prevents opportunities to avoid TLB flushes or perform them in finer granularity.
The use of mmu_gather for modified PTEs has benefits in various scenarios even if pages are not released. For instance, if only a single page needs to be flushed out of a range of many pages, only that page would be flushed. If a THP page is flushed, on x86 a single TLB invlpg instruction can be used instead of 512 instructions (or a full TLB flush, which would Linux would actually use by default). mprotect() over multiple VMAs requires a single flush.
Use mmu_gather in change_pXX_range(). As the pages are not released, only record the flushed range using tlb_flush_pXX_range().
Handle THP similarly and get rid of flush_cache_range() which becomes redundant since tlb_start_vma() calls it when needed.
--- fs/exec.c | 6 ++- include/linux/huge_mm.h | 5 ++- include/linux/mm.h | 5 ++- mm/huge_memory.c | 10 ++++- mm/mprotect.c | 93 ++++++++++++++++++++++------------------- mm/userfaultfd.c | 6 ++- 6 files changed, 75 insertions(+), 50 deletions(-)
fs/exec.c +5 -1
@@ -759,6 +759,7 @@ int setup_arg_pages(struct linux_binprm *bprm,unsigned long stack_size;unsigned long stack_expand;unsigned long rlim_stack; +struct mmu_gather tlb;#ifdef CONFIG_STACK_GROWSUP /* Limit stack size */
@@ -813,8 +814,11 @@ int setup_arg_pages(struct linux_binprm *bprm, vm_flags |= mm->def_flags; vm_flags |=VM_STACK_INCOMPLETE_SETUP ; - ret =mprotect_fixup (vma, &prev, vma->vm_start, vma->vm_end, +tlb_gather_mmu (&tlb, mm); + ret =mprotect_fixup (&tlb, vma, &prev, vma->vm_start, vma->vm_end, vm_flags); +tlb_finish_mmu (&tlb); +if (ret) goto out_unlock;BUG_ON (prev != vma);
include/linux/huge_mm.h +3 -2
@@ -36,8 +36,9 @@ int zap_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, pud_t *pud, unsigned long addr);bool move_huge_pmd (struct vm_area_struct *vma,unsigned long old_addr,unsigned long new_addr,pmd_t *old_pmd,pmd_t *new_pmd); -int change_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr, -pgprot_t newprot,unsigned long cp_flags); +int change_huge_pmd (struct mmu_gather *tlb,struct vm_area_struct *vma, +pmd_t *pmd,unsigned long addr,pgprot_t newprot, +unsigned long cp_flags);vm_fault_t vmf_insert_pfn_pmd_prot (struct vm_fault *vmf,pfn_t pfn,pgprot_t pgprot,bool write);
include/linux/mm.h +3 -2
@@ -1969,10 +1969,11 @@ extern unsigned long move_page_tables(struct vm_area_struct *vma,#define MM_CP_UFFD_WP_ALL (MM_CP_UFFD_WP | \ MM_CP_UFFD_WP_RESOLVE) -extern unsigned long change_protection (struct vm_area_struct *vma,unsigned long start, +extern unsigned long change_protection (struct mmu_gather *tlb, +struct vm_area_struct *vma,unsigned long start,unsigned long end,pgprot_t newprot,unsigned long cp_flags); -extern int mprotect_fixup (struct vm_area_struct *vma, +extern int mprotect_fixup (struct mmu_gather *tlb,struct vm_area_struct *vma,struct vm_area_struct **pprev,unsigned long start,unsigned long end,unsigned long newflags);
mm/huge_memory.c +8 -2
@@ -1692,8 +1692,9 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,* or if prot_numa but THP migration is not supported * - HPAGE_PMD_NR if protections changed and TLB flush necessary */ -int change_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd, -unsigned long addr,pgprot_t newprot,unsigned long cp_flags) +int change_huge_pmd (struct mmu_gather *tlb,struct vm_area_struct *vma, +pmd_t *pmd,unsigned long addr,pgprot_t newprot, +unsigned long cp_flags) {struct mm_struct *mm = vma->vm_mm;spinlock_t *ptl;
@@ -1704,6 +1705,8 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,bool uffd_wp = cp_flags &MM_CP_UFFD_WP ;bool uffd_wp_resolve = cp_flags &MM_CP_UFFD_WP_RESOLVE ; +tlb_change_page_size (tlb,HPAGE_PMD_SIZE ); +if (prot_numa && !thp_migration_supported ())return 1 ;
@@ -1799,6 +1802,9 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, } ret =HPAGE_PMD_NR ;set_pmd_at (mm, addr, pmd, entry); + +tlb_flush_pmd_range (tlb, addr,HPAGE_PMD_SIZE ); +BUG_ON (vma_is_anonymous (vma) && !preserve_write &&pmd_write (entry)); unlock:spin_unlock (ptl);
mm/mprotect.c +51 -42
248 lines not on this page · raw message
mm/userfaultfd.c +5 -1
@@ -16,6 +16,7 @@#include <linux/hugetlb.h> #include <linux/shmem_fs.h> #include <asm/tlbflush.h> +#include <asm/tlb.h> #include "internal.h" static __always_inline
@@ -687,6 +688,7 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, atomic_t *mmap_changing) {struct vm_area_struct *dst_vma; +struct mmu_gather tlb;pgprot_t newprot;int err;
@@ -728,8 +730,10 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start,else newprot =vm_get_page_prot (dst_vma->vm_flags); -change_protection (dst_vma, start, start + len, newprot, +tlb_gather_mmu (&tlb, dst_mm); +change_protection (&tlb, dst_vma, start, start + len, newprot, enable_wp ?MM_CP_UFFD_WP :MM_CP_UFFD_WP_RESOLVE ); +tlb_finish_mmu (&tlb); err =0 ; out_unlock:
Signature · 1 line
2.25.1
PATCH v3 4/5 mm/mprotect: do not flush on permission promotion
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
Currently, using mprotect() to unprotect a memory region or uffd to unprotect a memory region causes a TLB flush. At least on x86, as protection is promoted, no TLB flush is needed.
Add an arch-specific pte_may_need_flush() which tells whether a TLB flush is needed based on the old PTE and the new one. Implement an x86 pte_may_need_flush().
For x86, besides the simple logic that PTE protection promotion or changes of software bits does require a flush, also add logic that considers the dirty-bit. If the dirty-bit is clear and write-protect is set, no TLB flush is needed, as x86 updates the dirty-bit atomically on write, and if the bit is clear, the PTE is reread.
--- arch/x86/include/asm/pgtable_types.h | 2 + arch/x86/include/asm/tlbflush.h | 82 ++++++++++++++++++++++++++++ include/asm-generic/tlb.h | 14 +++++ mm/huge_memory.c | 9 +-- mm/mprotect.c | 3 +- 5 files changed, 105 insertions(+), 5 deletions(-)
arch/x86/include/asm/pgtable_types.h +2 -0
@@ -110,9 +110,11 @@#if defined(CONFIG_X86_64 ) || defined(CONFIG_X86_PAE )#define _PAGE_NX (_AT(pteval_t, 1) << _PAGE_BIT_NX)#define _PAGE_DEVMAP (_AT(u64, 1) << _PAGE_BIT_DEVMAP) +#define _PAGE_SOFTW4 (_AT(pteval_t, 1) << _PAGE_BIT_SOFTW4)#else #define _PAGE_NX (_AT(pteval_t, 0))#define _PAGE_DEVMAP (_AT(pteval_t, 0)) +#define _PAGE_SOFTW4 (_AT(pteval_t, 0))#endif #define _PAGE_PROTNONE (_AT(pteval_t, 1) << _PAGE_BIT_PROTNONE)
arch/x86/include/asm/tlbflush.h +82 -0
88 lines not on this page · raw message
include/asm-generic/tlb.h +14 -0
@@ -654,6 +654,20 @@ static inline void tlb_flush_p4d_range(struct mmu_gather *tlb, } while (0)#endif +#ifndef pte_needs_flush +static inline bool pte_needs_flush (pte_t oldpte,pte_t newpte) +{ +return true; +} +#endif + +#ifndef huge_pmd_needs_flush +static inline bool huge_pmd_needs_flush (pmd_t oldpmd,pmd_t newpmd) +{ +return true; +} +#endif +#endif /* CONFIG_MMU */ #endif /* _ASM_GENERIC__TLB_H */
mm/huge_memory.c +5 -4
@@ -1698,7 +1698,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, {struct mm_struct *mm = vma->vm_mm;spinlock_t *ptl; -pmd_t entry; +pmd_t oldpmd, entry;bool preserve_write;int ret;bool prot_numa = cp_flags &MM_CP_PROT_NUMA ;
@@ -1784,9 +1784,9 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,* pmdp_invalidate() is required to make sure we don't miss * dirty/young flags set by hardware. */ - entry =pmdp_invalidate (vma, addr, pmd); + oldpmd =pmdp_invalidate (vma, addr, pmd); - entry =pmd_modify (entry, newprot); + entry =pmd_modify (oldpmd, newprot);if (preserve_write) entry =pmd_mk_savedwrite (entry);if (uffd_wp) {
@@ -1803,7 +1803,8 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, ret =HPAGE_PMD_NR ;set_pmd_at (mm, addr, pmd, entry); -tlb_flush_pmd_range (tlb, addr,HPAGE_PMD_SIZE ); +if (huge_pmd_needs_flush (oldpmd, entry)) +tlb_flush_pmd_range (tlb, addr,HPAGE_PMD_SIZE );BUG_ON (vma_is_anonymous (vma) && !preserve_write &&pmd_write (entry)); unlock:
mm/mprotect.c +2 -1
@@ -152,7 +152,8 @@ static unsigned long change_pte_range(struct mmu_gather *tlb, ptent =pte_mkwrite (ptent); }ptep_modify_prot_commit (vma, addr, pte, oldpte, ptent); -tlb_flush_pte_range (tlb, addr,PAGE_SIZE ); +if (pte_needs_flush (oldpte, ptent)) +tlb_flush_pte_range (tlb, addr,PAGE_SIZE ); pages++; }else if (is_swap_pte (oldpte)) {swp_entry_t entry =pte_to_swp_entry (oldpte);
Signature · 1 line
2.25.1
PATCH v3 5/5 mm: avoid unnecessary flush on change_huge_pmd()
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
Calls to change_protection_range() on THP can trigger, at least on x86, two TLB flushes for one page: one immediately, when pmdp_invalidate() is called by change_huge_pmd(), and then another one later (that can be batched) when change_protection_range() finishes.
The first TLB flush is only necessary to prevent the dirty bit (and with a lesser importance the access bit) from changing while the PTE is modified. However, this is not necessary as the x86 CPUs set the dirty-bit atomically with an additional check that the PTE is (still) present. One caveat is Intel's Knights Landing that has a bug and does not do so.
Leverage this behavior to eliminate the unnecessary TLB flush in change_huge_pmd(). Introduce a new arch specific pmdp_invalidate_ad() that only invalidates the access and dirty bit from further changes.
--- arch/x86/include/asm/pgtable.h | 5 +++++ arch/x86/mm/pgtable.c | 10 ++++++++++ include/linux/pgtable.h | 20 ++++++++++++++++++++ mm/huge_memory.c | 4 ++-- mm/pgtable-generic.c | 8 ++++++++ 5 files changed, 45 insertions(+), 2 deletions(-)
arch/x86/include/asm/pgtable.h +5 -0
@@ -1173,6 +1173,11 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma, } }#endif + +#define __HAVE_ARCH_PMDP_INVALIDATE_AD +extern pmd_t pmdp_invalidate_ad (struct vm_area_struct *vma, +unsigned long address,pmd_t *pmdp); +/* * Page table pages are page-aligned. The lower half of the top * level is used for userspace and the top half for the kernel.
arch/x86/mm/pgtable.c +10 -0
@@ -608,6 +608,16 @@ int pmdp_clear_flush_young(struct vm_area_struct *vma, return young; } + +pmd_t pmdp_invalidate_ad (struct vm_area_struct *vma,unsigned long address, +pmd_t *pmdp) +{ +pmd_t old =pmdp_establish (vma, address, pmdp,pmd_mkinvalid (*pmdp)); + +if (cpu_feature_enabled (X86_BUG_PTE_LEAK )) +flush_pmd_tlb_range (vma, address, address +HPAGE_PMD_SIZE ); +return old; +}#endif /**
include/linux/pgtable.h +20 -0
@@ -570,6 +570,26 @@ extern pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp);#endif +#ifndef __HAVE_ARCH_PMDP_INVALIDATE_AD + +/* +* pmdp_invalidate_ad() invalidates the PMD while changing a transparent +* hugepage mapping in the page tables. This function is similar to +* pmdp_invalidate(), but should only be used if the access and dirty bits would +* not be cleared by the software in the new PMD value. The function ensures +* that hardware changes of the access and dirty bits updates would not be lost. +* +* Doing so can allow in certain architectures to avoid a TLB flush in most +* cases. Yet, another TLB flush might be necessary later if the PMD update +* itself requires such flush (e.g., if protection was set to be stricter). Yet, +* even when a TLB flush is needed because of the update, the caller may be able +* to batch these TLB flushing operations, so fewer TLB flush operations are +* needed. +*/ +extern pmd_t pmdp_invalidate_ad (struct vm_area_struct *vma, +unsigned long address,pmd_t *pmdp); +#endif +#ifndef __HAVE_ARCH_PTE_SAMEstatic inline int pte_same (pte_t pte_a,pte_t pte_b) {
mm/huge_memory.c +2 -2
@@ -1781,10 +1781,10 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,* The race makes MADV_DONTNEED miss the huge pmd and don't clear it * which may break userspace. * -* pmdp_invalidate() is required to make sure we don't miss +* pmdp_invalidate_ad() is required to make sure we don't miss * dirty/young flags set by hardware. */ - oldpmd =pmdp_invalidate (vma, addr, pmd); + oldpmd =pmdp_invalidate_ad (vma, addr, pmd); entry =pmd_modify (oldpmd, newprot);if (preserve_write)
mm/pgtable-generic.c +8 -0
@@ -201,6 +201,14 @@ pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address, }#endif +#ifndef __HAVE_ARCH_PMDP_INVALIDATE_AD +pmd_t pmdp_invalidate_ad (struct vm_area_struct *vma,unsigned long address, +pmd_t *pmdp) +{ +return pmdp_invalidate (vma, address, pmdp); +} +#endif +#ifndef pmdp_collapse_flushpmd_t pmdp_collapse_flush (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp)
Signature · 1 line
2.25.1
Andrew Morton
maintainerQuoted 4 lines · On Tue, 8 Mar 2022 20:10:38 -0800 Nadav Amit <[email protected]> wrote:
This patch-set is intended to remove unnecessary TLB flushes during
mprotect()syscalls. Once this patch-set make it through, similar and further optimizations forMADV_COLDand userfaultfd would be possible.
Thanks. I'll save this away for consideration after the -rc1 release. Which gives others plenty of time for review (hint) (request).
Quoted 14 lines · Andrew Morton
Quoted 4 lines · On Tue, 8 Mar 2022 20:10:38 -0800 Nadav Amit <[email protected]> wrote:
This patch-set is intended to remove unnecessary TLB flushes during
mprotect()syscalls. Once this patch-set make it through, similar and further optimizations forMADV_COLDand userfaultfd would be possible.Thanks. I'll save this away for consideration after the -rc1 release. Which gives others plenty of time for review (hint) (request).
I see that some patches did not make it through to the mailing list.
I will resend later today.