low-latency zap_page_range
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 6 lines
The lock hold time in
zap_page_rangeis horrid. This patch breaks the work up into chunks and relinquishes the lock after each iteration. This drastically lowers latency by creating a preemption point, as well as lowering lock contention.void
zap_page_range(struct vm_area_struct *vma, unsigned long address, unsigned long size)
Arjan sent me something similar, done by AKPM, only he did this a little differently. He added an argument to zap_page_range which allowed to work it in the old way, if set. Then, he set it so all places would use low latency EXCEPT a reading from /dev/zero. I assume it was some locking somewhere in devices/char/mem.c, though I was unable to figure which in particular.
Andrew, care to unconfuse me?
-- Pete
Quoted 6 lines · Pete Zaitcev
Quoted 5 lines
The lock hold time in
zap_page_rangeis horrid. This patch breaks the work up into chunks and relinquishes the lock after each iteration. This drastically lowers latency by creating a preemption point, as well as lowering lock contention.
Quoted 9 lines
Quoted 1 line
void
zap_page_range(struct vm_area_struct *vma, unsigned long address, unsigned long size)Arjan sent me something similar, done by AKPM, only he did this a little differently. He added an argument to
zap_page_rangewhich allowed to work it in the old way, if set. Then, he set it so all places would use low latency EXCEPT a reading from/dev/zero. I assume it was some locking somewhere indevices/char/mem.c, though I was unable to figure which in particular.
There are actually quite a few places in the ll patch which don't pass ZPR_COND_RESCHED into zap_page_range. Places which are themselves called under locks, places where not enough pages are being zapped to make it necessary, etc. vmtruncate_list, some mremap code, others.
Plus some historical notes: there used to be a couple more flags which could be passed into zap_page_range() to tell it whether to run flush_cache_range and flush_tlb_range. That was an irrelevant cleanup. But those calls were later unconditionally sucked into zap_page_range() anyway.
-
Quoted 5 lines · Andrew Morton
There are actually quite a few places in the ll patch which don't pass
ZPR_COND_RESCHEDintozap_page_range. Places which are themselves called under locks, places where not enough pages are being zapped to make it necessary, etc.vmtruncate_list, some mremap code, others.
The bonus of the approach in my patch is that we won't reschedule unless the lock count is zero (due to the preemptive kernel), so we do not need to worry about the above. The only concern would be if there is ever a case where the critical region is the entire chunk we want to zap (e.g. we must zap the entire inputted range atomically) but I do not see that ever being the case.
Robert Love