dlm_controld: Improvement on plock searching efficiency

This series introduces a RB tree for improving plock resources searching efficiency. We met this performance issue when running Samba on top of cluster filesystem, profiling during nbench runs with num-progs=500, the dlm_controld process can often be observed consuming an entire CPU core.

Oprofile points the finger at the dlm_controld/plock.c:find_resource() function, which taking into account the optimization should be the following code:

static struct resource *search_resource(struct lockspace *ls, uint64_t number)

{
        struct resource *r;

        list_for_each_entry(r, &ls->plock_resources, list) {
                if (r->number == number)
                        return r;
        }
        return NULL;
}

List is not quite efficient in searching, the search time is O(n). RB tree seems to be a good option since its search time is O(logn). So I introduce the RB tree just for improving the searching efficiency.

And in our testing, the RB tree conversion shows a considerable reduction in dlm_controld CPU usage:

libdlm 3.00.01-0.8.11 (SLE11 SP1)
=================================

samples  %        image name               app name                 symbol name
3958388  17.6510  vmlinux-2.6.32.27-0.2-default vmlinux-2.6.32.27-0.2-default

prepare_to_wait
1714244 7.6441 dlm_controld dlm_controld
find_resource

libdlm 3.00.01-0.25.6 (rbtree)
=================================

samples  %        image name               app name                 symbol name
1991559  10.0145  vmlinux-2.6.32.27-0.2-default vmlinux-2.6.32.27-0.2-default

mutex_spin_on_owner
...
12753 0.0641 dlm_controld dlm_controld
find_resource

In this patch series, the RB tree implementation is borrowed from the kernel source, and my changes are:

Jiaju Zhang (5):

 group/dlm_controld/Makefile.am  |    2 +-
 group/dlm_controld/dlm_daemon.h |    2 +
 group/dlm_controld/main.c       |    1 +
 group/dlm_controld/plock.c      |   62 +++++++++++++++++++++++++++++++++++---
 group/include/Makefile.am       |    2 +-
 5 files changed, 62 insertions(+), 7 deletions(-)

Thanks, the patches look good, I'll push them out to dlm.git. I'm curious how many resources (files) you had before you saw significant benefit from this patch?

Dave

Oh, sorry I don't know the exact resources either since this issue was reported and the patches were verified by one of the users, I haven't tested it myself;)

But I know their testing environment:
- 2-node cluster
- ocfs2 with user-space cluster stack, which depends on dlm to provide distributed locking
- CTDB as resource agent, which enabled Samba in the clustered environment
- LVS on top of the above.
and the testing they did is:
smbtorture <ocfs2_volume> --num-progs=500 BENCH-NBENCH

As far as I know, smbtorture is a testsuite for SMB server, but I haven't digged more into that to see the details of what smbtorture will do to filesystem;)

Many thanks for your review!

Thanks,
Jiaju