Skip to content

Deferred dput() and iput() -- reducing lock contention

Mike Waychison linux-fsdevel lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest8 patches 13 repliesIn review

ChangeLog from the cover letter

[v1] - Jan 16, 2009
- Initial version

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/8Deferred batching of dput()fs/dcache.c fs/super.c include/linux/dcache.h+261 -312 replies
2/8Parallel dput()fs/dcache.c+115 -410 replies
3/8Deferred batching of iput()fs/dcache.c fs/inode.c include/linux/fs.h+212 -22 replies
4/8Fixing iput() called from put_super pathfs/block_dev.c fs/ext2/super.c fs/gfs2/glock.c +6 more+24 -140 replies
5/8Parallelize iput()fs/inode.c include/linux/fs.h+167 -650 replies
6/8hugetlbfs drop_inode updatefs/hugetlbfs/inode.c+42 -300 replies
7/8Make drop_caches flush pending dput()s and iput()sfs/drop_caches.c+1 -00 replies
8/8Make the sync path drain dentries and inodesfs/sync.c+5 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH v1 0/8] Deferred dput() and iput() -- reducing lock contention

We've noticed that at times it can become very easy to have a system begin to livelock on dcache_lock/inode_lock (specifically in atomic_dec_and_lock()) when a lot of dentries are getting finalized at the same time (massive delete and large fdtable destructions are two paths I've seen cause problems).

This patchset is an attempt to try and reduce the locking overheads associated with final dput() and final iput(). This is done by batching dentries and inodes into per-process queues and processing them in 'parallel' to consolidate some of the locking.

Besides various workload testing, I threw together a load (at the end of this email) that causes massive fdtables (50K sockets by default) to get destroyed on each cpu in the system. It also populates the dcache for procfs on those tasks for good measure. Comparing lock_stat results (hardware is a Sun x4600 M2 populated with 8 4-core 2.3GHz packages (32 CPUs) + 128GiB RAM):

BEFORE PATCHSET:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
class name con-bounces contentions waittime-min waittime-max waittime-total acq-bounces acquisitions holdtime-min holdtime-max holdtime-total
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

                             dcache_lock:       5666485       10086172          11.16     2274036.47  3821690090.32       11042789       14926411           2.78       42138.28    65887138.46
                             -----------
                             dcache_lock        3016578          [<ffffffff810bd7e7>] d_alloc+0x190/0x1e1
                             dcache_lock        1844569          [<ffffffff810bcdfc>] d_instantiate+0x2c/0x51
                             dcache_lock        4223784          [<ffffffff8119bd88>] _atomic_dec_and_lock+0x3c/0x5c
                             dcache_lock         207544          [<ffffffff810bc36a>] d_rehash+0x1b/0x43
                             -----------
                             dcache_lock        2305449          [<ffffffff810bcdfc>] d_instantiate+0x2c/0x51
                             dcache_lock        1954160          [<ffffffff810bd7e7>] d_alloc+0x190/0x1e1
                             dcache_lock        4169163          [<ffffffff8119bd88>] _atomic_dec_and_lock+0x3c/0x5c
                             dcache_lock         866247          [<ffffffff810bc36a>] d_rehash+0x1b/0x43

...............................................................................................................................................................................................

                              inode_lock:        202813         203064          12.91      376655.85    37696597.26        5136095        8001156           3.37       15142.77     5033144.13
                              ----------
                              inode_lock          20192          [<ffffffff810bfdbc>] new_inode+0x27/0x63
                              inode_lock              1          [<ffffffff810c6f58>] sync_inode+0x19/0x39
                              inode_lock              5          [<ffffffff810c76b5>] __mark_inode_dirty+0xe2/0x165
                              inode_lock              2          [<ffffffff810c6e92>] __writeback_single_inode+0x1bf/0x26c
                              ----------
                              inode_lock          20198          [<ffffffff810bfdbc>] new_inode+0x27/0x63
                              inode_lock              1          [<ffffffff810c76b5>] __mark_inode_dirty+0xe2/0x165
                              inode_lock              5          [<ffffffff810c70fc>] generic_sync_sb_inodes+0x33/0x31a
                              inode_lock         165016          [<ffffffff8119bd88>] _atomic_dec_and_lock+0x3c/0x5c

...............................................................................................................................................................................................

                       &sbsec->isec_lock:         34428          34431          16.77       67593.54     3780131.15        1415432        3200261           4.06       13357.96     1180726.21

AFTER PATCHSET [Note that inode_lock doesn't even show up anymore]:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
class name con-bounces contentions waittime-min waittime-max waittime-total acq-bounces acquisitions holdtime-min holdtime-max holdtime-total
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

                             dcache_lock:       2732103        5254824          11.04     1314926.88  2187466928.10        5551173        9751332           2.43       78012.24    42830806.29
                             -----------
                             dcache_lock        3015590          [<ffffffff810bdb04>] d_alloc+0x190/0x1e1
                             dcache_lock        1966978          [<ffffffff810bd118>] d_instantiate+0x2c/0x51
                             dcache_lock         258657          [<ffffffff810bc3cd>] d_rehash+0x1b/0x43
                             dcache_lock             30          [<ffffffff810b7ad4>] __link_path_walk+0x42d/0x665
                             -----------
                             dcache_lock        2493589          [<ffffffff810bd118>] d_instantiate+0x2c/0x51
                             dcache_lock        1862921          [<ffffffff810bdb04>] d_alloc+0x190/0x1e1
                             dcache_lock         882054          [<ffffffff810bc3cd>] d_rehash+0x1b/0x43
                             dcache_lock          15504          [<ffffffff810bc5e1>] process_postponed_dentries+0x3e/0x29f

...............................................................................................................................................................................................

ChangeLog:
[v1] - Jan 16, 2009
- Initial version

Patches in series:

  1)  Deferred batching of dput()
  2)  Parallel dput()
  3)  Deferred batching of iput()
  4)  Fixing iput called from put_super path
  5)  Parallelize iput()
  6)  hugetlbfs drop_inode update
  7)  Make drop_caches flush pending dput()s and iput()s
  8)  Make the sync path drain dentries and inodes

Caveats:

* It's not clear to me how to make 'sync(2)' do what it should. Currently we
flush pending dputs and iputs before writing anything out, but presumably, there may be hidden iputs in the do_sync path that really ought to be synchronous.
* I don't like the changes in patch 4 and it should probably be changed to
have the ->put_super callback paths call a synchronous version of iput() (perhaps sync_iput()?) instead of having to tag the inodes with S_SYNCIPUT.
* cramfs, gfs2, ocfs2 need to be updated for the new drop_inode semantics.
---

 fs/block_dev.c         |    2 
 fs/dcache.c            |  374 +++++++++++++++++++++++++++++++++++++----
 fs/drop_caches.c       |    1 
 fs/ext2/super.c        |    2 
 fs/gfs2/glock.c        |    2 
 fs/gfs2/ops_fstype.c   |    2 
 fs/hugetlbfs/inode.c   |   72 +++++---
 fs/inode.c             |  441 +++++++++++++++++++++++++++++++++++++++++-------
 fs/ntfs/super.c        |    2 
 fs/smbfs/inode.c       |    2 
 fs/super.c             |    6 -
 fs/sync.c              |    5 +
 include/linux/dcache.h |    1 
 include/linux/fs.h     |   18 ++
 14 files changed, 787 insertions(+), 143 deletions(-)
Signature · 324 lines
// Copyright 2008 Google Inc. All Rights Reserved.
// Author: [email protected] (Mike Waychison)
//
//    This program is free software: you can redistribute it and/or modify
//    it under the terms of the GNU General Public License as published by
//    the Free Software Foundation, either version 3 of the License, or
//    (at your option) any later version.
//
//    This program is distributed in the hope that it will be useful,
//    but WITHOUT ANY WARRANTY; without even the implied warranty of
//    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
//    GNU General Public License for more details.
//
//    You should have received a copy of the GNU General Public License
//    along with this program.  If not, see <http://www.gnu.org/licenses/>.
//
// Description: Meant to make a kernel go nuts by stressing dcache with
// implosion.

#include <sys/mman.h>
#include <sys/resource.h>
#include <sys/select.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/time.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <dirent.h>
#include <fcntl.h>
#include <inttypes.h>
#include <sched.h>
#include <signal.h>
#include <unistd.h>
#include <stdio.h>
#include <time.h>
#include <errno.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>

#include <iostream>
#include <string>

using std::cout;
using std::string;

static char *who_am_i;

static bool wait_on_kids = true;
static int files_per_cpu = 50000;
static int wait_before_slaughter = 0;

static const int kMagicSignal = SIGUSR2;

static const int kMaxCPUs = __NCPUBITS;

typedef struct { volatile int counter; } atomic_t;
#define LOCK_PREFIX "lock "
static inline void atomic_inc(atomic_t *v)
{
  __asm__ __volatile__(
    LOCK_PREFIX "incl %0"
    :"=m" (v->counter)
    :"m" (v->counter));
}
#define atomic_read(v)		((v)->counter)
#define atomic_set(v,i)		(((v)->counter) = (i))

static void *shared_area;

static atomic_t *children_done;

#define Abort() DoAbort(__LINE__)
static void DoAbort(int line) {
  cout << "Aborted at line " << line << " errno=" << errno << "\n";
  exit(1);
}

static void InitSharedArea() {
  shared_area = mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_SHARED, 0, NULL);
  if (shared_area == MAP_FAILED)
    Abort();
  children_done = static_cast<atomic_t *>(shared_area);
  atomic_set(children_done, 0);
}

static void get_cpu_bitmap(cpu_set_t *bitmap) {
  CPU_ZERO(bitmap);
  if (sched_getaffinity(0, sizeof(*bitmap), bitmap) < 0)
    Abort();
}

static void run_child(int cpu) {
  // Start by setting our own SIGTERM handler to avoid google3 fallout
  signal(SIGTERM, SIG_DFL);

  // Pin ourselves to the cpu
  cpu_set_t cpumask;
  CPU_ZERO(&cpumask);
  CPU_SET(cpu, &cpumask);
  if (sched_setaffinity(0, sizeof(cpumask), &cpumask) < 0)
    Abort();

  // Start off by creating tons of open sockets.  Well just make dummy sockets
  // for now.
  printf("Making sockets\n");
  for (int i = 0; i < files_per_cpu; i++) {
    if (socket(PF_INET, SOCK_STREAM, 0) < 0) {
      printf("Ran out of file descriptors? errno == %d\n", errno);
      kill(getppid(), SIGABRT);
      exit(0);
    }
  }
  printf("Done making sockets\n");

  // Now populate the proc_inode_cache
  pid_t my_pid = getpid();
  char fd_path[100];
  sprintf(fd_path, "/proc/%d/fd", my_pid);
  struct stat stat_dat2;
  if (lstat(fd_path, &stat_dat2) < 0)
    Abort();
  cout << "Mode : " << stat_dat2.st_mode << "\n";
  DIR *dir = opendir(fd_path);
  if (dir == static_cast<DIR *>(NULL))
    Abort();

  printf("Done opening fd directory\n");
  dirent *dirent;
  while ((dirent = readdir(dir)) != NULL) {
    // Now stat it to make sure it gets populated in dcache
    char file_path[100];
    sprintf(file_path, "/proc/%d/fd/%s", my_pid, dirent->d_name);
    struct stat stat_dat;
    if (lstat(file_path, &stat_dat) < 0)
      Abort();
  }
  closedir(dir);
  printf("Done looking at directory\n");

  // Now tell the parent that we are done
  atomic_inc(children_done);

  // And wait around to die
  for (;;) {
    timeval tv = {1, 0};
    // Poke the parent
    if (atomic_read(children_done) != 0)
      kill(getppid(), kMagicSignal);
    select(0, NULL, NULL, NULL, &tv);
  }

  Abort();
}

static void handle_sigusr1(int signum) {
}

static void make_child(int cpu, pid_t *child_pids) {
  // Set up signal so that we know when the child is done.
  if (signal(kMagicSignal, handle_sigusr1) == SIG_ERR)
    Abort();

  pid_t pid;
  if ((pid = fork()) < 0)
    Abort();
  if (pid == 0) {
    run_child(cpu);
    return;
  }

  // Parent
  child_pids[cpu] = pid;

  // Nothing left to do for this child
}

static void crank_up_filemax() {
  int fd;
  if ((fd = open("/proc/sys/fs/file-max", O_RDWR)) < 0)
    Abort();
  char msg[100];
  sprintf(msg, "%d\n", files_per_cpu * kMaxCPUs);
  write(fd, msg, strlen(msg));
  close(fd);
}

static void UsageAndDie() {
  cout << "Usage: Hurt a kernel with fd implosion\n";
  cout << who_am_i << " [options]\n";
  cout << "  --dont_wait_on_kids : Exit before kids are dead\n";
  cout << "  --files_per_cpu <n> : How many files should be created per cpu"
          " (default 50000)\n";
  cout << "  --wait_before_slaugther <secs> : Do we sleep before killing "
          "children? (default 0)";
  exit(1);
}

static int ParseInt(char *arg)
{
  char *endp;
  int ret;
  ret = (int)strtol(arg, &endp, 0);
  if (*endp) {
    cout << "Argument '" << arg << "' not an integer\n";
    UsageAndDie();
  }
  return ret;
}

static void ParseArgs(int argc, char **argv)
{
  who_am_i = argv[0];
  argc--;
  argv++;

  while (argc) {
    if (!strcmp(argv[0], "--wait_on_kids")) {
      if (argc < 2)
        UsageAndDie();
      wait_on_kids = ParseInt(argv[1]);
      argc -= 2;
      argv += 2;
      continue;
    }
    if (!strcmp(argv[0], "--files_per_cpu")) {
      if (argc < 2)
        UsageAndDie();
      files_per_cpu = ParseInt(argv[1]);
      argc -= 2;
      argv += 2;
      continue;
    }
    if (!strcmp(argv[0], "--dont_wait_on_kids")) {
      wait_on_kids = false;
      argc -= 1;
      argv += 1;
      continue;
    }
    cout << "Don't understand option " << argv[0] << "\n";
    UsageAndDie();
  }
}
int main(int argc, char **argv) {
  ParseArgs(argc, argv);
  if (getuid() != 0) {
    cout << "This program must be run as root\n";
    Abort();
  }

  InitSharedArea();

  crank_up_filemax();

  // Up our count of allowable open files
  struct rlimit new_limits = {files_per_cpu + 20,
                              files_per_cpu + 20};
  if (setrlimit(RLIMIT_NOFILE, &new_limits) < 0)
    Abort();

  cpu_set_t cpu_bitmap;
  get_cpu_bitmap(&cpu_bitmap);

  pid_t child_pids[kMaxCPUs];

  // Build all the children
  int childCount = 0;
  for (int i = 0; i < kMaxCPUs; i++) {
    if (CPU_ISSET(i, &cpu_bitmap)) {
      cout << "About to run child on cpu" << i << "\n";
      make_child(i, child_pids);
      childCount++;
    }
  }

  // Wait for all the children to finish
  for (;;) {
    if (atomic_read(children_done) == childCount) {
      atomic_set(children_done, 0);
      break;
    }
    timeval tv = {1, 0};
    select(0, NULL, NULL, NULL, &tv);
  }

  sleep(wait_before_slaughter);

  cout << "About to start killing children\n";

  timeval before_tv;
  gettimeofday(&before_tv, NULL);

  // Now kill all the children
  for (int i = 0; i < kMaxCPUs; i++) {
    if (CPU_ISSET(i, &cpu_bitmap)) {
      kill(child_pids[i], SIGTERM);
    }
  }

  // Wait on em
  if (wait_on_kids) {
    for (int i = 0; i < kMaxCPUs; i++) {
      if (CPU_ISSET(i, &cpu_bitmap)) {
        int status;
        waitpid(child_pids[i], &status, 0);
      }
    }
  }

  timeval after_tv;
  gettimeofday(&after_tv, NULL);

  int seconds = after_tv.tv_sec - before_tv.tv_sec;
  uint64_t millisecs = seconds * 1000;
  millisecs += (after_tv.tv_usec - before_tv.tv_usec) / 1000;
  seconds = millisecs / 1000;
  millisecs %= 1000;

  char buf[100];
  sprintf(buf, "%d.%03" PRIu64, seconds, millisecs);
  cout << "do_exit path for run took " << buf << " seconds\n";

  return 0;
}