From: Paul Wankadia Subject: Re: [PATCH 1/4] Make hash table scale to thousands of entries Date: Fri, 16 Jan 2009 14:59:49 +1100 Message-ID: <408431290901151959o7c504c79k88681dddb468359f@mail.gmail.com> References: <1231526823-18595-1-git-send-email-vaurora@redhat.com> <1231983710.3100.38.camel@zeus.themaw.net> <1232032573.3100.53.camel@zeus.themaw.net> <408431290901151233q33699529r6119788c998866ed@mail.gmail.com> <408431290901151257i70400351yc3ab9360136c8e9c@mail.gmail.com> <1232065653.3166.19.camel@zeus.themaw.net> <1232066406.3166.28.camel@zeus.themaw.net> <1232070195.3166.36.camel@zeus.themaw.net> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1985382474744970208==" Return-path: In-Reply-To: <1232070195.3166.36.camel@zeus.themaw.net> List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: autofs-bounces@linux.kernel.org Errors-To: autofs-bounces@linux.kernel.org To: Ian Kent Cc: autofs@linux.kernel.org --===============1985382474744970208== Content-Type: multipart/alternative; boundary=0016e652f714b84b130460919a3c --0016e652f714b84b130460919a3c Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit On Fri, Jan 16, 2009 at 12:43 PM, Ian Kent wrote: > > > If we can simply flag the cached version as stale, then lazy reloads > > > > should be possible. > > > > > > Maybe not. > > > > > > The layer of code above the lookup modules does the re-loads but it is > > > general wrt. the map type so for a "nobrowse" map it won't read the > map. > > > I'm not sure yet that putting special case exceptions in that code is a > > > good thing to do or even straight forward. But, for the large map case, > > > > Or maybe I'm talking nonsense as it looks like I've actually got plenty > > of map type context in the layer above, hehe. > > > > > we can consider reading the entire map an expensive operation, so if we > > > did change the layer above and we found the map needed to be modified > > > then marked the map as stale we would need to scan the file map for the > > > key (as the map is out of date) and then read the whole map later. That > > > means we would end up reading the map at least one and a half times (on > > > average) instead of once (assuming no lookups between marking the map > > > stale and re-reading). > > > > > > I'm tending toward reading the map during the lookup so the map is up > to > > > date when we look for the key in the cache. > > Or maybe not. > > If we take advantage of the existing code to mark the map as stale and > handle the re-read and we accept the overhead of scanning the file map > while waiting for the update something like this could be all that's > needed. > > autofs-5.0.4 - always read file maps > > From: Ian Kent > > > --- > > daemon/lookup.c | 9 ++++++--- > modules/lookup_file.c | 11 ++++++++++- > 2 files changed, 16 insertions(+), 4 deletions(-) > > > diff --git a/daemon/lookup.c b/daemon/lookup.c > index 741d846..b954045 100644 > --- a/daemon/lookup.c > +++ b/daemon/lookup.c > @@ -283,10 +283,13 @@ static int do_read_map(struct autofs_point *ap, > struct map_source *map, time_t a > * for the fail cases to function correctly and to cache the > * lookup handle. > * > - * We always need to whole map for direct mounts in order to > - * mount the triggers. > + * We always need to read the whole map for direct mounts in > + * order to mount the triggers. We also want to read the whole > + * map if it's a file map to avoid potentially lengthy linear > + * file scanning. > */ > - if (!(ap->flags & MOUNT_FLAG_GHOST) && ap->type != LKP_DIRECT) > + if (strcmp(map->type, "file") || I suspect that you want && instead. + (!(ap->flags & MOUNT_FLAG_GHOST) && ap->type != LKP_DIRECT)) > return NSS_STATUS_SUCCESS; > > if (!map->stale) > diff --git a/modules/lookup_file.c b/modules/lookup_file.c > index 95b9f6f..9a1c39b 100644 > --- a/modules/lookup_file.c > +++ b/modules/lookup_file.c > @@ -93,7 +93,6 @@ int lookup_init(const char *mapfmt, int argc, const char > *const *argv, void **co > argv[0]); > return 1; > } > - > ctxt->mtime = st.st_mtime; > > if (!mapfmt) > @@ -1064,7 +1063,16 @@ int lookup_mount(struct autofs_point *ap, const char > *name, int name_len, void * > if (ap->type == LKP_INDIRECT && *key != '/') { > char *lkp_key; > > + /* > + * We can skip the map lookup and cache update altogether > + * if we know the map hasn't been modified since it was > + * last read. > + */ > cache_readlock(mc); > + me = cache_lookup_first(mc); > + if (me && ctxt->mtime <= me->age) You have to call stat(2) or fstat(2) at some point or else ctxt->mtime will never change. + goto do_lookup; My brain is bleeding. + > me = cache_lookup_distinct(mc, key); > if (me && me->multi) > lkp_key = strdup(me->multi->key); > @@ -1088,6 +1096,7 @@ int lookup_mount(struct autofs_point *ap, const char > *name, int name_len, void * > } > > cache_readlock(mc); > +do_lookup: > me = cache_lookup(mc, key); > /* Stale mapent => check for entry in alternate source or wildcard > */ > if (me && !me->mapent) { > > > --0016e652f714b84b130460919a3c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
On Fri, Jan 16, 2009 at 12:43 PM, Ian Kent <raven@themaw.net&g= t; wrote:

> > > If we can simply flag the cached version as stale, then lazy= reloads
> > > should be possible.
> >
> > Maybe not.
> >
> > The layer of code above the lookup modules does the re-loads but = it is
> > general wrt. the map type so for a "nobrowse" map it wo= n't read the map.
> > I'm not sure yet that putting special case exceptions in that= code is a
> > good thing to do or even straight forward. But, for the large map= case,
>
> Or maybe I'm talking nonsense as it looks like I've actually g= ot plenty
> of map type context in the layer above, hehe.
>
> > we can consider reading the entire map an expensive operation, so= if we
> > did change the layer above and we found the map needed to be modi= fied
> > then marked the map as stale we would need to scan the file map f= or the
> > key (as the map is out of date) and then read the whole map later= . That
> > means we would end up reading the map at least one and a half tim= es (on
> > average) instead of once (assuming no lookups between marking the= map
> > stale and re-reading).
> >
> > I'm tending toward reading the map during the lookup so the m= ap is up to
> > date when we look for the key in the cache.

Or maybe not.

If we take advantage of the existing code to mark the map as stale and
handle the re-read and we accept the overhead of scanning the file map
while waiting for the update something like this could be all that's needed.

autofs-5.0.4 - always read file maps

From: Ian Kent <raven@themaw.net= >


---

 daemon/lookup.c       |    9 ++++++---
 modules/lookup_file.c |   11 ++++++++++-
 2 files changed, 16 insertions(+), 4 deletions(-)


diff --git a/daemon/lookup.c b/daemon/lookup.c
index 741d846..b954045 100644
--- a/daemon/lookup.c
+++ b/daemon/lookup.c
@@ -283,10 +283,13 @@ static int do_read_map(struct autofs_point *ap, struc= t map_source *map, time_t a
        * for the fail cases to function correctly and= to cache the
        * lookup handle.
        *
-        * We always need to whole map for direct mount= s in order to
-        * mount the triggers.
+        * We always need to read the whole map for dir= ect mounts in
+        * order to mount the triggers. We also want to= read the whole
+        * map if it's a file map to avoid potentia= lly lengthy linear
+        * file scanning.
        */
-       if (!(ap->flags & MOUNT_FLAG_GHOST) &&= ; ap->type !=3D LKP_DIRECT)
+       if (strcmp(map->type, "file") ||

I suspect that you want && instead.

+           (!(ap->flags & MOUNT_FLAG_GHOST= ) && ap->type !=3D LKP_DIRECT))
               return NSS_STATUS_S= UCCESS;

       if (!map->stale)
diff --git a/modules/lookup_file.c b/modules/lookup_file.c
index 95b9f6f..9a1c39b 100644
--- a/modules/lookup_file.c
+++ b/modules/lookup_file.c
@@ -93,7 +93,6 @@ int lookup_init(const char *mapfmt, int argc, const char = *const *argv, void **co
                    argv= [0]);
               return 1;
       }
-
       ctxt->mtime =3D st.st_mtime;

       if (!mapfmt)
@@ -1064,7 +1063,16 @@ int lookup_mount(struct autofs_point *ap, const char= *name, int name_len, void *
       if (ap->type =3D=3D LKP_INDIRECT && = *key !=3D '/') {
               char *lkp_key;

+               /*
+                * We can skip the = map lookup and cache update altogether
+                * if we know the m= ap hasn't been modified since it was
+                * last read.
+                */
               cache_readlock(mc);=
+               me =3D cache_lookup_firs= t(mc);
+               if (me && ctxt-&= gt;mtime <=3D me->age)

You have to call stat(2) = or fstat(2) at some point or else ctxt->mtime will never change.

=
+                     &nb= sp; goto do_lookup;

My brain is bleeding.

+
               me =3D cache_lookup= _distinct(mc, key);
               if (me && m= e->multi)
                    &nbs= p;  lkp_key =3D strdup(me->multi->key);
@@ -1088,6 +1096,7 @@ int lookup_mount(struct autofs_point *ap, const char = *name, int name_len, void *
       }

       cache_readlock(mc);
+do_lookup:
       me =3D cache_lookup(mc, key);
       /* Stale mapent =3D> check for entry in alte= rnate source or wildcard */
       if (me && !me->mapent) {



--0016e652f714b84b130460919a3c-- --===============1985382474744970208== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ autofs mailing list autofs@linux.kernel.org http://linux.kernel.org/mailman/listinfo/autofs --===============1985382474744970208==--