Skip to contentSkip to conversation

stat of /users/no-such-user takes 15 seconds

DISCUSSION Brian Long 5 participants 31 messages Nov 22, 2005 – Dec 1, 2005

Subsystems (2)
lore
Share & tools
lore

Brian Long

author
#0 lore raw

I have an auto.master deployed to thousands of hosts and it looks like this:

/misc   /etc/auto.misc          --timeout=60
/auto   /etc/auto.indirect      rsize=32768,wsize=32768,tcp
/users  auto_home               rw,hard,intr,rsize=32768,wsize=32768,tcp

As you can see, /users comes from NIS auto.home map. In our case, auto.home contains over 34,000 entries. I've noticed in RHEL 3 U5 and beyond (autofs-4.1.3-130), trying to stat (ls) /users/no-such-user takes roughly 12-15 seconds. In RHEL 3 U3 (autofs-4.1.3-12), it returns immediately. This is a regression in my opinion.

After looking at tcpdump data, it appears automounter is downloading the entire auto.home map when it fails to lookup "no-such-user". This results in a > 700KB transfer from the NIS server :(

In the older autofs version, it does NOT do this. I have opened a Red Hat issue tracker (82760), but in parallel I wanted to ask this list if this is known behavior and if there is any way I can disable the download of the entire auto.home map using /etc/sysconfig/autofs or auto.master options to the automount program.

Thank you.

/Brian/

Signature · 4 lines
       Brian Long                      |         |           |
       IT Data Center Systems          |       .|||.       .|||.
       Cisco Linux Developer           |   ..:|||||||:...:|||||||:..
       Phone: (919) 392-7363           |   C i s c o   S y s t e m s
author
#1 lore raw
Quoted 17 lines · Brian Long

I have an auto.master deployed to thousands of hosts and it looks like this:

/misc   /etc/auto.misc          --timeout=60
/auto   /etc/auto.indirect      rsize=32768,wsize=32768,tcp
/users  auto_home               rw,hard,intr,rsize=32768,wsize=32768,tcp

As you can see, /users comes from NIS auto.home map. In our case, auto.home contains over 34,000 entries. I've noticed in RHEL 3 U5 and beyond (autofs-4.1.3-130), trying to stat (ls) /users/no-such-user takes roughly 12-15 seconds. In RHEL 3 U3 (autofs-4.1.3-12), it returns immediately. This is a regression in my opinion.

After looking at tcpdump data, it appears automounter is downloading the entire auto.home map when it fails to lookup "no-such-user". This results in a > 700KB transfer from the NIS server :(

Quoted 5 lines

In the older autofs version, it does NOT do this. I have opened a Red Hat issue tracker (82760), but in parallel I wanted to ask this list if this is known behavior and if there is any way I can disable the download of the entire auto.home map using /etc/sysconfig/autofs or auto.master options to the automount program.

FYI, in looking at the strace output from the automount process once it downloads auto.home, it goes through all 34,000+ entries and tries to rmdir /users/<entry>. This fails 34,000 times. Why the heck would it behave like this?

/Brian/

Signature · 4 lines
       Brian Long                      |         |           |
       IT Data Center Systems          |       .|||.       .|||.
       Cisco Linux Developer           |   ..:|||||||:...:|||||||:..
       Phone: (919) 392-7363           |   C i s c o   S y s t e m s
maintainer
#2 lore raw
Quoted 24 lines · Brian Long
Quoted 17 lines · Brian Long

I have an auto.master deployed to thousands of hosts and it looks like this:

/misc   /etc/auto.misc          --timeout=60
/auto   /etc/auto.indirect      rsize=32768,wsize=32768,tcp
/users  auto_home               rw,hard,intr,rsize=32768,wsize=32768,tcp

As you can see, /users comes from NIS auto.home map. In our case, auto.home contains over 34,000 entries. I've noticed in RHEL 3 U5 and beyond (autofs-4.1.3-130), trying to stat (ls) /users/no-such-user takes roughly 12-15 seconds. In RHEL 3 U3 (autofs-4.1.3-12), it returns immediately. This is a regression in my opinion.

After looking at tcpdump data, it appears automounter is downloading the entire auto.home map when it fails to lookup "no-such-user". This results in a > 700KB transfer from the NIS server :(

Quoted 5 lines

In the older autofs version, it does NOT do this. I have opened a Red Hat issue tracker (82760), but in parallel I wanted to ask this list if this is known behavior and if there is any way I can disable the download of the entire auto.home map using /etc/sysconfig/autofs or auto.master options to the automount program.

Quoted 4 lines

FYI, in looking at the strace output from the automount process once it downloads auto.home, it goes through all 34,000+ entries and tries to rmdir /users/<entry>. This fails 34,000 times. Why the heck would it behave like this?

Wasn't aware I did that. That's something I'll have to sort out.

Ian

#3 lore raw
Quoted 3 lines · Ian Kent
Quoted 2 lines · Brian Long
Quoted 1 line · Brian Long

I have an auto.master deployed to thousands of hosts and it looks like

-- snip --

Quoted 1 line
Quoted 1 line
Quoted 1 line

auto.home contains over 34,000 entries. I've noticed in RHEL 3 U5 and

-- snip --

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

After looking at tcpdump data, it appears automounter is downloading the entire auto.home map when it fails to lookup "no-such-user". This results in a > 700KB transfer from the NIS server :(

I may have missed something in subsequent discussion, but why isn't automount querying just the missing map row? If NIS agreed that it did not exist, fine, report an error to the client's opendir(). If it *did* exist, the new information could be added immediately and the filesystem could be mounted promptly.

Raven asked what people thought should be the semantics when a map is proven to be out of date. If there were no constraints from other parts of automount, I would have a fast path like described above, but to detect deletions I would have automount poll the order number (modification timestamp) from the server every so often -- once an hour? If the order number has changed, re-enumerate the map. To minimize thrashing, if possible, I'd do a "diff" between the latest map and the kernel's data: add rows absent in the kernel, remove rows absent in the map, and don't touch rows present in both (unless per-row mount options had changed).

If the fast path's NIS query revealed a map row we didn't know about, early polling should be triggered, in case there were other additions or deletions. This only applies if browsing is turned on. Or since userspace has to obtain the whole map anyway to recognize deletions, maybe it should cache the map, but only in the case of browsing should it install every row in the kernel.

Suppose mounting is attempted but fails. I think automount should also trigger early polling. This costs you a query for one row (the order number), and if the map has changed then your nonexistent mount point will go away -- or not go away, cheaply, if the problem is with the NFS server.

Remember that NIS UDP timeouts are a fact of life on an overloaded network.

Browsing is nice but gets expensive if there are 34,000 potential mount points. If browsing is turned off, the code that enumerates the map should delete rows in the kernel but not in the map, but should bypass adding rows in the map but not the kernel. Could this be part of the cause of the original problem -- browsing being turned on inappropriately?

James F. Carter Voice 310 825 2897 FAX 310 206 6673
UCLA-Mathnet; 6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA 90095-1555
Email: [email protected] http://www.math.ucla.edu/~jimc (q.v. for PGP key)

#4 lore raw
Quoted 3 lines · [email protected] (Jim Carter) writes:

[...] Or since userspace has to obtain the whole map anyway to recognize deletions, maybe it should cache the map, but only in the case of browsing should it install every row in the kernel.

I believe that LCUP (LDAP Client Update Protocol, RFC 3928) could be used for this purpose. This LDAP extension would make it possible for a client to request the server to notify it of any modifications to a map, including deletion of entries.

This only applies to LDAP of course, and far from all LDAP servers support LCUP. However, situations like this look like just what LCUP was designed for.

Thorild Selén
Datorföreningen Update / Update Computer Club, Uppsala, SE

maintainer
#5 lore raw
Quoted 5 lines · Thorild Selen
Quoted 3 lines · [email protected] (Jim Carter) writes:

[...] Or since userspace has to obtain the whole map anyway to recognize deletions, maybe it should cache the map, but only in the case of browsing should it install every row in the kernel.

Quoted 8 lines

I believe that LCUP (LDAP Client Update Protocol, RFC 3928) could be used for this purpose. This LDAP extension would make it possible for a client to request the server to notify it of any modifications to a map, including deletion of entries.

This only applies to LDAP of course, and far from all LDAP servers support LCUP. However, situations like this look like just what LCUP was designed for.

But, at this stage, autofs isn't going to be listening it's going to be asking?

Ian

#6 lore raw
Quoted 2 lines · Ian Kent

But, at this stage, autofs isn't going to be listening it's going to be asking?

LCUP provides a syncOnly mode if you don't want to listen persistently but instead occasionally ask for changes. If I've understood it correctly, it would work basically like this:

1. The automounter downloads a table from the LDAP server, attaching a Sync Request control (with no synchronisation cookie) to the search request.

2. The LDAP server returns the table, together with a synchronisation cookie.

3. Later, the automounter may ask the server: "What changes have been made to this table since you handed out this synchronisation cookie?" (Yes, I know, there is no such thing as a table from the LDAP server's point of view, but you get the point.) The server will then tell the client about any changes since then, including added/deleted entries, and return a new synchronisation cookie to be used in the next sync request. Repeat each time we want to check for modifications.

NOTE: I believe the server is not required to keep this synchronisation state around. The server may refuse, in which case we go back to 1. and download the table from scratch.

The advantage with this would be that the automounter can keep its tables perfectly up to date without having to retrieve the tables in their entirety over and over again.

(Alternatively, LCUP has a persistent mode where you constantly listen for changes instead of requesting them when required. That would probably not scale as well as using the syncOnly mode.)

Thorild Selén
Datorföreningen Update / Update Computer Club, Uppsala, SE

maintainer
#7 lore raw

On Fri, 25 Nov 2005, Thorild Selen wrote:

Thanks for this info.

Quoted 7 lines
Quoted 2 lines · Ian Kent

But, at this stage, autofs isn't going to be listening it's going to be asking?

LCUP provides a syncOnly mode if you don't want to listen persistently but instead occasionally ask for changes. If I've understood it correctly, it would work basically like this:

OK. I see.

I'll investigate. I can't say when I'll get to this as I've got a bunch of stuff that just keeps building up while I'm working on missing basic features.

Ian

author
#8 lore raw
In reply to Jim Carter
Quoted 11 lines · Jim Carter
Quoted 3 lines · Ian Kent
Quoted 2 lines · Brian Long
Quoted 1 line · Brian Long

I have an auto.master deployed to thousands of hosts and it looks like

-- snip --

Quoted 1 line
Quoted 1 line
Quoted 1 line

auto.home contains over 34,000 entries. I've noticed in RHEL 3 U5 and

-- snip --

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

After looking at tcpdump data, it appears automounter is downloading the entire auto.home map when it fails to lookup "no-such-user". This results in a > 700KB transfer from the NIS server :(

Quoted 5 lines

I may have missed something in subsequent discussion, but why isn't automount querying just the missing map row? If NIS agreed that it did not exist, fine, report an error to the client's opendir(). If it *did* exist, the new information could be added immediately and the filesystem could be mounted promptly.

Exactly; I believe this is a bug in the cache code as Jeff Moyer mentioned (and provided a patch for).

Quoted 5 lines

Browsing is nice but gets expensive if there are 34,000 potential mount points. If browsing is turned off, the code that enumerates the map should delete rows in the kernel but not in the map, but should bypass adding rows in the map but not the kernel. Could this be part of the cause of the original problem -- browsing being turned on inappropriately?

To my knowledge, browsing is not enabled by default. We definitely do not have it enabled in our environment on purpose.

/Brian/

Signature · 4 lines
       Brian Long                      |         |           |
       IT Data Center Systems          |       .|||.       .|||.
       Cisco Linux Developer           |   ..:|||||||:...:|||||||:..
       Phone: (919) 392-7363           |   C i s c o   S y s t e m s
maintainer
#9 lore raw
Quoted 12 lines · Brian Long
Quoted 11 lines · Jim Carter
Quoted 3 lines · Ian Kent
Quoted 2 lines · Brian Long
Quoted 1 line · Brian Long

I have an auto.master deployed to thousands of hosts and it looks like

-- snip --

Quoted 1 line
Quoted 1 line
Quoted 1 line

auto.home contains over 34,000 entries. I've noticed in RHEL 3 U5 and

-- snip --

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

After looking at tcpdump data, it appears automounter is downloading the entire auto.home map when it fails to lookup "no-such-user". This results in a > 700KB transfer from the NIS server :(

Quoted 8 lines
Quoted 5 lines

I may have missed something in subsequent discussion, but why isn't automount querying just the missing map row? If NIS agreed that it did not exist, fine, report an error to the client's opendir(). If it *did* exist, the new information could be added immediately and the filesystem could be mounted promptly.

Exactly; I believe this is a bug in the cache code as Jeff Moyer mentioned (and provided a patch for).

That's not quite the way it works.

autofs does query only the entry being requested when a mount is triggered. It's not possible to update the information there and then because the lookup is done in a subprocess while the autofs main program continues. So it's necessary to tell the parent process it needs to update the map info. Also, if there's one change you don't know how many others there might be.

Judging by the report the case that I don't handle correctly is the missing from NIS and not present in the mapent cache.

The patch I asked Jeff to review deals with this and another bug I noticed while checking out the reported problem. Please have a read of the description attached to the patch.

Most of the time mounts are not triggered by a readdir, they are triggered from the VFS lookup or revalidate callbacks. These can come from a number of sources such as chdir open or other calls that need to walk a file path.

Quoted 9 lines
Quoted 6 lines

Browsing is nice but gets expensive if there are 34,000 potential mount points. If browsing is turned off, the code that enumerates the map should delete rows in the kernel but not in the map, but should bypass adding rows in the map but not the kernel. Could this be part of the cause of the original problem -- browsing being turned on inappropriately?

To my knowledge, browsing is not enabled by default. We definitely do not have it enabled in our environment on purpose.

This is an interesting point.

Context is not held in the kernel except for the directories that are created in the autofs filesystem to provide a browsable map. Certainly a large amount of directories can slow things down heaps. That could certainly be a problem for the future.

Yes, browsing is turned of by default.

Ian

#10 lore raw
In reply to Brian Long

==> Regarding [autofs] stat of /users/no-such-user takes 15 seconds; Brian Long <[email protected]> adds:

brilong> I have an auto.master deployed to thousands of hosts and it looks brilong> like this:

brilong> /misc /etc/auto.misc --timeout=60 /auto /etc/auto.indirect brilong> rsize=32768,wsize=32768,tcp /users auto_home brilong> rw,hard,intr,rsize=32768,wsize=32768,tcp

brilong> As you can see, /users comes from NIS auto.home map. In our case, brilong> auto.home contains over 34,000 entries. I've noticed in RHEL 3 U5 brilong> and beyond (autofs-4.1.3-130), trying to stat (ls) brilong> /users/no-such-user takes roughly 12-15 seconds. In RHEL 3 U3 brilong> (autofs-4.1.3-12), it returns immediately. This is a regression brilong> in my opinion.

brilong> After looking at tcpdump data, it appears automounter is brilong> downloading the entire auto.home map when it fails to lookup brilong> "no-such-user". This results in a > 700KB transfer from the NIS brilong> server :(

brilong> In the older autofs version, it does NOT do this. I have opened a brilong> Red Hat issue tracker (82760), but in parallel I wanted to ask brilong> this list if this is known behavior and if there is any way I can brilong> disable the download of the entire auto.home map using brilong> /etc/sysconfig/autofs or auto.master options to the automount brilong> program.

This is likely a side-effect of the cache expiry code. Try this patch on for size.

-Jeff

modules/lookup_yp.c renamed from modules/lookup_yp.c.orig +10 -4
@@ -224,7 +224,7 @@ int lookup_mount(const char *root, const
 	int key_len;
 	char *mapent;
 	int mapent_len;
-	struct mapent_cache *me;
+	struct mapent_cache *me, *exists;
 	int ret;
 	time_t now = time(NULL);
 	time_t t_last_read;
@@ -240,7 +240,13 @@ int lookup_mount(const char *root, const
 	if (key_len > KEY_MAX_LEN)
 		return 1;
 
-	/* check map and if change is detected re-read map */
+	/*
+	 * check map and if change is detected re-read map
+	 */
+
+	/* First check to see if this entry exists in the cache */
+	exists = cache_lookup(key);
+
 	ret = lookup_one(root, key, key_len, ctxt);
 
 	debug("ret = %d", ret);
@@ -270,8 +276,8 @@ int lookup_mount(const char *root, const
 		if (ap.type == LKP_INDIRECT)
 			lookup_wild(root, ctxt);
 
-		/* Have parent update its map */
-		if (t_last_read > ap.exp_runfreq)
+		/* Have parent update its map if the key existed before */
+		if (exists && t_last_read > ap.exp_runfreq)
 			kill(getppid(), SIGHUP);
 	}
author
#11 lore raw
Quoted 2 lines · Jeff Moyer

This is likely a side-effect of the cache expiry code. Try this patch on for size.

I took the SRPM from autofs-4.1.3-154 (RHEL 3 U6) and applied your patch. I installed this RPM on a test host, rebooted the host and ls /users/nosuchuser came back immediately.

Is this patch something that would be accepted upstream Ian? Jeff, any chance this patch can be put into RHEL 3 U7 and RHEL 4 U3? I know it's last-minute, but I really need this fixed (and supported by Red Hat).

If there is any way you can poke at my IT 82760, I would greatly appreciate it.

/Brian/

Signature · 4 lines
       Brian Long                      |         |           |
       IT Data Center Systems          |       .|||.       .|||.
       Cisco Linux Developer           |   ..:|||||||:...:|||||||:..
       Phone: (919) 392-7363           |   C i s c o   S y s t e m s
maintainer
#12 lore raw
Quoted 5 lines · Brian Long
Quoted 2 lines · Jeff Moyer

This is likely a side-effect of the cache expiry code. Try this patch on for size.

Quoted 7 lines

I took the SRPM from autofs-4.1.3-154 (RHEL 3 U6) and applied your patch. I installed this RPM on a test host, rebooted the host and ls /users/nosuchuser came back immediately.

Is this patch something that would be accepted upstream Ian? Jeff, any chance this patch can be put into RHEL 3 U7 and RHEL 4 U3? I know it's last-minute, but I really need this fixed (and supported by Red Hat).

It's never a problem including this type of fix once we've decided the root cause which is happening now. And then this also needs to be done for the LDAP lookup module and probably best to do same for file lookup module and tested .....

The real problem for me is time and keeping track of the bug fixes for the different autofs versions and which patches have been applied while there in my queue for the next version.

Ian

#13 lore raw

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

Quoted 5 lines · Brian Long
Quoted 5 lines · Jeff Moyer
Quoted 2 lines

This is likely a side-effect of the cache expiry code. Try this patch

on > for size.

Quoted 7 lines
Quoted 7 lines

I took the SRPM from autofs-4.1.3-154 (RHEL 3 U6) and applied your patch. I installed this RPM on a test host, rebooted the host and ls /users/nosuchuser came back immediately.

Is this patch something that would be accepted upstream Ian? Jeff, any chance this patch can be put into RHEL 3 U7 and RHEL 4 U3? I know it's last-minute, but I really need this fixed (and supported by Red Hat).

raven> It's never a problem including this type of fix once we've decided raven> the root cause which is happening now. And then this also needs to raven> be done for the LDAP lookup module and probably best to do same for raven> file lookup module and tested .....

raven> The real problem for me is time and keeping track of the bug fixes raven> for the different autofs versions and which patches have been raven> applied while there in my queue for the next version.

And this is important. Perhaps we should limit the bug reports here to those reported on the upstream variant of autofs, currently 4.1.4 (and potentially with some patches from your ftp site).

I would be completely okay with that.

-Jeff

maintainer
#14 lore raw
Quoted 21 lines · Jeff Moyer

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

Quoted 5 lines · Brian Long
Quoted 5 lines · Jeff Moyer
Quoted 2 lines

This is likely a side-effect of the cache expiry code. Try this patch

on > for size.

Quoted 7 lines
Quoted 7 lines

I took the SRPM from autofs-4.1.3-154 (RHEL 3 U6) and applied your patch. I installed this RPM on a test host, rebooted the host and ls /users/nosuchuser came back immediately.

Is this patch something that would be accepted upstream Ian? Jeff, any chance this patch can be put into RHEL 3 U7 and RHEL 4 U3? I know it's last-minute, but I really need this fixed (and supported by Red Hat).

raven> It's never a problem including this type of fix once we've decided raven> the root cause which is happening now. And then this also needs to raven> be done for the LDAP lookup module and probably best to do same for raven> file lookup module and tested .....

Quoted 7 lines

raven> The real problem for me is time and keeping track of the bug fixes raven> for the different autofs versions and which patches have been raven> applied while there in my queue for the next version.

And this is important. Perhaps we should limit the bug reports here to those reported on the upstream variant of autofs, currently 4.1.4 (and potentially with some patches from your ftp site).

Something like that would be good, however people will likely still post and it would be poor relations to turn them down.

The understanding that here fixes are always going to be for the current version and that issues with previous versions are discussion only.

Ian

maintainer
#15 lore raw
In reply to Brian Long
Quoted 7 lines · Brian Long

I have an auto.master deployed to thousands of hosts and it looks like this:

/misc   /etc/auto.misc          --timeout=60
/auto   /etc/auto.indirect      rsize=32768,wsize=32768,tcp
/users  auto_home               rw,hard,intr,rsize=32768,wsize=32768,tcp
Quoted 5 lines

As you can see, /users comes from NIS auto.home map. In our case, auto.home contains over 34,000 entries. I've noticed in RHEL 3 U5 and beyond (autofs-4.1.3-130), trying to stat (ls) /users/no-such-user takes roughly 12-15 seconds. In RHEL 3 U3 (autofs-4.1.3-12), it returns immediately. This is a regression in my opinion.

Bummer. I thought that update code was OK.

Quoted 4 lines

After looking at tcpdump data, it appears automounter is downloading the entire auto.home map when it fails to lookup "no-such-user". This results in a > 700KB transfer from the NIS server :(

This is exactly the sort of problem I was concerned about when people asked for periodic reload of maps. I didn't do that and clearly I was right.

Having said that it possibly shouldn't be doing this either. I'll have to check the lookup code to see what I'm doing.

Probably I'm signaling a re-load because, since the entry is now missing, the map may need to be updated.

I hasten to point out there was quite a bit of demand for having "up to date maps"! Having a HUP signal to force a re-load wasn't enough.

Given your situation and the fact that we need to know about changes, what do you think the semantics of handling map updates should be?

Seems to me there are only two options when an out of date map entry is encountered. Update, remove or add the given entry and accept there may be more you don't know about or re-read the whole map. Now even the simple addition and removal is not simple as the lookup is done in a subprocess so somehow the process leader needs to be told what to get!

Did someone ask what would be the advantages of turning autofs into a threaded app?

Quoted 6 lines

In the older autofs version, it does NOT do this. I have opened a Red Hat issue tracker (82760), but in parallel I wanted to ask this list if this is known behavior and if there is any way I can disable the download of the entire auto.home map using /etc/sysconfig/autofs or auto.master options to the automount program.

I'm sure that the folks at RedHat will help us with this.

There's nothing atm but perhaps an environment variable to tell autofs whether to be lazy about updates. If I can just come up with a simple way to tell the process leader what it needs to do and, if only a record update, the key???

I'll think about it.

Ian

#16 lore raw

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

Quoted 7 lines · Brian Long
Quoted 7 lines

I have an auto.master deployed to thousands of hosts and it looks like this:

/misc /etc/auto.misc --timeout=60 /auto /etc/auto.indirect rsize=32768,wsize=32768,tcp /users auto_home rw,hard,intr,rsize=32768,wsize=32768,tcp

Quoted 5 lines
Quoted 5 lines

As you can see, /users comes from NIS auto.home map. In our case, auto.home contains over 34,000 entries. I've noticed in RHEL 3 U5 and beyond (autofs-4.1.3-130), trying to stat (ls) /users/no-such-user takes roughly 12-15 seconds. In RHEL 3 U3 (autofs-4.1.3-12), it returns immediately. This is a regression in my opinion.

raven> Bummer. I thought that update code was OK.

Quoted 3 lines
Quoted 3 lines

After looking at tcpdump data, it appears automounter is downloading the entire auto.home map when it fails to lookup "no-such-user". This results in a > 700KB transfer from the NIS server :(

raven> This is exactly the sort of problem I was concerned about when raven> people asked for periodic reload of maps. I didn't do that and raven> clearly I was right.

raven> Having said that it possibly shouldn't be doing this either. I'll raven> have to check the lookup code to see what I'm doing.

raven> Probably I'm signaling a re-load because, since the entry is now raven> missing, the map may need to be updated.

raven> I hasten to point out there was quite a bit of demand for having "up raven> to date maps"! Having a HUP signal to force a re-load wasn't enough.

raven> Given your situation and the fact that we need to know about raven> changes, what do you think the semantics of handling map updates raven> should be?

raven> Seems to me there are only two options when an out of date map entry raven> is encountered. Update, remove or add the given entry and accept raven> there may be more you don't know about or re-read the whole map. Now raven> even the simple addition and removal is not simple as the lookup is raven> done in a subprocess so somehow the process leader needs to be told raven> what to get!

Ian, the point here is that the map key isn't out of date. We don't detect that the entry didn't exist in the first place. See my other post to this thread, where I've attached a patch. Comments on that are, of course, encouraged.

-Jeff

maintainer
#17 lore raw
Quoted 46 lines · Jeff Moyer

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

Quoted 7 lines · Brian Long
Quoted 7 lines

I have an auto.master deployed to thousands of hosts and it looks like this:

/misc /etc/auto.misc --timeout=60 /auto /etc/auto.indirect rsize=32768,wsize=32768,tcp /users auto_home rw,hard,intr,rsize=32768,wsize=32768,tcp

Quoted 5 lines
Quoted 5 lines

As you can see, /users comes from NIS auto.home map. In our case, auto.home contains over 34,000 entries. I've noticed in RHEL 3 U5 and beyond (autofs-4.1.3-130), trying to stat (ls) /users/no-such-user takes roughly 12-15 seconds. In RHEL 3 U3 (autofs-4.1.3-12), it returns immediately. This is a regression in my opinion.

raven> Bummer. I thought that update code was OK.

Quoted 3 lines
Quoted 3 lines

After looking at tcpdump data, it appears automounter is downloading the entire auto.home map when it fails to lookup "no-such-user". This results in a > 700KB transfer from the NIS server :(

raven> This is exactly the sort of problem I was concerned about when raven> people asked for periodic reload of maps. I didn't do that and raven> clearly I was right.

raven> Having said that it possibly shouldn't be doing this either. I'll raven> have to check the lookup code to see what I'm doing.

raven> Probably I'm signaling a re-load because, since the entry is now raven> missing, the map may need to be updated.

raven> I hasten to point out there was quite a bit of demand for having "up raven> to date maps"! Having a HUP signal to force a re-load wasn't enough.

raven> Given your situation and the fact that we need to know about raven> changes, what do you think the semantics of handling map updates raven> should be?

raven> Seems to me there are only two options when an out of date map entry raven> is encountered. Update, remove or add the given entry and accept raven> there may be more you don't know about or re-read the whole map. Now raven> even the simple addition and removal is not simple as the lookup is raven> done in a subprocess so somehow the process leader needs to be told raven> what to get!

Quoted 4 lines

Ian, the point here is that the map key isn't out of date. We don't detect that the entry didn't exist in the first place. See my other post to this thread, where I've attached a patch. Comments on that are, of course, encouraged.

Think I get it now.

Your point is that the case where the map lookup to the server fails to find the key and the entry doesn't exist in the local entry database is not handled, yes.

Ian

maintainer
#18 lore raw
In reply to Jeff Moyer
Quoted 46 lines · Jeff Moyer

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

Quoted 7 lines · Brian Long
Quoted 7 lines

I have an auto.master deployed to thousands of hosts and it looks like this:

/misc /etc/auto.misc --timeout=60 /auto /etc/auto.indirect rsize=32768,wsize=32768,tcp /users auto_home rw,hard,intr,rsize=32768,wsize=32768,tcp

Quoted 5 lines
Quoted 5 lines

As you can see, /users comes from NIS auto.home map. In our case, auto.home contains over 34,000 entries. I've noticed in RHEL 3 U5 and beyond (autofs-4.1.3-130), trying to stat (ls) /users/no-such-user takes roughly 12-15 seconds. In RHEL 3 U3 (autofs-4.1.3-12), it returns immediately. This is a regression in my opinion.

raven> Bummer. I thought that update code was OK.

Quoted 3 lines
Quoted 3 lines

After looking at tcpdump data, it appears automounter is downloading the entire auto.home map when it fails to lookup "no-such-user". This results in a > 700KB transfer from the NIS server :(

raven> This is exactly the sort of problem I was concerned about when raven> people asked for periodic reload of maps. I didn't do that and raven> clearly I was right.

raven> Having said that it possibly shouldn't be doing this either. I'll raven> have to check the lookup code to see what I'm doing.

raven> Probably I'm signaling a re-load because, since the entry is now raven> missing, the map may need to be updated.

raven> I hasten to point out there was quite a bit of demand for having "up raven> to date maps"! Having a HUP signal to force a re-load wasn't enough.

raven> Given your situation and the fact that we need to know about raven> changes, what do you think the semantics of handling map updates raven> should be?

raven> Seems to me there are only two options when an out of date map entry raven> is encountered. Update, remove or add the given entry and accept raven> there may be more you don't know about or re-read the whole map. Now raven> even the simple addition and removal is not simple as the lookup is raven> done in a subprocess so somehow the process leader needs to be told raven> what to get!

Quoted 4 lines

Ian, the point here is that the map key isn't out of date. We don't detect that the entry didn't exist in the first place. See my other post to this thread, where I've attached a patch. Comments on that are, of course, encouraged.

Could you review this patch please Jeff.

I will try to find some time to setup and test it on the weekend.

==============

This patch addresses two problems.

Change cache_add to cache_update in lookup modules. Perform an update rather than an add so that only stale entries are actually removed instead of adding an additional entry then removing the old one. This add/remove causes a directory removal of almost every entry in the map when it should'nt.

Prevent re-read of map and return fail from the lookup module when a key is not present in the cache and is not present in the map source.

Requires autofs-4.1.4-misc-mixes.patch to have already been applied.

lib/cache.c renamed from lib/cache.c.update-fix +0 -10
@@ -312,11 +312,6 @@ void cache_clean(const char *root, time_
 			if (!path)
 				return;
 
-			if (is_mounted(_PATH_MOUNTED, path)) {
-				free(path);
-				continue;
-			}
-
 			if (me->age < age) {
 				pred->next = me->next;
 				free(me->key);
@@ -337,11 +332,6 @@ void cache_clean(const char *root, time_
 		if (!path)
 			return;
 
-		if (is_mounted(_PATH_MOUNTED, path)) {
-			free(path);
-			continue;
-		}
-
 		if (me->age < age) {
 			mapent_hash[i] = me->next;
 			rmdir_path(path);
modules/lookup_file.c renamed from modules/lookup_file.c.update-fix +5 -5
@@ -252,7 +252,7 @@ static int read_map(const char *root, ti
 	while(1) {
 		entry = read_one(f, key, mapent);
 		if (entry)
-			cache_add(root, key, mapent, age);
+			cache_update(root, key, mapent, age);
 
 		if (feof(f))
 			break;
@@ -383,7 +383,7 @@ int lookup_mount(const char *root, const
 	char key[KEY_MAX_LEN + 1];
 	int key_len;
 	char mapent[MAPENT_MAX_LEN + 1];
-	struct mapent_cache *me;
+	struct mapent_cache *me, *exists;
 	time_t now = time(NULL);
 	time_t t_last_read;
 	int need_hup = 0;
@@ -407,12 +407,12 @@ int lookup_mount(const char *root, const
 
 	/* only if it has been modified */
 	if (st.st_mtime > ctxt->mtime) {
+		exists = cache_lookup(key);
+
 		ret = lookup_one(root, key, key_len, ctxt);
-		if (!ret)
+		if (!exists && !ret)
 			return 1;
 
-		debug("ret = %d", ret);
-
 		if (t_last_read > ap.exp_runfreq)
 			if (ret & (CHE_UPDATED | CHE_MISSING))
 				need_hup = 1;
modules/lookup_yp.c renamed from modules/lookup_yp.c.update-fix +10 -5
@@ -102,7 +102,7 @@ int yp_all_callback(int status, char *yp
 	strncpy(mapent, val, vallen);
 	*(mapent + vallen) = '\0';
 
-	cache_add(root, key, mapent, age);
+	cache_update(root, key, mapent, age);
 
 	return 0;
 }
@@ -220,7 +220,7 @@ int lookup_mount(const char *root, const
 	int key_len;
 	char *mapent;
 	int mapent_len;
-	struct mapent_cache *me;
+	struct mapent_cache *me, *exists;
 	time_t now = time(NULL);
 	time_t t_last_read;
 	int need_hup = 0;
@@ -237,12 +237,17 @@ int lookup_mount(const char *root, const
 		return 1;
 
 	/* check map and if change is detected re-read map */
+	/*
+	 * check map and if change is detected re-read map
+	 */
+
+	/* First check to see if this entry exists in the cache */
+	exists = cache_lookup(key);
+
 	ret = lookup_one(root, key, key_len, ctxt);
-	if (!ret)
+	if (!exists && !ret)
 		return 1;
 
-	debug("ret = %d", ret);
-
 	if (ret < 0) {
 		warn(MODPREFIX 
 		     "lookup for %s failed: %s", name, yperr_string(-ret));
modules/lookup_ldap.c renamed from modules/lookup_ldap.c.update-fix +6 -10
@@ -463,10 +463,8 @@ static int lookup_one(const char *root, 
 	}
 
 	if (!me) {
-		cache_delete(root, qKey, 0);
-
 		for (i = 0; values[i]; i++) {	
-			rv = cache_add(root, qKey, values[i], age);
+			rv = cache_update(root, qKey, values[i], age);
 			if (!rv)
 				return 0;
 		}
@@ -571,10 +569,8 @@ static int lookup_wild(const char *root,
 	}
 
 	if (!me) {
-		cache_delete(root, "*", 0);
-
 		for (i = 0; values[i]; i++) {	
-			rv = cache_add(root, "*", values[i], age);
+			rv = cache_update(root, "*", values[i], age);
 			if (!rv)
 				return 0;
 		}
@@ -598,7 +594,7 @@ int lookup_mount(const char *root, const
 	int key_len;
 	char mapent[MAPENT_MAX_LEN + 1];
 	char *mapname;
-	struct mapent_cache *me;
+	struct mapent_cache *me *exists;
 	time_t now = time(NULL);
 	time_t t_last_read;
 	int need_hup = 0;
@@ -611,13 +607,13 @@ int lookup_mount(const char *root, const
 	if (key_len > KEY_MAX_LEN)
 		return 1;
 
+	exists = cache_lookup(key);
+
 	ret = lookup_one(root, key, "nisObject", "cn", "nisMapEntry", ctxt);
 	ret2 = lookup_one(root, key,
 			    "automount", "cn", "automountInformation", ctxt);
 	
-   	debug("ret = %d, ret2 = %d", ret, ret2);
-
-	if (!ret && !ret2)
+	if (!exists && !ret && !ret2)
 		return 1;
 
 	me = cache_lookup_first();
#19 lore raw

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

Quoted 3 lines · Jeff Moyer
Quoted 3 lines

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

Quoted 7 lines · Brian Long
Quoted 7 lines
Quoted 1 line
Quoted 1 line

I have an auto.master deployed to thousands of hosts and it looks

like >> this:

Quoted 2 lines
Quoted 2 lines

/misc /etc/auto.misc --timeout=60 /auto /etc/auto.indirect >>

rsize=32768,wsize=32768,tcp /users auto_home >>
rw,hard,intr,rsize=32768,wsize=32768,tcp

Quoted 1 line
Quoted 1 line
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line

As you can see, /users comes from NIS auto.home map. In our case, >>

auto.home contains over 34,000 entries. I've noticed in RHEL 3 U5 and

Quoted 1 line
Quoted 1 line

beyond (autofs-4.1.3-130), trying to stat (ls) /users/no-such-user

takes >> roughly 12-15 seconds. In RHEL 3 U3 (autofs-4.1.3-12), it returns >> immediately. This is a regression in my opinion.

raven> Bummer. I thought that update code was OK.

Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line

After looking at tcpdump data, it appears automounter is downloading

the >> entire auto.home map when it fails to lookup "no-such-user". This >> results in a > 700KB transfer from the NIS server :(

raven> This is exactly the sort of problem I was concerned about when raven> people asked for periodic reload of maps. I didn't do that and raven> clearly I was right.

Quoted 1 line
Quoted 1 line

raven> Having said that it possibly shouldn't be doing this either. I'll raven> have to check the lookup code to see what I'm doing.

Quoted 1 line
Quoted 1 line

raven> Probably I'm signaling a re-load because, since the entry is now raven> missing, the map may need to be updated.

Quoted 1 line
Quoted 1 line

raven> I hasten to point out there was quite a bit of demand for having "up raven> to date maps"! Having a HUP signal to force a re-load wasn't enough.

Quoted 1 line
Quoted 1 line

raven> Given your situation and the fact that we need to know about raven> changes, what do you think the semantics of handling map updates raven> should be?

Quoted 1 line
Quoted 1 line

raven> Seems to me there are only two options when an out of date map entry raven> is encountered. Update, remove or add the given entry and accept raven> there may be more you don't know about or re-read the whole map. Now raven> even the simple addition and removal is not simple as the lookup is raven> done in a subprocess so somehow the process leader needs to be told raven> what to get!

Quoted 4 lines
Quoted 4 lines

Ian, the point here is that the map key isn't out of date. We don't detect that the entry didn't exist in the first place. See my other post to this thread, where I've attached a patch. Comments on that are, of course, encouraged.

raven> Could you review this patch please Jeff.

raven> I will try to find some time to setup and test it on the weekend.

Yup, that looks great. My only nit is that we never check the return codes from cache_add or cache_update. But, that's something to worry about in another patch.

Brian, have you tested the new patch?

Thanks,

Jeff

author
#20 lore raw
Quoted 2 lines

Brian, have you tested the new patch?

My TAM is building autofs test RPMs for me today with this latest patch and I'll start testing them on as many hosts as I can. It appears this patch is too late for U7 (according to my TAM's interaction with RH Engineering mgmt).

/Brian/

Signature · 4 lines
       Brian Long                      |         |           |
       IT Data Center Systems          |       .|||.       .|||.
       Cisco Linux Developer           |   ..:|||||||:...:|||||||:..
       Phone: (919) 392-7363           |   C i s c o   S y s t e m s
maintainer
#21 lore raw
Quoted 7 lines · Brian Long
Quoted 2 lines

Brian, have you tested the new patch?

My TAM is building autofs test RPMs for me today with this latest patch and I'll start testing them on as many hosts as I can. It appears this patch is too late for U7 (according to my TAM's interaction with RH Engineering mgmt).

That might not be such a bad thing as I'm not sure we are entirely finished with this problem yet.

Ian

#22 lore raw
In reply to Ian Kent

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

Quoted 2 lines · Jeff Moyer
Quoted 2 lines

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

raven> Could you review this patch please Jeff.

After reviewing this further, I did find a problem. See below.

Quoted 2 lines
modules/lookup_file.c renamed from modules/lookup_file.c.update-fix +0 -0

[snip]

modules/lookup_file.c int lookup_mount(const char *root, const · line 414

Quoted 10 lines
modules/lookup_file.c renamed from modules/lookup_file.c.update-fix +3 -1
@@ -407,12 +407,12 @@ int lookup_mount(const char *root, const
 
 	/* only if it has been modified */
 	if (st.st_mtime > ctxt->mtime) {
+		exists = cache_lookup(key);
+
 		ret = lookup_one(root, key, key_len, ctxt);
-		if (!ret)
+		if (!exists && !ret)
 			return 1;

This check is wrong. ret will be zero only if there is a failure reading the map or adding an entry to the cache. I think the code should look like so:

	if (st.st_mtime > ctxt->mtime) {
		exists = cache_lookup(key);

		ret = lookup_one(root, key, key_len, ctxt);
		if (!ret)
			return 1;

		if (ret == CHE_MISSING && exists) {          <----------
			if (!cache_delete(root, key, CHE_RMPATH))
				rmdir_path(key);

			/* Maybe update wild card map entry */
			if (ap.type == LKP_INDIRECT)
				lookup_wild(root, ctxt);

			/* Have parent update its map */
			if (t_last_read > ap.exp_runfreq)
				kill(getppid(), SIGHUP);
		} else if (ret == CHE_UPDATED) {
			/* Have parent update its map */
			if (t_last_read > ap.exp_runfreq)
				kill(getppid(), SIGHUP);
		}
	}

As a matter of cleanliness, it would be good to use the CHE_ defines when testing for error. This mistake likely would not have been made if the code had read:

		if (ret == CHE_FAIL)
			return 1;

instead of "if (!ret)".

-Jeff

maintainer
#23 lore raw
Quoted 9 lines · Jeff Moyer

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

Quoted 2 lines · Jeff Moyer
Quoted 2 lines

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

raven> Could you review this patch please Jeff.

After reviewing this further, I did find a problem. See below.

All good points and the test is wrong.
I'll update the patch and post another.

That just leaves the re-read for the other cases which may be causing a long delay. I'm not sure this slows the lookup unless sending the signal is causing starvation of cpu cycles some how. Perhaps the change from cache_add to cache_update will help more than I think.

Ian

#24 lore raw

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

Quoted 3 lines · Jeff Moyer
Quoted 3 lines

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

Quoted 4 lines · Jeff Moyer
Quoted 4 lines
Quoted 1 line
Quoted 1 line

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15

seconds; Ian Kent <[email protected]> adds:

Quoted 1 line
Quoted 1 line

raven> Could you review this patch please Jeff.

Quoted 1 line
Quoted 1 line

After reviewing this further, I did find a problem. See below.

raven> All good points and the test is wrong. I'll update the patch and raven> post another.

raven> That just leaves the re-read for the other cases which may be raven> causing a long delay. I'm not sure this slows the lookup unless raven> sending the signal is causing starvation of cpu cycles some raven> how. Perhaps the change from cache_add to cache_update will help raven> more than I think.

Ian,

The patch I'm attaching is the one against the RHEL autofs package. This is what I have come up with, and it is WRONG. =) It causes a regression in the case where there are multiple keys of the same name in a map; the patched automounter will always use the last key instead of the first. Remember dealing with this back in February? Fun.

I'll keep poking at it, I just wanted to make you aware of the problem.

-Jeff

author
#25 lore raw
Quoted 25 lines · Jeff Moyer

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

Quoted 3 lines · Jeff Moyer
Quoted 3 lines

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

Quoted 4 lines · Jeff Moyer
Quoted 4 lines
Quoted 1 line
Quoted 1 line

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15

seconds; Ian Kent <[email protected]> adds:

Quoted 1 line
Quoted 1 line

raven> Could you review this patch please Jeff.

Quoted 1 line
Quoted 1 line

After reviewing this further, I did find a problem. See below.

raven> All good points and the test is wrong. I'll update the patch and raven> post another.

raven> That just leaves the re-read for the other cases which may be raven> causing a long delay. I'm not sure this slows the lookup unless raven> sending the signal is causing starvation of cpu cycles some raven> how. Perhaps the change from cache_add to cache_update will help raven> more than I think.

Ian,

Quoted 7 lines

The patch I'm attaching is the one against the RHEL autofs package. This is what I have come up with, and it is WRONG. =) It causes a regression in the case where there are multiple keys of the same name in a map; the patched automounter will always use the last key instead of the first. Remember dealing with this back in February? Fun.

I'll keep poking at it, I just wanted to make you aware of the problem.

Does this affect the autofs-4.1.3-166 RPMs I'm testing? :)

/Brian/

Signature · 4 lines
       Brian Long                      |         |           |
       IT Data Center Systems          |       .|||.       .|||.
       Cisco Linux Developer           |   ..:|||||||:...:|||||||:..
       Phone: (919) 392-7363           |   C i s c o   S y s t e m s
#26 lore raw

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Brian Long <[email protected]> adds:

Quoted 3 lines · Jeff Moyer
Quoted 3 lines

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

Quoted 3 lines · Jeff Moyer
Quoted 3 lines
Quoted 1 line
Quoted 1 line

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15

seconds; >> Ian Kent <[email protected]> adds:

Quoted 1 line
Quoted 1 line
Quoted 4 lines · Jeff Moyer
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 >>

seconds; Ian Kent <[email protected]> adds:

Quoted 2 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line

raven> Could you review this patch please Jeff.

Quoted 2 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line

After reviewing this further, I did find a problem. See below.

raven> All good points and the test is wrong. I'll update the patch and raven> post another.

Quoted 1 line
Quoted 1 line

raven> That just leaves the re-read for the other cases which may be raven> causing a long delay. I'm not sure this slows the lookup unless raven> sending the signal is causing starvation of cpu cycles some raven> how. Perhaps the change from cache_add to cache_update will help raven> more than I think.

Quoted 9 lines
Quoted 9 lines

Ian,

The patch I'm attaching is the one against the RHEL autofs package. This is what I have come up with, and it is WRONG. =) It causes a regression in the case where there are multiple keys of the same name in a map; the patched automounter will always use the last key instead of the first. Remember dealing with this back in February? Fun.

I'll keep poking at it, I just wanted to make you aware of the problem.

brilong> Does this affect the autofs-4.1.3-166 RPMs I'm testing? :)

Yes. And here is the patch, which I forgot to attach to my last message. In case this is at all unclear, the attached patch is bad. Don't deploy. ;) To clarify even further, I tested the file maps case.

-Jeff

modules/lookup_file.c renamed from modules/lookup_file.c.orig +6 -4
@@ -248,7 +248,7 @@ static int read_map(const char *root, ti
 	while(1) {
 		entry = read_one(f, key, mapent);
 		if (entry)
-			cache_add(root, key, mapent, age);
+			cache_update(root, key, mapent, age);
 
 		if (feof(f))
 			break;
@@ -380,7 +380,7 @@ int lookup_mount(const char *root, const
 	char key[KEY_MAX_LEN + 1];
 	int key_len;
 	char mapent[MAPENT_MAX_LEN + 1];
-	struct mapent_cache *me;
+	struct mapent_cache *me, *exists;
 	time_t now = time(NULL);
 	time_t t_last_read;
 	int ret;
@@ -403,13 +403,15 @@ int lookup_mount(const char *root, const
 
 	/* only if it has been modified */
 	if (st.st_mtime > ctxt->mtime) {
+		exists = cache_lookup(key);
+
 		ret = lookup_one(root, key, key_len, ctxt);
-		if (!ret)
+		if (ret == CHE_FAIL)
 			return 1;
 
 		debug("ret = %d", ret);
 
-		if (ret == CHE_MISSING) {
+		if (ret == CHE_MISSING && exists) {
 			if (!cache_delete(root, key, CHE_RMPATH))
 				rmdir_path(key);
 
modules/lookup_yp.c renamed from modules/lookup_yp.c.orig +9 -6
@@ -110,7 +110,7 @@ int yp_all_callback(int status, char *yp
 	strncpy(mapent, val, vallen);
 	*(mapent + vallen) = '\0';
 
-	cache_add(root, key, mapent, age);
+	cache_update(root, key, mapent, age);
 
 	return 0;
 }
@@ -224,7 +224,7 @@ int lookup_mount(const char *root, const
 	int key_len;
 	char *mapent;
 	int mapent_len;
-	struct mapent_cache *me;
+	struct mapent_cache *me, *exists;
 	int ret;
 	time_t now = time(NULL);
 	time_t t_last_read;
@@ -240,11 +240,14 @@ int lookup_mount(const char *root, const
 	if (key_len > KEY_MAX_LEN)
 		return 1;
 
-	/* check map and if change is detected re-read map */
-	ret = lookup_one(root, key, key_len, ctxt);
+	/*
+	 *  check map and if change is detected re-read map
+	 */
 
-	debug("ret = %d", ret);
+	/* First check to see if this entry exists in the cache */
+	exists = cache_lookup(key);
 
+	ret = lookup_one(root, key, key_len, ctxt);
 	if (!ret)
 		return 1;
 
@@ -261,7 +264,7 @@ int lookup_mount(const char *root, const
 		/* Have parent update its map */
 		if (t_last_read > ap.exp_runfreq)
 			kill(getppid(), SIGHUP);
-	} else if (ret == CHE_MISSING) {
+	} else if (ret == CHE_MISSING && exists) {
 		if (!cache_delete(root, key, 0)) {
 			rmdir_if_mount_fails = 1;
 		}
modules/lookup_ldap.c renamed from modules/lookup_ldap.c.orig +5 -5
@@ -460,10 +460,8 @@ static int lookup_one(const char *root, 
 	}
 
 	if (!me) {
-		cache_delete(root, qKey, 0);
-
 		for (i = 0; values[i]; i++) {	
-			rv = cache_add(root, qKey, values[i], age);
+			rv = cache_update(root, qKey, values[i], age);
 			if (!rv)
 				return 0;
 		}
@@ -594,7 +592,7 @@ int lookup_mount(const char *root, const
 	int key_len;
 	char mapent[MAPENT_MAX_LEN + 1];
 	char *mapname;
-	struct mapent_cache *me;
+	struct mapent_cache *me, *exists;
 	time_t now = time(NULL);
 	time_t t_last_read;
 
@@ -606,6 +604,8 @@ int lookup_mount(const char *root, const
 	if (key_len > KEY_MAX_LEN)
 		return 1;
 
+	exists = cache_lookup(key);
+
 	ret = lookup_one(root, key, "nisObject", "cn", "nisMapEntry", ctxt);
 	ret2 = lookup_one(root, key,
 			    "automount", "cn", "automountInformation", ctxt);
@@ -620,7 +620,7 @@ int lookup_mount(const char *root, const
 	me = cache_lookup_first();
 	t_last_read = now - me->age;
 
-	if (ret == CHE_MISSING && ret2 == CHE_MISSING && ret3 == CHE_MISSING) {
+	if (ret == ret2 == ret3 == CHE_MISSING && exists) {
 		if (!cache_delete(root, key, CHE_RMPATH))
 			rmdir_path(key);
maintainer
#27 lore raw
In reply to Jeff Moyer
Quoted 25 lines · Jeff Moyer

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

Quoted 3 lines · Jeff Moyer
Quoted 3 lines

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

Quoted 4 lines · Jeff Moyer
Quoted 4 lines
Quoted 1 line
Quoted 1 line

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15

seconds; Ian Kent <[email protected]> adds:

Quoted 1 line
Quoted 1 line

raven> Could you review this patch please Jeff.

Quoted 1 line
Quoted 1 line

After reviewing this further, I did find a problem. See below.

raven> All good points and the test is wrong. I'll update the patch and raven> post another.

raven> That just leaves the re-read for the other cases which may be raven> causing a long delay. I'm not sure this slows the lookup unless raven> sending the signal is causing starvation of cpu cycles some raven> how. Perhaps the change from cache_add to cache_update will help raven> more than I think.

Ian,

Quoted 7 lines

The patch I'm attaching is the one against the RHEL autofs package. This is what I have come up with, and it is WRONG. =) It causes a regression in the case where there are multiple keys of the same name in a map; the patched automounter will always use the last key instead of the first. Remember dealing with this back in February? Fun.

I'll keep poking at it, I just wanted to make you aware of the problem.

I'll check that when I make the patch but I'm fairly sure that I changed the cache code to prevent this at some point. If I can identify that I'll post it and specify it as a pre-patch. I also seem to remember that LDAP lookups were an issue and I still needed to deal with that.

Ian

#28 lore raw

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

Quoted 3 lines · Jeff Moyer
Quoted 3 lines

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

Quoted 3 lines · Jeff Moyer
Quoted 3 lines
Quoted 1 line
Quoted 1 line

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15

seconds; >> Ian Kent <[email protected]> adds:

Quoted 1 line
Quoted 1 line
Quoted 4 lines · Jeff Moyer
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 >>

seconds; Ian Kent <[email protected]> adds:

Quoted 2 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line

raven> Could you review this patch please Jeff.

Quoted 2 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line

After reviewing this further, I did find a problem. See below.

raven> All good points and the test is wrong. I'll update the patch and raven> post another.

Quoted 1 line
Quoted 1 line

raven> That just leaves the re-read for the other cases which may be raven> causing a long delay. I'm not sure this slows the lookup unless raven> sending the signal is causing starvation of cpu cycles some raven> how. Perhaps the change from cache_add to cache_update will help raven> more than I think.

Quoted 9 lines
Quoted 9 lines

Ian,

The patch I'm attaching is the one against the RHEL autofs package. This is what I have come up with, and it is WRONG. =) It causes a regression in the case where there are multiple keys of the same name in a map; the patched automounter will always use the last key instead of the first. Remember dealing with this back in February? Fun.

I'll keep poking at it, I just wanted to make you aware of the problem.

raven> I'll check that when I make the patch but I'm fairly sure that I raven> changed the cache code to prevent this at some point. If I can raven> identify that I'll post it and specify it as a pre-patch. I also raven> seem to remember that LDAP lookups were an issue and I still needed raven> to deal with that.

Well, ldap maps get tricky, since they don't guarantee the order of the results. Aside from that, I'm not sure what you mean. ;) Basically, you really shouldn't have multiple entries with the same key when using ldap.

-Jeff

maintainer
#29 lore raw
Quoted 42 lines · Jeff Moyer

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

Quoted 3 lines · Jeff Moyer
Quoted 3 lines

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

Quoted 3 lines · Jeff Moyer
Quoted 3 lines
Quoted 1 line
Quoted 1 line

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15

seconds; >> Ian Kent <[email protected]> adds:

Quoted 1 line
Quoted 1 line
Quoted 4 lines · Jeff Moyer
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 >>

seconds; Ian Kent <[email protected]> adds:

Quoted 2 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line

raven> Could you review this patch please Jeff.

Quoted 2 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line

After reviewing this further, I did find a problem. See below.

raven> All good points and the test is wrong. I'll update the patch and raven> post another.

Quoted 1 line
Quoted 1 line

raven> That just leaves the re-read for the other cases which may be raven> causing a long delay. I'm not sure this slows the lookup unless raven> sending the signal is causing starvation of cpu cycles some raven> how. Perhaps the change from cache_add to cache_update will help raven> more than I think.

Quoted 9 lines
Quoted 9 lines

Ian,

The patch I'm attaching is the one against the RHEL autofs package. This is what I have come up with, and it is WRONG. =) It causes a regression in the case where there are multiple keys of the same name in a map; the patched automounter will always use the last key instead of the first. Remember dealing with this back in February? Fun.

I'll keep poking at it, I just wanted to make you aware of the problem.

raven> I'll check that when I make the patch but I'm fairly sure that I raven> changed the cache code to prevent this at some point. If I can raven> identify that I'll post it and specify it as a pre-patch. I also raven> seem to remember that LDAP lookups were an issue and I still needed raven> to deal with that.

Quoted 3 lines

Well, ldap maps get tricky, since they don't guarantee the order of the results. Aside from that, I'm not sure what you mean. ;) Basically, you really shouldn't have multiple entries with the same key when using ldap.

Yep. But at the time I was thinking LDAP users might often think of multi-mount entries as multiple nisMapEntry or automountInformation attributes under the same cn or automountKey but it looks like I opened the possibility of duplicate keys when I did it. My typical brain thinking one thing, fingers typing something else and eyes seeing niether!

An example of the type of usage common in LDAP is say a group "www" that has multiple occurances of an attribute to describe group membership:

cn=www,ou=Groups, o=xx.com.au
objectclass=top
objectclass=groupOfUniqueNames
cn=www
description=WWW Users
uniquemember=uid=abcd,ou=People,o=xx.com.au
uniquemember=uid=abce,ou=People,o=xx.com.au
uniquemember=uid=abcf,ou=People,o=xx.com.au
uniquemember=uid=abcg,ou=People,o=xx.com.au
uniquemember=uid=abch,ou=People,o=xx.com.au

Ian

maintainer
#30 lore raw
In reply to Jeff Moyer
Quoted 23 lines · Jeff Moyer

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

Quoted 3 lines · Jeff Moyer
Quoted 3 lines

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15 seconds; Ian Kent <[email protected]> adds:

Quoted 4 lines · Jeff Moyer
Quoted 4 lines
Quoted 1 line
Quoted 1 line

==> Regarding Re: [autofs] stat of /users/no-such-user takes 15

seconds; Ian Kent <[email protected]> adds:

Quoted 1 line
Quoted 1 line

raven> Could you review this patch please Jeff.

Quoted 1 line
Quoted 1 line

After reviewing this further, I did find a problem. See below.

raven> All good points and the test is wrong. I'll update the patch and raven> post another.

raven> That just leaves the re-read for the other cases which may be raven> causing a long delay. I'm not sure this slows the lookup unless raven> sending the signal is causing starvation of cpu cycles some raven> how. Perhaps the change from cache_add to cache_update will help raven> more than I think.

Quoted 8 lines

Ian,

The patch I'm attaching is the one against the RHEL autofs package. This is what I have come up with, and it is WRONG. =) It causes a regression in the case where there are multiple keys of the same name in a map; the patched automounter will always use the last key instead of the first. Remember dealing with this back in February? Fun.

Maybe this one is a little better.
I've removed the LDAP update changes as I think their not needed.

Please check it out Jeff.

=================

This patch addresses three problems.

Change cache_update to only update the first occurance of a duplicate map entry on a map read.

Change cache_add to cache_update in lookup modules. Perform an update rather than an add so that only stale entries are actually removed instead of adding an additional entry then removing the old one. This add/remove causes a directory removal of almost every entry in the map when it should'nt.

Prevent re-read of map and return fail from the lookup module when a key is not present in the cache and is not present in the map source.

lib/cache.c renamed from lib/cache.c.update-fix +4 -10
@@ -233,6 +233,10 @@ int cache_update(const char *root, const
 		}
 		ret = CHE_UPDATED;
 	} else {
+		/* Already seen one of these */
+		if (me->age == age)
+			return CHE_OK;
+
 		if (strcmp(me->mapent, mapent) != 0) {
 			pent = malloc(strlen(mapent) + 1);
 			if (pent == NULL) {
@@ -312,11 +316,6 @@ void cache_clean(const char *root, time_
 			if (!path)
 				return;
 
-			if (is_mounted(_PATH_MOUNTED, path)) {
-				free(path);
-				continue;
-			}
-
 			if (me->age < age) {
 				pred->next = me->next;
 				free(me->key);
@@ -337,11 +336,6 @@ void cache_clean(const char *root, time_
 		if (!path)
 			return;
 
-		if (is_mounted(_PATH_MOUNTED, path)) {
-			free(path);
-			continue;
-		}
-
 		if (me->age < age) {
 			mapent_hash[i] = me->next;
 			rmdir_path(path);
modules/lookup_file.c renamed from modules/lookup_file.c.update-fix +9 -6
@@ -252,7 +252,7 @@ static int read_map(const char *root, ti
 	while(1) {
 		entry = read_one(f, key, mapent);
 		if (entry)
-			cache_add(root, key, mapent, age);
+			cache_update(root, key, mapent, age);
 
 		if (feof(f))
 			break;
@@ -383,7 +383,7 @@ int lookup_mount(const char *root, const
 	char key[KEY_MAX_LEN + 1];
 	int key_len;
 	char mapent[MAPENT_MAX_LEN + 1];
-	struct mapent_cache *me;
+	struct mapent_cache *me, *exists;
 	time_t now = time(NULL);
 	time_t t_last_read;
 	int need_hup = 0;
@@ -407,16 +407,19 @@ int lookup_mount(const char *root, const
 
 	/* only if it has been modified */
 	if (st.st_mtime > ctxt->mtime) {
+		exists = cache_lookup(key);
+
 		ret = lookup_one(root, key, key_len, ctxt);
-		if (!ret)
+		if (ret == CHE_FAIL)
 			return 1;
 
-		debug("ret = %d", ret);
-
+		/* Check if we need to update the map */
 		if (t_last_read > ap.exp_runfreq)
-			if (ret & (CHE_UPDATED | CHE_MISSING))
+			if ((ret & CHE_UPDATED) ||
+			    (exists && (ret & CHE_MISSING)))
 				need_hup = 1;
 
+		/* Update our map for the lookup */
 		if (ret == CHE_MISSING) {
 			int wild = CHE_MISSING;
 
modules/lookup_yp.c renamed from modules/lookup_yp.c.update-fix +12 -7
@@ -102,7 +102,7 @@ int yp_all_callback(int status, char *yp
 	strncpy(mapent, val, vallen);
 	*(mapent + vallen) = '\0';
 
-	cache_add(root, key, mapent, age);
+	cache_update(root, key, mapent, age);
 
 	return 0;
 }
@@ -220,7 +220,7 @@ int lookup_mount(const char *root, const
 	int key_len;
 	char *mapent;
 	int mapent_len;
-	struct mapent_cache *me;
+	struct mapent_cache *me, *exists;
 	time_t now = time(NULL);
 	time_t t_last_read;
 	int need_hup = 0;
@@ -236,13 +236,17 @@ int lookup_mount(const char *root, const
 	if (key_len > KEY_MAX_LEN)
 		return 1;
 
-	/* check map and if change is detected re-read map */
+	/*
+	 * check map and if change is detected re-read map
+	 */
+
+	/* First check to see if this entry exists in the cache */
+	exists = cache_lookup(key);
+
 	ret = lookup_one(root, key, key_len, ctxt);
-	if (!ret)
+	if (ret == CHE_FAIL)
 		return 1;
 
-	debug("ret = %d", ret);
-
 	if (ret < 0) {
 		warn(MODPREFIX 
 		     "lookup for %s failed: %s", name, yperr_string(-ret));
@@ -253,7 +257,8 @@ int lookup_mount(const char *root, const
 	t_last_read = me ? now - me->age : ap.exp_runfreq + 1;
 
 	if (t_last_read > ap.exp_runfreq)
-		if (ret & (CHE_UPDATED | CHE_MISSING))
+		if ((ret & CHE_UPDATED) ||
+		    (exists && (ret & CHE_MISSING)))
 			need_hup = 1;
 
 	if (ret == CHE_MISSING) {
modules/lookup_ldap.c renamed from modules/lookup_ldap.c.update-fix +11 -7
@@ -598,7 +598,7 @@ int lookup_mount(const char *root, const
 	int key_len;
 	char mapent[MAPENT_MAX_LEN + 1];
 	char *mapname;
-	struct mapent_cache *me;
+	struct mapent_cache *me, *exists;
 	time_t now = time(NULL);
 	time_t t_last_read;
 	int need_hup = 0;
@@ -611,22 +611,26 @@ int lookup_mount(const char *root, const
 	if (key_len > KEY_MAX_LEN)
 		return 1;
 
+	exists = cache_lookup(key);
+
 	ret = lookup_one(root, key, "nisObject", "cn", "nisMapEntry", ctxt);
 	ret2 = lookup_one(root, key,
 			    "automount", "cn", "automountInformation", ctxt);
 	
-   	debug("ret = %d, ret2 = %d", ret, ret2);
-
-	if (!ret && !ret2)
+	if (ret == CHE_FAIL && ret2 == CHE_FAIL)
 		return 1;
 
 	me = cache_lookup_first();
 	t_last_read = me ? now - me->age : ap.exp_runfreq + 1;
 
-	if (t_last_read > ap.exp_runfreq) 
-		if ((ret & (CHE_MISSING | CHE_UPDATED)) && 
-		    (ret2 & (CHE_MISSING | CHE_UPDATED)))
+	if (t_last_read > ap.exp_runfreq) {
+		if ((ret & CHE_UPDATED) ||
+		     (exists && (ret & CHE_MISSING)))
 			need_hup = 1;
+		else if ((ret2 & CHE_UPDATED) ||
+		     (exists && (ret2 & CHE_MISSING)))
+			need_hup = 1;
+	}
 
 	if (ret == CHE_MISSING && ret2 == CHE_MISSING) {
 		int wild = CHE_MISSING;