documentation issues
Hi,
a few things.
A user attempting to use autofs for the first time will assume that autofs will mount mount poins that already exist. Nowhere is it stated that mount points will only get created when accessed (through a "direct path").
Such a user (such as me) trying to use this thing (even with a tutorial on Ubuntu site) will be flabbergasted that nothing happens; he/she expects mount entries to appear, but nothing happens.
This renders the "nobrowse" default dysfunctional for a desktop user, who expects to see something before going there. It only works for automated systems, really.
Further, you will only become aware of "browse" after having been able to use the system, because otherwise you won't go reading entire manuals just to see what is wrong.
So in my case someone explained to me that the mount points would only get created when accessed, at which point I finally was able to have some result.
So I say two things:
- browse only works for static maps, and what person with a static map is going to run into performance issues when mount points are created in advance? Can we please remove this from the default config? (turning it off).
- browse being turned off by default combined with a complete lack of mention that you need to access these "nonexistent" directory paths / components before you will be able to see them, turns autofs into a complete mystery for someone who is not privy to the details yet.
Further I would say that on a desktop OS, the most likely use case is going to be this:
- have several shares in your home network
- want to access them through a GUI
- prefer automounting since mounting at boot can be problematic
- need to see them before you can access them in a GUI
- only alternative is "bookmarks" or "places" that work regardless of the things not being visible/listed in the directory.
Then I will also say,
that there is no documentation on the format of the map files other than auto.master. The documentation for map files is this:
cat /etc/auto.misc.
That is the documentation.
After you've figured out that e.g. auto.smb wants the servername as the key ($1) the documentation becomes this:
/etc/auto.smb <servername>
Quoted 1 line
Quoted 1 line
watch results pour out.
But auto.smb doesn't mention that it wants the servername as the key.
Of course, it does mention that:
# You'll be able to access Windows and Samba shares in your network
# under /cifs/host.domain/share
But at this point you still don't know that the "host.domain" part will be invisible until you access it.
I'm sorry if I sound abrasive but I was doing freaking straces and even kernel traces before realizing how this was supposed to work. There is just no documentation on it. The most important thing is missing. Is not mentioned.
I wasted so much time trying to figure out how this was supposed to work... in the game of Zelda 1 I believe in some dungeon there is a wall somewhere that you can walk through. You need to go through that wall to get ahead in the game, but there is no visual indication that you can go through. It is a secret.
Well. So is this :P.
How is anyone supposed to know that automounting a "mount point" does not require a mount point to exist in advance?????
I had even (previously, for something else) been reading kernel source code in namei.c or whatever and came across autofs in e.g. lookup_slow or something. Even then it never dawned to me that the typical use case would be for some mount point to not exist yet.
So you start this thing and you cannot write to the mount point any longer so you know it is getting managed.
But nothing ever shows. I even .... well let's not say that here :P.
;-) I had done "touch a" and the file was not created; however I had put some command in auto.smb that would output $1 to some file. So the file said "a" but it never occured to me that this was the result of my "touch a" command ;-). Stupid!.
So I would plead for just two things for other people:
a) get rid of the browse_mode = no, in the configuration file, for static mounts
b) say something about mounts not being created automatically in the man page for auto.master, if nothing else. If you also do (a), this only applies now to programmatic mounts.
=============================================================
You could also do nothing but change the default auto.master:
-------------------------------------------------------------
# This is the sample auto.master file installed on your system. # This map contains entries for mount points that are to be managed
# by autofs. # # Each of these mount points will be mounted with the required map # file as the parameter to the autofs filesystem module that will # manage this mount point for you in one of its threads.
# Each entry mentions a mount point that must already exist on your # system, as well as a map file to be loaded onto that mount point. # Its format is as follows:
# <mount-point> [map-type[,format]:]<map> [options]
# where <mount-point> indicates a directory on your filesystem and # <map> indicates the map file to be used for it.
# For further details see the man page for auto.master(5).
# Example:
#
# /misc /etc/auto.misc browse # # This will create entries from /etc/auto.misc under /misc. The browse # option guarantees that its entries will be created straight away as # sub-directories to /misc.
# Without the browse option, they would only be created after being # accessed, e.g. by direct reference. The browse option is only
# applicable to static map files. # # # /net -hosts # # This mounts all accessible NFS shares from hosts in your network # under /net/<hostname>/ when you first access that path.
# NOTE: mounts done from a hosts map will be mounted with the
# "nosuid" and "nodev" options unless the "suid" and "dev"
# options are explicitly given.
#
#
# Include *.autofs from /etc/auto.master.d/ and the included files
# must conform to the format of this file.
+dir:/etc/auto.master.d
# Include another central master map if it can be found using
# nsswitch sources. Note that duplicate entries will be discarded # as keys must be unique and the keys from this file will take # precedence. That means you can only specify each mount point once.
+auto.master
# Note that map files can be scripts that take the accessed path as a
# parameter. E.g. the following would mount all Samba shares from a # certain host under /cifs/<hostname>/* when /cifs/<hostname> was # accessed:
# /cifs program:/etc/auto.smb # # See /etc/auto.smb for more documentation.
-------------------------------------------------------------
I mean you could rewrite more documentation but this is the simplest I could do right here right now.
I think that would clarify a lot for everyone but of course perhaps you would want to rewrite more so the central documentation is also up to date, but please see this as a more modest attempt for making it more accessible to a new user.
Regards,
Bart Schouten.
--
To unsubscribe from this list: send the line "unsubscribe autofs" in
Ian Kent
maintainerQuoted 8 lines · Xen
Hi,
a few things.
A user attempting to use autofs for the first time will assume that autofs will mount mount poins that already exist. Nowhere is it stated that mount points will only get created when accessed (through a "direct path").
Umm, "direct path" is an unfortunate choice here as you are talking about autofs indirect mount maps.
Direct mount maps create a mount trigger for each map entry and so show up as mount points in the file system.
There are reasons for it being this way.
Quoted 8 lines
Such a user (such as me) trying to use this thing (even with a tutorial on Ubuntu site) will be flabbergasted that nothing happens; he/she expects mount entries to appear, but nothing happens.
This renders the "nobrowse" default dysfunctional for a desktop user, who expects to see something before going there. It only works for automated systems, really.
Your assuming that autofs primary usage is targeted at the desktop. It's often not the best use case for that.
There are other more suitable automount alternatives for GUI desktops.
Quoted 10 lines
Further, you will only become aware of "browse" after having been able to use the system, because otherwise you won't go reading entire manuals just to see what is wrong.
So in my case someone explained to me that the mount points would only get created when accessed, at which point I finally was able to have some result.
Quoted 6 lines
So I say two things:
- browse only works for static maps, and what person with a static map is going to run into performance issues when mount points are created in advance? Can we please remove this from the default config? (turning it off).
Well, quite a lot of current users actually.
Consider what can happen with a few hundred or a few thousand entries in an indirect map.
Even though the browse option was added it has a specific set of problems.
First the kernel does not know if you are just listing a directory or actually need to mount something.
So at the moment, certain types of system call (the module can't know the exact system call either) don't cause a call back to perform a mount but often user space will later make a system call that will cause a mount to occur anyway.
That leads to sometimes seeing directories that are those of the autofs file system not those of the mounted file system, so they have completely different attributes to those after an automount has been done, a problem in itself.
And, it can often lead to a mount storm, mounting everything in the directory.
OK for a few dozen mounts in a map but really bad for anything with more than a few hundred entries.
Another reason for the default is that, because of this, historically there was no "browse" option and this maintains what is (should be) expected.
Quoted 5 lines
- browse being turned off by default combined with a complete lack of mention that you need to access these "nonexistent" directory paths / components before you will be able to see them, turns autofs into a complete mystery for someone who is not privy to the details yet.
It's been a long time for me so I probably have forgotten what it's like for someone starting out.
And so, how would I know where in the man pages to put that so it gets noticed by new users, perhaps from your perspective, you could point out a few key places where it should be added, or better still send a patch.
Quoted 11 lines
Further I would say that on a desktop OS, the most likely use case is going to be this:
- have several shares in your home network
- want to access them through a GUI
- prefer automounting since mounting at boot can be problematic
- need to see them before you can access them in a GUI
- only alternative is "bookmarks" or "places" that work regardless of the things not being visible/listed in the directory.
Either use a different automounting mechanism or use a direct mount map.
That would satisfy all these needs, all be it with the map entries always being mounted on access.
The limitation of these is you can't have a direct map path that is within another automounted path, if you need to do that you need to use a more complicated setup.
Quoted 5 lines
Then I will also say,
that there is no documentation on the format of the map files other than auto.master. The documentation for map files is this:
That's not quite right.
Man page autofs(5) has a bit of information on both the Sun and amd format maps.
Quoted 14 lines
cat
/etc/auto.misc.That is the documentation.
After you've figured out that e.g. auto.smb wants the servername as the key ($1) the documentation becomes this:
/etc/auto.smb<servername>Quoted 1 line
Quoted 1 line
watch results pour out.
But auto.smb doesn't mention that it wants the servername as the key.
Quoted 7 lines
Of course, it does mention that:
# You'll be able to access Windows and Samba shares in your network
# under /cifs/host.domain/shareBut at this point you still don't know that the "host.domain" part will be invisible until you access it.
By definition a program map can't know what keys may be passed to it. And, yes, even autofs(5) doesn't say explicitly that a program map doesn't know what keys may be passed to it, perhaps that should be ammended.
Quoted 32 lines
I'm sorry if I sound abrasive but I was doing freaking straces and even kernel traces before realizing how this was supposed to work. There is just no documentation on it. The most important thing is missing. Is not mentioned.
I wasted so much time trying to figure out how this was supposed to work... in the game of Zelda 1 I believe in some dungeon there is a wall somewhere that you can walk through. You need to go through that wall to get ahead in the game, but there is no visual indication that you can go through. It is a secret.
Well. So is this :P.
How is anyone supposed to know that automounting a "mount point" does not require a mount point to exist in advance?????
I had even (previously, for something else) been reading kernel source code in
namei.cor whatever and came across autofs in e.g.lookup_slowor something. Even then it never dawned to me that the typical use case would be for some mount point to not exist yet.So you start this thing and you cannot write to the mount point any longer so you know it is getting managed.
But nothing ever shows. I even .... well let's not say that here :P.
;-) I had done "touch a" and the file was not created; however I had put some command in auto.smb that would output $1 to some file. So the file said "a" but it never occured to me that this was the result of my "touch a" command ;-). Stupid!.
Quoted 4 lines
So I would plead for just two things for other people:
a) get rid of the
browse_mode= no, in the configuration file, for static mounts
No, that has it's own set of problems which people need to be aware of before choosing to use the "browse" option.
Quoted 3 lines
b) say something about mounts not being created automatically in the man page for auto.master, if nothing else. If you also do (a), this only applies now to programmatic mounts.
auto.master(5) isn't the place for that, autofs(5) is better and, as I say, make some suggestions of provide a patch.
Quoted 67 lines
=============================================================
You could also do nothing but change the default auto.master:
-------------------------------------------------------------# This is the sample auto.master file installed on your system. # This map contains entries for mount points that are to be managed
# by autofs. # # Each of these mount points will be mounted with the required map # file as the parameter to the autofs filesystem module that will # manage this mount point for you in one of its threads.# Each entry mentions a mount point that must already exist on your # system, as well as a map file to be loaded onto that mount point. # Its format is as follows:
# <mount-point> [map-type[,format]:]<map> [options]
# where <mount-point> indicates a directory on your filesystem and # <map> indicates the map file to be used for it.# For further details see the man page for
auto.master(5).# Example:
## /misc /etc/auto.misc browse # # This will create entries from /etc/auto.misc under /misc. The browse # option guarantees that its entries will be created straight away as # sub-directories to /misc.# Without the browse option, they would only be created after being # accessed, e.g. by direct reference. The browse option is only
# applicable to static map files. # # # /net -hosts # # This mounts all accessible NFS shares from hosts in your network # under /net/<hostname>/ when you first access that path.# NOTE: mounts done from a hosts map will be mounted with the
# "nosuid" and "nodev" options unless the "suid" and "dev"
# options are explicitly given.
#
#
# Include *.autofs from/etc/auto.master.d/and the included files# must conform to the format of this file.+dir:/etc/auto.master.d
# Include another central master map if it can be found using
# nsswitch sources. Note that duplicate entries will be discarded # as keys must be unique and the keys from this file will take # precedence. That means you can only specify each mount point once.+auto.master
# Note that map files can be scripts that take the accessed path as a
# parameter. E.g. the following would mount all Samba shares from a # certain host under /cifs/<hostname>/* when /cifs/<hostname> was # accessed:
Quoted 5 lines
# /cifs program:/etc/auto.smb # # See /etc/auto.smb for more documentation.-------------------------------------------------------------
I'll have a look at doing this but beware, often distributions use there own default master map which I have no control over.
Quoted 8 lines
I mean you could rewrite more documentation but this is the simplest I could do right here right now.
I think that would clarify a lot for everyone but of course perhaps you would want to rewrite more so the central documentation is also up to date, but please see this as a more modest attempt for making it more accessible to a new user.
Yes, adding the fact that mount point directories aren't pre created for indirect maps somewhere in auto.master(5) and in a couple of places in autofs(5) is a good idea. You are right it's something that can catch the unaware.
Ian
--
To unsubscribe from this list: send the line "unsubscribe autofs" in
Ian Kent schreef op 06-07-2016 2:57:
Quoted 10 lines · Xen
Quoted 8 lines
Hi,
a few things.
A user attempting to use autofs for the first time will assume that autofs will mount mount poins that already exist. Nowhere is it stated that mount points will only get created when accessed (through a "direct path").
Quoted 7 lines
Umm, "direct path" is an unfortunate choice here as you are talking about autofs indirect mount maps.
Direct mount maps create a mount trigger for each map entry and so show up as mount points in the file system.
There are reasons for it being this way.
So you are berating me for choice of words ;-).
The only difference between direct and indirect seems to be that with direct the share is mounted "directly" on top of the specified mount point (actually, none is specified, it is mounted on the key returned from the map) and as a result it is not possible to have lazy initialization (unless you merged e.g. the root filesystem with an overlay filesystem) and so you are required to create this mountpoint, whereas with "indirect" you control the mount point itself and can choose whether to create the subdirectories or not (if you can acquire a list). "Directly" seems to mean "directly on the filesystem" whereas "indirectly" seems to mean: inside our own little enclave.
And always as a subdirectory of that. Not even technically necessary, you could even have a "indirect map" that triggered mounting I guess, don't know. I guess not.
Quoted 6 lines
Quoted 6 lines
Such a user (such as me) trying to use this thing (even with a tutorial on Ubuntu site) will be flabbergasted that nothing happens; he/she expects mount entries to appear, but nothing happens.
Quoted 8 lines
Quoted 3 lines
This renders the "nobrowse" default dysfunctional for a desktop user, who expects to see something before going there. It only works for automated systems, really.
Your assuming that autofs primary usage is targeted at the desktop. It's often not the best use case for that.
There are other more suitable automount alternatives for GUI desktops.
Linus Torvalds says he developed Linux as a desktop OS. And how chagrined he is that it didn't succeed in just that. Well, now we know the reason for that ;-).
I know there are probably alternatives; but they are mostly useless for someone who wants to do it on his own. Desktop environments often do not even have mount operations on directories (even) when they are described in fstab. There is no other way that I know of to automatically mount e.g. samba shares, unless you navigate directly to it (and then they end up in some weird location in the filesystem, if at all). There is usually also no "user based" fstab; where you can describe mounts specifically for your user, that are then mounted through e.g. fuse, or something else that can do suid. Perhaps you mean udisks or udisksd or udisksctl but I'm not even sure that supports network-filesystems, they are certainly not shown in udisksctl status.
Dolphin can not actually mount a share to some fixed location, the Nemo I am using doesn't even succeed in browsing the network. For me there are only two solutions that I see, either fstab or autofs. Once you know how to do it, running /etc/auto.smb <host> quickly returns a list of shares (if you've set the credentials file) that you can then edit to become a static map with the browse option.
I do not see what is better suited currently to auto-mounting samba-shares. Anything that would work well, would work from both the shell and the GUI, in that the systems have to be linked, and ideally what the GUI does is stored in state files that work equally well from the shell. Any persistent mount would have to be stored somewhere on disk. Then the link would need to be automounted, OR mounted through e.g. libpam-mount. There is nothing wrong with mounting upon logon. And that allows for specific user-dependent stuff. (An automount depending on what user first accesses it, is problematic). (As for Samba). So perhaps logon-mounts are better suited (just restore the mount from the previous session, in that sense) because the $UID problem is hard to solve for autofs, you must either duplicate all entries, and hardcore the UIDs you need, for every user, or I don't know what, prevent other users (including root) from accessing the share you need. You'd basically need map files in auto.master.d for every user, including a map file for each person to specify the shares, if you want browsable, or a modified version of auto.smb that mounts individual shares when accessed. (In the latter case you'd have no browse map).
But this script doesn't know what it is being called on, so now you need a variable I guess for each of those .d/ files, so that their entries would be something like:
/media/<user>/nas /etc/auto-short.smb -Duser=<user> which would then be used for the samba $UID.
If there is a better way to do this, I don't know. I see no alternatives other than libpam-mount and of course some GUI could do it, but then you'd really just have a form of duplicated functionality between autofs and it.
Quoted 9 lines
Quoted 4 lines
- browse only works for static maps, and what person with a static map is going to run into performance issues when mount points are created in advance? Can we please remove this from the default config? (turning it off).
Well, quite a lot of current users actually.
Consider what can happen with a few hundred or a few thousand entries in an indirect map.
Alright, but what indirect maps can cause browse tables to be shown? Only static maps, right? A programmatic map can only return a single entry. I have not looked into the other ones, but I suppose LDAP is more like a programmatic map than anything else.
So then consider that we only use a static map here. An actual file with thousands of entries? Of course it can happen. But what would be the point of that still (as opposed to LDAP or programmatic).
Quoted 4 lines
Even though the browse option was added it has a specific set of problems.
First the kernel does not know if you are just listing a directory or actually need to mount something.
Do you mean the directory containing the (indirect) entries (e.g. the managed directory) or one of those entries themselves?
Quoted 7 lines
So at the moment, certain types of system call (the module can't know the exact system call either) don't cause a call back to perform a mount but often user space will later make a system call that will cause a mount to occur anyway.
That leads to sometimes seeing directories that are those of the autofs file system not those of the mounted file system, so they have completely different attributes to those after an automount has been done, a problem in itself.
Oh... such as ownership etc.
The typical "I mount this, and now suddenly it is not root.root but xen.staff".
Quoted 1 line
And, it can often lead to a mount storm, mounting everything in the directory.
But how can that happen? If I do; "for f in *; ls $f; done" sure.
Quoted 3 lines
OK for a few dozen mounts in a map but really bad for anything with more than a few hundred entries.
After thinking a little I came up with what could perhaps be some default use case. You have terminals mounting home directories. On the "terminal machine" you may have just a single user. Through LDAP you find the home directory. There is obviously no sense for all users in the LDAP system to now have precreated mount points. The only thing that could make sense is to have an "invisible" user directory that now gets mounted.
I can see no reason to have hundreds of mount points in a directory unless they cannot conform to a specific pattern. If they follow a pattern, it is likely that only a few of them need to be mounted at any time. In that case being able to browse it seems nonsensical. An aggregate of hundreds of different mounts at the same time seems very weird. Weird to have a diverse set of unique names to begin with. As if all of them were manually added in some way. If they aren't manually added, but automatically, then why have a static list? Would you not be better off putting that in a database? And if you'd have so many, we'd assume they could be different users, or different websites, etc. We also assume not all of them are going to be needed, just a subset. But so now you have a system on which this list is statically created but dynamically updated. This list sits on the disk of the local system. But you're not querying a database (could be just as easy) that could also store this list and return the entries one by one when needed. The static file is the database.
Seems an improper use case but yeah. I don't know how you can have a static list that is huge while not choosing to use a database to manage it. Because a static map; now you need routines to update it, delete entries from it.
Maybe I'm missing something here.
Quoted 2 lines
Another reason for the default is that, because of this, historically there was no "browse" option and this maintains what is (should be) expected.
Yeah I thought so, because of the comments near the timeout value. However that is a reason to never change anything ever. Someone mistakenly choosing "browse" will very quickly find out.
If this person really has hundreds of items in his list, he will notice within a second that they are all visible. If this person really has hundreds to thousands of items, he or she may also be expected to know the ins and outs of the system, and to be aware of the browse option, or be able to find out. This is not going to be a novice user.
On the other hand, anyone new to the system will have modest goals. However those modest goals are frustrated and refuted because the system is geared towards that professional that already knows everything, and for whom adding nobrowse, or changing the config file, would probably be a nobrainer. The new user however may be flabbergasted that it either does not work, or not the way she feels it should work (making a list, and then not seeing it in action).
Quoted 11 lines
Quoted 4 lines
- browse being turned off by default combined with a complete lack of mention that you need to access these "nonexistent" directory paths / components before you will be able to see them, turns autofs into a complete mystery for someone who is not privy to the details yet.
It's been a long time for me so I probably have forgotten what it's like for someone starting out.
And so, how would I know where in the man pages to put that so it gets noticed by new users, perhaps from your perspective, you could point out a few key places where it should be added, or better still send a patch.
This happens a lot, it is no ones fault. I guess people who knows the ins and outs of something usually jump smack in the middle, and forget to mention what the system is really like from the outside. A fish doesn't notice the water, we don't notice the air.
We are too much immersed in it.
I think there are 3-4 things I would explain.
First the auto.master file on my system mentions man autofs as a place for seeing the format of the (sub)map files. However there is nothing in there. So that part is missing, but in a sense the contents of auto.misc is already somewhat sufficient for that, but still annoying that there is no man page.
The second part is that it is often hard to understand what is meant by "key". In the master file, key means the mount point that supposedly already exists in the filesystem. In the sub-maps, key means the subdirectory but also the component of the path getting accessed by the user. Both types of files are called map files, and both kind of keys are called keys. This is aggravated by the fact that there is no man page for the sub-maps, that I know of . Auto.smb then speaks of a key, but it doesn't say what it is. Of course after a while you realize it is to be the host, but then you don't know how it knows about this host. Since at that point you don't realize that this is input from the filesystem (e.g. a stat operation) on "hidden directories" you are really clueless as to where this information is coming from ;-).
You don't imagine that every freaking access of a directory or path component in the managed directory is going to result in network access.
You could do a "for f in seq 1 10000000; do vdir $f; done" and have fun.
(Don't know if that would still be the case for "browse" maps).
So this hidden directory, secret door thing is really the most confusing of all.
It's somewhat similar to something I might like to do, but still you don't really expect it ;-).
So the 3-4 things are:
create a man page or section for the format of the sub-maps
indicate more clearly that programmatic maps in principle return a single entry
indicate that programmatic maps currently require access of a path component and that this path component (user action) is the key to the script. indicate that this is true for "no browse" on "static maps" as well, and that directories are only created when they are accessed. And that no-browse is the configured default.
Currently the documentation for the master map and for (indirect) sub-maps is really in auto.master and auto.misc.
Reading config files is usually a lot more pleasant than man pages ;-) so that's not such a huge problem, but you can't learn anything from a very limited set of examples either ;-).
For instance: direct map won't work if not prefixed with /, but will also not give error if not. Indirect map (I think) won't work if prefixed with /, but will also not give error if not.
I could say more but my thumb is hurting (infected) so I gotta quit this.
I don't mind writing documentation. But I'm not familiar yet with man pages and their technical structure and all.
I'll have to see. Thank you for the interest in any case :).
Quoted 12 lines
Quoted 11 lines
Further I would say that on a desktop OS, the most likely use case is going to be this:
- have several shares in your home network
- want to access them through a GUI
- prefer automounting since mounting at boot can be problematic
- need to see them before you can access them in a GUI
- only alternative is "bookmarks" or "places" that work regardless of the things not being visible/listed in the directory.
Quoted 8 lines
Either use a different automounting mechanism or use a direct mount map.
That would satisfy all these needs, all be it with the map entries always being mounted on access.
The limitation of these is you can't have a direct map path that is within another automounted path, if you need to do that you need to use a more complicated setup.
I think that limitation indicates why using relative paths is so much more pleasant. A map file using relative files can be "moved around". A map file using absolute paths cannot.
Could you not make a direct map out of the key of the master file, and the keys of the sub-maps?
If there is really a need for huge static map files (I haven't studied LDAP) in principle all you need is a syntax to differentiate between the two, right.
/- ( /this/is/my/abs/path )
or
/this/is/my/abs ( -/path )
could be identical
normally ( /bla/bla ) indicates an absolute path and can only be used in a direct map, from what I've seen. You *could* cause it to turn this "indirect map" into a direct one.
ie. ( bla/bla ) == indirect
( /bla/bla ) == direct.
A direct map is always a static file right? But then it doesn't matter what part of the path is specified in what file.
Or
( bla/bla ) == indirect
( -/bla/bla ) == directSo you get:
/start/of/the (-/path )
or
/start/of/the/- ( path )
Then any path in the master file ending with /- would be a direct path.
And would cause the (static) map file to be treated as direct.
Quoted 10 lines
Quoted 6 lines
Then I will also say,
that there is no documentation on the format of the map files other than auto.master. The documentation for map files is this:
That's not quite right.
Man page autofs(5) has a bit of information on both the Sun and amd format maps.
Oh I am sorry. I think many users are not versed in the numbers for the man pages. I for one do not notice the difference and I almost never use one of those numbers. Sorry, I seriously missed that.
I only noticed just now because autofs(8) references autofs(5), and I'm like ....huh?
Okay, allright. Missed that.
Seriously bad system if you ask me, but okay.
So I guess you can have direct maps with LDAP etc. too.
I guess the wildcard also solves the many-user problem.
Well, but only for the sub-map. You couldn't get autofs to manage /media, so it needs to be /media/user/<aggregate> in which user can't suddenly be a wildcard unless you use a direct map. You could expand that to include the master file though:
/media/*/nas /etc/nas-shares.map
And then in /etc/nas-shares.map:
share1 -fstype=xxx,uid=& ://nas/share1 share2 -fstype=xxx,uid=& ://nas/share2
I can't see whether LDAP supports browse, I assume it must, since it supports direct.
Quoted 3 lines
By definition a program map can't know what keys may be passed to it. And, yes, even autofs(5) doesn't say explicitly that a program map doesn't know what keys may be passed to it, perhaps that should be ammended.
In principle even a program map could read all of the available keys from whatever database it consults, and return all of them; or return something only (as it does now, perhaps) when a given key matches (is found).
If you wanted that you would need the program script to e.g. support a wildcard as parameter. Meaning, all scripts must then check for "*" to be passed to it, in order to return the full list that they can acquire (if even possible) or at least return an error of some kind to say that they don't support it.
You'd need a parameter for it, and any script would need to be aware of that parameter.
That's either gonna be a wildcard or something given on $2, in that case every existing script would already be compatible with it, since it simply disregards $2.
So, for instance, if $2 is "browse" the script may choose to return a browse map, or list of shares, with whatever options are available (it could even return the UID/USER of the share given options, so that the created directory would have the same UID as the to be mounted share!) -- meaning it might simply return ALL entries it could possibly return in the default format, which is the meaning of the wildcard, and the program (autofs daemon or module(?)) would be able to generate the appropriate directory entries based on that. So in that case you're probably going to be rid of the problem of the non-matching state of pre- and post-mount of directories.
Quoted 5 lines
Quoted 2 lines
a) get rid of the
browse_mode= no, in the configuration file, for static mountsNo, that has it's own set of problems which people need to be aware of before choosing to use the "browse" option.
Not everyone.
I have like 5 mounts in this share. 5 shares in this mount. I have not noticed issues to begin with. Maybe there are issues like checking permissions on the directory prior to mounting it, and then being wrong about that once it is mounted. But still, that doesn't seem to happen. Actually, something funny is going on. 2 of the 8 shares that are 'automatically' loaded are not accessible. "No such file or directory".
No, that's nothing but a permission error. Still I am allowed to see them but not use them. For a static file no issue; just delete them.
I don't really see what I need to be aware of prior to using browse?
If it is the most sensible "novice" use case, why not allow it by default? It things go mayhem people can turn it off, or you could even turn it off when the number of mounts in a map exceeds 20 or 30 or 50. It's an arbitrary number, but what if I am wrong about LDAP.
What about you mean it is turned off by default for LDAP, but you do not consider static files with that?
What about it does function for ldap. What about we just need a separate default for LDAP and its like, than we do for static?
Are you seriously considering static files, or do you mean LDAP, yp, hisplus, and hesoid?
For anyone using an actual database, I concur.
For static, I just disagree. That is the start case for anyone.
Quoted 7 lines
Quoted 4 lines
b) say something about mounts not being created automatically in the man page for auto.master, if nothing else. If you also do (a), this only applies now to programmatic mounts.
auto.master(5)isn't the place for that, autofs(5) is better and, as I say, make some suggestions of provide a patch.
autofs(8) is loaded before autofs(5). It's a good document, autofs(5), but most people may not even notice it. Personally I would probably choose to call them:
autofs.master(5) and autofs.map(5). But then, I also consider auto.master a very nondescript name.
In this way no one can ever get confused, so you might just rename autofs(5) to autofs.map(5).
Or even auto.map(5) if that is necessary, but I wouldn't do that.
Quoted 2 lines
I'll have a look at doing this but beware, often distributions use there own default master map which I have no control over.
I have no clue what mine does, this is Debian, and then Ubuntu, and then Kubuntu ;-) but it is just the Debian package still.
You don't have to follow my example or suggestion, maybe there is a better improvement than this, it was just an example, I guess. Or I could guess.
That was just a thing I was able to change without editing man files ;-).
I will look into providing a patch, but I hope you can rename autofs(5) to autofs.map(5) before doing anything like that. I don't feel like improving a system I know myself haven't had access to, in that sense. I don't mean you have to do that straight away, but if that is a possible direction to go, or change to make, then I could supply a patch or anything based on the system that would result from that, seeing that autofs(5) is already pretty good in that sense.
Quoted 11 lines
Quoted 7 lines
I mean you could rewrite more documentation but this is the simplest I could do right here right now.
I think that would clarify a lot for everyone but of course perhaps you would want to rewrite more so the central documentation is also up to date, but please see this as a more modest attempt for making it more accessible to a new user.
Yes, adding the fact that mount point directories aren't pre created for indirect maps somewhere in
auto.master(5)and in a couple of places in autofs(5) is a good idea. You are right it's something that can catch the unaware.
That would also make the difference between direct and indirect more clear. (I still don't know exactly what is the entire difference).
That's also something I could change: how a direct map is explained in auto.master(5). But then, I didn't have access to autofs(5) to explain it to me better ;-).
Anyway, I think it is a good project, I didn't know autofs would be so good when it comes down to it.
It's usually documentation that puts you off ;-).
I often feel like I have to find the door in some invisible wall.
This time it was extreme because I couldn't for the life of me understand what a "key" would be in the sub-map file.
Regards, Bart.
--
To unsubscribe from this list: send the line "unsubscribe autofs" in
Ian Kent
maintainerQuoted 16 lines · Xen
Ian Kent schreef op 06-07-2016 2:57:
Quoted 15 lines · Xen
Quoted 9 lines
Hi,
a few things.
A user attempting to use autofs for the first time will assume that autofs will mount mount poins that already exist. Nowhere is it stated that mount points will only get created when accessed (through a "direct
path").Umm, "direct path" is an unfortunate choice here as you are talking about autofs
indirect mount maps.
Quoted 7 lines
Quoted 5 lines
Direct mount maps create a mount trigger for each map entry and so show up as
mount points in the file system.There are reasons for it being this way.
So you are berating me for choice of words ;-).
No, I'm trying to establish a common naming convention so both of us can avoid possible confusion later.
There are a bunch of conventions used (at least I use) that have specific meaning to me in relation to autofs. If I don't jump on those early there is often confusion about what I'm trying to say and there can be misunderstandings even when I do.
Quoted 11 lines
The only difference between direct and indirect seems to be that with direct the share is mounted "directly" on top of the specified mount point (actually, none is specified, it is mounted on the key returned from the map) and as a result it is not possible to have lazy initialization (unless you merged e.g. the root filesystem with an overlay filesystem) and so you are required to create this mountpoint, whereas with "indirect" you control the mount point itself and can choose whether to create the subdirectories or not (if you can acquire a list). "Directly" seems to mean "directly on the filesystem" whereas "indirectly" seems to mean: inside our own little enclave.
Yes, that's about it.
But I don't understand your use of "lazy initialization".
Yes, autofs will create a mount point path for each entry in a direct mount map. And yes, there will be an autofs file system mounted on each path to act as a trigger to cause the corresponding file system to be mounted during a kernel path walk (ie. when accessed).
So I don't quite understand what your concern is.
Sure, if you have hundreds or thousands of these there can be a performance problem but that's not what you were trying to do I thought.
It sounded to me like these were a good fit for your case .....
Quoted 87 lines
And always as a subdirectory of that. Not even technically necessary, you could even have a "indirect map" that triggered mounting I guess, don't know. I guess not.
Quoted 7 lines
Quoted 7 lines
Such a user (such as me) trying to use this thing (even with a tutorial
on Ubuntu site) will be flabbergasted that nothing happens; he/she expects mount entries to appear, but nothing happens.Quoted 8 lines
Quoted 3 lines
This renders the "nobrowse" default dysfunctional for a desktop user, who expects to see something before going there. It only works for automated systems, really.
Your assuming that autofs primary usage is targeted at the desktop. It's often not the best use case for that.
There are other more suitable automount alternatives for GUI desktops.
Linus Torvalds says he developed Linux as a desktop OS. And how chagrined he is that it didn't succeed in just that. Well, now we know the reason for that ;-).
I know there are probably alternatives; but they are mostly useless for someone who wants to do it on his own. Desktop environments often do not even have mount operations on directories (even) when they are described in fstab. There is no other way that I know of to automatically mount e.g. samba shares, unless you navigate directly to it (and then they end up in some weird location in the filesystem, if at all). There is usually also no "user based" fstab; where you can describe mounts specifically for your user, that are then mounted through e.g. fuse, or something else that can do suid. Perhaps you mean udisks or udisksd or udisksctl but I'm not even sure that supports network-filesystems, they are certainly not shown in udisksctl status.
Dolphin can not actually mount a share to some fixed location, the Nemo I am using doesn't even succeed in browsing the network. For me there are only two solutions that I see, either fstab or autofs. Once you know how to do it, running
/etc/auto.smb<host> quickly returns a list of shares (if you've set the credentials file) that you can then edit to become a static map with the browse option.I do not see what is better suited currently to auto-mounting samba-shares. Anything that would work well, would work from both the shell and the GUI, in that the systems have to be linked, and ideally what the GUI does is stored in state files that work equally well from the shell. Any persistent mount would have to be stored somewhere on disk. Then the link would need to be automounted, OR mounted through e.g. libpam-mount. There is nothing wrong with mounting upon logon. And that allows for specific user-dependent stuff. (An automount depending on what user first accesses it, is problematic). (As for Samba). So perhaps logon-mounts are better suited (just restore the mount from the previous session, in that sense) because the $UID problem is hard to solve for autofs, you must either duplicate all entries, and hardcore the UIDs you need, for every user, or I don't know what, prevent other users (including root) from accessing the share you need. You'd basically need map files in auto.master.d for every user, including a map file for each person to specify the shares, if you want browsable, or a modified version of auto.smb that mounts individual shares when accessed. (In the latter case you'd have no browse map).
But this script doesn't know what it is being called on, so now you need a variable I guess for each of those .d/ files, so that their entries would be something like:
/media/<user>/nas
/etc/auto-short.smb-Duser=<user> which would then be used for the samba $UID.If there is a better way to do this, I don't know. I see no alternatives other than libpam-mount and of course some GUI could do it, but then you'd really just have a form of duplicated functionality between autofs and it.
Quoted 9 lines
Quoted 6 lines
- browse only works for static maps, and what person with a static map is going to run into performance issues when mount points are created in
advance? Can we please remove this from the default config? (turning it
off).Well, quite a lot of current users actually.
Quoted 8 lines
Quoted 3 lines
Consider what can happen with a few hundred or a few thousand entries in an
indirect map.Alright, but what indirect maps can cause browse tables to be shown? Only static maps, right? A programmatic map can only return a single entry. I have not looked into the other ones, but I suppose LDAP is more like a programmatic map than anything else.
Of course, yes, program maps can never know in advance what key may be passed to them.
The problem is usually seen with user applications, cron jobs that scan directories, GUI file management utilities etc.
It's has been quite a problem over time.
LDAP is a map source, like a file, or nis etc.
Maps stored in LDAP behave much the same as maps stored in files.
Quoted 4 lines
So then consider that we only use a static map here. An actual file with thousands of entries? Of course it can happen. But what would be the point of that still (as opposed to LDAP or programmatic).
People do seem to like program maps but plenty of people use static or semi -static maps, ie. a bunch of entries followed by the wildcard map key "*" usually with some macro or "&" substution.
Similarly, what might match the wildcard map entry can't be known in advance.
Quoted 11 lines
Quoted 8 lines
Even though the browse option was added it has a specific set of problems.
First the kernel does not know if you are just listing a directory or actually
need to mount something.Do you mean the directory containing the (indirect) entries (e.g. the managed directory) or one of those entries themselves?
It's hard enough for me to understand VFS path walking sufficiently without having to describe it too.
All I can say is there is a problem determining whether a call back to the deamon is needed or not at the times the kernel module is called. That can sometimes result in all entries in the autofs managed mount point directory being mounted.
Another aspect of this problem is that the same system call may need to have the directory trigger a mount while not so at other times it's called and there's no way to tell what the usage is within in the kernel.
So sometimes people get caught because they want their stat(2) call to return the mounted file system results but not to cause a mount at other times when it might lead to a mount storm.
These people are usually quite passionate as well insisting that it be fixed and sometimes refuse to accept it is a problem.
Usuall all I can say to them is don't use the "browse" option.....
It is a difficult problem and I'm inevitably caught in the middle between the "browse" and "nobrowse" camps.
Not all cases of accessing such a directory will see this problem either but user space seems to have a habit of using system calls that don't trigger a mount together with others that do.
Causing the so called mount storm.
Having said that though, with a lot of work, I could probably do something about it.
TBH I've avoided it because it is difficult, perhaps in version 6.
It would require not pre-creating mount point directories and adding a readdir implementation to the kernel to fill in the names (map keys) that aren't already mounted. But that means significantly more (in terms of quantity of) communication between the daemon and the kernel module.
That means re-writing the user space <-> kernel space communications completely in an incompatible manner to what is used now, not to mention significant changes to the user space code itself.
A big job really.
Quoted 17 lines
Quoted 8 lines
So at the moment, certain types of system call (the module can't know the exact
system call either) don't cause a call back to perform a mount but often user
space will later make a system call that will cause a mount to occur anyway.Quoted 6 lines
That leads to sometimes seeing directories that are those of the autofs file
system not those of the mounted file system, so they have completely different
attributes to those after an automount has been done, a problem in itself.Oh... such as ownership etc.
Quoted 7 lines
The typical "I mount this, and now suddenly it is not root.root but xen.staff".
Quoted 2 lines
And, it can often lead to a mount storm, mounting everything in the directory.
But how can that happen? If I do; "for f in *; ls $f; done" sure.
It happens, from what I've seen, quite a bit.
Perhaps you would be surprised.
Quoted 7 lines
Quoted 6 lines
OK for a few dozen mounts in a map but really bad for anything with more than a
few hundred entries.
Quoted 7 lines
After thinking a little I came up with what could perhaps be some default use case. You have terminals mounting home directories. On the "terminal machine" you may have just a single user. Through LDAP you find the home directory. There is obviously no sense for all users in the LDAP system to now have precreated mount points. The only thing that could make sense is to have an "invisible" user directory that now gets mounted.
Yes, you would need to be using the default "nobrowse", or a limited set of keys with a wildcard entry.
Quoted 18 lines
I can see no reason to have hundreds of mount points in a directory unless they cannot conform to a specific pattern. If they follow a pattern, it is likely that only a few of them need to be mounted at any time. In that case being able to browse it seems nonsensical. An aggregate of hundreds of different mounts at the same time seems very weird. Weird to have a diverse set of unique names to begin with. As if all of them were manually added in some way. If they aren't manually added, but automatically, then why have a static list? Would you not be better off putting that in a database? And if you'd have so many, we'd assume they could be different users, or different websites, etc. We also assume not all of them are going to be needed, just a subset. But so now you have a system on which this list is statically created but dynamically updated. This list sits on the disk of the local system. But you're not querying a database (could be just as easy) that could also store this list and return the entries one by one when needed. The static file is the database.
Quoted 4 lines
Seems an improper use case but yeah. I don't know how you can have a static list that is huge while not choosing to use a database to manage it. Because a static map; now you need routines to update it, delete entries from it.
Yes, but network sources such as NIS and LDAP are treated much the same as a file map.
Checking if a file map has been updated since a key was last used is easier with a file map and straight forward with a NIS map but can't sensibly be done for an LDAP map.
Quoted 11 lines
Maybe I'm missing something here.
Quoted 3 lines
Another reason for the default is that, because of this, historically there was
no "browse" option and this maintains what is (should be) expected.Yeah I thought so, because of the comments near the timeout value. However that is a reason to never change anything ever. Someone mistakenly choosing "browse" will very quickly find out.
And wouldn't itself be sufficient but for the problems I'm attempting to describe above.
Quoted 39 lines
If this person really has hundreds of items in his list, he will notice within a second that they are all visible. If this person really has hundreds to thousands of items, he or she may also be expected to know the ins and outs of the system, and to be aware of the browse option, or be able to find out. This is not going to be a novice user.
On the other hand, anyone new to the system will have modest goals. However those modest goals are frustrated and refuted because the system is geared towards that professional that already knows everything, and for whom adding nobrowse, or changing the config file, would probably be a nobrainer. The new user however may be flabbergasted that it either does not work, or not the way she feels it should work (making a list, and then not seeing it in action).
Quoted 9 lines
Quoted 4 lines
- browse being turned off by default combined with a complete lack of mention that you need to access these "nonexistent" directory paths / components before you will be able to see them, turns autofs into a complete mystery for someone who is not privy to the details yet.
It's been a long time for me so I probably have forgotten what it's like for
someone starting out.Quoted 5 lines
And so, how would I know where in the man pages to put that so it gets noticed
by new users, perhaps from your perspective, you could point out a few key
places where it should be added, or better still send a patch.This happens a lot, it is no ones fault. I guess people who knows the ins and outs of something usually jump smack in the middle, and forget to mention what the system is really like from the outside. A fish doesn't notice the water, we don't notice the air.
We are too much immersed in it.
Quoted 7 lines
I think there are 3-4 things I would explain.
First the auto.master file on my system mentions man autofs as a place for seeing the format of the (sub)map files. However there is nothing in there. So that part is missing, but in a sense the contents of auto.misc is already somewhat sufficient for that, but still annoying that there is no man page.
That doesn't sound right, there might be a problem with your distribution package.
[raven@pluto autofs-dev.git]$ wc -l man/autofs.5 611 man/autofs.5
[raven@pluto autofs-dev.git]$ wc -l man/autofs.8.in 81 man/autofs.8.in
The first of these describes the Sun and amd map format map entries, not a particularly good job of it I admit, but that is where the changes you recommend should go.
The second describes the service start and stop, etc.
Quoted 13 lines
The second part is that it is often hard to understand what is meant by "key". In the master file, key means the mount point that supposedly already exists in the filesystem. In the sub-maps, key means the subdirectory but also the component of the path getting accessed by the user. Both types of files are called map files, and both kind of keys are called keys. This is aggravated by the fact that there is no man page for the sub-maps, that I know of . Auto.smb then speaks of a key, but it doesn't say what it is. Of course after a while you realize it is to be the host, but then you don't know how it knows about this host. Since at that point you don't realize that this is input from the filesystem (e.g. a stat operation) on "hidden directories" you are really clueless as to where this information is coming from ;-).
Sure, and that probably needs to be spelled out in autofs(5). I know people do get confused by this distinction.
Quoted 12 lines
You don't imagine that every freaking access of a directory or path component in the managed directory is going to result in network access.
You could do a "for f in
seq 1 10000000; do vdir $f; done" and have fun.(Don't know if that would still be the case for "browse" maps).
So this hidden directory, secret door thing is really the most confusing of all.
Quoted 6 lines
It's somewhat similar to something I might like to do, but still you don't really expect it ;-).
So the 3-4 things are:
create a man page or section for the format of the sub-maps
Already exists, needs work.
Lets call these mount maps (not the best but the convention used) as opposed to the master map.
Quoted 2 lines
indicate more clearly that programmatic maps in principle return a single entry
Maybe but isn't this in autofs(5) already close:
Executable Maps
A map can be marked as executable. A program map will be called with the key as an argument. It may return no lines of output if there's an error, or one or more lines containing a map entry (with \ quoting line breaks). The map entry corresponds to what would normally follow a map key.
Quoted 3 lines
indicate that programmatic maps currently require access of a path component and that this path component (user action) is the key to the script.
But that is the case for all maps:
Isn't this near the top of autofs(5) sufficient:
key [-options] location
key
For indirect mounts this is the part of the path name between the mount point and the path into the filesystem when it is mounted. Usually you can think about the key as a sub-directory name below the autofs managed mount point.
For direct mounts this is the full path of each mount point. This map is always associated with the /- mount point in the master map.
Quoted 2 lines
indicate that this is true for "no browse" on "static maps" as well, and that directories are only created when they are accessed.
That seems to be missing from the man page.
Quoted 1 line
And that no-browse is the configured default.
Not sure about that either.
This (from auto.master(5)) says that I have made "browse" the internal program default way back when version 5 was first developed but I install a configuration that turns it off because of (because of the above problems, which are not spelled out in the man page):
[no]browse
This is an autofs specific option that is a pseudo mount option and so is given without a leading dash. Use of the browse option pre-creates mount point directories for indirect mount maps so the map keys can be seen in a directory listing without being mounted. Use of this option can cause performance problem if the indirect map is large so it should be used with caution. The internal program default is to enable browse mode for indirect mounts but the default installed configuration overrides this by setting BROWSE_MODE to "no" because of the potential performance problem.
Don't forget that distributions can, and often do, install their own version of auto.master and possibly the autofs configuration.
And looking at the man pages shows that I have missed changing some of the configuration option names from the recent configuration file changes, something else that needs to be done.
Quoted 45 lines
Currently the documentation for the master map and for (indirect) sub-maps is really in auto.master and auto.misc.
Reading config files is usually a lot more pleasant than man pages ;-) so that's not such a huge problem, but you can't learn anything from a very limited set of examples either ;-).
For instance: direct map won't work if not prefixed with /, but will also not give error if not. Indirect map (I think) won't work if prefixed with /, but will also not give error if not.
I could say more but my thumb is hurting (infected) so I gotta quit this.
I don't mind writing documentation. But I'm not familiar yet with man pages and their technical structure and all.
I'll have to see. Thank you for the interest in any case :).
Quoted 15 lines
Quoted 11 lines
Further I would say that on a desktop OS, the most likely use case is going to be this:
- have several shares in your home network
- want to access them through a GUI
- prefer automounting since mounting at boot can be problematic
- need to see them before you can access them in a GUI
- only alternative is "bookmarks" or "places" that work regardless of the things not being visible/listed in the directory.Either use a different automounting mechanism or use a direct mount map.
Quoted 8 lines
That would satisfy all these needs, all be it with the map entries always being
mounted on access.The limitation of these is you can't have a direct map path that is within
another automounted path, if you need to do that you need to use a more complicated setup.
Quoted 6 lines
I think that limitation indicates why using relative paths is so much more pleasant. A map file using relative files can be "moved around". A map file using absolute paths cannot.
Could you not make a direct map out of the key of the master file, and the keys of the sub-maps?
Can't really do that.
The closest thing to that would be multi-mount map entries, again from autofs(5):
Multiple Mounts
A multi-mount map can be used to name multiple filesystems to mount. It takes the form:
key [ -options ] \
[[/] location \
[/relative-mount-point [ -options ] location...]...
This may extend over multiple lines, quoting the line-breaks with `\´. If present, the per-mountpoint mount-options are appended to the default mount-options. This behaviour may be overridden by the append_options configuration setting.
Quoted 11 lines
If there is really a need for huge static map files (I haven't studied LDAP) in principle all you need is a syntax to differentiate between the two, right.
/- ( /this/is/my/abs/path )
or
/this/is/my/abs ( -/path )
could be identical
Not really, the master map key "/-" is required to specify the direct mount maps that are to be used, anything else is an indirect mount, and "/-" is only valid within the master map (although I can't remember if I explicitly check for it within mount maps).
Quoted 7 lines
normally ( /bla/bla ) indicates an absolute path and can only be used in a direct map, from what I've seen. You *could* cause it to turn this "indirect map" into a direct one.
ie. ( bla/bla ) == indirect
( /bla/bla ) == direct.
In the Sun map format maps indirect mount map keys are a single path component only.
The situation is different if using amd format maps but I could work out how direct mount maps could be usefully used so that mount type has not been implemented. Even on Solaris where they are supposed to be fully supported I could make sense of them.
Quoted 13 lines
A direct map is always a static file right? But then it doesn't matter what part of the path is specified in what file.
Or
( bla/bla ) == indirect ( -/bla/bla ) == directSo you get:
/start/of/the (-/path )
Quoted 7 lines
or
/start/of/the/- ( path )
Then any path in the master file ending with /- would be a direct path.
And would cause the (static) map file to be treated as direct.
Nope, that is unlikely to work and would lead to a lot of problems.
One example might be with:
/start/of/the/-/path
people will ask where have the directories below /start/of/the/ gone and the answer will be they have been covered by the autofs file system mount.
Next it would be but I should be able to see these, surely that's easy to fix.
At hat point I would probably leave the conversation.
In any case a similar thing, but more explicit and I think easier to follow and maintain are the autofs multi-mount map entries, what I think you called relative path above.
It isn't clear but that syntax should function within both direct and indirect mount maps.
The problem with all this is that substitution cannot be used in the key.
The reason for that is same as why the amd regex key matching has not been implemented.
A linear map scan would be needed for every lookup in order to match what has been sent to us from the kernel. And as much as you might feel that wouldn't be a problem because maps are generally small I can assure you it would make like really unpleasant for me!
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
Then I will also say,
Quoted 8 lines
Quoted 3 lines
that there is no documentation on the format of the map files other than
auto.master. The documentation for map files is this:That's not quite right.
Man page autofs(5) has a bit of information on both the Sun and amd format maps.
Quoted 6 lines
Oh I am sorry. I think many users are not versed in the numbers for the man pages. I for one do not notice the difference and I almost never use one of those numbers. Sorry, I seriously missed that.
I only noticed just now because autofs(8) references autofs(5), and I'm like ....huh?
Yeah, that's what we have ....
Quoted 7 lines
Okay, allright. Missed that.
Seriously bad system if you ask me, but okay.
So I guess you can have direct maps with LDAP etc. too.
Quoted 8 lines
I guess the wildcard also solves the many-user problem.
Well, but only for the sub-map. You couldn't get autofs to manage /media, so it needs to be /media/user/<aggregate> in which user can't suddenly be a wildcard unless you use a direct map. You could expand that to include the master file though:
/media/*/nas
/etc/nas-shares.map
Can't do that, each /media/*/nas needs to be mounted at program start so that automounts can be done?
Chicken before the egg problem there.
You might be able to achieve something sensible with a wildcard and offset mounts (the relative paths above) and macro and & substution.
It is a bit difficult though.
Quoted 8 lines
And then in
/etc/nas-shares.map:share1 -fstype=xxx,uid=& ://nas/share1 share2 -fstype=xxx,uid=& ://nas/share2I can't see whether LDAP supports browse, I assume it must, since it supports direct.
Yep, same as file maps except remote maps cannot use plus map inclusion, like the
+auto.master
you might see in default installed master maps.
Quoted 9 lines
Quoted 8 lines
By definition a program map can't know what keys may be passed to it. And, yes, even autofs(5) doesn't say explicitly that a program map doesn't know
what keys may be passed to it, perhaps that should be ammended.
Quoted 4 lines
In principle even a program map could read all of the available keys from whatever database it consults, and return all of them; or return something only (as it does now, perhaps) when a given key matches (is found).
I've thought of that too.
But I've resisted because I'm concerned about confusing existing program maps people have in unpleasant ways.
But that suggestion has been made before and .....
Quoted 7 lines
If you wanted that you would need the program script to e.g. support a wildcard as parameter. Meaning, all scripts must then check for "*" to be passed to it, in order to return the full list that they can acquire (if even possible) or at least return an error of some kind to say that they don't support it.
Quoted 6 lines
You'd need a parameter for it, and any script would need to be aware of that parameter.
That's either gonna be a wildcard or something given on $2, in that case every existing script would already be compatible with it, since it simply disregards $2.
That's not a bad thought, but what would $1 be so that older scripts wouldn't choke on it.
Quoted 11 lines
So, for instance, if $2 is "browse" the script may choose to return a browse map, or list of shares, with whatever options are available (it could even return the UID/USER of the share given options, so that the created directory would have the same UID as the to be mounted share!) -- meaning it might simply return ALL entries it could possibly return in the default format, which is the meaning of the wildcard, and the program (autofs daemon or module(?)) would be able to generate the appropriate directory entries based on that. So in that case you're probably going to be rid of the problem of the non-matching state of pre- and post-mount of directories.
which pretty much defeats the only possible solution I can think of for "browse" option mounts ...
Quoted 34 lines
Quoted 8 lines
Quoted 4 lines
a) get rid of the
browse_mode= no, in the configuration file, for static mountsNo, that has it's own set of problems which people need to be aware of before
choosing to use the "browse" option.Not everyone.
I have like 5 mounts in this share. 5 shares in this mount. I have not noticed issues to begin with. Maybe there are issues like checking permissions on the directory prior to mounting it, and then being wrong about that once it is mounted. But still, that doesn't seem to happen. Actually, something funny is going on. 2 of the 8 shares that are 'automatically' loaded are not accessible. "No such file or directory".
No, that's nothing but a permission error. Still I am allowed to see them but not use them. For a static file no issue; just delete them.
I don't really see what I need to be aware of prior to using browse?
If it is the most sensible "novice" use case, why not allow it by default? It things go mayhem people can turn it off, or you could even turn it off when the number of mounts in a map exceeds 20 or 30 or 50. It's an arbitrary number, but what if I am wrong about LDAP.
What about you mean it is turned off by default for LDAP, but you do not consider static files with that?
What about it does function for ldap. What about we just need a separate default for LDAP and its like, than we do for static?
Quoted 6 lines
Are you seriously considering static files, or do you mean LDAP, yp, hisplus, and hesoid?
For anyone using an actual database, I concur.
For static, I just disagree. That is the start case for anyone.
Like I said, I wanted to have the default as "browse" but got hit with too many complaints and was only willing to change the package installed default configuration to satisfy them.
I'm not going to try that again.
Quoted 17 lines
Quoted 4 lines
Quoted 8 lines
Quoted 4 lines
b) say something about mounts not being created automatically in the man
page for auto.master, if nothing else. If you also do (a), this only applies now to programmatic mounts.
auto.master(5)isn't the place for that, autofs(5) is better and, as I say, make
some suggestions of provide a patch.autofs(8) is loaded before autofs(5). It's a good document, autofs(5), but most people may not even notice it. Personally I would probably choose to call them:
Quoted 5 lines
autofs.master(5)andautofs.map(5). But then, I also consider auto.master a very nondescript name.In this way no one can ever get confused, so you might just rename autofs(5) to
autofs.map(5).
Yeah maybe ...
Quoted 18 lines
Or even
auto.map(5)if that is necessary, but I wouldn't do that.Quoted 3 lines
I'll have a look at doing this but beware, often distributions use there own
default master map which I have no control over.I have no clue what mine does, this is Debian, and then Ubuntu, and then Kubuntu ;-) but it is just the Debian package still.
You don't have to follow my example or suggestion, maybe there is a better improvement than this, it was just an example, I guess. Or I could guess.
That was just a thing I was able to change without editing man files ;-).
Quoted 7 lines
I will look into providing a patch, but I hope you can rename autofs(5) to
autofs.map(5)before doing anything like that. I don't feel like improving a system I know myself haven't had access to, in that sense. I don't mean you have to do that straight away, but if that is a possible direction to go, or change to make, then I could supply a patch or anything based on the system that would result from that, seeing that autofs(5) is already pretty good in that sense.
At some point I'll need to return to this and make a list. Unfortunately that time isn't now.
Quoted 29 lines
Quoted 10 lines
Quoted 9 lines
I mean you could rewrite more documentation but this is the simplest I could do right here right now.
I think that would clarify a lot for everyone but of course perhaps you
would want to rewrite more so the central documentation is also up to date, but please see this as a more modest attempt for making it more accessible to a new user.Quoted 6 lines
Yes, adding the fact that mount point directories aren't pre created for
indirect maps somewhere inauto.master(5)and in a couple of places in autofs(5)
is a good idea. You are right it's something that can catch the unaware.That would also make the difference between direct and indirect more clear. (I still don't know exactly what is the entire difference).
That's also something I could change: how a direct map is explained in
auto.master(5). But then, I didn't have access to autofs(5) to explain it to me better ;-).Anyway, I think it is a good project, I didn't know autofs would be so good when it comes down to it.
It's usually documentation that puts you off ;-).
Quoted 8 lines
I often feel like I have to find the door in some invisible wall.
This time it was extreme because I couldn't for the life of me understand what a "key" would be in the sub-map file.
Regards, Bart.
--
To unsubscribe from this list: send the line "unsubscribe autofs" in
--
To unsubscribe from this list: send the line "unsubscribe autofs" in
unsubscribe autof
Quoted 817 lines · Ian Kent
Quoted 15 lines · Xen
Ian Kent schreef op 06-07-2016 2:57:
Quoted 14 lines · Xen
Quoted 9 lines
Hi,
a few things.
A user attempting to use autofs for the first time will assume that autofs will mount mount poins that already exist. Nowhere is it stated that mount points will only get created when accessed (through a "direct
path").Umm, "direct path" is an unfortunate choice here as you are talking about autofs
indirect mount maps.Quoted 6 lines
Quoted 5 lines
Direct mount maps create a mount trigger for each map entry and so show up as
mount points in the file system.There are reasons for it being this way.
So you are berating me for choice of words ;-).
No, I'm trying to establish a common naming convention so both of us can avoid possible confusion later.
There are a bunch of conventions used (at least I use) that have specific meaning to me in relation to autofs. If I don't jump on those early there is often confusion about what I'm trying to say and there can be misunderstandings even when I do.
Quoted 10 lines
The only difference between direct and indirect seems to be that with direct the share is mounted "directly" on top of the specified mount point (actually, none is specified, it is mounted on the key returned from the map) and as a result it is not possible to have lazy initialization (unless you merged e.g. the root filesystem with an overlay filesystem) and so you are required to create this mountpoint, whereas with "indirect" you control the mount point itself and can choose whether to create the subdirectories or not (if you can acquire a list). "Directly" seems to mean "directly on the filesystem" whereas "indirectly" seems to mean: inside our own little enclave.
Yes, that's about it.
But I don't understand your use of "lazy initialization".
Yes, autofs will create a mount point path for each entry in a direct mount map. And yes, there will be an autofs file system mounted on each path to act as a trigger to cause the corresponding file system to be mounted during a kernel path walk (ie. when accessed).
So I don't quite understand what your concern is.
Sure, if you have hundreds or thousands of these there can be a performance problem but that's not what you were trying to do I thought.It sounded to me like these were a good fit for your case .....
Quoted 81 lines
And always as a subdirectory of that. Not even technically necessary, you could even have a "indirect map" that triggered mounting I guess, don't know. I guess not.
Quoted 5 lines
Quoted 5 lines
Such a user (such as me) trying to use this thing (even with a tutorial
on Ubuntu site) will be flabbergasted that nothing happens; he/she expects mount entries to appear, but nothing happens.Quoted 7 lines
Quoted 3 lines
This renders the "nobrowse" default dysfunctional for a desktop user, who expects to see something before going there. It only works for automated systems, really.
Your assuming that autofs primary usage is targeted at the desktop. It's often not the best use case for that.
There are other more suitable automount alternatives for GUI desktops.
Linus Torvalds says he developed Linux as a desktop OS. And how chagrined he is that it didn't succeed in just that. Well, now we know the reason for that ;-).
I know there are probably alternatives; but they are mostly useless for someone who wants to do it on his own. Desktop environments often do not even have mount operations on directories (even) when they are described in fstab. There is no other way that I know of to automatically mount e.g. samba shares, unless you navigate directly to it (and then they end up in some weird location in the filesystem, if at all). There is usually also no "user based" fstab; where you can describe mounts specifically for your user, that are then mounted through e.g. fuse, or something else that can do suid. Perhaps you mean udisks or udisksd or udisksctl but I'm not even sure that supports network-filesystems, they are certainly not shown in udisksctl status.
Dolphin can not actually mount a share to some fixed location, the Nemo I am using doesn't even succeed in browsing the network. For me there are only two solutions that I see, either fstab or autofs. Once you know how to do it, running
/etc/auto.smb<host> quickly returns a list of shares (if you've set the credentials file) that you can then edit to become a static map with the browse option.I do not see what is better suited currently to auto-mounting samba-shares. Anything that would work well, would work from both the shell and the GUI, in that the systems have to be linked, and ideally what the GUI does is stored in state files that work equally well from the shell. Any persistent mount would have to be stored somewhere on disk. Then the link would need to be automounted, OR mounted through e.g. libpam-mount. There is nothing wrong with mounting upon logon. And that allows for specific user-dependent stuff. (An automount depending on what user first accesses it, is problematic). (As for Samba). So perhaps logon-mounts are better suited (just restore the mount from the previous session, in that sense) because the $UID problem is hard to solve for autofs, you must either duplicate all entries, and hardcore the UIDs you need, for every user, or I don't know what, prevent other users (including root) from accessing the share you need. You'd basically need map files in auto.master.d for every user, including a map file for each person to specify the shares, if you want browsable, or a modified version of auto.smb that mounts individual shares when accessed. (In the latter case you'd have no browse map).
But this script doesn't know what it is being called on, so now you need a variable I guess for each of those .d/ files, so that their entries would be something like:
/media/<user>/nas
/etc/auto-short.smb-Duser=<user> which would then be used for the samba $UID.If there is a better way to do this, I don't know. I see no alternatives other than libpam-mount and of course some GUI could do it, but then you'd really just have a form of duplicated functionality between autofs and it.
Quoted 8 lines
Quoted 6 lines
- browse only works for static maps, and what person with a static map is going to run into performance issues when mount points are created in
advance? Can we please remove this from the default config? (turning it
off).Well, quite a lot of current users actually.
Quoted 7 lines
Quoted 3 lines
Consider what can happen with a few hundred or a few thousand entries in an
indirect map.Alright, but what indirect maps can cause browse tables to be shown? Only static maps, right? A programmatic map can only return a single entry. I have not looked into the other ones, but I suppose LDAP is more like a programmatic map than anything else.
Of course, yes, program maps can never know in advance what key may be passed to them.
The problem is usually seen with user applications, cron jobs that scan directories, GUI file management utilities etc.
It's has been quite a problem over time.
LDAP is a map source, like a file, or nis etc.
Maps stored in LDAP behave much the same as maps stored in files.
Quoted 3 lines
So then consider that we only use a static map here. An actual file with thousands of entries? Of course it can happen. But what would be the point of that still (as opposed to LDAP or programmatic).
People do seem to like program maps but plenty of people use static or semi -static maps, ie. a bunch of entries followed by the wildcard map key "*" usually with some macro or "&" substution.
Similarly, what might match the wildcard map entry can't be known in advance.
Quoted 9 lines
Quoted 7 lines
Even though the browse option was added it has a specific set of problems.
First the kernel does not know if you are just listing a directory or actually
need to mount something.Do you mean the directory containing the (indirect) entries (e.g. the managed directory) or one of those entries themselves?
It's hard enough for me to understand VFS path walking sufficiently without having to describe it too.
All I can say is there is a problem determining whether a call back to the deamon is needed or not at the times the kernel module is called. That can sometimes result in all entries in the autofs managed mount point directory being mounted.
Another aspect of this problem is that the same system call may need to have the directory trigger a mount while not so at other times it's called and there's no way to tell what the usage is within in the kernel.
So sometimes people get caught because they want their stat(2) call to return the mounted file system results but not to cause a mount at other times when it might lead to a mount storm.
These people are usually quite passionate as well insisting that it be fixed and sometimes refuse to accept it is a problem.
Usuall all I can say to them is don't use the "browse" option.....
It is a difficult problem and I'm inevitably caught in the middle between the "browse" and "nobrowse" camps.
Not all cases of accessing such a directory will see this problem either but user space seems to have a habit of using system calls that don't trigger a mount together with others that do.
Causing the so called mount storm.
Having said that though, with a lot of work, I could probably do something about it.
TBH I've avoided it because it is difficult, perhaps in version 6.
It would require not pre-creating mount point directories and adding a readdir implementation to the kernel to fill in the names (map keys) that aren't already mounted. But that means significantly more (in terms of quantity of) communication between the daemon and the kernel module.
That means re-writing the user space <-> kernel space communications completely in an incompatible manner to what is used now, not to mention significant changes to the user space code itself.
A big job really.
Quoted 13 lines
Quoted 7 lines
So at the moment, certain types of system call (the module can't know the exact
system call either) don't cause a call back to perform a mount but often user
space will later make a system call that will cause a mount to occur anyway.Quoted 6 lines
That leads to sometimes seeing directories that are those of the autofs file
system not those of the mounted file system, so they have completely different
attributes to those after an automount has been done, a problem in itself.Quoted 8 lines
Oh... such as ownership etc.
The typical "I mount this, and now suddenly it is not root.root but xen.staff".
Quoted 2 lines
And, it can often lead to a mount storm, mounting everything in the directory.
But how can that happen? If I do; "for f in *; ls $f; done" sure.
It happens, from what I've seen, quite a bit.
Perhaps you would be surprised.Quoted 11 lines
Quoted 4 lines
OK for a few dozen mounts in a map but really bad for anything with more than a
few hundred entries.After thinking a little I came up with what could perhaps be some default use case. You have terminals mounting home directories. On the "terminal machine" you may have just a single user. Through LDAP you find the home directory. There is obviously no sense for all users in the LDAP system to now have precreated mount points. The only thing that could make sense is to have an "invisible" user directory that now gets mounted.
Yes, you would need to be using the default "nobrowse", or a limited set of keys with a wildcard entry.
Quoted 17 lines
I can see no reason to have hundreds of mount points in a directory unless they cannot conform to a specific pattern. If they follow a pattern, it is likely that only a few of them need to be mounted at any time. In that case being able to browse it seems nonsensical. An aggregate of hundreds of different mounts at the same time seems very weird. Weird to have a diverse set of unique names to begin with. As if all of them were manually added in some way. If they aren't manually added, but automatically, then why have a static list? Would you not be better off putting that in a database? And if you'd have so many, we'd assume they could be different users, or different websites, etc. We also assume not all of them are going to be needed, just a subset. But so now you have a system on which this list is statically created but dynamically updated. This list sits on the disk of the local system. But you're not querying a database (could be just as easy) that could also store this list and return the entries one by one when needed. The static file is the database.
Quoted 4 lines
Seems an improper use case but yeah. I don't know how you can have a static list that is huge while not choosing to use a database to manage it. Because a static map; now you need routines to update it, delete entries from it.
Yes, but network sources such as NIS and LDAP are treated much the same as a file map.
Checking if a file map has been updated since a key was last used is easier with a file map and straight forward with a NIS map but can't sensibly be done for an LDAP map.
Quoted 9 lines
Maybe I'm missing something here.
Quoted 3 lines
Another reason for the default is that, because of this, historically there was
no "browse" option and this maintains what is (should be) expected.Yeah I thought so, because of the comments near the timeout value. However that is a reason to never change anything ever. Someone mistakenly choosing "browse" will very quickly find out.
And wouldn't itself be sufficient but for the problems I'm attempting to describe above.
Quoted 36 lines
If this person really has hundreds of items in his list, he will notice within a second that they are all visible. If this person really has hundreds to thousands of items, he or she may also be expected to know the ins and outs of the system, and to be aware of the browse option, or be able to find out. This is not going to be a novice user.
On the other hand, anyone new to the system will have modest goals. However those modest goals are frustrated and refuted because the system is geared towards that professional that already knows everything, and for whom adding nobrowse, or changing the config file, would probably be a nobrainer. The new user however may be flabbergasted that it either does not work, or not the way she feels it should work (making a list, and then not seeing it in action).
Quoted 8 lines
Quoted 4 lines
- browse being turned off by default combined with a complete lack of mention that you need to access these "nonexistent" directory paths / components before you will be able to see them, turns autofs into a complete mystery for someone who is not privy to the details yet.
It's been a long time for me so I probably have forgotten what it's like for
someone starting out.Quoted 5 lines
And so, how would I know where in the man pages to put that so it gets noticed
by new users, perhaps from your perspective, you could point out a few key
places where it should be added, or better still send a patch.This happens a lot, it is no ones fault. I guess people who knows the ins and outs of something usually jump smack in the middle, and forget to mention what the system is really like from the outside. A fish doesn't notice the water, we don't notice the air.
We are too much immersed in it.
Quoted 7 lines
I think there are 3-4 things I would explain.
First the auto.master file on my system mentions man autofs as a place for seeing the format of the (sub)map files. However there is nothing in there. So that part is missing, but in a sense the contents of auto.misc is already somewhat sufficient for that, but still annoying that there is no man page.
That doesn't sound right, there might be a problem with your distribution package.
[raven@pluto autofs-dev.git]$ wc -l man/autofs.5 611 man/autofs.5
[raven@pluto autofs-dev.git]$ wc -l man/autofs.8.in 81 man/autofs.8.inThe first of these describes the Sun and amd map format map entries, not a particularly good job of it I admit, but that is where the changes you recommend should go.
The second describes the service start and stop, etc.
Quoted 4 lines
The second part is that it is often hard to understand what is meant by "key". In the master file, key means the mount point that supposedly already exists in the filesystem. In the sub-maps, key means the subdirectory but also the component of the path getting accessed by the
Quoted 8 lines
user. Both types of files are called map files, and both kind of keys are called keys. This is aggravated by the fact that there is no man page for the sub-maps, that I know of . Auto.smb then speaks of a key, but it doesn't say what it is. Of course after a while you realize it is to be the host, but then you don't know how it knows about this host. Since at that point you don't realize that this is input from the filesystem (e.g. a stat operation) on "hidden directories" you are really clueless as to where this information is coming from ;-).
Sure, and that probably needs to be spelled out in autofs(5). I know people do get confused by this distinction.
Quoted 11 lines
You don't imagine that every freaking access of a directory or path component in the managed directory is going to result in network access.
You could do a "for f in
seq 1 10000000; do vdir $f; done" and have fun.(Don't know if that would still be the case for "browse" maps).
So this hidden directory, secret door thing is really the most confusing of all.
Quoted 6 lines
It's somewhat similar to something I might like to do, but still you don't really expect it ;-).
So the 3-4 things are:
create a man page or section for the format of the sub-maps
Already exists, needs work.
Lets call these mount maps (not the best but the convention used) as opposed to the master map.Quoted 2 lines
indicate more clearly that programmatic maps in principle return a single entry
Maybe but isn't this in autofs(5) already close:
Executable Maps
A map can be marked as executable. A program map will be called with the key as an argument. It may return no lines of output if there's an error, or one or more lines containing a map entry (with \ quoting line breaks). The map entry corresponds to what would normally follow a map key.Quoted 3 lines
indicate that programmatic maps currently require access of a path component and that this path component (user action) is the key to the script.
But that is the case for all maps:
Isn't this near the top of autofs(5) sufficient:
key [-options] location
key
For indirect mounts this is the part of the path name between the mount point and the path into the filesystem when it is mounted. Usually you can think about the key as a sub-directory name below the autofs managed mount point.For direct mounts this is the full path of each mount point. This map is always associated with the /- mount point in the master map.
Quoted 2 lines
indicate that this is true for "no browse" on "static maps" as well, and that directories are only created when they are accessed.
That seems to be missing from the man page.
Quoted 1 line
And that no-browse is the configured default.
Not sure about that either.
This (from
auto.master(5)) says that I have made "browse" the internal program default way back when version 5 was first developed but I install a configuration that turns it off because of (because of the above problems, which are not spelled out in the man page):[no]browse
This is an autofs specific option that is a pseudo mount option and so is given without a leading dash. Use of the browse option pre-creates mount point directories for indirect mount maps so the map keys can be seen in a directory listing without being mounted. Use of this option can cause performance problem if the indirect map is large so it should be used with caution. The internal program default is to enable browse mode for indirect mounts but the default installed configuration overrides this by settingBROWSE_MODEto "no" because of the potential performance problem.Don't forget that distributions can, and often do, install their own version of auto.master and possibly the autofs configuration.
And looking at the man pages shows that I have missed changing some of the configuration option names from the recent configuration file changes, something else that needs to be done.
Quoted 41 lines
Currently the documentation for the master map and for (indirect) sub-maps is really in auto.master and auto.misc.
Reading config files is usually a lot more pleasant than man pages ;-) so that's not such a huge problem, but you can't learn anything from a very limited set of examples either ;-).
For instance: direct map won't work if not prefixed with /, but will also not give error if not. Indirect map (I think) won't work if prefixed with /, but will also not give error if not.
I could say more but my thumb is hurting (infected) so I gotta quit this.
I don't mind writing documentation. But I'm not familiar yet with man pages and their technical structure and all.
I'll have to see. Thank you for the interest in any case :).
Quoted 13 lines
Quoted 10 lines
Further I would say that on a desktop OS, the most likely use case is going to be this:
- have several shares in your home network
- want to access them through a GUI
- prefer automounting since mounting at boot can be problematic
- need to see them before you can access them in a GUI
- only alternative is "bookmarks" or "places" that work regardless of the things not being visible/listed in the directory.Either use a different automounting mechanism or use a direct mount map.
Quoted 8 lines
That would satisfy all these needs, all be it with the map entries always being
mounted on access.The limitation of these is you can't have a direct map path that is within
another automounted path, if you need to do that you need to use a more complicated setup.Quoted 6 lines
I think that limitation indicates why using relative paths is so much more pleasant. A map file using relative files can be "moved around". A map file using absolute paths cannot.
Could you not make a direct map out of the key of the master file, and the keys of the sub-maps?
Can't really do that.
The closest thing to that would be multi-mount map entries, again from autofs(5):
Multiple Mounts
A multi-mount map can be used to name multiple filesystems to mount. It takes the form:key [ -options ] \
[[/] location \
[/relative-mount-point [ -options ] location...]...This may extend over multiple lines, quoting the line-breaks with `\´. If present, the per-mountpoint mount-options are appended to the default mount-options. This behaviour may be overridden by the
append_optionsconfiguration setting.Quoted 11 lines
If there is really a need for huge static map files (I haven't studied LDAP) in principle all you need is a syntax to differentiate between the two, right.
/- ( /this/is/my/abs/path )
or
/this/is/my/abs ( -/path )
could be identical
Not really, the master map key "/-" is required to specify the direct mount maps that are to be used, anything else is an indirect mount, and "/-" is only valid within the master map (although I can't remember if I explicitly check for it within mount maps).
Quoted 6 lines
normally ( /bla/bla ) indicates an absolute path and can only be used in a direct map, from what I've seen. You *could* cause it to turn this "indirect map" into a direct one.
ie. ( bla/bla ) == indirect
( /bla/bla ) == direct.In the Sun map format maps indirect mount map keys are a single path component only.
The situation is different if using amd format maps but I could work out how direct mount maps could be usefully used so that mount type has not been implemented. Even on Solaris where they are supposed to be fully supported I could make sense of them.
Quoted 12 lines
A direct map is always a static file right? But then it doesn't matter what part of the path is specified in what file.
Or
( bla/bla ) == indirect ( -/bla/bla ) == directSo you get:
/start/of/the (-/path )
Quoted 7 lines
or
/start/of/the/- ( path )
Then any path in the master file ending with /- would be a direct path.
And would cause the (static) map file to be treated as direct.
Nope, that is unlikely to work and would lead to a lot of problems.
One example might be with:
/start/of/the/-/path
people will ask where have the directories below /start/of/the/ gone and the answer will be they have been covered by the autofs file system mount.
Next it would be but I should be able to see these, surely that's easy to fix.
At hat point I would probably leave the conversation.
In any case a similar thing, but more explicit and I think easier to follow and maintain are the autofs multi-mount map entries, what I think you called relative path above.
It isn't clear but that syntax should function within both direct and indirect mount maps.
The problem with all this is that substitution cannot be used in the key.
The reason for that is same as why the amd regex key matching has not been implemented.
A linear map scan would be needed for every lookup in order to match what has been sent to us from the kernel. And as much as you might feel that wouldn't be a problem because maps are generally small I can assure you it would make like really unpleasant for me!
Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
Then I will also say,
that there is no documentation on the format of the map files other than
auto.master. The documentation for map files is this:That's not quite right.
Quoted 8 lines
Quoted 2 lines
Man page autofs(5) has a bit of information on both the Sun and amd format maps.
Oh I am sorry. I think many users are not versed in the numbers for the man pages. I for one do not notice the difference and I almost never use one of those numbers. Sorry, I seriously missed that.
I only noticed just now because autofs(8) references autofs(5), and I'm like ....huh?
Yeah, that's what we have ....
Quoted 6 lines
Okay, allright. Missed that.
Seriously bad system if you ask me, but okay.
So I guess you can have direct maps with LDAP etc. too.
Quoted 8 lines
I guess the wildcard also solves the many-user problem.
Well, but only for the sub-map. You couldn't get autofs to manage /media, so it needs to be /media/user/<aggregate> in which user can't suddenly be a wildcard unless you use a direct map. You could expand that to include the master file though:
/media/*/nas
/etc/nas-shares.mapCan't do that, each /media/*/nas needs to be mounted at program start so that automounts can be done?
Chicken before the egg problem there.
You might be able to achieve something sensible with a wildcard and offset mounts (the relative paths above) and macro and & substution.
It is a bit difficult though.
Quoted 7 lines
And then in
/etc/nas-shares.map:share1 -fstype=xxx,uid=& ://nas/share1 share2 -fstype=xxx,uid=& ://nas/share2I can't see whether LDAP supports browse, I assume it must, since it supports direct.
Yep, same as file maps except remote maps cannot use plus map inclusion, like the
+auto.master
you might see in default installed master maps.Quoted 11 lines
Quoted 7 lines
By definition a program map can't know what keys may be passed to it. And, yes, even autofs(5) doesn't say explicitly that a program map doesn't know
what keys may be passed to it, perhaps that should be ammended.In principle even a program map could read all of the available keys from whatever database it consults, and return all of them; or return something only (as it does now, perhaps) when a given key matches (is found).
I've thought of that too.
But I've resisted because I'm concerned about confusing existing program maps people have in unpleasant ways.
But that suggestion has been made before and .....
Quoted 6 lines
If you wanted that you would need the program script to e.g. support a wildcard as parameter. Meaning, all scripts must then check for "*" to be passed to it, in order to return the full list that they can acquire (if even possible) or at least return an error of some kind to say that they don't support it.
Quoted 6 lines
You'd need a parameter for it, and any script would need to be aware of that parameter.
That's either gonna be a wildcard or something given on $2, in that case every existing script would already be compatible with it, since it simply disregards $2.
That's not a bad thought, but what would $1 be so that older scripts wouldn't choke on it.
Quoted 10 lines
So, for instance, if $2 is "browse" the script may choose to return a browse map, or list of shares, with whatever options are available (it could even return the UID/USER of the share given options, so that the created directory would have the same UID as the to be mounted share!) -- meaning it might simply return ALL entries it could possibly return in the default format, which is the meaning of the wildcard, and the program (autofs daemon or module(?)) would be able to generate the appropriate directory entries based on that. So in that case you're probably going to be rid of the problem of the non-matching state of pre- and post-mount of directories.
which pretty much defeats the only possible solution I can think of for "browse" option mounts ...
Quoted 31 lines
Quoted 6 lines
Quoted 3 lines
a) get rid of the
browse_mode= no, in the configuration file, for static mountsNo, that has it's own set of problems which people need to be aware of before
choosing to use the "browse" option.Not everyone.
I have like 5 mounts in this share. 5 shares in this mount. I have not noticed issues to begin with. Maybe there are issues like checking permissions on the directory prior to mounting it, and then being wrong about that once it is mounted. But still, that doesn't seem to happen. Actually, something funny is going on. 2 of the 8 shares that are 'automatically' loaded are not accessible. "No such file or directory".
No, that's nothing but a permission error. Still I am allowed to see them but not use them. For a static file no issue; just delete them.
I don't really see what I need to be aware of prior to using browse?
If it is the most sensible "novice" use case, why not allow it by default? It things go mayhem people can turn it off, or you could even turn it off when the number of mounts in a map exceeds 20 or 30 or 50. It's an arbitrary number, but what if I am wrong about LDAP.
What about you mean it is turned off by default for LDAP, but you do not consider static files with that?
What about it does function for ldap. What about we just need a separate default for LDAP and its like, than we do for static?
Quoted 6 lines
Are you seriously considering static files, or do you mean LDAP, yp, hisplus, and hesoid?
For anyone using an actual database, I concur.
For static, I just disagree. That is the start case for anyone.
Like I said, I wanted to have the default as "browse" but got hit with too many complaints and was only willing to change the package installed default configuration to satisfy them.
I'm not going to try that again.
Quoted 13 lines
Quoted 9 lines
Quoted 6 lines
b) say something about mounts not being created automatically in the man
page for auto.master, if nothing else. If you also do (a), this only applies now to programmatic mounts.
auto.master(5)isn't the place for that, autofs(5) is better and, as I say, make
some suggestions of provide a patch.autofs(8) is loaded before autofs(5). It's a good document, autofs(5), but most people may not even notice it. Personally I would probably choose to call them:
Quoted 5 lines
autofs.master(5)andautofs.map(5). But then, I also consider auto.master a very nondescript name.In this way no one can ever get confused, so you might just rename autofs(5) to
autofs.map(5).Yeah maybe ...
Quoted 16 lines
Or even
auto.map(5)if that is necessary, but I wouldn't do that.Quoted 3 lines
I'll have a look at doing this but beware, often distributions use there own
default master map which I have no control over.I have no clue what mine does, this is Debian, and then Ubuntu, and then Kubuntu ;-) but it is just the Debian package still.
You don't have to follow my example or suggestion, maybe there is a better improvement than this, it was just an example, I guess. Or I could guess.
That was just a thing I was able to change without editing man files ;-).
Quoted 7 lines
I will look into providing a patch, but I hope you can rename autofs(5) to
autofs.map(5)before doing anything like that. I don't feel like improving a system I know myself haven't had access to, in that sense. I don't mean you have to do that straight away, but if that is a possible direction to go, or change to make, then I could supply a patch or anything based on the system that would result from that, seeing that autofs(5) is already pretty good in that sense.At some point I'll need to return to this and make a list. Unfortunately that time isn't now.
Quoted 28 lines
Quoted 8 lines
Quoted 8 lines
I mean you could rewrite more documentation but this is the simplest I could do right here right now.
I think that would clarify a lot for everyone but of course perhaps you
would want to rewrite more so the central documentation is also up to date, but please see this as a more modest attempt for making it more accessible to a new user.Quoted 6 lines
Yes, adding the fact that mount point directories aren't pre created for
indirect maps somewhere inauto.master(5)and in a couple of places in autofs(5)
is a good idea. You are right it's something that can catch the unaware.That would also make the difference between direct and indirect more clear. (I still don't know exactly what is the entire difference).
That's also something I could change: how a direct map is explained in
auto.master(5). But then, I didn't have access to autofs(5) to explain it to me better ;-).Anyway, I think it is a good project, I didn't know autofs would be so good when it comes down to it.
It's usually documentation that puts you off ;-).
I often feel like I have to find the door in some invisible wall.
Quoted 8 lines
Quoted 6 lines
This time it was extreme because I couldn't for the life of me understand what a "key" would be in the sub-map file.
Regards, Bart.
--
To unsubscribe from this list: send the line "unsubscribe autofs" in--
To unsubscribe from this list: send the line "unsubscribe autofs" in
--
To unsubscribe from this list: send the line "unsubscribe autofs" in
Ian Kent schreef op 06-07-2016 14:48:
Quoted 1 line
But I don't understand your use of "lazy initialization".
I meant creating the mount point only when it was accessed.
Quoted 3 lines
So I don't quite understand what your concern is.
Sure, if you have hundreds or thousands of these there can be a performance problem but that's not what you were trying to do I thought.
I'm not sure either at this point but I'm too hungry to read back on what I had written before :p.
Quoted 7 lines
Of course, yes, program maps can never know in advance what key may be passed to them.
The problem is usually seen with user applications, cron jobs that scan directories, GUI file management utilities etc.
It's has been quite a problem over time.
Yes that is my concern. I do not like the idea of invisible access points at all, unless I was doing it for security, but in that sense you can't really hide anything in a Linux system. Invisible mount points only make sense when you have LOTS of them and you only need few at a time.
Quoted 3 lines
LDAP is a map source, like a file, or nis etc.
Maps stored in LDAP behave much the same as maps stored in files.
Right so you can browse them if you want.
Quoted 6 lines
Quoted 5 lines
So then consider that we only use a static map here. An actual file with thousands of entries? Of course it can happen. But what would be the point of that still (as opposed to LDAP or programmatic).
Quoted 6 lines
People do seem to like program maps but plenty of people use static or semi -static maps, ie. a bunch of entries followed by the wildcard map key "*"
usually with some macro or "&" substution.Similarly, what might match the wildcard map entry can't be known in advance.
Right. That makes it programmatic; the (server) response will determine if anything was actually there. On the other hand a wildcard (in a direct map at least) may determine existing (or match existing) filesystem components that you could read in advance. And you need to, in a direct map. But direct maps are mounted anyway (or the mountpoint is created, I mean). So it currently only applies to indirect maps where the wildcard is the final component, and it then behaves like a programmatic thing where it only actually gets mounted (and created) if e.g. the server reponds with a share.
Quoted 2 lines
It's hard enough for me to understand VFS path walking sufficiently without having to describe it too.
I don't know. I studied it in some detail, I just don't understand the RCU algorithm. At least, I have studied path_lookupat and the entire procedure leading up to lookup_fast and lookup_slow. I'd have to check my printouts to say more. But let's just say I'm aware of any (recursive) operation involving symlinks, where intermediate symlinks get expanded before the original path is being continued, etc. I am also aware of the presence of autofs code, particularly in the function that specifies one or two other mount things.
I believe that is called by lookup_slow....
Quoted 4 lines
All I can say is there is a problem determining whether a call back to the deamon is needed or not at the times the kernel module is called. That can sometimes result in all entries in the autofs managed mount point directory being mounted.
So the mount point is accessed but the module doesn't know if anything really needs to happen at that point. And stat operations or anything else all use the same path lookup. Of course I don't know what happens pre-mount and post-mount to the directory itself. If post-mount the directory is really on another filesystem, the "mnt" value will change. And it will return another dentry for that other mount. I remember "path" being mnt+dentry.
Quoted 4 lines
Another aspect of this problem is that the same system call may need to have the directory trigger a mount while not so at other times it's called and there's no way to tell what the usage is within in the kernel.
So you are getting called by lookup_slow or something of the kind near it, but you're not sure why. But these lookups are being done for each individual path componenent. However a "cd" to the directory should mount it, but a mere "stat" may not. Is that the case? Seems odd that it would be difficult, but I believe you on your word.
Maybe "cd" shouldn't mount it until the contents is accessed, but I don't know.
Quoted 4 lines
So sometimes people get caught because they want their stat(2) call to return the mounted file system results but not to cause a mount at other times when it might lead to a mount storm.
Hmm. It seems like the first desire is a problem. At least, I don't know how directory lookup is done. I don't know if this information is the result of a stat call or something else. I do remember seeing code for dentry-moving but it was confusing.
For sure you want "ls /directory/" to return the mounted contents. But I'm not sure if its possible for your module to differentiate. Ls -d "/directory/" should perhaps not cause mounting. I can see quickly though that "ls" causes an "open" call but "ls -d" does not.
But then, "open" will not access any path components beyond the directory itself. So if your module is only called by path_lookup it is going to be impossible to know what for.
Particularly if an "ls" does not actually access path componenets (beyond the directory itself) there would be no way to know what is being asked.
Quoted 1 line
Usuall all I can say to them is don't use the "browse" option.....
:).
Quoted 2 lines
It is a difficult problem and I'm inevitably caught in the middle between the "browse" and "nobrowse" camps.
You appear to be very conscientious about it.
Quoted 3 lines
Not all cases of accessing such a directory will see this problem either but user space seems to have a habit of using system calls that don't trigger a mount together with others that do.
such as that "ls" vs "ls -d"?
Quoted 1 line
Causing the so called mount storm.
Still seems odd but I believe you, but. I guess what it could mean is "ls" not only trying to list every share (browse) but also to perform calls on those shares (path componenents retrieved) that might require additional information. For example, if I am not content just knowing the shares, but also want to know the number of files in them.... which definitely some programs or user agents would do. Then I'm in trouble because this requires mounting. Then, if that is the case, a simple
"list /nas/" might result in the subfolders (/nas/media, /nas/backup, etc.) to also be queried and you have what you call a mount-storm.
But this is the result of user space convention and behaviour. Surely people cannot request browse while at the same time wanting to deny actions on the returned results when these actions stem completely from userspace?
If some user agent or GUI is trying to list the number of files in each directory (which at this point means a share) then that's not something autofs could ever deal with. And I don't think it is a problem autofs needs to solve.
If I am at all correct about this and I see what you mean right now, this "GUI" behaviour would then simply be incompatible with unmounted folders *that-might-need-to-get-mounted-but-not-always* and this GUI would need to be aware to not do that thing.
Quoted 11 lines
Having said that though, with a lot of work, I could probably do something about it.
TBH I've avoided it because it is difficult, perhaps in version 6.
It would require not pre-creating mount point directories and adding a readdir implementation to the kernel to fill in the names (map keys) that aren't already mounted. But that means significantly more (in terms of quantity of) communication between the daemon and the kernel module.
Quoted 5 lines
That means re-writing the user space <-> kernel space communications completely in an incompatible manner to what is used now, not to mention significant changes to the user space code itself.
A big job really.
And not a big joy perhaps.
So basically you are going to not precreate, but still hijack kernel calls to return the listings for the directories that do not yet exist (the listing of those directories).
Which means those directories could be shown, but not opened, or something of the kind. Then, the user agent .. well I don't know what difference it makes actually if the mount storm is the result of userspace calls that directly cause it. So I guess I'm mistaken here and the issue is even more complex.... :(.
I don't know how a readdir on the "top" directory could /directly/ causes a mountstorm on the listed entries. But then, I don't know much ;-).
Quoted 2 lines
It happens, from what I've seen, quite a bit.
Perhaps you would be surprised.
Illuminate me :P. What is the most common cause of it?
Quoted 2 lines
Yes, but network sources such as NIS and LDAP are treated much the same as a file map.
Yes, I didn't realize at the start. I have always found LDAP to be a rather incomprehensible thing to work with. Without hands-on experience it is hard to get a good grasp of how it works. Or even how to organise it (if you do create one) as there seem to be standards for everything.
Quoted 3 lines
Checking if a file map has been updated since a key was last used is easier with a file map and straight forward with a NIS map but can't sensibly be done for an LDAP map.
Right.
Quoted 9 lines
Quoted 9 lines
Maybe I'm missing something here.
Quoted 3 lines
Another reason for the default is that, because of this, historically there was
no "browse" option and this maintains what is (should be) expected.
Quoted 6 lines
Quoted 3 lines
Yeah I thought so, because of the comments near the timeout value. However that is a reason to never change anything ever. Someone mistakenly choosing "browse" will very quickly find out.
And wouldn't itself be sufficient but for the problems I'm attempting to describe above.
You mean the mountstorm problems are reasons to keep the default at no-browse?
It still seems at this point that this is only relevant for LDAP/NIS entries.
Quoted 8 lines
Quoted 5 lines
First the auto.master file on my system mentions man autofs as a place for seeing the format of the (sub)map files. However there is nothing in there. So that part is missing, but in a sense the contents of auto.misc is already somewhat sufficient for that, but still annoying that there is no man page.
That doesn't sound right, there might be a problem with your distribution package.
No, as I say later this is incomprehension on my part ;-). Or merely the fact that needing to use "number" for accessing "sections" of the "manual" is so rare (if you are not a developer accessing library functions and/or kernel functions) that a regular user will almost NEVER use a section number in daily operation.
Most configuration files are called "openvpn.conf" or something of the kind, and hence their manual page is also called that, and you don't need to do "man 5 openvpn.conf", because "man openvpn.conf" is sufficient.
Using numbers would be superfluous in 99% of cases, or even more than that. As opposed to perhaps requesting help on function calls, such as mount() as opposed to /bin/mount.
So as a developer you may be used to it. But a regular user is not. Anyone who does not do C programming, is not.
So, my mistake, I guess.
Quoted 15 lines
Quoted 15 lines
You don't imagine that every freaking access of a directory or path component in the managed directory is going to result in network access.
You could do a "for f in
seq 1 10000000; do vdir $f; done" and have fun.(Don't know if that would still be the case for "browse" maps).
So this hidden directory, secret door thing is really the most confusing of all.
It's somewhat similar to something I might like to do, but still you don't really expect it ;-).
Quoted 7 lines
Quoted 3 lines
So the 3-4 things are:
create a man page or section for the format of the sub-maps
Already exists, needs work.
Lets call these mount maps (not the best but the convention used) as opposed to the master map.
Alright. Still confusing though.
Quoted 4 lines
Quoted 3 lines
indicate more clearly that programmatic maps in principle return a single entry
Quoted 8 lines
Maybe but isn't this in autofs(5) already close:
Executable Maps
A map can be marked as executable. A program map will be called with the key as an argument. It may return no lines of output if there's an error, or one or more lines containing a map entry (with \ quoting line breaks). The map entry corresponds to what would normally follow a map key.
I guess.. it's just that you read over "a map entry" as meaning "only a single map entry" since just before it says "one or more lines". So it is something you only notice if you reread it in detail. After finding out that it doesn't work ;-).
Quoted 11 lines
Isn't this near the top of autofs(5) sufficient:
key [-options] location
key
For indirect mounts this is the part of the path name between the mount point and the path into the filesystem when it is mounted. Usually you can think about the key as a sub-directory name below the autofs managed mount point.
Quoted 3 lines
For direct mounts this is the full path of each mount point. This map is always associated with the /- mount point in the master map.
Yes it is, particularly because it mentions that it is a sub-directory name.
Then, I begin to wonder what "usually" means. See, from the looks of it it is completely identical to sub-directory name, because:
- they key is going to be directly used as the name of the subdirectory
- programmatically, the script does not even have any say about the key, it cannot even return a key, does not determine the key. The only thing that can ever happen is that the key as given is turned into a subdirectory (when valid).
I mean the logical abstraction between "key" and "value" may make sense from a purely abstract logical point of view ;-). But it has no bearing on actual domain-specific termininology, it is just a computer science term so to speak.
What it actually comes down to is:
- mount point for the actual share, or
- subdirectory created as mountpoint for the actual share(s).
Where "value" would mean:
- path to the mount (share), or
- network path to the share that gets mounted on the subdirectory (the key).
the "mount" utility calls these source and target, respectively (source for the path, target for the mount point). Otherwise it calls them "device" and "mountpoint".
So although "key" also has meaning with regards to "database lookup" I don't see how it is ever not directly used as the mountpoint itself. And this is what creates the confusion because with the "browse" option it is not just going to be a "lookup key".
Further there is no difference between "subdirectory mount point" and "lookup key", where these are fundamentally separate concepts, and could even imply, in principle, that the actual directory to use is *also* retrieved by the key.
So conceptually the idea that "key" instantly means "subdirectory mount-point" is confusing.
In fact it is odd to begin with, from the perspective of a user, that "auto.smb" mounts the entire share list of the requested "host" or "key" in one go under the location that was accessed.
This is not the most common scenario (well, maybe if you are in some big network). This is not most common not only because it mounts /all/ the shares at once(!) but also because generally people want to have some level of control over the directory it ends up in, and what is the host for some server (hostname) might not be where you actually want to mount it.
So I suggest that even this auto.smb is a bit of a weird entity. I have used it to generate a map list for me that I then adjusted and used as a static map file.
I know it does the same as that -net or -host thing. It's just that for a desktop user (again, you say that is not the target audience, but still) what you really want is a fixed location the user chooses (in auto.master) and then a browseable list of shares. Apparently this list needs to be static. What I envisioned before would be a parameter to a script that causes it to output a list of shares rather than the share entries themselves, or just the complete list of it.
This would give it "browse" capability. If you had that, you would need to be able to pass a parameter to the script as the key, but this is rather obscure.
What it comes down to is that this is what I would envision for myself:
Auto.master:
/cifs /etc/auto.smb diskstation <-- diskstation is the key
Now the program/daemon/whatever passes diskstation as key to the script. It also passes a wildcard parameter such as "*". The script now returns a list of shares for "diskstation". Subsequently when the daemon needs to mount, it calls the script with diskstation and the requested mount point/share. Alternatively, the returned output from the first call must be complete and is saved by the daemon. It is then used in place of a static map (the script is now nothing but a static map generator).
Personally I prefer the former: the script has two modes: list and return entry, and it therefore has two parameters: one to identify what to list, and the second to determine what entry to return from what was listed.
Otherwise, without the parameter, you'd need to hardcode it in the script and duplicate the script for every host, etc.
In fact as a user you may not even want all shares to be present on your local system. So you might equally well decide to use a static map. So perhaps you will use "smb.auto list" or "smb.auto list diskstation" or "smb.auto diskstation *" manually. And perhaps it generates all of the mappings for you that you can then prune and save.
Perhaps, in absense of the key (the entry, or wildcard) a call such as "smb.auto diskstation" should simply generate the share list, whereas "smb.auto diskstation media" generates the key for only that entry.
So you see me use a term like "entry" to determine what you today call a "key" and the word "key" to determine what you today call a "entry" ;-).
"entry" means "element of pre-existing list", "key" means "value to use for a lookup operation in a database".
"key" can also mean "that what you need to mount or access something".
so if you combine the two, you have like three things:
- subdirectory (mount point)
- key to use for database lookup
- entry in the list of shares
but you can only solve it if you differentiate between "lookup operation" and "element of a list".
a lookup operation may /return/ a list, which is why I said:
"diskstation" (hostname) is a key
list of shares is the result of that lookup operation based on that key entry is an element of that list you can mount.
This applies to a form of browse scenario. If you have invisible mounpoint, you get:
"diskstation" (hostname) is a key that is going to be a direct subdirectory the list of shares it returns is going to be a subdirectory list to that first subdirectory this will contain the actual shares, so you get top/key/entries.
And you see we basically have the same thing except that this is one level deeper in our hierarchy.
In my desired use case, you get:
top/entry
But in the current implementation of e.g. smb.auto, you get:
top/key/entry
So it is clear diskstation (hostname) IS a key, except that in the current case or scenario this key is ALSO used as the direct descendent of the toplevel directory used for the configuration in auto.master.
Which gives rise to the idea that maybe "key" and "subdirectory mount point" should not be equivalent or identical. Not necessarily identical.
In fact, smb.auto DOES produce a browsable list. It is just /beneath/ the subdirectory for its "key" (the hostname) and mounts ALL of the shares at once.
So you now already have that mountstorm (it just gets mounted by default) it's just a level deeper.
You now don't have a mountstorm on the level of hosts, but you have a mountstorm on the level of shares (per host) but it is still the same thing.
So the design specifies that keys must become mountpoints in the "sub-top-level" directory (the directory managed by autofs for indirect mounts).
Whereas perhaps conceptually you would need to differentiate between key, and an entry used for mounting, and the difference is that this entry used for mounting can be an element of the list returned by the key.
So you get key -> list -> entries. In the case of current auto.smb, what happens is:
key -> (subdirectory) -> list (itself an entry) -> entries.
Or in other words:
key -> list -> entries, but mounted beneath top/key, as key is used as an extra directory in between.
Current auto.smb:
<managed mount point> / <key> / entries / contents of shares
My preference:
<managed mount point> / entries / contents of shares
And why? Only because I want to specify mount points for specific hosts manually.
So the question becomes how you can combine such things.
The requirement becomes for the script to return the mount point. A static list does return mount points. A script does not, the mount point is specified as the "key" given to it. So if you want the script to determine the mount point the way a static list does, you need it to return, in essence, a static list.
Even if your static list is not browseable, so too can a returned "dynamic list" also be not browseable. But at this point your script can return a mount point of its own. That is again problematic if the key is derived from filesystem access.
So you see how hard this is to reconcile.
If we insist in accessing hidden shares and deriving keys from user action or filesystem action, or filesystem access, then keys are predetermined the moment the system is actually called. The script or database can only return whether something is gonna be there or not.
Then the only way to reconcile it is (in my case) for the key to be not the only argument to the script, but I give the script my argument I define in auto.master:
/cifs /etc/auto.smb diskstation
Now my script knows that e.g. $1 is gonna be the filesystem requested path, and $2 is gonna be the custom parameter I determined in advance.
I don't need to use the key by itself ($1) to do the lookup. My real key is $2. But I will use the second key ($1) to return a share for the host determined by the first key ($2).
In essence what happens with auto.smb is that the key ($1) is used as the host (first key) and smb.auto will itself derive and load the list of shares, and then instantly mount that (it retrieves all of the second keys and instantly uses that and mounts everything).
autofs allows for this second list of keys to be instantly used and utilized.
But it's not the common use case because you are now fixed to that location the user accesses and of course you could use that location (the key) ($1) to look up a host in a different way, that is perfectly possible. If your script marries "diskstation" to "ds" or "ds" to "diskstation" then you could automount /cifs/ds and it would request shares from host "diskstation".
But you're still stuck with that intermediate level, that extra level of subdirectories.
It's not possible currently to use the "first key" to determine the host and then get the "second keys" ($1) to determine the entries. Well, actually it is, but not in a browseable fashion:
/cifs /etc/auto.smb -Dhost=diskstation
I believe this could work. You pass your real "primary" key as that variable, and your script returns shares based on that key, but with the "filesystem key" ($1) as the name of the share to return.
Still, rather convoluted at this point, and not possible to browse this list.
Because it doesn't return a list. Using $2 to pass that variable wouldn't really be different.
But the script is only called when an entry is accessed (through the filesystem) and as such is never called without a secondary key.
The current auto.smb doesn't have secondary keys, it loads and mounts all of them instantly and automatically, and there is no "automount" for the secondary level.
Well, there is, it just mounts all of it in one go.
So how do you skip the intermediate level?
- call the script on purpose to generate a 'static list' instead of just for entries alone.
- use the static list to precreate directories (mount points), or
- use the static list to return readdir information, or to filter requests for directories, or don't use the static list at all and just keep it the same.
So we also see here that the static list *could* be used for the thing you described above, that alternative readdir implementation.
Although it's a bit of a hack, I'm sure.
You could specify some interval for invalidating the list. At which point it is renewed by another "get list" call to the script.
So in essence I am really advocating "return list" functionality for scripts here.
And there is no easy way to implement that while staying compatible with current scripts because you would need to: call the script with an empty parameter (that will never return an entry) or with a wildcard (that will probably never return an entry) and in essence, that might just as well mean, that if you want to take this approach:
- use the wildcard as the "list" operation, scripts that do not know about the wildcard will simply return zero entries.
- use the extra parameter as $2, in other words: allow for additional parameters in auto.master in that form of:
/cifs /etc/auto.smb diskstation random
But there are already recognised options. So what we have today (-D) I guess already works, it is just a bit obscure, and it seems to be the equivalent of what you would do.
Then an alternative auto.smb (I called it auto-short.smb) would:
- be called as:
/cifs /etc/auto.smb -Dhost=diskstation
- return a list of shares when called with "host=diskstation /etc/auto.smb *"
- alternatively you could have it be called with: "/etc/auto.smb * diskstation"
- make sure the shell doesn't expand that * ;-).
- autofs calls this list operation each time auto.master is reread.
- autofs calls this list operation each time some invalidation timeout occurs
- not really any other way to trigger it.
- the static list being returned (really a dynamic list now) can be used for readdir, or even access filtering (it contains all the valid keys).
Now when you have something like this, you can:
- call the filesystem component you use to access the actual shares, the "entry" or simply "share" or "directory". But "key" becomes more comprehensible now as well. "key" now corresponds to "entries in the returned list". Each "entry in the returned list" is now a "mapping between key (directory) and value (device-path). I realize "mount point" might confuse with the mountpoint defined in the master file.
"submount" would be ugly. "subdir" is most convenient. "subkey" is illustrative. "entry" describes the entire thing, not just the key. But in essence we are just talking about a mount-point here.
Nevertheless in principle the "first key" given should not even need to be the mount point in fact. It is a key to a lookup-table.
If you wanted to do it right I think you would need to do away with hidden (or principally unknown) mount points, and basically always work with static lists (or dynamically returned lists) and/or give every script a first parameter $1 that is a key that you actually use for it.
Then the second paramter $2 would be the key that the filesystem has actually given you for accessing a path.
In this way "access path" becomes secondary.
Perhaps a script in this way still cannot control the mount point (name) but at least now it provides it in advance. Of course there are situations in which you cannot know it in advance or in which it is irrelevant to know it in advance.
You can imagine a list to be much too big to actually retrieve it. In itself a sign of a bad system, I think. If it was that big, you would need hierarchy to make it smaller.
So I don't know if there is ever any real need for obscurity, and for hosts to be accessed while not being able to list them in advance.
I conjecture that this use case in itself would be odd, that you create auto-mounts for an unspecified number of hosts you don't really know about in advance.
It makes for zero-configuration, true. You don't have to keep a list anywhere or even arrange for it to be generated.
On the other hand if a list could not be had, the script would just return nothing and the daemon will assume no-browse no-information-in-advance lookups the way it does today.
Basically if you keep the current $1 and use it to pass a wildcard, no script is likely to ever break on that (it will just return nothing). Then $2 can be your primary key. Not every script needs it.
But complete zero-configuration is to me an odd thing: a magical directory that will mount every host given to it including all of its shares, does not seem sensible to me in the sense that you would at /least/ want some configuration for every system right?
One size fits all: the same system for every computer, just place it somewhere in your filesystem and you have access to everything and everything.
No, normally you would want to tailor it because you do not have access to an infinite amount of hosts and the /cifs/host/share structure therefore makes not much sense.
/device/share may make more sense, perhaps added with /hosts/hosts-meeting-a-certain-pattern/share
You already have a pretty big system if you use the latter. Anyway.
I think I should conclude this :p.
What seems sensible to me at least is to add something that doesn't break compability by using a wildcard * for scripts that will return a certain format of list (or just all shares it can find in the regular format) with a $2 parameter designating something such as a "host" or perhaps even "department".
In this way you solve the issue that the current auto.smb requires the "hostname" to be first key given to it because there is no other way to pass a key. Then the actual keys can be passed as "second parameter" in this sense that they still become $1, but at least now you can specify your host to it in advance instead of guessing it on the wind ;-).
So if I have to say anything I would say that I recommend this structure:
- $1 remains the filesystem key
- $2 becomes the additional key a user may specify in auto.master
- $1 as * will designate the request for a list lookup
- the list can in the future be used for readdir implementation, and currently for precreated mountpoints in case "browse" is used.
Quoted 14 lines
Quoted 2 lines
indicate that this is true for "no browse" on "static maps" as well, and that directories are only created when they are accessed.
That seems to be missing from the man page.
Quoted 1 line
And that no-browse is the configured default.
Not sure about that either.
This (from
auto.master(5)) says that I have made "browse" the internal program default way back when version 5 was first developed but I install a configuration that turns it off because of (because of the above problems, which are not spelled out in the man page):
Quoted 12 lines
[no]browse
This is an autofs specific option that is a pseudo mount option and so is given without a leading dash. Use of the browse option pre-creates mount point directories for indirect mount maps so the map keys can be seen in a directory listing without being mounted. Use of this option can cause performance problem if the indirect map is large so it should be used with caution. The internal program default is to enable browse mode for indirect mounts but the default installed configuration overrides this by settingBROWSE_MODEto "no" because of the potential performance problem.
I know, but if you, as a user, do not understand what the hell it is doing in the first place, and you have no time for extensive manual reading in advance (like me) you will never get to that option because you still don't understand the basics, and the more advanced stuff (like that) that will be too hard to understand.
That information is too difficult to comprehend (or digest) when you are someone who still doesn't understand how it works.
Quoted 8 lines
Quoted 6 lines
I think that limitation indicates why using relative paths is so much more pleasant. A map file using relative files can be "moved around". A map file using absolute paths cannot.
Could you not make a direct map out of the key of the master file, and the keys of the sub-maps?
Can't really do that.
It's not that I saw it as necessary, I just still think the difference and distinction between "direct" and "indirect" is rather I dunno, weird I guess ;-).
Quoted 11 lines
The closest thing to that would be multi-mount map entries, again from autofs(5):
Multiple Mounts
A multi-mount map can be used to name multiple filesystems to mount. It takes the form:key [ -options ] \
[[/] location \
[/relative-mount-point [ -options ] location...]...
Quoted 5 lines
This may extend over multiple lines, quoting the line-breaks
with
`\´. If present, the per-mountpoint mount-options are appended to the default mount-options. This behaviour may be overridden by theappend_optionsconfiguration setting.
Something I am already using (almost) and I believe this is what auto.smb uses by default too. It's just that I use it to mount parts of the same share. (Samba smbd often has issues with file-system boundaries so I create a share out of each filesystem and share them individually, then mount them as above).
Quoted 25 lines
Quoted 7 lines
normally ( /bla/bla ) indicates an absolute path and can only be used in a direct map, from what I've seen. You *could* cause it to turn this "indirect map" into a direct one.
ie. ( bla/bla ) == indirect
( /bla/bla ) == direct.In the Sun map format maps indirect mount map keys are a single path component only.
The situation is different if using amd format maps but I could work out how direct mount maps could be usefully used so that mount type has not been implemented. Even on Solaris where they are supposed to be fully supported I could make sense of them.
Quoted 5 lines
/start/of/the/- ( path )
Then any path in the master file ending with /- would be a direct path.
And would cause the (static) map file to be treated as direct.
Quoted 8 lines
Nope, that is unlikely to work and would lead to a lot of problems.
One example might be with:
/start/of/the/-/path
people will ask where have the directories below /start/of/the/ gone and the answer will be they have been covered by the autofs file system mount.
But that's no different from a regular mount on that location.
Quoted 3 lines
Next it would be but I should be able to see these, surely that's easy to fix.
At hat point I would probably leave the conversation.
:-). Haha. But come on, you can't take that seriously, the same thing happens with a regular mount, even a bind mount does that. It's absolutely not possible to reveal the contents of some subtree when another filesystem had mounted on top of it, except by remounting the entire subtree elsewhere.
(I am trying out some gimmick that requires me to mount the root device under something like /run/mount/rootfsclone, because I am bind mounting on top of something that I still want to be accessible. The only way to make it accessible is to mount the entire rootfs somewhere else. I basically need a persistent location for individual devices without anything mounted on top of them; ie. a pure one-file-system mount tree.
Then you have those individual mount-trees (filesystem trees) and THEN you have the complete FHS that you normally use.
That seems like the only sensible way to approach a lot of issues. Having a /mounts/ with every (custom) mounted device present there.
Recently the number of incomprensible mounts reported by "mount" has grown so big that the entire tool has become unusable, mostly due to cgroups. And tmpfs's, and debugfs's, and what not.
I now find I need to create a wrapper for mount so that "mount" stays at least usable to the user. Such that every subdirectory of /sys, and every subdirectory of /run, and every subdirectory of /dev/, is actually hidden from display. And from /proc as well. And the same applies to DF.
(I really need to find a way to better save and redistribute these modifications).
Quoted 10 lines
In any case a similar thing, but more explicit and I think easier to follow and maintain are the autofs multi-mount map entries, what I think you called relative path above.
It isn't clear but that syntax should function within both direct and indirect mount maps.
The problem with all this is that substitution cannot be used in the key.
Quoted 7 lines
The reason for that is same as why the amd regex key matching has not been implemented.
A linear map scan would be needed for every lookup in order to match what has been sent to us from the kernel. And as much as you might feel that wouldn't be a problem because maps are generally small I can assure you it would make life really unpleasant for me!
:). You have lost of enemies; if you don't do things right :P. LOL.
I am not thinking about this now, but I guess you mean that my proposed solution of saving those static lists (or dynamic lists) would actually help this :p.
Actually what you are saying is that even doing lookups on those lists I proposed would be quite annoying.
However currently this lookup is already done on the server (if it is some database) or in the static map (if they are not browseable). Right?
Databases are generally better at lookup than small-scale applications. Now you need hash-tables etc. Anyway, I can't think about it right now.
Quoted 8 lines
Quoted 2 lines
I only noticed just now because autofs(8) references autofs(5), and I'm like ....huh?
Yeah, that's what we have ....
Quoted 4 lines
Quoted 1 line
/media/*/nas
/etc/nas-shares.mapCan't do that, each /media/*/nas needs to be mounted at program start so that automounts can be done?
Actually the shell easily expands such a pattern to all matching trees?
e.g. if you create /usr/local/doc, and then do
"ls /usr/*/doc/" you will get the output of:
/usr/local/doc /usr/share/doc
So with that output, you mount all of them, all the while knowing that the middle component (here) is a wildcard. Then the daemon knows about this wildcard and can use it while processing /etc/nas-shares.map for each of them; but they are all individual mounts at that point.
e.g. /media/me/nas and /media/pamela/nas are now individual autofs mounts.
but their "/etc/nas-shares.map" is called with a different wildcard.
I mean that either /media/me/nas will already exist (precreated by the user) or your program creates it (I think it already does that).
Actually it cannot create it DUH, so the user will need to precreate them.
I know /media is usually seen as a managed directory but this is annoying in the first place. Such an important name (/media) and I cannot do anything with it because it is already taken. At the same time accessing it from your home directory is usually quite annoying. So in order to make sense of it I must precreate /media/xen and /media/dannielle and prevent the daemon (that /media thing) from wiping them.... buh.
The funny thing is that /media/xen is not even owned by xen.
The subfolders are, but the thing itself isn't (how stupid?).
So in this case perhaps not as convenient, but in the general case if I precreate:
/mytree/me/nas
/mytree/quinty/nas
etc.
Then the daemon could read
/mytree/*/nas
on program start and see all the shares that match. All the directories that match.
Then it knows that "me" and "quinty" are wildcard entries for their respective mounts.
Quoted 14 lines
Quoted 9 lines
Quoted 8 lines
By definition a program map can't know what keys may be passed to it. And, yes, even autofs(5) doesn't say explicitly that a program map doesn't know
what keys may be passed to it, perhaps that should be ammended.Quoted 4 lines
In principle even a program map could read all of the available keys from whatever database it consults, and return all of them; or return something only (as it does now, perhaps) when a given key matches (is found).
Quoted 4 lines
I've thought of that too.
But I've resisted because I'm concerned about confusing existing program maps people have in unpleasant ways.
At some point you need to be willing to break previous compability.
I understand that Linux is usually a running-upgrade system and even if people do upgrade entire distributions they want the system to keep functioning as normal without changing anything.
But e.g. the Debian update program can tell you stuff like "This package does something different now, and you need to ensure it can either handle or ignore the wildcard".
on every package installation.
Quoted 2 lines
That's not a bad thought, but what would $1 be so that older scripts wouldn't choke on it.
I had suggested * because there is no hostname called *.
But scripts may very well mount local filesystems as well. And they might very well use a shell to do stuff like "[ -d $key ]" or "[ -d "$key" ]" and....
And as always, stuff breaks a lot when people don't use "$key". If people's scripts are badly written then [ -d $key ] will malfunction.
In order to cover that you would have to use something like "_", no one is going to use _ as an actual filename.
Quoted 13 lines
Quoted 12 lines
So, for instance, if $2 is "browse" the script may choose to return a browse map, or list of shares, with whatever options are available (it could even return the UID/USER of the share given options, so that the created directory would have the same UID as the to be mounted share!) -- meaning it might simply return ALL entries it could possibly return in the default format, which is the meaning of the wildcard, and the program (autofs daemon or module(?)) would be able to generate the appropriate directory entries based on that. So in that case you're probably going to be rid of the problem of the non-matching state of pre- and post-mount of directories.
Quoted 2 lines
which pretty much defeats the only possible solution I can think of for "browse" option mounts ...
You mean the one suggested above. But if you make an alternative readdir having a list available won't bug you right?
I mean, having mount information available in advance cannot be a problem, only a help.
Quoted 5 lines
Like I said, I wanted to have the default as "browse" but got hit with too many complaints and was only willing to change the package installed default configuration to satisfy them.
I'm not going to try that again.
"Design by committee" they call it ;-).
Someone told me that OpenSUSE has had "browse" as the default for at least 5 years.
You did think about making a distinction between static and LDAP/NIS maps? I'm not sure I am not crossing my bounds here now ;-). Seeing that I already suggested it.
Quoted 14 lines
Quoted 2 lines
In this way no one can ever get confused, so you might just rename autofs(5) to
autofs.map(5).Yeah maybe ...
Quoted 7 lines
I will look into providing a patch, but I hope you can rename autofs(5) to
autofs.map(5)before doing anything like that. I don't feel like improving a system I know myself haven't had access to, in that sense. I don't mean you have to do that straight away, but if that is a possible direction to go, or change to make, then I could supply a patch or anything based on the system that would result from that, seeing that autofs(5) is already pretty good in that sense.
Quoted 2 lines
At some point I'll need to return to this and make a list. Unfortunately that time isn't now.
Here is just my short summarize:
- suggested compromise between browse and nobrowse by only making browse for static lists the default to please new people unaware of what will happen
- suggested perhaps using _ as the wildcard key, not using that "browse" flag as $2, but allowing a different "primary" key as $2 if the script wants it.
- suggested changing autofs(5) to autofs.map(5) such that users will be able to find it.
And the rest is up to you, I think you don't need anyone rewriting it for you now, but I could still try my hand if that was necessary. I just think using better terminology is dependent on the first two suggestions here, particularly perhaps the second, and otherwise you'll have to explain current terminology better and create some way to separate "key" in auto.master from "key" in "autofs.map" ;-).
However if scripts could accept an additional "key" and the default operation or capability would for them to be able to return a "list" then it would become more clear that the additional "key" would be the index to retrieve that list, and that the entries resulting from that may have their own "keys" but these "keys" are then used for the individual "mount points".
So then you have reason to call the entry keys "entry keys" and the other ones "lookup keys" for instance. Then, static maps use "entry keys" but scripts have "lookup keys" as their 2nd parameter. The script then has "entry key" as the first, and "lookup key" as the second parameter. Then, entry key is equivalent to subdirectory, and lookup key is often going to be equivalent to e.g. hostname.
A script without list functionality only accepts "entry keys" but may very well use them as lookup keys as well.
A database also uses them as equivalent. But at least now you have something of a model to conceptually separate them. Since browse already functions for e.g. LDAP, in many cases you will not use a lookup key for that, simply because (a) /etc/autofs_ldap_auth.conf already contains such paramters and (b) the other parameters are specified in the mapname-format for LDAP.
So what we call "lookup key" would already be encoded in the LDAP URL but such a thing is not possible for scripts.
So you could call scripts the exception to the rule in not having this. Of course it exists,: the "-D" parameters.
(I think? Haven't tested it yet).
So I don't know.
For a database the entry key is always going to be a lookup key to the database. Entry keys can also result from a list fetch.
So they can result from filesystem access or from some list retrieval.
But if "browse" becomes more normal and standard, you would see that it becomes easier to talk about this "entries from the list you get when quering the database/script for it".
Then the gap between "entries from the list" and "subdirectories created" is smaller than the gap between "filesystem access" and "subdirectories created".
Meaning: key (database designator) -> list -> entries (database entries) is conceptually easier to grasp than key (filesystem access) -> entry (database entry).
Or key (filesystem access) -> database lookup -> entry.
Simply because it is hard to understand that user action (filesystem lookup) is the input to the entire system and is actually the keys being used.
Even if it's not always to be used, having a level in between that ties "key" to "pre-existing shares" or "pre-existing entries" becomes easier. When a user thinks of "key" they think about either existing shares or existing hosts.
Then conceptually it is logical to assume that the script or database can give back this list. And that either this list is precreated on the harddrive, or at least that it is used to match the "keys" against, for instance.
(By harddrive I mean filesystem (hierarchy)).
This is why I think standard operation for at least smaller lists should be a form of browse or at least a form of list retrieval, if you are going to have dynamically generated lists in any case.
Then, it conforms more to a sense of what a general computer user is going to have of the idea of auto-mounting mount-points. It (he/she) thinks those mount-points are already going to exist, prior to being automounted.
This in turn makes it easier to explain that the key is really a subdirectory, and that there can be another key if needed to specify the host, or something of the kind.
Then, the most common use case for a new or general computer user wanting to try this is going to be met, because that use case is going to involve auto-mounting network-shares of which there is a limited number.
And it will involve something like
/nas/media
/nas/backup
/nas/home
etc.
Now I'm not sure if it covers the entire use. But at least you'd also have a script that can generate a static map for you that you can then use in place of the scripts, if you want to prune the mounts that are actually visible to you.
Apart from the distributions themselves there is not really any software in the Linux world that is tailored to either the desktop user, or the server user, but not both, is there?
Apart from the desktop environments.
Anyway this was a long read and write, I guess, for both of us :P.
Time to get some food again or at least some water :p.
See ya.
--
To unsubscribe from this list: send the line "unsubscribe autofs" in
Ian Kent
maintainerQuoted 21 lines · Xen
Ian Kent schreef op 06-07-2016 14:48:
Quoted 1 line
But I don't understand your use of "lazy initialization".
I meant creating the mount point only when it was accessed.
Quoted 4 lines
So I don't quite understand what your concern is. Sure, if you have hundreds or thousands of these there can be a performance
problem but that's not what you were trying to do I thought.I'm not sure either at this point but I'm too hungry to read back on what I had written before :p.
Quoted 7 lines
Of course, yes, program maps can never know in advance what key may be passed to
them.The problem is usually seen with user applications, cron jobs that scan directories, GUI file management utilities etc.
Quoted 7 lines
Quoted 1 line
It's has been quite a problem over time.
Yes that is my concern. I do not like the idea of invisible access points at all, unless I was doing it for security, but in that sense you can't really hide anything in a Linux system. Invisible mount points only make sense when you have LOTS of them and you only need few at a time.
And that is a fairly common case.
Don't forget the implications of scanning these directories during expire. It doesn't take much to increase the overhead of that.
Like I said before, the amount of work to do this is huge.
What is needed is a re-write of the user space <-> kernel space communication essentially to add readdir functionality.
So that listing a directory would show the possible map mount points without the need to create the mount point directories.
That is still open to excessive unnecessary mounting but not nearly as much as pre-created directories.
Quoted 7 lines
Quoted 5 lines
LDAP is a map source, like a file, or nis etc.
Maps stored in LDAP behave much the same as maps stored in files.
Right so you can browse them if you want.
Yep.
Quoted 18 lines
Quoted 9 lines
Quoted 8 lines
So then consider that we only use a static map here. An actual file with
thousands of entries? Of course it can happen. But what would be the point of that still (as opposed to LDAP or programmatic).Quoted 8 lines
People do seem to like program maps but plenty of people use static or semi
-static maps, ie. a bunch of entries followed by the wildcard map key "*"
usually with some macro or "&" substution.Similarly, what might match the wildcard map entry can't be known in advance.
Quoted 9 lines
Right. That makes it programmatic; the (server) response will determine if anything was actually there. On the other hand a wildcard (in a direct map at least) may determine existing (or match existing) filesystem components that you could read in advance. And you need to, in a direct map. But direct maps are mounted anyway (or the mountpoint is created, I mean). So it currently only applies to indirect maps where the wildcard is the final component, and it then behaves like a programmatic thing where it only actually gets mounted (and created) if e.g. the server reponds with a share.
A matter of terminology really, I don't see them as programmatic myself.
But also direct mount maps cannot contain a wildcard entry.
They must be distinct and known at the time the map is read because each entry is used to mount an autofs file system to act as a trigger for the actual mount.
That's why they can't be nested in any way since that could cover other direct mounts below the nesting point when a mount is triggered.
Quoted 15 lines
Quoted 5 lines
It's hard enough for me to understand VFS path walking sufficiently without
having to describe it too.I don't know. I studied it in some detail, I just don't understand the RCU algorithm. At least, I have studied
path_lookupatand the entire procedure leading up tolookup_fastandlookup_slow. I'd have to check my printouts to say more. But let's just say I'm aware of any (recursive) operation involving symlinks, where intermediate symlinks get expanded before the original path is being continued, etc. I am also aware of the presence of autofs code, particularly in the function that specifies one or two other mount things.
Quoted 1 line
I believe that is called by
lookup_slow....
Yeah, while there are no locks held when following a mount point (where automounting might be done) during the path walk you can't do most of the remainder of the path walking code lockless or rather without taking references to the struct path, you might see the terms ref-walk and I think rcu-walk in the code, that's the main difference between lookup_slow() and lookup_fast() IIRC.
The name rcu-walk I think is possibly not the best naming.
It implies a lockless and reference-less path walk which is much faster for certain work loads. But if the underlying structures are in danger of changing in ways that can't be properly handled by the read-copy-update (RCU) mechanism the path walk needs to revert to a ref-count(ed) walk which also allows locking.
That really just means taking a reference to the struct path (and the ability to take a lock on certain other structures when needed) so it can't go away from under you while you are using it, starting over when you hit such a case.
Quoted 11 lines
Quoted 10 lines
All I can say is there is a problem determining whether a call back to the
deamon is needed or not at the times the kernel module is called. That can
sometimes result in all entries in the autofs managed mount point directory
being mounted.
Quoted 7 lines
So the mount point is accessed but the module doesn't know if anything really needs to happen at that point. And stat operations or anything else all use the same path lookup. Of course I don't know what happens pre-mount and post-mount to the directory itself. If post-mount the directory is really on another filesystem, the "mnt" value will change. And it will return another dentry for that other mount. I remember "path" being mnt+dentry.
Actually the main automounting function of the module will be called back with essentially the path component only.
Most of the logic to decide if a mount should be done has now been taken into the VFS mount following logic.
And all it has are the lookup flags, like LOOKUP_DIRECTORY, LOOKUP_CREATE, etc.
Essentially all that's known is the class of lookup, directory, create, continue or other, which is a loose division of the related system calls, where continue means an intermediate path component.
The directory, create and continue classes trigger automounts, other classes usually don't (which are generally stat type system calls et. al.).
Quoted 8 lines
Quoted 7 lines
Another aspect of this problem is that the same system call may need to have the
directory trigger a mount while not so at other times it's called and there's no
way to tell what the usage is within in the kernel.
Quoted 8 lines
So you are getting called by
lookup_slowor something of the kind near it, but you're not sure why. But these lookups are being done for each individual path componenent. However a "cd" to the directory should mount it, but a mere "stat" may not. Is that the case? Seems odd that it would be difficult, but I believe you on your word.Maybe "cd" shouldn't mount it until the contents is accessed, but I don't know.
Yeah, but "cd" isn't a system call it's a bunch of code that makes system calls which must result in a mount because this directory is to be set in the VFS as a process working directory, it must be fully resolved at that point.
Quoted 19 lines
Quoted 8 lines
So sometimes people get caught because they want their stat(2) call to return
the mounted file system results but not to cause a mount at other times when it
might lead to a mount storm.Hmm. It seems like the first desire is a problem. At least, I don't know how directory lookup is done. I don't know if this information is the result of a stat call or something else. I do remember seeing code for dentry-moving but it was confusing.
For sure you want "ls /directory/" to return the mounted contents. But I'm not sure if its possible for your module to differentiate. Ls -d "/directory/" should perhaps not cause mounting. I can see quickly though that "ls" causes an "open" call but "ls -d" does not.
Quoted 7 lines
But then, "open" will not access any path components beyond the directory itself. So if your module is only called by
path_lookupit is going to be impossible to know what for.Particularly if an "ls" does not actually access path componenets (beyond the directory itself) there would be no way to know what is being asked.
There's one other thing to be aware of in all this.
Adding a "/" to the end of a path tells the kernel path walk this must resolve to a directory so the LOOKUP_DIRECTORY flag will be set and automounting will occur in that case.
That is about all the control available in user space and TBH not a big change to code if you really need stat to return the actual mounted file system information, given that non-directories can't be mount points and so aren't a problem.
Quoted 14 lines
Quoted 3 lines
Usuall all I can say to them is don't use the "browse" option.....
:).
Quoted 3 lines
It is a difficult problem and I'm inevitably caught in the middle between the
"browse" and "nobrowse" camps.You appear to be very conscientious about it.
Quoted 7 lines
Quoted 5 lines
Not all cases of accessing such a directory will see this problem either but
user space seems to have a habit of using system calls that don't trigger a
mount together with others that do.such as that "ls" vs "ls -d"?
Can't remember the difference between these two codewise but, for example, doing a stat(2) call then an opendir(3) and not being able to handle the fact that this changes from a directory that doesn't have a mount on it to one that does between the two system calls.
Which has caused find(1) to get very confused (in the somewhat distant past now though), ;)
Quoted 25 lines
Quoted 3 lines
Causing the so called mount storm.
Still seems odd but I believe you, but. I guess what it could mean is "ls" not only trying to list every share (browse) but also to perform calls on those shares (path componenents retrieved) that might require additional information. For example, if I am not content just knowing the shares, but also want to know the number of files in them.... which definitely some programs or user agents would do. Then I'm in trouble because this requires mounting. Then, if that is the case, a simple
"list /nas/" might result in the subfolders (/nas/media, /nas/backup, etc.) to also be queried and you have what you call a mount-storm.
But this is the result of user space convention and behaviour. Surely people cannot request browse while at the same time wanting to deny actions on the returned results when these actions stem completely from userspace?
If some user agent or GUI is trying to list the number of files in each directory (which at this point means a share) then that's not something autofs could ever deal with. And I don't think it is a problem autofs needs to solve.
Quoted 4 lines
If I am at all correct about this and I see what you mean right now, this "GUI" behaviour would then simply be incompatible with unmounted folders *that-might-need-to-get-mounted-but-not-always* and this GUI would need to be aware to not do that thing.
That's about right, and isn't a problem until someone with a largish map set for browse decides to access it with a GUI, then the complains start, ;)
But it isn't an unreasonable expectation it's just problematic to make it work the way we would like.
Quoted 24 lines
Quoted 24 lines
Having said that though, with a lot of work, I could probably do something about
it.TBH I've avoided it because it is difficult, perhaps in version 6.
It would require not pre-creating mount point directories and adding a readdir
implementation to the kernel to fill in the names (map keys) that aren't already
mounted. But that means significantly more (in terms of quantity of) communication between the daemon and the kernel module.That means re-writing the user space <-> kernel space communications completely
in an incompatible manner to what is used now, not to mention significant
changes to the user space code itself.
Quoted 7 lines
Quoted 1 line
A big job really.
And not a big joy perhaps.
So basically you are going to not precreate, but still hijack kernel calls to return the listings for the directories that do not yet exist (the listing of those directories).
Yes, similar to what the NFS READDIR or READDIR_PLUS RPC methods do. It's not a new idea by any stretch.
Quoted 10 lines
Which means those directories could be shown, but not opened, or something of the kind. Then, the user agent .. well I don't know what difference it makes actually if the mount storm is the result of userspace calls that directly cause it. So I guess I'm mistaken here and the issue is even more complex.... :(.
I don't know how a readdir on the "top" directory could /directly/ causes a mountstorm on the listed entries. But then, I don't know much ;-).
It's usually not actually the readdir that causes the problems it's associated calls done by code surrounding the readdir that generally it.
I mean if stat(2) was to cause a mount to occur then there would be a lot of mount storms reading directories.
Implementing an in kernel readdir won't get rid of the possibility of everything in a directory being mounted on access but it would go a fair way to avoiding cases where you didn't want or need everything mounted upon listing a directory.
Just how effect it would be has been something I've thought about many times, still not sure of the actual answer.
Quoted 6 lines
Quoted 4 lines
It happens, from what I've seen, quite a bit.
Perhaps you would be surprised.Illuminate me :P. What is the most common cause of it?
GUI applications.
Cron jobs scanning directories and causing so many mounts they cripple the system.
The point of the statement was more about number of people that happen to have large maps than anything.
Quoted 8 lines
Quoted 3 lines
Yes, but network sources such as NIS and LDAP are treated much the same as a
file map.Yes, I didn't realize at the start. I have always found LDAP to be a rather incomprehensible thing to work with. Without hands-on experience it is hard to get a good grasp of how it works. Or even how to organise it (if you do create one) as there seem to be standards for everything.
You think just checking it out looks incomprehensible, the autofs LDAP lookup module code is very verbose (incomprehensible really) and quite hard to work with.
I'd be much happier if LDAP never existed.
Quoted 47 lines
Quoted 7 lines
Checking if a file map has been updated since a key was last used is easier with
a file map and straight forward with a NIS map but can't sensibly be done for an
LDAP map.Right.
Quoted 9 lines
Quoted 9 lines
Maybe I'm missing something here.
Quoted 3 lines
Another reason for the default is that, because of this, historically there was
no "browse" option and this maintains what is (should be) expected.Quoted 7 lines
Quoted 3 lines
Yeah I thought so, because of the comments near the timeout value. However that is a reason to never change anything ever. Someone mistakenly choosing "browse" will very quickly find out.
And wouldn't itself be sufficient but for the problems I'm attempting to
describe above.You mean the mountstorm problems are reasons to keep the default at no-browse?
It still seems at this point that this is only relevant for LDAP/NIS entries.
Quoted 11 lines
Quoted 7 lines
First the auto.master file on my system mentions man autofs as a place for seeing the format of the (sub)map files. However there is nothing in
there. So that part is missing, but in a sense the contents of auto.misc
is already somewhat sufficient for that, but still annoying that there is no man page.That doesn't sound right, there might be a problem with your distribution
package.
Quoted 5 lines
No, as I say later this is incomprehension on my part ;-). Or merely the fact that needing to use "number" for accessing "sections" of the "manual" is so rare (if you are not a developer accessing library functions and/or kernel functions) that a regular user will almost NEVER use a section number in daily operation.
LOL, I got that later, yes.
Quoted 6 lines
Most configuration files are called "openvpn.conf" or something of the kind, and hence their manual page is also called that, and you don't need to do "man 5 openvpn.conf", because "man openvpn.conf" is sufficient.
Quoted 6 lines
Using numbers would be superfluous in 99% of cases, or even more than that. As opposed to perhaps requesting help on function calls, such as
mount()as opposed to/bin/mount.So as a developer you may be used to it. But a regular user is not. Anyone who does not do C programming, is not.
Yeah, maybe, but you still need to account for the case when there are two things of the same name, one configuration oriented and one system orientated, one belonging in section 5 and one belonging section 8.
In the autofs case autofs.conf(5) has it's own man page for configuration and autofs(8) for startup information and autofs(5) for mount map information.
Actually seems mostly sensible to me, since while mount maps are configuration in a sense they don't quite fit with the actual application configuration of autofs.conf(5).
Quoted 69 lines
So, my mistake, I guess.
Quoted 17 lines
Quoted 17 lines
You don't imagine that every freaking access of a directory or path component in the managed directory is going to result in network access.
You could do a "for f in
seq 1 10000000; do vdir $f; done" and have fun.(Don't know if that would still be the case for "browse" maps).
So this hidden directory, secret door thing is really the most confusing
of all.It's somewhat similar to something I might like to do, but still you don't really expect it ;-).
Quoted 8 lines
Quoted 3 lines
So the 3-4 things are:
create a man page or section for the format of the sub-maps
Already exists, needs work.
Lets call these mount maps (not the best but the convention used) as opposed to
the master map.Alright. Still confusing though.
Quoted 4 lines
Quoted 3 lines
indicate more clearly that programmatic maps in principle return a single entry
Quoted 8 lines
Maybe but isn't this in autofs(5) already close:
Executable Maps
A map can be marked as executable. A program map will be called with the key as an argument. It may return no lines of output if there's an error, or one or more lines containing a map entry (with \ quoting line breaks). The map entry corresponds to what would normally follow a map key.I guess.. it's just that you read over "a map entry" as meaning "only a single map entry" since just before it says "one or more lines". So it is something you only notice if you reread it in detail. After finding out that it doesn't work ;-).
Quoted 11 lines
Isn't this near the top of autofs(5) sufficient:
key [-options] location
key
For indirect mounts this is the part of the path name between the mount point and the path into the filesystem when it is mounted. Usually you can think about the key as a sub-directory name below the autofs managed mount point.Quoted 3 lines
For direct mounts this is the full path of each mount point. This map is always associated with the /- mount point in the master map.
Yes it is, particularly because it mentions that it is a sub-directory name.
Quoted 8 lines
Then, I begin to wonder what "usually" means. See, from the looks of it it is completely identical to sub-directory name, because:
- they key is going to be directly used as the name of the subdirectory
- programmatically, the script does not even have any say about the key, it cannot even return a key, does not determine the key. The only thing that can ever happen is that the key as given is turned into a subdirectory (when valid).
Yeah, I get what your saying and it makes sense.
But it could be a wildcard ....
Quoted 17 lines
I mean the logical abstraction between "key" and "value" may make sense from a purely abstract logical point of view ;-). But it has no bearing on actual domain-specific termininology, it is just a computer science term so to speak.
What it actually comes down to is:
- mount point for the actual share, or
- subdirectory created as mountpoint for the actual share(s).Where "value" would mean:
- path to the mount (share), or
- network path to the share that gets mounted on the subdirectory (the key).
Quoted 8 lines
the "mount" utility calls these source and target, respectively (source for the path, target for the mount point). Otherwise it calls them "device" and "mountpoint".
So although "key" also has meaning with regards to "database lookup" I don't see how it is ever not directly used as the mountpoint itself. And this is what creates the confusion because with the "browse" option it is not just going to be a "lookup key".
From automount(8)s POV it is always a lookup key that it wants to match to a mount location.
Quoted 6 lines
Further there is no difference between "subdirectory mount point" and "lookup key", where these are fundamentally separate concepts, and could even imply, in principle, that the actual directory to use is *also* retrieved by the key.
Quoted 6 lines
So conceptually the idea that "key" instantly means "subdirectory mount-point" is confusing.
In fact it is odd to begin with, from the perspective of a user, that "auto.smb" mounts the entire share list of the requested "host" or "key" in one go under the location that was accessed.
Yeah but I'm not going to get to excited about an example program map that gets installed by the package.
Quoted 7 lines
This is not the most common scenario (well, maybe if you are in some big network). This is not most common not only because it mounts /all/ the shares at once(!) but also because generally people want to have some level of control over the directory it ends up in, and what is the host for some server (hostname) might not be where you actually want to mount it.
Sure it is a bit of a hurdle but the autofs internal hosts map works that way and has been an expected automount feature for a long time.
I agree that doesn't cater for new users and I don't find creating and maintaining web presences at all pleasant, quite the opposite, so a wiki for this sort of information is probably not going to happen any time soon.
Quoted 32 lines
So I suggest that even this auto.smb is a bit of a weird entity. I have used it to generate a map list for me that I then adjusted and used as a static map file.
I know it does the same as that -net or -host thing. It's just that for a desktop user (again, you say that is not the target audience, but still) what you really want is a fixed location the user chooses (in auto.master) and then a browseable list of shares. Apparently this list needs to be static. What I envisioned before would be a parameter to a script that causes it to output a list of shares rather than the share entries themselves, or just the complete list of it.
This would give it "browse" capability. If you had that, you would need to be able to pass a parameter to the script as the key, but this is rather obscure.
What it comes down to is that this is what I would envision for myself:
Auto.master:
/cifs
/etc/auto.smbdiskstation <-- diskstation is
the keyNow the program/daemon/whatever passes diskstation as key to the script. It also passes a wildcard parameter such as "*". The script now returns a list of shares for "diskstation". Subsequently when the daemon needs to mount, it calls the script with diskstation and the requested mount point/share. Alternatively, the returned output from the first call must be complete and is saved by the daemon. It is then used in place of a static map (the script is now nothing but a static map generator).
Quoted 4 lines
Personally I prefer the former: the script has two modes: list and return entry, and it therefore has two parameters: one to identify what to list, and the second to determine what entry to return from what was listed.
I have thought about it a number of times.
It is necessary to remember that whatever is done must be backward compatible for existing users program maps.
One though was similar to your suggestion above, where diskstation would be introduced as a mount option, understood to be a program map parameter by automount(8) for certain lookup operations like the lookup module readmap entry point.
That's about the closest I've come to something that should work. Still I haven't actually implemented it.
There's still a lot of detail about this approach that is unclear though, such as what should the parameter be, how many would be needed, where to store it internally, etc.
Quoted 63 lines
Otherwise, without the parameter, you'd need to hardcode it in the script and duplicate the script for every host, etc.
In fact as a user you may not even want all shares to be present on your local system. So you might equally well decide to use a static map. So perhaps you will use "smb.auto list" or "smb.auto list diskstation" or "smb.auto diskstation *" manually. And perhaps it generates all of the mappings for you that you can then prune and save.
Perhaps, in absense of the key (the entry, or wildcard) a call such as "smb.auto diskstation" should simply generate the share list, whereas "smb.auto diskstation media" generates the key for only that entry.
So you see me use a term like "entry" to determine what you today call a "key" and the word "key" to determine what you today call a "entry" ;-).
"entry" means "element of pre-existing list", "key" means "value to use for a lookup operation in a database".
"key" can also mean "that what you need to mount or access something".
so if you combine the two, you have like three things:
- subdirectory (mount point)
- key to use for database lookup
- entry in the list of sharesbut you can only solve it if you differentiate between "lookup operation" and "element of a list".
a lookup operation may /return/ a list, which is why I said:
"diskstation" (hostname) is a key
list of shares is the result of that lookup operation based on that key entry is an element of that list you can mount.This applies to a form of browse scenario. If you have invisible mounpoint, you get:
"diskstation" (hostname) is a key that is going to be a direct subdirectory
the list of shares it returns is going to be a subdirectory list to that first subdirectory
this will contain the actual shares, so you get top/key/entries.And you see we basically have the same thing except that this is one level deeper in our hierarchy.
In my desired use case, you get:
top/entry
But in the current implementation of e.g. smb.auto, you get:
top/key/entry
So it is clear diskstation (hostname) IS a key, except that in the current case or scenario this key is ALSO used as the direct descendent of the toplevel directory used for the configuration in auto.master.
Quoted 7 lines
Which gives rise to the idea that maybe "key" and "subdirectory mount point" should not be equivalent or identical. Not necessarily identical.
In fact, smb.auto DOES produce a browsable list. It is just /beneath/ the subdirectory for its "key" (the hostname) and mounts ALL of the shares at once.
Actually it doesn't actually mount the shares.
It mounts autofs trigger mounts that will cause the actual share to be mounted on access.
Quoted 29 lines
So you now already have that mountstorm (it just gets mounted by default) it's just a level deeper.
You now don't have a mountstorm on the level of hosts, but you have a mountstorm on the level of shares (per host) but it is still the same thing.
So the design specifies that keys must become mountpoints in the "sub-top-level" directory (the directory managed by autofs for indirect mounts).
Whereas perhaps conceptually you would need to differentiate between key, and an entry used for mounting, and the difference is that this entry used for mounting can be an element of the list returned by the key.
So you get key -> list -> entries. In the case of current auto.smb, what happens is:
key -> (subdirectory) -> list (itself an entry) -> entries.
Or in other words:
key -> list -> entries, but mounted beneath top/key, as key is used as an extra directory in between.
Current auto.smb:
Quoted 8 lines
<managed mount point> / <key> / entries / contents of shares
My preference:
<managed mount point> / entries / contents of shares
And why? Only because I want to specify mount points for specific hosts manually.
auto.smb is an example program map.
Your free to implement your own program map any way you wish.
The thing that gets passed to the program map is a string that represents the trailing path component of a path the kernel believes needs something done to it, be it symlink it somewhere, bind mount it somewhere or lookup a location and mount that on it etc.
You must adhere to that restriction because that's how automounts are triggered.
Quoted 37 lines
So the question becomes how you can combine such things.
The requirement becomes for the script to return the mount point. A static list does return mount points. A script does not, the mount point is specified as the "key" given to it. So if you want the script to determine the mount point the way a static list does, you need it to return, in essence, a static list.
Even if your static list is not browseable, so too can a returned "dynamic list" also be not browseable. But at this point your script can return a mount point of its own. That is again problematic if the key is derived from filesystem access.
So you see how hard this is to reconcile.
If we insist in accessing hidden shares and deriving keys from user action or filesystem action, or filesystem access, then keys are predetermined the moment the system is actually called. The script or database can only return whether something is gonna be there or not.
Then the only way to reconcile it is (in my case) for the key to be not the only argument to the script, but I give the script my argument I define in auto.master:
/cifs
/etc/auto.smbdiskstationNow my script knows that e.g. $1 is gonna be the filesystem requested path, and $2 is gonna be the custom parameter I determined in advance.
I don't need to use the key by itself ($1) to do the lookup. My real key is $2. But I will use the second key ($1) to return a share for the host determined by the first key ($2).
Quoted 7 lines
In essence what happens with auto.smb is that the key ($1) is used as the host (first key) and smb.auto will itself derive and load the list of shares, and then instantly mount that (it retrieves all of the second keys and instantly uses that and mounts everything).
autofs allows for this second list of keys to be instantly used and utilized.
One other consideration is the lazy mounting (and umounting) of these multi -level mount trees that autofs manages.
It's very easy to get that wrong and end up with an inoperable mount point that cannot be recovered.
At the moment automount(8) will do that for you and occasionally gets confused itself.
To do what is wanted here it would be necessary have a way to tell automount(8) leave the mountpoint alone.
Quoted 58 lines
But it's not the common use case because you are now fixed to that location the user accesses and of course you could use that location (the key) ($1) to look up a host in a different way, that is perfectly possible. If your script marries "diskstation" to "ds" or "ds" to "diskstation" then you could automount /cifs/ds and it would request shares from host "diskstation".
But you're still stuck with that intermediate level, that extra level of subdirectories.
It's not possible currently to use the "first key" to determine the host and then get the "second keys" ($1) to determine the entries. Well, actually it is, but not in a browseable fashion:
/cifs
/etc/auto.smb-Dhost=diskstationI believe this could work. You pass your real "primary" key as that variable, and your script returns shares based on that key, but with the "filesystem key" ($1) as the name of the share to return.
Still, rather convoluted at this point, and not possible to browse this list.
Because it doesn't return a list. Using $2 to pass that variable wouldn't really be different.
But the script is only called when an entry is accessed (through the filesystem) and as such is never called without a secondary key.
The current auto.smb doesn't have secondary keys, it loads and mounts all of them instantly and automatically, and there is no "automount" for the secondary level.
Well, there is, it just mounts all of it in one go.
So how do you skip the intermediate level?
- call the script on purpose to generate a 'static list' instead of just for entries alone.
- use the static list to precreate directories (mount points), or
- use the static list to return readdir information, or to filter requests for directories, or don't use the static list at all and just keep it the same.
So we also see here that the static list *could* be used for the thing you described above, that alternative readdir implementation.
Although it's a bit of a hack, I'm sure.
You could specify some interval for invalidating the list. At which point it is renewed by another "get list" call to the script.
Quoted 8 lines
So in essence I am really advocating "return list" functionality for scripts here.
And there is no easy way to implement that while staying compatible with current scripts because you would need to: call the script with an empty parameter (that will never return an entry) or with a wildcard (that will probably never return an entry) and in essence, that might just as well mean, that if you want to take this approach:
Umm .. I think the other possibility is that if you pass an empty string to an old program map you might cause it to crash.
Quoted 209 lines
- use the wildcard as the "list" operation, scripts that do not know about the wildcard will simply return zero entries.
- use the extra parameter as $2, in other words: allow for additional parameters in auto.master in that form of:/cifs
/etc/auto.smbdiskstation randomBut there are already recognised options. So what we have today (-D) I guess already works, it is just a bit obscure, and it seems to be the equivalent of what you would do.
Then an alternative auto.smb (I called it auto-short.smb) would:
- be called as:
/cifs
/etc/auto.smb-Dhost=diskstation- return a list of shares when called with "host=diskstation
/etc/auto.smb*"
- alternatively you could have it be called with: "/etc/auto.smb* diskstation"- make sure the shell doesn't expand that * ;-).
- autofs calls this list operation each time auto.master is reread.
- autofs calls this list operation each time some invalidation timeout occurs
- not really any other way to trigger it.
- the static list being returned (really a dynamic list now) can be used for readdir, or even access filtering (it contains all the valid keys).
Now when you have something like this, you can:
- call the filesystem component you use to access the actual shares, the "entry" or simply "share" or "directory". But "key" becomes more comprehensible now as well. "key" now corresponds to "entries in the returned list". Each "entry in the returned list" is now a "mapping between key (directory) and value (device-path). I realize "mount point" might confuse with the mountpoint defined in the master file.
"submount" would be ugly. "subdir" is most convenient. "subkey" is illustrative. "entry" describes the entire thing, not just the key. But in essence we are just talking about a mount-point here.
Nevertheless in principle the "first key" given should not even need to be the mount point in fact. It is a key to a lookup-table.
If you wanted to do it right I think you would need to do away with hidden (or principally unknown) mount points, and basically always work with static lists (or dynamically returned lists) and/or give every script a first parameter $1 that is a key that you actually use for it.
Then the second paramter $2 would be the key that the filesystem has actually given you for accessing a path.
In this way "access path" becomes secondary.
Perhaps a script in this way still cannot control the mount point (name) but at least now it provides it in advance. Of course there are situations in which you cannot know it in advance or in which it is irrelevant to know it in advance.
You can imagine a list to be much too big to actually retrieve it. In itself a sign of a bad system, I think. If it was that big, you would need hierarchy to make it smaller.
So I don't know if there is ever any real need for obscurity, and for hosts to be accessed while not being able to list them in advance.
I conjecture that this use case in itself would be odd, that you create auto-mounts for an unspecified number of hosts you don't really know about in advance.
It makes for zero-configuration, true. You don't have to keep a list anywhere or even arrange for it to be generated.
On the other hand if a list could not be had, the script would just return nothing and the daemon will assume no-browse no-information-in-advance lookups the way it does today.
Basically if you keep the current $1 and use it to pass a wildcard, no script is likely to ever break on that (it will just return nothing). Then $2 can be your primary key. Not every script needs it.
But complete zero-configuration is to me an odd thing: a magical directory that will mount every host given to it including all of its shares, does not seem sensible to me in the sense that you would at /least/ want some configuration for every system right?
One size fits all: the same system for every computer, just place it somewhere in your filesystem and you have access to everything and everything.
No, normally you would want to tailor it because you do not have access to an infinite amount of hosts and the /cifs/host/share structure therefore makes not much sense.
/device/share may make more sense, perhaps added with /hosts/hosts-meeting-a-certain-pattern/share
You already have a pretty big system if you use the latter. Anyway.
I think I should conclude this :p.
What seems sensible to me at least is to add something that doesn't break compability by using a wildcard * for scripts that will return a certain format of list (or just all shares it can find in the regular format) with a $2 parameter designating something such as a "host" or perhaps even "department".
In this way you solve the issue that the current auto.smb requires the "hostname" to be first key given to it because there is no other way to pass a key. Then the actual keys can be passed as "second parameter" in this sense that they still become $1, but at least now you can specify your host to it in advance instead of guessing it on the wind ;-).
So if I have to say anything I would say that I recommend this structure:
- $1 remains the filesystem key
- $2 becomes the additional key a user may specify in auto.master
- $1 as * will designate the request for a list lookup
- the list can in the future be used for readdir implementation, and currently for precreated mountpoints in case "browse" is used.Quoted 17 lines
Quoted 3 lines
indicate that this is true for "no browse" on "static maps" as well, and
that directories are only created when they are accessed.That seems to be missing from the man page.
Quoted 1 line
And that no-browse is the configured default.
Not sure about that either.
This (from
auto.master(5)) says that I have made "browse" the internal program
default way back when version 5 was first developed but I install a configuration that turns it off because of (because of the above problems, which
are not spelled out in the man page):Quoted 12 lines
[no]browse
This is an autofs specific option that is a pseudo mount option and so is given without a leading dash. Use of the browse option pre-creates mount point directories for indirect mount maps so the map keys can be seen in a directory listing without being mounted. Use of this option can cause performance problem if the indirect map is large so it should be used with caution. The internal program default is to enable browse mode for indirect mounts but the default installed configuration overrides this by settingBROWSE_MODEto "no" because of the potential performance problem.I know, but if you, as a user, do not understand what the hell it is doing in the first place, and you have no time for extensive manual reading in advance (like me) you will never get to that option because you still don't understand the basics, and the more advanced stuff (like that) that will be too hard to understand.
That information is too difficult to comprehend (or digest) when you are someone who still doesn't understand how it works.
Quoted 9 lines
Quoted 7 lines
I think that limitation indicates why using relative paths is so much more pleasant. A map file using relative files can be "moved around". A
map file using absolute paths cannot.Could you not make a direct map out of the key of the master file, and the keys of the sub-maps?
Can't really do that.
It's not that I saw it as necessary, I just still think the difference and distinction between "direct" and "indirect" is rather I dunno, weird I guess ;-).
Quoted 11 lines
The closest thing to that would be multi-mount map entries, again from autofs(5):
Multiple Mounts
A multi-mount map can be used to name multiple filesystems to mount. It takes the form:key [ -options ] \
[[/] location \
[/relative-mount-point [ -options ] location...]...Quoted 5 lines
This may extend over multiple lines, quoting the line-breaks
with
`\´. If present, the per-mountpoint mount-options are appended to the default mount-options. This behaviour may be overridden by theappend_optionsconfiguration setting.
Quoted 5 lines
Something I am already using (almost) and I believe this is what auto.smb uses by default too. It's just that I use it to mount parts of the same share. (Samba smbd often has issues with file-system boundaries so I create a share out of each filesystem and share them individually, then mount them as above).
It does, yes.
Quoted 75 lines
Quoted 36 lines
Quoted 11 lines
normally ( /bla/bla ) indicates an absolute path and can only be used in
a direct map, from what I've seen. You *could* cause it to turn this "indirect map" into a direct one.ie. ( bla/bla ) == indirect
( /bla/bla ) == direct.In the Sun map format maps indirect mount map keys are a single path component
only.The situation is different if using amd format maps but I could work out how
direct mount maps could be usefully used so that mount type has not been
implemented. Even on Solaris where they are supposed to be fully supported I
could make sense of them.Quoted 6 lines
/start/of/the/- ( path )
Then any path in the master file ending with /- would be a direct path.
And would cause the (static) map file to be treated as direct.
Nope, that is unlikely to work and would lead to a lot of problems.
Quoted 7 lines
One example might be with:
/start/of/the/-/path
people will ask where have the directories below /start/of/the/ gone and the
answer will be they have been covered by the autofs file system mount.But that's no different from a regular mount on that location.
Quoted 4 lines
Next it would be but I should be able to see these, surely that's easy to fix.
At hat point I would probably leave the conversation.
:-). Haha. But come on, you can't take that seriously, the same thing happens with a regular mount, even a bind mount does that. It's absolutely not possible to reveal the contents of some subtree when another filesystem had mounted on top of it, except by remounting the entire subtree elsewhere.
(I am trying out some gimmick that requires me to mount the root device under something like
/run/mount/rootfsclone, because I am bind mounting on top of something that I still want to be accessible. The only way to make it accessible is to mount the entire rootfs somewhere else. I basically need a persistent location for individual devices without anything mounted on top of them; ie. a pure one-file-system mount tree.Then you have those individual mount-trees (filesystem trees) and THEN you have the complete FHS that you normally use.
That seems like the only sensible way to approach a lot of issues. Having a /mounts/ with every (custom) mounted device present there.
Recently the number of incomprensible mounts reported by "mount" has grown so big that the entire tool has become unusable, mostly due to cgroups. And tmpfs's, and debugfs's, and what not.
Quoted 7 lines
I now find I need to create a wrapper for mount so that "mount" stays at least usable to the user. Such that every subdirectory of /sys, and every subdirectory of /run, and every subdirectory of
/dev/, is actually hidden from display. And from /proc as well. And the same applies to DF.(I really need to find a way to better save and redistribute these modifications).
Oddly enough, a pseudo mount option like "hidden", visible in the /proc files but left out by utilities like mount, df etc. is what could be done, and should be done, IMHO, but no-one wants to do it!
Quoted 38 lines
Quoted 24 lines
In any case a similar thing, but more explicit and I think easier to follow and
maintain are the autofs multi-mount map entries, what I think you called
relative path above.It isn't clear but that syntax should function within both direct and indirect
mount maps.The problem with all this is that substitution cannot be used in the key.
The reason for that is same as why the amd regex key matching has not been
implemented.Quoted 7 lines
A linear map scan would be needed for every lookup in order to match what has
been sent to us from the kernel. And as much as you might feel that wouldn't be
a problem because maps are generally small I can assure you it would make life
really unpleasant for me!:). You have lost of enemies; if you don't do things right :P. LOL.
I am not thinking about this now, but I guess you mean that my proposed solution of saving those static lists (or dynamic lists) would actually help this :p.
Quoted 5 lines
Actually what you are saying is that even doing lookups on those lists I proposed would be quite annoying.
However currently this lookup is already done on the server (if it is some database) or in the static map (if they are not browseable). Right?
And in the locally cached map and quite frequently queried as part of the expire processing.
Direct maps and maps that have the browse option must be read completely at startup and are always present in local storage.
Quoted 21 lines
Databases are generally better at lookup than small-scale applications. Now you need hash-tables etc. Anyway, I can't think about it right now.
Quoted 9 lines
Quoted 3 lines
I only noticed just now because autofs(8) references autofs(5), and I'm
like ....huh?Yeah, that's what we have ....
Quoted 5 lines
Quoted 1 line
/media/*/nas
/etc/nas-shares.mapCan't do that, each /media/*/nas needs to be mounted at program start so that
automounts can be done?
Quoted 7 lines
Actually the shell easily expands such a pattern to all matching trees?
e.g. if you create
/usr/local/doc, and then do"ls
/usr/*/doc/" you will get the output of:
/usr/local/doc/usr/share/doc
automount is not a shell and doesn't have code to expand such things and an indirect mount may have a single map associated with the given mount point path.
Way back it was possible to associate multiple maps with a given mount point path but that functionality was removed because it was not a feature of the Sun automount system which autofs seeks to implement.
Quoted 6 lines
So with that output, you mount all of them, all the while knowing that the middle component (here) is a wildcard. Then the daemon knows about this wildcard and can use it while processing
/etc/nas-shares.mapfor each of them; but they are all individual mounts at that point.
Quoted 8 lines
e.g. /media/me/nas and /media/pamela/nas are now individual autofs mounts.
but their "
/etc/nas-shares.map" is called with a different wildcard.I mean that either /media/me/nas will already exist (precreated by the user) or your program creates it (I think it already does that).
This is so far from what autofs is meant to implement I can't see it not interfering with what it is supposed to be doing.
Quoted 306 lines
Actually it cannot create it DUH, so the user will need to precreate them.
I know /media is usually seen as a managed directory but this is annoying in the first place. Such an important name (/media) and I cannot do anything with it because it is already taken. At the same time accessing it from your home directory is usually quite annoying. So in order to make sense of it I must precreate /media/xen and /media/dannielle and prevent the daemon (that /media thing) from wiping them.... buh.
The funny thing is that /media/xen is not even owned by xen.
The subfolders are, but the thing itself isn't (how stupid?).
So in this case perhaps not as convenient, but in the general case if I precreate:
/mytree/me/nas
/mytree/quinty/nasetc.
Then the daemon could read
/mytree/*/nas
on program start and see all the shares that match. All the directories that match.
Then it knows that "me" and "quinty" are wildcard entries for their respective mounts.
Quoted 14 lines
Quoted 9 lines
Quoted 8 lines
By definition a program map can't know what keys may be passed to it. And, yes, even autofs(5) doesn't say explicitly that a program map doesn't know
what keys may be passed to it, perhaps that should be ammended.Quoted 4 lines
In principle even a program map could read all of the available keys from whatever database it consults, and return all of them; or return something only (as it does now, perhaps) when a given key matches (is found).
Quoted 5 lines
I've thought of that too.
But I've resisted because I'm concerned about confusing existing program maps
people have in unpleasant ways.At some point you need to be willing to break previous compability.
I understand that Linux is usually a running-upgrade system and even if people do upgrade entire distributions they want the system to keep functioning as normal without changing anything.
But e.g. the Debian update program can tell you stuff like "This package does something different now, and you need to ensure it can either handle or ignore the wildcard".
on every package installation.
Quoted 3 lines
That's not a bad thought, but what would $1 be so that older scripts wouldn't
choke on it.I had suggested * because there is no hostname called *.
But scripts may very well mount local filesystems as well. And they might very well use a shell to do stuff like "[ -d $key ]" or "[ -d "$key" ]" and....
And as always, stuff breaks a lot when people don't use "$key". If people's scripts are badly written then [ -d $key ] will malfunction.
In order to cover that you would have to use something like "_", no one is going to use _ as an actual filename.
Quoted 13 lines
Quoted 12 lines
So, for instance, if $2 is "browse" the script may choose to return a browse map, or list of shares, with whatever options are available (it could even return the UID/USER of the share given options, so that the created directory would have the same UID as the to be mounted share!) -- meaning it might simply return ALL entries it could possibly return in the default format, which is the meaning of the wildcard, and the program (autofs daemon or module(?)) would be able to generate the appropriate directory entries based on that. So in that case you're probably going to be rid of the problem of the non-matching state of pre- and post-mount of directories.
Quoted 3 lines
which pretty much defeats the only possible solution I can think of for "browse"
option mounts ...You mean the one suggested above. But if you make an alternative readdir having a list available won't bug you right?
I mean, having mount information available in advance cannot be a problem, only a help.
Quoted 6 lines
Like I said, I wanted to have the default as "browse" but got hit with too many
complaints and was only willing to change the package installed default configuration to satisfy them.I'm not going to try that again.
"Design by committee" they call it ;-).
Someone told me that OpenSUSE has had "browse" as the default for at least 5 years.
You did think about making a distinction between static and LDAP/NIS maps? I'm not sure I am not crossing my bounds here now ;-). Seeing that I already suggested it.
Quoted 17 lines
Quoted 2 lines
In this way no one can ever get confused, so you might just rename autofs(5) to
autofs.map(5).Yeah maybe ...
Quoted 10 lines
I will look into providing a patch, but I hope you can rename autofs(5)
toautofs.map(5)before doing anything like that. I don't feel like improving a system I know myself haven't had access to, in that sense. I
don't mean you have to do that straight away, but if that is a possible
direction to go, or change to make, then I could supply a patch or anything based on the system that would result from that, seeing that autofs(5) is already pretty good in that sense.Quoted 2 lines
At some point I'll need to return to this and make a list. Unfortunately that time isn't now.
Here is just my short summarize:
- suggested compromise between browse and nobrowse by only making browse for static lists the default to please new people unaware of what will happen
- suggested perhaps using _ as the wildcard key, not using that "browse" flag as $2, but allowing a different "primary" key as $2 if the script wants it.
- suggested changing autofs(5) to
autofs.map(5)such that users will be able to find it.And the rest is up to you, I think you don't need anyone rewriting it for you now, but I could still try my hand if that was necessary. I just think using better terminology is dependent on the first two suggestions here, particularly perhaps the second, and otherwise you'll have to explain current terminology better and create some way to separate "key" in auto.master from "key" in "autofs.map" ;-).
However if scripts could accept an additional "key" and the default operation or capability would for them to be able to return a "list" then it would become more clear that the additional "key" would be the index to retrieve that list, and that the entries resulting from that may have their own "keys" but these "keys" are then used for the individual "mount points".
So then you have reason to call the entry keys "entry keys" and the other ones "lookup keys" for instance. Then, static maps use "entry keys" but scripts have "lookup keys" as their 2nd parameter. The script then has "entry key" as the first, and "lookup key" as the second parameter. Then, entry key is equivalent to subdirectory, and lookup key is often going to be equivalent to e.g. hostname.
A script without list functionality only accepts "entry keys" but may very well use them as lookup keys as well.
A database also uses them as equivalent. But at least now you have something of a model to conceptually separate them. Since browse already functions for e.g. LDAP, in many cases you will not use a lookup key for that, simply because (a)
/etc/autofs_ldap_auth.confalready contains such paramters and (b) the other parameters are specified in the mapname-format for LDAP.So what we call "lookup key" would already be encoded in the LDAP URL but such a thing is not possible for scripts.
So you could call scripts the exception to the rule in not having this. Of course it exists,: the "-D" parameters.
(I think? Haven't tested it yet).
So I don't know.
For a database the entry key is always going to be a lookup key to the database.
Entry keys can also result from a list fetch.So they can result from filesystem access or from some list retrieval.
But if "browse" becomes more normal and standard, you would see that it becomes easier to talk about this "entries from the list you get when quering the database/script for it".
Then the gap between "entries from the list" and "subdirectories created" is smaller than the gap between "filesystem access" and "subdirectories created".
Meaning: key (database designator) -> list -> entries (database entries) is conceptually easier to grasp than key (filesystem access) -> entry (database entry).
Or key (filesystem access) -> database lookup -> entry.
Simply because it is hard to understand that user action (filesystem lookup) is the input to the entire system and is actually the keys being used.
Even if it's not always to be used, having a level in between that ties "key" to "pre-existing shares" or "pre-existing entries" becomes easier. When a user thinks of "key" they think about either existing shares or existing hosts.
Then conceptually it is logical to assume that the script or database can give back this list. And that either this list is precreated on the harddrive, or at least that it is used to match the "keys" against, for instance.
(By harddrive I mean filesystem (hierarchy)).
This is why I think standard operation for at least smaller lists should be a form of browse or at least a form of list retrieval, if you are going to have dynamically generated lists in any case.
Then, it conforms more to a sense of what a general computer user is going to have of the idea of auto-mounting mount-points. It (he/she) thinks those mount-points are already going to exist, prior to being automounted.
This in turn makes it easier to explain that the key is really a subdirectory, and that there can be another key if needed to specify the host, or something of the kind.
Then, the most common use case for a new or general computer user wanting to try this is going to be met, because that use case is going to involve auto-mounting network-shares of which there is a limited number.
And it will involve something like
/nas/media
/nas/backup
/nas/homeetc.
Now I'm not sure if it covers the entire use. But at least you'd also have a script that can generate a static map for you that you can then use in place of the scripts, if you want to prune the mounts that are actually visible to you.
Apart from the distributions themselves there is not really any software in the Linux world that is tailored to either the desktop user, or the server user, but not both, is there?
Apart from the desktop environments.
Anyway this was a long read and write, I guess, for both of us :P.
Time to get some food again or at least some water :p.
See ya.
Quoted 2 lines
--
To unsubscribe from this list: send the line "unsubscribe autofs" in
--
To unsubscribe from this list: send the line "unsubscribe autofs" in
Quoted 5 lines · Ian Kent
What is needed is a re-write of the user space <-> kernel space communication essentially to add readdir functionality.
So that listing a directory would show the possible map mount points without the need to create the mount point directories.
Oh my god.. we need this!
We have huge direct maps here and with the changing of /etc/mtab to a link to /proc/self/mounts things have gotten horrible. We've worked with suse to help speed up the start of autofs (it was taking 6+ hours to simply start the first time due to how automount reads the mtab file, stats paths, does the mkdir, etc). But things like mkinitrd running with our huge maps loaded can take 20-30 min due to all the parsing different steps do for various filesystem checks. I've learned to stop automount and do a full umount -t autofs -a before running something like a zypper update with a kernel upgrade.
Signature · 7 lines
Mike Marion-Unix SysAdmin/Sr. Staff IT Engineer-http://www.qualcomm.com Brian: "So enlighten us, what's your idea of the perfect man?" Helen: "Oh, I don't know... Solid, sensitive and dependable, but with a sense of whimsy, a love of music and children. A sense of commitment but a respect for my independence; adventurous yet a homebody... I really hadn't thought about it that much." Joe: "Sounds great Helen, but Kermit is a frog." => Wings-- To unsubscribe from this list: send the line "unsubscribe autofs" in