Skip to contentSkip to conversation

Export especific file extension

NFS Fábio Jr. 3 participants 9 messages Jul 27, 2009 – Jul 28, 2009

Subsystems (2)
lore
Share & tools
lore

Fábio Jr.

author
#0 lore raw

Hello all.

Is there a way to export only some specific file extensions from a=20 directory, like exporting only the files in /home/fabio with the .jpg=20 extension?

[]s

=46=E1bio Jr.

-----------------------------------------------------------------------=
-------
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

#1 lore raw
Quoted 4 lines · =46=E1bio Jr. wrote:

Hello all.

Is there a way to export only some specific file extensions from a=20 directory, like exporting only the files in /home/fabio with the .jpg=

=20

Quoted 1 line

extension?

You could do this pretty easily by exporting a 'fuse' filesystem layere= d=20
on top of your home directory.

Rob Gardner
HP

-----------------------------------------------------------------------=
-------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30= -Day=20
trial. Simplify your report design, integration and deployment - and fo= cus on=20
what you do best, core application coding. Discover what's new with=20 Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

author
#2 lore raw

Rob Gardner escreveu:

Quoted 5 lines
Quoted 4 lines · F=E1bio Jr. wrote:

Hello all.

Is there a way to export only some specific file extensions from a=20 directory, like exporting only the files in /home/fabio with the .jp=

g=20

Quoted 7 lines
Quoted 1 line

extension?

You could do this pretty easily by exporting a 'fuse' filesystem=20 layered on top of your home directory.

Rob Gardner
HP

Thanks Rob for the reply,

This means that if I use this FUSE, I can export via NFS only the file=20 extension that I need? let me explain my problem, not really a problem,= =20
but a doubt.

I have one storage that have my aplication files. This storage is=20 mounted via NFS in my 3 aplication servers. Now I will put another=20 server, to serve only static files (jpg,png,css,js basically), and=20 though that maybe there is a way to export only these files from the=20 storage. The problem is that these files are not organized in diferent=20 folders.

Maybe my first question doesn't explained my real necessity, but your=20 answer will make me search a little more about FUSE.

Thanks.

[]s

=46=E1bio Jr.

-----------------------------------------------------------------------=
-------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30= -Day=20
trial. Simplify your report design, integration and deployment - and fo= cus on=20
what you do best, core application coding. Discover what's new with=20 Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

#3 lore raw
Quoted 8 lines · =46=E1bio Jr. wrote:

Rob Gardner escreveu:
=20

Quoted 6 lines

F=E1bio Jr. wrote:

   =20
Quoted 4 lines

Hello all.

Is there a way to export only some specific file extensions from a=20 directory, like exporting only the files in /home/fabio with the .j=

pg=20

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

extension?

     =20

You could do this pretty easily by exporting a 'fuse' filesystem=20 layered on top of your home directory.

Quoted 7 lines
Quoted 4 lines

Rob Gardner
HP

   =20

Thanks Rob for the reply,

This means that if I use this FUSE, I can export via NFS only the fil=

e=20

Quoted 1 line

extension that I need? let me explain my problem, not really a proble=

m,=20

Quoted 7 lines

but a doubt.

I have one storage that have my aplication files. This storage is=20 mounted via NFS in my 3 aplication servers. Now I will put another=20 server, to serve only static files (jpg,png,css,js basically), and=20 though that maybe there is a way to export only these files from the=20 storage. The problem is that these files are not organized in diferen=

t=20

Quoted 3 lines

folders.

Maybe my first question doesn't explained my real necessity, but your=

=20

Quoted 1 line

answer will make me search a little more about FUSE.

I think your first question explained your need clearly. FUSE is a=20 filesystem layer that lets you filter requests through a user program=20 such as a python script. So, for instance, every time a process opens a= =20
file, a function is called in your script, and you can look at the name= =20
of the file being opened, and there decide whether or not to allow the=20 file to be opened. I think this would solve your problem easily. Though= =20
all the storage is "exported", only files with certain names (ie, *.jpg= ,=20
etc) could be opened. You could also decide which files get enumerated=20 with readdir, etc.

Rob Gardner
HP

-----------------------------------------------------------------------=
-------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30= -Day=20
trial. Simplify your report design, integration and deployment - and fo= cus on=20
what you do best, core application coding. Discover what's new with=20 Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

author
#4 lore raw

Rob Gardner escreveu:

Quoted 8 lines
Quoted 7 lines · F=E1bio Jr. wrote:

Rob Gardner escreveu:
=20

Quoted 5 lines

F=E1bio Jr. wrote:
=20

Quoted 3 lines

Hello all.

Is there a way to export only some specific file extensions from a=

=20

Quoted 12 lines
Quoted 12 lines
Quoted 10 lines
Quoted 3 lines

directory, like exporting only the files in /home/fabio with the=20 .jpg extension?

     =20

You could do this pretty easily by exporting a 'fuse' filesystem=20 layered on top of your home directory.

Rob Gardner
HP

   =20

Thanks Rob for the reply,

Quoted 8 lines
Quoted 8 lines

This means that if I use this FUSE, I can export via NFS only the=20 file extension that I need? let me explain my problem, not really a=20 problem, but a doubt.

I have one storage that have my aplication files. This storage is=20 mounted via NFS in my 3 aplication servers. Now I will put another=20 server, to serve only static files (jpg,png,css,js basically), and=20 though that maybe there is a way to export only these files from the=

=20

Quoted 4 lines
Quoted 4 lines

storage. The problem is that these files are not organized in=20 diferent folders.

Maybe my first question doesn't explained my real necessity, but you=

r=20

Quoted 4 lines
Quoted 1 line

answer will make me search a little more about FUSE.

I think your first question explained your need clearly. FUSE is a=20 filesystem layer that lets you filter requests through a user program=

=20

Quoted 1 line

such as a python script. So, for instance, every time a process opens=

=20

Quoted 4 lines

a file, a function is called in your script, and you can look at the=20 name of the file being opened, and there decide whether or not to=20 allow the file to be opened. I think this would solve your problem=20 easily. Though all the storage is "exported", only files with certain=

=20

Quoted 7 lines

names (ie, *.jpg, etc) could be opened. You could also decide which=20 files get enumerated with readdir, etc.

Rob Gardner
HP

Oh yes, I think I didn't understand you answer.. but now all became=20 clear in my mind. I'm afraid that using this solution maybe cause an=20 increase on storage processor load, because for every request the scrip= t=20
must be executed. I already have some issues with server availability,=20 and perhaps the solution of a problem becomes the worsening of another.

Still thanks for the reply and thanks for helping to clear my mind.

[]s

=46=E1bio Jr.

-----------------------------------------------------------------------=
-------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30= -Day=20
trial. Simplify your report design, integration and deployment - and fo= cus on=20
what you do best, core application coding. Discover what's new with=20 Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

#5 lore raw
Quoted 13 lines · =46=E1bio Jr. wrote:

Rob Gardner escreveu:
=20

Quoted 11 lines

F=E1bio Jr. wrote:

   =20
Quoted 9 lines

Rob Gardner escreveu:
=20

     =20
Quoted 6 lines

F=E1bio Jr. wrote:
=20

       =20
Quoted 3 lines

Hello all.

Is there a way to export only some specific file extensions from =

a=20

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines

directory, like exporting only the files in /home/fabio with the=20 .jpg extension?

     =20
         =20

You could do this pretty easily by exporting a 'fuse' filesystem=20 layered on top of your home directory.

Rob Gardner
HP

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
   =20
       =20

Thanks Rob for the reply,

This means that if I use this FUSE, I can export via NFS only the=20 file extension that I need? let me explain my problem, not really a=

=20

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

problem, but a doubt.

I have one storage that have my aplication files. This storage is=20 mounted via NFS in my 3 aplication servers. Now I will put another=20 server, to serve only static files (jpg,png,css,js basically), and=20 though that maybe there is a way to export only these files from th=

e=20

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

storage. The problem is that these files are not organized in=20 diferent folders.

Maybe my first question doesn't explained my real necessity, but yo=

ur=20

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

answer will make me search a little more about FUSE.

     =20

I think your first question explained your need clearly. FUSE is a=20 filesystem layer that lets you filter requests through a user progra=

m=20

Quoted 1 line
Quoted 1 line

such as a python script. So, for instance, every time a process open=

s=20

Quoted 1 line
Quoted 1 line

a file, a function is called in your script, and you can look at the=

=20

Quoted 3 lines
Quoted 3 lines

name of the file being opened, and there decide whether or not to=20 allow the file to be opened. I think this would solve your problem=20 easily. Though all the storage is "exported", only files with certai=

n=20

Quoted 11 lines
Quoted 8 lines

names (ie, *.jpg, etc) could be opened. You could also decide which=20 files get enumerated with readdir, etc.

Rob Gardner
HP

   =20

Oh yes, I think I didn't understand you answer.. but now all became=20 clear in my mind. I'm afraid that using this solution maybe cause an=20 increase on storage processor load, because for every request the scr=

ipt=20

Quoted 1 line

must be executed. I already have some issues with server availability=

,=20

Quoted 1 line

and perhaps the solution of a problem becomes the worsening of anothe=

r.

Quoted 2 lines

Still thanks for the reply and thanks for helping to clear my mind.

It's a valid concern, but fuse does not "execute a script" for every=20 operation. The script is always running, sort of like a server for the=20 pseudo-filesystem. Each requests causes a few lines in the script to be= =20
executed. There is no process creation and dispatch for each operation,= =20
only a process wakeup for each operation. It is impressively lightweigh= t=20
and it's worth trying before dismissing it as a resource drain.

But anyway, here's another idea for you-- Export a new directory that=20 contains links to only those the files you want to export.

Rob Gardner
HP

-----------------------------------------------------------------------=
-------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30= -Day=20
trial. Simplify your report design, integration and deployment - and fo= cus on=20
what you do best, core application coding. Discover what's new with=20 Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

author
#6 lore raw

Rob Gardner escreveu:

Quoted 5 lines
Quoted 4 lines · F=E1bio Jr. wrote:

Rob Gardner escreveu:
=20

Quoted 2 lines

F=E1bio Jr. wrote:
=20

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Rob Gardner escreveu:
=20

    =20
Quoted 5 lines

F=E1bio Jr. wrote:

         =20
Quoted 3 lines

Hello all.

Is there a way to export only some specific file extensions from=

=20

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

a directory, like exporting only the files in /home/fabio with=20 the .jpg extension?

               =20

You could do this pretty easily by exporting a 'fuse' filesystem=20 layered on top of your home directory.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

Rob Gardner
HP

           =20

Thanks Rob for the reply,

This means that if I use this FUSE, I can export via NFS only the=20 file extension that I need? let me explain my problem, not really =

a=20

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

problem, but a doubt.

I have one storage that have my aplication files. This storage is=20 mounted via NFS in my 3 aplication servers. Now I will put another=

=20

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

server, to serve only static files (jpg,png,css,js basically), and=

=20

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

though that maybe there is a way to export only these files from=20 the storage. The problem is that these files are not organized in=20 diferent folders.

Maybe my first question doesn't explained my real necessity, but=20 your answer will make me search a little more about FUSE.

     =20
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I think your first question explained your need clearly. FUSE is a=20 filesystem layer that lets you filter requests through a user=20 program such as a python script. So, for instance, every time a=20 process opens a file, a function is called in your script, and you=20 can look at the name of the file being opened, and there decide=20 whether or not to allow the file to be opened. I think this would=20 solve your problem easily. Though all the storage is "exported",=20 only files with certain names (ie, *.jpg, etc) could be opened. You=

=20

Quoted 9 lines
Quoted 9 lines
Quoted 7 lines

could also decide which files get enumerated with readdir, etc.

Rob Gardner
HP

   =20

Oh yes, I think I didn't understand you answer.. but now all became=20 clear in my mind. I'm afraid that using this solution maybe cause an=

=20

Quoted 9 lines
Quoted 6 lines

increase on storage processor load, because for every request the=20 script must be executed. I already have some issues with server=20 availability, and perhaps the solution of a problem becomes the=20 worsening of another.

Still thanks for the reply and thanks for helping to clear my mind.

It's a valid concern, but fuse does not "execute a script" for every=20 operation. The script is always running, sort of like a server for th=

e=20

Quoted 4 lines

pseudo-filesystem. Each requests causes a few lines in the script to=20 be executed. There is no process creation and dispatch for each=20 operation, only a process wakeup for each operation. It is=20 impressively lightweight and it's worth trying before dismissing it a=

s=20

Quoted 2 lines

a resource drain.

Hmm.. that's interesting, but, how many requests this script can handle= ?=20
This can vary in diferent systems or configs, or even can be measured?

Quoted 1 line

But anyway, here's another idea for you-- Export a new directory that=

=20

Quoted 2 lines

contains links to only those the files you want to export.

Unfortunately, even if I wanted to, it's not possible to do this in my=20 scenario, since I have aprox. 4 milion files only in jpeg format in=20 different folders, not counting the other extensions. The major problem= =20
was the planning phase, that not counted with so many files and not=20 prepared the files organization to better performance. But for smallers= =20
systems, its a very good solution.

[]s

=46=E1bio jr.

-----------------------------------------------------------------------=
-------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30= -Day=20
trial. Simplify your report design, integration and deployment - and fo= cus on=20
what you do best, core application coding. Discover what's new with=20 Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

#7 lore raw
Quoted 20 lines · =46=E1bio Jr. wrote:

Rob Gardner escreveu:
=20

Quoted 18 lines

F=E1bio Jr. wrote:

   =20
Quoted 16 lines

Rob Gardner escreveu:
=20

     =20
Quoted 13 lines

F=E1bio Jr. wrote:
=20

       =20
Quoted 10 lines

Rob Gardner escreveu:
=20

    =20
         =20
Quoted 6 lines

F=E1bio Jr. wrote:

         =20
           =20
Quoted 3 lines

Hello all.

Is there a way to export only some specific file extensions fro=

m=20

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

a directory, like exporting only the files in /home/fabio with=20 the .jpg extension?

               =20
             =20

You could do this pretty easily by exporting a 'fuse' filesystem=

=20

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines

layered on top of your home directory.

Rob Gardner
HP

           =20
           =20

Thanks Rob for the reply,

This means that if I use this FUSE, I can export via NFS only the=

=20

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

file extension that I need? let me explain my problem, not really=

a=20

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

problem, but a doubt.

I have one storage that have my aplication files. This storage is=

=20

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

mounted via NFS in my 3 aplication servers. Now I will put anothe=

r=20

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

server, to serve only static files (jpg,png,css,js basically), an=

d=20

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

though that maybe there is a way to export only these files from=20 the storage. The problem is that these files are not organized in=

=20

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

diferent folders.

Maybe my first question doesn't explained my real necessity, but=20 your answer will make me search a little more about FUSE.

     =20
         =20

I think your first question explained your need clearly. FUSE is a=

=20

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

filesystem layer that lets you filter requests through a user=20 program such as a python script. So, for instance, every time a=20 process opens a file, a function is called in your script, and you=

=20

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

can look at the name of the file being opened, and there decide=20 whether or not to allow the file to be opened. I think this would=20 solve your problem easily. Though all the storage is "exported",=20 only files with certain names (ie, *.jpg, etc) could be opened. Yo=

u=20

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

could also decide which files get enumerated with readdir, etc.

Rob Gardner
HP

   =20
       =20

Oh yes, I think I didn't understand you answer.. but now all became=

=20

Quoted 1 line
Quoted 1 line
Quoted 1 line

clear in my mind. I'm afraid that using this solution maybe cause a=

n=20

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

increase on storage processor load, because for every request the=20 script must be executed. I already have some issues with server=20 availability, and perhaps the solution of a problem becomes the=20 worsening of another.

Still thanks for the reply and thanks for helping to clear my mind.

     =20

It's a valid concern, but fuse does not "execute a script" for every=

=20

Quoted 1 line
Quoted 1 line

operation. The script is always running, sort of like a server for t=

he=20

Quoted 1 line
Quoted 1 line

pseudo-filesystem. Each requests causes a few lines in the script to=

=20

Quoted 3 lines
Quoted 3 lines

be executed. There is no process creation and dispatch for each=20 operation, only a process wakeup for each operation. It is=20 impressively lightweight and it's worth trying before dismissing it =

as=20

Quoted 4 lines
Quoted 3 lines

a resource drain.

   =20

Hmm.. that's interesting, but, how many requests this script can hand=

le?=20

Quoted 1 line

This can vary in diferent systems or configs, or even can be measured=

?

Quoted 1 line

=20

OK now you're talking about throughput... that's different than=20 processor load. The fuser script is single threaded, I think, so it=20 probably won't scale too well.

You could go into the kernel and hack a filter into say, the nfs lookup= =20
code, and cause lookup failures for files named '*.jpg' on certain=20 exports. That would scale, though would take quite a bit more work.

Quoted 1 line
Quoted 1 line

But anyway, here's another idea for you-- Export a new directory tha=

t=20

Quoted 4 lines
Quoted 3 lines

contains links to only those the files you want to export.

   =20

Unfortunately, even if I wanted to, it's not possible to do this in m=

y=20

Quoted 2 lines

scenario, since I have aprox. 4 milion files only in jpeg format in=20 different folders, not counting the other extensions. The major probl=

em=20

Quoted 2 lines

was the planning phase, that not counted with so many files and not=20 prepared the files organization to better performance. But for smalle=

rs=20

Quoted 1 line

systems, its a very good solution.

I guess you're in a better position to judge the suitability of a=20 proposed solution... I can think of various reasons why creating links= =20
might not be a good solution, but I don't see why having millions of=20 files presents any problem, as the process of creating links can be=20 automated rather easily.

Rob Gardner
HP

-----------------------------------------------------------------------=
-------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30= -Day=20
trial. Simplify your report design, integration and deployment - and fo= cus on=20
what you do best, core application coding. Discover what's new with=20 Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs

#8 lore raw
Quoted 3 lines · Rob Gardner

You could go into the kernel and hack a filter into say, the nfs lookup code, and cause lookup failures for files named '*.jpg' on certain exports. That would scale, though would take quite a bit more work.

This sounds more like a problem that should be solved on the application side....

E.g. surely there's a way to configure apache (or whatever) to map paths like $foo.jpg to /nfs/image-server/$foo.jpg?

--b.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs