Skip to contentSkip to conversation

What is needed to build an AFS fileserver on top of BTRFS?

DISCUSSION David Howells 4 participants 7 messages Dec 17, 2013 Btrfs file system

lore
Share & tools
lore
#0 lore raw

It has occurred to me and others that something like BTRFS could be a good fit to build an AFS fileserver directly on top of. The question is what facilities would be needed from BTRFS to make this work?

So I thought I'd kick off a shopping list;-)

(1) 64-bit data version numbers that increase monotonically with each write.

Yes, this is likely to cause some performance degredation as it introduces an ordering over data writes and metadata writes to a file. Maybe writes can be batched to improve performance?

(2) Storage for ACLs and AFS UIDs. Having shareable ACLs might also be useful.

Xattrs would likely do for this.

(3) The ability to snapshot a filesystem to make backups and for pushing to read-only volume servers.

(4) A 32-bit vnode number and 32-bit vnode uniquifier/generation number.

These don't necessarily have to be stored by BTRFS directly but could instead be in a separate database file that gets snapshotted also.

(5) The ability to set the vnode number, vnode uniquifier and data version number to specific values. Necessary to clone volumes and restore volume dumps.

David

#1 lore raw
Quoted 19 lines · David Howells

It has occurred to me and others that something like BTRFS could be a good fit to build an AFS fileserver directly on top of. The question is what facilities would be needed from BTRFS to make this work?

So I thought I'd kick off a shopping list;-)

(1) 64-bit data version numbers that increase monotonically with each write.

Yes, this is likely to cause some performance degredation as it introduces an ordering over data writes and metadata writes to a file. Maybe writes can be batched to improve performance?

(2) Storage for ACLs and AFS UIDs. Having shareable ACLs might also be useful.

Xattrs would likely do for this.

(3) The ability to snapshot a filesystem to make backups and for pushing to read-only volume servers.

Quoted 8 lines

(4) A 32-bit vnode number and 32-bit vnode uniquifier/generation number.

These don't necessarily have to be stored by BTRFS directly but could instead be in a separate database file that gets snapshotted also.

(5) The ability to set the vnode number, vnode uniquifier and data version number to specific values. Necessary to clone volumes and restore volume dumps.

Hmmm, what exactly are vnodes? Could we put them in xattrs?

-chris

#2 lore raw
Quoted 1 line · Chris Mason

Hmmm, what exactly are vnodes? Could we put them in xattrs?

vnode numbers are AFS's equivalent of inode numbers. Since they're one per file, they could be the object filename.

Probably there would have to be a table of {vnode,latest_uniquifier} as the uniquifier must still go up even if the vnode is unused for a while, so there could also be a table of {vnode,btrfs_file}.

David

#3 lore raw
Quoted 6 lines · David Howells
Quoted 1 line · Chris Mason

Hmmm, what exactly are vnodes? Could we put them in xattrs?

vnode numbers are AFS's equivalent of inode numbers. Since they're one per file, they could be the object filename.

Yes, in fact, the volume, vnode number, uniqifier, and DV are effectively the "name" the fileserver uses for the underlying inode. Note that if the fileserver is maintaining the vnode indices, then you don't actually _need_ to store a uniqifier for normal operation, because at any given time, a volume can contain at most one vnode with a particular vnode number, and that vnode's uniqifier is stored in the index. The uniqifier is used on-the-wire to distinguish different files that existed at different points in time with the same vnode number.

Quoted 3 lines

Probably there would have to be a table of {vnode,latest_uniquifier} as the uniquifier must still go up even if the vnode is unused for a while, so there could also be a table of {vnode,btrfs_file}.

No, you don't actually have to do this. The OpenAFS fileserver maintains a single uniqifier for an entire volume, and simply increments it every time a vnode is created.

-- Jeff

#4 lore raw
In reply to David Howells
Quoted 10 lines · David Howells

It has occurred to me and others that something like BTRFS could be a good fit to build an AFS fileserver directly on top of. The question is what facilities would be needed from BTRFS to make this work? So I thought I'd kick off a shopping list;-)

(1) 64-bit data version numbers that increase monotonically with
each write. Yes, this is likely to cause some performance degredation as it introduces an ordering over data writes and metadata writes to a file. Maybe writes can be batched to improve performance?

Do these have to be per-file? If not, then you might be able to get
away with using the transid, which is a filesystem-global monotonically-increasing number.

btrfs batches disk writes already, and uses the transid to
differentiate these -- the writes come at 30 second intervals (by default, although there's an option to change the period). There may be multiple distinct changes to a single file within that transaction (although obviously, only the state of the file after the last one gets written to disk). I don't know exactly what you need it for, so this may or may not be appropriate here.

Ceph uses transids for [something, mumble, wavy-hand] -- I don't
know if the use-case for Ceph is equivalent to the use-case for AFS.

Quoted 2 lines

(2) Storage for ACLs and AFS UIDs. Having shareable ACLs might also
be useful. Xattrs would likely do for this.

This would seem like a reasonable place to put them, given that
that's what POSIX ACLs do, and we have POSIX ACL support already.

Quoted 2 lines

(3) The ability to snapshot a filesystem to make backups and for pushing to read-only volume servers.

We have snapshots of subvolumes, but not the filesystem as a whole.

Quoted 8 lines

(4) A 32-bit vnode number and 32-bit vnode uniquifier/generation
number. These don't necessarily have to be stored by BTRFS directly but could instead be in a separate database file that gets snapshotted also.

(5) The ability to set the vnode number, vnode uniquifier and data version number to specific values. Necessary to clone volumes and restore volume dumps.

What's a vnode meant to represent? I'm not familiar with the
terminology.

Hugo.

Signature · 4 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
       --- "Are you the man who rules the Universe?" "Well,  I ---       
                              try not to."                               
#5 lore raw
Quoted 9 lines · Hugo Mills
Quoted 5 lines

(1) 64-bit data version numbers that increase monotonically with
each write. Yes, this is likely to cause some performance degredation as it introduces an ordering over data writes and metadata writes to a file. Maybe writes can be batched to improve performance?

Do these have to be per-file? If not, then you might be able to get
away with using the transid, which is a filesystem-global monotonically-increasing number.

Yes. If you send a write RPC op to the server, you get back the new version number. If the new version number is not the old version number + 1 you know there was a collision with a write from another client and you have to flush your cache for that file and request a new "callback" (ie. a promise to notify you if someone else changes the file).

Quoted 4 lines
Quoted 2 lines

(3) The ability to snapshot a filesystem to make backups and for pushing to read-only volume servers.

We have snapshots of subvolumes, but not the filesystem as a whole.

By "filesystem" I meant the current state of an AFS volume. Very likely this would be represented by a BTRFS subvolume, if I understand it correctly. You might have several AFS volumes represented within a BTRFS filesystem. They would be manipulated independently.

Quoted 6 lines
Quoted 3 lines

(5) The ability to set the vnode number, vnode uniquifier and data version number to specific values. Necessary to clone volumes and restore volume dumps.

What's a vnode meant to represent? I'm not familiar with the
terminology.

AFS's equivalent of an inode with a 32-bit number representing it. See my reply to Chris's question about the same thing.

David

#6 lore raw
Quoted 12 lines · David Howells
Quoted 9 lines · Hugo Mills
Quoted 5 lines

(1) 64-bit data version numbers that increase monotonically with
each write. Yes, this is likely to cause some performance degredation as it introduces an ordering over data writes and metadata writes to a file. Maybe writes can be batched to improve performance?

Do these have to be per-file? If not, then you might be able to get
away with using the transid, which is a filesystem-global monotonically-increasing number.

Quoted 5 lines

Yes. If you send a write RPC op to the server, you get back the new version number. If the new version number is not the old version number + 1 you know there was a collision with a write from another client and you have to flush your cache for that file and request a new "callback" (ie. a promise to notify you if someone else changes the file).

Right. So, the DV must increment by exactly one for each successful StoreData (and not for other changes). This is important because clients cache data and metadata independently, and cached data is labeled with the file's DV. This means that even if metadata for a file has to be refetched for some reason (for example, an expired callback), the _data_ doesn't have to be refetched unless it has actually changed, or been evicted from the client's cache due to cache pressure.

-- Jeff