Skip to contentSkip to conversation

Volatile Ranges (v6)

RFC PATCH In review John Stultz 3 participants 8 messages Apr 7, 2012 – Apr 9, 2012

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover Volatile Ranges (v6)

Just wanted to send out another iteration of the volatile range code for review and comment.

This revision handles some improved coalescing logic, fixes some bugs in the range tree management, and also utilizes a hash tabe to avoid bloating the address_space structure with range_tree pointers.

Still looking for guidence on what a better interface for this might be as well as thoughts on how to best drop the ranges under memory pressure (vmtruncate_region is pretty close to what I want, but lockdep makes it clear that its not really safe to call from a shrinker).

Another detail is that by hanging the volatile ranges off of the address_space, the volatility for tmpfs files persists even when no one has an open fd on the file. This could cause some surprises if application A marked some pages volatile and died, then application B opened the file and had pages dropped out underneith it while it was being used. I suspect I need to clean up the volatility when all fds are dropped. But any extra insight would be useful here.

Thanks for the continued advice and feedback!
-john

CC Andrew Morton <[email protected]>
CC Android Kernel Team <[email protected]>
CC Robert Love <[email protected]>
CC Mel Gorman <[email protected]>
CC Hugh Dickins <[email protected]>
CC Dave Hansen <[email protected]>
CC Rik van Riel <[email protected]>
CC Dmitry Adamushko <[email protected]>
CC Dave Chinner <[email protected]>
CC Neil Brown <[email protected]>
CC Andrea Righi <[email protected]>
CC Aneesh Kumar K.V <[email protected]>

John Stultz (2):
[RFC] Range tree implementation
[RFC] fadvise: Add _VOLATILE,_ISVOLATILE, and _NONVOLATILE flags

 fs/inode.c                |    2 +
 include/linux/fadvise.h   |    5 +
 include/linux/rangetree.h |   56 ++++++
 include/linux/volatile.h  |   14 ++
 lib/Makefile              |    2 +-
 lib/rangetree.c           |  128 +++++++++++++
 mm/Makefile               |    2 +-
 mm/fadvise.c              |   16 ++-
 mm/volatile.c             |  440 +++++++++++++++++++++++++++++++++++++++++++++
 9 files changed, 662 insertions(+), 3 deletions(-)
 create mode 100644 include/linux/rangetree.h
 create mode 100644 include/linux/volatile.h
 create mode 100644 lib/rangetree.c
 create mode 100644 mm/volatile.c
Signature · 1 line
1.7.3.2.146.gca209
#1 lore raw

RFC PATCH 1/2 Range tree implementation

After Andrew suggested something like his mumbletree idea to better store a list of ranges, I worked on a few different approaches, and this is what I've finally managed to get working.

I suspect range-tree isn't a totally accurate name, but I couldn't quite make out the difference between range trees and interval trees, so I just picked one to call it. Do let me know if you have a better name.

The idea of storing ranges in a tree is nice, but has a number of complications. When adding a range, its possible that a large range will consume and merge a number of smaller ranges. When removing a range, its possible you may end up splitting an existing range, causing one range to become two. This makes it very difficult to provide generic list_head like behavior, as the parent structures would need to be duplicated and removed, and that has lots of memory ownership issues.

So, this is a much simplified and more list_head like implementation. You can add a node to a tree, or remove a node to a tree, but the generic implementation doesn't do the merging or splitting for you. But it does provide helpers to find overlapping and adjacent ranges.

Andrew also really wanted this range-tree implementation to be resuable so we don't duplicate the file locking logic. I'm not totally convinced that the requirements between the volatile ranges and file locking are really equivelent, but this reduced impelementation may make it possible.

Do let me know what you think or if you have other ideas for better ways to do the same.

Changelog:
v2:
* Reworked code to use an rbtree instead of splaying

v3:
* Added range_tree_next_in_range() to avoid having to start lookups from the root every time.
* Fixed some comments and return NULL instead of 0, as suggested by Aneesh Kumar K.V

v6:
* Fixed range_tree_in_range() so that it finds the earliest range, rather then the first. This allows the next_in_range() function to properly cover all the ranges in the tree.
* Minor clenaups to simplify some of the functions

CC Andrew Morton <[email protected]>
CC Android Kernel Team <[email protected]>
CC Robert Love <[email protected]>
CC Mel Gorman <[email protected]>
CC Hugh Dickins <[email protected]>
CC Dave Hansen <[email protected]>
CC Rik van Riel <[email protected]>
CC Dmitry Adamushko <[email protected]>
CC Dave Chinner <[email protected]>
CC Neil Brown <[email protected]>
CC Andrea Righi <[email protected]>
CC Aneesh Kumar K.V <[email protected]>
---
 include/linux/rangetree.h |   56 ++++++++++++++++++++
 lib/Makefile              |    2 +-
 lib/rangetree.c           |  128 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 185 insertions(+), 1 deletions(-)
 create mode 100644 include/linux/rangetree.h
 create mode 100644 lib/rangetree.c
include/linux/rangetree.h new file +56 -0
new file mode 100644
index 0000000..c61ce7c
@@ -0,0 +1,56 @@
+#ifndef _LINUX_RANGETREE_H
+#define _LINUX_RANGETREE_H
+
+#include <linux/types.h>
+#include <linux/rbtree.h>
+
+struct range_tree_node {
+	struct rb_node rb;
+	u64 start;
+	u64 end;
+};
+
+struct range_tree_root {
+	struct rb_root head;
+};
+
+static inline void range_tree_init(struct range_tree_root *root)
+{
+	root->head = RB_ROOT;
+}
+
+static inline void range_tree_node_init(struct range_tree_node *node)
+{
+	rb_init_node(&node->rb);
+	node->start = 0;
+	node->end = 0;
+}
+
+static inline int range_tree_empty(struct range_tree_root *root)
+{
+	return RB_EMPTY_ROOT(&root->head);
+}
+
+static inline
+struct range_tree_node *range_tree_root_node(struct range_tree_root *root)
+{
+	struct range_tree_node *ret;
+	ret = container_of(root->head.rb_node, struct range_tree_node, rb);
+	return ret;
+}
+
+extern struct range_tree_node *range_tree_in_range(struct range_tree_root *root,
+							 u64 start, u64 end);
+extern struct range_tree_node *range_tree_in_range_adjacent(
+						struct range_tree_root *root,
+							 u64 start, u64 end);
+extern struct range_tree_node *range_tree_next_in_range(
+						struct range_tree_node *node,
+						u64 start, u64 end);
+extern void range_tree_add(struct range_tree_root *root,
+						struct range_tree_node *node);
+extern void range_tree_remove(struct range_tree_root *root,
+						struct range_tree_node *node);
+#endif
+
+
lib/Makefile +1 -1
index 18515f0..f43ef0d 100644
@@ -12,7 +12,7 @@ lib-y := ctype.o string.o vsprintf.o cmdline.o \
 	 idr.o int_sqrt.o extable.o prio_tree.o \
 	 sha1.o md5.o irq_regs.o reciprocal_div.o argv_split.o \
 	 proportions.o prio_heap.o ratelimit.o show_mem.o \
-	 is_single_threaded.o plist.o decompress.o
+	 is_single_threaded.o plist.o decompress.o rangetree.o
 
 lib-$(CONFIG_MMU) += ioremap.o
 lib-$(CONFIG_SMP) += cpumask.o
lib/rangetree.c new file +128 -0

128 lines not on this page · raw message

Signature · 1 line
1.7.3.2.146.gca209
#2 lore raw

Re: [PATCH 1/2] [RFC] Range tree implementation

Quoted 24 lines · John Stultz

After Andrew suggested something like his mumbletree idea to better store a list of ranges, I worked on a few different approaches, and this is what I've finally managed to get working.

I suspect range-tree isn't a totally accurate name, but I couldn't quite make out the difference between range trees and interval trees, so I just picked one to call it. Do let me know if you have a better name.

The idea of storing ranges in a tree is nice, but has a number of complications. When adding a range, its possible that a large range will consume and merge a number of smaller ranges. When removing a range, its possible you may end up splitting an existing range, causing one range to become two. This makes it very difficult to provide generic list_head like behavior, as the parent structures would need to be duplicated and removed, and that has lots of memory ownership issues.

So, this is a much simplified and more list_head like implementation. You can add a node to a tree, or remove a node to a tree, but the generic implementation doesn't do the merging or splitting for you. But it does provide helpers to find overlapping and adjacent ranges.

Quoted 8 lines

Andrew also really wanted this range-tree implementation to be resuable so we don't duplicate the file locking logic. I'm not totally convinced that the requirements between the volatile ranges and file locking are really equivelent, but this reduced impelementation may make it possible.

Do let me know what you think or if you have other ideas for better ways to do the same.

Hi John,

I have implemented an interval lookup tree using the augmented features of the in-kernel rbtree for the KVM tools project. We currently use it for several things as a framework code such as MMIO memory mapping.

Quoted 1 line

From what I see in the patch, you haven't fully implemented the

interval structure (it needs to keep track of additional parameters when building it, and the search is a bit different and is based on those parameters).

I could send that code as a patch against the kernel tree if something like that is actually required for the kernel at this point.

Thanks.

#3 lore raw

Re: [PATCH 1/2] [RFC] Range tree implementation

Quoted 40 lines · Sasha Levin
Quoted 24 lines · John Stultz

After Andrew suggested something like his mumbletree idea to better store a list of ranges, I worked on a few different approaches, and this is what I've finally managed to get working.

I suspect range-tree isn't a totally accurate name, but I couldn't quite make out the difference between range trees and interval trees, so I just picked one to call it. Do let me know if you have a better name.

The idea of storing ranges in a tree is nice, but has a number of complications. When adding a range, its possible that a large range will consume and merge a number of smaller ranges. When removing a range, its possible you may end up splitting an existing range, causing one range to become two. This makes it very difficult to provide generic list_head like behavior, as the parent structures would need to be duplicated and removed, and that has lots of memory ownership issues.

So, this is a much simplified and more list_head like implementation. You can add a node to a tree, or remove a node to a tree, but the generic implementation doesn't do the merging or splitting for you. But it does provide helpers to find overlapping and adjacent ranges.

Quoted 8 lines

Andrew also really wanted this range-tree implementation to be resuable so we don't duplicate the file locking logic. I'm not totally convinced that the requirements between the volatile ranges and file locking are really equivelent, but this reduced impelementation may make it possible.

Do let me know what you think or if you have other ideas for better ways to do the same.

Hi John,

I have implemented an interval lookup tree using the augmented features of the in-kernel rbtree for the KVM tools project. We currently use it for several things as a framework code such as MMIO memory mapping.

Quoted 4 lines

From what I see in the patch, you haven't fully implemented the
interval structure (it needs to keep track of additional parameters when building it, and the search is a bit different and is based on those parameters).

Any more details on whats missing/different? Or the pros/cons of the different approaches?

Quoted 3 lines

I could send that code as a patch against the kernel tree if something like that is actually required for the kernel at this point.

Sure. I'm not married to this implementation (its just the only one so far that solves my needs - Dmitry already pointed out that the priotree might be close to sufficient, but I've not yet tried to adapt it), and whatever goes upstream needs to be generic enough to solve the related problems that a number of folks are all working on solving. So seeing your approach might be good too.

thanks
-john

#4 lore raw

Re: [PATCH 1/2] [RFC] Range tree implementation

Quoted 45 lines · John Stultz
Quoted 44 lines · Sasha Levin
Quoted 25 lines · John Stultz

After Andrew suggested something like his mumbletree idea to better store a list of ranges, I worked on a few different approaches, and this is what I've finally managed to get working.

I suspect range-tree isn't a totally accurate name, but I couldn't quite make out the difference between range trees and interval trees, so I just picked one to call it. Do let me know if you have a better name.

The idea of storing ranges in a tree is nice, but has a number of complications. When adding a range, its possible that a large range will consume and merge a number of smaller ranges. When removing a range, its possible you may end up splitting an existing range, causing one range to become two. This makes it very difficult to provide generic list_head like behavior, as the parent structures would need to be duplicated and removed, and that has lots of memory ownership issues.

So, this is a much simplified and more list_head like implementation. You can add a node to a tree, or remove a node to a tree, but the generic implementation doesn't do the merging or splitting for you. But it does provide helpers to find overlapping and adjacent ranges.

Quoted 8 lines

Andrew also really wanted this range-tree implementation to be resuable so we don't duplicate the file locking logic. I'm not totally convinced that the requirements between the volatile ranges and file locking are really equivelent, but this reduced impelementation may make it possible.

Do let me know what you think or if you have other ideas for better ways to do the same.

Hi John,

I have implemented an interval lookup tree using the augmented features of the in-kernel rbtree for the KVM tools project. We currently use it for several things as a framework code such as MMIO memory mapping.

Quoted 7 lines
Quoted 4 lines

 From what I see in the patch, you haven't fully implemented the interval structure (it needs to keep track of additional parameters when building it, and the search is a bit different and is based on those parameters).

Any more details on whats missing/different? Or the pros/cons of the different approaches?

I took the implementation from 'Introduction to Algorithms', which called for keeping the max high all the nodes in each sub-tree in the root of that tree. I don't remember what stood behind that but I can look it up when I get back to the book :)

Quoted 9 lines
Quoted 3 lines

I could send that code as a patch against the kernel tree if something like that is actually required for the kernel at this point.

Sure.  I'm not married to this implementation (its just the only one so far that solves my needs - Dmitry already pointed out that the priotree might be close to sufficient, but I've not yet tried to adapt it), and whatever goes upstream needs to be generic enough to solve the related problems that a number of folks are all working on solving.  So seeing your approach might be good too.

The code is located here:
https://github.com/penberg/linux-kvm/blob/master/tools/kvm/util/rbtree-interval.c

Note that we didn't deal with intersecting ranges simply because there was no call for that (we used it to map devices in a virtual memory range, and those can't intersect). But it's not much of an issue expanding the range intersection function to deal with that.

If the code above looks ok to you I'll format it as a patch and re-send it.

Thanks.

#5 lore raw

RFC PATCH 2/2 fadvise: Add _VOLATILE,_ISVOLATILE, and _NONVOLATILE flags

In reply to John Stultz

This patch provides new fadvise flags that can be used to mark file pages as volatile, which will allow it to be discarded if the kernel wants to reclaim memory.

This is useful for userspace to allocate things like caches, and lets the kernel destructively (but safely) reclaim them when there's memory pressure.

It's different from FADV_DONTNEED since the pages are not immediately discarded; they are only discarded under pressure.

This is very much influenced by the Android Ashmem interface by Robert Love so credits to him and the Android developers. In many cases the code & logic come directly from the ashmem patch. The intent of this patch is to allow for ashmem-like behavior, but embeds the idea a little deeper into the VM code, instead of isolating it into a specific driver.

I'm very much a newbie at the VM code, so At this point, I just want to try to get some input on the patch, so if you have another idea for using something other then fadvise, or other thoughts on how the volatile ranges are stored, I'd be really interested in hearing them. So let me know if you have any comments for feedback!

Also many thanks to Dave Hansen who helped design and develop the initial version of this patch, and has provided continued review and mentoring for me in the VM code.

v2:
* After the valid critique that just dropping pages would poke holes in volatile ranges, and instead we should zap an entire range if we drop any of it, I changed the code to more closely mimic the ashmem implementation, which zaps entire ranges via a shrinker using an lru list that tracks which range has been marked volatile the longest.

v3:
* Reworked to use range tree implementation.

v4:
* Renamed functions to avoid confusion.
* More consistant PAGE_CACHE_SHIFT usage, suggested by Dmitry Adamushko
* Fixes exit without unlocking issue found by Dmitry Adamushko
* Migrate to rbtree based rangetree implementation
* Simplified locking to use global lock (we were grabbing global lru lock every time anyway).
* Avoid ENOMEM isses by allocating before we get into complicated code.
* Add some documentation to the volatile.c file from Neil Brown

v5:
* More fixes suggested by Dmitry Adamushko
* Improve range colescing so that we don't coalesce neighboring purged ranges.
* Utilize range_tree_next_in_range to avoid doing every lookup from the tree's root.

v6:
* Immediately zap range if we coalesce overlapping purged range.
* Use hash table to do mapping->rangetree lookup instead of bloating the address_space structure

Known issues:
* Lockdep doesn't like calling vmtruncate_range() from a shrinker. Any help here on how to address this would be appreciated. I've tried switching to invalidate_inode_pages2_range, but that always returns EBUSY in my testing, and I don't really want to launder dirty pages, instead I want to zap them.

* Volatile range persistence needs to be though through. Currently the volatility follows the inode in memory, which for tmpfs sticks around. This means application A could open a file, mark it volatile, close it. Then application B opens the file, and as its using it finds the pages earlier marked volatile disappearing under it. I think it probably makes more sense to drop all volatile ranges after all the fds to the file have been closed. This may also be something that changes if we switch up to a different interface. Suggestions here would be great.

CC Andrew Morton <[email protected]>
CC Android Kernel Team <[email protected]>
CC Robert Love <[email protected]>
CC Mel Gorman <[email protected]>
CC Hugh Dickins <[email protected]>
CC Dave Hansen <[email protected]>
CC Rik van Riel <[email protected]>
CC Dmitry Adamushko <[email protected]>
CC Dave Chinner <[email protected]>
CC Neil Brown <[email protected]>
CC Andrea Righi <[email protected]>
CC Aneesh Kumar K.V <[email protected]>
---
 fs/inode.c               |    2 +
 include/linux/fadvise.h  |    5 +
 include/linux/volatile.h |   14 ++
 mm/Makefile              |    2 +-
 mm/fadvise.c             |   16 ++-
 mm/volatile.c            |  440 ++++++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 477 insertions(+), 2 deletions(-)
 create mode 100644 include/linux/volatile.h
 create mode 100644 mm/volatile.c
fs/inode.c +2 -0
index 9f4f5fe..590d314 100644
@@ -17,6 +17,7 @@
 #include <linux/prefetch.h>
 #include <linux/buffer_head.h> /* for inode_has_buffers */
 #include <linux/ratelimit.h>
+#include <linux/volatile.h>
 #include "internal.h"
 
 /*
@@ -244,6 +245,7 @@ void __destroy_inode(struct inode *inode)
 	if (inode->i_default_acl && inode->i_default_acl != ACL_NOT_CACHED)
 		posix_acl_release(inode->i_default_acl);
 #endif
+	mapping_clear_volatile_ranges(&inode->i_data);
 	this_cpu_dec(nr_inodes);
 }
 EXPORT_SYMBOL(__destroy_inode);
include/linux/fadvise.h +5 -0
index e8e7471..443951c 100644
@@ -18,4 +18,9 @@
 #define POSIX_FADV_NOREUSE	5 /* Data will be accessed once.  */
 #endif
 
+#define POSIX_FADV_VOLATILE	8  /* _can_ toss, but don't toss now */
+#define POSIX_FADV_NONVOLATILE	9  /* Remove VOLATILE flag */
+
+
+
 #endif	/* FADVISE_H_INCLUDED */
include/linux/volatile.h new file +14 -0
new file mode 100644
index 0000000..5460d7b
@@ -0,0 +1,14 @@
+#ifndef _LINUX_VOLATILE_H
+#define _LINUX_VOLATILE_H
+
+#include <linux/fs.h>
+
+extern long mapping_range_volatile(struct address_space *mapping,
+				pgoff_t start_index, pgoff_t end_index);
+extern long mapping_range_nonvolatile(struct address_space *mapping,
+				pgoff_t start_index, pgoff_t end_index);
+extern long mapping_range_isvolatile(struct address_space *mapping,
+				pgoff_t start_index, pgoff_t end_index);
+extern void mapping_clear_volatile_ranges(struct address_space *mapping);
+
+#endif /* _LINUX_VOLATILE_H */
mm/Makefile +1 -1
index 50ec00e..7b6c7a8 100644
@@ -13,7 +13,7 @@ obj-y			:= filemap.o mempool.o oom_kill.o fadvise.o \
 			   readahead.o swap.o truncate.o vmscan.o shmem.o \
 			   prio_tree.o util.o mmzone.o vmstat.o backing-dev.o \
 			   page_isolation.o mm_init.o mmu_context.o percpu.o \
-			   $(mmu-y)
+			   volatile.o $(mmu-y)
 obj-y += init-mm.o
 
 ifdef CONFIG_NO_BOOTMEM
mm/fadvise.c +15 -1
index 469491e0..3e33845 100644
@@ -17,6 +17,7 @@
 #include <linux/fadvise.h>
 #include <linux/writeback.h>
 #include <linux/syscalls.h>
+#include <linux/volatile.h>
 
 #include <asm/unistd.h>
 
@@ -106,7 +107,7 @@ SYSCALL_DEFINE(fadvise64_64)(int fd, loff_t offset, loff_t len, int advice)
 		nrpages = end_index - start_index + 1;
 		if (!nrpages)
 			nrpages = ~0UL;
-		
+
 		ret = force_page_cache_readahead(mapping, file,
 				start_index,
 				nrpages);
@@ -128,6 +129,19 @@ SYSCALL_DEFINE(fadvise64_64)(int fd, loff_t offset, loff_t len, int advice)
 			invalidate_mapping_pages(mapping, start_index,
 						end_index);
 		break;
+	case POSIX_FADV_VOLATILE:
+		/* First and last PARTIAL page! */
+		start_index = offset >> PAGE_CACHE_SHIFT;
+		end_index = endbyte >> PAGE_CACHE_SHIFT;
+		ret = mapping_range_volatile(mapping, start_index, end_index);
+		break;
+	case POSIX_FADV_NONVOLATILE:
+		/* First and last PARTIAL page! */
+		start_index = offset >> PAGE_CACHE_SHIFT;
+		end_index = endbyte >> PAGE_CACHE_SHIFT;
+		ret = mapping_range_nonvolatile(mapping, start_index,
+								end_index);
+		break;
 	default:
 		ret = -EINVAL;
 	}
mm/volatile.c new file +440 -0

440 lines not on this page · raw message

Signature · 1 line
1.7.3.2.146.gca209
#6 lore raw
In reply to John Stultz
Quoted 7 lines · John Stultz

Another detail is that by hanging the volatile ranges off of the address_space, the volatility for tmpfs files persists even when no one has an open fd on the file. This could cause some surprises if application A marked some pages volatile and died, then application B opened the file and had pages dropped out underneith it while it was being used. I suspect I need to clean up the volatility when all fds are dropped.

And how do you handle the regions that have already been purged by this moment? Unless B has some specific mechanism to verify the consistency of the content, a sensible way would be to always mark off the regions as non-volatile before accessing them and verify the return code to see if there are holes.

More generally, what if B opens the file while A is still working with it? Besides the use of normal synchronization mechanisms, B should not make any assumption on the current state of the regions (unless there is a high-level protocol between A and B to share this info). So an explicit mark-off-as-non_volatile could be a simple generic mechanism.

--Dmitry

#7 lore raw
Quoted 13 lines · Dmitry Adamushko
Quoted 6 lines · John Stultz

Another detail is that by hanging the volatile ranges off of the address_space, the volatility for tmpfs files persists even when no one has an open fd on the file. This could cause some surprises if application A marked some pages volatile and died, then application B opened the file and had pages dropped out underneith it while it was being used. I suspect I need to clean up the volatility when all fds are dropped.

And how do you handle the regions that have already been purged by this moment? Unless B has some specific mechanism to verify the consistency of the content, a sensible way would be to always mark off the regions as non-volatile before accessing them and verify the return code to see if there are holes.

Quoted 6 lines

More generally, what if B opens the file while A is still working with it? Besides the use of normal synchronization mechanisms, B should not make any assumption on the current state of the regions (unless there is a high-level protocol between A and B to share this info). So an explicit mark-off-as-non_volatile could be a simple generic mechanism.

So yes, marking as non-volatile before you use pages would be a way to avoid the issue. But it still rubs me the wrong way.

I think the main issue I have with it is that it makes volatility the assumed state. So unless you mark it non-volatile to begin with, the file could be volatile somewhere. I feel like volatility should be the special state, not the assumed one, so normal applications that don't think about volatility are less-likely to be surprised.

Now, when you have concurrent users of a file, you have to coordinate, and things can change under you. That's an expectation people already have. But if volatile ranges persist, its sort of introducing a form of concurrency to non-concurrent access. Where a killed application can reach from the grave and zap a page in file someone else is using. I think this is too unexpected.

The case that bit me in particular was in testing this patch, I had an application (call it A) that had a bug and was marking a larger range volatile then it re-set to non-volatile. Then when using the same file later with a different test application (call it B), I was seeing those further pages be zapped unexpectedly. It took me a while to realize that it wasn't a problem with the B application, or the patch itself, but was a persistent range that was set much earlier by A.

So I suspect it would be better if the volatile ranges should be something that are cleared out when the last fd is closed.

thanks
-john