PRAM: Persistent over-kexec memory storage
Subsystems (6)
cover PRAM: Persistent over-kexec memory storage
Hi,
This patchset implements persistent over-kexec memory storage or PRAM, which is intended to be used for saving memory pages of the currently executing kernel and restoring them after a kexec in the newly booted one. This can be utilized for speeding up reboot by leaving process memory and/or FS caches in-place. The patchset introduces the PRAM kernel API serving for that purpose and makes use of this API to make tmpfs 'persistent', i.e. makes it possible to save tmpfs tree on unmount and restore it on the next mount even if the system is kexec'd between the mount and unmount.
For further details, please see below.
-- The problem --
If Ksplice is not available or cannot be applied, a kernel update requires restarting the system, which implies reinitialization of all running application. Since this is a disk-bound operation, it can take quite a lot of time. What is worse, if the host serves as a web or database or whatever else server, apart from huge downtime the system reboot will cause any existent connection to be dropped, which may not always be tolerated.
Although the kernel boot can be speeded up significantly by employing kexec, which jumps directly to the new kernel skipping the BIOS and boot loader stages, it has nothing to do with running applications, which still need to be restarted.
-- The solution --
There is the rapidly developing criu project (www.criu.org), which targets on saving running application states to disk to be restored later. It is already accepted by the community and hopefully it will soon be able to dump and restore every Linux process. Obviously criu can be successfully used to omit full application reinitialization on reboot, but criu'ing may still take a lot of time. To illustrate, imagine a database server that cached to its internal buffers 100 GB of data. Writing the image of that process sequentially at 100 MB/s will take more that 15 minutes. Multiplied by two, since the image must be read after reboot, it gives half an hour of downtime! The server's clients will probably disconnect by timeout until the system is up and running, which cancels all the benefits of criu'ing.
However, the disk read/write, which is the bottleneck in the criu scheme, can be avoided if kexec is used for rebooting. The point is kexec does not reset the RAM state leaving all data written to memory intact. This fact is already utilized by kdump to gather the full memory image on kernel panic. If it were possible to save arbitrary data and restore them after kexec, it could be utilized to completely avoid disk accesses when criu'ing.
This patchset implements the kernel API for saving data to be restored after kexec and employs it to make tmpfs 'persistent' as described below.
-- Usage --
1) Boot kernel with 'pram_banned=MEMRANGE' boot option.
MEMRANGE=MEMMIN-MEMMAX specifies memory range where kexec will load the new kernel code. It is used to avoid conflicts with persistent memory as described in implementation details. MEMRANGE=0-128M should be enough.
2) Mount tmpfs with 'pram=NAME' option.
NAME is an arbitrary string specifying persistent memory node. Different tmpfs trees may be saved to PRAM if different names are passed.
# mkdir -p /mnt/crdump
# mount -t tmpfs -o pram=mytmpfs none /mnt/crdump
3) Checkpoint the process tree you'd want to pass over kexec to tmpfs.
# criu dump -D /mnt/crdump -t $PID
4) Unmount tmpfs.
It will be automatically saved to PRAM on unmount.
# umount /mnt/crdump
5) Load the new kernel image.
Kexec needs some tweaking for PRAM to work. First, one should pass PRAM super block pfn via 'pram' boot option. The pfn is exported via the sysfs file /sys/kernel/pram. Second, kexec must be forced to load the kernel code to MEMRANGE (see p.1).
# kexec --load /vmlinuz --initrd=initrd.img \
--append="$(cat /proc/cmdline | sed -e 's/pram=[^ ]*//g') pram=$(cat /sys/kernel/pram)" \
--mem-min=$MEMMIN --mem-max=$MEMMAX
6) Boot to the new kernel.
# reboot
7) Mount tmpfs with 'pram=NAME' option.
It should find the PRAM node with the tmpfs tree saved on previous unmount and restore it.
# mount -t tmpfs -o pram=mytmpfs none /mnt/crdump
8) Restore the process saved in p.3.
# criu restore -d -D /mnt/crdump
9) Remove the dump and unmount tmpfs
# rm -f /mnt/crdump
# umount /mnt/crdump-- Implementation details --
* Saving a memory page is simply incrementing its refcounter so the page will not get freed when the last user puts it. So the data saved to PRAM may be safely used as usual.
* To preserve persistent memory in the newly booted kernel, PRAM marks all the pages saved as reserved at early boot so that they will not be recycled. For the new kernel to find persistent memory metadata, one should pass PRAM super block pfn, which is exported via /sys/kernel/pram, in the 'pram' boot param.
* Since some memory is required for completing boot sequence, PRAM tracks all memory regions that have ever been reserved by other parts of the kernel and avoids using them for persistent memory. Since the device configuration cannot change during kexec, and the newly booted kernel is likely to have the same set of device drivers, it should work in most cases.
* Since kexec may load the new kernel code to any memory region, it can destroy persistent memory. To exclude this, kexec should be forced to load the new kernel code to a memory region that is banned for PRAM. For that purpose, there is the 'pram_banned' boot param and --mem-min and --mem-max otpions of the kexec utility.
* If a conflict still happens, it will be identified and all persistent memory will be discarded to prevent further errors. It is guaranteed by checksumming all data saved to PRAM.
* tmpfs is saved to PRAM on unmount and loaded on mount if 'pram=NAME' mount option is passed. NAME specifies the PRAM node to save data to. This is to allow saving several tmpfs trees.
* Saving tmpfs to PRAM is not well elaborated at present and serves rather as a proof of concept. Namely, only regular files without multiple hard links are supported and tmpfs may not be swapped out. If these requirements are not met, save to PRAM will be aborted spewing a message to the kernel log. This is not very difficult to fix, but at present one should turn off swap to test the feature.
-- Future plans --
What we'd like to do:
* Implement swap entries 'freezing' to allow saving a swapped out tmpfs.
* Implement full support of tmpfs including saving dirs, special files, etc.
* Implement SPLICE_F_MOVE, SPLICE_F_GIFT flags for splicing data from/to shmem. This would allow avoiding memory copying on checkpoint/restore.
* Save uptodate fs cache on umount to be restored on mount after kexec.
Thanks,
Vladimir Davydov (13):
mm: add PRAM API stubs and Kconfig
mm: PRAM: implement node load and save functions
mm: PRAM: implement page stream operations
mm: PRAM: implement byte stream operations
mm: PRAM: link nodes by pfn before reboot
mm: PRAM: introduce super block
mm: PRAM: preserve persistent memory at boot
mm: PRAM: checksum saved data
mm: PRAM: ban pages that have been reserved at boot time
mm: PRAM: allow to ban arbitrary memory ranges
mm: PRAM: allow to free persistent memory from userspace
mm: shmem: introduce shmem_insert_page
mm: shmem: enable saving to PRAM
arch/x86/kernel/setup.c | 2 + arch/x86/mm/init_32.c | 5 + arch/x86/mm/init_64.c | 5 + include/linux/pram.h | 62 +++ include/linux/shmem_fs.h | 29 ++ mm/Kconfig | 14 + mm/Makefile | 1 + mm/bootmem.c | 4 + mm/memblock.c | 7 +- mm/pram.c | 1279 ++++++++++++++++++++++++++++++++++++++++++++++ mm/shmem.c | 97 +++- mm/shmem_pram.c | 378 ++++++++++++++ 12 files changed, 1878 insertions(+), 5 deletions(-) create mode 100644 include/linux/pram.h create mode 100644 mm/pram.c create mode 100644 mm/shmem_pram.c
Signature · 7 lines
1.7.10.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC PATCH 1/13 mm: add PRAM API stubs and Kconfig
Persistent memory subsys or PRAM is intended to be used for saving memory pages of the currently executing kernel and restoring them after a kexec in the newly booted one. This can be utilized for speeding up reboot by leaving process memory and/or FS caches in-place.
The proposed API:
* Persistent memory is divided into nodes, which can be saved or loaded independently of each other. The nodes are identified by unique name strings. PRAM node is created (removed) when save (load) is initiated by calling pram_prepare_save() (pram_prepare_load()), see below.
* For saving/loading data from a PRAM node an instance of the pram_stream struct is used. The struct is initialized by calling pram_prepare_save() for saving data or pram_prepare_load() for loading data. After save (load) is complete, pram_finish_save() (pram_finish_load()) must be called. If an error occurred during save, the saved data and the PRAM node may be freed by calling pram_discard_save() instead of pram_finish_save().
* Each pram_stream has a type, which determines the set of operations that may be used for saving/loading data. The type is defined by the pram_stream_type enum. Currently there are two stream types
available: PRAM_PAGE_STREAM to save/load memory pages, and PRAM_BYTE_STREAM to save/load byte strings. For page streams pram_save_page() and pram_load_page() may be used, and for byte streams pram_write() and pram_read() may be used for saving and loading data respectively.
Thus a sequence of operations for saving/loading data from PRAM should look like:
* For saving data to PRAM:
/* create PRAM node and initialize stream for saving data to it */
pram_prepare_save()
/* save data to the node */
pram_save_page()[,...] /* for page stream, or
pram_write()[,...] * ... for byte stream */
/* commit the save or discard and delete the node */
pram_finish_save() /* on success, or
pram_discard_save() * ... in case of error */* For loading data from PRAM:
/* remove PRAM node from the list and initialize stream for
* loading data from it */
pram_prepare_load()
/* load data from the node */
pram_load_page()[,...] /* for page stream, or
pram_read()[,...] * ... for byte stream */
/* free the node */
pram_finish_load()
---
include/linux/pram.h | 38 +++++++++++++++
mm/Kconfig | 9 ++++
mm/Makefile | 1 +
mm/pram.c | 131 ++++++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 179 insertions(+)
create mode 100644 include/linux/pram.h
create mode 100644 mm/pram.cinclude/linux/pram.h new file +38 -0
@@ -0,0 +1,38 @@ +#ifndef _LINUX_PRAM_H +#define _LINUX_PRAM_H + +#include <linux/gfp.h> +#include <linux/types.h> +#include <linux/mm_types.h> + +struct pram_stream ; + +#define PRAM_NAME_MAX 256/* including nul */ + +enum pram_stream_type { +PRAM_PAGE_STREAM , +PRAM_BYTE_STREAM , +}; + +extern int pram_prepare_save (struct pram_stream *ps, +const char *name,enum pram_stream_type type,gfp_t gfp_mask); +extern void pram_finish_save (struct pram_stream *ps); +extern void pram_discard_save (struct pram_stream *ps); + +extern int pram_prepare_load (struct pram_stream *ps, +const char *name,enum pram_stream_type type); +extern void pram_finish_load (struct pram_stream *ps); + +#define PRAM_PAGE_LRU 0x01/* page is on the LRU */ + +/* page-stream specific methods */ +extern int pram_save_page (struct pram_stream *ps, +struct page *page,int flags); +extern struct page *pram_load_page (struct pram_stream *ps,int *flags); + +/* byte-stream specific methods */ +extern ssize_t pram_write (struct pram_stream *ps, +const void *buf,size_t count); +extern size_t pram_read (struct pram_stream *ps,void *buf,size_t count); + +#endif /* _LINUX_PRAM_H */
mm/Kconfig +9 -0
@@ -471,3 +471,12 @@ config FRONTSWAP and swap data is stored as normal on the matching swap device. If unsure, say Y to enable frontswap. + +config PRAM + bool "Persistent over-kexec memory storage" + default n + help + This option adds the kernel API that enables saving memory pages of + the currently executing kernel and restoring them after a kexec in + the newly booted one. This can be utilized for speeding up reboot by + leaving process memory and/or FS caches in-place.
mm/Makefile +1 -0
@@ -58,3 +58,4 @@ obj-$(CONFIG_DEBUG_KMEMLEAK) += kmemleak.o obj-$(CONFIG_DEBUG_KMEMLEAK_TEST) += kmemleak-test.o obj-$(CONFIG_CLEANCACHE) += cleancache.o obj-$(CONFIG_MEMORY_ISOLATION) += page_isolation.o +obj-$(CONFIG_PRAM) += pram.o
mm/pram.c new file +131 -0
131 lines not on this page · raw message
Signature · 7 lines
1.7.10.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC PATCH 2/13 mm: PRAM: implement node load and save functions
In reply to Vladimir DavydovPersistent memory is divided into nodes, which can be saved and loaded independently of each other. PRAM nodes are kept on the list and identified by unique names. Whenever a save operation is initiated by calling pram_prepare_save(), a new node is created and linked to the list. When the save operation has been committed by calling pram_finish_save(), the node becomes loadable. A load operation can be then initiated by calling pram_prepare_load(), which deletes the node from the list and prepares the corresponding stream for loading data from it. After the load has been finished, the pram_finish_load() function must be called to free the node. Nodes are also deleted when a save operation is discarded, i.e. pram_discard_save() is called instead of pram_finish_save().
--- include/linux/pram.h | 7 ++- mm/pram.c | 158 ++++++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 159 insertions(+), 6 deletions(-)
include/linux/pram.h +6 -1
@@ -5,7 +5,12 @@#include <linux/types.h> #include <linux/mm_types.h> -struct pram_stream ; +struct pram_node ; + +struct pram_stream { +gfp_t gfp_mask; +struct pram_node *node; +};#define PRAM_NAME_MAX 256/* including nul */
mm/pram.c +153 -5
209 lines not on this page · raw message
Signature · 7 lines
1.7.10.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC PATCH 3/13 mm: PRAM: implement page stream operations
In reply to Vladimir DavydovUsing the pram_save_page() function, one can populate PRAM nodes with memory pages, which can be then loaded using the pram_load_page() function. Saving a memory page to PRAM is implemented as storing the pfn in the PRAM node and incrementing its ref count so that it will not get freed after the last user puts it.
--- include/linux/pram.h | 3 + mm/pram.c | 166 +++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 167 insertions(+), 2 deletions(-)
include/linux/pram.h +3 -0
@@ -6,10 +6,13 @@#include <linux/mm_types.h> struct pram_node ; +struct pram_link ;struct pram_stream {gfp_t gfp_mask;struct pram_node *node; +struct pram_link *link;/* current link */ +unsigned int page_index;/* next page index in link */ };#define PRAM_NAME_MAX 256/* including nul */
mm/pram.c +164 -2
227 lines not on this page · raw message
Signature · 7 lines
1.7.10.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC PATCH 4/13 mm: PRAM: implement byte stream operations
In reply to Vladimir DavydovThis patch adds ability to save arbitrary byte strings to PRAM using pram_write() to be restored later using pram_read(). These two operations are implemented on top of pram_save_page() and pram_load_page() respectively.
--- include/linux/pram.h | 4 +++ mm/pram.c | 86 ++++++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 88 insertions(+), 2 deletions(-)
include/linux/pram.h +4 -0
@@ -13,6 +13,10 @@ struct pram_stream {struct pram_node *node;struct pram_link *link;/* current link */ unsigned int page_index;/* next page index in link */ + +/* byte-stream specific */ +struct page *data_page; +unsigned int data_offset; };#define PRAM_NAME_MAX 256/* including nul */
mm/pram.c +84 -2
116 lines not on this page · raw message
Signature · 7 lines
1.7.10.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC PATCH 5/13 mm: PRAM: link nodes by pfn before reboot
In reply to Vladimir DavydovSince page structs, which are used for linking PRAM nodes, are cleared on boot, organize all PRAM nodes into a list singly-linked by pfn's before reboot to facilitate the node list restore in the new kernel.
--- mm/pram.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+)
mm/pram.c +50 -0
@@ -1,11 +1,15 @@#include <linux/err.h> #include <linux/gfp.h> #include <linux/highmem.h> +#include <linux/init.h> #include <linux/kernel.h> #include <linux/list.h> #include <linux/mm.h> +#include <linux/module.h> #include <linux/mutex.h> +#include <linux/notifier.h> #include <linux/pram.h> +#include <linux/reboot.h> #include <linux/sched.h> #include <linux/string.h> #include <linux/types.h>
@@ -42,6 +46,9 @@ struct pram_link {* singly-linked list of PRAM link structures (see above), the node has a * pointer to the head of. * +* To facilitate data restore in the new kernel, before reboot all PRAM nodes +* are organized into a list singly-linked by pfn's (see pram_reboot()). +* * The structure occupies a memory page. */ struct pram_node {
@@ -49,6 +56,7 @@ struct pram_node {__u32 type;/* data type, see enum pram_stream_type */ __u64 data_len;/* data size, only for byte streams */ __u64 link_pfn;/* points to the first link of the node */ +__u64 node_pfn;/* points to the next node in the node list */ __u8 name[PRAM_NAME_MAX ]; };
@@ -57,6 +65,10 @@ struct pram_node {#define PRAM_LOAD 2#define PRAM_ACCMODE_MASK 3 +/* +* For convenience sake PRAM nodes are kept in an auxiliary doubly-linked list +* connected through the lru field of the page struct. +*/ static LIST_HEAD (pram_nodes);/* linked through page::lru */ static DEFINE_MUTEX (pram_mutex);/* serializes open/close */
@@ -521,3 +533,41 @@ size_t pram_read(struct pram_stream *ps, void *buf, size_t count) }return read_count; } + +/* +* Build the list of PRAM nodes. +*/ +static void __pram_reboot (void ) +{ +struct page *page; +struct pram_node *node; +unsigned long node_pfn =0 ; + +list_for_each_entry_reverse (page, &pram_nodes, lru) { + node =page_address (page); +if (WARN_ON (node->flags &PRAM_ACCMODE_MASK )) +continue ; + node->node_pfn = node_pfn; + node_pfn =page_to_pfn (page); + } +} + +static int pram_reboot (struct notifier_block *notifier, +unsigned long val,void *v) +{ +if (val !=SYS_RESTART ) +return NOTIFY_DONE ; +__pram_reboot (); +return NOTIFY_OK ; +} + +static struct notifier_block pram_reboot_notifier = { + .notifier_call = pram_reboot, +}; + +static int __init pram_init (void ) +{ +register_reboot_notifier (&pram_reboot_notifier); +return 0 ; +} +module_init (pram_init);
Signature · 7 lines
1.7.10.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC PATCH 6/13 mm: PRAM: introduce super block
In reply to Vladimir DavydovThe PRAM super block is the starting point for restoring persistent memory. If the kernel locates the super block at boot time, it will preserve the persistent memory structure from the previous kernel. To point the kernel to the location of the super block, one should pass its pfn via the 'pram' boot param. For that purpose, the pram super block pfn is exported via /sys/kernel/pram. If none is passed, persistent memory will not be preserved, and a new super block will be allocated.
The current patch introduces only super block handling. Memory preservation will be implemented later.
--- mm/pram.c | 94 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 92 insertions(+), 2 deletions(-)
mm/pram.c +92 -2
@@ -3,15 +3,18 @@#include <linux/highmem.h> #include <linux/init.h> #include <linux/kernel.h> +#include <linux/kobject.h> #include <linux/list.h> #include <linux/mm.h> #include <linux/module.h> #include <linux/mutex.h> #include <linux/notifier.h> +#include <linux/pfn.h> #include <linux/pram.h> #include <linux/reboot.h> #include <linux/sched.h> #include <linux/string.h> +#include <linux/sysfs.h> #include <linux/types.h> /*
@@ -66,12 +69,39 @@ struct pram_node {#define PRAM_ACCMODE_MASK 3/* +* The PRAM super block contains data needed to restore the persistent memory +* structure on boot. The pointer to it (pfn) should be passed via the 'pram' +* boot param if one wants to restore persistent data saved by the previously +* executing kernel. For that purpose the kernel exports the pfn via +* /sys/kernel/pram. If none is passed, persistent memory if any will not be +* preserved and a new clean page will be allocated for the super block. +* +* The structure occupies a memory page. +*/ +struct pram_super_block { +__u64 node_pfn;/* points to the first element of +* the node list */ +}; + +static unsigned long __initdata pram_sb_pfn; +static struct pram_super_block *pram_sb; + +/* * For convenience sake PRAM nodes are kept in an auxiliary doubly-linked list * connected through the lru field of the page struct. */ static LIST_HEAD (pram_nodes);/* linked through page::lru */ static DEFINE_MUTEX (pram_mutex);/* serializes open/close */ +/* +* The PRAM super block pfn, see above. +*/ +static int __init parse_pram_sb_pfn (char *arg) +{ +return kstrtoul (arg,16 , &pram_sb_pfn); +} +early_param ("pram" , parse_pram_sb_pfn); +static inline struct page *pram_alloc_page (gfp_t gfp_mask) {return alloc_page (gfp_mask);
@@ -161,6 +191,7 @@ static void pram_stream_init(struct pram_stream *ps,* Returns 0 on success, -errno on failure. * * Error values: +* %ENODEV: PRAM not available * %ENAMETOOLONG: name len >= PRAM_NAME_MAX * %ENOMEM: insufficient memory available * %EEXIST: node with specified name already exists
@@ -175,6 +206,9 @@ int pram_prepare_save(struct pram_stream *ps,struct pram_node *node;int err =0 ; +if (!pram_sb) +return -ENODEV ; +BUG_ON (type !=PRAM_PAGE_STREAM && type !=PRAM_BYTE_STREAM );
@@ -250,6 +284,7 @@ void pram_discard_save(struct pram_stream *ps)* Returns 0 on success, -errno on failure. * * Error values: +* %ENODEV: PRAM not available * %ENOENT: node with specified name does not exist * %EBUSY: save to required node has not finished yet * %EPERM: specified type conflicts with type of required node
@@ -262,6 +297,9 @@ int pram_prepare_load(struct pram_stream *ps,struct pram_node *node;int err =0 ; +if (!pram_sb) +return -ENODEV ; +mutex_lock (&pram_mutex); node =pram_find_node (name);if (!node) {
@@ -550,6 +588,7 @@ static void __pram_reboot(void) node->node_pfn = node_pfn; node_pfn =page_to_pfn (page); } + pram_sb->node_pfn = node_pfn; } static int pram_reboot(struct notifier_block *notifier,
@@ -557,7 +596,8 @@ static int pram_reboot(struct notifier_block *notifier, {if (val !=SYS_RESTART )return NOTIFY_DONE ; -__pram_reboot (); +if (pram_sb) +__pram_reboot ();return NOTIFY_OK ; }
@@ -565,9 +605,59 @@ static struct notifier_block pram_reboot_notifier = { .notifier_call = pram_reboot, }; +static ssize_t show_pram_sb_pfn (struct kobject *kobj, +struct kobj_attribute *attr,char *buf) +{ +unsigned long pfn = pram_sb ?PFN_DOWN (__pa (pram_sb)) :0 ; +return sprintf (buf,"%lx\n" , pfn); +} + +static struct kobj_attribute pram_sb_pfn_attr = +__ATTR (pram,0444 , show_pram_sb_pfn,NULL ); + +static struct attribute *pram_attrs[] = { + &pram_sb_pfn_attr.attr, +NULL , +}; + +static struct attribute_group pram_attr_group = { + .attrs = pram_attrs, +}; + +/* returns non-zero on success */ +static int __init pram_init_sb (void ) +{ +unsigned long pfn; +struct pram_node *node; + +if (!pram_sb) { +struct page *page; + + page =pram_alloc_page (GFP_KERNEL | __GFP_ZERO); +if (!page) { +pr_err ("PRAM: Failed to allocate super block\n" ); +return 0 ; + } + pram_sb =page_address (page); + } + +/* build auxiliary doubly-linked list of nodes connected through +* page::lru for convenience sake */ + pfn = pram_sb->node_pfn; +while (pfn) { + node =pfn_to_kaddr (pfn); +pram_insert_node (node); + pfn = node->node_pfn; + } +return 1 ; +} +static int __init pram_init (void ) { -register_reboot_notifier (&pram_reboot_notifier); +if (pram_init_sb ()) { +register_reboot_notifier (&pram_reboot_notifier); +sysfs_update_group (kernel_kobj, &pram_attr_group); + }return 0 ; }module_init (pram_init);
Signature · 7 lines
1.7.10.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC PATCH 7/13 mm: PRAM: preserve persistent memory at boot
In reply to Vladimir DavydovPersistent memory preservation is done by reserving memory pages belonging to PRAM at early boot so that they will not be recycled. If memory reservation fails for some reason (e.g. memory region is busy), persistent memory will be lost.
Currently, PRAM preservation is only implemented for x86.
--- arch/x86/kernel/setup.c | 2 + arch/x86/mm/init_32.c | 4 + arch/x86/mm/init_64.c | 4 + include/linux/pram.h | 8 ++ mm/Kconfig | 1 + mm/pram.c | 203 +++++++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 222 insertions(+)
arch/x86/kernel/setup.c +2 -0
@@ -69,6 +69,7 @@#include <linux/crash_dump.h> #include <linux/tboot.h> #include <linux/jiffies.h> +#include <linux/pram.h> #include <video/edid.h>
@@ -1127,6 +1128,7 @@ void __init setup_arch(char **cmdline_p)acpi_initrd_override ((void *)initrd_start, initrd_end - initrd_start);#endif +pram_reserve ();reserve_crashkernel ();vsmp_init ();
arch/x86/mm/init_32.c +4 -0
@@ -31,6 +31,7 @@#include <linux/initrd.h> #include <linux/cpumask.h> #include <linux/gfp.h> +#include <linux/pram.h> #include <asm/asm.h> #include <asm/bios_ebda.h>
@@ -779,6 +780,9 @@ void __init mem_init(void) after_bootmem =1 ; + totalram_pages += pram_reserved_pages; + reservedpages -= pram_reserved_pages; + codesize = (unsigned long ) &_etext - (unsigned long ) &_text; datasize = (unsigned long ) &_edata - (unsigned long ) &_etext; initsize = (unsigned long ) &__init_end - (unsigned long ) &__init_begin;
arch/x86/mm/init_64.c +4 -0
@@ -32,6 +32,7 @@#include <linux/memory_hotplug.h> #include <linux/nmi.h> #include <linux/gfp.h> +#include <linux/pram.h> #include <asm/processor.h> #include <asm/bios_ebda.h>
@@ -1077,6 +1078,9 @@ void __init mem_init(void) reservedpages = max_pfn - totalram_pages - absent_pages; after_bootmem =1 ; + totalram_pages += pram_reserved_pages; + reservedpages -= pram_reserved_pages; + codesize = (unsigned long ) &_etext - (unsigned long ) &_text; datasize = (unsigned long ) &_edata - (unsigned long ) &_etext; initsize = (unsigned long ) &__init_end - (unsigned long ) &__init_begin;
include/linux/pram.h +8 -0
@@ -47,4 +47,12 @@ extern ssize_t pram_write(struct pram_stream *ps, const void *buf, size_t count);extern size_t pram_read (struct pram_stream *ps,void *buf,size_t count); +#ifdef CONFIG_PRAM +extern unsigned long pram_reserved_pages; +extern void pram_reserve (void ); +#else +#define pram_reserved_pages 0UL +static inline void pram_reserve (void ) { } +#endif +#endif /* _LINUX_PRAM_H */
mm/Kconfig +1 -0
@@ -474,6 +474,7 @@ config FRONTSWAP config PRAM bool "Persistent over-kexec memory storage" + depends on X86 default n help This option adds the kernel API that enables saving memory pages of
mm/pram.c +203 -0
242 lines not on this page · raw message
Signature · 7 lines
1.7.10.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC PATCH 8/13 mm: PRAM: checksum saved data
In reply to Vladimir DavydovChecksum PRAM pages with crc32 to ensure persistent memory is not corrupted during reboot.
--- mm/Kconfig | 4 ++ mm/pram.c | 128 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 130 insertions(+), 2 deletions(-)
mm/Kconfig +4 -0
@@ -475,6 +475,10 @@ config FRONTSWAP config PRAM bool "Persistent over-kexec memory storage" depends on X86 + select CRC32 + select LIBCRC32C + select CRYPTO_CRC32C + select CRYPTO_CRC32C_INTEL default n help This option adds the kernel API that enables saving memory pages of
mm/pram.c +126 -2
211 lines not on this page · raw message
Signature · 7 lines
1.7.10.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC PATCH 9/13 mm: PRAM: ban pages that have been reserved at boot time
In reply to Vladimir DavydovObviously, not all memory ranges can be used for saving persistent over-kexec data, because some of them are reserved by the system core and various device drivers at boot time. If a memory range used for initialization of a particular device turns out to be busy because PRAM uses it for storing its data, the device driver initialization stage or even the whole system boot sequence may fail.
As a workaround the current implementation uses a rather dirty hack. It tracks all memory regions that have ever been reserved during the boot sequence and avoids using pages belonging to those regions for storing persistent data. Since the device configuration cannot change during kexec and the newly booted kernel is likely to have a similar boot-time device driver set, this hack should work in most cases.
--- arch/x86/mm/init_32.c | 1 + arch/x86/mm/init_64.c | 1 + include/linux/pram.h | 4 + mm/bootmem.c | 4 + mm/memblock.c | 7 +- mm/pram.c | 211 ++++++++++++++++++++++++++++++++++++++++++++++++- 6 files changed, 225 insertions(+), 3 deletions(-)
arch/x86/mm/init_32.c +1 -0
@@ -782,6 +782,7 @@ void __init mem_init(void) totalram_pages += pram_reserved_pages; reservedpages -= pram_reserved_pages; +pram_show_banned (); codesize = (unsigned long ) &_etext - (unsigned long ) &_text; datasize = (unsigned long ) &_edata - (unsigned long ) &_etext;
arch/x86/mm/init_64.c +1 -0
@@ -1080,6 +1080,7 @@ void __init mem_init(void) totalram_pages += pram_reserved_pages; reservedpages -= pram_reserved_pages; +pram_show_banned (); codesize = (unsigned long ) &_etext - (unsigned long ) &_text; datasize = (unsigned long ) &_edata - (unsigned long ) &_etext;
include/linux/pram.h +4 -0
@@ -50,9 +50,13 @@ extern size_t pram_read(struct pram_stream *ps, void *buf, size_t count);#ifdef CONFIG_PRAM extern unsigned long pram_reserved_pages;extern void pram_reserve (void ); +extern void pram_ban_region (unsigned long start,unsigned long end); +extern void pram_show_banned (void );#else #define pram_reserved_pages 0ULstatic inline void pram_reserve (void ) { } +static inline void pram_ban_region (unsigned long start,unsigned long end) { } +static inline void pram_show_banned (void ) { }#endif #endif /* _LINUX_PRAM_H */
mm/bootmem.c +4 -0
@@ -16,6 +16,7 @@#include <linux/kmemleak.h> #include <linux/range.h> #include <linux/memblock.h> +#include <linux/pram.h> #include <asm/bug.h> #include <asm/io.h>
@@ -328,6 +329,9 @@ static int __init __reserve(bootmem_data_t *bdata, unsigned long sidx,bdebug ("silent double reserve of PFN %lx\n" , idx + bdata->node_min_pfn); } + +pram_ban_region (sidx + bdata->node_min_pfn, + eidx + bdata->node_min_pfn -1 );return 0 ; }
mm/memblock.c +6 -1
@@ -19,6 +19,7 @@#include <linux/debugfs.h> #include <linux/seq_file.h> #include <linux/memblock.h> +#include <linux/pram.h> static struct memblock_region memblock_memory_init_regions[INIT_MEMBLOCK_REGIONS] __initdata_memblock; static struct memblock_region memblock_reserved_init_regions[INIT_MEMBLOCK_REGIONS] __initdata_memblock;
@@ -553,13 +554,17 @@ int __init_memblock memblock_free(phys_addr_t base, phys_addr_t size) int __init_memblock memblock_reserve(phys_addr_t base, phys_addr_t size) {struct memblock_type *_rgn = &memblock.reserved; +int err;memblock_dbg ("memblock_reserve: [%#016llx-%#016llx] %pF\n" , (unsigned long long )base, (unsigned long long )base + size, (void *)_RET_IP_); -return memblock_add_region (_rgn, base, size,MAX_NUMNODES ); + err =memblock_add_region (_rgn, base, size,MAX_NUMNODES ); +if (!err) +pram_ban_region (PFN_DOWN (base),PFN_UP (base + size) -1 ); +return err; }/**
mm/pram.c +209 -2
279 lines not on this page · raw message
Signature · 7 lines
1.7.10.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC PATCH 10/13 mm: PRAM: allow to ban arbitrary memory ranges
In reply to Vladimir DavydovBanning for PRAM memory ranges that have been reserved at boot time is not enough for avoiding all conflicts. The point is that kexec may load the new kernel code to some address range that have never been reserved possibly overwriting persistent data.
Fortunately, it is possible to specify a memory range kexec will load the new kernel code into. Thus, to avoid kexec-vs-PRAM conflicts, it is enough to disallow for PRAM some memory range large enough to load the new kernel and make kexec load the new kernel code into that range.
For that purpose, This patch adds ability to specify arbitrary banned ranges using the 'pram_banned' boot option.
--- mm/pram.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+)
mm/pram.c +45 -0
@@ -127,6 +127,15 @@ static bool __meminitdata pram_reservation_in_progress;* persistent data. Since the device configuration cannot change during kexec * and the newly booted kernel is likely to have a similar boot-time device * driver set, this hack should work in most cases. +* +* This solution has one exception. The point is that kexec may load the new +* kernel code to some address range that have never been reserved and thus +* banned for PRAM by the current kernel possibly overwriting persistent data. +* Fortunately, it is possible to specify an exact range kexec will load the +* new kernel code into. Thus, to avoid kexec-vs-PRAM conflicts, one should +* disallow for PRAM some memory range large enough to load the new kernel (see +* the 'pram_banned' boot param) and make kexec load the new kernel code into +* that range. */ /*
@@ -378,6 +387,42 @@ out: }/* +* A comma separated list of memory regions that PRAM is not allowed to use. +*/ +static int __init parse_pram_banned (char *arg) +{ +char *cur = arg, *tmp; +unsigned long long start, end; + +do { + start =memparse (cur, &tmp); +if (cur == tmp) { +pr_warning ("pram_banned: Memory value expected\n" ); +return -EINVAL ; + } + cur = tmp; +if (*cur !='-' ) { +pr_warning ("pram_banned: '-' expected\n" ); +return -EINVAL ; + } + cur++; + end =memparse (cur, &tmp); +if (cur == tmp) { +pr_warning ("pram_banned: Memory value expected\n" ); +return -EINVAL ; + } +if (end <= start) { +pr_warning ("pram_banned: end <= start\n" ); +return -EINVAL ; + } +pram_ban_region (PFN_DOWN (start),PFN_UP (end) -1 ); + }while (*cur++ ==',' ); + +return 0 ; +} +early_param ("pram_banned" , parse_pram_banned); + +/* * Bans pfn range [start..end] (inclusive) for PRAM. */ void __meminit pram_ban_region (unsigned long start,unsigned long end)
Signature · 7 lines
1.7.10.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC PATCH 11/13 mm: PRAM: allow to free persistent memory from userspace
In reply to Vladimir DavydovTo free all space utilized for persistent memory, one can write 0 to /sys/kernel/pram. This will destroy all PRAM nodes that are not currently being read or written.
--- mm/pram.c | 39 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-)
mm/pram.c +38 -1
@@ -697,6 +697,32 @@ static void pram_truncate_node(struct pram_node *node) } +/* +* Free all nodes that are not under operation. +*/ +static void pram_truncate (void ) +{ +struct page *page, *tmp; +struct pram_node *node; +LIST_HEAD (dispose); + +mutex_lock (&pram_mutex); +list_for_each_entry_safe (page, tmp, &pram_nodes, lru) { + node =page_address (page); +if (!(node->flags &PRAM_ACCMODE_MASK )) +list_move (&page->lru, &dispose); + } +mutex_unlock (&pram_mutex); + +while (!list_empty (&dispose)) { + page = list_first_entry(&dispose, struct page, lru); +list_del (&page->lru); + node =page_address (page); +pram_truncate_node (node); +pram_free_page (node); + } +} +static void pram_stream_init (struct pram_stream *ps,struct pram_node *node,gfp_t gfp_mask) {
@@ -1189,8 +1215,19 @@ static ssize_t show_pram_sb_pfn(struct kobject *kobj,return sprintf (buf,"%lx\n" , pfn); } +static ssize_t store_pram_sb_pfn (struct kobject *kobj, +struct kobj_attribute *attr,const char *buf,size_t count) +{ +int val; + +if (kstrtoint (buf,0 , &val) || val) +return -EINVAL ; +pram_truncate (); +return count; +} +static struct kobj_attribute pram_sb_pfn_attr = -__ATTR (pram,0444 , show_pram_sb_pfn,NULL ); +__ATTR (pram,0644 , show_pram_sb_pfn, store_pram_sb_pfn);static struct attribute *pram_attrs[] = { &pram_sb_pfn_attr.attr,
Signature · 7 lines
1.7.10.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC PATCH 12/13 mm: shmem: introduce shmem_insert_page
In reply to Vladimir DavydovThe function inserts a memory page to a shmem file under an arbitrary offset. If there is something at the specified offset (page or swap), the function fails.
The function will be sued by the next patch.
--- include/linux/shmem_fs.h | 3 ++ mm/shmem.c | 68 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+)
include/linux/shmem_fs.h +3 -0
@@ -62,4 +62,7 @@ static inline struct page *shmem_read_mapping_page( mapping_gfp_mask(mapping)); } +extern int shmem_insert_page (struct inode *inode, +pgoff_t index,struct page *page,bool on_lru); +#endif
mm/shmem.c +68 -0
74 lines not on this page · raw message
Signature · 7 lines
1.7.10.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC PATCH 13/13 mm: shmem: enable saving to PRAM
In reply to Vladimir DavydovThis patch illustrates how PRAM API can be used for making tmpfs 'persistent'. It adds 'pram=' option to tmpfs, which specifies the PRAM node to load/save FS tree from/to.
If the option is passed on mount, shmem will look for the corresponding PRAM node and load the FS tree from it. On the subsequent unmount, it will save FS tree to that PRAM node.
A typical usage scenario looks like:
# mount -t tmpfs -o pram=mytmpfs none /mnt # echo something > /mnt/smth # umount /mnt <possibly kexec> # mount -t tmpfs -o pram=mytmpfs none /mnt # cat /mnt/smth
Each FS tree is saved into two PRAM nodes, one acting as a byte stream and the other acting as a page stream. The byte stream is used for saving files metadata (name, permissions, etc) and data page offsets while the page stream accommodates file content pages.
Current implementation serves for demonstration purposes and so is quite
simplified: it supports only regular files in the root directory without multiple hard links, and it does not save swapped out files aborting if any. However, it can be elaborated to fully support tmpfs.
--- include/linux/shmem_fs.h | 26 ++++ mm/Makefile | 2 +- mm/shmem.c | 29 +++- mm/shmem_pram.c | 378 ++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 430 insertions(+), 5 deletions(-) create mode 100644 mm/shmem_pram.c
include/linux/shmem_fs.h +26 -0
@@ -23,6 +23,11 @@ struct shmem_inode_info {struct inode vfs_inode; }; +#define SHMEM_PRAM_NAME_MAX 128 +struct shmem_pram_info { +char name[SHMEM_PRAM_NAME_MAX ]; +}; +struct shmem_sb_info {unsigned long max_blocks;/* How many blocks are allowed */ struct percpu_counter used_blocks;/* How many are allocated */
@@ -33,6 +38,7 @@ struct shmem_sb_info {kgid_t gid;/* Mount gid for root directory */ umode_t mode;/* Mount mode for root directory */ struct mempolicy *mpol;/* default memory policy for mappings */ +struct shmem_pram_info *pram; };static inline struct shmem_inode_info *SHMEM_I (struct inode *inode)
@@ -62,7 +68,27 @@ static inline struct page *shmem_read_mapping_page( mapping_gfp_mask(mapping)); } +struct pagevec ; +extern int shmem_insert_page (struct inode *inode,pgoff_t index,struct page *page,bool on_lru); +extern unsigned shmem_find_get_pages_and_swap (struct address_space *mapping, +pgoff_t start,unsigned int nr_pages, +struct page **pages,pgoff_t *indices); +extern void shmem_deswap_pagevec (struct pagevec *pvec); + +#ifdef CONFIG_PRAM +extern int shmem_parse_pram (const char *str,struct shmem_pram_info **pram); +extern void shmem_show_pram (struct seq_file *seq,struct shmem_pram_info *pram); +extern void shmem_save_pram (struct super_block *sb); +extern void shmem_load_pram (struct super_block *sb); +#else +static inline int shmem_parse_pram (const char *str, +struct shmem_pram_info **pram) {return 1 ; } +static inline void shmem_show_pram (struct seq_file *seq, +struct shmem_pram_info *pram) { } +static inline void shmem_save_pram (struct super_block *sb) { } +static inline void shmem_load_pram (struct super_block *sb) { } +#endif #endif
mm/Makefile +1 -1
@@ -58,4 +58,4 @@ obj-$(CONFIG_DEBUG_KMEMLEAK) += kmemleak.o obj-$(CONFIG_DEBUG_KMEMLEAK_TEST) += kmemleak-test.o obj-$(CONFIG_CLEANCACHE) += cleancache.o obj-$(CONFIG_MEMORY_ISOLATION) += page_isolation.o -obj-$(CONFIG_PRAM) += pram.o +obj-$(CONFIG_PRAM) += pram.o shmem_pram.o
mm/shmem.c +25 -4
@@ -399,7 +399,7 @@ out:/* * Like find_get_pages, but collecting swap entries as well as pages. */ -static unsigned shmem_find_get_pages_and_swap (struct address_space *mapping, +unsigned shmem_find_get_pages_and_swap (struct address_space *mapping,pgoff_t start,unsigned int nr_pages,struct page **pages,pgoff_t *indices) {
@@ -465,7 +465,7 @@ static int shmem_free_swap(struct address_space *mapping,/* * Pagevec may contain swap entries, so shuffle up pages before releasing. */ -static void shmem_deswap_pagevec (struct pagevec *pvec) +void shmem_deswap_pagevec (struct pagevec *pvec) {int i, j;
@@ -2535,6 +2535,10 @@ static int shmem_parse_options(char *options, struct shmem_sb_info *sbinfo, mpol =NULL ;if (mpol_parse_str (value, &mpol)) goto bad_val; + }else if (!strcmp (this_char,"pram" )) { +kfree (sbinfo->pram); +if (shmem_parse_pram (value, &sbinfo->pram)) + goto bad_val; }else {printk (KERN_ERR "tmpfs: Bad mount option %s\n" , this_char);
@@ -2561,6 +2565,7 @@ static int shmem_remount_fs(struct super_block *sb, int *flags, char *data)int error = -EINVAL ; config.mpol =NULL ; + config.pram =NULL ;if (shmem_parse_options (data, &config, true))return error;
@@ -2592,6 +2597,9 @@ static int shmem_remount_fs(struct super_block *sb, int *flags, char *data)mpol_put (sbinfo->mpol); sbinfo->mpol = config.mpol;/* transfers initial ref */ } + +kfree (sbinfo->pram); + sbinfo->pram = config.pram; out:spin_unlock (&sbinfo->stat_lock);return error;
@@ -2615,6 +2623,7 @@ static int shmem_show_options(struct seq_file *seq, struct dentry *root)seq_printf (seq,",gid=%u" ,from_kgid_munged (&init_user_ns, sbinfo->gid));shmem_show_mpol (seq, sbinfo->mpol); +shmem_show_pram (seq, sbinfo->pram); return 0; }#endif /* CONFIG_TMPFS */
@@ -2625,6 +2634,7 @@ static void shmem_put_super(struct super_block *sb)percpu_counter_destroy (&sbinfo->used_blocks);mpol_put (sbinfo->mpol); +kfree (sbinfo->pram);kfree (sbinfo); sb->s_fs_info =NULL ; }
@@ -2838,14 +2848,25 @@ static const struct vm_operations_struct shmem_vm_ops = {static struct dentry *shmem_mount (struct file_system_type *fs_type,int flags,const char *dev_name,void *data) { -return mount_nodev (fs_type, flags, data, shmem_fill_super); +struct dentry *root; + + root =mount_nodev (fs_type, flags, data, shmem_fill_super); +if (!IS_ERR (root)) +shmem_load_pram (root->d_sb); +return root; +} + +static void shmem_kill_sb (struct super_block *sb) +{ +shmem_save_pram (sb); +kill_litter_super (sb); }static struct file_system_type shmem_fs_type = { .owner =THIS_MODULE , .name ="tmpfs" , .mount = shmem_mount, - .kill_sb = kill_litter_super, + .kill_sb = shmem_kill_sb, .fs_flags =FS_USERNS_MOUNT , };
mm/shmem_pram.c new file +378 -0
378 lines not on this page · raw message
Signature · 7 lines
1.7.10.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>