Live Patching Function on 2.6.11.7

The patch was over 50k, so I separate it to each architecture and in line..

This patch add function called "Live patching" which is defined on OSDL's carrier grade linux requiremnt definition to linux 2.6.11.7 kernel. The live patching allows process to patch on-line (without restarting process) on i386 and x86_64 architectures, by overwriting jump assembly code on entry point of functions which you want to fix, to patched functions.
The live patching function is very common on high-availability system such as carrier system, and this patch realize it also on linux. (Patch & process restart time is very critical on such high-availability system, live patch allows you to milliseconds order process stopping time to apply new patch.)

The basis is below:
1. Live patch command loads the patch modules to target process's memory area,
2. Live patch command resolve patch symbol.
3. Live patch command overwrite jump code to the entry point of function which you want to fix, to the patch module's symbol.

Kernel patch and user mode tools are required, and both of them are available at http://pannus.sourceforge.net
Please take a look and give us comments!

This patch add following system calls and function.
o mmap3: maps patch to target process's memory area with security check.
o accesspvm: access(read/write) target process's memory area.
o init_pend: initialization of live patch sequence on target process.
o rt_handlereturn: run initialize root of each patch (same as signal handler).
o check_init: check that the initialization is finished or not.
o munmap3: unmap patch from target process's memory area.

Takashi Ikebe
NTT Network Service Systems Laboratories
9-11, Midori-Cho 3-Chome Musashino-Shi,
Tokyo 180-8585 Japan
Tel : +81 422 59 4246, Fax : +81 422 60 4012
e-mail : [email protected]

diff -urpN linux-2.6.11.7-vanilla/arch/x86_64/kernel/Makefile linux-2.6.11.7-pannus-x86_64/arch/x86_64/kernel/Makefile
@@ -7,7 +7,8 @@ EXTRA_AFLAGS	:= -traditional
 obj-y	:= process.o semaphore.o signal.o entry.o traps.o irq.o \
 		ptrace.o time.o ioport.o ldt.o setup.o i8259.o sys_x86_64.o \
 		x8664_ksyms.o i387.o syscall.o vsyscall.o \
-		setup64.o bootflag.o e820.o reboot.o quirks.o
+		setup64.o bootflag.o e820.o reboot.o quirks.o \
+		accesspvm.o exechandle.o
 
 obj-$(CONFIG_X86_MCE)         += mce.o
 obj-$(CONFIG_X86_MCE_INTEL)	+= mce_intel.o

diff -urpN linux-2.6.11.7-vanilla/arch/x86_64/kernel/asm-offsets.c linux-2.6.11.7-pannus-x86_64/arch/x86_64/kernel/asm-offsets.c
@@ -33,6 +33,7 @@ int main(void)
 	ENTRY(flags);
 	ENTRY(addr_limit);
 	ENTRY(preempt_count);
+	ENTRY(inipending);
 	BLANK();
 #undef ENTRY
 #define ENTRY(entry) DEFINE(pda_ ## entry, offsetof(struct x8664_pda, entry))

diff -urpN linux-2.6.11.7-vanilla/include/asm-x86_64/exechandle.h linux-2.6.11.7-pannus-x86_64/include/asm-x86_64/exechandle.h
@@ -0,0 +1,21 @@
+#ifndef _ASM_X86_64_EXECHANDLE_H
+#define _ASM_X86_64_EXECHANDLE_H
+
+#include <asm/types.h>
+#include <asm/signal.h>
+
+
+struct initaction
+{
+	void (*inithandler)(int);
+	void (*restorer)(void);
+};
+
+struct k_initaction
+{
+        struct initaction ia;
+};
+
+void do_init(struct pt_regs *regs, sigset_t *oldset);
+
+#endif

diff -urpN linux-2.6.11.7-vanilla/include/asm-x86_64/thread_info.h linux-2.6.11.7-pannus-x86_64/include/asm-x86_64/thread_info.h
@@ -33,6 +33,9 @@ struct thread_info {
 
 	mm_segment_t		addr_limit;	
 	struct restart_block    restart_block;
+
+	__u32			inipending;	/* pending flags for live patch */
+	__u32			inifinish;	/* finish flags for live patch */
 };
 #endif
 

diff -urpN linux-2.6.11.7-vanilla/include/asm-x86_64/unistd.h linux-2.6.11.7-pannus-x86_64/include/asm-x86_64/unistd.h
@@ -563,8 +563,21 @@ __SYSCALL(__NR_add_key, sys_add_key)
 __SYSCALL(__NR_request_key, sys_request_key)
 #define __NR_keyctl		250
 __SYSCALL(__NR_keyctl, sys_keyctl)
+#define __NR_mmap3	251
+__SYSCALL(__NR_mmap3, sys_mmap3)
+#define __NR_accesspvm	252
+__SYSCALL(__NR_accesspvm, sys_accesspvm)
+#define __NR_init_pend	253
+__SYSCALL(__NR_init_pend, sys_init_pend)
+#define __NR_rt_handlereturn	254
+__SYSCALL(__NR_rt_handlereturn, stub_rt_handlereturn)
+#define __NR_check_init	255
+__SYSCALL(__NR_check_init, sys_check_init)
+#define __NR_munmap3	256
+__SYSCALL(__NR_munmap3, sys_munmap3)
 
-#define __NR_syscall_max __NR_keyctl
+
+#define __NR_syscall_max __NR_munmap3
 #ifndef __NO_STUBS
 
 /* user-visible error numbers are in the range -1 - -4095 */

@@ -751,6 +764,7 @@ static inline pid_t waitpid(int pid, int
 extern long sys_mmap(unsigned long addr, unsigned long len,
 			unsigned long prot, unsigned long flags,
 			unsigned long fd, unsigned long off);
+extern long sys_mmap3(void *);
 
 extern int sys_modify_ldt(int func, void *ptr, unsigned long bytecount);
 

diff -urpN linux-2.6.11.7-vanilla/include/linux/capability.h linux-2.6.11.7-pannus-x86_64/include/linux/capability.h
@@ -288,6 +288,10 @@ typedef __u32 kernel_cap_t;
 
 #define CAP_AUDIT_CONTROL    30
 
+/* Allow use of memory access system calls for Live Patching */
+
+#define CAP_SYS_PANNUS       31
+
 #ifdef __KERNEL__
 /* 
  * Bounding set

diff -urpN linux-2.6.11.7-vanilla/include/linux/mm.h linux-2.6.11.7-pannus-x86_64/include/linux/mm.h
@@ -614,6 +614,7 @@ extern int install_page(struct mm_struct
 extern int install_file_pte(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, unsigned long pgoff, pgprot_t prot);
 extern int handle_mm_fault(struct mm_struct *mm,struct vm_area_struct *vma, unsigned long address, int write_access);
 extern int make_pages_present(unsigned long addr, unsigned long end);
+extern int make_pages_present2(unsigned long addr, unsigned long end, struct task_struct *tsk);
 extern int access_process_vm(struct task_struct *tsk, unsigned long addr, void *buf, int len, int write);
 void install_arg_page(struct vm_area_struct *, struct page *, unsigned long);
 

@@ -730,10 +731,16 @@ extern void exit_mmap(struct mm_struct *
 
 extern unsigned long get_unmapped_area(struct file *, unsigned long, unsigned long, unsigned long, unsigned long);
 
+extern unsigned long get_unmapped_area2(struct file *, unsigned long, unsigned long, unsigned long, unsigned long, struct task_struct *);
+
 extern unsigned long do_mmap_pgoff(struct file *file, unsigned long addr,
 	unsigned long len, unsigned long prot,
 	unsigned long flag, unsigned long pgoff);
 
+extern unsigned long do_mmap_pgoff2(struct file *file, unsigned long addr,
+	unsigned long len, unsigned long prot,
+	unsigned long flag, unsigned long pgoff, struct task_struct *);
+
 static inline unsigned long do_mmap(struct file *file, unsigned long addr,
 	unsigned long len, unsigned long prot,
 	unsigned long flag, unsigned long offset)

diff -urpN linux-2.6.11.7-vanilla/include/linux/mman.h linux-2.6.11.7-pannus-x86_64/include/linux/mman.h
@@ -64,4 +64,17 @@ calc_vm_flag_bits(unsigned long flags)
 	       _calc_vm_trans(flags, MAP_LOCKED,     VM_LOCKED    );
 }
 
+/*
+ * Struct of argument to pass the mmap3 system call.
+ */ 
+typedef struct _mmap3_arg_struct {
+	unsigned long addr; /* address where file is loaded */
+	unsigned long len; /* length of data to be maped */
+	unsigned long prot; /* permission of the memory where the file is mapped */
+	unsigned long flags; /* flag of mapped memory */
+	unsigned long fd; /* file descriptor of data to be mapped */
+	unsigned long pgoff; /* page offset of data to be mapped */
+	unsigned long pid; /* process ID */
+} mmap3_arg_struct_t;
+
 #endif /* _LINUX_MMAN_H */

diff -urpN linux-2.6.11.7-vanilla/include/linux/sched.h linux-2.6.11.7-pannus-x86_64/include/linux/sched.h
@@ -21,6 +21,7 @@
 #include <asm/ptrace.h>
 #include <asm/mmu.h>
 #include <asm/cputime.h>
+#include <asm/exechandle.h>
 
 #include <linux/smp.h>
 #include <linux/sem.h>

@@ -197,9 +198,19 @@ extern unsigned long
 arch_get_unmapped_area(struct file *, unsigned long, unsigned long,
 		       unsigned long, unsigned long);
 extern unsigned long
+arch_get_unmapped_area2(struct file *, unsigned long, unsigned long,
+		       unsigned long, unsigned long, struct task_struct *);
+
+extern unsigned long
 arch_get_unmapped_area_topdown(struct file *filp, unsigned long addr,
 			  unsigned long len, unsigned long pgoff,
 			  unsigned long flags);
+
+extern unsigned long
+arch_get_unmapped_area_topdown2(struct file *filp, unsigned long addr,
+			  unsigned long len, unsigned long pgoff,
+			  unsigned long flags, struct task_struct *);
+
 extern void arch_unmap_area(struct vm_area_struct *area);
 extern void arch_unmap_area_topdown(struct vm_area_struct *area);
 

@@ -211,6 +222,11 @@ struct mm_struct {
 	unsigned long (*get_unmapped_area) (struct file *filp,
 				unsigned long addr, unsigned long len,
 				unsigned long pgoff, unsigned long flags);
+	unsigned long (*get_unmapped_area2) (struct file *filp,
+				unsigned long addr, unsigned long len,
+				unsigned long pgoff, unsigned long flags, 
+				struct task_struct * tsk);
+
 	void (*unmap_area) (struct vm_area_struct *area);
 	unsigned long mmap_base;		/* base of mmap area */
 	unsigned long free_area_cache;		/* first hole */

@@ -685,6 +701,7 @@ struct task_struct {
   	struct mempolicy *mempolicy;
 	short il_next;
 #endif
+	struct k_initaction k_ia; /*Inialization info for live patch */
 };
 
 static inline pid_t process_group(struct task_struct *tsk)

@@ -1173,6 +1190,7 @@ static inline void arch_pick_mmap_layout
 {
 	mm->mmap_base = TASK_UNMAPPED_BASE;
 	mm->get_unmapped_area = arch_get_unmapped_area;
+	mm->get_unmapped_area2 = arch_get_unmapped_area2;
 	mm->unmap_area = arch_unmap_area;
 }
 #endif

diff -urpN linux-2.6.11.7-vanilla/kernel/fork.c linux-2.6.11.7-pannus-x86_64/kernel/fork.c
@@ -2,6 +2,7 @@
  *  linux/kernel/fork.c
  *
  *  Copyright (C) 1991, 1992  Linus Torvalds
+ *  Copyright (C)  2004-2005 NTT Corporation
  */
 
 /*

@@ -412,6 +413,12 @@ void mm_release(struct task_struct *tsk,
 		u32 __user * tidptr = tsk->clear_child_tid;
 		tsk->clear_child_tid = NULL;
 
+		/* initialize flag and information for live patch */
+		tsk->thread_info->inipending=0;
+		tsk->thread_info->inifinish=0;
+		tsk->k_ia.ia.inithandler=NULL;
+		tsk->k_ia.ia.restorer=NULL;
+
 		/*
 		 * We don't check the error code - if userspace has
 		 * not set up a proper pointer then tough luck.

diff -urpN linux-2.6.11.7-vanilla/mm/memory.c linux-2.6.11.7-pannus-x86_64/mm/memory.c
@@ -2209,6 +2209,27 @@ int make_pages_present(unsigned long add
 	return ret == len ? 0 : -1;
 }
 
+int make_pages_present2(unsigned long addr, unsigned long end, struct task_struct *tsk)
+{
+	int ret, len, write;
+	struct vm_area_struct * vma;
+
+	vma = find_vma(tsk->mm, addr);
+	if (!vma)
+		return -1;
+	write = (vma->vm_flags & VM_WRITE) != 0;
+	if (addr >= end)
+		BUG();
+	if (end > vma->vm_end)
+		BUG();
+	len = (end+PAGE_SIZE-1)/PAGE_SIZE-addr/PAGE_SIZE;
+	ret = get_user_pages(tsk, tsk->mm, addr,
+			len, write, 0, NULL, NULL);
+	if (ret < 0)
+		return ret;
+	return ret == len ? 0 : -1;
+}
+
 /* 
  * Map a vmalloc()-space virtual address to the physical page.
  */

I;m curious as to what people decided this was a necessary requirement.

Why can't you use ptrace for all this?

Hello,

The requirements are comes from Network Equipment Providers, Telecom Carriers, and Hardware Vendors, You can see the attendee from below link;
http://groups.osdl.org/world_map/full_roster/

GDB based approach seems not fit to our requirements. GDB(ptrace) based functions are basically need to be done when target process is stopping. In addition to that current PTRACE_PEEK/POKE* allows us to copy only a *word* size...
From our experience, sometimes patches became to dozens to hundreds at one patching, and in this case GDB based approach cause target process's availability descent.

Takashi Ikebe
NTT Network Service Systems Laboratories
9-11, Midori-Cho 3-Chome Musashino-Shi,
Tokyo 180-8585 Japan
Tel : +81 422 59 4246, Fax : +81 422 60 4012
e-mail : [email protected]

While true, this is easily fixable. There is even an interface precedent on OpenBSD (and possibly other platforms as well).

Daniel Jacobowitz
CodeSourcery, LLC

If we're going to be stealing ideas for debugging interfaces from other operating systems, could we steal from Solaris instead of anything ptrace-based?

Nicholas Miell <[email protected]>

Indeed, look at the systr_pmem_read() and systr_pmem_write() functions:

http://www.xmailserver.org/sysctr.html

- Davide

systr_pmem_read() and systr_pmem_write() just calls ptrace PTRACE_PEEKTEXT/DATA repeatedly.... In this case we need to *stop* target process whenever patch modules is loading.... It cause target process availability worse.

Takashi Ikebe
NTT Network Service Systems Laboratories
9-11, Midori-Cho 3-Chome Musashino-Shi,
Tokyo 180-8585 Japan
Tel : +81 422 59 4246, Fax : +81 422 60 4012
e-mail : [email protected]

Hi Davide

I found in my tests that there is no need to have a f_ep_lock spinlock attached to each struct file, using 8 bytes on 64bits platforms. The lock is hold for a very short time period and can be global, with almost no change in performance for applications using epoll, and a gain for all others.

Thank you
Eric Dumazet

[PATCH] eventpoll : Suppress a short lived lock from struct file

@@ -179,6 +179,8 @@
 	spinlock_t lock;
 };
 
+static DEFINE_SPINLOCK(f_ep_lock);
+
 /*
  * This structure is stored inside the "private_data" member of the file
  * structure and rapresent the main data sructure for the eventpoll

@@ -426,7 +428,6 @@
 {
 
 	INIT_LIST_HEAD(&file->f_ep_links);
-	spin_lock_init(&file->f_ep_lock);
 }
 
 

@@ -967,9 +968,9 @@
 		goto eexit_2;
 
 	/* Add the current item to the list of active epoll hook for this file */
-	spin_lock(&tfile->f_ep_lock);
+	spin_lock(&f_ep_lock);
 	list_add_tail(&epi->fllink, &tfile->f_ep_links);
-	spin_unlock(&tfile->f_ep_lock);
+	spin_unlock(&f_ep_lock);
 
 	/* We have to drop the new item inside our item list to keep track of it */
 	write_lock_irqsave(&ep->lock, flags);

@@ -1160,7 +1161,6 @@
 {
 	int error;
 	unsigned long flags;
-	struct file *file = epi->ffd.file;
 
 	/*
 	 * Removes poll wait queue hooks. We _have_ to do this without holding

@@ -1173,10 +1173,10 @@
 	ep_unregister_pollwait(ep, epi);
 
 	/* Remove the current item from the list of epoll hooks */
-	spin_lock(&file->f_ep_lock);
+	spin_lock(&f_ep_lock);
 	if (EP_IS_LINKED(&epi->fllink))
 		EP_LIST_DEL(&epi->fllink);
-	spin_unlock(&file->f_ep_lock);
+	spin_unlock(&f_ep_lock);
 
 	/* We need to acquire the write IRQ lock before calling ep_unlink() */
 	write_lock_irqsave(&ep->lock, flags);

@@ -597,7 +597,6 @@
 #ifdef CONFIG_EPOLL
 	/* Used by fs/eventpoll.c to link all the hooks to this file */
 	struct list_head	f_ep_links;
-	spinlock_t		f_ep_lock;
 #endif /* #ifdef CONFIG_EPOLL */
 	struct address_space	*f_mapping;
 };

Have you tested the impact of this change on big SMP/NUMA machines? I hate to see an Altrix crashing to its knees :-)

Peter Zijlstra <[email protected]>

I tested on a small NUMA machine (2 nodes), with a epoll enabled application, that use around 100 epoll ctl per second.

Of course, one may write a special benchmark on a BIG SMP/NUMA machine that
defeat these patch, using thousands of epoll ctl per second, but, a normal (well written ?)
epoll application doesnt constantly add/remove epoll ctl.

Should we waste 8 bytes per 'struct file' for a very unlikely micro benchmark ?

Eric

Eric, I can't really say I like this one. Not at least after extensive tests run on top of it. You are asking to add a bottleneck to save 8 bytes on an entity that taken alone in more than 120 bytes. Consider that when you have a "struct file" allocated, the cost on the system is not only the struct itself, but all the allocations associated with it. For example, if you consider that a case where you might feel a "struct file" pressure is when you have hundreds of thousands of network connections, the 8 bytes saved compared to all the buffers associated with those sockets boils down to basically nothing.

- Davide

fair enough :)

Well, the filp_cachep slab is created with SLAB_HWCACHE_ALIGN, enforcing a alignment of 64 bytes or even 128 bytes.

So it can be usefull to let the size of struct file goes from 0x84 to 0x80, because we can gain 64 or 128 bytes per file (0x80 bytes really allocated instead of 0xc0 or even 0x100 on Pentium 4).

In my case, I use other patches outside the scope of eventpoll (like declaring f_security only #ifdef CONFIG_SECURITY_SELINUX), and really gain 128 bytes of low memory per file. It reduces cache pressure for a given workload, and reduce lowmem pressure.

Before :

# grep filp /proc/slabinfo
filp               66633  66750    256   15    1 : tunables  120   60    8 : slabdata   4450   4450     60

After :

# grep filp /proc/slabinfo
filp               82712  82987    128   31    1 : tunables  120   60    8 : slabdata   2677   2677     20

It may appears to you as a penalty, but at least for me it is a noticeable gain.

Another candidate to "file struct" size reduction is the big struct file_ra_state that is included in all files, even sockets that dont use it, but that's a different story :)

Eric

Some platforms even support the necessary PTRADE_{READ,WRITE}DATA operations already, sparc is one such platform.

i don't really buy that it can't be done or you complex patches are necessary to be honest --- and there are various alternative APIs that could help as others have pointed out

could you perhaps explain some *real* *world* applications/systems where this is necessary and why existing APIs won't work with them perhaps?

solving a real-world problem is much more interesting to listen to that filling in a check-box on a (somewhat dubious) specification

In the telecom space it's quite common to want to modify multiple running binaries with as little downtime as possible. (Beyond a threshold it becomes FCC-reportable in the US, and everyone wants to avoid that...)

Our old proprietary OS had explicit support for replacing running binary code on the fly, so customers have gotten used to the ability. Now they want equivalent functionality with our linux-based stuff.

We've done some proprietary stuff (ie. pre-OSDL CGL) in this area, but it was apparently a real pain and was quite restrictive on the application writers. (I was not involved with that portion of the project.)

For general application support I suspect some kernel support will be required. Whether this is the way to go or whether it can be done using existing mechanisms, I'm not knowledgeable enough to comment.

Chris

OK

That's beside the point.

*Why* do they need this is what I asked. A sensible real world example would be useful.

I used to work in telco space, we had some such systems and similar things. Some from Nortel even.

None of the things I saw did anything that I can image really need a complicated kernel patch for.

In fact, I'm not convinced *any* of these uses really needed live-patching at all.

I would just like some examples of real-world needs and an explanation of why it's needed. Not handy-waving.

I raised a thread like this about 1 year ago. I was asking for it from the point of view of a Telco. After some discussions on this list, I came to agree with the posts on the list by other people that the feature is not needed. At least certainly not needed in the Telco space. 99.999% uptime is much better acheived with the use of clustering, rather than trying to upgrade software in a Live situation. In a clustered environment, one offloads all the tasks from machine A and spread them across the cluster. Once the machine A is not doing any work at all, you can upgrade, reboot, whatever you like, and then add it back to the cluster. This approach is much less risky than live module updates. If the equipment is not clustered, it will at least be 2 to 1 redundent, so you just upgrade the redundent device, manually force a fail over, and then upgrade the other device. Again, no live update required.

I can only think of one other system that might benefit from live updates, and that is set top boxes, so bugs can be fixed without the user knowing. This also can be worked around by downloading the bug fixes and only installing the bugs fixes when the user is not viewing the TV. E.g. When the box has been placed in standby by the user.

James

hardly mission critical and usually don't have the resources to do complicate things

much better to rexec/reboot as needed

again it doesn't need live patching (satellite and cable boxes update themselves routinely, some certainly do need to periodically reboot to do this and it's apparently not a problem)

For example, please think about telephone switching software. The software does not allow to stops over 100 milliseconds at worst case.(this depends on how many customers are using, the more customers are using, the allowable stopping time goes to shorter.)
And the system is required over 99.999% availability, so typically, the system is constructed as ACT-SBY high-availability clustering model. In the case of critical error the system takeover the service process to SBY node.
Not to descent the service availability, software fix due to bug, should not stop the service, and live patching is very historical function in telecoms world.
Every carrier, NEPs(Network Equipment Provider) provide/use this function to keep network service(such as telephone) available. This function is very essential whenever the carrier use the linux as center of it's system.

Therefore the live patching function should not stop the target process(service process) as possible as. the more times we stop the target process, the service goes unavailable...

Takashi Ikebe
NTT Network Service Systems Laboratories
9-11, Midori-Cho 3-Chome Musashino-Shi,
Tokyo 180-8585 Japan
Tel : +81 422 59 4246, Fax : +81 422 60 4012
e-mail : [email protected]

Out of interest, how do you ensure the process doesn't stop for that long right now? Linux doesn't guarantee you'll get scheduled (strictly speaking) in n milliseconds usually.

Lots of really complicated and unnecessary things are common in the telecoms world.

For the example you gave I can think of several ways to migrate data to a new process (if need be) in a timely manner without interruption. None of these *require* live patching.

How does this *require* live patching?

Those are just marketing words.

Love patching seems like a very complicated thing to get right and it could potentially blow up.

I'm guessing any suggestion of fixing the applications behavior would be lost with some argument along the lines of: "this application was written in 1824 by Ada Lovelace using pre-Roswell Alien Technology and was certified NEBS compliant by the Deli Lama and god herself, so clearly we can't touch a single line of it" or similar right?

I'm sorry, I can not catch the point well,
but what I want to say is, we do not want stop the service due to bug fix.

As you said, if we can migrate the data to new process without stopping service, it is OK, but the real applications need to takeover data very much(sometimes it's over gigabyte....depends on service, and causes service disruption...). So, live patching seems reasonable to us. And unfortunately the APIs which people suggested are all based on ptrace system calls. as you know, ptrace based data read/write needs to stop the target process during read/write. We would like to minimize target process stopping time.

well, it's possibly, but the same problems occur on gdb. I think this depends on user's manner...

I briefly describe the way of live patching below;
1. Load the patch modules with pannus -l command.
- load the patch module with first memory mapping system call(ptrace PEEKDATA can be same work, but it needs to stop target process..)
- search patch module's initialize area and execute them with execinit.c(similler to signal handler)
- target process exec initialization.
2. Activate the patch modules with pannus -a command.
- stop the target process and check current instruction not to conflict.
- if it is not conflict, overwrite the jump assembly to function's entrypoiny where you want to fix, to patch module's one.
- restart the process.

Will this be answer??

Takashi Ikebe
NTT Network Service Systems Laboratories
9-11, Midori-Cho 3-Chome Musashino-Shi,
Tokyo 180-8585 Japan
Tel : +81 422 59 4246, Fax : +81 422 60 4012
e-mail : [email protected]

man mmap
man 5 ipc

That still doesn't tell me why it's necessary to do something so complicated

there is a still a stop/start here

why not just hand the state of to a different process? how is that slower?

maybe, but i'm far from convinced it's necessary and therefore warrants a big ugly kernel patch

The general case load doesn't apply here. Those doing call switching know what they have running, and know that it won't give the scheduler any opportunities to not run what must be run, in time. Given a sufficiently short run queue, the scheduler is quite predictable.

And there is difference between the entire system being down for over 100 milliseconds, and a given call being delayed that long. Under sufficient load (busiest hour on Mothers Day, say), the system must continue to operate, though some switching may be delayed longer than normal, though still within specified limits.

The call switching folks have been doing live patching at least since I worked on it, over 25 years ago. This is not just marketing.

No sense in being disrespectful to Takashi-san. This patch may or may not be the best way to provide the functionality they require. I don't even know if a kernel patch is needed.

But the tone of this thread won't lead anyone to better answers anytime soon.

                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.650.933.1373, 1.925.600.0401

That still doesn't explain *why* live patching is needed.

I suspect it was needed in the past, on embedded computers so small they could only run one program at a time.

I see no reason why changing programs on the fly couldn't be done nicer with SHM segments today - just start up the new program in parallel with the old one, have it attach to the SHM region and handshake with the old program to take over operations.

At that point the old program can let go of file descriptors (eg. those to devices), yield the CPU and the new program can open those file descriptors. The SHM area contains all of the state information needed, so the program can continue running like it always would.

This may well be lower latency than live patching, and probably lower complexity/risk too...

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Hello,

I think most important thing for carrier system is service availability.
The live patch only stops process(which have 3 threads) 180 nanoseconds with 2functions, 2 variable changes on my linux desktop(Xeon 2.8G dual). (on sample 1)

I believe process status copy consume more time, may be below sequences are needed;
- Stop the service on ACT-process.
- Copy on memory/on transaction status to shared memory.
- Takeover shared memory key to SBY process and release the shared memory
- SBY process access to shared memory.
- SBY process checks the memory and reset broken sessions.
- SBY process restart the service.

Some part may be parallelize, but seems the more data make service disruption time longer...(It seems exceeds 100 milliseconds depends on data size..) and process will be more complicated....makes more bugs...

Takashi Ikebe
NTT Network Service Systems Laboratories
9-11, Midori-Cho 3-Chome Musashino-Shi,
Tokyo 180-8585 Japan
Tel : +81 422 59 4246, Fax : +81 422 60 4012
e-mail : [email protected]

No need for this, the process could ALWAYS store its status in a shared memory status. This is just as fast as private memory, only more flexible.

Which means the SBY process can attach to the shared memory region while the ACT process is running. It can then communicate with the ACT process through a socket ...

... and the SBY process can take over immediately. The state machine running the SBY software can continue using the same data structures the ACT process was using beforehand, since they're in a shared memory region.

The data size should not be an issue, since the primary copy of the state is in the shared memory area.

The state machine in the SBY process can directly run using those data structures, so no copying is needed.

The only overhead will be inter-process communication, having the first process close file descriptors, yielding the CPU to the second process, which then opens up the devices again. We both know how long a context switch and an open() syscall take - negligable.

The old version of the program can shut itself down after it knows the new version has taken over - in the background, without disrupting the now active process.

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

I don't think so, because ACT process must stop service logic to takeover, if the service use network listen port.(ACT process need to stop service and close socket to take over.)

this makes software developer crazy....

For me, is seems very dangerous to estimate the primary copy is not broken through status takeover..

I think your assumption works on some type of process, but not for all the process.
Some process use critical resources such as fixed network listen port can not speed up so.
More importantly, the only process who prepare to use this mechanism only allows to use quick process takeover. This cause software development difficult.
The live patching does not require to implement such special techniques on applications.

Takashi Ikebe
NTT Network Service Systems Laboratories
9-11, Midori-Cho 3-Chome Musashino-Shi,
Tokyo 180-8585 Japan
Tel : +81 422 59 4246, Fax : +81 422 60 4012
e-mail : [email protected]

are you serious? how is live patching of .text easier than some of the other suggestions which all are more or less sane and things like gdb, oprofile, etc. will deal with w/o problems?

patching code in a running process is way complicated and messy, if you think this is the easier solution i guess i have little more to say

that would also be a problem for live patching too, if you have bad state, you have bad state --- live patching doesn't change that

hand the fd off to another process

no, i can mmap state or similar, hand fd's off and switch to another process in a context switch... hot patching i bet is going to be slower

how about you show up some code that needs this?

i honestly doubt in most cases you can hand live patch faster and more easily than having the application sensibly written and passing it off

please, prove me wrong, show us some code

this is like saying live patching is a complicated in-kernel solution for badly written userspace isn't it?

Sorry, I may mistake the point,

What I want to say is takeover may makes memory unstable, because there are extra operations to reserve current (unstable) status to memory. Live patching never force target process to reserve status to memory. Is this make sense?

I think the point is how long does it takes to hand the fd off to another process.(means how long time the network port is unavailable??)

Please see and try http://pannus.sourceforge.net
There includes commands and some samples.
On live patching, you never need to use shared memory, just prepare fixed code, and just compile it as shared ibject, that's all. pretty easy and fast to replace the functions.

Takashi Ikebe
NTT Network Service Systems Laboratories
9-11, Midori-Cho 3-Chome Musashino-Shi,
Tokyo 180-8585 Japan
Tel : +81 422 59 4246, Fax : +81 422 60 4012
e-mail : [email protected]

mmap is coherent between processes

Not really. I don't see how it makes it any better or easier, just different.

Probably under 1 ms. Not long anyhow.

pannus-sample.tgz contains some pretty contrived examples, nothing that anyone could really sensibly comment on

it requires magic like a compiler and knowledge of the original application.

if the application was written sensibly someone without access to the application code could change this live taking over the previous applications state even more easily --- and the code would be more straightforward. so i still fail to see why this is needed.

Hello,

Well, Live patching is just a patch, so I think the developer of patch should know the original source code well.

Well, as you said some application can do that, but some application can not continue service with your suggestion.
please think about the process which use connection type communication such as TCP(it's only example) between users and server. During status copy, all the session between users and server are disconnected... can not save the exiting service at all.
It's one example, but similar problems may occurs whenever processed use the resources which are mainly controlled by kernel.

Takashi Ikebe
NTT Network Service Systems Laboratories
9-11, Midori-Cho 3-Chome Musashino-Shi,
Tokyo 180-8585 Japan
Tel : +81 422 59 4246, Fax : +81 422 60 4012
e-mail : [email protected]

In which case they could fix the application.

Such as?

They don't have to be.

Yes they can.

What resources? We can migrate memory and file descriptors? What is missing?

Anyhow, you seem hell bent on this despite showing any real evidence it's useful or desirable... maybe a different audience for your patches would help?

http://selenic.com/mailman/listinfo/kernel-mentors might be of value to you.

Hi,
I think basic assumption between us and you is not match..
Our assumption, the live patching is not for debug, but for the real operation method to fix very very important process which can not stop. Live patchin fix the important process's bug without disrupting process.

Yes, so they provide us the patch module, and we want to apply the patch as live patching.

???
To takeover the application status, connection type communications(SOCK_STREAM) are need to be disconnected by close(). Same network port is not allowed to bind by multiple processes....

How can you do that??
Users don't want to disconnect,(and also we don't want to disconnect) but server process need to it to takeover the status.

For example,
current process's resouces of rlimit.
you nerver set current rusage to new process.
especialy, ru_utime and ru_stime is very important to critical applications. I don't know much about resources, but there may be more....(I hope not..)

Takashi Ikebe
NTT Network Service Systems Laboratories
9-11, Midori-Cho 3-Chome Musashino-Shi,
Tokyo 180-8585 Japan
Tel : +81 422 59 4246, Fax : +81 422 60 4012
e-mail : [email protected]

No, I think at a high-level they do.

I understand that.

It might be though you could probably do what you want with some kind of enhanced ptrace or debugging interface that would also be of value to other people and probably simple than your proposed patch.

I understand that.

AF_UNIX socket with SCM_RIGHTS

how so? what is magical about these that can't be dealt with in userspace should it span 2+ processes?

hmm.. most internet base services will use TCPv4 TCPv6 SCTP... AF_UNIX can not use as inter-nodes communication.

Takashi Ikebe
NTT Network Service Systems Laboratories
9-11, Midori-Cho 3-Chome Musashino-Shi,
Tokyo 180-8585 Japan
Tel : +81 422 59 4246, Fax : +81 422 60 4012
e-mail : [email protected]

You can send file descriptors (the actually file descriptors themselves, not their contents) to another process over a socket.

A nearly ten-year old example is attached (ie. this isn't new or magical or specific to Linux).

/* sendfd.c - v. crude example of passing fds */

/*
* I tested this on HPUX10 using gcc -D_XOPEN_SOURCE_EXTENDED sendfd.c -lxnet *
* Expected output is "hello world"
*/

#include <stdio.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/socket.h>
#include <sys/un.h>

/* Note: msg_control may be msg_accrights, etc. */

int sendfd(int conn, int fd)

{
    size_t len = sizeof(struct cmsghdr) + sizeof(int);
    struct cmsghdr *hdr = (struct cmsghdr *)malloc(len);
    struct msghdr msg;
    int rc;

    hdr->cmsg_len = len;
    hdr->cmsg_level = SOL_SOCKET;
    hdr->cmsg_type = SCM_RIGHTS;
    *(int *)CMSG_DATA(hdr) = fd;

    msg.msg_name = NULL;
    msg.msg_namelen = 0;
    msg.msg_iov = NULL;
    msg.msg_iovlen = 0;
    msg.msg_control = hdr;
    msg.msg_controllen = len;

    rc = sendmsg(conn, &msg, 0);
    free(hdr);
    return rc;
}

int recvfd(int conn)

{
    size_t len = sizeof(struct cmsghdr) + sizeof(int);
    struct cmsghdr *hdr = (struct cmsghdr *)malloc(len);
    struct msghdr msg;
    int rc;

    msg.msg_iov = NULL;
    msg.msg_iovlen = 0;
    msg.msg_control = hdr;
    msg.msg_controllen = len;

    rc = recvmsg(conn, &msg, 0);

    if (rc >= 0 
	&& hdr->cmsg_len == len 
	&& hdr->cmsg_level == SOL_SOCKET 
	&& hdr->cmsg_type == SCM_RIGHTS)
    {
	int fd = *(int *)CMSG_DATA(hdr);
	free(hdr);
	return fd;
    }

    free(hdr);
    return -1;
}

int main()

{
    int fds[2];
    int fd = -1;
    int rc = socketpair(AF_UNIX, SOCK_STREAM, 0, fds);
    pid_t pid;

    if (rc)
    {
	perror("socketpair");
	return 1;
    }

    /* punt */
    system("echo hello world >testfile");

    switch (pid = fork())
    {
    case 0:
	// open this in the child proc
	fd = open("testfile",O_RDONLY);
	if (fd < 0)
	    perror("open(testfile)");
	rc = sendfd(fds[1], fd);
	if (rc)
	    perror("sendfd");
	_exit(0);
    case -1:
	perror("fork");
	return 1;
    default:
	waitpid(pid,NULL,0);
    }

    fd = recvfd(fds[0]);

    if (fd < 0)
    {
	perror("recvfd");
	return 1;
    }

    close(fds[0]);
    close(fds[1]);

    dup2(fd,0);
    close(fd);
    execl("/bin/cat","cat",NULL);
}

Hmm interest enough,
But please see man.

NOTES
On Linux, the only supported domain for this call is AF_UNIX (or
syn-
onymously, AF_LOCAL). (Most implementations have the same
restric-

       tion.)

Only for AF_UNIX..

Well, as many said Live patching is very historical & authoritative function on especially carrier, telecom vendor.
If linux want to be adopted on mission critical world, this function is esseintial.

Takashi Ikebe
NTT Network Service Systems Laboratories
9-11, Midori-Cho 3-Chome Musashino-Shi,
Tokyo 180-8585 Japan
Tel : +81 422 59 4246, Fax : +81 422 60 4012
e-mail : [email protected]

I'm sure that means AF_UNIX is restricted for the socket you use to pass the file descriptors, not a restriction on the file descriptors themselves. I don't see why the kernel would care what the descriptors are.

Linux doesn't have it now, do it's not historical in the Linux space.

But Linux is used in mission critical places and we don't have that feature.

Yes, if you want to use Linux in those scenarios you will need to change the telco programs to use shared memory and file descriptor passing, instead of live patching.

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Unfortunately we're also dealing (in many cases) with pre-existing software coming over from other OS's. The beancounters want to avoid rewriting the millions of lines of application code, so they'd rather add the missing support to the kernel.

If it doesn't go into mainline, we'll just end up with a bunch of different telco-patches being maintained on the side. I highly doubt all the applications will get fixed any time soon.

Chris

In article <[email protected]>,

No, Chris means filedescriptor passing.

You can pass any existing open filedescriptor to another process using an AF_UNIX socket.

For example, the existing running process creates a UNIX socket in /var/run/mysocket that the new process can connect() to. The processes can then not only exchange all kinds of information, the old process can even send open filedescriptors over to the new process without closing/re-opening.

See "man 7 unix", ANCILLARY MESSAGES -> SCM_RIGHTS

ANCILLARY MESSAGES
Ancillary data is sent and received using sendmsg(2) and recvmsg(2). For historical reasons the ancillary message types listed below are specified with a SOL_SOCKET type even though they are PF_UNIX specific. To send them set the cmsg_level field of the struct cmsghdr to SOL_SOCKET and the cmsg_type field to the type. For more information see cmsg(3).

SCM_RIGHTS
Send or receive a set of open file descriptors from another pro- cess. The data portion contains an integer array of the file
descriptors. The passed file descriptors behave as though they
have been created with dup(2).

Mike.

Sorry, I misunderstand it, forget above comment, both methods are possible to destroy memory.

Takashi Ikebe
NTT Network Service Systems Laboratories
9-11, Midori-Cho 3-Chome Musashino-Shi,
Tokyo 180-8585 Japan
Tel : +81 422 59 4246, Fax : +81 422 60 4012
e-mail : [email protected]

True enough.

When a requirement is based in 30 years of tradition and practice, it takes work to to back it up to the essentials that would distinguish accurately between adequate and inadequate alternatives. And that I presume is what you mean by the emphasized *why*.

                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.650.933.1373, 1.925.600.0401

The more optimization a modern compiler does the less practical a patching approach seems for anything but very trivial fixes.

I'd try a shared library based approach for on the fly updates.

Ralf

The version that I've seen imposed requirements on the application for this to work properly.

There are tradeoffs either way.

Chris

How exactly is this different from ptrace?
Seems just like a ptrace memcpy extension
Is the patching really that time critical that you cant do it with normal ptrace?

Hello,

Only few patch modules are not so critical, however sometimes large number of patches are applied at one time. In that case, time is very critical with normal ptrace. As you know, normal ptrace need to target process STOP whenever change the memory/registers.
Our approach is "do not stop the target process's execution as possible as", because the target process can provide service during patch on SMP machine (do not want to stop service due to patch).
If we load hundreds of patch modules at one time, I think it will goes quite time critical..

Thank you, we'll fix this soon.

OK, we'll fix this soon.

Well, this can change as you said, but I think, this makes target process stopping time increase.
Because, to control target process's (patch module's) initialization, the command process should know the target process's status and then stop with ptrace.
Currently rt_handlereturn works on target process's context like signal handler return, so, I think there is minimum time loss on target process. If command process controls the target process's initialization, this seems target process's stopping time increasing.
Well, may be our idea is wrong, please tell us.

Thank you your advice!

Takashi Ikebe
NTT Network Service Systems Laboratories
9-11, Midori-Cho 3-Chome Musashino-Shi,
Tokyo 180-8585 Japan
Tel : +81 422 59 4246, Fax : +81 422 60 4012
e-mail : [email protected]

If you want that exact functionality, do this:

At program start, spawn a new thread:
1) Open a UNIX socket (/var/run/someapp_live_patch.sock)
2) poll() that socket for a connection.
3) When you get a connection, do your own security checks
4) If it's ok, then map the specified file into memory
5) Read a table of crap to patch from the file
6) Do the patching, being careful to avoid the millions of races involved for each CPU, *especially* regarding the separate icache and dcache on CPUs like PPC and such.
7) Go back to step 2

If you want equivalent functionality but much safer and not CPU dependent and full of hand-coded assembly:

1) open(), mmap(), and mlock() the file (/var/lib/someapp/data)
2) Spawn normal operation threads
3) Spawn a new hot-patch thread:
1) Open a UNIX socket (/var/run/someapp_live_patch.sock)
2) poll() that socket for a connection.
3) When you get one, coordinate with the new process as it attaches itself to /var/lib/someapp/data
4) Handle shared locking of parts of /var/lib/someapp/data
5) Send it your listen() file-descriptors over the socket.
6) Wait for the other process to signal it's ready.
7) Stop accepting new connections on the socket.
8) Send file-descriptors for current connections
9) Cleanup and quit

When live-patching:

1)  connect to the socket /var/run/someapp_live_patch.sock
2)  open(), mmap() and mlock() /var/lib/someapp/data
3)  Coordinate with the other process via the socket
4)  Receive the listen() file-descriptors over the socket.
5)  Set up the shared data locking
6)  Spawn normal operation threads
7)  Signal readiness
8)  Receive file-descriptors for current connections
9)  Spawn threads for them too.

10) Spawn a new hot-patch thread as above

Cheers,
Kyle Moffett

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$ L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+ PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r !y?(-) ------END GEEK CODE BLOCK------

Kyle, thank you so much for your detailed information.
If you design completely new software, your suggestion is very useful!

Unfortunately, we carrier have very many exiting software and try to run on Linux.
We need to seek the way which can apply to exiting software also...

Takashi Ikebe
NTT Network Service Systems Laboratories
9-11, Midori-Cho 3-Chome Musashino-Shi,
Tokyo 180-8585 Japan
Tel : +81 422 59 4246, Fax : +81 422 60 4012
e-mail : [email protected]

If you notice, the above method has only minimal changes from your mmap3 stuff, except without needing kernel support. One thing to remember, though, as there _is_ a very clean method to do this from userspace, therefore you are not likely to get much sympathy on this list.

I suggest you try adding a new hotpatch thread to your code, as above, then use it to implement the mmap3 and other tasks necessary for live patching instead of in kernel space.

Cheers,
Kyle Moffett

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$ L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+ PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r !y?(-) ------END GEEK CODE BLOCK------

Hi!

"We want to do the wrong thing because we think its easier".

Okay, you are free to do that, but don't try to push that into mainline kernel. Maintain your own patches; if that seems too hard, do the right thing.
Pavel

Boycott Kodak -- for their patent abuse against Java.

ptrace can all do this, even with an existing kernel.
Your full patch is just a funky ptrace equivalent as far as I can see.

-Andi

You *really* want to take the time to re-write the software to do things The Linux Way. If you're looking at doing on-the-fly patching, you're probably also carrying around a lot of *other* ugly cruft to make this creeping horror work on Linux. In fact, I'd not be surprised if you have a shim layer to make the compatibility layer for the *previous* system work on Linux...

I'm reminded of a (possibly apocryphal) quote from an ATT spokesperson from 1988 or so, when a misplaced comma in a patch kept crashing the long-distance phone network. When asked "Why don't you just reboot the affected switches?" his response was "This assumes that the switch had ever been booted in the first place". (Apparently, the *whole thing* had been on-the-fly replaced/patched without an actual reload happening...)

Gaaahhh! :)

I think that's the common sense in every carrier.
If we reboot the switch, the service will be disrupted.
The phone network is lifeline, and does not allow to be disrupt by just bug fix. I think same kind of function is needed in many real enterprise/mission-critical/business area.

All do with ptrace may affect target process's time critical task. (need to stop target process whenever fix)
All implement in user application costs too much, need to implement all the application...(and I do not know this approach really works on time critical applications yet.) There are clear demand to realize this common and GPL-ed function....

That is definitely not common sense. It may be good business practice, but those are two *entirely* different things.

Yes. My personal favorite solution to this problem is HeartBeat, some Open-Source software that is very good at maintaining high availability. With a properly written multi-system clustering switch application that utilizes the Linux Virtual-Server tools, you could reasonably efficiently run a system such that you can reboot any individual system without any loss of service.

But you miss the point. Linux is *NOT* about "business", or "enterprise", or "mission-critical". Linux is (at least to many hackers) about hacking, having fun, and Good Design(TM).

So don't do it with ptrace!!! I've given you one other method that uses minimal changes to existing software and emulates the crappy mmap3 call you keep trying to push.

What about one of the dozen other offered methods?

So why not write a utility library? You'd need to "implement all in the kernel", too, and since it can be done better in userspace, let's keep out the bloat while we're at it.

So test it! You're clearly working for a big corporation with the money and resources to develop something like this, so do so, and if you get something that works well, *and* uses good design, we'll welcome patches!

The kernel is not about business, demand, or what the CEO of some big-name company wants. The kernel strives for the goal of "Good Engineering (TM)".

Cheers,
Kyle Moffett

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$ L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+ PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r !y?(-) ------END GEEK CODE BLOCK------

        ~~~~~~~~~~~~~~~~

I had very strong urge to reply with "<plonk>" here.

Clearly noone but you wants to make kernel more ugly just for "faster ptrace". If you want faster ptrace, fine, advertise it as such and provide nice and small patch to make it faster.

If you are going to handwave about "clear demand", well, find some other list to troll on.
Pavel

Boycott Kodak -- for their patent abuse against Java.

Sorry, but what are your exact time requirements for this?

Remember any x86-64 CPU is really fast and it can do a _lot_ of ptrace operations in a very short time.

Just a vague "it may be too slow" is not enough justification to push a lot of redundant code into the kernel. Also if ptrace should be really too slow (which I doubt, but you are welcome to show some numbers together with real time requirements from a real system) then we could optimize ptrace for this, e.g. by adding a ptrace subcommand to copy whole memory blocks more efficiently or maybe even do a mmap like thing.

But unless someone actually demonstrates this is needed it seems far overkill.

I think you have a lot of unproved and doubtful assumptions here.

-Andi