kevent: Generic event handling mechanism (socket, pipes, files, signals, AIO, timers, posix timers...).
Subsystems (13)
A message that is not in the archive (sent off-list or not yet received): [email protected]
cover kevent: Generic event handling mechanism (socket, pipes, files, signals, AIO, timers, posix timers...).
Generic event handling mechanism (socket, pipes, files, signals, AIO, timers, posix timers...).
Kevent is a generic subsytem which allows to handle event notifications. It supports both level and edge triggered events. It is similar to poll/epoll in some cases, but it is more scalable, it is faster and allows to work with essentially eny kind of events.
It can serve as a storage for different AIO models as well - in case syscall or other request is ready immediately kevent returns that event in submission point.
Events are provided into kernel through control syscall and can be read back through ring buffer or using usual syscalls.
Kevent update (i.e. readiness switching) happens directly from internals of the appropriate state machine of the underlying subsytem (like network, filesystem, timer or any other).
Homepage:
http://tservice.net.ru/~s0mbre/old/?section=projects&item=kevent
Documentation page:
http://linux-net.osdl.org/index.php/Kevent
Git tree:
http://tservice.net.ru/~s0mbre/archive/kevent/kevent.git/
This is the latest patchset sent for inclusion/declining decision, kevent will be moved into maintenance state with usual per-kernel releases if no decision will be made.
P.S. If you want to be removed from Cc: list just drop me a mail.
Changes from 'take36' patchset:
* ported to 2.6.21-rc1
* documentation cleanups by Frederik Deweerdt.
Changes from 'take35' patchset:
* Fixed typo in Makefile about kevent based replacement for epoll (not included into patchset) which led to compilation failure.
* Changed AIO description text.
Changes from 'take34' patchset:
* Ported to the 2.6.20-rc7 (9be5b038b1c9d1927c367bf91683458e10d5d4eb) tree.
Changes from 'take33' patchset:
* Added optional header pointer and its size into aio_sendfile_path(), which allows to send header and file in one syscall instead of send(header), open file, sendfile(file).
Changes from 'take32' patchset:
* Updated documentation (aio_sendfile_path()).
* Fixed typo in forward declaration.
Changes from 'take31' patchset:
* Added aio_sendfile_path() - this syscall allows to asynchronosly transfer file specified by provided pathname to destination socket. Opened file descriptor is returned.
* Added trivial scheduler which selects execution thread. It allows to specify given thread 'by-hands', but since kaio provides '-1' it uses round-robin to get processing thread. In theory it can be bound to scheduler statistics or gamma-ray receiver data.
* Number of bug fixes in kevent based AIO mpage_readpages().
Benchmark of the 100 1Mb files transfer (files are in VFS already) using sync sendfile or this new version shows about 10Mb/sec performance win for aio_sendfile_path().
Changes from 'take30' patchset:
* AIO state machine.
* aio_sendfile() implementation.
* moved kevent_user_get/kevent_user_put into header.
* use *zalloc where needed.
Changes from 'take29' patchset:
* new private userspace notifications - allows to queue any userspace private event and then mark it as ready using kevent_ctl(KEVENT_READY) command
* KEVENT_REQ_READY flag - if set kevent will be marked as ready at enqueue time
* port to 2.6.20-rc2 tree (54abb5fcdae74a811ed440ec6556cabc6b24f404 commit)
* use struct kmem_cache instead of kmem_cache_t
* added notificaion type into search key, this allows to have the same id for different types of notifications
Changes from 'take28' patchset:
* optimized af_unix to use socket notifications
* changed ALWAYS_QUEUE behaviour with poll/select notifications - previously kevent was not queued into poll wait queue when ALWAYS_QUEUE flag is set
* added KEVENT_POLL_POLLRDHUP definition into ukevent.h header
* libevent-1.2 patch (Jamal, your request is completed, so I'm waiting two weeks before starting final countdown :)
All regression tests passed successfully except test_evbuffer(), which is crashed on my amd64 linux 2.6 test machine for all types of notifications, probably it was fixed in libevent-1.2a version, I did not check. Patch and README can be found at project homepage.
Changes from 'take27' patchset:
* made kevent default yes in non embedded case.
* added falgs to callback structures - currently used to check if kevent can be requested from kernelspace only (posix timers) or userspace (all others)
Changes from 'take26' patchset:
* made kevent visible in config only in case of embedded setup.
* added comment about KEVENT_MAX number.
* spell fix.
Changes from 'take25' patchset:
* use timespec as timeout parameter.
* added high-resolution timer to handle absolute timeouts.
* added flags to waiting and initialization syscalls.
* kevent_commit() has new_uidx parameter.
* kevent_wait() has old_uidx parameter, which, if not equal to u->uidx, results in immediate wakeup (usefull for the case when entries
are added asynchronously from kernel (not supported for now)).
* added interface to mark any event as ready.
* event POSIX timers support.
* return -ENOSYS if there is no registered event type.
* provided file descriptor must be checked for fifo type (spotted by Eric Dumazet).
* signal notifications.
* documentation update.
* lighttpd patch updated (the latest benchmarks with lighttpd patch can be found in blog).
Changes from 'take24' patchset:
* new (old (new)) ring buffer implementation with kernel and user indexes.
* added initialization syscall instead of opening /dev/kevent
* kevent_commit() syscall to commit ring buffer entries
* changed KEVENT_REQ_WAKEUP_ONE flag to KEVENT_REQ_WAKEUP_ALL, kevent wakes only first thread always if that flag is not set
* KEVENT_REQ_ALWAYS_QUEUE flag. If set, kevent will be queued into ready queue instead of copying back to userspace when kevent is ready immediately when it is added.
* lighttpd patch (Hail! Although nothing really outstanding compared to epoll)
Changes from 'take23' patchset:
* kevent PIPE notifications
* KEVENT_REQ_LAST_CHECK flag, which allows to perform last check at dequeueing time
* fixed poll/select notifications (were broken due to tree manipulations)
* made Documentation/kevent.txt look nice in 80-col terminal
* fix for copy_to_user() failure report for the first kevent (Andrew Morton)
* minor function renames
Changes from 'take22' patchset:
* new ring buffer implementation in process' memory
* wakeup-one-thread flag
* edge-triggered behaviour
Changes from 'take21' patchset:
* minor cleanups (different return values, removed unneded variables, whitespaces and so on)
* fixed bug in kevent removal in case when kevent being removed
is the same as overflow_kevent (spotted by Eric Dumazet)
Changes from 'take20' patchset:
* new ring buffer implementation
* removed artificial limit on possible number of kevents
Changes from 'take19' patchset:
* use __init instead of __devinit
* removed 'default N' from config for user statistic
* removed kevent_user_fini() since kevent can not be unloaded
* use KERN_INFO for statistic output
Changes from 'take18' patchset:
* use __init instead of __devinit
* removed 'default N' from config for user statistic
* removed kevent_user_fini() since kevent can not be unloaded
* use KERN_INFO for statistic output
Changes from 'take17' patchset:
* Use RB tree instead of hash table.
At least for a web sever, frequency of addition/deletion of new kevent is comparable with number of search access, i.e. most of the time events are added, accesed only couple of times and then removed, so it justifies RB tree usage over AVL tree, since the latter does have much slower deletion time (max O(log(N)) compared to 3 ops),
although faster search time (1.44*O(log(N)) vs. 2*O(log(N))).
So for kevents I use RB tree for now and later, when my AVL tree implementation is ready, it will be possible to compare them.
* Changed readiness check for socket notifications.
With both above changes it is possible to achieve more than 3380 req/second compared to 2200, sometimes 2500 req/second for epoll() for trivial web-server and httperf client on the same hardware.
It is possible that above kevent limit is due to maximum allowed kevents in a time limit, which is 4096 events.
Changes from 'take16' patchset:
* misc cleanups (__read_mostly, const ...)
* created special macro which is used for mmap size (number of pages) calculation
* export kevent_socket_notify(), since it is used in network protocols which can be built as modules (IPv6 for example)
Changes from 'take15' patchset:
* converted kevent_timer to high-resolution timers, this forces timer API update at
http://linux-net.osdl.org/index.php/Kevent
* use struct ukevent* instead of void * in syscalls (documentation has been updated)
* added warning in kevent_add_ukevent() if ring has broken index (for testing)
Changes from 'take14' patchset:
* added kevent_wait()
This syscall waits until either timeout expires or at least one event becomes ready. It also commits that @num events from @start are processed by userspace and thus can be be removed or rearmed (depending on it's flags). It can be used for commit events read by userspace through mmap interface. Example userspace code (evtest.c) can be found on project's homepage.
* added socket notifications (send/recv/accept)
Changes from 'take13' patchset:
* do not get lock aroung user data check in __kevent_search()
* fail early if there were no registered callbacks for given type of kevent
* trailing whitespace cleanup
Changes from 'take12' patchset:
* remove non-chardev interface for initialization
* use pointer to kevent_mring instead of unsigned longs
* use aligned 64bit type in raw user data (can be used by high-res timer if needed)
* simplified enqueue/dequeue callbacks and kevent initialization
* use nanoseconds for timeout
* put number of milliseconds into timer's return data
* move some definitions into user-visible header
* removed filenames from comments
Changes from 'take11' patchset:
* include missing headers into patchset
* some trivial code cleanups (use goto instead of if/else games and so on)
* some whitespace cleanups
* check for ready_callback() callback before main loop which should save us some ticks
Changes from 'take10' patchset:
* removed non-existent prototypes
* added helper function for kevent_registered_callbacks
* fixed 80 lines comments issues
* added shared between userspace and kernelspace header instead of embedd them in one
* core restructuring to remove forward declarations
* s o m e w h i t e s p a c e c o d y n g s t y l e c l e a n u p
* use vm_insert_page() instead of remap_pfn_range()
Changes from 'take9' patchset:
* fixed ->nopage method
Changes from 'take8' patchset:
* fixed mmap release bug
* use module_init() instead of late_initcall()
* use better structures for timer notifications
Changes from 'take7' patchset:
* new mmap interface (not tested, waiting for other changes to be acked)
- use nopage() method to dynamically substitue pages
- allocate new page for events only when new added kevent requres it
- do not use ugly index dereferencing, use structure instead
- reduced amount of data in the ring (id and flags), maximum 12 pages on x86 per kevent fd
Changes from 'take6' patchset:
* a lot of comments!
* do not use list poisoning for detection of the fact, that entry is in the list
* return number of ready kevents even if copy*user() fails
* strict check for number of kevents in syscall
* use ARRAY_SIZE for array size calculation
* changed superblock magic number
* use SLAB_PANIC instead of direct panic() call
* changed -E* return values
* a lot of small cleanups and indent fixes
Changes from 'take5' patchset:
* removed compilation warnings about unused wariables when lockdep is not turned on
* do not use internal socket structures, use appropriate (exported) wrappers instead
* removed default 1 second timeout
* removed AIO stuff from patchset
Changes from 'take4' patchset:
* use miscdevice instead of chardevice
* comments fixes
Changes from 'take3' patchset:
* removed serializing mutex from kevent_user_wait()
* moved storage list processing to RCU
* removed lockdep screaming - all storage locks are initialized in the same function, so it was
learned
to differentiate between various cases
* remove kevent from storage if is marked as broken after callback
* fixed a typo in mmaped buffer implementation which would end up in wrong index calcualtion
Changes from 'take2' patchset:
* split kevent_finish_user() to locked and unlocked variants
* do not use KEVENT_STAT ifdefs, use inline functions instead
* use array of callbacks of each type instead of each kevent callback initialization
* changed name of ukevent guarding lock
* use only one kevent lock in kevent_user for all hash buckets instead of per-bucket locks
* do not use kevent_user_ctl structure instead provide needed arguments as syscall parameters
* various indent cleanups
* added optimisation, which is aimed to help when a lot of kevents are being copied from
userspace
* mapped buffer (initial) implementation (no userspace yet)
Changes from 'take1' patchset:
- rebased against 2.6.18-git tree
- removed ioctl controlling
- added new syscall kevent_get_events(int fd, unsigned int min_nr, unsigned int max_nr, unsigned int timeout, void __user *buf, unsigned flags)
- use old syscall kevent_ctl for creation/removing, modification and initial kevent initialization
- use mutuxes instead of semaphores
- added file descriptor check and return error if provided descriptor does not match kevent file operations
- various indent fixes
- removed aio_sendfile() declarations.
Thank you.
Signed-off-by Evgeniy Polyakov <[email protected]>
PATCH 1/10 kevent: Description.
Description.
Documentation/kevent.txt new file +276 -0
@@ -0,0 +1,276 @@ +Description. + +Kevent documentation +==================== + +http://tservice.net.ru/~s0mbre/old/?section=projects&item=kevent + +int kevent_init(struct kevent_ring *ring, unsigned int num, + unsigned int flags); + +num - size of the ring buffer in events +ring - pointer to the allocated ring buffer +flags - various flags, see KEVENT_FLAGS_* definitions. + +Return value: kevent control file descriptor or a negative error value. + +struct kevent_ring { + unsigned int ring_kidx, ring_over; + struct ukevent event[0]; +} + +ring_kidx - index in the ring buffer where kernel will put new events + when kevent_wait() or kevent_get_events() get called. +ring_over - number of overflows that occured on ring_kidx since the start + The overflow counter is used to prevent the situation where + two threads are going to free the same events, but one of them + was scheduled away for too long. + In that case, ring indexes may have wrapped, so when the + long-sleeping thread will be awakened, it may free the wrong events. + +An example userspace code (ring_buffer.c) can be found on project's +homepage. + +Each kevent syscall can be seen as a cancellation point in the glibc. +I.e. when a thread has been cancelled during a kevent syscall, this +thread can be safely removed and no events will be lost, since each +syscall (kevent_wait() or kevent_get_events()) will copy the event into +a special ring buffer, accessible from other threads or even processes +(if shared memory is used). + +When a kevent is removed (not dequeued when it is ready, but just +removed), even if it was ready, it is not copied into the ring buffer, +since if it is removed, no one cares about it (otherwise the user +would wait until it becomes ready and would get it the usual way: using +kevent_get_events() or kevent_wait()). There is thus no need to copy +it to the ring buffer. + +------------------------------------------------------------------------------- + + +int kevent_ctl(int fd, unsigned int cmd, unsigned int num, struct ukevent *arg); + +fd - is the file descriptor referring to the kevent queue to manipulate. +It is created by opening "/dev/kevent" char device, which is created with +dynamic minor number and major number assigned for misc devices. + +cmd - is the requested operation. It can be one of the following: + KEVENT_CTL_ADD - add event notification + KEVENT_CTL_REMOVE - remove event notification + KEVENT_CTL_MODIFY - modify existing notification + KEVENT_CTL_READY - mark existing events as ready, if the number of + events is zero, it just wakes up parked in syscall thread + +num - number of struct ukevent in the array pointed to by arg +arg - array of struct ukevent + +Return value: number of events processed or a negative error value. + +When called, kevent_ctl will carry out the operation specified in the +cmd parameter. +------------------------------------------------------------------------------- + + int kevent_get_events(int ctl_fd, unsigned int min_nr, unsigned int max_nr, + struct timespec timeout, struct ukevent *buf, unsigned flags); + +ctl_fd - file descriptor referring to the kevent queue +min_nr - minimum number of completed events that kevent_get_events + will block waiting for +max_nr - number of struct ukevent in buf +timeout - time to wait before returning less than min_nr events. If this + is -1, then wait forever. +buf - pointer to an array of struct ukevent. +flags - various flags, see KEVENT_FLAGS_* definitions. + +Return value: number of events copied or a negative error value. + +kevent_get_events will wait timeout milliseconds for at least min_nr completed +events, copying completed struct ukevents to buf and deleting any +KEVENT_REQ_ONESHOT event requests. In nonblocking mode it returns as many +events as possible, but not more than max_nr. In blocking mode it waits until +timeout or if at least min_nr events are ready. + +This function copies event into ring buffer if it was initialized, if ring buffer +is full, KEVENT_RET_COPY_FAILED flag is set in ret_flags field. +------------------------------------------------------------------------------- + + int kevent_wait(int ctl_fd, unsigned int num, unsigned int old_uidx, + struct timespec timeout, unsigned int flags); + +ctl_fd - file descriptor referring to the kevent queue +num - number of processed kevents +old_uidx - the last index user is aware of +timeout - time to wait until there is free space in kevent queue +flags - various flags, see KEVENT_FLAGS_* definitions. + +Return value: number of events copied into ring buffer or a negative +error value. + +This syscall waits until either timeout expires or at least one event becomes +ready. It also copies events into special ring buffer. If ring buffer is full, +it waits until there are ready events and then return. +If kevent is a one-shot kevent it is removed in this syscall. +If kevent is a edge-triggered (KEVENT_REQ_ET flag is set in 'req_flags'), +it will be requeued in this syscall for performance reasons. +------------------------------------------------------------------------------- + + int kevent_commit(int ctl_fd, unsigned int new_idx, unsigned int over); + +ctl_fd - file descriptor referring to the kevent queue +new_uidx - the last committed kevent +over - overflow count for given $new_idx value + +Return value: + number of committed kevents or negative error value. + +This function commits, i.e. marks as empty, slots in the ring buffer, so +they can be reused when userspace completes that entries processing. + +Overflow counter is used to prevent situation when two threads are going +to free the same events, but one of them was scheduled away for too long, +so ring indexes were wrapped, so when that thread will be awakened, it +will free not those events, which it suppose to free. + +It is possible that returned number of committed events will be smaller than +requested number - it is possible when several threads try to commit the +same events. +------------------------------------------------------------------------------- + +long aio_sendfile(int kevent_fd, int sock_fd, int in_fd, off_t offset, size_t count); + +kevent_fd - file descriptor referring to the kevent queue +sock_fd - destination socket file descriptor +in_fd - source file descriptor +offset - offset from the beginning of the source file +count - number of bytes to transfer + +Async sendfile implementation. +Returned coockie can be used to determine which entry has been returned by +kevent_get_events() - it will be stored in event.ptr. +event.ret_data will contain number of bytes actually transferred. +------------------------------------------------------------------------------- + +long aio_sendfile_path(int kevent_fd, int sock_fd, + void *header, size_t header_size, + char *filename, off_t offset, size_t count); + +kevent_fd - file descriptor referring to the kevent queue +sock_fd - destination socket file descriptor +header - header data, which will be sent before file's data +header_size - size of the header +pathname - source filename +offset - offset from the beginning of the source file +count - number of bytes to transfer +------------------------------------------------------------------------------- + +The bulk of the interface is entirely done through the ukevent struct. +It is used to add event requests, modify existing event requests, +specify which event requests to remove, and return completed events. + +struct ukevent contains the following members: + +struct kevent_id id + Id of this request, e.g. socket number, file descriptor and so on +__u32 type + Event type, e.g. KEVENT_SOCK, KEVENT_INODE, KEVENT_TIMER and so on +__u32 event + Event itself, e.g. SOCK_ACCEPT, INODE_CREATED, TIMER_FIRED +__u32 req_flags + Per-event request flags, + + KEVENT_REQ_ONESHOT + event will be removed when it is ready + + KEVENT_REQ_WAKEUP_ALL + Kevent wakes up only first thread interested in given event, + or all threads if this flag is set. + + KEVENT_REQ_ET + Edge Triggered behaviour. It is an optimisation which allows to move + ready and dequeued (i.e. copied to userspace) event to move into set + of interest for given storage (socket, inode and so on) again. It is + very usefull for cases when the same event should be used many times + (like reading from pipe). It is similar to epoll()'s EPOLLET flag. + + KEVENT_REQ_LAST_CHECK + if set allows to perform the last check on kevent (call appropriate + callback) when kevent is marked as ready and has been removed from + ready queue. If it will be confirmed that kevent is ready + (k->callbacks.callback(k) returns true) then kevent will be copied + to userspace, otherwise it will be requeued back to storage. + Second (checking) call is performed with this bit cleared, so callback + can detect when it was called from kevent_storage_ready() - bit is set, + or kevent_dequeue_ready() - bit is cleared. If kevent will be requeued, + bit will be set again. + + KEVENT_REQ_ALWAYS_QUEUE + If this flag is set kevent will be queued into ready queue if it is + ready at enqueue time, otherwise it will be copied back to userspace + and will not be queued into the storage. + + KEVENT_REQ_READY + If this flag is set, kevent will be marked as ready immediately at enqueue + time. + +__u32 ret_flags + Per-event return flags + + KEVENT_RET_BROKEN + Kevent is broken + + KEVENT_RET_DONE + Kevent processing was finished successfully + + KEVENT_RET_COPY_FAILED + Kevent was not copied into ring buffer due to some error conditions. + +__u32 ret_data + Event return data. Event originator fills it with anything it likes + (for example timer notifications put number of milliseconds when timer + has fired +union { __u32 user[2]; void *ptr; } + User's data. It is not used, just copied to/from user. The whole structure + is aligned to 8 bytes already, so the last union is aligned properly. + +------------------------------------------------------------------------------- + +Kevent waiting syscall flags. + +KEVENT_FLAGS_ABSTIME - provided timespec parameter contains absolute time, + for example Aug 27, 2194, or time(NULL) + 10. + +------------------------------------------------------------------------------- + +Usage + +For KEVENT_CTL_ADD, all fields relevant to the event type must be filled +(id, type, event, req_flags). +After kevent_ctl(..., KEVENT_CTL_ADD, ...) returns each struct's ret_flags +should be checked to see if the event is already broken or done. + +For KEVENT_CTL_MODIFY, the id, req_flags, and user and event fields must be +set and an existing kevent request must have matching id and user fields. If +match is found, req_flags and event are replaced with the newly supplied +values and requeueing is started, so modified kevent can be checked and +probably marked as ready immediately. If a match can't be found, the +passed in ukevent's ret_flags has KEVENT_RET_BROKEN set. KEVENT_RET_DONE is +always set. + +For KEVENT_CTL_REMOVE, the id and user fields must be set and an existing +kevent request must have matching id and user fields. If a match is found, +the kevent request is removed. If a match can't be found, the passed in +ukevent's ret_flags has KEVENT_RET_BROKEN set. KEVENT_RET_DONE is always set. + +For kevent_get_events, the entire structure is returned. + +------------------------------------------------------------------------------- + +Usage cases + +kevent_timer +struct ukevent should contain following fields: + type - KEVENT_TIMER + event - KEVENT_TIMER_FIRED + req_flags - KEVENT_REQ_ONESHOT if you want to fire that timer only once + id.raw[0] - number of seconds after commit when this timer shout expire + id.raw[1] - additional to number of seconds number of nanoseconds
PATCH 2/10 kevent: Core files.
Core files.
This patch includes core kevent files:
* userspace controlling
* kernelspace interfaces
* initialization
* notification state machines
Some bits of documentation can be found on project's homepage (and links from there):
http://tservice.net.ru/~s0mbre/old/?section=projects&item=kevent
Signed-off-by Evgeniy Polyakov <[email protected]>
arch/i386/kernel/syscall_table.S +7 -0
@@ -319,3 +319,10 @@ ENTRY(sys_call_table) .long sys_move_pages .long sys_getcpu .long sys_epoll_pwait + .long sys_kevent_get_events + .long sys_kevent_ctl /* 320 */ + .long sys_kevent_wait + .long sys_kevent_commit + .long sys_kevent_init + .long sys_aio_sendfile + .long sys_aio_sendfile_path
arch/x86_64/ia32/ia32entry.S +8 -1
@@ -714,9 +714,16 @@ ia32_sys_call_table: .quad compat_sys_get_robust_list .quad sys_splice .quad sys_sync_file_range - .quad sys_tee + .quad sys_tee /* 315 */ .quad compat_sys_vmsplice .quad compat_sys_move_pages .quad sys_getcpu .quad sys_epoll_pwait + .quad sys_kevent_get_events + .quad sys_kevent_ctl /* 320 */ + .quad sys_kevent_wait + .quad sys_kevent_commit + .quad sys_kevent_init + .quad sys_aio_sendfile + .quad sys_aio_sendfile_path ia32_syscall_end:
include/asm-i386/unistd.h +8 -1
@@ -325,10 +325,17 @@#define __NR_move_pages 317#define __NR_getcpu 318#define __NR_epoll_pwait 319 +#define __NR_kevent_get_events 320 +#define __NR_kevent_ctl 321 +#define __NR_kevent_wait 322 +#define __NR_kevent_commit 323 +#define __NR_kevent_init 324 +#define __NR_aio_sendfile 325 +#define __NR_aio_sendfile_path 326#ifdef __KERNEL__ -#define NR_syscalls 320 +#define NR_syscalls 327#define __ARCH_WANT_IPC_PARSE_VERSION#define __ARCH_WANT_OLD_READDIR
include/asm-x86_64/unistd.h +16 -2
@@ -619,8 +619,22 @@ __SYSCALL(__NR_sync_file_range, sys_sync_file_range) __SYSCALL(__NR_vmsplice, sys_vmsplice)#define __NR_move_pages 279 __SYSCALL(__NR_move_pages, sys_move_pages) - -#define __NR_syscall_max __NR_move_pages +#define __NR_kevent_get_events 280 +__SYSCALL(__NR_kevent_get_events, sys_kevent_get_events) +#define __NR_kevent_ctl 281 +__SYSCALL(__NR_kevent_ctl, sys_kevent_ctl) +#define __NR_kevent_wait 282 +__SYSCALL(__NR_kevent_wait, sys_kevent_wait) +#define __NR_kevent_commit 283 +__SYSCALL(__NR_kevent_commit, sys_kevent_commit) +#define __NR_kevent_init 284 +__SYSCALL(__NR_kevent_init, sys_kevent_init) +#define __NR_aio_sendfile 285 +__SYSCALL(__NR_aio_sendfile, sys_aio_sendfile) +#define __NR_aio_sendfile_path 286 +__SYSCALL(__NR_aio_sendfile_path, sys_aio_sendfile_path) + +#define __NR_syscall_max __NR_aio_sendfile_path#ifndef __NO_STUBS#define __ARCH_WANT_OLD_READDIR
include/linux/kevent.h new file +268 -0
268 lines not on this page · raw message
include/linux/kevent_storage.h new file +11 -0
@@ -0,0 +1,11 @@ +#ifndef __KEVENT_STORAGE_H +#define __KEVENT_STORAGE_H + +struct kevent_storage +{ +void *origin;/* Originator's pointer, e.g. struct sock or struct file. Can be NULL. */ +struct list_head list;/* List of queued kevents. */ +spinlock_t lock;/* Protects users queue. */ +}; + +#endif /* __KEVENT_STORAGE_H */
include/linux/syscalls.h +15 -0
@@ -54,6 +54,8 @@ struct compat_stat;struct compat_timeval ;struct robust_list_head ;struct getcpu_cache ; +struct ukevent ; +struct kevent_ring ;#include <linux/types.h> #include <linux/aio_abi.h>
@@ -603,6 +605,19 @@ asmlinkage long sys_set_robust_list(struct robust_list_head __user *head, size_t len);asmlinkage long sys_getcpu (unsigned __user *cpu,unsigned __user *node,struct getcpu_cache __user *cache); +asmlinkage long sys_kevent_get_events (int ctl_fd,unsigned int min,unsigned int max, +struct timespec timeout,struct ukevent __user *buf,unsigned flags); +asmlinkage long sys_kevent_ctl (int ctl_fd,unsigned int cmd,unsigned int num,struct ukevent __user *buf); +asmlinkage long sys_kevent_wait (int ctl_fd,unsigned int num,unsigned int old_uidx, +struct timespec timeout,unsigned int flags); +asmlinkage long sys_kevent_commit (int ctl_fd,unsigned int new_uidx,unsigned int over); +asmlinkage long sys_kevent_init (struct kevent_ring __user *ring,unsigned int num,unsigned int flags); +int kernel_execve (const char *filename,char *const argv[],char *const envp[]); +asmlinkage long sys_aio_sendfile (int kevent_fd,int sock_fd,int in_fd,off_t offset,size_t count); +asmlinkage long sys_aio_sendfile_path (int kevent_fd,int sock_fd, +void __user *header,size_t header_size, +char __user *filename,off_t offset,size_t count); +#endif
include/linux/ukevent.h new file +191 -0
191 lines not on this page · raw message
init/Kconfig +2 -0
@@ -243,6 +243,8 @@ config AUDITSYSCALL such as SELinux. To use audit's filesystem watch feature, please ensure that INOTIFY is configured. +source "kernel/kevent/Kconfig" + config IKCONFIG tristate "Kernel .config support" ---help---
kernel/Makefile +1 -0
@@ -46,6 +46,7 @@ obj-$(CONFIG_DETECT_SOFTLOCKUP) += softlockup.o obj-$(CONFIG_GENERIC_HARDIRQS) += irq/ obj-$(CONFIG_SECCOMP) += seccomp.o obj-$(CONFIG_RCU_TORTURE_TEST) += rcutorture.o +obj-$(CONFIG_KEVENT) += kevent/ obj-$(CONFIG_RELAY) += relay.o obj-$(CONFIG_SYSCTL) += utsname_sysctl.o obj-$(CONFIG_UTS_NS) += utsname.o
kernel/kevent/Kconfig new file +82 -0
82 lines not on this page · raw message
kernel/kevent/Makefile new file +8 -0
@@ -0,0 +1,8 @@ +obj-y := kevent.o kevent_user.o +obj-$(CONFIG_KEVENT_TIMER) += kevent_timer.o +obj-$(CONFIG_KEVENT_POLL) += kevent_poll.o +obj-$(CONFIG_KEVENT_SOCKET) += kevent_socket.o +obj-$(CONFIG_KEVENT_PIPE) += kevent_pipe.o +obj-$(CONFIG_KEVENT_SIGNAL) += kevent_signal.o +obj-$(CONFIG_KEVENT_UNOTIFY) += kevent_unotify.o +obj-$(CONFIG_KEVENT_AIO) += kevent_aio.o
kernel/kevent/kevent.c new file +267 -0
267 lines not on this page · raw message
kernel/kevent/kevent_user.c new file +1360 -0
1,360 lines not on this page · raw message
kernel/sys_ni.c +9 -0
@@ -123,6 +123,15 @@ cond_syscall(ppc_rtas);cond_syscall (sys_spu_run);cond_syscall (sys_spu_create); +cond_syscall (sys_kevent_get_events); +cond_syscall (sys_kevent_ctl); +cond_syscall (sys_kevent_wait); +cond_syscall (sys_kevent_commit); +cond_syscall (sys_kevent_init); + +cond_syscall (sys_aio_sendfile); +cond_syscall (sys_aio_sendfile_path); +/* mmu depending weak syscall entries */ cond_syscall (sys_mprotect);cond_syscall (sys_msync);
PATCH 3/10 kevent: poll/select() notifications.
poll/select() notifications.
This patch includes generic poll/select notifications. kevent_poll works simialr to epoll and has the same issues (callback is invoked not from internal state machine of the caller, but through process awake, a lot of allocations and so on).
Signed-off-by Evgeniy Polyakov <[email protected]>
fs/file_table.c +3 -0
@@ -20,6 +20,7 @@#include <linux/cdev.h> #include <linux/fsnotify.h> #include <linux/sysctl.h> +#include <linux/kevent.h> #include <linux/percpu_counter.h> #include <asm/atomic.h>
@@ -119,6 +120,7 @@ struct file *get_empty_filp(void) f->f_uid = tsk->fsuid; f->f_gid = tsk->fsgid;eventpoll_init_file (f); +kevent_init_file (f);/* f->f_version: 0 */ return f;
@@ -164,6 +166,7 @@ void fastcall __fput(struct file *file)* in the file cleanup chain. */ eventpoll_release (file); +kevent_cleanup_file (file);locks_remove_flock (file);if (file->f_op && file->f_op->release)
include/linux/fs.h +10 -0
@@ -281,6 +281,7 @@ extern int dir_notify_enable;#include <linux/init.h> #include <linux/pid.h> #include <linux/mutex.h> +#include <linux/kevent_storage.h> #include <asm/atomic.h> #include <asm/semaphore.h>
@@ -409,6 +410,8 @@ struct address_space_operations {int (*readpages)(struct file *filp,struct address_space *mapping,struct list_head *pages,unsigned nr_pages); +int (*aio_readpages)(struct file *filp,struct address_space *mapping, +struct list_head *pages,unsigned nr_pages,void *priv);/* * ext3 requires that a successful prepare_write() call be followed
@@ -580,6 +583,10 @@ struct inode {struct mutex inotify_mutex;/* protects the watches list */ #endif +#if definedCONFIG_KEVENT_SOCKET || definedCONFIG_KEVENT_PIPE +struct kevent_storage st; +#endif +unsigned long i_state;unsigned long dirtied_when;/* jiffies of first dirtying */
@@ -739,6 +746,9 @@ struct file {struct list_head f_ep_links;spinlock_t f_ep_lock;#endif /* #ifdef CONFIG_EPOLL */ +#ifdef CONFIG_KEVENT_POLL +struct kevent_storage st; +#endif struct address_space *f_mapping; };extern spinlock_t files_lock;
kernel/kevent/kevent_poll.c new file +234 -0
234 lines not on this page · raw message
PATCH 4/10 kevent: Socket notifications.
Socket notifications.
This patch includes socket send/recv/accept notifications.
Signed-off-by Evgeniy Polyakov <[email protected]>
fs/inode.c +7 -0
@@ -21,6 +21,7 @@#include <linux/cdev.h> #include <linux/bootmem.h> #include <linux/inotify.h> +#include <linux/kevent.h> #include <linux/mount.h> /*
@@ -164,12 +165,18 @@ static struct inode *alloc_inode(struct super_block *sb) } inode->i_private =NULL ; inode->i_mapping = mapping; +#if definedCONFIG_KEVENT_SOCKET || definedCONFIG_KEVENT_PIPE +kevent_storage_init (inode, &inode->st); +#endif }return inode; }void destroy_inode (struct inode *inode) { +#if definedCONFIG_KEVENT_SOCKET || definedCONFIG_KEVENT_PIPE +kevent_storage_fini (&inode->st); +#endif BUG_ON (inode_has_buffers (inode));security_inode_free (inode);if (inode->i_sb->s_op->destroy_inode)
include/net/sock.h +17 -15
@@ -49,6 +49,7 @@#include <linux/skbuff.h> /* struct sk_buff */ #include <linux/mm.h> #include <linux/security.h> +#include <linux/kevent.h> #include <linux/filter.h>
@@ -451,6 +452,21 @@ static inline int sk_stream_memory_free(struct sock *sk)extern void sk_stream_rfree (struct sk_buff *skb); +struct socket_alloc { +struct socket socket; +struct inode vfs_inode; +}; + +static inline struct socket *SOCKET_I (struct inode *inode) +{ + return &container_of(inode, struct socket_alloc, vfs_inode)->socket; +} + +static inline struct inode *SOCK_INODE (struct socket *socket) +{ + return &container_of(socket, struct socket_alloc, socket)->vfs_inode; +} +static inline void sk_stream_set_owner_r (struct sk_buff *skb,struct sock *sk) { skb->sk = sk;
@@ -478,6 +494,7 @@ static inline void sk_add_backlog(struct sock *sk, struct sk_buff *skb) sk->sk_backlog.tail = skb; } skb->next =NULL ; +kevent_socket_notify (sk,KEVENT_SOCKET_RECV ); }#define sk_wait_event (__sk, __timeo, __condition) \
@@ -679,21 +696,6 @@ static inline struct kiocb *siocb_to_kiocb(struct sock_iocb *si)return si->kiocb; } -struct socket_alloc { -struct socket socket; -struct inode vfs_inode; -}; - -static inline struct socket *SOCKET_I (struct inode *inode) -{ - return &container_of(inode, struct socket_alloc, vfs_inode)->socket; -} - -static inline struct inode *SOCK_INODE (struct socket *socket) -{ - return &container_of(socket, struct socket_alloc, socket)->vfs_inode; -} -extern void __sk_stream_mem_reclaim (struct sock *sk);extern int sk_stream_mem_schedule (struct sock *sk,int size,int kind);
include/net/tcp.h +1 -0
@@ -863,6 +863,7 @@ static inline int tcp_prequeue(struct sock *sk, struct sk_buff *skb) tp->ucopy.memory =0 ; }else if (skb_queue_len (&tp->ucopy.prequeue) ==1 ) {wake_up_interruptible (sk->sk_sleep); +kevent_socket_notify (sk,KEVENT_SOCKET_RECV |KEVENT_SOCKET_SEND );if (!inet_csk_ack_scheduled (sk)) inet_csk_reset_xmit_timer(sk, ICSK_TIME_DACK, (3 * TCP_RTO_MIN) / 4,
kernel/kevent/kevent_socket.c new file +144 -0
144 lines not on this page · raw message
net/core/sock.c +9 -1
@@ -1393,6 +1393,7 @@ static void sock_def_wakeup(struct sock *sk)if (sk->sk_sleep &&waitqueue_active (sk->sk_sleep))wake_up_interruptible_all (sk->sk_sleep);read_unlock (&sk->sk_callback_lock); +kevent_socket_notify (sk,KEVENT_SOCKET_RECV |KEVENT_SOCKET_SEND ); }static void sock_def_error_report (struct sock *sk)
@@ -1402,6 +1403,7 @@ static void sock_def_error_report(struct sock *sk)wake_up_interruptible (sk->sk_sleep);sk_wake_async (sk,0 ,POLL_ERR );read_unlock (&sk->sk_callback_lock); +kevent_socket_notify (sk,KEVENT_SOCKET_RECV |KEVENT_SOCKET_SEND ); }static void sock_def_readable (struct sock *sk,int len)
@@ -1411,6 +1413,7 @@ static void sock_def_readable(struct sock *sk, int len)wake_up_interruptible (sk->sk_sleep);sk_wake_async (sk,1 ,POLL_IN );read_unlock (&sk->sk_callback_lock); +kevent_socket_notify (sk,KEVENT_SOCKET_RECV |KEVENT_SOCKET_SEND ); }static void sock_def_write_space (struct sock *sk)
@@ -1430,6 +1433,7 @@ static void sock_def_write_space(struct sock *sk) }read_unlock (&sk->sk_callback_lock); +kevent_socket_notify (sk,KEVENT_SOCKET_SEND |KEVENT_SOCKET_RECV ); }static void sock_def_destruct (struct sock *sk)
@@ -1480,6 +1484,8 @@ void sock_init_data(struct socket *sock, struct sock *sk) sk->sk_state =TCP_CLOSE ; sk->sk_socket = sock; +kevent_sk_reinit (sk); +sock_set_flag (sk,SOCK_ZAPPED );if (sock)
@@ -1546,8 +1552,10 @@ void fastcall release_sock(struct sock *sk)if (sk->sk_backlog.tail)__release_sock (sk); sk->sk_lock.owner =NULL ; -if (waitqueue_active (&sk->sk_lock.wq)) +if (waitqueue_active (&sk->sk_lock.wq)) {wake_up (&sk->sk_lock.wq); +kevent_socket_notify (sk,KEVENT_SOCKET_RECV |KEVENT_SOCKET_SEND ); + }spin_unlock_bh (&sk->sk_lock.slock); }EXPORT_SYMBOL (release_sock);
net/core/stream.c +1 -0
@@ -36,6 +36,7 @@ void sk_stream_write_space(struct sock *sk)wake_up_interruptible (sk->sk_sleep);if (sock->fasync_list && !(sk->sk_shutdown &SEND_SHUTDOWN ))sock_wake_async (sock,2 ,POLL_OUT ); +kevent_socket_notify (sk,KEVENT_SOCKET_SEND |KEVENT_SOCKET_RECV ); } }
net/ipv4/tcp_input.c +1 -0
@@ -3143,6 +3143,7 @@ static void tcp_ofo_queue(struct sock *sk)__skb_unlink (skb, &tp->out_of_order_queue);__skb_queue_tail (&sk->sk_receive_queue, skb); +kevent_socket_notify (sk,KEVENT_SOCKET_RECV ); tp->rcv_nxt =TCP_SKB_CB (skb)->end_seq;if (skb->h.th->fin)tcp_fin (skb, sk, skb->h.th);
net/ipv4/tcp_ipv4.c +2 -0
@@ -61,6 +61,7 @@#include <linux/jhash.h> #include <linux/init.h> #include <linux/times.h> +#include <linux/kevent.h> #include <net/icmp.h> #include <net/inet_hashtables.h>
@@ -1391,6 +1392,7 @@ int tcp_v4_conn_request(struct sock *sk, struct sk_buff *skb)reqsk_free (req); }else {inet_csk_reqsk_queue_hash_add (sk, req,TCP_TIMEOUT_INIT ); +kevent_socket_notify (sk,KEVENT_SOCKET_ACCEPT ); }return 0 ;
net/socket.c +3 -0
@@ -84,6 +84,7 @@#include <linux/kmod.h> #include <linux/audit.h> #include <linux/wireless.h> +#include <linux/kevent.h> #include <asm/uaccess.h> #include <asm/unistd.h>
@@ -483,6 +484,8 @@ static struct socket *sock_alloc(void) inode->i_uid = current->fsuid; inode->i_gid = current->fsgid; +kevent_socket_reinit (sock); +get_cpu_var (sockets_in_use)++;put_cpu_var (sockets_in_use);return sock;
net/unix/af_unix.c +21 -1
@@ -1563,8 +1563,10 @@ static int unix_dgram_recvmsg(struct kiocb *iocb, struct socket *sock,struct scm_cookie tmp_scm;struct sock *sk = sock->sk;struct unix_sock *u =unix_sk (sk); +struct sock *other;int noblock = flags &MSG_DONTWAIT ;struct sk_buff *skb; +int err; err = -EOPNOTSUPP ;
@@ -1580,6 +1582,12 @@ static int unix_dgram_recvmsg(struct kiocb *iocb, struct socket *sock, goto out_unlock;wake_up_interruptible (&u->peer_wait); + other =unix_peer_get (sk); +if (other) { +kevent_socket_notify (other,KEVENT_SOCKET_SEND ); +sock_put (other); + }else +kevent_socket_notify (sk,KEVENT_SOCKET_RECV );if (msg->msg_name)unix_copy_addr (msg, skb->sk);
@@ -1674,7 +1682,7 @@ static int unix_stream_recvmsg(struct kiocb *iocb, struct socket *sock, {struct sock_iocb *siocb =kiocb_to_siocb (iocb);struct scm_cookie tmp_scm; -struct sock *sk = sock->sk; +struct sock *sk = sock->sk, *other;struct unix_sock *u =unix_sk (sk);struct sockaddr_un *sunaddr=msg->msg_name;int copied =0 ;
@@ -1803,6 +1811,14 @@ static int unix_stream_recvmsg(struct kiocb *iocb, struct socket *sock, } }while (size); + other =unix_peer_get (sk); +if (other) { +kevent_socket_notify (other,KEVENT_SOCKET_SEND ); +sock_put (other); + } +if (sk->sk_shutdown &RCV_SHUTDOWN || !other) +kevent_socket_notify (sk,KEVENT_SOCKET_RECV ); +mutex_unlock (&u->readlock);scm_recv (sock, msg, siocb->scm, flags); out:
@@ -1824,6 +1840,10 @@ static int unix_shutdown(struct socket *sock, int mode)sock_hold (other);unix_state_wunlock (sk); sk->sk_state_change (sk); +kevent_socket_notify (sk,KEVENT_SOCKET_SEND |KEVENT_SOCKET_RECV ); +if (other) +kevent_socket_notify (other,KEVENT_SOCKET_SEND |KEVENT_SOCKET_RECV ); +kevent_socket_notify (sk,KEVENT_SOCKET_SEND |KEVENT_SOCKET_RECV );if (other && (sk->sk_type ==SOCK_STREAM || sk->sk_type ==SOCK_SEQPACKET )) {
PATCH 5/10 kevent: Timer notifications.
Timer notifications.
Timer notifications can be used for fine grained per-process time management, since interval timers are very inconvenient to use, and they are limited.
This subsystem uses high-resolution timers.
id.raw[0] is used as number of seconds
id.raw[1] is used as number of nanoseconds
Signed-off-by Evgeniy Polyakov <[email protected]>
kernel/kevent/kevent_timer.c new file +114 -0
@@ -0,0 +1,114 @@ +/* +* 2006 Copyright (c) Evgeniy Polyakov <[email protected]> +* All rights reserved. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program; if not, write to the Free Software +* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +*/ + +#include <linux/kernel.h> +#include <linux/types.h> +#include <linux/list.h> +#include <linux/slab.h> +#include <linux/spinlock.h> +#include <linux/hrtimer.h> +#include <linux/jiffies.h> +#include <linux/kevent.h> + +struct kevent_timer +{ +struct hrtimer ktimer; +struct kevent_storage ktimer_storage; +struct kevent *ktimer_event; +}; + +static int kevent_timer_func (struct hrtimer *timer) +{ + struct kevent_timer *t = container_of(timer, struct kevent_timer, ktimer); +struct kevent *k = t->ktimer_event; + +kevent_storage_ready (&t->ktimer_storage,NULL ,KEVENT_MASK_ALL ); +hrtimer_forward (timer, timer->base->softirq_time, +ktime_set (k->event.id.raw[0 ], k->event.id.raw[1 ])); +return HRTIMER_RESTART ; +} + +static struct lock_class_key kevent_timer_key; + +static int kevent_timer_enqueue (struct kevent *k) +{ +int err; +struct kevent_timer *t; + + t =kmalloc (sizeof (struct kevent_timer ),GFP_KERNEL ); +if (!t) +return -ENOMEM ; + +hrtimer_init (&t->ktimer,CLOCK_MONOTONIC ,HRTIMER_REL ); + t->ktimer.expires =ktime_set (k->event.id.raw[0 ], k->event.id.raw[1 ]); + t->ktimer.function = kevent_timer_func; + t->ktimer_event = k; + + err =kevent_storage_init (&t->ktimer, &t->ktimer_storage); +if (err) + goto err_out_free; +lockdep_set_class (&t->ktimer_storage.lock, &kevent_timer_key); + + err =kevent_storage_enqueue (&t->ktimer_storage, k); +if (err) + goto err_out_st_fini; + +hrtimer_start (&t->ktimer, t->ktimer.expires,HRTIMER_REL ); + +return 0 ; + +err_out_st_fini: +kevent_storage_fini (&t->ktimer_storage); +err_out_free: +kfree (t); + +return err; +} + +static int kevent_timer_dequeue (struct kevent *k) +{ +struct kevent_storage *st = k->st; + struct kevent_timer *t = container_of(st, struct kevent_timer, ktimer_storage); + +hrtimer_cancel (&t->ktimer); +kevent_storage_dequeue (st, k); +kfree (t); + +return 0 ; +} + +static int kevent_timer_callback (struct kevent *k) +{ + k->event.ret_data[0 ] =jiffies_to_msecs (jiffies); +return 1 ; +} + +static int __init kevent_init_timer (void ) +{ +struct kevent_callbacks tc = { + .callback = &kevent_timer_callback, + .enqueue = &kevent_timer_enqueue, + .dequeue = &kevent_timer_dequeue, + .flags =0 , + }; + +return kevent_add_callbacks (&tc,KEVENT_TIMER ); +} +module_init (kevent_init_timer); +
PATCH 6/10 kevent: Pipe notifications.
Pipe notifications.
fs/pipe.c +6 -0
@@ -16,6 +16,7 @@#include <linux/uio.h> #include <linux/highmem.h> #include <linux/pagemap.h> +#include <linux/kevent.h> #include <linux/audit.h> #include <asm/uaccess.h>
@@ -314,6 +315,7 @@ redo:break ; }if (do_wakeup) { +kevent_pipe_notify (inode,KEVENT_SOCKET_SEND );wake_up_interruptible_sync (&pipe->wait);kill_fasync (&pipe->fasync_writers,SIGIO ,POLL_OUT ); }
@@ -323,6 +325,7 @@ redo:/* Signal writers asynchronously that there is more room. */ if (do_wakeup) { +kevent_pipe_notify (inode,KEVENT_SOCKET_SEND );wake_up_interruptible (&pipe->wait);kill_fasync (&pipe->fasync_writers,SIGIO ,POLL_OUT ); }
@@ -485,6 +488,7 @@ redo2:break ; }if (do_wakeup) { +kevent_pipe_notify (inode,KEVENT_SOCKET_RECV );wake_up_interruptible_sync (&pipe->wait);kill_fasync (&pipe->fasync_readers,SIGIO ,POLL_IN ); do_wakeup =0 ;
@@ -496,6 +500,7 @@ redo2: out:mutex_unlock (&inode->i_mutex);if (do_wakeup) { +kevent_pipe_notify (inode,KEVENT_SOCKET_RECV );wake_up_interruptible (&pipe->wait);kill_fasync (&pipe->fasync_readers,SIGIO ,POLL_IN ); }
@@ -591,6 +596,7 @@ pipe_release(struct inode *inode, int decr, int decw)free_pipe_info (inode); }else {wake_up_interruptible (&pipe->wait); +kevent_pipe_notify (inode,KEVENT_SOCKET_SEND |KEVENT_SOCKET_RECV );kill_fasync (&pipe->fasync_readers,SIGIO ,POLL_IN );kill_fasync (&pipe->fasync_writers,SIGIO ,POLL_OUT ); }
kernel/kevent/kevent_pipe.c new file +123 -0
123 lines not on this page · raw message
PATCH 7/10 kevent: Signal notifications.
Signal notifications.
This type of notifications allows to deliver signals through kevent queue. One can find example application signal.c on project homepage.
If KEVENT_SIGNAL_NOMASK bit is set in raw_u64 id then signal will be delivered only through queue, otherwise both delivery types are used - old through update of mask of pending signals and through queue.
If signal is delivered only through kevent queue, mask of pending signals is not updated at all, which is equal to putting signal into blocked mask, but with delivery of that signal through kevent queue.
Signed-off-by Evgeniy Polyakov <[email protected]>
include/linux/sched.h +5 -0
@@ -82,6 +82,7 @@ struct sched_param {#include <linux/resource.h> #include <linux/timer.h> #include <linux/hrtimer.h> +#include <linux/kevent_storage.h> #include <linux/task_io_accounting.h> #include <asm/processor.h>
@@ -1050,6 +1051,10 @@ struct task_struct {#ifdef CONFIG_TASK_DELAY_ACCT struct task_delay_info *delays;#endif +#ifdef CONFIG_KEVENT_SIGNAL +struct kevent_storage st; +u32 kevent_signals; +#endif #ifdef CONFIG_FAULT_INJECTION int make_it_fail;#endif
kernel/fork.c +8 -0
@@ -49,6 +49,7 @@#include <linux/delayacct.h> #include <linux/taskstats_kern.h> #include <linux/random.h> +#include <linux/kevent.h> #include <asm/pgtable.h> #include <asm/pgalloc.h>
@@ -118,6 +119,9 @@ void __put_task_struct(struct task_struct *tsk)WARN_ON (atomic_read (&tsk->usage));WARN_ON (tsk == current); +#ifdef CONFIG_KEVENT_SIGNAL +kevent_storage_fini (&tsk->st); +#endif security_task_free (tsk);free_uid (tsk->user);put_group_info (tsk->group_info);
@@ -1128,6 +1132,10 @@ static struct task_struct *copy_process(unsigned long clone_flags,if (retval) goto bad_fork_cleanup_namespaces; +#ifdef CONFIG_KEVENT_SIGNAL +kevent_storage_init (p, &p->st); +#endif + p->set_child_tid = (clone_flags &CLONE_CHILD_SETTID ) ? child_tidptr :NULL ;/* * Clear TID on mm_release()?
kernel/kevent/kevent_signal.c new file +94 -0
94 lines not on this page · raw message
kernel/signal.c +26 -0
@@ -23,6 +23,7 @@#include <linux/ptrace.h> #include <linux/signal.h> #include <linux/capability.h> +#include <linux/kevent.h> #include <linux/freezer.h> #include <linux/pid_namespace.h> #include <linux/nsproxy.h>
@@ -738,6 +739,9 @@ static int send_signal(int sig, struct siginfo *info, struct task_struct *t, {struct sigqueue * q =NULL ;int ret =0 ; + +if (kevent_signal_notify (t, sig)) +return 1 ;/* * fast-pathed signals for kernel-internal things like SIGSTOP
@@ -817,6 +821,17 @@ specific_send_sig_info(int sig, struct siginfo *info, struct task_struct *t) ret =send_signal (sig, info, t, &t->pending);if (!ret && !sigismember (&t->blocked, sig))signal_wake_up (t, sig ==SIGKILL ); +#ifdef CONFIG_KEVENT_SIGNAL +/* +* Kevent allows to deliver signals through kevent queue, +* it is possible to setup kevent to not deliver +* signal through the usual way, in that case send_signal() +* returns 1 and signal is delivered only through kevent queue. +* We simulate successfull delivery notification through this hack: +*/ +if (ret ==1 ) + ret =0 ; +#endif out:return ret; }
@@ -1006,6 +1021,17 @@ __group_send_sig_info(int sig, struct siginfo *info, struct task_struct *p)* to avoid several races. */ ret =send_signal (sig, info, p, &p->signal->shared_pending); +#ifdef CONFIG_KEVENT_SIGNAL +/* +* Kevent allows to deliver signals through kevent queue, +* it is possible to setup kevent to not deliver +* signal through the usual way, in that case send_signal() +* returns 1 and signal is delivered only through kevent queue. +* We simulate successfull delivery notification through this hack: +*/ +if (ret ==1 ) + ret =0 ; +#endif if (unlikely (ret))return ret;
PATCH 8/10 kevent: Kevent posix timer notifications.
Kevent posix timer notifications.
Simple extension to POSIX timers which allows
to deliver notification of the timer expiration
through kevent queue.
Example application posix_timer.c can be found
in archive on project homepage.
Signed-off-by Evgeniy Polyakov <[email protected]>
include/asm-generic/siginfo.h +3 -0
@@ -235,6 +235,7 @@ typedef struct siginfo {#define SIGEV_NONE 1/* other notification: meaningless */ #define SIGEV_THREAD 2/* deliver via thread creation */ #define SIGEV_THREAD_ID 4/* deliver to thread */ +#define SIGEV_KEVENT 8/* deliver through kevent queue */ /* * This works because the alignment is ok on all current architectures
@@ -260,6 +261,8 @@ typedef struct sigevent {void (*_function)(sigval_t );void *_attribute;/* really pthread_attr_t */ } _sigev_thread; + +int kevent_fd; } _sigev_un; } sigevent_t;
include/linux/posix-timers.h +4 -0
@@ -4,6 +4,7 @@#include <linux/spinlock.h> #include <linux/list.h> #include <linux/sched.h> +#include <linux/kevent_storage.h> union cpu_time_count {cputime_t cpu;
@@ -49,6 +50,9 @@ struct k_itimer {sigval_t it_sigev_value;/* value word of sigevent struct */ struct task_struct *it_process;/* process to send signal to */ struct sigqueue *sigq;/* signal queue entry. */ +#ifdef CONFIG_KEVENT_TIMER +struct kevent_storage st; +#endif union { struct { struct hrtimer timer;
kernel/posix-timers.c +173 -47
266 lines not on this page · raw message
PATCH 9/10 kevent: Private userspace notifications.
Private userspace notifications.
Allows to register notifications of any private userspace events over kevent. Events can be marked as ready using kevent_ctl(KEVENT_READY) command.
Signed-off-by Evgeniy Polyakov <[email protected]>
kernel/kevent/kevent_unotify.c new file +62 -0
@@ -0,0 +1,62 @@ +/* +* 2006 Copyright (c) Evgeniy Polyakov <[email protected]> +* All rights reserved. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program; if not, write to the Free Software +* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +*/ + +#include <linux/kernel.h> +#include <linux/kevent.h> + +static int kevent_unotify_callback (struct kevent *k) +{ +return 1 ; +} + +int kevent_unotify_enqueue (struct kevent *k) +{ +int err; + + err =kevent_storage_enqueue (&k->user->st, k); +if (err) + goto err_out_exit; + +if (k->event.req_flags &KEVENT_REQ_ALWAYS_QUEUE ) +kevent_requeue (k); + +return 0 ; + +err_out_exit: +return err; +} + +int kevent_unotify_dequeue (struct kevent *k) +{ +kevent_storage_dequeue (k->st, k); +return 0 ; +} + +static int __init kevent_init_unotify (void ) +{ +struct kevent_callbacks sc = { + .callback = &kevent_unotify_callback, + .enqueue = &kevent_unotify_enqueue, + .dequeue = &kevent_unotify_dequeue, + .flags =0 , + }; + +return kevent_add_callbacks (&sc,KEVENT_UNOTIFY ); +} +module_init (kevent_init_unotify);
PATCH 10/10 kevent: Kevent based generic AIO.
Kevent based generic AIO.
This patch only implements network AIO, which is _COMPLETELY_ impossible and broken in _ANY_ micro-thread design. For details and test consider following link:
http://tservice.net.ru/~s0mbre/blog/2007/02/10#2007_02_10
Designing AIO without network in mind can only be result of heavy hang-over.
Kevent AIO is implemented as state machine.
There is a patch which implements async open/send_header/sendfile/close.
aio_sendfile()/aio_sendfile_path() contains of two major parts: AIO state machine and page processing code.
The former is just a small subsystem, which allows to queue callback for theirs invocation in process' context on behalf of pool of kernel threads. It allows to queue caches of callbacks to the local thread or to any other specified. Each cache of callbacks is processed until there are callbacks in it, callbacks can requeue themselfs into the same cache.
Real work is being done in page processing code - code which populates pages into VFS cache and then sends pages to the destination socket via ->sendpage(). Unlike previous aio_sendfile() implementation, new one does not require low-level filesystem specific callbacks (->get_block()) at all, instead I extended struct address_space_operations to contain new member called ->aio_readpages(), which is exactly the same as ->readpage() (read: mpage_readpages()) except different BIO allocation and sumbission routines. I changed mpage_readpages() to provide mpage_alloc() and mpage_bio_submit() to the new function called __mpage_readpages(), which is exactly old mpage_readpages() with provided callback invocation instead of usage for old functions. mpage_readpages_aio() provides kevent specific callbacks, which calls old functions, but with different destructor callbacks, which are essentially the same, except that they reschedule AIO processing.
aio_sendfile_path() is essentially aio_sendfile(), except that it takes source filename as parameter, has a pointer to private header and its size (which allows to send header and file's content in one syscall instead of three (open, send, sendfile) and returns opened file descriptor.
Benchmark of the 100 1MB files transfer (files are in VFS already) using sync sendfile() against aio_sendfile_path() shows about 10MB/sec performance win (78 MB/s vs 66-72 MB/s over 1 Gb network, sendfile sending server is one-way AMD Athlong 64 3500+) for aio_sendfile_path().
AIO state machine is a base for network AIO (which becomes quite trivial), but I will not start implementation until roadback of kevent as a whole and AIO implementation become more clear.
Signed-off-by Evgeniy Polyakov <[email protected]>
fs/bio.c +1 -1
@@ -120,7 +120,7 @@ void bio_free(struct bio *bio, struct bio_set *bio_set)/* * default destructor for a bio allocated with bio_alloc_bioset() */ -static void bio_fs_destructor (struct bio *bio) +void bio_fs_destructor (struct bio *bio) {bio_free (bio, fs_bio_set); }
fs/ext3/inode.c +8 -0
@@ -1650,6 +1650,13 @@ ext3_readpages(struct file *file, struct address_space *mapping,return mpage_readpages (mapping, pages, nr_pages, ext3_get_block); } +static int +ext3_readpages_aio (struct file *file,struct address_space *mapping, +struct list_head *pages,unsigned nr_pages,void *priv) +{ +return mpage_readpages_aio (mapping, pages, nr_pages, ext3_get_block, priv); +} +static void ext3_invalidatepage (struct page *page,unsigned long offset) {journal_t *journal =EXT3_JOURNAL (page->mapping->host);
@@ -1768,6 +1775,7 @@ static int ext3_journalled_set_page_dirty(struct page *page) }static const struct address_space_operations ext3_ordered_aops = { + .aio_readpages = ext3_readpages_aio, .readpage = ext3_readpage, .readpages = ext3_readpages, .writepage = ext3_ordered_writepage,
fs/mpage.c +76 -41
188 lines not on this page · raw message
include/linux/mpage.h +2 -0
@@ -16,6 +16,8 @@ typedef int (writepage_t)(struct page *page, struct writeback_control *wbc);int mpage_readpages (struct address_space *mapping,struct list_head *pages,unsigned nr_pages,get_block_t get_block); +int mpage_readpages_aio (struct address_space *mapping,struct list_head *pages, +unsigned nr_pages,get_block_t get_block,void *priv);int mpage_readpage (struct page *page,get_block_t get_block);int mpage_writepages (struct address_space *mapping,struct writeback_control *wbc,get_block_t get_block);
kernel/kevent/kevent_aio.c new file +939 -0
939 lines not on this page · raw message
Quoted 2 lines · Evgeniy Polyakov
Changes from 'take36' patchset:
* ported to 2.6.21-rc1
Hi Evgeniy,
The changes below were needed to be able to compile take37 on 2.6.21-rc1 with:CONFIG_KEVENT=yCONFIG_KEVENT_USER_STAT=yCONFIG_KEVENT_TIMER=yCONFIG_KEVENT_POLL=yCONFIG_KEVENT_SOCKET=yCONFIG_KEVENT_PIPE=yCONFIG_KEVENT_SIGNAL=yCONFIG_KEVENT_UNOTIFY=yCONFIG_KEVENT_AIO=y
Regards,
Frederik
kernel/kevent/kevent_timer.c +3 -3
@@ -33,7 +33,7 @@ struct kevent_timerstruct kevent *ktimer_event; }; -static int kevent_timer_func (struct hrtimer *timer) +static enum hrtimer_restart kevent_timer_func (struct hrtimer *timer) { struct kevent_timer *t = container_of(timer, struct kevent_timer, ktimer);struct kevent *k = t->ktimer_event;
@@ -55,7 +55,7 @@ static int kevent_timer_enqueue(struct kevent *k)if (!t)return -ENOMEM ; -hrtimer_init (&t->ktimer,CLOCK_MONOTONIC ,HRTIMER_REL ); +hrtimer_init (&t->ktimer,CLOCK_MONOTONIC ,HRTIMER_MODE_REL ); t->ktimer.expires =ktime_set (k->event.id.raw[0 ], k->event.id.raw[1 ]); t->ktimer.function = kevent_timer_func; t->ktimer_event = k;
@@ -69,7 +69,7 @@ static int kevent_timer_enqueue(struct kevent *k)if (err) goto err_out_st_fini; -hrtimer_start (&t->ktimer, t->ktimer.expires,HRTIMER_REL ); +hrtimer_start (&t->ktimer, t->ktimer.expires,HRTIMER_MODE_REL );return 0 ;
kernel/kevent/kevent_user.c +6 -6
@@ -152,7 +152,7 @@ static struct kevent_user *kevent_user_alloc(struct kevent_ring __user *ring, un u->pring = ring; u->ring_size = num; -hrtimer_init (&u->timer,CLOCK_REALTIME ,HRTIMER_ABS ); +hrtimer_init (&u->timer,CLOCK_REALTIME ,HRTIMER_MODE_ABS );kevent_storage_init (u, &u->st);
@@ -881,7 +881,7 @@ out_remove: }/* Used to wakeup waiting syscalls in case high-resolution timer is used. */ -static int kevent_user_wake (struct hrtimer *timer) +static enum hrtimer_restart kevent_user_wake (struct hrtimer *timer) { struct kevent_user *u = container_of(timer, struct kevent_user, timer);
@@ -910,10 +910,10 @@ static int kevent_user_wait(struct file *file, struct kevent_user *u,if (flags &KEVENT_FLAGS_ABSTIME ) {hrtimer_cancel (&u->timer); -hrtimer_init (&u->timer,CLOCK_REALTIME ,HRTIMER_ABS ); +hrtimer_init (&u->timer,CLOCK_REALTIME ,HRTIMER_MODE_ABS ); u->timer.expires =ktime_set (timeout.tv_sec, timeout.tv_nsec); u->timer.function = &kevent_user_wake; -hrtimer_start (&u->timer, u->timer.expires,HRTIMER_ABS ); +hrtimer_start (&u->timer, u->timer.expires,HRTIMER_MODE_ABS );if (unlikely (kevent_debug_abstime ==0 )) {printk (KERN_INFO "kevent: author was wrong, " "someone uses absolute time in %s, "
@@ -1214,10 +1214,10 @@ asmlinkage long sys_kevent_wait(int ctl_fd, unsigned int num, unsigned int old_uif (flags &KEVENT_FLAGS_ABSTIME ) {hrtimer_cancel (&u->timer); -hrtimer_init (&u->timer,CLOCK_REALTIME ,HRTIMER_ABS ); +hrtimer_init (&u->timer,CLOCK_REALTIME ,HRTIMER_MODE_ABS ); u->timer.expires =ktime_set (timeout.tv_sec, timeout.tv_nsec); u->timer.function = &kevent_user_wake; -hrtimer_start (&u->timer, u->timer.expires,HRTIMER_ABS ); +hrtimer_start (&u->timer, u->timer.expires,HRTIMER_MODE_ABS );if (unlikely (kevent_debug_abstime ==0 )) {printk (KERN_INFO "kevent: author was wrong, " "someone uses absolute time in %s, "
Quoted 18 lines · Frederik Deweerdt
Quoted 2 lines · Evgeniy Polyakov
Changes from 'take36' patchset:
* ported to 2.6.21-rc1Hi Evgeniy,
The changes below were needed to be able to compile take37 on 2.6.21-rc1 with:
CONFIG_KEVENT=yCONFIG_KEVENT_USER_STAT=yCONFIG_KEVENT_TIMER=yCONFIG_KEVENT_POLL=yCONFIG_KEVENT_SOCKET=yCONFIG_KEVENT_PIPE=yCONFIG_KEVENT_SIGNAL=yCONFIG_KEVENT_UNOTIFY=yCONFIG_KEVENT_AIO=y
Quoted 2 lines
Regards,
Frederik
Shame on me - I started to make to many such errors last time. I've applied a patch, thanks a lot Frederik.
Signature · 1 line
Evgeniy Polyakov
cover dst: Distributed storage.
In reply to a message not in the archiveDistributed storage.
I'm pleased to announce the 8'th release of the distributed storage subsystem (DST). This is a maintenance release and includes bug fixes only.
DST allows to form a storage on top of local and remote nodes and combine them into linear or mirroring setup, which in turn can be exported to remote nodes.
Short changelog:
* cleanup sysfs files on error path.
Patch by Chris Madden <[email protected]>
Overall list of features of the DST can be found on project's homepage:
http://tservice.net.ru/~s0mbre/old/?section=projects&item=dst
Thank you.
Signed-off-by Evgeniy Polyakov <[email protected]>
PATCH 1/4 dst: Distributed storage documentation.
Distributed storage documentation.
Algorithms used in the system, userspace interfaces (sysfs dirs and files), design and implementation details are described here.
Signed-off-by Evgeniy Polyakov <[email protected]>
Documentation/dst/algorithms.txt new file +115 -0
@@ -0,0 +1,115 @@ +Each storage by itself is just a set of contiguous logical blocks, with +allowed number of operations. Nodes, each of which has own start and size, +are placed into storage by appropriate algorithm, which remaps +logical sector number into real node's sector. One can create +own algorithms, since DST has pluggable interface for that. +Currently mirrored and linear algorithms are supported. + +Let's briefly describe how they work. + +Linear algorithm. +Simple approach of concatenating storages into single device with +increased size is used in this algorithm. Essentially new device +has size equal to sum of sizes of underlying nodes and nodes are +placed one after another. + + /----- Node 1 ---\ /------ Node 3 ----\ +start end start end + |==================|========================|==================| + | start end | + | \------- Node 2 ---------/ | + | | +start end + \-------------------------- DST storage ----------------------/ + + /\ + || + || + + IO operations + + Figure 1. + 3 nodes combined into single storage using linear algorithm. + +Mirror algorithm. +In this algorithms nodes are placed under each other, so when +operation comes to the first one, it can be mirrored to all +underlying nodes. In case of reading, actual data is obtained from +the nearest node - algoritm keeps track of previous operation +and knows where it was stopped, so that subsequent seek to the +start of the new request will take the shortest time. +Writing is always mirrored to all underlying nodes. + + IO operations + || + || + \/ + +|---------------- DST storage -------------------| +| prev position | +|-------|------------ Node 1 --------------------| +| prev pos | +|-------------------- Node 2 -----|--------------| +|prev pos | +|---|---------------- Node 3 --------------------| + + Figure 2. + 3 nodes combined into single storage using mirror algorithm. + +Each algorithm must implement number of callbacks, +which must be registered during initialization time. + +struct dst_alg_ops +{ + int (*add_node)(struct dst_node *n); + void (*del_node)(struct dst_node *n); + int (*remap)(struct dst_request *req); + int (*error)(struct kst_state *state, int err); + struct module *owner; +}; + +@add_node. +This callback is invoked when new node is being added into the storage, +but before node is actually added into the storage, so that it could +be accessed from it. When it is called, all appropriate initialization +of the underlying device is already completed (system has been connected +to remote node or got a reference to the local block device). At this +stage algorithm can add node into private map. +It must return zero on success or negative value otherwise. + +@del_node. +This callback is invoked when node is being deleted from the storage, +i.e. when its reference counter hits zero. It is called before +any cleaning is performed. +It must return zero on success or negative value otherwise. + +@remap. +This callback is invoked each time new bio hits the storage. +Request structure contains BIO itself, pointer to the node, which originally +stores the whole region under given IO request, and various parameters +used by storage core to process this block request. +It must return zero on success or negative value otherwise. It is upto +this method to call all cleaning if remapping failed, for example it must +call kst_bio_endio() for given callback in case of error, which in turn +will call bio_endio(). Note, that dst_request structure provided in this +callback is allocated on stack, so if there is a need to use it outside +of the given function, it must be cloned (it will happen automatically +in state's push callback, but that copy will not be shared by any other +user). + +@error. +This callback is invoked for each error, which happend when processed +requests for remote nodes or when talking to remote size +of the local export node (state contains data related to data +transfers over the network). +If this function has fixed given error, it must return 0 or negative +error value otherwise. + +@owner. +This is module reference counter updated automatically by DST core. + +Algorithm must provide its name and above structure to the +dst_alloc_alg() function, which will return a reference to the newly +created algorithm. +To remove it, one needs to call dst_remove_alg() with given algorithm +pointer.
Documentation/dst/dst.txt new file +69 -0
69 lines not on this page · raw message
Documentation/dst/sysfs.txt new file +30 -0
@@ -0,0 +1,30 @@ +This file describes sysfs files created for each storage. + +1. Per-storage files. +Each storage has its own dir /sysfs/devices/$storage_name, +which contains following files: + +alg - contains name of the algorithm used to created given storage +name - name of the storage +nodes - map of the storage (list of nodes and their sizes and starts) +remove_all_nodes - writable file which allows to remove all nodes from given + storage +n-$start-$cookie - per node directory, where + $start - start of the given node in sectors, + $cookie - unique node's id used by DST + +2. Per-node files. +Node's files are located in /sysfs/devices/$storage_name/n-$start-$cookie +directory, described above. + +chunks - private file for mirroring algorithm, contains map of update/dirty + sectors of the node, '-' means update, '+' is dirty and has to be + resynced sector +clean - writable file, writing leads to marking node as clean (in sync) +dirty - writable file, writing leads to marking node as dirty (not in sync) +size - size of the given node in sectors +start - start of the given node in the storage in sectors +type - contains type of the node in the following format: $type: $dev + where $type is either 'L' or 'R' - local or remote acordingly, + and $dev is device name for local node (/dev/sda1 for example) + or address of the remote node (192.168.4.81:1025 for example)
PATCH 2/4 dst: Core distributed storage files.
Core distributed storage files.
Include userspace interfaces, initialization, block layer bindings and other core functionality.
Signed-off-by Evgeniy Polyakov <[email protected]>
drivers/block/Kconfig +2 -0
@@ -451,6 +451,8 @@ config ATA_OVER_ETH This driver provides Support for ATA over Ethernet block devices like the Coraid EtherDrive (R) Storage Blade. +source "drivers/block/dst/Kconfig" + source "drivers/s390/block/Kconfig" endmenu
drivers/block/Makefile +1 -0
@@ -29,3 +29,4 @@ obj-$(CONFIG_VIODASD) += viodasd.o obj-$(CONFIG_BLK_DEV_SX8) += sx8.o obj-$(CONFIG_BLK_DEV_UB) += ub.o +obj-$(CONFIG_DST) += dst/
drivers/block/dst/Kconfig new file +21 -0
@@ -0,0 +1,21 @@ +config DST + tristate "Distributed storage" + depends on NET + select CONNECTOR + select LIBCRC32C + ---help--- + This driver allows to create a distributed storage. + +config DST_ALG_LINEAR + tristate "Linear distribution algorithm" + depends on DST + ---help--- + This module allows to create linear mapping of the nodes + in the distributed storage. + +config DST_ALG_MIRROR + tristate "Mirror distribution algorithm" + depends on DST + ---help--- + This module allows to create a mirror of the noes in the + distributed storage.
drivers/block/dst/Makefile new file +6 -0
@@ -0,0 +1,6 @@ +obj-$(CONFIG_DST) += dst.o + +dst-y := dcore.o kst.o + +obj-$(CONFIG_DST_ALG_LINEAR) += alg_linear.o +obj-$(CONFIG_DST_ALG_MIRROR) += alg_mirror.o
drivers/block/dst/dcore.c new file +1608 -0
1,608 lines not on this page · raw message
include/linux/connector.h +3 -1
@@ -36,9 +36,11 @@#define CN_VAL_CIFS 0x1#define CN_W1_IDX 0x3/* w1 communication */ #define CN_W1_VAL 0x1 +#define CN_DST_IDX 0x4/* Distributed storage */ +#define CN_DST_VAL 0x1 -#define CN_NETLINK_USERS 4 +#define CN_NETLINK_USERS 5/* * Maximum connector's message size.
include/linux/dst.h new file +377 -0
377 lines not on this page · raw message
PATCH 3/4 dst: Network state machine.
Network state machine.
Includes network async processing state machine and related tasks.
Signed-off-by Evgeniy Polyakov <[email protected]>
drivers/block/dst/kst.c new file +1475 -0
@@ -0,0 +1,1475 @@ +/* +* 2007+ Copyright (c) Evgeniy Polyakov <[email protected]> +* All rights reserved. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +*/ + +#include <linux/kernel.h> +#include <linux/module.h> +#include <linux/list.h> +#include <linux/slab.h> +#include <linux/socket.h> +#include <linux/kthread.h> +#include <linux/net.h> +#include <linux/in.h> +#include <linux/poll.h> +#include <linux/bio.h> +#include <linux/dst.h> + +#include <net/sock.h> + +struct kst_poll_helper +{ +poll_table pt; +struct kst_state *st; +}; + +static LIST_HEAD (kst_worker_list); +static DEFINE_MUTEX (kst_worker_mutex); + +/* +* This function creates bound socket for local export node. +*/ +static int kst_sock_create (struct kst_state *st,struct saddr *addr, +int type,int proto,int backlog) +{ +int err; + + err =sock_create (addr->sa_family, type, proto, &st->socket); +if (err) + goto err_out_exit; + + err = st->socket->ops->bind (st->socket, (struct sockaddr *)addr, + addr->sa_data_len); + + err = st->socket->ops->listen (st->socket, backlog); +if (err) + goto err_out_release; + + st->socket->sk->sk_allocation =GFP_NOIO ; + +return 0 ; + +err_out_release: +sock_release (st->socket); +err_out_exit: +return err; +} + +static void kst_sock_release (struct kst_state *st) +{ +if (st->socket) { +sock_release (st->socket); + st->socket =NULL ; + } +} + +void kst_wake (struct kst_state *st) +{ +if (st) { +struct kst_worker *w = st->node->w; +unsigned long flags; + +spin_lock_irqsave (&w->ready_lock, flags); +if (list_empty (&st->ready_entry)) +list_add_tail (&st->ready_entry, &w->ready_list); +spin_unlock_irqrestore (&w->ready_lock, flags); + +wake_up (&w->wait); + } +} +EXPORT_SYMBOL_GPL (kst_wake); + +/* +* Polling machinery. +*/ +static int kst_state_wake_callback (wait_queue_t *wait,unsigned mode, +int sync,void *key) +{ + struct kst_state *st = container_of(wait, struct kst_state, wait); +kst_wake (st); +return 1 ; +} + +static void kst_queue_func (struct file *file,wait_queue_head_t *whead, +poll_table *pt) +{ + struct kst_state *st = container_of(pt, struct kst_poll_helper, pt)->st; + + st->whead = whead; +init_waitqueue_func_entry (&st->wait, kst_state_wake_callback); +add_wait_queue (whead, &st->wait); +} + +static void kst_poll_exit (struct kst_state *st) +{ +if (st->whead) { +remove_wait_queue (st->whead, &st->wait); + st->whead =NULL ; + } +} + +/* +* This function removes request from state tree and ordering list. +*/ +void kst_del_req (struct dst_request *req) +{ +list_del_init (&req->request_list_entry); +} +EXPORT_SYMBOL_GPL (kst_del_req); + +static struct dst_request *kst_req_first (struct kst_state *st) +{ +struct dst_request *req =NULL ; + +if (!list_empty (&st->request_list)) + req = list_entry(st->request_list.next, struct dst_request, + request_list_entry); +return req; +} + +/* +* This function dequeues first request from the queue and tree. +*/ +static struct dst_request *kst_dequeue_req (struct kst_state *st) +{ +struct dst_request *req; + +mutex_lock (&st->request_lock); + req =kst_req_first (st); +if (req) +kst_del_req (req); +mutex_unlock (&st->request_lock); +return req; +} + +/* +* This function enqueues request into tree, indexed by start of the request, +* and also puts request into ordered queue. +*/ +int kst_enqueue_req (struct kst_state *st,struct dst_request *req) +{ +if (unlikely (req->flags &DST_REQ_CHECK_QUEUE )) { +struct dst_request *r; + +list_for_each_entry (r, &st->request_list, request_list_entry) { +if (bio_rw (r->bio) !=bio_rw (req->bio)) +continue ; + +if (r->start >= req->start + req->size) +continue ; + +if (r->start + r->size <= req->start) +continue ; + +return -EEXIST ; + } + } + +list_add_tail (&req->request_list_entry, &st->request_list); +return 0 ; +} +EXPORT_SYMBOL_GPL (kst_enqueue_req); + +/* +* BIOs for local exporting node are freed via this function. +*/ +static void kst_export_put_bio (struct bio *bio) +{ +int i; +struct bio_vec *bv; + +dprintk ("%s: bio: %p, size: %u, idx: %d, num: %d, req: %p.\n" , + __func__, bio, bio->bi_size, bio->bi_idx, + bio->bi_vcnt, bio->bi_private); + +bio_for_each_segment (bv, bio, i) +__free_page (bv->bv_page); +bio_put (bio); +} + +/* +* This is a generic request completion function for requests, +* queued for async processing. +* If it is local export node, state machine is different, +* see details below. +*/ +void kst_complete_req (struct dst_request *req,int err) +{ +dprintk ("%s: bio: %p, req: %p, size: %llu, orig_size: %llu, " +"bi_size: %u, err: %d, flags: %u.\n" , + __func__, req->bio, req, req->size, req->orig_size, + req->bio->bi_size, err, req->flags); + +if (req->flags &DST_REQ_EXPORT ) { +if (err || !(req->flags &DST_REQ_EXPORT_WRITE )) { + req->bio_endio (req, err); + goto out; + } + + req->bio->bi_rw =WRITE ; +generic_make_request (req->bio); + }else { + req->bio_endio (req, err); + } +out: +dst_free_request (req); +} +EXPORT_SYMBOL_GPL (kst_complete_req); + +static void kst_flush_requests (struct kst_state *st) +{ +struct dst_request *req; + +while ((req =kst_dequeue_req (st)) !=NULL ) +kst_complete_req (req, -EIO ); +} + +static int kst_poll_init (struct kst_state *st) +{ +struct kst_poll_helper ph; + + ph.st = st; +init_poll_funcptr (&ph.pt, &kst_queue_func); + + st->socket->ops->poll (NULL , st->socket, &ph.pt); +return 0 ; +} + +/* +* Main state creation function. +* It creates new state according to given operations +* and links it into worker structure and node. +*/ +static struct kst_state *kst_state_init (struct dst_node *node, +unsigned int permissions, +struct kst_state_ops *ops,void *data) +{ +struct kst_state *st; +int err; + + st =kzalloc (sizeof (struct kst_state ),GFP_KERNEL ); +if (!st) +return ERR_PTR (-ENOMEM ); + + st->permissions = permissions; + st->node = node; + st->ops = ops; +INIT_LIST_HEAD (&st->ready_entry); +INIT_LIST_HEAD (&st->entry); +INIT_LIST_HEAD (&st->request_list); +mutex_init (&st->request_lock); + + err = st->ops->init (st, data); +if (err) + goto err_out_free; +mutex_lock (&node->w->state_mutex); +list_add_tail (&st->entry, &node->w->state_list); +mutex_unlock (&node->w->state_mutex); + +kst_wake (st); + +return st; + +err_out_free: +kfree (st); +return ERR_PTR (err); +} + +/* +* This function is called when node is removed, +* or when state is destroyed for connected to local exporting +* node client. +*/ +void kst_state_exit (struct kst_state *st) +{ +struct kst_worker *w = st->node->w; + +mutex_lock (&w->state_mutex); +list_del_init (&st->entry); +mutex_unlock (&w->state_mutex); + + st->ops->exit (st); + +if (st == st->node->state) + st->node->state =NULL ; + +kfree (st); +} + +static int kst_error (struct kst_state *st,int err) +{ +if ((err == -ECONNRESET || err == -EPIPE ) && st->ops->recovery) + err = st->ops->recovery (st, err); + +return st->node->st->alg->ops->error (st, err); +} + +/* +* This is main state processing function. +* It tries to complete request and invoke appropriate +* callbacks in case of errors or successfull operation finish. +*/ +static int kst_thread_process_state (struct kst_state *st) +{ +int err, empty; +unsigned int revents; +struct dst_request *req, *tmp; + +mutex_lock (&st->request_lock); +if (st->ops->ready) { + err = st->ops->ready (st); +if (err) { +mutex_unlock (&st->request_lock); +if (err <0 ) +kst_state_exit (st); +return err; + } + } + + err =0 ; + empty =1 ; + req =NULL ; +list_for_each_entry_safe (req, tmp, &st->request_list, request_list_entry) { + empty =0 ; + revents = st->socket->ops->poll (st->socket->file, + st->socket,NULL ); +if (!revents) +break ; + err = req->callback (req, revents); +if (req->size && !err) + err =1 ; + +if (err <0 || !req->size) { +if (!req->size) + err =0 ; +kst_del_req (req); +kst_complete_req (req, err); + } + +if (err) +break ; + } + +dprintk ("%s: broke the loop: err: %d, list_empty: %d.\n" , + __func__, err,list_empty (&st->request_list)); +mutex_unlock (&st->request_lock); + +if (err <0 ) { +dprintk ("%s: req: %p, err: %d, st: %p, node->state: %p.\n" , + __func__, req, err, st, st->node->state); + +if (st != st->node->state) { +/* +* Accepted client has state not related to storage +* node, so it must be freed explicitely. +* We do not try to fix clients connections to local +* export nodes, just drop the client. +*/ + +kst_state_exit (st); +return err; + } + + err =kst_error (st, err); +if (err) +return err; + +kst_wake (st); + } + +if (list_empty (&st->request_list) && !empty) +kst_wake (st); + +return err; +} + +/* +* Main worker thread - one per storage. +*/ +static int kst_thread_func (void *data) +{ +struct kst_worker *w = data; +struct kst_state *st; +unsigned long flags; +int err =0 ; + +while (!kthread_should_stop ()) { +wait_event_interruptible_timeout (w->wait, + !list_empty (&w->ready_list) || +kthread_should_stop (), +HZ ); + + st =NULL ; +spin_lock_irqsave (&w->ready_lock, flags); +if (!list_empty (&w->ready_list)) { + st = list_entry(w->ready_list.next, struct kst_state, + ready_entry); +list_del_init (&st->ready_entry); + } +spin_unlock_irqrestore (&w->ready_lock, flags); + +if (!st) +continue ; + + err =kst_thread_process_state (st); + } + +return err; +} + +/* +* Worker initialization - this object will host andprocess all states, +* which in turn host requests for remote targets. +*/ +struct kst_worker *kst_worker_init (int id) +{ +struct kst_worker *w; +int err; + + w =kzalloc (sizeof (struct kst_worker ),GFP_KERNEL ); +if (!w) +return ERR_PTR (-ENOMEM ); + + w->id = id; +init_waitqueue_head (&w->wait); +spin_lock_init (&w->ready_lock); +mutex_init (&w->state_mutex); + +INIT_LIST_HEAD (&w->ready_list); +INIT_LIST_HEAD (&w->state_list); + + w->req_pool =mempool_create_slab_pool (256 , dst_request_cache); +if (!w->req_pool) { + err = -ENOMEM ; + goto err_out_free; + } + + w->thread =kthread_run (&kst_thread_func, w,"kst%d" , w->id); +if (IS_ERR (w->thread)) { + err =PTR_ERR (w->thread); + goto err_out_destroy; + } + +mutex_lock (&kst_worker_mutex); +list_add_tail (&w->entry, &kst_worker_list); +mutex_unlock (&kst_worker_mutex); + +return w; + +err_out_destroy: +mempool_destroy (w->req_pool); +err_out_free: +kfree (w); +return ERR_PTR (err); +} + +void kst_worker_exit (struct kst_worker *w) +{ +struct kst_state *st, *n; + +mutex_lock (&kst_worker_mutex); +list_del (&w->entry); +mutex_unlock (&kst_worker_mutex); + +kthread_stop (w->thread); + +list_for_each_entry_safe (st, n, &w->state_list, entry) { +kst_state_exit (st); + } + +mempool_destroy (w->req_pool); +kfree (w); +} + +/* +* Common state exit callback. +* Removes itself from worker's list of states, +* releases socket and flushes all requests. +*/ +static void kst_common_exit (struct kst_state *st) +{ +unsigned long flags; + +kst_poll_exit (st); + +spin_lock_irqsave (&st->node->w->ready_lock, flags); +list_del_init (&st->ready_entry); +spin_unlock_irqrestore (&st->node->w->ready_lock, flags); + +kst_flush_requests (st); +kst_sock_release (st); +} + +/* +* Listen socket contains security attributes in request_list, +* so it can not be flushed via usual way. +*/ +static void kst_listen_flush (struct kst_state *st) +{ +struct dst_secure *s, *tmp; + +list_for_each_entry_safe (s, tmp, &st->request_list, sec_entry) { +list_del (&s->sec_entry); +kfree (s); + } +} + +static void kst_listen_exit (struct kst_state *st) +{ +kst_listen_flush (st); +kst_common_exit (st); +} + +/* +* BIO vector receiving function - does not block, but may sleep because +* of scheduling policy. +*/ +static int kst_data_recv_bio_vec (struct kst_state *st,struct bio_vec *bv, +unsigned int offset,unsigned int size) +{ +struct msghdr msg; +struct kvec iov; +void *kaddr; +int err; + + kaddr =kmap (bv->bv_page); + + iov.iov_base = kaddr + bv->bv_offset + offset; + iov.iov_len = size; + + msg.msg_iov = (struct iovec *)&iov; + msg.msg_iovlen =1 ; + msg.msg_name =NULL ; + msg.msg_namelen =0 ; + msg.msg_control =NULL ; + msg.msg_controllen =0 ; + msg.msg_flags =MSG_DONTWAIT |MSG_NOSIGNAL ; + + err =kernel_recvmsg (st->socket, &msg, &iov,1 , iov.iov_len, + msg.msg_flags); +kunmap (bv->bv_page); + +return err; +} + +/* +* BIO vector sending function - does not block, but may sleep because +* of scheduling policy. +*/ +static int kst_data_send_bio_vec (struct kst_state *st,struct bio_vec *bv, +unsigned int offset,unsigned int size) +{ +return kernel_sendpage (st->socket, bv->bv_page, + bv->bv_offset + offset, size, +MSG_DONTWAIT |MSG_NOSIGNAL ); +} + +static u32 dst_csum_bvec (struct bio_vec *bv,unsigned int offset,unsigned int size) +{ +void *addr; +u32 csum; + + addr =kmap_atomic (bv->bv_page,KM_USER0 ); + csum =dst_csum_data (addr + bv->bv_offset + offset, size); +kunmap_atomic (addr,KM_USER0 ); + +return csum; +} + +typedef int (*kst_data_process_bio_vec_t )(struct kst_state *st, +struct bio_vec *bv,unsigned int offset,unsigned int size); + +/* +* @req: processing request. +* Contains BIO and all related to its processing info. +* +* This function sends or receives requested number of pages from given BIO. +* +* In case of errors negative value is returned and @size, +* @index and @off are set to the: +* - number of bytes not yet processed (i.e. the rest of the bytes to be +* processed). +* - index of the last bio_vec started to be processed (header sent). +* - offset of the first byte to be processed in the bio_vec. +* +* If there are no errors, zero is returned. +* -EAGAIN is not an error and is transformed into zero return value, +* called must check if @size is zero, in that case whole BIO is processed +* and thus req->bio_endio() can be called, othervise new request must be allocated +* to be processed later. +*/ +static int kst_data_process_bio (struct dst_request *req) +{ +int err = -ENOSPC ; +struct dst_remote_request r; +kst_data_process_bio_vec_t func; +unsigned int cur_size; +int use_csum =test_bit (DST_NODE_USE_CSUM , &req->node->flags); + +if (bio_rw (req->bio) ==WRITE ) { + r.cmd =cpu_to_be32 (DST_WRITE ); + func = kst_data_send_bio_vec; + }else { + r.cmd =cpu_to_be32 (DST_READ ); + func = kst_data_recv_bio_vec; + } + +dprintk ("%s: start: [%c], start: %llu, idx: %d, num: %d, " +"size: %llu, offset: %u, flags: %x, use_csum: %d.\n" , + __func__, (bio_rw (req->bio) ==WRITE )?'W' :'R' , + req->start, req->idx, req->num, req->size, req->offset, + req->flags, use_csum); + +while (req->idx < req->num) { +struct bio_vec *bv =bio_iovec_idx (req->bio, req->idx); + + cur_size =min_t (u64, bv->bv_len - req->offset, req->size); + +dprintk ("%s: page: %p, off: %u, len: %u, req->offset: %u, " +"req->size: %llu, cur_size: %u, flags: %x, " +"use_csum: %d, req->csum: %x.\n" , + __func__, bv->bv_page, bv->bv_offset, bv->bv_len, + req->offset, req->size, cur_size, + req->flags, use_csum, req->tmp_csum); + +if (cur_size ==0 ) { +printk (KERN_ERR "%s: %d/%d: start: %llu, " +"bv_offset: %u, bv_len: %u, " +"req_offset: %u, req_size: %llu, " +"req: %p, bio: %p, err: %d.\n" , + __func__, req->idx, req->num, req->start, + bv->bv_offset, bv->bv_len, + req->offset, req->size, + req, req->bio, err); +BUG (); + } + +if (!(req->flags &DST_REQ_HEADER_SENT )) { + r.sector =cpu_to_be64 (req->start); + r.offset =cpu_to_be32 (bv->bv_offset + req->offset); + r.size =cpu_to_be32 (cur_size); + r.csum =0 ; + +if (use_csum &&bio_rw (req->bio) ==WRITE && + !req->tmp_offset) { + req->tmp_offset = req->offset; + r.csum =cpu_to_be32 (dst_csum_bvec (bv, + req->offset, cur_size)); + } + + err =dst_data_send_header (req->state->socket, &r); +dprintk ("%s: %d/%d: sending header: cmd: %u, start: %llu, " +"bv_offset: %u, bv_len: %u, " +"a offset: %u, offset: %u, " +"cur_size: %u, err: %d.\n" , + __func__, req->idx, req->num,be32_to_cpu (r.cmd), + req->start, bv->bv_offset, bv->bv_len, + bv->bv_offset + req->offset, + req->offset, cur_size, err); + +if (err !=sizeof (struct dst_remote_request )) { +if (err >=0 ) + err = -EINVAL ; +break ; + } + + req->flags |=DST_REQ_HEADER_SENT ; + } + +if (use_csum && (bio_rw (req->bio) !=WRITE ) && + !(req->flags &DST_REQ_CHEKSUM_RECV )) { +struct dst_remote_request tmp_req; + + err =dst_data_recv_header (req->state->socket, &tmp_req,0 ); +dprintk ("%s: %d/%d: receiving header: start: %llu, " +"bv_offset: %u, bv_len: %u, " +"a offset: %u, offset: %u, " +"cur_size: %u, err: %d.\n" , + __func__, req->idx, req->num, + req->start, bv->bv_offset, bv->bv_len, + bv->bv_offset + req->offset, + req->offset, cur_size, err); + +if (err !=sizeof (struct dst_remote_request )) { +if (err >=0 ) + err = -EINVAL ; +break ; + } + +if (req->tmp_csum) { +printk ("%s: req: %p, old csum: %x, new: %x.\n" , + __func__, req, req->tmp_csum, +be32_to_cpu (tmp_req.csum)); +BUG_ON (1 ); + } + +dprintk ("%s: req: %p, old csum: %x, new: %x.\n" , + __func__, req, req->tmp_csum, +be32_to_cpu (tmp_req.csum)); + req->tmp_csum =be32_to_cpu (tmp_req.csum); + + req->flags |=DST_REQ_CHEKSUM_RECV ; + } + + err =func (req->state, bv, req->offset, cur_size); +if (err <=0 ) +break ; + + req->offset += err; + req->size -= err; + +if (req->offset != bv->bv_len) { +dprintk ("%s: %d/%d: this: start: %llu, bv_offset: %u, " +"bv_len: %u, offset: %u, " +"cur_size: %u, err: %d.\n" , + __func__, req->idx, req->num, req->start, + bv->bv_offset, bv->bv_len, + req->offset, cur_size, err); + err = -EAGAIN ; +break ; + } + +if (use_csum &&bio_rw (req->bio) !=WRITE ) { +u32 csum =dst_csum_bvec (bv, req->tmp_offset, + bv->bv_len - req->tmp_offset); + +dprintk ("%s: req: %p, csum: %x, received csum: %x.\n" , + __func__, req, csum, req->tmp_csum); + +if (csum != req->tmp_csum) { +printk ("%s: %d/%d: broken checksum: start: %llu, " +"bv_offset: %u, bv_len: %u, " +"a offset: %u, offset: %u, " +"cur_size: %u, orig_size: %llu.\n" , + __func__, req->idx, req->num, + req->start, bv->bv_offset, bv->bv_len, + bv->bv_offset + req->offset, + req->offset, cur_size, req->orig_size); +printk ("%s: broken checksum: req: %p, csum: %x, " +"should be: %x, flags: %x, " +"req->tmp_offset: %u, rw: %lu.\n" , + __func__, req, csum, req->tmp_csum, + req->flags, req->tmp_offset,bio_rw (req->bio)); + + req->offset -= err; + req->size += err; + + err = -EREMOTEIO ; +break ; + } + } + + req->offset =0 ; + req->idx++; + req->flags &= ~(DST_REQ_HEADER_SENT |DST_REQ_CHEKSUM_RECV ); + req->tmp_csum =0 ; + req->start +=to_sector (bv->bv_len); + } + +if (err <=0 && err != -EAGAIN ) { +if (err ==0 ) + err = -ECONNRESET ; + }else + err =0 ; + +if (err <0 || (req->idx == req->num && req->size)) { +dprintk ("%s: return: idx: %d, num: %d, offset: %u, " +"size: %llu, err: %d.\n" , + __func__, req->idx, req->num, req->offset, + req->size, err); + } +dprintk ("%s: end: start: %llu, idx: %d, num: %d, " +"size: %llu, offset: %u.\n" , + __func__, req->start, req->idx, req->num, + req->size, req->offset); + +return err; +} + +void kst_bio_endio (struct dst_request *req,int err) +{ +if (err &&printk_ratelimit ()) +printk ("%s: freeing bio: %p, bi_size: %u, " +"orig_size: %llu, req: %p, err: %d.\n" , + __func__, req->bio, req->bio->bi_size, req->orig_size, + req, err); +bio_endio (req->bio, req->orig_size, err); +} +EXPORT_SYMBOL_GPL (kst_bio_endio); + +/* +* This callback is invoked by worker thread to process given request. +*/ +int kst_data_callback (struct dst_request *req,unsigned int revents) +{ +int err; + +dprintk ("%s: req: %p, num: %d, idx: %d, bio: %p, " +"revents: %x, flags: %x.\n" , + __func__, req, req->num, req->idx, req->bio, + revents, req->flags); + +if (req->flags &DST_REQ_EXPORT_READ ) +return 1 ; + + err =kst_data_process_bio (req); + +if (revents & (POLLERR |POLLHUP |POLLRDHUP )) + err = -EPIPE ; + +return err; +} +EXPORT_SYMBOL_GPL (kst_data_callback); + +struct dst_request *dst_clone_request (struct dst_request *req,mempool_t *pool) +{ +struct dst_request *new_req; + + new_req =mempool_alloc (pool,GFP_NOIO ); +if (!new_req) +return NULL ; + +memset (new_req,0 ,sizeof (struct dst_request )); + +dprintk ("%s: req: %p, new_req: %p.\n" , __func__, req, new_req); + +if (req) { + new_req->bio = req->bio; + new_req->state = req->state; + new_req->node = req->node; + new_req->idx = req->idx; + new_req->num = req->num; + new_req->size = req->size; + new_req->orig_size = req->orig_size; + new_req->offset = req->offset; + new_req->tmp_offset = req->tmp_offset; + new_req->tmp_csum = req->tmp_csum; + new_req->start = req->start; + new_req->flags = req->flags; + new_req->bio_endio = req->bio_endio; + new_req->priv = req->priv; + } + +return new_req; +} +EXPORT_SYMBOL_GPL (dst_clone_request); + +void dst_free_request (struct dst_request *req) +{ +dprintk ("%s: free req: %p, pool: %p, bio: %p, state: %p, node: %p.\n" , + __func__, req, req->node->w->req_pool, + req->bio, req->state, req->node); +mempool_free (req, req->node->w->req_pool); +} +EXPORT_SYMBOL_GPL (dst_free_request); + +/* +* This is main data processing function, eventually invoked from block layer. +* It tries to complte request, but if it is about to block, it allocates +* new request and queues it to main worker to be processed when events allow. +*/ +static int kst_data_push (struct dst_request *req) +{ +struct kst_state *st = req->state; +struct dst_request *new_req; +unsigned int revents; +int err, locked =0 ; + +dprintk ("%s: start: %llu, size: %llu, bio: %p.\n" , + __func__, req->start, req->size, req->bio); + +if (!list_empty (&st->request_list) || (req->flags &DST_REQ_ALWAYS_QUEUE )) + goto alloc_new_req; + +if (mutex_trylock (&st->request_lock)) { + locked =1 ; + +if (!list_empty (&st->request_list)) + goto alloc_new_req; + + revents = st->socket->ops->poll (NULL , st->socket,NULL ); +if (revents &POLLOUT ) { + err =kst_data_process_bio (req); +if (err <0 ) + goto out_unlock; + +if (!req->size) + goto out_bio_endio; + } + } + +alloc_new_req: + err = -ENOMEM ; + new_req =dst_clone_request (req, req->node->w->req_pool); +if (!new_req) + goto out_unlock; + + new_req->callback = &kst_data_callback; + +if (!locked) +mutex_lock (&st->request_lock); + + locked =1 ; + + err =kst_enqueue_req (st, new_req); +if (err) + goto out_unlock; +mutex_unlock (&st->request_lock); + + err =0 ; + goto out; + +out_bio_endio: + req->bio_endio (req, err); +out_unlock: +if (locked) +mutex_unlock (&st->request_lock); + locked =0 ; + +if (err) { + err =kst_error (st, err); +if (!err) + goto alloc_new_req; + } + +if (err &&printk_ratelimit ()) { +printk ("%s: error [%c], start: %llu, idx: %d, num: %d, " +"size: %llu, offset: %u, err: %d.\n" , + __func__, (bio_rw (req->bio) ==WRITE )?'W' :'R' , + req->start, req->idx, req->num, req->size, + req->offset, err); + } + +out: + +kst_wake (st); +return err; +} + +/* +* Remote node initialization callback. +*/ +static int kst_data_init (struct kst_state *st,void *data) +{ +int err; + + st->socket = data; + st->socket->sk->sk_allocation =GFP_NOIO ; +/* +* Why not? +*/ + st->socket->sk->sk_sndbuf = st->socket->sk->sk_sndbuf =1024 *1024 *10 ; + + err =kst_poll_init (st); +if (err) +return err; + +return 0 ; +} + +/* +* Remote node recovery function - tries to reconnect to given target. +*/ +static int kst_data_recovery (struct kst_state *st,int err) +{ +struct socket *sock; +struct sockaddr addr; +int addrlen; +struct dst_request *req; + +if (err != -ECONNRESET && err != -EPIPE ) { +dprintk ("%s: state %p does not know how " +"to recover from error %d.\n" , + __func__, st, err); +return err; + } + + err =sock_create (st->socket->ops->family, st->socket->type, + st->socket->sk->sk_protocol, &sock); +if (err <0 ) + goto err_out_exit; + + sock->sk->sk_sndtimeo = sock->sk->sk_rcvtimeo = +msecs_to_jiffies (DST_DEFAULT_TIMEO ); + + err = sock->ops->getname (st->socket, &addr, &addrlen,2 ); +if (err) + goto err_out_destroy; + + err = sock->ops->connect (sock, &addr, addrlen,0 ); +if (err) + goto err_out_destroy; + +kst_poll_exit (st); +kst_sock_release (st); + +mutex_lock (&st->request_lock); + err = st->ops->init (st, sock); +if (!err) { +/* +* After reconnection is completed all requests +* must be resent from the state they were finished previously, +* but with new headers. +*/ +list_for_each_entry (req, &st->request_list, request_list_entry) + req->flags &= ~(DST_REQ_HEADER_SENT |DST_REQ_CHEKSUM_RECV ); + } +mutex_unlock (&st->request_lock); +if (err <0 ) + goto err_out_destroy; + +kst_wake (st); +dprintk ("%s: reconnected.\n" , __func__); + +return 0 ; + +err_out_destroy: +sock_release (sock); +err_out_exit: +dprintk ("%s: revovery failed: st: %p, err: %d.\n" , __func__, st, err); +return err; +} + +/* +* Local exporting node end IO callbacks. +*/ +static int kst_export_write_end_io (struct bio *bio,unsigned int size,int err) +{ +dprintk ("%s: bio: %p, size: %u, idx: %d, num: %d, err: %d.\n" , + __func__, bio, bio->bi_size, bio->bi_idx, bio->bi_vcnt, err); + +if (bio->bi_size) +return 1 ; + +kst_export_put_bio (bio); +return 0 ; +} + +static int kst_export_read_end_io (struct bio *bio,unsigned int size,int err) +{ +struct dst_request *req = bio->bi_private; +struct kst_state *st = req->state; +int use_csum =test_bit (DST_NODE_USE_CSUM , &req->node->flags); + +dprintk ("%s: bio: %p, req: %p, size: %u, idx: %d, num: %d, err: %d.\n" , + __func__, bio, req, bio->bi_size, bio->bi_idx, + bio->bi_vcnt, err); + +if (bio->bi_size) +return 1 ; + +if (err) { +kst_export_put_bio (bio); +return 0 ; + } + + bio->bi_size = req->size = req->orig_size; + bio->bi_rw =WRITE ; +if (use_csum) + req->flags &= ~(DST_REQ_HEADER_SENT |DST_REQ_CHEKSUM_RECV ); + +/* +* This is a race with kst_data_callback(), which checks +* this bit to determine if it can or can not process given +* request. This does not harm actually, since subsequent +* state wakeup will call it again and thus will pick +* given request in time. +*/ + req->flags &= ~DST_REQ_EXPORT_READ ; +kst_wake (st); +return 0 ; +} + +/* +* This callback is invoked each time new request from remote +* node to given local export node is received. +* It allocates new block IO request and queues it for processing. +*/ +static int kst_export_ready (struct kst_state *st) +{ +struct dst_remote_request r; +struct bio *bio; +int err, nr, i; +struct dst_request *req; +unsigned int revents = st->socket->ops->poll (NULL , st->socket,NULL ); + +if (revents & (POLLERR |POLLHUP )) { + err = -EPIPE ; + goto err_out_exit; + } + +if (!(revents &POLLIN ) || !list_empty (&st->request_list)) +return 0 ; + + err =dst_data_recv_header (st->socket, &r,1 ); +if (err !=sizeof (struct dst_remote_request )) { + err = -ECONNRESET ; + goto err_out_exit; + } + +kst_convert_header (&r); + +dprintk ("\n%s: st: %p, cmd: %u, sector: %llu, size: %u, " +"csum: %x, offset: %u.\n" , + __func__, st, r.cmd, r.sector, + r.size, r.csum, r.offset); + + err = -EINVAL ; +if (r.cmd !=DST_READ && r.cmd !=DST_WRITE && r.cmd !=DST_REMOTE_CFG ) + goto err_out_exit; + +if ((s64)(r.sector +to_sector (r.size)) <0 || + (r.sector +to_sector (r.size)) > st->node->size || + r.offset >=PAGE_SIZE ) + goto err_out_exit; + +if (r.cmd ==DST_REMOTE_CFG ) { + r.sector = st->node->size; + +if (test_bit (DST_NODE_USE_CSUM , &st->node->flags)) + r.csum =1 ; + +kst_convert_header (&r); + + err =dst_data_send_header (st->socket, &r); +if (err !=sizeof (struct dst_remote_request )) { + err = -EINVAL ; + goto err_out_exit; + } +kst_wake (st); +return 0 ; + } + + nr =DIV_ROUND_UP (r.size,PAGE_SIZE ); + +while (r.size) { +int nr_pages =min (BIO_MAX_PAGES , nr); +unsigned int size; +struct page *page; + + err = -ENOMEM ; + req =dst_clone_request (NULL , st->node->w->req_pool); +if (!req) + goto err_out_exit; + + bio =bio_alloc (GFP_NOIO , nr_pages); +if (!bio) + goto err_out_free_req; + + req->flags =DST_REQ_EXPORT |DST_REQ_HEADER_SENT | +DST_REQ_CHEKSUM_RECV ; + req->bio = bio; + req->state = st; + req->node = st->node; + req->callback = &kst_data_callback; + req->bio_endio = &kst_bio_endio; + + req->tmp_offset =0 ; + req->tmp_csum = r.csum; + +/* +* Yes, looks a bit weird. +* Logic is simple - for local exporting node all operations +* are reversed compared to usual nodes, since usual nodes +* process remote data and local export node process remote +* requests, so that writing data means sending data to +* remote node and receiving on the local export one. +* +* So, to process writing to the exported node we need first +* to receive data from the net (i.e. to perform READ +* operationin terms of usual node), and then put it to the +* storage (WRITE command, so it will be changed before +* calling generic_make_request()). +* +* To process read request from the exported node we need +* first to read it from storage (READ command for BIO) +* and then send it over the net (perform WRITE operation +* in terms of network). +*/ +if (r.cmd ==DST_WRITE ) { + req->flags |=DST_REQ_EXPORT_WRITE ; + bio->bi_end_io = kst_export_write_end_io; + }else { + req->flags |=DST_REQ_EXPORT_READ ; + bio->bi_end_io = kst_export_read_end_io; + } + bio->bi_rw =READ ; + bio->bi_private = req; + bio->bi_sector = r.sector; + bio->bi_bdev = st->node->bdev; + +for (i =0 ; i < nr_pages; ++i) { + page =alloc_page (GFP_NOIO ); +if (!page) +break ; + + size =min_t (u32,PAGE_SIZE - r.offset, r.size); + + err =bio_add_page (bio, page, size,0 ); +dprintk ("%s: %d/%d: page: %p, size: %u, " +"offset: %u (used zero), err: %d.\n" , + __func__, i, nr_pages, page, size, + r.offset, err); +if (err <=0 ) +break ; + +if (err == size) + nr--; + + r.size -= err; + r.sector +=to_sector (err); + +if (!r.size) +break ; + } + +if (!bio->bi_vcnt) { + err = -ENOMEM ; + goto err_out_put; + } + + req->size = req->orig_size = bio->bi_size; + req->start = bio->bi_sector; + req->idx =0 ; + req->num = bio->bi_vcnt; + +dprintk ("%s: submitting: bio: %p, req: %p, start: %llu, " +"size: %llu, idx: %d, num: %d, offset: %u, csum: %x.\n" , + __func__, bio, req, req->start, req->size, + req->idx, req->num, req->offset, req->tmp_csum); + + err =kst_enqueue_req (st, req); +if (err) + goto err_out_put; + +if (r.cmd ==DST_READ ) { +generic_make_request (bio); + } + } + +kst_wake (st); +return 0 ; + +err_out_put: +bio_put (bio); +err_out_free_req: +dst_free_request (req); +err_out_exit: +return err; +} + +static void kst_export_exit (struct kst_state *st) +{ +struct dst_node *n = st->node; + +kst_common_exit (st); +dst_node_put (n); +} + +static struct kst_state_ops kst_data_export_ops = { + .init = &kst_data_init, + .push = &kst_data_push, + .exit = &kst_export_exit, + .ready = &kst_export_ready, +}; + +/* +* This callback is invoked each time listening socket for +* given local export node becomes ready. +* It creates new state for connected client and queues for processing. +*/ +static int kst_listen_ready (struct kst_state *st) +{ +struct socket *newsock; +struct saddr addr; +struct kst_state *newst; +int err; +unsigned int revents, permissions =0 ; +struct dst_secure *s; + + revents = st->socket->ops->poll (NULL , st->socket,NULL ); +if (!(revents &POLLIN )) +return 1 ; + + err =sock_create (st->socket->ops->family, st->socket->type, + st->socket->sk->sk_protocol, &newsock); +if (err) + goto err_out_exit; + + err = st->socket->ops->accept (st->socket, newsock,0 ); +if (err) + goto err_out_put; + +if (newsock->ops->getname (newsock, (struct sockaddr *)&addr, + (int *)&addr.sa_data_len,2 ) <0 ) { + err = -ECONNABORTED ; + goto err_out_put; + } + +list_for_each_entry (s, &st->request_list, sec_entry) { +void *sec_addr, *new_addr; + + sec_addr = ((void *)&s->sec.addr) + s->sec.check_offset; + new_addr = ((void *)&addr) + s->sec.check_offset; + +if (!memcmp (sec_addr, new_addr, + addr.sa_data_len - s->sec.check_offset)) { + permissions = s->sec.permissions; +break ; + } + } + +/* +* So far only reading and writing are supported. +* Block device does not know about anything else, +* but as far as I recall, there was a prognosis, +* that computer will never require more than 640kb of RAM. +*/ +if (permissions ==0 ) { + err = -EPERM ; + goto err_out_put; + } + +if (st->socket->ops->family ==AF_INET ) { +struct sockaddr_in *sin = (struct sockaddr_in *)&addr; +printk (KERN_INFO "%s: Client: %u.%u.%u.%u:%d.\n" , __func__, +NIPQUAD (sin->sin_addr.s_addr),ntohs (sin->sin_port)); + }else if (st->socket->ops->family ==AF_INET6 ) { +struct sockaddr_in6 *sin = (struct sockaddr_in6 *)&addr; +printk (KERN_INFO "%s: Client: " +"%04x:%04x:%04x:%04x:%04x:%04x:%04x:%04x:%d" , + __func__, +NIP6 (sin->sin6_addr),ntohs (sin->sin6_port)); + } + +dst_node_get (st->node); + newst =kst_state_init (st->node, permissions, + &kst_data_export_ops, newsock); +if (IS_ERR (newst)) { + err =PTR_ERR (newst); + goto err_out_put; + } + +/* +* Negative return value means error, positive - stop this state +* processing. Zero allows to check state for pending requests. +* Listening socket contains security objects in request list, +* since it does not have any requests. +*/ +return 1 ; + +err_out_put: +sock_release (newsock); +err_out_exit: +return 1 ; +} + +static int kst_listen_init (struct kst_state *st,void *data) +{ +int err = -ENOMEM , i; +struct dst_le_template *tmp = data; +struct dst_secure *s; + +for (i=0 ; i<tmp->le->secure_attr_num; ++i) { + s =kmalloc (sizeof (struct dst_secure ),GFP_KERNEL ); +if (!s) + goto err_out_exit; + +memcpy (&s->sec, tmp->data,sizeof (struct dst_secure_user )); + +list_add_tail (&s->sec_entry, &st->request_list); + tmp->data +=sizeof (struct dst_secure_user ); + +if (s->sec.addr.sa_family ==AF_INET ) { +struct sockaddr_in *sin = + (struct sockaddr_in *)&s->sec.addr; +printk (KERN_INFO "%s: Client: %u.%u.%u.%u:%d, " +"permissions: %x.\n" , + __func__,NIPQUAD (sin->sin_addr.s_addr), +ntohs (sin->sin_port), s->sec.permissions); + }else if (s->sec.addr.sa_family ==AF_INET6 ) { +struct sockaddr_in6 *sin = + (struct sockaddr_in6 *)&s->sec.addr; +printk (KERN_INFO "%s: Client: " +"%04x:%04x:%04x:%04x:%04x:%04x:%04x:%04x:%d, " +"permissions: %x.\n" , + __func__,NIP6 (sin->sin6_addr), +ntohs (sin->sin6_port), s->sec.permissions); + } + } + + err =kst_sock_create (st, &tmp->le->rctl.addr, tmp->le->rctl.type, + tmp->le->rctl.proto, tmp->le->backlog); +if (err) + goto err_out_exit; + + err =kst_poll_init (st); +if (err) + goto err_out_release; + +return 0 ; + +err_out_release: +kst_sock_release (st); +err_out_exit: +kst_listen_flush (st); +return err; +} + +/* +* Operations for different types of states. +* There are three: +* data state - created for remote node, when distributed storage connects +* to remote node, which contain data. +* listen state - created for local export node, when remote distributed +* storage's node connects to given node to get/put data. +* data export state - created for each client connected to above listen +* state. +*/ +static struct kst_state_ops kst_listen_ops = { + .init = &kst_listen_init, + .exit = &kst_listen_exit, + .ready = &kst_listen_ready, +}; +static struct kst_state_ops kst_data_ops = { + .init = &kst_data_init, + .push = &kst_data_push, + .exit = &kst_common_exit, + .recovery = &kst_data_recovery, +}; + +struct kst_state *kst_listener_state_init (struct dst_node *node, +struct dst_le_template *tmp) +{ +return kst_state_init (node,DST_PERM_READ |DST_PERM_WRITE , + &kst_listen_ops, tmp); +} + +struct kst_state *kst_data_state_init (struct dst_node *node, +struct socket *newsock) +{ +return kst_state_init (node,DST_PERM_READ |DST_PERM_WRITE , + &kst_data_ops, newsock); +} + +/* +* Remove all workers and associated states. +*/ +void kst_exit_all (void ) +{ +struct kst_worker *w, *n; + +list_for_each_entry_safe (w, n, &kst_worker_list, entry) { +kst_worker_exit (w); + } +}
PATCH 4/4 dst: Algorithms used in distributed storage.
Algorithms used in distributed storage.
Mirror and linear mapping code.
Signed-off-by Evgeniy Polyakov <[email protected]>
drivers/block/dst/alg_linear.c new file +104 -0
@@ -0,0 +1,104 @@ +/* +* 2007+ Copyright (c) Evgeniy Polyakov <[email protected]> +* All rights reserved. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +*/ + +#include <linux/module.h> +#include <linux/kernel.h> +#include <linux/init.h> +#include <linux/dst.h> + +static struct dst_alg *alg_linear; + +/* +* This callback is invoked when node is removed from storage. +*/ +static void dst_linear_del_node (struct dst_node *n) +{ +} + +/* +* This callback is invoked when node is added to storage. +*/ +static int dst_linear_add_node (struct dst_node *n) +{ +struct dst_storage *st = n->st; + +dprintk ("%s: disk_size: %llu, node_size: %llu.\n" , + __func__, st->disk_size, n->size); + +mutex_lock (&st->tree_lock); + n->start = st->disk_size; + st->disk_size += n->size; +mutex_unlock (&st->tree_lock); + +return 0 ; +} + +static int dst_linear_remap (struct dst_request *req) +{ +int err; + +if (req->node->bdev) { +generic_make_request (req->bio); +return 0 ; + } + + err =kst_check_permissions (req->state, req->bio); +if (err) +return err; + +return req->state->ops->push (req); +} + +/* +* Failover callback - it is invoked each time error happens during +* request processing. +*/ +static int dst_linear_error (struct kst_state *st,int err) +{ +if (err) +set_bit (DST_NODE_FROZEN , &st->node->flags); +else +clear_bit (DST_NODE_FROZEN , &st->node->flags); +return 0 ; +} + +static struct dst_alg_ops alg_linear_ops = { + .remap = dst_linear_remap, + .add_node = dst_linear_add_node, + .del_node = dst_linear_del_node, + .error = dst_linear_error, + .owner =THIS_MODULE , +}; + +static int __devinit alg_linear_init (void ) +{ + alg_linear =dst_alloc_alg ("alg_linear" , &alg_linear_ops); +if (!alg_linear) +return -ENOMEM ; + +return 0 ; +} + +static void __devexit alg_linear_exit(void) +{ +dst_remove_alg (alg_linear); +} + +module_init (alg_linear_init); +module_exit (alg_linear_exit); + +MODULE_LICENSE ("GPL" ); +MODULE_AUTHOR ("Evgeniy Polyakov <[email protected]>" ); +MODULE_DESCRIPTION ("Linear distributed algorithm." );
drivers/block/dst/alg_mirror.c new file +1113 -0
1,113 lines not on this page · raw message