Skip to contentSkip to conversation

Current state of MARS

RFC New Thomas Schoebel-Theuer 1 participant 32 messages Dec 31, 2015 Staging Page 1 of 2

lore
Share & tools
lore
#0 lore raw

cover Current state of MARS

MARS Light is an asynchronous replication system for block storage over long distances [1,2,3]. It is the base for HA over long distances (more than 50km) and over network bottlenecks, e.g. high / varying packet loss rates.

The out-of-tree (OOT) version of MARS (branch light0.1.y) is in production at 1&1 Internet SE, currently at more than 1600 servers of various types, more than 6 petabytes of total storage, and has collected more than 6 millions of operating hours with less than 10 customer-visible incidents caused by MARS (most of them in the early beta phase).

Quoted 1 line

From my last LKML posting in 2014, I got the following major TODOs

from you:

1) no kernel prepatch anymore, no additional EXPORT_SYMBOL()s anywhere in the rest of the kernel.

=> accomplished in the enclosed patchset (however not yet fully bug-free). See also branches WIP-PORTABLE, WIP-BASE and WIP-PROPOSED-UPSTREAM at [1] for some out-of-tree (OOT) versions of MARS.

=> this means there will be zero impact from MARS to the rest of the kernel when MARS is not used. Additionally I placed it into drivers/staging/ (configurabale via migration script ./rework-mars-for-upstream.pl ).

2) replace the symlink tree by something else.

=> not yet done, due to lack of time. Sorry. I am planning to work on it in the forthcoming year, and hope to get enough time for it.

I think the attached patchset is not yet ready for submission, but please help me by giving early feedback. I would be very glad if some experienced upstream hacker would mentor me and help me at various points.

Here are some of my thoughts about further development:

In future, I would like to store the old symlink information (list of key => value pairs) in status files instead, one instance for each resource, and one for global configuration data (notice that the information _needs_ to be persistent for coping with node failures, and that dynamic storage is needed anyway for huge masses of transaction logfiles).

This should enhance scalability in the distributed system. Here is a use case: almost our full 1&1 webhosting machine park has been migrated from DRBD to MARS, retaining the conventional pair structure. Thus more than 800 DRBD clusters have been turned into MARS clusters.

In future, the whole 1&1 machine park could collapse into a _single_ big cluster consisting of thousands of machines. This has some advantages, not limited to greater flexibility with dynamic join-resource / leave-resource operations throughout that big cluster. For example, userspace can more easily detect failures cluster-wide and datacenter-wide, because the Lamport Clock algorithm used by MARS [2,3] is already a kind of "heartbeat", even remembering the timestamps of last successful information exchange. Thus it should be relatively easy to implement quorum algorithms etc on top of it in userspace.

Note that I am currently paid by 1&1 to ensure a seamless upgrade path for servers to any new data format, without downtime (other than temporarily switching the primary roles).

If you don't want the old symlink format in the upstream kernel, I would first implement the new format OOT (and strip out the old code for submission). Otherwise, I would agree to do this intree, hopefully with help and advise from some interested kernel hackers.

Cheers and a happy new year,

Thomas

[1] https://github.com/schoebel/mars
[2] https://github.com/schoebel/mars/blob/master/docu/MARS_Froscon2015.pdf
[3] https://github.com/schoebel/mars/blob/master/docu/mars-manual.pdf

Thomas Schoebel-Theuer (31):
mars: add new module lamport
mars: add new module brick_say
mars: add new module brick_mem
mars: add new module brick_checking
mars: add new module meta
mars: add new module brick
mars: add new module lib_pairing_heap
mars: add new module lib_queue
mars: add new module lib_rank
mars: add new module lib_limiter
mars: add new module lib_timing
mars: add new module vfs_compat
mars: add new module xio
mars: add new module xio_net
mars: add new module lib_mapfree
mars: add new module lib_log
mars: add new module xio_bio
mars: add new module xio_sio
mars: add new module xio_client
mars: add new module xio_if
mars: add new module xio_copy
mars: add new module xio_trans_logger
mars: add new module xio_server
mars: add new module light_strategy
mars: add new module light_net
mars: add new module light_server_strategy
mars: add new module mars_proc
mars: add new module mars_light
mars: add new module Makefile
mars: add new module Kconfig
mars: activate build

 drivers/staging/Kconfig                            |    2 +
 drivers/staging/Makefile                           |    1 +
 drivers/staging/mars/Kconfig                       |  266 +
 drivers/staging/mars/Makefile                      |   61 +
 drivers/staging/mars/brick.c                       |  728 +++
 drivers/staging/mars/brick_mem.c                   | 1081 ++++
 drivers/staging/mars/brick_say.c                   |  916 +++
 drivers/staging/mars/lamport.c                     |   61 +
 drivers/staging/mars/lib/lib_limiter.c             |  129 +
 drivers/staging/mars/lib/lib_rank.c                |   87 +
 drivers/staging/mars/lib/lib_timing.c              |   71 +
 drivers/staging/mars/mars_light/light_net.c        |  109 +
 .../mars/mars_light/light_server_strategy.c        |  403 ++
 drivers/staging/mars/mars_light/light_strategy.c   | 2132 +++++++
 drivers/staging/mars/mars_light/mars_light.c       | 5880 ++++++++++++++++++++
 drivers/staging/mars/mars_light/mars_proc.c        |  369 ++
 drivers/staging/mars/xio_bricks/lib_log.c          |  505 ++
 drivers/staging/mars/xio_bricks/lib_mapfree.c      |  380 ++
 drivers/staging/mars/xio_bricks/xio.c              |  161 +
 drivers/staging/mars/xio_bricks/xio_bio.c          |  845 +++
 drivers/staging/mars/xio_bricks/xio_client.c       | 1055 ++++
 drivers/staging/mars/xio_bricks/xio_copy.c         | 1005 ++++
 drivers/staging/mars/xio_bricks/xio_if.c           |  961 ++++
 drivers/staging/mars/xio_bricks/xio_net.c          | 1830 ++++++
 drivers/staging/mars/xio_bricks/xio_server.c       |  486 ++
 drivers/staging/mars/xio_bricks/xio_sio.c          |  571 ++
 drivers/staging/mars/xio_bricks/xio_trans_logger.c | 3309 +++++++++++
 include/linux/brick/brick.h                        |  642 +++
 include/linux/brick/brick_checking.h               |  104 +
 include/linux/brick/brick_mem.h                    |  218 +
 include/linux/brick/brick_say.h                    |   96 +
 include/linux/brick/lamport.h                      |   26 +
 include/linux/brick/lib_limiter.h                  |   49 +
 include/linux/brick/lib_pairing_heap.h             |  110 +
 include/linux/brick/lib_queue.h                    |  166 +
 include/linux/brick/lib_rank.h                     |  135 +
 include/linux/brick/lib_timing.h                   |  181 +
 include/linux/brick/meta.h                         |  106 +
 include/linux/brick/vfs_compat.h                   |   45 +
 include/linux/mars_light/light_strategy.h          |  236 +
 include/linux/mars_light/mars_proc.h               |   34 +
 include/linux/xio/lib_log.h                        |  329 ++
 include/linux/xio/lib_mapfree.h                    |   84 +
 include/linux/xio/xio.h                            |  313 ++
 include/linux/xio/xio_bio.h                        |   85 +
 include/linux/xio/xio_client.h                     |  105 +
 include/linux/xio/xio_copy.h                       |  115 +
 include/linux/xio/xio_if.h                         |  108 +
 include/linux/xio/xio_net.h                        |  171 +
 include/linux/xio/xio_server.h                     |   91 +
 include/linux/xio/xio_sio.h                        |   68 +
 include/linux/xio/xio_trans_logger.h               |  263 +
 52 files changed, 27284 insertions(+)
 create mode 100644 drivers/staging/mars/Kconfig
 create mode 100644 drivers/staging/mars/Makefile
 create mode 100644 drivers/staging/mars/brick.c
 create mode 100644 drivers/staging/mars/brick_mem.c
 create mode 100644 drivers/staging/mars/brick_say.c
 create mode 100644 drivers/staging/mars/lamport.c
 create mode 100644 drivers/staging/mars/lib/lib_limiter.c
 create mode 100644 drivers/staging/mars/lib/lib_rank.c
 create mode 100644 drivers/staging/mars/lib/lib_timing.c
 create mode 100644 drivers/staging/mars/mars_light/light_net.c
 create mode 100644 drivers/staging/mars/mars_light/light_server_strategy.c
 create mode 100644 drivers/staging/mars/mars_light/light_strategy.c
 create mode 100644 drivers/staging/mars/mars_light/mars_light.c
 create mode 100644 drivers/staging/mars/mars_light/mars_proc.c
 create mode 100644 drivers/staging/mars/xio_bricks/lib_log.c
 create mode 100644 drivers/staging/mars/xio_bricks/lib_mapfree.c
 create mode 100644 drivers/staging/mars/xio_bricks/xio.c
 create mode 100644 drivers/staging/mars/xio_bricks/xio_bio.c
 create mode 100644 drivers/staging/mars/xio_bricks/xio_client.c
 create mode 100644 drivers/staging/mars/xio_bricks/xio_copy.c
 create mode 100644 drivers/staging/mars/xio_bricks/xio_if.c
 create mode 100644 drivers/staging/mars/xio_bricks/xio_net.c
 create mode 100644 drivers/staging/mars/xio_bricks/xio_server.c
 create mode 100644 drivers/staging/mars/xio_bricks/xio_sio.c
 create mode 100644 drivers/staging/mars/xio_bricks/xio_trans_logger.c
 create mode 100644 include/linux/brick/brick.h
 create mode 100644 include/linux/brick/brick_checking.h
 create mode 100644 include/linux/brick/brick_mem.h
 create mode 100644 include/linux/brick/brick_say.h
 create mode 100644 include/linux/brick/lamport.h
 create mode 100644 include/linux/brick/lib_limiter.h
 create mode 100644 include/linux/brick/lib_pairing_heap.h
 create mode 100644 include/linux/brick/lib_queue.h
 create mode 100644 include/linux/brick/lib_rank.h
 create mode 100644 include/linux/brick/lib_timing.h
 create mode 100644 include/linux/brick/meta.h
 create mode 100644 include/linux/brick/vfs_compat.h
 create mode 100644 include/linux/mars_light/light_strategy.h
 create mode 100644 include/linux/mars_light/mars_proc.h
 create mode 100644 include/linux/xio/lib_log.h
 create mode 100644 include/linux/xio/lib_mapfree.h
 create mode 100644 include/linux/xio/xio.h
 create mode 100644 include/linux/xio/xio_bio.h
 create mode 100644 include/linux/xio/xio_client.h
 create mode 100644 include/linux/xio/xio_copy.h
 create mode 100644 include/linux/xio/xio_if.h
 create mode 100644 include/linux/xio/xio_net.h
 create mode 100644 include/linux/xio/xio_server.h
 create mode 100644 include/linux/xio/xio_sio.h
 create mode 100644 include/linux/xio/xio_trans_logger.h
Signature · 1 line
2.6.4
#1 lore raw

RFC 1/31 mars: add new module lamport

---
 drivers/staging/mars/lamport.c | 61 ++++++++++++++++++++++++++++++++++++++++++
 include/linux/brick/lamport.h  | 26 ++++++++++++++++++
 2 files changed, 87 insertions(+)
 create mode 100644 drivers/staging/mars/lamport.c
 create mode 100644 include/linux/brick/lamport.h
drivers/staging/mars/lamport.c new file +61 -0
new file mode 100644
index 0000000..373093f
@@ -0,0 +1,61 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/semaphore.h>
+
+#include <linux/brick/lamport.h>
+
+/*  TODO: replace with spinlock if possible (first check) */
+struct semaphore lamport_sem = __SEMAPHORE_INITIALIZER(lamport_sem, 1);
+struct timespec lamport_now = {};
+
+void get_lamport(struct timespec *now)
+{
+	int diff;
+
+	down(&lamport_sem);
+
+	*now = CURRENT_TIME;
+	diff = timespec_compare(now, &lamport_now);
+	if (diff >= 0) {
+		timespec_add_ns(now, 1);
+		memcpy(&lamport_now, now, sizeof(lamport_now));
+		timespec_add_ns(&lamport_now, 1);
+	} else {
+		timespec_add_ns(&lamport_now, 1);
+		memcpy(now, &lamport_now, sizeof(*now));
+	}
+
+	up(&lamport_sem);
+}
+
+void set_lamport(struct timespec *old)
+{
+	int diff;
+
+	down(&lamport_sem);
+
+	diff = timespec_compare(old, &lamport_now);
+	if (diff >= 0) {
+		memcpy(&lamport_now, old, sizeof(lamport_now));
+		timespec_add_ns(&lamport_now, 1);
+	}
+
+	up(&lamport_sem);
+}
include/linux/brick/lamport.h new file +26 -0
new file mode 100644
index 0000000..9aac0ce
@@ -0,0 +1,26 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef LAMPORT_H
+#define LAMPORT_H
+
+#include <linux/time.h>
+
+extern void get_lamport(struct timespec *now);
+extern void set_lamport(struct timespec *old);
+
+#endif
Signature · 1 line
2.6.4
#2 lore raw

RFC 2/31 mars: add new module brick_say

In reply to Thomas Schoebel-Theuer
---
 drivers/staging/mars/brick_say.c | 916 +++++++++++++++++++++++++++++++++++++++
 include/linux/brick/brick_say.h  |  96 ++++
 2 files changed, 1012 insertions(+)
 create mode 100644 drivers/staging/mars/brick_say.c
 create mode 100644 include/linux/brick/brick_say.h
drivers/staging/mars/brick_say.c new file +916 -0
new file mode 100644
index 0000000..7a51273
@@ -0,0 +1,916 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/string.h>
+
+#include <linux/brick/brick_say.h>
+#include <linux/brick/lamport.h>
+
+/*******************************************************************/
+
+/*  messaging */
+
+#include <linux/fs.h>
+#include <linux/blkdev.h>
+#include <linux/file.h>
+#include <linux/sched.h>
+#include <linux/preempt.h>
+#include <linux/hardirq.h>
+#include <linux/smp.h>
+#include <linux/slab.h>
+#include <linux/kthread.h>
+#include <linux/syscalls.h>
+
+#include <linux/uaccess.h>
+
+#include <linux/brick/vfs_compat.h>
+
+#ifndef GFP_BRICK
+#define GFP_BRICK			GFP_NOIO
+#endif
+
+#define SAY_ORDER			0
+#define SAY_BUFMAX			(PAGE_SIZE << SAY_ORDER)
+#define SAY_BUF_LIMIT			(SAY_BUFMAX - 1500)
+#define MAX_FILELEN			16
+#define MAX_IDS				1000
+
+const char *say_class[MAX_SAY_CLASS] = {
+	[SAY_DEBUG] = "debug",
+	[SAY_INFO] = "info",
+	[SAY_WARN] = "warn",
+	[SAY_ERROR] = "error",
+	[SAY_FATAL] = "fatal",
+	[SAY_TOTAL] = "total",
+};
+
+int brick_say_logging = 1;
+
+module_param_named(say_logging, brick_say_logging, int, 0);
+int brick_say_debug;
+
+module_param_named(say_debug, brick_say_debug, int, 0);
+
+int brick_say_syslog_min = 1;
+int brick_say_syslog_max = -1;
+int brick_say_syslog_flood_class = 3;
+int brick_say_syslog_flood_limit = 20;
+int brick_say_syslog_flood_recovery = 300;
+
+int delay_say_on_overflow =
+#ifdef CONFIG_MARS_DEBUG
+	1;
+#else
+	0;
+#endif
+
+static atomic_t say_alloc_channels = ATOMIC_INIT(0);
+static atomic_t say_alloc_names = ATOMIC_INIT(0);
+static atomic_t say_alloc_pages = ATOMIC_INIT(0);
+
+static unsigned long flood_start_jiffies;
+static int flood_count;
+
+struct say_channel {
+	char *ch_name;
+	struct say_channel *ch_next;
+	spinlock_t ch_lock[MAX_SAY_CLASS];
+	char *ch_buf[MAX_SAY_CLASS][2];
+
+	short ch_index[MAX_SAY_CLASS];
+	struct file *ch_filp[MAX_SAY_CLASS][2];
+	int ch_overflow[MAX_SAY_CLASS];
+	bool ch_written[MAX_SAY_CLASS];
+	bool ch_rollover;
+	bool ch_must_exist;
+	bool ch_is_dir;
+	bool ch_delete;
+	int ch_status_written;
+	int ch_id_max;
+	void *ch_ids[MAX_IDS];
+
+	wait_queue_head_t ch_progress;
+};
+
+struct say_channel *default_channel;
+
+static struct say_channel *channel_list;
+
+static rwlock_t say_lock = __RW_LOCK_UNLOCKED(say_lock);
+
+static struct task_struct *say_thread;
+
+static DECLARE_WAIT_QUEUE_HEAD(say_event);
+
+bool say_dirty;
+
+#define use_atomic()							\
+	((preempt_count() & (PREEMPT_MASK | SOFTIRQ_MASK | HARDIRQ_MASK | NMI_MASK)) != 0 || irqs_disabled())
+
+static
+void wait_channel(struct say_channel *ch, int class)
+{
+	if (delay_say_on_overflow && ch->ch_index[class] > SAY_BUF_LIMIT) {
+		if (!use_atomic()) {
+			say_dirty = true;
+			wake_up_interruptible(&say_event);
+			wait_event_interruptible_timeout(ch->ch_progress,
+				ch->ch_index[class] < SAY_BUF_LIMIT,
+				HZ / 10);
+		}
+	}
+}
+
+static
+struct say_channel *find_channel(const void *id)
+{
+	struct say_channel *res = default_channel;
+	struct say_channel *ch;
+
+	read_lock(&say_lock);
+	for (ch = channel_list; ch; ch = ch->ch_next) {
+		int i;
+
+		for (i = 0; i < ch->ch_id_max; i++) {
+			if (ch->ch_ids[i] == id) {
+				res = ch;
+				goto found;
+			}
+		}
+	}
+found:
+	read_unlock(&say_lock);
+	return res;
+}
+
+static
+void _remove_binding(struct task_struct *whom)
+{
+	struct say_channel *ch;
+	int i;
+
+	for (ch = channel_list; ch; ch = ch->ch_next) {
+		for (i = 0; i < ch->ch_id_max; i++) {
+			if (ch->ch_ids[i] == whom)
+				ch->ch_ids[i] = NULL;
+		}
+	}
+}
+
+void bind_to_channel(struct say_channel *ch, struct task_struct *whom)
+{
+	int i;
+
+	write_lock(&say_lock);
+	_remove_binding(whom);
+	for (i = 0; i < ch->ch_id_max; i++) {
+		if (!ch->ch_ids[i]) {
+			ch->ch_ids[i] = whom;
+			goto done;
+		}
+	}
+	if (likely(ch->ch_id_max < MAX_IDS - 1))
+		ch->ch_ids[ch->ch_id_max++] = whom;
+	else
+		goto err;
+done:
+	write_unlock(&say_lock);
+	goto out_return;
+err:
+	write_unlock(&say_lock);
+
+	say_to(default_channel, SAY_ERROR, "ID overflow for thread '%s'\n", whom->comm);
+out_return:;
+}
+
+struct say_channel *get_binding(struct task_struct *whom)
+{
+	struct say_channel *ch;
+	int i;
+
+	read_lock(&say_lock);
+	for (ch = channel_list; ch; ch = ch->ch_next) {
+		for (i = 0; i < ch->ch_id_max; i++) {
+			if (ch->ch_ids[i] == whom)
+				goto found;
+		}
+	}
+	ch = NULL;
+found:
+	read_unlock(&say_lock);
+	return ch;
+}
+
+void remove_binding_from(struct say_channel *ch, struct task_struct *whom)
+{
+	bool found = false;
+	int i;
+
+	write_lock(&say_lock);
+	for (i = 0; i < ch->ch_id_max; i++) {
+		if (ch->ch_ids[i] == whom) {
+			ch->ch_ids[i] = NULL;
+			found = true;
+			break;
+		}
+	}
+	if (!found)
+		_remove_binding(whom);
+	write_unlock(&say_lock);
+}
+
+void remove_binding(struct task_struct *whom)
+{
+	write_lock(&say_lock);
+	_remove_binding(whom);
+	write_unlock(&say_lock);
+}
+
+void rollover_channel(struct say_channel *ch)
+{
+	if (!ch)
+		ch = find_channel(current);
+	if (likely(ch))
+		ch->ch_rollover = true;
+}
+
+void rollover_all(void)
+{
+	struct say_channel *ch;
+
+	read_lock(&say_lock);
+	for (ch = channel_list; ch; ch = ch->ch_next)
+		ch->ch_rollover = true;
+	read_unlock(&say_lock);
+}
+
+void del_channel(struct say_channel *ch)
+{
+	if (unlikely(!ch))
+		goto out_return;
+	if (unlikely(ch == default_channel)) {
+		say_to(default_channel, SAY_ERROR, "thread '%s' tried to delete the default channel\n", current->comm);
+		goto out_return;
+	}
+
+	ch->ch_delete = true;
+out_return:;
+}
+
+static
+void _del_channel(struct say_channel *ch)
+{
+	struct say_channel *tmp;
+	struct say_channel **_tmp;
+	int i, j;
+
+	if (!ch)
+		goto out_return;
+	write_lock(&say_lock);
+	for (_tmp = &channel_list; (tmp = *_tmp) != NULL; _tmp = &tmp->ch_next) {
+		if (tmp == ch) {
+			*_tmp = tmp->ch_next;
+			break;
+		}
+	}
+	write_unlock(&say_lock);
+
+	for (i = 0; i < MAX_SAY_CLASS; i++) {
+		for (j = 0; j < 2; j++) {
+			if (ch->ch_filp[i][j]) {
+				filp_close(ch->ch_filp[i][j], NULL);
+				ch->ch_filp[i][j] = NULL;
+			}
+		}
+		for (j = 0; j < 2; j++) {
+			char *buf = ch->ch_buf[i][j];
+
+			if (buf) {
+				__free_pages(virt_to_page((unsigned long)buf), SAY_ORDER);
+				atomic_dec(&say_alloc_pages);
+			}
+		}
+	}
+	if (ch->ch_name) {
+		atomic_dec(&say_alloc_names);
+		kfree(ch->ch_name);
+	}
+	kfree(ch);
+	atomic_dec(&say_alloc_channels);
+out_return:;
+}
+
+static
+struct say_channel *_make_channel(const char *name, bool must_exist)
+{
+	struct say_channel *res = NULL;
+	struct kstat kstat = {};
+	int i, j;
+	unsigned long mode = use_atomic() ? GFP_ATOMIC : GFP_BRICK;
+
+	mm_segment_t oldfs;
+	bool is_dir = false;
+	int status;
+
+	oldfs = get_fs();
+	set_fs(get_ds());
+	status = vfs_stat((char *)name, &kstat);
+	set_fs(oldfs);
+
+	if (unlikely(status < 0)) {
+		if (must_exist) {
+			say(SAY_ERROR, "cannot create channel '%s', status = %d\n", name, status);
+			goto done;
+		}
+	} else {
+		is_dir = S_ISDIR(kstat.mode);
+	}
+
+restart:
+	res = kzalloc(sizeof(struct say_channel), mode);
+	if (unlikely(!res)) {
+		schedule();
+		goto restart;
+	}
+	atomic_inc(&say_alloc_channels);
+	res->ch_must_exist = must_exist;
+	res->ch_is_dir = is_dir;
+	init_waitqueue_head(&res->ch_progress);
+restart2:
+	res->ch_name = kstrdup(name, mode);
+	if (unlikely(!res->ch_name)) {
+		schedule();
+		goto restart2;
+	}
+	atomic_inc(&say_alloc_names);
+	for (i = 0; i < MAX_SAY_CLASS; i++) {
+		spin_lock_init(&res->ch_lock[i]);
+		for (j = 0; j < 2; j++) {
+			char *buf;
+
+restart3:
+			buf = (void *)__get_free_pages(mode, SAY_ORDER);
+			if (unlikely(!buf)) {
+				schedule();
+				goto restart3;
+			}
+			atomic_inc(&say_alloc_pages);
+			res->ch_buf[i][j] = buf;
+		}
+	}
+done:
+	return res;
+}
+
+struct say_channel *make_channel(const char *name, bool must_exist)
+{
+	struct say_channel *res = NULL;
+	struct say_channel *ch;
+
+	read_lock(&say_lock);
+	for (ch = channel_list; ch; ch = ch->ch_next) {
+		if (!strcmp(ch->ch_name, name)) {
+			res = ch;
+			break;
+		}
+	}
+	read_unlock(&say_lock);
+
+	if (unlikely(!res)) {
+		res = _make_channel(name, must_exist);
+		if (unlikely(!res))
+			goto done;
+
+		write_lock(&say_lock);
+
+		for (ch = channel_list; ch; ch = ch->ch_next) {
+			if (ch != res && unlikely(!strcmp(ch->ch_name, name))) {
+				_del_channel(res);
+				res = ch;
+				goto race_found;
+			}
+		}
+
+		res->ch_next = channel_list;
+		channel_list = res;
+
+race_found:
+		write_unlock(&say_lock);
+	}
+
+done:
+	return res;
+}
+
+/*  tell gcc to check for varargs errors */
+static
+void _say(struct say_channel *ch, int class, va_list args, bool use_args, const char *fmt, ...)  __printf(5, 6);
+
+static
+void _say(struct say_channel *ch, int class, va_list args, bool use_args, const char *fmt, ...)
+{
+	char *start;
+	int offset;
+	int rest;
+	int written;
+
+	if (unlikely(!ch))
+		goto out_return;
+	if (unlikely(ch->ch_delete && ch != default_channel)) {
+		say_to(default_channel, SAY_ERROR, "thread '%s' tried to write on deleted channel\n", current->comm);
+		goto out_return;
+	}
+
+	offset = ch->ch_index[class];
+	start = ch->ch_buf[class][0] + offset;
+	rest = SAY_BUFMAX - 1 - offset;
+	if (unlikely(rest <= 0)) {
+		ch->ch_overflow[class]++;
+		goto out_return;
+	}
+
+	if (use_args) {
+		va_list args2;
+
+		va_start(args2, fmt);
+		written = vscnprintf(start, rest, fmt, args2);
+		va_end(args2);
+	} else {
+		written = vscnprintf(start, rest, fmt, args);
+	}
+
+	if (likely(rest > written)) {
+		start[written] = '\0';
+		ch->ch_index[class] += written;
+		say_dirty = true;
+	} else {
+		/*  indicate overflow */
+		start[0] = '\0';
+		ch->ch_overflow[class]++;
+	}
+out_return:;
+}
+
+void say_to(struct say_channel *ch, int class, const char *fmt, ...)
+{
+	va_list args;
+	unsigned long flags;
+
+	if (!class && !brick_say_debug)
+		goto out_return;
+	if (!ch)
+		ch = find_channel(current);
+
+	if (likely(ch)) {
+		if (!ch->ch_is_dir)
+			class = SAY_TOTAL;
+		if (likely(class >= 0 && class < MAX_SAY_CLASS)) {
+			wait_channel(ch, class);
+			spin_lock_irqsave(&ch->ch_lock[class], flags);
+
+			va_start(args, fmt);
+			_say(ch, class, args, false, fmt);
+			va_end(args);
+
+			spin_unlock_irqrestore(&ch->ch_lock[class], flags);
+		}
+	}
+
+	ch = default_channel;
+	if (likely(ch)) {
+		class = SAY_TOTAL;
+		wait_channel(ch, class);
+		spin_lock_irqsave(&ch->ch_lock[class], flags);
+
+		va_start(args, fmt);
+		_say(ch, class, args, false, fmt);
+		va_end(args);
+
+		spin_unlock_irqrestore(&ch->ch_lock[class], flags);
+
+		wake_up_interruptible(&say_event);
+	}
+out_return:;
+}
+
+void brick_say_to(struct say_channel *ch,
+	int class,
+	bool dump,
+	const char *prefix,
+	const char *file,
+	int line,
+	const char *func,
+	const char *fmt,
+	...)
+{
+	const char *channel_name = "-";
+	struct timespec s_now;
+	struct timespec l_now;
+	int filelen;
+	int orig_class;
+	va_list args;
+	unsigned long flags;
+
+	if (!class && !brick_say_debug)
+		goto out_return;
+	s_now = CURRENT_TIME;
+	get_lamport(&l_now);
+
+	if (!ch)
+		ch = find_channel(current);
+
+	orig_class = class;
+
+	/*  limit the filename */
+	filelen = strlen(file);
+	if (filelen > MAX_FILELEN)
+		file += filelen - MAX_FILELEN;
+
+	if (likely(ch)) {
+		channel_name = ch->ch_name;
+		if (!ch->ch_is_dir)
+			class = SAY_TOTAL;
+		if (likely(class >= 0 && class < MAX_SAY_CLASS)) {
+			wait_channel(ch, class);
+			spin_lock_irqsave(&ch->ch_lock[class], flags);
+
+			_say(ch, class, NULL, true,
+			     "%ld.%09ld %ld.%09ld %s %s[%d] %s:%d %s(): ",
+			     s_now.tv_sec, s_now.tv_nsec,
+			     l_now.tv_sec, l_now.tv_nsec,
+			     prefix,
+			     current->comm, (int)smp_processor_id(),
+			     file, line,
+			     func);
+
+			va_start(args, fmt);
+			_say(ch, class, args, false, fmt);
+			va_end(args);
+
+			spin_unlock_irqrestore(&ch->ch_lock[class], flags);
+		}
+	}
+
+	ch = default_channel;
+	if (likely(ch)) {
+		wait_channel(ch, SAY_TOTAL);
+		spin_lock_irqsave(&ch->ch_lock[SAY_TOTAL], flags);
+
+		_say(ch, SAY_TOTAL, NULL, true,
+		     "%ld.%09ld %ld.%09ld %s_%-5s %s %s[%d] %s:%d %s(): ",
+		     s_now.tv_sec, s_now.tv_nsec,
+		     l_now.tv_sec, l_now.tv_nsec,
+		     prefix, say_class[orig_class],
+		     channel_name,
+		     current->comm, (int)smp_processor_id(),
+		     file, line,
+		     func);
+
+		va_start(args, fmt);
+		_say(ch, SAY_TOTAL, args, false, fmt);
+		va_end(args);
+
+		spin_unlock_irqrestore(&ch->ch_lock[SAY_TOTAL], flags);
+
+	}
+#ifdef CONFIG_MARS_DEBUG
+	if (dump)
+		brick_dump_stack();
+#endif
+	wake_up_interruptible(&say_event);
+out_return:;
+}
+
+static
+void try_open_file(struct file **file, char *filename, bool creat)
+{
+	struct address_space *mapping;
+	int flags = O_APPEND | O_WRONLY | O_LARGEFILE;
+	int prot = 0600;
+
+	if (creat)
+		flags |= O_CREAT;
+
+	*file = filp_open(filename, flags, prot);
+	if (unlikely(IS_ERR(*file))) {
+		*file = NULL;
+		goto out_return;
+	}
+	mapping = (*file)->f_mapping;
+	if (likely(mapping))
+		mapping_set_gfp_mask(mapping, mapping_gfp_mask(mapping) & ~(__GFP_IO | __GFP_FS));
+out_return:;
+}
+
+static
+void out_to_file(struct file *file, char *buf, int len)
+{
+	loff_t log_pos = 0;
+
+	mm_segment_t oldfs;
+
+	if (file) {
+		oldfs = get_fs();
+		set_fs(get_ds());
+		(void)vfs_write(file, buf, len, &log_pos);
+		set_fs(oldfs);
+	}
+}
+
+static inline
+void reset_flood(void)
+{
+	if (flood_start_jiffies &&
+	    time_is_before_jiffies(flood_start_jiffies + brick_say_syslog_flood_recovery * HZ)) {
+		flood_start_jiffies = 0;
+		flood_count = 0;
+	}
+}
+
+static
+void printk_with_class(int class, char *buf)
+{
+	switch (class) {
+	case SAY_INFO:
+		printk(KERN_INFO "%s", buf);
+		break;
+	case SAY_WARN:
+		printk(KERN_WARNING "%s", buf);
+		break;
+	case SAY_ERROR:
+	case SAY_FATAL:
+		printk(KERN_ERR "%s", buf);
+		break;
+	default:
+		printk(KERN_DEBUG "%s", buf);
+	}
+}
+
+static
+void out_to_syslog(int class, char *buf, int len)
+{
+	reset_flood();
+	if (class >= brick_say_syslog_min && class <= brick_say_syslog_max) {
+		buf[len] = '\0';
+		printk_with_class(class, buf);
+	} else if (class >= brick_say_syslog_flood_class && brick_say_syslog_flood_class >= 0 && class != SAY_TOTAL) {
+		flood_start_jiffies = jiffies;
+		if (++flood_count <= brick_say_syslog_flood_limit) {
+			buf[len] = '\0';
+			printk_with_class(class, buf);
+		}
+	}
+}
+
+static inline
+char *_make_filename(struct say_channel *ch, int class, int transact, int add_tmp)
+{
+	char *filename;
+
+restart:
+	filename = kmalloc(1024, GFP_KERNEL);
+	if (unlikely(!filename)) {
+		schedule();
+		goto restart;
+	}
+	atomic_inc(&say_alloc_names);
+	if (ch->ch_is_dir) {
+		snprintf(filename,
+			1023,
+			"%s/%d.%s.%s%s",
+			ch->ch_name,
+			class,
+			say_class[class],
+			transact ? "status" : "log",
+			add_tmp ? ".tmp" : "");
+	} else {
+		snprintf(filename, 1023, "%s.%s%s", ch->ch_name, transact ? "status" : "log", add_tmp ? ".tmp" : "");
+	}
+	return filename;
+}
+
+static
+void _rollover_channel(struct say_channel *ch)
+{
+	int start = 0;
+	int class;
+
+	ch->ch_rollover = false;
+	ch->ch_status_written = 0;
+
+	if (!ch->ch_is_dir)
+		start = SAY_TOTAL;
+
+	for (class = start; class < MAX_SAY_CLASS; class++) {
+		char *old = _make_filename(ch, class, 1, 1);
+		char *new = _make_filename(ch, class, 1, 0);
+
+		if (likely(old && new)) {
+			int i;
+
+			mm_segment_t oldfs;
+
+			for (i = 0; i < 2; i++) {
+				if (ch->ch_filp[class][i]) {
+					filp_close(ch->ch_filp[class][i], NULL);
+					ch->ch_filp[class][i] = NULL;
+				}
+			}
+
+			oldfs = get_fs();
+			set_fs(get_ds());
+#ifdef __USE_COMPAT
+			_compat_rename(old, new);
+#else
+			sys_rename(old, new);
+#endif
+			set_fs(oldfs);
+		}
+
+		if (likely(old)) {
+			kfree(old);
+			atomic_dec(&say_alloc_names);
+		}
+		if (likely(new)) {
+			kfree(new);
+			atomic_dec(&say_alloc_names);
+		}
+	}
+}
+
+static
+void treat_channel(struct say_channel *ch, int class)
+{
+	int len;
+	int overflow;
+	int transact;
+	int start;
+	char *buf;
+	char *tmp;
+	unsigned long flags;
+
+	spin_lock_irqsave(&ch->ch_lock[class], flags);
+
+	buf = ch->ch_buf[class][0];
+	tmp = ch->ch_buf[class][1];
+	ch->ch_buf[class][1] = buf;
+	ch->ch_buf[class][0] = tmp;
+	len = ch->ch_index[class];
+	ch->ch_index[class] = 0;
+	overflow = ch->ch_overflow[class];
+	ch->ch_overflow[class] = 0;
+
+	spin_unlock_irqrestore(&ch->ch_lock[class], flags);
+
+	wake_up_interruptible(&ch->ch_progress);
+
+	ch->ch_status_written += len;
+	out_to_syslog(class, buf, len);
+	start = 0;
+	if (!brick_say_logging)
+		start++;
+	for (transact = start; transact < 2; transact++) {
+		if (unlikely(!ch->ch_filp[class][transact])) {
+			char *filename = _make_filename(ch, class, transact, transact);
+
+			if (likely(filename)) {
+				try_open_file(&ch->ch_filp[class][transact], filename, transact);
+				kfree(filename);
+				atomic_dec(&say_alloc_names);
+			}
+		}
+		out_to_file(ch->ch_filp[class][transact], buf, len);
+	}
+
+	if (unlikely(overflow > 0)) {
+		struct timespec s_now = CURRENT_TIME;
+		struct timespec l_now;
+
+		get_lamport(&l_now);
+		len = scnprintf(buf,
+			       SAY_BUFMAX,
+			       "%ld.%09ld %ld.%09ld %s %d OVERFLOW %d times\n",
+			       s_now.tv_sec, s_now.tv_nsec,
+			       l_now.tv_sec, l_now.tv_nsec,
+			       ch->ch_name,
+			       class,
+			       overflow);
+		ch->ch_status_written += len;
+		out_to_syslog(class, buf, len);
+		for (transact = 0; transact < 2; transact++)
+			out_to_file(ch->ch_filp[class][transact], buf, len);
+	}
+}
+
+static
+int _say_thread(void *data)
+{
+	while (!kthread_should_stop()) {
+		struct say_channel *ch;
+		int i;
+
+		wait_event_interruptible_timeout(say_event, say_dirty, HZ);
+		say_dirty = false;
+
+restart_rollover:
+		read_lock(&say_lock);
+		for (ch = channel_list; ch; ch = ch->ch_next) {
+			if (ch->ch_rollover && ch->ch_status_written > 0) {
+				read_unlock(&say_lock);
+				_rollover_channel(ch);
+				goto restart_rollover;
+			}
+		}
+		read_unlock(&say_lock);
+
+restart:
+		read_lock(&say_lock);
+		for (ch = channel_list; ch; ch = ch->ch_next) {
+			int start = 0;
+
+			if (!ch->ch_is_dir)
+				start = SAY_TOTAL;
+			for (i = start; i < MAX_SAY_CLASS; i++) {
+				if (ch->ch_index[i] > 0) {
+					read_unlock(&say_lock);
+					treat_channel(ch, i);
+					goto restart;
+				}
+			}
+			if (ch->ch_delete) {
+				read_unlock(&say_lock);
+				_del_channel(ch);
+				goto restart;
+			}
+		}
+		read_unlock(&say_lock);
+	}
+
+	return 0;
+}
+
+void init_say(void)
+{
+	default_channel = make_channel(CONFIG_MARS_LOGDIR, true);
+	say_thread = kthread_create(_say_thread, NULL, "brick_say");
+	if (IS_ERR(say_thread)) {
+		say_thread = NULL;
+	} else {
+		get_task_struct(say_thread);
+		wake_up_process(say_thread);
+	}
+
+}
+
+void exit_say(void)
+{
+	int memleak_channels;
+	int memleak_names;
+	int memleak_pages;
+
+	if (say_thread) {
+		kthread_stop(say_thread);
+		put_task_struct(say_thread);
+		say_thread = NULL;
+	}
+
+	default_channel = NULL;
+	while (channel_list)
+		_del_channel(channel_list);
+
+	memleak_channels = atomic_read(&say_alloc_channels);
+	memleak_names = atomic_read(&say_alloc_names);
+	memleak_pages = atomic_read(&say_alloc_pages);
+	if (unlikely(memleak_channels || memleak_names || memleak_pages))
+		printk("MEMLEAK: channels=%d names=%d pages=%d\n", memleak_channels, memleak_names, memleak_pages);
+}
+
+#ifdef CONFIG_MARS_DEBUG
+
+static int dump_max = 5;
+
+void brick_dump_stack(void)
+{
+	if (dump_max > 0) {
+		dump_max--; /*	racy, but does no harm */
+		dump_stack();
+	}
+}
+
+#endif
include/linux/brick/brick_say.h new file +96 -0

96 lines not on this page · raw message

Signature · 1 line
2.6.4
#3 lore raw

RFC 3/31 mars: add new module brick_mem

In reply to Thomas Schoebel-Theuer
---
 drivers/staging/mars/brick_mem.c | 1081 ++++++++++++++++++++++++++++++++++++++
 include/linux/brick/brick_mem.h  |  218 ++++++++
 2 files changed, 1299 insertions(+)
 create mode 100644 drivers/staging/mars/brick_mem.c
 create mode 100644 include/linux/brick/brick_mem.h
drivers/staging/mars/brick_mem.c new file +1081 -0
new file mode 100644
index 0000000..03a3d28
@@ -0,0 +1,1081 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/mm.h>
+#include <linux/fs.h>
+#include <linux/slab.h>
+#include <linux/delay.h>
+
+#include <linux/atomic.h>
+
+#include <linux/brick/brick_mem.h>
+#include <linux/brick/brick_say.h>
+#include <linux/brick/lamport.h>
+
+#define USE_KERNEL_PAGES		/*  currently mandatory (vmalloc does not work) */
+
+#define MAGIC_BLOCK			((int)0x8B395D7B)
+#define MAGIC_BEND			((int)0x8B395D7C)
+#define MAGIC_MEM1			((int)0x8B395D7D)
+#define MAGIC_MEM2			((int)0x9B395D8D)
+#define MAGIC_MEND1			((int)0x8B395D7E)
+#define MAGIC_MEND2			((int)0x9B395D8E)
+#define MAGIC_STR			((int)0x8B395D7F)
+#define MAGIC_SEND			((int)0x9B395D8F)
+
+#define INT_ACCESS(ptr, offset) (*(int *)(((char *)(ptr)) + (offset)))
+
+#define _BRICK_FMT(_fmt, _class)					\
+	"%ld.%09ld %ld.%09ld MEM_%-5s %s[%d] %s:%d %s(): "		\
+		_fmt,							\
+		_s_now.tv_sec, _s_now.tv_nsec,				\
+		_l_now.tv_sec, _l_now.tv_nsec,				\
+		say_class[_class],					\
+		current->comm, (int)smp_processor_id(),			\
+		__BASE_FILE__,						\
+		__LINE__,						\
+		__func__
+
+#define _BRICK_MSG(_class, _dump, _fmt, _args...)			\
+	do {								\
+		struct timespec _s_now = CURRENT_TIME;			\
+		struct timespec _l_now;					\
+		get_lamport(&_l_now);					\
+		say(_class, _BRICK_FMT(_fmt, _class), ##_args);		\
+		if (_dump)						\
+			dump_stack();					\
+	} while (0)
+
+#define BRICK_ERR(_fmt, _args...) _BRICK_MSG(SAY_ERROR, true,  _fmt, ##_args)
+#define BRICK_WRN(_fmt, _args...) _BRICK_MSG(SAY_WARN,	false, _fmt, ##_args)
+#define BRICK_INF(_fmt, _args...) _BRICK_MSG(SAY_INFO,	false, _fmt, ##_args)
+
+/***********************************************************************/
+
+/*  limit handling */
+
+#include <linux/swap.h>
+
+long long brick_global_memavail;
+long long brick_global_memlimit;
+
+atomic64_t brick_global_block_used = ATOMIC64_INIT(0);
+
+void get_total_ram(void)
+{
+	struct sysinfo i = {};
+
+	si_meminfo(&i);
+	/* si_swapinfo(&i); */
+	brick_global_memavail = (long long)i.totalram * (PAGE_SIZE / 1024);
+	BRICK_INF("total RAM = %lld [KiB]\n", brick_global_memavail);
+}
+
+/***********************************************************************/
+
+/*  small memory allocation (use this only for len < PAGE_SIZE) */
+
+#ifdef BRICK_DEBUG_MEM
+static atomic_t phys_mem_alloc = ATOMIC_INIT(0);
+static atomic_t mem_redirect_alloc = ATOMIC_INIT(0);
+static atomic_t mem_count[BRICK_DEBUG_MEM];
+static atomic_t mem_free[BRICK_DEBUG_MEM];
+static int  mem_len[BRICK_DEBUG_MEM];
+
+#define PLUS_SIZE			(6 * sizeof(int))
+#else
+#define PLUS_SIZE			(2 * sizeof(int))
+#endif
+
+static inline
+void *__brick_mem_alloc(int len)
+{
+	void *res;
+
+	if (len >= PAGE_SIZE) {
+#ifdef BRICK_DEBUG_MEM
+		atomic_inc(&mem_redirect_alloc);
+#endif
+		res = _brick_block_alloc(0, len, 0);
+	} else {
+		for (;;) {
+			res = kmalloc(len, GFP_BRICK);
+			if (likely(res))
+				break;
+			msleep(1000);
+		}
+#ifdef BRICK_DEBUG_MEM
+		atomic_inc(&phys_mem_alloc);
+#endif
+	}
+	return res;
+}
+
+static inline
+void __brick_mem_free(void *data, int len)
+{
+	if (len >= PAGE_SIZE) {
+		_brick_block_free(data, len, 0);
+#ifdef BRICK_DEBUG_MEM
+		atomic_dec(&mem_redirect_alloc);
+#endif
+	} else {
+		kfree(data);
+#ifdef BRICK_DEBUG_MEM
+		atomic_dec(&phys_mem_alloc);
+#endif
+	}
+}
+
+void *_brick_mem_alloc(int len, int line)
+{
+	void *res;
+
+#ifdef CONFIG_MARS_DEBUG
+	might_sleep();
+#endif
+
+	res = __brick_mem_alloc(len + PLUS_SIZE);
+
+#ifdef BRICK_DEBUG_MEM
+	if (unlikely(line < 0))
+		line = 0;
+	else if (unlikely(line >= BRICK_DEBUG_MEM))
+		line = BRICK_DEBUG_MEM - 1;
+	INT_ACCESS(res, 0 * sizeof(int)) = MAGIC_MEM1;
+	INT_ACCESS(res, 1 * sizeof(int)) = len;
+	INT_ACCESS(res, 2 * sizeof(int)) = line;
+	INT_ACCESS(res, 3 * sizeof(int)) = MAGIC_MEM2;
+	res += 4 * sizeof(int);
+	INT_ACCESS(res, len + 0 * sizeof(int)) = MAGIC_MEND1;
+	INT_ACCESS(res, len + 1 * sizeof(int)) = MAGIC_MEND2;
+	atomic_inc(&mem_count[line]);
+	mem_len[line] = len;
+#else
+	INT_ACCESS(res, 0 * sizeof(int)) = len;
+	res += PLUS_SIZE;
+#endif
+	return res;
+}
+
+void _brick_mem_free(void *data, int cline)
+{
+#ifdef BRICK_DEBUG_MEM
+	void *test = data - 4 * sizeof(int);
+	int magic1 = INT_ACCESS(test, 0 * sizeof(int));
+	int len = INT_ACCESS(test, 1 * sizeof(int));
+	int line = INT_ACCESS(test, 2 * sizeof(int));
+	int magic2 = INT_ACCESS(test, 3 * sizeof(int));
+
+	if (unlikely(magic1 != MAGIC_MEM1)) {
+		BRICK_ERR("line %d memory corruption: magix1 %08x != %08x, len = %d\n",
+			cline,
+			magic1,
+			MAGIC_MEM1,
+			len);
+		goto _out_return;
+	}
+	if (unlikely(magic2 != MAGIC_MEM2)) {
+		BRICK_ERR("line %d memory corruption: magix2 %08x != %08x, len = %d\n",
+			cline,
+			magic2,
+			MAGIC_MEM2,
+			len);
+		goto _out_return;
+	}
+	if (unlikely(line < 0 || line >= BRICK_DEBUG_MEM)) {
+		BRICK_ERR("line %d memory corruption: alloc line = %d, len = %d\n", cline, line, len);
+		goto _out_return;
+	}
+	INT_ACCESS(test, 0) = 0xffffffff;
+	magic1 = INT_ACCESS(data, len + 0 * sizeof(int));
+	if (unlikely(magic1 != MAGIC_MEND1)) {
+		BRICK_ERR("line %d memory corruption: magix1 %08x != %08x, len = %d\n",
+			cline,
+			magic1,
+			MAGIC_MEND1,
+			len);
+		goto _out_return;
+	}
+	magic2 = INT_ACCESS(data, len + 1 * sizeof(int));
+	if (unlikely(magic2 != MAGIC_MEND2)) {
+		BRICK_ERR("line %d memory corruption: magix2 %08x != %08x, len = %d\n",
+			cline,
+			magic2,
+			MAGIC_MEND2,
+			len);
+		goto _out_return;
+	}
+	INT_ACCESS(data, len) = 0xffffffff;
+	atomic_dec(&mem_count[line]);
+	atomic_inc(&mem_free[line]);
+#else
+	void *test = data - PLUS_SIZE;
+	int len = INT_ACCESS(test, 0 * sizeof(int));
+
+#endif
+	data = test;
+	__brick_mem_free(data, len + PLUS_SIZE);
+#ifdef BRICK_DEBUG_MEM
+_out_return:;
+#endif
+}
+
+/***********************************************************************/
+
+/*  string memory allocation */
+
+#ifdef CONFIG_MARS_DEBUG_MEM_STRONG
+# define STRING_CANARY							\
+	"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" \
+	"yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy" \
+	"zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz" \
+	"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" \
+	"yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy" \
+	"zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz" \
+	"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" \
+	"yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy" \
+	"zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz" \
+	" FILE = "	__FILE__					\
+	" VERSION = "	__VERSION__					\
+	" xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx STRING_error xxx\n"
+# define STRING_PLUS (sizeof(int) * 3 + sizeof(STRING_CANARY))
+#elif defined(BRICK_DEBUG_MEM)
+# define STRING_PLUS (sizeof(int) * 4)
+#else
+# define STRING_PLUS 0
+#endif
+
+#ifdef BRICK_DEBUG_MEM
+static atomic_t phys_string_alloc = ATOMIC_INIT(0);
+static atomic_t string_count[BRICK_DEBUG_MEM];
+static atomic_t string_free[BRICK_DEBUG_MEM];
+
+#endif
+
+char *_brick_string_alloc(int len, int line)
+{
+	char *res;
+
+#ifdef CONFIG_MARS_DEBUG
+	might_sleep();
+	if (unlikely(len > PAGE_SIZE))
+		BRICK_WRN("line = %d string too long: len = %d\n", line, len);
+#endif
+	if (len <= 0)
+		len = BRICK_STRING_LEN;
+
+	for (;;) {
+		res = kzalloc(len + STRING_PLUS, GFP_BRICK);
+		if (likely(res))
+			break;
+		msleep(1000);
+	}
+
+#ifdef BRICK_DEBUG_MEM
+#ifdef CONFIG_MARS_DEBUG_MEM_STRONG
+	memset(res + 1, '?', len - 1);
+#endif
+	atomic_inc(&phys_string_alloc);
+	if (unlikely(line < 0))
+		line = 0;
+	else if (unlikely(line >= BRICK_DEBUG_MEM))
+		line = BRICK_DEBUG_MEM - 1;
+	INT_ACCESS(res, 0) = MAGIC_STR;
+	INT_ACCESS(res, sizeof(int)) = len;
+	INT_ACCESS(res, sizeof(int) * 2) = line;
+	res += sizeof(int) * 3;
+#ifdef CONFIG_MARS_DEBUG_MEM_STRONG
+	strcpy(res + len, STRING_CANARY);
+#else
+	INT_ACCESS(res, len) = MAGIC_SEND;
+#endif
+	atomic_inc(&string_count[line]);
+#endif
+	return res;
+}
+
+void _brick_string_free(const char *data, int cline)
+{
+#ifdef BRICK_DEBUG_MEM
+	int magic;
+	int len;
+	int line;
+	char *orig = (void *)data;
+
+	data -= sizeof(int) * 3;
+	magic = INT_ACCESS(data, 0);
+	if (unlikely(magic != MAGIC_STR)) {
+		BRICK_ERR("cline %d stringmem corruption: magix %08x != %08x\n", cline, magic, MAGIC_STR);
+		goto _out_return;
+	}
+	len = INT_ACCESS(data, sizeof(int));
+	line = INT_ACCESS(data, sizeof(int) * 2);
+	if (unlikely(len <= 0)) {
+		BRICK_ERR("cline %d stringmem corruption: line = %d len = %d\n", cline, line, len);
+		goto _out_return;
+	}
+	if (unlikely(len > PAGE_SIZE))
+		BRICK_ERR("cline %d string too long: line = %d len = %d string='%s'\n", cline, line, len, orig);
+	if (unlikely(line < 0 || line >= BRICK_DEBUG_MEM)) {
+		BRICK_ERR("cline %d stringmem corruption: line = %d (len = %d)\n", cline, line, len);
+		goto _out_return;
+	}
+#ifdef CONFIG_MARS_DEBUG_MEM_STRONG
+	if (unlikely(strcmp(orig + len, STRING_CANARY))) {
+		BRICK_ERR("cline %d stringmem corruption: bad canary '%s', line = %d len = %d\n",
+			  cline, STRING_CANARY, line, len);
+		goto _out_return;
+	}
+	orig[len]--;
+	memset(orig, '!', len);
+#else
+	magic = INT_ACCESS(orig, len);
+	if (unlikely(magic != MAGIC_SEND)) {
+		BRICK_ERR("cline %d stringmem corruption: end_magix %08x != %08x, line = %d len = %d\n",
+			  cline, magic, MAGIC_SEND, line, len);
+		goto _out_return;
+	}
+	INT_ACCESS(orig, len) = 0xffffffff;
+#endif
+	atomic_dec(&string_count[line]);
+	atomic_inc(&string_free[line]);
+	atomic_dec(&phys_string_alloc);
+#endif
+	kfree(data);
+#ifdef BRICK_DEBUG_MEM
+_out_return:;
+#endif
+}
+
+/***********************************************************************/
+
+/*  block memory allocation */
+
+static
+int len2order(int len)
+{
+	int order = 0;
+
+	if (unlikely(len <= 0)) {
+		BRICK_ERR("trying to use %d bytes\n", len);
+		return 0;
+	}
+
+	while ((PAGE_SIZE << order) < len)
+		order++;
+
+	if (unlikely(order > BRICK_MAX_ORDER)) {
+		BRICK_ERR("trying to use %d bytes (oder = %d, max = %d)\n", len, order, BRICK_MAX_ORDER);
+		return BRICK_MAX_ORDER;
+	}
+	return order;
+}
+
+#ifdef CONFIG_MARS_MEM_PREALLOC
+static atomic_t _alloc_count[BRICK_MAX_ORDER+1];
+int brick_mem_alloc_count[BRICK_MAX_ORDER+1] = {};
+int brick_mem_alloc_max[BRICK_MAX_ORDER+1] = {};
+int brick_mem_freelist_max[BRICK_MAX_ORDER+1] = {};
+
+#endif
+
+#ifdef BRICK_DEBUG_MEM
+static atomic_t phys_block_alloc = ATOMIC_INIT(0);
+
+/*  indexed by line */
+static atomic_t block_count[BRICK_DEBUG_MEM];
+static atomic_t block_free[BRICK_DEBUG_MEM];
+static int  block_len[BRICK_DEBUG_MEM];
+
+/*  indexed by order */
+static atomic_t op_count[BRICK_MAX_ORDER+1];
+static atomic_t raw_count[BRICK_MAX_ORDER+1];
+static int alloc_line[BRICK_MAX_ORDER+1];
+static int alloc_len[BRICK_MAX_ORDER+1];
+
+#endif
+
+#ifdef CONFIG_MARS_DEBUG_MEM_STRONG
+
+#define MAX_INFO_LISTS			1024
+
+#define INFO_LIST_HASH(addr) ((unsigned long)(addr) / (PAGE_SIZE * 2) % MAX_INFO_LISTS)
+
+struct mem_block_info {
+	struct list_head inf_head;
+	void *inf_data;
+	int inf_len;
+	int inf_line;
+	bool inf_used;
+};
+
+static struct list_head inf_anchor[MAX_INFO_LISTS];
+static rwlock_t inf_lock[MAX_INFO_LISTS];
+
+static
+void _new_block_info(void *data, int len, int cline)
+{
+	struct mem_block_info *inf;
+	int hash;
+
+	for (;;) {
+		inf = kmalloc(sizeof(struct mem_block_info), GFP_BRICK);
+		if (likely(inf))
+			break;
+		msleep(1000);
+	}
+	inf->inf_data = data;
+	inf->inf_len = len;
+	inf->inf_line = cline;
+	inf->inf_used = true;
+
+	hash = INFO_LIST_HASH(data);
+
+	write_lock(&inf_lock[hash]);
+	list_add(&inf->inf_head, &inf_anchor[hash]);
+	write_unlock(&inf_lock[hash]);
+}
+
+static
+struct mem_block_info *_find_block_info(void *data, bool remove)
+{
+	struct mem_block_info *res = NULL;
+	struct list_head *tmp;
+	int hash = INFO_LIST_HASH(data);
+
+	if (remove)
+		write_lock(&inf_lock[hash]);
+	else
+		read_lock(&inf_lock[hash]);
+	for (tmp = inf_anchor[hash].next; tmp != &inf_anchor[hash]; tmp = tmp->next) {
+		struct mem_block_info *inf = container_of(tmp, struct mem_block_info, inf_head);
+
+		if (inf->inf_data != data)
+			continue;
+		if (remove)
+			list_del_init(tmp);
+		res = inf;
+		break;
+	}
+	if (remove)
+		write_unlock(&inf_lock[hash]);
+	else
+		read_unlock(&inf_lock[hash]);
+	return res;
+}
+
+#endif /*  CONFIG_MARS_DEBUG_MEM_STRONG */
+
+static inline
+void *__brick_block_alloc(gfp_t gfp, int order, int cline)
+{
+	void *res;
+
+	for (;;) {
+#ifdef USE_KERNEL_PAGES
+		res = (void *)__get_free_pages(gfp, order);
+#else
+		res = __vmalloc(PAGE_SIZE << order, gfp, PAGE_KERNEL_IO);
+#endif
+		if (likely(res))
+			break;
+		msleep(1000);
+	}
+
+#ifdef CONFIG_MARS_DEBUG_MEM_STRONG
+	_new_block_info(res, PAGE_SIZE << order, cline);
+#endif
+#ifdef BRICK_DEBUG_MEM
+	atomic_inc(&phys_block_alloc);
+	atomic_inc(&raw_count[order]);
+#endif
+	atomic64_add((PAGE_SIZE/1024) << order, &brick_global_block_used);
+
+	return res;
+}
+
+static inline
+void __brick_block_free(void *data, int order, int cline)
+{
+#ifdef CONFIG_MARS_DEBUG_MEM_STRONG
+	struct mem_block_info *inf = _find_block_info(data, true);
+
+	if (likely(inf)) {
+		int inf_len = inf->inf_len;
+		int inf_line = inf->inf_line;
+
+		kfree(inf);
+		if (unlikely(inf_len != (PAGE_SIZE << order))) {
+			BRICK_ERR("line %d: address %p: bad freeing size %d (correct should be %d, previous line = %d)\n",
+				cline,
+				data,
+				(int)(PAGE_SIZE << order),
+				inf_len,
+				inf_line);
+			goto err;
+		}
+	} else {
+		BRICK_ERR("line %d: trying to free non-existent address %p (order = %d)\n", cline, data, order);
+		goto err;
+	}
+#endif
+#ifdef USE_KERNEL_PAGES
+	__free_pages(virt_to_page((unsigned long)data), order);
+#else
+	vfree(data);
+#endif
+#ifdef CONFIG_MARS_DEBUG_MEM_STRONG
+err:
+#endif
+#ifdef BRICK_DEBUG_MEM
+	atomic_dec(&phys_block_alloc);
+	atomic_dec(&raw_count[order]);
+#endif
+	atomic64_sub((PAGE_SIZE/1024) << order, &brick_global_block_used);
+}
+
+#ifdef CONFIG_MARS_MEM_PREALLOC
+int brick_allow_freelist = 1;
+
+int brick_pre_reserve[BRICK_MAX_ORDER+1] = {};
+
+/* Note: we have no separate lists per CPU.
+ * This should not hurt because the freelists are only used
+ * for higher-order pages which should be rather low-frequency.
+ */
+static spinlock_t freelist_lock[BRICK_MAX_ORDER+1];
+static void *brick_freelist[BRICK_MAX_ORDER+1];
+static atomic_t freelist_count[BRICK_MAX_ORDER+1];
+
+static
+void *_get_free(int order, int cline)
+{
+	void *data;
+	unsigned long flags;
+
+	spin_lock_irqsave(&freelist_lock[order], flags);
+	data = brick_freelist[order];
+	if (likely(data)) {
+		void *next = *(void **)data;
+
+#ifdef BRICK_DEBUG_MEM /*  check for corruptions */
+		long pattern = *(((long *)data)+1);
+		void *copy = *(((void **)data)+2);
+
+		if (unlikely(pattern != 0xf0f0f0f0f0f0f0f0 || next != copy)) { /*  found a corruption */
+			/*  prevent further trouble by leaving a memleak */
+			brick_freelist[order] = NULL;
+			spin_unlock_irqrestore(&freelist_lock[order], flags);
+			BRICK_ERR("line %d:freelist corruption at %p (pattern = %lx next %p != %p, murdered = %d), order = %d\n",
+				  cline, data, pattern, next, copy, atomic_read(&freelist_count[order]), order);
+			return NULL;
+		}
+#endif
+		brick_freelist[order] = next;
+		atomic_dec(&freelist_count[order]);
+	}
+	spin_unlock_irqrestore(&freelist_lock[order], flags);
+#ifdef CONFIG_MARS_DEBUG_MEM_STRONG
+	if (data) {
+		struct mem_block_info *inf = _find_block_info(data, false);
+
+		if (likely(inf)) {
+			if (unlikely(inf->inf_len != (PAGE_SIZE << order))) {
+				BRICK_ERR("line %d: address %p: bad freelist size %d (correct should be %d, previous line = %d)\n",
+					  cline, data, (int)(PAGE_SIZE << order), inf->inf_len, inf->inf_line);
+			}
+			inf->inf_line = cline;
+			inf->inf_used = true;
+		} else {
+			BRICK_ERR("line %d: freelist address %p is invalid (order = %d)\n", cline, data, order);
+		}
+	}
+#endif
+	return data;
+}
+
+static
+void _put_free(void *data, int order)
+{
+	void *next;
+	unsigned long flags;
+
+#ifdef BRICK_DEBUG_MEM /*  fill with pattern */
+	memset(data, 0xf0, PAGE_SIZE << order);
+#endif
+
+	spin_lock_irqsave(&freelist_lock[order], flags);
+	next = brick_freelist[order];
+	*(void **)data = next;
+#ifdef BRICK_DEBUG_MEM /*  insert redundant copy for checking */
+	*(((void **)data)+2) = next;
+#endif
+	brick_freelist[order] = data;
+	spin_unlock_irqrestore(&freelist_lock[order], flags);
+	atomic_inc(&freelist_count[order]);
+}
+
+static
+void _free_all(void)
+{
+	int order;
+
+	for (order = BRICK_MAX_ORDER; order >= 0; order--) {
+		for (;;) {
+			void *data = _get_free(order, __LINE__);
+
+			if (!data)
+				break;
+			__brick_block_free(data, order, __LINE__);
+		}
+	}
+}
+
+int brick_mem_reserve(void)
+{
+	int order;
+	int status = 0;
+
+	for (order = BRICK_MAX_ORDER; order >= 0; order--) {
+		int max = brick_pre_reserve[order];
+		int i;
+
+		brick_mem_freelist_max[order] += max;
+		BRICK_INF("preallocating %d at order %d (new maxlevel = %d)\n",
+			max,
+			order,
+			brick_mem_freelist_max[order]);
+
+		max = brick_mem_freelist_max[order] - atomic_read(&freelist_count[order]);
+		if (max >= 0) {
+			for (i = 0; i < max; i++) {
+				void *data = __brick_block_alloc(GFP_KERNEL, order, __LINE__);
+
+				if (likely(data))
+					_put_free(data, order);
+				else
+					status = -ENOMEM;
+			}
+		} else {
+			for (i = 0; i < -max; i++) {
+				void *data = _get_free(order, __LINE__);
+
+				if (likely(data))
+					__brick_block_free(data, order, __LINE__);
+			}
+		}
+	}
+	return status;
+}
+#else
+int brick_mem_reserve(struct mem_reservation *r)
+{
+	BRICK_INF("preallocation is not compiled in\n");
+	return 0;
+}
+#endif
+
+void *_brick_block_alloc(loff_t pos, int len, int line)
+{
+	void *data;
+	int count;
+
+#ifdef BRICK_DEBUG_MEM
+#ifdef BRICK_DEBUG_ORDER0
+	const int plus0 = PAGE_SIZE;
+
+#else
+	const int plus0 = 0;
+
+#endif
+	const int plus = len <= PAGE_SIZE ? plus0 : PAGE_SIZE * 2;
+
+#else
+	const int plus = 0;
+
+#endif
+	int order = len2order(len + plus);
+
+	if (unlikely(order < 0)) {
+		BRICK_ERR("trying to allocate %d bytes (max = %d)\n", len, (int)(PAGE_SIZE << order));
+		return NULL;
+	}
+
+#ifdef CONFIG_MARS_DEBUG
+	might_sleep();
+#endif
+
+#ifdef CONFIG_MARS_MEM_PREALLOC
+	count = atomic_add_return(1, &_alloc_count[order]);
+	brick_mem_alloc_count[order] = count;
+	if (count > brick_mem_alloc_max[order])
+		brick_mem_alloc_max[order] = count;
+#endif
+
+#ifdef BRICK_DEBUG_MEM
+	atomic_inc(&op_count[order]);
+	/*  statistics */
+	alloc_line[order] = line;
+	alloc_len[order] = len;
+#endif
+
+#ifdef CONFIG_MARS_MEM_PREALLOC
+	/* Dynamic increase of limits, in order to reduce
+	 * fragmentation on higher-order pages.
+	 * This comes on cost of higher memory usage.
+	 */
+	if (order > 0 && count > brick_mem_freelist_max[order])
+		brick_mem_freelist_max[order] = count;
+#endif
+
+#ifdef CONFIG_MARS_MEM_PREALLOC
+	data = _get_free(order, line);
+	if (!data)
+#endif
+		data = __brick_block_alloc(GFP_BRICK, order, line);
+
+#ifdef BRICK_DEBUG_MEM
+	if (order > 0) {
+		if (unlikely(line < 0))
+			line = 0;
+		else if (unlikely(line >= BRICK_DEBUG_MEM))
+			line = BRICK_DEBUG_MEM - 1;
+		atomic_inc(&block_count[line]);
+		block_len[line] = len;
+		if (order > 1) {
+			INT_ACCESS(data, 0 * sizeof(int)) = MAGIC_BLOCK;
+			INT_ACCESS(data, 1 * sizeof(int)) = line;
+			INT_ACCESS(data, 2 * sizeof(int)) = len;
+			data += PAGE_SIZE;
+			INT_ACCESS(data, -1 * sizeof(int)) = MAGIC_BLOCK;
+			INT_ACCESS(data, len) = MAGIC_BEND;
+		} else if (order == 1) {
+			INT_ACCESS(data, PAGE_SIZE + 0 * sizeof(int)) = MAGIC_BLOCK;
+			INT_ACCESS(data, PAGE_SIZE + 1 * sizeof(int)) = line;
+			INT_ACCESS(data, PAGE_SIZE + 2 * sizeof(int)) = len;
+		}
+	}
+#endif
+	return data;
+}
+
+void _brick_block_free(void *data, int len, int cline)
+{
+	int order;
+
+#ifdef CONFIG_MARS_DEBUG_MEM_STRONG
+	struct mem_block_info *inf;
+	char *real_data;
+
+#endif
+#ifdef BRICK_DEBUG_MEM
+	int prev_line = 0;
+
+#ifdef BRICK_DEBUG_ORDER0
+	const int plus0 = PAGE_SIZE;
+
+#else
+	const int plus0 = 0;
+
+#endif
+	const int plus = len <= PAGE_SIZE ? plus0 : PAGE_SIZE * 2;
+
+#else
+	const int plus = 0;
+
+#endif
+
+	order = len2order(len + plus);
+#ifdef CONFIG_MARS_DEBUG_MEM_STRONG
+	real_data = data;
+	if (order > 1)
+		real_data -= PAGE_SIZE;
+	inf = _find_block_info(real_data, false);
+	if (likely(inf)) {
+		prev_line = inf->inf_line;
+		if (unlikely(inf->inf_len != (PAGE_SIZE << order))) {
+			BRICK_ERR("line %d: address %p: bad freeing size %d (correct should be %d, previous line = %d)\n",
+				  cline, data, (int)(PAGE_SIZE << order), inf->inf_len, prev_line);
+			goto _out_return;
+		}
+		if (unlikely(!inf->inf_used)) {
+			BRICK_ERR("line %d: address %p: double freeing (previous line = %d)\n",
+				cline,
+				data,
+				prev_line);
+			goto _out_return;
+		}
+		inf->inf_line = cline;
+		inf->inf_used = false;
+	} else {
+		BRICK_ERR("line %d: trying to free non-existent address %p (order = %d)\n", cline, data, order);
+		goto _out_return;
+	}
+#endif
+#ifdef BRICK_DEBUG_MEM
+	if (order > 1) {
+		void *test = data - PAGE_SIZE;
+		int magic = INT_ACCESS(test, 0);
+		int line = INT_ACCESS(test, sizeof(int));
+		int oldlen = INT_ACCESS(test, sizeof(int)*2);
+		int magic1 = INT_ACCESS(data, -1 * sizeof(int));
+		int magic2;
+
+		if (unlikely(magic1 != MAGIC_BLOCK)) {
+			BRICK_ERR("line %d memory corruption: %p magix1 %08x != %08x (previous line = %d)\n",
+				cline,
+				data,
+				magic1,
+				MAGIC_BLOCK,
+				prev_line);
+			goto _out_return;
+		}
+		if (unlikely(magic != MAGIC_BLOCK)) {
+			BRICK_ERR("line %d memory corruption: %p magix %08x != %08x (previous line = %d)\n",
+				cline,
+				data,
+				magic,
+				MAGIC_BLOCK,
+				prev_line);
+			goto _out_return;
+		}
+		if (unlikely(line < 0 || line >= BRICK_DEBUG_MEM)) {
+			BRICK_ERR("line %d memory corruption %p: alloc line = %d (previous line = %d)\n",
+				cline,
+				data,
+				line,
+				prev_line);
+			goto _out_return;
+		}
+		if (unlikely(oldlen != len)) {
+			BRICK_ERR("line %d memory corruption %p: len != oldlen (%d != %d, previous line = %d))\n",
+				cline,
+				data,
+				len,
+				oldlen,
+				prev_line);
+			goto _out_return;
+		}
+		magic2 = INT_ACCESS(data, len);
+		if (unlikely(magic2 != MAGIC_BEND)) {
+			BRICK_ERR("line %d memory corruption %p: magix %08x != %08x (previous line = %d)\n",
+				cline,
+				data,
+				magic,
+				MAGIC_BEND,
+				prev_line);
+			goto _out_return;
+		}
+		INT_ACCESS(test, 0) = 0xffffffff;
+		INT_ACCESS(data, len) = 0xffffffff;
+		data = test;
+		atomic_dec(&block_count[line]);
+		atomic_inc(&block_free[line]);
+	} else if (order == 1) {
+		void *test = data + PAGE_SIZE;
+		int magic = INT_ACCESS(test, 0 * sizeof(int));
+		int line = INT_ACCESS(test, 1 * sizeof(int));
+		int oldlen = INT_ACCESS(test, 2 * sizeof(int));
+
+		if (unlikely(magic != MAGIC_BLOCK)) {
+			BRICK_ERR("line %d memory corruption %p: magix %08x != %08x (previous line = %d)\n",
+				cline,
+				data,
+				magic,
+				MAGIC_BLOCK,
+				prev_line);
+			goto _out_return;
+		}
+		if (unlikely(line < 0 || line >= BRICK_DEBUG_MEM)) {
+			BRICK_ERR("line %d memory corruption %p: alloc line = %d (previous line = %d)\n",
+				cline,
+				data,
+				line,
+				prev_line);
+			goto _out_return;
+		}
+		if (unlikely(oldlen != len)) {
+			BRICK_ERR("line %d memory corruption %p: len != oldlen (%d != %d, previous line = %d))\n",
+				cline,
+				data,
+				len,
+				oldlen,
+				prev_line);
+			goto _out_return;
+		}
+		atomic_dec(&block_count[line]);
+		atomic_inc(&block_free[line]);
+	}
+#endif
+#ifdef CONFIG_MARS_MEM_PREALLOC
+	if (order > 0 && brick_allow_freelist && atomic_read(&freelist_count[order]) <= brick_mem_freelist_max[order]) {
+		_put_free(data, order);
+	} else
+#endif
+		__brick_block_free(data, order, cline);
+
+#ifdef CONFIG_MARS_MEM_PREALLOC
+	brick_mem_alloc_count[order] = atomic_dec_return(&_alloc_count[order]);
+#endif
+#ifdef BRICK_DEBUG_MEM
+_out_return:;
+#endif
+}
+
+struct page *brick_iomap(void *data, int *offset, int *len)
+{
+	int _offset = ((unsigned long)data) & (PAGE_SIZE-1);
+	struct page *page;
+
+	*offset = _offset;
+	if (*len > PAGE_SIZE - _offset)
+		*len = PAGE_SIZE - _offset;
+	if (is_vmalloc_addr(data))
+		page = vmalloc_to_page(data);
+	else
+		page = virt_to_page(data);
+	return page;
+}
+
+/***********************************************************************/
+
+/*  module */
+
+void brick_mem_statistics(bool final)
+{
+#ifdef BRICK_DEBUG_MEM
+	int i;
+	int count = 0;
+	int places = 0;
+
+	BRICK_INF("======== page allocation:\n");
+#ifdef CONFIG_MARS_MEM_PREALLOC
+	for (i = 0; i <= BRICK_MAX_ORDER; i++) {
+		BRICK_INF("pages order = %2d operations = %9d freelist_count = %4d / %3d raw_count = %5d alloc_count = %5d alloc_len = %5d line = %5d max_count = %5d\n",
+			  i,
+			  atomic_read(&op_count[i]),
+			  atomic_read(&freelist_count[i]),
+			  brick_mem_freelist_max[i],
+			  atomic_read(&raw_count[i]),
+			  brick_mem_alloc_count[i],
+			  alloc_len[i],
+			  alloc_line[i],
+			  brick_mem_alloc_max[i]);
+	}
+#endif
+	for (i = 0; i < BRICK_DEBUG_MEM; i++) {
+		int val = atomic_read(&block_count[i]);
+
+		if (val) {
+			count += val;
+			places++;
+			BRICK_INF("line %4d: %6d allocated (last size = %4d, freed = %6d)\n",
+				  i,
+				  val,
+				  block_len[i],
+				  atomic_read(&block_free[i]));
+		}
+	}
+	if (!final || !count) {
+		BRICK_INF("======== %d block allocations in %d places (phys=%d)\n",
+			  count, places, atomic_read(&phys_block_alloc));
+	} else {
+		BRICK_ERR("======== %d block allocations in %d places (phys=%d)\n",
+			  count, places, atomic_read(&phys_block_alloc));
+	}
+	count = places = 0;
+	for (i = 0; i < BRICK_DEBUG_MEM; i++) {
+		int val = atomic_read(&mem_count[i]);
+
+		if (val) {
+			count += val;
+			places++;
+			BRICK_INF("line %4d: %6d allocated (last size = %4d, freed = %6d)\n",
+				  i,
+				  val,
+				  mem_len[i],
+				  atomic_read(&mem_free[i]));
+		}
+	}
+	if (!final || !count) {
+		BRICK_INF("======== %d memory allocations in %d places (phys=%d,redirect=%d)\n",
+			  count, places,
+			  atomic_read(&phys_mem_alloc), atomic_read(&mem_redirect_alloc));
+	} else {
+		BRICK_ERR("======== %d memory allocations in %d places (phys=%d,redirect=%d)\n",
+			  count, places,
+			  atomic_read(&phys_mem_alloc), atomic_read(&mem_redirect_alloc));
+	}
+	count = places = 0;
+	for (i = 0; i < BRICK_DEBUG_MEM; i++) {
+		int val = atomic_read(&string_count[i]);
+
+		if (val) {
+			count += val;
+			places++;
+			BRICK_INF("line %4d: %6d allocated (freed = %6d)\n",
+				  i,
+				  val,
+				  atomic_read(&string_free[i]));
+		}
+	}
+	if (!final || !count) {
+		BRICK_INF("======== %d string allocations in %d places (phys=%d)\n",
+			  count, places, atomic_read(&phys_string_alloc));
+	} else {
+		BRICK_ERR("======== %d string allocations in %d places (phys=%d)\n",
+			  count, places, atomic_read(&phys_string_alloc));
+	}
+#endif
+}
+
+/*  module init stuff */
+
+int __init init_brick_mem(void)
+{
+	int i;
+
+#ifdef CONFIG_MARS_MEM_PREALLOC
+	for (i = BRICK_MAX_ORDER; i >= 0; i--)
+		spin_lock_init(&freelist_lock[i]);
+#endif
+#ifdef CONFIG_MARS_DEBUG_MEM_STRONG
+	for (i = 0; i < MAX_INFO_LISTS; i++) {
+		INIT_LIST_HEAD(&inf_anchor[i]);
+		rwlock_init(&inf_lock[i]);
+	}
+#else
+	(void)i;
+#endif
+
+	get_total_ram();
+
+	return 0;
+}
+
+void exit_brick_mem(void)
+{
+	BRICK_INF("deallocating memory...\n");
+#ifdef CONFIG_MARS_MEM_PREALLOC
+	_free_all();
+#endif
+
+	brick_mem_statistics(true);
+}
include/linux/brick/brick_mem.h new file +218 -0

218 lines not on this page · raw message

Signature · 1 line
2.6.4
#4 lore raw

RFC 4/31 mars: add new module brick_checking

In reply to Thomas Schoebel-Theuer
---
 include/linux/brick/brick_checking.h | 104 +++++++++++++++++++++++++++++++++++
 1 file changed, 104 insertions(+)
 create mode 100644 include/linux/brick/brick_checking.h
include/linux/brick/brick_checking.h new file +104 -0
new file mode 100644
index 0000000..a02f1bf
@@ -0,0 +1,104 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef BRICK_CHECKING_H
+#define BRICK_CHECKING_H
+
+/***********************************************************************/
+
+/*  checking */
+
+#if defined(CONFIG_MARS_DEBUG) || defined(CONFIG_MARS_CHECKS)
+#define BRICK_CHECKING			true
+#else
+#define BRICK_CHECKING			false
+#endif
+
+#define _CHECK_ATOMIC(atom, OP, minval)					\
+do {									\
+	if (BRICK_CHECKING) {						\
+		int __test = atomic_read(atom);				\
+		if (unlikely(__test OP(minval))) {			\
+			atomic_set(atom, minval);			\
+			BRICK_ERR("%d: atomic " #atom " " #OP " " #minval " (%d)\n", __LINE__, __test);\
+		}							\
+	}								\
+} while (0)
+
+#define CHECK_ATOMIC(atom, minval)					\
+	_CHECK_ATOMIC(atom, <, minval)
+
+#define CHECK_HEAD_EMPTY(head)						\
+do {									\
+	if (BRICK_CHECKING && unlikely(!list_empty(head) && (head)->next)) {\
+		list_del_init(head);					\
+		BRICK_ERR("%d: list_head " #head " (%p) not empty\n", __LINE__, head);\
+	}								\
+} while (0)
+
+#ifdef CONFIG_MARS_DEBUG_MEM
+#define CHECK_PTR_DEAD(ptr, label)					\
+do {									\
+	if (BRICK_CHECKING && unlikely((ptr) == (void *)0x5a5a5a5a5a5a5a5a)) {\
+		BRICK_FAT("%d: pointer '" #ptr "' is DEAD\n", __LINE__);\
+		goto label;						\
+	}								\
+} while (0)
+#else
+#define CHECK_PTR_DEAD(ptr, label) /*empty*/
+#endif
+
+#define CHECK_PTR_NULL(ptr, label)					\
+do {									\
+	CHECK_PTR_DEAD(ptr, label);					\
+	if (BRICK_CHECKING && unlikely(!(ptr))) {			\
+		BRICK_FAT("%d: pointer '" #ptr "' is NULL\n", __LINE__);\
+		goto label;						\
+	}								\
+} while (0)
+
+#ifdef CONFIG_MARS_DEBUG
+#define CHECK_PTR(ptr, label)						\
+do {									\
+	CHECK_PTR_NULL(ptr, label);					\
+	if (BRICK_CHECKING && unlikely(!virt_addr_valid(ptr))) {	\
+		BRICK_FAT("%d: pointer '" #ptr "' (%p) is no valid virtual KERNEL address\n", __LINE__, ptr);\
+		goto label;						\
+	}								\
+} while (0)
+#else
+#define CHECK_PTR(ptr, label) CHECK_PTR_NULL(ptr, label)
+#endif
+
+#define CHECK_ASPECT(a_ptr, o_ptr, label)				\
+do {									\
+	if (BRICK_CHECKING && unlikely((a_ptr)->object != o_ptr)) {	\
+		BRICK_FAT("%d: aspect pointer '" #a_ptr "' (%p) belongs to object %p, not to " #o_ptr " (%p)\n",\
+			__LINE__, a_ptr, (a_ptr)->object, o_ptr);	\
+		goto label;						\
+	}								\
+} while (0)
+
+#define _CHECK(ptr, label)						\
+do {									\
+	if (BRICK_CHECKING && unlikely(!(ptr))) {			\
+		BRICK_FAT("%d: condition '" #ptr "' is VIOLATED\n", __LINE__);\
+		goto label;						\
+	}								\
+} while (0)
+
+#endif
Signature · 1 line
2.6.4
#5 lore raw

RFC 5/31 mars: add new module meta

In reply to Thomas Schoebel-Theuer
---
 include/linux/brick/meta.h | 106 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 106 insertions(+)
 create mode 100644 include/linux/brick/meta.h
include/linux/brick/meta.h new file +106 -0
new file mode 100644
index 0000000..a92b2b6
@@ -0,0 +1,106 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef META_H
+#define META_H
+
+/***********************************************************************/
+
+/*  metadata descriptions */
+
+/* The idea is to describe your C structures in such a way that
+ * transfers to disk or over a network become self-describing.
+ *
+ * In essence, this is a kind of version-independent marshalling.
+ *
+ * Advantage:
+ * When you extend your original C struct (and of course update the
+ * corresponding meta structure), old data on disk (or network peers
+ * running an old version of your program) will remain valid.
+ * Upon read, newly added fields missing in the old version will be simply
+ * not filled in and therefore remain zeroed (if you don't forget to
+ * initially clear your structures via memset() / initializers / etc).
+ * Note that this works only if you never rename or remove existing
+ * fields; you should only add new ones.
+ * [TODO: add macros for description of ignored / renamed fields to
+ *  overcome this limitation]
+ * You may increase the size of integers, for example from 32bit to 64bit
+ * or even higher; sign extension will be automatically carried out
+ * when necessary.
+ * Also, you may change the order of fields, because the metadata interpreter
+ * will check each field individually; field offsets are automatically
+ * maintained.
+ *
+ * Disadvantage: this adds some (small) overhead.
+ */
+
+enum field_type {
+	FIELD_DONE,
+	FIELD_REF,
+	FIELD_SUB,
+	FIELD_STRING,
+	FIELD_RAW,
+	FIELD_INT,
+	FIELD_UINT,
+};
+
+struct meta {
+	/* char field_name[MAX_FIELD_LEN]; */
+	char *field_name;
+
+	short field_type;
+	short field_data_size;
+	short field_transfer_size;
+	int   field_offset;
+	const struct meta *field_ref;
+};
+
+#define _META_INI(NAME, STRUCT, TYPE, TSIZE)				\
+	.field_name = #NAME,						\
+	.field_type = TYPE,						\
+	.field_data_size = sizeof(((STRUCT *)NULL)->NAME),		\
+	.field_transfer_size = (TSIZE),					\
+	.field_offset = offsetof(STRUCT, NAME)				\
+
+#define META_INI_TRANSFER(NAME, STRUCT, TYPE, TSIZE)			\
+	{ _META_INI(NAME, STRUCT, TYPE, TSIZE) }
+
+#define META_INI(NAME, STRUCT, TYPE)					\
+	{ _META_INI(NAME, STRUCT, TYPE, 0) }
+
+#define _META_INI_AIO(NAME, STRUCT, AIO)				\
+	.field_name = #NAME,						\
+	.field_type = FIELD_REF,					\
+	.field_data_size = sizeof(*(((STRUCT *)NULL)->NAME)),		\
+	.field_offset = offsetof(STRUCT, NAME),				\
+	.field_ref = AIO
+
+#define META_INI_AIO(NAME, STRUCT, AIO) { _META_INI_AIO(NAME, STRUCT, AIO) }
+
+#define _META_INI_SUB(NAME, STRUCT, SUB)				\
+	.field_name = #NAME,						\
+	.field_type = FIELD_SUB,					\
+	.field_data_size = sizeof(((STRUCT *)NULL)->NAME),		\
+	.field_offset = offsetof(STRUCT, NAME),				\
+	.field_ref = SUB
+
+#define META_INI_SUB(NAME, STRUCT, SUB) { _META_INI_SUB(NAME, STRUCT, SUB) }
+
+extern const struct meta *find_meta(const struct meta *meta, const char *field_name);
+/* extern void free_meta(void *data, const struct meta *meta); */
+
+#endif
Signature · 1 line
2.6.4
#6 lore raw

RFC 6/31 mars: add new module brick

In reply to Thomas Schoebel-Theuer
---
 drivers/staging/mars/brick.c | 728 +++++++++++++++++++++++++++++++++++++++++++
 include/linux/brick/brick.h  | 642 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 1370 insertions(+)
 create mode 100644 drivers/staging/mars/brick.c
 create mode 100644 include/linux/brick/brick.h
drivers/staging/mars/brick.c new file +728 -0
new file mode 100644
index 0000000..9c3d5b9
@@ -0,0 +1,728 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/string.h>
+
+#define _STRATEGY
+
+#include <linux/brick/brick.h>
+#include <linux/brick/brick_mem.h>
+
+/************************************************************/
+
+/*  init / exit functions */
+
+void _generic_output_init(struct generic_brick *brick,
+	const struct generic_output_type *type,
+	struct generic_output *output)
+{
+	output->brick = brick;
+	output->type = type;
+	output->ops = type->master_ops;
+	output->nr_connected = 0;
+	INIT_LIST_HEAD(&output->output_head);
+}
+
+void _generic_output_exit(struct generic_output *output)
+{
+	list_del_init(&output->output_head);
+	output->brick = NULL;
+	output->type = NULL;
+	output->ops = NULL;
+	output->nr_connected = 0;
+}
+
+int generic_brick_init(const struct generic_brick_type *type, struct generic_brick *brick)
+{
+	brick->aspect_context.brick_index = get_brick_nr();
+	brick->type = type;
+	brick->ops = type->master_ops;
+	brick->nr_inputs = 0;
+	brick->nr_outputs = 0;
+	brick->power.off_led = true;
+	init_waitqueue_head(&brick->power.event);
+	INIT_LIST_HEAD(&brick->tmp_head);
+	return 0;
+}
+
+void generic_brick_exit(struct generic_brick *brick)
+{
+	list_del_init(&brick->tmp_head);
+	brick->type = NULL;
+	brick->ops = NULL;
+	brick->nr_inputs = 0;
+	brick->nr_outputs = 0;
+	put_brick_nr(brick->aspect_context.brick_index);
+}
+
+int generic_input_init(struct generic_brick *brick,
+	int index,
+	const struct generic_input_type *type,
+	struct generic_input *input)
+{
+	if (index < 0 || index >= brick->type->max_inputs)
+		return -EINVAL;
+	if (brick->inputs[index])
+		return -EEXIST;
+	input->brick = brick;
+	input->type = type;
+	input->connect = NULL;
+	INIT_LIST_HEAD(&input->input_head);
+	brick->inputs[index] = input;
+	brick->nr_inputs++;
+	return 0;
+}
+
+void generic_input_exit(struct generic_input *input)
+{
+	list_del_init(&input->input_head);
+	input->brick = NULL;
+	input->type = NULL;
+	input->connect = NULL;
+}
+
+int generic_output_init(struct generic_brick *brick,
+	int index,
+	const struct generic_output_type *type,
+	struct generic_output *output)
+{
+	if (index < 0 || index >= brick->type->max_outputs)
+		return -ENOMEM;
+	if (brick->outputs[index])
+		return -EEXIST;
+	_generic_output_init(brick, type, output);
+	brick->outputs[index] = output;
+	brick->nr_outputs++;
+	return 0;
+}
+
+int generic_size(const struct generic_brick_type *brick_type)
+{
+	int size = brick_type->brick_size;
+	int i;
+
+	size += brick_type->max_inputs * sizeof(void *);
+	for (i = 0; i < brick_type->max_inputs; i++)
+		size += brick_type->default_input_types[i]->input_size;
+	size += brick_type->max_outputs * sizeof(void *);
+	for (i = 0; i < brick_type->max_outputs; i++)
+		size += brick_type->default_output_types[i]->output_size;
+	return size;
+}
+
+int generic_connect(struct generic_input *input, struct generic_output *output)
+{
+	BRICK_DBG("generic_connect(input=%p, output=%p)\n", input, output);
+	if (unlikely(!input || !output))
+		return -EINVAL;
+	if (unlikely(input->connect))
+		return -EEXIST;
+	if (unlikely(!list_empty(&input->input_head)))
+		return -EINVAL;
+	/*  helps only against the most common errors */
+	if (unlikely(input->brick == output->brick))
+		return -EDEADLK;
+
+	input->connect = output;
+	output->nr_connected++;
+	list_add(&input->input_head, &output->output_head);
+	return 0;
+}
+
+int generic_disconnect(struct generic_input *input)
+{
+	struct generic_output *connect;
+
+	BRICK_DBG("generic_disconnect(input=%p)\n", input);
+	if (!input)
+		return -EINVAL;
+	connect = input->connect;
+	if (connect) {
+		connect->nr_connected--;
+		input->connect = NULL;
+		list_del_init(&input->input_head);
+	}
+	return 0;
+}
+
+/************************************************************/
+
+/*  general */
+
+int _brick_msleep(int msecs, bool shorten)
+{
+	unsigned long timeout;
+
+	flush_signals(current);
+	if (msecs <= 0) {
+		schedule();
+		return 0;
+	}
+	timeout = msecs_to_jiffies(msecs) + 1;
+
+	timeout = schedule_timeout_interruptible(timeout);
+
+	if (!shorten) {
+		while ((long)timeout > 0)
+			timeout = schedule_timeout_uninterruptible(timeout);
+	}
+
+	return jiffies_to_msecs(timeout);
+}
+
+/************************************************************/
+
+/*  number management */
+
+static char *nr_table;
+int nr_max = 256;
+
+int get_brick_nr(void)
+{
+	char *new;
+	int nr;
+
+	if (unlikely(!nr_table))
+		nr_table = brick_zmem_alloc(nr_max);
+
+	for (;;) {
+		for (nr = 1; nr < nr_max; nr++) {
+			if (!nr_table[nr]) {
+				nr_table[nr] = 1;
+				return nr;
+			}
+		}
+		new = brick_zmem_alloc(nr_max << 1);
+		memcpy(new, nr_table, nr_max);
+		brick_mem_free(nr_table);
+		nr_table = new;
+		nr_max <<= 1;
+	}
+}
+
+void put_brick_nr(int nr)
+{
+	if (likely(nr_table && nr > 0 && nr < nr_max))
+		nr_table[nr] = 0;
+}
+
+/************************************************************/
+
+/*  object stuff */
+
+/************************************************************/
+
+/*  brick stuff */
+
+static int nr_brick_types;
+static const struct generic_brick_type *brick_types[MAX_BRICK_TYPES];
+
+int generic_register_brick_type(const struct generic_brick_type *new_type)
+{
+	int i;
+	int found = -1;
+
+	BRICK_DBG("generic_register_brick_type() name=%s\n", new_type->type_name);
+	for (i = 0; i < nr_brick_types; i++) {
+		if (!brick_types[i]) {
+			found = i;
+			continue;
+		}
+		if (!strcmp(brick_types[i]->type_name, new_type->type_name))
+			return 0;
+	}
+	if (found < 0) {
+		if (nr_brick_types >= MAX_BRICK_TYPES) {
+			BRICK_ERR("sorry, cannot register bricktype %s.\n", new_type->type_name);
+			return -ENOMEM;
+		}
+		found = nr_brick_types++;
+	}
+	brick_types[found] = new_type;
+	BRICK_DBG("generic_register_brick_type() done.\n");
+	return 0;
+}
+
+int generic_unregister_brick_type(const struct generic_brick_type *old_type)
+{
+	BRICK_DBG("generic_unregister_brick_type()\n");
+	return -1; /*  NYI */
+}
+
+int generic_brick_init_full(
+	void *data,
+	int size,
+	const struct generic_brick_type *brick_type,
+	const struct generic_input_type **input_types,
+	const struct generic_output_type **output_types)
+{
+	struct generic_brick *brick = data;
+	int status;
+	int i;
+
+	if (unlikely(!data)) {
+		BRICK_ERR("invalid memory\n");
+		return -EINVAL;
+	}
+
+	/*  call the generic constructors */
+
+	status = generic_brick_init(brick_type, brick);
+	if (status)
+		return status;
+	data += brick_type->brick_size;
+	size -= brick_type->brick_size;
+	if (size < 0) {
+		BRICK_ERR("Not enough MEMORY\n");
+		return -ENOMEM;
+	}
+	if (!input_types) {
+		input_types = brick_type->default_input_types;
+		if (unlikely(!input_types)) {
+			BRICK_ERR("no input types specified\n");
+			return -EINVAL;
+		}
+	}
+	brick->inputs = data;
+	data += sizeof(void *) * brick_type->max_inputs;
+	size -= sizeof(void *) * brick_type->max_inputs;
+	if (size < 0)
+		return -ENOMEM;
+	for (i = 0; i < brick_type->max_inputs; i++) {
+		struct generic_input *input = data;
+		const struct generic_input_type *type = *input_types++;
+
+		if (!type || type->input_size <= 0)
+			return -EINVAL;
+		BRICK_DBG("generic_brick_init_full: calling generic_input_init()\n");
+		status = generic_input_init(brick, i, type, input);
+		if (status < 0)
+			return status;
+		data += type->input_size;
+		size -= type->input_size;
+		if (size < 0)
+			return -ENOMEM;
+	}
+	if (!output_types) {
+		output_types = brick_type->default_output_types;
+		if (unlikely(!output_types)) {
+			BRICK_ERR("no output types specified\n");
+			return -EINVAL;
+		}
+	}
+	brick->outputs = data;
+	data += sizeof(void *) * brick_type->max_outputs;
+	size -= sizeof(void *) * brick_type->max_outputs;
+	if (size < 0)
+		return -ENOMEM;
+	for (i = 0; i < brick_type->max_outputs; i++) {
+		struct generic_output *output = data;
+		const struct generic_output_type *type = *output_types++;
+
+		if (!type || type->output_size <= 0)
+			return -EINVAL;
+		BRICK_DBG("generic_brick_init_full: calling generic_output_init()\n");
+		generic_output_init(brick, i, type, output);
+		if (status < 0)
+			return status;
+		data += type->output_size;
+		size -= type->output_size;
+		if (size < 0)
+			return -ENOMEM;
+	}
+
+	/*  call the specific constructors */
+	if (brick_type->brick_construct) {
+		BRICK_DBG("generic_brick_init_full: calling brick_construct()\n");
+		status = brick_type->brick_construct(brick);
+		if (status < 0)
+			return status;
+	}
+	for (i = 0; i < brick_type->max_inputs; i++) {
+		struct generic_input *input = brick->inputs[i];
+
+		if (!input)
+			continue;
+		if (!input->type) {
+			BRICK_ERR("input has no associated type!\n");
+			continue;
+		}
+		if (input->type->input_construct) {
+			BRICK_DBG("generic_brick_init_full: calling input_construct()\n");
+			status = input->type->input_construct(input);
+			if (status < 0)
+				return status;
+		}
+	}
+	for (i = 0; i < brick_type->max_outputs; i++) {
+		struct generic_output *output = brick->outputs[i];
+
+		if (!output)
+			continue;
+		if (!output->type) {
+			BRICK_ERR("output has no associated type!\n");
+			continue;
+		}
+		if (output->type->output_construct) {
+			BRICK_DBG("generic_brick_init_full: calling output_construct()\n");
+			status = output->type->output_construct(output);
+			if (status < 0)
+				return status;
+		}
+	}
+	return 0;
+}
+
+int generic_brick_exit_full(struct generic_brick *brick)
+{
+	int i;
+	int status;
+
+	/*  first, check all outputs */
+	for (i = 0; i < brick->type->max_outputs; i++) {
+		struct generic_output *output = brick->outputs[i];
+
+		if (!output)
+			continue;
+		if (!output->type) {
+			BRICK_ERR("output has no associated type!\n");
+			continue;
+		}
+		if (output->nr_connected) {
+			BRICK_ERR("output is connected!\n");
+			return -EPERM;
+		}
+	}
+	/*  ok, test succeeded. start destruction... */
+	for (i = 0; i < brick->type->max_outputs; i++) {
+		struct generic_output *output = brick->outputs[i];
+
+		if (!output)
+			continue;
+		if (!output->type) {
+			BRICK_ERR("output has no associated type!\n");
+			continue;
+		}
+		if (output->type->output_destruct) {
+			BRICK_DBG("generic_brick_exit_full: calling output_destruct()\n");
+			status = output->type->output_destruct(output);
+			if (status < 0)
+				return status;
+			_generic_output_exit(output);
+			brick->outputs[i] = NULL; /*  others may remain leftover */
+		}
+	}
+	for (i = 0; i < brick->type->max_inputs; i++) {
+		struct generic_input *input = brick->inputs[i];
+
+		if (!input)
+			continue;
+		if (!input->type) {
+			BRICK_ERR("input has no associated type!\n");
+			continue;
+		}
+		if (input->type->input_destruct) {
+			status = generic_disconnect(input);
+			if (status < 0)
+				return status;
+			BRICK_DBG("generic_brick_exit_full: calling input_destruct()\n");
+			status = input->type->input_destruct(input);
+			if (status < 0)
+				return status;
+			brick->inputs[i] = NULL; /*  others may remain leftover */
+			generic_input_exit(input);
+		}
+	}
+	if (brick->type->brick_destruct) {
+		BRICK_DBG("generic_brick_exit_full: calling brick_destruct()\n");
+		status = brick->type->brick_destruct(brick);
+		if (status < 0)
+			return status;
+	}
+	generic_brick_exit(brick);
+	return 0;
+}
+
+/**********************************************************************/
+
+/*  default implementations */
+
+struct generic_object *generic_alloc(struct generic_object_layout *object_layout,
+	const struct generic_object_type *object_type)
+{
+	struct generic_object *object;
+	void *data;
+	int object_size;
+	int aspect_nr_max;
+	int total_size;
+	int hint_size;
+
+	CHECK_PTR_NULL(object_type, err);
+	CHECK_PTR(object_layout, err);
+
+	object_size = object_type->default_size;
+	aspect_nr_max = nr_max;
+	total_size = object_size + aspect_nr_max * sizeof(void *);
+	hint_size = object_layout->size_hint;
+	if (likely(total_size <= hint_size)) {
+		total_size = hint_size;
+	} else { /*  usually happens only at the first time */
+		object_layout->size_hint = total_size;
+	}
+
+	data = brick_zmem_alloc(total_size);
+
+	atomic_inc(&object_layout->alloc_count);
+	atomic_inc(&object_layout->total_alloc_count);
+
+	object = data;
+	object->object_type = object_type;
+	object->object_layout = object_layout;
+	object->aspects = data + object_size;
+	object->aspect_nr_max = aspect_nr_max;
+	object->free_offset = object_size + aspect_nr_max * sizeof(void *);
+	object->max_offset = total_size;
+
+	if (object_type->init_fn) {
+		int status = object_type->init_fn(object);
+
+		if (status < 0)
+			goto err_free;
+	}
+
+	return object;
+
+err_free:
+	brick_mem_free(data);
+err:
+	return NULL;
+}
+
+void generic_free(struct generic_object *object)
+{
+	const struct generic_object_type *object_type;
+	struct generic_object_layout *object_layout;
+	int i;
+
+	CHECK_PTR(object, done);
+	object_type = object->object_type;
+	CHECK_PTR_NULL(object_type, done);
+	object_layout = object->object_layout;
+	CHECK_PTR(object_layout, done);
+	_CHECK_ATOMIC(&object->obj_count, !=, 0);
+
+	atomic_dec(&object_layout->alloc_count);
+	for (i = 0; i < object->aspect_nr_max; i++) {
+		const struct generic_aspect_type *aspect_type;
+		struct generic_aspect *aspect = object->aspects[i];
+
+		if (!aspect)
+			continue;
+		object->aspects[i] = NULL;
+		aspect_type = aspect->aspect_type;
+		CHECK_PTR_NULL(aspect_type, done);
+		if (aspect_type->exit_fn)
+			aspect_type->exit_fn(aspect);
+		if (aspect->shortcut)
+			continue;
+		brick_mem_free(aspect);
+		atomic_dec(&object_layout->aspect_count);
+	}
+	if (object_type->exit_fn)
+		object_type->exit_fn(object);
+	brick_mem_free(object);
+done:;
+}
+
+static inline
+struct generic_aspect *_new_aspect(const struct generic_aspect_type *aspect_type, struct generic_object *obj)
+{
+	struct generic_aspect *res = NULL;
+	int size;
+	int rest;
+
+	size = aspect_type->aspect_size;
+	rest = obj->max_offset - obj->free_offset;
+	if (likely(size <= rest)) {
+		/* Optimisation: re-use single memory allocation for both
+		 * the object and the new aspect.
+		 */
+		res = ((void *)obj) + obj->free_offset;
+		obj->free_offset += size;
+		res->shortcut = true;
+	} else {
+		struct generic_object_layout *object_layout = obj->object_layout;
+
+		CHECK_PTR(object_layout, done);
+		/* Maintain the size hint.
+		 * In future, only small aspects should be integrated into
+		 * the same memory block, and the hint should not grow larger
+		 * than PAGE_SIZE if it was smaller before.
+		 */
+		if (size < PAGE_SIZE / 2) {
+			int max;
+
+			max = obj->free_offset + size;
+			/* This is racy, but races won't do any harm because
+			 * it is just a hint, not essential.
+			 */
+			if ((max < PAGE_SIZE || object_layout->size_hint > PAGE_SIZE) &&
+			    object_layout->size_hint < max)
+				object_layout->size_hint = max;
+		}
+
+		res = brick_zmem_alloc(size);
+		atomic_inc(&object_layout->aspect_count);
+		atomic_inc(&object_layout->total_aspect_count);
+	}
+	res->object = obj;
+	res->aspect_type = aspect_type;
+
+	if (aspect_type->init_fn) {
+		int status = aspect_type->init_fn(res);
+
+		if (unlikely(status < 0)) {
+			BRICK_ERR("aspect init %p %p %p status = %d\n", aspect_type, obj, res, status);
+			goto done;
+		}
+	}
+
+done:
+	return res;
+}
+
+struct generic_aspect *generic_get_aspect(struct generic_brick *brick, struct generic_object *obj)
+{
+	struct generic_aspect *res = NULL;
+	int nr;
+
+	CHECK_PTR(brick, done);
+	CHECK_PTR(obj, done);
+
+	nr = brick->aspect_context.brick_index;
+	if (unlikely(nr <= 0 || nr >= obj->aspect_nr_max)) {
+		BRICK_ERR("bad nr = %d\n", nr);
+		goto done;
+	}
+
+	res = obj->aspects[nr];
+	if (!res) {
+		const struct generic_object_type *object_type = obj->object_type;
+		const struct generic_brick_type *brick_type = brick->type;
+		const struct generic_aspect_type *aspect_type;
+		int object_type_nr;
+
+		CHECK_PTR_NULL(object_type, done);
+		CHECK_PTR_NULL(brick_type, done);
+		object_type_nr = object_type->object_type_nr;
+		aspect_type = brick_type->aspect_types[object_type_nr];
+		CHECK_PTR_NULL(aspect_type, done);
+
+		res = _new_aspect(aspect_type, obj);
+
+		obj->aspects[nr] = res;
+	}
+	CHECK_PTR(res, done);
+	CHECK_PTR(res->object, done);
+	_CHECK(res->object == obj, done);
+
+done:
+	return res;
+}
+
+/***************************************************************/
+
+/*  helper stuff */
+
+void set_button(struct generic_switch *sw, bool val, bool force)
+{
+	bool oldval = sw->button;
+
+	sw->force_off |= force;
+	if (sw->force_off)
+		val = false;
+	if (val != oldval) {
+		sw->button = val;
+		wake_up_interruptible(&sw->event);
+	}
+}
+
+void set_on_led(struct generic_switch *sw, bool val)
+{
+	bool oldval = sw->on_led;
+
+	if (val != oldval) {
+		sw->on_led = val;
+		wake_up_interruptible(&sw->event);
+	}
+}
+
+void set_off_led(struct generic_switch *sw, bool val)
+{
+	bool oldval = sw->off_led;
+
+	if (val != oldval) {
+		sw->off_led = val;
+		wake_up_interruptible(&sw->event);
+	}
+}
+
+void set_button_wait(struct generic_brick *brick, bool val, bool force, int timeout)
+{
+	set_button(&brick->power, val, force);
+	if (brick->ops)
+		(void)brick->ops->brick_switch(brick);
+	if (val)
+		wait_event_interruptible_timeout(brick->power.event, brick->power.on_led, timeout);
+	else
+		wait_event_interruptible_timeout(brick->power.event, brick->power.off_led, timeout);
+}
+
+/***************************************************************/
+
+/*  meta stuff */
+
+const struct meta *find_meta(const struct meta *meta, const char *field_name)
+{
+	const struct meta *tmp;
+
+	for (tmp = meta; tmp->field_name; tmp++) {
+		if (!strcmp(field_name, tmp->field_name))
+			return tmp;
+	}
+	return NULL;
+}
+
+/***********************************************************************/
+
+/*  module init stuff */
+
+int __init init_brick(void)
+{
+	nr_table = brick_zmem_alloc(nr_max);
+	return 0;
+}
+
+void exit_brick(void)
+{
+	if (nr_table) {
+		brick_mem_free(nr_table);
+		nr_table = NULL;
+	}
+}
include/linux/brick/brick.h new file +642 -0

642 lines not on this page · raw message

Signature · 1 line
2.6.4
#7 lore raw

RFC 7/31 mars: add new module lib_pairing_heap

In reply to Thomas Schoebel-Theuer
---
 include/linux/brick/lib_pairing_heap.h | 110 +++++++++++++++++++++++++++++++++
 1 file changed, 110 insertions(+)
 create mode 100644 include/linux/brick/lib_pairing_heap.h
include/linux/brick/lib_pairing_heap.h new file +110 -0
new file mode 100644
index 0000000..eb97097
@@ -0,0 +1,110 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef PAIRING_HEAP_H
+#define PAIRING_HEAP_H
+
+/* Algorithm: see http://en.wikipedia.org/wiki/Pairing_heap
+ * This is just an efficient translation from recursive to iterative form.
+ *
+ * Note: find_min() is so trivial that we don't implement it.
+ */
+
+/* generic version: KEYDEF is kept separate, allowing you to
+ * embed this structure into other container structures already
+ * possessing some key (just provide an empty KEYDEF in this case).
+ */
+#define _PAIRING_HEAP_TYPEDEF(KEYTYPE, KEYDEF)				\
+									\
+struct pairing_heap_##KEYTYPE {						\
+	KEYDEF								\
+	struct pairing_heap_##KEYTYPE *next;				\
+	struct pairing_heap_##KEYTYPE *subheaps;			\
+};									\
+/* this comment is for keeping TRAILING_SEMICOLON happy */
+
+/* less generic version: define the key inside.
+ */
+#define PAIRING_HEAP_TYPEDEF(KEYTYPE)					\
+	_PAIRING_HEAP_TYPEDEF(KEYTYPE, KEYTYPE key;)
+
+/* generic methods: allow arbitrary CMP() functions.
+ */
+#define _PAIRING_HEAP_FUNCTIONS(_STATIC, KEYTYPE, CMP)			\
+									\
+_STATIC									\
+struct pairing_heap_##KEYTYPE *_ph_merge_##KEYTYPE(struct pairing_heap_##KEYTYPE *heap1,\
+	struct pairing_heap_##KEYTYPE *heap2)				\
+{									\
+	if (!heap1)							\
+		return heap2;						\
+	if (!heap2)							\
+		return heap1;						\
+	if (CMP(heap1, heap2) < 0) {					\
+		heap2->next = heap1->subheaps;				\
+		heap1->subheaps = heap2;				\
+		return heap1;						\
+	}								\
+	heap1->next = heap2->subheaps;					\
+	heap2->subheaps = heap1;					\
+	return heap2;							\
+}									\
+									\
+_STATIC									\
+void ph_insert_##KEYTYPE(struct pairing_heap_##KEYTYPE **heap, struct pairing_heap_##KEYTYPE *new)\
+{									\
+	new->next = NULL;						\
+	new->subheaps = NULL;						\
+	*heap = _ph_merge_##KEYTYPE(*heap, new);			\
+}									\
+									\
+_STATIC									\
+void ph_delete_min_##KEYTYPE(struct pairing_heap_##KEYTYPE **heap)	\
+{									\
+	struct pairing_heap_##KEYTYPE *tmplist = NULL;			\
+	struct pairing_heap_##KEYTYPE *ptr;				\
+	struct pairing_heap_##KEYTYPE *next;				\
+	struct pairing_heap_##KEYTYPE *res;				\
+	if (!*heap) {							\
+		return;							\
+	}								\
+	for (ptr = (*heap)->subheaps; ptr; ptr = next) {		\
+		struct pairing_heap_##KEYTYPE *p2 = ptr->next;		\
+		next = p2;						\
+		if (p2) {						\
+			next = p2->next;				\
+			ptr = _ph_merge_##KEYTYPE(ptr, p2);		\
+		}							\
+		ptr->next = tmplist;					\
+		tmplist = ptr;						\
+	}								\
+	res = NULL;							\
+	for (ptr = tmplist; ptr; ptr = next) {				\
+		next = ptr->next;					\
+		res = _ph_merge_##KEYTYPE(res, ptr);			\
+	}								\
+	*heap = res;							\
+}
+
+/* some default CMP() function */
+#define PAIRING_HEAP_COMPARE(a, b) ((a)->key < (b)->key ? -1 : ((a)->key > (b)->key ? 1 : 0))
+
+/* less generic version: use the default CMP() function */
+#define PAIRING_HEAP_FUNCTIONS(_STATIC, KEYTYPE)			\
+	_PAIRING_HEAP_FUNCTIONS(_STATIC, KEYTYPE, PAIRING_HEAP_COMPARE)
+
+#endif
Signature · 1 line
2.6.4
#8 lore raw

RFC 8/31 mars: add new module lib_queue

In reply to Thomas Schoebel-Theuer
---
 include/linux/brick/lib_queue.h | 166 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 166 insertions(+)
 create mode 100644 include/linux/brick/lib_queue.h
include/linux/brick/lib_queue.h new file +166 -0
new file mode 100644
index 0000000..f1b1a9e
@@ -0,0 +1,166 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef LIB_QUEUE_H
+#define LIB_QUEUE_H
+
+#define QUEUE_ANCHOR(PREFIX, KEYTYPE, HEAPTYPE)				\
+	/* parameters */						\
+	/* readonly from outside */					\
+	atomic_t q_queued;						\
+	atomic_t q_flying;						\
+	atomic_t q_total;						\
+	/* tunables */							\
+	int q_batchlen;							\
+	int q_io_prio;							\
+	bool q_ordering;						\
+	/* private */							\
+	wait_queue_head_t *q_event;					\
+	spinlock_t q_lock;						\
+	struct list_head q_anchor;					\
+	struct pairing_heap_##HEAPTYPE *heap_high;			\
+	struct pairing_heap_##HEAPTYPE *heap_low;			\
+	long long q_last_insert; /* jiffies */				\
+	KEYTYPE heap_margin;						\
+	KEYTYPE last_pos;						\
+	/* this comment is for keeping TRAILING_SEMICOLON happy */
+
+#define QUEUE_FUNCTIONS(PREFIX, ELEM_TYPE, HEAD, KEYFN, KEYCMP, HEAPTYPE)\
+									\
+static inline								\
+void q_##PREFIX##_trigger(struct PREFIX##_queue *q)			\
+{									\
+	if (q->q_event) {						\
+		wake_up_interruptible(q->q_event);			\
+	}								\
+}									\
+									\
+static inline								\
+void q_##PREFIX##_init(struct PREFIX##_queue *q)			\
+{									\
+	INIT_LIST_HEAD(&q->q_anchor);					\
+	q->heap_low = NULL;						\
+	q->heap_high = NULL;						\
+	spin_lock_init(&q->q_lock);					\
+	atomic_set(&q->q_queued, 0);					\
+	atomic_set(&q->q_flying, 0);					\
+}									\
+									\
+static inline								\
+void q_##PREFIX##_insert(struct PREFIX##_queue *q, ELEM_TYPE * elem)	\
+{									\
+	unsigned long flags;						\
+									\
+	spin_lock_irqsave(&q->q_lock, flags);				\
+									\
+	if (q->q_ordering) {						\
+		struct pairing_heap_##HEAPTYPE **use = &q->heap_high;	\
+		if (KEYCMP(KEYFN(elem), &q->heap_margin) <= 0) {	\
+			use = &q->heap_low;				\
+		}							\
+		ph_insert_##HEAPTYPE(use, &elem->ph);			\
+	} else {							\
+		list_add_tail(&elem->HEAD, &q->q_anchor);		\
+	}								\
+	atomic_inc(&q->q_queued);					\
+	atomic_inc(&q->q_total);					\
+	q->q_last_insert = jiffies;					\
+									\
+	spin_unlock_irqrestore(&q->q_lock, flags);			\
+									\
+	q_##PREFIX##_trigger(q);					\
+}									\
+									\
+static inline								\
+void q_##PREFIX##_pushback(struct PREFIX##_queue *q, ELEM_TYPE * elem)	\
+{									\
+	unsigned long flags;						\
+									\
+	if (q->q_ordering) {						\
+		atomic_dec(&q->q_total);				\
+		q_##PREFIX##_insert(q, elem);				\
+		return;							\
+	}								\
+									\
+	spin_lock_irqsave(&q->q_lock, flags);				\
+									\
+	list_add(&elem->HEAD, &q->q_anchor);				\
+	atomic_inc(&q->q_queued);					\
+									\
+	spin_unlock_irqrestore(&q->q_lock, flags);			\
+}									\
+									\
+static inline								\
+ELEM_TYPE *q_##PREFIX##_fetch(struct PREFIX##_queue *q)			\
+{									\
+	ELEM_TYPE *elem = NULL;						\
+	unsigned long flags;						\
+									\
+	spin_lock_irqsave(&q->q_lock, flags);				\
+									\
+	if (q->q_ordering) {						\
+		if (!q->heap_high) {					\
+			q->heap_high = q->heap_low;			\
+			q->heap_low = NULL;				\
+			q->heap_margin = 0;				\
+			q->last_pos = 0;				\
+		}							\
+		if (q->heap_high) {					\
+			elem = container_of(q->heap_high, ELEM_TYPE, ph);\
+									\
+			if (unlikely(KEYCMP(KEYFN(elem), &q->last_pos) < 0)) {\
+				printk(KERN_ERR "backskip pos %lld -> %lld\n",\
+					(long long)q->last_pos, (long long)KEYFN(elem));\
+			}						\
+			memcpy(&q->last_pos, KEYFN(elem), sizeof(q->last_pos));\
+									\
+			if (KEYCMP(KEYFN(elem), &q->heap_margin) > 0) { \
+				memcpy(&q->heap_margin, KEYFN(elem), sizeof(q->heap_margin));\
+			}						\
+			ph_delete_min_##HEAPTYPE(&q->heap_high);	\
+			atomic_dec(&q->q_queued);			\
+		}							\
+	} else if (!list_empty(&q->q_anchor)) {				\
+		struct list_head *next = q->q_anchor.next;		\
+		list_del_init(next);					\
+		atomic_dec(&q->q_queued);				\
+		elem = container_of(next, ELEM_TYPE, HEAD);		\
+	}								\
+									\
+	spin_unlock_irqrestore(&q->q_lock, flags);			\
+									\
+	q_##PREFIX##_trigger(q);					\
+									\
+	return elem;							\
+}									\
+									\
+static inline								\
+void q_##PREFIX##_inc_flying(struct PREFIX##_queue *q)			\
+{									\
+	atomic_inc(&q->q_flying);					\
+	q_##PREFIX##_trigger(q);					\
+}									\
+									\
+static inline								\
+void q_##PREFIX##_dec_flying(struct PREFIX##_queue *q)			\
+{									\
+	atomic_dec(&q->q_flying);					\
+	q_##PREFIX##_trigger(q);					\
+}									\
+									\
+
+#endif
Signature · 1 line
2.6.4
#9 lore raw

RFC 9/31 mars: add new module lib_rank

In reply to Thomas Schoebel-Theuer
---
 drivers/staging/mars/lib/lib_rank.c |  87 +++++++++++++++++++++++
 include/linux/brick/lib_rank.h      | 135 ++++++++++++++++++++++++++++++++++++
 2 files changed, 222 insertions(+)
 create mode 100644 drivers/staging/mars/lib/lib_rank.c
 create mode 100644 include/linux/brick/lib_rank.h
drivers/staging/mars/lib/lib_rank.c new file +87 -0
new file mode 100644
index 0000000..3bdd1d5
@@ -0,0 +1,87 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+/*  (c) 2012 Thomas Schoebel-Theuer */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+#include <linux/brick/lib_rank.h>
+
+void ranking_compute(struct rank_data *rkd, const struct rank_info rki[], int x)
+{
+	int points = 0;
+	int i;
+
+	for (i = 0; ; i++) {
+		int x0;
+		int x1;
+		int y0;
+		int y1;
+
+		x0 = rki[i].rki_x;
+		if (x < x0)
+			break;
+
+		x1 = rki[i+1].rki_x;
+
+		if (unlikely(x1 == RKI_DUMMY)) {
+			points = rki[i].rki_y;
+			break;
+		}
+
+		if (x > x1)
+			continue;
+
+		y0 = rki[i].rki_y;
+		y1 = rki[i+1].rki_y;
+
+		/*  linear interpolation */
+		points = ((long long)(x - x0) * (long long)(y1 - y0)) / (x1 - x0) + y0;
+		break;
+	}
+	rkd->rkd_tmp += points;
+}
+
+int ranking_select(struct rank_data rkd[], int rkd_count)
+{
+	int res = -1;
+	long long max = LLONG_MIN / 2;
+	int i;
+
+	for (i = 0; i < rkd_count; i++) {
+		struct rank_data *tmp = &rkd[i];
+		long long rest = tmp->rkd_current_points;
+
+		if (rest <= 0)
+			continue;
+		/* rest -= tmp->rkd_got; */
+		if (rest > max) {
+			max = rest;
+			res = i;
+		}
+	}
+	/* Prevent underflow in the long term
+	 * and reset the "clocks" after each round of
+	 * weighted round-robin selection.
+	 */
+	if (max < 0 && res >= 0) {
+		for (i = 0; i < rkd_count; i++)
+			rkd[i].rkd_got += max;
+	}
+	return res;
+}
include/linux/brick/lib_rank.h new file +135 -0

135 lines not on this page · raw message

Signature · 1 line
2.6.4
#10 lore raw

RFC 10/31 mars: add new module lib_limiter

In reply to Thomas Schoebel-Theuer
---
 drivers/staging/mars/lib/lib_limiter.c | 129 +++++++++++++++++++++++++++++++++
 include/linux/brick/lib_limiter.h      |  49 +++++++++++++
 2 files changed, 178 insertions(+)
 create mode 100644 drivers/staging/mars/lib/lib_limiter.c
 create mode 100644 include/linux/brick/lib_limiter.h
drivers/staging/mars/lib/lib_limiter.c new file +129 -0
new file mode 100644
index 0000000..12ac8de
@@ -0,0 +1,129 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/brick/lib_limiter.h>
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/jiffies.h>
+#include <linux/sched.h>
+
+#define LIMITER_TIME_RESOLUTION		NSEC_PER_SEC
+
+int rate_limit(struct rate_limiter *lim, int amount)
+{
+	int delay = 0;
+	long long now;
+
+	now = cpu_clock(raw_smp_processor_id());
+
+	/* Compute the maximum delay along the path
+	 * down to the root of the hierarchy tree.
+	 */
+	while (lim != NULL) {
+		long long window = now - lim->lim_stamp;
+
+		/* Sometimes, raw CPU clocks may do weired things...
+		 * Smaller windows in the denominator than 1s could fake unrealistic rates.
+		 */
+		if (unlikely(lim->lim_min_window <= 0))
+			lim->lim_min_window = 1000;
+		if (unlikely(lim->lim_max_window <= lim->lim_min_window))
+			lim->lim_max_window = lim->lim_min_window + 8000;
+		if (unlikely(window < (long long)lim->lim_min_window * (LIMITER_TIME_RESOLUTION / 1000)))
+			window = (long long)lim->lim_min_window * (LIMITER_TIME_RESOLUTION / 1000);
+
+		/* Update total statistics.
+		 * They will intentionally wrap around.
+		 * Userspace must take care of that.
+		 */
+		lim->lim_total_ops++;
+		lim->lim_total_sum += amount;
+
+		/* Only use incremental accumulation at repeated calls, but
+		 * never after longer pauses.
+		 */
+		if (likely(lim->lim_stamp &&
+			   window < (long long)lim->lim_max_window * (LIMITER_TIME_RESOLUTION / 1000))) {
+			long long rate_raw;
+			int rate;
+
+			/* Races are possible, but taken into account.
+			 * There is no real harm from rarely lost updates.
+			 */
+			if (likely(amount > 0)) {
+				lim->lim_accu += amount;
+				lim->lim_cumul += amount;
+				lim->lim_count++;
+			}
+
+			rate_raw = lim->lim_accu * LIMITER_TIME_RESOLUTION / window;
+			rate = rate_raw;
+			if (unlikely(rate_raw > INT_MAX))
+				rate = INT_MAX;
+			lim->lim_rate = rate;
+
+			/*  limit exceeded? */
+			if (lim->lim_max_rate > 0 && rate > lim->lim_max_rate) {
+				int this_delay = (window * rate / lim->lim_max_rate - window) / (LIMITER_TIME_RESOLUTION / 1000);
+
+				/*  compute maximum */
+				if (this_delay > delay && this_delay > 0)
+					delay = this_delay;
+			}
+
+			/* Try to keep the next window below min_window
+			 */
+			window -= lim->lim_min_window * (LIMITER_TIME_RESOLUTION / 1000);
+			if (window > 0) {
+				long long used_up = (long long)lim->lim_rate * window / LIMITER_TIME_RESOLUTION;
+
+				if (used_up > 0) {
+					lim->lim_stamp += window;
+					lim->lim_accu -= used_up;
+					if (unlikely(lim->lim_accu < 0))
+						lim->lim_accu = 0;
+				}
+			}
+		} else { /*  reset, start over with new measurement cycle */
+			if (unlikely(amount < 0))
+				amount = 0;
+			lim->lim_accu = amount;
+			lim->lim_stamp = now - lim->lim_min_window * (LIMITER_TIME_RESOLUTION / 1000);
+			lim->lim_rate = 0;
+		}
+		lim = lim->lim_father;
+	}
+	return delay;
+}
+
+void rate_limit_sleep(struct rate_limiter *lim, int amount)
+{
+	int sleep = rate_limit(lim, amount);
+
+	if (sleep > 0) {
+		unsigned long timeout;
+
+		if (unlikely(lim->lim_max_delay <= 0))
+			lim->lim_max_delay = 1000;
+		if (sleep > lim->lim_max_delay)
+			sleep = lim->lim_max_delay;
+		timeout = msecs_to_jiffies(sleep);
+		while ((long)timeout > 0)
+			timeout = schedule_timeout_uninterruptible(timeout);
+	}
+}
include/linux/brick/lib_limiter.h new file +49 -0

49 lines not on this page · raw message

Signature · 1 line
2.6.4
#11 lore raw

RFC 11/31 mars: add new module lib_timing

In reply to Thomas Schoebel-Theuer
---
 drivers/staging/mars/lib/lib_timing.c |  71 +++++++++++++
 include/linux/brick/lib_timing.h      | 181 ++++++++++++++++++++++++++++++++++
 2 files changed, 252 insertions(+)
 create mode 100644 drivers/staging/mars/lib/lib_timing.c
 create mode 100644 include/linux/brick/lib_timing.h
drivers/staging/mars/lib/lib_timing.c new file +71 -0
new file mode 100644
index 0000000..7421dc4
@@ -0,0 +1,71 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/brick/lib_timing.h>
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+#ifdef CONFIG_DEBUG_KERNEL
+
+int report_timing(struct timing_stats *tim, char *str, int maxlen)
+{
+	int len = 0;
+	int time = 1;
+	int resol = 1;
+
+	static const char * const units[] = {
+		"us",
+		"ms",
+		"s",
+		"ERROR"
+	};
+	const char *unit = units[0];
+	int unit_index = 0;
+	int i;
+
+	for (i = 0; i < TIMING_MAX; i++) {
+		int this_len = scnprintf(str,
+
+			maxlen,
+			"<%d%s = %d (%lld) ",
+			resol,
+			unit,
+			tim->tim_count[i],
+			(long long)tim->tim_count[i] * time);
+		str += this_len;
+		len += this_len;
+		maxlen -= this_len;
+		if (maxlen <= 1)
+			break;
+		resol <<= 1;
+		time <<= 1;
+		if (resol >= 1000) {
+			resol = 1;
+			unit = units[++unit_index];
+		}
+	}
+	return len;
+}
+
+#endif /*  CONFIG_DEBUG_KERNEL */
+
+struct threshold global_io_threshold = {
+	.thr_limit = 30 * 1000000, /*  30 seconds */
+	.thr_factor = 100,
+	.thr_plus = 0,
+};
include/linux/brick/lib_timing.h new file +181 -0

181 lines not on this page · raw message

Signature · 1 line
2.6.4
#12 lore raw

RFC 12/31 mars: add new module vfs_compat

In reply to Thomas Schoebel-Theuer
---
 include/linux/brick/vfs_compat.h | 45 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)
 create mode 100644 include/linux/brick/vfs_compat.h
include/linux/brick/vfs_compat.h new file +45 -0
new file mode 100644
index 0000000..a119581
@@ -0,0 +1,45 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef _MARS_COMPAT
+#define _MARS_COMPAT
+
+/* TRANSITIONAL compatibility to BOTH the old prepatch
+ * and the new wrappers around vfs_*().
+ */
+#ifndef MARS_MAJOR
+#define __USE_COMPAT
+#endif
+
+#ifdef __USE_COMPAT
+
+int _compat_symlink(const char __user *oldname,
+		    const char __user *newname,
+		    struct timespec *mtime);
+
+int _compat_mkdir(const char __user *pathname,
+		  int mode);
+
+int _compat_rename(const char __user *oldname,
+		   const char __user *newname);
+
+int _compat_unlink(const char __user *pathname);
+
+#else
+#include <linux/syscalls.h>
+#endif
+#endif
Signature · 1 line
2.6.4
#13 lore raw

RFC 13/31 mars: add new module xio

In reply to Thomas Schoebel-Theuer
---
 drivers/staging/mars/xio_bricks/xio.c | 161 +++++++++++++++++
 include/linux/xio/xio.h               | 313 ++++++++++++++++++++++++++++++++++
 2 files changed, 474 insertions(+)
 create mode 100644 drivers/staging/mars/xio_bricks/xio.c
 create mode 100644 include/linux/xio/xio.h
drivers/staging/mars/xio_bricks/xio.c new file +161 -0
new file mode 100644
index 0000000..94aeb60
@@ -0,0 +1,161 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/string.h>
+#include <linux/uaccess.h>
+#include <linux/file.h>
+#include <linux/fs.h>
+
+#include <linux/xio/xio.h>
+
+/************************************************************/
+
+/*  infrastructure */
+
+struct banning xio_global_ban = {};
+atomic_t xio_global_io_flying = ATOMIC_INIT(0);
+
+/************************************************************/
+
+/*  object stuff */
+
+const struct generic_object_type aio_type = {
+	.object_type_name = "aio",
+	.default_size = sizeof(struct aio_object),
+	.object_type_nr = OBJ_TYPE_AIO,
+};
+
+/************************************************************/
+
+/*  brick stuff */
+
+/*******************************************************************/
+
+/*  meta descriptions */
+
+const struct meta xio_info_meta[] = {
+	META_INI(current_size,	  struct xio_info, FIELD_INT),
+	META_INI(tf_align,	  struct xio_info, FIELD_INT),
+	META_INI(tf_min_size,	  struct xio_info, FIELD_INT),
+	{}
+};
+
+const struct meta xio_aio_user_meta[] = {
+	META_INI(_object_cb.cb_error, struct aio_object, FIELD_INT),
+	META_INI(io_pos,	   struct aio_object, FIELD_INT),
+	META_INI(io_len,	   struct aio_object, FIELD_INT),
+	META_INI(io_may_write,	  struct aio_object, FIELD_INT),
+	META_INI(io_prio,	   struct aio_object, FIELD_INT),
+	META_INI(io_cs_mode,	   struct aio_object, FIELD_INT),
+	META_INI(io_timeout,	   struct aio_object, FIELD_INT),
+	META_INI(io_total_size,   struct aio_object, FIELD_INT),
+	META_INI(io_checksum,	   struct aio_object, FIELD_RAW),
+	META_INI(io_flags,	   struct aio_object, FIELD_INT),
+	META_INI(io_rw,    struct aio_object, FIELD_INT),
+	META_INI(io_id,    struct aio_object, FIELD_INT),
+	META_INI(io_skip_sync,	  struct aio_object, FIELD_INT),
+	{}
+};
+
+const struct meta xio_timespec_meta[] = {
+	META_INI_TRANSFER(tv_sec,  struct timespec, FIELD_UINT, 8),
+	META_INI_TRANSFER(tv_nsec, struct timespec, FIELD_UINT, 4),
+	{}
+};
+
+/************************************************************/
+
+/*  crypto stuff */
+
+#include <linux/scatterlist.h>
+#include <linux/crypto.h>
+
+static struct crypto_hash *xio_tfm;
+static struct semaphore tfm_sem;
+int xio_digest_size;
+
+void xio_digest(unsigned char *digest, void *data, int len)
+{
+	struct hash_desc desc = {
+		.tfm = xio_tfm,
+		.flags = 0,
+	};
+	struct scatterlist sg;
+
+	memset(digest, 0, xio_digest_size);
+
+	/*  TODO: use per-thread instance, omit locking */
+	down(&tfm_sem);
+
+	crypto_hash_init(&desc);
+	sg_init_table(&sg, 1);
+	sg_set_buf(&sg, data, len);
+	crypto_hash_update(&desc, &sg, sg.length);
+	crypto_hash_final(&desc, digest);
+	up(&tfm_sem);
+}
+
+void aio_checksum(struct aio_object *aio)
+{
+	unsigned char checksum[xio_digest_size];
+	int len;
+
+	if (aio->io_cs_mode <= 0 || !aio->io_data)
+		goto out_return;
+	xio_digest(checksum, aio->io_data, aio->io_len);
+
+	len = sizeof(aio->io_checksum);
+	if (len > xio_digest_size)
+		len = xio_digest_size;
+	memcpy(&aio->io_checksum, checksum, len);
+out_return:;
+}
+
+/*******************************************************************/
+
+/*  init stuff */
+
+int __init init_xio(void)
+{
+	XIO_INF("init_xio()\n");
+
+	sema_init(&tfm_sem, 1);
+
+	xio_tfm = crypto_alloc_hash("md5", 0, CRYPTO_ALG_ASYNC);
+	if (!xio_tfm) {
+		XIO_ERR("cannot alloc crypto hash\n");
+		return -ENOMEM;
+	}
+	if (IS_ERR(xio_tfm)) {
+		XIO_ERR("alloc crypto hash failed, status = %d\n", (int)PTR_ERR(xio_tfm));
+		return PTR_ERR(xio_tfm);
+	}
+	xio_digest_size = crypto_hash_digestsize(xio_tfm);
+	XIO_INF("digest_size = %d\n", xio_digest_size);
+
+	return 0;
+}
+
+void exit_xio(void)
+{
+	XIO_INF("exit_xio()\n");
+
+	if (xio_tfm)
+		crypto_free_hash(xio_tfm);
+}
include/linux/xio/xio.h new file +313 -0

313 lines not on this page · raw message

Signature · 1 line
2.6.4
#14 lore raw

RFC 14/31 mars: add new module xio_net

In reply to Thomas Schoebel-Theuer
---
 drivers/staging/mars/xio_bricks/xio_net.c | 1830 +++++++++++++++++++++++++++++
 include/linux/xio/xio_net.h               |  171 +++
 2 files changed, 2001 insertions(+)
 create mode 100644 drivers/staging/mars/xio_bricks/xio_net.c
 create mode 100644 include/linux/xio/xio_net.h
drivers/staging/mars/xio_bricks/xio_net.c new file +1830 -0
new file mode 100644
index 0000000..dcc443c
@@ -0,0 +1,1830 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/string.h>
+#include <linux/moduleparam.h>
+#include <linux/lzo.h>
+#include <linux/utsname.h>
+
+#include <linux/xio/xio.h>
+#include <linux/xio/xio_net.h>
+
+/******************************************************************/
+
+/*  provisionary version detection */
+
+#ifndef TCP_MAX_REORDERING
+#define __HAS_IOV_ITER
+#endif
+
+#ifdef sk_net_refcnt
+/* see eeb1bd5c40edb0e2fd925c8535e2fdebdbc5cef2 */
+#define __HAS_STRUCT_NET
+#endif
+
+/******************************************************************/
+
+#define USE_BUFFERING
+
+#define SEND_PROTO_VERSION		2
+
+enum COMPRESS_TYPES {
+	COMPRESS_NONE = 0,
+	COMPRESS_LZO = 1,
+	/* insert further methods here */
+};
+
+int xio_net_compress_data;
+
+const u16 net_global_flags = 0
+#ifdef __HAVE_LZO
+	| COMPRESS_LZO
+#endif
+	;
+
+/******************************************************************/
+
+/* Internal data structures for low-level transfer of C structures
+ * described by struct meta.
+ * Only these low-level fields need to have a fixed size like s64.
+ * The size and bytesex of the higher-level C structures is converted
+ * automatically; therefore classical "int" or "long long" etc is viable.
+ */
+
+#define MAX_FIELD_LEN			(32 + 16)
+
+/* Please keep this at a size of 64 bytes by
+ * reuse of *spare* fields.
+ */
+struct xio_desc_cache {
+	u8    cache_sender_proto;
+	u8    cache_recver_proto;
+	s8    cache_is_bigendian;
+	u8    cache_spare0;
+	s16   cache_items;
+	u16   cache_spare1;
+	u32   cache_spare2;
+	u32   cache_spare3;
+	u64   cache_spare4[4];
+	u64   cache_sender_cookie;
+	u64   cache_recver_cookie;
+};
+
+/* Please keep this also at a size of 64 bytes by
+ * reuse of *spare* fields.
+ */
+struct xio_desc_item {
+	s8    field_type;
+	s8    field_spare0;
+	s16   field_data_size;
+	s16   field_sender_size;
+	s16   field_sender_offset;
+	s16   field_recver_size;
+	s16   field_recver_offset;
+	s32   field_spare;
+	char  field_name[MAX_FIELD_LEN];
+};
+
+/* This must not be mirror symmetric between big and little endian
+ */
+#define XIO_DESC_MAGIC			0x73D0A2EC6148F48Ell
+
+struct xio_desc_header {
+	u64 h_magic;
+	u64 h_cookie;
+	s16 h_meta_len;
+	s16 h_index;
+	u32 h_spare1;
+	u64 h_spare2;
+};
+
+#define MAX_INT_TRANSFER		16
+
+/******************************************************************/
+
+/* Bytesex conversion / sign extension
+ */
+
+#ifdef __LITTLE_ENDIAN
+static const bool myself_is_bigendian;
+
+#endif
+#ifdef __BIG_ENDIAN
+static const bool myself_is_bigendian = true;
+
+#endif
+
+static inline
+void swap_bytes(void *data, int len)
+{
+	char *a = data;
+	char *b = data + len - 1;
+
+	while (a < b) {
+		char tmp = *a;
+
+		*a = *b;
+		*b = tmp;
+		a++;
+		b--;
+	}
+}
+
+#define SWAP_FIELD(x) swap_bytes(&(x), sizeof(x))
+
+static inline
+void swap_mc(struct xio_desc_cache *mc, int len)
+{
+	struct xio_desc_item *mi;
+
+	SWAP_FIELD(mc->cache_sender_cookie);
+	SWAP_FIELD(mc->cache_recver_cookie);
+	SWAP_FIELD(mc->cache_items);
+
+	len -= sizeof(*mc);
+
+	for (mi = (void *)(mc + 1); len > 0; mi++, len -= sizeof(*mi)) {
+		SWAP_FIELD(mi->field_data_size);
+		SWAP_FIELD(mi->field_sender_size);
+		SWAP_FIELD(mi->field_sender_offset);
+		SWAP_FIELD(mi->field_recver_size);
+		SWAP_FIELD(mi->field_recver_offset);
+	}
+}
+
+static inline
+char get_sign(const void *data, int len, bool is_bigendian, bool is_signed)
+{
+	if (is_signed) {
+		char x = is_bigendian ?
+			((const char *)data)[0] :
+			((const char *)data)[len - 1];
+		if (x < 0)
+			return -1;
+	}
+	return 0;
+}
+
+/******************************************************************/
+
+/* Low-level network traffic
+ */
+
+int xio_net_default_port = CONFIG_MARS_DEFAULT_PORT;
+
+module_param_named(xio_port, xio_net_default_port, int, 0);
+
+int xio_net_bind_before_listen = 1;
+
+module_param_named(xio_net_bind_before_listen, xio_net_bind_before_listen, int, 0);
+
+int xio_net_bind_before_connect = 1;
+
+/* TODO: make all the socket options configurable.
+ * TODO: implement signal handling.
+ * TODO: add authentication.
+ * TODO: add encryption.
+ */
+
+struct xio_tcp_params default_tcp_params = {
+	.ip_tos = IPTOS_LOWDELAY,
+	.tcp_window_size = 8 * 1024 * 1024, /*	for long distance replications */
+	.tcp_nodelay = 0,
+	.tcp_timeout = 2,
+	.tcp_keepcnt = 3,
+	.tcp_keepintvl = 3, /*	keepalive ping time */
+	.tcp_keepidle = 4,
+};
+
+static char *id;
+
+char *my_id(void)
+{
+	struct new_utsname *u;
+
+	if (!id) {
+		/* down_read(&uts_sem); // FIXME: this is currenty not EXPORTed from the kernel! */
+		u = utsname();
+		if (u)
+			id = brick_strdup(u->nodename);
+		/* up_read(&uts_sem); */
+	}
+	return id;
+}
+
+static
+void __setsockopt(struct socket *sock, int level, int optname, char *optval, int optsize)
+{
+	int status = kernel_setsockopt(sock, level, optname, optval, optsize);
+
+	if (status < 0) {
+		XIO_WRN("cannot set %d socket option %d to value %d, status = %d\n",
+			 level, optname, *(int *)optval, status);
+	}
+}
+
+#define _setsockopt(sock, level, optname, val) __setsockopt(sock, level, optname, (char *)&(val), sizeof(val))
+
+int xio_create_sockaddr(struct sockaddr_storage *addr, const char *spec)
+{
+	struct sockaddr_in *sockaddr = (void *)addr;
+	const char *new_spec;
+	const char *tmp_spec;
+	int status = 0;
+
+	memset(addr, 0, sizeof(*addr));
+	sockaddr->sin_family = AF_INET;
+	sockaddr->sin_port = htons(xio_net_default_port);
+
+	/* Try to translate hostnames to IPs if possible.
+	 */
+	if (xio_translate_hostname)
+		new_spec = xio_translate_hostname(spec);
+	else
+		new_spec = brick_strdup(spec);
+	tmp_spec = new_spec;
+
+	/* This is PROVISIONARY!
+	 * TODO: add IPV6 syntax and many more features :)
+	 */
+	if (!*tmp_spec)
+		goto done;
+	if (*tmp_spec != ':') {
+		unsigned char u0 = 0, u1 = 0, u2 = 0, u3 = 0;
+
+		status = sscanf(tmp_spec, "%hhu.%hhu.%hhu.%hhu", &u0, &u1, &u2, &u3);
+		if (status != 4) {
+			XIO_ERR("invalid sockaddr IP syntax '%s', status = %d\n", tmp_spec, status);
+			status = -EINVAL;
+			goto done;
+		}
+		XIO_DBG("decoded IP = %u.%u.%u.%u\n", u0, u1, u2, u3);
+		sockaddr->sin_addr.s_addr = (__be32)u0 | (__be32)u1 << 8 | (__be32)u2 << 16 | (__be32)u3 << 24;
+	}
+	/*  deocde port number (when present) */
+	tmp_spec = spec;
+	while (*tmp_spec && *tmp_spec++ != ':')
+		/*empty*/;
+	if (*tmp_spec) {
+		int port = 0;
+
+		status = kstrtoint(tmp_spec, 10, &port);
+		if (unlikely(status)) {
+			XIO_ERR("invalid sockaddr PORT syntax '%s', status = %d\n", tmp_spec, status);
+			status = -EINVAL;
+			goto done;
+		}
+		XIO_DBG("decoded PORT = %d\n", port);
+		sockaddr->sin_port = htons(port);
+	}
+	status = 0;
+done:
+	brick_string_free(new_spec);
+	return status;
+}
+
+static int current_debug_nr; /*  no locking, just for debugging */
+
+static
+void _set_socketopts(struct socket *sock)
+{
+	struct timeval t = {
+		.tv_sec = default_tcp_params.tcp_timeout,
+	};
+	int x_true = 1;
+
+	/* TODO: improve this by a table-driven approach
+	 */
+	sock->sk->sk_rcvtimeo = sock->sk->sk_sndtimeo = default_tcp_params.tcp_timeout * HZ;
+	sock->sk->sk_reuse = 1;
+	_setsockopt(sock, SOL_SOCKET, SO_SNDBUFFORCE, default_tcp_params.tcp_window_size);
+	_setsockopt(sock, SOL_SOCKET, SO_RCVBUFFORCE, default_tcp_params.tcp_window_size);
+	_setsockopt(sock, SOL_IP, SO_PRIORITY, default_tcp_params.ip_tos);
+	_setsockopt(sock, IPPROTO_TCP, TCP_NODELAY, default_tcp_params.tcp_nodelay);
+	_setsockopt(sock, SOL_SOCKET, SO_KEEPALIVE, x_true);
+	_setsockopt(sock, IPPROTO_TCP, TCP_KEEPCNT, default_tcp_params.tcp_keepcnt);
+	_setsockopt(sock, IPPROTO_TCP, TCP_KEEPINTVL, default_tcp_params.tcp_keepintvl);
+	_setsockopt(sock, IPPROTO_TCP, TCP_KEEPIDLE, default_tcp_params.tcp_keepidle);
+	_setsockopt(sock, SOL_SOCKET, SO_SNDTIMEO, t);
+	_setsockopt(sock, SOL_SOCKET, SO_RCVTIMEO, t);
+
+	if (sock->file) { /*  switch back to blocking mode */
+		sock->file->f_flags &= ~O_NONBLOCK;
+	}
+}
+
+static int _xio_send_raw(struct xio_socket *msock, const void *buf, int len, int flags);
+static int _xio_recv_raw(struct xio_socket *msock, void *buf, int minlen, int maxlen, int flags);
+
+static
+void xio_proto_check(struct xio_socket *msock)
+{
+	u8 service_version = 0;
+	u16 service_flags = 0;
+	int status;
+
+	status = _xio_recv_raw(msock, &service_version, 1, 1, 0);
+	if (unlikely(status < 0)) {
+		XIO_DBG("#%d protocol exchange failed at receiving, status = %d\n",
+			 msock->s_debug_nr,
+			 status);
+		goto out_return;
+	}
+
+	/*  take the the minimum of both protocol versions */
+	if (service_version > msock->s_send_proto)
+		service_version = msock->s_send_proto;
+	msock->s_send_proto = service_version;
+
+	status = _xio_recv_raw(msock, &service_flags, 2, 2, 0);
+	if (unlikely(status < 0)) {
+		XIO_DBG("#%d protocol exchange failed at receiving, status = %d\n",
+			 msock->s_debug_nr,
+			 status);
+		goto out_return;
+	}
+
+	msock->s_recv_flags = service_flags;
+out_return:;
+}
+
+static
+int xio_proto_exchange(struct xio_socket *msock, const char *msg)
+{
+	int status;
+
+	msock->s_send_proto = SEND_PROTO_VERSION;
+	status = xio_send_raw(msock, &msock->s_send_proto, 1, false);
+	if (unlikely(status < 0)) {
+		XIO_DBG("#%d protocol exchange on %s failed at sending, status = %d\n",
+			 msock->s_debug_nr,
+			 msg,
+			 status);
+		goto done;
+	}
+
+	msock->s_send_flags = net_global_flags;
+	status = xio_send_raw(msock, &msock->s_send_flags, 2, false);
+	if (unlikely(status < 0)) {
+		XIO_DBG("#%d flags exchange on %s failed at sending, status = %d\n",
+			 msock->s_debug_nr,
+			 msg,
+			 status);
+		goto done;
+	}
+
+done:
+	return status;
+}
+
+int xio_create_socket(struct xio_socket *msock, struct sockaddr_storage *src_addr, struct sockaddr_storage *dst_addr)
+{
+	struct socket *sock;
+	struct sockaddr *src_sockaddr = (void *)src_addr;
+	struct sockaddr *dst_sockaddr = (void *)dst_addr;
+	int status = -EEXIST;
+
+	if (unlikely(atomic_read(&msock->s_count))) {
+		XIO_ERR("#%d socket already in use\n", msock->s_debug_nr);
+		goto final;
+	}
+	if (unlikely(msock->s_socket)) {
+		XIO_ERR("#%d socket already open\n", msock->s_debug_nr);
+		goto final;
+	}
+	atomic_set(&msock->s_count, 1);
+
+#ifdef __HAS_STRUCT_NET
+	status = sock_create_kern(&init_net, AF_INET, SOCK_STREAM, IPPROTO_TCP, &msock->s_socket);
+#else
+	status = sock_create_kern(AF_INET, SOCK_STREAM, IPPROTO_TCP, &msock->s_socket);
+#endif
+	if (unlikely(status < 0 || !msock->s_socket)) {
+		msock->s_socket = NULL;
+		XIO_WRN("cannot create socket, status = %d\n", status);
+		goto final;
+	}
+	msock->s_debug_nr = ++current_debug_nr;
+	sock = msock->s_socket;
+	CHECK_PTR(sock, done);
+	msock->s_alive = true;
+
+	_set_socketopts(sock);
+
+	if (!dst_sockaddr) { /* we are server */
+		struct sockaddr_in bind_addr;
+
+		if (unlikely(!src_sockaddr)) {
+			XIO_ERR("no srcaddr given for bind()\n");
+			status = -EINVAL;
+			goto done;
+		}
+
+		memcpy(&bind_addr, src_sockaddr, sizeof(bind_addr));
+		if (!xio_net_bind_before_listen)
+			memset(&bind_addr.sin_addr, 0, sizeof(bind_addr.sin_addr));
+
+		status = kernel_bind(sock, (struct sockaddr *)&bind_addr, sizeof(bind_addr));
+		if (unlikely(status < 0)) {
+			XIO_WRN("#%d bind failed, status = %d\n", msock->s_debug_nr, status);
+			goto done;
+		}
+		status = kernel_listen(sock, 16);
+		if (status < 0)
+			XIO_WRN("#%d listen failed, status = %d\n", msock->s_debug_nr, status);
+	} else {
+		/* When both src and dst are given, explicitly bind local address.
+		 * Needed for multihomed hosts.
+		 */
+		if (src_sockaddr && xio_net_bind_before_connect) {
+			struct sockaddr_in bind_addr;
+
+			memcpy(&bind_addr, src_sockaddr, sizeof(bind_addr));
+			bind_addr.sin_port = 0;
+
+			status = kernel_bind(sock, (struct sockaddr *)&bind_addr, sizeof(struct sockaddr));
+			if (unlikely(status < 0)) {
+				XIO_WRN("#%d bind before connect failed (ignored), status = %d\n",
+					 msock->s_debug_nr, status);
+			}
+		}
+
+		status = kernel_connect(sock, dst_sockaddr, sizeof(*dst_sockaddr), 0);
+		/* Treat non-blocking connects as successful.
+		 * Any potential errors will show up later during traffic.
+		 */
+		if (status == -EINPROGRESS) {
+			XIO_DBG("#%d connect in progress\n", msock->s_debug_nr);
+			status = 0;
+		}
+		if (unlikely(status < 0)) {
+			XIO_DBG("#%d connect failed, status = %d\n", msock->s_debug_nr, status);
+			goto done;
+		}
+		status = xio_proto_exchange(msock, "connect");
+	}
+
+done:
+	if (status < 0)
+		xio_put_socket(msock);
+	else
+		XIO_DBG("successfully created socket #%d\n", msock->s_debug_nr);
+final:
+	return status;
+}
+
+int xio_accept_socket(struct xio_socket *new_msock, struct xio_socket *old_msock)
+{
+	int status = -ENOENT;
+	struct socket *new_socket = NULL;
+	bool ok;
+
+	ok = xio_get_socket(old_msock);
+	if (likely(ok)) {
+		struct socket *sock = old_msock->s_socket;
+
+		if (unlikely(!sock))
+			goto err;
+
+		status = kernel_accept(sock, &new_socket, O_NONBLOCK);
+		if (unlikely(status < 0))
+			goto err;
+		if (unlikely(!new_socket)) {
+			status = -EBADF;
+			goto err;
+		}
+
+		_set_socketopts(new_socket);
+
+		memset(new_msock, 0, sizeof(struct xio_socket));
+		new_msock->s_socket = new_socket;
+		atomic_set(&new_msock->s_count, 1);
+		new_msock->s_alive = true;
+		new_msock->s_debug_nr = ++current_debug_nr;
+		XIO_DBG("#%d successfully accepted socket #%d\n", old_msock->s_debug_nr, new_msock->s_debug_nr);
+
+		status = xio_proto_exchange(new_msock, "accept");
+err:
+		xio_put_socket(old_msock);
+	}
+	return status;
+}
+
+bool xio_get_socket(struct xio_socket *msock)
+{
+	if (unlikely(atomic_read(&msock->s_count) <= 0)) {
+		XIO_ERR("#%d bad nesting on msock = %p\n", msock->s_debug_nr, msock);
+		return false;
+	}
+
+	atomic_inc(&msock->s_count);
+
+	if (unlikely(!msock->s_socket || !msock->s_alive)) {
+		xio_put_socket(msock);
+		return false;
+	}
+	return true;
+}
+
+void xio_put_socket(struct xio_socket *msock)
+{
+	if (unlikely(atomic_read(&msock->s_count) <= 0)) {
+		XIO_ERR("#%d bad nesting on msock = %p sock = %p\n", msock->s_debug_nr, msock, msock->s_socket);
+	} else if (atomic_dec_and_test(&msock->s_count)) {
+		struct socket *sock = msock->s_socket;
+		int i;
+
+		XIO_DBG("#%d closing socket %p\n", msock->s_debug_nr, sock);
+		if (likely(sock && cmpxchg(&msock->s_alive, true, false)))
+			kernel_sock_shutdown(sock, SHUT_RDWR);
+		if (likely(sock && !msock->s_alive)) {
+			XIO_DBG("#%d releasing socket %p\n", msock->s_debug_nr, sock);
+			sock_release(sock);
+		}
+		for (i = 0; i < MAX_DESC_CACHE; i++) {
+			if (msock->s_desc_send[i])
+				brick_block_free(msock->s_desc_send[i], PAGE_SIZE);
+			if (msock->s_desc_recv[i])
+				brick_block_free(msock->s_desc_recv[i], PAGE_SIZE);
+		}
+		brick_block_free(msock->s_buffer, PAGE_SIZE);
+		memset(msock, 0, sizeof(struct xio_socket));
+	}
+}
+
+void xio_shutdown_socket(struct xio_socket *msock)
+{
+	if (msock->s_socket) {
+		bool ok = xio_get_socket(msock);
+
+		if (likely(ok)) {
+			struct socket *sock = msock->s_socket;
+
+			if (likely(sock && cmpxchg(&msock->s_alive, true, false))) {
+				XIO_DBG("#%d shutdown socket %p\n", msock->s_debug_nr, sock);
+				kernel_sock_shutdown(sock, SHUT_RDWR);
+			}
+			xio_put_socket(msock);
+		}
+	}
+}
+
+bool xio_socket_is_alive(struct xio_socket *msock)
+{
+	bool res = false;
+
+	if (!msock->s_socket || !msock->s_alive)
+		goto done;
+	if (unlikely(atomic_read(&msock->s_count) <= 0)) {
+		XIO_ERR("#%d bad nesting on msock = %p sock = %p\n", msock->s_debug_nr, msock, msock->s_socket);
+		goto done;
+	}
+	res = true;
+done:
+	return res;
+}
+
+long xio_socket_send_space_available(struct xio_socket *msock)
+{
+	struct socket *raw_sock = msock->s_socket;
+	long res = 0;
+
+	if (!msock->s_alive || !raw_sock || !raw_sock->sk)
+		goto done;
+	if (unlikely(atomic_read(&msock->s_count) <= 0)) {
+		XIO_ERR("#%d bad nesting on msock = %p sock = %p\n", msock->s_debug_nr, msock, msock->s_socket);
+		goto done;
+	}
+
+	res = raw_sock->sk->sk_sndbuf - raw_sock->sk->sk_wmem_queued;
+	if (res < 0)
+		res = 0;
+	res += msock->s_pos;
+
+done:
+	return res;
+}
+
+static
+int _xio_send_raw(struct xio_socket *msock, const void *buf, int len, int flags)
+{
+	int sleeptime = 1000 / HZ;
+	int sent = 0;
+	int status = 0;
+
+	msock->s_send_cnt = 0;
+	while (len > 0) {
+		int this_len = len;
+		struct socket *sock = msock->s_socket;
+
+		if (unlikely(!sock || !xio_net_is_alive || brick_thread_should_stop())) {
+			XIO_WRN("interrupting, sent = %d\n", sent);
+			status = -EIDRM;
+			break;
+		}
+
+		{
+			struct kvec iov = {
+				.iov_base = (void *)buf,
+				.iov_len = this_len,
+			};
+			struct msghdr msg = {
+#ifndef __HAS_IOV_ITER
+				.msg_iov = (struct iovec *)&iov,
+#endif
+				.msg_flags = 0 | MSG_NOSIGNAL,
+			};
+			status = kernel_sendmsg(sock, &msg, &iov, 1, this_len);
+		}
+
+		if (status == -EAGAIN) {
+			if (msock->s_send_abort > 0 && ++msock->s_send_cnt > msock->s_send_abort) {
+				XIO_WRN("#%d reached send abort %d\n", msock->s_debug_nr, msock->s_send_abort);
+				status = -EINTR;
+				break;
+			}
+			brick_msleep(sleeptime);
+			/*  linearly increasing backoff */
+			if (sleeptime < 100)
+				sleeptime += 1000 / HZ;
+			continue;
+		}
+		msock->s_send_cnt = 0;
+		if (unlikely(status == -EINTR)) { /*  ignore it */
+			flush_signals(current);
+			brick_msleep(50);
+			continue;
+		}
+		if (unlikely(!status)) {
+			XIO_WRN("#%d EOF from socket upon send_page()\n", msock->s_debug_nr);
+			brick_msleep(50);
+			status = -ECOMM;
+			break;
+		}
+		if (unlikely(status < 0)) {
+			XIO_WRN("#%d bad socket sendmsg, len=%d, this_len=%d, sent=%d, status = %d\n",
+				msock->s_debug_nr,
+				len,
+				this_len,
+				sent,
+				status);
+			break;
+		}
+
+		len -= status;
+		buf += status;
+		sent += status;
+		sleeptime = 1000 / HZ;
+	}
+
+	msock->s_send_bytes += sent;
+	if (status >= 0)
+		status = sent;
+
+	return status;
+}
+
+int xio_send_raw(struct xio_socket *msock, const void *buf, int len, bool cork)
+{
+#ifdef USE_BUFFERING
+	int sent = 0;
+	int rest = len;
+
+#endif
+	int status = -EINVAL;
+
+	if (!xio_get_socket(msock))
+		goto final;
+
+#ifdef USE_BUFFERING
+restart:
+	if (!msock->s_buffer) {
+		msock->s_pos = 0;
+		msock->s_buffer = brick_block_alloc(0, PAGE_SIZE);
+	}
+
+	if (msock->s_pos + rest < PAGE_SIZE) {
+		memcpy(msock->s_buffer + msock->s_pos, buf, rest);
+		msock->s_pos += rest;
+		sent += rest;
+		rest = 0;
+		status = sent;
+		if (cork)
+			goto done;
+	}
+
+	if (msock->s_pos > 0) {
+		status = _xio_send_raw(msock, msock->s_buffer, msock->s_pos, 0);
+		if (status < 0)
+			goto done;
+
+		brick_block_free(msock->s_buffer, PAGE_SIZE);
+		msock->s_buffer = NULL;
+		msock->s_pos = 0;
+	}
+
+	if (rest >= PAGE_SIZE) {
+		status = _xio_send_raw(msock, buf, rest, 0);
+		goto done;
+	} else if (rest > 0) {
+		goto restart;
+	}
+	status = sent;
+
+done:
+#else
+	status = _xio_send_raw(msock, buf, len, 0);
+#endif
+	if (status < 0 && msock->s_shutdown_on_err)
+		xio_shutdown_socket(msock);
+
+	xio_put_socket(msock);
+
+final:
+	return status;
+}
+
+/**
+ * xio_recv_raw() - Get [min, max] number of bytes
+ * @msock:	socket to read from
+ * @buf:	buffer to put the data in
+ * @minlen:	minimum number of bytes to read
+ * @maxlen:	maximum number of bytes to read
+ *
+ * Returns a negative error code or a number between [@minlen, @maxlen].
+ * Short reads are mapped to an error.
+ *
+ * Hint: by setting @minlen to 1, you can read any number up to @maxlen.
+ * However, the most important use case is @minlen == @maxlen.
+ *
+ * Note: buf may be NULL. In this case, the data is simply consumed,
+ * like /dev/null
+ */
+static
+int _xio_recv_raw(struct xio_socket *msock, void *buf, int minlen, int maxlen, int flags)
+{
+	void *dummy = NULL;
+	int sleeptime = 1000 / HZ;
+	int status = -EIDRM;
+	int done = 0;
+
+	if (!buf)
+		buf = dummy = brick_block_alloc(0, maxlen);
+
+	if (!xio_get_socket(msock))
+		goto final;
+
+	if (minlen < maxlen) {
+		struct socket *sock = msock->s_socket;
+
+		if (sock && sock->file) {
+			/* Use nonblocking reads to consume as much data
+			 * as possible
+			 */
+			sock->file->f_flags |= O_NONBLOCK;
+		}
+	}
+
+	msock->s_recv_cnt = 0;
+	while (done < minlen || (!minlen && !done)) {
+		struct kvec iov = {
+			.iov_base = buf + done,
+			.iov_len = maxlen - done,
+		};
+		struct msghdr msg = {
+#ifndef __HAS_IOV_ITER
+			.msg_iovlen = 1,
+			.msg_iov = (struct iovec *)&iov,
+#endif
+			.msg_flags = flags | MSG_NOSIGNAL,
+		};
+		struct socket *sock = msock->s_socket;
+
+		if (unlikely(!sock)) {
+			XIO_WRN("#%d socket has disappeared\n", msock->s_debug_nr);
+			status = -EIDRM;
+			goto err;
+		}
+
+		if (!xio_net_is_alive || brick_thread_should_stop()) {
+			XIO_WRN("#%d interrupting, done = %d\n", msock->s_debug_nr, done);
+			status = -EIDRM;
+			goto err;
+		}
+
+		status = kernel_recvmsg(sock, &msg, &iov, 1, maxlen-done, msg.msg_flags);
+
+		if (!xio_net_is_alive || brick_thread_should_stop()) {
+			XIO_WRN("#%d interrupting, done = %d\n", msock->s_debug_nr, done);
+			status = -EIDRM;
+			goto err;
+		}
+
+		if (status == -EAGAIN) {
+			if (msock->s_recv_abort > 0 && ++msock->s_recv_cnt > msock->s_recv_abort) {
+				XIO_WRN("#%d reached recv abort %d\n", msock->s_debug_nr, msock->s_recv_abort);
+				status = -EINTR;
+				goto err;
+			}
+			brick_msleep(sleeptime);
+			if (minlen <= 0)
+				break;
+			/*  linearly increasing backoff */
+			if (sleeptime < 100)
+				sleeptime += 1000 / HZ;
+			continue;
+		}
+		msock->s_recv_cnt = 0;
+		if (!status) { /*  EOF */
+			XIO_WRN("#%d got EOF from socket (done=%d, req_size=%d)\n",
+				msock->s_debug_nr,
+				done,
+				maxlen - done);
+			status = -EPIPE;
+			goto err;
+		}
+		if (status < 0) {
+			XIO_WRN("#%d bad recvmsg, status = %d\n", msock->s_debug_nr, status);
+			goto err;
+		}
+		done += status;
+		sleeptime = 1000 / HZ;
+	}
+	status = done;
+	msock->s_recv_bytes += done;
+
+err:
+	if (status < 0 && msock->s_shutdown_on_err)
+		xio_shutdown_socket(msock);
+	xio_put_socket(msock);
+final:
+	if (dummy)
+		brick_block_free(dummy, maxlen);
+	return status;
+}
+
+int xio_recv_raw(struct xio_socket *msock, void *buf, int minlen, int maxlen)
+{
+	/* Check the very first received byte for higher-level protocol
+	 * information. This safes one ping-pong cycle at
+	 * xio_proto_exchange() because the sender can immediately
+	 * start sending bulk data without need to wait there.
+	 * This is important for latency, thus we exceptionally break
+	 * the layering hierarchy here. Also, we start sending at
+	 * the lowest possible protocol version and may increase
+	 * the protocol capabilities dynamically at runtime,
+	 * somewhen later. This bears some slight nondeterminism,
+	 * but we take it into account for performance reasons.
+	 */
+	if (unlikely(!msock->s_recv_bytes))
+		xio_proto_check(msock);
+
+	return _xio_recv_raw(msock, buf, minlen, maxlen, 0);
+}
+
+int xio_send_compressed(struct xio_socket *msock, const void *buf, s32 len, int compress, bool cork)
+{
+	void *compr_data = NULL;
+
+	s16 compr_code = 0;
+	int status;
+
+	switch (compress) {
+	case COMPRESS_LZO:
+#ifdef __HAVE_LZO
+		/*  tolerate mixes of different proto versions */
+		if (msock->s_send_proto >= 2 && (msock->s_recv_flags & COMPRESS_LZO)) {
+			size_t compr_len = 0;
+			int lzo_status;
+			void *wrkmem;
+
+			compr_data = brick_mem_alloc(lzo1x_worst_compress(len));
+			wrkmem = brick_mem_alloc(LZO1X_1_MEM_COMPRESS);
+
+			lzo_status = lzo1x_1_compress(buf, len, compr_data, &compr_len, wrkmem);
+
+			brick_mem_free(wrkmem);
+			if (likely(lzo_status == LZO_E_OK && compr_len < len)) {
+				compr_code = COMPRESS_LZO;
+				buf = compr_data;
+				len = compr_len;
+			}
+		}
+#endif
+		break;
+
+		/* implement further methods here */
+
+	default:
+		/* ignore unknown compress codes */
+		break;
+	}
+
+	/*  allow mixing of different proto versions */
+	if (likely(msock->s_send_proto >= 2)) {
+		status = xio_send_raw(msock, &compr_code, sizeof(compr_code), true);
+		if (unlikely(status < 0))
+			goto done;
+		if (compr_code > 0) {
+			status = xio_send_raw(msock, &len, sizeof(len), true);
+			if (unlikely(status < 0))
+				goto done;
+		}
+	}
+
+	status = xio_send_raw(msock, buf, len, cork);
+
+done:
+	brick_mem_free(compr_data);
+	return status;
+}
+
+int xio_recv_compressed(struct xio_socket *msock, void *buf, int minlen, int maxlen)
+{
+	void *compr_data = NULL;
+
+	s16 compr_code = COMPRESS_NONE;
+	int status;
+
+	/*  allow mixing of different proto versions */
+	if (msock->s_send_proto >= 2) {
+		status = xio_recv_raw(msock, &compr_code, sizeof(compr_code), sizeof(compr_code));
+		if (unlikely(status < 0))
+			goto done;
+	}
+
+	switch (compr_code) {
+	case COMPRESS_NONE:
+		status = xio_recv_raw(msock, buf, minlen, maxlen);
+		break;
+
+	case COMPRESS_LZO:
+#ifdef __HAVE_LZO
+		{
+			s32 compr_len = 0;
+			size_t this_len;
+			int lzo_status;
+
+			status = xio_recv_raw(msock, &compr_len, sizeof(compr_len), sizeof(compr_len));
+			if (unlikely(status < 0))
+				goto done;
+			if (unlikely(compr_len <= 0 || compr_len >= maxlen)) {
+				XIO_ERR("bad comp_len = %d, real minlen = %d maxlen = %d\n",
+					 compr_len, minlen, maxlen);
+				status = -EOVERFLOW;
+				goto done;
+			}
+
+			compr_data = brick_mem_alloc(compr_len);
+
+			status = xio_recv_raw(msock, compr_data, compr_len, compr_len);
+			if (unlikely(status < 0))
+				goto done;
+
+			this_len = maxlen;
+			lzo_status = lzo1x_decompress_safe(compr_data, compr_len, buf, &this_len);
+
+			status = this_len;
+			if (unlikely(lzo_status != LZO_E_OK)) {
+				XIO_ERR("bad decompression, lzo_status = %d\n", lzo_status);
+				status = -EBADE;
+				goto done;
+			}
+			if (unlikely(this_len < minlen || this_len > maxlen)) {
+				XIO_WRN("bad decompression length this_len = %ld, minlen = %d maxlen = %d\n",
+					(long)this_len,
+					minlen,
+					maxlen);
+				status = -EBADMSG;
+				goto done;
+			}
+			break;
+		}
+#else
+		XIO_WRN("cannot LZO decompress\n");
+		status = -EBADMSG;
+		break;
+#endif
+
+	/* implement further methods here */
+
+	default:
+		XIO_WRN("got unknown compr_code = %d\n", compr_code);
+		status = -EBADRQC;
+	}
+
+done:
+	brick_mem_free(compr_data);
+	return status;
+}
+
+/*********************************************************************/
+
+/* Mid-level field data exchange
+ */
+
+static
+void dump_meta(const struct meta *meta)
+{
+	int count = 0;
+
+	for (; meta->field_name != NULL; meta++) {
+		XIO_ERR("%2d %4d %4d %4d %p '%s'\n",
+			 meta->field_type,
+			 meta->field_data_size,
+			 meta->field_transfer_size,
+			 meta->field_offset,
+			 meta->field_ref,
+			 meta->field_name);
+		count++;
+	}
+	XIO_ERR("-------- %d fields.\n", count);
+}
+
+static
+int _add_fields(struct xio_desc_item *mi, const struct meta *meta, int offset, const char *prefix, int maxlen)
+{
+	int count = 0;
+
+	for (; meta->field_name != NULL; meta++) {
+		const char *new_prefix;
+		int new_offset;
+		int len;
+
+		short this_size;
+
+		new_prefix = mi->field_name;
+		new_offset = offset + meta->field_offset;
+
+		if (unlikely(maxlen < sizeof(struct xio_desc_item))) {
+			XIO_ERR("desc cache item overflow\n");
+			count = -1;
+			goto done;
+		}
+
+		len = scnprintf(mi->field_name, MAX_FIELD_LEN, "%s.%s", prefix, meta->field_name);
+		if (unlikely(len >= MAX_FIELD_LEN)) {
+			XIO_ERR("field len overflow on '%s.%s'\n", prefix, meta->field_name);
+			count = -1;
+			goto done;
+		}
+		mi->field_type = meta->field_type;
+		this_size = meta->field_data_size;
+		mi->field_data_size = this_size;
+		mi->field_sender_size = this_size;
+		this_size = meta->field_transfer_size;
+		if (this_size > 0)
+			mi->field_sender_size = this_size;
+		mi->field_sender_offset = new_offset;
+		mi->field_recver_offset = -1;
+
+		mi++;
+		maxlen -= sizeof(struct xio_desc_item);
+		count++;
+
+		if (meta->field_type == FIELD_SUB) {
+			int sub_count;
+
+			sub_count = _add_fields(mi, meta->field_ref, new_offset, new_prefix, maxlen);
+			if (sub_count < 0)
+				return sub_count;
+
+			mi += sub_count;
+			count += sub_count;
+			maxlen -= sub_count * sizeof(struct xio_desc_item);
+		}
+	}
+done:
+	return count;
+}
+
+static
+struct xio_desc_cache *make_sender_cache(struct xio_socket *msock, const struct meta *meta, int *cache_index)
+{
+	int orig_len = PAGE_SIZE;
+	int maxlen = orig_len;
+	struct xio_desc_cache *mc;
+	struct xio_desc_item *mi;
+	int i;
+	int status;
+
+	for (i = 0; i < MAX_DESC_CACHE; i++) {
+		mc = msock->s_desc_send[i];
+		if (!mc)
+			break;
+		if (mc->cache_sender_cookie == (u64)meta)
+			goto done;
+	}
+
+	if (unlikely(i >= MAX_DESC_CACHE - 1)) {
+		XIO_ERR("#%d desc cache overflow\n", msock->s_debug_nr);
+		return NULL;
+	}
+
+	mc = brick_block_alloc(0, maxlen);
+
+	memset(mc, 0, maxlen);
+	mc->cache_sender_cookie = (u64)meta;
+	/*  further bits may be used in future */
+	mc->cache_sender_proto = msock->s_send_proto;
+	mc->cache_recver_proto = msock->s_recv_proto;
+
+	maxlen -= sizeof(struct xio_desc_cache);
+	mi = (void *)(mc + 1);
+
+	status = _add_fields(mi, meta, 0, "", maxlen);
+
+	if (likely(status > 0)) {
+		mc->cache_items = status;
+		mc->cache_is_bigendian = myself_is_bigendian;
+		msock->s_desc_send[i] = mc;
+		*cache_index = i;
+	} else {
+		brick_block_free(mc, orig_len);
+		mc = NULL;
+	}
+
+done:
+	return mc;
+}
+
+static
+int _make_recver_cache(struct xio_desc_cache *mc, const struct meta *meta, int offset, const char *prefix)
+{
+	char *tmp = brick_string_alloc(MAX_FIELD_LEN);
+	int count = 0;
+	int i;
+
+	for (; meta->field_name != NULL; meta++, count++) {
+		snprintf(tmp, MAX_FIELD_LEN, "%s.%s", prefix, meta->field_name);
+		for (i = 0; i < mc->cache_items; i++) {
+			struct xio_desc_item *mi = ((struct xio_desc_item *)(mc + 1)) + i;
+
+			if (meta->field_type == mi->field_type &&
+			    !strcmp(tmp, mi->field_name)) {
+				mi->field_recver_size = meta->field_data_size;
+				mi->field_recver_offset = offset + meta->field_offset;
+				if (meta->field_type == FIELD_SUB) {
+					int sub_count = _make_recver_cache(mc,
+
+						meta->field_ref,
+						mi->field_recver_offset,
+						tmp);
+					if (unlikely(sub_count <= 0)) {
+						count = 0;
+						goto done;
+					}
+				}
+				goto found;
+			}
+		}
+		if (unlikely(!count)) {
+			XIO_ERR("field '%s' is missing\n", meta->field_name);
+			goto done;
+		}
+		XIO_WRN("field %2d '%s' is missing\n", count, meta->field_name);
+found:;
+	}
+done:
+	brick_string_free(tmp);
+	return count;
+}
+
+static
+int make_recver_cache(struct xio_desc_cache *mc, const struct meta *meta)
+{
+	int count;
+	int i;
+
+	mc->cache_recver_cookie = (u64)meta;
+	count = _make_recver_cache(mc, meta, 0, "");
+
+	for (i = 0; i < mc->cache_items; i++) {
+		struct xio_desc_item *mi = ((struct xio_desc_item *)(mc + 1)) + i;
+
+		if (unlikely(mi->field_recver_offset < 0))
+			XIO_WRN("field '%s' is not transferred\n", mi->field_name);
+	}
+	return count;
+}
+
+#define _CHECK_STATUS(_txt_)						\
+do {									\
+	if (unlikely(status < 0)) {					\
+		XIO_DBG("%s status = %d\n", _txt_, status);		\
+		goto err;						\
+	}								\
+} while (0)
+
+static
+int _desc_send_item(struct xio_socket *msock,
+	const void *data,
+	const struct xio_desc_cache *mc,
+	int index,
+	bool cork)
+{
+	struct xio_desc_item *mi = ((struct xio_desc_item *)(mc + 1)) + index;
+	const void *item = data + mi->field_sender_offset;
+
+	s16 data_len = mi->field_data_size;
+	s16 transfer_len = mi->field_sender_size;
+	int status;
+	bool is_signed = false;
+	int res = -1;
+
+	switch (mi->field_type) {
+	case FIELD_REF:
+		XIO_ERR("field '%s' NYI type = %d\n", mi->field_name, mi->field_type);
+		goto err;
+	case FIELD_SUB:
+		/* skip this */
+		res = 0;
+		break;
+	case FIELD_INT:
+		is_signed = true;
+		/* fallthrough */
+	case FIELD_UINT:
+		if (unlikely(data_len <= 0 || data_len > MAX_INT_TRANSFER)) {
+			XIO_ERR("field '%s' bad data_len = %d\n", mi->field_name, data_len);
+			goto err;
+		}
+		if (unlikely(transfer_len > MAX_INT_TRANSFER)) {
+			XIO_ERR("field '%s' bad transfer_len = %d\n", mi->field_name, transfer_len);
+			goto err;
+		}
+
+		if (likely(data_len == transfer_len))
+			goto raw;
+
+		if (transfer_len > data_len) {
+			int diff = transfer_len - data_len;
+			char empty[diff];
+			char sign;
+
+			sign = get_sign(item, data_len, myself_is_bigendian, is_signed);
+			memset(empty, sign, diff);
+
+			if (myself_is_bigendian) {
+				status = xio_send_raw(msock, empty, diff, true);
+				_CHECK_STATUS("send_diff");
+				status = xio_send_raw(msock, item, data_len, cork);
+				_CHECK_STATUS("send_item");
+
+			} else {
+				status = xio_send_raw(msock, item, data_len, true);
+				_CHECK_STATUS("send_item");
+				status = xio_send_raw(msock, empty, diff, cork);
+				_CHECK_STATUS("send_diff");
+			}
+
+			res = data_len;
+			break;
+		} else if (unlikely(transfer_len <= 0)) {
+			XIO_ERR("bad transfer_len = %d\n", transfer_len);
+			goto err;
+		} else { /*  transfer_len < data_len */
+			char check = get_sign(item, data_len, myself_is_bigendian, is_signed);
+			int start;
+			int end;
+			int i;
+
+			if (is_signed &&
+			    unlikely(get_sign(item, transfer_len, myself_is_bigendian, true) != check)) {
+				XIO_ERR("cannot sign-reduce signed integer from %d to %d bytes, byte %d !~ %d\n",
+					 data_len,
+					 transfer_len,
+					 ((char *)item)[transfer_len - 1],
+					 check);
+				goto err;
+			}
+
+			if (myself_is_bigendian) {
+				start = 0;
+				end = data_len - transfer_len;
+			} else {
+				start = transfer_len;
+				end = data_len;
+			}
+
+			for (i = start; i < end; i++) {
+				if (unlikely(((char *)item)[i] != check)) {
+					XIO_ERR("cannot sign-reduce %ssigned integer from %d to %d bytes at pos %d, byte %d != %d\n",
+						 is_signed ? "" : "un",
+						 data_len,
+						 transfer_len,
+						 i,
+						 ((char *)item)[i],
+						 check);
+					goto err;
+				}
+			}
+
+			/*  just omit the higher/lower bytes */
+			data_len = transfer_len;
+			if (myself_is_bigendian)
+				item += end;
+			goto raw;
+		}
+	case FIELD_STRING:
+		item = *(void **)item;
+		data_len = 0;
+		if (item)
+			data_len = strlen(item) + 1;
+
+		status = xio_send_raw(msock, &data_len, sizeof(data_len), true);
+		_CHECK_STATUS("send_string_len");
+		/* fallthrough */
+	case FIELD_RAW:
+raw:
+		if (unlikely(data_len < 0)) {
+			XIO_ERR("field '%s' bad data_len = %d\n", mi->field_name, data_len);
+			goto err;
+		}
+		status = xio_send_raw(msock, item, data_len, cork);
+		_CHECK_STATUS("send_raw");
+		res = data_len;
+		break;
+	default:
+		XIO_ERR("field '%s' unknown type = %d\n", mi->field_name, mi->field_type);
+	}
+err:
+	return res;
+}
+
+static
+int _desc_recv_item(struct xio_socket *msock, void *data, const struct xio_desc_cache *mc, int index, int line)
+{
+	struct xio_desc_item *mi = ((struct xio_desc_item *)(mc + 1)) + index;
+	void *item = NULL;
+
+	s16 data_len = mi->field_recver_size;
+	s16 transfer_len = mi->field_sender_size;
+	int status;
+	bool is_signed = false;
+	int res = -1;
+
+	if (likely(data && data_len > 0 && mi->field_recver_offset >= 0))
+		item = data + mi->field_recver_offset;
+
+	switch (mi->field_type) {
+	case FIELD_REF:
+		XIO_ERR("field '%s' NYI type = %d\n", mi->field_name, mi->field_type);
+		goto err;
+	case FIELD_SUB:
+		/* skip this */
+		res = 0;
+		break;
+	case FIELD_INT:
+		is_signed = true;
+		/* fallthrough */
+	case FIELD_UINT:
+		if (unlikely(data_len <= 0 || data_len > MAX_INT_TRANSFER)) {
+			XIO_ERR("field '%s' bad data_len = %d\n", mi->field_name, data_len);
+			goto err;
+		}
+		if (unlikely(transfer_len > MAX_INT_TRANSFER)) {
+			XIO_ERR("field '%s' bad transfer_len = %d\n", mi->field_name, transfer_len);
+			goto err;
+		}
+
+		if (likely(data_len == transfer_len))
+			goto raw;
+
+		if (transfer_len > data_len) {
+			int diff = transfer_len - data_len;
+			char empty[diff];
+			char check;
+
+			memset(empty, 0, diff);
+
+			if (myself_is_bigendian) {
+				status = xio_recv_raw(msock, empty, diff, diff);
+				_CHECK_STATUS("recv_diff");
+			}
+
+			status = xio_recv_raw(msock, item, data_len, data_len);
+			_CHECK_STATUS("recv_item");
+			if (unlikely(mc->cache_is_bigendian != myself_is_bigendian && item))
+				swap_bytes(item, data_len);
+
+			if (!myself_is_bigendian) {
+				status = xio_recv_raw(msock, empty, diff, diff);
+				_CHECK_STATUS("recv_diff");
+			}
+
+			/*  check that sign extension did no harm */
+			check = get_sign(empty, diff, mc->cache_is_bigendian, is_signed);
+			while (--diff >= 0) {
+				if (unlikely(empty[diff] != check)) {
+					XIO_ERR("field '%s' %sSIGNED INTEGER OVERFLOW on size reduction from %d to %d, byte %d != %d\n",
+						 mi->field_name,
+						 is_signed ? "" : "UN",
+						 transfer_len,
+						 data_len,
+						 empty[diff],
+						 check);
+					goto err;
+				}
+			}
+			if (is_signed && item &&
+			    unlikely(get_sign(item, data_len, myself_is_bigendian, true) != check)) {
+				XIO_ERR("field '%s' SIGNED INTEGER OVERLOW on reduction from size %d to %d, byte %d !~ %d\n",
+					 mi->field_name,
+					 transfer_len,
+					 data_len,
+					 ((char *)item)[data_len - 1],
+					 check);
+				goto err;
+			}
+
+			res = data_len;
+			break;
+		} else if (unlikely(transfer_len <= 0)) {
+			XIO_ERR("field '%s' bad transfer_len = %d\n", mi->field_name, transfer_len);
+			goto err;
+		} else if (unlikely(!item)) { /*  shortcut without checks */
+			data_len = transfer_len;
+			goto raw;
+		} else { /*  transfer_len < data_len */
+			int diff = data_len - transfer_len;
+			char *transfer_ptr = item;
+			char sign;
+
+			if (myself_is_bigendian)
+				transfer_ptr += diff;
+
+			status = xio_recv_raw(msock, transfer_ptr, transfer_len, transfer_len);
+			_CHECK_STATUS("recv_transfer");
+			if (unlikely(mc->cache_is_bigendian != myself_is_bigendian))
+				swap_bytes(transfer_ptr, transfer_len);
+
+			/*  sign-extend from transfer_len to data_len */
+			sign = get_sign(transfer_ptr, transfer_len, myself_is_bigendian, is_signed);
+			if (myself_is_bigendian)
+				memset(item, sign, diff);
+			else
+				memset(item + transfer_len, sign, diff);
+			res = data_len;
+			break;
+		}
+	case FIELD_STRING:
+		data_len = 0;
+		status = xio_recv_raw(msock, &data_len, sizeof(data_len), sizeof(data_len));
+		_CHECK_STATUS("recv_string_len");
+
+		if (unlikely(mc->cache_is_bigendian != myself_is_bigendian))
+			swap_bytes(&data_len, sizeof(data_len));
+
+		if (data_len > 0 && item) {
+			char *str = _brick_string_alloc(data_len, line);
+
+			*(void **)item = str;
+			item = str;
+		}
+
+		transfer_len = data_len;
+		/* fallthrough */
+	case FIELD_RAW:
+raw:
+		if (unlikely(data_len < 0)) {
+			XIO_ERR("field = '%s' implausible data_len = %d\n", mi->field_name, data_len);
+			goto err;
+		}
+		if (likely(data_len > 0)) {
+			if (unlikely(transfer_len != data_len)) {
+				XIO_ERR("cannot handle generic mismatch in transfer sizes, field = '%s', %d != %d\n",
+					mi->field_name,
+					transfer_len,
+					data_len);
+				goto err;
+			}
+			status = xio_recv_raw(msock, item, data_len, data_len);
+			_CHECK_STATUS("recv_raw");
+		}
+		res = data_len;
+		break;
+	default:
+		XIO_ERR("field '%s' unknown type = %d\n", mi->field_name, mi->field_type);
+	}
+err:
+	return res;
+}
+
+static inline
+int _desc_send_struct(struct xio_socket *msock, int cache_index, const void *data, int h_meta_len, bool cork)
+{
+	const struct xio_desc_cache *mc = msock->s_desc_send[cache_index];
+
+	struct xio_desc_header header = {
+		.h_magic = XIO_DESC_MAGIC,
+		.h_cookie = mc->cache_sender_cookie,
+		.h_meta_len = h_meta_len,
+		.h_index = data ? cache_index : -1,
+	};
+	int index;
+	int count = 0;
+	int status = 0;
+
+	status = xio_send_raw(msock, &header, sizeof(header), cork || data);
+	_CHECK_STATUS("send_header");
+
+	if (unlikely(h_meta_len > 0)) {
+		status = xio_send_raw(msock, mc, h_meta_len, true);
+		_CHECK_STATUS("send_meta");
+	}
+
+	if (likely(data)) {
+		for (index = 0; index < mc->cache_items; index++) {
+			status = _desc_send_item(msock, data, mc, index, cork || index < mc->cache_items-1);
+			_CHECK_STATUS("send_cache_item");
+			count++;
+		}
+	}
+
+	if (status >= 0)
+		status = count;
+err:
+	return status;
+}
+
+static
+int desc_send_struct(struct xio_socket *msock, const void *data, const struct meta *meta, bool cork)
+{
+	struct xio_desc_cache *mc;
+	int i;
+	int h_meta_len = 0;
+	int status = -EINVAL;
+
+	for (i = 0; i < MAX_DESC_CACHE; i++) {
+		mc = msock->s_desc_send[i];
+		if (!mc)
+			break;
+		if (mc->cache_sender_cookie == (u64)meta)
+			goto found;
+	}
+
+	mc = make_sender_cache(msock, meta, &i);
+	if (unlikely(!mc))
+		goto done;
+
+	h_meta_len = mc->cache_items * sizeof(struct xio_desc_item) + sizeof(struct xio_desc_cache);
+
+found:
+	status = _desc_send_struct(msock, i, data, h_meta_len, cork);
+
+done:
+	return status;
+}
+
+static
+int desc_recv_struct(struct xio_socket *msock, void *data, const struct meta *meta, int line)
+{
+	struct xio_desc_header header = {};
+	struct xio_desc_cache *mc;
+	int cache_index;
+	int index;
+	int count = 0;
+	int status = 0;
+	bool need_swap = false;
+
+	status = xio_recv_raw(msock, &header, sizeof(header), sizeof(header));
+	_CHECK_STATUS("recv_header");
+
+	if (unlikely(header.h_magic != XIO_DESC_MAGIC)) {
+		need_swap = true;
+		SWAP_FIELD(header.h_magic);
+		if (unlikely(header.h_magic != XIO_DESC_MAGIC)) {
+			XIO_WRN("#%d called from line %d bad packet header magic = %llx\n",
+				msock->s_debug_nr,
+				line,
+				header.h_magic);
+			status = -ENOMSG;
+			goto err;
+		}
+		SWAP_FIELD(header.h_cookie);
+		SWAP_FIELD(header.h_meta_len);
+		SWAP_FIELD(header.h_index);
+	}
+
+	cache_index = header.h_index;
+	if (cache_index < 0) { /*  EOR */
+		goto done;
+	}
+	if (unlikely(cache_index >= MAX_DESC_CACHE - 1)) {
+		XIO_WRN("#%d called from line %d bad cache index %d\n", msock->s_debug_nr, line, cache_index);
+		status = -EBADF;
+		goto err;
+	}
+
+	mc = msock->s_desc_recv[cache_index];
+	if (unlikely(!mc)) {
+		if (unlikely(header.h_meta_len <= 0)) {
+			XIO_WRN("#%d called from line %d missing meta information\n", msock->s_debug_nr, line);
+			status = -ENOMSG;
+			goto err;
+		}
+
+		mc = _brick_block_alloc(0, PAGE_SIZE, line);
+
+		status = xio_recv_raw(msock, mc, header.h_meta_len, header.h_meta_len);
+		if (unlikely(status < 0))
+			brick_block_free(mc, PAGE_SIZE);
+		_CHECK_STATUS("recv_meta");
+
+		if (unlikely(need_swap))
+			swap_mc(mc, header.h_meta_len);
+
+		status = make_recver_cache(mc, meta);
+		if (unlikely(status < 0)) {
+			brick_block_free(mc, PAGE_SIZE);
+			goto err;
+		}
+		msock->s_desc_recv[cache_index] = mc;
+	} else if (unlikely(header.h_meta_len > 0)) {
+		XIO_WRN("#%d called from line %d has %d unexpected meta bytes\n",
+			msock->s_debug_nr,
+			line,
+			header.h_meta_len);
+		status = -EMSGSIZE;
+		goto err;
+	} else if (unlikely(mc->cache_recver_cookie != (u64)meta)) {
+		XIO_ERR("#%d protocol error %p != %p\n", msock->s_debug_nr, meta, (void *)mc->cache_recver_cookie);
+		dump_meta((void *)mc->cache_recver_cookie);
+		dump_meta(meta);
+		status = -EPROTO;
+		goto err;
+	}
+
+	for (index = 0; index < mc->cache_items; index++) {
+		status = _desc_recv_item(msock, data, mc, index, line);
+		_CHECK_STATUS("recv_cache_item");
+		count++;
+	}
+
+done:
+	if (status >= 0)
+		status = count;
+err:
+	return status;
+}
+
+int xio_send_struct(struct xio_socket *msock, const void *data, const struct meta *meta)
+{
+	return desc_send_struct(msock, data, meta, false);
+}
+
+int _xio_recv_struct(struct xio_socket *msock, void *data, const struct meta *meta, int line)
+{
+	return desc_recv_struct(msock, data, meta, line);
+}
+
+/*********************************************************************/
+
+/* High-level transport of xio structures
+ */
+
+const struct meta xio_cmd_meta[] = {
+	META_INI_SUB(cmd_stamp, struct xio_cmd, xio_timespec_meta),
+	META_INI(cmd_code, struct xio_cmd, FIELD_INT),
+	META_INI(cmd_int1, struct xio_cmd, FIELD_INT),
+	META_INI(cmd_str1, struct xio_cmd, FIELD_STRING),
+	{}
+};
+
+int xio_send_aio(struct xio_socket *msock, struct aio_object *aio)
+{
+	struct xio_cmd cmd = {
+		.cmd_code = CMD_AIO,
+		.cmd_int1 = aio->io_id,
+	};
+	int seq = 0;
+	int status;
+
+	if (aio->io_rw != 0 && aio->io_data && aio->io_cs_mode < 2)
+		cmd.cmd_code |= CMD_FLAG_HAS_DATA;
+
+	get_lamport(&cmd.cmd_stamp);
+
+	status = desc_send_struct(msock, &cmd, xio_cmd_meta, true);
+	if (status < 0)
+		goto done;
+
+	seq = 0;
+	status = desc_send_struct(msock, aio, xio_aio_user_meta, cmd.cmd_code & CMD_FLAG_HAS_DATA);
+	if (status < 0)
+		goto done;
+
+	if (cmd.cmd_code & CMD_FLAG_HAS_DATA)
+		status = xio_send_compressed(msock, aio->io_data, aio->io_len, xio_net_compress_data, false);
+done:
+	return status;
+}
+
+int xio_recv_aio(struct xio_socket *msock, struct aio_object *aio, struct xio_cmd *cmd)
+{
+	int status;
+
+	status = desc_recv_struct(msock, aio, xio_aio_user_meta, __LINE__);
+	if (status < 0)
+		goto done;
+
+	set_lamport(&cmd->cmd_stamp);
+
+	if (cmd->cmd_code & CMD_FLAG_HAS_DATA) {
+		if (!aio->io_data)
+			aio->io_data = brick_block_alloc(0, aio->io_len);
+		status = xio_recv_compressed(msock, aio->io_data, aio->io_len, aio->io_len);
+		if (unlikely(status < 0))
+			XIO_WRN("#%d aio_len = %d, status = %d\n", msock->s_debug_nr, aio->io_len, status);
+	}
+done:
+	return status;
+}
+
+int xio_send_cb(struct xio_socket *msock, struct aio_object *aio)
+{
+	struct xio_cmd cmd = {
+		.cmd_code = CMD_CB,
+		.cmd_int1 = aio->io_id,
+	};
+	int seq = 0;
+	int status;
+
+	if (aio->io_rw == 0 && aio->io_data && aio->io_cs_mode < 2)
+		cmd.cmd_code |= CMD_FLAG_HAS_DATA;
+
+	get_lamport(&cmd.cmd_stamp);
+
+	status = desc_send_struct(msock, &cmd, xio_cmd_meta, true);
+	if (status < 0)
+		goto done;
+
+	seq = 0;
+	status = desc_send_struct(msock, aio, xio_aio_user_meta, cmd.cmd_code & CMD_FLAG_HAS_DATA);
+	if (status < 0)
+		goto done;
+
+	if (cmd.cmd_code & CMD_FLAG_HAS_DATA)
+		status = xio_send_compressed(msock, aio->io_data, aio->io_len, xio_net_compress_data, false);
+done:
+	return status;
+}
+
+int xio_recv_cb(struct xio_socket *msock, struct aio_object *aio, struct xio_cmd *cmd)
+{
+	int status;
+
+	status = desc_recv_struct(msock, aio, xio_aio_user_meta, __LINE__);
+	if (status < 0)
+		goto done;
+
+	set_lamport(&cmd->cmd_stamp);
+
+	if (cmd->cmd_code & CMD_FLAG_HAS_DATA) {
+		if (!aio->io_data) {
+			XIO_WRN("#%d no internal buffer available\n", msock->s_debug_nr);
+			status = -EINVAL;
+			goto done;
+		}
+		status = xio_recv_compressed(msock, aio->io_data, aio->io_len, aio->io_len);
+	}
+done:
+	return status;
+}
+
+/***************** module init stuff ************************/
+
+char *(*xio_translate_hostname)(const char *name) = NULL;
+
+bool xio_net_is_alive;
+
+int __init init_xio_net(void)
+{
+	XIO_INF("init_net()\n");
+	xio_net_is_alive = true;
+	return 0;
+}
+
+void exit_xio_net(void)
+{
+	xio_net_is_alive = false;
+	brick_string_free(id);
+	id = NULL;
+	XIO_INF("exit_net()\n");
+}
include/linux/xio/xio_net.h new file +171 -0

171 lines not on this page · raw message

Signature · 1 line
2.6.4
#15 lore raw

RFC 15/31 mars: add new module lib_mapfree

In reply to Thomas Schoebel-Theuer
---
 drivers/staging/mars/xio_bricks/lib_mapfree.c | 380 ++++++++++++++++++++++++++
 include/linux/xio/lib_mapfree.h               |  84 ++++++
 2 files changed, 464 insertions(+)
 create mode 100644 drivers/staging/mars/xio_bricks/lib_mapfree.c
 create mode 100644 include/linux/xio/lib_mapfree.h
drivers/staging/mars/xio_bricks/lib_mapfree.c new file +380 -0
new file mode 100644
index 0000000..6b464d7
@@ -0,0 +1,380 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/xio/lib_mapfree.h>
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/string.h>
+#include <linux/list.h>
+#include <linux/types.h>
+#include <linux/blkdev.h>
+#include <linux/spinlock.h>
+#include <linux/wait.h>
+#include <linux/file.h>
+
+/*  time to wait between background mapfree operations */
+int mapfree_period_sec = 10;
+
+/*  some grace space where no regular cleanup should occur */
+int mapfree_grace_keep_mb = 16;
+
+static
+DECLARE_RWSEM(mapfree_mutex);
+
+static
+LIST_HEAD(mapfree_list);
+
+void mapfree_pages(struct mapfree_info *mf, int grace_keep)
+{
+	struct address_space *mapping;
+	pgoff_t start;
+	pgoff_t end;
+
+	if (unlikely(!mf))
+	    goto done;
+	if (unlikely(!mf->mf_filp))
+		goto done;
+
+	mapping = mf->mf_filp->f_mapping;
+	if (unlikely(!mapping))
+		goto done;
+
+	if (grace_keep < 0) { /*  force full flush */
+		start = 0;
+		end = -1;
+	} else {
+		unsigned long flags;
+		loff_t tmp;
+		loff_t min;
+
+		spin_lock_irqsave(&mf->mf_lock, flags);
+
+		min = tmp = mf->mf_min[0];
+		if (likely(mf->mf_min[1] < min))
+			min = mf->mf_min[1];
+		if (tmp) {
+			mf->mf_min[1] = tmp;
+			mf->mf_min[0] = 0;
+		}
+
+		spin_unlock_irqrestore(&mf->mf_lock, flags);
+
+		min -= (loff_t)grace_keep * (1024 * 1024); /*  megabytes */
+		end = 0;
+
+		if (min > 0 || mf->mf_last) {
+			start = mf->mf_last / PAGE_SIZE;
+			/*  add some grace overlapping */
+			if (likely(start > 0))
+				start--;
+			mf->mf_last = min;
+			end = min / PAGE_SIZE;
+		} else	{ /*  there was no progress for at least 2 rounds */
+			start = 0;
+			if (!grace_keep) /*  also flush thoroughly */
+				end = -1;
+		}
+
+		XIO_DBG("file = '%s' start = %lu end = %lu\n", mf->mf_name, start, end);
+	}
+
+	if (end > start || end == -1)
+		invalidate_mapping_pages(mapping, start, end);
+
+done:;
+}
+
+static
+void _mapfree_put(struct mapfree_info *mf)
+{
+	if (atomic_dec_and_test(&mf->mf_count)) {
+		XIO_DBG("closing file '%s' filp = %p\n", mf->mf_name, mf->mf_filp);
+		list_del_init(&mf->mf_head);
+		CHECK_HEAD_EMPTY(&mf->mf_dirty_anchor);
+		if (likely(mf->mf_filp)) {
+			mapfree_pages(mf, -1);
+			filp_close(mf->mf_filp, NULL);
+		}
+		brick_string_free(mf->mf_name);
+		brick_mem_free(mf);
+	}
+}
+
+void mapfree_put(struct mapfree_info *mf)
+{
+	if (likely(mf)) {
+		down_write(&mapfree_mutex);
+		_mapfree_put(mf);
+		up_write(&mapfree_mutex);
+	}
+}
+
+struct mapfree_info *mapfree_get(const char *name, int flags)
+{
+	struct mapfree_info *mf = NULL;
+	struct list_head *tmp;
+
+	if (!(flags & O_DIRECT)) {
+		down_read(&mapfree_mutex);
+		for (tmp = mapfree_list.next; tmp != &mapfree_list; tmp = tmp->next) {
+			struct mapfree_info *_mf = container_of(tmp, struct mapfree_info, mf_head);
+
+			if (_mf->mf_flags == flags && !strcmp(_mf->mf_name, name)) {
+				mf = _mf;
+				atomic_inc(&mf->mf_count);
+				break;
+			}
+		}
+		up_read(&mapfree_mutex);
+
+		if (mf)
+			goto done;
+	}
+
+	for (;;) {
+		struct address_space *mapping;
+		struct inode *inode = NULL;
+		int ra = 1;
+		int prot = 0600;
+
+		mm_segment_t oldfs;
+
+		mf = brick_zmem_alloc(sizeof(struct mapfree_info));
+
+		mf->mf_name = brick_strdup(name);
+
+		mf->mf_flags = flags;
+		INIT_LIST_HEAD(&mf->mf_head);
+		INIT_LIST_HEAD(&mf->mf_dirty_anchor);
+		atomic_set(&mf->mf_count, 1);
+		spin_lock_init(&mf->mf_lock);
+		mf->mf_max = -1;
+
+		oldfs = get_fs();
+		set_fs(get_ds());
+		mf->mf_filp = filp_open(name, flags, prot);
+		set_fs(oldfs);
+
+		XIO_DBG("file '%s' flags = %d prot = %d filp = %p\n", name, flags, prot, mf->mf_filp);
+
+		if (unlikely(!mf->mf_filp || IS_ERR(mf->mf_filp))) {
+			int err = PTR_ERR(mf->mf_filp);
+
+			XIO_ERR("can't open file '%s' status=%d\n", name, err);
+			mf->mf_filp = NULL;
+			_mapfree_put(mf);
+			mf = NULL;
+			break;
+		}
+
+		mapping = mf->mf_filp->f_mapping;
+		if (likely(mapping))
+			inode = mapping->host;
+		if (unlikely(!mapping || !inode)) {
+			XIO_ERR("file '%s' has no mapping\n", name);
+			mf->mf_filp = NULL;
+			_mapfree_put(mf);
+			mf = NULL;
+			break;
+		}
+
+		mapping_set_gfp_mask(mapping, mapping_gfp_mask(mapping) & ~(__GFP_IO | __GFP_FS));
+
+		mf->mf_max = i_size_read(inode);
+
+		if (S_ISBLK(inode->i_mode)) {
+			XIO_INF("changing blkdev readahead from %lu to %d\n",
+				inode->i_bdev->bd_disk->queue->backing_dev_info.ra_pages,
+				ra);
+			inode->i_bdev->bd_disk->queue->backing_dev_info.ra_pages = ra;
+		}
+
+		if (flags & O_DIRECT) { /*  never share them */
+			break;
+		}
+
+		/*  maintain global list of all open files */
+		down_write(&mapfree_mutex);
+		for (tmp = mapfree_list.next; tmp != &mapfree_list; tmp = tmp->next) {
+			struct mapfree_info *_mf = container_of(tmp, struct mapfree_info, mf_head);
+
+			if (unlikely(_mf->mf_flags == flags && !strcmp(_mf->mf_name, name))) {
+				XIO_WRN("race on creation of '%s' detected\n", name);
+				_mapfree_put(mf);
+				mf = _mf;
+				atomic_inc(&mf->mf_count);
+				goto leave;
+			}
+		}
+		list_add_tail(&mf->mf_head, &mapfree_list);
+leave:
+		up_write(&mapfree_mutex);
+		break;
+	}
+done:
+	return mf;
+}
+
+void mapfree_set(struct mapfree_info *mf, loff_t min, loff_t max)
+{
+	unsigned long flags;
+
+	if (likely(mf)) {
+		spin_lock_irqsave(&mf->mf_lock, flags);
+		if (!mf->mf_min[0] || mf->mf_min[0] > min)
+			mf->mf_min[0] = min;
+		if (max >= 0 && mf->mf_max < max)
+			mf->mf_max = max;
+		spin_unlock_irqrestore(&mf->mf_lock, flags);
+	}
+}
+
+static
+int mapfree_thread(void *data)
+{
+	while (!brick_thread_should_stop()) {
+		struct mapfree_info *mf = NULL;
+		struct list_head *tmp;
+		long long eldest = 0;
+
+		brick_msleep(500);
+
+		if (mapfree_period_sec <= 0)
+			continue;
+
+		down_read(&mapfree_mutex);
+
+		for (tmp = mapfree_list.next; tmp != &mapfree_list; tmp = tmp->next) {
+			struct mapfree_info *_mf = container_of(tmp, struct mapfree_info, mf_head);
+
+			if (unlikely(!_mf->mf_jiffies)) {
+				_mf->mf_jiffies = jiffies;
+				continue;
+			}
+			if ((long long)jiffies - _mf->mf_jiffies > mapfree_period_sec * HZ &&
+			    (!mf || _mf->mf_jiffies < eldest)) {
+				mf = _mf;
+				eldest = _mf->mf_jiffies;
+			}
+		}
+		if (mf)
+			atomic_inc(&mf->mf_count);
+
+		up_read(&mapfree_mutex);
+
+		if (!mf)
+			continue;
+
+		mapfree_pages(mf, mapfree_grace_keep_mb);
+
+		mf->mf_jiffies = jiffies;
+		mapfree_put(mf);
+	}
+	return 0;
+}
+
+/***************** dirty IOs on the fly  *****************/
+
+void mf_insert_dirty(struct mapfree_info *mf, struct dirty_info *di)
+{
+	unsigned long flags;
+
+	if (likely(di->dirty_aio && mf)) {
+		spin_lock_irqsave(&mf->mf_lock, flags);
+		list_del(&di->dirty_head);
+		list_add(&di->dirty_head, &mf->mf_dirty_anchor);
+		spin_unlock_irqrestore(&mf->mf_lock, flags);
+	}
+}
+
+void mf_remove_dirty(struct mapfree_info *mf, struct dirty_info *di)
+{
+	unsigned long flags;
+
+	if (!list_empty(&di->dirty_head) && mf) {
+		spin_lock_irqsave(&mf->mf_lock, flags);
+		list_del_init(&di->dirty_head);
+		spin_unlock_irqrestore(&mf->mf_lock, flags);
+	}
+}
+
+void mf_get_dirty(struct mapfree_info *mf, loff_t *min, loff_t *max, int min_stage, int max_stage)
+{
+	unsigned long flags;
+
+	struct list_head *tmp;
+
+	if (unlikely(!mf))
+	    goto done;
+
+	spin_lock_irqsave(&mf->mf_lock, flags);
+	for (tmp = mf->mf_dirty_anchor.next; tmp != &mf->mf_dirty_anchor; tmp = tmp->next) {
+		struct dirty_info *di = container_of(tmp, struct dirty_info, dirty_head);
+		struct aio_object *aio = di->dirty_aio;
+
+		if (unlikely(!aio))
+			continue;
+		if (di->dirty_stage < min_stage || di->dirty_stage > max_stage)
+			continue;
+		if (aio->io_pos < *min)
+			*min = aio->io_pos;
+		if (aio->io_pos + aio->io_len > *max)
+			*max = aio->io_pos + aio->io_len;
+	}
+	spin_unlock_irqrestore(&mf->mf_lock, flags);
+done:;
+}
+
+void mf_get_any_dirty(const char *filename, loff_t *min, loff_t *max, int min_stage, int max_stage)
+{
+	struct list_head *tmp;
+
+	down_read(&mapfree_mutex);
+	for (tmp = mapfree_list.next; tmp != &mapfree_list; tmp = tmp->next) {
+		struct mapfree_info *mf = container_of(tmp, struct mapfree_info, mf_head);
+
+		if (!strcmp(mf->mf_name, filename))
+			mf_get_dirty(mf, min, max, min_stage, max_stage);
+	}
+	up_read(&mapfree_mutex);
+}
+
+/***************** module init stuff ************************/
+
+static
+struct task_struct *mf_thread;
+
+int __init init_xio_mapfree(void)
+{
+	XIO_DBG("init_mapfree()\n");
+	mf_thread = brick_thread_create(mapfree_thread, NULL, "xio_mapfree");
+	if (unlikely(!mf_thread)) {
+		XIO_ERR("could not create mapfree thread\n");
+		return -ENOMEM;
+	}
+	return 0;
+}
+
+void exit_xio_mapfree(void)
+{
+	XIO_DBG("exit_mapfree()\n");
+	if (likely(mf_thread)) {
+		brick_thread_stop(mf_thread);
+		mf_thread = NULL;
+	}
+}
include/linux/xio/lib_mapfree.h new file +84 -0

84 lines not on this page · raw message

Signature · 1 line
2.6.4
#16 lore raw

RFC 16/31 mars: add new module lib_log

In reply to Thomas Schoebel-Theuer
---
 drivers/staging/mars/xio_bricks/lib_log.c | 505 ++++++++++++++++++++++++++++++
 include/linux/xio/lib_log.h               | 329 +++++++++++++++++++
 2 files changed, 834 insertions(+)
 create mode 100644 drivers/staging/mars/xio_bricks/lib_log.c
 create mode 100644 include/linux/xio/lib_log.h
drivers/staging/mars/xio_bricks/lib_log.c new file +505 -0
new file mode 100644
index 0000000..a8382e5
@@ -0,0 +1,505 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/bio.h>
+
+#include <linux/xio/lib_log.h>
+
+atomic_t global_aio_flying = ATOMIC_INIT(0);
+
+void exit_logst(struct log_status *logst)
+{
+	int count;
+
+	log_flush(logst);
+
+	/*  TODO: replace by event */
+	count = 0;
+	while (atomic_read(&logst->aio_flying) > 0) {
+		if (!count++)
+			XIO_DBG("waiting for IO terminating...");
+		brick_msleep(500);
+	}
+	if (logst->read_aio) {
+		XIO_DBG("putting read_aio\n");
+		GENERIC_INPUT_CALL(logst->input, aio_put, logst->read_aio);
+		logst->read_aio = NULL;
+	}
+	if (logst->log_aio) {
+		XIO_DBG("putting log_aio\n");
+		GENERIC_INPUT_CALL(logst->input, aio_put, logst->log_aio);
+		logst->log_aio = NULL;
+	}
+}
+
+void init_logst(struct log_status *logst, struct xio_input *input, loff_t start_pos, loff_t end_pos)
+{
+	exit_logst(logst);
+
+	memset(logst, 0, sizeof(struct log_status));
+
+	logst->input = input;
+	logst->brick = input->brick;
+	logst->start_pos = start_pos;
+	logst->log_pos = start_pos;
+	logst->end_pos = end_pos;
+	init_waitqueue_head(&logst->event);
+}
+
+#define XIO_LOG_CB_MAX			32
+
+struct log_cb_info {
+	struct aio_object *aio;
+	struct log_status *logst;
+	struct semaphore mutex;
+	atomic_t refcount;
+	int nr_cb;
+	void (*endios[XIO_LOG_CB_MAX])(void *private, int error);
+	void *privates[XIO_LOG_CB_MAX];
+};
+
+static
+void put_log_cb_info(struct log_cb_info *cb_info)
+{
+	if (atomic_dec_and_test(&cb_info->refcount))
+		brick_mem_free(cb_info);
+}
+
+static
+void _do_callbacks(struct log_cb_info *cb_info, int error)
+{
+	int i;
+
+	down(&cb_info->mutex);
+	for (i = 0; i < cb_info->nr_cb; i++) {
+		void (*end_fn)(void *private, int error);
+
+		end_fn = cb_info->endios[i];
+		cb_info->endios[i] = NULL;
+		if (end_fn)
+			end_fn(cb_info->privates[i], error);
+	}
+	up(&cb_info->mutex);
+}
+
+static
+void log_write_endio(struct generic_callback *cb)
+{
+	struct log_cb_info *cb_info = cb->cb_private;
+	struct log_status *logst;
+
+	LAST_CALLBACK(cb);
+	CHECK_PTR(cb_info, err);
+
+	logst = cb_info->logst;
+	CHECK_PTR(logst, done);
+
+	_do_callbacks(cb_info, cb->cb_error);
+
+done:
+	put_log_cb_info(cb_info);
+	atomic_dec(&logst->aio_flying);
+	atomic_dec(&global_aio_flying);
+	if (logst->signal_event)
+		wake_up_interruptible(logst->signal_event);
+
+	goto out_return;
+err:
+	XIO_FAT("internal pointer corruption\n");
+out_return:;
+}
+
+void log_flush(struct log_status *logst)
+{
+	struct aio_object *aio = logst->log_aio;
+	struct log_cb_info *cb_info;
+	int align_size;
+	int gap;
+
+	if (!aio || !logst->count)
+		goto out_return;
+	gap = 0;
+	align_size = (logst->align_size / PAGE_SIZE) * PAGE_SIZE;
+	if (align_size > 0) {
+		/*  round up to next alignment border */
+		int align_offset = logst->offset & (align_size-1);
+
+		if (align_offset > 0) {
+			int restlen = aio->io_len - logst->offset;
+
+			gap = align_size - align_offset;
+			if (unlikely(gap > restlen))
+				gap = restlen;
+		}
+	}
+	if (gap > 0) {
+		/*  don't leak information from kernelspace */
+		memset(aio->io_data + logst->offset, 0, gap);
+		logst->offset += gap;
+	}
+	aio->io_len = logst->offset;
+	memcpy(&logst->log_pos_stamp, &logst->tmp_pos_stamp, sizeof(logst->log_pos_stamp));
+
+	cb_info = logst->private;
+	logst->private = NULL;
+	SETUP_CALLBACK(aio, log_write_endio, cb_info);
+	cb_info->logst = logst;
+	aio->io_rw = 1;
+
+	atomic_inc(&logst->aio_flying);
+	atomic_inc(&global_aio_flying);
+
+	GENERIC_INPUT_CALL(logst->input, aio_io, aio);
+	GENERIC_INPUT_CALL(logst->input, aio_put, aio);
+
+	logst->log_pos += logst->offset;
+	logst->offset = 0;
+	logst->count = 0;
+	logst->log_aio = NULL;
+
+	put_log_cb_info(cb_info);
+out_return:;
+}
+
+void *log_reserve(struct log_status *logst, struct log_header *lh)
+{
+	struct log_cb_info *cb_info = logst->private;
+	struct aio_object *aio;
+	void *data;
+
+	short total_len = lh->l_len + OVERHEAD;
+	int offset;
+	int status;
+
+	if (unlikely(lh->l_len <= 0 || lh->l_len > logst->max_size)) {
+		XIO_ERR("trying to write %d bytes, max allowed = %d\n", lh->l_len, logst->max_size);
+		goto err;
+	}
+
+	aio = logst->log_aio;
+	if ((aio && total_len > aio->io_len - logst->offset)
+	   || !cb_info || cb_info->nr_cb >= XIO_LOG_CB_MAX) {
+		log_flush(logst);
+	}
+
+	aio = logst->log_aio;
+	if (!aio) {
+		if (unlikely(logst->private)) {
+			XIO_ERR("oops\n");
+			brick_mem_free(logst->private);
+		}
+		logst->private = brick_zmem_alloc(sizeof(struct log_cb_info));
+		cb_info = logst->private;
+		sema_init(&cb_info->mutex, 1);
+		atomic_set(&cb_info->refcount, 2);
+
+		aio = xio_alloc_aio(logst->brick);
+		cb_info->aio = aio;
+
+		aio->io_pos = logst->log_pos;
+		aio->io_len = logst->chunk_size ? logst->chunk_size : total_len;
+		aio->io_may_write = WRITE;
+		aio->io_prio = logst->io_prio;
+
+		for (;;) {
+			status = GENERIC_INPUT_CALL(logst->input, aio_get, aio);
+			if (likely(status >= 0))
+				break;
+			if (status != -ENOMEM && status != -EAGAIN) {
+				XIO_ERR("aio_get() failed, status = %d\n", status);
+				goto err_free;
+			}
+			brick_msleep(100);
+		}
+
+		if (unlikely(aio->io_len < total_len)) {
+			XIO_ERR("io_len = %d total_len = %d\n", aio->io_len, total_len);
+			goto put;
+		}
+
+		logst->offset = 0;
+		logst->log_aio = aio;
+	}
+
+	offset = logst->offset;
+	data = aio->io_data;
+	DATA_PUT(data, offset, START_MAGIC);
+	DATA_PUT(data, offset, (char)FORMAT_VERSION);
+	logst->validflag_offset = offset;
+	DATA_PUT(data, offset, (char)0); /*  valid_flag */
+	DATA_PUT(data, offset, total_len); /*  start of next header */
+	DATA_PUT(data, offset, lh->l_stamp.tv_sec);
+	DATA_PUT(data, offset, lh->l_stamp.tv_nsec);
+	DATA_PUT(data, offset, lh->l_pos);
+	logst->reallen_offset = offset;
+	DATA_PUT(data, offset, lh->l_len);
+	DATA_PUT(data, offset, (short)0); /*  spare */
+	DATA_PUT(data, offset, (int)0); /*  spare */
+	DATA_PUT(data, offset, lh->l_code);
+	DATA_PUT(data, offset, (short)0); /*  spare */
+
+	/*  remember the last timestamp */
+	memcpy(&logst->tmp_pos_stamp, &lh->l_stamp, sizeof(logst->tmp_pos_stamp));
+
+	logst->payload_offset = offset;
+	logst->payload_len = lh->l_len;
+
+	return data + offset;
+
+put:
+	GENERIC_INPUT_CALL(logst->input, aio_put, aio);
+	logst->log_aio = NULL;
+	return NULL;
+
+err_free:
+	obj_free(aio);
+	if (logst->private) {
+		/*  TODO: if callbacks are already registered, call them here with some error code */
+		brick_mem_free(logst->private);
+		logst->private = NULL;
+	}
+err:
+	return NULL;
+}
+
+bool log_finalize(struct log_status *logst, int len, void (*endio)(void *private, int error), void *private)
+{
+	struct aio_object *aio = logst->log_aio;
+	struct log_cb_info *cb_info = logst->private;
+	struct timespec now;
+	void *data;
+	int offset;
+	int restlen;
+	int nr_cb;
+	int crc;
+	bool ok = false;
+
+	CHECK_PTR(aio, err);
+
+	if (unlikely(len > logst->payload_len)) {
+		XIO_ERR("trying to write more than reserved (%d > %d)\n", len, logst->payload_len);
+		goto err;
+	}
+	restlen = aio->io_len - logst->offset;
+	if (unlikely(len + END_OVERHEAD > restlen)) {
+		XIO_ERR("trying to write more than available (%d > %d)\n", len, (int)(restlen - END_OVERHEAD));
+		goto err;
+	}
+	if (unlikely(!cb_info || cb_info->nr_cb >= XIO_LOG_CB_MAX)) {
+		XIO_ERR("too many endio() calls\n");
+		goto err;
+	}
+
+	data = aio->io_data;
+
+	crc = 0;
+	if (logst->do_crc) {
+		unsigned char checksum[xio_digest_size];
+
+		xio_digest(checksum, data + logst->payload_offset, len);
+		crc = *(int *)checksum;
+	}
+
+	/* Correct the length in the header.
+	 */
+	offset = logst->reallen_offset;
+	DATA_PUT(data, offset, (short)len);
+
+	/* Write the trailer.
+	 */
+	offset = logst->payload_offset + len;
+	DATA_PUT(data, offset, END_MAGIC);
+	DATA_PUT(data, offset, crc);
+	DATA_PUT(data, offset, (char)1);  /*  valid_flag copy */
+	DATA_PUT(data, offset, (char)0);  /*  spare */
+	DATA_PUT(data, offset, (short)0); /*  spare */
+	DATA_PUT(data, offset, logst->seq_nr + 1);
+	get_lamport(&now);    /*  when the log entry was ready. */
+	DATA_PUT(data, offset, now.tv_sec);
+	DATA_PUT(data, offset, now.tv_nsec);
+
+	if (unlikely(offset > aio->io_len)) {
+		XIO_FAT("length calculation was wrong: %d > %d\n", offset, aio->io_len);
+		goto err;
+	}
+	logst->offset = offset;
+
+	/* This must come last. In case of incomplete
+	 * or even overlapping disk transfers, this indicates
+	 * the completeness / integrity of the payload at
+	 * the time of starting the transfer.
+	 */
+	offset = logst->validflag_offset;
+	DATA_PUT(data, offset, (char)1);
+
+	nr_cb = cb_info->nr_cb++;
+	cb_info->endios[nr_cb] = endio;
+	cb_info->privates[nr_cb] = private;
+
+	/*  report success */
+	logst->seq_nr++;
+	logst->count++;
+	ok = true;
+
+err:
+	return ok;
+}
+
+static
+void log_read_endio(struct generic_callback *cb)
+{
+	struct log_status *logst = cb->cb_private;
+
+	LAST_CALLBACK(cb);
+	CHECK_PTR(logst, err);
+	logst->error_code = cb->cb_error;
+	logst->got = true;
+	wake_up_interruptible(&logst->event);
+	goto out_return;
+err:
+	XIO_FAT("internal pointer corruption\n");
+out_return:;
+}
+
+int log_read(struct log_status *logst, bool sloppy, struct log_header *lh, void **payload, int *payload_len)
+{
+	struct aio_object *aio;
+	int old_offset;
+	int status;
+
+restart:
+	status = 0;
+	aio = logst->read_aio;
+	if (!aio || logst->do_free) {
+		loff_t this_len;
+
+		if (aio) {
+			GENERIC_INPUT_CALL(logst->input, aio_put, aio);
+			logst->read_aio = NULL;
+			logst->log_pos += logst->offset;
+			logst->offset = 0;
+		}
+
+		this_len = logst->end_pos - logst->log_pos;
+		if (this_len > logst->chunk_size) {
+			this_len = logst->chunk_size;
+		} else if (unlikely(this_len <= 0)) {
+			XIO_ERR("tried bad IO len %lld, start_pos = %lld log_pos = %lld end_pos = %lld\n",
+				this_len,
+				logst->start_pos,
+				logst->log_pos,
+				logst->end_pos);
+			status = -EOVERFLOW;
+			goto done;
+		}
+
+		aio = xio_alloc_aio(logst->brick);
+		aio->io_pos = logst->log_pos;
+		aio->io_len = this_len;
+		aio->io_prio = logst->io_prio;
+
+		status = GENERIC_INPUT_CALL(logst->input, aio_get, aio);
+		if (unlikely(status < 0)) {
+			if (status != -ENODATA)
+				XIO_ERR("aio_get() failed, status = %d\n", status);
+			goto done_free;
+		}
+		if (unlikely(aio->io_len <= OVERHEAD)) { /*  EOF */
+			status = 0;
+			goto done_put;
+		}
+
+		SETUP_CALLBACK(aio, log_read_endio, logst);
+		aio->io_rw = READ;
+		logst->offset = 0;
+		logst->got = false;
+		logst->do_free = false;
+
+		GENERIC_INPUT_CALL(logst->input, aio_io, aio);
+
+		wait_event_interruptible_timeout(logst->event, logst->got, 60 * HZ);
+		status = -ETIME;
+		if (!logst->got)
+			goto done_put;
+		status = logst->error_code;
+		if (status < 0)
+			goto done_put;
+		logst->read_aio = aio;
+	}
+
+	status = log_scan(aio->io_data + logst->offset,
+			  aio->io_len - logst->offset,
+			  aio->io_pos,
+			  logst->offset,
+			  sloppy,
+			  lh,
+			  payload,
+			  payload_len,
+			  &logst->seq_nr);
+
+	if (unlikely(status == 0)) {
+		XIO_ERR("bad logfile scan\n");
+		status = -EINVAL;
+	}
+	if (unlikely(status < 0))
+		goto done_put;
+
+	/*  memoize success */
+	logst->offset += status;
+	if (logst->offset + (logst->max_size + OVERHEAD) * 2 >= aio->io_len)
+		logst->do_free = true;
+
+done:
+	if (status == -ENODATA) {
+		/*  indicate EOF */
+		status = 0;
+	}
+	return status;
+
+done_put:
+	old_offset = logst->offset;
+	if (aio) {
+		GENERIC_INPUT_CALL(logst->input, aio_put, aio);
+		logst->read_aio = NULL;
+		logst->log_pos += logst->offset;
+		logst->offset = 0;
+	}
+	if (status == -EAGAIN && old_offset > 0)
+		goto restart;
+	goto done;
+
+done_free:
+	obj_free(aio);
+	logst->read_aio = NULL;
+	goto done;
+
+}
+
+/***************** module init stuff ************************/
+
+int __init init_log_format(void)
+{
+	XIO_INF("init_log_format()\n");
+	return 0;
+}
+
+void exit_log_format(void)
+{
+	XIO_INF("exit_log_format()\n");
+}
include/linux/xio/lib_log.h new file +329 -0

329 lines not on this page · raw message

Signature · 1 line
2.6.4
#17 lore raw

RFC 17/31 mars: add new module xio_bio

In reply to Thomas Schoebel-Theuer
---
 drivers/staging/mars/xio_bricks/xio_bio.c | 845 ++++++++++++++++++++++++++++++
 include/linux/xio/xio_bio.h               |  85 +++
 2 files changed, 930 insertions(+)
 create mode 100644 drivers/staging/mars/xio_bricks/xio_bio.c
 create mode 100644 include/linux/xio/xio_bio.h
drivers/staging/mars/xio_bricks/xio_bio.c new file +845 -0
new file mode 100644
index 0000000..ef18325
@@ -0,0 +1,845 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+/*  Bio brick (interface to blkdev IO via kernel bios) */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/string.h>
+#include <linux/bio.h>
+
+#include <linux/xio/xio.h>
+#include <linux/brick/lib_timing.h>
+#include <linux/xio/lib_mapfree.h>
+
+#include <linux/xio/xio_bio.h>
+static struct timing_stats timings[2];
+
+struct threshold bio_submit_threshold = {
+	.thr_ban = &xio_global_ban,
+	.thr_parent = &global_io_threshold,
+	.thr_limit = BIO_SUBMIT_MAX_LATENCY,
+	.thr_factor = 100,
+	.thr_plus = 0,
+};
+
+struct threshold bio_io_threshold[2] = {
+	[0] = {
+		.thr_ban = &xio_global_ban,
+		.thr_parent = &global_io_threshold,
+		.thr_limit = BIO_IO_R_MAX_LATENCY,
+		.thr_factor = 10,
+		.thr_plus = 10000,
+	},
+	[1] = {
+		.thr_ban = &xio_global_ban,
+		.thr_parent = &global_io_threshold,
+		.thr_limit = BIO_IO_W_MAX_LATENCY,
+		.thr_factor = 10,
+		.thr_plus = 10000,
+	},
+};
+
+/************************ own type definitions ***********************/
+
+/************************ own helper functions ***********************/
+
+/* This is called from the kernel bio layer.
+ */
+static
+void bio_callback(struct bio *bio)
+{
+	struct bio_aio_aspect *aio_a = bio->bi_private;
+	struct bio_brick *brick;
+	unsigned long flags;
+
+	CHECK_PTR(aio_a, err);
+	CHECK_PTR(aio_a->output, err);
+	brick = aio_a->output->brick;
+	CHECK_PTR(brick, err);
+
+	aio_a->status_code = bio->bi_error;
+
+	spin_lock_irqsave(&brick->lock, flags);
+	list_del(&aio_a->io_head);
+	list_add_tail(&aio_a->io_head, &brick->completed_list);
+	atomic_inc(&brick->completed_count);
+	spin_unlock_irqrestore(&brick->lock, flags);
+
+	wake_up_interruptible(&brick->response_event);
+	goto out_return;
+err:
+	XIO_FAT("cannot handle bio callback\n");
+out_return:;
+}
+
+/* Map from kernel address/length to struct page (if not already known),
+ * check alignment constraints, create bio from it.
+ * Return the length (may be smaller than requested).
+ */
+static
+int make_bio(struct bio_brick *brick,
+	void *data,
+	int len,
+	loff_t pos,
+	struct bio_aio_aspect *private,
+	struct bio **_bio)
+{
+	unsigned long long sector;
+	int sector_offset;
+	int data_offset;
+	int page_offset;
+	int page_len;
+	int bvec_count;
+	int rest_len = len;
+	int result_len = 0;
+	int status;
+	int i;
+	struct bio *bio = NULL;
+	struct block_device *bdev;
+
+	status = -EINVAL;
+	CHECK_PTR(brick, out);
+	bdev = brick->bdev;
+	CHECK_PTR(bdev, out);
+
+	if (unlikely(rest_len <= 0)) {
+		XIO_ERR("bad bio len %d\n", rest_len);
+		goto out;
+	}
+
+	sector = pos >> 9;		       /*  TODO: make dynamic */
+	sector_offset = pos & ((1 << 9) - 1);  /*  TODO: make dynamic */
+	data_offset = ((unsigned long)data) & ((1 << 9) - 1);  /*  TODO: make dynamic */
+
+	if (unlikely(sector_offset > 0)) {
+		XIO_ERR("odd sector offset %d\n", sector_offset);
+		goto out;
+	}
+	if (unlikely(sector_offset != data_offset)) {
+		XIO_ERR("bad alignment: sector_offset %d != data_offset %d\n", sector_offset, data_offset);
+		goto out;
+	}
+	if (unlikely(rest_len & ((1 << 9) - 1))) {
+		XIO_ERR("odd length %d\n", rest_len);
+		goto out;
+	}
+
+	page_offset = ((unsigned long)data) & (PAGE_SIZE-1);
+	page_len = rest_len + page_offset;
+	bvec_count = (page_len - 1) / PAGE_SIZE + 1;
+	if (bvec_count > brick->bvec_max) {
+		bvec_count = brick->bvec_max;
+	} else if (unlikely(bvec_count <= 0)) {
+		XIO_WRN("bvec_count=%d\n", bvec_count);
+		bvec_count = 1;
+	}
+
+	bio = bio_alloc(GFP_BRICK, bvec_count);
+	status = -ENOMEM;
+
+	for (i = 0; i < bvec_count && rest_len > 0; i++) {
+		struct page *page;
+		int this_rest = PAGE_SIZE - page_offset;
+		int this_len = rest_len;
+
+		if (this_len > this_rest)
+			this_len = this_rest;
+
+		page = brick_iomap(data, &page_offset, &this_len);
+		if (unlikely(!page)) {
+			XIO_ERR("cannot iomap() kernel address %p\n", data);
+			status = -EINVAL;
+			goto out;
+		}
+
+		bio->bi_io_vec[i].bv_page = page;
+		bio->bi_io_vec[i].bv_len = this_len;
+		bio->bi_io_vec[i].bv_offset = page_offset;
+
+		data += this_len;
+		rest_len -= this_len;
+		result_len += this_len;
+		page_offset = 0;
+	}
+
+	if (unlikely(rest_len != 0)) {
+		XIO_ERR("computation of bvec_count %d was wrong, diff=%d\n", bvec_count, rest_len);
+		status = -EINVAL;
+		goto out;
+	}
+
+	bio->bi_vcnt = i;
+	bio->bi_iter.bi_idx = 0;
+	bio->bi_iter.bi_size = result_len;
+	bio->bi_iter.bi_sector = sector;
+	bio->bi_bdev = bdev;
+	bio->bi_private = private;
+	bio->bi_end_io = bio_callback;
+	bio->bi_rw = 0; /*  must be filled in later */
+	status = result_len;
+
+out:
+	if (unlikely(status < 0)) {
+		XIO_ERR("error %d\n", status);
+		if (bio) {
+			bio_put(bio);
+			bio = NULL;
+		}
+	}
+	*_bio = bio;
+	return status;
+}
+
+/***************** own brick * input * output operations *****************/
+
+#define PRIO_INDEX(aio) ((aio)->io_prio + 1)
+
+static int bio_get_info(struct bio_output *output, struct xio_info *info)
+{
+	struct bio_brick *brick = output->brick;
+	struct inode *inode;
+	int status = -ENOENT;
+
+	if (unlikely(!brick->mf ||
+		     !brick->mf->mf_filp ||
+		     !brick->mf->mf_filp->f_mapping)) {
+		goto done;
+	}
+	inode = brick->mf->mf_filp->f_mapping->host;
+	if (unlikely(!inode))
+		goto done;
+
+	info->tf_align = 512;
+	info->tf_min_size = 512;
+	brick->total_size = i_size_read(inode);
+	info->current_size = brick->total_size;
+	XIO_DBG("determined device size = %lld\n", info->current_size);
+	status = 0;
+
+done:
+	return status;
+}
+
+static int bio_io_get(struct bio_output *output, struct aio_object *aio)
+{
+	struct bio_aio_aspect *aio_a;
+	int status = -EINVAL;
+
+	CHECK_PTR(output, done);
+	CHECK_PTR(output->brick, done);
+
+	if (aio->obj_initialized) {
+		obj_get(aio);
+		return aio->io_len;
+	}
+
+	aio_a = bio_aio_get_aspect(output->brick, aio);
+	CHECK_PTR(aio_a, done);
+	aio_a->output = output;
+	aio_a->bio = NULL;
+
+	if (!aio->io_data) { /*  buffered IO. */
+		if (unlikely(aio->io_len <= 0))
+			goto done;
+		status = -ENOMEM;
+		aio->io_data = brick_block_alloc(aio->io_pos, (aio_a->alloc_len = aio->io_len));
+		aio_a->do_dealloc = true;
+	}
+
+	status = make_bio(output->brick, aio->io_data, aio->io_len, aio->io_pos, aio_a, &aio_a->bio);
+	if (unlikely(status < 0 || !aio_a->bio)) {
+		XIO_ERR("could not create bio, status = %d\n", status);
+		goto done;
+	}
+
+	if (unlikely(aio->io_prio < XIO_PRIO_HIGH))
+		aio->io_prio = XIO_PRIO_HIGH;
+	else if (unlikely(aio->io_prio > XIO_PRIO_LOW))
+		aio->io_prio = XIO_PRIO_LOW;
+
+	aio->io_len = status;
+	obj_get_first(aio);
+	status = 0;
+
+done:
+	return status;
+}
+
+static
+void _bio_io_put(struct bio_output *output, struct aio_object *aio)
+{
+	struct bio_aio_aspect *aio_a;
+
+	aio->io_total_size = output->brick->total_size;
+
+	aio_a = bio_aio_get_aspect(output->brick, aio);
+	CHECK_PTR(aio_a, err);
+
+	if (likely(aio_a->bio)) {
+		bio_put(aio_a->bio);
+		aio_a->bio = NULL;
+	}
+	if (aio_a->do_dealloc) {
+		brick_block_free(aio->io_data, aio_a->alloc_len);
+		aio->io_data = NULL;
+	}
+	obj_free(aio);
+
+	goto out_return;
+err:
+	XIO_FAT("cannot work\n");
+out_return:;
+}
+
+#define BIO_AIO_PUT(output, aio)					\
+	({								\
+		if (obj_put(aio)) {					\
+			_bio_io_put(output, aio);			\
+		}							\
+	})
+
+static
+void bio_io_put(struct bio_output *output, struct aio_object *aio)
+{
+	BIO_AIO_PUT(output, aio);
+}
+
+static
+void _bio_io_io(struct bio_output *output, struct aio_object *aio, bool cork)
+{
+	struct bio_brick *brick = output->brick;
+	struct bio_aio_aspect *aio_a = bio_aio_get_aspect(output->brick, aio);
+	struct bio *bio;
+	unsigned long long latency;
+	unsigned long flags;
+	int rw;
+	int status = -EINVAL;
+
+	CHECK_PTR(aio_a, err);
+	bio = aio_a->bio;
+	CHECK_PTR(bio, err);
+
+	obj_get(aio);
+	atomic_inc(&brick->fly_count[PRIO_INDEX(aio)]);
+
+	bio_get(bio);
+
+	rw = aio->io_rw & 1;
+	if (brick->do_noidle && !cork)
+		rw |= REQ_NOIDLE;
+	if (!aio->io_skip_sync) {
+		if (brick->do_sync)
+			rw |= REQ_SYNC;
+	}
+
+	aio_a->start_stamp = cpu_clock(raw_smp_processor_id());
+	spin_lock_irqsave(&brick->lock, flags);
+	list_add_tail(&aio_a->io_head, &brick->submitted_list[rw & 1]);
+	spin_unlock_irqrestore(&brick->lock, flags);
+
+	bio->bi_rw = rw;
+	latency = TIME_STATS(
+		&timings[rw & 1],
+		submit_bio(rw, bio)
+		);
+
+	threshold_check(&bio_submit_threshold, latency);
+
+	status = 0;
+#ifdef BIO_EOPNOTSUPP /* missing since b25de9d6da49b1a8760a89672283128aa8c78345 */
+	if (unlikely(bio_flagged(bio, BIO_EOPNOTSUPP)))
+		status = -EOPNOTSUPP;
+#endif
+
+	if (likely(status >= 0))
+		goto done;
+
+	bio_put(bio);
+	atomic_dec(&brick->fly_count[PRIO_INDEX(aio)]);
+
+err:
+	XIO_ERR("IO error %d\n", status);
+	CHECKED_CALLBACK(aio, status, done);
+	atomic_dec(&xio_global_io_flying);
+
+done:;
+}
+
+static
+void bio_io_io(struct bio_output *output, struct aio_object *aio)
+{
+	CHECK_PTR(aio, fatal);
+
+	obj_get(aio);
+	atomic_inc(&xio_global_io_flying);
+
+	if (aio->io_prio == XIO_PRIO_LOW ||
+	    (aio->io_prio == XIO_PRIO_NORMAL && aio->io_rw)) {
+		struct bio_aio_aspect *aio_a = bio_aio_get_aspect(output->brick, aio);
+		struct bio_brick *brick = output->brick;
+		unsigned long flags;
+
+		spin_lock_irqsave(&brick->lock, flags);
+		list_add_tail(&aio_a->io_head, &brick->queue_list[PRIO_INDEX(aio)]);
+		atomic_inc(&brick->queue_count[PRIO_INDEX(aio)]);
+		spin_unlock_irqrestore(&brick->lock, flags);
+		brick->submitted = true;
+
+		wake_up_interruptible(&brick->submit_event);
+		goto out_return;
+	}
+
+	/*  realtime IO: start immediately */
+	_bio_io_io(output, aio, false);
+	BIO_AIO_PUT(output, aio);
+	goto out_return;
+fatal:
+	XIO_FAT("cannot handle aio %p on output %p\n", aio, output);
+out_return:;
+}
+
+static
+int bio_response_thread(void *data)
+{
+	struct bio_brick *brick = data;
+
+	XIO_INF("bio response thread has started on '%s'.\n", brick->brick_path);
+
+	for (;;) {
+		LIST_HEAD(tmp_list);
+		unsigned long flags;
+		int thr_limit;
+		int sleeptime;
+		int count;
+		int i;
+
+		thr_limit = bio_io_threshold[0].thr_limit;
+		if (bio_io_threshold[1].thr_limit < thr_limit)
+			thr_limit = bio_io_threshold[1].thr_limit;
+
+		sleeptime = HZ / 10;
+		if (thr_limit > 0) {
+			sleeptime = thr_limit / (1000000 * 2 / HZ);
+			if (unlikely(sleeptime < 2))
+				sleeptime = 2;
+		}
+
+		wait_event_interruptible_timeout(
+			brick->response_event,
+			atomic_read(&brick->completed_count) > 0,
+			sleeptime);
+
+		spin_lock_irqsave(&brick->lock, flags);
+		list_replace_init(&brick->completed_list, &tmp_list);
+		spin_unlock_irqrestore(&brick->lock, flags);
+
+		count = 0;
+		for (;;) {
+			struct list_head *tmp;
+			struct bio_aio_aspect *aio_a;
+			struct aio_object *aio;
+			unsigned long long latency;
+			int code;
+
+			if (list_empty(&tmp_list)) {
+				if (brick_thread_should_stop() &&
+				    atomic_read(&brick->fly_count[0]) +
+				    atomic_read(&brick->fly_count[1]) +
+				    atomic_read(&brick->fly_count[2]) <= 0)
+					goto done;
+				break;
+			}
+
+			tmp = tmp_list.next;
+			list_del_init(tmp);
+			atomic_dec(&brick->completed_count);
+
+			aio_a = container_of(tmp, struct bio_aio_aspect, io_head);
+			aio = aio_a->object;
+
+			latency = cpu_clock(raw_smp_processor_id()) - aio_a->start_stamp;
+			threshold_check(&bio_io_threshold[aio->io_rw & 1], latency);
+
+			code = aio_a->status_code;
+
+			if (code < 0) {
+				XIO_ERR("IO error %d\n", code);
+			} else {
+				aio_checksum(aio);
+				aio->io_flags |= AIO_UPTODATE;
+			}
+
+			SIMPLE_CALLBACK(aio, code);
+
+			atomic_dec(&brick->fly_count[PRIO_INDEX(aio)]);
+			atomic_inc(&brick->total_completed_count[PRIO_INDEX(aio)]);
+			count++;
+
+			if (likely(aio_a->bio))
+				bio_put(aio_a->bio);
+			BIO_AIO_PUT(aio_a->output, aio);
+
+			atomic_dec(&xio_global_io_flying);
+		}
+
+		/* Try to detect slow requests as early as possible,
+		 * even before they have completed.
+		 */
+		for (i = 0; i < 2; i++) {
+			unsigned long long eldest = 0;
+
+			spin_lock_irqsave(&brick->lock, flags);
+			if (!list_empty(&brick->submitted_list[i])) {
+				struct bio_aio_aspect *aio_a;
+
+				aio_a = container_of(brick->submitted_list[i].next, struct bio_aio_aspect, io_head);
+				eldest = aio_a->start_stamp;
+			}
+			spin_unlock_irqrestore(&brick->lock, flags);
+
+			if (eldest)
+				threshold_check(&bio_io_threshold[i], cpu_clock(raw_smp_processor_id()) - eldest);
+		}
+
+		if (count) {
+			brick->submitted = true;
+			wake_up_interruptible(&brick->submit_event);
+		}
+	}
+done:
+	XIO_INF("bio response thread has stopped.\n");
+	return 0;
+}
+
+static
+bool _bg_should_run(struct bio_brick *brick)
+{
+	return (atomic_read(&brick->queue_count[2]) > 0 &&
+		atomic_read(&brick->fly_count[0]) + atomic_read(&brick->fly_count[1]) <= brick->bg_threshold &&
+		(brick->bg_maxfly <= 0 || atomic_read(&brick->fly_count[2]) < brick->bg_maxfly));
+}
+
+static
+int bio_submit_thread(void *data)
+{
+	struct bio_brick *brick = data;
+
+	XIO_INF("bio submit thread has started on '%s'.\n", brick->brick_path);
+
+	while (!brick_thread_should_stop()) {
+		int prio;
+
+		wait_event_interruptible_timeout(
+			brick->submit_event,
+			brick->submitted,
+			HZ / 2);
+
+		brick->submitted = false;
+
+		for (prio = 0; prio < XIO_PRIO_NR; prio++) {
+			LIST_HEAD(tmp_list);
+			unsigned long flags;
+
+			if (prio == XIO_PRIO_NR-1 && !_bg_should_run(brick))
+				break;
+
+			spin_lock_irqsave(&brick->lock, flags);
+			list_replace_init(&brick->queue_list[prio], &tmp_list);
+			spin_unlock_irqrestore(&brick->lock, flags);
+
+			while (!list_empty(&tmp_list)) {
+				struct list_head *tmp = tmp_list.next;
+				struct bio_aio_aspect *aio_a;
+				struct aio_object *aio;
+				bool cork;
+
+				list_del_init(tmp);
+
+				aio_a = container_of(tmp, struct bio_aio_aspect, io_head);
+				aio = aio_a->object;
+				if (unlikely(!aio)) {
+					XIO_ERR("invalid aio\n");
+					continue;
+				}
+
+				atomic_dec(&brick->queue_count[PRIO_INDEX(aio)]);
+				cork = atomic_read(&brick->queue_count[PRIO_INDEX(aio)]) > 0;
+
+				_bio_io_io(aio_a->output, aio, cork);
+
+				BIO_AIO_PUT(aio_a->output, aio);
+			}
+		}
+	}
+
+	XIO_INF("bio submit thread has stopped.\n");
+	return 0;
+}
+
+static int bio_switch(struct bio_brick *brick)
+{
+	int status = 0;
+
+	if (brick->power.button) {
+		if (brick->power.on_led)
+			goto done;
+
+		xio_set_power_off_led((void *)brick, false);
+
+		if (!brick->bdev) {
+			static int index;
+			const char *path = brick->brick_path;
+			int flags = O_RDWR | O_EXCL | O_LARGEFILE;
+			struct address_space *mapping;
+			struct inode *inode = NULL;
+			struct request_queue *q;
+
+			brick->mf = mapfree_get(path, flags);
+			if (unlikely(!brick->mf || !brick->mf->mf_filp)) {
+				status = -ENOENT;
+				XIO_ERR("cannot open file '%s'\n", path);
+				goto done;
+			}
+			mapfree_pages(brick->mf, -1);
+			mapping = brick->mf->mf_filp->f_mapping;
+			if (likely(mapping))
+				inode = mapping->host;
+			if (unlikely(!mapping || !inode)) {
+				XIO_ERR("internal problem with '%s'\n", path);
+				status = -EINVAL;
+				goto done;
+			}
+			if (unlikely(!S_ISBLK(inode->i_mode) || !inode->i_bdev)) {
+				XIO_ERR("sorry, '%s' is not a block device\n", path);
+				status = -ENODEV;
+				goto done;
+			}
+
+			mapping_set_gfp_mask(mapping, mapping_gfp_mask(mapping) & ~(__GFP_IO | __GFP_FS));
+
+			q = bdev_get_queue(inode->i_bdev);
+			if (unlikely(!q)) {
+				XIO_ERR("internal queue '%s' does not exist\n", path);
+				status = -EINVAL;
+				goto done;
+			}
+
+			XIO_INF("'%s' ra_pages OLD=%lu NEW=%d\n",
+				path,
+				q->backing_dev_info.ra_pages,
+				brick->ra_pages);
+			q->backing_dev_info.ra_pages = brick->ra_pages;
+
+			brick->bvec_max = queue_max_hw_sectors(q) >> (PAGE_SHIFT - 9);
+			if (brick->bvec_max > BIO_MAX_PAGES)
+				brick->bvec_max = BIO_MAX_PAGES;
+			else if (brick->bvec_max <= 1)
+				brick->bvec_max = 1;
+			brick->total_size = i_size_read(inode);
+			XIO_INF("'%s' size=%lld bvec_max=%d\n",
+				 path, brick->total_size, brick->bvec_max);
+
+			brick->response_thread = brick_thread_create(bio_response_thread,
+				brick,
+				"xio_bio_r%d",
+				index);
+			brick->submit_thread = brick_thread_create(bio_submit_thread, brick, "xio_bio_s%d", index);
+			status = -ENOMEM;
+			if (likely(brick->submit_thread && brick->response_thread)) {
+				brick->bdev = inode->i_bdev;
+				brick->mode_ptr = &brick->mf->mf_mode;
+				index++;
+				status = 0;
+			}
+		}
+	}
+
+	xio_set_power_on_led((void *)brick, brick->power.button && brick->bdev != NULL);
+
+done:
+	if (status < 0 || !brick->power.button) {
+		if (brick->submit_thread) {
+			brick_thread_stop(brick->submit_thread);
+			brick->submit_thread = NULL;
+		}
+		if (brick->response_thread) {
+			brick_thread_stop(brick->response_thread);
+			brick->response_thread = NULL;
+		}
+		if (brick->mf) {
+			mapfree_put(brick->mf);
+			brick->mf = NULL;
+		}
+		brick->mode_ptr = NULL;
+		brick->bdev = NULL;
+		if (!brick->power.button) {
+			xio_set_power_off_led((void *)brick, true);
+			brick->total_size = 0;
+		}
+	}
+	return status;
+}
+
+/*************** informational * statistics **************/
+
+static noinline
+char *bio_statistics(struct bio_brick *brick, int verbose)
+{
+	char *res = brick_string_alloc(4096);
+	int pos = 0;
+
+	pos += report_timing(&timings[0], res + pos, 4096 - pos);
+	pos += report_timing(&timings[1], res + pos, 4096 - pos);
+
+	snprintf(res + pos, 4096 - pos,
+		 "total completed[0] = %d completed[1] = %d completed[2] = %d | queued[0] = %d queued[1] = %d queued[2] = %d flying[0] = %d flying[1] = %d flying[2] = %d completing = %d\n",
+		 atomic_read(&brick->total_completed_count[0]),
+		 atomic_read(&brick->total_completed_count[1]),
+		 atomic_read(&brick->total_completed_count[2]),
+		 atomic_read(&brick->fly_count[0]),
+		 atomic_read(&brick->queue_count[0]),
+		 atomic_read(&brick->queue_count[1]),
+		 atomic_read(&brick->queue_count[2]),
+		 atomic_read(&brick->fly_count[1]),
+		 atomic_read(&brick->fly_count[2]),
+		 atomic_read(&brick->completed_count));
+
+	return res;
+}
+
+static noinline
+void bio_reset_statistics(struct bio_brick *brick)
+{
+	atomic_set(&brick->total_completed_count[0], 0);
+	atomic_set(&brick->total_completed_count[1], 0);
+	atomic_set(&brick->total_completed_count[2], 0);
+}
+
+/*************** object * aspect constructors * destructors **************/
+
+static int bio_aio_aspect_init_fn(struct generic_aspect *_ini)
+{
+	struct bio_aio_aspect *ini = (void *)_ini;
+
+	INIT_LIST_HEAD(&ini->io_head);
+	return 0;
+}
+
+static void bio_aio_aspect_exit_fn(struct generic_aspect *_ini)
+{
+	struct bio_aio_aspect *ini = (void *)_ini;
+
+	(void)ini;
+}
+
+XIO_MAKE_STATICS(bio);
+
+/********************* brick constructors * destructors *******************/
+
+static int bio_brick_construct(struct bio_brick *brick)
+{
+	spin_lock_init(&brick->lock);
+	INIT_LIST_HEAD(&brick->queue_list[0]);
+	INIT_LIST_HEAD(&brick->queue_list[1]);
+	INIT_LIST_HEAD(&brick->queue_list[2]);
+	INIT_LIST_HEAD(&brick->submitted_list[0]);
+	INIT_LIST_HEAD(&brick->submitted_list[1]);
+	INIT_LIST_HEAD(&brick->completed_list);
+	init_waitqueue_head(&brick->submit_event);
+	init_waitqueue_head(&brick->response_event);
+	return 0;
+}
+
+static int bio_brick_destruct(struct bio_brick *brick)
+{
+	return 0;
+}
+
+static int bio_output_construct(struct bio_output *output)
+{
+	return 0;
+}
+
+static int bio_output_destruct(struct bio_output *output)
+{
+	return 0;
+}
+
+/************************ static structs ***********************/
+
+static struct bio_brick_ops bio_brick_ops = {
+	.brick_switch = bio_switch,
+	.brick_statistics = bio_statistics,
+	.reset_statistics = bio_reset_statistics,
+};
+
+static struct bio_output_ops bio_output_ops = {
+	.xio_get_info = bio_get_info,
+	.aio_get = bio_io_get,
+	.aio_put = bio_io_put,
+	.aio_io = bio_io_io,
+};
+
+const struct bio_input_type bio_input_type = {
+	.type_name = "bio_input",
+	.input_size = sizeof(struct bio_input),
+};
+
+static const struct bio_input_type *bio_input_types[] = {
+	&bio_input_type,
+};
+
+const struct bio_output_type bio_output_type = {
+	.type_name = "bio_output",
+	.output_size = sizeof(struct bio_output),
+	.master_ops = &bio_output_ops,
+	.output_construct = &bio_output_construct,
+	.output_destruct = &bio_output_destruct,
+};
+
+static const struct bio_output_type *bio_output_types[] = {
+	&bio_output_type,
+};
+
+const struct bio_brick_type bio_brick_type = {
+	.type_name = "bio_brick",
+	.brick_size = sizeof(struct bio_brick),
+	.max_inputs = 0,
+	.max_outputs = 1,
+	.master_ops = &bio_brick_ops,
+	.aspect_types = bio_aspect_types,
+	.default_input_types = bio_input_types,
+	.default_output_types = bio_output_types,
+	.brick_construct = &bio_brick_construct,
+	.brick_destruct = &bio_brick_destruct,
+};
+
+/***************** module init stuff ************************/
+
+int __init init_xio_bio(void)
+{
+	XIO_INF("init_bio()\n");
+	_bio_brick_type = (void *)&bio_brick_type;
+	return bio_register_brick_type();
+}
+
+void exit_xio_bio(void)
+{
+	XIO_INF("exit_bio()\n");
+	bio_unregister_brick_type();
+}
include/linux/xio/xio_bio.h new file +85 -0

85 lines not on this page · raw message

Signature · 1 line
2.6.4
#18 lore raw

RFC 18/31 mars: add new module xio_sio

In reply to Thomas Schoebel-Theuer
---
 drivers/staging/mars/xio_bricks/xio_sio.c | 571 ++++++++++++++++++++++++++++++
 include/linux/xio/xio_sio.h               |  68 ++++
 2 files changed, 639 insertions(+)
 create mode 100644 drivers/staging/mars/xio_bricks/xio_sio.c
 create mode 100644 include/linux/xio/xio_sio.h
drivers/staging/mars/xio_bricks/xio_sio.c new file +571 -0
new file mode 100644
index 0000000..5822847
@@ -0,0 +1,571 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/version.h>
+#include <linux/string.h>
+#include <linux/list.h>
+#include <linux/types.h>
+#include <linux/blkdev.h>
+#include <linux/highmem.h>
+#include <linux/spinlock.h>
+#include <linux/wait.h>
+
+#include <linux/xio/xio.h>
+
+/************************ own type definitions ***********************/
+
+#include <linux/xio/xio_sio.h>
+
+/***************** own brick * input * output operations *****************/
+
+static int sio_io_get(struct sio_output *output, struct aio_object *aio)
+{
+	struct file *file;
+
+	if (unlikely(!output->brick->power.on_led))
+		return -EBADFD;
+
+	if (aio->obj_initialized) {
+		obj_get(aio);
+		return aio->io_len;
+	}
+
+	file = output->mf->mf_filp;
+	if (file) {
+		loff_t total_size = i_size_read(file->f_mapping->host);
+
+		aio->io_total_size = total_size;
+		/* Only check reads.
+		 * Writes behind EOF are always allowed (sparse files)
+		 */
+		if (!aio->io_may_write) {
+			loff_t len = total_size - aio->io_pos;
+
+			if (unlikely(len <= 0)) {
+				/* Special case: allow reads starting _exactly_ at EOF when a timeout is specified.
+				 */
+				if (len < 0 || aio->io_timeout <= 0) {
+					XIO_DBG("ENODATA %lld\n", len);
+					return -ENODATA;
+				}
+			}
+			/*  Shorten below EOF, but allow special case */
+			if (aio->io_len > len && len > 0)
+				aio->io_len = len;
+		}
+	}
+
+	/* Buffered IO.
+	 */
+	if (!aio->io_data) {
+		struct sio_aio_aspect *aio_a = sio_aio_get_aspect(output->brick, aio);
+
+		if (unlikely(!aio_a))
+			return -EILSEQ;
+		if (unlikely(aio->io_len <= 0)) {
+			XIO_ERR("bad io_len = %d\n", aio->io_len);
+			return -ENOMEM;
+		}
+		aio->io_data = brick_block_alloc(aio->io_pos, (aio_a->alloc_len = aio->io_len));
+		aio_a->do_dealloc = true;
+		/* atomic_inc(&output->total_alloc_count); */
+		/* atomic_inc(&output->alloc_count); */
+	}
+
+	obj_get_first(aio);
+	return aio->io_len;
+}
+
+static void sio_io_put(struct sio_output *output, struct aio_object *aio)
+{
+	struct file *file;
+	struct sio_aio_aspect *aio_a;
+
+	if (!obj_put(aio))
+		goto out_return;
+	file = output->mf->mf_filp;
+	aio->io_total_size = i_size_read(file->f_mapping->host);
+
+	aio_a = sio_aio_get_aspect(output->brick, aio);
+	if (aio_a && aio_a->do_dealloc) {
+		brick_block_free(aio->io_data, aio_a->alloc_len);
+		/* atomic_dec(&output->alloc_count); */
+	}
+
+	obj_free(aio);
+out_return:;
+}
+
+static
+int write_aops(struct sio_output *output, struct aio_object *aio)
+{
+	struct file *file = output->mf->mf_filp;
+	loff_t pos = aio->io_pos;
+	void *data = aio->io_data;
+	int  len = aio->io_len;
+	int ret = 0;
+
+	mm_segment_t oldfs;
+
+	oldfs = get_fs();
+	set_fs(get_ds());
+	ret = vfs_write(file, data, len, &pos);
+	set_fs(oldfs);
+	return ret;
+}
+
+static
+int read_aops(struct sio_output *output, struct aio_object *aio)
+{
+	loff_t pos = aio->io_pos;
+	int len = aio->io_len;
+	int ret;
+
+	mm_segment_t oldfs;
+
+	oldfs = get_fs();
+	set_fs(get_ds());
+	ret = vfs_read(output->mf->mf_filp, aio->io_data, len, &pos);
+	set_fs(oldfs);
+
+	if (unlikely(ret < 0))
+		XIO_ERR("%p %p status=%d\n", output, aio, ret);
+	return ret;
+}
+
+static void sync_file(struct sio_output *output)
+{
+	struct file *file = output->mf->mf_filp;
+	int ret;
+
+#if defined(S_BIAS) || (defined(RHEL_MAJOR) && (RHEL_MAJOR < 7))
+	ret = vfs_fsync(file, file->f_path.dentry, 1);
+#else
+	ret = vfs_fsync(file, 1);
+#endif
+	if (unlikely(ret))
+		XIO_ERR("syncing pages failed: %d\n", ret);
+	goto out_return;
+out_return:;
+}
+
+static
+void _complete(struct sio_output *output, struct aio_object *aio, int err)
+{
+	obj_check(aio);
+
+	if (err < 0) {
+		XIO_ERR("IO error %d at pos=%lld len=%d (aio=%p io_data=%p)\n",
+			err,
+			aio->io_pos,
+			aio->io_len,
+			aio,
+			aio->io_data);
+	} else {
+		aio_checksum(aio);
+		aio->io_flags |= AIO_UPTODATE;
+	}
+
+	CHECKED_CALLBACK(aio, err, err_found);
+
+done:
+	sio_io_put(output, aio);
+
+	atomic_dec(&output->work_count);
+	atomic_dec(&xio_global_io_flying);
+	goto out_return;
+err_found:
+	XIO_FAT("giving up...\n");
+	goto done;
+out_return:;
+}
+
+/* This is called by the threads
+ */
+static
+void _sio_io_io(struct sio_threadinfo *tinfo, struct aio_object *aio)
+{
+	struct sio_output *output = tinfo->output;
+	bool barrier = false;
+	int status;
+
+	obj_check(aio);
+
+	atomic_inc(&tinfo->fly_count);
+
+	if (unlikely(!output->mf || !output->mf->mf_filp)) {
+		status = -EINVAL;
+		goto done;
+	}
+
+	if (barrier) {
+		XIO_INF("got barrier request\n");
+		sync_file(output);
+	}
+
+	if (aio->io_rw == READ) {
+		status = read_aops(output, aio);
+	} else {
+		status = write_aops(output, aio);
+		if (barrier || output->brick->o_fdsync)
+			sync_file(output);
+	}
+
+	mapfree_set(output->mf, aio->io_pos, aio->io_pos + aio->io_len);
+
+done:
+	_complete(output, aio, status);
+
+	atomic_dec(&tinfo->fly_count);
+}
+
+/* This is called from outside
+ */
+static
+void sio_io_io(struct sio_output *output, struct aio_object *aio)
+{
+	int index;
+	struct sio_threadinfo *tinfo;
+	struct sio_aio_aspect *aio_a;
+	unsigned long flags;
+
+	obj_check(aio);
+
+	aio_a = sio_aio_get_aspect(output->brick, aio);
+	if (unlikely(!aio_a)) {
+		XIO_FAT("cannot get aspect\n");
+		SIMPLE_CALLBACK(aio, -EINVAL);
+		goto out_return;
+	}
+
+	if (unlikely(!output->brick->power.on_led)) {
+		SIMPLE_CALLBACK(aio, -EBADFD);
+		goto out_return;
+	}
+
+	atomic_inc(&xio_global_io_flying);
+	atomic_inc(&output->work_count);
+	obj_get(aio);
+
+	mapfree_set(output->mf, aio->io_pos, -1);
+
+	index = 0;
+	if (aio->io_rw == READ) {
+		spin_lock_irqsave(&output->g_lock, flags);
+		index = output->index++;
+		spin_unlock_irqrestore(&output->g_lock, flags);
+		index = (index % WITH_THREAD) + 1;
+	}
+
+	tinfo = &output->tinfo[index];
+
+	atomic_inc(&tinfo->total_count);
+	atomic_inc(&tinfo->queue_count);
+
+	spin_lock_irqsave(&tinfo->lock, flags);
+	list_add_tail(&aio_a->io_head, &tinfo->aio_list);
+	spin_unlock_irqrestore(&tinfo->lock, flags);
+
+	wake_up_interruptible(&tinfo->event);
+out_return:;
+}
+
+static int sio_thread(void *data)
+{
+	struct sio_threadinfo *tinfo = data;
+
+	XIO_INF("sio thread has started.\n");
+	/* set_user_nice(current, -20); */
+
+	while (!brick_thread_should_stop()) {
+		struct list_head *tmp = NULL;
+		struct aio_object *aio;
+		struct sio_aio_aspect *aio_a;
+		unsigned long flags;
+
+		wait_event_interruptible_timeout(
+			tinfo->event,
+			!list_empty(&tinfo->aio_list) || brick_thread_should_stop(),
+			HZ);
+
+		tinfo->last_jiffies = jiffies;
+
+		spin_lock_irqsave(&tinfo->lock, flags);
+
+		if (!list_empty(&tinfo->aio_list)) {
+			tmp = tinfo->aio_list.next;
+			list_del_init(tmp);
+			atomic_dec(&tinfo->queue_count);
+		}
+
+		spin_unlock_irqrestore(&tinfo->lock, flags);
+
+		if (!tmp)
+			continue;
+
+		aio_a = container_of(tmp, struct sio_aio_aspect, io_head);
+		aio = aio_a->object;
+		_sio_io_io(tinfo, aio);
+	}
+
+	XIO_INF("sio thread has stopped.\n");
+	return 0;
+}
+
+static int sio_get_info(struct sio_output *output, struct xio_info *info)
+{
+	struct file *file = output->mf->mf_filp;
+
+	if (unlikely(!file || !file->f_mapping || !file->f_mapping->host))
+		return -EINVAL;
+
+	info->tf_align = 1;
+	info->tf_min_size = 1;
+	info->current_size = i_size_read(file->f_mapping->host);
+	XIO_DBG("determined file size = %lld\n", info->current_size);
+	return 0;
+}
+
+/*************** informational * statistics **************/
+
+static noinline
+char *sio_statistics(struct sio_brick *brick, int verbose)
+{
+	struct sio_output *output = brick->outputs[0];
+	char *res = brick_string_alloc(1024);
+	int queue_sum = 0;
+	int fly_sum = 0;
+	int total_sum = 0;
+	int i;
+
+	for (i = 1; i <= WITH_THREAD; i++) {
+		struct sio_threadinfo *tinfo = &output->tinfo[i];
+
+		queue_sum += atomic_read(&tinfo->queue_count);
+		fly_sum += atomic_read(&tinfo->fly_count);
+		total_sum += atomic_read(&tinfo->total_count);
+	}
+
+	snprintf(res, 1024,
+		 "queued read = %d write = %d flying read = %d write = %d total  read = %d write = %d\n",
+		 queue_sum, atomic_read(&output->tinfo[0].queue_count),
+		 fly_sum,   atomic_read(&output->tinfo[0].fly_count),
+		 total_sum, atomic_read(&output->tinfo[0].total_count)
+		);
+	return res;
+}
+
+static noinline
+void sio_reset_statistics(struct sio_brick *brick)
+{
+	struct sio_output *output = brick->outputs[0];
+	int i;
+
+	for (i = 0; i <= WITH_THREAD; i++) {
+		struct sio_threadinfo *tinfo = &output->tinfo[i];
+
+		atomic_set(&tinfo->total_count, 0);
+	}
+}
+
+/*************** object * aspect constructors * destructors **************/
+
+static int sio_aio_aspect_init_fn(struct generic_aspect *_ini)
+{
+	struct sio_aio_aspect *ini = (void *)_ini;
+
+	INIT_LIST_HEAD(&ini->io_head);
+	return 0;
+}
+
+static void sio_aio_aspect_exit_fn(struct generic_aspect *_ini)
+{
+	struct sio_aio_aspect *ini = (void *)_ini;
+
+	(void)ini;
+	CHECK_HEAD_EMPTY(&ini->io_head);
+}
+
+XIO_MAKE_STATICS(sio);
+
+/********************* brick constructors * destructors *******************/
+
+static int sio_brick_construct(struct sio_brick *brick)
+{
+	return 0;
+}
+
+static int sio_switch(struct sio_brick *brick)
+{
+	static int sio_nr;
+	struct sio_output *output = brick->outputs[0];
+	const char *path = output->brick->brick_path;
+	int status = 0;
+
+	if (brick->power.button) {
+		int flags = O_CREAT | O_RDWR | O_LARGEFILE;
+		int index;
+
+		if (brick->power.on_led)
+			goto done;
+
+		if (brick->o_direct) {
+			flags |= O_DIRECT;
+			XIO_INF("using O_DIRECT on %s\n", path);
+		}
+
+		xio_set_power_off_led((void *)brick, false);
+
+		output->mf = mapfree_get(path, flags);
+		if (unlikely(IS_ERR(output->mf))) {
+			XIO_ERR("could not open file = '%s' flags = %d\n", path, flags);
+			status = -ENOENT;
+			goto done;
+		}
+
+		output->index = 0;
+		for (index = 0; index <= WITH_THREAD; index++) {
+			struct sio_threadinfo *tinfo = &output->tinfo[index];
+
+			tinfo->last_jiffies = jiffies;
+			tinfo->thread = brick_thread_create(sio_thread, tinfo, "xio_sio%d", sio_nr++);
+			if (unlikely(!tinfo->thread)) {
+				XIO_ERR("cannot create thread\n");
+				status = -ENOENT;
+				goto done;
+			}
+		}
+		xio_set_power_on_led((void *)brick, true);
+	}
+done:
+	if (unlikely(status < 0) || !brick->power.button) {
+		int index;
+		int count;
+
+		xio_set_power_on_led((void *)brick, false);
+		for (;;) {
+			count = atomic_read(&output->work_count);
+			if (count <= 0)
+				break;
+			XIO_DBG("working on %d requests\n", count);
+			brick_msleep(1000);
+		}
+		for (index = 0; index <= WITH_THREAD; index++) {
+			struct sio_threadinfo *tinfo = &output->tinfo[index];
+
+			if (!tinfo->thread)
+				continue;
+			XIO_DBG("stopping thread %d\n", index);
+			brick_thread_stop(tinfo->thread);
+			tinfo->thread = NULL;
+		}
+		if (output->mf) {
+			XIO_DBG("closing file\n");
+			mapfree_put(output->mf);
+			output->mf = NULL;
+		}
+		xio_set_power_off_led((void *)brick, true);
+	}
+	return status;
+}
+
+static int sio_output_construct(struct sio_output *output)
+{
+	int index;
+
+	spin_lock_init(&output->g_lock);
+	for (index = 0; index <= WITH_THREAD; index++) {
+		struct sio_threadinfo *tinfo = &output->tinfo[index];
+
+		tinfo->output = output;
+		spin_lock_init(&tinfo->lock);
+		init_waitqueue_head(&tinfo->event);
+		INIT_LIST_HEAD(&tinfo->aio_list);
+	}
+
+	return 0;
+}
+
+static int sio_output_destruct(struct sio_output *output)
+{
+	return 0;
+}
+
+/************************ static structs ***********************/
+
+static struct sio_brick_ops sio_brick_ops = {
+	.brick_switch = sio_switch,
+	.brick_statistics = sio_statistics,
+	.reset_statistics = sio_reset_statistics,
+};
+
+static struct sio_output_ops sio_output_ops = {
+	.aio_get = sio_io_get,
+	.aio_put = sio_io_put,
+	.aio_io = sio_io_io,
+	.xio_get_info = sio_get_info,
+};
+
+const struct sio_input_type sio_input_type = {
+	.type_name = "sio_input",
+	.input_size = sizeof(struct sio_input),
+};
+
+static const struct sio_input_type *sio_input_types[] = {
+	&sio_input_type,
+};
+
+const struct sio_output_type sio_output_type = {
+	.type_name = "sio_output",
+	.output_size = sizeof(struct sio_output),
+	.master_ops = &sio_output_ops,
+	.output_construct = &sio_output_construct,
+	.output_destruct = &sio_output_destruct,
+};
+
+static const struct sio_output_type *sio_output_types[] = {
+	&sio_output_type,
+};
+
+const struct sio_brick_type sio_brick_type = {
+	.type_name = "sio_brick",
+	.brick_size = sizeof(struct sio_brick),
+	.max_inputs = 0,
+	.max_outputs = 1,
+	.master_ops = &sio_brick_ops,
+	.aspect_types = sio_aspect_types,
+	.default_input_types = sio_input_types,
+	.default_output_types = sio_output_types,
+	.brick_construct = &sio_brick_construct,
+};
+
+/***************** module init stuff ************************/
+
+int __init init_xio_sio(void)
+{
+	XIO_INF("init_sio()\n");
+	_sio_brick_type = (void *)&sio_brick_type;
+	return sio_register_brick_type();
+}
+
+void exit_xio_sio(void)
+{
+	XIO_INF("exit_sio()\n");
+	sio_unregister_brick_type();
+}
include/linux/xio/xio_sio.h new file +68 -0

68 lines not on this page · raw message

Signature · 1 line
2.6.4
#19 lore raw

RFC 19/31 mars: add new module xio_client

In reply to Thomas Schoebel-Theuer
---
 drivers/staging/mars/xio_bricks/xio_client.c | 1055 ++++++++++++++++++++++++++
 include/linux/xio/xio_client.h               |  105 +++
 2 files changed, 1160 insertions(+)
 create mode 100644 drivers/staging/mars/xio_bricks/xio_client.c
 create mode 100644 include/linux/xio/xio_client.h
drivers/staging/mars/xio_bricks/xio_client.c new file +1055 -0
new file mode 100644
index 0000000..6fdc261
@@ -0,0 +1,1055 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/string.h>
+#include <linux/jiffies.h>
+
+#include <linux/xio/xio.h>
+
+/************************ own type definitions ***********************/
+
+#include <linux/xio/xio_client.h>
+
+#define CLIENT_HASH_MAX			(PAGE_SIZE / sizeof(struct list_head))
+
+int xio_client_abort = 10;
+
+int max_client_channels = 1;
+
+int max_client_bulk = 16;
+
+/************************ own helper functions ***********************/
+
+static int thread_count;
+
+static
+void _do_resubmit(struct client_channel *ch)
+{
+	struct client_output *output = ch->output;
+	unsigned long flags;
+
+	spin_lock_irqsave(&output->lock, flags);
+	if (!list_empty(&ch->wait_list)) {
+		struct list_head *first = ch->wait_list.next;
+		struct list_head *last = ch->wait_list.prev;
+		struct list_head *old_start = output->aio_list.next;
+
+#define list_connect __list_del /*  the original routine has a misleading name: in reality it is more general */
+		list_connect(&output->aio_list, first);
+		list_connect(last, old_start);
+		INIT_LIST_HEAD(&ch->wait_list);
+	}
+	spin_unlock_irqrestore(&output->lock, flags);
+}
+
+static
+void _kill_thread(struct client_threadinfo *ti, const char *name)
+{
+	struct task_struct *thread = ti->thread;
+
+	if (thread) {
+		XIO_DBG("stopping %s thread\n", name);
+		ti->thread = NULL;
+		brick_thread_stop(thread);
+	}
+}
+
+static
+void _kill_channel(struct client_channel *ch)
+{
+	XIO_DBG("channel = %p\n", ch);
+	if (xio_socket_is_alive(&ch->socket)) {
+		XIO_DBG("shutdown socket\n");
+		xio_shutdown_socket(&ch->socket);
+	}
+	_kill_thread(&ch->receiver, "receiver");
+	if (ch->is_open) {
+		XIO_DBG("close socket\n");
+		xio_put_socket(&ch->socket);
+	}
+	ch->recv_error = 0;
+	ch->is_used = false;
+	ch->is_open = false;
+	ch->is_connected = false;
+	/* Re-Submit any waiting requests
+	 */
+	_do_resubmit(ch);
+}
+
+static inline
+void _kill_all_channels(struct client_bundle *bundle)
+{
+	int i;
+
+	/*  first pass: shutdown in parallel without waiting */
+	for (i = 0; i < MAX_CLIENT_CHANNELS; i++) {
+		struct client_channel *ch = &bundle->channel[i];
+
+		if (xio_socket_is_alive(&ch->socket)) {
+			XIO_DBG("shutdown socket %d\n", i);
+			xio_shutdown_socket(&ch->socket);
+		}
+	}
+	/*  separate pass (may wait) */
+	for (i = 0; i < MAX_CLIENT_CHANNELS; i++)
+		_kill_channel(&bundle->channel[i]);
+}
+
+static int receiver_thread(void *data);
+
+static
+int _setup_channel(struct client_bundle *bundle, int ch_nr)
+{
+	struct client_channel *ch = &bundle->channel[ch_nr];
+	struct sockaddr_storage src_sockaddr;
+	struct sockaddr_storage dst_sockaddr;
+	int status;
+
+	ch->ch_nr = ch_nr;
+	if (unlikely(ch->receiver.thread)) {
+		XIO_WRN("receiver thread %d unexpectedly not dead\n", ch_nr);
+		_kill_thread(&ch->receiver, "receiver");
+	}
+
+	status = xio_create_sockaddr(&src_sockaddr, my_id());
+	if (unlikely(status < 0)) {
+		XIO_DBG("no src sockaddr, status = %d\n", status);
+		goto done;
+	}
+
+	status = xio_create_sockaddr(&dst_sockaddr, bundle->host);
+	if (unlikely(status < 0)) {
+		XIO_DBG("no dst sockaddr, status = %d\n", status);
+		goto done;
+	}
+
+	status = xio_create_socket(&ch->socket, &src_sockaddr, &dst_sockaddr);
+	if (unlikely(status < 0)) {
+		XIO_DBG("no socket, status = %d\n", status);
+		goto really_done;
+	}
+	ch->socket.s_shutdown_on_err = true;
+	ch->socket.s_send_abort = xio_client_abort;
+	ch->socket.s_recv_abort = xio_client_abort;
+	ch->is_open = true;
+
+	ch->receiver.thread = brick_thread_create(receiver_thread,
+		ch,
+		"xio_receiver%d.%d.%d",
+		bundle->thread_count,
+		ch_nr,
+		ch->thread_count++);
+	if (unlikely(!ch->receiver.thread)) {
+		XIO_ERR("cannot start receiver thread %d, status = %d\n", ch_nr, status);
+		status = -ENOENT;
+		goto done;
+	}
+	ch->is_used = true;
+
+done:
+	if (status < 0) {
+		XIO_INF("cannot connect channel %d to remote host '%s' (status = %d) -- retrying\n",
+			 ch_nr,
+			 bundle->host ? bundle->host : "NULL",
+			 status);
+		_kill_channel(ch);
+	}
+
+really_done:
+	return status;
+}
+
+static
+void _kill_bundle(struct client_bundle *bundle)
+{
+	XIO_DBG("\n");
+	_kill_thread(&bundle->sender, "sender");
+	_kill_all_channels(bundle);
+}
+
+static
+void _maintain_bundle(struct client_bundle *bundle)
+{
+	int i;
+
+	/* Re-open _any_ failed channel, even old ones.
+	 * Reason: the number of channels might change during operation.
+	 */
+	for (i = 0; i < MAX_CLIENT_CHANNELS; i++) {
+		struct client_channel *ch = &bundle->channel[i];
+
+		if (!ch->is_used ||
+		    (!ch->recv_error && xio_socket_is_alive(&ch->socket)))
+			continue;
+
+		XIO_DBG("killing channel %d\n", i);
+		_kill_channel(ch);
+		/* Re-setup including connect optiona is done later.
+		 */
+	}
+}
+
+static
+struct client_channel *_get_channel(struct client_bundle *bundle, int min_channel, int max_channel)
+{
+	struct client_channel *res;
+	long best_space;
+	int best_channel;
+	int i;
+
+	if (unlikely(max_channel <= 0 || max_channel > MAX_CLIENT_CHANNELS))
+		max_channel = MAX_CLIENT_CHANNELS;
+	if (unlikely(min_channel < 0 || min_channel >= max_channel)) {
+		min_channel = max_channel - 1;
+		if (unlikely(min_channel < 0))
+			min_channel = 0;
+	}
+
+	/* Fast path.
+	 * Speculate that the next channel is already usable,
+	 * and that it has enough room.
+	 */
+	best_channel = bundle->old_channel + 1;
+	if (best_channel >= max_channel)
+		best_channel = min_channel;
+	res = &bundle->channel[best_channel];
+	if (res->is_connected && !res->recv_error && xio_socket_is_alive(&res->socket)) {
+		res->current_space = xio_socket_send_space_available(&res->socket);
+		if (res->current_space > (PAGE_SIZE + PAGE_SIZE / 4))
+			goto found;
+	}
+
+	/* Slow path. Do all the teady work.
+	 */
+	_maintain_bundle(bundle);
+
+	res = NULL;
+	best_space = -1;
+	best_channel = -1;
+	for (i = min_channel; i < max_channel; i++) {
+		struct client_channel *ch = &bundle->channel[i];
+		long this_space;
+
+		/*  create new channels when necessary */
+		if (unlikely(!ch->is_open)) {
+			int status;
+
+			/*  only create one new channel at a time */
+			status = _setup_channel(bundle, i);
+			XIO_DBG("setup channel %d status=%d\n", i, status);
+			if (unlikely(status < 0))
+				continue;
+
+			this_space = xio_socket_send_space_available(&ch->socket);
+			ch->current_space = this_space;
+			/* Always prefer the newly opened channel */
+			res = ch;
+			best_channel = i;
+			break;
+		}
+
+		/*  select the best usable channel */
+		this_space = xio_socket_send_space_available(&ch->socket);
+		ch->current_space = this_space;
+		if (this_space > best_space) {
+			best_space = this_space;
+			best_channel = i;
+			res = ch;
+		}
+	}
+
+	if (unlikely(!res)) {
+		XIO_WRN("cannot setup communication channel '%s' @%s\n",
+			 bundle->path,
+			 bundle->host);
+		goto done;
+	}
+
+	/*  send initial connect command */
+	if (unlikely(!res->is_connected)) {
+		struct xio_cmd cmd = {
+			.cmd_code = CMD_CONNECT,
+			.cmd_str1 = bundle->path,
+		};
+		int status = xio_send_struct(&res->socket, &cmd, xio_cmd_meta);
+
+		XIO_DBG("send CMD_CONNECT status = %d\n", status);
+		if (unlikely(status < 0)) {
+			XIO_WRN("connect '%s' @%s on channel %d failed, status = %d\n",
+				 bundle->path,
+				 bundle->host,
+				 best_channel,
+				 status);
+			_kill_channel(res);
+			res = NULL;
+			goto done;
+		}
+		res->is_connected = true;
+	}
+
+found:
+	bundle->old_channel = best_channel;
+
+done:
+	return res;
+}
+
+static
+int _request_info(struct client_channel *ch)
+{
+	struct xio_cmd cmd = {
+		.cmd_code = CMD_GETINFO,
+	};
+	int status;
+
+	XIO_DBG("\n");
+	status = xio_send_struct(&ch->socket, &cmd, xio_cmd_meta);
+	XIO_DBG("send CMD_GETINFO status = %d\n", status);
+	if (unlikely(status < 0))
+		XIO_DBG("send of getinfo failed, status = %d\n", status);
+	return status;
+}
+
+static int sender_thread(void *data);
+
+static
+int _setup_bundle(struct client_bundle *bundle, const char *str)
+{
+	int status = -ENOMEM;
+
+	XIO_DBG("\n");
+	_kill_bundle(bundle);
+	brick_string_free(bundle->path);
+
+	bundle->path = brick_strdup(str);
+
+	status = -EINVAL;
+	bundle->host = strchr(bundle->path, '@');
+	if (unlikely(!bundle->host)) {
+		brick_string_free(bundle->path);
+		bundle->path = NULL;
+		XIO_ERR("parameter string '%s' contains no remote specifier with '@'-syntax\n", str);
+		goto done;
+	}
+	*bundle->host++ = '\0';
+
+	bundle->thread_count = thread_count++;
+	bundle->sender.thread = brick_thread_create(sender_thread, bundle, "xio_sender%d", bundle->thread_count);
+	if (unlikely(!bundle->sender.thread)) {
+		XIO_ERR("cannot start sender thread for '%s' @%s\n",
+			 bundle->path,
+			 bundle->host);
+		status = -ENOENT;
+		goto done;
+	}
+
+	status = 0;
+
+done:
+	XIO_DBG("status = %d\n", status);
+	return status;
+}
+
+/***************** own brick * input * output operations *****************/
+
+static int client_get_info(struct client_output *output, struct xio_info *info)
+{
+	int status;
+
+	output->got_info = false;
+	output->get_info = true;
+	wake_up_interruptible_all(&output->bundle.sender_event);
+
+	wait_event_interruptible_timeout(output->info_event, output->got_info, 60 * HZ);
+	status = -ETIME;
+	if (output->got_info && info) {
+		memcpy(info, &output->info, sizeof(*info));
+		status = 0;
+	}
+
+	return status;
+}
+
+static int client_io_get(struct client_output *output, struct aio_object *aio)
+{
+	int maxlen;
+
+	if (aio->obj_initialized) {
+		obj_get(aio);
+		return aio->io_len;
+	}
+
+	/* Limit transfers to page boundaries.
+	 * Currently, this is more restrictive than necessary.
+	 * TODO: improve performance by doing better when possible.
+	 * This needs help from the server in some efficient way.
+	 */
+	maxlen = PAGE_SIZE - (aio->io_pos & (PAGE_SIZE-1));
+	if (aio->io_len > maxlen)
+		aio->io_len = maxlen;
+
+	if (!aio->io_data) { /*  buffered IO */
+		struct client_aio_aspect *aio_a = client_aio_get_aspect(output->brick, aio);
+
+		if (!aio_a)
+			return -EILSEQ;
+
+		aio->io_data = brick_block_alloc(aio->io_pos, (aio_a->alloc_len = aio->io_len));
+
+		aio_a->do_dealloc = true;
+		aio->io_flags = 0;
+	}
+
+	obj_get_first(aio);
+	return 0;
+}
+
+static void client_io_put(struct client_output *output, struct aio_object *aio)
+{
+	struct client_aio_aspect *aio_a;
+
+	if (!obj_put(aio))
+		goto out_return;
+	aio_a = client_aio_get_aspect(output->brick, aio);
+	if (aio_a && aio_a->do_dealloc)
+		brick_block_free(aio->io_data, aio_a->alloc_len);
+	obj_free(aio);
+out_return:;
+}
+
+static
+void _hash_insert(struct client_output *output, struct client_aio_aspect *aio_a)
+{
+	struct aio_object *aio = aio_a->object;
+	unsigned long flags;
+	int hash_index;
+
+	spin_lock_irqsave(&output->lock, flags);
+	list_del(&aio_a->io_head);
+	list_add_tail(&aio_a->io_head, &output->aio_list);
+	list_del(&aio_a->hash_head);
+	aio->io_id = ++output->last_id;
+	hash_index = aio->io_id % CLIENT_HASH_MAX;
+	list_add_tail(&aio_a->hash_head, &output->hash_table[hash_index]);
+	spin_unlock_irqrestore(&output->lock, flags);
+}
+
+static void client_io_io(struct client_output *output, struct aio_object *aio)
+{
+	struct client_aio_aspect *aio_a;
+	int error = -EINVAL;
+
+	aio_a = client_aio_get_aspect(output->brick, aio);
+	if (unlikely(!aio_a))
+		goto error;
+
+	while (output->brick->max_flying > 0 && atomic_read(&output->fly_count) > output->brick->max_flying)
+		brick_msleep(1000 * 2 / HZ);
+
+	if (!output->brick->power.on_led)
+		XIO_ERR("IO submission on dead instance\n");
+
+	atomic_inc(&xio_global_io_flying);
+	atomic_inc(&output->fly_count);
+	obj_get(aio);
+
+	aio_a->submit_jiffies = jiffies;
+	_hash_insert(output, aio_a);
+
+	wake_up_interruptible_all(&output->bundle.sender_event);
+
+	goto out_return;
+error:
+	XIO_ERR("IO error = %d\n", error);
+	SIMPLE_CALLBACK(aio, error);
+	client_io_put(output, aio);
+out_return:;
+}
+
+static
+int receiver_thread(void *data)
+{
+	struct client_channel *ch = data;
+	struct client_output *output = ch->output;
+	int status = 0;
+
+	while (!brick_thread_should_stop()) {
+		struct xio_cmd cmd = {};
+		struct list_head *tmp;
+		struct client_aio_aspect *aio_a = NULL;
+		struct aio_object *aio = NULL;
+		unsigned long flags;
+
+		if (ch->recv_error) {
+			/* The protocol may be out of sync.
+			 * Consume some data to avoid distributed deadlocks.
+			 */
+			(void)xio_recv_raw(&ch->socket, &cmd, 0, sizeof(cmd));
+			brick_msleep(100);
+			status = ch->recv_error;
+			continue;
+		}
+
+		status = xio_recv_struct(&ch->socket, &cmd, xio_cmd_meta);
+		if (status <= 0) {
+			if (!xio_socket_is_alive(&ch->socket)) {
+				XIO_DBG("socket is dead\n");
+				brick_msleep(1000);
+				continue;
+			}
+			goto done;
+		}
+
+		switch (cmd.cmd_code & CMD_FLAG_MASK) {
+		case CMD_NOTIFY:
+			local_trigger();
+			break;
+		case CMD_CONNECT:
+			if (cmd.cmd_int1 < 0) {
+				status = cmd.cmd_int1;
+				XIO_ERR("remote brick connect '%s' @%s failed, remote status = %d\n",
+					 output->bundle.path,
+					 output->bundle.host,
+					 status);
+				goto done;
+			}
+			break;
+		case CMD_CB:
+		{
+			int hash_index = cmd.cmd_int1 % CLIENT_HASH_MAX;
+
+			spin_lock_irqsave(&output->lock, flags);
+			for (tmp = output->hash_table[hash_index].next; tmp != &output->hash_table[hash_index]; tmp = tmp->next) {
+				struct aio_object *tmp_aio;
+
+				aio_a = container_of(tmp, struct client_aio_aspect, hash_head);
+				tmp_aio = aio_a->object;
+				CHECK_PTR(tmp_aio, err);
+				if (tmp_aio->io_id != cmd.cmd_int1)
+					continue;
+				aio = tmp_aio;
+				list_del_init(&aio_a->hash_head);
+				list_del_init(&aio_a->io_head);
+				break;
+
+err:
+				spin_unlock_irqrestore(&output->lock, flags);
+				status = -EBADR;
+				goto done;
+			}
+			spin_unlock_irqrestore(&output->lock, flags);
+
+			if (unlikely(!aio)) {
+				XIO_WRN("got unknown callback id %d on '%s' @%s\n",
+					 cmd.cmd_int1,
+					 output->bundle.path,
+					 output->bundle.host);
+				/*  try to consume the corresponding payload */
+				aio = client_alloc_aio(output->brick);
+				status = xio_recv_cb(&ch->socket, aio, &cmd);
+				obj_free(aio);
+				goto done;
+			}
+
+			status = xio_recv_cb(&ch->socket, aio, &cmd);
+			if (unlikely(status < 0)) {
+				XIO_WRN("interrupted data transfer during callback on '%s' @%s, status = %d\n",
+					 output->bundle.path,
+					 output->bundle.host,
+					 status);
+				_hash_insert(output, aio_a);
+				goto done;
+			}
+
+			if (aio->_object_cb.cb_error < 0)
+				XIO_DBG("ERROR %d\n", aio->_object_cb.cb_error);
+			SIMPLE_CALLBACK(aio, aio->_object_cb.cb_error);
+
+			client_io_put(output, aio);
+
+			atomic_dec(&output->fly_count);
+			atomic_dec(&xio_global_io_flying);
+			break;
+		}
+		case CMD_GETINFO:
+			status = xio_recv_struct(&ch->socket, &output->info, xio_info_meta);
+			if (status < 0) {
+				XIO_WRN("got bad info from remote '%s' @%s, status = %d\n",
+					 output->bundle.path,
+					 output->bundle.host,
+					 status);
+				goto done;
+			}
+			output->got_info = true;
+			wake_up_interruptible_all(&output->info_event);
+			break;
+		default:
+			XIO_ERR("got bad command %d from remote '%s' @%s, terminating.\n",
+				 cmd.cmd_code,
+				 output->bundle.path,
+				 output->bundle.host);
+			status = -EBADR;
+			goto done;
+		}
+done:
+		brick_string_free(cmd.cmd_str1);
+		if (unlikely(status < 0)) {
+			if (!ch->recv_error) {
+				XIO_DBG("signalling recv_error = %d\n", status);
+				ch->recv_error = status;
+			}
+			brick_msleep(100);
+		}
+		/*  wake up sender in any case */
+		wake_up_interruptible_all(&output->bundle.sender_event);
+	}
+
+	if (unlikely(status < 0)) {
+		XIO_WRN("receiver thread '%s' @%s terminated with status = %d\n",
+			 output->bundle.path,
+			 output->bundle.host,
+			 status);
+	}
+
+	xio_shutdown_socket(&ch->socket);
+	return status;
+}
+
+static
+void _do_timeout(struct client_output *output, struct list_head *anchor, int *rounds, bool force)
+{
+	struct client_brick *brick = output->brick;
+	struct list_head *tmp;
+	struct list_head *next;
+	LIST_HEAD(tmp_list);
+	long io_timeout = brick->power.io_timeout;
+	unsigned long flags;
+
+	if (list_empty(anchor))
+		goto out_return;
+	if (io_timeout <= 0)
+		io_timeout = global_net_io_timeout;
+
+	if (!xio_net_is_alive)
+		force = true;
+
+	if (!force && io_timeout <= 0)
+		goto out_return;
+	io_timeout *= HZ;
+
+	spin_lock_irqsave(&output->lock, flags);
+	for (tmp = anchor->next, next = tmp->next; tmp != anchor; tmp = next, next = tmp->next) {
+		struct client_aio_aspect *aio_a;
+
+		aio_a = container_of(tmp, struct client_aio_aspect, io_head);
+
+		if (!force &&
+		    !time_is_before_jiffies(aio_a->submit_jiffies + io_timeout)) {
+			continue;
+		}
+
+		list_del_init(&aio_a->hash_head);
+		list_del_init(&aio_a->io_head);
+		list_add_tail(&aio_a->tmp_head, &tmp_list);
+	}
+	spin_unlock_irqrestore(&output->lock, flags);
+
+	while (!list_empty(&tmp_list)) {
+		struct client_aio_aspect *aio_a;
+		struct aio_object *aio;
+
+		tmp = tmp_list.next;
+		list_del_init(tmp);
+		aio_a = container_of(tmp, struct client_aio_aspect, tmp_head);
+		aio = aio_a->object;
+
+		if (unlikely(!(*rounds)++)) {
+			XIO_WRN("'%s' @%s timeout after %ld: signalling IO error at pos = %lld len = %d\n",
+				 output->bundle.path,
+				 output->bundle.host,
+				 io_timeout,
+				 aio->io_pos,
+				 aio->io_len);
+		}
+
+		atomic_inc(&output->timeout_count);
+
+		SIMPLE_CALLBACK(aio, -ESTALE);
+
+		client_io_put(output, aio);
+
+		atomic_dec(&output->fly_count);
+		atomic_dec(&xio_global_io_flying);
+	}
+out_return:;
+}
+
+static
+void _do_timeout_all(struct client_output *output, bool force)
+{
+	int rounds = 0;
+	int i;
+
+	for (i = 0; i < MAX_CLIENT_CHANNELS; i++) {
+		struct client_channel *ch = &output->bundle.channel[i];
+
+		if (!ch->is_used)
+			continue;
+		_do_timeout(output, &ch->wait_list, &rounds, force);
+	}
+	_do_timeout(output, &output->aio_list, &rounds, force);
+	if (unlikely(rounds > 0)) {
+		XIO_WRN("'%s' @%s had %d timeouts, force = %d\n",
+			 output->bundle.path,
+			 output->bundle.host,
+			 rounds,
+			 force);
+	}
+}
+
+static int sender_thread(void *data)
+{
+	struct client_bundle *bundle = data;
+	struct client_output *output = container_of(bundle, struct client_output, bundle);
+	struct client_brick *brick = output->brick;
+	struct client_channel *ch = NULL;
+	bool do_timeout = false;
+	int ch_skip = max_client_bulk;
+	int status = -ESHUTDOWN;
+	unsigned long flags;
+
+	while (!brick_thread_should_stop()) {
+		struct list_head *tmp = NULL;
+		struct client_aio_aspect *aio_a;
+		struct aio_object *aio;
+		int min_nr;
+		int max_nr;
+
+		/*  timeouting is a rather expensive operation, don't do it too often */
+		if (do_timeout) {
+			do_timeout = false;
+			_maintain_bundle(&output->bundle);
+			_do_timeout_all(output, false);
+		}
+
+		wait_event_interruptible_timeout(output->bundle.sender_event,
+						 !list_empty(&output->aio_list) ||
+						 output->get_info,
+						 2 * HZ);
+
+		if (output->get_info) {
+			ch = _get_channel(bundle, 0, 1);
+			if (unlikely(!ch)) {
+				do_timeout = true;
+				brick_msleep(1000);
+				continue;
+			}
+			status = _request_info(ch);
+			if (unlikely(status < 0)) {
+				XIO_WRN("cannot send info request '%s' @%s, status = %d\n",
+					 output->bundle.path,
+					 output->bundle.host,
+					 status);
+				do_timeout = true;
+				brick_msleep(1000);
+				continue;
+			}
+			output->get_info = false;
+		}
+
+		/* Grab the next aio from the queue
+		 */
+		spin_lock_irqsave(&output->lock, flags);
+		tmp = output->aio_list.next;
+		if (tmp == &output->aio_list) {
+			spin_unlock_irqrestore(&output->lock, flags);
+			XIO_DBG("empty %d %d\n", output->get_info, brick_thread_should_stop());
+			do_timeout = true;
+			continue;
+		}
+		list_del_init(tmp);
+		/*  notice: hash_head remains in its list! */
+		spin_unlock_irqrestore(&output->lock, flags);
+
+		aio_a = container_of(tmp, struct client_aio_aspect, io_head);
+		aio = aio_a->object;
+
+		if (brick->limit_mode) {
+			int amount = 0;
+
+			if (aio->io_cs_mode < 2)
+				amount = (aio->io_len - 1) / 1024 + 1;
+			rate_limit_sleep(&client_limiter, amount);
+		}
+
+		/*  try to spread reads over multiple channels.... */
+		min_nr = 0;
+		max_nr = max_client_channels;
+		if (!aio->io_rw) {
+			/* optionally separate reads from writes */
+			if (brick->separate_reads && max_nr > 1)
+				min_nr = 1;
+		} else if (!brick->allow_permuting_writes) {
+			max_nr = 1;
+		}
+		if (!ch || ch->recv_error ||
+		    !xio_socket_is_alive(&ch->socket) ||
+		    ch->ch_nr >= max_nr || --ch_skip < 0) {
+			ch = _get_channel(bundle, min_nr, max_nr);
+			if (unlikely(!ch)) {
+				/*  notice: this will re-assign hash_head without harm */
+				_hash_insert(output, aio_a);
+				do_timeout = true;
+				brick_msleep(1000);
+				continue;
+			}
+			/* estimate: add some headroom for overhead */
+			ch_skip = ch->current_space / PAGE_SIZE +
+				ch->current_space / (PAGE_SIZE * 8);
+			if (ch_skip > max_client_bulk)
+				ch_skip = max_client_bulk;
+		}
+
+		spin_lock_irqsave(&output->lock, flags);
+		list_add(tmp, &ch->wait_list);
+		/*  notice: hash_head is already there! */
+		spin_unlock_irqrestore(&output->lock, flags);
+
+		status = xio_send_aio(&ch->socket, aio);
+		if (unlikely(status < 0)) {
+			_hash_insert(output, aio_a);
+			do_timeout = true;
+			ch = NULL;
+			/*  retry submission on next occasion.. */
+			XIO_WRN("aio send '%s' @%s failed, status = %d\n",
+				 output->bundle.path,
+				 output->bundle.host,
+				 status);
+
+			brick_msleep(100);
+			continue;
+		}
+	}
+
+	if (unlikely(status < 0)) {
+		XIO_WRN("sender thread '%s' @%s terminated with status = %d\n",
+			 output->bundle.path,
+			 output->bundle.host,
+			 status);
+	}
+
+	_kill_all_channels(bundle);
+
+	/* Signal error on all pending IO requests.
+	 * We have no other chance (except probably delaying
+	 * this until destruction which is probably not what
+	 * we want).
+	 */
+	_do_timeout_all(output, true);
+	wake_up_interruptible_all(&output->bundle.sender_event);
+	XIO_DBG("sender terminated\n");
+	return status;
+}
+
+static int client_switch(struct client_brick *brick)
+{
+	struct client_output *output = brick->outputs[0];
+	int status = 0;
+
+	if (brick->power.button) {
+		if (brick->power.on_led)
+			goto done;
+		xio_set_power_off_led((void *)brick, false);
+		status = _setup_bundle(&output->bundle, brick->brick_name);
+		if (likely(status >= 0)) {
+			output->get_info = true;
+			brick->connection_state = 1;
+			xio_set_power_on_led((void *)brick, true);
+		}
+	} else {
+		if (brick->power.off_led)
+			goto done;
+		xio_set_power_on_led((void *)brick, false);
+		_kill_bundle(&output->bundle);
+		_do_timeout_all(output, true);
+		output->got_info = false;
+		brick->connection_state = 0;
+		xio_set_power_off_led((void *)brick, !output->bundle.sender.thread);
+	}
+done:
+	return status;
+}
+
+/*************** informational * statistics **************/
+
+static
+char *client_statistics(struct client_brick *brick, int verbose)
+{
+	struct client_output *output = brick->outputs[0];
+	char *res = brick_string_alloc(1024);
+
+	snprintf(res, 1024,
+		 "max_flying = %d io_timeout = %d | timeout_count = %d fly_count = %d\n",
+		 brick->max_flying,
+		 brick->power.io_timeout,
+		 atomic_read(&output->timeout_count),
+		 atomic_read(&output->fly_count));
+
+	return res;
+}
+
+static
+void client_reset_statistics(struct client_brick *brick)
+{
+	struct client_output *output = brick->outputs[0];
+
+	atomic_set(&output->timeout_count, 0);
+}
+
+/*************** object * aspect constructors * destructors **************/
+
+static int client_aio_aspect_init_fn(struct generic_aspect *_ini)
+{
+	struct client_aio_aspect *ini = (void *)_ini;
+
+	INIT_LIST_HEAD(&ini->io_head);
+	INIT_LIST_HEAD(&ini->hash_head);
+	INIT_LIST_HEAD(&ini->tmp_head);
+	return 0;
+}
+
+static void client_aio_aspect_exit_fn(struct generic_aspect *_ini)
+{
+	struct client_aio_aspect *ini = (void *)_ini;
+
+	CHECK_HEAD_EMPTY(&ini->io_head);
+	CHECK_HEAD_EMPTY(&ini->hash_head);
+}
+
+XIO_MAKE_STATICS(client);
+
+/********************* brick constructors * destructors *******************/
+
+static int client_brick_construct(struct client_brick *brick)
+{
+	return 0;
+}
+
+static int client_output_construct(struct client_output *output)
+{
+	int i;
+
+	output->hash_table = brick_block_alloc(0, PAGE_SIZE);
+
+	for (i = 0; i < CLIENT_HASH_MAX; i++)
+		INIT_LIST_HEAD(&output->hash_table[i]);
+
+	for (i = 0; i < MAX_CLIENT_CHANNELS; i++) {
+		struct client_channel *ch = &output->bundle.channel[i];
+
+		ch->output = output;
+		INIT_LIST_HEAD(&ch->wait_list);
+	}
+
+	init_waitqueue_head(&output->bundle.sender_event);
+
+	spin_lock_init(&output->lock);
+	INIT_LIST_HEAD(&output->aio_list);
+	init_waitqueue_head(&output->info_event);
+	return 0;
+}
+
+static int client_output_destruct(struct client_output *output)
+{
+	brick_string_free(output->bundle.path);
+	output->bundle.path = NULL;
+	brick_block_free(output->hash_table, PAGE_SIZE);
+	return 0;
+}
+
+/************************ static structs ***********************/
+
+static struct client_brick_ops client_brick_ops = {
+	.brick_switch = client_switch,
+	.brick_statistics = client_statistics,
+	.reset_statistics = client_reset_statistics,
+};
+
+static struct client_output_ops client_output_ops = {
+	.xio_get_info = client_get_info,
+	.aio_get = client_io_get,
+	.aio_put = client_io_put,
+	.aio_io = client_io_io,
+};
+
+const struct client_input_type client_input_type = {
+	.type_name = "client_input",
+	.input_size = sizeof(struct client_input),
+};
+
+static const struct client_input_type *client_input_types[] = {
+	&client_input_type,
+};
+
+const struct client_output_type client_output_type = {
+	.type_name = "client_output",
+	.output_size = sizeof(struct client_output),
+	.master_ops = &client_output_ops,
+	.output_construct = &client_output_construct,
+	.output_destruct = &client_output_destruct,
+};
+
+static const struct client_output_type *client_output_types[] = {
+	&client_output_type,
+};
+
+const struct client_brick_type client_brick_type = {
+	.type_name = "client_brick",
+	.brick_size = sizeof(struct client_brick),
+	.max_inputs = 0,
+	.max_outputs = 1,
+	.master_ops = &client_brick_ops,
+	.aspect_types = client_aspect_types,
+	.default_input_types = client_input_types,
+	.default_output_types = client_output_types,
+	.brick_construct = &client_brick_construct,
+};
+
+/***************** module init stuff ************************/
+
+struct rate_limiter client_limiter = {
+	.lim_max_rate = 0,
+};
+
+int global_net_io_timeout = 30;
+
+module_param_named(net_io_timeout, global_net_io_timeout, int, 0);
+
+int __init init_xio_client(void)
+{
+	XIO_INF("init_client()\n");
+	_client_brick_type = (void *)&client_brick_type;
+	return client_register_brick_type();
+}
+
+void exit_xio_client(void)
+{
+	XIO_INF("exit_client()\n");
+	client_unregister_brick_type();
+}
include/linux/xio/xio_client.h new file +105 -0

105 lines not on this page · raw message

Signature · 1 line
2.6.4
#20 lore raw

RFC 20/31 mars: add new module xio_if

In reply to Thomas Schoebel-Theuer
---
 drivers/staging/mars/xio_bricks/xio_if.c | 961 +++++++++++++++++++++++++++++++
 include/linux/xio/xio_if.h               | 108 ++++
 2 files changed, 1069 insertions(+)
 create mode 100644 drivers/staging/mars/xio_bricks/xio_if.c
 create mode 100644 include/linux/xio/xio_if.h
drivers/staging/mars/xio_bricks/xio_if.c new file +961 -0
new file mode 100644
index 0000000..65e023c
@@ -0,0 +1,961 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+/* Interface to a Linux device.
+ * 1 Input, 0 Outputs.
+ */
+
+#define REQUEST_MERGING
+#define ALWAYS_UNPLUG			true
+#define PREFETCH_LEN			PAGE_SIZE
+
+/*  low-level device parameters */
+#define IF_MAX_SEGMENT_SIZE		PAGE_SIZE
+#define USE_MAX_SECTORS			(IF_MAX_SEGMENT_SIZE >> 9)
+#define USE_MAX_PHYS_SEGMENTS		(IF_MAX_SEGMENT_SIZE >> 9)
+#define USE_MAX_SEGMENT_SIZE		IF_MAX_SEGMENT_SIZE
+#define USE_LOGICAL_BLOCK_SIZE		512
+#define USE_SEGMENT_BOUNDARY		(PAGE_SIZE-1)
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/string.h>
+
+#include <linux/bio.h>
+#include <linux/major.h>
+#include <linux/genhd.h>
+#include <linux/blkdev.h>
+
+#include <linux/xio/xio.h>
+#include <linux/brick/lib_limiter.h>
+
+#ifndef XIO_MAJOR
+#define XIO_MAJOR			(DRBD_MAJOR + 1)
+#endif
+
+/************************ global tuning ***********************/
+
+int if_throttle_start_size;
+
+struct rate_limiter if_throttle = {
+	.lim_max_rate = 5000,
+};
+
+/************************ own type definitions ***********************/
+
+#include <linux/xio/xio_if.h>
+
+#define IF_HASH_MAX			(PAGE_SIZE / sizeof(struct if_hash_anchor))
+#define IF_HASH_CHUNK			(PAGE_SIZE * 32)
+
+struct if_hash_anchor {
+	spinlock_t hash_lock;
+	struct list_head hash_anchor;
+};
+
+/************************ own static definitions ***********************/
+
+/*  TODO: check bounds, ensure that free minor numbers are recycled */
+static int device_minor;
+
+/*************** object * aspect constructors * destructors **************/
+
+/************************ linux operations ***********************/
+
+static
+void _if_start_io_acct(struct if_input *input, struct bio_wrapper *biow)
+{
+	struct bio *bio = biow->bio;
+	const int rw = bio_data_dir(bio);
+	const int cpu = part_stat_lock();
+
+	(void)cpu;
+	part_round_stats(cpu, &input->disk->part0);
+	part_stat_inc(cpu, &input->disk->part0, ios[rw]);
+	part_stat_add(cpu, &input->disk->part0, sectors[rw], bio->bi_iter.bi_size >> 9);
+	part_inc_in_flight(&input->disk->part0, rw);
+	part_stat_unlock();
+	biow->start_time = jiffies;
+}
+
+static
+void _if_end_io_acct(struct if_input *input, struct bio_wrapper *biow)
+{
+	unsigned long duration = jiffies - biow->start_time;
+	struct bio *bio = biow->bio;
+	const int rw = bio_data_dir(bio);
+	const int cpu = part_stat_lock();
+
+	(void)cpu;
+	part_stat_add(cpu, &input->disk->part0, ticks[rw], duration);
+	part_round_stats(cpu, &input->disk->part0);
+	part_dec_in_flight(&input->disk->part0, rw);
+	part_stat_unlock();
+}
+
+/* callback
+ */
+static
+void if_endio(struct generic_callback *cb)
+{
+	struct if_aio_aspect *aio_a = cb->cb_private;
+	struct if_input *input;
+	int k;
+	int rw;
+	int error;
+
+	LAST_CALLBACK(cb);
+	if (unlikely(!aio_a || !aio_a->object)) {
+		XIO_FAT("aio_a = %p aio = %p, something is very wrong here!\n", aio_a, aio_a->object);
+		goto out_return;
+	}
+	input = aio_a->input;
+	CHECK_PTR(input, err);
+
+	rw = aio_a->object->io_rw;
+
+	for (k = 0; k < aio_a->bio_count; k++) {
+		struct bio_wrapper *biow;
+		struct bio *bio;
+
+		biow = aio_a->orig_biow[k];
+		aio_a->orig_biow[k] = NULL;
+		CHECK_PTR(biow, err);
+
+		CHECK_ATOMIC(&biow->bi_comp_cnt, 1);
+		if (!atomic_dec_and_test(&biow->bi_comp_cnt))
+			continue;
+
+		bio = biow->bio;
+		CHECK_PTR_NULL(bio, err);
+
+		_if_end_io_acct(input, biow);
+
+		error = CALLBACK_ERROR(aio_a->object);
+		if (unlikely(error < 0)) {
+			int bi_size = bio->bi_iter.bi_size;
+
+			XIO_ERR("NYI: error=%d RETRY LOGIC %u\n", error, bi_size);
+		} else { /*  bio conventions are slightly different... */
+			error = 0;
+			bio->bi_iter.bi_size = 0;
+		}
+		bio->bi_error = error;
+		bio_endio(bio);
+		bio_put(bio);
+		brick_mem_free(biow);
+	}
+	atomic_dec(&input->flying_count);
+	if (rw)
+		atomic_dec(&input->write_flying_count);
+	else
+		atomic_dec(&input->read_flying_count);
+	goto out_return;
+err:
+	XIO_FAT("error in callback, giving up\n");
+out_return:;
+}
+
+/* Kick off plugged aios
+ */
+static
+void _if_unplug(struct if_input *input)
+{
+	/* struct if_brick *brick = input->brick; */
+	LIST_HEAD(tmp_list);
+	unsigned long flags;
+
+#ifdef CONFIG_MARS_DEBUG
+	might_sleep();
+#endif
+
+	spin_lock_irqsave(&input->req_lock, flags);
+	if (!list_empty(&input->plug_anchor)) {
+		/*  move over the whole list */
+		list_replace_init(&input->plug_anchor, &tmp_list);
+		atomic_set(&input->plugged_count, 0);
+	}
+	spin_unlock_irqrestore(&input->req_lock, flags);
+
+	while (!list_empty(&tmp_list)) {
+		struct if_aio_aspect *aio_a;
+		struct aio_object *aio;
+		int hash_index;
+
+		aio_a = container_of(tmp_list.next, struct if_aio_aspect, plug_head);
+		list_del_init(&aio_a->plug_head);
+
+		hash_index = aio_a->hash_index;
+		spin_lock_irqsave(&input->hash_table[hash_index].hash_lock, flags);
+		list_del_init(&aio_a->hash_head);
+		spin_unlock_irqrestore(&input->hash_table[hash_index].hash_lock, flags);
+
+		aio = aio_a->object;
+
+		if (unlikely(aio_a->current_len > aio_a->max_len))
+			XIO_ERR("request len %d > %d\n", aio_a->current_len, aio_a->max_len);
+		aio->io_len = aio_a->current_len;
+
+		atomic_inc(&input->flying_count);
+		atomic_inc(&input->total_fire_count);
+		if (aio->io_rw)
+			atomic_inc(&input->write_flying_count);
+		else
+			atomic_inc(&input->read_flying_count);
+		if (aio->io_skip_sync)
+			atomic_inc(&input->total_skip_sync_count);
+
+		GENERIC_INPUT_CALL(input, aio_io, aio);
+		GENERIC_INPUT_CALL(input, aio_put, aio);
+	}
+}
+
+/* accept a linux bio, convert to aio and call buf_io() on it.
+ */
+static
+blk_qc_t if_make_request(struct request_queue *q, struct bio *bio)
+{
+	struct if_input *input = q->queuedata;
+	struct if_brick *brick = input->brick;
+
+	/* Original flags of the source bio
+	 */
+	const int  rw = bio_data_dir(bio);
+	const int  sectors = bio_sectors(bio);
+
+	const bool ahead = bio_flagged(bio, __REQ_RAHEAD) && rw == READ;
+	const bool barrier = bio_flagged(bio, __REQ_SOFTBARRIER);
+	const bool syncio = bio_flagged(bio, __REQ_SYNC);
+	const bool unplug = false;
+	const bool meta = bio_flagged(bio, __REQ_META);
+	const bool discard = bio_flagged(bio, __REQ_DISCARD);
+	const bool noidle = bio_flagged(bio, __REQ_NOIDLE);
+
+	const int  prio = bio_prio(bio);
+
+	/* Transform into XIO flags
+	 */
+	const int  io_prio =
+		(prio == IOPRIO_CLASS_RT || (meta | syncio)) ?
+		XIO_PRIO_HIGH :
+		(prio == IOPRIO_CLASS_IDLE) ?
+		XIO_PRIO_LOW :
+		XIO_PRIO_NORMAL;
+	const bool do_unplug = ALWAYS_UNPLUG | unplug | noidle;
+	const bool do_skip_sync = brick->skip_sync && !(barrier | syncio);
+
+	struct bio_wrapper *biow;
+	struct aio_object *aio = NULL;
+	struct if_aio_aspect *aio_a;
+
+	struct bio_vec bvec;
+	struct bvec_iter i;
+
+	loff_t pos = ((loff_t)bio->bi_iter.bi_sector) << 9; /*	TODO: make dynamic */
+	int total_len = bio->bi_iter.bi_size;
+
+	bool assigned = false;
+	int error = -EINVAL;
+
+	bind_to_channel(brick->say_channel, current);
+
+	might_sleep();
+
+	blk_queue_split(q, &bio, q->bio_split);
+
+	if (unlikely(!sectors)) {
+		_if_unplug(input);
+		/* THINK: usually this happens only at write barriers.
+		 * We have no "barrier" operation in XIO, since
+		 * callback semantics should always denote
+		 * "writethrough accomplished".
+		 * In case of exceptional semantics, we need to do
+		 * something here. For now, we do just nothing.
+		 */
+		error = 0;
+		bio->bi_error = error;
+		bio_endio(bio);
+		goto done;
+	}
+
+	/*  throttling of too big write requests */
+	if (rw && if_throttle_start_size > 0) {
+		int kb = (total_len + 512) / 1024;
+
+		if (kb >= if_throttle_start_size)
+			rate_limit_sleep(&if_throttle, kb);
+	}
+
+	(void)ahead; /*  shut up gcc */
+	if (unlikely(discard)) { /*  NYI */
+		error = 0;
+		bio->bi_error = error;
+		bio_endio(bio);
+		goto done;
+	}
+
+	biow = brick_mem_alloc(sizeof(struct bio_wrapper));
+	biow->bio = bio;
+	atomic_set(&biow->bi_comp_cnt, 0);
+
+	if (rw)
+		atomic_inc(&input->total_write_count);
+	else
+		atomic_inc(&input->total_read_count);
+	_if_start_io_acct(input, biow);
+
+	/* Get a reference to the bio.
+	 * Will be released after bio_endio().
+	 */
+	bio_get(bio);
+
+	/* FIXME: THIS IS PROVISIONARY (use event instead)
+	 */
+	while (unlikely(!brick->power.on_led))
+		brick_msleep(100);
+
+	bio_for_each_segment(bvec, bio, i) {
+		struct page *page = bvec.bv_page;
+		int bv_len = bvec.bv_len;
+		int offset = bvec.bv_offset;
+
+		void *data;
+
+#ifdef ARCH_HAS_KMAP
+#error FIXME/TODO: the current infrastructure cannot deal with HIGHMEM / kmap()
+#error HINT: XIO is supposed to run on big 64bit (storage) servers.
+#endif
+		data = page_address(page);
+		error = -EINVAL;
+		if (unlikely(!data))
+			break;
+
+		data += offset;
+
+		while (bv_len > 0) {
+			struct list_head *tmp;
+			int hash_index;
+			int this_len = 0;
+			unsigned long flags;
+
+			aio = NULL;
+			aio_a = NULL;
+
+			hash_index = (pos / IF_HASH_CHUNK) % IF_HASH_MAX;
+
+#ifdef REQUEST_MERGING
+			spin_lock_irqsave(&input->hash_table[hash_index].hash_lock, flags);
+			for (tmp = input->hash_table[hash_index].hash_anchor.next; tmp != &input->hash_table[hash_index].hash_anchor; tmp = tmp->next) {
+				struct if_aio_aspect *tmp_a;
+				struct aio_object *tmp_aio;
+				int i;
+
+				tmp_a = container_of(tmp, struct if_aio_aspect, hash_head);
+				tmp_aio = tmp_a->object;
+				if (tmp_a->orig_page != page || tmp_aio->io_rw != rw || tmp_a->bio_count >= MAX_BIO || tmp_a->current_len + bv_len > tmp_a->max_len)
+					continue;
+
+				if (tmp_aio->io_data + tmp_a->current_len == data)
+					goto merge_end;
+				continue;
+
+merge_end:
+				tmp_a->current_len += bv_len;
+				aio = tmp_aio;
+				aio_a = tmp_a;
+				this_len = bv_len;
+				if (!do_skip_sync)
+					aio->io_skip_sync = false;
+
+				for (i = 0; i < aio_a->bio_count; i++) {
+					if (aio_a->orig_biow[i]->bio == bio)
+						goto unlock;
+				}
+
+				CHECK_ATOMIC(&biow->bi_comp_cnt, 0);
+				atomic_inc(&biow->bi_comp_cnt);
+				aio_a->orig_biow[aio_a->bio_count++] = biow;
+				assigned = true;
+				goto unlock;
+			} /*  foreach hash collision list member */
+
+unlock:
+			spin_unlock_irqrestore(&input->hash_table[hash_index].hash_lock, flags);
+#endif
+			if (!aio) {
+				int prefetch_len;
+
+				error = -ENOMEM;
+				aio = if_alloc_aio(brick);
+				aio_a = if_aio_get_aspect(brick, aio);
+				if (unlikely(!aio_a))
+					goto err;
+
+#ifdef PREFETCH_LEN
+				prefetch_len = PREFETCH_LEN - offset;
+/**/
+				if (prefetch_len > total_len)
+					prefetch_len = total_len;
+				if (pos + prefetch_len > brick->dev_size)
+					prefetch_len = brick->dev_size - pos;
+				if (prefetch_len < bv_len)
+					prefetch_len = bv_len;
+#else
+				prefetch_len = bv_len;
+#endif
+
+				SETUP_CALLBACK(aio, if_endio, aio_a);
+
+				aio_a->input = input;
+				aio->io_rw = aio->io_may_write = rw;
+				aio->io_pos = pos;
+				aio->io_len = prefetch_len;
+				aio->io_data = data; /*  direct IO */
+				aio->io_prio = io_prio;
+				aio_a->orig_page = page;
+
+				error = GENERIC_INPUT_CALL(input, aio_get, aio);
+				if (unlikely(error < 0))
+					goto err;
+
+				this_len = aio->io_len; /*  now may be shorter than originally requested. */
+				aio_a->max_len = this_len;
+				if (this_len > bv_len)
+					this_len = bv_len;
+				aio_a->current_len = this_len;
+				if (rw)
+					atomic_inc(&input->total_aio_write_count);
+				else
+					atomic_inc(&input->total_aio_read_count);
+				CHECK_ATOMIC(&biow->bi_comp_cnt, 0);
+				atomic_inc(&biow->bi_comp_cnt);
+				aio_a->orig_biow[0] = biow;
+				aio_a->bio_count = 1;
+				assigned = true;
+
+				/* When a bio with multiple biovecs is split into
+				 * multiple aios, only the last one should be
+				 * working in synchronous writethrough mode.
+				 */
+				aio->io_skip_sync = true;
+				if (!do_skip_sync && i.bi_idx + 1 >= bio->bi_iter.bi_idx)
+					aio->io_skip_sync = false;
+
+				atomic_inc(&input->plugged_count);
+
+				aio_a->hash_index = hash_index;
+				spin_lock_irqsave(&input->hash_table[hash_index].hash_lock, flags);
+				list_add_tail(&aio_a->hash_head, &input->hash_table[hash_index].hash_anchor);
+				spin_unlock_irqrestore(&input->hash_table[hash_index].hash_lock, flags);
+
+				spin_lock_irqsave(&input->req_lock, flags);
+				list_add_tail(&aio_a->plug_head, &input->plug_anchor);
+				spin_unlock_irqrestore(&input->req_lock, flags);
+			} /*  !aio */
+
+			pos += this_len;
+			data += this_len;
+			bv_len -= this_len;
+		} /*  while bv_len > 0 */
+	} /*  foreach bvec */
+
+	error = 0;
+
+err:
+	if (error < 0) {
+		XIO_ERR("cannot submit request from bio, status=%d\n", error);
+		if (!assigned) {
+			bio->bi_error = error;
+			bio_endio(bio);
+		}
+	}
+
+	if (do_unplug ||
+	   (brick && brick->max_plugged > 0 && atomic_read(&input->plugged_count) > brick->max_plugged)) {
+		_if_unplug(input);
+	}
+
+done:
+	remove_binding_from(brick->say_channel, current);
+
+	return BLK_QC_T_NONE;
+}
+
+static
+int xio_congested(void *data, int bdi_bits)
+{
+	struct if_input *input = data;
+	int ret = 0;
+
+	if (bdi_bits & (1 << WB_sync_congested) &&
+	    atomic_read(&input->read_flying_count) > 0) {
+		ret |= (1 << WB_sync_congested);
+	}
+	if (bdi_bits & (1 << WB_async_congested) &&
+	    atomic_read(&input->write_flying_count) > 0) {
+		ret |= (1 << WB_async_congested);
+	}
+	return ret;
+}
+
+static
+loff_t if_get_capacity(struct if_brick *brick)
+{
+	/* Don't read always, read only when unknown.
+	 * brick->dev_size may be different from underlying sizes,
+	 * e.g. when the size symlink indicates a logically smaller
+	 * device than physically.
+	 */
+	if (brick->dev_size <= 0) {
+		struct xio_info info = {};
+		struct if_input *input = brick->inputs[0];
+		int status;
+
+		status = GENERIC_INPUT_CALL(input, xio_get_info, &info);
+		if (unlikely(status < 0)) {
+			XIO_ERR("cannot get device info, status=%d\n", status);
+			return 0;
+		}
+		XIO_INF("determined default capacity: %lld bytes\n", info.current_size);
+		brick->dev_size = info.current_size;
+	}
+	return brick->dev_size;
+}
+
+static
+void if_set_capacity(struct if_input *input, loff_t capacity)
+{
+	CHECK_PTR(input->disk, done);
+	CHECK_PTR(input->disk->disk_name, done);
+	XIO_INF("new capacity of '%s': %lld bytes\n", input->disk->disk_name, capacity);
+	input->capacity = capacity;
+	set_capacity(input->disk, capacity >> 9);
+	if (likely(input->bdev && input->bdev->bd_inode))
+		i_size_write(input->bdev->bd_inode, capacity);
+done:;
+}
+
+static const struct block_device_operations if_blkdev_ops;
+
+static int if_switch(struct if_brick *brick)
+{
+	struct if_input *input = brick->inputs[0];
+	struct request_queue *q;
+	struct gendisk *disk;
+	int minor;
+	int status = 0;
+
+	down(&brick->switch_sem);
+
+	/*  brick is in operation */
+	if (brick->power.button && brick->power.on_led) {
+		loff_t capacity;
+
+		capacity = if_get_capacity(brick);
+		if (capacity > 0 && capacity != input->capacity) {
+			XIO_INF("changing capacity from %lld to %lld\n",
+				(long long)input->capacity,
+				(long long)capacity);
+			if_set_capacity(input, capacity);
+		}
+	}
+
+	/*  brick should be switched on */
+	if (brick->power.button && brick->power.off_led) {
+		loff_t capacity;
+
+		xio_set_power_off_led((void *)brick,  false);
+		brick->say_channel = get_binding(current);
+
+		status = -ENOMEM;
+		q = blk_alloc_queue(GFP_BRICK);
+		if (!q) {
+			XIO_ERR("cannot allocate device request queue\n");
+			goto is_down;
+		}
+		q->queuedata = input;
+		input->q = q;
+
+		disk = alloc_disk(1);
+		if (!disk) {
+			XIO_ERR("cannot allocate gendisk\n");
+			goto is_down;
+		}
+
+		minor = device_minor++; /* TODO: protect against races (e.g. atomic_t) */
+		set_disk_ro(disk, true);
+
+		disk->queue = q;
+		disk->major = XIO_MAJOR; /* TODO: make this dynamic for >256 devices */
+		disk->first_minor = minor;
+		disk->fops = &if_blkdev_ops;
+		snprintf(disk->disk_name, sizeof(disk->disk_name),  "%s", brick->brick_name);
+		disk->private_data = input;
+		input->disk = disk;
+		capacity = if_get_capacity(brick);
+		XIO_DBG("created device name %s, capacity=%lld\n", disk->disk_name, capacity);
+		if_set_capacity(input, capacity);
+
+		blk_queue_make_request(q, if_make_request);
+		blk_set_stacking_limits(&q->limits);
+		blk_queue_max_hw_sectors(q, USE_MAX_SECTORS);
+		blk_queue_max_segments(q, PAGE_SIZE);
+		blk_queue_max_segment_size(q, USE_MAX_SEGMENT_SIZE);
+		blk_queue_logical_block_size(q, USE_LOGICAL_BLOCK_SIZE);
+		blk_queue_segment_boundary(q, USE_SEGMENT_BOUNDARY);
+		blk_queue_bounce_limit(q, BLK_BOUNCE_ANY);
+		blk_queue_max_write_same_sectors(q, 0);
+		q->queue_lock = &input->req_lock; /*  needed! */
+
+		input->bdev = bdget(MKDEV(disk->major, minor));
+		/* we have no partitions. we contain only ourselves. */
+		input->bdev->bd_contains = input->bdev;
+
+		q->backing_dev_info.congested_fn = xio_congested;
+		q->backing_dev_info.congested_data = input;
+
+		/*  point of no return */
+		XIO_DBG("add_disk()\n");
+		add_disk(disk);
+		set_disk_ro(disk, false);
+
+		/*  report success */
+		xio_set_power_on_led((void *)brick, true);
+		status = 0;
+	}
+
+	/*  brick should be switched off */
+	if (!brick->power.button && !brick->power.off_led) {
+		int opened;
+		int plugged;
+		int flying;
+
+		xio_set_power_on_led((void *)brick, false);
+		disk = input->disk;
+		if (!disk)
+			goto is_down;
+
+		opened = atomic_read(&brick->open_count);
+		if (unlikely(opened > 0)) {
+			XIO_INF("device '%s' is open %d times, cannot shutdown\n", disk->disk_name, opened);
+			status = -EBUSY;
+			goto done; /*  don't indicate "off" status */
+		}
+		plugged = atomic_read(&input->plugged_count);
+		if (unlikely(plugged > 0)) {
+			XIO_INF("device '%s' has %d plugged requests, cannot shutdown\n", disk->disk_name, plugged);
+			status = -EBUSY;
+			goto done; /*  don't indicate "off" status */
+		}
+		flying = atomic_read(&input->flying_count);
+		if (unlikely(flying > 0)) {
+			XIO_INF("device '%s' has %d flying requests, cannot shutdown\n", disk->disk_name, flying);
+			status = -EBUSY;
+			goto done; /*  don't indicate "off" status */
+		}
+		XIO_DBG("calling del_gendisk()\n");
+		del_gendisk(input->disk);
+		/* There might be subtle races */
+		while (atomic_read(&input->flying_count) > 0) {
+			XIO_WRN("device '%s' unexpectedly has %d flying requests\n", disk->disk_name, flying);
+			brick_msleep(1000);
+		}
+		if (input->bdev) {
+			XIO_DBG("calling bdput()\n");
+			bdput(input->bdev);
+			input->bdev = NULL;
+		}
+		XIO_DBG("calling put_disk()\n");
+		put_disk(input->disk);
+		input->disk = NULL;
+		q = input->q;
+		if (q) {
+			blk_cleanup_queue(q);
+			input->q = NULL;
+		}
+		status = 0;
+is_down:
+		xio_set_power_off_led((void *)brick, true);
+	}
+
+done:
+	up(&brick->switch_sem);
+	return status;
+}
+
+/*************** interface to the outer world (kernel) **************/
+
+static int if_open(struct block_device *bdev, fmode_t mode)
+{
+	struct if_input *input;
+	struct if_brick *brick;
+
+	if (unlikely(!bdev || !bdev->bd_disk)) {
+		XIO_ERR("----------------------- INVAL ------------------------------\n");
+		return -EINVAL;
+	}
+
+	input = bdev->bd_disk->private_data;
+
+	if (unlikely(!input || !input->brick)) {
+		XIO_ERR("----------------------- BAD IF SETUP ------------------------------\n");
+		return -EINVAL;
+	}
+	brick = input->brick;
+
+	down(&brick->switch_sem);
+
+	if (unlikely(!brick->power.on_led)) {
+		XIO_INF("----------------------- BUSY %d ------------------------------\n",
+			atomic_read(&brick->open_count));
+		up(&brick->switch_sem);
+		return -EBUSY;
+	}
+
+	atomic_inc(&brick->open_count);
+
+	XIO_INF("----------------------- OPEN %d ------------------------------\n", atomic_read(&brick->open_count));
+
+	up(&brick->switch_sem);
+	return 0;
+}
+
+static
+void
+if_release(struct gendisk *gd, fmode_t mode)
+{
+	struct if_input *input = gd->private_data;
+	struct if_brick *brick = input->brick;
+	int nr;
+
+	XIO_INF("----------------------- CLOSE %d ------------------------------\n", atomic_read(&brick->open_count));
+
+	if (atomic_dec_and_test(&brick->open_count)) {
+		while ((nr = atomic_read(&input->flying_count)) > 0) {
+			XIO_INF("%d IO requests not yet completed\n", nr);
+			brick_msleep(1000);
+		}
+
+		XIO_DBG("status button=%d on_led=%d off_led=%d\n",
+			brick->power.button,
+			brick->power.on_led,
+			brick->power.off_led);
+		local_trigger();
+	}
+}
+
+static const struct block_device_operations if_blkdev_ops = {
+	.owner = THIS_MODULE,
+	.open = if_open,
+	.release = if_release,
+
+};
+
+/*************** informational * statistics **************/
+
+static
+char *if_statistics(struct if_brick *brick, int verbose)
+{
+	struct if_input *input = brick->inputs[0];
+	char *res = brick_string_alloc(512);
+	int tmp0 = atomic_read(&input->total_reada_count);
+	int tmp1 = atomic_read(&input->total_read_count);
+	int tmp2 = atomic_read(&input->total_aio_read_count);
+	int tmp3 = atomic_read(&input->total_write_count);
+	int tmp4 = atomic_read(&input->total_aio_write_count);
+
+	snprintf(res, 512,
+		 "total reada = %d reads = %d aio_reads = %d (%d%%) writes = %d aio_writes = %d (%d%%) empty = %d fired = %d skip_sync = %d | plugged = %d flying = %d (reads = %d writes = %d)\n",
+		 tmp0,
+		 tmp1,
+		 tmp2,
+		 tmp1 ? tmp2 * 100 / tmp1 : 0,
+		 tmp3,
+		 tmp4,
+		 tmp3 ? tmp4 * 100 / tmp3 : 0,
+		 atomic_read(&input->total_empty_count),
+		 atomic_read(&input->total_fire_count),
+		 atomic_read(&input->total_skip_sync_count),
+		 atomic_read(&input->plugged_count),
+		 atomic_read(&input->flying_count),
+		 atomic_read(&input->read_flying_count),
+		 atomic_read(&input->write_flying_count));
+	return res;
+}
+
+static
+void if_reset_statistics(struct if_brick *brick)
+{
+	struct if_input *input = brick->inputs[0];
+
+	atomic_set(&input->total_read_count, 0);
+	atomic_set(&input->total_write_count, 0);
+	atomic_set(&input->total_empty_count, 0);
+	atomic_set(&input->total_fire_count, 0);
+	atomic_set(&input->total_skip_sync_count, 0);
+	atomic_set(&input->total_aio_read_count, 0);
+	atomic_set(&input->total_aio_write_count, 0);
+}
+
+/***************** own brick * input * output operations *****************/
+
+/*  none */
+
+/*************** object * aspect constructors * destructors **************/
+
+static int if_aio_aspect_init_fn(struct generic_aspect *_ini)
+{
+	struct if_aio_aspect *ini = (void *)_ini;
+
+	INIT_LIST_HEAD(&ini->plug_head);
+	INIT_LIST_HEAD(&ini->hash_head);
+	return 0;
+}
+
+static void if_aio_aspect_exit_fn(struct generic_aspect *_ini)
+{
+	struct if_aio_aspect *ini = (void *)_ini;
+
+	CHECK_HEAD_EMPTY(&ini->plug_head);
+	CHECK_HEAD_EMPTY(&ini->hash_head);
+}
+
+XIO_MAKE_STATICS(if);
+
+/*********************** constructors * destructors ***********************/
+
+static int if_brick_construct(struct if_brick *brick)
+{
+	sema_init(&brick->switch_sem, 1);
+	atomic_set(&brick->open_count, 0);
+	return 0;
+}
+
+static int if_brick_destruct(struct if_brick *brick)
+{
+	return 0;
+}
+
+static int if_input_construct(struct if_input *input)
+{
+	int i;
+
+	input->hash_table = brick_block_alloc(0, PAGE_SIZE);
+	for (i = 0; i < IF_HASH_MAX; i++) {
+		spin_lock_init(&input->hash_table[i].hash_lock);
+		INIT_LIST_HEAD(&input->hash_table[i].hash_anchor);
+	}
+	INIT_LIST_HEAD(&input->plug_anchor);
+	spin_lock_init(&input->req_lock);
+	atomic_set(&input->flying_count, 0);
+	atomic_set(&input->read_flying_count, 0);
+	atomic_set(&input->write_flying_count, 0);
+	atomic_set(&input->plugged_count, 0);
+	return 0;
+}
+
+static int if_input_destruct(struct if_input *input)
+{
+	int i;
+
+	for (i = 0; i < IF_HASH_MAX; i++)
+		CHECK_HEAD_EMPTY(&input->hash_table[i].hash_anchor);
+	CHECK_HEAD_EMPTY(&input->plug_anchor);
+	brick_block_free(input->hash_table, PAGE_SIZE);
+	return 0;
+}
+
+static int if_output_construct(struct if_output *output)
+{
+	return 0;
+}
+
+/************************ static structs ***********************/
+
+static struct if_brick_ops if_brick_ops = {
+	.brick_switch = if_switch,
+	.brick_statistics = if_statistics,
+	.reset_statistics = if_reset_statistics,
+};
+
+static struct if_output_ops if_output_ops;
+
+const struct if_input_type if_input_type = {
+	.type_name = "if_input",
+	.input_size = sizeof(struct if_input),
+	.input_construct = &if_input_construct,
+	.input_destruct = &if_input_destruct,
+};
+
+static const struct if_input_type *if_input_types[] = {
+	&if_input_type,
+};
+
+const struct if_output_type if_output_type = {
+	.type_name = "if_output",
+	.output_size = sizeof(struct if_output),
+	.master_ops = &if_output_ops,
+	.output_construct = &if_output_construct,
+};
+
+static const struct if_output_type *if_output_types[] = {
+	&if_output_type,
+};
+
+const struct if_brick_type if_brick_type = {
+	.type_name = "if_brick",
+	.brick_size = sizeof(struct if_brick),
+	.max_inputs = 1,
+	.max_outputs = 0,
+	.master_ops = &if_brick_ops,
+	.aspect_types = if_aspect_types,
+	.default_input_types = if_input_types,
+	.default_output_types = if_output_types,
+	.brick_construct = &if_brick_construct,
+	.brick_destruct = &if_brick_destruct,
+};
+
+/***************** module init stuff ************************/
+
+void exit_xio_if(void)
+{
+	int status;
+
+	XIO_INF("exit_if()\n");
+	status = if_unregister_brick_type();
+	unregister_blkdev(XIO_MAJOR, "xio");
+}
+
+int __init init_xio_if(void)
+{
+	int status;
+
+	(void)if_aspect_types; /*  not used, shut up gcc */
+
+	XIO_INF("init_if()\n");
+	status = register_blkdev(XIO_MAJOR, "xio");
+	if (status)
+		return status;
+	status = if_register_brick_type();
+	if (status)
+		goto err_device;
+	return status;
+err_device:
+	XIO_ERR("init_if() status=%d\n", status);
+	exit_xio_if();
+	return status;
+}
include/linux/xio/xio_if.h new file +108 -0

108 lines not on this page · raw message

Signature · 1 line
2.6.4
#21 lore raw

RFC 21/31 mars: add new module xio_copy

In reply to Thomas Schoebel-Theuer
---
 drivers/staging/mars/xio_bricks/xio_copy.c | 1005 ++++++++++++++++++++++++++++
 include/linux/xio/xio_copy.h               |  115 ++++
 2 files changed, 1120 insertions(+)
 create mode 100644 drivers/staging/mars/xio_bricks/xio_copy.c
 create mode 100644 include/linux/xio/xio_copy.h
drivers/staging/mars/xio_bricks/xio_copy.c new file +1005 -0
new file mode 100644
index 0000000..aa5bc56
@@ -0,0 +1,1005 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+/*  Copy brick (just for demonstration) */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/string.h>
+
+#include <linux/xio/xio.h>
+#include <linux/brick/lib_limiter.h>
+
+#ifndef READ
+#define READ				0
+#define WRITE				1
+#endif
+
+#define COPY_CHUNK			(PAGE_SIZE)
+#define NR_COPY_REQUESTS		(32 * 1024 * 1024 / COPY_CHUNK)
+
+#define STATES_PER_PAGE			(PAGE_SIZE / sizeof(struct copy_state))
+#define MAX_SUB_TABLES			(NR_COPY_REQUESTS / STATES_PER_PAGE + (NR_COPY_REQUESTS % STATES_PER_PAGE ? 1 : 0)\
+									\
+)
+#define MAX_COPY_REQUESTS		(PAGE_SIZE / sizeof(struct copy_state *) * STATES_PER_PAGE)
+
+#define GET_STATE(brick, index)						\
+	((brick)->st[(index) / STATES_PER_PAGE][(index) % STATES_PER_PAGE])
+
+/************************ own type definitions ***********************/
+
+#include <linux/xio/xio_copy.h>
+
+int xio_copy_overlap = 1;
+
+int xio_copy_read_prio = XIO_PRIO_NORMAL;
+
+int xio_copy_write_prio = XIO_PRIO_NORMAL;
+
+int xio_copy_read_max_fly;
+
+int xio_copy_write_max_fly;
+
+#define is_read_limited(brick)						\
+	(xio_copy_read_max_fly > 0 && atomic_read(&(brick)->copy_read_flight) >= xio_copy_read_max_fly)
+
+#define is_write_limited(brick)						\
+	(xio_copy_write_max_fly > 0 && atomic_read(&(brick)->copy_write_flight) >= xio_copy_write_max_fly)
+
+/************************ own helper functions ***********************/
+
+/* TODO:
+ * The clash logic is untested / alpha stage (Feb. 2011).
+ *
+ * For now, the output is never used, so this cannot do harm.
+ *
+ * In order to get the output really working / enterprise grade,
+ * some larger test effort should be invested.
+ */
+static inline
+void _clash(struct copy_brick *brick)
+{
+	brick->trigger = true;
+	set_bit(0, &brick->clash);
+	atomic_inc(&brick->total_clash_count);
+	wake_up_interruptible(&brick->event);
+}
+
+static inline
+int _clear_clash(struct copy_brick *brick)
+{
+	int old;
+
+	old = test_and_clear_bit(0, &brick->clash);
+	return old;
+}
+
+/* Current semantics:
+ *
+ * All writes are always going to the original input A. They are _not_
+ * replicated to B.
+ *
+ * In order to get B really uptodate, you have to replay the right
+ * transaction logs there (at the right time).
+ * [If you had no writes on A at all during the copy, of course
+ * this is not necessary]
+ *
+ * When utilize_mode is on, reads can utilize the already copied
+ * region from B, but only as long as this region has not been
+ * invalidated by writes (indicated by low_dirty).
+ *
+ * TODO: implement replicated writes, together with some transaction
+ * replay logic applying the transaction logs _only_ after
+ * crashes during inconsistency caused by partial replication of writes.
+ */
+static
+int _determine_input(struct copy_brick *brick, struct aio_object *aio)
+{
+	int rw;
+	int below;
+	int behind;
+	loff_t io_end;
+
+	if (!brick->utilize_mode || brick->low_dirty)
+		return INPUT_A_IO;
+
+	io_end = aio->io_pos + aio->io_len;
+	below = io_end <= brick->copy_start;
+	behind = !brick->copy_end || aio->io_pos >= brick->copy_end;
+	rw = aio->io_may_write | aio->io_rw;
+	if (rw) {
+		if (!behind) {
+			brick->low_dirty = true;
+			if (!below) {
+				_clash(brick);
+				wake_up_interruptible(&brick->event);
+			}
+		}
+		return INPUT_A_IO;
+	}
+
+	if (below)
+		return INPUT_B_IO;
+
+	return INPUT_A_IO;
+}
+
+#define GET_INDEX(pos)	  (((pos) / COPY_CHUNK) % NR_COPY_REQUESTS)
+#define GET_OFFSET(pos)   ((pos) % COPY_CHUNK)
+
+static
+void __clear_aio(struct copy_brick *brick, struct aio_object *aio, int queue)
+{
+	struct copy_input *input;
+
+	input = queue ? brick->inputs[INPUT_B_COPY] : brick->inputs[INPUT_A_COPY];
+	GENERIC_INPUT_CALL(input, aio_put, aio);
+}
+
+static
+void _clear_aio(struct copy_brick *brick, int index, int queue)
+{
+	struct copy_state *st = &GET_STATE(brick, index);
+	struct aio_object *aio = st->table[queue];
+
+	if (aio) {
+		if (unlikely(st->active[queue])) {
+			XIO_ERR("clearing active aio, index = %d queue = %d\n", index, queue);
+			st->active[queue] = false;
+		}
+		__clear_aio(brick, aio, queue);
+		st->table[queue] = NULL;
+	}
+}
+
+static
+void _clear_all_aio(struct copy_brick *brick)
+{
+	int i;
+
+	for (i = 0; i < NR_COPY_REQUESTS; i++) {
+		GET_STATE(brick, i).state = COPY_STATE_START;
+		_clear_aio(brick, i, 0);
+		_clear_aio(brick, i, 1);
+	}
+}
+
+static
+void _clear_state_table(struct copy_brick *brick)
+{
+	int i;
+
+	for (i = 0; i < MAX_SUB_TABLES; i++) {
+		struct copy_state *sub_table = brick->st[i];
+
+		memset(sub_table, 0, PAGE_SIZE);
+	}
+}
+
+static
+void copy_endio(struct generic_callback *cb)
+{
+	struct copy_aio_aspect *aio_a;
+	struct aio_object *aio;
+	struct copy_brick *brick;
+	struct copy_state *st;
+	int index;
+	int queue;
+	int error = 0;
+
+	LAST_CALLBACK(cb);
+	aio_a = cb->cb_private;
+	CHECK_PTR(aio_a, err);
+	aio = aio_a->object;
+	CHECK_PTR(aio, err);
+	brick = aio_a->brick;
+	CHECK_PTR(brick, err);
+
+	queue = aio_a->queue;
+	index = GET_INDEX(aio->io_pos);
+	st = &GET_STATE(brick, index);
+
+	if (unlikely(queue < 0 || queue >= 2)) {
+		XIO_ERR("bad queue %d\n", queue);
+		error = -EINVAL;
+		goto exit;
+	}
+	st->active[queue] = false;
+	if (unlikely(st->table[queue])) {
+		XIO_ERR("table corruption at %d %d (%p => %p)\n", index, queue, st->table[queue], aio);
+		error = -EEXIST;
+		goto exit;
+	}
+	if (unlikely(cb->cb_error < 0)) {
+		error = cb->cb_error;
+		__clear_aio(brick, aio, queue);
+		/* This is racy, but does no harm.
+		 * Worst case just produces more error output.
+		 */
+		if (!brick->copy_error_count++)
+			XIO_WRN("IO error %d on index %d, old state = %d\n", cb->cb_error, index, st->state);
+	} else {
+		if (unlikely(st->table[queue])) {
+			XIO_ERR("overwriting index %d, state = %d\n", index, st->state);
+			_clear_aio(brick, index, queue);
+		}
+		st->table[queue] = aio;
+	}
+
+exit:
+	if (unlikely(error < 0)) {
+		st->error = error;
+		_clash(brick);
+	}
+	if (aio->io_rw)
+		atomic_dec(&brick->copy_write_flight);
+	else
+		atomic_dec(&brick->copy_read_flight);
+	brick->trigger = true;
+	wake_up_interruptible(&brick->event);
+	goto out_return;
+err:
+	XIO_FAT("cannot handle callback\n");
+out_return:;
+}
+
+static
+int _make_aio(struct copy_brick *brick,
+	int index,
+	int queue,
+	void *data,
+	loff_t pos,
+	loff_t end_pos,
+	int rw,
+	int cs_mode)
+{
+	struct aio_object *aio;
+	struct copy_aio_aspect *aio_a;
+	struct copy_input *input;
+	int offset;
+	int len;
+	int status = -EAGAIN;
+
+	if (brick->clash || end_pos <= 0)
+		goto done;
+
+	aio = copy_alloc_aio(brick);
+	status = -ENOMEM;
+
+	aio_a = copy_aio_get_aspect(brick, aio);
+	if (unlikely(!aio_a)) {
+		XIO_FAT("cannot get own apsect\n");
+		goto done;
+	}
+
+	aio_a->brick = brick;
+	aio_a->queue = queue;
+	aio->io_may_write = rw;
+	aio->io_rw = rw;
+	aio->io_data = data;
+	aio->io_pos = pos;
+	aio->io_cs_mode = cs_mode;
+	offset = GET_OFFSET(pos);
+	len = COPY_CHUNK - offset;
+	if (pos + len > end_pos)
+		len = end_pos - pos;
+	aio->io_len = len;
+	aio->io_prio = rw ?
+		xio_copy_write_prio :
+		xio_copy_read_prio;
+	if (aio->io_prio < XIO_PRIO_HIGH || aio->io_prio > XIO_PRIO_LOW)
+		aio->io_prio = brick->io_prio;
+
+	SETUP_CALLBACK(aio, copy_endio, aio_a);
+
+	input = queue ? brick->inputs[INPUT_B_COPY] : brick->inputs[INPUT_A_COPY];
+	status = GENERIC_INPUT_CALL(input, aio_get, aio);
+	if (unlikely(status < 0)) {
+		XIO_ERR("status = %d\n", status);
+		obj_free(aio);
+		goto done;
+	}
+	if (unlikely(aio->io_len < len))
+		XIO_DBG("shorten len %d < %d\n", aio->io_len, len);
+	if (queue == 0) {
+		GET_STATE(brick, index).len = aio->io_len;
+	} else if (unlikely(aio->io_len < GET_STATE(brick, index).len)) {
+		XIO_DBG("shorten len %d < %d at index %d\n", aio->io_len, GET_STATE(brick, index).len, index);
+		GET_STATE(brick, index).len = aio->io_len;
+	}
+
+	GET_STATE(brick, index).active[queue] = true;
+	if (rw)
+		atomic_inc(&brick->copy_write_flight);
+	else
+		atomic_inc(&brick->copy_read_flight);
+	GENERIC_INPUT_CALL(input, aio_io, aio);
+
+done:
+	return status;
+}
+
+static
+void _update_percent(struct copy_brick *brick, bool force)
+{
+	if (force
+	   || brick->copy_last > brick->copy_start + 8 * 1024 * 1024
+	   || time_is_before_jiffies(brick->last_jiffies + 5 * HZ)
+	   || (brick->copy_last == brick->copy_end && brick->copy_end > 0)) {
+		brick->copy_start = brick->copy_last;
+		brick->last_jiffies = jiffies;
+		brick->power.percent_done = brick->copy_end > 0 ? brick->copy_start * 100 / brick->copy_end : 0;
+		XIO_INF("'%s' copied %lld / %lld bytes (%d%%)\n",
+			brick->brick_path,
+			brick->copy_last,
+			brick->copy_end,
+			brick->power.percent_done);
+	}
+}
+
+/* The heart of this brick.
+ * State transition function of the finite automaton.
+ * In case no progress is possible (e.g. preconditions not
+ * yet true), the state is left as is (idempotence property:
+ * calling this too often does no harm, just costs performance).
+ */
+static
+int _next_state(struct copy_brick *brick, int index, loff_t pos)
+{
+	struct aio_object *aio0;
+	struct aio_object *aio1;
+	struct copy_state *st;
+	char state;
+	char next_state;
+	bool do_restart = false;
+	int progress = 0;
+	int status;
+
+	st = &GET_STATE(brick, index);
+	next_state = st->state;
+
+restart:
+	state = next_state;
+
+	do_restart = false;
+
+	switch (state) {
+	case COPY_STATE_RESET:
+		/* This state is only entered after errors or
+		 * in restarting situations.
+		 */
+		_clear_aio(brick, index, 1);
+		_clear_aio(brick, index, 0);
+		next_state = COPY_STATE_START;
+		/* fallthrough */
+	case COPY_STATE_START:
+		/* This is the relgular starting state.
+		 * It must be zero, automatically entered via memset()
+		 */
+		if (st->table[0] || st->table[1]) {
+			XIO_ERR("index %d not startable\n", index);
+			progress = -EPROTO;
+			goto idle;
+		}
+
+		_clear_aio(brick, index, 1);
+		_clear_aio(brick, index, 0);
+		st->writeout = false;
+		st->error = 0;
+
+		if (brick->is_aborting ||
+		    is_read_limited(brick))
+			goto idle;
+
+		status = _make_aio(brick, index, 0, NULL, pos, brick->copy_end, READ, brick->verify_mode ? 2 : 0);
+		if (unlikely(status < 0)) {
+			XIO_DBG("status = %d\n", status);
+			progress = status;
+			break;
+		}
+
+		next_state = COPY_STATE_READ1;
+		if (!brick->verify_mode)
+			break;
+
+		next_state = COPY_STATE_START2;
+		/* fallthrough */
+	case COPY_STATE_START2:
+		status = _make_aio(brick, index, 1, NULL, pos, brick->copy_end, READ, 2);
+		if (unlikely(status < 0)) {
+			XIO_DBG("status = %d\n", status);
+			progress = status;
+			break;
+		}
+		next_state = COPY_STATE_READ2;
+		/* fallthrough */
+	case COPY_STATE_READ2:
+		aio1 = st->table[1];
+		if (!aio1) { /*  idempotence: wait by unchanged state */
+			goto idle;
+		}
+		/* fallthrough = > wait for both aios to appear */
+	case COPY_STATE_READ1:
+	case COPY_STATE_READ3:
+		aio0 = st->table[0];
+		if (!aio0) { /*  idempotence: wait by unchanged state */
+			goto idle;
+		}
+		if (brick->copy_limiter) {
+			int amount = (aio0->io_len - 1) / 1024 + 1;
+
+			rate_limit_sleep(brick->copy_limiter, amount);
+		}
+		/*  on append mode: increase the end pointer dynamically */
+		if (brick->append_mode > 0 && aio0->io_total_size && aio0->io_total_size > brick->copy_end)
+			brick->copy_end = aio0->io_total_size;
+		/*  do verify (when applicable) */
+		aio1 = st->table[1];
+		if (aio1 && state != COPY_STATE_READ3) {
+			int len = aio0->io_len;
+			bool ok;
+
+			if (len != aio1->io_len) {
+				ok = false;
+			} else if (aio0->io_cs_mode) {
+				static unsigned char null[sizeof(aio0->io_checksum)];
+
+				ok = !memcmp(aio0->io_checksum, aio1->io_checksum, sizeof(aio0->io_checksum));
+				if (ok)
+					ok = memcmp(aio0->io_checksum, null, sizeof(aio0->io_checksum)) != 0;
+			} else if (!aio0->io_data || !aio1->io_data) {
+				ok = false;
+			} else {
+				ok = !memcmp(aio0->io_data, aio1->io_data, len);
+			}
+
+			_clear_aio(brick, index, 1);
+
+			if (ok)
+				brick->verify_ok_count++;
+			else
+				brick->verify_error_count++;
+
+			if (ok || !brick->repair_mode) {
+				/* skip start of writing, goto final treatment of writeout */
+				next_state = COPY_STATE_CLEANUP;
+				break;
+			}
+		}
+
+		if (aio0->io_cs_mode > 1) { /*	re-read, this time with data */
+			_clear_aio(brick, index, 0);
+			status = _make_aio(brick, index, 0, NULL, pos, brick->copy_end, READ, 0);
+			if (unlikely(status < 0)) {
+				XIO_DBG("status = %d\n", status);
+				progress = status;
+				next_state = COPY_STATE_RESET;
+				break;
+			}
+			next_state = COPY_STATE_READ3;
+			break;
+		}
+		next_state = COPY_STATE_WRITE;
+		/* fallthrough */
+	case COPY_STATE_WRITE:
+		if (is_write_limited(brick))
+			goto idle;
+		/* Obey ordering to get a strict "append" behaviour.
+		 * We assume that we don't need to wait for completion
+		 * of the previous write to avoid a sparse result file
+		 * under all circumstances, i.e. we only assure that
+		 * _starting_ the writes is in order.
+		 * This is only correct when all lower bricks obey the
+		 * order of io_io() operations.
+		 * Currenty, bio and aio are obeying this. Be careful when
+		 * implementing new IO bricks!
+		 */
+		if (st->prev >= 0 && !GET_STATE(brick, st->prev).writeout)
+			goto idle;
+		aio0 = st->table[0];
+		if (unlikely(!aio0 || !aio0->io_data)) {
+			XIO_ERR("src buffer for write does not exist, state %d at index %d\n", state, index);
+			progress = -EILSEQ;
+			break;
+		}
+		if (unlikely(brick->is_aborting)) {
+			progress = -EINTR;
+			break;
+		}
+		/* start writeout */
+		status = _make_aio(brick, index, 1, aio0->io_data, pos, pos + aio0->io_len, WRITE, 0);
+		if (unlikely(status < 0)) {
+			XIO_DBG("status = %d\n", status);
+			progress = status;
+			next_state = COPY_STATE_RESET;
+			break;
+		}
+		/* Attention! overlapped IO behind EOF could
+		 * lead to temporary inconsistent state of the
+		 * file, because the write order may be different from
+		 * strict O_APPEND behaviour.
+		 */
+		if (xio_copy_overlap)
+			st->writeout = true;
+		next_state = COPY_STATE_WRITTEN;
+		/* fallthrough */
+	case COPY_STATE_WRITTEN:
+		aio1 = st->table[1];
+		if (!aio1) { /*  idempotence: wait by unchanged state */
+			goto idle;
+		}
+		st->writeout = true;
+		/* rechecking means to start over again.
+		 * ATTENTIION! this may lead to infinite request
+		 * submission loops, intentionally.
+		 * TODO: implement some timeout means.
+		 */
+		if (brick->recheck_mode && brick->repair_mode) {
+			next_state = COPY_STATE_RESET;
+			break;
+		}
+		next_state = COPY_STATE_CLEANUP;
+		/* fallthrough */
+	case COPY_STATE_CLEANUP:
+		_clear_aio(brick, index, 1);
+		_clear_aio(brick, index, 0);
+		next_state = COPY_STATE_FINISHED;
+		/* fallthrough */
+	case COPY_STATE_FINISHED:
+		/* Indicate successful completion by remaining in this state.
+		 * Restart of the finite automaton must be done externally.
+		 */
+		goto idle;
+	default:
+		XIO_ERR("illegal state %d at index %d\n", state, index);
+		_clash(brick);
+		progress = -EILSEQ;
+	}
+
+	do_restart = (state != next_state);
+
+idle:
+	if (unlikely(progress < 0)) {
+		if (st->error >= 0)
+			st->error = progress;
+		XIO_DBG("progress = %d\n", progress);
+		progress = 0;
+		_clash(brick);
+	} else if (do_restart) {
+		goto restart;
+	} else if (st->state != next_state) {
+		progress++;
+	}
+
+	/*  save the resulting state */
+	st->state = next_state;
+	return progress;
+}
+
+static
+int _run_copy(struct copy_brick *brick)
+{
+	int max;
+	loff_t pos;
+	loff_t limit = -1;
+
+	short prev;
+	int progress;
+
+	if (unlikely(_clear_clash(brick))) {
+		XIO_DBG("clash\n");
+		if (atomic_read(&brick->copy_read_flight) + atomic_read(&brick->copy_write_flight) > 0) {
+			/* wait until all pending copy IO has finished
+			 */
+			_clash(brick);
+			XIO_DBG("re-clash\n");
+			brick_msleep(100);
+			return 0;
+		}
+		_clear_all_aio(brick);
+		_clear_state_table(brick);
+	}
+
+	/* Do at most max iterations in the below loop
+	 */
+	max = NR_COPY_REQUESTS - atomic_read(&brick->io_flight) * 2;
+
+	prev = -1;
+	progress = 0;
+	for (pos = brick->copy_last; pos < brick->copy_end || brick->append_mode > 1; pos = ((pos / COPY_CHUNK) + 1) * COPY_CHUNK) {
+		int index = GET_INDEX(pos);
+		struct copy_state *st = &GET_STATE(brick, index);
+
+		if (max-- <= 0)
+			break;
+		st->prev = prev;
+		prev = index;
+		/*  call the finite state automaton */
+		if (!(st->active[0] | st->active[1])) {
+			progress += _next_state(brick, index, pos);
+			limit = pos;
+		}
+	}
+
+	/*  check the resulting state: can we advance the copy_last pointer? */
+	if (likely(progress && !brick->clash)) {
+		int count = 0;
+
+		for (pos = brick->copy_last; pos <= limit; pos = ((pos / COPY_CHUNK) + 1) * COPY_CHUNK) {
+			int index = GET_INDEX(pos);
+			struct copy_state *st = &GET_STATE(brick, index);
+
+			if (st->state != COPY_STATE_FINISHED)
+				break;
+			if (unlikely(st->error < 0)) {
+				/* check for fatal consistency errors */
+				if (st->error == -EMEDIUMTYPE) {
+					brick->copy_error = st->error;
+					brick->abort_mode = true;
+					XIO_WRN("Consistency is violated\n");
+				}
+				if (!brick->copy_error) {
+					brick->copy_error = st->error;
+					XIO_WRN("IO error = %d\n", st->error);
+				}
+				if (brick->abort_mode)
+					brick->is_aborting = true;
+				break;
+			}
+			/*  rollover */
+			st->state = COPY_STATE_START;
+			count += st->len;
+			/*  check contiguity */
+			if (unlikely(GET_OFFSET(pos) + st->len != COPY_CHUNK))
+				break;
+		}
+		if (count > 0) {
+			brick->copy_last += count;
+			get_lamport(&brick->copy_last_stamp);
+			_update_percent(brick, false);
+		}
+	}
+	return progress;
+}
+
+static
+bool _is_done(struct copy_brick *brick)
+{
+	if (brick_thread_should_stop())
+		brick->is_aborting = true;
+	return brick->is_aborting &&
+		atomic_read(&brick->copy_read_flight) + atomic_read(&brick->copy_write_flight) <= 0;
+}
+
+static int _copy_thread(void *data)
+{
+	struct copy_brick *brick = data;
+	int rounds = 0;
+
+	XIO_DBG("--------------- copy_thread %p starting\n", brick);
+	brick->copy_error = 0;
+	brick->copy_error_count = 0;
+	brick->verify_ok_count = 0;
+	brick->verify_error_count = 0;
+
+	_update_percent(brick, true);
+
+	xio_set_power_on_led((void *)brick, true);
+	brick->trigger = true;
+
+	while (!_is_done(brick)) {
+		loff_t old_start = brick->copy_start;
+		loff_t old_end = brick->copy_end;
+		int progress = 0;
+
+		if (old_end > 0) {
+			progress = _run_copy(brick);
+			if (!progress || ++rounds > 1000)
+				rounds = 0;
+		}
+
+		wait_event_interruptible_timeout(brick->event,
+						 progress > 0 ||
+						 brick->trigger ||
+						 brick->copy_start != old_start ||
+						 brick->copy_end != old_end ||
+						 _is_done(brick),
+						 1 * HZ);
+		brick->trigger = false;
+	}
+
+	/* check for fatal consistency errors */
+	if (brick->copy_error == -EMEDIUMTYPE) {
+		/* reset the whole area */
+		brick->copy_start = 0;
+		brick->copy_last = 0;
+		XIO_WRN("resetting the full copy area\n");
+	}
+	_update_percent(brick, true);
+
+	XIO_DBG("--------------- copy_thread terminating (%d read requests / %d write requests flying, copy_start = %lld copy_end = %lld)\n",
+		 atomic_read(&brick->copy_read_flight),
+		 atomic_read(&brick->copy_write_flight),
+		 brick->copy_start,
+		 brick->copy_end);
+
+	_clear_all_aio(brick);
+	xio_set_power_off_led((void *)brick, true);
+	XIO_DBG("--------------- copy_thread done.\n");
+	return 0;
+}
+
+/***************** own brick * input * output operations *****************/
+
+static int copy_get_info(struct copy_output *output, struct xio_info *info)
+{
+	struct copy_input *input = output->brick->inputs[INPUT_B_IO];
+
+	return GENERIC_INPUT_CALL(input, xio_get_info, info);
+}
+
+static int copy_io_get(struct copy_output *output, struct aio_object *aio)
+{
+	struct copy_input *input;
+	int index;
+	int status;
+
+	index = _determine_input(output->brick, aio);
+	input = output->brick->inputs[index];
+	status = GENERIC_INPUT_CALL(input, aio_get, aio);
+	if (status >= 0)
+		atomic_inc(&output->brick->io_flight);
+	return status;
+}
+
+static void copy_io_put(struct copy_output *output, struct aio_object *aio)
+{
+	struct copy_input *input;
+	int index;
+
+	index = _determine_input(output->brick, aio);
+	input = output->brick->inputs[index];
+	GENERIC_INPUT_CALL(input, aio_put, aio);
+	if (atomic_dec_and_test(&output->brick->io_flight)) {
+		output->brick->trigger = true;
+		wake_up_interruptible(&output->brick->event);
+	}
+}
+
+static void copy_io_io(struct copy_output *output, struct aio_object *aio)
+{
+	struct copy_input *input;
+	int index;
+
+	index = _determine_input(output->brick, aio);
+	input = output->brick->inputs[index];
+	GENERIC_INPUT_CALL(input, aio_io, aio);
+}
+
+static int copy_switch(struct copy_brick *brick)
+{
+	static int version;
+
+	XIO_DBG("power.button = %d\n", brick->power.button);
+	if (brick->power.button) {
+		if (brick->power.on_led)
+			goto done;
+		xio_set_power_off_led((void *)brick, false);
+		brick->is_aborting = false;
+		if (!brick->thread) {
+			brick->copy_last = brick->copy_start;
+			get_lamport(&brick->copy_last_stamp);
+			brick->thread = brick_thread_create(_copy_thread, brick, "xio_copy%d", version++);
+			if (brick->thread) {
+				brick->trigger = true;
+			} else {
+				xio_set_power_off_led((void *)brick, true);
+				XIO_ERR("could not start copy thread\n");
+			}
+		}
+	} else {
+		if (brick->power.off_led)
+			goto done;
+		xio_set_power_on_led((void *)brick, false);
+		if (brick->thread) {
+			XIO_INF("stopping thread...\n");
+			brick_thread_stop(brick->thread);
+		}
+	}
+done:
+	return 0;
+}
+
+/*************** informational * statistics **************/
+
+static
+char *copy_statistics(struct copy_brick *brick, int verbose)
+{
+	char *res = brick_string_alloc(1024);
+
+	snprintf(res, 1024,
+		 "copy_start = %lld copy_last = %lld copy_end = %lld copy_error = %d copy_error_count = %d verify_ok_count = %d verify_error_count = %d low_dirty = %d is_aborting = %d clash = %lu | total clash_count = %d | io_flight = %d copy_read_flight = %d copy_write_flight = %d\n",
+		 brick->copy_start,
+		 brick->copy_last,
+		 brick->copy_end,
+		 brick->copy_error,
+		 brick->copy_error_count,
+		 brick->verify_ok_count,
+		 brick->verify_error_count,
+		 brick->low_dirty,
+		 brick->is_aborting,
+		 brick->clash,
+		 atomic_read(&brick->total_clash_count),
+		 atomic_read(&brick->io_flight),
+		 atomic_read(&brick->copy_read_flight),
+		 atomic_read(&brick->copy_write_flight));
+
+	return res;
+}
+
+static
+void copy_reset_statistics(struct copy_brick *brick)
+{
+	atomic_set(&brick->total_clash_count, 0);
+}
+
+/*************** object * aspect constructors * destructors **************/
+
+static int copy_aio_aspect_init_fn(struct generic_aspect *_ini)
+{
+	struct copy_aio_aspect *ini = (void *)_ini;
+
+	(void)ini;
+	return 0;
+}
+
+static void copy_aio_aspect_exit_fn(struct generic_aspect *_ini)
+{
+	struct copy_aio_aspect *ini = (void *)_ini;
+
+	(void)ini;
+}
+
+XIO_MAKE_STATICS(copy);
+
+/********************* brick constructors * destructors *******************/
+
+static
+void _free_pages(struct copy_brick *brick)
+{
+	int i;
+
+	for (i = 0; i < MAX_SUB_TABLES; i++) {
+		struct copy_state *sub_table = brick->st[i];
+
+		if (!sub_table)
+			continue;
+
+		brick_block_free(sub_table, PAGE_SIZE);
+	}
+	brick_block_free(brick->st, PAGE_SIZE);
+}
+
+static int copy_brick_construct(struct copy_brick *brick)
+{
+	int i;
+
+	brick->st = brick_block_alloc(0, PAGE_SIZE);
+	memset(brick->st, 0, PAGE_SIZE);
+
+	for (i = 0; i < MAX_SUB_TABLES; i++) {
+		struct copy_state *sub_table;
+
+		/*  this should be usually optimized away as dead code */
+		if (unlikely(i >= MAX_SUB_TABLES)) {
+			XIO_ERR("sorry, subtable index %d is too large.\n", i);
+			_free_pages(brick);
+			return -EINVAL;
+		}
+
+		sub_table = brick_block_alloc(0, PAGE_SIZE);
+		brick->st[i] = sub_table;
+		memset(sub_table, 0, PAGE_SIZE);
+	}
+
+	init_waitqueue_head(&brick->event);
+	sema_init(&brick->mutex, 1);
+	return 0;
+}
+
+static int copy_brick_destruct(struct copy_brick *brick)
+{
+	_free_pages(brick);
+	return 0;
+}
+
+static int copy_output_construct(struct copy_output *output)
+{
+	return 0;
+}
+
+static int copy_output_destruct(struct copy_output *output)
+{
+	return 0;
+}
+
+/************************ static structs ***********************/
+
+static struct copy_brick_ops copy_brick_ops = {
+	.brick_switch = copy_switch,
+	.brick_statistics = copy_statistics,
+	.reset_statistics = copy_reset_statistics,
+};
+
+static struct copy_output_ops copy_output_ops = {
+	.xio_get_info = copy_get_info,
+	.aio_get = copy_io_get,
+	.aio_put = copy_io_put,
+	.aio_io = copy_io_io,
+};
+
+const struct copy_input_type copy_input_type = {
+	.type_name = "copy_input",
+	.input_size = sizeof(struct copy_input),
+};
+
+static const struct copy_input_type *copy_input_types[] = {
+	&copy_input_type,
+	&copy_input_type,
+	&copy_input_type,
+	&copy_input_type,
+};
+
+const struct copy_output_type copy_output_type = {
+	.type_name = "copy_output",
+	.output_size = sizeof(struct copy_output),
+	.master_ops = &copy_output_ops,
+	.output_construct = &copy_output_construct,
+	.output_destruct = &copy_output_destruct,
+};
+
+static const struct copy_output_type *copy_output_types[] = {
+	&copy_output_type,
+};
+
+const struct copy_brick_type copy_brick_type = {
+	.type_name = "copy_brick",
+	.brick_size = sizeof(struct copy_brick),
+	.max_inputs = 4,
+	.max_outputs = 1,
+	.master_ops = &copy_brick_ops,
+	.aspect_types = copy_aspect_types,
+	.default_input_types = copy_input_types,
+	.default_output_types = copy_output_types,
+	.brick_construct = &copy_brick_construct,
+	.brick_destruct = &copy_brick_destruct,
+};
+
+/***************** module init stuff ************************/
+
+int __init init_xio_copy(void)
+{
+	XIO_INF("init_copy()\n");
+	return copy_register_brick_type();
+}
+
+void exit_xio_copy(void)
+{
+	XIO_INF("exit_copy()\n");
+	copy_unregister_brick_type();
+}
include/linux/xio/xio_copy.h new file +115 -0

115 lines not on this page · raw message

Signature · 1 line
2.6.4
#22 lore raw

RFC 22/31 mars: add new module xio_trans_logger

In reply to Thomas Schoebel-Theuer
---
 drivers/staging/mars/xio_bricks/xio_trans_logger.c | 3309 ++++++++++++++++++++
 include/linux/xio/xio_trans_logger.h               |  263 ++
 2 files changed, 3572 insertions(+)
 create mode 100644 drivers/staging/mars/xio_bricks/xio_trans_logger.c
 create mode 100644 include/linux/xio/xio_trans_logger.h
drivers/staging/mars/xio_bricks/xio_trans_logger.c new file +3309 -0
new file mode 100644
index 0000000..04d4c63
@@ -0,0 +1,3309 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+/*  Trans_Logger brick */
+
+#define XIO_DEBUGGING
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/string.h>
+#include <linux/bio.h>
+
+#include <linux/xio/xio.h>
+#include <linux/brick/lib_rank.h>
+#include <linux/brick/lib_limiter.h>
+
+#include <linux/xio/xio_trans_logger.h>
+
+/*  variants */
+#define KEEP_UNIQUE
+#define DELAY_CALLERS			/*  this is _needed_ for production systems */
+/* When possible, queue 1 executes phase3_startio() directly without
+ * intermediate queueing into queue 3 = > may be irritating, but has better
+ * performance. NOTICE: when some day the IO scheduling should be
+ * different between queue 1 and 3, you MUST disable this in order
+ * to distinguish between them!
+ */
+#define SHORTCUT_1_to_3
+
+/*  commenting this out is dangerous for data integrity! use only for testing! */
+#define USE_MEMCPY
+#define DO_WRITEBACK			/*  otherwise FAKE IO */
+#define REPLAY_DATA
+
+/*  tuning */
+#ifdef BRICK_DEBUG_MEM
+#define CONF_TRANS_CHUNKSIZE		(128 * 1024 - PAGE_SIZE * 2)
+#else
+#define CONF_TRANS_CHUNKSIZE		(128 * 1024)
+#endif
+#define CONF_TRANS_MAX_AIO_SIZE		PAGE_SIZE
+#define CONF_TRANS_ALIGN		0
+
+#define XIO_RPL(_args...) /*empty*/
+
+struct trans_logger_hash_anchor {
+	struct rw_semaphore hash_mutex;
+	struct list_head hash_anchor;
+};
+
+#define NR_HASH_PAGES			64
+
+#define MAX_HASH_PAGES			(PAGE_SIZE / sizeof(struct trans_logger_hash_anchor *))
+#define HASH_PER_PAGE			(PAGE_SIZE / sizeof(struct trans_logger_hash_anchor))
+#define HASH_TOTAL			(NR_HASH_PAGES * HASH_PER_PAGE)
+
+/************************ global tuning ***********************/
+
+int trans_logger_completion_semantics = 1;
+
+int trans_logger_do_crc =
+#ifdef CONFIG_MARS_DEBUG
+	true;
+#else
+	false;
+#endif
+
+int trans_logger_mem_usage; /*	in KB */
+
+int trans_logger_max_interleave = -1;
+
+int trans_logger_resume = 1;
+
+int trans_logger_replay_timeout = 1; /*  in s */
+
+struct writeback_group global_writeback = {
+	.lock = __RW_LOCK_UNLOCKED(global_writeback.lock),
+	.group_anchor = LIST_HEAD_INIT(global_writeback.group_anchor),
+	.until_percent = 30,
+};
+
+static
+void add_to_group(struct writeback_group *gr, struct trans_logger_brick *brick)
+{
+	unsigned long flags;
+
+	write_lock_irqsave(&gr->lock, flags);
+	list_add_tail(&brick->group_head, &gr->group_anchor);
+	write_unlock_irqrestore(&gr->lock, flags);
+}
+
+static
+void remove_from_group(struct writeback_group *gr, struct trans_logger_brick *brick)
+{
+	unsigned long flags;
+
+	write_lock_irqsave(&gr->lock, flags);
+	list_del_init(&brick->group_head);
+	gr->leader = NULL;
+	write_unlock_irqrestore(&gr->lock, flags);
+}
+
+static
+struct trans_logger_brick *elect_leader(struct writeback_group *gr)
+{
+	struct trans_logger_brick *res = gr->leader;
+	struct list_head *tmp;
+	unsigned long flags;
+
+	if (res && gr->until_percent >= 0) {
+		loff_t used = atomic64_read(&res->shadow_mem_used);
+
+		if (used > gr->biggest * gr->until_percent / 100)
+			goto done;
+	}
+
+	read_lock_irqsave(&gr->lock, flags);
+	for (tmp = gr->group_anchor.next; tmp != &gr->group_anchor; tmp = tmp->next) {
+		struct trans_logger_brick *test = container_of(tmp, struct trans_logger_brick, group_head);
+		loff_t new_used = atomic64_read(&test->shadow_mem_used);
+
+		if (!res || new_used > atomic64_read(&res->shadow_mem_used)) {
+			res = test;
+			gr->biggest = new_used;
+		}
+	}
+	read_unlock_irqrestore(&gr->lock, flags);
+
+	gr->leader = res;
+
+done:
+	return res;
+}
+
+/************************ own type definitions ***********************/
+
+static inline
+int lh_cmp(loff_t *a, loff_t *b)
+{
+	if (*a < *b)
+		return -1;
+	if (*a > *b)
+		return 1;
+	return 0;
+}
+
+static inline
+int tr_cmp(struct pairing_heap_logger *_a, struct pairing_heap_logger *_b)
+{
+	struct logger_head *a = container_of(_a, struct logger_head, ph);
+	struct logger_head *b = container_of(_b, struct logger_head, ph);
+
+	return lh_cmp(a->lh_pos, b->lh_pos);
+}
+
+_PAIRING_HEAP_FUNCTIONS(static, logger, tr_cmp);
+
+static inline
+loff_t *lh_get(struct logger_head *th)
+{
+	return th->lh_pos;
+}
+
+QUEUE_FUNCTIONS(logger, struct logger_head, lh_head, lh_get, lh_cmp, logger);
+
+/************************* logger queue handling ***********************/
+
+static inline
+void qq_init(struct logger_queue *q, struct trans_logger_brick *brick)
+{
+	q_logger_init(q);
+	q->q_event = &brick->worker_event;
+	q->q_brick = brick;
+}
+
+static inline
+void qq_inc_flying(struct logger_queue *q)
+{
+	q_logger_inc_flying(q);
+}
+
+static inline
+void qq_dec_flying(struct logger_queue *q)
+{
+	q_logger_dec_flying(q);
+}
+
+static inline
+void qq_aio_insert(struct logger_queue *q, struct trans_logger_aio_aspect *aio_a)
+{
+	struct aio_object *aio = aio_a->object;
+
+	obj_get(aio); /*  must be paired with __trans_logger_io_put() */
+	atomic_inc(&q->q_brick->inner_balance_count);
+
+	q_logger_insert(q, &aio_a->lh);
+}
+
+static inline
+void qq_wb_insert(struct logger_queue *q, struct writeback_info *wb)
+{
+	q_logger_insert(q, &wb->w_lh);
+}
+
+static inline
+void qq_aio_pushback(struct logger_queue *q, struct trans_logger_aio_aspect *aio_a)
+{
+	obj_check(aio_a->object);
+
+	q->pushback_count++;
+
+	q_logger_pushback(q, &aio_a->lh);
+}
+
+static inline
+void qq_wb_pushback(struct logger_queue *q, struct writeback_info *wb)
+{
+	q->pushback_count++;
+	q_logger_pushback(q, &wb->w_lh);
+}
+
+static inline
+struct trans_logger_aio_aspect *qq_aio_fetch(struct logger_queue *q)
+{
+	struct logger_head *test;
+	struct trans_logger_aio_aspect *aio_a = NULL;
+
+	test = q_logger_fetch(q);
+
+	if (test) {
+		aio_a = container_of(test, struct trans_logger_aio_aspect, lh);
+		obj_check(aio_a->object);
+	}
+	return aio_a;
+}
+
+static inline
+struct writeback_info *qq_wb_fetch(struct logger_queue *q)
+{
+	struct logger_head *test;
+	struct writeback_info *res = NULL;
+
+	test = q_logger_fetch(q);
+
+	if (test)
+		res = container_of(test, struct writeback_info, w_lh);
+	return res;
+}
+
+/************************ own helper functions ***********************/
+
+static inline
+int hash_fn(loff_t pos)
+{
+	/*  simple and stupid */
+	long base_index = pos >> REGION_SIZE_BITS;
+
+	base_index += base_index / HASH_TOTAL / 7;
+	return base_index % HASH_TOTAL;
+}
+
+static inline
+struct trans_logger_aio_aspect *_hash_find(struct list_head *start,
+	loff_t pos,
+	int *max_len,
+	bool use_collect_head,
+	bool find_unstable)
+{
+	struct list_head *tmp;
+	struct trans_logger_aio_aspect *res = NULL;
+	int len = *max_len;
+
+	/* The lists are always sorted according to age (newest first).
+	 * Caution: there may be duplicates in the list, some of them
+	 * overlapping with the search area in many different ways.
+	 */
+	for (tmp = start->next; tmp != start; tmp = tmp->next) {
+		struct trans_logger_aio_aspect *test_a;
+		struct aio_object *test;
+		int diff;
+
+		if (use_collect_head)
+			test_a = container_of(tmp, struct trans_logger_aio_aspect, collect_head);
+		else
+			test_a = container_of(tmp, struct trans_logger_aio_aspect, hash_head);
+		test = test_a->object;
+
+		obj_check(test);
+
+		/*  are the regions overlapping? */
+		if (pos >= test->io_pos + test->io_len || pos + len <= test->io_pos)
+			continue; /*  not relevant */
+
+		/*  searching for unstable elements (only in special cases) */
+		if (find_unstable && test_a->is_stable)
+			break;
+
+		diff = test->io_pos - pos;
+		if (diff <= 0) {
+			int restlen = test->io_len + diff;
+
+			res = test_a;
+			if (restlen < len)
+				len = restlen;
+			break;
+		}
+		if (diff < len)
+			len = diff;
+	}
+
+	*max_len = len;
+	return res;
+}
+
+static
+struct trans_logger_aio_aspect *hash_find(struct trans_logger_brick *brick,
+	loff_t pos,
+	int *max_len,
+	bool find_unstable)
+{
+
+	int hash = hash_fn(pos);
+	struct trans_logger_hash_anchor *sub_table = brick->hash_table[hash / HASH_PER_PAGE];
+	struct trans_logger_hash_anchor *start = &sub_table[hash % HASH_PER_PAGE];
+	struct trans_logger_aio_aspect *res;
+
+	atomic_inc(&brick->total_hash_find_count);
+
+	down_read(&start->hash_mutex);
+
+	res = _hash_find(&start->hash_anchor, pos, max_len, false, find_unstable);
+
+	/* Ensure the found aio can't go away...
+	 */
+	if (res && res->object)
+		obj_get(res->object);
+
+	up_read(&start->hash_mutex);
+
+	return res;
+}
+
+static
+void hash_insert(struct trans_logger_brick *brick, struct trans_logger_aio_aspect *elem_a)
+{
+	int hash = hash_fn(elem_a->object->io_pos);
+	struct trans_logger_hash_anchor *sub_table = brick->hash_table[hash / HASH_PER_PAGE];
+	struct trans_logger_hash_anchor *start = &sub_table[hash % HASH_PER_PAGE];
+
+	CHECK_HEAD_EMPTY(&elem_a->hash_head);
+	obj_check(elem_a->object);
+
+	/*  only for statistics: */
+	atomic_inc(&brick->hash_count);
+	atomic_inc(&brick->total_hash_insert_count);
+
+	down_write(&start->hash_mutex);
+
+	list_add(&elem_a->hash_head, &start->hash_anchor);
+	elem_a->is_hashed = true;
+
+	up_write(&start->hash_mutex);
+}
+
+/* Find the transitive closure of overlapping requests
+ * and collect them into a list.
+ */
+static
+void hash_extend(struct trans_logger_brick *brick, loff_t *_pos, int *_len, struct list_head *collect_list)
+{
+	loff_t pos = *_pos;
+	int len = *_len;
+	int hash = hash_fn(pos);
+	struct trans_logger_hash_anchor *sub_table = brick->hash_table[hash / HASH_PER_PAGE];
+	struct trans_logger_hash_anchor *start = &sub_table[hash % HASH_PER_PAGE];
+	struct list_head *tmp;
+	bool extended;
+
+	if (collect_list)
+		CHECK_HEAD_EMPTY(collect_list);
+
+	atomic_inc(&brick->total_hash_extend_count);
+
+	down_read(&start->hash_mutex);
+
+	do {
+		extended = false;
+
+		for (tmp = start->hash_anchor.next; tmp != &start->hash_anchor; tmp = tmp->next) {
+			struct trans_logger_aio_aspect *test_a;
+			struct aio_object *test;
+			loff_t diff;
+
+			test_a = container_of(tmp, struct trans_logger_aio_aspect, hash_head);
+			test = test_a->object;
+
+			obj_check(test);
+
+			/*  are the regions overlapping? */
+			if (pos >= test->io_pos + test->io_len || pos + len <= test->io_pos)
+				continue; /*  not relevant */
+
+			/*  collision detection */
+			if (test_a->is_collected)
+				goto collision;
+
+			/*  no writeback of non-persistent data */
+			if (!(test_a->is_persistent & test_a->is_completed))
+				goto collision;
+
+			/*  extend the search region when necessary */
+			diff = pos - test->io_pos;
+			if (diff > 0) {
+				len += diff;
+				pos = test->io_pos;
+				extended = true;
+			}
+			diff = (test->io_pos + test->io_len)	- (pos + len);
+			if (diff > 0) {
+				len += diff;
+				extended = true;
+			}
+		}
+	} while (extended); /*	start over for transitive closure */
+
+	*_pos = pos;
+	*_len = len;
+
+	for (tmp = start->hash_anchor.next; tmp != &start->hash_anchor; tmp = tmp->next) {
+		struct trans_logger_aio_aspect *test_a;
+		struct aio_object *test;
+
+		test_a = container_of(tmp, struct trans_logger_aio_aspect, hash_head);
+		test = test_a->object;
+
+		/*  are the regions overlapping? */
+		if (pos >= test->io_pos + test->io_len || pos + len <= test->io_pos)
+			continue; /*  not relevant */
+
+		/*  collect */
+		CHECK_HEAD_EMPTY(&test_a->collect_head);
+		if (unlikely(test_a->is_collected))
+			XIO_ERR("collision detection did not work\n");
+		test_a->is_collected = true;
+		obj_check(test);
+		list_add_tail(&test_a->collect_head, collect_list);
+	}
+
+collision:
+	up_read(&start->hash_mutex);
+}
+
+/* Atomically put all elements from the list.
+ * All elements must reside in the same collision list.
+ */
+static inline
+void hash_put_all(struct trans_logger_brick *brick, struct list_head *list)
+{
+	struct list_head *tmp;
+	struct trans_logger_hash_anchor *start = NULL;
+	int first_hash = -1;
+
+	for (tmp = list->next; tmp != list; tmp = tmp->next) {
+		struct trans_logger_aio_aspect *elem_a;
+		struct aio_object *elem;
+		int hash;
+
+		elem_a = container_of(tmp, struct trans_logger_aio_aspect, collect_head);
+		elem = elem_a->object;
+		CHECK_PTR(elem, err);
+		obj_check(elem);
+
+		hash = hash_fn(elem->io_pos);
+		if (!start) {
+			struct trans_logger_hash_anchor *sub_table = brick->hash_table[hash / HASH_PER_PAGE];
+
+			start = &sub_table[hash % HASH_PER_PAGE];
+			first_hash = hash;
+			down_write(&start->hash_mutex);
+		} else if (unlikely(hash != first_hash)) {
+			XIO_ERR("oops, different hashes: %d != %d\n", hash, first_hash);
+		}
+
+		if (!elem_a->is_hashed)
+			continue;
+
+		list_del_init(&elem_a->hash_head);
+		elem_a->is_hashed = false;
+		atomic_dec(&brick->hash_count);
+	}
+
+err:
+	if (start)
+		up_write(&start->hash_mutex);
+}
+
+static inline
+void hash_ensure_stableness(struct trans_logger_brick *brick, struct trans_logger_aio_aspect *aio_a)
+{
+	if (!aio_a->is_stable) {
+		struct aio_object *aio = aio_a->object;
+		int hash = hash_fn(aio->io_pos);
+		struct trans_logger_hash_anchor *sub_table = brick->hash_table[hash / HASH_PER_PAGE];
+		struct trans_logger_hash_anchor *start = &sub_table[hash % HASH_PER_PAGE];
+
+		down_write(&start->hash_mutex);
+
+		aio_a->is_stable = true;
+
+		up_write(&start->hash_mutex);
+	}
+}
+
+static
+void _inf_callback(struct trans_logger_input *input, bool force)
+{
+	if (!force &&
+	    input->inf_last_jiffies &&
+	    input->inf_last_jiffies + 4 * HZ > (long long)jiffies)
+		goto out_return;
+	if (input->inf.inf_callback && input->is_operating) {
+		input->inf_last_jiffies = jiffies;
+
+		input->inf.inf_callback(&input->inf);
+
+		input->inf_last_jiffies = jiffies;
+	} else {
+		XIO_DBG("%p skipped callback, callback = %p is_operating = %d\n",
+			input,
+			input->inf.inf_callback,
+			input->is_operating);
+	}
+out_return:;
+}
+
+static inline
+int _congested(struct trans_logger_brick *brick)
+{
+	return atomic_read(&brick->q_phase[0].q_queued) ||
+		atomic_read(&brick->q_phase[0].q_flying) ||
+		atomic_read(&brick->q_phase[1].q_queued) ||
+		atomic_read(&brick->q_phase[1].q_flying) ||
+		atomic_read(&brick->q_phase[2].q_queued) ||
+		atomic_read(&brick->q_phase[2].q_flying) ||
+		atomic_read(&brick->q_phase[3].q_queued) ||
+		atomic_read(&brick->q_phase[3].q_flying);
+}
+
+/***************** own brick * input * output operations *****************/
+
+atomic_t   global_mshadow_count = ATOMIC_INIT(0);
+
+atomic64_t global_mshadow_used = ATOMIC64_INIT(0);
+
+static
+int trans_logger_get_info(struct trans_logger_output *output, struct xio_info *info)
+{
+	struct trans_logger_input *input = output->brick->inputs[TL_INPUT_READ];
+
+	return GENERIC_INPUT_CALL(input, xio_get_info, info);
+}
+
+static
+int _make_sshadow(struct trans_logger_output *output,
+	struct trans_logger_aio_aspect *aio_a,
+	struct trans_logger_aio_aspect *mshadow_a)
+{
+	struct trans_logger_brick *brick = output->brick;
+	struct aio_object *aio = aio_a->object;
+	struct aio_object *mshadow;
+	int diff;
+
+	mshadow = mshadow_a->object;
+	if (unlikely(aio->io_len > mshadow->io_len)) {
+		XIO_ERR("oops %d -> %d\n", aio->io_len, mshadow->io_len);
+		aio->io_len = mshadow->io_len;
+	}
+	if (unlikely(mshadow_a == aio_a)) {
+		XIO_ERR("oops %p == %p\n", mshadow_a, aio_a);
+		return -EINVAL;
+	}
+
+	diff = aio->io_pos - mshadow->io_pos;
+	if (unlikely(diff < 0)) {
+		XIO_ERR("oops diff = %d\n", diff);
+		return -EINVAL;
+	}
+
+	/* Attach aio to the existing shadow ("slave shadow").
+	 */
+	aio_a->shadow_data = mshadow_a->shadow_data + diff;
+	aio_a->do_dealloc = false;
+	if (!aio->io_data) { /*  buffered IO */
+		aio->io_data = aio_a->shadow_data;
+		aio_a->do_buffered = true;
+		atomic_inc(&brick->total_sshadow_buffered_count);
+	}
+	aio->io_flags = mshadow->io_flags;
+	aio_a->shadow_aio = mshadow_a;
+	aio_a->my_brick = brick;
+
+	/* Get an ordinary internal reference
+	 */
+	obj_get_first(aio); /*	must be paired with __trans_logger_io_put() */
+	atomic_inc(&brick->inner_balance_count);
+
+	/* The internal reference from slave to master is already
+	 * present due to hash_find(),
+	 * such that the master cannot go away before the slave.
+	 * It is compensated by master transition in __trans_logger_io_put()
+	 */
+	atomic_inc(&brick->inner_balance_count);
+
+	atomic_inc(&brick->sshadow_count);
+	atomic_inc(&brick->total_sshadow_count);
+
+	if (unlikely(aio->io_len <= 0)) {
+		XIO_ERR("oops, len = %d\n", aio->io_len);
+		return -EINVAL;
+	}
+
+	return aio->io_len;
+}
+
+static
+int _read_io_get(struct trans_logger_output *output, struct trans_logger_aio_aspect *aio_a)
+{
+	struct trans_logger_brick *brick = output->brick;
+	struct aio_object *aio = aio_a->object;
+	struct trans_logger_input *input = brick->inputs[TL_INPUT_READ];
+	struct trans_logger_aio_aspect *mshadow_a;
+
+	/* Look if there is a newer version on the fly, shadowing
+	 * the old one.
+	 * When a shadow is found, use it as buffer for the aio.
+	 */
+	mshadow_a = hash_find(brick, aio->io_pos, &aio->io_len, false);
+	if (!mshadow_a)
+		return GENERIC_INPUT_CALL(input, aio_get, aio);
+
+	return _make_sshadow(output, aio_a, mshadow_a);
+}
+
+static
+int _write_io_get(struct trans_logger_output *output, struct trans_logger_aio_aspect *aio_a)
+{
+	struct trans_logger_brick *brick = output->brick;
+	struct aio_object *aio = aio_a->object;
+	void *data;
+
+#ifdef KEEP_UNIQUE
+	struct trans_logger_aio_aspect *mshadow_a;
+
+#endif
+
+#ifdef CONFIG_MARS_DEBUG
+	if (unlikely(aio->io_len <= 0)) {
+		XIO_ERR("oops, io_len = %d\n", aio->io_len);
+		return -EINVAL;
+	}
+#endif
+
+#ifdef KEEP_UNIQUE
+	mshadow_a = hash_find(brick, aio->io_pos, &aio->io_len, true);
+	if (mshadow_a)
+		return _make_sshadow(output, aio_a, mshadow_a);
+#endif
+
+#ifdef DELAY_CALLERS
+	/*  delay in case of too many master shadows / memory shortage */
+	wait_event_interruptible_timeout(brick->caller_event,
+					 !brick->delay_callers &&
+					 (brick_global_memlimit < 1024 ||
+					  atomic64_read(&global_mshadow_used) / 1024 < brick_global_memlimit),
+					 HZ / 2);
+#endif
+
+	/*  create a new master shadow */
+	data = brick_block_alloc(aio->io_pos, (aio_a->alloc_len = aio->io_len));
+	atomic64_add(aio->io_len, &brick->shadow_mem_used);
+#ifdef CONFIG_MARS_DEBUG
+	memset(data, 0x11, aio->io_len);
+#endif
+	aio_a->shadow_data = data;
+	aio_a->do_dealloc = true;
+	if (!aio->io_data) { /*  buffered IO */
+		aio->io_data = data;
+		aio_a->do_buffered = true;
+		atomic_inc(&brick->total_mshadow_buffered_count);
+	}
+	aio_a->my_brick = brick;
+	aio->io_flags = 0;
+	aio_a->shadow_aio = aio_a; /*  cyclic self-reference = > indicates master shadow */
+
+	atomic_inc(&brick->mshadow_count);
+	atomic_inc(&brick->total_mshadow_count);
+	atomic_inc(&global_mshadow_count);
+	atomic64_add(aio->io_len, &global_mshadow_used);
+
+	atomic_inc(&brick->inner_balance_count);
+	obj_get_first(aio); /*	must be paired with __trans_logger_io_put() */
+
+	return aio->io_len;
+}
+
+static
+int trans_logger_io_get(struct trans_logger_output *output, struct aio_object *aio)
+{
+	struct trans_logger_brick *brick;
+	struct trans_logger_aio_aspect *aio_a;
+	loff_t base_offset;
+
+	CHECK_PTR(output, err);
+	brick = output->brick;
+	CHECK_PTR(brick, err);
+	CHECK_PTR(aio, err);
+
+	aio_a = trans_logger_aio_get_aspect(brick, aio);
+	CHECK_PTR(aio_a, err);
+	CHECK_ASPECT(aio_a, aio, err);
+
+	atomic_inc(&brick->outer_balance_count);
+
+	if (aio->obj_initialized) { /*	setup already performed */
+		obj_check(aio);
+		obj_get(aio); /*  must be paired with __trans_logger_io_put() */
+		return aio->io_len;
+	}
+
+	get_lamport(&aio_a->stamp);
+
+	if (aio->io_len > CONF_TRANS_MAX_AIO_SIZE && CONF_TRANS_MAX_AIO_SIZE > 0)
+		aio->io_len = CONF_TRANS_MAX_AIO_SIZE;
+
+	/*  ensure that REGION_SIZE boundaries are obeyed by hashing */
+	base_offset = aio->io_pos & (loff_t)(REGION_SIZE - 1);
+	if (aio->io_len > REGION_SIZE - base_offset)
+		aio->io_len = REGION_SIZE - base_offset;
+
+	if (aio->io_may_write == READ)
+		return _read_io_get(output, aio_a);
+
+	if (unlikely(brick->stopped_logging)) { /*  only in EMERGENCY mode */
+		aio_a->is_emergency = true;
+		/* Wait until writeback has finished.
+		 * We have to this because writeback is out-of-order.
+		 * Otherwise consistency could be violated for some time.
+		 */
+		while (_congested(brick)) {
+			/*  in case of emergency, busy-wait should be acceptable */
+			brick_msleep(HZ / 10);
+		}
+		return _read_io_get(output, aio_a);
+	}
+
+	/* FIXME: THIS IS PROVISIONARY (use event instead)
+	 */
+	while (unlikely(!brick->power.on_led))
+		brick_msleep(HZ / 10);
+
+	return _write_io_get(output, aio_a);
+
+err:
+	return -EINVAL;
+}
+
+static void pos_complete(struct trans_logger_aio_aspect *orig_aio_a);
+
+static
+void __trans_logger_io_put(struct trans_logger_brick *brick, struct trans_logger_aio_aspect *aio_a)
+{
+	struct aio_object *aio;
+	struct trans_logger_aio_aspect *shadow_a;
+	struct trans_logger_input *input;
+
+restart:
+	CHECK_PTR(aio_a, err);
+	aio = aio_a->object;
+	CHECK_PTR(aio, err);
+
+	obj_check(aio);
+
+	/*  are we a shadow (whether master or slave)? */
+	shadow_a = aio_a->shadow_aio;
+	if (shadow_a) {
+		bool finished;
+
+		CHECK_PTR(shadow_a, err);
+		CHECK_PTR(shadow_a->object, err);
+		obj_check(shadow_a->object);
+
+		finished = obj_put(aio);
+		atomic_dec(&brick->inner_balance_count);
+		if (unlikely(finished && aio_a->is_hashed)) {
+			XIO_ERR("trying to put a hashed aio, pos = %lld len = %d\n", aio->io_pos, aio->io_len);
+			finished = false; /*  leaves a memleak */
+		}
+
+		if (!finished)
+			goto out_return;
+
+		CHECK_HEAD_EMPTY(&aio_a->lh.lh_head);
+		CHECK_HEAD_EMPTY(&aio_a->hash_head);
+		CHECK_HEAD_EMPTY(&aio_a->replay_head);
+		CHECK_HEAD_EMPTY(&aio_a->collect_head);
+		CHECK_HEAD_EMPTY(&aio_a->sub_list);
+		CHECK_HEAD_EMPTY(&aio_a->sub_head);
+
+		if (aio_a->is_collected && likely(aio_a->wb_error >= 0))
+			pos_complete(aio_a);
+
+		CHECK_HEAD_EMPTY(&aio_a->pos_head);
+
+		if (shadow_a != aio_a) { /*  we are a slave shadow */
+			/* XIO_DBG("slave\n"); */
+			atomic_dec(&brick->sshadow_count);
+			CHECK_HEAD_EMPTY(&aio_a->hash_head);
+			obj_free(aio);
+			/*  now put the master shadow */
+			aio_a = shadow_a;
+			goto restart;
+		}
+		/*  we are a master shadow */
+		CHECK_PTR(aio_a->shadow_data, err);
+		if (aio_a->do_dealloc) {
+			brick_block_free(aio_a->shadow_data, aio_a->alloc_len);
+			atomic64_sub(aio->io_len, &brick->shadow_mem_used);
+			aio_a->shadow_data = NULL;
+			aio_a->do_dealloc = false;
+		}
+		if (aio_a->do_buffered)
+			aio->io_data = NULL;
+		atomic_dec(&brick->mshadow_count);
+		atomic_dec(&global_mshadow_count);
+		atomic64_sub(aio->io_len, &global_mshadow_used);
+		obj_free(aio);
+		goto out_return;
+	}
+
+	/*  only READ is allowed on non-shadow buffers */
+	if (unlikely(aio->io_rw != READ && !aio_a->is_emergency))
+		XIO_FAT("bad operation %d on non-shadow\n", aio->io_rw);
+
+	/*  no shadow = > call through */
+	input = brick->inputs[TL_INPUT_READ];
+	CHECK_PTR(input, err);
+
+	GENERIC_INPUT_CALL(input, aio_put, aio);
+
+err:;
+out_return:;
+}
+
+static
+void _trans_logger_io_put(struct trans_logger_output *output, struct aio_object *aio)
+{
+	struct trans_logger_aio_aspect *aio_a;
+
+	aio_a = trans_logger_aio_get_aspect(output->brick, aio);
+	CHECK_PTR(aio_a, err);
+	CHECK_ASPECT(aio_a, aio, err);
+
+	__trans_logger_io_put(output->brick, aio_a);
+	goto out_return;
+err:
+	XIO_FAT("giving up...\n");
+out_return:;
+}
+
+static
+void trans_logger_io_put(struct trans_logger_output *output, struct aio_object *aio)
+{
+	struct trans_logger_brick *brick = output->brick;
+
+	atomic_dec(&brick->outer_balance_count);
+	_trans_logger_io_put(output, aio);
+}
+
+static
+void _trans_logger_endio(struct generic_callback *cb)
+{
+	struct trans_logger_aio_aspect *aio_a;
+	struct trans_logger_brick *brick;
+
+	aio_a = cb->cb_private;
+	CHECK_PTR(aio_a, err);
+	if (unlikely(&aio_a->cb != cb)) {
+		XIO_FAT("bad callback -- hanging up\n");
+		goto err;
+	}
+	brick = aio_a->my_brick;
+	CHECK_PTR(brick, err);
+
+	NEXT_CHECKED_CALLBACK(cb, err);
+
+	atomic_dec(&brick->any_fly_count);
+	atomic_inc(&brick->total_cb_count);
+	wake_up_interruptible_all(&brick->worker_event);
+	goto out_return;
+err:
+	XIO_FAT("cannot handle callback\n");
+out_return:;
+}
+
+static
+void trans_logger_io_io(struct trans_logger_output *output, struct aio_object *aio)
+{
+	struct trans_logger_brick *brick = output->brick;
+	struct trans_logger_aio_aspect *aio_a;
+	struct trans_logger_aio_aspect *shadow_a;
+	struct trans_logger_input *input;
+
+	obj_check(aio);
+
+	aio_a = trans_logger_aio_get_aspect(brick, aio);
+	CHECK_PTR(aio_a, err);
+	CHECK_ASPECT(aio_a, aio, err);
+
+	/*  statistics */
+	if (aio->io_rw)
+		atomic_inc(&brick->total_write_count);
+	else
+		atomic_inc(&brick->total_read_count);
+	/*  is this a shadow buffer? */
+	shadow_a = aio_a->shadow_aio;
+	if (shadow_a) {
+		CHECK_HEAD_EMPTY(&aio_a->lh.lh_head);
+		CHECK_HEAD_EMPTY(&aio_a->hash_head);
+		CHECK_HEAD_EMPTY(&aio_a->pos_head);
+
+		obj_get(aio); /*  must be paired with __trans_logger_io_put() */
+		atomic_inc(&brick->inner_balance_count);
+
+		qq_aio_insert(&brick->q_phase[0], aio_a);
+		wake_up_interruptible_all(&brick->worker_event);
+		goto out_return;
+	}
+
+	/*  only READ is allowed on non-shadow buffers */
+	if (unlikely(aio->io_rw != READ && !aio_a->is_emergency))
+		XIO_FAT("bad operation %d on non-shadow\n", aio->io_rw);
+
+	atomic_inc(&brick->any_fly_count);
+
+	aio_a->my_brick = brick;
+
+	INSERT_CALLBACK(aio, &aio_a->cb, _trans_logger_endio, aio_a);
+
+	input = output->brick->inputs[TL_INPUT_READ];
+
+	GENERIC_INPUT_CALL(input, aio_io, aio);
+	goto out_return;
+err:
+	XIO_FAT("cannot handle IO\n");
+out_return:;
+}
+
+/***************************** writeback info *****************************/
+
+/* save final completion status when necessary
+ */
+static
+void pos_complete(struct trans_logger_aio_aspect *orig_aio_a)
+{
+	struct trans_logger_brick *brick = orig_aio_a->my_brick;
+	struct trans_logger_input *log_input = orig_aio_a->log_input;
+	loff_t finished;
+	struct list_head *tmp;
+
+	CHECK_PTR(brick, err);
+	CHECK_PTR(log_input, err);
+
+	atomic_inc(&brick->total_writeback_count);
+
+	tmp = &orig_aio_a->pos_head;
+
+	down(&log_input->inf_mutex);
+
+	finished = orig_aio_a->log_pos;
+	/*  am I the first member? (means "youngest" list entry) */
+	if (tmp == log_input->pos_list.next) {
+		if (unlikely(finished <= log_input->inf.inf_min_pos))
+			XIO_ERR("backskip in log writeback: %lld -> %lld\n", log_input->inf.inf_min_pos, finished);
+		if (unlikely(finished > log_input->inf.inf_max_pos))
+			XIO_ERR("min_pos > max_pos: %lld > %lld\n", finished, log_input->inf.inf_max_pos);
+		log_input->inf.inf_min_pos = finished;
+		get_lamport(&log_input->inf.inf_min_pos_stamp);
+		_inf_callback(log_input, false);
+	} else {
+		struct trans_logger_aio_aspect *prev_aio_a;
+
+		prev_aio_a = container_of(tmp->prev, struct trans_logger_aio_aspect, pos_head);
+		if (unlikely(finished <= prev_aio_a->log_pos)) {
+			XIO_ERR("backskip: %lld -> %lld\n", finished, prev_aio_a->log_pos);
+		} else {
+			/* Transitively transfer log_pos to the predecessor
+			 * to correctly reflect the committed region.
+			 */
+			prev_aio_a->log_pos = finished;
+		}
+	}
+
+	list_del_init(tmp);
+	atomic_dec(&log_input->pos_count);
+
+	up(&log_input->inf_mutex);
+err:;
+}
+
+static
+void free_writeback(struct writeback_info *wb)
+{
+	struct list_head *tmp;
+
+	if (unlikely(wb->w_error < 0)) {
+		XIO_ERR("writeback error = %d at pos = %lld len = %d, writeback is incomplete\n",
+			wb->w_error,
+			wb->w_pos,
+			wb->w_len);
+	}
+
+	/* Now complete the original requests.
+	 */
+	while ((tmp = wb->w_collect_list.next) != &wb->w_collect_list) {
+		struct trans_logger_aio_aspect *orig_aio_a;
+		struct aio_object *orig_aio;
+
+		list_del_init(tmp);
+
+		orig_aio_a = container_of(tmp, struct trans_logger_aio_aspect, collect_head);
+		orig_aio = orig_aio_a->object;
+
+		obj_check(orig_aio);
+		if (unlikely(!orig_aio_a->is_collected)) {
+			XIO_ERR("request %lld (len = %d) was not collected\n",
+				orig_aio->io_pos,
+				orig_aio->io_len);
+		}
+		if (unlikely(wb->w_error < 0))
+			orig_aio_a->wb_error = wb->w_error;
+
+		__trans_logger_io_put(orig_aio_a->my_brick, orig_aio_a);
+	}
+
+	brick_mem_free(wb);
+}
+
+/* Generic endio() for writeback_info
+ */
+static
+void wb_endio(struct generic_callback *cb)
+{
+	struct trans_logger_aio_aspect *sub_aio_a;
+	struct aio_object *sub_aio;
+	struct trans_logger_brick *brick;
+	struct writeback_info *wb;
+	atomic_t *dec;
+	int rw;
+	void (**_endio)(struct generic_callback *cb);
+	void (*endio)(struct generic_callback *cb);
+
+	LAST_CALLBACK(cb);
+	sub_aio_a = cb->cb_private;
+	CHECK_PTR(sub_aio_a, err);
+	sub_aio = sub_aio_a->object;
+	CHECK_PTR(sub_aio, err);
+	wb = sub_aio_a->wb;
+	CHECK_PTR(wb, err);
+	brick = wb->w_brick;
+	CHECK_PTR(brick, err);
+
+	if (cb->cb_error < 0)
+		wb->w_error = cb->cb_error;
+
+	atomic_dec(&brick->wb_balance_count);
+
+	rw = sub_aio_a->orig_rw;
+	dec = rw ? &wb->w_sub_write_count : &wb->w_sub_read_count;
+	CHECK_ATOMIC(dec, 1);
+	if (!atomic_dec_and_test(dec))
+		goto done;
+
+	_endio = rw ? &wb->write_endio : &wb->read_endio;
+	endio = *_endio;
+	*_endio = NULL;
+	if (likely(endio))
+		endio(cb);
+	else
+		XIO_ERR("internal: no endio defined\n");
+done:
+	wake_up_interruptible_all(&brick->worker_event);
+	goto out_return;
+err:
+	XIO_FAT("hanging up....\n");
+out_return:;
+}
+
+/* Atomically create writeback info, based on "snapshot" of current hash
+ * state.
+ * Notice that the hash can change during writeback IO, thus we need
+ * struct writeback_info to precisely catch that information at a single
+ * point in time.
+ */
+static
+struct writeback_info *make_writeback(struct trans_logger_brick *brick, loff_t pos, int len)
+{
+	struct writeback_info *wb;
+	struct trans_logger_input *read_input;
+	struct trans_logger_input *write_input;
+	int write_input_nr;
+
+	/* Allocate structure representing a bunch of adjacent writebacks
+	 */
+	wb = brick_zmem_alloc(sizeof(struct writeback_info));
+	if (unlikely(len < 0))
+		XIO_ERR("len = %d\n", len);
+
+	wb->w_brick = brick;
+	wb->w_pos = pos;
+	wb->w_len = len;
+	wb->w_lh.lh_pos = &wb->w_pos;
+	INIT_LIST_HEAD(&wb->w_lh.lh_head);
+	INIT_LIST_HEAD(&wb->w_collect_list);
+	INIT_LIST_HEAD(&wb->w_sub_read_list);
+	INIT_LIST_HEAD(&wb->w_sub_write_list);
+
+	/* Atomically fetch transitive closure on all requests
+	 * overlapping with the current search region.
+	 */
+	hash_extend(brick, &wb->w_pos, &wb->w_len, &wb->w_collect_list);
+
+	if (list_empty(&wb->w_collect_list))
+		goto collision;
+
+	pos = wb->w_pos;
+	len = wb->w_len;
+
+	if (unlikely(len < 0))
+		XIO_ERR("len = %d\n", len);
+
+	/* Determine the "channels" we want to operate on
+	 */
+	read_input = brick->inputs[TL_INPUT_READ];
+	write_input_nr = TL_INPUT_WRITEBACK;
+	write_input = brick->inputs[write_input_nr];
+	if (!write_input->connect) {
+		write_input_nr = TL_INPUT_READ;
+		write_input = read_input;
+	}
+
+	/* Create sub_aios for read of old disk version (phase1)
+	 */
+	if (brick->log_reads) {
+		while (len > 0) {
+			struct trans_logger_aio_aspect *sub_aio_a;
+			struct aio_object *sub_aio;
+			struct trans_logger_input *log_input;
+			int this_len;
+			int status;
+
+			sub_aio = trans_logger_alloc_aio(brick);
+
+			sub_aio->io_pos = pos;
+			sub_aio->io_len = len;
+			sub_aio->io_may_write = READ;
+			sub_aio->io_rw = READ;
+			sub_aio->io_data = NULL;
+
+			sub_aio_a = trans_logger_aio_get_aspect(brick, sub_aio);
+			CHECK_PTR(sub_aio_a, err);
+			CHECK_ASPECT(sub_aio_a, sub_aio, err);
+
+			sub_aio_a->my_input = read_input;
+			log_input = brick->inputs[brick->log_input_nr];
+			sub_aio_a->log_input = log_input;
+			atomic_inc(&log_input->log_obj_count);
+			sub_aio_a->my_brick = brick;
+			sub_aio_a->orig_rw = READ;
+			sub_aio_a->wb = wb;
+
+			status = GENERIC_INPUT_CALL(read_input, aio_get, sub_aio);
+			if (unlikely(status < 0)) {
+				XIO_FAT("cannot get sub_aio, status = %d\n", status);
+				goto err;
+			}
+
+			list_add_tail(&sub_aio_a->sub_head, &wb->w_sub_read_list);
+			atomic_inc(&wb->w_sub_read_count);
+			atomic_inc(&brick->wb_balance_count);
+
+			this_len = sub_aio->io_len;
+			pos += this_len;
+			len -= this_len;
+		}
+		/* Re-init for startover
+		 */
+		pos = wb->w_pos;
+		len = wb->w_len;
+	}
+
+	/* Always create sub_aios for writeback (phase3)
+	 */
+	while (len > 0) {
+		struct trans_logger_aio_aspect *sub_aio_a;
+		struct aio_object *sub_aio;
+		struct trans_logger_aio_aspect *orig_aio_a;
+		struct aio_object *orig_aio;
+		struct trans_logger_input *log_input;
+		void *data;
+		int this_len = len;
+		int diff;
+		int status;
+
+		atomic_inc(&brick->total_hash_find_count);
+
+		orig_aio_a = _hash_find(&wb->w_collect_list, pos, &this_len, true, false);
+		if (unlikely(!orig_aio_a)) {
+			XIO_FAT("could not find data\n");
+			goto err;
+		}
+
+		orig_aio = orig_aio_a->object;
+		diff = pos - orig_aio->io_pos;
+		if (unlikely(diff < 0)) {
+			XIO_FAT("bad diff %d\n", diff);
+			goto err;
+		}
+		data = orig_aio_a->shadow_data + diff;
+
+		sub_aio = trans_logger_alloc_aio(brick);
+
+		sub_aio->io_pos = pos;
+		sub_aio->io_len = this_len;
+		sub_aio->io_may_write = WRITE;
+		sub_aio->io_rw = WRITE;
+		sub_aio->io_data = data;
+
+		sub_aio_a = trans_logger_aio_get_aspect(brick, sub_aio);
+		CHECK_PTR(sub_aio_a, err);
+		CHECK_ASPECT(sub_aio_a, sub_aio, err);
+
+		sub_aio_a->orig_aio_a = orig_aio_a;
+		sub_aio_a->my_input = write_input;
+		log_input = orig_aio_a->log_input;
+		sub_aio_a->log_input = log_input;
+		atomic_inc(&log_input->log_obj_count);
+		sub_aio_a->my_brick = brick;
+		sub_aio_a->orig_rw = WRITE;
+		sub_aio_a->wb = wb;
+
+		status = GENERIC_INPUT_CALL(write_input, aio_get, sub_aio);
+		if (unlikely(status < 0)) {
+			XIO_FAT("cannot get sub_aio, status = %d\n", status);
+			wb->w_error = status;
+			goto err;
+		}
+
+		list_add_tail(&sub_aio_a->sub_head, &wb->w_sub_write_list);
+		atomic_inc(&wb->w_sub_write_count);
+		atomic_inc(&brick->wb_balance_count);
+
+		this_len = sub_aio->io_len;
+		pos += this_len;
+		len -= this_len;
+	}
+
+	return wb;
+
+err:
+	XIO_ERR("cleaning up...\n");
+collision:
+	if (wb)
+		free_writeback(wb);
+	return NULL;
+}
+
+static inline
+void _fire_one(struct list_head *tmp, bool do_update)
+{
+	struct trans_logger_aio_aspect *sub_aio_a;
+	struct aio_object *sub_aio;
+	struct trans_logger_input *sub_input;
+
+	sub_aio_a = container_of(tmp, struct trans_logger_aio_aspect, sub_head);
+	sub_aio = sub_aio_a->object;
+
+	if (unlikely(sub_aio_a->is_fired)) {
+		XIO_ERR("trying to fire twice\n");
+		goto out_return;
+	}
+	sub_aio_a->is_fired = true;
+
+	SETUP_CALLBACK(sub_aio, wb_endio, sub_aio_a);
+
+	sub_input = sub_aio_a->my_input;
+
+#ifdef DO_WRITEBACK
+	GENERIC_INPUT_CALL(sub_input, aio_io, sub_aio);
+#else
+	SIMPLE_CALLBACK(sub_aio, 0);
+#endif
+	if (do_update) { /*  CHECK: shouldn't we do this always? */
+		GENERIC_INPUT_CALL(sub_input, aio_put, sub_aio);
+	}
+out_return:;
+}
+
+static inline
+void fire_writeback(struct list_head *start, bool do_update)
+{
+	struct list_head *tmp;
+
+	/* Caution! The wb structure may get deallocated
+	 * during _fire_one() in some cases (e.g. when the
+	 * callback is directly called by the aio_io operation).
+	 * Ensure that no ptr dereferencing can take
+	 * place after working on the last list member.
+	 */
+	tmp = start->next;
+	while (tmp != start) {
+		struct list_head *next = tmp->next;
+
+		list_del_init(tmp);
+		_fire_one(tmp, do_update);
+		tmp = next;
+	}
+}
+
+static inline
+void update_max_pos(struct trans_logger_aio_aspect *orig_aio_a)
+{
+	loff_t max_pos = orig_aio_a->log_pos;
+	struct trans_logger_input *log_input = orig_aio_a->log_input;
+
+	CHECK_PTR(log_input, done);
+
+	down(&log_input->inf_mutex);
+
+	if (unlikely(max_pos < log_input->inf.inf_min_pos))
+		XIO_ERR("new max_pos < min_pos: %lld < %lld\n", max_pos, log_input->inf.inf_min_pos);
+	if (log_input->inf.inf_max_pos < max_pos) {
+		log_input->inf.inf_max_pos = max_pos;
+		get_lamport(&log_input->inf.inf_max_pos_stamp);
+		_inf_callback(log_input, false);
+	}
+
+	up(&log_input->inf_mutex);
+done:;
+}
+
+static inline
+void update_writeback_info(struct writeback_info *wb)
+{
+	struct list_head *start = &wb->w_collect_list;
+	struct list_head *tmp;
+
+	/* Notice: in case of log rotation, each list member
+	 * may belong to a different log_input.
+	 */
+	for (tmp = start->next; tmp != start; tmp = tmp->next) {
+		struct trans_logger_aio_aspect *orig_aio_a;
+
+		orig_aio_a = container_of(tmp, struct trans_logger_aio_aspect, collect_head);
+		update_max_pos(orig_aio_a);
+	}
+}
+
+/***************************** worker thread *****************************/
+
+/*********************************************************************
+ * Phase 0: write transaction log entry for the original write request.
+ */
+
+static
+void _complete(struct trans_logger_brick *brick, struct trans_logger_aio_aspect *orig_aio_a, int error, bool pre_io)
+{
+	struct aio_object *orig_aio;
+
+	orig_aio = orig_aio_a->object;
+	CHECK_PTR(orig_aio, err);
+
+	if (orig_aio_a->is_completed ||
+	    (pre_io &&
+	     (trans_logger_completion_semantics >= 2 ||
+	      (trans_logger_completion_semantics >= 1 && !orig_aio->io_skip_sync)))) {
+		goto done;
+	}
+
+	if (cmpxchg(&orig_aio_a->is_completed, false, true))
+		goto done;
+
+	atomic_dec(&brick->log_fly_count);
+
+	if (likely(error >= 0)) {
+		aio_checksum(orig_aio);
+		orig_aio->io_flags &= ~AIO_WRITING;
+		orig_aio->io_flags |= AIO_UPTODATE;
+	}
+	CHECKED_CALLBACK(orig_aio, error, err);
+
+done:
+	goto out_return;
+err:
+	XIO_ERR("giving up...\n");
+out_return:;
+}
+
+static
+void phase0_preio(void *private)
+{
+	struct trans_logger_aio_aspect *orig_aio_a;
+	struct trans_logger_brick *brick;
+
+	orig_aio_a = private;
+	CHECK_PTR(orig_aio_a, err);
+	CHECK_PTR(orig_aio_a->object, err);
+	brick = orig_aio_a->my_brick;
+	CHECK_PTR(brick, err);
+
+	/*  signal completion to the upper layer */
+/**/
+
+	obj_check(orig_aio_a->object);
+	_complete(brick, orig_aio_a, 0, true);
+	obj_check(orig_aio_a->object);
+	goto out_return;
+err:
+	XIO_ERR("giving up...\n");
+out_return:;
+}
+
+static
+void phase0_endio(void *private, int error)
+{
+	struct aio_object *orig_aio;
+	struct trans_logger_aio_aspect *orig_aio_a;
+	struct trans_logger_brick *brick;
+
+	orig_aio_a = private;
+	CHECK_PTR(orig_aio_a, err);
+
+	brick = orig_aio_a->my_brick;
+	CHECK_PTR(brick, err);
+	orig_aio = orig_aio_a->object;
+	CHECK_PTR(orig_aio, err);
+
+	orig_aio_a->is_persistent = true;
+	qq_dec_flying(&brick->q_phase[0]);
+
+	_CHECK(orig_aio_a->shadow_aio, err);
+
+	/*  signal completion to the upper layer */
+	_complete(brick, orig_aio_a, error, false);
+
+	/* Queue up for the next phase.
+	 */
+	qq_aio_insert(&brick->q_phase[1], orig_aio_a);
+
+	/* Undo the above pinning
+	 */
+	__trans_logger_io_put(brick, orig_aio_a);
+
+	banning_reset(&brick->q_phase[0].q_banning);
+
+	wake_up_interruptible_all(&brick->worker_event);
+	goto out_return;
+err:
+	XIO_ERR("giving up...\n");
+out_return:;
+}
+
+static
+bool phase0_startio(struct trans_logger_aio_aspect *orig_aio_a)
+{
+	struct aio_object *orig_aio;
+	struct trans_logger_brick *brick;
+	struct trans_logger_input *input;
+	struct log_status *logst;
+	loff_t log_pos;
+	void *data;
+	bool ok;
+
+	CHECK_PTR(orig_aio_a, err);
+	orig_aio = orig_aio_a->object;
+	CHECK_PTR(orig_aio, err);
+	brick = orig_aio_a->my_brick;
+	CHECK_PTR(brick, err);
+	input = orig_aio_a->log_input;
+	CHECK_PTR(input, err);
+	logst = &input->logst;
+	logst->do_crc = trans_logger_do_crc;
+
+	{
+		struct log_header l = {
+			.l_stamp = orig_aio_a->stamp,
+			.l_pos = orig_aio->io_pos,
+			.l_len = orig_aio->io_len,
+			.l_code = CODE_WRITE_NEW,
+		};
+		data = log_reserve(logst, &l);
+	}
+	if (unlikely(!data))
+		goto err;
+
+	hash_ensure_stableness(brick, orig_aio_a);
+
+	memcpy(data, orig_aio_a->shadow_data, orig_aio->io_len);
+
+	/* Pin aio->obj_count so it can't go away
+	 * after _complete().
+	 * This may happen rather early in phase0_preio().
+	 */
+	obj_get(orig_aio); /*  must be paired with __trans_logger_io_put() */
+	atomic_inc(&brick->inner_balance_count);
+	atomic_inc(&brick->log_fly_count);
+
+	ok = log_finalize(logst, orig_aio->io_len, phase0_endio, orig_aio_a);
+	if (unlikely(!ok)) {
+		atomic_dec(&brick->log_fly_count);
+		goto err;
+	}
+	log_pos = logst->log_pos + logst->offset;
+	orig_aio_a->log_pos = log_pos;
+
+	/*  update new log_pos in the symlinks */
+	down(&input->inf_mutex);
+	input->inf.inf_log_pos = log_pos;
+	memcpy(&input->inf.inf_log_pos_stamp, &logst->log_pos_stamp, sizeof(input->inf.inf_log_pos_stamp));
+	_inf_callback(input, false);
+
+#ifdef CONFIG_MARS_DEBUG
+	if (!list_empty(&input->pos_list)) {
+		struct trans_logger_aio_aspect *last_aio_a;
+
+		last_aio_a = container_of(input->pos_list.prev, struct trans_logger_aio_aspect, pos_head);
+		if (last_aio_a->log_pos >= orig_aio_a->log_pos)
+			XIO_ERR("backskip in pos_list, %lld >= %lld\n", last_aio_a->log_pos, orig_aio_a->log_pos);
+	}
+#endif
+	list_add_tail(&orig_aio_a->pos_head, &input->pos_list);
+	atomic_inc(&input->pos_count);
+	up(&input->inf_mutex);
+
+	qq_inc_flying(&brick->q_phase[0]);
+
+	phase0_preio(orig_aio_a);
+
+	return true;
+
+err:
+	return false;
+}
+
+static
+bool prep_phase_startio(struct trans_logger_aio_aspect *aio_a)
+{
+	struct aio_object *aio = aio_a->object;
+	struct trans_logger_aio_aspect *shadow_a;
+	struct trans_logger_brick *brick;
+
+	CHECK_PTR(aio, err);
+	shadow_a = aio_a->shadow_aio;
+	CHECK_PTR(shadow_a, err);
+	brick = aio_a->my_brick;
+	CHECK_PTR(brick, err);
+
+	if (aio->io_rw == READ) {
+		/*  nothing to do: directly signal success. */
+		struct aio_object *shadow = shadow_a->object;
+
+		if (unlikely(shadow == aio))
+			XIO_ERR("oops, we should be a slave shadow, but are a master one\n");
+#ifdef USE_MEMCPY
+		if (aio_a->shadow_data != aio->io_data) {
+			if (unlikely(aio->io_len <= 0 || aio->io_len > PAGE_SIZE))
+				XIO_ERR("implausible io_len = %d\n", aio->io_len);
+			memcpy(aio->io_data, aio_a->shadow_data, aio->io_len);
+		}
+#endif
+		aio->io_flags |= AIO_UPTODATE;
+
+		CHECKED_CALLBACK(aio, 0, err);
+
+		__trans_logger_io_put(brick, aio_a);
+
+		return true;
+	}
+	/*  else WRITE */
+	CHECK_HEAD_EMPTY(&aio_a->lh.lh_head);
+	CHECK_HEAD_EMPTY(&aio_a->hash_head);
+	if (unlikely(aio->io_flags & (AIO_READING | AIO_WRITING)))
+		XIO_ERR("bad flags %d\n", aio->io_flags);
+	/* In case of non-buffered IO, the buffer is
+	 * under control of the user. In particular, he
+	 * may change it without telling us.
+	 * Therefore we make a copy (or "snapshot") here.
+	 */
+	aio->io_flags |= AIO_WRITING;
+#ifdef USE_MEMCPY
+	if (aio_a->shadow_data != aio->io_data) {
+		if (unlikely(aio->io_len <= 0 || aio->io_len > PAGE_SIZE))
+			XIO_ERR("implausible io_len = %d\n", aio->io_len);
+		memcpy(aio_a->shadow_data, aio->io_data, aio->io_len);
+	}
+#endif
+	aio_a->is_dirty = true;
+	aio_a->shadow_aio->is_dirty = true;
+#ifndef KEEP_UNIQUE
+	if (unlikely(aio_a->shadow_aio != aio_a))
+		XIO_ERR("something is wrong: %p != %p\n", aio_a->shadow_aio, aio_a);
+#endif
+	if (likely(!aio_a->is_hashed)) {
+		struct trans_logger_input *log_input;
+
+		log_input = brick->inputs[brick->log_input_nr];
+		aio_a->log_input = log_input;
+		atomic_inc(&log_input->log_obj_count);
+		hash_insert(brick, aio_a);
+	} else {
+		XIO_ERR("tried to hash twice\n");
+	}
+	return phase0_startio(aio_a);
+
+err:
+	XIO_ERR("cannot work\n");
+	brick_msleep(1000);
+	return false;
+}
+
+/*********************************************************************
+ * Phase 1: read original version of data.
+ * This happens _after_ phase 0, deliberately.
+ * We are explicitly dealing with old and new versions.
+ * The new version is hashed in memory all the time (such that parallel
+ * READs will see them), so we have plenty of time for getting the
+ * old version from disk somewhen later, e.g. when IO contention is low.
+ */
+
+static
+void phase1_endio(struct generic_callback *cb)
+{
+	struct trans_logger_aio_aspect *sub_aio_a;
+	struct writeback_info *wb;
+	struct trans_logger_brick *brick;
+
+	CHECK_PTR(cb, err);
+	sub_aio_a = cb->cb_private;
+	CHECK_PTR(sub_aio_a, err);
+	wb = sub_aio_a->wb;
+	CHECK_PTR(wb, err);
+	brick = wb->w_brick;
+	CHECK_PTR(brick, err);
+
+	if (unlikely(cb->cb_error < 0)) {
+		XIO_FAT("IO error %d\n", cb->cb_error);
+		goto err;
+	}
+
+	qq_dec_flying(&brick->q_phase[1]);
+
+	banning_reset(&brick->q_phase[1].q_banning);
+
+	/*  queue up for the next phase */
+	qq_wb_insert(&brick->q_phase[2], wb);
+	wake_up_interruptible_all(&brick->worker_event);
+	goto out_return;
+err:
+	XIO_FAT("hanging up....\n");
+out_return:;
+}
+
+static void phase3_endio(struct generic_callback *cb);
+
+static bool phase3_startio(struct writeback_info *wb);
+
+static
+bool phase1_startio(struct trans_logger_aio_aspect *orig_aio_a)
+{
+	struct aio_object *orig_aio;
+	struct trans_logger_brick *brick;
+	struct writeback_info *wb = NULL;
+
+	CHECK_PTR(orig_aio_a, err);
+	orig_aio = orig_aio_a->object;
+	CHECK_PTR(orig_aio, err);
+	brick = orig_aio_a->my_brick;
+	CHECK_PTR(brick, err);
+
+	if (orig_aio_a->is_collected)
+		goto done;
+	if (!orig_aio_a->is_hashed)
+		goto done;
+
+	wb = make_writeback(brick, orig_aio->io_pos, orig_aio->io_len);
+	if (unlikely(!wb))
+		goto collision;
+
+	if (unlikely(list_empty(&wb->w_sub_write_list))) {
+		XIO_ERR("sub_write_list is empty, orig pos = %lld len = %d (collected=%d), extended pos = %lld len = %d\n",
+			orig_aio->io_pos,
+			orig_aio->io_len,
+			(int)orig_aio_a->is_collected,
+			wb->w_pos,
+			wb->w_len);
+		goto err;
+	}
+
+	wb->read_endio = phase1_endio;
+	wb->write_endio = phase3_endio;
+	atomic_set(&wb->w_sub_log_count, atomic_read(&wb->w_sub_read_count));
+
+	if (brick->log_reads) {
+		qq_inc_flying(&brick->q_phase[1]);
+		fire_writeback(&wb->w_sub_read_list, false);
+	} else { /*  shortcut */
+#ifndef SHORTCUT_1_to_3
+		qq_wb_insert(&brick->q_phase[3], wb);
+		wake_up_interruptible_all(&brick->worker_event);
+#else
+		return phase3_startio(wb);
+#endif
+	}
+
+done:
+	return true;
+
+err:
+	if (wb)
+		free_writeback(wb);
+collision:
+	return false;
+}
+
+/*********************************************************************
+ * Phase 2: log the old disk version.
+ */
+
+static inline
+void _phase2_endio(struct writeback_info *wb)
+{
+	struct trans_logger_brick *brick = wb->w_brick;
+
+	/*  queue up for the next phase */
+	qq_wb_insert(&brick->q_phase[3], wb);
+	wake_up_interruptible_all(&brick->worker_event);
+	goto out_return;
+out_return:;
+}
+
+static
+void phase2_endio(void *private, int error)
+{
+	struct trans_logger_aio_aspect *sub_aio_a;
+	struct trans_logger_brick *brick;
+	struct writeback_info *wb;
+
+	sub_aio_a = private;
+	CHECK_PTR(sub_aio_a, err);
+	wb = sub_aio_a->wb;
+	CHECK_PTR(wb, err);
+	brick = wb->w_brick;
+	CHECK_PTR(brick, err);
+
+	qq_dec_flying(&brick->q_phase[2]);
+
+	if (unlikely(error < 0)) {
+		XIO_FAT("IO error %d\n", error);
+		goto err; /*  FIXME: this leads to hanging requests. do better. */
+	}
+
+	CHECK_ATOMIC(&wb->w_sub_log_count, 1);
+	if (atomic_dec_and_test(&wb->w_sub_log_count)) {
+		banning_reset(&brick->q_phase[2].q_banning);
+		_phase2_endio(wb);
+	}
+	goto out_return;
+err:
+	XIO_FAT("hanging up....\n");
+out_return:;
+}
+
+static
+bool _phase2_startio(struct trans_logger_aio_aspect *sub_aio_a)
+{
+	struct aio_object *sub_aio = NULL;
+	struct writeback_info *wb;
+	struct trans_logger_input *input;
+	struct trans_logger_brick *brick;
+	struct log_status *logst;
+	void *data;
+	bool ok;
+
+	CHECK_PTR(sub_aio_a, err);
+	sub_aio = sub_aio_a->object;
+	CHECK_PTR(sub_aio, err);
+	wb = sub_aio_a->wb;
+	CHECK_PTR(wb, err);
+	brick = wb->w_brick;
+	CHECK_PTR(brick, err);
+	input = sub_aio_a->log_input;
+	CHECK_PTR(input, err);
+	logst = &input->logst;
+	logst->do_crc = trans_logger_do_crc;
+
+	{
+		struct log_header l = {
+			.l_stamp = sub_aio_a->stamp,
+			.l_pos = sub_aio->io_pos,
+			.l_len = sub_aio->io_len,
+			.l_code = CODE_WRITE_OLD,
+		};
+		data = log_reserve(logst, &l);
+	}
+
+	if (unlikely(!data))
+		goto err;
+
+	memcpy(data, sub_aio->io_data, sub_aio->io_len);
+
+	ok = log_finalize(logst, sub_aio->io_len, phase2_endio, sub_aio_a);
+	if (unlikely(!ok))
+		goto err;
+
+	qq_inc_flying(&brick->q_phase[2]);
+
+	return true;
+
+err:
+	XIO_FAT("cannot log old data, pos = %lld len = %d\n",
+		sub_aio ? sub_aio->io_pos : 0,
+		sub_aio ? sub_aio->io_len : 0);
+	return false;
+}
+
+static
+bool phase2_startio(struct writeback_info *wb)
+{
+	struct trans_logger_brick *brick;
+	bool ok = true;
+
+	CHECK_PTR(wb, err);
+	brick = wb->w_brick;
+	CHECK_PTR(brick, err);
+
+	if (brick->log_reads && atomic_read(&wb->w_sub_log_count) > 0) {
+		struct list_head *start;
+		struct list_head *tmp;
+
+		start = &wb->w_sub_read_list;
+		for (tmp = start->next; tmp != start; tmp = tmp->next) {
+			struct trans_logger_aio_aspect *sub_aio_a;
+			struct aio_object *sub_aio;
+
+			sub_aio_a = container_of(tmp, struct trans_logger_aio_aspect, sub_head);
+			sub_aio = sub_aio_a->object;
+
+			if (!_phase2_startio(sub_aio_a))
+				ok = false;
+		}
+		wake_up_interruptible_all(&brick->worker_event);
+	} else {
+		_phase2_endio(wb);
+	}
+	return ok;
+err:
+	return false;
+}
+
+/*********************************************************************
+ * Phase 3: overwrite old disk version with new version.
+ */
+
+static
+void phase3_endio(struct generic_callback *cb)
+{
+	struct trans_logger_aio_aspect *sub_aio_a;
+	struct writeback_info *wb;
+	struct trans_logger_brick *brick;
+
+	CHECK_PTR(cb, err);
+	sub_aio_a = cb->cb_private;
+	CHECK_PTR(sub_aio_a, err);
+	wb = sub_aio_a->wb;
+	CHECK_PTR(wb, err);
+	brick = wb->w_brick;
+	CHECK_PTR(brick, err);
+
+	if (unlikely(cb->cb_error < 0)) {
+		XIO_FAT("IO error %d\n", cb->cb_error);
+		goto err;
+	}
+
+	hash_put_all(brick, &wb->w_collect_list);
+
+	qq_dec_flying(&brick->q_phase[3]);
+	atomic_inc(&brick->total_writeback_cluster_count);
+
+	free_writeback(wb);
+
+	banning_reset(&brick->q_phase[3].q_banning);
+
+	wake_up_interruptible_all(&brick->worker_event);
+
+	goto out_return;
+err:
+	XIO_FAT("hanging up....\n");
+out_return:;
+}
+
+static
+bool phase3_startio(struct writeback_info *wb)
+{
+	struct list_head *start = &wb->w_sub_read_list;
+	struct list_head *tmp;
+
+	/* Cleanup read requests (if they exist from previous phases)
+	 */
+	while ((tmp = start->next) != start) {
+		struct trans_logger_aio_aspect *sub_aio_a;
+		struct aio_object *sub_aio;
+		struct trans_logger_input *sub_input;
+
+		list_del_init(tmp);
+
+		sub_aio_a = container_of(tmp, struct trans_logger_aio_aspect, sub_head);
+		sub_aio = sub_aio_a->object;
+		sub_input = sub_aio_a->my_input;
+
+		GENERIC_INPUT_CALL(sub_input, aio_put, sub_aio);
+	}
+
+	update_writeback_info(wb);
+
+	/* Start writeback IO
+	 */
+	qq_inc_flying(&wb->w_brick->q_phase[3]);
+	fire_writeback(&wb->w_sub_write_list, true);
+	return true;
+}
+
+/*********************************************************************
+ * The logger thread.
+ * There is only a single instance, dealing with all requests in parallel.
+ */
+
+static
+int run_aio_queue(struct logger_queue *q,
+	bool (*startio)(struct trans_logger_aio_aspect *sub_aio_a),
+	int max,
+	bool do_limit)
+{
+	struct trans_logger_brick *brick = q->q_brick;
+	int total_len = 0;
+	bool found = false;
+	bool ok;
+	int res = 0;
+
+	do {
+		struct trans_logger_aio_aspect *aio_a;
+
+		aio_a = qq_aio_fetch(q);
+		if (!aio_a)
+			goto done;
+
+		if (do_limit && likely(aio_a->object))
+			total_len += aio_a->object->io_len;
+
+		ok = startio(aio_a);
+		if (unlikely(!ok)) {
+			qq_aio_pushback(q, aio_a);
+			goto done;
+		}
+		res++;
+		found = true;
+		__trans_logger_io_put(aio_a->my_brick, aio_a);
+	} while (--max > 0);
+
+done:
+	if (found) {
+		rate_limit(&global_writeback.limiter, (total_len - 1) / 1024 + 1);
+		wake_up_interruptible_all(&brick->worker_event);
+	}
+	return res;
+}
+
+static
+int run_wb_queue(struct logger_queue *q, bool (*startio)(struct writeback_info *wb), int max)
+{
+	struct trans_logger_brick *brick = q->q_brick;
+	int total_len = 0;
+	bool found = false;
+	bool ok;
+	int res = 0;
+
+	do {
+		struct writeback_info *wb;
+
+		wb = qq_wb_fetch(q);
+		if (!wb)
+			goto done;
+
+		total_len += wb->w_len;
+
+		ok = startio(wb);
+		if (unlikely(!ok)) {
+			qq_wb_pushback(q, wb);
+			goto done;
+		}
+		res++;
+		found = true;
+	} while (--max > 0);
+
+done:
+	if (found) {
+		rate_limit(&global_writeback.limiter, (total_len - 1) / 1024 + 1);
+		wake_up_interruptible_all(&brick->worker_event);
+	}
+	return res;
+}
+
+/* Ranking tables.
+ */
+static
+struct rank_info float_queue_rank_log[] = {
+	{     0,    0 },
+	{     1,  100 },
+	{ RKI_DUMMY }
+};
+
+static
+struct rank_info float_queue_rank_io[] = {
+	{     0,    0 },
+	{     1,    1 },
+	{ RKI_DUMMY }
+};
+
+static
+struct rank_info float_fly_rank_log[] = {
+	{     0,    0 },
+	{     1,    1 },
+	{    32,   10 },
+	{ RKI_DUMMY }
+};
+
+static
+struct rank_info float_fly_rank_io[] = {
+	{     0,    0 },
+	{     1,   10 },
+	{     2,  -10 },
+	{ 10000, -200 },
+	{ RKI_DUMMY }
+};
+
+static
+struct rank_info nofloat_queue_rank_log[] = {
+	{     0,    0 },
+	{     1,   10 },
+	{ RKI_DUMMY }
+};
+
+static
+struct rank_info nofloat_queue_rank_io[] = {
+	{     0,    0 },
+	{     1,   10 },
+	{   100,  100 },
+	{ RKI_DUMMY }
+};
+
+#define nofloat_fly_rank_log float_fly_rank_log
+
+static
+struct rank_info nofloat_fly_rank_io[] = {
+	{     0,    0 },
+	{     1,   10 },
+	{   128,    8 },
+	{   129, -200 },
+	{ RKI_DUMMY }
+};
+
+static
+struct rank_info *queue_ranks[2][LOGGER_QUEUES] = {
+	[0] = {
+		[0] = float_queue_rank_log,
+		[1] = float_queue_rank_io,
+		[2] = float_queue_rank_io,
+		[3] = float_queue_rank_io,
+	},
+	[1] = {
+		[0] = nofloat_queue_rank_log,
+		[1] = nofloat_queue_rank_io,
+		[2] = nofloat_queue_rank_io,
+		[3] = nofloat_queue_rank_io,
+	},
+};
+static
+struct rank_info *fly_ranks[2][LOGGER_QUEUES] = {
+	[0] = {
+		[0] = float_fly_rank_log,
+		[1] = float_fly_rank_io,
+		[2] = float_fly_rank_io,
+		[3] = float_fly_rank_io,
+	},
+	[1] = {
+		[0] = nofloat_fly_rank_log,
+		[1] = nofloat_fly_rank_io,
+		[2] = nofloat_fly_rank_io,
+		[3] = nofloat_fly_rank_io,
+	},
+};
+
+static
+struct rank_info extra_rank_aio_flying[] = {
+	{     0,    0 },
+	{     1,   10 },
+	{    16,   30 },
+	{    31,    0 },
+	{    32, -200 },
+	{ RKI_DUMMY }
+};
+
+static
+struct rank_info global_rank_aio_flying[] = {
+	{     0,    0 },
+	{    63,    0 },
+	{    64, -200 },
+	{ RKI_DUMMY }
+};
+
+static
+int _do_ranking(struct trans_logger_brick *brick, struct rank_data rkd[])
+{
+	int res;
+	int i;
+	int floating_mode;
+	int aio_flying;
+	bool delay_callers;
+
+	ranking_start(rkd, LOGGER_QUEUES);
+
+	/*  check the memory situation... */
+	delay_callers = false;
+	floating_mode = 1;
+	if (brick_global_memlimit >= 1024) {
+		int global_mem_used = atomic64_read(&global_mshadow_used) / 1024;
+
+		trans_logger_mem_usage = global_mem_used;
+
+		floating_mode = (global_mem_used < brick_global_memlimit / 2) ? 0 : 1;
+
+		if (global_mem_used >= brick_global_memlimit)
+			delay_callers = true;
+
+	} else if (brick->shadow_mem_limit >= 8) {
+		int local_mem_used = atomic64_read(&brick->shadow_mem_used) / 1024;
+
+		floating_mode = (local_mem_used < brick->shadow_mem_limit / 2) ? 0 : 1;
+
+		if (local_mem_used >= brick->shadow_mem_limit)
+			delay_callers = true;
+	}
+	if (delay_callers) {
+		if (!brick->delay_callers) {
+			brick->delay_callers = true;
+			atomic_inc(&brick->total_delay_count);
+		}
+	} else if (brick->delay_callers) {
+		brick->delay_callers = false;
+		wake_up_interruptible(&brick->caller_event);
+	}
+
+	/*  global limit for flying aios */
+	ranking_compute(&rkd[0], global_rank_aio_flying, atomic_read(&global_aio_flying));
+
+	/*  local limit for flying aios */
+	aio_flying = 0;
+	for (i = TL_INPUT_LOG1; i <= TL_INPUT_LOG2; i++) {
+		struct trans_logger_input *input = brick->inputs[i];
+
+		aio_flying += atomic_read(&input->logst.aio_flying);
+	}
+
+	/*  obey the basic rules... */
+	for (i = 0; i < LOGGER_QUEUES; i++) {
+		int queued = atomic_read(&brick->q_phase[i].q_queued);
+		int flying;
+
+		/* This must come first.
+		 * When a queue is empty, you must not credit any positive points.
+		 * Otherwise, (almost) infinite selection of untreatable
+		 * queues may occur.
+		 */
+		if (queued <= 0)
+			continue;
+
+		if (banning_is_hit(&brick->q_phase[i].q_banning))
+			break;
+
+		if (i == 0) {
+			/*  limit aio IO parallelism on transaction log */
+			ranking_compute(&rkd[0], extra_rank_aio_flying, aio_flying);
+		} else if (i == 1 && !floating_mode) {
+			struct trans_logger_brick *leader;
+			int lim;
+
+			if (!aio_flying && atomic_read(&brick->q_phase[0].q_queued) > 0)
+				break;
+
+			leader = elect_leader(&global_writeback);
+			if (leader != brick)
+				break;
+
+			if (banning_is_hit(&xio_global_ban))
+				break;
+
+			lim = rate_limit(&global_writeback.limiter, 0);
+			if (lim > 0)
+				break;
+		}
+
+		ranking_compute(&rkd[i], queue_ranks[floating_mode][i], queued);
+
+		flying = atomic_read(&brick->q_phase[i].q_flying);
+
+		ranking_compute(&rkd[i], fly_ranks[floating_mode][i], flying);
+	}
+
+	/*  finalize it */
+	ranking_stop(rkd, LOGGER_QUEUES);
+
+	res = ranking_select(rkd, LOGGER_QUEUES);
+
+	return res;
+}
+
+static
+void _init_input(struct trans_logger_input *input, loff_t start_pos, loff_t end_pos)
+{
+	struct trans_logger_brick *brick = input->brick;
+	struct log_status *logst = &input->logst;
+
+	init_logst(logst, (void *)input, start_pos, end_pos);
+	logst->signal_event = &brick->worker_event;
+	logst->align_size = CONF_TRANS_ALIGN;
+	logst->chunk_size = CONF_TRANS_CHUNKSIZE;
+	logst->max_size = CONF_TRANS_MAX_AIO_SIZE;
+
+	input->inf.inf_min_pos = start_pos;
+	input->inf.inf_max_pos = end_pos;
+	get_lamport(&input->inf.inf_max_pos_stamp);
+	memcpy(&input->inf.inf_min_pos_stamp, &input->inf.inf_max_pos_stamp, sizeof(input->inf.inf_min_pos_stamp));
+
+	logst->log_pos = start_pos;
+	input->inf.inf_log_pos = start_pos;
+	input->inf_last_jiffies = jiffies;
+	input->inf.inf_is_replaying = false;
+	input->inf.inf_is_logging = false;
+
+	input->is_operating = true;
+}
+
+static
+void _init_inputs(struct trans_logger_brick *brick, bool is_first)
+{
+	struct trans_logger_input *input;
+	int old_nr = brick->old_input_nr;
+	int log_nr = brick->log_input_nr;
+	int new_nr = brick->new_input_nr;
+
+	if (!is_first &&
+	    (new_nr == log_nr ||
+	     log_nr != old_nr)) {
+		goto done;
+	}
+	if (unlikely(new_nr < TL_INPUT_LOG1 || new_nr > TL_INPUT_LOG2)) {
+		XIO_ERR("bad new_input_nr = %d\n", new_nr);
+		goto done;
+	}
+
+	input = brick->inputs[new_nr];
+	CHECK_PTR(input, done);
+
+	if (input->is_operating || !input->connect)
+		goto done;
+
+	down(&input->inf_mutex);
+
+	_init_input(input, 0, 0);
+	input->inf.inf_is_logging = is_first;
+
+	/*  from now on, new requests should go to the new input */
+	brick->log_input_nr = new_nr;
+	XIO_INF("switched over to new logfile %d (old = %d)\n", new_nr, old_nr);
+
+	/* Flush the old log buffer and update its symlinks.
+	 * Notice: for some short time, _both_ logfiles may grow
+	 * due to (harmless) races with log_flush().
+	 */
+	if (likely(!is_first)) {
+		struct trans_logger_input *other_input = brick->inputs[old_nr];
+
+		down(&other_input->inf_mutex);
+		log_flush(&other_input->logst);
+		_inf_callback(other_input, true);
+		up(&other_input->inf_mutex);
+	}
+
+	_inf_callback(input, true);
+
+	up(&input->inf_mutex);
+done:;
+}
+
+static
+int _nr_flying_inputs(struct trans_logger_brick *brick)
+{
+	int count = 0;
+	int i;
+
+	for (i = TL_INPUT_LOG1; i <= TL_INPUT_LOG2; i++) {
+		struct trans_logger_input *input = brick->inputs[i];
+		struct log_status *logst = &input->logst;
+
+		if (input->is_operating)
+			count += logst->count;
+	}
+	return count;
+}
+
+static
+void _flush_inputs(struct trans_logger_brick *brick)
+{
+	int i;
+
+	for (i = TL_INPUT_LOG1; i <= TL_INPUT_LOG2; i++) {
+		struct trans_logger_input *input = brick->inputs[i];
+		struct log_status *logst = &input->logst;
+
+		if (input->is_operating && logst->count > 0) {
+			atomic_inc(&brick->total_flush_count);
+			log_flush(logst);
+		}
+	}
+}
+
+static
+void _exit_inputs(struct trans_logger_brick *brick, bool force)
+{
+	int i;
+
+	for (i = TL_INPUT_LOG1; i <= TL_INPUT_LOG2; i++) {
+		struct trans_logger_input *input = brick->inputs[i];
+		struct log_status *logst = &input->logst;
+
+		if (input->is_operating &&
+		    (force || !input->connect)) {
+			bool old_replaying = input->inf.inf_is_replaying;
+			bool old_logging = input->inf.inf_is_logging;
+
+			XIO_DBG("cleaning up input %d (log = %d old = %d), old_replaying = %d old_logging = %d\n",
+				i,
+				brick->log_input_nr,
+				brick->old_input_nr,
+				old_replaying,
+				old_logging);
+			exit_logst(logst);
+			/*  no locking here: we should be the only thread doing this. */
+			_inf_callback(input, true);
+			input->inf_last_jiffies = 0;
+			input->inf.inf_is_replaying = false;
+			input->inf.inf_is_logging = false;
+			input->is_operating = false;
+			if (i == brick->old_input_nr && i != brick->log_input_nr) {
+				struct trans_logger_input *other_input = brick->inputs[brick->log_input_nr];
+
+				down(&other_input->inf_mutex);
+				brick->old_input_nr = brick->log_input_nr;
+				other_input->inf.inf_is_replaying = old_replaying;
+				other_input->inf.inf_is_logging = old_logging;
+				_inf_callback(other_input, true);
+				up(&other_input->inf_mutex);
+			}
+		}
+	}
+}
+
+/* Performance-critical:
+ * Calling log_flush() too often may result in
+ * increased overhead (and thus in lower throughput).
+ * Call it only when the IO scheduler need not do anything else.
+ * OTOH, calling it too seldom may hold back
+ * IO completion for the end user for too long time.
+ *
+ * Be careful to flush any leftovers in the log buffer, at least after
+ * some short delay.
+ *
+ * Description of flush_mode:
+ *  0 = flush unconditionally
+ *  1 = flush only when nothing can be appended to the transaction log
+ *  2 = see 1 && flush only when the user is waiting for an answer
+ *  3 = see 1 && not 2 && flush only when there is no other activity (background mode)
+ * Notice: 3 makes only sense for leftovers where the user is _not_ waiting for
+ */
+static inline
+void flush_inputs(struct trans_logger_brick *brick, int flush_mode)
+{
+	if (flush_mode < 1 ||
+	    /*	there is nothing to append any more */
+	    (atomic_read(&brick->q_phase[0].q_queued) <= 0 &&
+	     /*  and the user is waiting for an answer */
+	     (flush_mode < 2 ||
+	      atomic_read(&brick->log_fly_count) > 0 ||
+	     /*  else flush any leftovers in background, when there is no writeback activity */
+	      (flush_mode == 3 &&
+	       atomic_read(&brick->q_phase[1].q_flying) + atomic_read(&brick->q_phase[3].q_flying) <= 0))))
+		_flush_inputs(brick);
+}
+
+static
+void trans_logger_log(struct trans_logger_brick *brick)
+{
+	struct rank_data rkd[LOGGER_QUEUES] = {};
+	long long old_jiffies = jiffies;
+	long long work_jiffies = jiffies;
+	int interleave = 0;
+	int nr_flying;
+
+	brick->replay_code = 0; /*  indicates "running" */
+	brick->disk_io_error = 0;
+
+	_init_inputs(brick, true);
+
+	xio_set_power_on_led((void *)brick, true);
+
+	while (!brick_thread_should_stop() || _congested(brick)) {
+		int winner;
+		int nr;
+
+		wait_event_interruptible_timeout(
+			brick->worker_event,
+			({
+				winner = _do_ranking(brick, rkd);
+				if (winner < 0) { /*  no more work to do */
+					int flush_mode = 2 - ((int)(jiffies - work_jiffies)) / (HZ * 2);
+
+					flush_inputs(brick, flush_mode);
+					interleave = 0;
+				} else { /*  reset the timer whenever something is to do */
+					work_jiffies = jiffies;
+				}
+				winner >= 0;
+			}),
+			HZ / 10);
+
+		atomic_inc(&brick->total_round_count);
+
+		if (brick->cease_logging)
+			brick->stopped_logging = true;
+		else if (brick->stopped_logging && !_congested(brick))
+			brick->stopped_logging = false;
+
+		_init_inputs(brick, false);
+
+		switch (winner) {
+		case 0:
+			interleave = 0;
+			nr = run_aio_queue(&brick->q_phase[0],
+				prep_phase_startio,
+				brick->q_phase[0].q_batchlen,
+				true);
+			goto done;
+		case 1:
+			if (interleave >= trans_logger_max_interleave && trans_logger_max_interleave >= 0) {
+				interleave = 0;
+				flush_inputs(brick, 3);
+			}
+			nr = run_aio_queue(&brick->q_phase[1], phase1_startio, brick->q_phase[1].q_batchlen, true);
+			interleave += nr;
+			goto done;
+		case 2:
+			interleave = 0;
+			nr = run_wb_queue(&brick->q_phase[2], phase2_startio, brick->q_phase[2].q_batchlen);
+			goto done;
+		case 3:
+			if (interleave >= trans_logger_max_interleave && trans_logger_max_interleave >= 0) {
+				interleave = 0;
+				flush_inputs(brick, 3);
+			}
+			nr = run_wb_queue(&brick->q_phase[3], phase3_startio, brick->q_phase[3].q_batchlen);
+			interleave += nr;
+done:
+			if (unlikely(nr <= 0)) {
+				/* This should not happen!
+				 * However, in error situations, the ranking
+				 * algorithm cannot foresee anything.
+				 */
+				brick->q_phase[winner].no_progress_count++;
+				banning_hit(&brick->q_phase[winner].q_banning, 10000);
+				flush_inputs(brick, 0);
+			}
+			ranking_select_done(rkd, winner, nr);
+			break;
+
+		default:
+			break;
+		}
+
+		/* Update symlinks even during pauses.
+		 */
+		if (winner < 0 && ((long long)jiffies) - old_jiffies >= HZ) {
+			int i;
+
+			old_jiffies = jiffies;
+			for (i = TL_INPUT_LOG1; i <= TL_INPUT_LOG2; i++) {
+				struct trans_logger_input *input = brick->inputs[i];
+
+				down(&input->inf_mutex);
+				_inf_callback(input, false);
+				up(&input->inf_mutex);
+			}
+		}
+
+		_exit_inputs(brick, false);
+	}
+
+	for (;;) {
+		_exit_inputs(brick, true);
+		nr_flying = _nr_flying_inputs(brick);
+		if (nr_flying <= 0)
+			break;
+		XIO_INF("%d inputs are operating\n", nr_flying);
+		brick_msleep(1000);
+	}
+}
+
+/***************************** log replay *****************************/
+
+static
+void replay_endio(struct generic_callback *cb)
+{
+	struct trans_logger_aio_aspect *aio_a = cb->cb_private;
+	struct trans_logger_brick *brick;
+	bool ok;
+	unsigned long flags;
+
+	LAST_CALLBACK(cb);
+	CHECK_PTR(aio_a, err);
+	brick = aio_a->my_brick;
+	CHECK_PTR(brick, err);
+
+	if (unlikely(cb->cb_error < 0)) {
+		brick->disk_io_error = cb->cb_error;
+		XIO_ERR("IO error = %d\n", cb->cb_error);
+	}
+
+	spin_lock_irqsave(&brick->replay_lock, flags);
+	ok = !list_empty(&aio_a->replay_head);
+	list_del_init(&aio_a->replay_head);
+	spin_unlock_irqrestore(&brick->replay_lock, flags);
+
+	if (likely(ok))
+		atomic_dec(&brick->replay_count);
+	else
+		XIO_ERR("callback with empty replay_head (replay_count=%d)\n", atomic_read(&brick->replay_count));
+	wake_up_interruptible_all(&brick->worker_event);
+	goto out_return;
+err:
+	XIO_FAT("cannot handle replay IO\n");
+out_return:;
+}
+
+static
+bool _has_conflict(struct trans_logger_brick *brick, struct trans_logger_aio_aspect *aio_a)
+{
+	struct aio_object *aio = aio_a->object;
+	struct list_head *tmp;
+	bool res = false;
+	unsigned long flags;
+
+/**/
+
+	spin_lock_irqsave(&brick->replay_lock, flags);
+
+	for (tmp = brick->replay_list.next; tmp != &brick->replay_list; tmp = tmp->next) {
+		struct trans_logger_aio_aspect *tmp_a;
+		struct aio_object *tmp_aio;
+
+		tmp_a = container_of(tmp, struct trans_logger_aio_aspect, replay_head);
+		tmp_aio = tmp_a->object;
+		if (tmp_aio->io_pos + tmp_aio->io_len > aio->io_pos && tmp_aio->io_pos < aio->io_pos + aio->io_len) {
+			res = true;
+			break;
+		}
+	}
+
+	spin_unlock_irqrestore(&brick->replay_lock, flags);
+	return res;
+}
+
+static
+void wait_replay(struct trans_logger_brick *brick, struct trans_logger_aio_aspect *aio_a)
+{
+	const int max = 512; /*  limit parallelism somewhat */
+	int conflicts = 0;
+	bool ok = false;
+	bool was_empty;
+	unsigned long flags;
+
+	wait_event_interruptible_timeout(brick->worker_event,
+					 atomic_read(&brick->replay_count) < max
+					 && (_has_conflict(brick, aio_a) ? conflicts++ : (ok = true), ok),
+					 60 * HZ);
+
+	atomic_inc(&brick->total_replay_count);
+	if (conflicts)
+		atomic_inc(&brick->total_replay_conflict_count);
+
+	spin_lock_irqsave(&brick->replay_lock, flags);
+	was_empty = !!list_empty(&aio_a->replay_head);
+	if (likely(was_empty))
+		atomic_inc(&brick->replay_count);
+	else
+		list_del(&aio_a->replay_head);
+	list_add(&aio_a->replay_head, &brick->replay_list);
+	spin_unlock_irqrestore(&brick->replay_lock, flags);
+
+	if (unlikely(!was_empty)) {
+		XIO_ERR("replay_head was already used (ok=%d, conflicts=%d, replay_count=%d)\n",
+			ok,
+			conflicts,
+			atomic_read(&brick->replay_count));
+	}
+}
+
+static
+int replay_data(struct trans_logger_brick *brick, loff_t pos, void *buf, int len)
+{
+	struct trans_logger_input *input = brick->inputs[TL_INPUT_WRITEBACK];
+	int status;
+
+	if (!input->connect)
+		input = brick->inputs[TL_INPUT_READ];
+
+	/* TODO for better efficiency:
+	 * Instead of starting IO here, just put the data into the hashes
+	 * and queues such that ordinary IO will be corrected.
+	 * Writeback will be lazy then.
+	 * The switch infrastructure must be changed before this
+	 * becomes possible.
+	 */
+#ifdef REPLAY_DATA
+	while (len > 0) {
+		struct aio_object *aio;
+		struct trans_logger_aio_aspect *aio_a;
+
+		status = -ENOMEM;
+		aio = trans_logger_alloc_aio(brick);
+		aio_a = trans_logger_aio_get_aspect(brick, aio);
+		CHECK_PTR(aio_a, done);
+		CHECK_ASPECT(aio_a, aio, done);
+
+		aio->io_pos = pos;
+		aio->io_data = NULL;
+		aio->io_len = len;
+		aio->io_may_write = WRITE;
+		aio->io_rw = WRITE;
+
+		status = GENERIC_INPUT_CALL(input, aio_get, aio);
+		if (unlikely(status < 0)) {
+			XIO_ERR("cannot get aio, status = %d\n", status);
+			goto done;
+		}
+		if (unlikely(!aio->io_data)) {
+			status = -ENOMEM;
+			XIO_ERR("cannot get aio, status = %d\n", status);
+			goto done;
+		}
+		if (unlikely(aio->io_len <= 0 || aio->io_len > len)) {
+			status = -EINVAL;
+			XIO_ERR("bad aio len = %d (requested = %d)\n", aio->io_len, len);
+			goto done;
+		}
+
+		wait_replay(brick, aio_a);
+
+		memcpy(aio->io_data, buf, aio->io_len);
+
+		SETUP_CALLBACK(aio, replay_endio, aio_a);
+		aio_a->my_brick = brick;
+
+		GENERIC_INPUT_CALL(input, aio_io, aio);
+
+		if (unlikely(aio->io_len <= 0)) {
+			status = -EINVAL;
+			XIO_ERR("bad aio len = %d (requested = %d)\n", aio->io_len, len);
+			goto done;
+		}
+
+		pos += aio->io_len;
+		buf += aio->io_len;
+		len -= aio->io_len;
+
+		GENERIC_INPUT_CALL(input, aio_put, aio);
+	}
+#endif
+	status = 0;
+done:
+	return status;
+}
+
+static
+void trans_logger_replay(struct trans_logger_brick *brick)
+{
+	struct trans_logger_input *input = brick->inputs[brick->log_input_nr];
+	struct log_header lh = {};
+	loff_t start_pos;
+	loff_t end_pos;
+	loff_t finished_pos = -1;
+	loff_t new_finished_pos = -1;
+	long long old_jiffies = jiffies;
+	int nr_flying;
+	int backoff = 0;
+	int status = 0;
+
+	brick->replay_code = 0; /*  indicates "running" */
+	brick->disk_io_error = 0;
+
+	start_pos = brick->replay_start_pos;
+	end_pos = brick->replay_end_pos;
+	brick->replay_current_pos = start_pos;
+
+	_init_input(input, start_pos, end_pos);
+
+	input->inf.inf_min_pos = start_pos;
+	input->inf.inf_max_pos = end_pos;
+	input->inf.inf_log_pos = end_pos;
+	input->inf.inf_is_replaying = true;
+	input->inf.inf_is_logging = false;
+
+	XIO_INF("starting replay from %lld to %lld\n", start_pos, end_pos);
+
+	xio_set_power_on_led((void *)brick, true);
+
+	for (;;) {
+		void *buf = NULL;
+		int len = 0;
+
+		if (brick_thread_should_stop() ||
+		   (!brick->continuous_replay_mode && finished_pos >= brick->replay_end_pos)) {
+			status = 0; /*	treat as EOF */
+			break;
+		}
+
+		status = log_read(&input->logst, false, &lh, &buf, &len);
+
+		new_finished_pos = input->logst.log_pos + input->logst.offset;
+		XIO_RPL("read	%lld %lld\n", finished_pos, new_finished_pos);
+
+		if (status == -EAGAIN) {
+			loff_t remaining = brick->replay_end_pos - new_finished_pos;
+
+			XIO_DBG("got -EAGAIN, remaining = %lld\n", remaining);
+			if (brick->replay_tolerance > 0 && remaining < brick->replay_tolerance) {
+				XIO_WRN("logfile is truncated at position %lld (end_pos = %lld, remaining = %lld, tolerance = %d)\n",
+					 new_finished_pos,
+					 brick->replay_end_pos,
+					 remaining,
+					 brick->replay_tolerance);
+				finished_pos = new_finished_pos;
+				brick->replay_code = status;
+				break;
+			}
+			brick_msleep(backoff);
+			if (backoff < trans_logger_replay_timeout * 1000) {
+				backoff += 100;
+			} else {
+				XIO_WRN("logfile replay not possible at position %lld (end_pos = %lld, remaining = %lld), please check/repair your logfile in userspace by some tool!\n",
+					 new_finished_pos,
+					 brick->replay_end_pos,
+					 remaining);
+				brick->replay_code = status;
+				break;
+			}
+			continue;
+		}
+		if (unlikely(status < 0)) {
+			brick->replay_code = status;
+			XIO_WRN("cannot read logfile data, status = %d\n", status);
+			break;
+		}
+
+		if ((!status && len <= 0) ||
+		   new_finished_pos > brick->replay_end_pos) { /*  EOF -> wait until brick_thread_should_stop() */
+			XIO_DBG("EOF at %lld (old = %lld, end_pos = %lld)\n",
+				new_finished_pos,
+				finished_pos,
+				brick->replay_end_pos);
+			if (!brick->continuous_replay_mode) {
+				/*  notice: finished_pos remains at old value here! */
+				break;
+			}
+			brick_msleep(1000);
+			continue;
+		}
+
+		if (lh.l_code != CODE_WRITE_NEW) {
+			/* ignore other records silently */;
+		} else if (unlikely(brick->disk_io_error)) {
+			status = brick->disk_io_error;
+			brick->replay_code = status;
+			XIO_ERR("IO error %d\n", status);
+			break;
+		} else if (likely(buf && len)) {
+			if (brick->replay_limiter)
+				rate_limit_sleep(brick->replay_limiter, (len - 1) / 1024 + 1);
+			status = replay_data(brick, lh.l_pos, buf, len);
+			XIO_RPL("replay %lld %lld (pos=%lld status=%d)\n",
+				finished_pos,
+				new_finished_pos,
+				lh.l_pos,
+				status);
+			if (unlikely(status < 0)) {
+				brick->replay_code = status;
+				XIO_ERR("cannot replay data at pos = %lld len = %d, status = %d\n",
+					lh.l_pos,
+					len,
+					status);
+				break;
+			} else {
+				finished_pos = new_finished_pos;
+			}
+		}
+
+		/*  do this _after_ any opportunities for errors... */
+		if ((atomic_read(&brick->replay_count) <= 0 ||
+		     ((long long)jiffies) - old_jiffies >= HZ * 3) &&
+		    finished_pos >= 0) {
+			/*  for safety, wait until the IO queue has drained. */
+			wait_event_interruptible_timeout(brick->worker_event,
+				atomic_read(&brick->replay_count) <= 0,
+				30 * HZ);
+
+			if (unlikely(brick->disk_io_error)) {
+				status = brick->disk_io_error;
+				brick->replay_code = status;
+				XIO_ERR("IO error %d\n", status);
+				break;
+			}
+
+			down(&input->inf_mutex);
+			input->inf.inf_min_pos = finished_pos;
+			get_lamport(&input->inf.inf_min_pos_stamp);
+			old_jiffies = jiffies;
+			_inf_callback(input, false);
+			up(&input->inf_mutex);
+		}
+		_exit_inputs(brick, false);
+	}
+
+	XIO_INF("waiting for finish...\n");
+
+	wait_event_interruptible_timeout(brick->worker_event, atomic_read(&brick->replay_count) <= 0, 60 * HZ);
+
+	if (unlikely(finished_pos > brick->replay_end_pos)) {
+		XIO_ERR("finished_pos too large: %lld + %d = %lld > %lld\n",
+			input->logst.log_pos,
+			input->logst.offset,
+			finished_pos,
+			brick->replay_end_pos);
+	}
+
+	if (finished_pos >= 0 && !brick->disk_io_error) {
+		input->inf.inf_min_pos = finished_pos;
+		brick->replay_current_pos = finished_pos;
+	}
+
+	get_lamport(&input->inf.inf_min_pos_stamp);
+
+	if (status >= 0 && finished_pos == brick->replay_end_pos) {
+		XIO_INF("replay finished at %lld\n", finished_pos);
+		brick->replay_code = 1;
+	} else if (status == -EAGAIN && finished_pos + brick->replay_tolerance > brick->replay_end_pos) {
+		XIO_INF("TOLERANCE: logfile is incomplete at %lld (of %lld)\n", finished_pos, brick->replay_end_pos);
+		brick->replay_code = 2;
+	} else if (status < 0) {
+		if (finished_pos < 0)
+			finished_pos = new_finished_pos;
+		if (finished_pos + brick->replay_tolerance > brick->replay_end_pos) {
+			XIO_INF("TOLERANCE: logfile is incomplete at %lld (of %lld), status = %d\n",
+				finished_pos,
+				brick->replay_end_pos,
+				status);
+		} else {
+			XIO_ERR("replay error %d at %lld (of %lld)\n", status, finished_pos, brick->replay_end_pos);
+		}
+		brick->replay_code = status;
+	} else {
+		XIO_INF("replay stopped prematurely at %lld (of %lld)\n", finished_pos, brick->replay_end_pos);
+		brick->replay_code = 2;
+	}
+
+	for (;;) {
+		_exit_inputs(brick, true);
+		nr_flying = _nr_flying_inputs(brick);
+		if (nr_flying <= 0)
+			break;
+		XIO_INF("%d inputs are operating\n", nr_flying);
+		brick_msleep(1000);
+	}
+
+	local_trigger();
+
+	while (!brick_thread_should_stop())
+		brick_msleep(500);
+}
+
+/************************ logger thread * switching ************************/
+
+static
+int trans_logger_thread(void *data)
+{
+	struct trans_logger_output *output = data;
+	struct trans_logger_brick *brick = output->brick;
+
+	XIO_INF("........... logger has started.\n");
+
+	if (brick->replay_mode)
+		trans_logger_replay(brick);
+	else
+		trans_logger_log(brick);
+	XIO_INF("........... logger has stopped.\n");
+	xio_set_power_on_led((void *)brick, false);
+	xio_set_power_off_led((void *)brick, true);
+	return 0;
+}
+
+static
+int trans_logger_switch(struct trans_logger_brick *brick)
+{
+	static int index;
+	struct trans_logger_output *output = brick->outputs[0];
+
+	if (brick->power.button) {
+		if (!brick->thread && brick->power.off_led) {
+			xio_set_power_off_led((void *)brick, false);
+
+			brick->thread = brick_thread_create(trans_logger_thread, output, "xio_logger%d", index++);
+			if (unlikely(!brick->thread)) {
+				XIO_ERR("cannot create logger thread\n");
+				return -ENOENT;
+			}
+		}
+	} else {
+		xio_set_power_on_led((void *)brick, false);
+		if (brick->thread) {
+			XIO_INF("stopping thread...\n");
+			brick_thread_stop(brick->thread);
+			brick->thread = NULL;
+		}
+	}
+	return 0;
+}
+
+/*************** informational * statistics **************/
+
+static
+char *trans_logger_statistics(struct trans_logger_brick *brick, int verbose)
+{
+	char *res = brick_string_alloc(1024);
+
+	snprintf(res, 1023,
+		 "mode replay=%d continuous=%d replay_code=%d disk_io_error=%d log_reads=%d | cease_logging=%d stopped_logging=%d congested=%d | replay_start_pos = %lld replay_end_pos = %lld | new_input_nr = %d log_input_nr = %d (old = %d) inf_min_pos1 = %lld inf_max_pos1 = %lld inf_min_pos2 = %lld inf_max_pos2 = %lld | total hash_insert=%d hash_find=%d hash_extend=%d replay=%d replay_conflict=%d  (%d%%) callbacks=%d reads=%d writes=%d flushes=%d (%d%%) wb_clusters=%d writebacks=%d (%d%%) shortcut=%d (%d%%) mshadow=%d sshadow=%d mshadow_buffered=%d sshadow_buffered=%d rounds=%d restarts=%d delays=%d phase0=%d phase1=%d phase2=%d phase3=%d | current #aios = %d shadow_mem_used=%ld/%lld replay_count=%d mshadow=%d/%d sshadow=%d hash_count=%d balance=%d/%d/%d/%d pos_count1=%d pos_count2=%d log_aios1=%d log_aios2=%d any_fly=%d log_fly=%d aio_flying1=%d aio_flying2=%d phase0=%d+%d <%d/%d> phase1=%d+%d <%d/%d> phase2=%d+%d <%d/%d> phase3=%d+%d <%d/%d>\n",
+		 brick->replay_mode,
+		 brick->continuous_replay_mode,
+		 brick->replay_code,
+		 brick->disk_io_error,
+		 brick->log_reads,
+		 brick->cease_logging,
+		 brick->stopped_logging,
+		 _congested(brick),
+		 brick->replay_start_pos,
+		 brick->replay_end_pos,
+		 brick->new_input_nr,
+		 brick->log_input_nr,
+		 brick->old_input_nr,
+		 brick->inputs[TL_INPUT_LOG1]->inf.inf_min_pos,
+		 brick->inputs[TL_INPUT_LOG1]->inf.inf_max_pos,
[… more lines in the raw message]

Lines of this file are not shown. View the raw message.

include/linux/xio/xio_trans_logger.h new file +263 -0

263 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.6.4

Part of the diff is not shown here. View the raw message.

#23 lore raw

RFC 23/31 mars: add new module xio_server

In reply to Thomas Schoebel-Theuer
---
 drivers/staging/mars/xio_bricks/xio_server.c | 486 +++++++++++++++++++++++++++
 include/linux/xio/xio_server.h               |  91 +++++
 2 files changed, 577 insertions(+)
 create mode 100644 drivers/staging/mars/xio_bricks/xio_server.c
 create mode 100644 include/linux/xio/xio_server.h
drivers/staging/mars/xio_bricks/xio_server.c new file +486 -0
new file mode 100644
index 0000000..95a3327
@@ -0,0 +1,486 @@
+/*
+ * MARS Long Distance Replication Software
+ *
+ * Copyright (C) 2010-2014 Thomas Schoebel-Theuer
+ * Copyright (C) 2011-2014 1&1 Internet AG
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+/*  Server brick (just for demonstration) */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/string.h>
+
+#include <linux/brick/brick.h>
+#include <linux/xio/xio.h>
+#include <linux/xio/xio_bio.h>
+#include <linux/xio/xio_sio.h>
+
+/************************ own type definitions ***********************/
+
+#include <linux/xio/xio_server.h>
+
+static struct xio_socket server_socket[NR_SERVER_SOCKETS];
+static struct task_struct *server_threads[NR_SERVER_SOCKETS];
+
+/************************ own helper functions ***********************/
+
+int cb_thread(void *data)
+{
+	struct server_brick *brick = data;
+	struct xio_socket *sock = &brick->handler_socket;
+	bool aborted = false;
+	bool ok = xio_get_socket(sock);
+	int status = -EINVAL;
+
+	XIO_DBG("--------------- cb_thread starting on socket #%d, ok = %d\n", sock->s_debug_nr, ok);
+	if (!ok)
+		goto done;
+
+	brick->cb_running = true;
+	wake_up_interruptible(&brick->startup_event);
+
+	while (!brick_thread_should_stop() || !list_empty(&brick->cb_read_list) || !list_empty(&brick->cb_write_list) || atomic_read(&brick->in_flight) > 0) {
+		struct server_aio_aspect *aio_a;
+		struct aio_object *aio;
+		struct list_head *tmp;
+		unsigned long flags;
+
+		wait_event_interruptible_timeout(
+			brick->cb_event,
+			!list_empty(&brick->cb_read_list) ||
+			!list_empty(&brick->cb_write_list),
+			1 * HZ);
+
+		spin_lock_irqsave(&brick->cb_lock, flags);
+		tmp = brick->cb_write_list.next;
+		if (tmp == &brick->cb_write_list) {
+			tmp = brick->cb_read_list.next;
+			if (tmp == &brick->cb_read_list) {
+				spin_unlock_irqrestore(&brick->cb_lock, flags);
+				brick_msleep(1000 / HZ);
+				continue;
+			}
+		}
+		list_del_init(tmp);
+		spin_unlock_irqrestore(&brick->cb_lock, flags);
+
+		aio_a = container_of(tmp, struct server_aio_aspect, cb_head);
+		aio = aio_a->object;
+		status = -EINVAL;
+		CHECK_PTR(aio, err);
+
+		status = 0;
+		/* Report a remote error when consistency cannot be guaranteed,
+		 * e.g. emergency mode during sync.
+		 */
+		if (brick->conn_brick && brick->conn_brick->mode_ptr && *brick->conn_brick->mode_ptr < 0
+		    && aio->object_cb)
+			aio->object_cb->cb_error = *brick->conn_brick->mode_ptr;
+		if (!aborted) {
+			down(&brick->socket_sem);
+			status = xio_send_cb(sock, aio);
+			up(&brick->socket_sem);
+		}
+
+err:
+		if (unlikely(status < 0) && !aborted) {
+			aborted = true;
+			XIO_WRN("cannot send response, status = %d\n", status);
+			/* Just shutdown the socket and forget all pending
+			 * requests.
+			 * The _client_ is responsible for resending
+			 * any lost operations.
+			 */
+			xio_shutdown_socket(sock);
+		}
+
+		if (aio_a->data) {
+			brick_block_free(aio_a->data, aio_a->len);
+			aio->io_data = NULL;
+		}
+		if (aio_a->do_put) {
+			GENERIC_INPUT_CALL(brick->inputs[0], aio_put, aio);
+			atomic_dec(&brick->in_flight);
+		} else {
+			obj_free(aio);
+		}
+	}
+
+	xio_shutdown_socket(sock);
+	xio_put_socket(sock);
+
+done:
+	XIO_DBG("---------- cb_thread terminating, status = %d\n", status);
+	wake_up_interruptible(&brick->startup_event);
+	return status;
+}
+
+static
+void server_endio(struct generic_callback *cb)
+{
+	struct server_aio_aspect *aio_a;
+	struct aio_object *aio;
+	struct server_brick *brick;
+	int rw;
+	unsigned long flags;
+
+	aio_a = cb->cb_private;
+	CHECK_PTR(aio_a, err);
+	aio = aio_a->object;
+	CHECK_PTR(aio, err);
+	LAST_CALLBACK(cb);
+	if (unlikely(cb != &aio->_object_cb))
+		XIO_ERR("bad cb pointer %p != %p\n", cb, &aio->_object_cb);
+
+	brick = aio_a->brick;
+	if (unlikely(!brick)) {
+		XIO_WRN("late IO callback -- cannot do anything\n");
+		goto out_return;
+	}
+
+	rw = aio->io_rw;
+
+	spin_lock_irqsave(&brick->cb_lock, flags);
+	if (rw)
+		list_add_tail(&aio_a->cb_head, &brick->cb_write_list);
+	else
+		list_add_tail(&aio_a->cb_head, &brick->cb_read_list);
+	spin_unlock_irqrestore(&brick->cb_lock, flags);
+
+	wake_up_interruptible(&brick->cb_event);
+	goto out_return;
+err:
+	XIO_FAT("cannot handle callback - giving up\n");
+out_return:;
+}
+
+int server_io(struct server_brick *brick, struct xio_socket *sock, struct xio_cmd *cmd)
+{
+	struct aio_object *aio;
+	struct server_aio_aspect *aio_a;
+	int amount;
+	int status = -ENOTRECOVERABLE;
+
+	if (!brick->cb_running || !brick->handler_running || !xio_socket_is_alive(sock))
+		goto done;
+
+	aio = server_alloc_aio(brick);
+	status = -ENOMEM;
+	aio_a = server_aio_get_aspect(brick, aio);
+	if (unlikely(!aio_a)) {
+		obj_free(aio);
+		goto done;
+	}
+
+	status = xio_recv_aio(sock, aio, cmd);
+	if (status < 0) {
+		obj_free(aio);
+		goto done;
+	}
+
+	aio_a->brick = brick;
+	aio_a->data = aio->io_data;
+	aio_a->len = aio->io_len;
+	SETUP_CALLBACK(aio, server_endio, aio_a);
+
+	amount = 0;
+	if (!aio->io_cs_mode < 2)
+		amount = (aio->io_len - 1) / 1024 + 1;
+	rate_limit_sleep(&server_limiter, amount);
+
+	status = GENERIC_INPUT_CALL(brick->inputs[0], aio_get, aio);
+	if (unlikely(status < 0)) {
+		XIO_WRN("aio_get execution error = %d\n", status);
+		SIMPLE_CALLBACK(aio, status);
+		status = 0; /*	continue serving requests */
+		goto done;
+	}
+	aio_a->do_put = true;
+	atomic_inc(&brick->in_flight);
+	GENERIC_INPUT_CALL(brick->inputs[0], aio_io, aio);
+
+done:
+	return status;
+}
+
+/***************** own brick * input * output operations *****************/
+
+static int server_get_info(struct server_output *output, struct xio_info *info)
+{
+	struct server_input *input = output->brick->inputs[0];
+
+	return GENERIC_INPUT_CALL(input, xio_get_info, info);
+}
+
+static int server_io_get(struct server_output *output, struct aio_object *aio)
+{
+	struct server_input *input = output->brick->inputs[0];
+
+	return GENERIC_INPUT_CALL(input, aio_get, aio);
+}
+
+static void server_io_put(struct server_output *output, struct aio_object *aio)
+{
+	struct server_input *input = output->brick->inputs[0];
+
+	GENERIC_INPUT_CALL(input, aio_put, aio);
+}
+
+static void server_io_io(struct server_output *output, struct aio_object *aio)
+{
+	struct server_input *input = output->brick->inputs[0];
+
+	GENERIC_INPUT_CALL(input, aio_io, aio);
+}
+
+int server_switch(struct server_brick *brick)
+{
+	struct xio_socket *sock = &brick->handler_socket;
+	int status = 0;
+
+	if (brick->power.button) {
+		static int version;
+		bool ok;
+
+		if (brick->power.on_led)
+			goto done;
+
+		ok = xio_get_socket(sock);
+		if (unlikely(!ok)) {
+			status = -ENOENT;
+			goto err;
+		}
+
+		xio_set_power_off_led((void *)brick, false);
+
+		brick->version = version++;
+		brick->handler_thread = brick_thread_create(handler_thread, brick, "xio_handler%d", brick->version);
+		if (unlikely(!brick->handler_thread)) {
+			XIO_ERR("cannot create handler thread\n");
+			status = -ENOENT;
+			goto err;
+		}
+
+		xio_set_power_on_led((void *)brick, true);
+	} else if (!brick->power.off_led) {
+		struct task_struct *thread;
+
+		xio_set_power_on_led((void *)brick, false);
+
+		xio_shutdown_socket(sock);
+
+		thread = brick->handler_thread;
+		if (thread) {
+			brick->handler_thread = NULL;
+			brick->handler_running = false;
+			XIO_DBG("#%d stopping handler thread....\n", sock->s_debug_nr);
+			brick_thread_stop(thread);
+		}
+
+		xio_put_socket(sock);
+		XIO_DBG("#%d socket s_count = %d\n", sock->s_debug_nr, atomic_read(&sock->s_count));
+
+		xio_set_power_off_led((void *)brick, true);
+	}
+err:
+	if (unlikely(status < 0)) {
+		xio_set_power_off_led((void *)brick, true);
+		xio_shutdown_socket(sock);
+		xio_put_socket(sock);
+	}
+done:
+	return status;
+}
+
+/*************** informational * statistics **************/
+
+static
+char *server_statistics(struct server_brick *brick, int verbose)
+{
+	char *res = brick_string_alloc(1024);
+
+	snprintf(res, 1024,
+		 "cb_running = %d handler_running = %d in_flight = %d\n",
+		 brick->cb_running,
+		 brick->handler_running,
+		 atomic_read(&brick->in_flight));
+
+	return res;
+}
+
+static
+void server_reset_statistics(struct server_brick *brick)
+{
+}
+
+/*************** object * aspect constructors * destructors **************/
+
+static int server_aio_aspect_init_fn(struct generic_aspect *_ini)
+{
+	struct server_aio_aspect *ini = (void *)_ini;
+
+	INIT_LIST_HEAD(&ini->cb_head);
+	return 0;
+}
+
+static void server_aio_aspect_exit_fn(struct generic_aspect *_ini)
+{
+	struct server_aio_aspect *ini = (void *)_ini;
+
+	CHECK_HEAD_EMPTY(&ini->cb_head);
+}
+
+XIO_MAKE_STATICS(server);
+
+/********************* brick constructors * destructors *******************/
+
+static int server_brick_construct(struct server_brick *brick)
+{
+	init_waitqueue_head(&brick->startup_event);
+	init_waitqueue_head(&brick->cb_event);
+	sema_init(&brick->socket_sem, 1);
+	spin_lock_init(&brick->cb_lock);
+	INIT_LIST_HEAD(&brick->cb_read_list);
+	INIT_LIST_HEAD(&brick->cb_write_list);
+	return 0;
+}
+
+static int server_brick_destruct(struct server_brick *brick)
+{
+	CHECK_HEAD_EMPTY(&brick->cb_read_list);
+	CHECK_HEAD_EMPTY(&brick->cb_write_list);
+	return 0;
+}
+
+static int server_output_construct(struct server_output *output)
+{
+	return 0;
+}
+
+/************************ static structs ***********************/
+
+static struct server_brick_ops server_brick_ops = {
+	.brick_switch = server_switch,
+	.brick_statistics = server_statistics,
+	.reset_statistics = server_reset_statistics,
+};
+
+static struct server_output_ops server_output_ops = {
+	.xio_get_info = server_get_info,
+	.aio_get = server_io_get,
+	.aio_put = server_io_put,
+	.aio_io = server_io_io,
+};
+
+const struct server_input_type server_input_type = {
+	.type_name = "server_input",
+	.input_size = sizeof(struct server_input),
+};
+
+static const struct server_input_type *server_input_types[] = {
+	&server_input_type,
+};
+
+const struct server_output_type server_output_type = {
+	.type_name = "server_output",
+	.output_size = sizeof(struct server_output),
+	.master_ops = &server_output_ops,
+	.output_construct = &server_output_construct,
+};
+
+static const struct server_output_type *server_output_types[] = {
+	&server_output_type,
+};
+
+const struct server_brick_type server_brick_type = {
+	.type_name = "server_brick",
+	.brick_size = sizeof(struct server_brick),
+	.max_inputs = 1,
+	.max_outputs = 0,
+	.master_ops = &server_brick_ops,
+	.aspect_types = server_aspect_types,
+	.default_input_types = server_input_types,
+	.default_output_types = server_output_types,
+	.brick_construct = &server_brick_construct,
+	.brick_destruct = &server_brick_destruct,
+};
+
+/*********************************************************************/
+
+/*  strategy layer */
+
+int server_show_statist;
+
+/***************** module init stuff ************************/
+
+struct rate_limiter server_limiter = {
+	.lim_max_rate = 0,
+};
+
+void exit_xio_server(void)
+{
+	int i;
+
+	XIO_INF("exit_server()\n");
+	server_unregister_brick_type();
+
+	for (i = 0; i < NR_SERVER_SOCKETS; i++) {
+		if (server_threads[i]) {
+			XIO_INF("stopping server thread %d...\n", i);
+			brick_thread_stop(server_threads[i]);
+		}
+		XIO_INF("closing server socket %d...\n", i);
+		xio_put_socket(&server_socket[i]);
+	}
+}
+
+int __init init_xio_server(void)
+{
+	int i;
+
+	XIO_INF("init_server()\n");
+
+	for (i = 0; i < NR_SERVER_SOCKETS; i++) {
+		struct sockaddr_storage sockaddr = {};
+		char tmp[64];
+		int status;
+
+		if (xio_translate_hostname)
+			snprintf(tmp, sizeof(tmp), "%s:%d", my_id(), xio_net_default_port + i);
+		else
+			snprintf(tmp, sizeof(tmp), ":%d", xio_net_default_port + i);
+
+		status = xio_create_sockaddr(&sockaddr, tmp);
+		if (unlikely(status < 0)) {
+			exit_xio_server();
+			return status;
+		}
+
+		status = xio_create_socket(&server_socket[i], &sockaddr, NULL);
+		if (unlikely(status < 0)) {
+			XIO_ERR("could not create server socket %d, status = %d\n", i, status);
+			exit_xio_server();
+			return status;
+		}
+
+		server_threads[i] = brick_thread_create(server_thread, &server_socket[i], "xio_server_%d", i);
+		if (unlikely(!server_threads[i] || IS_ERR(server_threads[i]))) {
+			XIO_ERR("could not create server thread %d\n", i);
+			exit_xio_server();
+			return -ENOENT;
+		}
+	}
+
+	return server_register_brick_type();
+}
include/linux/xio/xio_server.h new file +91 -0

91 lines not on this page · raw message

Signature · 1 line
2.6.4