Bus Resource Management

Hi all,

I would appreciate any comments.

Thanks,
Adam

Linux Bus Resource Managment
by Adam Belay <[email protected]>
last updated: August 19, 2004
-------------------------------------------------------------------------------

context:
This document describes my Linux Resource Management Rewrite. Although It is written with the specific challanges of the x86 architecture in mind, most material will apply to every architecture.

Background Information
======================
Bus resources are channels used by drivers to communicate with physical hardware. In most modern systems, it is possible to dynamically assign and reassign resources to devices. This process is known as "Resource Management".

There are three types of resources common to most architectures:

1.) I/O ports 2.) I/O memory 3.) interrupts

In addition, x86, arm and others have DMA channels.

A device that is decoding its resources is said to be "enabled". Conversely, a device that ignores resources is said to be "disabled". Just like devices, resources have parents and children. They can be nested and form a tree-like topology. A resource's parent is known as a "producer". A resource's child is a "consumer". Every resource belongs to a physical device. A device may decode several resources.

There have been many attempts to improve resource management for the x86 architecture. As a result there are several firmware interfaces designed to support this feature. In addition, most buses implement their own resource management capabilities.

Resource management interfaces include:

legacy:
isapnp, pnpbios, eisa, mca, pcmcia (pcmcia is still widely used)

modern:
pci, pccard, pci-X, pci express, acpi, open firmware, others?

Older technologies, such as ISA are not capable of supporting resource management, and require special measures to operate correctly.

The Problem
===========

The Resource Management Challange

Although allowing the operating system to configure resources is a significant advantage, it also introduces complex problems. Every device has different resource requirements. The method of determining these requirements is bus specific. Some devices are configured by the BIOS before the operating system is loaded. Others must always be configured by software. Still, others may be configured by the BIOS if cold plugged, but require operating system configuration if hot plugged. To make matters worse, both resource producers, such as pci bridges, and resource consumers, such as network adapters, could potentially be hot plugged.

In a perfect world there would be a virtually unlimited amount of resources available. Every device would be granted its requirements, without the risk of an allocation failure. Unfortunantly, many (not all) systems have very limited resource availability. Although this is not a problem for the operating system when the BIOS preconfigures devices, any hot pluggable device requires configuration by the kernel. Also, in some systems, the BIOS will only allocate resources to devices required for boot. Finally, resource configuration must be restored after a suspend to ram.

Legacy compatibility makes the situation even more complex. For example, many ISA devices will only decode the first 10 bits of a 16 bit I/O address. As a result, ghosts will appear at each address with a matching first 10 bits. This very quickly consumes available I/O space, and creates a greater potential for resource conflicts. Even modern systems still exhibit this problem with vga controllers. Another example would be that many interrupts are unavailable because the are reserved for legacy hardware. Interrupt 13 is saved for a math coprocessor, whether or not it physically is wired to hardware.

Although new specifications generally have improved the resource management situation, they also present new challenges. For example, pci express will allow for multipe bridges even on small desktop boxes. Also, as hotplugable devices become more prevalent, the need for dynamic resource allocation and even "rebalancing" (the moving of a device to a different resource vector) increases.

Linux Specific Problems

The Linux Kernel currently has an API for managing some types of resources. Resources are generally reserved by drivers at the time they are required. PCI is an exception in that it requests I/O and Memory before the driver is loaded. Resources are assigned at the time of detection in some cases, and during driver probing in others. Every bus driver has its own method of assigning resources, and these implementations vary greatly between buses. Furthermore, most buses are unaware of the resources assigned by other buses if they have not been requested by the existing resource API at the time of allocation. Assignment is not protected by semaphores in all cases, and there are several race conditions.

The kernel is unaware of resource based device dependencies. The net effect is that that the kernel cannot always determine who is producing a resource, and lifetime rules for devices are often incorrect.

Despite its many flaws, the existing implementation is able to handle common cases quite well. As a result, code repetition, limited user interface, and lack of driver model integration are equally important problems.

Proposed Solution
=================

In order to improve the functionality of Linux Resource Management, I propose the following changes:

1.) Driver Model integration of resource management
- resource information is stored in "struct device".
- "struct bus_type" has hooks for common resource management operations.
- device dependency and producer/consumer relationships are considered.
- an API is exported that will allow for any device to be enabled or disabled in a uniform way.
- resources utilize kobject.

2.) Sysfs user interface
- userspace applications can determine resource usage and dependencies.
- userspace applications can disable and enable devices
- userspace applications can assign resources to a device
- userspace applications can pause the operation of a device, and rebalance its resources

3.) Smarter Allocation
- a central API controls allocation
- functions are available to determine special case conflicts such as 10 bit decode
- all devices are detected before allocation begins
- special provisions are taken for legacy hardware

4.) Support for all resource types
- currently only I/O and Mem are available for pre-driver reservations
- add support for interrupts and dma
- also pci bus numbers could be a resource type

5.) Report resource availability acurately
- acpi enumerates devices in the driver model tree and reports resource usage
- all other buses also utilize the new resource management API
- allow for buses to specify their own rules for determining if a resource is vaild

It is likely that even more needs to be changed. (open for discussion)

The Implementation
==================

The new implementation attempts to maintain compatibility with the existing API. I encourage everyone to look at the proof-of-concept code attached at the end of this email. The majority of it compiles, but it still needs to be debugged. I'm sharing it at this alpha stage because I think it could benefit from comments and suggestions.

Here is a _very_ brief summary...

::: Core resource management

This is the foundation of the resource management API

struct resource {
	struct kobject kobj;
	const char *name;
	unsigned long start;
	unsigned long stop;
	unsigned long flags;
	struct resource *parent;
	struct resource *child;
	struct resource *sibling;
	struct res_type *type;
	struct device *dev;
};

int resource_register(struct resource *res)
- reserves the resource and shows its new status in sysfs

int resource_unregister(struct resource *res)
- frees the resource

::: Resource types

A resource type is defined for each type of resource (ex. io, mem, irq etc)

struct res_type {
const char *name;

struct subsystem        subsys;
struct rw_semaphore     rwsem;
struct resource         *root;
int                     ranged;

};

int res_type_register(struct res_type *type)
- registers a resource type and inits it for use

void res_type_unregister(struct res_type *type)
- unregisters a resource type

::: Other functions

int resource_assign(struct resource *res, int ordered, unsigned long *next)
- attempts to allocate a resource at the location specified in res->start
- if it fails, next points to the next available location

int res_assign_range(struct resource *res, unsigned long min, unsigned long

		     max, unsigned long length, unsigned long align);

int device_enable(struct device *dev)
int device_disable(struct device *dev)

also the following has been added to struct bus_type:

(*enable)(struct device *dev)
(*disable)(struct device *dev)
(*assign)(struct device *dev)
(*verify)(struct resource *res)

More information is available in the code/comments.

Conclusion
==========

In conclusion, resource management can be difficult, but I believe Linux, with improvement, could handle it well. Because this issue is extremely complex, I think it is important that we publicly discuss it, and work out a final API. I hope that the code I have contributed will be useful, but I'm open to making any changes necessary for it to be accepted into the kernel. Implementing this properly will be a long and difficult process, and I'd like to get as many people involved as possible so we can potentially speed things up.

Cheers,
Adam

diff -urN a/drivers/base/Makefile b/drivers/base/Makefile
@@ -3,7 +3,7 @@
 obj-y			:= core.o sys.o interface.o bus.o \
 			   driver.o class.o class_simple.o platform.o \
 			   cpu.o firmware.o init.o map.o dmapool.o
-obj-y			+= power/
+obj-y			+= power/ resource/
 obj-$(CONFIG_FW_LOADER)	+= firmware_class.o
 obj-$(CONFIG_NUMA)	+= node.o
 

diff -urN a/drivers/base/core.c b/drivers/base/core.c
@@ -20,6 +20,7 @@
 
 #include "base.h"
 #include "power/power.h"
+#include "resource/resource.h"
 
 int (*platform_notify)(struct device * dev) = NULL;
 int (*platform_notify_remove)(struct device * dev) = NULL;

@@ -229,6 +230,8 @@
 		goto Error;
 	if ((error = device_pm_add(dev)))
 		goto PMError;
+	if ((error = device_res_add(dev)))
+		goto RESError;
 	if ((error = bus_add_device(dev)))
 		goto BusError;
 	down_write(&devices_subsys.rwsem);

@@ -244,6 +247,8 @@
 	return error;
  BusError:
 	device_pm_remove(dev);
+ RESError:
+	device_res_remove(dev);
  PMError:
 	kobject_del(&dev->kobj);
  Error:

diff -urN a/drivers/base/init.c b/drivers/base/init.c
@@ -13,6 +13,7 @@
 extern int devices_init(void);
 extern int buses_init(void);
 extern int classes_init(void);
+extern int resource_init(void);
 extern int firmware_init(void);
 extern int platform_bus_init(void);
 extern int system_bus_init(void);

@@ -31,6 +32,7 @@
 	devices_init();
 	buses_init();
 	classes_init();
+	resource_init();
 	firmware_init();
 
 	/* These are also core pieces, but must come after the

diff -urN a/drivers/base/resource/Makefile b/drivers/base/resource/Makefile
@@ -0,0 +1,11 @@
+#
+# Makefile for Linux Resource Management.
+#
+
+res-proc-$(CONFIG_PROC_FS) = proc.o
+
+obj-y := main.o resource.o types.o assign.o sysfs.o slot.o device.o $(res-proc-y) compat.o
+
+ifeq ($(CONFIG_DEBUG_DRIVER),y)
+EXTRA_CFLAGS += -DDEBUG
+endif

diff -urN a/drivers/base/resource/main.c b/drivers/base/resource/main.c
@@ -0,0 +1,32 @@
+/*
+ * drivers/base/resource/main.c - links the driver model to resource management
+ *
+ * Copyright (c) 2004 Adam Belay
+ *
+ * This file is released under the GPLv2
+ *
+ * based on drivers/base/power/main.c (c) 2003 Patrick Mochel
+ */
+
+#include <linux/config.h>
+#include <linux/device.h>
+#include <linux/ioport.h>
+#include "resource.h"
+
+
+int device_res_add(struct device * dev)
+{
+	int error;
+
+	pr_debug("RES: Adding info for %s:%s\n",
+		 dev->bus ? dev->bus->name : "No Bus", dev->kobj.name);
+	error = res_sysfs_add(dev);
+	return error;
+}
+
+void device_res_remove(struct device * dev)
+{
+	pr_debug("RES: Removing info for %s:%s\n",
+		 dev->bus ? dev->bus->name : "No Bus", dev->kobj.name);
+	res_sysfs_remove(dev);
+}

diff -urN a/drivers/base/resource/resource.h b/drivers/base/resource/resource.h
@@ -0,0 +1,34 @@
+/*
+ * resource.h - internal resource management header
+ *
+ *
+ * This file is released under the GPLv2
+ *
+ * Copyright (c) 2004	Adam Belay
+ *
+ */
+
+#include <linux/config.h>
+
+
+struct resource_attribute {
+	struct attribute	attr;
+	ssize_t (*show)(struct resource *res, char *buf);
+	ssize_t (*store)(struct resource *res, const char *buf, size_t count);
+};
+
+extern int __resource_register(struct resource *res, struct resource *from,
+			       struct resource **conflict);
+extern int __resource_unregister(struct resource *res);
+
+#ifdef CONFIG_PROC_FS
+extern int res_proc_init(void);
+#else
+static inline int res_proc_init(void) { return 0; }
+#endif /* CONFIG_PROC_FS */
+
+extern int res_sysfs_add(struct device *dev);
+extern void res_sysfs_remove(struct device *dev);
+
+extern int device_res_add(struct device *);
+extern void device_res_remove(struct device *);

diff -urN a/include/linux/device.h b/include/linux/device.h
@@ -41,6 +41,11 @@
 	RESUME_ENABLE,
 };
 
+enum {
+	DEVICE_DISABLED,
+	DEVICE_ENABLED,
+};
+
 struct device;
 struct device_driver;
 struct class;

@@ -62,6 +67,12 @@
 	struct device * (*add)	(struct device * parent, char * bus_id);
 	int		(*hotplug) (struct device *dev, char **envp,
 				    int num_envp, char *buffer, int buffer_size);
+
+	int		(*assign)(struct device *dev);
+	int		(*verify)(struct resource *res, unsigned long *next);
+	int		(*enable)(struct device *dev);
+	int		(*disable)(struct device *dev);
+
 	int		(*suspend)(struct device * dev, u32 state);
 	int		(*resume)(struct device * dev);
 };

@@ -276,6 +287,9 @@
 	u32		detach_state;	/* State to enter when device is
 					   detached from its driver. */
 
+	int		state;		/* resource management state */
+	int		num_resources;
+	struct res_slot	* resource;
 	u64		*dma_mask;	/* dma mask (if dma'able device) */
 	u64		coherent_dma_mask;/* Like dma_mask, but for
 					     alloc_coherent mappings as

I agree that run-time hotplug policy and re-balancing would all be very snappy from user-space - users should be in charge when setting policy. Heck, humans may even be needed to make some decisions regarding resource conflicts...

But I'm wondering where the proper line is between that and the resource management that we have to do on boot before user-space exists.

cheers,
-Len

For the minimum, I'm leaning toward in-kernel configuration of devices needed for boot and their parents. I would like to see features such as rebalancing performed from userspace. However, if there is a specific reason for this to be in the kernel, I'd be happy to implement it. Right now I'm just trying to establish an infustructure that we can adapt to our needs.

Also there is the option of using initrd and further limiting the number of devices that have to be configured before userspace is loaded.

When and how devices and their drivers should be configure for resources and other settings is really a matter that needs to be discussed. Do we want more userspace involvement or less userspace involvement? Also should the kernel be able to operate independently of userspace in every situation? One advantage I see with userspace involement is that it would allow for a cache of previous configuration settings. This would make it possible to ensure that all devices are configured consistently between boots. It could apply not only to resource settings, but also to driver settings currently handled by module params. We could even control the binding of drivers to devices.

Thanks,
Adam