64-bit kdev_t - just for playing

Hi Joel,

I have been playing with supporting 4000 disks on IA32 machines. There are bunch of issues we need to resolve before we could do that.

I am using scsi_debug to simulate 4000 disks. (Ofcourse, I had to hack "sd" to support more than 256 disks). Anyway, I noticed that I lost almost 350MB of my lowmem, when I simulated 4000 disks. We are working on most of these. But there are userlevel issues to be resolved. Here is the list ...

1) deadline_drq, blkdev_request consume 80 MB of low memory.
- Jens is looking at it. He is working on a patch to allocate
requests dynamically.

2) sysfs inode use up 50 MB of low memory
- 4000 disks without partitions create (4000 * 35) = 140,000 inodes in
/sysfs. So, it uses 50 MB of lowmem.

3) dcache is eating up 25 MB of low memory.

4) kmalloc() slabs are consuming 55 MB. We are in the process
of identifying the heavy consumers and fixing them.
- Jens is fixing hash table size issues with io-schedulers.
- I have patch to allocate "hd_struct" dynamically. So, if your disks does not have any partitions, you don't use any more memory.

5) glibc and utility issues - lots of stuff are broken (Need a new libc)
- mknod
- ls
- raw command
- etc..

I have not done any IO on these yet. When I mount all of these and do IO on them, we might see new issues. So with all these, I will be doubtful if we can ever reach 16k disks on IA32.

Thanks,
Badari

These are actually the same thing (the inode pinning references are actually held by the dentries IIRC). shaggy and I are brewing up something to show to gregkh and mochel in the near future.

-- wli

That's the conversation I'm trying to kickstart.

Wow, this is cool. Thanks for telling me about this.

We're going to have to find a way. IA32 is going to be around
for long enough, I think. Easily 8k disks, as soon as the folks who are doing 4k disks today want to multipath.

Joel

Life's Little Instruction Book #226

	"When someone hugs you, let them be the first to let go."

Joel Becker
Senior Member of Technical Staff
Oracle Corporation
E-mail: [email protected]
Phone: (650) 506-8127

Hi,

Could you please post your changes to sd.c and anything relevant to it? Thanks.

bye, Roman

Roman,

Here is the patch for sd to allow more than 256 disks. There are few issues with the patch that need to be resolved.

1) With the patch I get 16 bits for minor. Since 4 bits are used for partition, we get 12 bits to represent disks. So each major can support 2^12 = 4096 disks. Disks 0 - 4095 are mapped to major=8, disks 4096 - 8191 to major = 65 and so on..

This means ..

(i) I need to create nodes in /dev/ to match new <major, minor> for these disks. Currently "mknod" is broken due to glibc issues with dev_t.

(ii) We need to worry about backward compatibility. For example:
17th disk used to have <65, 0>. Now its major, minor is <8, 256>. So /dev/ entires need to be re-created to match these, everytime you reboot 2.4/2.5 etc. Greg KH udev might fix this for us.

2) Do we still need 16 majors for disks ?

We could change my patch to work around major/minor assignment issues and maintain backward compatibility.

Thanks,
Badari

@@ -56,7 +56,9 @@
  * Remaining dev_t-handling stuff
  */
 #define SD_MAJORS	16
-#define SD_DISKS	(SD_MAJORS << 4)
+#define SD_DISKS_PER_MAJOR_SHIFT	(KDEV_MINOR_BITS - 4)
+#define SD_DISKS_PER_MAJOR	(1 << SD_DISKS_PER_MAJOR_SHIFT)
+#define SD_DISKS	(SD_MAJORS << SD_DISKS_PER_MAJOR_SHIFT)
 
 /*
  * Time out in seconds for disks and Magneto-opticals (which are slower).

@@ -1328,17 +1330,23 @@ static int sd_attach(struct scsi_device 
 	sdkp->index = index;
 
 	gd->de = sdp->de;
-	gd->major = sd_major(index >> 4);
-	gd->first_minor = (index & 15) << 4;
+	gd->major = sd_major(index >> SD_DISKS_PER_MAJOR_SHIFT);
+	gd->first_minor = (index & (SD_DISKS_PER_MAJOR - 1)) << 4;
 	gd->minors = 16;
 	gd->fops = &sd_fops;
 
-	if (index >= 26) {
+	if (index < 26) {
+		sprintf(gd->disk_name, "sd%c", 'a' + index % 26);
+	} else if (index < (26*27)) {
 		sprintf(gd->disk_name, "sd%c%c",
 			'a' + index/26-1,'a' + index % 26);
 	} else {
-		sprintf(gd->disk_name, "sd%c", 'a' + index % 26);
-	}
+		const unsigned int m1 = (index/ 26 - 1) / 26 - 1;
+		const unsigned int m2 = (index / 26 - 1) % 26;
+		const unsigned int m3 = index % 26;
+		sprintf(gd->disk_name, "sd%c%c%c", 
+			'a' + m1, 'a' + m2, 'a' + m3);
+	}	
 
 	sd_init_onedisk(sdkp, gd);
 

Followup to:  <[email protected]>
By author:    Badari Pulavarty <[email protected]>
In newsgroup: linux.dev.kernel

No, we don't. On the other hand, we really should change to 64 partitions/disk, same as for non-SCSI disks. 16 really is too small.

-hpa

<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64

when you think of huge raid arrays that present themselves to the system as a single drive even 64 partitions can be limiting.

I don't really know what an appropriate limit would be, this may be something that has a default setting for static dev entries and they dynamic assignments can get more

David Lang

That's what logical volume management is for. Problem solved.

Sincerely,
Lars Marowsky-Brée <[email protected]>

SuSE Labs - Research & Development, SuSE Linux AG
  
"If anything can go wrong, it will." "Chance favors the prepared (mind)."
  -- Capt. Edward A. Murphy            -- Louis Pasteur

Hi,

I expected something like this. As soon as we want to maintain compatibility it can get quite ugly.
I would prefer to leave the dev_t range below 0x10000 unchanged for 2.6 (this means in the kernel the minor bits stay at 8) and new dev_t entries a created dynamically. Work which is currently still done in sd.c (e.g. numbering/naming) can be incrementally moved to the generic code and during 2.7 it can be dropped from sd.c. This way other drivers can automatically benefit from this and don't have to reproduce the hacks done in sd.c. Also if someone wants more partitions it would be a simple switch in sd.c and all dev_t numbers are allocated dynamically.

I am really interested why nobody wants to do now that small deciding step to use dynamic dev_t numbers. Everyone agrees that we want to use dynamic numbers, but almost everyone wants to hold on to static dev_t numbers. What will be the next step after enlarging dev_t? Playing around with the major/minor split will not help in the long term and only creates new problems. On the other hand at this point it's trivial now to maintain compability for old device numbers and create and use dynamic numbers.

bye, Roman

Because we first want to have the numbers before using them?

Hi,

Yes, I know this mantra now and that's not the problem (or will be fixed shortly).
This still doesn't answer what will come next. You must have _some_ idea, otherwise you wouldn't add a new interface, remove other infrastructure and provide a patch which modifies MKDEV & co. All of this only leads us away from the goal of dynamic device numbers. Why?

bye, Roman

Followup to:  <Pine.LNX.4.44.0304031548090.12110-100000@serv>
By author:    Roman Zippel <[email protected]>
In newsgroup: linux.dev.kernel

I have an idea, why don't you read the archives of this mailing list for the past eight years and learn, once again, why dynamic numbers are broken for nearly all applications (disks and ptys being, perhaps, the few case where they actually work.)

This has been hashed and rehashed on this list so many times it's not even funny.

-hpa

<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64

Hi,

I ignore the flame bait in the hopes to get a reasonable answer this time, if you have a personal problem with me, you are free to tell me this directly, but flaming will help nobody.

The dynamic number discussion I remember were not only about numbering but also about naming. A specific device had to be addressable with a specific number. We currently have a fixed mapping between numbers and names (as documented in devices.txt). Disks are an area where this doesn't work, the only exception is IDE, because most machines have a builtin controller, so it's easy to map to a fixed number. SCSI already has to use dynamic numbers, because it wouldn't otherwise fit into a 16bit dev_t and with SAN even 64bit will not be enough. This makes SCSI very annoying, when at the next boot everything has a new name, because a disk was added/removed.

Consequently it's impossible that the kernel guarantees a static name/ number for a specific device. devfs showed that the kernel is the wrong place for name policies. Is see no other possibility than to number devices dynamically and leave it to userspace to name them or did I miss an important point in previous discussions?
Maybe you could also answer me, what exactly you need a 64bit dev_t for? If anything of this has been discussed previously, I'd also be happy about any URLs. Thanks.

bye, Roman

Hi,

Ok, Peter refuses to give me an answer to that, this either means I'm unworthy an answer or he doesn't know it either.
Is there really nobody who can answer me these questions?

bye, Roman

Well, I'm not Peter, or Andries or anyone else for that matter... but if a userspace tool can be built that will enforce the user's (or distro's) desired naming policies for completely dynamic numbered devices, I'm all for that. It sure seems that it would greatly simplify much code in the kernel and completely eliminate the need to move beyond 32-bit dev_t at all. (I mean really, who's ever going to have more than 4G devices on a single Linux system image?)

However, there are significant hurdles to implementing this tool:

- defining an adequately powerful file format to define the desired naming policy
- getting the tool to run _both_ in early-userspace and real-userspace environments, using the _same_ policy file (or copies thereof)

Hi,

The kernel only exports information, which you can get from sysfs. This will unlikely become a requirement for 2.6, so how this done can still change a bit.

Early-userspace is not that much of a problem, as it only has to map the device information from sysfs to a device name. The required configuration file can be quite simple, every device has a list of properties, which closely resembles sysfs (one property == one value).

bye, Roman

[...]

[...]

That was a quick conclusion, after less than two hours :-)

Anyway, I agree with your general concern. It only seems good engineering practice to also look at the numbering schemes that are supposed to go with the device number enlargement.

Or, alternatively, to make sure that it's trivial to make further enlargements (or shrinkages), if the need for them should arise. I didn't look at the issue in detail, but perhaps the latter is the case ?

- Werner

  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/

the biggest problem I see with dynamic numbers is that it needs a userspace devfs type solution for creating and maintaining the device nodes that are then used. While this isn't rocket science it's also somthing that is hard to get people to agree to (remember the devfs names that everyone gripes about are not what richard started with it's what he switched to to get things into the kernel, they changed many times during that process)

I don't think many people will argue that dynamic assignments are evil, but I think you will find a lot of people very nervous about switching to them and the risk involved with doing so.

David Lang

Agreed, this is the difficult part. This probably means that the kernel will have to come with a default initrd-like setup that is built and attached by "make bzImage" and the like. I thought that people were quite actively working towards something like this ?

Actually, here we may have an advantage now. The "old way" was to write the driver, allocate a number, release the driver, and suggest a device name. If somebody didn't like the name, they could easily change it to something else. Nobody had to agree with the author.

devfs was different. The one thing everybody has to agree on is the kernel. And gratuitous differences in "vendor kernels" are frowned upon, and are no less of a pain in individual kernels either. So now everyone had to accept the name the author assigned - or fight with the author until the name changed.

(Of course, one could have mounted devfs in some dark corner and set up a link tree pointing to it, which wouldn't have been too different from using sysfs. devfs was never really positioned to take such a role, and seems to carry too much overhead trying to be a "full" /dev replacement to be justifiable as an "almost" /dev replacement.)

By completely removing such policy from the kernel, we return to the status quo ante: user-visible names only need to be agreed on during early system bringup. After that, the "device file manager" takes over, and that one can just use a local device name database.

- Werner

  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/

Yes, this is being worked on actively. This will use the 2.5.x initramfs infrastructure, and when it's up and going there will be early-userspace tools included in the kernel tarball to do the basic things that need doing (essentially responding to hotplug events and creating/removing devices nodes as needed).

Yes, this is true. Ideally, the early-userspace version and the normal-userspace version will be the same tool (possibly compiled differently, but otherwise identical). The early-userspace version should not have to create many nodes for a "normal" kernel boot, so most of the remaining ones can be created once the user's preferred naming policy can be located and used. Really, the early-userspace version only needs to create two nodes that I can think of to satisfy the "normal" kernel boot process: /dev/console and /dev/root. However, both of these need to be created with knowledge of parameters passed on the kernel command line (as the kernel does now), so that when the hotplug process sees the appropriate device "appear" it can create the proper device node.

Followup to:  <[email protected]>
By author:    David Lang <[email protected]>
In newsgroup: linux.dev.kernel

They aren't evil, they're just very, very hard to get right.

So far, *none* of the schemes used for dynamics have gotten it right. They just ignore a fair number of the problems. People keep focusing on disks, and they are nearly uniformly the almost-trivial case in comparison with especially character devices, where you don't have the layer of indirection called /etc/fstab, persistent labels, etc.

It is also independent of the need to switch to a larger dev_t. Claiming that we can squeeze more out of the existing device scheme if we have an ideal-world dynamic scheme is unrealistic because:

a) There are, genuinely, systems with more than 65,536 devices or anonymous mounts. That rules out the current dev_t just by itself.

b) Despite the fact that people have tried since the mid-90's, we still don't have a sane way to manage such dynamicity.

c) We are now in what pretty much amounts to a crisis situation. We have needed to enlarge dev_t for well over half a decade. Therefore, it is too late to say "well, given X we wouldn't need it." We need something done in *this* kernel cycle.

Given that it has taken, literally, 8 years to get to this point, and based on collective global experience with numberspaces, I'm arguing for enlarging it far more than anyone can currently imagine being necessary.

dev_t is already 64 bits in glibc, and the glibc<->kernel interface needs to be fixed *anyway*. We have to take the pain of migration, we might as well go all the way.

-hpa

<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64

If I understand the words of Andries and Linus correctly, that is the plan.

Hi,

http://marc.theaimsgroup.com/?l=linux-kernel&m=95547434315472&w=2

Absolutely nobody denies that we need a larger dev_t. It's really a poor argumentation that you have to come up with this.

Maybe you didn't notice, that only now the block device layer is clean enough to go dynamic. Maybe you didn't notice that scsi devices are already dynamically numbered and that there are already user space tools to translate them to constant device names.

And since more than 8 years Linus is trying to tell you that static device numbers don't work. When do you start to listen?

bye, Roman

Roman,
There are a couple things being discussed here. One is the size
of dev_t. The other is dynamic numbers. It would seem that most folks agree with a larger dev_t and a more dynamic numbering system. Let's assume we want both for now (folks who don't, please keep out for a second). There are three courses of action that seem to be advocated.

1) Ship 2.6 with 16bit dev_t, work on a larger dev_t and perfect dynamic devices in 2.7.
2) Ship 2.6 with a (32|64)bit dev_t, work on a perfect dynamic scheme in 2.7.
3) Hold 2.6 until it can ship with (32|64)bit dev_t and perfect dynamic devices.

Many folks, Peter and myself included, are claiming that choice
(1) is absolutely untenable. We need more device space today, not in 3 years when 2.7 becomes 2.8.
If I understand you correctly (and here is why I mailed), you
feel that choice (2) is the worst of the choices. You feel that we should either choose course (1) or course (3). I'm not sure which of those you prefer.

Joel

Life's Little Instruction Book #335

	"Every so often, push your luck."

Joel Becker
Senior Member of Technical Staff
Oracle Corporation
E-mail: [email protected]
Phone: (650) 506-8127

Hi,

That misunderstanding is hopefully easy to resolve:

(4) Ship 2.6 with a (32|64)bit dev_t with an experimental dynamic scheme and keep the device numbers below 0x10000 as they are now.

bye, Roman

Hi,

Let's try it with a real example:
I have two onboard SCSI channels, the first one is for external devices and the second for internal devices. This means the internal drives are renumbered depending on whether I have my MO-drive turned on or off. Now I also want to access the MO-drive as normal user, but the device name can be /dev/sda or /dev/sdd.
So what options do I have here?
1. devfs: it would work in this situation, but it doesn't offer really stable device names.
2. scsidev: it's more flexible, but it is limited to scsi devices.

Another example:
Let's take a legacy free pc, where a serial port is only available via a usb port replicator. Is there really a good reason, why that port isn't available via /dev/ttyS0? Why has the user to find out, how that thing is called? Or why should a terminal program maintain a growing list of possible serial device names?

The information exported via sysfs offers now for the first time a more general mechanism to manage these problems. I'm trying very hard here to get an answer to the question "How are we going to manage a lot of devices?", instead I only get answers to the question "How do we get a lot of devices?". If we concentrate only on the latter, we neglect a major part of the problem.

I have to come back to the two questions I already asked earlier:
1. How do we want to manage devices in the future?
2. What compromises can we make for 2.6?

The first question requires a look at how we want to register devices, how is device information exported to the user and how the user can access the devices.
The second question requires a look at what we already have and what is still missing.

I think there is a general agreement that we need to manage more dynamically, so what do have already? Block devices are already the furthest and also need a larger dev_t the most. A lot of drivers which only use add_disk/register_blkdev (but not blk_register_region) care about the device number only during registration and don't need it otherwise. scsi already exports a lot of information via sysfs.
So if we just look at scsi we already have everything to manage devices more dynamically, userspace only has to make use of that information now. To manage now a lot of scsi devices, we only have to generate new device numbers for them, preferably in a way that a) other drivers can benefit from it and b) it doesn't break existing setups. The required kernel changes are minimal and certainly no reason to hold up any kernel release, but simply changing the kernel major/minor split breaks compatibility and should be avoided.

Character devices still need more work and that is mostly 2.7 material, but basically it's possible to give more control to user space similiar to block devices.

The only questions I still can't answer myself are:
1. What exactly do we need a 64bit dev_t for and what has "collective global experience with numberspaces" to do with the local device numberspace?
2. What advantage does a more static scheme have and what are the problems ignored by the dynamic schemes?
Above examples should help to demonstrate how to deal with the various problems.

Peter and Andries, I would really appreciate it, if you would stop ignoring me and start answering my questions or explain what is wrong in above explanation.
If anyone else can answer these question, you are of course welcome to do so, otherwise I hope you have more luck than me to get some answers here.

bye, Roman

Hi,

I should have inserted a "privately" there, because there a few mails and even a small irc conversation inbetween.

It was really a mistake from Peter to point me to the archives, it's really amazing to how much crap from the static device number crew had to be objected by Linus and Al.

bye, Roman

Hi!

Ugh? 64bits not enough? I don't think so. You could do 20/44 bits split for example... Or allocate 60000 majors just for disks. But I do not think you have 2^32 disks in any SAN.

				Pavel
Written on sharp zaurus, because my Velo1 broke. If you have Velo you don't need...

Hi,

Ok, I checked the archives and found some interesting mails:

http://www.ussg.iu.edu/hypermail/linux/kernel/0105.1/1170.html
http://www.ussg.iu.edu/hypermail/linux/kernel/0105.1/1180.html
http://www.ussg.iu.edu/hypermail/linux/kernel/0105.1/1072.html
http://www.ussg.iu.edu/hypermail/linux/kernel/0105.1/1310.html
http://www.ussg.iu.edu/hypermail/linux/kernel/0105.1/1348.html

This is from the last dev_t discussion I was able to find and my apologies to Linus for dragging him into this, personally I prefer a discussion based on arguments, but you want to feel the whip of authority. (There are also some juicy mails from Al, but you can look for these yourself.)

Linus argues here for dynamic numbers and I was not able to find a single mail, where he completely changed his mind since then. If you know something I don't, I'd be really happy to hear about it (actually I found 9 (nine!) year old mails, where he argues for a more dynamic system).

In above discussion, Alan was one of the few who actually came up with reasonable arguments, some of his concerns were:
- he needed a solution for 2.4:
we are at 2.5 now and the kernel is mostly ready for dynamic device numbers
- compatibility:
it's trivial to preserve dev numbers below 0x10000, new drivers start above this
- hardcoded ioctl knowledge:
this is partly a compatibility problem and in the meantime it's generally accepted that they are a bad idea most of the time and drivers should use e.g. sysfs instead.

Am I now worthy of an answer, so you could please explain "why dynamic numbers are broken for nearly all applications"? What were I supposed to learn from the archives? Maybe you should read them yourself, because I didn't found a single discussion with a clear outcome.

bye, Roman