Skip to contentSkip to conversation

cciss: fix for 2TB support

PATCH In review Mike Miller (OS Dev) 6 participants 15 messages Feb 21, 2007 – Feb 24, 2007

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/2 cciss: fix for 2TB support

Patch 1/2

This patch changes the way we determine if a logical volume is larger than 2TB. The original test looked for a total_size of 0. Originally we added 1 to the total_size. That would make our read_capacity return size 0 for >2TB lv's. We assumed that we could not have a lv size of 0 so it seemed OK until we were in a clustered system. The backup node would see a size of 0 due to the reservation on the drive. That caused the driver to switch to 16-byte CDB's which are not supported on older controllers. After that everything was broken.
It may seem petty but I don't see the value in trying to determine if the LBA is beyond the 2TB boundary. That's why when we switch we use 16-byte CDB's for all read/write operations.
Please consider this for inclusion.

------------------------------------------------------------------------------------------

drivers/block/cciss.c +24 -15
index 05dfe35..916aab0 100644
@@ -1291,13 +1291,19 @@ static void cciss_update_drive_info(int 
 	if (inq_buff == NULL)
 		goto mem_msg;
 
+ 	/* testing to see if 16-byte CDBs are already being used */
+ 	if (h->cciss_read == CCISS_READ_16) {
+ 		cciss_read_capacity_16(h->ctlr, drv_index, 1,
+ 			&total_size, &block_size);
+ 		goto geo_inq;
+ 	}
+  
 	cciss_read_capacity(ctlr, drv_index, 1,
 			    &total_size, &block_size);
 
-	/* total size = last LBA + 1 */
-	/* FFFFFFFF + 1 = 0, cannot have a logical volume of size 0 */
-	/* so we assume this volume this must be >2TB in size */
-	if (total_size == (__u32) 0) {
+  	/* if read_capacity returns all F's this volume is >2TB in size */
+  	/* so we switch to 16-byte CDB's for all read/write ops */
+  	if (total_size == 0xFFFFFFFF) {
 		cciss_read_capacity_16(ctlr, drv_index, 1,
 		&total_size, &block_size);
 		h->cciss_read = CCISS_READ_16;
@@ -1306,6 +1312,7 @@ static void cciss_update_drive_info(int 
 		h->cciss_read = CCISS_READ_10;
 		h->cciss_write = CCISS_WRITE_10;
 	}
+geo_inq:
 	cciss_geometry_inquiry(ctlr, drv_index, 1, total_size, block_size,
 			       inq_buff, &h->drv[drv_index]);
 
@@ -1917,13 +1924,14 @@ static void cciss_geometry_inquiry(int c
 			drv->raid_level = inq_buff->data_byte[8];
 		}
 		drv->block_size = block_size;
-		drv->nr_blocks = total_size;
+		drv->nr_blocks = total_size + 1;
 		t = drv->heads * drv->sectors;
 		if (t > 1) {
-			unsigned rem = sector_div(total_size, t);
+			sector_t real_size = total_size + 1;
+			unsigned long rem = sector_div(real_size, t);
 			if (rem)
-				total_size++;
-			drv->cylinders = total_size;
+				real_size++;
+			drv->cylinders = real_size;
 		}
 	} else {		/* Get geometry failed */
 		printk(KERN_WARNING "cciss: reading geometry failed\n");
@@ -1953,16 +1961,16 @@ cciss_read_capacity(int ctlr, int logvol
 				ctlr, buf, sizeof(ReadCapdata_struct),
 					1, logvol, 0, NULL, TYPE_CMD);
 	if (return_code == IO_OK) {
-		*total_size = be32_to_cpu(*(__u32 *) buf->total_size)+1;
+		*total_size = be32_to_cpu(*(__u32 *) buf->total_size);
 		*block_size = be32_to_cpu(*(__u32 *) buf->block_size);
 	} else {		/* read capacity command failed */
 		printk(KERN_WARNING "cciss: read capacity failed\n");
 		*total_size = 0;
 		*block_size = BLOCK_SIZE;
 	}
-	if (*total_size != (__u32) 0)
+	if (*total_size != 0)
 		printk(KERN_INFO "      blocks= %llu block_size= %d\n",
-		(unsigned long long)*total_size, *block_size);
+		(unsigned long long)*total_size+1, *block_size);
 	kfree(buf);
 	return;
 }
@@ -1989,7 +1997,7 @@ cciss_read_capacity_16(int ctlr, int log
 				1, logvol, 0, NULL, TYPE_CMD);
 	}
 	if (return_code == IO_OK) {
-		*total_size = be64_to_cpu(*(__u64 *) buf->total_size)+1;
+		*total_size = be64_to_cpu(*(__u64 *) buf->total_size);
 		*block_size = be32_to_cpu(*(__u32 *) buf->block_size);
 	} else {		/* read capacity command failed */
 		printk(KERN_WARNING "cciss: read capacity failed\n");
@@ -1997,7 +2005,7 @@ cciss_read_capacity_16(int ctlr, int log
 		*block_size = BLOCK_SIZE;
 	}
 	printk(KERN_INFO "      blocks= %llu block_size= %d\n",
-	       (unsigned long long)*total_size, *block_size);
+	       (unsigned long long)*total_size+1, *block_size);
 	kfree(buf);
 	return;
 }
@@ -3119,8 +3127,9 @@ #endif				/* CCISS_DEBUG */
 		}
 		cciss_read_capacity(cntl_num, i, 0, &total_size, &block_size);
 
-		/* total_size = last LBA + 1 */
-		if(total_size == (__u32) 0) {
+		/* If read_capacity returns all F's the logical is >2TB */
+		/* so we switch to 16-byte CDBs for all read/write ops */
+		if(total_size == 0xFFFFFFFF) {
 			cciss_read_capacity_16(cntl_num, i, 0,
 			&total_size, &block_size);
 			hba[cntl_num]->cciss_read = CCISS_READ_16;
#1 lore raw
Quoted 14 lines · On Wed, 21 Feb 2007 15:10:39 -0600 "Mike Miller (OS Dev)"…

Patch 1/2

This patch changes the way we determine if a logical volume is larger than 2TB. The original test looked for a total_size of 0. Originally we added 1 to the total_size. That would make our read_capacity return size 0 for >2TB lv's. We assumed that we could not have a lv size of 0 so it seemed OK until we were in a clustered system. The backup node would see a size of 0 due to the reservation on the drive. That caused the driver to switch to 16-byte CDB's which are not supported on older controllers. After that everything was broken.
It may seem petty but I don't see the value in trying to determine if the LBA is beyond the 2TB boundary. That's why when we switch we use 16-byte CDB's for all read/write operations.
Please consider this for inclusion.

Quoted 3 lines

...

+ if (total_size == 0xFFFFFFFF) {

I seem to remember having already questioned this. total_size is sector_t, which can be either 32-bit or 64-bit. Are you sure that comparison works as intended in both cases?

drivers/block/cciss.c #endif /* CCISS_DEBUG */ · line 3135

Quoted 4 lines
drivers/block/cciss.c +1 -0
+		if(total_size == 0xFFFFFFFF) {
 			cciss_read_capacity_16(cntl_num, i, 0,
 			&total_size, &block_size);
 			hba[cntl_num]->cciss_read = CCISS_READ_16;

Here too.

#2 lore raw

PATCH 1/1 Speedup divides by cpu_power in scheduler

I noticed expensive divides done in try_to_wakeup() and find_busiest_group() on a bi dual core Opteron machine (total of 4 cores), moderatly loaded (15.000 context switch per second)

oprofile numbers :

CPU: AMD64 processors, speed 2600.05 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit mask of 0x00 (No unit mask) count 50000
samples % symbol name
...
613914 1.0498 try_to_wake_up
834 0.0013 :ffffffff80227ae1: div %rcx
77513 0.1191 :ffffffff80227ae4: mov %rax,%r11

608893 1.0413 find_busiest_group

  1841  0.0031 :ffffffff802260bf:       div    %rdi
140109  0.2394 :ffffffff802260c2:       test   %sil,%sil

Some of these divides can use the reciprocal divides we introduced some time ago (currently used in slab AFAIK)

We can assume a load will fit in a 32bits number, because with a SCHED_LOAD_SCALE=128 value, its still a theorical limit of 33554432

When/if we reach this limit one day, probably cpus will have a fast hardware divide and we can zap the reciprocal divide trick.

I did not convert the divide in cpu_avg_load_per_task(), because tracking nr_running changes may be not worth it ? We could use a static table of 32 reciprocal values but it would add a conditional branch and table lookup.

include/linux/sched.h +6 -1
@@ -669,7 +669,12 @@ struct sched_group {
 	 * CPU power of this group, SCHED_LOAD_SCALE being max power for a
 	 * single CPU. This is read only (except for setup, hotplug CPU).
 	 */
-	unsigned long cpu_power;
+	unsigned int cpu_power;
+	/*
+	 * reciprocal value of cpu_power to avoid expensive divides
+	 * (see include/linux/reciprocal_div.h)
+	 */
+	u32 reciprocal_cpu_power;
 };
 
 struct sched_domain {
kernel/sched.c renamed from kernel/sched.c.orig +36 -12

111 lines not on this page · raw message

Acked-by on v1 1/1
#3 lore raw

Re: [PATCH] Speedup divides by cpu_power in scheduler

Quoted 5 lines · Eric Dumazet

I noticed expensive divides done in try_to_wakeup() and find_busiest_group() on a bi dual core Opteron machine (total of 4 cores), moderatly loaded (15.000 context switch per second)

oprofile numbers :

nice patch! Ack for -mm testing:

one general suggestion: could you rename ->cpu_power to ->__cpu_power? That makes it perfectly clear that this field's semantics have changed and that it should never be manipulated directly without also changing ->reciprocal_cpu_power, and will also flag any out of tree code trivially.

Quoted 7 lines

+ * Divide a load by a sched group cpu_power : (load / sg->cpu_power)
+ * Since cpu_power is a 'constant', we can use a reciprocal divide.
+ */

kernel/sched.c renamed from kernel/sched.c.orig +4 -0
+static inline u32 sg_div_cpu_power(const struct sched_group *sg, u32 load)
+{
+	return reciprocal_divide(load, sg->reciprocal_cpu_power);
+}

kernel/sched.c static unsigned int static_prio_timeslic · line 202

Quoted 10 lines
kernel/sched.c renamed from kernel/sched.c.orig +10 -0
+/*
+ * Each time a sched group cpu_power is changed,
+ * we must compute its reciprocal value
+ */
+static inline void sg_inc_cpu_power(struct sched_group *sg, u32 val)
+{
+	sg->cpu_power += val;
+	BUG_ON(sg->cpu_power == 0);
+	sg->reciprocal_cpu_power = reciprocal_value(sg->cpu_power);
+}

Could you remove the BUG_ON() - it will most likely cause the non-inlining of these functions if CONFIG_CC_OPTIMIZE_FOR_SIZE=y and CONFIG_FORCED_INLINING is disabled (which is a popular combination in distro kernels, it reduces the kernel's size by over 30%). And it's not like we'll be able to overlook a divide by zero crash in reciprocal_value() anyway, if cpu_power were to be zero ;-)

Ingo

#4 lore raw

PATCH 1/1 Speedup divides by cpu_power in scheduler

In reply to Eric Dumazet

I noticed expensive divides done in try_to_wakeup() and find_busiest_group() on a bi dual core Opteron machine (total of 4 cores), moderatly loaded (15.000 context switch per second)

oprofile numbers :

CPU: AMD64 processors, speed 2600.05 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit mask of 0x00 (No unit mask) count 50000
samples % symbol name
...
613914 1.0498 try_to_wake_up
834 0.0013 :ffffffff80227ae1: div %rcx
77513 0.1191 :ffffffff80227ae4: mov %rax,%r11

608893 1.0413 find_busiest_group

  1841  0.0031 :ffffffff802260bf:       div    %rdi
140109  0.2394 :ffffffff802260c2:       test   %sil,%sil

Some of these divides can use the reciprocal divides we introduced some time ago (currently used in slab AFAIK)

We can assume a load will fit in a 32bits number, because with a SCHED_LOAD_SCALE=128 value, its still a theorical limit of 33554432

When/if we reach this limit one day, probably cpus will have a fast hardware divide and we can zap the reciprocal divide trick.

Ingo suggested to rename cpu_power to __cpu_power to make clear it should not be modified without changing its reciprocal value too.

I did not convert the divide in cpu_avg_load_per_task(), because tracking nr_running changes may be not worth it ? We could use a static table of 32 reciprocal values but it would add a conditional branch and table lookup.

include/linux/sched.h +7 -1
@@ -668,8 +668,14 @@ struct sched_group {
 	/*
 	 * CPU power of this group, SCHED_LOAD_SCALE being max power for a
 	 * single CPU. This is read only (except for setup, hotplug CPU).
+	 * Note : Never change cpu_power without recompute its reciprocal
 	 */
-	unsigned long cpu_power;
+	unsigned int __cpu_power;
+	/*
+	 * reciprocal value of cpu_power to avoid expensive divides
+	 * (see include/linux/reciprocal_div.h)
+	 */
+	u32 reciprocal_cpu_power;
 };
 
 struct sched_domain {
kernel/sched.c +48 -25

173 lines not on this page · raw message

Acked-by on v1 1/1
#5 lore raw

Re: [PATCH, take 2] Speedup divides by cpu_power in scheduler

Quoted 2 lines · Eric Dumazet

Ingo suggested to rename cpu_power to __cpu_power to make clear it should not be modified without changing its reciprocal value too.

thanks,

Quoted 4 lines

I did not convert the divide in cpu_avg_load_per_task(), because tracking nr_running changes may be not worth it ? We could use a static table of 32 reciprocal values but it would add a conditional branch and table lookup.

not worth it i think. Lets wait for it to show up in an oprofile? (if ever)

Ingo

#6 lore raw
In reply to Andrew Morton
Quoted 25 lines · Andrew Morton
Quoted 14 lines · On Wed, 21 Feb 2007 15:10:39 -0600 "Mike Miller (OS Dev)"…

Patch 1/2

This patch changes the way we determine if a logical volume is larger than 2TB. The original test looked for a total_size of 0. Originally we added 1 to the total_size. That would make our read_capacity return size 0 for >2TB lv's. We assumed that we could not have a lv size of 0 so it seemed OK until we were in a clustered system. The backup node would see a size of 0 due to the reservation on the drive. That caused the driver to switch to 16-byte CDB's which are not supported on older controllers. After that everything was broken.
It may seem petty but I don't see the value in trying to determine if the LBA is beyond the 2TB boundary. That's why when we switch we use 16-byte CDB's for all read/write operations.
Please consider this for inclusion.

Quoted 3 lines

...

+ if (total_size == 0xFFFFFFFF) {

I seem to remember having already questioned this. total_size is sector_t, which can be either 32-bit or 64-bit. Are you sure that comparison works as intended in both cases?

Quoted 6 lines

drivers/block/cciss.c #endif /* CCISS_DEBUG */ · line 3135

Quoted 4 lines
drivers/block/cciss.c +1 -0
+		if(total_size == 0xFFFFFFFF) {
 			cciss_read_capacity_16(cntl_num, i, 0,
 			&total_size, &block_size);
 			hba[cntl_num]->cciss_read = CCISS_READ_16;

Here too.

It has worked in all of the configs I've tested. Should I change it from sector_t to a __64? I have not tested all possible configs.

-- mikem

#7 lore raw
Quoted 12 lines · On Thu, 22 Feb 2007 10:51:23 -0600 "Mike Miller (OS Dev)"…
Quoted 10 lines · Andrew Morton
Quoted 2 lines · On Wed, 21 Feb 2007 15:10:39 -0600 "Mike Miller (OS Dev)"…

Patch 1/2
+ if (total_size == 0xFFFFFFFF) {

I seem to remember having already questioned this. total_size is sector_t, which can be either 32-bit or 64-bit. Are you sure that comparison works as intended in both cases?

Quoted 9 lines
Quoted 6 lines

drivers/block/cciss.c #endif /* CCISS_DEBUG */ · line 3135

Quoted 4 lines
drivers/block/cciss.c +1 -0
+		if(total_size == 0xFFFFFFFF) {
 			cciss_read_capacity_16(cntl_num, i, 0,
 			&total_size, &block_size);
 			hba[cntl_num]->cciss_read = CCISS_READ_16;

Here too.

It has worked in all of the configs I've tested. Should I change it from sector_t to a __64? I have not tested all possible configs.

I'd suggest using -1: that just works.

#8 lore raw
Quoted 18 lines · Andrew Morton
Quoted 18 lines · On Thu, 22 Feb 2007 10:51:23 -0600 "Mike Miller (OS Dev)"…
Quoted 16 lines · Andrew Morton
Quoted 2 lines · On Wed, 21 Feb 2007 15:10:39 -0600 "Mike Miller (OS Dev)"…

Patch 1/2
+ if (total_size == 0xFFFFFFFF) {

I seem to remember having already questioned this. total_size is sector_t, which can be either 32-bit or 64-bit. Are you sure that comparison works as intended in both cases?

drivers/block/cciss.c #endif /* CCISS_DEBUG */ · line 3135

Quoted 4 lines
drivers/block/cciss.c +1 -0
+		if(total_size == 0xFFFFFFFF) {
 			cciss_read_capacity_16(cntl_num, i, 0,
 			&total_size, &block_size);
 			hba[cntl_num]->cciss_read = CCISS_READ_16;

Here too.

Quoted 5 lines
Quoted 3 lines

It has worked in all of the configs I've tested. Should I change it from sector_t to a __64? I have not tested all possible configs.

I'd suggest using -1: that just works.

Actually, no, that won't work.

This is a SCSI heuristic for determining when to use the 16 byte version of the read capacity command. The 10 byte command can only return 32 bits of information (this is in sectors, so it returns up to 2TB of bytes).

The heuristic requirement is that if the size is exactly 0xffffffff then you should try the 16 byte command (which can return 64 bits of information). If that fails then you assume the 0xfffffff is a real size otherwize, you assume it was truncated and take the real result from the 16 byte command.

You can see a far more elaborate version of this in operation in sd.c:sd_read_capacity().

The only thing I'd suggest is to use 0xFFFFFFFFULL as the constant to prevent sign extension issues.

James

#9 lore raw
Quoted 38 lines · James Bottomley
Quoted 18 lines · Andrew Morton
Quoted 18 lines · On Thu, 22 Feb 2007 10:51:23 -0600 "Mike Miller (OS Dev)"…
Quoted 16 lines · Andrew Morton
Quoted 2 lines · On Wed, 21 Feb 2007 15:10:39 -0600 "Mike Miller (OS Dev)"…

Patch 1/2
+ if (total_size == 0xFFFFFFFF) {

I seem to remember having already questioned this. total_size is sector_t, which can be either 32-bit or 64-bit. Are you sure that comparison works as intended in both cases?

Review

Quoted 4 lines
(quoted diff) +1 -0
+		if(total_size == 0xFFFFFFFF) {
 			cciss_read_capacity_16(cntl_num, i, 0,
 			&total_size, &block_size);
 			hba[cntl_num]->cciss_read = CCISS_READ_16;

Here too.

Quoted 5 lines
Quoted 3 lines

It has worked in all of the configs I've tested. Should I change it from sector_t to a __64? I have not tested all possible configs.

I'd suggest using -1: that just works.

Actually, no, that won't work.

This is a SCSI heuristic for determining when to use the 16 byte version of the read capacity command. The 10 byte command can only return 32 bits of information (this is in sectors, so it returns up to 2TB of bytes).

The heuristic requirement is that if the size is exactly 0xffffffff then you should try the 16 byte command (which can return 64 bits of information). If that fails then you assume the 0xfffffff is a real size otherwize, you assume it was truncated and take the real result from the 16 byte command.

Quoted 9 lines

You can see a far more elaborate version of this in operation in sd.c:sd_read_capacity().

The only thing I'd suggest is to use 0xFFFFFFFFULL as the constant to prevent sign extension issues.

James

Will this patch for my patch work for now?

drivers/block/cciss.c +2 -2
index 1abf1f5..a1f1d9f 100644
@@ -1303,7 +1303,7 @@ static void cciss_update_drive_info(int 
 
   	/* if read_capacity returns all F's this volume is >2TB in size */
   	/* so we switch to 16-byte CDB's for all read/write ops */
-  	if (total_size == 0xFFFFFFFF) {
+  	if (total_size == 0xFFFFFFFFULL) {
 		cciss_read_capacity_16(ctlr, drv_index, 1,
 		&total_size, &block_size);
 		h->cciss_read = CCISS_READ_16;
@@ -3129,7 +3129,7 @@ #endif				/* CCISS_DEBUG */
 
 		/* If read_capacity returns all F's the logical is >2TB */
 		/* so we switch to 16-byte CDBs for all read/write ops */
-		if(total_size == 0xFFFFFFFF) {
+		if(total_size == 0xFFFFFFFFULL) {
 			cciss_read_capacity_16(cntl_num, i, 0,
 			&total_size, &block_size);
 			hba[cntl_num]->cciss_read = CCISS_READ_16;
#10 lore raw
Quoted 1 line · Mike Miller (OS Dev)

Will this patch for my patch work for now?

Yes, I think that should be fine ... it's only a theoretical worry; at the moment sector_t is unsigned ... but just in case.

James

#11 lore raw
Quoted 9 lines · James Bottomley
Quoted 1 line · Mike Miller (OS Dev)

Will this patch for my patch work for now?

Yes, I think that should be fine ... it's only a theoretical worry; at the moment sector_t is unsigned ... but just in case.

James

Andrew,
Are you waiting for a new patch from me? Or is my patch's patch sufficient?

-- mikem

#12 lore raw
Quoted 5 lines · On Fri, 23 Feb 2007 14:52:29 -0600 "Mike Miller (OS Dev)"…
Quoted 3 lines · James Bottomley
Quoted 1 line · Mike Miller (OS Dev)

Will this patch for my patch work for now?

Quoted 9 lines
Quoted 6 lines

Yes, I think that should be fine ... it's only a theoretical worry; at the moment sector_t is unsigned ... but just in case.

James

Andrew,
Are you waiting for a new patch from me? Or is my patch's patch sufficient?

It looked OK. But I'm travelling at present, will get back into things late next week.

#13 lore raw
In reply to Andrew Morton
Quoted 8 lines · Andrew Morton
Quoted 2 lines

+ if (total_size == 0xFFFFFFFF) {

I seem to remember having already questioned this. total_size is sector_t, which can be either 32-bit or 64-bit. Are you sure that comparison works as intended in both cases?

Quoted 6 lines

drivers/block/cciss.c #endif /* CCISS_DEBUG */ · line 3135

Quoted 4 lines
drivers/block/cciss.c +1 -0
+		if(total_size == 0xFFFFFFFF) {
 			cciss_read_capacity_16(cntl_num, i, 0,
 			&total_size, &block_size);
 			hba[cntl_num]->cciss_read = CCISS_READ_16;

Here too.

Andrew,
Using this test program and changing the type of x to int, long, long long signed and unsigned the comparison always worked on x86, x86_64, and ia64. It looks to me like the comparsion will always do what we expect. Unless you see some other problem.

#include <stdio.h>

int main(int argc, char *argv[])

{
	unsigned long long x;
	
	x =  0x00000000ffffffff;

	printf(sizeof(x) == 8 ? 
		"x = %lld, sizeof(x) = %d\n" :
		"x = %ld, sizeof(x) = %d\n",  x, sizeof(x));
	if (x == 0xffffffff)
		printf("equal\n");
	else
		printf("not equal\n");

}

-- mikem

#14 lore raw
Quoted 29 lines

-----Original Message-----
From: Mike Miller (OS Dev) [mailto:[email protected]]

Andrew,
Using this test program and changing the type of x to int, long, long long signed and unsigned the comparison always worked on x86, x86_64, and ia64. It looks to me like the comparsion will always do what we expect. Unless you see some other problem.

#include <stdio.h>

int main(int argc, char *argv[])

{
	unsigned long long x;
	
	x =  0x00000000ffffffff;

	printf(sizeof(x) == 8 ? 
		"x = %lld, sizeof(x) = %d\n" :
		"x = %ld, sizeof(x) = %d\n",  x, sizeof(x));
	if (x == 0xffffffff)
		printf("equal\n");
	else
		printf("not equal\n");
Quoted 4 lines

}

-- mikem

BTW: also changed x to be 8 f's, 16 f's, and 8 and 8 as shown.