fio 2.0.9 fails: shmget: No space left on device

DISCUSSION Konstantin Boyandin 2 participants 6 messages Sep 21, 2012

lore

Share & tools

lore

Konstantin Boyandin

author

2012-09-21 11:51 UTC #0 lore raw

Hello,

When trying to run the command line

fio --bs=4k --size=128m --direct=1 --runtime=10 --rw=randread --numjobs=32 --group_reporting --time_based --name=128m

I see only this:

shmget: No space left on device
error: failed to setup shm segment

Environment:

# cat /etc/redhat-release
CentOS release 6.3 (Final)

# uname -a
Linux vpseer 2.6.32-042stab059.7 #1 SMP Tue Jul 24 19:12:01 MSK 2012
x86_64 x86_64 x86_64 GNU/Linux

# virt-what
openvz

# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/simfs            52428800   1590532  50838268   4% /
none                    526336         4    526332   1% /dev
none                    526336         0    526336   0% /dev/shm

If additional info required, please tell me.
Thanks.

Sincerely,
Konstantin

Jens Axboe

2012-09-21 12:16 UTC #1 lore raw

Quoted 11 lines · Konstantin Boyandin

Hello,
When trying to run the command line
fio --bs=4k --size=128m --direct=1 --runtime=10 --rw=randread --numjobs=32 --group_reporting --time_based --name=128m
I see only this:
shmget: No space left on device
error: failed to setup shm segment

Hmmm. Can you send me the output of:

$ grep . /proc/sys/kernel/shm*
$ ipcs -l
$ ipcs -m

Signature · 1 line

Jens Axboe

Konstantin Boyandin

author

2012-09-21 12:34 UTC #2 lore raw

Hello Jens,

Quoted 8 lines · Jens Axboe

Quoted 7 lines · Konstantin Boyandin
Hello,
When trying to run the command line
fio --bs=4k --size=128m --direct=1 --runtime=10 --rw=randread --numjobs=32 --group_reporting --time_based --name=128m

Quoted 8 lines

Quoted 4 lines
I see only this:
shmget: No space left on device
error: failed to setup shm segment
Hmmm. Can you send me the output of:
$ grep . /proc/sys/kernel/shm*

/proc/sys/kernel/shmall:2097152
/proc/sys/kernel/shmmax:33554432
/proc/sys/kernel/shmmni:4096
/proc/sys/kernel/shm_rmid_forced:0

Quoted 1 line

$ ipcs -l

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 32768
max total shared memory (kbytes) = 8388608
min seg size (bytes) = 1

------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767

------ Messages: Limits --------
max queues system wide = 6365
max size of message (bytes) = 8192
default max size of queue (bytes) = 16384

Quoted 1 line

$ ipcs -m

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status

The VM wasn't restarted after the fio failed run.

fio was run by root user.

Sincerely,
Konstantin

Jens Axboe

2012-09-21 12:38 UTC #3 lore raw

Quoted 45 lines · Konstantin Boyandin

Hello Jens,
Quoted 8 lines · Jens Axboe
Quoted 7 lines · Konstantin Boyandin
Hello,
When trying to run the command line
fio --bs=4k --size=128m --direct=1 --runtime=10 --rw=randread --numjobs=32 --group_reporting --time_based --name=128m
Quoted 8 lines
Quoted 4 lines
I see only this:
shmget: No space left on device
error: failed to setup shm segment
Hmmm. Can you send me the output of:
$ grep . /proc/sys/kernel/shm*
/proc/sys/kernel/shmall:2097152
/proc/sys/kernel/shmmax:33554432
/proc/sys/kernel/shmmni:4096
/proc/sys/kernel/shm_rmid_forced:0
Quoted 1 line
$ ipcs -l
------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 32768
max total shared memory (kbytes) = 8388608
min seg size (bytes) = 1
------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767
------ Messages: Limits --------
max queues system wide = 6365
max size of message (bytes) = 8192
default max size of queue (bytes) = 16384

Quoted 8 lines

Quoted 1 line
$ ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
The VM wasn't restarted after the fio failed run.
fio was run by root user.

Oh, try with this please.

init.c +1 -1

index da1f472..b3215f5 100644
@@ -254,7 +254,7 @@ static int setup_thread_area(void)
 		shm_id = shmget(0, size, IPC_CREAT | 0600);
 		if (shm_id != -1)
 			break;
-		if (errno != EINVAL && errno != ENOMEM) {
+		if (errno != EINVAL && errno != ENOMEM && errno != ENOSPC) {
 			perror("shmget");
 			break;
 		}

Signature · 1 line

Jens Axboe

Konstantin Boyandin

author

2012-09-21 13:02 UTC #4 lore raw

Hello Jens,

Quoted 62 lines · Jens Axboe

Quoted 45 lines · Konstantin Boyandin
Hello Jens,
Quoted 8 lines · Jens Axboe
Quoted 7 lines · Konstantin Boyandin
Hello,
When trying to run the command line
fio --bs=4k --size=128m --direct=1 --runtime=10 --rw=randread --numjobs=32 --group_reporting --time_based --name=128m
Quoted 8 lines
Quoted 4 lines
I see only this:
shmget: No space left on device
error: failed to setup shm segment
Hmmm. Can you send me the output of:
$ grep . /proc/sys/kernel/shm*
/proc/sys/kernel/shmall:2097152
/proc/sys/kernel/shmmax:33554432
/proc/sys/kernel/shmmni:4096
/proc/sys/kernel/shm_rmid_forced:0
Quoted 1 line
$ ipcs -l
------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 32768
max total shared memory (kbytes) = 8388608
min seg size (bytes) = 1
------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767
------ Messages: Limits --------
max queues system wide = 6365
max size of message (bytes) = 8192
default max size of queue (bytes) = 16384
Quoted 8 lines
Quoted 1 line
$ ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
The VM wasn't restarted after the fio failed run.
fio was run by root user.
Oh, try with this please.
init.c +0 -0
index da1f472..b3215f5 100644

init.c static int setup_thread_area(void) · line 260

Quoted 8 lines

init.c +1 -1

@@ -254,7 +254,7 @@ static int setup_thread_area(void)
 		shm_id = shmget(0, size, IPC_CREAT | 0600);
 		if (shm_id != -1)
 			break;
-		if (errno != EINVAL && errno != ENOMEM) {
+		if (errno != EINVAL && errno != ENOMEM && errno != ENOSPC) {
 			perror("shmget");
 			break;
 		}

The patch worked fine, thank you!
fio works now as expected.

Sincerely,
Konstantin

Jens Axboe

2012-09-21 13:07 UTC #5 lore raw

Quoted 60 lines · Konstantin Boyandin

Hello Jens,
Quoted 57 lines · Jens Axboe
Quoted 45 lines · Konstantin Boyandin
Hello Jens,
Quoted 8 lines · Jens Axboe
Quoted 7 lines · Konstantin Boyandin
Hello,
When trying to run the command line
fio --bs=4k --size=128m --direct=1 --runtime=10 --rw=randread --numjobs=32 --group_reporting --time_based --name=128m
Quoted 8 lines
Quoted 4 lines
I see only this:
shmget: No space left on device
error: failed to setup shm segment
Hmmm. Can you send me the output of:
$ grep . /proc/sys/kernel/shm*
/proc/sys/kernel/shmall:2097152
/proc/sys/kernel/shmmax:33554432
/proc/sys/kernel/shmmni:4096
/proc/sys/kernel/shm_rmid_forced:0
Quoted 1 line
$ ipcs -l
------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 32768
max total shared memory (kbytes) = 8388608
min seg size (bytes) = 1
------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767
------ Messages: Limits --------
max queues system wide = 6365
max size of message (bytes) = 8192
default max size of queue (bytes) = 16384
Quoted 8 lines
Quoted 1 line
$ ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
The VM wasn't restarted after the fio failed run.
fio was run by root user.
Oh, try with this please.

Quoted 16 lines

init.c static int setup_thread_area(void) · line 260

Quoted 13 lines

init.c +1 -1

index da1f472..b3215f5 100644
@@ -254,7 +254,7 @@ static int setup_thread_area(void)
 		shm_id = shmget(0, size, IPC_CREAT | 0600);
 		if (shm_id != -1)
 			break;
-		if (errno != EINVAL && errno != ENOMEM) {
+		if (errno != EINVAL && errno != ENOMEM && errno != ENOSPC) {
 			perror("shmget");
 			break;
 		}

The patch worked fine, thank you!
fio works now as expected.

Excellent. Thanks for the report, fix has been committed.

Signature · 1 line

Jens Axboe