vm: shmat() address for 32 vs 64 bit kernel
I were needed to run a legacy application which uses Oracle 8 database. Tried to run it on a 64bit kernel, which is running on my machine for a long time, together with 32bits userland. Everything worked but not oracle, which segfaults at startup on 64bits kernel, but works just fine when running the same but 32bits kernel (userspace is the same).
After much debugging I found this difference which may be relevant. It's the address returned by shmat() routine.
Here are the relevant stuff for 32bits:
shmget(0x4d21f3bc, 579067904, IPC_CREAT|IPC_EXCL|0640) = 174358532shmat(174358532, 0, 0) = 0x9569f000
Here, the segment ends at (0x9569f000+579067904)/1024/1024 = 2942Mb.
And here's what happens with 64bits kernel:
shmget(0xc9a7c840, 579067904, IPC_CREAT|IPC_EXCL|0640) = 294912shmat(294912, 0, 0) = 0xd5548000
Here, the segment ends at 3965Mb.
And right after that shmat, oracle process terminates with SIGSEGV (on 64bits kernel, but continues just fine on 32bits).
I understand full well many bad points here: proprietary software, old legacy code, not supported, buggy.
But the question really is: why the difference in the first place, for 32bits userland, between 32 and 64bits kernels? Can this diff be "fixed" somehow? For example, I remember kernel/user memory split kconfig option - is there similar thing for 32bits userspace on 64bits kernel?
(And for the record, in case others will encounter the same issue:
the process started it all sees this:
SQL> startup
ORA-24323: value not allowed
ORA-03113: end-of-file on communication channel
and in the alert file we see this:
Errors in file /usr/oracle/8.1/rdbms/log/ora_3365.trc:
ORA-00445: background process "PMON" did not start after 120 seconds )
Thanks!
/mjt
Quoted 23 lines · Michael Tokarev
I were needed to run a legacy application which uses Oracle 8 database. Tried to run it on a 64bit kernel, which is running on my machine for a long time, together with 32bits userland. Everything worked but not oracle, which segfaults at startup on 64bits kernel, but works just fine when running the same but 32bits kernel (userspace is the same).
After much debugging I found this difference which may be relevant. It's the address returned by
shmat()routine.Here are the relevant stuff for 32bits:
shmget(0x4d21f3bc, 579067904, IPC_CREAT|IPC_EXCL|0640)= 174358532shmat(174358532, 0, 0)=0x9569f000Here, the segment ends at (0x9569f000+579067904)/1024/1024 = 2942Mb.
And here's what happens with 64bits kernel:
shmget(0xc9a7c840, 579067904, IPC_CREAT|IPC_EXCL|0640)= 294912shmat(294912, 0, 0)=0xd5548000
Quoted 7 lines
Here, the segment ends at 3965Mb.
And right after that shmat, oracle process terminates with SIGSEGV (on 64bits kernel, but continues just fine on 32bits).
I understand full well many bad points here: proprietary software, old legacy code, not supported, buggy.
Fortunately, compatibility is compatibility, and none of these are relevant. :)
Quoted 6 lines
But the question really is: why the difference in the first place, for 32bits userland, between 32 and 64bits kernels? Can this diff be "fixed" somehow? For example, I remember kernel/user memory split kconfig option - is there similar thing for 32bits userspace on 64bits kernel?
On 32-bit systems, the kernel is mapped into the last gigabyte of the 4 GB address space for every process. On 64-bit systems, the kernel is mapped in the negative half of the 64-bit address space for all processes, which leaves the top gigabyte of the 4 GB address space free for userspace use.
Most of the time this just means more memory for the processes that need it, but (as you have discovered) sometimes it can cause problems, which is why the setarch command has the --3gb flag.
Quoted 16 lines
(And for the record, in case others will encounter the same issue:
the process started it all sees this:SQL> startup
ORA-24323: value not allowed
ORA-03113: end-of-file on communication channeland in the alert file we see this:
Errors in file
/usr/oracle/8.1/rdbms/log/ora_3365.trc:
ORA-00445: background process "PMON" did not start after 120 seconds )Thanks!
/mjt
Signature · 1 line
Nicholas Miell <[email protected]>
Quoted 10 lines · Nicholas Miell
Quoted 9 lines · Michael Tokarev
shmat(174358532, 0, 0)=0x9569f000
Here, the segment ends at (0x9569f000+579067904)/1024/1024 = 2942Mb.And here's what happens with 64bits kernel:
shmget(0xc9a7c840, 579067904, IPC_CREAT|IPC_EXCL|0640)= 294912 Here, the segment ends at 3965Mb.
[]
Quoted 5 lines
Quoted 2 lines
I understand full well many bad points here: proprietary software, old legacy code, not supported, buggy.
Fortunately, compatibility is compatibility, and none of these are relevant. :)
Well, compatibility for 64bits kernel and 32bits userland it not the first priority, so to say.
[]
Quoted 9 lines
On 32-bit systems, the kernel is mapped into the last gigabyte of the 4 GB address space for every process. On 64-bit systems, the kernel is mapped in the negative half of the 64-bit address space for all processes, which leaves the top gigabyte of the 4 GB address space free for userspace use.
Most of the time this just means more memory for the processes that need it, but (as you have discovered) sometimes it can cause problems, which is why the setarch command has the --3gb flag.
Oh. Nice thing, I just discovered personality(2) stuff ;) Thank you very much for the pointer!
Except of some.. interesting stuff here. It does not quite work as *I*'d expect (not to say it's wrong but just unexpected). Namely, when setarch is also 32bits, it can't execute anything after setting ADDR_LIMIT_3GB (or PER_LINUX32_3GB). execve(2) returns immediately with EFAULT error. But if doing the same from a 64bit executable, it all works as expected. Or when doing the same from 32bit exe running on a 32bit kernel. I.e. the only problematic case for personality(PER_LINUX32_3GB) call is 32bit userspace on 64bit kernel.
personality(PER_LINUX32_3GB) = 0execve("/bin/echo", ["/bin/echo", "foo"], [/* 13 vars */]) = -1 EFAULT (Bad address)
Should it work? ;)
For now, it is running correctly when using a 64bits setarch executable in-between. Sure it will not work anymore if I'll reboot into 32bits kernel ;) (I run 64bits now because I'm trying to move data between two databases, one old 32bits and another is 64 bits).
Thank you!
/mjt