"Trying to free nonexistent swap-page" error message.

Hello,

I have searched the archives for this error message before, but no one seems to have given a good answer. (Though the question has been posted before.) I am not sure if this is a kernel problem, a hardware problem or a Oracle problem. (Or a combination of them.)

One one of our Linux Oracle servers the following messages has started to appear :

Jun 29 07:16:32 blanco kernel: swap_free: Trying to free nonexistent swap-page
Jun 29 07:16:32 blanco kernel: swap_free: Trying to free nonexistent swap-page

They seem to always come in pairs, and usually with about three hours between them.

The database had to be restored from backup because of massive table corruption recently, but these messages also appeared before we had to restore it. (But we believe they might have caused the corruption.)

I also find some of these:

Jun 29 06:25:01 blanco kernel: EXT2-fs error (device sd(8,10)): ext2_readdir: bad entry in directory #172258: rec_len %% 4 != 0 - offset=192, inode=812610409, rec_len=11833, name_len=115
Jun 29 06:25:32 blanco kernel: EXT2-fs error (device sd(8,10)): ext2_readdir: bad entry in directory #172258: rec_len %% 4 != 0 - offset=192, inode=812610409, rec_len=11833, name_len=115

Machine is a 2x933MhZ P3 with 2GB of memory. Kernel version is now 2.2.19, but the same problem appeared with 2.2.18 as well. The database is in moderate to heavy use 24/7 and with a lot (~ 500 - 3000) processes during business hours.

The machine has only 128MB of swap, is this to little since it has a full 2GB of memory?

The kernel is stock 2.2.19 with the following patch applied:

@@ -11,7 +11,7 @@
 #define NR_CPUS 1
 #endif
 
-#define NR_TASKS	512	/* On x86 Max about 4000 */
+#define NR_TASKS	4000	/* On x86 Max about 4000 */
 
 #define MAX_TASKS_PER_USER (NR_TASKS/2)
 #define MIN_TASKS_LEFT_FOR_ROOT 4

--
Regards
Johan Seland
Programmer
Net Fonds ASA

Johan Seland

My first guess would be some sort of hardware/software problem with your SCSI controller, cables, disk, etc. I'm not sure about the swap problem, but the ext2 problems are caused by corruption of the disk or memory.

It is not just a single-bit error either, because rec_len % 4 != 0 AND it is larger than a page size, so the value is totally bogus, as is the inode number. Interestingly, converting the above ext2 numbers into ascii gives:

0x69 0x73 0x6f 0x30 0x39 0x2e 0x73 => iso09.s

(in the order they are layed out in ext2_dir_entry_2). Coincidence or bug? I would suggest a full fsck for the filesystem, as it is likely that there are other problems.

Now when you say "servers" do you mean you have the same problem on multiple machines? Are they identical, or different?

Cheers, Andreas

Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

My first guess would be hardware also, except in this case I've seen similiar things on three different dual processor G4 systems running 2.2, and they work fine with 2.4.

Does Oracle for Linux us pthreads or the 'clone()' system call?

Can you try running the included pthreads program on an 2.2.19 SMP system (but make it's idle, since if this is a genric 2.2 SMP bug it will probably crash the system)

Compile with the following command:
gcc -o pt pthread-test.c -lpthread

Run it repeatedly:
I=0; while [ $? -eq 0 ] ; let I=I+1; do ./pt ; done ; echo $I

on 2.2.19 on a mac dual G4, the pthreads program will sometimes get segfaults and illegal instructions, and if I run it long enough, I will eventually get the following in dmesg:

swap_free: Trying to free nonexistent swap-page
swap_free: offset exceeds max
swap_free: Trying to free nonexistent swap-page
swap_free: Trying to free nonexistent swap-page
swap_free: offset exceeds max
swap_free: Trying to free nonexistent swap-page
swap_free: Trying to free nonexistent swap-page
swap_free: offset exceeds max
swap_free: offset exceeds max
swap_free: offset exceeds max
swap_free: Trying to free nonexistent swap-page
swap_free: Trying to free nonexistent swap-page
swap_free: Trying to free nonexistent swap-page
swap_free: Trying to free nonexistent swap-page
swap_free: offset exceeds max
swap_free: offset exceeds max
swap_free: offset exceeds max

If I keep running it, I will eventually wind up with a kernel panic on an illegal instruction. Something is corrupting memory, and in my case, the kernel panics are caused by a '0x00000008' being written over a random location in the kernel code.

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <pthread.h>

#define NTH 8

void
thread(void *arg)

{
        printf("THREAD: pid %d\n", getpid());
 
        return;
}

int
main(int argc, char **argv)

{
        int i;
        pthread_t t[NTH];
 
        for (i=0; i<NTH; i++)
           pthread_create(t+i, NULL, thread, NULL);
 
        for (i=0; i<NTH; i++)
           pthread_join(t[i], NULL);
 
        return 0;
}

Troy Benjegerdes | master of mispeeling | 'da hozer' |  [email protected]
-----"If this message isn't misspelled, I didn't write it" -- Me -----
"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's 
why I draw cartoons. It's my life." -- Charles Shulz

I have replaced all the memory with fresh 4x512MB REGISTRED ECC RAM, and it has been running stable for 8 days now. I also replaced the SCSI cables. 2.4 is not an option.

I am not really sure. How do I find out?

Sorry, I am not going to risk bringing down the database with your program. Its to critical for our business. (The database contains all stock quotes for Oslo, Stockholm, Frankfurt, NYSE, AMEX and NASDAQ stock exchanges and they are updated 24/7. I can only bring it down in weekends, and only if it really important.)

However I have access to a few other dual boxen. (2x550, 2x350, 2x166 and a dual SparcStation 20). I can run your program on them with stock 2.2.19 SMP if you want me to.

Med vennlig hilsen 
Johan Seland
for Net Fonds ASA

If you have several identical machines running oracle, and the machine is stable after changing the RAM & scsi cables, my guess it was a RAM or cable problem.

I don't think it will show anything.. I've manged to find a dual PPro to run on and didn't see any problems. I think it's a PPC specific problem.

Thanks though.

Troy Benjegerdes | master of mispeeling | 'da hozer' |  [email protected]
-----"If this message isn't misspelled, I didn't write it" -- Me -----
"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's 
why I draw cartoons. It's my life." -- Charles Shulz