Skip to contentSkip to conversation

Call trace during snapshot start/stop sequence

BUG Arkadiusz Bubała 2 participants 6 messages Nov 27, 2013 – Dec 5, 2013

lore
Share & tools
lore
#0 lore raw

Hello,

we're running test script that starts and stops
snapshots in a loop while overfilling them. After a few days of running system hangs. We've captured following call trace:

[116649.755761] XFS (dm-42): metadata I/O error: block 0xfa2b06
("xlog_iodone") error 5 buf count 1024
[116649.947247] XFS (dm-42): Log I/O Error Detected.  Shutting down
filesystem
[116650.073881] XFS (dm-42): Please umount the filesystem and rectify
the problem(s)
[116650.207186] BUG: unable to handle kernel paging request at
00000000000010a8
[116650.335185] IP: [<ffffffff8102e1d6>] __ticket_spin_lock+0x6/0x20
[116650.451052] PGD 0
[116650.518151] Oops: 0002 [#1] SMP
[116650.599477] CPU 0
[116650.622838] Modules linked in: iscsi_scst(O) scst_vdisk(O) scst(O)
drbd(O) twofish_x86_64 twofish_generic twofish_common
serpent_sse2_x86_64 lrw xts gf1]
[116651.479730]
[116651.540674] Pid: 30173, comm: kworker/0:5 Tainted: G           O
3.4.63-oe64-00000-g1a33902 #38 Intel Corporation S1200BTL/S1200BTL
[116651.772395] RIP: 0010:[<ffffffff8102e1d6>]  [<ffffffff8102e1d6>]
__ticket_spin_lock+0x6/0x20
[116651.921067] RSP: 0018:ffff88010a9c7e30  EFLAGS: 00010246
[116652.031355] RAX: 0000000000000100 RBX: 00000000000010a8 RCX:
0000000000000000
[116652.163291] RDX: 0000000000000092 RSI: 0000000000000002 RDI:
00000000000010a8
[116652.294384] RBP: ffff88022d424c00 R08: 0000000000000000 R09:
ffff8802334ba740
[116652.425190] R10: 0000000000000000 R11: ffffffff812c24b0 R12:
0000000000001000
[116652.555680] R13: 0000000000000002 R14: 0000000000000000 R15:
ffff880237017500
[116652.685235] FS:  0000000000000000(0000) GS:ffff880237000000(0000)
knlGS:0000000000000000
[116652.825924] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[116652.937639] CR2: 00000000000010a8 CR3: 0000000001874000 CR4:
00000000000407f0
[116653.066047] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[116653.193821] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[116653.320481] Process kworker/0:5 (pid: 30173, threadinfo
ffff88010a9c6000, task ffff880233f7f0d0)
[116653.466833] Stack:
[116653.530557]  ffffffff815f4b45 ffffffff812685f2 ffff88022f97b180
ffff88023700e300
[116653.661317]  ffff880015def170 ffff88023700e180 ffff88022f97b180
ffffffff81051843
[116653.792457]  000000062f97b1a8 ffff88022f97b180 ffff88023700e180
ffff88022f97b1a8
[116653.923833] Call Trace:
[116653.995006]  [<ffffffff815f4b45>] ? _raw_spin_lock+0x5/0x10
[116654.103462]  [<ffffffff812685f2>] ? xlog_state_done_syncing+0x32/0xc0
[116654.221716]  [<ffffffff81051843>] ? process_one_work+0xf3/0x320
[116654.333195]  [<ffffffff810534f2>] ? worker_thread+0xe2/0x280
[116654.441031]  [<ffffffff81053410>] ? gcwq_mayday_timeout+0x80/0x80
[116654.553512]  [<ffffffff8105776b>] ? kthread+0x9b/0xb0
[116654.652866]  [<ffffffff815fbde4>] ? kernel_thread_helper+0x4/0x10
[116654.764568]  [<ffffffff810576d0>] ? kthread_bind+0x80/0x80
[116654.869162]  [<ffffffff815fbde0>] ? gs_change+0x13/0x13
[116654.970752] Code: 13 ff ff ff 48 c7 c2 21 e0 02 81 48 c7 c1 24 e0 02
81 e9 00 ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 31 c0 30 c0
b4 01<f0>
[116655.320953] RIP  [<ffffffff8102e1d6>] __ticket_spin_lock+0x6/0x20
[116655.432251]  RSP<ffff88010a9c7e30>
[116655.511021] CR2: 00000000000010a8
[116655.586770] ---[ end trace 109ecd45394d669c ]---
[116655.678353] BUG: unable to handle kernel paging request at
fffffffffffffff8
[116655.798563] IP: [<ffffffff81057437>] kthread_data+0x7/0x10
[116655.900375] PGD 1876067 PUD 1877067 PMD 0
[116655.984863] Oops: 0000 [#2] SMP
[116656.057855] CPU 0
[116656.081214] Modules linked in: iscsi_scst(O) scst_vdisk(O) scst(O)
drbd(O) twofish_x86_64 twofish_generic
[116656.243502] XFS (dm-47): xfs_log_force: error 5 returned.
[116656.400665]  twofish_common serpent_sse2_x86_64 lrw xts gf128mul
serpent_generic blowfish_x86_64 blowfish_generic blowfish_common cast5
sha512_generic s]
[116656.599500]
[116656.599503] Pid: 30173, comm: kworker/0:5 Tainted: G      D    O
3.4.63-oe64-00000-g1a33902 #38 Intel Corporation S1200BTL/S1200BTL
[116656.599505] RIP: 0010:[<ffffffff81057437>]  [<ffffffff81057437>]
kthread_data+0x7/0x10
[116656.599510] RSP: 0018:ffff88010a9c7a40  EFLAGS: 00010002
[116656.599511] RAX: 0000000000000000 RBX: ffff880233f7f368 RCX:
ffffffff81a21500
[116656.599512] RDX: 000005f8a818d26c RSI: 0000000000000000 RDI:
ffff880233f7f0d0
[116656.599513] RBP: 0000000000000000 R08: 0000000000000001 R09:
ffffea000742d440
[116656.599514] R10: 0000000000000400 R11: ffffffff81068c40 R12:
ffff880233f7f0d0
[116656.599515] R13: ffff880235c90000 R14: ffff880237012180 R15:
0000000000000000
[116656.599516] FS:  0000000000000000(0000) GS:ffff880237000000(0000)
knlGS:0000000000000000
[116656.599517] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[116656.599518] CR2: fffffffffffffff8 CR3: 0000000001874000 CR4:
00000000000407f0
[116656.599519] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[116656.599521] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[116656.599522] Process kworker/0:5 (pid: 30173, threadinfo
ffff88010a9c6000, task ffff880233f7f0d0)
[116656.599523] Stack:
[116656.599523]  ffffffff810530fb ffff880233f7f368 ffff880233f7f0d0
ffff880233f7f0d0
[116656.599525]  ffffffff815f3ee2 ffff88010a9c7ad8 ffff880236810240
ffff8801f7f5c930
[116656.599526]  ffff8801f7f5c900 000000010a9c7ae8 ffffffff810fe66e
ffff880231cf75c0
[116656.599528] Call Trace:
[116656.599530]  [<ffffffff810530fb>] ? wq_worker_sleeping+0xb/0x70
[116656.599533]  [<ffffffff815f3ee2>] ? __schedule+0x372/0x600
[116656.599535]  [<ffffffff810fe66e>] ? d_lookup+0x2e/0x60
[116656.599538]  [<ffffffff8102e255>] ? default_spin_lock_flags+0x5/0x10
[116656.599540]  [<ffffffff810413f1>] ? do_exit+0x5d1/0x8e0
[116656.599542]  [<ffffffff812c24b0>] ? __const_udelay+0x40/0x40
[116656.599544]  [<ffffffff815f4b7b>] ? _raw_spin_lock_irqsave+0x2b/0x50
[116656.599545]  [<ffffffff8103db5e>] ? kmsg_dump+0x6e/0x120
[116656.599547]  [<ffffffff815f5dba>] ? oops_end+0xea/0xf0
[116656.599549]  [<ffffffff81030a94>] ? no_context+0x1c4/0x2d0
[116656.599551]  [<ffffffff815f8371>] ? do_page_fault+0x2b1/0x500
[116656.599552]  [<ffffffff8103e6de>] ? printk+0x4e/0x60
[116656.599555]  [<ffffffff8105fb97>] ? check_preempt_curr+0x57/0x80
[116656.599557]  [<ffffffff8105f21b>] ? __wake_up_common+0x5b/0x90
[116656.599559]  [<ffffffff81227465>] ? xfs_alert_tag+0x75/0xa0
[116656.599561]  [<ffffffff815f51a5>] ? page_fault+0x25/0x30
[116656.599563]  [<ffffffff812c24b0>] ? __const_udelay+0x40/0x40
[116656.599565]  [<ffffffff8102e1d6>] ? __ticket_spin_lock+0x6/0x20
[116656.599566]  [<ffffffff815f4b45>] ? _raw_spin_lock+0x5/0x10
[116656.599568]  [<ffffffff812685f2>] ? xlog_state_done_syncing+0x32/0xc0
[116656.599571]  [<ffffffff81051843>] ? process_one_work+0xf3/0x320
[116656.599572]  [<ffffffff810534f2>] ? worker_thread+0xe2/0x280
[116656.599574]  [<ffffffff81053410>] ? gcwq_mayday_timeout+0x80/0x80
[116656.599575]  [<ffffffff8105776b>] ? kthread+0x9b/0xb0
[116656.599577]  [<ffffffff815fbde4>] ? kernel_thread_helper+0x4/0x10
[116656.599579]  [<ffffffff810576d0>] ? kthread_bind+0x80/0x80
[116656.599581]  [<ffffffff815fbde0>] ? gs_change+0x13/0x13
[116656.599581] Code: fe ff ff 90 65 48 8b 04 25 80 c5 00 00 48 8b 80 40
02 00 00 8b 40 f0 c3 66 66 66 90 66 66 66 90 66 66 66 90 48 8b 87 40 02
00 00<48>
[116656.599592] RIP  [<ffffffff81057437>] kthread_data+0x7/0x10
[116656.599594]  RSP<ffff88010a9c7a40>
[116656.599595] CR2: fffffffffffffff8
[116656.599596] ---[ end trace 109ecd45394d669d ]---
[116656.599597] Fixing recursive fault but reboot is needed!
[116669.563245] XFS (dm-51): xfs_log_force: error 5 returned.

It looks like a race condition.

Test script source:
#!/bin/bash

DEV=$1

if [ -z $DEV ]; then
echo "This program requires device name as parameter" exit 1
fi

function overload()
{
COUNT=$1
temp_COUNT=$COUNT;

while [ -f ./run ]; do

    while [ $COUNT -ge 1 ]; do
        if [ -f ./run ]; then
            dd bs=1024 count=102400 if=/dev/zero of=/$2/"_"$COUNT&>  /dev/null
        fi;
        let COUNT=$COUNT-1
    done;

    rm $2/*;
    COUNT=$temp_COUNT;

done;
}

function create_vg()
{
#create physical volume
pvcreate /dev/sda
if [[ $? -gt 0 ]]; then
echo "[ FAIL ] Unable to create physical volume" exit 1
fi
#create volume group
vgcreate -v -s 32M vg0 /dev/sda
if [[ $? -gt 0 ]]; then
echo "[ FAIL ] Unable to create volume group" exit 1
fi
VG="vg0"
}

function create_lv()

{
    local LV="$1"
    #create logical volume
    lvcreate -l 500 -n "$VG+$LV" "$VG"
    if [[ $? -gt 0 ]]; then
        echo "[ FAIL ] Unable to create LV"
        exit 1
    fi
    mkfs -t xfs -f -l lazy-count=0 /dev/$VG/"$VG+$LV"&>/dev/null
    if [[ $? -gt 0 ]]; then
        echo "[ FAIL ] Can't create filesystem"
        exit 1
    fi
}

function create_snapshots()

{
    for ((i=0; i<  20; i++)); do
        if [[ $i -lt 10 ]]; then
            lvcreate -l "64" -n "snap0$i" "$VG"
            if [[ $? -gt 0 ]]; then
                echo "[ FAIL ] Unable to create snapshot LV"
                exit 1
            fi
        else
            lvcreate -l "64" -n "snap$i" "$VG"
            if [[ $? -gt 0 ]]; then
                echo "[ FAIL ] Unable to create snapshot LV"
                exit 1
            fi
        fi
    done
}

function assign_snapshots()

{
    for ((i=0; i<  20; i++)); do
        if [[ $i -lt 10 ]]; then
            lvrename "$VG" "snap0$i" "lv0+snap0$i"
            if [[ $? -gt 0 ]]; then
                echo "[ FAIL ] Unable to rename snapshot LV"
                exit 1
            fi
        else
            lvrename "$VG" "snap$i" "lv1+snap$i"
            if [[ $? -gt 0 ]]; then
                echo "[ FAIL ] Unable to rename snapshot LV"
                exit 1
            fi
        fi
    done
}

function mount_volume()
{
local MVG=$1
local MLV=$2
mkdir -p "/test/mount/$MVG+$MLV"
if [[ $? -gt 0 ]]; then
echo "[ FAIL ] Unable to create mounting point"
exit 1
fi
mount -t xfs -o defaults,usrquota,grpquota,nouuid,noatime,nodiratime "/dev/$MVG/$MVG+$MLV" "/test/mount/$MVG+$MLV" if [[ $? -gt 0 ]]; then
echo "[ FAIL ] Unable to mount LV"
exit 1
fi
}

function start_overload()

{
    touch ./run
    mkdir -p /test/mount/$1+$2/test
    overload 50 "/test/mount/$1+$2/test" $3&
    echo "overload $1 /test/mount/$1+$2/test $3&"
    sleep 4;
    echo "[ OK ] copying files to $2 started"
}

function get_snapshot_status()

{
    lvdisplay /dev/$1/$2 | awk ' $0~"LV snapshot status" { print $4 } '
}

function remove_snapshot()
{
local LVG=$1
local LLV=$2
local LSNAP=$3
umount "/test/mount/$LSNAP"
if [[ $? -gt 0 ]]; then
echo "[ FAIL ] Can't umount snapshot"
fi
lvremove -sf "/dev/$LVG/$LSNAP"
if [[ $? -gt 0 ]]; then
echo "[ FAIL ] Can't remove snapshot"
fi
}

function create_snapshot()

{
    local LVG=$1
    local LLV=$2
    local LSNAP=$3
    for((it=0; it<7; it++)); do
        local ERROR=0
        local STATUS=`get_snapshot_status $LVG $LSNAP`
        if [[ "$STATUS" == "active" ]]; then
            remove_snapshot $LVG "$LLV+$LSNAP"
        fi
        STATUS=`get_snapshot_status $LVG $LSNAP`
        if [[ "$STATUS" == "active" ]]; then
            remove_snapshot  $LVG "$LLV+$LSNAP"
        fi
        CHUNKSIZE=512
        for ((ile=0;ile<it/2;ile++)); do
            CHUNKSIZE=$((CHUNKSIZE/2))
        done
        lvconvert -s -c $CHUNKSIZE "/dev/$LVG/$LVG+$LLV" "/dev/$LVG/$LSNAP"
        if [[ $? -gt 0 ]]; then
            ERROR=1
        fi
        #mount snapshot
        mkdir -p "/test/mount/$LSNAP"
        mount -t xfs -o nouuid,noatime "/dev/$LVG/$LSNAP" "/test/mount/$LSNAP"
        if [[ $? -gt 0 ]]; then
            ERROR=2
        fi
        create_time=`date "+%Y-%m-%d %H:%M:%S"`
        if [ $ERROR -ne 0 ]; then
            remove_snapshot $LVG $LLV $LSNAP
            sleep 5
        else
            break;
        fi
    done

}

function start_snap()
{
local i;
for((i=0; i<20; i++)); do
echo "Starting snap$i : date"
local START=date +%s
if [[ $i -lt 10 ]]; then
snapname="lv0+snap0"$i
create_snapshot $VG "lv0" $snapname
else
snapname="lv1+snap"$i
create_snapshot $VG "lv1" $snapname
fi
if [ -z "lvs | grep $snapname | grep $VG+lv" ]; then
echo "[ FAIL ] $snapname not activated !!!"
else
echo "[ OK ] $snapname activated."
fi
if [ -z "mount | grep $snapname" ]; then
echo "[ FAIL ] $snapname not mounted !!!">> $LOGFILE
else
echo "[ OK ] $snapname mounted."
fi
local STOP=$[date +%s-$START]
echo "Starting time : $STOP s."
echo "---------------------------"
sleep 2
done;
}

function stop_snap()
{
local i
for((i=0; i<20; i++)); do
echo "Stopping snap$i : date"
local START=date +%s
if [[ $i -lt 10 ]]; then
snapname="lv0+snap0"$i
remove_snapshot $VG "lv0" $snapname
else
snapname="lv1+snap"$i
remove_snapshot $VG "lv1" $snapname
fi
if [ "lvs | grep $snapname | grep $VG+lv" ]; then
echo "[ FAIL ] $snapname still active !!!"
else
echo "[ OK ] $snapname deactivated."
fi;
if [ "mount | grep $snapname" ]; then
echo "[ FAIL ] $snapname still mounted !!!">> $LOGFILE
else
echo "[ OK ] $snapname umounted."
fi;
local STOP=$[date +%s-$START]
echo "Stopping time : $STOP s."
echo "---------------------------"
sleep 2
done;
}

echo "-------- Creating vg0 on $DEV..." create_vg
echo "[ OK ] Volume group created successfully" echo "-------- Creating logical volumes on $VG..." create_lv "lv0"
create_lv "lv1"
echo "[ OK ] Logical volumes created successfully" echo "-------- Mounting logical volumes..." mount_volume "$VG" "lv0"
mount_volume "$VG" "lv1"
echo "[ OK ] Logical volumes mounted successfully" echo "-------- Creating snapshots..." create_snapshots
echo "[ OK ] Snapshots created successfully" echo "-------- Assigning snapshots..." assign_snapshots
echo "[ OK ] Snapshots assigned successfully" echo "-------- Start overload..." start_overload "vg0" "lv0"
start_overload "vg0" "lv1"

while true; do
start_snap 2> /dev/null
stop_snap 2> /dev/null
done

rm ./run

Signature · 11 lines
Best regards
Arkadiusz Bubała
Open-E Poland Sp. z o.o.
www.open-e.com



_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#1 lore raw
Quoted 6 lines · Arkadiusz Bubała

Hello,

we're running test script that starts and stops
snapshots in a loop while overfilling them. After a few days of running system hangs. We've captured following call trace:

Quoted 6 lines
[116649.755761] XFS (dm-42): metadata I/O error: block 0xfa2b06
("xlog_iodone") error 5 buf count 1024
[116649.947247] XFS (dm-42): Log I/O Error Detected.  Shutting down
filesystem
[116650.073881] XFS (dm-42): Please umount the filesystem and rectify

the problem(s)

So, an EIO error on a log IO, resulting in a shutdown....

Quoted 2 lines
[116650.207186] BUG: unable to handle kernel paging request at

00000000000010a8

That's an interesting offset - quite large for a null pointer dereference.

Quoted 10 lines
[116650.335185] IP: [<ffffffff8102e1d6>] __ticket_spin_lock+0x6/0x20
[116650.451052] PGD 0
[116650.518151] Oops: 0002 [#1] SMP
[116650.599477] CPU 0
[116650.622838] Modules linked in: iscsi_scst(O) scst_vdisk(O) scst(O)
drbd(O) twofish_x86_64 twofish_generic twofish_common
serpent_sse2_x86_64 lrw xts gf1]
[116651.479730]
[116651.540674] Pid: 30173, comm: kworker/0:5 Tainted: G           O

3.4.63-oe64-00000-g1a33902 #38 Intel Corporation S1200BTL/S1200BTL

Running a custom built 3.4.63 kernel with a bunch of out of tree modules installed. can you reproduce this on a vanilla 3.12 kernel?

Quoted 7 lines
[116653.923833] Call Trace:
[116653.995006]  [<ffffffff815f4b45>] ? _raw_spin_lock+0x5/0x10
[116654.103462]  [<ffffffff812685f2>] ? xlog_state_done_syncing+0x32/0xc0
[116654.221716]  [<ffffffff81051843>] ? process_one_work+0xf3/0x320
[116654.333195]  [<ffffffff810534f2>] ? worker_thread+0xe2/0x280
[116654.441031]  [<ffffffff81053410>] ? gcwq_mayday_timeout+0x80/0x80
[116654.553512]  [<ffffffff8105776b>] ? kthread+0x9b/0xb0

Which is this line:

STATIC void
xlog_state_done_syncing(
xlog_in_core_t *iclog,
int aborted)

{
        struct xlog        *log = iclog->ic_log;

        spin_lock(&log->l_icloglock);

So, the icloglock is at offset 296 bytes into the struct xlog, and the iclog structure is only 256 bytes in size itself, so that structure offset is way outside anything the code should be trying to access (ignoring the null pointer issue). Even if we assume that the 0x1000 bit is a memory corruption, offset 0xa8 lands in a hole in the struct xlog_in_core, and isin the middle of a bunch of log size constants in the struct xlog (l_sectBBsize to be exact).

So this doesn't make much sense to me.

BTW, you should compile you kernels with frame pointers enabled so that the kernel emits stack traces that can be trusted rather than just dumping a list of symbols found on the stack...

Quoted 1 line

It looks like a race condition.

Looks more like memory corruption to me....

Quoted 1 line

Test script source:

I'll see if I can reproduce it locally.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#2 lore raw
Quoted 69 lines · Dave Chinner
Quoted 6 lines · Arkadiusz Bubała

Hello,

we're running test script that starts and stops
snapshots in a loop while overfilling them. After a few days of running system hangs. We've captured following call trace:

Quoted 6 lines
[116649.755761] XFS (dm-42): metadata I/O error: block 0xfa2b06
("xlog_iodone") error 5 buf count 1024
[116649.947247] XFS (dm-42): Log I/O Error Detected.  Shutting down
filesystem
[116650.073881] XFS (dm-42): Please umount the filesystem and rectify

the problem(s)

So, an EIO error on a log IO, resulting in a shutdown....

Quoted 2 lines
[116650.207186] BUG: unable to handle kernel paging request at

00000000000010a8

That's an interesting offset - quite large for a null pointer dereference.

Quoted 10 lines
[116650.335185] IP: [<ffffffff8102e1d6>] __ticket_spin_lock+0x6/0x20
[116650.451052] PGD 0
[116650.518151] Oops: 0002 [#1] SMP
[116650.599477] CPU 0
[116650.622838] Modules linked in: iscsi_scst(O) scst_vdisk(O) scst(O)
drbd(O) twofish_x86_64 twofish_generic twofish_common
serpent_sse2_x86_64 lrw xts gf1]
[116651.479730]
[116651.540674] Pid: 30173, comm: kworker/0:5 Tainted: G           O

3.4.63-oe64-00000-g1a33902 #38 Intel Corporation S1200BTL/S1200BTL

Running a custom built 3.4.63 kernel with a bunch of out of tree modules installed. can you reproduce this on a vanilla 3.12 kernel?

Quoted 7 lines
[116653.923833] Call Trace:
[116653.995006]  [<ffffffff815f4b45>] ? _raw_spin_lock+0x5/0x10
[116654.103462]  [<ffffffff812685f2>] ? xlog_state_done_syncing+0x32/0xc0
[116654.221716]  [<ffffffff81051843>] ? process_one_work+0xf3/0x320
[116654.333195]  [<ffffffff810534f2>] ? worker_thread+0xe2/0x280
[116654.441031]  [<ffffffff81053410>] ? gcwq_mayday_timeout+0x80/0x80
[116654.553512]  [<ffffffff8105776b>] ? kthread+0x9b/0xb0

Which is this line:

STATIC void
xlog_state_done_syncing(
xlog_in_core_t *iclog,
int aborted)

{
        struct xlog        *log = iclog->ic_log;

        spin_lock(&log->l_icloglock);

So, the icloglock is at offset 296 bytes into the struct xlog, and the iclog structure is only 256 bytes in size itself, so that structure offset is way outside anything the code should be trying to access (ignoring the null pointer issue). Even if we assume that the 0x1000 bit is a memory corruption, offset 0xa8 lands in a hole in the struct xlog_in_core, and isin the middle of a bunch of log size constants in the struct xlog (l_sectBBsize to be exact).

So this doesn't make much sense to me.

BTW, you should compile you kernels with frame pointers enabled so that the kernel emits stack traces that can be trusted rather than just dumping a list of symbols found on the stack...

Quoted 7 lines
Quoted 1 line

It looks like a race condition.

Looks more like memory corruption to me....

Quoted 1 line

Test script source:

I'll see if I can reproduce it locally.

The script is full of bugs, and i don't have time to debug it - it hard codes /dev/sda in places despite taking the device as a CLI parameter. It has hard coded mount points. It sometimes fails to make the filesystem on the base LV after it's been created. start_snap() appears to fail for some reason, as it doesn't result in mounted snapshots. stop_snap fails as well:

Starting snap19 : Thursday 28 November 10:01:26 EST 2013
Logical volume lv1+snap19 converted to snapshot.
[ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ OK ] lv1+snap19 activated.
Starting time : 37 s.
---------------------------
Stopping snap0 : Thursday 28 November 10:02:06 EST 2013 [ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ FAIL ] lv0+snap00 still active !!!
[ OK ] lv0+snap00 umounted.
Stopping time : 0 s.

I've got no idea is this is intended behaviour, but it sure doesn't seem right to me...

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#3 lore raw

Hello,
thank you for valuable information.

Quoted 5 lines · Dave Chinner
Quoted 5 lines

Running a custom built 3.4.63 kernel with a bunch of out of tree modules installed. can you reproduce this on a vanilla 3.12 kernel?

Ok, we'll try.

Quoted 33 lines

The script is full of bugs, and i don't have time to debug it - it hard codes /dev/sda in places despite taking the device as a CLI parameter. It has hard coded mount points. It sometimes fails to make the filesystem on the base LV after it's been created. start_snap() appears to fail for some reason, as it doesn't result in mounted snapshots. stop_snap fails as well:

Starting snap19 : Thursday 28 November 10:01:26 EST 2013
Logical volume lv1+snap19 converted to snapshot.
[ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ OK ] lv1+snap19 activated.
Starting time : 37 s.
---------------------------
Stopping snap0 : Thursday 28 November 10:02:06 EST 2013 [ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ FAIL ] lv0+snap00 still active !!!
[ OK ] lv0+snap00 umounted.
Stopping time : 0 s.

Quoted 5 lines

I've got no idea is this is intended behaviour, but it sure doesn't seem right to me...

Yes, sometimes umount and remove operations fail. This script tests system stability and these messages are debug info only. I've fixed it. Now it takes two parameters: device and mount point.

#!/bin/bash

DEV=$1
MOUNTPOINT=$2

function print_usage()
{
echo "Usage: $0 device mountpoint"
exit 1;
}

if [ -z "$DEV" ]; then
print_usage
fi

if [ -z "$MOUNTPOINT" ]; then
print_usage
fi

function overload()
{
COUNT=$1
temp_COUNT=$COUNT;

while [ -f ./run ]; do

    while [ $COUNT -ge 1 ]; do
        if [ -f ./run ]; then
            dd bs=1024 count=102400 if=/dev/zero of=/$2/"_"$COUNT &> 
/dev/null
        fi;
        let COUNT=$COUNT-1
    done;

    rm $2/*;
    COUNT=$temp_COUNT;

done;
}

function create_vg()
{
#create physical volume
pvcreate $DEV
if [[ $? -gt 0 ]]; then
echo "[ FAIL ] Unable to create physical volume" exit 1
fi
#create volume group
vgcreate -v -s 32M $VG $DEV
if [[ $? -gt 0 ]]; then
echo "[ FAIL ] Unable to create volume group" exit 1
fi
}

function create_lv()

{
    local LV="$1"
    #create logical volume
    lvcreate -l 500 -n "$VG+$LV" "$VG"
    if [[ $? -gt 0 ]]; then
        echo "[ FAIL ] Unable to create LV"
        exit 1
    fi
    mkfs -t xfs -f -l lazy-count=0 /dev/$VG/"$VG+$LV" &>/dev/null
    if [[ $? -gt 0 ]]; then
        echo "[ FAIL ] Can't create filesystem"
        exit 1
    fi
}

function create_snapshots()

{
    for ((i=0; i < 20; i++)); do
        if [[ $i -lt 10 ]]; then
            lvcreate -l "64" -n "snap0$i" "$VG"
            if [[ $? -gt 0 ]]; then
                echo "[ FAIL ] Unable to create snapshot LV"
                exit 1
            fi
        else
            lvcreate -l "64" -n "snap$i" "$VG"
            if [[ $? -gt 0 ]]; then
                echo "[ FAIL ] Unable to create snapshot LV"
                exit 1
            fi
        fi
    done
}

function assign_snapshots()

{
    for ((i=0; i < 20; i++)); do
        if [[ $i -lt 10 ]]; then
            lvrename "$VG" "snap0$i" "lv0+snap0$i"
            if [[ $? -gt 0 ]]; then
                echo "[ FAIL ] Unable to rename snapshot LV"
                exit 1
            fi
        else
            lvrename "$VG" "snap$i" "lv1+snap$i"
            if [[ $? -gt 0 ]]; then
                echo "[ FAIL ] Unable to rename snapshot LV"
                exit 1
            fi
        fi
    done
}

function mount_volume()
{
local MVG=$1
local MLV=$2
mkdir -p "$MOUNTPOINT/$MVG+$MLV"
if [[ $? -gt 0 ]]; then
echo "[ FAIL ] Unable to create mounting point"
exit 1
fi
mount -t xfs -o
defaults,usrquota,grpquota,nouuid,noatime,nodiratime "/dev/$MVG/$MVG+$MLV" "$MOUNTPOINT/$MVG+$MLV"
if [[ $? -gt 0 ]]; then
echo "[ FAIL ] Unable to mount LV"
exit 1
fi
}

function start_overload()

{
    touch ./run
    mkdir -p "$MOUNTPOINT/$1+$2/test"
    overload 50 "$MOUNTPOINT/$1+$2/test" $3 &
    echo "overload $1 $MOUNTPOINT/$1+$2/test $3 &"
    sleep 4;
    echo "[ OK ] copying files to $2 started"
}

function get_snapshot_status()

{
    lvdisplay /dev/$1/$2 | awk ' $0~"LV snapshot status" { print $4 } '
}

function remove_snapshot()
{
local LVG=$1
local LLV=$2
local LSNAP=$3
umount "$MOUNTPOINT/$LSNAP"
if [[ $? -gt 0 ]]; then
echo "[ FAIL ] Can't umount snapshot"
fi
lvremove -sf "/dev/$LVG/$LSNAP"
if [[ $? -gt 0 ]]; then
echo "[ FAIL ] Can't remove snapshot"
fi
}

function create_snapshot()

{
    local LVG=$1
    local LLV=$2
    local LSNAP=$3
    for((it=0; it<7; it++)); do
        local ERROR=0
        local STATUS=`get_snapshot_status $LVG $LSNAP`
        if [[ "$STATUS" == "active" ]]; then
            remove_snapshot $LVG "$LLV+$LSNAP"
        fi
        STATUS=`get_snapshot_status $LVG $LSNAP`
        if [[ "$STATUS" == "active" ]]; then
            remove_snapshot  $LVG "$LLV+$LSNAP"
        fi
        CHUNKSIZE=512
        for ((ile=0;ile<it/2;ile++)); do
            CHUNKSIZE=$((CHUNKSIZE/2))
        done
        lvconvert -s -c $CHUNKSIZE "/dev/$LVG/$LVG+$LLV" "/dev/$LVG/$LSNAP"
        if [[ $? -gt 0 ]]; then
            ERROR=1
        fi
        #mount snapshot
        mkdir -p "$MOUNTPOINT/$LSNAP"
        mount -t xfs -o nouuid,noatime "/dev/$LVG/$LSNAP" 
"$MOUNTPOINT/$LSNAP"
        if [[ $? -gt 0 ]]; then
            ERROR=2
        fi
        create_time=`date "+%Y-%m-%d %H:%M:%S"`
        if [ $ERROR -ne 0 ]; then
            remove_snapshot $LVG $LLV $LSNAP
            sleep 5
        else
            break;
        fi
    done

}

function start_snap()
{
local i;
for((i=0; i<20; i++)); do
echo "Starting snap$i : date"
local START=date +%s
if [[ $i -lt 10 ]]; then
snapname="lv0+snap0"$i
create_snapshot $VG "lv0" $snapname
else
snapname="lv1+snap"$i
create_snapshot $VG "lv1" $snapname
fi
if [ -z "lvs | grep $snapname | grep $VG+lv" ]; then
echo "[ FAIL ] $snapname not activated !!!"
else
echo "[ OK ] $snapname activated."
fi
if [ -z "mount | grep $snapname" ]; then
echo "[ FAIL ] $snapname not mounted !!!" >> $LOGFILE
else
echo "[ OK ] $snapname mounted."
fi
local STOP=$[date +%s-$START]
echo "Starting time : $STOP s."
echo "---------------------------"
sleep 2
done;
}

function stop_snap()
{
local i
for((i=0; i<20; i++)); do
echo "Stopping snap$i : date"
local START=date +%s
if [[ $i -lt 10 ]]; then
snapname="lv0+snap0"$i
remove_snapshot $VG "lv0" $snapname
else
snapname="lv1+snap"$i
remove_snapshot $VG "lv1" $snapname
fi
if [ "lvs | grep $snapname | grep $VG+lv" ]; then
echo "[ FAIL ] $snapname still active !!!"
else
echo "[ OK ] $snapname deactivated."
fi;
if [ "mount | grep $snapname" ]; then
echo "[ FAIL ] $snapname still mounted !!!" >> $LOGFILE
else
echo "[ OK ] $snapname umounted."
fi;
local STOP=$[date +%s-$START]
echo "Stopping time : $STOP s."
echo "---------------------------"
sleep 2
done;
}

VG="vg0"

echo "-------- Creating $VG on $DEV..." create_vg
echo "[ OK ] Volume group created successfully" echo "-------- Creating logical volumes on $VG..." create_lv "lv0"
create_lv "lv1"
echo "[ OK ] Logical volumes created successfully" echo "-------- Mounting logical volumes..." mount_volume "$VG" "lv0"
mount_volume "$VG" "lv1"
echo "[ OK ] Logical volumes mounted successfully" echo "-------- Creating snapshots..." create_snapshots
echo "[ OK ] Snapshots created successfully" echo "-------- Assigning snapshots..." assign_snapshots
echo "[ OK ] Snapshots assigned successfully" echo "-------- Start overload..." start_overload "$VG" "lv0"
start_overload "$VG" "lv1"

while true; do
start_snap 2> /dev/null
stop_snap 2> /dev/null
done

rm ./run

Signature · 9 lines
Best regards
Arkadiusz Bubała
Open-E Poland Sp. z o.o.
www.open-e.com

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#4 lore raw
Quoted 43 lines · Arkadiusz Bubała

Hello,
thank you for valuable information.

Quoted 4 lines · Dave Chinner
Quoted 4 lines

Running a custom built 3.4.63 kernel with a bunch of out of tree modules installed. can you reproduce this on a vanilla 3.12 kernel?

Ok, we'll try.

Quoted 33 lines

The script is full of bugs, and i don't have time to debug it - it hard codes /dev/sda in places despite taking the device as a CLI parameter. It has hard coded mount points. It sometimes fails to make the filesystem on the base LV after it's been created. start_snap() appears to fail for some reason, as it doesn't result in mounted snapshots. stop_snap fails as well:

Starting snap19 : Thursday 28 November 10:01:26 EST 2013
Logical volume lv1+snap19 converted to snapshot.
[ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ OK ] lv1+snap19 activated.
Starting time : 37 s.
---------------------------
Stopping snap0 : Thursday 28 November 10:02:06 EST 2013 [ FAIL ] Can't umount snapshot
[ FAIL ] Can't remove snapshot
[ FAIL ] lv0+snap00 still active !!!
[ OK ] lv0+snap00 umounted.
Stopping time : 0 s.

Quoted 5 lines
Quoted 4 lines

I've got no idea is this is intended behaviour, but it sure doesn't seem right to me...

Yes, sometimes umount and remove operations fail.

They always fail here - snapshots are not being mounted at all (nothing in dmesg about XFS filesystems being mounted during the test at all), so the test does not appear to be doing what you expect to be doing...

Quoted 2 lines

This script tests
system stability and these messages are debug info only.

It ran overnight on a TOT 3.13-rc1 kernel with memory leak and poisoning turned on, issuing those fail messages and nothing broke...

Quoted 1 line

I've fixed it. Now it takes two parameters: device and mount point.

I'll try it again...

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#5 lore raw

Hello,

I tried to repeat issue on another environment and but after about 15h of running mentioned test I got a deadlock. Test ran on the latest mainline 3.4.71 kernel. Here's the dumped stack:

[74017.711401] dd              D ffffffff8160a4a0     0   546   3277 0x20020000
[74017.711404]  ffff88018ca40000 0000000000000086 0000000300000000 ffff8801f751fad8
[74017.711406]  ffffffffffffffa6 ffff88042cd66ae0 0000000000000000 0000000000102400
[74017.711408]  0000000000000200 0000000000000000 0000000000000200 0000000000102652
[74017.711410] Call Trace:
[74017.711411]  [<ffffffff81221133>] ? xfs_iunlock+0x63/0x80
[74017.711413]  [<ffffffff810a9728>] ? find_get_page+0x18/0x90
[74017.711414]  [<ffffffff810a9880>] ? __lock_page+0x70/0x70
[74017.711416]  [<ffffffff815f530d>] ? io_schedule+0x4d/0x70
[74017.711417]  [<ffffffff810a9889>] ? sleep_on_page+0x9/0x10
[74017.711419]  [<ffffffff815f398f>] ? __wait_on_bit+0x4f/0x80
[74017.711420]  [<ffffffff81219d40>] ? xfs_get_blocks_direct+0x10/0x10
[74017.711422]  [<ffffffff810a9b9c>] ? wait_on_page_bit+0x6c/0x80
[74017.711424]  [<ffffffff81057e50>] ? autoremove_wake_function+0x30/0x30
[74017.711425]  [<ffffffff810a9fb0>] ? grab_cache_page_write_begin+0xc0/0x100
[74017.711427]  [<ffffffff81219d40>] ? xfs_get_blocks_direct+0x10/0x10
[74017.711430]  [<ffffffff8111563e>] ? block_write_begin+0x3e/0xa0
[74017.711431]  [<ffffffff81219830>] ? xfs_vm_write_begin+0x40/0x80
[74017.711433]  [<ffffffff810a8fff>] ? generic_file_buffered_write+0x10f/0x270
[74017.711435]  [<ffffffff8121eb74>] ? xfs_file_aio_write_checks+0xf4/0x140
[74017.711437]  [<ffffffff8121ec83>] ? xfs_file_buffered_aio_write+0xc3/0x140
[74017.711439]  [<ffffffff8121f084>] ? xfs_file_aio_write+0xf4/0x150
[74017.711440]  [<ffffffff810fa44c>] ? do_filp_open+0x4c/0xc0
[74017.711442]  [<ffffffff810ea594>] ? do_sync_write+0xc4/0x100
[74017.711444]  [<ffffffff81121cd2>] ? fsnotify+0x1b2/0x2c0
[74017.711446]  [<ffffffff8128b0ac>] ? security_file_permission+0x1c/0x90
[74017.711447]  [<ffffffff810ead90>] ? vfs_write+0xd0/0x170
[74017.711449]  [<ffffffff810eb543>] ? sys_write+0x53/0x90
[74017.711451]  [<ffffffff815fd399>] ? ia32_do_call+0x13/0x13
[74017.711452] lvconvert       D ffffffff8160a4a0     0  1857   1612 0x20020000
[74017.711455]  ffff88041fd08be0 0000000000000082 ffff880252aeda18 0000000000000001
[74017.711457]  ffffffff810be0ed ffff88042cd66ae0 ffffea000b5d7de0 0000000000000040
[74017.711458]  0000000000000202 0000000000000000 ffff880252aedfd8 0000004000000000
[74017.711460] Call Trace:
[74017.711462]  [<ffffffff810be0ed>] ? zone_statistics+0x9d/0xa0
[74017.711465]  [<ffffffff810d13cc>] ? alloc_vmap_area+0x6c/0x310
[74017.711467]  [<ffffffff815f58cd>] ? rwsem_down_failed_common+0xcd/0x140
[74017.711469]  [<ffffffff810b0222>] ? __alloc_pages_nodemask+0x152/0x720
[74017.711471]  [<ffffffff812c3c33>] ? call_rwsem_down_write_failed+0x13/0x20
[74017.711473]  [<ffffffff812889e0>] ? cap_netlink_send+0x10/0x10
[74017.711475]  [<ffffffff815f426c>] ? down_write+0x1c/0x20
[74017.711477]  [<ffffffff810ecbd9>] ? grab_super+0x29/0x90
[74017.711478]  [<ffffffff810ee4d1>] ? get_active_super+0x71/0x90
[74017.711480]  [<ffffffff8111b01b>] ? freeze_bdev+0x7b/0xd0
[74017.711482]  [<ffffffff814b3450>] ? dm_suspend+0x1e0/0x280
[74017.711484]  [<ffffffff814b6d9c>] ? __find_device_hash_cell+0xdc/0x150
[74017.711486]  [<ffffffff814b7ca0>] ? dev_wait+0xb0/0xb0
[74017.711488]  [<ffffffff814b7e6c>] ? dev_suspend+0x1cc/0x220
[74017.711490]  [<ffffffff814b8b23>] ? dm_ctl_ioctl+0x303/0x3a0
[74017.711492]  [<ffffffff815f5995>] ? _raw_spin_lock+0x5/0x10
[74017.711494]  [<ffffffff81105bc5>] ? vfsmount_lock_local_lock+0x15/0x20
[74017.711497]  [<ffffffff81130b94>] ? compat_sys_ioctl+0x104/0x1160
[74017.711499]  [<ffffffff81105bc5>] ? vfsmount_lock_local_lock+0x15/0x20
[74017.711501]  [<ffffffff8110713a>] ? mntput_no_expire+0x1a/0x140
[74017.711503]  [<ffffffff81105be5>] ? vfsmount_lock_local_unlock+0x15/0x20
[74017.711505]  [<ffffffff810ef5db>] ? vfs_fstatat+0x7b/0x80
[74017.711507]  [<ffffffff810389ce>] ? sys32_stat64+0x2e/0x50
[74017.711509]  [<ffffffff815fd399>] ? ia32_do_call+0x13/0x13
[74017.711511] kworker/4:2     D 0000000000000000     0 45754      2 0x00000000
[74017.711513]  ffff88020f12f0d0 0000000000000046 0000000000000001 0000000000000400
[74017.711516]  0000000000000000 ffff8801f9c04d30 0000000000000400 0000000040000000
[74017.711518]  0000000000000001 000000000003cb84 ffff8804275b8780 ffff88020f12f0d0
[74017.711519] Call Trace:
[74017.711521]  [<ffffffff8121c660>] ? xfs_buf_ioend+0x90/0x90
[74017.711523]  [<ffffffff8121c6bb>] ? xfs_buf_iorequest+0x3b/0x40
[74017.711524]  [<ffffffff81267476>] ? xlog_bdstrat+0x36/0x40
[74017.711526]  [<ffffffff8102e255>] ? default_spin_lock_flags+0x5/0x10
[74017.711528]  [<ffffffff815f59cb>] ? _raw_spin_lock_irqsave+0x2b/0x50
[74017.711530]  [<ffffffff812692e1>] ? _xfs_log_force_lsn+0x2d1/0x310
[74017.711531]  [<ffffffff810630f0>] ? try_to_wake_up+0x280/0x280
[74017.711534]  [<ffffffff81266cc5>] ? xfs_trans_commit+0x245/0x250
[74017.711535]  [<ffffffff8122a574>] ? xfs_sync_worker+0x54/0x60
[74017.711537]  [<ffffffff81051843>] ? process_one_work+0xf3/0x320
[74017.711539]  [<ffffffff810534f2>] ? worker_thread+0xe2/0x280
[74017.711541]  [<ffffffff81053410>] ? gcwq_mayday_timeout+0x80/0x80
[74017.711542]  [<ffffffff8105776b>] ? kthread+0x9b/0xb0
[74017.711544]  [<ffffffff815fcc24>] ? kernel_thread_helper+0x4/0x10
[74017.711546]  [<ffffffff810576d0>] ? kthread_bind+0x80/0x80
[74017.711548]  [<ffffffff815fcc20>] ? gs_change+0x13/0x13
Signature · 9 lines
Best regards
Arkadiusz Bubała
Open-E Poland Sp. z o.o.
www.open-e.com

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs