Skip to contentSkip to conversation

Xfs Access to block zero exception and system crash

DISCUSSION Sagar Borikar 1 participant 3 messages Jul 8, 2008 – Jul 10, 2008 2 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

Sagar Borikar

author
#2 lore raw

Sure Eric, I'll keep you posted with the results w/o loop back file. When you say that the deadlock could be due to vm, is it due to lack of memory? I checked meminfo and I found that sufficient buffers and committed_as were persent when xfs is stalled.

Thanks
Sagar

Quoted 4 lines · Sagar Borikar

That's right Eric but I am still surprised that why should we get a dead lock in this scenario as it is a plain copy of file in multiple directories. Our customer is reporting similar kind of lockup in our platform.

ok, I guess I had missed that, sorry.

Quoted 5 lines

I do understand that we are chasing the access to block zero exception

and XFS forced shutdown which I mentioned earlier. But we also see quite a few smbd processes which are writing data to XFS are in uninterruptible sleep state and the system locks up too.

Ok; then the next step is probably to do sysrq-t and see where things are stuck. It might be better to see if you can reproduce w/o the loopback file, too, since that's just another layer to go through that might be changing things.

Quoted 4 lines

So I thought
the test which I am running could be pointing to similar issue which we are observing on our platform. But does this indicate that the problem lies with x86 XFS too ?

or maybe the vm ...

Quoted 4 lines

Also I presume in enterprise market such kind of simultaneous write situation may happen. Has anybody reported similar issues to you? As you observed it over x86 and 2.6.24 kernel, could you say what would be root cause of this?

Haven't really seen it before that I recall, and at this point can't say for sure what it might be.

-Eric

Quoted 71 lines

Sorry for lots of questions at same time :) But I am happy that
you were able to see the deadlock in x86 on your setup with 2.6.24

Thanks
Sagar

Quoted 62 lines · Eric Sandeen
Quoted 4 lines · Sagar Borikar

Hi Eric,

Did you see any issues in your test?

I got a deadlock but that's it; I don't think that's the bug you want

to chase...

-Eric

Quoted 47 lines

Thanks
Sagar

Quoted 41 lines · Sagar Borikar
Quoted 11 lines · Eric Sandeen
Quoted 6 lines · Sagar Borikar

Could you kindly try with my test? I presume you should see failure soon. I tried this on
2 different x86 systems 2 times ( after rebooting the system ) and I saw it every time.

Quoted 5 lines

Sure. Is there a reason you're doing this on a loopback file? That probably stresses the vm a bit more, and might get even trickier if the loopback file is sparse...

Initially I thought to do that since I didn't want to have a strict

allocation limit but allowing allocations to grow as needed until the backing filesystem runs out of free space due to type of the test case I had. But then I dropped the plan and created a non-sparse loopback device. There was no specific reason to create loopback but as it was simplest option to do it.

Quoted 8 lines

But anyway, on an x86_64 machine with 2G of memory and a non-sparse 10G loopback file on 2.6.24.7-92.fc8, your test runs w/o problems for me, though the system does get sluggish. I let it run a bit then ran repair and it found no problems, I'll run it

overnight to see if anything else turns up.

That will be great. Thanks indeed.
Sagar

Quoted 3 lines

-Eric

author
#3 lore raw
In reply to a message not in the archive
Quoted 1 line · Sagar Borikar

That's right Eric but I am still surprised that why should we get a

dead

Quoted 3 lines

lock in this scenario as it is a plain copy of file in multiple directories. Our customer is reporting similar kind of lockup in our platform.

ok, I guess I had missed that, sorry.

Quoted 3 lines

I do understand that we are chasing the access to block zero exception and XFS forced shutdown which I mentioned earlier. But we also see quite a few smbd processes which are writing data to XFS are

in

Quoted 1 line

uninterruptible sleep state and the system locks up too.

Ok; then the next step is probably to do sysrq-t and see where things are stuck. It might be better to see if you can reproduce w/o the loopback file, too, since that's just another layer to go through that might be changing things.

<Sagar> I ran it on actual device w/o loopback file and even there observed that XFS transactions going into uninterruptible sleep state and the copies were stalled. I had to hard reboot the system to bring XFS out of that state since soft reboot didn't work, it was waiting for file system to get unmounted. I shall provide the sysrq-t update later.

Quoted 2 lines

So I thought
the test which I am running could be pointing to similar issue which

we

Quoted 3 lines

are observing on our platform. But does this indicate that the problem

lies with x86 XFS too ?

or maybe the vm ...

Quoted 4 lines

Also I presume in enterprise market such kind
of simultaneous write situation may happen. Has anybody reported similar issues to you? As you observed it over x86 and 2.6.24 kernel, could you say what would be root cause of this?

Haven't really seen it before that I recall, and at this point can't say for sure what it might be.

-Eric

Quoted 1 line

Sorry for lots of questions at same time :) But I am happy that

you

Quoted 14 lines

were able to see the deadlock in x86 on your setup with 2.6.24

Thanks
Sagar

Quoted 7 lines · Eric Sandeen
Quoted 4 lines · Sagar Borikar

Hi Eric,

Did you see any issues in your test?

I got a deadlock but that's it; I don't think that's the bug you want

to

Quoted 12 lines
Quoted 12 lines

chase...

-Eric

Quoted 6 lines

Thanks
Sagar

Sagar Borikar wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines · Eric Sandeen
Quoted 1 line · Sagar Borikar

Could you kindly try with my test? I presume you should see

failure

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

soon. I tried this on
2 different x86 systems 2 times ( after rebooting the system )

and I

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

saw it every time.

Sure. Is there a reason you're doing this on a loopback file?

That

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

probably stresses the vm a bit more, and might get even trickier

if the

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

loopback file is sparse...

Initially I thought to do that since I didn't want to have a strict

allocation limit but
allowing allocations to grow as needed until the backing

filesystem

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

runs out of free space
due to type of the test case I had. But then I dropped the plan and

created a non-sparse
loopback device. There was no specific reason to create loopback

but

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

as it was
simplest option to do it.

Quoted 1 line

But anyway, on an x86_64 machine with 2G of memory and a

non-sparse 10G

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

loopback file on 2.6.24.7-92.fc8, your test runs w/o problems for

me,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

though the system does get sluggish. I let it run a bit then ran

repair

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

and it found no problems, I'll run it overnight to see if anything

else

Quoted 12 lines
Quoted 11 lines
Quoted 10 lines
Quoted 10 lines
Quoted 3 lines

turns up.

That will be great. Thanks indeed.
Sagar

Quoted 3 lines

-Eric

author
#4 lore raw

Eric,

Could be a slight digression but can you let me know why the fragmentation factor is going to 99% immediately? I observed this on both x86 and MIPS platform. Also to alleviate this issue, if I specify allocsize=512m what would be the consequences? Since default allocsize is 64k right? Also while mounting we are setting up default option for mounting file system.

Thanks
Sagar

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Sagar Borikar
Sent: Wednesday, July 09, 2008 10:28 PM
To: Eric Sandeen
Cc: [email protected]
Subject: RE: Xfs Access to block zero exception and system crash

Quoted 1 line · Sagar Borikar

That's right Eric but I am still surprised that why should we get a

dead

Quoted 3 lines

lock in this scenario as it is a plain copy of file in multiple directories. Our customer is reporting similar kind of lockup in our platform.

ok, I guess I had missed that, sorry.

Quoted 3 lines

I do understand that we are chasing the access to block zero exception and XFS forced shutdown which I mentioned earlier. But we also see quite a few smbd processes which are writing data to XFS are

in

Quoted 1 line

uninterruptible sleep state and the system locks up too.

Ok; then the next step is probably to do sysrq-t and see where things are stuck. It might be better to see if you can reproduce w/o the loopback file, too, since that's just another layer to go through that might be changing things.

<Sagar> I ran it on actual device w/o loopback file and even there observed that XFS transactions going into uninterruptible sleep state and the copies were stalled. I had to hard reboot the system to bring XFS out of that state since soft reboot didn't work, it was waiting for file system to get unmounted. I shall provide the sysrq-t update later.

Quoted 2 lines

So I thought
the test which I am running could be pointing to similar issue which

we

Quoted 3 lines

are observing on our platform. But does this indicate that the problem

lies with x86 XFS too ?

or maybe the vm ...

Quoted 4 lines

Also I presume in enterprise market such kind
of simultaneous write situation may happen. Has anybody reported similar issues to you? As you observed it over x86 and 2.6.24 kernel, could you say what would be root cause of this?

Haven't really seen it before that I recall, and at this point can't say for sure what it might be.

-Eric

Quoted 1 line

Sorry for lots of questions at same time :) But I am happy that

you

Quoted 14 lines

were able to see the deadlock in x86 on your setup with 2.6.24

Thanks
Sagar

Quoted 7 lines · Eric Sandeen
Quoted 4 lines · Sagar Borikar

Hi Eric,

Did you see any issues in your test?

I got a deadlock but that's it; I don't think that's the bug you want

to

Quoted 12 lines
Quoted 12 lines

chase...

-Eric

Quoted 6 lines

Thanks
Sagar

Sagar Borikar wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines · Eric Sandeen
Quoted 1 line · Sagar Borikar

Could you kindly try with my test? I presume you should see

failure

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

soon. I tried this on
2 different x86 systems 2 times ( after rebooting the system )

and I

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

saw it every time.

Sure. Is there a reason you're doing this on a loopback file?

That

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

probably stresses the vm a bit more, and might get even trickier

if the

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

loopback file is sparse...

Initially I thought to do that since I didn't want to have a strict

allocation limit but
allowing allocations to grow as needed until the backing

filesystem

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

runs out of free space
due to type of the test case I had. But then I dropped the plan and

created a non-sparse
loopback device. There was no specific reason to create loopback

but

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

as it was
simplest option to do it.

Quoted 1 line

But anyway, on an x86_64 machine with 2G of memory and a

non-sparse 10G

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

loopback file on 2.6.24.7-92.fc8, your test runs w/o problems for

me,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

though the system does get sluggish. I let it run a bit then ran

repair

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

and it found no problems, I'll run it overnight to see if anything

else

Quoted 12 lines
Quoted 11 lines
Quoted 10 lines
Quoted 10 lines
Quoted 3 lines

turns up.

That will be great. Thanks indeed.
Sagar

Quoted 3 lines

-Eric