Skip to contentSkip to conversation

bo problem, file-max limit reached within two days

DISCUSSION Knut Petersen 3 participants 6 messages Apr 2, 2012 – Apr 12, 2012

lore
Share & tools
lore
#1 lore raw
In reply to Knut Petersen
Quoted 7 lines · Knut Petersen

After a few unattended hours Xorg was still running, but only a terminal window had survived the night :-(

Xorg: git, a few days old, kernel: 3.3.

Xorg.0.log: Nothing unusual

Quoted 8 lines

dmesg:

[156859.078080] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[179417.374016] VFS: file-max limit 204863 reached

I built a new X server and tried kernel 3.2.12 - that does not seem to help as both the number of open files and the number of gem objects still grow without obvious reasons.

It's the EFILE that is truly worrying. We have a patch in the queue to help ease the ENOSPC issue, but the EFILE implies a bo reference leak. And that I have not found yet.

Happy hunting,
-Chris

Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
#0 lore raw

After a few unattended hours Xorg was still running, but only a terminal window had survived the night :-(

Xorg: git, a few days old, kernel: 3.3.

Xorg.0.log: Nothing unusual

dmesg:

[156859.078080] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[156869.197310] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[156870.008424] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[156871.268130] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[156872.988325] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[156879.187113] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[156880.046793] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[156898.079853] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[170448.897871] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[170449.982378] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[170450.399949] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[170451.999312] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[170458.007902] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[170458.795343] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[170756.249755] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[170756.572389] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[170769.156889] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[170769.392065] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[170770.703797] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[170772.407662] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[170779.460247] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[170788.713283] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[170795.466570] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[170796.491909] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[170798.477484] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[170804.722188] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[170805.534740] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[179417.374016] VFS: file-max limit 204863 reached
[179418.373995] VFS: file-max limit 204863 reached
[179419.373969] VFS: file-max limit 204863 reached
[179420.373982] VFS: file-max limit 204863 reached
[179422.373995] VFS: file-max limit 204863 reached
[179423.374002] VFS: file-max limit 204863 reached
[179425.373970] VFS: file-max limit 204863 reached
[179427.404335] VFS: file-max limit 204863 reached
[179428.373994] VFS: file-max limit 204863 reached
[179429.373992] VFS: file-max limit 204863 reached
[179430.381375] VFS: file-max limit 204863 reached
[179431.374017] VFS: file-max limit 204863 reached
[179432.380678] VFS: file-max limit 204863 reached
[179433.373972] VFS: file-max limit 204863 reached
[179434.385198] VFS: file-max limit 204863 reached
[179435.374010] VFS: file-max limit 204863 reached
[179437.373992] VFS: file-max limit 204863 reached
[179438.373992] VFS: file-max limit 204863 reached
[179440.374007] VFS: file-max limit 204863 reached
[179442.420661] VFS: file-max limit 204863 reached
[179443.373989] VFS: file-max limit 204863 reached
[179444.373998] VFS: file-max limit 204863 reached
[179445.394205] VFS: file-max limit 204863 reached
[179446.373974] VFS: file-max limit 204863 reached
[179447.395051] VFS: file-max limit 204863 reached
[179448.373964] VFS: file-max limit 204863 reached
[179449.402791] VFS: file-max limit 204863 reached
[179450.373994] VFS: file-max limit 204863 reached
[179452.374004] VFS: file-max limit 204863 reached
[179453.374010] VFS: file-max limit 204863 reached
[179455.374067] VFS: file-max limit 204863 reached
[179455.797582] VFS: file-max limit 204863 reached
[179472.450203] VFS: file-max limit 204863 reached
[179473.373970] VFS: file-max limit 204863 reached
[179474.374087] VFS: file-max limit 204863 reached
[179475.424852] VFS: file-max limit 204863 reached
[179476.401695] VFS: file-max limit 204863 reached
[179477.427067] VFS: file-max limit 204863 reached
[179478.398170] VFS: file-max limit 204863 reached
[179479.425794] VFS: file-max limit 204863 reached
[179480.373985] VFS: file-max limit 204863 reached
[179482.374007] VFS: file-max limit 204863 reached
[179483.373988] VFS: file-max limit 204863 reached
[179485.373987] VFS: file-max limit 204863 reached

I built a new X server and tried kernel 3.2.12 - that does not seem to help as both the number of open files and the number of gem objects still grow without obvious reasons.

Any idea?

cu,
Knut

#2 lore raw
In reply to Chris Wilson

Am 02.04.2012 10:44, schrieb Chris Wilson:

Quoted 16 lines · Knut Petersen
Quoted 7 lines

After a few unattended hours Xorg was still running, but only a terminal window had survived the night :-(

Xorg: git, a few days old, kernel: 3.3.

Xorg.0.log: Nothing unusual

Quoted 8 lines

dmesg:

[156859.078080] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[179417.374016] VFS: file-max limit 204863 reached

I built a new X server and tried kernel 3.2.12 - that does not seem to help as both the number of open files and the number of gem objects still grow without obvious reasons.

Quoted 7 lines

It's the EFILE that is truly worrying. We have a patch in the queue to help ease the ENOSPC issue, but the EFILE implies a bo reference leak. And that I have not found yet.

Happy hunting,
-Chris

Well, it takes more than two days to trigger the EFILE limit here, but yesterday the ENOSPC bo problem appeared 6.5 hours after booting the system with current Xorg git and kernel 3.3.1. I wonder if this is one problem or if we face two independent bugs.

cu,
Knut

#3 lore raw
Quoted 21 lines · Knut Petersen

Am 02.04.2012 10:44, schrieb Chris Wilson:

Quoted 20 lines · Knut Petersen
Quoted 7 lines

After a few unattended hours Xorg was still running, but only a terminal window had survived the night :-(

Xorg: git, a few days old, kernel: 3.3.

Xorg.0.log: Nothing unusual

Quoted 8 lines

dmesg:

[156859.078080] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[179417.374016] VFS: file-max limit 204863 reached

I built a new X server and tried kernel 3.2.12 - that does not seem to help as both the number of open files and the number of gem objects still grow without obvious reasons.

It's the EFILE that is truly worrying. We have a patch in the queue to help ease the ENOSPC issue, but the EFILE implies a bo reference leak. And that I have not found yet.

Quoted 7 lines
Quoted 3 lines

Happy hunting,
-Chris

Well, it takes more than two days to trigger the EFILE limit here, but yesterday the ENOSPC bo problem appeared 6.5 hours after booting the system with current Xorg git and kernel 3.3.1. I wonder if this is one problem or if we face two independent bugs.

Can you try the drm-intel-next-queued branch from my git repo at

http://cgit.freedesktop.org/~danvet/drm-intel/

That contains a patch from Chris Wilson to mitigate mmio offset exhaustion, one possible reason for a -ENOSPC failure. -Daniel

Signature · 3 lines
Daniel Vetter
Mail: [email protected]
Mobile: +41 (0)79 365 57 48
#5 lore raw
In reply to Knut Petersen
Quoted 21 lines · Knut Petersen

Am 02.04.2012 10:44, schrieb Chris Wilson:

Quoted 20 lines · Knut Petersen
Quoted 7 lines

After a few unattended hours Xorg was still running, but only a terminal window had survived the night :-(

Xorg: git, a few days old, kernel: 3.3.

Xorg.0.log: Nothing unusual

Quoted 8 lines

dmesg:

[156859.078080] [drm:drm_gem_create_mmap_offset] *ERROR* failed to allocate offset for bo 0
[179417.374016] VFS: file-max limit 204863 reached

I built a new X server and tried kernel 3.2.12 - that does not seem to help as both the number of open files and the number of gem objects still grow without obvious reasons.

It's the EFILE that is truly worrying. We have a patch in the queue to help ease the ENOSPC issue, but the EFILE implies a bo reference leak. And that I have not found yet.

Quoted 7 lines
Quoted 3 lines

Happy hunting,
-Chris

Well, it takes more than two days to trigger the EFILE limit here, but yesterday the ENOSPC bo problem appeared 6.5 hours after booting the system with current Xorg git and kernel 3.3.1. I wonder if this is one problem or if we face two independent bugs.

Where there is one, there is likely to be a cluster. However both ENOSPC and EFILE can be explained by a bo leak, just differing by how they were used prior to the leak. Eventually we run out of fd handles on the system (EFILE). If we mapped those into our process space before the leak then we hit the much smaller address space limit (ENOSPC). -Chris

Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
#4 lore raw
In reply to Daniel Vetter

Am 12.04.2012 10:03, schrieb Daniel Vetter:

Quoted 1 line

Can you try the drm-intel-next-queued branch from my git repo at http://cgit.freedesktop.org/~danvet/drm-intel/ That contains a patch from Chris Wilson to mitigate mmio offset exhaustion, one possible reason for a -ENOSPC failure. -Daniel

It´s a short time for a test, but drm-intel-next-queued does not look better than other kernels:

date; cat /proc/sys/fs/file-nr; cat /sys/kernel/debug/dri/0/i915_gem_objects

Do 12. Apr 13:21:50 CEST 2012
9323 0 204854
1275 objects, 81797120 bytes
1251 [1251] objects, 84410368 [84410368] bytes in gtt
19 [19] active objects, 11051008 [11051008] bytes 2 [2] pinned objects, 5373952 [5373952] bytes 1230 [1230] inactive objects, 67985408 [67985408] bytes 0 [0] freed objects, 0 [0] bytes
3 pinned mappable objects, 13762560 bytes
786 fault mappable objects, 16318464 bytes 268435456 [268435456] gtt total

Do 12. Apr 14:09:41 CEST 2012
12387 0 204854
4317 objects, 82264064 bytes
4282 [4282] objects, 84713472 [84713472] bytes in gtt
15 [15] active objects, 11304960 [11304960] bytes 2 [2] pinned objects, 5373952 [5373952] bytes 4265 [4265] inactive objects, 68034560 [68034560] bytes 0 [0] freed objects, 0 [0] bytes
3 pinned mappable objects, 13762560 bytes
3809 fault mappable objects, 27975680 bytes 268435456 [268435456] gtt total

Do 12. Apr 14:47:18 CEST 2012
15616 0 204854
7015 objects, 148389888 bytes
905 [905] objects, 73834496 [73834496] bytes in gtt
15 [15] active objects, 11304960 [11304960] bytes 2 [2] pinned objects, 5373952 [5373952] bytes 888 [888] inactive objects, 57155584 [57155584] bytes 0 [0] freed objects, 0 [0] bytes
3 pinned mappable objects, 13762560 bytes
699 fault mappable objects, 13668352 bytes 268435456 [268435456] gtt total

Files, total number of objects, inactive objects and fault mappable objects raise at a rate comparable to kernels 3.3.1, 3.2.* and 3.1*.

I searched about 15 months of log files - both error messages appeared for the first time about 14 days ago. Kernel 3.2 is much older, and the system survived _much_ longer uptimes previously. Either it´s not a kernel problem, or that kernel problem is exposed by some recent (max 6 weeks) change in Xorg, KDE, ... etc. Unfortunately there were a lot of changes in that time.

cu,
Knut