2.6.8.1-mm2 breaks vmware

Andrew, I know you are not interested in closed source vmware, I'm just blatantly Cc'ing you in case you would have some suggestion of the top of your head. As stuff slowly trickles from -mm to mainline, this could eventually end up biting more people.

In short, there are two (afaict) separate problem:

(1) vmmon.ko gives this:

vmmon: Your kernel is br0ken. get_user_pages(current, current->mm, b7dd1000, 1, 1, 0, &page, NULL) returned -14.
vmmon: I'll try accessing page tables directly, but you should know that your
vmmon: kernel is br0ken and you should uninstall all additional patches you
vmmon: have installed!
vmmon: FYI, copy_from_user(b7dd1000) returns 0 (if not 0 maybe your kernel is not br0ken)

(2) vmware fails to start any guest os, telling it cannot allocate memory:

VMX|[msg.msg.noMem] Cannot allocate memory.

(1) happened with 2.6.6-mm4 and with 2.6.8.1-mm2.
(2) only happened with 2.6.8.1-mm2 (with 2.6.6-mm4 vmware worked despite the warning.

So I backed out these patches from 2.6.8.1-mm2:

	flexible-mmap-2.6.7-mm3-A8.patch
	flex-mmap-for-ppc64.patch
	flex-mmap-for-s390x.patch
	sysctl-tunable-for-flexmmap.patch
	get_user_pages-latency-fix.patch
	increase-mlock-limit-to-32k.patch
	mlock-as-user-for-268-rc2-mm2.patch
	(All conveniently available for reference at ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.8.1/2.6.8.1-mm2/broken-out/
         if anyone is interested)

I had a vague hunch that flex-mmap stuff might affect (2) and get_user_pages patch perhaps (1).

After this, problem (1) went away for 2.6.8.1-mm2, but problem (2) remained.

Then I tried 2.6.8.1 vanilla. It does NOT suffer from either (1) or (2).

All experiments are done with Petr Vandrovec's newest vmware-any-any-update81 (apart from 2.6.6-mm4 that had some older any-to-any patch) and VMwareWorkstation-3.2.0-2230.

Before I continue backing stuff out: does anyone have ideas or suggestions what -mm patches might be suspectible problem (2)? 2.6.8.1 -> 2.6.8.1-mm2 is rather large patch and so is 2.6.6-mm4 -> 2.6.8.1-mm2, and the patches listed above were everything I thought might be suspectible.

Could get_user_pages-latency-fix.patch explain (1)? My kernel expertise is not sufficient to tell.

-- v --

[email protected]

Try -mm3, please. It'll have the same problem.

These have all been lumped together in mm3.

Try setting /proc/sys/vm/legacy_va_layout to 1

It won't be this.

Unlikely to be these.

Try setting /proc/sys/vm/overcommit_memory to 1

Maybe Peter could take a look sometime?

Doubtful.

I can try. I'm a bit confused what I should learn from trying -mm3 -- it doesn't have any big changes on this area, right?

As I said, backing these out did not cure (2) (vmware failing to allocate memory). And as Arjan said, flex-mmap is in Fedora kernels, and they have no problems.

Also, flex-mmap has been introduced after 2.6.6-mm4, right? That would mean it can't explain (1) ("get_user_pages() returns -EFAULT even though copy_from_user() return 0")

That would mean (1) is cured by reversing get_user_pages-latency-fix.patch, increase-mlock-limit-to-32k.patch or mlock-as-user-for-268-rc2-mm2.patch.

But what can explain (1), then? flex-mmap is not in 2.6.6-mm4, still it gives that warning.

I just noticed I had missed get_user_pages-handle-VM_IO.patch - I'll try backing that out first. I'll report back if I find anything interesting with different patch mixtures.

Forgot to tell: problem (2) (vmware failing to allocate memory) is consistent with 2.6.8.1-mm2 even when I try to configure the guest os to use 8MB memory. 128MB or 256MB is no problem with 2.6.8.1 mainline and it as /proc/sys/vm/overcommit_memory set to 0. (I have 512MB of RAM and 512MB of swap.)

I already harrashed him, but I couldn't tell off the top of his head.

Ok.

-- v --

[email protected]

Well, I just tried 2.6.8.1-mm2 minus get_user_pages-handle-VM_IO.patch but that didn't help with the "cannot allocate memory" problem. Curiously, I didn't get the "get_user_pages() returns -EFAULT" warning with this kernel.

Also, neither "echo 1 > /proc/sys/vm/overcommit_memory" nor "echo 1 > /proc/sys/vm/legacy_va_layout" helped.

It seems this is "cannot allocate memory" might have something to do with /dev/mem mmap() permissions - here's a strace:

--8<-----------------------------------------------------------------------

5022  geteuid32()                       = 1414
5022  setresuid32(-1, 0, -1)            = 0
5022  open("/dev/mem", O_RDWR)          = 8
5022  getuid32()                        = 1414
5022  setresuid32(-1, 1414, -1)         = 0
5022  old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, 8, 0x3d6000) = -1 EPERM (Operation not permitted)
5022  close(8)                          = 0
5022  ioctl(4, 0xf0, 0x3d6000)          = 0
5022  time(NULL)                        = 1093011756
5022  open("/etc/localtime", O_RDONLY)  = 8
5022  fstat64(8, {st_mode=S_IFREG|0644, st_size=682, ...}) = 0
5022  close(8)                          = 0
5022  writev(3, [{"Aug 20 17:22:36: ", 17}, {"VMX|", 4}, {"Msg_Post: Error\n", 16}], 3) = 37
5022  writev(3, [{"Aug 20 17:22:36: ", 17}, {"VMX|", 4}, {"[msg.msg.noMem] Cannot allocate "..., 40}], 3) = 61
5022  writev(3, [{"Aug 20 17:22:36: ", 17}, {"VMX|", 4}, {"--------------------------------"..., 41}], 3) = 62

--8<-----------------------------------------------------------------------

I just put get_user_pages-handle-VM_IO.patch back and reverted dev-mem-restriction-patch.patch - I'll report back when it has compiled.

-- v --

[email protected]

Ok, 2.6.8.1-mm2 minus dev-mem-restriction-patch.patch fixes the "cannot allocate memory" problem.

With this kernel I still get the

--8<-----------------------------------------------------------------------
vmmon: Your kernel is br0ken. get_user_pages(current, current->mm, b7dd1000, 1, 1, 0, &page, NULL) returned -14.
vmmon: I'll try accessing page tables directly, but you should know that your
vmmon: kernel is br0ken and you should uninstall all additional patches you vmmon: have installed!
vmmon: FYI, copy_from_user(b7dd1000) returns 0 (if not 0 maybe your kernel is not br0ken) --8<-----------------------------------------------------------------------

warning, but vmware appears to work now (well apart from altgr not working, but that has been broken since 2.4 -> 2.6 transition.)

I'm still not 100% which of the patches causes that get_user_pages() warning.

-- v --

[email protected]

Thanks for working that out.

Strange. I'd have assumed that the Fedora kernels include that patch.

If you could work that out sometime, it would help.

we do. Maybe you have an older (rare) vmware version ???

question is wtf vmware is doing with that memory.. it's outside the bios area after all.... petr?

Yes, VMwareWorkstation-3.2.0-2230.

Looks like I'll need to break my piggy bank and consider upgrading. Commercial software... ;@)

-- v --

[email protected]

Salut,

Does VMware > 4.0 still require cmov?

Tonnerre

Yes, this requirement is not going to go away: VMware >= 4.0 / GSX >= 3.0 require processors which support CMOV instruction.
Petr

qemu isnt as fast and isnt as featured but doesn't need cmov. The current one runs stuff like Win98 rather well including PCI video card emulation.

Alan

No problem. (If only compile-reboot-startx-launch vmware was faster on my computer...)

I'll try to do that as soon as I find time.

-- v --

[email protected]

* 2.6.8.1-mm2 minus just dev-mem-restriction-patch.patch fixes the "cannot allocate memory" problem.

* 2.6.8.1-mm2 minus dev-mem-restriction-patch.patch and get_user_pages-handle-VM_IO.patch fixes both the "cannot allocate memory" and "get_user_pages() returns -EFAULT" problems.

"Cannot allocate memory" may be specific to older vmware 3.2.0 since it hasn't been reported by anyone else (even though the patch is present in Fedora).

-- v --

[email protected]

Sorry that it took so long, I was without connectivity during weekend, trying to find what's going on.

VMware Workstation 3.2.x (builds 2230 to 2242) uses very strange way to allocate memory below 4GB range (older prods than 3.2.0 crash on systems with page tables above 4GB, and VMware 4.x use better technology).

Whole situation (with originally released vmmon) looked like that vmware binary issued ioctl() to allocate memory, marked that page PG_RESERVED, and returned physical page number to userspace. Userspace then opened /dev/mem, and mapped that page to the process. On cleanup /dev/mem was unmapped, PG_RESERVED bit was cleared, and page released (in my updates PG_RESERVED setting/clearing is removed, as it badly intereferes with page's refcounting).

This is broken by two patches in MM:
* get_user_pages_handle-VM_IO, which marks all /dev/mem memory mmaps as VM_IO (thus breaking get_user_pages() on these pages, which is later needed) and
* dev-mem-restricton-patch which prevents main memory to be mapped through /dev/mem (if page obtained from kernel is above 1MB).

During weekend I was able to create binary patch for VMware Workstation 3.2.1 (patch available at http://platan.vc.cvut.cz/ftp/pub/vmware/vmware-any-any-update82.tar.gz)
which turns that messy ioctl & /dev/mem mmap to simpler, safer and better /dev/vmmon mmap (and unmap + ioctl to simple unmap) (as used by WS4+/GSX3+). After that WS 3.2.1 works on your 2.6.8.1-mm3 without problems (after applying two patches I'm sending you separately to get -mm3 to work at all on my notebook).

Andrew, because I have a binary patch for affected products, I think that you should leave your tree as is. Although I do not agree with dev-mem-restriction-patch, maybe some other people will feel safer with this restriction in place.

You can redirect VMware Workstation 3.2.x and VMware GSX Server 2.x customers to me, and I'll create binary patches for these other products on demand (I do not want to waste my time, and GSX customers should either use supported OSes, or latest released GSX version anyway).

Ville, please try applying vmware-any-any-update82.tar.gz. During application it must say that it found 'VMware Workstation 3.2.1-build 2242'. If it will say that it found build-2230 to 2242, new binary pattern was not recognized and I'll need your vmware binary, as I did not find build 2230 in my archive. Or you can upgrade to the build 2242.

Best regards,
Petr Vandrovec [email protected]

I've sent andrew a patch that allows such mmaps again for PG_RESERVED pages. The approach vmware did is rather questionable (as you say) and thankfully fixed later, but breaking it is not entirely required.

Incredible.

I was sure I was out of look with the aged vmware 3.2.0, and now suddenly there is patch for both the kernel and the application :).

I didn't even realize 2242 had been released ;). (As I said, I've lately been using vmware less and less.)

I just upgraded to 2242 + any2any82 and it works fine (*). I'm still using the 2.6.8.1-mm2 with the updated dev-mem restriction patch. I'll retest this as soon as I get to upgrade my kernel again to something that doesn't contain Arjan's new dev-mem patch. I'll let you know.

*) Apart from that even 3.2.1 2242 does not fix the altgr key that does not
seem to work with 2.6 new input system. I presume the new input layer is the reason; I do know altgr used to work with 2.4 host kernel and winxp guest.

-- v --

[email protected]

For the record, Petr found a solution for this one, too.

For whatever reason, xev gives the following for altgr (right alt) these days:

    state 0x0, keycode 113 (keysym 0xfe03, ISO_Level3_Shift), same_screen YES,
                       ^^^

Adding

xkeymap.keycode.113 = 312

into /etc/vmware/config or into the per guest .cfg lets altgr work again with vmware 3.2.1.

-- v --

[email protected]

unlikely candidates; these are in the fedora kernel too and no problems reported with vmware like this...
(and we do get reports when vmware breaks ;)