pci passthrough - VF reset at boot is dropping assigned MAC

Running qemu-kvm.git as of today (ffce28f, April 18, 2011) the virtual function passed to the VM is losing its assigned mac address. That is, prior to launching qemu-kvm, the following command is run to set the MAC address:

ip link set dev eth2 vf 0 mac 02:12:34:56:79:20

Yet, when the VM boots the MAC address is random which is what happens when the VF is reset. Looking through the commit logs between 0.13.0 -- the version in Fedora 14 -- and latest git I found the following:

commit d9488459ff2ab113293586c1c36b1679bb15deee

Author: Alex Williamson <[email protected]>
Date:   Thu Mar 17 15:24:31 2011 -0600

device-assignment: Reset device on system reset

On system reset, we currently try to quiesce DMA by clearing the command register. This assumes that nothing re-enables bus master support without first de-programming the device. Use a bigger hammer to help the guest not shoot itself by issuing a function reset via sysfs on each system reset.

Signed-off-by Alex Williamson <[email protected]>

Acked-by Chris Wright <[email protected]>

Signed-off-by Marcelo Tosatti <[email protected]>

Is this the cause of the MAC address reset and is this behavior intended?

Quoted 28 lines · David Ahern

Running qemu-kvm.git as of today (ffce28f, April 18, 2011) the virtual function passed to the VM is losing its assigned mac address. That is, prior to launching qemu-kvm, the following command is run to set the MAC address:

ip link set dev eth2 vf 0 mac 02:12:34:56:79:20

Yet, when the VM boots the MAC address is random which is what happens when the VF is reset. Looking through the commit logs between 0.13.0 -- the version in Fedora 14 -- and latest git I found the following:

commit d9488459ff2ab113293586c1c36b1679bb15deee

Author: Alex Williamson <[email protected]>
Date:   Thu Mar 17 15:24:31 2011 -0600

device-assignment: Reset device on system reset

On system reset, we currently try to quiesce DMA by clearing the command register. This assumes that nothing re-enables bus master support without first de-programming the device. Use a bigger hammer to help the guest not shoot itself by issuing a function reset via sysfs on each system reset.

Signed-off-by Alex Williamson <[email protected]>

Acked-by Chris Wright <[email protected]>

Signed-off-by Marcelo Tosatti <[email protected]>

Quoted 1 line

Is this the cause of the MAC address reset and is this behavior intended?

Ugh, I hope not, it's certainly not an intended side effect. Can you see if the problem still happens if you revert this patch? If it does,

we might need more device specific reset functions to save and restore that extra bit of state. I assume this is still the 82576 VF you were asking about before? Thanks,

Alex

Quoted 29 lines · Alex Williamson

Quoted 28 lines · David Ahern

Running qemu-kvm.git as of today (ffce28f, April 18, 2011) the virtual function passed to the VM is losing its assigned mac address. That is, prior to launching qemu-kvm, the following command is run to set the MAC address:

ip link set dev eth2 vf 0 mac 02:12:34:56:79:20

Yet, when the VM boots the MAC address is random which is what happens when the VF is reset. Looking through the commit logs between 0.13.0 -- the version in Fedora 14 -- and latest git I found the following:

commit d9488459ff2ab113293586c1c36b1679bb15deee

Author: Alex Williamson <[email protected]>
Date:   Thu Mar 17 15:24:31 2011 -0600

device-assignment: Reset device on system reset

On system reset, we currently try to quiesce DMA by clearing the command register. This assumes that nothing re-enables bus master support without first de-programming the device. Use a bigger hammer to help the guest not shoot itself by issuing a function reset via sysfs on each system reset.

Signed-off-by Alex Williamson <[email protected]>

Acked-by Chris Wright <[email protected]>

Signed-off-by Marcelo Tosatti <[email protected]>

Quoted 4 lines

Quoted 1 line

Is this the cause of the MAC address reset and is this behavior intended?

Ugh, I hope not, it's certainly not an intended side effect. Can you see if the problem still happens if you revert this patch? If it does,

I commented out the write() in the reset function and indeed the mac address was not reset on VM boot.

So yes, it does change. However, if I set the VF mac instead of using a randomly generated one, I get:

# modprobe -r igbvf
# ip link set eth2 vf 6 mac 02:00:10:91:73:01
# modprobe igbvf
# dmesg | grep "igbvf 0000\:01\:11.5\: Address\:"
igbvf 0000:01:11.5: Address: d2:c8:17:d6:97:f7
igbvf 0000:01:11.5: Address: 4e:ee:2a:d8:12:7c
igbvf 0000:01:11.5: Address: 02:00:10:91:73:01
# modprobe -r igbvf
# echo 1 > /sys/bus/pci/devices/0000:01:11.5/reset
# modprobe igbvf
# dmesg | grep "igbvf 0000\:01\:11.5\: Address\:"
igbvf 0000:01:11.5: Address: d2:c8:17:d6:97:f7
igbvf 0000:01:11.5: Address: 4e:ee:2a:d8:12:7c
igbvf 0000:01:11.5: Address: 02:00:10:91:73:01
igbvf 0000:01:11.5: Address: 02:00:10:91:73:01

So now it sticks. You're going to get random mac addresses on the VFs every time you reload the igb driver (ie. ever boot) anyway (at least with these sr-iov cards), so if you need consistent macs, they probably need to be set before launching the VM anyway. Thanks,

Alex

# modprobe -r igbvf
# ip link set eth2 vf 6 mac 02:00:10:91:73:01
# modprobe igbvf
# dmesg | grep "igbvf 0000\:01\:11.5\: Address\:"
igbvf 0000:01:11.5: Address: d2:c8:17:d6:97:f7
igbvf 0000:01:11.5: Address: 4e:ee:2a:d8:12:7c
igbvf 0000:01:11.5: Address: 02:00:10:91:73:01
# modprobe -r igbvf
# echo 1 > /sys/bus/pci/devices/0000:01:11.5/reset
# modprobe igbvf
# dmesg | grep "igbvf 0000\:01\:11.5\: Address\:"
igbvf 0000:01:11.5: Address: d2:c8:17:d6:97:f7
igbvf 0000:01:11.5: Address: 4e:ee:2a:d8:12:7c
igbvf 0000:01:11.5: Address: 02:00:10:91:73:01
igbvf 0000:01:11.5: Address: 02:00:10:91:73:01

So now it sticks. You're going to get random mac addresses on the VFs every time you reload the igb driver (ie. ever boot) anyway (at least with these sr-iov cards), so if you need consistent macs, they probably need to be set before launching the VM anyway. Thanks,

Alex

Quoted 16 lines · Alex Williamson

# modprobe -r igbvf
# ip link set eth2 vf 6 mac 02:00:10:91:73:01
# modprobe igbvf
# dmesg | grep "igbvf 0000\:01\:11.5\: Address\:"
igbvf 0000:01:11.5: Address: d2:c8:17:d6:97:f7
igbvf 0000:01:11.5: Address: 4e:ee:2a:d8:12:7c
igbvf 0000:01:11.5: Address: 02:00:10:91:73:01
# modprobe -r igbvf
# echo 1 > /sys/bus/pci/devices/0000:01:11.5/reset
# modprobe igbvf
# dmesg | grep "igbvf 0000\:01\:11.5\: Address\:"
igbvf 0000:01:11.5: Address: d2:c8:17:d6:97:f7
igbvf 0000:01:11.5: Address: 4e:ee:2a:d8:12:7c
igbvf 0000:01:11.5: Address: 02:00:10:91:73:01
igbvf 0000:01:11.5: Address: 02:00:10:91:73:01

Quoted 7 lines

So now it sticks. You're going to get random mac addresses on the VFs every time you reload the igb driver (ie. ever boot) anyway (at least with these sr-iov cards), so if you need consistent macs, they probably need to be set before launching the VM anyway. Thanks,

Alex

Ok, I was able to repeat the above commands from the host command line.

However, when qemu-kvm starts the MAC is reset.

# ip link show | less

2: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000

    link/ether 00:1b:21:98:b7:10 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 02:12:34:56:80:20

--> that's the MAC address I set

I start qemu-kvm (unpatched version) and the host side sees the address changed:

# ip link show | less

2: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000

    link/ether 00:1b:21:98:b7:10 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 7a:17:3f:98:0f:db

Can you try that aspect on your end - seeing if the MAC address maintains after starting qemu-kvm?

Quoted 50 lines · David Ahern

Quoted 16 lines · Alex Williamson

# modprobe -r igbvf
# ip link set eth2 vf 6 mac 02:00:10:91:73:01
# modprobe igbvf
# dmesg | grep "igbvf 0000\:01\:11.5\: Address\:"
igbvf 0000:01:11.5: Address: d2:c8:17:d6:97:f7
igbvf 0000:01:11.5: Address: 4e:ee:2a:d8:12:7c
igbvf 0000:01:11.5: Address: 02:00:10:91:73:01
# modprobe -r igbvf
# echo 1 > /sys/bus/pci/devices/0000:01:11.5/reset
# modprobe igbvf
# dmesg | grep "igbvf 0000\:01\:11.5\: Address\:"
igbvf 0000:01:11.5: Address: d2:c8:17:d6:97:f7
igbvf 0000:01:11.5: Address: 4e:ee:2a:d8:12:7c
igbvf 0000:01:11.5: Address: 02:00:10:91:73:01
igbvf 0000:01:11.5: Address: 02:00:10:91:73:01

Quoted 7 lines

So now it sticks. You're going to get random mac addresses on the VFs every time you reload the igb driver (ie. ever boot) anyway (at least with these sr-iov cards), so if you need consistent macs, they probably need to be set before launching the VM anyway. Thanks,

Alex

Ok, I was able to repeat the above commands from the host command line.

However, when qemu-kvm starts the MAC is reset.

# ip link show | less

2: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000

    link/ether 00:1b:21:98:b7:10 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 02:12:34:56:80:20

--> that's the MAC address I set

I start qemu-kvm (unpatched version) and the host side sees the address changed:

# ip link show | less

2: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000

    link/ether 00:1b:21:98:b7:10 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 7a:17:3f:98:0f:db

Quoted 2 lines

Can you try that aspect on your end - seeing if the MAC address maintains after starting qemu-kvm?

I don't see this happening on my system, once manually set the mac never changes. I can restart and reset the VM and the host and guest both continue seeing the set mac address. I tested it with both a recent rhel6.1 host kernel as well as upstream 2.6.39-rc4. If I switch to a VF with an unset mac, those will change on each VM reset or restart.

Quoted 51 lines · Alex Williamson

Quoted 50 lines · David Ahern

Quoted 16 lines · Alex Williamson

# modprobe -r igbvf
# ip link set eth2 vf 6 mac 02:00:10:91:73:01
# modprobe igbvf
# dmesg | grep "igbvf 0000\:01\:11.5\: Address\:"
igbvf 0000:01:11.5: Address: d2:c8:17:d6:97:f7
igbvf 0000:01:11.5: Address: 4e:ee:2a:d8:12:7c
igbvf 0000:01:11.5: Address: 02:00:10:91:73:01
# modprobe -r igbvf
# echo 1 > /sys/bus/pci/devices/0000:01:11.5/reset
# modprobe igbvf
# dmesg | grep "igbvf 0000\:01\:11.5\: Address\:"
igbvf 0000:01:11.5: Address: d2:c8:17:d6:97:f7
igbvf 0000:01:11.5: Address: 4e:ee:2a:d8:12:7c
igbvf 0000:01:11.5: Address: 02:00:10:91:73:01
igbvf 0000:01:11.5: Address: 02:00:10:91:73:01

Quoted 7 lines

So now it sticks. You're going to get random mac addresses on the VFs every time you reload the igb driver (ie. ever boot) anyway (at least with these sr-iov cards), so if you need consistent macs, they probably need to be set before launching the VM anyway. Thanks,

Alex

Ok, I was able to repeat the above commands from the host command line.

However, when qemu-kvm starts the MAC is reset.

# ip link show | less

2: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000

    link/ether 00:1b:21:98:b7:10 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 02:12:34:56:80:20

--> that's the MAC address I set

I start qemu-kvm (unpatched version) and the host side sees the address changed:

# ip link show | less

2: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000

    link/ether 00:1b:21:98:b7:10 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 7a:17:3f:98:0f:db

Quoted 8 lines

Quoted 2 lines

Can you try that aspect on your end - seeing if the MAC address maintains after starting qemu-kvm?

I don't see this happening on my system, once manually set the mac never changes. I can restart and reset the VM and the host and guest both continue seeing the set mac address. I tested it with both a recent rhel6.1 host kernel as well as upstream 2.6.39-rc4. If I switch to a VF with an unset mac, those will change on each VM reset or restart.

Blacklist igbvf in the host and you will. That must be the difference: I was preventing the vf driver from loading in the host -- it's not needed there, so why load it?

I rebooted for a fresh run. Loaded the igbvf driver before starting the VM using my tools. With the igbvf driver loaded in the host the MAC address for the VF was not reset.

As for why I blacklisted it -- udev. What a PITA with VFs. I saw the feature for Fedora 15 which should address this.

Quoted 9 lines

Quoted 5 lines

I don't see this happening on my system, once manually set the mac never changes. I can restart and reset the VM and the host and guest both continue seeing the set mac address. I tested it with both a recent rhel6.1 host kernel as well as upstream 2.6.39-rc4. If I switch to a VF with an unset mac, those will change on each VM reset or restart.

Blacklist igbvf in the host and you will. That must be the difference: I was preventing the vf driver from loading in the host -- it's not needed there, so why load it?

I already have it blacklisted. It's not needed if you're using the VFs they way we are, but there are other uses.

Quoted 6 lines

I rebooted for a fresh run. Loaded the igbvf driver before starting the VM using my tools. With the igbvf driver loaded in the host the MAC address for the VF was not reset.

As for why I blacklisted it -- udev. What a PITA with VFs. I saw the feature for Fedora 15 which should address this.

Yes, my VM is up to renaming the VFs eth1340 since the mac changes every boot. I'm still confused though as I did a whole round of testing after a reboot where igbvf was never loaded and the set mac address stuck across VM restarts and resets.

Alex

Quoted 25 lines · Alex Williamson

Quoted 9 lines

Quoted 5 lines

I don't see this happening on my system, once manually set the mac never changes. I can restart and reset the VM and the host and guest both continue seeing the set mac address. I tested it with both a recent rhel6.1 host kernel as well as upstream 2.6.39-rc4. If I switch to a VF with an unset mac, those will change on each VM reset or restart.

Blacklist igbvf in the host and you will. That must be the difference: I was preventing the vf driver from loading in the host -- it's not needed there, so why load it?

I already have it blacklisted. It's not needed if you're using the VFs they way we are, but there are other uses.

Quoted 6 lines

I rebooted for a fresh run. Loaded the igbvf driver before starting the VM using my tools. With the igbvf driver loaded in the host the MAC address for the VF was not reset.

As for why I blacklisted it -- udev. What a PITA with VFs. I saw the feature for Fedora 15 which should address this.

Yes, my VM is up to renaming the VFs eth1340 since the mac changes every boot. I'm still confused though as I did a whole round of testing after a reboot where igbvf was never loaded and the set mac address stuck across VM restarts and resets.

Quoted 2 lines

Alex

The resetting of the VM MAC address is fixed in 2.6.39-rc4, so it's a Fedora 14, 2.6.35.12 problem.