ofed on xen

Hi All,

I would like to setup a node under xen and use infiniband in it. Here is the thing I would like to do :

node :
07:00.0 InfiniBand: Mellanox Technologies MT26418 [ConnectX IB DDR,
PCIe 2.0 5GT/s] (rev a0)

dom0: infiniband acess

      -> domu1: infiniband acess
      -> domu2: infiniband acess

so I would like to share the infiniband connection between a single card.

Is it possible? if yes, how? if not, is there plan to support something like that, or what kind of setup virtualisation people are using with infiniband?

I installed xen 3.4.2 (kernel 2.6.18.8) under centos 5.3 and ofed OFED-1.4-mlnx8. Then the infiniband card worked for dom0, but I found no way to make it working on any domu...

I then download OFED-1.5.1-rc1, and compile fine. but when i tried to load the modules in dom0, it crashed with the following message :
...
mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (April 4, 2008)
PM: Adding info for No Bus:mlx4_0
mlx4_en: Mellanox ConnectX HCA Ethernet driver v1.5.0 (Dec 2009)
ADDRCONF(NETDEV_UP): ib0: link is not ready
ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready

BUG: soft lockup detected on CPU#1!

Call Trace:
<IRQ> [<ffffffff8025d01e>] softlockup_tick+0xd6/0xeb
[<ffffffff8020e2b5>] timer_interrupt+0x471/0x4cf
[<ffffffff8814c70d>] :e1000:e1000_swfw_sync_release+0x2c/0x54
[<ffffffff8025d318>] handle_IRQ_event+0x4e/0x95
[<ffffffff8025d418>] __do_IRQ+0xb9/0x11e
[<ffffffff8020be09>] do_IRQ+0x44/0x50
[<ffffffff80236b8e>] tasklet_action+0xa8/0x12e
[<ffffffff8039c289>] evtchn_do_upcall+0x193/0x284
[<ffffffff80236a59>] __do_softirq+0x8a/0x117
[<ffffffff80209dee>] do_hypervisor_callback+0x1e/0x2c
<EOI> [<ffffffff8020522a>] hypercall_page+0x22a/0x1000
[<ffffffff8020522a>] hypercall_page+0x22a/0x1000
[<ffffffff803ea398>] pci_conf1_read+0x0/0xca
[<ffffffff8039b5ca>] force_evtchn_callback+0xa/0xb
[<ffffffff8032bfb4>] pci_bus_read_config_word+0x6f/0x7e
[<ffffffff88375012>] :mlx4_en:vpd_read_dword+0x4e/0x7a
[<ffffffff8837509c>] :mlx4_en:mlx4_en_cache_vpd+0x5e/0x144
[<ffffffff88370557>] :mlx4_en:mlx4_en_add+0x403/0x53c
[<ffffffff8810c4ba>] :mlx4_core:mlx4_add_device+0x2e/0xa1
[<ffffffff8810c6a1>] :mlx4_core:mlx4_register_interface+0x75/0x9a
[<ffffffff8024fa40>] sys_init_module+0x1872/0x1a42
[<ffffffff80209812>] tracesys+0xab/0xb5

thanks,

Jonathan

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi All,

ok, I managed to make ofed work withing a single domU. So the real question now is how do we setup ofed with xen to have multiple domu using the same card?

thanks,

Jonathan

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html