Tarkvaraline RDMA

Sissejuhatus

rxe abil saab pidada tarkvaralist RDMA lahendust. Võib olla on sellise tegevuse peamine põhjus nö väga odavalt infiniband laadse elamuse saamine.

RDMA üldiselt

Väited

• infiniband võrgu jaoks on rdma midagi sellist nagu on ethernet võrgu jaoks tcp/ip

Tarkvaralised seadmed

Ühes arvutis sobib öelda, eeldusel, et enp6s19 seade on tavaline füüsiline ethernet võrguseade (nt Proxmox platvormil töötavas virtuaalses arvutis virtio seade)

root@pve-sdn-01:~# modprobe rdma_rxe
root@pve-sdn-01:~# rdma link add rxe0 type rxe netdev enp6s19

ning teises arvutis sobib öelda

root@pve-sdn-02:~# modprobe rdma_rxe
root@pve-sdn-02:~# rdma link add rxe0 type rxe netdev enp6s19

Tulemusena on arvutites rdma seade

root@pve-sdn-02:~# rdma link
link rxe0/1 state ACTIVE physical_state LINK_UP netdev enp6s19

root@pve-sdn-02:~# ibv_devinfo 
hca_id:	rxe0
	transport:			InfiniBand (0)
	fw_ver:				0.0.0
	node_guid:			be24:11ff:fec2:de40
	sys_image_guid:			be24:11ff:fec2:de40
	vendor_id:			0xffffff
	vendor_part_id:			0
	hw_ver:				0x0
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		1024 (3)
			sm_lid:			0
			port_lid:		0
			port_lmc:		0x00
			link_layer:		Ethernet

rdma kontakti kontrolliks sobib proovida nt, ühes arvutis

root@pve-sdn-02:~# ib_send_bw -i 1 -d rxe0

************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
                    Send BW Test
 Dual-port       : OFF		Device         : rxe0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : OFF
 RX depth        : 512
 CQ Moderation   : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 1
 Max inline data : 0[B]
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x001f PSN 0xde6173
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:116:202
 remote address: LID 0000 QPN 0x001f PSN 0x82142e
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:116:201
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 65536      1000             0.00               118.36 		   0.001894
---------------------------------------------------------------------------------------

ja teises arvutis

root@pve-sdn-01:~# ib_send_bw -i 1 192.168.116.202 -d rxe0
---------------------------------------------------------------------------------------
                    Send BW Test
 Dual-port       : OFF		Device         : rxe0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : OFF
 TX depth        : 128
 CQ Moderation   : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 1
 Max inline data : 0[B]
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x001f PSN 0x82142e
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:116:201
 remote address: LID 0000 QPN 0x001f PSN 0xde6173
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:116:202
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 65536      1000             164.59             118.29 		   0.001893
---------------------------------------------------------------------------------------

Kasulikud lisamaterjalid

• man rxe - Software RDMA over Ethernet