Tarkvaraline RDMA

Sissejuhatus

rxe abil saab pidada tarkvaralist RDMA lahendust. Võib olla on sellise tegevuse peamine põhjust nö väga odavalt infiniband laadse elamuse saamine.

Tarkvaralised seadmed

Ühes arvutis sobib öelda, eeldusel, et enp6s19 seade on tavaline füüsiline ethernet võrguseade (nt Proxmox platvormil töötavas virtuaalses arvutis virtio seade)

root@pve-sdn-01:~# modprobe rdma_rxe
root@pve-sdn-01:~# rdma link add rxe0 type rxe netdev enp6s19

ning teises arvutis sobib öelda

root@pve-sdn-02:~# modprobe rdma_rxe
root@pve-sdn-02:~# rdma link add rxe0 type rxe netdev enp6s19

Tulemusena on arvutites rdma seade

root@pve-sdn-02:~# rdma link
link rxe0/1 state ACTIVE physical_state LINK_UP netdev enp6s19

root@pve-sdn-02:~# ibv_devinfo 
hca_id:	rxe0
	transport:			InfiniBand (0)
	fw_ver:				0.0.0
	node_guid:			be24:11ff:fec2:de40
	sys_image_guid:			be24:11ff:fec2:de40
	vendor_id:			0xffffff
	vendor_part_id:			0
	hw_ver:				0x0
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		1024 (3)
			sm_lid:			0
			port_lid:		0
			port_lmc:		0x00
			link_layer:		Ethernet

rdma kontakti kontrolliks sobib proovida nt, ühes arvutis

root@pve-sdn-02:~# ib_send_bw -i 1 -d rxe0

************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
                    Send BW Test
 Dual-port       : OFF		Device         : rxe0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : OFF
 RX depth        : 512
 CQ Moderation   : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 1
 Max inline data : 0[B]
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x001f PSN 0xde6173
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:116:202
 remote address: LID 0000 QPN 0x001f PSN 0x82142e
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:116:201
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 65536      1000             0.00               118.36 		   0.001894
---------------------------------------------------------------------------------------

ja teises arvutis

root@pve-sdn-01:~# ib_send_bw -i 1 192.168.116.202 -d rxe0
---------------------------------------------------------------------------------------
                    Send BW Test
 Dual-port       : OFF		Device         : rxe0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : OFF
 TX depth        : 128
 CQ Moderation   : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 1
 Max inline data : 0[B]
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x001f PSN 0x82142e
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:116:201
 remote address: LID 0000 QPN 0x001f PSN 0xde6173
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:116:202
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 65536      1000             164.59             118.29 		   0.001893
---------------------------------------------------------------------------------------

Kasulikud lisamaterjalid

• man rxe - Software RDMA over Ethernet