Mellanox ConnectX-3 40GbE 網卡測試 - 3C

這張網卡裝在先前 5950X + WS X570-ACE 的機器上
型號是 MCX313A-BCCT

https://i.imgur.com/x7LwWyd.jpg

https://i.imgur.com/UFNWn7W.jpg

由於是買別人淘汰的卡, 一張 2800 就弄到了, 比全新的 10G 卡還便宜
不過目前拍賣上最便宜的是 HP 544FLR, 目前沒屋頂一張不到 800,
不過這張卡造型特殊需要轉一下才能用, 詳情可以去 YT 搜尋 "測速怪"
Proxmox VE 7.0 能偵測到, 使用的驅動程式是 mlx4_en
插上光纖讓兩台 5950X 互連, 確定兩張卡都亮燈
接著進入測試環節

首先用 iperf3 測試

# iperf3 -c 172.16.50.32
Connecting to host 172.16.50.32, port 5201
[ 5] local 172.16.50.31 port 56320 connected to 172.16.50.32 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 3.69 GBytes 31.7 Gbits/sec 0 1.29 MBytes
[ 5] 1.00-2.00 sec 3.67 GBytes 31.5 Gbits/sec 0 1.40 MBytes
[ 5] 2.00-3.00 sec 3.24 GBytes 27.9 Gbits/sec 0 1.47 MBytes
[ 5] 3.00-4.00 sec 3.09 GBytes 26.6 Gbits/sec 0 1.47 MBytes
[ 5] 4.00-5.00 sec 3.08 GBytes 26.5 Gbits/sec 0 1.47 MBytes
[ 5] 5.00-6.00 sec 3.07 GBytes 26.4 Gbits/sec 0 1.47 MBytes
[ 5] 6.00-7.00 sec 3.09 GBytes 26.5 Gbits/sec 0 1.47 MBytes
[ 5] 7.00-8.00 sec 3.09 GBytes 26.5 Gbits/sec 0 1.47 MBytes
[ 5] 8.00-9.00 sec 3.09 GBytes 26.5 Gbits/sec 0 1.47 MBytes
[ 5] 9.00-10.00 sec 3.10 GBytes 26.6 Gbits/sec 0 1.47 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 32.2 GBytes 27.7 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 32.2 GBytes 27.7 Gbits/sec receiver

前 3 秒爽了一下, 後來就掉速到 26Gbps, 有點傷心, 試了幾次都這樣
爬文調了一些設定看起來都沒啥鳥用
不過其中有一篇官網的建議測試方式

https://community.mellanox.com/s/article/iperf--iperf2--iperf3

其中有一句話是

we recommend to use iperf2 (2.0.x) in Linux

結果越想越不對勁, 改用 iperf2 做雙線測試
# iperf -c 172.16.50.32 -P 2
------------------------------------------------------------
Client connecting to 172.16.50.32, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 172.16.50.31 port 49590 connected with 172.16.50.32 port 5001
[ 4] local 172.16.50.31 port 49592 connected with 172.16.50.32 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0000-10.0005 sec 21.4 GBytes 18.4 Gbits/sec
[ 4] 0.0000-10.0002 sec 21.5 GBytes 18.5 Gbits/sec
[SUM] 0.0000-10.0002 sec 42.9 GBytes 36.9 Gbits/sec
[ CT] final connect times (min/avg/max/stdev) = 0.618/0.674/0.730/0.618 ms (tot/

結果真得逼近 40Gbps

接著查一下這張卡的 MTU 可設定範圍

# ip -d link show enp10s0
4: enp10s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DE
link/ether e4:1d:2d:18:a5:a0 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 m
numtxqueues 256 numrxqueues 128 gso_max_size 65536 gso_max_segs 65535 portid

可以看出 MTU 可設定範圍是 68-9900, 那麼改用 9900 來試試

# iperf -c 172.16.50.32 -P 2
------------------------------------------------------------
Client connecting to 172.16.50.32, TCP port 5001
TCP window size: 325 KByte (default)
------------------------------------------------------------
[ 3] local 172.16.50.31 port 49670 connected with 172.16.50.32 port 5001
[ 4] local 172.16.50.31 port 49672 connected with 172.16.50.32 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0000-10.0004 sec 23.1 GBytes 19.8 Gbits/sec
[ 3] 0.0000-10.0002 sec 23.1 GBytes 19.8 Gbits/sec
[SUM] 0.0000-10.0002 sec 46.1 GBytes 39.6 Gbits/sec
[ CT] final connect times (min/avg/max/stdev) = 1.115/1.224/1.334/1.115 ms (tot/

結果從 36.9 提升到 39.6, 看起來 MTU 要喬一下才能操到緊繃
那麼回頭測一下單線

# iperf3 -c 172.16.50.32
Connecting to host 172.16.50.32, port 5201
[ 5] local 172.16.50.31 port 56512 connected to 172.16.50.32 port 5201
... 略 ...
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 31.7 GBytes 27.2 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 31.7 GBytes 27.2 Gbits/sec receiver

看起來 MTU 9900 對單線沒啥鳥用, 不死心再用 4096 試試

[ 5] 0.00-10.00 sec 37.1 GBytes 31.9 Gbits/sec 0 sender

結果穩上 31.9, 再試試其他值, 後來試出 2048 更好

[ 5] 0.00-10.00 sec 38.0 GBytes 32.7 Gbits/sec 201 sender

修改參數, 把傳輸量加大到 200G 試試, iperf3 -c 172.16.50.32 -n 200G

[ 5] 0.00-56.10 sec 200 GBytes 30.6 Gbits/sec 197 sender

再回頭測試多線傳輸

# iperf -c 172.16.50.32 -P 2 -n 200G
------------------------------------------------------------
Client connecting to 172.16.50.32, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 4] local 172.16.50.31 port 50464 connected with 172.16.50.32 port 5001
[ 3] local 172.16.50.31 port 50462 connected with 172.16.50.32 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0000-90.2807 sec 200 GBytes 19.0 Gbits/sec
[ 4] 0.0000-90.3109 sec 200 GBytes 19.0 Gbits/sec
[SUM] 0.0000-90.3109 sec 400 GBytes 38.0 Gbits/sec
[ CT] final connect times (min/avg/max/stdev) = 0.737/0.784/0.831/0.831 ms (tot/

看起來 2048 有利單線但不利多線
改回 9900, 再測試很多線的 CPU 用量

# iperf -c 172.16.50.32 -P 128 -n 1G
[SUM] 0.0000-26.9918 sec 128 GBytes 40.7 Gbits/sec

# iperf -c 172.16.50.32 -P 32 -n 1G
[SUM] 0.0000-6.9327 sec 32.0 GBytes 39.6 Gbits/sec

看起來 32 線和 128 線大概都是消耗 4~5% 的 CPU 用量, 不過這可是 5950X 喔, 其他 CP
為了盡量發揮滿載吞吐量, 還是犧牲單線最佳值吧, 就決定用 9900 了

感謝大家看我如何浪費生命

--
世界越快 心則慢
網路越慢 心則快

--