Mellanox ConnectX-3 40GbE 網卡測試 - 3C

※ 引述《tacovirus (大怒神のスイッチ)》之銘言：
: 這張網卡裝在先前 5950X + WS X570-ACE 的機器上
: 型號是 MCX313A-BCCT
: https://i.imgur.com/x7LwWyd.jpg
: https://i.imgur.com/UFNWn7W.jpg
: 由於是買別人淘汰的卡, 一張 2800 就弄到了, 比全新的 10G 卡還便宜
: 不過目前拍賣上最便宜的是 HP 544FLR, 目前沒屋頂一張不到 800,
: 不過這張卡造型特殊需要轉一下才能用, 詳情可以去 YT 搜尋 "測速怪"
: Proxmox VE 7.0 能偵測到, 使用的驅動程式是 mlx4_en
: ...

40GbE 爽卡已經搞定, 接下來要解決穩定問題
為了讓系統能夠在 40G 網卡故障時, 用比較弱的卡接手工作
所以使用 Linux Bonding 做一組 Active Backup
用 PVE 介面來設定大概長這樣

https://imgur.com/a/vy513pX

接著來實測, 三秒時拔掉光纖觀察, 然後插回去

# iperf3 -c 172.16.50.32 -t 60
Connecting to host 172.16.50.32, port 5201
[ 5] local 172.16.50.31 port 42834 connected to 172.16.50.32 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 3.97 GBytes 34.1 Gbits/sec 0 1.38 MBytes
[ 5] 1.00-2.00 sec 3.19 GBytes 27.4 Gbits/sec 0 1.49 MBytes
[ 5] 2.00-3.00 sec 2.78 GBytes 23.9 Gbits/sec 0 1.49 MBytes
[ 5] 3.00-4.00 sec 95.0 MBytes 797 Mbits/sec 1011 321 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 352 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 944 Mbits/sec 0 352 KBytes
[ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 0 352 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 944 Mbits/sec 0 352 KBytes
[ 5] 8.00-9.00 sec 97.5 MBytes 818 Mbits/sec 0 352 KBytes
[ 5] 9.00-10.00 sec 2.98 GBytes 25.6 Gbits/sec 242 1.42 MBytes
[ 5] 10.00-11.00 sec 3.43 GBytes 29.5 Gbits/sec 0 1.42 MBytes
[ 5] 11.00-12.00 sec 3.53 GBytes 30.3 Gbits/sec 0 1.42 MBytes
[ 5] 12.00-13.00 sec 3.82 GBytes 32.8 Gbits/sec 0 1.42 MBytes

實測結果看起來確實有接手, 恢復正常時也會自己切回 40G

接著來觀察流量損失

# iperf -c 172.16.50.32 -P 3
------------------------------------------------------------
Client connecting to 172.16.50.32, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 172.16.50.31 port 52454 connected with 172.16.50.32 port 5001
[ 4] local 172.16.50.31 port 52456 connected with 172.16.50.32 port 5001
[ 5] local 172.16.50.31 port 52458 connected with 172.16.50.32 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0000-10.0007 sec 14.6 GBytes 12.5 Gbits/sec
[ 5] 0.0000-10.0004 sec 14.5 GBytes 12.4 Gbits/sec
[ 3] 0.0000-10.0004 sec 14.6 GBytes 12.5 Gbits/sec
[SUM] 0.0000-10.0004 sec 43.7 GBytes 37.5 Gbits/sec
[ CT] final connect times (min/avg/max/stdev) = 0.286/0.385/0.512/0.255 ms (tot/

頻寬從 39.5 下降到 37.5, 損失還在可接受範圍
看起來可以放心使用 iSCSI 了

--
世界越快 心則慢
網路越慢 心則快

--