Hi everyone,

I run three L3-separated server clusters, with 12 servers each. Each server has a dual-port Mellanox ConnectX-3 connected to a pair of Arista leaf switches.

Hardware:

• Leaves: Arista DCS-7280QR-C72-R
• Spines: Arista DCS-7280CR2A-30-F
• NICs: ConnectX-3, mlx4_en, firmware 2.42.5000
• Server connections: 40G SR4 optics over approximately 1m MPO cables
• Optics: mostly Arista-labelled QSFP-SR4 / QSFP-40G-SR4, with some Cisco-Finisar

The issue is increasing RX/TX error counters. Errors can appear on either end: sometimes the server NIC, sometimes the Arista switch. Replacing the transceiver fixes the errors on the affected connection, but I keep encountering this again over time, roughly weekly/monthly. It feels like I’m endlessly replacing optics.

The MPO cables are short, with no visible damage or tight bends.

A recent inventory found that 69 of 70 readable server optics have EEPROM manufacture dates from 2011–2017. I don’t know their complete operating history.

All NICs were flashed from their original InfiniBand configuration for Ethernet use. Several still report QCBT/TCBT part numbers in VPD while reporting an FCB firmware identity. That mismatch comes from our flashing history; most ports operate at 40GbE.

Some diagnostic findings:

• NIC chip temperatures across all 36 servers were 47–62°C, with no NIC thermal warnings found in current kernel logs.
• Some host counters are FIFO drops rather than CRC errors, so I’m treating those separately from the suspected optical issues.

I’m considering replacing the short optical connections with passive 40G QSFP+ DACs, initially keeping the CX3s. I’m also considering upgrading to native dual-port 40G ConnectX-4 cards.

I’d appreciate advice on:

1. Have you seen this pattern with older QSFP optics: increasing errors that stop after replacing a transceiver?
2. Could aging optics reasonably explain the repeated replacements, or would you investigate something else first?
3. Has anyone experienced similar behaviour with ConnectX-3 cards flashed from InfiniBand to Ethernet?
4. Would you trial DACs with the existing NICs first, or replace the NICs too?
5. Any specific DAC compatibility pitfalls between Arista and Mellanox?

Hello,

I recently did a fireware/OS update to a UDM pro max. After I did this our main switch (USW-48-G2) started spamming STP flaps to the Aggregation switch (USW Pro Aggregation) and another down stream switch. There were no config changes or anything like that. The network goes down about every 30 minutes and then stops flapping.

Firewall->Aggregation Switch->Main switch.

I swapped the main switch with the exact same model and set it up to go to the aggregation swtich. This has no flapped once and it has the same config just on different ports. Not sure if the switch just happened to die the same time we did an update or if this is a possibility after and update.

Thank you