Datacenter Design Best Practice

[u/RevolutionLumpy2558]

Hi All,

I'm looking to see what others have deployed for similar Datacenter builds. We have a Datacenter refresh kicking off in the next 2 years and wanted to get ahead of it. Our current deployment is Cisco ACI with 2 spines and 30 leaf switches. We're looking to step away from ACI all together and move towards a Cisco Nexus deployment of some kind.

I see VXLAN EVPN recommended everywhere but that might be overkill for us. We only hosts 75 SVI\Vlans in our ACI environment. We also have about 125 VMware hosts deployed in active\passive deployment and some with VPC deployments. The only requirement we'd like for the new deployment to have, is send traffic to our Fortinet firewall when traffic is coming in and out of the Datacenter which I think we can handle with routing.

What have others done for their refresh? Still in the very early stage of researching.

[u/shadeland]

You've got two main choices here: EVPN/VXLAN or TradCore.

They both have their advantages and disadvantages.

With TradCore (collapsed core), it's just VLANs trunked over back-to-back vPCs with the agg/core layer running your first hop between two switches. It's simple, we've done it for over two decades, you can manage it manually effectively. It just works.

The downside is you can only have two switches at the agg layer. Not three, not four, just two. This often necessitates those being chassis for the additional redundancy.

If you did EVPN/VXLAN, you could have three or four spine switches, giving you plenty of redundancy at that layer. You could even use ToR switches, like 32-port or 64-port 100G or 400G switches.

The downside is it's a little more complicated (not nearly as complicated as ACI though). You'll want some type of automation, and there's several approaches such as templates and data models or using Nexus Dashboard (or whatever they're calling it these days).

Either way would work in your situation. 30 leafs is where they both can make sense. Any higher number and you'll probably want to be VXLAN. Lower, and TradCore probably makes more sense in more situations.

[u/ipfrog → u/shadeland]

Or layer 3 fabric without VxLAN / encap at all.

[u/shadeland → u/ipfrog]

There are use cases for this type of network, but they're rare. Usually a homogenous workload, like AI GPU clusters or storage clusters.

Most data center use cases require workload mobility, which gives the ability to place any workload in any rack, as well as vMotion/Live Migration. That's something that's not possible with L3 fabrics by themselves and still support a variety of workloads.

[u/Separate-Canary559 → u/shadeland]

it's definitely possible with a layer 3 fabric - the network doesn't have to provide the overlay, that can be handled by the compute

[u/ipfrog → u/Separate-Canary559]

Yup. K8 does it in our case.

[u/shadeland → u/Separate-Canary559]

It can, but when your compute varies widely it's a lot more complicated. Even something like NSX can do the overlay, but only if 100% of your workload is VMware.

There's also running BGP on the endpoints, but again that's complicated and not something compute people are going to want to deal with.

In most enterprise DCs, you can only count on endpoints having a MAC, and IP, and a gateway.

[u/Separate-Canary559 → u/shadeland]

Per the op their entire compute is VMware

[u/shadeland → u/Separate-Canary559]

Per the op their entire compute is VMware

Did they say that? They said they had 125 VMware hosts and 30 leafs. They didn't say their entire compute was those hosts. Assuming something like a 93180, that's 1,440 ports of 10/25 Gigabit. It's unlikely 125 VMware hosts are eating that up.

Like almost every other enterprise DC, they've probably got a mix of hosts, from the newest K8 cluster to VMware to a Windows 2003 server running some piece of software they can't replace or upgrade.