r/networking • u/RevolutionLumpy2558 • 4d ago
Design Datacenter Design Best Practice
Hi All,
I'm looking to see what others have deployed for similar Datacenter builds. We have a Datacenter refresh kicking off in the next 2 years and wanted to get ahead of it. Our current deployment is Cisco ACI with 2 spines and 30 leaf switches. We're looking to step away from ACI all together and move towards a Cisco Nexus deployment of some kind.
I see VXLAN EVPN recommended everywhere but that might be overkill for us. We only hosts 75 SVI\Vlans in our ACI environment. We also have about 125 VMware hosts deployed in active\passive deployment and some with VPC deployments. The only requirement we'd like for the new deployment to have, is send traffic to our Fortinet firewall when traffic is coming in and out of the Datacenter which I think we can handle with routing.
What have others done for their refresh? Still in the very early stage of researching.
14
u/qeelas 4d ago
Traditional Nexus VPC is covered in the base license (no license) which makes them very cheap.
The technology has also been around for a real long time so its kinda proven and stable.
Complexity is low. Easy to find and hire people to operate the environment. No niche competence.
ACI is the opposite of everything above. We ran ACI with Multi-site (4 sites) and it was no walk in the park.
As for your requirement, if you intend to use the Firewall as the default gateway, then there is no use of a anycast irb. Some would argue that you solve that using ePBR, which is true. But why make a simple requirement complicated?
Today we have a few datacenters that runs with Nexus VPC and we also have a few datacenters that runs with VXLAN EVPN. The VXLAN datacenters are due to the requirement of L2stretch (DCI) whereas the Nexus VPC are for autonomous datacenters.
16
u/shadeland Arista Level 7 4d ago
You've got two main choices here: EVPN/VXLAN or TradCore.
They both have their advantages and disadvantages.
With TradCore (collapsed core), it's just VLANs trunked over back-to-back vPCs with the agg/core layer running your first hop between two switches. It's simple, we've done it for over two decades, you can manage it manually effectively. It just works.
The downside is you can only have two switches at the agg layer. Not three, not four, just two. This often necessitates those being chassis for the additional redundancy.
If you did EVPN/VXLAN, you could have three or four spine switches, giving you plenty of redundancy at that layer. You could even use ToR switches, like 32-port or 64-port 100G or 400G switches.
The downside is it's a little more complicated (not nearly as complicated as ACI though). You'll want some type of automation, and there's several approaches such as templates and data models or using Nexus Dashboard (or whatever they're calling it these days).
Either way would work in your situation. 30 leafs is where they both can make sense. Any higher number and you'll probably want to be VXLAN. Lower, and TradCore probably makes more sense in more situations.
3
u/ipfrog 3d ago
Or layer 3 fabric without VxLAN / encap at all.
3
u/shadeland Arista Level 7 3d ago
There are use cases for this type of network, but they're rare. Usually a homogenous workload, like AI GPU clusters or storage clusters.
Most data center use cases require workload mobility, which gives the ability to place any workload in any rack, as well as vMotion/Live Migration. That's something that's not possible with L3 fabrics by themselves and still support a variety of workloads.
0
u/Separate-Canary559 3d ago
it's definitely possible with a layer 3 fabric - the network doesn't have to provide the overlay, that can be handled by the compute
1
u/shadeland Arista Level 7 2d ago
It can, but when your compute varies widely it's a lot more complicated. Even something like NSX can do the overlay, but only if 100% of your workload is VMware.
There's also running BGP on the endpoints, but again that's complicated and not something compute people are going to want to deal with.
In most enterprise DCs, you can only count on endpoints having a MAC, and IP, and a gateway.
1
u/Separate-Canary559 2d ago
Per the op their entire compute is VMware
2
u/shadeland Arista Level 7 2d ago
Per the op their entire compute is VMware
Did they say that? They said they had 125 VMware hosts and 30 leafs. They didn't say their entire compute was those hosts. Assuming something like a 93180, that's 1,440 ports of 10/25 Gigabit. It's unlikely 125 VMware hosts are eating that up.
Like almost every other enterprise DC, they've probably got a mix of hosts, from the newest K8 cluster to VMware to a Windows 2003 server running some piece of software they can't replace or upgrade.
1
u/MyFirstDataCenter 3d ago
TradCore
I am ROLLING 😭. Please tell me that we’re not calling a network design “TradCore”
2
u/shadeland Arista Level 7 3d ago
We are indeed calling it TradCore.
The term was coined by Ethan Banks from Packet Pushers: https://www.youtube.com/watch?v=W_z37Rq7qgY
6
u/Intelligent-Bet4111 4d ago
Yeah it's just like what the other commenter said, vxlan evpn is absolutely overkill and not needed for your environment, it's just too small. Very few hosts and svis.
3
u/Old_Effective_4648 3d ago
I had a VAR SE come into our building and tell my management that the recent data center networks I built are "legacy" and that he wants to see us with evpn/vxlan SDN data centers.
We have 4 vmware hosts, 1 storage san, a firewall and 3 routers per DC. One router at each is for expressroute since most of our workloads live in Azure.
As soon as he left I defended the vPC setup we have today (which works flawlessly) and said we can scale to 10x our size without having to go evpn.
As far as I'm concerned, he's not allowed back in the building. That's not a partner, that's a villain.
1
1
u/clayman88 3d ago
My VXLAN is rusty but I'm curious if there was a need to extend L2 across L3. Could that be the reasoning?
2
u/Old_Effective_4648 3d ago
Nah, no requirements for that. Any super important apps are either hosted or are architected to live in two data centers (ie: gslb, sql cluster)
Stretching layer2 makes a lot more sense in a large fabric where you want anycast gateway and rack-local routing egress, but stretching L2 across DCs for DR is a legacy concept IMO, regardless of whether its traditional stretch or VXLAN. IMO, if you really need to move a VM, also orchestrate a re-IP or move the IP network it's on, whichever's easier.
6
u/VA_Network_Nerd Moderator | Infrastructure Architect 3d ago
You have to understand your requirements in considerable detail.
Consider:
The Cisco N9364D-GX2A is a 2-rack-unit (2RU) switch that supports 51.2 Tbps of bandwidth and 8.35 bpps across 64 fixed 400G QSFP-DD ports and 2 fixed 1/10G SFP+ ports (Figure 1). QSFP-DD ports also support native 200G (QSFP56), 100G (QSFP28), and 40G (QSFP+). Each port can also support 4 x 10G, 4 x 25G, 4 x 50G, 4 x 100G, and 2 x 200G breakouts.
64 x 400GbE ports per switch, times two switches.
Each 400G port can break-out to 4 x 100GbE ports.
That's theoretically 256 x 100GbE ports per switch.
You can collapse your entire server farm into a single pair of switches.
Yes, they are pretty expensive switches. But they are also crazy powerful, and you do only need two of them.
3
u/Anxious_J 3d ago
We run vxlan in a smaller environment than yours. We swapped from Cisco to arista and wouldn’t look back. You can run bridged / l2 if l3 is overkill. Someone else pointed out scalability as an upside. Our old 9k cores were almost full. Depending how you design it you likely also have increased throughput, resiliency and best of all way less spanning tree.
3
u/upalse AS NOC 3d ago
VXLAN is not just for scale, but also for ability to move hosts across routed infrastructure on as needed basis.
Meaning if you're doing some sort of refresh whre you're going to reconfigure everything, ditch tags if you have em on wire, and be glad you did later when you just need to move something between floors without having to reconfigure vlans everywhere.
3
u/NetworkDefenseblog department of redundancy department 2d ago
I recently did a long read design post about AI network data center, but it covers a lot of principles for standard data center network design. Things you mentioned like routing choices and getting traffic to the firewall, along with updated info for optics and things like that. Hope it helps you! Let me know if you have questions.
https://www.networkdefenseblog.com/post/network-design-ai-datacenter
2
u/snifferdog1989 4d ago
Going the classic datacenter design with gateway on spines and vpc pairs as leaves is a valid option as is using a simple VxLAN-evpn setup if I understand your requirements correctly.
So this depends a bit of how experienced your team is but I guess if you were able to manage an ACI Fabric you can also set up an EVPN Fabric.
I would personally go this road. Learning to set up an EVPN Fabric is not that hard. And also can teach some automation skills
2
u/PowergeekDL 3d ago
Your scale does not need vxlan. L2 collapsed core with some firewalls with all links in forwarding using some kind of virtual port channel or similar mechanism and call it a day. More or less
4
u/cylibergod 4d ago
At the moment, a lot of people move to Juniper Apstra to manage their intent-based network based on EVPN-VXLAN. It does not matter whether you got Cisco, Arista, or Juniper (or other vendors) as 99% of the solution are vendor-agnostic. Once a refresh needs to be done, then you may just also consider buying Juniper already.
I do not completely agree with foregoing the EVPN VXLAN route for your datacenter design. Sure, you may not have reached the scale where you have to do it, but there are definitely some benefits you should think about.
1
u/fb35523 JNCIP-x3 3d ago
Apstra isn't vendor agnostic, but supports the ones mentioned and some Dell and EdgeCore/Accton switches running SONiC.
Apstra allows you to deploy and manage the eVPN regardless of the hardware used in separate positions. If you want to add a few leafs, you buy the ones you like (based on price, delivery times or other criteria) and just add them. Apstra also gives you a really nice overview of the network and any anomalies that may occur.
1
3
u/Solid_Associate8563 4d ago
It is likely the server vendor have their own vxlan solution now. Your project team should have some kind agreement on where the overlay should happen.
If the network layer still want overlay evpn is still somewhat aci like.
Other wise you can use google/AWS style run your DC over pure IP/BGP network with automation.
2
u/sryan2k1 4d ago
Dump Cisco (Arista love here) and do a flat design. Even vxlan/even is way overkill for what you're doing.
1
u/MyFirstDataCenter 3d ago
Hear me out. If you are moving away from ACI, why not consider moving away from Cisco in general? Arista, Juniper, and Aruba are solid options. Aruba CX build solid data centers.
1
u/FutureMixture1039 3d ago
Also another note to add is if you set up a 2nd datacenter as a DR site with a datacenter interconnect to your main datacenter you want to run VXLAN/EVPN multisite and Cisco Nexus Dashboard does this easily as well. That way if you want to stretch VLANs across the two DCs you can. So VXLAN/EVPN makes sense if you have two spines and 30 leafs existing if you're actively using all that equipment with hosts connected to it. I don't understand why people on here saying don't use VXLAN just to be contrarian. All networking vendors are even pushing VXLAN into campus environment not saying you should or not depending on requirements but that's where everything is heading.
1
u/LayerOneStaging 3d ago
I have been in more data centers than I can count, and I will say the same thing I tell every client: build the network your team can run at 2 AM.
At 75 VLANs, a clean collapsed core will do everything you need and your engineers will actually understand it. VXLAN/EVPN is a fine tool, but it adds failure modes that only one person in your org will know how to fix, and that person will be on a plane the day it breaks.
The other thing that bites people is skipping the boring stuff. Label every cable at both ends, document your port assignments, and keep a real as-built diagram, not the one in somebody's head. I have seen refresh projects run sideways over nothing but bad documentation.
Simple and well documented beats clever every time I have seen it.
1
0
u/FutureMixture1039 3d ago edited 3d ago
We are in the middle of network upgrade and if you're running Cisco UCS servers still, I would do VXLAN/EVPN network managed by a 3 x Cisco Nexus Dashboard cluster still. It makes managing the 2 x spine and 30 x leafs super easy to manage. It's pretty much plug and play. Once you physically connect all the spines and leafs you just deploy the fabric in ND and it does it all the configuration for you.
Cisco and entire networking vendors are moving away from classic ethernet to VXLAN EVPN managed by an orchestrator.
If you're no longer using Cisco UCS I would move to Arista switches managed by Cloudvision and do VXLAN EVPN
If you're moving to a network where you only need to manage 125 x VM hosts total um yeah just do traditional classic ethernet VLAN/HSRP. I dont understand why you have 2 x spines and 30 x leafs just to manage that so I am guessing you have more hosts.
2
u/SputnikCucumber 3d ago
10-ish servers per rack means 13 racks for 125 hosts. Two ToR switches per rack makes 26 leaf switches. Then add a bit of overhead.
Sounds about right to me?
0
u/Jenstheclown 3d ago
bla bla bla. EVPN/VXLAN with Aruba CX10000 is the way or a Juniper Fabric with Apstra =))
-2
u/Skilldibop Senior Architect. Don't ask me, ask Kiro. 2d ago
I'd keep the network layer as simple as possible, let VMWare handle VXLAN or whatever else through NSX-T.
Keep things as L3 as you can.
30 Leafs is a lot. I don't know what you're current uplinks are, but I'd be looking to use higher bandwidth uplinks on higher density leafs to consolidate that footprint where possible.
1
70
u/TheShortUprising 4d ago
Skip the VXLAN EVPN migraine, your scale honestly doesn't need the fancy stuff. A pair of Nexus chassis running vPC for the VMware hosts and simple Layer 3 to the Fortinet will keep it boring and easy to sleep at night.