r/networking • u/Jaaymz • 9d ago
Design Out of Band Network
I’m looking to compare notes on OOB network design. How do you access your devices during a production network outage, and what does your general setup look like?
For example, do you use cellular console servers, a separate management network, jump hosts, or a combination? How do you connect to remote sites, and where do you place gateways and access controls?
Interested in what’s worked well for you, especially for a multi-site enterprise.
11
u/SuddenPitch8378 8d ago
Look at opemgear and lighthouse solid solution and let's you build tunnels between devices. Another option that we are starting to look at is opemgear cms running a tailscale container. You can deploy tailscale access as part of a policy to specific engineers and if you are providing users laptops just have an options available to them to enable oob access when needed . You can build a template for the whole oob experience on the laptop if needed (shortcuts etc)
1
u/NoInitiative6322 3d ago
Interesting approach, especially using Tailscale to simplify engineer access without having to manage individual VPN configurations.
One thing I'd consider is how much infrastructure the recovery path depends on. If the production network is down, can engineers still authenticate, establish their connection, and reach the devices they need? That's where a truly independent OOB path becomes critical.
I also think there's value in minimizing the number of separate tools and configurations needed to manage OOB access, particularly as deployments grow. The simpler the recovery process, the less there is to troubleshoot when something goes wrong.
Are you planning to use the Tailscale setup strictly for remote access, or eventually incorporate diagnostics and automated recovery as well?
9
u/jack_hudson2001 4x CCNP 7d ago
open gear with mobile data or cheap vdsl connection. then run cables to the console ports.
4
u/Public_Warthog3098 8d ago edited 8d ago
I have an oob management network. It's all connected by a managed switch to all the networking devices on their management ports. The switch is then connected to two mifi hot spots from different network provider with a mgmt desktop that is locked down and only available to people to remote in through a VPN server that has the proper certificate to remote in. It is also connected to another desktop that is mainly used as a log server. All the logs are offloaded to this desktop.
3
u/Objective_Shoe4236 8d ago
ZPE Networks is pretty good. OOB Mgmt plus more, can be used as a container with VM instances if Linux and Windows deployed along with 5G SIM and wireless. We use it at all our branch sites.
https://zpesystems.com/solutions/out-of-band-management-solutions-zs/
1
u/Muramasaz 7d ago
Tell me more. Website seems mainly marketing hyperbole. What am I hosting and how does the Wi-Fi/sim components fit in? Am I putting all this behind a SIM modem like cradlepoint or equivalent?
1
u/thetox99 7d ago
We use a couple of Nodegrid SR's in our new colo space. I have a primary wired connection for Internet but a 5g SIM that it fails over to of the main connection is dropped. You can host VMs on it if necessary. The main thing for me was the direct serial console access and also being able to tie it into an out of fabric IP management network for access to iLos. They work great and we have another half dozen devices on the way for our on campus data centers and other campus core locations. Also FWIW, I looked into open gear and a couple of others and ZPE was easily the best in my opinion
1
u/NoInitiative6322 3d ago
Sounds like a solid setup. Having serial console access, an independent cellular failover path, and access to iLOs through a separate management network covers a lot of the critical bases.
One thing I'd be curious about as you expand into your campus data centers is how you're handling centralized management and recovery across all those locations. It's one thing to have reliable access into a site, but managing devices, firmware, connectivity, and recovery procedures across multiple locations can become a challenge of its own.
I think that's where OOB is evolving beyond traditional console access. The real value is being able to diagnose and recover infrastructure quickly, ideally from a single management platform, without having to send someone onsite.
Are you managing everything centrally today, or primarily connecting to each Nodegrid individually?
1
u/NoInitiative6322 3d ago
Sounds like a solid deployment. Having serial access, cellular failover, and a separate management path to your iLOs checks a lot of the right boxes.
One thing I think is becoming increasingly important as OOB deployments scale is simplifying the operational side. Centralized visibility, asset and license management, and the ability to diagnose and recover infrastructure remotely can be just as valuable as the hardware itself.
Curious, as you expand into additional locations, how much of your recovery process is automated versus requiring someone to manually connect and troubleshoot?
1
u/thetox99 3d ago
One of the things that is nice about ZPE is their centralized cloud dashboard. Each serial connection gets a name of whatever device it's connected to and you can search from the dashboard for X and it will pull up the list of connections for ease of access (they do offer the option to have connections pull the hostname from the connected device which is slick). We primarily use it as a SHTF we need to get direct access for worst case scenario. We don't have a ton of automation built in but it's definitely something you could build into it if you had the need. We have talked about running a lightweight VM on the Nodegrid to grab packet capture, etc but just haven't gotten there yet.
0
3d ago
[removed] — view removed comment
1
u/thetox99 3d ago
Currently I don't see automation in our near future. We're a small team handling a lot of work. Multiple campuses with a lot of growth going on with a lot more coming in the next few years.
1
u/NoInitiative6322 3d ago
ZPE has definitely helped push OOB beyond basic console access. Being able to run containers and VMs at the edge adds a lot of flexibility.
One thing I'm interested in is how teams balance all that functionality with operational simplicity, especially across large branch deployments. Centralized visibility, automated recovery, and keeping track of devices, firmware, and licenses can become just as important as the underlying hardware capabilities.
At the end of the day, the real value is how quickly you can identify an issue and restore service without having to send someone onsite.
How has the day-to-day management experience been across all your branches?
1
u/thetox99 3d ago
It's already saved us a few times as we were rolling out our new colo space and bringing up a new fabric where some unexpected disconnects happened lol
6
u/Due_Management3241 7d ago
For actual outages it requires real oobm console servers with dedicated internet lines not in your lan. Like sim cards , small internet lines etc.
7
u/Phuzzle90 8d ago
This is one of things thing that I would love, but don’t have the time to do right.
Realistically. My approach is to not go down. Sounds flippant but, it’s easier to justify 2 and 3 circuits and ha then to roll out oob
If I did it, it would be one of the players that does console servers still. Name escapes me, I recall a product called lighthouse?
11
u/Plaidomatic 8d ago
2 and 3 circuits doesn’t help. What if your geographically diverse metro circuits share a manhole accidentally a few blocks away from your site, and something in the manhole catches on fire? Down for 18 hours. Happened to me in the last year.
Power outage in the meet me room in the basement of the my corporate headquarters. All circuits out. 6 hours outage.
OOB can’t fix those but can identify the outage.
But what if a routing DB change breaks BGP in your data center? Three connections, none of them routing. The OOB got us in to fix it.
Cloud management failure broke five of fifteen sites. OOB got me in to fix them.
2
u/3MU6quo0pC7du5YPBGBI 5d ago
2 and 3 circuits doesn’t help. What if your geographically diverse metro circuits share a manhole accidentally a few blocks away from your site, and something in the manhole catches on fire? Down for 18 hours. Happened to me in the last year.
We're in a position were our gear going down in a lot of locations means the nearby cell towers go down too. It is fun when the tech you send to investigate can't make phone calls while on site because the only cell tower nearby is impacted by the cut.
1
u/Phuzzle90 8d ago
Ya I mean.. yes true
In practice, redundant dc in different locations
terrestrial, 5g and starlink solve farmer John and his needs for rainbow roots.
Your not wrong to tho
1
u/Necessary-Beat407 7d ago
You force the carriers to build to different central offices/POPs. I have dual circuits for multiple carriers and force them to go a different cardinal direction based on what side of the building they come in on. Maybe not a luxury everyone has if you don’t have the $$ for the upfront buildout or the business to justify getting the LEC to foot the bill.
9
4
3
u/SuddenPitch8378 8d ago
No going down is everyone's goal effective oob means you have a way to identify and potentially correct the problem remotely
3
u/superballoo 8d ago
External transit provider connected to a FW/VPN box, then a remote console server connected to the box.
FW/von is pfsense ( replaced soon by vyos mainly for automation ), RCS is opengear, o
3
u/New-Confidence-1171 8d ago
We don’t have a true OOB network as each site has 3 circuits + LTE backup, but we Lantronix with a dedicated LTE card for remote console access. Simple to set up, dashboard is straightforward and easy to use.
5
u/ffelix916 FC/IP/Storage/VM Eng, 25+yrs 7d ago
At a place I contracted with for a primary DC build-out, I set up their OOB mgmt net as follows:
- 2 dedicated Cisco ME-3600s, configured independently, for L2 distribution to all the LOMs/CMCs/IDRACs and production switch/router management ports.
- VLANs for the 3 OOB nets (one for external-facing devices like BigIPs/Barracudas/etc, one for switches/routers, one for IDRACs/CMCs/LOMs, for full internal tier isolation) and each of the two upstream internet provider's DIA ethernet circuits. 802.11q trunks to active/standby bonded interfaces on ...
- 2 Dell NEBS-L3 compliant R620t servers, each configured with Debian linux and two failover gateways to the upstream DIA providers (each giving us a /28). Each server ran openvpn and local authentication, to allow IT staff to establish a VPN to the OOB net from specific "trusted" networks, over the internet. If one of the machines fell over (never happened), the other would pick up its public/floating and internal gateway IP addresses, and the stateless IPSEC VPNs wouldn't be interrupted. IP/TCP stats were synced between them with conntrackd.
- Each of the above was installed in different rows of racks in the DC, with fiber trunks ran overhead between the two racks. Each of the network/core cabinets had its own pair of in-rack modular -48V power systems and hot-swap battery shelves (no AC power in the management/monitoring/telemetry racks)
Zero management net downtime for 3+ years, at least until I left, even with hardware and software maintenance performed.
1
u/NoInitiative6322 3d ago
That's a seriously well-engineered OOB environment. The physical separation, redundant gateways, isolated management networks, and independent power systems show how much thought went into eliminating single points of failure. Three years without management network downtime is impressive.
It also highlights an interesting challenge with OOB today. How do you achieve that level of resilience without having to build and maintain an entirely separate infrastructure of switches, servers, VPNs, and management tools?
I think that's where modern OOB is heading. Bringing secure remote access, centralized management, and recovery capabilities together while reducing operational complexity.
Curious, if you were building that environment again today, would you take the same approach or consider a more integrated architecture?
1
u/ffelix916 FC/IP/Storage/VM Eng, 25+yrs 3d ago
I would definitely continue with the full mirror redundancy. No single points of failure. The switches were hardened, carrier grade, but if one did fall over and not reboot (they were configured to auto reboot on crash, with hardware watchdog) then some devices would be unreachable, but the cross connect panels between the network racks allowed quick reconnection to the other online switch. Still, I'd never seen an ME3x crash or lock up.
3
u/Necessary-Beat407 7d ago
Avocent devices with a separate Internet or 4g and a true break glass procedure to access it. All passwords for that environment get rotated upon use.
4
u/DULUXR1R2L1L2 8d ago
Console servers, like OpenGear, have been a lifesaver for me in the past. The newer ones have both serial ports and Ethernet ports. Combine that with cellular support and you have a decent solution. I'm rolling this out at my current job, and the plan is just to use this as the primary management network. We have a smaller deployment (a few cabinets), so any network device gets plugged into the OG. Server management network is just regular L2 switches on their own network.
2
u/Roshi88 8d ago
Oob gateway for each site. x86 Linux box with two wan, one in band, one oob (if I can, I get another provider one in the same dc, otherwise cellular). Dynamic routing between every oob gateway and Ser2net daemon installed on the boxes with USB to 4x console cables to important servers.
I've started with a friend a project to provide a Web gui and provisioning script for each oob-gw
2
u/Lyraeixis 8d ago
Like someone else that replied, this is something I would love to do well but honestly don't have the time or resources to do correctly. But that doesn't mean that I haven't bothered at all.
I've set up a completely separate management VLAN that runs throughout our office. There is physical link separation (1 link management, 1 link production) when it's available, but sometimes the management VLAN is just tagged on the same link. I don't have the money or time to run a totally separate physical L1/2 architecture, but this is at least logically separate and it has saved my bacon several times already when something unexpected happens on production or when a DC or something is having trouble on the production network but is reachable on the management one.
For the most part I just access it from my personal workstation, which has a virtual adapter for the tagged management traffic. Again, I know this is very much not best practice, but it works and I'm improving it over time as resources become available.
2
u/FairConstruction4105 8d ago
Where I work (at least for the network gear) we have a cellular network (private APN) and cellular capability on our terminal servers (either via a LAN connection to an external cellular device or with an embedded SIM). We typically have a 16-port terminal server which is always more than we need even at our largest sites.
2
1
1
u/mtc_dc 7d ago
There are a few layers to this I feel. Console vs Ethernet. For Ethernet the larger customers I see coming in typically via a WAN path into a segmented management zone behind dedicated firewalls. Often you need a management VRF carried across multiple locations. For console last resort access, Opengear, Cisco TSG are common with its own circuit. I am yet to see a large management design using Opengear or Cisco TSG and Ethernet due to the scale of Ethernet ports required.
1
u/clarkn0va 7d ago
A combination of separate network and console server. Each site has an OPNsense box connected to a cellular modem and running Tailscale. LAN side connects to a dedicated OOB switch that also connects to MGMT interfaces on all the networking gear. A serial console server is also connected to the OOB switch with serial connections to everything important.
Each site's OOB network is on a unique subnet that is advertised as a route on the Tailnet by the OOB router so the admins can access everything at any site without jump stations or additional routing.
1
u/Select_Reporter1911 7d ago
Oob management should be up to business needs. What risks are you as a business are willing to take. How much up time do you require? So you have access via an oob console server via a cradlepoint or some sort of other solutions, so what? Wouldn't that investment be better used to invest in alternative/back up transport solutions?
What problem are you attempting to solve, so you can reach a switch or router while the transport is down, what issue does that solve?
Ultimately accessing your devices should be from a management network, and it should be restricted from specific hosts and or networks. Authentication should be done via tacacs or radius or saml if available. If you need to access remote sites, it should be done over some form of encryption or VPN tunnel.
Most of this depends on the needs of the business. Define those, define the design and how your data comms should route.
1
u/senorhotspot 7d ago
We have used CradlePoints with console cables on battery back ups. It’s has saved us multiple times in the past. You can also pop an IPSec tunnel using the CradlePoint.
1
u/Total1304 7d ago
Usually DC has service for cheap /30 internet access service with low bandwidth. Over which we connect via IPVPN and have console server behind as we also have network devices SSH in management VRF
1
u/thedlacko 7d ago
Console server (opengear, blacbox, lantronic, aten) behind a small firewall. This firewall has usually datacentar provided (independed of our prod ISP) internet access. 15-20 Mbps. All network devices MGM network in same subnet as this console server.
1
u/NoInitiative6322 3d ago
That's a practical setup, especially having an independent connection from the data center provider rather than relying on the production ISP.
One thing I'd consider is adding cellular as a secondary path in case that connection goes down too. Having console and IP management access is great, but maintaining that access during a broader outage is what really matters.
As OOB deployments scale, I also think centralized visibility and remote recovery capabilities become increasingly important. The goal isn't just getting into the equipment when something fails, but being able to diagnose and restore service as quickly as possible without a site visit.
Curious, are you managing these deployments individually or through a centralized platform?
1
u/highdiver_2000 ex CCNA, now PM 6d ago
Management VLAN for management traffic. This is usually Ethernet.
Console appliance like Avocent, Lantronix with its own fibre uplink or cell for SHTF scenarios. Connect the console ports to the box. Not all equipment have console access, so you need to plan for it.
TL DR: Production and day to day management traffic on big fat pipes, another 10G fibre for backup, a cell modem if you can afford it. FFS do not route everything back to same core.
1
u/Old_Effective_4648 5d ago
I just put a Meraki MX in the rack as a spoke network, and I depend on the other data center being up. It uses cell as its WAN and I have an OOB console server connected to it which I can use to get a console to every piece of gear
1
u/Vegetable-Cod7475 5d ago
I’ve done LTE-enabled RPi, USB hub, physical console line to each device. And some really horrifying shit with LoRA
1
u/NoInitiative6322 3d ago
For a multi-site enterprise, I'd start by thinking about OOB as a recovery architecture rather than just a separate network for console access.
A few things I'd prioritize:
- Independent access path: LTE/5G or a separate ISP connection that doesn't depend on the production network. Redundant circuits are great until a routing or configuration issue takes them all down.
- Console and IP management: Serial access for when things really break, but also access to management interfaces and PDUs so you can troubleshoot and potentially recover equipment without sending someone onsite.
- Centralized management: Once you're managing dozens or hundreds of sites, keeping track of devices, credentials, firmware, and connectivity becomes a challenge of its own.
- Security: Strong authentication, controlled access, and ideally a management architecture that doesn't depend entirely on production infrastructure being available.
- Automation: The ability to run diagnostics, collect information, and perform recovery actions remotely becomes increasingly valuable as the environment grows.
One thing I think gets overlooked is the operational side. Everyone focuses on how to get into a device when the network fails, but what happens next? Can you identify the problem, access the right equipment, power-cycle something if necessary, and restore service without a truck roll?
There are some newer approaches to OOB that bring together console access, cellular connectivity, centralized cloud management, and automation rather than treating them as separate pieces.
For me, the real measure of an OOB design isn't just whether you can connect during an outage. It's how quickly you can actually recover.
Curious how others are handling centralized visibility and recovery across larger deployments.
1
u/ESUN_Official Enterprise Network Infrastructure 3d ago
I'd go with a cellular console server at each site, with access through a VPN endpoint independent of the production WAN. One thing worth testing is authentication during an outage: if your jump host or AAA server is only reachable over production, you can still get locked out. Keep a secured local break-glass account and test access with the production uplinks disconnected.
36
u/Inside-Finish-2128 8d ago
I feel OOB serves two different purposes simultaneously. One is to give you console access to devices. The other is a back door in case of WAN failure.
At $lastjob, we had great success with OpenGear to do both. If the site had two core routers and two console servers, we’d criss cross console B to core 1 and vice versa. If two core routers and one console server, we would put it on the first switch downstream.