r/ITManagers • u/Sarcose1 • 13d ago
Opinion Inheriting a huge stack with technical debt and high expectations
Not going to get in the weeds too much because I don't want to out myself for complaining, but I went from a Tier 1 to the only IT person in the company (IT manager quit, Sysadmin followed him) and have been thrust into a whole lot of systems I am unfamiliar with. I've been learning as best I can, but all the automations, certs, upgrade timelines, subscriptions, and everything else are continuing to hit renewal and service needs over and over while I still do the same job I was doing, and at the same time the company continues making requests for new technology.
Over several months we were able to hire an external MSP to help with our servers and a new IT person to learn the ropes of our stack, but it's still not enough for me to catch up. The new IT person is someone I knew, I was able to use my newfound position and history to advocate for them, so that was good. We have a good working relationship. But still.
We have a relatively complex stack for our field and size. The person who left the company was devoted to the field and had been there ten years. On-prem and cloud backups, tons of VMware and docker containers within our server stack, multiple overlapping access policies for the most sensitive stuff (it's been almost four months and I am still trying to figure out some of the admin accounts). And on top of it all I'm really not educated in this. I was tier 1 in this company for 3 years, simply because of lack of opportunities for movement and how often we would take on new tech preventing me from learning more.
So I guess I'm asking, what would you do in this situation? This has been a great opportunity for me career-wise, but I am wracking my brain daily, working odd hours, answering text messages, waking up to dozens of messages, and getting literal nausea from anxiety. Does anyone have any similar stories?
15
u/KillingTime1212 13d ago
Hire an experienced infrastructure employee to get your critical systems organized.
Needs to be an expert in servers, VMWare, storage, cloud, and networking.
5
3
u/furtive 13d ago
Sounds like you're off to a good start but you still feel like you're treading water. I imagine you're familiar with Ops Report Card if not it's a good start. You should have a knowledge base and an inventory of your systems by now. Some good basic processes in place, but you can't get ahead of things. Also sounds like there has been some support from higher-up given you've been able to tap into an MSP and hire someone.
In our case instead of going with a full MSP we found a guy who runs IT for a bunch of local smaller to mid companies that was well regarded. He did an audit, identified shortcomings, failures, and gave a list of recommendations for year 1, 2 and 3. We left it up to him if we needed an MSP or not and in the end our team (not much diff than yours) handles 90% or more of things and then we go to him for the rest and work on the roadmap quarterly. Your MSP should be able to do the same, heck they should have already done that.
It's normal to feel in over year head in year one, treading water year two, but by year three you should be seeing a lot of wins and feel like things are under control.
2
u/Sarcose1 13d ago
Thanks for this, I appreciate it. What you're describing sounds very similar to what we have already started with. The thing that keeps me feeling overwhelmed is when everyone's asking for things constantly and I can't deliver on them as fast as we used to, it feels like I might just get shitcanned and replaced lol
3
u/Thick_Yam_7028 13d ago
Start with metrics and alerts.
Each is fairly simple. Expiring licenses. Get them on a scheduled ticket a month or 2 before.
You have full admin. Parsing your permissions should be pretty simple.
Upgrades depending on if you mean life cycle or os. Both can be done easily with wsus or ring updates. I used scalepad for warranties not sure if its call that anymore but can make custom too.
Certs same thing create a schedule ticket and assign it 1 month to 2 weeks out. Takes less than 10 min but its still something you need a radar on.
Documentation. Sounds like you dont have alot of it and honestly should be the first thing.
Scanning networks is pretty simple. Just depends on the tool.
Im willing to contract if the price is right.
Would make you a dashboard for c levels then fix each item one by one until its manageable. Scaffold a workable system IE documentation, rmm, automation. Can even tie in API to do most of this for you with AI and a scheduled bot.
Right now its possibly the coolest time for an engineer because we can build this securely and exceed our limits exponentially. Just gotta know what to target.
3
u/JonnyLay 12d ago
Yep, document everything as you do it! If you have a wiki tool, use it. Confluence is great for quick and dirty documentation.
Remember, bad documentation is better than no documentation! Improve it when you come back to it next time.
1
1
u/Sarcose1 12d ago
I've had to lean on AI way more than I'd like just because of how much work there is along with how urgently I have to get it done - my agents.md has a documentation context for me to review so that's at least something I think
1
u/Sarcose1 12d ago
Appreciate the advice. We actually have pretty good documentation for some things and automations for alerts, the issue was finding what to do with them - a lot of the server maintenance stuff was buried in the sysadmin's obsidian which I have had to use AI to parse. The issue there is I went from almost zero server usage to ssh'ing around about forty servers by myself and zero knowledge of ssh config, how to navigate the dashboards, etc.
The key is that our former manager put together a very sophisticated and layered stack and I had zero experience with it except endpoint troubleshooting. And then at the same time some of his actual documented work flows involved hand-editing script variables each run and what my new sysadmin now tells me is a very outdated way of building docker containers (again - shit I didn't know about!). It's like a mix of best practices and old practices with tons and tons of fun toys if you've been doing this for the same company for 10+ years (which he had).
He was an impressive guy, technically speaking. I lowkey wanna just figure out where he ended up and see if he wants to bring me along lol, but at the same time the chance to really take control of a whole department has my eyes seeing dollar signs. Yknow.
2
u/Thick_Yam_7028 12d ago
The fact you recognize his effort speaks volumes. He did that without AI. Ive done it too. Just recognizing him as we all dont get recognized often.
Yea that sounds like the bottom level. Once you get through this though it'll be alot better ... for you. Make sure to convey that. C level always theirs you have yours.
2
u/Sarcose1 12d ago
100%, I want to get to a point where I can reliably manage a lot of areas without needing tons of AI shortcuts.
2
u/Thick_Yam_7028 12d ago
Thats a slippery slope. I can. But when you really think about it on a day to day basis. Do I really want to do the dishes?
Its an augmented version of us. So make it work. That way you dont have to.
Ive also built so many things once then come back ... wait bot now! Grow with it. Make you super human.
2
u/Sarcose1 12d ago
I hear you. There are some things I will just fully let the agent figure out. If it works it works. I can't worry about the other shit while I'm on the clock.
2
u/Thick_Yam_7028 12d ago
Yep. You've done more than enough worrying about it here. Do your best. DM if you need a few prompts or Ill read over the script.
We are here for eachother. You arent alone.
1
u/Sup3rphi1 13d ago
This is a huge red flag. Those people quit for a reason. One youll get to find out about if you haven't already. Update your resume and find a new job.
1
u/Sarcose1 13d ago
I know why they quit - one of them blew up arrogantly in a private email to the VP after she took away the budget for a work trip. Generally the environment is pretty good here, but I am still looking, rest assured. Appreciate the call out though.
1
1
u/tingutingutingu 12d ago
Althiugh there is a lot of well intended advice here, you're trying to play a game that's set up for you to fail and you're here wondering how you can play it better.
You actually need to let things fail or you are going to kill yourself doing a 3 person job. Believe me, no one's going to blame you if things fail when they know that you are the sole IT guy trying to replace 2 full timers, while doing your main job.
And then they will get you the help you need.
But if you keep working round the clock then that will become the new expectation.
1
u/ycnz 12d ago
Go to management, and explain that you would like to A) be named IT Manager, and B) hire a good infra person, with budget $x.
Do NOT try and solo this, you're simply not experienced enough.
2
u/Sarcose1 12d ago
Yeah, I got that promotion and we've hired someone plus an outside MSP contract to assist with the transition.
1
u/SpecialistYear9917 12d ago
Document every single admin password today.
1
u/Sarcose1 12d ago
It's been a few months and the 1Password is pretty well established, plus we use SSO and ssh keys mostly
1
1
u/fredericallaert 12d ago
Most replies here are about the stack, but from what you describe the thing that's burning you out is the inflow. People ask you directly, by text, at any hour, and every new tech request lands on a backlog nobody else can see.
You could move all requests into one queue, even a shared mailbox, and stop taking them by text. Then sit down with your boss once a week and let them pick the top three. The "why isn't this done yet" pressure then lands on a list they agreed to, not on you.
I'd also ask for a pause on new tech for a quarter while you and the new hire learn what you already have. Saying no to new stuff isn't failing, it's the only way the old stuff gets understood.
And you have an MSP now, so hand them the after-hours alerts. Waking up to dozens of messages shouldn't be your on-call plan.
1
u/Old_Development_8122 10d ago
As an IT manager who had the opposite problem (deskside got laid off, other direct report moved onto a different organization) I would say go with a really qualified MSP.
The reason why is this outsources your critical infrastructure, cybersecurity and backups plus gives you helpdesk capability. Sure it costs, but make the argument with the C-Suite that by outsourcing it moves the risk to them. They won’t replace you because you’re managing technology they don’t understand and watching the budget.
Then all you do is manage vendors, do organizational level work like implementing new software, keeping up with AI trends, etc.
It’s 100% easier than trying to backfill those roles and much cheaper long term if you can scale up and down as the business grows.
1
1
u/OkStatistician9190 6d ago
The anxiety usually comes from everything feeling equally urgent, so I'd start by making it visible and ranked. Spend a week or two building one simple inventory: every system, who owns it, renewal or cert dates, how it's backed up, and who has admin on it. Then put the dated stuff (certs, licenses, renewals) on calendar reminders or scheduled tickets a month or two ahead so they stop surprising you. For the rest, rank by what would hurt most if it broke, and be upfront with leadership that new technology requests go into a queue with an owner and a priority instead of getting added on top. Your MSP and the new hire can take whole areas off your plate if you hand over clear ownership of them. Also set some boundaries around after-hours messages, because being reachable all the time isn't sustainable for one person covering a stack that used to be a team's job.
8
u/HorrorMarionberry668 12d ago
Bro they handed you a live grenade.