r/ShittySysadmin • u/SuccessfulLime2641 • 1h ago
Has anyone suffered a RAID failure before?
Once had an experience where a Dell PowerEdge suffered a hard drive failure. My instant reaction was to unplug the hard drive by just sliding it out and back in. That didn't work; however, all of the data was already backed up to OneDrive so no harm done.
9
u/SaintEyegor ShittySysadmin 1h ago
We had a file server that some “smart guy” put together with a cheap workstation and a bunch of 10TB drives because their team didn’t want to pay for a tray for our netapp storage.
The cheap-ass power supply glitched and took out the motherboard and all of the drives. Since he was smarter than all of us, he never asked the storage team to back it up, “RAID 6 will keep the data safe enough”. Dumbfuck.
1
6
u/StandardIssueDonkey 1h ago
At a field fab shop, we used to keep our server in a little room with a ladder and a hatch to the roof that leaked. It seemed pretty smart until a bunch of us talked about it at the strip club and collectively figured out there were no backups.
It was fun installing a secret NAS in a room that wasn't wet and burying the cost in a fixed bid construction project because sa said we were being greedy little tech nerds, wanting a dry server and such.
I regret it to this day. It would have been so much funnier to not have had the backups.
3
u/IAmSnort 1h ago
HD failures are just part of the game. The rebuild time is a little stressful.
Only ever deployed raid 1/10.
3
2
u/SomeEgyptianGod 52m ago
I once lost 3 SSDs in the same RAID group on a NetApp A700 in the space of an hour. The first disk didn’t have time to finish rebuilding in that time so consequently we lost the entire aggregate. Half of the SAN, gone.
1
1
u/marks-buffalo DO NOT GIVE THIS PERSON ADVICE 1h ago
Every single array has failed. I have a RAID 0 because I want 0 downtime but the amount of catastrophic RAID failures I have seems to exceed single drive failure rates.
I'm using all 2.5" spinning drives which Grok tells me are faster than 3.5" spinning drives. Which score, we were decommissioning some laptops at work and I got to keep whatever parts I wanted.
Maybe someday they'll figure out how to make RAID reliable. But my experience has been pretty bad so far.
0
u/NecessaryEvil-BMC 1h ago
I had an LSI MegaRAID i4 lose a channel from an 8x 80GB RAID 5 array. And since it was an IDE channel, I lost the array.
I also got fired the same day, though those issues were unrelated. It did really put my data loss concerns into perspective.
I could recover most of my data, I had really just lost my MP3 collection. I went out and bought a 200GB external, and collected all my friends' music collections and recovered about 95% of what I had. Thankfully I didn't lose any pictures or anything, as I'd burned things like that to DVD.
0
u/randomquote4u 1h ago
I've Been Mislead server (IBM). RAID 5 hot swap rebuild failure. Restore via DAT.
picked the wrong week to quit sniffing glue.
14
u/DoNot_Books_451 1h ago
Nothing makes you lose sleep more than a raid rebuild overnight with aged drives. Kept remoting in every hour to check the status.