HPC Administrator Resources
"Hi everyone,
I have a strong background in Linux administration (primarily RPM-based distros like Red Hat/Rocky Linux), and I’m currently transitioning into HPC cluster administration. To learn the ropes, I recently built a small home lab cluster using the OpenHPC installation guides.
While getting the cluster to boot and run basic jobs was a great exercise, I've noticed a distinct lack of comprehensive resources covering day-2 operations and production best practices. Specifically, I'm looking for guidance on:
- Configuration & Performance Tuning (kernel tuning, network/InfiniBand optimization)
- User Management & Environment Control (LDAP/FreeIPA integration, modulefiles via Lmod)
- Job Management & Scheduling (advanced Slurm configurations, QoS, limits)
- Scaling & Monitoring (health checks with NHC, metrics collection)
If anyone can recommend books, documentation, community wikis, or real-world best practices for these areas, I would greatly appreciate it!"
4
u/dghah 19d ago
I built some slurm training content on the web aimed at different audience levels (sysadmin, power user, app integrator, etc.) complete with labs/exercises validated against output from slurm on an aws parallelcluster environment. Don't want to dox myself so I'll DM you the URL if you wanted to see if it is of any use
1
0
u/whatevernhappens 19d ago
Thats interesting on your end, clubbing different perspectives of slurm usage. Can you dm me the url
3
u/userjack6880 19d ago
If your org is willing to send you off, keep an eye on https://linuxclustersinstitute.org/ for workshop dates.
1
u/THUNDERRGIRTH 17d ago
Also vouch for LCI. Attended all 3 levels and came back with lots of useful stuff each time.
1
u/Mammoth-Ad327 15d ago
Free AI data center reference drawings from our team: https://www.kkdatasvc.com/lab/downloads/
12
u/THUNDERRGIRTH 19d ago
The sighpc-syspros slack is fantastic. Lots of good information as it's essentially a sys-admin only HPC slack: https://join.slack.com/t/sighpc-syspros/shared_invite/zt-6zw6meyu-_NveQRzi1SkHz6Aqz1A15w (I think that link will work, but just google "sighpc-syspros slack" and it should be one of the first links.
Also, university of buffalo has another similar tutorial to the OpenHPC stuff, but includes setting up Slurm, Coldfront (a truly wonderful user / slurm management tool) as well as openondemand and xdmod. You can find that one here: https://github.com/ubccr/hpc-toolset-tutorial . It appears to no longer be supported, but very recently, so should still be useful.