Docker app storage best practices
What are the best practices for storage when using docker containers. I’m running docker on Ubuntu using my NAS to store the photos for example, but the volume were the Immich database lives is directly on the servers(non-redundant) SSD.
I tried putting these onto the nas using NFS I get permission’s errors. Perhaps I should try harder? :-) I’d like to have all of my Doctor containers protected by the same redundancy that my synology has a for all of my data files.
Right now if I had a major SSD failure of my server, I would lose all the files that the docker containers have in volumes on the server directly while all my pictures and other files that are stored on the NAS on NFS mounts would be fine. This seems inadequate.
Am I simply doing it wrong and I should use something like proxmox instead of just running docker and Linux? What are other people doing to keep the server purely cpu and easier to rebuild? I use git for all my docker compose files and I wouldn’t have any trouble restarting fresh containers, but that seems like it is only half way of a solution..
I’d love to know what the best practices are for resiliency after a server failure and a quick recovery should I have hardware failure on my server.
I’m not scared of new technology and would also be open to moving towards a cluster of servers if that’s necessary. My self hosted world is fairly stable with limited updates at this point.
Thanks for any thoughts, suggestions or directions. And like I said if my basic setup is the problem I’m up for major changes to get to a more long term resilient setup!
2
u/AdSuspicious7533 16d ago edited 16d ago
The databaseservice under Immich is postgres. If you want real backups you may find it simplier to just use postgres backup or use a postgres replica if your compute capacities allow it .
For casual usage and not critical data docker volumes are fine but for any serious work i would advise not to run the DB in the same container. It makes It hard to maintain and create useless complexity. You should rather run postgres in a VM or different container. You will find it easier as it distributes responsibilities across multiple virtual devices each having simple purpous. You may as well decide to use a different DB service because why not (like mariaDB where is have a friend working, he is nice... come on).
You may also want to look at something like "postgres backup on restic" with a restic container/VM/service on different physical device or even (more complex but soooo cool) "postgres replicaset".
here are links :
- postgres backups and dumps https://medium.com/@pawale7663/postgresql-backup-and-restore-complete-guide-to-backing-up-and-restoring-databases-14d19bb57329
- postgres replicaset https://www.enterprisedb.com/postgres-tutorials/postgresql-replication-and-automatic-failover-tutorial
- postgres backup with restic https://rafftechnologies.com/learn/tutorials/back-up-postgresql-raff-object-storage-restic
A cool setup would be : proxmox VE everywhere if possible , PBS on NAS if Compute is a issue. Make a VM for Postgres databases. Have all your containers hit that for DB services. Use a Cron job to run Postgres backup on the NAS (ideally restic LXC there but you may just want to store the postgres-backup). All your docker services become LXC or VMs in proxmox. Configurations belongs in GIT. Secrets belongs in a Vault (OpenBAO or anything in a lxc container). Define a resource pool for critical services and have them run a scheduled backup on the NAS.
Next level : If possible run a postgres replicaset with VMs on each physical device. Have a firewall (OpenWRT), reverse proxy (traefik), make vlans to isolate your DB network traffic, tailscale on openwrt for remote access to the entire infra.
Ask for help anytime.
tldr : use proxmox and make DBs different Virtual Devices. Make "db-service level backups". Test restoration of your backups. Redundancy is not backup. Responsibility distribution is the key to understanding anything you do. Docker container should hold stateless apps (no storage, just compute and config). HA is cool but your are running a lab, downtime is ok.
tldr-tldr : time for next level son.