jooola Could extend your reasoning on this one ? I don't see any problem with managing docker-compose specs using ansible.
Sure! Last time I used the compose plugin from the docker community selection I had several issues, especially with managing the state of the containers. Additionally it is only compatible with compose version 2, 3 is current and it still relies on the pretty old, deprecated and outdated python library for docker. IIRC they are working on implementing the more up-to-date go docker library, but there is no release scheduled yet.
We could surely place compose files using ansible and run
docker compose commands in the directories, but this might introduce several other issues, at least ansibles recommendation is not to run commands if plugins are available. However, I am not a huge ansible expert and I might be wrong here.
jooola Funkwhale's infra doesn't really need to "scale", maybe vertically, but I doubt it needs to scale horizontally. Or if scaling is needed, distributing the services on multiples server might be a better solutions.
Well, it would be great to have scalable storage and if open.audio grows in the same way it did the last months, we might want to scale it. But even if that is the case we can move open.audio to k8s while having everything in k8s would be insanely expensive (either money-wise for managed k8s or required effort to manage it on our own).
I think @JuniorJPDJ mentioned k8s because we both have a professional background in this area and really like the tech, but funkwhale is likely too small.
jooola What are the requirements in terms on security ?
From the past experience I want to be able to revoke access if people are not around anymore. So distributing a root password is likely not an option. Generally I would try to give as less access to data and secrets as possible but enough to keep people able to work. I think right now we have all secrets stored in a self hosted bitwarden, it might be ideal if we are able to either use this store or switch to one which is a password manager and allows us to read the secrets from the server, too.
Between having the secrets stored in an ansible-vault file, or in the Gitlab CI variables, to setting up a Hashicorp Vault, I think we can find something that fits our needs ?
Sure, the question was more or less regarding the idea of managing pure configs inside of a git repository. Right now I moved all the secrets into env vars, which are referenced in the configuration files but not inside git itself. This is something, but obviously fails if we need to recover a service on a new maschine since the secrets would be missing. If we pick something like ansible, its all so much easier. (Except the concerns I mentioned above)
jooola What is the details of the current hardware ? Is that meant to change ? What is the budget ?
We have a Hetzner AX41 with two additional 960 GB SATA SSD Datacenter Edition https://www.hetzner.com/dedicated-rootserver/ax41/?country=de (about 75€/month). I don't see any need to change this right now, my main concern would be to make the software running there somewhat more maintainable and decrease the bus factor in case of incidents.
I don't think we have a proper voted budget yet. We currently spend around 100-150€/month for our infrastructure, we have additional S3-storage and the domains are quite expensive as well. I think we should stay in this range for a while in order to make sure we keep the lights on as long as possible.
jooola How should the data storage be handled ? How much storage does the project need ?
Well, our server is full from time to time, mostly because open.audio takes 500-600GB storage. I am currently in the process of migrating the files to wasabi, so we have some more headroom on the server again. Gitlab itself is also using quite some storage for artifacts, the registries ... but I don't have numbers at hand right now. Maybe @egon0 can help here?
jooola I think an ansible based project would be the simplest and most maintainable solution.
I'd agree in general, except of the concerns I raised above regarding the compose stacks. And I really don't want to mess with systemd unit files to manage containers, which seems to be quite common these days, too.