Moving away from Gitlab?

gcrk

Gitlab is our major development platform and we are using it for years. Most of the time its doing a good job, however I do have some issues with Gitlab as a platform and I therefore want to start a discussion about moving to another tool.

Please note that this isn't something we will be able to achieve in the closer future. We are highly locked into Gitlab for several reasons:

Our development process and project management is aligned with the features provided by Gitlab
We are using Gitlab CI for all our pipelines and migrating is a lot of work
We are using Gitlabs internal container registry
We are using Gitlabs Service Desk to manage inbound mails regarding open.audio
We are using Gitlab Pages to serve static pages

This means a migration is a huge amount of work and should be well thought and discussed.

However, there are some problems with Gitlab that might require us to take those steps anyways.

On the one hand Gitlabs usage of resources is growing with every release and hosting it gets more and more expensive while the responsiveness and performance of Gitlab declined over the last months. We were constantly increasing the resources for our Gitlab VM, right now its by far the most expensive tool we host in terms of CPU, memory and io-load. Since we are a software development team, this is somewhat expected and alright. However, Gitlab is constantly growing, adding more services and clearly aims huge deployments with hundreds or thousands of developers, and we are not the target group.

Another important aspect is that we rely heavily on Gitlab Ultimate. Gitlab hides quite some security features behind their proprietary versions. This is also true for the approval system we implemented in order to make merging of changes less depended on single, over powered maintainers. Right now Gitlab grants us a free Ultimate license, however this is always granted on a yearly base and might stop at any time. However, we are unable to pay for the ultimate licenses, which brings us in a bad position. If Gitlab decides to stop their free Ultimate licensing for open source projects, we have to either do without the features we currently depend on or get ourself a huge bunch of funding in order to pay for the license.

A few words about self hosting: I think its a great strength of the Funkwhale Collective that we basically host all the infrastructure we need to build Funkwhale on our own. This makes us independent from everyone, even though we need to spend some resources on this. So I am in favor of a self-hosted solution and moving to Gitlab.com would only mitigate a small portion of the problem anyways.

Thats my point of view, but I'd appreciate if the people involved in development share their opinions. Personally I tend to migrating away from Gitlab and move to Forgejo. Let me know what you think so we can decide on our long-term goal here and eventually slowly move away from Gitlab in the future.

jooola

I don't think moving to a new software is a good idea.

The problems you are mentioning can be solved without such a big change:

We are using Gitlabs internal container registry

This is available for all offerings: https://docs.gitlab.com/ee/user/packages/container_registry/

We are using Gitlabs Service Desk to manage inbound mails regarding open.audio

This is available for all offerings: https://docs.gitlab.com/ee/user/project/service_desk/
I doubt there is no good open source alternative out there.

We are using Gitlab Pages to serve static pages

This is available for all offerings: https://docs.gitlab.com/ee/user/project/pages/
Using a simple web server could also do the job.

gcrk On the one hand Gitlabs usage of resources is growing with every release and hosting it gets more and more expensive while the responsiveness and performance of Gitlab declined over the last months. We were constantly increasing the resources for our Gitlab VM, right now its by far the most expensive tool we host in terms of CPU, memory and io-load. Since we are a software development team, this is somewhat expected and alright. However, Gitlab is constantly growing, adding more services and clearly aims huge deployments with hundreds or thousands of developers, and we are not the target group.

I think you can disable the features you don't need, and do some optimization. Cleanup up spam users and mining project could also do some good.

gcrk quite some security features behind their proprietary versions. This is also true for the approval system we implemented in order to make merging of changes less depended on single, over powered maintainers. Right now Gitlab grants us a free Ultimate license, however this is always granted on a yearly base and might stop at any time. However, we are unable to pay for the ultimate licenses, which brings us in a bad position. If Gitlab decides to stop their free Ultimate licensing for open source projects, we have to either do without the features we currently depend on or get ourself a huge bunch of funding in order to pay for t

Another important aspect is that we rely heavily on Gitlab Ultimate. Gitlab hides quite some security features behind their proprietary versions. This is also true for the approval system we implemented in order to make merging of changes less depended on single, over powered maintainers.

I doubt you cannot have a similar approval workflow with the free offering of GitLab. Could you define what is currently in use and only available in ultimate ?

A few words about self hosting: I think its a great strength of the Funkwhale Collective that we basically host all the infrastructure we need to build Funkwhale on our own. This makes us independent from everyone, even though we need to spend some resources on this. So I am in favor of a self-hosted solution and moving to Gitlab.com would only mitigate a small portion of the problem anyways.

If self-hosting is a pain, the project could migrate to a shared GitLab instance and maybe share the costs. This would be a more acceptable change, as you won't break the entire ecosystem relying on GitLab.

There are many out there that would probably be really happy to host the Funkwhale project. I assume you want a European based instance? Maybe https://framagit.org/ is an idea? Maybe the CCC has a global instance?

I could add some more points, but I think there is already some points to clear before going further.
Cheers!

MORE:
https://docs.gitlab.com/omnibus/settings/memory_constrained_envs.html
https://www.chatons.org/search/by-service?field_software_target_id=284

petitminion

I don't know Forgejo and I don't manage our gitlab instance. That being said I have the impression that it's less expensive to keep our dev env has it is know ( even if we have to paid for a bigger server) than to have to migrate.

gcrk

jooola I think I didn't made clear enough what my list is about. This is just a list of features we use that would make it hard for us to switch the platform. As you correctly stated, all of this is available in CE as well. Its just making the price for moving higher.

The major reason for us to rely on Ultimate is actually the Approval workflow. You cannot have required approvals in Gitlab CE. We tried having something similar in Gitlab CE and it basically meant that someone has to check by hand if something is approved, where the result was that I had to manage all the merges. Now we have a way lower bus factor there, which freed up a lot of my time for other things.

The only other EE-Feature we are using right now as far as I can tell are Epics.

But to make this clear, I am fully aware that none of those is a really hard argument against Gitlab. Its more about people being concerned we are using non-FOSS in our development and that the direction Gitlab is heading makes it harder and harder for us to run it while it doesn't really add a lot of value with updates. I know that some members of the dev team sympathize with the idea and direction of Forgejo and it seem to align much better to our standards than Gitlab does.

Using a hosted instance of Gitlab is a feasible solution as well.

jooola

gcrk I think I didn’t made clear enough what my list is about. This is just a list of features we use that would make it hard for us to switch the platform. As you correctly stated, all of this is available in CE as well. Its just making the price for moving higher.

Right!

gcrk Its more about people being concerned we are using non-FOSS in our development and that the direction Gitlab is heading makes it harder and harder for us to run it while it doesn’t really add a lot of value with updates.

Could you paste some links to back this statement?

gcrk

jooola Could you paste some links to back this statement?

Not sure which one exactly.

The concerns regarding running non-oss is merely brought up in personal conversations.

In regards to Gitlabs direction: As I already said its getting heavier and heaver with each release. Thats something we notice when looking at the resource consumption.

Additionally, Gitlab is going to implement some AI nonsense: https://about.gitlab.com/solutions/code-suggestions/
They are more and more buying into "Cloud", which doesn't serve us. When I read the latest release blog posts, more and more features are targeting bigger companies and enterprises. This wouldn't bother me if it wouldn't at the same time increase the cost of running Gitlab.

aschrijver https://codeberg.org runs on Forgejo (with some codeberg-specific modifications on top), and Forgejo in turn synchronises 2-ways with upstream Gitea (which went corporate).

I am running Forgejo privately and I am deeply in love, I have to admit. Its fast, its small, easy to operate and brings all the stuff you need without paywalling any features. test-driving the migration is indeed a good idea, however with the amount of issues we have its probably hard to verify how well it worked 😃 If I find some time, I am going to test the importer.

aschrijver

Just came here to say that you may test-drive a migration to Forgejo by signing up to codeberg.org and using the project importer. It supports imports from Github and Gitlab, and quite well, including issues and wiki's and the like. Codeberg have their own CI service based on Woodpecker (FOSS drone.io fork) and they have Codeberg Pages to host your own site.

https://codeberg.org runs on Forgejo (with some codeberg-specific modifications on top), and Forgejo in turn synchronises 2-ways with upstream Gitea (which went corporate).

petitminion

gcrk When I read the latest release blog posts, more and more features are targeting bigger companies and enterprises. This wouldn’t bother me if it wouldn’t at the same time increase the cost of running Gitlab.

Is it really a cost ? I mean do we have cpu consumption issue with the infra ? If not we could wait and see if this became a problem. Also what is the cost of migrating all our stuff comparing to have a server a with more power ?

gcrk

petitminion We need to increase the resources for the VM from time to time. This is not only about CPU, its about ioload, memory consumption and others as well. I am not maintaining the VM, so I can only report this as I was told by @egon0.

Additionally Gitlab constantly fails, the internal registry isn't reachable which makes Pipelines fail. Gitlab is slow or not reachable at all, which blocks me and maybe others from working.

Our infrastructure cost is already close to our monthly income, so if we want to spend more, we also need to raise more monthly money.

The cost of migration is surely not small, even if we are able to migrate all the issues automatically, we would still need to migrate the CI pipelines, the registries while we would loose all the links to issues made in commits or the source code (I assume the issue IDs would change when doing a migration).

petitminion

gcrk isn't there a gitlab instance at techcultivation that we could use freely ?

gcrk

petitminion Not that I know