• Support

  • Help with 1.3 upgrade using traefik as reverse proxy not nginx

I've done some more looking at what has changed in the compose file after noticing a few differences between the new one and my one (the api exposed port 5000 before and all the containers were on the default network).
A comment in the gitlab got me thinking about how the front end contacts the back end as that is my problem here. So I wondered if the install method had changed since I first installed my instance.
So I was looking at this section of the install page:
https://docs.funkwhale.audio/administrator/installation/docker.html#override-default-nginx-templates
Point 2 seems to have an issue:
the file for 1.3.0 for the second command here doesn't exist:

curl -L -o /srv/funkwhale/nginx/funkwhale.template "https://dev.funkwhale.audio/funkwhale/funkwhale/raw/${FUNKWHALE_VERSION}/deploy/docker.nginx.template"
curl -L -o /srv/funkwhale/nginx/funkwhale_proxy.conf "https://dev.funkwhale.audio/funkwhale/funkwhale/raw/${FUNKWHALE_VERSION}/deploy/docker.funkwhale_proxy.conf"

This would be:
https://dev.funkwhale.audio/funkwhale/funkwhale/raw/1.3.0/deploy/docker.funkwhale_proxy.conf

Though the above is slightly getting off track.

My /srv/funkwhale/nginx/funkwhale_proxy.conf does seem up to date with: https://dev.funkwhale.audio/funkwhale/funkwhale/-/blob/develop/deploy/funkwhale_proxy.conf

The upgrade instructions for 1.2.10 do say to update these files:
https://docs.funkwhale.audio/admin/upgrading.html#multi-container-installation
But the upgrade instructions for 1.3.0 do not mention this:
https://docs.funkwhale.audio/administrator/upgrade/docker.html

The docker.nginx.template located here is quite different to mine:
https://dev.funkwhale.audio/funkwhale/funkwhale/raw/1.3.0/deploy/docker.nginx.template
This file simply contains the line:

../front/docker/funkwhale.conf.template

Which points to a front folder that I do not have.

I feel like I might have come unstuck in the upgrade tree somewhere? I'm unsure if it matters but my install started as a mono container install and moved to the multi container install when the migration instructions were put out.

A bit more investigation...

/srv/funkwhale/nginx/funkwhale.template has:

upstream funkwhale-api {
    # depending on your setup, you may want to update this
    server api:5000;
}

where as https://dev.funkwhale.audio/funkwhale/funkwhale/-/blob/develop/front/docker/funkwhale.conf.template has:

upstream funkwhale-api {    
server ${FUNKWHALE_API_HOST}:${FUNKWHALE_API_PORT};
}

Which would make sense of my problem seeing that the containers were taken off the default network and port 5000 isn't exposed anymore.
But when I pulled https://dev.funkwhale.audio/funkwhale/funkwhale/-/blob/develop/front/docker/funkwhale.conf.template into /srv/funkwhale/nginx/funkwhale.template I still get the same error so I'm still missing something.

Did I miss an upgrade step somewhere?

FUNKWHALE_API_HOST isn't in my .env nor is it mentioned in https://dev.funkwhale.audio/funkwhale/funkwhale/-/blob/develop/deploy/env.prod.sample
It appears to be FUNKWHALE_API_IP ?
So i think the code defaults to using the container name? "api"

So I think what has happened is that this all worked with my traefik setup before now because before these changes:
https://dev.funkwhale.audio/funkwhale/funkwhale/-/commit/96c2eacb0f187b5032726e34845bff345a9c164e
https://dev.funkwhale.audio/funkwhale/funkwhale/-/commit/097f803faba58f5a8fc02050dec0b6e1997d7499
The containers were on the same network and could communicate using the container name ("api" in this case), whereas now they are not and are meant to be communicating using the external IP and port of the actual API?
This would then mean that I will need to set traefik to expose the API container on port 5000 and forward this through my router now?

Sorry for all the messages I thought I'd do some digging and try and reduce the legwork for anyone looking into this.\
And sorry if I'm barking up totally the wrong tree, sometimes a little knowledge is a dangerous thing!

I noticed that I'd made a mistake by not uncommenting the lines in the compose file " Uncomment if you want to use your previous nginx config". I also changed the funkwhale.template to read FUNKWHALE_API_IP rather than _HOST

I removed the front container and tried again. Sadly this didn't work though and I can see that the front container is still pulling in the following setup by looking at the log:

server api:5000

In my .env I have these set:

FUNKWHALE_API_IP=127.0.0.1
FUNKWHALE_API_PORT=5000
  • gcrk replied to this.

    Hi @MattDHarding

    I guess you run into https://dev.funkwhale.audio/funkwhale/funkwhale/-/issues/2146, which will be fixed in 1.3.1 (You can check the MR for the fix and apply it manually, its a path change in docker-compoye.yml.

    Also, why did you commented the ports section of the front container in you compose file? We added them for backwards compatibility, and if you want to use localhost in order to access the service, you need to have the portmapping. The internal port of the container is 80, so localhost:5000 wont work.

      gcrk yea I copied the fix for 2146 and that got my front container running thanks. But then I hit trouble for exactly the reason you point out with the localhost:5000 thing when I try to log in:

      front_1         | 2023/06/26 14:47:57 [error] 39#39: *7 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.5, server: _, request: "POST /api/v1/users/login HTTP/1.1", upstream: "http://127.0.0.1:5000/api/v1/users/login", host: "funkwhale.domain.xyz", referrer: "https://funkwhale.domain.xyz/login"

      What should these settings be in .env? I've always had them as above and it worked before 1.3.0?

      Also, why did you commented the ports section of the front container in you compose file?

      I'm using traefik as a reverse proxy and this is just the way traefik works I believe (I'm no traefik expert though). The container doesn't bind the port itself, the traefik labels tell traefik what ports to use for the container then it does it's stuff.

        Just to be clear, I'm not trying to access funkwhale on a weird port or use localhost, I setup traefik to serve funkwhale on funkwhale.domain.xyz

        I think that my previous setup worked because the previous version of the compose file
        before:
        https://dev.funkwhale.audio/funkwhale/funkwhale/-/commit/96c2eacb0f187b5032726e34845bff345a9c164e
        https://dev.funkwhale.audio/funkwhale/funkwhale/-/commit/097f803faba58f5a8fc02050dec0b6e1997d7499
        had all the containers on the same network "default" so the nginx (as it was then) container talked to api:5000 because it could use the container name.

        I'm a bit confused as to what the setup should be now, because I'm unsure how the front container talks to the api container in 1.3.0

        • gcrk replied to this.

          MattDHarding had all the containers on the same network "default" so the nginx (as it was then) container talked to api:5000 because it could use the container name.

          The default network is applied automatically if none is specified, so the diff you linked shouldn't make a difference

          MattDHarding I'm a bit confused as to what the setup should be now, because I'm unsure how the front container talks to the api container in 1.3.0

          The frontend container never talks to the API, the frontend container ships the frontend and the frontend will be running in your browser which needs to contact the API. In 1.3.0 we changed the setup, previously the API was serving the frontend static files, which was not optimal. In the new setup the first point of contact is the front container is the entry point and it will forward requests to the API to the API container.

          MattDHarding FUNKWHALE_API_IP=127.0.0.1

          If you have this in your env, this is probably cause of the issue, because the frontend container will try to contact port 5000 on itself, which isn't running there. I think you should even remove the line from your env since the docker container will pick up a sane default if nothing is set.

                  gcrk The default network is applied automatically if none is specified, so the diff you linked shouldn't make a difference

                  Ah that makes some sense, I didn't know that.

                  gcrk The frontend container never talks to the API, the frontend container ships the frontend and the frontend will be running in your browser which needs to contact the API.

                  Oh of course, this is a brain fart I keep making.

                  So, I reverted the above change that I mentioned above so my funkwhale.template and commented out those lines from my .env like you suggested:

                  #FUNKWHALE_API_IP=127.0.0.1
#FUNKWHALE_API_PORT=5000

                  I get pretty much the same issue, any ideas? The front container constantly restarts printing this error.

                  front_1         | 2023/06/27 15:34:36 [emerg] 1#1: host not found in upstream "api:5000" in /etc/nginx/conf.d/default.conf:2
front_1         | nginx: [emerg] host not found in upstream "api:5000" in /etc/nginx/conf.d/default.conf:2
                  • gcrk replied to this.

                        MattDHarding Well, I don't know what traefik does here. Maybe investigate the networks.

                        You can list your docker networks using docker network ls. There should be one named like the directory your compose file is living in. eg funkwhale_default. Now you can inspect this: docker inspect funkwhale_default. Maybe you can send me the output.

                        - "traefik.docker.network=web"

                        This looke a bit like traefik is changing the networking, if thats the case and the API container is put to a web network, it probably is not part of default anymore and cannot be reached by the frontend container (which tests if the upstream is available before forwarding there)

                            gcrk First off, thanks for all your help so far.

                            You're right about checking the docker network. I have to put the front container on the "web" network as that is what my traefik uses to communicate. But putting the front container on the web network then doesn't put it on the default network.

                            Putting all the containers into the default network then allows the frontend to run ok, but when I try to login it tells me my password is wrong and I get the following output from the containers:

                            front_1         | 172.18.0.5 - - [28/Jun/2023:07:13:15 +0000] "POST /api/v1/users/settings HTTP/1.1" 401 58 "https://funkwhale.domain.xyz/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "82.19.72.223"
api_1           | [2023-06-28 07:13:15 +0000] [10] [INFO] ('192.168.16.7', 59754) - "WebSocket /api/v1/activity" 403
api_1           | [2023-06-28 07:13:15 +0000] [10] [INFO] connection failed (403 Forbidden)
front_1         | 172.18.0.5 - - [28/Jun/2023:07:13:15 +0000] "GET /api/v1/activity HTTP/1.1" 403 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "82.19.72.223"

                            Any ideas?

                            • gcrk replied to this.

                                MattDHarding Thats only a little helpful, mind checking the browser console for log messages? Or check the network tab of the browser dev tools for failing requests

                                    gcrk The nerwork response for that message is:

                                    {"non_field_errors": ["Unable to log in with provided credentials"]}

                                    Is it possible that I messed up the database migration and that user isn't in my database now? Is there an easy way to check?

                                    Edit, there is also this output from the api container:

                                    front_1         | 172.18.0.5 - - [28/Jun/2023:07:41:34 +0000] "GET /api/v1/activity HTTP/1.1" 403 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "82.19.72.223"
api_1           | [2023-06-28 07:41:34 +0000] [10] [INFO] connection closed
api_1           | [2023-06-28 07:41:34 +0000] [10] [ERROR] Exception in ASGI application
api_1           | Traceback (most recent call last):
api_1           |   File "/venv/lib/python3.10/site-packages/uvicorn/protocols/websockets/websockets_impl.py", line 238, in run_asgi
api_1           |     result = await self.app(self.scope, self.asgi_receive, self.asgi_send)
api_1           |   File "/venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
api_1           |     return await self.app(scope, receive, send)
api_1           |   File "/venv/lib/python3.10/site-packages/channels/routing.py", line 62, in __call__
api_1           |     return await application(scope, receive, send)
api_1           |   File "/venv/lib/python3.10/site-packages/channels/sessions.py", line 47, in __call__
api_1           |     return await self.inner(dict(scope, cookies=cookies), receive, send)
api_1           |   File "/venv/lib/python3.10/site-packages/channels/sessions.py", line 263, in __call__
api_1           |     return await self.inner(wrapper.scope, receive, wrapper.send)
api_1           |   File "/venv/lib/python3.10/site-packages/channels/auth.py", line 185, in __call__
api_1           |     return await super().__call__(scope, receive, send)
api_1           |   File "/venv/lib/python3.10/site-packages/channels/middleware.py", line 24, in __call__
api_1           |     return await self.inner(scope, receive, send)
api_1           |   File "/venv/lib/python3.10/site-packages/channels/routing.py", line 116, in __call__
api_1           |     return await application(
api_1           |   File "/venv/lib/python3.10/site-packages/channels/consumer.py", line 94, in app
api_1           |     return await consumer(scope, receive, send)
api_1           |   File "/venv/lib/python3.10/site-packages/channels/consumer.py", line 58, in __call__
api_1           |     await await_many_dispatch(
api_1           |   File "/venv/lib/python3.10/site-packages/channels/utils.py", line 50, in await_many_dispatch
api_1           |     await dispatch(result)
api_1           |   File "/venv/lib/python3.10/site-packages/asgiref/sync.py", line 435, in __call__
api_1           |     ret = await asyncio.wait_for(future, timeout=None)
api_1           |   File "/usr/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
api_1           |     return await fut
api_1           |   File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
api_1           |     result = self.fn(*self.args, **self.kwargs)
api_1           |   File "/venv/lib/python3.10/site-packages/channels/db.py", line 13, in thread_handler
api_1           |     return super().thread_handler(loop, *args, **kwargs)
api_1           |   File "/venv/lib/python3.10/site-packages/asgiref/sync.py", line 476, in thread_handler
api_1           |     return func(*args, **kwargs)
api_1           |   File "/venv/lib/python3.10/site-packages/channels/consumer.py", line 125, in dispatch
api_1           |     handler(message)
api_1           |   File "/venv/lib/python3.10/site-packages/channels/generic/websocket.py", line 105, in websocket_disconnect
api_1           |     self.disconnect(message["code"])
api_1           |   File "/app/funkwhale_api/common/consumers.py", line 22, in disconnect
api_1           |     groups = self.scope["user"].get_channels_groups() + self.groups
api_1           |   File "/venv/lib/python3.10/site-packages/django/utils/functional.py", line 247, in inner
api_1           |     return func(self._wrapped, *args)
api_1           | AttributeError: 'AnonymousUser' object has no attribute 'get_channels_groups'
                                    • gcrk replied to this.

                                        MattDHarding You should always be able to visit https://your-instance.org/api/admin and login there for the django admin. This should work even if the frontend is broken.

                                        Can you screenshot the network tab of the browser dev tools while trying to login? The exception you posted should only be related to the websockets, which is barely used and has nothing to do with the login...

                                            gcrk Is this what you wanted to see?
                                            [upl-image-preview url=https://forum.funkwhale.audio/assets/files/2023-06-28/1687961538-148035-image.png]

                                            I can't log in on https://funkwhale.lepton.link/api/admin either, so I'm wondering if I need to look at the database migration again, I thought it went fine but perhaps not.

                                            • gcrk replied to this.

                                                MattDHarding It looks indeed like your instance is technically working fine, but has an empty database. Did you changed the mountpoint for postgres by any chance?

                                                    gcrk I didn't, it looks like I was a dummy and mistyped something during the database migration and blanked my database (db_dump.sql is empty). Seeing that this is a personal instance and I had nothing of consequence in the database I think I'll just install my instance from scratch.

                                                    Thanks for your help here in resolving my problems 🙂

                                                    Edit: the bottom line here for anyone struggling with traefik and the new docker-compose.yml file for 1.3 is to add back in the default network to all the containers.

                                                    • gcrk replied to this.

                                                        MattDHarding Edit: the bottom line here for anyone struggling with traefik and the new docker-compose.yml file for 1.3 is to add back in the default network to all the containers.

                                                        or alternatively make traefik join the implicit network.

                                                        An addition: I would recommend making those modifications to docker-compose.override.yml next to docker-compose.yml. This way the compose file can be updated while the custom overrides stay in place.

                                                          21 days later

                                                          Just wondering if funkwhale could possibly make this more difficult to manage/maintain. I like pulling my hair out every single time there is an upgrade

                                                          • gcrk replied to this.

                                                            dishcandanty Thanks for the super constructive feedback, helps a lot and makes spending spare time on the project even more fun. Super appreciated!