Issue after upgrading from 1.2.10 to 1.3.1 (docker)

homer77

funkwhale-nginx-1         | 192.168.80.1 - - [21/Aug/2023:18:02:02 +0000] "GET /index.html HTTP/1.1" 499 0 "-" "python-requests (funkwhale/1.3.1; +https://pod.mydomain.net)" "89.246.124.14"
funkwhale-api-1           | 2023-08-21 18:02:02,290 django.request ERROR    Internal Server Error: /index.html
funkwhale-api-1           | Traceback (most recent call last):
funkwhale-api-1           |   File "/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 466, in _make_request
funkwhale-api-1           |     six.raise_from(e, None)
funkwhale-api-1           |   File "<string>", line 3, in raise_from
funkwhale-api-1           |   File "/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 461, in _make_request
funkwhale-api-1           |     httplib_response = conn.getresponse()
funkwhale-api-1           |   File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
funkwhale-api-1           |     response.begin()
funkwhale-api-1           |   File "/usr/lib/python3.10/http/client.py", line 318, in begin
funkwhale-api-1           |     version, status, reason = self._read_status()
funkwhale-api-1           |   File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
funkwhale-api-1           |     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
funkwhale-api-1           |   File "/usr/lib/python3.10/socket.py", line 705, in readinto
funkwhale-api-1           |     return self._sock.recv_into(b)
funkwhale-api-1           |   File "/usr/lib/python3.10/ssl.py", line 1274, in recv_into
funkwhale-api-1           |     return self.read(nbytes, buffer)
funkwhale-api-1           |   File "/usr/lib/python3.10/ssl.py", line 1130, in read
funkwhale-api-1           |     return self._sslobj.read(len, buffer)
funkwhale-api-1           | TimeoutError: The read operation timed out
funkwhale-api-1           | 
funkwhale-api-1           | During handling of the above exception, another exception occurred:
funkwhale-api-1           | 
funkwhale-api-1           | Traceback (most recent call last):
funkwhale-api-1           |   File "/venv/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
funkwhale-api-1           |     resp = conn.urlopen(
funkwhale-api-1           |   File "/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 798, in urlopen
funkwhale-api-1           |     retries = retries.increment(
funkwhale-api-1           |   File "/venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 550, in increment
funkwhale-nginx-1         | 192.168.80.1 - - [21/Aug/2023:18:02:03 +0000] "GET /index.html HTTP/1.1" 499 0 "-" "python-requests (funkwhale/1.3.1; +https://pod.mydomain.net)" "89.246.124.14"
funkwhale-api-1           | 2023-08-21 18:02:03,942 django.request ERROR    Internal Server Error: /index.html
funkwhale-api-1           | Traceback (most recent call last):
funkwhale-api-1           |   File "/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 466, in _make_request
funkwhale-api-1           |     six.raise_from(e, None)
funkwhale-api-1           |   File "<string>", line 3, in raise_from
funkwhale-api-1           |   File "/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 461, in _make_request
funkwhale-api-1           |     httplib_response = conn.getresponse()
funkwhale-nginx-1         | 192.168.80.1 - - [21/Aug/2023:18:02:12 +0000] "GET /index.html HTTP/1.1" 499 0 "-" "python-requests (funkwhale/1.3.1; +https://pod.mydomain.net)" "89.246.124.14"
funkwhale-api-1           | 2023-08-21 18:02:12,403 django.request ERROR    Internal Server Error: /index.html

After upgrading I receive an 500 Server Error and the failure message (excerpt) above.
I first thought that's again a nginx proxy related error because upgrade broke my customizations. But after twiddeling around with it the problem seems to relate to that api container. Obviously it can't reach mydomain. But I don't understand why.

Regards and thanks for ideas,

Homer

gcrk

funkwhale-api-1           | 2023-08-21 18:02:03,942 django.request ERROR    Internal Server Error: /index.html

If the API fails on index.html, there is most likely an issue with your reverse proxy conf. Please check https://docs.funkwhale.audio/administrator/upgrade/docker.html#update-your-reverse-proxy-configuration or show your config so we can debug it together 🙂

homer77

gcrk When I execute a bash in the nginx container I receive

root@d1f14c3338a3:/# curl http://api:5000

<!doctype html>
<html lang="en">
<head>
  <title>Server Error (500)</title>
</head>
<body>
  <h1>Server Error (500)</h1><p></p>
</body>
</html>

So the api container seems to be accessible at the 5000 port but gets a 500.

upstream funkwhale-api {
    # depending on your setup, you may want to update this
    server api:5000;
}


# required for websocket support
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 5001;
    server_name ${FUNKWHALE_HOSTNAME};

   root /frontend;

    # If you are using S3 to host your files, remember to add your S3 URL to the
    # media-src and img-src headers (e.g. img-src 'self' https://<your-S3-URL> $

    add_header Content-Security-Policy "default-src 'self'; script-src 'self'; $
    add_header Referrer-Policy "strict-origin-when-cross-origin";
    add_header X-Frame-Options "SAMEORIGIN" always;

    location / {
        include /etc/nginx/funkwhale_proxy.conf;                                
        # this is needed if you have file import via upload enabled
        client_max_body_size ${NGINX_MAX_BODY_SIZE};
        proxy_pass   http://funkwhale-api/;
    }

    location /front/ {
        add_header Content-Security-Policy "default-src 'self'; script-src 'sel$
        add_header Referrer-Policy "strict-origin-when-cross-origin";
        add_header Service-Worker-Allowed "/";
        alias /frontend/;
       expires 30d;
        add_header Pragma public;
        add_header Cache-Control "public, must-revalidate, proxy-revalidate";
    }

    location /front/embed.html {
        add_header Content-Security-Policy "default-src 'self'; script-src 'sel$
        add_header Referrer-Policy "strict-origin-when-cross-origin";

        add_header X-Frame-Options "" always;
        alias /frontend/embed.html;
        expires 30d;
        add_header Pragma public;
        add_header Cache-Control "public, must-revalidate, proxy-revalidate";
    }

    location /federation/ {
        include /etc/nginx/funkwhale_proxy.conf;
        proxy_pass   http://funkwhale-api/federation/;
    }
        
 # You can comment this if you do not plan to use the Subsonic API
    location /rest/ {
        include /etc/nginx/funkwhale_proxy.conf;
        proxy_pass   http://funkwhale-api/api/subsonic/rest/;
    }

    location /.well-known/ {
        include /etc/nginx/funkwhale_proxy.conf;
        proxy_pass   http://funkwhale-api/.well-known/;
    }
        
    location /media/__sized__/ {
        alias ${MEDIA_ROOT}/__sized__/;
    }

    location /media/attachments/ {
        alias ${MEDIA_ROOT}/attachments/;
    }

   location /_protected/media { 
        internal;
        alias   ${MEDIA_ROOT};  

    }
    # Comment the previous location and uncomment this one if you're storing
    # media files in a S3 bucket
    # location ~ /_protected/media/(.+) {
    #     internal;
    #     # Needed to ensure DSub auth isn't forwarded to S3/Minio, see #932    
    #     proxy_set_header Authorization "";
    #     proxy_pass $1;
    # }

    location /_protected/music {
        # this is an internal location that is used to serve
        # audio files once correct permission / authentication
        # has been checked on API side
        # Set this to the same value as your MUSIC_DIRECTORY_PATH setting
        internal;
        alias   ${MUSIC_DIRECTORY_PATH};
    }

    location /staticfiles/ {
        # django static files
        alias ${STATIC_ROOT}/;
    }
}

That's my funkwhale.template that worked with 1.29.10. But it didn't longer after the update.

I downloaded the newest template but had to customize as I get the encrypted domain from another nginx on my Synology Device.

upstream fw {
    # depending on your setup, you may want to update this
#    server ${FUNKWHALE_API_IP}:${FUNKWHALE_API_PORT};
    server api:5000;
}
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {


    listen 5001;
    server_name ${FUNKWHALE_HOSTNAME};
#    location / { return 301 https://$host$request_uri; }
    root /frontend;

    # HSTS
    add_header Strict-Transport-Security "max-age=31536000";

    # Security related headers

    # If you are using S3 to host your files, remember to add your S3 URL to the
    # media-src and img-src headers (e.g. img-src 'self' https://<your-S3-URL> $
                                                                                
    add_header Content-Security-Policy "default-src 'self'; script-src 'self'; $

    # compression settings
    gzip on;
    gzip_comp_level    5;
    gzip_min_length    256;
    gzip_proxied       any;
    gzip_vary          on;

    gzip_types
       application/javascript
        application/vnd.geo+json
        application/vnd.ms-fontobject
        application/x-font-ttf
        application/x-web-app-manifest+json
        font/opentype
        image/bmp
        image/svg+xml
        image/x-icon
        text/cache-manifest
        text/css
        text/plain
        text/vcard
        text/vnd.rim.location.xloc   
        text/vtt
        text/x-component
        text/x-cross-domain-policy;
        
    location / {
        include /etc/nginx/funkwhale_proxy.conf;
        client_max_body_size ${NGINX_MAX_BODY_SIZE};
        proxy_pass   http://fw;
       application/javascript
        application/vnd.geo+json
        application/vnd.ms-fontobject
        application/x-font-ttf
        application/x-web-app-manifest+json
        font/opentype
        image/bmp
        image/svg+xml
        image/x-icon
        text/cache-manifest
        text/css
        text/plain
        text/vcard
        text/vnd.rim.location.xloc   
        text/vtt
        text/x-component
        text/x-cross-domain-policy;
        
    location / {
        include /etc/nginx/funkwhale_proxy.conf;
        client_max_body_size ${NGINX_MAX_BODY_SIZE};
        proxy_pass   http://fw;
    }
}

gcrk

homer77 I downloaded the newest template but had to customize as I get the encrypted domain from another nginx on my Synology Device.

Okay, so if you already have a nginx reverse proxy, why don't you use this and send all requests to the frontend container? This would work!

homer77 gcrk When I execute a bash in the nginx container I receive

However, this seems to be broken. Can you show me your env config? Issues might be FUNKWHALE_PROTOCOL or FUNKWHALE_API_PORT settings.

Maybe related (just a guess): https://dev.funkwhale.audio/funkwhale/funkwhale/-/issues/2190

homer77

EDIT2: OK, it's solved. After removing my old nginx config and adding the front container I could forward its port 80 to dockers 5002 and so to Synology's mechanisms. Et voilá, instance is working like before 😅
Obviously I missed out some developments in the last years by not updating the docker-compose.yml since then 🤪
Thanks for your efforts! @gcrk

EDIT: Oh boy! ... I just realized that there have been some bigger changes in the docker-compose.yml including a frontend container?! ... I ... will maybe try this out first and see if that's just the cause of the mess 🙈

~~gcrk Because unfortunately the inbuilt nginx of the SynologyOS is quite constraint and customizations by this quite limited and complicated. So I yet used to do it like that.~~

FUNKWHALE_PROTOCOL is https since ever.
FUNKWHALE_API_IP=api
FUNKWHALE_API_PORT=5000

~~I used to have the funkwhale nginx container listen to 5001 which the Synology nginx gave to my ssl-subdomain pod.mydomain.net. As port 5000 was already in use.~~

Just for testing I deactivated the nginx container and set API_PORT to 5001.
I call funkwhale on my home-network-ip:5001. Same result. Same error messages that api container can't reach index.html at pod.mydomain.net.
So the api fails because it can't reach the api at its own domain? ... I don't get the logic behind it 🤔 Shouldn't it provide exactly that output? ...

gcrk

homer77 Yes, we had some deep changes in order to untangle front and backend to make the architecture easier and potentially easier to scale (although I always prefer more smaller instances than one huge instance). This requires everyone to do some migration steps, but we tried to write them all down in the changelog. If you have any more questions, let me know! 🙂

homer77

gcrk
Yeah ... well ... that points out why I missed it. I read https://blog.funkwhale.audio/funkwhale-131-release.html, then jumped directly to the upgrading docs and so didn't see anything of the notes at https://dev.funkwhale.audio/funkwhale/funkwhale/-/releases/1.3.0 🤦
Well - at least nothings broken for good and I can continue to listen to my music 😋 I still have to explore what had actually changed on the surface 🤔

gcrk

@Sporiff Note for future releases: Always link the Changelog 🙂

homer77 I still have to explore what had actually changed on the surface 🤔

Here the changelog might come in handy, however most work we did was rebuilding the frontend in Vue3 and Typescript, so there are not a lot of new features, but it should be way more responsive, hopefully.