Differences between stable and develop docker images?

phw

I have a strange issue: I have a multi container docker setup that first ran on the previous stable versions (1.1.4), then I changed it to use FUNKWHALE_VERSION=develop and frequently updated it to the latest develop version. Currently it happily runs with 1.2.0+git.742f843c which at time of writing this is the latest commit on develop that bumps the version to 1.2.0.

Now if I switch to FUNKWHALE_VERSION=1.2.0 and update the containers Fukwhale fails to serve any request. The funkwhale_api_1 container gives the following error:

[2022-01-01 13:02:57 +0000] [12] [INFO] Application startup complete.
2022-01-01 13:04:05,077 plugins      DEBUG    Calling handlers for filter urls
2022-01-01 13:04:05,587 django.request ERROR    Internal Server Error: /about/pod
Traceback (most recent call last):
  File "/venv/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/venv/lib/python3.8/site-packages/urllib3/connection.py", line 358, in connect
    conn = self._new_conn()
  File "/venv/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f5ea5469610>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/venv/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='music.uploadedlobster.com', port=443): Max retries exceeded with url: /front/index.html (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f5ea5469610>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/app/funkwhale_api/common/middleware.py", line 239, in __call__
    return serve_spa(request)
  File "/app/funkwhale_api/common/middleware.py", line 38, in serve_spa
    html = get_spa_html(settings.FUNKWHALE_SPA_HTML_ROOT)
  File "/app/funkwhale_api/common/middleware.py", line 112, in get_spa_html
    return get_spa_file(spa_url, "index.html")
  File "/app/funkwhale_api/common/middleware.py", line 129, in get_spa_file
    response = session.get_session().get(utils.join_url(spa_url, name),)
  File "/venv/lib/python3.8/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/app/funkwhale_api/common/session.py", line 11, in request
    return super().request(*args, **kwargs)
  File "/venv/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/venv/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/venv/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='music.uploadedlobster.com', port=443): Max retries exceeded with url: /front/index.html (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f5ea5469610>: Failed to establish a new connection: [Errno 111] Connection refused'))

So something seems to be different for how the stable images are behaving compared to develop. But it is not clear what it is.

Maybe part of the issue is that the nginx that is part of the multi container funkwhale setup in my configuration does not do the HTTPS handling. Instead there is another nginx proxy that provides the SSL certificates for multiple docker based services on the same server (based on the setup at https://github.com/phw/docker-nginx-proxy-letsencrypt). But still I don't see how this should matter and why the develop version runs just fine.

phw

I looked at the differences between stable and develop branch at https://dev.funkwhale.audio/funkwhale/funkwhale/-/compare/develop...stable?from_project_id=17 . Some adhoc changes that I think will get pushed to develop also, but nothing that would explain the difference in behavior to me.

Totally unclear why the funkwhale_api docker instance cannot access the URL for getting the frontend when running the stable image. The configuration is otherwise exactly the same.

If I run the stable image and explicitly open /front/ then it also downloads it, so the access to that URL is generally working. And FUNKWHALE_SPA_HTML_ROOT seems to point to the correct location.

phw

Interesting, FUNKWHALE_VERSION=stable also works without issues and ends up with version 1.2.0+git.77f3f444 , which is the currently latest commit on the stable branch.

So it is just the 1.2 tag for the docker image that is misbehaving. Had this image been built different?

AMoonRabbit

If you want to use stable release, then I'd recommend changing the docker-compose file to reference funkwhale/funkwhale:latest instead of specifying "stable". That's what I did when I reverted from develop back to release when 1.2 was officially released.

phw

Thanks, but latest is currently the exact same image as the 1.2.0 tag, so has the same issue.

stable on the other hand gives me the latest commit on the stable branch, and that works. The only change I think might probably matter was commit 77f3f44. I'm not totally sure what issue this was supposed to fix, but the rest of the commits seems unrelated to image building.

@gcrk probably can tell better. But from my side probably just the general information, that the 1.2.0 tag image has the issue described above, and 1.2.0+git.742f843c is fine again. So hopefully a 1.2.1 stable release will work directly again 🙂

gcrk

Looks like I did a bad job with this release. Lets try to shed some light on this.

In the last development period I changed how the pipeline builds and tags docker images. One nice thing is that we have multi arch containers now. Sadly I messed up and the actual release pipeline has a bug, so we don't had built docker containers for 1.2.0. So I built them manually on my work notebook and maybe something went wrong there? I am not sure right now. But I really hope 1.2.1 will solve all this.

Huh, what an exciting release 😐

phw

gcrk Was the first release under new maintainership, and it has a great amount of really good changes. So I think you did a good job in coordinating and planing and getting things finished for a release 🙂

The final builds got a bit messed up, but I'm positive that the 1.2.1 release will go smoother. To me it looks like that the build issues have been resolved.

gcrk

Thanks for the kind words. I still think some self criticism is totally appropriate in order to have some learning for the future.

Anyway, I am not sure what to do with the 1.2.0 image. Remove it? Since it seems to be broken anyway.

phw

gcrk I'm not sure if it is broken for everyone, or it's still in some way special for me. I just still don't understand the actual cause for this connection issue.

But I would actually say let's make a 1.2.1 release soon to get a clean and working 1.2.x release out.

phw

I can confirm that the 1.2.1 image works again without this issue.