Traefik configuration

gcrk

For libraries I get:

Could not fetch remote library

    Error while fetching the library: HTTPSConnectionPool(host='node1.funkwhale.test', port=443): Max retries exceeded with url: /federation/music/libraries/cfc9cbb1-6f73-44ca-8102-48240fec843e (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f2f662b13a0>, 'Connection to node1.funkwhale.test timed out. (connect timeout=10)'))

For channels:

 Error while fetching object

    This object cannot be retrieved

@devilcius Is this something your MR fixes?

devilcius

gcrk I believe that's the same error that I first had, and then it disappeared when retrying. I'm don't think that is related with the MR, but I haven't test it.

devilcius

gcrk. Regarding the networking: hosts file only serves as a substitute to configure dnsmasq, it's not related with the communication between the containers. I believe this is managed by traefik.

And docker network ls return this:

NETWORK ID     NAME             DRIVER    SCOPE
ef8adcd8b5a9   bridge           bridge    local
0f06ee76e03e   federation       bridge    local
67018ab16dcc   host             host      local
70e51560fcf3   node1_default    bridge    local
591600d80d43   node1_internal   bridge    local
adb052a80098   node2_default    bridge    local
750c3ba932a1   node2_internal   bridge    local

gcrk

devilcius I believe this is managed by traefik.

This can't be true, ping from node1 to node2 still works when traefik is not running anymore. Docker networking is so weird sometimes. My current guess is, that it takes the /etc/hosts file when creating a network, so if the entries are available BEFORE the network is created, it works.

gcrk

@devilcius Can you show me docker network inspect federation?

devilcius

gcrk


[
    {
        "Name": "federation",
        "Id": "f2635a0743326ccc75c573b63c6a94bb83abd3afd17cc42710290e7f3d9e8575",
        "Created": "2022-04-15T18:50:14.69865116+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {},
        "Labels": {}
    }
]

devilcius

I retry from scratch and save the logs so we can maybe find out what's going on:

sudo docker system prune -a
sudo rm -rf data/*
docker network create federation
docker-compose -f dev.yml build

I launch every instance as described in my original post.

I try 3 times in a row to subscribe from node2 to node1 channel:

api_1 | 2022-04-15 17:18:47,358 funkwhale_api.federation.authentication INFO Discarding HTTP request from actor/domain https://node2.funkwhale.test/federation/actors/marcos, HTTPSConnectionPool(host='node2.funkwhale.test', port=443): Read timed out. (read timeout=10)

create public library in node1, try to connect (x3) from node2:

api_1           | 2022-04-15 17:27:19,845 django.request WARNING  Bad Request: /api/v1/federation/libraries/fetch/
api_1           | INFO:     172.21.0.7:51560 - "POST /api/v1/federation/libraries/fetch/ HTTP/1.1" 400 Bad Request

tried once again:

Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:99.0) Gecko/20100101 Firefox/99.0" "172.18.0.1"
nginx_1 | 172.18.0.2 - - [15/Apr/2022:17:32:11 +0000] "GET /federation/actors/marcos HTTP/1.1" 499 0 "-" "python-requests (funkwhale/1.2.3; +https://node1.funkwhale.test)" "172.18.0.1"

Since there's a 499 http error, I stop both instances and added include /etc/nginx/funkwhale_proxy.conf; to docker/nginx/conf.dev. I did that yesterday too. butI didn't mentioned in my original post because I modified at the beginning of my debug journey and thought it had no impact.

I try to connect to library again. No errors in the server logs but GUI returns error. I reload the page and try again. Success!

[upl-image-preview url=https://forum.funkwhale.audio/assets/files/2022-04-15/1650044737-229972-image.png]

I try with the channel, success!
[upl-image-preview url=https://forum.funkwhale.audio/assets/files/2022-04-15/1650044841-451832-image.png]

petitminion

Adding the new nginx rule don't resolve the errors for me using docker compose build --no-cache 😢

pruning every (docker and fw data) don't work either

devilcius

petitminion
What's the error and when does it occur, when building?

devilcius

petitminion
Apparently node2 sees node1 since it get a 400 response, so I presume it's not the same problem as @gcrk is experiencing.

node2-api-1 | 2022-04-15 14:23:40,424 django.request WARNING Bad Request: /api/v1/federation/libraries/fetch/

I checked my logs from yesterday and I also had this same error at the beginning (the logs are shared in funkwhale-dev in matrix). After setting proxy_ignore_client_abort on; under server block in nginx conf.dev and restarting both instances it did succeed.

petitminion

devilcius same as here https://forum.funkwhale.audio/d/120-traefik-configuration/28

node1 can't resolve node 2 :

docker exec 108017f72b24 nslookup node2.funkwhale.test
Server:         127.0.0.11
Address:        127.0.0.11:53

Name:   node2.funkwhale.test
Address: 127.0.0.1

*** Can't find node2.funkwhale.test: No answer

but node1 can ping node2 :

PING node2.funkwhale.test (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: seq=0 ttl=64 time=0.035 ms
64 bytes from 127.0.0.1: seq=1 ttl=64 time=0.112 ms
^C
--- node2.funkwhale.test ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss

petitminion

devilcius this didn't work 😢

gcrk

@petitminion How about having a Jitsi session to get this sorted on your machine? I got a fully working dev setup today and can probably dig quite deep with you.

devilcius

gcrk
I'm glad you got it working! Did you use dnsmasq?

petitminion

gcrk Yes it would be great ! o/

petitminion

I got it working by adding this to dev.yml in the api service :

    extra_hosts:
      - "node1.funkwhale.test:172.17.0.1"
      - "node2.funkwhale.test:172.17.0.1"

and we should also add this to .env :

FUNKWHALE_WEB_WORKERS = 4

should we update the doc with this solution ?

petitminion

extra_hosts:
  - "node1.funkwhale.test:172.17.0.1"
  - "node2.funkwhale.test:172.17.0.1"

has to be added to the celery container too

« Previous Page