Traefik configuration

petitminion

main error in traefik level=debug msg="http: TLS handshake error from 172.19.0.1:59130: remote error: tls: unknown certificate" and docker-traefik-1 | time="2022-03-17T16:14:51Z" level=debug msg="No default certificate, generating one" tlsStoreName=default

the problem seems to come from the fact traefik do not load the dynamic conf :

docker-traefik-1  | time="2022-03-17T15:54:23Z" level=debug msg="Configuration received from provider file: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=file

Config Files

static conf in traefik.yml

services:
  traefik:
    image: traefik:v2.6
    command:
      - --log.level=DEBUG
      #- --providers.file.filename=/cert.yml
      - --providers.file.directory=/dynamic_conf/
      - --providers.file.watch=true
      - --api.insecure=true
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false
      - --providers.docker.endpoint=unix:///var/run/docker.sock
      - --entrypoints.web.address=:80
      - --entrypoints.web-secure.address=:443

    restart: always
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
      - "8040:8040"
    networks:
      - federation
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./ssl/test.key:/ssl/test.key
      - ./ssl/test.crt:/ssl/test.crt
      # - ./ssl/:/ssl/
      - ./dynamic_conf/cert.yml:/dynamic_conf/cert.yml:ro
      - ./cert.yml:/cert.yml
    # labels:
    # - "traefik.enable=true"
    # - "traefik.http.routers.traefik=true"

networks:
  federation:
    name: federation

dynamic conf :

tls:
  #  certificates:
  #    certFile: /ssl/test.crt
  #    keyFile: /ssl/test.key
  #    stores:
  #      - default
  certificates:
    - certFile: /ssl/test.crt
      keyFile: /ssl/test.key

Nginx traefik V2 labels :

      - "traefik.http.routers.${COMPOSE_PROJECT_NAME}.rule=Host(`${COMPOSE_PROJECT_NAME}.funkwhale.localhost`)"
      - "traefik.http.routers.${COMPOSE_PROJECT_NAME}.tls=true"
      - "traefik.http.routers.${COMPOSE_PROJECT_NAME}.tls.domains[0].main=${COMPOSE_PROJECT_NAME}.funkwhale.localhost"
      - "traefik.enable=true"
      - "traefik.federation.protocol=http"
      - "traefik.federation.port=80"
      - "traefik.docker.network=federation"

I change a little bit the ssl cert generation but this shouldn't change anything :
./ssl/openssl.con

# cf https://stackoverflow.com/questions/10175812/how-to-create-a-self-signed-certificate-with-openssl/27931596#27931596
# create with openssl req -config openssl.conf -new -x509 -sha256 -newkey rsa:2048 -nodes -keyout test.key -days 365 -out test.crt
# sudo cp test.crt /usr/local/share/ca-certificates/ ; sudo update-ca-certificates # note that the cert need to end with .crt to be picked.
# verify : openssl verify test.crt

[ req ]
default_bits        = 2048
default_keyfile     = server-key.pem
distinguished_name  = subject
req_extensions      = req_ext
x509_extensions     = x509_ext
string_mask         = utf8only

# The Subject DN can be formed using X501 or RFC 4514 (see RFC 4519 for a description).
#   Its sort of a mashup. For example, RFC 4514 does not provide emailAddress.
[ subject ]
countryName         = Country Name (2 letter code)
countryName_default     = US

stateOrProvinceName     = State or Province Name (full name)
stateOrProvinceName_default = NY

localityName            = Locality Name (eg, city)
localityName_default        = New York

organizationName         = Organization Name (eg, company)
organizationName_default    = Example, LLC

# Use a friendly name here because it's presented to the user. The server's DNS
#   names are placed in Subject Alternate Names. Plus, DNS names here is deprecated
#   by both IETF and CA/Browser Forums. If you place a DNS name here, then you
#   must include the DNS name in the SAN too (otherwise, Chrome and others that
#   strictly follow the CA/Browser Baseline Requirements will fail).
commonName          = Common Name (e.g. server FQDN or YOUR name)
commonName_default      = *funkwhale.localhost

emailAddress            = Email Address
emailAddress_default        = test@example.com

# Section x509_ext is used when generating a self-signed certificate. I.e., openssl req -x509 ...
[ x509_ext ]

subjectKeyIdentifier        = hash
authorityKeyIdentifier    = keyid,issuer

# You only need digitalSignature below. *If* you don't allow
#   RSA Key transport (i.e., you use ephemeral cipher suites), then
#   omit keyEncipherment because that's key transport.
basicConstraints        = CA:FALSE
keyUsage            = digitalSignature, keyEncipherment, keyCertSign
subjectAltName          = @alternate_names
nsComment           = "OpenSSL Generated Certificate"

# RFC 5280, Section 4.2.1.12 makes EKU optional
#   CA/Browser Baseline Requirements, Appendix (B)(3)(G) makes me confused
#   In either case, you probably only need serverAuth.
# extendedKeyUsage    = serverAuth, clientAuth

# Section req_ext is used when generating a certificate signing request. I.e., openssl req ...
[ req_ext ]

subjectKeyIdentifier        = hash

basicConstraints        = CA:FALSE
keyUsage            = digitalSignature, keyEncipherment
subjectAltName          = @alternate_names
nsComment           = "OpenSSL Generated Certificate"

# RFC 5280, Section 4.2.1.12 makes EKU optional
#   CA/Browser Baseline Requirements, Appendix (B)(3)(G) makes me confused
#   In either case, you probably only need serverAuth.
# extendedKeyUsage    = serverAuth, clientAuth

[ alternate_names ]

DNS.1       = funkwhale.localhost
DNS.2       = node1.funkwhale.localhost
DNS.3       = node2.funkwhale.localhost
DNS.4       = node3.funkwhale.localhost
DNS.5       = localhost
DNS.6       = 127.0.0.1

# Add these if you need them. But usually you don't want them or
#   need them in production. You may need them for development.
# DNS.5       = localhost
# DNS.6       = localhost.localdomain
# DNS.7       = 127.0.0.1

# IPv6 localhost
# DNS.8     = ::1

petitminion

The certs and the dynamic configuration are accessible from inside the traefik container

petitminion

oh wait I have somethingggggg :
level=debug msg="Adding certificate for domain(s) example company,127.0.0.1,funkwhale.loaclhost,localhost,node1.funkwhale.localhost,node2.funkwhale.localhost,node3.funkwhale.loacalhost"

amazing idk what changed

petitminion

I read in some forum you have to choose only one configuration type (cli, yml or tml) but you can't use both..
https://doc.traefik.io/traefik/v2.0/reference/dynamic-configuration/file/

petitminion

okey so I have a configuration reading the dynamic config but now traefik can't read the certificate Unable to append certificate to store: unable to generate TLS certificate : tls: failed to find any PEM data in certificate input" tlsStoreName=default
This error come from https://go.googlesource.com/go/+/go1.9/src/crypto/tls/tls.go and I means cert is empty

Config

dynamic

tls:
  certificates:
    certFile: /ssl/test.crt
    keyFile: /ssl/test.key
    stores:
      - default

  stores:
    default:
      defaultCertificate:
        certFile: /ssl/test.crt
        keyFile: /ssl/test.key

traefik

      - --log.level=DEBUG
      #- --providers.file=true
      #- --providers.file.filename=/cert.yml
      - --providers.file.directory=/dynamic_conf/
      - --providers.file.debugloggeneratedtemplate=true
      #- --providers.file.watch=true
      - --api.insecure=true
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false
      - --providers.docker.endpoint=unix:///var/run/docker.sock
      - --entrypoints.web.address=:80
      - --entrypoints.web-secure.address=:443

petitminion

found the error : missing - like :

tls:
  certificates:
    - certFile: /ssl/test.crt
      keyFile: /ssl/test.key
      stores:
        - default

  stores:
    default:
      defaultCertificate:
        certFile: /ssl/test.crt
        keyFile: /ssl/test.key

petitminion

I stil have a level=debug msg="http: TLS handshake error from 172.19.0.1:59386: remote error: tls: unknown certificate error

petitminion

petitminion It seems this error is cause request on localhost that do not came from funkwhale. In my computeur 172.19.0.1 is a docker container but they are all closed unless the traefik so I don't get where this requests come from..

devilcius

How I configured development setup with federation. Step by step.

System: Ubuntu 20.04
docker-compose version 1.25.0, build unknown
Docker version 20.10.14, build a224086
TL;DR: instructions at https://docs.funkwhale.audio/contributing.html#working-with-federation-locally work. Although we might improve them for clarity.

First of all:

sudo docker system prune -a

Probably overkill, but I want to be sure that all networks are deleted, and no built cache interferes in any sense. I don't trust my limited understanding about docker.

git clone git@dev.funkwhale.audio:funkwhale/funkwhale.git funkwhale-federation

cd funkwhale-federation

Add this entries to /etc/hosts:

172.17.0.1      node1.funkwhale.test
172.17.0.1      node2.funkwhale.test

Create federation network:

docker network create federation

Follow the steps from here https://docs.funkwhale.audio/contributing.html#working-with-federation-locally

- sudo cp docker/ssl/test.crt /usr/local/share/ca-certificates/
- sudo update-ca-certificates

Create .env file with this content:

# This will ensure we don't bind any port on the host, and thus enable
# multiple instances of funkwhale to be spawned concurrently.
VUE_PORT_BINDING=
# This disable certificate verification
EXTERNAL_REQUESTS_VERIFY_SSL=false
# this ensure you don't have incorrect urls pointing to http resources
FUNKWHALE_PROTOCOL=https
# Disable host ports binding for the nginx container, as traefik is serving everything
NGINX_PORTS_MAPPING=80

Build container:

docker-compose -f dev.yml build

In one terminal tab:

docker-compose -f docker/traefik.yml up

In a second tab:

export COMPOSE_PROJECT_NAME=node1
export VUE_PORT=8888
docker-compose -f dev.yml run --rm api python manage.py migrate
docker-compose -f dev.yml run --rm api python manage.py createsuperuser
docker-compose -f dev.yml up nginx api front nginx api celeryworker

In a third tab:

export COMPOSE_PROJECT_NAME=node2
export VUE_PORT=9999
docker-compose -f dev.yml run --rm api python manage.py migrate
docker-compose -f dev.yml run --rm api python manage.py createsuperuser
docker-compose -f dev.yml up nginx api front nginx api celeryworker

Open https://node1.funkwhale.test/ in Firefox (accept risk and continue.…):

Log in then add content (upload icon, top right column), => Publish your work in a channel => add new channel => artist discography | name: node1 channel => create channel. Once created I create an new album and add some songs to that album. Then publish.

Open new browser tab https://node2.funkwhale.test/ (accept risk and continue...)
Channels => add new => introduce @node1channel@node1.funkwhale.test

It fails:

node1:
INFO: 172.19.0.7:33444 - "GET /federation/actors/node1channel HTTP/1.1" 403 Forbidden

I try with RSS:

api_1 | INFO: 172.19.0.7:33816 - "GET /api/v1/channels/node1channel/rss HTTP/1.1" 401 Unauthorized

I decide to try with libraries. I create one in node1, with "everyone across all instances" visibility. I try to follow from node2, it fails with another 403 http error, but when I try a second time it succeeds!

[upl-image-preview url=https://forum.funkwhale.audio/assets/files/2022-04-14/1649957613-39120-image.png]

So I decide to try again with the channel. This time it works!

[upl-image-preview url=https://forum.funkwhale.audio/assets/files/2022-04-14/1649957547-512373-image.png]

I don't know what happened, since I didn't modify anything before the successful attempts. My guess is that the instances needed some time to sync?

devilcius

Now that I remember I did modify something. While testing it, I fixed three errors related to the id prop type in VUE views. Maybe it might be related with the subscription channel working again. In any case here's the MR: https://dev.funkwhale.audio/funkwhale/funkwhale/-/merge_requests/1765

petitminion

As a note this error : node1.funkwhale.test/:1 Uncaught DOMException: Failed to read the 'localStorage' property from 'Window': Access is denied for this document. at a (http://node1.funkwhale.test/front/node_modules/.vite/vuex-persistedstate.js?v=8833859e:65:56) at http://node1.funkwhale.test/front/src/store/index.js:34:5

happens if youtried to reach funkwhale on http instead of https....I spend too much time looking about this error hahaha
I still happens in chrome tho' (even allowing third party cookies)

devilcius

petitminion I don't quite understand your point. You mean that the error pops up if using http? The whole setup presumes using https.

petitminion

devilcius yes thats what i meant. And yeah I know the point is to use https I just thought I was trying to connect with https but in fact I was trying with http

petitminion

note 2 : This do not work for me :

export COMPOSE_PROJECT_NAME=node1
export VUE_PORT=8888

I need to enter this variables in the .env file so they are taked into account when building.

devilcius

petitminion the problem is that to test federation you need to export the variable with different values for each instance, otherwise it won't work,. For this setup you cannot use .env file since it's shared between all instances. How come export on the terminal doesn't work for you? What OS are you using?

petitminion

devilcius
I though when docker start the .env variables will be saved inside each container, so after launching I can use another .env...but yeah it seems logic this can create errors..

I'm using kubuntu 21.1. My console is konsole 21.08.1.

petitminion

I have everything up and running but federation do not seems to work :

 Could not fetch remote library

    Error while fetching the library: HTTPSConnectionPool(host='node2.funkwhale.test', port=443): Max retries exceeded with url: /federation/music/libraries/11ef55ef-314d-4c6d-b868-c0dd5757b5ed (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f0b84441af0>: Failed to establish a new connection: [Errno 111] Connection refused'))

devilcius

petitminion see my previous comment

devilcius

petitminion So if you enter export COMPOSE_PROJECT_NAME=node1 and then echo $COMPOSE_PROJECT_NAME you don't get node1?

petitminion

Seems like the console know the variable but docker doesn't :

export COMPOSE_PROJECT_NAME=node1 ; export VUE_PORT=8888
$COMPOSE_PROJECT_NAME
Command 'node1' not found, did you mean:
  command 'node' from deb nodejs (12.21.0~dfsg-3ubuntu1)
Try: sudo apt install <deb name>
sudo  docker compose -f dev.yml up nginx api front nginx api celeryworker 
WARN[0000] The "COMPOSE_PROJECT_NAME" variable is not set. Defaulting to a blank string. 
WARN[0000] The "COMPOSE_PROJECT_NAME" variable is not set. Defaulting to a blank string. 
WARN[0000] network federation: network.external.name is deprecated in favor of network.name

devilcius

petitminion you missed the echo part, 🙂 But the variable is set, there's no doubt about that.

petitminion

devilcius as you can see in the previous command $COMPOSE_PROJECT_NAME throw node1 but docker doesn't seem to take it into account : The "COMPOSE_PROJECT_NAME" variable is not set. Defaulting to a blank string