Figured it out! I was, in fact, missing step 4 in the reverse proxy setup. It even says so in the config:
# If you are using S3 to host your files, remember to add your S3 URL to the
# media-src and img-src headers (e.g. img-src 'self' https://<your-S3-URL> data:)
Did that and all is well now!