Table of contents
- The Issue
- The Solution(s)
- The Implementation of the advanced solution (Just for fun!)
From our past blog Best Logging Solution for Docker [Basic Version],
We saw how to make logging a piece of cake using Loki & Grafana.
But I think there is one minor issue If your servers are on the public internet and you have enabled port forwarding for your use case, then anyone using your Loki URL, can add it as a data source in their Grafana Installation and will be able to see all of your sensitive logs, which is the last thing on earth we want.
It's simple right?
When you run Loki, out of the box, it does not come with any kind of additional security layer. So basically anyone who can access that URL, can access and see your logs, simple as that!
Loki Official Documentation says that Grafana Loki does not come with any included authentication layer.
Now as I am thinking about it, feels like we can use some simple tactics to enhance security. Those are :
If not required, don't launch your servers on the public subnet. (hence reducing the possibility of someone connecting from outside your network)
Do not use port-forwarding, instead use DNS which is supported by the docker network. (for example, instead of
http://Server_IP:3100we can use
Use some firewall/SecurityGroups to whitelist services/users to allow limited access to required ports.
Advance - You can use a reverse proxy(NGINX) with Basic Auth. So even if someone has connectivity to the endpoint, they need to enter basic auth credentials to connect :)
The Implementation of the advanced solution (Just for fun!)
Step-1: Keep your environment ready
I am assuming, you already have a docker environment ready that is already using Loki without any auth method.
You can take a reference from my previous blog as well.
Do you know what I did?
I referred to this official docker-compose file to get started in a few seconds.
version: "3" networks: loki: services: loki: image: grafana/loki:2.6.1 ports: - "3100:3100" command: -config.file=/etc/loki/local-config.yaml networks: - loki promtail: image: grafana/promtail:2.6.1 volumes: - /var/log:/var/log command: -config.file=/etc/promtail/config.yml networks: - loki grafana: image: grafana/grafana:latest ports: - "3000:3000" networks: - loki
Step-2: Prepare the NGINX reverse proxy with Basic Auth
Luckily I stumbled upon this cool open-source project which does the job for you :
So now what? How to use this?
You just have to give the required env vars and it will launch NGINX accordingly.
- BASIC_AUTH_USERNAME & BASIC_AUTH_PASSWORD => Basic Auth Creds
- PROXY_PASS => where you want to forward your request from NGINX reverse proxy
- PORT => The port, on which you want NGINX to listen.
Note: Here we will add our Loki URL in the PROXY_PASS section so it can forward our request to Loki.
Step-3: Combining all the pieces together
So Now, as we know how can we secure our Loki Installation, let's add this reverse proxy in our
version: "3" networks: loki: services: loki: image: grafana/loki:2.6.1 ports: - 3100 command: -config.file=/etc/loki/local-config.yaml networks: - loki promtail: image: grafana/promtail:2.6.1 volumes: - /var/log:/var/log command: -config.file=/etc/promtail/config.yml networks: - loki grafana: image: grafana/grafana:latest ports: - "3000:3000" networks: - loki nginx: image: quay.io/dtan4/nginx-basic-auth-proxy environment: - BASIC_AUTH_USERNAME=loki_user - BASIC_AUTH_PASSWORD=loki_super_secret_password - PROXY_PASS=http://loki:3100 - PORT=3200 - SERVER_NAME=_ ports: - 3200:3200 - 8090:8090 networks: - loki
Here I also disabled port-forwarding for Loki, because that is not required. Also, notice that in the PROXY_PASS env var, I gave
loki:3100 which is the service name in the
docker-compose file and is internally resolvable by the containers in the same docker network.
Here, we are running this NGINX server on port 3200 + we did port forwarding on the same port, so we can use localhost:3200 to connect to the NGINX server.
Also, port 8090 is used for NGINX metrics. You can check that on ->
Step-4: See it in action!
Now, up this docker-compose file and see what happens at the Grafana end.
So, let's try to add a data source, just like before.
Trial-1: On old port 3100
As expected, we have disabled the port forwarding, it should fail.
Trial-2: With NGINX Port (w/o creds)
Again, this also fails as above.
Just for curiosity, let's
curl and see.
Oh ok, let's try with the creds which we supplied to NGINX in docker-compose.
- BASIC_AUTH_USERNAME=loki_user - BASIC_AUTH_PASSWORD=loki_super_secret_password
Trial-3: NGINX with Basic Auth Creds
Now using the same endpoint, Enable the Basic Auth, and add those basic auth credentials
Voilà! - we were able to connect to our Loki Data source with an additional layer of security i.e. Basic Auth.
I know most of you are using Kubernetes out there and only a small set of users can relate to this(and implement it) but it is a really small fun exercise on how to use NGINX reverse proxy, setup Basic Auth, docker-compose(or docker swarm) inter-service communication and securing Loki installation. 😄
and the other reason to be happy is:
That was it! Let me know what you have to say in the comment box.
Thanks for reading :)
Did you find this article valuable?
Support Kratik Jain by becoming a sponsor. Any amount is appreciated!