Testing PMM with Docker

If you work with MySQL or PostgreSQL, probably you at least heard about PMM (Percona Monitoring and Management). If you want to test it, you may also be interested in this article: I’ll show you how to setup a MySQL and a MariaDB instance and monitor them in the quickest way. Which is Docker.

Docker is also my favourite way of running PMM in production. To be clear, I wouldn’t run a production DBMS in Docker. But now we are talking about testing something quickly on your laptop, without wasting too much time. And I don’t know a faster way.

Note: PMM is not a single piece of software. It is a solution created by making use of existing software projects, like Grafana, Prometheus and Orchestrator. In this post I want to hide this complexity, even if the cost is a possible oversimplification. I hope you don’t mind if I’ll keep writing PMM even when I could mean Grafana or Prometheus.

For more information, see Overview of Percona Monitoring and Management Architecture.

Running and monitoring MySQL/MariaDB in Docker

Linus Torvalds said “Talking is cheap, show me the code”. Let’s avoid long discussions, let’s code! But after that, you’ll also find some notes which explain why we did some weird things. After all, we should always understand what we do (ha ha ha).

Talking is cheap…

Here is what we will do, at a high level:

  • Create a Docker network;
  • Start PMM in Docker;
  • Start a MySQL or MariaDB container, or both;
  • Install the PMM client in the DBMSs containers.

…show me the code!

Enough babbling. Here is the code.


docker network create test-pmm
docker pull percona/pmm-server:latest
docker create \
    -v /opt/prometheus/data \
    -v /opt/consul-data \
    -v /var/lib/mysql \
    -v /var/lib/grafana \
    --name pmm-data \
    --network test-pmm \
    percona/pmm-server:latest /bin/true
docker run -d \
    -p 80:80 \
    --volumes-from pmm-data \
    --name pmm-server \
    --restart always \
    --network test-pmm \
    percona/pmm-server:latest

Now, visit http://localhost with your browser. You should see PMM up and working. But it is not interesting, because it is not monitoring anything… well, almost. Actually PMM contains Prometheus, which always monitors itself. So you can click on Insight -> Prometheus.

PMM allows to select the instance to monitor: pmm-server, x-mysql-8.0, x-mariadb-10.0
We can see pmm-server (it monitors itself),
as well as the containers we created

Now create your MySQL container and/or your MariaDB container:


docker run --name x-mysql-8.0 -h x-mysql-8.0 \
    -e MYSQL_ROOT_PASSWORD=secret -d --network test-pmm mysql:8.0
docker run --name x-mariadb-10.3 -h x-mariadb-10.3 \
    -e MYSQL_ROOT_PASSWORD=secret -d --network test-pmm mariadb:10.3

Now open BASH in your containers:

docker exec -ti x-mysql-8.0 bash

Start mysql client and run these queries (the PMM agent needs to have a MySQL user):


CREATE USER pmm@127.0.0.1 IDENTIFIED BY 'my_secret';
GRANT SELECT, PROCESS, SUPER, REPLICATION CLIENT, RELOAD
    ON *.*
    TO pmm@127.0.0.1;
GRANT SELECT, UPDATE, DELETE, DROP
    ON performance_schema.*
    TO pmm@127.0.0.1;

And now, in the containers BASH, let’s run these commands to install and configure pmm-admin:


apt-get update
apt-get install -y wget
wget https://repo.percona.com/apt/percona-release_latest.generic_all.deb
dpkg -i percona-release_latest.generic_all.deb
apt --fix-broken install -y
dpkg -i percona-release_latest.generic_all.deb
apt-get update
mkdir /etc/init
apt-get install pmm-client
pmm-admin config --server pmm-server:80
pmm-admin add linux:metrics
pmm-admin add mysql:metrics --host 127.0.0.1 --user pmm --password my_secret
pmm-admin add mysql:queries --host 127.0.0.1 --user pmm --password my_secret

At this point, if you go back to Grafana, you should see MySQL metrics appear.

Uninstall

At some point, you may want to destroy these container and the bridge network, leaving the system clean. It’s simple:


docker rm -f pmm-server
docker rm -f pmm-data
docker rm -f x-mariadb-10.3
docker rm -f x-mysql-8.0
docker network rm test-pmm
InnoDB buffer pool requests, monitored by PMM
Metrics collection just began. Thank you, PMM!

Notes

Since we have done some weird things, it’s good to document them.

Hostnames

It is not possible to change the hostname of a running Docker container. Trying to run the hostname command will fail. Fortunately, we can set the hostname at creation time, with the -h option.

But why is it important? Because Grafana shows a menu with the names of the monitored instanced. We want to see readable names, not random sequences of characters.

Add Percona repository twice

The error generated when trying to add Percona repository
The error message when trying to add a repository
makes me sad

The first time that you try to add Percona repo (dpkg -i percona-release_latest.generic_all.deb) the command will fail. Most likely because the lsb-release package is not present, and it fails to install it. After that, we run apt to fix dependency problems. Then we retry to add the repository, and this time we succeed.

I reported this bug: ENG-97.
UPDATE: Not a bug, see link and comments below.

Create /etc/init

Trying to install the pmm-client package will fail if the /etc/init directory is not present. In MySQL and MariaDB containers, we have to create it manually.

I reported this bug: PMM-3520.
UPDATE: Fix released.

Strange failures in MySQL container

The pmm-admin add commands failed for me in the MySQL container. Well, they only apparently failed. When I checked PMM, I saw that the container was monitored. So, ignore errors in the consolle and check if PMM is showing metrics.

pmm-admin reports an exit status of 1
This exit status makes me sad,
but I know that everything is ok.

So, everything worked? Let me know if you had any problems.

Toodle pip,
Federico

To also monitor ProxySQL, see this post:
Monitoring ProxySQL with PMM in Docker

Comments (10)

  1. Hi Federico,

    Installation of pmm-client in containers is not supported at this point so this is not a bug but lack of feature
    The intended use is to install it on the main host even if you deploy MySQL/MariaDB with docker rather than conventional package.

    1. Privet Peter. Acknowledged, thank you for your answer.
      Anyway, even if not officially supported, I’m glad that the workaround above allow to install the client in a container. In this way, we don’t have to touch the host.

  2. Hi Federico,

    In addition to Peter’s mention of pmm-client not being currently supported in containers, I will mention that we have a publicly available experimental Docker container image designed for microservice infrastructures. This can be used in Kubernetes via a provided helm chart or directly in Docker using environment variables. Basic usage instructions are included in this forum post:

    https://www.percona.com/forums/questions-discussions/percona-monitoring-and-management/52822-how-to-install-pmm-agent-into-docker-container?p=52872#post52872

    Regarding the percona-release deb package requiring lsb-release, it is included in even the most minimal physical, hypervisor and cloud instances on all of our support platorms. However, lsb-release is generally not included in the base platform Docker images as LSB is not always necessary in microservice environments.

    Thanks again for giving us the opportunity to point these things out and helping us understand better how users plan on evaluating our software. These examples assist us in becoming a better provider of advanced open source database solutions for our customers and the community.

    1. Thank you for all the clarifications David.
      I don’t know if what I described here is representative of how users test things. The “Docker philosophy” is probably to have pmm-admin in a separate container (a separate one for each monitored system), but this adds a lot of complexity and I don’t see a real benefit (at least, not if I just want to make some tests).

  3. I needed to put you this very small note to finally say thank you once again for the breathtaking guidelines you have featured at this time. It’s simply seriously open-handed with you to supply without restraint exactly what many people would have marketed for an e book to help make some profit on their own, especially considering that you could have done it in case you desired. The guidelines likewise served to become a good way to be sure that other individuals have the identical desire similar to mine to understand whole lot more with respect to this condition. I’m sure there are many more pleasant instances ahead for individuals who view your website.

    1. Hi Allison! Your kind feedback is much appreciated. This article and others could end up in an ebook someday, but even in that case, the book will be free. Everyone who appreciates this contents is free to contribute, improving my work. I will benefit from this. And of course, sharing knowledge is also a form of personal branding. I still do business – but I try to do it honestly and, as you said, in a open-handed way.

  4. I wanted to compose you the tiny remark to finally thank you over again with your extraordinary concepts you’ve featured here. This has been shockingly generous of you in giving openly precisely what a lot of folks could possibly have marketed as an electronic book in making some profit on their own, especially given that you might have done it if you ever desired. The tricks likewise worked like the fantastic way to recognize that some people have the identical eagerness just like my very own to know the truth more when it comes to this issue. Certainly there are a lot more pleasurable sessions up front for people who find out your website.

Leave a Reply

Your email address will not be published. Required fields are marked *