In this post I wanted to make a demo of Nginx VTS + Prometheus + Grafana. For the demo, it was necessary that upstream could issue different http codes. The following projects could do this: Mockify , written in Golang, and WireMock , written in Java.
A small table of contents
- installation and configuration of Nginx VTS + Prometheus + Grafana;
- Mockify is a lightweight, configurable emulation API written in Golang;
- Comparison of CPU usage for Mockify, written in Golang, and WireMock, written in Java.
Test stand virtual machine:
inxi CPU: 8x Single Core Intel Xeon E312xx (Sandy Bridge) (-SMP-) speed: 2594 MHz Kernel: 3.10.0-957.1.3.el7.x86_64 x86_64 Up: 58m Mem: 474.9/32011.6 MiB (1.5%) Storage: 80.00 GiB (2.7% used) Procs: 149 Shell: bash 4.2.46 inxi: 3.0.35
Config prometheus:
global: scrape_interval: 5s # Set the scrape interval to every 15 seconds. Default is every 1 minute. evaluation_interval: 5s # Evaluate rules every 15 seconds. The default is every 1 minute. # scrape_timeout is set to the global default (10s). # Alertmanager configuration alerting: alertmanagers: - static_configs: - targets: # - alertmanager:9093 # Load rules once and periodically evaluate them according to the global 'evaluation_interval'. rule_files: # - "first_rules.yml" # - "second_rules.yml" scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] - job_name: 'nginx_vts_exporter' static_configs: - targets: ['localhost:9913']
Config Grafana standard. Dashboard ID 2949.
For Nginx VTS, you need to compile nginx with the nginx-module-vts module. We do this using Nginx-builder . His config:
nginx_version: 1.16.1 output_package: rpm modules: - module: name: nginx-module-vts git_url: https://github.com/vozlt/nginx-module-vts.git git_tag: v0.1.18
Install the assembled nginx. Here is its main config (do not forget to specify vhost_traffic_status_zone;):
user nginx; worker_processes auto; worker_rlimit_nofile 40960; error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid; events { use epoll; worker_connections 1024; multi_accept on; } http { vhost_traffic_status_zone; include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; #access_log /var/log/nginx/access.log main; access_log off; sendfile on; tcp_nodelay on; tcp_nopush on; keepalive_timeout 65; include /etc/nginx/conf.d/*.conf; open_file_cache max=200000 inactive=20s; open_file_cache_valid 30s; open_file_cache_min_uses 2; open_file_cache_errors on; }
Create virtual host configs:
touch vhost1.conf vhost2.conf vhost3.conf vhost4.conf vhost5.conf
Their contents:
vhost1.conf:
server { listen 80; server_name vhost1; location / { proxy_pass http://127.0.0.1:8001/102; } }
vhost2.conf:
server { listen 80; server_name vhost2; location / { proxy_pass http://127.0.0.1:8001/204; } }
vhost3.conf:
server { listen 80; server_name vhost3; location / { proxy_pass http://localhost:8001/304; } }
vhost4.conf:
server { listen 80; server_name vhost4; location / { proxy_pass http://localhost:8001/403; } }
vhost5.conf:
server { listen 80; server_name vhost5; location / { proxy_pass http://localhost:8001/503; } }
Install nginx-vts-exporter:
wget https://github.com/hnlq715/nginx-vts-exporter/releases/download/v0.10.3/nginx-vts-exporter-0.10.3.linux-amd64.tar.gz tar zxvf nginx-vts-exporter-0.10.3.linux-amd64.tar.gz cp nginx-vts-exporter-0.10.3.linux-amd64/nginx-vts-exporter /usr/local/bin/nginx-vts-exporter
Create the file /etc/systemd/system/nginx_vts_exporter.service:
[Unit] Description=Nginx vts exporter Wants=network-online.target After=network-online.target [Service] ExecStart=/usr/local/bin/nginx-vts-exporter -nginx.scrape_uri=http://localhost:7070/status/format/json Restart=always RestartSec=3 [Install] WantedBy=default.target
Create the VTS configuration file /etc/nginx/conf.d/vts-exporter.conf
server { listen 7070; listen [::]:7070; location / { } location /status { vhost_traffic_status_display; vhost_traffic_status_display_format html; } }
Add DNS of virtual hosts to / etc / hosts:
127.0.0.1 vhost1 127.0.0.1 vhost2 127.0.0.1 vhost3 127.0.0.1 vhost4 127.0.0.1 vhost5
We are optimizing the kernel for testing, since wiremock gave out a timeout with a large number of connections. Now it may not be necessary, as it reduced the number of connections when starting the apache benchmark. Create the file /etc/sysctl.d/90-nginx.conf with the contents:
fs.file-max=100000 net.netfilter.nf_conntrack_max=1548576 net.ipv4.ip_local_port_range=10000 65000 net.ipv4.tcp_tw_reuse=1 net.core.somaxconn=15600 net.ipv4.tcp_fin_timeout=15 net.ipv4.tcp_tw_recycle=1 net.core.rmem_default=31457280 net.core.rmem_max=12582912 net.core.wmem_default=31457280 net.core.wmem_max=12582912 net.core.netdev_max_backlog=65536 net.core.optmem_max=25165824 net.ipv4.tcp_rmem=8192 87380 16777216 net.ipv4.udp_rmem_min=16384 net.ipv4.tcp_wmem=8192 65536 16777216
Apply settings
sysctl -p /etc/sysctl.d/90-nginx.conf
Install mockify-rpm
yum -y install yum-plugin-copr yum copr enable antonpatsev/mockify-rpm yum -y install mockify systemctl start mockify
Install Apache Benchmark:
yum install -y httpd-tools
We start a little nginx testing:
while true; do ab -c 1 -n 1 -t 1 http://vhost1/; sleep 2; done while true; do ab -c 1 -n 1 -t 1 http://vhost2/; sleep 2; done while true; do ab -c 1 -n 1 -t 1 http://vhost3/; sleep 2; done while true; do ab -c 1 -n 1 -t 1 http://vhost4/; sleep 2; done while true; do ab -c 1 -n 1 -t 1 http://vhost5/; sleep 2; done
Screenshots:
Install wiremock:
yum -y install yum-plugin-copr yum copr enable antonpatsev/wiremock-rpm yum -y install wiremock wiremock-popular-json systemctl start wiremock
Also, in the vhost1-vhost5 files in nginx, you need to change the port from 8001 to 8080.
Sorry for the jagged screenshots.
Below is the CPU and MEM mockify load when testing vhost1-vhost5
Below is CPU load and MEM wiremock when testing vhost1
Below is the CPU and MEM wiremock load when testing vhost1-vhost2
Below is CPU load and MEM wiremock when testing vhost1-vhost3
Below is CPU load and MEM wiremock when testing vhost1-vhost4
Below is the CPU and MEM wiremock load when testing vhost1-vhost5. Sometimes the load on the CPU grew up to 700%.
Conclusions:
According to Nginx VTS, I would like more metrics without editing configs.
By Wiremock vs Mockify: use Mockify. It uses less CPU and MEM.
And finally, the assembly of Golang applications in Fedora Copr using the example of Mockify.
Use the repository https://github.com/patsevanton/mockify-rpm as an example.