Recently, we built a dynamic module for Nginx, and when everything was ready, it turned out that our module was not compatible with Nginx, which was already installed on the server. We could not find a ready-made solution to the problem and we began to fight it on our own. We spent a lot of time, but got a new experience and, most importantly, a working solution. Which I would like to share.
Let's start with a description of the dynamic module assembly process using the example https://github.com/vozlt/nginx-module-vts . And then we will show what error occurs and what to do with it.
Inputs:
Debian 9 OS
Nginx v1.10.3 from the standard Debian repository. nginx -V output:
nginx version: nginx/1.10.3 built with OpenSSL 1.1.0k 28 May 2019 (running with OpenSSL 1.1.1c 28 May 2019) TLS SNI support enabled configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-DhOtPd/nginx-1.10.3=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/ nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path =/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module - -with-http_stub_status_module --with-http_realip_mod ule --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module= dynamic --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-mail=dynamic --with-mail_ssl_module --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-auth-pam --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-dav-ext-module --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-echo --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-upstream-fair --add-dynamic-module=/build/nginx-DhOtPd/nginx -1.10.3/debian/modules/ngx_http_substitutions_filter_module
The output of this command contains information that is necessary to build dynamic modules, namely, everything that is written after configure arguments: and before the first - --add-dynamic-module , that is:
--with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-DhOtPd/nginx-1.10.3=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/ nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path =/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_mod ule --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module= dynamic --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-mail=dynamic --with-mail_ssl_module
So, we proceed to the assembly of the module:
- For assembly, it is convenient to use a docker container so as not to clutter up the main system, take the debian: 9 image as the basis, for the convenience of copying the finished module from the container, you can specify volume. The command to start the container in this case will look like this:
docker run --rm -it -v /tmp/nginx_module:/nginx_module debian:9 bash
- After starting the container, install the necessary packages. Please note that the packages are selected according to the output of the nginx -V command, so you may not need some libraries. In any case, the configure script will warn you about the absence of dependencies:
apt update apt install git make gcc autoconf wget libpcre3-dev libpcre++-dev zlib1g-dev libxml2-dev libxslt-dev libgd-dev libgeoip-dev
The libssl-dev package is missing from this list, because the nginx installed on the server is built with the version 1.1.0k of OpenSSL, and at the time of writing when installing the libssl-dev package from the repository, the version of OpenSSL in it was already 1.1.0l. Therefore, the OpenSSL source code for the correct version must be downloaded separately.
cd /usr/local/src/ wget http://nginx.org/download/nginx-1.10.3.tar.gz wget https://www.openssl.org/source/old/1.1.0/openssl-1.1.0k.tar.gz git clone git://github.com/vozlt/nginx-module-vts.git tar xvfz nginx-1.10.3.tar.gz tar xzvf openssl-1.1.0k.tar.gz cd nginx-1.10.3
- Now you are ready to run the configure script. We specify all parameters from the output of the nginx -V command as the script parameters, which I wrote about at the beginning of the article. We also add the --add-dynamic-module parameter to build the nginx-module-vts module and specify the path to the directory with the OpenSSL source files through the --with-openssl parameter. The final command will look like this:
./configure --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-DhOtPd/nginx-1.10.3=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-mail=dynamic --with-mail_ssl_module --with-openssl=../openssl-1.1.0k/ --add-dynamic-module=../nginx-module-vts/
- After the
configure script finishes working, we start the assembly of modules; we don’t need to collect all nginx :
make modules
Note that this command is only available with nginx version 1.9.13.
- After the assembly process is completed, copy the resulting so file to
volume , which was mounted when the container started:
cp objs/ngx_http_vhost_traffic_status_module.so /nginx_module/
- Next, we place the file on the target server, in this case, the directory with the modules is located on the path
/usr/share/nginx/modules . In order to enable the module in nginx we create the file /etc/nginx/modules-enabled/mod-http-vhost-traffic-status.conf with the following contents:
load_module modules/ngx_http_vhost_traffic_status_module.so;
And create a symbolic link to this file in the /etc/nginx/modules-enabled directory.
After the work done, we call the nginx -t command and ... we get the error:
nginx: [emerg] module "/usr/share/nginx/modules/ngx_http_vhost_traffic_status_module.so" is not binary compatible in /etc/nginx/modules-enabled/50-mod-http-vhost-traffic-status.conf:1
And here, usually, there is some bewilderment, because the source versions and the nginx version installed on the server are the same, the OpenSSL versions too, the parameters for the configure script are also copied entirely. So what's the deal?
To understand this problem, I had to understand how nginx checks if the dynamic module is suitable for it. The code that is located in the nginx sources in the src/core/ngx_module.h ( https://github.com/nginx/nginx/blob/master/src/core/ngx_module.h ) is responsible for this check. There are a number of checks in this file, in this case 34, during which nginx sets for variables of the form NGX_MODULE_SIGNATURE_0 (1,2,3, etc.)
values 1 or 0. Next is the variable NGX_MODULE_SIGNATURE
which collects the results of all checks in one line. Accordingly, the problem is that the signature string of the new module does not match the signature string of the existing nginx. To check the values of signature strings, you can use the following commands:
- The signature line of the module that was built:
strings /usr/share/nginx/modules/ngx_http_vhost_traffic_status_module.so| fgrep '8,4,8' 8,4,8,000011111101011111111111110110111
- Signature line of the
nginx binary file installed on the server:
strings /usr/sbin/nginx| fgrep '8,4,8' 8,4,8,000011111101011111111111110111111
When comparing these lines with the naked eye, it can be seen that in the module signature line, the fourth variable from the end turned out to be 0, while nginx has 1. It is necessary to refer to the src/core/ngx_module.h file to understand what exactly is missing in the module src/core/ngx_module.h and find the fourth variable from the end, it looks something like this:
#if (NGX_HTTP_REALIP) #define NGX_MODULE_SIGNATURE_29 "1" #else #define NGX_MODULE_SIGNATURE_29 "0" #endif #if (NGX_HTTP_HEADERS) #define NGX_MODULE_SIGNATURE_30 "1" #else #define NGX_MODULE_SIGNATURE_30 "0" #endif #if (NGX_HTTP_DAV) #define NGX_MODULE_SIGNATURE_31 "1" #else #define NGX_MODULE_SIGNATURE_31 "0" #endif #if (NGX_HTTP_CACHE) #define NGX_MODULE_SIGNATURE_32 "1" #else #define NGX_MODULE_SIGNATURE_32 "0" #endif #if (NGX_HTTP_UPSTREAM_ZONE) #define NGX_MODULE_SIGNATURE_33 "1" #else #define NGX_MODULE_SIGNATURE_33 "0" #endif #define NGX_MODULE_SIGNATURE
We are interested in the variable NGX_HTTP_HEADERS , when building nginx it was 1, and when building the module 0. In order for the module to NGX_HTTP_HEADERS with NGX_HTTP_HEADERS you need to adjust the config file in the directory with the source code of the module by adding the line have=NGX_HTTP_HEADERS . auto/have have=NGX_HTTP_HEADERS . auto/have at the beginning of the config. file config. The following is the beginning of the config file after making the changes:
ngx_addon_name=ngx_http_vhost_traffic_status_module have=NGX_STAT_STUB . auto/have have=NGX_HTTP_HEADERS . auto/have ...
Next, do make clean , and then re-run configure and make modules . After assembling the module, we check its signature string and see that now it corresponds to the nginx string:
$ strings /usr/sbin/nginx| fgrep '8,4,8' 8,4,8,000011111101011111111111110111111 $ strings /usr/share/nginx/modules/ngx_http_vhost_traffic_status_module.so | fgrep '8,4,8' 8,4,8,000011111101011111111111110111111
After that, the module connects without any problems and you can collect advanced statistics with nginx .
In your case, perhaps the signature line will look different, as well as the variables that are responsible for each position in this line. However, now you can understand what exactly is wrong with the assembly of the module and fix the problem.
I hope this article saves someone a few hours of time, since I personally several times faced the problem that the module did not fit nginx , as a result, as a rule, I solved this through the full assembly of nginx along with the desired module.
Also read other articles on our blog: