利用 docker buildx 静态编译 nginx

语言: CN / TW / HK

时间 2022-01-26 18:10:49 Zhangguanzhang

内部有需求需要静态编译 nginx，尝试了下，搞出来了。先是按照官方 nginx Dockerfile 的逻辑走不通，后面下载 nginx 官方源码编译才行。

buildx 使用

nginx Dockerfile

先说下官方的失败尝试。先 clone 项目：

git clone https://github.com/nginxinc/docker-nginx.git
cd docker-nginx

分为 stable 和 mainline 。大概研究了下，发现 case "$apkArch" in x86_64|aarch64) 的情况是利用包管理直接安装的，其他的架构才是源码编译安装。改了下这个 case ，先用 arm64 的试下走源码编译。然后看下逻辑是下载源码进去 make all ：

curl -f -O https://hg.nginx.org/pkg-oss/archive/${NGINX_VERSION}-${PKG_RELEASE}.tar.gz
tar xzvf ${NGINX_VERSION}-${PKG_RELEASE}.tar.gz
&& cd pkg-oss-${NGINX_VERSION}-${PKG_RELEASE} \
&& cd alpine \
&& make all

而源码里 Makefile 的前面内容如下：

ifeq ($(MODULE_TARGET), plus)
APKBUILD_TEMPLATE=	APKBUILD-plus-module.in
MODULE_SUFFIX=	-plus
MODULE_SUMMARY_PREFIX=NGINX Plus
TARGET_VERSION=$(PLUS_VERSION)
MODULE_PACKAGE_PREFIX=nginx-plus-module
else
APKBUILD_TEMPLATE=	APKBUILD-module.in
MODULE_SUMMARY_PREFIX=nginx
TARGET_VERSION=$(BASE_VERSION)
MODULE_PACKAGE_PREFIX=nginx-module
endif

因为不是构建 nginx-plus 所以我们看 cat APKBUILD-module.in 里找到了下面的：

build() {
	cd "$builddir"

	_nproc=`getconf _NPROCESSORS_ONLN`
	if [ $_nproc -gt 1 ]; then
		_make_opts="-j$_nproc"
	fi

	%%MODULE_PREBUILD%%

	cd "$builddir"

	CFLAGS= %%MODULE_ENV%% ./configure %%BASE_CONFIGURE_ARGS%% %%MODULE_CONFIGURE_ARGS%% --with-cc-opt="$CFLAGS %%MODULE_CC_OPT_DEBUG%%" --with-ld-opt="$LDFLAGS %%MODULE_LD_OPT_DEBUG%%" --with-debug
	make $_make_opts modules
	for so in `find objs/ -maxdepth 1 -type f -name "*.so"`; do \
		debugso=`echo ${so} | sed -e 's|\.so$|-debug.so|'` ; \
		mv ${so} ${debugso} ; \
        done
	CFLAGS= %%MODULE_ENV%% ./configure %%BASE_CONFIGURE_ARGS%% %%MODULE_CONFIGURE_ARGS%% --with-cc-opt="$CFLAGS %%MODULE_CC_OPT%%" --with-ld-opt="$LDFLAGS %%MODULE_LD_OPT%%"
	make $_make_opts modules
}

CFLAGS 和 LDFLAGS 是支持环境变量传入的，所以改下 Dockerfile 加入下面的：

ARG CFLAGS
ARG LDFLAGS

然后开始构建：

docker buildx build -t zhangguanzhang/nginx:arm64-static . \
    --platform linux/arm64 \
    --build-arg="CFLAGS='-static -s'" --build-arg=LDFLAGS=-static

然后失败，报错 ./configure: error: the invalid value in --with-ld-opt="-static" ：

#6 481.1 >>> nginx: Unpacking /tmp/tmp.OciAKf/pkg-oss-1.21.5-1/alpine/abuild-base/nginx-1.21.5.tar.gz...
#6 482.9 checking for OS
#6 482.9  + Linux 5.4.0-91-generic aarch64
#6 482.9 checking for C compiler ... found
#6 483.6  + using GNU C compiler
#6 483.7  + gcc version: 10.3.1 20211027 (Alpine 10.3.1_git20211027) 
#6 483.7 checking for gcc -pipe switch ... found
#6 484.3 checking for --with-ld-opt="-static" ... not found
#6 484.3 ./configure: error: the invalid value in --with-ld-opt="-static"

然后在 Dockerfile 里 apk add 加了 glibc-static 还是一样报错。然后尝试下官方的

自己编译

docker run --rm -ti --name t1 test alpine

前置依赖

if [ -f /etc/apk/repositories ];then sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories; fi && \
    if [ -f /etc/apt/sources.list ];then sed -ri 's/(deb|security).debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list; fi && \
    if [ ! -e /etc/nsswitch.conf ];then echo 'hosts: files dns myhostname' > /etc/nsswitch.conf; fi

apk add --no-cache --virtual .build-deps \
                gcc \
                libc-dev \
                make \
                openssl-dev \
                pcre2-dev \
                zlib-dev \
              openssl-libs-static zlib-static  \
                linux-headers \
                libxslt-dev \
                gd-dev \
                geoip-dev \
                perl-dev \
                libedit-dev \
                bash \
                alpine-sdk \
                findutils
bash

下载源码包

wget https://nginx.org/download/nginx-1.21.6.tar.gz
tar zxf nginx-1.21.6.tar.gz
cd nginx-1.21.6

找下官方的编译参数，下面是我 arm64 上：

$ docker run --rm --entrypoint nginx nginx:alpine-perl -V
nginx version: nginx/1.21.6
built by gcc 10.3.1 20211027 (Alpine 10.3.1_git20211027) 
built with OpenSSL 1.1.1l  24 Aug 2021
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --with-perl_modules_path=/usr/lib/perl5/vendor_perl --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-Os -fomit-frame-pointer -g' --with-ld-opt=-Wl,--as-needed,-O1,--sort-common

改下 --with-cc-opt 和 --with-ld-opt 后编译

./configure \
  --prefix=/etc/nginx \
  --sbin-path=/usr/sbin/nginx \
  --modules-path=/usr/lib/nginx/modules \
  --conf-path=/etc/nginx/nginx.conf \
  --error-log-path=/var/log/nginx/error.log \
  --http-log-path=/var/log/nginx/access.log \
  --pid-path=/var/run/nginx.pid \
  --lock-path=/var/run/nginx.lock \
  --http-client-body-temp-path=/var/cache/nginx/client_temp \
  --http-proxy-temp-path=/var/cache/nginx/proxy_temp \
  --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \
  --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \
  --http-scgi-temp-path=/var/cache/nginx/scgi_temp \
  --with-perl_modules_path=/usr/lib/perl5/vendor_perl \
  --user=nginx \
  --group=nginx \
  --with-compat \
  --with-file-aio \
  --with-threads \
  --with-http_addition_module \
  --with-http_auth_request_module \
  --with-http_dav_module \
  --with-http_flv_module \
  --with-http_gunzip_module \
  --with-http_gzip_static_module \
  --with-http_mp4_module \
  --with-http_random_index_module \
  --with-http_realip_module \
  --with-http_secure_link_module \
  --with-http_slice_module \
  --with-http_ssl_module \
  --with-http_stub_status_module \
  --with-http_sub_module \
  --with-http_v2_module \
  --with-mail \
  --with-mail_ssl_module \
  --with-stream \
  --with-stream_realip_module \
  --with-stream_ssl_module \
  --with-stream_ssl_preread_module \
  --with-cc-opt='-static -s' \
  --with-ld-opt=-static

make

make install

然后 Dockerfile 构建:

FROM alpine AS build

ARG NGINX_VERSION=1.21.6
WORKDIR /opt
RUN if [ -f /etc/apk/repositories ];then sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories; fi && \
    if [ -f /etc/apt/sources.list ];then sed -ri 's/(deb|security).debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list; fi && \
    if [ ! -e /etc/nsswitch.conf ];then echo 'hosts: files dns myhostname' > /etc/nsswitch.conf; fi && \
    apk add --no-cache --virtual .build-deps \
                gcc \
                libc-dev \
                make \
                openssl-dev \
                pcre2-dev \
                zlib-dev \
              openssl-libs-static zlib-static  \
                linux-headers \
                libxslt-dev \
                gd-dev \
                geoip-dev \
                perl-dev \
                libedit-dev \
                bash \
                alpine-sdk \
                findutils && \
    wget http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz && tar zxf nginx-${NGINX_VERSION}.tar.gz

RUN cd nginx-${NGINX_VERSION} && \
    ./configure \
    --prefix=/etc/nginx \
    --sbin-path=/usr/sbin/nginx \
    --modules-path=/usr/lib/nginx/modules \
    --conf-path=/etc/nginx/nginx.conf \
    --error-log-path=/var/log/nginx/error.log \
    --http-log-path=/var/log/nginx/access.log \
    --pid-path=/var/run/nginx.pid \
    --lock-path=/var/run/nginx.lock \
    --http-client-body-temp-path=/var/cache/nginx/client_temp \
    --http-proxy-temp-path=/var/cache/nginx/proxy_temp \
    --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \
    --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \
    --http-scgi-temp-path=/var/cache/nginx/scgi_temp \
    --with-perl_modules_path=/usr/lib/perl5/vendor_perl \
    --user=nginx \
    --group=nginx \
    --with-compat \
    --with-file-aio \
    --with-threads \
    --with-http_addition_module \
    --with-http_auth_request_module \
    --with-http_dav_module \
    --with-http_flv_module \
    --with-http_gunzip_module \
    --with-http_gzip_static_module \
    --with-http_mp4_module \
    --with-http_random_index_module \
    --with-http_realip_module \
    --with-http_secure_link_module \
    --with-http_slice_module \
    --with-http_ssl_module \
    --with-http_stub_status_module \
    --with-http_sub_module \
    --with-http_v2_module \
    --with-mail \
    --with-mail_ssl_module \
    --with-stream \
    --with-stream_realip_module \
    --with-stream_ssl_module \
    --with-stream_ssl_preread_module \
    --with-cc-opt='-static -s' \
    --with-ld-opt=-static && \
    make && \
    make install && \
    cp `which nginx` /nginx-$(cat /etc/apk/arch)

FROM scratch AS bin
COPY --from=build /nginx-* /

构建

docker buildx build  . --platform linux/amd64,linux/arm64 \
    --target bin --output . \
    --build-arg=NGINX_VERSION=1.21.6

结果：

# ll                          
total 4                                    
drwxr-xr-x 4 root root   62 Jan 26 16:21 ./             
drwxr-xr-x 6 root root   98 Jan 26 15:31 ../   
-rw-r--r-- 1 root root 2718 Jan 26 16:05 Dockerfile 
drwxr-xr-x 2 root root   26 Jan 26 16:21 linux_amd64/                                          
drwxr-xr-x 2 root root   27 Jan 26 16:21 linux_arm64/ 
# ll linux_a*
linux_amd64:
total 20424
drwxr-xr-x 2 root root       26 Jan 26 16:21 ./
drwxr-xr-x 4 root root       62 Jan 26 16:21 ../
-rwxr-xr-x 1 root root 20910696 Jan 26 16:12 nginx-x86_64*

linux_arm64:
total 20444
drwxr-xr-x 2 root root       27 Jan 26 16:21 ./
drwxr-xr-x 4 root root       62 Jan 26 16:21 ../
-rwxr-xr-x 1 root root 20932656 Jan 26 16:21 nginx-aarch64*
# ./linux_amd64/nginx-x86_64 -V
nginx version: nginx/1.21.6
built by gcc 10.3.1 20211027 (Alpine 10.3.1_git20211027) 
built with OpenSSL 1.1.1l  24 Aug 2021
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --with-perl_modules_path=/usr/lib/perl5/vendor_perl --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-static -s' --with-ld-opt=-static
# file ./linux_amd64/nginx-x86_64 
./linux_amd64/nginx-x86_64: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped
# file ./linux_arm64/nginx-aarch64 
./linux_arm64/nginx-aarch64: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, with debug_info, not stripped

符号链接可以 strip -s $(which nginx) 去掉减少大小。

「其他文章」