keepalived static link build

語言: CN / TW / HK

之前那篇 ipvs svc 的文章,內部已經上生產了,客戶的環境可能完全內網,包管理安裝 keepalived 不現實,所以 keepalived 是部署容器裡的。在容災測試的時候,例如 3 臺機器部署好業務,然後跑壓測指令碼模擬使用者使用,發現關臺機器的時候故障時間很短,但是這個機器開機的期間,還是很大概率故障時間很長,體現在介面的錯誤數量很多。大概看了下,是 keepalived 啟動慢,先試啟動 docker daemon,然後容器啟動是順序不固定,可能 keepalived 很後起來,於是就想著看看能不能 keepalived 拿出來,也就是靜態編譯。

buildx 使用

見文章 buildx 使用

折騰

在官方倉庫提了 issue is there any way to static build 後,和開發者溝通嘗試過不少姿勢都不行,然後有個大佬 hack 下編譯成功了。開發者參照改了下後我試了下最新原始碼試可以整出來了。

主要是在 alpine 容器構建:

git clone https://github.com/acassen/keepalived.git
cd keepalived
docker run -v $PWD:/opt --workdir /opt --rm -ti alpine

構建依賴參考倉庫裡的 Dockerfile.in , static 庫之類的可以 alpinelinux 上去搜索

if [ -f /etc/apk/repositories ];then sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories; fi && \
    if [ -f /etc/apt/sources.list ];then sed -ri 's/(deb|security).debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list; fi && \
    if [ ! -e /etc/nsswitch.conf ];then echo 'hosts: files dns myhostname' > /etc/nsswitch.conf; fi

apk --no-cache add \
    binutils \
    file \
    file-dev \
    gcc \
    glib \
    glib-dev \
    ipset \
    ipset-dev \
    iptables \
    iptables-dev \
    libmnl-dev \
    libnftnl-dev \
    libnl3 \
    libnl3-dev \
    make \
    musl-dev \
    net-snmp-dev \
    openssl \
    openssl-dev \
    openssl-libs-static \
    pcre2 \
    pcre2-dev \
    autoconf \
    automake zlib-static  alpine-sdk linux-headers  libmnl-static

當然,你會發現沒有 configure 指令碼,可以參照 INSTALL 裡執行 ./autogen.sh 生成,install 裡可以參考下,有些包名在 os 上可能換了名字。同時如果要折騰的話,建議看下 --help 的內容。

./autogen.sh
./configure --help

靜態編譯的配置:

CFLAGS='-static -s' LDFLAGS=-static ./configure  --disable-dynamic-linking \
    --prefix=/usr \
    --exec-prefix=/usr \
    --bindir=/usr/bin \
    --sbindir=/usr/sbin \
    --sysconfdir=/etc \
    --datadir=/usr/share \
    --localstatedir=/var \
    --mandir=/usr/share/man \
    --enable-bfd \
    --enable-snmp \
    --enable-snmp-rfc \
    --enable-nftables \
    --enable-regex \
    --enable-json  --with-init=systemd --enable-vrrp --enable-libnl-dynamic

配置資訊:

Keepalived version       : 2.2.7
Compiler                 : gcc gcc (Alpine 10.3.1_git20211027) 10.3.1 20211027
Preprocessor flags       : -D_GNU_SOURCE -DNETSNMP_NO_INLINE
Compiler flags           : -g -static -s -Wall -Wextra -Wunused -Wstrict-prototypes -Wabi -Wabsolute-value -Waddress-of-packed-member -Walloca -Walloc-zero -Warith-conversion -Warray-bounds=2 -Wattribute-alias=2 -Wbad-function-cast -Wc11-c2x-compat -Wcast-align -Wcast-qual -Wdate-time -Wdisabled-optimization -Wdouble-promotion -Wduplicated-branches -Wduplicated-cond -Wfloat-conversion -Wfloat-equal -Wformat-overflow -Wformat-security -Wformat-signedness -Wformat-truncation -Wframe-larger-than=5120 -Wimplicit-fallthrough=3 -Winit-self -Winline -Winvalid-pch -Wjump-misses-init -Wlogical-op -Wmissing-declarations -Wmissing-field-initializers -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs -Wnormalized -Wnull-dereference -Wold-style-definition -Woverlength-strings -Wpointer-arith -Wredundant-decls -Wshadow -Wshift-overflow=2 -Wstack-protector -Wstrict-overflow=4 -Wstringop-overflow=2 -Wstringop-truncation -Wsuggest-attribute=cold -Wsuggest-attribute=const -Wsuggest-attribute=format -Wsuggest-attribute=malloc -Wsuggest-attribute=noreturn -Wsuggest-attribute=pure -Wsync-nand -Wtrampolines -Wundef -Wuninitialized -Wunknown-pragmas -Wunsafe-loop-optimizations -Wunsuffixed-float-constants -Wunused-const-variable=2 -Wvariadic-macros -Wwrite-strings -fPIE -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -O2
Linker flags             : -static -pie -Wl,-z,relro -Wl,-z,now -L/usr/lib
Extra Lib                : -lm -lssl -lcrypto -lnftnl -lmnl -lpcre2-8 -lnetsnmpmibs -lnetsnmpagent -lnetsnmp -lcrypto
Use IPVS Framework       : Yes
IPVS use libnl           : No
IPVS syncd attributes    : Yes
IPVS 64 bit stats        : Yes
HTTP_GET regex support   : Yes
fwmark socket support    : Yes
Use VRRP Framework       : Yes
Use VRRP VMAC            : Yes
Use VRRP authentication  : Yes
With track_process       : Yes
With linkbeat            : Yes
Use BFD Framework        : Yes
SNMP vrrp support        : Yes
SNMP checker support     : Yes
SNMP RFCv2 support       : Yes
SNMP RFCv3 support       : Yes
SNMP send V3 for V2      : Yes
DBUS support             : No
Use JSON output          : Yes
libnl version            : None
Use IPv4 devconf         : Yes
Use iptables             : No
Use nftables             : Yes
init type                : systemd
systemd notify           : No
Strict config checks     : No
Build documentation      : No
Default runtime options  : -D

*** WARNING - this build will not support IPVS with IPv6. Please install libnl/libnl-3 dev libraries to support IPv6 with IPVS.

libnl/libnl-3 這個我試了下加不進去, apk add libnl-dev 可以加上去,但是我看了 alpine 裡 keepalived -v 的 configure 裡也沒我上面開的多。強開我試了 CPPFLAGS='-I/usr/include/libnl3' LDLIBS='-lnl3 -lnl-genl-3' 和下載 libnl-3 編譯安裝後都不行,想折騰和傳遞引數啥的話,多看看 configure 檔案裡的內容。

編譯和安裝:

make && make install

資訊:

/opt # keepalived -v
Keepalived v2.2.7 (02/23,2022), git commit v2.2.7-22-geb533a93

Copyright(C) 2001-2022 Alexandre Cassen, <[email protected]>

Built with kernel headers for Linux 5.10.41
Running on Linux 5.4.0-99-generic #112-Ubuntu SMP Thu Feb 3 13:50:55 UTC 2022
Distro: Alpine Linux v3.15

configure options: --disable-dynamic-linking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --mandir=/usr/share/man --enable-bfd --enable-snmp --enable-snmp-rfc --enable-nftables --enable-regex --enable-json --with-init=systemd --enable-vrrp --enable-libnl-dynamic CFLAGS=-static -s LDFLAGS=-static

Config options:  NFTABLES LVS REGEX VRRP VRRP_AUTH VRRP_VMAC JSON BFD OLD_CHKSUM_COMPAT SNMP_V3_FOR_V2 SNMP_VRRP SNMP_CHECKER SNMP_RFCV2 SNMP_RFCV3 INIT=systemd

System options:  VSYSLOG MEMFD_CREATE IPV6_MULTICAST_ALL IPV4_DEVCONF RTA_ENCAP RTA_EXPIRES RTA_NEWDST RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_SUPPRESS_IFGROUP FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTEXT_FILTER_SKIP_STATS FRA_L3MDEV FRA_UID_RANGE RTAX_FASTOPEN_NO_COOKIE RTA_VIA FRA_PROTOCOL FRA_IP_PROTO FRA_SPORT_RANGE FRA_DPORT_RANGE RTA_TTL_PROPAGATE IFA_FLAGS LWTUNNEL_ENCAP_MPLS LWTUNNEL_ENCAP_ILA NET_LINUX_IF_H_COLLISION NETINET_LINUX_IF_ETHER_H_COLLISION IPVS_DEST_ATTR_ADDR_FAMILY IPVS_SYNCD_ATTRIBUTES IPVS_64BIT_STATS IPVS_TUN_TYPE IPVS_TUN_CSUM IPVS_TUN_GRE VRRP_IPVLAN IFLA_LINK_NETNSID INET6_ADDR_GEN_MODE VRF SO_MARK
/opt # file `which keepalived`
/usr/sbin/keepalived: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped
/opt # ldd `which keepalived`
/lib/ld-musl-x86_64.so.1: /usr/sbin/keepalived: Not a valid dynamic program

buildx 一步到位

FROM alpine as build
RUN if [ -f /etc/apk/repositories ];then sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories; fi && \
    if [ -f /etc/apt/sources.list ];then sed -ri 's/(deb|security).debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list; fi && \
    if [ ! -e /etc/nsswitch.conf ];then echo 'hosts: files dns myhostname' > /etc/nsswitch.conf; fi  && \
    apk --no-cache add \
        binutils \
        file \
        file-dev \
        gcc \
        glib \
        glib-dev \
        ipset \
        ipset-dev \
        iptables \
        iptables-dev \
        libmnl-dev \
        libnftnl-dev \
        libnl3 \
        libnl3-dev \
        make \
        musl-dev \
        net-snmp-dev \
        openssl \
        openssl-dev \
        openssl-libs-static \
        pcre2 \
        pcre2-dev \
        autoconf \
        automake zlib-static  alpine-sdk linux-headers  libmnl-static git
WORKDIR /opt
RUN git clone https://github.com/acassen/keepalived.git

RUN set -ex && \
    cd /opt/keepalived && \
    ./autogen.sh && \
    CFLAGS='-static -s' LDFLAGS=-static ./configure  --disable-dynamic-linking \
    --prefix=/usr \
    --exec-prefix=/usr \
    --bindir=/usr/bin \
    --sbindir=/usr/sbin \
    --sysconfdir=/etc \
    --datadir=/usr/share \
    --localstatedir=/var \
    --mandir=/usr/share/man \
    --enable-bfd \
    --enable-snmp \
    --enable-snmp-rfc \
    --enable-nftables \
    --enable-regex \
    --enable-json  --with-init=systemd --enable-vrrp --enable-libnl-dynamic
RUN set -ex && \
    cd /opt/keepalived && \
    make && \
    make DESTDIR=/install_root install && \
    find /install_root && \
# delete the docs
    rm -rf /install_root/usr/share

FROM scratch AS bin
COPY --from=build /install_root /

構建:

docker buildx build  . --platform linux/amd64,linux/arm64 \
    --target bin --output . 

資訊:

$ ./usr/sbin/keepalived -v
Keepalived v2.2.7 (02/23,2022), git commit v2.2.7-22-geb533a93

Copyright(C) 2001-2022 Alexandre Cassen, <[email protected]>

Built with kernel headers for Linux 5.10.41
Running on Linux 5.4.0-99-generic #112-Ubuntu SMP Thu Feb 3 13:50:55 UTC 2022
Distro: Ubuntu 20.04.3 LTS

configure options: --disable-dynamic-linking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --mandir=/usr/share/man --enable-bfd --enable-snmp --enable-snmp-rfc --enable-nftables --enable-regex --enable-json --with-init=systemd --enable-vrrp --enable-libnl-dynamic CFLAGS=-static -s LDFLAGS=-static

Config options:  NFTABLES LVS REGEX VRRP VRRP_AUTH VRRP_VMAC JSON BFD OLD_CHKSUM_COMPAT SNMP_V3_FOR_V2 SNMP_VRRP SNMP_CHECKER SNMP_RFCV2 SNMP_RFCV3 INIT=systemd

System options:  VSYSLOG MEMFD_CREATE IPV6_MULTICAST_ALL IPV4_DEVCONF RTA_ENCAP RTA_EXPIRES RTA_NEWDST RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_SUPPRESS_IFGROUP FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTEXT_FILTER_SKIP_STATS FRA_L3MDEV FRA_UID_RANGE RTAX_FASTOPEN_NO_COOKIE RTA_VIA FRA_PROTOCOL FRA_IP_PROTO FRA_SPORT_RANGE FRA_DPORT_RANGE RTA_TTL_PROPAGATE IFA_FLAGS LWTUNNEL_ENCAP_MPLS LWTUNNEL_ENCAP_ILA NET_LINUX_IF_H_COLLISION NETINET_LINUX_IF_ETHER_H_COLLISION IPVS_DEST_ATTR_ADDR_FAMILY IPVS_SYNCD_ATTRIBUTES IPVS_64BIT_STATS IPVS_TUN_TYPE IPVS_TUN_CSUM IPVS_TUN_GRE VRRP_IPVLAN IFLA_LINK_NETNSID INET6_ADDR_GEN_MODE VRF SO_MARK

$ ldd ./usr/sbin/keepalived 
	not a dynamic executable
$ file ./usr/sbin/keepalived
./usr/sbin/keepalived: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped
$ ls -lh ./usr/sbin/keepalived
-rwxr-xr-x 1 root root 4.4M Feb 24 17:56 ./usr/sbin/keepalived