【雲原生】zookeeper + kafka on k8s 環境部署

語言: CN / TW / HK

一、概述

  • Apache ZooKeeper 是一個集中式服務,用於維護配置信息、命名、提供分佈式同步和提供組服務,ZooKeeper 致力於開發和維護一個開源服務器,以實現高度可靠的分佈式協調,其實也可以認為就是一個分佈式數據庫,只是結構比較特殊,是樹狀結構。官網文檔:https://zookeeper.apache.org/doc/r3.8.0/ ,關於Zookeeper的介紹,也可以參考我之前的文章:分佈式開源協調服務——Zookeeper

- - Kafka是最初由Linkedin公司開發,是一個分佈式、支持分區的(partition)、多副本的(replica),基於zookeeper協調的分佈式消息系統。官方文檔:https://kafka.apache.org/documentation/關於Kafka的介紹,也可以參考我之前的文章:Kafka原理介紹+安裝+基本操作

在這裏插入圖片描述

二、Zookeeper on k8s 部署

在這裏插入圖片描述

1)添加源

部署包地址:https://artifacthub.io/packages/helm/zookeeper/zookeeper

bash helm repo add bitnami https://charts.bitnami.com/bitnami helm pull bitnami/zookeeper tar -xf zookeeper-10.2.1.tgz

2)修改配置

  • 修改zookeeper/values.yaml

```bash image: registry: myharbor.com repository: bigdata/zookeeper tag: 3.8.0-debian-11-r36 ...

replicaCount: 3

...

service: type: NodePort nodePorts: #NodePort 默認範圍是 30000-32767 client: "32181" tls: "32182"

...

persistence: storageClass: "zookeeper-local-storage" size: "10Gi" # 目錄需要提前在宿主機上創建 local: - name: zookeeper-0 host: "local-168-182-110" path: "/opt/bigdata/servers/zookeeper/data/data1" - name: zookeeper-1 host: "local-168-182-111" path: "/opt/bigdata/servers/zookeeper/data/data1" - name: zookeeper-2 host: "local-168-182-112" path: "/opt/bigdata/servers/zookeeper/data/data1"

...

Enable Prometheus to access ZooKeeper metrics endpoint

metrics: enabled: true - 添加`zookeeper/templates/pv.yaml`yaml {{- range .Values.persistence.local }}


apiVersion: v1 kind: PersistentVolume metadata: name: {{ .name }} labels: name: {{ .name }} spec: storageClassName: {{ $.Values.persistence.storageClass }} capacity: storage: {{ $.Values.persistence.size }} accessModes: - ReadWriteOnce local: path: {{ .path }} nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - {{ .host }}


{{- end }}

`` - 添加zookeeper/templates/storage-class.yaml`

bash kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: {{ .Values.persistence.storageClass }} provisioner: kubernetes.io/no-provisioner

3)開始安裝

```bash

先準備好鏡像

docker pull docker.io/bitnami/zookeeper:3.8.0-debian-11-r36 docker tag docker.io/bitnami/zookeeper:3.8.0-debian-11-r36 myharbor.com/bigdata/zookeeper:3.8.0-debian-11-r36 docker push myharbor.com/bigdata/zookeeper:3.8.0-debian-11-r36

開始安裝

helm install zookeeper ./zookeeper -n zookeeper --create-namespace ``` NOTES

```bash NAME: zookeeper LAST DEPLOYED: Sun Sep 18 18:24:03 2022 NAMESPACE: zookeeper STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: CHART NAME: zookeeper CHART VERSION: 10.2.1 APP VERSION: 3.8.0

Please be patient while the chart is being deployed

ZooKeeper can be accessed via port 2181 on the following DNS name from within your cluster:

zookeeper.zookeeper.svc.cluster.local

To connect to your ZooKeeper server run the following commands:

export POD_NAME=$(kubectl get pods --namespace zookeeper -l "app.kubernetes.io/name=zookeeper,app.kubernetes.io/instance=zookeeper,app.kubernetes.io/component=zookeeper" -o jsonpath="{.items[0].metadata.name}")
kubectl exec -it $POD_NAME -- zkCli.sh

To connect to your ZooKeeper server from outside the cluster execute the following commands:

export NODE_IP=$(kubectl get nodes --namespace zookeeper -o jsonpath="{.items[0].status.addresses[0].address}")
export NODE_PORT=$(kubectl get --namespace zookeeper -o jsonpath="{.spec.ports[0].nodePort}" services zookeeper)
zkCli.sh $NODE_IP:$NODE_PORT

``` 在這裏插入圖片描述 查看pod狀態

bash kubectl get pods,svc -n zookeeper -owide 在這裏插入圖片描述

4)測試驗證

```bash

登錄zookeeper pod

kubectl exec -it zookeeper-0 -n zookeeper -- zkServer.sh status kubectl exec -it zookeeper-1 -n zookeeper -- zkServer.sh status kubectl exec -it zookeeper-2 -n zookeeper -- zkServer.sh status

kubectl exec -it zookeeper-0 -n zookeeper -- bash ``` 在這裏插入圖片描述

5)Prometheus監控

Prometheus:https://prometheus.k8s.local/targets?search=zookeeper 在這裏插入圖片描述

可以通過命令查看採集數據

bash kubectl get --raw http://10.244.0.52:9141/metrics kubectl get --raw http://10.244.1.101:9141/metrics kubectl get --raw http://10.244.2.137:9141/metrics Grafana:https://grafana.k8s.local/ 賬號:admin,密碼通過下面命令獲取

bash kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo 導入grafana模板,集羣資源監控:10465 官方模塊下載地址:https://grafana.com/grafana/dashboards/ 在這裏插入圖片描述

6)卸載

```bash helm uninstall zookeeper -n zookeeper

kubectl delete pod -n zookeeper kubectl get pod -n zookeeper|awk 'NR>1{print $1}' --force kubectl patch ns zookeeper -p '{"metadata":{"finalizers":null}}' kubectl delete ns zookeeper --force ```

三、Kafka on k8s 部署

1)添加源

部署包地址:https://artifacthub.io/packages/helm/bitnami/kafka bash helm repo add bitnami https://charts.bitnami.com/bitnami helm pull bitnami/kafka tar -xf kafka-18.4.2.tgz

2)修改配置

  • 修改kafka/values.yaml

```bash image: registry: myharbor.com repository: bigdata/kafka tag: 3.2.1-debian-11-r16

...

replicaCount: 3

...

service: type: NodePort nodePorts: client: "30092" external: "30094"

...

externalAccess enabled: true service: type: NodePort nodePorts: - 30001 - 30002 - 30003 useHostIPs: true

...

persistence: storageClass: "kafka-local-storage" size: "10Gi" # 目錄需要提前在宿主機上創建 local: - name: kafka-0 host: "local-168-182-110" path: "/opt/bigdata/servers/kafka/data/data1" - name: kafka-1 host: "local-168-182-111" path: "/opt/bigdata/servers/kafka/data/data1" - name: kafka-2 host: "local-168-182-112" path: "/opt/bigdata/servers/kafka/data/data1"

...

metrics: kafka: enabled: true image: registry: myharbor.com repository: bigdata/kafka-exporter tag: 1.6.0-debian-11-r8 jmx: enabled: true image: registry: myharbor.com repository: bigdata/jmx-exporter tag: 0.17.1-debian-11-r1 annotations: prometheus.io/path: "/metrics"

...

zookeeper: enabled: false

...

externalZookeeper servers: - zookeeper-0.zookeeper-headless.zookeeper - zookeeper-1.zookeeper-headless.zookeeper - zookeeper-2.zookeeper-headless.zookeeper - 添加`kafka/templates/pv.yaml`yaml {{- range .Values.persistence.local }}


apiVersion: v1 kind: PersistentVolume metadata: name: {{ .name }} labels: name: {{ .name }} spec: storageClassName: {{ $.Values.persistence.storageClass }} capacity: storage: {{ $.Values.persistence.size }} accessModes: - ReadWriteOnce local: path: {{ .path }} nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - {{ .host }}


{{- end }} `` - 添加kafka/templates/storage-class.yaml`

bash kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: {{ .Values.persistence.storageClass }} provisioner: kubernetes.io/no-provisioner

3)開始安裝

```bash

先準備好鏡像

docker pull docker.io/bitnami/kafka:3.2.1-debian-11-r16 docker tag docker.io/bitnami/kafka:3.2.1-debian-11-r16 myharbor.com/bigdata/kafka:3.2.1-debian-11-r16 docker push myharbor.com/bigdata/kafka:3.2.1-debian-11-r16

node-export

docker pull docker.io/bitnami/kafka-exporter:1.6.0-debian-11-r8 docker tag docker.io/bitnami/kafka-exporter:1.6.0-debian-11-r8 myharbor.com/bigdata/kafka-exporter:1.6.0-debian-11-r8 docker push myharbor.com/bigdata/kafka-exporter:1.6.0-debian-11-r8

JXM

docker.io/bitnami/jmx-exporter:0.17.1-debian-11-r1 docker tag docker.io/bitnami/jmx-exporter:0.17.1-debian-11-r1 myharbor.com/bigdata/jmx-exporter:0.17.1-debian-11-r1 docker push myharbor.com/bigdata/jmx-exporter:0.17.1-debian-11-r1

開始安裝

helm install kafka ./kafka -n kafka --create-namespace ``` NOTES

```bash NAME: kafka LAST DEPLOYED: Sun Sep 18 20:57:02 2022 NAMESPACE: kafka STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: CHART NAME: kafka CHART VERSION: 18.4.2 APP VERSION: 3.2.1


WARNING

By specifying "serviceType=LoadBalancer" and not configuring the authentication
you have most likely exposed the Kafka service externally without any
authentication mechanism.

For security reasons, we strongly suggest that you switch to "ClusterIP" or
"NodePort". As alternative, you can also configure the Kafka authentication.

Please be patient while the chart is being deployed

Kafka can be accessed by consumers via port 9092 on the following DNS name from within your cluster:

kafka.kafka.svc.cluster.local

Each Kafka broker can be accessed by producers via port 9092 on the following DNS name(s) from within your cluster:

kafka-0.kafka-headless.kafka.svc.cluster.local:9092
kafka-1.kafka-headless.kafka.svc.cluster.local:9092
kafka-2.kafka-headless.kafka.svc.cluster.local:9092

To create a pod that you can use as a Kafka client run the following commands:

kubectl run kafka-client --restart='Never' --image docker.io/bitnami/kafka:3.2.1-debian-11-r16 --namespace kafka --command -- sleep infinity
kubectl exec --tty -i kafka-client --namespace kafka -- bash

PRODUCER:
    kafka-console-producer.sh \

        --broker-list kafka-0.kafka-headless.kafka.svc.cluster.local:9092,kafka-1.kafka-headless.kafka.svc.cluster.local:9092,kafka-2.kafka-headless.kafka.svc.cluster.local:9092 \
        --topic test

CONSUMER:
    kafka-console-consumer.sh \

        --bootstrap-server kafka.kafka.svc.cluster.local:9092 \
        --topic test \
        --from-beginning

To connect to your Kafka server from outside the cluster, follow the instructions below:

Kafka brokers domain: You can get the external node IP from the Kafka configuration file with the following commands (Check the EXTERNAL listener)

    1. Obtain the pod name:

    kubectl get pods --namespace kafka -l "app.kubernetes.io/name=kafka,app.kubernetes.io/instance=kafka,app.kubernetes.io/component=kafka"

    2. Obtain pod configuration:

    kubectl exec -it KAFKA_POD -- cat /opt/bitnami/kafka/config/server.properties | grep advertised.listeners

Kafka brokers port: You will have a different node port for each Kafka broker. You can get the list of configured node ports using the command below:

    echo "$(kubectl get svc --namespace kafka -l "app.kubernetes.io/name=kafka,app.kubernetes.io/instance=kafka,app.kubernetes.io/component=kafka,pod" -o jsonpath='{.items[*].spec.ports[0].nodePort}' | tr ' ' '\n')"

``` 在這裏插入圖片描述 查看pod狀態

bash kubectl get pods,svc -n kafka -owide 在這裏插入圖片描述

4)測試驗證

```bash

登錄zookeeper pod

kubectl exec -it kafka-0 -n kafka -- bash ```

1、創建Topic(一個副本一個分區)

```bash --create: 指定創建topic動作

--topic:指定新建topic的名稱

--bootstrap-server: 指定kafka連接地址

--config:指定當前topic上有效的參數值,參數列表參考文檔為: Topic-level configuration

--partitions:指定當前創建的kafka分區數量,默認為1個

--replication-factor:指定每個分區的複製因子個數,默認1個 ```

```bash kafka-topics.sh --create --topic test001 --bootstrap-server kafka.kafka:9092 --partitions 1 --replication-factor 1

查看

kafka-topics.sh --describe --bootstrap-server kafka.kafka:9092 --topic test001

``` 在這裏插入圖片描述

2、查看Topic列表

bash kafka-topics.sh --list --bootstrap-server kafka.kafka:9092

3、生產者/消費者測試

【生產者】

```bash kafka-console-producer.sh --broker-list kafka.kafka:9092 --topic test001

{"id":"1","name":"n1","age":"20"} {"id":"2","name":"n2","age":"21"} {"id":"3","name":"n3","age":"22"} ``` 【消費者】

```bash

從頭開始消費

kafka-console-consumer.sh --bootstrap-server kafka.kafka:9092 --topic test001 --from-beginning

指定從分區的某個位置開始消費,這裏只指定了一個分區,可以多寫幾行或者遍歷對應的所有分區

kafka-console-consumer.sh --bootstrap-server kafka.kafka:9092 --topic test001 --partition 0 --offset 100 --group test001 ```

4、查看數據積壓

bash kafka-consumer-groups.sh --bootstrap-server kafka.kafka:9092 --describe --group test001

5、刪除topic

bash kafka-topics.sh --delete --topic test001 --bootstrap-server kafka.kafka:9092

5)Prometheus監控

Prometheus:https://prometheus.k8s.local/targets?search=kafka 在這裏插入圖片描述

可以通過命令查看採集數據

bash kubectl get --raw http://10.244.2.165:9308/metrics Grafana:https://grafana.k8s.local/ 賬號:admin,密碼通過下面命令獲取

bash kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo 導入grafana模板,集羣資源監控:11962 官方模塊下載地址:https://grafana.com/grafana/dashboards/

6)卸載

```bash helm uninstall kafka -n kafka

kubectl delete pod -n kafka kubectl get pod -n kafka|awk 'NR>1{print $1}' --force kubectl patch ns kafka -p '{"metadata":{"finalizers":null}}' kubectl delete ns kafka --force ``` zookeeper + kafka on k8s 環境部署 就先到這裏了,小夥伴有任何疑問,歡迎給我留言,後續會持續分享【雲原生和大數據】相關的問題~