【雲原生】zookeeper + kafka on k8s 環境部署
一、概述
Apache ZooKeeper是一個集中式服務，用於維護配置資訊、命名、提供分散式同步和提供組服務，ZooKeeper 致力於開發和維護一個開源伺服器，以實現高度可靠的分散式協調，其實也可以認為就是一個分散式資料庫，只是結構比較特殊，是樹狀結構。官網文件：http://zookeeper.apache.org/doc/r3.8.0/ ，關於Zookeeper的介紹，也可以參考我之前的文章：分散式開源協調服務——Zookeeper
-
Kafka是最初由Linkedin公司開發，是一個分散式、支援分割槽的（partition）、多副本的（replica），基於zookeeper協調的分散式訊息系統。官方文件：http://kafka.apache.org/documentation/關於Kafka的介紹，也可以參考我之前的文章：Kafka原理介紹+安裝+基本操作
二、Zookeeper on k8s 部署
1）新增源
部署包地址：http://artifacthub.io/packages/helm/zookeeper/zookeeper
bash
helm repo add bitnami http://charts.bitnami.com/bitnami
helm pull bitnami/zookeeper
tar -xf zookeeper-10.2.1.tgz
2）修改配置
- 修改
zookeeper/values.yaml
```bash image: registry: myharbor.com repository: bigdata/zookeeper tag: 3.8.0-debian-11-r36 ...
replicaCount: 3
...
service: type: NodePort nodePorts: #NodePort 預設範圍是 30000-32767 client: "32181" tls: "32182"
...
persistence: storageClass: "zookeeper-local-storage" size: "10Gi" # 目錄需要提前在宿主機上建立 local: - name: zookeeper-0 host: "local-168-182-110" path: "/opt/bigdata/servers/zookeeper/data/data1" - name: zookeeper-1 host: "local-168-182-111" path: "/opt/bigdata/servers/zookeeper/data/data1" - name: zookeeper-2 host: "local-168-182-112" path: "/opt/bigdata/servers/zookeeper/data/data1"
...
Enable Prometheus to access ZooKeeper metrics endpoint
metrics:
enabled: true
- 新增`zookeeper/templates/pv.yaml`yaml
{{- range .Values.persistence.local }}
apiVersion: v1 kind: PersistentVolume metadata: name: {{ .name }} labels: name: {{ .name }} spec: storageClassName: {{ $.Values.persistence.storageClass }} capacity: storage: {{ $.Values.persistence.size }} accessModes: - ReadWriteOnce local: path: {{ .path }} nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - {{ .host }}
{{- end }}
``
- 新增zookeeper/templates/storage-class.yaml`
bash
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: {{ .Values.persistence.storageClass }}
provisioner: kubernetes.io/no-provisioner
3）開始安裝
```bash
先準備好映象
docker pull docker.io/bitnami/zookeeper:3.8.0-debian-11-r36 docker tag docker.io/bitnami/zookeeper:3.8.0-debian-11-r36 myharbor.com/bigdata/zookeeper:3.8.0-debian-11-r36 docker push myharbor.com/bigdata/zookeeper:3.8.0-debian-11-r36
開始安裝
helm install zookeeper ./zookeeper -n zookeeper --create-namespace ``` NOTES
```bash NAME: zookeeper LAST DEPLOYED: Sun Sep 18 18:24:03 2022 NAMESPACE: zookeeper STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: CHART NAME: zookeeper CHART VERSION: 10.2.1 APP VERSION: 3.8.0
Please be patient while the chart is being deployed
ZooKeeper can be accessed via port 2181 on the following DNS name from within your cluster:
zookeeper.zookeeper.svc.cluster.local
To connect to your ZooKeeper server run the following commands:
export POD_NAME=$(kubectl get pods --namespace zookeeper -l "app.kubernetes.io/name=zookeeper,app.kubernetes.io/instance=zookeeper,app.kubernetes.io/component=zookeeper" -o jsonpath="{.items[0].metadata.name}")
kubectl exec -it $POD_NAME -- zkCli.sh
To connect to your ZooKeeper server from outside the cluster execute the following commands:
export NODE_IP=$(kubectl get nodes --namespace zookeeper -o jsonpath="{.items[0].status.addresses[0].address}")
export NODE_PORT=$(kubectl get --namespace zookeeper -o jsonpath="{.spec.ports[0].nodePort}" services zookeeper)
zkCli.sh $NODE_IP:$NODE_PORT
```
檢視pod狀態
bash
kubectl get pods,svc -n zookeeper -owide
4）測試驗證
```bash
登入zookeeper pod
kubectl exec -it zookeeper-0 -n zookeeper -- zkServer.sh status kubectl exec -it zookeeper-1 -n zookeeper -- zkServer.sh status kubectl exec -it zookeeper-2 -n zookeeper -- zkServer.sh status
kubectl exec -it zookeeper-0 -n zookeeper -- bash
```
5）Prometheus監控
Prometheus：http://prometheus.k8s.local/targets?search=zookeeper
可以通過命令檢視採集資料
bash
kubectl get --raw http://10.244.0.52:9141/metrics
kubectl get --raw http://10.244.1.101:9141/metrics
kubectl get --raw http://10.244.2.137:9141/metrics
Grafana：http://grafana.k8s.local/
賬號：admin，密碼通過下面命令獲取
bash
kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
匯入grafana模板，叢集資源監控：10465
官方模組下載地址：http://grafana.com/grafana/dashboards/
6）解除安裝
```bash helm uninstall zookeeper -n zookeeper
kubectl delete pod -n zookeeper kubectl get pod -n zookeeper|awk 'NR>1{print $1}' --force
kubectl patch ns zookeeper -p '{"metadata":{"finalizers":null}}'
kubectl delete ns zookeeper --force
```
三、Kafka on k8s 部署
1）新增源
部署包地址：http://artifacthub.io/packages/helm/bitnami/kafka
bash
helm repo add bitnami http://charts.bitnami.com/bitnami
helm pull bitnami/kafka
tar -xf kafka-18.4.2.tgz
2）修改配置
- 修改
kafka/values.yaml
```bash image: registry: myharbor.com repository: bigdata/kafka tag: 3.2.1-debian-11-r16
...
replicaCount: 3
...
service: type: NodePort nodePorts: client: "30092" external: "30094"
...
externalAccess enabled: true service: type: NodePort nodePorts: - 30001 - 30002 - 30003 useHostIPs: true
...
persistence: storageClass: "kafka-local-storage" size: "10Gi" # 目錄需要提前在宿主機上建立 local: - name: kafka-0 host: "local-168-182-110" path: "/opt/bigdata/servers/kafka/data/data1" - name: kafka-1 host: "local-168-182-111" path: "/opt/bigdata/servers/kafka/data/data1" - name: kafka-2 host: "local-168-182-112" path: "/opt/bigdata/servers/kafka/data/data1"
...
metrics: kafka: enabled: true image: registry: myharbor.com repository: bigdata/kafka-exporter tag: 1.6.0-debian-11-r8 jmx: enabled: true image: registry: myharbor.com repository: bigdata/jmx-exporter tag: 0.17.1-debian-11-r1 annotations: prometheus.io/path: "/metrics"
...
zookeeper: enabled: false
...
externalZookeeper
servers:
- zookeeper-0.zookeeper-headless.zookeeper
- zookeeper-1.zookeeper-headless.zookeeper
- zookeeper-2.zookeeper-headless.zookeeper
- 新增`kafka/templates/pv.yaml`yaml
{{- range .Values.persistence.local }}
apiVersion: v1 kind: PersistentVolume metadata: name: {{ .name }} labels: name: {{ .name }} spec: storageClassName: {{ $.Values.persistence.storageClass }} capacity: storage: {{ $.Values.persistence.size }} accessModes: - ReadWriteOnce local: path: {{ .path }} nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - {{ .host }}
{{- end }}
``
- 新增kafka/templates/storage-class.yaml`
bash
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: {{ .Values.persistence.storageClass }}
provisioner: kubernetes.io/no-provisioner
3）開始安裝
```bash
先準備好映象
docker pull docker.io/bitnami/kafka:3.2.1-debian-11-r16 docker tag docker.io/bitnami/kafka:3.2.1-debian-11-r16 myharbor.com/bigdata/kafka:3.2.1-debian-11-r16 docker push myharbor.com/bigdata/kafka:3.2.1-debian-11-r16
node-export
docker pull docker.io/bitnami/kafka-exporter:1.6.0-debian-11-r8 docker tag docker.io/bitnami/kafka-exporter:1.6.0-debian-11-r8 myharbor.com/bigdata/kafka-exporter:1.6.0-debian-11-r8 docker push myharbor.com/bigdata/kafka-exporter:1.6.0-debian-11-r8
JXM
docker.io/bitnami/jmx-exporter:0.17.1-debian-11-r1 docker tag docker.io/bitnami/jmx-exporter:0.17.1-debian-11-r1 myharbor.com/bigdata/jmx-exporter:0.17.1-debian-11-r1 docker push myharbor.com/bigdata/jmx-exporter:0.17.1-debian-11-r1
開始安裝
helm install kafka ./kafka -n kafka --create-namespace ``` NOTES
```bash NAME: kafka LAST DEPLOYED: Sun Sep 18 20:57:02 2022 NAMESPACE: kafka STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: CHART NAME: kafka CHART VERSION: 18.4.2 APP VERSION: 3.2.1
WARNING
By specifying "serviceType=LoadBalancer" and not configuring the authentication
you have most likely exposed the Kafka service externally without any
authentication mechanism.
For security reasons, we strongly suggest that you switch to "ClusterIP" or
"NodePort". As alternative, you can also configure the Kafka authentication.
Please be patient while the chart is being deployed
Kafka can be accessed by consumers via port 9092 on the following DNS name from within your cluster:
kafka.kafka.svc.cluster.local
Each Kafka broker can be accessed by producers via port 9092 on the following DNS name(s) from within your cluster:
kafka-0.kafka-headless.kafka.svc.cluster.local:9092
kafka-1.kafka-headless.kafka.svc.cluster.local:9092
kafka-2.kafka-headless.kafka.svc.cluster.local:9092
To create a pod that you can use as a Kafka client run the following commands:
kubectl run kafka-client --restart='Never' --image docker.io/bitnami/kafka:3.2.1-debian-11-r16 --namespace kafka --command -- sleep infinity
kubectl exec --tty -i kafka-client --namespace kafka -- bash
PRODUCER:
kafka-console-producer.sh \
--broker-list kafka-0.kafka-headless.kafka.svc.cluster.local:9092,kafka-1.kafka-headless.kafka.svc.cluster.local:9092,kafka-2.kafka-headless.kafka.svc.cluster.local:9092 \
--topic test
CONSUMER:
kafka-console-consumer.sh \
--bootstrap-server kafka.kafka.svc.cluster.local:9092 \
--topic test \
--from-beginning
To connect to your Kafka server from outside the cluster, follow the instructions below:
Kafka brokers domain: You can get the external node IP from the Kafka configuration file with the following commands (Check the EXTERNAL listener)
1. Obtain the pod name:
kubectl get pods --namespace kafka -l "app.kubernetes.io/name=kafka,app.kubernetes.io/instance=kafka,app.kubernetes.io/component=kafka"
2. Obtain pod configuration:
kubectl exec -it KAFKA_POD -- cat /opt/bitnami/kafka/config/server.properties | grep advertised.listeners
Kafka brokers port: You will have a different node port for each Kafka broker. You can get the list of configured node ports using the command below:
echo "$(kubectl get svc --namespace kafka -l "app.kubernetes.io/name=kafka,app.kubernetes.io/instance=kafka,app.kubernetes.io/component=kafka,pod" -o jsonpath='{.items[*].spec.ports[0].nodePort}' | tr ' ' '\n')"
```
檢視pod狀態
bash
kubectl get pods,svc -n kafka -owide
4）測試驗證
```bash
登入zookeeper pod
kubectl exec -it kafka-0 -n kafka -- bash ```
1、建立Topic（一個副本一個分割槽）
```bash --create： 指定建立topic動作
--topic：指定新建topic的名稱
--bootstrap-server： 指定kafka連線地址
--config：指定當前topic上有效的引數值，引數列表參考文件為: Topic-level configuration
--partitions：指定當前建立的kafka分割槽數量，預設為1個
--replication-factor：指定每個分割槽的複製因子個數，預設1個 ```
```bash kafka-topics.sh --create --topic test001 --bootstrap-server kafka.kafka:9092 --partitions 1 --replication-factor 1
檢視
kafka-topics.sh --describe --bootstrap-server kafka.kafka:9092 --topic test001
```
2、檢視Topic列表
bash
kafka-topics.sh --list --bootstrap-server kafka.kafka:9092
3、生產者/消費者測試
【生產者】
```bash kafka-console-producer.sh --broker-list kafka.kafka:9092 --topic test001
{"id":"1","name":"n1","age":"20"} {"id":"2","name":"n2","age":"21"} {"id":"3","name":"n3","age":"22"} ``` 【消費者】
```bash
從頭開始消費
kafka-console-consumer.sh --bootstrap-server kafka.kafka:9092 --topic test001 --from-beginning
指定從分割槽的某個位置開始消費，這裡只指定了一個分割槽，可以多寫幾行或者遍歷對應的所有分割槽
kafka-console-consumer.sh --bootstrap-server kafka.kafka:9092 --topic test001 --partition 0 --offset 100 --group test001 ```
4、檢視資料積壓
bash
kafka-consumer-groups.sh --bootstrap-server kafka.kafka:9092 --describe --group test001
5、刪除topic
bash
kafka-topics.sh --delete --topic test001 --bootstrap-server kafka.kafka:9092
5）Prometheus監控
Prometheus：http://prometheus.k8s.local/targets?search=kafka
可以通過命令檢視採集資料
bash
kubectl get --raw http://10.244.2.165:9308/metrics
Grafana：http://grafana.k8s.local/
賬號：admin，密碼通過下面命令獲取
bash
kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
匯入grafana模板，叢集資源監控：11962
官方模組下載地址：http://grafana.com/grafana/dashboards/
6）解除安裝
```bash helm uninstall kafka -n kafka
kubectl delete pod -n kafka kubectl get pod -n kafka|awk 'NR>1{print $1}' --force
kubectl patch ns kafka -p '{"metadata":{"finalizers":null}}'
kubectl delete ns kafka --force
```
zookeeper + kafka on k8s 環境部署 就先到這裡了，小夥伴有任何疑問，歡迎給我留言，後續會持續分享【雲原生和大資料】相關的問題~
- Python Flask 介紹
- Python 基礎語法介紹（二）
- Python 高階程式設計之IO模型與協程（四）
- 分散式儲存系統 Ceph 介紹與環境部署
- 【雲原生】Grafana Alerting 告警模組介紹與實戰操作
- 【雲原生】Minio on k8s 講解與實戰操作
- 高效能分散式物件儲存——MinIO（環境部署）
- 【雲原生】Flink on k8s 講解與實戰操作
- 【雲原生】Presto/Trino on k8s 環境部署
- 【雲原生】Hadoop on k8s 環境部署
- 【雲原生】Hadoop HA on k8s 環境部署
- 【雲原生】Azkaban on k8s 講解與實戰操作
- 【雲原生】zookeeper kafka on k8s 環境部署
- 【雲原生】MySQL on k8s 環境部署
- 【雲原生】Hive on k8s 環境部署
- 企業級日誌系統架構——ELK（Elasticsearch、Filebeat、Kafka、Logstash、Kibana）
- Kubernetes（k8s）許可權管理RBAC詳解
- Kubernetes（k8s）kube-proxy、Service詳解
- Linux防火牆——Firewalld原理與實戰操作
- Kubernetes（k8s）基礎概念介紹