k8s series 24: calico初級(監控)

語言: CN / TW / HK

theme: channing-cyan highlight: an-old-hope


「這是我參與11月更文挑戰的第12天,活動詳情檢視:2021最後一次更文挑戰」。

前言

calico官方提供了prometheus的監控,且還詳細說明了相關指標

因為我們是預設安裝的calico,並沒有啟用Typha元件,所以kubernetes叢集中就只有Felix和kube-controlles兩大元件在執行

Felix的詳細指標: https://docs.projectcalico.org/reference/felix/prometheus

kube-controlles的詳細指標:https://docs.projectcalico.org/reference/kube-controllers/prometheus

calico元件配置

雖然官方提供了相關prometheus的監控,但預設配置是禁用的,需要手動開啟,且還需要提供端點供prometheus拉取監控資料

Felix配置

啟用Felix的prometheus指標 js calicoctl patch felixConfiguration default --patch '{"spec":{"prometheusMetricsEnabled": true}}'

建立Felix指標端點

js kubectl apply -f - <<EOF apiVersion: v1 kind: Service metadata: name: felix-metrics-svc namespace: kube-system spec: selector: k8s-app: calico-node ports: - port: 9091 targetPort: 9091 EOF

kube-controlles配置

kube-controlles的prometheus指標預設是啟用的,需要無須改動,如果想更改它的監控埠,可以使用如下命令,如果埠改為0,則為禁用

```js

預設埠監控在9094

calicoctl patch kubecontrollersconfiguration default --patch '{"spec":{"prometheusMetricsPort": 9094}}' ``` 建立kube-controlles指標端點

js kubectl apply -f - <<EOF apiVersion: v1 kind: Service metadata: name: kube-controllers-metrics-svc namespace: kube-system spec: selector: k8s-app: calico-kube-controllers ports: - port: 9094 targetPort: 9094 EOF

兩個元件的Service建立成功後,檢視一下

image.png

prometheus安裝配置

在安裝prometheus之前需要提前 建立相關服務賬號和許可權

建立namespace

建立一個獨立的命令空間,供監控使用

js kubectl apply -f -<<EOF apiVersion: v1 kind: Namespace metadata: name: calico-monitoring labels: app: ns-calico-monitoring role: monitoring EOF

建立服務賬號

建立一個具從calico採集資料的賬號,然後授於相關許可權

下面配置分為三部分,建立角色,建立賬號,繫結角色賬號 ```js kubectl apply -f - <<EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: calico-prometheus-user rules: - apiGroups: [""] resources: - endpoints - services - pods verbs: ["get", "list", "watch"] - nonResourceURLs: ["/metrics"] verbs: ["get"]


apiVersion: v1 kind: ServiceAccount metadata: name: calico-prometheus-user namespace: calico-monitoring


apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: calico-prometheus-user roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: calico-prometheus-user subjects: - kind: ServiceAccount name: calico-prometheus-user namespace: calico-monitoring EOF ```

prometheus配置檔案

建立prometheus的配置檔案,如果二制進安裝過prometheus,應該發現下列的配置幾乎是一樣的,後期想修改相關配置,直接編輯既可 js kubectl apply -f - <<EOF apiVersion: v1 kind: ConfigMap metadata: name: prometheus-config namespace: calico-monitoring data: prometheus.yml: |- global: scrape_interval: 15s external_labels: monitor: 'tutorial-monitor' scrape_configs: - job_name: 'prometheus' scrape_interval: 5s static_configs: - targets: ['localhost:9090'] - job_name: 'felix_metrics' scrape_interval: 5s scheme: http kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_service_name] regex: felix-metrics-svc replacement: $1 action: keep - job_name: 'typha_metrics' scrape_interval: 5s scheme: http kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_service_name] regex: typha-metrics-svc replacement: $1 action: keep - job_name: 'kube_controllers_metrics' scrape_interval: 5s scheme: http kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_service_name] regex: kube-controllers-metrics-svc replacement: $1 action: keep EOF

安裝prometheus

以上步驟成功後,執行下列安裝步驟 js kubectl apply -f - <<EOF apiVersion: v1 kind: Pod metadata: name: prometheus-pod namespace: calico-monitoring labels: app: prometheus-pod role: monitoring spec: serviceAccountName: calico-prometheus-user containers: - name: prometheus-pod image: prom/prometheus resources: limits: memory: "128Mi" cpu: "500m" volumeMounts: - name: config-volume mountPath: /etc/prometheus/prometheus.yml subPath: prometheus.yml ports: - containerPort: 9090 volumes: - name: config-volume configMap: name: prometheus-config EOF

檢視安裝進度,如果返回的狀態是Running說明安裝完成

js kubectl get pods prometheus-pod -n calico-monitoring

訪問prometheus

因為我們沒有給promethesu建立Service,所以這裡先使用埠轉發,簡單驗證一下prometheus是否獲取到了calico的資料

js kubectl port-forward --address 0.0.0.0 pod/prometheus-pod 9090:9090 -n calico-monitoring

訪問 http://ip:9090 埠

Grafana安裝配置

在配置Grafana之前,需要宣告prometheus訪問方式,便於訪問資料顯示圖表

建立prometheus Service

js kubectl apply -f - <<EOF apiVersion: v1 kind: Service metadata: name: prometheus-dashboard-svc namespace: calico-monitoring spec: selector: app: prometheus-pod role: monitoring ports: - port: 9090 targetPort: 9090 EOF

建立grafana配置

建立grafana連線資料庫的型別,地址,埠,以及連線方式 js kubectl apply -f - <<EOF apiVersion: v1 kind: ConfigMap metadata: name: grafana-config namespace: calico-monitoring data: prometheus.yaml: |- { "apiVersion": 1, "datasources": [ { "access":"proxy", "editable": true, "name": "calico-demo-prometheus", "orgId": 1, "type": "prometheus", "url": "http://prometheus-dashboard-svc.calico-monitoring.svc:9090", "version": 1 } ] } EOF

Felix的儀表盤配置

js kubectl apply -f https://docs.projectcalico.org/manifests/grafana-dashboards.yaml

安裝Grafana

直接應用如下配置,會從grafana官方下載最新的映象 js kubectl apply -f - <<EOF apiVersion: v1 kind: Pod metadata: name: grafana-pod namespace: calico-monitoring labels: app: grafana-pod role: monitoring spec: containers: - name: grafana-pod image: grafana/grafana:latest resources: limits: memory: "128Mi" cpu: "500m" volumeMounts: - name: grafana-config-volume mountPath: /etc/grafana/provisioning/datasources - name: grafana-dashboards-volume mountPath: /etc/grafana/provisioning/dashboards - name: grafana-storage-volume mountPath: /var/lib/grafana ports: - containerPort: 3000 volumes: - name: grafana-storage-volume emptyDir: {} - name: grafana-config-volume configMap: name: grafana-config - name: grafana-dashboards-volume configMap: name: grafana-dashboards-config EOF

訪問grafana

因暫時沒有寫Service配置,先轉發埠來訪問,驗證一下監控是否正常 js kubectl port-forward --address 0.0.0.0 pod/grafana-pod 3000:3000 -n calico-monitoring 訪問http://IP:3000 訪問Grafana的web-ui登陸頁,預設賬號密碼都是: admin/admin

登陸成功後,會提示修改密碼或跳過,後續在設定中修改

image.png

登陸好之看,是沒有任何東西的,需要訪問一下這個地址: http://ip:3000/d/calico-felix-dashboard/felix-dashboard-calico?orgId=1

會開啟calico給我們提供的Dashborad,這裡點一下加星,後面就可以在主頁上找到該面版了

image.png

建立Service

直接使用expose命令建立一個NodePort型別的Service ```js

建立Service

kubectl expose pod grafana-pod --port=3000 --target-port=3000 --type=NodePort -n calico-monitoring

檢視暴露的埠

kubectl get svc -n calico-monitoring ``` 訪問叢集節點ip+30538埠 就可以開啟grafana了 image.png

解除安裝

如果覺得該套監控比較佔用叢集資源,如果單純的只是想看看效果,可執行下列命令來刪除這套監控

```js kubectl delete service felix-metrics-svc -n kube-system kubectl delete service typha-metrics-svc -n kube-system kubectl delete service kube-controllers-metrics-svc -n kube-system kubectl delete namespace calico-monitoring kubectl delete ClusterRole calico-prometheus-user kubectl delete clusterrolebinding calico-prometheus-user

kubectl delete namespace calico-monitoring ```