K8s搭建redis集群

上文讲了使用裸机搭建redis集群，本文讲如何使用K8s搭建redis集群

环境

跟上文一样，redis至少需要3个主节点，所以我们使用三台机器各有三个主节点pod和三个从节点pod，主节点pod端口为6379，从节点pod端口号为6380。先创建三个主节点

创建命名空间kubectl create namespace redis

redis-config.yaml

redis启动配置文件，使用CofigMap来管理比较方便

apiVersion: v1
kind: ConfigMap
metadata:
 name: redis-config
 namespace: redis
data:
 update-node.sh: |
  #!/bin/sh
  REDIS_NODES="/data/nodes.conf"
  sed -i -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${MY_POD_IP}/" ${REDIS_NODES}
  exec "$@"
 redis.conf: |+
  port 6379
  protected-mode no
  cluster-enabled yes
  cluster-config-file nodes.conf
  cluster-node-timeout 15000
  #cluster-announce-ip ${MY_POD_IP}
  #cluster-announce-port 6379
  #cluster-announce-bus-port 17001                         
  logfile "/data/redis.log"

说明：2个文件，node-update.sh用来redis启动时执行脚本，具体后面再介绍为何要增加一个启动脚本；redis.conf为redis启动配置文件。

redis-pv.yaml

redis持久化存储pv，pv使用nfs服务存储，集群6个节点分别创建6个pv不同存储

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv0
  labels:
    pv: nfs-pv0
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 192.168.0.10
    path: /home/nfs/redis-cluster0

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv1
  labels:
    pv: nfs-pv1
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 192.168.0.10
    path: /home/nfs/redis-cluster1

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv2
  labels:
    pv: nfs-pv2
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 192.168.0.20
    path: /home/nfs/redis-cluster2

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv3
  labels:
    pv: nfs-pv3
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 192.168.0.20
    path: /home/nfs/redis-cluster3

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv4
  labels:
    pv: nfs-pv4
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 192.168.0.30
    path: /home/nfs/redis-cluster4

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv5
  labels:
    pv: nfs-pv5
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 192.168.0.30
    path: /home/nfs/redis-cluster5

这里我将每两个pv放在一台主机上

redis-cluster.yaml

使用StatefulSet创建redis-cluster集群节点和headless service

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: redis-cluster
  name: redis-cluster
  namespace: redis
spec:
  replicas: 3
  selector:
    matchLabels:
      app: redis-cluster
  serviceName: redis-cluster
  template:
    metadata:
      labels:
        app: redis-cluster
    spec:
      tolerations:
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"
      containers:
        - command: 
            ["/bin/bash", "/usr/local/etc/redis/update-node.sh", "/usr/local/bin/redis-server", "/usr/local/etc/redis/redis.conf"]
          #args:
          #  - /usr/local/etc/redis/redis.conf
          #  - --cluster-announce-ip
          #  - "$(MY_POD_IP)"
          env:
            - name: MY_POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: TZ
              value: Asia/Shanghai
          image: 'redis:5.0.12'
          imagePullPolicy: IfNotPresent
          name: redis
          ports:
            - containerPort: 6379
              hostPort: 6379
              name: redis-port
              protocol: TCP
          volumeMounts:
            - mountPath: /data
              name: redis-cluster-data
              subPath: data
              readOnly: false
            - mountPath: /usr/local/etc/redis
              name: redis-config
              readOnly: false
      dnsPolicy: ClusterFirst
      volumes:
        - name: redis-config
          configMap:
           name: redis-config
  volumeClaimTemplates:  
  - metadata:
      name: redis-cluster-data
      namespace: redis
    spec:
      accessModes: [ "ReadWriteMany" ]
      resources:
        requests:
          storage: 1Gi

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: redis-cluster
  name: redis-cluster
  namespace: redis
spec:
  ports:
    - name: redis-port
      port: 6379
      protocol: TCP
      targetPort: 6379
  selector:
    app: redis-cluster
  type: ClusterIP
  clusterIP: None

说明2个点：第一，update-node.sh脚本需要加上bash环境变量，否则脚本启动执行时提示没有权限，这也是网上很多文章并没有提到的；
第二，env环境变量中加了2个，${MY_POD_IP}是节点重启时更新pod的ip地址，TZ就是时区更改；

下面进行从节点安装

redis-slaves-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
 name: redis-slaves-config
 namespace: redis
data:
 update-node.sh: |
  #!/bin/sh
  REDIS_NODES="/data/nodes.conf"
  sed -i -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${MY_POD_IP}/" ${REDIS_NODES}
  exec "$@"
 redis.conf: |+
  port 6380
  protected-mode no
  cluster-enabled yes
  cluster-config-file nodes.conf
  cluster-node-timeout 15000
  #cluster-announce-ip ${MY_POD_IP}
  #cluster-announce-port 6380
  #cluster-announce-bus-port 17001                         
  logfile "/data/redis.log"

redis-slaves-cluster.yaml

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: redis-slaves-cluster
  name: redis-slaves-cluster
  namespace: redis
spec:
  replicas: 3
  selector:
    matchLabels:
      app: redis-slaves-cluster
  serviceName: redis-slaves-cluster
  template:
    metadata:
      labels:
        app: redis-slaves-cluster
    spec:
      tolerations:
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"
      containers:
        - command: 
            ["/bin/bash", "/usr/local/etc/redis/update-node.sh", "redis-server", "/usr/local/etc/redis/redis.conf"]
          #args:
          #  - /usr/local/etc/redis/redis.conf
          #  - --cluster-announce-ip
          #  - "$(MY_POD_IP)"
          env:
            - name: MY_POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: TZ
              value: Asia/Shanghai
          image: 'redis:5.0.12'
          imagePullPolicy: IfNotPresent
          name: redis
          ports:
            - containerPort: 6380
              hostPort: 6380
              name: redis-port
              protocol: TCP
          volumeMounts:
            - mountPath: /data
              name: redis-cluster-data
              subPath: data
              readOnly: false
            - mountPath: /usr/local/etc/redis
              name: redis-config
              readOnly: false
      dnsPolicy: ClusterFirst
      volumes:
        - name: redis-config
          configMap:
           name: redis-slaves-config
  volumeClaimTemplates:  
  - metadata:
      name: redis-cluster-data
      namespace: redis
    spec:
      accessModes: [ "ReadWriteMany" ]
      resources:
        requests:
          storage: 1Gi

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: redis-slaves-cluster
  name: redis-slaves-cluster
  namespace: redis
spec:
  ports:
    - name: redis-port
      port: 6380
      protocol: TCP
      targetPort: 6380
  selector:
    app: redis-slaves-cluster
  type: ClusterIP
  clusterIP: None

初始化集群

kubectl exec -it redis-cluster-0 -n redis -- redis-cli --cluster-replicas 1 --cluster create $(kubectl get pods -l app=redis-cluster -n redis -o jsonpath='{range.items[*]}{.status.podIP}:6379 ' | awk -F " " '{NF=3}1' ) $(kubectl get pods -l app=redis-slaves-cluster -n redis -o jsonpath='{range.items[*]}{.status.podIP}:6380 ' | awk -F " " '{NF=3}1' )

验证集群

kubectl exec -it redis-cluster-0 -n redis-- redis-cli cluster info

遇到的问题

集群初始化时，一直等待 Waiting for the cluster to join

最开始部署时，从使用docker-comopose部署的方法套用过来，由于redis.conf配置文件中参数cluster-announce-ip配置了宿主机的ip，当初始化时集群节点之间需要通讯，但是再k8s中宿主机ip已经不适用，导致无法通讯一致卡住在那个位置。
禁用#cluster-announce-ip

集群创建时，[ERR] Node 10.244.7.224:7001 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

由于前面初始化集群时卡住，导致部分节点的nodes.conf文件中更新了新节点数据，需要删除数据。可以删除文件，也可以通过如下命令
redis-cli -p 6389 -c 登录每个redis节点重置:

flushdb
cluster reset

pvc如何和pv一对一绑定？

说明：分静态和动态绑定
首先静态，之前一直没想明白，写好了pv的yaml文件，service里面定义好pvc之后，如何让pv和pvc进行绑定？只要我们定义的pv存储空间大小和pvc申请大小一致就会匹配绑定bound,当然也可以通过lebel标签来关联pvc到特定的pv。
另外，redis集群多个节点，持久化存储目录肯定不能一样，如果yaml中直接指定PVC的名字，只能指定一个名字，这样是办不到关联到多个不同的PV，于是需要volumeClaimTemplates（PVC模板）。

  volumeClaimTemplates:
  - metadata:
      name: redis-cluster-data
      namespace: default
    spec:
      accessModes: [ "ReadWriteMany" ]
      resources:
        requests:
          storage: 1Gi

动态，因为NFS不支持动态存储，所以我们需要借用这个存储插件StorageClass,关联pvc,动态创建pv。分3个步骤，1，定义storage;2，部署授权，因为storage自动创建pv需要经过kube-apiserver,所以要进行授权；3，部署自动创建pv的服务nfs-client-provisioner；4，部署redis服务。

参考链接地址：https://blog.csdn.net/qq_25611295/article/details/86065053

pv卷和pod挂载不了

在创建完pod以后，使用kubectl describe pod -n redis 容器名查看到先是显示mounted confused，网上搜到答案是nfs没有开启

yum -y install nfs-utils
systemctl start nfs-utils
systemctl enable nfs-utils

后面又出现/home/nfs/redis-cluster-0:such file or directory not exists，将每个节点都新建了相关文件夹

mkdir -p /home/nfs/redis-cluster-0
mkdir -p /home/nfs/redis-cluster-1
mkdir -p /home/nfs/redis-cluster-2
mkdir -p /home/nfs/redis-cluster-3
mkdir -p /home/nfs/redis-cluster-4
mkdir -p /home/nfs/redis-cluster-5

重新部署后pod挂载到pv上了

这里还有一点，pv如果在挂载时删除，是一直在Terminating中的，先查看已经lost的pvc

kubectl get pvc -n redis

将不用的删除，在编辑需要删除的pv

kubectl edit pv nfs-pv2 -n redis

这是pv就可以顺利被删掉

https://segmentfault.com/a/1190000039196137
https://www.cnblogs.com/wangxu01/articles/11684063.html
https://blog.csdn.net/wangmiaoyan/article/details/106135710
https://juejin.cn/post/6844903806719754254

2021 年 6 月
一	二	三	四	五	六	日
	1	2	3	4	5	6
7	8	9	10	11	12	13
14	15	16	17	18	19	20
21	22	23	24	25	26	27
28	29	30

环境