kubernetes集群时,针对存储这块用的都是nfs,但是前期规划的时候把nfs盘的数据和系统盘放一起了,当时太着急忘记这快了,现在因为数据越来越多了,系统盘无法进行扩容,只能重新购买一个盘,把nfs的数据迁移到新盘中,这里最基本的要求就是不能有数据丢失.
环境介绍
我现在kubernetes集群已经安装好了,并且在集群里也配置好了nfs-storage,为了后期验证数据的完整性,我需要先创建一个数据库,然后在数据库中创建一些数据,然后先迁移一下数据,然后在验证数据是否完整.
[root@node-01 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE mysql-6f6c97f4d5-6q4w5 1/1 Running 0 6s nfs-client-provisioner-79fcb887b9-hvdrl 1/1 Running 0 8m56s [root@node-01 ~]# kubectl exec -it mysql-6f6c97f4d5-6q4w5 /bin/bash kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. bash-4.2# mysql -uroot -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.7.40 MySQL Community Server (GPL) Copyright (c) 2000, 2022, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> create database wulaoer; Query OK, 1 row affected (0.00 sec) mysql> use wulaoer; Database changed mysql> CREATE TABLE users ( -> id INT AUTO_INCREMENT PRIMARY KEY, -> name VARCHAR(100), -> age INT, -> email VARCHAR(100) -> ); Query OK, 0 rows affected (0.03 sec) mysql> INSERT INTO users (name, age, email) -> VALUES ('Alice', 25, 'alice@example.com'), -> ('Bob', 30, 'bob@example.com'), -> ('Charlie', 28, 'charlie@example.com'); Query OK, 3 rows affected (0.01 sec) Records: 3 Duplicates: 0 Warnings: 0 mysql> select * from users; +----+---------+------+---------------------+ | id | name | age | email | +----+---------+------+---------------------+ | 1 | Alice | 25 | alice@example.com | | 2 | Bob | 30 | bob@example.com | | 3 | Charlie | 28 | charlie@example.com | +----+---------+------+---------------------+ 3 rows in set (0.00 sec)
迁移数据
我的数据库已经启动了,并且在数据库中插入了一些数据,我只需要把nfs进行迁移即可,因为在迁移过程中肯定不能写入数据,所以,我需要先停止nfs,然后把nfs中的数据copy到新磁盘中,然后在重启nfs服务即可.
[root@node-01 ~]# systemctl stop nfs-server [root@node-01 ~]# cd /nfs/ [root@node-01 nfs]# ls data [root@node-01 nfs]# mv data data.back [root@node-01 nfs]# mkdir data [root@node-01 nfs]# mount /dev/vdb1 /nfs/data [root@node-01 nfs]# chown -R 777 /nfs/data [root@node-01 nfs]# mv data.back/* /nfs/data/
到此数据已经迁移过来了,我们需要做的是先把nfs服务启动后,然后查看一下mysql中的数据.
[root@node-01 nfs]# systemctl start nfs-server [root@node-01 nfs]# kubectl get pod NAME READY STATUS RESTARTS AGE mysql-6f6c97f4d5-6q4w5 1/1 Running 0 6m48s nfs-client-provisioner-79fcb887b9-hvdrl 1/1 Running 0 15m [root@node-01 nfs]# kubectl exec -it mysql-6f6c97f4d5-6q4w5 /bin/bash kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. bash-4.2# mysql -uroot -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 3 Server version: 5.7.40 MySQL Community Server (GPL) Copyright (c) 2000, 2022, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> show dataabses; ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'dataabses' at line 1
数据copy过去后,没有重启直接进入pod,然后查看一下数据库,提示错误,重启mysql然后重新进入
[root@node-01 nfs]# kubectl delete pod mysql-6f6c97f4d5-bp84h --force Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "mysql-6f6c97f4d5-bp84h" force deleted [root@node-01 nfs]# kubectl get pod NAME READY STATUS RESTARTS AGE mysql-6f6c97f4d5-8vwb6 1/1 Running 0 1s nfs-client-provisioner-79fcb887b9-lzmxz 1/1 Running 0 6m57s nfs-sc-client-provisioner-7b48cf79c-bzqn2 1/1 Running 0 6m51s [root@node-01 nfs]# kubectl exec -it mysql-6f6c97f4d5-8vwb6 /bin/bash kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. bash-4.2# mysql -uroot -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.7.40 MySQL Community Server (GPL) Copyright (c) 2000, 2022, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | sys | | wulaoer | +--------------------+ 5 rows in set (0.01 sec) mysql> use wulaoer; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> select * from users; +----+---------------+-------------------+ | id | name | email | +----+---------------+-------------------+ | 1 | John Doe | john@example.com | | 2 | Jane Smith | jane@example.com | | 3 | Alice Johnson | alice@example.com | +----+---------------+-------------------+ 3 rows in set (0.00 sec)
数据没有报错了,主要原因是因为数据无法找到nfs中的数据了,所以需要重启重新加载数据.
nfs高可用
我们目前使用的是一台nfs作为整个集群的存储,集群中如果部署的中间件是集群方式的话,单个nfs无法满足集群效果,因为nfs只要挂了,所有挂在的中间件都会出现数据丢失的问题,为了保证数据不丢失,所以我们这里选择创建两个nfs,针对中间件集群选择多个nfs存储,这样就避免了nfs单点故障的问题了.
接着上面的使用,目前只有一个StorageClass,我们需要在创新创建一个StorageClass,然后把原来的pvc中的数据迁移过来看一下数据是否有丢失,服务是否需要重启.
[root@node-01 k8s]# cat nfs-storage.yml # nfs-storage 存储类和 Provisioner apiVersion: v1 kind: ServiceAccount metadata: name: nfs-client-provisioner namespace: default --- apiVersion: apps/v1 kind: Deployment metadata: name: nfs-client-provisioner namespace: default spec: replicas: 1 selector: matchLabels: app: nfs-client-provisioner template: metadata: labels: app: nfs-client-provisioner spec: serviceAccountName: nfs-client-provisioner containers: - name: provisioner image: registry.cn-beijing.aliyuncs.com/mydlq/nfs-subdir-external-provisioner:v4.0.0 env: - name: PROVISIONER_NAME value: "k8s-sigs.io/nfs-subdir-external-provisioner" # 设置正确的 PROVISIONER_NAME - name: NFS_SERVER value: "192.168.6.100" # 更新为实际的 NFS 服务器 IP 地址 - name: NFS_PATH value: "/nfs/data" # 更新为实际的 NFS 共享目录路径 volumeMounts: - name: nfs-client-root mountPath: /persistentvolumes volumes: - name: nfs-client-root nfs: server: "192.168.6.100" # 同样确保 NFS 服务器的地址是正确的 path: "/nfs/data" --- # nfs-storage 存储类的 RBAC 配置 apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: nfs-client-provisioner-runner rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["list", "watch", "create", "update", "patch"] - apiGroups: [""] resources: ["endpoints"] verbs: ["create", "delete", "get", "list", "watch", "patch", "update"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: run-nfs-client-provisioner subjects: - kind: ServiceAccount name: nfs-client-provisioner namespace: default roleRef: kind: ClusterRole name: nfs-client-provisioner-runner apiGroup: rbac.authorization.k8s.io
因为StorageClass需要单独一个文件,合并一个文件报错了
[root@node-01 k8s]# cat nfs-storage-sc.yml # 5. StorageClass apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: nfs-storage provisioner: k8s-sigs.io/nfs-subdir-external-provisioner parameters: archiveOnDelete: "true"
以上是命名为nfs-storage的StorageClass,下面创建一个命名为nfs-sc 的StorageClass,下面是yml文件内容.
[root@node-01 k8s]# cat nfs-sc.yml # nfs-sc 存储类和 Provisioner apiVersion: v1 kind: ServiceAccount metadata: name: nfs-sc-client-provisioner namespace: default --- apiVersion: apps/v1 kind: Deployment metadata: name: nfs-sc-client-provisioner namespace: default spec: replicas: 1 selector: matchLabels: app: nfs-sc-client-provisioner template: metadata: labels: app: nfs-sc-client-provisioner spec: serviceAccountName: nfs-sc-client-provisioner containers: - name: provisioner image: registry.cn-beijing.aliyuncs.com/mydlq/nfs-subdir-external-provisioner:v4.0.0 env: - name: PROVISIONER_NAME value: "k8s-sigs.io/nfs-subdir-external-provisioner-sc" # 设置正确的 PROVISIONER_NAME - name: NFS_SERVER value: "192.168.6.102" # 更新为实际的 NFS 服务器 IP 地址 - name: NFS_PATH value: "/nfs/data" # 更新为实际的 NFS 共享目录路径 volumeMounts: - name: nfs-client-root mountPath: /persistentvolumes volumes: - name: nfs-client-root nfs: server: "192.168.6.102" # 同样确保 NFS 服务器的地址是正确的 path: "/nfs/data" --- # nfs-sc 存储类的 RBAC 配置 apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: nfs-sc-client-provisioner-runner rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["list", "watch", "create", "update", "patch"] - apiGroups: [""] resources: ["endpoints"] verbs: ["create", "delete", "get", "list", "watch", "patch", "update"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: run-nfs-sc-client-provisioner subjects: - kind: ServiceAccount name: nfs-sc-client-provisioner namespace: default roleRef: kind: ClusterRole name: nfs-sc-client-provisioner-runner apiGroup: rbac.authorization.k8s.io [root@node-01 k8s]# cat nfs-sc-sc.yml # 5. StorageClass apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: nfs-sc provisioner: k8s-sigs.io/nfs-subdir-external-provisioner-sc parameters: archiveOnDelete: "true" # 启用归档删除
yml文件已经有了,下面创建,每个StorageClass要单独一个pod,要不两个StorageClass之间会有影响.
[root@node-01 k8s]# kubectl apply -f nfs-sc.yml serviceaccount/nfs-sc-client-provisioner created deployment.apps/nfs-sc-client-provisioner created clusterrole.rbac.authorization.k8s.io/nfs-sc-client-provisioner-runner created clusterrolebinding.rbac.authorization.k8s.io/run-nfs-sc-client-provisioner created [root@node-01 k8s]# kubectl apply -f nfs-sc-sc.yml storageclass.storage.k8s.io/nfs-sc created [root@node-01 k8s]# kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE nfs-sc k8s-sigs.io/nfs-subdir-external-provisioner Delete Immediate false 4s nfs-storage k8s-sigs.io/nfs-subdir-external-provisioner Delete Immediate false 41m [root@node-01 k8s]# kubectl get pod NAME READY STATUS RESTARTS AGE mysql-6f6c97f4d5-6q4w5 1/1 Running 0 26m nfs-client-provisioner-79fcb887b9-hvdrl 1/1 Running 0 35m nfs-sc-client-provisioner-5f9847bcd8-rnrkv 1/1 Running 0 14s
下面开始创建pvc
[root@node-01 k8s]# cat pvc.yml kind: PersistentVolumeClaim #创建PVC资源 apiVersion: v1 metadata: name: nginx-pvc #PVC的名称 spec: accessModes: #定义对PV的访问模式，代表PV可以被多个PVC以读写模式挂载 - ReadWriteMany resources: #定义PVC资源的参数 requests: #设置具体资源需求 storage: 200Mi #表示申请200MI的空间资源 storageClassName: nfs-sc [root@node-01 k8s]# cat mysql-pvc.yml kind: PersistentVolumeClaim #创建PVC资源 apiVersion: v1 metadata: name: mysql-pv-claim #PVC的名称 spec: accessModes: #定义对PV的访问模式，代表PV可以被多个PVC以读写模式挂载 - ReadWriteMany resources: #定义PVC资源的参数 requests: #设置具体资源需求 storage: 200Mi #表示申请200MI的空间资源 storageClassName: nfs-storage [root@node-01 k8s]# kubectl apply -f mysql-pvc.yml persistentvolumeclaim/mysql-pv-claim created [root@node-01 k8s]# kubectl apply -f pvc.yml persistentvolumeclaim/nginx-pvc created [root@node-01 k8s]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE mysql-pv-claim Bound pvc-271b41d4-2e23-44fb-9cc2-ac0e54465a51 200Mi RWX nfs-storage 8s nginx-pvc Bound pvc-05d058e0-b2de-4774-b34a-41ef0bdff3f1 200Mi RWX nfs-sc 3s
现在两个StorageClass里都创建了PVC和PV,我们只需要在两个nfs中验证一下,有这两个PVC的数据,说明我们的StorageClass创建成功.
[root@node-01 k8s]# ls /nfs/data default-mysql-pv-claim-pvc-271b41d4-2e23-44fb-9cc2-ac0e54465a51 [root@node-03 ~]# ls /nfs/data/ default-nginx-pvc-pvc-05d058e0-b2de-4774-b34a-41ef0bdff3f1
两个nfs服务中都有数据了,下面我们删除nginx-pvc整个PVC,然后在nfs-storage上创建看看是否受影响.
[root@node-01 k8s]# kubectl delete -f pvc.yml persistentvolumeclaim "nginx-pvc" deleted [root@node-01 k8s]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE mysql-pv-claim Bound pvc-271b41d4-2e23-44fb-9cc2-ac0e54465a51 200Mi RWX nfs-storage 12m [root@node-01 k8s]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-271b41d4-2e23-44fb-9cc2-ac0e54465a51 200Mi RWX Delete Bound default/mysql-pv-claim nfs-storage 13m [root@node-03 ~]# ls /nfs/data/ archived-pvc-05d058e0-b2de-4774-b34a-41ef0bdff3f1
在nfs-sc上从PVC删除完成了,但是nfs中数据不会清理,需要手动删除,然后重新指向nginx-pvc的StorageClass到nfs-storage
[root@node-01 k8s]# cat pvc.yml kind: PersistentVolumeClaim #创建PVC资源 apiVersion: v1 metadata: name: nginx-pvc #PVC的名称 spec: accessModes: #定义对PV的访问模式，代表PV可以被多个PVC以读写模式挂载 - ReadWriteMany resources: #定义PVC资源的参数 requests: #设置具体资源需求 storage: 200Mi #表示申请200MI的空间资源 storageClassName: nfs-storage [root@node-01 k8s]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE mysql-pv-claim Bound pvc-271b41d4-2e23-44fb-9cc2-ac0e54465a51 200Mi RWX nfs-storage 27m nginx-pvc Bound pvc-4e92b33f-f922-4bb8-9c84-c7cc0d00ff3c 200Mi RWX nfs-storage 116s
如果创建的PVC出现Pending状态,有可能是缓存,重启一下nfs的pod即可.
您可以选择一种方式赞助本站
支付宝扫一扫赞助
微信钱包扫描赞助
赏