云原生
Kubernetes基础
容器技术介绍
Docker快速入门
Containerd快速入门
K8S主要资源罗列
认识YAML
API资源对象
Kubernetes安全掌控
Kubernetes网络
Kubernetes高级调度
Kubernetes 存储
Kubernetes集群维护
Skywalking全链路监控
ConfigMap&Secret场景应用
Kubernetes基础概念及核心组件
水平自动扩容和缩容HPA
Jenkins
k8s中部署jenkins并利用master-slave模式实现CICD
Jenkins构建过程中常见问题排查与解决
Jenkins部署在k8s集群之外使用动态slave模式
Jenkins基于Helm的应用发布
Jenkins Pipeline语法
EFKStack
EFK日志平台部署管理
海量数据下的EFK架构优化升级
基于Loki的日志收集系统
Ingress
基于Kubernetes的Ingress-Nginx解决方案
Ingress-Nginx高级配置
使用 Ingress-Nginx 进行灰度(金丝雀)发布
Ingress-nginx优化配置
APM
Skywalking全链路监控
基于Helm部署Skywalking
应用接入Skywalking
服务网格
Istio
基于Istio的微服务可观察性
基于Istio的微服务Gateway实战
Kubernetes高可用集群部署
Kuberntes部署MetalLB负载均衡器
Ceph
使用cephadm部署ceph集群
使用Rook部署Ceph存储集群
openstack
glance上传镜像失败
mariadb运行不起来
创建域和项目错误_1
创建域和项目错误_2
安装计算节点
时钟源
网络创建失败
本文档使用 MrDoc 发布
-
+
首页
Kubernetes集群维护
# 1、查看Kubernetes集群资源使用情况和日志 1、 查看资源使用情况 1）kubectl top查看Node使用CPU和内存情况 ```Bash $ kubectl top node #查看所有node $ kubectl top node k8s01 #查看指定node ``` 2）kubectl top查看Pod使用CPU和内存情况 ```Bash $ kubectl top pod #查看所有Pod $ kubectl top pod php-apache-64b6b9d449-t9h4z #查看指定Pod ``` 注意: top功能需要先安装metrics-server，安装步骤参考6.17章节 2、查看日志 1）K8s相关日志 Linux系统里记录的日志 ```Bash $ journalctl -u kubelet ``` K8s各组件日志 首先查看Pod name ```Bash $ kubectl get po -n kube-system # calico-kube-controllers-xxxx, calico-node-xxx, coredns-xxx, etcd-xxx, kube-apiserver-xxx, kube-controller-manager-xxx, kube-proxy-xxx, kube-scheduler-xxx, metrics-server-xxx ``` 查看指定Pod日志 ```Bash $ kubectl logs -n kube-system calico-kube-controllers-798cc86c47-44525 $ kubectl logs -n kube-system kube-scheduler-k8s01 ``` 另外，可以加上-f选项动态查看指定pod日志，类似tail -f 2）应用日志 跟查看K8s组件日志一样，将Pod名字改为想查看的Pod名字即可 ```Bash $ kubectl logs php-apache-64b6b9d449-t9h4z ``` 另外，也可以进入到Pod内部去查看应用日志 ```Bash $ kubectl exec -it pod-name -n namespace-name -- bash ##进入后，再去查看具体的日志 ``` 有时候，我们的应用也会将日志目录给映射到Node上或者共享存储里，那样查看日志就方便多了。 # 2、维护Kubernetes集群CA证书 ## 2.1 Kubernetes集群中的CA证书 如果使用Kubeadm部署集群，CA证书会自动生成，但如果用二进制方式部署则需要手动生成。 服务器上CA证书在哪里？ ```Bash tree /etc/kubernetes/pki/ /etc/kubernetes/pki/ ├── apiserver.crt ├── apiserver-etcd-client.crt ├── apiserver-etcd-client.key ├── apiserver.key ├── apiserver-kubelet-client.crt ├── apiserver-kubelet-client.key ├── ca.crt ├── ca.key ├── etcd │ ├── ca.crt │ ├── ca.key │ ├── healthcheck-client.crt │ ├── healthcheck-client.key │ ├── peer.crt │ ├── peer.key │ ├── server.crt │ └── server.key ├── front-proxy-ca.crt ├── front-proxy-ca.key ├── front-proxy-client.crt ├── front-proxy-client.key ├── sa.key └── sa.pub ``` Kubernetes为了安全，使用的是双向认证（ 除了客户端需要验证服务器的证书，服务器也要通过客户端证书验证客户端的身份。） 1、 CA证书 kubeadm安装的集群中我们都是用3套CA证书来管理和签发其他证书，一套CA给ETCD使用，一套是给kubernates内部组件使用，还有一套是给配置聚合层使用的，当然如果你觉得管理3套CA比较麻烦，您也可以用一套来管理。 1）Etcd证书 Etcd证书位于/etc/kubernetes/pki/etcd目录下，可以用ps查看Etcd的进程以及参数： ```Bash # ps aux |grep etcd |grep -v 'kube-apiserver' root 1796 2.0 3.0 11215492 102036 ? Ssl 10:18 0:29 etcd --advertise-client-urls=https://192.168.222.101:2379 --cert-file=/etc/kubernetes/pki/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/etcd --experimental-initial-corrupt-check=true --experimental-watch-progress-notify-interval=5s --initial-advertise-peer-urls=https://192.168.222.101:2380 --initial-cluster=aminglinux01=https://192.168.222.101:2380 --key-file=/etc/kubernetes/pki/etcd/server.key --listen-client-urls=https://127.0.0.1:2379,https://192.168.222.101:2379 --listen-metrics-urls=http://127.0.0.1:2381 --listen-peer-urls=https://192.168.222.101:2380 --name=aminglinux01 --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt --peer-client-cert-auth=true --peer-key-file=/etc/kubernetes/pki/etcd/peer.key --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt --snapshot-count=10000 --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt ``` 证书以及说明 ```Bash ├── etcd │ ├── ca.crt ## 用于Etcd集群节点之间相互认证的CA证书 │ ├── ca.key ## 同上 │ ├── healthcheck-client.crt ## 当Etcd访问其它服务时，它作为客户端使用的CA证书 │ ├── healthcheck-client.key ## 同上 │ ├── peer.crt ## Etcd集群节点之间相互认证的peer证书，这是公钥 │ ├── peer.key ## 同上，这是私钥 │ ├── server.crt ## Etcd对外提供服务时，比如apiserver连接etcd时，它作为服务端的CA证书，这是公钥 │ └── server.key ## 同上，这是私钥 ``` 2） Kube-apiserver证书 Apiserver对应的证书目录在/etc/kubernetes/pki，可以用ps查看进程 ```Bash ps aux |grep apiserver root 1761 3.1 14.6 1254140 482468 ? Ssl 10:18 4:38 kube-apiserver --advertise-address=192.168.222.101 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.15.0.0/16 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key ``` 证书以及说明： ```Bash tree /etc/kubernetes/pki/ /etc/kubernetes/pki/ ├── apiserver.crt ##Apiserver作为服务端用到的CA证书 ├── apiserver.key ##同上 ├── apiserver-etcd-client.crt ##Apiserver作为客户端访问Etcd服务时用到的CA证书 ├── apiserver-etcd-client.key ##同上 ├── apiserver-kubelet-client.crt ##Apiserver访问kublet时，它作为客户端用到的证书 ├── apiserver-kubelet-client.key ##同上 ├── ca.crt ##用来签发k8s中其它证书CA证书，是一个根证书 ├── ca.key ##同上 ├── front-proxy-ca.crt ##配置聚合层（Apiserver扩展）的CA证书 ├── front-proxy-ca.key ##同上 ├── front-proxy-client.crt ##置聚合层（Apiserver扩展）的客户端证书 ├── front-proxy-client.key ##同上 ├── sa.key ##验证service account token用的私钥 └── sa.pub ##验证service account token用的公钥 ``` 3） kube-controller-manager用到的证书 查看进程： ```Bash ps aux |grep controller root 1809 0.8 3.7 826096 122324 ? Ssl 10:18 1:27 kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/etc/kubernetes/pki/ca.crt --cluster-cidr=10.18.0.0/16 --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt --cluster-signing-key-file=/etc/kubernetes/pki/ca.key --controllers=*,bootstrapsigner,tokencleaner --kubeconfig=/etc/kubernetes/controller-manager.conf --leader-elect=true --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --root-ca-file=/etc/kubernetes/pki/ca.crt --service-account-private-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.15.0.0/16 --use-service-account-credentials=true systemd+ 3328 0.0 1.7 1125840 56504 ? Ssl 10:18 0:02 /usr/bin/kube-controllers ``` 说明： ps看到的进程用到的ca证书如下 ```Bash /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/ca.key /etc/kubernetes/pki/front-proxy-ca.crt /etc/kubernetes/pki/sa.key ``` 这些证书其实是Apiserver相关的证书，而kube-controller-manager用到的证书在/etc/kubernetes/controller-manager.conf这个配置文件里 ```Bash cat /etc/kubernetes/controller-manager.conf apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJek1EVXlNekExTlRreE1Wb1hEVE16TURVeU1EQTFOVGt4TVZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTlViCjFYdENSOFdLU0R4d3g1Sis5SkVPS0dOditHSDF6L1p6cnFKaHpSaFl5RHdSQVpwM3lmVmJPbCtoVEdlc25qQ3MKbTVIM1AxNU4zWElMVlhDR3RPa0E1MFY3bENWNVIwaGxWUEUvWFZib0Y3ZllldU9lMmFqVkJaL01kN3hMeFVGegppQVhDdkVrZFVya0VHOUNWRm5IeGxRZHF0MEhaSXVISDB2ajBBMitXRkFMNDVzNTlvdktzM1Q5UlVHTnljRkx6CnE5VlNIT3hBcWh5QUd1dmdqZjgvZ3Q4eSs1blQxSlhBdWVEaktKRlVnd1JXVEQ0b1V5cERDNkFYMnRZbjdJTVcKUG1SNTJIbklCVThzajVwRUF0MVRuVFp0SURlL0ZHMXlNRlJmZGZFRnY4ZlpLdGlqZzRZNndycitQbnZjVXRMMApnbEZIWjFoM1NGL0xSbml2U05VQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZON1h5cVpsckxnWEg0bUhZb3YvYzVXWUhuVTVNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBSUNYVnZ6T013Ni9vYlp0REF4egpoemY1eU5IdFByYXZVb1lmbHhPRlAwdWhMdEMvT3hjL3lRZCtDZm5lVUtveDMxeXFLQXNwZXlzREZYOVpiZ1d0Ckt6bHJGVGcySjJLUm5PVnB0N21vMHlvS2lHemRvMFN1OXdncHZqa1p3OW84dWY0Qk5nYmdaMlJlbFVjTUVLRzcKTHczalR1ckJjRVJ3L3BwU2RnbDNxOHFIaVZCWUJpTVlSYXpwclJJK05YclErcHhXSHJ6WFRKamZvRGZVSHE0ZQo4bTJhZ011eGUzT1h4b1RZbnd5NDRldmtkUFNzb1UwRlc4ZEJnTXlQRnNOSjRYbnBaOVFqcjFodk1zVG02WXZTCmNudTFNbUFvQTdPZS93WWUyMXlMMHkvN3EzODNqcUltdUdoN3NodlhoZWFHMUxnNVZBT3FuQ3IvelVxYktJbzEKMThzPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== server: https://192.168.222.101:6443 name: kubernetes contexts: - context: cluster: kubernetes user: system:kube-controller-manager name: system:kube-controller-manager@kubernetes current-context: system:kube-controller-manager@kubernetes kind: Config preferences: {} users: - name: system:kube-controller-manager user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURGakNDQWY2Z0F3SUJBZ0lJRFRWVStoaEl3Wm93RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TXpBMU1qTXdOVFU1TVRGYUZ3MHlOREExTWpJd05UVTVNVFJhTUNreApKekFsQmdOVkJBTVRIbk41YzNSbGJUcHJkV0psTFdOdmJuUnliMnhzWlhJdGJXRnVZV2RsY2pDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU5pMXN5b1BFZm53amNKdkFJUE9wVWF0VnJkeFNHTDYKcHQ4amY0YjJML05zY1RXcmZCdGt4MzVhbTZoQnFxRW9vUHRhNnBqaktNWWJNUGFWM0Y5SGRKd0NQbTdQbC9FOQowMFluQ2dKbUpkLytuckRLbk02czNRb2Iwa3VSamNHbVY0anNnTDhDZ3NKeldxSFdQOHRadGZNaWtHOTNwbEp0CmViL3VWMDRxMm5VbmxkWTNRWS90WHdGVUNGcjFXYVIxdHdjQS9ZelF5M0FOcVR0OEcwM2tFbFcwYkpkZXE3T3gKNzlGcCsrczlDajBQLzdUTHZ3WTFxNG1MM0hxdEFCQU5jZDBiYXR2cXhZNys3cnZVUHpsV3p0ZFNMaW9ha2U0LwptbDFhL0NqR0RnNDlmdkFOck9rU1E0a1B2c1dOM3Nscit6RDFJUit3a0RuL1ZQdklydDRGWEE4Q0F3RUFBYU5XCk1GUXdEZ1lEVlIwUEFRSC9CQVFEQWdXZ01CTUdBMVVkSlFRTU1Bb0dDQ3NHQVFVRkJ3TUNNQXdHQTFVZEV3RUIKL3dRQ01BQXdId1lEVlIwakJCZ3dGb0FVM3RmS3BtV3N1QmNmaVlkaWkvOXpsWmdlZFRrd0RRWUpLb1pJaHZjTgpBUUVMQlFBRGdnRUJBSmNFaXNhTEc5THZNNCs3aGRWcEt3U0hrUm1OR2F1NTQ2ODE0ZHZUZUlETWRHTE1lMUFoCnU2VzlhV055MzcvK2Z4eTg2eEEwYmpacE52ZnpSRWUyWnJqR1k2NXVMcjJreHJnNGljMWRSc1Y0aG9Fa2RVTFIKZG5ZdWd4Tk9FZ2xTNWZDUndGUXB1MkZ4KzFjbC9YS2ZrOUozUVdOUlhZWThVYkR1NTNHQW41U3NCcDJRV3I4OApDR3dJazFTUWUvMk9MY1JIRExlVThXdlMwN3NDVVZZWHl1VEd1c25qMnlqNUh6bExMZ29lLzYxbVNDWUpsTDFKCkVxT3NiVEdJUjlnTVdiRy9RZGlCWE5VNWZQWmFQTVBoNzdBUUY1UU9tcXhsUHJlcyt6TEJCNVgrdUJZOEw5dEwKYjh1QXFpMHFqRUNob0VPVjFqNnN2NThHditKMjk0ejJkZU09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBMkxXektnOFIrZkNOd204QWc4NmxScTFXdDNGSVl2cW0zeU4vaHZZdjgyeHhOYXQ4CkcyVEhmbHFicUVHcW9TaWcrMXJxbU9Nb3hoc3c5cFhjWDBkMG5BSSticytYOFQzVFJpY0tBbVlsMy82ZXNNcWMKenF6ZENodlNTNUdOd2FaWGlPeUF2d0tDd25OYW9kWS95MW0xOHlLUWIzZW1VbTE1dis1WFRpcmFkU2VWMWpkQgpqKzFmQVZRSVd2VlpwSFczQndEOWpORExjQTJwTzN3YlRlUVNWYlJzbDE2cnM3SHYwV243NnowS1BRLy90TXUvCkJqV3JpWXZjZXEwQUVBMXgzUnRxMityRmp2N3V1OVEvT1ZiTzExSXVLaHFSN2orYVhWcjhLTVlPRGoxKzhBMnMKNlJKRGlRKyt4WTNleVd2N01QVWhIN0NRT2Y5VSs4aXUzZ1ZjRHdJREFRQUJBb0lCQUJ0MlF5TVVUOUhka3dsTgpxY0lLUU5XSkI5RXRVT0FDSlgxbmNUUzRJL0YwRDVDZWo1YWxHY2JBVXJOcXh1V3ZVVjhvOFV1NGxhVXRNdkk4Cm9YV05oNUJ4bEZuVWdzdTJhdnliMjkvYjgvYkR2SFpvNXFBbU5jaWZKL0lkbXRvc2F6RlZ6eWJsZVk2TXNiS0sKaFFFTytFaThXNU5VbFVZaXkyZndHeTR6cmdWa2FCekRtOC9JdFJWS0VqS0R1QlJmQjZyM1l2Ujl4NTBMSi91NwpXUnlTVzNOTmZCZG93cE55clRZSHdRKzdjZ0RaYzgvRWkzVEZDa1lVOFAxV3RwanNTQmdxa2g4am5PODhxc0tFCk5mZWt0TEZJTTRRMGVDTTdJK0RWazVjR0ZldnFRYmxlcFdhNEMvOGRVQ0dJRWxUSXRWbzhSTEpKMDFVQ3lWWFgKenp5MzBTa0NnWUVBNmNJZENGNGloS3piTzFlbG03MVlub3ZmZ3hJSnZiN3N1UW4xeGdGQ3FrUlA3ZzFMYktoUApGd05mL2U0ekJNYklkQWJKL1liNjl6ZnVXV3krZ0IzT2o1MUk1ekdGZnFVOTU0RGJYcFdrajZRbERDMHp1V1p6Ckg0cE5IYVNTbUdLSHlNVnBPMEU4Q2RNbVpTT3czMUhwaXF4cys5cXpLY2VnR240RS9RTzh1dk1DZ1lFQTdWUlIKc0xpU3dYdzdLc09SZ08yMDcvb3dYb1ozNUdSaHl2WG1PK2Iwc2NmUTFOSTltWm54YkJCZ0JYYkFMRnFWc210eAo4NFFWbHdnUGg5cTlwUHBqR2phZHM5cU5XTk95Ry9CZ2hzSkE5SkxNVVhuMVdvOEl0VVRnVEF5Y1c2OTNCbXBBCjJ5aGN5Tk0vU0RpWEgxQTA2VExUc3dQK3ZNTGgrZ1JuR0RTaEtYVUNnWUVBdkhlRC91WWV3SWU1Ymp2NVN2cTcKZzREVW9aR2Vodnh6VWJFbEZoS1lGRDFsa0ZIcHVONXVmZWg3c1hrbWF6aDJENDRWdHpIb0VBQnBUSEl2bVkzdQpwNzJ4TksycHF2VkJWdkIrYmVFS3Y4RWhHWk1zTzlQTkIrWHl6TzZPUHd4WjI5YkdSRDhSVC82TTIwaU9aMDljCmt6RDM0WXR2bmtUVDRlZ3V6MnYrODE4Q2dZQkN0VUdrNzZ4b2NEZmRSRTFBdFEvM3FCWXlhNCtGaGVXcSs4VTAKTHkyMlR0alhDU1ZCYUhEZ2xCVzVVM0NleHZOdXpCNmllUXpzZjlxRnNDbEFGeFVHQ0RYNW1NUEVKUFRhUDQyUwpNMFc0dVE1dkZGMnZqNmE5OU5jcTZSeUY5T0w2emFEZk9SQWpicVZKOG9ZZnloYWxPYUVobjB6WjE5bFd3aWdNCk9MYllCUUtCZ1FDbGF5dDl4UGlUQkk3amJXdk8zK1VUN05wRDJQRkdXSzlBUWJOc2NCcUVKV21BaGVMaFdKZHgKUEFJaTgrbmRiQjQwellFWXRqZlp1QVk5TkQraWVCakNKOTZ2TTZjM0FXNVAwckhJREJORHFWd2ZNRjJjYUlhTQpZNlNWa08wV3NvSSt4akRYNU9nbzkxR04rWGR5VnNvSkFJRE5ZUzhQSGF1ZVc2L282K1NDcWc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo= ``` Kubernetes 这里的设计是这样的：kube-controller-mananger、kube-scheduler、kube-proxy、kubelet等组件，采用一个kubeconfig 文件中配置的信息来访问 kube-apiserver。该文件中包含了 kube-apiserver 的地址，验证 kube-apiserver 服务器证书的 CA 证书，自己的客户端证书和私钥等访问信息。 4）Kube-scheduler 跟Kube-controller-namager一样，Kube-scheduler用的也是kubeconfig ```Bash cat /etc/kubernetes/scheduler.conf apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJek1EVXlNekExTlRreE1Wb1hEVE16TURVeU1EQTFOVGt4TVZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTlViCjFYdENSOFdLU0R4d3g1Sis5SkVPS0dOditHSDF6L1p6cnFKaHpSaFl5RHdSQVpwM3lmVmJPbCtoVEdlc25qQ3MKbTVIM1AxNU4zWElMVlhDR3RPa0E1MFY3bENWNVIwaGxWUEUvWFZib0Y3ZllldU9lMmFqVkJaL01kN3hMeFVGegppQVhDdkVrZFVya0VHOUNWRm5IeGxRZHF0MEhaSXVISDB2ajBBMitXRkFMNDVzNTlvdktzM1Q5UlVHTnljRkx6CnE5VlNIT3hBcWh5QUd1dmdqZjgvZ3Q4eSs1blQxSlhBdWVEaktKRlVnd1JXVEQ0b1V5cERDNkFYMnRZbjdJTVcKUG1SNTJIbklCVThzajVwRUF0MVRuVFp0SURlL0ZHMXlNRlJmZGZFRnY4ZlpLdGlqZzRZNndycitQbnZjVXRMMApnbEZIWjFoM1NGL0xSbml2U05VQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZON1h5cVpsckxnWEg0bUhZb3YvYzVXWUhuVTVNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBSUNYVnZ6T013Ni9vYlp0REF4egpoemY1eU5IdFByYXZVb1lmbHhPRlAwdWhMdEMvT3hjL3lRZCtDZm5lVUtveDMxeXFLQXNwZXlzREZYOVpiZ1d0Ckt6bHJGVGcySjJLUm5PVnB0N21vMHlvS2lHemRvMFN1OXdncHZqa1p3OW84dWY0Qk5nYmdaMlJlbFVjTUVLRzcKTHczalR1ckJjRVJ3L3BwU2RnbDNxOHFIaVZCWUJpTVlSYXpwclJJK05YclErcHhXSHJ6WFRKamZvRGZVSHE0ZQo4bTJhZ011eGUzT1h4b1RZbnd5NDRldmtkUFNzb1UwRlc4ZEJnTXlQRnNOSjRYbnBaOVFqcjFodk1zVG02WXZTCmNudTFNbUFvQTdPZS93WWUyMXlMMHkvN3EzODNqcUltdUdoN3NodlhoZWFHMUxnNVZBT3FuQ3IvelVxYktJbzEKMThzPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== server: https://192.168.222.101:6443 name: kubernetes contexts: - context: cluster: kubernetes user: system:kube-scheduler name: system:kube-scheduler@kubernetes current-context: system:kube-scheduler@kubernetes kind: Config preferences: {} users: - name: system:kube-scheduler user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUREVENDQWZXZ0F3SUJBZ0lJQkFIcmNyRTJGMzR3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TXpBMU1qTXdOVFU1TVRGYUZ3MHlOREExTWpJd05UVTVNVFJhTUNBeApIakFjQmdOVkJBTVRGWE41YzNSbGJUcHJkV0psTFhOamFHVmtkV3hsY2pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCCkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU13TXVRd05TTks3NURBTWk2WTlLaHpYQ1dHOXVNSEpjNmFNb1krOUs2WjQKbmRQNGJHNlU1WU5WbE9xVkd5UWoyejFEU05yLzYvdnJTRS84ZEU3SitGT2VDZEhCTWp6ZjhjdExOU0U1WGo2Kwo4eFYwMTNHNlo4aTh4d3JwZjZUZ2VpTmJ0UjJkaytlUXJCNDh0MmxMWUE2blZSSVl5RDRZR3EyNTdaRlM4Z3dTCmZlN3YzUU9Ud1RYeDlTeHRwc2x2YzV5NFllUXBuWVN4L1lLeUhwUWFFMjBLZCtxcWFQTjBnMFVOSTF0ZmdwbDAKc002T2F5RERwNVZSeml5TkhaL2RpK0JZYldOSnlaWWsxZ3h0dVdMeGd6eGxVRGRaVk9KaFRxSk5yc081L2RRRwpRNkxDaXJWUWk4aWpwbHBlTkRucFVicFRydno5TWc0MEpqSTJWQU92K0YwQ0F3RUFBYU5XTUZRd0RnWURWUjBQCkFRSC9CQVFEQWdXZ01CTUdBMVVkSlFRTU1Bb0dDQ3NHQVFVRkJ3TUNNQXdHQTFVZEV3RUIvd1FDTUFBd0h3WUQKVlIwakJCZ3dGb0FVM3RmS3BtV3N1QmNmaVlkaWkvOXpsWmdlZFRrd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQgpBRnVkRjJ2VmNtQjdJVFNYa29GUTlWUVFwQXVxLzZLZDFvcG9XVE9yQW85ZG9Bc2JMWlNKKzY3UjlUeFNEbFRnClFLZEdEYTJZNDZtZGtpOXgzUFdXZ0h3S0xyMGlWS0s0aEc4Zy8zbEhhMXJDTnJHMFo0UG1GR3c1Q3RXazZtWkgKZFlJaEs2ODZJRjcvdHZMbFJsdk12NDdXKzVNZGtFZXdKamNlWEtqR2ZDRmRaa2ErTmFySTV5MXhwbkxOYmtwbwpsY1dTKzUyb3BVZGlpcEJGOUt6bzdwL0ZuWEcweVQvQnFwbXdHUUdDUlZCdll5L1NwanpTb0hKK25vRytIK2dKCjIvWXR0eVdlQnEvQ2xCOHhhUHRJRDVLT0psa2VCOGRFOENoSjk1TkxKdmN5aWpUcmhuLzNBSGYxUzZsWmtkTHoKbUcySDZacE5zRHdCUjlPcjMvMmNINFE9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBekF5NURBMUkwcnZrTUF5THBqMHFITmNKWWIyNHdjbHpwb3loajcwcnBuaWQwL2hzCmJwVGxnMVdVNnBVYkpDUGJQVU5JMnYvcisrdElUL3gwVHNuNFU1NEowY0V5UE4veHkwczFJVGxlUHI3ekZYVFgKY2JwbnlMekhDdWwvcE9CNkkxdTFIWjJUNTVDc0hqeTNhVXRnRHFkVkVoaklQaGdhcmJudGtWTHlEQko5N3UvZApBNVBCTmZIMUxHMm15Vzl6bkxoaDVDbWRoTEg5Z3JJZWxCb1RiUXAzNnFwbzgzU0RSUTBqVzErQ21YU3d6bzVyCklNT25sVkhPTEkwZG45Mkw0Rmh0WTBuSmxpVFdERzI1WXZHRFBHVlFOMWxVNG1GT29rMnV3N245MUFaRG9zS0sKdFZDTHlLT21XbDQwT2VsUnVsT3UvUDB5RGpRbU1qWlVBNi80WFFJREFRQUJBb0lCQUdibDc1YzFSOGtGZVZBRgpzNUswZTMwcHI1QjdsU1VjU3RidFNqZnJKSTBCZkUyOWxDaTdxM1hzZlhsQ2tmcEtucStwTTU5RXVhTkRoaWJYCjMyc3dES3RXSWd1RnlxaktJZXZyMURJWjlQM2RXMFl4c3NlbVFkb0g0TFFQQmhDR0p2R1h1S0RqcXdkMmZHV1AKSnNyUmVQSkt6ZXFmVnJPQmdrcWpYVHNRd2dEMWxySm9mUzFyUk0vUXlGYzNZYWUya1l3SEhtbUV3K25pWFJ3YwpPM0FkNHZDZGVYN2lwbmUrTFE1dGpCWWVwVHJlU0VlNS80alNyV1ZaN0F3VDAvZW02NnFJbGRMcHZPQXRpZm9xCjFQSXc1VHBPNGxlSHF1K2VKRlJLcU9sM3h0Sm1hbUIrODJTT1c2Z0pkNmlubEZ5UmRBNXhJeG1RN2pPWFVhTksKSnVkRi9oMENnWUVBKy9DWmxHSzVQRGJ4a0pGek1sYjBoNFk2OTBad096NThIRHMwVWxPRVkzN2VCSVF4MjV0Uwp3TlJiTHB0dU9WNmhLeWEzVTdvSTN6YzBzZHVtdk1hSXVtQWVOb08vZUR3V0tlRm1JSFBRbzJrZXVmVFAyUyt0Cm5ydndGcFVxdnk0UW4rRURBTTljUVZyMEN2TFdMSmNYSTVxK3ZVdUJlNjVnaFFlL2xnT1dzMDhDZ1lFQXoxYU0KUXZWNzFtUWxLWktUZE5GNTdiK211UTQvLytMblZianFIVjNRM2VaQkFneVhrb254SDV6eG0vdDdIZkxRbGM1ZAo5TThIWThEdmg1VnFOMVdDYnJvb2lQaFF6Z2lDM01qamJoc2FwcVppNk1pTFJ0UXlBTFFiOGVWYWs3RHBINjNnCmdiN1dxUTByUHBjTFA0bk5lVFQxb0Z5ZjhVMXJBR1l1RVNpd1hwTUNnWUVBaGxOTHJ4L0wzMXowQXI1cUN3YnQKcld4VitBMG9QWTRkSWZnMjkwWHNLaGcwUzNnb0ZReUZYS1pjVDBjeng0YTZxNFVpN1pNY3M5WjlVKzdtd1hPZwo1cTNrcmZBa24vaDVZSzc2Z29iOVVJTDFqUFFIOWFaRURZUHFpRC9UNE1hd3VtMS82bWlrcVh2UzdodDNNYU1GCkJVaTJOYnNTT0YxS3ZPTGF0U01Jc0dVQ2dZQTNxV3A2UjJENWkyVVdwZzBFSFlCNFBTMUxIUEk5aGZMSTVwMmoKU0o1Y0ZBWjd3Rm1vczNPU0l4WVI1QStIV0xwNm5TNUlISlJGU1hTRWw2NHVNTHlWY1ZDVWhPVmNpcGhuSVY4OQpIZlducTU5K0V1aWhHVEpiVWY3Mmo3WWpWQ2tob2hKVmdxQXFQaWpQNHNqWVEreHZqN0lwWFBScldYZFNZaHdYCjZ5NStGUUtCZ1FDUjR3VVdKS29pdkdIeExVUkQ0U0NEWnorYlFOaVFMUkJ6RjR1WUVSOWxjTlp1cXdxY3dPaXkKQXpCY0tMR3RlaGxpcjJOR21XWUpTZ0dBenRQWmZ5RnFPcUlHMU9QT3F3c1dqTjNISkdoREFGWktWLzBUb3dsMQo2MVBUOHV2ODhZTGZOeU1mSTh4UzY5QWNhQlpYWk1TTkFkdlZSKysrcFBmZDhrSEQxYVhDdUE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo= ``` 可以看到，配置文件里的证书内容和Kube-controller-namager一样。 5）Kubelet Kubelet用的也是kubeconfig ```Bash cat /etc/kubernetes/kubelet.conf apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJek1EVXlNekExTlRreE1Wb1hEVE16TURVeU1EQTFOVGt4TVZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTlViCjFYdENSOFdLU0R4d3g1Sis5SkVPS0dOditHSDF6L1p6cnFKaHpSaFl5RHdSQVpwM3lmVmJPbCtoVEdlc25qQ3MKbTVIM1AxNU4zWElMVlhDR3RPa0E1MFY3bENWNVIwaGxWUEUvWFZib0Y3ZllldU9lMmFqVkJaL01kN3hMeFVGegppQVhDdkVrZFVya0VHOUNWRm5IeGxRZHF0MEhaSXVISDB2ajBBMitXRkFMNDVzNTlvdktzM1Q5UlVHTnljRkx6CnE5VlNIT3hBcWh5QUd1dmdqZjgvZ3Q4eSs1blQxSlhBdWVEaktKRlVnd1JXVEQ0b1V5cERDNkFYMnRZbjdJTVcKUG1SNTJIbklCVThzajVwRUF0MVRuVFp0SURlL0ZHMXlNRlJmZGZFRnY4ZlpLdGlqZzRZNndycitQbnZjVXRMMApnbEZIWjFoM1NGL0xSbml2U05VQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZON1h5cVpsckxnWEg0bUhZb3YvYzVXWUhuVTVNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBSUNYVnZ6T013Ni9vYlp0REF4egpoemY1eU5IdFByYXZVb1lmbHhPRlAwdWhMdEMvT3hjL3lRZCtDZm5lVUtveDMxeXFLQXNwZXlzREZYOVpiZ1d0Ckt6bHJGVGcySjJLUm5PVnB0N21vMHlvS2lHemRvMFN1OXdncHZqa1p3OW84dWY0Qk5nYmdaMlJlbFVjTUVLRzcKTHczalR1ckJjRVJ3L3BwU2RnbDNxOHFIaVZCWUJpTVlSYXpwclJJK05YclErcHhXSHJ6WFRKamZvRGZVSHE0ZQo4bTJhZ011eGUzT1h4b1RZbnd5NDRldmtkUFNzb1UwRlc4ZEJnTXlQRnNOSjRYbnBaOVFqcjFodk1zVG02WXZTCmNudTFNbUFvQTdPZS93WWUyMXlMMHkvN3EzODNqcUltdUdoN3NodlhoZWFHMUxnNVZBT3FuQ3IvelVxYktJbzEKMThzPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== server: https://192.168.222.101:6443 name: kubernetes contexts: - context: cluster: kubernetes user: system:node:aminglinux01 name: system:node:aminglinux01@kubernetes current-context: system:node:aminglinux01@kubernetes kind: Config preferences: {} users: - name: system:node:aminglinux01 user: client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem client-key: /var/lib/kubelet/pki/kubelet-client-current.pem ``` 这个certificate-authority-data对应的数据和上面几个组件一样。而最下面的user配置段里有client-certificate和client-key，为kubelet作为客户端时用的CA证书。 2、续签证书 CA证书是有时效性的，如果过期了会影响到业务。如何查看证书何时到期呢？ ```Bash $ openssl x509 -noout -dates -in /etc/kubernetes/pki/apiserver.crt notBefore=Sep 27 12:25:11 2023 GMT # 什么时候产生 notAfter=Sep 26 12:30:12 2024 GMT # 什么时候失效 ``` 可见证书有效期为1年。 如果你的Kubernetes集群是由kubeadm搭建，那么还有一种方法，使用kubeadm查看整个集群所有证书有效期： ```Bash $ kubeadm certs check-expiration [check-expiration] Reading configuration from the cluster... [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Sep 26, 2024 12:30 UTC 329d ca no apiserver Sep 26, 2024 12:30 UTC 329d ca no apiserver-etcd-client Sep 26, 2024 12:30 UTC 329d etcd-ca no apiserver-kubelet-client Sep 26, 2024 12:30 UTC 329d ca no controller-manager.conf Sep 26, 2024 12:30 UTC 329d ca no etcd-healthcheck-client Sep 26, 2024 12:30 UTC 329d etcd-ca no etcd-peer Sep 26, 2024 12:30 UTC 329d etcd-ca no etcd-server Sep 26, 2024 12:30 UTC 329d etcd-ca no front-proxy-client Sep 26, 2024 12:30 UTC 329d front-proxy-ca no scheduler.conf Sep 26, 2024 12:30 UTC 329d ca no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Sep 24, 2033 12:30 UTC 9y no etcd-ca Sep 24, 2033 12:30 UTC 9y no front-proxy-ca Sep 24, 2033 12:30 UTC 9y no ``` 如果到期，使用kubeadm可以续签证书，方法是： ```Bash $ kubeadm certs renew all ``` 看输出，最后面有一句提醒，You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates. 需要重启这些服务：kube-apiserver, kube-controller-manager, kube-scheduler and etcd # 3、Kubernetes集群版本升级 1）为什么要升级 - ① 为了使用新功能 - ② 当前版本存在bug - ③ 当前版本存在安全漏洞 2）注意事项： - ① 不支持跨版本升级（这个跨版本指的是主要版本和次要版本，比如1.24.2，其中1为主要版本，24为次要版本，2为补丁版本） - 示例: ```Bash 1.20.2 --> 1.21.4 支持 1.20.2 --> 1.22.3 不支持 1.25.0 --> 1.25.4 支持 ``` - ② 升级前做备份 - ③ 升级前拿测试环境做演练 3）升级流程 - ① Node层面 - 先升级Master k8s01（如果有多个Master，需要一台一台升级） --> 再升级Worker节点k8s02和k8s03 - ② 软件层面 - 升级kubeadm --> 节点执行drain操作 --> 升级各组件（apiserver, coredns, kube-proxy, controller-manager, scheduler等）--> 取消drain --> 升级kubelet和kubectl 官方升级文档： https://kubernetes.io/zh-cn/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/ 4）升级步骤 示例： 1.26.9 --> 1.27.6 ① 升级Master 查看最新版本 ```Bash $ yum list --showduplicates kubeadm ``` 升级kubeadm ```Bash $ yum install -y kubeadm-1.27.6-0 ##需要指定版本号 ``` 驱逐Master上的Pod ```Bash $ kubectl drain master01 --ignore-daemonsets ``` 查看集群是否能够升级 ```Bash $ kubeadm upgrade plan ``` 执行升级 ```Bash $ kubeadm upgrade apply v1.27.6 ``` 升级kubelet和kubectl ```Bash $ yum install -y kubelet-1.27.6-0 kubectl-1.27.6-0 ``` 重启kubelet ```Bash $ systemctl daemon-reload $ systemctl restart kubelet ``` 恢复调度，上线 ```Bash $ kubectl uncordon master01 ``` ② 升级Work第一个节点 升级kubeadm（node01上执行） ```Bash $ yum -y install kubeadm-1.27.6-0 ##需要指定版本号 ``` 驱逐node01上的Pod（master01上执行） ```Bash $ kubectl drain node01 --ignore-daemonsets --delete-emptydir-data ``` 升级kubelet配置（node01上执行） ```Bash $ kubeadm upgrade node ``` 升级kubelet和kubectl（node01上执行） ```Bash $ yum install -y kubelet-1.27.6-0 kubectl-1.27.6-0 ``` 重启kubelet（node01上执行） ```Bash $ systemctl daemon-reload $ systemctl restart kubelet ``` 恢复调度，上线（master01上执行） ```Bash $ kubectl uncordon node01 ``` ③ 升级Work第二个节点 升级kubeadm（node02上执行） ```Bash $ yum install -y kubeadm-1.27.6-0 ##需要指定版本号 ``` 驱逐node02上的Pod（master01上执行） ```Bash $ kubectl drain node02 --ignore-daemonsets --delete-emptydir-data ``` 升级kubelet配置（node02上执行） ```Bash $ kubeadm upgrade node ``` 升级kubelet和kubectl（node02上执行） ```Bash $ yum install -y kubelet-1.27.6-0 kubectl-1.27.6-0 ``` 重启kubelet（node02上执行） ```Bash $ systemctl daemon-reload $ systemctl restart kubelet ``` 恢复调度，上线（master01上执行） ```Bash $ kubectl uncordon node02 ``` 如果有其它Node，继续模仿上面的③操作即可。 查看集群状态 ```Bash $ kubectl get node NAME STATUS ROLES AGE VERSION master01 Ready control-plane 35d v1.27.6 master02 Ready control-plane 35d v1.27.6 master03 Ready control-plane 35d v1.27.6 node01 NotReady worker 35d v1.27.6 node02 Ready worker 35d v1.27.6 node03 Ready worker 35d v1.27.6 ``` # 4、Kubernetes节点上线和下线 ## 4.1 新节点上线 1）准备工作 关闭防火墙、SELINUX ``` $ systemctl stop firewalld && systemctl disable firewalld $ sed -i 's/enforcing/disabled/g' /etc/selinux/config $ setenforce 0 ``` 配置主机名 node节点,名称为node04 ``` $ hostnamectl set-hostname node04 ``` 配置Host文件 ``` $ cat > /etc/hosts <<EOF 10.0.1.200 master01 10.0.1.201 master02 10.0.1.202 master03 10.0.1.203 node01 10.0.1.204 node02 10.0.1.205 node03 10.0.1.206 node04 EOF ``` 时间同步配置 ``` $ yum install -y chrony $ systemctl start chronyd && systemctl enable chronyd ``` 配置内核转发及网桥过滤 ``` $ cat > /etc/modules-load.d/k8s.conf << EOF overlay br_netfilter EOF 添加网桥过滤及内核转发配置文件 $ cat > /etc/sysctl.d/k8s.conf <<EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 vm.swappiness = 0 EOF 加载br_netfilter模块 $ modprobe overlay $ modprobe br_netfilter 查看是否加载 $ lsmod | grep br_netfilter br_netfilter 22256 0 bridge 151336 1 br_netfilter 加载网桥过滤及内核转发配置文件 $ sysctl -p /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 vm.swappiness = 0 ``` 安装ipset及ipvsadm ``` $ dnf install -y ipset ipvsadm ``` 配置ipvsadm模块加载方式，添加需要加载的模块 ``` $ cat > /etc/sysconfig/modules/ipvs.modules <<EOF #!/bin/bash modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack EOF ``` 授权、运行、检查是否加载 ``` $ chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack ``` 关闭SWAP分区 ``` $ swapoff -a # 临时 --有用 $ sed -ri 's/.*swap.*/#&/' /etc/fstab ``` 配置containerd 官方文档：https://github.com/containerd/containerd/blob/main/docs/getting-started.md 下载并解压 地址：https://github.com/containerd/containerd/releases ``` $ wget https://github.com/containerd/containerd/releases/download/v1.7.6/containerd-1.7.6-linux-amd64.tar.gz $ tar xzvf /usr/local containerd-1.7.6-linux-amd64.tar.gz -C /usr/local ``` 配置systemd ``` $ wget -O /usr/lib/systemd/system/containerd.service https://raw.githubusercontent.com/containerd/containerd/main/containerd.service $ systemctl daemon-reload $ systemctl enable --now containerd ``` 配置runc 地址： https://github.com/opencontainers/runc/releases ``` $ wget https://github.com/opencontainers/runc/archive/refs/tags/v1.1.8.tar.gz $ install -m 755 runc.amd64 /usr/local/sbin/runc ``` 配置CNI plugins 地址：https://github.com/containernetworking/plugins/releases ``` $ wget https://github.com/containernetworking/plugins/releases/download/v1.3.0/cni-plugins-linux-amd64-v1.3.0.tgz $ mkdir -p /opt/cni/bin $ tar xzvf cni-plugins-linux-amd64-v1.1.1.tgz -C /opt/cni/bin ``` 配置cgroup ``` $ mkdir /etc/containerd $ /usr/local/bin/containerd config default > /etc/containerd/config.toml $ sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml $ sed -i 's/sandbox_image = "registry.k8s.io/pause:3.8"/sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"/g' /etc/containerd/config.toml $ systemctl restart containerd ``` 3）配置kubernetes仓库 ```Bash $ cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF ``` 说明：kubernetes用的是RHEL7的源，和8是通用的 4）安装kubeadm和kubelet ```Bash $ yum install -y kubelet-1.27.6 kubeadm-1.27.6 kubectl-1.27.6 ``` 启动kubelet服务 ```Bash $ systemctl start kubelet.service $ systemctl enable kubelet.service ``` 为了实现docker使用的cgroupdriver与kubelet使用的cgroup的一致性，建议修改如下文件内容。 ```Bash $ vim /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS="--cgroup-driver=systemd" 或 sed -i 's/KUBELET_EXTRA_ARGS=/KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"/g' /etc/sysconfig/kubelet ``` 5）设置crictl连接 containerd ```Bash $ crictl config --set runtime-endpoint=unix:///run/containerd/containerd.sock ``` 6）到master节点上，获取join token ```Bash $ kubeadm token create --print-join-command ``` 7）到新节点，加入集群 ```Bash $ kubeadm join k8s.zhoumx.cc:6443 --token 17c6pl.9lwnl4q39wa0lx68 --discovery-token-ca-cert-hash sha256:1299ad3afbf3e37c0421bb6abbf2a45191accdbeca410b358546db69d2e1c293 ``` 8）master上查看node信息 ```Bash $ kubectl get node NAME STATUS ROLES AGE VERSION master01 Ready control-plane 35d v1.27.6 master02 Ready control-plane 35d v1.27.6 master03 Ready control-plane 35d v1.27.6 node01 Ready worker 35d v1.27.6 node02 Ready worker 35d v1.27.6 node03 Ready worker 35d v1.27.6 node04 NotReady <none> 5s v1.27.6 ``` 等待calico pod完成后再次查看node信息 ```Bash $ kubectl get node NAME STATUS ROLES AGE VERSION master01 Ready control-plane 35d v1.27.6 master02 Ready control-plane 35d v1.27.6 master03 Ready control-plane 35d v1.27.6 node01 Ready worker 35d v1.27.6 node02 Ready worker 35d v1.27.6 node03 Ready worker 35d v1.27.6 node04 Ready <none> 2m41s v1.27.6 ``` ## 4.2 节点下线 1）下线之前，先创建一个测试Deployment ```Bash 命令行创建deployment，指定Pod副本为7 $ kubectl create deployment testdp2 --image=nginx:1.23.2 --replicas=7 ``` 查看Pod ```Bash $ kubectl get po -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES testdp2-7b965b84bf-5gmr7 1/1 Running 0 2m39s 10.224.59.214 master02 <none> <none> testdp2-7b965b84bf-7b6h4 1/1 Running 0 2m39s 10.224.241.97 master01 <none> <none> testdp2-7b965b84bf-7wz42 1/1 Running 0 2m39s 10.224.235.28 master03 <none> <none> testdp2-7b965b84bf-bbdbp 1/1 Running 0 2m39s 10.224.186.211 node03 <none> <none> testdp2-7b965b84bf-qqwwd 1/1 Running 0 2m39s 10.224.196.143 node01 <none> <none> testdp2-7b965b84bf-qwkg2 1/1 Running 0 2m39s 10.224.140.65 node02 <none> <none> testdp2-7b965b84bf-wmqzk 1/1 Running 0 2m39s 10.224.248.195 node04 <none> <none> ``` 2）驱逐下线节点上的Pod，并设置不可调度（master01上执行） ```Bash $ kubectl drain node04 --ignore-daemonsets ``` 3）恢复可调度（master01上执行） ```Bash $ kubectl uncordon node04 ``` 4）移除节点 ```Bash $ kubectl delete node node04 ``` 查看node信息 ```Bash $ kubectl get node NAME STATUS ROLES AGE VERSION master01 Ready control-plane 35d v1.27.6 master02 Ready control-plane 35d v1.27.6 master03 Ready control-plane 35d v1.27.6 node01 Ready worker 35d v1.27.6 node02 Ready worker 35d v1.27.6 node03 Ready worker 35d v1.27.6 ```
阿星
2024年1月6日 21:17
转发文档
收藏文档
上一篇
下一篇
手机扫码
复制链接
手机扫一扫转发分享
复制链接
Markdown文件
PDF文档（打印）
分享
链接
类型
密码
更新密码