K8s环境持续部署

K8s环境持续部署

目录

[toc]

推荐文章

https:COPYindex.html /usr/share/nginx/html/

• 编辑index.html文件

<h1>VERSION:main</h1>

• 配置好sonar参数

创建sonar-project.properties文件

# 定义唯一的关键字sonar.projectKey=devops6-npm-service# 定义项目名称sonar.projectName=devops6-npm-service# 定义项目的版本信息sonar.projectVersion=1.0# 指定扫描代码的目录位置（多个逗号分隔）sonar.sources=src# 执行项目编码sonar.sourceEncoding=UTF-8# 指定sonar Serversonar.host.url=http:# 认证信息sonar.login=adminsonar.password=Admin@123

然后提交项目代码。

创建一条新pipeline

• 以原devops6-maven-service_CI为基础拷贝一条新流水线devops6-npm-service_K8SCI：

⚠️ 注意：但是这里的Git选项参数一直没效果，测试不出来，很奇怪。。。因此，就直接用默认的选项参数就好。

• 开始进行构建

可以看待，前面几个过程都是ok的，这里上传制品过程，不需要。

创建harbor仓库

• 镜像命名规范：

• 创建仓库

• 修改本地 docker 服务使用 http 协议和私有仓库通信（否则会报错）

#配置可信任（如果仓库是HTTPS访问不用配置）#在 daemon.json 中添加以下参数[root@harbor ~]# vim /etc/docker/daemon.json #创建此文件，并写入以下内容{"insecure-registries":["172.29.9.120"] }#重启docker 服务[root@harbor ~]# systemctl daemon-reload &&systemctlrestartdocker

编写CI pipeline

• 创建k8sci.jenkinsfile

@Library("devops06@main") _defcheckout =neworg.devops.CheckOut()defbuild =neworg.devops.Build()defsonar =neworg.devops.Sonar()env.branchName ="${env.branchName}"-"origin/"println(env.branchName)pipeline {agent {label "build"}options {skipDefaultCheckout true} stages{stage("CheckOut"){steps{script{checkout.CheckOut()env.commitID =checkout.GetCommitID()println("commitID:${env.commitID}")currentBuild.displayName ="第${BUILD_NUMBER}次构建-${env.commitID}"currentBuild.description ="构建分支名称：${env.branchName}"}}}stage("Build"){steps{script{build.Build()}}} stage("CodeScan"){when {environment name:'skipSonar',value:'false'}steps{script{sonar.SonarScannerByPlugin() }}}stage("ImageBuild"){steps{script{appName ="${JOB_NAME}".split('_')[0] repoName =appName.split('-')[0] imageName ="${repoName}/${appName}"imageTag ="${env.branchName}-${env.commitID}"sh """#登录镜像仓库docker login -u admin -p Harbor12345 172.29.9.120# 构建镜像docker build -t 172.29.9.120/${imageName}:${imageTag} .# 上传镜像docker push 172.29.9.120/${imageName}:${imageTag}# 删除镜像sleep 2docker rmi 172.29.9.120/${imageName}:${imageTag}"""}}} }}

• 在回放里运行

• 然后将代码提交到共享库里。

==准备k8s环境==

• 启动好k8s环境

• 在devops06机器上安装kubectl工具

cat>/etc/yum.repos.d/kubernetes.repo<<EOF[kubernetes]name=Kubernetesbaseurl=https:enabled=1gpgcheck=0repo_gpgcheck=0gpgkey=https:EOF[root@Devops6 ~]#yum install -y kubectl-1.20.0 --disableexcludes=kubernetes

• 然后把k8s集群master节点下的~/.kube/config文件拷贝到devops6机器~/.kube/目录下

• 然后在k8s的2个节点先配置好 修改本地 docker 服务使用 http 协议和私有仓库通信（否则会报错）

#配置可信任（如果仓库是HTTPS访问不用配置）#在 daemon.json 中添加以下参数[root@harbor ~]# vim /etc/docker/daemon.json #创建此文件，并写入以下内容{"insecure-registries":["172.29.9.120"] }#重启docker 服务[root@harbor ~]# systemctl daemon-reload &&systemctlrestartdocker

记得：只需要在node1 node2上配置就行。

• 自己k8s集群需不是一个ingress-controller。

这里之前已经部署好ingress-controller了。

• 配置下kubectl的自动补全

#安装软件包yuminstall-yepel-releasebash-completion#执行命令source/usr/share/bash-completion/bash_completionsource<(kubectlcompletion bash)echo"source <(kubectl completion bash)">>~/.bashrcsource~/.bashrc

==创建一个devops6-deploy-repo仓库==

• 创建Deployment.yaml

apiVersion:apps/v1kind:Deploymentmetadata:name:devops6-npm-servicespec:replicas:3revisionHistoryLimit:3selector:matchLabels:app:devops6-npm-servicetemplate:metadata:labels:app:devops6-npm-servicespec:containers:- image:172.29.9.120/devops6/devops6-npm-service:main-ed12ce10name:devops6-npm-serviceports:- containerPort:80---apiVersion:v1kind:Servicemetadata:name:devops6-npm-servicespec:type:ClusterIPselector:app:devops6-npm-serviceports:- name:httpprotocol:TCPport:80targetPort:80---apiVersion:networking.k8s.io/v1kind:Ingressmetadata:name:devops6-npm-serviceannotations:kubernetes.io/ingress.class:nginxspec:rules:- host:devops.test.comhttp:paths:- path:/pathType:Prefixbackend:service:name:devops6-npm-serviceport:name:http

• 自己部署应用测试

创建新命名空间devops6

[root@Devops6 ~]#kubectl create ns devops6

部署应用：

[root@Devops6 ~]#kubectl -n devops6 apply -f Deployment.yaml deployment.apps/devops6-npm-servicecreatedservice/devops6-npm-servicecreatedingress.networking.k8s.io/devops6-npm-servicecreated

注意：这里直接加上命名空间后，应用就会直接被部署到此命名空间了。

[root@Devops6 ~]#kubectl get po -ndevops6NAMEREADYSTATUSRESTARTSAGEdevops6-npm-service-bd4978ff9-27bpp1/1Running032sdevops6-npm-service-bd4978ff9-clkhm1/1Running032sdevops6-npm-service-bd4978ff9-x2sw81/1Running032s

配置ingress域名解析：

[root@Devops6 ~]#vim /etc/hosts172.29.9.31devops.test.com[root@Devops6 ~]#kubectl get ingress -ndevops6NAMECLASSHOSTSADDRESSPORTSAGEdevops6-npm-service<none>devops.test.com172.29.9.31808m11s

测试效果：

[root@Devops6 ~]#curl devops.test.com<!DOCTYPEhtml><html><head><metacharset="utf-8"><metaname="viewport"content="width=device-width,initial-scale=1.0"><title>vuedemo</title></head><body><divid="app"><h1>VERSION:main</h1></div><!--builtfileswillbeautoinjected--></body></html>

==新建devops6-npm-service版本分支，特性分支==

• 以main为基础新建devops6-npm-service版本分支RELEASE-1.1.1

修改index.html的内容为RELEASE-1.1.1。

• 运行一次CI流水线

要记得改下这里的jenkinsfile文件：

成功生成镜像：

• 我们再来更新一个版本的应用程序看下

[root@Devops6 ~]#vim Deployment.yaml将-image:172.29.9.120/devops6/devops6-npm-service:main-ed12ce10替换为172.29.9.120/devops6/devops6-npm-service:RELEASE-1.1.1-7d906f68#然后部署应用：[root@Devops6 ~]#kubectl apply -f Deployment.yaml -ndevops6#验证[root@Devops6 ~]#curl devops.test.com<!DOCTYPEhtml><html><head><metacharset="utf-8"><metaname="viewport"content="width=device-width,initial-scale=1.0"><title>vuedemo</title></head><body><divid="app"><h1>VERSION:RELEASE-1.1.1</h1></div><!--builtfileswillbeautoinjected--></body></html>#可以看到，已经更新成功了。

• 我们来测试下回滚

回滚命令：

## 查看历史kubectlrollouthistorydeployment/anyops-devopsdocker-ui## 查看具体某一个历史版本信息kubectlrollouthistorydeployment/anyops-devopsdocker-ui--revision=2## 回滚上个版本 kubectlrolloutundodeployment/anyops-devopsdocker-ui-nanyops## 回滚指定版本kubectlrolloutundodeployment/nginx--to-revision=2

查看当前应用版本：

[root@Devops6 ~]#kubectl rollout history deployment devops6-npm-service -ndevops6deployment.apps/devops6-npm-serviceREVISIONCHANGE-CAUSE1<none>2<none>#可以看到有2个历史版本

我们打算回滚到上个历史版本：

先来查看下当前应用版本：

watch-n1"curl devops.test.com"

watch-n1"curl -s devops.test.com"

开始回滚：

[root@Devops6 ~]#kubectl rollout undo deployment devops6-npm-service -ndevops6deployment.apps/devops6-npm-servicerolledback

回滚结果：

回滚成功。

• 这里调的是gitlab的api

调用gitlab api自动更新配置文件。

• jenkins装一个插件HTTP Request

• devops6-npm-service项目ProjectID为11。

• 创建devops6-npm-service目录

将Deployment.yaml里的image改为__IMAGE_NAME__

• pipeline代码

Gitlab.groovy

packageorg.devopsdefGetRepoFile(projectId,filePath,branchName){apiUrl ="/projects/${projectId}/repository/files/${filePath}/raw?ref=${branchName}"response =HttpReq('GET',apiUrl)returnresponse}defUpdateRepoFile(projectId,filePath,fileContent,branchName){apiUrl ="projects/${projectId}/repository/files/${filePath}"reqBody ="""{"branch":"${branchName}","encoding":"base64","content":"${fileContent}","commit_message":"update a new file"}"""response =HttpReqByPlugin('PUT',apiUrl,reqBody)println(response)}defCreateRepoFile(projectId,filePath,fileContent,branchName){apiUrl ="projects/${projectId}/repository/files/${filePath}"reqBody ="""{"branch":"${branchName}","encoding":"base64","content":"${fileContent}","commit_message":"update a new file"}"""response =HttpReqByPlugin('POST',apiUrl,reqBody)println(response)}defHttpReqByPlugin(reqType,reqUrl,reqBody){defgitServer ="http:withCredentials([string(credentialsId:'5782c77d-ce9d-44e5-b9ba-1ba2097fc31d',variable:'GITLABTOKEN')]) {response =httpRequest acceptType:'APPLICATION_JSON_UTF8',consoleLogResponseBody:true,contentType:'APPLICATION_JSON_UTF8',customHeaders:[[maskValue:false,name:'PRIVATE-TOKEN',value:"${GITLABTOKEN}"]],httpMode:"${reqType}",url:"${gitServer}/${reqUrl}",wrapAsMultipart:false,requestBody:"${reqBody}"}returnresponse}defHttpReq(method,apiUrl){withCredentials([string(credentialsId:'5782c77d-ce9d-44e5-b9ba-1ba2097fc31d',variable:'gitlabtoken')]) {response =sh returnStdout:true,script:"""curl --location --request ${method} \http:--header "PRIVATE-TOKEN:${gitlabtoken}""""}try{response =readJSON text:response -"\n"} catch(e){println(e)}returnresponse }defGetProjectIDByName(projectName,groupName){apiUrl ="projects?search=${projectName}"response =HttpReq("GET",apiUrl)if(response !=[]){for(p inresponse) {if(p["namespace"]["name"] ==groupName){returnresponse[0]["id"]}}}}defGetBranchCommitID(projectID,branchName){apiUrl ="projects/${projectID}/repository/branches/${branchName}"response =HttpReq("GET",apiUrl)returnresponse.commit.short_id}

k8sci.jenkinsfile

@Library("devops06@main") _defcheckout =neworg.devops.CheckOut()defbuild =neworg.devops.Build()defsonar =neworg.devops.Sonar()defmygit =neworg.devops.Gitlab()env.branchName ="${env.branchName}"-"origin/"println(env.branchName)pipeline {agent {label "build"}options {skipDefaultCheckout true} stages{stage("CheckOut"){steps{script{checkout.CheckOut()env.commitID =checkout.GetCommitID()println("commitID:${env.commitID}")currentBuild.displayName ="第${BUILD_NUMBER}次构建-${env.commitID}"currentBuild.description ="构建分支名称：${env.branchName}"}}}stage("Build"){steps{script{build.Build()}}} stage("CodeScan"){when {environment name:'skipSonar',value:'false'}steps{script{sonar.SonarScannerByPlugin() }}}stage("ImageBuild"){steps{script{appName ="${JOB_NAME}".split('_')[0] repoName =appName.split('-')[0] imageName ="${repoName}/${appName}"imageTag ="${env.branchName}-${env.commitID}"env.fullImageName ="172.29.9.120/${imageName}:${imageTag}"sh """#登录镜像仓库docker login -u admin -p Harbor12345 172.29.9.120# 构建镜像docker build -t ${env.fullImageName} .# 上传镜像docker push ${env.fullImageName}# 删除镜像sleep 2docker rmi ${env.fullImageName}"""}}}stage("UpdateEnvFile"){steps{script {projectId =11fileName ="Deployment.yaml"branchName ="main"fileData =mygit.GetRepoFile(projectId,fileName,branchName)sh "rm -fr ${fileName}"writeFile file:fileName ,text:fileDataenv.deployFile =fileNamesh "sed -i 's#__IMAGE_NAME__#${env.fullImageName}#g'${env.deployFile} "sh "ls -l ;cat ${fileName}"newYaml =sh returnStdout:true,script:"cat ${env.deployFile}"println(newYaml)base64Content =newYaml.bytes.encodeBase64().toString()appName ="${JOB_NAME}".split('_')[0] env.groupName =appName.split('-')[0] env.projectName =appNametry{mygit.UpdateRepoFile(projectId,"${env.projectName}%2f${env.branchName}.yaml",base64Content,"main")} catch(e){mygit.CreateRepoFile(projectId,"${env.projectName}%2f${env.branchName}.yaml",base64Content,"main")}}}} }}

• 再次新建版本分支RELEASE-2.1.1，jenkins里记得配置下该分支名。

• 运行

点击approve

符合预期。

【GitOps最重要的一个特性： ==自动更新部署文件==】

更新生成版本文件

• 步骤： 拿到 env仓库中的deployments.yaml模板文件， 然后替换内容，更新到版本库。

• 更新对象：

• • RELEASE_VERSION(镜像TAG)

• stage("UpdateEnvFile"){steps{script {projectId =11fileName ="Deployment.yaml"branchName ="main"fileData =mygit.GetRepoFile(projectId,fileName,branchName)sh "rm -fr ${fileName}"writeFile file:fileName ,text:fileDataenv.deployFile =fileNamesh "sed -i 's#__IMAGE_NAME__#${env.fullImageName}#g'${env.deployFile} "sh "ls -l ;cat ${fileName}"newYaml =sh returnStdout:true,script:"cat ${env.deployFile}"println(newYaml)base64Content =newYaml.bytes.encodeBase64().toString()appName ="${JOB_NAME}".split('_')[0] env.groupName =appName.split('-')[0] env.projectName =appNametry{mygit.UpdateRepoFile(projectId,"${env.projectName}%2f${env.branchName}.yaml",base64Content,"main")} catch(e){mygit.CreateRepoFile(projectId,"${env.projectName}%2f${env.branchName}.yaml",base64Content,"main")}}}}

• 更新后的版本文件

编写CD pipeline

• 创建一个CD的job devops6-npm-service_K8SCD

配置git仓库：

• 编写pipeline

k8scd.jenkinsfile文件

@Library("devops06@main") _ defmygit =neworg.devops.Gitlab()env.branchName ="${env.branchName}"-"origin/"println(env.branchName)pipeline {agent {label "build"}options {skipDefaultCheckout true}stages{stage("GetManifests"){steps{script{projectId =11env.deployFile ="${env.branchName}.yaml"env.appName ="${JOB_NAME}".split('_')[0] filePath ="${env.appName}%2f${env.deployFile}"branchName ="main"fileData =mygit.GetRepoFile(projectId,filePath,branchName)sh "rm -fr ${env.deployFile}"writeFile file:env.deployFile ,text:fileDatash "ls -l ;cat ${env.deployFile}"}}}stage("Deploy"){steps{script{env.namespace ="${env.appName}".split('-')[0] sh """## 发布应用kubectl apply -f ${env.deployFile} -n ${env.namespace}"""5.times{sh "sleep 2;kubectl -n ${env.namespace} get pod |grep ${env.appName}"}}}}stage("RollBack"){input {message "是否进行回滚"ok "提交"submitter ""parameters {choice(choices:['yes','no'],name:'opts')}}steps{script{switch("${opts}") {case"yes":sh "kubectl rollout undo deployment/${env.appName} -n ${env.namespace} "breakcase"no":break}}}}}}

• 运行流水线

我们来提前观察下此时版本：

符合预期。

完整代码

完整代码如下：

链接：https:jenkins:2.346.3-2-lts-jdk11sonarqube:9.9.0-communityharborv2.6.2