kube-benech
kube-beach
CIS 安全基准
• CIS安全基准介绍
• K8s安全基准工具 kube-bench
CIS安全基准
互联网安全中心（CIS，Center for Internet Security），是一个非盈利组织，致力为互联网提供免费的安全防御解决方案。
3、测试
bash
[root@k8s-master1 ~]#kube-bench --help
部署完成。😘
CIS基准测试工具：kube-beach使用
使用kube-bench run进行测试，该指令有以下常用参数：
常用参数：
-s,--targets 指定要基础测试的目标，这个目标需要匹配cfg/<version>中的文件名称，已有目标：master,controlplane,node,etcd,policies
• --version：指定k8s版本，如果未指定会自动检测
• --benchmark：手动指定CIS基准版本，不能与--version一起使用
ls /etc/kube-bench/cfg/
目前通常使用的是：1.6.0版本。
ls /etc/kube-bench/cfg/cis-1.6/
案例：检查master组件安全配置
kube-bench run --targets=master
执行后会逐个检查安全配置并输出修复方案及汇总信息输出：
[PASS]：测试通过
[FAIL]：测试未通过，重点关注，在测试结果会给出修复建议
[WARN]：警告，可做了解
[INFO]：信息
🍀 测试过程：
bash
[root@k8s-master1 ~]#kube-bench run --targets=masterCIS基准测试：
修复建议：
汇总信息：
注意：这个检查项就是按这个CIS_Kubernetes_Benchmark_v1.6.0.pdf上来做的配置。
🍀 找出FAILED的配置项：
bash
[FAIL] 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated)[FAIL] 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) #不太熟，忽略掉。[FAIL] 1.2.16 Ensure that the admission control plugin PodSecurityPolicy is set (Automated) #后面会单独讲解。[FAIL] 1.2.21 Ensure that the --profiling argument is set to false (Automated) #pprod 性能分析程序，go语言。默认是打开的。#审计日志[FAIL] 1.2.22 Ensure that the --audit-log-path argument is set (Automated)[FAIL] 1.2.23 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated)[FAIL] 1.2.24 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated)[FAIL] 1.2.25 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated)[FAIL] 1.3.2 Ensure that the --profiling argument is set to false (Automated)[FAIL] 1.4.1 Ensure that the --profiling argument is set to false (Automated)- 检查项
因为当前k8s环境的etcd是通过静态pod启动起来的，因此这个检查项可忽略。
[FAIL] 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated)
- 进一步测试
bash
[FAIL] 1.2.21 Ensure that the --profiling argument is set to false (Automated) #pprod 性能分析程序，go语言。默认是打开的。#建议配置如下1.2.21EdittheAPIserverpodspecificationfile/etc/kubernetes/manifests/kube-apiserver.yamlonthemasternodeandsetthebelowparameter.--profiling=false开始配置：
bash
[root@k8s-master1 ~]#vim /etc/kubernetes/manifests/kube-apiserver.yaml
配置完/etc/kubernetes/manifests/kube-apiserver.yaml配置文件后，可以看到kube-apiserver-k8s-master1pod发生了重启，此时kubelet也是活动的。（这里是不用重启kublet服务的）
此时，我们再次运行下kube-bench run --targets=master命令，观察现象：
可以看到，原来的1.2.21选项此时的检查结果为PASS了，符合预期。
配置完成。😘
例如：检查node组件安全配置
检查node时，会重点检查这个kubelet。
bash
[root@k8s-master1 ~]#kube-bench run --targets=node[INFO] 4 Worker Node Security Configuration[INFO] 4.1 Worker Node Configuration Files[PASS] 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated)[PASS] 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated)[PASS] 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Manual)[PASS] 4.1.4 Ensure that the proxy kubeconfig file ownership is set to root:root (Manual)[PASS] 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated)[PASS] 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Manual)[PASS] 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Manual)[PASS] 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual)[PASS] 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated)[PASS] 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated)[INFO] 4.2 Kubelet[PASS] 4.2.1 Ensure that the anonymous-auth argument is set to false (Automated)[PASS] 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated)[PASS] 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated)[PASS] 4.2.4 Ensure that the --read-only-port argument is set to 0 (Manual)[PASS] 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual)[FAIL] 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated)[PASS] 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated)[PASS] 4.2.8 Ensure that the --hostname-override argument is not set (Manual)[WARN] 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual)[WARN] 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual)[PASS] 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Manual)[PASS] 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual)[WARN] 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual)==Remediationsnode==4.2.6IfusingaKubeletconfigfile,editthefiletosetprotectKernelDefaults:true.Ifusingcommandlinearguments,editthekubeletservicefile/lib/systemd/system/kubelet.serviceoneachworkernodeandsetthebelowparameterinKUBELET_SYSTEM_PODS_ARGSvariable.--protect-kernel-defaults=trueBasedonyoursystem,restartthekubeletservice.Forexample:systemctldaemon-reloadsystemctlrestartkubelet.service4.2.9IfusingaKubeletconfigfile,editthefiletoseteventRecordQPS:toanappropriatelevel.Ifusingcommandlinearguments,editthekubeletservicefile/lib/systemd/system/kubelet.serviceoneachworkernodeandsetthebelowparameterinKUBELET_SYSTEM_PODS_ARGSvariable.Basedonyoursystem,restartthekubeletservice.Forexample:systemctldaemon-reloadsystemctlrestartkubelet.service4.2.10IfusingaKubeletconfigfile,editthefiletosettlsCertFiletothelocationofthecertificatefiletousetoidentifythisKubelet,andtlsPrivateKeyFiletothelocationofthecorrespondingprivatekeyfile.Ifusingcommandlinearguments,editthekubeletservicefile/lib/systemd/system/kubelet.serviceoneachworkernodeandsetthebelowparametersinKUBELET_CERTIFICATE_ARGSvariable.--tls-cert-file=<path/to/tls-certificate-file>--tls-private-key-file=<path/to/tls-key-file>Basedonyoursystem,restartthekubeletservice.Forexample:systemctldaemon-reloadsystemctlrestartkubelet.service4.2.13IfusingaKubeletconfigfile,editthefiletosetTLSCipherSuites:toTLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256ortoasubsetofthesevalues.Ifusingexecutablearguments,editthekubeletservicefile/lib/systemd/system/kubelet.serviceoneachworkernodeandsetthe--tls-cipher-suitesparameterasfollows,ortoasubsetofthesevalues.--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256Basedonyoursystem,restartthekubeletservice.Forexample:systemctldaemon-reloadsystemctlrestartkubelet.service==Summarynode==19checksPASS1checksFAIL3checksWARN0checksINFO==Summarytotal==19checksPASS1checksFAIL3checksWARN0checksINFO[root@k8s-master1 ~]#
例如：检查etcd组件安全配置
案例：如何跳过检查项
测试项目配置文件：/etc/kube-bench/cfg/cis-1.6/
bash
•id：编号•text：提示的文本•audit：•tests：测试项目•remediation：修复方案•scored：如果为true，kube-bench无法正常测试，则会生成FAIL，如果为false，无法正常测试，则会生成WARN。#如果设置为false，则会生成WARN。•type：如果为manual则会生成WARN，如果为skip，则会生成INFO案例测试：
bash
[root@k8s-master1 ~]#vim /etc/kube-bench/cfg/cis-1.6/master.yaml
kube-bench run --targets=mster的检测未通过项。
bash
[FAIL] 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated)[FAIL] 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) #不太熟，忽略掉。[FAIL] 1.2.16 Ensure that the admission control plugin PodSecurityPolicy is set (Automated) #后面会单独讲解。[FAIL] 1.2.21 Ensure that the --profiling argument is set to false (Automated) #pprod 性能分析程序，go语言。默认是打开的。#审计日志[FAIL] 1.2.22 Ensure that the --audit-log-path argument is set (Automated)[FAIL] 1.2.23 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated)[FAIL] 1.2.24 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated)[FAIL] 1.2.25 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated)[FAIL] 1.3.2 Ensure that the --profiling argument is set to false (Automated)[FAIL] 1.4.1 Ensure that the --profiling argument is set to false (Automated)我们可以看一下官网：
kube-controller-manager和kube-apiserver的这个profiling功能还没弃用，但是kube-scheduler的这个profiling功能已经被弃用了；
但是这个kube-bench这个脚本的检测程序还在，说明kube-bench并没有及时地区更新；
此时，想要把kube-scheduler组件的profiling检查项给忽略掉，该怎么办呢？
因此需要用到如下选项：type：如果为manual则会生成WARN，如果为skip，则会生成INFO
bash
[root@k8s-master1 ~]#vim /etc/kube-bench/cfg/cis-1.6/master.yaml
再次执行kube-bench run --targets=master，可以看到1.4.1检查项此时为INFO了；
测试结束。😘
注意事项
bash
[FAIL] 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated)[FAIL] 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) #不太熟，忽略掉。[FAIL] 1.2.16 Ensure that the admission control plugin PodSecurityPolicy is set (Automated) #后面会单独讲解。PodSecurityPolicy这个本来就是个独立的功能，可用可不用；[FAIL] 1.2.21 Ensure that the --profiling argument is set to false (Automated) #pprod 性能分析程序，go语言。默认是打开的。#审计日志[FAIL] 1.2.22 Ensure that the --audit-log-path argument is set (Automated)[FAIL] 1.2.23 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated)[FAIL] 1.2.24 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated)[FAIL] 1.2.25 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated)[FAIL] 1.3.2 Ensure that the --profiling argument is set to false (Automated)[FAIL] 1.4.1 Ensure that the --profiling argument is set to false (Automated)一般在master上是可以检查node的；（当然直接在node节点安装kube-bench跑程序也是可以的）
例如有3个master节点，只需要在1个master节点执行检查项就行，然后在all master节点配置相同的加固箱就好；（同样，node加固项也是需要做相同的操作）
可以以json格式输出的：
kube-bench run --help，--json Prints the results as JSON。
关于我
我的博客主旨：
- 排版美观，语言精炼；
- 文档即手册，步骤明细，拒绝埋坑，提供源码；
- 本人实战文档都是亲测成功的，各位小伙伴在实际操作过程中如有什么疑问，可随时联系本人帮您解决问题，让我们一起进步！
🍀 微信二维码 x2675263825 （舍得）， qq：2675263825。
🍀 微信公众号 《云原生架构师实战》
🍀 个人博客站点