hugo-teek is loading...

2024-12-25 k8s +2 容器运行时 Containerd 4568 字 23 分钟 -- 次

实战-centos7上containerd的安装-v1-5-5-20211023-测试成功

最后更新于: 2025-12-06 17:39:04

实战：centos7上containerd的安装(测试成功)-2021.10.23

💘 实践：centos7上containerd的安装(成功测试)-2021.10.23

实验环境

1实验环境：
21、win10笔记本
32、1台centos7.6虚机(vmwrokstation虚机)
4	cri-containerd-cni-1.5.5-linux-amd64.tar.gz

实验软件

链接：https://pan.baidu.com/s/1pfn6G9GOnpTG7AE1NIDF_Q 提取码：h34l –来自百度网盘超级会员V6的分享

cri-containerd-cni-1.5.5-linux-amd64.tar.gz

1、安装libseccomp依赖包

这里我使用的系统是 CentOS 7.6，首先需要安装 libseccomp 依赖：

 1#先查看系统有没有libseccomp软件包，没的话就安装，一般需要安装
 2[root@containerd ~]#rpm -qa |grep libseccomp
 3[root@containerd ~]#
 4
 5#我们先来看一下libseccomp依赖版属于哪个软件包
 6[root@containerd ~]#yum search libseccomp -y
 7Loaded plugins: fastestmirror
 8Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
 9Determining fastest mirrors
10 * base: mirrors.aliyun.com
11 * extras: mirrors.aliyun.com
12 * updates: mirrors.aliyun.com
13================================================= N/S matched: libseccomp ==================================================
14libseccomp-devel.i686 : Development files used to build applications with libseccomp support
15libseccomp-devel.x86_64 : Development files used to build applications with libseccomp support
16libseccomp.i686 : Enhanced seccomp library
17libseccomp.x86_64 : Enhanced seccomp library
18
19  Name and summary matches only, use "search all" for everything.
20[root@containerd ~]#
21
22#yum安装libseccomp依赖包
23[root@containerd ~]#yum install libseccomp -y
24
25#验证是否安装
26[root@containerd ~]#rpm -qa |grep libseccomp
27libseccomp-2.3.1-4.el7.x86_64
28[root@containerd ~]#

2、下载containerd软件包并解压

由于 containerd 需要调用 runc，所以我们也需要先安装 runc，不过 containerd 提供了一个包含相关依赖的压缩包 cri-containerd-cni-${VERSION}.${OS}-${ARCH}.tar.gz，可以直接使用这个包来进行安装。首先从 release 页面下载最新版本的压缩包，当前为 1.5.5 版本（注意：最新的1.5.7版本在CentOS7下面执行 runc 会报错：

该issue：

https://github.com/containerd/containerd

1[root@containerd ~]#wget https://github.com/containerd/containerd/releases/download/v1.5.5/cri-containerd-cni-1.5.5-linux-amd64.tar.gz
2# 如果有限制，也可以替换成下面的 URL 加速下载
3wget https://download.fastgit.org/containerd/containerd/releases/download/v1.5.5/cri-containerd-cni-1.5.5-linux-amd64.tar.gz

加速下载还是挺快的：

可以通过 tar 的 -t 选项直接看到压缩包中包含哪些文件：

 1[root@containerd ~]#tar -tf cri-containerd-cni-1.5.5-linux-amd64.tar.gz
 2etc/
 3etc/crictl.yaml
 4etc/cni/
 5etc/cni/net.d/
 6etc/cni/net.d/10-containerd-net.conflist
 7etc/systemd/
 8etc/systemd/system/
 9etc/systemd/system/containerd.service
10usr/
11usr/local/
12usr/local/sbin/
13usr/local/sbin/runc
14usr/local/bin/
15usr/local/bin/containerd-shim-runc-v1
16usr/local/bin/critest
17usr/local/bin/ctr
18usr/local/bin/containerd-shim
19usr/local/bin/crictl
20usr/local/bin/containerd
21usr/local/bin/ctd-decoder
22usr/local/bin/containerd-stress
23usr/local/bin/containerd-shim-runc-v2
24opt/
25opt/cni/
26opt/cni/bin/
27opt/cni/bin/firewall
28opt/cni/bin/macvlan
29opt/cni/bin/vrf
30opt/cni/bin/ipvlan
31opt/cni/bin/dhcp
32opt/cni/bin/flannel
33opt/cni/bin/bandwidth
34opt/cni/bin/portmap
35opt/cni/bin/ptp
36opt/cni/bin/host-local
37opt/cni/bin/sbr
38opt/cni/bin/loopback
39opt/cni/bin/static
40opt/cni/bin/host-device
41opt/cni/bin/tuning
42opt/cni/bin/bridge
43opt/cni/bin/vlan
44opt/containerd/
45opt/containerd/cluster/
46opt/containerd/cluster/gce/
47opt/containerd/cluster/gce/env
48opt/containerd/cluster/gce/configure.sh
49opt/containerd/cluster/gce/cloud-init/
50opt/containerd/cluster/gce/cloud-init/master.yaml
51opt/containerd/cluster/gce/cloud-init/node.yaml
52opt/containerd/cluster/gce/cni.template
53opt/containerd/cluster/version
54[root@containerd ~]#

直接将压缩包解压到系统的各个目录中：

1[root@containerd ~]#tar -C / -xzf cri-containerd-cni-1.5.5-linux-amd64.tar.gz

当然要记得将 /usr/local/bin 和 /usr/local/sbin 追加到 ~/.bashrc 文件的 PATH 环境变量中：

1[root@containerd ~]#vim ~/.bashrc
2export PATH=$PATH:/usr/local/bin:/usr/local/sbin #添加这行

然后执行下面的命令使其立即生效：

1[root@containerd ~]#source ~/.bashrc

注意：centos7默认的PAH就已经包含了 /usr/local/bin 和 /usr/local/sbin 路径，但老师这里还是手动添加了这2个路径，我这里就继续添加下吧。

3、生成containerd 的默认配置文件config.toml

containerd 的默认配置文件为 /etc/containerd/config.toml，我们可以通过如下所示的命令生成一个默认的配置：

1[root@containerd ~]#mkdir -p /etc/containerd
2[root@containerd ~]#containerd config default > /etc/containerd/config.toml

由于上面我们下载的 containerd 压缩包中包含一个 etc/systemd/system/containerd.service 的文件，这样我们就可以通过 systemd 来配置 containerd 作为守护进程运行了，内容如下所示：

 1[root@containerd ~]#ll /etc/systemd/system/containerd.service
 2-rw-r--r-- 1 1001 116 1270 Jul 30 01:12 /etc/systemd/system/containerd.service
 3[root@containerd ~]#cat !$
 4cat /etc/systemd/system/containerd.service
 5# Copyright The containerd Authors.
 6#
 7# Licensed under the Apache License, Version 2.0 (the "License");
 8# you may not use this file except in compliance with the License.
 9# You may obtain a copy of the License at
10#
11#     http://www.apache.org/licenses/LICENSE-2.0
12#
13# Unless required by applicable law or agreed to in writing, software
14# distributed under the License is distributed on an "AS IS" BASIS,
15# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16# See the License for the specific language governing permissions and
17# limitations under the License.
18
19[Unit]
20Description=containerd container runtime
21Documentation=https://containerd.io
22After=network.target local-fs.target
23
24[Service]
25ExecStartPre=-/sbin/modprobe overlay
26ExecStart=/usr/local/bin/containerd
27
28Type=notify
29Delegate=yes
30KillMode=process
31Restart=always
32RestartSec=5
33# Having non-zero Limit*s causes performance problems due to accounting overhead
34# in the kernel. We recommend using cgroups to do container-local accounting.
35LimitNPROC=infinity
36LimitCORE=infinity
37LimitNOFILE=infinity
38# Comment TasksMax if your systemd version does not supports it.
39# Only systemd 226 and above support this version.
40TasksMax=infinity
41OOMScoreAdjust=-999
42
43[Install]
44WantedBy=multi-user.target
45[root@containerd ~]#

这里有两个重要的参数：

Delegate: 这个选项允许 containerd 以及运行时自己管理自己创建容器的 cgroups。如果不设置这个选项，systemd 就会将进程移到自己的 cgroups 中，从而导致 containerd 无法正确获取容器的资源使用情况。
KillMode: 这个选项用来处理 containerd 进程被杀死的方式。**默认情况下，systemd 会在进程的 cgroup 中查找并杀死 containerd 的所有子进程。**KillMode 字段可以设置的值如下。
- control-group（默认值）：当前控制组里面的所有子进程，都会被杀掉
- process：只杀主进程
- mixed：主进程将收到 SIGTERM 信号，子进程收到 SIGKILL 信号
- none：没有进程会被杀掉，只是执行服务的 stop 命令

我们需要将 KillMode 的值设置为 process，这样可以确保升级或重启 containerd 时不杀死现有的容器。

注意：这里自己对这个KillMode参数的说明比较模糊，后期注意下。

4、追加1：配置镜像加速器地址

我们首先来查看下上面默认生成的配置文件 /etc/containerd/config.toml：

  1[root@containerd ~]#cat /etc/containerd/config.toml
  2disabled_plugins = []
  3imports = []
  4oom_score = 0
  5plugin_dir = ""
  6required_plugins = []
  7root = "/var/lib/containerd"
  8state = "/run/containerd"
  9version = 2
 10
 11[cgroup]
 12  path = ""
 13
 14[debug]
 15  address = ""
 16  format = ""
 17  gid = 0
 18  level = ""
 19  uid = 0
 20
 21[grpc]
 22  address = "/run/containerd/containerd.sock"
 23  gid = 0
 24  max_recv_message_size = 16777216
 25  max_send_message_size = 16777216
 26  tcp_address = ""
 27  tcp_tls_cert = ""
 28  tcp_tls_key = ""
 29  uid = 0
 30
 31[metrics]
 32  address = ""
 33  grpc_histogram = false
 34
 35[plugins]
 36
 37  [plugins."io.containerd.gc.v1.scheduler"]
 38    deletion_threshold = 0
 39    mutation_threshold = 100
 40    pause_threshold = 0.02
 41    schedule_delay = "0s"
 42    startup_delay = "100ms"
 43
 44  [plugins."io.containerd.grpc.v1.cri"]
 45    disable_apparmor = false
 46    disable_cgroup = false
 47    disable_hugetlb_controller = true
 48    disable_proc_mount = false
 49    disable_tcp_service = true
 50    enable_selinux = false
 51    enable_tls_streaming = false
 52    ignore_image_defined_volumes = false
 53    max_concurrent_downloads = 3
 54    max_container_log_line_size = 16384
 55    netns_mounts_under_state_dir = false
 56    restrict_oom_score_adj = false
 57    sandbox_image = "k8s.gcr.io/pause:3.5"
 58    selinux_category_range = 1024
 59    stats_collect_period = 10
 60    stream_idle_timeout = "4h0m0s"
 61    stream_server_address = "127.0.0.1"
 62    stream_server_port = "0"
 63    systemd_cgroup = false
 64    tolerate_missing_hugetlb_controller = true
 65    unset_seccomp_profile = ""
 66
 67    [plugins."io.containerd.grpc.v1.cri".cni]
 68      bin_dir = "/opt/cni/bin"
 69      conf_dir = "/etc/cni/net.d"
 70      conf_template = ""
 71      max_conf_num = 1
 72
 73    [plugins."io.containerd.grpc.v1.cri".containerd]
 74      default_runtime_name = "runc"
 75      disable_snapshot_annotations = true
 76      discard_unpacked_layers = false
 77      no_pivot = false
 78      snapshotter = "overlayfs"
 79
 80      [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
 81        base_runtime_spec = ""
 82        container_annotations = []
 83        pod_annotations = []
 84        privileged_without_host_devices = false
 85        runtime_engine = ""
 86        runtime_root = ""
 87        runtime_type = ""
 88
 89        [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]
 90
 91      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
 92
 93        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
 94          base_runtime_spec = ""
 95          container_annotations = []
 96          pod_annotations = []
 97          privileged_without_host_devices = false
 98          runtime_engine = ""
 99          runtime_root = ""
100          runtime_type = "io.containerd.runc.v2"
101
102          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
103            BinaryName = ""
104            CriuImagePath = ""
105            CriuPath = ""
106            CriuWorkPath = ""
107            IoGid = 0
108            IoUid = 0
109            NoNewKeyring = false
110            NoPivotRoot = false
111            Root = ""
112            ShimCgroup = ""
113            SystemdCgroup = false
114
115      [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
116        base_runtime_spec = ""
117        container_annotations = []
118        pod_annotations = []
119        privileged_without_host_devices = false
120        runtime_engine = ""
121        runtime_root = ""
122        runtime_type = ""
123
124        [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]
125
126    [plugins."io.containerd.grpc.v1.cri".image_decryption]
127      key_model = "node"
128
129    [plugins."io.containerd.grpc.v1.cri".registry]
130      config_path = ""
131
132      [plugins."io.containerd.grpc.v1.cri".registry.auths]
133
134      [plugins."io.containerd.grpc.v1.cri".registry.configs]
135
136      [plugins."io.containerd.grpc.v1.cri".registry.headers]
137
138      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
139
140    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
141      tls_cert_file = ""
142      tls_key_file = ""
143
144  [plugins."io.containerd.internal.v1.opt"]
145    path = "/opt/containerd"
146
147  [plugins."io.containerd.internal.v1.restart"]
148    interval = "10s"
149
150  [plugins."io.containerd.metadata.v1.bolt"]
151    content_sharing_policy = "shared"
152
153  [plugins."io.containerd.monitor.v1.cgroups"]
154    no_prometheus = false
155
156  [plugins."io.containerd.runtime.v1.linux"]
157    no_shim = false
158    runtime = "runc"
159    runtime_root = ""
160    shim = "containerd-shim"
161    shim_debug = false
162
163  [plugins."io.containerd.runtime.v2.task"]
164    platforms = ["linux/amd64"]
165
166  [plugins."io.containerd.service.v1.diff-service"]
167    default = ["walking"]
168
169  [plugins."io.containerd.snapshotter.v1.aufs"]
170    root_path = ""
171
172  [plugins."io.containerd.snapshotter.v1.btrfs"]
173    root_path = ""
174
175  [plugins."io.containerd.snapshotter.v1.devmapper"]
176    async_remove = false
177    base_image_size = ""
178    pool_name = ""
179    root_path = ""
180
181  [plugins."io.containerd.snapshotter.v1.native"]
182    root_path = ""
183
184  [plugins."io.containerd.snapshotter.v1.overlayfs"]
185    root_path = ""
186
187  [plugins."io.containerd.snapshotter.v1.zfs"]
188    root_path = ""
189
190[proxy_plugins]
191
192[stream_processors]
193
194  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
195    accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
196    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
197    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
198    path = "ctd-decoder"
199    returns = "application/vnd.oci.image.layer.v1.tar"
200
201  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
202    accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
203    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
204    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
205    path = "ctd-decoder"
206    returns = "application/vnd.oci.image.layer.v1.tar+gzip"
207
208[timeouts]
209  "io.containerd.timeout.shim.cleanup" = "5s"
210  "io.containerd.timeout.shim.load" = "5s"
211  "io.containerd.timeout.shim.shutdown" = "3s"
212  "io.containerd.timeout.task.state" = "2s"
213
214[ttrpc]
215  address = ""
216  gid = 0
217  uid = 0
218[root@containerd ~]#

这个配置文件比较复杂，我们可以将重点放在其中的 plugins 配置上面，仔细观察我们可以发现每一个顶级配置块的命名都是 plugins."io.containerd.xxx.vx.xxx" 这种形式，每一个顶级配置块都表示一个插件，其中 io.containerd.xxx.vx 表示插件的类型，vx 后面的 xxx 表示插件的 ID，我们可以通过 ctr 查看插件列表：

 1[root@containerd ~]#ctr plugin ls
 2TYPE                            ID                       PLATFORMS      STATUS
 3io.containerd.content.v1        content                  -              ok
 4io.containerd.snapshotter.v1    aufs                     linux/amd64    skip
 5io.containerd.snapshotter.v1    btrfs                    linux/amd64    skip
 6io.containerd.snapshotter.v1    devmapper                linux/amd64    error
 7io.containerd.snapshotter.v1    native                   linux/amd64    ok
 8io.containerd.snapshotter.v1    overlayfs                linux/amd64    ok
 9io.containerd.snapshotter.v1    zfs                      linux/amd64    skip
10io.containerd.metadata.v1       bolt                     -              ok
11io.containerd.differ.v1         walking                  linux/amd64    ok
12io.containerd.gc.v1             scheduler                -              ok
13io.containerd.service.v1        introspection-service    -              ok
14io.containerd.service.v1        containers-service       -              ok
15io.containerd.service.v1        content-service          -              ok
16io.containerd.service.v1        diff-service             -              ok
17io.containerd.service.v1        images-service           -              ok
18io.containerd.service.v1        leases-service           -              ok
19io.containerd.service.v1        namespaces-service       -              ok
20io.containerd.service.v1        snapshots-service        -              ok
21io.containerd.runtime.v1        linux                    linux/amd64    ok
22io.containerd.runtime.v2        task                     linux/amd64    ok
23io.containerd.monitor.v1        cgroups                  linux/amd64    ok
24io.containerd.service.v1        tasks-service            -              ok
25io.containerd.internal.v1       restart                  -              ok
26io.containerd.grpc.v1           containers               -              ok
27io.containerd.grpc.v1           content                  -              ok
28io.containerd.grpc.v1           diff                     -              ok
29io.containerd.grpc.v1           events                   -              ok
30io.containerd.grpc.v1           healthcheck              -              ok
31io.containerd.grpc.v1           images                   -              ok
32io.containerd.grpc.v1           leases                   -              ok
33io.containerd.grpc.v1           namespaces               -              ok
34io.containerd.internal.v1       opt                      -              ok
35io.containerd.grpc.v1           snapshots                -              ok
36io.containerd.grpc.v1           tasks                    -              ok
37io.containerd.grpc.v1           version                  -              ok
38io.containerd.grpc.v1           cri                      linux/amd64    ok
39[root@containerd ~]#

顶级配置块下面的子配置块表示该插件的各种配置，比如 cri 插件下面就分为 containerd、cni 和 registry 的配置，而 containerd 下面又可以配置各种 runtime，还可以配置默认的 runtime。比如现在我们要为镜像配置一个加速器，那么就需要在 cri 配置块下面的 registry 配置块下面进行配置 registry.mirrors：

注意缩进：

1[root@containerd ~]#vim /etc/containerd/config.toml
2[plugins."io.containerd.grpc.v1.cri".registry]
3  [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
4    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
5      endpoint = ["https://kvuwuws2.mirror.aliyuncs.com"]
6    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."k8s.gcr.io"]
7      endpoint = ["https://registry.aliyuncs.com/k8sxio"]

registry.mirrors."xxx": 表示需要配置 mirror 的镜像仓库，例如 registry.mirrors."docker.io" 表示配置 docker.io 的 mirror。
endpoint: 表示提供 mirror 的镜像加速服务，比如我们可以注册一个阿里云的镜像服务来作为 docker.io 的 mirror。

🍀 注意：contanerd镜像加速

🍀 另外在默认配置中还有两个关于存储的配置路径：

1root = "/var/lib/containerd"
2state = "/run/containerd"

其中 root 是用来保存持久化数据，包括 Snapshots, Content, Metadata 以及各种插件的数据，每一个插件都有自己单独的目录，Containerd 本身不存储任何数据，它的所有功能都来自于已加载的插件。

而另外的 state 是用来保存运行时的临时数据的，包括 sockets、pid、挂载点、运行时状态以及不需要持久化的插件数据。

到此，关于containerd安装结束！

5、追加2：将 containerd 的 cgroup driver 配置为 systemd

对于使用 systemd 作为 init system 的 Linux 的发行版，使用 systemd 作为容器的 cgroup driver 可以确保节点在资源紧张的情况更加稳定，所以推荐将 containerd 的 cgroup driver 配置为 systemd。

修改方法：

修改前面生成的配置文件 /etc/containerd/config.toml，在 plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options 配置块下面将 SystemdCgroup 设置为 true：

 1#通过搜索SystemdCgroup进行定位
 2vim /etc/containerd/config.toml
 3[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
 4  ...
 5  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
 6    SystemdCgroup = true
 7    ....
 8
 9#注意：最终输出shell命令：
10sed -i "s/SystemdCgroup = false/SystemdCgroup = true/g" /etc/containerd/config.toml

备注：

如果 linux 系统使用 systemd 初始化系统，初始化进程会生成一个 root cgroup，每个 systemd unit 都将会被分配一个cgroup，同样可以配置容器运行时如 containerd 选择使用 cgroupfs 或 systemd 作为 cgroup 驱动，containerd 默认使用的是 cgroupfs，但对于使用了 systemd 的 linux 发行版来说就同时存在两个 cgroup 管理器，对于该服务器上启动的容器使用的是cgroupfs，而对于其他 systemd 管理的进程使用的是 systemd，这样在服务器资源负载高的情况下可能会变的不稳定。因此对于使用了 systemd 的 linux 系统，推荐将容器运行时的 cgroup 驱动使用 systemd。

6、启动containerd服务

现在我们就可以启动 containerd 了，直接执行下面的命令即可：

1[root@containerd ~]#systemctl daemon-reload
2[root@containerd ~]#systemctl restart containerd

7、验证

启动完成后就可以使用 containerd 的本地 CLI 工具 ctr 和 crictl 了，比如查看版本：

1➜  ~ ctr version
2➜  ~ crictl version

FAQ

注意：libseccomp-devel.x86_64和libseccomp.x86_64 2个软件包有什么区别？

01.老师亲自实战安装的是libseccomp-devel.x86_64软件，但文档上提供的是libseccomp.x86_64软件，2者实验都是没问题的； 02.自己最后建议按文档上的来吧，自己也是按文档上的来吧，也就是说使用yum install -y libseccomp.x86_64即可！
[root@containerd ~]#yum search libseccomp Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile
base: mirrors.aliyun.com
extras: mirrors.aliyun.com
updates: mirrors.aliyun.com ======================================================================== N/S matched: libseccomp ======================================================================== libseccomp-devel.i686 : Development files used to build applications with libseccomp support libseccomp-devel.x86_64 : Development files used to build applications with libseccomp support libseccomp.i686 : Enhanced seccomp library libseccomp.x86_64 : Enhanced seccomp library

Name and summary matches only, use “search all” for everything. [root@containerd ~]#

注意：如果有配置config.toml文件，一定要记得先reload一下配置文件，再重启服务

1#配置完成后记得重启下containerd
2systemctl daemon-reload
3systemctl restart containerd

注意：containerd的k8s集群，pod跨节点请求出现了点问题

master1上无法访问其他节点pod，但在pod所在节点上可以访问：

 1[root@node2 ~]#curl http://10.244.2.5
 2<!DOCTYPE html>
 3<html>
 4<head>
 5<title>Welcome to nginx!</title>
 6<style>
 7html { color-scheme: light dark; }
 8body { width: 35em; margin: 0 auto;
 9font-family: Tahoma, Verdana, Arial, sans-serif; }
10</style>
11</head>
12<body>
13<h1>Welcome to nginx!</h1>
14<p>If you see this page, the nginx web server is successfully installed and
15working. Further configuration is required.</p>
16
17<p>For online documentation and support please refer to
18<a href="http://nginx.org/">nginx.org</a>.<br/>
19Commercial support is available at
20<a href="http://nginx.com/">nginx.com</a>.</p>
21
22<p><em>Thank you for using nginx.</em></p>
23</body>
24</html>

这个问题好无语啊。。。。感觉像是flannel插件哪里出问题了。。。。。

跨节点也是ping测试失败的。。。。

master节点上没有cni0，以及flannel.1地址：

我重建下allnode的flannel pod，再尝试一次：

重建flannel pod后发现master1上出现了flannel.1地址：

此时再次测试，观看效果：

ping测试，跨节点也是没有任何问题的；

curl测试nginx也是没问题的：

结论：

注意：老师重建flannel网络命令
ifconfig cni0 down && ip link delete cni0 ifconfig flannel.1 down && ip link delete flannel.1

输出脚本

2021-10-31更新脚本：

 1cd /root/
 2yum install libseccomp -y
 3
 4wget https://download.fastgit.org/containerd/containerd/releases/download/v1.5.5/cri-containerd-cni-1.5.5-linux-amd64.tar.gz
 5tar -C / -xzf cri-containerd-cni-1.5.5-linux-amd64.tar.gz
 6
 7echo "export PATH=$PATH:/usr/local/bin:/usr/local/sbin" >> ~/.bashrc
 8source ~/.bashrc
 9
10mkdir -p /etc/containerd
11containerd config default > /etc/containerd/config.toml
12
13sed -i "s/SystemdCgroup = false/SystemdCgroup = true/g" /etc/containerd/config.toml
14
15# 将 containerd 的 cgroup driver 配置为 systemd，需要手动配置！
16
17systemctl daemon-reload
18systemctl enable containerd --now
19
20cri version
21crictl version