手动在kubernetes集群中添加minon

Tech Comments

背景

去年年中我使用ansible给项目部署了一个小型的kubernetes集群，部署源码可以在这里找到:contrib。随着业务的发展，目前的集群规模很有可能在明年迎来瓶颈，因此需要 扩容.当然可以继续通过之前的contrib去部署，不过有几点担忧：

  1. 目前的系统稳定地提供着服务，ansible脚本可能会对集群产生不可逆的影响
  2. 对于contrib的代码不是特别熟悉，不清楚它具体做了哪些步骤，总体来说还是不放心

同时，作为一个system admin，对于集群必须了如指掌才行。因此，在这里做一次演练，手动为已有集群添加节点，增进了解，也为今年的扩容做好准备工作。

Let’s do it

0.已有环境

当前系统里有2台机器:

  1. node-1-master
    master上运行着 apiserver, control-manager, scheduler, kubelet, kube-proxy
  2. node-2-slave-1
    slave1上运行着 kubelet, kube-proxy

我们需要增加一台node-3-slave-2增加集群的运算能力。
备注: 目前我们所有的kubernetes机器全都是基于CentOS 7及以上，kubernetes的版本为1.3

1.申请新机器

编辑/etc/hosts，在其中增加master和本机的dns条目(集群中都是以hostname来标示每一台机器，比如https://node-1-master，这些dns条目在互联网上是无法解析的，必须在每台机器的 /etc/hosts 配置)。

1
2
xxx.xxx.xxx.xxx node-1-master
xxx.xxx.xxx.xxx node-3-slave-2

2.安装docker

1
2
yum install docker
systemctl enable docker # 开机自动启动

3.网络配置

3.1 安装firewalld
1
yum install firewalld

如果已经装好了，这步跳过，判断是否安装可以使用rpm -q firewalld检查。

3.2 配置kube-proxy网络环境
1
2
3
4
5
6
7
firewall-cmd --zone=public --add-port=8472/udp --permanent
firewall-cmd --zone=public --add-port=10250/tcp --permanent

firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 1 -i flannel.1 -o docker0 -j ACCEPT -m comment --comment "flannel subnet"
firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 1 -i docker0 -j ACCEPT -m comment --comment "kube-proxy redirects"

systemctl restart firewalld
3.2 安装配置flannel

安装flannel:

1
yum install flannel

装完之后配置flannel参数, 文件位置在/etc/sysconfig/flanneld:

1
2
3
4
5
6
7
8
9
10
11
12
# Flanneld configuration options

# etcd url location.  Point this to the server where etcd runs
FLANNEL_ETCD_ENDPOINTS="http://node-1-master:2379"   # 已修改

# etcd config key.  This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_PREFIX="/cluster.local/network"         # 已修改

# Any additional options that you want to pass
# By default, we just add a good guess for the network interface on Vbox.  Otherwise, Flannel will probably make the right guess.
FLANNEL_OPTIONS=""

以上环境变量根据自己集群的实际情况配置

重启相关服务：

1
2
3
systemctl enable flanneld
systemctl restart flanneld #自动会从etcd获取新的ip地址段
systemctl restart docker

4.安装kubelet和kube-proxy

在centos上两个服务只需要安装kubnernetes-node就可以了:

1
yum install kubernetes-node
4.1 修改公用配置

配置文件路径在/etc/kubernetes/config:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
###
# kubernetes system config
#
# The following values are used to configure various aspects of all
# kubernetes services, including
#
#   kube-apiserver.service
#   kube-controller-manager.service
#   kube-scheduler.service
#   kubelet.service
#   kube-proxy.service
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"

# journal message level, 0 is debug
KUBE_LOG_LEVEL="--v=0"

# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=false"

# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=https://node-1-master"               # 已修改

一般情况下只需要修改KUBE_MASTER的配置就可以了，让它指向master机器的地址

4.2 修改kubelet配置

文件位置在/etc/kubernetes/kubelet:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
###
# kubernetes kubelet (minion) config

# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=0.0.0.0"                        # 已修改

# The port for the info server to serve on
# KUBELET_PORT="--port=10250"

# You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=node-3-slave-2"      # 已修改

# location of the api-server
KUBELET_API_SERVER="--api-servers=https://node-1-master"   # 已修改

# pod infrastructure container
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"

# Add your own!
# 这条配置是在第5步的时候加的!!!!!!!!!
KUBELET_ARGS="--kubeconfig=/etc/kubernetes/kubelet.kubeconfig --config=/etc/kubernetes/manifests"
4.3 修改kube-proxy配置

文件位置在/etc/kubernetes/proxy:

1
2
3
4
5
6
7
8
###
# kubernetes proxy config

# default config should be adequate

# Add your own!
# 这条配置是在第5步的时候加的!!!!!!!!!
KUBE_PROXY_ARGS="--kubeconfig=/etc/kubernetes/proxy.kubeconfig"

5.权限相关

5.1 生成kubelet和kube-proxy的token

在这个步骤中我们生成新机器的kubelet和kube-proxy的token并把它注册到master机器中，值得注意的是，以下所有子步骤中的脚本都要在master执行。

  1. 执行以下脚本在控制台获得一个token:

    1
    dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64 | tr -d "=+/" | dd bs=32 count=1 2>/dev/null

  2. 将上一步生成的token复制到master机器的/etc/kubernetes/tokens/known_hosts.csv文件中:

    1
    2
    # 多加一行，表示该token代表slave2上的kubelet
    YOUR_KUBELET_TOKEN,system:kubelet-node-3-slave-2,system:kubelet-node-3-slave-2

  3. 重复步骤1，2，生成kube-proxy的token

    1
    YOUR_PROXY_TOKEN,system:proxy-node-3-slave-2,system:proxy-node-3-slave-2

  4. 重启相关服务

    1
    2
    3
    systemctl restart kube-apiserver
    systemctl restart kube-controller-manager
    systemctl restart kube-scheduler
5.2 服务的认证

  1. 复制ca证书
    将master中的/etc/kubernetes/certs/ca.crt复制到新机器同样的位置

  2. 在配置中多加一行(参照4.2,4.3的注释)，同时创建两个文件:
    2.1. /etc/kubernetes/kubelet.kubeconfig

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    apiVersion: v1
    kind: Config
    current-context: kubelet-to-cluster.local
    preferences: {}
    clusters:
    - cluster:
        certificate-authority: /etc/kubernetes/certs/ca.crt
        server: https://node-1-master
      name: cluster.local
    contexts:
    - context:
        cluster: cluster.local
        user: kubelet
      name: kubelet-to-cluster.local
    users:
    - name: kubelet
      user:
        token: YOUR_KUBELET_TOKEN

    2.2. /etc/kubernetes/proxy.kubeconfig

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    apiVersion: v1
    kind: Config
    current-context: proxy-to-cluster.local
    preferences: {}
    contexts:
    - context:
        cluster: cluster.local
        user: proxy
      name: proxy-to-cluster.local
    clusters:
    - cluster:
        certificate-authority: /etc/kubernetes/certs/ca.crt
        server: https://node-1-master
      name: cluster.local
    users:
    - name: proxy
      user:
        token: YOUR_PROXY_TOKEN

  3. 重启并配置开机自动启动

    1
    2
    3
    4
    systemctl enable kubelet
    systemctl enable kube-proxy
    systemctl restart kubelet
    systemctl restart kube-proxy

xiduiSoftware Engineer

个人简介。