Erlo

git+ jenkins+harbor+k8s(kubernetes) 实现自动化部署

时间:2020-07-28   阅读:42次   来源:博客园
页面报错
点赞

系统环境:

centos 7 

git:gitee.com 当然随便一个git服务端都行

jenkins: lts版本,部署在服务器上,没有通过部署在k8s集群中

harbor: offline版本,用来存储docker镜像

 Kubernetes 集群为了方便快捷,使用了kubeadm方式搭建,是三台,并且启用了IPVS,具体服务器用途说明如下:

 
HOSTNAME IP地址 服务器用途
master.test.cn 192.168.184.31 k8s-master
node1.test.cn 192.168.184.32 k8s-node1
node2.test.cn 192.168.184.33 k8s-node2
soft.test.cn 192.168.184.34 harbor、jenkins

 

一、Kubernetes 搭建

本节点参考:https://www.cnblogs.com/lovesKey/p/10888006.html

1.1 系统配置

先将4台主机写好hosts 

1 [root@master ~]# cat /etc/hosts
2 192.168.184.31    master.test.cn
3 192.168.184.32    node1.test.cn fanli.test.cn
4 192.168.184.33    node2.test.cn
5 192.168.184.34    soft.test.cn jenkins.test.cn harbor.test.cn

 

关闭swap:
临时关闭

swapoff -a

永久关闭(删除或注释掉swap那一行重启即可)

vim /etc/fstab

关闭所有防火墙

systemctl stop firewalld
systemctl disable firewalld

禁用SELINUX:

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

重启使SELINUX生效

将桥接的IPv4流量传递到iptables的链,使设置生效:

cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

1.2 kube-proxy开启ipvs的前置条件

由于ipvs已经加入到了内核的主干,所以为kube-proxy开启ipvs的前提需要加载以下的内核模块:

ip_vs

ip_vs_rr

ip_vs_wrr

ip_vs_sh

nf_conntrack_ipv4

在所有的Kubernetes节点master,node1和node2上执行以下脚本:

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

可以看到已经生效了: nf_conntrack_ipv4
15053 5 nf_defrag_ipv4 12729 1 nf_conntrack_ipv4 ip_vs_sh 12688 0 ip_vs_wrr 12697 0 ip_vs_rr 12600 16 ip_vs 145497 22 ip_vs_rr,ip_vs_sh,ip_vs_wrr nf_conntrack 139264 9 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4,nf_conntrack_ipv6 libcrc32c 12644 4 xfs,ip_vs,nf_nat,nf_conntrack

脚本创建了的/etc/sysconfig/modules/ipvs.modules文件,保证在节点重启后能自动加载所需模块。 使用lsmod | grep -e ip_vs -e nf_conntrack_ipv4命令查看是否已经正确加载所需的内核模块。

在所有节点上安装ipset软件包,同时为了方便查看ipvs规则我们要安装ipvsadm(可选)

 yum install ipset ipvsadm

1.3 安装Docker(所有节点)

       配置docker国内源(阿里云)

wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo

 安装最新版docker-ce

yum -y install docker-ce
systemctl enable docker && systemctl start docker
docker --version

配置kubernetes的源(阿里云)

cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

手动导入gpgkey或者关闭 

gpgcheck=0

 

 

rpmkeys --import https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
rpmkeys --import https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

 

开始安装kubeadm和kubelet:

yum install -y kubelet kubeadm kubectl
systemctl enable kubelet
systemctl start kubelet

 

1.4开始部署Kubernetes

初始化master

kubeadm init 
--apiserver-advertise-address=192.168.184.31 
--image-repository lank8s.cn 
--kubernetes-version v1.18.6 
--pod-network-cidr=10.244.0.0/16

 

关注输出内容

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.184.31:6443 --token a9vg9z.dlboqvfuwwzauufq 
    --discovery-token-ca-cert-hash sha256:c2ade88a856f15de80240ff4994661a6daa668113cea0c4a4073f701f05192cb

执行下面命令 初始化当前用户配置 使用kubectl会用到  master节点执行:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

安装pod网络插件

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

在各个node上执行 下面加入命令(加入集群中)

kubeadm join 192.168.233.251:6443 --token a9vg9z.dlboqvfuwwzauufq --discovery-token-ca-cert-hash sha256:c2ade88a856f15de80240ff4994661a6daa668113cea0c4a4073f701f05192cb

 

使用kubectl get po -A -o wide确保所有的Pod都处于Running状态。

[root@master ~]# kubectl get po -A -o wide
NAMESPACE       NAME                                        READY   STATUS      RESTARTS   AGE    IP               NODE             NOMINATED NODE   READINESS GATES
kube-system     coredns-5c579bbb7b-flzkd                    1/1     Running     5          2d     10.244.1.14      node1.test.cn    <none>           <none>
kube-system     coredns-5c579bbb7b-qz5m8                    1/1     Running     4          2d     10.244.2.9       node2.test.cn    <none>           <none>
kube-system     etcd-master.test.cn                         1/1     Running     5          2d     192.168.184.31   master.test.cn   <none>           <none>
kube-system     kube-apiserver-master.test.cn               1/1     Running     5          2d     192.168.184.31   master.test.cn   <none>           <none>
kube-system     kube-controller-manager-master.test.cn      1/1     Running     5          2d     192.168.184.31   master.test.cn   <none>           <none>
kube-system     kube-flannel-ds-amd64-bhmps                 1/1     Running     6          2d     192.168.184.33   node2.test.cn    <none>           <none>
kube-system     kube-flannel-ds-amd64-mbpvb                 1/1     Running     6          2d     192.168.184.32   node1.test.cn    <none>           <none>
kube-system     kube-flannel-ds-amd64-xnw2l                 1/1     Running     6          2d     192.168.184.31   master.test.cn   <none>           <none>
kube-system     kube-proxy-8nkgs                            1/1     Running     6          2d     192.168.184.32   node1.test.cn    <none>           <none>
kube-system     kube-proxy-jxtfk                            1/1     Running     4          2d     192.168.184.31   master.test.cn   <none>           <none>
kube-system     kube-proxy-ls7xg                            1/1     Running     4          2d     192.168.184.33   node2.test.cn    <none>           <none>
kube-system     kube-scheduler-master.test.cn               1/1     Running     4          2d     192.168.184.31   master.test.cn   <none>           <none>

kube-proxy开启ipvs

#修改ConfigMap的kube-system/kube-proxy中的config.conf,把 mode: "" 改为mode: “ipvs" 保存退出即可
[root@k8smaster centos]# kubectl edit cm kube-proxy -n kube-system
configmap/kube-proxy edited
###删除之前的proxy pod
[root@master centos]# kubectl get pod -n kube-system |grep kube-proxy |awk '{system("kubectl delete pod "$1" -n kube-system")}'
pod "kube-proxy-2m5jh" deleted
pod "kube-proxy-nfzfl" deleted
pod "kube-proxy-shxdt" deleted
#查看proxy运行状态
[root@master centos]# kubectl get pod -n kube-system | grep kube-proxy
kube-proxy-54qnw                              1/1     Running   0          24s
kube-proxy-bzssq                              1/1     Running   0          14s
kube-proxy-cvlcm                              1/1     Running   0          37s
#查看日志,如果有 `Using ipvs Proxier.` 说明kube-proxy的ipvs 开启成功!
[root@master centos]# kubectl logs kube-proxy-54qnw -n kube-system
I0518 20:24:09.319160       1 server_others.go:176] Using ipvs Proxier.
W0518 20:24:09.319751       1 proxier.go:386] IPVS scheduler not specified, use rr by default
I0518 20:24:09.320035       1 server.go:562] Version: v1.14.2
I0518 20:24:09.334372       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0518 20:24:09.334853       1 config.go:102] Starting endpoints config controller
I0518 20:24:09.334916       1 controller_utils.go:1027] Waiting for caches to sync for endpoints config controller
I0518 20:24:09.334945       1 config.go:202] Starting service config controller
I0518 20:24:09.334976       1 controller_utils.go:1027] Waiting for caches to sync for service config controller
I0518 20:24:09.435153       1 controller_utils.go:1034] Caches are synced for service config controller
I0518 20:24:09.435271       1 controller_utils.go:1034] Caches are synced for endpoints config controller

查看 node节点是否ready

[root@master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
master.test.cn Ready master 2d v1.18.6
node1.test.cn Ready <none> 2d v1.18.6
node2.test.cn Ready <none> 2d v1.18.6

 

至此 K8S 用kubeadm方式并且使用IPVS方案 安装成功

二、harbor 安装

 2.1 准备

harbor是通过docker-compose 启动的,我们首先要在soft.test.cn 节点 安装docker-compose

curl -L https://github.com/docker/compose/releases/download/1.26.1/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose 

 

2.2 下载Harbor

harbor的官方地址:https://github.com/goharbor/harbor/releases 

然后按照官方安装文档进行安装操作:https://github.com/goharbor/harbor/blob/master/docs/install-config/_index.md

这里的设置方法,我把自己的域名粘贴上来了,如果自己实际操作,需要将harbor.test.cn替换成你自己的域名

 

下载离线包

wget https://github.com/goharbor/harbor/releases/download/v1.10.4/harbor-offline-installer-v1.10.4.tgz

 

解压安装包

[root@soft ~]# tar zxvf harbor-offline-installer-v1.10.4.tgz

 

2.3 设置 https 

生成 CA 证书私钥

openssl genrsa -out ca.key 4096

生成 CA 证书

openssl req -x509 -new -nodes -sha512 -days 3650 
 -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=harbor.test.cn" 
 -key ca.key 
 -out ca.crt

2.4 生成服务器证书

证书通常包含文件和文件,例如  .crt.key yourdomain.com.crt yourdomain.com.key

生成私钥

openssl genrsa -out harbor.test.cn.key 4096

 

生成证书签名请求 (CSR)

调整选项中的值以反映您的组织。如果使用 FQDN 连接港口主机,则必须将其指定为公共名称 () 属性,并在密钥和 CSR 文件名中使用它。-subj CN

openssl req -sha512 -new 
    -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=harbor.test.cn" 
    -key harbor.test.cn.key 
    -out harbor.test.cn.csr

 

生成 x509 v3 扩展文件

无论您使用 FQDN 还是 IP 地址连接到港口主机,都必须创建此文件,以便可以为符合主题替代名称 (SAN) 和 x509 v3 扩展要求的港湾主机生成证书。替换条目以反映您的域。

cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1=test.cn
DNS.2=test
DNS.3=harbor.test.cn
EOF

使用该文件为港口主机生成证书

将 CRS 和 CRT 文件名中的替换为harbor主机名

openssl x509 -req -sha512 -days 3650 
    -extfile v3.ext 
    -CA ca.crt -CAkey ca.key -CAcreateserial 
    -in harbor.test.cn.csr 
    -out harbor.test.cn.crt

 

 

2.5 向harbor和docker提供证书

生成ca.crt 、yourdomain.com.crt和 yourdomain.com.key文件后,必须将它们提供给harbor和docker,并重新配置harbor 以使用它们

将服务器证书和密钥复制到港湾主机上的证书文件夹中

cp harbor.test.cn.crt /data/cert/
cp harbor.test.cn.key /data/cert/

 

转换harbor.test.cn.crt为harbor.test.cn.cert ,供 Docker 使用

Docker 守护进程将crt文件解释为 CA 证书,cert文件解释为客户端证书

openssl x509 -inform PEM -in yourdomain.com.crt -out yourdomain.com.cert

将服务器证书、密钥和 CA 文件复制到港湾主机上的 Docker 证书文件夹中。您必须先创建相应的文件夹

cp harbor.test.cn.cert /etc/docker/certs.d/harbor.test.cn/
cp harbor.test.cn.key /etc/docker/certs.d/harbor.test.cn/
cp ca.crt /etc/docker/certs.d/harbor.test.cn/

 

重启docker

systemctl restart docker

 

2.6 部署harbor

修改配置文件harbor.yml

两处需要修改

1.修改hostname

2. 修改certificate 和key的路径

 

运行脚本以启用 HTTPS

[root@soft ~]# cd harbor
[root@soft harbor]# ./prepare 

 

结束后执行install.sh

[root@soft harbor]# ./install.sh 

 

harbor相关命令

查看结果,可以看到已经都是up的状态,同时启动了80和443端口映射

[root@soft harbor]# docker-compose ps
      Name                     Command                  State                          Ports                   
---------------------------------------------------------------------------------------------------------------
harbor-core         /harbor/harbor_core              Up (healthy)                                              
harbor-db           /docker-entrypoint.sh            Up (healthy)   5432/tcp                                   
harbor-jobservice   /harbor/harbor_jobservice  ...   Up (healthy)                                              
harbor-log          /bin/sh -c /usr/local/bin/ ...   Up (healthy)   127.0.0.1:1514->10514/tcp                  
harbor-portal       nginx -g daemon off;             Up (healthy)   8080/tcp                                   
nginx               nginx -g daemon off;             Up (healthy)   0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp
redis               redis-server /etc/redis.conf     Up (healthy)   6379/tcp                                   
registry            /home/harbor/entrypoint.sh       Up (healthy)   5000/tcp                                   
registryctl         /home/harbor/start.sh            Up (healthy)      

 

停止harbor运行

[root@soft harbor]# docker-compose down -v
Stopping nginx             ... done
Stopping harbor-jobservice ... done
Stopping harbor-core       ... done
Stopping harbor-portal     ... done
Stopping redis             ... done
Stopping registryctl       ... done
Stopping harbor-db         ... done
Stopping registry          ... done
Stopping harbor-log        ... done
Removing nginx             ... done
Removing harbor-jobservice ... done
Removing harbor-core       ... done
Removing harbor-portal     ... done
Removing redis             ... done
Removing registryctl       ... done
Removing harbor-db         ... done
Removing registry          ... done
Removing harbor-log        ... done
Removing network harbor_harbor

 

启动harbor进程

[root@soft harbor]# docker-compose up -d
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating registry      ... done
Creating harbor-db     ... done
Creating redis         ... done
Creating harbor-portal ... done
Creating registryctl   ... done
Creating harbor-core   ... done
Creating nginx             ... done
Creating harbor-jobservice ... done

 

2.7 登录测试

浏览器登录https://harbor.test.cn 需要电脑配置hosts.账号admin 密码Harbor12345

创建一个jenkins用户,用于jenkins使用

 

 新建一个jenkins项目用于jenkins后续部署用

 

 添加成员

 

 

 

 

 在需要上传镜像的服务器上修改docker仓库连接方式为http,否则默认https无法连接。这里以harbor.test.cn上我修改的sonarqube镜像为例

vim /etc/docker/daemon.json
加入 {
"insecure-registries" : ["harbor.test.cn"] }

 

重启Docker生效

systemctl restart docker

 三、Jenkins

安装

Jenkins 官网地址:https://www.jenkins.io/zh/

可以根据不同的系统安装,可以安装lts版,长期支持版,也可以安装每周更新版,我这里选择安装长期支持版:

[root@soft ~]# wget https://mirrors.tuna.tsinghua.edu.cn/jenkins/redhat-stable/jenkins-2.235.2-1.1.noarch.rpm

 

安装jdk

[root@soft harbor]# yum install -y java-1.8.0-openjdk

 

安装jenkins

[root@soft harbor]# yum localinstall -y jenkins-2.235.2-1.1.noarch.rpm 

 

 之后我们设置开机启动并启动jenkins

[root@soft harbor]# systemctl enable jenkins 
[root@soft harbor]# systemctl start jenkins

 

 

 我们访问jenkins页面:http://192.168.184.34:8080/,可以看到jenkins已经可以初始化了

按照提示的路径查看密码

 

 

 

选择安装插件,第一个为默认安装,第二个为手动这里选择默认的

 

 

 

安装完插件后,创建新用户

 

 

  我们安装插件CloudBees Docker Build and Publish plugin

 

 

 

 安装完成后我们新建item

 

 

 

在源码管理的这里按需选择,用svn就选svn,用git就选git,我这里选择了gitee网站上自己写的测试代码 django构建的,同样分支默认应该为master,我这里按需填写了develop,自己按需填写。

注意Dockerfile 一定要和代码放在同一处(当前目录最顶层),这样docker build and publish 插件才能生效

 

 

 

Dockerfile

FROM python:3.6-alpine

ENV PYTHONUNBUFFERED 1

WORKDIR /app

RUN pip install django -i https://pypi.douban.com/simple

COPY . /app

CMD python /app/manage.py runserver 0.0.0.0:8000

 

 

 

 

 

之后构建处选择docker build and publish 

 

 

 

 

 

 给soft.test.cn主机的/var/run/docker.sock 666权限

[root@soft harbor]# chmod 666  /var/run/docker.sock

 

再次添加构建,选择execute shell 

 

 

 命令中这么写为了在master主机上执行kubectl ,去部署pod

 

 

 

ssh 192.168.184.31 "cd /data/jenkins/${JOB_NAME} && sh rollout.sh ${BUILD_NUMBER}"

 

我的脚本是这么写的:

[root@master fanli_admin]# cat rollout.sh 
#!/bin/bash
workdir="/data/jenkins/fanli_admin"
project="fanli_admin"
job_number=$(date +%s)
cd ${workdir}
oldversion=$(cat ${project}_delpoyment.yaml | grep "image:" | awk -F ':' '{print $NF}')
newversion=$1

echo "
                         
                 

评论留言

还没有评论留言,赶紧来抢楼吧~~

吐槽小黑屋()

* 这里是“吐槽小黑屋”,所有人可看,只保留当天信息。

  • Erlo吐槽

    Erlo.vip2020-08-12 05:15:29Hello、欢迎使用吐槽小黑屋,这就是个吐槽的地方。
  • 返回顶部