[설치] Openshift Version UP (3.3 -> 3.4)

 [ UPGRADING A CLUSTER ]

Install Guide에 따르면 버전업그레이드 Automated 방식과 Manual 방식이 있으며, 본 내용은 Manual 방식으로 진행된다.

해당 내용은 'OpenShift_Container_Platform-3.4-Installation_and_Configuration-en-US.pdf'의 Chapter5를 참고한다.

1. 사전 준비 작업

- Download new docker images (ose 3.4)

   ㄴ pushing new docker images to Docker Registry

- Update Yum Repository ( rhel-7-server-extras-rpms / rhel-7-server-ose-3.4-rpms)

   ㄴ yum clean all, yum repolist

2. 작업절차

- push docker images - update yum repository - 각 서버 openshift 패키지 업그레이드 - Etcd 백업 - Master 서버작업    ㄴ Upgrading Master Components    ㄴ Updating Policy Definitions - Nodes 서버작업 - Upgrading the Router - Updating the Default Image Streams and Templates - Upgrading the EFK Logging Stack - Upgrading Cluster Metrics

3. 업그레이드 수행

1단계 : openshift 3.4 docker image 다운로드 및 push

Download Images

Push Images

# 다운로드 docker push registry.access.redhat.com/openshift3/ose-haproxy-router:v3.4.1.5 &&\ docker push registry.access.redhat.com/openshift3/ose-deployer:v3.4.1.5 &&\ docker push registry.access.redhat.com/openshift3/ose-sti-builder:v3.4.1.5 &&\ docker push registry.access.redhat.com/openshift3/ose-docker-builder:v3.4.1.5 &&\ docker push registry.access.redhat.com/openshift3/ose-pod:v3.4.1.5 &&\ docker push registry.access.redhat.com/openshift3/ose-docker-registry:v3.4.1.5 docker pull registry.access.redhat.com/openshift3/logging-deployer:3.4.1 &&\ docker pull registry.access.redhat.com/openshift3/logging-elasticsearch:3.4.1 &&\ docker pull registry.access.redhat.com/openshift3/logging-kibana:3.4.1 &&\ docker pull registry.access.redhat.com/openshift3/logging-fluentd:3.4.1 &&\ docker pull registry.access.redhat.com/openshift3/logging-curator:3.4.1 &&\ docker pull registry.access.redhat.com/openshift3/logging-auth-proxy:3.4.1 &&\ docker pull registry.access.redhat.com/openshift3/metrics-deployer:3.4.1 &&\ docker pull registry.access.redhat.com/openshift3/metrics-hawkular-metrics:3.4.1 &&\ docker pull registry.access.redhat.com/openshift3/metrics-cassandra:3.4.1 &&\ docker pull registry.access.redhat.com/openshift3/metrics-heapster:3.4.1 # Retagging docker tag registry.access.redhat.com/openshift3/ose-haproxy-router:v3.4.1.5  openshift3/ose-haproxy-router:v3.4.1.10 # Create tarballs docker save -o ose34.tar \ openshift3/ose-haproxy-router:v3.4.1.10 &&\ openshift3/ose-deployer:v3.4.1.10 &&\ openshift3/ose-sti-builder:v3.4.1.10 &&\ openshift3/ose-docker-builder:v3.4.1.10 &&\ openshift3/ose-pod:v3.4.1.10 &&\ openshift3/ose-docker-registry:v3.4.1.10 docker save -o ose34-metric.tar \ openshift3/logging-deployer:3.4.1 &&\ openshift3/logging-elasticsearch:3.4.1 &&\ openshift3/logging-kibana:3.4.1 &&\ openshift3/logging-fluentd:3.4.1 &&\ openshift3/logging-curator:3.4.1 &&\ openshift3/logging-auth-proxy:3.4.1 &&\ openshift3/metrics-deployer:3.4.1 &&\ openshift3/metrics-hawkular-metrics:3.4.1 &&\ openshift3/metrics-cassandra:3.4.1 &&\ openshift3/metrics-heapster:3.4.1

# loading docker images docker load -i ose34.tar docker load -i ose34-metric.tar # push docker images docker push openshift3/ose-haproxy-router:v3.4.1.10 &&\ docker push openshift3/ose-deployer:v3.4.1.10 &&\ docker push openshift3/ose-sti-builder:v3.4.1.10 &&\ docker push openshift3/ose-docker-builder:v3.4.1.10 &&\ docker push openshift3/ose-pod:v3.4.1.10 &&\ docker push openshift3/ose-docker-registry:v3.4.1.10 docker push openshift3/logging-deployer:3.4.1 &&\ docker push openshift3/logging-elasticsearch:3.4.1 &&\ docker push openshift3/logging-kibana:3.4.1 &&\ docker push openshift3/logging-fluentd:3.4.1 &&\ docker push openshift3/logging-curator:3.4.1 &&\ docker push openshift3/logging-auth-proxy:3.4.1 &&\ docker push openshift3/metrics-deployer:3.4.1 &&\ docker push openshift3/metrics-hawkular-metrics:3.4.1 &&\ docker push openshift3/metrics-cassandra:3.4.1 &&\ docker push openshift3/metrics-heapster:3.4.1

2단계 : Yum repository 변경, 3.3 패키지 대시 3.4 패키지 재정의

$ yum clean all

$ yum repolist


3단계 : 각 서버 openshift 패키지 업그레이드

$ yum install atomic-openshift-utils

$ yum install atomic-openshift-excluder atomic-openshift-dockerexcluder

4단계 : etcd 백업(master에 embedded etcd인 경우)

* etcdctl 명령어가 먹히지 않을경우 yum install etcd 로 설치

$ ETCD_DATA_DIR=/var/lib/etcd

$ etcdctl backup --data-dir $ETCD_DATA_DIR --backup-dir $ETCD_DATA_DIR.bak.170321

drwxr-xr-x.  3 etcd    etcd      19 Mar 21 13:08 etcd

drwx------.  3 root    root      19 Mar 21 14:05 etcd.bak.170321

5단계 : Upgrading Master Components

$ atomic-openshift-excluder unexclude

$ yum update etcd

$ systemctl restart etcd  (1대만 실행할 경우 remote etcd의 버전이 안맞아 에러 발생, 동시에 진행)

request version incompatibility (remote version is too low: remote[6648eed0773ccc39]

$ yum upgrade atomic-openshift\*

$ systemctl restart atomic-openshift-master-controllers

$ systemctl restart atomic-openshift-master-api

$ journalctl -r -u atomic-openshift-master-controllers

$ journalctl -r -u atomic-openshift-master-api

$ systemctl restart atomic-openshift-node

$ systemctl restart openvswitch

$ journalctl -r -u openvswitch

$ journalctl -r -u atomic-openshift-node

$ yum update docker (1.10.3 -> 1.12.6)

$ systemctl reboot (master서버 순차적으로 진행)

$ atomic-openshift-excluder exclude

$ oadm policy reconcile-cluster-roles

$ oadm policy reconcile-cluster-roles \

--additive-only=true \

--confirm

---> 결과

clusterrole/cluster-reader

clusterrole/storage-admin

clusterrole/admin

clusterrole/edit

clusterrole/basic-user

clusterrole/cluster-status

clusterrole/system:deployer

clusterrole/system:registry

clusterrole/registry-admin

<----

$ oadm policy reconcile-cluster-role-bindings \

--exclude-groups=system:authenticated \

--exclude-groups=system:authenticated:oauth \

--exclude-groups=system:unauthenticated \

--exclude-users=system:anonymous \

--additive-only=true \

--confirm

$ oadm policy reconcile-sccs \

--additive-only=true \

--confirm

6단계 : Upgrading Nodes

$ atomic-openshift-excluder unexclude (노드서버)

# pod가 생성되지 않게 하는 설정....pod 이관작업이므로 1대씩 수행

$ oadm manage-node node1.ocp.com --schedulable=false

$ oadm manage-node node2.ocp.com --schedulable=false

$ oc get nodes

node1.ocp.com     Ready,SchedulingDisabled   19d

node2.ocp.com     Ready,SchedulingDisabled   19d

# 아래 작업은 해당 노드의 pod를 다른 노드로 옮기는 명령어로 보인다. 1대씩 수행하자

$ oadm manage-node node1.ocp.com --evacuate --force

$ oadm manage-node node2.ocp.com --evacuate --force

> 결과 

---------------------------------------------------------------------------------------------

[root@master1 ~]# oadm manage-node node1.ocp.com --evacuate --force

Flag --evacuate has been deprecated, use 'oadm manage-node drain NODE' instead

Migrating these pods on node: node1.ocp.com

NAME                          READY     STATUS    RESTARTS   AGE

eap-app-3-4d8bw             1/1       Running        1          1h

eap-app-3-build               0/1       Completed     0          1h

docker-registry-3-35bxu     1/1       Running        2          19d

router-2-4wu3k                1/1       Running        2          19d

---------------------------------------------------------------------------------------------

$ yum upgrade atomic-openshift\* (노드서버)

$ systemctl restart atomic-openshift-node

$ systemctl restart openvswitch

$ journalctl -r -u atomic-openshift-node

$ journalctl -r -u openvswitch

$ yum update docker

$ systemctl restart docker

$ systemctl restart atomic-openshift-node

$ systemctl reboot

$ oadm manage-node node1.ocp.com --schedulable=true --> 1번 작업 완료 다음 노드 수행....스케쥴 false 부터

$ oadm manage-node node2.ocp.com --schedulable=true

$ atomic-openshift-excluder exclude

7단계 : Upgrading the Router

$ oc edit dc/router -n default (버전 변경)

image: openshift3/ose-haproxy-router:v3.4.1.10

8단계 : Updating the Default Image Streams and Templates

oc create -n openshift -f /usr/share/openshift/examples/image-streams/image-streams-rhel7.json;

oc create -n openshift -f /usr/share/openshift/examples/image-streams/dotnet_imagestreams.json;

oc create -n openshift -f /usr/share/openshift/examples/db-templates;

oc create -n openshift -f /usr/share/openshift/examples/quickstart-templates;

oc create -n openshift -f /usr/share/openshift/examples/xpaas-streams;

oc create -n openshift -f /usr/share/openshift/examples/xpaas-templates;

oc replace -n openshift -f /usr/share/openshift/examples/image-streams/image-streams-rhel7.json && \

oc replace -n openshift -f /usr/share/openshift/examples/db-templates && \

oc replace -n openshift -f /usr/share/openshift/examples/quickstart-templates && \

oc replace -n openshift -f /usr/share/openshift/examples/xpaas-streams && \

oc replace -n openshift -f /usr/share/openshift/examples/xpaas-templates

# 1.4 버전 imageStream, templates 위치 확인

$ rpm -ql openshift-ansible-roles | grep examples | grep v1.4

$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/image-streams/image-streams-rhel7.json

$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/image-streams/dotnet_imagestreams.json

$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/image-streams/image-streams-rhel7.json

$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/image-streams/dotnet_imagestreams.json

# 1.4 버전 추가

oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/quickstart-templates/

oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/db-templates/

oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/infrastructure-templates/

oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/xpaas-templates/

oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/xpaas-streams/

oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/quickstart-templates/ && \

oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/db-templates/ && \

oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/infrastructure-templates/ && \

oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/xpaas-templates/ && \

oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/xpaas-streams/

9단계 : Upgrading the EFK Logging Stack

$ oc project logging

$ oc apply -n openshift -f \

/usr/share/ansible/openshift-ansible/roles/openshift_hosted_templates/files/v1.4/enterprise/logging-deployer.yaml

$ oc process logging-deployer-account-template | oc apply -f -    ---> 에러발생

error: template "logging-deployer-account-template" could not be found

error: no objects passed to apply

$ oadm policy add-cluster-role-to-user oauth-editor \

system:serviceaccount:logging:logging-deployer

$ oadm policy add-cluster-role-to-user rolebinding-reader \

system:serviceaccount:logging:aggregated-logging-elasticsearch

$ oadm policy add-scc-to-user privileged \

system:serviceaccount:logging:aggregated-logging-fluentd

$ oc new-app logging-deployer-template -p MODE=upgrade

10단계 : Upgrading Cluster Metrics

$ oadm policy add-role-to-user view \

system:serviceaccount:openshift-infra:hawkular \

-n openshift-infra

$ oc new-app --as=system:serviceaccount:openshift-infra:metrics-deployer \

-f /usr/share/openshift/examples/infrastructure-templates/enterprise/metrics-deployer.yaml \

-p HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.app.ocp.com \

-p MODE=refresh \

-p IMAGE_PREFIX=10.1.0.48:5000/openshift3/ \

-p IMAGE_VERSION=3.4.1

위 작업을 통해 업그레이드 작업을 마치고 기능검증 진행.

- Pod 생성

- EAP 이미지로 scale out 확인

- metric 정상 동작 확인 등

- 마 침 -