Deploying a User Provisioned Infrastructure (UPI) environment for Openshift 4.2
1. Prerequisite
Please refer to the Openshift Documentation for more information regarding supported config
This is created with below software version:
- Terraform v0.11.12
- RHCOS v4.2
- Openshift v4.2.16
- Openshift Installer v4.4
- govc v0.22.1
Below are the prerequisite:
-
Download Terraform Install unzip
sudo apt-get install wget unzip
Download respective version and extract the archive
export VER="0.11.12" wget https://releases.hashicorp.com/terraform/${VER}/terraform_${VER}_linux_amd64.zip unzip terraform_${VER}_linux_amd64.zip
Move file fo /usr/local/bin directory
sudo mv terraform /usr/local/bin/
-
Download and import Red Hat CoreOS Image for vSphere Download govc and RHCOS image
mkdir ocp-install cd ocp-install/ curl -L https://github.com/vmware/govmomi/releases/download/v0.22.1/govc_linux_amd64.gz | sudo gunzip > /usr/local/bin/govc curl -O https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.2/latest/rhcos-4.2.0-x86_64-openstack.qcow2 chmod +x /usr/local/bin/govc
Login from GOVC
export GOVC_URL='vcsa-01a.corp.local' export GOVC_USERNAME='[email protected]' export GOVC_PASSWORD='<password>' export GOVC_NETWORK='LabNet' export GOVC_DATASTORE='ds-site-a-nfs03' export GOVC_INSECURE=1
Check if the login works
$ govc about Name: VMware vCenter Server Vendor: VMware, Inc. Version: 6.7.0 Build: 14070654 OS type: linux-x64 API type: VirtualCenter API version: 6.7.2 Product ID: vpx UUID: ccf97624-bfc3-4eb1-8fab-9a5acccaa46b
Upload OVA image and mask as template
$ govc import.spec rhcos-4.2.0-x86_64-vmware.ova | python -m json.tool > rhcos.json
Customize the Network you want to use in rhcos.json
"NetworkMapping": [ { "Name": "VM Network", "Network": "LabNet" }
List the resource pool available
$ govc find / -type p /DC-SiteA/host/Mgmt-Edge-Cluster/Resources /DC-SiteA/host/Compute-Cluster/Resources
Upload RHCOS image to the resource pool
govc import.ova -options=./rhcos.json -name=rhcos-4.2 -pool=/DC-SiteA/host/Compute-Cluster/Resources rhcos-4.2.0-x86_64-vmware.ova
Mark the VM as a template:
govc vm.markastemplate vm/rhcos-4.2
-
Create applicable DNS record for Openshift deployment Please refer to the Openshift DNS requirement for more information regarding supported config
$ORIGIN apps.example.com. * A 10.x.y.38 * A 10.x.y.39 * A 10.x.y.40 $ORIGIN openshift4.example.com. _etcd-server-ssl._tcp SRV 0 10 2380 etcd-0 _etcd-server-ssl._tcp SRV 0 10 2380 etcd-1 _etcd-server-ssl._tcp SRV 0 10 2380 etcd-2 bootstrap-0 A 10.x.y.34 control-plane-0 A 10.x.y.35 control-plane-1 A 10.x.y.36 control-plane-2 A 10.x.y.37 api A 10.x.y.34 api A 10.x.y.35 api A 10.x.y.36 api A 10.x.y.37 api-int A 10.x.y.34 api-int A 10.x.y.35 api-int A 10.x.y.36 api-int A 10.x.y.37 etcd-0 A 10.x.y.35 etcd-1 A 10.x.y.36 etcd-2 A 10.x.y.37 compute-0 A 10.x.y.38 compute-1 A 10.x.y.39 compute-2 A 10.x.y.40
-
Create a working install-config file for the installer to consume
cd ocp-install/ curl -O https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.2.16/openshift-install-linux-4.2.16.tar.gz curl -O https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.2.16/openshift-client-linux-4.2.16.tar.gz tar xzvf openshift-install-linux-4.2.16.tar.gz tar xzvf openshift-client-linux-4.2.16.tar.gz mv oc kubectl openshift-install /usr/local/bin/ vi install-config.yaml
Generate SSH Key for passwordless login
ssh-keygen -t rsa -b 4096 -N '' -f ~/.ssh/id_rsa eval "$(ssh-agent -s)" ssh-add ~/.ssh/id_rsa
Extract the SSH public key
cat ~/.ssh/id_rsa.pub
Generate Manifest files and edit the master scheduler
openshift-install --dir ~/ocp-install/config create manifests $ cat manifests/cluster-scheduler-02-config.yml apiVersion: config.openshift.io/v1 kind: Scheduler metadata: creationTimestamp: null name: cluster spec: mastersSchedulable: false policy: name: "" status: {}
install-config.yaml file
apiVersion: v1 baseDomain: corp.local metadata: name: openshift4 compute: - hyperthreading: Enabled name: worker replicas: 3 controlPlane: hyperthreading: Enabled name: master replicas: 3 networking: clusterNetwork: - cidr: 10.128.0.0/14 hostPrefix: 23 networkType: OpenShiftSDN serviceNetwork: - 172.30.0.0/16 platform: vsphere: vcenter: vcsa-01a.corp.local username: "[email protected]" password: "VMware1!" datacenter: DC-SiteA defaultDatastore: ds-site-a-nfs03 pullSecret: '<pull-secret>' sshKey: <ssh-pub-key>
Generate ignition config files that will need to be added to the terraform tfvars
openshift-install --dir ~/ocp-install/config create ignition-configs
There are three ignition files generated: bootstrap.ign, master.ign, worker.ign. Bootstrap.ign is quite a long text file so you need to upload it to HTTP server. In this example, I provision IIS server in Windows Server 2012 and put those 3 files in the directory.
Create file append-bootstrap as follow:
{ "ignition": { "config": { "append": [ { "source": "http://192.168.110.10:8080/ignition/bootstrap.ign", "verification": {} } ] }, "timeouts": {}, "version": "2.1.0" }, "networkd": {}, "passwd": {}, "storage": {}, "systemd": {} }
Store them in the same directory
2. Download UPI Terraform included in Openshift installer repository and update provisioning variables
-
Download the terraform files
git clone -b release-4.4 https://github.com/openshift/installer vi openshift/installer/upi/vsphere/terraform.tfvars
# Put static IPs here bootstrap_ip = "192.168.110.81" control_plane_ips = ["192.168.110.82","192.168.110.83","192.168.110.84"] compute_ips = ["192.168.110.85","192.168.110.86","192.168.110.87"] machine_cidr = "192.168.110.0/24" # Put cluster information here cluster_id = "openshift4" cluster_domain = "openshift4.corp.local" base_domain = "corp.local" # vSphere information vsphere_server = "vcsa-01a.corp.local" vsphere_user = "[email protected]" vsphere_password = "VMware1!" vsphere_cluster = "Compute-Cluster" vsphere_datacenter = "DC-SiteA" vsphere_datastore = "ds-site-a-nfs03” vm_template = "rhcos-4.2" vm_network = "LabNet" # Define the number of nodes control_plane_count = 3 compute_count = 3 bootstrap_ignition_url = "http://192.168.110.10:8080/ignition/append-bootstrap.ign" control_plane_ignition = <<END_OF_MASTER_IGNITION <put your master ignition file content here> END_OF_MASTER_IGNITION compute_ignition = <<END_OF_WORKER_IGNITION <put your worker ignition file content here> END_OF_WORKER_IGNITION
-
Edit the DNS variable in ignition.tf
vi openshift/installer/upi/vsphere/machine/ignition.tf
DNS1=192.168.110.10
Remove The DNS Section from main.tf
/* module "dns" { source = "./route53" base_domain = "${var.base_domain}" cluster_domain = "${var.cluster_domain}" bootstrap_count = "${var.bootstrap_complete ? 0 : 1}" bootstrap_ips = ["${module.bootstrap.ip_addresses}"] control_plane_count = "${var.control_plane_count}" control_plane_ips = ["${module.control_plane.ip_addresses}"] compute_count = "${var.compute_count}" compute_ips = ["${module.compute.ip_addresses}"] } */
3. Run Terraform to initialize and provision the environment
cd installer/upi/vsphere/
terraform init
terraform plan
terraform apply -auto-approve
The environment should now be deployed in vSphere
4. Inform the installer that the bootstrap process has completed and delete bootstrap VM
openshift-install wait-for bootstrap-complete
terraform apply -auto-approve -var 'bootstrap_complete=true'
5. Logging in to the cluster
-
Export the kubeadmin credentials
export KUBECONFIG=auth/kubeconfig
-
Verify with oc command
$ oc whoami system:admin $ oc get nodes NAME STATUS ROLES AGE VERSION compute-0 Ready worker 22m v1.14.6+97c81d00e compute-1 Ready worker 22m v1.14.6+97c81d00e compute-2 Ready worker 22m v1.14.6+97c81d00e control-plane-0 Ready master 22m v1.14.6+97c81d00e control-plane-1 Ready master 22m v1.14.6+97c81d00e control-plane-2 Ready master 22m v1.14.6+97c81d00e
Source
https://blog.openshift.com/deploying-a-user-provisioned-infrastructure-environment-for-openshift-4-1-on-vsphere/
https://github.com/openshift/installer
https://labs.consol.de/container/platform/openshift/2020/01/31/ocp43-installation-vmware.html