1. Prerequisite

Please refer to the Openshift Documentation for more information regarding supported config

This is created with below software version:

  • Terraform v0.11.12
  • RHCOS v4.2
  • Openshift v4.2.16
  • Openshift Installer v4.4
  • govc v0.22.1

Below are the prerequisite:

  • Download Terraform Install unzip

    sudo apt-get install wget unzip
    

    Download respective version and extract the archive

    export VER="0.11.12"
    wget https://releases.hashicorp.com/terraform/${VER}/terraform_${VER}_linux_amd64.zip
    unzip terraform_${VER}_linux_amd64.zip
    

    Move file fo /usr/local/bin directory

    sudo mv terraform /usr/local/bin/
    
  • Download and import Red Hat CoreOS Image for vSphere Download govc and RHCOS image

    mkdir ocp-install
    cd ocp-install/
    curl -L https://github.com/vmware/govmomi/releases/download/v0.22.1/govc_linux_amd64.gz | sudo gunzip > /usr/local/bin/govc
    curl -O https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.2/latest/rhcos-4.2.0-x86_64-openstack.qcow2
    chmod +x /usr/local/bin/govc
    

    Login from GOVC

    export GOVC_URL='vcsa-01a.corp.local'
    export GOVC_USERNAME='[email protected]'
    export GOVC_PASSWORD='<password>'
    export GOVC_NETWORK='LabNet'
    export GOVC_DATASTORE='ds-site-a-nfs03'
    export GOVC_INSECURE=1
    

    Check if the login works

    $ govc about
    Name:         VMware vCenter Server
    Vendor:       VMware, Inc.
    Version:      6.7.0
    Build:        14070654
    OS type:      linux-x64
    API type:     VirtualCenter
    API version:  6.7.2
    Product ID:   vpx
    UUID:         ccf97624-bfc3-4eb1-8fab-9a5acccaa46b
    

    Upload OVA image and mask as template

    $ govc import.spec rhcos-4.2.0-x86_64-vmware.ova | python -m json.tool > rhcos.json
    

    Customize the Network you want to use in rhcos.json

    "NetworkMapping": [
      {
          "Name": "VM Network",
          "Network": "LabNet"
      }
    

    List the resource pool available

    $ govc find / -type p
    /DC-SiteA/host/Mgmt-Edge-Cluster/Resources
    /DC-SiteA/host/Compute-Cluster/Resources
    

    Upload RHCOS image to the resource pool

    govc import.ova -options=./rhcos.json -name=rhcos-4.2  -pool=/DC-SiteA/host/Compute-Cluster/Resources  rhcos-4.2.0-x86_64-vmware.ova
    

    Mark the VM as a template:

    govc vm.markastemplate vm/rhcos-4.2
    
  • Create applicable DNS record for Openshift deployment Please refer to the Openshift DNS requirement for more information regarding supported config

    $ORIGIN apps.example.com.
    * A 10.x.y.38
    * A 10.x.y.39
    * A 10.x.y.40
    $ORIGIN openshift4.example.com.
    _etcd-server-ssl._tcp SRV 0 10 2380 etcd-0
    _etcd-server-ssl._tcp SRV 0 10 2380 etcd-1
    _etcd-server-ssl._tcp SRV 0 10 2380 etcd-2
    bootstrap-0 A 10.x.y.34
    control-plane-0 A 10.x.y.35
    control-plane-1 A 10.x.y.36
    control-plane-2 A 10.x.y.37
    api A 10.x.y.34
    api A 10.x.y.35
    api A 10.x.y.36
    api A 10.x.y.37
    api-int A 10.x.y.34
    api-int A 10.x.y.35
    api-int A 10.x.y.36
    api-int A 10.x.y.37
    etcd-0 A 10.x.y.35
    etcd-1 A 10.x.y.36
    etcd-2 A 10.x.y.37
    compute-0 A 10.x.y.38
    compute-1 A 10.x.y.39
    compute-2 A 10.x.y.40
    
  • Create a working install-config file for the installer to consume

    cd ocp-install/
    curl -O https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.2.16/openshift-install-linux-4.2.16.tar.gz
    curl -O https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.2.16/openshift-client-linux-4.2.16.tar.gz
    tar xzvf openshift-install-linux-4.2.16.tar.gz
    tar xzvf openshift-client-linux-4.2.16.tar.gz
    mv oc kubectl openshift-install /usr/local/bin/
    vi install-config.yaml
    

    Generate SSH Key for passwordless login

    ssh-keygen -t rsa -b 4096 -N '' -f ~/.ssh/id_rsa
    eval "$(ssh-agent -s)"
    ssh-add ~/.ssh/id_rsa
    

    Extract the SSH public key

    cat ~/.ssh/id_rsa.pub
    

    Generate Manifest files and edit the master scheduler

    openshift-install --dir ~/ocp-install/config create manifests
    
    $ cat manifests/cluster-scheduler-02-config.yml
    apiVersion: config.openshift.io/v1
    kind: Scheduler
    metadata:
      creationTimestamp: null
      name: cluster
    spec:
      mastersSchedulable: false
      policy:
        name: ""
    status: {}
    

    install-config.yaml file

    apiVersion: v1
    baseDomain: corp.local
    metadata:
      name: openshift4
    compute:
    - hyperthreading: Enabled   
      name: worker
      replicas: 3
    controlPlane:
      hyperthreading: Enabled   
      name: master
      replicas: 3
    networking:
      clusterNetwork:
      - cidr: 10.128.0.0/14
        hostPrefix: 23
      networkType: OpenShiftSDN
      serviceNetwork:
      - 172.30.0.0/16
    platform:
      vsphere:
        vcenter: vcsa-01a.corp.local
        username: "[email protected]"
        password: "VMware1!"
        datacenter: DC-SiteA
        defaultDatastore: ds-site-a-nfs03
    pullSecret: '<pull-secret>'
    sshKey: <ssh-pub-key>
    

    Generate ignition config files that will need to be added to the terraform tfvars

    openshift-install --dir ~/ocp-install/config create ignition-configs
    

    There are three ignition files generated: bootstrap.ign, master.ign, worker.ign. Bootstrap.ign is quite a long text file so you need to upload it to HTTP server. In this example, I provision IIS server in Windows Server 2012 and put those 3 files in the directory.

    Create file append-bootstrap as follow:

    {
    "ignition": {
      "config": {
        "append": [
          {
            "source": "http://192.168.110.10:8080/ignition/bootstrap.ign",
            "verification": {}
          }
        ]
      },
      "timeouts": {},
      "version": "2.1.0"
    },
    "networkd": {},
    "passwd": {},
    "storage": {},
    "systemd": {}
    }
    

    Store them in the same directory

2. Download UPI Terraform included in Openshift installer repository and update provisioning variables

  • Download the terraform files

    git clone -b release-4.4 https://github.com/openshift/installer
    vi openshift/installer/upi/vsphere/terraform.tfvars
    
    # Put static IPs here
    bootstrap_ip = "192.168.110.81"
    control_plane_ips = ["192.168.110.82","192.168.110.83","192.168.110.84"]
    compute_ips = ["192.168.110.85","192.168.110.86","192.168.110.87"]
    machine_cidr = "192.168.110.0/24"
    
    # Put cluster information here
    cluster_id = "openshift4"
    cluster_domain = "openshift4.corp.local"
    base_domain = "corp.local"
    
    # vSphere information
    vsphere_server = "vcsa-01a.corp.local"
    vsphere_user = "[email protected]"
    vsphere_password = "VMware1!"
    vsphere_cluster = "Compute-Cluster"
    vsphere_datacenter = "DC-SiteA"
    vsphere_datastore = "ds-site-a-nfs03”
    vm_template = "rhcos-4.2"
    vm_network = "LabNet"
    
    # Define the number of nodes
    control_plane_count = 3
    compute_count = 3
    
    bootstrap_ignition_url = "http://192.168.110.10:8080/ignition/append-bootstrap.ign"
    
    control_plane_ignition = <<END_OF_MASTER_IGNITION
    <put your master ignition file content here>
    END_OF_MASTER_IGNITION
    
    compute_ignition = <<END_OF_WORKER_IGNITION
    <put your worker ignition file content here>
    END_OF_WORKER_IGNITION
    
  • Edit the DNS variable in ignition.tf

    vi openshift/installer/upi/vsphere/machine/ignition.tf
    
    DNS1=192.168.110.10
    

    Remove The DNS Section from main.tf

    /*
    module "dns" {
    source = "./route53"
    
    base_domain = "${var.base_domain}"
    cluster_domain = "${var.cluster_domain}"
    bootstrap_count = "${var.bootstrap_complete ? 0 : 1}"
    bootstrap_ips = ["${module.bootstrap.ip_addresses}"]
    control_plane_count = "${var.control_plane_count}"
    control_plane_ips = ["${module.control_plane.ip_addresses}"]
    compute_count = "${var.compute_count}"
    compute_ips = ["${module.compute.ip_addresses}"]
    }
    */
    

3. Run Terraform to initialize and provision the environment

cd installer/upi/vsphere/
terraform init
terraform plan
terraform apply -auto-approve

The environment should now be deployed in vSphere

4. Inform the installer that the bootstrap process has completed and delete bootstrap VM

openshift-install wait-for bootstrap-complete

terraform apply -auto-approve -var 'bootstrap_complete=true'

5. Logging in to the cluster

  • Export the kubeadmin credentials

    export KUBECONFIG=auth/kubeconfig
    
  • Verify with oc command

    $ oc whoami
    system:admin
    
    $ oc get nodes
    NAME              STATUS   ROLES    AGE   VERSION
    compute-0         Ready    worker   22m   v1.14.6+97c81d00e
    compute-1         Ready    worker   22m   v1.14.6+97c81d00e
    compute-2         Ready    worker   22m   v1.14.6+97c81d00e
    control-plane-0   Ready    master   22m   v1.14.6+97c81d00e
    control-plane-1   Ready    master   22m   v1.14.6+97c81d00e
    control-plane-2   Ready    master   22m   v1.14.6+97c81d00e
    

Source

https://blog.openshift.com/deploying-a-user-provisioned-infrastructure-environment-for-openshift-4-1-on-vsphere/
https://github.com/openshift/installer
https://labs.consol.de/container/platform/openshift/2020/01/31/ocp43-installation-vmware.html