VMware vSphere Integrated Containers provider (#206)

* Add Virtual Kubelet provider for VIC

Initial virtual kubelet provider for VMware VIC.  This provider currently
handles creating and starting of a pod VM via the VIC portlayer and persona
server.  Image store handling via the VIC persona server.  This provider
currently requires the feature/wolfpack branch of VIC.

* Added pod stop and delete.  Also added node capacity.

Added the ability to stop and delete pod VMs via VIC.  Also retrieve
node capacity information from the VCH.

* Cleanup and readme file

Some file clean up and added a Readme.md markdown file for the VIC
provider.

* Cleaned up errors, added function comments, moved operation code

1. Cleaned up error handling.  Set standard for creating errors.
2. Added method prototype comments for all interface functions.
3. Moved PodCreator, PodStarter, PodStopper, and PodDeleter to a new folder.

* Add mocking code and unit tests for podcache, podcreator, and podstarter

Used the unit test framework used in VIC to handle assertions in the provider's
unit test.  Mocking code generated using OSS project mockery, which is compatible
with the testify assertion framework.

* Vendored packages for the VIC provider

Requires feature/wolfpack branch of VIC and a few specific commit sha of
projects used within VIC.

* Implementation of POD Stopper and Deleter unit tests (#4)

* Updated files for initial PR
This commit is contained in:
Loc Nguyen
2018-06-04 15:41:32 -07:00
committed by Ria Bhatia
parent 98a111e8b7
commit 513cebe7b7
6296 changed files with 1123685 additions and 8 deletions

117
vendor/github.com/vmware/govmomi/scripts/vcsa/README.md generated vendored Normal file
View File

@@ -0,0 +1,117 @@
# vCenter cluster testbed automation
## Overview
This directory contains scripts to automate VCSA/ESXi install and cluster configuration for developing and testing.
## Dependencies
### govc
Install the latest release via https://github.com/vmware/govmomi/releases
### jq
Used here to derive static VCSA networking from its parent ESXi host.
But, you should already be using and loving jq for other tasks: http://stedolan.github.io/jq/
## Scripts
### create-esxi-vm.sh
This script creates a VM running stateless ESXi, booted via cdrom/iso.
It will create 2 disks by default:
* vSAN cache disk (Virtual SSD)
* vSAN store disk
The two vSAN disks will be unformatted, leaving them to be autoclaimed
by a vSAN enabled cluster.
Note that for a cluster to use vSAN, it will need at least 3 of these
ESXi VMs.
To create an ESXi VM for standalone use, use the `-s` flag and optionally increase the default disk size with the `-d`
flag:
```
./create-esxi-vm.sh -s -d 56 $GOVC_URL my-esxi-vm
```
The script can also be used directly against Workstation. Without a username in the url, govc will use local ticket
authentication. No password is used in this case, but the script will still use this value to set the password for
`root` in the ESX vm. You may also want to decrease the default disk `-d` and memory `-m` sizes. Example:
```
GOVC_NETWORK=NAT ./create-esxi-vm.sh -d 16 -m 4 -s :password-for-esx60@localhost $USER-esxbox
```
For use against Fusion, use `.` as the hostname:
```
./create-esxi-vm.sh -d 16 -m 4 -s root:password-for-esx60@. $USER-esxbox
```
### create-vcsa-vm.sh
This script creates a VM with VCSA (Virtual Center Server Appliance) installed.
### create-cluster.sh
The first argument to the script is the IP address of VCSA.
There must be at least three arguments that follow, IP addresses of ESXi hosts, to form the cluster.
The script then creates the following managed objects:
* Datacenter (dc1)
* ClusterComputeResource (cluster1)
* DistributedVirtualSwitch (DSwitch)
* DistributedVirtualPortgroup (PublicNetwork)
* DistributedVirtualPortgroup (InternalNetwork)
All of the given host systems are:
* Added to the ClusterComputeResource (cluster1)
* Added to the DistributedVirtualSwitch (DSwitch)
* Enabled for vSAN traffic (vmk0)
* Firewall configured to enable the remoteSerialPort rule
Cluster configuration includes:
* DRS enabled
* vSAN autoclaim of host system disks (results in shared Datastore "vsanDatastore")
## Example
This example will install VCSA, 3 ESXi VMs and create a cluster.
```
export GOVC_URL="root:password@some-esx-host"
./create-vcsa-vm.sh -n "${USER}-vcsa" $GOVC_URL
printf "${USER}-esxi-%03d\n" {1..3} | xargs -P3 -n1 ./create-esxi-vm.sh $GOVC_URL
govc vm.ip -k "${USER}-vcsa" "${USER}-esxi-*" | xargs ./create-cluster.sh
```
## Licenses
Optional, if you want to assign licenses to VCSA or ESXi hosts.
Where "ESX_LICENSE" should be that of "vSphere 6 per CPU, Enterprise Plus".
And "VCSA_LICENSE" should be that of "vCenter Server 6, Standard".
```
ESX_LICENSE=... VCSA_LICENSE=... ./assign-licenses.sh
```

View File

@@ -0,0 +1,31 @@
#!/bin/bash -e
# Copyright 2017 VMware, Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
export GOVC_INSECURE=true
govc license.add "$VCSA_LICENSE" "$ESX_LICENSE" >/dev/null
govc license.assign "$VCSA_LICENSE" >/dev/null
govc ls -t HostSystem /dc1/host/cluster1 | xargs -I% -n1 govc license.assign -host % "$ESX_LICENSE" >/dev/null
echo "Assigned licenses..."
govc license.assigned.ls
echo ""
echo "License usage..."
govc license.ls

View File

@@ -0,0 +1,132 @@
#!/bin/bash -e
# Copyright 2017-2018 VMware, Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Configure a vCenter cluster with vSAN datastore, DVS and DVPGs
export GOVC_INSECURE=1
export GOVC_USERNAME=${GOVC_USERNAME:-"Administrator@vsphere.local"}
if [ -z "$GOVC_PASSWORD" ] ; then
# extract password from $GOVC_URL
GOVC_PASSWORD=$(govc env GOVC_PASSWORD)
fi
usage() {
echo "Usage: $0 [-d DATACENTER] [-c CLUSTER] VCSA_IP ESX_IP..." 1>&2
exit 1
}
# Defaults
dc_name="dc1"
cluster_name="cluster1"
vsan_vnic="vmk0"
while getopts c:d: flag
do
case $flag in
c)
cluster_name=$OPTARG
;;
d)
dc_name=$OPTARG
;;
*)
usage
;;
esac
done
shift $((OPTIND-1))
if [ $# -lt 2 ] ; then
usage
fi
vc_ip=$1
shift
unset GOVC_DATACENTER
export GOVC_URL="${GOVC_USERNAME}:${GOVC_PASSWORD}@${vc_ip}"
cluster_path="/$dc_name/host/$cluster_name"
dvs_path="/$dc_name/network/DSwitch"
public_network="/$dc_name/network/PublicNetwork"
internal_network="/$dc_name/network/InternalNetwork"
if [ -z "$(govc ls "/$dc_name")" ] ; then
echo "Creating datacenter ${dc_name}..."
govc datacenter.create "$dc_name"
fi
export GOVC_DATACENTER="$dc_name"
if [ -z "$(govc ls "$cluster_path")" ] ; then
echo "Creating cluster ${cluster_path}..."
govc cluster.create "$cluster_name"
fi
if [ -z "$(govc ls "$dvs_path")" ] ; then
echo "Creating dvs ${dvs_path}..."
govc dvs.create -product-version 5.5.0 -folder "$(dirname "$dvs_path")" "$(basename "$dvs_path")"
fi
if [ -z "$(govc ls "$public_network")" ] ; then
govc dvs.portgroup.add -dvs "$dvs_path" -type earlyBinding -nports 16 "$(basename "$public_network")"
fi
if [ -z "$(govc ls "$internal_network")" ] ; then
govc dvs.portgroup.add -dvs "$dvs_path" -type ephemeral "$(basename "$internal_network")"
fi
hosts=()
vsan_hosts=()
for host_ip in "$@" ; do
host_path="$cluster_path/$host_ip"
hosts+=($host_path)
if [ -z "$(govc ls "$host_path")" ] ; then
echo "Adding host ($host_ip) to cluster $cluster_name"
govc cluster.add -cluster "$cluster_path" -noverify -force \
-hostname "$host_ip" -username root -password "$GOVC_PASSWORD"
fi
unclaimed=$(govc host.storage.info -host "$host_path" -unclaimed | tail -n+2 | wc -l)
if [ "$unclaimed" -eq 2 ] ; then
echo "Enabling vSAN traffic on ${vsan_vnic} for ${host_path}..."
govc host.vnic.service -host "$host_path" -enable vsan "$vsan_vnic"
vsan_hosts+=($host_path)
else
echo "Skipping vSAN configuration for ${host_path}: $unclaimed unclaimed disks"
fi
done
govc dvs.add -dvs "$dvs_path" -pnic vmnic1 "${hosts[@]}"
echo "Enabling DRS for ${cluster_path}..."
govc cluster.change -drs-enabled "$cluster_path"
if [ ${#vsan_hosts[@]} -ge 3 ] ; then
echo "Enabling vSAN for ${cluster_path}..."
govc cluster.change -vsan-enabled -vsan-autoclaim "$cluster_path"
fi
echo "Enabling HA for ${cluster_path}..."
govc cluster.change -ha-enabled "$cluster_path"
echo "Granting Admin permissions for user root..."
govc permissions.set -principal root -role Admin
echo "Done."

View File

@@ -0,0 +1,283 @@
#!/bin/bash -e
# Copyright 2017-2018 VMware, Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Create a VM and boot stateless ESXi via cdrom/iso
set -o pipefail
usage() {
cat <<'EOF'
Usage: $0 [-d DISK_GB] [-m MEM_GB] [-i ESX_ISO] [-s] ESX_URL VM_NAME
GOVC_* environment variables also apply, see https://github.com/vmware/govmomi/tree/master/govc#usage
If GOVC_USERNAME is set, it is used to create an account on the ESX vm. Default is to use the existing root account.
If GOVC_PASSWORD is set, the account password will be set to this value. Default is to use the given ESX_URL password.
EOF
}
disk=48
mem=16
# 6.5U1 (EP5) - https://docs.vmware.com/en/VMware-vSphere/6.5/rn/vsphere-esxi-651-release-notes.html
iso=VMware-VMvisor-6.5.0-7526137.x86_64.iso
while getopts d:hi:m:s flag
do
case $flag in
d)
disk=$OPTARG
;;
h)
usage
exit
;;
i)
iso=$OPTARG
;;
m)
mem=$OPTARG
;;
s)
standalone=true
;;
*)
usage 1>&2
exit 1
;;
esac
done
shift $((OPTIND-1))
if [ $# -ne 2 ] ; then
usage
exit 1
fi
if [[ "$iso" == *"-Installer-"* ]] ; then
echo "Invalid iso name (need stateless, not installer): $iso" 1>&2
exit 1
fi
export GOVC_INSECURE=1
export GOVC_URL=$1
network=${GOVC_NETWORK:-"VM Network"}
username=$GOVC_USERNAME
password=$GOVC_PASSWORD
unset GOVC_USERNAME GOVC_PASSWORD
guest=${GUEST:-"vmkernel6Guest"}
if [ -z "$password" ] ; then
# extract password from $GOVC_URL
password=$(govc env GOVC_PASSWORD)
fi
shift
name=$1
shift
echo -n "Checking govc version..."
govc version -require 0.15.0
if [ "$(govc env -x GOVC_URL_HOST)" = "." ] ; then
if [ "$(uname -s)" = "Darwin" ]; then
PATH="/Applications/VMware Fusion.app/Contents/Library:$PATH"
fi
dir="${name}.vmwarevm"
vmx="$dir/${name}.vmx"
if [ -d "$dir" ] ; then
if vmrun list | grep -q "$vmx" ; then
vmrun stop "$vmx" hard
fi
rm -rf "$dir"
fi
mkdir "$dir"
vmware-vdiskmanager -c -s "${disk}GB" -a lsilogic -t 1 "$dir/${name}.vmdk" 2>/dev/null
cat > "$vmx" <<EOF
config.version = "8"
virtualHW.version = "11"
numvcpus = "2"
memsize = "$((mem*1024))"
displayName = "$name"
guestOS = "vmkernel6"
vhv.enable = "TRUE"
scsi0.present = "TRUE"
scsi0.virtualDev = "lsilogic"
scsi0:0.present = "TRUE"
scsi0:0.fileName = "${name}.vmdk"
ide1:0.present = "TRUE"
ide1:0.fileName = "$(realpath "$iso")"
ide1:0.deviceType = "cdrom-image"
ethernet0.present = "TRUE"
ethernet0.connectionType = "nat"
ethernet0.virtualDev = "e1000"
ethernet0.wakeOnPcktRcv = "FALSE"
ethernet0.linkStatePropagation.enable = "TRUE"
vmci0.present = "TRUE"
hpet0.present = "TRUE"
tools.syncTime = "TRUE"
pciBridge0.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
EOF
vmrun start "$vmx" nogui
vm_ip=$(vmrun getGuestIPAddress "$vmx" -wait)
else
export GOVC_DATASTORE=${GOVC_DATASTORE:-$(basename "$(govc ls datastore)")}
if [ "$(govc about -json | jq -r .About.ProductLineId)" == "embeddedEsx" ] ; then
policy=$(govc host.portgroup.info -json | jq -r ".Portgroup[] | select(.Spec.Name == \"$network\") | .Spec.Policy.Security")
if [ -n "$policy" ] && [ "$(jq -r <<<"$policy" .AllowPromiscuous)" != "true" ] ; then
echo "Enabling promiscuous mode for $network on $(govc env -x GOVC_URL_HOST)..."
govc host.portgroup.change -allow-promiscuous "$network"
fi
fi
boot=$(basename "$iso")
if ! govc datastore.ls "$boot" > /dev/null 2>&1 ; then
govc datastore.upload "$iso" "$boot"
fi
echo "Creating vm ${name}..."
govc vm.create -on=false -net "$network" -m $((mem*1024)) -c 2 -g "$guest" -net.adapter=vmxnet3 -disk.controller pvscsi "$name"
echo "Adding a second nic for ${name}..."
govc vm.network.add -net "$network" -net.adapter=vmxnet3 -vm "$name"
echo "Enabling nested hv for ${name}..."
govc vm.change -vm "$name" -nested-hv-enabled
echo "Enabling Mac Learning dvFilter for ${name}..."
seq 0 1 | xargs -I% govc vm.change -vm "$name" \
-e ethernet%.filter4.name=dvfilter-maclearn \
-e ethernet%.filter4.onFailure=failOpen
echo "Adding cdrom device to ${name}..."
id=$(govc device.cdrom.add -vm "$name")
echo "Inserting $boot into $name cdrom device..."
govc device.cdrom.insert -vm "$name" -device "$id" "$boot"
if [ -n "$standalone" ] ; then
echo "Creating $name disk for use by ESXi..."
govc vm.disk.create -vm "$name" -name "$name"/disk1 -size "${disk}G"
fi
echo "Powering on $name VM..."
govc vm.power -on "$name"
echo "Waiting for $name ESXi IP..."
vm_ip=$(govc vm.ip "$name")
! govc events -n 100 "vm/$name" | grep -E 'warning|error'
fi
esx_url="root:@${vm_ip}"
echo "Waiting for $name hostd (via GOVC_URL=$esx_url)..."
while true; do
if govc about -u "$esx_url" 2>/dev/null; then
break
fi
printf "."
sleep 1
done
if [ -z "$standalone" ] ; then
# Create disk for vSAN after boot so they are unclaimed
echo "Creating $name disks for use by vSAN..."
govc vm.disk.create -vm "$name" -name "$name"/vsan-cache -size "$((disk/2))G"
govc vm.disk.create -vm "$name" -name "$name"/vsan-store -size "${disk}G"
fi
# Set target to the ESXi VM
GOVC_URL="$esx_url"
if [ -z "$standalone" ] ; then
echo "Rescanning ${name} HBA for new devices..."
disk=($(govc host.storage.info -rescan | grep /vmfs/devices/disks | awk '{print $1}' | sort))
echo "Marking ${name} disk ${disk[0]} as SSD..."
govc host.storage.mark -ssd "${disk[0]}"
echo "Marking ${name} disk ${disk[1]} as HDD..."
govc host.storage.mark -ssd=false "${disk[1]}"
fi
echo "Enabling MOB for ${name}..."
govc host.option.set Config.HostAgent.plugins.solo.enableMob true
echo "Configuring NTP for ${name}..."
govc host.date.change -server 0.pool.ntp.org
for id in TSM TSM-SSH ntpd ; do
printf "Enabling service %s for ${name}...\n" $id
govc host.service enable $id
govc host.service start $id
done
if [ -z "$username" ] ; then
username=root
action="update"
else
action="create"
fi
echo "Disabling VSAN device monitoring for ${name}..."
govc host.esxcli system settings advanced set -o /LSOM/VSANDeviceMonitoring -i 0
# A setting of 1 means that vSwp files are created thin, with 0% Object Space Reservation
govc host.esxcli system settings advanced set -o /VSAN/SwapThickProvisionDisabled -i 1
govc host.esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 1
echo "ESX host account $action for user $username on ${name}..."
govc host.account.$action -id $username -password "$password"
echo "Granting Admin permissions for user $username on ${name}..."
govc permissions.set -principal $username -role Admin
echo "Enabling guest ARP inspection to get vm IPs without vmtools on ${name}..."
govc host.esxcli system settings advanced set -o /Net/GuestIPHack -i 1
echo "Opening firewall for serial port traffic for ${name}..."
govc host.esxcli network firewall ruleset set -r remoteSerialPort -e true
echo "Setting hostname for ${name}..."
govc host.esxcli system hostname set -H "$name"
if which sshpass >/dev/null && [ -e ~/.ssh/id_rsa.pub ] ; then
echo "Adding ssh authorized key to ${name}..."
sshpass -p "$password" scp \
-oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oLogLevel=error \
~/.ssh/id_rsa.pub "root@$vm_ip:/etc/ssh/keys-root/authorized_keys"
fi
echo "Done: GOVC_URL=${username}:${password}@${vm_ip}"

View File

@@ -0,0 +1,144 @@
#!/bin/bash -e
# Copyright 2017-2018 VMware, Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Create a VCSA VM
usage() {
echo "Usage: $0 [-n VM_NAME] [-i VCSA_OVA] [-a IP] ESX_URL" 1>&2
exit 1
}
export GOVC_INSECURE=1
name=vcsa
# 6.5 U1e (EP5) - https://docs.vmware.com/en/VMware-vSphere/6.5/rn/vsphere-vcenter-server-65u1e-release-notes.html
ova=VMware-vCenter-Server-Appliance-6.5.0.14000-7515524_OVF10.ova
while getopts a:i:n: flag
do
case $flag in
a)
ip=$OPTARG
;;
i)
ova=$OPTARG
;;
n)
name=$OPTARG
;;
*)
usage
;;
esac
done
if [ -d "$ova" ] ; then
ova=$(ls "$ova"/*.ovf)
fi
shift $((OPTIND-1))
if [ $# -ne 1 ] ; then
usage
fi
export GOVC_URL=$1
network=${GOVC_NETWORK:-$(basename "$(govc ls network)")}
product=$(govc about -json | jq -r .About.ProductLineId)
# Use the same password as GOVC_URL
password=$(govc env GOVC_PASSWORD)
if [ -z "$password" ] ; then
echo "password not set"
exit 1
fi
opts=(
cis.vmdir.password=$password
cis.appliance.root.passwd=$password
cis.appliance.root.shell=/bin/bash
cis.deployment.node.type=embedded
cis.vmdir.domain-name=vsphere.local
cis.vmdir.site-name=VCSA
cis.appliance.net.addr.family=ipv4
cis.appliance.ssh.enabled=True
cis.ceip_enabled=False
cis.deployment.autoconfig=True
)
if [ -z "$ip" ] ; then
mode=dhcp
ntp=0.pool.ntp.org
else
mode=static
# Derive net config from the ESX server
config=$(govc host.info -k -json | jq -r .HostSystems[].Config)
gateway=$(jq -r .Network.IpRouteConfig.DefaultGateway <<<"$config")
dns=$(jq -r .Network.DnsConfig.Address[0] <<<"$config")
ntp=$(jq -r .DateTimeInfo.NtpConfig.Server[0] <<<"$config")
route=$(jq -r ".Network.RouteTableInfo.IpRoute[] | select(.DeviceName == \"vmk0\") | select(.Gateway == \"0.0.0.0\")" <<<"$config")
prefix=$(jq -r .PrefixLength <<<"$route")
opts+=(cis.appliance.net.addr=$ip
cis.appliance.net.prefix=$prefix
cis.appliance.net.dns.servers=$dns
cis.appliance.net.gateway=$gateway)
fi
opts+=(
cis.appliance.ntp.servers="$ntp"
cis.appliance.net.mode=$mode
)
if [ "$product" = "ws" ] ; then
# workstation does not suport NFC
dir=$(govc datastore.info -json | jq -r .Datastores[0].Info.Url)
ovftool --name="$name" --acceptAllEulas "$ova" "$dir"
vmx="$name/${name}.vmx"
printf "guestinfo.%s\n" "${opts[@]}" >> "$dir/$vmx"
govc vm.register "$vmx"
govc vm.network.change -vm "$name" -net NAT ethernet-0
else
props=$(printf -- "guestinfo.%s\n" "${opts[@]}" | \
jq --slurp -R 'split("\n") | map(select(. != "")) | map(split("=")) | map({"Key": .[0], "Value": .[1]})')
cat <<EOF | govc import.${ova##*.} -options - "$ova"
{
"Name": "$name",
"Deployment": "tiny",
"DiskProvisioning": "thin",
"IPProtocol": "IPv4",
"Annotation": "VMware vCenter Server Appliance",
"PowerOn": false,
"WaitForIP": false,
"InjectOvfEnv": true,
"NetworkMapping": [
{
"Name": "Network 1",
"Network": "${network}"
}
],
"PropertyMapping": $props
}
EOF
fi
govc vm.power -on "$name"
govc vm.ip "$name"