d8dfun/virtual-kubelet
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

VMware vSphere Integrated Containers provider (#206)

Browse Source 
* Add Virtual Kubelet provider for VIC

Initial virtual kubelet provider for VMware VIC.  This provider currently
handles creating and starting of a pod VM via the VIC portlayer and persona
server.  Image store handling via the VIC persona server.  This provider
currently requires the feature/wolfpack branch of VIC.

* Added pod stop and delete.  Also added node capacity.

Added the ability to stop and delete pod VMs via VIC.  Also retrieve
node capacity information from the VCH.

* Cleanup and readme file

Some file clean up and added a Readme.md markdown file for the VIC
provider.

* Cleaned up errors, added function comments, moved operation code

1. Cleaned up error handling.  Set standard for creating errors.
2. Added method prototype comments for all interface functions.
3. Moved PodCreator, PodStarter, PodStopper, and PodDeleter to a new folder.

* Add mocking code and unit tests for podcache, podcreator, and podstarter

Used the unit test framework used in VIC to handle assertions in the provider's
unit test.  Mocking code generated using OSS project mockery, which is compatible
with the testify assertion framework.

* Vendored packages for the VIC provider

Requires feature/wolfpack branch of VIC and a few specific commit sha of
projects used within VIC.

* Implementation of POD Stopper and Deleter unit tests (#4)

* Updated files for initial PR
This commit is contained in:
Loc Nguyen
2018-06-04 15:41:32 -07:00
committed by Ria Bhatia
parent 98a111e8b7
commit 513cebe7b7
6296 changed files with 1123685 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

57
vendor/github.com/vmware/vic/doc/application_workflow/README.md generated vendored Normal file
View File

@@ -0,0 +1,57 @@
# Container Workflow
The following are guideline suggestions for those who want to use VIC to develop and deploy a containerized application. These guidelines pertain to VIC 0.7.0. While VIC continues to progress towards VIC 1.0, the current feature set requires some care in creating a containerized application and deploying it. We present these guidelines from a developer and a devops perspective.
An example workflow is presented here, in the form of a modified voting app, based on [Docker's voting app](https://github.com/docker/example-voting-app).
## Container Workflow Feature Set
General feature set (e.g. Docker run, ps, inspect, etc) used at the CLI will not be discussed. Only feature set important for containerizing apps and deploying them are discussed. These include volume and network support. It is also worth mentioning that basic Docker Compose support is available for application deployment.
#### Currently Available Features
1. Docker Compose (basic)
2. Registry pull from docker hub and private registry
3. Named Data Volumes
4. Anonymous Data Volumes
5. Bridged Networks
6. External Networks
7. Port Mapping
8. Network Links/Alias
#### Future Features
Be aware the following feature are not yet available and must be taken into account when containerizing an app and deploying it.
1. Docker build
2. Registry pushing
3. Concurrent data volume sharing between containers
4. Local host folder mapping to a container volume
5. Local host file mapping to a container
6. Docker copy files into a container, both running and stopped
7. Docker container inspect does not return all container network for a container
## Workflow Guidelines
Anything that can be performed with Docker Compose can be performed manually via the Docker CLI and via scripting using the CLI. This makes Compose a good baseline reference and our guidelines will use it for demonstration purposes. Our guideline uses Docker Compose 1.8.1. The list above in the Future Features section puts constraints on what types of containerized application can be deployed on VIC 0.7.0.
Please note, these guidelines and recommendations exist for the current feature set in VIC 0.7.0. As VIC approaches 1.0, many of these constraints will go away.
#### Guidelines for Building Container Images
The current lack of docker build and registry pushing means users will need to use regular Docker to build a container and to push it to the global hub or your corporate private registry. The example workflow using Docker's voting app will illustrate how to get around this constraint.
#### Guidelines for Sharing Config
VIC 0.7.0 current lack of data volume sharing and docker copy will put constraints on how configuration are provided to a containerized application. An example of configuration is your web server config files. Our recommendation for getting around the current limitation is to pass in configuration via command line arguments or environment variables. Add a script to the container image that ingest the command line argument/environment variable and pass these configuration to the contained application. A benefit of using environment variables to transfer configuration is the containerized app will more closely follow the popular 12-factor app model.
With no direct support for sharing volumes between containers processes that must share files have the following options:
1. build them into the same image and run in the same container
2. add a script to the container that mounts an NFS share (containers must be on the same network)
a. Run container with NFS server sharing a data volume
b. Mount NFS share in whichever containers need to share
TODO: Provide example of both
## Example Applications
We have taken Docker's voting app example and used the above guidelines to modify it for use on VIC 0.7.0. Please follow to this [page](voting_app.md) for more information.

14
vendor/github.com/vmware/vic/doc/application_workflow/harbor_registry/README.md generated vendored Normal file
View File

@@ -0,0 +1,14 @@
# Using vSphere Integrated Container Engine with VMware's Harbor
In this example, we will install VMware's Harbor registry and show how to get vSphere Integrated Container Engine (VIC Engine) 0.8.0 working with Harbor. With 0.8.0, the engine does not have an install-time mechanism to set up a self-signed certificate so we will show the manual steps for post-install setup as a workaround. We will not show how to setup Harbor with LDAP. For that, the reader may visit the [Harbor documentation](https://github.com/vmware/harbor/tree/master/docs) site for more information. Since there is a lot of documentation on the Harbor site for various setup, we will focus on setting up Harbor with a self-signed certificate and setting up VIC Engine to work with this Harbor instance.
## Prerequisite
The following example requires a vCenter installation.
Note: Certificate verification requires all machines using certificates are time/date accurate. This can be achieved using several options, suchas, vSphere web client, vSphere thick client for Windows or govc. In the following, we deploy this example on a vCenter where all ESXi hosts in the cluster have been set up with NTP and were sync'd prior to installing VIC Engine or Harbor.
## Workflows
1. [Deploy a VCH to use with Harbor](deploy_vch_with_harbor.md)
2. [Post-Install Usage](post_install_usage.md)

76
vendor/github.com/vmware/vic/doc/application_workflow/harbor_registry/deploy_vch_with_harbor.md generated vendored Normal file
View File

@@ -0,0 +1,76 @@
# Deploying vSphere Integrated Container Engine with Harbor
## Prerequisite
Harbor requires 60GB or more free space on your datastore.
## Workflow
We will use VIC Engine 0.8.0 and Harbor 0.5.0 for this example. We will use Ubuntu as OS on our user machine.
If no server certificate and private key are provided during installation, Harbor will self generate these. It will also provide a self-generated CA (certificate authority) certificate if no server certificate and private key are provided during installation. The OVA installation guide for Harbor can be found in the [Harbor docs](https://github.com/vmware/harbor/blob/master/docs/installation_guide_ova.md). Harbor requires both an IP address and FQDN (fully qualified domain name) for the the server. There is also a DHCP install method available for debugging purposes, but it is not a recommended production deployment model.
We will assume a Harbor instance has been installed without server certificate and private key. We will also assume we have downloaded the CA cert using the Harbor instuctions. The last steps left to get Harbor working with vSphere Integrated Container Engine is to update standard docker with the Harbor CA cert and deploy a new VCH with the CA cert. The instructions are provided below.
<br><br>
## Update the user working machine with the CA.crt for standard docker
We must update the standard docker on our laptop so it knows of our CA certificate. Docker can look for additional CA certificates outside of the OS's CA bundle folder if we put new CA certificates in the right location, documented [here](https://docs.docker.com/engine/security/certificates/).
We create the necessary folder, copy our CA cert file there, and restart docker. This should be all that is necessary. We take the additional steps to verify that we can log onto our Harbor server.
```
loc@Devbox:~/mycerts$ sudo su
[sudo] password for loc:
root@Devbox:/home/loc/mycerts# mkdir -p /etc/docker/certs.d/<Harbor FQDN>
root@Devbox:/home/loc/mycerts# mkdir -p /etc/docker/certs.d/<Harbor IP>
root@Devbox:/home/loc/mycerts# cp ca.crt /etc/docker/certs.d/<Harbor FQDN>/
root@Devbox:/home/loc/mycerts# cp ca.crt /etc/docker/certs.d/<Harbor IP>/
root@Devbox:/home/loc/mycerts# exit
exit
loc@Devbox:~/mycerts$ sudo systemctl daemon-reload
loc@Devbox:~/mycerts$ sudo systemctl restart docker
loc@Devbox:~$ docker logout <Harbor FQDN>
Remove login credentials for <Harbor FQDN>
loc@Devbox:~$ docker logout <Harbor IP>
Remove login credentials for <Harbor IP>
loc@Devbox:~$ docker login <Harbor FQDN>
Username: loc
Password:
Login Succeeded
loc@Devbox:~$ docker login <Harbor IP>
Username: loc
Password:
Login Succeeded
loc@Devbox:~$ docker logout <Harbor FQDN>
Remove login credentials for <Harbor FQDN>
loc@Devbox:~$ docker logout <Harbor IP>
Remove login credentials for <Harbor IP>
```
Notice we create folders for both FQDN and IP in the docker cert folder and copy the CA cert to both. This will allow us to log into the Harbor from Docker using both FQDN and IP address.
<br><br>
## Install a VCH with the new CA certificate
In this step, we deploy a VCH and specify our CA cert via a --registry-ca parameter in vic-machine. This parameter is a list, meaning we can easily add multiple CA certs by specifying multiple --registry-ca parameters.
For simplicity, we will install a VCH with the --no-tls flag. This indicates we will not need TLS from a docker CLI to the VCH. This does NOT imply that access to Harbor will be performed without TLS.
```
root@Devbox:/home/loc/go/src/github.com/vmware/vic/bin# ./vic-machine-linux create --target=<vCenter_IP> --image-store="vsanDatastore" --name=vic-docker --user=root -password=<vCenter_password> --compute-resource="/dc1/host/cluster1/Resources" --bridge-network DPortGroup --force --no-tls --registry-ca=ca.crt
WARN[2016-11-11T11:46:37-08:00] Configuring without TLS - all communications will be insecure
...
INFO[2016-11-11T11:47:57-08:00] Installer completed successfully
```
<br>
Proceed to [Post-Install Usage](post_install_usage.md) for examples of how to use this deployed VCH with Harbor.

72
vendor/github.com/vmware/vic/doc/application_workflow/harbor_registry/post_install_usage.md generated vendored Normal file
View File

@@ -0,0 +1,72 @@
# Using vSphere Integrated Container Engine with Harbor
Here we show an example of using a deployed VCH with Harbor as a private registry. We assume that one has been setup using either static IP or FQDN. We also assume standard docker has been updated with the certificate authority cert that can verify the deployed Harbor's server cert.
<br><br>
## Workflow
1. Develop or obtain a docker container image on a computer (or terminal) using standard docker. Tag the image for Harbor and push the image to the server.
2. Pull down the image from Harbor to a deployed VCH and use it.
<br><br>
## Push a container image to Harbor using standard docker
In this step, we pull the busybox container image from the docker hub down to our laptop, which had the CA certificate updated for docker use earlier. Then we tag the image for uploading to our Harbor registry and push the image up to it. Please note, we log onto the Harbor server before pushing the image up to it.
```
loc@Devbox:~/mycerts$ docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
56bec22e3559: Pull complete
Digest: sha256:29f5d56d12684887bdfa50dcd29fc31eea4aaf4ad3bec43daf19026a7ce69912
Status: Downloaded newer image for busybox:latest
loc@Devbox:~/mycerts$
loc@Devbox:~/mycerts$ docker tag busybox <Harbor FQDN or static IP>/test/busybox
loc@Devbox:~/mycerts$ docker login <Harbor FQDN or static IP>
Username: loc
Password:
Login Succeeded
loc@Devbox:~/mycerts$ docker push <Harbor FQDN or static IP>/test/busybox
The push refers to a repository [<Harbor FQDN or static IP>/test/busybox]
e88b3f82283b: Pushed
latest: digest: sha256:29f5d56d12684887bdfa50dcd29fc31eea4aaf4ad3bec43daf19026a7ce69912 size: 527
```
## Pull the container image down to the VCH
Now, in another terminal, we can pull the image from Harbor to our VCH.
```
loc@Devbox:~$ export DOCKER_HOST=tcp://<Deployed VCH IP>:2375
loc@Devbox:~$ export DOCKER_API_VERSION=1.23
loc@Devbox:~$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
loc@Devbox:~$ docker pull <Harbor FQDN or static IP>/test/busybox
Using default tag: latest
Pulling from test/busybox
Error: image test/busybox not found
loc@Devbox:~$ docker login <Harbor FQDN or static IP>
Username: loc
Password:
Login Succeeded
loc@Devbox:~$ docker pull <Harbor FQDN or static IP>/test/busybox
Using default tag: latest
Pulling from test/busybox
56bec22e3559: Pull complete
a3ed95caeb02: Pull complete
Digest: sha256:97af7f861fb557c1eaafb721946af5c7aefaedd51f78d38fa1828d7ccaae4141
Status: Downloaded newer image for test/busybox:latest
loc@Devbox:~$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
<Harbor FQDN or static IP>/test/busybox latest e292aa76ad3b 5 weeks ago 1.093 MB
loc@Devbox:~$
```
Note above, on our first attempt to pull the image down, it failed, with a 'not found' error message. Once we log into the Harbor server, our attempt to pull down the image succeeds.

BIN
vendor/github.com/vmware/vic/doc/application_workflow/images/elgg.png generated vendored Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 58 KiB

102
vendor/github.com/vmware/vic/doc/application_workflow/private_registry.md generated vendored Normal file
View File

@@ -0,0 +1,102 @@
# Running a private registry with VIC
In this example, we will run a private Docker registry on VIC and push and pull images using VIC. VMware also offers an enterprise ready registry named [Harbor](https://github.com/vmware/harbor) that can be used in place of the base Docker registry.
## Prerequisite
Before going through this example, we need to re-emphasize some concepts around Docker and VIC. The following examples are shown using Linux. For Windows and Mac users, these examples should not differ much.
Installing VIC will also require installing Docker locally. When Docker is installed, we will get both a client (CLI or command line interface) and a daemon that handles all local container operations. Local containers are those that run on the user local machine instead of a VMWare vSphere/ESXi environment. The CLI is important as it will be most user's touchpoint for working with containers on VIC and on their local system. The distinction between using the CLI against the two environment is very important in this example. By default, the CLI will use the local Docker daemon. After setting some environment variables, the CLI can be instructed to send all operations to VIC instead of the local Docker daemon. The two environment variable are DOCKER_HOST and DOCKER_API_VERSION.
In this example, we are deploying an insecure registry with no authentication for simplicity. We will also be targeting an ESXi environment.
## Workflow
In terminal #1: (local Docker)
1. Open a terminal and make sure that it will use the local Docker daemon. At a command prompt, issue
```
$> unset DOCKER_HOST
```
2. Install a VCH for a private registry using vic-machine
3. Run [Docker's registry](https://docs.docker.com/registry/) on the first VCH
4. Install a second VCH for running applications, making sure to specify --insecure-registry to ensure this second VIC can pull images from the insecure registry in the first VCH.
5. At a terminal command prompt, using regular Docker, tag the images to be destined for the registry.
6. Modify the docker systemd config file to allow pushing to an insecure registry
7. Restart the docker daemon
8. Push the image using the full tagged name (including host IP and port)
In terminal #2: (VIC VCH)
1. Open a terminal and make sure it is using the second VCH. At a command prompt, issue
```
$> export DOCKER_HOST=tcp://<VCH_IP>:<VCH_PORT>
$> export DOCKER_API_VERSION=1.23
```
2. Pull the image from the registry VCH
### Example run
terminal 1:
```
$> unset DOCKER_HOST
$> ./vic-machine-linux create --target=192.168.218.207 --image-store=datastore1 --name=vic-registry --user=root --password=vagrant --compute-resource="/ha-datacenter/host/esxbox.localdomain/Resources" --bridge-network=vic-network --no-tls --volume-store=datastore1/registry:default --force
...
INFO[2016-10-08T17:31:06-07:00] Initialization of appliance successful
INFO[2016-10-08T17:31:06-07:00]
INFO[2016-10-08T17:31:06-07:00] vic-admin portal:
INFO[2016-10-08T17:31:06-07:00] http://192.168.218.138:2378
INFO[2016-10-08T17:31:06-07:00]
INFO[2016-10-08T17:31:06-07:00] Docker environment variables:
INFO[2016-10-08T17:31:06-07:00] DOCKER_HOST=192.168.218.138:2375
INFO[2016-10-08T17:31:06-07:00]
INFO[2016-10-08T17:31:06-07:00]
INFO[2016-10-08T17:31:06-07:00] Connect to docker:
INFO[2016-10-08T17:31:06-07:00] docker -H 192.168.218.138:2375 info
INFO[2016-10-08T17:31:06-07:00] Installer completed successfully
$> DOCKER_HOST=tcp://192.168.218.138:2375 DOCKER_API_VERSION=1.23 docker run -d -p 5000:5000 --name registry registry:2
$> ./vic-machine-linux create --target=192.168.218.207 --image-store=datastore1 --name=vic-app --user=root --password=vagrant --compute-resource="/ha-datacenter/host/esxbox.localdomain/Resources" --bridge-network=vic-network --no-tls --volume-store=datastore1/vic-app:default --force --insecure-registry 192.168.218.138
...
INFO[2016-10-08T17:31:06-07:00] Initialization of appliance successful
INFO[2016-10-08T17:31:06-07:00]
INFO[2016-10-08T17:31:06-07:00] vic-admin portal:
INFO[2016-10-08T17:31:06-07:00] http://192.168.218.131:2378
INFO[2016-10-08T17:31:06-07:00]
INFO[2016-10-08T17:31:06-07:00] Docker environment variables:
INFO[2016-10-08T17:31:06-07:00] DOCKER_HOST=192.168.218.131:2375
INFO[2016-10-08T17:31:06-07:00]
INFO[2016-10-08T17:31:06-07:00]
INFO[2016-10-08T17:31:06-07:00] Connect to docker:
INFO[2016-10-08T17:31:06-07:00] docker -H 192.168.218.131:2375 info
INFO[2016-10-08T17:31:06-07:00] Installer completed successfully
$> sudo vi /lib/systemd/system/docker.service
$> sudo systemctl daemon-reload
$> sudo systemctl restart docker
$> docker tag busybox 192.168.218.138:5000/test/busybox
$> docker push 192.168.218.138:5000/test/busybox
```
terminal 2:
```
$> export DOCKER_HOST=tcp://192.168.218.131:2375
$> export DOCKER_API_VERSION=1.23
$> docker pull 192.168.218.138:5000/test/busybox
```
Note, in this example, we disabled TLS for simplicity. Also, we did not show what was modified in /lib/systemd/system/docker.service. That is shown below for the example above.
```
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd --tls=false -H fd:// --insecure-registry 192.168.218.138:5000
```
In the second step, we specify the necessary environment variables before the docker run command. On our Linux machine, this sets the variables for the duration of the operation, and once the docker run finishes, those variables are reverted to their previous values. We use the registry:2 image. It is important not to specify registry:2.0 in this example. Registry 2.0 has issues that prevents the example above from working.

106
vendor/github.com/vmware/vic/doc/application_workflow/voting_app.md generated vendored Normal file
View File

@@ -0,0 +1,106 @@
# Voting App on VIC
The [voting app](https://github.com/docker/example-voting-app) is one of Docker's example multi-tiered app. We will neither discuss the design of the app nor explain how it works. Instead, we will focus on it's Docker Compose yml file and use the [guidelines](README.md) to modify that yml file to make it work on VIC 0.7.0. You can find the modified compose yml file in the folder [vic/demos/compose/voting-app/](../../demos/compose/voting-app). We have only included the modified yml file as the below workflow will use the original source from github.
## Workflow
### Original Compose File
```
version: "2"
services:
vote:
build: ./vote
command: python app.py
volumes:
- ./vote:/app
ports:
- "5000:80"
redis:
image: redis:alpine
ports: ["6379"]
worker:
build: ./worker
db:
image: postgres:9.4
result:
build: ./result
command: nodemon --debug server.js
volumes:
- ./result:/app
ports:
- "5001:80"
- "5858:5858"
```
We see the above compose file are using two features that are not yet supported in VIC 0.7.0. The first is docker build. The second is local folder mapping to container volume. Let's walk through modifying this app and deploy it onto a vSphere environment.
### Getting the App Prepared
First, clone the repository from github. Note, we have included the modified compose file in our /demos folder, but in this exercise, we are going to modify this app from the sources from github.
Second, to get around the docker build directive, we follow the previously mentioned guidelines and use regular docker to build each component that requires a build. Then we tag the the images to upload to our private registry (or private account on Docker Hub). In this example, we are going to use VMWare's victest account on Docker Hub. You will not be able to use this account, but you can create your own and use that in place of the victest keywords below. Please note, the steps shown below are performed in a terminal using regular docker (as opposed to VIC's docker personality daemon). Note, it is possible to build and tag an image in one step. Below, the steps are broken into separate steps.
**build the images:**
$> cd example-voting-app
$> docker build -t vote ./vote
$> docker build -t vote-worker ./worker
$> docker build -t vote-result ./result
**tag the images for a registry:**
$> docker tag vote victest/vote
$> docker tag vote-worker victest/vote-worker
$> docker tag vote-result victest/vote-result
**push the images to the registry:**
$> docker login (... and provide credentials)
$> docker push victest/vote
$> docker push victest/vote-worker
$> docker push victest/vote-result
Second, we analyze the application. There doesn't appear to be a real need to map the local folder to the container volume so we remove the local folder mapping. We also remove all the build directives from the yml file.
### Updated Compose File for VIC 0.7.0
```
version: "2"
services:
vote:
image: victest/vote
command: python app.py
ports:
- "5000:80"
redis:
image: redis:alpine
ports: ["6379"]
worker:
image: victest/vote-worker
db:
image: postgres:9.4
result:
image: victest/vote-result
command: nodemon --debug server.js
ports:
- "5001:80"
- "5858:5858"
```
### Deploy to Your VCH
We assume a VCH has already been deployed with vic-machine and VCH_IP is the IP address of the deployed VCH. This IP should have been presented after the VCH was successfully installed. We also assume we are still in the example-voting-app folder, with the modified compose yml file.
$> docker-compose -H VCH_IP up -d
Now, use your web browser to navigate to "http://VCH_IP:5000" and "http://VCH_IP:5001" to verify the voting app is running.
That's really all there is to deploying **this** app. It is a contrived app and more complex containerized apps may have more steps to perform before it will on VIC 0.7.0.

136
vendor/github.com/vmware/vic/doc/application_workflow/webserving.md generated vendored Normal file
View File

@@ -0,0 +1,136 @@
# Web-serving on vSphere Integrated Containers Engine
We take the Web-serving benchmark from CloudSuite (http://cloudsuite.ch/webserving/) as an example, to demonstrate how customers who are interested in the LEMP implementation of a web-serving application could deploy it on vSphere Integrated Containers Engine 0.7.0 using Docker Compose. This demo has three tiers deployed on three containerVMs: an Nginx Web server, a Memcached server, and a MySQL database server. The Web server runs Elgg (a social networking engine) and connects the Memcached server and the database server through the network.
## Workflow
### Build docker image for the Web server (on regular docker)
Note, in the original web-server docker image from Cloudesuite, the email verification for new user is not enabled. This demo is here for illustration only. **You can also skip this section and proceed to "[Compose File for vSphere Integrated Containers Engine](#compose-file-for-vsphere-integrated-containers-engine)" if you do not want to build your own image**.
Step I:
Download the original installation files from https://github.com/ParsaLab/cloudsuite/tree/master/benchmarks/web-serving/web_server
Step II:
In the Dockerfile, add “Run apt-get install –y sendmail” and “EXPOSE 25”
Step III:
Replace “bootstrap.sh” with the following:
```
#!/bin/bash
hname=$(hostname)
line=$(cat /etc/hosts | grep '127.0.0.1')
line2=" web_server web_server.localdomain"
sed -i "/\b\(127.0.0.1\)\b/d" /etc/hosts
echo "$line $line2 $hname" >> /etc/hosts
cat /etc/hosts
service sendmail stop
service sendmail start
service php5-fpm restart
service nginx restart
```
Step IV: (In this example, we will deploy the image to the docker hub.)
- Build the image:
```
$> docker build -t repo/directory:tag .
```
- Login to your registry: (we use the default docker hub in this example)
```
$> docker login (input your credentials when needed)
```
- upload your image:
```
$> docker push repo/directory:tag
```
### Build docker image for the MySQL server (on regular docker)
This example application uses the database to store the address of the Web server. The original Dockerfile from Cloudsuite populates this with "http://web_server:8080", which is not usable in production. Using the suggestions we provided earlier, we modify the execute.sh script to replace the "web_server" text with the actual IP of our VCH. This script is executed when this database container is executed. In the Docker Compose file, we specify the IP address of our target VCH. You will see that in the modified compose yml file below.
This example illustrates passing config in via environment variables and having the script use those values to modify internal config in the running container. Another option is to use a script and command line arguments to pass config to a containerized app. Below, we will modify the Dockerfile and script. **You can also skip this section and proceed to "[Compose File for vSphere Integrated Containers Engine](#compose-file-for-vsphere-integrated-containers-engine)" if you do not want to build your own image**.
Step I:
Download the original installation files from https://github.com/ParsaLab/cloudsuite/tree/master/benchmarks/web-serving/db_server
Step II:
In the Dockerfile, comment out the following lines:
```
ENV web_host web_server
RUN sed -i -e"s/HOST_IP/${web_host}:8080/" /elgg_db.dump
CMD bash -c "/execute.sh ${root_password}"
```
Step III:
Replace “files/execute.sh” with the following:
```
#!/bin/bash
set -x
service mysql restart
# Wait for mysql to come up
while :; do mysql -uroot -p${root_password} -e "status" && break; sleep 1; done
mysql -uroot -p$root_password -e "create database ELGG_DB;"
bash -c 'sed -i -e"s/HOST_IP/${web_host}:8080/" /elgg_db.dump'
cat /elgg_db.dump | grep 8080
# Need bash -c for redirection
bash -c "mysql -uroot -p$root_password ELGG_DB < /elgg_db.dump"
mysql -uroot -p$root_password -e "GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '$root_password' WITH GRANT OPTION; FLUSH PRIVILEGES;"
service mysql stop
/usr/sbin/mysqld
```
Step IV: Same as Step IV when creating the docker image for the Web server.
### Compose File for vSphere Integrated Containers Engine
```
version: '2'
networks:
my_net:
driver: bridge
services:
web_server:
image: victest/web_elgg
container_name: web_server
networks:
- my_net
ports:
- "8080:8080"
mysql_server:
image: victest/web_db
container_name: mysql_server
command: [bash, -c, "/execute.sh"]
networks:
- my_net
environment:
- web_host=192.168.60.130 # This is the VCH_IP
- root_password=root # Password for the root user
memcache_server:
image: cloudsuite/web-serving:memcached_server
container_name: memcache_server
networks:
- my_net
```
### Deploy to Your VCH
Once you already have a VCH deployed by vic-machine, go to the folder where you have the above “docker-compose.yml” file and execute the following command to start the Web-serving application:
```
$> docker-compose -H VCH_IP:VCH_PORT up –d
```
Here VCH_IP and VCH_PORT can be found from the standard output when you use “vic-machine create” to launch the VCH. Now we are ready to view the website. Open a browser and navigate to http://VCH_IP:8080. We make sure to use the IP address of the VCH we deployed on as the IP of our Web server here. You should be able to see the following page:
![Web serving demo](images/elgg.png)
You can login in as the admin user (username: admin; password: admin1234), or register as a new user with a valid email address (Gmail does not work). You can also create your own content, invite friends, or chat with others. Enjoy!

14
vendor/github.com/vmware/vic/doc/bundle/NOTICE generated vendored Normal file
View File

@@ -0,0 +1,14 @@
NOTICE
vSphere Integrated Containers Engine
Copyright (c) 2016-2017 VMware, Inc. All Rights Reserved.
This product is licensed to you under the Apache License, Version 2.0 (the
"License"). You may not use this product except in compliance with the
License.
This product may include a number of components with separate copyright
notices and license terms. Your use of the source code for these components is
subject to the terms and conditions of the component's license, as noted in
the LICENSE file.

124
vendor/github.com/vmware/vic/doc/bundle/OVA-spec.md generated vendored Normal file
View File

@@ -0,0 +1,124 @@
# VIC Unified Management OVA specification
<!-- TOC -->
- [VIC Unified Management OVA specification](#vic-unified-management-ova-specification)
- [Appliance Specification](#appliance-specification)
- [Filesystem layout](#filesystem-layout)
- [Network layout](#network-layout)
- [Components logging](#components-logging)
- [Components startup / shutdown](#components-startup--shutdown)
- [OVF Properties list](#ovf-properties-list)
- [Appliance level properties](#appliance-level-properties)
- [Harbor](#harbor)
- [Admiral](#admiral)
- [Upgrade Strategy](#upgrade-strategy)
- [Upgrade from previous Harbor-only OVA](#upgrade-from-previous-harbor-only-ova)
- [Future version upgrade](#future-version-upgrade)
<!-- /TOC -->
**User Statement**: As a customer of vSphere Integrated Containers, I want an easy way to install all the product components on my vSphere infrastructure, with a single download, and using a familiar workflow that leverages my competencies on vSphere. When an upgrade or a patch is released, I want to follow a simple guide to perform the upgrade of one or more components of vSphere Integrated Containers.
**Details**: to accomplish these goals, we need to create a single OVA that installs all the VIC components and performs their configuration using OVF properties, resulting in a running service that can be accessed by the vSphere admin for further configuration.
**Acceptance Criteria**:
1. The vSphere administrator downloads the VIC Management OVA from a known location
2. The vSphere administrator deploys the VIC Management OVA from the vSphere client, provides values for Harbor and Admiral configuration options
3. The deployment should also tie the Harbor and Admiral instances together
4. Once deployed the vSphere administrator should be able to
- Access the Admiral / Harbor integrated UI
- SSH into the machine to run vic-machine commands
5. The vSphere administrator should also be able to download the VIC Engine tarball from the OVA.
## Appliance Specification
The Appliance is based on Photon OS, version 1.0 (build 62c543d), it's built using [packer](https://packer.io) and produces two main artifacts, one for local development (packaged as a Vagrant box) and a releasable OVA artifact.
The OVA appliance build process is codified in a JSON document that contains the packer specification for the build, the build will be conducted as part of the existing CI system in use by the VIC Engine project, it will also be possible to build locally on the developer workstation to aid with the development.
All the components integrated in the appliance (Harbor, Admiral, Lightwave) will be run as Docker containers, using Docker runtime version 1.12.6, Docker Compose will also be available to orchestrate discrete services inside the components if needed.
Use of non-contained processes inside the appliance is severely discouraged.
## Filesystem layout
The appliance will ship with two disks, internally named `system` and `data`, `system` only contains the operating system and the docker containers of the services, `data` will hold all the user-generated data and data that has to be persisted by the components.
The data disk will be initialized on first boot using a fixed size, the filesystem will be automatically expanded by a startup script if the underlying disk size has changed (e.g. the operator shuts down the VM and increase the size of the data disk from vSphere), this will allow the operator to extend the size of the components data store up to the limit of a single VMDK.
## Network layout
The appliance will have a single IP, which will be configured using OVF properties upon deployment, network changes will be automatically picked up by the appliance startup script if the operator changes OVF properties post-deployment.
In order to ease deployment and lower install complexity, all the services will be run on different ports, there will be no effort to tie them together using a reverse proxy.
## Components logging
All the docker containers will log locally to syslog, using the docker syslog driver, this will give a familiar environment for VMware admins to troubleshoot issues with the appliance.
## Components startup / shutdown
Startup and shutdown of the components will be performed with systemd units, each component will use a single systemd unit, components that require service orchestration must use docker-compose to specify relations and startup/shutdown order.
## OVF Properties list
The OVF properties list includes all the fields that the user has to input upon deployment for initial configuration, all the parameters (except when noted) are changeable post deployment by accessing the VM properties on vCenter, startup scripts will reconfigure the appliance with the new parameters.
### Appliance level properties
- Root Password - `appliance.root_pwd`: The initial password of the root user. Subsequent changes of password should be performed in operating system. (8-128 characters)
- Permit Root Login - `appliance.permit_root_login`: Specifies whether root user can log in using SSH. (disabled by default)
- Network IP Address - `network.ip0`: The IP address of this interface. Leave blank if DHCP is desired.
- Network Netmask - `network.netmask0`: The netmask or prefix for this interface. Leave blank if DHCP is desired.
- Default Gateway - `network.gateway`: The default gateway address for this VM. Leave blank if DHCP is desired.
- Domain Name Servers - `network.DNS`: The domain name server IP Address for this VM(comma separated). Leave blank if DHCP is desired.
- Domain Search Path - `network.searchpath`: The domain search path(comma or space separated domain names) for this VM. Leave blank if DHCP is desired.
- Domain Name - `network.domain`: The domain name of this VM. Run command man resolv.conf for more explanation. Leave blank if DHCP is desired or the domain name is not needed for static IP.
- Email Server - `appliance.email_server`: The mail server to send out emails.
- Email Server Port - `appliance.email_server_port`: The port of mail server.
- Email Username - `appliance.email_username`: The user from whom the password reset email is sent. Usually this is a system email address.
- Email Password - `appliance.email_password`: The password of the user from whom the password reset email is sent.
- Email From - `appliance.email_from`: The name of the email sender.
- Email SSL `appliance.email_ssl`: Whether to enable secure mail transmission.
### Harbor
- Deploy - `harbor.deploy`: Specifies whether Harbor is enabled on the appliance.
- Port - `harbor.port`: The port on which Harbor will bind the NGINX frontend, defaults to 443
- Harbor Admin Password - `harbor.admin_password`: The initial password of Harbor admin. It only works for the first time when Harbor starts. It has no effect after the first launch of Harbor. Change the admin password from UI after launching Harbor. (8-20 characters)
- Database Password - `harbor.db_password`: The initial password of the root user of MySQL database. Subsequent changes of password should be performed in operating system. (8-128 characters)
- Garbage Collection - `harbor.gc_enabled`: When setting this to true, Harbor performs garbage collection everytime it boots up.
- SSL Cert - `harbor.ssl_cert`: Paste in the content of a certificate file. Leave blank for a generated self-signed certificate.
- SSL Cert Key - `harbor.ssl_cert_key`: Paste in the content of a certificate key file. Leave blank for a generated key.
### Admiral
- Deploy - `admiral.deploy`: Specifies whether Admiral is enabled on the appliance.
- Port - `admiral.port`: The (secure) port on which Admiral will bind the HTTP service, defaults to 8443
- SSL Cert - `admiral.ssl_cert`: Paste in the content of a certificate file. Leave blank for a generated self-signed certificate.
- SSL Cert Key - `admiral.ssl_cert_key`: Paste in the content of a certificate key file. Leave blank for a generated key.
## Upgrade Strategy
### Upgrade from previous Harbor-only OVA
Upgrade from previous Harbor-only OVA will consist in attaching both the system and data disk from the old Harbor-only OVA as additional disks of the newly deployed VIC OVA.
The procedure will roughly look like this:
1. Shutdown of the old Harbor-only Appliance.
2. Deploy of the new VIC Appliance.
- New VIC Appliance deployed with all the old credentials (DB Password, Admin Password, etc...)
3. User will attach both disks from the Harbor-only Appliance to the newly deployed VIC Appliance as SCSI 0:2 and SCSI 0:3 (order is not important).
4. During bootup, a systemd unit will recognize the external data and, if there is no new data existing in the VIC appliance, will proceed with the migration procedure.
5. The migration procedure will copy the data over to the new data disk.
6. A Migration script provided by the Harbor team will perform some data transformation on the migrated data and will proceed with the customization of the configuration (which is stored in the system disk of the old Harbor-only appliance).
7. The VI Admin will follow a progress bar on the console of the VM that will show progress of the procedure.
8. Once the procedure is done, a message will appear, asking the user to shutdown and detach the old Harbor-only disks from the new OVA.
9. The user will then start the new VIC Appliance again and the procedure will be complete.
### Future version upgrade
Upgrade strategy (to subsequent version of the VIC Appliance) will include moving the data disk to a newly deployed OVA, the startup scripts will recognize existing data on the data disk and new versions of the components will start up with the previous data attached and available, it will be up to the individual components to do perform any application-level upgrade procedure (i.e. database schema updates) going forward.

1
vendor/github.com/vmware/vic/doc/bundle/README generated vendored Normal file
View File

@@ -0,0 +1 @@
Go to https://github.com/vmware/vic for more information

14
vendor/github.com/vmware/vic/doc/design/README.md generated vendored Normal file
View File

@@ -0,0 +1,14 @@
# vSphere Integrated Containers - Architecture and Design
The bulk of the design notes currently relate to [components and their role in the system](components.md) - high level descriptions primarily.
Networking is broken out into a subfolder on the expectation that it will become a significantly larger area once NSX integration is addressed - [the 1.0 MVP network design](networking/README.md) does not encompass NSX.
Documentation about component interactions is ongoing with the initial docs being:
* [configuration](configuration.md)
* [communication between VCH appliance and containerVMs](communications.md)
* portlayer component communication
* security
* [installation and self-provisioning - usage examples](../user/usage.md)
* [installation and self-provisioning - technical](vic-machine.md)

18
vendor/github.com/vmware/vic/doc/design/application-topologies.md generated vendored Normal file
View File

@@ -0,0 +1,18 @@
# Application topologies
This describes the core application topologies that will be used to validate function and determine relative priority of feature implementation. This is by no means a complete list and the ordering may change over time as we gain additional feedback from users.
## Prime topology - platform 2.5 tiered application
### Description
This is the topology currently being used as the primary touchstone for feature support and priority. It's based off [the docker voting app](https://github.com/docker/example-voting-app) as a great example of a mapping from a traditional tiered application to a containerized environment. This is not assumed to be a pure [twelve factor app](http://12factor.net/) because while that significantly simplifies the infrastructure requirements it's not a viable assumption for all workloads.
There are two scenarios described in the breakdown:
1. unmodified voting app
2. voting app using non-containerized database with direct exposure to an external network on the front-end
[Application workflow](https://github.com/vmware/vic/doc/application_workflow) has examples on building these applications. The [voting app](https://github.com/vmware/vic/doc/application_workflow/voting_app.md) illustrates scenario 1, discussed above. In this example, the entire voting app and all tiers are created as a self-enclosed set of containers. Docker-compose is used to orchestrate the deployment.
The second scenario is assumed to be a better reflection of probable deployment scenarios as people are moving to containerized workloads, hence the description of this as a 2.5 platform application.
TODO: Add an example to the application workflow docs illustrating scenario 2.

21
vendor/github.com/vmware/vic/doc/design/arch/arch.md generated vendored Normal file
View File

@@ -0,0 +1,21 @@
### vSphere Integrated Containers Architecture
#### Overview
VIC is a product designed to tightly integrate container workflow, lifecycle and provisioning with the vSphere SDDC. In VIC, a container is a hardware-virtualized first-class citizen on the hypervisor provisioned into a _Virtual Container Host_ (VCH) and able to directly integrate with vSphere infrastructure capabilities, such as networking and storage features.
[Learn more about the differences between the VIC model and a traditional software-virtualized container](vic-container-abstraction.md)
The architecture of VIC is designed to allow for significant modularity and flexibility and includes the following key components:
##### Port Layer Abstractions
vSphere currently lacks the notion of container primitives and abstractions through which they can be manipulated. It has a rich API with bindings for various languages (Eg. [govmomi](https://github.com/vmware/govmomi)) but these are all necessarily oriented around the notion of a VM.
While it would be possible to write a rudimentary VIC-like container engine by driving the vSphere APIs directly from within a daemon of some kind, the tight coupling between the low-level vSphere calls and the high-level daemon API would result in very little re-usable code and monolith that's potentially difficult to maintain. An API layer that encapsulates low-level container primitives that is both container engine and operating system agnostic would be preferable.
A secondary benefit of such an API is that it could easily be extended for compatibility with emerging standards which operate at a similar layer, such as [runc](https://github.com/opencontainers/runc).
The Port Layer is designed in such a way that the libraries can be built into static binaries or remotable services. They can be combined together into a single service endpoint or distributed for greater flexibility.
[Learn more about the Port Layer](vic-port-layer-overview.md)

4
vendor/github.com/vmware/vic/doc/design/arch/component_architecture.md generated vendored Normal file
View File

@@ -0,0 +1,4 @@
# self-provisioning proxy
# vmomi proxy agent

58
vendor/github.com/vmware/vic/doc/design/arch/vic-container-abstraction.md generated vendored Normal file
View File

@@ -0,0 +1,58 @@
### The VIC Container Abstraction
VIC provisions containers _as_ VMs, rather than _in_ VMs. In understanding the VIC container abstraction, it is helpful to compare and contrast against the virtualization of a traditional container host.
#### Traditional Container Host
Let's take a Linux VM running Docker as an example. The container host is a VM running a Linux OS with the necessary libraries, kernel version and daemon installed. The container host will have a fixed amount of memory and vCPU resource that can be used by the containers provisioned into it.
The container host operating system along with the Docker daemon have to provide the following:
* **The control plane** - an endpoint through which control operations are performed, executing in the same OS as its controlling
* **A container abstraction** - library extensions to the guest OS need to provide a private namespace and resource constraints
* **Network virtualization** - simple bridge networking or overlay networking
* **Layered filesystem** - not an absolute requirement for a container, but typically conflated in most implementations
* **OS Kernel** - a dependency for the container executable to execute on, typically shared between containers
The hypervisor in this mode provides hardware virtualization of the entire container host VM, one or more VMDKs providing local disk for the OS, one or more vNICs to provide network connectivity for the OS and possibly paravirtualization capabilities allowing the containers to directly access hypervisor infrastructure.
#### The VIC model
VIC containers operate quite differently. In the above model, it would be reasonable to describe a container as being run _in_ a VM. In the VIC model, a container is run _as_ a VM. For the purposes of this project, we will refer to this as a _containerVM_.
So what does this mean in practice? Well, firstly a container host isn't a VM, it's a resource pool - this is why we call it a _Virtual_ Container Host. It's an abstract dynamically-configurable resource boundary into which containers can be provisioned. As for the other functions highlighted above:
* **The control plane** - functionally the same endpoint as above, but controlling vSphere and running in its own OS
* **A container abstraction** - is a VM. A VM provides resource constraints and a private namespace, like a container
* **Network virtualization** - provided entirely by vSphere. NSX, distributed port-groups. Each container gets a vNIC
* **Layered filesystem** - provided entirely by vSphere. VMDK snapshots in the initial release
* **OS Kernel** - provided as a minimal ISO from which the containerVM is either booted or forked
In this mode, there is necessarily a 1:1 coupling between a container and a VM. A container image is attached to the VM as a disk, the VM is either booted or forked from the kernel ISO, then the containerVM chroots into the container filesystem effectively becoming the container.
#### Differences
This model leads to some very distinct differences between a VIC container and a traditional container, none of which impact the portability of the container abstraction between these systems, but which are important to understand.
##### Container
1. There is no default shared filesystem between the container and its host
* Volumes are attached to the container as disks and are completely isolated from each other
* A shared filesystem could be provided by something like an NFS volume driver
2. The way that you do low-level management and monitoring of a container is different. There is no VCH shell.
* Any API-level control plane query, such as `docker ps`, works as expected
* Low-level management and monitoring uses exactly the same tools and processes as for a VM
3. The kernel running in the container is not shared with any other container
* This means that there is no such thing as an optional _privileged_ mode. Every container is privileged and fully isolated.
* When a containerVM kernel is forked rather than booted, much of its immutable memory is shared with a parent _template_
4. There is no such thing as unspecified memory or CPU limits
* A Linux container will have access to all of the CPU and memory resource available in its host if not specified
* A containerVM must have memory and CPU limits defined, either derived from a default or specified explicitly
##### Virtual Container Host
A container host in VIC is a _Virtual_ Container Host (VCH). A VCH is not in itself a VM - it is an abstract dynamic resource boundary that is defined and controlled by vSphere into which containerVMs can be provisioned. As such, a VCH can be a subset of a physical host or a subset of a cluster of hosts.
However a container host also represents an API endpoint with an isolated namespace for accessing the control plane, so a functionally equivalent service must be provisioned to the vSphere infrastructure that provides the same endpoint for each VCH. There are various ways in which such an service could be deployed, but the simplest representation is to run it in a VM.
Given that a VCH in many cases will represent a subset of resource from a cluster of physical hosts, it is actually closer in concept to something like Docker Swarm than a traditional container host.
There are also necessarily implementation differences, transparent to the user, which are required to support this abstraction. For example, given that a container is entirely isolated from other containers and its host is just an esoteric resource boundary, any control operations performed within the container - launching processes, streaming stout/stderr, setting environment variables, network specialization - must be done either by modifying the container image disk before it is attached; or through a special control channel embedded in the container (see [Tether](vic-port-layer-overview.md#the-tether-process)).

43
vendor/github.com/vmware/vic/doc/design/arch/vic-port-layer-overview.md generated vendored Normal file
View File

@@ -0,0 +1,43 @@
#### Port Layer Abstractions
The Port Layer abstractions in VIC are designed to augment the vSphere APIs with low-level container primitives from which a simple container engine could be implemented. The design criteria of the Port Layer is as follows:
* The Port Layer should be primarily oriented around the notion of _isolation domains_. It should provide the means to easily express rich and flexible criteria for isolating containers and their resources, without being explicit about the mechanism through which this should be achieved.
* The Port Layer is designed to be invoked by higher-level software abstraction. It is not designed to be exposed directly to users.
* The Port Layer should be developed as Open Source Software to allow for 3rd party integration
* The Port Layer should be container engine and operating system agnostic
* The Port Layer should be designed in such a way as to optimize control plane performance
* The Port Layer should ensure a single source of truth for all state. Eg. VM power-off == container stop
From an architectural perspective, the Port Layer should be considered functionally equivalent to a project like https://github.com/docker/libcontainer in as much as it provides low-level platform-specific primitives. It is easy to see how such an abstraction could be container engine agnostic since it provides capabilities at a much lower layer. Our goal however is that it should also be operating system agnostic, which is a more challenging goal at such a low layer.
##### Operating System Independence
VMs are already completely operating system agnostic, since they virtualize at the hardware layer and all control plane operations through the vSphere APIs are therefore also necessarily OS agnostic. Guest differences are encapsulated in different builds of "VMware Tools" which is an optional in-guest agent that mediates between the guest and the hypervisor.
The Port Layer in VIC will function in exactly the same way. Control plane operations will be expressed through an OS agnostic API and distinct differences between operating system implementations will be encapuslated in the _Tether_ process that runs in each containerVM.
##### The Tether Process
A traditional container runtime, such as Linux/LXC, allows the control plane and the containers to share a kernel within a common address space. Each container gets its own private namespace, but the shared kernel allows the control plane to have visibility into the containers and also allows for processes to be started and stopped inside them.
A containerVM by contrast uses completely separate isolated kernels for the control plane and containers. The control plane can either run in the hypervisor kernel or in a distinct guest OS kernel in a separate VM, possibly even on a separate physical host. This isolation is by design: the job of a containerVM is to run only the container process in its own kernel with as minimal a guest OS stack as feasibly possible while ensuring the same strong degree of isolation as any other VM. Even the hypervisor doesn't have visibility inside the guest without an in-guest agent installed.
As such, in order for the container control plane to provide a shell into a container, to start and stop processes or to provide monitoring statistics, there must be some kind of guest agent in the containerVM. We call this guest agent a _Tether_ process. This is not the same agent as VMware Tools, but a minimal agent designed specifically for VIC.
The Tether API and Tether codebase is where all OS differences will be encapsulated. As such, the Tether API should be considered private to the Port Layer - it exists exclusively for the benefit of the internal control plane operations, not to be invoked directly by anything that implements the Port Layer.
##### Interoperability
So what kind of container primitives should the Port Layer provide and how are those intended to interoperate with established container standards?
It stands to reason that Networking, Storage and Execution are obvious areas for low-layer primitives. These primitives already exist in the vSphere APIs and the VIC Port Layer is designed to provide a framework which builds on those APIs by providing both plumbing code and opinionated mappings between container concepts and vSphere concepts.
For example, what is a container storage Volume and how should one be configured? The Port Layer API should be responsible for deciding what vSphere construct most appropriately represents a Volume and also that it is configured appropriately. It should do this based on the parameters passed in, the vSphere features currently installed in the system and the resources that the tenant has the authorization to access. It can pass back a handle to that Volume that can then be used in the creation of a container. By doing this, the Port Layer made a choice about the most appropriate underlying representation and it also made sure it was appropriately configured and indexed.
To some extent there is an inevitable overlap with the goals of other projects in this sphere, such as https://github.com/opencontainers/runc. While this is hardly surprising given that the Port Layer is attempting to make opinionated choices in exactly the same problem domain, it would be wrong to infer that this makes it an intentional fragmentation or competing API. It is our explicit intention that the two should be entirely complimentary and that the Port Layer should be the lowest level of abstraction that a VIC implementation of runc would end up calling. If the abstractions are correct, it should be just as possible to build an implementation of https://github.com/coreos/rkt/blob/master/Documentation/app-container.md using the same APIs.

94
vendor/github.com/vmware/vic/doc/design/communications.md generated vendored Normal file
View File

@@ -0,0 +1,94 @@
# Communications
This document discusses possible communication channels between components, but not what flows over them. This is essentially Layer2/3 with Layer7 discussed in the documents relating to specific components.
## ContainerVM to Appliance
There are two types of communication that occur between a containerVM and the [VCH appliance](components.md#appliance):
1. application level network traffic
2. VIC control channel, which includes user interaction with the container console
This section addresses the VIC control channel
### Async - guestinfo
The containerVM guestinfo is used as a persistent, asynch publishing mechanism for specific data related to container state, primarily:
1. assigned DHCP address
2. error reporting for [containerVM components](components.md#container)
3. exit status of the container process
Appliance to containerVM communication also uses guestinfo as a channel and is addressed in a little more depth under [configuration](configuration.md#appliance)
### Synchronous
The following are different approaches to handling the interactive communication path between containerVM and appliance, each with pros and cons.
#### Network serial port
The network serial port is a mechanism provided by vSphere that allows serial traffic to be forwarded over a TCP connection; it can operate in either client or server mode.
Pros:
* native to vSphere
* serial support can be assumed in most OSes
Cons:
* requires Enterprise license
* requires route from host to appliance VM
* implies firewall config
* generally means the appliance needs to be on the management network
* serial does not appear to be efficient for large volumes of data
* inhibits vMotion if not targeted at a Virtual Serial Port Concentrator (vSPC)
* serial port access and access semantics differ significantly between OSes
#### vSPC relay
A vSPC is explicitly to enable vMotion for VMs that have network serial connections. When connecting to a vSPC the connection is an extended TELNET protocol rather than raw TCP. In the case of a regular vSPC appliance, the vSPC has to bridge between the management network (network serial originates from the ESX host, not the VM) and a network the VIC appliance is attached to.
The vSPC relay is an agent on ESX that does dumb forwarding of the incoming Telnet connection via vSocket into the VCH appliance, which provides a very basic vSPC implementation. In this case the containerVM would be configured with a network serial port with a vSPC target that is the ESX host running the applianceVM.
Pros:
* appliance does not require connectivity with the management network (when combined with [vmomi agent](#components.md#vmomi-authenticating-agent) )
* containerVM vMotion
Cons:
* requires Enterprise license
* require knowledge of applianceVM host and reconfiguration if applianceVM vMotion occurs
### Pipe backed serial port with relay agent
The pipe backed serial port is a mechanism provided by vSphere that maps serial port IO to a named pipe on the ESX host. This allows an agent on the host to act as a relay for that data.
Pros:
* does not require Enterprise license
* serial support can be assumed in most OSes
* does not inhibit vMotion
Cons:
* requires agent on all hosts in the cluster that will run containerVMs
* requires route from host to appliance VM
* serial port access and access semantics differ significantly between OSes
#### vSocket relay
[vSocket relay](components.md#vsocket-relay-agent) makes use of the paravirtual vSocket mechanism for VM to Host communication. This model requires a agent running on all ESX hosts in the cluster that can receive incoming vSocket communications and forward them to the appliance.
Pros:
* does not require Enterprise license
* common access and usage semantics across OSes
* does not inhibit vMotion
* improved efficiency over serial (uses PIO ports)
Cons:
* requires driver in the OS (easily supplied in the bootstrap image if driver exists)
* requires agent on all hosts in the cluster that will run containerVMs
* requires route from host to appliance VM
#### vSocket peer-to-peer relay
An extension of this approach has the vSocket relay agent forwarding to another vSocket relay agent on the host running the applianceVM, with that connection then relayed via vSocket into the appliance. The pros and cons below are in addition to those without the peer-to-peer relay.
Pros:
* appliance does not require connectivity with the management network (when combined with [vmomi agent](#components.md#vmomi-authenticating-agent) )
Cons:
* require knowledge of applianceVM host and reconfiguration if applianceVM vMotion occurs

252
vendor/github.com/vmware/vic/doc/design/components.md generated vendored Normal file
View File

@@ -0,0 +1,252 @@
# vSphere Integrated Containers - Architecture and Design
## Approach
The development approach taken for VIC is iterative refinement, both for the high level product and recursively throughout the lower level components that go into it. This, as with most Agile processes, is driven by a need to have a useful product in short timeframes, and to manage risk - whether risk related to larger delivery deadlines (VIC may be OSS but it's still a VMware product), changing technical landscapes, or the unknown. The day-by-day development process we follow is [documented in the contributor docs](../../CONTRIBUTING.md).
The upshot of this approach is that many of these components are outline sketches at best as we start out, with specific areas being filled out over time. Avoiding any comment on business priorities, the _technical_ priorities for the project are:
1. build an appliance foundation to permit secure orchestration of Virtual Infrastructure by regular users
2. build primitives that support container style semantics, implemented with VMs
2. present a Docker API endpoint though which users can orchestrate vSphere resources, within the scope of the Virtual Container Host
The primary _goal_ of the project is to be useful to users - very simple to say, but much more involved to achieve. Our expectation is that most users will consume VIC via the Docker API, therefore if you have specific requirements around docker function support please raise them as issues if they are not already documented in #445 - if they are, add a +1. We will be focusing on function required on the production end of the CI pipeline initially, but knowing what _you_ require for our initial releases to be useful in that scope is invaluable.
## Components
- [Component Architecture](#component-architecture)
- [Container](#container)
- [Container Base](#container-base)
- [Tether](#tether)
- [Container Logging](#container-logging)
- [Appliance](#appliance)
- [Appliance Base](#appliance-base)
- [vicadmin](#vicadmin)
- [Docker API server](#docker-api-server)
- [imagec](#imagec)
- [Port Layer](#port-layer)
- [Port Layer - Execution](#port-layer---execution)
- [Port Layer - Interaction](#port-layer---interaction)
- [Port Layer - Networking](#port-layer---networking)
- [Port Layer - Storage](#port-layer---storage)
- [Install and management](#install-and-management)
- [vic-machine](#vic-machine)
- [Validating Proxy](#validating-proxy)
- [ESX Agents](#esx-agents)
- [VMOMI Authenticating Agent](#vmomi-authenticating-agent)
- [vSocket Relay Agent](#vsocket-relay-agent)
- [NSX Authenticating Agent](#nsx-authenticating-agent)
## Component Architecture
This is a component architecture for the system, encompassing some of the main control flows. It is intended to be used as a visual map for which components interact and how those interaction occur; it does not indicate what those interaction are. Components are marked as _new code_ are entirely new for vSphere Integrated Containers, where as others are either part of vSphere or we have a functioning reference implementation from Project Bonneville.
![system component architecture](images/component_architecture.png)
## Container
### Container Base
ContainerVMs are bootstrapped from a PhotonOS based liveCD, containing just enough to boot Linux and set up the container filesystem, before performing a switch_root into the container root filesystem. The end result is a VM:
* running the PhotonOS kernel, with appropriate kernel modules for demand loading
* the specified container filesystem mounted as `/`
* [a custom init binary](#tether) that provides the command & control channel for container interaction
The container process runs as root with full privileges, however there's no way to make persistent changes to anything except the container filesystem - the core operating files are read-only on the ISO and refresh each time the container is started.
[Issues relating to this component](https://github.com/vmware/vic/labels/component%2Fcontainer-base)
### Tether
The tether is an init replacement used in containerVMs that provides the command & control channel necessary to perform any operation inside the container. This includes launching of the container process, setting of environment variables, configuration of networking, etc. The tether is currently based on a modified SSH server tailored specifically for this purpose.
[Additional detail](tether.md)
[Issues relating to tether](https://github.com/vmware/vic/labels/component%2Ftether)
### Container Logging
Container logging, as distinct from other logging mechanisms, is the capture mechanism for container output. This fills the same niche as [Docker log drivers](https://docs.docker.com/engine/admin/logging/overview/) and has the potential to be a direct consumer of docker log drivers.
As of the v0.1 release logs are being persisted on the datastore along with the VM.
[Issues relating to container logging](https://github.com/vmware/vic/labels/component%2Fcontainer-logging)
## Appliance
### Appliance Base
The appliance VM is bootstrapped from a PhotonOS based liveCD that embeds elements of VIC relevant to Virtual Container Host functions. The appliance is diskless _in essence_, but may well use a non-persistent disk to cache transient data such as docker images in-flight from [Docker Hub](https://hub.docker.com/) but not yet persisted to a datastore.
The ISO used to bootstrap the appliance is generated from the following make targets - only the first must be specified as the others are pre-requisites:
```
make appliance appliance-staging iso-base
```
The intent behind having the appliance be diskless and bootstrap off an ISO each time is:
* improved security - coupled with [the configuration mechanism](configuration.md#Configuration-persistence-mechanism) there's no way to make persistent changes without vSphere API access
* robustness - a reboot will return the appliance to a known good state, assuming an administrator has not altered the configuration in the meantime
* simplicity of update - using [vic-machine](#vic-machine) to update a VCH should be as simple as pointing the appliance VM at a new ISO version and rebooting, so long as there's no metadata migration needed; in that case the migration should be viable as a background task prior to VCH update.
[Issues relating to appliance base](https://github.com/vmware/vic/labels/component%2Fappliance-base)
### vicadmin
This is more speculative than any of the other components at this point. We fully expect there to be a need for user level inspection/administration of a deployed Virtual Container Host, however we've not yet identified the functions this should provide.
Current list of functions:
* log collection
Speculative list of functions (via docker-machine as a client?):
* docker API user management
* reconfigure operations (e.g. add --insecure-registry)
[Issues relating to vicadmin](https://github.com/vmware/vic/labels/component%2Fvicadmin)
### Docker API server
This is the portion of a Virtual Container Host that provides a Docker API endpoint for users to interact with; it is also referred to as a 'personality'. The longer term design has multiple personalities running within a single VCH, such that the same endpoint can serve mixed API versions, and the same VCH can serve multiple API flavours.
As of the v0.1 release this makes use of the [docker engine-api](https://github.com/docker/engine-api) project to ensure API compatibility with docker.
The following is commentary about why the docker API server is not actual Docker. It's not usual design doc content, but is a question that comes up enough that it becomes core to understanding the design.
[Issues relating to docker API](https://github.com/vmware/vic/labels/component%2Fdocker-api-server)
#### Commentary
There are frequently questions asked about why we're not using Docker directly and supplying exec, graph and libnetwork drivers, which is the approach we took with [Project Bonneville](http://blogs.vmware.com/cloudnative/introducing-project-bonneville/). People who are intimately familiar with the container space may also ask why we're not using the OCI components such as runc.
Being very open here, the answer comes back to the reasoning behind the [approach](#approach) - it was a choice made when starting VIC in January 2016 to manage technical and project risks, and it's a trade against business risk (community perception, customer concerns about compatibility, and others).
Some of the major technical assumptions in the Docker code base do not hold true for VIC (as of decision nexus date):
1. container filesystems exist on the same host the daemon is running on
2. namespaces can be created independently of containers (e.g. libnetwork sets up interfaces prior to graphdriver.Create)
3. operations on namespaces are _relatively_ cheap in terms of performance
4. container lifecycles are tied to that of the docker daemon
5. no orthogonal second management plane
As Docker continues with decomposition of the monolithic docker daemon into components, our ability to consume those components directly increases, and there are very good reasons to do so - not least is the fact it reduces the development and maintenance burden on the VIC team. In line with the stated approach, as docker components become available that can be consumed, we will.
A good example of the planned approach regarding docker is our use of the engine-api project -- VIC was initially using a Docker API server generated from a swagger specification. Once the engine-api project became somewhat stable we contributed the swagger spec to that project and are now consumers of the engine-api directly.
This leads to another question regarding why we don't simply invest our project resources in helping Docker decompose the daemon. Again the answer comes back to risk management - Docker's priorities are not VMware's and, without a formal partnership, assuming the decomposition stays on a schedule that's viable for VIC deadlines is, well, risky. This is a fine line to walk as we _want_ to be good community members and, from a more selfish perspective, don't want to invest in quickly deprecated code. Our intent is to, as with the engine-api project, move to the docker and OCI components as they become semi-stable and invest resource in them at that point - how early in the life of a given component we can do that is a function of deadlines, resources and politics.
### imagec
The component follows the naming pattern introduced by OCI with 'runc' and is a docker registry client library, purely concerned with the pull/push aspects of the registry with login becoming a transitive dependency.
[Current open issues relating to imagec](https://github.com/vmware/vic/labels/component%2Fimagec)
### Port Layer
The port layer presents container primitive operations, with APIs that express intent rather than the intricacies of how.
#### Port Layer - Execution
This component handles management of containers such as create, start, stop, kill, etc, and is broken distinct from interaction primarily because the uptime requirements may be different, [see Interaction](#portlayer---interaction).
[Issues relating to portlayer execution](https://github.com/vmware/vic/labels/component%2Fportlayer%2Fexecution)
#### Port Layer - Interaction
This component deals handles interaction with a running container and is broken distinct from execution primarily because the uptime requirements may be different.
If the execution portion of the port layer is unavailable then only container management operations around creating, starting, stopping, et al are impacted, and then only for as long as the VCH is unavailable.
If the interaction portions are unavailable it impacts ongoing use of interactive sessions and potentially loses chunks of the container output (unless serialized to vSphere infrastructure as an intermediate step - [container logging](#container-logging), [container base](#container-base), and [tether](#tether)) are the other components that factor into the log persistence discussion).
[Issues relating to portlayer interaction](https://github.com/vmware/vic/labels/component%2Fportlayer%2Finteraction)
#### Port Layer - Networking
A Virtual Container Host presents a cluster of resources as a single endpoint, so while we must also supply transparent cross host networking, it must be there without requiring any user configuration.
In conjunction with [the provisioning workflows](#vic-machine) it should also allow mapping of specific vSphere/NSX networks into the docker network namespace, and mapping of existing network entities (e.g. database servers) into the docker container namespace with defined aliases.
Initial design and implementation details for MVP are [here](networking/README.md).
[Issues relating to portlayer networking](https://github.com/vmware/vic/labels/component%2Fportlayer%2Fnetwork)
#### Port Layer - Storage
This provides the storage manipulation portions of the port layer, including container image storage, layering along with volume creation and manipulation. [imagec](#imagec) uses this component to translate registry images into a layered format the can be used directly by vSphere, namely VMDK disk chains.
[Issues relating to portlayer storage](https://github.com/vmware/vic/labels/component%2Fportlayer%2Fstorage)
#### VMOMI gateway
This is the GuestOS side of the [VMOMI authenticating proxy](#vmomi-authenticating-proxy) that provides a 'paravirtual' vmomi interface to the underlying vSphere infrastructure. This is structured as a library for direct consumption by other components, for example [vic-machine](#vic-machine), and may be wrapped in a service presenting a URI endpoint for shared consumption between components such as portlayer and vicadmin. This latter approach is less secure by virtue of exposing that mechanism generally within the Appliance, but has the benefit of simplicity regarding managing multiple connections over a single channel.
[Issues relating to portlayer storage](https://github.com/vmware/vic/labels/component%2Fvmomi%2Fgateway)
## Install and management
### vic-machine
The _docker-machine_ mechanism is something of a de-facto standard for provisioning docker hosts.
There is a significant amount of vSphere specific behaviour that needs to be exposed that may well go beyond what docker-machine provides - hence supplying a vic-machine binary.
Ideally we'll provide vic-machine in a form that makes it viable as a docker-machine plugin, allowing some reuse of existing knowledge, with VIC specific options and behaviours presented to the user via plugin options.
It is possible that the value provided by keeping the _docker-machine_ naming is overshadowed by the flexibility that changing the name provides (perhaps _vic-machine_) - lacking concrete knowledge one way or another, this component is currently named vic-machine so as to avoid confusion with the docker binary.
While deployment of a Virtual Container Host is relatively simple if performed by someone with vSphere administrative credentials, conversations with customers have shown that the self-provisioning facet of docker is a significant portion of it's value. This component, in conjunction with [the validating proxy](#validating-proxy), provides self-provisioning capabilities and the requisite delegation of authority and specification of restrictions.
[Issues relating to vic-machine](https://github.com/vmware/vic/labels/component%2Fvic-machine)
### Validating Proxy
The self-provisioning workflow for vSphere Integrated Containers is an authority delegation and resource restriction model. That requires that there be an active endpoint accessible to both the user and the viadmin that is capable of generating delegation tokens to be passed to the user, and validating those that are received from a user. The validating proxy fills this niche; the described proxy is very, very simple and does not include elements such as directory services integration.
[Issues relating to validating proxy](https://github.com/vmware/vic/labels/component%2Fvalidating-proxy)
## ESX Agents
### VMOMI Authenticating Agent
As with [access to NSX management](#nsx-authenticating-agent), vSphere orchestration requires credentials and therefore credential management. The authenticating agent is intended to move the credential management problem out of the VCH and into the vSphere infrastructure. Authorizing the VCH to perform specific operations by virtue of being a specific VM (and potentially cryptographically verified), rather than generating and embedding credentials into the guest, means:
* a VCH can be safely considered untrusted
* IDS style inspection and validation can be performed on each infrastructure operation performed by a VCH
* no access to management networks is required for infrastructure orchestration
[Issues relating to vmomi agent](https://github.com/vmware/vic/labels/component%2Fvmomi-authenticating-agent)
# vSocket Relay Agent
Network serial ports as a communication channel have several drawbacks:
* serial is not intended for high bandwidth, high frequency data
* inhibits forking & vMotion without vSPC, and a vSPC requires an appliance in FT/HA configuration
* requires a VCH have a presence on the management networks
* requires opening a port on the ESX firewall
The alternative we're looking at is vSocket (uses PIO based VMCI communication), however that it Host<->VM only so we need a mechanism to relay that communication to the VCH. Initially it's expected that the Host->VCH communication still be a TCP connection for a staged delivery approach, with the longer term being an agent<->agent relay between the two hosts.
[Issues relating to vsocket relay agent](https://github.com/vmware/vic/labels/component%2Fvsocket-relay-agent)
### NSX Authenticating Agent
As with [access to vSphere](#vmomi-authenticating-agent), NSX management interaction requires credentials and therefore credential management. The authenticating agent is indended to move the credential management problem out of the VCH and into the vSphere infrastructure. Authorizing the VCH to perform specific operations by virtue of being a specific VM (and potentially cryptographically verified), rather than generating and embedding credentials into the guest, means:
* a VCH can be safely considered untrusted
* IDS style inspection and validation can be performed on each infrastructure operation performed by a VCH
* no access to management networks is required for infrastructure orchestration
[Issues relating to nsx agent](https://github.com/vmware/vic/labels/component%2Fnsx-authenticating-agent)

83
vendor/github.com/vmware/vic/doc/design/configuration.md generated vendored Normal file
View File

@@ -0,0 +1,83 @@
# Configuration
Three primary elements make up a functioning VCH - these are referred to as components in the rest of this document, along with their subcomponents:
1. [vic-machine](vic-machine.md) - the mechanism by which the VCH is deployed, configured, and inspected
2. appliance - a VM that runs the VCH logic ([docker personality](components.md#docker-api-server), [log server](components.md#vicadmin), [port-layer](arch/vic-port-layer-overview.md), et al)
3. containerVMs - VMs that _are_ containers ([tether](tether.md), and container process)
This document discusses how configuration between components is done, and why it's done in that manner.
## Component interactions
For both the VCH and containerVMs the configuration is used for two way communication, with a component updating its own configuration as an asynch publishing mechanism for low-frequency, persistent information. The fields listed below are examples rather than comprehensive lists.
### vic-machine configuration of appliance
Configures VCH on initial deploy:
* _vSphere target and credentials_ (hidden)
* networks (external, client, management, bridge, mapped vsphere networks)
* vsphere compute path (e.g. resource pool)
* image store URI
* container store URI
* permissible URIs for volumes
* white/black list of registries
* _etc_
vic-machine also:
* updates VCH config as part of management operations
* inspects (read-only) containerVM configuration for diagnostics
### appliance
Configuration of containers and environment:
* Configures containerVMs with (example config elements below)
* _appliance_ (hidden)
* ID
* command (executable, args, environment, working directory)
* networks (MAC, CIDR, gateway, name, nameservers)
* configures logical networks (whether IPAM segregation or SDN logical networks)
* configures VMDKs (volumes, images, container read/write layers)
Publishes the following:
* DHCP supplied network configuration
* system & subcomponent status
### ContainerVM
Publishes the following:
* DHCP supplied network configuration
* container start status
* container process exit code
## Configuration persistence mechanism
VIC uses the vSphere `extraConfig` and `guestinfo` mechanisms for storing configuration, the former used for config associated with either appliance or contianerVM that should **not** be visible to the GuestOS, and the latter for config that should. In reality the mechanisms are the same, with guestinfo being a subset of extraConfig.
Access to guestinfo data within the GuestOS is done via the vmware-rpctool, or library providing the same capability e.g. [vmware/vmw-guestinfo](https://github.com/vmware/vmw-guestinfo/). The various subcomponents in the appliance and the tether in containerVMs will access this configuration directly via library calls - there is no intermediate step that presents this data via arguments or environment variables.
The reasons for taking this approach:
1. it keeps the appliance VM completely stateless - a reboot will return it to a known good state
2. no cross component dependency - containerVMs do not require an VCH for ongoing operations, including vSphere High Availability restart
3. the configuration is protected from tampering, even if superuser priviledges are obtained
4. configuration is visible both from within the VM and remotely via vSphere APIs
ExtraConfig data is used for data that should not be visible to the GuestOS - the best example of this are the vSphere target and credentials (ideally an SSO token), which are hidden from the appliance and only visible to the [vmomi agent](components.md#vmomi-authenticating-agent) that supplies the authenticated vSphere API connection. This means that even if superuser priviledges are obtained on the appliance, the risk is limited to a controlled connection rather than leaking any form of access token or even the URI of the vSphere endpoint.
## Accessing configuration - implementation
The defacto model for holding configuration in Go is a struct with Fields that contain the specifics. The idiomatic way of populating a struct from a serialized form is with a Decoder from the [Go encoding](https://golang.org/pkg/encoding/) package.
It is expected that use of the configuration be performed via two endcoder/decoder pairs - one that operates directly on guestinfo via the vmx-guestinfo library, the other that takes a [VirtualMachineConfigSpec](pubs.vmware.com/vsphere-60/topic/com.vmware.wssdk.apiref.doc/vim.vm.ConfigSpec.html) and operates on the ExtraConfig field of the spec.
The implementation will understand the following set of annotations and map the fields to appropriate extraConfig keys - in the case where the key describes a boolean state, omitting the annotation implies the opposite:
* `hidden` - hidden from GuestOS
* `read-only` - value can only be modified via vSphere APIs
* `non-persistent` - value will be lost on VM reboot
* `volatile` - field is not exported directly, but via a function that freshens the value each time)
* `atomic:<group>` - speculative to allow atomic updates to multiple fields

26
vendor/github.com/vmware/vic/doc/design/container_logging.md generated vendored Normal file
View File

@@ -0,0 +1,26 @@
# Container VM output logging in VIC
When a container VM process produces output, that output is sent down through a serial connection into the vSphere backend where it is written to a logfile. When a user wants to see this output, they use the `docker logs` command, which reads from this logfile and displays the log entries on the command line in order from oldest to newest.
However, in order for users to be able to use the `--since` option with `docker logs`, we must also couple timestamps to these log entries so that the user can selectively filter log messages to be displayed based on when those entries occurred.
### Requirements
The container VM logging mechanism must:
1. Create a header for each log entry containing both the size of that entry, the time at which that entry occurred, and the stream from which that entry originated (stdout, stderr).
2. Allow these entries to be read and at a later time, starting with the first entry occurring at or beyond the timestamp supplied by the user with the `--since` option to `docker logs`, or starting with the first entry in the logfile if `--since` was not used.
### Implementation
A package for containerVM logging in VIC is added as `github.com/vmware/vic/lib/iolog`. In this package are two files, `log_writer.go` and `log_reader.go`. These files contain an implementation of Go's `io.Writer` and `io.ReadCloser` interfaces, respectively.
The responsibilities of the `LogWriter` are:
1. To create a header for the entry, containing the timestamp at which the entry occurred, the size of the entry, and the stream that produced the entry.
2. To write these bytes to the serial port associated with the containerVM logfile on the backend.
3. To flush the remaining bytes in the supplied buffer upon `Close()`
The responsibilities of the `LogReader` are:
1. To read in a header and decode it into the timestamp, size and stream of the entry.
2. To then read the following `size` bytes that contain the actual message.
3. To copy the message bytes into the underlying `Read` stream's `[]byte` slice.
4. To preserve unwritten bytes in a call to `Read` in memory so that they may be written during the next call, in the case where the supplied `[]byte` slice was smaller than the log message we are trying to write.

BIN
vendor/github.com/vmware/vic/doc/design/containerd/images/vic-layers.png generated vendored Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 147 KiB

276
vendor/github.com/vmware/vic/doc/design/containerd/port-layer-integration.md generated vendored Normal file
View File

@@ -0,0 +1,276 @@
**Containerd Integration Draft**
**Summary**
Docker’s Containerd project aims to present an API surface representing all of the core functions of a container runtime on a single host. This includes image resolution, image pull, container and image lifecycle management, events, metrics, execution of processes and interactive sessions.
Network and volume management are out of scope for containerd which will expect to reference already-existing networks and storage.
http://containerd.io/
https://github.com/docker/containerd
**Relevance to VIC**
The scope of the Docker personality in VICe (VIC engine) is currently as an enterprise container _runtime_, very much the same scope as containerd. The main difference between the two is that in VIC, clustering, HA and other enterprise features are integrated transparently below the user API boundary. With regular containerd, it is assumed that these features will exist above the API in the application layer and that containerd manages containers on a single literal host (guest OS on physical harware or VM). The VIC integration will see containerd managing container workloads in a "virtual host" (vSphere resource pool), but the goal is for this to be completely transparent to the containerd consumer.
The goals of integrating VICe with containerd are the following:
* Be API complete with an industry-standard API that better encapsulates the scope of VIC.
* Reach some degree of compatibility with the OCI image format. This is a requirement of containerd.
* Reuse more common Docker code. It is anticipated that containerd will be part of a larger refactoring effort that will enable us to integrate better with Docker code.
* Extend compatibility beyond Docker, assuming containerd succeeds as a runtime compatibility layer.
* Work with the community to ensure that containerd has flexible enough integration points to support VIC
**Scope**
The scope of this document covers integration between containerd and the container primitives layer in VIC, called the Port Layer. It does not cover how the Docker engine integrates with containerd.
**Design Considerations**
*OCI compatibility?*
The question of image format and subsequent implications on low-level runtime can be separated from containerd to some extent. The design explicitly allows for differences in container image format, providing that a /rootfs is the output of image post-processing. It also potentially allows for differences in container runtime via the use of the executor interface that both runc and containerd-shim implement. That said, containerd is designed to be OCI-compatible and that is the way the industry is likely to converge.
As such, OCI compatibility is a goal for the VIC containerd integration.
*Runc / Runv*
Runc as it stands expects many things to exist before it creates a container. It expects an unpacked /rootfs with a config.json containing runtime configuration for that specific container. It expects networking and storage to already exist. When a container is created, the config.json is parsed and a dummy process is created in /proc and the created container gets a PID. Hooks can also be run at this point. When the container is started, the process in the created container is executed and runc will block until the process has completed. While the process is running, runc is its parent.
Writing an equivalent runc for vSphere presents some architectural challenges. For a start, the control plane which creates the rootfs for a container cannot see it once it’s created. It can only be seen by the container guest. That immediately presents a problem for config.json, because a runc equivalent won’t be able to see it to configure the container. There is no /proc filesystem - the closest equivalent is a moRef for a VM. There is no "parent" process model - raw calls to vSphere are async and the Port Layer calls are also async. Runc expects to write local state to /var/run, for which VIC has no equivalent.
The biggest difference though is that runc and containerd-shim is a per-container model. VICe's closest equivalent to Runc is infact the in-guest Tether init process - there is one per container and it's the glue between the VM lifecycle and the lifecycle of the process(es) running in the VM. However, Tether processes are only visible by proxy via the Port Layer which itself is a daemon model, so integrating Tether with a runc shim is a non-starter.
As such, closely aping runc’s behavior is not a useful goal for VICe. It is a plumbing detail and the whole purpose of containerd is to abstract the plumbing layer and make it easier to consume. The most obvious integration point for VICe is the Executor interface, which is also implemented by runc, so even if aping runc’s behavior is not the goal, we should end up with something that operates in a very similar manner.
**Integration Points**
![VICe layers](images/vic-layers.png)
![Containerd architecture](https://github.com/docker/containerd/blob/0a5544d8c4dab44dfc682f5ad07f1cd011c0a115/design/architecture.png)
Since the goal is to integrate with containerd, the most important consideration is what the integration points should be. At this point the following have been identified:
1. Snapshots
2. Executor
3. Content store
4. Metadata store
5. Events
Containerd has components which are designed to take an image from being pulled down through to a container being invoked (see above) which map pretty consistently to our integration points. The data flow envisaged looks something like this:
![Data flow](https://github.com/docker/containerd/blob/0a5544d8c4dab44dfc682f5ad07f1cd011c0a115/design/data-flow.png)
In order to understand the process an image goes through, it’s also helpful to understand the OCI image format: [https://github.com/opencontainers/image-spec](https://github.com/opencontainers/image-spec)
A remote registry, the content blobstore and the snapshot store in containerd can be thought of as a three layered cache. Some amount of processing is done when data moves between the caches. When an image is pulled from a remote registry, it is broken up into metadata and blobs and cached in the metadata and content stores respectively. It is then post-processed onto snapshots (we should assume are also cached) which when combined using a layered filesystem present as a filesystem bundle to the Runtime Controller.
As such, it should be assumed that the caches will be searched in reverse order when a container create call is made on containerd: Do the snapshots already exist? If not, is the data already in the content store to create them? If not, does the image URI resolve to a remote registry they can be pulled from?
If this is how it expects to work, then the content blobstore is potentially optional. If every snapshot was created from a blob and the snapshots are not garbage collected, then a cache hit will return a snapshot and a cache miss will always result in a pull. The unpacking of the blob into a snapshot then becomes a one-off event in tempfs.
*Snapshots*
So the snapshot integration should look very like it does today. Snapshots map to VMDKs and the snapshot manager will use the storage part of the Port Layer to manage snapshot lifecycle as well as listing available images. In the case where containerd shuts down and starts up again, it will need to query the Port Layer to rebuild its own local cache (if any)*
*(Note that the VIC appliance local filesystem is intentionally ephemeral, so no data persisted on the local filesystem is permanent. We must assume that operations like upgrade or other control plane outages should have no impact on the running containers or image cache and as such, there must be a means for containerd to learn about existing state when it starts up).
*Executor*
The executor interface is responsible both for all aspects of container lifecycle, including interaction with the container - (stdout, stderr, stdin and tty).
As it stands, the interface expect a single out-of-process instance of an executor per container and that the executor will keep interaction channels open if containerd is restarted. This is something of a mismatch with our existing Port Layer which is a single service offering async API calls for running and interacting with containers.
This presents two options: Fake up shim processes that represent each container or add an ID field to the Executor interface. The latter approach is our preference for the main reason that shim lifecycle management becomes a pain in the event that the VICe endpoint VM is restarted.
*Content store*
In the containerd design, the content store is a content-addressable blobstore. This seems to be something that intentionally exists outside of the scope of containerd itself as a way to optimize the local caching of image data. It doesn't seem necessary or desirable to implement this in the VIC appliance as sizing ramfs to cope with a blob store would be challenging. That said, if the three-level cache discussed above is the nature of the design, a content store implementation may well be optional.
*Metadata store*
Metadata is what ties everything together. It will need to link image URIs to digests and links to blobs and snapshots. Given that VIC does not have a persistent filesystem layer, metadata must be persisted into vSphere in some way.
The question of how this should be done remains open. The KV store on the Port Layer is likely not rich enough to represent the data relationships, so an alternative needs to be sought.
*Events*
At present, the events interface is one of the least clearly defined. However, it’s not hard to imagine how events from vSphere could propagate through containerd via the Port Layer.
**State Management**
One theme that has to be very carefully considered is the question of where state is persisted with containerd integration. Given that there is no persistent filesystem in the VIC appliance and vSphere is the single source of truth for runtime state, it’s important to consider the lifespan of runtime state and where it should be persisted. This does not include caches.
<table>
<tr>
<td>State</td>
<td>Standard containerd/runc</td>
<td>VIC containerd</td>
</tr>
<tr>
<td>OCI image metadata</td>
<td>Local filesystem</td>
<td>Port Layer KV store -> vSphere</td>
</tr>
<tr>
<td>OCI image blob data</td>
<td>Local filesystem</td>
<td>Local ramfs (not persisted)*</td>
</tr>
<tr>
<td>OCI flat bundle data</td>
<td>Local filesystem</td>
<td>Single VMDK (support?)</td>
</tr>
<tr>
<td>OCI config data</td>
<td>Local filesystem</td>
<td>cVM guestInfo</td>
</tr>
<tr>
<td>Image / RW snapshots</td>
<td>Btrfs / DeviceMapper</td>
<td>VMDK snapshots</td>
</tr>
<tr>
<td>Ctr create/start/stop</td>
<td>/proc PID</td>
<td>VM exists / powered on / off</td>
</tr>
<tr>
<td>Running container state</td>
<td>Local filesystem (via runc)</td>
<td>Tether via Port Layer</td>
</tr>
<tr>
<td>Container metrics</td>
<td>Host pprof (+ agent?)</td>
<td>vSphere via Port Layer</td>
</tr>
<tr>
<td>Container logs</td>
<td>Local filesystem</td>
<td>File-back serial port</td>
</tr>
<tr>
<td>Stdout/err/in streams</td>
<td>Fifo on local filesystem</td>
<td>Interaction Port Layer</td>
</tr>
<tr>
<td>Exec’d process state</td>
<td>/proc PID ?</td>
<td>Tether via Port Layer</td>
</tr>
</table>
* There’s a question of whether VIC should support the notion of creating a flat bundle without snapshots. OCI supports this idea by extracting layers on top of each other on the local filesystem. There are potential performance advantages, but these would be significantly offset by the lack of layer reuse.
**Workflows**
The most common and most simple workflows to consider are image pull, container create, container start and delete. Interesting to note that there is no "stop" on the executor interface, implying no restart. Here is the executor interface that we propose to integrate with:
```
type Executor interface {
Create(ctx context.Context, id string, o CreateOpts) (*Container, error)
Pause(context.Context, *Container) error
Resume(context.Context, *Container) error
List(context.Context) ([]*Container, error)
Load(ctx context.Context, id string) (*Container, error)
Delete(context.Context, *Container) error
Start(context.Context, *Container) error
StartProcess(context.Context, *Container, StartProcessOpts) (Process, error)
SignalProcess(ctx context.Context, c *Container, id string, sig os.Signal) error
DeleteProcess(ctx context.Context, c *Container, id string) error
}
```
*Image Pull*
Image pull will be handled by the "Distribution" subsystem, which it is to be assumed operates similarly to VIC imagec component. Given a URI, it should be able to pull down any images that don’t already exist in a local cache.
The question of whether snapshots are created as output of image pull or whether these are created as output of container create is one that gets to the heart of whether the output of image pull needs to be cached at all. At present this isn’t clear, but it would be preferable for it to be the latter.
Image metadata will need to persist beyond the lifespan of the VIC appliance and as such will need to be written into vSphere.
*Container Create*
Assuming the image snapshots are already created, an additional RW layer needs to be created, assuming the config doesn’t have readonly=true.
There is then a configuration pre-processing that needs to happen where the config.json is generated for the container. This will encapsulate a mixture of the image spec, containerd command-line arguments, constants set for the installation and possibly environmental factors.
There will then need to be a validation step of the generated config to make sure that it makes sense. For example, referenced networks and storage need to exist. If the config passes validation, create() is called on the executor interface, which would propagate to the Port Layer. A VMConfigSpec will then be generated and a VM will be created. This is the equivalent of runc create (where a dummy process is created and gets a PID).
*Container Run*
Container run should operate in a very similar manner to the way it does today in that the VM is powered on and a handshake established with the Tether.
The question is how we handle TTY/attach and streaming of stdout and stderr. In the current version of containerd, the containerd-shim maintains a Fifo on the filesystem that can be reconnected to if containerd shuts down and restarts.
<TBD>
**Questions**
There are a number of questions that still need to be worked out. Here’s are the most obvious:
*Do we need to create a process per container in the VIC appliance to fit with the executor model?*
Both runc and the containerd shim are a process-per-container model. Our Port Layer is a long-running service. We will need some kind of shim layer on the Port Layer to mate with the executor interface anyway. Does having entries in /proc for each container help in any way?
*What do we do with the config.json in a bundle?*
When a bundle is extracted to a local filesystem, config.json is used as input to runc in terms of how to run the container. Config.json in the normal case will be a composite of image config and runtime config generated by containerd. For us to integrate seamlessly, we should expect the output of containerd to still be a config.json, but this will then need to be parsed by the Port Layer in creation of the VMConfigSpec. The config.json does not then need to end up in the bundle filesystem mounted to the container since runtime configuration goes into VM guestInfo.
From discussions with the containerd team, it doesn't seem necessary to cache the config.json anywhere or make it available to the container. In fact, containerd is not responsible for modifying config.json at all.
*What parts of config.json can/should we ignore?*
VIC is not the only hypervisor OCI implementation out there and it should be noted that there are a bunch of areas in the OCI config spec that simply can’t apply to VIC. There is no distinction in the spec between the OS that supports the container infrastructure and the OS that the container runs on - one implies the other. In the case of VIC, the two are completely distinct.
Other hypervisor implementations list a bunch of areas that they either [don’t support or ignore](https://github.com/01org/cc-oci-runtime/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aopen%20label%3Adomain%3AOCI). The spec itself lists a bunch of parts that are "optional". We should put a proposal together in another document.
*Role of the Supervisor component*
The Supervisor is a multiplexer for shims, so it's very similar in scope and function to our Port Layer.
*Are snapshots created during image pull or container create?*
In the current model, snapshots are created on pull. Containerd seems to be designed to have snapshots created as part of container create. This needs to be clarified as it will have significant performance implications on create as well as caching implications for the image data.
*What to do with pause and resume on the executor interface?*
This could map to VM suspend and resume, but to date this is not something we’ve tried or even seen a need for. According to the containerd team, this is for the purpose of committing images.
*How to handle TTY/attach and streaming of stdout and stderr*
The containerd-shim uses Fifo on the filesystem to keep these streams open even if containerd shuts down. There isn’t a clear integration point to handle this, so we need to work out how to map this to our current implementation and whether an extension point needs to be created.
*How to handle container **logs*
In VICe, the portlayer handles retrieving logs, and the logs are streamed to the personality server using the attach mechanism. The personality server performs some validation but is mostly hands off.
*How to handle docker info and stats*
VICe returns vSphere specific information for docker info. VIC will need hooks to provide this info. Also, how to stream VCH and container stats up to the Docker’s prometheus telemetry hooks?

106
vendor/github.com/vmware/vic/doc/design/dinv-dev-tool.md generated vendored Normal file
View File

@@ -0,0 +1,106 @@
# Goal
The goal for this investigation, is to find a suitable tool to enable "easy-button" provisioning of Docker hosts on top of VIC, also referred to as DinV, or Docker-in-VIC.
The short term goal is to provide a tool for the user to provision a prebuilt docker image containing a full fledged docker host and make it available to the user, automating the provisioning and security part (deployment, creation of certificates, registry cache persistency) as much as possible, giving the developers a tool to self-provision docker hosts with an easy button.
# Options
We're currently evaluating two options: use Docker Machine, or build our own tool.
# Investigation: Why docker machine isn't a good choice.
Docker machine is an open source tool published by docker, helps users in getting started with creating docker hosts, it works on various platforms, including desktops (Virtualbox, HyperV, Fusion), mega clouds (aws, azure, google cloud) and private clouds (vsphere, openstack).
Docker machine has a modular plugin architecture for the infrastructure part, but lacks a modular provisioning system, which means we cannot provision anything outside the OS/binaries that are built into its core.
VMware has contributed drivers for its platforms (Fusion, vSphere and vCloud Air), the people at Docker started to modularize these drivers with a component called libmachine.
Libmachine, despite the name, is not entirely modular, its external interface only includes the infrastructure portion (which also requires ssh)[1] and doesn't have an interface for OS provisioning we can plug into, the only way to have a new OS or provisioning system supported, is to have a PR accepted into docker machine core.
There is no other way around this, even crafting an infrastructure plugin to do both infra provisioning and the certificate automation part won't work, docker machine expects to rely on libmachine for all the day2 operations it handles (upgrade, regenerate-certificates and even create), this renders Docker machine unsuitable for our needs.
Same goes for the security aspect, certificate creation is performed on the local machine and copied over SSH and there are no hooks whatsoever in this process.
# Investigation: How we should build our tool.
Developers and Ops people are accustomed to the modus operandi of docker machine, our tool has a very similar purpose, but with a specific target which is the docker api exposed by VIC (or others in the future), modeling our own tool on the same flow that docker machine has will ensure an easy adoption curve with our users.
Building our own tool means also making ourselves future proof with regards to new personalities and use cases with VIC, two examples that come to mind: Kubernetes and Docker Datacenter.
Our tool will initially focus heavily on the DinV use case, meaning we'll be a consumer of the docker API that the VCH provides, plus helpers that will automate the provisioning part of the process (some of these parts can be borrowed from docker/machine packages, like for certificate creation).
The tool will be dependent on the VCH implementing the `docker cp` operation (https://github.com/vmware/vic/issues/769).
We will provide an official "DinV" image, based on photon, which we will support, as well as the source for the container that customers can use to "roll their own" DinV image, with their own customization (including things like running `sshd` inside the container). For customization purposes, it would be ideal to support LinuxKit, but that will require VIC to support running container processes as PID 1.
# Conclusions
Building our own tool is the most sensible option right now, modeling it after docker machine will ensure an easy migration path for people accustomed to the tool, and will make it future proof for new type of deployments to come.
[1]
```go
// Driver defines how a host is created and controlled. Different types of
// driver represent different ways hosts can be created (e.g. different
// hypervisors, different cloud providers)
type Driver interface {
// Create a host using the driver's config
Create() error
// DriverName returns the name of the driver
DriverName() string
// GetCreateFlags returns the mcnflag.Flag slice representing the flags
// that can be set, their descriptions and defaults.
GetCreateFlags() []mcnflag.Flag
// GetIP returns an IP or hostname that this host is available at
// e.g. 1.2.3.4 or docker-host-d60b70a14d3a.cloudapp.net
GetIP() (string, error)
// GetMachineName returns the name of the machine
GetMachineName() string
// GetSSHHostname returns hostname for use with ssh
GetSSHHostname() (string, error)
// GetSSHKeyPath returns key path for use with ssh
GetSSHKeyPath() string
// GetSSHPort returns port for use with ssh
GetSSHPort() (int, error)
// GetSSHUsername returns username for use with ssh
GetSSHUsername() string
// GetURL returns a Docker compatible host URL for connecting to this host
// e.g. tcp://1.2.3.4:2376
GetURL() (string, error)
// GetState returns the state that the host is in (running, stopped, etc)
GetState() (state.State, error)
// Kill stops a host forcefully
Kill() error
// PreCreateCheck allows for pre-create operations to make sure a driver is ready for creation
PreCreateCheck() error
// Remove a host
Remove() error
// Restart a host. This may just call Stop(); Start() if the provider does not
// have any special restart behaviour.
Restart() error
// SetConfigFromFlags configures the driver with the object that was returned
// by RegisterCreateFlags
SetConfigFromFlags(opts DriverOptions) error
// Start a host
Start() error
// Stop a host gracefully
Stop() error
}
```

78
vendor/github.com/vmware/vic/doc/design/docker-api-filtering.md generated vendored Normal file
View File

@@ -0,0 +1,78 @@
### Docker API Filtering
The [docker api server](components.md#docker-api-server) is the endpoint for the docker client. In many of the docker
client commands a user is able to specify filter criteria. That criteria will be evaluated in
the docker persona and will limit the results of the docker command.
#### Docker Filters
The docker client provides filter options across a variety of commands. A good example of
docker filter options is the `docker images` command. The following options are available
as of Docker 1.13:
```
-f, --filter value Filter output based on conditions provided (default [])
- dangling=true
- label=<key> or label=<key>=<value>
- before=(<image-name>[:tag]|<image-id>|<image@digest>)
- since=(<image-name>[:tag]|<image-id>|<image@digest>)
- reference=(<image-name>[:tag])
```
Here's an example using the images filter: `docker images -f label=prod`. That command will
return only the images with a label key of `prod`.
#### Filter Design
The vic filtering logic is provided in the filter package of the vic docker personality. That
package should hold all of the vic implemented logic for docker personality filtering. The
filtering logic is a combination of new vic code and the reuse of docker validation code.
The package contains the common filtering and validation logic that can be used across the
vic docker personality. Additionally the package contains object specific filtering logic
isolated in object specific files. For example all vic specific container filtering will
be provided in the following file: `/lib/apiservers/engine/backends/filter/container.go`.
The vic container filtering will take advantage of the common functionality as well as
implement vic specific container filtering logic.
##### Common Filtering
There are a few filter options that span multiple docker client commands. That filtering
logic is contained in the `/lib/apiservers/engine/backends/filter/filter.go` file. The
following common filtering is provided:
- ID
- Name
- Label
- Before
- Since
If a docker command provides any of these filtering options then the vic implementation
should take advantage of this common filtering mechanism.
##### Object Specific Filtering
As mentioned above object specific filtering should be isolated in a file identifying the
object. For example all vic specific container filtering will be provided in the
following file: `/lib/apiservers/engine/backends/filter/container.go`.
Object specific filtering should take advantage of the aforementioned common filtering
as well as vendored docker validation code. Then any additional custom vic filtering
should be provided in the file.
##### Docker filter support
While all docker filtering options are valid not all filtering options are supported
by vic. In order to segregate validity from support each command that provides filtering
must provide maps indicating valid vs. supported. If all docker filtering options are
supported by vic then only one map will be needed. The container object and specifically
the `docker ps` command provides a good example of the [maps](https://github.com/vmware/vic/blob/master/lib/apiservers/engine/backends/container.go#L60).
There are two maps for the container object since not all of the filter criteria is
currently supported by vic. Those two maps are provided to the common filter validation
code to check for validity and supportability.
If a command is not currently supported by vic the validation logic will provide the
following error message:
```
Error response from daemon: filter dangling is not currently supported by vic
```

303
vendor/github.com/vmware/vic/doc/design/docker-cp.md generated vendored Normal file
View File

@@ -0,0 +1,303 @@
## DOCKER CP DESIGN DOC
### Initial Rundown of docker cp
Docker cp has many possible scenarios that we will have to account for and they will be described in this section.
We will be supporting docker cp in it's entirety. This means we will need to copy files to and from a container regardless of whether it is on or off. This will lead to 4 scenarios.
1. The container is *on*, and we want to copy data *from* it.
2. The container is *on*, and we want to copy data *to* it.
3. The container is *off*, and we want to copy data *from* it.(will constitute multiple calls from the personality)
4. The container is *off*, an we want to copy data *to* it.(will constitute multiple calls from the personality)
According to the docker remote api the target for a copy must be in `identity (no compression), gzip, bzip2, xz`. As the docker cli handles packaging a *to* request we can expect on of these formats. We must package our *from* response in one of these formats as well.
_NOTE_: docker has discussed copying between containers in the past. This will be yet another set of 4 possibilities and endpoints. Something that we can add to this plan later if it is needed in the future.
Here is a more complex look at the call operation/state flow.
|Operation| state| volumes | scenario |
|---|---|---|---|
| Export | ContainerRunning | No | Single call from personality to portlayer. Guest Tools will be used on the target rather than `Read` |
| Export | ContainerRunning | Yes | Single call from personality to portlayer. Guest Tools will be used on the target rather than `Read` |
| Export | ContainerStopped | No | Single call from personality to portlayer. `Read` is called based on the supplied filespec. |
| Export | ContainerStopped | Yes | Multiple calls from personality to portlayer. One for the r/w layer, and n more calls where n is the number of volumes. `Read` is invoked n+1 times based on each call. Powerstatus likely won't matter here because we mount non-persistent disks|
| Import | ContainerRunning | No | Single call from personality to portlayer. Guest tools will be used on the target rather than a `Write` |
| Import | ContainerRunning | Yes | Single call from personality to portlayer. Guest tools will be used on the target rather than a `Write` |
| Import | ContainerStopped | No | Single call from personality to portlayer. `Write` will be used to mount the r/w layer and then write the contents based on the supplied filespec |
| Import | ContainerStopped | Yes | Multiple calls from personality to portlayer. One for the r/w layer, and n more calls where n is the number of volumes. `Write` is invoked n+1 times based on each call from the personality. If the container is started during this time we cannot mount the volumes or the r/w layer and we will report a failure requesting the user to try again. Alternatively, we block start events until operation completion. |
### Personality to Portlayer Communication
Since all 4 of these must be supported we will require both distinctions for when the container is off and which endpoint to call to distinguish between *to* and *from* operations.
The proposed solution is to have two endpoints designed for the portlayer api swagger server. Both endpoints will exist along the same request path as two different verbs. The request path should be as such :
The portlayer functional target for `ExportArchive` and `ImportArchive` will need to be called multiple times from the personality for each device that constitutes the full filesystem of the container. A `PathSpec` struct will be used to mark inclusion and exclusion paths. We will also need to know when to strip the paths provided to the portlayer since the view of a volume would be `/` while in a running container the target path could be `/path/to/volume/` where the path before the final `/` exists on the container r/w layer(or another volume, but we are not worrying about that now.) Because there will be multiple calls for the potential Import of the archive(write calls) we will need to have a way to pass or reference the stream of information to the portlayer multiple times. This may constitute another input for the function header in order to identify the stream that we care about.
the portlayer functional targets will look as such:
__ExportArchive__
```
// ID : container/vm target ID mainly for power state checking in the portlayer to determine Exports logical behavior.
// deviceID : target deviceID for copying
// fileSpec : just a map[string]string of keyed paths to exclusion and strip operations determined by the desired behavior of the read
//
func Read(ID, deviceID, filespec map[string]string) (io.reader, error)
// no need for the data bool since we know that this operation involves data. we can set that as always true on the portlayer side of things.
```
__ImportArchive__
```
// StoreID : ID of the store where the target device resides be it image store, volume store, or something else in the future.
// ID : target container/vm for the portlayer to check powerstate to determine the logical approach of the ImportArchive function
// deviceID : device id found in the targeted store that is to receive the archive
// fileSpec : just a map[string]string of keyed paths to exclusion and strip operations determined by the desired behavior of the write
// tarStream : the actual tar stream to be imported to the target
func Write(ID, deviceID string, fileSpec map[string]string, tarStream io.Writer) error
```
Note: this call will involve multiple docker endpoints, Stat is needed with we plan to support the `-L` functionality. We will also need the interaction piece for streaming tar archive's to the portlayer with the same tar stream from docker being streamed to multiple calls(hopefully we might have a way to not stream the same data several times over the network).
## Personality design
### Copy Operation
The personality has Three endpoints that are called depending on the to/from situation above(5 if you count the two deprecated calls that were made previously.). The portlayer will divide the behavior of to/from as two separate endpoints. These endpoints will behave in an `Export/Import` behavior. Aside from the Import/Export behavior the docker cli also calls `ContainerStatPath` when determining whether to follow symlinks via the `-L` option. We will need this endpoint implemented all the way through to the portlayer if we want to support following symlinks.
Error codes expected by docker :
```
200
400
403 - in the event of a readonly filesystem
404
409
500
```
Docker does not do much in the way of string checking as far as the path is concerned. This is likely not a big concern for the personality as the actual copy command will fail if a bad path string is provided.
For the multiple calls to the portlayer we will need to assemble Pathspecs which detail inclusion, exclusion, rebase,and strip configurations for the tar stream on both reads and writes. Below will be some scenarios and what the path spec should look like for those scenarios.
Exclude the tether path. Exclude any paths that docker does not allow for a copy. the filespec map with a serialized struct or keys for `exclusion`, `strip`.
examples of a pathspec(please note that these are simple scenarios for the time being):
```
For a CopyTo to path "/volumes" :
container has one volume mounted at /volumes/data
this will invoke two write calls.
first pathspec:
// this pathspec would be empty since you would be writing against the r/w of the containerFS and the starting path would be '/' of the containerFS. No stripping of the write headers necessary.
spec.rebase="volumes"
spec.strip = ""
spec.includes = map[string]string {"": struct{}{}}
spec.excludes = map[string]string {"volumes/data": struct{}{}}
Second Pathspec:
spec.Strip = "volumes/data/" // stripped since the volume will be mounted and the starting path will be "/"
spec.Rebase = ""
spec.includes = map[string]string {"": struct{}{}}
}
```
```
For a Copy from path "/volumes/" :
container has one volume mounted at /volumes/data
this will invoke two read calls
// this pathspec would be empty since you would be writing against the r/w of the containerFS and the starting path would be '/' of the containerFS. No rebase necessary for the headers.
spec.Rebase = "volumes"
spec.Strip = "volumes"
// in this case strip and rebase are the same since this is a 1st level directory e.g for a target of /volumes/data rebase would be "data" and strip would be "volumes/data"
spec.include = map[string]string {"volumes":struct{}{}}
spec.Exclude = map[string]string {"": struct{}{}}
// this pathspec will be for the volume filesystem
spec.Rebase = "volumes"
spec.Strip = ""
spec.include = map[string]string {"":struct{}{}}
```
NOTE: use TarAppender in docker/pkg/archive to possibly merge the different tar streams during an ExportArchive.
### Stat Operation
## Portlayer Design
There will be two situations to be concerned with in the portlayer. When the container is *on* and when the container is *off*.
in the event of the *off* scenario we will convey the requested operations to the `Read`/`Write` calls in the storage portion of the portlayer. The results of those calls should be returned to the user.
Regardless of whether the container is on or off. Some investigation has shown that it works in the scenario of both containers being on, off, and one on and one off. We will need to architect a way for this to work for us. If the target is a volume that is attached to a container we will need to understand what is needed in the case of a vmdk volume target. This behavior should not be an issue for the nfs volume store targets.
The portlayer should behave at a basic level as such:
```
1. look up appropriate storage object
2. check object usage
3. mount object or initiate copy based on (2)
3b. check for error in initialization of copy due to power state change, repeat (2) if found
4. mounted disk prevents incompatible operations, or, tether copy in progress squashes in guest shutdown
```
some additiona notes surrounding the portlayer and swagger endpoint design:
use query string to pass FilterSpec so that it's not needed to be packaged in a custom format in body along with the tar
this assumes that callers have some knowledge of mount topology and correctly populate the FilterSpec to avoid recursing into mounts that will be directly addressed by a later call.
because it's possible for the various storage elements of a container to be in different states (e.g. container is not running so r/w and volumeA are not in use, but volumeB from the container is now used by container X) we require that separate calls be made to Import/Export for each of the distinct storage elements. This allows those calls to be routed appropriately based on current owners and usage of the element in question.
### Stat Operation
We will need a `stat` endpoint which will be used to target devices with a filesystem that can return fileinfo back to the caller. This will need to be tolerant of the power status of the target, if the target container/vm is on then we can use guest tools. Otherwise, mounting the appropriate device for the stat is necessary. Like the Read/Write calls we should expect a store ID and a device ID in addition to the target of the stat operation. if the compute is not active then we should mount the target device specified for the stat.
```
underlying filesystem stat :
// storename : this is the store that the target device resides on
// deviceID : this is the target device for the stat operation
// target : is the filesys
//
// stat will mount the target and stat the filesystem target in order to obtain the requested filesystem info.
func Stat(storename string, deviceID string, spec archive.FilterSpec) (FileInfo, error)
// it is the responsibility of the caller to determine the status of this device. If it is already mounted or in use the caller must determine the action to be taken.
```
### Portlayer Import/Export Behavior for writing and reading to storage devices.
The following are the core interfaces that allow hiding of the current usage state of a given storage element such as a volume:
```
When a container is online and a copy is attempted guest tools will be utilized to move the files onto the container. Below are the defined interfaces that will be used for reading and writing to the devices. Note that there can be multiple datasources and datasinks for the same device backing. This is due to online and offline behavior.
A docker stop should be blocked until the copy.(currently this is not the case)
// DataSource defines the methods for exporting data from a specific storage element as a tar stream
type DataSource interface {
// Close releases all resources associated with this source. Shared resources should be reference counted.
io.Closer
// Export performs an export of the specified files, returning the data as a tar stream. This is single use; once
// the export has completed it should not be assumed that the source remains functional.
//
// spec: specifies which files will be included/excluded in the export and allows for path rebasing/stripping
// data: if true the actual file data is included, if false only the file headers are present
Export(op trace.Operation, spec *archive.FilterSpec, data bool) (io.ReadCloser, error)
// Source returns the mechanism by which the data source is accessed
// Examples:
// vmdk mounted locally: *os.File
// nfs volume: XDR-client
// via guesttools: toolbox client
Source() interface{}
}
// DataSink defines the methods for importing data to a specific storage element from a tar stream
type DataSink interface {
// Close releases all resources associated with this sink. Shared resources should be reference counted.
io.Closer
// Import performs an import of the tar stream to the source held by this DataSink. This is single use; once
// the export has completed it should not be assumed that the sink remains functional.
//
// spec: specifies which files will be included/excluded in the import and allows for path rebasing/stripping
// tarStream: the tar stream to from which to import data
Import(op trace.Operation, spec *archive.FilterSpec, tarStream io.ReadCloser) error
// Sink returns the mechanism by which the data sink is accessed
// Examples:
// vmdk mounted locally: *os.File
// nfs volume: XDR-client
// via guesttools: toolbox client
Sink() interface{}
}
// Importer defines the methods needed to write data into a storage element. This should be implemented by the various
// store types.
type Importer interface {
// Import allows direct construction and invocation of a data sink for the specified ID.
Import(op trace.Operation, id string, spec *archive.FilterSpec, tarStream io.ReadCloser) error
// NewDataSink constructs a data sink for the specified ID within the context of the Importer. This is a single
// use sink which may hold resources until Closed.
NewDataSink(op trace.Operation, id string) (DataSink, error)
}
// Exporter defines the methods needed to read data from a storage element, optionally diff with an ancestor. This
// shoiuld be implemented by the various store types.
type Exporter interface {
// Export allows direct construction and invocation of a data source for the specified ID.
Export(op trace.Operation, id, ancestor string, spec *archive.FilterSpec, data bool) (io.ReadCloser, error)
// NewDataSource constructs a data source for the specified ID within the context of the Exporter. This is a single
// use source which may hold resources until Closed.
NewDataSource(op trace.Operation, id string) (DataSource, error)
}
```
This provides a pair of helper functions per store, supporting generalized implementation of the above Import/Export logic:
```
// Resolver defines methods for mapping ids to URLS, and urls to owners of that device
type Resolver interface {
// URL returns a url to the data source representing `id`
// For historic reasons this is not the same URL that other parts of the storage component use, but an actual
// URL suited for locating the storage element without having additional precursor knowledge.
// Of the form `ds://[datastore]/path/on/datastore/element.vmdk`
URL(op trace.Operation, id string) (*url.URL, error)
// Owners returns a list of VMs that are using the resource specified by `url`
Owners(op trace.Operation, url *url.URL, filter func(vm *mo.VirtualMachine) bool) ([]*vm.VirtualMachine, error)
}
```
Example usage:
```
func (h *handler) ImportArchive(store, id string, spec *archive.FilterSpec, tar io.ReadCloser) middleware.Responder {
op := trace.NewOperation(context.Background(), "ImportArchive: %s:%s", store, id)
s, ok := storage.GetImporter(store)
if !ok {
op.Errorf("Failed to locate import capable store %s", store)
op.Debugf("Available importers are: %+q", storage.GetImporters())
return storage.NewImportArchiveNotFound()
}
err := s.Import(op, id, spec, tar)
if err != nil {
// This should be usefully typed errors
return storage.NewExportArchiveInternalServerError()
}
return storage.NewImportArchiveOK()
}
```

44
vendor/github.com/vmware/vic/doc/design/docker-rename.md generated vendored Normal file
View File

@@ -0,0 +1,44 @@
This is the design document for implementing “docker rename” on vSphere Integrated Containers Engine (VIC Engine).
## Design
### The problem:
Rename involves the containerVM’s display name and the name of the datastore folder (and the files in the folder) of the containerVM on the vsphere UI, the container cache of the VIC Engine and the container network configuration (e.g., network alias). Currently both the containerVM's display name and the folder name are created during VM creation using containerName-containerID, which is done for matching the container information obtained from `docker ps` to the VM displayed on the vSphere UI. Renaming the VM display name on the UI can be achieved by using govc, which however does not update the datastore folder name. In this case, the vi admin would observe inconsistent VM display name and datastore folder name, and it becomes difficult the admin to reference the datastore folder based on the new VM display name.
### Solution:
- We use `containerName-containerShortID` to assemble the VM display name. We do not use containerName-containerID in order to avoid the scenario wherein the containerName gets truncated to satisfy the maximum length of a VM display name in vSphere. In addition, we use the `containerID` to set the name of the datastore folder, thus there is no need to worry about the VM display name and datastore folder name being inconsistent.
- VM configuration during creation:
- vSAN: Since vSAN requires the VM display name to be the same as the datastore folder name during VM creation, we set both the VM display name and the datastore folder name to `containerName-containerShortID` during VM creation.
- Non-vSAN: We set the VM display name to `containerName-containerShortID` and set the datastore folder name to `containeriD`.
- VM guestinfo: We replace `guestinfo.vice./common/name` with `common/name` in the `ExtraConfig` of a containerVM and set the scope of `common/name` to `hidden`. Then when `docker rename` is called after the VM is created, we can update the `common/name` with the new name. By doing this, we can persist the new name for a running containerVM.
- Docker support for rename: When a customer calls `docker rename`, we update the VM display name to the new name in both the docker persona and the portlayer.
- Network:
- Network alias should be updated.
- If `--link` is used when creating the container, HostConfig of relevant containers should be automatically updated based on the backend data.
- `/etc/hosts`: it will not have the containerName, since `guestinfo.vice/common/name` does not exist.
- Storage: Nothing needs to be updated if we set the datastore folder name to containerID.
- Backward compatibility for containers created by a VCH of an older version that does not support `docker rename`: During VCH upgrade, if an existing containerVM has `guestinfo.vice./common/name` instead of `common/name`, a data migration plugin will add `common/name` to the `ExtraConfig` of this containerVM and update the value accordingly, so that the docker operations that are supported by the old VCH still work. However, since we won't be able to update the binaries of the existing containerVMs, we disable `docker rename` on these containerVMs.
## Testing and Acceptance Criteria
Robot scripts will be written to test the following:
1. VM configuration:
- After a containerVM is created, use govc to check the display name (containerName-containerShortID) and datastore folder name (containerID if on a non-vSAN setup).
2. Docker support for rename:
- The basic functionality of `docker rename`
- Check validity of network alias and HostConfig
- `docker-compose up` when there are existing containers for the same service but the configuration or image has been changed
- `docker-compose up –force-recreate` when there are existing containers for the same service even if the configuration or image has not been changed
3. Backward compatibility
- Add a test case in the upgrade test. Create a container using a VCH that does not support `docker rename`. After upgrading the VCH, the basic docker operations that are supported by the old VCH should work.

27
vendor/github.com/vmware/vic/doc/design/image-management.md generated vendored Normal file
View File

@@ -0,0 +1,27 @@
### How VIC manages filesystem images
A big part of a container runtime is how it uses and manages filesystem images. There are various approaches to managing container images, but the net result needs to be enough binaries to run the main container process. Those binaries may infact be a single statically-linked executable, or it could be operating system dependencies, third party libraries etc.
Given that each containerVM in VIC does not share a filesystem with anything else, for a container VM to be functional, it requires the following:
* OS binaries from which to boot. These are contained in the ISO attached to the containerVM
* A disk, representing the container filesystem, mounted at "/". The disk may contain OS libraries, but must contain the binaries corresponding to the entry point of the container. The disk will typically be R/W and the underlying implementation may capture all reads into a layer which can be committed into a snapshot
* 0 or more R/W disks which persist beyond the lifespan of the container
#### Port Layer Storage
#### Imagec
Image resolution and the transfer of image data between an external repository and a local image store is something that should be able to happen out-of-band. This also means that it does not need to be a service that is tightly-coupled to any one VCH. A VCH should be able to make asynchronous requests to an image service which maintains a local image store. There is no reason the image store could not be shared between a number of VCHs. When an image is added to the store, the VCH should then be able to query the image store via the PortLayer for metadata about the image.
Given the out-of-band nature of this transaction, plus the potential for the exchange of image data to be be automated or even managed by a different control plane, it makes sense to provide a binary to allow for this. This binary is [imagec](../../../master/cmd/imagec/README.md) described [here](components.md#imagec)
#### Image metadata
Image metadata arguably consists of two distinct types: Immutable metadata that is added to the image on creation/push and contextual data that can be added by the user of an image to facilitate publishing/referencing. Given that a single image can be deployed by multiple users, there is also a scoping aspect to this distinction: Immutable image metadata is coupled to the image and is consumed by all users of the image; whereras user metadata exists within the context of a particular use of the image. As such, user metadata is scoped to a VCH and image metatdata is scoped to an image store. Both need to be persisted.
#### High-level control flow
This diagram gives a high-level overview of the control flow between the various components.
![system component architecture](images/docker-pull-workflow.png)

BIN
vendor/github.com/vmware/vic/doc/design/images/component_architecture.png generated vendored Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 139 KiB

5722
vendor/github.com/vmware/vic/doc/design/images/component_architecture.svg generated vendored Normal file
View File

File diff suppressed because it is too large Load Diff
Side by Side

After

Width:  |  Height:  |  Size: 563 KiB

BIN
vendor/github.com/vmware/vic/doc/design/images/docker-pull-workflow.png generated vendored Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 88 KiB

BIN
vendor/github.com/vmware/vic/doc/design/images/docker-push-logic-flow.png generated vendored Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 120 KiB

297
vendor/github.com/vmware/vic/doc/design/images/docker_create_sequence.svg generated vendored Normal file
View File

@@ -0,0 +1,297 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by Microsoft Visio, SVG Export docker_create_sequence.svg Docker Create -->
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="8.51389in" height="10.0835in"
viewBox="0 0 613 726.014" xml:space="preserve" color-interpolation-filters="sRGB" class="st17">
<v:documentProperties v:langID="1033" v:viewMarkup="false">
<v:userDefs>
<v:ud v:nameU="msvNoAutoConnect" v:prompt="" v:val="VT0(1):26"/>
</v:userDefs>
</v:documentProperties>
<style type="text/css">
<![CDATA[
.st1 {fill:#ffffff;stroke:#1e5054;stroke-linecap:round;stroke-linejoin:round;stroke-width:2}
.st2 {stroke:#1e5054;stroke-dasharray:7,5;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
.st3 {fill:#102c2f;font-family:Calibri;font-size:1.00001em;font-weight:bold}
.st4 {font-size:1em}
.st5 {marker-end:url(#mrkr4-61);stroke:#af1513;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
.st6 {fill:#af1513;fill-opacity:1;stroke:#af1513;stroke-opacity:1;stroke-width:0.23584905660377}
.st7 {fill:#ffffff;stroke:none;stroke-linecap:butt;stroke-width:7.2}
.st8 {fill:#610c0b;font-family:Calibri;font-size:0.666664em}
.st9 {marker-end:url(#mrkr4-69);stroke:#af1513;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.3;stroke-width:1}
.st10 {fill:#af1513;fill-opacity:0.3;stroke:#af1513;stroke-opacity:0.3;stroke-width:0.23584905660377}
.st11 {fill:#1e5054;fill-opacity:0;stroke:#1e5054;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
.st12 {fill:#ffffff;stroke:#1e5054;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
.st13 {fill:#1e5155;font-family:Calibri;font-size:0.666664em;font-weight:bold}
.st14 {fill:#1e5054;font-family:Calibri;font-size:0.666664em}
.st15 {marker-end:url(#mrkr3-99);stroke:#af1513;stroke-dasharray:7,5;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
.st16 {fill:#ffffff;stroke:#af1513;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
.st17 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
]]>
</style>
<defs id="Markers">
<g id="lend4">
<path d="M 2 1 L 0 0 L 2 -1 L 2 1 " style="stroke:none"/>
</g>
<marker id="mrkr4-61" class="st6" v:arrowType="4" v:arrowSize="3" v:setback="8.48" refX="-8.48" orient="auto"
markerUnits="strokeWidth" overflow="visible">
<use xlink:href="#lend4" transform="scale(-4.24,-4.24) "/>
</marker>
<marker id="mrkr4-69" class="st10" v:arrowType="4" v:arrowSize="3" v:setback="8.48" refX="-8.48" orient="auto"
markerUnits="strokeWidth" overflow="visible">
<use xlink:href="#lend4" transform="scale(-4.24,-4.24) "/>
</marker>
<g id="lend3">
<path d="M 2 1 L 0 0 L 2 -1 " style="stroke-linecap:round;stroke-linejoin:round;fill:none"/>
</g>
<marker id="mrkr3-99" class="st6" v:arrowType="3" v:arrowSize="3" orient="auto" markerUnits="strokeWidth"
overflow="visible">
<use xlink:href="#lend3" transform="scale(-4.24,-4.24) "/>
</marker>
</defs>
<g v:mID="0" v:index="1" v:groupContext="foregroundPage">
<v:userDefs>
<v:ud v:nameU="msvThemeOrder" v:val="VT0(0):26"/>
</v:userDefs>
<title>Docker Create</title>
<v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/>
<v:layer v:name="Connector" v:index="0"/>
<g id="group1-1" transform="translate(19,-633.214)" v:mID="1" v:groupContext="group">
<v:userDefs>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="ShowActor" v:prompt="" v:val="VT0(1):5"/>
<v:ud v:nameU="ShowDestruction" v:prompt="" v:val="VT0(0):5"/>
</v:userDefs>
<title>Actor lifeline</title>
<desc>docker client</desc>
<g id="shape2-2" v:mID="2" v:groupContext="shape" transform="translate(0,-4.5)">
<title>Sheet.2</title>
<rect x="0" y="700.814" width="54" height="25.2" class="st1"/>
</g>
<g id="shape3-4" v:mID="3" v:groupContext="shape" transform="translate(18,-37.8)">
<title>Sheet.3</title>
<path d="M3.6 726.01 L3.6 708.01 L3.6 715.21 L0 715.21 L0 700.81 L1.8 699.01 L16.2 699.01 L18 700.81 L18 715.21 L14.4
715.21 L14.4 708.01 L14.4 726.01 L9 726.01 L9 711.61 L9 726.01 L3.6 726.01 ZM12.6 693.61 A3.6 3.6 -180
1 0 5.4 693.61 A3.6 3.6 -180 1 0 12.6 693.61 Z" class="st1"/>
</g>
<g id="shape4-6" v:mID="4" v:groupContext="shape" transform="translate(18,615.214)">
<title>Sheet.4</title>
</g>
<g id="shape5-8" v:mID="5" v:groupContext="shape" transform="translate(753.014,726.014) rotate(90)">
<title>Sheet.5</title>
<path d="M0 726.01 L594 726.01" class="st2"/>
</g>
<g id="shape1-11" v:mID="1" v:groupContext="groupContent">
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="27" cy="708.914" width="54" height="34.2"/>
<rect x="0" y="691.814" width="54" height="34.2" class="st1"/>
<text x="10.01" y="705.31" class="st3" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>docker <tspan
x="13.22" dy="1.2em" class="st4">client</tspan></text> </g>
</g>
<g id="group6-15" transform="translate(145,-633.214)" v:mID="6" v:groupContext="group">
<v:userDefs>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="ShowActor" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="ShowDestruction" v:prompt="" v:val="VT0(0):5"/>
</v:userDefs>
<title>Object lifeline</title>
<desc>docker API</desc>
<g id="shape7-16" v:mID="7" v:groupContext="shape" transform="translate(0,-3.6)">
<title>Sheet.7</title>
<rect x="0" y="700.814" width="54" height="25.2" class="st1"/>
</g>
<g id="shape8-18" v:mID="8" v:groupContext="shape" transform="translate(18,-36)">
<title>Sheet.8</title>
<path d="M9 726.01 a0 0 0 1 1 0 0 a0 0 0 1 1 0 0 Z" class="st1"/>
</g>
<g id="shape9-20" v:mID="9" v:groupContext="shape" transform="translate(18,605.571)">
<title>Sheet.9</title>
</g>
<g id="shape10-22" v:mID="10" v:groupContext="shape" transform="translate(753.014,726.014) rotate(90)">
<title>Sheet.10</title>
<path d="M0 726.01 L585 726.01" class="st2"/>
</g>
<g id="shape6-25" v:mID="6" v:groupContext="groupContent">
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="27" cy="709.814" width="54" height="32.4"/>
<rect x="0" y="693.614" width="54" height="32.4" class="st1"/>
<text x="10.01" y="706.21" class="st3" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>docker <tspan
x="18.57" dy="1.2em" class="st4">API</tspan></text> </g>
</g>
<g id="group21-29" transform="translate(306,-640.414)" v:mID="21" v:groupContext="group">
<v:userDefs>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="ShowActor" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="ShowDestruction" v:prompt="" v:val="VT0(0):5"/>
</v:userDefs>
<title>Object lifeline.21</title>
<desc>Port Layer</desc>
<g id="shape22-30" v:mID="22" v:groupContext="shape">
<title>Sheet.22</title>
<rect x="0" y="700.814" width="54" height="25.2" class="st1"/>
</g>
<g id="shape23-32" v:mID="23" v:groupContext="shape" transform="translate(18,-28.8)">
<title>Sheet.23</title>
<path d="M9 726.01 a0 0 0 1 1 0 0 a0 0 0 1 1 0 0 Z" class="st1"/>
</g>
<g id="shape24-34" v:mID="24" v:groupContext="shape" transform="translate(18,621)">
<title>Sheet.24</title>
</g>
<g id="shape25-36" v:mID="25" v:groupContext="shape" transform="translate(753.014,726.014) rotate(90)">
<title>Sheet.25</title>
<path d="M0 726.01 L603 726.01" class="st2"/>
</g>
<g id="shape21-39" v:mID="21" v:groupContext="groupContent">
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="27" cy="713.414" width="54" height="25.2"/>
<rect x="0" y="700.814" width="54" height="25.2" class="st1"/>
<text x="16.37" y="709.81" class="st3" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Port <tspan
x="13.51" dy="1.2em" class="st4">Layer</tspan></text> </g>
</g>
<g id="group26-43" transform="translate(540,-640.414)" v:mID="26" v:groupContext="group">
<v:userDefs>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="ShowActor" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="ShowDestruction" v:prompt="" v:val="VT0(0):5"/>
</v:userDefs>
<title>Object lifeline.26</title>
<desc>vCenter</desc>
<g id="shape27-44" v:mID="27" v:groupContext="shape">
<title>Sheet.27</title>
<rect x="0" y="700.814" width="54" height="25.2" class="st1"/>
</g>
<g id="shape28-46" v:mID="28" v:groupContext="shape" transform="translate(18,-28.8)">
<title>Sheet.28</title>
<path d="M9 726.01 a0 0 0 1 1 0 0 a0 0 0 1 1 0 0 Z" class="st1"/>
</g>
<g id="shape29-48" v:mID="29" v:groupContext="shape" transform="translate(18,621)">
<title>Sheet.29</title>
</g>
<g id="shape30-50" v:mID="30" v:groupContext="shape" transform="translate(753.014,726.014) rotate(90)">
<title>Sheet.30</title>
<path d="M0 726.01 L603 726.01" class="st2"/>
</g>
<g id="shape26-53" v:mID="26" v:groupContext="groupContent">
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="27" cy="713.414" width="54" height="25.2"/>
<rect x="0" y="700.814" width="54" height="25.2" class="st1"/>
<text x="7.51" y="717.01" class="st3" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>vCenter</text> </g>
</g>
<g id="shape31-56" v:mID="31" v:groupContext="shape" v:layerMember="0" transform="translate(46,-606.214)">
<title>Message.22</title>
<desc>TLS (HTTP connection)</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="63" cy="717.014" width="82.45" height="17.6036"/>
<path d="M0 717.01 L117.52 717.01" class="st5"/>
<rect v:rectContext="textBkgnd" x="26.686" y="712.215" width="72.6281" height="9.59985" class="st7"/>
<text x="26.69" y="719.41" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>TLS (HTTP connection)</text> </g>
<g id="shape83-64" v:mID="83" v:groupContext="shape" v:layerMember="0" transform="translate(172,-606.214)">
<title>Message.22.83</title>
<desc>possible audit log mapping for all operations over this conne...</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="197.5" cy="717.014" width="222.44" height="17.6036"/>
<path d="M0 717.01 L386.52 717.01" class="st9"/>
<rect v:rectContext="textBkgnd" x="91.1908" y="712.215" width="212.619" height="9.59985" class="st7"/>
<text x="91.19" y="719.41" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>possible audit log mapping for all operations over this connection</text> </g>
<g id="group87-72" transform="translate(46,-511.714)" v:mID="87" v:groupContext="group">
<v:userDefs>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<title>Optional fragment</title>
<g id="shape87-73" v:mID="87" v:groupContext="groupContent">
<rect x="0" y="667.514" width="125" height="58.5" class="st11"/>
</g>
<g id="shape88-75" v:mID="88" v:groupContext="shape" transform="translate(0,-40.8964)">
<title>Sheet.88</title>
<desc>opt</desc>
<v:userDefs>
<v:ud v:nameU="FoldSize" v:prompt="" v:val="VT0(0.1):0"/>
<v:ud v:nameU="BackFillColor" v:prompt="" v:val="VT5(#96afcf)"/>
<v:ud v:nameU="BackLineColor" v:prompt="" v:val="VT5(#1f477d)"/>
<v:ud v:nameU="BackCharColor" v:prompt="" v:val="VT5(#000000)"/>
<v:ud v:nameU="DarkerColor" v:prompt="" v:val="VT5(#000000)"/>
<v:ud v:nameU="BackFillColor" v:prompt="" v:val="VT5(#1e5155)"/>
<v:ud v:nameU="BackLineColor" v:prompt="" v:val="VT5(#1e5155)"/>
<v:ud v:nameU="BackCharColor" v:prompt="" v:val="VT5(#ffffff)"/>
<v:ud v:nameU="DarkerColor" v:prompt="" v:val="VT5(#1e5155)"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="32" cy="717.213" width="64" height="17.6036"/>
<path d="M0 726.01 L56.8 726.01 L64 718.81 L64 708.41 L0 708.41 L0 726.01 Z" class="st12"/>
<text x="4" y="719.61" class="st13" v:langID="1033"><v:paragraph/><v:tabList/>opt</text> </g>
<g id="shape89-78" v:mID="89" v:groupContext="shape" transform="translate(0,-23.2929)">
<title>Sheet.89</title>
<desc>image does not exist</desc>
<v:userDefs>
<v:ud v:nameU="msvQuickStyleVariationRestore" v:prompt="" v:val="VT0(2):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="38.3821" cy="717.213" width="76.77" height="17.6036"/>
<text x="4" y="719.61" class="st14" v:langID="1033"><v:paragraph/><v:tabList/>image does not exist</text> </g>
</g>
<g id="shape90-80" v:mID="90" v:groupContext="shape" v:layerMember="0" transform="translate(46,-570.214)">
<title>Message.22.90</title>
<desc>create</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="60.75" cy="717.014" width="40" height="17.6036"/>
<path d="M0 717.01 L113.02 717.01" class="st5"/>
<rect v:rectContext="textBkgnd" x="50.4279" y="712.215" width="20.6441" height="9.59985" class="st7"/>
<text x="50.43" y="719.41" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>create</text> </g>
<g id="shape91-87" v:mID="91" v:groupContext="shape" v:layerMember="0" transform="translate(46,-516.214)">
<title>Message.22.91</title>
<desc>pull</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="60.75" cy="717.014" width="40" height="17.6036"/>
<path d="M0 717.01 L113.02 717.01" class="st5"/>
<rect v:rectContext="textBkgnd" x="54.7112" y="712.215" width="12.0778" height="9.59985" class="st7"/>
<text x="54.71" y="719.41" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>pull</text> </g>
<g id="shape93-94" v:mID="93" v:groupContext="shape" v:layerMember="0" transform="translate(171,-525.214)">
<title>Return Message.26</title>
<desc>404</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(1):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="-63" cy="717.014" width="40" height="17.6036"/>
<path d="M0 717.01 L-126 717.01" class="st15"/>
<rect v:rectContext="textBkgnd" x="-69.0822" y="712.215" width="12.1643" height="9.59985" class="st7"/>
<text x="-69.08" y="719.41" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>404</text> </g>
<g id="shape94-102" v:mID="94" v:groupContext="shape" transform="translate(-549.514,470.8) rotate(-90)">
<title>Activation</title>
<v:userDefs>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<rect x="0" y="717.014" width="324" height="9" class="st16"/>
</g>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 16 KiB

727
vendor/github.com/vmware/vic/doc/design/images/docker_pull_sequence.svg generated vendored Normal file
View File

@@ -0,0 +1,727 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by Microsoft Visio, SVG Export docker_pull_sequence.svg Page-1 -->
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="10.5278in" height="10.1639in"
viewBox="0 0 758.002 731.801" xml:space="preserve" color-interpolation-filters="sRGB" class="st19">
<v:documentProperties v:langID="1033">
<v:userDefs>
<v:ud v:nameU="msvNoAutoConnect" v:prompt="" v:val="VT0(1):26"/>
</v:userDefs>
</v:documentProperties>
<style type="text/css">
<![CDATA[
.st1 {fill:#1e5054;fill-opacity:0;stroke:#1e5054;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
.st2 {fill:#ffffff;stroke:#1e5054;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
.st3 {fill:#1e5155;font-family:Calibri;font-size:0.666664em;font-weight:bold}
.st4 {fill:#1e5054;font-family:Calibri;font-size:0.666664em}
.st5 {fill:#ffffff;stroke:#1e5054;stroke-linecap:round;stroke-linejoin:round;stroke-width:2}
.st6 {fill:none}
.st7 {stroke:#1e5054;stroke-dasharray:7.00001,5.00001;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
.st8 {fill:#102c2f;font-family:Calibri;font-size:1.00001em;font-weight:bold}
.st9 {font-size:1em}
.st10 {marker-end:url(#mrkr4-78);stroke:#af1513;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
.st11 {fill:#af1513;fill-opacity:1;stroke:#af1513;stroke-opacity:1;stroke-width:0.23584925362398}
.st12 {fill:#ffffff;stroke:none;stroke-linecap:butt}
.st13 {fill:#610c0b;font-family:Calibri;font-size:0.666664em}
.st14 {fill:#ffffff;stroke:none;stroke-linecap:butt;stroke-width:7.2}
.st15 {fill:#ffffff;stroke:#af1513;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
.st16 {marker-end:url(#mrkr3-130);stroke:#af1513;stroke-dasharray:7.00001,5.00001;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
.st17 {marker-end:url(#mrkr4-281);stroke:#af1513;stroke-linecap:round;stroke-linejoin:round;stroke-opacity:0.3;stroke-width:1}
.st18 {fill:#af1513;fill-opacity:0.3;stroke:#af1513;stroke-opacity:0.3;stroke-width:0.23584925362398}
.st19 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
]]>
</style>
<defs id="Markers">
<g id="lend4">
<path d="M 2 1 L 0 0 L 2 -1 L 2 1 " style="stroke:none"/>
</g>
<marker id="mrkr4-78" class="st11" v:arrowType="4" v:arrowSize="3" v:setback="8.47999" refX="-8.479992916105" orient="auto"
markerUnits="strokeWidth" overflow="visible">
<use xlink:href="#lend4" transform="scale(-4.2399964580525,-4.2399964580525) "/>
</marker>
<g id="lend3">
<path d="M 2 1 L 0 0 L 2 -1 " style="stroke-linecap:round;stroke-linejoin:round;fill:none"/>
</g>
<marker id="mrkr3-130" class="st11" v:arrowType="3" v:arrowSize="3" orient="auto" markerUnits="strokeWidth"
overflow="visible">
<use xlink:href="#lend3" transform="scale(-4.2399964580525,-4.2399964580525) "/>
</marker>
<marker id="mrkr4-281" class="st18" v:arrowType="4" v:arrowSize="3" v:setback="8.47999" refX="-8.479992916105"
orient="auto" markerUnits="strokeWidth" overflow="visible">
<use xlink:href="#lend4" transform="scale(-4.2399964580525,-4.2399964580525) "/>
</marker>
</defs>
<g v:mID="0" v:index="1" v:groupContext="foregroundPage">
<v:userDefs>
<v:ud v:nameU="msvThemeOrder" v:val="VT0(0):26"/>
</v:userDefs>
<title>Docker Pull</title>
<v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9.00001"
v:shadowOffsetY="-9.00001"/>
<v:layer v:name="Connector" v:index="0"/>
<g id="group1-1" transform="translate(298,-99.0001)" v:mID="1" v:groupContext="group">
<v:userDefs>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<title>Loop fragment</title>
<g id="shape2-2" v:mID="2" v:groupContext="shape" transform="translate(0.000385679,0)">
<title>shape47-2</title>
<rect x="0" y="308.8" width="216" height="423" class="st1"/>
</g>
<g id="shape3-4" v:mID="3" v:groupContext="shape" transform="translate(0.000385679,-405.396)">
<title>Sheet.3</title>
<desc>loop</desc>
<v:userDefs>
<v:ud v:nameU="FoldSize" v:prompt="" v:val="VT0(0.1):0"/>
<v:ud v:nameU="BackFillColor" v:prompt="" v:val="VT5(#96afcf)"/>
<v:ud v:nameU="BackLineColor" v:prompt="" v:val="VT5(#1f477d)"/>
<v:ud v:nameU="BackCharColor" v:prompt="" v:val="VT5(#000000)"/>
<v:ud v:nameU="DarkerColor" v:prompt="" v:val="VT5(#000000)"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="31.9996" cy="722.999" width="64.01" height="17.6036"/>
<path d="M0 731.8 L56.8 731.8 L64 724.6 L64 714.2 L0 714.2 L0 731.8 Z" class="st2"/>
<text x="4" y="725.4" class="st3" v:langID="1033"><v:paragraph/><v:tabList/>loop</text> </g>
<g id="shape4-7" v:mID="4" v:groupContext="shape" transform="translate(1.07914E-12,-387.794)">
<title>Sheet.4</title>
<desc>fsLayers</desc>
<v:userDefs>
<v:ud v:nameU="msvQuickStyleVariationRestore" v:prompt="" v:val="VT0(2):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="32" cy="722.999" width="64.01" height="17.6036"/>
<text x="4" y="725.4" class="st4" v:langID="1033"><v:paragraph/><v:tabList/>fsLayers</text> </g>
</g>
<g id="group5-9" transform="translate(253.001,-45)" v:mID="5" v:groupContext="group">
<v:userDefs>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="ShowActor" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="ShowDestruction" v:prompt="" v:val="VT0(0):5"/>
</v:userDefs>
<title>Object lifeline.11</title>
<desc>imagec</desc>
<g id="shape6-10" v:mID="6" v:groupContext="shape" transform="translate(-2.39808E-14,-594.001)">
<title>Sheet.6</title>
<rect x="0" y="706.601" width="54.0001" height="25.2" class="st5"/>
</g>
<g id="shape7-12" v:mID="7" v:groupContext="shape" transform="translate(18,-622.801)">
<title>Sheet.7</title>
</g>
<g id="shape9-13" v:mID="9" v:groupContext="shape" transform="translate(758.801,137.8) rotate(90)">
<title>Sheet.9</title>
<path d="M0 731.8 L594 731.8 L0 731.8 Z" class="st6"/>
<path d="M0 731.8 L594 731.8" class="st7"/>
</g>
<g id="shape10-16" v:mID="10" v:groupContext="shape" transform="translate(-2.39808E-14,-594.001)">
<title>shape11-19</title>
<desc>imagec</desc>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="26.9996" cy="719.201" width="54.01" height="25.2"/>
<rect x="0" y="706.601" width="54.0001" height="25.2" class="st5"/>
<text x="9.31" y="722.8" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>imagec</text> </g>
</g>
<g id="group11-19" transform="translate(19.0004,-45)" v:mID="11" v:groupContext="group">
<v:userDefs>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="ShowActor" v:prompt="" v:val="VT0(1):5"/>
<v:ud v:nameU="ShowDestruction" v:prompt="" v:val="VT0(0):5"/>
</v:userDefs>
<title>Actor lifeline</title>
<desc>docker client</desc>
<g id="shape12-20" v:mID="12" v:groupContext="shape" transform="translate(-7.99361E-15,-598.501)">
<title>Sheet.12</title>
<rect x="0" y="706.601" width="54.0001" height="25.2" class="st5"/>
</g>
<g id="shape13-22" v:mID="13" v:groupContext="shape" transform="translate(18,-631.801)">
<title>Sheet.13</title>
<path d="M3.6 731.8 L3.6 713.8 L3.6 721 L0 721 L0 706.6 L1.8 704.8 L16.2 704.8 L18 706.6 L18 721 L14.4 721 L14.4
713.8 L14.4 731.8 L9 731.8 L9 717.4 L9 731.8 L3.6 731.8 ZM12.6 699.4 A3.6 3.6 -180 1 0 5.4 699.4 A3.6
3.6 -180 1 0 12.6 699.4 Z" class="st5"/>
</g>
<g id="shape15-24" v:mID="15" v:groupContext="shape" transform="translate(758.801,137.8) rotate(90)">
<title>Sheet.15</title>
<path d="M0 731.8 L594 731.8 L0 731.8 Z" class="st6"/>
<path d="M0 731.8 L594 731.8" class="st7"/>
</g>
<g id="shape16-27" v:mID="16" v:groupContext="shape" transform="translate(-7.99361E-15,-594.001)">
<title>shape1-32</title>
<desc>docker client</desc>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="26.9996" cy="714.701" width="54.01" height="34.2"/>
<rect x="0" y="697.601" width="54.0001" height="34.2" class="st5"/>
<text x="10.01" y="711.1" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>docker <tspan
x="13.22" dy="1.2em" class="st9">client</tspan></text> </g>
</g>
<g id="group17-31" transform="translate(145.001,-54.0001)" v:mID="17" v:groupContext="group">
<v:userDefs>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="ShowActor" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="ShowDestruction" v:prompt="" v:val="VT0(0):5"/>
</v:userDefs>
<title>Object lifeline</title>
<desc>docker API</desc>
<g id="shape18-32" v:mID="18" v:groupContext="shape" transform="translate(0,-588.601)">
<title>Sheet.18</title>
<rect x="0" y="706.601" width="54.0001" height="25.2" class="st5"/>
</g>
<g id="shape19-34" v:mID="19" v:groupContext="shape" transform="translate(18,-621.001)">
<title>Sheet.19</title>
</g>
<g id="shape21-35" v:mID="21" v:groupContext="shape" transform="translate(758.801,146.8) rotate(90)">
<title>Sheet.21</title>
<path d="M0 731.8 L585 731.8 L0 731.8 Z" class="st6"/>
<path d="M0 731.8 L585 731.8" class="st7"/>
</g>
<g id="shape22-38" v:mID="22" v:groupContext="shape" transform="translate(0,-585.001)">
<title>shape6-46</title>
<desc>docker API</desc>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="26.9996" cy="715.601" width="54.01" height="32.4"/>
<rect x="0" y="699.401" width="54.0001" height="32.4" class="st5"/>
<text x="10.01" y="712" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>docker <tspan
x="18.57" dy="1.2em" class="st9">API</tspan></text> </g>
</g>
<g id="group23-42" transform="translate(361.001,-351)" v:mID="23" v:groupContext="group">
<v:userDefs>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="ShowActor" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="ShowDestruction" v:prompt="" v:val="VT0(0):5"/>
</v:userDefs>
<title>Object lifeline.16</title>
<desc>registry</desc>
<g id="shape24-43" v:mID="24" v:groupContext="shape" transform="translate(0,-288)">
<title>Sheet.24</title>
<rect x="0" y="706.601" width="54.0001" height="25.2" class="st5"/>
</g>
<g id="shape25-45" v:mID="25" v:groupContext="shape" transform="translate(18,-316.8)">
<title>Sheet.25</title>
</g>
<g id="shape27-46" v:mID="27" v:groupContext="shape" transform="translate(758.801,443.8) rotate(90)">
<title>Sheet.27</title>
<path d="M0 731.8 L288 731.8 L0 731.8 Z" class="st6"/>
<path d="M0 731.8 L288 731.8" class="st7"/>
</g>
<g id="shape28-49" v:mID="28" v:groupContext="shape" transform="translate(0,-288)">
<title>shape16-60</title>
<desc>registry</desc>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="26.9996" cy="719.201" width="54.01" height="25.2"/>
<rect x="0" y="706.601" width="54.0001" height="25.2" class="st5"/>
<text x="8.08" y="722.8" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>registry</text> </g>
</g>
<g id="group29-52" transform="translate(505.001,-36)" v:mID="29" v:groupContext="group">
<v:userDefs>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="ShowActor" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="ShowDestruction" v:prompt="" v:val="VT0(0):5"/>
</v:userDefs>
<title>Object lifeline.21</title>
<desc>Port Layer</desc>
<g id="shape30-53" v:mID="30" v:groupContext="shape" transform="translate(0,-603.001)">
<title>Sheet.30</title>
<rect x="0" y="706.601" width="54.0001" height="25.2" class="st5"/>
</g>
<g id="shape31-55" v:mID="31" v:groupContext="shape" transform="translate(18,-631.801)">
<title>Sheet.31</title>
</g>
<g id="shape33-56" v:mID="33" v:groupContext="shape" transform="translate(758.801,128.8) rotate(90)">
<title>Sheet.33</title>
<path d="M0 731.8 L603 731.8 L0 731.8 Z" class="st6"/>
<path d="M0 731.8 L603 731.8" class="st7"/>
</g>
<g id="shape34-59" v:mID="34" v:groupContext="shape" transform="translate(0,-603.001)">
<title>shape21-73</title>
<desc>Port Layer</desc>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="26.9996" cy="719.201" width="54.01" height="25.2"/>
<rect x="0" y="706.601" width="54.0001" height="25.2" class="st5"/>
<text x="16.37" y="715.6" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Port <tspan
x="13.51" dy="1.2em" class="st9">Layer</tspan></text> </g>
</g>
<g id="group35-63" transform="translate(685.001,-36)" v:mID="35" v:groupContext="group">
<v:userDefs>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="ShowActor" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="ShowDestruction" v:prompt="" v:val="VT0(0):5"/>
</v:userDefs>
<title>Object lifeline.26</title>
<desc>vCenter</desc>
<g id="shape36-64" v:mID="36" v:groupContext="shape" transform="translate(2.79776E-14,-603.001)">
<title>Sheet.36</title>
<rect x="0" y="706.601" width="54.0001" height="25.2" class="st5"/>
</g>
<g id="shape37-66" v:mID="37" v:groupContext="shape" transform="translate(18,-631.801)">
<title>Sheet.37</title>
</g>
<g id="shape39-67" v:mID="39" v:groupContext="shape" transform="translate(758.801,128.8) rotate(90)">
<title>Sheet.39</title>
<path d="M0 731.8 L603 731.8 L0 731.8 Z" class="st6"/>
<path d="M0 731.8 L603 731.8" class="st7"/>
</g>
<g id="shape40-70" v:mID="40" v:groupContext="shape" transform="translate(2.79776E-14,-603.001)">
<title>shape26-87</title>
<desc>vCenter</desc>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="26.9996" cy="719.201" width="54.01" height="25.2"/>
<rect x="0" y="706.601" width="54.0001" height="25.2" class="st5"/>
<text x="7.51" y="722.8" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>vCenter</text> </g>
</g>
<g id="shape41-73" v:mID="41" v:groupContext="shape" v:layerMember="0" transform="translate(46.0004,-621.001)">
<title>Message.22</title>
<desc>TLS (HTTP connection)</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="62.9997" cy="731.801" width="82.46" height="17.6036"/>
<path d="M0 731.8 L117.52 731.8 L117.52 731.8" class="st10"/>
<rect v:rectContext="textBkgnd" x="26.6855" y="727.001" width="72.6281" height="9.59985" class="st12"/>
<text x="26.69" y="734.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>TLS (HTTP connection)</text> </g>
<g id="shape42-81" v:mID="42" v:groupContext="shape" v:layerMember="0" transform="translate(46.0004,-585.001)">
<title>Message.22.32</title>
<desc>login</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="62.9997" cy="731.801" width="40.01" height="17.6036"/>
<path d="M0 731.8 L117.52 731.8 L117.52 731.8" class="st10"/>
<rect v:rectContext="textBkgnd" x="55.0703" y="727.001" width="15.859" height="9.59985" class="st14"/>
<text x="55.07" y="734.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>login</text> </g>
<g id="shape43-88" v:mID="43" v:groupContext="shape" v:layerMember="0" transform="translate(46.0004,-549.001)">
<title>Message.22.33</title>
<desc>pull</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="62.9997" cy="731.801" width="40.01" height="17.6036"/>
<path d="M0 731.8 L117.52 731.8 L117.52 731.8" class="st10"/>
<rect v:rectContext="textBkgnd" x="56.9608" y="727.001" width="12.0778" height="9.59985" class="st14"/>
<text x="56.96" y="734.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>pull</text> </g>
<g id="shape44-95" v:mID="44" v:groupContext="shape" v:layerMember="0" transform="translate(172.001,-585.001)">
<title>Message.22.34</title>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<path d="M0 731.8 L99.52 731.8 L99.52 731.8" class="st10"/>
</g>
<g id="shape45-100" v:mID="45" v:groupContext="shape" v:layerMember="0" transform="translate(280.001,-585.001)">
<title>Message.22.35</title>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<path d="M0 731.8 L99.52 731.8 L99.52 731.8" class="st10"/>
</g>
<g id="shape46-105" v:mID="46" v:groupContext="shape" transform="translate(-555.3,632.801) rotate(-90)">
<title>Activation</title>
<v:userDefs>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<rect x="0" y="722.801" width="450" height="9.00001" class="st15"/>
</g>
<g id="group47-107" transform="translate(46.0004,-570.601)" v:mID="47" v:groupContext="group">
<v:userDefs>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<title>Optional fragment</title>
<g id="shape48-108" v:mID="48" v:groupContext="shape">
<title>shape37-125</title>
<rect x="0" y="697.601" width="342" height="34.2" class="st1"/>
</g>
<g id="shape49-110" v:mID="49" v:groupContext="shape" transform="translate(-9.19265E-14,-16.5964)">
<title>Sheet.49</title>
<desc>opt</desc>
<v:userDefs>
<v:ud v:nameU="FoldSize" v:prompt="" v:val="VT0(0.1):0"/>
<v:ud v:nameU="BackFillColor" v:prompt="" v:val="VT5(#96afcf)"/>
<v:ud v:nameU="BackLineColor" v:prompt="" v:val="VT5(#1f477d)"/>
<v:ud v:nameU="BackCharColor" v:prompt="" v:val="VT5(#000000)"/>
<v:ud v:nameU="DarkerColor" v:prompt="" v:val="VT5(#000000)"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="31.9996" cy="722.999" width="64.01" height="17.6036"/>
<path d="M0 731.8 L56.8 731.8 L64 724.6 L64 714.2 L0 714.2 L0 731.8 Z" class="st2"/>
<text x="4" y="725.4" class="st3" v:langID="1033"><v:paragraph/><v:tabList/>opt</text> </g>
</g>
<g id="shape50-113" v:mID="50" v:groupContext="shape" v:layerMember="0" transform="translate(176.501,-549.001)">
<title>Message.22.44</title>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<path d="M0 731.8 L95.02 731.8 L95.02 731.8" class="st10"/>
</g>
<g id="shape51-118" v:mID="51" v:groupContext="shape" v:layerMember="0" transform="translate(280.001,-549.001)">
<title>Message.22.45</title>
<desc>resolve</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="53.9997" cy="731.801" width="40.01" height="17.6036"/>
<path d="M0 731.8 L99.52 731.8 L99.52 731.8" class="st10"/>
<rect v:rectContext="textBkgnd" x="42.2265" y="727.001" width="23.5467" height="9.59985" class="st14"/>
<text x="42.23" y="734.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>resolve</text> </g>
<g id="shape52-125" v:mID="52" v:groupContext="shape" v:layerMember="0" transform="translate(280.001,-540.001)">
<title>Return Message.26</title>
<desc>manifest</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(1):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="53.9997" cy="731.801" width="40.01" height="17.6036"/>
<path d="M108 731.8 L0 731.8" class="st16"/>
<rect v:rectContext="textBkgnd" x="39.754" y="727.001" width="28.4917" height="9.59985" class="st14"/>
<text x="39.75" y="734.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>manifest</text> </g>
<g id="shape53-133" v:mID="53" v:groupContext="shape" v:layerMember="0" transform="translate(284.501,-477.001)">
<title>Message.22.53</title>
<desc>getLayer</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="123.75" cy="731.801" width="40.01" height="17.6036"/>
<path d="M0 731.8 L239.02 731.8 L239.02 731.8" class="st10"/>
<rect v:rectContext="textBkgnd" x="109.744" y="727.001" width="28.0113" height="9.59985" class="st14"/>
<text x="109.74" y="734.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>getLayer</text> </g>
<g id="shape54-140" v:mID="54" v:groupContext="shape" v:layerMember="0" transform="translate(284.501,-423)">
<title>Return Message.26.54</title>
<desc>handle</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(1):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="123.75" cy="731.801" width="40.01" height="17.6036"/>
<path d="M247.5 731.8 L0 731.8" class="st16"/>
<rect v:rectContext="textBkgnd" x="112.621" y="727.001" width="22.2575" height="9.59985" class="st14"/>
<text x="112.62" y="734.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>handle</text> </g>
<g id="shape55-147" v:mID="55" v:groupContext="shape" v:layerMember="0" transform="translate(284.501,-306)">
<title>Message.22.59</title>
<desc>createLayer</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="123.75" cy="731.801" width="48.06" height="17.6036"/>
<path d="M0 731.8 L239.02 731.8 L239.02 731.8" class="st10"/>
<rect v:rectContext="textBkgnd" x="104.635" y="727.001" width="38.2299" height="9.59985" class="st14"/>
<text x="104.63" y="734.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>createLayer</text> </g>
<g id="shape56-154" v:mID="56" v:groupContext="shape" v:layerMember="0" transform="translate(532.001,-306)">
<title>Message.22.60</title>
<desc>create VMDK</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="89.9997" cy="731.801" width="52.74" height="17.6036"/>
<path d="M0 731.8 L171.52 731.8 L171.52 731.8" class="st10"/>
<rect v:rectContext="textBkgnd" x="68.545" y="727.001" width="42.9096" height="9.59985" class="st14"/>
<text x="68.55" y="734.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>create VMDK</text> </g>
<g id="shape57-161" v:mID="57" v:groupContext="shape" v:layerMember="0" transform="translate(532.001,-477.001)">
<title>Message.22.61</title>
<desc>exists?</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="89.9997" cy="731.801" width="40.01" height="17.6036"/>
<path d="M0 731.8 L171.52 731.8 L171.52 731.8" class="st10"/>
<rect v:rectContext="textBkgnd" x="79.037" y="727.001" width="21.9254" height="9.59985" class="st14"/>
<text x="79.04" y="734.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>exists?</text> </g>
<g id="shape58-168" v:mID="58" v:groupContext="shape" v:layerMember="0" transform="translate(532.001,-423)">
<title>Self Message.24.63</title>
<desc>map result to handle</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(1):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="-0.000385679" cy="713.801" width="77.45" height="17.6036"/>
<path d="M0 695.8 L18 695.8 L18 731.8 L8.48 731.8 L8.48 731.8" class="st10"/>
<rect v:rectContext="textBkgnd" x="-33.8123" y="709.001" width="67.6241" height="9.59985" class="st14"/>
<text x="-33.81" y="716.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>map result to handle</text> </g>
<g id="shape59-175" v:mID="59" v:groupContext="shape" v:layerMember="0" transform="translate(532.001,-468.001)">
<title>Return Message.26.64</title>
<desc>true/false/blob?</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(1):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="89.9997" cy="731.801" width="63.05" height="17.6036"/>
<path d="M180 731.8 L0 731.8" class="st16"/>
<rect v:rectContext="textBkgnd" x="63.3906" y="727.001" width="53.2183" height="9.59985" class="st14"/>
<text x="63.39" y="734.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>true/false/blob?</text> </g>
<g id="shape60-182" v:mID="60" v:groupContext="shape" v:layerMember="0" transform="translate(532.001,-288)">
<title>Message.22.65</title>
<desc>attach VMDK</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="89.9997" cy="731.801" width="52.7" height="17.6036"/>
<path d="M0 731.8 L171.52 731.8 L171.52 731.8" class="st10"/>
<rect v:rectContext="textBkgnd" x="68.5626" y="727.001" width="42.8744" height="9.59985" class="st14"/>
<text x="68.56" y="734.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>attach VMDK</text> </g>
<g id="shape61-189" v:mID="61" v:groupContext="shape" transform="translate(-195.3,641.801) rotate(-90)">
<title>Activation.69</title>
<v:userDefs>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<rect x="0" y="722.801" width="216" height="9.00001" class="st15"/>
</g>
<g id="shape62-191" v:mID="62" v:groupContext="shape" v:layerMember="0" transform="translate(536.501,-198)">
<title>Self Message.24.70</title>
<desc>extract data</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(1):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="-0.000385679" cy="713.801" width="48.99" height="17.6036"/>
<path d="M0 695.8 L18 695.8 L18 731.8 L8.48 731.8 L8.48 731.8" class="st10"/>
<rect v:rectContext="textBkgnd" x="-19.582" y="709.001" width="39.1635" height="9.59985" class="st14"/>
<text x="-19.58" y="716.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>extract data</text> </g>
<g id="shape63-198" v:mID="63" v:groupContext="shape" v:layerMember="0" transform="translate(536.501,-234)">
<title>Self Message.24.71</title>
<desc>mount VMDK</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(1):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="-0.000385679" cy="713.801" width="53.79" height="17.6036"/>
<path d="M0 695.8 L18 695.8 L18 731.8 L8.48 731.8 L8.48 731.8" class="st10"/>
<rect v:rectContext="textBkgnd" x="-21.9806" y="709.001" width="43.9604" height="9.59985" class="st14"/>
<text x="-21.98" y="716.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>mount VMDK</text> </g>
<g id="shape64-205" v:mID="64" v:groupContext="shape" v:layerMember="0" transform="translate(536.501,-162)">
<title>Self Message.24.73</title>
<desc>unmount VMDK</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(1):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="-0.000385679" cy="713.801" width="62.19" height="17.6036"/>
<path d="M0 695.8 L18 695.8 L18 731.8 L8.48 731.8 L8.48 731.8" class="st10"/>
<rect v:rectContext="textBkgnd" x="-26.1837" y="709.001" width="52.3666" height="9.59985" class="st14"/>
<text x="-26.18" y="716.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>unmount VMDK</text> </g>
<g id="shape65-212" v:mID="65" v:groupContext="shape" v:layerMember="0" transform="translate(536.501,-144)">
<title>Message.22.74</title>
<desc>detach VMDK</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="87.7497" cy="731.801" width="54.37" height="17.6036"/>
<path d="M0 731.8 L167.02 731.8 L167.02 731.8" class="st10"/>
<rect v:rectContext="textBkgnd" x="65.4767" y="727.001" width="44.5463" height="9.59985" class="st14"/>
<text x="65.48" y="734.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>detach VMDK</text> </g>
<g id="group66-219" transform="translate(298,-180)" v:mID="66" v:groupContext="group">
<v:userDefs>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<title>Optional fragment.75</title>
<g id="shape67-220" v:mID="67" v:groupContext="shape" transform="translate(0.00118568,0)">
<title>shape75-237</title>
<rect x="0" y="493.301" width="189" height="238.5" class="st1"/>
</g>
<g id="shape68-222" v:mID="68" v:groupContext="shape" transform="translate(0,-203.293)">
<title>Sheet.68</title>
<desc>layer does not exist</desc>
<v:userDefs>
<v:ud v:nameU="msvQuickStyleVariationRestore" v:prompt="" v:val="VT0(2):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="36.51" cy="722.999" width="73.03" height="17.6036"/>
<text x="4" y="725.4" class="st4" v:langID="1033"><v:paragraph/><v:tabList/>layer does not exist</text> </g>
<g id="shape69-224" v:mID="69" v:groupContext="shape" transform="translate(0.00118568,-220.896)">
<title>Sheet.69</title>
<desc>opt</desc>
<v:userDefs>
<v:ud v:nameU="FoldSize" v:prompt="" v:val="VT0(0.1):0"/>
<v:ud v:nameU="BackFillColor" v:prompt="" v:val="VT5(#96afcf)"/>
<v:ud v:nameU="BackLineColor" v:prompt="" v:val="VT5(#1f477d)"/>
<v:ud v:nameU="BackCharColor" v:prompt="" v:val="VT5(#000000)"/>
<v:ud v:nameU="DarkerColor" v:prompt="" v:val="VT5(#000000)"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="31.9996" cy="722.999" width="64.01" height="17.6036"/>
<path d="M0 731.8 L56.8 731.8 L64 724.6 L64 714.2 L0 714.2 L0 731.8 Z" class="st2"/>
<text x="4" y="725.4" class="st3" v:langID="1033"><v:paragraph/><v:tabList/>opt</text> </g>
</g>
<g id="shape70-227" v:mID="70" v:groupContext="shape" transform="translate(-447.3,632.801) rotate(-90)">
<title>Activation.72</title>
<v:userDefs>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<rect x="0" y="722.801" width="450" height="9.00001" class="st15"/>
</g>
<g id="shape71-229" v:mID="71" v:groupContext="shape" v:layerMember="0" transform="translate(284.501,-315)">
<title>Self Message.24</title>
<desc>cache and/or stream</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(1):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="58.4997" cy="713.801" width="77.36" height="17.6036"/>
<path d="M0 695.8 L18 695.8 L18 731.8 L8.48 731.8 L8.48 731.8" class="st10"/>
<rect v:rectContext="textBkgnd" x="24.7325" y="709.001" width="67.5346" height="9.59985" class="st14"/>
<text x="24.73" y="716.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>cache and/or stream</text> </g>
<g id="shape72-236" v:mID="72" v:groupContext="shape" v:layerMember="0" transform="translate(284.501,-369)">
<title>Return Message.26.51</title>
<desc>blob data</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(1):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="51.7497" cy="731.801" width="40.64" height="17.6036"/>
<path d="M103.5 731.8 L0 731.8" class="st16"/>
<rect v:rectContext="textBkgnd" x="36.3417" y="727.001" width="30.8161" height="9.59985" class="st14"/>
<text x="36.34" y="734.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>blob data</text> </g>
<g id="shape73-243" v:mID="73" v:groupContext="shape" v:layerMember="0" transform="translate(284.501,-378)">
<title>Message.22.78</title>
<desc>get fsLayer</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="51.7497" cy="731.801" width="45.22" height="17.6036"/>
<path d="M0 731.8 L95.02 731.8 L95.02 731.8" class="st10"/>
<rect v:rectContext="textBkgnd" x="34.0547" y="727.001" width="35.3903" height="9.59985" class="st14"/>
<text x="34.05" y="734.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>get fsLayer</text> </g>
<g id="shape74-250" v:mID="74" v:groupContext="shape" v:layerMember="0" transform="translate(275.501,-108)">
<title>Return Message.26.79</title>
<desc>handle</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(1):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="126" cy="731.801" width="40.01" height="17.6036"/>
<path d="M252 731.8 L0 731.8" class="st16"/>
<rect v:rectContext="textBkgnd" x="114.871" y="727.001" width="22.2575" height="9.59985" class="st14"/>
<text x="114.87" y="734.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>handle</text> </g>
<g id="shape75-257" v:mID="75" v:groupContext="shape" v:layerMember="0" transform="translate(536.501,-108)">
<title>Self Message.24.80</title>
<desc>store layer metadata</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(1):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="40.4997" cy="713.801" width="77.73" height="17.6036"/>
<path d="M0 695.8 L18 695.8 L18 731.8 L8.48 731.8 L8.48 731.8" class="st10"/>
<rect v:rectContext="textBkgnd" x="6.54886" y="709.001" width="67.9016" height="9.59985" class="st14"/>
<text x="6.55" y="716.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>store layer metadata</text> </g>
<g id="shape76-264" v:mID="76" v:groupContext="shape" v:layerMember="0" transform="translate(172.001,-99.0001)">
<title>Return Message.26.81</title>
<desc>ongoing status update</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(1):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="51.7497" cy="731.801" width="82.41" height="17.6036"/>
<path d="M103.5 731.8 L0 731.8" class="st16"/>
<rect v:rectContext="textBkgnd" x="15.4591" y="727.001" width="72.5812" height="9.59985" class="st14"/>
<text x="15.46" y="734.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>ongoing status update</text> </g>
<g id="shape77-271" v:mID="77" v:groupContext="shape" v:layerMember="0" transform="translate(46.0004,-99.0001)">
<title>Return Message.26.82</title>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(1):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<path d="M121.5 731.8 L0 731.8" class="st16"/>
</g>
<g id="shape78-276" v:mID="78" v:groupContext="shape" v:layerMember="0" transform="translate(172.001,-621.001)">
<title>Message.22.83</title>
<desc>possible audit log mapping for all operations over this conne...</desc>
<v:userDefs>
<v:ud v:nameU="IsReturn" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="IsAsync" v:prompt="" v:val="VT0(0):26"/>
<v:ud v:nameU="IsSelfMessage" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<v:textBlock v:margins="rect(4,4,4,4)"/>
<v:textRect cx="270" cy="731.801" width="222.45" height="17.6036"/>
<path d="M0 731.8 L531.52 731.8 L531.52 731.8" class="st17"/>
<rect v:rectContext="textBkgnd" x="163.691" y="727.001" width="212.619" height="9.59985" class="st14"/>
<text x="163.69" y="734.2" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>possible audit log mapping for all operations over this connection</text> </g>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 41 KiB

BIN
vendor/github.com/vmware/vic/doc/design/images/vic-machine-high-level.png generated vendored Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 21 KiB

BIN
vendor/github.com/vmware/vic/doc/design/images/vic-machine-high-level.vsdx generated vendored Normal file
View File

Binary file not shown.

333
vendor/github.com/vmware/vic/doc/design/networking/README.md generated vendored Normal file
View File

@@ -0,0 +1,333 @@
# Networking in vIC
This document provides a high level overview of how the networking will
work in vIC for MVP. After reading this document you should have a good
idea of how different components in vIC are connected to each other,
and how they use those networks.
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
- [High level overview of networks for MVP](#high-level-overview-of-networks-for-mvp)
- [vSphere Management Network](#vsphere-management-network)
- [Docker Management Endpoint Network](#docker-management-endpoint-network)
- [External Network (ie. route to Internet)](#external-network-ie-route-to-internet)
- [Container Bridge Network(s)](#container-bridge-networks)
- [Docker Networking and the Port Layer](#docker-networking-and-the-port-layer)
- [Use Cases](#use-cases)
- [Container with published port](#container-with-published-port)
- [Description](#description)
- [User Workflow](#user-workflow)
- [Expected Outcome](#expected-outcome)
- [Simple bridge network](#simple-bridge-network)
- [Description](#description-1)
- [User Workflow](#user-workflow-1)
- [Expected Outcome](#expected-outcome-1)
- [Bridged containers with exposed port](#bridged-containers-with-exposed-port)
- [Description](#description-2)
- [User Workflow](#user-workflow-2)
- [Expected Outcome](#expected-outcome-2)
- [Container using external network](#container-using-external-network)
- [Description](#description-3)
- [User Workflow](#user-workflow-3)
- [Expected Outcome](#expected-outcome-3)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
## High level overview of networks for MVP
There are four network types connected to a VCH. At VCH creation there
will be at least one of each network. Additional network configurations
are explained later in this document.
![Networks connected to VCH](pics/vch-net.png)
### vSphere Management Network
The VCH will use the vSphere Management Network to communicate with
vCenter and the hosts as needed. In addition, the tether within the
containers are expected to use the management network to connect with
the VCH (using either serial-over-LAN or vSocket, etc).
![VCH to Management Network](pics/vch-management-net.png)
### Docker Management Endpoint Network
We need to allow customers to isolate the docker endpoint from the
more public external network. This will basically allow for a
"management" network for Docker without requiring the Docker clients
to have access to the vSphere Management network.
![VCH to Docker Management Endpoint Network](pics/vch-docker-net.png)
vIC will support published ports on the default bridged network using
`run -p` or `run -P` however that solution makes the VCH a single
point of failure for the external networking of containers. We are introducing the concept of an external network to remove this single
point and to allow for greater scalability and flexibility in
*published* services offered by containers.
### External Network (ie. route to Internet)
Typically the interface on the Docker Host is shared by both the
Docker endpoint and the containers that are publishing ports. In vIC
we are promoting a more scalable solution where the containers that
want to publish network services should be configured on a *external*
network and not use NAT through the VCH.
For the VCH, an external network is connected to the VCH and used
by the VCH as the default interface for *published* ports of containers
when those containers are using the *default* bridge network.
In addition, once the external network is defined, containers can
be deployed directly on the external interface and do not need
to go through the VCH.
![VCH to External Network](pics/vch-external-net.png)
### Container Bridge Network(s)
The bridge networks allow the containers to talk to each other. In
addition the bridge network is used when the container is
configured to expose a port via the VCH. The user can map ports
on the external network of the VCH to a port on a specific
container. Therefore the VCH is also on all bridge networks.
![VCH Bridge Network](pics/vch-bridge-net.png)
> Note that vSphere Integrated containers does not support the host
networking driver for docker. Since each container is running in a
VM the networking stacks cannot be shared as with host mode networking.
## Docker Networking and the Port Layer
The Port Layer implements network scopes that generally map into the
networks that show during a `docker network ls`. For MVP, networks
that are tied into the infrastructure will be configured within vSphere.
Which of those pre-configured networks are available to a VCH is
determined by the networks that are part of the VCH provisioning, or
added via a VCH reconfigure.
## Use Cases
For our MVP we will start with a few sample use cases that should be
working and tested. Each use case should work independently as well as
concurrently.
### Container with published port
#### Description
Launch a container and expose a port via the VCH using `run -p`.
Test by connecting to the container via the external mapped
port on the external interface of the VCH.
>**NOTE:** This is used for compatibility with the way that most
docker containers publish ports. vIC promotes the use of the external
networks for published ports as a more resilient and scalable solution.
#### User Workflow
$ docker run -p 8080:80 --name test1 my_container my_app
#### Expected Outcome
Port layer will:
* connect *test1* to the default bridge network
* expose port 80 on container *test1*
* setup NAT mapping port 8080 on the external interface of the VCH
to port 8080 on the internal bridge interface of *test1*
User expects port 80 on *test1* to be accessible via the
external network interface on the VCH at port 8080.
### Simple bridge network
#### Description
Create a new non-default bridge network and setup two containers on the
network. Ensure those containers can locate and communicate with
each other.
#### User Workflow
$ docker network create -d bridge my-bridge-network
$ docker network ls
...
NETWORK ID NAME DRIVER
615d565d498c my-bridge-network bridge
...
$ docker run -d --net=my-bridge-network \
--name=server my_server_image server_app
$ docker run -it --name=client --net=my-bridge-network busybox
/ # ping server
PING server (172.18.0.2): 56 data bytes
64 bytes from 172.18.0.2: seq=0 ttl=64 time=0.073 ms
64 bytes from 172.18.0.2: seq=1 ttl=64 time=0.092 ms
64 bytes from 172.18.0.2: seq=2 ttl=64 time=0.088 ms
#### Expected Outcome
Port layer will:
* Allocate(\*) a new bridge network for `network create` command
* Provide IP and name services to the containers created/attached
to that network
User expects server and client can ping each other by name.
(\*) *Note that for MVP the bridge networks may all reside on a single
port group and only be separated by IP space. This will be addressed
post MVP with NSX integration.*
### Bridged containers with exposed port
#### Description
Combine the two examples above. Connect two containers on a
bridge network and setup one of the containers to publish
a port via the VCH.
#### User Workflow
We're going to assume server_app binds to port 5000.
$ docker network create -d bridge my-bridge-network
$ docker network ls
...
NETWORK ID NAME DRIVER
615d565d498c my-bridge-network bridge
...
$ docker run -d -p 5000:5000 --net=my-bridge-network \
--name=server my_server_image server_app
$ docker run -it --name=client --net=my-bridge-network busybox
/ # ping -c 3 server
PING server (172.18.0.2): 56 data bytes
64 bytes from 172.18.0.2: seq=0 ttl=64 time=0.073 ms
64 bytes from 172.18.0.2: seq=1 ttl=64 time=0.092 ms
64 bytes from 172.18.0.2: seq=2 ttl=64 time=0.088 ms
/ # telnet server 5000
GET /
Hello world!Connection closed by foreign host
$ telnet vch_external_interface 5000
Trying 192.168.218.137...
Connected to 192.168.218.137.
Escape character is '^]'.
GET /
Hello world!Connection closed by foreign host.
#### Expected Outcome
Port layer will:
* Allocate(\*) a new bridge network for `network create` command
* Provide IP and name services to the containers created/attached
to that network
* setup NAT mapping port 5000 on the external interface of the VCH
to port 5000 on the *my-bridge-network* interface of *server*
User expects:
* *server* and *client* can ping each other by name.
* can connect to *server* on port 5000 from *client* container
and by connecting to port 5000 on the VCH external interface.
### Container using external network
> **NOTE:** For MVP all the "external" networks will be configured
on the vSphere side and associated with a VCH at VCH creation time
or during a VCH reconfigure. Port Layer/Docker endpoint self service
for external networks is a post-MVP feature.
#### Description
In this use case we assume a couple external networks have been setup
on the vSphere side.
`default-external` is `10.2.0.0/16` and has a gateway @ `10.2.0.1`
`vic-production` is `208.91.3.0/24` and has a gateway @ `208.91.3.1`
The VCH is attached to the *default-external* network at 208.91.3.2.
Therefore `docker network ls` shows the following:
$ docker network ls
NETWORK ID NAME DRIVER
e2113b821ead none null
37470ed9992f default-external bridge
ea96a6b919de vic-production bridge
b7e91524f3e2 bridge bridge
The user has a container providing a web service that they want to
expose outside of the vIC environment.
#### User Workflow
For MVP, the networks have been setup and added to the VCH during
VCH creation or reconfigure.
The output of `docker network inspect default-external` is:
[
{
"Name": "default-external",
"Id": "37470ed9992f6ab922e155d8e902ca03710574d96ffbfde1b3faf541de2a701f",
"Scope": "external",
"Driver": "bridge",
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "10.2.0.0/16",
"Gateway": "10.2.0.1"
}
]
},
"Containers": {},
"Options": {}
}
]
And the output of `docker network inspect vic-production` is:
[
{
"Name": "vic-production",
"Id": "ea96a6b919de4ca2bd627bfdf0683ca04e5a2c3360968d3c6445cb18fab6d210",
"Scope": "external",
"Driver": "bridge",
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "208.91.3.0/24",
"Gateway": "208.91.3.1"
}
]
},
"Containers": {},
"Options": {}
}
]
The user would do the following to setup a server on the
vic-production network.
$ docker run -d --expose=80 --net=vic-production --name server my_webapp
$ docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' server
208.91.3.2
$ telnet 208.91.3.2 80
Trying 208.91.3.2...
Connected to 208.91.3.2.
Escape character is '^]'.
GET /
Hello world!Connection closed by foreign host.
>**NOTE:** The user could also just use -p 80 or -p 80:80 instead of
--expose=80. Trying to map to different ports with -p will result in a
configuration error.
#### Expected Outcome
The Port Layer will add the container *server* to the
*vic-production* network.
The user expects the *server* container to have it's port exposed on
the external network *vic-production*.

BIN
vendor/github.com/vmware/vic/doc/design/networking/pics/vch-bridge-net.png generated vendored Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 18 KiB

BIN
vendor/github.com/vmware/vic/doc/design/networking/pics/vch-docker-net.png generated vendored Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 16 KiB

BIN
vendor/github.com/vmware/vic/doc/design/networking/pics/vch-external-net.png generated vendored Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 21 KiB

BIN
vendor/github.com/vmware/vic/doc/design/networking/pics/vch-management-net.png generated vendored Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 18 KiB

BIN
vendor/github.com/vmware/vic/doc/design/networking/pics/vch-net.png generated vendored Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 20 KiB

159
vendor/github.com/vmware/vic/doc/design/registries/whitelist-registry.md generated vendored Normal file
View File

@@ -0,0 +1,159 @@
# Whitelist Registries
vSphere Integrated Containers 1.2 (VIC) added the ability to whitelist registry access in an installed VCH. When one or more registries are whitelisted for the VCH at install time the VCH goes into 'whitelist mode'. From this point on, the VCH will only allow access to registries in its list of whitelisted registries. In this mode, users will not be able to access any non-whitelisted registries, public or private.
## Specifying Whitelist Registries at Installation
Whitelisted registries can be declared during a VCH installation with *vic-machine* parameter, `--whitelist-registry`. Two other vic-machine parameters affect whitelist registries, `--registry-ca` and `--insecure-registry`. When whitelisted registries are declared during installation, the latter two parameters acts as modifiers to the whitelisted registries.
Registry-ca declares additional certificates to verify access to registry servers secured with TLS. If a registry is declared as a whitelist registry and not an insecure registry (discussed below), the VCH must have access to the server certificate to verify access. The Photon-based VM that VIC uses has a base set of well-known certificates from public CAs. If a whitelist registry uses a certificate that is not in that set of well-known certificates, the certificate must be uploaded to the VCH via vic-machine's `--registry-ca` parameter.
Insecure-registry declares a registry server that can be used without requiring TLS certificate verification. This modifies the whitelist label and takes precedence. For instance, if a registry is declared with `--whitelist-registry` and with `--insecure-registry`, the VCH will assume the registry is an insecure whitelisted registry. If the registry is listed with only `--whitelist-registry`, then the VCH will attempt to verify access using certificates.
If a registry is declared with `--insecure-registry` but not with `--whitelist-registry`, vic-machine will add the insecure registries to the list of whitelist registries *IF* at least one whitelist registry was declared.
A note about certificates. During installation, vic-machine will attempt to verify the registry server is actually a valid registry server. It will also attempt to validate that the certificates declared in `--registry-ca` are valid for the secure whitelisted registries. Vic-machine only performs best effort validation of registry servers. It will not remove the server's access from the VCH if it cannot validate the server.
Acceptable values for whitelist registry values are numbered IP, FQDN, CIDR formatted range, and wildcard domains. If a CIDR format is used, e.g. 192.168.1.1/24, then the VCH will whitelist any IP address within that subnet. Vic-machine will not try to validate CIDR defined ranges. If a wildcard domain is provided, e.g. *.company.com, the VCH will whitelist any IP address or FQDN address that it can validate against the domain provided during installation. A numeric IP address will cause the VCH to perform a reverse DNS lookup to validate against that wild card domain. As with CIDR values, vic-machine will not attempt to validate wildcard domains during installation. Examples are provided below.
The parameter `--whitelist-registry` creates a list of registries. If multiple whitelist registries need to be declared, repeat `--whitelist-registry` multiple times during installation for each registry.
### Example: vch installation with vic-machine
This example installs 2 whitelist registries and 1 insecure registry.
```
./vic-machine-linux create --target=10.2.2.5 --image-store=datastore1 --name=vic-docker --user=root --password=xxxxx --compute-resource="/ha-datacenter/host/office2-sfo2-dhcp121.mycompany.com/Resources" --bridge-network=vic-network --debug=0 --volume-store=datastore1/test:default --tls-cname=*.mycompany.com --whitelist-registry="10.2.40.40:443" --whitelist-registry=10.2.2.1/24 --whitelist-registry=*.mycompany.com --insecure-registry=192.168.100.207 --registry-ca=/home/admin/mycerts/ca.crt
```
### Example: vic-machine's output during installation
Below is a snippet from the vic-machine output for the above command.
```
May 15 2017 16:36:12.453-07:00 WARN Unable to confirm insecure registry 192.168.100.207 is a valid registry at this time.
May 15 2017 16:36:12.505-07:00 INFO Insecure registries = 192.168.100.207
May 15 2017 16:36:12.505-07:00 INFO Whitelist registries = 10.2.40.40:443, 10.2.2.1/24, *.mycompany.com, 192.168.100.207
```
Had the above command also included --debug=1 (or higher), the following would be the output
```
May 15 2017 16:36:12.453-07:00 WARN Unable to confirm insecure registry 192.168.100.207 is a valid registry at this time.
May 15 2017 16:36:12.505-07:00 DEBUG Secure registry 10.2.40.40:443 confirmed.
May 15 2017 16:36:12.505-07:00 DEBUG Skipping registry validation for 10.2.2.1/24
May 15 2017 16:36:12.505-07:00 DEBUG Skipping registry validation for *.eng.vmware.com
May 15 2017 16:36:12.505-07:00 INFO Insecure registries = 192.168.100.207
May 15 2017 16:36:12.505-07:00 INFO Whitelist registries = 10.2.40.40:443, 10.2.2.1/24, *.mycompany.com, 192.168.100.207
```
There are a few things to note from this snippet.
1. The confirmation of the insecure registry was not attempted.
2. The whitelist registry that is secured was confirmed.
3. Both CIDR and wildcard domain declared as whitelist were skipped during validation.
4. The final whitelist registry list contains all registries declared with both --whitelist-registry and --insecure-registry.
5. While not stated yet, the IP address above that contained :443 will not prevent users to use just the server address during docker commands. This will be discussed below.
## Using Docker Commands Against a VCH in 'Whitelist mode'
VIC currently supports Docker commands that are most applicable for production deployment of containers. The commands that are affected by whitelist mode are docker info, docker login, and docker pull. Below are examples of docker commands issued for the VCH installed with the command above. Let's assume vic-machine properly installs a VCH with the above command and it reports the VCH has an FQDN of myvch.mycompany.com.
### Example: docker -H myvch.mycompany.com info
```
devbox:~/$ docker -H myvch.mycompany.com:2376 --tlsverify --tlscacert="vic-docker/ca.pem" --tlscert="vic-docker/cert.pem" --tlskey="vic-docker/key.pem" info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: v1.1.0-rc3-0-c913391
Storage Driver: vSphere Integrated Containers v1.1.0-rc3-0-c913391 Backend Engine
VolumeStores: default
vSphere Integrated Containers v1.1.0-rc3-0-c913391 Backend Engine: RUNNING
VCH CPU limit: 10414 MHz
VCH memory limit: 58.61 GiB
VCH CPU usage: 3103 MHz
VCH memory usage: 56.03 GiB
VMware Product: VMware ESXi
VMware OS: vmnix-x86
VMware OS version: 6.0.0
Insecure Registries: 192.168.100.207
Registry Whitelist Mode: enabled
Whitelisted Registries: 10.2.40.40:443, 10.2.2.1/24, *.mycompany.com, 192.168.100.207
Plugins:
Volume: vsphere
Network: bridge
Swarm: inactive
Operating System: vmnix-x86
OSType: vmnix-x86
Architecture: x86_64
CPUs: 10414
Total Memory: 58.61 GiB
ID: vSphere Integrated Containers
Docker Root Dir:
Debug Mode (client): false
Debug Mode (server): false
Registry: registry-1.docker.io
Experimental: false
Live Restore Enabled: false
```
There are a few things to note in the output of this docker info call.
1. Insecure Registry and whitelist registry lists are shown.
2. There is a message, 'Registry Whitelist Mode: enabled'. If no whitelist registries are declared during installation, this message will not be shown.
3. 'Registry: registry-1.docker.io' is displayed even though that address was not whitelisted. This is the address for docker hub. It does not mean docker hub is accessible (shown in example below). It is simply the default registry that is attempted when attempting to login or pull without a registry address.
### Example: docker -H myvch.mycompany.com login 10.2.40.40
```
devbox:~/$ docker -H myvch.mycompany.com:2376 --tlsverify --tlscacert="vic-docker/ca.pem" --tlscert="vic-docker/cert.pem" --tlskey="vic-docker/key.pem" login 10.2.40.40
Username:
Password:
Login Succeeded
```
In this example, a command was issued to log onto a registry that was declared during installation. Note, :443 was included during installation but left off during docker login. The VCH will accept either form of the address.
### Example: docker -H myvch.mycompany.com login
```
devbox:~/$ docker -H myvch.mycompany.com:2376 --tlsverify --tlscacert="vic-docker/ca.pem" --tlscert="vic-docker/cert.pem" --tlskey="vic-docker/key.pem" login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: user
Password:
Error response from daemon: Access denied to unauthorized registry (registry-1.docker.io) while VCH is in whitelist mode
```
Notice when the registry address is left off, it attempts to access docker hub (which was indicated in the docker info output above), but the VCH denies access.
### Example: docker -H myvch.mycompany.com pull 10.2.40.40/test/busybox
```
devbox:~/$ docker -H myvch.mycompany.com:2376 --tlsverify --tlscacert="vic-docker/ca.pem" --tlscert="vic-docker/cert.pem" --tlskey="vic-docker/key.pem" pull 10.2.40.40/test/busybox
Using default tag: latest
Pulling from test/busybox
c05511d7505a: Pull complete
a3ed95caeb02: Pull complete
Digest: sha256:85f3a6aadbb0f25e148d9cfbcf23fbb206f7e6159ea168c33ac51e76fdff4b8e
Status: Downloaded newer image for test/busybox:latest
```
This succeeds as it should.
### Example: docker pull busybox
```
devbox:~/$ docker -H myvch.mycompany.com:2376 --tlsverify --tlscacert="vic-docker/ca.pem" --tlscert="vic-docker/cert.pem" --tlskey="vic-docker/key.pem" pull busybox
Using default tag: latest
Access denied to unauthorized registry (docker.io) while VCH is in whitelist mode
```
An attempt to pull from docker hub fails with a message that access was denied while the VCH is in whitelist mode.

54
vendor/github.com/vmware/vic/doc/design/serial_communication_comparing_solutions.md generated vendored Normal file
View File

@@ -0,0 +1,54 @@
# Comparison between vSPC and CAF for serial communication
This document provides a comparison between two options that were considered to handle vMotion during an active connection between a user and a containerVM's serial port. The two options investigated are:
- Virtual Serial Port Concentrator
- Common Agent Framework
In the first section, a very brief description of both solutions is provided. In the second section, a comparison between the capabilities of the two solutions is discussed. Finally, a solution is proposed with detailed action items.
# First: Introduction
### Virtual Serial Port Concentrator (vSPC)
vSPC is a telnet server proxy that aggregates and multiplexes serial port communication from multiple VMs to a remote system. vSPC allows vMotion to continue even if there exists an active connection between a remote system and the serial port of a VM.
For vSPC to work, the telnet server proxy must support the following:
1. VMware's telnet exetension commands
2. Handle extension commands for vMotion notification
3. forward connections to a remote system
A more in-depth description of this solution is very well documented in https://www.vmware.com/support/developer/vc-sdk/visdk41pubs/vsp41_usingproxy_virtual_serial_ports.pdf
### Common Agent Framework (CAF)
Multiple VMware's management solutions (e.d Loginsight) require the installation of guest agents. Even though multiple agents have common functionalities, this commonality is not efficiently utilized.
CAF attempts to solve many of these problems by providing a common framework that simplifies standard deployment/upgrade mechanism, common scalable communication layers, common in-guest authorization, common provider frameworks, common data representation(s), etc.
CAF provides a secure, scalable, high-bandwidth message bus that is currently implemented via a message broker proxy that talks to a rabbitmq node (or cluster)
Serial communication from containerVMs to the VIC appliance can leverage the CAF to provide a high-bandwidth and secure communication channel.
# Second: Comparison
Comparison Metric | vSPC | CAF
------------------|------|------
Ease of implementation/integration|relatively easy|more challenging<sup>1</sup>
Communication Bandwidth|relatively small|relatively large
Connection Persistence during vMotion|Supported|Supported
No ESX host agent| Not required | Not required<sup>2</sup>
No Network stack required for containerVMs|None required|None required<sup>3</sup>
<sup>1</sup>CAF will require a lot of across-team communication and a major code refactoring
<sup>2</sup>Even though, AFAIK, there is no ESX host agent requirement, there is an extra overhead associated with refactoring the code to utilize the CAF API. Furthermore, the `rabbitmqproxy` in the ESX host should be configured to be able to communicate with the broker
<sup>3</sup>The Rabbitmq node (or cluster) has to be connected to the management network
# Third: Proposal
Based on the priority criteria put forward in issue #3937, vSPC seems to be the right way to go because of its ease of implememntation/integration with VIC's code base. The only significant difference between using CAF and vSPC is the high-bandwidth communication we are projected to get should we use CAF. Until performance becomes an issue, vSPC seems to be a very good candidate.
Adopting the vSPC solution will require the following:
1. Implementing an extensible telnet server
2. Implementing a vSPC
3. integrating the vSPC with VIC
Step 1 is required because, AFAIK, the current go telnet implementations do not support extending the telnet commands which is a requirement on our end. The output of this step will be a stand-alone library
Step 2 is required because, AFAIK, the only vSPC implementation I have seen is written in python. The output of this step will be a stand-alone binary

42
vendor/github.com/vmware/vic/doc/design/telemetry/README.md generated vendored Normal file
View File

@@ -0,0 +1,42 @@
# Telemetry Support
This is the design proposal for vSphere Integrated Containers Engine's initial integration with Vmware Analytics Cloud (VAC for short). This data provides Vmware with usage metrics. Currently, vSphere Integrated Containers Engine will piggyback on one of the data field that vSphere 6 and 6.5 are already collecting, specifically the OS full name used in the VM. By providing a custom OS name for both the VCH and container VM, Vmware can determine the install base of VIC, the distribution of version installed, and the number of container VM used in a VCH.
## Design
vSphere Integrated Containers Engine will update the name of the guest full name during VM creation time. The vSphere SDK allows providing a custom text for the guest OS name if guest id is either otherGuest or otherGuest64 and alternate guest name has a value. Vmware Analytics Cloud collects customer opted-in metrics (via CEIP) and will use the custom guest name to differentiate between regular VMs and vSphere Integrated Containers Engine's VMs.
For these values to get updated, vic-machine will populate the two fields described above in the VM config spec during creation of the VCH. During container create, the portlayer will get the default linux VM spec from vic/lib/guest/linux.go with those two fields populated. Future OS or variants of container VM's linux OS can add default VM spec go files (e.g. vic/lib/guest/redhat.go). The following is the proposed VCH's and container VM's OS names:
| VM Type | OS Full Name |
| --- | --- |
| VCH | Photon - VCH v.v.v, b, c |
| container VM | Photon - Container v.v.v, b, c |
| container VM | Redhat - Container v.v.v, b, c |
| container VM | Windows - Container v.v.v, b, c |
In the above, v.v.v represents the actual version of vSphere Integrated Containers Engine, b represents build number, and c is the first 7 digit of the git commit sha for the source code. Note, VIC currently supports a custom Photon OS kernel as the OS for both the VCH and container VM. The proposed naming scheme above will allow the engine to accommodate other OSes as well as other Linux variants.
## Initial Metrics
Here are all the metrics that will be collected specifically for VIC Product (includes VIC Engine, Harbor, and Admiral):
| Metric | Available with Guest OS Name? | Note |
| --- | --- | --- |
| 1. Number of ESX hosts in the cluster | Y | |
| 2. Version number of ESX hosts | Y | |
| 3. Version number of VC hosts | Y | |
| 4. Presence of Harbor or Admiral with version number | N | Harbor and Admiral projects will handle this |
| 5. Total number of VMs | Y | |
| 6. Total number of VCHs | Y | |
| 7. Total number of container VMs | Y | |
| 8. Number of VCHs per host | Y | |
| 9. Number of container VMs per VCH | Y | |
| 10. Average lifetime of the Container VMS | Y | Will require a join of the VM and VC events tables |
## Testing and Acceptance Criteria
For metrics 1-9 (minus #4):
1. Robot scripts will be written to use govc to query the VCH's and a created container's guest name.
2. A sanity check will also be performed by checking whether the VCH and container VM can be differentiated on the Vmware Analytics Cloud query page. This will be manual test only as the query cannot be performed outside of Vmware's firewall.

56
vendor/github.com/vmware/vic/doc/design/test/kubernetes.md generated vendored Normal file
View File

@@ -0,0 +1,56 @@
# Kubernetes initial testing notes
## Required HW setup:
- Ubuntu 16.04
- 4 CPU
- 16GB memory
- 80GB disk
- (might need 8CPU/32GB of memory, as the above recommended was quite slow still)
## Initial install:
- `sudo apt-get update`
- `sudo apt-add-repository ppa:juju/stable`
- `sudo apt-add-repository ppa:conjure-up/next`
- `sudo apt update`
- `sudo apt install conjure-up`
## Configure container hypervisor:
- `newgrp lxd`
- `sudo lxd init`
Walkthrough the config (just hit enter) - select NO when asked to setup IPv6
## Point juju at the new hypervisor:
- `juju bootstrap localhost lxd-test`
## Start the k8s cluster:
- `conjure-up canonical-kubernetes`
Just hit enter a few times until you get to the summary and hit Q
- `watch -c juju status --color`
Wait for quite a while until the cluster is completely up/active/idle. Can take upwards of an hour!
## Finalize setup:
- `mkdir -p ~/.kube`
- `juju scp kubernetes-master/0:config ~/.kube/config`
- `juju scp kubernetes-master/0:kubectl ~/bin/kubectl`
## Verify it is working and the cluster is up:
- `kubectl cluster-info`
## Example commands:
- `kubectl run -i -t busybox --image=busybox --restart=Never`
- `kubectl run nginx --image=nginx`
### Show the running pods:
- `kubectl get pods`
### To scale up the cluster:
- `juju add-unit kubernetes-worker`
### To show the controller:
- `juju switch`
### To destroy the cluster:
- `juju destroy-controller lxd-test --destroy-all-models`

47
vendor/github.com/vmware/vic/doc/design/test/nsx.md generated vendored Normal file
View File

@@ -0,0 +1,47 @@
## NSX Initial testing notes
##Required HW setup:
- vSphere 6.0/6.5 Cluster setup (6.5 wan't supported for a while, but the latest releases supports it)
- An ESX host in the cluster with a minimum of 4 CPU to host the NSX Manager Appliance
##Initial install:
###Deploy a Nimbus Cluster using 5-2-Cluster test.
###Add a beefy ESXi to the cluster to host the NSX appliance.
- `nimbus-esxdeploy --disk=40000000 --nics=2 --memory=90000 --cpus=4 nsx-esx 3620759`
###Update host password for the ESXi:
- `export GOVC_URL=root:@10.x.x.x`
- `govc host.account.update -id root -password xxxxxx`
###Add host to the VC Cluster:
- `export GOVC_URL="Administrator@vSphere.local":password@10.x.x.x`
- `govc cluster.add -hostname=10.x.x.x -username=root -dc=ha-datacenter -password=xxxx -noverify=true`
###Install the NSX manager using OVFTool.
- `ovftool nsx-manager-1.1.0.0.0.4788147.ova nsx-manager-1.1.0.0.0.4788147.ovf`
- `ovftool --datastore=${datastore} --name=${name} --net:"Network 1"="${network}" --diskMode=thin --powerOn --X:waitForIp --X:injectOvfEnv --X:enableHiddenProperties --prop:vami.domain.NSX=mgmt.local --prop:vami.searchpath.NSX=mgmt.local --prop:vami.DNS.NSX=8.8.8.8 --prop:vm.vmname=NSX nsx-manager-1.1.0.0.0.4788147.ovf 'vi://${user}:${password}@${host}`
###Add ESX nodes into the NSX Manager using the NSX REST API.
- `FABRIC->Nodes using ESXi credentials: root/password`
###Create the Transport Zone using the NSX REST api
###Create a logical Switch with the VLAN based Transport Zone
###Add the Logical Switch as a Transport node to the ESXi host
###Check if the switch is visible from running govc command (Upgrade to the latest govc 0.12.0 to get this working)
- `govc ls network`
###Install VIC Appliance and Run Regression tests

64
vendor/github.com/vmware/vic/doc/design/tether.md generated vendored Normal file
View File

@@ -0,0 +1,64 @@
# Tether
The tether provides two distinct sets of function:
1. the channel necessary to present the process input/output as if it were bound directly to the users local console.
2. configuration of the operating system - where regular docker can manipulate elements such as network interfaces directly, or use bind mounting of files to inject data and configuration into a container, this approach is not available when the container is running as an isolated VM; the tether has to take on all responsibility for correct configuration of the underlying OS before launching the container process.
## Operating System configuration
### Hostname
### Name resolution
### Filesystem mounts
### Network interfaces
## Management behaviours
This is a somewhat arbitrary divide, but these are essentially ongoing concerns rather than one-offs on initial start of the containerVM
### Hotplug
It's not yet determined whether the containerVM bootstrap images will include udevd, systemd, or if tether will take on that responsibility directly; this could be use of libudev or direct handling of netlink messages given the very limited scope of hotplug devices we need support for (CPU, memory, scsi disks, NICs).
### Signal handling
### Process reaping
Tether runs as pid1 when in a containerVM and has to discharge the associated responsibilities. This primarily means reaping orphaned children so that the process list does not get cluttered with zombies. On a more prosaic note, it's been observed that a connection to a bash container will not exit fully after typing `exit` if reaping isn't performed.
### Secrets
### Forking
This relates specifically to VMfork, aka _Instant Clone_ - the ability to freeze a VM and spin off arbitrary numbers of children that inherit that parent VMs memory state and configuration. This requires cooperation between ESX, GuestOS, and the application processes, to handle changes to configuration such as time, MAC and IP addresses, ARP caches, open network connections, etc.
More directly, VMfork requires that the fork be triggered from within the GuestOS as a means of ensuring it is in a suitable state, meaning the tether has to handle triggering of forking and recovery post-fork
## External communication
The vast bulk of container operations can be performed without a synchronous connection to containerVM, however `attach` is a core element of the docker command set and the one that is probably most heavily used outside of production deployment. This requires that we be able to present the user with a console for the container process.
The initial communication will be via plain network serial port, configured in client mode:
Overall flow:
1. ContainerVM powered on
2. ContainerVM configured so com1 is a client network serial port targeting the appliance VM
2. ESX initiates a serial-over-TCP connection to the applianceVM - this link is treated as a reliable bytestream
3. ApplianceVM accepts the connection
4. ApplianceVM acts as an SSH client over the new socket - authentication is negligible for now
5. ApplianceVM uses a custom global request type to retrieve list of containers running in containerVM (exec presents as a separate container); containerVM replies with list
6. ApplianceVM opens a channel, requesting a specific container; containerVM acknowledges and copies I/O between process and channel
From the Personality to the Portlayer:
* request for container X from Personality to Interaction component - X is a container handle
* Interaction configures containerVM network serial port to point at it's IP and connects the serial port
* Interaction blocks waiting for SSH session with X, record request in connection map
* Interaction starts stream copying from HTTP websocket to SSH channel
Incoming TCP connection to Interaction component:
* request set of IDs on the executor - (5) in overall flow
* create entries for each of the IDs in connection map
* if there is a recorded request for channel to a container X
* establish attach channel
* notify waiters for X

193
vendor/github.com/vmware/vic/doc/design/vic-machine/configure.md generated vendored Normal file
View File

@@ -0,0 +1,193 @@
# VCH Configure (P0)
This document defines the VCH configure command line, and the related changes in vic-machine, to make VCH configure process more reasonable.
Here are the key points considerred in this Definition:
- Batch Operation
We'd like to change VCH configurations more than one item at one time to make the command easy to use
- Reuse all options used in vic-machine create
In vic-machine create, we already have 57 options to define VCH configurations, though not all of them can be updated, many of them should be able to be changed, and this number will grow over time, so we definitely do not want to introduce 3X that number of options to configure (configure means add/modify/remove)
## Options
First option is to configure all in one command
### Implicit Add/Modify and Explicit Delete
All existing vic-machine create options are supported in vic-machine configure command. The meaning of those options are as following:
- anything not specified means no change
- any option specified in configure command means replacement.
- if the old value is not specified during creation, this new value will be added
- if there is old value specified, no matter it's a list value or not, we'll replace the old value with the new one specified in configure command.
here are the possibilities to change list option ```--registry-ca```, if the create command option is ```--registry-ca ca1 --registry-ca ca2```
* ```--registry-ca ca2 --registry-ca ca3```, means new configuration will be [ca2, ca3], ca1 is deleted, and ca3 is added
* ```--registry-ca ca2```, means new configuration will be [ca2], ca1 is deleted
* ```--registry-ca ""```, means ca1 and ca2 are all deleted // Here the empty string value is required, and the representation in different os platform is different.
* nothing specified for registry-ca, then existing [ca1, ca2] is not changed
The benefit of this option is it can update all things in one command, and no new option introduced. User can use exactly same command option in both vic-machine create and vic-machine configure.
### Separate Configure to Sub Commands
The idea of this option is to update a few configuration each time, which is one delta update option. For example, to update volume-store, we can use following command
```
vic-machine configure volume-store --rm ds://datastore/volume:default
vic-machine configure volume-store --add nfs://host:port/container:nfs
```
The command format is like ```vic-machine configure <configure object> --<configure keyword> --<configure option> <configuration value>```
Pros:
- The whole configuration options are split to many sub commands. Then in each sub command, only a small number of options are available, and that does not increase over time. Only the sub command number will increase.
- Consistent with existing update firewall command format (though the command name might be different)
Cons:
- We'll need to reinvent option names
For example, to update client network, user need to specify all options if they want to use static ip. Then in one update command, we'll need all those options. Here is the command looks like ```vic-machine configure bridge-network --port-group value --gateway value --ip value```
The existing option name in vic-machine create is ```--client-network value --client-network-gateway value --client-network-ip value```, we'll need to remove the prefix cause that is already shown as sub-command name.
Decision: We'll go with the first option
## Show Existing VCH configuration
Right now, there is no user friendly way to view all configurations of existing VCH. To support update operation, we need to make it easier, otherwise, update only makes things weird.
vic-machine inspect command is available now, just it shows only the execution endpoint of VCH, instead of the configuration. We can extend that command, to support configuration inspection as well.
- vic-machine inspect <--configuration, --conf>
show VCH configuration only
- vic-machine inspect <--execution, -e>
show VCH execution endpoint (default behavior)
### Configuration Inspection
For configuration inspection, two output format should be supported
- yml format
This should print out all VCH configuration with predefined categories, e.g. general information, storage, network, etc. in user friendly format. Secret information will be hidden or skipped.
Following is sample format:
```
General:
Target: "vc.domain.com"
Thumbprint: "98:97:D1:7B:DF:..."
OpsUser: "Operator"
Name: "VCH-1"
Proxies:
- "https://host:port"
- "http://host:port"
Compute:
Path: " myCluster/Resources/myRP"
CpuLimit: 0 MHz
MemoryLimit: 0 MHZ
Storage:
ImageStore: "datastore"
VolumeStore:
- "ds://datastore/volumes/default": "default"
- "nfs://userid@host:port/volumes/": "nfs"
Networks:
BridgeNetwork:
PortGroup: "bridge"
IpRange: "172.16.0.0/12"
ClientNetwork:
PortGroup: "client"
Gateway: "10.1.0.0/16,10.2.0.0/16:10.0.0.1"
...
DNSServer:
- "8.8.8.8"
- "8.8.4.4"
Certificate:
...
Registries:
InsecureRegistries:
- "..."
RegistryCA:
- "ca file"
Endpoint:
Cpu: 1
Memory: 2048 MHZ
Debug: 3
```
- Command Option Format
The yml format output is more readable for VIC admin, but if users want to recreate their vic-machine create or configure command, it's hard to do based on that format, because there is no one on one mapping between the configuration and command options.
Here we'd support another option as "vic-machine inspect --conf", this will print out the vic-machine create command options minus secret information.
The sample output:
```
--target vc.domain.com
--name VCH-1
--compute-resource /vcqaDC/host/cls/Resources
--ops-user operator
--image-store datastore1
--volume-store ds://datastore/volumes/default:default
--volume-store nfs://userid@host:port/volumes/:nfs
--bridge-network bridge
--bridge-network-range 172.16.0.0/12
...
```
Based on this output, it's easy to modify existing configuration to configure command options
Note: Special character might be escaped based on current OS platform. But even with this, the command options cannot be copied to other platform.
- Subcommand Inspect (corresponding to configure subcommand option)
The configure subcommand option might change command option based on each different sub command requirement, to get current configuration, we'll need to have command inspect for each subcommand. So the option is to extend vic-machine inspect command to include sub command as well.
For example, ```vic-machine inspect bridge-network``` will print out current configuration in the command format, to make sure user can easily generate their configure command based on that.
Note: this is not needed atm, because we're not doing sub configure command
## Update In Progress Status
Same to vic-machine upgrade, concurrent configure requests for same VCH is not allowed. vic-machine should be able to detect if one VCH is running configure/upgrade. If true, return useful error message. And also should not leave that configure flag in there after configure is stopped, or even interrupted.
Issue https://github.com/vmware/vic/issues/4069 is tracking this problem.
## Backward Compatibility
It's possible that after vic engine released new version, user is trying to configure older version's VCH with newer vic-machine.
There are few options:
- Run configure using same version's vic-machine (this option should technically always work)
For any customer that does not want to change vic-machine version, they need to run configure using same version's vic-machine.
- Combine upgrade process into configure
Upgrade is actually one special kind of configure, technically it's easy to have upgrade run together with configure. User might want to configure VCH, and also want to upgrade it after or before configure. In that case, a combined configure and upgrade command can provide a better user experience.
Here is the combined command ```vic-machine configure --upgrade <all configure and upgrade options supported here>```
- Support backward compatibility in vic-machine
This will need to add backward compatible code in vic-machine. Even initially there is only few changes, that will expand massively over time, which is not consistent with our data migration design, so this is not our option.
# VCH configure Rollback (P0.5)
Similar to VCH upgrade workflow, before configure VCH, snapshot should be took to keep existing status. If the configure failed eventually, VCH should be reverted to old status automatically.
The Configure Rollback is talking about after a successful configure, vic-machine is still able to rollback VCH's old configuration. The command is like vic-machine configure --rollback
This feature is nice to have, but as vic-machine upgrade already have this capability, mark it as P0.5.
- Set Env
`vic-machine inspect` should set user env like what `vic-machine create` did, to make sure user can run docker command correctly.
# Manifest File
vic-machine create already has more than 50 options, and the number will increase over time. With this configure feature added, most of those options need to be updated as well. To improve user experience, in my opinion, we need Manifest file to simplify vic-machine command generation.
[vic-machine design](vic-machine.md#vch-manifest) mentioned manifest idea, here we'll start from a simple implementation. The manifest file format will be same to the yml file format, mentioned in [configuration inspection](#configuration-inspection)
- vic-machine inspect --manifest filename will dump all configurations in yml file format to the manifest file.
Here is how to manage VCH using manifest files.
- vic-machine create --manifest will load the manifest file and create VCH based on it.
- vic-machine configure --manifest will load the manifest file and update VCH configuration based on it.
In addition, all the command options can be used together with manifest file, and the command option will override the value in manifest file
TODO: Manifest management command line is not well defined yet. Here is one possible solution for it.
- vic-machine update manifest
All command options in `vic-machine create` can be used for this command.
This manifest file approach will simplify vic-machine create/configure workflow, and leverage user readable configuration format. But it will be an additional work for existing vic-machine create.

508
vendor/github.com/vmware/vic/doc/design/vic-machine/service.md generated vendored Normal file
View File

@@ -0,0 +1,508 @@
1. [Overview](#overview)
2. [Design goals and project scope](#design-goals-and-project-scope)
3. [Implementation Decisions](#implementation-decisions)
1. [Technology Choices](#technology-choices)
2. [Delivery](#delivery)
3. [Cardinality](#cardinality)
4. [Service Upgrade](#service-upgrade)
5. [Compatibility](#compatibility)
6. [Authentication](#authentication)
7. [Certificate Management](#certificate-management)
1. [Host Certificates](#host-certificates)
2. [Client Certificates](#client-certificates)
8. [Log Management](#log-management)
1. [Operation logs](#operation-logs)
2. [Server logs](#server-logs)
9. [Cross-Origin Requests &amp; Cross-Site Request Forgery](#cross-origin-requests--cross-site-request-forgery)
10. [ISO Management](#iso-management)
11. [Communicating modifiability](#communicating-modifiability)
12. [Use of a query parameter for compute-resource](#use-of-a-query-parameter-for-compute-resource)
13. [Use of a request body for delete](#use-of-a-request-body-for-delete)
4. [The REST API](#the-rest-api)
1. [API Versioning](#api-versioning)
2. [Headers](#headers)
3. [Resources](#resources)
4. [Query Parameters](#query-parameters)
5. [Operations](#operations)
1. [Get version information](#get-version-information)
2. [List VCHs](#list-vchs)
3. [Create a VCH](#create-a-vch)
4. [Inspect a VCH](#inspect-a-vch)
5. [Access the creation and reconfiguration log for a VCH](#access-the-creation-and-reconfiguration-log-for-a-vch)
6. [Access the host certificate for a VCH](#access-the-host-certificate-for-a-vch)
7. [Reconfigure a VCH](#reconfigure-a-vch)
8. [Delete a VCH](#delete-a-vch)
9. [Upgrade a VCH](#upgrade-a-vch)
10. [Debug a VCH](#debug-a-vch)
11. [View firewall rule settings](#view-firewall-rule-settings)
12. [Update firewall rules](#update-firewall-rules)
5. [The WebSockets API](#the-websockets-api)
6. [Proposed changes to existing functionality](#proposed-changes-to-existing-functionality)
1. [Support for authenticating via SAML or with a session key in vic-machine](#support-for-authenticating-via-saml-or-with-a-session-key-in-vic-machine)
2. [VCH creation spawns a custom task](#vch-creation-spawns-a-custom-task)
3. [Refactoring of vic-machine](#refactoring-of-vic-machine)
4. [Deprecation of demo VCH installer](#deprecation-of-demo-vch-installer)
5. [Client certificate management using Admiral](#client-certificate-management-using-admiral)
6. [Reduce friction around use of operations credentials](#reduce-friction-around-use-of-operations-credentials)
7. [Associating contact information with a VCH](#associating-contact-information-with-a-vch)
7. [Testing](#testing)
8. [See Also](#see-also)
## Overview
The `vic-machine-{darwin,linux,windows}` command line utilities are used to perform Virtual Container Host (VCH) lifecycle operations.
The `vic-machine-server` exposes similar functionality via a REST API to allow for a wider range of interaction models, such as the integration of VCH management into web interfaces (including the vSphere H5 client).
## Design goals and project scope
This API will become a public, documented API at some point in the future (and, to some extent, all APIs are public), but functionality will be delivered incrementally and the first versions will be versioned in a way that communicates breaking changes are expected and backwards compatibility may not be maintained.
The initial priority is for this API to implement sufficient functionality to enable development of a vSphere plugin, as described here: https://vmware.invisionapp.com/share/GDC9QEDAZ. This may require functionality which is _not_ currently implemented by the `vic-machine` command line utilities.
The API may not initially _implement_ all functionality currently implemented by the `vic-machine` command line utilities. However, to avoid painting ourselves into a corner, it is desirable to consider all functionality in the design. Further, the _design_ should consider plausible future work to ensure that eventual extension of the API is possible.
Because this API would likely be used in conjunction with (or by developers familiar with) other VMware APIs, it is desirable to provide a similar "look & feel" where possible.
Because this API may be used in conjunction with (or by developers familiar with) other container-related APIs, providing a similar "look & feel" to those APIs is also desirable.
This document will attempt to adhere to VMware standards and guidance around API design.
## Implementation Decisions
### Technology Choices
The API will be defined using OpenAPI (Swagger), today's de facto standard for REST APIs.
The service will be built in Go, using `go-swagger`. This allows any developer on the team to contribute, maintain, and modify the service with a minimal learning curve.
WebSockets will be used to provide streaming information about the status on long-running operations so that sophisticated clients do not need to poll for updates. See "[The WebSockets API](#the-websockets-api)" for more information.
Unit tests will be defined using the Testify libraries.
End-to-end tests will be defined using the Robot Framework.
### Delivery
The new service is implemented as an additional flavor of `vic-machine`: `vic-machine-server`. This standalone linux binary (a swagger server) is invoked with port and certificate information to serve the REST API on one or more interfaces.
This service is packaged as a container and included in the VIC OVA, following existing best practices.
The service does not require a access to a persistent data directory, but it useful to provide one as a destination for log files.
A configuration file (stored on the OVA's data directory) may eventually be used to provide the service with high-level configuration information, such as a syslog server and perhaps the admiral CA certificate (see "[Client certificate management using Admiral](#client-certificate-management-using-admiral)".
### Cardinality
It is not assumed that a single instance of `vic-machine-server` will run in a given environment or which manage a given vSphere resource. It is also expected that one or more instances of `vic-machine-server` and the `vic-machine` CLI will be used side-by-side. This means that in-process locking will not be sufficient to protect resources from concurrent access or modification.
In the future, specific use cases may be identified for multiple instances of `vic-machine-server` around availability, scalability, or isolation.
### Service Upgrade
The VIC OVA is upgraded in a side-by-side fashion:
1. The new OVA is deployed.
2. The old OVA is shutdown.
3. The data disk is moved from the old to the new.
4. A data migration script is run.
* In the case of the upgrading to the first version of the OVA that includes the `vic-machine` server, sane default values could be assumed (e.g., by copying syslog settings from another service).
5. The new OVA is started.
This does involve downtime of the service (but not of the VCHs or containers). Clients (including the H5 plugin) should handle this appropriately.
### Compatibility
We expect the compatibility between `vic-machine-server` and VCHs to be similar to `vic-machine-{darwin,linux,windows}`:
* Creation will only be supported for VCH of the same version
* Reconfigure will only be supported for VCH of the same version
* Upgrade will only be supported for VCH of the same or lesser versions
* Rollback will only be supported to the same VCH version
* Deletion will be supported for VCH of the same version, and will be best-effort for older versions
We expect the vSphere H5 client plugin to support a single version of `vic-machine-server` and be upgraded in lock-step.
- [ ] Do we need the plugin to provide at least basic support for the "N+1" version of `vic-machine-server` as well so that it remains functional between when the OVA is upgraded and when the plugin is upgraded?
### Authentication
As with the `vic-machine` CLI, vSphere credentials must be supplied each time an operation is invoked using `vic-machine-server`. The service itself will not store or manage credentials. This means that **all** operations must either be explicitly tied to a vSphere operation or unauthenticated. This is similar to the model for the `vic-machine` CLI, where all operations require vSphere credentials except for those to display help and version information.
See "[Headers](#headers)" for information about how credentials will be passed, "[Certificate Management](#certificate-management)" for a discussion of how this design influences the design for certificate management, and "[Log Management](#log-management)" for a discussion of how this design influences the design for log persistence and access.
### Certificate Management
When invoking the `vic-machine` CLI, access to PKI files is managed out-of-band. PKI files can be placed on the filesystem, and paths passed to the CLI. Generated PKI files are placed on the filesystem for subsequent access, and the OS handles access control for those files.
With the REST API, these workflows need to be handled explicitly: the REST API must allow PKI files to be supplied as a part of requests and must allow for retrieval of generated PKI files, with appropriate access controls.
#### Host Certificates
Host certificates are persisted in the VCH's guest info and are available via the vSphere API today. `vic-machine-server` allows users to access them via its API, regardless of how the VCH was created.
#### Client Certificates
Due to the complexity of ensuring the secrecy of client certificate private keys, the API will not support generation of client certificates. (If this becomes a requirement, generated certificates could be protected with a user-supplied passphrase and stored in the same manner as log files.)
For an alternative approach that allows for creation of VCHs without requiring users to specify client certificates, see "[Client certificate management using Admiral](#client-certificate-management-using-admiral)".
### Log Management
#### Operation logs
When invoking the `vic-machine` CLI, real-time information is provided to stdout and log files can be persisted on the filesystem. The REST API needs to provide equivalent functionality.
Logs will be streamed to the VCH's datastore folder as `vic-machine-server` executes. Access to logs is then restricted to those vSphere users who can read those files from the datastore.
#### Server logs
Logs for the server itself are written to a configurable directory. These logs include server lifecycle information as well as information about each request. Operation IDs are used to associate a request with the log messages related to the handling of that request.
Logs are not rotated by the server. When deployed as a part of the OVA, `logrotate` is used to provide this functionality.
### Cross-Origin Requests & Cross-Site Request Forgery
Because the vSphere H5 client plugin and the `vic-machine-server` will be served from different hosts, the H5 client plugin will be making cross-origin requests. As these would normally be prevented by the browser's same-origin policy, the service must support responding to all requests with an `Access-Control-Allow-Origin` header with an appropriate value, and respond appropriately to `Options` requests for all resources.
The same-origin policy is intended to prevent cross-origin requests because the browser might inadvertently provide credentials on behalf of the user. Given that each individual request to the API must include credentials, and that we won't be using cookies or persisting those credentials in any other way, the same-origin policy does not provide protection in our case.
To avoid additional configuration complexity, we will use a combination of `Access-Control-Allow-Origin: *`, ` Access-Control-Allow-Credentials: false` (the default value), and `Access-Control-Allow-Headers: Authorization, X-VMWARE-TICKET` to express that cross-origin requests are allowed from any origin, and that an `Authorization` or `X-VMWARE-TICKET` header may be included, but that the user-agent should not include cookies or HTTP authentication information based on the user agent's previous interactions with the API. This `Access-Control-Allow-Credentials` restriction will not affect the H5 client plugin, as it will be _explicitly_ including credentials by setting an `Authorization` or `X-VMWARE-TICKET` header, not relying on `XMLHttpRequest.withCredentials`.
### ISO Management
Several VCH operations (create, configure, and upgrade) take a pair of ISOs as input. These ISOs are used for booting the VCH appliance and container VMs respectively.
The current implementation involves maintaining the master ISOs on the OVA and duplicating these ISOs for each VCH that is created. We may wish to change this behavior in the future to make it easier for customers to build their own bootstrap ISO, or to transition to something like direct boot.
Even considering only the current model, uploading these ISOs as a part of service API calls would introduce complexity and inefficiency.
For simplicity and flexibility, we will provide an API that lists "flavors" of ISOs which are known to the OVA and which may be used for API operations. By default, a single "flavor" will be included with the OVA: the stock appliance and boostrap ISOs shipped with that release. We will provide instructions for users who wish to add additional "flavors" of the appliance ISO (e.g., in order to use a RHEL kernel or systemd), which might involve putting the ISOs in a particular directory on the OVA or adding paths to the ISOs in some sort of manifest file. For now, custom bootstrap ISOs will not be supported (but it is easy to imagine adding support following this same pattern).
In the future, it may be desirable to improve our handling of ISOs to reduce duplication (e.g., by storing them on the datastore instead of within the OVA). This is, however, orthogonal to the introduction of an API.
- [ ] Come up with a better term than "flavor".
### Communicating modifiability
VCH properties may or may not be modifiable for a variety of reasons. Some properties, such as the id, may never be modifiable. Other properties may not be modifiable without a power-cycle of the VCH. Yet others may be dependent on the state of other resource, such as whether containers are using an attached network.
In all cases, the server would enforce unmodifiablility when performing an operation, but to provide a good experience for direct and indirect users of the API, it would be helpful to communicate which properties of a given VCH are not currently modifiable and why. (That is, it's better to grey out a field in the interface and provide a help tooltip with an explanation than to allow users to attempt to an operation which will inevitably fail.)
A variety of approaches exist for this:
* Capturing the general mutability rules in a formal language so that clients can evaluate those rules against the current state of the resource. This powerful pattern would allow interfaces built on the API to clearly communicate why a field cannot be modified, and offer options for remediation, without attempting the modification. However, this would require substantial effort for both the client and server.
* Evaluating mutability rules on the server to determining point-in-time mutability, and including that information as a part of GET requests. This reduces the burden on the client, while still allowing for a good user experience. However, this significantly bloats the API and leads to GET/PUT asymmetry.
* Documenting the mutability rules in a human-readable way. This would allow implementors of interfaces built on the API to read the rules and select some or all of them to evaluate in their client code. This can be cumbersome when clients wish to support multiple versions of the API, but seems to be the approach used by most common REST APIs today.
With each of these approaches, it is also possible for the client or sever to express a level of intrusiveness. For example, a server might communicate "the VCH must be restarted to modify this" instead of simply "this cannot be modified in the current state." Similarly, a client might communicate "I want to make this change, even if it requires restarting the VCH" or "I want to make this change, even if it requires powering off all containers."
More complex logic can be introduced in the future, but in the interest of simplicity of the API and ease of implementation, it seems desirable to start simply with documenting the mutability rules in a human-readable way. As a next step, it may be useful to allow clients to communicate "I want to make this change, even if it requires restarting the VCH."
### Use of a query parameter for compute-resource
A datacenter represents an aggregation of resources within which a VCH may exist (spanning compute, storage, and networking), and is therefore included as a hierarchical path element. However, the compute-resource (i.e., cluster or resource pool) is, conceptually, a one-dimensional filter. Analogous filters for storage-resource or networking-resource could exist to identify VCHs using a particular datastore or network respectively. Expressing such filters as query parameters avoids the incorrect connotation of hierarchy that a path element would imply, and ensures composability.
(In the future, one could even imagine a more flexible filter mechanism that allowed for filtering on _any_ property of a VCH. Under such a model, compute-resource could be deprecated in favor of a more verbose expression, or viewed as a shorthand.)
### Use of a request body for delete
While the semantics for including a body with a delete request are not generally defined ([RFC 7231](https://tools.ietf.org/html/rfc7231#section-4.3.5)), there does not seem to be a better way to influence the behavior of the operation. (Use of query parameters or path segments would be contrary to the [stylistic pattern](https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm) of uniform interface.)
An alternative approach would be to model deletion as an action, but that may not be as intuitive for some consumers of the API.
## The REST API
### API Versioning
- [ ] To do
### Headers
Where possible, the standard `Authorization` header will be used for authentication.
Currently, only one scheme is supported:
* "basic", which will allow direct authentication with username and password
Eventually, another may be added:
* "Bearer", which will allow authentication via SAML, including from the H5 client plugin.
Additionally, a session ticket may be specified using the `X-VMWARE-TICKET` header.
### Resources
The base resource for all API operations will be `/container`.
Additional resources will exist to represent:
1. a vSphere target (ESX, vCenter, or Datacenter)
* The root resource followed by `/target/{target-network-address}`
* The target-network-address parameter must be a valid network address (FQDN or IP address) of a vSphere Server (ESX or vCenter)
* The root resource followed by `/target/{target-network-address}/datacenter/{datacenter-id}`
* The target-network-address parameter must be a valid network address (FQDN or IP address) of a vCenter Server
* The datacenter-id parameter must be an identifier for a resource of type Datacenter located within that vCenter Server
2. the collection of VCHs within (1)
* Any resource from (1) followed by `/vch`
3. a VCH within (2)
* Any resource from (2) followed by `/{vch-id}`
4. a host certificate within (3)
* Any resource from (3) followed by `/certificate`
5. log data within (3)
* Any resource from (3) followed by `/log`
Note: Given the use cases for this API, the exclusive use of identifiers (vs. names) seems acceptable. If necessary, lookup-by-name can be implemented using the `filter.names` query pattern from in the vSphere REST API.
### Query Parameters
For all requests to all resources except the root resource: An optional "thumbprint" parameter will be supported to allow an API client to indicate the expected thumbprint of the target vSphere system. This parameter need not be supplied if the target system has a certificate signed by a trusted certificate authority. There will be no equivalent to the "force" command-line argument.
- [ ] Figure out how certificate authority management will work. (Presumably there's something this can piggyback on.)
For many requests, as detailed below: An optional "compute-resource" parameter will be supported to scope a request to a particular compute resource within a vSphere target. This is equivalent to the "compute-resource" command-line argument except that it takes an identifier instead of a name. (For context, see [Use of a query parameter for compute-resource](#use-of-a-query-parameter-for-compute-resource).)
### Operations
#### Get version information
```
GET /container
GET /container/version
```
A `GET` request on the base resource will return a JSON object containing metadata. Initially, the only piece of metadata included will be the version number. Eventually, this may include a list of known appliance ISOs.
A `GET` request on the `version` sub-resource will return just the version.
- [ ] Should this also capture the required vSphere permissions for various operations? (If so, how?)
Corresponding CLI: `vic-machine-{darwin,linux,windows} version`
#### Display static message
```
GET /container/hello
```
A `GET` request on the `hello` resource will return a static "welcome" message for users who have been redirected to the server to accept SSL/TLS certificates..
Corresponding CLI: N/A
#### List VCHs
```
GET /container/target/{target-network-address}/[datacenter/{datacenter-id}]/vch?[thumbprint={thumbprint}]&[compute-resource={compute-resource}]
```
Making a `GET` request on `/vch` under a target and optionally a datacenter will return information about the VCHs on that target, in that datacenter.
- [ ] Pagination
Corresponding CLI: `vic-machine-{darwin,linux,windows} ls`
#### Create a VCH
```
POST /container/target/{target-network-address}/[datacenter/{datacenter-id}]/vch?[thumbprint={thumbprint}]
```
Making a `POST` request on `/vch` under a target and optionally a datacenter will create a VCH on that target, in that datacenter. Information about the VCH will be provided in the body of the request in a format similar to this.
Note that validation of the request occurs synchronously, with any errors being returned using an appropriate response code and status. Eventually, portions of the creation will proceed asynchronously, with errors being reported via a vSphere task that is returned once the synchronous validation is complete. (See "[VCH creation spawns a custom task](#vch-creation-spawns-a-custom-task)".)
Corresponding CLI: `vic-machine-{darwin,linux,windows} create`
#### Inspect a VCH
```
GET /container/target/{target-network-address}/[datacenter/{datacenter-id}]/vch/{vch-id}?[thumbprint={thumbprint}]
```
Making a `GET` request on a VCH resource will return information about the VCH. Information about the VCH will be provided in the body of the response in the same format as create.
Corresponding CLI: `vic-machine-{darwin,linux,windows} inspect`
#### Access the creation and reconfiguration log for a VCH
```
GET /container/target/{target-network-address}/[datacenter/{datacenter-id}]/vch/{vch-id}/log?[thumbprint={thumbprint}]
```
Making a `GET` request on `/log` under a VCH resource will return the contents of that VCH's log. The log is created during VCH creation and appended to during subsequent operations. This request is different than most others in that the return type is `text/plain`.
Note that log information will be _persisted_ in multiple files (a timestamped file for each creation/mutation operation). The contents of those files will be combined into a single stream for consumption. In the future, more granular access could be provided.
Corresponding CLI: N/A
#### Access the host certificate for a VCH
```
GET /container/target/{target-network-address}/[datacenter/{datacenter-id}]/vch/{vch-id}/certificate?[thumbprint={thumbprint}]
```
Making a `GET` request on `/certificate` under a VCH resource will return the certificate the VCH uses when acting as a server, which clients may wish to access to download and add to a trust store. This request is different than most others in that the return type is `application/x-pem-file`.
Corresponding CLI: N/A
#### Reconfigure a VCH
```
PUT /container/target/{target-network-address}/[datacenter/{datacenter-id}]/vch/{vch-id}?[thumbprint={thumbprint}]
PATCH /container/target/{target-network-address}/[datacenter/{datacenter-id}]/vch/{vch-id}?[thumbprint={thumbprint}]
```
Making a `PUT` request on a VCH resource will update that VCH's configuration. Information about the VCH will be provided in the body of the request in the same format as create.
In trying to strike a balance between the Robustness Principle and the Principle of Least Astonishment, we will allow for fields which cannot be modified to appear in the body of a `PUT` as long as the value of those fields match the current state of the object. This allows us to be relatively liberal in what we accept while avoiding the potential surprise of having edits dropped from the request. When the value of a field which cannot be modified does not match the current state, a 409 Conflict will be returned. To preserve the idempotency requirement for `PUT`, modifications to mutable portions of the body must not cause immutable portions of the body to change as a side-effect.
Making a `PATCH` request on a VCH resource (with a body as described in RFC 7396) will update a subset of that VCH's configuration.
As `PATCH` is an explicit request to update a set of fields, fields which cannot be modified must not appear in the body of the `PATCH` request, even if the modification would be a no-op.
Corresponding CLI: `vic-machine-{darwin,linux,windows} configure`
#### Delete a VCH
```
DELETE /container/target/{target-network-address}/[datacenter/{datacenter-id}]/vch/{vch-id}?[thumbprint={thumbprint}]
```
Making a `DELETE` request on a VCH resource will delete that VCH. By default, the VCH and any powered off containers will be deleted. A request body may be provided to indicate whether powered on containers and/or volume stores should be deleted.
Corresponding CLI: `vic-machine-{darwin,linux,windows} delete`
#### Upgrade a VCH
```
POST /container/target/{target-network-address}/[datacenter/{datacenter-id}]/vch/{vch-id}?action=upgrade&[thumbprint={thumbprint}]
```
Making a `POST` request on a VCH resource with an action of upgrade will initiate an upgrade of the VCH. The body of the request will be a JSON object containing the following optional properties: `bootstrap-iso` (a reference to a known bootstrap ISO on the OVA) and `rollback` (a boolean value).
Corresponding CLI: `vic-machine-{darwin,linux,windows} upgrade`
#### Debug a VCH
```
POST /container/target/{target-network-address}/[datacenter/{datacenter-id}]/vch/{vch-id}?action=debug&[thumbprint={thumbprint}]
```
Making a `POST` request on a VCH resource with an action of debug will modify the debug settings for the VCH. The body of the request will be a JSON object containing the following optional properties: `enable-ssh` (a boolean value), `authorized-key` (a string representation of a public key), `rootpw` (a string).
Corresponding CLI: `vic-machine-{darwin,linux,windows} debug`
#### View firewall rule settings
```
GET /container/target/{target-network-address}/[datacenter/{datacenter-id}]?[thumbprint={thumbprint}]&[compute-resource={compute-resource}]
```
Making a `GET` request on a vSphere target (with an optional datacenter and compute resource) will return information about the state of the host firewall on those resources. This allows a user to easily determine whether the hosts are in an appropriate state, and allows interfaces to display alerts when they are not. This also provides some measure of symmetry with the update firewall rules operation below.
Corresponding CLI: N/A
#### Update firewall rules
```
POST /container/target/{target-network-address}/[datacenter/{datacenter-id}]?action=firewall:[allow|deny]&[thumbprint={thumbprint}]&[compute-resource={compute-resource}]
```
Making a `POST` request on a vSphere target (with an optional datacenter and compute resource) with an action of `firewall:allow` or `firewall:deny` will update the host firewall on those resources.
Corresponding CLI: `vic-machine-{darwin,linux,windows} update`
## The WebSockets API
A WebSockets-based API will be used to provide streaming access to log data.
- [ ] Specify this.
## Proposed changes to existing functionality
### VCH creation spawns a custom task
Currently, VCH creation consists of three main steps:
1. Validation of the request and the state of the system.
2. A series of vSphere operations to create the VCH.
3. Starting the container VM and its services.
A single parent task could be created for the middle portion of this workflow, to be used as a starting point for a user wishing to query for the status of the creation operation.
When initiating VCH creation via the service, the first (validation) step would occur synchronously. Once the second (vSphere) step begins and the custom task is created, the API would return the handle to the vSphere task, which tracks the asynchronous portion of the request.
### Refactoring of `vic-machine`
The CLI and REST API should each be a thin layer around a common idiomatic Go API. That is, the CLI and REST API should be interaction and translation layers which do not include "business logic". Refactoring may be necessary to achieve this.
Eventually, the CLI could be be re-imagined and implemented as a client of the REST API. The advantages and disadvantages of such a change should be carefully considered.
### Deprecation of demo VCH installer
The demo VCH installer is currently shipped as a part of the OVA as a container with a web application that listens on port 1337. It provides users with a simple web interface to provision a VCH for demo/testing purposes by invoking the `vic-machine` CLI.
Re-implementing this UI using the interface using the new API would be feasible. However, it seems to make sense to consider the use cases for this application in the design of the vSphere H5 client plugin. And as there are no known backwards compatibility guarantees (or important use cases/workflows which would be impacted), it would make sense to deprecate and eventually remove the demo VCH installer once the vSphere H5 client plugin is available.
### Client certificate management using Admiral
Currently, `vic-machine` supports using user-specified client certificates or generating certificates for users to use. As an alternate model, it could delegate client certificate management to Admiral.
As an outline of how this might work:
* When Admiral is installed, it could generate a CA certificate and expose that certificate's public key (or make use of the vSphere CA).
* When a VCH is created, the Admiral CA certificate's public key could be supplied as the client certificate CA.
* When a user in created in Admiral, it could generate a client certificate signed by its client CA certificate (or sign a public key supplied by the user).
* When VCH is authenticating a request, the Admiral CA certificate's public key (from guest info) could be blended with dynamic configuration information from Admiral to limit access to only those users who have been granted access to the VCH's project in Admiral.
* When a user accesses a VCH, they would use the client certificate assigned by Admiral.
This provides several key benefits:
* Each user would have a single certificate which authorizes them to use all VCHs they have been granted access to.
* Operations performed on a VCH can be tied to a user.
* This would help address the lack of certificate revocation mechanisms and allow Admiral to be used to dynamically manage user authorization.
Most relevantly, and perhaps most importantly: this means that a VCH creation workflow does not require upload or download of client certificates, allowing for a simplified user experience.
The work that would be required for this would include:
1. Discovery/lookup of the Admiral CA certificate by `vic-machine-service`.
2. Certificate blending within the VCH. (Validating that the client certificates supplied by Admiral's dynamic configuration are signed by a client CA certificate configured on the VCH, and then limiting access to only those client certificates.)
3. Per-user client certificate generation (or signing) within Admiral.
4. Support for per-user client certificates in Admiral dynamic config.
5. To allow operations performed on a VCH to be tied to a user:
1. Expanding operation logging across the personality/portlayer boundary.
2. Logging the public details of the client certificate used for authorization.
### Reduce friction around use of operations credentials
Because a VCH must be able to interact with vSphere for a variety of operations on behalf of its users (which may not be vSphere administrators), it requires access to a set of vSphere credentials.
When using the `vic-machine` CLI, the default behavior is to persist the credentials which were used for the VCH creation itself. Alternatively, administrators may supply a set of "operations credentials".
This default option cannot be supported for API consumers using a session to authenticate, including the UI. As baseline functionality, we can require that operations credentials are supplied when authenticating with the API in this way. An administrator may chose to supply their own credentials as the operations credentials if they wish to mimic the CLI's default behavior.
Requiring administrators to supply operations credentials works, but is likely to be a point of friction. Beyond the basic inconvenience of treating more information as required in the wizard, an operations user needs to have a non-trivial set of permissions which introduces an additional source of user error.
Support for automatically creating a per-VCH solution user and assigning it the correct permissions seems like the best way to address these pain points. This design also has other advantages: solution users are the accepted mechanism for machine-to-machine interaction, solution users authenticate using a certificate instead of a password, and this would allow us to follow the principle of least privilege by default.
However, issuing requests as a solution user is believed to present significant technical challenges due to immature golang support for SOAP and SAML.
As an interim solution, we can:
1. Introduce logic to validate that supplied operations credentials have the necessary permissions, so that missing permissions can be identified early and clearly.
2. Allow administrators to have vic-machine attempt to automatically grant any missing permissions, so that permissions can be granted corrected quickly and easily.
3. Support automatically creating a non-solution user on behalf of the administrator, and granting it a minimal set of permissions, to further reduce the work required of the administrator.
Eventually, we can then:
4. Enhance the container operations code to support use of a solution user and update the automatic creation logic to create a solution user.
Each of these items builds on the previous, and all three act as building blocks towards #4. Each of these items would be useful for UI, API, and CLI.
### Associating contact information with a VCH
The original VCH configuration had an "environment contact" and an "administrative contact", which were stored using vSphere notes. This allowed a vSphere administrator to associate their contact information with a VCH, so that other administrators would know who to contact before making changes to the system.
This information could also be included on the vicadmin page, even for unauthenticated users, to enable users of all types to know who to talk to for help. (Perhaps the contact information should be free-form to allow for things like referring to a ticket-tracking system.)
- [ ] Customers with large environments must have a way to manage ownership information of entities. Can we learn from that?
## Testing
The service will require two types of tests:
1. Unit tests to verify functionality of handler logic.
2. End-to-end tests to verify the API functionality from a client's point of view (and to serve as the first "client", and as a secondary form of documentation).
Additionally, appropriate testing will be needed for each of the items in the "proposed changes to existing VIC functionality" section.

335
vendor/github.com/vmware/vic/doc/design/vic-machine/upgrade.md generated vendored Normal file
View File

@@ -0,0 +1,335 @@
# Upgrade
After installing a VCH, a VIC engine admin will need to manage the lifecycle of the VCH. One aspect of this lifecycle management is upgrading and patching. For our purposes, patching and upgrading will be treated the same.
The VIC engine product will release a new, complete, software download bundle for both patches and upgrades, versus, having a sparse patch or a separate upgrade model.
## Requirements
### Scope
vic-machine can upgrade VCH endpointVM no matter it's running or not, so vic-machine upgrade cannot rely on services running in VCH endpointVM.
For the VCH endpointVM, vic-machine upgrade will detect the version difference between the old VCH and itself, this includes detecting the guestinfo changes between the two versions. Then vic-machine upgrade will migrate the guestinfo metadata, and upload updated iso files. If anything incorrect happens during this process, vic-machine upgrade will revert back to old version and status. The endpoint API will have downtime during upgrade, but the running containers will not.
The containers managed by this VCH will be left running existing version. If the container is running, it will work well if it's not coupled with endpointVM through port-forwarding. After VCH is upgraded, the container management will be resumed even there is container configuration change. Portlayer will be backward compatible to make sure old version's container still work well.
But if there is security patch for container VM kernel or tether, user still need to recreate new container and then replace the old containers at once or gradually based on business requirement. vic-machine upgrade does not take care of container update, which is the same behavior with vanilla docker.
The exception is that in near future we might replace serial port with VMCI, to solve the VM vMotion issue. If after that change, portlayer lose backward compatibility, which means portlayer cannot talk with old container through tether, after endpointVM upgrade, part of container VM management function will loss, which means container list, inspect, network, volume, start/stop/remove still work, but container interaction, including attach, exec, log will not work.
Summary of upgrade requirement:
- Only upgrade VCH endpointVM
- Portlayer support backward compatibility
- If container functionality is impact, user is responsible to replace old container with new container after VCH endpointVM upgrade
- If there is vSphere credential changes, upgrade will replace old credentials with new one
### Version Difference
VIC version includes three parts, "release tag"-"build id"-"github commit hash". vic-machine upgrade will rely on build id to detect one version is newer or older than another one.
User need to provide newer binary with bigger build id, to upgrade existing VCH. This also requires our build system to keep increase build id number no matter any kind of system change.
Note: after introduced data migration between builds, we introduced one more internal used data migration version. That version is related to data migration plugin only, so will not be shown in vic-machine version. This will be described in data migration section.
### Impact
The VCH endpointVM (control plane) will have downtime during upgrade. Container lifecyles are not coupled with that of the endpointVM and, when using mapped vsphere networks (vic-machine --container-network argument) instead of port forwarding, the network data paths are not dependent on the endpointVM either.
There will be impact on container interaction while the endpointVM is down:
- no direct access to container logs
- no attach ability
- network outage for NAT based port forwarding
All of those facets should resume normal operation after upgrade is complete.
Note: Exception might happen if there is container communication changes. Additional operations are required to fix the problem.
### VCH status
vic-machine ls and inspect will show the VCH versions and upgrade status.
### One Step Roll Back
If anything wrong happens during upgrade, vic-machine will rollback VCH to original version and status.
And a little bit further, even vic-machine upgrade succeeds, user is still able to run vic-machine upgrade --rollback to do one step roll back, which means the endpointVM will be rollback to wherever version it was before this time's upgrade.
The benefit is that if the docker endpoint API does not actually work after upgrade (due to vic-machine upgrade mis judgement) or user appliacation still prefer old version's API for whatever reason, they still could get back to old working version easily.
Limitation:
- If there is new version container created after upgrade, rollback will fail. In that case, user should delete new container through docker CLI or portlayer API.
- Only one step roll-back is supported. As long as new upgrade is executed, no matter it's succeeds or not, the old version's rollback persisence will be removed.
### Resume Interrupted Upgrade
If upgrade failed, it will rollback to old version dynamically to make sure user application is not stopped due to upgrade failure. But there is one scenario still might break everything, that is user input Ctrl+C during upgrade, cause vic-machine upgrade is interrupte, it is not able to roll back everything it did, and then it is useful to resume this kind of operation from vic-machine.
Open Issue: if this is high priority user requirement?
### Downgrade
Verified docker-engine upgrade/downgrade, which works well between 1.11.2 and 1.12. The running container will be stopped after upgrade/downgrade, but is good to start again in new version. And all images and containers information is not lost. But think of the complexity to support version downgrade, we'll not go with this at this moment, only with a limited one step roll back.
### Internet connection
Internet connection is not required to upgrade/downgrade VCH, but newer version's binary should be available for vic-machine.
## Design/Implementation - Phase1
This section described the first simple implementation of upgrade. After this phase, user could upgrade VCH endpointVM from build to build, as long as there is no metadata changes, which means no changes in guestinfo, key value store and image metadata.
This code is already merged, and works for security patch update.
### Versioning
#### Freeze following attributes definition in VCH configuration and container VM configuration
- VCH configuration: ID, Name, Version
- Container VM serial port configuration
- Container VM log/debug file location
#### Configuration Version
Both VCH and container configuration will have version and the value is same with what vic-machine version command shown. After upgrade, VCH configuration version should be updated to new version, but container VM configuration version will still be old one.
New version's VCH will work with both old and new versions containers.
#### Embed iso file version
vic-machine need to identify the iso file version before and after upgrade. Two options here:
- Add version into iso file Primary Volume Descriptor, and read back from vic-machine
- Leverage iso file name including version, e.g. appliance-0.5.0.iso
vic-machine should check iso file version during deployment, and after upload to datastore, version should be appended to iso file name, to make sure mutilple iso files could co-exist in the same VCH, and used for different version's container.
Image file name should not be changed in datastore, cause the file path is used to create container VM. And then vic-machine will leverage file name for version checking during upgrade, to avoid download iso file from datastore back to where vic-machine is running.
Note: No feasible golang library found for this function, so will write our own library to read iso metadata.
#### Refactor current VCH configuration and container VM configuration structure
To make data migration easier, we'll need to refactor current VCH and container VM configuration structure.
VirtualContainerHostConfigSpec is too big, which includes everything in one structure. We're not be able to update part of configuration structure, it will help to update part of data with separated structures based on functionality, e.g. network, storage, and common attributes.
For ContainerVM configuration, there are a few attributes not used for container VM setup, e.g. in ExecutorConfig, Key, LayerID, RepoName, are all not container VM configuration related. Move out irrelated attributes can help the structure stability in the future.
### Transactional Operation/Roll Back
VM snapshot is used to keep current status for upgrade roll back. Before upgrade, vic-machine will create one snapshot of VCH endpointVM. If anything wrong happens, vic-machine can switch back to the pre-upgrade snapshot.
The one step roll-back will roll back the snapshot as well.
Note: VM snapshot does not persist data in serial port, as we will use datastore files through serial port for vch logs, after roll back, log files will have all error message during upgrade if VCH endpointVM is ever started with new configuration.
### Upgrade Workflow
Here is the upgrade workflow
- Find existing endpointVM
- Query VCH configuration
- Verify new version and existing VCH's version
- Cleanup snapshot and images created from last time's upgrade, if there is any. This step is to make sure the rollback after upgrade is not confused.
- Read back old VCH configuration, and migrate to new version. (data migration will be in phase 2 impl)
- Upload iso files
- Prepare combined endpointVM configuration, both hardware spec (ISO path) and the extraconfig portion (VCH version)
--- everything until this portion can be done without interrupting ongoing operation ---
- Snapshot endpointVM
- Poweroff endpointVM if it's not
- Update VCH configuration together with the migrated configuration data
- Reconfigure VCH endpointVM
- Power on endpointVM, and wait VCH service initialization
- Anything wrong in the above steps, roll back to upgrade snapshot
- Ensure endpointVM is powered on and initialized correctly after rollback to snapshot
- Cleanup env after failed upgrade (remove upgrade snapshot, remove uploaded iso files)
- Leave snapshot and old image files for one step roll-back
This ensures that a failure to upload ISOs for whatever reason is detected before we take down the existing version. It limits the failure modes after shutting down the endpointVM to:
1. failed to update endpointVM configuration
2. failed to power on the endpointVM (e.g. system resource constraints)
3. failed to boot endpointVM
4. failed to acquire network addresses (this has been seen in real world examples - we may wish to attempt to preserve/reuse the IPs the endpointVM had prior to update, which should still be present in the extraconfig)
5. failed to rebuild system state from vsphere infrastructure
## Design/Implementation - Phase2
Although we did lots of code refactor for VCH endpointVM configuration, it's still unavoidable to continue changing that structure. To make sure we don't break upgrade after GA, we need a solution for the data migration anyway.
- EndpointVM guestinfo, which is used to persist VCH endpointVM configuration
- Container guestinfo, which is used to persist container configuration
- KeyValue Store and image metadata, following information is persisted
- parentMap
- per-image metadata
- image layer cache
- network portlayer data
- EndpointVM log files
- vSphere Object Management Logic
### Data Migration Framework
Guestinfo is versioned by auto generated build number, git commit hash, build timestamp etc, which are all not controllable during development. So we'll add another sequential version for data migration version control only. For each version, there will have one data migration plugin registered into data migration framework.
Each time to migrate data, either for container or endpointVM, the framework will compare latest version with the existing version to generate data migration path, e.g. r2->r3->f4->latest or r4->latest. And then corresponding plugins will be executed sequentially to migrate data.
Container data and endpointVM data will be migrated separately, because they have different lifecycle. All the metadata in endpointVM, including guestinfo, keyvalue store, image metadata, log files are all part of endpointVM scope, which will be considered as one set and migrated at the same time.
As old container binary will not be updated by vic-machine upgrade, we'll not be able to write container data back to container VM guestinfo. In that case, we need to try to avoid container data change even there is version difference between VCH endpointVM and container. And the framework should explicitly provide methods to detect if there is data version difference.
Based on the design above, we had following migration framework interface:
```
// MigrateApplianceConfigure migrate VCH appliance configuration, including guestinfo, keyvaluestore, or any other kinds of change
// InIt accept VCH appliance guestinfo map, and return all configurations need to be made in guestinfo, keyvaluestore, and can have
// more kinds of change in the future. Each kind is one map key/value pair.
// If there is error returned, returned map might have half-migrated value, this is why we don't persist any data in plugin.
func MigrateApplianceConfigure(ctx context.Context, s *session.Session, conf map[string]string) (map[string]string, bool, error)
// MigrateContainerConfigure migrate container configuration
// Migrated data will be returned in map, and input object is not changed.
// If there is error returned, returned map might have half-migrated value.
func MigrateContainerConfigure(conf map[string]string) (map[string]string, bool, error)
func IsContainerDataOlder(conf map[string]string) (bool, error)
func IsApplianceDataOlder(conf map[string]string) (bool, error)
```
Here is the interface for plugin and plugin manager:
```
type DataMigration interface {
// Register plugin to data migration system
Register(version int, target string, plugin Plugin) error
// Migrate data with current version ID, return true if has any plugin executed
Migrate(ctx context.Context, s *session.Session, target string, currentVersion int, data interface{}) (int, error)
// GetLatestVersion return the latest plugin id for specified target
GetLatestVersion(target string) int
}
type Plugin interface {
Migrate(ctx context.Context, s *session.Session, data interface{}) error
}
```
Notes:
- Developers who change guestinfo, keyvalue store, image metadata, etc, will be the owner to develop migration plugin, and be responsible to increase data migration version.
- Each migration version should have one and only one corresponding plugin.
- If endpointVM configuration and container configuration are changed at the same time, two different plugins should be added, and registered to different plugin category.
- If both endpointVM configuration and keyvalue store are changed, two plugins are recommended as well.
Reminder: extraconfig package should always be backward compatible. If it breaks this assumption, upgrade is broken.
### EndpointVM Migration Process
vic-machine upgrade will upgrade endpointVM and data migration framework will be called to migrate data, which is described in phase1 workflow. There is no new command parameters added for data migration.
#### Guestinfo migration
The plugin should migrate data in memory and return changed value directly. No need to persist any data.
The difficulty is how to migrate secret data in guestinfo. The encryption key is persisted through raw guestinfo.ovfEnv variable, which is readable from in guest, but not through vSphere API, no matter it's VC or ESXi. Decrypt and re-encrypt secret data is necessary because otherwise if old vc login credential is changed, upgrade will always fail.
To solve this issue, we'll need to download endpointVM vmx file and read back guestinfo.ovfEnv value to decrypt encrypted data. In another word, this will be another variable dependent by upgrade framework, which cannot be changed in the future, otherwise, upgrade is broken.
#### KeyValue Store and Image metadata Migration
Different to guestinfo object, VIC engine will not change configuration from guestinfo to anywhere else anytime soon. But from start, keyvalue store persistent position is in argument. Right now, vic engine has a few datastore files for keyvalue store and image metadata, but those information is not configurable, which means hardcoded in portlayer.
Based on this idea, data migration framework will not assume where to load these data, and what version is that, instead, it assume all data for endpointVM is sharing the same version written in VM guestinfo. After compare the version difference, it will invoke plugins sequentially to migrate data including all endpointVM data. The plugin, for this kind of data, should read from datastore, change data and then commit back to datastore, or to anywhere defined in new version. So the input of keyvalue store plugin will be the endpointVM configuration data.
The problem of this solution is that each single plugin will persist its own change, not like the guestinfo update, which is migrated in memory and persisted by migration framework. So the rollback for guestinfo is easy, but not possible for keyvalue store, unless we have data roll back plugin mechanism.
The solution for this issue is to have one model to write key/value store and image metadata migration plugin. Every time we need something new and have to be migrated in new version, we should create new datastore files with a suffix matching the version number, e.g. metadata.v3, and then in the migration plugin, copy all existing data into new versioned files, and modify over there. The old versioned datastore files is not changed by the plugin, so even if eventually we dropped the upgrade, the old binary could still work with the old files without any problem.
#### EndpointVM log files
EndpointVM log files can be handled similar to KeyValue store.
#### vSphere Object Management Logic
There will have new vSphere API come up, so the logic to manage vsphere objects will be enhanced, for example, as Caglar said the vmdk can be managed directly in vSphere 6.5, instead of through vm operations in vSphere 6.0. We'll switch to these new interfaces for image and volume management sometime later.
vic-machine is not supposed to migrate old image data or volume data to new vmdk files, so portlayer will need to be backward compatible.
### Container Data Migration Process
vic-machine upgrade will leave container in old version, and do not update it even container is restarted. So portlayer will be responsible for container's backward compatibility. Here is how portlayer talk to old containers.
- Load container configuration
- Check if data migration is required. If yes, migrate data, and conver to new version's data structure in memory
- Read/write from/to new version's data for whatever container operations
- While need to write container configuration back to container VM guestinfo, check if data migration is done. If yes, skip writing.
Risks:
- As the new data is not written back, there will have few container information inconsistence
Currently, portlayer will write container status into guestinfo before start/stop, and tether will write some. If portlayer does not write this information because of data version mismatch, the container status will have problem if we still rely on portlayer to write it.
In the future, if portlayer add more functions to modify container, and those information is persisted in guestinfo, those change will not work for old container, for example, container rename.
- If portlayer and container communication channel is changed, from serial port to VMCI, and no backward compatibility support, container attach/exec/log etc., interaction related command will not work, but container image/volume/network/start/stop/remove should still work well.
In the above cases, user should think about replace old container with new created one. And if that happens, during endpointVM upgrade, vic-machine should have clear information to mention the container functional limitation after upgrade, and the suggested solution for it.
### One Step Roll Back
To support one step roll back, we'll have one new option in vic-machine upgrade command as following
```
vic-machine upgrade --rollback --<same other upgrade options>
```
Following is the workflow
- Find existing VCH endpointVM
- Check if upgrade snapshot available, and consistent with previous iso file version, if not, stop rollback
- Check if there is any newer version's container created already, if yes, print warning message and stop rollback
- Check if old iso files with same snapshot version still exist, if not, cannot rollback
- Snapshot current endpointVM with version in name
- Switch to old version's snapshot
- Power on endpointVM, and wait VCH service initialization
- Anything wrong in the above steps, roll back to new version's snapshot
- Ensure endpointVM is powered on and initialized after rollback to newer version
- Remove new version's upgrade snapshot if rollback failed
- Remove new version's upgrade snapshot and new version's iso files if rollback succeeds (after rollback, user is still able to reupgrade through the above operation, do not leave anything to avoid confusing)
### Resume Interrupted Upgrade
For user interrupted upgrade through Ctrl+C during upgrade is running, the VCH endpointVM is left in partial migrated status, which is hard to handle manually. Another option in vic-machine upgrade will help on this.
```
vic-machine upgrade --resume --<same other upgrade options>
```
Following is the workflow
- Check if there is one upgrade in progress, we'll rely on upgrade snapshot to see if the VCH is in upgrade. If anything is interrupted before upgrade snapshot is created, VCH endpointVM is not actually changed, user should use general upgrade command to continue.
- Check if docker API is available, if yes, last time's upgrade is already done, no need to resume (if vic-machine upgrade succeeds, but something else still does not work correctly, user should use one step rollback, instead of resume here)
- Check if there is any newer version's container created already, if yes, stop resume. Which means new VCH endpoint API already works, but occasionally break by something else, resume cannot help on that.
- Check if old iso files with same snapshot version still exist, if not, cannot resume
- Upload new iso files no matter they exist or not, to solve any potential iso file broken issue
- Switch to old version's snapshot
- Query endpointVM guestinfo, and migrate to new version.
- Power off endpointVM if it's not
- Update VCH configuration together with the migrated configuration data
- Reconfigure VCH endpointVM
- Power on endpointVM, and wait VCH service initialization
- Anything wrong in the above steps, roll back to upgrade snapshot
- Ensure endpointVM is powered on and initialized correctly after rollback to snapshot
- Cleanup env after failed upgrade (remove upgrade snapshot, remove uploaded iso files)
- Leave snapshot and old image files for one step roll-back
## Restrictions
User cannot run two upgrades for same VCH at the same time.
vic-machine will check if there is already another upgrade snapshot is created before it starts to create snapshot. But as create vsphere snapshot will take some time, e.g. one minute, if at this time, another upgrade process is started, it will start upgrade again cause the snapshot of previous task is not finished yet.
## Integration Test
### Upgrade test w/o data migration
CI test already have test to upgrade from previous release to latest version, and run regression test after upgrade. With data migration framework added, no new integration test is required. But for any data migration added in the future, upgrade test should cover the new features regression.
### One Step Roll Back
After upgrade, run one step roll back and check the old version's VCH. Following scenarios should be covered.
- Roll back works without new container created, and sample function works (do not run regression test for that might have new features not suppored in old VCH)
- Roll back failed for new container created
- Roll back works after removed new version's container, and sample function works
- After roll back, VCH old version is recovered, and new feature is not supported
### Upgrade Resume
During upgrade, break the upgrade process, and check if vic-machine can resume the upgrade. Following scenarios should be covered.
- Before snapshot is created, stop upgrade, resume does not work for no upgrade is found
- Upgrade succeeds, resume should not start for docker endpoint API is ready
- After snapshot is created, stop upgrade, and remove the old version iso files, resume should fail
- After snapshot is created, stop upgrade, and do not touch everything else, resume should work, regression passed
- After resume, one step roll back should still work

367
vendor/github.com/vmware/vic/doc/design/vic-machine/vic-machine.md generated vendored Normal file
View File

@@ -0,0 +1,367 @@
# On the care and feeding of VCHs - vic-machine
vic-machine is both the management client for Virtual Container Hosts and the mechanism by which they are initially deployed.
## Roles, responsibilities, and multi-tenacy models
This document notes three separate roles in the course of deploying and managing a VCH:
* vSphere Administrator (_viadmin_)
* VCH Administrator (_admin_)
* VCH User (_user_)
This workflow is the bridge between the infrastructure administration portion of an organisation and the users. The specific intent here is to allow each of the roles to operate at a level of detail appropriate to that role, with minimal impact or dependency on the others. The:
* _viadmin_ - identified by having administrative access to vSphere
- operates at a business decision level, mapping relative priority of projects and teams to accessible resources and permissible resource limits. Delegates usage authority of those resources to the _admin_ within specific limits enforced by vSphere.
* _admin_ - identified by delegated authority in the form of access to a signed VCH manifest file
- controls sub-division, if any, of assigned resources among finer-grained projects and teams
- deploys a VCH to manage a set of assigned resources. Delegates consumption decisions about those resources to _user_
* _user_ - identified by granted API access to a specific VCH
- controls the specifics of _what_ is done with the available resources, by way of an API client such as the Docker client
This model provides for a form of multi-tenancy, with the _viadmin_ able to specify a service account that will be configured with appropriate RBAC rules. It's unclear at this time if it's viable to create sub-users with further restricted RBAC rulesets, so the working assumption is that all sub-division performed by _admin_ operate with the same service account, and the sub-division is enforced by configuration of vSphere constructs such as resource pools, but without the authority isolation provided by different vSphere users. How a VCH prevents manipulation of those sub-division limits is covered in [the security architecture](security.md).
It is not intended that there be RBAC within a VCH at this time.
How this multi-tenacy model is used is left up to the business, but there are two primary models that we consider during development:
* team based - the VCH is assigned to a team for their use, potentially running a mix of independent workloads
* application based - a VCH is used as the management construct for a given appliancation, with all containers being portions of that app
## VCH manifest
Certain information is required to deploy a VCH, with the _viadmin_ and _admin_ each contributing portions of that data. The VCH manifest is the mechanism by which that cooperation happens, and is the token via which authority delegation from _viadmin_ to _admin_ occurs. Creation of a manifest is conceptually a compositing process with the following inputs and an immutable result:
1. input manifest (if omitted, this creates a new manifest from scratch)
2. a restriction set of some kind ( e.g. compute resource path, datastore prefix, registry whitelist)
- as a note, user credentials are also considered a restriction as they control permissible operations
The _viadmin_ uses vic-machine to create a _base manifest_ containing the following restrictions:
* target vSphere environment
* vSphere user and credential for created VCHs:
- must already exist, or
- must be created by vic-machine during manifest creation, or
- demand created during VCH deployment, requiring stored admin credentials:
* stored _viadmin_ credentials if necessary:
- encrypted credentials and [validating proxy](components.md#validating-proxy) URI, or
- unencrypted credentials (_viadmin_ and _admin_ roles are held by the same entity and manifests are stored securely)
The _base manifest_ is the minimum set of information necessary from the _viadmin_ role. There is additional information that will almost always be required for full function of a VCH:
* vSphere network to use for container network
- must already exist
* vSphere switch to use for container network:
- must already exist
- if switch, port group or network is demand created during VCH deployment, requiring stored admin credentials
- in an enterprise environment, creation of a portgroup without VXLAN or VLAN is unlikely to suffice, so pre-existing portgroup is highly recommended
To create a VCH, disambiguation of which resource to use from the available set is necessary. If there is no ambiguity, i.e only one of each, then this can be omitted:
* _client_ network - necessary for any access to the VCH
* default container network:
- _private_ port group or dedicated vSwitch, or
- IPAM config for _client_ network (defaults to DHCP)
* datastore paths for (may all be the same path):
- images
- containers
- volumes
## Usage Examples
These are various examples intended to illustrate how vic-machine is intended to be used. These are illustrations rather than the finalized naming so if something doesn't make sense ask about it.
### Fully specified command line
```
vic-machine create
-target=https://root@****:vcenter.fqdn/path/to/datacenter
-compute=mycluster/edw-pool/web/edw-web
-path=/folder/path/orthogonal/to/compute
-image-store=ds://datastore/containers/edw
-container-store=ds://datastore/edw/edw/web
-client-network="VM Network"
-management-network="management-net"
-volume-store=ds://edw-san/reports:reports
-volume-store=ds://edw-san/data:data
-container-network=private-vlan12d7f2:bridge
-container-network-ipam=private-vlan12d7f2:172.17.0.128-192/24,172.17.0.1
-container-network=edw-corp-net:backend
-alias=oracledb.edw.corp.net:db.backend
```
The first block of these options control the core configuration of the VCH:
* target - the destination ESX or VC datacenter
* compute - resource pool, host, cluster, or other compute boundary into which the VCH should be deployed.
* path - the folder path into which the VCH VMs will be placed - this is VC only
* image-store - the datastore location where images will be placed, whether pulled via `docker pull` or created via `docker commit`.
* container-store - the datastore location where containerVMs will be created. This does not have to be on the same datastore as the images, but both must be visible to all hosts on which containerVMs are to be created.
* client-network - the network on which users will connect to the VCH to issue DOCKER_API commands.
* management-network - the network providing access to a VMOMI SDK, in deployments where access to the management network is required.
The second block of options control how exisiting vSphere resources are presented to users of the VCH. These are specified as a `source:destination` mapping with vSphere identifier as the source; it is the viadmin role that is providing this data and mapping mydomain:targetdomain aligns with how people tend to think.
* -volume-store - datastore prefixes under which volumes can be created. These prefixes are mapped to labels that can be reference via the --opts mechanism when calling `docker volume create`
* -container-network - vSphere networks that should be exposed via `docker network` commands. As with the datastores this is a mapping from vSphere name to docker name. Ideally the docker name should express something about the purpose of presenting the network such as `internet`, `intranet`, or `databases`
* -container-network-ipam - this is an optional argument furnishing additional information for controlling IP address management on the network, in the form `ipaddress-range/mask,gateway`. If not specified DHCP is used.
* -alias - this allows containers to address a specific FQDN as if it were itself a container managed by the VCH, specified in the form of `FQDN:alias.network`.
### Manifest file - create
Note that the various resource paths end in a `/` - this signifies that it's acceptable for additional refinement to be provided when using this manifest file:
```
vic-machine manifest
-manifest=file::///home/joe/vch/edw.manifest
-target=https://root@****:vcenter.fqdn/path/to/datacenter
-compute=mycluster/edw-pool/
-image-store=ds://vsan-datastore/containers/edw
-container-store=ds://vsan-datastore/edw/edw/
-client-network="VM Network"
-management-network="management-net"
-volume-store=ds://edw-san/reports/:reports
-volume-store=ds://edw-san/data/:data
-container-network=private-vlan12d7f2:bridge
-container-network=edw-corp-net:backend
-container-network-ipam=edw-corp-net:10.118.78.128-192/24,10.118.78.1
-container-network=internet-corp-net:frontend
-alias=oracledb.edw.corp.net:db.backend
```
### Manifest file - use
When using a manifest file it's necessary to fully qualify resources that are left open (i.e. trailing `/` in the manifest). In the case of mappings where it is `source/:destination`, the destination label is used as the resource reference; this avoids a requirement to know the manifest path, although there is no intent at this time to obfuscate specifications beyond encrypting credentials and target URI.
```
vic-machine create
-manifest=file::///home/joe/vch/edw.manifest
-compute=webteam
-container-store=web
-volume-store=reports/web:report
-container-network=bridge
-name=web-team-private
```
The exact consumption mechanic of manifests is still quite uncertain. The example above is working on the premise that all mappings in the manifest are optional and are bound only if referenced, whether simply by label or as part of a refinement. The example above does not reference the _frontend_ network, so no containers can be attached to it.
### List existing VCHs
List the existing VCHs under the specified compute resource. The example below will list all VCHs in the target vCenter.
```
vic-machine ls
-FROM=https://root@****:vcenter/
ID Path Notes
vm-239 mycluster/edw-pool/web/edw-web Notes from the VCH config that get truncated after...
vm-2372 mycluster/random/someother Another VCH in the random pool
vm-15 mycluster/xyz VCH in the root of the cluster
```
The following example lists VCHs under a specific compute resource
```
vic-machine ls
-target=https://root@****:vcenter/example-datacenter
-compute=/mycluster/edw-pool/
```
The following example lists VCHs under a specific folder
```
vic-machine ls
-target=https://root@****:vcenter/example-datacenter
-path=/a/folder/path
```
The following example lists VCHs under a specific resource, described by manifest. Using the example above, this would list all VCHs under /example-datacenter/host/mycluster/Resources/edw-pool/
```
vic-machine ls
-manifest=file::///home/joe/vch/edw.manifest
```
## Inspect existing VCH
Inspect the configuration of an existing VCH by compute path:
```
vic-machine inspect
-target=https://root@****:vcenter/example-datacenter
-compute=host/mycluster/Resources/edw-pool/web-team-vch
```
Inspect the configuration of an existing VCH by folder path:
```
vic-machine inspect
-target=https://root@****:vcenter/example-datacenter
-path=/a/folder/path/web-team-vch
```
Inspect the configuration of an existing VCH by moref - I feel leaking vCenter abstractions is unavoidable if allowing any specifier other than full paths. In this case the identifier is part of target because it is sufficient to be unambiguous without datacenter/path pair:
```
vic-machine inspect
-target=https://root@****:vcenter/moref=vm-10324
```
## Updating an existing VCH configuration
## Implementation approach
Recommended initial approach is to have several components that get rolled into both vic-machine. Initial implementation should focus on:
* CLI parsing into config structs
* validation of config struct values
* creation of VCH from config struct
![vic-machine high level logic](images/vic-machine-high-level.png)
Components:
* command line argument parsing
- maps command line arguments to internal config data structures
- _package: main, path: cmd/vic-machine_
* validation of config structure values
- given an internal config data structure it's necessary to check the specifics against the target vSphere to ensure they are valid - past validity is no indicator of current validity
- this may also translate from symbolic names to morefs, resulting in a manifest that is resilient to name changes, but fragile across vSphere instances with identical naming schemes. This should be a user selectable behaviour, I prefer it on by default.
- _package: spec, path: lib/spec_
* creation of VCH from configuration
- when a manifest has been validated against a vSphere, this component creates the corresponding vSphere objects - this is primarily the VCH applianceVM, but may also include port groups and other objects.
- _package: management, path: install/management_
* [vmomi gateway](components.md#vmomi_gateway)
* manifest creation and consistency
- logic to validate that layered restrictions don't violate prior restrictions, e.g. refining [datastore1]/a/path to [datastore1]/a/path/to/vch is acceptable, but not to [datastore1]/a/second/path
- this should operate on the internal config data structures - probably a sliding window of two layers of the composite manifest each loaded into a config struct
- signing and signature validation
- _package: manifest, path: install/manifest_
* load/save manifest to/from internal config data structures
- this is the mapping from the config held in the composite manifest to the current end configuration, held in a serializable config structure
- signing and signature validation of manifest layers
- _package: manifest, path: install/manifest_
The reason for this breakdown is because some of these elements need to be duplicated in the:
* [validating proxy](components.md#validating-proxy):
- load/save manifest
- manifest consistency
- validation
- reification
- vmomi gateway
* applianceVM
- validation
- vmomi gateway
## =================================================
below this point is working notes.
## Installing - per vSphere target
### Inputs
1. vSphere SDK endpoint
2. vSphere administrative credentials
### Actions
1. deploy ESX agents
2. upload ISOs to common location
3. create custom tasks, alerts, and icons
4. create VCH tag (enable filtering of VCHs)
5. install UI plugin
## Installing - per VCH
### Inputs
* VCH user (existing or new) **
* resource lists:
- pool **
- imagestore datastore paths **
- container datastore paths **
- volume datastore paths (restriction)
- network mappings:
- one network minimum for VCH comms**
- other network mappings
* resource allotments:
- cpu
- memory
- network IO
- disk IO
- datastore quotas (per datastore path)
* certificates
- users - for access to VCH
- hosts - for container access to external hosts
- network - for VCH/container access to networks (gating proxies)
* registry lists
- whitelist
- blacklist
* default container resource reservations and limits *
* containerVM naming convention (displayName for vSphere) *
### Actions
Some of the elevated privilege operations could be delegated during self-provisioning to avoid manifestations of un-utilized authority, e.g. resource pool, user, and rbac entries for a potential but uncreated VCH. This delegation of higher authority requires additional care in the self-provisoning path.
### Requiring elevated privileges
1. create vSphere user for VCH
2. create RBAC entries for VCH resources - resource pool, datastores, networks, et al
3. obtain credentials for VCH user (e.g. SSO token)
* should be revokable
* should only have expiration date if no concern about clean VCH retirement
4. create and size VCH resource pool/vApp
* if vApp then should also configure the start/stop behaviours
* this may encompass disabling certain operations via the UI
5. place credentials in VCH applianceVM extraConfig
## Requiring VCH user privileges
1. validate supplied configuration
2. construct extraConfig/guestinfo configuration for applianceVM
3. create VCH applianceVM
* this may encompass disabling certain operations via the UI
4. upload ISOs if not shared
5. initialize applianceVM
At this point install transitions to managing - reporting VCH status from initial install is the same as reporting that information for any VCH regardless of age.
## Deleting - per vSphere target
### Inputs
1. vSphere SDK endpoint
2. vSphere administrative credentials
3. VCH identification
- VCH resource pool path in govc format, e.g. /ha-datacenter/vm
- VCH name
- VCH ID, which is the value returned by vic-machine ls
The VCH name and VCH resource pool path are identical to the value used in vic-machine create command. Which can uniquely identify one VCH instance.
The VCH ID is the value returned by vic-machine ls, which probably be the VM mob-id, query from VC or ESX.
Either VCH ID or the VCH resource pool path and VCH name should be specified.
### Actions
1. Get VCH VM
2. Read back VCH configuration from VM guestinfo
3. Delete following resources based on VCH configuration.
- Container VMs managed by the VCH
- Container datastore paths and resource pool path configured during installation will be used here, to detect if the VM belongs to this VCH. If yes, these VMs will be removed.
- Container VMs will be removed if they are in stopped status.
- If container VMs are in powered on state, delete will return failure if -force is not specified.
- If container VMs are in powered on state, and -force is specified, vic-machine delete will power off and remove them those VMs.
- volumes managed by the VCH
- -force option is required to delete volumes together with VCH uninstallation.
- Volume datastore path (sample: ds://datastore1/volume/vch1) configured during installation will be used here, to detect if the volumes are created by this VCH.
- If volume directory is empty, vic-machine delete will delete this directory as well, otherwise warning it.
- images managed by the VCH
- Images are shared by VCHs. Currently no reference or metadata to specify which image is referenced by which VCH, so for TP3, vic-machine delete will not touch images.
- vSphere networks created
- vic-machine create or VCH port-layer-server will create network, so vic-machine delete should delete any new networks. (Still need to confirm for how to get created networks)
- VCH specific metadata from vsphere objects if any
- Cause image will not be removed at this time, image metadata will not touched as well. But for container metadata, if any, will be deleted. (Need to finalize where it is)
- the resource pool and appliance VM
### Samples
```
vic-machine delete
--target root:password@192.168.1.1/dc1
--vch-path cluster/pool
--name vch1
--force
```
This command will delete VCH /dc1/host/cluster/pool/vch1 appliance, all containers VMs, networks and volumes created by this VCH.
## Managing a VCH
* report VCH status and information (API endpoint, log server, et al)
* update VCH configuration - implies possible restart of component
* shutdown/reboot VCH
* upgrade VCH - should have an entirely separate doc for this

53
vendor/github.com/vmware/vic/doc/design/volumes.md generated vendored Normal file
View File

@@ -0,0 +1,53 @@
# Volume support
Volumes are mutable disk devices, backed by `.vmdk`, whos lifetime is decoupled from that of the container VM.
### Requirements
- To create a `volume` by `ID` in PL on the specified datastore + path via `volumestore` name and return a URI pointing it.
- To create "anonymous" `volumes and identify them with a generated `ID`.
- To (statefully) remove a `volume` via the PL by its URI.
- To reference an existing `volume` in the `spec` for a new container by its vmkd path
- Store (immutable) metadata in `"key"/[]byte{"value"}` form.
### Filesystem requirements
Mirrors what exists in the storage layer for images. Currently there is an implementation for `ext4`, but the underlying `disk` implementation is opaque to this. In short, we'll use `ext4` for now, but using any other filesystem shouldn't be a huge change.
### Datastore destination
The VI admin will specify datastore + paths which volumes should be created on, giving each of them a `volumestore` to identify them. For instance `[datastore1] /path/to/volumes` can be identified by volumestore name `datastoreWithVolumes`. The the docker user can supply this volumestore name to as a driver arg when creating the volume. The result will be a `.vmdk` created in the `[datastore1] /path/to/volumes/VIC/<vch uuid>/volumes/<volume name>/<volume name>.vmdk`
### Definitions
```
// Unique identifier for the volume. Enumeration of volumes is done via these tags.
type Tag string
type Volume struct {
// Identifies the volume
Tag Tag
// The datastore the volume lives on
Datastore *object.Datastore
// Metadata the volume is included with. Is persisted along side the volume vmdk.
Info map[string][]byte
// Namespace in the storage layer to look up this volume.
SelfLink url.URL
}
// VolumeStorer is an interface to create, remove, enumerate, and get Volumes.
type VolumeStorer interface {
// Creates a volume on the given volume store, of the given size, with the given metadata.
VolumeCreate(ctx context.Context, ID string, store *url.URL, capacityKB uint64, info map[string][]byte) (*Volume, error)
// Get an existing volume via it's ID.
VolumeGet(ctx context.Context, ID string) (*Volume, error)
// Destroys a volume
VolumeDestroy(ctx context.Context, ID string) error
// Lists all volumes
VolumesList(ctx context.Context) ([]*Volume, error)
}
```

69
vendor/github.com/vmware/vic/doc/design/volumes/ContainerCreateWithVolumes.md generated vendored Normal file
View File

@@ -0,0 +1,69 @@
### Docker Personality
The docker personality needed no changes to support nfs volumes. This was due to our Volume Store architecture. Adding a new type of Volume Store still appears as a volumestore to the docker personality. In the future we are planning to distinguish between volume types by customizing our `driver` field. e.g. `vsphere-vmdk` and `vsphere-nfs`.
This will change the way the personality operates, changes will need to be made on how we identify the volume store in order to know which kind of driver we need to return on ls. Driver can also be specified at volume creation time and it is very possible that we will require users to know which volume store is of which driver type. In order to facilitate this we should modify the output of `docker info` to list not only the available volume stores, but also the drivers associated with each store. Alternatively/additionally we can coax which type of driver belongs to a store programatically, this will require some work in the portlayer and docker around fetching a store by name and determining it's type.
In the future some mount options like `squashfs`, `sync`, and `retries` may be desired as configuration options at volume creation time. If this is the case work will need to be done on the personality to parse the additional `driver arguments`. Naturally, the portlayer will also need some additional argument parsing logic to facilitate communicating these options to the tether at mount/run time.
This will also require us to keep a basic set of options for anonymous volumes. At create time users can actually specify several options after the ending `:` character, some investigation will be needed as to how we can use this. The basic options can be used when nothing is specified at the end of the `-v` option on create. We also have a hard coded default capacity for anonymous volumes as this cannot be specified from the docker cli. In the future we should make this configurable at vic-machine create time, [Ticket Reference](https://github.com/vmware/vic/issues/5172).
examples:
```
docker create -v "/mnt/pnt:<some basic opts>"
and
docker create -v "<name>:/mnt/pnt:<some basic opts>"
```
Note: Most importantly from a vic-machine standpoint, A `default` store __must__ be tagged at vic-machine create time for anonymous volumes to be supported in a deployment. Without one any anonymous volume creation requests will be rejected and an error will be returned to the user.
#### Inputs
+ **mount path** is the destination of where the vdmk will be mounted inside the container. _This is required_ if it is the only value set the user is specifying an anonymous volume and we generate a UUID as the name for the volume and this UUID is propagated to the volume metadata.
+ **name** is the value that will be listed a as the name of the volume and the md5 sum of this name will be used as the label for the block device and the target of the mount(portlayer join operation). If this is specified it must be validated. The name is also used to namespace volumes within their respective volumestores. Because of this there is currently and issue out for having a VCH which ends up with two volumes having the same name, [Ticket with information](https://github.com/vmware/vic/issues/5173)
+ **general args** are as follows [rw|ro], [z|Z], [[r]shared|[r]slave|[r]private], and [nocopy]. These should be parsed and placed into the DriverArgs that are specified to the portlayer. right now we only support rw/ro. __TODO__ we do want to research the [nocopy] option. Theses could possibly change in the future for our support depending on how we want to manage other types of volumes beside block based.
__NOTE:__ : in MountPoint for the volume metadata(docker perspective) we need to include something that says "Mountpoint is a block device" or something along those lines.
### Join call for attaching a volume to a vm
This call, which will be implemented in the volume portion of the storage layer within the portlayer srever, will involve a config spec change. The three things needed for this call are the handle to the container, a filled volume struct, and the driver options for the device addition(such as rw/ro). We will add a value to the extraconfig->executorConfig which will append a new Mountspec for the device to be mounted. The Op type will be an "Add"
```
[]DeviceChange{
op:Add,
state(?):exists,
VirtualDevice{
file:<vmdkPath->(should come from volume struct)>
}
}
[]Extraconfig.append{
executorConfig:
label:<generated on creation, should be md5 sum>
MountPoint:<where to mount the vmdk in the container>
}
```
The function signature should look as such
```
func (v *VolumeStore) Join(container_handle *Handle, volume *Volume, diskOpts map[string]string)
```
this will be added to a new file called vm.go as part of the vsphere package under the storage layer code in the Portlayer.
__NOTE__ : there are now two implementations for volume join: an [NFS implementation](https://github.com/vmware/vic/blob/master/lib/portlayer/storage/nfs/vm.go#L42-L57) and a [VMDK implementation](https://github.com/vmware/vic/blob/master/lib/portlayer/storage/vsphere/vm.go#L28-L48)

164
vendor/github.com/vmware/vic/doc/design/volumes/nfsvolumestores.md generated vendored Normal file
View File

@@ -0,0 +1,164 @@
# Specification to support containers with NFS based shared volumes in VIC
Container users want to be able to access shared storage between their containers programatically. Docker solves this by way of host specific bind mounts adding [NFS volumes](https://docs.docker.com/engine/reference/commandline/volume_create/#/driver-specific-options). VIC can streamline this functionality for the user by abstracting this away from the container user and allowing the VI admin to configure the VCH with NFS based volumes access by way of our `VolumeStore`. This way, the VI admin can add an NFS based VolumeStore to the VCH, and the container user need only create volumes on it without needing to know the details of the NFS target.
### Requirements
Allow the VI admin to
1. add an NFSv3 based `VolumeStore`
Allow the container user to
1. specify one or many NFS based `VolumeStore`s to create the container volumes on
1. verify network connectivity from created containers to the NFS targets
1. create volumes on the NFS based `VolumeStore`
1. create 1 or greater containers with NFS based volumes at the given location in the container filesystem namespace
1. validate the volume is no longer in use and delete it
### Non Requirements
1. Instantiation or provisioning of shared storage
2. Exposing shared storage configuration via VIC (e.g. IOPS per client, storage policy, etc.)
3. Management of shared storage via VIC (e.g. container quiesce for storage maintenance, quota manipulation of the target, etc.)
### Networking
Containers will need to be on an appropriate network for the NFS volume store to be accessible.
There are ways this could be done:
- allow association of volume-store with container-network to allow direct coupling
- note in the volume store list what network is required
In our current networking model, this can potentially result in the container using the endpoint vm to NAT NFS traffic to/from the NFS target. This is a potential bottleneck and single point of failure. The other mode we support is adding an NFS container network, and then adding containers requiring the target to the same network. This removes the point of failure but has other issues (*).
_(*) Note: Without microsegmentation support, the services run in the container can potentially be exposed to the network the NFS target is on. This means containers connecting to a specific NFS target all have direct connectivity to eachothers ports._
Ultimately, (once we have microsegmentation support), we'd like to add the container to the appropriate container network in order for the container to have connectivity to the NFS target.
### Implementation
Adding shared storage to our model fits with the `VolumeStore` interface. At install, a VI admin can specify an NFS target as a `VolumeStore` (potentially) using a `nfs://host/<path>` URI with a volume store name. And provided the VCH has access to the network the target is on, the container user only needs to pass the volume store name as one of the `volume create` driver opts to create a volume which will be backed by this shared storage target. Then many containers can be created with the specified volume attached.
#### Runtime network connectivity validation
We need to inform the user when a container is being created without the appropriate network required to get connectivity to the NFS target. The container will attempt to mount the `Target` on `start` and fail early if the volume cannot be mounted. It will be up to the user to communicate with the VI admin and create the container on the appropriate network (*). If the container _is_ on the appropriate network _OR_ the `Target` can be reached via the NAT, the container should mount the volume successfully and move on with `start`.
(*) Note: This requires a doc change.
We want to fail early in the case of issues mounting the volume. Possible errors are
* network connectivity releated
* `Target` permission related
The expectation is the error will be actionable by the user such that if it is a configuration issue related to networking or access, the user can either try the operation again with the right container network configuration, or contact the admin with the action item to allow access to the storage device.
#### VolumeStore
The `VolumeStore` interface is used by the storage layer to implement the volume storage layer on different backend implementations. The current implementation used by VIC is to manipulate vsphere `.vmdk` backed block devices on the Datastore. We have create a similar implementation for `NFS` based volume stores.
The advantage to using the interface is the storage layer maintains consistency of the volumes regardless of the storage backend used. For instance it checks all containers during `volume destroy` to see if the named volume is still referenced by another container (whether the container is powered `on` or `off`).
[For reference](https://github.com/vmware/vic/blob/master/lib/portlayer/storage/volume.go#L36)
```
35 // VolumeStorer is an interface to create, remove, enumerate, and get Volumes.
36 type VolumeStorer interface {
37 »···// Creates a volume on the given volume store, of the given size, with the given metadata.
38 »···VolumeCreate(op trace.Operation, ID string, store *url.URL, capacityKB uint64, info map[string][]byte) (*Volume, error)
39
40 »···// Destroys a volume
41 »···VolumeDestroy(op trace.Operation, vol *Volume) error
42
43 »···// Lists all volumes
44 »···VolumesList(op trace.Operation) ([]*Volume, error)
...
48 }
```
When we create the NFS `VolumeStore`, we'll store the NFS target parameters (`host` + `path`) in the implementation's struct. This is the only information we'll need to mount the NFS target on the container. NFS based volume stores will work for the default volume store as well.
```
// VolumeStore stores nfs-related volume store information
type VolumeStore struct {
// volume store name
Name string
// Service is the interface to the nfs target.
Service MountServer
// SelfLink to volume store.
SelfLink *url.URL
}
```
On creation the volume store target path is mounted to the VCH and the volumes directory and metadata direectories are made under that target path. The client used is one that sits in the user space of the portlayer, this is from the vendored go-nfs client in our vendor directory [For Reference](github.com/fdawg4l/go-nfs-client/nfs). This avoids the issue of The `linux` VFS implementation throwing `sync` errors when mounts are unavailable.
In a container, the volume will be mounted during tether boot time, using the man (2) version of the nfs mount call found in the golang syscall package. Currently, the mount options are not configurable, there are plans to address this going forward as an improvement to the nfs volume implementation. These will be configurable by the VI admin at volume store creation time. They will be configured as query parameters for the url supplied as an nfs target.
Currently a decision must still be made on how we respond to failed mounts due to a target being unreachable at mount time. This is possible when the target endpoint goes down, or the network topology falls out from under the container. Currently, we have no bidirectional communication between the tether and the portlayer. The current option is to fail the launch of the container, though this is not optimal without returning an appropriate error message to the user(we want the user to be able to rectify the issue, or as above to be able to submit an actionable ticket to their vi admin).
#### VolumeCreate
In the vsphere model, a volume is a `.vmdk` backed block device. Creation of a volume entails attaching a new disk to the VCH, preparing it with a filesystem, and detaching it. The resulting `.vmdk` lives in its own folder in the volume store directory (specified during install w/ `vic-machine`). We're going to follow the same model except there is nothing to prepare. Each volume will be a directory (which the container client will mount directly) and live at the top of the volume store directory (which we will prepare during install). We will create the directory for the volume content as well as a directory that sits next to the `volumes` directory. Under the `volumes` directory a directory named after the volume name will be made, that will be the mount target at tether mount time (this path will look like `<vs path>/volumes/<volume id>/<volume contents>`). Likewise, the metadata will be housed under a path denoted under a metadata directory and fetched along with the volume(this path will look like `<vs path>/metadata/<volume id>/dockerMetadata`, the file name is determined by the personality that makes the create request).
Some pseudocode:
```
func VolumeCreate() {
// volPath := vicVolumePath(nameOfVolume)
// mkdir volPath
// metadataPath := vicVolumeMetaDataPath(nameOfVolume, infomap)
.. mkdir metadataPath
// return volPath
}
```
#### VolumeDestroy
Likewise destroying the volume is simply removing the volume's top level directory as well as removing the metadata file.
```
func VolumeDestroy() {
// volPath := vicVolumePath(nameOfVolume)
// rm -rf volPath
// metadataPath := vicVolumeMetaDataPath(nameOfVolume, infomap)
// rm -rf metadataPath
// return $?
}
```
#### VolumeList
Listing the volumes is just listing the diretories at the top of the volume store location
```
func VolumesList() {
// volums := ls -l vicVolumePath(.)
// volumeMetadatas := ls -l vicVolumeMetaDataDir(.)
// volSlice := MatchVolumesAndMetada(volumes, volumeMetadatas)
// return volMap
}
```
### Testing
#### Functional
1. Create a VCH with an NFS backed `VolumeStore`, create a volume on the `VolumeStore`, create 2 containers with the volume attached, touch a file from the first container, verify it exists on the 2nd. Destroy the 2nd container, attempt to destroy the volume and expect a failure. Poweroff the first container, reattempt destroy of the volume, it should fail. Then destroy the container and destroy the volume.
2. Create a VCH with a nonreachable NFS backed `VolumeStore`. Check that vic-machine warns about the lack of creation of that volume store. Possibly check for the appropriate logging in the portlayer logs.
3. Create a VCH with an NFS backed `VolumeStore`, create a volume on said store. Attach this volume to two different cotainers. Touch a file in the volume with container 1. Confirm that this file shows up in container 2. Remove the VCH and confirm that the volume store is removed as well.
4. Create a VCH with an NFS volume store. create a volume and attach it to a container. Touch a file inside the volume attached to that container. Tear down the VCH. Create a new VCh with the same NFS volume store target. Check that the created volume is still there. Attach it to a container. Check that the touched file appears inside the volume.
5. Create a volume store with multiple NFS Volume Stores. Create volumes on each store. Mount each of those volumes to the same container. Touch files to them. attach all volumes to a second container. Make sure that all touched files are present (multiple volume version of test 3)
6. Create a VCH with an nfs volume store. Create a volume attach it to a running container. Attempt to remove the volume. Expect to get a `Volume in Use` error.
7. Create a VCH with an nfs store tagged as the `default store`. Test all the functionalities of the anonymous volume functionality (this will likely be multiple tests there are already several tests to mirror from our vmdk based volume store).
Note: there are more tests, and these tests should also be added to the integration tests ticket.
#### Unit
Whether the `VolumeStore` implementation uses the local VCH to mount the NFS or uses a client library to manipulate the target, the Storer implementation should sit in front of an interface which can be mocked. The mock should write to the local filesystem so the storer interface can be tested end to end without requiring an NFS server.
The NFS volume store implementation has unit tests. These unit tests can be run with or without an NFS server. To test it against an active NFS server you must run `go test` with the environment var `NFS_TEST_URL` set to the nfs target e.g. `NFS_TEST_URL=nfs://my.nfs.com/my/share/point`.
### Possible questions
1. Is there any mechanism by which we can indicate available space in the volumestore? Is this necessary data for functional usage.
* Answer: See Non-requirement 3
1. Should we allow for read-only volume store? - e.g. publishing datasets for consumption
* Answer: Needs investigation. What is RO here (the target or the directory) and what would the container user want to see or expect when such a target was used? This could involve us passing in more arguments at volume creation time.
1. Failure handling; what do we do if a mount is unavailable, does the container go down?
* Answer: Needs investigation. We're relying on the kernel nfs client in the container to handle failures to the target. There is little we can do during run-time, but we can check availability during container create at a minimum. This has been detailed further above. Currently, the result of an unavailable mount is a silent failure. This is actually detrimental to customers, if they do not check the status of their mounts manually (very unlikely) then it is very possible they will run a container believing that data is being persisted. [Tracking Ticket Reference](https://github.com/vmware/vic/issues/4850)
1. NFS target mount options- How do you pass a `uid`/`gid` or `nosquash` mapping? How do you map `uid`/`gid` to the container at all?
* Answer: Requirement and usage needs to be thought through at a higher level. Mapping of users into containers and mapping of credentials into containers need to be solved in the system as a whole. However things like `sync` and `retries` will be specified as a driver option invoked with `vic-machine`, and passed out of band to the container. The container users will not be able to specify mount options specific to the target. Currently, `uid` and `gid` are supported via passing them in as query arguments when specifying the NFS volume store target. These are the only configurable options for now, more discussion will be needed surrounding how we want to handle allowing configuration at target creation time and volume creation time. It is likely that customer feedback( and seeing how customers want to use this) will help drive part of the architecture for mount options.

52
vendor/github.com/vmware/vic/doc/processes/BRANCHING.md generated vendored Normal file
View File

@@ -0,0 +1,52 @@
# VIC Branching
This document is used to define a strategy to enable development and testing of
release candidate, patch found issues, while allowing development of future
releases by way of branching and tagging.
## Requirements
_(Not in priority order)_
* To insulate development of future releases from RC / GA testing and development.
* Allow development of future releases.
* Avoid breaking RC releases while in testing.
_Non requirements_
* Define how to enable development of new features on MASTER or otherwise.
* Define how to test RC / GA releases.
* Define criteria for backporting or forward porting of found bugfixes, features, etc.
* Define release trains for patching GA released builds.
* Define release criteria for RC
## Proposal
We can keep changes isolated from RC / GA testing by way of `git` branches.
###Branching###
* Use [master](http://github.com/vmware/vic) for future release work.
* Use RC branch (`releases/MAJOR.MINOR.MACRO`) for RC release work.
* `TAG` branch for each RC
* `TAG` with `MACRO++` for each patch.
###Accounting###
* Targeting
* Bugs found in RC branch need an issue before merging fix to branch targeted to RC, e.g. `targeted/<RC branch name>`
* Ideally patch should be merged to `MASTER` first if it exists there too.
* Only if issue is targeted for RC, a different PR with the same issue number for the RC branch.
* Only close issue after each relevant and targeted release has had a fixed merged to it. We can use targeting to verify this.
* Bugs found in `MASTER` or any RC branch need to be tagged, e.g.`exists/<branches>`
* The release branch will live as long as our support contract exists on that branch, once support sunsets we can remove the branch
* The CI system will build all branches within the `releases/*` naming convention on push or tag event and publish the binary to our public binary location
```
MASTER -------------------------------------------------->
\ \
0.7---------------- \
\ \ \
TAG 0.7.1 TAG 0.7.2 \
\
0.8---->
```

65
vendor/github.com/vmware/vic/doc/processes/RELEASE-PROCESS.md generated vendored Normal file
View File

@@ -0,0 +1,65 @@
# VIC Engine Release Process
## Release cycle
VIC Engine is developed using a feature driven model. Requirements are broken
down into features and then sized to ensure they will fit into our monthly
releases. Any features that do not are then reworked until they will fit. Any
features under active development that do not make the monthly release will be
pushed into the next months version.
See [CONTRIBUTING.md](../../CONTRIBUTING.md#reporting-bugs-and-creating-issues)
for details on how issues are prioritized and tracked.
## Release Versioning
The VIC Engine project follows Semantic Versioning 2.0.0. This is described in
detail at http://semver.org/
## Example VIC Engine Versions Major revisions:
v1.0.0-beta1 v1.0.0-beta2 v1.0.0-beta v1.0.0 v2.0.0-alpha v2.0.0-beta
v2.0.0
Minor revisions:
v1.1.0-beta v1.1.0 v2.2.0
Update or patch revisions:
v1.12.1 v1.12.20
## Release details
VIC Engine is released in both source and binary form. The source is tagged
using github tagging methods. This is manual for now.
The binary releases are posted at https://console.cloud.google.com/storage/browser/vic-engine-releases/
### Update README.md and documentation
The main repo README contains a project status relating to the latest tagged
release along with guidance on how to build, deploy, et al. The latter should
be updated by any commits changing those workflows, but the status and what's
new needs to be addressed as part of the release process.
* Create a PR, "Release x.y", with the corresponding doc updates once tagging
is imminent merge that PR as the last thing that occurs prior to tagging tag
the release as described in the next section.
###NOTE: Make sure that the last commit before tag does NOT have [skip ci] in the title
### Tag The Release
* Follow the above Release Versioning for choosing the new version. Gather the
Release Notes file. Go to https://github.com/vmware/vic/releases and click
on Draft a new release. Add the tag version that meets the requirements for
this release.
* Title the release "vSphere Integrated Containers Engine Version X.Y.Z" where
X.Y.Z meets the versioning requirements.
* Paste the release note file contents into the Write field and preview.
* If this is release is non-production ready select the "This is a pre-release"
box.
* After a successful build on tag event, you can find the release here:
https://console.cloud.google.com/storage/browser/vic-engine-releases/
* Update the github releases page with the link to the final release tarball

325
vendor/github.com/vmware/vic/doc/user/usage.md generated vendored Normal file
View File

@@ -0,0 +1,325 @@
## Installing Virtual Integrated Containers Engine
The intent is that vSphere Integrated Containers Engine (VIC Engine) should not _require_ an installation step - deploying a [Virtual Container Host](../design/arch/vic-container-abstraction.md#virtual-container-host) (VCH) directly without any prior steps should always be possible. At the current time this is the only approach available.
## Deploying a Virtual Container Host
### Requirements
- ESXi/vCenter - the target virtualization environment.
- ESXi - Enterprise license
- vCenter - Enterprise plus license, only very simple configurations have been tested.
- Bridge network - when installed in a vCenter environment vic-machine does not automatically create a bridge network. An existing vSwitch or Distributed Portgroup should be specified via the -bridge-network flag, should not be the same as the external network, and should not have DHCP.
### Privileges and credentials
There is an operations user mechanism provided to allow a VCH to operate with less privileged credentials than are required for deploying a new VCH. These options are:
* `--ops-user`
* `--ops-password`
If neither option is specified then the user supplied via `--target` or `--user` will be used and a warning will be output. If only the `--ops-user` option is provided there will be an interactive prompt for the password.
At this time vic-machine _does not create_ the operations user, nor does it configure it with minimum set of RBAC permissions necessary for operation - this is actively being worked on (#1689)
Deploying a VCH requires credentials able to:
* create and configure a resource pool (vApp on vCenter)
* create a vSwitch (ESX only)
* create and configure the endpointVM within that pool
* upload files to datastores
* inspect much of the vSphere inventory
However operation of a VCH requires only a subset of those privileges:
* create and configure containerVMs within the VCH resource pool/vApp
* create/delete/attach/detach disks on the endpointVM
* attach containerVMs to supplied networks
* upload to/download from datastore
### Example
Replace the `<fields>` in the following example with values specific to your environment - this will install VCH to the specified resource pool of ESXi or vCenter, and the container VMs will be created under that resource pool. This example will use DHCP for the API endpoint and will not configure client authentication.
- --target is the URL of the destination vSphere environment in the form `https://user:password@ip-or-fqdn/datacenter-name`. Protocol, user, and password are OPTIONAL. Datacenter is OPTIONAL if targeting ESXi or only one datacenter is configured.
- --compute-resource is the compute resource where VCH will be deployed to. This should be the name of a cluster or resource pool, e.g. `myCluster` (vCenter), or `myPool` .
- --thumbprint is the thumbprint of the server's certificate, required if the certificate cannot be validated with a trusted certificate authority (`--force` will accept whatever certificate is presented)
- Note: environment variables can be set instead of these options:
- --target => VIC\_MACHINE\_TARGET
- --user => VIC\_MACHINE\_USER
- --password => VIC\_MACHINE\_PASSWORD
- --thumbprint => VIC\_MACHINE_THUMBPRINT
```
vic-machine-linux create --target <target-host>[/datacenter] --user <root> --password <password> --thumbprint <certificate thumbprint> --compute-resource <cluster or resource pool name> --image-store <datastore name> --name <vch-name> --no-tlsverify
```
This will, if successful, produce output similar to the following when deploying VIC Engine onto an ESXi (output was generated with --client-network-ip specified instead of `--no-tlsverify` denoted above):
```
INFO[2016-11-07T22:01:22Z] Using client-network-ip as cname for server certificates - use --tls-cname to override: x.x.x.x
WARN[2016-11-07T22:01:22Z] Using administrative user for VCH operation - use --ops-user to improve security (see -x for advanced help)
INFO[2016-11-07T22:01:22Z] Generating CA certificate/key pair - private key in ./XXX/ca-key.pem
INFO[2016-11-07T22:01:22Z] Generating server certificate/key pair - private key in ./XXX/server-key.pem
INFO[2016-11-07T22:01:22Z] Generating client certificate/key pair - private key in ./XXX/key.pem
INFO[2016-11-07T22:01:22Z] Generated browser friendly PFX client certificate - certificate in ./XXX/cert.pfx
INFO[2016-11-07T22:01:22Z] ### Installing VCH ####
INFO[2016-11-07T22:01:22Z] Validating supplied configuration
INFO[2016-11-07T22:01:22Z] Configuring static IP for additional networks using port group "VM Network"
INFO[2016-11-07T22:01:23Z] Firewall status: DISABLED on "/ha-datacenter/host/esx.localdomain/esx.localdomain"
WARN[2016-11-07T22:01:23Z] Firewall configuration will be incorrect if firewall is reenabled on hosts:
WARN[2016-11-07T22:01:23Z] "/ha-datacenter/host/esx.localdomain/esx.localdomain"
WARN[2016-11-07T22:01:23Z] Firewall must permit 2377/tcp outbound if firewall is reenabled
INFO[2016-11-07T22:01:23Z] License check OK
INFO[2016-11-07T22:01:23Z] DRS check SKIPPED - target is standalone host
INFO[2016-11-07T22:01:23Z]
INFO[2016-11-07T22:01:23Z] Creating Resource Pool "XXX"
INFO[2016-11-07T22:01:23Z] Creating directory [datastore1] volumes
INFO[2016-11-07T22:01:23Z] Datastore path is [datastore1] volumes
INFO[2016-11-07T22:01:23Z] Creating appliance on target
INFO[2016-11-07T22:01:23Z] Network role "management" is sharing NIC with "client"
INFO[2016-11-07T22:01:23Z] Network role "external" is sharing NIC with "client"
INFO[2016-11-07T22:01:23Z] Uploading images for container
INFO[2016-11-07T22:01:23Z] "bootstrap.iso"
INFO[2016-11-07T22:01:23Z] "appliance.iso"
INFO[2016-11-07T22:01:30Z] Waiting for IP information
INFO[2016-11-07T22:01:44Z] Waiting for major appliance components to launch
INFO[2016-11-07T22:01:54Z] Initialization of appliance successful
INFO[2016-11-07T22:01:54Z]
INFO[2016-11-07T22:01:54Z] vic-admin portal:
INFO[2016-11-07T22:01:54Z] https://x.x.x.x:2378
INFO[2016-11-07T22:01:54Z]
INFO[2016-11-07T22:01:54Z] Published ports can be reached at:
INFO[2016-11-07T22:01:54Z] x.x.x.x
INFO[2016-11-07T22:01:54Z]
INFO[2016-11-07T22:01:54Z] Docker environment variables:
INFO[2016-11-07T22:01:54Z] DOCKER_TLS_VERIFY=1 DOCKER_CERT_PATH=/home/vagrant/vicsmb/src/github.com/vmware/vic/bin/XXX DOCKER_HOST=x.x.x.x:2376
INFO[2016-11-07T22:01:54Z]
INFO[2016-11-07T22:01:54Z] Environment saved in XXX/XXX.env
INFO[2016-11-07T22:01:54Z]
INFO[2016-11-07T22:01:54Z] Connect to docker:
INFO[2016-11-07T22:01:54Z] docker -H x.x.x.x:2376 --tlsverify --tlscacert="./XXX/ca.pem" --tlscert="./XXX/cert.pem" --tlskey="./XXX/key.pem" info
INFO[2016-11-07T22:01:54Z] Installer completed successfully
```
## Deleting a Virtual Container Host
Specify the same resource pool and VCH name used to create a VCH, then the VCH will removed, together with the created containers, images, and volumes, if --force is provided. Here is an example command and output - replace the `<fields>` in the example with values specific to your environment.
```
vic-machine-linux delete --target <target-host>[/datacenter] --user <root> --password <password> --compute-resource <cluster or resource pool name> --name <vch-name>
INFO[2016-06-27T00:09:25Z] ### Removing VCH ####
INFO[2016-06-27T00:09:26Z] Removing VMs
INFO[2016-06-27T00:09:26Z] Removing images
INFO[2016-06-27T00:09:26Z] Removing volumes
INFO[2016-06-27T00:09:26Z] Removing appliance VM network devices
INFO[2016-06-27T00:09:27Z] Removing Portgroup XXX
INFO[2016-06-27T00:09:27Z] Removing VirtualSwitch XXX
INFO[2016-06-27T00:09:27Z] Removing Resource Pool XXX
INFO[2016-06-27T00:09:27Z] Completed successfully
```
## Inspecting a Virtual Container Host
Specify the same resource pool and VCH name used to create a VCH, vic-machine inspect can show the VCH information.
```
vic-machine-linux inspect --target <target-host>[/datacenter] --user <root> --password <password> --compute-resource <cluster or resource pool name> --name <vch-name>
INFO[2016-10-08T23:40:28Z] ### Inspecting VCH ####
INFO[2016-10-08T23:40:29Z]
INFO[2016-10-08T23:40:29Z] VCH ID: VirtualMachine:286
INFO[2016-10-08T23:40:29Z]
INFO[2016-10-08T23:40:29Z] Installer version: v0.6.0-0-0fac2c0
INFO[2016-10-08T23:40:29Z] VCH version: v0.6.0-0-0fac2c0
INFO[2016-10-08T23:40:29Z]
INFO[2016-10-08T23:40:29Z] VCH upgrade status:
INFO[2016-10-08T23:40:29Z] Installer has same version as VCH
INFO[2016-10-08T23:40:29Z] No upgrade available with this installer version
INFO[2016-10-08T23:40:29Z]
INFO[2016-10-08T23:40:29Z] vic-admin portal:
INFO[2016-10-08T23:40:29Z] https://x.x.x.x:2378
INFO[2016-10-08T23:40:29Z]
INFO[2016-10-08T23:40:29Z] Docker environment variables:
INFO[2016-10-08T23:40:29Z] DOCKER_HOST=x.x.x.x:2376
INFO[2016-10-08T23:40:29Z]
INFO[2016-10-08T23:40:29Z]
INFO[2016-10-08T23:40:29Z] Connect to docker:
INFO[2016-10-08T23:40:29Z] docker -H x.x.x.x:2376 --tls info
INFO[2016-10-08T23:40:29Z] Completed successfully
```
## Enabling SSH to Virtual Container Host appliance
Specify the same resource pool and VCH name used to create a VCH, vic-machine debug will enable SSH on the appliance VM and then display the VCH information, now with SSH entry.
```
vic-machine-linux debug --target <target-host>[/datacenter] --user <root> --password <password> --compute-resource <cluster or resource pool name> --name <vch-name> --enable-ssh --rootpw <other password> --authorized-key <keyfile>
INFO[2016-10-08T23:41:16Z] ### Configuring VCH for debug ####
INFO[2016-10-08T23:41:16Z]
INFO[2016-10-08T23:41:16Z] VCH ID: VirtualMachine:286
INFO[2016-10-08T23:41:16Z]
INFO[2016-10-08T23:41:16Z] Installer version: v0.6.0-0-0fac2c0
INFO[2016-10-08T23:41:16Z] VCH version: v0.6.0-0-0fac2c0
INFO[2016-10-08T23:41:16Z]
INFO[2016-10-08T23:41:16Z] SSH to appliance
INFO[2016-10-08T23:41:16Z] ssh root@x.x.x.x
INFO[2016-10-08T23:41:16Z]
INFO[2016-10-08T23:41:16Z] vic-admin portal:
INFO[2016-10-08T23:41:16Z] https://x.x.x.x:2378
INFO[2016-10-08T23:41:16Z]
INFO[2016-10-08T23:41:16Z] Docker environment variables:
INFO[2016-10-08T23:41:16Z] DOCKER_HOST=x.x.x.x:2376
INFO[2016-10-08T23:41:16Z]
INFO[2016-10-08T23:41:16Z]
INFO[2016-10-08T23:41:16Z] Connect to docker:
INFO[2016-10-08T23:41:16Z] docker -H x.x.x.x:2376 --tls info
INFO[2016-10-08T23:41:16Z] Completed successfully
```
## List Virtual Container Hosts
vic-machine ls can list all VCHs in your VC/ESXi, or list all VCHs under the provided resource pool by compute-resource parameter.
```
vic-machine-linux ls --target <target-host> --user <root> --password <password>
INFO[2016-08-08T16:21:57-05:00] ### Listing VCHs ####
ID PATH NAME
VirtualMachine:vm-189 /dc1/host/cluster1/Resources/test1/test1-2 test1-2-1
VirtualMachine:vm-189 /dc2/host/cluster2/Resources/test2/test2-2 test2-2-1
```
```
vic-machine-linux ls --target <target-host>/dc1 --user <root> --password <password> --compute-resource test1
INFO[2016-08-08T16:25:50-02:00] ### Listing VCHs ####
ID PATH NAME
VirtualMachine:vm-189 /dc1/host/cluster1/Resources/test1/test1-2 test1-2-1
```
## Configuring Volumes in a Virtual Container Host
Volumes are implemented as VMDKs and mounted as block devices on a containerVM. This means that they cannot be used concurrently by multiple running, containers. Attempting to start a container that has a volume attached that is in use by a running container will result in an error.
The location in which volumes are created can be specified at creation time via the `--volume-store` argument. This can be supplied multiple times to configure multiple datastores or paths:
```
vic-machine-linux create --volume-store=datastore1/some/path:default --volume-store=ssdDatastore/other/path:fast ...
```
The volume store to use is specified via driver optons when creating volumes (capacity is in MB):
```
docker volume create --name=reports --opts VolumeStore=fast --opt Capacity=1024
```
Providing a volume store named `default` allows the driver options to be omitted in the example above and enables anoymous volumes including those defined in Dockerfiles, e.g.:
```
docker run -v /var/lib/data -it busybox
```
## Exposing vSphere networks within a Virtual Container Host
vSphere networks can be directly mapped into the VCH for use by containers. This allows a container to expose services to the wider world without using port-forwarding:
```
vic-machine-linux create --container-network=vpshere-network:descriptive-name
```
A container can then be attached directly to this network and a bridge network via:
```
docker create --net=descriptive-name haproxy
docker network connect bridge <container-id>
```
Currently the container does **not** have a firewall configured in this circumstance ([#692](https://github.com/vmware/vic/issues/692)).
## TLS configuration
There are three TLS configurations available for the API endpoint - the default configuration is _mutual authentication_.
If there is insufficient information via the create options to use that configuration you will see the following help output:
```
ERRO[2016-11-07T19:53:44Z] Common Name must be provided when generating certificates for client authentication:
INFO[2016-11-07T19:53:44Z] --tls-cname=<FQDN or static IP> # for the appliance VM
INFO[2016-11-07T19:53:44Z] --tls-cname=<*.yourdomain.com> # if DNS has entries in that form for DHCP addresses (less secure)
INFO[2016-11-07T19:53:44Z] --no-tlsverify # disables client authentication (anyone can connect to the VCH)
INFO[2016-11-07T19:53:44Z] --no-tls # disables TLS entirely
INFO[2016-11-07T19:53:44Z]
ERRO[2016-11-07T19:53:44Z] Create cannot continue: unable to generate certificates
ERRO[2016-11-07T19:53:44Z] --------------------
ERRO[2016-11-07T19:53:44Z] vic-machine-linux failed: provide Common Name for server certificate
```
The [`--tls-cert-path`](#certificate-names-and---tls-cert-path) option applies to all of the TLS configurations other than --no-tls.
#### Disabled, `--no-tls`
Disabling TLS completely is strongly discouraged as it allows trivial snooping of API traffic by entities on the same network. When using this option the API will be served over HTTP, not HTTPS.
#### Server authentication, `--no-tlsverify`
In this configuration the API endpoint has a certificate that clients will use to validate the identity of the server. This allows the client to trust the server, but the server does not require
authentication or authorization of the client. If no certificate is provided then a self-signed certificate will be generated.
If using a pre-created certificate, the following options are used:
- `--tls-server-key` - path to key file in PEM format
- `--tls-server-cert` - path to certificate file in PEM format
#### Mutual authentication (tlsverify)
Mutual authentication, also referred to as _tlsverify_, means that the client must authenticate by presenting a certificate to the server in addition to the server authentication with the client.
In this configuration the vicadmin server also requires authentication, which can be via client certificate.
If using pre-created certificates the following option must be provided in addition to the server authentication options above.
- `--tls-ca` - path to certificate authority to vet client certificates against in PEM format. May be specified multiple times.
As a convenience, vic-machine will generate authority, server, and client certificates if a Common Name is provided.
- `--tls-cname` - FQDN or static IP of the API endpoint. This can be an FQDN wildcard to allow use with DHCP if DNS has entries for the DHCP allocated addresses.
These are self-signed certificates and therefore clients will need to be explicit about the certificate authority in order to perform validation.
See [the docker documentation](https://docs.docker.com/engine/security/https/) about TLSVERIFY for details of docker client
configuration (note that the _DOCKER_TLS_VERIFY_ environment variable must be removed from the environment completely to prevent it taking effect)
The following can be used for minimal customization of the generated certificates:
- `--organisation` - shown to clients to help distinguish certificates
- `--certificate-key-size`
If using a static IP for the API endpoint via the following option, a server certificate will be generated using that IP address as the Common Name unless certificates are provided
or `--no-tlsverify` is specified.
- `--client-network-ip` - IP or FQDN to use for the API endpoint
#### Certificate names and `--tls-cert-path`
If using the `--tls-server-key` and `--tls-server-cert` options, any filename and path can be provided for the server certificate and key. However when generating certificates the following standard names are used:
* server-cert.pem
* server-key.pem
* cert.pem
* key.pem
* ca.pem
The default value of `--tls-cert-path` is that of the `--name` parameter, in the current working directory, and is used as:
1. the location to check for existing certificates, by the default names detailed above.
2. the location to save generated certificates, which will occur only if existing certificates are not found
The certificate authority (CA) in the certificate path will only be loaded if no CA is specified via the `--tls-ca` option.
If a warning in the form below is received during creation it means that client authentication was enabled (a certificate authority was provided), but neither that authority nor the ones configured
on the system were able to verify the provided server certificate. This can be a valid configuration, but should be checked:
```
Unable to verify server certificate with configured CAs: <additional detail>
```
_NOTE: while it is possible to mix generated and pre-created client and server certificates additional care must be taken to ensure a working setup_
Sample `vic-machine` output when loading existing certificates for a tlsverify configuration:
```
INFO[2016-11-11T23:58:02Z] Using client-network-ip as cname where needed - use --tls-cname to override: 192.168.78.127
INFO[2016-11-11T23:58:02Z] Loaded server certificate ./xxx/server-cert.pem
INFO[2016-11-11T23:58:02Z] Loaded CA with default name from certificate path xxx
INFO[2016-11-11T23:58:02Z] Loaded client certificate with default name from certificate path xxx
```
#### Using client certificates with wget or curl
To use client certificates with wget and curl requires adding the following options:
```
wget --certificate=/path/to/cert --private-key=/path/to/key
curl --cert=/path/to/cert --key=/path/to/key
```
[Issues relating to Virtual Container Host deployment](https://github.com/vmware/vic/labels/component%2Fvic-machine)

5
vendor/github.com/vmware/vic/doc/user_doc/README.md generated vendored Normal file
View File

@@ -0,0 +1,5 @@
# vSphere Integrated Containers Engine Documentation #
The Markdown source files for vSphere Integrated Containers Engine have migrated to https://github.com/vmware/vic-product/.
To access the documentation in navigable and searchable HTML and PDF formats, go to https://vmware.github.io/vic-product/#getting-started.