d8dfun/virtual-kubelet
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

VMware vSphere Integrated Containers provider (#206)

Browse Source 
* Add Virtual Kubelet provider for VIC

Initial virtual kubelet provider for VMware VIC.  This provider currently
handles creating and starting of a pod VM via the VIC portlayer and persona
server.  Image store handling via the VIC persona server.  This provider
currently requires the feature/wolfpack branch of VIC.

* Added pod stop and delete.  Also added node capacity.

Added the ability to stop and delete pod VMs via VIC.  Also retrieve
node capacity information from the VCH.

* Cleanup and readme file

Some file clean up and added a Readme.md markdown file for the VIC
provider.

* Cleaned up errors, added function comments, moved operation code

1. Cleaned up error handling.  Set standard for creating errors.
2. Added method prototype comments for all interface functions.
3. Moved PodCreator, PodStarter, PodStopper, and PodDeleter to a new folder.

* Add mocking code and unit tests for podcache, podcreator, and podstarter

Used the unit test framework used in VIC to handle assertions in the provider's
unit test.  Mocking code generated using OSS project mockery, which is compatible
with the testify assertion framework.

* Vendored packages for the VIC provider

Requires feature/wolfpack branch of VIC and a few specific commit sha of
projects used within VIC.

* Implementation of POD Stopper and Deleter unit tests (#4)

* Updated files for initial PR
This commit is contained in:
Loc Nguyen
2018-06-04 15:41:32 -07:00
committed by Ria Bhatia
parent 98a111e8b7
commit 513cebe7b7
6296 changed files with 1123685 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

159
vendor/github.com/vmware/vic/doc/design/registries/whitelist-registry.md generated vendored Normal file
View File

@@ -0,0 +1,159 @@
# Whitelist Registries
vSphere Integrated Containers 1.2 (VIC) added the ability to whitelist registry access in an installed VCH. When one or more registries are whitelisted for the VCH at install time the VCH goes into 'whitelist mode'. From this point on, the VCH will only allow access to registries in its list of whitelisted registries. In this mode, users will not be able to access any non-whitelisted registries, public or private.
## Specifying Whitelist Registries at Installation
Whitelisted registries can be declared during a VCH installation with *vic-machine* parameter, `--whitelist-registry`. Two other vic-machine parameters affect whitelist registries, `--registry-ca` and `--insecure-registry`. When whitelisted registries are declared during installation, the latter two parameters acts as modifiers to the whitelisted registries.
Registry-ca declares additional certificates to verify access to registry servers secured with TLS. If a registry is declared as a whitelist registry and not an insecure registry (discussed below), the VCH must have access to the server certificate to verify access. The Photon-based VM that VIC uses has a base set of well-known certificates from public CAs. If a whitelist registry uses a certificate that is not in that set of well-known certificates, the certificate must be uploaded to the VCH via vic-machine's `--registry-ca` parameter.
Insecure-registry declares a registry server that can be used without requiring TLS certificate verification. This modifies the whitelist label and takes precedence. For instance, if a registry is declared with `--whitelist-registry` and with `--insecure-registry`, the VCH will assume the registry is an insecure whitelisted registry. If the registry is listed with only `--whitelist-registry`, then the VCH will attempt to verify access using certificates.
If a registry is declared with `--insecure-registry` but not with `--whitelist-registry`, vic-machine will add the insecure registries to the list of whitelist registries *IF* at least one whitelist registry was declared.
A note about certificates. During installation, vic-machine will attempt to verify the registry server is actually a valid registry server. It will also attempt to validate that the certificates declared in `--registry-ca` are valid for the secure whitelisted registries. Vic-machine only performs best effort validation of registry servers. It will not remove the server's access from the VCH if it cannot validate the server.
Acceptable values for whitelist registry values are numbered IP, FQDN, CIDR formatted range, and wildcard domains. If a CIDR format is used, e.g. 192.168.1.1/24, then the VCH will whitelist any IP address within that subnet. Vic-machine will not try to validate CIDR defined ranges. If a wildcard domain is provided, e.g. *.company.com, the VCH will whitelist any IP address or FQDN address that it can validate against the domain provided during installation. A numeric IP address will cause the VCH to perform a reverse DNS lookup to validate against that wild card domain. As with CIDR values, vic-machine will not attempt to validate wildcard domains during installation. Examples are provided below.
The parameter `--whitelist-registry` creates a list of registries. If multiple whitelist registries need to be declared, repeat `--whitelist-registry` multiple times during installation for each registry.
### Example: vch installation with vic-machine
This example installs 2 whitelist registries and 1 insecure registry.
```
./vic-machine-linux create --target=10.2.2.5 --image-store=datastore1 --name=vic-docker --user=root --password=xxxxx --compute-resource="/ha-datacenter/host/office2-sfo2-dhcp121.mycompany.com/Resources" --bridge-network=vic-network --debug=0 --volume-store=datastore1/test:default --tls-cname=*.mycompany.com --whitelist-registry="10.2.40.40:443" --whitelist-registry=10.2.2.1/24 --whitelist-registry=*.mycompany.com --insecure-registry=192.168.100.207 --registry-ca=/home/admin/mycerts/ca.crt
```
### Example: vic-machine's output during installation
Below is a snippet from the vic-machine output for the above command.
```
May 15 2017 16:36:12.453-07:00 WARN Unable to confirm insecure registry 192.168.100.207 is a valid registry at this time.
May 15 2017 16:36:12.505-07:00 INFO Insecure registries = 192.168.100.207
May 15 2017 16:36:12.505-07:00 INFO Whitelist registries = 10.2.40.40:443, 10.2.2.1/24, *.mycompany.com, 192.168.100.207
```
Had the above command also included --debug=1 (or higher), the following would be the output
```
May 15 2017 16:36:12.453-07:00 WARN Unable to confirm insecure registry 192.168.100.207 is a valid registry at this time.
May 15 2017 16:36:12.505-07:00 DEBUG Secure registry 10.2.40.40:443 confirmed.
May 15 2017 16:36:12.505-07:00 DEBUG Skipping registry validation for 10.2.2.1/24
May 15 2017 16:36:12.505-07:00 DEBUG Skipping registry validation for *.eng.vmware.com
May 15 2017 16:36:12.505-07:00 INFO Insecure registries = 192.168.100.207
May 15 2017 16:36:12.505-07:00 INFO Whitelist registries = 10.2.40.40:443, 10.2.2.1/24, *.mycompany.com, 192.168.100.207
```
There are a few things to note from this snippet.
1. The confirmation of the insecure registry was not attempted.
2. The whitelist registry that is secured was confirmed.
3. Both CIDR and wildcard domain declared as whitelist were skipped during validation.
4. The final whitelist registry list contains all registries declared with both --whitelist-registry and --insecure-registry.
5. While not stated yet, the IP address above that contained :443 will not prevent users to use just the server address during docker commands. This will be discussed below.
## Using Docker Commands Against a VCH in 'Whitelist mode'
VIC currently supports Docker commands that are most applicable for production deployment of containers. The commands that are affected by whitelist mode are docker info, docker login, and docker pull. Below are examples of docker commands issued for the VCH installed with the command above. Let's assume vic-machine properly installs a VCH with the above command and it reports the VCH has an FQDN of myvch.mycompany.com.
### Example: docker -H myvch.mycompany.com info
```
devbox:~/$ docker -H myvch.mycompany.com:2376 --tlsverify --tlscacert="vic-docker/ca.pem" --tlscert="vic-docker/cert.pem" --tlskey="vic-docker/key.pem" info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: v1.1.0-rc3-0-c913391
Storage Driver: vSphere Integrated Containers v1.1.0-rc3-0-c913391 Backend Engine
VolumeStores: default
vSphere Integrated Containers v1.1.0-rc3-0-c913391 Backend Engine: RUNNING
VCH CPU limit: 10414 MHz
VCH memory limit: 58.61 GiB
VCH CPU usage: 3103 MHz
VCH memory usage: 56.03 GiB
VMware Product: VMware ESXi
VMware OS: vmnix-x86
VMware OS version: 6.0.0
Insecure Registries: 192.168.100.207
Registry Whitelist Mode: enabled
Whitelisted Registries: 10.2.40.40:443, 10.2.2.1/24, *.mycompany.com, 192.168.100.207
Plugins:
Volume: vsphere
Network: bridge
Swarm: inactive
Operating System: vmnix-x86
OSType: vmnix-x86
Architecture: x86_64
CPUs: 10414
Total Memory: 58.61 GiB
ID: vSphere Integrated Containers
Docker Root Dir:
Debug Mode (client): false
Debug Mode (server): false
Registry: registry-1.docker.io
Experimental: false
Live Restore Enabled: false
```
There are a few things to note in the output of this docker info call.
1. Insecure Registry and whitelist registry lists are shown.
2. There is a message, 'Registry Whitelist Mode: enabled'. If no whitelist registries are declared during installation, this message will not be shown.
3. 'Registry: registry-1.docker.io' is displayed even though that address was not whitelisted. This is the address for docker hub. It does not mean docker hub is accessible (shown in example below). It is simply the default registry that is attempted when attempting to login or pull without a registry address.
### Example: docker -H myvch.mycompany.com login 10.2.40.40
```
devbox:~/$ docker -H myvch.mycompany.com:2376 --tlsverify --tlscacert="vic-docker/ca.pem" --tlscert="vic-docker/cert.pem" --tlskey="vic-docker/key.pem" login 10.2.40.40
Username:
Password:
Login Succeeded
```
In this example, a command was issued to log onto a registry that was declared during installation. Note, :443 was included during installation but left off during docker login. The VCH will accept either form of the address.
### Example: docker -H myvch.mycompany.com login
```
devbox:~/$ docker -H myvch.mycompany.com:2376 --tlsverify --tlscacert="vic-docker/ca.pem" --tlscert="vic-docker/cert.pem" --tlskey="vic-docker/key.pem" login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: user
Password:
Error response from daemon: Access denied to unauthorized registry (registry-1.docker.io) while VCH is in whitelist mode
```
Notice when the registry address is left off, it attempts to access docker hub (which was indicated in the docker info output above), but the VCH denies access.
### Example: docker -H myvch.mycompany.com pull 10.2.40.40/test/busybox
```
devbox:~/$ docker -H myvch.mycompany.com:2376 --tlsverify --tlscacert="vic-docker/ca.pem" --tlscert="vic-docker/cert.pem" --tlskey="vic-docker/key.pem" pull 10.2.40.40/test/busybox
Using default tag: latest
Pulling from test/busybox
c05511d7505a: Pull complete
a3ed95caeb02: Pull complete
Digest: sha256:85f3a6aadbb0f25e148d9cfbcf23fbb206f7e6159ea168c33ac51e76fdff4b8e
Status: Downloaded newer image for test/busybox:latest
```
This succeeds as it should.
### Example: docker pull busybox
```
devbox:~/$ docker -H myvch.mycompany.com:2376 --tlsverify --tlscacert="vic-docker/ca.pem" --tlscert="vic-docker/cert.pem" --tlskey="vic-docker/key.pem" pull busybox
Using default tag: latest
Access denied to unauthorized registry (docker.io) while VCH is in whitelist mode
```
An attempt to pull from docker hub fails with a message that access was denied while the VCH is in whitelist mode.