d8dfun/virtual-kubelet
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

VMware vSphere Integrated Containers provider (#206)

Browse Source 
* Add Virtual Kubelet provider for VIC

Initial virtual kubelet provider for VMware VIC.  This provider currently
handles creating and starting of a pod VM via the VIC portlayer and persona
server.  Image store handling via the VIC persona server.  This provider
currently requires the feature/wolfpack branch of VIC.

* Added pod stop and delete.  Also added node capacity.

Added the ability to stop and delete pod VMs via VIC.  Also retrieve
node capacity information from the VCH.

* Cleanup and readme file

Some file clean up and added a Readme.md markdown file for the VIC
provider.

* Cleaned up errors, added function comments, moved operation code

1. Cleaned up error handling.  Set standard for creating errors.
2. Added method prototype comments for all interface functions.
3. Moved PodCreator, PodStarter, PodStopper, and PodDeleter to a new folder.

* Add mocking code and unit tests for podcache, podcreator, and podstarter

Used the unit test framework used in VIC to handle assertions in the provider's
unit test.  Mocking code generated using OSS project mockery, which is compatible
with the testify assertion framework.

* Vendored packages for the VIC provider

Requires feature/wolfpack branch of VIC and a few specific commit sha of
projects used within VIC.

* Implementation of POD Stopper and Deleter unit tests (#4)

* Updated files for initial PR
This commit is contained in:
Loc Nguyen
2018-06-04 15:41:32 -07:00
committed by Ria Bhatia
parent 98a111e8b7
commit 513cebe7b7
6296 changed files with 1123685 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

193
vendor/github.com/vmware/vic/doc/design/vic-machine/configure.md generated vendored Normal file
View File

@@ -0,0 +1,193 @@
# VCH Configure (P0)
This document defines the VCH configure command line, and the related changes in vic-machine, to make VCH configure process more reasonable.
Here are the key points considerred in this Definition:
- Batch Operation
We'd like to change VCH configurations more than one item at one time to make the command easy to use
- Reuse all options used in vic-machine create
In vic-machine create, we already have 57 options to define VCH configurations, though not all of them can be updated, many of them should be able to be changed, and this number will grow over time, so we definitely do not want to introduce 3X that number of options to configure (configure means add/modify/remove)
## Options
First option is to configure all in one command
### Implicit Add/Modify and Explicit Delete
All existing vic-machine create options are supported in vic-machine configure command. The meaning of those options are as following:
- anything not specified means no change
- any option specified in configure command means replacement.
- if the old value is not specified during creation, this new value will be added
- if there is old value specified, no matter it's a list value or not, we'll replace the old value with the new one specified in configure command.
here are the possibilities to change list option ```--registry-ca```, if the create command option is ```--registry-ca ca1 --registry-ca ca2```
* ```--registry-ca ca2 --registry-ca ca3```, means new configuration will be [ca2, ca3], ca1 is deleted, and ca3 is added
* ```--registry-ca ca2```, means new configuration will be [ca2], ca1 is deleted
* ```--registry-ca ""```, means ca1 and ca2 are all deleted // Here the empty string value is required, and the representation in different os platform is different.
* nothing specified for registry-ca, then existing [ca1, ca2] is not changed
The benefit of this option is it can update all things in one command, and no new option introduced. User can use exactly same command option in both vic-machine create and vic-machine configure.
### Separate Configure to Sub Commands
The idea of this option is to update a few configuration each time, which is one delta update option. For example, to update volume-store, we can use following command
```
vic-machine configure volume-store --rm ds://datastore/volume:default
vic-machine configure volume-store --add nfs://host:port/container:nfs
```
The command format is like ```vic-machine configure <configure object> --<configure keyword> --<configure option> <configuration value>```
Pros:
- The whole configuration options are split to many sub commands. Then in each sub command, only a small number of options are available, and that does not increase over time. Only the sub command number will increase.
- Consistent with existing update firewall command format (though the command name might be different)
Cons:
- We'll need to reinvent option names
For example, to update client network, user need to specify all options if they want to use static ip. Then in one update command, we'll need all those options. Here is the command looks like ```vic-machine configure bridge-network --port-group value --gateway value --ip value```
The existing option name in vic-machine create is ```--client-network value --client-network-gateway value --client-network-ip value```, we'll need to remove the prefix cause that is already shown as sub-command name.
Decision: We'll go with the first option
## Show Existing VCH configuration
Right now, there is no user friendly way to view all configurations of existing VCH. To support update operation, we need to make it easier, otherwise, update only makes things weird.
vic-machine inspect command is available now, just it shows only the execution endpoint of VCH, instead of the configuration. We can extend that command, to support configuration inspection as well.
- vic-machine inspect <--configuration, --conf>
show VCH configuration only
- vic-machine inspect <--execution, -e>
show VCH execution endpoint (default behavior)
### Configuration Inspection
For configuration inspection, two output format should be supported
- yml format
This should print out all VCH configuration with predefined categories, e.g. general information, storage, network, etc. in user friendly format. Secret information will be hidden or skipped.
Following is sample format:
```
General:
Target: "vc.domain.com"
Thumbprint: "98:97:D1:7B:DF:..."
OpsUser: "Operator"
Name: "VCH-1"
Proxies:
- "https://host:port"
- "http://host:port"
Compute:
Path: " myCluster/Resources/myRP"
CpuLimit: 0 MHz
MemoryLimit: 0 MHZ
Storage:
ImageStore: "datastore"
VolumeStore:
- "ds://datastore/volumes/default": "default"
- "nfs://userid@host:port/volumes/": "nfs"
Networks:
BridgeNetwork:
PortGroup: "bridge"
IpRange: "172.16.0.0/12"
ClientNetwork:
PortGroup: "client"
Gateway: "10.1.0.0/16,10.2.0.0/16:10.0.0.1"
...
DNSServer:
- "8.8.8.8"
- "8.8.4.4"
Certificate:
...
Registries:
InsecureRegistries:
- "..."
RegistryCA:
- "ca file"
Endpoint:
Cpu: 1
Memory: 2048 MHZ
Debug: 3
```
- Command Option Format
The yml format output is more readable for VIC admin, but if users want to recreate their vic-machine create or configure command, it's hard to do based on that format, because there is no one on one mapping between the configuration and command options.
Here we'd support another option as "vic-machine inspect --conf", this will print out the vic-machine create command options minus secret information.
The sample output:
```
--target vc.domain.com
--name VCH-1
--compute-resource /vcqaDC/host/cls/Resources
--ops-user operator
--image-store datastore1
--volume-store ds://datastore/volumes/default:default
--volume-store nfs://userid@host:port/volumes/:nfs
--bridge-network bridge
--bridge-network-range 172.16.0.0/12
...
```
Based on this output, it's easy to modify existing configuration to configure command options
Note: Special character might be escaped based on current OS platform. But even with this, the command options cannot be copied to other platform.
- Subcommand Inspect (corresponding to configure subcommand option)
The configure subcommand option might change command option based on each different sub command requirement, to get current configuration, we'll need to have command inspect for each subcommand. So the option is to extend vic-machine inspect command to include sub command as well.
For example, ```vic-machine inspect bridge-network``` will print out current configuration in the command format, to make sure user can easily generate their configure command based on that.
Note: this is not needed atm, because we're not doing sub configure command
## Update In Progress Status
Same to vic-machine upgrade, concurrent configure requests for same VCH is not allowed. vic-machine should be able to detect if one VCH is running configure/upgrade. If true, return useful error message. And also should not leave that configure flag in there after configure is stopped, or even interrupted.
Issue https://github.com/vmware/vic/issues/4069 is tracking this problem.
## Backward Compatibility
It's possible that after vic engine released new version, user is trying to configure older version's VCH with newer vic-machine.
There are few options:
- Run configure using same version's vic-machine (this option should technically always work)
For any customer that does not want to change vic-machine version, they need to run configure using same version's vic-machine.
- Combine upgrade process into configure
Upgrade is actually one special kind of configure, technically it's easy to have upgrade run together with configure. User might want to configure VCH, and also want to upgrade it after or before configure. In that case, a combined configure and upgrade command can provide a better user experience.
Here is the combined command ```vic-machine configure --upgrade <all configure and upgrade options supported here>```
- Support backward compatibility in vic-machine
This will need to add backward compatible code in vic-machine. Even initially there is only few changes, that will expand massively over time, which is not consistent with our data migration design, so this is not our option.
# VCH configure Rollback (P0.5)
Similar to VCH upgrade workflow, before configure VCH, snapshot should be took to keep existing status. If the configure failed eventually, VCH should be reverted to old status automatically.
The Configure Rollback is talking about after a successful configure, vic-machine is still able to rollback VCH's old configuration. The command is like vic-machine configure --rollback
This feature is nice to have, but as vic-machine upgrade already have this capability, mark it as P0.5.
- Set Env
`vic-machine inspect` should set user env like what `vic-machine create` did, to make sure user can run docker command correctly.
# Manifest File
vic-machine create already has more than 50 options, and the number will increase over time. With this configure feature added, most of those options need to be updated as well. To improve user experience, in my opinion, we need Manifest file to simplify vic-machine command generation.
[vic-machine design](vic-machine.md#vch-manifest) mentioned manifest idea, here we'll start from a simple implementation. The manifest file format will be same to the yml file format, mentioned in [configuration inspection](#configuration-inspection)
- vic-machine inspect --manifest filename will dump all configurations in yml file format to the manifest file.
Here is how to manage VCH using manifest files.
- vic-machine create --manifest will load the manifest file and create VCH based on it.
- vic-machine configure --manifest will load the manifest file and update VCH configuration based on it.
In addition, all the command options can be used together with manifest file, and the command option will override the value in manifest file
TODO: Manifest management command line is not well defined yet. Here is one possible solution for it.
- vic-machine update manifest
All command options in `vic-machine create` can be used for this command.
This manifest file approach will simplify vic-machine create/configure workflow, and leverage user readable configuration format. But it will be an additional work for existing vic-machine create.

508
vendor/github.com/vmware/vic/doc/design/vic-machine/service.md generated vendored Normal file
View File

@@ -0,0 +1,508 @@
1. [Overview](#overview)
2. [Design goals and project scope](#design-goals-and-project-scope)
3. [Implementation Decisions](#implementation-decisions)
1. [Technology Choices](#technology-choices)
2. [Delivery](#delivery)
3. [Cardinality](#cardinality)
4. [Service Upgrade](#service-upgrade)
5. [Compatibility](#compatibility)
6. [Authentication](#authentication)
7. [Certificate Management](#certificate-management)
1. [Host Certificates](#host-certificates)
2. [Client Certificates](#client-certificates)
8. [Log Management](#log-management)
1. [Operation logs](#operation-logs)
2. [Server logs](#server-logs)
9. [Cross-Origin Requests &amp; Cross-Site Request Forgery](#cross-origin-requests--cross-site-request-forgery)
10. [ISO Management](#iso-management)
11. [Communicating modifiability](#communicating-modifiability)
12. [Use of a query parameter for compute-resource](#use-of-a-query-parameter-for-compute-resource)
13. [Use of a request body for delete](#use-of-a-request-body-for-delete)
4. [The REST API](#the-rest-api)
1. [API Versioning](#api-versioning)
2. [Headers](#headers)
3. [Resources](#resources)
4. [Query Parameters](#query-parameters)
5. [Operations](#operations)
1. [Get version information](#get-version-information)
2. [List VCHs](#list-vchs)
3. [Create a VCH](#create-a-vch)
4. [Inspect a VCH](#inspect-a-vch)
5. [Access the creation and reconfiguration log for a VCH](#access-the-creation-and-reconfiguration-log-for-a-vch)
6. [Access the host certificate for a VCH](#access-the-host-certificate-for-a-vch)
7. [Reconfigure a VCH](#reconfigure-a-vch)
8. [Delete a VCH](#delete-a-vch)
9. [Upgrade a VCH](#upgrade-a-vch)
10. [Debug a VCH](#debug-a-vch)
11. [View firewall rule settings](#view-firewall-rule-settings)
12. [Update firewall rules](#update-firewall-rules)
5. [The WebSockets API](#the-websockets-api)
6. [Proposed changes to existing functionality](#proposed-changes-to-existing-functionality)
1. [Support for authenticating via SAML or with a session key in vic-machine](#support-for-authenticating-via-saml-or-with-a-session-key-in-vic-machine)
2. [VCH creation spawns a custom task](#vch-creation-spawns-a-custom-task)
3. [Refactoring of vic-machine](#refactoring-of-vic-machine)
4. [Deprecation of demo VCH installer](#deprecation-of-demo-vch-installer)
5. [Client certificate management using Admiral](#client-certificate-management-using-admiral)
6. [Reduce friction around use of operations credentials](#reduce-friction-around-use-of-operations-credentials)
7. [Associating contact information with a VCH](#associating-contact-information-with-a-vch)
7. [Testing](#testing)
8. [See Also](#see-also)
## Overview
The `vic-machine-{darwin,linux,windows}` command line utilities are used to perform Virtual Container Host (VCH) lifecycle operations.
The `vic-machine-server` exposes similar functionality via a REST API to allow for a wider range of interaction models, such as the integration of VCH management into web interfaces (including the vSphere H5 client).
## Design goals and project scope
This API will become a public, documented API at some point in the future (and, to some extent, all APIs are public), but functionality will be delivered incrementally and the first versions will be versioned in a way that communicates breaking changes are expected and backwards compatibility may not be maintained.
The initial priority is for this API to implement sufficient functionality to enable development of a vSphere plugin, as described here: https://vmware.invisionapp.com/share/GDC9QEDAZ. This may require functionality which is _not_ currently implemented by the `vic-machine` command line utilities.
The API may not initially _implement_ all functionality currently implemented by the `vic-machine` command line utilities. However, to avoid painting ourselves into a corner, it is desirable to consider all functionality in the design. Further, the _design_ should consider plausible future work to ensure that eventual extension of the API is possible.
Because this API would likely be used in conjunction with (or by developers familiar with) other VMware APIs, it is desirable to provide a similar "look & feel" where possible.
Because this API may be used in conjunction with (or by developers familiar with) other container-related APIs, providing a similar "look & feel" to those APIs is also desirable.
This document will attempt to adhere to VMware standards and guidance around API design.
## Implementation Decisions
### Technology Choices
The API will be defined using OpenAPI (Swagger), today's de facto standard for REST APIs.
The service will be built in Go, using `go-swagger`. This allows any developer on the team to contribute, maintain, and modify the service with a minimal learning curve.
WebSockets will be used to provide streaming information about the status on long-running operations so that sophisticated clients do not need to poll for updates. See "[The WebSockets API](#the-websockets-api)" for more information.
Unit tests will be defined using the Testify libraries.
End-to-end tests will be defined using the Robot Framework.
### Delivery
The new service is implemented as an additional flavor of `vic-machine`: `vic-machine-server`. This standalone linux binary (a swagger server) is invoked with port and certificate information to serve the REST API on one or more interfaces.
This service is packaged as a container and included in the VIC OVA, following existing best practices.
The service does not require a access to a persistent data directory, but it useful to provide one as a destination for log files.
A configuration file (stored on the OVA's data directory) may eventually be used to provide the service with high-level configuration information, such as a syslog server and perhaps the admiral CA certificate (see "[Client certificate management using Admiral](#client-certificate-management-using-admiral)".
### Cardinality
It is not assumed that a single instance of `vic-machine-server` will run in a given environment or which manage a given vSphere resource. It is also expected that one or more instances of `vic-machine-server` and the `vic-machine` CLI will be used side-by-side. This means that in-process locking will not be sufficient to protect resources from concurrent access or modification.
In the future, specific use cases may be identified for multiple instances of `vic-machine-server` around availability, scalability, or isolation.
### Service Upgrade
The VIC OVA is upgraded in a side-by-side fashion:
1. The new OVA is deployed.
2. The old OVA is shutdown.
3. The data disk is moved from the old to the new.
4. A data migration script is run.
* In the case of the upgrading to the first version of the OVA that includes the `vic-machine` server, sane default values could be assumed (e.g., by copying syslog settings from another service).
5. The new OVA is started.
This does involve downtime of the service (but not of the VCHs or containers). Clients (including the H5 plugin) should handle this appropriately.
### Compatibility
We expect the compatibility between `vic-machine-server` and VCHs to be similar to `vic-machine-{darwin,linux,windows}`:
* Creation will only be supported for VCH of the same version
* Reconfigure will only be supported for VCH of the same version
* Upgrade will only be supported for VCH of the same or lesser versions
* Rollback will only be supported to the same VCH version
* Deletion will be supported for VCH of the same version, and will be best-effort for older versions
We expect the vSphere H5 client plugin to support a single version of `vic-machine-server` and be upgraded in lock-step.
- [ ] Do we need the plugin to provide at least basic support for the "N+1" version of `vic-machine-server` as well so that it remains functional between when the OVA is upgraded and when the plugin is upgraded?
### Authentication
As with the `vic-machine` CLI, vSphere credentials must be supplied each time an operation is invoked using `vic-machine-server`. The service itself will not store or manage credentials. This means that **all** operations must either be explicitly tied to a vSphere operation or unauthenticated. This is similar to the model for the `vic-machine` CLI, where all operations require vSphere credentials except for those to display help and version information.
See "[Headers](#headers)" for information about how credentials will be passed, "[Certificate Management](#certificate-management)" for a discussion of how this design influences the design for certificate management, and "[Log Management](#log-management)" for a discussion of how this design influences the design for log persistence and access.
### Certificate Management
When invoking the `vic-machine` CLI, access to PKI files is managed out-of-band. PKI files can be placed on the filesystem, and paths passed to the CLI. Generated PKI files are placed on the filesystem for subsequent access, and the OS handles access control for those files.
With the REST API, these workflows need to be handled explicitly: the REST API must allow PKI files to be supplied as a part of requests and must allow for retrieval of generated PKI files, with appropriate access controls.
#### Host Certificates
Host certificates are persisted in the VCH's guest info and are available via the vSphere API today. `vic-machine-server` allows users to access them via its API, regardless of how the VCH was created.
#### Client Certificates
Due to the complexity of ensuring the secrecy of client certificate private keys, the API will not support generation of client certificates. (If this becomes a requirement, generated certificates could be protected with a user-supplied passphrase and stored in the same manner as log files.)
For an alternative approach that allows for creation of VCHs without requiring users to specify client certificates, see "[Client certificate management using Admiral](#client-certificate-management-using-admiral)".
### Log Management
#### Operation logs
When invoking the `vic-machine` CLI, real-time information is provided to stdout and log files can be persisted on the filesystem. The REST API needs to provide equivalent functionality.
Logs will be streamed to the VCH's datastore folder as `vic-machine-server` executes. Access to logs is then restricted to those vSphere users who can read those files from the datastore.
#### Server logs
Logs for the server itself are written to a configurable directory. These logs include server lifecycle information as well as information about each request. Operation IDs are used to associate a request with the log messages related to the handling of that request.
Logs are not rotated by the server. When deployed as a part of the OVA, `logrotate` is used to provide this functionality.
### Cross-Origin Requests & Cross-Site Request Forgery
Because the vSphere H5 client plugin and the `vic-machine-server` will be served from different hosts, the H5 client plugin will be making cross-origin requests. As these would normally be prevented by the browser's same-origin policy, the service must support responding to all requests with an `Access-Control-Allow-Origin` header with an appropriate value, and respond appropriately to `Options` requests for all resources.
The same-origin policy is intended to prevent cross-origin requests because the browser might inadvertently provide credentials on behalf of the user. Given that each individual request to the API must include credentials, and that we won't be using cookies or persisting those credentials in any other way, the same-origin policy does not provide protection in our case.
To avoid additional configuration complexity, we will use a combination of `Access-Control-Allow-Origin: *`, ` Access-Control-Allow-Credentials: false` (the default value), and `Access-Control-Allow-Headers: Authorization, X-VMWARE-TICKET` to express that cross-origin requests are allowed from any origin, and that an `Authorization` or `X-VMWARE-TICKET` header may be included, but that the user-agent should not include cookies or HTTP authentication information based on the user agent's previous interactions with the API. This `Access-Control-Allow-Credentials` restriction will not affect the H5 client plugin, as it will be _explicitly_ including credentials by setting an `Authorization` or `X-VMWARE-TICKET` header, not relying on `XMLHttpRequest.withCredentials`.
### ISO Management
Several VCH operations (create, configure, and upgrade) take a pair of ISOs as input. These ISOs are used for booting the VCH appliance and container VMs respectively.
The current implementation involves maintaining the master ISOs on the OVA and duplicating these ISOs for each VCH that is created. We may wish to change this behavior in the future to make it easier for customers to build their own bootstrap ISO, or to transition to something like direct boot.
Even considering only the current model, uploading these ISOs as a part of service API calls would introduce complexity and inefficiency.
For simplicity and flexibility, we will provide an API that lists "flavors" of ISOs which are known to the OVA and which may be used for API operations. By default, a single "flavor" will be included with the OVA: the stock appliance and boostrap ISOs shipped with that release. We will provide instructions for users who wish to add additional "flavors" of the appliance ISO (e.g., in order to use a RHEL kernel or systemd), which might involve putting the ISOs in a particular directory on the OVA or adding paths to the ISOs in some sort of manifest file. For now, custom bootstrap ISOs will not be supported (but it is easy to imagine adding support following this same pattern).
In the future, it may be desirable to improve our handling of ISOs to reduce duplication (e.g., by storing them on the datastore instead of within the OVA). This is, however, orthogonal to the introduction of an API.
- [ ] Come up with a better term than "flavor".
### Communicating modifiability
VCH properties may or may not be modifiable for a variety of reasons. Some properties, such as the id, may never be modifiable. Other properties may not be modifiable without a power-cycle of the VCH. Yet others may be dependent on the state of other resource, such as whether containers are using an attached network.
In all cases, the server would enforce unmodifiablility when performing an operation, but to provide a good experience for direct and indirect users of the API, it would be helpful to communicate which properties of a given VCH are not currently modifiable and why. (That is, it's better to grey out a field in the interface and provide a help tooltip with an explanation than to allow users to attempt to an operation which will inevitably fail.)
A variety of approaches exist for this:
* Capturing the general mutability rules in a formal language so that clients can evaluate those rules against the current state of the resource. This powerful pattern would allow interfaces built on the API to clearly communicate why a field cannot be modified, and offer options for remediation, without attempting the modification. However, this would require substantial effort for both the client and server.
* Evaluating mutability rules on the server to determining point-in-time mutability, and including that information as a part of GET requests. This reduces the burden on the client, while still allowing for a good user experience. However, this significantly bloats the API and leads to GET/PUT asymmetry.
* Documenting the mutability rules in a human-readable way. This would allow implementors of interfaces built on the API to read the rules and select some or all of them to evaluate in their client code. This can be cumbersome when clients wish to support multiple versions of the API, but seems to be the approach used by most common REST APIs today.
With each of these approaches, it is also possible for the client or sever to express a level of intrusiveness. For example, a server might communicate "the VCH must be restarted to modify this" instead of simply "this cannot be modified in the current state." Similarly, a client might communicate "I want to make this change, even if it requires restarting the VCH" or "I want to make this change, even if it requires powering off all containers."
More complex logic can be introduced in the future, but in the interest of simplicity of the API and ease of implementation, it seems desirable to start simply with documenting the mutability rules in a human-readable way. As a next step, it may be useful to allow clients to communicate "I want to make this change, even if it requires restarting the VCH."
### Use of a query parameter for compute-resource
A datacenter represents an aggregation of resources within which a VCH may exist (spanning compute, storage, and networking), and is therefore included as a hierarchical path element. However, the compute-resource (i.e., cluster or resource pool) is, conceptually, a one-dimensional filter. Analogous filters for storage-resource or networking-resource could exist to identify VCHs using a particular datastore or network respectively. Expressing such filters as query parameters avoids the incorrect connotation of hierarchy that a path element would imply, and ensures composability.
(In the future, one could even imagine a more flexible filter mechanism that allowed for filtering on _any_ property of a VCH. Under such a model, compute-resource could be deprecated in favor of a more verbose expression, or viewed as a shorthand.)
### Use of a request body for delete
While the semantics for including a body with a delete request are not generally defined ([RFC 7231](https://tools.ietf.org/html/rfc7231#section-4.3.5)), there does not seem to be a better way to influence the behavior of the operation. (Use of query parameters or path segments would be contrary to the [stylistic pattern](https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm) of uniform interface.)
An alternative approach would be to model deletion as an action, but that may not be as intuitive for some consumers of the API.
## The REST API
### API Versioning
- [ ] To do
### Headers
Where possible, the standard `Authorization` header will be used for authentication.
Currently, only one scheme is supported:
* "basic", which will allow direct authentication with username and password
Eventually, another may be added:
* "Bearer", which will allow authentication via SAML, including from the H5 client plugin.
Additionally, a session ticket may be specified using the `X-VMWARE-TICKET` header.
### Resources
The base resource for all API operations will be `/container`.
Additional resources will exist to represent:
1. a vSphere target (ESX, vCenter, or Datacenter)
* The root resource followed by `/target/{target-network-address}`
* The target-network-address parameter must be a valid network address (FQDN or IP address) of a vSphere Server (ESX or vCenter)
* The root resource followed by `/target/{target-network-address}/datacenter/{datacenter-id}`
* The target-network-address parameter must be a valid network address (FQDN or IP address) of a vCenter Server
* The datacenter-id parameter must be an identifier for a resource of type Datacenter located within that vCenter Server
2. the collection of VCHs within (1)
* Any resource from (1) followed by `/vch`
3. a VCH within (2)
* Any resource from (2) followed by `/{vch-id}`
4. a host certificate within (3)
* Any resource from (3) followed by `/certificate`
5. log data within (3)
* Any resource from (3) followed by `/log`
Note: Given the use cases for this API, the exclusive use of identifiers (vs. names) seems acceptable. If necessary, lookup-by-name can be implemented using the `filter.names` query pattern from in the vSphere REST API.
### Query Parameters
For all requests to all resources except the root resource: An optional "thumbprint" parameter will be supported to allow an API client to indicate the expected thumbprint of the target vSphere system. This parameter need not be supplied if the target system has a certificate signed by a trusted certificate authority. There will be no equivalent to the "force" command-line argument.
- [ ] Figure out how certificate authority management will work. (Presumably there's something this can piggyback on.)
For many requests, as detailed below: An optional "compute-resource" parameter will be supported to scope a request to a particular compute resource within a vSphere target. This is equivalent to the "compute-resource" command-line argument except that it takes an identifier instead of a name. (For context, see [Use of a query parameter for compute-resource](#use-of-a-query-parameter-for-compute-resource).)
### Operations
#### Get version information
```
GET /container
GET /container/version
```
A `GET` request on the base resource will return a JSON object containing metadata. Initially, the only piece of metadata included will be the version number. Eventually, this may include a list of known appliance ISOs.
A `GET` request on the `version` sub-resource will return just the version.
- [ ] Should this also capture the required vSphere permissions for various operations? (If so, how?)
Corresponding CLI: `vic-machine-{darwin,linux,windows} version`
#### Display static message
```
GET /container/hello
```
A `GET` request on the `hello` resource will return a static "welcome" message for users who have been redirected to the server to accept SSL/TLS certificates..
Corresponding CLI: N/A
#### List VCHs
```
GET /container/target/{target-network-address}/[datacenter/{datacenter-id}]/vch?[thumbprint={thumbprint}]&[compute-resource={compute-resource}]
```
Making a `GET` request on `/vch` under a target and optionally a datacenter will return information about the VCHs on that target, in that datacenter.
- [ ] Pagination
Corresponding CLI: `vic-machine-{darwin,linux,windows} ls`
#### Create a VCH
```
POST /container/target/{target-network-address}/[datacenter/{datacenter-id}]/vch?[thumbprint={thumbprint}]
```
Making a `POST` request on `/vch` under a target and optionally a datacenter will create a VCH on that target, in that datacenter. Information about the VCH will be provided in the body of the request in a format similar to this.
Note that validation of the request occurs synchronously, with any errors being returned using an appropriate response code and status. Eventually, portions of the creation will proceed asynchronously, with errors being reported via a vSphere task that is returned once the synchronous validation is complete. (See "[VCH creation spawns a custom task](#vch-creation-spawns-a-custom-task)".)
Corresponding CLI: `vic-machine-{darwin,linux,windows} create`
#### Inspect a VCH
```
GET /container/target/{target-network-address}/[datacenter/{datacenter-id}]/vch/{vch-id}?[thumbprint={thumbprint}]
```
Making a `GET` request on a VCH resource will return information about the VCH. Information about the VCH will be provided in the body of the response in the same format as create.
Corresponding CLI: `vic-machine-{darwin,linux,windows} inspect`
#### Access the creation and reconfiguration log for a VCH
```
GET /container/target/{target-network-address}/[datacenter/{datacenter-id}]/vch/{vch-id}/log?[thumbprint={thumbprint}]
```
Making a `GET` request on `/log` under a VCH resource will return the contents of that VCH's log. The log is created during VCH creation and appended to during subsequent operations. This request is different than most others in that the return type is `text/plain`.
Note that log information will be _persisted_ in multiple files (a timestamped file for each creation/mutation operation). The contents of those files will be combined into a single stream for consumption. In the future, more granular access could be provided.
Corresponding CLI: N/A
#### Access the host certificate for a VCH
```
GET /container/target/{target-network-address}/[datacenter/{datacenter-id}]/vch/{vch-id}/certificate?[thumbprint={thumbprint}]
```
Making a `GET` request on `/certificate` under a VCH resource will return the certificate the VCH uses when acting as a server, which clients may wish to access to download and add to a trust store. This request is different than most others in that the return type is `application/x-pem-file`.
Corresponding CLI: N/A
#### Reconfigure a VCH
```
PUT /container/target/{target-network-address}/[datacenter/{datacenter-id}]/vch/{vch-id}?[thumbprint={thumbprint}]
PATCH /container/target/{target-network-address}/[datacenter/{datacenter-id}]/vch/{vch-id}?[thumbprint={thumbprint}]
```
Making a `PUT` request on a VCH resource will update that VCH's configuration. Information about the VCH will be provided in the body of the request in the same format as create.
In trying to strike a balance between the Robustness Principle and the Principle of Least Astonishment, we will allow for fields which cannot be modified to appear in the body of a `PUT` as long as the value of those fields match the current state of the object. This allows us to be relatively liberal in what we accept while avoiding the potential surprise of having edits dropped from the request. When the value of a field which cannot be modified does not match the current state, a 409 Conflict will be returned. To preserve the idempotency requirement for `PUT`, modifications to mutable portions of the body must not cause immutable portions of the body to change as a side-effect.
Making a `PATCH` request on a VCH resource (with a body as described in RFC 7396) will update a subset of that VCH's configuration.
As `PATCH` is an explicit request to update a set of fields, fields which cannot be modified must not appear in the body of the `PATCH` request, even if the modification would be a no-op.
Corresponding CLI: `vic-machine-{darwin,linux,windows} configure`
#### Delete a VCH
```
DELETE /container/target/{target-network-address}/[datacenter/{datacenter-id}]/vch/{vch-id}?[thumbprint={thumbprint}]
```
Making a `DELETE` request on a VCH resource will delete that VCH. By default, the VCH and any powered off containers will be deleted. A request body may be provided to indicate whether powered on containers and/or volume stores should be deleted.
Corresponding CLI: `vic-machine-{darwin,linux,windows} delete`
#### Upgrade a VCH
```
POST /container/target/{target-network-address}/[datacenter/{datacenter-id}]/vch/{vch-id}?action=upgrade&[thumbprint={thumbprint}]
```
Making a `POST` request on a VCH resource with an action of upgrade will initiate an upgrade of the VCH. The body of the request will be a JSON object containing the following optional properties: `bootstrap-iso` (a reference to a known bootstrap ISO on the OVA) and `rollback` (a boolean value).
Corresponding CLI: `vic-machine-{darwin,linux,windows} upgrade`
#### Debug a VCH
```
POST /container/target/{target-network-address}/[datacenter/{datacenter-id}]/vch/{vch-id}?action=debug&[thumbprint={thumbprint}]
```
Making a `POST` request on a VCH resource with an action of debug will modify the debug settings for the VCH. The body of the request will be a JSON object containing the following optional properties: `enable-ssh` (a boolean value), `authorized-key` (a string representation of a public key), `rootpw` (a string).
Corresponding CLI: `vic-machine-{darwin,linux,windows} debug`
#### View firewall rule settings
```
GET /container/target/{target-network-address}/[datacenter/{datacenter-id}]?[thumbprint={thumbprint}]&[compute-resource={compute-resource}]
```
Making a `GET` request on a vSphere target (with an optional datacenter and compute resource) will return information about the state of the host firewall on those resources. This allows a user to easily determine whether the hosts are in an appropriate state, and allows interfaces to display alerts when they are not. This also provides some measure of symmetry with the update firewall rules operation below.
Corresponding CLI: N/A
#### Update firewall rules
```
POST /container/target/{target-network-address}/[datacenter/{datacenter-id}]?action=firewall:[allow|deny]&[thumbprint={thumbprint}]&[compute-resource={compute-resource}]
```
Making a `POST` request on a vSphere target (with an optional datacenter and compute resource) with an action of `firewall:allow` or `firewall:deny` will update the host firewall on those resources.
Corresponding CLI: `vic-machine-{darwin,linux,windows} update`
## The WebSockets API
A WebSockets-based API will be used to provide streaming access to log data.
- [ ] Specify this.
## Proposed changes to existing functionality
### VCH creation spawns a custom task
Currently, VCH creation consists of three main steps:
1. Validation of the request and the state of the system.
2. A series of vSphere operations to create the VCH.
3. Starting the container VM and its services.
A single parent task could be created for the middle portion of this workflow, to be used as a starting point for a user wishing to query for the status of the creation operation.
When initiating VCH creation via the service, the first (validation) step would occur synchronously. Once the second (vSphere) step begins and the custom task is created, the API would return the handle to the vSphere task, which tracks the asynchronous portion of the request.
### Refactoring of `vic-machine`
The CLI and REST API should each be a thin layer around a common idiomatic Go API. That is, the CLI and REST API should be interaction and translation layers which do not include "business logic". Refactoring may be necessary to achieve this.
Eventually, the CLI could be be re-imagined and implemented as a client of the REST API. The advantages and disadvantages of such a change should be carefully considered.
### Deprecation of demo VCH installer
The demo VCH installer is currently shipped as a part of the OVA as a container with a web application that listens on port 1337. It provides users with a simple web interface to provision a VCH for demo/testing purposes by invoking the `vic-machine` CLI.
Re-implementing this UI using the interface using the new API would be feasible. However, it seems to make sense to consider the use cases for this application in the design of the vSphere H5 client plugin. And as there are no known backwards compatibility guarantees (or important use cases/workflows which would be impacted), it would make sense to deprecate and eventually remove the demo VCH installer once the vSphere H5 client plugin is available.
### Client certificate management using Admiral
Currently, `vic-machine` supports using user-specified client certificates or generating certificates for users to use. As an alternate model, it could delegate client certificate management to Admiral.
As an outline of how this might work:
* When Admiral is installed, it could generate a CA certificate and expose that certificate's public key (or make use of the vSphere CA).
* When a VCH is created, the Admiral CA certificate's public key could be supplied as the client certificate CA.
* When a user in created in Admiral, it could generate a client certificate signed by its client CA certificate (or sign a public key supplied by the user).
* When VCH is authenticating a request, the Admiral CA certificate's public key (from guest info) could be blended with dynamic configuration information from Admiral to limit access to only those users who have been granted access to the VCH's project in Admiral.
* When a user accesses a VCH, they would use the client certificate assigned by Admiral.
This provides several key benefits:
* Each user would have a single certificate which authorizes them to use all VCHs they have been granted access to.
* Operations performed on a VCH can be tied to a user.
* This would help address the lack of certificate revocation mechanisms and allow Admiral to be used to dynamically manage user authorization.
Most relevantly, and perhaps most importantly: this means that a VCH creation workflow does not require upload or download of client certificates, allowing for a simplified user experience.
The work that would be required for this would include:
1. Discovery/lookup of the Admiral CA certificate by `vic-machine-service`.
2. Certificate blending within the VCH. (Validating that the client certificates supplied by Admiral's dynamic configuration are signed by a client CA certificate configured on the VCH, and then limiting access to only those client certificates.)
3. Per-user client certificate generation (or signing) within Admiral.
4. Support for per-user client certificates in Admiral dynamic config.
5. To allow operations performed on a VCH to be tied to a user:
1. Expanding operation logging across the personality/portlayer boundary.
2. Logging the public details of the client certificate used for authorization.
### Reduce friction around use of operations credentials
Because a VCH must be able to interact with vSphere for a variety of operations on behalf of its users (which may not be vSphere administrators), it requires access to a set of vSphere credentials.
When using the `vic-machine` CLI, the default behavior is to persist the credentials which were used for the VCH creation itself. Alternatively, administrators may supply a set of "operations credentials".
This default option cannot be supported for API consumers using a session to authenticate, including the UI. As baseline functionality, we can require that operations credentials are supplied when authenticating with the API in this way. An administrator may chose to supply their own credentials as the operations credentials if they wish to mimic the CLI's default behavior.
Requiring administrators to supply operations credentials works, but is likely to be a point of friction. Beyond the basic inconvenience of treating more information as required in the wizard, an operations user needs to have a non-trivial set of permissions which introduces an additional source of user error.
Support for automatically creating a per-VCH solution user and assigning it the correct permissions seems like the best way to address these pain points. This design also has other advantages: solution users are the accepted mechanism for machine-to-machine interaction, solution users authenticate using a certificate instead of a password, and this would allow us to follow the principle of least privilege by default.
However, issuing requests as a solution user is believed to present significant technical challenges due to immature golang support for SOAP and SAML.
As an interim solution, we can:
1. Introduce logic to validate that supplied operations credentials have the necessary permissions, so that missing permissions can be identified early and clearly.
2. Allow administrators to have vic-machine attempt to automatically grant any missing permissions, so that permissions can be granted corrected quickly and easily.
3. Support automatically creating a non-solution user on behalf of the administrator, and granting it a minimal set of permissions, to further reduce the work required of the administrator.
Eventually, we can then:
4. Enhance the container operations code to support use of a solution user and update the automatic creation logic to create a solution user.
Each of these items builds on the previous, and all three act as building blocks towards #4. Each of these items would be useful for UI, API, and CLI.
### Associating contact information with a VCH
The original VCH configuration had an "environment contact" and an "administrative contact", which were stored using vSphere notes. This allowed a vSphere administrator to associate their contact information with a VCH, so that other administrators would know who to contact before making changes to the system.
This information could also be included on the vicadmin page, even for unauthenticated users, to enable users of all types to know who to talk to for help. (Perhaps the contact information should be free-form to allow for things like referring to a ticket-tracking system.)
- [ ] Customers with large environments must have a way to manage ownership information of entities. Can we learn from that?
## Testing
The service will require two types of tests:
1. Unit tests to verify functionality of handler logic.
2. End-to-end tests to verify the API functionality from a client's point of view (and to serve as the first "client", and as a secondary form of documentation).
Additionally, appropriate testing will be needed for each of the items in the "proposed changes to existing VIC functionality" section.

335
vendor/github.com/vmware/vic/doc/design/vic-machine/upgrade.md generated vendored Normal file
View File

@@ -0,0 +1,335 @@
# Upgrade
After installing a VCH, a VIC engine admin will need to manage the lifecycle of the VCH. One aspect of this lifecycle management is upgrading and patching. For our purposes, patching and upgrading will be treated the same.
The VIC engine product will release a new, complete, software download bundle for both patches and upgrades, versus, having a sparse patch or a separate upgrade model.
## Requirements
### Scope
vic-machine can upgrade VCH endpointVM no matter it's running or not, so vic-machine upgrade cannot rely on services running in VCH endpointVM.
For the VCH endpointVM, vic-machine upgrade will detect the version difference between the old VCH and itself, this includes detecting the guestinfo changes between the two versions. Then vic-machine upgrade will migrate the guestinfo metadata, and upload updated iso files. If anything incorrect happens during this process, vic-machine upgrade will revert back to old version and status. The endpoint API will have downtime during upgrade, but the running containers will not.
The containers managed by this VCH will be left running existing version. If the container is running, it will work well if it's not coupled with endpointVM through port-forwarding. After VCH is upgraded, the container management will be resumed even there is container configuration change. Portlayer will be backward compatible to make sure old version's container still work well.
But if there is security patch for container VM kernel or tether, user still need to recreate new container and then replace the old containers at once or gradually based on business requirement. vic-machine upgrade does not take care of container update, which is the same behavior with vanilla docker.
The exception is that in near future we might replace serial port with VMCI, to solve the VM vMotion issue. If after that change, portlayer lose backward compatibility, which means portlayer cannot talk with old container through tether, after endpointVM upgrade, part of container VM management function will loss, which means container list, inspect, network, volume, start/stop/remove still work, but container interaction, including attach, exec, log will not work.
Summary of upgrade requirement:
- Only upgrade VCH endpointVM
- Portlayer support backward compatibility
- If container functionality is impact, user is responsible to replace old container with new container after VCH endpointVM upgrade
- If there is vSphere credential changes, upgrade will replace old credentials with new one
### Version Difference
VIC version includes three parts, "release tag"-"build id"-"github commit hash". vic-machine upgrade will rely on build id to detect one version is newer or older than another one.
User need to provide newer binary with bigger build id, to upgrade existing VCH. This also requires our build system to keep increase build id number no matter any kind of system change.
Note: after introduced data migration between builds, we introduced one more internal used data migration version. That version is related to data migration plugin only, so will not be shown in vic-machine version. This will be described in data migration section.
### Impact
The VCH endpointVM (control plane) will have downtime during upgrade. Container lifecyles are not coupled with that of the endpointVM and, when using mapped vsphere networks (vic-machine --container-network argument) instead of port forwarding, the network data paths are not dependent on the endpointVM either.
There will be impact on container interaction while the endpointVM is down:
- no direct access to container logs
- no attach ability
- network outage for NAT based port forwarding
All of those facets should resume normal operation after upgrade is complete.
Note: Exception might happen if there is container communication changes. Additional operations are required to fix the problem.
### VCH status
vic-machine ls and inspect will show the VCH versions and upgrade status.
### One Step Roll Back
If anything wrong happens during upgrade, vic-machine will rollback VCH to original version and status.
And a little bit further, even vic-machine upgrade succeeds, user is still able to run vic-machine upgrade --rollback to do one step roll back, which means the endpointVM will be rollback to wherever version it was before this time's upgrade.
The benefit is that if the docker endpoint API does not actually work after upgrade (due to vic-machine upgrade mis judgement) or user appliacation still prefer old version's API for whatever reason, they still could get back to old working version easily.
Limitation:
- If there is new version container created after upgrade, rollback will fail. In that case, user should delete new container through docker CLI or portlayer API.
- Only one step roll-back is supported. As long as new upgrade is executed, no matter it's succeeds or not, the old version's rollback persisence will be removed.
### Resume Interrupted Upgrade
If upgrade failed, it will rollback to old version dynamically to make sure user application is not stopped due to upgrade failure. But there is one scenario still might break everything, that is user input Ctrl+C during upgrade, cause vic-machine upgrade is interrupte, it is not able to roll back everything it did, and then it is useful to resume this kind of operation from vic-machine.
Open Issue: if this is high priority user requirement?
### Downgrade
Verified docker-engine upgrade/downgrade, which works well between 1.11.2 and 1.12. The running container will be stopped after upgrade/downgrade, but is good to start again in new version. And all images and containers information is not lost. But think of the complexity to support version downgrade, we'll not go with this at this moment, only with a limited one step roll back.
### Internet connection
Internet connection is not required to upgrade/downgrade VCH, but newer version's binary should be available for vic-machine.
## Design/Implementation - Phase1
This section described the first simple implementation of upgrade. After this phase, user could upgrade VCH endpointVM from build to build, as long as there is no metadata changes, which means no changes in guestinfo, key value store and image metadata.
This code is already merged, and works for security patch update.
### Versioning
#### Freeze following attributes definition in VCH configuration and container VM configuration
- VCH configuration: ID, Name, Version
- Container VM serial port configuration
- Container VM log/debug file location
#### Configuration Version
Both VCH and container configuration will have version and the value is same with what vic-machine version command shown. After upgrade, VCH configuration version should be updated to new version, but container VM configuration version will still be old one.
New version's VCH will work with both old and new versions containers.
#### Embed iso file version
vic-machine need to identify the iso file version before and after upgrade. Two options here:
- Add version into iso file Primary Volume Descriptor, and read back from vic-machine
- Leverage iso file name including version, e.g. appliance-0.5.0.iso
vic-machine should check iso file version during deployment, and after upload to datastore, version should be appended to iso file name, to make sure mutilple iso files could co-exist in the same VCH, and used for different version's container.
Image file name should not be changed in datastore, cause the file path is used to create container VM. And then vic-machine will leverage file name for version checking during upgrade, to avoid download iso file from datastore back to where vic-machine is running.
Note: No feasible golang library found for this function, so will write our own library to read iso metadata.
#### Refactor current VCH configuration and container VM configuration structure
To make data migration easier, we'll need to refactor current VCH and container VM configuration structure.
VirtualContainerHostConfigSpec is too big, which includes everything in one structure. We're not be able to update part of configuration structure, it will help to update part of data with separated structures based on functionality, e.g. network, storage, and common attributes.
For ContainerVM configuration, there are a few attributes not used for container VM setup, e.g. in ExecutorConfig, Key, LayerID, RepoName, are all not container VM configuration related. Move out irrelated attributes can help the structure stability in the future.
### Transactional Operation/Roll Back
VM snapshot is used to keep current status for upgrade roll back. Before upgrade, vic-machine will create one snapshot of VCH endpointVM. If anything wrong happens, vic-machine can switch back to the pre-upgrade snapshot.
The one step roll-back will roll back the snapshot as well.
Note: VM snapshot does not persist data in serial port, as we will use datastore files through serial port for vch logs, after roll back, log files will have all error message during upgrade if VCH endpointVM is ever started with new configuration.
### Upgrade Workflow
Here is the upgrade workflow
- Find existing endpointVM
- Query VCH configuration
- Verify new version and existing VCH's version
- Cleanup snapshot and images created from last time's upgrade, if there is any. This step is to make sure the rollback after upgrade is not confused.
- Read back old VCH configuration, and migrate to new version. (data migration will be in phase 2 impl)
- Upload iso files
- Prepare combined endpointVM configuration, both hardware spec (ISO path) and the extraconfig portion (VCH version)
--- everything until this portion can be done without interrupting ongoing operation ---
- Snapshot endpointVM
- Poweroff endpointVM if it's not
- Update VCH configuration together with the migrated configuration data
- Reconfigure VCH endpointVM
- Power on endpointVM, and wait VCH service initialization
- Anything wrong in the above steps, roll back to upgrade snapshot
- Ensure endpointVM is powered on and initialized correctly after rollback to snapshot
- Cleanup env after failed upgrade (remove upgrade snapshot, remove uploaded iso files)
- Leave snapshot and old image files for one step roll-back
This ensures that a failure to upload ISOs for whatever reason is detected before we take down the existing version. It limits the failure modes after shutting down the endpointVM to:
1. failed to update endpointVM configuration
2. failed to power on the endpointVM (e.g. system resource constraints)
3. failed to boot endpointVM
4. failed to acquire network addresses (this has been seen in real world examples - we may wish to attempt to preserve/reuse the IPs the endpointVM had prior to update, which should still be present in the extraconfig)
5. failed to rebuild system state from vsphere infrastructure
## Design/Implementation - Phase2
Although we did lots of code refactor for VCH endpointVM configuration, it's still unavoidable to continue changing that structure. To make sure we don't break upgrade after GA, we need a solution for the data migration anyway.
- EndpointVM guestinfo, which is used to persist VCH endpointVM configuration
- Container guestinfo, which is used to persist container configuration
- KeyValue Store and image metadata, following information is persisted
- parentMap
- per-image metadata
- image layer cache
- network portlayer data
- EndpointVM log files
- vSphere Object Management Logic
### Data Migration Framework
Guestinfo is versioned by auto generated build number, git commit hash, build timestamp etc, which are all not controllable during development. So we'll add another sequential version for data migration version control only. For each version, there will have one data migration plugin registered into data migration framework.
Each time to migrate data, either for container or endpointVM, the framework will compare latest version with the existing version to generate data migration path, e.g. r2->r3->f4->latest or r4->latest. And then corresponding plugins will be executed sequentially to migrate data.
Container data and endpointVM data will be migrated separately, because they have different lifecycle. All the metadata in endpointVM, including guestinfo, keyvalue store, image metadata, log files are all part of endpointVM scope, which will be considered as one set and migrated at the same time.
As old container binary will not be updated by vic-machine upgrade, we'll not be able to write container data back to container VM guestinfo. In that case, we need to try to avoid container data change even there is version difference between VCH endpointVM and container. And the framework should explicitly provide methods to detect if there is data version difference.
Based on the design above, we had following migration framework interface:
```
// MigrateApplianceConfigure migrate VCH appliance configuration, including guestinfo, keyvaluestore, or any other kinds of change
// InIt accept VCH appliance guestinfo map, and return all configurations need to be made in guestinfo, keyvaluestore, and can have
// more kinds of change in the future. Each kind is one map key/value pair.
// If there is error returned, returned map might have half-migrated value, this is why we don't persist any data in plugin.
func MigrateApplianceConfigure(ctx context.Context, s *session.Session, conf map[string]string) (map[string]string, bool, error)
// MigrateContainerConfigure migrate container configuration
// Migrated data will be returned in map, and input object is not changed.
// If there is error returned, returned map might have half-migrated value.
func MigrateContainerConfigure(conf map[string]string) (map[string]string, bool, error)
func IsContainerDataOlder(conf map[string]string) (bool, error)
func IsApplianceDataOlder(conf map[string]string) (bool, error)
```
Here is the interface for plugin and plugin manager:
```
type DataMigration interface {
// Register plugin to data migration system
Register(version int, target string, plugin Plugin) error
// Migrate data with current version ID, return true if has any plugin executed
Migrate(ctx context.Context, s *session.Session, target string, currentVersion int, data interface{}) (int, error)
// GetLatestVersion return the latest plugin id for specified target
GetLatestVersion(target string) int
}
type Plugin interface {
Migrate(ctx context.Context, s *session.Session, data interface{}) error
}
```
Notes:
- Developers who change guestinfo, keyvalue store, image metadata, etc, will be the owner to develop migration plugin, and be responsible to increase data migration version.
- Each migration version should have one and only one corresponding plugin.
- If endpointVM configuration and container configuration are changed at the same time, two different plugins should be added, and registered to different plugin category.
- If both endpointVM configuration and keyvalue store are changed, two plugins are recommended as well.
Reminder: extraconfig package should always be backward compatible. If it breaks this assumption, upgrade is broken.
### EndpointVM Migration Process
vic-machine upgrade will upgrade endpointVM and data migration framework will be called to migrate data, which is described in phase1 workflow. There is no new command parameters added for data migration.
#### Guestinfo migration
The plugin should migrate data in memory and return changed value directly. No need to persist any data.
The difficulty is how to migrate secret data in guestinfo. The encryption key is persisted through raw guestinfo.ovfEnv variable, which is readable from in guest, but not through vSphere API, no matter it's VC or ESXi. Decrypt and re-encrypt secret data is necessary because otherwise if old vc login credential is changed, upgrade will always fail.
To solve this issue, we'll need to download endpointVM vmx file and read back guestinfo.ovfEnv value to decrypt encrypted data. In another word, this will be another variable dependent by upgrade framework, which cannot be changed in the future, otherwise, upgrade is broken.
#### KeyValue Store and Image metadata Migration
Different to guestinfo object, VIC engine will not change configuration from guestinfo to anywhere else anytime soon. But from start, keyvalue store persistent position is in argument. Right now, vic engine has a few datastore files for keyvalue store and image metadata, but those information is not configurable, which means hardcoded in portlayer.
Based on this idea, data migration framework will not assume where to load these data, and what version is that, instead, it assume all data for endpointVM is sharing the same version written in VM guestinfo. After compare the version difference, it will invoke plugins sequentially to migrate data including all endpointVM data. The plugin, for this kind of data, should read from datastore, change data and then commit back to datastore, or to anywhere defined in new version. So the input of keyvalue store plugin will be the endpointVM configuration data.
The problem of this solution is that each single plugin will persist its own change, not like the guestinfo update, which is migrated in memory and persisted by migration framework. So the rollback for guestinfo is easy, but not possible for keyvalue store, unless we have data roll back plugin mechanism.
The solution for this issue is to have one model to write key/value store and image metadata migration plugin. Every time we need something new and have to be migrated in new version, we should create new datastore files with a suffix matching the version number, e.g. metadata.v3, and then in the migration plugin, copy all existing data into new versioned files, and modify over there. The old versioned datastore files is not changed by the plugin, so even if eventually we dropped the upgrade, the old binary could still work with the old files without any problem.
#### EndpointVM log files
EndpointVM log files can be handled similar to KeyValue store.
#### vSphere Object Management Logic
There will have new vSphere API come up, so the logic to manage vsphere objects will be enhanced, for example, as Caglar said the vmdk can be managed directly in vSphere 6.5, instead of through vm operations in vSphere 6.0. We'll switch to these new interfaces for image and volume management sometime later.
vic-machine is not supposed to migrate old image data or volume data to new vmdk files, so portlayer will need to be backward compatible.
### Container Data Migration Process
vic-machine upgrade will leave container in old version, and do not update it even container is restarted. So portlayer will be responsible for container's backward compatibility. Here is how portlayer talk to old containers.
- Load container configuration
- Check if data migration is required. If yes, migrate data, and conver to new version's data structure in memory
- Read/write from/to new version's data for whatever container operations
- While need to write container configuration back to container VM guestinfo, check if data migration is done. If yes, skip writing.
Risks:
- As the new data is not written back, there will have few container information inconsistence
Currently, portlayer will write container status into guestinfo before start/stop, and tether will write some. If portlayer does not write this information because of data version mismatch, the container status will have problem if we still rely on portlayer to write it.
In the future, if portlayer add more functions to modify container, and those information is persisted in guestinfo, those change will not work for old container, for example, container rename.
- If portlayer and container communication channel is changed, from serial port to VMCI, and no backward compatibility support, container attach/exec/log etc., interaction related command will not work, but container image/volume/network/start/stop/remove should still work well.
In the above cases, user should think about replace old container with new created one. And if that happens, during endpointVM upgrade, vic-machine should have clear information to mention the container functional limitation after upgrade, and the suggested solution for it.
### One Step Roll Back
To support one step roll back, we'll have one new option in vic-machine upgrade command as following
```
vic-machine upgrade --rollback --<same other upgrade options>
```
Following is the workflow
- Find existing VCH endpointVM
- Check if upgrade snapshot available, and consistent with previous iso file version, if not, stop rollback
- Check if there is any newer version's container created already, if yes, print warning message and stop rollback
- Check if old iso files with same snapshot version still exist, if not, cannot rollback
- Snapshot current endpointVM with version in name
- Switch to old version's snapshot
- Power on endpointVM, and wait VCH service initialization
- Anything wrong in the above steps, roll back to new version's snapshot
- Ensure endpointVM is powered on and initialized after rollback to newer version
- Remove new version's upgrade snapshot if rollback failed
- Remove new version's upgrade snapshot and new version's iso files if rollback succeeds (after rollback, user is still able to reupgrade through the above operation, do not leave anything to avoid confusing)
### Resume Interrupted Upgrade
For user interrupted upgrade through Ctrl+C during upgrade is running, the VCH endpointVM is left in partial migrated status, which is hard to handle manually. Another option in vic-machine upgrade will help on this.
```
vic-machine upgrade --resume --<same other upgrade options>
```
Following is the workflow
- Check if there is one upgrade in progress, we'll rely on upgrade snapshot to see if the VCH is in upgrade. If anything is interrupted before upgrade snapshot is created, VCH endpointVM is not actually changed, user should use general upgrade command to continue.
- Check if docker API is available, if yes, last time's upgrade is already done, no need to resume (if vic-machine upgrade succeeds, but something else still does not work correctly, user should use one step rollback, instead of resume here)
- Check if there is any newer version's container created already, if yes, stop resume. Which means new VCH endpoint API already works, but occasionally break by something else, resume cannot help on that.
- Check if old iso files with same snapshot version still exist, if not, cannot resume
- Upload new iso files no matter they exist or not, to solve any potential iso file broken issue
- Switch to old version's snapshot
- Query endpointVM guestinfo, and migrate to new version.
- Power off endpointVM if it's not
- Update VCH configuration together with the migrated configuration data
- Reconfigure VCH endpointVM
- Power on endpointVM, and wait VCH service initialization
- Anything wrong in the above steps, roll back to upgrade snapshot
- Ensure endpointVM is powered on and initialized correctly after rollback to snapshot
- Cleanup env after failed upgrade (remove upgrade snapshot, remove uploaded iso files)
- Leave snapshot and old image files for one step roll-back
## Restrictions
User cannot run two upgrades for same VCH at the same time.
vic-machine will check if there is already another upgrade snapshot is created before it starts to create snapshot. But as create vsphere snapshot will take some time, e.g. one minute, if at this time, another upgrade process is started, it will start upgrade again cause the snapshot of previous task is not finished yet.
## Integration Test
### Upgrade test w/o data migration
CI test already have test to upgrade from previous release to latest version, and run regression test after upgrade. With data migration framework added, no new integration test is required. But for any data migration added in the future, upgrade test should cover the new features regression.
### One Step Roll Back
After upgrade, run one step roll back and check the old version's VCH. Following scenarios should be covered.
- Roll back works without new container created, and sample function works (do not run regression test for that might have new features not suppored in old VCH)
- Roll back failed for new container created
- Roll back works after removed new version's container, and sample function works
- After roll back, VCH old version is recovered, and new feature is not supported
### Upgrade Resume
During upgrade, break the upgrade process, and check if vic-machine can resume the upgrade. Following scenarios should be covered.
- Before snapshot is created, stop upgrade, resume does not work for no upgrade is found
- Upgrade succeeds, resume should not start for docker endpoint API is ready
- After snapshot is created, stop upgrade, and remove the old version iso files, resume should fail
- After snapshot is created, stop upgrade, and do not touch everything else, resume should work, regression passed
- After resume, one step roll back should still work

367
vendor/github.com/vmware/vic/doc/design/vic-machine/vic-machine.md generated vendored Normal file
View File

@@ -0,0 +1,367 @@
# On the care and feeding of VCHs - vic-machine
vic-machine is both the management client for Virtual Container Hosts and the mechanism by which they are initially deployed.
## Roles, responsibilities, and multi-tenacy models
This document notes three separate roles in the course of deploying and managing a VCH:
* vSphere Administrator (_viadmin_)
* VCH Administrator (_admin_)
* VCH User (_user_)
This workflow is the bridge between the infrastructure administration portion of an organisation and the users. The specific intent here is to allow each of the roles to operate at a level of detail appropriate to that role, with minimal impact or dependency on the others. The:
* _viadmin_ - identified by having administrative access to vSphere
- operates at a business decision level, mapping relative priority of projects and teams to accessible resources and permissible resource limits. Delegates usage authority of those resources to the _admin_ within specific limits enforced by vSphere.
* _admin_ - identified by delegated authority in the form of access to a signed VCH manifest file
- controls sub-division, if any, of assigned resources among finer-grained projects and teams
- deploys a VCH to manage a set of assigned resources. Delegates consumption decisions about those resources to _user_
* _user_ - identified by granted API access to a specific VCH
- controls the specifics of _what_ is done with the available resources, by way of an API client such as the Docker client
This model provides for a form of multi-tenancy, with the _viadmin_ able to specify a service account that will be configured with appropriate RBAC rules. It's unclear at this time if it's viable to create sub-users with further restricted RBAC rulesets, so the working assumption is that all sub-division performed by _admin_ operate with the same service account, and the sub-division is enforced by configuration of vSphere constructs such as resource pools, but without the authority isolation provided by different vSphere users. How a VCH prevents manipulation of those sub-division limits is covered in [the security architecture](security.md).
It is not intended that there be RBAC within a VCH at this time.
How this multi-tenacy model is used is left up to the business, but there are two primary models that we consider during development:
* team based - the VCH is assigned to a team for their use, potentially running a mix of independent workloads
* application based - a VCH is used as the management construct for a given appliancation, with all containers being portions of that app
## VCH manifest
Certain information is required to deploy a VCH, with the _viadmin_ and _admin_ each contributing portions of that data. The VCH manifest is the mechanism by which that cooperation happens, and is the token via which authority delegation from _viadmin_ to _admin_ occurs. Creation of a manifest is conceptually a compositing process with the following inputs and an immutable result:
1. input manifest (if omitted, this creates a new manifest from scratch)
2. a restriction set of some kind ( e.g. compute resource path, datastore prefix, registry whitelist)
- as a note, user credentials are also considered a restriction as they control permissible operations
The _viadmin_ uses vic-machine to create a _base manifest_ containing the following restrictions:
* target vSphere environment
* vSphere user and credential for created VCHs:
- must already exist, or
- must be created by vic-machine during manifest creation, or
- demand created during VCH deployment, requiring stored admin credentials:
* stored _viadmin_ credentials if necessary:
- encrypted credentials and [validating proxy](components.md#validating-proxy) URI, or
- unencrypted credentials (_viadmin_ and _admin_ roles are held by the same entity and manifests are stored securely)
The _base manifest_ is the minimum set of information necessary from the _viadmin_ role. There is additional information that will almost always be required for full function of a VCH:
* vSphere network to use for container network
- must already exist
* vSphere switch to use for container network:
- must already exist
- if switch, port group or network is demand created during VCH deployment, requiring stored admin credentials
- in an enterprise environment, creation of a portgroup without VXLAN or VLAN is unlikely to suffice, so pre-existing portgroup is highly recommended
To create a VCH, disambiguation of which resource to use from the available set is necessary. If there is no ambiguity, i.e only one of each, then this can be omitted:
* _client_ network - necessary for any access to the VCH
* default container network:
- _private_ port group or dedicated vSwitch, or
- IPAM config for _client_ network (defaults to DHCP)
* datastore paths for (may all be the same path):
- images
- containers
- volumes
## Usage Examples
These are various examples intended to illustrate how vic-machine is intended to be used. These are illustrations rather than the finalized naming so if something doesn't make sense ask about it.
### Fully specified command line
```
vic-machine create
-target=https://root@****:vcenter.fqdn/path/to/datacenter
-compute=mycluster/edw-pool/web/edw-web
-path=/folder/path/orthogonal/to/compute
-image-store=ds://datastore/containers/edw
-container-store=ds://datastore/edw/edw/web
-client-network="VM Network"
-management-network="management-net"
-volume-store=ds://edw-san/reports:reports
-volume-store=ds://edw-san/data:data
-container-network=private-vlan12d7f2:bridge
-container-network-ipam=private-vlan12d7f2:172.17.0.128-192/24,172.17.0.1
-container-network=edw-corp-net:backend
-alias=oracledb.edw.corp.net:db.backend
```
The first block of these options control the core configuration of the VCH:
* target - the destination ESX or VC datacenter
* compute - resource pool, host, cluster, or other compute boundary into which the VCH should be deployed.
* path - the folder path into which the VCH VMs will be placed - this is VC only
* image-store - the datastore location where images will be placed, whether pulled via `docker pull` or created via `docker commit`.
* container-store - the datastore location where containerVMs will be created. This does not have to be on the same datastore as the images, but both must be visible to all hosts on which containerVMs are to be created.
* client-network - the network on which users will connect to the VCH to issue DOCKER_API commands.
* management-network - the network providing access to a VMOMI SDK, in deployments where access to the management network is required.
The second block of options control how exisiting vSphere resources are presented to users of the VCH. These are specified as a `source:destination` mapping with vSphere identifier as the source; it is the viadmin role that is providing this data and mapping mydomain:targetdomain aligns with how people tend to think.
* -volume-store - datastore prefixes under which volumes can be created. These prefixes are mapped to labels that can be reference via the --opts mechanism when calling `docker volume create`
* -container-network - vSphere networks that should be exposed via `docker network` commands. As with the datastores this is a mapping from vSphere name to docker name. Ideally the docker name should express something about the purpose of presenting the network such as `internet`, `intranet`, or `databases`
* -container-network-ipam - this is an optional argument furnishing additional information for controlling IP address management on the network, in the form `ipaddress-range/mask,gateway`. If not specified DHCP is used.
* -alias - this allows containers to address a specific FQDN as if it were itself a container managed by the VCH, specified in the form of `FQDN:alias.network`.
### Manifest file - create
Note that the various resource paths end in a `/` - this signifies that it's acceptable for additional refinement to be provided when using this manifest file:
```
vic-machine manifest
-manifest=file::///home/joe/vch/edw.manifest
-target=https://root@****:vcenter.fqdn/path/to/datacenter
-compute=mycluster/edw-pool/
-image-store=ds://vsan-datastore/containers/edw
-container-store=ds://vsan-datastore/edw/edw/
-client-network="VM Network"
-management-network="management-net"
-volume-store=ds://edw-san/reports/:reports
-volume-store=ds://edw-san/data/:data
-container-network=private-vlan12d7f2:bridge
-container-network=edw-corp-net:backend
-container-network-ipam=edw-corp-net:10.118.78.128-192/24,10.118.78.1
-container-network=internet-corp-net:frontend
-alias=oracledb.edw.corp.net:db.backend
```
### Manifest file - use
When using a manifest file it's necessary to fully qualify resources that are left open (i.e. trailing `/` in the manifest). In the case of mappings where it is `source/:destination`, the destination label is used as the resource reference; this avoids a requirement to know the manifest path, although there is no intent at this time to obfuscate specifications beyond encrypting credentials and target URI.
```
vic-machine create
-manifest=file::///home/joe/vch/edw.manifest
-compute=webteam
-container-store=web
-volume-store=reports/web:report
-container-network=bridge
-name=web-team-private
```
The exact consumption mechanic of manifests is still quite uncertain. The example above is working on the premise that all mappings in the manifest are optional and are bound only if referenced, whether simply by label or as part of a refinement. The example above does not reference the _frontend_ network, so no containers can be attached to it.
### List existing VCHs
List the existing VCHs under the specified compute resource. The example below will list all VCHs in the target vCenter.
```
vic-machine ls
-FROM=https://root@****:vcenter/
ID Path Notes
vm-239 mycluster/edw-pool/web/edw-web Notes from the VCH config that get truncated after...
vm-2372 mycluster/random/someother Another VCH in the random pool
vm-15 mycluster/xyz VCH in the root of the cluster
```
The following example lists VCHs under a specific compute resource
```
vic-machine ls
-target=https://root@****:vcenter/example-datacenter
-compute=/mycluster/edw-pool/
```
The following example lists VCHs under a specific folder
```
vic-machine ls
-target=https://root@****:vcenter/example-datacenter
-path=/a/folder/path
```
The following example lists VCHs under a specific resource, described by manifest. Using the example above, this would list all VCHs under /example-datacenter/host/mycluster/Resources/edw-pool/
```
vic-machine ls
-manifest=file::///home/joe/vch/edw.manifest
```
## Inspect existing VCH
Inspect the configuration of an existing VCH by compute path:
```
vic-machine inspect
-target=https://root@****:vcenter/example-datacenter
-compute=host/mycluster/Resources/edw-pool/web-team-vch
```
Inspect the configuration of an existing VCH by folder path:
```
vic-machine inspect
-target=https://root@****:vcenter/example-datacenter
-path=/a/folder/path/web-team-vch
```
Inspect the configuration of an existing VCH by moref - I feel leaking vCenter abstractions is unavoidable if allowing any specifier other than full paths. In this case the identifier is part of target because it is sufficient to be unambiguous without datacenter/path pair:
```
vic-machine inspect
-target=https://root@****:vcenter/moref=vm-10324
```
## Updating an existing VCH configuration
## Implementation approach
Recommended initial approach is to have several components that get rolled into both vic-machine. Initial implementation should focus on:
* CLI parsing into config structs
* validation of config struct values
* creation of VCH from config struct
![vic-machine high level logic](images/vic-machine-high-level.png)
Components:
* command line argument parsing
- maps command line arguments to internal config data structures
- _package: main, path: cmd/vic-machine_
* validation of config structure values
- given an internal config data structure it's necessary to check the specifics against the target vSphere to ensure they are valid - past validity is no indicator of current validity
- this may also translate from symbolic names to morefs, resulting in a manifest that is resilient to name changes, but fragile across vSphere instances with identical naming schemes. This should be a user selectable behaviour, I prefer it on by default.
- _package: spec, path: lib/spec_
* creation of VCH from configuration
- when a manifest has been validated against a vSphere, this component creates the corresponding vSphere objects - this is primarily the VCH applianceVM, but may also include port groups and other objects.
- _package: management, path: install/management_
* [vmomi gateway](components.md#vmomi_gateway)
* manifest creation and consistency
- logic to validate that layered restrictions don't violate prior restrictions, e.g. refining [datastore1]/a/path to [datastore1]/a/path/to/vch is acceptable, but not to [datastore1]/a/second/path
- this should operate on the internal config data structures - probably a sliding window of two layers of the composite manifest each loaded into a config struct
- signing and signature validation
- _package: manifest, path: install/manifest_
* load/save manifest to/from internal config data structures
- this is the mapping from the config held in the composite manifest to the current end configuration, held in a serializable config structure
- signing and signature validation of manifest layers
- _package: manifest, path: install/manifest_
The reason for this breakdown is because some of these elements need to be duplicated in the:
* [validating proxy](components.md#validating-proxy):
- load/save manifest
- manifest consistency
- validation
- reification
- vmomi gateway
* applianceVM
- validation
- vmomi gateway
## =================================================
below this point is working notes.
## Installing - per vSphere target
### Inputs
1. vSphere SDK endpoint
2. vSphere administrative credentials
### Actions
1. deploy ESX agents
2. upload ISOs to common location
3. create custom tasks, alerts, and icons
4. create VCH tag (enable filtering of VCHs)
5. install UI plugin
## Installing - per VCH
### Inputs
* VCH user (existing or new) **
* resource lists:
- pool **
- imagestore datastore paths **
- container datastore paths **
- volume datastore paths (restriction)
- network mappings:
- one network minimum for VCH comms**
- other network mappings
* resource allotments:
- cpu
- memory
- network IO
- disk IO
- datastore quotas (per datastore path)
* certificates
- users - for access to VCH
- hosts - for container access to external hosts
- network - for VCH/container access to networks (gating proxies)
* registry lists
- whitelist
- blacklist
* default container resource reservations and limits *
* containerVM naming convention (displayName for vSphere) *
### Actions
Some of the elevated privilege operations could be delegated during self-provisioning to avoid manifestations of un-utilized authority, e.g. resource pool, user, and rbac entries for a potential but uncreated VCH. This delegation of higher authority requires additional care in the self-provisoning path.
### Requiring elevated privileges
1. create vSphere user for VCH
2. create RBAC entries for VCH resources - resource pool, datastores, networks, et al
3. obtain credentials for VCH user (e.g. SSO token)
* should be revokable
* should only have expiration date if no concern about clean VCH retirement
4. create and size VCH resource pool/vApp
* if vApp then should also configure the start/stop behaviours
* this may encompass disabling certain operations via the UI
5. place credentials in VCH applianceVM extraConfig
## Requiring VCH user privileges
1. validate supplied configuration
2. construct extraConfig/guestinfo configuration for applianceVM
3. create VCH applianceVM
* this may encompass disabling certain operations via the UI
4. upload ISOs if not shared
5. initialize applianceVM
At this point install transitions to managing - reporting VCH status from initial install is the same as reporting that information for any VCH regardless of age.
## Deleting - per vSphere target
### Inputs
1. vSphere SDK endpoint
2. vSphere administrative credentials
3. VCH identification
- VCH resource pool path in govc format, e.g. /ha-datacenter/vm
- VCH name
- VCH ID, which is the value returned by vic-machine ls
The VCH name and VCH resource pool path are identical to the value used in vic-machine create command. Which can uniquely identify one VCH instance.
The VCH ID is the value returned by vic-machine ls, which probably be the VM mob-id, query from VC or ESX.
Either VCH ID or the VCH resource pool path and VCH name should be specified.
### Actions
1. Get VCH VM
2. Read back VCH configuration from VM guestinfo
3. Delete following resources based on VCH configuration.
- Container VMs managed by the VCH
- Container datastore paths and resource pool path configured during installation will be used here, to detect if the VM belongs to this VCH. If yes, these VMs will be removed.
- Container VMs will be removed if they are in stopped status.
- If container VMs are in powered on state, delete will return failure if -force is not specified.
- If container VMs are in powered on state, and -force is specified, vic-machine delete will power off and remove them those VMs.
- volumes managed by the VCH
- -force option is required to delete volumes together with VCH uninstallation.
- Volume datastore path (sample: ds://datastore1/volume/vch1) configured during installation will be used here, to detect if the volumes are created by this VCH.
- If volume directory is empty, vic-machine delete will delete this directory as well, otherwise warning it.
- images managed by the VCH
- Images are shared by VCHs. Currently no reference or metadata to specify which image is referenced by which VCH, so for TP3, vic-machine delete will not touch images.
- vSphere networks created
- vic-machine create or VCH port-layer-server will create network, so vic-machine delete should delete any new networks. (Still need to confirm for how to get created networks)
- VCH specific metadata from vsphere objects if any
- Cause image will not be removed at this time, image metadata will not touched as well. But for container metadata, if any, will be deleted. (Need to finalize where it is)
- the resource pool and appliance VM
### Samples
```
vic-machine delete
--target root:password@192.168.1.1/dc1
--vch-path cluster/pool
--name vch1
--force
```
This command will delete VCH /dc1/host/cluster/pool/vch1 appliance, all containers VMs, networks and volumes created by this VCH.
## Managing a VCH
* report VCH status and information (API endpoint, log server, et al)
* update VCH configuration - implies possible restart of component
* shutdown/reboot VCH
* upgrade VCH - should have an entirely separate doc for this