d8dfun/virtual-kubelet
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Remove providers (#670)

Browse Source 
* Move all but mock provider out of tree

These have all been moved to repos under github.com/virtual-kubelet.

* Introduce a providers.Store

This essentially moves the the old register/ handling into a first class
object that can be controlled from the CLI rather than through build
tags deep in the code.

This actually would have made it a bit easier to build the provider
repos and makes the cmd/ code more re-usable.
This commit is contained in:
Brian Goff
2019-06-18 03:11:11 -07:00
committed by Pires
parent 9bcc381ca3
commit a00c2f4b8b
811 changed files with 159 additions and 362521 deletions
Download Patch File Download Diff File Expand all files Collapse all files

57
providers/alibabacloud/README.md
View File

@@ -1,57 +0,0 @@
# Alibaba Cloud ECI
<img src="eci.svg" width="200" height="200" />
Alibaba Cloud ECI(Elastic Container Instance) is a service that allow you run containers without having to manage servers or clusters.
You can find more infomation via [alibaba cloud ECI web portal](https://www.aliyun.com/product/eci)
## Alibaba Cloud ECI Virtual-Kubelet Provider
Alibaba ECI provider is an adapter to connect between k8s and ECI service to implement pod from k8s cluster on alibaba cloud platform
## Prerequisites
To using ECI service on alibaba cloud, you may need open ECI service on [web portal](https://www.aliyun.com/product/eci), and then the ECI service will be available
## Deployment of the ECI provider in your cluster
configure and launch virtual kubelet
```
export ECI_REGION=cn-hangzhou
export ECI_SECURITY_GROUP=sg-123
export ECI_VSWITCH=vsw-123
export ECI_ACCESS_KEY=123
export ECI_SECRET_KEY=123
VKUBELET_TAINT_KEY=alibabacloud.com/eci virtual-kubelet --provider alibabacloud
```
confirm the virtual kubelet is connected to k8s cluster
```
$kubectl get node
NAME STATUS ROLES AGE VERSION
cn-shanghai.i-uf69qodr5ntaxleqdhhk Ready <none> 1d v1.9.3
virtual-kubelet Ready agent 10s v1.8.3
```
## Schedule K8s Pod to ECI via virtual kubelet
You can assign pod to virtual kubelet via node-selector and toleration.
```
apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
nodeName: virtual-kubelet
containers:
- name: nginx
image: nginx
tolerations:
- key: alibabacloud.com/eci
operator: "Exists"
effect: NoSchedule
```
# Alibaba Cloud Serverless Kubernetes
Alibaba Cloud serverless kubernetes allows you to quickly create kubernetes container applications without
having to manage and maintain clusters and servers. It is based on ECI and fully compatible with the Kuberentes API.
You can find more infomation via [alibaba cloud serverless kubernetes product doc](https://www.alibabacloud.com/help/doc-detail/94078.htm)

56
providers/alibabacloud/config.go
View File

@@ -1,56 +0,0 @@
package alibabacloud
import (
"io"
"github.com/BurntSushi/toml"
"github.com/virtual-kubelet/virtual-kubelet/providers"
)
type providerConfig struct {
Region string
OperatingSystem string
CPU string
Memory string
Pods string
VSwitch string
SecureGroup string
ClusterName string
}
func (p *ECIProvider) loadConfig(r io.Reader) error {
var config providerConfig
if _, err := toml.DecodeReader(r, &config); err != nil {
return err
}
p.region = config.Region
if p.region == "" {
p.region = "cn-hangzhou"
}
p.vSwitch = config.VSwitch
p.secureGroup = config.SecureGroup
p.cpu = config.CPU
if p.cpu == "" {
p.cpu = "20"
}
p.memory = config.Memory
if p.memory == "" {
p.memory = "100Gi"
}
p.pods = config.Pods
if p.pods == "" {
p.pods = "20"
}
p.operatingSystem = config.OperatingSystem
if p.operatingSystem == "" {
p.operatingSystem = providers.OperatingSystemLinux
}
p.clusterName = config.ClusterName
if p.clusterName == "" {
p.clusterName = "default"
}
return nil
}

896
providers/alibabacloud/eci.go
View File

@@ -1,896 +0,0 @@
package alibabacloud
import (
"bytes"
"context"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"io/ioutil"
"os"
"strconv"
"strings"
"time"
"github.com/aliyun/alibaba-cloud-sdk-go/sdk/requests"
"github.com/virtual-kubelet/virtual-kubelet/errdefs"
"github.com/virtual-kubelet/virtual-kubelet/log"
"github.com/virtual-kubelet/virtual-kubelet/manager"
"github.com/virtual-kubelet/virtual-kubelet/node/api"
"github.com/virtual-kubelet/virtual-kubelet/providers/alibabacloud/eci"
v1 "k8s.io/api/core/v1"
k8serr "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/api/resource"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/types"
)
// The service account secret mount path.
const serviceAccountSecretMountPath = "/var/run/secrets/kubernetes.io/serviceaccount"
const podTagTimeFormat = "2006-01-02T15-04-05Z"
const timeFormat = "2006-01-02T15:04:05Z"
// ECIProvider implements the virtual-kubelet provider interface and communicates with Alibaba Cloud's ECI APIs.
type ECIProvider struct {
eciClient *eci.Client
resourceManager *manager.ResourceManager
resourceGroup string
region string
nodeName string
operatingSystem string
clusterName string
cpu string
memory string
pods string
internalIP string
daemonEndpointPort int32
secureGroup string
vSwitch string
}
// AuthConfig is the secret returned from an ImageRegistryCredential
type AuthConfig struct {
Username string `json:"username,omitempty"`
Password string `json:"password,omitempty"`
Auth string `json:"auth,omitempty"`
Email string `json:"email,omitempty"`
ServerAddress string `json:"serveraddress,omitempty"`
IdentityToken string `json:"identitytoken,omitempty"`
RegistryToken string `json:"registrytoken,omitempty"`
}
var validEciRegions = []string{
"cn-hangzhou",
"cn-shanghai",
"cn-beijing",
"us-west-1",
}
// isValidECIRegion checks to make sure we're using a valid ECI region
func isValidECIRegion(region string) bool {
regionLower := strings.ToLower(region)
regionTrimmed := strings.Replace(regionLower, " ", "", -1)
for _, validRegion := range validEciRegions {
if regionTrimmed == validRegion {
return true
}
}
return false
}
// NewECIProvider creates a new ECIProvider.
func NewECIProvider(config string, rm *manager.ResourceManager, nodeName, operatingSystem string, internalIP string, daemonEndpointPort int32) (*ECIProvider, error) {
var p ECIProvider
var err error
p.resourceManager = rm
if config != "" {
f, err := os.Open(config)
if err != nil {
return nil, err
}
defer f.Close()
if err := p.loadConfig(f); err != nil {
return nil, err
}
}
if r := os.Getenv("ECI_CLUSTER_NAME"); r != "" {
p.clusterName = r
}
if p.clusterName == "" {
p.clusterName = "default"
}
if r := os.Getenv("ECI_REGION"); r != "" {
p.region = r
}
if p.region == "" {
return nil, errors.New("Region can't be empty please set ECI_REGION\n")
}
if r := p.region; !isValidECIRegion(r) {
unsupportedRegionMessage := fmt.Sprintf("Region %s is invalid. Current supported regions are: %s",
r, strings.Join(validEciRegions, ", "))
return nil, errors.New(unsupportedRegionMessage)
}
var accessKey, secretKey string
if ak := os.Getenv("ECI_ACCESS_KEY"); ak != "" {
accessKey = ak
}
if sk := os.Getenv("ECI_SECRET_KEY"); sk != "" {
secretKey = sk
}
if sg := os.Getenv("ECI_SECURITY_GROUP"); sg != "" {
p.secureGroup = sg
}
if vsw := os.Getenv("ECI_VSWITCH"); vsw != "" {
p.vSwitch = vsw
}
if p.secureGroup == "" {
return nil, errors.New("secureGroup can't be empty\n")
}
if p.vSwitch == "" {
return nil, errors.New("vSwitch can't be empty\n")
}
p.eciClient, err = eci.NewClientWithAccessKey(p.region, accessKey, secretKey)
if err != nil {
return nil, err
}
p.cpu = "1000"
p.memory = "4Ti"
p.pods = "1000"
if cpuQuota := os.Getenv("ECI_QUOTA_CPU"); cpuQuota != "" {
p.cpu = cpuQuota
}
if memoryQuota := os.Getenv("ECI_QUOTA_MEMORY"); memoryQuota != "" {
p.memory = memoryQuota
}
if podsQuota := os.Getenv("ECI_QUOTA_POD"); podsQuota != "" {
p.pods = podsQuota
}
p.operatingSystem = operatingSystem
p.nodeName = nodeName
p.internalIP = internalIP
p.daemonEndpointPort = daemonEndpointPort
return &p, err
}
// CreatePod accepts a Pod definition and creates
// an ECI deployment
func (p *ECIProvider) CreatePod(ctx context.Context, pod *v1.Pod) error {
//Ignore daemonSet Pod
if pod != nil && pod.OwnerReferences != nil && len(pod.OwnerReferences) != 0 && pod.OwnerReferences[0].Kind == "DaemonSet" {
msg := fmt.Sprintf("Skip to create DaemonSet pod %q", pod.Name)
log.G(ctx).WithField("Method", "CreatePod").Info(msg)
return nil
}
request := eci.CreateCreateContainerGroupRequest()
request.RestartPolicy = string(pod.Spec.RestartPolicy)
// get containers
containers, err := p.getContainers(pod, false)
initContainers, err := p.getContainers(pod, true)
if err != nil {
return err
}
// get registry creds
creds, err := p.getImagePullSecrets(pod)
if err != nil {
return err
}
// get volumes
volumes, err := p.getVolumes(pod)
if err != nil {
return err
}
// assign all the things
request.Containers = containers
request.InitContainers = initContainers
request.Volumes = volumes
request.ImageRegistryCredentials = creds
CreationTimestamp := pod.CreationTimestamp.UTC().Format(podTagTimeFormat)
tags := []eci.Tag{
eci.Tag{Key: "ClusterName", Value: p.clusterName},
eci.Tag{Key: "NodeName", Value: p.nodeName},
eci.Tag{Key: "NameSpace", Value: pod.Namespace},
eci.Tag{Key: "PodName", Value: pod.Name},
eci.Tag{Key: "UID", Value: string(pod.UID)},
eci.Tag{Key: "CreationTimestamp", Value: CreationTimestamp},
}
ContainerGroupName := containerGroupName(pod)
request.Tags = tags
request.SecurityGroupId = p.secureGroup
request.VSwitchId = p.vSwitch
request.ContainerGroupName = ContainerGroupName
msg := fmt.Sprintf("CreateContainerGroup request %+v", request)
log.G(ctx).WithField("Method", "CreatePod").Info(msg)
response, err := p.eciClient.CreateContainerGroup(request)
if err != nil {
return err
}
msg = fmt.Sprintf("CreateContainerGroup successed. %s, %s, %s", response.RequestId, response.ContainerGroupId, ContainerGroupName)
log.G(ctx).WithField("Method", "CreatePod").Info(msg)
return nil
}
func containerGroupName(pod *v1.Pod) string {
return fmt.Sprintf("%s-%s", pod.Namespace, pod.Name)
}
// UpdatePod is a noop, ECI currently does not support live updates of a pod.
func (p *ECIProvider) UpdatePod(ctx context.Context, pod *v1.Pod) error {
return nil
}
// DeletePod deletes the specified pod out of ECI.
func (p *ECIProvider) DeletePod(ctx context.Context, pod *v1.Pod) error {
eciId := ""
for _, cg := range p.GetCgs() {
if getECITagValue(&cg, "PodName") == pod.Name && getECITagValue(&cg, "NameSpace") == pod.Namespace {
eciId = cg.ContainerGroupId
break
}
}
if eciId == "" {
return errdefs.NotFoundf("DeletePod can't find Pod %s-%s", pod.Namespace, pod.Name)
}
request := eci.CreateDeleteContainerGroupRequest()
request.ContainerGroupId = eciId
_, err := p.eciClient.DeleteContainerGroup(request)
return wrapError(err)
}
// GetPod returns a pod by name that is running inside ECI
// returns nil if a pod by that name is not found.
func (p *ECIProvider) GetPod(ctx context.Context, namespace, name string) (*v1.Pod, error) {
pods, err := p.GetPods(ctx)
if err != nil {
return nil, err
}
for _, pod := range pods {
if pod.Name == name && pod.Namespace == namespace {
return pod, nil
}
}
return nil, nil
}
// GetContainerLogs returns the logs of a pod by name that is running inside ECI.
func (p *ECIProvider) GetContainerLogs(ctx context.Context, namespace, podName, containerName string, opts api.ContainerLogOpts) (io.ReadCloser, error) {
eciId := ""
for _, cg := range p.GetCgs() {
if getECITagValue(&cg, "PodName") == podName && getECITagValue(&cg, "NameSpace") == namespace {
eciId = cg.ContainerGroupId
break
}
}
if eciId == "" {
return nil, errors.New(fmt.Sprintf("GetContainerLogs can't find Pod %s-%s", namespace, podName))
}
request := eci.CreateDescribeContainerLogRequest()
request.ContainerGroupId = eciId
request.ContainerName = containerName
request.Tail = requests.Integer(opts.Tail)
// get logs from cg
logContent := ""
retry := 10
for i := 0; i < retry; i++ {
response, err := p.eciClient.DescribeContainerLog(request)
if err != nil {
msg := fmt.Sprint("Error getting container logs, retrying")
log.G(ctx).WithField("Method", "GetContainerLogs").Info(msg)
time.Sleep(5000 * time.Millisecond)
} else {
logContent = response.Content
break
}
}
return ioutil.NopCloser(strings.NewReader(logContent)), nil
}
// Get full pod name as defined in the provider context
func (p *ECIProvider) GetPodFullName(namespace string, pod string) string {
return fmt.Sprintf("%s-%s", namespace, pod)
}
// RunInContainer executes a command in a container in the pod, copying data
// between in/out/err and the container's stdin/stdout/stderr.
func (p *ECIProvider) RunInContainer(ctx context.Context, namespace, podName, containerName string, cmd []string, attach api.AttachIO) error {
return nil
}
// GetPodStatus returns the status of a pod by name that is running inside ECI
// returns nil if a pod by that name is not found.
func (p *ECIProvider) GetPodStatus(ctx context.Context, namespace, name string) (*v1.PodStatus, error) {
pod, err := p.GetPod(ctx, namespace, name)
if err != nil {
return nil, err
}
if pod == nil {
return nil, nil
}
return &pod.Status, nil
}
func (p *ECIProvider) GetCgs() []eci.ContainerGroup {
cgs := make([]eci.ContainerGroup, 0)
request := eci.CreateDescribeContainerGroupsRequest()
for {
cgsResponse, err := p.eciClient.DescribeContainerGroups(request)
if err != nil || len(cgsResponse.ContainerGroups) == 0 {
break
}
request.NextToken = cgsResponse.NextToken
for _, cg := range cgsResponse.ContainerGroups {
if getECITagValue(&cg, "NodeName") != p.nodeName {
continue
}
cn := getECITagValue(&cg, "ClusterName")
if cn == "" {
cn = "default"
}
if cn != p.clusterName {
continue
}
cgs = append(cgs, cg)
}
if request.NextToken == "" {
break
}
}
return cgs
}
// GetPods returns a list of all pods known to be running within ECI.
func (p *ECIProvider) GetPods(ctx context.Context) ([]*v1.Pod, error) {
pods := make([]*v1.Pod, 0)
for _, cg := range p.GetCgs() {
c := cg
pod, err := containerGroupToPod(&c)
if err != nil {
msg := fmt.Sprint("error converting container group to pod", cg.ContainerGroupId, err)
log.G(context.TODO()).WithField("Method", "GetPods").Info(msg)
continue
}
pods = append(pods, pod)
}
return pods, nil
}
// Capacity returns a resource list containing the capacity limits set for ECI.
func (p *ECIProvider) Capacity(ctx context.Context) v1.ResourceList {
return v1.ResourceList{
"cpu": resource.MustParse(p.cpu),
"memory": resource.MustParse(p.memory),
"pods": resource.MustParse(p.pods),
}
}
// NodeConditions returns a list of conditions (Ready, OutOfDisk, etc), for updates to the node status
// within Kubernetes.
func (p *ECIProvider) NodeConditions(ctx context.Context) []v1.NodeCondition {
// TODO: Make these dynamic and augment with custom ECI specific conditions of interest
return []v1.NodeCondition{
{
Type: "Ready",
Status: v1.ConditionTrue,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletReady",
Message: "kubelet is ready.",
},
{
Type: "OutOfDisk",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletHasSufficientDisk",
Message: "kubelet has sufficient disk space available",
},
{
Type: "MemoryPressure",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletHasSufficientMemory",
Message: "kubelet has sufficient memory available",
},
{
Type: "DiskPressure",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletHasNoDiskPressure",
Message: "kubelet has no disk pressure",
},
{
Type: "NetworkUnavailable",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "RouteCreated",
Message: "RouteController created a route",
},
}
}
// NodeAddresses returns a list of addresses for the node status
// within Kubernetes.
func (p *ECIProvider) NodeAddresses(ctx context.Context) []v1.NodeAddress {
// TODO: Make these dynamic and augment with custom ECI specific conditions of interest
return []v1.NodeAddress{
{
Type: "InternalIP",
Address: p.internalIP,
},
}
}
// NodeDaemonEndpoints returns NodeDaemonEndpoints for the node status
// within Kubernetes.
func (p *ECIProvider) NodeDaemonEndpoints(ctx context.Context) *v1.NodeDaemonEndpoints {
return &v1.NodeDaemonEndpoints{
KubeletEndpoint: v1.DaemonEndpoint{
Port: p.daemonEndpointPort,
},
}
}
// OperatingSystem returns the operating system that was provided by the config.
func (p *ECIProvider) OperatingSystem() string {
return p.operatingSystem
}
func (p *ECIProvider) getImagePullSecrets(pod *v1.Pod) ([]eci.ImageRegistryCredential, error) {
ips := make([]eci.ImageRegistryCredential, 0, len(pod.Spec.ImagePullSecrets))
for _, ref := range pod.Spec.ImagePullSecrets {
secret, err := p.resourceManager.GetSecret(ref.Name, pod.Namespace)
if err != nil {
return ips, err
}
if secret == nil {
return nil, fmt.Errorf("error getting image pull secret")
}
// TODO: Check if secret type is v1.SecretTypeDockercfg and use DockerConfigKey instead of hardcoded value
// TODO: Check if secret type is v1.SecretTypeDockerConfigJson and use DockerConfigJsonKey to determine if it's in json format
// TODO: Return error if it's not one of these two types
switch secret.Type {
case v1.SecretTypeDockercfg:
ips, err = readDockerCfgSecret(secret, ips)
case v1.SecretTypeDockerConfigJson:
ips, err = readDockerConfigJSONSecret(secret, ips)
default:
return nil, fmt.Errorf("image pull secret type is not one of kubernetes.io/dockercfg or kubernetes.io/dockerconfigjson")
}
if err != nil {
return ips, err
}
}
return ips, nil
}
func readDockerCfgSecret(secret *v1.Secret, ips []eci.ImageRegistryCredential) ([]eci.ImageRegistryCredential, error) {
var err error
var authConfigs map[string]AuthConfig
repoData, ok := secret.Data[string(v1.DockerConfigKey)]
if !ok {
return ips, fmt.Errorf("no dockercfg present in secret")
}
err = json.Unmarshal(repoData, &authConfigs)
if err != nil {
return ips, fmt.Errorf("failed to unmarshal auth config %+v", err)
}
for server, authConfig := range authConfigs {
ips = append(ips, eci.ImageRegistryCredential{
Password: authConfig.Password,
Server: server,
UserName: authConfig.Username,
})
}
return ips, err
}
func readDockerConfigJSONSecret(secret *v1.Secret, ips []eci.ImageRegistryCredential) ([]eci.ImageRegistryCredential, error) {
var err error
repoData, ok := secret.Data[string(v1.DockerConfigJsonKey)]
if !ok {
return ips, fmt.Errorf("no dockerconfigjson present in secret")
}
var authConfigs map[string]map[string]AuthConfig
err = json.Unmarshal(repoData, &authConfigs)
if err != nil {
return ips, err
}
auths, ok := authConfigs["auths"]
if !ok {
return ips, fmt.Errorf("malformed dockerconfigjson in secret")
}
for server, authConfig := range auths {
ips = append(ips, eci.ImageRegistryCredential{
Password: authConfig.Password,
Server: server,
UserName: authConfig.Username,
})
}
return ips, err
}
func (p *ECIProvider) getContainers(pod *v1.Pod, init bool) ([]eci.CreateContainer, error) {
podContainers := pod.Spec.Containers
if init {
podContainers = pod.Spec.InitContainers
}
containers := make([]eci.CreateContainer, 0, len(podContainers))
for _, container := range podContainers {
c := eci.CreateContainer{
Name: container.Name,
Image: container.Image,
Commands: append(container.Command, container.Args...),
Ports: make([]eci.ContainerPort, 0, len(container.Ports)),
}
for _, p := range container.Ports {
c.Ports = append(c.Ports, eci.ContainerPort{
Port: requests.Integer(strconv.FormatInt(int64(p.ContainerPort), 10)),
Protocol: string(p.Protocol),
})
}
c.VolumeMounts = make([]eci.VolumeMount, 0, len(container.VolumeMounts))
for _, v := range container.VolumeMounts {
c.VolumeMounts = append(c.VolumeMounts, eci.VolumeMount{
Name: v.Name,
MountPath: v.MountPath,
ReadOnly: requests.Boolean(strconv.FormatBool(v.ReadOnly)),
})
}
c.EnvironmentVars = make([]eci.EnvironmentVar, 0, len(container.Env))
for _, e := range container.Env {
c.EnvironmentVars = append(c.EnvironmentVars, eci.EnvironmentVar{Key: e.Name, Value: e.Value})
}
cpuRequest := 1.00
if _, ok := container.Resources.Requests[v1.ResourceCPU]; ok {
cpuRequest = float64(container.Resources.Requests.Cpu().MilliValue()) / 1000.00
}
c.Cpu = requests.Float(fmt.Sprintf("%.3f", cpuRequest))
memoryRequest := 2.0
if _, ok := container.Resources.Requests[v1.ResourceMemory]; ok {
memoryRequest = float64(container.Resources.Requests.Memory().Value()) / 1024.0 / 1024.0 / 1024.0
}
c.Memory = requests.Float(fmt.Sprintf("%.3f", memoryRequest))
c.ImagePullPolicy = string(container.ImagePullPolicy)
c.WorkingDir = container.WorkingDir
containers = append(containers, c)
}
return containers, nil
}
func (p *ECIProvider) getVolumes(pod *v1.Pod) ([]eci.Volume, error) {
volumes := make([]eci.Volume, 0, len(pod.Spec.Volumes))
for _, v := range pod.Spec.Volumes {
// Handle the case for the EmptyDir.
if v.EmptyDir != nil {
volumes = append(volumes, eci.Volume{
Type: eci.VOL_TYPE_EMPTYDIR,
Name: v.Name,
EmptyDirVolumeEnable: requests.Boolean(strconv.FormatBool(true)),
})
continue
}
// Handle the case for the NFS.
if v.NFS != nil {
volumes = append(volumes, eci.Volume{
Type: eci.VOL_TYPE_NFS,
Name: v.Name,
NfsVolumeServer: v.NFS.Server,
NfsVolumePath: v.NFS.Path,
NfsVolumeReadOnly: requests.Boolean(strconv.FormatBool(v.NFS.ReadOnly)),
})
continue
}
// Handle the case for ConfigMap volume.
if v.ConfigMap != nil {
ConfigFileToPaths := make([]eci.ConfigFileToPath, 0)
configMap, err := p.resourceManager.GetConfigMap(v.ConfigMap.Name, pod.Namespace)
if v.ConfigMap.Optional != nil && !*v.ConfigMap.Optional && k8serr.IsNotFound(err) {
return nil, fmt.Errorf("ConfigMap %s is required by Pod %s and does not exist", v.ConfigMap.Name, pod.Name)
}
if configMap == nil {
continue
}
for k, v := range configMap.Data {
var b bytes.Buffer
enc := base64.NewEncoder(base64.StdEncoding, &b)
enc.Write([]byte(v))
ConfigFileToPaths = append(ConfigFileToPaths, eci.ConfigFileToPath{Path: k, Content: b.String()})
}
if len(ConfigFileToPaths) != 0 {
volumes = append(volumes, eci.Volume{
Type: eci.VOL_TYPE_CONFIGFILEVOLUME,
Name: v.Name,
ConfigFileToPaths: ConfigFileToPaths,
})
}
continue
}
if v.Secret != nil {
ConfigFileToPaths := make([]eci.ConfigFileToPath, 0)
secret, err := p.resourceManager.GetSecret(v.Secret.SecretName, pod.Namespace)
if v.Secret.Optional != nil && !*v.Secret.Optional && k8serr.IsNotFound(err) {
return nil, fmt.Errorf("Secret %s is required by Pod %s and does not exist", v.Secret.SecretName, pod.Name)
}
if secret == nil {
continue
}
for k, v := range secret.Data {
var b bytes.Buffer
enc := base64.NewEncoder(base64.StdEncoding, &b)
enc.Write(v)
ConfigFileToPaths = append(ConfigFileToPaths, eci.ConfigFileToPath{Path: k, Content: b.String()})
}
if len(ConfigFileToPaths) != 0 {
volumes = append(volumes, eci.Volume{
Type: eci.VOL_TYPE_CONFIGFILEVOLUME,
Name: v.Name,
ConfigFileToPaths: ConfigFileToPaths,
})
}
continue
}
// If we've made it this far we have found a volume type that isn't supported
return nil, fmt.Errorf("Pod %s requires volume %s which is of an unsupported type\n", pod.Name, v.Name)
}
return volumes, nil
}
func containerGroupToPod(cg *eci.ContainerGroup) (*v1.Pod, error) {
var podCreationTimestamp, containerStartTime metav1.Time
CreationTimestamp := getECITagValue(cg, "CreationTimestamp")
if CreationTimestamp != "" {
if t, err := time.Parse(podTagTimeFormat, CreationTimestamp); err == nil {
podCreationTimestamp = metav1.NewTime(t)
}
}
if t, err := time.Parse(timeFormat, cg.Containers[0].CurrentState.StartTime); err == nil {
containerStartTime = metav1.NewTime(t)
}
// Use the Provisioning State if it's not Succeeded,
// otherwise use the state of the instance.
eciState := cg.Status
containers := make([]v1.Container, 0, len(cg.Containers))
containerStatuses := make([]v1.ContainerStatus, 0, len(cg.Containers))
for _, c := range cg.Containers {
container := v1.Container{
Name: c.Name,
Image: c.Image,
Command: c.Commands,
Resources: v1.ResourceRequirements{
Requests: v1.ResourceList{
v1.ResourceCPU: resource.MustParse(fmt.Sprintf("%.2f", c.Cpu)),
v1.ResourceMemory: resource.MustParse(fmt.Sprintf("%.1fG", c.Memory)),
},
},
}
container.Resources.Limits = v1.ResourceList{
v1.ResourceCPU: resource.MustParse(fmt.Sprintf("%.2f", c.Cpu)),
v1.ResourceMemory: resource.MustParse(fmt.Sprintf("%.1fG", c.Memory)),
}
containers = append(containers, container)
containerStatus := v1.ContainerStatus{
Name: c.Name,
State: eciContainerStateToContainerState(c.CurrentState),
LastTerminationState: eciContainerStateToContainerState(c.PreviousState),
Ready: eciStateToPodPhase(c.CurrentState.State) == v1.PodRunning,
RestartCount: int32(c.RestartCount),
Image: c.Image,
ImageID: "",
ContainerID: getContainerID(cg.ContainerGroupId, c.Name),
}
// Add to containerStatuses
containerStatuses = append(containerStatuses, containerStatus)
}
pod := v1.Pod{
TypeMeta: metav1.TypeMeta{
Kind: "Pod",
APIVersion: "v1",
},
ObjectMeta: metav1.ObjectMeta{
Name: getECITagValue(cg, "PodName"),
Namespace: getECITagValue(cg, "NameSpace"),
ClusterName: getECITagValue(cg, "ClusterName"),
UID: types.UID(getECITagValue(cg, "UID")),
CreationTimestamp: podCreationTimestamp,
},
Spec: v1.PodSpec{
NodeName: getECITagValue(cg, "NodeName"),
Volumes: []v1.Volume{},
Containers: containers,
},
Status: v1.PodStatus{
Phase: eciStateToPodPhase(eciState),
Conditions: eciStateToPodConditions(eciState, podCreationTimestamp),
Message: "",
Reason: "",
HostIP: "",
PodIP: cg.IntranetIp,
StartTime: &containerStartTime,
ContainerStatuses: containerStatuses,
},
}
return &pod, nil
}
func getContainerID(cgID, containerName string) string {
if cgID == "" {
return ""
}
containerResourceID := fmt.Sprintf("%s/containers/%s", cgID, containerName)
h := sha256.New()
h.Write([]byte(strings.ToUpper(containerResourceID)))
hashBytes := h.Sum(nil)
return fmt.Sprintf("eci://%s", hex.EncodeToString(hashBytes))
}
func eciStateToPodPhase(state string) v1.PodPhase {
switch state {
case "Scheduling":
return v1.PodPending
case "ScheduleFailed":
return v1.PodFailed
case "Pending":
return v1.PodPending
case "Running":
return v1.PodRunning
case "Failed":
return v1.PodFailed
case "Succeeded":
return v1.PodSucceeded
}
return v1.PodUnknown
}
func eciStateToPodConditions(state string, transitionTime metav1.Time) []v1.PodCondition {
switch state {
case "Running", "Succeeded":
return []v1.PodCondition{
v1.PodCondition{
Type: v1.PodReady,
Status: v1.ConditionTrue,
LastTransitionTime: transitionTime,
}, v1.PodCondition{
Type: v1.PodInitialized,
Status: v1.ConditionTrue,
LastTransitionTime: transitionTime,
}, v1.PodCondition{
Type: v1.PodScheduled,
Status: v1.ConditionTrue,
LastTransitionTime: transitionTime,
},
}
}
return []v1.PodCondition{}
}
func eciContainerStateToContainerState(cs eci.ContainerState) v1.ContainerState {
t1, err := time.Parse(timeFormat, cs.StartTime)
if err != nil {
return v1.ContainerState{}
}
startTime := metav1.NewTime(t1)
// Handle the case where the container is running.
if cs.State == "Running" || cs.State == "Succeeded" {
return v1.ContainerState{
Running: &v1.ContainerStateRunning{
StartedAt: startTime,
},
}
}
t2, err := time.Parse(timeFormat, cs.FinishTime)
if err != nil {
return v1.ContainerState{}
}
finishTime := metav1.NewTime(t2)
// Handle the case where the container failed.
if cs.State == "Failed" || cs.State == "Canceled" {
return v1.ContainerState{
Terminated: &v1.ContainerStateTerminated{
ExitCode: int32(cs.ExitCode),
Reason: cs.State,
Message: cs.DetailStatus,
StartedAt: startTime,
FinishedAt: finishTime,
},
}
}
// Handle the case where the container is pending.
// Which should be all other eci states.
return v1.ContainerState{
Waiting: &v1.ContainerStateWaiting{
Reason: cs.State,
Message: cs.DetailStatus,
},
}
}
func getECITagValue(cg *eci.ContainerGroup, key string) string {
for _, tag := range cg.Tags {
if tag.Key == key {
return tag.Value
}
}
return ""
}

16
providers/alibabacloud/eci.svg
View File

@@ -1,16 +0,0 @@
<?xml version="1.0" encoding="utf-8"?>
<!-- Generator: Adobe Illustrator 21.0.0, SVG Export Plug-In . SVG Version: 6.00 Build 0) -->
<svg version="1.0" id="layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
viewBox="0 0 48 48" style="enable-background:new 0 0 48 48;" xml:space="preserve">
<style type="text/css">
.st0{fill:#00C1DE;}
</style>
<g>
<polygon class="st0" points="41.4,22.7 41.4,25 43.7,25.6 43.7,31.8 24,36.3 4.3,31.8 4.3,25.6 6.5,25 6.5,22.7 2,23.7 2,33.7
24,38.7 46,33.7 46,23.7 "/>
<polygon class="st0" points="38.1,10.6 24,8 9.9,10.6 24,12.9 "/>
<polygon class="st0" points="22.8,15.1 8.8,12.6 8.8,30.2 22.8,33.4 "/>
<path class="st0" d="M25.2,33.4l14-3.2V12.6l-14,2.5V33.4z M35.2,15.3l2-0.4v13.7l-2,0.5V15.3z M31.3,15.9l2-0.4v13.9l-2,0.5V15.9z
M27.5,16.6l2-0.4v14l-2,0.5V16.6z"/>
</g>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 878 B

6
providers/alibabacloud/eci.toml
View File

@@ -1,6 +0,0 @@
Region = "cn-hangzhou"
OperatingSystem = "Linux"
CPU = "20"
Memory = "100Gi"
Pods = "20"
ClusterName = "default"

81
providers/alibabacloud/eci/client.go
View File

@@ -1,81 +0,0 @@
package eci
//Licensed under the Apache License, Version 2.0 (the "License");
//you may not use this file except in compliance with the License.
//You may obtain a copy of the License at
//
//http://www.apache.org/licenses/LICENSE-2.0
//
//Unless required by applicable law or agreed to in writing, software
//distributed under the License is distributed on an "AS IS" BASIS,
//WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
//See the License for the specific language governing permissions and
//limitations under the License.
//
// Code generated by Alibaba Cloud SDK Code Generator.
// Changes may cause incorrect behavior and will be lost if the code is regenerated.
import (
"github.com/aliyun/alibaba-cloud-sdk-go/sdk"
"github.com/aliyun/alibaba-cloud-sdk-go/sdk/auth"
)
// Client is the sdk client struct, each func corresponds to an OpenAPI
type Client struct {
sdk.Client
}
// NewClient creates a sdk client with environment variables
func NewClient() (client *Client, err error) {
client = &Client{}
err = client.Init()
return
}
// NewClientWithOptions creates a sdk client with regionId/sdkConfig/credential
// this is the common api to create a sdk client
func NewClientWithOptions(regionId string, config *sdk.Config, credential auth.Credential) (client *Client, err error) {
client = &Client{}
err = client.InitWithOptions(regionId, config, credential)
return
}
// NewClientWithAccessKey is a shortcut to create sdk client with accesskey
// usage: https://help.aliyun.com/document_detail/66217.html
func NewClientWithAccessKey(regionId, accessKeyId, accessKeySecret string) (client *Client, err error) {
client = &Client{}
err = client.InitWithAccessKey(regionId, accessKeyId, accessKeySecret)
return
}
// NewClientWithStsToken is a shortcut to create sdk client with sts token
// usage: https://help.aliyun.com/document_detail/66222.html
func NewClientWithStsToken(regionId, stsAccessKeyId, stsAccessKeySecret, stsToken string) (client *Client, err error) {
client = &Client{}
err = client.InitWithStsToken(regionId, stsAccessKeyId, stsAccessKeySecret, stsToken)
return
}
// NewClientWithRamRoleArn is a shortcut to create sdk client with ram roleArn
// usage: https://help.aliyun.com/document_detail/66222.html
func NewClientWithRamRoleArn(regionId string, accessKeyId, accessKeySecret, roleArn, roleSessionName string) (client *Client, err error) {
client = &Client{}
err = client.InitWithRamRoleArn(regionId, accessKeyId, accessKeySecret, roleArn, roleSessionName)
return
}
// NewClientWithEcsRamRole is a shortcut to create sdk client with ecs ram role
// usage: https://help.aliyun.com/document_detail/66223.html
func NewClientWithEcsRamRole(regionId string, roleName string) (client *Client, err error) {
client = &Client{}
err = client.InitWithEcsRamRole(regionId, roleName)
return
}
// NewClientWithRsaKeyPair is a shortcut to create sdk client with rsa key pair
// attention: rsa key pair auth is only Japan regions available
func NewClientWithRsaKeyPair(regionId string, publicKeyId, privateKey string, sessionExpiration int) (client *Client, err error) {
client = &Client{}
err = client.InitWithRsaKeyPair(regionId, publicKeyId, privateKey, sessionExpiration)
return
}

138
providers/alibabacloud/eci/create_container_group.go
View File

@@ -1,138 +0,0 @@
package eci
//Licensed under the Apache License, Version 2.0 (the "License");
//you may not use this file except in compliance with the License.
//You may obtain a copy of the License at
//
//http://www.apache.org/licenses/LICENSE-2.0
//
//Unless required by applicable law or agreed to in writing, software
//distributed under the License is distributed on an "AS IS" BASIS,
//WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
//See the License for the specific language governing permissions and
//limitations under the License.
//
// Code generated by Alibaba Cloud SDK Code Generator.
// Changes may cause incorrect behavior and will be lost if the code is regenerated.
import (
"github.com/aliyun/alibaba-cloud-sdk-go/sdk/requests"
"github.com/aliyun/alibaba-cloud-sdk-go/sdk/responses"
)
// CreateContainerGroup invokes the eci.CreateContainerGroup API synchronously
// api document: https://help.aliyun.com/api/eci/createcontainergroup.html
func (client *Client) CreateContainerGroup(request *CreateContainerGroupRequest) (response *CreateContainerGroupResponse, err error) {
response = CreateCreateContainerGroupResponse()
err = client.DoAction(request, response)
return
}
// CreateContainerGroupWithChan invokes the eci.CreateContainerGroup API asynchronously
// api document: https://help.aliyun.com/api/eci/createcontainergroup.html
// asynchronous document: https://help.aliyun.com/document_detail/66220.html
func (client *Client) CreateContainerGroupWithChan(request *CreateContainerGroupRequest) (<-chan *CreateContainerGroupResponse, <-chan error) {
responseChan := make(chan *CreateContainerGroupResponse, 1)
errChan := make(chan error, 1)
err := client.AddAsyncTask(func() {
defer close(responseChan)
defer close(errChan)
response, err := client.CreateContainerGroup(request)
if err != nil {
errChan <- err
} else {
responseChan <- response
}
})
if err != nil {
errChan <- err
close(responseChan)
close(errChan)
}
return responseChan, errChan
}
// CreateContainerGroupWithCallback invokes the eci.CreateContainerGroup API asynchronously
// api document: https://help.aliyun.com/api/eci/createcontainergroup.html
// asynchronous document: https://help.aliyun.com/document_detail/66220.html
func (client *Client) CreateContainerGroupWithCallback(request *CreateContainerGroupRequest, callback func(response *CreateContainerGroupResponse, err error)) <-chan int {
result := make(chan int, 1)
err := client.AddAsyncTask(func() {
var response *CreateContainerGroupResponse
var err error
defer close(result)
response, err = client.CreateContainerGroup(request)
callback(response, err)
result <- 1
})
if err != nil {
defer close(result)
callback(nil, err)
result <- 0
}
return result
}
// CreateContainerGroupRequest is the request struct for api CreateContainerGroup
type CreateContainerGroupRequest struct {
*requests.RpcRequest
Containers []CreateContainer `position:"Query" name:"Container" type:"Repeated"`
InitContainers []CreateContainer `position:"Query" name:"InitContainer" type:"Repeated"`
ResourceOwnerId requests.Integer `position:"Query" name:"ResourceOwnerId"`
SecurityGroupId string `position:"Query" name:"SecurityGroupId"`
ImageRegistryCredentials []ImageRegistryCredential `position:"Query" name:"ImageRegistryCredential" type:"Repeated"`
Tags []Tag `position:"Query" name:"Tag" type:"Repeated"`
ResourceOwnerAccount string `position:"Query" name:"ResourceOwnerAccount"`
RestartPolicy string `position:"Query" name:"RestartPolicy"`
OwnerAccount string `position:"Query" name:"OwnerAccount"`
OwnerId requests.Integer `position:"Query" name:"OwnerId"`
VSwitchId string `position:"Query" name:"VSwitchId"`
Volumes []Volume `position:"Query" name:"Volume" type:"Repeated"`
ContainerGroupName string `position:"Query" name:"ContainerGroupName"`
ZoneId string `position:"Query" name:"ZoneId"`
}
type CreateContainer struct {
Name string `name:"Name"`
Image string `name:"Image"`
Memory requests.Float `name:"Memory"`
Cpu requests.Float `name:"Cpu"`
WorkingDir string `name:"WorkingDir"`
ImagePullPolicy string `name:"ImagePullPolicy"`
Commands []string `name:"Command" type:"Repeated"`
Args []string `name:"Arg" type:"Repeated"`
VolumeMounts []VolumeMount `name:"VolumeMount" type:"Repeated"`
Ports []ContainerPort `name:"Port" type:"Repeated"`
EnvironmentVars []EnvironmentVar `name:"EnvironmentVar" type:"Repeated"`
}
// CreateContainerGroupImageRegistryCredential is a repeated param struct in CreateContainerGroupRequest
type ImageRegistryCredential struct {
Server string `name:"Server"`
UserName string `name:"UserName"`
Password string `name:"Password"`
}
// CreateContainerGroupResponse is the response struct for api CreateContainerGroup
type CreateContainerGroupResponse struct {
*responses.BaseResponse
RequestId string
ContainerGroupId string
}
// CreateCreateContainerGroupRequest creates a request to invoke CreateContainerGroup API
func CreateCreateContainerGroupRequest() (request *CreateContainerGroupRequest) {
request = &CreateContainerGroupRequest{
RpcRequest: &requests.RpcRequest{},
}
request.InitWithApiInfo("Eci", "2018-08-08", "CreateContainerGroup", "eci", "openAPI")
return
}
// CreateCreateContainerGroupResponse creates a response to parse from CreateContainerGroup response
func CreateCreateContainerGroupResponse() (response *CreateContainerGroupResponse) {
response = &CreateContainerGroupResponse{
BaseResponse: &responses.BaseResponse{},
}
return
}

107
providers/alibabacloud/eci/delete_container_group.go
View File

@@ -1,107 +0,0 @@
package eci
//Licensed under the Apache License, Version 2.0 (the "License");
//you may not use this file except in compliance with the License.
//You may obtain a copy of the License at
//
//http://www.apache.org/licenses/LICENSE-2.0
//
//Unless required by applicable law or agreed to in writing, software
//distributed under the License is distributed on an "AS IS" BASIS,
//WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
//See the License for the specific language governing permissions and
//limitations under the License.
//
// Code generated by Alibaba Cloud SDK Code Generator.
// Changes may cause incorrect behavior and will be lost if the code is regenerated.
import (
"github.com/aliyun/alibaba-cloud-sdk-go/sdk/requests"
"github.com/aliyun/alibaba-cloud-sdk-go/sdk/responses"
)
// DeleteContainerGroup invokes the eci.DeleteContainerGroup API synchronously
// api document: https://help.aliyun.com/api/eci/deletecontainergroup.html
func (client *Client) DeleteContainerGroup(request *DeleteContainerGroupRequest) (response *DeleteContainerGroupResponse, err error) {
response = CreateDeleteContainerGroupResponse()
err = client.DoAction(request, response)
return
}
// DeleteContainerGroupWithChan invokes the eci.DeleteContainerGroup API asynchronously
// api document: https://help.aliyun.com/api/eci/deletecontainergroup.html
// asynchronous document: https://help.aliyun.com/document_detail/66220.html
func (client *Client) DeleteContainerGroupWithChan(request *DeleteContainerGroupRequest) (<-chan *DeleteContainerGroupResponse, <-chan error) {
responseChan := make(chan *DeleteContainerGroupResponse, 1)
errChan := make(chan error, 1)
err := client.AddAsyncTask(func() {
defer close(responseChan)
defer close(errChan)
response, err := client.DeleteContainerGroup(request)
if err != nil {
errChan <- err
} else {
responseChan <- response
}
})
if err != nil {
errChan <- err
close(responseChan)
close(errChan)
}
return responseChan, errChan
}
// DeleteContainerGroupWithCallback invokes the eci.DeleteContainerGroup API asynchronously
// api document: https://help.aliyun.com/api/eci/deletecontainergroup.html
// asynchronous document: https://help.aliyun.com/document_detail/66220.html
func (client *Client) DeleteContainerGroupWithCallback(request *DeleteContainerGroupRequest, callback func(response *DeleteContainerGroupResponse, err error)) <-chan int {
result := make(chan int, 1)
err := client.AddAsyncTask(func() {
var response *DeleteContainerGroupResponse
var err error
defer close(result)
response, err = client.DeleteContainerGroup(request)
callback(response, err)
result <- 1
})
if err != nil {
defer close(result)
callback(nil, err)
result <- 0
}
return result
}
// DeleteContainerGroupRequest is the request struct for api DeleteContainerGroup
type DeleteContainerGroupRequest struct {
*requests.RpcRequest
ResourceOwnerId requests.Integer `position:"Query" name:"ResourceOwnerId"`
ContainerGroupId string `position:"Query" name:"ContainerGroupId"`
ResourceOwnerAccount string `position:"Query" name:"ResourceOwnerAccount"`
OwnerAccount string `position:"Query" name:"OwnerAccount"`
OwnerId requests.Integer `position:"Query" name:"OwnerId"`
}
// DeleteContainerGroupResponse is the response struct for api DeleteContainerGroup
type DeleteContainerGroupResponse struct {
*responses.BaseResponse
RequestId string `json:"RequestId" xml:"RequestId"`
}
// CreateDeleteContainerGroupRequest creates a request to invoke DeleteContainerGroup API
func CreateDeleteContainerGroupRequest() (request *DeleteContainerGroupRequest) {
request = &DeleteContainerGroupRequest{
RpcRequest: &requests.RpcRequest{},
}
request.InitWithApiInfo("Eci", "2018-08-08", "DeleteContainerGroup", "eci", "openAPI")
return
}
// CreateDeleteContainerGroupResponse creates a response to parse from DeleteContainerGroup response
func CreateDeleteContainerGroupResponse() (response *DeleteContainerGroupResponse) {
response = &DeleteContainerGroupResponse{
BaseResponse: &responses.BaseResponse{},
}
return
}

122
providers/alibabacloud/eci/describe_container_groups.go
View File

@@ -1,122 +0,0 @@
package eci
//Licensed under the Apache License, Version 2.0 (the "License");
//you may not use this file except in compliance with the License.
//You may obtain a copy of the License at
//
//http://www.apache.org/licenses/LICENSE-2.0
//
//Unless required by applicable law or agreed to in writing, software
//distributed under the License is distributed on an "AS IS" BASIS,
//WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
//See the License for the specific language governing permissions and
//limitations under the License.
//
// Code generated by Alibaba Cloud SDK Code Generator.
// Changes may cause incorrect behavior and will be lost if the code is regenerated.
import (
"github.com/aliyun/alibaba-cloud-sdk-go/sdk/requests"
"github.com/aliyun/alibaba-cloud-sdk-go/sdk/responses"
)
// DescribeContainerGroups invokes the eci.DescribeContainerGroups API synchronously
// api document: https://help.aliyun.com/api/eci/describecontainergroups.html
func (client *Client) DescribeContainerGroups(request *DescribeContainerGroupsRequest) (response *DescribeContainerGroupsResponse, err error) {
response = CreateDescribeContainerGroupsResponse()
err = client.DoAction(request, response)
return
}
// DescribeContainerGroupsWithChan invokes the eci.DescribeContainerGroups API asynchronously
// api document: https://help.aliyun.com/api/eci/describecontainergroups.html
// asynchronous document: https://help.aliyun.com/document_detail/66220.html
func (client *Client) DescribeContainerGroupsWithChan(request *DescribeContainerGroupsRequest) (<-chan *DescribeContainerGroupsResponse, <-chan error) {
responseChan := make(chan *DescribeContainerGroupsResponse, 1)
errChan := make(chan error, 1)
err := client.AddAsyncTask(func() {
defer close(responseChan)
defer close(errChan)
response, err := client.DescribeContainerGroups(request)
if err != nil {
errChan <- err
} else {
responseChan <- response
}
})
if err != nil {
errChan <- err
close(responseChan)
close(errChan)
}
return responseChan, errChan
}
// DescribeContainerGroupsWithCallback invokes the eci.DescribeContainerGroups API asynchronously
// api document: https://help.aliyun.com/api/eci/describecontainergroups.html
// asynchronous document: https://help.aliyun.com/document_detail/66220.html
func (client *Client) DescribeContainerGroupsWithCallback(request *DescribeContainerGroupsRequest, callback func(response *DescribeContainerGroupsResponse, err error)) <-chan int {
result := make(chan int, 1)
err := client.AddAsyncTask(func() {
var response *DescribeContainerGroupsResponse
var err error
defer close(result)
response, err = client.DescribeContainerGroups(request)
callback(response, err)
result <- 1
})
if err != nil {
defer close(result)
callback(nil, err)
result <- 0
}
return result
}
// DescribeContainerGroupsRequest is the request struct for api DescribeContainerGroups
type DescribeContainerGroupsRequest struct {
*requests.RpcRequest
ResourceOwnerId requests.Integer `position:"Query" name:"ResourceOwnerId"`
NextToken string `position:"Query" name:"NextToken"`
Limit requests.Integer `position:"Query" name:"Limit"`
Tags *[]DescribeContainerGroupsTag `position:"Query" name:"Tag" type:"Repeated"`
ContainerGroupId string `position:"Query" name:"ContainerGroupId"`
ResourceOwnerAccount string `position:"Query" name:"ResourceOwnerAccount"`
OwnerAccount string `position:"Query" name:"OwnerAccount"`
OwnerId requests.Integer `position:"Query" name:"OwnerId"`
VSwitchId string `position:"Query" name:"VSwitchId"`
ContainerGroupName string `position:"Query" name:"ContainerGroupName"`
ZoneId string `position:"Query" name:"ZoneId"`
}
// DescribeContainerGroupsTag is a repeated param struct in DescribeContainerGroupsRequest
type DescribeContainerGroupsTag struct {
Key string `name:"Key"`
Value string `name:"Value"`
}
// DescribeContainerGroupsResponse is the response struct for api DescribeContainerGroups
type DescribeContainerGroupsResponse struct {
*responses.BaseResponse
RequestId string `json:"RequestId" xml:"RequestId"`
NextToken string `json:"NextToken" xml:"NextToken"`
TotalCount int `json:"TotalCount" xml:"TotalCount"`
ContainerGroups []ContainerGroup `json:"ContainerGroups" xml:"ContainerGroups"`
}
// CreateDescribeContainerGroupsRequest creates a request to invoke DescribeContainerGroups API
func CreateDescribeContainerGroupsRequest() (request *DescribeContainerGroupsRequest) {
request = &DescribeContainerGroupsRequest{
RpcRequest: &requests.RpcRequest{},
}
request.InitWithApiInfo("Eci", "2018-08-08", "DescribeContainerGroups", "eci", "openAPI")
return
}
// CreateDescribeContainerGroupsResponse creates a response to parse from DescribeContainerGroups response
func CreateDescribeContainerGroupsResponse() (response *DescribeContainerGroupsResponse) {
response = &DescribeContainerGroupsResponse{
BaseResponse: &responses.BaseResponse{},
}
return
}

112
providers/alibabacloud/eci/describe_container_log.go
View File

@@ -1,112 +0,0 @@
package eci
//Licensed under the Apache License, Version 2.0 (the "License");
//you may not use this file except in compliance with the License.
//You may obtain a copy of the License at
//
//http://www.apache.org/licenses/LICENSE-2.0
//
//Unless required by applicable law or agreed to in writing, software
//distributed under the License is distributed on an "AS IS" BASIS,
//WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
//See the License for the specific language governing permissions and
//limitations under the License.
//
// Code generated by Alibaba Cloud SDK Code Generator.
// Changes may cause incorrect behavior and will be lost if the code is regenerated.
import (
"github.com/aliyun/alibaba-cloud-sdk-go/sdk/requests"
"github.com/aliyun/alibaba-cloud-sdk-go/sdk/responses"
)
// DescribeContainerLog invokes the eci.DescribeContainerLog API synchronously
// api document: https://help.aliyun.com/api/eci/describecontainerlog.html
func (client *Client) DescribeContainerLog(request *DescribeContainerLogRequest) (response *DescribeContainerLogResponse, err error) {
response = CreateDescribeContainerLogResponse()
err = client.DoAction(request, response)
return
}
// DescribeContainerLogWithChan invokes the eci.DescribeContainerLog API asynchronously
// api document: https://help.aliyun.com/api/eci/describecontainerlog.html
// asynchronous document: https://help.aliyun.com/document_detail/66220.html
func (client *Client) DescribeContainerLogWithChan(request *DescribeContainerLogRequest) (<-chan *DescribeContainerLogResponse, <-chan error) {
responseChan := make(chan *DescribeContainerLogResponse, 1)
errChan := make(chan error, 1)
err := client.AddAsyncTask(func() {
defer close(responseChan)
defer close(errChan)
response, err := client.DescribeContainerLog(request)
if err != nil {
errChan <- err
} else {
responseChan <- response
}
})
if err != nil {
errChan <- err
close(responseChan)
close(errChan)
}
return responseChan, errChan
}
// DescribeContainerLogWithCallback invokes the eci.DescribeContainerLog API asynchronously
// api document: https://help.aliyun.com/api/eci/describecontainerlog.html
// asynchronous document: https://help.aliyun.com/document_detail/66220.html
func (client *Client) DescribeContainerLogWithCallback(request *DescribeContainerLogRequest, callback func(response *DescribeContainerLogResponse, err error)) <-chan int {
result := make(chan int, 1)
err := client.AddAsyncTask(func() {
var response *DescribeContainerLogResponse
var err error
defer close(result)
response, err = client.DescribeContainerLog(request)
callback(response, err)
result <- 1
})
if err != nil {
defer close(result)
callback(nil, err)
result <- 0
}
return result
}
// DescribeContainerLogRequest is the request struct for api DescribeContainerLog
type DescribeContainerLogRequest struct {
*requests.RpcRequest
ResourceOwnerId requests.Integer `position:"Query" name:"ResourceOwnerId"`
ContainerName string `position:"Query" name:"ContainerName"`
StartTime string `position:"Query" name:"StartTime"`
ContainerGroupId string `position:"Query" name:"ContainerGroupId"`
ResourceOwnerAccount string `position:"Query" name:"ResourceOwnerAccount"`
Tail requests.Integer `position:"Query" name:"Tail"`
OwnerAccount string `position:"Query" name:"OwnerAccount"`
OwnerId requests.Integer `position:"Query" name:"OwnerId"`
}
// DescribeContainerLogResponse is the response struct for api DescribeContainerLog
type DescribeContainerLogResponse struct {
*responses.BaseResponse
RequestId string `json:"RequestId" xml:"RequestId"`
ContainerName string `json:"ContainerName" xml:"ContainerName"`
Content string `json:"Content" xml:"Content"`
}
// CreateDescribeContainerLogRequest creates a request to invoke DescribeContainerLog API
func CreateDescribeContainerLogRequest() (request *DescribeContainerLogRequest) {
request = &DescribeContainerLogRequest{
RpcRequest: &requests.RpcRequest{},
}
request.InitWithApiInfo("Eci", "2018-08-08", "DescribeContainerLog", "eci", "openAPI")
return
}
// CreateDescribeContainerLogResponse creates a response to parse from DescribeContainerLog response
func CreateDescribeContainerLogResponse() (response *DescribeContainerLogResponse) {
response = &DescribeContainerLogResponse{
BaseResponse: &responses.BaseResponse{},
}
return
}

22
providers/alibabacloud/eci/struct_config_file_to_path.go
View File

@@ -1,22 +0,0 @@
package eci
//Licensed under the Apache License, Version 2.0 (the "License");
//you may not use this file except in compliance with the License.
//You may obtain a copy of the License at
//
//http://www.apache.org/licenses/LICENSE-2.0
//
//Unless required by applicable law or agreed to in writing, software
//distributed under the License is distributed on an "AS IS" BASIS,
//WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
//See the License for the specific language governing permissions and
//limitations under the License.
//
// Code generated by Alibaba Cloud SDK Code Generator.
// Changes may cause incorrect behavior and will be lost if the code is regenerated.
// ConfigFileVolumeConfigFileToPath is a nested struct in eci response
type ConfigFileToPath struct {
Content string `name:"Content"`
Path string `name:"Path"`
}

34
providers/alibabacloud/eci/struct_container.go
View File

@@ -1,34 +0,0 @@
package eci
//Licensed under the Apache License, Version 2.0 (the "License");
//you may not use this file except in compliance with the License.
//You may obtain a copy of the License at
//
//http://www.apache.org/licenses/LICENSE-2.0
//
//Unless required by applicable law or agreed to in writing, software
//distributed under the License is distributed on an "AS IS" BASIS,
//WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
//See the License for the specific language governing permissions and
//limitations under the License.
//
// Code generated by Alibaba Cloud SDK Code Generator.
// Changes may cause incorrect behavior and will be lost if the code is regenerated.
// Container is a nested struct in eci response
type Container struct {
Name string `json:"Name" xml:"Name" `
Image string `json:"Image" xml:"Image"`
Memory float64 `json:"Memory" xml:"Memory"`
Cpu float64 `json:"Cpu" xml:"Cpu"`
RestartCount int `json:"RestartCount" xml:"RestartCount"`
WorkingDir string `json:"WorkingDir" xml:"WorkingDir"`
ImagePullPolicy string `json:"ImagePullPolicy" xml:"ImagePullPolicy"`
Commands []string `json:"Commands" xml:"Commands"`
Args []string `json:"Args" xml:"Args"`
PreviousState ContainerState `json:"PreviousState" xml:"PreviousState"`
CurrentState ContainerState `json:"CurrentState" xml:"CurrentState"`
VolumeMounts []VolumeMount `json:"VolumeMounts" xml:"VolumeMounts"`
Ports []ContainerPort `json:"Ports" xml:"Ports"`
EnvironmentVars []EnvironmentVar `json:"EnvironmentVars" xml:"EnvironmentVars"`
}

38
providers/alibabacloud/eci/struct_container_group.go
View File

@@ -1,38 +0,0 @@
package eci
//Licensed under the Apache License, Version 2.0 (the "License");
//you may not use this file except in compliance with the License.
//You may obtain a copy of the License at
//
//http://www.apache.org/licenses/LICENSE-2.0
//
//Unless required by applicable law or agreed to in writing, software
//distributed under the License is distributed on an "AS IS" BASIS,
//WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
//See the License for the specific language governing permissions and
//limitations under the License.
//
// Code generated by Alibaba Cloud SDK Code Generator.
// Changes may cause incorrect behavior and will be lost if the code is regenerated.
// ContainerGroup is a nested struct in eci response
type ContainerGroup struct {
ContainerGroupId string `json:"ContainerGroupId" xml:"ContainerGroupId"`
ContainerGroupName string `json:"ContainerGroupName" xml:"ContainerGroupName"`
RegionId string `json:"RegionId" xml:"RegionId"`
ZoneId string `json:"ZoneId" xml:"ZoneId"`
Memory float64 `json:"Memory" xml:"Memory"`
Cpu float64 `json:"Cpu" xml:"Cpu"`
VSwitchId string `json:"VSwitchId" xml:"VSwitchId"`
SecurityGroupId string `json:"SecurityGroupId" xml:"SecurityGroupId"`
RestartPolicy string `json:"RestartPolicy" xml:"RestartPolicy"`
IntranetIp string `json:"IntranetIp" xml:"IntranetIp"`
Status string `json:"Status" xml:"Status"`
InternetIp string `json:"InternetIp" xml:"InternetIp"`
CreationTime string `json:"CreationTime" xml:"CreationTime"`
SucceededTime string `json:"SucceededTime" xml:"SucceededTime"`
Tags []Tag `json:"Tags" xml:"Tags"`
Events []Event `json:"Events" xml:"Events"`
Containers []Container `json:"Containers" xml:"Containers"`
Volumes []Volume `json:"Volumes" xml:"Volumes"`
}

26
providers/alibabacloud/eci/struct_container_port.go
View File

@@ -1,26 +0,0 @@
package eci
import (
"github.com/aliyun/alibaba-cloud-sdk-go/sdk/requests"
)
//Licensed under the Apache License, Version 2.0 (the "License");
//you may not use this file except in compliance with the License.
//You may obtain a copy of the License at
//
//http://www.apache.org/licenses/LICENSE-2.0
//
//Unless required by applicable law or agreed to in writing, software
//distributed under the License is distributed on an "AS IS" BASIS,
//WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
//See the License for the specific language governing permissions and
//limitations under the License.
//
// Code generated by Alibaba Cloud SDK Code Generator.
// Changes may cause incorrect behavior and will be lost if the code is regenerated.
// ContainerPort is a nested struct in eci response
type ContainerPort struct {
Port requests.Integer `name:"Port"`
Protocol string `name:"Protocol"`
}

25
providers/alibabacloud/eci/struct_container_state.go
View File

@@ -1,25 +0,0 @@
package eci
//Licensed under the Apache License, Version 2.0 (the "License");
//you may not use this file except in compliance with the License.
//You may obtain a copy of the License at
//
//http://www.apache.org/licenses/LICENSE-2.0
//
//Unless required by applicable law or agreed to in writing, software
//distributed under the License is distributed on an "AS IS" BASIS,
//WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
//See the License for the specific language governing permissions and
//limitations under the License.
//
// Code generated by Alibaba Cloud SDK Code Generator.
// Changes may cause incorrect behavior and will be lost if the code is regenerated.
// CurrentState is a nested struct in eci response
type ContainerState struct {
State string `json:"State" xml:"State"`
DetailStatus string `json:"DetailStatus" xml:"DetailStatus"`
ExitCode int `json:"ExitCode" xml:"ExitCode"`
StartTime string `json:"StartTime" xml:"StartTime"`
FinishTime string `json:"FinishTime" xml:"FinishTime"`
}

22
providers/alibabacloud/eci/struct_environment_var.go
View File

@@ -1,22 +0,0 @@
package eci
//Licensed under the Apache License, Version 2.0 (the "License");
//you may not use this file except in compliance with the License.
//You may obtain a copy of the License at
//
//http://www.apache.org/licenses/LICENSE-2.0
//
//Unless required by applicable law or agreed to in writing, software
//distributed under the License is distributed on an "AS IS" BASIS,
//WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
//See the License for the specific language governing permissions and
//limitations under the License.
//
// Code generated by Alibaba Cloud SDK Code Generator.
// Changes may cause incorrect behavior and will be lost if the code is regenerated.
// EnvironmentVar is a nested struct in eci response
type EnvironmentVar struct {
Key string `name:"Key"`
Value string `name:"Value"`
}

26
providers/alibabacloud/eci/struct_event.go
View File

@@ -1,26 +0,0 @@
package eci
//Licensed under the Apache License, Version 2.0 (the "License");
//you may not use this file except in compliance with the License.
//You may obtain a copy of the License at
//
//http://www.apache.org/licenses/LICENSE-2.0
//
//Unless required by applicable law or agreed to in writing, software
//distributed under the License is distributed on an "AS IS" BASIS,
//WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
//See the License for the specific language governing permissions and
//limitations under the License.
//
// Code generated by Alibaba Cloud SDK Code Generator.
// Changes may cause incorrect behavior and will be lost if the code is regenerated.
// Event is a nested struct in eci response
type Event struct {
Count int `json:"Count" xml:"Count"`
Type string `json:"Type" xml:"Type"`
Name string `json:"Name" xml:"Name"`
Message string `json:"Message" xml:"Message"`
FirstTimestamp string `json:"FirstTimestamp" xml:"FirstTimestamp"`
LastTimestamp string `json:"LastTimestamp" xml:"LastTimestamp"`
}

22
providers/alibabacloud/eci/struct_tag.go
View File

@@ -1,22 +0,0 @@
package eci
//Licensed under the Apache License, Version 2.0 (the "License");
//you may not use this file except in compliance with the License.
//You may obtain a copy of the License at
//
//http://www.apache.org/licenses/LICENSE-2.0
//
//Unless required by applicable law or agreed to in writing, software
//distributed under the License is distributed on an "AS IS" BASIS,
//WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
//See the License for the specific language governing permissions and
//limitations under the License.
//
// Code generated by Alibaba Cloud SDK Code Generator.
// Changes may cause incorrect behavior and will be lost if the code is regenerated.
// Label is a nested struct in eci response
type Tag struct {
Key string `name:"Key"`
Value string `name:"Value"`
}

35
providers/alibabacloud/eci/struct_volume.go
View File

@@ -1,35 +0,0 @@
package eci
import "github.com/aliyun/alibaba-cloud-sdk-go/sdk/requests"
//Licensed under the Apache License, Version 2.0 (the "License");
//you may not use this file except in compliance with the License.
//You may obtain a copy of the License at
//
//http://www.apache.org/licenses/LICENSE-2.0
//
//Unless required by applicable law or agreed to in writing, software
//distributed under the License is distributed on an "AS IS" BASIS,
//WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
//See the License for the specific language governing permissions and
//limitations under the License.
//
// Code generated by Alibaba Cloud SDK Code Generator.
// Changes may cause incorrect behavior and will be lost if the code is regenerated.
// Volume is a nested struct in eci response
const (
VOL_TYPE_NFS = "NFSVolume"
VOL_TYPE_EMPTYDIR = "EmptyDirVolume"
VOL_TYPE_CONFIGFILEVOLUME = "ConfigFileVolume"
)
type Volume struct {
Type string `name:"Type"`
Name string `name:"Name"`
NfsVolumePath string `name:"NFSVolume.Path"`
NfsVolumeServer string `name:"NFSVolume.Server"`
NfsVolumeReadOnly requests.Boolean `name:"NFSVolume.ReadOnly"`
EmptyDirVolumeEnable requests.Boolean `name:"EmptyDirVolume.Enable"`
ConfigFileToPaths []ConfigFileToPath `name:"ConfigFileVolume.ConfigFileToPath" type:"Repeated"`
}

25
providers/alibabacloud/eci/struct_volume_mount.go
View File

@@ -1,25 +0,0 @@
package eci
import "github.com/aliyun/alibaba-cloud-sdk-go/sdk/requests"
//Licensed under the Apache License, Version 2.0 (the "License");
//you may not use this file except in compliance with the License.
//You may obtain a copy of the License at
//
//http://www.apache.org/licenses/LICENSE-2.0
//
//Unless required by applicable law or agreed to in writing, software
//distributed under the License is distributed on an "AS IS" BASIS,
//WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
//See the License for the specific language governing permissions and
//limitations under the License.
//
// Code generated by Alibaba Cloud SDK Code Generator.
// Changes may cause incorrect behavior and will be lost if the code is regenerated.
// VolumeMount is a nested struct in eci response
type VolumeMount struct {
MountPath string `name:"MountPath"`
ReadOnly requests.Boolean `name:"ReadOnly"`
Name string `name:"Name"`
}

26
providers/alibabacloud/errors.go
View File

@@ -1,26 +0,0 @@
package alibabacloud
import (
"net/http"
"github.com/aliyun/alibaba-cloud-sdk-go/sdk/errors"
"github.com/virtual-kubelet/virtual-kubelet/errdefs"
)
func wrapError(err error) error {
if err == nil {
return nil
}
se, ok := err.(*errors.ServerError)
if !ok {
return err
}
switch se.HttpStatus() {
case http.StatusNotFound:
return errdefs.AsNotFound(err)
default:
return err
}
}

87
providers/aws/README.md
View File

@@ -1,87 +0,0 @@
# AWS Fargate
[AWS Fargate](https://aws.amazon.com/fargate/) is a technology that allows you to run containers
without having to manage servers or clusters. With AWS Fargate, you no longer have to provision,
configure and scale clusters of virtual machines to run containers. This removes the need to choose
server types, decide when to scale your clusters, or optimize cluster packing. Fargate lets you
focus on designing and building your applications instead of managing the infrastructure that runs
them.
Fargate makes it easy to scale your applications. You no longer have to worry about provisioning
enough compute resources. You can launch tens or tens of thousands of containers in seconds.
With Fargate, billing is at a per second granularity and you only pay for what you use. You pay for
the amount of vCPU and memory resources your containerized application requests. vCPU and memory
resources are calculated from the time your container images are pulled until they terminate,
rounded up to the nearest second.
## AWS Fargate virtual-kubelet provider
> Virtual-kubelet and the AWS Fargate virtual-kubelet provider are in very early stages of development.<br>
> DO NOT run them in any Kubernetes production environment or connect to any Fargate production cluster.
AWS Fargate virtual-kubelet provider connects your Kubernetes cluster to a Fargate cluster in AWS.
The Fargate cluster is exposed as a virtual node with the CPU and memory capacity that you choose.
Pods scheduled on the virtual node run on Fargate like they would run on a standard Kubernetes node.
See our [AWS Open Source Blog post](https://aws.amazon.com/blogs/opensource/aws-fargate-virtual-kubelet/) for detailed step-by-step instructions on how to run virtual-kubelet with AWS Fargate. If you are already familiar with virtual-kubelet, the rest of this README contains an overview of how to setup AWS Fargate.
## Prerequisites
If you have never used Fargate before, the easiest way to get started is to run Fargate's
[First run experience](https://console.aws.amazon.com/ecs/home?region=us-east-1#/firstRun). This
will setup Fargate in your AWS account with the default settings. It will create a default Fargate
cluster, IAM roles, a default VPC with an internet gateway and a default security group. You can
always fine-tune individual settings later.
Once you have your first application on Fargate running, move on to the next section below.
You may also want to install the
[AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/installing.html)
and visit the [AWS ECS console](https://console.aws.amazon.com/ecs) to take a closer look at your
Fargate resources.
## Configuration
In order to run virtual-kubelet for AWS Fargate, you need a simple configuration file. We have
provided a [sample configuration file](fargate.toml) for you that contains reasonable defaults and
brief descriptions for each field.
Create a copy of the sample configuration file and customize it.
If you ran the first-run experience, you only need to provide a subnet and set
AssignPublicIPv4Address to true. You can leave the security groups list blank to use the default
security group. You can learn your subnet ID in
[AWS console VPC subnets page](https://console.aws.amazon.com/vpc/home?#subnets). You
also need to update your [security group](https://console.aws.amazon.com/vpc/home?#securityGroups)
to allow traffic to your pods.
## Authentication via IAM
Virtual-kubelet needs permission to schedule pods on Fargate on your behalf. The easiest way to do
so is to run virtual-kubelet on a worker node in your Kubernetes cluster in EC2. Attach an IAM role
to the worker node EC2 instance and give it permission to your Fargate cluster.
## Connecting virtual-kubelet to your Kubernetes cluster
Copy the virtual-kubelet binary and your configuration file to your Kubernetes worker node in EC2.
```console
virtual-kubelet --provider aws --provider-config fargate.toml
```
In your Kubernetes cluster, confirm that the virtual-kubelet shows up as a node.
```console
kubectl get nodes
NAME STATUS ROLES AGE VERSION
virtual-kubelet Ready agent 5s v1.8.3
```
To disconnect, stop the virtual-kubelet process.
## Deploying Kubernetes pods in AWS Fargate
Virtual-kubelet currently supports only a subset of regular kubelet functionality. In order to not
break existing pod deployments, pods that are to be deployed on Fargate require an explicit node
selector that points to the virtual node.

149
providers/aws/config.go
View File

@@ -1,149 +0,0 @@
package aws
import (
"fmt"
"io"
"os"
"github.com/virtual-kubelet/virtual-kubelet/providers"
"github.com/virtual-kubelet/virtual-kubelet/providers/aws/fargate"
"github.com/BurntSushi/toml"
"k8s.io/apimachinery/pkg/api/resource"
)
const (
// Provider configuration defaults.
// https://docs.aws.amazon.com/AmazonECS/latest/developerguide/platform_versions.html
defaultPlatformVersion = "LATEST"
defaultClusterName = "default"
defaultAssignPublicIPv4Address = false
defaultOperatingSystem = providers.OperatingSystemLinux
// Default resource capacity advertised by Fargate provider.
// These are intentionally low to prevent any accidental overuse.
defaultCPUCapacity = "20"
defaultMemoryCapacity = "40Gi"
defaultStorageCapacity = "40Gi"
defaultPodCapacity = "20"
// Minimum resource capacity advertised by Fargate provider.
// These values correspond to the minimum Fargate task size.
minCPUCapacity = "250m"
minMemoryCapacity = "512Mi"
minPodCapacity = "1"
)
// ProviderConfig represents the contents of the provider configuration file.
type providerConfig struct {
Region string
ClusterName string
Subnets []string
SecurityGroups []string
AssignPublicIPv4Address bool
ExecutionRoleArn string
CloudWatchLogGroupName string
PlatformVersion string
OperatingSystem string
CPU string
Memory string
Storage string
Pods string
}
// loadConfigFile loads the given Fargate provider configuration file.
func (p *FargateProvider) loadConfigFile(filePath string) error {
f, err := os.Open(filePath)
if err != nil {
return err
}
defer f.Close()
err = p.loadConfig(f)
return err
}
// loadConfig loads the given Fargate provider TOML configuration stream.
func (p *FargateProvider) loadConfig(r io.Reader) error {
var config providerConfig
var q resource.Quantity
// Set defaults for optional fields.
config.ClusterName = defaultClusterName
config.AssignPublicIPv4Address = defaultAssignPublicIPv4Address
config.PlatformVersion = defaultPlatformVersion
config.OperatingSystem = defaultOperatingSystem
config.CPU = defaultCPUCapacity
config.Memory = defaultMemoryCapacity
config.Storage = defaultStorageCapacity
config.Pods = defaultPodCapacity
// Read the user-supplied configuration.
_, err := toml.DecodeReader(r, &config)
if err != nil {
return err
}
// Validate aggregate configuration.
if config.Region == "" {
return fmt.Errorf("Region is a required field")
}
if !fargate.FargateRegions.Include(config.Region) {
return fmt.Errorf(
"Fargate is available only in regions %v and not available in %v",
fargate.FargateRegions.Names(), config.Region)
}
if config.Subnets == nil || len(config.Subnets) == 0 {
return fmt.Errorf("Subnets is a required field")
}
if config.SecurityGroups == nil {
config.SecurityGroups = []string{}
}
if config.OperatingSystem != providers.OperatingSystemLinux {
return fmt.Errorf("Fargate does not support operating system %v", config.OperatingSystem)
}
if config.CloudWatchLogGroupName != "" && config.ExecutionRoleArn == "" {
return fmt.Errorf("Execution role required if CloudWatch log group is specified")
}
// Validate advertised capacity.
if q, err = resource.ParseQuantity(config.CPU); err != nil {
return fmt.Errorf("Invalid CPU value %v", config.CPU)
}
if q.Cmp(resource.MustParse(minCPUCapacity)) == -1 {
return fmt.Errorf("CPU value %v is less than the minimum %v", config.CPU, minCPUCapacity)
}
if q, err = resource.ParseQuantity(config.Memory); err != nil {
return fmt.Errorf("Invalid memory value %v", config.Memory)
}
if q.Cmp(resource.MustParse(minMemoryCapacity)) == -1 {
return fmt.Errorf("Memory value %v is less than the minimum %v", config.Memory, minMemoryCapacity)
}
if q, err = resource.ParseQuantity(config.Storage); err != nil {
return fmt.Errorf("Invalid storage value %v", config.Storage)
}
if q, err = resource.ParseQuantity(config.Pods); err != nil {
return fmt.Errorf("Invalid pods value %v", config.Pods)
}
if q.Cmp(resource.MustParse(minPodCapacity)) == -1 {
return fmt.Errorf("Pod value %v is less than the minimum %v", config.Pods, minPodCapacity)
}
// Populate provider fields.
p.region = config.Region
p.subnets = config.Subnets
p.securityGroups = config.SecurityGroups
p.clusterName = config.ClusterName
p.assignPublicIPv4Address = config.AssignPublicIPv4Address
p.executionRoleArn = config.ExecutionRoleArn
p.cloudWatchLogGroupName = config.CloudWatchLogGroupName
p.platformVersion = config.PlatformVersion
p.operatingSystem = config.OperatingSystem
p.capacity.cpu = config.CPU
p.capacity.memory = config.Memory
p.capacity.storage = config.Storage
p.capacity.pods = config.Pods
return nil
}

50
providers/aws/fargate.toml
View File

@@ -1,50 +0,0 @@
#
# Example configuration file for AWS Fargate virtual-kubelet provider.
#
# Usage:
# virtual-kubelet --provider aws --provider-config fargate.toml
#
# AWS region where Fargate resources are provisioned. Mandatory.
Region = "us-east-1"
# AWS Fargate cluster name. Optional. Defaults to "default".
# If a cluster with this name does not exist in the region, virtual-kubelet will create it.
# Creating a dedicated Fargate cluster for each virtual-kubelet is recommended.
ClusterName = "default"
# List of subnets that pod ENIs are connected to. Mandatory.
Subnets = ["subnet-12345678"]
# List of security groups associated with pod ENIs. Optional.
# If omitted, pod ENIs inherit their VPC's default security group.
SecurityGroups = ["sg-12345678"]
# Whether pod ENIs are assigned a public IPv4 address. Optional. Defaults to false.
# If your pod requires internet access (e.g. to download container images from ECR or Docker Hub),
# this should be set to "true" for pods on public subnets with internet gateways,
# and to "false" for pods on private subnets with NAT gateways.
AssignPublicIPv4Address = false
# Role assumed by AWS Fargate to execute your pod. Optional.
# https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_execution_IAM_role.html
ExecutionRoleArn = ""
# Amazon CloudWatch log group name used to store container logs. Optional.
# If omitted, container logs will not be available.
# If specified, an execution role with access to CloudWatch logs is required.
CloudWatchLogGroupName = ""
# AWS Fargate platform version. Optional. Defaults to "LATEST".
# https://docs.aws.amazon.com/AmazonECS/latest/developerguide/platform_versions.html
PlatformVersion = "LATEST"
# Operating system for pods. Optional. Defaults to "Linux".
OperatingSystem = "Linux"
# AWS Fargate capacity advertised by virtual-kubelet. Optional. Defaults to the values below.
# Capacity is specified using Kubernetes resource format.
# https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/
CPU = "20"
Memory = "40Gi"
Pods = "20"

54
providers/aws/fargate/client.go
View File

@@ -1,54 +0,0 @@
package fargate
import (
"log"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/cloudwatchlogs"
"github.com/aws/aws-sdk-go/service/cloudwatchlogs/cloudwatchlogsiface"
"github.com/aws/aws-sdk-go/service/ecs"
"github.com/aws/aws-sdk-go/service/ecs/ecsiface"
)
// Client communicates with the regional AWS Fargate service.
type Client struct {
region string
svc *ecs.ECS
api ecsiface.ECSAPI
logsapi cloudwatchlogsiface.CloudWatchLogsAPI
}
var client *Client
// NewClient creates a new Fargate client in the given region.
func newClient(region string) (*Client, error) {
var client Client
// Initialize client session configuration.
config := aws.NewConfig()
config.Region = aws.String(region)
session, err := session.NewSessionWithOptions(
session.Options{
Config: *config,
SharedConfigState: session.SharedConfigEnable,
},
)
if err != nil {
return nil, err
}
// Create the Fargate service client.
client.region = region
client.svc = ecs.New(session)
client.api = client.svc
// Create the CloudWatch service client.
client.logsapi = cloudwatchlogs.New(session)
log.Println("Created Fargate service client.")
return &client, nil
}

358
providers/aws/fargate/cluster.go
View File

@@ -1,358 +0,0 @@
package fargate
import (
"fmt"
"io"
"io/ioutil"
"log"
"strings"
"sync"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/cloudwatchlogs"
"github.com/aws/aws-sdk-go/service/ecs"
"github.com/virtual-kubelet/virtual-kubelet/errdefs"
"github.com/virtual-kubelet/virtual-kubelet/node/api"
k8sTypes "k8s.io/apimachinery/pkg/types"
)
const (
clusterFailureReasonMissing = "MISSING"
)
// ClusterConfig contains a Fargate cluster's configurable parameters.
type ClusterConfig struct {
Region string
Name string
NodeName string
Subnets []string
SecurityGroups []string
AssignPublicIPv4Address bool
ExecutionRoleArn string
CloudWatchLogGroupName string
PlatformVersion string
}
// Cluster represents a Fargate cluster.
type Cluster struct {
region string
name string
nodeName string
arn string
subnets []string
securityGroups []string
assignPublicIPv4Address bool
executionRoleArn string
cloudWatchLogGroupName string
platformVersion string
pods map[string]*Pod
sync.RWMutex
}
// NewCluster creates a new Cluster object.
func NewCluster(config *ClusterConfig) (*Cluster, error) {
var err error
// Cluster name cannot contain '_' as it is used as a separator in task tags.
if strings.Contains(config.Name, "_") {
return nil, fmt.Errorf("cluster name should not contain the '_' character")
}
// Check if Fargate is available in the given region.
if !FargateRegions.Include(config.Region) {
return nil, fmt.Errorf("Fargate is not available in region %s", config.Region)
}
// Create the client to the regional Fargate service.
client, err = newClient(config.Region)
if err != nil {
return nil, fmt.Errorf("failed to create Fargate client: %v", err)
}
// Initialize the cluster.
cluster := &Cluster{
region: config.Region,
name: config.Name,
nodeName: config.NodeName,
subnets: config.Subnets,
securityGroups: config.SecurityGroups,
assignPublicIPv4Address: config.AssignPublicIPv4Address,
executionRoleArn: config.ExecutionRoleArn,
cloudWatchLogGroupName: config.CloudWatchLogGroupName,
platformVersion: config.PlatformVersion,
pods: make(map[string]*Pod),
}
// If a node name is not specified, use the Fargate cluster name.
if cluster.nodeName == "" {
cluster.nodeName = cluster.name
}
// Check if the cluster already exists.
err = cluster.describe()
if err != nil && !strings.Contains(err.Error(), clusterFailureReasonMissing) {
return nil, err
}
// If not, try to create it.
// This might fail if the principal doesn't have the necessary permission.
if cluster.arn == "" {
err = cluster.create()
if err != nil {
return nil, err
}
}
// Load existing pod state from Fargate to the local cache.
err = cluster.loadPodState()
if err != nil {
return nil, err
}
return cluster, nil
}
// Create creates a new Fargate cluster.
func (c *Cluster) create() error {
api := client.api
input := &ecs.CreateClusterInput{
ClusterName: aws.String(c.name),
}
log.Printf("Creating Fargate cluster %s in region %s", c.name, c.region)
output, err := api.CreateCluster(input)
if err != nil {
err = fmt.Errorf("failed to create cluster: %v", err)
log.Println(err)
return err
}
c.arn = aws.StringValue(output.Cluster.ClusterArn)
log.Printf("Created Fargate cluster %s in region %s", c.name, c.region)
return nil
}
// Describe loads information from an existing Fargate cluster.
func (c *Cluster) describe() error {
api := client.api
input := &ecs.DescribeClustersInput{
Clusters: aws.StringSlice([]string{c.name}),
}
log.Printf("Looking for Fargate cluster %s in region %s.", c.name, c.region)
output, err := api.DescribeClusters(input)
if err != nil || len(output.Clusters) == 0 {
if len(output.Failures) > 0 {
err = fmt.Errorf("reason: %s", *output.Failures[0].Reason)
}
err = fmt.Errorf("failed to describe cluster: %v", err)
log.Println(err)
return err
}
log.Printf("Found Fargate cluster %s in region %s.", c.name, c.region)
c.arn = aws.StringValue(output.Clusters[0].ClusterArn)
return nil
}
// LoadPodState rebuilds pod and container objects in this cluster by loading existing tasks from
// Fargate. This is done during startup and whenever the local state is suspected to be out of sync
// with the actual state in Fargate. Caching state locally minimizes the number of service calls.
func (c *Cluster) loadPodState() error {
api := client.api
log.Printf("Loading pod state from cluster %s.", c.name)
taskArns := make([]*string, 0)
// Get a list of all Fargate tasks running on this cluster.
err := api.ListTasksPages(
&ecs.ListTasksInput{
Cluster: aws.String(c.name),
DesiredStatus: aws.String(ecs.DesiredStatusRunning),
LaunchType: aws.String(ecs.LaunchTypeFargate),
},
func(page *ecs.ListTasksOutput, lastPage bool) bool {
taskArns = append(taskArns, page.TaskArns...)
return !lastPage
},
)
if err != nil {
err := fmt.Errorf("failed to load pod state: %v", err)
log.Println(err)
return err
}
log.Printf("Found %d tasks on cluster %s.", len(taskArns), c.name)
pods := make(map[string]*Pod)
// For each task running on this Fargate cluster...
for _, taskArn := range taskArns {
// Describe the task.
describeTasksOutput, err := api.DescribeTasks(
&ecs.DescribeTasksInput{
Cluster: aws.String(c.name),
Tasks: []*string{taskArn},
},
)
if err != nil || len(describeTasksOutput.Tasks) != 1 {
log.Printf("Failed to describe task %s. Skipping.", *taskArn)
continue
}
task := describeTasksOutput.Tasks[0]
// Describe the task definition.
describeTaskDefinitionOutput, err := api.DescribeTaskDefinition(
&ecs.DescribeTaskDefinitionInput{
TaskDefinition: task.TaskDefinitionArn,
},
)
if err != nil {
log.Printf("Failed to describe task definition %s. Skipping.", *task.TaskDefinitionArn)
continue
}
taskDef := describeTaskDefinitionOutput.TaskDefinition
// A pod's tag is stored in its task definition's Family field.
tag := aws.StringValue(taskDef.Family)
// Rebuild the pod object.
// Not all tasks are necessarily pods. Skip tasks that do not have a valid tag.
pod, err := NewPodFromTag(c, tag)
if err != nil {
log.Printf("Skipping unknown task %s: %v", *taskArn, err)
continue
}
pod.uid = k8sTypes.UID(aws.StringValue(task.StartedBy))
pod.taskDefArn = aws.StringValue(task.TaskDefinitionArn)
pod.taskArn = aws.StringValue(task.TaskArn)
if taskDef.TaskRoleArn != nil {
pod.taskRoleArn = aws.StringValue(taskDef.TaskRoleArn)
}
pod.taskStatus = aws.StringValue(task.LastStatus)
pod.taskRefreshTime = time.Now()
// Rebuild the container objects.
for _, cntrDef := range taskDef.ContainerDefinitions {
cntr, _ := newContainerFromDefinition(cntrDef, task.CreatedAt)
pod.taskCPU += aws.Int64Value(cntr.definition.Cpu)
pod.taskMemory += aws.Int64Value(cntr.definition.Memory)
pod.containers[aws.StringValue(cntrDef.Name)] = cntr
log.Printf("Found pod %s/%s on cluster %s.", pod.namespace, pod.name, c.name)
}
pods[tag] = pod
}
// Update local state.
c.Lock()
c.pods = pods
c.Unlock()
return nil
}
// GetPod returns a Kubernetes pod deployed on this cluster.
func (c *Cluster) GetPod(namespace string, name string) (*Pod, error) {
c.RLock()
defer c.RUnlock()
tag := buildTaskDefinitionTag(c.name, namespace, name)
pod, ok := c.pods[tag]
if !ok {
return nil, errdefs.NotFoundf("pod %s/%s is not found", namespace, name)
}
return pod, nil
}
// GetPods returns all Kubernetes pods deployed on this cluster.
func (c *Cluster) GetPods() ([]*Pod, error) {
c.RLock()
defer c.RUnlock()
pods := make([]*Pod, 0, len(c.pods))
for _, pod := range c.pods {
pods = append(pods, pod)
}
return pods, nil
}
// InsertPod inserts a Kubernetes pod to this cluster.
func (c *Cluster) InsertPod(pod *Pod, tag string) {
c.Lock()
defer c.Unlock()
c.pods[tag] = pod
}
// RemovePod removes a Kubernetes pod from this cluster.
func (c *Cluster) RemovePod(tag string) {
c.Lock()
defer c.Unlock()
delete(c.pods, tag)
}
// GetContainerLogs returns the logs of a container from this cluster.
func (c *Cluster) GetContainerLogs(namespace, podName, containerName string, opts api.ContainerLogOpts) (io.ReadCloser, error) {
if c.cloudWatchLogGroupName == "" {
return nil, fmt.Errorf("logs not configured, please specify a \"CloudWatchLogGroupName\"")
}
prefix := fmt.Sprintf("%s_%s", buildTaskDefinitionTag(c.name, namespace, podName), containerName)
describeResult, err := client.logsapi.DescribeLogStreams(&cloudwatchlogs.DescribeLogStreamsInput{
LogGroupName: aws.String(c.cloudWatchLogGroupName),
LogStreamNamePrefix: aws.String(prefix),
})
if err != nil {
return nil, err
}
// Nothing logged yet.
if len(describeResult.LogStreams) == 0 {
return nil, nil
}
logs := ""
err = client.logsapi.GetLogEventsPages(&cloudwatchlogs.GetLogEventsInput{
Limit: aws.Int64(int64(opts.Tail)),
LogGroupName: aws.String(c.cloudWatchLogGroupName),
LogStreamName: describeResult.LogStreams[0].LogStreamName,
}, func(page *cloudwatchlogs.GetLogEventsOutput, lastPage bool) bool {
for _, event := range page.Events {
logs += *event.Message
logs += "\n"
}
// Due to a issue in the aws-sdk last page is never true, but the we can stop
// as soon as no further results are returned.
// See https://github.com/aws/aws-sdk-ruby/pull/730.
return len(page.Events) > 0
})
if err != nil {
return nil, err
}
return ioutil.NopCloser(strings.NewReader(logs)), nil
}

246
providers/aws/fargate/container.go
View File

@@ -1,246 +0,0 @@
package fargate
import (
"fmt"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/ecs"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/resource"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
const (
// Container status strings.
containerStatusProvisioning = "PROVISIONING"
containerStatusPending = "PENDING"
containerStatusRunning = "RUNNING"
containerStatusStopped = "STOPPED"
// Container log configuration options.
containerLogOptionRegion = "awslogs-region"
containerLogOptionGroup = "awslogs-group"
containerLogOptionStreamPrefix = "awslogs-stream-prefix"
// Default container resource limits.
containerDefaultCPULimit int64 = VCPU / 4
containerDefaultMemoryLimit int64 = 512 // * MiB
)
// Container is the representation of a Kubernetes container in Fargate.
type container struct {
definition ecs.ContainerDefinition
startTime time.Time
finishTime time.Time
}
// NewContainer creates a new container from a Kubernetes container spec.
func newContainer(spec *corev1.Container) (*container, error) {
var cntr container
// Translate the Kubernetes container spec to a Fargate container definition.
cntr.definition = ecs.ContainerDefinition{
Name: aws.String(spec.Name),
Image: aws.String(spec.Image),
EntryPoint: aws.StringSlice(spec.Command),
Command: aws.StringSlice(spec.Args),
}
if spec.WorkingDir != "" {
cntr.definition.WorkingDirectory = aws.String(spec.WorkingDir)
}
// Add environment variables.
if spec.Env != nil {
for _, env := range spec.Env {
cntr.definition.Environment = append(
cntr.definition.Environment,
&ecs.KeyValuePair{
Name: aws.String(env.Name),
Value: aws.String(env.Value),
})
}
}
// Translate the Kubernetes container resource requirements to Fargate units.
cntr.setResourceRequirements(&spec.Resources)
return &cntr, nil
}
// NewContainerFromDefinition creates a new container from a Fargate container definition.
func newContainerFromDefinition(def *ecs.ContainerDefinition, startTime *time.Time) (*container, error) {
var cntr container
cntr.definition = *def
if startTime != nil {
cntr.startTime = *startTime
}
return &cntr, nil
}
// ConfigureLogs configures container logs to be sent to the given CloudWatch log group.
func (cntr *container) configureLogs(region string, logGroupName string, streamPrefix string) {
streamPrefix = fmt.Sprintf("%s_%s", streamPrefix, *cntr.definition.Name)
// Fargate requires awslogs log driver.
cntr.definition.LogConfiguration = &ecs.LogConfiguration{
LogDriver: aws.String(ecs.LogDriverAwslogs),
Options: map[string]*string{
containerLogOptionRegion: aws.String(region),
containerLogOptionGroup: aws.String(logGroupName),
containerLogOptionStreamPrefix: aws.String(streamPrefix),
},
}
}
// GetStatus returns the status of a container running in Fargate.
func (cntr *container) getStatus(runtimeState *ecs.Container) corev1.ContainerStatus {
var reason string
var state corev1.ContainerState
var isReady bool
if runtimeState.Reason != nil {
reason = *runtimeState.Reason
}
switch *runtimeState.LastStatus {
case containerStatusProvisioning,
containerStatusPending:
state = corev1.ContainerState{
Waiting: &corev1.ContainerStateWaiting{
Reason: reason,
Message: "",
},
}
case containerStatusRunning:
if cntr.startTime.IsZero() {
cntr.startTime = time.Now()
}
isReady = true
state = corev1.ContainerState{
Running: &corev1.ContainerStateRunning{
StartedAt: metav1.NewTime(cntr.startTime),
},
}
case containerStatusStopped:
if cntr.finishTime.IsZero() {
cntr.finishTime = time.Now()
}
var exitCode int32
if runtimeState.ExitCode != nil {
exitCode = int32(*runtimeState.ExitCode)
}
state = corev1.ContainerState{
Terminated: &corev1.ContainerStateTerminated{
ExitCode: exitCode,
Signal: 0,
Reason: reason,
Message: "",
StartedAt: metav1.NewTime(cntr.startTime),
FinishedAt: metav1.NewTime(cntr.finishTime),
ContainerID: "",
},
}
}
return corev1.ContainerStatus{
Name: *runtimeState.Name,
State: state,
Ready: isReady,
RestartCount: 0,
Image: *cntr.definition.Image,
ImageID: "",
ContainerID: "",
}
}
// SetResourceRequirements translates Kubernetes container resource requirements to Fargate units.
func (cntr *container) setResourceRequirements(reqs *corev1.ResourceRequirements) {
//
// Kubernetes container resource requirements consist of "limits" and "requests" for each
// resource type. Limits are the maximum amount of resources allowed. Requests are the minimum
// amount of resources reserved for the container. Both are optional. If requests are omitted,
// they default to limits. If limits are also omitted, they both default to an
// implementation-defined value.
//
// Fargate container resource requirements consist of CPU shares and memory limits. Memory is a
// hard limit, which when exceeded, causes the container to be killed. MemoryReservation is a
// the amount of resources reserved for the container. At least one must be specified.
//
// Use the defaults if the container does not have any resource requirements.
cpu := containerDefaultCPULimit
memory := containerDefaultMemoryLimit
memoryReservation := containerDefaultMemoryLimit
// Compute CPU requirements.
if reqs != nil {
var quantity resource.Quantity
var ok bool
// Fargate tasks do not share resources with other tasks. Therefore the task and each
// container in it must be allocated their resource limits. Hence limits are preferred
// over requests.
if reqs.Limits != nil {
quantity, ok = reqs.Limits[corev1.ResourceCPU]
}
if !ok && reqs.Requests != nil {
quantity, ok = reqs.Requests[corev1.ResourceCPU]
}
if ok {
// Because Fargate task CPU limit is the sum of the task's containers' CPU shares,
// the container's CPU share equals its CPU limit.
//
// Convert CPU unit from Kubernetes milli-CPUs to EC2 vCPUs.
cpu = quantity.ScaledValue(resource.Milli) * VCPU / 1000
}
}
// Compute memory requirements.
if reqs != nil {
var reqQuantity resource.Quantity
var limQuantity resource.Quantity
var reqOk bool
var limOk bool
// Find the memory request and limit, if available.
if reqs.Requests != nil {
reqQuantity, reqOk = reqs.Requests[corev1.ResourceMemory]
}
if reqs.Limits != nil {
limQuantity, limOk = reqs.Limits[corev1.ResourceMemory]
}
// If one is omitted, use the other one's value.
if !limOk && reqOk {
limQuantity = reqQuantity
} else if !reqOk && limOk {
reqQuantity = limQuantity
}
// If at least one is specified...
if reqOk || limOk {
// Convert memory unit from bytes to MiBs, rounding up to the next MiB.
// This is necessary because Fargate container definition memory reservations and
// limits are both in MiBs.
memoryReservation = (reqQuantity.Value() + MiB - 1) / MiB
memory = (limQuantity.Value() + MiB - 1) / MiB
}
}
// Set final values.
cntr.definition.Cpu = aws.Int64(cpu)
cntr.definition.Memory = aws.Int64(memory)
cntr.definition.MemoryReservation = aws.Int64(memoryReservation)
}

275
providers/aws/fargate/container_test.go
View File

@@ -1,275 +0,0 @@
package fargate
import (
"fmt"
"testing"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/ecs"
"gotest.tools/assert"
is "gotest.tools/assert/cmp"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/resource"
)
const (
anyCPURequest = "500m"
anyCPULimit = "2"
anyMemoryRequest = "768Mi"
anyMemoryLimit = "2Gi"
anyContainerName = "any container name"
anyContainerImage = "any container image"
anyContainerReason = "any reason"
anyContainerExitCode = 42
)
var (
anyContainerSpec = corev1.Container{
Name: anyContainerName,
Image: anyContainerImage,
Command: []string{"anyCmd"},
Args: []string{"anyArg1", "anyArg2"},
WorkingDir: "/any/working/dir",
Env: []corev1.EnvVar{
{Name: "anyEnvName1", Value: "anyEnvValue1"},
{Name: "anyEnvName2", Value: "anyEnvValue2"},
},
}
)
// TestCreateContainer verifies whether Kubernetes container specs are translated to
// Fargate container definitions correctly.
func TestContainerDefinition(t *testing.T) {
cntrSpec := anyContainerSpec
cntr, err := newContainer(&cntrSpec)
assert.NilError(t, err, "failed to create container")
assert.Check(t, is.Equal(cntrSpec.Name, *cntr.definition.Name), "incorrect name")
assert.Check(t, is.Equal(cntrSpec.Image, *cntr.definition.Image), "incorrect image")
assert.Check(t, is.Equal(cntrSpec.Command[0], *cntr.definition.EntryPoint[0]), "incorrect command")
for i, env := range cntrSpec.Args {
assert.Check(t, is.Equal(env, *cntr.definition.Command[i]), "incorrect args")
}
assert.Check(t, is.Equal(cntrSpec.WorkingDir, *cntr.definition.WorkingDirectory), "incorrect working dir")
for i, env := range cntrSpec.Env {
assert.Check(t, is.Equal(env.Name, *cntr.definition.Environment[i].Name), "incorrect env name")
assert.Check(t, is.Equal(env.Value, *cntr.definition.Environment[i].Value), "incorrect env value")
}
}
// TestContainerResourceRequirementsDefaults verifies whether the container gets default CPU
// and memory resources when none is specified.
func TestContainerResourceRequirementsDefaults(t *testing.T) {
cntrSpec := anyContainerSpec
cntr, err := newContainer(&cntrSpec)
assert.NilError(t, err, "failed to create container")
assert.Check(t, is.Equal(containerDefaultCPULimit, *cntr.definition.Cpu), "incorrect CPU limit")
assert.Check(t, is.Equal(containerDefaultMemoryLimit, *cntr.definition.Memory), "incorrect memory limit")
}
// TestContainerResourceRequirementsWithRequestsNoLimits verifies whether the container gets
// correct CPU and memory requests when only requests are specified.
func TestContainerResourceRequirementsWithRequestsNoLimits(t *testing.T) {
cntrSpec := anyContainerSpec
cntrSpec.Resources = corev1.ResourceRequirements{
Requests: map[corev1.ResourceName]resource.Quantity{
corev1.ResourceCPU: resource.MustParse(anyCPURequest),
corev1.ResourceMemory: resource.MustParse(anyMemoryRequest),
},
}
cntr, err := newContainer(&cntrSpec)
assert.NilError(t, err, "failed to create container")
assert.Check(t, is.Equal(int64(512), *cntr.definition.Cpu), "incorrect CPU limit")
assert.Check(t, is.Equal(int64(768), *cntr.definition.Memory), "incorrect memory limit")
}
// TestContainerResourceRequirementsWithLimitsNoRequests verifies whether the container gets
// correct CPU and memory limits when only limits are specified.
func TestContainerResourceRequirementsWithLimitsNoRequests(t *testing.T) {
cntrSpec := anyContainerSpec
cntrSpec.Resources = corev1.ResourceRequirements{
Limits: map[corev1.ResourceName]resource.Quantity{
corev1.ResourceCPU: resource.MustParse(anyCPULimit),
corev1.ResourceMemory: resource.MustParse(anyMemoryLimit),
},
}
cntr, err := newContainer(&cntrSpec)
assert.NilError(t, err, "failed to create container")
assert.Check(t, is.Equal(int64(2048), *cntr.definition.Cpu), "incorrect CPU limit")
assert.Check(t, is.Equal(int64(2048), *cntr.definition.Memory), "incorrect memory limit")
}
// TestContainerResourceRequirementsWithRequestsAndLimits verifies whether the container gets
// correct CPU and memory limits when both requests and limits are specified.
func TestContainerResourceRequirementsWithRequestsAndLimits(t *testing.T) {
cntrSpec := anyContainerSpec
cntrSpec.Resources = corev1.ResourceRequirements{
Requests: map[corev1.ResourceName]resource.Quantity{
corev1.ResourceCPU: resource.MustParse(anyCPURequest),
corev1.ResourceMemory: resource.MustParse(anyMemoryRequest),
},
Limits: map[corev1.ResourceName]resource.Quantity{
corev1.ResourceCPU: resource.MustParse(anyCPULimit),
corev1.ResourceMemory: resource.MustParse(anyMemoryLimit),
},
}
cntr, err := newContainer(&cntrSpec)
assert.NilError(t, err, "failed to create container")
assert.Check(t, is.Equal(int64(2048), *cntr.definition.Cpu), "incorrect CPU limit")
assert.Check(t, is.Equal(int64(2048), *cntr.definition.Memory), "incorrect memory limit")
}
// TestContainerResourceRequirements verifies whether Kubernetes container resource requirements
// are translated to Fargate container resource requests correctly.
func TestContainerResourceRequirementsTranslations(t *testing.T) {
type testCase struct {
requestedCPU string
requestedMemory string
expectedCPU int64
expectedMemoryInMiBs int64
}
// Expected and observed CPU quantities are in units of 1/1024th vCPUs.
var testCases = []testCase{
// Missing or partial resource requests.
{"", "", 256, 512},
{"100m", "", 102, 512},
{"", "256Mi", 256, 256},
// Minimum CPU request.
{"1m", "1Mi", 1, 1},
// Small memory request rounded up to the next MiB.
{"250m", "1Ki", 256, 1},
{"250m", "100Ki", 256, 1},
{"250m", "500Ki", 256, 1},
{"250m", "1024Ki", 256, 1},
{"250m", "1025Ki", 256, 2},
// Common combinations.
{"200m", "300Mi", 204, 300},
{"500m", "500Mi", 512, 500},
{"1000m", "512Mi", 1024, 512},
{"1", "512Mi", 1024, 512},
{"1500m", "1000Mi", 1536, 1000},
{"1500m", "1024Mi", 1536, 1024},
{"2", "2Gi", 2048, 2048},
{"4", "30Gi", 4096, 30 * 1024},
// Very large requests.
{"8", "42Gi", 8192, 42 * 1024},
{"10", "128Gi", 10240, 128 * 1024},
}
for _, tc := range testCases {
t.Run(
fmt.Sprintf("cpu:%s,memory:%s", tc.requestedCPU, tc.requestedMemory),
func(t *testing.T) {
reqs := corev1.ResourceRequirements{
Limits: map[corev1.ResourceName]resource.Quantity{},
}
if tc.requestedCPU != "" {
reqs.Limits[corev1.ResourceCPU] = resource.MustParse(tc.requestedCPU)
}
if tc.requestedMemory != "" {
reqs.Limits[corev1.ResourceMemory] = resource.MustParse(tc.requestedMemory)
}
cntrSpec := anyContainerSpec
cntrSpec.Resources = reqs
cntr, err := newContainer(&cntrSpec)
assert.NilError(t, err, "failed to create container")
assert.Check(t,
*cntr.definition.Cpu == tc.expectedCPU && *cntr.definition.Memory == tc.expectedMemoryInMiBs,
"requested (cpu:%v memory:%v) expected (cpu:%v memory:%v) observed (cpu:%v memory:%v)",
tc.requestedCPU, tc.requestedMemory,
tc.expectedCPU, tc.expectedMemoryInMiBs,
*cntr.definition.Cpu, *cntr.definition.Memory)
})
}
}
// TestContainerStatus verifies whether Kubernetes containers report their status correctly for
// all Fargate container state transitions.
func TestContainerStatus(t *testing.T) {
cntrSpec := anyContainerSpec
cntr, err := newContainer(&cntrSpec)
assert.NilError(t, err, "failed to create container")
// Fargate container status provisioning.
state := ecs.Container{
Name: aws.String(anyContainerName),
Reason: aws.String(anyContainerReason),
LastStatus: aws.String(containerStatusProvisioning),
ExitCode: aws.Int64(0),
}
status := cntr.getStatus(&state)
assert.Check(t, is.Equal(anyContainerName, status.Name), "incorrect name")
assert.Check(t, status.State.Waiting != nil, "incorrect state")
assert.Check(t, is.Equal(anyContainerReason, status.State.Waiting.Reason), "incorrect reason")
assert.Check(t, is.Nil(status.State.Running), "incorrect state")
assert.Check(t, is.Nil(status.State.Terminated), "incorrect state")
assert.Check(t, !status.Ready, "incorrect ready")
assert.Check(t, is.Equal(anyContainerImage, status.Image), "incorrect image")
// Fargate container status pending.
state.LastStatus = aws.String(containerStatusPending)
status = cntr.getStatus(&state)
assert.Check(t, is.Equal(anyContainerName, status.Name), "incorrect name")
assert.Check(t, status.State.Waiting != nil, "incorrect state")
assert.Check(t, is.Equal(anyContainerReason, status.State.Waiting.Reason), "incorrect reason")
assert.Check(t, is.Nil(status.State.Running), "incorrect state")
assert.Check(t, is.Nil(status.State.Terminated), "incorrect state")
assert.Check(t, !status.Ready, "incorrect ready")
assert.Check(t, is.Equal(anyContainerImage, status.Image), "incorrect image")
// Fargate container status running.
state.LastStatus = aws.String(containerStatusRunning)
status = cntr.getStatus(&state)
assert.Check(t, is.Equal(anyContainerName, status.Name), "incorrect name")
assert.Check(t, is.Nil(status.State.Waiting), "incorrect state")
assert.Check(t, status.State.Running != nil, "incorrect state")
assert.Check(t, !status.State.Running.StartedAt.IsZero(), "incorrect startedat")
assert.Check(t, is.Nil(status.State.Terminated), "incorrect state")
assert.Check(t, status.Ready, "incorrect ready")
assert.Check(t, is.Equal(anyContainerImage, status.Image), "incorrect image")
// Fargate container status stopped.
state.LastStatus = aws.String(containerStatusStopped)
state.ExitCode = aws.Int64(anyContainerExitCode)
status = cntr.getStatus(&state)
assert.Check(t, is.Equal(anyContainerName, status.Name), "incorrect name")
assert.Check(t, is.Nil(status.State.Waiting), "incorrect state")
assert.Check(t, is.Nil(status.State.Running), "incorrect state")
assert.Check(t, status.State.Terminated != nil, "incorrect state")
assert.Check(t, is.Equal(int32(anyContainerExitCode), status.State.Terminated.ExitCode), "incorrect exitcode")
assert.Check(t, is.Equal(anyContainerReason, status.State.Terminated.Reason), "incorrect reason")
assert.Check(t, !status.State.Terminated.StartedAt.IsZero(), "incorrect startedat")
assert.Check(t, !status.State.Terminated.FinishedAt.IsZero(), "incorrect finishedat")
assert.Check(t, !status.Ready, "incorrect ready")
assert.Check(t, is.Equal(anyContainerImage, status.Image), "incorrect image")
}

47
providers/aws/fargate/fargate.go
View File

@@ -1,47 +0,0 @@
package fargate
const (
// EC2 compute resource units.
// VCPU is one virtual CPU core in EC2.
VCPU int64 = 1024
// MiB is 2^20 bytes.
MiB int64 = 1024 * 1024
// GiB is 2^30 bytes.
GiB int64 = 1024 * MiB
)
// TaskSize represents a Fargate task size.
type taskSize struct {
cpu int64
memory memorySizeRange
}
// MemorySizeRange represents a range of Fargate task memory sizes.
type memorySizeRange struct {
min int64
max int64
inc int64
}
var (
// Fargate task size table.
// https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#task_size
//
// VCPU Memory (in MiBs, available in 1GiB increments)
// ==== ===================
// 256 512, 1024 ... 2048
// 512 1024 ... 4096
// 1024 2048 ... 8192
// 2048 4096 ... 16384
// 4096 8192 ... 30720
//
taskSizeTable = []taskSize{
{VCPU / 4, memorySizeRange{512 * MiB, 512 * MiB, 1}},
{VCPU / 4, memorySizeRange{1 * GiB, 2 * GiB, 1 * GiB}},
{VCPU / 2, memorySizeRange{1 * GiB, 4 * GiB, 1 * GiB}},
{1 * VCPU, memorySizeRange{2 * GiB, 8 * GiB, 1 * GiB}},
{2 * VCPU, memorySizeRange{4 * GiB, 16 * GiB, 1 * GiB}},
{4 * VCPU, memorySizeRange{8 * GiB, 30 * GiB, 1 * GiB}},
}
)

520
providers/aws/fargate/pod.go
View File

@@ -1,520 +0,0 @@
package fargate
import (
"fmt"
"log"
"strconv"
"strings"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/ecs"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/resource"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
k8sTypes "k8s.io/apimachinery/pkg/types"
)
const (
// Prefixes for objects created in Fargate.
taskDefFamilyPrefix = "vk-podspec"
taskTagPrefix = "vk-pod"
// Task status strings.
taskStatusProvisioning = "PROVISIONING"
taskStatusPending = "PENDING"
taskStatusRunning = "RUNNING"
taskStatusStopped = "STOPPED"
// Task attachment types.
taskAttachmentENI = "ElasticNetworkInterface"
taskAttachmentENIPrivateIPv4Address = "privateIPv4Address"
// Reason used for task state changes.
taskGenericReason = "Initiated by user"
// Annotation to configure the task role.
taskRoleAnnotation = "iam.amazonaws.com/role"
)
// Pod is the representation of a Kubernetes pod in Fargate.
type Pod struct {
// Kubernetes pod properties.
namespace string
name string
uid k8sTypes.UID
// Fargate task properties.
cluster *Cluster
taskDefArn string
taskArn string
taskRoleArn string
taskStatus string
taskRefreshTime time.Time
taskCPU int64
taskMemory int64
containers map[string]*container
}
// NewPod creates a new Kubernetes pod on Fargate.
func NewPod(cluster *Cluster, pod *corev1.Pod) (*Pod, error) {
api := client.api
// Initialize the pod.
fgPod := &Pod{
namespace: pod.Namespace,
name: pod.Name,
uid: pod.UID,
cluster: cluster,
containers: make(map[string]*container),
}
tag := fgPod.buildTaskDefinitionTag()
// Create a task definition matching the pod spec.
taskDef := &ecs.RegisterTaskDefinitionInput{
Family: aws.String(tag),
RequiresCompatibilities: []*string{aws.String(ecs.CompatibilityFargate)},
NetworkMode: aws.String(ecs.NetworkModeAwsvpc),
ContainerDefinitions: []*ecs.ContainerDefinition{},
}
// For each container in the pod...
for _, containerSpec := range pod.Spec.Containers {
// Create a container definition.
cntr, err := newContainer(&containerSpec)
if err != nil {
return nil, err
}
// Configure container logs to be sent to CloudWatch Logs if enabled.
if cluster.cloudWatchLogGroupName != "" {
cntr.configureLogs(cluster.region, cluster.cloudWatchLogGroupName, tag)
}
// Add the container's resource requirements to its pod's total resource requirements.
fgPod.taskCPU += *cntr.definition.Cpu
fgPod.taskMemory += *cntr.definition.Memory
// Insert the container to its pod.
fgPod.containers[containerSpec.Name] = cntr
// Insert container definition to the task definition.
taskDef.ContainerDefinitions = append(taskDef.ContainerDefinitions, &cntr.definition)
}
// Set task resource limits.
err := fgPod.mapTaskSize()
if err != nil {
return nil, err
}
taskDef.Cpu = aws.String(strconv.Itoa(int(fgPod.taskCPU)))
taskDef.Memory = aws.String(strconv.Itoa(int(fgPod.taskMemory)))
// Set a custom task execution IAM role if configured.
if cluster.executionRoleArn != "" {
taskDef.ExecutionRoleArn = aws.String(cluster.executionRoleArn)
}
// Set a custom task IAM role if configured.
if val, ok := pod.Annotations[taskRoleAnnotation]; ok {
taskDef.TaskRoleArn = aws.String(val)
fgPod.taskRoleArn = val
}
// Register the task definition with Fargate.
log.Printf("RegisterTaskDefinition input:%+v", taskDef)
output, err := api.RegisterTaskDefinition(taskDef)
log.Printf("RegisterTaskDefinition err:%+v output:%+v", err, output)
if err != nil {
err = fmt.Errorf("failed to register task definition: %v", err)
return nil, err
}
// Save the registered task definition ARN.
fgPod.taskDefArn = *output.TaskDefinition.TaskDefinitionArn
if cluster != nil {
cluster.InsertPod(fgPod, tag)
}
return fgPod, nil
}
// NewPodFromTag creates a new pod identified by a tag.
func NewPodFromTag(cluster *Cluster, tag string) (*Pod, error) {
data := strings.Split(tag, "_")
if len(data) < 4 ||
data[0] != taskDefFamilyPrefix ||
data[1] != cluster.name {
return nil, fmt.Errorf("invalid tag")
}
pod := &Pod{
namespace: data[2],
name: data[3],
cluster: cluster,
containers: make(map[string]*container),
}
return pod, nil
}
// Start deploys and runs a Kubernetes pod on Fargate.
func (pod *Pod) Start() error {
api := client.api
// Pods always get an ENI with a private IPv4 address in customer subnet.
// Assign a public IPv4 address to the ENI only if requested.
assignPublicIPAddress := ecs.AssignPublicIpDisabled
if pod.cluster.assignPublicIPv4Address {
assignPublicIPAddress = ecs.AssignPublicIpEnabled
}
// Start the task.
runTaskInput := &ecs.RunTaskInput{
Cluster: aws.String(pod.cluster.name),
Count: aws.Int64(1),
LaunchType: aws.String(ecs.LaunchTypeFargate),
NetworkConfiguration: &ecs.NetworkConfiguration{
AwsvpcConfiguration: &ecs.AwsVpcConfiguration{
AssignPublicIp: aws.String(assignPublicIPAddress),
SecurityGroups: aws.StringSlice(pod.cluster.securityGroups),
Subnets: aws.StringSlice(pod.cluster.subnets),
},
},
PlatformVersion: aws.String(pod.cluster.platformVersion),
StartedBy: aws.String(pod.buildTaskTag()),
TaskDefinition: aws.String(pod.taskDefArn),
}
log.Printf("RunTask input:%+v", runTaskInput)
runTaskOutput, err := api.RunTask(runTaskInput)
log.Printf("RunTask err:%+v output:%+v", err, runTaskOutput)
if err != nil || len(runTaskOutput.Tasks) == 0 {
if len(runTaskOutput.Failures) != 0 {
err = fmt.Errorf("reason: %s", *runTaskOutput.Failures[0].Reason)
}
err = fmt.Errorf("failed to run task: %v", err)
return err
}
// Save the task ARN.
pod.taskArn = *runTaskOutput.Tasks[0].TaskArn
return nil
}
// Stop stops a running Kubernetes pod on Fargate.
func (pod *Pod) Stop() error {
api := client.api
// Stop the task.
stopTaskInput := &ecs.StopTaskInput{
Cluster: aws.String(pod.cluster.name),
Reason: aws.String(taskGenericReason),
Task: aws.String(pod.taskArn),
}
log.Printf("StopTask input:%+v", stopTaskInput)
stopTaskOutput, err := api.StopTask(stopTaskInput)
log.Printf("StopTask err:%+v output:%+v", err, stopTaskOutput)
if err != nil {
err = fmt.Errorf("failed to stop task: %v", err)
return err
}
// Deregister the task definition.
_, err = api.DeregisterTaskDefinition(&ecs.DeregisterTaskDefinitionInput{
TaskDefinition: aws.String(pod.taskDefArn),
})
if err != nil {
log.Printf("Failed to deregister task definition: %v", err)
}
// Remove the pod from its cluster.
if pod.cluster != nil {
pod.cluster.RemovePod(pod.buildTaskDefinitionTag())
}
return nil
}
// GetSpec returns the specification of a Kubernetes pod on Fargate.
func (pod *Pod) GetSpec() (*corev1.Pod, error) {
task, err := pod.describe()
if err != nil {
return nil, err
}
return pod.getSpec(task)
}
// GetStatus returns the status of a Kubernetes pod on Fargate.
func (pod *Pod) GetStatus() corev1.PodStatus {
task, err := pod.describe()
if err != nil {
return corev1.PodStatus{Phase: corev1.PodUnknown}
}
return pod.getStatus(task)
}
// BuildTaskDefinitionTag returns the task definition tag for this pod.
func (pod *Pod) buildTaskDefinitionTag() string {
return buildTaskDefinitionTag(pod.cluster.name, pod.namespace, pod.name)
}
// buildTaskDefinitionTag builds a task definition tag from its components.
func buildTaskDefinitionTag(clusterName string, namespace string, name string) string {
// vk-podspec_cluster_namespacae_podname
return fmt.Sprintf("%s_%s_%s_%s", taskDefFamilyPrefix, clusterName, namespace, name)
}
// BuildTaskTag returns the pod's task tag, used for mapping a task back to its pod.
func (pod *Pod) buildTaskTag() string {
return fmt.Sprintf("%s", pod.uid)
}
// mapTaskSize maps Kubernetes pod resource requirements to a Fargate task size.
func (pod *Pod) mapTaskSize() error {
//
// Kubernetes pods do not have explicit resource requirements; their containers do. Pod resource
// requirements are the sum of the pod's containers' requirements.
//
// Fargate tasks have explicit CPU and memory limits. Both are required and specify the maximum
// amount of resources for the task. The limits must match a task size on taskSizeTable.
//
var cpu int64
var memory int64
// Find the smallest Fargate task size that can satisfy the total resource request.
for _, row := range taskSizeTable {
if pod.taskCPU <= row.cpu {
for mem := row.memory.min; mem <= row.memory.max; mem += row.memory.inc {
if pod.taskMemory <= mem/MiB {
cpu = row.cpu
memory = mem / MiB
break
}
}
if cpu != 0 {
break
}
}
}
log.Printf("Mapped resource requirements (cpu:%v, memory:%v) to task size (cpu:%v, memory:%v)",
pod.taskCPU, pod.taskMemory, cpu, memory)
// Fail if the resource requirements cannot be satisfied by any Fargate task size.
if cpu == 0 {
return fmt.Errorf("resource requirements (cpu:%v, memory:%v) are too high",
pod.taskCPU, pod.taskMemory)
}
// Fargate task CPU size is specified in vCPU/1024s and memory size is specified in MiBs.
pod.taskCPU = cpu
pod.taskMemory = memory
return nil
}
// Describe retrieves the status of a Kubernetes pod from Fargate.
func (pod *Pod) describe() (*ecs.Task, error) {
api := client.api
// Describe the task.
describeTasksInput := &ecs.DescribeTasksInput{
Cluster: aws.String(pod.cluster.name),
Tasks: []*string{aws.String(pod.taskArn)},
}
describeTasksOutput, err := api.DescribeTasks(describeTasksInput)
if err != nil || len(describeTasksOutput.Tasks) == 0 {
if len(describeTasksOutput.Failures) != 0 {
err = fmt.Errorf("reason: %s", *describeTasksOutput.Failures[0].Reason)
}
err = fmt.Errorf("failed to describe task: %v", err)
return nil, err
}
task := describeTasksOutput.Tasks[0]
pod.taskStatus = *task.LastStatus
pod.taskRefreshTime = time.Now()
return task, nil
}
// GetSpec returns the specification of a Kubernetes pod on Fargate.
func (pod *Pod) getSpec(task *ecs.Task) (*corev1.Pod, error) {
containers := make([]corev1.Container, 0, len(task.Containers))
for _, c := range task.Containers {
cntrDef := pod.containers[*c.Name].definition
cntr := corev1.Container{
Name: *c.Name,
Image: *cntrDef.Image,
Command: aws.StringValueSlice(cntrDef.EntryPoint),
Args: aws.StringValueSlice(cntrDef.Command),
Resources: corev1.ResourceRequirements{
Limits: corev1.ResourceList{
corev1.ResourceCPU: resource.MustParse(fmt.Sprintf("%d", *cntrDef.Cpu)),
corev1.ResourceMemory: resource.MustParse(fmt.Sprintf("%dMi", *cntrDef.Memory)),
},
Requests: corev1.ResourceList{
corev1.ResourceCPU: resource.MustParse(fmt.Sprintf("%d", *cntrDef.Cpu)),
corev1.ResourceMemory: resource.MustParse(fmt.Sprintf("%dMi", *cntrDef.MemoryReservation)),
},
},
Ports: make([]corev1.ContainerPort, 0, len(cntrDef.PortMappings)),
Env: make([]corev1.EnvVar, 0, len(cntrDef.Environment)),
}
if cntrDef.WorkingDirectory != nil {
cntr.WorkingDir = *cntrDef.WorkingDirectory
}
for _, mapping := range cntrDef.PortMappings {
cntr.Ports = append(cntr.Ports, corev1.ContainerPort{
ContainerPort: int32(*mapping.ContainerPort),
HostPort: int32(*mapping.HostPort),
Protocol: corev1.ProtocolTCP,
})
}
for _, env := range cntrDef.Environment {
cntr.Env = append(cntr.Env, corev1.EnvVar{
Name: *env.Name,
Value: *env.Value,
})
}
containers = append(containers, cntr)
}
annotations := make(map[string]string)
if pod.taskRoleArn != "" {
annotations[taskRoleAnnotation] = pod.taskRoleArn
}
podSpec := corev1.Pod{
TypeMeta: metav1.TypeMeta{
Kind: "Pod",
APIVersion: "v1",
},
ObjectMeta: metav1.ObjectMeta{
Namespace: pod.namespace,
Name: pod.name,
UID: pod.uid,
Annotations: annotations,
},
Spec: corev1.PodSpec{
NodeName: pod.cluster.nodeName,
Volumes: []corev1.Volume{},
Containers: containers,
},
Status: pod.getStatus(task),
}
return &podSpec, nil
}
// GetStatus returns the status of a Kubernetes pod on Fargate.
func (pod *Pod) getStatus(task *ecs.Task) corev1.PodStatus {
// Translate task status to pod phase.
phase := corev1.PodUnknown
switch pod.taskStatus {
case taskStatusProvisioning:
phase = corev1.PodPending
case taskStatusPending:
phase = corev1.PodPending
case taskStatusRunning:
phase = corev1.PodRunning
case taskStatusStopped:
phase = corev1.PodSucceeded
}
// Set pod conditions based on task's last known status.
isScheduled := corev1.ConditionFalse
isInitialized := corev1.ConditionFalse
isReady := corev1.ConditionFalse
switch pod.taskStatus {
case taskStatusProvisioning:
isScheduled = corev1.ConditionTrue
case taskStatusPending:
isScheduled = corev1.ConditionTrue
case taskStatusRunning:
isScheduled = corev1.ConditionTrue
isInitialized = corev1.ConditionTrue
isReady = corev1.ConditionTrue
case taskStatusStopped:
isScheduled = corev1.ConditionTrue
isInitialized = corev1.ConditionTrue
isReady = corev1.ConditionTrue
}
conditions := []corev1.PodCondition{
corev1.PodCondition{
Type: corev1.PodScheduled,
Status: isScheduled,
},
corev1.PodCondition{
Type: corev1.PodInitialized,
Status: isInitialized,
},
corev1.PodCondition{
Type: corev1.PodReady,
Status: isReady,
},
}
// Set the pod start time as the task creation time.
var startTime metav1.Time
if task.CreatedAt != nil {
startTime = metav1.NewTime(*task.CreatedAt)
}
// Set the pod IP address from the task ENI information.
privateIPv4Address := ""
for _, attachment := range task.Attachments {
if *attachment.Type == taskAttachmentENI {
for _, detail := range attachment.Details {
if *detail.Name == taskAttachmentENIPrivateIPv4Address {
privateIPv4Address = *detail.Value
}
}
}
}
// Get statuses from all containers in this pod.
containerStatuses := make([]corev1.ContainerStatus, 0, len(task.Containers))
for _, cntr := range task.Containers {
containerStatuses = append(containerStatuses, pod.containers[*cntr.Name].getStatus(cntr))
}
// Build the pod status structure to be reported.
status := corev1.PodStatus{
Phase: phase,
Conditions: conditions,
Message: "",
Reason: "",
HostIP: privateIPv4Address,
PodIP: privateIPv4Address,
StartTime: &startTime,
InitContainerStatuses: nil,
ContainerStatuses: containerStatuses,
QOSClass: corev1.PodQOSBestEffort,
}
return status
}

134
providers/aws/fargate/pod_test.go
View File

@@ -1,134 +0,0 @@
package fargate
import (
"fmt"
"testing"
"gotest.tools/assert"
is "gotest.tools/assert/cmp"
)
// TestTaskSizeTableInvariants verifies that the task size table is in ascending order by CPU.
// This is necessary for Pod::mapTaskSize to function correctly.
func TestTaskSizeTableInvariants(t *testing.T) {
prevRow := taskSizeTable[0]
for _, row := range taskSizeTable {
assert.Check(t, row.cpu >= prevRow.cpu, "Task size table must be in ascending order by CPU")
prevRow = row
}
}
// TestPodResourceRequirements verifies whether Kubernetes pod resource requirements
// are translated to Fargate task resource requests correctly.
func TestPodResourceRequirements(t *testing.T) {
type testCase struct {
podCPU int64
podMemory int64
taskCPU int64
taskMemory int64
}
testCases := []testCase{
{0, 0, 256, 512},
{1, 1, 256, 512},
{200, 256, 256, 512},
{200, 512, 256, 512},
{256, 3072, 512, 3072},
{256, 512, 256, 512},
{256, 1024, 256, 1024},
{256, 2048, 256, 2048},
{512, 1024, 512, 1024},
{512, 2048, 512, 2048},
{512, 3072, 512, 3072},
{512, 4096, 512, 4096},
{1024, 2 * 1024, 1024, 2 * 1024},
{1024, 3 * 1024, 1024, 3 * 1024},
{1024, 4 * 1024, 1024, 4 * 1024},
{1024, 5 * 1024, 1024, 5 * 1024},
{1024, 6 * 1024, 1024, 6 * 1024},
{1024, 7 * 1024, 1024, 7 * 1024},
{1024, 8 * 1024, 1024, 8 * 1024},
{2048, 4 * 1024, 2048, 4 * 1024},
{2048, 5 * 1024, 2048, 5 * 1024},
{2048, 6 * 1024, 2048, 6 * 1024},
{2048, 7 * 1024, 2048, 7 * 1024},
{2048, 8 * 1024, 2048, 8 * 1024},
{2048, 9 * 1024, 2048, 9 * 1024},
{2048, 10 * 1024, 2048, 10 * 1024},
{2048, 11 * 1024, 2048, 11 * 1024},
{2048, 12 * 1024, 2048, 12 * 1024},
{2048, 13 * 1024, 2048, 13 * 1024},
{2048, 14 * 1024, 2048, 14 * 1024},
{2048, 15 * 1024, 2048, 15 * 1024},
{2048, 16 * 1024, 2048, 16 * 1024},
{4096, 8 * 1024, 4096, 8 * 1024},
{4096, 9 * 1024, 4096, 9 * 1024},
{4096, 10 * 1024, 4096, 10 * 1024},
{4096, 11 * 1024, 4096, 11 * 1024},
{4096, 12 * 1024, 4096, 12 * 1024},
{4096, 13 * 1024, 4096, 13 * 1024},
{4096, 14 * 1024, 4096, 14 * 1024},
{4096, 15 * 1024, 4096, 15 * 1024},
{4096, 16 * 1024, 4096, 16 * 1024},
{4096, 17 * 1024, 4096, 17 * 1024},
{4096, 18 * 1024, 4096, 18 * 1024},
{4096, 19 * 1024, 4096, 19 * 1024},
{4096, 20 * 1024, 4096, 20 * 1024},
{4096, 21 * 1024, 4096, 21 * 1024},
{4096, 22 * 1024, 4096, 22 * 1024},
{4096, 23 * 1024, 4096, 23 * 1024},
{4096, 24 * 1024, 4096, 24 * 1024},
{4096, 25 * 1024, 4096, 25 * 1024},
{4096, 26 * 1024, 4096, 26 * 1024},
{4096, 27 * 1024, 4096, 27 * 1024},
{4096, 28 * 1024, 4096, 28 * 1024},
{4096, 29 * 1024, 4096, 29 * 1024},
{4096, 30 * 1024, 4096, 30 * 1024},
{4097, 30 * 1024, 0, 0},
{4096, 30*1024 + 1, 0, 0},
{4096, 32 * 1024, 0, 0},
{8192, 64 * 1024, 0, 0},
}
for _, tc := range testCases {
t.Run(
fmt.Sprintf("cpu:%v,memory:%v", tc.podCPU, tc.podMemory),
func(t *testing.T) {
pod := &Pod{
taskCPU: tc.podCPU,
taskMemory: tc.podMemory,
}
err := pod.mapTaskSize()
if tc.taskCPU != 0 {
// Test case is expected to succeed.
assert.Check(t, err,
"mapTaskSize failed for (cpu:%v memory:%v)",
tc.podCPU, tc.podMemory)
if err != nil {
return
}
} else {
// Test case is expected to fail.
assert.Check(t, is.ErrorContains(err, ""), "mapTaskSize expected to fail but succeeded for (cpu:%v memory:%v)",
tc.podCPU, tc.podMemory)
return
}
assert.Check(t, pod.taskCPU >= tc.podCPU, "pod assigned less cpu than requested")
assert.Check(t, pod.taskMemory >= tc.podMemory, "pod assigned less memory than requested")
assert.Check(t,
pod.taskCPU == tc.taskCPU && pod.taskMemory == tc.taskMemory,
"requested (cpu:%v memory:%v) expected (cpu:%v memory:%v) observed (cpu:%v memory:%v)\n",
tc.podCPU, tc.podMemory, tc.taskCPU, tc.taskMemory, pod.taskCPU, pod.taskMemory)
})
}
}

53
providers/aws/fargate/region.go
View File

@@ -1,53 +0,0 @@
package fargate
import (
"strings"
)
// Regions is the set of AWS regions where a service is available.
// https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/
type Regions []string
var (
// FargateRegions are AWS regions where Fargate is available.
FargateRegions = Regions{
"ap-northeast-1", // Asia Pacific (Tokyo)
"ap-northeast-2", // Asia Pacific (Seoul)
"ap-southeast-1", // Asia Pacific (Singapore)
"ap-southeast-2", // Asia Pacific (Sydney)
"ap-south-1", // Asia Pacific (Mumbai)
"ca-central-1", // Canada (Central)
"eu-central-1", // EU (Frankfurt)
"eu-west-1", // EU (Ireland)
"eu-west-2", // EU (London)
"us-east-1", // US East (N. Virginia)
"us-east-2", // US East (Ohio)
"us-west-1", // US West (N. California)
"us-west-2", // US West (Oregon)
}
)
// Include returns whether the region set includes the given region.
func (r Regions) Include(region string) bool {
region = strings.ToLower(region)
region = strings.Trim(region, " ")
for _, name := range r {
if name == region {
return true
}
}
return false
}
// Names returns an array of region names.
func (r Regions) Names() []string {
names := make([]string, 0, len(r))
for _, name := range r {
names = append(names, name)
}
return names
}

284
providers/aws/helpers_test.go
View File

@@ -1,284 +0,0 @@
package aws_test
import (
"fmt"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/ec2"
)
// createVpcWithInternetAccess create a VPC with one subnet and internet access
// and tags all created resources
func createVpcWithInternetAccess(ec2Client *ec2.EC2) (*string, error) {
vpcCreateResponse, err := ec2Client.CreateVpc(&ec2.CreateVpcInput{
CidrBlock: aws.String("172.31.0.0/16"),
})
if err != nil {
return nil, err
}
vpcID := vpcCreateResponse.Vpc.VpcId
err = tagResource(ec2Client, vpcID)
if err != nil {
return nil, err
}
subnetResponse, err := ec2Client.CreateSubnet(&ec2.CreateSubnetInput{
CidrBlock: aws.String("172.31.0.0/16"),
VpcId: vpcID,
})
if err != nil {
return nil, err
}
subnetID := subnetResponse.Subnet.SubnetId
err = tagResource(ec2Client, subnetID)
if err != nil {
return nil, err
}
igResponse, err := ec2Client.CreateInternetGateway(&ec2.CreateInternetGatewayInput{})
if err != nil {
return nil, err
}
igID := igResponse.InternetGateway.InternetGatewayId
err = tagResource(ec2Client, igID)
if err != nil {
return nil, err
}
_, err = ec2Client.AttachInternetGateway(&ec2.AttachInternetGatewayInput{
InternetGatewayId: igID,
VpcId: vpcID,
})
if err != nil {
return nil, err
}
routeTableResponse, err := ec2Client.CreateRouteTable(&ec2.CreateRouteTableInput{
VpcId: vpcID,
})
if err != nil {
return nil, err
}
routeTableID := routeTableResponse.RouteTable.RouteTableId
err = tagResource(ec2Client, routeTableID)
if err != nil {
return nil, err
}
_, err = ec2Client.AssociateRouteTable(&ec2.AssociateRouteTableInput{
RouteTableId: routeTableID,
SubnetId: subnetID,
})
if err != nil {
return nil, err
}
_, err = ec2Client.CreateRoute(&ec2.CreateRouteInput{
DestinationCidrBlock: aws.String("0.0.0.0/0"),
GatewayId: igID,
RouteTableId: routeTableID,
})
if err != nil {
return nil, err
}
return subnetID, nil
}
// tagResource tries to tag an EC2 resource in a loop to workaround EC2 eventual consistency
func tagResource(ec2Client *ec2.EC2, resourceID *string) error {
fmt.Printf("Tagging: %s\n", *resourceID)
return retry(func() error {
_, err := ec2Client.CreateTags(&ec2.CreateTagsInput{
Resources: []*string{resourceID},
Tags: []*ec2.Tag{&ec2.Tag{
Key: aws.String("Name"),
Value: aws.String("vk-aws-e2e-test"),
}},
})
return err
})
}
// deleteVpc deletes all resources of the created VPC by enumarating all tagged
// resources and deleting them with multiple retries to cope with EC2 eventual
// consistency
func deleteVpc(ec2Client *ec2.EC2) error {
// Remove any routing tables
retry(func() error {
resourceIDs, err := findResourceByTag(ec2Client, "route-table")
if err != nil {
return err
}
describeResponse, err := ec2Client.DescribeRouteTables(&ec2.DescribeRouteTablesInput{
RouteTableIds: resourceIDs,
})
if err != nil {
return err
}
for _, routeTable := range describeResponse.RouteTables {
for _, association := range routeTable.Associations {
_, err = ec2Client.DisassociateRouteTable(&ec2.DisassociateRouteTableInput{
AssociationId: association.RouteTableAssociationId,
})
if err != nil {
return err
}
}
_, err = ec2Client.DeleteRouteTable(&ec2.DeleteRouteTableInput{
RouteTableId: routeTable.RouteTableId,
})
if err != nil {
return err
}
}
return nil
})
// Remove associatated internet gateways
retry(func() error {
resourceIDs, err := findResourceByTag(ec2Client, "internet-gateway")
if err != nil {
return err
}
describeResponse, err := ec2Client.DescribeInternetGateways(&ec2.DescribeInternetGatewaysInput{
InternetGatewayIds: resourceIDs,
})
if err != nil {
return err
}
for _, internetGateway := range describeResponse.InternetGateways {
for _, attachment := range internetGateway.Attachments {
_, err = ec2Client.DetachInternetGateway(&ec2.DetachInternetGatewayInput{
InternetGatewayId: internetGateway.InternetGatewayId,
VpcId: attachment.VpcId,
})
if err != nil {
return err
}
}
_, err = ec2Client.DeleteInternetGateway(&ec2.DeleteInternetGatewayInput{
InternetGatewayId: internetGateway.InternetGatewayId,
})
if err != nil {
return err
}
}
return nil
})
// Remove subnets
retry(func() error {
resourceIDs, err := findResourceByTag(ec2Client, "subnet")
if err != nil {
return err
}
for _, resourceID := range resourceIDs {
_, err = ec2Client.DeleteSubnet(&ec2.DeleteSubnetInput{
SubnetId: resourceID,
})
if err != nil {
return err
}
}
return nil
})
// Remove the VPC itself
retry(func() error {
resourceIDs, err := findResourceByTag(ec2Client, "vpc")
if err != nil {
return err
}
for _, resourceID := range resourceIDs {
_, err = ec2Client.DeleteVpc(&ec2.DeleteVpcInput{
VpcId: resourceID,
})
if err != nil {
return err
}
}
return nil
})
return nil
}
// findResourceByTag finds EC2 resources by a tag
func findResourceByTag(ec2Client *ec2.EC2, resourceType string) ([]*string, error) {
describeResponse, err := ec2Client.DescribeTags(&ec2.DescribeTagsInput{
Filters: []*ec2.Filter{
&ec2.Filter{
Name: aws.String("key"),
Values: []*string{aws.String("Name")},
},
&ec2.Filter{
Name: aws.String("value"),
Values: []*string{aws.String("vk-aws-e2e-test")},
},
&ec2.Filter{
Name: aws.String("resource-type"),
Values: []*string{aws.String(resourceType)},
},
},
})
if err != nil {
return nil, err
}
resourceIDs := make([]*string, len(describeResponse.Tags))
for i, tag := range describeResponse.Tags {
resourceIDs[i] = tag.ResourceId
}
return resourceIDs, nil
}
type fn func() error
// retry retries an action up to 10 times
func retry(action fn) error {
attempts := 10
sleep := time.Second * 10
for {
if attempts == 0 {
return fmt.Errorf("action failed, maximum attempts reached")
}
err := action()
if err == nil {
return nil
}
fmt.Printf("action failed, err: %s retrying...\n", err)
time.Sleep(sleep)
}
}

338
providers/aws/provider.go
View File

@@ -1,338 +0,0 @@
package aws
import (
"context"
"fmt"
"io"
"log"
"time"
"github.com/virtual-kubelet/virtual-kubelet/manager"
"github.com/virtual-kubelet/virtual-kubelet/node/api"
"github.com/virtual-kubelet/virtual-kubelet/providers/aws/fargate"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/resource"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// FargateProvider implements the virtual-kubelet provider interface.
type FargateProvider struct {
resourceManager *manager.ResourceManager
nodeName string
operatingSystem string
internalIP string
daemonEndpointPort int32
// AWS resources.
region string
subnets []string
securityGroups []string
// Fargate resources.
cluster *fargate.Cluster
clusterName string
capacity capacity
assignPublicIPv4Address bool
executionRoleArn string
cloudWatchLogGroupName string
platformVersion string
lastTransitionTime time.Time
}
// Capacity represents the provisioned capacity on a Fargate cluster.
type capacity struct {
cpu string
memory string
storage string
pods string
}
var (
errNotImplemented = fmt.Errorf("not implemented by Fargate provider")
)
// NewFargateProvider creates a new Fargate provider.
func NewFargateProvider(
config string,
rm *manager.ResourceManager,
nodeName string,
operatingSystem string,
internalIP string,
daemonEndpointPort int32) (*FargateProvider, error) {
// Create the Fargate provider.
log.Println("Creating Fargate provider.")
p := FargateProvider{
resourceManager: rm,
nodeName: nodeName,
operatingSystem: operatingSystem,
internalIP: internalIP,
daemonEndpointPort: daemonEndpointPort,
}
// Read the Fargate provider configuration file.
err := p.loadConfigFile(config)
if err != nil {
err = fmt.Errorf("failed to load configuration file %s: %v", config, err)
return nil, err
}
log.Printf("Loaded provider configuration file %s.", config)
// Find or create the configured Fargate cluster.
clusterConfig := fargate.ClusterConfig{
Region: p.region,
Name: p.clusterName,
NodeName: nodeName,
Subnets: p.subnets,
SecurityGroups: p.securityGroups,
AssignPublicIPv4Address: p.assignPublicIPv4Address,
ExecutionRoleArn: p.executionRoleArn,
CloudWatchLogGroupName: p.cloudWatchLogGroupName,
PlatformVersion: p.platformVersion,
}
p.cluster, err = fargate.NewCluster(&clusterConfig)
if err != nil {
err = fmt.Errorf("failed to create Fargate cluster: %v", err)
return nil, err
}
p.lastTransitionTime = time.Now()
log.Printf("Created Fargate provider: %+v.", p)
return &p, nil
}
// CreatePod takes a Kubernetes Pod and deploys it within the Fargate provider.
func (p *FargateProvider) CreatePod(ctx context.Context, pod *corev1.Pod) error {
log.Printf("Received CreatePod request for %+v.\n", pod)
fgPod, err := fargate.NewPod(p.cluster, pod)
if err != nil {
log.Printf("Failed to create pod: %v.\n", err)
return err
}
err = fgPod.Start()
if err != nil {
log.Printf("Failed to start pod: %v.\n", err)
return err
}
return nil
}
// UpdatePod takes a Kubernetes Pod and updates it within the provider.
func (p *FargateProvider) UpdatePod(ctx context.Context, pod *corev1.Pod) error {
log.Printf("Received UpdatePod request for %s/%s.\n", pod.Namespace, pod.Name)
return errNotImplemented
}
// DeletePod takes a Kubernetes Pod and deletes it from the provider.
func (p *FargateProvider) DeletePod(ctx context.Context, pod *corev1.Pod) error {
log.Printf("Received DeletePod request for %s/%s.\n", pod.Namespace, pod.Name)
fgPod, err := p.cluster.GetPod(pod.Namespace, pod.Name)
if err != nil {
log.Printf("Failed to get pod: %v.\n", err)
return err
}
err = fgPod.Stop()
if err != nil {
log.Printf("Failed to stop pod: %v.\n", err)
return err
}
return nil
}
// GetPod retrieves a pod by name from the provider (can be cached).
func (p *FargateProvider) GetPod(ctx context.Context, namespace, name string) (*corev1.Pod, error) {
log.Printf("Received GetPod request for %s/%s.\n", namespace, name)
pod, err := p.cluster.GetPod(namespace, name)
if err != nil {
log.Printf("Failed to get pod: %v.\n", err)
return nil, err
}
spec, err := pod.GetSpec()
if err != nil {
log.Printf("Failed to get pod spec: %v.\n", err)
return nil, err
}
log.Printf("Responding to GetPod: %+v.\n", spec)
return spec, nil
}
// GetContainerLogs retrieves the logs of a container by name from the provider.
func (p *FargateProvider) GetContainerLogs(ctx context.Context, namespace, podName, containerName string, opts api.ContainerLogOpts) (io.ReadCloser, error) {
log.Printf("Received GetContainerLogs request for %s/%s/%s.\n", namespace, podName, containerName)
return p.cluster.GetContainerLogs(namespace, podName, containerName, opts)
}
// GetPodFullName retrieves the full pod name as defined in the provider context.
func (p *FargateProvider) GetPodFullName(namespace string, pod string) string {
return ""
}
// RunInContainer executes a command in a container in the pod, copying data
// between in/out/err and the container's stdin/stdout/stderr.
func (p *FargateProvider) RunInContainer(ctx context.Context, namespace, podName, containerName string, cmd []string, attach api.AttachIO) error {
return errNotImplemented
}
// GetPodStatus retrieves the status of a pod by name from the provider.
func (p *FargateProvider) GetPodStatus(ctx context.Context, namespace, name string) (*corev1.PodStatus, error) {
log.Printf("Received GetPodStatus request for %s/%s.\n", namespace, name)
pod, err := p.cluster.GetPod(namespace, name)
if err != nil {
log.Printf("Failed to get pod: %v.\n", err)
return nil, err
}
status := pod.GetStatus()
log.Printf("Responding to GetPodStatus: %+v.\n", status)
return &status, nil
}
// GetPods retrieves a list of all pods running on the provider (can be cached).
func (p *FargateProvider) GetPods(ctx context.Context) ([]*corev1.Pod, error) {
log.Println("Received GetPods request.")
pods, err := p.cluster.GetPods()
if err != nil {
log.Printf("Failed to get pods: %v.\n", err)
return nil, err
}
var result []*corev1.Pod
for _, pod := range pods {
spec, err := pod.GetSpec()
if err != nil {
log.Printf("Failed to get pod spec: %v.\n", err)
continue
}
result = append(result, spec)
}
log.Printf("Responding to GetPods: %+v.\n", result)
return result, nil
}
// Capacity returns a resource list with the capacity constraints of the provider.
func (p *FargateProvider) Capacity(ctx context.Context) corev1.ResourceList {
log.Println("Received Capacity request.")
return corev1.ResourceList{
corev1.ResourceCPU: resource.MustParse(p.capacity.cpu),
corev1.ResourceMemory: resource.MustParse(p.capacity.memory),
corev1.ResourceStorage: resource.MustParse(p.capacity.storage),
corev1.ResourcePods: resource.MustParse(p.capacity.pods),
}
}
// NodeConditions returns a list of conditions (Ready, OutOfDisk, etc), which is polled
// periodically to update the node status within Kubernetes.
func (p *FargateProvider) NodeConditions(ctx context.Context) []corev1.NodeCondition {
log.Println("Received NodeConditions request.")
lastHeartbeatTime := metav1.Now()
lastTransitionTime := metav1.NewTime(p.lastTransitionTime)
lastTransitionReason := "Fargate cluster is ready"
lastTransitionMessage := "ok"
// Return static thumbs-up values for all conditions.
return []corev1.NodeCondition{
{
Type: corev1.NodeReady,
Status: corev1.ConditionTrue,
LastHeartbeatTime: lastHeartbeatTime,
LastTransitionTime: lastTransitionTime,
Reason: lastTransitionReason,
Message: lastTransitionMessage,
},
{
Type: corev1.NodeOutOfDisk,
Status: corev1.ConditionFalse,
LastHeartbeatTime: lastHeartbeatTime,
LastTransitionTime: lastTransitionTime,
Reason: lastTransitionReason,
Message: lastTransitionMessage,
},
{
Type: corev1.NodeMemoryPressure,
Status: corev1.ConditionFalse,
LastHeartbeatTime: lastHeartbeatTime,
LastTransitionTime: lastTransitionTime,
Reason: lastTransitionReason,
Message: lastTransitionMessage,
},
{
Type: corev1.NodeDiskPressure,
Status: corev1.ConditionFalse,
LastHeartbeatTime: lastHeartbeatTime,
LastTransitionTime: lastTransitionTime,
Reason: lastTransitionReason,
Message: lastTransitionMessage,
},
{
Type: corev1.NodeNetworkUnavailable,
Status: corev1.ConditionFalse,
LastHeartbeatTime: lastHeartbeatTime,
LastTransitionTime: lastTransitionTime,
Reason: lastTransitionReason,
Message: lastTransitionMessage,
},
{
Type: "KubeletConfigOk",
Status: corev1.ConditionTrue,
LastHeartbeatTime: lastHeartbeatTime,
LastTransitionTime: lastTransitionTime,
Reason: lastTransitionReason,
Message: lastTransitionMessage,
},
}
}
// NodeAddresses returns a list of addresses for the node status within Kubernetes.
func (p *FargateProvider) NodeAddresses(ctx context.Context) []corev1.NodeAddress {
log.Println("Received NodeAddresses request.")
return []corev1.NodeAddress{
{
Type: corev1.NodeInternalIP,
Address: p.internalIP,
},
}
}
// NodeDaemonEndpoints returns NodeDaemonEndpoints for the node status within Kubernetes.
func (p *FargateProvider) NodeDaemonEndpoints(ctx context.Context) *corev1.NodeDaemonEndpoints {
log.Println("Received NodeDaemonEndpoints request.")
return &corev1.NodeDaemonEndpoints{
KubeletEndpoint: corev1.DaemonEndpoint{
Port: p.daemonEndpointPort,
},
}
}
// OperatingSystem returns the operating system the provider is for.
func (p *FargateProvider) OperatingSystem() string {
log.Println("Received OperatingSystem request.")
return p.operatingSystem
}

365
providers/aws/provider_test.go
View File

@@ -1,365 +0,0 @@
package aws_test
import (
"context"
"fmt"
"io/ioutil"
"os"
"strings"
"testing"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/cloudwatchlogs"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/aws/aws-sdk-go/service/ecs"
"github.com/aws/aws-sdk-go/service/iam"
"github.com/virtual-kubelet/virtual-kubelet/node/api"
vkAWS "github.com/virtual-kubelet/virtual-kubelet/providers/aws"
v1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/resource"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/types"
)
const (
// E2E test configuration.
testName = "vk-fargate-e2e-test"
defaultTestRegion = "us-east-1"
// Environment variables that modify the test behavior.
envSkipTests = "SKIP_AWS_E2E"
envTestRegion = "VK_TEST_FARGATE_REGION"
)
// executorRoleAssumePolicy is the policy used by task execution role.
const executorRoleAssumePolicy = `{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ecs-tasks.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}`
// testConfig contains the Fargate provider test configuration template.
const testConfig = `
Region = "%s"
ClusterName = "%s"
Subnets = [ "%s" ]
SecurityGroups = [ ]
AssignPublicIPv4Address = true
ExecutionRoleArn = "%s"
CloudWatchLogGroupName = "%s"
`
var (
ecsClient *ecs.ECS
testRegion string
subnetID *string
executorRoleName *string
logGroupName *string
)
// TestMain wraps the tests with the extra setup and teardown of AWS resources.
func TestMain(m *testing.M) {
var err error
// Skip the tests in this package if the environment variable is set.
if os.Getenv(envSkipTests) == "1" {
fmt.Println("Skipping AWS E2E tests.")
os.Exit(0)
}
// Query the test region.
region, ok := os.LookupEnv(envTestRegion)
if ok {
testRegion = region
} else {
testRegion = defaultTestRegion
}
fmt.Printf("Starting provider tests in region %s\n", testRegion)
// Create the session and clients.
session := session.New(&aws.Config{
Region: aws.String(testRegion),
})
ecsClient = ecs.New(session)
ec2Client := ec2.New(session)
cloudwatchClient := cloudwatchlogs.New(session)
iamClient := iam.New(session)
// Create a test VPC with one subnet and internet access.
// Internet access is required to pull public images from the docker registry.
subnetID, err = createVpcWithInternetAccess(ec2Client)
if err != nil {
fmt.Printf("Failed to create VPC: %+v\n", err)
os.Exit(-1)
}
// Create the AWS CloudWatch Logs log group used by containers.
logGroupName = aws.String("/ecs/" + testName)
_, err = cloudwatchClient.CreateLogGroup(&cloudwatchlogs.CreateLogGroupInput{
LogGroupName: logGroupName,
})
if err != nil {
fmt.Printf("Failed to create CloudWatch Logs log group: %+v\n", err)
os.Exit(-1)
}
// Create the role used by Fargate to write logs and pull ECR images.
executorRoleName = aws.String(testName)
_, err = iamClient.CreateRole(&iam.CreateRoleInput{
RoleName: executorRoleName,
AssumeRolePolicyDocument: aws.String(executorRoleAssumePolicy),
})
if err != nil {
fmt.Printf("Failed to create task execution role: %+v", err)
os.Exit(-1)
}
// Attach the default policy allowing log writes and ECR pulls.
iamClient.AttachRolePolicy(&iam.AttachRolePolicyInput{
PolicyArn: aws.String("arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"),
RoleName: executorRoleName,
})
if err != nil {
fmt.Printf("Failed to attach role policy: %+v", err)
os.Exit(-1)
}
// Run the tests.
exitCode := m.Run()
// Delete the task execution role.
iamClient.DetachRolePolicy(&iam.DetachRolePolicyInput{
PolicyArn: aws.String("arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"),
RoleName: executorRoleName,
})
if err != nil {
fmt.Printf("Failed to delete task execution role: %+v", err)
}
// Delete the role.
_, err = iamClient.DeleteRole(&iam.DeleteRoleInput{
RoleName: executorRoleName,
})
if err != nil {
fmt.Printf("Failed to delete task execution role: %+v", err)
}
// Delete the log group.
_, err = cloudwatchClient.DeleteLogGroup(&cloudwatchlogs.DeleteLogGroupInput{
LogGroupName: logGroupName,
})
if err != nil {
fmt.Printf("Failed to delete CloudWatch Logs log group: %+v\n", err)
}
// Delete the test VPC.
err = deleteVpc(ec2Client)
if err != nil {
fmt.Printf("Failed to delete VPC: %+v\n", err)
}
os.Exit(exitCode)
}
// TestAWSFargateProviderPodLifecycle validates basic pod lifecycle by starting and stopping a pod.
func TestAWSFargateProviderPodLifecycle(t *testing.T) {
// Create a cluster for the E2E test.
createResponse, err := ecsClient.CreateCluster(&ecs.CreateClusterInput{
ClusterName: aws.String(testName),
})
if err != nil {
t.Error(err)
}
clusterID := createResponse.Cluster.ClusterArn
time.Sleep(10 * time.Second)
t.Run("Create, list and delete pod", func(t *testing.T) {
// Write provider config file with test configuration.
config := fmt.Sprintf(testConfig, testRegion, testName, *subnetID, *executorRoleName, *logGroupName)
fmt.Printf("Fargate provider test configuration:%s", config)
tmpfile, err := ioutil.TempFile("", "example")
if err != nil {
t.Fatal(err)
}
defer os.Remove(tmpfile.Name())
if _, err = tmpfile.Write([]byte(config)); err != nil {
t.Fatal(err)
}
if err = tmpfile.Close(); err != nil {
t.Fatal(err)
}
// Start the Fargate provider.
provider, err := vkAWS.NewFargateProvider(
tmpfile.Name(), nil, testName, "Linux", "1.2.3.4", 10250)
if err != nil {
t.Fatal(err)
}
// Confirm that there are no pods on the cluster.
pods, err := provider.GetPods(context.Background())
if err != nil {
t.Error(err)
}
if len(pods) != 0 {
t.Errorf("Expect zero pods, but received %d pods\n%v", len(pods), pods)
}
// Create a test pod.
podName := fmt.Sprintf("test_%d", time.Now().UnixNano()/1000)
pod := &v1.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: podName,
Namespace: "default",
UID: types.UID("unique"),
},
Spec: v1.PodSpec{
Containers: []v1.Container{v1.Container{
Name: "echo-container",
Image: "busybox",
Command: []string{
"/bin/sh",
},
Args: []string{
"-c",
"echo \"Started\";" +
"echo \"TEST_ENV=$TEST_ENV\";" +
"while true; do sleep 1; done",
},
Env: []v1.EnvVar{
{Name: "TEST_ENV", Value: "AnyValue"},
},
Resources: v1.ResourceRequirements{
Limits: v1.ResourceList{
v1.ResourceCPU: resource.MustParse("200m"),
v1.ResourceMemory: resource.MustParse("450Mi"),
},
Requests: v1.ResourceList{
v1.ResourceCPU: resource.MustParse("100m"),
v1.ResourceMemory: resource.MustParse("256Mi"),
},
},
}},
},
}
err = provider.CreatePod(context.Background(), pod)
if err != nil {
t.Fatal(err)
}
// Now there should be exactly one pod.
pods, err = provider.GetPods(context.Background())
if err != nil {
t.Error(err)
}
if len(pods) != 1 {
t.Errorf("Expect one pods, but received %d pods\n%v", len(pods), pods)
}
// Wait until the pod is running.
err = waitUntilPodStatus(provider, podName, v1.PodRunning)
if err != nil {
t.Error(err)
}
// Wait a few seconds for the logs to settle.
time.Sleep(10 * time.Second)
logs, err := provider.GetContainerLogs(context.Background(), "default", podName, "echo-container", api.ContainerLogOpts{Tail: 100})
if err != nil {
t.Fatal(err)
}
defer logs.Close()
b, err := ioutil.ReadAll(logs)
if err != nil {
t.Fatal(err)
}
// Test log output.
receivedLogs := strings.Split(string(b), "\n")
expectedLogs := []string{
"Started",
pod.Spec.Containers[0].Env[0].Name + "=" + pod.Spec.Containers[0].Env[0].Value,
}
for i, line := range receivedLogs {
fmt.Printf("Log[#%d]: %v\n", i, line)
if len(expectedLogs) > i && receivedLogs[i] != expectedLogs[i] {
t.Errorf("Expected log line %d to be %q, but received %q", i, line, receivedLogs[i])
}
}
// Delete the pod.
err = provider.DeletePod(context.Background(), pod)
if err != nil {
t.Fatal(err)
}
err = waitUntilPodStatus(provider, podName, v1.PodSucceeded)
if err != nil {
t.Error(err)
}
// The cluster should be empty again.
pods, err = provider.GetPods(context.Background())
if err != nil {
t.Error(err)
}
if len(pods) != 0 {
t.Errorf("Expect zero pods, but received %d pods\n%v", len(pods), pods)
}
})
// Delete the test cluster.
_, err = ecsClient.DeleteCluster(&ecs.DeleteClusterInput{
Cluster: clusterID,
})
if err != nil {
t.Error(err)
}
}
// waitUntilPodStatus polls pod status until the desired state is reached.
func waitUntilPodStatus(provider *vkAWS.FargateProvider, podName string, desiredStatus v1.PodPhase) error {
ctx := context.Background()
context.WithTimeout(ctx, time.Duration(time.Second*60))
for {
select {
case <-ctx.Done():
return ctx.Err()
default:
status, err := provider.GetPodStatus(context.Background(), "default", podName)
if err != nil {
if strings.Contains(err.Error(), "is not found") {
return nil
}
return err
}
if status.Phase == desiredStatus {
return nil
}
time.Sleep(3 * time.Second)
}
}
}

79
providers/azurebatch/README.md
View File

@@ -1,79 +0,0 @@
# Kubernetes Virtual Kubelet with Azure Batch
[Azure Batch](https://docs.microsoft.com/en-us/azure/batch/) provides a HPC Computing environment in Azure for distributed tasks. Azure Batch handles scheduling of discrete jobs and tasks accross pools of VM's. It is commonly used for batch processing tasks such as rendering.
The Virtual kubelet integration allows you to take advantage of this from within Kubernetes. The primary usecase for the provider is to make it easy to use GPU based workload from normal Kubernetes clusters. For example, creating Kubernetes Jobs which train or execute ML models using Nvidia GPU's or using FFMPEG.
Azure Batch allows for [low priority nodes](https://docs.microsoft.com/en-us/azure/batch/batch-low-pri-vms) which can also help to reduce cost for non-time sensitive workloads.
__The [ACI provider](../azure/README.md) is the best option unless you're looking to utilise some specific features of Azure Batch__.
## Status: Experimental
This provider is currently in the exterimental stages. Contributions welcome!
## Quick Start
The following Terraform template deploys an AKS cluster with the Virtual Kubelet, Azure Batch Account and GPU enabled Azure Batch pool. The Batch pool contains 1 Dedicated NC6 Node and 2 Low Priority NC6 Nodes.
1. Setup Terraform for Azure following [this guide here](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/terraform-install-configure)
2. From the commandline move to the deployment folder `cd ./providers/azurebatch/deployment` then edit `vars.example.tfvars` adding in your Service Principal details
3. Download the latest version of the Community Kubernetes Provider for Terraform. Get the correct link [from here](https://github.com/sl1pm4t/terraform-provider-kubernetes/releases) and use it as follows: (Current official Terraform K8s provider doesn't support `Deployments`)
```shell
curl -L -o - PUT_RELASE_BINARY_LINK_YOU_FOUND_HERE | gunzip > terraform-provider-kubernetes
chmod +x ./terraform-provider-kubernetes
```
4. Use `terraform init` to initialize the template
5. Use `terraform plan -var-file=./vars.example.tfvars` and `terraform apply -var-file=./vars.example.tfvars` to deploy the template
6. Run `kubectl describe deployment/vkdeployment` to check the virtual kubelet is running correctly.
7. Run `kubectl create -f examplegpupod.yaml`
8. Run `pods=$(kubectl get pods --selector=app=examplegpupod --show-all --output=jsonpath={.items..metadata.name})` then `kubectl logs $pods` to view the logs. Should see:
```text
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done
```
### Tweaking the Quickstart
You can update [main.tf](./main.tf) to increase the number of nodes allocated to the Azure Batch pool or update [./aks/main.tf](./aks/main.tf) to increase the number of agent nodes allocated to your AKS cluster.
## Advanced Setup
## Prerequistes
1. An Azure Batch Account configurated
2. An Azure Batch Pool created with necessary VM spec. VM's in the pool must have:
- `docker` installed and correctly configured
- `nvidia-docker` and `cuda` drivers installed
3. K8s cluster
4. Azure Service Principal with access to the Azure Batch Account
## Setup
The provider expects the following environment variables to be configured:
```
ClientID: AZURE_CLIENT_ID
ClientSecret: AZURE_CLIENT_SECRET
ResourceGroup: AZURE_RESOURCE_GROUP
SubscriptionID: AZURE_SUBSCRIPTION_ID
TenantID: AZURE_TENANT_ID
PoolID: AZURE_BATCH_POOLID
JobID (optional):AZURE_BATCH_JOBID
AccountLocation: AZURE_BATCH_ACCOUNT_LOCATION
AccountName: AZURE_BATCH_ACCOUNT_NAME
```
## Running
The provider will assign pods to machines in the Azure Batch Pool. Each machine can, by default, process only one pod at a time
running more than 1 pod per machine isn't currently supported and will result in errors.
Azure Batch queues tasks when no machines are available so pods will sit in `podPending` state while waiting for a VM to become available.

465
providers/azurebatch/batch.go
View File

@@ -1,465 +0,0 @@
package azurebatch
import (
"context"
"encoding/json"
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
"os"
"strings"
"github.com/Azure/azure-sdk-for-go/services/batch/2017-09-01.6.0/batch"
"github.com/Azure/go-autorest/autorest"
"github.com/Azure/go-autorest/autorest/azure"
"github.com/Azure/go-autorest/autorest/to"
"github.com/lawrencegripper/pod2docker"
vklog "github.com/virtual-kubelet/virtual-kubelet/log"
"github.com/virtual-kubelet/virtual-kubelet/manager"
"github.com/virtual-kubelet/virtual-kubelet/node/api"
v1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/resource"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
const (
podJSONKey string = "virtualkubelet_pod"
)
// Provider the base struct for the Azure Batch provider
type Provider struct {
batchConfig *Config
ctx context.Context
cancelCtx context.CancelFunc
fileClient *batch.FileClient
resourceManager *manager.ResourceManager
listTasks func() (*[]batch.CloudTask, error)
addTask func(batch.TaskAddParameter) (autorest.Response, error)
getTask func(taskID string) (batch.CloudTask, error)
deleteTask func(taskID string) (autorest.Response, error)
getFileFromTask func(taskID, path string) (result batch.ReadCloser, err error)
nodeName string
operatingSystem string
cpu string
memory string
pods string
internalIP string
daemonEndpointPort int32
}
// Config - Basic azure config used to interact with ARM resources.
type Config struct {
ClientID string
ClientSecret string
SubscriptionID string
TenantID string
ResourceGroup string
PoolID string
JobID string
AccountName string
AccountLocation string
}
// AcsCredential represents the credential file for ACS
type acsCredential struct {
Cloud string `json:"cloud"`
TenantID string `json:"tenantId"`
SubscriptionID string `json:"subscriptionId"`
ClientID string `json:"aadClientId"`
ClientSecret string `json:"aadClientSecret"`
ResourceGroup string `json:"resourceGroup"`
Region string `json:"location"`
VNetName string `json:"vnetName"`
VNetResourceGroup string `json:"vnetResourceGroup"`
}
func newACSCredential(p string) (*acsCredential, error) {
logger := vklog.G(context.TODO()).WithField("method", "NewAcsCredential").WithField("file", p)
logger.Debug("Reading ACS credential file")
b, err := ioutil.ReadFile(p)
if err != nil {
return nil, fmt.Errorf("Reading ACS credential file %q failed: %v", p, err)
}
// Unmarshal the authentication file.
var cred acsCredential
if err := json.Unmarshal(b, &cred); err != nil {
return nil, err
}
logger.Debug("Load ACS credential file successfully")
return &cred, nil
}
// NewBatchProvider Creates a batch provider
func NewBatchProvider(configString string, rm *manager.ResourceManager, nodeName, operatingSystem string, internalIP string, daemonEndpointPort int32) (*Provider, error) {
fmt.Println("Starting create provider")
config := &Config{}
if azureCredsFilepath := os.Getenv("AZURE_CREDENTIALS_LOCATION"); azureCredsFilepath != "" {
creds, err := newACSCredential(azureCredsFilepath)
if err != nil {
return nil, err
}
config.ClientID = creds.ClientID
config.ClientSecret = creds.ClientSecret
config.SubscriptionID = creds.SubscriptionID
config.TenantID = creds.TenantID
}
err := getAzureConfigFromEnv(config)
if err != nil {
log.Println("Failed to get auth information")
}
return NewBatchProviderFromConfig(config, rm, nodeName, operatingSystem, internalIP, daemonEndpointPort)
}
// NewBatchProviderFromConfig Creates a batch provider
func NewBatchProviderFromConfig(config *Config, rm *manager.ResourceManager, nodeName, operatingSystem string, internalIP string, daemonEndpointPort int32) (*Provider, error) {
p := Provider{}
p.batchConfig = config
// Set sane defaults for Capacity in case config is not supplied
p.cpu = "20"
p.memory = "100Gi"
p.pods = "20"
p.resourceManager = rm
p.operatingSystem = operatingSystem
p.nodeName = nodeName
p.internalIP = internalIP
p.daemonEndpointPort = daemonEndpointPort
p.ctx, p.cancelCtx = context.WithCancel(context.Background())
auth := getAzureADAuthorizer(config, azure.PublicCloud.BatchManagementEndpoint)
batchBaseURL := getBatchBaseURL(config.AccountName, config.AccountLocation)
_, err := getPool(p.ctx, batchBaseURL, config.PoolID, auth)
if err != nil {
log.Panicf("Error retreiving Azure Batch pool: %v", err)
}
_, err = createOrGetJob(p.ctx, batchBaseURL, config.JobID, config.PoolID, auth)
if err != nil {
log.Panicf("Error retreiving/creating Azure Batch job: %v", err)
}
taskClient := batch.NewTaskClientWithBaseURI(batchBaseURL)
taskClient.Authorizer = auth
p.listTasks = func() (*[]batch.CloudTask, error) {
res, err := taskClient.List(p.ctx, config.JobID, "", "", "", nil, nil, nil, nil, nil)
if err != nil {
return &[]batch.CloudTask{}, err
}
currentTasks := res.Values()
for res.NotDone() {
err = res.Next()
if err != nil {
return &[]batch.CloudTask{}, err
}
pageTasks := res.Values()
if pageTasks != nil || len(pageTasks) != 0 {
currentTasks = append(currentTasks, pageTasks...)
}
}
return &currentTasks, nil
}
p.addTask = func(task batch.TaskAddParameter) (autorest.Response, error) {
return taskClient.Add(p.ctx, config.JobID, task, nil, nil, nil, nil)
}
p.getTask = func(taskID string) (batch.CloudTask, error) {
return taskClient.Get(p.ctx, config.JobID, taskID, "", "", nil, nil, nil, nil, "", "", nil, nil)
}
p.deleteTask = func(taskID string) (autorest.Response, error) {
return taskClient.Delete(p.ctx, config.JobID, taskID, nil, nil, nil, nil, "", "", nil, nil)
}
p.getFileFromTask = func(taskID, path string) (batch.ReadCloser, error) {
return p.fileClient.GetFromTask(p.ctx, config.JobID, taskID, path, nil, nil, nil, nil, "", nil, nil)
}
fileClient := batch.NewFileClientWithBaseURI(batchBaseURL)
fileClient.Authorizer = auth
p.fileClient = &fileClient
return &p, nil
}
// CreatePod accepts a Pod definition
func (p *Provider) CreatePod(ctx context.Context, pod *v1.Pod) error {
log.Println("Creating pod...")
podCommand, err := pod2docker.GetBashCommand(pod2docker.PodComponents{
InitContainers: pod.Spec.InitContainers,
Containers: pod.Spec.Containers,
PodName: pod.Name,
Volumes: pod.Spec.Volumes,
})
if err != nil {
return err
}
bytes, err := json.Marshal(pod)
if err != nil {
panic(err)
}
task := batch.TaskAddParameter{
DisplayName: to.StringPtr(string(pod.UID)),
ID: to.StringPtr(getTaskIDForPod(pod.Namespace, pod.Name)),
CommandLine: to.StringPtr(fmt.Sprintf(`/bin/bash -c "%s"`, podCommand)),
UserIdentity: &batch.UserIdentity{
AutoUser: &batch.AutoUserSpecification{
ElevationLevel: batch.Admin,
Scope: batch.Pool,
},
},
EnvironmentSettings: &[]batch.EnvironmentSetting{
{
Name: to.StringPtr(podJSONKey),
Value: to.StringPtr(string(bytes)),
},
},
}
_, err = p.addTask(task)
if err != nil {
return err
}
return nil
}
// GetPodStatus retrieves the status of a given pod by name.
func (p *Provider) GetPodStatus(ctx context.Context, namespace, name string) (*v1.PodStatus, error) {
log.Println("Getting pod status ....")
pod, err := p.GetPod(ctx, namespace, name)
if err != nil {
return nil, err
}
if pod == nil {
return nil, nil
}
return &pod.Status, nil
}
// UpdatePod accepts a Pod definition
func (p *Provider) UpdatePod(ctx context.Context, pod *v1.Pod) error {
log.Println("Pod Update called: No-op as not implemented")
return nil
}
// DeletePod accepts a Pod definition
func (p *Provider) DeletePod(ctx context.Context, pod *v1.Pod) error {
taskID := getTaskIDForPod(pod.Namespace, pod.Name)
task, err := p.deleteTask(taskID)
if err != nil {
log.Println(task)
log.Println(err)
return wrapError(err)
}
log.Printf(fmt.Sprintf("Deleting task: %v", taskID))
return nil
}
// GetPod returns a pod by name
func (p *Provider) GetPod(ctx context.Context, namespace, name string) (*v1.Pod, error) {
log.Println("Getting Pod ...")
task, err := p.getTask(getTaskIDForPod(namespace, name))
if err != nil {
if task.Response.StatusCode == http.StatusNotFound {
return nil, nil
}
log.Println(err)
return nil, err
}
pod, err := getPodFromTask(&task)
if err != nil {
panic(err)
}
status, _ := convertTaskToPodStatus(&task)
pod.Status = *status
return pod, nil
}
const (
startingUpHeader = "Container still starting..\nShowing startup logs from Azure Batch node instead:\n"
stdoutHeader = "----- STDOUT -----\n"
stderrHeader = "----- STDERR -----\n"
)
// GetContainerLogs returns the logs of a container running in a pod by name.
func (p *Provider) GetContainerLogs(ctx context.Context, namespace, podName, containerName string, opts api.ContainerLogOpts) (io.ReadCloser, error) {
log.Println("Getting pod logs ....")
taskID := getTaskIDForPod(namespace, podName)
logFileLocation := fmt.Sprintf("wd/%s.log", containerName)
containerLogReader, err := p.getFileFromTask(taskID, logFileLocation)
if containerLogReader.Response.Response != nil && containerLogReader.StatusCode == http.StatusNotFound {
stdoutReader, err := p.getFileFromTask(taskID, "stdout.txt")
if err != nil {
return nil, err
}
stderrReader, err := p.getFileFromTask(taskID, "stderr.txt")
if err != nil {
return nil, err
}
stdout := io.MultiReader(strings.NewReader(startingUpHeader), strings.NewReader(stdoutHeader), *stdoutReader.Value, strings.NewReader("\n"))
stderr := io.MultiReader(strings.NewReader(stderrHeader), *stderrReader.Value, strings.NewReader("\n"))
return &readCloser{
Reader: io.MultiReader(stdout, stderr),
closer: func() error {
(*stdoutReader.Value).Close()
(*stderrReader.Value).Close()
return nil
}}, nil
}
if err != nil {
return nil, err
}
// TODO(@cpuguy83): don't convert stream to a string
result, err := formatLogJSON(containerLogReader)
if err != nil {
return nil, fmt.Errorf("Container log formating failed err: %v", err)
}
return ioutil.NopCloser(strings.NewReader(result)), nil
}
type readCloser struct {
io.Reader
closer func() error
}
func (r *readCloser) Close() error {
return r.closer()
}
// Get full pod name as defined in the provider context
// TODO: Implementation
func (p *Provider) GetPodFullName(namespace string, pod string) string {
return ""
}
// RunInContainer executes a command in a container in the pod, copying data
// between in/out/err and the container's stdin/stdout/stderr.
// TODO: Implementation
func (p *Provider) RunInContainer(ctx context.Context, namespace, name, container string, cmd []string, attach api.AttachIO) error {
log.Printf("receive ExecInContainer %q\n", container)
return nil
}
// GetPods retrieves a list of all pods scheduled to run.
func (p *Provider) GetPods(ctx context.Context) ([]*v1.Pod, error) {
log.Println("Getting pods...")
tasksPtr, err := p.listTasks()
if err != nil {
panic(err)
}
if tasksPtr == nil {
return []*v1.Pod{}, nil
}
tasks := *tasksPtr
pods := make([]*v1.Pod, len(tasks), len(tasks))
for i, t := range tasks {
pod, err := getPodFromTask(&t)
if err != nil {
panic(err)
}
pods[i] = pod
}
return pods, nil
}
// Capacity returns a resource list containing the capacity limits
func (p *Provider) Capacity(ctx context.Context) v1.ResourceList {
return v1.ResourceList{
"cpu": resource.MustParse(p.cpu),
"memory": resource.MustParse(p.memory),
"pods": resource.MustParse(p.pods),
"nvidia.com/gpu": resource.MustParse("1"),
}
}
// NodeConditions returns a list of conditions (Ready, OutOfDisk, etc), for updates to the node status
// within Kubernetes.
func (p *Provider) NodeConditions(ctx context.Context) []v1.NodeCondition {
return []v1.NodeCondition{
{
Type: "Ready",
Status: v1.ConditionTrue,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletReady",
Message: "kubelet is ready.",
},
{
Type: "OutOfDisk",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletHasSufficientDisk",
Message: "kubelet has sufficient disk space available",
},
{
Type: "MemoryPressure",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletHasSufficientMemory",
Message: "kubelet has sufficient memory available",
},
{
Type: "DiskPressure",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletHasNoDiskPressure",
Message: "kubelet has no disk pressure",
},
{
Type: "NetworkUnavailable",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "RouteCreated",
Message: "RouteController created a route",
},
}
}
// NodeAddresses returns a list of addresses for the node status
// within Kubernetes.
func (p *Provider) NodeAddresses(ctx context.Context) []v1.NodeAddress {
// TODO: Make these dynamic and augment with custom ACI specific conditions of interest
return []v1.NodeAddress{
{
Type: "InternalIP",
Address: p.internalIP,
},
}
}
// NodeDaemonEndpoints returns NodeDaemonEndpoints for the node status
// within Kubernetes.
func (p *Provider) NodeDaemonEndpoints(ctx context.Context) *v1.NodeDaemonEndpoints {
return &v1.NodeDaemonEndpoints{
KubeletEndpoint: v1.DaemonEndpoint{
Port: p.daemonEndpointPort,
},
}
}
// OperatingSystem returns the operating system for this provider.
func (p *Provider) OperatingSystem() string {
return p.operatingSystem
}

216
providers/azurebatch/batch_helpers.go
View File

@@ -1,216 +0,0 @@
package azurebatch
import (
"bufio"
"context"
"crypto/md5"
"encoding/json"
"fmt"
"log"
"os"
"strings"
"time"
"github.com/Azure/azure-sdk-for-go/services/batch/2017-09-01.6.0/batch"
"github.com/Azure/go-autorest/autorest"
"github.com/Azure/go-autorest/autorest/adal"
"github.com/Azure/go-autorest/autorest/azure"
)
func mustWriteString(builder *strings.Builder, s string) {
_, err := builder.WriteString(s)
if err != nil {
panic(err)
}
}
func mustWrite(builder *strings.Builder, b []byte) {
_, err := builder.Write(b)
if err != nil {
panic(err)
}
}
// NewServicePrincipalTokenFromCredentials creates a new ServicePrincipalToken using values of the
// passed credentials map.
func newServicePrincipalTokenFromCredentials(c *Config, scope string) (*adal.ServicePrincipalToken, error) {
oauthConfig, err := adal.NewOAuthConfig(azure.PublicCloud.ActiveDirectoryEndpoint, c.TenantID)
if err != nil {
panic(err)
}
return adal.NewServicePrincipalToken(*oauthConfig, c.ClientID, c.ClientSecret, scope)
}
// GetAzureADAuthorizer return an authorizor for Azure SP
func getAzureADAuthorizer(c *Config, azureEndpoint string) autorest.Authorizer {
spt, err := newServicePrincipalTokenFromCredentials(c, azureEndpoint)
if err != nil {
panic(fmt.Sprintf("Failed to create authorizer: %v", err))
}
auth := autorest.NewBearerAuthorizer(spt)
return auth
}
func getPool(ctx context.Context, batchBaseURL, poolID string, auth autorest.Authorizer) (*batch.PoolClient, error) {
poolClient := batch.NewPoolClientWithBaseURI(batchBaseURL)
poolClient.Authorizer = auth
poolClient.RetryAttempts = 0
pool, err := poolClient.Get(ctx, poolID, "*", "", nil, nil, nil, nil, "", "", nil, nil)
// If we observe an error which isn't related to the pool not existing panic.
// 404 is expected if this is first run.
if err != nil && pool.Response.Response == nil {
log.Printf("Failed to get pool. nil response %v", poolID)
return nil, err
} else if err != nil && pool.StatusCode == 404 {
log.Printf("Pool doesn't exist 404 received Error: %v PoolID: %v", err, poolID)
return nil, err
} else if err != nil {
log.Printf("Failed to get pool. Response:%v", pool.Response)
return nil, err
}
if pool.State == batch.PoolStateActive {
log.Println("Pool active and running...")
return &poolClient, nil
}
return nil, fmt.Errorf("Pool not in active state: %v", pool.State)
}
func createOrGetJob(ctx context.Context, batchBaseURL, jobID, poolID string, auth autorest.Authorizer) (*batch.JobClient, error) {
jobClient := batch.NewJobClientWithBaseURI(batchBaseURL)
jobClient.Authorizer = auth
// check if job exists already
currentJob, err := jobClient.Get(ctx, jobID, "", "", nil, nil, nil, nil, "", "", nil, nil)
if err == nil && currentJob.State == batch.JobStateActive {
log.Println("Wrapper job already exists...")
return &jobClient, nil
} else if currentJob.Response.StatusCode == 404 {
log.Println("Wrapper job missing... creating...")
wrapperJob := batch.JobAddParameter{
ID: &jobID,
PoolInfo: &batch.PoolInformation{
PoolID: &poolID,
},
}
_, err := jobClient.Add(ctx, wrapperJob, nil, nil, nil, nil)
if err != nil {
return nil, err
}
return &jobClient, nil
} else if currentJob.State == batch.JobStateDeleting {
log.Printf("Job is being deleted... Waiting then will retry")
time.Sleep(time.Minute)
return createOrGetJob(ctx, batchBaseURL, jobID, poolID, auth)
}
return nil, err
}
func getBatchBaseURL(batchAccountName, batchAccountLocation string) string {
return fmt.Sprintf("https://%s.%s.batch.azure.com", batchAccountName, batchAccountLocation)
}
func envHasValue(env string) bool {
val := os.Getenv(env)
if val == "" {
return false
}
return true
}
// GetConfigFromEnv - Retreives the azure configuration from environment variables.
func getAzureConfigFromEnv(config *Config) error {
if envHasValue("AZURE_CLIENT_ID") {
config.ClientID = os.Getenv("AZURE_CLIENT_ID")
}
if envHasValue("AZURE_CLIENT_SECRET") {
config.ClientSecret = os.Getenv("AZURE_CLIENT_SECRET")
}
if envHasValue("AZURE_RESOURCE_GROUP") {
config.ResourceGroup = os.Getenv("AZURE_RESOURCE_GROUP")
}
if envHasValue("AZURE_SUBSCRIPTION_ID") {
config.SubscriptionID = os.Getenv("AZURE_SUBSCRIPTION_ID")
}
if envHasValue("AZURE_TENANT_ID") {
config.TenantID = os.Getenv("AZURE_TENANT_ID")
}
if envHasValue("AZURE_BATCH_POOLID") {
config.PoolID = os.Getenv("AZURE_BATCH_POOLID")
}
if envHasValue("AZURE_BATCH_JOBID") {
config.JobID = os.Getenv("AZURE_BATCH_JOBID")
}
if envHasValue("AZURE_BATCH_ACCOUNT_LOCATION") {
config.AccountLocation = os.Getenv("AZURE_BATCH_ACCOUNT_LOCATION")
}
if envHasValue("AZURE_BATCH_ACCOUNT_NAME") {
config.AccountName = os.Getenv("AZURE_BATCH_ACCOUNT_NAME")
}
if config.JobID == "" {
hostname, err := os.Hostname()
if err != nil {
log.Panic(err)
}
config.JobID = hostname
}
if config.ClientID == "" ||
config.ClientSecret == "" ||
config.ResourceGroup == "" ||
config.SubscriptionID == "" ||
config.PoolID == "" ||
config.TenantID == "" {
return &ConfigError{CurrentConfig: config, ErrorDetails: "Missing configuration"}
}
return nil
}
func getTaskIDForPod(namespace, name string) string {
ID := []byte(fmt.Sprintf("%s-%s", namespace, name))
return string(fmt.Sprintf("%x", md5.Sum(ID)))
}
type jsonLog struct {
Log string `json:"log"`
}
func formatLogJSON(readCloser batch.ReadCloser) (string, error) {
//Read line by line as file isn't valid json. Each line is a single valid json object.
scanner := bufio.NewScanner(*readCloser.Value)
var b strings.Builder
for scanner.Scan() {
result := jsonLog{}
err := json.Unmarshal(scanner.Bytes(), &result)
if err != nil {
return "", err
}
mustWriteString(&b, result.Log)
}
return b.String(), nil
}
// ConfigError - Error when reading configuration values.
type ConfigError struct {
CurrentConfig *Config
ErrorDetails string
}
func (e *ConfigError) Error() string {
configJSON, err := json.Marshal(e.CurrentConfig)
if err != nil {
return e.ErrorDetails
}
return e.ErrorDetails + ": " + string(configJSON)
}

141
providers/azurebatch/batch_status.go
View File

@@ -1,141 +0,0 @@
package azurebatch
import (
"encoding/json"
"fmt"
"github.com/Azure/azure-sdk-for-go/services/batch/2017-09-01.6.0/batch"
apiv1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
func getPodFromTask(task *batch.CloudTask) (pod *apiv1.Pod, err error) {
if task == nil || task.EnvironmentSettings == nil {
return nil, fmt.Errorf("invalid input task: %v", task)
}
ok := false
jsonData := ""
settings := *task.EnvironmentSettings
for _, s := range settings {
if *s.Name == podJSONKey && s.Value != nil {
ok = true
jsonData = *s.Value
}
}
if !ok {
return nil, fmt.Errorf("task doesn't have pod json stored in it: %v", task.EnvironmentSettings)
}
pod = &apiv1.Pod{}
err = json.Unmarshal([]byte(jsonData), pod)
if err != nil {
return nil, err
}
return
}
func convertTaskToPodStatus(task *batch.CloudTask) (status *apiv1.PodStatus, err error) {
pod, err := getPodFromTask(task)
if err != nil {
return
}
// Todo: Review indivudal container status response
status = &apiv1.PodStatus{
Phase: convertTaskStatusToPodPhase(task),
Conditions: []apiv1.PodCondition{},
Message: "",
Reason: "",
HostIP: "",
PodIP: "127.0.0.1",
StartTime: &pod.CreationTimestamp,
}
for _, container := range pod.Spec.Containers {
containerStatus := apiv1.ContainerStatus{
Name: container.Name,
State: convertTaskStatusToContainerState(task),
Ready: true,
RestartCount: 0,
Image: container.Image,
ImageID: "",
ContainerID: "",
}
status.ContainerStatuses = append(status.ContainerStatuses, containerStatus)
}
return
}
func convertTaskStatusToPodPhase(t *batch.CloudTask) (podPhase apiv1.PodPhase) {
switch t.State {
case batch.TaskStatePreparing:
podPhase = apiv1.PodPending
case batch.TaskStateActive:
podPhase = apiv1.PodPending
case batch.TaskStateRunning:
podPhase = apiv1.PodRunning
case batch.TaskStateCompleted:
podPhase = apiv1.PodFailed
if t.ExecutionInfo != nil && t.ExecutionInfo.ExitCode != nil && *t.ExecutionInfo.ExitCode == 0 {
podPhase = apiv1.PodSucceeded
}
}
return
}
func convertTaskStatusToContainerState(t *batch.CloudTask) (containerState apiv1.ContainerState) {
startTime := metav1.Time{}
if t.ExecutionInfo != nil {
if t.ExecutionInfo.StartTime != nil {
startTime.Time = t.ExecutionInfo.StartTime.Time
}
}
switch t.State {
case batch.TaskStatePreparing:
containerState = apiv1.ContainerState{
Waiting: &apiv1.ContainerStateWaiting{
Message: "Waiting for machine in AzureBatch",
Reason: "Preparing",
},
}
case batch.TaskStateActive:
containerState = apiv1.ContainerState{
Waiting: &apiv1.ContainerStateWaiting{
Message: "Waiting for machine in AzureBatch",
Reason: "Queued",
},
}
case batch.TaskStateRunning:
containerState = apiv1.ContainerState{
Running: &apiv1.ContainerStateRunning{
StartedAt: startTime,
},
}
case batch.TaskStateCompleted:
termStatus := apiv1.ContainerState{
Terminated: &apiv1.ContainerStateTerminated{
FinishedAt: metav1.Time{
Time: t.StateTransitionTime.Time,
},
StartedAt: startTime,
},
}
if t.ExecutionInfo != nil && t.ExecutionInfo.ExitCode != nil {
exitCode := *t.ExecutionInfo.ExitCode
termStatus.Terminated.ExitCode = exitCode
if exitCode != 0 {
termStatus.Terminated.Message = *t.ExecutionInfo.FailureInfo.Message
}
}
}
return
}

167
providers/azurebatch/batch_status_test.go
View File

@@ -1,167 +0,0 @@
package azurebatch
import (
"reflect"
"testing"
"github.com/Azure/go-autorest/autorest/to"
"github.com/Azure/azure-sdk-for-go/services/batch/2017-09-01.6.0/batch"
apiv1 "k8s.io/api/core/v1"
)
func Test_getPodFromTask(t *testing.T) {
type args struct {
task *batch.CloudTask
}
tests := []struct {
name string
task batch.CloudTask
wantPod *apiv1.Pod
wantErr bool
}{
{
name: "SimplePod",
task: batch.CloudTask{
EnvironmentSettings: &[]batch.EnvironmentSetting{
{
Name: to.StringPtr(podJSONKey),
Value: to.StringPtr(`{"metadata":{"creationTimestamp":null},"spec":{"containers":[{"name":"web","image":"nginx:1.12","ports":[{"name":"http","containerPort":80,"protocol":"TCP"}],"resources":{}}]},"status":{}}`),
},
},
},
wantPod: &apiv1.Pod{
Spec: apiv1.PodSpec{
Containers: []apiv1.Container{
{
Name: "web",
Image: "nginx:1.12",
Ports: []apiv1.ContainerPort{
{
Name: "http",
Protocol: apiv1.ProtocolTCP,
ContainerPort: 80,
},
},
},
},
},
},
},
{
name: "InvalidJson",
task: batch.CloudTask{
EnvironmentSettings: &[]batch.EnvironmentSetting{
{
Name: to.StringPtr(podJSONKey),
Value: to.StringPtr("---notjson--"),
},
},
},
wantErr: true,
},
{
name: "NilEnvironment",
task: batch.CloudTask{
EnvironmentSettings: nil,
},
wantErr: true,
},
{
name: "NilString",
task: batch.CloudTask{
EnvironmentSettings: &[]batch.EnvironmentSetting{
{
Name: to.StringPtr(podJSONKey),
Value: nil,
},
},
},
wantErr: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
gotPod, err := getPodFromTask(&tt.task)
if (err != nil) != tt.wantErr {
t.Errorf("getPodFromTask() error = %v, wantErr %v", err, tt.wantErr)
return
}
if !reflect.DeepEqual(gotPod, tt.wantPod) {
t.Errorf("getPodFromTask() = %v, want %v", gotPod, tt.wantPod)
}
})
}
}
func Test_convertTaskStatusToPodPhase(t *testing.T) {
type args struct {
t *batch.CloudTask
}
tests := []struct {
name string
task batch.CloudTask
wantPodPhase apiv1.PodPhase
}{
{
name: "PreparingTask",
task: batch.CloudTask{
State: batch.TaskStatePreparing,
},
wantPodPhase: apiv1.PodPending,
},
{
//Active tasks are sitting in a queue waiting for a node
// so maps best to pending state
name: "ActiveTask",
task: batch.CloudTask{
State: batch.TaskStateActive,
},
wantPodPhase: apiv1.PodPending,
},
{
name: "RunningTask",
task: batch.CloudTask{
State: batch.TaskStateRunning,
},
wantPodPhase: apiv1.PodRunning,
},
{
name: "CompletedTask_ExitCode0",
task: batch.CloudTask{
State: batch.TaskStateCompleted,
ExecutionInfo: &batch.TaskExecutionInformation{
ExitCode: to.Int32Ptr(0),
},
},
wantPodPhase: apiv1.PodSucceeded,
},
{
name: "CompletedTask_ExitCode127",
task: batch.CloudTask{
State: batch.TaskStateCompleted,
ExecutionInfo: &batch.TaskExecutionInformation{
ExitCode: to.Int32Ptr(127),
},
},
wantPodPhase: apiv1.PodFailed,
},
{
name: "CompletedTask_nilExecInfo",
task: batch.CloudTask{
State: batch.TaskStateCompleted,
ExecutionInfo: nil,
},
wantPodPhase: apiv1.PodFailed,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if gotPodPhase := convertTaskStatusToPodPhase(&tt.task); !reflect.DeepEqual(gotPodPhase, tt.wantPodPhase) {
t.Errorf("convertTaskStatusToPodPhase() = %v, want %v", gotPodPhase, tt.wantPodPhase)
}
})
}
}

180
providers/azurebatch/batch_test.go
View File

@@ -1,180 +0,0 @@
package azurebatch
import (
"context"
"crypto/md5"
"fmt"
"io/ioutil"
"net/http"
"os"
"strings"
"testing"
"github.com/Azure/azure-sdk-for-go/services/batch/2017-09-01.6.0/batch"
"github.com/Azure/go-autorest/autorest"
"github.com/virtual-kubelet/virtual-kubelet/node/api"
apiv1 "k8s.io/api/core/v1"
)
func Test_deletePod(t *testing.T) {
podNamespace := "bob"
podName := "marley"
concatName := []byte(fmt.Sprintf("%s-%s", podNamespace, podName))
expectedDeleteTaskID := fmt.Sprintf("%x", md5.Sum(concatName))
provider := Provider{}
provider.deleteTask = func(taskID string) (autorest.Response, error) {
if taskID != expectedDeleteTaskID {
t.Errorf("Deleted wrong task! Expected delete: %v Actual: %v", taskID, expectedDeleteTaskID)
}
return autorest.Response{}, nil
}
pod := &apiv1.Pod{}
pod.Name = podName
pod.Namespace = podNamespace
err := provider.DeletePod(context.Background(), pod)
if err != nil {
t.Error(err)
}
}
func Test_deletePod_doesntExist(t *testing.T) {
pod := &apiv1.Pod{}
pod.Namespace = "bob"
pod.Name = "marley"
provider := Provider{}
provider.deleteTask = func(taskID string) (autorest.Response, error) {
return autorest.Response{}, fmt.Errorf("Task doesn't exist")
}
err := provider.DeletePod(context.Background(), pod)
if err == nil {
t.Error("Expected error but none seen")
}
}
func Test_createPod(t *testing.T) {
pod := &apiv1.Pod{}
pod.Namespace = "bob"
pod.Name = "marley"
provider := Provider{}
provider.addTask = func(task batch.TaskAddParameter) (autorest.Response, error) {
if task.CommandLine == nil || *task.CommandLine == "" {
t.Error("Missing commandline args")
}
derefVars := *task.EnvironmentSettings
if len(derefVars) != 1 || *derefVars[0].Name != podJSONKey {
t.Error("Missing pod json")
}
return autorest.Response{}, nil
}
err := provider.CreatePod(context.Background(), pod)
if err != nil {
t.Errorf("Unexpected error creating pod %v", err)
}
}
func Test_createPod_errorResponse(t *testing.T) {
pod := &apiv1.Pod{}
pod.Namespace = "bob"
pod.Name = "marley"
provider := Provider{}
provider.addTask = func(task batch.TaskAddParameter) (autorest.Response, error) {
return autorest.Response{}, fmt.Errorf("Failed creating task")
}
err := provider.CreatePod(context.Background(), pod)
if err == nil {
t.Error("Expected error but none seen")
}
}
func Test_readLogs_404Response_expectReturnStartupLogs(t *testing.T) {
pod := &apiv1.Pod{}
pod.Namespace = "bob"
pod.Name = "marley"
containerName := "sam"
provider := Provider{}
provider.getFileFromTask = func(taskID, path string) (batch.ReadCloser, error) {
if path == "wd/sam.log" {
// Autorest - Seriously? Can't find a better way to make a 404 :(
return batch.ReadCloser{Response: autorest.Response{Response: &http.Response{StatusCode: 404}}}, nil
} else if path == "stderr.txt" {
response := ioutil.NopCloser(strings.NewReader("stderrResponse"))
return batch.ReadCloser{Value: &response}, nil
} else if path == "stdout.txt" {
response := ioutil.NopCloser(strings.NewReader("stdoutResponse"))
return batch.ReadCloser{Value: &response}, nil
} else {
t.Errorf("Unexpected Filepath: %v", path)
}
return batch.ReadCloser{}, fmt.Errorf("Failed in test mock of getFileFromTask")
}
logs, err := provider.GetContainerLogs(context.Background(), pod.Namespace, pod.Name, containerName, api.ContainerLogOpts{})
if err != nil {
t.Fatalf("GetContainerLogs return error: %v", err)
}
defer logs.Close()
r, err := ioutil.ReadAll(logs)
if err != nil {
t.Fatal(err)
}
result := string(r)
if !strings.Contains(result, "stderrResponse") || !strings.Contains(result, "stdoutResponse") {
t.Errorf("Result didn't contain expected content have: %v", result)
}
}
func Test_readLogs_JsonResponse_expectFormattedLogs(t *testing.T) {
pod := &apiv1.Pod{}
pod.Namespace = "bob"
pod.Name = "marley"
containerName := "sam"
provider := Provider{}
provider.getFileFromTask = func(taskID, path string) (batch.ReadCloser, error) {
if path == "wd/sam.log" {
fileReader, err := os.Open("./testdata/logresponse.json")
if err != nil {
t.Error(err)
}
readCloser := ioutil.NopCloser(fileReader)
return batch.ReadCloser{Value: &readCloser, Response: autorest.Response{Response: &http.Response{StatusCode: 200}}}, nil
}
t.Errorf("Unexpected Filepath: %v", path)
return batch.ReadCloser{}, fmt.Errorf("Failed in test mock of getFileFromTask")
}
logs, err := provider.GetContainerLogs(context.Background(), pod.Namespace, pod.Name, containerName, api.ContainerLogOpts{})
if err != nil {
t.Errorf("GetContainerLogs return error: %v", err)
}
defer logs.Close()
r, err := ioutil.ReadAll(logs)
if err != nil {
t.Fatal(err)
}
result := string(r)
if !strings.Contains(string(result), "Copy output data from the CUDA device to the host memory") || strings.Contains(result, "{") {
t.Errorf("Result didn't contain expected content have or had json: %v", result)
}
}

59
providers/azurebatch/deployment/aks/main.tf
View File

@@ -1,59 +0,0 @@
resource "random_id" "workspace" {
keepers = {
# Generate a new id each time we switch to a new resource group
group_name = "${var.resource_group_name}"
}
byte_length = 8
}
#an attempt to keep the AKS name (and dns label) somewhat unique
resource "random_integer" "random_int" {
min = 100
max = 999
}
resource "azurerm_kubernetes_cluster" "aks" {
name = "aks-${random_integer.random_int.result}"
location = "${var.resource_group_location}"
dns_prefix = "aks-${random_integer.random_int.result}"
resource_group_name = "${var.resource_group_name}"
kubernetes_version = "1.9.2"
linux_profile {
admin_username = "${var.linux_admin_username}"
ssh_key {
key_data = "${var.linux_admin_ssh_publickey}"
}
}
agent_pool_profile {
name = "agentpool"
count = "2"
vm_size = "Standard_DS2_v2"
os_type = "Linux"
}
service_principal {
client_id = "${var.client_id}"
client_secret = "${var.client_secret}"
}
}
output "cluster_client_certificate" {
value = "${base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_certificate)}"
}
output "cluster_client_key" {
value = "${base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_key)}"
}
output "cluster_ca" {
value = "${base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.cluster_ca_certificate)}"
}
output "host" {
value = "${azurerm_kubernetes_cluster.aks.kube_config.0.host}"
}

31
providers/azurebatch/deployment/aks/variables.tf
View File

@@ -1,31 +0,0 @@
variable "client_id" {
type = "string"
description = "Client ID"
}
variable "client_secret" {
type = "string"
description = "Client secret."
}
variable "resource_group_name" {
type = "string"
description = "Name of the azure resource group."
default = "akc-rg"
}
variable "resource_group_location" {
type = "string"
description = "Location of the azure resource group."
default = "eastus"
}
variable "linux_admin_username" {
type = "string"
description = "User name for authentication to the Kubernetes linux agent virtual machines in the cluster."
}
variable "linux_admin_ssh_publickey" {
type = "string"
description = "Configure all the linux virtual machines in the cluster with the SSH RSA public key string. The key should include three parts, for example 'ssh-rsa AAAAB...snip...UcyupgH azureuser@linuxvm'"
}

146
providers/azurebatch/deployment/azurebatch/main.tf
View File

@@ -1,146 +0,0 @@
resource "random_string" "batchname" {
keepers = {
# Generate a new id each time we switch to a new resource group
group_name = "${var.resource_group_name}"
}
length = 8
upper = false
special = false
number = false
}
resource "azurerm_template_deployment" "test" {
name = "tfdeployment"
resource_group_name = "${var.resource_group_name}"
# these key-value pairs are passed into the ARM Template's `parameters` block
parameters {
"batchAccountName" = "${random_string.batchname.result}"
"storageAccountID" = "${var.storage_account_id}"
"poolBoostrapScriptUrl" = "${var.pool_bootstrap_script_url}"
"location" = "${var.resource_group_location}"
"poolID" = "${var.pool_id}"
"vmSku" = "${var.vm_sku}"
"lowPriorityNodeCount" = "${var.low_priority_node_count}"
"dedicatedNodeCount" = "${var.dedicated_node_count}"
}
deployment_mode = "Incremental"
template_body = <<DEPLOY
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"batchAccountName": {
"type": "string",
"metadata": {
"description": "Batch Account Name"
}
},
"poolID": {
"type": "string",
"metadata": {
"description": "GPU Pool ID"
}
},
"dedicatedNodeCount": {
"type": "string"
},
"lowPriorityNodeCount": {
"type": "string"
},
"vmSku": {
"type": "string"
},
"storageAccountID": {
"type": "string"
},
"poolBoostrapScriptUrl": {
"type": "string"
},
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]",
"metadata": {
"description": "Location for all resources."
}
}
},
"resources": [
{
"type": "Microsoft.Batch/batchAccounts",
"name": "[parameters('batchAccountName')]",
"apiVersion": "2015-12-01",
"location": "[parameters('location')]",
"tags": {
"ObjectName": "[parameters('batchAccountName')]"
},
"properties": {
"autoStorage": {
"storageAccountId": "[parameters('storageAccountID')]"
}
}
},
{
"type": "Microsoft.Batch/batchAccounts/pools",
"name": "[concat(parameters('batchAccountName'), '/', parameters('poolID'))]",
"apiVersion": "2017-09-01",
"scale": null,
"properties": {
"vmSize": "STANDARD_NC6",
"interNodeCommunication": "Disabled",
"maxTasksPerNode": 1,
"taskSchedulingPolicy": {
"nodeFillType": "Spread"
},
"startTask": {
"commandLine": "/bin/bash -c ./init.sh",
"resourceFiles": [
{
"blobSource": "[parameters('poolBoostrapScriptUrl')]",
"fileMode": "777",
"filePath": "./init.sh"
}
],
"userIdentity": {
"autoUser": {
"elevationLevel": "Admin",
"scope": "Pool"
}
},
"waitForSuccess": true,
"maxTaskRetryCount": 0
},
"deploymentConfiguration": {
"virtualMachineConfiguration": {
"imageReference": {
"publisher": "Canonical",
"offer": "UbuntuServer",
"sku": "16.04-LTS",
"version": "latest"
},
"nodeAgentSkuId": "batch.node.ubuntu 16.04"
}
},
"scaleSettings": {
"fixedScale": {
"targetDedicatedNodes": "[parameters('dedicatedNodeCount')]",
"targetLowPriorityNodes": "[parameters('lowPriorityNodeCount')]",
"resizeTimeout": "PT15M"
}
}
},
"dependsOn": [
"[resourceId('Microsoft.Batch/batchAccounts', parameters('batchAccountName'))]"
]
}
]
}
DEPLOY
}
output "name" {
value = "${random_string.batchname.result}"
}

43
providers/azurebatch/deployment/azurebatch/variables.tf
View File

@@ -1,43 +0,0 @@
variable "pool_id" {
type = "string"
description = "Name of the Azure Batch pool to create."
default = "pool1"
}
variable "vm_sku" {
type = "string"
description = "VM SKU to use - Default to NC6 GPU SKU."
default = "STANDARD_NC6"
}
variable "pool_bootstrap_script_url" {
type = "string"
description = "Publicly accessible url used for boostrapping nodes in the pool. Installing GPU drivers, for example."
}
variable "storage_account_id" {
type = "string"
description = "Name of the storage account to be used by Azure Batch"
}
variable "resource_group_name" {
type = "string"
description = "Name of the azure resource group."
default = "akc-rg"
}
variable "resource_group_location" {
type = "string"
description = "Location of the azure resource group."
default = "eastus"
}
variable "low_priority_node_count" {
type = "string"
description = "The number of low priority nodes to allocate to the pool"
}
variable "dedicated_node_count" {
type = "string"
description = "The number dedicated nodes to allocate to the pool"
}

19
providers/azurebatch/deployment/examplegpupod.yaml
View File

@@ -1,19 +0,0 @@
apiVersion: v1
kind: Pod
metadata:
name: cuda-vector-add
labels:
app: examplegpupod
spec:
restartPolicy: OnFailure
containers:
- name: cuda-vector-add
# https://github.com/kubernetes/kubernetes/blob/v1.7.11/test/images/nvidia-cuda/Dockerfile
image: "k8s.gcr.io/cuda-vector-add:v0.1"
resources:
limits:
nvidia.com/gpu: 1 # requesting 1 GPU
nodeName: virtual-kubelet
tolerations:
- key: azure.com/batch
effect: NoSchedule

20
providers/azurebatch/deployment/examplenvidiasmi.yaml
View File

@@ -1,20 +0,0 @@
apiVersion: v1
kind: Pod
metadata:
name: exampegpujob
spec:
containers:
- image: nvidia/cuda
command: ["nvidia-smi"]
imagePullPolicy: Always
name: nvidia
resources:
requests:
memory: 1G
cpu: 1
limits:
nvidia.com/gpu: 1 # requesting 1 GPU
nodeName: virtual-kubelet
tolerations:
- key: azure.com/batch
effect: NoSchedule

53
providers/azurebatch/deployment/main.tf
View File

@@ -1,53 +0,0 @@
resource "azurerm_resource_group" "batchrg" {
name = "${var.resource_group_name}"
location = "${var.resource_group_location}"
}
module "aks" {
source = "aks"
//Defaults to using current ssh key: recomend changing as needed
linux_admin_username = "aks"
linux_admin_ssh_publickey = "${file("~/.ssh/id_rsa.pub")}"
client_id = "${var.client_id}"
client_secret = "${var.client_secret}"
resource_group_name = "${azurerm_resource_group.batchrg.name}"
resource_group_location = "${azurerm_resource_group.batchrg.location}"
}
module "storage" {
source = "storage"
pool_bootstrap_script_path = "./scripts/poolstartup.sh"
resource_group_name = "${azurerm_resource_group.batchrg.name}"
resource_group_location = "${azurerm_resource_group.batchrg.location}"
}
module "azurebatch" {
source = "azurebatch"
storage_account_id = "${module.storage.id}"
pool_bootstrap_script_url = "${module.storage.pool_boostrap_script_url}"
resource_group_name = "${azurerm_resource_group.batchrg.name}"
resource_group_location = "${azurerm_resource_group.batchrg.location}"
dedicated_node_count = 1
low_priority_node_count = 2
}
module "virtualkubelet" {
source = "virtualkubelet"
virtualkubelet_docker_image = "${var.virtualkubelet_docker_image}"
cluster_client_key = "${module.aks.cluster_client_key}"
cluster_client_certificate = "${module.aks.cluster_client_certificate}"
cluster_ca = "${module.aks.cluster_ca}"
cluster_host = "${module.aks.host}"
azure_batch_account_name = "${module.azurebatch.name}"
resource_group_location = "${azurerm_resource_group.batchrg.location}"
}

49
providers/azurebatch/deployment/scripts/poolstartup.sh
View File

@@ -1,49 +0,0 @@
export DEBIAN_FRONTEND=noninteractive
export TEMP_DISK=/mnt
apt-get install -y -q --no-install-recommends \
build-essential
# Add dockerce repo
apt-get update -y -q --no-install-recommends
apt-get install -y -q -o Dpkg::Options::="--force-confnew" --no-install-recommends \
apt-transport-https ca-certificates curl software-properties-common cgroup-lite
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
apt-get update
#Install latest cuda driver..
CUDA_REPO_PKG=cuda-repo-ubuntu1604_9.1.85-1_amd64.deb
wget -O /tmp/${CUDA_REPO_PKG} http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/${CUDA_REPO_PKG}
sudo dpkg -i /tmp/${CUDA_REPO_PKG}
sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
rm -f /tmp/${CUDA_REPO_PKG}
sudo apt-get update -y -q --no-install-recommends
sudo apt-get install cuda-drivers -y -q --no-install-recommends
# install nvidia-docker
curl -fSsL https://nvidia.github.io/nvidia-docker/gpgkey | apt-key add -
curl -fSsL https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/nvidia-docker.list | \
tee /etc/apt/sources.list.d/nvidia-docker.list
apt-get update -y -q --no-install-recommends
apt-get install -y -q --no-install-recommends -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confnew" nvidia-docker2
systemctl restart docker.service
nvidia-docker version
# prep docker
systemctl stop docker.service
rm -rf /var/lib/docker
mkdir -p /etc/docker
mkdir -p $TEMPDISK/docker
chmod 777 $TEMPDISK/docker
echo "{ \"data-root\": \"$TEMP_DISK/docker\", \"hosts\": [ \"unix:///var/run/docker.sock\", \"tcp://127.0.0.1:2375\" ] }" > /etc/docker/daemon.json.merge
python -c "import json;a=json.load(open('/etc/docker/daemon.json.merge'));b=json.load(open('/etc/docker/daemon.json'));a.update(b);f=open('/etc/docker/daemon.json','w');json.dump(a,f);f.close();"
rm -f /etc/docker/daemon.json.merge
sed -i 's|^ExecStart=/usr/bin/dockerd.*|ExecStart=/usr/bin/dockerd|' /lib/systemd/system/docker.service
systemctl daemon-reload
systemctl start docker.service

77
providers/azurebatch/deployment/storage/main.tf
View File

@@ -1,77 +0,0 @@
resource "random_string" "storage" {
keepers = {
# Generate a new id each time we switch to a new resource group
group_name = "${var.resource_group_name}"
}
length = 8
upper = false
special = false
number = false
}
resource "azurerm_storage_account" "batchstorage" {
name = "${lower(random_string.storage.result)}"
resource_group_name = "${var.resource_group_name}"
location = "${var.resource_group_location}"
account_tier = "Standard"
account_replication_type = "LRS"
}
resource "azurerm_storage_container" "boostrapscript" {
name = "scripts"
resource_group_name = "${var.resource_group_name}"
storage_account_name = "${azurerm_storage_account.batchstorage.name}"
container_access_type = "private"
}
resource "azurerm_storage_blob" "initscript" {
name = "init.sh"
resource_group_name = "${var.resource_group_name}"
storage_account_name = "${azurerm_storage_account.batchstorage.name}"
storage_container_name = "${azurerm_storage_container.boostrapscript.name}"
type = "block"
source = "${var.pool_bootstrap_script_path}"
}
data "azurerm_storage_account_sas" "scriptaccess" {
connection_string = "${azurerm_storage_account.batchstorage.primary_connection_string}"
https_only = true
resource_types {
service = false
container = false
object = true
}
services {
blob = true
queue = false
table = false
file = false
}
start = "${timestamp()}"
expiry = "${timeadd(timestamp(), "8776h")}"
permissions {
read = true
write = false
delete = false
list = false
add = false
create = false
update = false
process = false
}
}
output "pool_boostrap_script_url" {
value = "${azurerm_storage_blob.initscript.url}${data.azurerm_storage_account_sas.scriptaccess.sas}"
}
output "id" {
value = "${azurerm_storage_account.batchstorage.id}"
}

14
providers/azurebatch/deployment/storage/variables.tf
View File

@@ -1,14 +0,0 @@
variable "resource_group_name" {
description = "Resource group name"
type = "string"
}
variable "resource_group_location" {
description = "Resource group location"
type = "string"
}
variable "pool_bootstrap_script_path" {
description = "The filepath of the pool boostrapping script"
type = "string"
}

23
providers/azurebatch/deployment/variables.tf
View File

@@ -1,23 +0,0 @@
variable "client_id" {
type = "string"
description = "Client ID"
}
variable "client_secret" {
type = "string"
description = "Client secret."
}
variable "resource_group_name" {
description = "Resource group name"
type = "string"
}
variable "resource_group_location" {
description = "Resource group location"
type = "string"
}
variable "virtualkubelet_docker_image" {
type = "string"
}

14
providers/azurebatch/deployment/vars.example.tfvars
View File

@@ -1,14 +0,0 @@
// Provide the Client ID of a service principal for use by AKS
client_id = "00000000-0000-0000-0000-000000000000"
// Provide the Client Secret of a service principal for use by AKS
client_secret = "00000000-0000-0000-0000-000000000000"
// The resource group you would like to deploy too
resource_group_name = "vkgpu"
// The location of all resources
resource_group_location = "westeurope"
// Virtual Kubelet docker image
virtualkubelet_docker_image = "microsoft/virtual-kubelet"

126
providers/azurebatch/deployment/virtualkubelet/main.tf
View File

@@ -1,126 +0,0 @@
provider "kubernetes" {
host = "${var.cluster_host}"
client_certificate = "${var.cluster_client_certificate}"
client_key = "${var.cluster_client_key}"
cluster_ca_certificate = "${var.cluster_ca}"
}
resource "kubernetes_secret" "vkcredentials" {
metadata {
name = "vkcredentials"
}
data {
cert.pem = "${var.cluster_client_certificate}"
key.pem = "${var.cluster_client_key}"
}
}
resource "kubernetes_deployment" "vkdeployment" {
metadata {
name = "vkdeployment"
}
spec {
selector {
app = "virtualkubelet"
}
template {
metadata {
labels {
app = "virtualkubelet"
}
}
spec {
container {
name = "vk"
image = "${var.virtualkubelet_docker_image}"
args = [
"--provider",
"azurebatch",
"--taint",
"azure.com/batch",
"--namespace",
"default",
]
port {
container_port = 10250
protocol = "TCP"
name = "kubeletport"
}
volume_mount {
name = "azure-credentials"
mount_path = "/etc/aks/azure.json"
}
volume_mount {
name = "credentials"
mount_path = "/etc/virtual-kubelet"
}
env = [
{
name = "AZURE_BATCH_ACCOUNT_LOCATION"
value = "${var.resource_group_location}"
},
{
name = "AZURE_BATCH_ACCOUNT_NAME"
value = "${var.azure_batch_account_name}"
},
{
name = "AZURE_BATCH_POOLID"
value = "${var.azure_batch_pool_id}"
},
{
name = "KUBELET_PORT"
value = "10250"
},
{
name = "AZURE_CREDENTIALS_LOCATION"
value = "/etc/aks/azure.json"
},
{
name = "APISERVER_CERT_LOCATION"
value = "/etc/virtual-kubelet/cert.pem"
},
{
name = "APISERVER_KEY_LOCATION"
value = "/etc/virtual-kubelet/key.pem"
},
{
name = "VKUBELET_POD_IP"
value_from {
field_ref {
field_path = "status.podIP"
}
}
},
]
}
volume {
name = "azure-credentials"
host_path {
path = "/etc/kubernetes/azure.json"
}
}
volume {
name = "credentials"
secret {
secret_name = "vkcredentials"
}
}
}
}
}
}

41
providers/azurebatch/deployment/virtualkubelet/variables.tf
View File

@@ -1,41 +0,0 @@
variable "cluster_client_certificate" {
type = "string"
description = "Cluster client Certificate"
default = "eastus"
}
variable "cluster_client_key" {
type = "string"
description = "Cluster client Certificate Key"
}
variable "cluster_ca" {
type = "string"
description = "Cluster Certificate Authority"
}
variable "cluster_host" {
type = "string"
description = "Cluster Admin API host"
}
variable "virtualkubelet_docker_image" {
type = "string"
description = "The docker image to use for deploying the virtual kubelet"
}
variable "azure_batch_account_name" {
type = "string"
description = "The name of the Azure Batch account to use"
}
variable "azure_batch_pool_id" {
type = "string"
description = "The PoolID to use in Azure batch"
default = "pool1"
}
variable "resource_group_location" {
description = "Resource group location"
type = "string"
}

48
providers/azurebatch/errors.go
View File

@@ -1,48 +0,0 @@
package azurebatch
import (
"net/http"
"github.com/Azure/go-autorest/autorest"
"github.com/Azure/go-autorest/autorest/azure"
"github.com/virtual-kubelet/virtual-kubelet/errdefs"
)
func wrapError(err error) error {
if err == nil {
return nil
}
switch {
case isStatus(err, http.StatusNotFound):
return errdefs.AsNotFound(err)
default:
return err
}
}
type causal interface {
Cause() error
}
func isStatus(err error, status int) bool {
if err == nil {
return false
}
switch e := err.(type) {
case *azure.RequestError:
if e.StatusCode != 0 {
return e.StatusCode == status
}
return isStatus(e.Original, status)
case autorest.DetailedError:
if e.StatusCode != 0 {
return e.StatusCode == status
}
return isStatus(e.Original, status)
case causal:
return isStatus(e.Cause(), status)
}
return false
}

6
providers/azurebatch/testdata/logresponse.json vendored
View File

@@ -1,6 +0,0 @@
{"log":"[Vector addition of 50000 elements]\n","stream":"stdout","time":"2018-05-30T17:02:49.967357287Z"}
{"log":"Copy input data from the host memory to the CUDA device\n","stream":"stdout","time":"2018-05-30T17:02:49.967417086Z"}
{"log":"CUDA kernel launch with 196 blocks of 256 threads\n","stream":"stdout","time":"2018-05-30T17:02:49.967423286Z"}
{"log":"Copy output data from the CUDA device to the host memory\n","stream":"stdout","time":"2018-05-30T17:02:49.967427386Z"}
{"log":"Test PASSED\n","stream":"stdout","time":"2018-05-30T17:02:49.967431286Z"}
{"log":"Done\n","stream":"stdout","time":"2018-05-30T17:02:49.967435286Z"}

48
providers/cri/README.md
View File

@@ -1,48 +0,0 @@
# Virtual Kubelet CRI Provider
This is a Virtual Kubelet Provider implementation that manages real pods and containers in a CRI-based container runtime.
## Purpose
The purpose of the CRI Provider is for testing and prototyping ONLY. It is not to be used for any other purpose!
The whole point of the Virtual Kubelet project is to provide an interface for container runtimes that don't conform to the standard node-based model. The [Kubelet](https://github.com/kubernetes/kubernetes/tree/master/pkg/kubelet) codebase is the comprehensive standard CRI node agent and this Provider is not attempting to recreate that.
This Provider implementation should be seen as a bare-bones minimum implementation for making it easier to test the core of the Virtual Kubelet project against real pods and containers - in other words, more comprehensive than MockProvider.
This Provider implementation is also designed such that it can be used for prototyping new architectural features which can be developed against local Linux infrastructure. If the CRI provider can be shown to work successfully within a Linux guest, there can be a much higher degree of confidence that the abstraction should work for other Providers.
## Dependencies
The simplest way to run the CRI provider is to install [containerd 1.1](https://github.com/containerd/containerd/releases), which already has the CRI plugin installed.
## Configuring
* Copy `/etc/kubernetes/admin.conf` from your master node and place it somewhere local to Virtual Kubelet
* Find a `client.crt` and `client.key` that will allow you to authenticate with the API server and copy them somewhere local
## Running
Start containerd
```cli
sudo nohup containerd > /tmp/containerd.out 2>&1 &
```
Create a script that will set up the environment and run Virtual Kubelet with the correct provider
```
#!/bin/bash
export VKUBELET_POD_IP=<IP of the Linux node>
export APISERVER_CERT_LOCATION="/etc/virtual-kubelet/client.crt"
export APISERVER_KEY_LOCATION="/etc/virtual-kubelet/client.key"
export KUBELET_PORT="10250"
cd bin
./virtual-kubelet --provider cri --kubeconfig admin.conf
```
The Provider assumes that the containerd socket is available at `/run/containerd/containerd.sock` which is the default location. It will write container logs at `/var/log/vk-cri/` and mount volumes at `/run/vk-cri/volumes/`. You need to make sure that you run as a user that has permissions to read and write to these locations.
## Limitations
* The CRI provider does everything that the Provider interface currently allows it to do, principally managing the lifecycle of pods, returning logs and very little else.
* It will create emptyDir, configmap and secret volumes as necessary, but won't update configmaps or secrets if they change as this has yet to be implemented in the base
* It does not support any kind of persistent volumes
* It will try to run kube-proxy when it starts and can successfully do that. However, as we transition VK to a model in which it treats services and routing in the abstract, this capability will be refactored as a means of testing that feature.
* Networking should currently be considered non-functional

181
providers/cri/client.go
View File

@@ -1,181 +0,0 @@
package cri
import (
"fmt"
log "github.com/sirupsen/logrus"
"golang.org/x/net/context"
criapi "k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1alpha2"
)
// Call RunPodSandbox on the CRI client
func runPodSandbox(client criapi.RuntimeServiceClient, config *criapi.PodSandboxConfig) (string, error) {
request := &criapi.RunPodSandboxRequest{Config: config}
log.Debugf("RunPodSandboxRequest: %v", request)
r, err := client.RunPodSandbox(context.Background(), request)
log.Debugf("RunPodSandboxResponse: %v", r)
if err != nil {
return "", err
}
log.Printf("New pod sandbox created: %v", r.PodSandboxId)
return r.PodSandboxId, nil
}
// Call StopPodSandbox on the CRI client
func stopPodSandbox(client criapi.RuntimeServiceClient, id string) error {
if id == "" {
return fmt.Errorf("ID cannot be empty")
}
request := &criapi.StopPodSandboxRequest{PodSandboxId: id}
log.Debugf("StopPodSandboxRequest: %v", request)
r, err := client.StopPodSandbox(context.Background(), request)
log.Debugf("StopPodSandboxResponse: %v", r)
if err != nil {
return err
}
log.Printf("Stopped sandbox %s\n", id)
return nil
}
// Call RemovePodSandbox on the CRI client
func removePodSandbox(client criapi.RuntimeServiceClient, id string) error {
if id == "" {
return fmt.Errorf("ID cannot be empty")
}
request := &criapi.RemovePodSandboxRequest{PodSandboxId: id}
log.Debugf("RemovePodSandboxRequest: %v", request)
r, err := client.RemovePodSandbox(context.Background(), request)
log.Debugf("RemovePodSandboxResponse: %v", r)
if err != nil {
return err
}
log.Printf("Removed sandbox %s\n", id)
return nil
}
// Call ListPodSandbox on the CRI client
func getPodSandboxes(client criapi.RuntimeServiceClient) ([]*criapi.PodSandbox, error) {
filter := &criapi.PodSandboxFilter{}
request := &criapi.ListPodSandboxRequest{
Filter: filter,
}
log.Debugf("ListPodSandboxRequest: %v", request)
r, err := client.ListPodSandbox(context.Background(), request)
log.Debugf("ListPodSandboxResponse: %v", r)
if err != nil {
return nil, err
}
return r.GetItems(), err
}
// Call PodSandboxStatus on the CRI client
func getPodSandboxStatus(client criapi.RuntimeServiceClient, psId string) (*criapi.PodSandboxStatus, error) {
if psId == "" {
return nil, fmt.Errorf("Pod ID cannot be empty in GPSS")
}
request := &criapi.PodSandboxStatusRequest{
PodSandboxId: psId,
Verbose: false,
}
log.Debugf("PodSandboxStatusRequest: %v", request)
r, err := client.PodSandboxStatus(context.Background(), request)
log.Debugf("PodSandboxStatusResponse: %v", r)
if err != nil {
return nil, err
}
return r.Status, nil
}
// Call CreateContainer on the CRI client
func createContainer(client criapi.RuntimeServiceClient, config *criapi.ContainerConfig, podConfig *criapi.PodSandboxConfig, pId string) (string, error) {
request := &criapi.CreateContainerRequest{
PodSandboxId: pId,
Config: config,
SandboxConfig: podConfig,
}
log.Debugf("CreateContainerRequest: %v", request)
r, err := client.CreateContainer(context.Background(), request)
log.Debugf("CreateContainerResponse: %v", r)
if err != nil {
return "", err
}
log.Printf("Container created: %s\n", r.ContainerId)
return r.ContainerId, nil
}
// Call StartContainer on the CRI client
func startContainer(client criapi.RuntimeServiceClient, cId string) error {
if cId == "" {
return fmt.Errorf("ID cannot be empty")
}
request := &criapi.StartContainerRequest{
ContainerId: cId,
}
log.Debugf("StartContainerRequest: %v", request)
r, err := client.StartContainer(context.Background(), request)
log.Debugf("StartContainerResponse: %v", r)
if err != nil {
return err
}
log.Printf("Container started: %s\n", cId)
return nil
}
// Call ContainerStatus on the CRI client
func getContainerCRIStatus(client criapi.RuntimeServiceClient, cId string) (*criapi.ContainerStatus, error) {
if cId == "" {
return nil, fmt.Errorf("Container ID cannot be empty in GCCS")
}
request := &criapi.ContainerStatusRequest{
ContainerId: cId,
Verbose: false,
}
log.Debugf("ContainerStatusRequest: %v", request)
r, err := client.ContainerStatus(context.Background(), request)
log.Debugf("ContainerStatusResponse: %v", r)
if err != nil {
return nil, err
}
return r.Status, nil
}
// Call ListContainers on the CRI client
func getContainersForSandbox(client criapi.RuntimeServiceClient, psId string) ([]*criapi.Container, error) {
filter := &criapi.ContainerFilter{}
filter.PodSandboxId = psId
request := &criapi.ListContainersRequest{
Filter: filter,
}
log.Debugf("ListContainerRequest: %v", request)
r, err := client.ListContainers(context.Background(), request)
log.Debugf("ListContainerResponse: %v", r)
if err != nil {
return nil, err
}
return r.Containers, nil
}
// Pull and image on the CRI client and return the image ref
func pullImage(client criapi.ImageServiceClient, image string) (string, error) {
request := &criapi.PullImageRequest{
Image: &criapi.ImageSpec{
Image: image,
},
}
log.Debugf("PullImageRequest: %v", request)
r, err := client.PullImage(context.Background(), request)
log.Debugf("PullImageResponse: %v", r)
if err != nil {
return "", err
}
return r.ImageRef, nil
}

935
providers/cri/cri.go
View File

@@ -1,935 +0,0 @@
// +build linux
package cri
import (
"bufio"
"context"
"fmt"
"io"
"io/ioutil"
"net"
"os"
"path/filepath"
"runtime"
"strings"
"syscall"
"time"
log "github.com/sirupsen/logrus"
"github.com/virtual-kubelet/virtual-kubelet/errdefs"
"github.com/virtual-kubelet/virtual-kubelet/manager"
"github.com/virtual-kubelet/virtual-kubelet/node/api"
"github.com/virtual-kubelet/virtual-kubelet/providers"
"google.golang.org/grpc"
v1 "k8s.io/api/core/v1"
k8serr "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/api/resource"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
types "k8s.io/apimachinery/pkg/types"
criapi "k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1alpha2"
)
// TODO: Make these configurable
const CriSocketPath = "/run/containerd/containerd.sock"
const PodLogRoot = "/var/log/vk-cri/"
const PodVolRoot = "/run/vk-cri/volumes/"
const PodLogRootPerms = 0755
const PodVolRootPerms = 0755
const PodVolPerms = 0755
const PodSecretVolPerms = 0755
const PodSecretVolDir = "/secrets"
const PodSecretFilePerms = 0644
const PodConfigMapVolPerms = 0755
const PodConfigMapVolDir = "/configmaps"
const PodConfigMapFilePerms = 0644
// CRIProvider implements the virtual-kubelet provider interface and manages pods in a CRI runtime
// NOTE: CRIProvider is not inteded as an alternative to Kubelet, rather it's intended for testing and POC purposes
// As such, it is far from functionally complete and never will be. It provides the minimum function necessary
type CRIProvider struct {
resourceManager *manager.ResourceManager
podLogRoot string
podVolRoot string
nodeName string
operatingSystem string
internalIP string
daemonEndpointPort int32
podStatus map[types.UID]CRIPod // Indexed by Pod Spec UID
runtimeClient criapi.RuntimeServiceClient
imageClient criapi.ImageServiceClient
}
type CRIPod struct {
id string // This is the CRI Pod ID, not the UID from the Pod Spec
containers map[string]*criapi.ContainerStatus // ContainerStatus is a superset of Container, so no need to store both
status *criapi.PodSandboxStatus // PodStatus is a superset of PodSandbox, so no need to store both
}
// Build an internal representation of the state of the pods and containers on the node
// Call this at the start of every function that needs to read any pod or container state
func (p *CRIProvider) refreshNodeState() error {
allPods, err := getPodSandboxes(p.runtimeClient)
if err != nil {
return err
}
newStatus := make(map[types.UID]CRIPod)
for _, pod := range allPods {
psId := pod.Id
pss, err := getPodSandboxStatus(p.runtimeClient, psId)
if err != nil {
return err
}
containers, err := getContainersForSandbox(p.runtimeClient, psId)
if err != nil {
return err
}
var css = make(map[string]*criapi.ContainerStatus)
for _, c := range containers {
cstatus, err := getContainerCRIStatus(p.runtimeClient, c.Id)
if err != nil {
return err
}
css[cstatus.Metadata.Name] = cstatus
}
newStatus[types.UID(pss.Metadata.Uid)] = CRIPod{
id: pod.Id,
status: pss,
containers: css,
}
}
p.podStatus = newStatus
return nil
}
// Initialize the CRI APIs required
func getClientAPIs(criSocketPath string) (criapi.RuntimeServiceClient, criapi.ImageServiceClient, error) {
// Set up a connection to the server.
conn, err := getClientConnection(criSocketPath)
if err != nil {
return nil, nil, fmt.Errorf("failed to connect: %v", err)
}
rc := criapi.NewRuntimeServiceClient(conn)
if rc == nil {
return nil, nil, fmt.Errorf("failed to create runtime service client")
}
ic := criapi.NewImageServiceClient(conn)
if ic == nil {
return nil, nil, fmt.Errorf("failed to create image service client")
}
return rc, ic, err
}
func unixDialer(addr string, timeout time.Duration) (net.Conn, error) {
return net.DialTimeout("unix", addr, timeout)
}
// Initialize CRI client connection
func getClientConnection(criSocketPath string) (*grpc.ClientConn, error) {
conn, err := grpc.Dial(criSocketPath, grpc.WithInsecure(), grpc.WithTimeout(10*time.Second), grpc.WithDialer(unixDialer))
if err != nil {
return nil, fmt.Errorf("failed to connect: %v", err)
}
return conn, nil
}
// Create a new CRIProvider
func NewCRIProvider(nodeName, operatingSystem string, internalIP string, resourceManager *manager.ResourceManager, daemonEndpointPort int32) (*CRIProvider, error) {
runtimeClient, imageClient, err := getClientAPIs(CriSocketPath)
if err != nil {
return nil, err
}
provider := CRIProvider{
resourceManager: resourceManager,
podLogRoot: PodLogRoot,
podVolRoot: PodVolRoot,
nodeName: nodeName,
operatingSystem: operatingSystem,
internalIP: internalIP,
daemonEndpointPort: daemonEndpointPort,
podStatus: make(map[types.UID]CRIPod),
runtimeClient: runtimeClient,
imageClient: imageClient,
}
err = os.MkdirAll(provider.podLogRoot, PodLogRootPerms)
if err != nil {
return nil, err
}
err = os.MkdirAll(provider.podVolRoot, PodVolRootPerms)
if err != nil {
return nil, err
}
return &provider, err
}
// Take the labels from the Pod spec and turn the into a map
// Note: None of the "special" labels appear to have any meaning outside of Kubelet
func createPodLabels(pod *v1.Pod) map[string]string {
labels := make(map[string]string)
for k, v := range pod.Labels {
labels[k] = v
}
return labels
}
// Create a hostname from the Pod spec
func createPodHostname(pod *v1.Pod) string {
specHostname := pod.Spec.Hostname
// specDomain := pod.Spec.Subdomain
if len(specHostname) == 0 {
specHostname = pod.Spec.NodeName // TODO: This is what kube-proxy expects. Double-check
}
// if len(specDomain) == 0 {
return specHostname
// }
// TODO: Cannot apply the domain until we get the cluster domain from the equivalent of kube-config
// If specified, the fully qualified Pod hostname will be "<hostname>.<subdomain>.<pod namespace>.svc.<cluster domain>".
// If not specified, the pod will not have a domainname at all.
// return fmt.Sprintf("%s.%s.%s.svc.%s", specHostname, specDomain, Pod.Spec.Namespace, //cluster domain)
}
// Create DNS config from the Pod spec
func createPodDnsConfig(pod *v1.Pod) *criapi.DNSConfig {
return nil // Use the container engine defaults for now
}
// Convert protocol spec to CRI
func convertCRIProtocol(in v1.Protocol) criapi.Protocol {
switch in {
case v1.ProtocolTCP:
return criapi.Protocol_TCP
case v1.ProtocolUDP:
return criapi.Protocol_UDP
}
return criapi.Protocol(-1)
}
// Create CRI port mappings from the Pod spec
func createPortMappings(pod *v1.Pod) []*criapi.PortMapping {
result := []*criapi.PortMapping{}
for _, c := range pod.Spec.Containers {
for _, p := range c.Ports {
result = append(result, &criapi.PortMapping{
HostPort: p.HostPort,
ContainerPort: p.ContainerPort,
Protocol: convertCRIProtocol(p.Protocol),
HostIp: p.HostIP,
})
}
}
return result
}
// A Pod is privileged if it contains a privileged container. Look for one in the Pod spec
func existsPrivilegedContainerInSpec(pod *v1.Pod) bool {
for _, c := range pod.Spec.Containers {
if c.SecurityContext != nil &&
c.SecurityContext.Privileged != nil &&
*c.SecurityContext.Privileged {
return true
}
}
return false
}
// Create CRI LinuxPodSandboxConfig from the Pod spec
// TODO: This mapping is currently incomplete
func createPodSandboxLinuxConfig(pod *v1.Pod) *criapi.LinuxPodSandboxConfig {
return &criapi.LinuxPodSandboxConfig{
CgroupParent: "",
SecurityContext: &criapi.LinuxSandboxSecurityContext{
NamespaceOptions: nil, // type *NamespaceOption
SelinuxOptions: nil, // type *SELinuxOption
RunAsUser: nil, // type *Int64Value
RunAsGroup: nil, // type *Int64Value
ReadonlyRootfs: false,
SupplementalGroups: []int64{},
Privileged: existsPrivilegedContainerInSpec(pod),
SeccompProfilePath: "",
},
Sysctls: make(map[string]string),
}
}
// Greate CRI PodSandboxConfig from the Pod spec
// TODO: This is probably incomplete
func generatePodSandboxConfig(pod *v1.Pod, logDir string, attempt uint32) (*criapi.PodSandboxConfig, error) {
podUID := string(pod.UID)
config := &criapi.PodSandboxConfig{
Metadata: &criapi.PodSandboxMetadata{
Name: pod.Name,
Namespace: pod.Namespace,
Uid: podUID,
Attempt: attempt,
},
Labels: createPodLabels(pod),
Annotations: pod.Annotations,
LogDirectory: logDir,
DnsConfig: createPodDnsConfig(pod),
Hostname: createPodHostname(pod),
PortMappings: createPortMappings(pod),
Linux: createPodSandboxLinuxConfig(pod),
}
return config, nil
}
// Convert environment variables to CRI format
func createCtrEnvVars(in []v1.EnvVar) []*criapi.KeyValue {
out := make([]*criapi.KeyValue, len(in))
for i := range in {
e := in[i]
out[i] = &criapi.KeyValue{
Key: e.Name,
Value: e.Value,
}
}
return out
}
// Create CRI container labels from Pod and Container spec
func createCtrLabels(container *v1.Container, pod *v1.Pod) map[string]string {
labels := make(map[string]string)
// Note: None of the "special" labels appear to have any meaning outside of Kubelet
return labels
}
// Create CRI container annotations from Pod and Container spec
func createCtrAnnotations(container *v1.Container, pod *v1.Pod) map[string]string {
annotations := make(map[string]string)
// Note: None of the "special" annotations appear to have any meaning outside of Kubelet
return annotations
}
// Search for a particular volume spec by name in the Pod spec
func findPodVolumeSpec(pod *v1.Pod, name string) *v1.VolumeSource {
for _, volume := range pod.Spec.Volumes {
if volume.Name == name {
return &volume.VolumeSource
}
}
return nil
}
// Convert mount propagation type to CRI format
func convertMountPropagationToCRI(input *v1.MountPropagationMode) criapi.MountPropagation {
if input != nil {
switch *input {
case v1.MountPropagationHostToContainer:
return criapi.MountPropagation_PROPAGATION_HOST_TO_CONTAINER
case v1.MountPropagationBidirectional:
return criapi.MountPropagation_PROPAGATION_BIDIRECTIONAL
}
}
return criapi.MountPropagation_PROPAGATION_PRIVATE
}
// Create a CRI specification for the container mounts from the Pod and Container specs
func createCtrMounts(container *v1.Container, pod *v1.Pod, podVolRoot string, rm *manager.ResourceManager) ([]*criapi.Mount, error) {
mounts := []*criapi.Mount{}
for _, mountSpec := range container.VolumeMounts {
podVolSpec := findPodVolumeSpec(pod, mountSpec.Name)
if podVolSpec == nil {
log.Printf("Container volume mount %s not found in Pod spec", mountSpec.Name)
continue
}
// Common fields to all mount types
newMount := criapi.Mount{
ContainerPath: filepath.Join(mountSpec.MountPath, mountSpec.SubPath),
Readonly: mountSpec.ReadOnly,
Propagation: convertMountPropagationToCRI(mountSpec.MountPropagation),
}
// Iterate over the volume types we care about
if podVolSpec.HostPath != nil {
newMount.HostPath = podVolSpec.HostPath.Path
} else if podVolSpec.EmptyDir != nil {
// TODO: Currently ignores the SizeLimit
newMount.HostPath = filepath.Join(podVolRoot, mountSpec.Name)
// TODO: Maybe not the best place to modify the filesystem, but clear enough for now
err := os.MkdirAll(newMount.HostPath, PodVolPerms)
if err != nil {
return nil, fmt.Errorf("Error making emptyDir for path %s: %v", newMount.HostPath, err)
}
} else if podVolSpec.Secret != nil {
spec := podVolSpec.Secret
podSecretDir := filepath.Join(podVolRoot, PodSecretVolDir, mountSpec.Name)
newMount.HostPath = podSecretDir
err := os.MkdirAll(newMount.HostPath, PodSecretVolPerms)
if err != nil {
return nil, fmt.Errorf("Error making secret dir for path %s: %v", newMount.HostPath, err)
}
secret, err := rm.GetSecret(spec.SecretName, pod.Namespace)
if spec.Optional != nil && !*spec.Optional && k8serr.IsNotFound(err) {
return nil, fmt.Errorf("Secret %s is required by Pod %s and does not exist", spec.SecretName, pod.Name)
}
if err != nil {
return nil, fmt.Errorf("Error getting secret %s from API server: %v", spec.SecretName, err)
}
if secret == nil {
continue
}
// TODO: Check podVolSpec.Secret.Items and map to specified paths
// TODO: Check podVolSpec.Secret.StringData
// TODO: What to do with podVolSpec.Secret.SecretType?
for k, v := range secret.Data {
// TODO: Arguably the wrong place to be writing files, but clear enough for now
// TODO: Ensure that these files are deleted in failure cases
fullPath := filepath.Join(podSecretDir, k)
err = ioutil.WriteFile(fullPath, v, PodSecretFilePerms) // Not encoded
if err != nil {
return nil, fmt.Errorf("Could not write secret file %s", fullPath)
}
}
} else if podVolSpec.ConfigMap != nil {
spec := podVolSpec.ConfigMap
podConfigMapDir := filepath.Join(podVolRoot, PodConfigMapVolDir, mountSpec.Name)
newMount.HostPath = podConfigMapDir
err := os.MkdirAll(newMount.HostPath, PodConfigMapVolPerms)
if err != nil {
return nil, fmt.Errorf("Error making configmap dir for path %s: %v", newMount.HostPath, err)
}
configMap, err := rm.GetConfigMap(spec.Name, pod.Namespace)
if spec.Optional != nil && !*spec.Optional && k8serr.IsNotFound(err) {
return nil, fmt.Errorf("Configmap %s is required by Pod %s and does not exist", spec.Name, pod.Name)
}
if err != nil {
return nil, fmt.Errorf("Error getting configmap %s from API server: %v", spec.Name, err)
}
if configMap == nil {
continue
}
// TODO: Check podVolSpec.ConfigMap.Items and map to paths
// TODO: Check podVolSpec.ConfigMap.BinaryData
for k, v := range configMap.Data {
// TODO: Arguably the wrong place to be writing files, but clear enough for now
// TODO: Ensure that these files are deleted in failure cases
fullPath := filepath.Join(podConfigMapDir, k)
err = ioutil.WriteFile(fullPath, []byte(v), PodConfigMapFilePerms)
if err != nil {
return nil, fmt.Errorf("Could not write configmap file %s", fullPath)
}
}
} else {
continue
}
mounts = append(mounts, &newMount)
}
return mounts, nil
}
// Test a bool pointer. If nil, return default value
func valueOrDefaultBool(input *bool, defVal bool) bool {
if input != nil {
return *input
}
return defVal
}
// Create CRI LinuxContainerConfig from Pod and Container spec
// TODO: Currently incomplete
func createCtrLinuxConfig(container *v1.Container, pod *v1.Pod) *criapi.LinuxContainerConfig {
v1sc := container.SecurityContext
var sc *criapi.LinuxContainerSecurityContext
if v1sc != nil {
sc = &criapi.LinuxContainerSecurityContext{
Capabilities: nil, // type: *Capability
Privileged: valueOrDefaultBool(v1sc.Privileged, false), // No default Pod value
NamespaceOptions: nil, // type: *NamespaceOption
SelinuxOptions: nil, // type: *SELinuxOption
RunAsUser: nil, // type: *Int64Value
RunAsGroup: nil, // type: *Int64Value
RunAsUsername: "",
ReadonlyRootfs: false,
SupplementalGroups: []int64{},
ApparmorProfile: "",
SeccompProfilePath: "",
NoNewPrivs: false,
}
}
return &criapi.LinuxContainerConfig{
Resources: nil, // type: *LinuxContainerResources
SecurityContext: sc,
}
}
// Generate the CRI ContainerConfig from the Pod and container specs
// TODO: Probably incomplete
func generateContainerConfig(container *v1.Container, pod *v1.Pod, imageRef, podVolRoot string, rm *manager.ResourceManager, attempt uint32) (*criapi.ContainerConfig, error) {
// TODO: Probably incomplete
config := &criapi.ContainerConfig{
Metadata: &criapi.ContainerMetadata{
Name: container.Name,
Attempt: attempt,
},
Image: &criapi.ImageSpec{Image: imageRef},
Command: container.Command,
Args: container.Args,
WorkingDir: container.WorkingDir,
Envs: createCtrEnvVars(container.Env),
Labels: createCtrLabels(container, pod),
Annotations: createCtrAnnotations(container, pod),
Linux: createCtrLinuxConfig(container, pod),
LogPath: fmt.Sprintf("%s-%d.log", container.Name, attempt),
Stdin: container.Stdin,
StdinOnce: container.StdinOnce,
Tty: container.TTY,
}
mounts, err := createCtrMounts(container, pod, podVolRoot, rm)
if err != nil {
return nil, err
}
config.Mounts = mounts
return config, nil
}
// Provider function to create a Pod
func (p *CRIProvider) CreatePod(ctx context.Context, pod *v1.Pod) error {
log.Printf("receive CreatePod %q", pod.Name)
var attempt uint32 // TODO: Track attempts. Currently always 0
logPath := filepath.Join(p.podLogRoot, string(pod.UID))
volPath := filepath.Join(p.podVolRoot, string(pod.UID))
err := p.refreshNodeState()
if err != nil {
return err
}
pConfig, err := generatePodSandboxConfig(pod, logPath, attempt)
if err != nil {
return err
}
log.Debugf("%v", pConfig)
existing := p.findPodByName(pod.Namespace, pod.Name)
// TODO: Is re-using an existing sandbox with the UID the correct behavior?
// TODO: Should delete the sandbox if container creation fails
var pId string
if existing == nil {
err = os.MkdirAll(logPath, 0755)
if err != nil {
return err
}
err = os.MkdirAll(volPath, 0755)
if err != nil {
return err
}
// TODO: Is there a race here?
pId, err = runPodSandbox(p.runtimeClient, pConfig)
if err != nil {
return err
}
} else {
pId = existing.status.Metadata.Uid
}
for _, c := range pod.Spec.Containers {
log.Printf("Pulling image %s", c.Image)
imageRef, err := pullImage(p.imageClient, c.Image)
if err != nil {
return err
}
log.Printf("Creating container %s", c.Name)
cConfig, err := generateContainerConfig(&c, pod, imageRef, volPath, p.resourceManager, attempt)
log.Debugf("%v", cConfig)
if err != nil {
return err
}
cId, err := createContainer(p.runtimeClient, cConfig, pConfig, pId)
if err != nil {
return err
}
log.Printf("Starting container %s", c.Name)
err = startContainer(p.runtimeClient, cId)
}
return err
}
// Update is currently not required or even called by VK, so not implemented
func (p *CRIProvider) UpdatePod(ctx context.Context, pod *v1.Pod) error {
log.Printf("receive UpdatePod %q", pod.Name)
return nil
}
// Provider function to delete a pod and its containers
func (p *CRIProvider) DeletePod(ctx context.Context, pod *v1.Pod) error {
log.Printf("receive DeletePod %q", pod.Name)
err := p.refreshNodeState()
if err != nil {
return err
}
ps, ok := p.podStatus[pod.UID]
if !ok {
return errdefs.NotFoundf("Pod %s not found", pod.UID)
}
// TODO: Check pod status for running state
err = stopPodSandbox(p.runtimeClient, ps.status.Id)
if err != nil {
// Note the error, but shouldn't prevent us trying to delete
log.Print(err)
}
// Remove any emptyDir volumes
// TODO: Is there other cleanup that needs to happen here?
err = os.RemoveAll(filepath.Join(p.podVolRoot, string(pod.UID)))
if err != nil {
log.Print(err)
}
err = removePodSandbox(p.runtimeClient, ps.status.Id)
return err
}
// Provider function to return a Pod spec - mostly used for its status
func (p *CRIProvider) GetPod(ctx context.Context, namespace, name string) (*v1.Pod, error) {
log.Printf("receive GetPod %q", name)
err := p.refreshNodeState()
if err != nil {
return nil, err
}
pod := p.findPodByName(namespace, name)
if pod == nil {
return nil, errdefs.NotFoundf("Pod %s in namespace %s could not be found on the node", name, namespace)
}
return createPodSpecFromCRI(pod, p.nodeName), nil
}
// Reads a log file into a string
func readLogFile(filename string, opts api.ContainerLogOpts) (io.ReadCloser, error) {
file, err := os.Open(filename)
if err != nil {
return nil, err
}
defer file.Close()
// TODO: This is not an efficient algorithm for tailing very large logs files
scanner := bufio.NewScanner(file)
lines := []string{}
for scanner.Scan() {
lines = append(lines, scanner.Text())
}
if opts.Tail > 0 && opts.Tail < len(lines) {
lines = lines[len(lines)-opts.Tail:]
}
return ioutil.NopCloser(strings.NewReader(strings.Join(lines, ""))), nil
}
// Provider function to read the logs of a container
func (p *CRIProvider) GetContainerLogs(ctx context.Context, namespace, podName, containerName string, opts api.ContainerLogOpts) (io.ReadCloser, error) {
log.Printf("receive GetContainerLogs %q", containerName)
err := p.refreshNodeState()
if err != nil {
return nil, err
}
pod := p.findPodByName(namespace, podName)
if pod == nil {
return nil, errdefs.NotFoundf("Pod %s in namespace %s not found", podName, namespace)
}
container := pod.containers[containerName]
if container == nil {
return nil, errdefs.NotFoundf("Cannot find container %s in pod %s namespace %s", containerName, podName, namespace)
}
return readLogFile(container.LogPath, opts)
}
// Get full pod name as defined in the provider context
// TODO: Implementation
func (p *CRIProvider) GetPodFullName(namespace string, pod string) string {
return ""
}
// RunInContainer executes a command in a container in the pod, copying data
// between in/out/err and the container's stdin/stdout/stderr.
// TODO: Implementation
func (p *CRIProvider) RunInContainer(ctx context.Context, namespace, name, container string, cmd []string, attach api.AttachIO) error {
log.Printf("receive ExecInContainer %q\n", container)
return nil
}
// Find a pod by name and namespace. Pods are indexed by UID
func (p *CRIProvider) findPodByName(namespace, name string) *CRIPod {
var found *CRIPod
for _, pod := range p.podStatus {
if pod.status.Metadata.Name == name && pod.status.Metadata.Namespace == namespace {
found = &pod
break
}
}
return found
}
// Provider function to return the status of a Pod
func (p *CRIProvider) GetPodStatus(ctx context.Context, namespace, name string) (*v1.PodStatus, error) {
log.Printf("receive GetPodStatus %q", name)
err := p.refreshNodeState()
if err != nil {
return nil, err
}
pod := p.findPodByName(namespace, name)
if pod == nil {
return nil, errdefs.NotFoundf("Pod %s in namespace %s could not be found on the node", name, namespace)
}
return createPodStatusFromCRI(pod), nil
}
// Converts CRI container state to ContainerState
func createContainerStateFromCRI(state criapi.ContainerState, status *criapi.ContainerStatus) *v1.ContainerState {
var result *v1.ContainerState
switch state {
case criapi.ContainerState_CONTAINER_UNKNOWN:
fallthrough
case criapi.ContainerState_CONTAINER_CREATED:
result = &v1.ContainerState{
Waiting: &v1.ContainerStateWaiting{
Reason: status.Reason,
Message: status.Message,
},
}
case criapi.ContainerState_CONTAINER_RUNNING:
result = &v1.ContainerState{
Running: &v1.ContainerStateRunning{
StartedAt: metav1.NewTime(time.Unix(0, status.StartedAt)),
},
}
case criapi.ContainerState_CONTAINER_EXITED:
result = &v1.ContainerState{
Terminated: &v1.ContainerStateTerminated{
ExitCode: status.ExitCode,
Reason: status.Reason,
Message: status.Message,
StartedAt: metav1.NewTime(time.Unix(0, status.StartedAt)),
FinishedAt: metav1.NewTime(time.Unix(0, status.FinishedAt)),
},
}
}
return result
}
// Converts CRI container spec to Container spec
func createContainerSpecsFromCRI(containerMap map[string]*criapi.ContainerStatus) ([]v1.Container, []v1.ContainerStatus) {
containers := make([]v1.Container, 0, len(containerMap))
containerStatuses := make([]v1.ContainerStatus, 0, len(containerMap))
for _, c := range containerMap {
// TODO: Fill out more fields
container := v1.Container{
Name: c.Metadata.Name,
Image: c.Image.Image,
//Command: Command is buried in the Info JSON,
}
containers = append(containers, container)
// TODO: Fill out more fields
containerStatus := v1.ContainerStatus{
Name: c.Metadata.Name,
Image: c.Image.Image,
ImageID: c.ImageRef,
ContainerID: c.Id,
Ready: c.State == criapi.ContainerState_CONTAINER_RUNNING,
State: *createContainerStateFromCRI(c.State, c),
// LastTerminationState:
// RestartCount:
}
containerStatuses = append(containerStatuses, containerStatus)
}
return containers, containerStatuses
}
// Converts CRI pod status to a PodStatus
func createPodStatusFromCRI(p *CRIPod) *v1.PodStatus {
_, cStatuses := createContainerSpecsFromCRI(p.containers)
// TODO: How to determine PodSucceeded and PodFailed?
phase := v1.PodPending
if p.status.State == criapi.PodSandboxState_SANDBOX_READY {
phase = v1.PodRunning
}
startTime := metav1.NewTime(time.Unix(0, p.status.CreatedAt))
return &v1.PodStatus{
Phase: phase,
Conditions: []v1.PodCondition{},
Message: "",
Reason: "",
HostIP: "",
PodIP: p.status.Network.Ip,
StartTime: &startTime,
ContainerStatuses: cStatuses,
}
}
// Creates a Pod spec from data obtained through CRI
func createPodSpecFromCRI(p *CRIPod, nodeName string) *v1.Pod {
cSpecs, _ := createContainerSpecsFromCRI(p.containers)
// TODO: Fill out more fields here
podSpec := v1.Pod{
TypeMeta: metav1.TypeMeta{
Kind: "Pod",
APIVersion: "v1",
},
ObjectMeta: metav1.ObjectMeta{
Name: p.status.Metadata.Name,
Namespace: p.status.Metadata.Namespace,
// ClusterName: TODO: What is this??
UID: types.UID(p.status.Metadata.Uid),
CreationTimestamp: metav1.NewTime(time.Unix(0, p.status.CreatedAt)),
},
Spec: v1.PodSpec{
NodeName: nodeName,
Volumes: []v1.Volume{},
Containers: cSpecs,
},
Status: *createPodStatusFromCRI(p),
}
// log.Printf("Created Pod Spec %v", podSpec)
return &podSpec
}
// Provider function to return all known pods
// TODO: Should this be all pods or just running pods?
func (p *CRIProvider) GetPods(ctx context.Context) ([]*v1.Pod, error) {
log.Printf("receive GetPods")
var pods []*v1.Pod
err := p.refreshNodeState()
if err != nil {
return nil, err
}
for _, ps := range p.podStatus {
pods = append(pods, createPodSpecFromCRI(&ps, p.nodeName))
}
return pods, nil
}
// Find the total memory in the guest OS
func getSystemTotalMemory() uint64 {
in := &syscall.Sysinfo_t{}
err := syscall.Sysinfo(in)
if err != nil {
return 0
}
return uint64(in.Totalram) * uint64(in.Unit)
}
// Provider function to return the capacity of the node
func (p *CRIProvider) Capacity(ctx context.Context) v1.ResourceList {
log.Printf("receive Capacity")
err := p.refreshNodeState()
if err != nil {
log.Printf("Error getting pod status: %v", err)
}
var cpuQ resource.Quantity
cpuQ.Set(int64(runtime.NumCPU()))
var memQ resource.Quantity
memQ.Set(int64(getSystemTotalMemory()))
return v1.ResourceList{
"cpu": cpuQ,
"memory": memQ,
"pods": resource.MustParse("1000"),
}
}
// Provider function to return node conditions
// TODO: For now, use the same node conditions as the MockProvider
func (p *CRIProvider) NodeConditions(ctx context.Context) []v1.NodeCondition {
// TODO: Make this configurable
return []v1.NodeCondition{
{
Type: "Ready",
Status: v1.ConditionTrue,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletReady",
Message: "kubelet is ready.",
},
{
Type: "OutOfDisk",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletHasSufficientDisk",
Message: "kubelet has sufficient disk space available",
},
{
Type: "MemoryPressure",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletHasSufficientMemory",
Message: "kubelet has sufficient memory available",
},
{
Type: "DiskPressure",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletHasNoDiskPressure",
Message: "kubelet has no disk pressure",
},
{
Type: "NetworkUnavailable",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "RouteCreated",
Message: "RouteController created a route",
},
}
}
// Provider function to return a list of node addresses
func (p *CRIProvider) NodeAddresses(ctx context.Context) []v1.NodeAddress {
log.Printf("receive NodeAddresses - returning %s", p.internalIP)
return []v1.NodeAddress{
{
Type: "InternalIP",
Address: p.internalIP,
},
}
}
// Provider function to return the daemon endpoint
func (p *CRIProvider) NodeDaemonEndpoints(ctx context.Context) *v1.NodeDaemonEndpoints {
log.Printf("receive NodeDaemonEndpoints - returning %v", p.daemonEndpointPort)
return &v1.NodeDaemonEndpoints{
KubeletEndpoint: v1.DaemonEndpoint{
Port: p.daemonEndpointPort,
},
}
}
// Provider function to return the guest OS
func (p *CRIProvider) OperatingSystem() string {
log.Printf("receive OperatingSystem - returning %s", providers.OperatingSystemLinux)
return providers.OperatingSystemLinux
}

85
providers/huawei/README.md
View File

@@ -1,85 +0,0 @@
# Huawei CCI
Huawei CCI [(Cloud Container Instance)](https://www.huaweicloud.com/product/cci.html) service provides serverless container management,
and does not require users to manage the cluster and the server.
Only through simple configuration, users can enjoy the agility and high performance of the container.
CCI supports stateless workloads (Deployment) and stateful workload (StatefulSet).
On the basis of Kubernetes, we have made a series of important enhancements such as secure container,
elastic load balancing, elastic scalability, Blue Green Deployment and so on.
## Huawei CCI Virtual Kubelet Provider
Huawei CCI virtual kubelet provider configures a CCI project as node in any of your Kubernetes cluster,
such as Huawei CCE [(Cloud Container Engine)](https://www.huaweicloud.com/en-us/product/cce.html).
CCE supports native Kubernetes applications and tools as private cluster, allowing you to easily set up a container runtime environment.
Pod which is scheduled to the virtual kubelet provider will run in the CCI, that will makes good use of the high performance of CCI.
The diagram below illustrates how Huawei CCI virtual kubelet provider works.
![diagram](cci-provider.svg)
**NOTE:** The Huawei CCI virtual-kubelet provider is in the early stages of development,
and don't use it in a production environment.
## Prerequisites
You must install the provider in a Kubernetes cluster and connect to the CCI, and also need create an account for CCI.
Once you've created your account, then need to record the aksk, region for the configuration in next step.
## Configuration
Before run CCI Virtual Kubelet Provider, you must do as the following steps.
1. Create a configuration profile.
You need to provide the fields you specify like in the [example fils](cci.toml).
2. Copy your AKSK and save them in environment variable:
```console
export APP_KEY="<AppKey>"
export APP_SECRET="<AppSecret>"
```
## Connect to CCI from your cluster via Virtual Kubelet
On the Kubernetes work node, starting a virtual-kubelet process as follows.
```console
virtual-kubelet --provider huawei --provider-config cci.toml
```
Then run ``kubectl get nodes`` in your cluster to validate the provider has been running as a node.
```console
kubectl get nodes
NAME STATUS AGE
virtual-kubelet Ready 5m
cce-192.168.0.178 Ready 10d
cce-192.168.0.233 Ready 10d
```
If want to stop the virtual kubelet, just stop the virtual kubelet process.
## Schedule pod to CCI via Virtual Kubelet
```console
apiVersion: v1
kind: Pod
metadata:
name: myapp
labels:
app: myapp
spec:
nodeName: virtual-kubelet
containers:
- name: nginx
image: 1and1internet/ubuntu-16-nginx
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
tolerations:
- key: huawei.com/cci
effect: NoSchedule
```
Replace the nodeName to the virtual-kubelet nodename and save the configuration to a file ``virtual-kubelet-pod.yaml``.
Then run ``kubectl create -f virtual-kubelet-pod.yaml`` to create the pod. Run ``kubectl get pods -owide`` to get pods.
```console
kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
myapp-7c7877989-vbffm 1/1 Running 0 39s 172.17.0.3 virtual-kubelet
```

309
providers/huawei/auth/auth.go
View File

@@ -1,309 +0,0 @@
package auth
// HWS API Gateway Signature
// Analog to AWS Signature Version 4, with some HWS specific parameters
// Please refer to: http://docs.aws.amazon.com/general/latest/gr/signature-version-4.html
import (
"bytes"
"crypto/hmac"
"crypto/sha256"
"fmt"
"io/ioutil"
"net/http"
"sort"
"strings"
"time"
)
// BasicDateFormat and BasicDateFormatShort define aws-date format
const (
BasicDateFormat = "20060102T150405Z"
BasicDateFormatShort = "20060102"
TerminationString = "sdk_request"
Algorithm = "SDK-HMAC-SHA256"
PreSKString = "SDK"
HeaderXDate = "x-sdk-date"
HeaderDate = "date"
HeaderHost = "host"
HeaderAuthorization = "Authorization"
HeaderContentSha256 = "x-sdk-content-sha256"
// todo: use the region and service.
DefaultRegion = "default"
DefaultService = "apigateway"
)
func shouldEscape(c byte) bool {
if 'A' <= c && c <= 'Z' || 'a' <= c && c <= 'z' || '0' <= c && c <= '9' || c == '_' || c == '-' || c == '~' || c == '.' {
return false
}
return true
}
func escape(s string) string {
hexCount := 0
for i := 0; i < len(s); i++ {
c := s[i]
if shouldEscape(c) {
hexCount++
}
}
if hexCount == 0 {
return s
}
t := make([]byte, len(s)+2*hexCount)
j := 0
for i := 0; i < len(s); i++ {
switch c := s[i]; {
case shouldEscape(c):
t[j] = '%'
t[j+1] = "0123456789ABCDEF"[c>>4]
t[j+2] = "0123456789ABCDEF"[c&15]
j += 3
default:
t[j] = s[i]
j++
}
}
return string(t)
}
func hmacsha256(key []byte, data string) ([]byte, error) {
h := hmac.New(sha256.New, []byte(key))
if _, err := h.Write([]byte(data)); err != nil {
return nil, err
}
return h.Sum(nil), nil
}
// Build a CanonicalRequest from a regular request string
//
// See http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
// CanonicalRequest =
// HTTPRequestMethod + '\n' +
// CanonicalURI + '\n' +
// CanonicalQueryString + '\n' +
// CanonicalHeaders + '\n' +
// SignedHeaders + '\n' +
// HexEncode(Hash(RequestPayload))
func CanonicalRequest(r *http.Request) (string, error) {
var hexencode string
var err error
if hex := r.Header.Get(HeaderContentSha256); hex != "" {
hexencode = hex
} else {
data, err := RequestPayload(r)
if err != nil {
return "", err
}
hexencode, err = HexEncodeSHA256Hash(data)
}
return fmt.Sprintf("%s\n%s\n%s\n%s\n%s\n%s", r.Method, CanonicalURI(r), CanonicalQueryString(r), CanonicalHeaders(r), SignedHeaders(r), hexencode), err
}
// CanonicalURI returns request uri
func CanonicalURI(r *http.Request) string {
pattens := strings.Split(r.URL.Path, "/")
var uri []string
for _, v := range pattens {
switch v {
case "":
continue
case ".":
continue
case "..":
if len(uri) > 0 {
uri = uri[:len(uri)-1]
}
default:
uri = append(uri, escape(v))
}
}
urlpath := "/"
if len(uri) > 0 {
urlpath = urlpath + strings.Join(uri, "/") + "/"
}
return urlpath
}
// CanonicalQueryString
func CanonicalQueryString(r *http.Request) string {
var a []string
for key, value := range r.URL.Query() {
k := escape(key)
for _, v := range value {
var kv string
if v == "" {
kv = k
} else {
kv = fmt.Sprintf("%s=%s", k, escape(v))
}
a = append(a, kv)
}
}
sort.Strings(a)
query := strings.Join(a, "&")
r.URL.RawQuery = query
return query
}
// CanonicalHeaders
func CanonicalHeaders(r *http.Request) string {
var a []string
for key, value := range r.Header {
sort.Strings(value)
var q []string
for _, v := range value {
q = append(q, trimString(v))
}
a = append(a, strings.ToLower(key)+":"+strings.Join(q, ","))
}
a = append(a, HeaderHost+":"+r.Host)
sort.Strings(a)
return fmt.Sprintf("%s\n", strings.Join(a, "\n"))
}
// SignedHeaders
func SignedHeaders(r *http.Request) string {
var a []string
for key := range r.Header {
a = append(a, strings.ToLower(key))
}
a = append(a, HeaderHost)
sort.Strings(a)
return fmt.Sprintf("%s", strings.Join(a, ";"))
}
// RequestPayload
func RequestPayload(r *http.Request) ([]byte, error) {
if r.Body == nil {
return []byte(""), nil
}
b, err := ioutil.ReadAll(r.Body)
r.Body = ioutil.NopCloser(bytes.NewBuffer(b))
return b, err
}
// Return the Credential Scope. See http://docs.aws.amazon.com/general/latest/gr/sigv4-create-string-to-sign.html
func CredentialScope(t time.Time, regionName, serviceName string) string {
return fmt.Sprintf("%s/%s/%s/%s", t.UTC().Format(BasicDateFormatShort), regionName, serviceName, TerminationString)
}
// Create a "String to Sign". See http://docs.aws.amazon.com/general/latest/gr/sigv4-create-string-to-sign.html
func StringToSign(canonicalRequest, credentialScope string, t time.Time) string {
hash := sha256.New()
hash.Write([]byte(canonicalRequest))
return fmt.Sprintf("%s\n%s\n%s\n%x",
Algorithm, t.UTC().Format(BasicDateFormat), credentialScope, hash.Sum(nil))
}
// Generate a "signing key" to sign the "String To Sign". See http://docs.aws.amazon.com/general/latest/gr/sigv4-calculate-signature.html
func GenerateSigningKey(secretKey, regionName, serviceName string, t time.Time) ([]byte, error) {
key := []byte(PreSKString + secretKey)
var err error
dateStamp := t.UTC().Format(BasicDateFormatShort)
data := []string{dateStamp, regionName, serviceName, TerminationString}
for _, d := range data {
key, err = hmacsha256(key, d)
if err != nil {
return nil, err
}
}
return key, nil
}
// Create the HWS Signature. See http://docs.aws.amazon.com/general/latest/gr/sigv4-calculate-signature.html
func SignStringToSign(stringToSign string, signingKey []byte) (string, error) {
hm, err := hmacsha256(signingKey, stringToSign)
return fmt.Sprintf("%x", hm), err
}
// HexEncodeSHA256Hash returns hexcode of sha256
func HexEncodeSHA256Hash(body []byte) (string, error) {
hash := sha256.New()
if body == nil {
body = []byte("")
}
_, err := hash.Write(body)
return fmt.Sprintf("%x", hash.Sum(nil)), err
}
// Get the finalized value for the "Authorization" header. The signature parameter is the output from SignStringToSign
func AuthHeaderValue(signature, accessKey, credentialScope, signedHeaders string) string {
return fmt.Sprintf("%s Credential=%s/%s, SignedHeaders=%s, Signature=%s", Algorithm, accessKey, credentialScope, signedHeaders, signature)
}
func trimString(s string) string {
var trimedString []byte
inQuote := false
var lastChar byte
s = strings.TrimSpace(s)
for _, v := range []byte(s) {
if byte(v) == byte('"') {
inQuote = !inQuote
}
if lastChar == byte(' ') && byte(v) == byte(' ') && !inQuote {
continue
}
trimedString = append(trimedString, v)
lastChar = v
}
return string(trimedString)
}
type Signer interface {
Sign(*http.Request) error
}
// Signature HWS meta
type SignerHws struct {
AppKey string
AppSecret string
Region string
Service string
}
// SignRequest set Authorization header
func (s *SignerHws) Sign(r *http.Request) error {
var t time.Time
var err error
var dt string
if dt = r.Header.Get(HeaderXDate); dt != "" {
t, err = time.Parse(BasicDateFormat, dt)
} else if dt = r.Header.Get(HeaderDate); dt != "" {
t, err = time.Parse(time.RFC1123, dt)
}
if err != nil || dt == "" {
r.Header.Del(HeaderDate)
t = time.Now()
r.Header.Set(HeaderXDate, t.UTC().Format(BasicDateFormat))
}
canonicalRequest, err := CanonicalRequest(r)
if err != nil {
return err
}
Region := DefaultRegion
Service := DefaultService
if s.Region != "" {
Region = s.Region
}
if s.Service != "" {
Service = s.Service
}
credentialScope := CredentialScope(t, Region, Service)
stringToSign := StringToSign(canonicalRequest, credentialScope, t)
key, err := GenerateSigningKey(s.AppSecret, Region, Service, t)
if err != nil {
return err
}
signature, err := SignStringToSign(stringToSign, key)
if err != nil {
return err
}
signedHeaders := SignedHeaders(r)
authValue := AuthHeaderValue(signature, s.AppKey, credentialScope, signedHeaders)
r.Header.Set(HeaderAuthorization, authValue)
return nil
}

784
providers/huawei/cci-provider.svg
View File

@@ -1,784 +0,0 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- 由 Microsoft Visio, SVG Export 生成 cci-provider.svg Page-1 -->
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="8.02082in" height="4.62889in"
viewBox="0 0 577.499 333.28" xml:space="preserve" color-interpolation-filters="sRGB" class="st19">
<v:documentProperties v:langID="2052" v:metric="true" v:viewMarkup="false">
<v:userDefs>
<v:ud v:nameU="msvSubprocessMaster" v:prompt="" v:val="VT4(Rectangle)"/>
<v:ud v:nameU="msvNoAutoConnect" v:val="VT0(1):26"/>
</v:userDefs>
</v:documentProperties>
<style type="text/css">
<![CDATA[
.st1 {fill:#ffffff;stroke:#41719c;stroke-width:0.75}
.st2 {visibility:visible}
.st3 {fill:#5b9bd5;fill-opacity:0.22;filter:url(#filter_2);stroke:#5b9bd5;stroke-opacity:0.22}
.st4 {fill:#ffffff;stroke:#5b9bd5;stroke-width:0.25}
.st5 {fill:#e2efd9;stroke:#61973d;stroke-dasharray:5.25,3.75;stroke-width:0.75}
.st6 {fill:none;stroke:none;stroke-width:0.25}
.st7 {fill:#000000;font-family:Calibri;font-size:1.33333em}
.st8 {fill:#000000;font-family:Calibri;font-size:1.00001em}
.st9 {fill:#c5e0b3;stroke:#c7c8c8;stroke-width:0.25}
.st10 {fill:#000000;font-family:Calibri;font-size:1.00001em;font-weight:bold}
.st11 {marker-end:url(#mrkr1-32);stroke:#000000;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
.st12 {fill:#000000;fill-opacity:1;stroke:#000000;stroke-opacity:1;stroke-width:0.28409090909091}
.st13 {fill:#9cc3e5;stroke:#c7c8c8;stroke-width:0.25}
.st14 {fill:#000000;font-family:Calibri;font-size:1.16666em;font-weight:bold}
.st15 {fill:#000000;font-family:Calibri;font-size:1.33333em;font-weight:bold}
.st16 {fill:#ffffff;stroke:#41719c;stroke-dasharray:3.5,2.5;stroke-width:0.5}
.st17 {fill:#e2efd9;stroke:#61973d;stroke-width:0.75}
.st18 {fill:#000000;font-family:Calibri;font-size:1.16666em}
.st19 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
]]>
</style>
<defs id="Markers">
<g id="lend1">
<path d="M 1 -1 L 0 0 L 1 1 " style="stroke-linecap:round;stroke-linejoin:round;fill:none"/>
</g>
<marker id="mrkr1-32" class="st12" v:arrowType="1" v:arrowSize="2" orient="auto" markerUnits="strokeWidth"
overflow="visible">
<use xlink:href="#lend1" transform="scale(-3.52,-3.52) "/>
</marker>
</defs>
<defs id="Filters">
<filter id="filter_2">
<feGaussianBlur stdDeviation="2"/>
</filter>
</defs>
<g v:mID="0" v:index="1" v:groupContext="foregroundPage">
<v:userDefs>
<v:ud v:nameU="msvThemeOrder" v:val="VT0(0):26"/>
</v:userDefs>
<title>页-1</title>
<v:pageProperties v:drawingScale="0.0393701" v:pageScale="0.0393701" v:drawingUnits="24" v:shadowOffsetX="8.50394"
v:shadowOffsetY="-8.50394"/>
<g id="shape17-1" v:mID="17" v:groupContext="shape" transform="translate(18.7496,-157.931)">
<title>圆角的矩形.17</title>
<v:userDefs>
<v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.34251968503937):1"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.34251968503937):1"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.34251968503937):1"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.34251968503937):1"/>
</v:userDefs>
<path d="M24.66 333.28 L221.95 333.28 A24.661 24.661 -180 0 0 246.61 308.62 L246.61 206.51 A24.661 24.661 -180 0 0 221.95
181.85 L24.66 181.85 A24.661 24.661 -180 0 0 -0 206.51 L0 308.62 A24.661 24.661 -180 0 0 24.66 333.28 Z"
class="st1"/>
</g>
<g id="shape85-3" v:mID="85" v:groupContext="shape" transform="translate(33.1466,-173.969)">
<title>圆角的矩形</title>
<v:userDefs>
<v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<g id="shadow85-4" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
transform="matrix(1,0,0,1,0.345598,1.97279)" class="st2">
<path d="M11.34 333.28 L102.05 333.28 A11.3384 11.3384 -180 0 0 113.39 321.94 L113.39 287.93 A11.3384 11.3384 -180
0 0 102.05 276.59 L11.34 276.59 A11.3384 11.3384 -180 0 0 0 287.93 L0 321.94 A11.3384 11.3384 -180 0
0 11.34 333.28 Z" class="st3"/>
</g>
<path d="M11.34 333.28 L102.05 333.28 A11.3384 11.3384 -180 0 0 113.39 321.94 L113.39 287.93 A11.3384 11.3384 -180 0
0 102.05 276.59 L11.34 276.59 A11.3384 11.3384 -180 0 0 0 287.93 L0 321.94 A11.3384 11.3384 -180 0 0 11.34
333.28 Z" class="st4"/>
</g>
<g id="shape5-8" v:mID="5" v:groupContext="shape" transform="translate(89.6157,-242.747)">
<title>圆角的矩形.5</title>
<v:userDefs>
<v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.11811023622047):1"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.11811023622047):1"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.11811023622047):1"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.11811023622047):1"/>
</v:userDefs>
<path d="M8.5 333.28 L76.54 333.28 A8.5038 8.5038 -180 0 0 85.04 324.78 L85.04 295.01 A8.5038 8.5038 -180 0 0 76.54 286.51
L8.5 286.51 A8.5038 8.5038 -180 0 0 -0 295.01 L0 324.78 A8.5038 8.5038 -180 0 0 8.5 333.28 Z" class="st5"/>
</g>
<g id="shape7-10" v:mID="7" v:groupContext="shape" transform="translate(140.639,-281.014)">
<title>工作表.7</title>
<desc>CCE-Cluster1</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="62.3622" cy="316.272" width="124.73" height="34.0157"/>
<rect x="0" y="299.264" width="124.724" height="34.0157" class="st6"/>
<text x="20.54" y="321.07" class="st7" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>CCE-Cluster1</text> </g>
<g id="shape8-13" v:mID="8" v:groupContext="shape" transform="translate(100.954,-255.503)">
<title>工作表.8</title>
<desc>POD</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="28.3465" cy="316.272" width="56.7" height="34.0157"/>
<rect x="0" y="299.264" width="56.6929" height="34.0157" class="st6"/>
<text x="17.58" y="319.87" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>POD</text> </g>
<g id="shape9-16" v:mID="9" v:groupContext="shape" transform="translate(75.4425,-238.495)">
<title>工作表.9</title>
<desc>nodeName: VK</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="55.2756" cy="316.272" width="110.56" height="34.0157"/>
<rect x="0" y="299.264" width="110.551" height="34.0157" class="st6"/>
<text x="18.81" y="319.87" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>nodeName: VK</text> </g>
<g id="shape11-19" v:mID="11" v:groupContext="shape" transform="translate(159.065,-184.636)">
<title>圆角的矩形.11</title>
<v:userDefs>
<v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.13779527559055):1"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.13779527559055):1"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.13779527559055):1"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.13779527559055):1"/>
</v:userDefs>
<g id="shadow11-20" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
transform="matrix(1,0,0,1,0.345598,1.97279)" class="st2">
<path d="M9.92 333.28 L89.29 333.28 A9.9211 9.9211 -180 0 0 99.21 323.36 L99.21 303.52 A9.9211 9.9211 -180 0 0 89.29
293.6 L9.92 293.6 A9.9211 9.9211 -180 0 0 0 303.52 L0 323.36 A9.9211 9.9211 -180 0 0 9.92 333.28 Z"
class="st3"/>
</g>
<path d="M9.92 333.28 L89.29 333.28 A9.9211 9.9211 -180 0 0 99.21 323.36 L99.21 303.52 A9.9211 9.9211 -180 0 0 89.29
293.6 L9.92 293.6 A9.9211 9.9211 -180 0 0 0 303.52 L0 323.36 A9.9211 9.9211 -180 0 0 9.92 333.28 Z"
class="st9"/>
</g>
<g id="shape12-24" v:mID="12" v:groupContext="shape" transform="translate(167.568,-187.471)">
<title>工作表.12</title>
<desc>Virtual-Kubelet</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="42.5197" cy="316.272" width="85.04" height="34.0157"/>
<rect x="0" y="299.264" width="85.0394" height="34.0157" class="st6"/>
<text x="4.48" y="319.87" class="st10" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Virtual-Kubelet</text> </g>
<g id="shape14-27" v:mID="14" v:groupContext="shape" transform="translate(433.184,-143.181) rotate(50.8696)">
<title>工作表.14</title>
<path d="M0 333.28 L53.9 333.28" class="st11"/>
</g>
<g id="shape22-33" v:mID="22" v:groupContext="shape" transform="translate(159.065,-184.636)">
<title>圆角的矩形.22</title>
<v:userDefs>
<v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.13779527559055):1"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.13779527559055):1"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.13779527559055):1"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.13779527559055):1"/>
</v:userDefs>
<g id="shadow22-34" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
transform="matrix(1,0,0,1,0.345598,1.97279)" class="st2">
<path d="M9.92 333.28 L89.29 333.28 A9.9211 9.9211 -180 0 0 99.21 323.36 L99.21 303.52 A9.9211 9.9211 -180 0 0 89.29
293.6 L9.92 293.6 A9.9211 9.9211 -180 0 0 0 303.52 L0 323.36 A9.9211 9.9211 -180 0 0 9.92 333.28 Z"
class="st3"/>
</g>
<path d="M9.92 333.28 L89.29 333.28 A9.9211 9.9211 -180 0 0 99.21 323.36 L99.21 303.52 A9.9211 9.9211 -180 0 0 89.29
293.6 L9.92 293.6 A9.9211 9.9211 -180 0 0 0 303.52 L0 323.36 A9.9211 9.9211 -180 0 0 9.92 333.28 Z"
class="st9"/>
</g>
<g id="shape23-38" v:mID="23" v:groupContext="shape" transform="translate(167.568,-187.471)">
<title>工作表.23</title>
<desc>Virtual-Kubelet</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="42.5197" cy="316.272" width="85.04" height="34.0157"/>
<rect x="0" y="299.264" width="85.0394" height="34.0157" class="st6"/>
<text x="4.48" y="319.87" class="st10" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Virtual-Kubelet</text> </g>
<g id="shape37-41" v:mID="37" v:groupContext="shape" transform="translate(93.9143,-192.618)">
<title>圆角的矩形.37</title>
<v:userDefs>
<v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.061023622047244):1"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.061023622047244):1"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.061023622047244):1"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.061023622047244):1"/>
</v:userDefs>
<g id="shadow37-42" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
transform="matrix(1,0,0,1,0.345598,1.97279)" class="st2">
<path d="M4.39 333.28 L39.54 333.28 A4.39363 4.39363 -180 0 0 43.94 328.89 L43.94 309.33 A4.39363 4.39363 -180 0
0 39.54 304.93 L4.39 304.93 A4.39363 4.39363 -180 0 0 0 309.33 L0 328.89 A4.39363 4.39363 -180 0 0 4.39
333.28 Z" class="st3"/>
</g>
<path d="M4.39 333.28 L39.54 333.28 A4.39363 4.39363 -180 0 0 43.94 328.89 L43.94 309.33 A4.39363 4.39363 -180 0 0 39.54
304.93 L4.39 304.93 A4.39363 4.39363 -180 0 0 0 309.33 L0 328.89 A4.39363 4.39363 -180 0 0 4.39 333.28 Z"
class="st13"/>
</g>
<g id="shape38-46" v:mID="38" v:groupContext="shape" transform="translate(98.1663,-194.602)">
<title>工作表.38</title>
<desc>POD</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="18.4252" cy="320.808" width="36.86" height="24.9449"/>
<rect x="0" y="308.335" width="36.8504" height="24.9449" class="st6"/>
<text x="7.66" y="324.41" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>POD</text> </g>
<g id="shape39-49" v:mID="39" v:groupContext="shape" transform="translate(42.5364,-192.618)">
<title>圆角的矩形.39</title>
<v:userDefs>
<v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.061023622047244):1"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.061023622047244):1"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.061023622047244):1"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.061023622047244):1"/>
</v:userDefs>
<g id="shadow39-50" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
transform="matrix(1,0,0,1,0.345598,1.97279)" class="st2">
<path d="M4.39 333.28 L39.54 333.28 A4.39363 4.39363 -180 0 0 43.94 328.89 L43.94 309.33 A4.39363 4.39363 -180 0
0 39.54 304.93 L4.39 304.93 A4.39363 4.39363 -180 0 0 0 309.33 L0 328.89 A4.39363 4.39363 -180 0 0 4.39
333.28 Z" class="st3"/>
</g>
<path d="M4.39 333.28 L39.54 333.28 A4.39363 4.39363 -180 0 0 43.94 328.89 L43.94 309.33 A4.39363 4.39363 -180 0 0 39.54
304.93 L4.39 304.93 A4.39363 4.39363 -180 0 0 0 309.33 L0 328.89 A4.39363 4.39363 -180 0 0 4.39 333.28 Z"
class="st13"/>
</g>
<g id="shape40-54" v:mID="40" v:groupContext="shape" transform="translate(44.6624,-193.185)">
<title>工作表.40</title>
<desc>POD</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="18.4252" cy="320.808" width="36.86" height="24.9449"/>
<rect x="0" y="308.335" width="36.8504" height="24.9449" class="st6"/>
<text x="7.66" y="324.41" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>POD</text> </g>
<g id="shape42-57" v:mID="42" v:groupContext="shape" transform="translate(363.159,-240.509)">
<title>圆角的矩形.42</title>
<v:userDefs>
<v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.11811023622047):1"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.11811023622047):1"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.11811023622047):1"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.11811023622047):1"/>
</v:userDefs>
<path d="M8.5 333.28 L76.54 333.28 A8.5038 8.5038 -180 0 0 85.04 324.78 L85.04 295.01 A8.5038 8.5038 -180 0 0 76.54 286.51
L8.5 286.51 A8.5038 8.5038 -180 0 0 -0 295.01 L0 324.78 A8.5038 8.5038 -180 0 0 8.5 333.28 Z" class="st5"/>
</g>
<g id="shape43-59" v:mID="43" v:groupContext="shape" transform="translate(312.135,-279.131)">
<title>工作表.43</title>
<desc>CCE-Cluster1</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="62.3622" cy="316.272" width="124.73" height="34.0157"/>
<rect x="0" y="299.264" width="124.724" height="34.0157" class="st6"/>
<text x="20.54" y="321.07" class="st7" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>CCE-Cluster1</text> </g>
<g id="shape44-62" v:mID="44" v:groupContext="shape" transform="translate(374.498,-253.265)">
<title>工作表.44</title>
<desc>POD</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="28.3465" cy="316.272" width="56.7" height="34.0157"/>
<rect x="0" y="299.264" width="56.6929" height="34.0157" class="st6"/>
<text x="17.58" y="319.87" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>POD</text> </g>
<g id="shape45-65" v:mID="45" v:groupContext="shape" transform="translate(348.986,-236.257)">
<title>工作表.45</title>
<desc>nodeName: VK</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="55.2756" cy="316.272" width="110.56" height="34.0157"/>
<rect x="0" y="299.264" width="110.551" height="34.0157" class="st6"/>
<text x="18.81" y="319.87" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>nodeName: VK</text> </g>
<g id="shape46-68" v:mID="46" v:groupContext="shape" transform="translate(432.608,-182.399)">
<title>圆角的矩形.46</title>
<v:userDefs>
<v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.13779527559055):1"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.13779527559055):1"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.13779527559055):1"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.13779527559055):1"/>
</v:userDefs>
<g id="shadow46-69" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
transform="matrix(1,0,0,1,0.345598,1.97279)" class="st2">
<path d="M9.92 333.28 L89.29 333.28 A9.9211 9.9211 -180 0 0 99.21 323.36 L99.21 303.52 A9.9211 9.9211 -180 0 0 89.29
293.6 L9.92 293.6 A9.9211 9.9211 -180 0 0 0 303.52 L0 323.36 A9.9211 9.9211 -180 0 0 9.92 333.28 Z"
class="st3"/>
</g>
<path d="M9.92 333.28 L89.29 333.28 A9.9211 9.9211 -180 0 0 99.21 323.36 L99.21 303.52 A9.9211 9.9211 -180 0 0 89.29
293.6 L9.92 293.6 A9.9211 9.9211 -180 0 0 0 303.52 L0 323.36 A9.9211 9.9211 -180 0 0 9.92 333.28 Z"
class="st9"/>
</g>
<g id="shape48-73" v:mID="48" v:groupContext="shape" transform="translate(706.728,-140.943) rotate(50.8696)">
<title>工作表.48</title>
<path d="M0 333.28 L53.9 333.28" class="st11"/>
</g>
<g id="shape50-78" v:mID="50" v:groupContext="shape" transform="translate(312.135,-155.693)">
<title>圆角的矩形.50</title>
<v:userDefs>
<v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.34251968503937):1"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.34251968503937):1"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.34251968503937):1"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.34251968503937):1"/>
</v:userDefs>
<path d="M24.66 333.28 L221.95 333.28 A24.661 24.661 -180 0 0 246.61 308.62 L246.61 206.51 A24.661 24.661 -180 0 0 221.95
181.85 L24.66 181.85 A24.661 24.661 -180 0 0 -0 206.51 L0 308.62 A24.661 24.661 -180 0 0 24.66 333.28 Z"
class="st1"/>
</g>
<g id="shape51-80" v:mID="51" v:groupContext="shape" transform="translate(400.009,-240.136)">
<title>圆角的矩形.51</title>
<v:userDefs>
<v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.11811023622047):1"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.11811023622047):1"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.11811023622047):1"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.11811023622047):1"/>
</v:userDefs>
<path d="M8.5 333.28 L76.54 333.28 A8.5038 8.5038 -180 0 0 85.04 324.78 L85.04 295.01 A8.5038 8.5038 -180 0 0 76.54 286.51
L8.5 286.51 A8.5038 8.5038 -180 0 0 -0 295.01 L0 324.78 A8.5038 8.5038 -180 0 0 8.5 333.28 Z" class="st5"/>
</g>
<g id="shape52-82" v:mID="52" v:groupContext="shape" transform="translate(314.97,-278.404)">
<title>工作表.52</title>
<desc>CCE-ClusterN</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="62.3622" cy="316.272" width="124.73" height="34.0157"/>
<rect x="0" y="299.264" width="124.724" height="34.0157" class="st6"/>
<text x="19.43" y="321.07" class="st7" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>CCE-ClusterN</text> </g>
<g id="shape53-85" v:mID="53" v:groupContext="shape" transform="translate(411.348,-252.892)">
<title>工作表.53</title>
<desc>POD</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="28.3465" cy="316.272" width="56.7" height="34.0157"/>
<rect x="0" y="299.264" width="56.6929" height="34.0157" class="st6"/>
<text x="17.58" y="319.87" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>POD</text> </g>
<g id="shape54-88" v:mID="54" v:groupContext="shape" transform="translate(385.836,-235.884)">
<title>工作表.54</title>
<desc>nodeName: VK</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="55.2756" cy="316.272" width="110.56" height="34.0157"/>
<rect x="0" y="299.264" width="110.551" height="34.0157" class="st6"/>
<text x="18.81" y="319.87" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>nodeName: VK</text> </g>
<g id="shape55-91" v:mID="55" v:groupContext="shape" transform="translate(318.536,-181.69)">
<title>圆角的矩形.55</title>
<v:userDefs>
<v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.13779527559055):1"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.13779527559055):1"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.13779527559055):1"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.13779527559055):1"/>
</v:userDefs>
<g id="shadow55-92" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
transform="matrix(1,0,0,1,0.345598,1.97279)" class="st2">
<path d="M9.92 333.28 L89.29 333.28 A9.9211 9.9211 -180 0 0 99.21 323.36 L99.21 303.52 A9.9211 9.9211 -180 0 0 89.29
293.6 L9.92 293.6 A9.9211 9.9211 -180 0 0 0 303.52 L0 323.36 A9.9211 9.9211 -180 0 0 9.92 333.28 Z"
class="st3"/>
</g>
<path d="M9.92 333.28 L89.29 333.28 A9.9211 9.9211 -180 0 0 99.21 323.36 L99.21 303.52 A9.9211 9.9211 -180 0 0 89.29
293.6 L9.92 293.6 A9.9211 9.9211 -180 0 0 0 303.52 L0 323.36 A9.9211 9.9211 -180 0 0 9.92 333.28 Z"
class="st9"/>
</g>
<g id="shape56-96" v:mID="56" v:groupContext="shape" transform="translate(329.28,-185.233)">
<title>工作表.56</title>
<desc>Virtual-Kubelet</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="42.5197" cy="316.272" width="85.04" height="34.0157"/>
<rect x="0" y="299.264" width="85.0394" height="34.0157" class="st6"/>
<text x="4.48" y="319.87" class="st10" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Virtual-Kubelet</text> </g>
<g id="shape65-99" v:mID="65" v:groupContext="shape" transform="translate(164.734,-18.75)">
<title>圆角的矩形.65</title>
<v:userDefs>
<v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.34251968503937):1"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.34251968503937):1"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.34251968503937):1"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.34251968503937):1"/>
</v:userDefs>
<path d="M24.66 333.28 L221.95 333.28 A24.661 24.661 -180 0 0 246.61 308.62 L246.61 235.48 A24.661 24.661 -180 0 0 221.95
210.82 L24.66 210.82 A24.661 24.661 -180 0 0 -0 235.48 L0 308.62 A24.661 24.661 -180 0 0 24.66 333.28 Z"
class="st1"/>
</g>
<g id="shape66-101" v:mID="66" v:groupContext="shape" transform="translate(275.285,-220.17)">
<title>工作表.66</title>
<desc>. . .</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="18.4252" cy="320.808" width="36.86" height="24.9449"/>
<rect x="0" y="308.335" width="36.8504" height="24.9449" class="st6"/>
<text x="9.65" y="325.01" class="st14" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>. . .</text> </g>
<g id="shape67-104" v:mID="67" v:groupContext="shape" transform="translate(269.616,-113.386)">
<title>工作表.67</title>
<desc>CCI</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="18.4252" cy="320.808" width="36.86" height="24.9449"/>
<rect x="0" y="308.335" width="36.8504" height="24.9449" class="st6"/>
<text x="7.82" y="325.61" class="st15" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>CCI</text> </g>
<g id="shape68-107" v:mID="68" v:groupContext="shape" transform="translate(665.854,270.762) rotate(127.093)">
<title>工作表.68</title>
<path d="M0 333.28 L52.84 333.28" class="st11"/>
</g>
<g id="shape69-112" v:mID="69" v:groupContext="shape" transform="translate(181.417,-34.9075)">
<title>圆角的矩形.69</title>
<v:userDefs>
<v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.11417322834646):1"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.11417322834646):1"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.11417322834646):1"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.11417322834646):1"/>
</v:userDefs>
<path d="M8.22 333.28 L73.98 333.28 A8.22034 8.22034 -180 0 0 82.2 325.06 L82.2 263.55 A8.22034 8.22034 -180 0 0 73.98
255.33 L8.22 255.33 A8.22034 8.22034 -180 0 0 0 263.55 L0 325.06 A8.22034 8.22034 -180 0 0 8.22 333.28 Z"
class="st16"/>
</g>
<g id="shape70-114" v:mID="70" v:groupContext="shape" transform="translate(185.315,-34.9075)">
<title>工作表.70</title>
<desc>Project-VK1</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="36.4961" cy="319.107" width="73" height="28.3465"/>
<rect x="0" y="304.934" width="72.9921" height="28.3465" class="st6"/>
<text x="7.77" y="322.71" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Project-VK1</text> </g>
<g id="shape71-117" v:mID="71" v:groupContext="shape" transform="translate(197.008,-60.4193)">
<title>圆角的矩形.71</title>
<v:userDefs>
<v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.07007239014478):1"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.07007239014478):1"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.07007239014478):1"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.07007239014478):1"/>
</v:userDefs>
<path d="M5.05 333.28 L45.41 333.28 A5.04513 5.04513 -180 0 0 50.45 328.23 L50.45 291.55 A5.04513 5.04513 -180 0 0 45.41
286.51 L5.05 286.51 A5.04513 5.04513 -180 0 0 0 291.55 L0 328.23 A5.04513 5.04513 -180 0 0 5.05 333.28 Z"
class="st17"/>
</g>
<g id="shape76-119" v:mID="76" v:groupContext="shape" transform="translate(202.814,-71.3327)">
<title>工作表.76</title>
<desc>POD</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="18.4252" cy="320.808" width="36.86" height="24.9449"/>
<rect x="0" y="308.335" width="36.8504" height="24.9449" class="st6"/>
<text x="7.66" y="324.41" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>POD</text> </g>
<g id="shape77-122" v:mID="77" v:groupContext="shape" transform="translate(536.955,91.1513) rotate(80.0665)">
<title>工作表.77</title>
<path d="M0 333.28 L78.62 333.28" class="st11"/>
</g>
<g id="shape78-127" v:mID="78" v:groupContext="shape" transform="translate(311.528,-34.9075)">
<title>圆角的矩形.78</title>
<v:userDefs>
<v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.11417322834646):1"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.11417322834646):1"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.11417322834646):1"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.11417322834646):1"/>
</v:userDefs>
<path d="M8.22 333.28 L73.98 333.28 A8.22034 8.22034 -180 0 0 82.2 325.06 L82.2 263.55 A8.22034 8.22034 -180 0 0 73.98
255.33 L8.22 255.33 A8.22034 8.22034 -180 0 0 0 263.55 L0 325.06 A8.22034 8.22034 -180 0 0 8.22 333.28 Z"
class="st16"/>
</g>
<g id="shape79-129" v:mID="79" v:groupContext="shape" transform="translate(315.779,-34.9075)">
<title>工作表.79</title>
<desc>Project-VKN</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="36.4961" cy="320.808" width="73" height="24.9449"/>
<rect x="0" y="308.335" width="72.9921" height="24.9449" class="st6"/>
<text x="6.94" y="324.41" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Project-VKN</text> </g>
<g id="shape80-132" v:mID="80" v:groupContext="shape" transform="translate(328.535,-60.4193)">
<title>圆角的矩形.80</title>
<v:userDefs>
<v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.07007239014478):1"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.07007239014478):1"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.07007239014478):1"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.07007239014478):1"/>
</v:userDefs>
<path d="M5.05 333.28 L45.41 333.28 A5.04513 5.04513 -180 0 0 50.45 328.23 L50.45 291.55 A5.04513 5.04513 -180 0 0 45.41
286.51 L5.05 286.51 A5.04513 5.04513 -180 0 0 0 291.55 L0 328.23 A5.04513 5.04513 -180 0 0 5.05 333.28 Z"
class="st17"/>
</g>
<g id="shape81-134" v:mID="81" v:groupContext="shape" transform="translate(334.342,-71.3327)">
<title>工作表.81</title>
<desc>POD</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="18.4252" cy="320.808" width="36.86" height="24.9449"/>
<rect x="0" y="308.335" width="36.8504" height="24.9449" class="st6"/>
<text x="7.66" y="324.41" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>POD</text> </g>
<g id="shape82-137" v:mID="82" v:groupContext="shape" transform="translate(695.381,214.76) rotate(100.926)">
<title>工作表.82</title>
<path d="M0 333.28 L75.87 333.28" class="st11"/>
</g>
<g id="shape83-142" v:mID="83" v:groupContext="shape" transform="translate(269.616,-68.5984)">
<title>工作表.83</title>
<desc>. . .</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="18.4252" cy="320.808" width="36.86" height="24.9449"/>
<rect x="0" y="308.335" width="36.8504" height="24.9449" class="st6"/>
<text x="9.65" y="325.01" class="st14" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>. . .</text> </g>
<g id="shape86-145" v:mID="86" v:groupContext="shape" transform="translate(64.6135,-169.493)">
<title>工作表.86</title>
<desc>node</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="25.2261" cy="320.808" width="50.46" height="24.9449"/>
<rect x="0" y="308.335" width="50.4521" height="24.9449" class="st6"/>
<text x="10.7" y="325.01" class="st18" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>node</text> </g>
<g id="shape87-148" v:mID="87" v:groupContext="shape" transform="translate(431.191,-170.09)">
<title>圆角的矩形.87</title>
<v:userDefs>
<v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
</v:userDefs>
<g id="shadow87-149" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
transform="matrix(1,0,0,1,0.345598,1.97279)" class="st2">
<path d="M11.34 333.28 L102.05 333.28 A11.3384 11.3384 -180 0 0 113.39 321.94 L113.39 287.93 A11.3384 11.3384 -180
0 0 102.05 276.59 L11.34 276.59 A11.3384 11.3384 -180 0 0 0 287.93 L0 321.94 A11.3384 11.3384 -180 0
0 11.34 333.28 Z" class="st3"/>
</g>
<path d="M11.34 333.28 L102.05 333.28 A11.3384 11.3384 -180 0 0 113.39 321.94 L113.39 287.93 A11.3384 11.3384 -180 0
0 102.05 276.59 L11.34 276.59 A11.3384 11.3384 -180 0 0 0 287.93 L0 321.94 A11.3384 11.3384 -180 0 0 11.34
333.28 Z" class="st4"/>
</g>
<g id="shape88-153" v:mID="88" v:groupContext="shape" transform="translate(491.958,-188.739)">
<title>圆角的矩形.88</title>
<v:userDefs>
<v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.061023622047244):1"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.061023622047244):1"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.061023622047244):1"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.061023622047244):1"/>
</v:userDefs>
<g id="shadow88-154" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
transform="matrix(1,0,0,1,0.345598,1.97279)" class="st2">
<path d="M4.39 333.28 L39.54 333.28 A4.39363 4.39363 -180 0 0 43.94 328.89 L43.94 309.33 A4.39363 4.39363 -180 0
0 39.54 304.93 L4.39 304.93 A4.39363 4.39363 -180 0 0 0 309.33 L0 328.89 A4.39363 4.39363 -180 0 0 4.39
333.28 Z" class="st3"/>
</g>
<path d="M4.39 333.28 L39.54 333.28 A4.39363 4.39363 -180 0 0 43.94 328.89 L43.94 309.33 A4.39363 4.39363 -180 0 0 39.54
304.93 L4.39 304.93 A4.39363 4.39363 -180 0 0 0 309.33 L0 328.89 A4.39363 4.39363 -180 0 0 4.39 333.28 Z"
class="st13"/>
</g>
<g id="shape89-158" v:mID="89" v:groupContext="shape" transform="translate(496.21,-190.723)">
<title>工作表.89</title>
<desc>POD</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="18.4252" cy="320.808" width="36.86" height="24.9449"/>
<rect x="0" y="308.335" width="36.8504" height="24.9449" class="st6"/>
<text x="7.66" y="324.41" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>POD</text> </g>
<g id="shape90-161" v:mID="90" v:groupContext="shape" transform="translate(440.58,-188.739)">
<title>圆角的矩形.90</title>
<v:userDefs>
<v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15748031496063):24"/>
<v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
<v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.061023622047244):1"/>
<v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.061023622047244):1"/>
<v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.061023622047244):1"/>
<v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.061023622047244):1"/>
</v:userDefs>
<g id="shadow90-162" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1"
transform="matrix(1,0,0,1,0.345598,1.97279)" class="st2">
<path d="M4.39 333.28 L39.54 333.28 A4.39363 4.39363 -180 0 0 43.94 328.89 L43.94 309.33 A4.39363 4.39363 -180 0
0 39.54 304.93 L4.39 304.93 A4.39363 4.39363 -180 0 0 0 309.33 L0 328.89 A4.39363 4.39363 -180 0 0 4.39
333.28 Z" class="st3"/>
</g>
<path d="M4.39 333.28 L39.54 333.28 A4.39363 4.39363 -180 0 0 43.94 328.89 L43.94 309.33 A4.39363 4.39363 -180 0 0 39.54
304.93 L4.39 304.93 A4.39363 4.39363 -180 0 0 0 309.33 L0 328.89 A4.39363 4.39363 -180 0 0 4.39 333.28 Z"
class="st13"/>
</g>
<g id="shape91-166" v:mID="91" v:groupContext="shape" transform="translate(442.706,-189.306)">
<title>工作表.91</title>
<desc>POD</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="18.4252" cy="320.808" width="36.86" height="24.9449"/>
<rect x="0" y="308.335" width="36.8504" height="24.9449" class="st6"/>
<text x="7.66" y="324.41" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>POD</text> </g>
<g id="shape92-169" v:mID="92" v:groupContext="shape" transform="translate(462.657,-165.614)">
<title>工作表.92</title>
<desc>node</desc>
<v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/>
<v:textRect cx="25.2261" cy="320.808" width="50.46" height="24.9449"/>
<rect x="0" y="308.335" width="50.4521" height="24.9449" class="st6"/>
<text x="10.7" y="325.01" class="st18" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>node</text> </g>
</g>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 51 KiB

451
providers/huawei/cci.go
View File

@@ -1,451 +0,0 @@
package huawei
import (
"bytes"
"context"
"crypto/tls"
"encoding/json"
"errors"
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
"os"
"strings"
"time"
"github.com/virtual-kubelet/virtual-kubelet/errdefs"
"github.com/virtual-kubelet/virtual-kubelet/manager"
"github.com/virtual-kubelet/virtual-kubelet/node/api"
"github.com/virtual-kubelet/virtual-kubelet/providers/huawei/auth"
v1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/resource"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/types"
)
const (
podAnnotationNamespaceKey = "virtual-kubelet-namespace"
podAnnotationPodNameKey = "virtual-kubelet-podname"
podAnnotationClusterNameKey = "virtual-kubelet-clustername"
podAnnotationUIDkey = "virtual-kubelet-uid"
podAnnotationNodeName = "virtual-kubelet-nodename"
podAnnotationCreationTimestamp = "virtual-kubelet-creationtimestamp"
)
var defaultApiEndpoint string = "https://cciback.cn-north-1.huaweicloud.com"
// CCIProvider implements the virtual-kubelet provider interface and communicates with Huawei's CCI APIs.
type CCIProvider struct {
appKey string
appSecret string
apiEndpoint string
region string
service string
project string
internalIP string
daemonEndpointPort int32
nodeName string
operatingSystem string
client *Client
resourceManager *manager.ResourceManager
cpu string
memory string
pods string
}
// Client represents the client config for Huawei.
type Client struct {
Signer auth.Signer
HTTPClient http.Client
}
// NewCCIProvider creates a new CCI provider.
func NewCCIProvider(config string, rm *manager.ResourceManager, nodeName, operatingSystem string, internalIP string, daemonEndpointPort int32) (*CCIProvider, error) {
p := CCIProvider{}
if config != "" {
f, err := os.Open(config)
if err != nil {
return nil, err
}
defer f.Close()
if err := p.loadConfig(f); err != nil {
return nil, err
}
}
if appKey := os.Getenv("CCI_APP_KEP"); appKey != "" {
p.appKey = appKey
}
if p.appKey == "" {
return nil, errors.New("AppKey can not be empty please set CCI_APP_KEP")
}
if appSecret := os.Getenv("CCI_APP_SECRET"); appSecret != "" {
p.appSecret = appSecret
}
if p.appSecret == "" {
return nil, errors.New("AppSecret can not be empty please set CCI_APP_SECRET")
}
p.client = new(Client)
p.client.Signer = &auth.SignerHws{
AppKey: p.appKey,
AppSecret: p.appSecret,
Region: p.region,
Service: p.service,
}
tr := &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
p.client.HTTPClient = http.Client{
Transport: tr,
}
p.resourceManager = rm
p.apiEndpoint = defaultApiEndpoint
p.nodeName = nodeName
p.operatingSystem = operatingSystem
p.internalIP = internalIP
p.daemonEndpointPort = daemonEndpointPort
if err := p.createProject(); err != nil {
return nil, err
}
return &p, nil
}
func (p *CCIProvider) createProject() error {
// Create the createProject request url
uri := p.apiEndpoint + "/api/v1/namespaces"
// build the request
project := &v1.Namespace{
TypeMeta: metav1.TypeMeta{
Kind: "Namespace",
APIVersion: "v1",
},
ObjectMeta: metav1.ObjectMeta{
Name: p.project,
},
}
var bodyReader io.Reader
body, err := json.Marshal(project)
if err != nil {
return err
}
if body != nil {
bodyReader = bytes.NewReader(body)
}
r, err := http.NewRequest("POST", uri, bodyReader)
if err != nil {
return err
}
if err = p.signRequest(r); err != nil {
return fmt.Errorf("Sign the request failed: %v", err)
}
_, err = p.client.HTTPClient.Do(r)
return err
}
func (p *CCIProvider) signRequest(r *http.Request) error {
r.Header.Add("content-type", "application/json; charset=utf-8")
if err := p.client.Signer.Sign(r); err != nil {
return fmt.Errorf("Sign the request failed: %v", err)
}
return nil
}
func (p *CCIProvider) setPodAnnotations(pod *v1.Pod) {
metav1.SetMetaDataAnnotation(&pod.ObjectMeta, podAnnotationNamespaceKey, pod.Namespace)
metav1.SetMetaDataAnnotation(&pod.ObjectMeta, podAnnotationClusterNameKey, pod.ClusterName)
metav1.SetMetaDataAnnotation(&pod.ObjectMeta, podAnnotationPodNameKey, pod.Name)
metav1.SetMetaDataAnnotation(&pod.ObjectMeta, podAnnotationUIDkey, string(pod.UID))
metav1.SetMetaDataAnnotation(&pod.ObjectMeta, podAnnotationNodeName, pod.Spec.NodeName)
metav1.SetMetaDataAnnotation(&pod.ObjectMeta, podAnnotationCreationTimestamp, pod.CreationTimestamp.String())
pod.Namespace = p.project
pod.Name = pod.Namespace + "-" + pod.Name
pod.UID = ""
pod.Spec.NodeName = ""
pod.CreationTimestamp = metav1.Time{}
}
func (p *CCIProvider) deletePodAnnotations(pod *v1.Pod) error {
pod.Name = pod.Annotations[podAnnotationPodNameKey]
pod.Namespace = pod.Annotations[podAnnotationNamespaceKey]
pod.UID = types.UID(pod.Annotations[podAnnotationUIDkey])
pod.ClusterName = pod.Annotations[podAnnotationClusterNameKey]
pod.Spec.NodeName = pod.Annotations[podAnnotationNodeName]
if pod.Annotations[podAnnotationCreationTimestamp] != "" {
t, err := time.Parse("2006-01-02 15:04:05.999999999 -0700 MST", pod.Annotations[podAnnotationCreationTimestamp])
if err != nil {
return err
}
podCreationTimestamp := metav1.NewTime(t)
pod.CreationTimestamp = podCreationTimestamp
}
delete(pod.Annotations, podAnnotationPodNameKey)
delete(pod.Annotations, podAnnotationNamespaceKey)
delete(pod.Annotations, podAnnotationUIDkey)
delete(pod.Annotations, podAnnotationClusterNameKey)
delete(pod.Annotations, podAnnotationNodeName)
delete(pod.Annotations, podAnnotationCreationTimestamp)
pod.Annotations = nil
return nil
}
// CreatePod takes a Kubernetes Pod and deploys it within the huawei CCI provider.
func (p *CCIProvider) CreatePod(ctx context.Context, pod *v1.Pod) error {
// Create the createPod request url
p.setPodAnnotations(pod)
uri := p.apiEndpoint + "/api/v1/namespaces/" + p.project + "/pods"
// build the request
var bodyReader io.Reader
body, err := json.Marshal(pod)
if err != nil {
return err
}
if body != nil {
bodyReader = bytes.NewReader(body)
}
r, err := http.NewRequest("POST", uri, bodyReader)
if err != nil {
return err
}
if err = p.signRequest(r); err != nil {
return fmt.Errorf("Sign the request failed: %v", err)
}
_, err = p.client.HTTPClient.Do(r)
return err
}
// UpdatePod takes a Kubernetes Pod and updates it within the huawei CCI provider.
func (p *CCIProvider) UpdatePod(ctx context.Context, pod *v1.Pod) error {
return nil
}
// DeletePod takes a Kubernetes Pod and deletes it from the huawei CCI provider.
func (p *CCIProvider) DeletePod(ctx context.Context, pod *v1.Pod) error {
// Create the deletePod request url
podName := pod.Namespace + "-" + pod.Name
uri := p.apiEndpoint + "/api/v1/namespaces/" + p.project + "/pods/" + podName
// build the request
r, err := http.NewRequest("DELETE", uri, nil)
if err != nil {
return err
}
if err = p.signRequest(r); err != nil {
return fmt.Errorf("Sign the request failed: %v", err)
}
resp, err := p.client.HTTPClient.Do(r)
if err != nil {
return err
}
return errorFromResponse(resp)
}
func errorFromResponse(resp *http.Response) error {
if resp.StatusCode < 400 {
return nil
}
body, _ := ioutil.ReadAll(io.LimitReader(resp.Body, 16*1024))
err := fmt.Errorf("error during http request, status=%d: %q", resp.StatusCode, string(body))
switch resp.StatusCode {
case http.StatusNotFound:
return errdefs.AsNotFound(err)
default:
return err
}
}
// GetPod retrieves a pod by name from the huawei CCI provider.
func (p *CCIProvider) GetPod(ctx context.Context, namespace, name string) (*v1.Pod, error) {
// Create the getPod request url
podName := namespace + "-" + name
uri := p.apiEndpoint + "/api/v1/namespaces/" + p.project + "/pods/" + podName
r, err := http.NewRequest("GET", uri, nil)
if err != nil {
return nil, fmt.Errorf("Create get POD request failed: %v", err)
}
if err = p.signRequest(r); err != nil {
return nil, fmt.Errorf("Sign the request failed: %v", err)
}
resp, err := p.client.HTTPClient.Do(r)
if err != nil {
return nil, err
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, err
}
var pod v1.Pod
if err = json.Unmarshal(body, &pod); err != nil {
return nil, err
}
if err := p.deletePodAnnotations(&pod); err != nil {
return nil, err
}
return &pod, nil
}
// GetContainerLogs retrieves the logs of a container by name from the huawei CCI provider.
func (p *CCIProvider) GetContainerLogs(ctx context.Context, namespace, podName, containerName string, opts api.ContainerLogOpts) (io.ReadCloser, error) {
return ioutil.NopCloser(strings.NewReader("")), nil
}
// Get full pod name as defined in the provider context
// TODO: Implementation
func (p *CCIProvider) GetPodFullName(namespace string, pod string) string {
return ""
}
// RunInContainer executes a command in a container in the pod, copying data
// between in/out/err and the container's stdin/stdout/stderr.
// TODO: Implementation
func (p *CCIProvider) RunInContainer(ctx context.Context, namespace, name, container string, cmd []string, attach api.AttachIO) error {
log.Printf("receive ExecInContainer %q\n", container)
return nil
}
// GetPodStatus retrieves the status of a pod by name from the huawei CCI provider.
func (p *CCIProvider) GetPodStatus(ctx context.Context, namespace, name string) (*v1.PodStatus, error) {
pod, err := p.GetPod(ctx, namespace, name)
if err != nil {
return nil, err
}
if pod == nil {
return nil, nil
}
return &pod.Status, nil
}
// GetPods retrieves a list of all pods running on the huawei CCI provider.
func (p *CCIProvider) GetPods(ctx context.Context) ([]*v1.Pod, error) {
// Create the getPod request url
uri := p.apiEndpoint + "/api/v1/namespaces/" + p.project + "/pods"
r, err := http.NewRequest("GET", uri, nil)
if err != nil {
return nil, fmt.Errorf("Create get POD request failed: %v", err)
}
if err = p.signRequest(r); err != nil {
return nil, fmt.Errorf("Sign the request failed: %v", err)
}
resp, err := p.client.HTTPClient.Do(r)
if err != nil {
return nil, err
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, err
}
var pods []*v1.Pod
if err = json.Unmarshal(body, &pods); err != nil {
return nil, err
}
for _, pod := range pods {
if err := p.deletePodAnnotations(pod); err != nil {
return nil, err
}
}
return pods, nil
}
// Capacity returns a resource list with the capacity constraints of the huawei CCI provider.
func (p *CCIProvider) Capacity(ctx context.Context) v1.ResourceList {
return v1.ResourceList{
"cpu": resource.MustParse(p.cpu),
"memory": resource.MustParse(p.memory),
"pods": resource.MustParse(p.pods),
}
}
// NodeConditions returns a list of conditions (Ready, OutOfDisk, etc), which is
// polled periodically to update the node status within Kubernetes.
func (p *CCIProvider) NodeConditions(ctx context.Context) []v1.NodeCondition {
// TODO: Make these dynamic and augment with custom CCI specific conditions of interest
return []v1.NodeCondition{
{
Type: "Ready",
Status: v1.ConditionTrue,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletReady",
Message: "kubelet is ready.",
},
{
Type: "OutOfDisk",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletHasSufficientDisk",
Message: "kubelet has sufficient disk space available",
},
{
Type: "MemoryPressure",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletHasSufficientMemory",
Message: "kubelet has sufficient memory available",
},
{
Type: "DiskPressure",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletHasNoDiskPressure",
Message: "kubelet has no disk pressure",
},
{
Type: "NetworkUnavailable",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "RouteCreated",
Message: "RouteController created a route",
},
}
}
// NodeAddresses returns a list of addresses for the node status
// within Kubernetes.
func (p *CCIProvider) NodeAddresses(ctx context.Context) []v1.NodeAddress {
// TODO: Make these dynamic and augment with custom CCI specific conditions of interest
return []v1.NodeAddress{
{
Type: "InternalIP",
Address: p.internalIP,
},
}
}
// NodeDaemonEndpoints returns NodeDaemonEndpoints for the node status
// within Kubernetes.
func (p *CCIProvider) NodeDaemonEndpoints(ctx context.Context) *v1.NodeDaemonEndpoints {
return &v1.NodeDaemonEndpoints{
KubeletEndpoint: v1.DaemonEndpoint{
Port: p.daemonEndpointPort,
},
}
}
// OperatingSystem returns the operating system the huawei CCI provider is for.
func (p *CCIProvider) OperatingSystem() string {
return p.operatingSystem
}

9
providers/huawei/cci.toml
View File

@@ -1,9 +0,0 @@
# example configuration file for Huawei CCI virtual-kubelet provider.
Project = "default"
Region = "southchina"
Service = "CCI"
OperatingSystem = "Linux"
CPU = "20"
Memory = "100Gi"
Pods = "20"

151
providers/huawei/cciMock.go
View File

@@ -1,151 +0,0 @@
package huawei
import (
"bytes"
"encoding/json"
"net/http"
"net/http/httptest"
"github.com/gorilla/mux"
v1 "k8s.io/api/core/v1"
)
// CCIMock implements a CCI service mock server.
type CCIMock struct {
server *httptest.Server
OnCreateProject func(*v1.Namespace) (int, interface{})
OnCreatePod func(*v1.Pod) (int, interface{})
OnGetPods func() (int, interface{})
OnGetPod func(string, string) (int, interface{})
}
// fakeSigner signature HWS meta
type fakeSigner struct {
AppKey string
AppSecret string
Region string
Service string
}
// Sign set Authorization header
func (s *fakeSigner) Sign(r *http.Request) error {
return nil
}
const (
cciProjectRoute = "/api/v1/namespaces"
cciPodsRoute = cciProjectRoute + "/{namespaceID}/pods"
cciPodRoute = cciPodsRoute + "/{podID}"
)
// NewCCIMock creates a CCI service mock server.
func NewCCIMock() *CCIMock {
mock := new(CCIMock)
mock.start()
return mock
}
// Start the CCI service mock service.
func (mock *CCIMock) start() {
if mock.server != nil {
return
}
router := mux.NewRouter()
router.HandleFunc(
cciProjectRoute,
func(w http.ResponseWriter, r *http.Request) {
var ns v1.Namespace
if err := json.NewDecoder(r.Body).Decode(&ns); err != nil {
panic(err)
}
if mock.OnCreatePod != nil {
statusCode, response := mock.OnCreateProject(&ns)
w.WriteHeader(statusCode)
b := new(bytes.Buffer)
json.NewEncoder(b).Encode(response)
w.Write(b.Bytes())
return
}
w.WriteHeader(http.StatusNotImplemented)
}).Methods("PUT")
router.HandleFunc(
cciPodsRoute,
func(w http.ResponseWriter, r *http.Request) {
var pod v1.Pod
if err := json.NewDecoder(r.Body).Decode(&pod); err != nil {
panic(err)
}
if mock.OnCreatePod != nil {
statusCode, response := mock.OnCreatePod(&pod)
w.WriteHeader(statusCode)
b := new(bytes.Buffer)
json.NewEncoder(b).Encode(response)
w.Write(b.Bytes())
return
}
w.WriteHeader(http.StatusNotImplemented)
}).Methods("PUT")
router.HandleFunc(
cciPodRoute,
func(w http.ResponseWriter, r *http.Request) {
namespace, _ := mux.Vars(r)["namespaceID"]
podname, _ := mux.Vars(r)["podID"]
if mock.OnGetPod != nil {
statusCode, response := mock.OnGetPod(namespace, podname)
w.WriteHeader(statusCode)
b := new(bytes.Buffer)
json.NewEncoder(b).Encode(response)
w.Write(b.Bytes())
return
}
w.WriteHeader(http.StatusNotImplemented)
}).Methods("GET")
router.HandleFunc(
cciPodsRoute,
func(w http.ResponseWriter, r *http.Request) {
if mock.OnGetPods != nil {
statusCode, response := mock.OnGetPods()
w.WriteHeader(statusCode)
b := new(bytes.Buffer)
json.NewEncoder(b).Encode(response)
w.Write(b.Bytes())
return
}
w.WriteHeader(http.StatusNotImplemented)
}).Methods("GET")
mock.server = httptest.NewServer(router)
}
// GetServerURL returns the mock server URL.
func (mock *CCIMock) GetServerURL() string {
if mock.server != nil {
return mock.server.URL
}
panic("Mock server is not initialized.")
}
// Close terminates the CCI mock server.
func (mock *CCIMock) Close() {
if mock.server != nil {
mock.server.Close()
mock.server = nil
}
}

214
providers/huawei/cci_test.go
View File

@@ -1,214 +0,0 @@
package huawei
import (
"context"
"net/http"
"os"
"testing"
"gotest.tools/assert"
is "gotest.tools/assert/cmp"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/util/uuid"
)
const (
fakeAppKey = "Whj8f5RAHsvQahveqCdo"
fakeAppSecret = "ymW5JgrdwrIvRS76YxyIqHNXe9s5ocIhaWWvPUhx"
fakeRegion = "southchina"
fakeService = "default"
fakeProject = "vk-project"
fakeNodeName = "vk"
)
// TestCreateProject test create project.
func TestCreateProject(t *testing.T) {
cciServerMocker, provider, err := prepareMocks()
if err != nil {
t.Fatal("Unable to prepare the mocks", err)
}
cciServerMocker.OnCreateProject = func(ns *v1.Namespace) (int, interface{}) {
assert.Check(t, ns != nil, "Project is nil")
assert.Check(t, is.Equal(fakeProject, ns.Name), "pod.Annotations[\"virtual-kubelet-podname\"] is not expected")
return http.StatusOK, &v1.Namespace{
TypeMeta: metav1.TypeMeta{
Kind: "Namespace",
APIVersion: "v1",
},
ObjectMeta: metav1.ObjectMeta{
Name: fakeProject,
},
}
}
if err := provider.createProject(); err != nil {
t.Fatal("Failed to create project", err)
}
}
// TestCreatePod test create pod.
func TestCreatePod(t *testing.T) {
cciServerMocker, provider, err := prepareMocks()
if err != nil {
t.Fatal("Unable to prepare the mocks", err)
}
podName := "pod-" + string(uuid.NewUUID())
podNamespace := "ns-" + string(uuid.NewUUID())
cciServerMocker.OnCreatePod = func(pod *v1.Pod) (int, interface{}) {
assert.Check(t, pod != nil, "Pod is nil")
assert.Check(t, pod.Annotations != nil, "pod.Annotations is expected")
assert.Check(t, is.Equal(podName, pod.Annotations[podAnnotationPodNameKey]), "pod.Annotations[\"virtual-kubelet-podname\"] is not expected")
assert.Check(t, is.Equal(podNamespace, pod.Annotations[podAnnotationNamespaceKey]), "pod.Annotations[\"virtual-kubelet-namespace\"] is not expected")
assert.Check(t, is.Equal(1, len(pod.Spec.Containers)), "1 Container is expected")
assert.Check(t, is.Equal("nginx", pod.Spec.Containers[0].Name), "Container nginx is expected")
return http.StatusOK, pod
}
pod := &v1.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: podName,
Namespace: podNamespace,
},
Spec: v1.PodSpec{
Containers: []v1.Container{
v1.Container{
Name: "nginx",
},
},
},
}
if err := provider.CreatePod(context.Background(), pod); err != nil {
t.Fatal("Failed to create pod", err)
}
}
// Tests get pod.
func TestGetPod(t *testing.T) {
cciServerMocker, provider, err := prepareMocks()
if err != nil {
t.Fatal("Unable to prepare the mocks", err)
}
podName := "pod-" + string(uuid.NewUUID())
podNamespace := "ns-" + string(uuid.NewUUID())
cciServerMocker.OnGetPod = func(namespace, name string) (int, interface{}) {
annotations := map[string]string{
podAnnotationPodNameKey: "podname",
podAnnotationNamespaceKey: "podnamespaces",
podAnnotationUIDkey: "poduid",
podAnnotationClusterNameKey: "podclustername",
podAnnotationNodeName: "podnodename",
}
return http.StatusOK, &v1.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: podName,
Namespace: podNamespace,
Annotations: annotations,
},
Spec: v1.PodSpec{
Containers: []v1.Container{
v1.Container{
Name: "nginx",
},
},
},
}
}
pod, err := provider.GetPod(context.Background(), podNamespace, podName)
if err != nil {
t.Fatal("Failed to get pod", err)
}
assert.Check(t, pod != nil, "Response pod should not be nil")
assert.Check(t, pod.Spec.Containers != nil, "Containers should not be nil")
assert.Check(t, is.Equal(pod.Name, "podname"), "Pod name is not expected")
assert.Check(t, is.Equal(pod.Namespace, "podnamespaces"), "Pod namespace is not expected")
assert.Check(t, is.Nil(pod.Annotations), "Pod Annotations should be nil")
assert.Check(t, is.Equal(string(pod.UID), "poduid"), "Pod UID is not expected")
assert.Check(t, is.Equal(pod.ClusterName, "podclustername"), "Pod clustername is not expected")
assert.Check(t, is.Equal(pod.Spec.NodeName, "podnodename"), "Pod node name is not expected")
}
// Tests get pod.
func TestGetPods(t *testing.T) {
cciServerMocker, provider, err := prepareMocks()
if err != nil {
t.Fatal("Unable to prepare the mocks", err)
}
podName := "pod-" + string(uuid.NewUUID())
podNamespace := "ns-" + string(uuid.NewUUID())
cciServerMocker.OnGetPods = func() (int, interface{}) {
annotations := map[string]string{
podAnnotationPodNameKey: "podname",
podAnnotationNamespaceKey: "podnamespaces",
podAnnotationUIDkey: "poduid",
podAnnotationClusterNameKey: "podclustername",
podAnnotationNodeName: "podnodename",
}
pod := v1.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: podName,
Namespace: podNamespace,
Annotations: annotations,
},
Spec: v1.PodSpec{
Containers: []v1.Container{
v1.Container{
Name: "nginx",
},
},
},
}
return http.StatusOK, []v1.Pod{pod}
}
pods, err := provider.GetPods(context.Background())
if err != nil {
t.Fatal("Failed to get pods", err)
}
pod := pods[0]
assert.Check(t, pod != nil, "Response pod should not be nil")
assert.Check(t, pod.Spec.Containers != nil, "Containers should not be nil")
assert.Check(t, is.Equal(pod.Name, "podname"), "Pod name is not expected")
assert.Check(t, is.Equal(pod.Namespace, "podnamespaces"), "Pod namespace is not expected")
assert.Check(t, is.Nil(pod.Annotations), "Pod Annotations should be nil")
assert.Check(t, is.Equal(string(pod.UID), "poduid"), "Pod UID is not expected")
assert.Check(t, is.Equal(pod.ClusterName, "podclustername"), "Pod clustername is not expected")
assert.Check(t, is.Equal(pod.Spec.NodeName, "podnodename"), "Pod node name is not expected")
}
func prepareMocks() (*CCIMock, *CCIProvider, error) {
cciServerMocker := NewCCIMock()
os.Setenv("CCI_APP_KEP", fakeAppKey)
os.Setenv("CCI_APP_SECRET", fakeAppSecret)
defaultApiEndpoint = cciServerMocker.GetServerURL()
provider, err := NewCCIProvider("cci.toml", nil, fakeNodeName, "Linux", "0.0.0.0", 10250)
if err != nil {
return nil, nil, err
}
provider.project = fakeProject
provider.client.Signer = &fakeSigner{
AppKey: fakeAppKey,
AppSecret: fakeAppSecret,
Region: fakeRegion,
Service: fakeService,
}
return cciServerMocker, provider, nil
}

54
providers/huawei/config.go
View File

@@ -1,54 +0,0 @@
package huawei
import (
"io"
"github.com/BurntSushi/toml"
"github.com/virtual-kubelet/virtual-kubelet/providers"
"k8s.io/apimachinery/pkg/util/uuid"
)
type providerConfig struct {
Project string
Region string
Service string
OperatingSystem string
CPU string
Memory string
Pods string
}
func (p *CCIProvider) loadConfig(r io.Reader) error {
var config providerConfig
if _, err := toml.DecodeReader(r, &config); err != nil {
return err
}
p.apiEndpoint = defaultApiEndpoint
p.service = "CCI"
p.region = config.Region
if p.region == "" {
p.region = "southchina"
}
p.cpu = config.CPU
if p.cpu == "" {
p.cpu = "20"
}
p.memory = config.Memory
if p.memory == "" {
p.memory = "100Gi"
}
p.pods = config.Pods
if p.pods == "" {
p.pods = "20"
}
p.project = config.Project
if p.project == "" {
p.project = string(uuid.NewUUID())
}
p.operatingSystem = config.OperatingSystem
if p.operatingSystem == "" {
p.operatingSystem = providers.OperatingSystemLinux
}
return nil
}

58
providers/mock/mock.go
View File

@@ -35,14 +35,14 @@ const (
// See: https://github.com/virtual-kubelet/virtual-kubelet/issues/632
/*
var (
_ providers.Provider = (*MockLegacyProvider)(nil)
_ providers.PodMetricsProvider = (*MockLegacyProvider)(nil)
_ providers.Provider = (*MockV0Provider)(nil)
_ providers.PodMetricsProvider = (*MockV0Provider)(nil)
_ node.PodNotifier = (*MockProvider)(nil)
)
*/
// MockLegacyProvider implements the virtual-kubelet provider interface and stores pods in memory.
type MockLegacyProvider struct {
// MockV0Provider implements the virtual-kubelet provider interface and stores pods in memory.
type MockV0Provider struct {
nodeName string
operatingSystem string
internalIP string
@@ -53,9 +53,9 @@ type MockLegacyProvider struct {
notifier func(*v1.Pod)
}
// MockProvider is like MockLegacyProvider, but implements the PodNotifier interface
// MockProvider is like MockV0Provider, but implements the PodNotifier interface
type MockProvider struct {
*MockLegacyProvider
*MockV0Provider
}
// MockConfig contains a mock virtual-kubelet's configurable parameters.
@@ -65,8 +65,8 @@ type MockConfig struct {
Pods string `json:"pods,omitempty"`
}
// NewMockProviderMockConfig creates a new MockLegacyProvider. Mock legacy provider does not implement the new asynchronous podnotifier interface
func NewMockLegacyProviderMockConfig(config MockConfig, nodeName, operatingSystem string, internalIP string, daemonEndpointPort int32) (*MockLegacyProvider, error) {
// NewMockProviderMockConfig creates a new MockV0Provider. Mock legacy provider does not implement the new asynchronous podnotifier interface
func NewMockV0ProviderMockConfig(config MockConfig, nodeName, operatingSystem string, internalIP string, daemonEndpointPort int32) (*MockV0Provider, error) {
//set defaults
if config.CPU == "" {
config.CPU = defaultCPUCapacity
@@ -77,7 +77,7 @@ func NewMockLegacyProviderMockConfig(config MockConfig, nodeName, operatingSyste
if config.Pods == "" {
config.Pods = defaultPodCapacity
}
provider := MockLegacyProvider{
provider := MockV0Provider{
nodeName: nodeName,
operatingSystem: operatingSystem,
internalIP: internalIP,
@@ -94,21 +94,21 @@ func NewMockLegacyProviderMockConfig(config MockConfig, nodeName, operatingSyste
return &provider, nil
}
// NewMockLegacyProvider creates a new MockLegacyProvider
func NewMockLegacyProvider(providerConfig, nodeName, operatingSystem string, internalIP string, daemonEndpointPort int32) (*MockLegacyProvider, error) {
// NewMockV0Provider creates a new MockV0Provider
func NewMockV0Provider(providerConfig, nodeName, operatingSystem string, internalIP string, daemonEndpointPort int32) (*MockV0Provider, error) {
config, err := loadConfig(providerConfig, nodeName)
if err != nil {
return nil, err
}
return NewMockLegacyProviderMockConfig(config, nodeName, operatingSystem, internalIP, daemonEndpointPort)
return NewMockV0ProviderMockConfig(config, nodeName, operatingSystem, internalIP, daemonEndpointPort)
}
// NewMockProviderMockConfig creates a new MockProvider with the given config
func NewMockProviderMockConfig(config MockConfig, nodeName, operatingSystem string, internalIP string, daemonEndpointPort int32) (*MockProvider, error) {
p, err := NewMockLegacyProviderMockConfig(config, nodeName, operatingSystem, internalIP, daemonEndpointPort)
p, err := NewMockV0ProviderMockConfig(config, nodeName, operatingSystem, internalIP, daemonEndpointPort)
return &MockProvider{MockLegacyProvider: p}, err
return &MockProvider{MockV0Provider: p}, err
}
// NewMockProvider creates a new MockProvider, which implements the PodNotifier interface
@@ -158,7 +158,7 @@ func loadConfig(providerConfig, nodeName string) (config MockConfig, err error)
}
// CreatePod accepts a Pod definition and stores it in memory.
func (p *MockLegacyProvider) CreatePod(ctx context.Context, pod *v1.Pod) error {
func (p *MockV0Provider) CreatePod(ctx context.Context, pod *v1.Pod) error {
ctx, span := trace.StartSpan(ctx, "CreatePod")
defer span.End()
@@ -215,7 +215,7 @@ func (p *MockLegacyProvider) CreatePod(ctx context.Context, pod *v1.Pod) error {
}
// UpdatePod accepts a Pod definition and updates its reference.
func (p *MockLegacyProvider) UpdatePod(ctx context.Context, pod *v1.Pod) error {
func (p *MockV0Provider) UpdatePod(ctx context.Context, pod *v1.Pod) error {
ctx, span := trace.StartSpan(ctx, "UpdatePod")
defer span.End()
@@ -236,7 +236,7 @@ func (p *MockLegacyProvider) UpdatePod(ctx context.Context, pod *v1.Pod) error {
}
// DeletePod deletes the specified pod out of memory.
func (p *MockLegacyProvider) DeletePod(ctx context.Context, pod *v1.Pod) (err error) {
func (p *MockV0Provider) DeletePod(ctx context.Context, pod *v1.Pod) (err error) {
ctx, span := trace.StartSpan(ctx, "DeletePod")
defer span.End()
@@ -277,7 +277,7 @@ func (p *MockLegacyProvider) DeletePod(ctx context.Context, pod *v1.Pod) (err er
}
// GetPod returns a pod by name that is stored in memory.
func (p *MockLegacyProvider) GetPod(ctx context.Context, namespace, name string) (pod *v1.Pod, err error) {
func (p *MockV0Provider) GetPod(ctx context.Context, namespace, name string) (pod *v1.Pod, err error) {
ctx, span := trace.StartSpan(ctx, "GetPod")
defer func() {
span.SetStatus(err)
@@ -301,7 +301,7 @@ func (p *MockLegacyProvider) GetPod(ctx context.Context, namespace, name string)
}
// GetContainerLogs retrieves the logs of a container by name from the provider.
func (p *MockLegacyProvider) GetContainerLogs(ctx context.Context, namespace, podName, containerName string, opts api.ContainerLogOpts) (io.ReadCloser, error) {
func (p *MockV0Provider) GetContainerLogs(ctx context.Context, namespace, podName, containerName string, opts api.ContainerLogOpts) (io.ReadCloser, error) {
ctx, span := trace.StartSpan(ctx, "GetContainerLogs")
defer span.End()
@@ -314,20 +314,20 @@ func (p *MockLegacyProvider) GetContainerLogs(ctx context.Context, namespace, po
// Get full pod name as defined in the provider context
// TODO: Implementation
func (p *MockLegacyProvider) GetPodFullName(namespace string, pod string) string {
func (p *MockV0Provider) GetPodFullName(namespace string, pod string) string {
return ""
}
// RunInContainer executes a command in a container in the pod, copying data
// between in/out/err and the container's stdin/stdout/stderr.
func (p *MockLegacyProvider) RunInContainer(ctx context.Context, namespace, name, container string, cmd []string, attach api.AttachIO) error {
func (p *MockV0Provider) RunInContainer(ctx context.Context, namespace, name, container string, cmd []string, attach api.AttachIO) error {
log.G(context.TODO()).Infof("receive ExecInContainer %q", container)
return nil
}
// GetPodStatus returns the status of a pod by name that is "running".
// returns nil if a pod by that name is not found.
func (p *MockLegacyProvider) GetPodStatus(ctx context.Context, namespace, name string) (*v1.PodStatus, error) {
func (p *MockV0Provider) GetPodStatus(ctx context.Context, namespace, name string) (*v1.PodStatus, error) {
ctx, span := trace.StartSpan(ctx, "GetPodStatus")
defer span.End()
@@ -345,7 +345,7 @@ func (p *MockLegacyProvider) GetPodStatus(ctx context.Context, namespace, name s
}
// GetPods returns a list of all pods known to be "running".
func (p *MockLegacyProvider) GetPods(ctx context.Context) ([]*v1.Pod, error) {
func (p *MockV0Provider) GetPods(ctx context.Context) ([]*v1.Pod, error) {
ctx, span := trace.StartSpan(ctx, "GetPods")
defer span.End()
@@ -361,7 +361,7 @@ func (p *MockLegacyProvider) GetPods(ctx context.Context) ([]*v1.Pod, error) {
}
// Capacity returns a resource list containing the capacity limits.
func (p *MockLegacyProvider) Capacity(ctx context.Context) v1.ResourceList {
func (p *MockV0Provider) Capacity(ctx context.Context) v1.ResourceList {
ctx, span := trace.StartSpan(ctx, "Capacity")
defer span.End()
@@ -374,7 +374,7 @@ func (p *MockLegacyProvider) Capacity(ctx context.Context) v1.ResourceList {
// NodeConditions returns a list of conditions (Ready, OutOfDisk, etc), for updates to the node status
// within Kubernetes.
func (p *MockLegacyProvider) NodeConditions(ctx context.Context) []v1.NodeCondition {
func (p *MockV0Provider) NodeConditions(ctx context.Context) []v1.NodeCondition {
ctx, span := trace.StartSpan(ctx, "NodeConditions")
defer span.End()
@@ -426,7 +426,7 @@ func (p *MockLegacyProvider) NodeConditions(ctx context.Context) []v1.NodeCondit
// NodeAddresses returns a list of addresses for the node status
// within Kubernetes.
func (p *MockLegacyProvider) NodeAddresses(ctx context.Context) []v1.NodeAddress {
func (p *MockV0Provider) NodeAddresses(ctx context.Context) []v1.NodeAddress {
ctx, span := trace.StartSpan(ctx, "NodeAddresses")
defer span.End()
@@ -440,7 +440,7 @@ func (p *MockLegacyProvider) NodeAddresses(ctx context.Context) []v1.NodeAddress
// NodeDaemonEndpoints returns NodeDaemonEndpoints for the node status
// within Kubernetes.
func (p *MockLegacyProvider) NodeDaemonEndpoints(ctx context.Context) *v1.NodeDaemonEndpoints {
func (p *MockV0Provider) NodeDaemonEndpoints(ctx context.Context) *v1.NodeDaemonEndpoints {
ctx, span := trace.StartSpan(ctx, "NodeDaemonEndpoints")
defer span.End()
@@ -454,13 +454,13 @@ func (p *MockLegacyProvider) NodeDaemonEndpoints(ctx context.Context) *v1.NodeDa
// OperatingSystem returns the operating system for this provider.
// This is a noop to default to Linux for now.
func (p *MockLegacyProvider) OperatingSystem() string {
func (p *MockV0Provider) OperatingSystem() string {
// This is harcoded due to: https://github.com/virtual-kubelet/virtual-kubelet/issues/632
return "Linux"
}
// GetStatsSummary returns dummy stats for all pods known by this provider.
func (p *MockLegacyProvider) GetStatsSummary(ctx context.Context) (*stats.Summary, error) {
func (p *MockV0Provider) GetStatsSummary(ctx context.Context) (*stats.Summary, error) {
ctx, span := trace.StartSpan(ctx, "GetStatsSummary")
defer span.End()

119
providers/nomad/README.md
View File

@@ -1,119 +0,0 @@
# HashiCorp Nomad Provider for Virtual Kubelet
HashiCorp [Nomad](https://nomadproject.io) provider for Virtual Kubelet connects your Kubernetes cluster
with Nomad cluster by exposing the Nomad cluster as a node in Kubernetes. By
using the provider, pods that are scheduled on the virtual Nomad node
registered on Kubernetes will run as jobs on Nomad clients as they
would on a Kubernetes node.
**This is an experimental project. This project isn't production ready.**
## Demo
![Virtual Kubelet Nomad Provider Demo](./images/virtual-kubelet-nomad-provider-showcase.gif "Virtual Kubelet Nomad Provider Demo")
## Prerequisites
This guide assumes the following:
* A Nomad cluster up and running.
* A Kubernetes cluster up and running.
* The Nomad API is accessible from the Kubernetes cluster.
* [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl) installed.
## Usage
The Nomad provider accepts the following two environment variables:
* `NOMAD_ADDR` - The Nomad API address. Set to `127.0.0.1:4646` by default.
* `NOMAD_REGION` - The Nomad region. Set to `global` by default.
```bash
export NOMAD_ADDR="127.0.0.1:4646"
export NOMAD_REGION="global"
```
### Run Virtual Kubelet with Nomad Provider
```bash
VK_TAINT_KEY="hashicorp.com/nomad" ./virtual-kubelet --provider="nomad"
```
Validate that the virtual kubelet node is registered.
```bash
kubectl get nodes
```
Expected output.
```bash
NAME STATUS ROLES AGE VERSION
minikube Ready master 55d v1.10.0
virtual-kubelet Ready agent 1m v1.13.1-vk-N/A
```
### Create a Pod in Kubernetes
```bash
kubectl apply -f pods/nginx-pod.yaml
```
Validate pod.
```bash
kubectl get pods
```
Expected output.
```bash
NAME READY STATUS RESTARTS AGE
nginx 1/1 Running 0 1m
```
Validate Nomad job.
```bash
nomad status
```
Expected output.
```bash
ID Type Priority Status Submit Date
nomad-virtual-kubelet-nginx service 100 running 2018-12-31T16:52:52+05:30
```
### Configuration Options
The Nomad provider has support for annotations to define Nomad [datacenters](https://www.nomadproject.io/docs/job-specification/job.html#datacenters).
Here is an example usage of the Nomad datacenter annotations in a pod spec.
```yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx
annotations:
"nomad.hashicorp.com/datacenters": "us-east1,us-west1"
spec:
containers:
- image: nginx
imagePullPolicy: Always
name: nginx
ports:
- containerPort: 80
- containerPort: 443
dnsPolicy: ClusterFirst
nodeSelector:
kubernetes.io/role: agent
beta.kubernetes.io/os: linux
type: virtual-kubelet
tolerations:
- key: virtual-kubelet.io/provider
operator: Exists
- key: hashicorp.com/nomad
effect: NoSchedule
```

297
providers/nomad/helpers.go
View File

@@ -1,297 +0,0 @@
package nomad
import (
"fmt"
"strconv"
"strings"
"time"
nomad "github.com/hashicorp/nomad/api"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// createNomadTasks takes the containers in a kubernetes pod and creates
// a list of Nomad tasks from them.
func (p *Provider) createNomadTasks(pod *v1.Pod) []*nomad.Task {
nomadTasks := make([]*nomad.Task, 0, len(pod.Spec.Containers))
for _, ctr := range pod.Spec.Containers {
portMap, networkResourcess := createPortMap(ctr.Ports)
image := ctr.Image
labels := pod.Labels
command := ctr.Command
args := ctr.Args
resources := createResources(ctr.Resources.Limits, networkResourcess)
envVars := createEnvVars(ctr.Env)
task := nomad.Task{
Name: ctr.Name,
Driver: "docker",
Config: map[string]interface{}{
"image": image,
"port_map": portMap,
"labels": labels,
// TODO: Add volumes support
"command": strings.Join(command, ""),
"args": args,
},
Resources: resources,
Env: envVars,
}
nomadTasks = append(nomadTasks, &task)
}
return nomadTasks
}
func createPortMap(ports []v1.ContainerPort) ([]map[string]interface{}, []*nomad.NetworkResource) {
var portMap []map[string]interface{}
var dynamicPorts []nomad.Port
var networkResources []*nomad.NetworkResource
for i, port := range ports {
portName := fmt.Sprintf("port_%s", strconv.Itoa(i+1))
if port.Name != "" {
portName = port.Name
}
portMap = append(portMap, map[string]interface{}{portName: port.ContainerPort})
dynamicPorts = append(dynamicPorts, nomad.Port{Label: portName})
}
return portMap, append(networkResources, &nomad.NetworkResource{DynamicPorts: dynamicPorts})
}
func createResources(limits v1.ResourceList, networkResources []*nomad.NetworkResource) *nomad.Resources {
taskMemory := int(limits.Memory().Value())
taskCPU := int(limits.Cpu().Value())
if taskMemory == 0 {
taskMemory = 128
}
if taskCPU == 0 {
taskCPU = 100
}
return &nomad.Resources{
Networks: networkResources,
MemoryMB: &taskMemory,
CPU: &taskCPU,
}
}
func createEnvVars(podEnvVars []v1.EnvVar) map[string]string {
envVars := map[string]string{}
for _, v := range podEnvVars {
envVars[v.Name] = v.Value
}
return envVars
}
func (p *Provider) createTaskGroups(name string, tasks []*nomad.Task) []*nomad.TaskGroup {
count := 1
restartDelay := 1 * time.Second
restartMode := "delay"
restartAttempts := 25
return []*nomad.TaskGroup{
&nomad.TaskGroup{
Name: &name,
Count: &count,
RestartPolicy: &nomad.RestartPolicy{
Delay: &restartDelay,
Mode: &restartMode,
Attempts: &restartAttempts,
},
Tasks: tasks,
},
}
}
func (p *Provider) createJob(name string, datacenters []string, taskGroups []*nomad.TaskGroup) *nomad.Job {
jobName := fmt.Sprintf("%s-%s", jobNamePrefix, name)
// Create a new nomad job
job := nomad.NewServiceJob(jobName, jobName, p.nomadRegion, 100)
job.Datacenters = datacenters
job.TaskGroups = taskGroups
return job
}
func (p *Provider) jobToPod(job *nomad.Job, allocs []*nomad.AllocationListStub) (*v1.Pod, error) {
containers := []v1.Container{}
containerStatues := []v1.ContainerStatus{}
jobStatus := *job.Status
jobCreatedAt := *job.SubmitTime
podCondition := convertJobStatusToPodCondition(jobStatus)
containerStatusesMap := allocToContainerStatuses(allocs)
// containerPorts are specified for task in a task
// group
var containerPorts []v1.ContainerPort
for _, tg := range job.TaskGroups {
for _, task := range tg.Tasks {
for _, taskNetwork := range task.Resources.Networks {
for _, dynamicPort := range taskNetwork.DynamicPorts {
// TODO: Dynamic ports aren't being reported via the
// Nomad `/jobs` endpoint.
containerPorts = append(containerPorts, v1.ContainerPort{
Name: dynamicPort.Label,
HostPort: int32(dynamicPort.Value),
HostIP: taskNetwork.IP,
})
}
}
containers = append(containers, v1.Container{
Name: task.Name,
Image: fmt.Sprintf("%s", task.Config["image"]),
Command: strings.Split(fmt.Sprintf("%s", task.Config["command"]), ""),
Args: strings.Split(fmt.Sprintf("%s", task.Config["args"]), " "),
Ports: containerPorts,
})
containerStatus := containerStatusesMap[task.Name]
containerStatus.Image = fmt.Sprintf("%s", task.Config["image"])
containerStatus.ImageID = fmt.Sprintf("%s", task.Config["image"])
containerStatues = append(containerStatues, containerStatus)
}
}
pod := v1.Pod{
TypeMeta: metav1.TypeMeta{
Kind: "Pod",
APIVersion: "v1",
},
ObjectMeta: metav1.ObjectMeta{
Name: *job.Name,
Namespace: "default",
CreationTimestamp: metav1.NewTime(time.Unix(jobCreatedAt, 0)),
},
Spec: v1.PodSpec{
NodeName: p.nodeName,
Volumes: []v1.Volume{},
Containers: containers,
},
Status: v1.PodStatus{
Phase: jobStatusToPodPhase(jobStatus),
Conditions: []v1.PodCondition{podCondition},
Message: "",
Reason: "",
HostIP: "", // TODO: find out the HostIP
PodIP: "", // TODO: find out the equalent for PodIP
ContainerStatuses: containerStatues,
},
}
return &pod, nil
}
func allocToContainerStatuses(allocs []*nomad.AllocationListStub) map[string]v1.ContainerStatus {
containerStatusesMap := map[string]v1.ContainerStatus{}
for _, alloc := range allocs {
for name, taskState := range alloc.TaskStates {
containerState, readyFlag := convertTaskStateToContainerState(taskState.State,
taskState.StartedAt,
taskState.FinishedAt,
)
containerStatusesMap[name] = v1.ContainerStatus{
Name: name,
RestartCount: int32(taskState.Restarts),
Ready: readyFlag,
State: containerState,
}
}
}
return containerStatusesMap
}
func jobStatusToPodPhase(status string) v1.PodPhase {
switch status {
case "pending":
return v1.PodPending
case "running":
return v1.PodRunning
// TODO: Make sure we take PodFailed into account.
case "dead":
return v1.PodFailed
}
return v1.PodUnknown
}
func convertJobStatusToPodCondition(jobStatus string) v1.PodCondition {
podCondition := v1.PodCondition{}
switch jobStatus {
case "pending":
podCondition = v1.PodCondition{
Type: v1.PodInitialized,
Status: v1.ConditionFalse,
}
case "running":
podCondition = v1.PodCondition{
Type: v1.PodReady,
Status: v1.ConditionTrue,
}
case "dead":
podCondition = v1.PodCondition{
Type: v1.PodReasonUnschedulable,
Status: v1.ConditionFalse,
}
default:
podCondition = v1.PodCondition{
Type: v1.PodReasonUnschedulable,
Status: v1.ConditionUnknown,
}
}
return podCondition
}
func convertTaskStateToContainerState(taskState string, startedAt time.Time, finishedAt time.Time) (v1.ContainerState, bool) {
containerState := v1.ContainerState{}
readyFlag := false
switch taskState {
case "pending":
containerState = v1.ContainerState{
Waiting: &v1.ContainerStateWaiting{},
}
case "running":
containerState = v1.ContainerState{
Running: &v1.ContainerStateRunning{
StartedAt: metav1.NewTime(startedAt),
},
}
readyFlag = true
// TODO: Make sure containers that are exiting with non-zero status codes
// are accounted for using events or something similar?
//case v1.PodSucceeded:
// podCondition = v1.PodCondition{
// Type: v1.PodReasonUnschedulable,
// Status: v1.ConditionFalse,
// }
// containerState = v1.ContainerState{
// Terminated: &v1.ContainerStateTerminated{
// ExitCode: int32(container.State.ExitCode),
// FinishedAt: metav1.NewTime(finishedAt),
// },
// }
case "dead":
containerState = v1.ContainerState{
Terminated: &v1.ContainerStateTerminated{
ExitCode: 0,
FinishedAt: metav1.NewTime(finishedAt),
},
}
default:
containerState = v1.ContainerState{}
}
return containerState, readyFlag
}

BIN
providers/nomad/images/virtual-kubelet-nomad-provider-showcase.gif
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.7 MiB

290
providers/nomad/nomad.go
View File

@@ -1,290 +0,0 @@
package nomad
import (
"context"
"fmt"
"io"
"io/ioutil"
"log"
"os"
"strings"
nomad "github.com/hashicorp/nomad/api"
"github.com/virtual-kubelet/virtual-kubelet/manager"
"github.com/virtual-kubelet/virtual-kubelet/node/api"
"github.com/virtual-kubelet/virtual-kubelet/providers"
v1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/resource"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// Nomad provider constants
const (
jobNamePrefix = "nomad-virtual-kubelet"
nomadDatacentersAnnotation = "nomad.hashicorp.com/datacenters"
defaultNomadAddress = "127.0.0.1:4646"
defaultNomadDatacenter = "dc1"
defaultNomadRegion = "global"
)
// Provider implements the virtual-kubelet provider interface and communicates with the Nomad API.
type Provider struct {
nomadClient *nomad.Client
resourceManager *manager.ResourceManager
nodeName string
operatingSystem string
nomadAddress string
nomadRegion string
cpu string
memory string
pods string
}
// NewProvider creates a new Provider
func NewProvider(rm *manager.ResourceManager, nodeName, operatingSystem string) (*Provider, error) {
p := Provider{}
p.resourceManager = rm
p.nodeName = nodeName
p.operatingSystem = operatingSystem
p.nomadAddress = os.Getenv("NOMAD_ADDR")
p.nomadRegion = os.Getenv("NOMAD_REGION")
if p.nomadAddress == "" {
p.nomadAddress = defaultNomadAddress
}
if p.nomadRegion == "" {
p.nomadRegion = defaultNomadRegion
}
c := nomad.DefaultConfig()
log.Printf("nomad client address: %s", p.nomadAddress)
nomadClient, err := nomad.NewClient(c.ClientConfig(p.nomadRegion, p.nomadAddress, false))
if err != nil {
log.Printf("Unable to create nomad client: %s", err)
return nil, err
}
p.nomadClient = nomadClient
return &p, nil
}
// CreatePod accepts a Pod definition and creates
// a Nomad job
func (p *Provider) CreatePod(ctx context.Context, pod *v1.Pod) error {
log.Printf("CreatePod %q\n", pod.Name)
// Ignore daemonSet Pod
if pod != nil && pod.OwnerReferences != nil && len(pod.OwnerReferences) != 0 && pod.OwnerReferences[0].Kind == "DaemonSet" {
log.Printf("Skip to create DaemonSet pod %q\n", pod.Name)
return nil
}
// Default datacenter name
datacenters := []string{defaultNomadDatacenter}
nomadDatacenters := pod.Annotations[nomadDatacentersAnnotation]
if nomadDatacenters != "" {
datacenters = strings.Split(nomadDatacenters, ",")
}
// Create a list of nomad tasks
nomadTasks := p.createNomadTasks(pod)
taskGroups := p.createTaskGroups(pod.Name, nomadTasks)
job := p.createJob(pod.Name, datacenters, taskGroups)
// Register nomad job
_, _, err := p.nomadClient.Jobs().Register(job, nil)
if err != nil {
return fmt.Errorf("couldn't start nomad job: %q", err)
}
return nil
}
// UpdatePod is a noop, nomad does not support live updates of a pod.
func (p *Provider) UpdatePod(ctx context.Context, pod *v1.Pod) error {
log.Println("Pod Update called: No-op as not implemented")
return nil
}
// DeletePod accepts a Pod definition and deletes a Nomad job.
func (p *Provider) DeletePod(ctx context.Context, pod *v1.Pod) (err error) {
// Deregister job
response, _, err := p.nomadClient.Jobs().Deregister(pod.Name, true, nil)
if err != nil {
return fmt.Errorf("couldn't stop or deregister nomad job: %s: %s", response, err)
}
log.Printf("deregistered nomad job %q response %q\n", pod.Name, response)
return nil
}
// GetPod returns the pod running in the Nomad cluster. returns nil
// if pod is not found.
func (p *Provider) GetPod(ctx context.Context, namespace, name string) (pod *v1.Pod, err error) {
jobID := fmt.Sprintf("%s-%s", jobNamePrefix, name)
// Get nomad job
job, _, err := p.nomadClient.Jobs().Info(jobID, nil)
if err != nil {
return nil, fmt.Errorf("couldn't retrieve nomad job: %s", err)
}
// Get nomad job allocations to get individual task statuses
jobAllocs, _, err := p.nomadClient.Jobs().Allocations(jobID, false, nil)
if err != nil {
return nil, fmt.Errorf("couldn't retrieve nomad job allocations: %s", err)
}
// Change a nomad job into a kubernetes pod
pod, err = p.jobToPod(job, jobAllocs)
if err != nil {
return nil, fmt.Errorf("couldn't convert a nomad job into a pod: %s", err)
}
return pod, nil
}
// GetContainerLogs retrieves the logs of a container by name from the provider.
func (p *Provider) GetContainerLogs(ctx context.Context, namespace, podName, containerName string, opts api.ContainerLogOpts) (io.ReadCloser, error) {
return ioutil.NopCloser(strings.NewReader("")), nil
}
// GetPodFullName as defined in the provider context
func (p *Provider) GetPodFullName(ctx context.Context, namespace string, pod string) string {
return fmt.Sprintf("%s-%s", jobNamePrefix, pod)
}
// RunInContainer executes a command in a container in the pod, copying data
// between in/out/err and the container's stdin/stdout/stderr.
// TODO: Implementation
func (p *Provider) RunInContainer(ctx context.Context, namespace, name, container string, cmd []string, attach api.AttachIO) error {
log.Printf("ExecInContainer %q\n", container)
return nil
}
// GetPodStatus returns the status of a pod by name that is running as a job
// in the Nomad cluster returns nil if a pod by that name is not found.
func (p *Provider) GetPodStatus(ctx context.Context, namespace, name string) (*v1.PodStatus, error) {
pod, err := p.GetPod(ctx, namespace, name)
if err != nil {
return nil, err
}
return &pod.Status, nil
}
// GetPods returns a list of all pods known to be running in Nomad nodes.
func (p *Provider) GetPods(ctx context.Context) ([]*v1.Pod, error) {
log.Printf("GetPods\n")
jobsList, _, err := p.nomadClient.Jobs().PrefixList(jobNamePrefix)
if err != nil {
return nil, fmt.Errorf("couldn't get job list from nomad: %s", err)
}
var pods = []*v1.Pod{}
for _, job := range jobsList {
// Get nomad job
j, _, err := p.nomadClient.Jobs().Info(job.ID, nil)
if err != nil {
return nil, fmt.Errorf("couldn't retrieve nomad job: %s", err)
}
// Get nomad job allocations to get individual task statuses
jobAllocs, _, err := p.nomadClient.Jobs().Allocations(job.ID, false, nil)
if err != nil {
return nil, fmt.Errorf("couldn't retrieve nomad job allocations: %s", err)
}
// Change a nomad job into a kubernetes pod
pod, err := p.jobToPod(j, jobAllocs)
if err != nil {
return nil, fmt.Errorf("couldn't convert a nomad job into a pod: %s", err)
}
pods = append(pods, pod)
}
return pods, nil
}
// Capacity returns a resource list containing the capacity limits set for Nomad.
func (p *Provider) Capacity(ctx context.Context) v1.ResourceList {
// TODO: Use nomad /nodes api to get a list of nodes in the cluster
// and then use the read node /node/:node_id endpoint to calculate
// the total resources of the cluster to report back to kubernetes.
return v1.ResourceList{
"cpu": resource.MustParse("20"),
"memory": resource.MustParse("100Gi"),
"pods": resource.MustParse("20"),
}
}
// NodeConditions returns a list of conditions (Ready, OutOfDisk, etc), for updates to the node status
// within Kubernetes.
func (p *Provider) NodeConditions(ctx context.Context) []v1.NodeCondition {
// TODO: Make these dynamic.
return []v1.NodeCondition{
{
Type: "Ready",
Status: v1.ConditionTrue,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletReady",
Message: "kubelet is ready.",
},
{
Type: "OutOfDisk",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletHasSufficientDisk",
Message: "kubelet has sufficient disk space available",
},
{
Type: "MemoryPressure",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletHasSufficientMemory",
Message: "kubelet has sufficient memory available",
},
{
Type: "DiskPressure",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletHasNoDiskPressure",
Message: "kubelet has no disk pressure",
},
{
Type: "NetworkUnavailable",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "RouteCreated",
Message: "RouteController created a route",
},
}
}
// NodeAddresses returns a list of addresses for the node status
// within Kubernetes.
func (p *Provider) NodeAddresses(ctx context.Context) []v1.NodeAddress {
// TODO: Use nomad api to get a list of node addresses.
return nil
}
// NodeDaemonEndpoints returns NodeDaemonEndpoints for the node status
// within Kubernetes.
func (p *Provider) NodeDaemonEndpoints(ctx context.Context) *v1.NodeDaemonEndpoints {
return &v1.NodeDaemonEndpoints{}
}
// OperatingSystem returns the operating system for this provider.
// This is a noop to default to Linux for now.
func (p *Provider) OperatingSystem() string {
return providers.OperatingSystemLinux
}

126
providers/nomad/nomad_test.go
View File

@@ -1,126 +0,0 @@
package nomad
import (
"context"
"fmt"
"os"
"testing"
"github.com/google/uuid"
nomad "github.com/hashicorp/nomad/api"
"github.com/hashicorp/nomad/testutil"
"gotest.tools/assert"
is "gotest.tools/assert/cmp"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/util/intstr"
)
// Client provides a client to the Nomad API
type Client struct {
config nomad.Config
}
func TestCreateGetDeletePod(t *testing.T) {
provider, err := makeProvider(t)
if err != nil {
t.Fatal("unable to create mock provider", err)
}
nomadClient, nomadServer := makeClient(t, nil)
defer nomadServer.Stop()
provider.nomadClient = nomadClient
provider.nomadAddress = nomadServer.HTTPAddr
podName := "pod-" + uuid.New().String()
podNamespace := "ns-" + uuid.New().String()
pod := &v1.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: podName,
Namespace: podNamespace,
},
Spec: v1.PodSpec{
Containers: []v1.Container{
v1.Container{
Name: "nginx",
Image: "nginx",
LivenessProbe: &v1.Probe{
Handler: v1.Handler{
HTTPGet: &v1.HTTPGetAction{
Port: intstr.FromString("8080"),
Path: "/",
},
},
InitialDelaySeconds: 10,
PeriodSeconds: 5,
TimeoutSeconds: 60,
SuccessThreshold: 3,
FailureThreshold: 5,
},
},
},
},
}
// Create pod
err = provider.CreatePod(context.Background(), pod)
if err != nil {
t.Fatal("failed to create pod", err)
}
// Get pod
pod, err = provider.GetPod(context.Background(), podNamespace, podName)
if err != nil {
t.Fatal("failed to get pod", err)
}
// Get pod tests
// Validate pod spec
assert.Check(t, pod != nil, "pod cannot be nil")
assert.Check(t, pod.Spec.Containers != nil, "containers cannot be nil")
assert.Check(t, is.Nil(pod.Annotations), "pod annotations should be nil")
assert.Check(t, is.Equal(pod.Name, fmt.Sprintf("%s-%s", jobNamePrefix, podName)), "pod name should be equal")
// Get pods
pods, err := provider.GetPods(context.Background())
if err != nil {
t.Fatal("failed to get pods", err)
}
// TODO: finish adding a few more assertions
assert.Check(t, is.Len(pods, 1), "number of pods should be 1")
// Delete pod
err = provider.DeletePod(context.Background(), pod)
if err != nil {
t.Fatal("failed to delete pod", err)
}
}
func makeClient(t *testing.T, cb testutil.ServerConfigCallback) (*nomad.Client, *testutil.TestServer) {
// Make client config
conf := nomad.DefaultConfig()
// Create server
server := testutil.NewTestServer(t, cb)
conf.Address = "http://" + server.HTTPAddr
// Create client
client, err := nomad.NewClient(conf)
if err != nil {
t.Fatalf("err: %v", err)
}
return client, server
}
func makeProvider(t *testing.T) (*Provider, error) {
// Set default region
os.Setenv("NOMAD_REGION", "global")
provider, err := NewProvider(nil, "fakeNomadNode", "linux")
if err != nil {
return nil, err
}
return provider, nil
}

22
providers/nomad/pods/nginx-pod.yaml
View File

@@ -1,22 +0,0 @@
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- image: nginx
imagePullPolicy: Always
name: nginx
ports:
- containerPort: 80
- containerPort: 443
dnsPolicy: ClusterFirst
nodeSelector:
kubernetes.io/role: agent
beta.kubernetes.io/os: linux
type: virtual-kubelet
tolerations:
- key: virtual-kubelet.io/provider
operator: Exists
- key: hashicorp.com/nomad
effect: NoSchedule

86
providers/openstack/README.md
View File

@@ -1,86 +0,0 @@
# OpenStack Zun
[OpenStack Zun](https://docs.openstack.org/zun/latest/) is an OpenStack Container service.
It aims to provide an API service for running application containers without the need to
manage servers or clusters.
## OpenStack Zun virtual-kubelet provider
OpenStack Zun virtual-kubelet provider connects your Kubernetes cluster to an OpenStack Cloud.
Your pods on OpenStack have access to OpenStack tenant networks since each pod is given
dedicated Neutron ports in your tenant subnets.
## Prerequisites
You need to have an OpenStack cloud with Zun service installed.
The quickest way to get everything setup is using
[Devstack](https://docs.openstack.org/zun/latest/contributor/quickstart.html).
If it is for production purpose, you follow the
[Zun installation guide](https://docs.openstack.org/zun/latest/install/index.html).
Another alternative is using
[Kolla](https://docs.openstack.org/kolla-ansible/latest/reference/compute/zun-guide.html).
## Authentication via Keystone
Virtual-kubelet needs permission to schedule pods on OpenStack Zun on your behalf.
You will need to retrieve your OpenStack credentials and store them as environment variables.
```console
export OS_DOMAIN_ID=default
export OS_REGION_NAME=RegionOne
export OS_PROJECT_NAME=demo
export OS_IDENTITY_API_VERSION=3
export OS_AUTH_URL=http://10.0.2.15/identity/v3
export OS_USERNAME=demo
export OS_PASSWORD=password
```
For users that have the OpenStack dashboard installed, there's a shortcut. If you visit the
project/access_and_security path in Horizon and click on the "Download OpenStack RC File" button
at the top right hand corner, you will download a bash file that exports all of your access details
to environment variables. To execute the file, run source admin-openrc.sh and you will be prompted
for your password.
## Connecting virtual-kubelet to your Kubernetes cluster
Start the virtual-kubelet process.
```console
virtual-kubelet --provider openstack
```
In your Kubernetes cluster, confirm that the virtual-kubelet shows up as a node.
```console
kubectl get nodes
NAME STATUS ROLES AGE VERSION
virtual-kubelet Ready agent 20d v1.13.1-vk-N/A
...
```
To disconnect, stop the virtual-kubelet process.
## Deploying Kubernetes pods in OpenStack Zun
In order to not break existing pod deployments, the OpenStack virtual node is given a taint.
Pods that are to be deployed on OpenStack require an explicit toleration that tolerates the
taint of the virtual node.
```
apiVersion: v1
kind: Pod
metadata:
name: myapp-pod
labels:
app: myapp
spec:
tolerations:
- key: "virtual-kubelet.io/provider"
operator: "Equal"
value: "openstack"
effect: "NoSchedule"
containers:
- name: myapp-container
image: busybox
command: ['sh', '-c', 'echo Hello Kubernetes! && sleep 3600']
```

56
providers/openstack/config.go
View File

@@ -1,56 +0,0 @@
package openstack
import (
"fmt"
"io"
"strings"
"github.com/BurntSushi/toml"
"github.com/virtual-kubelet/virtual-kubelet/providers"
)
type providerConfig struct {
Region string
OperatingSystem string
CPU string
Memory string
Pods string
}
func (p *ZunProvider) loadConfig(r io.Reader) error {
var config providerConfig
if _, err := toml.DecodeReader(r, &config); err != nil {
return err
}
p.region = config.Region
// Default to 20 mcpu
p.cpu = "20"
if config.CPU != "" {
p.cpu = config.CPU
}
// Default to 100Gi
p.memory = "100Gi"
if config.Memory != "" {
p.memory = config.Memory
}
// Default to 20 pods
p.pods = "20"
if config.Pods != "" {
p.pods = config.Pods
}
// Default to Linux if the operating system was not defined in the config.
if config.OperatingSystem == "" {
config.OperatingSystem = providers.OperatingSystemLinux
} else {
// Validate operating system from config.
ok, _ := providers.ValidOperatingSystems[config.OperatingSystem]
if !ok {
return fmt.Errorf("%q is not a valid operating system, try one of the following instead: %s", config.OperatingSystem, strings.Join(providers.ValidOperatingSystems.Names(), " | "))
}
}
p.operatingSystem = config.OperatingSystem
return nil
}

71
providers/openstack/types.go
View File

@@ -1,71 +0,0 @@
package openstack
// CapsuleSpec
type CapsuleSpec struct {
Volumes []Volume `json:"volumes,omitempty"`
Containers []Container `json:"containers,omitempty"`
RestartPolicy string `json:"restartPolicy,omitempty"`
}
type CapsuleTemplate struct {
Spec CapsuleSpec `json:"spec,omitempty"`
Kind string `json:"kind,omitempty"`
Metadata Metadata `json:"metadata,omitempty"`
}
type Metadata struct {
Labels map[string]string `json:"labels,omitempty"`
Name string `json:"name,omitempty"`
}
type Volume struct {
Name string `json:"name,omitempty"`
}
type Container struct {
// Name string `json:"name" protobuf:"bytes,1,opt,name=name"`
Image string `json:"image,omitempty" protobuf:"bytes,2,opt,name=image"`
Command []string `json:"command,omitempty" protobuf:"bytes,3,rep,name=command"`
Args []string `json:"args,omitempty" protobuf:"bytes,4,rep,name=args"`
WorkingDir string `json:"workDir,omitempty" protobuf:"bytes,5,opt,name=workingDir"`
// Ports []ContainerPort `json:"ports,omitempty" patchStrategy:"merge" patchMergeKey:"containerPort" protobuf:"bytes,6,rep,name=ports"`
Env map[string]string `json:"env,omitempty"`
//ENV is different with Kubernetes
// Env []EnvVar `json:"env,omitempty" patchStrategy:"merge" patchMergeKey:"name" protobuf:"bytes,6,rep,name=env"`
Resources ResourceRequirements `json:"resources,omitempty" protobuf:"bytes,7,opt,name=resources"`
// VolumeMounts []VolumeMount `json:"volumeMounts,omitempty" patchStrategy:"merge" patchMergeKey:"mountPath" protobuf:"bytes,8,rep,name=volumeMounts"`
ImagePullPolicy string `json:"imagePullPolicy,omitempty" protobuf:"bytes,8,opt,name=imagePullPolicy"`
// Stdin bool `json:"stdin,omitempty" protobuf:"varint,16,opt,name=stdin"`
// StdinOnce bool `json:"stdinOnce,omitempty" protobuf:"varint,17,opt,name=stdinOnce"`
// TTY bool `json:"tty,omitempty" protobuf:"varint,18,opt,name=tty"`
}
//type EnvVar struct {
// Name string `json:"name" protobuf:"bytes,1,opt,name=name"`
// Value string `json:"value,omitempty" protobuf:"bytes,2,opt,name=value"`
//}
// ContainerPort represents a network port in a single container.
//type ContainerPort struct {
// Name string `json:"name,omitempty" protobuf:"bytes,1,opt,name=name"`
// HostPort int32 `json:"hostPort,omitempty" protobuf:"varint,2,opt,name=hostPort"`
// ContainerPort int32 `json:"containerPort" protobuf:"varint,3,opt,name=containerPort"`
// Protocol Protocol `json:"protocol,omitempty" protobuf:"bytes,4,opt,name=protocol,casttype=Protocol"`
// HostIP string `json:"hostIP,omitempty" protobuf:"bytes,5,opt,name=hostIP"`
//}
type ResourceName string
type ResourceList map[ResourceName]float64
// ResourceRequirements describes the compute resource requirements.
type ResourceRequirements struct {
//Zun define the Limit is opposite.
//Limits ResourceList `json:"limits,omitempty" protobuf:"bytes,1,rep,name=limits,casttype=ResourceList,castkey=ResourceName"`
//Requests ResourceList `json:"requests,omitempty" protobuf:"bytes,2,rep,name=requests,casttype=ResourceList,castkey=ResourceName"`
Limits ResourceList `json:"requests,omitempty" protobuf:"bytes,1,rep,name=limits"`
}

519
providers/openstack/zun.go
View File

@@ -1,519 +0,0 @@
package openstack
import (
"context"
"encoding/json"
"fmt"
"io"
"io/ioutil"
"log"
"os"
"strconv"
"strings"
"time"
"github.com/gophercloud/gophercloud"
"github.com/gophercloud/gophercloud/openstack"
"github.com/gophercloud/gophercloud/openstack/container/v1/capsules"
"github.com/gophercloud/gophercloud/pagination"
"github.com/virtual-kubelet/virtual-kubelet/manager"
"github.com/virtual-kubelet/virtual-kubelet/node/api"
"github.com/virtual-kubelet/virtual-kubelet/providers"
v1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/resource"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/types"
)
// ZunProvider implements the virtual-kubelet provider interface and communicates with OpenStack's Zun APIs.
type ZunProvider struct {
ZunClient *gophercloud.ServiceClient
resourceManager *manager.ResourceManager
region string
nodeName string
operatingSystem string
cpu string
memory string
pods string
daemonEndpointPort int32
}
// NewZunProvider creates a new ZunProvider.
func NewZunProvider(config string, rm *manager.ResourceManager, nodeName string, operatingSystem string, daemonEndpointPort int32) (*ZunProvider, error) {
var p ZunProvider
var err error
p.resourceManager = rm
AuthOptions, err := openstack.AuthOptionsFromEnv()
if err != nil {
return nil, fmt.Errorf("Unable to get the Auth options from environment variables: %s", err)
}
Provider, err := openstack.AuthenticatedClient(AuthOptions)
if err != nil {
return nil, fmt.Errorf("Unable to get provider: %s", err)
}
p.ZunClient, err = openstack.NewContainerV1(Provider, gophercloud.EndpointOpts{
Region: os.Getenv("OS_REGION_NAME"),
})
if err != nil {
return nil, fmt.Errorf("Unable to get zun client")
}
p.ZunClient.Microversion = "1.32"
// Set sane defaults for Capacity in case config is not supplied
p.cpu = "20"
p.memory = "100Gi"
p.pods = "20"
p.operatingSystem = operatingSystem
p.nodeName = nodeName
p.daemonEndpointPort = daemonEndpointPort
return &p, err
}
// GetPod returns a pod by name that is running inside Zun
// returns nil if a pod by that name is not found.
func (p *ZunProvider) GetPod(ctx context.Context, namespace, name string) (*v1.Pod, error) {
capsule, err := capsules.Get(p.ZunClient, fmt.Sprintf("%s-%s", namespace, name)).ExtractV132()
if err != nil {
return nil, err
}
if capsule.MetaLabels["NodeName"] != p.nodeName {
return nil, nil
}
return capsuleToPod(capsule)
}
// GetPods returns a list of all pods known to be running within Zun.
func (p *ZunProvider) GetPods(ctx context.Context) ([]*v1.Pod, error) {
pager := capsules.List(p.ZunClient, nil)
pages := 0
err := pager.EachPage(func(page pagination.Page) (bool, error) {
pages++
return true, nil
})
if err != nil {
return nil, err
}
pods := make([]*v1.Pod, 0, pages)
err = pager.EachPage(func(page pagination.Page) (bool, error) {
CapsuleList, err := capsules.ExtractCapsulesV132(page)
if err != nil {
return false, err
}
for _, m := range CapsuleList {
c := m
if m.MetaLabels["NodeName"] != p.nodeName {
continue
}
p, err := capsuleToPod(&c)
if err != nil {
log.Println(err)
continue
}
pods = append(pods, p)
}
return true, nil
})
if err != nil {
return nil, err
}
return pods, nil
}
// CreatePod accepts a Pod definition and creates
// an Zun deployment
func (p *ZunProvider) CreatePod(ctx context.Context, pod *v1.Pod) error {
var capsuleTemplate CapsuleTemplate
capsuleTemplate.Kind = "capsule"
podUID := string(pod.UID)
podCreationTimestamp := pod.CreationTimestamp.String()
var metadata Metadata
metadata.Labels = map[string]string{
"PodName": pod.Name,
"ClusterName": pod.ClusterName,
"NodeName": pod.Spec.NodeName,
"Namespace": pod.Namespace,
"UID": podUID,
"CreationTimestamp": podCreationTimestamp,
}
metadata.Name = pod.Namespace + "-" + pod.Name
capsuleTemplate.Metadata = metadata
// get containers
containers, err := p.getContainers(ctx, pod)
if err != nil {
return err
}
capsuleTemplate.Spec.Containers = containers
data, err := json.MarshalIndent(capsuleTemplate, "", " ")
if err != nil {
return err
}
template := new(capsules.Template)
template.Bin = []byte(data)
createOpts := capsules.CreateOpts{
TemplateOpts: template,
}
_, err = capsules.Create(p.ZunClient, createOpts).ExtractV132()
if err != nil {
return err
}
return err
}
func (p *ZunProvider) getContainers(ctx context.Context, pod *v1.Pod) ([]Container, error) {
containers := make([]Container, 0, len(pod.Spec.Containers))
for _, container := range pod.Spec.Containers {
c := Container{
// Name: container.Name,
Image: container.Image,
Command: append(container.Command, container.Args...),
WorkingDir: container.WorkingDir,
ImagePullPolicy: string(container.ImagePullPolicy),
}
// Container ENV need to sync with K8s in Zun and gophercloud. Will change them.
// From map[string]string to []map[string]string
c.Env = map[string]string{}
for _, e := range container.Env {
c.Env[e.Name] = e.Value
}
if container.Resources.Limits != nil {
cpuLimit := float64(1)
if _, ok := container.Resources.Limits[v1.ResourceCPU]; ok {
cpuLimit = float64(container.Resources.Limits.Cpu().MilliValue()) / 1000.00
}
memoryLimit := 0.5
if _, ok := container.Resources.Limits[v1.ResourceMemory]; ok {
memoryLimit = float64(container.Resources.Limits.Memory().Value()) / 1000000000.00
}
c.Resources.Limits["cpu"] = cpuLimit
c.Resources.Limits["memory"] = memoryLimit * 1024
}
//TODO: Add Sync with Resource requirement
//TODO: Add port Sync
//TODO: Add volume support
containers = append(containers, c)
}
return containers, nil
}
// RunInContainer executes a command in a container in the pod, copying data
// between in/out/err and the container's stdin/stdout/stderr.
func (p *ZunProvider) RunInContainer(ctx context.Context, namespace, name, container string, cmd []string, attach api.AttachIO) error {
log.Printf("receive ExecInContainer %q\n", container)
return nil
}
// GetPodStatus returns the status of a pod by name that is running inside Zun
// returns nil if a pod by that name is not found.
func (p *ZunProvider) GetPodStatus(ctx context.Context, namespace, name string) (*v1.PodStatus, error) {
pod, err := p.GetPod(ctx, namespace, name)
if err != nil {
return nil, err
}
if pod == nil {
return nil, nil
}
return &pod.Status, nil
}
func (p *ZunProvider) GetContainerLogs(ctx context.Context, namespace, podName, containerName string, opts api.ContainerLogOpts) (io.ReadCloser, error) {
return ioutil.NopCloser(strings.NewReader("not support in Zun Provider")), nil
}
// NodeConditions returns a list of conditions (Ready, OutOfDisk, etc), for updates to the node status
// within Kubernetes.
func (p *ZunProvider) NodeConditions(ctx context.Context) []v1.NodeCondition {
// TODO: Make these dynamic and augment with custom Zun specific conditions of interest
return []v1.NodeCondition{
{
Type: "Ready",
Status: v1.ConditionTrue,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletReady",
Message: "kubelet is ready.",
},
{
Type: "OutOfDisk",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletHasSufficientDisk",
Message: "kubelet has sufficient disk space available",
},
{
Type: "MemoryPressure",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletHasSufficientMemory",
Message: "kubelet has sufficient memory available",
},
{
Type: "DiskPressure",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "KubeletHasNoDiskPressure",
Message: "kubelet has no disk pressure",
},
{
Type: "NetworkUnavailable",
Status: v1.ConditionFalse,
LastHeartbeatTime: metav1.Now(),
LastTransitionTime: metav1.Now(),
Reason: "RouteCreated",
Message: "RouteController created a route",
},
}
}
// NodeAddresses returns a list of addresses for the node status
// within Kubernetes.
func (p *ZunProvider) NodeAddresses(ctx context.Context) []v1.NodeAddress {
return nil
}
// NodeDaemonEndpoints returns NodeDaemonEndpoints for the node status
// within Kubernetes.
func (p *ZunProvider) NodeDaemonEndpoints(ctx context.Context) *v1.NodeDaemonEndpoints {
return &v1.NodeDaemonEndpoints{
KubeletEndpoint: v1.DaemonEndpoint{
Port: p.daemonEndpointPort,
},
}
}
// OperatingSystem returns the operating system for this provider.
// This is a noop to default to Linux for now.
func (p *ZunProvider) OperatingSystem() string {
return providers.OperatingSystemLinux
}
func capsuleToPod(capsule *capsules.CapsuleV132) (*v1.Pod, error) {
var podCreationTimestamp metav1.Time
var containerStartTime metav1.Time
podCreationTimestamp = metav1.NewTime(capsule.CreatedAt)
if len(capsule.Containers) > 0 {
containerStartTime = metav1.NewTime(capsule.Containers[0].StartedAt)
}
containerStartTime = metav1.NewTime(time.Time{})
// Deal with container inside capsule
containers := make([]v1.Container, 0, len(capsule.Containers))
containerStatuses := make([]v1.ContainerStatus, 0, len(capsule.Containers))
for _, c := range capsule.Containers {
containerMemoryMB := 0
if c.Memory != "" {
containerMemory, err := strconv.Atoi(c.Memory)
if err != nil {
log.Println(err)
}
containerMemoryMB = containerMemory
}
container := v1.Container{
Name: c.Name,
Image: c.Image,
Command: c.Command,
Resources: v1.ResourceRequirements{
Limits: v1.ResourceList{
v1.ResourceCPU: resource.MustParse(fmt.Sprintf("%g", float64(c.CPU))),
v1.ResourceMemory: resource.MustParse(fmt.Sprintf("%dM", containerMemoryMB)),
},
Requests: v1.ResourceList{
v1.ResourceCPU: resource.MustParse(fmt.Sprintf("%g", float64(c.CPU*1024/100))),
v1.ResourceMemory: resource.MustParse(fmt.Sprintf("%dM", containerMemoryMB)),
},
},
}
containers = append(containers, container)
containerStatus := v1.ContainerStatus{
Name: c.Name,
State: zunContainerStausToContainerStatus(&c),
LastTerminationState: zunContainerStausToContainerStatus(&c),
Ready: zunStatusToPodPhase(c.Status) == v1.PodRunning,
RestartCount: int32(0),
Image: c.Image,
ImageID: "",
ContainerID: c.UUID,
}
// Add to containerStatuses
containerStatuses = append(containerStatuses, containerStatus)
}
ip := ""
if capsule.Addresses != nil {
for _, v := range capsule.Addresses {
for _, addr := range v {
if addr.Version == float64(4) {
ip = addr.Addr
}
}
}
}
p := v1.Pod{
TypeMeta: metav1.TypeMeta{
Kind: "Pod",
APIVersion: "v1",
},
ObjectMeta: metav1.ObjectMeta{
Name: capsule.MetaLabels["PodName"],
Namespace: capsule.MetaLabels["Namespace"],
ClusterName: capsule.MetaLabels["ClusterName"],
UID: types.UID(capsule.UUID),
CreationTimestamp: podCreationTimestamp,
},
Spec: v1.PodSpec{
NodeName: capsule.MetaLabels["NodeName"],
Volumes: []v1.Volume{},
Containers: containers,
},
Status: v1.PodStatus{
Phase: zunStatusToPodPhase(capsule.Status),
Conditions: []v1.PodCondition{},
Message: "",
Reason: "",
HostIP: "",
PodIP: ip,
StartTime: &containerStartTime,
ContainerStatuses: containerStatuses,
},
}
return &p, nil
}
// UpdatePod is a noop, Zun currently does not support live updates of a pod.
func (p *ZunProvider) UpdatePod(ctx context.Context, pod *v1.Pod) error {
return nil
}
// DeletePod deletes the specified pod out of Zun.
func (p *ZunProvider) DeletePod(ctx context.Context, pod *v1.Pod) error {
err := capsules.Delete(p.ZunClient, fmt.Sprintf("%s-%s", pod.Namespace, pod.Name)).ExtractErr()
if err != nil {
return err
}
// wait for the capsule deletion
for i := 0; i < 300; i++ {
time.Sleep(1 * time.Second)
capsule, err := capsules.Get(p.ZunClient, fmt.Sprintf("%s-%s", pod.Namespace, pod.Name)).ExtractV132()
if _, ok := err.(gophercloud.ErrDefault404); ok {
// deletion complete
return nil
}
if err != nil {
return err
}
if capsule.Status == "Error" {
return fmt.Errorf("Capsule in ERROR state")
}
}
return fmt.Errorf("Timed out on waiting capsule deletion")
}
func zunContainerStausToContainerStatus(cs *capsules.Container) v1.ContainerState {
// Zun already container start time but not add support at gophercloud
//startTime := metav1.NewTime(time.Time(cs.StartTime))
// Zun container status:
//'Error', 'Running', 'Stopped', 'Paused', 'Unknown', 'Creating', 'Created',
//'Deleted', 'Deleting', 'Rebuilding', 'Dead', 'Restarting'
// Handle the case where the container is running.
if cs.Status == "Running" || cs.Status == "Stopped" {
return v1.ContainerState{
Running: &v1.ContainerStateRunning{
StartedAt: metav1.NewTime(time.Time(cs.StartedAt)),
},
}
}
// Handle the case where the container failed.
if cs.Status == "Error" || cs.Status == "Dead" {
return v1.ContainerState{
Terminated: &v1.ContainerStateTerminated{
ExitCode: int32(0),
Reason: cs.Status,
Message: cs.StatusDetail,
StartedAt: metav1.NewTime(time.Time(cs.StartedAt)),
FinishedAt: metav1.NewTime(time.Time(cs.UpdatedAt)),
},
}
}
// Handle the case where the container is pending.
// Which should be all other Zun states.
return v1.ContainerState{
Waiting: &v1.ContainerStateWaiting{
Reason: cs.Status,
Message: cs.StatusDetail,
},
}
}
func zunStatusToPodPhase(status string) v1.PodPhase {
switch status {
case "Running":
return v1.PodRunning
case "Stopped":
return v1.PodSucceeded
case "Error":
return v1.PodFailed
case "Dead":
return v1.PodFailed
case "Creating":
return v1.PodPending
case "Created":
return v1.PodPending
case "Restarting":
return v1.PodPending
case "Rebuilding":
return v1.PodPending
case "Paused":
return v1.PodPending
case "Deleting":
return v1.PodPending
case "Deleted":
return v1.PodPending
}
return v1.PodUnknown
}
// Capacity returns a resource list containing the capacity limits set for Zun.
func (p *ZunProvider) Capacity(ctx context.Context) v1.ResourceList {
return v1.ResourceList{
"cpu": resource.MustParse(p.cpu),
"memory": resource.MustParse(p.memory),
"pods": resource.MustParse(p.pods),
}
}

23
providers/register/provider_alibabacloud.go
View File

@@ -1,23 +0,0 @@
// +build alibabacloud_provider
package register
import (
"github.com/virtual-kubelet/virtual-kubelet/providers"
"github.com/virtual-kubelet/virtual-kubelet/providers/alibabacloud"
)
func init() {
register("alibabacloud", aliCloudInit)
}
func aliCloudInit(cfg InitConfig) (providers.Provider, error) {
return alibabacloud.NewECIProvider(
cfg.ConfigPath,
cfg.ResourceManager,
cfg.NodeName,
cfg.OperatingSystem,
cfg.InternalIP,
cfg.DaemonPort,
)
}

16
providers/register/provider_aws.go
View File

@@ -1,16 +0,0 @@
// +build aws_provider
package register
import (
"github.com/virtual-kubelet/virtual-kubelet/providers"
"github.com/virtual-kubelet/virtual-kubelet/providers/aws"
)
func init() {
register("aws", initAWS)
}
func initAWS(cfg InitConfig) (providers.Provider, error) {
return aws.NewFargateProvider(cfg.ConfigPath, cfg.ResourceManager, cfg.NodeName, cfg.OperatingSystem, cfg.InternalIP, cfg.DaemonPort)
}

23
providers/register/provider_azurebatch.go
View File

@@ -1,23 +0,0 @@
// +build azurebatch_provider
package register
import (
"github.com/virtual-kubelet/virtual-kubelet/providers"
"github.com/virtual-kubelet/virtual-kubelet/providers/azurebatch"
)
func init() {
register("azurebatch", initAzureBatch)
}
func initAzureBatch(cfg InitConfig) (providers.Provider, error) {
return azurebatch.NewBatchProvider(
cfg.ConfigPath,
cfg.ResourceManager,
cfg.NodeName,
cfg.OperatingSystem,
cfg.InternalIP,
cfg.DaemonPort,
)
}

22
providers/register/provider_cri.go
View File

@@ -1,22 +0,0 @@
// +build linux,cri_provider
package register
import (
"github.com/virtual-kubelet/virtual-kubelet/providers"
"github.com/virtual-kubelet/virtual-kubelet/providers/cri"
)
func init() {
register("cri", criInit)
}
func criInit(cfg InitConfig) (providers.Provider, error) {
return cri.NewCRIProvider(
cfg.NodeName,
cfg.OperatingSystem,
cfg.InternalIP,
cfg.ResourceManager,
cfg.DaemonPort,
)
}

23
providers/register/provider_huawei.go
View File

@@ -1,23 +0,0 @@
// +build huawei_provider
package register
import (
"github.com/virtual-kubelet/virtual-kubelet/providers"
"github.com/virtual-kubelet/virtual-kubelet/providers/huawei"
)
func init() {
register("huawei", initHuawei)
}
func initHuawei(cfg InitConfig) (providers.Provider, error) {
return huawei.NewCCIProvider(
cfg.ConfigPath,
cfg.ResourceManager,
cfg.NodeName,
cfg.OperatingSystem,
cfg.InternalIP,
cfg.DaemonPort,
)
}

34
providers/register/provider_mock.go
View File

@@ -1,34 +0,0 @@
// +build mock_provider
package register
import (
"github.com/virtual-kubelet/virtual-kubelet/providers"
"github.com/virtual-kubelet/virtual-kubelet/providers/mock"
)
func init() {
register("mock", initMock)
register("mocklegacy", initMockLegacy)
}
func initMock(cfg InitConfig) (providers.Provider, error) {
return mock.NewMockProvider(
cfg.ConfigPath,
cfg.NodeName,
cfg.OperatingSystem,
cfg.InternalIP,
cfg.DaemonPort,
)
}
func initMockLegacy(cfg InitConfig) (providers.Provider, error) {
return mock.NewMockLegacyProvider(
cfg.ConfigPath,
cfg.NodeName,
cfg.OperatingSystem,
cfg.InternalIP,
cfg.DaemonPort,
)
}

16
providers/register/provider_nomad.go
View File

@@ -1,16 +0,0 @@
// +build nomad_provider
package register
import (
"github.com/virtual-kubelet/virtual-kubelet/providers"
"github.com/virtual-kubelet/virtual-kubelet/providers/nomad"
)
func init() {
register("nomad", initNomad)
}
func initNomad(cfg InitConfig) (providers.Provider, error) {
return nomad.NewProvider(cfg.ResourceManager, cfg.NodeName, cfg.OperatingSystem)
}

21
providers/register/provider_openstack.go
View File

@@ -1,21 +0,0 @@
// +build openstack_provider
package register
import (
"github.com/virtual-kubelet/virtual-kubelet/providers"
"github.com/virtual-kubelet/virtual-kubelet/providers/openstack"
)
func init() {
register("openstack", initOpenStack)
}
func initOpenStack(cfg InitConfig) (providers.Provider, error) {
return openstack.NewZunProvider(
cfg.ConfigPath,
cfg.ResourceManager,
cfg.NodeName,
cfg.OperatingSystem,
cfg.DaemonPort)
}

16
providers/register/provider_web.go
View File

@@ -1,16 +0,0 @@
// +build web_provider
package register
import (
"github.com/virtual-kubelet/virtual-kubelet/providers"
"github.com/virtual-kubelet/virtual-kubelet/providers/web"
)
func init() {
register("web", initWeb)
}
func initWeb(cfg InitConfig) (providers.Provider, error) {
return web.NewBrokerProvider(cfg.NodeName, cfg.OperatingSystem, cfg.DaemonPort)
}

52
providers/register/register.go
View File

@@ -1,52 +0,0 @@
package register
import (
"sort"
"github.com/virtual-kubelet/virtual-kubelet/errdefs"
"github.com/virtual-kubelet/virtual-kubelet/manager"
"github.com/virtual-kubelet/virtual-kubelet/providers"
)
var providerInits = make(map[string]initFunc)
// InitConfig is the config passed to initialize a registered provider.
type InitConfig struct {
ConfigPath string
NodeName string
OperatingSystem string
InternalIP string
DaemonPort int32
ResourceManager *manager.ResourceManager
}
type initFunc func(InitConfig) (providers.Provider, error)
// GetProvider gets the provider specified by the given name
func GetProvider(name string, cfg InitConfig) (providers.Provider, error) {
f, ok := providerInits[name]
if !ok {
return nil, errdefs.NotFoundf("provider not found: %s", name)
}
return f(cfg)
}
// Exists checks if a provider is regstered
func Exists(name string) bool {
_, ok := providerInits[name]
return ok
}
// List gets the list of all provider names
func List() []string {
ls := make([]string, 0, len(providerInits))
for name := range providerInits {
ls = append(ls, name)
}
sort.Strings(ls)
return ls
}
func register(name string, f initFunc) {
providerInits[name] = f
}

73
providers/store.go Normal file
View File

@@ -0,0 +1,73 @@
package providers
import (
"sync"
"github.com/virtual-kubelet/virtual-kubelet/errdefs"
"github.com/virtual-kubelet/virtual-kubelet/manager"
)
// Store is used for registering/fetching providers
type Store struct {
mu sync.Mutex
ls map[string]InitFunc
}
func NewStore() *Store {
return &Store{
ls: make(map[string]InitFunc),
}
}
// Register registers a providers init func by name
func (s *Store) Register(name string, f InitFunc) error {
if f == nil {
return errdefs.InvalidInput("provided init function cannot not be nil")
}
s.mu.Lock()
s.ls[name] = f
s.mu.Unlock()
return nil
}
// Get gets the registered init func for the given name
// The returned function may be nil if the given name is not registered.
func (s *Store) Get(name string) InitFunc {
s.mu.Lock()
f := s.ls[name]
s.mu.Unlock()
return f
}
// List lists all the registered providers
func (s *Store) List() []string {
s.mu.Lock()
defer s.mu.Unlock()
ls := make([]string, 0, len(s.ls))
for p := range s.ls {
ls = append(ls, p)
}
return ls
}
// Exists returns if there is an init function registered under the provided name
func (s *Store) Exists(name string) bool {
s.mu.Lock()
_, ok := s.ls[name]
s.mu.Unlock()
return ok
}
// InitConfig is the config passed to initialize a registered provider.
type InitConfig struct {
ConfigPath string
NodeName string
OperatingSystem string
InternalIP string
DaemonPort int32
ResourceManager *manager.ResourceManager
}
type InitFunc func(InitConfig) (Provider, error)

157
providers/web/README.md
View File

@@ -1,157 +0,0 @@
Web provider for Virtual Kubelet
================================
Virtual Kubelet providers are written using the Go programming language. While
Go is a great general purpose programming language, it is however a fact that
other programming languages exist. The problem that Virtual Kubelet solves is as
applicable to applications written in those languages as it is for those written
using Go. This provider aims to serve as a bridge between technology stacks and
programming languages, as it were, by adapting the Virtual Kubelet provider
interface using a web endpoint, i.e., this provider is a thin layer that
forwards all calls that Kubernetes makes to the virtual kubelet to a
pre-configured HTTP endpoint. This frees the provider's implementor to write
their code in any programming language and technology stack as they see fit.
The `providers/web/web-rust` folder contains a sample provider implemented in
the Rust programming language. Here's a diagram that depicts the interaction
between Kubernetes, the virtual kubelet web provider and the Rust app.
+----------------+ +---------------------------+ +------------------------------+
| | | | HTTP | |
| Kubernetes | <-----> | Virtual Kubelet: Web | <------> | Provider written in Rust |
| | | | | |
+----------------+ +---------------------------+ +------------------------------+
Provider interface
------------------
The web provider uses an environment variable to determine the endpoint to use
for forwarding requests. The environment variable must be named
`WEB_ENDPOINT_URL` and must implement the following HTTP API:
| Path | Verb | Query | Request | Response | Description |
|-------------------|--------|-----------------------------------------|----------|---------------------------------------------------|---------------------------------------------------------------------------|
| /createPod | POST | - | Pod JSON | HTTP status code | Create a new pod |
| /updatePod | PUT | - | Pod JSON | HTTP status code | Update pod spec |
| /deletePod | DELETE | - | Pod JSON | HTTP status code | Delete an existing pod |
| /getPod | GET | namespace, name | - | Pod JSON | Given a pod namespace and name, return the pod JSON |
| /getContainerLogs | GET | namespace, podName, containerName, tail | - | Container logs | Given the namespace, pod name and container name, return `tail` log lines |
| /getPodStatus | GET | namespace, name | - | Pod status JSON | Given a pod namespace and name, return the pod's status JSON |
| /getPods | GET | - | - | Array of pod JSON strings | Fetch list of created pods |
| /capacity | GET | - | - | JSON map containing resource name and values | Fetch resource capacity values |
| /nodeConditions | GET | - | - | Array of node condition JSON strings | Get list of node conditions (Ready, OutOfDisk etc) |
| /nodeAddresses | GET | - | - | Array of node address values (type/address pairs) | Fetch a list of addresses for the node status |
A typical deployment configuration for this setup would be to have the provider
implementation be deployed as a container in the same pod as the virtual kubelet
itself (as a "sidecar").
Take her for a spin
-------------------
A sample web provider implementation is included in this repo in order to
showcase what this enables. The sample has been implemented in
[Rust](http://rust-lang.org). The easiest way to get this up and running is
to use the Helm chart available at `providers/web/charts/web-rust`. Open a
terminal and install the chart like so:
$ cd providers/web/charts
$ helm install -n web-provider ./web-rust
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
web-provider-virtual-kubelet-web-6b5b7446f6-279xl 2/2 Running 0 3h
If you list the nodes in the cluster after this you should see something that
looks like this:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
aks-nodepool1-35187879-0 Ready agent 37d v1.8.2
aks-nodepool1-35187879-1 Ready agent 37d v1.8.2
aks-nodepool1-35187879-3 Ready agent 37d v1.8.2
virtual-kubelet-web Ready agent 3h v1.8.3
In case the name of the node didn't give it away, the last entry in the output
above is the virtual kubelet. If you try to list the containers in the pod that
represents the virtual kubelet you should be able to see the sidecar Rust
container:
$ kubectl get pods -o=custom-columns=NAME:.metadata.name,CONTAINERS:.spec.containers[*].name
NAME CONTAINERS
web-provider-virtual-kubelet-web-6b5b7446f6-279xl webrust,virtualkubelet
In the output above, `webrust` is the sidecar container and `virtualkubelet` is
the broker that forwards requests to `webrust`. You can run a query on the
`/getPods` HTTP endpoint on the `webrust` container to see a list of the pods
that it has been asked to create. To do this we first use `kubectl` to setup a
port forwarding server like so:
$ kubectl port-forward web-provider-virtual-kubelet-web-6b5b7446f6-279xl 3000:3000
Now if we run `curl` on the `http://localhost:3000/getPods` URL you should see
the pod JSON getting dumped to the terminal. I ran my test on a Kubernetes
cluster deployed on [Azure](https://docs.microsoft.com/en-us/azure/aks/) which
happens to deploy a daemonset with some, what I imagine are "system" pods to
every node in the cluster. You can filter the output to see just the pod names
using the [jq](https://stedolan.github.io/jq/) tool like so:
$ curl -s http://localhost:3000/getPods | jq '.[] | { name: .metadata.name }'
{
"name": "kube-proxy-czz57"
}
{
"name": "kube-svc-redirect-7qlpd"
}
You can deploy workloads to the virtual kubelet as you normally do. Here's a
sample pod spec that uses `nodeSelector` to cause the deployment to be scheduled
on the virtual kubelet.
apiVersion: v1
kind: Pod
metadata:
name: vk-pod
labels:
foo: bar
spec:
containers:
- name: web1
image: nginx
nodeSelector:
type: virtual-kubelet
Let's go ahead and deploy the pod and run our `/getPods` query again:
$ kubectl apply -f ~/tmp/pod1.yaml
pod "vk-pod" created
$ curl -s http://localhost:3000/getPods | jq '.[] | { name: .metadata.name }'
{
"name": "kube-proxy-czz57"
}
{
"name": "kube-svc-redirect-7qlpd"
}
{
"name": "vk-pod"
}
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
vk-pod 0/1 Running 0 1m
web-provider-virtual-kubelet-web-6b5b7446f6-279xl 2/2 Running 6 4h
As you can tell, a new pod has been scheduled to run on our virtual kubelet
instance. Deleting pods works as one would expect:
$ kubectl delete -f ~/tmp/pod1.yaml
pod "vk-pod" deleted
$ curl -s http://localhost:3000/getPods | jq '.[] | { name: .metadata.name }'
{
"name": "kube-proxy-czz57"
}
{
"name": "kube-svc-redirect-7qlpd"
}

314
providers/web/broker.go
View File

@@ -1,314 +0,0 @@
// Package web provides an implementation of the virtual kubelet provider interface
// by forwarding all calls to a web endpoint. The web endpoint to which requests
// must be forwarded must be specified through an environment variable called
// `WEB_ENDPOINT_URL`. This endpoint must implement the following HTTP APIs:
// - POST /createPod
// - PUT /updatePod
// - DELETE /deletePod
// - GET /getPod?namespace=[namespace]&name=[pod name]
// - GE /getContainerLogs?namespace=[namespace]&podName=[pod name]&containerName=[container name]&tail=[tail value]
// - GET /getPodStatus?namespace=[namespace]&name=[pod name]
// - GET /getPods
// - GET /capacity
// - GET /nodeConditions
// - GET /nodeAddresses
package web
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
"net/url"
"os"
"time"
"github.com/cenkalti/backoff"
"github.com/virtual-kubelet/virtual-kubelet/errdefs"
"github.com/virtual-kubelet/virtual-kubelet/node/api"
v1 "k8s.io/api/core/v1"
)
// BrokerProvider implements the virtual-kubelet provider interface by forwarding kubelet calls to a web endpoint.
type BrokerProvider struct {
nodeName string
operatingSystem string
endpoint *url.URL
client *http.Client
daemonEndpointPort int32
}
// NewBrokerProvider creates a new BrokerProvider
func NewBrokerProvider(nodeName, operatingSystem string, daemonEndpointPort int32) (*BrokerProvider, error) {
var provider BrokerProvider
provider.nodeName = nodeName
provider.operatingSystem = operatingSystem
provider.client = &http.Client{}
provider.daemonEndpointPort = daemonEndpointPort
if ep := os.Getenv("WEB_ENDPOINT_URL"); ep != "" {
epurl, err := url.Parse(ep)
if err != nil {
return nil, err
}
provider.endpoint = epurl
}
return &provider, nil
}
// CreatePod accepts a Pod definition and forwards the call to the web endpoint
func (p *BrokerProvider) CreatePod(ctx context.Context, pod *v1.Pod) error {
return p.createUpdatePod(pod, "POST", "/createPod")
}
// UpdatePod accepts a Pod definition and forwards the call to the web endpoint
func (p *BrokerProvider) UpdatePod(ctx context.Context, pod *v1.Pod) error {
return p.createUpdatePod(pod, "PUT", "/updatePod")
}
// DeletePod accepts a Pod definition and forwards the call to the web endpoint
func (p *BrokerProvider) DeletePod(ctx context.Context, pod *v1.Pod) error {
urlPath, err := url.Parse("/deletePod")
if err != nil {
return err
}
// encode pod definition as JSON and post request
podJSON, err := json.Marshal(pod)
if err != nil {
return err
}
return checkResponseStatus(p.doRequest("DELETE", urlPath, podJSON))
}
// GetPod returns a pod by name that is being managed by the web server
func (p *BrokerProvider) GetPod(ctx context.Context, namespace, name string) (*v1.Pod, error) {
urlPathStr := fmt.Sprintf(
"/getPod?namespace=%s&name=%s",
url.QueryEscape(namespace),
url.QueryEscape(name))
var pod v1.Pod
err := p.doGetRequest(urlPathStr, &pod)
// if we get a "404 Not Found" then we return nil to indicate that no pod
// with this name was found
if err != nil && err.Error() == "404 Not Found" {
return nil, nil
}
return &pod, err
}
// GetContainerLogs returns the logs of a container running in a pod by name.
func (p *BrokerProvider) GetContainerLogs(ctx context.Context, namespace, podName, containerName string, opts api.ContainerLogOpts) (io.ReadCloser, error) {
urlPathStr := fmt.Sprintf(
"/getContainerLogs?namespace=%s&podName=%s&containerName=%s&tail=%d",
url.QueryEscape(namespace),
url.QueryEscape(podName),
url.QueryEscape(containerName),
opts.Tail)
r, err := p.doGetRequestRaw(urlPathStr)
if err := checkResponseStatus(r, err); err != nil {
return nil, err
}
return r.Body, nil
}
// Get full pod name as defined in the provider context
// TODO: Implementation
func (p *BrokerProvider) GetPodFullName(namespace string, pod string) string {
return ""
}
// RunInContainer executes a command in a container in the pod, copying data
// between in/out/err and the container's stdin/stdout/stderr.
// TODO: Implementation
func (p *BrokerProvider) RunInContainer(ctx context.Context, namespace, name, container string, cmd []string, attach api.AttachIO) error {
log.Printf("receive ExecInContainer %q\n", container)
return nil
}
// GetPodStatus retrieves the status of a given pod by name.
func (p *BrokerProvider) GetPodStatus(ctx context.Context, namespace, name string) (*v1.PodStatus, error) {
urlPathStr := fmt.Sprintf(
"/getPodStatus?namespace=%s&name=%s",
url.QueryEscape(namespace),
url.QueryEscape(name))
var podStatus v1.PodStatus
err := p.doGetRequest(urlPathStr, &podStatus)
// if we get a "404 Not Found" then we return nil to indicate that no pod
// with this name was found
if err != nil && err.Error() == "404 Not Found" {
return nil, nil
}
return &podStatus, err
}
// GetPods retrieves a list of all pods scheduled to run.
func (p *BrokerProvider) GetPods(ctx context.Context) ([]*v1.Pod, error) {
var pods []*v1.Pod
err := p.doGetRequest("/getPods", &pods)
return pods, err
}
// Capacity returns a resource list containing the capacity limits
func (p *BrokerProvider) Capacity(ctx context.Context) v1.ResourceList {
var resourceList v1.ResourceList
err := p.doGetRequest("/capacity", &resourceList)
// TODO: This API should support reporting an error.
if err != nil {
panic(err)
}
return resourceList
}
// NodeConditions returns a list of conditions (Ready, OutOfDisk, etc), for updates to the node status
func (p *BrokerProvider) NodeConditions(ctx context.Context) []v1.NodeCondition {
var nodeConditions []v1.NodeCondition
err := p.doGetRequest("/nodeConditions", &nodeConditions)
// TODO: This API should support reporting an error.
if err != nil {
panic(err)
}
return nodeConditions
}
// NodeAddresses returns a list of addresses for the node status
// within Kubernetes.
func (p *BrokerProvider) NodeAddresses(ctx context.Context) []v1.NodeAddress {
var nodeAddresses []v1.NodeAddress
err := p.doGetRequest("/nodeAddresses", &nodeAddresses)
// TODO: This API should support reporting an error.
if err != nil {
panic(err)
}
return nodeAddresses
}
// NodeDaemonEndpoints returns NodeDaemonEndpoints for the node status
// within Kubernetes.
func (p *BrokerProvider) NodeDaemonEndpoints(ctx context.Context) *v1.NodeDaemonEndpoints {
return &v1.NodeDaemonEndpoints{
KubeletEndpoint: v1.DaemonEndpoint{
Port: p.daemonEndpointPort,
},
}
}
// OperatingSystem returns the operating system for this provider.
func (p *BrokerProvider) OperatingSystem() string {
return p.operatingSystem
}
func (p *BrokerProvider) doGetRequest(urlPathStr string, v interface{}) error {
response, err := p.doGetRequestBytes(urlPathStr)
if err != nil {
return err
}
return json.Unmarshal(response, &v)
}
func (p *BrokerProvider) doGetRequestRaw(urlPathStr string) (*http.Response, error) {
urlPath, err := url.Parse(urlPathStr)
if err != nil {
return nil, err
}
return p.doRequest("GET", urlPath, nil)
}
func (p *BrokerProvider) doGetRequestBytes(urlPathStr string) ([]byte, error) {
return readResponse(p.doGetRequestRaw(urlPathStr))
}
func (p *BrokerProvider) createUpdatePod(pod *v1.Pod, method, postPath string) error {
// build the post url
postPathURL, err := url.Parse(postPath)
if err != nil {
return err
}
// encode pod definition as JSON and post request
podJSON, err := json.Marshal(pod)
if err != nil {
return err
}
return checkResponseStatus(p.doRequest(method, postPathURL, podJSON))
}
func (p *BrokerProvider) doRequest(method string, urlPath *url.URL, body []byte) (*http.Response, error) {
// build full URL
requestURL := p.endpoint.ResolveReference(urlPath)
// build the request
var bodyReader io.Reader
if body != nil {
bodyReader = bytes.NewReader(body)
}
request, err := http.NewRequest(method, requestURL.String(), bodyReader)
request.Header.Add("Content-Type", "application/json")
// issue request
retry := backoff.NewExponentialBackOff()
retry.MaxElapsedTime = 5 * time.Minute
var response *http.Response
err = backoff.Retry(func() error {
response, err = p.client.Do(request)
return err
}, retry)
if err != nil {
return nil, err
}
return response, nil
}
func checkResponseStatus(r *http.Response, err error) error {
if err != nil {
return err
}
if r.StatusCode < 200 || r.StatusCode > 299 {
switch r.StatusCode {
case http.StatusNotFound:
return errdefs.NotFound(r.Status)
default:
return errors.New(r.Status)
}
}
return nil
}
func readResponse(r *http.Response, err error) ([]byte, error) {
if r.Body != nil {
defer r.Body.Close()
}
if err := checkResponseStatus(r, err); err != nil {
return nil, err
}
lr := io.LimitReader(r.Body, 1e6)
return ioutil.ReadAll(lr)
}

21
providers/web/charts/web-rust/.helmignore
View File

@@ -1,21 +0,0 @@
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*~
# Various IDEs
.project
.idea/
*.tmproj

4
providers/web/charts/web-rust/Chart.yaml
View File

@@ -1,4 +0,0 @@
name: virtual-kubelet-web
version: 0.1.0
description: a Helm chart to install virtual kubelet in Kubernetes setup with a web provider

5
providers/web/charts/web-rust/templates/NOTES.txt
View File

@@ -1,5 +0,0 @@
The virtual kubelet is getting deployed on your cluster.
To verify that virtual kubelet has started, run:
kubectl --namespace={{ .Release.Namespace }} get pods -l "app={{ template "virtual-kubelet-web.fullname" . }}"

16
providers/web/charts/web-rust/templates/_helpers.tpl
View File

@@ -1,16 +0,0 @@
{{/* vim: set filetype=mustache: */}}
{{/*
Expand the name of the chart.
*/}}
{{- define "virtual-kubelet-web.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "virtual-kubelet-web.fullname" -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}

42
providers/web/charts/web-rust/templates/deployment.yaml
View File

@@ -1,42 +0,0 @@
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: {{ template "virtual-kubelet-web.fullname" . }}
labels:
app: {{ template "virtual-kubelet-web.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
replicas: 1
selector:
matchLabels:
app: {{ template "virtual-kubelet-web.name" . }}
template:
metadata:
labels:
app: {{ template "virtual-kubelet-web.name" . }}
release: {{ .Release.Name }}
spec:
containers:
- name: webrust
image: "{{ .Values.rustwebimage.repository }}:{{ .Values.rustwebimage.tag }}"
imagePullPolicy: {{ .Values.rustwebimage.pullPolicy }}
ports:
- containerPort: {{ .Values.rustwebimage.port }}
livenessProbe:
httpGet:
path: /
port: {{ .Values.rustwebimage.port }}
readinessProbe:
httpGet:
path: /
port: {{ .Values.rustwebimage.port }}
- name: virtualkubelet
image: "{{ .Values.vkimage.repository }}:{{ .Values.vkimage.tag }}"
imagePullPolicy: {{ .Values.vkimage.pullPolicy }}
env:
- name: WEB_ENDPOINT_URL
value: http://localhost:{{ .Values.rustwebimage.port }}
command: ["virtual-kubelet"]
args: ["--provider", "web", "--nodename", {{ default "web-provider" .Values.env.nodeName | quote }}]

11
providers/web/charts/web-rust/values.yaml
View File

@@ -1,11 +0,0 @@
rustwebimage:
repository: avranju/web-rust
tag: latest
pullPolicy: Always
port: 3000
vkimage:
repository: avranju/virtual-kubelet
tag: latest
pullPolicy: Always
env:
nodeName: virtual-kubelet-web

4
providers/web/web-rust/.gitignore vendored
View File

@@ -1,4 +0,0 @@
/target/
**/*.rs.bk
.idea/
.vscode/

988
providers/web/web-rust/Cargo.lock generated
View File

@@ -1,988 +0,0 @@
[[package]]
name = "aho-corasick"
version = "0.6.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"memchr 2.0.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "base64"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"byteorder 1.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
"safemem 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "base64"
version = "0.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"byteorder 1.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
"safemem 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "base64"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"byteorder 1.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
"safemem 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "bitflags"
version = "0.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "bitflags"
version = "1.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "bodyparser"
version = "0.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"iron 0.6.0 (registry+https://github.com/rust-lang/crates.io-index)",
"persistent 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)",
"plugin 0.2.6 (registry+https://github.com/rust-lang/crates.io-index)",
"serde 1.0.27 (registry+https://github.com/rust-lang/crates.io-index)",
"serde_json 1.0.9 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "byteorder"
version = "1.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "bytes"
version = "0.4.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"byteorder 1.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
"iovec 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "cfg-if"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "dtoa"
version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "env_logger"
version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"log 0.3.9 (registry+https://github.com/rust-lang/crates.io-index)",
"regex 0.2.5 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "fuchsia-zircon"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"fuchsia-zircon-sys 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "fuchsia-zircon"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"bitflags 1.0.1 (registry+https://github.com/rust-lang/crates.io-index)",
"fuchsia-zircon-sys 0.3.2 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "fuchsia-zircon-sys"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"bitflags 0.7.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "fuchsia-zircon-sys"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "futures"
version = "0.1.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "futures-cpupool"
version = "0.1.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"futures 0.1.17 (registry+https://github.com/rust-lang/crates.io-index)",
"num_cpus 1.8.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "httparse"
version = "1.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "hyper"
version = "0.10.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"base64 0.6.0 (registry+https://github.com/rust-lang/crates.io-index)",
"httparse 1.2.3 (registry+https://github.com/rust-lang/crates.io-index)",
"language-tags 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
"log 0.3.9 (registry+https://github.com/rust-lang/crates.io-index)",
"mime 0.2.6 (registry+https://github.com/rust-lang/crates.io-index)",
"num_cpus 1.8.0 (registry+https://github.com/rust-lang/crates.io-index)",
"time 0.1.38 (registry+https://github.com/rust-lang/crates.io-index)",
"traitobject 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
"typeable 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)",
"unicase 1.4.2 (registry+https://github.com/rust-lang/crates.io-index)",
"url 1.6.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "hyper"
version = "0.11.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"base64 0.9.0 (registry+https://github.com/rust-lang/crates.io-index)",
"bytes 0.4.5 (registry+https://github.com/rust-lang/crates.io-index)",
"futures 0.1.17 (registry+https://github.com/rust-lang/crates.io-index)",
"futures-cpupool 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)",
"httparse 1.2.3 (registry+https://github.com/rust-lang/crates.io-index)",
"language-tags 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
"log 0.3.9 (registry+https://github.com/rust-lang/crates.io-index)",
"mime 0.3.5 (registry+https://github.com/rust-lang/crates.io-index)",
"percent-encoding 1.0.1 (registry+https://github.com/rust-lang/crates.io-index)",
"relay 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
"time 0.1.38 (registry+https://github.com/rust-lang/crates.io-index)",
"tokio-core 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)",
"tokio-io 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)",
"tokio-proto 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
"tokio-service 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
"unicase 2.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "idna"
version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"matches 0.1.6 (registry+https://github.com/rust-lang/crates.io-index)",
"unicode-bidi 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)",
"unicode-normalization 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "iovec"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"libc 0.2.34 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "iron"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"hyper 0.10.13 (registry+https://github.com/rust-lang/crates.io-index)",
"log 0.3.9 (registry+https://github.com/rust-lang/crates.io-index)",
"mime_guess 1.8.3 (registry+https://github.com/rust-lang/crates.io-index)",
"modifier 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
"num_cpus 1.8.0 (registry+https://github.com/rust-lang/crates.io-index)",
"plugin 0.2.6 (registry+https://github.com/rust-lang/crates.io-index)",
"typemap 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)",
"url 1.6.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "itoa"
version = "0.3.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "kernel32-sys"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "kube_rust"
version = "1.0.0"
source = "git+https://github.com/avranju/kube-rust?rev=058de6366d0d75cb60b2d0fd5ba1abd2e7d83fff#058de6366d0d75cb60b2d0fd5ba1abd2e7d83fff"
dependencies = [
"base64 0.7.0 (registry+https://github.com/rust-lang/crates.io-index)",
"futures 0.1.17 (registry+https://github.com/rust-lang/crates.io-index)",
"hyper 0.11.10 (registry+https://github.com/rust-lang/crates.io-index)",
"serde 1.0.27 (registry+https://github.com/rust-lang/crates.io-index)",
"serde_derive 1.0.27 (registry+https://github.com/rust-lang/crates.io-index)",
"serde_json 1.0.9 (registry+https://github.com/rust-lang/crates.io-index)",
"serde_yaml 0.7.3 (registry+https://github.com/rust-lang/crates.io-index)",
"url 1.6.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "language-tags"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "lazy_static"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "lazycell"
version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "libc"
version = "0.2.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "linked-hash-map"
version = "0.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "log"
version = "0.3.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"log 0.4.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "log"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"cfg-if 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "matches"
version = "0.1.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "memchr"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"libc 0.2.34 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "mime"
version = "0.2.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"log 0.3.9 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "mime"
version = "0.3.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"unicase 2.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "mime_guess"
version = "1.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"mime 0.2.6 (registry+https://github.com/rust-lang/crates.io-index)",
"phf 0.7.21 (registry+https://github.com/rust-lang/crates.io-index)",
"phf_codegen 0.7.21 (registry+https://github.com/rust-lang/crates.io-index)",
"unicase 1.4.2 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "mio"
version = "0.6.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"fuchsia-zircon 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
"fuchsia-zircon-sys 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
"iovec 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
"kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
"lazycell 0.5.1 (registry+https://github.com/rust-lang/crates.io-index)",
"libc 0.2.34 (registry+https://github.com/rust-lang/crates.io-index)",
"log 0.3.9 (registry+https://github.com/rust-lang/crates.io-index)",
"miow 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
"net2 0.2.31 (registry+https://github.com/rust-lang/crates.io-index)",
"slab 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "miow"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
"net2 0.2.31 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
"ws2_32-sys 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "modifier"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "net2"
version = "0.2.31"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"cfg-if 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)",
"kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
"libc 0.2.34 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
"ws2_32-sys 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "num"
version = "0.1.41"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"num-bigint 0.1.41 (registry+https://github.com/rust-lang/crates.io-index)",
"num-complex 0.1.41 (registry+https://github.com/rust-lang/crates.io-index)",
"num-integer 0.1.35 (registry+https://github.com/rust-lang/crates.io-index)",
"num-iter 0.1.34 (registry+https://github.com/rust-lang/crates.io-index)",
"num-rational 0.1.40 (registry+https://github.com/rust-lang/crates.io-index)",
"num-traits 0.1.41 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "num-bigint"
version = "0.1.41"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"num-integer 0.1.35 (registry+https://github.com/rust-lang/crates.io-index)",
"num-traits 0.1.41 (registry+https://github.com/rust-lang/crates.io-index)",
"rand 0.3.19 (registry+https://github.com/rust-lang/crates.io-index)",
"rustc-serialize 0.3.24 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "num-complex"
version = "0.1.41"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"num-traits 0.1.41 (registry+https://github.com/rust-lang/crates.io-index)",
"rustc-serialize 0.3.24 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "num-integer"
version = "0.1.35"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"num-traits 0.1.41 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "num-iter"
version = "0.1.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"num-integer 0.1.35 (registry+https://github.com/rust-lang/crates.io-index)",
"num-traits 0.1.41 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "num-rational"
version = "0.1.40"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"num-bigint 0.1.41 (registry+https://github.com/rust-lang/crates.io-index)",
"num-integer 0.1.35 (registry+https://github.com/rust-lang/crates.io-index)",
"num-traits 0.1.41 (registry+https://github.com/rust-lang/crates.io-index)",
"rustc-serialize 0.3.24 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "num-traits"
version = "0.1.41"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "num_cpus"
version = "1.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"libc 0.2.34 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "percent-encoding"
version = "1.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "persistent"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"iron 0.6.0 (registry+https://github.com/rust-lang/crates.io-index)",
"plugin 0.2.6 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "phf"
version = "0.7.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"phf_shared 0.7.21 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "phf_codegen"
version = "0.7.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"phf_generator 0.7.21 (registry+https://github.com/rust-lang/crates.io-index)",
"phf_shared 0.7.21 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "phf_generator"
version = "0.7.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"phf_shared 0.7.21 (registry+https://github.com/rust-lang/crates.io-index)",
"rand 0.3.19 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "phf_shared"
version = "0.7.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"siphasher 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
"unicase 1.4.2 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "plugin"
version = "0.2.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"typemap 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "quote"
version = "0.3.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "rand"
version = "0.3.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"fuchsia-zircon 0.3.2 (registry+https://github.com/rust-lang/crates.io-index)",
"libc 0.2.34 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "redox_syscall"
version = "0.1.33"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "regex"
version = "0.2.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"aho-corasick 0.6.4 (registry+https://github.com/rust-lang/crates.io-index)",
"memchr 2.0.1 (registry+https://github.com/rust-lang/crates.io-index)",
"regex-syntax 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)",
"thread_local 0.3.5 (registry+https://github.com/rust-lang/crates.io-index)",
"utf8-ranges 1.0.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "regex-syntax"
version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "relay"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"futures 0.1.17 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "route-recognizer"
version = "0.1.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "router"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"iron 0.6.0 (registry+https://github.com/rust-lang/crates.io-index)",
"route-recognizer 0.1.12 (registry+https://github.com/rust-lang/crates.io-index)",
"url 1.6.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "rustc-serialize"
version = "0.3.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "safemem"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "scoped-tls"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "serde"
version = "1.0.27"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "serde_derive"
version = "1.0.27"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"quote 0.3.15 (registry+https://github.com/rust-lang/crates.io-index)",
"serde_derive_internals 0.19.0 (registry+https://github.com/rust-lang/crates.io-index)",
"syn 0.11.11 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "serde_derive_internals"
version = "0.19.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"syn 0.11.11 (registry+https://github.com/rust-lang/crates.io-index)",
"synom 0.11.3 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "serde_json"
version = "1.0.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"dtoa 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)",
"itoa 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)",
"num-traits 0.1.41 (registry+https://github.com/rust-lang/crates.io-index)",
"serde 1.0.27 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "serde_yaml"
version = "0.7.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"linked-hash-map 0.5.0 (registry+https://github.com/rust-lang/crates.io-index)",
"num-traits 0.1.41 (registry+https://github.com/rust-lang/crates.io-index)",
"serde 1.0.27 (registry+https://github.com/rust-lang/crates.io-index)",
"yaml-rust 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "siphasher"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "slab"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "slab"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "smallvec"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "syn"
version = "0.11.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"quote 0.3.15 (registry+https://github.com/rust-lang/crates.io-index)",
"synom 0.11.3 (registry+https://github.com/rust-lang/crates.io-index)",
"unicode-xid 0.0.4 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "synom"
version = "0.11.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"unicode-xid 0.0.4 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "take"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "thread_local"
version = "0.3.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"lazy_static 1.0.0 (registry+https://github.com/rust-lang/crates.io-index)",
"unreachable 1.0.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "time"
version = "0.1.38"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
"libc 0.2.34 (registry+https://github.com/rust-lang/crates.io-index)",
"redox_syscall 0.1.33 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "tokio-core"
version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"bytes 0.4.5 (registry+https://github.com/rust-lang/crates.io-index)",
"futures 0.1.17 (registry+https://github.com/rust-lang/crates.io-index)",
"iovec 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
"log 0.3.9 (registry+https://github.com/rust-lang/crates.io-index)",
"mio 0.6.11 (registry+https://github.com/rust-lang/crates.io-index)",
"scoped-tls 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
"slab 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)",
"tokio-io 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "tokio-io"
version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"bytes 0.4.5 (registry+https://github.com/rust-lang/crates.io-index)",
"futures 0.1.17 (registry+https://github.com/rust-lang/crates.io-index)",
"log 0.3.9 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "tokio-proto"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"futures 0.1.17 (registry+https://github.com/rust-lang/crates.io-index)",
"log 0.3.9 (registry+https://github.com/rust-lang/crates.io-index)",
"net2 0.2.31 (registry+https://github.com/rust-lang/crates.io-index)",
"rand 0.3.19 (registry+https://github.com/rust-lang/crates.io-index)",
"slab 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)",
"smallvec 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
"take 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
"tokio-core 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)",
"tokio-io 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)",
"tokio-service 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "tokio-service"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"futures 0.1.17 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "traitobject"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "typeable"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "typemap"
version = "0.3.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"unsafe-any 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "unicase"
version = "1.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"version_check 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "unicase"
version = "2.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"version_check 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "unicode-bidi"
version = "0.3.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"matches 0.1.6 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "unicode-normalization"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "unicode-xid"
version = "0.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "unreachable"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"void 1.0.2 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "unsafe-any"
version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"traitobject 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "url"
version = "1.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"idna 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)",
"matches 0.1.6 (registry+https://github.com/rust-lang/crates.io-index)",
"percent-encoding 1.0.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "utf8-ranges"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "version_check"
version = "0.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "virtual-kubelet-adapter"
version = "0.1.0"
source = "git+https://github.com/avranju/rust-virtual-kubelet-adapter?rev=4250103d31e2864725e47bdd23295e79ee12b6d0#4250103d31e2864725e47bdd23295e79ee12b6d0"
dependencies = [
"bodyparser 0.8.0 (registry+https://github.com/rust-lang/crates.io-index)",
"iron 0.6.0 (registry+https://github.com/rust-lang/crates.io-index)",
"kube_rust 1.0.0 (git+https://github.com/avranju/kube-rust?rev=058de6366d0d75cb60b2d0fd5ba1abd2e7d83fff)",
"num 0.1.41 (registry+https://github.com/rust-lang/crates.io-index)",
"persistent 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)",
"router 0.6.0 (registry+https://github.com/rust-lang/crates.io-index)",
"serde 1.0.27 (registry+https://github.com/rust-lang/crates.io-index)",
"serde_json 1.0.9 (registry+https://github.com/rust-lang/crates.io-index)",
"url 1.6.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "void"
version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "web-rust"
version = "0.1.0"
dependencies = [
"env_logger 0.4.3 (registry+https://github.com/rust-lang/crates.io-index)",
"kube_rust 1.0.0 (git+https://github.com/avranju/kube-rust?rev=058de6366d0d75cb60b2d0fd5ba1abd2e7d83fff)",
"log 0.4.1 (registry+https://github.com/rust-lang/crates.io-index)",
"time 0.1.38 (registry+https://github.com/rust-lang/crates.io-index)",
"virtual-kubelet-adapter 0.1.0 (git+https://github.com/avranju/rust-virtual-kubelet-adapter?rev=4250103d31e2864725e47bdd23295e79ee12b6d0)",
]
[[package]]
name = "winapi"
version = "0.2.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "winapi-build"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "ws2_32-sys"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "yaml-rust"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"linked-hash-map 0.5.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[metadata]
"checksum aho-corasick 0.6.4 (registry+https://github.com/rust-lang/crates.io-index)" = "d6531d44de723825aa81398a6415283229725a00fa30713812ab9323faa82fc4"
"checksum base64 0.6.0 (registry+https://github.com/rust-lang/crates.io-index)" = "96434f987501f0ed4eb336a411e0631ecd1afa11574fe148587adc4ff96143c9"
"checksum base64 0.7.0 (registry+https://github.com/rust-lang/crates.io-index)" = "5032d51da2741729bfdaeb2664d9b8c6d9fd1e2b90715c660b6def36628499c2"
"checksum base64 0.9.0 (registry+https://github.com/rust-lang/crates.io-index)" = "229d032f1a99302697f10b27167ae6d03d49d032e6a8e2550e8d3fc13356d2b4"
"checksum bitflags 0.7.0 (registry+https://github.com/rust-lang/crates.io-index)" = "aad18937a628ec6abcd26d1489012cc0e18c21798210f491af69ded9b881106d"
"checksum bitflags 1.0.1 (registry+https://github.com/rust-lang/crates.io-index)" = "b3c30d3802dfb7281680d6285f2ccdaa8c2d8fee41f93805dba5c4cf50dc23cf"
"checksum bodyparser 0.8.0 (registry+https://github.com/rust-lang/crates.io-index)" = "f023abfa58aad6f6bc4ae0630799e24d5ee0ab8bb2e49f651d9b1f9aa4f52f30"
"checksum byteorder 1.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "652805b7e73fada9d85e9a6682a4abd490cb52d96aeecc12e33a0de34dfd0d23"
"checksum bytes 0.4.5 (registry+https://github.com/rust-lang/crates.io-index)" = "d828f97b58cc5de3e40c421d0cf2132d6b2da4ee0e11b8632fa838f0f9333ad6"
"checksum cfg-if 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)" = "d4c819a1287eb618df47cc647173c5c4c66ba19d888a6e50d605672aed3140de"
"checksum dtoa 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)" = "09c3753c3db574d215cba4ea76018483895d7bff25a31b49ba45db21c48e50ab"
"checksum env_logger 0.4.3 (registry+https://github.com/rust-lang/crates.io-index)" = "3ddf21e73e016298f5cb37d6ef8e8da8e39f91f9ec8b0df44b7deb16a9f8cd5b"
"checksum fuchsia-zircon 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "f6c0581a4e363262e52b87f59ee2afe3415361c6ec35e665924eb08afe8ff159"
"checksum fuchsia-zircon 0.3.2 (registry+https://github.com/rust-lang/crates.io-index)" = "bd510087c325af53ba24f3be8f1c081b0982319adcb8b03cad764512923ccc19"
"checksum fuchsia-zircon-sys 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "43f3795b4bae048dc6123a6b972cadde2e676f9ded08aef6bb77f5f157684a82"
"checksum fuchsia-zircon-sys 0.3.2 (registry+https://github.com/rust-lang/crates.io-index)" = "08b3a6f13ad6b96572b53ce7af74543132f1a7055ccceb6d073dd36c54481859"
"checksum futures 0.1.17 (registry+https://github.com/rust-lang/crates.io-index)" = "118b49cac82e04121117cbd3121ede3147e885627d82c4546b87c702debb90c1"
"checksum futures-cpupool 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)" = "e86f49cc0d92fe1b97a5980ec32d56208272cbb00f15044ea9e2799dde766fdf"
"checksum httparse 1.2.3 (registry+https://github.com/rust-lang/crates.io-index)" = "af2f2dd97457e8fb1ae7c5a420db346af389926e36f43768b96f101546b04a07"
"checksum hyper 0.10.13 (registry+https://github.com/rust-lang/crates.io-index)" = "368cb56b2740ebf4230520e2b90ebb0461e69034d85d1945febd9b3971426db2"
"checksum hyper 0.11.10 (registry+https://github.com/rust-lang/crates.io-index)" = "4844b207be8393981c5fcb61c9372d7c96432fcc8f5c3431a255a9d19b5c298b"
"checksum idna 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)" = "014b298351066f1512874135335d62a789ffe78a9974f94b43ed5621951eaf7d"
"checksum iovec 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "b6e8b9c2247fcf6c6a1151f1156932be5606c9fd6f55a2d7f9fc1cb29386b2f7"
"checksum iron 0.6.0 (registry+https://github.com/rust-lang/crates.io-index)" = "1d8e17268922834707e1c29e8badbf9c712c9c43378e1b6a3388946baff10be2"
"checksum itoa 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)" = "8324a32baf01e2ae060e9de58ed0bc2320c9a2833491ee36cd3b4c414de4db8c"
"checksum kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)" = "7507624b29483431c0ba2d82aece8ca6cdba9382bff4ddd0f7490560c056098d"
"checksum kube_rust 1.0.0 (git+https://github.com/avranju/kube-rust?rev=058de6366d0d75cb60b2d0fd5ba1abd2e7d83fff)" = "<none>"
"checksum language-tags 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)" = "a91d884b6667cd606bb5a69aa0c99ba811a115fc68915e7056ec08a46e93199a"
"checksum lazy_static 1.0.0 (registry+https://github.com/rust-lang/crates.io-index)" = "c8f31047daa365f19be14b47c29df4f7c3b581832407daabe6ae77397619237d"
"checksum lazycell 0.5.1 (registry+https://github.com/rust-lang/crates.io-index)" = "3b585b7a6811fb03aa10e74b278a0f00f8dd9b45dc681f148bb29fa5cb61859b"
"checksum libc 0.2.34 (registry+https://github.com/rust-lang/crates.io-index)" = "36fbc8a8929c632868295d0178dd8f63fc423fd7537ad0738372bd010b3ac9b0"
"checksum linked-hash-map 0.5.0 (registry+https://github.com/rust-lang/crates.io-index)" = "2d2aab0478615bb586559b0114d94dd8eca4fdbb73b443adcb0d00b61692b4bf"
"checksum log 0.3.9 (registry+https://github.com/rust-lang/crates.io-index)" = "e19e8d5c34a3e0e2223db8e060f9e8264aeeb5c5fc64a4ee9965c062211c024b"
"checksum log 0.4.1 (registry+https://github.com/rust-lang/crates.io-index)" = "89f010e843f2b1a31dbd316b3b8d443758bc634bed37aabade59c686d644e0a2"
"checksum matches 0.1.6 (registry+https://github.com/rust-lang/crates.io-index)" = "100aabe6b8ff4e4a7e32c1c13523379802df0772b82466207ac25b013f193376"
"checksum memchr 2.0.1 (registry+https://github.com/rust-lang/crates.io-index)" = "796fba70e76612589ed2ce7f45282f5af869e0fdd7cc6199fa1aa1f1d591ba9d"
"checksum mime 0.2.6 (registry+https://github.com/rust-lang/crates.io-index)" = "ba626b8a6de5da682e1caa06bdb42a335aee5a84db8e5046a3e8ab17ba0a3ae0"
"checksum mime 0.3.5 (registry+https://github.com/rust-lang/crates.io-index)" = "e2e00e17be181010a91dbfefb01660b17311059dc8c7f48b9017677721e732bd"
"checksum mime_guess 1.8.3 (registry+https://github.com/rust-lang/crates.io-index)" = "dc7e82a15629bb4ecd9e72365bf33d1382be91e030f820edb8e2a21c02430da8"
"checksum mio 0.6.11 (registry+https://github.com/rust-lang/crates.io-index)" = "0e8411968194c7b139e9105bc4ae7db0bae232af087147e72f0616ebf5fdb9cb"
"checksum miow 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "8c1f2f3b1cf331de6896aabf6e9d55dca90356cc9960cca7eaaf408a355ae919"
"checksum modifier 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "41f5c9112cb662acd3b204077e0de5bc66305fa8df65c8019d5adb10e9ab6e58"
"checksum net2 0.2.31 (registry+https://github.com/rust-lang/crates.io-index)" = "3a80f842784ef6c9a958b68b7516bc7e35883c614004dd94959a4dca1b716c09"
"checksum num 0.1.41 (registry+https://github.com/rust-lang/crates.io-index)" = "cc4083e14b542ea3eb9b5f33ff48bd373a92d78687e74f4cc0a30caeb754f0ca"
"checksum num-bigint 0.1.41 (registry+https://github.com/rust-lang/crates.io-index)" = "bdc1494b5912f088f260b775799468d9b9209ac60885d8186a547a0476289e23"
"checksum num-complex 0.1.41 (registry+https://github.com/rust-lang/crates.io-index)" = "58de7b4bf7cf5dbecb635a5797d489864eadd03b107930cbccf9e0fd7428b47c"
"checksum num-integer 0.1.35 (registry+https://github.com/rust-lang/crates.io-index)" = "d1452e8b06e448a07f0e6ebb0bb1d92b8890eea63288c0b627331d53514d0fba"
"checksum num-iter 0.1.34 (registry+https://github.com/rust-lang/crates.io-index)" = "7485fcc84f85b4ecd0ea527b14189281cf27d60e583ae65ebc9c088b13dffe01"
"checksum num-rational 0.1.40 (registry+https://github.com/rust-lang/crates.io-index)" = "0c7cb72a95250d8a370105c828f388932373e0e94414919891a0f945222310fe"
"checksum num-traits 0.1.41 (registry+https://github.com/rust-lang/crates.io-index)" = "cacfcab5eb48250ee7d0c7896b51a2c5eec99c1feea5f32025635f5ae4b00070"
"checksum num_cpus 1.8.0 (registry+https://github.com/rust-lang/crates.io-index)" = "c51a3322e4bca9d212ad9a158a02abc6934d005490c054a2778df73a70aa0a30"
"checksum percent-encoding 1.0.1 (registry+https://github.com/rust-lang/crates.io-index)" = "31010dd2e1ac33d5b46a5b413495239882813e0369f8ed8a5e266f173602f831"
"checksum persistent 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "8e8fa0009c4f3d350281309909c618abddf10bb7e3145f28410782f6a5ec74c5"
"checksum phf 0.7.21 (registry+https://github.com/rust-lang/crates.io-index)" = "cb325642290f28ee14d8c6201159949a872f220c62af6e110a56ea914fbe42fc"
"checksum phf_codegen 0.7.21 (registry+https://github.com/rust-lang/crates.io-index)" = "d62594c0bb54c464f633175d502038177e90309daf2e0158be42ed5f023ce88f"
"checksum phf_generator 0.7.21 (registry+https://github.com/rust-lang/crates.io-index)" = "6b07ffcc532ccc85e3afc45865469bf5d9e4ef5bfcf9622e3cfe80c2d275ec03"
"checksum phf_shared 0.7.21 (registry+https://github.com/rust-lang/crates.io-index)" = "07e24b0ca9643bdecd0632f2b3da6b1b89bbb0030e0b992afc1113b23a7bc2f2"
"checksum plugin 0.2.6 (registry+https://github.com/rust-lang/crates.io-index)" = "1a6a0dc3910bc8db877ffed8e457763b317cf880df4ae19109b9f77d277cf6e0"
"checksum quote 0.3.15 (registry+https://github.com/rust-lang/crates.io-index)" = "7a6e920b65c65f10b2ae65c831a81a073a89edd28c7cce89475bff467ab4167a"
"checksum rand 0.3.19 (registry+https://github.com/rust-lang/crates.io-index)" = "9e7944d95d25ace8f377da3ac7068ce517e4c646754c43a1b1849177bbf72e59"
"checksum redox_syscall 0.1.33 (registry+https://github.com/rust-lang/crates.io-index)" = "07b8f011e3254d5a9b318fde596d409a0001c9ae4c6e7907520c2eaa4d988c99"
"checksum regex 0.2.5 (registry+https://github.com/rust-lang/crates.io-index)" = "744554e01ccbd98fff8c457c3b092cd67af62a555a43bfe97ae8a0451f7799fa"
"checksum regex-syntax 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)" = "8e931c58b93d86f080c734bfd2bce7dd0079ae2331235818133c8be7f422e20e"
"checksum relay 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "f301bafeb60867c85170031bdb2fcf24c8041f33aee09e7b116a58d4e9f781c5"
"checksum route-recognizer 0.1.12 (registry+https://github.com/rust-lang/crates.io-index)" = "cf3255338088df8146ba63d60a9b8e3556f1146ce2973bc05a75181a42ce2256"
"checksum router 0.6.0 (registry+https://github.com/rust-lang/crates.io-index)" = "dc63b6f3b8895b0d04e816b2b1aa58fdba2d5acca3cbb8f0ab8e017347d57397"
"checksum rustc-serialize 0.3.24 (registry+https://github.com/rust-lang/crates.io-index)" = "dcf128d1287d2ea9d80910b5f1120d0b8eede3fbf1abe91c40d39ea7d51e6fda"
"checksum safemem 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "e27a8b19b835f7aea908818e871f5cc3a5a186550c30773be987e155e8163d8f"
"checksum scoped-tls 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "f417c22df063e9450888a7561788e9bd46d3bb3c1466435b4eccb903807f147d"
"checksum serde 1.0.27 (registry+https://github.com/rust-lang/crates.io-index)" = "db99f3919e20faa51bb2996057f5031d8685019b5a06139b1ce761da671b8526"
"checksum serde_derive 1.0.27 (registry+https://github.com/rust-lang/crates.io-index)" = "f4ba7591cfe93755e89eeecdbcc668885624829b020050e6aec99c2a03bd3fd0"
"checksum serde_derive_internals 0.19.0 (registry+https://github.com/rust-lang/crates.io-index)" = "6e03f1c9530c3fb0a0a5c9b826bdd9246a5921ae995d75f512ac917fc4dd55b5"
"checksum serde_json 1.0.9 (registry+https://github.com/rust-lang/crates.io-index)" = "c9db7266c7d63a4c4b7fe8719656ccdd51acf1bed6124b174f933b009fb10bcb"
"checksum serde_yaml 0.7.3 (registry+https://github.com/rust-lang/crates.io-index)" = "e0f868d400d9d13d00988da49f7f02aeac6ef00f11901a8c535bd59d777b9e19"
"checksum siphasher 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)" = "0df90a788073e8d0235a67e50441d47db7c8ad9debd91cbf43736a2a92d36537"
"checksum slab 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)" = "17b4fcaed89ab08ef143da37bc52adbcc04d4a69014f4c1208d6b51f0c47bc23"
"checksum slab 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "fdeff4cd9ecff59ec7e3744cbca73dfe5ac35c2aedb2cfba8a1c715a18912e9d"
"checksum smallvec 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "4c8cbcd6df1e117c2210e13ab5109635ad68a929fcbb8964dc965b76cb5ee013"
"checksum syn 0.11.11 (registry+https://github.com/rust-lang/crates.io-index)" = "d3b891b9015c88c576343b9b3e41c2c11a51c219ef067b264bd9c8aa9b441dad"
"checksum synom 0.11.3 (registry+https://github.com/rust-lang/crates.io-index)" = "a393066ed9010ebaed60b9eafa373d4b1baac186dd7e008555b0f702b51945b6"
"checksum take 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "b157868d8ac1f56b64604539990685fa7611d8fa9e5476cf0c02cf34d32917c5"
"checksum thread_local 0.3.5 (registry+https://github.com/rust-lang/crates.io-index)" = "279ef31c19ededf577bfd12dfae728040a21f635b06a24cd670ff510edd38963"
"checksum time 0.1.38 (registry+https://github.com/rust-lang/crates.io-index)" = "d5d788d3aa77bc0ef3e9621256885555368b47bd495c13dd2e7413c89f845520"
"checksum tokio-core 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)" = "c87c27560184212c9dc45cd8f38623f37918248aad5b58fb65303b5d07a98c6e"
"checksum tokio-io 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)" = "514aae203178929dbf03318ad7c683126672d4d96eccb77b29603d33c9e25743"
"checksum tokio-proto 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "8fbb47ae81353c63c487030659494b295f6cb6576242f907f203473b191b0389"
"checksum tokio-service 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "24da22d077e0f15f55162bdbdc661228c1581892f52074fb242678d015b45162"
"checksum traitobject 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "efd1f82c56340fdf16f2a953d7bda4f8fdffba13d93b00844c25572110b26079"
"checksum typeable 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)" = "1410f6f91f21d1612654e7cc69193b0334f909dcf2c790c4826254fbb86f8887"
"checksum typemap 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)" = "653be63c80a3296da5551e1bfd2cca35227e13cdd08c6668903ae2f4f77aa1f6"
"checksum unicase 1.4.2 (registry+https://github.com/rust-lang/crates.io-index)" = "7f4765f83163b74f957c797ad9253caf97f103fb064d3999aea9568d09fc8a33"
"checksum unicase 2.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "284b6d3db520d67fbe88fd778c21510d1b0ba4a551e5d0fbb023d33405f6de8a"
"checksum unicode-bidi 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)" = "49f2bd0c6468a8230e1db229cff8029217cf623c767ea5d60bfbd42729ea54d5"
"checksum unicode-normalization 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)" = "51ccda9ef9efa3f7ef5d91e8f9b83bbe6955f9bf86aec89d5cce2c874625920f"
"checksum unicode-xid 0.0.4 (registry+https://github.com/rust-lang/crates.io-index)" = "8c1f860d7d29cf02cb2f3f359fd35991af3d30bac52c57d265a3c461074cb4dc"
"checksum unreachable 1.0.0 (registry+https://github.com/rust-lang/crates.io-index)" = "382810877fe448991dfc7f0dd6e3ae5d58088fd0ea5e35189655f84e6814fa56"
"checksum unsafe-any 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)" = "f30360d7979f5e9c6e6cea48af192ea8fab4afb3cf72597154b8f08935bc9c7f"
"checksum url 1.6.0 (registry+https://github.com/rust-lang/crates.io-index)" = "fa35e768d4daf1d85733418a49fb42e10d7f633e394fccab4ab7aba897053fe2"
"checksum utf8-ranges 1.0.0 (registry+https://github.com/rust-lang/crates.io-index)" = "662fab6525a98beff2921d7f61a39e7d59e0b425ebc7d0d9e66d316e55124122"
"checksum version_check 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)" = "6b772017e347561807c1aa192438c5fd74242a670a6cffacc40f2defd1dc069d"
"checksum virtual-kubelet-adapter 0.1.0 (git+https://github.com/avranju/rust-virtual-kubelet-adapter?rev=4250103d31e2864725e47bdd23295e79ee12b6d0)" = "<none>"
"checksum void 1.0.2 (registry+https://github.com/rust-lang/crates.io-index)" = "6a02e4885ed3bc0f2de90ea6dd45ebcbb66dacffe03547fadbb0eeae2770887d"
"checksum winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)" = "167dc9d6949a9b857f3451275e911c3f44255842c1f7a76f33c55103a909087a"
"checksum winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "2d315eee3b34aca4797b2da6b13ed88266e6d612562a0c46390af8299fc699bc"
"checksum ws2_32-sys 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "d59cefebd0c892fa2dd6de581e937301d8552cb44489cdff035c6187cb63fa5e"
"checksum yaml-rust 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "57ab38ee1a4a266ed033496cf9af1828d8d6e6c1cfa5f643a2809effcae4d628"

Some files were not shown because too many files have changed in this diff Show More