This makes the concept of a `Provider` wholely implemented in the cli
implementation in cmd/virtual-kubelet.
It allows us to slim down the interfaces used in vkubelet (and
vkubelet/api) to what is actually used there rather than a huge
interface that is only there to serve the CLI's needs.
* Light up UpdatePod
This PR updates the vkublet/pod.go createOrUpdate(..) method to actually handle
updates. It gets the pod from the provider as before, but now if it exists the method
checks the hash of the spec against the spec of the new pod. If they've changed, it
calls UpdatePod(..).
Also makes a small change to the Server struct to swap from kuberentes.Clientset to kubernetes.Interface
to better facilitate testing with fake ClientSet.
Co-Authored-By: Brian Goff <cpuguy83@gmail.com>
* Add the /runningpods/ api endpoint
This adds an API endpoint from the kubelet (/runningpods/). It is
an endpoint on kubelet which is considered a "debug" endpoint, so
it might be worth exposing through the options, but by default
it is exposed in most k8s configs AFAICT.
This adds two concepts, where one encompasses the other.
Startup timeout
Startup timeout is how long to wait for the entire kubelet
to get into a functional state. Right now, this only waits
for the pod informer cache for the pod controllerto become
in-sync with API server, but this could be extended to other
informers (like secrets informer).
Wait For Startup
This changes the behaviour of the virtual kubelet to wait
for the pod controller to start before registering the node.
It is to avoid the race condition where the node is registered,
but we cannot actually do any pod operations.
node.ResourceVersion must not be set when creating a node.
This issue prevents vk from resolving issues after the vk node instance
has been deleted (for whatever reason).
* Patch the node status instead of updating it.
Virtual-kubelet updates the node status periodically.
This change proposes we use the `Patch` API instead of `Update`,
to update the node status.
This avoids overwriting any node updates made by other controllers
in the system, for example a attach-detach controller.
Patch API does a strategic merge instead of overwriting
the entire object, which ensures parallel updates don't overwrite
each other.
Note: `PatchNodeStatus` reduces the time precision to the seconds-level
and therefore I corrected the test for this.
consider two controllers:
CONTROLLER 1 (virtual kubelet) | CONTROLLER 2
oldNode := nodes.Get(nodename) |
| node := nodes.Get(nodename)
| // update status with attached volumes info
| updateNode := Nodes.UpdateStatus(node)
// update vkubelet info on node status |
latestNode := Nodes.UpdateStatus(oldNode) |
<-- latestNode does not contain the volume info added by second controller.
with my patch change:
CONTROLLER 1 (virtual kubelet) | CONTROLLER 2
oldNode := Nodes.Get(nodename) |
| node := Nodes.Get(nodename)
| // update status with attached volumes info
| updateNode := Nodes.UpdateStatus(node)
node := oldNode.DeepCopy() |
// update vkubelet info on node status |
latestNode := util.PatchNodeStatus(oldNode, node) |
<-- latestNode contains the volume info added by second controller.
Testing Done: make test
* Introduce PatchNodeStatus into vkubelet.
* Pass only the node interface.
This adds a new interface that a provider can implement which enables
async notifications of pod status changes rather than the existing loop
which goes through every pod in k8s and checks the status in the
provider.
In practice this should be significantly more efficient since we are not
constantly listing all pods and then looking up the status in the
provider.
For providers that do not support this interface, the old method is
still used to sync state from the provider.
This commit does not update any of the providers to support this
interface.
This starts the work of having a `NodeProvider` which is responsible for
providing node details.
It splits the responsibilities of node management off to a new
controller.
The primary change here is to add the framework pieces for node
management and move the VK CLI to use this new controller.
It also adds support for node leases where available. This can be
enabled via the command line (disabled by default), but may fall back if
we find that leaess aren't supported on the cluster.
* Define and use an interface for logging.
This allows alternative implementations to use whatever logging package
they want.
Currently the interface just mimicks what logrus already implements,
with minor modifications to not rely on logrus itself. I think the
interface is pretty solid in terms of logging implementations being able
to do what they need to.
* Make tracing interface to coalesce logging/tracing
Allows us to share data between the tracer and the logger so we can
simplify log/trace handling wher we generally want data to go both
places.
Tickers always tick, so if we tick every 5 seconds and the work that we
perform at each tick takes 5 seconds, we end up just looping with no
sleep period.
Instead this is using a timer to ensure we actually get a full 5 second
sleep between loops.
We should consider an async API instead of polling the provider like
this.
* Don't start things in New
* Move http server handling up to daemon.
This removes the burdern of dealing with listeners, http servers, etc in
the core framework.
Instead provide helpers to attach the appropriate routes to the
caller's serve mux.
With this change, the vkubelet package only helps callers setup HTTP
rather than forcing a specific HTTP config on them.
* deps: bump to Kubernetes 1.13.1
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* version: new VK version
Signed-off-by: Paulo Pires <pjpires@gmail.com>
This uses the same number of workers as the pod sync workers.
We may want to start a worker queue here instead, but I think for now
this is ok, particularly because we are limiting the number of
goroutines being spun up at once.
* vendor: add vendored code
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* controller: use shared informers and a work queue
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* errors: use cpuguy83/strongerrors
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* aci: fix test that uses resource manager
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* readme: clarify skaffold run before e2e
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* cmd: use root context everywhere
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* sync: refactor pod lifecycle management
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* e2e: fix race in test when observing deletions
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* e2e: test pod forced deletion
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* cmd: fix root context potential leak
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* sync: rename metaKey
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* sync: remove calls to HandleError
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* Revert "errors: use cpuguy83/strongerrors"
This reverts commit f031fc6d.
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* manager: remove redundant lister constraint
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* sync: rename the pod event recorder
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* sync: amend misleading comment
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* mock: add tracing
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* sync: add tracing
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* test: observe timeouts
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* trace: remove unnecessary comments
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* sync: limit concurrency in deleteDanglingPods
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* sync: never store context, always pass in calls
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* sync: remove HandleCrash and just panic
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* sync: don't sync succeeded pods
Signed-off-by: Paulo Pires <pjpires@gmail.com>
* sync: ensure pod deletion from kubernetes
Signed-off-by: Paulo Pires <pjpires@gmail.com>
Updates the pod status in Kubernetes to "Failed" when the pod status is
not found from the provider.
Note that currently thet most providers return `nil, nil` when a pod is
not found. This works but should possibly return a typed error so we can
determine if the error means not found or something else... but this
works as is so I haven't changed it.
* Add k8s.io/client-go/tools/cache package
* Add cache controller
* Add pod creator and terminator
* Pod Synchronizer
* Clean up
* Add back reconcile
* Remove unnecessary space in log
* Incorprate feedbacks
* dep ensure
* Fix the syntax error
* Fix the merge errors
* Minor Refactor
* Set status
* Pass context together with the pod to the pod channel
* Change to use flag to specify the number of pod sync workers
* Remove the unused const
* Use Stable PROD Region WestUS in Test
EastUS2EUAP is not reliable
* Refactor provider init
This moves provider init out of vkubelet setup, instead preferring to
initialize vkubelet with a provider.
* Split API server configuration from setup.
This makes sure that configuration (which is done primarily through env
vars) is separate from actually standing up the servers.
This also makes sure to abort daemon initialization if the API servers
are not able to start.
Alibaba Cloud ECI(Elastic Container Instance) is a service that allow you
run containers without having to manage servers or clusters.
This commit add ECI provider for virtual kubelet, connects ECI with
kubernetes cluster.
Signed-off-by: xianwei.zw <xianwei.zw@alibaba-inc.com>
Signed-off-by: shidao.ytt <shidao.ytt@alibaba-inc.com>
This refactors a bit of the http handler code.
Moves error handling for handler functions to a generic handler.
This also has a side-effect of being able to propagate errors from the
provider to send the correct status code, provided the error type
matches a pre-defined interface.
