If we don't handle the "pod not found" case then we end up with the pod
getting re-queued over and over until the max retries are hit. It also
blocks the queue for other pod status updates for that pod
namespace/name.
This changes the tracing package to accept an error on SetStatus, which
is really what we always want anyway.
This also decouples the trace package from opencensus.
This moves the logic for re-creating the a missing node up into the CLI.
We can make this optional, but for now I've just preserved existing
functionality.
Example:
env name: FOO
value: "foo"
name: BAR
value: "bar"
name: FOOBAR
value: "${FOO}${BAR}" <-- should expand to: "foobar"
Added testcase for the same as well.
Change is based on kubelet_pods.go.
Simplified some of the existing code.
* Use the VKUBELET API for fetching the pods during testing, which
makes it easier to run testing without wiring up an entire cluster
* Runningpods does not require each provider to implement the
optional stats interface for testing.
Don't use actual "mock" provider in vkubelet to reduce awkward import
cycles.
Really the "mock" provider is for e2e tests. Instead create our own mock
in the vkubelet package specifically for unit tests which can also be
more easily customized to test edge cases.
This makes the concept of a `Provider` wholely implemented in the cli
implementation in cmd/virtual-kubelet.
It allows us to slim down the interfaces used in vkubelet (and
vkubelet/api) to what is actually used there rather than a huge
interface that is only there to serve the CLI's needs.
* Light up UpdatePod
This PR updates the vkublet/pod.go createOrUpdate(..) method to actually handle
updates. It gets the pod from the provider as before, but now if it exists the method
checks the hash of the spec against the spec of the new pod. If they've changed, it
calls UpdatePod(..).
Also makes a small change to the Server struct to swap from kuberentes.Clientset to kubernetes.Interface
to better facilitate testing with fake ClientSet.
Co-Authored-By: Brian Goff <cpuguy83@gmail.com>
For some reason the kubelet tries to do things that it gets perm
denied. This changes the base.yml in the skaffold file to have
the permissions. In turn, this broke the node deletion test. So,
instead of looking for the deletion event (imperative), we wait
for the node UID to change (declarative).
We're in effect testing the K8s scheduler logic in the test
by setting taints, as opposed to the actual VK itself. If we
want to make sure the taint is set, we can just observe the node
object.
Instead, bind to the pod to the VK node explicitly.
* Add the /runningpods/ api endpoint
This adds an API endpoint from the kubelet (/runningpods/). It is
an endpoint on kubelet which is considered a "debug" endpoint, so
it might be worth exposing through the options, but by default
it is exposed in most k8s configs AFAICT.
* Add tracing of the kubernetes cluster during testing
This adds tracing to the testing to get the kubelet's logs upon
failure. In addition, it keeps track of the pods, and the node
statuses throughout the test.
* Add arguments to make virtual kubelet's log more useful
This adds a 5 minute timeout to the end-to-end tests. The end-to-end
tests typically run in under 2 minutes. On Circle-CI the timeout is
10 minutes, at which point, Circle CI just shoots the tests in the
head so we don't get any logs.
All of Kubernetes logging is based on klog. Klog currently does
not output any logging information to logrus, so you're flying
somewhat blind to Kubernetes internals.
This exposes the full set of configurables that klog offers,
but decorates (prefixes) the klog configuration with "klog".
