This requires dropping otel down to v0.20 because the apiserver package
is importing it and some packages moved around with otel v1.
Even k8s v0.25 still uses this old version of otel, so we are stuck for
a bit (v0.26 will, as of now, use a newer otel version).
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Changes in klog and logr have made automatic bumps from dependabot
problematic.
We also shouldn't need klogv1 so removed that.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
With this one line, vk fails to build against k8s 1.24 libs.
The comment says:
// Deprecated: ClusterName is a legacy field that was always cleared by
// the system and never used; it will be removed completely in 1.25.
Seems to be removed in 1.24 though.
Auth is not automatically enabled because this requires some
bootstrapping to work.
I'll leave this for some future work.
In the meantime people can use the current code similar to how they used
the node-cli code to inject their own auth.
This moves API handling into the node object so now everything can be
done in one place.
TLS is required.
In the current form, auth must be setup by the caller.
This changes `ControllerManager` to `Node`.
`Node` is created from a client where the VK lib is responsible for
creating all the things except the client (unless client is nil, then we
use the env client).
This should be a good replacement for node-cli. It offers a simpler
API. *It only works with leases enabled* since this seems always
desired, however an option could be added to disable if needed.
The intent of this is to provide a simpler way to get a vk node up and
running while also being extensible. We can slowly add options, but
they should be focussed on a use-case rather than trying to support
every possible scenario... in which case the user can just use the
controllers directly.
Found that this caused a panic after many many test runs.
It seems like we should have returned early since the pingResult is nil.
We don't want to update a lease when ping fails.
This makes a controller that handles the startup for the node and pod
controller.
Later if we add an "api controller" it can also be added here.
This is just part of reducing some of the boiler plate code so it is
easier to get off of node-cli.
In error cases these goroutines never exit.
Trying to debug cases we end up with a bunch of these goroutines stuck
making it difficult to troubleshoot.
We could just make a buffered channel, however this will makes it less
clear, in cases of an error, what all is happening.
This drops another dependency on k8s.io/kubernetes.
This does have the unfortunate side effect that implementers will now
get a compile error until they update their code to use the new type.
Just as a note:
The stats types have moved to k8s.io/kubelet, however the stats types
are only there as of v1.20.
Currently we support older versions than v1.20, and even our go.mod
imports from v1.19.
For now we copy the types in. Later we can remove the type defs and
change them to type aliases to the k8s.io/kubelet types (which prevents
another compile time issue).
Anything relying on type assertions to determine if something implements
this method will, unfortunately, be broken and it will be hard to notice
until runtime. We need to make sure to call this out in the release
notes.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Our current retry policy is naive and only does 20 retries. It is
also based off of the rate limiter. If the user is somewhat aggressive in
rate limiting, but they have a temporary outage on API server, they
may want to continue to delay.
In facts, K8s has a built-in function to suggest delays:
https://pkg.go.dev/k8s.io/apimachinery/pkg/api/errors#SuggestsClientDelay
Signed-off-by: Sargun Dhillon <sargun@sargun.me>
This starts watching for events prior to the start of the controller.
This smells like a bug in the fakeclient bits, but it seems to fix
the problem.
Signed-off-by: Sargun Dhillon <sargun@sargun.me>
We had added an optimization that made it so we dedupe pod status updates
from the provider. This ignored two subfields that could be updated along
with status.
Because the details of subresource updating is a bit API server centric,
I wrote an envtest which checks for this behaviour.
Signed-off-by: Sargun Dhillon <sargun@sargun.me>
This is a fundamentally different API than that of the K8s workqueue
which is better suited for our needs. Specifically, we need a simple
queue which doesn't have complex features like delayed adds that
sit on "external" goroutines.
In addition, we need deep introspection into the operations of the
workqueue. Although you can get this on top of the K8s workqueue
by implementing a custom rate limiter, the problem is that
the underlying rate limiter's behaviour is still somewhat
opaque.
This basically has 100% code coverage.
If you share a ratelimiter between workqueues, it breaks.
WQ1: Starts processing item (When)
WQ1: Fails to process item (When)
WQ1: Fails to process item (When)
WQ1: Fails to process item (When)
--- At this point we've backed off a bit ---
WQ2: Starts processing item (with same key, When)
WQ2: Succeeds at processing item (Forget)
WQ1: Fails to process item (When) ---> THIS RESULTS IN AN ERROR
This results in an error because it "forgot" the previous
rate limit.
Copy/paste some more kubernetes code. This is to remove the dep on
kubernetes/kubernetes from within exec.go
See #940
Signed-off-by: Miek Gieben <miek@miek.nl>
This refactor is a preparation for another commit. I want to add instrumentation
around our queues. The code of how queues were handled was spread throughout
the code base, and that made adding such instrumentation nice and complicated.
This centralizes the queue management logic in queue.go, and only requires
the user to provide a (custom) rate limiter, if they want to, a name,
and a handler.
The lease code is moved into its own package to simplify testing, because
the goroutine leak tester was triggering incorrectly if other tests
were running, and it was measuring leaks from those tests.
This also identified buggy behaviour:
wq := workqueue.NewNamedRateLimitingQueue(workqueue.DefaultItemBasedRateLimiter(), "test")
wq.AddRateLimited("hi")
fmt.Printf("Added hi, len: %d\n", wq.Len())
wq.Forget("hi")
fmt.Printf("Forgot hi, len: %d\n", wq.Len())
wq.Done("hi")
fmt.Printf("Done hi, len: %d\n", wq.Len())
---
Prints all 0s because event non-delayed items are delayed. If you call Add
directly, then the last line prints a len of 2.
// Workqueue docs:
// Forget indicates that an item is finished being retried. Doesn't matter whether it's for perm failing
// or for success, we'll stop the rate limiter from tracking it. This only clears the `rateLimiter`, you
// still have to call `Done` on the queue.
^----- Even this seems untrue
This refactors the v1 lease controller. It makes two functional differences
to the lease controller:
* It no longer ties lease updates to node pings or node status updates
* There is no fallback mechanism to status updates
This also moves vk_envtest, allowing for future brown-box testing of the
lease controller with envtest
As described in the issue, if the following sequence happens, we fail to properly
update the pod status in api server:
1. Create pod in k8s
2. Provider creates the pod and syncs its status back
3. Pod in k8s ready/running, all fine
4. Virtual kubelet fails to update node status for some time for whatever reason (e.g. network connectivity issues)
5. Virtual node marked as NotReady with message: Kubelet stopped posting node status
6. kube-controller-manager of k8s, goes and marks all pods as Ready = false:
7. Virtual kubelet never sync's status of pod in provider back to k8s
We were having issues with golint not properly reporting declaration of functions
without proper documentation (comments). This is due to a config with golangci.
See: https://github.com/golangci/golangci-lint/issues/456
This moves node ping controller to using the new internal lock
API.
The reason for this is twofold:
* The channel approach that was used to notify other
controllers of changes could only be used once (at startup),
and couldn't be used in the future to broadcast node
ping status. The idea idea is here that we could move
to a sync.Cond style API and only wakeup other controllers
on change, as opposed to constantly polling each other
* The problem with sync.Cond is that it's not context friendly.
If we want to do stuff like wait on a sync.cond and use a context
or a timer or similar, it doesn't work whereas this API allows
context cancellations on condition change.
The idea is that as we have more controllers that act as centralized
sources of authority, they can broadcast out their state.
