Support for changing timezones

[sjthespian]

I have a somewhat unusual situation, I'm looking at using kured on systems that move -- for example a Kubernetes data center on a ship. This means I will regularly have to update the timezone to make sure reboots are happening in the middle of the night local time and/or update the start/end times if I'm keeping things in UTC.

Can you think of any options for this other than updating the settings in our config and redeploying the application? It would be nice to be able to update an annotation and/or send the application a message to change the timezone or other parameters w/o having to redeploy. I can see use cases where we might also want to change the start/end times or other command line options.

Thanks!

[evrardjp]

It's been a while we're pondering whether to use a CRD or not.
We've managed to dodge this at the moment.

I believe that storing that information in a configmap would be okayish, but I would have prefered a CRD to avoid watching on configmaps. What do you think @jackfrancis ?

[evrardjp]

Oh, and I forgot to say: "interesting use case!"

[sjthespian]

"interesting use case" is pretty-much my day-to-day life. :).

A CRD is an interesting idea, I assume you would run kured as an operator and let it create and manage the daemonset? Or would you just let the operator spin up pods to check for the reboot trigger and run the reboot, more like the k8s equivalent of a cron scheduler?

In my world this is all to make sure we safely reboot the cluster nodes after we patch the OS. I have a dream that at some point we will move to something like CoreOS or SUSE Linux Micro and get out of the OS management business. Then we could replace the reboots by rebuilding the nodes under the cluster instead of rebooting.

[evrardjp]

ah... the immutable story. If everybody would be using immutable distros, kured would almost not exist.

Every customer I have seen claims that their infra is immutable. When you asked them when they rebuilt (and rolled out an update of) their nodes last, it's a different story based on their maturity.

This is why some of my customers still use Kured. Not everything is immutable for them (against what some shiny vendors tell you). Sometimes you can't afford to rebuild the whole infra and roll it out over time. Sometimes you just need to roll out a patch (not a complete distro) fast. And sometimes you want to do a complete cleanup and rollout new nodes everywhere...

I am not yet sold on the CRD tbh. I like it because it gives a way to handle maintenance windows more elegantly. Your case enters that part. At the same time, it's far more complex than having a config map, which passes the right flags.

Do not forget that you can combine kured with things like stakater/reloader: https://github.com/stakater/Reloader . It would solve your "no need to redeploy", as a configmap change would auto trigger a reload. Editing a cm is not so far from editing a label/annotation I guess...

[evrardjp]

Did you think of the reloader case?

I am currently designing the v2, I was wondering if CRD/cm is something I need to prioritise.

[sjthespian]

I like the idea of a CRD, but honestly I'm not sure if the amount of work it will take to implement it makes sense. I'm looking at spinning up kured via. helm and removing it at the end of the window. In our case our L1 team would just run an AWX job at the start of the maintenance window and then a second one at the end.

As much as I would prefer to have this as a daemonset that is there all the time, I'm not sure how I would manage both changing maintenance windows and changing time zones, even with a CRD. Especially since I don't have an easy way to tell what time zone a ship is in... I can get the offset from GMT via. a service call, but converting that to a time zone name is non-trivial.

[evrardjp]

What about a different approach? Have really open ended maintenance windows BUT having blockers during the day? It shifts where the source of data is.

Alternatively to the blockers: use a different command to know whether to reboot or not ?

[github-actions]

This issue was automatically considered stale due to lack of activity. Please update it and/or join our slack channels to promote it, before it automatically closes (in 7 days).

[evrardjp]

I did a demo last week that might interest you @sjthespian .
In kured-v2, each maintenance window is a config map. Amongst the data in the config map is a nodeSelector.

With this model, you can select your nodes in a certain region in a specific maintenance window.
On top of that, the maintenance window start now takes the robfig/cron input, which means TZ are allowed!

I think it should do the trick. The only thing I need is beta testers and/or ppl willing to provide e2e tests for such features.