Push notifications.

It is fine hosting a service that gets requests then talks to FCM or the iOS version. But a service that one’s phone stays connected to 24/7 is really hard, and not kill one’s battery.

As an SRE I rarely touch customer-facing stuff, but from what I’ve seen of the devs they are often several layers removed from the public docs. Most are simply focused on their own cog. For this reason I am gravitating towards the projects they’ve open-sourced (k8s, grpc, bazel) and building from more of a clean slate. I’d much prefer open-source components that I can fit into a k8s cluster than rely have lock-in on some cloud service. They solve some nice things, but I’d like to run it locally if I want. For example I’d much prefer to have my pubsub stack rely on Redis Streams rather than GCP PubSub. Redis has such a small footprint, scales to 16k nodes and given how fast it is that is way more of a ceiling than I need. GCP’s UI is nice, but at the end of the day I’m going to be editing some config file and letting my CICD pipeline roll it out than going to the GCP console and clicking some buttons. But that’s just me.

10+ years at Google as an SRE. While borg =!= k8s, I’ve seen my fair share of platforms come and go. The trend seems to reward shifts towards declarative automation rather than imperative orchestration models. In the programming world, you’ll hear the term idempotent, similar idea. There is no substitute or wrapper that can take imperative and make it declarative without tons of work. Ansible is imperative where if something goes wrong it is easiest to nuke then try again. K8s is the culmination of various imperative-based automation systems at Google, attempts at replacing them with declarative, then try again, then finally start afresh with an open-source version of borg.

Not many companies need the scale of Google, with thousands of engineers trying to modify production with hardened interfaces that force developers to write their applications in such an opinionated way (stateful applications must use StatefulSet, dynamic configuration should go into a ConfigMap, separate your command line arguments from the command being executed from the environment variables, LoadBalancers are distinct from and are an implementation detail of Services…).

But with the good foundation that k8s provides and imposes, you set yourself up for letting the infrastructure team not care about what is running on what hardware. They can focus on doing hardware, networking, disk swapouts… Ops can focus on service uptime, readiness+liveness probers, standardized monitoring/logging, traffic routing and rollouts. Devs can focus on writing code. These standards reduce the leakage that often happens between these 3 groups.

Taking declarative to the next level, you build CICD pipelines that can take your yaml files in a github repo and automatically push them. To the next level you want to account for importing templates and standard libraries, so you look to Kustomize till you realize that it doesn’t give you the building blocks you need. You then start to adopt more declarative models where the source code (both java and json/yaml config files) can be built and the artifacts of that build step are what are fed into k8s, making your github repo the source of truth. Then all production fiddling is done with PRs rather than clicking buttons in an imperative way on some UI.

The more you see automation tools, the more you realize that declarative offers a more robust interface that can be glued to other declarative systems, albiet adding yet another layer of abstraction. This complexity is often not streamlined enough for people on this subreddit, as well as for lots of people writing self-hosted apps. Helm is about as both streamlined and exhaustive as you’re going to get.

I agree with many here that learning k8s is best if you’re needing to learn it for your job, or you have hopes of getting into the DevOps field.