<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet href="/scripts/pretty-feed-v3.xsl" type="text/xsl"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:h="http://www.w3.org/TR/html4/" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>The Backyard Rack</title><description>G&apos;day from the homelab! Tinkering with servers, self-hosting the good stuff, and keeping the cloud out of me backyard. She&apos;ll be right, mate!</description><link>https://www.backyard-rack.com</link><item><title>Welcome to &quot;The Backyard Rack&quot;</title><link>https://www.backyard-rack.com/blog/welcome-to-backyard-rack</link><guid isPermaLink="true">https://www.backyard-rack.com/blog/welcome-to-backyard-rack</guid><description>Farewell and Welcome (The Song Remains the Same)</description><pubDate>Fri, 20 Nov 2026 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;Welcome to &lt;strong&gt;The Backyard Rack&lt;/strong&gt;!&lt;/p&gt;
&lt;p&gt;If you&apos;ve followed my Kubernetes, Proxmox, OpenStack, and general infrastructure adventures over on Medium, you might have noticed something different. I&apos;ve packed up my YAML, grabbed my Helm charts, and moved out. I&apos;m no longer publishing on Medium. Not because I&apos;ve stopped writing; quite the opposite. It&apos;s because Medium has become a big hot festering pile of shit.&lt;/p&gt;
&lt;p&gt;There. Said it.&lt;/p&gt;
&lt;h2&gt;Medium, come on mate!&lt;/h2&gt;
&lt;p&gt;Once upon a time, Medium was a decent place to publish technical articles. You wrote something useful, hit publish, and people who were interested in the subject could actually find it. Occasionally, someone would read the whole thing without being asked to sign up or become a bloody paid member.&lt;/p&gt;
&lt;p&gt;What a concept, righto? But somewhere along the way, the experience changed. The platform increasingly feels like it&apos;s optimised for everything &lt;em&gt;except&lt;/em&gt; connecting writers with readers.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;And then there&apos;s the AI circus&lt;/strong&gt;. The internet is drowning in mass-produced, AI-generated content. Endless articles promising to &lt;strong&gt;revolutionise&lt;/strong&gt;, &lt;strong&gt;supercharge&lt;/strong&gt;, and &lt;strong&gt;unlock the power of&lt;/strong&gt; whatever technology was trending five minutes ago. (add quoatations wherever you see fit)&lt;/p&gt;
&lt;p&gt;You know the ones:&lt;/p&gt;
&lt;p&gt;&lt;em&gt;&quot;10 Kubernetes Secrets That Will Change Your Life!&quot;&lt;/em&gt; or &lt;em&gt;&quot;Kubernetes is Dead -- Here is why!&quot;&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;Written by &lt;em&gt;something&lt;/em&gt; that has apparently never seen a Kubernetes cluster, let alone watched one catch fire at 2 a.m. Meanwhile, those of us actually building things, breaking things, fixing things, and documenting what happened are left competing for attention in an increasingly noisy feed. I&apos;m not claiming AI is the root of all evil. It&apos;s a useful tool, and I like useful tools. But there&apos;s a difference between using AI to help you work and flooding the internet with confident-sounding technical sludge. And when discovery and traffic increasingly depend on opaque algorithms, it&apos;s hard to justify putting hours into detailed technical writing on a platform where you have so little control over who sees it.&lt;/p&gt;
&lt;p&gt;Hence, welcome to &lt;strong&gt;The Backyard Rack&lt;/strong&gt;!&lt;/p&gt;
&lt;h2&gt;And No, I&apos;m Not Moving to Substack Either&lt;/h2&gt;
&lt;p&gt;Every time someone complains about Medium, there&apos;s inevitably a well-meaning suggestion: &lt;em&gt;&quot;Have you considered Substack?&quot;&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Yes. For about three seconds.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Look, I&apos;m sure Substack is lovely if your lifelong ambition is to become a writer (hahahahaha). But I don&apos;t want to. &lt;strong&gt;I want to run a bloody blog.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;I write technical articles about Kubernetes, Proxmox, OpenStack, and other infrastructure nonsense. I want people to find them when they&apos;re Googling an obscure error at 2 a.m., not receive a surprise essay in their inbox while they&apos;re trying to find a parcel delivery notification. I don&apos;t need subscribers, email campaigns, open rates, conversion funnels, or a weekly obligation to produce &lt;em&gt;content&lt;/em&gt;. I&apos;m not launching a media company. I&apos;m documenting stuff I&apos;ve built, broken, and occasionally managed to fix.&lt;/p&gt;
&lt;p&gt;And I certainly don&apos;t want to replace one publishing platform&apos;s algorithmic nonsense with another platform&apos;s obsession with turning every writer into a newsletter &lt;em&gt;entrepreneur&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;Just a website (with RSS). Like we used to have. &lt;strong&gt;Radical concept, apparently.&lt;/strong&gt;&lt;/p&gt;
&lt;h2&gt;So I Built My Own Little Corner of the Internet&lt;/h2&gt;
&lt;p&gt;Welcome to &lt;strong&gt;The Backyard Rack&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;The name is exactly what it sounds like: a nod to the kind of infrastructure experiments that start with a perfectly reasonable idea and the unjustified confidence of a junior developer:&lt;/p&gt;
&lt;p&gt;&lt;em&gt;&quot;Surely this won&apos;t be too difficult.&quot;&lt;/em&gt;, Famous last words, my young Padawan!&lt;/p&gt;
&lt;p&gt;This is now the home for my technical articles, experiments, tutorials, and occasional infrastructure-induced rage.&lt;/p&gt;
&lt;p&gt;Expect plenty of:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Kubernetes&lt;/strong&gt;, because apparently I enjoy making simple things distributed (it&apos;s a microservice architecture you wouldn&apos;t understand).&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Proxmox and virtualisation&lt;/strong&gt;, there&apos;s always room for another VM (not with these RAM prices mate!).&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;OpenStack and cloud infrastructure&lt;/strong&gt;, for when Kubernetes alone isn&apos;t complicated enough.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Automation, networking, and self-hosting&lt;/strong&gt;, no amount of CVEs can deter us from living on the edge.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Real-world troubleshooting&lt;/strong&gt;, including the bits that didn&apos;t work the first time (it&apos;s called wisdom kiddo)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The focus hasn&apos;t changed. I just break things and document them so you don&apos;t need to break your things. (swearing is optional)&lt;/p&gt;
&lt;h2&gt;What&apos;s Different Here?&lt;/h2&gt;
&lt;p&gt;For starters, &lt;strong&gt;this is my site&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;No Medium membership walls (never had any anyways) between you and an article. No platform deciding whether a technical deep dive deserves to appear in someone&apos;s feed. No need to optimise every headline for an algorithm having a midlife crisis. Just articles, code, diagrams, and occasional bad jokes and puns.&lt;/p&gt;
&lt;p&gt;The site is built with &lt;strong&gt;Astro&lt;/strong&gt;, and I&apos;ve gradually been moving my existing articles across so they can continue to be useful without depending on Medium.&lt;/p&gt;
&lt;p&gt;I&apos;m also giving the content a bit of a tidy-up or a rewrite as I go. Think of it as moving house, except instead of discovering boxes full of old cables, I&apos;m rediscovering YAML manifests and wondering &quot;what the heck was I thinking that day dammit!&quot;&lt;/p&gt;
&lt;h2&gt;To Everyone Who Followed Me on Medium&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Thanks, seriously.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;If you found your way over from Medium, welcome back. If you&apos;ve just stumbled across this place while searching for an obscure Kubernetes error message, welcome aboard.&lt;/p&gt;
&lt;p&gt;Pull up a chair. The coffee&apos;s probably gone cold, and something in the cluster is almost certainly restarting.&lt;/p&gt;
&lt;h2&gt;Same Content. Less Corporate Bullshit.&lt;/h2&gt;
&lt;p&gt;I&apos;m not trying to build a content empire.&lt;/p&gt;
&lt;p&gt;I&apos;m not interested in publishing five AI-generated articles a day about why the &quot;latest CNCF project will &lt;em&gt;change everything&lt;/em&gt;&quot;. Oh, CNCF Ambassadors and MVPs, I&apos;ll poke that particular wound another day. You really are the worst sometimes, you know?&lt;/p&gt;
&lt;p&gt;I just want somewhere to write about interesting technical problems, explain how I approached them, and share the occasional spectacular failure.&lt;/p&gt;
&lt;p&gt;If that helps someone build something, learn something, or avoid a particularly nasty rabbit hole, then the effort was worth it.&lt;/p&gt;
&lt;p&gt;So that&apos;s the story. Medium was an interesting side-quest. But I&apos;m packing my shit and going home.&lt;/p&gt;
&lt;p&gt;Welcome to &lt;strong&gt;The Backyard Rack&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;Now, if you&apos;ll excuse me, I&apos;ve got a cluster that&apos;s been running suspiciously quietly for days...&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;— Kyr&lt;/strong&gt;&lt;/p&gt;</content:encoded><h:img src="/_astro/thumbnail.CN2cYG2M.png"/><enclosure url="/_astro/thumbnail.CN2cYG2M.png"/></item><item><title>Deploy OpenStack on Kubernetes in under 60 minutes</title><link>https://www.backyard-rack.com/blog/deploy-openstack-on-kubernetes</link><guid isPermaLink="true">https://www.backyard-rack.com/blog/deploy-openstack-on-kubernetes</guid><description>Build a Kubernetes-native OpenStack cloud in under an hour using Proxmox, K3s, Ceph and OpenStack-Helm.</description><pubDate>Wed, 10 Jun 2026 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;Build a Kubernetes-native OpenStack cloud in under an hour using Proxmox, K3s, Ceph and OpenStack-Helm.&lt;/p&gt;
&lt;p&gt;Instead of manually assembling every &lt;a href=&quot;https://www.openstack.org/&quot;&gt;OpenStack&lt;/a&gt; moving part, we use a bunch of automation scripts to built around &lt;a href=&quot;https://k3s.io/&quot;&gt;K3s&lt;/a&gt;, &lt;a href=&quot;https://rook.io/&quot;&gt;Rook-Ceph&lt;/a&gt;, and &gt; &lt;a href=&quot;https://github.com/openstack/openstack-helm&quot;&gt;OpenStack-Helm&lt;/a&gt; in order to deploy a fully functional OpenStack environment on a three-node Kubernetes cluster. Starting from a handful of virtual machines, we will bootstrap Kubernetes, provision distributed storage, prepare the networking stack, and deploy the OpenStack control plane and compute services with a series of repeatable, idempotent, scripted steps.&lt;/p&gt;
&lt;p&gt;The result is not a toy deployment (not a production deployment though) or a collection of isolated services. By the end of the process, you will have a working OpenStack cloud with Keystone, Glance, Nova, Neutron, Cinder, Heat, Horizon, and Skyline running on Kubernetes, backed by Ceph storage and ready to launch networks &amp;#x26; virtual machines for your homelab or your OpenStack development environment.&lt;/p&gt;
&lt;h2&gt;Prerequisites&lt;/h2&gt;
&lt;p&gt;Before we begin, it is worth noting that &lt;strong&gt;this deployment is intentionally and inevitably opinionated, due to nature and complexity of the objective&lt;/strong&gt;. Deploying OpenStack on Kubernetes involves a considerable number of moving parts, and the choices made throughout this guide reflect a specific approach that prioritizes simplicity, repeatability, and automation. That said, &lt;strong&gt;the underlying concepts are not tied to a particular environment&lt;/strong&gt;. While the examples use Proxmox and OPNsense, the same deployment can be adapted to other virtualization and networking platforms, provided that the required infrastructure, networking, and storage and load-balancing capabilities are provided.&lt;/p&gt;
&lt;p&gt;Throughout this article, it’s assumed that a virtualization and networking foundation is already in place. For that matter, &lt;a href=&quot;https://www.proxmox.com/&quot;&gt;Proxmox&lt;/a&gt; provides the compute layer used to host the Kubernetes nodes, while &lt;a href=&quot;https://opnsense.org/&quot;&gt;OPNsense&lt;/a&gt; is responsible for network services such as DNS, load balancing, and virtual IP management. These components are treated as existing infrastructure and their installation and configuration are therefore &lt;strong&gt;outside the scope&lt;/strong&gt; of this article.&lt;/p&gt;
&lt;p&gt;So this is what you are going to need:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;a &lt;strong&gt;Proxmox&lt;/strong&gt; cluster (&gt;= 8.4.2) with enough capacity to host 3 additional VMs (6 vCPUs, 16GB RAM, 2 NICs, 2 iSCSI disks) that will serve as Kubernetes nodes&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;an &lt;strong&gt;OPNSense&lt;/strong&gt; instance (≥= 25.1.12) supporting the SDN/Zone/VNet functionality of the Proxmox cluster and its VMs or LXCs&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;an SDN VXLAN &lt;strong&gt;Zone&lt;/strong&gt; and a tagged VLAN-Aware &lt;strong&gt;VNet&lt;/strong&gt; in Proxmox. Throughout the remainder of this article, these resources will be referred to as &lt;code&gt;ixzone&lt;/code&gt; and &lt;code&gt;ixvnet&lt;/code&gt;, respectively.
&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-01.DCbPydMR_KKSzo.webp&quot; alt=&quot;Creating the Proxmox SDN VXLAN zone (ixzone). The VXLAN zone is configured with an MTU of 1450 to accommodate encapsulation overhead and serves as the network foundation for tenant connectivity within the OpenStack environment.&quot;&gt;&lt;/p&gt;
&lt;p&gt;Creating the Proxmox SDN VXLAN zone (&lt;strong&gt;ixzone&lt;/strong&gt;). The VXLAN zone is configured with an MTU of &lt;strong&gt;1450&lt;/strong&gt; to accommodate encapsulation overhead and serves as the network foundation for tenant connectivity within the OpenStack environment.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;two interfaces&lt;/strong&gt; are configured in OPNsense. The first is attached to the untagged network and serves as the Kubernetes management network. The second is attached to VLAN 70 and provides connectivity for OpenStack external networks, including floating IP traffic. The configuration is shown in the screenshots below:&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-02.DHVReIM4_Z2e88iO.webp&quot; alt=&quot;OPNsense interface assignments used by the OpenStack deployment. The untagged physical interface vtnet1 (1) is assigned as the Kubernetes management network, while the VLAN interface vlan0.70 (2), created on top of vtnet1 and configured with VLAN tag 70 (3), is dedicated to OpenStack external networking and floating IP traffic.&quot;&gt;&lt;/p&gt;
&lt;p&gt;OPNsense interface assignments used by the OpenStack deployment. The untagged physical interface &lt;strong&gt;vtnet1&lt;/strong&gt; (1) is assigned as the Kubernetes management network, while the VLAN interface &lt;strong&gt;vlan0.70&lt;/strong&gt; (2), created on top of &lt;strong&gt;vtnet1&lt;/strong&gt; and configured with VLAN tag &lt;strong&gt;70&lt;/strong&gt; (3), is dedicated to OpenStack external networking and floating IP traffic.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-03.DG3SWxNJ_2ecQ3F.webp&quot; alt=&quot;VLAN configuration in OPNsense showing the vlan0.70 interface attached to the vtnet1 parent interface and configured with VLAN tag 70. This VLAN provides the network segment used by OpenStack external networking, enabling connectivity for provider networks and floating IP traffic.&quot;&gt;&lt;/p&gt;
&lt;p&gt;VLAN configuration in OPNsense showing the &lt;strong&gt;vlan0.70&lt;/strong&gt; interface attached to the &lt;strong&gt;vtnet1&lt;/strong&gt; parent interface and configured with VLAN tag &lt;strong&gt;70&lt;/strong&gt;. This VLAN provides the network segment used by OpenStack external networking, enabling connectivity for provider networks and floating IP traffic.&lt;/p&gt;
&lt;p&gt;Before proceeding, clone the deployment repository to the node (the one destined to be the master node of your K3s cluster) that will be used to orchestrate the installation. All automation scripts, configuration templates, and deployment assets referenced throughout this article are contained in the repository:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;git clone https://github.com/akyriako/osh.git
cd osh
&lt;/code&gt;&lt;/pre&gt;
&lt;blockquote&gt;
&lt;p&gt;Unless stated otherwise, all commands shown in the remainder of this guide are executed from within the cloned repository directory.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h2&gt;Architecture&lt;/h2&gt;
&lt;p&gt;At the end of this deployment, we will have a fully functional OpenStack cloud running entirely on top of a three-node &lt;a href=&quot;https://k3s.io/&quot;&gt;K3s&lt;/a&gt; cluster.&lt;/p&gt;
&lt;p&gt;The Kubernetes layer forms the foundation of the platform. A single control-plane node and two worker nodes host all OpenStack services as Kubernetes workloads, deployed through OpenStack-Helm. Rather than dedicating individual virtual or physical machines to specific OpenStack roles, the control plane, networking services, storage services, and compute components are distributed across the cluster and managed through Kubernetes scheduling.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-04.ZO2IQ8f5_Z18IoSY.webp&quot; alt=&quot;Alt text: Architecture diagram showing a three-node K3s cluster hosting OpenStack-Helm services. OPNsense with HAProxy sits at the network edge and distributes traffic to NGINX and Traefik ingress controllers running on the Kubernetes nodes. The cluster hosts OpenStack services including Keystone, Glance, Horizon, Skyline, Heat, Nova, Neutron, and Placement, together with supporting infrastructure such as MariaDB, RabbitMQ, Memcached, Open vSwitch, and libvirt. Persistent storage is provide&quot;&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Target architecture&lt;/strong&gt;: a three-node K3s cluster running containerized OpenStack services, backed by a distributed Ceph storage cluster and exposed through OPNsense-managed ingress, DNS, and load balancing.&lt;/p&gt;
&lt;p&gt;Persistent storage is provided by a Rook-managed Ceph cluster running on the same nodes. Each virtual machine contributes local storage devices to Ceph, creating a shared storage backend that is consumed both by Kubernetes and OpenStack services such as Cinder and Glance. This removes the need for an external storage appliance while still providing a distributed and resilient storage layer.&lt;/p&gt;
&lt;p&gt;At the edge of the environment, OPNsense acts as the networking entry point. &lt;a href=&quot;https://github.com/haproxy/haproxy&quot;&gt;HAProxy&lt;/a&gt; distributes incoming traffic across the Kubernetes cluster, while DNS records resolve OpenStack service endpoints to virtual IP addresses managed by &lt;a href=&quot;https://docs.opnsense.org/manual/unbound.html&quot;&gt;OPNsense Unbound DNS&lt;/a&gt; service.&lt;/p&gt;
&lt;p&gt;Two ingress paths are used within the cluster. NGINX Ingress exposes public OpenStack endpoints such as Horizon, Skyline, and Keystone, while Traefik (ingress controller that comes bundled with K3s) is retained for management interfaces and supporting services. This separation keeps administrative traffic distinct from the OpenStack APIs consumed by users and applications.&lt;/p&gt;
&lt;p&gt;The OpenStack control plane itself consists of &lt;a href=&quot;https://www.openstack.org/software/releases/dalmatian/components/keystone&quot;&gt;Keystone&lt;/a&gt; for identity management, &lt;a href=&quot;https://www.openstack.org/software/releases/dalmatian/components/glance&quot;&gt;Glance&lt;/a&gt; for image services, &lt;a href=&quot;https://www.openstack.org/software/releases/dalmatian/components/cinder&quot;&gt;Cinder&lt;/a&gt; for block storage, &lt;a href=&quot;https://www.openstack.org/software/releases/dalmatian/components/heat&quot;&gt;Heat&lt;/a&gt; for orchestration, &lt;a href=&quot;https://www.openstack.org/software/releases/dalmatian/components/horizon&quot;&gt;Horizon&lt;/a&gt; and &lt;a href=&quot;https://www.openstack.org/software/releases/dalmatian/components/skyline&quot;&gt;Skyline&lt;/a&gt; as UI layer, and the &lt;a href=&quot;https://www.openstack.org/software/releases/dalmatian/components/nova&quot;&gt;Nova&lt;/a&gt;, &lt;a href=&quot;https://www.openstack.org/software/releases/dalmatian/components/neutron&quot;&gt;Neutron&lt;/a&gt;, and &lt;a href=&quot;https://www.openstack.org/software/releases/dalmatian/components/placement&quot;&gt;Placement&lt;/a&gt; services required to provision and operate virtual machines. Open vSwitch provides the networking foundation for tenant connectivity, while libvirt and KVM enable Nova virtual machine instances to run directly on the Kubernetes nodes.&lt;/p&gt;
&lt;h2&gt;Prepare the Proxmox Virtual Machines&lt;/h2&gt;
&lt;p&gt;Each node has to be provisioned with &lt;strong&gt;two virtual network interfaces&lt;/strong&gt;. The first interface is attached to the VXLAN-backed management network (&lt;code&gt;ixvnet&lt;/code&gt;) and is used for Kubernetes, Ceph, and OpenStack internal communication. The second interface is connected to the same virtual network but carries VLAN tag &lt;code&gt;70&lt;/code&gt;, providing access to the external provider network that Neutron later exposes as the &lt;code&gt;public&lt;/code&gt; physical network.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-05.CGp20kVb_jwkkU.webp&quot; alt=&quot;Proxmox Virtual Environment hardware configuration screen showing a Kubernetes worker node. The virtual machine is configured with 14 GB of memory, 6 vCPUs, two virtual disks, and two network interfaces. One network interface is connected to the management network, while the second is attached to VLAN 70 for external provider network connectivity. The secondary disk is reserved for Ceph OSD storage.&quot;&gt;&lt;/p&gt;
&lt;p&gt;Proxmox virtual machine configuration used for the Kubernetes nodes that host OpenStack.&lt;/p&gt;
&lt;p&gt;The storage configuration includes &lt;strong&gt;two virtual disks&lt;/strong&gt;. The primary disk hosts the operating system and Kubernetes workloads, while the secondary disk is dedicated exclusively to Ceph. During the Ceph installation phase, this disk is discovered as &lt;code&gt;/dev/sdb&lt;/code&gt; and consumed by Rook to create an Object Storage Daemon (OSD), contributing storage capacity to the distributed Ceph cluster.&lt;/p&gt;
&lt;h2&gt;Install K3s Control Plane and Workers&lt;/h2&gt;
&lt;p&gt;The first step is to establish the Kubernetes cluster that will host the entire OpenStack deployment. Since &lt;a href=&quot;https://github.com/openstack/openstack-helm&quot;&gt;OpenStack-Helm&lt;/a&gt; packages OpenStack services as Helm charts, a functioning Kubernetes environment must be in place before any OpenStack components can be deployed.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;install-k3s.sh&lt;/code&gt; script bootstraps the cluster by installing K3s on the control-plane node and preparing it for the addition of worker nodes. During this phase, it installs and configures K3s, prepares cluster access, installs the tooling required throughout the remainder of the deployment, and exposes the default Traefik ingress controller (ingress for non-OpenStack workloads) on &lt;code&gt;HTTP/8080&lt;/code&gt; and &lt;code&gt;HTTPS/8443&lt;/code&gt;.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;K3s was chosen because it provides a fully conformant Kubernetes distribution while keeping the operational footprint small, making it well suited for lab environments and proof-of-concept deployments.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;sudo ./scripts/install-k3s.sh
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Once the control-plane installation is complete, the script generates a ready-to-use worker join-command. This command contains the information required for additional &lt;strong&gt;worker&lt;/strong&gt; nodes to securely join the cluster and &lt;strong&gt;must be executed on each worker node&lt;/strong&gt;.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;After running the join-command on each worker node and confirming that all nodes are part of the cluster, only then you are clear to move on to the next phase of the deployment.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h2&gt;Post-Installation Steps for K3s Nodes&lt;/h2&gt;
&lt;p&gt;With the Kubernetes cluster in place, there is one more preparatory step before we start deploying storage and OpenStack services. While K3s can run perfectly well with its default operating system settings, OpenStack and Ceph place considerably higher demands on the underlying nodes. As the number of pods, files, watches, and network connections grows, the default kernel and system limits that are perfectly acceptable for a small Kubernetes cluster can become restrictive for a cloud platform.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;post-k3s.sh&lt;/code&gt; script prepares each node for the workload that will follow. It applies a number of operating system and Kubernetes tuning parameters, including disabling swap and increasing file descriptor and inotify limits. These changes help ensure that Kubernetes, containerd, Ceph, and the OpenStack services have sufficient resources available as the environment grows.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;sudo ./scripts/post-k3s.sh
&lt;/code&gt;&lt;/pre&gt;
&lt;blockquote&gt;
&lt;p&gt;Run this script &lt;strong&gt;on every node&lt;/strong&gt; in the cluster, including both the control-plane and worker nodes. Although the changes are relatively small, they establish a more suitable baseline for the storage and virtualization workloads that will be introduced in the next phases of the deployment.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h2&gt;Prepare OPNsense HAProxy and Unbound DNS&lt;/h2&gt;
&lt;p&gt;Before deploying storage and OpenStack services, we need to establish how users and applications will reach them. In a public cloud providers, this responsibility is typically handled by managed load balancers and DNS services. In our environment, those capabilities are provided by OPNsense through a combination of &lt;a href=&quot;https://docs.opnsense.org/manual/firewall_vip.html&quot;&gt;Virtual IP addresses&lt;/a&gt;, &lt;a href=&quot;https://docs.opnsense.org/manual/unbound.html#overrides&quot;&gt;DNS records&lt;/a&gt;, and HAProxy listeners.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-06.BkCfjJ2T_1KIsHx.webp&quot; alt=&quot;OpenStack on Kubernetes article illustration&quot;&gt;&lt;/p&gt;
&lt;p&gt;1️⃣ The first step is to create two Virtual IP addresses that will act as stable entry points into the Kubernetes cluster. One is reserved for the NGINX ingress controller that will later expose the public OpenStack endpoints, while the other is used by Traefik, which provides access to cluster management and supporting services.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-07.D_NWX4Sy_Z1UjlDu.webp&quot; alt=&quot;OpenStack on Kubernetes article illustration&quot;&gt;&lt;/p&gt;
&lt;p&gt;2️⃣ Unbound DNS overrides are then configured so that requests for the OpenStack and management domains resolve to the appropriate Virtual IPs. This gives us predictable hostnames from the very beginning of the deployment and avoids having to track individual Kubernetes node addresses as services move around the cluster.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-08.Dlo5H3fa_bhfIs.webp&quot; alt=&quot;OpenStack on Kubernetes article illustration&quot;&gt;&lt;/p&gt;
&lt;p&gt;3️⃣ Finally, HAProxy has to be configured to distribute incoming traffic across all Kubernetes nodes. Rather than directing requests to a specific machine, HAProxy forwards them to whichever node is currently capable of serving the requested ingress traffic. This creates a single, consistent access layer in front of the cluster and allows the ingress controllers to remain highly available even if individual nodes become unavailable.&lt;/p&gt;
&lt;p&gt;First we need to create the &lt;strong&gt;Real Servers&lt;/strong&gt;:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Navigate to &lt;strong&gt;Services → HAProxy&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Open the &lt;strong&gt;Real Servers&lt;/strong&gt; configuration page.&lt;/li&gt;
&lt;li&gt;Create a total of &lt;strong&gt;12 server definitions&lt;/strong&gt;, 4 for each Kubernetes node.&lt;/li&gt;
&lt;li&gt;The first group represents the NGINX ingress endpoints (&lt;code&gt;80&lt;/code&gt; and &lt;code&gt;443&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;The second group represents the Traefik ingress endpoints (&lt;code&gt;8080&lt;/code&gt; and &lt;code&gt;8443&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;Apply the configuration before proceeding with the creation of backend pools and public listeners.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-09.CNO5VX2Y_Z13pLOE.webp&quot; alt=&quot;OPNsense HAProxy configuration screen showing the definition of real servers for a three-node K3s cluster. Each Kubernetes node is registered four times to represent the NGINX ingress endpoints on ports 80 and 443 and the Traefik ingress endpoints on ports 8080 and 8443. These server entries are later used to build backend pools and public listeners for OpenStack and cluster management traffic.&quot;&gt;&lt;/p&gt;
&lt;p&gt;Creating the HAProxy real server definitions used to forward ingress traffic to the Kubernetes cluster. Each node is registered multiple times, once for every ingress endpoint exposed by the cluster: HTTP (80) and HTTPS (443) for the NGINX ingress controller, and HTTP (8080) and HTTPS (8443) for Traefik. These backend definitions are later grouped into HAProxy pools and exposed through dedicated virtual IP addresses.&lt;/p&gt;
&lt;p&gt;Next we need to create the &lt;strong&gt;Backend Pools&lt;/strong&gt;:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Navigate to &lt;strong&gt;Services → HAProxy&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Open &lt;strong&gt;Virtual Services → Backend Pools&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Create a total of &lt;strong&gt;4 backend pools&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Add the corresponding server entries for all three Kubernetes nodes to each pool (the ones we created in the previous step).&lt;/li&gt;
&lt;li&gt;The resulting pools represent the NGINX (&lt;code&gt;80&lt;/code&gt;, &lt;code&gt;443&lt;/code&gt;) and Traefik (&lt;code&gt;8080&lt;/code&gt;, &lt;code&gt;8443&lt;/code&gt;) ingress endpoints.&lt;/li&gt;
&lt;li&gt;Apply the configuration before proceeding with the creation of public services.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-10.Dzrr3z75_Z1W3g2x.webp&quot; alt=&quot;OPNsense HAProxy configuration screen showing the creation of backend pools. Four pools are configured to group the NGINX ingress endpoints on ports 80 and 443 and the Traefik ingress endpoints on ports 8080 and 8443. Each pool contains the corresponding ingress servers from all three Kubernetes nodes, enabling HAProxy to load-balance traffic across the cluster.&quot;&gt;&lt;/p&gt;
&lt;p&gt;Creating the HAProxy backend pools that aggregate the ingress endpoints of all Kubernetes nodes. Four pools are defined: two for the NGINX ingress controller (&lt;code&gt;80&lt;/code&gt; and &lt;code&gt;443&lt;/code&gt;) and two for Traefik (&lt;code&gt;8080&lt;/code&gt; and &lt;code&gt;8443&lt;/code&gt;). Each pool contains the corresponding server definitions from every cluster node, allowing HAProxy to distribute traffic across the cluster rather than targeting a single host.&lt;/p&gt;
&lt;p&gt;Last we need to create the &lt;strong&gt;Public Services&lt;/strong&gt;:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Navigate to &lt;strong&gt;Services → HAProxy&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Open &lt;strong&gt;Virtual Services → Public Services&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Configure the listener to bind to the appropriate virtual IP address and port.&lt;/li&gt;
&lt;li&gt;Associate the listener with the corresponding backend pool.&lt;/li&gt;
&lt;li&gt;Create a total of &lt;strong&gt;4 public services&lt;/strong&gt;: two for the NGINX ingress VIP and two for the Traefik ingress VIP.&lt;/li&gt;
&lt;li&gt;Apply the configuration to activate the listeners.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-11.BUg3_jLj_KkV9X.webp&quot; alt=&quot;OPNsense HAProxy configuration screen showing the creation of public services. Each listener is bound to a virtual IP address and forwards traffic to a corresponding backend pool. Four listeners are configured in total, exposing the NGINX ingress controller on ports 80 and 443 and the Traefik ingress controller on ports 80 and 443 through separate virtual IP addresses.&quot;&gt;&lt;/p&gt;
&lt;p&gt;Creating the HAProxy public services that expose the Kubernetes ingress controllers through dedicated virtual IP addresses. Each public service binds a virtual IP and port combination to one of the backend pools created previously, providing a stable entry point for traffic entering the cluster. Two listeners are configured for the NGINX ingress controller (&lt;code&gt;80&lt;/code&gt; and &lt;code&gt;443&lt;/code&gt;) and two for Traefik (&lt;code&gt;80&lt;/code&gt; and &lt;code&gt;443&lt;/code&gt;), each mapped to their respective backend pools.&lt;/p&gt;
&lt;h2&gt;Install Ceph&lt;/h2&gt;
&lt;p&gt;With Kubernetes deployed and external access to the cluster established through HAProxy, DNS, and ingress controllers, the next component we need is storage. OpenStack services generate and consume persistent data from the very beginning of the deployment, whether it is virtual machine images, block volumes, databases, or service metadata. Rather than relying on local node storage, we will build a distributed storage layer that is shared across the entire cluster.&lt;/p&gt;
&lt;p&gt;To achieve this, the &lt;code&gt;install-ceph.sh&lt;/code&gt; script deploys a Rook-managed Ceph cluster on Kubernetes. Each node (Proxmox virtual machine) contributes a dedicated storage device, exposed to the virtual machine as a raw disk and attached as &lt;code&gt;/dev/sdb&lt;/code&gt;. These disks are used &lt;strong&gt;exclusively&lt;/strong&gt; by Ceph to create Object Storage Daemons (OSDs), which form the foundation of the distributed storage cluster.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;sudo ./scripts/install-ceph.sh
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;That will create:&lt;/p&gt;
&lt;p&gt;✅ a Ceph deployment (1 manager, 3 monitors, 3 OSDs on &lt;code&gt;/dev/sdb&lt;/code&gt; on each VM)&lt;br&gt;
✅ a&lt;code&gt;CephCluster&lt;/code&gt; named &lt;code&gt;ceph&lt;/code&gt;, and a &lt;code&gt;CephBlockPool&lt;/code&gt; named &lt;code&gt;rdb&lt;/code&gt;&lt;br&gt;
✅ a&lt;code&gt;StorageClass&lt;/code&gt; named &lt;code&gt;general&lt;/code&gt; based on provisioner &lt;code&gt;rook-ceph.rbd.csi.ceph.com&lt;/code&gt;&lt;br&gt;
✅ the Ceph management web interface (via Traefik ingress)&lt;/p&gt;
&lt;p&gt;Rook takes care of deploying and managing the Ceph components inside Kubernetes, while Ceph itself provides the resilient block storage backend consumed by both Kubernetes and OpenStack. As part of the installation, the script creates the Ceph cluster, provisions the OSDs on the dedicated disks, creates an RBD pool, and exposes a Kubernetes &lt;code&gt;StorageClass&lt;/code&gt; that will later be used by OpenStack-Helm itself.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-12.DmbHx0zB_Z23FNpH.webp&quot; alt=&quot;OpenStack on Kubernetes article illustration&quot;&gt;&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;At this stage, we are not yet deploying any OpenStack services. Instead, we are establishing one of the most critical building blocks of the platform: &lt;strong&gt;a shared storage layer that spans all nodes in the cluster&lt;/strong&gt;. By the time the installation completes, the Kubernetes environment has gained a distributed storage backend capable of supporting Glance image storage, Cinder volumes, and the persistent data requirements of the OpenStack control plane.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;The Ceph dashboard is also exposed through the management ingress, allowing you to verify the health of the cluster before proceeding with the OpenStack deployment.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-13.D-EHhLNs_1y61SK.webp&quot; alt=&quot;OpenStack on Kubernetes article illustration&quot;&gt;&lt;/p&gt;
&lt;h2&gt;Install OpenStack-Helm Dependencies&lt;/h2&gt;
&lt;p&gt;At this point, we have a functioning Kubernetes cluster, a distributed Ceph storage backend, and a network layer capable of exposing services outside the cluster. What we do not have yet is an environment that OpenStack can actually run on.&lt;/p&gt;
&lt;p&gt;OpenStack-Helm is a bit more than a collection of Helm charts. It expects Kubernetes to be organized in a particular way, with dedicated ingress paths, storage integrations, scheduling rules, and supporting components already in place. Before we can deploy a single OpenStack service, we need to prepare the cluster for that matter.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;install-osh-deps.sh&lt;/code&gt; script prepares the environment expected by the OpenStack-Helm charts. This step creates the required namespaces, pulls the OpenStack-Helm repositories, and installs the Rook-Ceph adapter so that OpenStack-Helm can consume the Ceph cluster created in the previous step. It also deploys thededicated NGINX ingress controller for the public OpenStack endpoints. This keeps OpenStack API and dashboard traffic separate from the management services exposed through Traefik . Later, services such as Keystone, Horizon, and Skyline will use this ingress path.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;sudo ./scripts/install-osh-deps.sh
&lt;/code&gt;&lt;/pre&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;em&gt;K3s ships with&lt;/em&gt; &lt;a href=&quot;https://docs.k3s.io/networking/networking-services#traefik-ingress-controller&quot;&gt;&lt;em&gt;Traefik&lt;/em&gt;&lt;/a&gt; &lt;em&gt;as its default ingress controller. Earlier in the deployment, we reconfigured Traefik to listen on ports&lt;/em&gt; &lt;code&gt;8080&lt;/code&gt; &lt;em&gt;and&lt;/em&gt; &lt;code&gt;8443&lt;/code&gt;&lt;em&gt;, reserving the standard&lt;/em&gt; &lt;code&gt;80&lt;/code&gt; &lt;em&gt;and&lt;/em&gt; &lt;code&gt;443&lt;/code&gt; &lt;em&gt;ports for the dedicated NGINX ingress controller that will expose the OpenStack APIs and dashboards.&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Another important part of this step is node preparation. OpenStack-Helm relies on specific node labels and scheduling expectations, so the script adjusts labels and taints to make the K3s nodes suitable for OpenStack workloads. Without this alignment, charts may install successfully but fail to schedule the required pods.&lt;/p&gt;
&lt;p&gt;Additionally it will deploy &lt;a href=&quot;https://github.com/kite-org/kite&quot;&gt;Kite&lt;/a&gt;, a modern lightweight Kubernetes web dashboard, and expose it via Traefik.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-14.DX6d8s2m_1py9g5.webp&quot; alt=&quot;OpenStack on Kubernetes article illustration&quot;&gt;&lt;/p&gt;
&lt;h2&gt;Install OpenStack Infrastructure Components&lt;/h2&gt;
&lt;p&gt;Now that Kubernetes has been prepared for OpenStack-Helm, we can start deploying the first OpenStack layer. This is not yet the cloud that users will interact with. There is no dashboard, no identity service, no images, and no virtual machines. What we are deploying here is the internal machinery that the rest of OpenStack depends on.&lt;/p&gt;
&lt;p&gt;The infrastructure phase brings up the shared services used by the OpenStack control plane. &lt;a href=&quot;https://mariadb.org/&quot;&gt;MariaDB&lt;/a&gt; provides the database backend, &lt;a href=&quot;https://www.rabbitmq.com/&quot;&gt;RabbitMQ&lt;/a&gt; handles service-to-service messaging, &lt;a href=&quot;https://www.memcached.org/&quot;&gt;Memcached&lt;/a&gt; supports caching, and &lt;a href=&quot;https://www.openvswitch.org/&quot;&gt;Open vSwitch&lt;/a&gt; lays the groundwork for the networking stack that Neutron will use later.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;sudo ./scripts/install-osh.sh --infra
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;In the next phase we will focus on the actual control plane: identity, image management, dashboards, and the services users will recognize as OpenStack.&lt;/p&gt;
&lt;h2&gt;Install OpenStack Core Components&lt;/h2&gt;
&lt;p&gt;With the underlying infrastructure now in place, we can begin deploying the OpenStack control plane.&lt;/p&gt;
&lt;p&gt;1️⃣ This phase introduces the first services that users will directly interact with and marks the transition from platform preparation to cloud deployment. While the previous steps focused on Kubernetes, storage, networking, and supporting infrastructure, this stage establishes the foundations of an operational OpenStack environment.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;sudo ./scripts/install-osh.sh --core
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The deployment begins with &lt;strong&gt;Keystone&lt;/strong&gt;, which serves as the identity service for the entire platform. Every authentication request, service registration, and API interaction ultimately passes through Keystone, making it the central point of trust within the cloud.&lt;/p&gt;
&lt;p&gt;Next comes &lt;strong&gt;Glance&lt;/strong&gt;, the image service responsible for storing and distributing virtual machine images. Whether launching a Linux distribution, a network appliance, or a custom workload, Glance becomes the catalog from which compute instances obtain their boot images.&lt;/p&gt;
&lt;p&gt;To provide a user-facing experience, &lt;strong&gt;Horizon&lt;/strong&gt; is deployed as the traditional OpenStack dashboard. Alongside it, we can optionally deploy &lt;strong&gt;Skyline&lt;/strong&gt; which offers a more modern web interface built around OpenStack APIs, giving users an alternative way to interact with cloud resources. Both dashboards are exposed through the dedicated NGINX ingress layer established earlier, making them accessible through the public OpenStack endpoints.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-15.Bz5vtWKy_ZAvAfX.webp&quot; alt=&quot;OpenStack Horizon login page displayed in a web browser. The interface contains fields for username, password, and domain selection, along with the OpenStack logo and sign-in button. Horizon serves as the traditional web dashboard for managing OpenStack resources and authenticates users through Keystone.&quot;&gt;&lt;/p&gt;
&lt;p&gt;Horizon is the traditional OpenStack dashboard and provides a web-based interface for managing cloud resources, including instances, networks, images, volumes, and projects.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;At this stage, our cloud is &lt;strong&gt;not yet&lt;/strong&gt; capable of launching virtual machines. The compute and networking services that make this possible are introduced later. What we laid here is the operational foundations of OpenStack: identity management, image management, and user access.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;The remaining phases will build upon this foundation by adding storage, orchestration, networking, and ultimately the ability to provision and operate virtual machines.&lt;/p&gt;
&lt;p&gt;2️⃣ Although the Kubernetes platform already has access to distributed storage through Ceph, OpenStack still needs a way to expose that storage to its users. This is the role of Cinder.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Cinder&lt;/strong&gt; introduces block storage as a first-class OpenStack service, allowing users and applications to provision persistent volumes independently of the virtual machines that consume them. These volumes can be attached, detached, resized, and managed throughout their lifecycle without affecting the underlying instances.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;sudo ./scripts/install-osh.sh --cinder
&lt;/code&gt;&lt;/pre&gt;
&lt;blockquote&gt;
&lt;p&gt;In this deployment, Cinder is backed by the Ceph cluster that was created earlier. This means that block volumes are not tied to a particular Kubernetes node or compute host. Instead, they are stored on the distributed Ceph backend, providing the resilience and flexibility expected from a real cloud storage service.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;blockquote&gt;
&lt;p&gt;While users may not immediately interact with Cinder when exploring the dashboards, many workloads depend on it. Databases, stateful applications, and long-lived virtual machines often require storage that persists beyond the lifetime of a single instance. Cinder provides that capability and completes another essential building block of the cloud platform.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;3️⃣ The final component introduced during this phase is Heat, OpenStack’s native orchestration service.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Heat&lt;/strong&gt; allows collections of OpenStack resources to be defined as templates and deployed as a single stack. Rather than creating networks, security groups, volumes, floating IPs, and compute instances individually, Heat coordinates the creation of these resources by interacting with the corresponding OpenStack APIs in the correct order.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;sudo ./scripts/install-osh.sh --heat
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;A Heat template can describe both the resources themselves and the relationships between them. For example, a template can specify that a volume must be attached to a server, or that a floating IP must be associated with a particular instance. Heat then orchestrates the deployment and lifecycle of the entire stack as a single unit.&lt;/p&gt;
&lt;p&gt;Although Heat is not required to operate the cloud, it is an integral part of the OpenStack ecosystem and provides the orchestration capabilities expected from a complete OpenStack deployment. By installing it at this stage, we ensure that the environment includes OpenStack’s native mechanism for template-driven resource provisioning and lifecycle management.&lt;/p&gt;
&lt;p&gt;4️⃣ The final service deployed, which is optional as we already have installed Horizon, is Skyline, OpenStack’s next-generation web dashboard.&lt;/p&gt;
&lt;p&gt;For many years, Horizon has been the primary graphical interface for OpenStack. &lt;strong&gt;Skyline&lt;/strong&gt; was introduced to provide a more modern user experience, built around OpenStack APIs and contemporary web technologies while exposing many of the same cloud management capabilities.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;sudo ./scripts/install-osh.sh --skyline
&lt;/code&gt;&lt;/pre&gt;
&lt;blockquote&gt;
&lt;p&gt;Skyline does not replace any core OpenStack functionality. Instead, it provides an alternative way for administrators and users to interact with the platform, complementing the traditional Horizon dashboard already deployed in the previous step.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-16.BJf4_hEH_pY9XF.webp&quot; alt=&quot;Skyline login page displayed through a web browser. The interface shows the OpenStack region selector, domain selection, username and password fields, and a custom cloud branding theme. The page is served through the OpenStack ingress layer and authenticates users against Keystone.&quot;&gt;&lt;/p&gt;
&lt;p&gt;Skyline provides a modern web interface for interacting with OpenStack services and complements the traditional Horizon dashboard deployed earlier in the guide.&lt;/p&gt;
&lt;p&gt;By exposing both Horizon and Skyline, the environment offers access to both the established OpenStack interface and its newer counterpart, allowing users to choose the experience that best fits their operational preferences.&lt;/p&gt;
&lt;h2&gt;Install Libvirt&lt;/h2&gt;
&lt;p&gt;Up to this point, every component we have deployed has been focused on supporting the OpenStack control plane. This step shifts our attention to the compute layer.&lt;/p&gt;
&lt;p&gt;OpenStack ultimately exists to provision and manage virtual machines, and those virtual machines need a hypervisor. In this deployment, that role is provided by &lt;a href=&quot;https://linux-kvm.org/page/Main_Page&quot;&gt;KVM&lt;/a&gt; and &lt;a href=&quot;https://libvirt.org/&quot;&gt;libvirt&lt;/a&gt; running directly on the Kubernetes nodes.&lt;/p&gt;
&lt;p&gt;The purpose of this step is to prepare the cluster for virtualization workloads &lt;strong&gt;before&lt;/strong&gt; the Nova compute services are introduced. Because the deployment runs on Kubernetes, libvirt is not installed in the traditional manner on dedicated hosts. Instead, it is deployed as a Kubernetes workload with the permissions and host access required to interact with the underlying KVM capabilities exposed by each node.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;sudo ./scripts/install-osh.sh --libvirt
&lt;/code&gt;&lt;/pre&gt;
&lt;blockquote&gt;
&lt;p&gt;It is important to note at this point, that t&lt;strong&gt;he libvirt pods do not immediately transition to a&lt;/strong&gt; &lt;code&gt;Running&lt;/code&gt; &lt;strong&gt;state&lt;/strong&gt;. &lt;em&gt;This is expected&lt;/em&gt;. At this stage, part of the networking infrastructure required by the compute stack is not yet available. Once Neutron and the remaining compute services are deployed in the next phase, libvirt completes its initialization and becomes fully operational.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;By the end of this step, the cluster has everything required to host virtual machines from a hypervisor perspective. The final piece missing is the OpenStack compute stack itself, which will connect Nova, Neutron, Placement, and libvirt into a complete virtualization platform.&lt;/p&gt;
&lt;h2&gt;Install OpenStack Compute Kit&lt;/h2&gt;
&lt;p&gt;This is the final major step of the deployment and the point at which our OpenStack cloud becomes fully operational.&lt;/p&gt;
&lt;p&gt;Up to now, we have assembled all the supporting layers required by the platform: Kubernetes provides orchestration, Ceph provides storage, the control plane services handle identity and image management, and libvirt has prepared the nodes for virtualization. What remains is to connect these pieces together and introduce the services responsible for scheduling, networking, and running virtual machines.&lt;/p&gt;
&lt;p&gt;The compute kit deploys three key OpenStack services: Placement, Nova, and Neutron.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;sudo ./scripts/install-osh.sh --compute
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;Placement&lt;/strong&gt; is responsible for resource inventory and allocation. It maintains an accurate view of the compute resources available across the environment and allows OpenStack services to determine where workloads can be scheduled.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Nova&lt;/strong&gt; provides the compute service itself. It is responsible for managing the lifecycle of virtual machine instances, coordinating with the hypervisor layer, consuming images from Glance, and interacting with the rest of the OpenStack control plane whenever an instance is created, modified, or removed.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Neutron&lt;/strong&gt; delivers networking services to the cloud. It provides the abstractions used to create tenant networks, routers, ports, security groups, and floating IPs, while integrating with the Open vSwitch infrastructure deployed earlier in the process.&lt;/p&gt;
&lt;p&gt;This step configures Nova and Neutron to operate with KVM virtualization, Open vSwitch networking, and the Ceph storage backend established in the previous steps. Together, these services transform the environment from a collection of infrastructure components into a functioning Infrastructure-as-a-Service platform. Once this phase completes, our cloud is capable of launching virtual machines. Users can authenticate through Keystone, select images stored in Glance, provision storage through Cinder, consume networking services through Neutron, and have Nova schedule and run workloads across the cluster.&lt;/p&gt;
&lt;p&gt;At this point, our OpenStack deployment is complete and ready to host its first instances.&lt;/p&gt;
&lt;h2&gt;Validation and Testing&lt;/h2&gt;
&lt;p&gt;1️⃣ Open your Horizon dashboard at &lt;code&gt;https://horizon.osh.proxmox.home&lt;/code&gt; (or the equivalent FQDN in your environment) and sign in using the default administrative credentials:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Username:&lt;/strong&gt; &lt;code&gt;admin&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Password:&lt;/strong&gt; &lt;code&gt;password&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Domain:&lt;/strong&gt; &lt;code&gt;Default&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;2️⃣ Open &lt;strong&gt;Project&lt;/strong&gt; &lt;strong&gt;→ API Access&lt;/strong&gt; where you should be able to see all OpenStack services API endpoints we deployed so far:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-17.B7_88Nfd_Z27uCOS.webp&quot; alt=&quot;OpenStack on Kubernetes article illustration&quot;&gt;&lt;/p&gt;
&lt;p&gt;3️⃣ Open &lt;strong&gt;Admin&lt;/strong&gt; &lt;strong&gt;→ Compute → Hypervisors → Compute Host&lt;/strong&gt; where you should be able to see all KVM hosts available. Make sure their &lt;strong&gt;Status&lt;/strong&gt; is &lt;code&gt;Enabled&lt;/code&gt; and their &lt;strong&gt;State&lt;/strong&gt; is &lt;code&gt;Up&lt;/code&gt;:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-18.DUVzDhUS_Z28moXu.webp&quot; alt=&quot;OpenStack on Kubernetes article illustration&quot;&gt;&lt;/p&gt;
&lt;p&gt;4️⃣ First, we are going to create a shared VPC. Open &lt;strong&gt;Admin&lt;/strong&gt; &lt;strong&gt;→ Network → Networks&lt;/strong&gt; and click &lt;strong&gt;Create Network.&lt;/strong&gt; Choose the following parameter values**:**&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Name:&lt;/strong&gt; &lt;code&gt;test_vpc&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Project:&lt;/strong&gt; &lt;code&gt;admin&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Provider Network Type:&lt;/strong&gt; &lt;code&gt;VXLAN&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Segmentation ID&lt;/strong&gt;: &lt;code&gt;959&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Enable Admin State&lt;/strong&gt;: &lt;code&gt;true&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Shared&lt;/strong&gt;: &lt;code&gt;true&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;MTU&lt;/strong&gt;: &lt;code&gt;1450&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Subnet Name&lt;/strong&gt;: &lt;code&gt;test_subnet&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Network Address&lt;/strong&gt;: &lt;code&gt;192.168.111.0/24&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The values used for &lt;strong&gt;Segmentation ID&lt;/strong&gt; (&lt;code&gt;959&lt;/code&gt;) and &lt;strong&gt;Network Address&lt;/strong&gt; (&lt;code&gt;192.168.111.0/24&lt;/code&gt;) are provided as examples only. The VXLAN segmentation ID can be any unused identifier within your environment, and the subnet CIDR can be replaced with any non-overlapping private network range that fits your addressing scheme.&lt;/p&gt;
&lt;p&gt;These are going to be the private addresses assigned to the virtual machines, valid only within the VPC itself. The only requirement is that both values are unique and do not conflict with existing OpenStack networks.&lt;/p&gt;
&lt;p&gt;5️⃣ Next, we are going to create an External Network. Open &lt;strong&gt;Admin&lt;/strong&gt; &lt;strong&gt;→ Network → Networks&lt;/strong&gt; and click &lt;strong&gt;Create Network.&lt;/strong&gt; Choose the following parameter values**:**&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Name:&lt;/strong&gt; &lt;code&gt;ext&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Project:&lt;/strong&gt; &lt;code&gt;admin&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Provider Network Type:&lt;/strong&gt; &lt;code&gt;flat&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Physical Network&lt;/strong&gt;: &lt;code&gt;public&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Enable Admin State&lt;/strong&gt;: &lt;code&gt;true&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Shared&lt;/strong&gt;: &lt;code&gt;true&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;External Network&lt;/strong&gt;: &lt;code&gt;true&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Subnet Name&lt;/strong&gt;: &lt;code&gt;ext_subnet&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Network Address&lt;/strong&gt;: &lt;code&gt;10.70.0.0/16&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Gateway IP&lt;/strong&gt;: &lt;code&gt;10.70.168.1&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Enable DHCP&lt;/strong&gt;: &lt;code&gt;false&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Allocation Pools&lt;/strong&gt;: &lt;code&gt;10.70.250.1,10.70.250.254&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;DNS Name Servers&lt;/strong&gt;: &lt;code&gt;10.70.168.1&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Unlike the tenant network created in the previous step, &lt;strong&gt;these values are not arbitrary&lt;/strong&gt; and must reflect the characteristics of your environment. The &lt;strong&gt;Physical Network&lt;/strong&gt; value (&lt;code&gt;public&lt;/code&gt;) refers to the Neutron provider network configured by the deployment and mapped to the secondary network interface (&lt;code&gt;eth1&lt;/code&gt;) of the Kubernetes nodes. The subnet configuration (&lt;code&gt;10.70.0.0/16&lt;/code&gt;), gateway (&lt;code&gt;10.70.168.1&lt;/code&gt;), allocation pool, and DNS server originate from the VLAN-backed network attached to that interface and managed by OPNsense. This network serves as the external connectivity layer for OpenStack and is used by services such as floating IPs and provider networks. If your environment uses a different VLAN, subnet, gateway, or address allocation scheme, adjust these values accordingly to match your own network configuration.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-19.Dptkyd9K_Z22vQ2o.webp&quot; alt=&quot;OPNsense DHCPv4 configuration screen for the VLAN-backed external network. The interface is configured with a 10.70.0.0/16 subnet, a DHCP allocation range from 10.70.168.2 to 10.70.168.254, and a DNS server at 10.70.168.1. These values are later referenced when configuring the OpenStack external network and floating IP address pool.&quot;&gt;&lt;/p&gt;
&lt;p&gt;The subnet allocated to VLAN 70, the DHCP address pool used for dynamic address assignment, and the DNS server advertised to clients. These values form the basis of the external network configuration created in Neutron.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-20.BL1rHBD4_1T3ONa.webp&quot; alt=&quot;OpenStack on Kubernetes article illustration&quot;&gt;&lt;/p&gt;
&lt;p&gt;6️⃣ Next, we are going to create a Router and Floating IPs.&lt;/p&gt;
&lt;p&gt;Go to &lt;strong&gt;Admin&lt;/strong&gt; &lt;strong&gt;→ Network → Routers&lt;/strong&gt; and click &lt;strong&gt;Create Router&lt;/strong&gt; using as &lt;strong&gt;Name:&lt;/strong&gt; &lt;code&gt;router1&lt;/code&gt;and &lt;strong&gt;External Network:&lt;/strong&gt; &lt;code&gt;ext&lt;/code&gt;. Then, open &lt;strong&gt;router1&lt;/strong&gt; and go to &lt;strong&gt;Interfaces → Add Interface&lt;/strong&gt; and select as private subnet &lt;code&gt;test_vpc/192.168.111.0/24&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-21.DnDhpQPa_ZVJDzO.webp&quot; alt=&quot;OpenStack on Kubernetes article illustration&quot;&gt;&lt;/p&gt;
&lt;p&gt;Go to &lt;strong&gt;Admin&lt;/strong&gt; &lt;strong&gt;→ Network → Floating IPs&lt;/strong&gt; click &lt;strong&gt;Allocate IP to Project&lt;/strong&gt;, choose &lt;strong&gt;Pool&lt;/strong&gt;: &lt;code&gt;ext 10.70.0.0/16&lt;/code&gt; and click &lt;strong&gt;Allocate Floating IP&lt;/strong&gt; to exit.&lt;/p&gt;
&lt;p&gt;7️⃣ With the networking stack in place, it is time to validate that the cloud can perform its primary function: launching virtual machines and providing network connectivity between them.&lt;/p&gt;
&lt;p&gt;Create 3 instances using the CirrOS image and the &lt;code&gt;m1.tiny&lt;/code&gt; flavor. Attach all three instances to the &lt;code&gt;test_vpc&lt;/code&gt; network created earlier. For one of the instances, also associate the floating IP allocated from the &lt;code&gt;ext&lt;/code&gt; external network.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-22.vSM4lsRS_ZLgFeR.webp&quot; alt=&quot;OpenStack on Kubernetes article illustration&quot;&gt;&lt;/p&gt;
&lt;p&gt;Our 3 CirrOS instances. Two instances use only tenant networking, while the third is associated with a floating IP from the external network, providing a path for validating external connectivity and SSH access.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-23.CdZo-Ci4_2g07QU.webp&quot; alt=&quot;OpenStack on Kubernetes article illustration&quot;&gt;&lt;/p&gt;
&lt;p&gt;Our network topology after the deployment of the tenant network, router, floating IP, and test instances.&lt;/p&gt;
&lt;p&gt;Once the instances have reached the &lt;strong&gt;Active&lt;/strong&gt; state, open their VNC consoles from Horizon and perform a series of connectivity checks:&lt;/p&gt;
&lt;p&gt;↔️ Start by verifying &lt;strong&gt;east-west connectivity within the tenant network&lt;/strong&gt;. From each instance, ping the private IP addresses of the other two instances to confirm that Neutron networking, Open vSwitch, and the VXLAN overlay are operating correctly.&lt;/p&gt;
&lt;p&gt;↕️ Next, &lt;strong&gt;verify north-south connectivity by testing outbound access from the instances to an external destination&lt;/strong&gt;. A successful response confirms that routing and external network access are functioning as expected.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-image-24.Bcu_4uCh_12JCzL.webp&quot; alt=&quot;VNC console of a CirrOS virtual machine running in OpenStack. The terminal shows successful ICMP communication with another instance on the tenant network (192.168.111.202) and successful connectivity to the external address 8.8.8.8. The output demonstrates both internal tenant-network communication and outbound internet access through the OpenStack networking stack.&quot;&gt;&lt;/p&gt;
&lt;p&gt;Connectivity validation from a virtual machine running inside the &lt;code&gt;test_vpc&lt;/code&gt; tenant network. The instance successfully communicates with another instance on the same VXLAN-backed network and reaches an external destination through the Neutron router and external provider network, confirming that both east-west and north-south networking are functioning correctly.&lt;/p&gt;
&lt;p&gt;As we can see in the image above:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;East-west connectivity test&lt;/strong&gt;: The instance (&lt;code&gt;192.168.111.195&lt;/code&gt;) successfully pings another virtual machine (&lt;code&gt;192.168.111.202&lt;/code&gt;) on the same tenant network, validating Neutron networking, Open vSwitch forwarding, and VXLAN encapsulation between instances.&lt;/li&gt;
&lt;li&gt;**North-south connectivity test:**The instance successfully reaches an external destination (&lt;code&gt;8.8.8.8&lt;/code&gt;), confirming that routing, source NAT, and connectivity through the external provider network are operating as expected.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;🏁 As last step we need to validate floating IP connectivity. From your workstation, establish an SSH connection to the instance that was assigned the floating IP. If the connection succeeds, traffic is successfully traversing the complete network path: OPNsense, the provider network, Neutron, Open vSwitch, and the target virtual machine.&lt;/p&gt;
&lt;h2&gt;Closing Thoughts&lt;/h2&gt;
&lt;p&gt;Running OpenStack on Kubernetes is no longer an experiment or a proof of concept. Projects such as OpenStack-Helm, Rook and Kubernetes have matured to the point where complex cloud platforms can be deployed, operated, and upgraded using the same cloud-native practices that have become commonplace the last years.&lt;/p&gt;
&lt;p&gt;At a time when digital sovereignty, technological self-determination, and control over critical infrastructure are rapidly becoming matters of national security, understanding how cloud platforms are built is valuable in its own right. The ability to operate and evolve strategic infrastructure without dependency on foreign providers, proprietary ecosystems, or external political decisions is no longer merely a technical consideration; it is increasingly becoming a question of resilience, autonomy and long-term competitiveness.&lt;/p&gt;
&lt;p&gt;Originally published on &lt;a href=&quot;https://akyriako.medium.com/deploy-openstack-on-kubernetes-in-under-60-minutes-f4ff01e2b424&quot;&gt;Medium&lt;/a&gt;.&lt;/p&gt;</content:encoded><h:img src="/_astro/thumbnail.Cz5oJbF6.png"/><enclosure url="/_astro/thumbnail.Cz5oJbF6.png"/></item><item><title>Cross-Cluster Leader Election for Kubernetes Workloads</title><link>https://www.backyard-rack.com/blog/cross-cluster-leader-election-for-kubernetes-workloads</link><guid isPermaLink="true">https://www.backyard-rack.com/blog/cross-cluster-leader-election-for-kubernetes-workloads</guid><description>Kubernetes-native leader election across multiple clusters using PostgreSQL and client-go.</description><pubDate>Sat, 11 Apr 2026 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;This article explores how to achieve cross-cluster leader election for Kubernetes workloads using PostgreSQL as a backend instead of the Kubernetes API. By plugging a custom database-backed lock into the &lt;a href=&quot;https://github.com/kubernetes/client-go/tree/master/tools/leaderelection&quot;&gt;client-go leader election module&lt;/a&gt;, multiple instances even across different clusters, can coordinate to ensure that only one acts as the leader at any given time.&lt;/p&gt;
&lt;p&gt;In distributed systems, especially in Kubernetes environments, it’s common to run multiple replicas of the same workload for availability and scalability. However, not all workloads are designed to run concurrently across all available replicas. Some require a &lt;strong&gt;single active instance&lt;/strong&gt; responsible for coordinating actions such as scheduling jobs, processing queues, managing shared state etc. This is where leader election becomes necessary.&lt;/p&gt;
&lt;h2&gt;Leader Election in Kubernetes&lt;/h2&gt;
&lt;p&gt;Kubernetes ensures that multiple instances of a &lt;code&gt;Pod&lt;/code&gt; can run reliably, &lt;strong&gt;but it does not inherently guarantee that only one of those instances performs a specific task; aka act a a leader&lt;/strong&gt;. Without coordination, this can lead to duplicated work, race conditions, or inconsistent system state.&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Leader election&lt;/em&gt; introduces a coordination mechanism where one instance is elected as the leader and is responsible for executing critical or even all sections of the workload, while the others remain passive or on standby. If the leader fails or becomes unavailable, another instance takes over. This pattern is commonly used in controllers, operators, and background workers. In a typical Kubernetes setup, leader election is scoped to a single cluster because it relies on the Kubernetes API as the coordination backend.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;This becomes a limitation when workloads span multiple clusters and require a single global leader.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Kubernetes provides a resource called &lt;code&gt;Lease&lt;/code&gt;, as part of the coordination API group, which is specifically designed for lightweight distributed coordination. A &lt;code&gt;Lease&lt;/code&gt; object contains metadata such as the identity of the current leader, the duration of the lease and timestamps for renewals.&lt;/p&gt;
&lt;p&gt;Leader election works by having participants attempt to acquire or renew a lease. The pod that successfully holds and continuously renews the lease is considered the leader of the replica set. If the lease is not renewed within a defined time window, it is assumed that the leader has failed, and other participants can attempt to acquire it.&lt;/p&gt;
&lt;p&gt;This mechanism is efficient because it avoids heavy resource contention and relies on simple update operations against the Kubernetes API server. However, it is inherently limited to a single cluster. &lt;strong&gt;When workloads span multiple clusters, this model no longer applies.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;In such cases, one option is to introduce a shared “control-plane” cluster and have all workloads coordinate through its Kubernetes API, which quickly becomes complex and difficult to manage. An alternative approach, which this article focuses on, is to use a PostgreSQL database as a shared backend for storing and updating the leader election state.&lt;/p&gt;
&lt;p&gt;This solution relies on the Kubernetes leader election module, specifically &lt;code&gt;k8s.io/client-go/tools/leaderelection/resourcelock&lt;/code&gt; which implements the core leader election algorithm including the lease acquisition and renewal logic, the leader transition handling and the respective timing configuration (lease duration, renew deadline, retry period etc.)&lt;/p&gt;
&lt;p&gt;Instead of using the default Kubernetes-backed lock (such as &lt;code&gt;Lease&lt;/code&gt; objects), the implementation provides a custom lock that satisfies the expected &lt;code&gt;resourcelock&lt;/code&gt; interface.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;This allows the leader election logic from &lt;a href=&quot;https://github.com/kubernetes/client-go/tree/master/tools/leaderelection&quot;&gt;client-go&lt;/a&gt; to remain unchanged, while replacing the storage backend with PostgreSQL. As a result, the same leader election semantics are preserved, &lt;strong&gt;but coordination is no longer tied to a single Kubernetes control plane.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h2&gt;Database setup and migration execution&lt;/h2&gt;
&lt;p&gt;To enable leader election across clusters, we need a shared and reliable coordination backend. In this implementation, PostgreSQL is used to store the leader election state. Every participating instance (each pod from each replica set, from every cluster) connects to the same database and reads or updates a single record that represents the current leader.&lt;/p&gt;
&lt;p&gt;The schema is minimal but you could extend it as you wish in order to hold the information you need. It is just a single table named &lt;code&gt;leader_election_locks&lt;/code&gt; that holds &lt;strong&gt;one row per logical election&lt;/strong&gt;. Each row is identified by a combination of &lt;code&gt;namespace&lt;/code&gt; and &lt;code&gt;name&lt;/code&gt;, and stores the serialized leader election record in JSON format. A &lt;code&gt;version&lt;/code&gt; field is used to implement optimistic concurrency control, ensuring that updates only succeed if no other instance has modified the record in the meantime. An &lt;code&gt;updated_at&lt;/code&gt; timestamp is also maintained for observability and debugging.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;The article assumes you have already a PostgreSQL instance in place. If not, you can easily spin up one using the &lt;a href=&quot;https://awesome-docker-compose.com/postgresql-official&quot;&gt;official docker-compose manifest&lt;/a&gt; or via &lt;a href=&quot;https://autobase.tech/&quot;&gt;Autobase for PostgreSQL&lt;/a&gt;.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;To apply the migration, the repository provides a small helper under &lt;code&gt;cmd/migration&lt;/code&gt;, that collects and executes the SQL files located in the &lt;code&gt;migrations&lt;/code&gt; directory:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-sql&quot;&gt;CREATE TABLE leader_election_locks (
    namespace   text        NOT NULL,
    name        text        NOT NULL,
    record_json jsonb       NOT NULL,
    version     bigint      NOT NULL DEFAULT 1,
    updated_at  timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (namespace, name)
);

CREATE INDEX idx_leader_election_locks_updated_at
    ON leader_election_locks (updated_at);
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Running it is straightforward:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;go run ./cmd/migration
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Once executed, the required table is created and ready to be used by the leader election logic. From that point on, all instances participating in the election will rely on this table as their shared coordination point.&lt;/p&gt;
&lt;h2&gt;Implementing the PostgreSQL resource lock&lt;/h2&gt;
&lt;p&gt;At the core of this solution is &lt;code&gt;pglock/lock.go&lt;/code&gt; , where we replaces the default Kubernetes resource lock with a PostgreSQL-backed alternative. Instead of relying on &lt;code&gt;Lease&lt;/code&gt; objects stored in the Kubernetes API, the custom &lt;code&gt;Lock&lt;/code&gt; component persists and manages the leader election state directly in the database.&lt;/p&gt;
&lt;p&gt;The PostgreSQL lock works because it satisfies the interface expected by the Kubernetes leader election library, namely the &lt;code&gt;resourcelock.Interface&lt;/code&gt; from &lt;a href=&quot;https://github.com/kubernetes/client-go/tree/master/tools/leaderelection&quot;&gt;client-go&lt;/a&gt;, and delegates all state operations to PostgreSQL.&lt;/p&gt;
&lt;p&gt;This interface defines how the leader election module interacts with the underlying coordination mechanism. It does not care whether the state is stored in a Kubernetes &lt;code&gt;Lease&lt;/code&gt;, a &lt;code&gt;ConfigMap&lt;/code&gt;, or, in this case, a PostgreSQL table. As long as the interface is implemented correctly, the leader election logic remains unchanged.&lt;/p&gt;
&lt;p&gt;The interface requires a small set of methods:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;&lt;code&gt;Get()&lt;/code&gt; retrieves the current leader election record&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;code&gt;Create()&lt;/code&gt; initializes the record if it does not exist&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;code&gt;Update()&lt;/code&gt; attempts to modify the record&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;code&gt;Identity()&lt;/code&gt; returns the unique identity of the participant&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;code&gt;Describe()&lt;/code&gt; provides a human-readable identifier for logging&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;By implementing these methods in the &lt;code&gt;Lock&lt;/code&gt; component in &lt;code&gt;pglock/lock.go&lt;/code&gt;, the PostgreSQL-backed lock becomes a drop-in replacement for the standard Kubernetes resource locks. The &lt;code&gt;Lock&lt;/code&gt; component identifies the specific election using a &lt;code&gt;namespace&lt;/code&gt; and &lt;code&gt;name&lt;/code&gt;, and keeps track of a local &lt;code&gt;version&lt;/code&gt;. This version is important, as it enables optimistic concurrency control when multiple instances attempt to update the same record.&lt;/p&gt;
&lt;p&gt;1️⃣ The leader election module periodically calls &lt;code&gt;Get()&lt;/code&gt; method which tries to retrieves the current election state, &lt;code&gt;resourcelock.LeaderElectionRecord&lt;/code&gt; from the database and allows each instance to understand who the current leader is and whether the lock is already held.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-go&quot;&gt;func (l *Lock) Get(ctx context.Context) (*resourcelock.LeaderElectionRecord, []byte, error) {
 const q = `


 var raw []byte
 var version int64

 err := l.pool.QueryRow(ctx, q, l.namespace, l.name).Scan(&amp;#x26;raw, &amp;#x26;version)
 if err != nil {
  if errors.Is(err, pgx.ErrNoRows) {
   return nil, nil, ErrNotFound
  }
  return nil, nil, fmt.Errorf(&quot;getting leader lock failed: %w&quot;, err)
 }

 var rec resourcelock.LeaderElectionRecord
 if err := json.Unmarshal(raw, &amp;#x26;rec); err != nil {
  return nil, nil, fmt.Errorf(&quot;unmarshalling leader record failed: %w&quot;, err)
 }

 l.mu.Lock()
 l.lastVersion = version
 l.mu.Unlock()

 return &amp;#x26;rec, raw, nil
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;2️⃣ If no record exists yet, the &lt;code&gt;Create()&lt;/code&gt; method is used to insert the a row. Since multiple instances may attempt this at the same time, the implementation relies on PostgreSQL’s unique constraint on &lt;code&gt;(namespace, name)&lt;/code&gt; to ensure only one succeeds. Any duplicate insert attempts are treated as a normal primary key violation, guaranteeing that way deterministic strong consistency.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-go&quot;&gt;func (l *Lock) Create(ctx context.Context, ler resourcelock.LeaderElectionRecord) error {
 raw, err := json.Marshal(ler)
 if err != nil {
  return fmt.Errorf(&quot;marshalling leader record failed: %w&quot;, err)
 }

 const q = `
INSERT INTO leader_election_locks(namespace, name, record_json, version, updated_at)
VALUES ($1, $2, $3, 1, now())
`
 _, err = l.pool.Exec(ctx, q, l.namespace, l.name, raw)
 if err != nil {
  var pgErr *pgconn.PgError
  if errors.As(err, &amp;#x26;pgErr) &amp;#x26;&amp;#x26; pgErr.Code == &quot;23505&quot; {
   return ErrAlreadyExists
  }
  return fmt.Errorf(&quot;creating leader lock failed: %w&quot;, err)
 }

 l.mu.Lock()
 l.lastVersion = 1
 l.mu.Unlock()

 return nil
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;3️⃣ The most critical part is the &lt;code&gt;Update()&lt;/code&gt; method. This is where leadership is renewed or transferred. Instead of blindly updating the row, the query includes the previously observed version. If the version still matches, the update succeeds and increments the version. If not, it means another instance has already modified the record, and the update is rejected.&lt;/p&gt;
&lt;p&gt;This approach ensures that only one instance can successfully update the leader record at a time, which is enough to maintain consistent leader election semantics. Combined with the retry and timing logic from &lt;a href=&quot;https://github.com/kubernetes/client-go/tree/master/tools/leaderelection&quot;&gt;client-go&lt;/a&gt;, it provides a reliable coordination mechanism without requiring any Kubernetes API interaction.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-go&quot;&gt;func (l *Lock) Update(ctx context.Context, ler resourcelock.LeaderElectionRecord) error {
 raw, err := json.Marshal(ler)
 if err != nil {
  return fmt.Errorf(&quot;marshalling leader record failed: %w&quot;, err)
 }

 l.mu.Lock()
 expectedVersion := l.lastVersion
 l.mu.Unlock()

 if expectedVersion == 0 {
  return fmt.Errorf(&quot;updating called before get; no cached version&quot;)
 }

 const q = `
UPDATE leader_election_locks
SET record_json = $3,
    version = version + 1,
    updated_at = now()
WHERE namespace = $1
  AND name = $2
  AND version = $4
RETURNING version
`
 var newVersion int64
 err = l.pool.QueryRow(ctx, q, l.namespace, l.name, raw, expectedVersion).Scan(&amp;#x26;newVersion)
 if err != nil {
  if errors.Is(err, pgx.ErrNoRows) {
   return ErrVersionConflict
  }
  return fmt.Errorf(&quot;updating leader lock failed: %w&quot;, err)
 }

 l.mu.Lock()
 l.lastVersion = newVersion
 l.mu.Unlock()

 return nil
}
&lt;/code&gt;&lt;/pre&gt;
&lt;blockquote&gt;
&lt;p&gt;This separation is what makes the approach robust. The proven leader election behavior from Kubernetes is preserved, while the storage backend is replaced with PostgreSQL, enabling coordination across clusters without changing the core algorithm.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h2&gt;Implementing a sample workload&lt;/h2&gt;
&lt;p&gt;To understand how everything fits together, we will build a simple client that requires a leader and consequently a leader election process. This sample will not demonstrate &lt;em&gt;how&lt;/em&gt; leader election works (that is already handled by &lt;a href=&quot;https://github.com/kubernetes/client-go/tree/master/tools/leaderelection&quot;&gt;client-go&lt;/a&gt;) but &lt;em&gt;where&lt;/em&gt; the coordination state lives and &lt;em&gt;how&lt;/em&gt; the application integrates with it.&lt;/p&gt;
&lt;p&gt;1️⃣ We need at first, build a unique identity that includes cluster, namespace, and pod information. In a single-cluster setup, a pod name would be usually enough. In a cross-cluster scenario on the other hand it isn’t, because without including the cluster dimension, two different clusters could produce identical identities, which would break the election semantics.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-perl&quot;&gt;identity := fmt.Sprintf(&quot;%s/%s/%s&quot;, cfg.ClusterName, cfg.Namespace, cfg.PodName)
lock := pglock.New(pool, cfg.ElectionNamespace, cfg.ElectionName, identity)
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Once identity is defined, the application initializes a new &lt;code&gt;Lock&lt;/code&gt;. This is where the PostgreSQL-backed implementation comes in. As we already discussed before, the &lt;code&gt;Lock&lt;/code&gt; component exposes a small set of methods &lt;code&gt;Get&lt;/code&gt;, &lt;code&gt;Create&lt;/code&gt;, and &lt;code&gt;Update&lt;/code&gt;, which represent the full lifecycle of the election state.&lt;/p&gt;
&lt;p&gt;2️⃣ The leader election loop itself is driven by the configuration defined in a structure called &lt;code&gt;leaderelection.LeaderElectionConfig&lt;/code&gt;. This is where the behavior is defined:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;&lt;code&gt;LeaseDuration&lt;/code&gt; defines how long leadership is considered valid without renewal (defaults to &lt;code&gt;15s&lt;/code&gt;)&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;code&gt;RenewDeadline&lt;/code&gt; defines how long the current leader keeps trying to renew before giving up (defaults to &lt;code&gt;10s&lt;/code&gt;)&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;code&gt;RetryPeriod&lt;/code&gt; defines how frequently non-leaders attempt to acquire the lock (default to &lt;code&gt;2s&lt;/code&gt;)&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;blockquote&gt;
&lt;p&gt;These three parameters are not arbitrary. They enforce a timing model where a leader must continuously prove liveness, while others compete to take over when that liveness is lost.
All three parameters default values as well are not arbitrary. They’re set to the default values that are used internally in Kubernetes API and is really not advised to change them.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;pre&gt;&lt;code class=&quot;language-go&quot;&gt;lec := leaderelection.LeaderElectionConfig{
  Lock:            lock,
  ReleaseOnCancel: true,
  LeaseDuration:   cfg.LeaseDuration,
  RenewDeadline:   cfg.RenewDeadline,
  RetryPeriod:     cfg.RetryPeriod,
  Name:            cfg.ElectionName,
  Callbacks: leaderelection.LeaderCallbacks{
   OnStartedLeading: func(ctx context.Context) {
    klog.Infof(&quot;elected as leader: identity=&apos;%s&apos;&quot;, identity)

    // Do what you have to do or call a function like leaderRun
    leaderRun(ctx, identity)
   },
   OnStoppedLeading: func() {
    klog.Infof(&quot;stepping down...&quot;)
    //os.Exit(0)

    // Do what you have to do or call a function to perform clean-up
   },
   OnNewLeader: func(id string) {
    if id == identity {
     return
    }
    klog.Infof(&quot;current leader: identity=&apos;%s&apos;&quot;, id)
   },
  },
 }
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;3️⃣ Once a new leader elector, &lt;code&gt;leaderelection.NewLeaderElector&lt;/code&gt;, is created and &lt;code&gt;Run()&lt;/code&gt; is called, the flow is consistent regardless of the backend:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-go&quot;&gt; elector, err := leaderelection.NewLeaderElector(lec)
 if err != nil {
  klog.Error(&quot;creating new leader elector failed: %v&quot;, err)
  os.Exit(1)
 }

 elector.Run(ctx)
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;code&gt;Run()&lt;/code&gt; binds the current process to the provided context. As long as the context is active, the instance participates in the election. When the context is cancelled (for example on &lt;code&gt;SIGTERM&lt;/code&gt; or &lt;code&gt;SIGINT&lt;/code&gt;), &lt;code&gt;Run()&lt;/code&gt; ensures that participation stops cleanly and leadership is released.&lt;/p&gt;
&lt;p&gt;Internally, &lt;code&gt;Run()&lt;/code&gt; starts a loop that continuously attempts (on &lt;strong&gt;every&lt;/strong&gt; pod of the replica set of the workload) to acquire or renew leadership. This loop is not exposed directly but is encapsulated inside the elector. It drives calls to the resource lock (via the methods we discussed already &lt;code&gt;Get&lt;/code&gt;, &lt;code&gt;Create&lt;/code&gt;, &lt;code&gt;Update&lt;/code&gt;) based on the configured timing parameters.&lt;/p&gt;
&lt;p&gt;The sample application, or any application you will implement using &lt;code&gt;pglock&lt;/code&gt; or any similar implementation for that matter, never calls these methods directly during election. &lt;code&gt;Run()&lt;/code&gt; is the component that decides &lt;em&gt;when&lt;/em&gt; and &lt;em&gt;how often&lt;/em&gt; they are invoked based on the configuration &lt;code&gt;leaderelection.LeaderElectionConfig&lt;/code&gt; we provided to the elector.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;Run()&lt;/code&gt; is also responsible for invoking the registered callbacks (see above &lt;code&gt;OnStartedLeading&lt;/code&gt;, &lt;code&gt;OnStoppedLeading&lt;/code&gt;, &lt;code&gt;OnNewLeader&lt;/code&gt;). It guarantees, regardless of resource lock implementation, that:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;&lt;code&gt;OnStartedLeading&lt;/code&gt; runs only after leadership is successfully acquired&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;code&gt;OnStoppedLeading&lt;/code&gt; is triggered when leadership is lost or the context ends&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;code&gt;OnNewLeader&lt;/code&gt; is triggered when a different leader is observed&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;4️⃣ The leader election loop as we discussed above, is driven by &lt;code&gt;Run()&lt;/code&gt;, but the application logic is defined through &lt;strong&gt;callbacks&lt;/strong&gt;. The two that matter the most, are &lt;code&gt;OnStartedLeading&lt;/code&gt; and &lt;code&gt;OnStoppedLeading&lt;/code&gt;. These are the points where your application pods react to changes in leadership.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;OnStartedLeading&lt;/code&gt; is invoked &lt;strong&gt;when the instance has successfully acquired the lock and is now considered the leader&lt;/strong&gt;. At this point, the election mechanism has already done the heavy lifting and there is no ambiguity about ownership. This callback is where you attach any logic that must run exclusively on that single instance that now acts as a leader.&lt;/p&gt;
&lt;p&gt;In the example, the callback calls &lt;code&gt;leaderRun(ctx, identity)&lt;/code&gt;. That function represents the &lt;em&gt;leader-only&lt;/em&gt; workload. It’s deliberately demonstrating a simple thing: once leadership is obtained, the process enters a controlled execution loop that continues as long as leadership is maintained or the parent context is cancelled:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-go&quot;&gt;func leaderRun(ctx context.Context, identity string) {
 ticker := time.NewTicker(5 * time.Second)
 defer ticker.Stop()

 klog.Infof(&quot;leader workload started: identity=&apos;%s&apos;&quot;, identity)

 for {
  select {
  case &amp;#x3C;-ctx.Done():
   klog.Infof(&quot;leader workload stopping: identity=&apos;%s&apos;&quot;, identity)
   return
  case &amp;#x3C;-ticker.C:
   // do what you have to do here. here is your application/business logic
   klog.Infof(&quot;leader workload tick: identity=&apos;%s&apos;&quot;, identity)
  }
 }
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;code&gt;OnStoppedLeading&lt;/code&gt; is triggered when the instance loses leadership. This can happen for several reasons: the process failed to renew the lease in time, another instance successfully updated the record, or the application context was cancelled. The important part is that this callback marks the end of exclusive ownership and you can either inform your logs or proceed to clean-up tasks depending on your application logic.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;This whole dance described above is the key takeaway of the election pattern. &lt;strong&gt;The workload is not aware of leader election directly. It simply runs while the context is active and stops when it is cancelled&lt;/strong&gt;. Leader election controls &lt;em&gt;when&lt;/em&gt; that context is valid.
In a real system, &lt;code&gt;leaderRun()&lt;/code&gt; or however you want to call that method for that matter, would be replaced with something meaningful: a controller loop, a job scheduler, or a queue processor. The structure would remain the same. The only requirement is that the workload respects context cancellation so that leadership transitions remain clean and predictable.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h2&gt;Simulating leader election across multiple instances&lt;/h2&gt;
&lt;p&gt;To see the behavior in practice we will let multiple instances of the sample application run simultaneously and compete to become the leader.&lt;/p&gt;
&lt;p&gt;1️⃣ First, build the application:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;go build -o demo ./cmd/app
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;2️⃣ Before starting the instances, ensure that:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;PostgreSQL is running and reachable&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;the migration has been applied&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;all instances use the same &lt;code&gt;POSTGRES_DSN&lt;/code&gt;, &lt;code&gt;ELECTION_NAMESPACE&lt;/code&gt;, and &lt;code&gt;ELECTION_NAME&lt;/code&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;code&gt;ELECTION_NAMESPACE&lt;/code&gt; and &lt;code&gt;ELECTION_NAME&lt;/code&gt; define &lt;strong&gt;the scope and identity of a leader election group&lt;/strong&gt;. Together, they uniquely identify a single election within the shared PostgreSQL backend.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;They are not related to Kubernetes namespaces&lt;/strong&gt; in a strict sense. In this implementation, they are simply logical identifiers used to group participants that should compete for the same leadership.&lt;/p&gt;
&lt;p&gt;At the database level, &lt;strong&gt;these two values map directly to the primary key&lt;/strong&gt; of the &lt;code&gt;leader_election_locks&lt;/code&gt; table. This means:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;every unique &lt;code&gt;(namespace, name)&lt;/code&gt; pair represents one independent election&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;all instances using the same pair will compete for the same leadership&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;instances using different values will not see or affect each other&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;code&gt;ELECTION_NAME&lt;/code&gt; defines &lt;em&gt;what&lt;/em&gt; is being elected. It typically represents a specific workload or responsibility, such as a scheduler, controller, or background processor.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;ELECTION_NAMESPACE&lt;/code&gt; is used to group or isolate elections. It can be used to separate environments (for example &lt;code&gt;dev&lt;/code&gt;, &lt;code&gt;staging&lt;/code&gt;, &lt;code&gt;prod&lt;/code&gt;) or logical domains within a system.&lt;/p&gt;
&lt;p&gt;A useful way to think about it is:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;&lt;code&gt;ELECTION_NAME&lt;/code&gt; → the role&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;code&gt;ELECTION_NAMESPACE&lt;/code&gt; → the scope&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;For example:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-ini&quot;&gt;ELECTION_NAMESPACE=prod
ELECTION_NAME=job-runner
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;All instances configured with these values will compete to become the single leader responsible for running the job runner in the production environment. If you change either value, you create a completely separate election. &lt;strong&gt;This allows multiple independent leader election processes to coexist in the same PostgreSQL database without interfering with each other.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;3️⃣ Each instance must have a unique identity. In a Kubernetes environment, this comes naturally from pod metadata. For local testing, you can simulate this by setting different pod name and namespace values.&lt;/p&gt;
&lt;p&gt;Start &lt;strong&gt;three instances&lt;/strong&gt; in separate terminals (we will leave the default value for &lt;code&gt;ELECTION_NAMESPACE&lt;/code&gt; and &lt;code&gt;ELECTION_NAME&lt;/code&gt;):&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;POSTGRES_DSN=&quot;postgres://user:pass@localhost:5432/db&quot; \
K8S_CLUSTER_NAME=cluster-a \
K8S_NAMESPACE=default \
K8S_POD_NAME=pod-1 \
./demo
&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;POSTGRES_DSN=&quot;postgres://user:pass@localhost:5432/db&quot; \
K8S_CLUSTER_NAME=cluster-b \
K8S_NAMESPACE=default \
K8S_POD_NAME=pod-2 \
./demo
&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;POSTGRES_DSN=&quot;postgres://user:pass@localhost:5432/db&quot; \
K8S_CLUSTER_NAME=cluster-c \
K8S_NAMESPACE=default \
K8S_POD_NAME=pod-3 \
./demo
&lt;/code&gt;&lt;/pre&gt;
&lt;blockquote&gt;
&lt;p&gt;Replace the &lt;code&gt;POSTGRES_DSN&lt;/code&gt; value with your own database connection string, matching your PostgreSQL user, password, host, port, and database name.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;4️⃣ Once all instances are running, you will observe that:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;only one instance&lt;/strong&gt; logs that it has become leader&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;the other instances report the current leader through &lt;code&gt;OnNewLeader&lt;/code&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;only the leader&lt;/strong&gt; executes the periodic workload&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;To observe a leadership change, terminate the leader process (for example with &lt;em&gt;Ctrl+C&lt;/em&gt;). After the lease expires, one of the remaining instances will acquire leadership. You will observe:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;the previous leader stops its workload&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;a new instance logs that it has started leading&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;the workload resumes on the new leader&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This demonstrates that all instances are coordinating exclusively through PostgreSQL. There is no Kubernetes API involved, yet the behavior matches the standard Kubernetes coordination API leader election semantics: a single active leader, automatic fail-over, and continuous renewal of ownership.&lt;/p&gt;
&lt;h2&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;This approach demonstrates that leader election for Kubernetes workloads does not have to be tied to the Kubernetes API. By implementing the &lt;code&gt;resourcelock.Interface&lt;/code&gt; and using PostgreSQL as the coordination backend, &lt;strong&gt;we retain the proven leader election semantics&lt;/strong&gt; from &lt;code&gt;client-go&lt;/code&gt; while &lt;strong&gt;extending them beyond a single cluster boundary&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;The main advantage of this design is flexibility. It enables coordination across clusters, regions, or even hybrid environments where a shared Kubernetes control plane is not available or not desirable. It also keeps the application structure unchanged. Workloads still rely on the same leader election model and callbacks, which makes the transition straightforward.&lt;/p&gt;
&lt;p&gt;There are, however, trade-offs. PostgreSQL becomes a critical dependency for coordination. Its availability and latency directly affect leader election behavior, especially lease renewal and failover timing. This also introduces a different operational concern compared to relying on the Kubernetes API, which is already part of the cluster control plane. Additionally, this model does not introduce stronger guarantees than the original leader election mechanism. It should still be treated as a coordination tool, not as strict distributed locking with &lt;strong&gt;fencing&lt;/strong&gt;.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Fencing&lt;/strong&gt; is a mechanism used in distributed systems to prevent a previous leader from continuing to act after it has lost leadership.
In simple terms, even if a node &lt;em&gt;thinks&lt;/em&gt; it is still the leader (due to network delays or pauses), fencing ensures that its actions are rejected. This is usually done by attaching a &lt;strong&gt;monotonically increasing token (fencing token)&lt;/strong&gt; to each leadership term. Only the node with the latest token is allowed to perform operations.
Without fencing, two instances might briefly act as leader at the same time, leading to conflicting updates.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;PostgreSQL itself, is not a hard requirement. The same approach can be applied to &lt;strong&gt;any&lt;/strong&gt; backend that can provide consistent read and conditional update semantics. The only requirement is to implement the expected interface correctly. This opens the door to using other systems such as Redis, etcd, or any data-store that can support similar concurrency patterns. The full implementation, including the PostgreSQL-backed lock and the sample application, is available here:&lt;/p&gt;
&lt;p&gt;This repository can be used as a starting point or adapted to fit different environments and backend choices, depending on the coordination requirements.&lt;/p&gt;
&lt;p&gt;Originally published on &lt;a href=&quot;https://medium.com/itnext/cross-cluster-leader-election-for-kubernetes-workloads-3548f77b0531&quot;&gt;Medium&lt;/a&gt;&lt;/p&gt;</content:encoded><h:img src="/_astro/thumbnail.D1t9B_QF.jpg"/><enclosure url="/_astro/thumbnail.D1t9B_QF.jpg"/></item><item><title>Configure a Shared Træfik Gateway on K3s</title><link>https://www.backyard-rack.com/blog/configure-shared-traefik-gateway-on-k3s</link><guid isPermaLink="true">https://www.backyard-rack.com/blog/configure-shared-traefik-gateway-on-k3s</guid><description>Træfik Gateway API on K3s: Shared Gateway, TLS and Helm caveats.</description><pubDate>Tue, 17 Feb 2026 00:00:00 GMT</pubDate><content:encoded>&lt;blockquote&gt;
&lt;p&gt;Ingress NGINX Controller for Kubernetes is retired on March 2026 and Kubernetes Ingress API will receive no future development. Configure Traefik as a shared Gateway in &lt;a href=&quot;https://k3s.io/&quot;&gt;K3s&lt;/a&gt;.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;The Kubernetes &lt;a href=&quot;https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/&quot;&gt;&lt;strong&gt;Ingress API&lt;/strong&gt;&lt;/a&gt; is feature-frozen and will not developed any further. New traffic management capabilities are being developed in the Kubernetes &lt;a href=&quot;https://kubernetes.io/docs/concepts/services-networking/gateway/&quot;&gt;&lt;strong&gt;Gateway API&lt;/strong&gt;&lt;/a&gt;, which provides a more expressive and extensible model for service networking, but is still lacking the feature parity with the Ingress API. As stated in Kubernetes official documentation:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;“The Kubernetes project recommends using&lt;/strong&gt; &lt;a href=&quot;https://gateway-api.sigs.k8s.io/&quot;&gt;&lt;strong&gt;Gateway&lt;/strong&gt;&lt;/a&gt; &lt;strong&gt;instead of&lt;/strong&gt; &lt;a href=&quot;https://kubernetes.io/docs/concepts/services-networking/ingress/&quot;&gt;&lt;strong&gt;Ingress&lt;/strong&gt;&lt;/a&gt;. The Ingress API has been frozen. This means that:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The Ingress API is generally available, and is subject to the &lt;a href=&quot;https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecating-parts-of-the-api&quot;&gt;stability guarantees&lt;/a&gt; for generally available APIs. The Kubernetes project has no plans to remove Ingress from Kubernetes.&lt;/li&gt;
&lt;li&gt;The Ingress API is no longer being developed, and will have no further changes or updates made to it.”&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;In the meantime, the &lt;a href=&quot;https://github.com/kubernetes/ingress-nginx/&quot;&gt;&lt;strong&gt;ingress-nginx&lt;/strong&gt;&lt;/a&gt; project (Ingress NGINX Controller for Kubernetes) &lt;a href=&quot;https://www.kubernetes.dev/blog/2025/11/12/ingress-nginx-retirement/&quot;&gt;retires on March 2026&lt;/a&gt;, forcing half of the Kubernetes universe racing against time to migrate to a viable alternative.&lt;/p&gt;
&lt;p&gt;Ingress and Gateway APIs &lt;strong&gt;can coexist&lt;/strong&gt; in the same cluster. Migration does not need to be immediate or global. Many environments adopt Gateway API incrementally by introducing &lt;strong&gt;shared Gateways&lt;/strong&gt; for new workloads while continuing to run existing services behind Ingress.&lt;/p&gt;
&lt;p&gt;Using a &lt;strong&gt;shared Gateway&lt;/strong&gt; has several advantages. It allows a single, centrally managed entry point to serve multiple applications, reduces duplication of configuration, and limits the permissions required by application teams. Applications only need to create route resources in their own namespaces, while certificates, listeners, and exposure policies remain under the control of the Gateway owner.&lt;/p&gt;
&lt;h3&gt;Gateway API Components&lt;/h3&gt;
&lt;p&gt;The Gateway API defines a set of resources that separate &lt;strong&gt;infrastructure concerns&lt;/strong&gt; from &lt;strong&gt;application routing concerns&lt;/strong&gt;. Each resource (4 stable at the time of writing) has a clear responsibility and scope:&lt;/p&gt;
&lt;h4&gt;GatewayClass&lt;/h4&gt;
&lt;p&gt;A &lt;code&gt;GatewayClass&lt;/code&gt; represents a type of Gateway implementation (exactly as the &lt;code&gt;IngressClass&lt;/code&gt; for the Ingress API). It is defined by the infrastructure or platform layer and is typically installed by a Gateway controller (for example, Traefik, Envoy Gateway, Kong and &lt;a href=&quot;https://gateway-api.sigs.k8s.io/implementations/#gateway-controller-implementation-status&quot;&gt;many others&lt;/a&gt;).&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Application teams do not interact with &lt;code&gt;GatewayClass&lt;/code&gt; directly.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h4&gt;Gateway&lt;/h4&gt;
&lt;p&gt;A &lt;code&gt;Gateway&lt;/code&gt; represents a network entry point into the cluster. It defines where and how traffic enters the system (per cluster if shared or per application if dedicated).&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Gateways are usually managed by platform or infrastructure teams and are often shared across multiple applications and namespaces.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h4&gt;Routes&lt;/h4&gt;
&lt;p&gt;Routes define &lt;strong&gt;application-level routing&lt;/strong&gt; to backend services and attach to a Gateway. &lt;code&gt;HTTPRoute&lt;/code&gt; is the most common route type and is used for good ole’ HTTP and HTTPS traffic. &lt;code&gt;GRPCRoute&lt;/code&gt; on the other hand, defines gRPC-specific rules for routing traffic from a Gateway to backend network endpoints.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Routes are typically created and managed by application teams in their own namespaces.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h3&gt;Configure K3s Helm Charts Manifests&lt;/h3&gt;
&lt;p&gt;&lt;a href=&quot;https://k3s.io/&quot;&gt;K3s&lt;/a&gt;, in its default configuration, comes pre-configured with Traefik and this where we are going to focus on. Traefik Gateway CRDs are installed but no &lt;code&gt;GatewayClass&lt;/code&gt; or &lt;code&gt;Gateway&lt;/code&gt; are enabled or deployed. K3s is bootstrapped with a Traefik Ingress.&lt;/p&gt;
&lt;p&gt;K3s uses a built-in mechanism to deploy and manage core cluster components using Helm charts. This mechanism is implemented through two custom resources: &lt;code&gt;HelmChart&lt;/code&gt; and &lt;code&gt;HelmChartConfig&lt;/code&gt;. Together, they allow K3s to install, configure, and upgrade packaged applications during cluster startup and runtime.&lt;/p&gt;
&lt;p&gt;This system is primarily used to bootstrap components such as Traefik, CoreDNS, metrics-server, and other optional add-ons. The presence of a &lt;code&gt;HelmChart&lt;/code&gt; resource signals the K3s Helm controller to install or upgrade the corresponding Helm release and a &lt;code&gt;HelmChartConfig&lt;/code&gt; resource provides user-defined overrides for a corresponding &lt;code&gt;HelmChart&lt;/code&gt;. K3s supports placing manifests under:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;/var/lib/rancher/k3s/server/manifests/
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This mechanism allows cluster configuration and enables automatic upfront bootstrapping and (re)configuration without the need of manual Helm commands. To configure Traefik, we need to update the &lt;code&gt;traefik-config.yaml&lt;/code&gt; as following:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Enable the &lt;code&gt;kubernetesGateway&lt;/code&gt; under &lt;code&gt;providers&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Add a new stanza &lt;code&gt;gateway.listeners&lt;/code&gt;. We need to add two listeners, one for HTTP and one HTTPS traffic and point them to the internal ports of the Traefik container, namely &lt;code&gt;8000&lt;/code&gt; and &lt;code&gt;8443&lt;/code&gt; respectively.&lt;/li&gt;
&lt;li&gt;Each listener should explicitly set &lt;code&gt;namespacePolicy.from&lt;/code&gt; to &lt;code&gt;All&lt;/code&gt;. Without this configuration, the value defaults to &lt;code&gt;Same&lt;/code&gt;, restricting Gateway access to workloads within the same namespace only.&lt;/li&gt;
&lt;li&gt;For the HTTPS listener set &lt;code&gt;mode&lt;/code&gt; to &lt;code&gt;Terminate&lt;/code&gt;. With this knob TLS connections are terminated at the Gateway. TLS handling is centralized at the Gateway and like this we avoid the need for individual applications to manage certificates or serve HTTPS themselves.&lt;/li&gt;
&lt;li&gt;When using &lt;code&gt;protocol: HTTPS&lt;/code&gt; with &lt;code&gt;mode: Terminate&lt;/code&gt;, the Gateway &lt;strong&gt;must&lt;/strong&gt; reference &lt;strong&gt;at least one TLS certificate&lt;/strong&gt;. Without a certificate, the Gateway controller cannot program the listener. Certificates are referenced using the stanza&lt;code&gt;certificateRefs&lt;/code&gt;, which point to Kubernetes &lt;code&gt;Secret&lt;/code&gt; objects of type &lt;code&gt;kubernetes.io/tls&lt;/code&gt;. The &lt;code&gt;Secret&lt;/code&gt; must exist in the same namespace as the &lt;code&gt;Gateway&lt;/code&gt;, contain a valid certificate and private key and of course match the hostnames served by the listener.&lt;/li&gt;
&lt;/ul&gt;
&lt;blockquote&gt;
&lt;p&gt;In a shared Gateway setup, certificate management is typically handled by the platform or infrastructure layer.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;pre&gt;&lt;code&gt;apiVersion: helm.cattle.io/v1  
kind: HelmChartConfig  
metadata:  
  name: traefik  
  namespace: kube-system  
spec:  
  valuesContent: |-  
    providers:  
      kubernetesIngress:  
        enabled: true  
      kubernetesGateway:  
        enabled: true  
    gateway:  
      listeners:  
        web:  
          port: 8000  
          protocol: HTTP  
          namespacePolicy:  
            from: All  
        websecure:  
          port: 8443  
          protocol: HTTPS  
          mode: Terminate  
          certificateRefs:  
            - kind: Secret  
              name: wildcard-cert-tls  
          namespacePolicy:  
            from: All
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Create the &lt;code&gt;wildcard-cert-tls&lt;/code&gt; (see next paragraph) and &lt;strong&gt;then&lt;/strong&gt; restart the K3s service to pick up the changes. It should provision a &lt;code&gt;GatewayClass&lt;/code&gt; and a &lt;code&gt;Gateway&lt;/code&gt; in namespace &lt;em&gt;kube-system&lt;/em&gt;.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;sudo systemctl restart k3s
&lt;/code&gt;&lt;/pre&gt;
&lt;blockquote&gt;
&lt;p&gt;Assuming you are running a Debian-family operating system; otherwise, adjust the command above accordingly.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h3&gt;Create a Wildcard Certificate&lt;/h3&gt;
&lt;p&gt;A wildcard certificate allows a single certificate to cover multiple hostnames under the same domain. For example, a certificate for &lt;code&gt;*.apps.mylab.home&lt;/code&gt; can be used for serving multiple applications e.g.:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;whoami.apps.mylab.home&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;grafana.apps.mylab.home&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;any other subdomain under &lt;code&gt;apps.mylab.home&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This is useful in shared Gateway deployments, where multiple applications attach routes to the same HTTPS listener.&lt;/p&gt;
&lt;p&gt;Let’s lay the create a wildcard certificate for our private published services&lt;/p&gt;
&lt;p&gt;1️⃣ Create a file &lt;strong&gt;wildcard.cnf&lt;/strong&gt; and paste the following:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;[req]  
default_bits       = 4096  
prompt             = no  
default_md         = sha256  
distinguished_name = dn  
x509_extensions    = v3_req  
  
[dn]  
CN = apps.mylab.home  
  
[v3_req]  
subjectAltName = @alt_names  
keyUsage = digitalSignature, keyEncipherment  
extendedKeyUsage = serverAuth  
  
[alt_names]  
DNS.1 = apps.mylab.home  
DNS.2 = *.apps.mylab.home
&lt;/code&gt;&lt;/pre&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;em&gt;Replace&lt;/em&gt; &lt;code&gt;CN&lt;/code&gt;&lt;em&gt;,&lt;/em&gt;&lt;code&gt;DNS.1&lt;/code&gt; &lt;em&gt;and&lt;/em&gt; &lt;code&gt;DNS.2&lt;/code&gt; &lt;em&gt;with the values specific to your environment.&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;2️⃣ Create the Certificate&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;openssl req -x509 -nodes -days 825 \  
  -newkey rsa:4096 \  
  -keyout tls.key \  
  -out tls.crt \  
  -config wildcard.cnf
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;3️⃣ Create the Kubernetes Secret&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;kubectl -n kube-system create secret tls wildcard-cert-tls \  
  --cert=tls.crt \  
  --key=tls.key
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;4️⃣ Create DNS Records&lt;/p&gt;
&lt;p&gt;You have to make sure that the external IPs that are assigned to your Gateway/Ingress (the node IPs in case of a vanilla K3s) are pointing, with or without a floating IP, to &lt;code&gt;*.apps.mylab.home&lt;/code&gt;. Adjust your DNS records accordingly.&lt;/p&gt;
&lt;p&gt;Once both Gateway and Secret are configured, the Gateway can serve HTTPS for any route whose hostname matches the certificate.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;If on the other hand you are planning to expose services to the public, TLS certificates should be issued by a publicly trusted certificate authority (like Let’s Encrypt for instance). In Kubernetes environments, this is commonly handled using &lt;a href=&quot;https://cert-manager.io/&quot;&gt;&lt;strong&gt;cert-manager&lt;/strong&gt;&lt;/a&gt; with a DNS-based ACME challenge (&lt;a href=&quot;https://cert-manager.io/docs/configuration/acme/dns01/&quot;&gt;DNS-01&lt;/a&gt;). DNS validation is required for wildcard certificates and/or for clusters that are not directly reachable over HTTP.
In this setup, cert-manager is responsible for obtaining and renewing the certificate, while the resulting TLS &lt;code&gt;Secret&lt;/code&gt; is referenced by the Gateway listener. The DNS provider configuration (DNS solver) must be able to create and remove validation records for the target domain.
This approach allows certificate management to remain automated and independent of application routing configuration.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h3&gt;Create a Workload&lt;/h3&gt;
&lt;p&gt;We are going to test the whole setting now with deploying and exposing &lt;a href=&quot;https://github.com/traefik/whoami&quot;&gt;traefik/whoami&lt;/a&gt; internally.&lt;/p&gt;
&lt;p&gt;1️⃣ Create and deploy the manifests for the application itself&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;apiVersion: v1  
kind: Namespace  
metadata:  
  name: whoami  
---  
apiVersion: apps/v1  
kind: Deployment  
metadata:  
  name: whoami  
  namespace: whoami  
spec:  
  replicas: 3  
  selector:  
    matchLabels:  
      app: whoami  
  template:  
    metadata:  
      labels:  
        app: whoami  
    spec:  
      containers:  
        - name: whoami  
          image: traefik/whoami:v1.11  
          ports:  
            - name: http  
              containerPort: 80  
---  
apiVersion: v1  
kind: Service  
metadata:  
  name: whoami  
  namespace: whoami  
spec:  
  selector:  
    app: whoami  
  ports:  
    - name: http  
      port: 80  
      targetPort: http
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;2️⃣ Create and deploy the manifests for the Route&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;apiVersion: gateway.networking.k8s.io/v1  
kind: HTTPRoute  
metadata:  
  name: whoami  
  namespace: whoami  
spec:  
  parentRefs:  
    - name: traefik-gateway  
      namespace: kube-system  
  hostnames:  
    - whoami.apps.mylab.home  
  rules:  
    - matches:  
        - path:  
            type: PathPrefix  
            value: /  
      backendRefs:  
        - name: whoami  
          port: 80
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-45e9374392ca.akninsYu_Z8Yflw.webp&quot; alt=&quot;&quot;&gt;&lt;/p&gt;
&lt;p&gt;Of course the Certificate is not trusted as it is a self-signed one.&lt;/p&gt;
&lt;h3&gt;Closing Thoughts&lt;/h3&gt;
&lt;p&gt;Retiring &lt;a href=&quot;https://github.com/kubernetes/ingress-nginx&quot;&gt;&lt;strong&gt;kubernetes/ingress-nginx&lt;/strong&gt;&lt;/a&gt; sets a genuinely bad precedent for the ecosystem because it shows that even &lt;em&gt;foundational&lt;/em&gt; “it’s everywhere” components can become unsustainable and then get shut down without a fully community-owned succession plan.&lt;/p&gt;
&lt;p&gt;There isn’t a single definitive estimate of the blast radius of this event, but the Kubernetes Steering Committee itself in a statement calls it “&lt;em&gt;the piece of critical infrastructure for about&lt;/em&gt; &lt;em&gt;&lt;strong&gt;half&lt;/strong&gt;&lt;/em&gt; &lt;em&gt;of cloud native environments.&lt;/em&gt;”. You get the picture right?&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;So act now, before you need to (re)act under pressure after March 2026&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;As long as Ingress API is getting no new features, sooner or later you’ll need considering moving to Gateway API (unless you have a strong technical reason to do otherewise).&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;⚠&lt;em&gt;️&lt;/em&gt; &lt;strong&gt;This article does not advocate replacing the Kubernetes Ingress NGINX Controller with Traefik Gateway. It simply explains how to configure a K3s cluster to use a shared Gateway, leaving the choice of controller to your specific needs.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Originally published on &lt;a href=&quot;https://medium.com/itnext/configure-a-shared-tr%C3%A6fik-gateway-on-k3s-b09fb0d32c6e&quot;&gt;Medium&lt;/a&gt;&lt;/p&gt;</content:encoded><h:img src="/_astro/thumbnail.BuMfVaWS.jpg"/><enclosure url="/_astro/thumbnail.BuMfVaWS.jpg"/></item><item><title>Managed Kubernetes on Proxmox with CAPI and Kamaji</title><link>https://www.backyard-rack.com/blog/managed-kubernetes-proxmox-capi-kamaji</link><guid isPermaLink="true">https://www.backyard-rack.com/blog/managed-kubernetes-proxmox-capi-kamaji</guid><description>Build a managed Kubernetes service on Proxmox using Cluster API and Kamaji for hosted control planes.</description><pubDate>Thu, 18 Dec 2025 00:00:00 GMT</pubDate><content:encoded>&lt;blockquote&gt;
&lt;p&gt;&lt;em&gt;⚠️&lt;/em&gt; If you are new to &lt;a href=&quot;https://github.com/kubernetes-sigs/cluster-api&quot;&gt;Cluster API (CAPI)&lt;/a&gt;, please refer to my previous article, &lt;a href=&quot;/blog/managed-kubernetes-proxmox-capi&quot;&gt;Build your own Managed Kubernetes Service on Proxmox with CAPI&lt;/a&gt;, for a detailed introduction.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h2&gt;Prerequisites&lt;/h2&gt;
&lt;p&gt;For this lab you need to bring with:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;a single-node or even better a multi-node &lt;a href=&quot;https://www.proxmox.com/&quot;&gt;Proxmox&lt;/a&gt; cluster. &lt;em&gt;Installing and configuring the Proxmox cluster itself is&lt;/em&gt; &lt;em&gt;&lt;strong&gt;out&lt;/strong&gt;&lt;/em&gt; &lt;em&gt;of the scope of this article&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;Relevant hands-on experience on Proxmox and Kubernetes.&lt;/li&gt;
&lt;li&gt;Relevant hands-on experience with CAPI.&lt;/li&gt;
&lt;li&gt;A workstation with homebrew, kubectl, &lt;a href=&quot;https://cluster-api.sigs.k8s.io/clusterctl/overview.html&quot;&gt;clusterctl&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;A virtual machine with &lt;a href=&quot;https://k3s.io/&quot;&gt;k3s&lt;/a&gt; (4 vCPUs, 6GB RAM no ballooning) for implementing the Kamaji part of the lab (In this case, the management cluster cannot be illustrated using &lt;a href=&quot;https://kind.sigs.k8s.io/&quot;&gt;kind&lt;/a&gt;, unlike in the previous article.&lt;/li&gt;
&lt;/ol&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;em&gt;⚠️&lt;/em&gt; This lab requires Proxmox version &lt;code&gt;8.4.12&lt;/code&gt;. Earlier versions exhibit issues with the integration with the &lt;a href=&quot;https://github.com/ionos-cloud/cluster-api-provider-proxmox&quot;&gt;CAPMOX&lt;/a&gt; provider, and later versions have not been tested by this guide.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;In this lab we are going to demonstrate how you could create your own Managed Kubernetes Service using CAPI with Proxmox as infrastructure provider, &lt;a href=&quot;https://kubernetes.io/docs/reference/setup-tools/kubeadm/&quot;&gt;kubeadm&lt;/a&gt; as bootstrap provider and for the role of control plane provider we are going to use &lt;a href=&quot;https://kamaji.clastix.io/&quot;&gt;Kamaji&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-43824dd5fa0f.C9qyU0Ly_ZetDB3.webp&quot; alt=&quot;&quot;&gt;&lt;/p&gt;
&lt;h2&gt;What is Kamaji?&lt;/h2&gt;
&lt;p&gt;&lt;a href=&quot;https://github.com/clastix/kamaji&quot;&gt;Kamaji&lt;/a&gt; is an open-source project developed by &lt;a href=&quot;https://clastix.io/&quot;&gt;Clastix&lt;/a&gt; that implements the Kubernetes control plane as a hosted service running entirely on Kubernetes itself. Instead of provisioning dedicated virtual machines or bare-metal servers for etcd, kube-apiserver, the controller manager, and the scheduler, Kamaji runs these components as Pods within a management cluster and exposes them to tenant clusters through a lightweight API. This hosted control plane model decouples control plane lifecycle management from the underlying infrastructure, significantly reducing operational overhead and resource duplication.&lt;/p&gt;
&lt;p&gt;Under the hood, Kamaji introduces two new API groups. The &lt;code&gt;TenantControlPlane&lt;/code&gt; resource, which defines the desired configuration for a control plane: Kubernetes version, certificate settings, replica count, scheduling constraints and exposure method (ClusterIP, LoadBalancer or NodePort). Separately, a &lt;code&gt;DataStore&lt;/code&gt; resource manages the backing store for one or more tenant control planes, which is by default &lt;a href=&quot;https://etcd.io/&quot;&gt;etcd&lt;/a&gt;, but optionally other better scaling alternatives such as MySQL, PostgreSQL or NATS (via Kine) can used as a replacement.&lt;/p&gt;
&lt;h2&gt;Preparing your Proxmox Users and API Tokens&lt;/h2&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;em&gt;❗️&lt;/em&gt;You can skip this section if you have already completed the setup by following the steps outlined in the previous article.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Before we can build images, the image-builder needs permission to interact with your Proxmox environment through its API. This access allows it to launch a virtual machine, perform the build process, and then turn that VM into a reusable template. Execute the following command in one of your Proxmox nodes:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;pveum user add image-builder@pve
pveum aclmod / -user image-builder@pve -role PVEAdmin
pveum user token add image-builder@pve capi -privsep 0
pveum user add capmox@pve
pveum aclmod / -user capmox@pve -role PVEAdmin
pveum user token add capmox@pve capi -privsep 0
&lt;/code&gt;&lt;/pre&gt;
&lt;h2&gt;Building Images for Proxmox with image-builder&lt;/h2&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;em&gt;‼️&lt;/em&gt;Given that this is a quite lengthy configuration step, and to avoid cluttering the article or making it unnecessarily long, please refer to the corresponding section in the previous article, if you have not already completed it. Otherwise, you can safely skip this step.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h2&gt;Installing Prerequisites&lt;/h2&gt;
&lt;blockquote&gt;
&lt;p&gt;To maintain a single guide for both macOS and Linux, we’ll use &lt;a href=&quot;https://brew.sh/&quot;&gt;Homebrew&lt;/a&gt; to install the binaries required for managing the cluster.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;1️⃣ Install Homebrew&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;/bin/bash -c &quot;$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)&quot;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Follow the instructions of the installer, to complete the installation.&lt;/p&gt;
&lt;p&gt;2️⃣ Install kubectl&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;brew install kubectl
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;3️⃣ Install clusterctl&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;brew install clusterctl
&lt;/code&gt;&lt;/pre&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;em&gt;⚠️&lt;/em&gt; This lab requires clusterctl version &lt;code&gt;v1.11.3&lt;/code&gt; or later.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;The &lt;code&gt;clusterctl&lt;/code&gt; CLI delivers a streamlined “day one” experience for CAPI by automatically retrieving and installing the necessary provider manifests. It embeds recommended best practices for provider management, helping you avoid mis-configurations and simplifying tasks like upgrades.&lt;/p&gt;
&lt;p&gt;Built on top of &lt;code&gt;clusterctl&lt;/code&gt;, the Cluster API Operator is a Kubernetes operator that brings a fully declarative workflow for managing provider lifecycles within your management cluster. It enhances the deployment and day-to-day operation of Cluster API, making routine tasks and GitOps-driven automation effortless.&lt;/p&gt;
&lt;h2&gt;Create Configuration for Proxmox&lt;/h2&gt;
&lt;p&gt;Create a a file named &lt;code&gt;capmox.env&lt;/code&gt;, and fill it in with the following content:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;export PROXMOX_URL=&quot;https://&amp;#x3C;PROXMOX_NODE_IP&gt;:8006&quot;
export PROXMOX_TOKEN=&apos;capmox@pve!capi&apos;
export PROXMOX_SECRET=&quot;&amp;#x3C;CAPMOX_API_TOKEN_SECRET&gt;&quot;                          
export PROXMOX_SKIP_TLS_VERIFY=&quot;true&quot;
&lt;/code&gt;&lt;/pre&gt;
&lt;blockquote&gt;
&lt;p&gt;Replace every occurrence of &lt;code&gt;&amp;#x3C;PROXMOX_NODE_IP&gt;&lt;/code&gt; with the IP address and CIDR corresponding to your bridge interface (&lt;code&gt;vmbr0&lt;/code&gt;, for this lab).&lt;/p&gt;
&lt;/blockquote&gt;
&lt;blockquote&gt;
&lt;p&gt;Do the same for &lt;code&gt;&amp;#x3C;CAPMOX_API_TOKEN_SECRET&gt;&lt;/code&gt;, using the appropriate value for your environment.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h2&gt;Setting up a Management Cluster&lt;/h2&gt;
&lt;p&gt;This is the cluster that runs one or more Infrastructure Providers and holds resources (for example, &lt;code&gt;Machines&lt;/code&gt;), when provisioning multiple workload clusters.&lt;/p&gt;
&lt;p&gt;⏩ &lt;strong&gt;SSH into the virtual machine that will host the K3s cluster.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;1️⃣ Create the cluster&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC=&quot;--disable traefik --disable servicelb&quot; sh -

sudo chmod 644 /etc/rancher/k3s/k3s.yaml
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown &amp;#x3C;USER&gt;:&amp;#x3C;USER&gt; ~/.kube/config
&lt;/code&gt;&lt;/pre&gt;
&lt;blockquote&gt;
&lt;p&gt;Replace &lt;code&gt;&amp;#x3C;USER&gt;&lt;/code&gt; with the appropriate user and group for your environment.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;If the management cluster is running on a different virtual machine, copy &lt;code&gt;~/.kube/config&lt;/code&gt; to your bootstrap machine where the required prerequisites (Homebrew, kubectl, and clusterctl) are already installed.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Make sure you change the &lt;code&gt;server&lt;/code&gt; value from &lt;code&gt;https://127.0.0.1:6443&lt;/code&gt; to the IP address assigned on the network device that uses the bridge &lt;code&gt;vmbr0&lt;/code&gt;.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;⏩ &lt;strong&gt;Switch now to your bootstrap machine.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;2️⃣ Install Cert-Manager&lt;/p&gt;
&lt;p&gt;Kamaji relies on Kubernetes dynamic admission control mechanisms, including validating and mutating webhook configurations. These webhooks communicate over TLS, with certificates managed by &lt;a href=&quot;https://github.com/cert-manager/cert-manager&quot;&gt;cert-manager&lt;/a&gt;, which therefore must be installed as a prerequisite.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-cpp&quot;&gt;helm repo add jetstack https://charts.jetstack.io
helm repo update

helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set installCRDs=true
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;3️⃣ Install the Kamaji Operator&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;helm repo add clastix https://clastix.github.io/charts
helm repo update

helm install kamaji clastix/kamaji \
    --version 0.0.0+latest \
    --namespace kamaji-system \
    --create-namespace \
    --set image.tag=latest
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;4️⃣ Install MetalLB&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;em&gt;⚠️&lt;/em&gt; Kamaji provisions the control plane as a set of Pods exposed through a Service. This Service must be assigned an IP address within the network defined by the &lt;code&gt;vmbr0&lt;/code&gt; interface so that it is reachable by worker nodes and by consumers of the cluster. For this reason, a solution such as &lt;a href=&quot;https://github.com/metallb/metallb&quot;&gt;MetalLB&lt;/a&gt; is required. Other alternatives may also be suitable; e.g., &lt;a href=&quot;https://github.com/kube-vip/kube-vip&quot;&gt;kube-vip&lt;/a&gt; or &lt;a href=&quot;https://github.com/k3s-io/klipper-lb&quot;&gt;klipper&lt;/a&gt; the embedded K3s service load balancer which in our case would mean you need to reinstall the cluster, since it was provisioned &lt;strong&gt;without&lt;/strong&gt; Traefik and Klipper.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;pre&gt;&lt;code class=&quot;language-yaml&quot;&gt;kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.3/config/manifests/metallb-native.yaml

kubectl apply -f - &amp;#x3C;&amp;#x3C;EOF
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.1.240-192.168.1.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: default-advertisement
  namespace: metallb-system
spec:
  ipAddressPools:
    - default-pool
EOF
&lt;/code&gt;&lt;/pre&gt;
&lt;blockquote&gt;
&lt;p&gt;️Replace the IP address pool range, in this case &lt;code&gt;192.168.1.240–192.168.1.250&lt;/code&gt;, so it matches your environment. Ensure that the selected IP range belongs to the &lt;code&gt;vmbr0&lt;/code&gt; interface network.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;5️⃣ Transform the cluster to a Management Cluster&lt;/p&gt;
&lt;p&gt;With clusterctl installed and prerequisites satisfied, run &lt;code&gt;clusterctl init&lt;/code&gt; to convert your Kubernetes cluster into a management cluster.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;source capmox.env

clusterctl init \
  --infrastructure proxmox \ 
  --ipam in-cluster \ 
  --control-plane kamaji
&lt;/code&gt;&lt;/pre&gt;
&lt;blockquote&gt;
&lt;p&gt;The command above will install &lt;a href=&quot;https://github.com/ionos-cloud/cluster-api-provider-proxmox/&quot;&gt;Cluster API Provider for Proxmox VE&lt;/a&gt; (CAPMOX) as infrastructure provider, Kubeadm as bootstrap provider and Kamaji as control-plane provider.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-c780edb49580.6p4awaHC_Z161MyC.webp&quot; alt=&quot;&quot;&gt;&lt;/p&gt;
&lt;h2&gt;Creating a Cluster with the Kamaji CP Provider&lt;/h2&gt;
&lt;p&gt;When you use a &lt;code&gt;KamajiControlPlane&lt;/code&gt; alongside your &lt;code&gt;Cluster&lt;/code&gt; and &lt;code&gt;Machine&lt;/code&gt; objects, the Kamaji control-plane provider watches for that resource and &lt;strong&gt;spins up the Kubernetes control-plane components as Pods in your management cluster&lt;/strong&gt;. It provisions etcd, generates and rotates certificates, and configures the API server, controller-manager and scheduler according to the specs in the &lt;code&gt;KamajiControlPlane&lt;/code&gt;. As Pod replicas become ready, Kamaji wires them into a highly available endpoint and exposes it (ClusterIP, LoadBalancer or NodePort), while your worker nodes simply run &lt;code&gt;kubeadm join&lt;/code&gt; against that hosted control plane, giving you &lt;strong&gt;a fully managed, multi-tenant control plane without requiring dedicated VMs&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-6467c721ac80.CCVpp9LR_Z1Oin8U.webp&quot; alt=&quot;&quot;&gt;
*Multiple Control Planes are running as Kubernetes Pods while their Worker Nodes are Proxmox VMs .*&lt;/p&gt;
&lt;p&gt;1️⃣ Create Configuration for Kamaji&lt;/p&gt;
&lt;p&gt;Before creating a tenant Kubernetes cluster, some configuration is required on the management cluster for Kamaji. Create a a file named &lt;code&gt;kamaji.env&lt;/code&gt;, and fill it in with the following content:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;# Cluster Configuration
export CLUSTER_NAME=&quot;capi-kamaji-quickstart&quot;
export CLUSTER_NAMESPACE=&quot;default&quot;
export CONTROL_PLANE_REPLICAS=3
export KUBERNETES_VERSION=&quot;v1.33.1&quot;
export CLUSTER_DATASTORE=&quot;default&quot;

# Networking Configuration
export IP_RANGE=&apos;[&quot;192.168.1.230-192.168.1.239&quot;]&apos;
export IP_PREFIX=24
export GATEWAY=&quot;192.168.1.1&quot;
export DNS_SERVERS=&apos;[&quot;192.168.1.1&quot;,&quot;8.8.8.8&quot;]&apos;
export NETWORK_BRIDGE=&quot;vmbr0&quot;
export NETWORK_MODEL=&quot;virtio&quot;

# Node Configuration
export SSH_USER=&quot;&amp;#x3C;USER&gt;&quot;
export SSH_AUTHORIZED_KEY=&quot;ssh-rsa AAAAB...&quot;
export NODE_REPLICAS=2
export NODE_LABELS=
export NODE_TAINTS=

# Resource Configuration
export SOURCE_NODE=&quot;pve-01&quot;
export TEMPLATE_ID=101
export ALLOWED_NODES=&apos;[&quot;pve-01&quot;, &quot;pve-02&quot;]&apos;
export MEMORY_MIB=&quot;2048&quot;                                            
export NUM_CORES=2
export NUM_SOCKETS=1
export BOOT_VOLUME_DEVICE=&quot;scsi0&quot;
export BOOT_VOLUME_SIZE=20
export FILE_STORAGE_FORMAT=&quot;raw&quot;
export STORAGE_NODE=&quot;local-lvm&quot;

export CLOUD_INIT_CONFIG=&quot;#cloud-config package_update=true packages=- net-tools&quot;
&lt;/code&gt;&lt;/pre&gt;
&lt;blockquote&gt;
&lt;p&gt;Replace &lt;code&gt;IP_RANGE&lt;/code&gt;, &lt;code&gt;GATEWAY&lt;/code&gt;, &lt;code&gt;DNS_SERVERS&lt;/code&gt;, &lt;code&gt;NETWORK_BRIDGE&lt;/code&gt;, &lt;code&gt;SSH_USER&lt;/code&gt;, &lt;code&gt;SSH_AUTHORIZED_KEY&lt;/code&gt;, &lt;code&gt;SOURCE_NODE&lt;/code&gt;, &lt;code&gt;TEMPLATE_ID&lt;/code&gt;, &lt;code&gt;ALLOWED_NODES&lt;/code&gt; values and any other variable for that matter, so they match &lt;strong&gt;your&lt;/strong&gt; environment.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;2️⃣ Generate the manifests&lt;/p&gt;
&lt;p&gt;Download the &lt;code&gt;capi-kamaji-proxmox-template.yaml&lt;/code&gt; template file and save it in your working directory, then use the following commands to generate a cluster manifest from this template.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;source kamaji.dev 

clusterctl generate cluster $CLUSTER_NAME \
    --from capi-kamaji-proxmox-template.yaml \
    &gt; capi-kamaji-proxmox-cluster.yaml
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;3️⃣ Allow CAPMOX to over-commit memory ⚠️&lt;/p&gt;
&lt;p&gt;When the infrastructure controller provisions a new VM on Proxmox, it first sums the RAM already allocated on the target host to determine if your &lt;code&gt;MachineTemplate&lt;/code&gt; will fit. If you haven’t explicitly enabled memory over-commit, this check will refuse to schedule any VM once the sum of allocated RAM reaches the node’s nominal capacity, causing new VMs to hang even though Proxmox could safely oversubscribe memory in practice.&lt;/p&gt;
&lt;p&gt;For that matter, add the following properties to the &lt;code&gt;ProxmoxCluster&lt;/code&gt; resource:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-yaml&quot;&gt;schedulerHints:
    memoryAdjustment: 0
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;For example:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-yaml&quot;&gt;apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: ProxmoxCluster
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: tenant-01-kamaji
  name: tenant-01-kamaji
  namespace: default
spec:
  ...
  dnsServers:
    - 192.168.1.1
    - 8.8.8.8
  ...
  schedulerHints:
    memoryAdjustment: 0
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;4️⃣ Create the cluster&lt;/p&gt;
&lt;p&gt;Execute the command:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;kubectl apply -f capi-kamaji-proxmox-cluster.yaml
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-4796dfacd3dd.DRTKjr7P_2hb8aA.webp&quot; alt=&quot;&quot;&gt;
*The worker nodes are being provisioned in Proxmox.*&lt;/p&gt;
&lt;p&gt;This process will take a few minutes. Once it completes, retrieve the &lt;strong&gt;kubeconfig&lt;/strong&gt; file for the newly created cluster by running:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;clusterctl get kubeconfig capi-kamaji-quickstart &gt; capi-kamaji-quickstart.kubeconfig
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;5️⃣ Deploy a CNI&lt;/p&gt;
&lt;p&gt;For demonstration purposes, &lt;a href=&quot;https://docs.tigera.io/calico/latest/about/&quot;&gt;Calico&lt;/a&gt; will be deployed as the CNI for the newly created cluster. In your lab, you may choose a different CNI if preferred. Execute the following command:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;kubectl --kubeconfig=./capi-kamaji-quickstart.kubeconfig \
  apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml
&lt;/code&gt;&lt;/pre&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;em&gt;When issues arise, you can access any node directly by opening an SSH session as the&lt;/em&gt; &lt;code&gt;root&lt;/code&gt; &lt;em&gt;user. This allows you to inspect system logs, review configuration files, and run diagnostic commands on each machine.&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-0053b9569ee6.BKMfENYj_ZFzGfH.webp&quot; alt=&quot;&quot;&gt;
*The workload cluster is now ready for use!*&lt;/p&gt;
&lt;p&gt;6️⃣ Install Kamaji Console (Optional)&lt;/p&gt;
&lt;p&gt;The &lt;a href=&quot;https://kamaji.clastix.io/guides/console/&quot;&gt;&lt;strong&gt;Kamaji Console&lt;/strong&gt;&lt;/a&gt; is a web-based user interface provided by the Kamaji project to observe and manage hosted Kubernetes control planes managed by Kamaji. There is limited functionality available in the free version, but it is included here for completeness.&lt;/p&gt;
&lt;p&gt;Before installing the console, you must provision the required Secret:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-yaml&quot;&gt;cat &amp;#x3C;&amp;#x3C;EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: kamaji-console
  namespace: kamaji-system
data:
  ADMIN_EMAIL: &amp;#x3C;ADMIN_EMAIL&gt;
  ADMIN_PASSWORD: &amp;#x3C;ADMIN_PASSWORD&gt;
  JWT_SECRET: &amp;#x3C;JWT_SECRET&gt;
  NEXTAUTH_URL: &amp;#x3C;NEXTAUTH_URL&gt;
EOF
&lt;/code&gt;&lt;/pre&gt;
&lt;blockquote&gt;
&lt;p&gt;Replace all values of the variables under the &lt;code&gt;data&lt;/code&gt; stanza so they match &lt;strong&gt;your&lt;/strong&gt; environment.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;And then install the console via its Helm Chart:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-javascript&quot;&gt;helm repo add clastix https://clastix.github.io/charts
helm repo update

helm install console clastix/kamaji-console -n kamaji-system
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-694d6105f631.he6pYTxj_1F1qbs.webp&quot; alt=&quot;&quot;&gt;&lt;/p&gt;
&lt;h2&gt;Next Steps&lt;/h2&gt;
&lt;p&gt;In the next article, we will explore how to use Kamaji together with tools such as &lt;a href=&quot;https://argo-cd.readthedocs.io/en/stable/&quot;&gt;ArgoCD&lt;/a&gt; and &lt;a href=&quot;https://sveltos.projectsveltos.io/&quot;&gt;Sveltos&lt;/a&gt; to take advantage of the automation capabilities provided by GitOps.&lt;/p&gt;
&lt;p&gt;Originally published on &lt;a href=&quot;https://medium.com/itnext/build-your-own-managed-kubernetes-service-on-proxmox-with-capi-and-kamaji-7858ad41cb09&quot;&gt;Medium&lt;/a&gt;&lt;/p&gt;</content:encoded><h:img src="/_astro/thumbnail.BQetYe3G.png"/><enclosure url="/_astro/thumbnail.BQetYe3G.png"/></item><item><title>Managed Kubernetes on Proxmox with Cluster API</title><link>https://www.backyard-rack.com/blog/managed-kubernetes-proxmox-capi</link><guid isPermaLink="true">https://www.backyard-rack.com/blog/managed-kubernetes-proxmox-capi</guid><description>Build your own Kubernetes-as-a-Service platform on Proxmox using Cluster API and kubeadm.</description><pubDate>Sat, 18 Oct 2025 00:00:00 GMT</pubDate><content:encoded>&lt;h2&gt;What is CAPI?&lt;/h2&gt;
&lt;p&gt;&lt;a href=&quot;https://github.com/kubernetes-sigs/cluster-api&quot;&gt;Cluster API&lt;/a&gt; (CAPI) is an open-source Kubernetes sub-project. Its goal is to bring Kubernetes-style, declarative APIs and controllers to the problem of bootstrapping, configuring, upgrading and operating entire Kubernetes clusters, treating clusters themselves (and the machines that compose them) as first-class Kubernetes resources rather than as external, manually-provisioned infrastructure.&lt;/p&gt;
&lt;p&gt;CAPI is deployed to a Kubernetes cluster, it leverages your familiar toolchain (kubectl, CRDs, manifests) to manage clusters at scale. When you request a new cluster, via a new custom resource named &lt;code&gt;Cluster&lt;/code&gt;, a &lt;strong&gt;provider-specific controller&lt;/strong&gt; provisions the necessary resources in the target infrastructure. As cluster demands change or when upgrades are required, simply updating the Kubernetes manifests and reapplying them triggers rolling updates or scale-out operations, all managed by the same reconciliation loops that govern standard Kubernetes workloads.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-8c265b60d7a5.CvR-OLXa_Z2rxKXh.webp&quot; alt=&quot;&quot;&gt;&lt;/p&gt;
&lt;p&gt;Adopting CAPI brings consistency and repeatability to cluster operations. Cluster operators can define golden-image references, networking topologies and scaling policies in version-controlled manifests. The reconciliation-driven approach ensures that drift is corrected automatically, while upgrade strategies (for example, in-place control-plane upgrades followed by a rolling reboot of worker nodes) can be codified and reused. This makes Cluster API especially valuable for organizations managing hundreds of clusters across multiple teams, regions or infrastructure providers, including public clouds and on-premises environments.&lt;/p&gt;
&lt;h2&gt;How is CAPI working?&lt;/h2&gt;
&lt;p&gt;CAPI is built around a few key abstractions and components that work together to model and manage Kubernetes clusters declaratively. At a high level, you have:&lt;/p&gt;
&lt;h3&gt;Management &amp;#x26; Workload Clusters&lt;/h3&gt;
&lt;p&gt;A &lt;em&gt;management cluster&lt;/em&gt; is a regular Kubernetes control plane where the CAPI controllers run. It holds all of the Cluster API Custom Resources (CRs) and drives the lifecycle of one or more &lt;em&gt;workload clusters;&lt;/em&gt; the actual Kubernetes clusters you are going to use for running your applications.&lt;/p&gt;
&lt;h3&gt;Custom Resource Definitions (CRDs)&lt;/h3&gt;
&lt;p&gt;CAPI introduces its own CRDs to represent clusters and the machines that comprise them:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;Cluster&lt;/code&gt;&lt;strong&gt;:&lt;/strong&gt; The top-level object defining a Kubernetes cluster (networking, Kubernetes version, etc.).&lt;/li&gt;
&lt;li&gt;&lt;code&gt;Machine&lt;/code&gt;, &lt;code&gt;MachineSet&lt;/code&gt; and &lt;code&gt;MachineDeployment&lt;/code&gt;&lt;strong&gt;:&lt;/strong&gt; Machines map to individual VMs or bare-metal hosts. A MachineSet ensures a stable number of Machines, while a MachineDeployment provides rolling-update semantics over MachineSets (much like a Deployment for Pods).&lt;/li&gt;
&lt;li&gt;&lt;code&gt;ControlPlane&lt;/code&gt;: Encapsulates control-plane-specific settings (certificate management, high-availability, version upgrades) and drives creation or scaling of control-plane Machines (the default ControlPlane provider is &lt;code&gt;KubeadmControlPlane&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;&lt;code&gt;MachineHealthCheck&lt;/code&gt;: Defines criteria for identifying unhealthy Nodes and triggers automatic replacement of the corresponding Machines.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;Provider Components&lt;/h3&gt;
&lt;p&gt;The reconciliation logic is split, abstracted and delegated into three &lt;strong&gt;pluggable&lt;/strong&gt; provider types, each implemented by its very own controller manager:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Infrastructure provider:&lt;/strong&gt; Talks to the target cloud or virtualization API (AWS, Azure, GCP, OpenStack, Proxmox, VMware etc.) to create and delete VMs, networks, load balancers and related resources.&lt;/li&gt;
&lt;/ul&gt;
&lt;blockquote&gt;
&lt;p&gt;You can find an indicative, but not extensive list, of the most common providers here: &lt;a href=&quot;https://image-builder.sigs.k8s.io/capi/capi#providers&quot;&gt;https://image-builder.sigs.k8s.io/capi/capi#providers&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Bootstrap provider:&lt;/strong&gt; Prepares a newly-provisioned VM to become a Kubernetes Node: generating certs, initializing the control plane on the first Machine, and running &lt;code&gt;kubeadm join&lt;/code&gt; on subsequent control-plane or worker Machines.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;ControlPlane provider:&lt;/strong&gt; Manages the lifecycle of control-plane Machines themselves (implementing rolling upgrades of API servers via e.g.&lt;code&gt;KubeadmControlPlane&lt;/code&gt;).&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;Controllers &amp;#x26; Reconciliation&lt;/h3&gt;
&lt;p&gt;For each CRD, a corresponding controller watches the &lt;code&gt;.spec&lt;/code&gt; you declare, compares it to the actual state in the cloud and Kubernetes, and takes actions to converge them. If you increase the replica count of a&lt;code&gt;MachineDeployment&lt;/code&gt;, the &lt;code&gt;MachineDeployment&lt;/code&gt; controller will create a new &lt;code&gt;MachineSet&lt;/code&gt; (or update an existing one), which in turn lets the Infrastructure and Bootstrap controllers provision and join new VMs. If someone manually deletes a VM, the next reconciliation cycle notices the missing Machine object and reprovisions it automatically. Finalizers ensure that when you delete a &lt;code&gt;Cluster&lt;/code&gt; resource, machines and cloud resources are torn down in the proper order before the &lt;code&gt;Cluster&lt;/code&gt; object is removed.&lt;/p&gt;
&lt;h2&gt;Prerequisites&lt;/h2&gt;
&lt;p&gt;For this lab you need to bring with:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;a &lt;a href=&quot;https://www.proxmox.com/&quot;&gt;Proxmox&lt;/a&gt; installation (single-node or even better a multi-node cluster). &lt;em&gt;Installing and configuring Proxmox is&lt;/em&gt; &lt;em&gt;&lt;strong&gt;out&lt;/strong&gt;&lt;/em&gt; &lt;em&gt;of the scope of this article&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;Relevant hands-on experience on Proxmox and Kubernetes.&lt;/li&gt;
&lt;li&gt;A workstation with homebrew, kubectl, clusterctl and kind.&lt;/li&gt;
&lt;li&gt;A virtual machine with &lt;a href=&quot;https://k3s.io/&quot;&gt;k3s&lt;/a&gt; for implementing the management cluster.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;In this lab we are going to demonstrate how you could create your own Managed Kubernetes Service using CAPI with Proxmox as infrastructure provider, kubeadm as bootstrap provider and for the role of control plane provider we are going to use &lt;strong&gt;kubeadm&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-90a267c817b3.BXVZf8ZI_Z2jbUOR.webp&quot; alt=&quot;&quot;&gt;&lt;/p&gt;
&lt;h2&gt;Preparing your Proxmox Users and API Tokens&lt;/h2&gt;
&lt;p&gt;Before we can build images, the image-builder needs permission to interact with your Proxmox environment through its API. This access allows it to launch a virtual machine, perform the build process, and then turn that VM into a reusable template. Execute the following command in one of your Proxmox nodes:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;pveum user add image-builder@pve
pveum aclmod / -user image-builder@pve -role PVEAdmin
pveum user token add image-builder@pve capi -privsep 0

pveum user add capmox@pve
pveum aclmod / -user capmox@pve -role PVEAdmin
pveum user token add capmox@pve capi -privsep 0
&lt;/code&gt;&lt;/pre&gt;
&lt;h2&gt;Building Images for Proxmox with image-builder&lt;/h2&gt;
&lt;p&gt;With &lt;a href=&quot;https://github.com/kubernetes-sigs/image-builder&quot;&gt;image-builder&lt;/a&gt; we try to standardize and automate the creation of virtual machine images that are pre-configured for Kubernetes and Cluster API, ensuring consistency, security, and speed during cluster provisioning. Rather than relying on untested or manually assembled OS snapshots, image-builder uses declarative “recipes” (built with tools like &lt;a href=&quot;https://developer.hashicorp.com/packer&quot;&gt;Packer&lt;/a&gt; and &lt;a href=&quot;https://www.redhat.com/en/ansible-collaborative&quot;&gt;Ansible&lt;/a&gt;) to bake in all the required components as kernel settings, container runtime, kubeadm binaries, networking plugins, and any custom security hardening into one baseline image. These images then serve as the immutable base for all control-plane and worker nodes, eliminating variability and drift across environments.&lt;/p&gt;
&lt;p&gt;By producing provider-specific artifacts, image-builder decouples the image creation lifecycle from cluster creation. The result is faster cluster bootstraps, reproducible upgrades aligned to Kubernetes versions, and a GitOps-friendly workflow where image updates propagate through your pipelines just like application code.&lt;/p&gt;
&lt;p&gt;So let’s perform the following steps to install, configure and create or base line with the image-builder:&lt;/p&gt;
&lt;p&gt;1️⃣ Clone (or fork and clone) the repo in your workstation:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;git clone git@github.com:kubernetes-sigs/image-builder.git
cd image-builder/images/capi
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;2️⃣ Install the dependencies:&lt;/p&gt;
&lt;p&gt;In order to install the dependencies automatically execute the following command:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;make deps
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This will install dependencies in the folder&lt;code&gt;image-builder/images/capi/.bin&lt;/code&gt; (if they are not already on your system). Then you need to add that to your &lt;code&gt;PATH&lt;/code&gt; environment variable before they can be used:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;export PATH=$PWD/.bin:$PATH
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;3️⃣ Configure for Proxmox&lt;/p&gt;
&lt;p&gt;Add a new file at&lt;code&gt;image-builder/images/capi/packer/proxmox/.env&lt;/code&gt; and provide the required environment variables, replacing with your own values where necessary:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-typescript&quot;&gt;export PROXMOX_URL=&quot;https://&amp;#x3C;PROXMOX_NODE_IP&gt;:8006/api2/json&quot;
export PROXMOX_USERNAME=image-builder@pve!capi
export PROXMOX_TOKEN=&amp;#x3C;IMAGE_BUILD_API_TOKEN&gt;
export PROXMOX_NODE=&quot;&amp;#x3C;PROXMOX_NODE_NAME&gt;&quot;
export PROXMOX_ISO_POOL=&quot;local&quot;
export PROXMOX_STORAGE_POOL=&quot;&amp;#x3C;PROXMOX_STORAGE_POOL&gt;&quot;
export DISK_FORMAT=&quot;&amp;#x3C;DISK_FORMAT&gt;&quot;
&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code class=&quot;language-text&quot;&gt;export PROXMOX_BRIDGE=&quot;vmbr0&quot;
export PROXMOX_NIC_MODEL=&quot;virtio&quot;
&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code class=&quot;language-text&quot;&gt;# If you are working with OPNSense, image-builder is not 
# assigning a Proxmox MAC address to the VM, and this seems 
# to be an issue with DHCP in VXLANs controlled by OPNSense!
# In theory you can configure your MTU and VLAN tag with the 
# following variables. For the time being leave them commented.
&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code class=&quot;language-text&quot;&gt;# export PROXMOX_MTU=&quot;1450&quot;
# export PROXMOX_VLAN=&quot;23&quot;
&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code class=&quot;language-text&quot;&gt;export PACKER_FLAGS=&quot;--var memory=2048 --var &apos;kubernetes_rpm_version=1.32.5&apos; --var &apos;kubernetes_semver=v1.32.5&apos; --var &apos;kubernetes_series=v1.32&apos; --var &apos;kubernetes_deb_version=1.32.5-1.1&apos;&quot;
&lt;/code&gt;&lt;/pre&gt;
&lt;blockquote&gt;
&lt;p&gt;Make sure you add the &lt;code&gt;.env&lt;/code&gt; file in your &lt;code&gt;.gitignore&lt;/code&gt; in order to avoid leaking API tokens when you commit the changes to your repo (if you have forked it in the first place).
The environment variables &lt;code&gt;PROXMOX_ISO_POOL&lt;/code&gt;, &lt;code&gt;PROXMOX_BRIDGE&lt;/code&gt;,&lt;code&gt;PROXMOX_STORAGE_POOL&lt;/code&gt; here are optional, just to demonstrate how to override the default values, because we might need to adjust some of them later on for a multi-node Proxmox cluster scenario.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;If you want to bake an image for a specific Kubernetes version (or pass specific variables to parameterize the baseline image) use the &lt;code&gt;PACKER_FLAGS&lt;/code&gt; variable. You can omit altogether but I don’t recommend it until you fully grasp how image-builder utilizes Packer behind the scenes.&lt;/p&gt;
&lt;p&gt;4️⃣ Configure &lt;code&gt;PROXMOX_STORAGE_POOL&lt;/code&gt; and &lt;code&gt;DISK_FORMAT&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;⚠️ &lt;em&gt;Very Important:&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;If you are running on a &lt;strong&gt;single-node Proxmox&lt;/strong&gt; installation you need to set &lt;code&gt;PROXMOX_STORAGE_POOL&lt;/code&gt; value to &lt;code&gt;local-lvm&lt;/code&gt;. Consequently, because &lt;code&gt;local-lvm&lt;/code&gt; is not accepting &lt;code&gt;qcow2&lt;/code&gt; disk formats, you need to set &lt;code&gt;DISK_FORMAT&lt;/code&gt; value to &lt;code&gt;raw&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;If, on the other hand, you are running a &lt;strong&gt;multi-node Proxmox cluster&lt;/strong&gt;, you need to set &lt;code&gt;PROXMOX_STORAGE_POOL&lt;/code&gt; value as the name of a storage &lt;strong&gt;that can be shared among nodes,&lt;/strong&gt; otherwise you will not be able to spin control-plane and worker nodes on other nodes than the one you uploaded your baseline image. In addition to that, you have to configure that very storage to accept &lt;code&gt;qcow2&lt;/code&gt; format. Set &lt;code&gt;DISK_FORMAT&lt;/code&gt; value to &lt;code&gt;qcow2&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;📓 &lt;em&gt;A short side-note:&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;In my case, I run a &lt;a href=&quot;https://www.truenas.com/&quot;&gt;TrueNAS&lt;/a&gt; server as a virtual machine in one of the Proxmox nodes and I expose an NFS share which I add as storage in Proxmox (funny circle I know; performance is not great so I do recommend building an external NAS if you have the budget).&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-dd2b751ff910.CDvmzlQJ_1GKIUa.webp&quot; alt=&quot;&quot;&gt;&lt;/p&gt;
&lt;p&gt;Installing a separate TrueNAS, is an effort worth investing in, as you could externalize &lt;strong&gt;all&lt;/strong&gt; the storage demands of &lt;strong&gt;all&lt;/strong&gt; your Kubernetes clusters to TrueNAS by using &lt;a href=&quot;https://github.com/democratic-csi/democratic-csi&quot;&gt;democratic-csi&lt;/a&gt; as CSI (but this is way out of the scope of this article, maybe I will write it up in a separate article in the future).&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-41452d2a3986.SucgnKAM_Z25AmTn.webp&quot; alt=&quot;&quot;&gt;&lt;/p&gt;
&lt;p&gt;5️⃣ Adjust Packer Template&lt;/p&gt;
&lt;p&gt;We are going to make some adjustments in the file that Packer is using to create the baseline image, as it is missing some details that come in very handy when working with Proxmox.&lt;/p&gt;
&lt;p&gt;Some of them are just missing, others just regulate the hardware specs that we are going to demand for our base line image template. So open the file below &lt;code&gt;image-builder/images/capi/packer/proxmox/packer.json.tmpl&lt;/code&gt;:&lt;/p&gt;
&lt;p&gt;Add in &lt;code&gt;network_adapters&lt;/code&gt; and new property called &lt;code&gt;model&lt;/code&gt;, as shown in the snippet below:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-json&quot;&gt;&quot;network_adapters&quot;: [
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Under &lt;code&gt;variables&lt;/code&gt;, add:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-json&quot;&gt;&quot;nic_model&quot;: &quot;{{env `PROXMOX_NIC_MODEL`}}&quot;,
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;change (remove or append according to -/+ prefixes):&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-json&quot;&gt;- &quot;disk_format&quot;: &quot;qcow2&quot;,
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;6️⃣ Build Image&lt;/p&gt;
&lt;p&gt;Execute the following commands:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;source packer/proxmox/.env
make build-proxmox-ubuntu-2204
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;That will build a base image for your control-plane and worker nodes, based on Ubuntu 22.04; the process takes quite some time (~25–35mins depending on the specs of your workstation, your internal &amp;#x26; external networks speed etc) so sit back and wait for it. In the meantime you can open your Proxmox console and observe how a new virtual machine is being created, gets its network configuration, package updates, and eventually having installed (via Ansible) all the necessary binaries that will make it suitable to serve as a Kubernetes node. When all fine tuning is complete, it will be automatically converted to a template.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-988e0c0130ef.BeGzI1A7_Z1nj1on.webp&quot; alt=&quot;&quot;&gt;&lt;/p&gt;
&lt;h2&gt;Installing Prerequisites&lt;/h2&gt;
&lt;p&gt;To maintain a single guide for both macOS and Linux, we’ll use &lt;a href=&quot;https://brew.sh/&quot;&gt;Homebrew&lt;/a&gt; to install the binaries required for managing the cluster.&lt;/p&gt;
&lt;p&gt;1️⃣ Install Homebrew&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;/bin/bash -c &quot;$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)&quot;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Follow the instructions of the installer, to complete the installation.&lt;/p&gt;
&lt;p&gt;2️⃣ Install KiND&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;brew install kind
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;3️⃣ Install kubectl&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-wasm&quot;&gt;brew install kubectl
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;4️⃣ Install clusterctl&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;brew install clusterctl
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The &lt;code&gt;clusterctl&lt;/code&gt; CLI delivers a streamlined “day one” experience for CAPI by automatically retrieving and installing the necessary provider manifests. It embeds recommended best practices for provider management, helping you avoid mis-configurations and simplifying tasks like upgrades.&lt;/p&gt;
&lt;p&gt;Built on top of &lt;code&gt;clusterctl&lt;/code&gt;, the Cluster API Operator is a Kubernetes operator that brings a fully declarative workflow for managing provider lifecycles within your management cluster. It enhances the deployment and day-to-day operation of Cluster API, making routine tasks and GitOps-driven automation effortless.&lt;/p&gt;
&lt;h2&gt;Create Configuration for Proxmox&lt;/h2&gt;
&lt;p&gt;Before creating a workload cluster, certain configurations required on your management cluster. This is done by setting up environment variables for the &lt;a href=&quot;https://github.com/ionos-cloud/cluster-api-provider-proxmox/&quot;&gt;Cluster API Provider for Proxmox VE&lt;/a&gt; (CAPMOX) and then generating a cluster manifest.&lt;/p&gt;
&lt;p&gt;Create a a file named &lt;code&gt;capmox.env&lt;/code&gt;, and fill it in with the following content:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;## -- Controller settings -- ##
export PROXMOX_URL=&quot;https://&amp;#x3C;PROXMOX_NODE_IP&gt;:8006&quot;
export PROXMOX_TOKEN=&apos;capmox@pve!capi&apos;
export PROXMOX_SECRET=&quot;&amp;#x3C;CAPMOX_API_TOKEN_SECRET&gt;&quot;                          
&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code class=&quot;language-text&quot;&gt;## -- Required workload cluster default settings -- ##
export PROXMOX_SOURCENODE=&quot;&amp;#x3C;PROXMOX_SOURCENODE_NAME&gt;&quot;                                     
export TEMPLATE_VMID=&quot;&amp;#x3C;PROXMOX_BASE_IMAGE_TEMPLATE_ID&gt;&quot;                                          
export ALLOWED_NODES=&quot;[&amp;#x3C;PROXMOX_NODE1_NAME&gt;,&amp;#x3C;PROXMOX_NODE2_NAME&gt;...]&quot;
export VM_SSH_KEYS=&quot;ssh-rsa AAAAB...&quot;
&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code class=&quot;language-text&quot;&gt;## -- networking configuration-- ##
export CONTROL_PLANE_ENDPOINT_IP=&quot;192.168.1.230&quot;                    
export NODE_IP_RANGES=&quot;[192.168.1.220-192.168.1.229]&quot;               
export GATEWAY=&quot;192.168.1.1&quot;                                        
export IP_PREFIX=&quot;24&quot;                                               
export DNS_SERVERS=&quot;[192.168.1.1]&quot;                                  
export BRIDGE=&quot;vmbr0&quot;
export NETWORK_MODEL=&quot;virtio&quot;                                       
&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code class=&quot;language-text&quot;&gt;## -- xl nodes -- ##
export BOOT_VOLUME_DEVICE=&quot;scsi0&quot;                                   
export BOOT_VOLUME_SIZE=&quot;20&quot;                                       
export NUM_SOCKETS=&quot;1&quot;                                              
export NUM_CORES=&quot;2&quot;                                                
export MEMORY_MIB=&quot;2048&quot;                                            
export FILE_STORAGE_FORMAT=&quot;raw&quot;
export STORAGE_NODE=&quot;local-lvm&quot;
export NODE_REPLICAS=1
&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code class=&quot;language-text&quot;&gt;export CLOUD_INIT_CONFIG=&quot;#cloud-config package_update=true packages=- net-tools&quot;## -- Controller settings -- ##
&lt;/code&gt;&lt;/pre&gt;
&lt;blockquote&gt;
&lt;p&gt;Replace all the &lt;code&gt;192.168.1.0/24&lt;/code&gt; IP address with the CIDR that matches your the bridge interface&lt;code&gt;vmbr0&lt;/code&gt;, or whichever bridge you chose to use for that matter.
Replace all the values in placeholders with the ones corresponding in your lab.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h2&gt;Setting up a Management Cluster&lt;/h2&gt;
&lt;p&gt;This is the cluster that runs one or more Infrastructure Providers and holds resources (for example, &lt;code&gt;Machines&lt;/code&gt;), when provisioning multiple workload clusters.&lt;/p&gt;
&lt;p&gt;1️⃣ Create the cluster&lt;/p&gt;
&lt;p&gt;Create a kind config file named &lt;code&gt;kind-management-cluster.yaml&lt;/code&gt;, with extra mounts to allow the Docker provider to access Docker on the host:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-yaml&quot;&gt;kind: Cluster
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;and execute the following command:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;kind create cluster --name=mgmt-capmox --config=kind-management-cluster.yaml
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;the kubeconfig of the newly generated cluster is already merged with your &lt;code&gt;KUBECONFIG&lt;/code&gt; and your kubectl is automatically set to use its context.&lt;/p&gt;
&lt;p&gt;If you are planning to try Kamaji as control-plane provider, install the management cluster, using &lt;a href=&quot;https://k3s.io/&quot;&gt;K3s&lt;/a&gt;, in a brand new virtual machine (that either uses the same bridge, &lt;code&gt;vmbr0&lt;/code&gt; in our lab, as the one you used above or one that can communicate with &lt;code&gt;vmbr0&lt;/code&gt;) , by executing the following command:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC=&quot;--disable=traefik --disable=servicelb&quot; sh -
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;You can find the kubeconfig of the newly created cluster under &lt;code&gt;/etc/rancher/k3s/k3s.yaml&lt;/code&gt;. Copy it somewhere in your workstation (preferably under &lt;code&gt;~/.kube/k3s.yaml&lt;/code&gt;) and run:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;export KUBECONFIG=~/.kube/k3s.yaml
&lt;/code&gt;&lt;/pre&gt;
&lt;blockquote&gt;
&lt;p&gt;Make sure you change the server value from &lt;code&gt;https://127.0.0.1:6443&lt;/code&gt; to the IP address assigned on the network device that uses the bridge &lt;code&gt;vmbr0&lt;/code&gt;.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;2️⃣ Transform the cluster to a Management Cluster&lt;/p&gt;
&lt;p&gt;With clusterctl installed and prerequisites satisfied, run &lt;code&gt;clusterctl init&lt;/code&gt; to convert your Kubernetes cluster into a management cluster.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;source capmox.env
&lt;/code&gt;&lt;/pre&gt;
&lt;blockquote&gt;
&lt;p&gt;The command above will install &lt;a href=&quot;https://github.com/ionos-cloud/cluster-api-provider-proxmox/&quot;&gt;Cluster API Provider for Proxmox VE&lt;/a&gt; (CAPMOX) as infrastructure provider and Kubeadm as control-plane &amp;#x26; bootstrap provider.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-4d14d8672b59.BqzoIPtm_1X5mgO.webp&quot; alt=&quot;&quot;&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;1. Infrastructure Provider (Proxmox) 2. Control-Plane Provider (Kubeadm) 3. Bootstrap Provider (Kubeadm)&lt;/em&gt;&lt;/p&gt;
&lt;h2&gt;Creating a Cluster with Kubeadm CP Provider&lt;/h2&gt;
&lt;p&gt;The kubeadm control plane provider watches &lt;code&gt;KubeadmControlPlane&lt;/code&gt; resources in your management cluster and drives the lifecycle of control-plane nodes. When you create or update a &lt;code&gt;KubeadmControlPlane&lt;/code&gt;, the provider generates a corresponding set of &lt;code&gt;Machine&lt;/code&gt; objects (using your &lt;code&gt;MachineTemplate&lt;/code&gt;) and orchestrates &lt;code&gt;kubeadm init&lt;/code&gt; on the first node, then &lt;code&gt;kubeadm join&lt;/code&gt; for additional replicas.&lt;/p&gt;
&lt;p&gt;It manages certificate generation and rotation, handles rolling upgrades by replacing one control-plane node at a time, and ensures the Kubernetes API servers remain available throughout. All configuration (Kubernetes version, component settings, high-availability topology) is defined declaratively in the &lt;code&gt;KubeadmControlPlane&lt;/code&gt; spec, and the provider’s reconciliation loop converges the actual state to match.&lt;/p&gt;
&lt;p&gt;When using the kubeadm control plane provider with the Proxmox infrastructure provider (CAPMOX), the control plane controller creates &lt;code&gt;Machine&lt;/code&gt; objects &lt;strong&gt;whose underlying VMs are provisioned on Proxmox&lt;/strong&gt; according to your &lt;code&gt;MachineTemplate&lt;/code&gt;. As each VM becomes available, the kubeadm provider runs &lt;code&gt;kubeadm init&lt;/code&gt; on the first node and &lt;code&gt;kubeadm join&lt;/code&gt; on subsequent replicas, while Proxmox handles the VM lifecycle, boot, networking, and storage. This integration lets you declaratively manage high-availability control planes on your Proxmox cluster using the familiar CAPI workflows.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-03cdb3b2c840.CPljUuaE_ZKVIvK.webp&quot; alt=&quot;&quot;&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Multiple Control Planes and Worker Nodes are running as VMs in Proxmox&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;1️⃣ Generate the manifests&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;clusterctl generate cluster&lt;/code&gt; command returns a YAML manifest for creating a workload cluster.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;source capmox.env
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This creates a manifest called&lt;code&gt;capi-quickstart.yaml&lt;/code&gt; with a predefined list of CAPI resources as &lt;code&gt;Cluster&lt;/code&gt;, &lt;code&gt;Machines&lt;/code&gt; and &lt;code&gt;MachineDeployments&lt;/code&gt;, etc. The generated file has some issues when it comes to play with Proxmox as target infrastructure and we have to make some manual changes.&lt;/p&gt;
&lt;p&gt;2️⃣ Change the CIDR block of the &lt;code&gt;Cluster&lt;/code&gt; object ⚠️&lt;/p&gt;
&lt;p&gt;You need to change the &lt;code&gt;cidrBlock&lt;/code&gt; value to something not overlapping with the CIDR block of your &lt;code&gt;vmbr0&lt;/code&gt; bridge (in this case, yours might differ) otherwise your Pods will starts stealing IP addresses from that network segment. So go and replace the value, I choose in my case a CIDR I know that I am not using somewhere else either in a physical or virtual LAN, &lt;code&gt;10.44.0.0/16&lt;/code&gt;.&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-yaml&quot;&gt;apiVersion: cluster.x-k8s.io/v1beta1
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;3️⃣ Allow CAPMOX to overcommit memory ⚠️&lt;/p&gt;
&lt;p&gt;When the infrastructure controller provisions a new VM on Proxmox, whether it’s a control-plane or worker node , it first sums the RAM already allocated on the target host to determine if your &lt;code&gt;MachineTemplate&lt;/code&gt; will fit. If you haven’t explicitly enabled memory over-commit, this check will refuse to schedule any VM once the sum of allocated RAM reaches the node’s nominal capacity, causing new VMs to hang even though Proxmox could safely oversubscribe memory in practice.&lt;/p&gt;
&lt;p&gt;For that matter, add the following properties to the &lt;code&gt;ProxmoxCluster&lt;/code&gt; resource:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-yaml&quot;&gt;schedulerHints:
    memoryAdjustment: 0
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;It should eventually look like this:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-yaml&quot;&gt;apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: ProxmoxCluster
metadata:
  name: capi-quickstart
  namespace: default
spec:
  allowedNodes:
  - pve-am06
  controlPlaneEndpoint:
    host: 192.168.1.230
    port: 6443
  dnsServers:
  - 192.168.1.1
  ipv4Config:
    addresses:
    - 192.168.1.220-192.168.1.229
    gateway: 192.168.1.1
    prefix: 24
  schedulerHints:
    memoryAdjustment: 0
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;4️⃣ Create the cluster&lt;/p&gt;
&lt;p&gt;Execute the command:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-typescript&quot;&gt;kubectl apply -f capi-quickstart.yaml
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Then sit back and watch Proxmox clone the template and begin provisioning control-plane and worker nodes; a process that will take a few minutes to complete.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-5c64d31e16b6.BHm86FaX_Z1KtCvQ.webp&quot; alt=&quot;&quot;&gt;&lt;/p&gt;
&lt;p&gt;When is done, get the &lt;strong&gt;kubeconfig&lt;/strong&gt; file of your new cluster by executing:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;clusterctl get kubeconfig capi-quickstart &gt; capi-quickstart.kubeconfig
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;5️⃣ Deploy a CNI&lt;/p&gt;
&lt;p&gt;For demonstration purposes, we are going to deploy &lt;a href=&quot;https://docs.tigera.io/calico/latest/about/&quot;&gt;Calico&lt;/a&gt; as the CNI solution of our newly created cluster. In your lab you can choose the CNI of your liking. Execute the following command:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;kubectl --kubeconfig=./capi-quickstart.kubeconfig \
  apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;img src=&quot;https://www.backyard-rack.com/_astro/medium-6de7d86070a4.CNGwg1RX_zw0E1.webp&quot; alt=&quot;&quot;&gt;&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;When issues arise, you can access any node directly by opening an SSH session as the &lt;code&gt;root&lt;/code&gt; user. This allows you to inspect system logs, review configuration files, and run diagnostic commands on each machine.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h2&gt;Next Steps&lt;/h2&gt;
&lt;p&gt;In the &lt;a href=&quot;https://akyriako.medium.com/build-your-own-managed-kubernetes-service-on-proxmox-with-capi-and-kamaji-7858ad41cb09&quot;&gt;next article&lt;/a&gt;, we’ll explore how to leverage &lt;strong&gt;Kamaji&lt;/strong&gt; to containerize the control plane of tenant clusters within the management cluster, leaving only the worker nodes running as virtual machines in Proxmox.&lt;/p&gt;
&lt;p&gt;Kamaji is an open-source operator developed by &lt;a href=&quot;https://clastix.io/&quot;&gt;Clastix&lt;/a&gt;, which imagines the Kubernetes control plane as a hosted service &lt;strong&gt;running entirely on Kubernetes&lt;/strong&gt; itself. Rather than provisioning dedicated virtual machines or bare-metal servers for etcd, kube-apiserver, controller manager and scheduler, Kamaji deploys those components as &lt;strong&gt;Pods&lt;/strong&gt; inside the management cluster and presents them to tenant clusters via a lightweight API. This Hosted Control Plane pattern decouples control plane lifecycle from underlying infrastructure and dramatically reduces both operational overhead and resource duplication.&lt;/p&gt;
&lt;p&gt;Originally published on &lt;a href=&quot;https://medium.com/itnext/build-your-own-managed-kubernetes-service-on-proxmox-with-capi-8d9786644818&quot;&gt;Medium&lt;/a&gt;&lt;/p&gt;</content:encoded><h:img src="/_astro/thumbnail._po21YLS.jpg"/><enclosure url="/_astro/thumbnail._po21YLS.jpg"/></item></channel></rss>