Scott's Weblog

Using Mitmproxy to Observe kubectl Traffic

Scott Lowe — Wed, 04 Mar 2026 17:30:00 EST

When I first started learning Kubernetes, I had the idea that observing the network traffic between a client system using kubectl and the Kubernetes API Server would be a useful thing to do. The source of the idea is unclear; I am unsure why I thought this would be useful as a learning tool. Regardless, I continued on with learning Kubernetes and never really pursued this idea—until this week. I found it can be a useful troubleshooting technique, but I will leave it up to you to determine if it is a useful learning technique. In this post, I will show you how to observe kubectl traffic using mitmproxy.

This technique is inspired by/informed by Ahmet Alp Balkan’s similarly-named blog post from 2019. Unfortunately, I found the instructions there to be incomplete (most likely just due to the passage of time and continued evolution of the tools involved).

I used the following tools and environments in my testing:

The tests were conducted on a Linux system running Ubuntu 24.04.4. The commands should work similarly on macOS.
Mitmproxy was installed from the Ubuntu repositories using apt.
kubectl version 1.33.3 was used to communicate to a self-managed cluster on AWS (in other words, not Amazon EKS) running Kubernetes 1.32.9. The cluster was bootstrapped using kubeadm. I wouldn’t expect any major/significant differences with other versions of kubectl or Kubernetes.
I was using a client certificate to authenticate to Kubernetes. It’s unclear to me how this might work—if it works at all—with alternate authentication mechanisms.

Prepare Client Certificates

Before you can start mitmproxy, you’ll first need to extract the client certificates from the Kubeconfig file. A couple of ways exist to do this; a blog post of mine from 2022 contains what I believe is the easiest way. The method involves yq (to extract information from the Kubeconfig) and base64 (to decode the client certificate and client key). Refer to the linked blog post for full details.

First, extract the client certificate (adjust the users[0] as needed based on the Kubeconfig file):
```
yq '.users[0].user.client-certificate-data' < kubeconfig | base64 -d >> client-cert.pem
```

Next, extract the client key:

yq 'users[0].user.client-key-data'< kubeconfig | base64 -d >> client-cert.pem

Running Mitmproxy and Watching Traffic

Now that you have the client certificate and key in hand, you’re ready to launch mitmproxy and observe some traffic. Mitmproxy requires a few specific configuration flags for this use case:

You’ll need to disable HTTP/2 support with --set http2=false.
Mitmproxy must be configured with the client certificate you extracted above (using --set client_certs=client-cert.pem).
In this particular case, since the cluster CA isn’t trusted, mitmproxy also has to be told not to verify the upstream certificate with --set ssl_insecure=true.

Putting all this together, the full command looks like this:

mitmproxy -p 5000 --set ssl_insecure=true --set http2=false --set client_certs=client-cert.pem

Then, in a separate window, run your kubectl command:

HTTPS_PROXY=:5000 kubectl get po -A --insecure-skip-tls-verify

You should see the traffic pop up in the window running mitmproxy, and can review the traffic flow(s) in detail.

As explanation: the HTTPS_PROXY part redirects traffic through the instance of mitmproxy you just launched, and --insecure-skip-tls-verify tells kubectl not to verify the upstream certificate—which it can’t do because it’s seeing mitmproxy’s certificate, not the cluster’s certificate.

There you go—now you can observe traffic between kubectl and the Kubernetes API server. As I mentioned at the start of this post, this technique is useful for troubleshooting. I did not find it useful as a learning tool, but others might.

I hope you find this useful. Thanks to Ahmet for setting me down the right path! If you have questions, feel free to reach out. I’m available on social platforms (X/Twitter, Mastodon, Bluesky, and LinkedIn), or you can drop me an email. I’d be happy to help if I’m able.

Running the Azure CLI in a Container

Scott Lowe — Mon, 02 Mar 2026 08:00:00 EST

Like perhaps some readers, I am quite particular about what gets installed on my systems. I try to keep my systems as “clean” as possible, doing my best to avoid tools that have an extensive list of dependencies that must be installed and updated. Where that isn’t possible—such as with the Azure CLI, which has a massive number of Python modules that are required in order for the tool to function—I will use various isolation mechanisms. For the Azure CLI, that’s typically been a Python virtual environment. Somewhat recently, though, I had an idea to try using a container. In this post, I’ll share what worked and what did not work when trying to run the Azure CLI in a container.

First, though, a disclaimer: I am not an Azure expert, nor am I a Python expert. I know enough to get by. If I share something here that’s incorrect, please contact me and constructively show me my errors so that I can fix them.

Before I started down this path, I was sure this would be a slam dunk. I mean, this is what containers are for, right? If you do some web searches for running the Azure CLI in a container, you’ll find articles that give you this command line:

docker run -it mcr.microsoft.com/azure-cli

(Note that this assumes you are using Docker instead of something like Podman.)

This does work, as long as you’re willing to operate in an interactive shell in the container. I was looking for something a bit different: I wanted to create an alias for az that executed the container in my current shell instead of putting me into a separate shell inside the container. Something like this, for example:

alias az="docker container run --rm mcr.microsoft.com/azure-cli az"

I quickly found that you need to map in your Azure configuration directory from outside the container, or your configuration won’t persist. (In theory you could remove the --rm parameter and keep the container around.) Now the docker command in your alias starts to look like this:

docker container run --rm -v ${HOME}/.azure:/root/.azure mcr.microsoft.com/azure-cli az

Oh, you need access to your SSH keys? You’ll have to map those in, too:

docker container run --rm -v ${HOME}/.azure:/root/.azure \
-v ${HOME}/.ssh:/root/.ssh mcr.microsoft.com/azure-cli az

Except that then I remember that I am running as root in the container, and so any files created by the Azure CLI will be owned as root outside the container, too. No worries; I’ll just run it as a different user ID. Unfortunately, that involves building and maintaining my own container image to specify that the Azure CLI should run as something other than root. At this point, you’ll likely come to the same conclusion I did, and decide that a Python virtual environment is fine.

I was about to classify this experiment as “Death by a Thousand Cuts”, because every time I turned around I was finding some annoyance or blocker caused by Docker and containers. And then I remembered Distrobox.

Distrobox uses a container, but the container is “tightly integrated” with the host. For example, a Distrobox container shares the home directory of the user. For this use case, this makes the situation appreciably simpler—I no longer need to map in volumes for the Azure configuration or for my SSH keys.

Getting this working with Distrobox was only a few steps:

Create a new Distrobox container (I chose to use Debian 13):
```
distrobox create -n azcli -i debian:13
```
Enter the new container with distrobox enter azcli and install a few tools. (This is one oddity of Distrobox: it will use the ~/.bashrc from your host, which may references tools that don’t exist in the container. In my situation, I needed to install Starship and a few command-line tools.)
Install the Azure CLI in the Distrobox container by following the Linux installation instructions. (One side note here: I chose Debian 13, but technically Debian 13 isn’t yet supported.)
In the Distrobox container, export the Azure CLI so it is accessible from the host system:
```
distrobox-export -b /usr/bin/az
```

If you omit step 4, then you are sort of back to where I was earlier in this article—you’ll enter the Distrobox container and run the az commands from a separate shell. However, once you complete step 4 (which is done from within the Distrobox container), then on the host you will run az. That’s it! All of the files, all the dependencies, etc., are all stored in the Distrobox container. Your system remains “clean.”

Wrapping Up and Additional Resources

After trying to use a “standard” Docker container, I was convinced that trying to run the Azure CLI in a container was more work than just falling back to a plain old Python virtual environment. With Distrobox, though, it’s a piece of cake! Just create the Distrobox container, install the Azure CLI, export it, and you’re done.

I hope you’ve found this article helpful. I encourage you to check out Distrobox, either via the Distrobox web site or via the GitHub repository. Feel free to find me online (X/Twitter, Mastodon, Bluesky, or LinkedIn) and let me know what sort of cool things you end up doing with Distrobox!

Technology Short Take 191

Scott Lowe — Fri, 27 Feb 2026 09:00:00 EST

Welcome to Technology Short Take #191! This is my semi-regular collection of links related to technology disciplines, including networking, security, cloud computing, storage, and programming/development. I hope that I’ve managed to curate an interesting and useful set of links for readers. Enjoy!

Networking

I learned a new acronym from Ivan Pepelnjak in his article on “not so passive” OPSFv2 interfaces in Cisco IOS/XR: QDS (Quick and Dirty Solution). I am totally using that from now on.
Via Ivan (hat tip to my colleague Russ for pointing me in this direction), I also learned that all is not well in network automation land with Ansible. Although some parts of it are apparently now resolved, this does raise questions about the viability of Ansible for network automation.
Doug Dawson talks about the new Amazon Eero Signal, a cellular backup to a primary broadband connection for users using their Eero mesh systems. I agree with Doug—the pricing seems high to me. I generally like the Eero devices, although there have been times when I really could use a bit more control over the settings.

Security

The APNIC blog features a guest article discussing the use of tunneling protocols to infiltrate networks. The article specifically discusses Generic Routing Encapsulation (GRE) and Virtual Extensible LAN (VXLAN). It’s worth a read if your network uses either of these technologies.

Cloud Computing/Cloud Management

Somewhere along the way I missed the CDK deprecation announcement. If I am not mistaken, this leaves Pulumi as the only cloud-independent IaC tool that lets you use a general purpose programming language.
Managing dependencies between containers can be difficult at times; this blog post by Nicolas Fränkel introduced me to wait4x, a tool one can use to help simplify these types of situations.

Operating Systems/Applications

I appreciated reading Mitchell Hashimoto’s AI adoption journey. I feel it does provide a more realistic approach to how someone in a role like his might leverage an LLM or coding agent.
Michael Christofides taught me about bloat in this article on read efficiency issues in Postgres queries.
Ricardo Katz discusses a few somewhat-recent security vulnerabilities in kgateway, an open source Gateway API implementation that leverages Envoy Proxy.
I am squarely in agreement with Martin Fowler on the topic of agentic email—this seems like a dangerous combination, so tread carefully. (I am squarely in disagreement with Martin’s characterization of servant leadership as gaslighting, but that’s another topic for another day.)
You know AI has gone too far when you find it has “continvoucly morged” your content.
Earlier this week I stumbled across hazelnut, a terminal-based file organizer for Linux systems. The macOS app Hazel inspired hazelnut. Neat!

Programming/Development

Tuan Nguyen’s article on parsing a YAML file in Golang was helpful for me recently. I had struggled with how to do this without a struct, and the article provides a couple of examples of using a map instead of a struct for this purpose.
Simon Willison is starting to write about agentic engineering patterns—that is, patterns for software engineering when working with agents.

Virtualization

Lightweight Linux VMs (sometimes called “micro-VMs”) are gaining in popularity. Deno recently introduced Deno Sandbox, built on lightweight Linux VMs, to provide code security for untrusted code.
William Lam provides some information on using cross-vCenter vMotion to help with migrating away from vSphere 7 (which is now reached end of support).
Who out there is messing around with Incus? I am considering trying it out.
I noticed this past week that you can now use nested virtualization to run hypervisors on “regular” EC2 instances.

Career/Soft Skills

Tom Hollingsworth writes about managing your attention, which is something I’m increasingly seeing. I spoke about this some time ago when I advised folks to turn off new email notifications and switch to checking your email less frequently so you could focus on getting work done.
I also saw this post by Scott H Young about managing your energy instead of your time, which—to me, at least—relates to Tom’s post in the previous bullet. In the follow-up post explaining what energy is, Young (or one of the linked articles/studies) refers to energy as a “mental resource.” It seems to me, as an uneducated layman, that managing your energy levels would also entail managing the cognitive zap you get from constant context switches due to notifications. I could be wrong (it wouldn’t be the first time, nor will it be the last).
One final thought in this thread: Scott Young’s article on how to make hard work feel less hard talks about making tasks less effortful. This sounds a lot like “reducing the friction” for common tasks, which I have discussed in the past (here is one example). Friction is something that Murat Demirbas has also recently discussed, although his example is more organizational than personal. Either way, the concept of friction in our work is definitely a valid concept, and something we should take into account as we work toward managing both our attention and our energy.

That’s all for now. I will return in a couple of weeks with the next installation, and a fresh set of links from around the internet. In the meantime, I invite you to drop me a line (my email is not hard to find) or hit me up on social media (X/Twitter, Mastodon, Bluesky, or LinkedIn) and let me know if you’ve found something useful. I love to hear from readers!

Technology Short Take 190

Scott Lowe — Fri, 06 Feb 2026 08:00:00 EST

Welcome to Technology Short Take #190! This is the first Tech Short Take of 2026, and it has been nearly three months (wow!) since the last one. I can’t argue that I fell off the blogging bandwagon over the end of 2025 and early 2026. I won’t get into all the reasons why (if you’re interested then feel free to reach out and I’ll fill you in). Enough about me—let’s get to the technical content! Here’s hoping you find something useful.

Networking

Here’s a little something on the lighter side about what causes network outages.
As the lead-in to this article on Starlink’s performance during a solar superstorm says, “It’s not science fiction.” Solar storms can have a real impact on Earth, and satellites in low-Earth orbit (LEO) are not exempt from that impact.

Servers/Hardware

William Lam reviews the Mini PC and SFF (small form factor) hardware announcements from CES 2026.

Security

The scale of DDoS attacks continues to grow, as evidenced by this report of a 15 Tbps attack on Microsoft Azure.
Microsoft having the ability to give away your Windows PC’s data encryption key is horrifying.

Cloud Computing/Cloud Management

While doing some reading on Terragrunt, I also came across this open source tool for performing operations against multiple Git repositories. I don’t have a use case for it, but it is cool!
While on the topic of Terragrunt: let me say that I appreciate the work that went into their online docs! From what I’ve read so far, they are well-written, clear, concise, and informative. Well done!
And while still on the topic of Terragrunt: it was this article from Axel Mendoza on why they use Terragrunt over Terraform/OpenTofu that sent me down the Terragrunt rabbit hole.
The most recent installation of Ricardo Sueiras’ AWS open source newsletter pointed me to a couple of tools that look really handy: s3sh (available from GitHub) and taws (also available from GitHub).
Nick Buraglio has a great comparison of hosted email options.

Operating Systems/Applications

Howard Oakley’s “loss of confidence” in macOS is one reason why I started using Linux more. I just kept finding things in macOS that didn’t work like they should, which was made worse by the fact that Apple (the company) just doesn’t seem to care.
It is still kind of raw, but I found a Markdown preview tool for Linux called Inlyne. Pretty cool! I am looking forward to seeing how it develops.
This treatise from Mike Swanson on “backseat software” is absolute gold. Go read it. Now. Seriously, this is good stuff. I was nodding so vigorously as I was reading it that I almost gave myself a headache!

Storage

Looking for some information on vSAN Encryption in VMware Cloud Foundation? I think Eric Sloof has exactly the document you need. (You may also find this related article from Eric useful as well.)

Virtualization

Eric Sloof writes about the evolution of the VCDX. As an early VCDX (number 39), I am of two minds regarding this change. On one hand, it makes sense; they need more than just architects to pursue this certification. On the other hand, they are (in some ways) “softening” the certification. It will be interesting to see if this helps the designation grow, or if it continues to fade into obscurity.

Career/Soft Skills

Murat Demirbas talks a bit about momentum, and how maintaining momentum (the “the messy daily pushes” as he puts it) should be how you organize and structure your workflows. I can see his point, certainly, although I would warn folks against interpreting his “produce something every day” advice as meaning that you need to publish something every day. That path leads to burn out. Produce something, sure, but allow for that something to be something that only you’ll see.
This article about getting better at technical writing contains some good guidelines.
This discussion of feeling uncomfortable and how it relates to growing personally really resonated with me. No, not because of Werner’s fear of public speaking; rather, it was about how it is necessary to move outside of your comfort zone in order to grow. I am getting ready to potentially embark on something that really pushes me outside my comfort zone, and it is scary. I also know, though, that it is the only way to grow.

That’s all, folks! (Bonus points for you if you recognize that reference.) I am always up to hear from readers, so I invite you to reach out using various social media platforms (X/Twitter, Mastodon, Bluesky, LinkedIn), Slack, or old-fashioned email. Thanks for reading!

Setting up a VPC Route Server with Pulumi

Scott Lowe — Wed, 28 Jan 2026 09:00:00 -0400

If you need to work with BGP in your AWS VPCs—so that BGP-learned routes can be injected into a VPC route table—then you will likely need a VPC Route Server. While you could set up a VPC Route Server manually, what’s the fun in that? In this post, I will walk you through a Pulumi program that will set up a VPC Route Server. Afterward, I will discuss some ways you could check the functionality of the VPC Route Server to show that it is indeed working as expected.

To make things as easy as possible, I have added a simple Pulumi program to my GitHub “learning-tools” repository in the aws/vpc-route-server directory. This program sets up a VPC Route Server and its associated components for you, and I will walk through this program in this blog post.

The first step is creating the VPC Route Server itself. The VPC Route Server has no prerequisities, and the primary configuration needed is setting the ASN (Autonomous System Number) the Route Server should use:

rs, err := vpc.NewRouteServer(ctx, "rs", &vpc.RouteServerArgs{
    AmazonSideAsn: pulumi.Int(65534),
    Tags: pulumi.StringMap{
        "Name":    pulumi.String("rs"),
        "Project": pulumi.String("vpc-route-server"),
    },
})

Next, you will need to associate the Route Server with a VPC. This requires a VPC ID, as you might expect; this ID could come from a configuration value passed in by the user, or from a VPC created earlier in this or another Pulumi program.

rsa, err := vpc.NewRouteServerVpcAssociation(ctx, "rsa", &vpc.RouteServerVpcAssociationArgs{
    RouteServerId: rs.ID(),
    VpcId:         pulumi.String(userSuppliedVpcId),
})

With that association in place, the next component is a Route Server Endpoint. This endpoint is created in a subnet, and will have an IP address assigned from that subnet. This IP address is what your BGP peer will use to establish a peering relationship to exchange routes.

rse, err := vpc.NewRouteServerEndpoint(ctx, "rse", &vpc.RouteServerEndpointArgs{
    RouteServerId: rs.RouteServerId,
    SubnetId:      pulumi.String(userSuppliedPrivateSubnetId),
    Tags: pulumi.StringMap{
        "Name":    pulumi.String("rse"),
        "Project": pulumi.String("vpc-route-server"),
    },
}, pulumi.DependsOn([]pulumi.Resource{rsa}))

In working with the VPC Route Server, I’ve found that sometimes the dependencies between components don’t quite work as expected. Because the endpoint references the rs resource (the Route Server itself), the endpoint has a implicit dependency on that resource. However, if the VPC association isn’t complete, then you can run into errors. To address that problem, the code above uses pulumi.DependsOn to create an explicit dependency on the VPC association. This way, both the Route Server and the association of the Route Server to a VPC are complete before the program attempts to create the endpoint.

The Route Server needs to know where to inject the routes it learns, and that’s handled by creating a propagation. The propagation links the Route Server to a route table in the associated VPC. Here’s the Pulumi code from the example in the “learning-tools” repository:

_, err = vpc.NewRouteServerPropagation(ctx, "rs-prop", &vpc.RouteServerPropagationArgs{
    RouteServerId: rs.RouteServerId,
    RouteTableId:  pulumi.String(userSuppliedPrivateRouteTableId),
}, pulumi.DependsOn([]pulumi.Resource{rsa}))

The use of pulumi.DependsOn makes the propagation, like the endpoint, dependent on the VPC association. This helps eliminate errors related to the order of resource creation or deletion.

The last step is creating a peer for the Route Server. This tells the Route Server what the “other side” of a BGP peering relationship looks like. The “other side” could be an EC2 instance running a BGP daemon, or it could be a Kubernetes node on a cluster running a CNI that supports BGP (like Cilium). Whatever the “other side” is, you’ll need to be sure the configuration you pass to the Route Server peer matches the configuration in use by the other side. Otherwise, establishing the BGP peering relationship will fail.

_, err = vpc.NewRouteServerPeer(ctx, "rs-peer-01", &vpc.RouteServerPeerArgs{
    BgpOptions: &vpc.RouteServerPeerBgpOptionsArgs{
        PeerAsn:               pulumi.Int(65001),
        PeerLivenessDetection: pulumi.String("bgp-keepalive"),
    },
    PeerAddress:           pulumi.String(userSuppliedPeerIpAddress),
    RouteServerEndpointId: rse.RouteServerEndpointId,
    Tags: pulumi.StringMap{
        "Name":    pulumi.String("rs-peer-01"),
        "Project": pulumi.String("vpc-route-server"),
    },
}, pulumi.DependsOn([]pulumi.Resource{rs, rse}))

In the definition of the Route Server peer, you point it to the IP address of some other device or instance running BGP. In the other device or instance, you will point it to the IP address of the Route Server endpoint. This way, each member of the BGP peering relationship knows how to connect to the other member.

All of the code above is found in the aws/vpc-route-server directory in my GitHub “learning-tools” repository.

Once you run pulumi up to create all the resources, verifying the creation is a matter of using the AWS CLI:

aws ec2 describe-route-servers # to get the Route Server ID
aws ec2 describe-route-server-endpoints
aws ec2 get-route-server-propagations --route-server-id <id>
aws ec2 describe-route-server-peers

Some of these commands don’t (yet) support some of the same filtering mechanisms as other AWS CLI commands, so you may find it necessary to use jq to filter the output on the client side.

Verifying the operation of the Route Server is a bit more complex—you will need something to run BGP and peer with the Route Server to exchange routes. As I mentioned earlier, Cilium can do this for a Kubernetes cluster, but you could also use an EC2 instance running FRR or some other BGP daemon. This is a topic I might explore in a future blog post; if it’s something you’re interested in seeing, please let me know.

If you have any questions or comments, please reach out to me. I love hearing from readers and would be happy to do my best to answer any queries or respond to feedback. You can reach me via email (my address is on this site), via various social media sites (X/Twitter, Bluesky, Mastodon), or via Slack (I’m in several Slack communities, including the Kubernetes Slack community). I hope you found this information helpful, and thanks for reading!

Technology Short Take 189

Scott Lowe — Fri, 31 Oct 2025 09:00:00 EDT

Welcome to Technology Short Take #189, Halloween Edition! OK, you caught me—this Tech Short Take is not scary. I’ll try harder next year. In the meantime, enjoy this collection of links about data center-related technologies. Although this installation is lighter on content than I would prefer, I am publishing anyway in the hopes of trying to get back to a somewhat-regular cadence. Here’s hoping you find something useful and informative!

Networking

Kevin Myers dissects EVPN/VXLAN interoperability with MicroTik and IP Infusion.
Ivan Pepelnjak has a new project: open source EVPN/VXLAN labs. I plan to tackle these myself in the near future!

Servers/Hardware

Kevin Houston takes a look at the availability of Xeon 6 CPUs across blade server vendors.

Security

Security researchers recently published some research on a new microarchitectural exploit called “VMScape.” The TL;DR on VMScape is that it allows hypervisor information to leak from a malicious VM. Oops! Olivier Lambert has a write-up that explains why the Xen hypervisor is not affected by this exploit. (Side note: be sure to read the comments—Olivier shares some useful information there.)
The leaking of source code for F5 appliances by a “nation-state affiliated cyber threat actor” has lead the CISA to call on all federal agencies to mitigate vulnerabilities in F5 appliances due to “an imminent threat to federal networks using F5 devices and software.” That’s not good.
In early October of this year, Red Hat confirmed the breach of one of its GitLab instances, leading to the theft of nearly 570GB of data across 28,000 repositories. Oof.

Cloud Computing/Cloud Management

Iain Smart reviews some common exploit paths when trying to build multi-tenant Kubernetes clusters. As Iain says, multi-tenancy is hard.
Digital Society discusses their migration off AWS and Digital Ocean to Hetzner.
Just this past week I learned of Pulumi’s resource hooks, a feature made available back in June of this year.
Hrittik Roy discusses vCluster Standalone, a multi-tenancy solution for Kubernetes that now no longer requires an underlying host cluster.
Mattis Fjellström discusses Terraform-related options for managing tags in AWS.

Operating Systems/Applications

Do you, like Cory Doctorow, believe an AI economic apocalypse is near?
I recently published a post on Git pre-commit hooks. The next day I wake up to an email from Ivan Pepelnjak, who has—not surprisingly—already explored Git pre-commit hooks and found a framework for making them easier to work with. Ivan uses this framework to perform a variety of checks against his blog posts. Now he has inspired me to go even farther with my pre-commit script!
Anthropic, along with the UK AI Institute and the Alan Turing Institute, have found that as few as 250 malicious documents can produce a “backdoor” vulnerability in a large language model (LLM). You can get more details, as well as a link to the paper published from their research, by reading the associated Anthropic blog post.
Here is a handy tool to generate a Mermaid diagram from your Terraform configuration.
Familiar with pwru? If not, read this guide to getting started with pwru.

That’s it for this time around. As always, I welcome your feedback—I’d love to hear from you! Feel free to reach me on Twitter/X, Mastodon, or Bluesky. Or email me, that is fine too (my address is here on the site, it is not too hard to find). I truly do enjoy hearing from readers. You can also find me in a variety of Slack communities, so feel free to DM me there if you would prefer. Thanks for reading!

Posts from the Past, October 2025

Scott Lowe — Wed, 22 Oct 2025 12:00:00 -0600

Every now and then, I publish one of these “Posts from the Past” articles that looks back on content I’ve created and posted over the life of this site. This year marks 20 years of content—I can hardly believe it! Don’t worry, though; you won’t have to go through 20 years of past posts. Here is a selection of posts from mid- to late October over the last decade or so. I hope you find something useful, informative, or at least entertaining!

October 2024

Last year I shared information on how to use Pulumi to stand up an Amazon Elastic Kubernetes Service (EKS) cluster with Bottlerocket OS on the Kubernetes nodes—without using any higher-level Pulumi components.

October 2022

In 2022, after getting irritated with what I felt was a poor user experience when accessing Azure Kubernetes Service (AKS) clusters created with Pulumi, I published this post on how to change the Kubeconfig file for a more streamlined user experience.

October 2021

Cluster API is the name of the game for multiple posts in October 2021. First I wrote this article on kustomize transformer configurations for Cluster API v1beta1 (so that you can use kustomize to manipulate Cluster API manifests), followed up later that month with an article on influencing Cluster API AMI selection.

I also touched upon using the external (out of tree) cloud provider for AWS that month, a topic I am revisiting soon as I explore integrating Talos Linux with AWS.

October 2020

More Cluster API content—this time discussing IaC considerations for Cluster API (think things like integrating workload clusters with existing AWS workloads or services).

October 2019

In October 2019 I explored using jk to programmatically create Kubernetes manifests, and discussed how to use kustomize with kubeadm configuration files.

October 2018

Plenty of articles discuss the use of kubeadm to bootstrap Kubernetes clusters (including a few I wrote!), but what of talking about using kubeadm to stand up an etcd cluster? I’ve got you covered!

October 2017

Let’s get recursive: check out this “Posts from the Past” article from October 2017!

October 2016

Amid a bunch of content related to Vagrant, I found this gem on using Ansible to manage your AWS infrastructure.

October 2015

Flashing back an entire decade, the OpenStack Summit was happening in Tokyo, Japan, and I was liveblogging sessions. Among the sessions I attended and liveblogged was a session on Carina, a containers-as-a-service offering. Read my liveblog of the Carina session.

Interested in finding more posts from the past? Feel free to browse the site archives, which have a link to every single post I’ve ever published. If you find something interesting, share it on social media and tag me! I’m on Twitter/X, Mastodon, and Bluesky.

Thanks for reading!

Using Git Pre-Commit Hooks

Scott Lowe — Mon, 20 Oct 2025 09:00:00 -0600

A while ago I wrote an article about linting Markdown files with markdownlint. In that article, I presented the use case of linting the Markdown source files for this site. While manually running linting checks is fine—there are times and situations when this is appropriate and necessary—this is the sort of task that is ideally suited for a Git pre-commit hook. In this post, I’ll discuss Git pre-commit hooks in the context of using them to run linting checks.

Before moving on, a disclaimer: I am not an expert on Git hooks. This post shares my limited experience and provides an example based on what I use for this site. I have no doubt that my current implementation will improve over time as my knowledge and experience grow.

What is a Git Hook?

As this page explains, a hook is a program “you can place in a hooks directory to trigger actions at certain points in git’s execution.” Generally, a hook is a script of some sort. Git supports different hooks that get invoked in response to specific actions in Git; in this particular instance, I’m focusing on the pre-commit hook. This hook gets invoked by git-commit (i.e., the user running a git commit command) and allows users to perform a series of checks or tests before actually making a commit. If the pre-commit script exits with a non-zero status, then Git aborts the commit.

Using a Pre-Commit Hook to Lint Markdown

For my use case, I wanted to run Markdownlint to lint the Markdown files (as described in the previous article on linting Markdown) before the commit. To do that, I came up with the following script:

#!/usr/bin/env bash

MDLINT="/usr/local/bin/markdownlint"

for file in $(git diff --cached --name-only --diff-filter=ACM | grep "\.md"); do
    if ! $MDLINT "$file"; then
        echo "Lint check failed on file '$file'."
        echo "Run markdownlint to identify the errors and try again."
        exit 1
    fi
done

To make this script active as a pre-commit hook, you must name it pre-commit, you must place it into Git’s hook directory (which defaults to .git/hooks in the current repository), and you must make it executable.

This script is readable enough for most folks to understand what it is doing, but there are a couple of things that might be helpful to point out:

The git diff --cached --name-only command will return a list of the files staged for commit using git add. This list of files is what will Markdownlint will check.
The --diff-filter=ACM parameter tells git diff to return only Added, Copied, and Modified files. This prevents git diff from returning the filename of a deleted file, which would then generate an error from Markdownlint.
Using grep here further restricts the list of files to include only Markdown files.
The exit 1 command is critical; without returning a non-zero exit code, Git wouldn’t know to abort the commit.

With the script in the right place and marked as executable, it’s automatically invoked by Git when I try commit any Markdown files. If the check succeeds, the commit proceeds as normal (my editor opens for me to edit the commit message); if the check fails, then I see an error message and the commit aborts. Exactly what I needed!

Additional Resources

I found the following articles useful when I was learning about pre-commit hooks and how to use them:

How to use git pre-commit hooks, the hard way and the easy way

Git Pre-Commit Hook: A Practical Guide (with Examples)

I hope you’ve found this post helpful in some fashion. As I previously mentioned, I don’t pretend to be an expert on this topic. If you spot an error or mistake in this post, then please reach out and let me know. You can reach me on Twitter, on the Fediverse, or in a variety of Slack communities. Positive feedback is also welcome—thanks for reading!

Technology Short Take 188

Scott Lowe — Fri, 19 Sep 2025 09:00:00 -0500

Welcome to Technology Short Take #188! I’m back once again with a small collection of articles and links related to a variety of data center-related technologies. I hope you find something useful!

Networking

Scott Hogg discusses using dual-stack network configurations on your DNS name servers as part of an overall IPv6 deployment plan.
Leon Adato explains why knowing what’s under the hood helps when it comes to networking telemetry.
Via Ivan Pepelnjak, I found Suresh Vina’s article on using netlab to build network labs.

Security

The use of malicious software packages continues to be a vector for attacks; here’s the latest discovery of malicious Go and npm packages.
James Kettle explains why HTTP/1.1 must die.
Gary Marcus and Nathan Hamiel discuss why the combination of LLMs plus coding agents has the potential to result in a security nightmare.
Weaponized RAR files are delivering a backdoor to Linux systems using only a filename—more details in the linked article.
WhatsApp patches a zero-click exploit targeting iOS and macOS devices. I don’t use WhatsApp, but I know lots of folks that do (it’s especially popular among my international friends). If you’re a WhatsApp user, be sure to update.
More software supply chain attacks, this time targeting CrowdStrike’s npm packages, have been observed as part of a larger campaign known as “Shai-Hulud.”

Cloud Computing/Cloud Management

Mike Roberts explains why you probably don’t want to use AWS’ new GitHub Actions Lambda Deployment action.
What would you do if AWS deleted your account and all your data?
AWS has finally released universal macOS installers for the AWS CLI.

Operating Systems/Applications

Adam Thompson shares how to improve Logitech MX Master 3 mouse support on Arch Linux.
Mitchell Hashimoto shares some lessons learned when they rewrote the Ghostty GTK application.
Some of the creators of Open Policy Agent (OPA) are joining Apple. Interesting.
Simon Späti shares details on their journey from macOS to Arch Linux with Omarchy.
Here is a useful post on systemd service hardening.
The details of this post on Apple’s new Memory Integrity Enforcement (MIE) are a bit beyond me, I’ll be honest. I’m of two minds on the new technology. On one hand, it’s nice to see progress at fixing memory safety-related bugs and exploits. On the other hand, this progress could only be achieved, as the article states, through combining “the unique strengths of Apple silicon hardware with our advanced operating system security.” In other words, more tightly coupling the hardware and the OS.

Programming/Development

Go has changed the way the language handles GOMAXPROCS in the 1.25 release.

Virtualization

In advance of VMware Explore 2025—which apparently is/was this week in Las Vegas—William Lam shares an update on the nested ESXi 8.0 and ESXi 9.0 virtual appliance.

Career/Soft Skills

Ashley Willis talks about the value and importance of the quiet season. I particularly found this sentence applicable to my own life: “So yes, I’m writing again. And maybe I’m less ’everywhere’ than I used to be. But I’d rather show up less often and have something worth saying than burn myself out trying to convince the world I’m still here.” Well said, IMO.
All too true.
I don’t know if I would go so far as to say I am an AI hater, but I am most definitely opposed to AI in its current form. (I don’t even like calling it “AI” when it’s really nothing more than a statistical model for putting words together. But that’s another discussion for another day…) As the author says, being a hater is a “kind of integrity” all its own—so I say, if you’re an AI hater, don’t be afraid to say so.

It’s time to wrap up now—but don’t be sad, I’ll be back soon with more content. In the meantime, if you’d like to reach out to me to provide some feedback on this article or any article, I’d love to hear from you! Feel free to contact me via Bluesky, via Mastodon, via X/Twitter, via Slack (I frequent a number of different communities, including the Kubernetes Slack instance) or even via email. Thanks for reading!

Creating a Talos Linux Cluster on AWS with Pulumi, 2025 Edition

Scott Lowe — Mon, 08 Sep 2025 09:35:00 EDT

A little over two years ago, I wrote a post on creating a Talos Linux cluster on AWS using Pulumi. At the time of that post, the Pulumi provider for Talos was still a prerelease version. Since then, the Talos provider has undergone some notable changes necessitating an update to the example code I have on GitHub. For your reading pleasure, therefore, I present you with the 2025 edition of a tutorial for using Pulumi to create a Talos Linux cluster on AWS.

The updated Pulumi code can be found in this GitHub repository. Note that I’ve tagged the original version from the 2023 blog post with the “2023-post” tag, in the event you’d like to see the original code. While I chose to write my Pulumi code in Go, note that Pulumi supports a number of different languages (such as JavaScript/TypeScript, Python, one of the .NET languages, Java, or even YAML). I leave it as an exercise for the reader to re-implement this functionality in a different language. This Pulumi program is based on the Talos documentation for standing up a cluster on AWS.

The Pulumi program has four major sections:

First, it creates the underlying base infrastructure needed for a Talos Linux cluster to run. This includes a VPC (and all the assorted other pieces, like subnets, gateways, routes, and route tables) and a load balancer. The load balancer is needed for the Kubernetes control plane, which we will bootstrap later in the program. This portion also creates the EC2 instances for the control plane.
Next, it uses the Talos Pulumi provider to generate the Talos configuration that will be applied to the instances after they are created. This configuration needs information from step 1; specifically, it needs the DNS name of the load balancer and the IP addresses of the control plane nodes.
Third, it applies the correct configuration to the control plane nodes, and launches some EC2 instances to serve as worker nodes in the Kubernetes cluster. The program then takes the Talos configuration from step 2 and applies it to the worker nodes.
The fourth and final step is to bootstrap the cluster.

For those who need a “TL;DR”: you can clone the associated repository, run pulumi up, and have a working Talos Linux cluster in a few minutes.

For those who want a bit more detail, read on.

Creating the AWS Infrastructure

The first part of the Pulumi program (running through about line 270) creates the necessary AWS infrastructure. The program leverages Crosswalk for AWS to simplify the creation of the underlying VPC and associated components. Most of the code involves creating security groups and security group rules.

This part of the program also creates the load balancer that will be used for the Kubernetes API.

Finally, the program launches three EC2 instances using one of the official Talos AMIs. In the original version of the code, you had to supply the AMI ID as a configuration value; this version looks up the AMI when it runs. The IDs and the private IP addresses for these EC2 instances are stored in two different arrays; we’ll need this information later.

Creating the Talos Configuration

The next part of the Pulumi program builds the Talos configuration, using the Pulumi provider and information (outputs) from the resources created earlier. Specifically, the Talos configuration uses the DNS name of the load balancer created earlier, and the IP addresses of the EC2 instances (which will become the control plane nodes for the Kubernetes cluster).

There are three parts to the configuration:

A client configuration file, which the talosctl command-line utility uses to connect to the Talos Linux nodes
Machine configuration for the control plane nodes
Machine configuration for the worker nodes

Configuring the Nodes

Once the Talos provider builds the correct machine configurations, the program then applies the configuration to each of the control plane nodes (worker nodes are handled later). The most efficient way would be to range over the array with the instance IP addresses and apply the configuration to each instance. However, in this particular case we need to build a dependency on applying the machine configuration, so the program does it “manually.”

Once the machine configuration has been applied to the control plane nodes, three more EC2 instances are launched for worker nodes, and then the correct machine configuration is applied to those nodes (this time iterating through the array).

Bootstrapping the Cluster

Finally, it comes down to bootstrapping the cluster (see line 397), which—again using the Talos provider for Pulumi—is accomplished with calling machine.NewBootstrap against the first control plane node to be created. The Pulumi program will complete after this; it doesn’t wait for the bootstrap process to complete (which will take a few more minutes).

At this point, you can use pulumi stack output to retrieve a configuration file for the talosctl command-line utility, and then run talosctl health to watch the cluster bootstrap. Once the cluster is up and running, talosctl kubeconfig will get you a Kubeconfig file you can use to access the Kubernetes API via kubectl. Nice!

Additional Resources

The updated Pulumi code is available on GitHub in this repository. Please note that I tested the code and it works for me, but it is supplied “as-is”. Don’t use this for your production environment without performing your own testing and validation!

Thanks for reading! I hope this post is helpful. If you have questions, you’re welcome to open an issue in the GitHub repository, or you can contact me directly. If you have a potential enhancement to the code, you’re invited to open a pull request. For more in-depth discussions, it’s not hard to get in touch with me—you can find me on Twitter, on Mastodon, and on various Slack communities (including the Kubernetes and Pulumi Slack communities).

Technology Short Take 187

Scott Lowe — Fri, 08 Aug 2025 09:00:00 EDT

Welcome to Technology Short Take #187! In this Technology Short Take, I have a curated collection of links on topics ranging from BGP to blade server hardware to writing notes using a “zettelkasten”-style approach, along with a few other topics thrown in here and there for fun. I hope you find something useful!

Networking

Jason Gintert justifies the value behind using IRRs and RPSL to keep your public BGP routing working well.
IPv6 has a new documentation prefix, as outlined by Ivan Pepelnjak in his blog post as well as by Nick Buraglio in his blog post.

Servers/Hardware

Kevin Houston takes a look at the latest blade server offerings from Cisco and HPE.

Security

This is an older article dating back to the start of 2025, but it outlines a potential security flaw that allowed hackers to hijack Subaru cars in the US and Canada.
Rory McCune tackles PKI in this post of his series on Kubernetes security fundamentals.
Daniel Stenberg (of curl fame) explains why the curl project doesn’t use CVSS scores and describes some of the problems that result from this decision.

Cloud Computing/Cloud Management

I’ve spoken about Cedar before here on this site. The first mention of Cedar was in TST 165; later, in TST 168, I pondered why AWS went through the effort of creating and open-sourcing Cedar. In a paper on Cedar (linked to via this post), I learned that consistent performance at scale was one benefit of Cedar when compared to other policy solutions, including Open Policy Agent (OPA). Good to know!
Also on the topic of Cedar: here’s an implementation of Kubernetes RBAC in Cedar.
This blog post provides some insight on how Amazon scaled EKS to support up to 100,000 nodes.
The Infisical team provides a case for using Terraform modules.
Looking for a list of useful, IaC-adjacent tools? Look no further.

Operating Systems/Applications

Marcus Hutchins explains his stance on AI. It’s a bit of a long read, but he goes into great detail for all the reasons why he’s opposed to AI in its current form (LLMs).
Julian Hofer discusses CLI tools (gh and glab, for GitHub and GitLab, respectively) and how they enhance the git CLI experience when working with these online services (which he calls “Git forges”).
The Register clues its readers in on some security changes Apple is supposedly making in iOS and macOS, leveraging a feature called “exclaves”.
Stig Brautaset describes YAML’s exponent problem.

Storage

Ben Dicken of PlanetScale has a good primer on IO devices and latency.

Virtualization

William Lam takes readers through the steps necessary to rename a vSAN virtual machine and its files.
I’ve been thinking recently that I need to give Incus a deeper look. This article on Incus installation and use (focused on storage pools and networking) will likely come in handy.

Career/Soft Skills

Ruben Schade explains why calling a technology “boring” is actually a compliment.
Scott Young provides some guidance to know when focusing on your strengths isn’t the best approach.
Over the past year or two, I’ve been experimenting with a “zettelkasten” style of note taking. Accordingly, I’ve been reading various articles and resources about this approach, what makes it work, and how to use it effectively. Among other articles, this article by Bob Doto on “bottom up” versus “top down” in note-taking was helpful.

That’s all for now. I welcome all feedback as long as it is constructive; feel free to reach out and tell me what I could improve! There are a variety of ways to get in touch with me; you can reach me via various social media sites (Twitter, Mastodon, and Bluesky), via Slack (I frequent the Kubernetes Slack community, among others), or via email (my address is on this site and it’s not too hard to guess). I’d love to hear from you!

Technology Short Take 186

Scott Lowe — Fri, 11 Jul 2025 10:00:00 EDT

Welcome to Technology Short Take #186! Yes, it’s been quite a while since I published a Technology Short Take; life has “gotten in the way,” so to speak, of gathering links to share with all of you. However, I think this crazy phase of my life is about to start settling down (I hope so, anyway), and I’m cautiously optimistic that I’ll be able to pick up the blogging pace once again. For now, though, here’s a collection of links I’ve gathered since the last Technology Short Take. I hope you find something useful here!

Networking

Nick Buraglio explains how to use Cloudflare Tunnel for IPv6-only connectivity.
A colleague recently introduced me to Ultra Ethernet, which is (as I understand it) designed for AI and HPC workloads and intends to “close the gap” between InfiniBand and Ethernet. Although, in reading this comparison between InfiniBand and Ethernet by WWT, it looks like the gap isn’t necessarily all that large. (Hat tip to Drew for the links.)

Security

CSO Online briefly touches on the scope of the Salt Typhoon hacking campaign, now revealed to include even more companies than previously thought.
161 security vulnerabilities, including three actively exploited zero-day vulnerabilities? Holy broken software! I know that no software is without security vulnerabilities, but this is just…wow. Be sure to read the linked article for more details on all the patched vulnerabilities to make sure you are appropriately protected.
Here’s an article describing how to bypass disk encryption on systems with automatic TPM2 unlock.

Cloud Computing/Cloud Management

There’s some good real-world experience contained in Jon Spriggs’ post about a few weird issues in his custom AWS EKS workers. I love that Jon also included the workarounds for each issue; this is really helpful.

Operating Systems/Applications

Matt of NSHipster explains how to use the 1Password CLI (op) to automatically inject secrets into environment files in conjunction with op run.
I appreciate Julia Evans’ post on what’s involved in getting a “modern” terminal setup. Julia’s approach is fish plus neovim plus a terminal emulator with 24-bit color support. While the autocomplete suggestions from fish look really appealing, I’ve settled into bash plus starship plus exa (or eza) and a handful of very useful other CLI tools (rg, fd, and fzf, notably).
It’s no secret that I have not (thus far) been a fan of LLMs; I suppose there’s just too much hype and not enough reality for my tastes. In spite of that—or perhaps because of that?—I appreciated David Crawshaw’s balanced and down to earth discussion of his personal experience using generative models while programming. I particularly appreciated David’s very honest acknowledgement of some of the flaws of LLMs, as well as how he shared both the good and the bad he’s uncovered in his experimentation.
I’ll be honest, I love seeing stuff like these anti-LLM scraping tools start to emerge.
What’s this—someone speaking in favor of systemd? Unthinkable!

That’s all for now! I know, this one is shorter than the typical Technology Short Take. Nevertheless, hopefully I shared something useful. If you have any feedback, please feel free to reach out! I don’t use Twitter/X very much these days, but I’m still there; you can also reach me on Mastodon, on Bluesky, or through one of the various Slack communities I frequent (such as the Kubernetes Slack instance). Thanks for reading!

Bootstrapping Dual-Stack Kubernetes on Flatcar with Kubeadm

Scott Lowe — Tue, 03 Jun 2025 09:00:00 EDT

Recently I needed to be able to stand up a dual-stack (IPv4/IPv6) Kubernetes cluster on Flatcar Container Linux using kubeadm. At first glance, this seemed like it would be relatively straightforward, but as I dug deeper into it there were a few quirks that emerged. Given these quirks, it seemed like a worthwhile process to write up and publish here. In this post, you’ll see how to use Butane and kubeadm to bootstrap a dual-stack IPv4/IPv6 Kubernetes cluster on AWS.

For those who are unfamiliar, Flatcar Container Linux is a container-optimized Linux distribution considered to be the spiritual successor to CoreOS. For configuring OS instances during provisioning, Flatcar uses Ignition (see here or here for more information). Ignition is intended to be machine-friendly, but not human-friendly. Users can use Butane to write human-friendly YAML configurations that then get transpiled into Ignition. So, when bootstrapping Kubernetes on Flatcar, users will generally use a Butane configuration that leverages kubeadm, as described in the Flatcar documentation.

While the Butane configurations in the documentation are a good start for bootstrapping Kubernetes on Flatcar, they don’t address the dual-stack use case. As outlined in the Kubernetes documentation for dual-stack support with kubeadm, you need to tell kubeadm to have the Kubernetes nodes register themselves with both the IPv4 and the IPv6 addresses, like this:

nodeRegistration:
  kubeletExtraArgs:
  - name: "node-ip"
    value: "10.100.0.2,fd00:1:2:3::2"

You can also use this syntax:

nodeRegistration:
  kubeletExtraArgs:
    node-ip: "10.100.0.2,fd00:1:2:3::2"

If you don’t specifically instruct the Kubelet to use both addresses, it will register with only one of the addresses (typically the IPv4 address), and you won’t have dual-stack support.

If you’re in a situation where you are bootstrapping Kubernetes with kubeadm after the OS boots, then you’re able to find out what IP addresses the instance is using and configure your kubeadm configuration file accordingly. In this case, however, we are running kubeadm immediately after boot (by supplying the Butane configuration as user data via Terraform with the terraform-ct provider). How are we going to know what addresses were picked up by the OS when it booted? How can we tell kubeadm which addresses to use? This is the first challenge I encountered when working out how to bootstrap a dual-stack cluster with Flatcar.

Fortunately, there is a solution: enter Afterburn. Afterburn dates from the CoreOS days but is supported for use on Flatcar (although some of the variable names change slightly; be sure to refer to the Flatcar documentation for the exact changes). Among other things, Afterburn enables users to get IP addresses in use by the system by using specific environment variables—perfect for this use case.

To use Afterburn, a few changes are needed to the Butane configuration that invokes kubeadm. Specifically:

The systemd unit for kubeadm needs to be dependent upon the Afterburn service (referenced as “coreos-metadata.service”).
The kubeadm systemd unit needs to load the environment file created by Afterburn.
An additional line is needed to invoke envsubst to supply the dynamic values provided by Afterburn.

The first change involves modifying the Requires= and After= lines in the [Unit] section of the systemd unit definition, like this (this is the Butane configuration creating the kubeadm systemd unit):

systemd:
  units:
    - name: kubeadm.service
      enabled: true
      contents: |
        [Unit]
        Description=Kubeadm service
        Requires=containerd.service coreos-metadata.service
        After=containerd.service coreos-metadata.service

The second change requires the addition of an EnvironmentFile= line at the beginning of the [Service] section. Building on the previous example, here’s what that looks like:

systemd:
  units:
    - name: kubeadm.service
      enabled: true
      contents: |
        [Unit]
        Description=Kubeadm service
        Requires=containerd.service coreos-metadata.service
        After=containerd.service coreos-metadata.service
        ConditionPathExists=!/etc/kubernetes/kubelet.conf
        [Service]
        EnvironmentFile=/run/metadata/flatcar

This sources the /run/metadata/flatcar file created by Afterburn. This simple text file contains environment variable definitions of various pieces of metadata gathered by Afterburn. Of particular interest in this case are the COREOS_EC2_IPV4_LOCAL and COREOS_EC2_IPV6 variables—these contain the local IPv4 and IPv6 addresses in use. These environment variables are now available for use by any of the rest of the commands in the systemd unit.

The third and final change uses envsubst to replace placeholders in the kubeadm configuration with the dynamic values gathered by Afterburn. Before looking at that, let’s examine the kubeadm configuration file being supplied by Butane:

storage:
  files:
    - path: /etc/kubeadm.yml
      contents:
        inline: |
          ---
          apiVersion: kubeadm.k8s.io/v1beta3
          kind: InitConfiguration
          bootstrapTokens:
          - groups:
            - system:bootstrappers:kubeadm:default-node-token
            token: ${token}
            ttl: 24h0m0s
            usages:
            - signing
            - authentication
          certificateKey: ${certificate_key}
          nodeRegistration:
            kubeletExtraArgs:
              volume-plugin-dir: ${volume_plugin_dir}
              cloud-provider: ${cloud_provider}
              node-ip: "$COREOS_EC2_IPV4_LOCAL,$COREOS_EC2_IPV6"
          skipPhases:
          - addon/kube-proxy
          ---
          apiVersion: kubeadm.k8s.io/v1beta3
          kind: ClusterConfiguration
          apiServer:
            certSANs:
              - ${external_kube_apiserver_host}
              - ${internal_kube_apiserver_host}
            extraArgs:
              cloud-provider: ${cloud_provider}
            timeoutForControlPlane: 5m0s
          controlPlaneEndpoint: "${internal_kube_apiserver_host}:${internal_kube_apiserver_port}"
          controllerManager:
            extraArgs:
              flex-volume-plugin-dir: ${volume_plugin_dir}
              cloud-provider: ${cloud_provider}
          kubernetesVersion: ${version_bare}
          networking:
            podSubnet: ${pod_cidr}
            serviceSubnet: ${service_cidr}
          clusterName: ${cluster_name}
          ---
          apiVersion: kubelet.config.k8s.io/v1beta1
          kind: KubeletConfiguration
          cgroupDriver: systemd

Some values in this configuration file are being supplied by the terraform-ct provider (these are referenced as ${variable}); they are templated in the Terraform configuration and variables are replaced. However, Terraform won’t know what the IP addresses are going to be until after the resource is created, at which point it’s too late. Hence, the node-ip line uses $VARIABLE syntax that will be picked up by envsubst and replaced with the dynamic values retrieved by Afterburn. So there are essentially two templating passes that take place: one at the Terraform layer, and a second one at the systemd/Butane layer. Confused yet?

With this configuration in mind, the third change needed in the kubeadm systemd unit created by Butane has to invoke envsubst. This change looks like this:

systemd:
  units:
    - name: kubeadm.service
      enabled: true
      contents: |
        [Unit]
        Description=Kubeadm service
        Requires=containerd.service coreos-metadata.service
        After=containerd.service coreos-metadata.service
        ConditionPathExists=!/etc/kubernetes/kubelet.conf
        [Service]
        EnvironmentFile=/run/metadata/flatcar
        Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin"
        ExecStartPre=/bin/bash -c 'envsubst < /etc/kubeadm.yml > /etc/kubeadm-final.yml'

The last line shown above is the key change. Shell redirection and pipes aren’t supported directly in systemd unit files, so it’s necessary to use /bin/bash -c (or /bin/sh -c) when invoking envsubst in order for it to work correctly. Making envsubst work properly in a systemd unit file was the second key challenge I ran into during this process.

Finally, the kubeadm systemd unit needs to invoke kubeadm itself to bootstrap the cluster. I did find at least one change from the Flatcar examples that was useful here, which is the inclusion of the --upload-certs flag. Here’s the final systemd unit for kubeadm for the first control plane node:

systemd:
  units:
    - name: kubeadm.service
      enabled: true
      contents: |
        [Unit]
        Description=Kubeadm service
        Requires=containerd.service coreos-metadata.service
        After=containerd.service coreos-metadata.service
        ConditionPathExists=!/etc/kubernetes/kubelet.conf
        [Service]
        EnvironmentFile=/run/metadata/flatcar
        Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin"
        ExecStartPre=/bin/bash -c 'envsubst < /etc/kubeadm.yml > /etc/kubeadm-final.yml'
        ExecStartPre=/opt/bin/kubeadm config images pull
        ExecStartPre=/opt/bin/kubeadm init --upload-certs --config /etc/kubeadm-final.yml
        ExecStartPre=/usr/bin/mkdir /home/${ssh_user}/.kube
        ExecStartPre=/usr/bin/cp /etc/kubernetes/admin.conf /home/${ssh_user}/.kube/config
        ExecStart=/usr/bin/chown -R ${ssh_user}:${ssh_user} /home/${ssh_user}/.kube
        [Install]
        WantedBy=multi-user.target

And here’s the systemd unit for kubeadm for additional control plane nodes:

systemd:
  units:
    - name: kubeadm.service
      enabled: true
      contents: |
        [Unit]
        Description=Kubeadm service
        Requires=containerd.service coreos-metadata.service
        After=containerd.service coreos-metadata.service
        ConditionPathExists=!/etc/kubernetes/kubelet.conf
        [Service]
        EnvironmentFile=/run/metadata/flatcar
        Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin"
        ExecStartPre=/bin/bash -c 'envsubst < /etc/kubeadm.yml > /etc/kubeadm-final.yml'
        ExecStartPre=/opt/bin/kubeadm config images pull
        ExecStartPre=sleep 10
        ExecStartPre=/opt/bin/kubeadm join --config /etc/kubeadm-final.yml
        ExecStartPre=/usr/bin/mkdir /home/${ssh_user}/.kube
        ExecStartPre=/usr/bin/cp /etc/kubernetes/admin.conf /home/${ssh_user}/.kube/config
        ExecStart=/usr/bin/chown -R ${ssh_user}:${ssh_user} /home/${ssh_user}/.kube
        [Install]
        WantedBy=multi-user.target

Finally, here’s the systemd unit for kubeadm for any worker nodes joined to the cluster:

systemd:
  units:
    - name: kubeadm.service
      enabled: true
      contents: |
        [Unit]
        Description=Kubeadm service
        Requires=containerd.service coreos-metadata.service
        After=containerd.service coreos-metadata.service
        ConditionPathExists=!/etc/kubernetes/kubelet.conf
        [Service]
        EnvironmentFile=/run/metadata/flatcar
        Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin"
        ExecStartPre=/bin/bash -c 'envsubst < /etc/kubeadm.yml > /etc/kubeadm-final.yml'
        ExecStartPre=/opt/bin/kubeadm config images pull
        ExecStart=/opt/bin/kubeadm join --config /etc/kubeadm-final.yml
        [Install]
        WantedBy=multi-user.target

The last step is supplying the full Butane configuration, including the kubeadm configuration file and the kubeadm systemd unit, as user data to the AWS instance when it boots. I did this in Terraform via the terraform-ct provider, but really just about any method you want to use will work.

Whew! That was a lot—let’s recap things:

Bootstrapping Kubernetes on Flatcar generally involves the use of a Butane configuration to perform certain tasks, create files, and define systemd unit when the system boots up. On AWS, you’ll pass this Butane configuraiton to the EC2 instance(s) as user data.
The Butane configuration needs to define both a kubeadm configuration file and a kubeadm systemd unit.
The kubeadm configuration file needs to include placeholders for the Afterburn-defined environment variables $COREOS_EC2_IPV4_LOCAL and $COREOS_EC2_IPV6 in the nodeRegistration portion of the InitConfiguration section.
The kubeadm systemd unit needs to be told to depend on Afterburn (aka “coreos-metadata.service”), needs to read in the environment file created by Afterburn, and needs to use envsubst (or whatever tool you choose) to replace the placeholders mentioned in step 3 with the actual values. This ensures that the Kubelet registers both an IPv4 and and IPv6 address, making dual-stack support active.
Shell redirection doesn’t work in systemd units, so you’ll need to use /bin/bash -c or /bin/sh -c to run whatever tool is needed to replace the placeholders.

With all this in place, you should be able to stand up Flatcar Container Linux on EC2 instances and have them bootstrap into a Kubernetes cluster with dual-stack IPv4/IPv6 support.

I hope this proves useful to other folks out there. If you have any questions, feel free to find me either on Mastodon or on Bluesky (I don’t really use Twitter/X any longer). You’re also welcome to hit me up on the Kubernetes Slack instance; I’m in both the #flatcar and #kubeadm channels. I also welcome feedback or suggested improvements to the information provided in this post, so don’t hesitate to reach out. Thanks for reading!

Technology Short Take 185

Scott Lowe — Fri, 10 Jan 2025 09:00:00 -0600

Welcome to Technology Short Take #185, the first of 2025! I’m excited for the opportunity to continue to bring readers articles and links of interest across data center- and cloud-related technologies (along with some original content along the way). I had originally intended for this post to be my last post of 2024, but personal challenges got in the way. Enough of that, though—on to the content!

Networking

Rasika Nayanajith has a detailed overview of Wi-Fi 7.
Just before the end of 2024, John Howard (whose blog I only recently discovered—lots of great content there!) posted a write-up on using Vault to provision TLS certificates onto a network appliance.
Federico Paolinelli shared information on running Podman pods as systemd units in the context of an architecture to terminate EVPN inside Kubernetes nodes. Federico’s blog is another one I only recently discovered; there’s lots of great networking content here.
Julio Perez has a nice write-up on Codespaces for network engineers and educators. (Hat tip to Ivan Pepelnjak for pointing me to Julio’s site.)

Servers/Hardware

Bryan Cantrill shares some thoughts on why Gelsinger was wrong for Intel.

Security

Brian Krebs’ article on the operations of a prolific voice phishing crew was both enlightening and frightening.
ESET has an article with information on Bootkitty, which they label as “the first UEFI bootkit designed for Linux systems.”

Cloud Computing/Cloud Management

I haven’t personally used mirrord, but this article on port forwarding with mirrord piqued my interest. I might have to give this a try soon.
Rory McCune asks the question, “When is read-only not read-only?” (This is in the context of a Kubernetes RBAC and, in Rory’s words, “a possible sharp corner”.)
Frederic Branczyk with Polar Signals discusses kubezonnet, a tool for identifying and measuring cross-zone pod network traffic in Kubernetes clusters.
This article is a good write-up on troubleshooting/identifying a memory leak that wasn’t actually there.
Dan Slimmon reviews the metrics that are relevant to latency- and throughput-optimized clusters.

Operating Systems/Applications

This article from Edward Zitron is a scathing indictment of generative AI, but I can’t really disagree with any of it. My hope—and the hope of others I’ve talked to that share the same perspective—is that the fall of GenAI doesn’t mean the fall of the entire tech economy.
Matthew Sanabria shares a list of tools worth changing to in 2025. None of these tools are new to me, but it’s nice to hear about another user’s direct experience. I will say that I have thus far dismissed LLMs—see my thoughts about generative AI in the previous bullet—but I may reconsider local use of an LLM via Ollama. I don’t have a GPU to throw at it, so the experience may be subpar.

Virtualization

William Lam kicks off his 2025 blogging content with an article on an easier way to simulate custom ESXi SMBIOS hardware strings.

Career/Soft Skills

Does social media count as a soft skill? I’m going to say “Yes” so that I can put Virginia Craft’s article on how to quit mainstream social media and join Mastodon in this section. Twitter/X is the only “mainstream social media” platform I use, and—if I’m honest—I’m using it less these days in favor of Mastodon and Bluesky.
Something Tom Hollingsworth said in his recent post looking back at 2024 really stuck with me: “I want to make sure I’m bringing you the kind of content that you want to read instead of just posting because I need to create something.” That really resonated with me. I know my blogging frequency has gone down in recent years, but I don’t want to sacrifice blogging quality just for the sake of posting more often. Thanks, Tom—it’s nice to know that other writers have the same struggles I do.

That’s all for now. If you have any feedback, or if you just want to reach out and say hello, you can find me online in a variety of places. My email address isn’t too hard to find, so you’re welcome to drop me an email message if that’s your preference. Otherwise, find me on Bluesky, on Mastodon, on Twitter, and in a variety of Slack communities. I’d love to hear from you!

Using Multiple AWS Regions with Pulumi and S3 Backend

Scott Lowe — Thu, 19 Dec 2024 12:00:00 -0600

For a while now, I’ve been using Direnv to manage environment variables when I enter or leave certain directories. Since I have to work with more than one AWS account, one of the use cases for me has been populating AWS-specific environment variables, like AWS_REGION or AWS_PROFILE. This generally works really well for me, but recently I ran into a bit of a corner case involving multiple AWS regions, Pulumi, and using S3 as the Pulumi backend. In this post, I’ll share the workaround that allows this configuration to work as expected.

I describe this as a “bit of a corner case” because it only affects specific configurations (which included my configuration):

You must be setting the AWS_REGION environment variable and not setting the aws:region configuration value used by the Pulumi AWS provider.
You must be using S3 as the backend for Pulumi, and using an S3 URL of s3://bucket-name.
You want to deploy resources into an AWS region that is different than the AWS region where the backend state bucket resides.

In my specific situation, my backend state bucket resides in the AWS us-west-2 (Oregon) region, as this offers the lowest latencies from my home office in Colorado. I control this via the PULUMI_BACKEND_URL environment variable using the s3://bucket-name syntax. I needed to deploy resources into the us-east-2 (Ohio) region, so I set the AWS_REGION environment variable accordingly. Because I used AWS_REGION, I did not set the aws:region Pulumi provider configuration value.

When I ran pulumi stack init to create a Pulumi stack for the Ohio resources, I got an error message containing the text “incorrect region, the bucket is not in ‘us-east-2’ region at endpoint ‘’, bucket is in ‘us-west-2’ region”.

After a bit of Googling, I realized the problem: Pulumi was looking for the backend state bucket in the region specified by AWS_REGION. How, then, does one deploy to one AWS region when needing to use a backend state bucket in a different region?

The answer sits buried in this section of the Pulumi documentation; specifically, in this callout (emphasis mine):

As of Pulumi CLI v3.33.1, instead of specifying the AWS Profile, add awssdk=v2 along with the region and profile to the query string.

Ah ha! All that’s required is to amend the PULUMI_BACKEND_URL setting to look something more like this:

s3://bucket-name?region=us-west-2&awssdk=v2&profile=aws-profile

That worked perfectly—Pulumi now knew to look for the backend state bucket in the us-west-2 (Oregon) region while deploying resources to the region specified by the AWS_REGION environment variable. In theory, since I didn’t need to use different AWS profiles for the backend state bucket versus where resources live, it might be possible to remove &profile=aws-profile from the S3 URL. I haven’t tested that yet.

There you have it—to use different AWS regions for your S3 backend state bucket and the resources you’re deploying, make sure your PULUMI_BACKEND_URL setting properly includes the region in the URL as described above.

I hope this helps someone else out there! Hit me up on Twitter, on Mastodon, or on Bluesky if you have any feedback or any questions.