I recently ran into an interesting issue in my cluster. One node could pull container images from AWS ECR without any problems, while another node refused to cooperate. Kubernetes kept throwing authentication errors even though everything looked correctly configured.

In this post, I will walk through:

- The problem I encountered
- The debugging process
- A few possible solutions
- The approach I eventually settled on

If you are running Kubernetes in a homelab and pulling images from Amazon Elastic Container Registry [(ECR)](https://aws.amazon.com/ecr/), this might save you some troubleshooting time.

## My Homelab Setup

My Kubernetes cluster runs on [k3s](https://fullstackwriter.dev/post/how-i-installed-kubernetes-on-my-homelab-using-k3s-on-proxmox-ubuntu?category=kubernetes) because it is lightweight and easy to operate in a small environment.

The cluster currently has two nodes:

| **Node**        | **Role**      |
| --------------- | ------------- |
| control-plane-1 | Control Plane |
| k3s-worker-1    | Worker        |

My application images are stored in AWS ECR and deployed to the cluster using a Kubernetes Deployment.

The deployment uses this image:

```bash
  <SERVICE_ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com/webappservice:webappservice-latest
```

## The Problem

I deployed the application with **two replicas**, expecting Kubernetes to schedule one pod on each node. That part worked, but the pods behaved differently depending on where they landed.

| **Node**      | **Pod Status**   |
| ------------- | ---------------- |
| Worker Node   | Running          |
| Control Plane | ImagePullBackOff |

The control plane pod kept failing with this error:

```bash
Back-off pulling image
ErrImagePull: failed to pull and unpack image
no basic auth credentials
```

This suggested the node could not authenticate with ECR.

## First Assumption: Missing Authentication

My first thought was that Kubernetes did not have credentials to access the registry. To fix that, I created a pull secret using the AWS CLI.

```bash
kubectl create secret docker-registry ecr-secret \
  --docker-server=<SERVICE_ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com \
  --docker-username=AWS \
  --docker-password=$(aws ecr get-login-password --region eu-west-2) \
  --namespace dev
```

Then I updated the deployment to reference the secret.

```bash
imagePullSecrets:
  - name: ecr-secret
```

However, the control plane node still failed to pull the image.

## Verifying the Node was the Problem

To confirm whether this was a Kubernetes issue or a node-level issue, I SSH'd into the control plane node and attempted to pull the image directly using containerd.

```bash
sudo ctr image pull <SERVICE_ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com/webappservice:latest
```

I got this error:

```bash
pull access denied
authorization failed: no basic auth credentials
```

This confirmed the issue was node authentication. Interestingly, the worker node worked fine.

At this point, I realized the worker node likely already had the image cached locally, which explained why its pod started successfully.

## Attempt 2: Node-Level Authentication

The next idea was configuring authentication at the node level.

One approach is using the [ECR credential helper](https://github.com/awslabs/amazon-ecr-credential-helper), which automatically fetches tokens from AWS whenever an image pull happens.

However, this turned out to be unreliable in my setup because k3s uses **containerd**, and containerd does not automatically read Docker credential helper configurations. This made the approach more complicated than necessary for a homelab cluster.

## The Simpler and More Reliable Solution

Instead of configuring authentication per node or per deployment, I decided to attach the ECR pull secret to the **default ServiceAccount in the namespace**.

This means any pod created in that namespace automatically inherits the image pull secret. To do this, I followed the steps below:

**Step 1: Create the ECR Pull Secret**

First, I recreated the secret.

```bash
kubectl create secret docker-registry ecr-secret \
  --docker-server=<SERVICE_ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com \
  --docker-username=AWS \
  --docker-password=$(aws ecr get-login-password --region eu-west-2) \
  --namespace dev
```

**Step 2: Attach the Secret to the Namespace ServiceAccount**

Next, I patched the default ServiceAccount in the `dev` namespace.

```bash
kubectl patch serviceaccount default \
  -n dev \
  -p '{"imagePullSecrets": [{"name": "ecr-secret"}]}'
```

Now every pod created in the namespace automatically uses the secret.

**Step 3: Restart the Failing Pod**

I deleted the failing pod, so Kubernetes would recreate it.

```bash
kubectl delete pod -n dev webappservice-hudjfi
```

The deployment created a new pod, and this time it started successfully.

**Verifying the Fix**

Running `kubectl describe pod` confirmed the container started correctly. Key parts of the output looked like this:

```bash
State: Running
Ready: True
```

The event logs also showed:

```bash
Container image already present on machine
Created container
Started container
```

At this point, both nodes were running the application successfully.

## Why I Chose the ServiceAccount Approach

There are a few ways to solve ECR authentication in Kubernetes.

| **Approach**                    | **Pros**                | **Cons**                     |
| ------------------------------- | ----------------------- | ---------------------------- |
| Node credential helper          | Automatic token refresh | Harder to configure with k3s |
| Static registry token           | Simple                  | Tokens expire after 12 hours |
| imagePullSecrets per deployment | Works                   | Repetitive configuration     |
| ServiceAccount imagePullSecret  | Simple and reusable     | Requires secret refresh      |

The **ServiceAccount approach** felt like the best tradeoff for a homelab environment because:

- It centralizes authentication
- It avoids repeating the configuration in every deployment
- It works consistently across nodes

Despite the flexibility the ServiceAccount approach brings, it still has the same problem of 12-hour expiry tokens that come with ECR tokens. Thankfully, this can be managed using a Kubernetes CronJob that refreshes the token periodically.

## Automatically Refreshing the ECR Secret

As mentioned earlier, authentication tokens from ECR expire every 12 hours. This does not affect running pods, but it can break new deployments or pods scheduled on fresh nodes if the secret has expired.

A simple way to solve this in a homelab cluster is to periodically refresh the secret using a Kubernetes CronJob.

![](https://paper-attachments.dropboxusercontent.com/s_88EBE4FB662784CC10B48EE40A4F16D609D3D7F24069A2CADFF36D32EB9F9E94_1772996213878_image.png)

This keeps the cluster working without manual intervention.

**Step 1: Create a ServiceAccount for the Job**

First, I created a ServiceAccount that the CronJob can use.

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
    name: ecr-secret-refresher
    namespace: dev
```

**Step 2: Allow the Job to Manage Secrets**

The job needs permission to update secrets in the namespace.

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
    name: ecr-secret-manager
    namespace: dev
rules:
    - apiGroups: ['']
      resources: ['secrets']
      verbs: ['get', 'create', 'delete']
```

Then bind the role to the ServiceAccount.

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
    name: ecr-secret-manager-binding
    namespace: dev
subjects:
    - kind: ServiceAccount
      name: ecr-secret-refresher
roleRef:
    kind: Role
    name: ecr-secret-manager
    apiGroup: rbac.authorization.k8s.io
```

**Step 3: Create the CronJob**

Before setting up the CronJob, the job container needs AWS credentials to call `ecr get-login-password`. In a managed cloud environment, I would typically handle this with an IAM role, but in a homelab the simplest approach is storing my AWS credentials in a Kubernetes secret and injecting them as environment variables.

```bash
kubectl create secret generic aws-credentials \
  --from-literal=AWS_ACCESS_KEY_ID=<your-access-key-id> \
  --from-literal=AWS_SECRET_ACCESS_KEY=<your-secret-access-key> \
  --namespace dev
```

Then, I created the CronJob that refreshes the secret every 11 hours.

```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
    name: refresh-ecr-secret
    namespace: dev
spec:
    schedule: '0 */11 * * *'
    jobTemplate:
        spec:
            template:
                spec:
                    serviceAccountName: ecr-secret-refresher
                    restartPolicy: OnFailure
                    containers:
                        - name: refresh-secret
                          image: amazon/aws-cli
                          env:
                              - name: AWS_REGION
                                value: eu-west-2
                          command:
                              - /bin/sh
                              - -c
                              - |
                                  TOKEN=$(aws ecr get-login-password --region $AWS_REGION)
                                  kubectl delete secret ecr-secret -n dev --ignore-not-found
                                  kubectl create secret docker-registry ecr-secret \
                                    --docker-server=<ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com \
                                    --docker-username=AWS \
                                    --docker-password=$TOKEN \
                                    -n dev
```

This job will:

1. Request a fresh login token from ECR
2. Delete the old secret
3. Recreate the secret with the new token

Since the namespace ServiceAccount already references `ecr-secret`, all new pods automatically use the refreshed credentials.

**Applying the Configuration**

I saved the resources in a file and apply them:

```bash
kubectl apply -f cronjob.yaml
```

Then I verified the CronJob with:

```bash
kubectl get cronjobs -n dev
```

![Cronjob](https://paper-attachments.dropboxusercontent.com/s_88EBE4FB662784CC10B48EE40A4F16D609D3D7F24069A2CADFF36D32EB9F9E94_1772997819523_Screenshot+2026-03-08+at+20.23.11.png)

## Final Thoughts

The most misleading part of this issue was that one pod was already running. It made the problem look like an intermittent auth failure when the real explanation was simpler: the worker node had the image cached, so it never needed to authenticate. The control plane node had no cache, no credentials, and nowhere to go.

Once that clicked, the fix was straightforward. Attaching the ECR pull secret to the namespace ServiceAccount means every pod in that namespace inherits it automatically, and the CronJob keeps the credentials fresh without any manual intervention.

If you're running a k3s homelab with images in ECR, this setup is worth the 10 minutes it takes to configure. It is less fragile than managing secrets per deployment and less complex than node-level credential helpers.


How to troubleshoot and pull images from Amazon Elastic Container Registry (ECR) into your Kubernetes

Why One Node Pulled and the Other Didn't: Debugging an ECR Image Pull Failure in Kubernetes Homelab

So you’ve installed a Linux virtual machine (VM). Congratulations! Now you’re staring at the terminal and thinking, _“What’s next?”_ _“How do I move forward?”_ _“What should I install to get the environment ready?”_ You’re not alone. We've all been there. I recently went through the same phase after setting up my Homelab. That’s a setup where you run your own servers, networks, or other infrastructure at home to learn, experiment, or support side projects. I remember wondering what to install and how to get everything running as a proper Homelab or even a lightweight production environment.

In this guide, I’ll walk you through the essential post-installation steps to get your Linux VM ready for deploying, hosting, and building software.

## 1. Initial System Updates and Package Management

Before you get into any complex setup, the first thing you should do is update your VM. This pulls in the latest security fixes, patches, and features. It might not seem exciting, but skipping this step is like leaving your front door unlocked. Outdated software is one of the easiest ways for things to go wrong.

```bash
sudo apt update
sudo apt upgrade -y
```

## 2. Install Essential Tools

With your Linux VM updated, the next step is to install some basic development tools. These will help you access the internet, manage code, and do everyday tasks more easily.

```bash
sudo apt install build-essential git curl wget vim -y
```

Here’s a quick breakdown of what each tool does:

-   `build-essential` is a package that includes tools like `gcc`, `g++`, and `make`, which are needed to compile and build software from source.
-   `git` is used to track changes in your code and collaborate with others on projects.
-   `curl` lets you fetch data from URLs directly from the terminal. It’s great for testing APIs or downloading files.
-   `wget` is another tool for downloading files from the internet. It works well for pulling entire files or even websites.
-   `vim` is a command-line text editor. It’s handy for quickly editing config files or writing code without needing a full-blown IDE.

## 3. User Management: Create a Non-Root User

When you first set up your Linux VM, you’ll usually have access as the root user. That’s convenient because you can do anything on the machine. However, it’s also risky. It's like driving without a seatbelt. The root account should be used only for core administrative tasks. You should create a regular user account for your daily work to stay safe and avoid mistakes.

```bash
# Create a new user
sudo adduser yourusername

# Add user to sudo group (for admin privileges when needed)
sudo usermod -aG sudo yourusername
```

## 3. Set Up SSH and Permission Keys for Remote Access

While using a username and password lets you access your VM, you probably don’t want to start sharing those credentials with applications that need to connect to it. For example, if you’ve set up a GitHub Action to deploy a Next.js app to your VM, using your username and password for remote access is not a good idea. Instead, you should use an SSH key. It gives you a secure way to connect to your VM without relying on a password.

Passwords are more vulnerable to brute force attacks. SSH keys help reduce that risk and give you a safer way to manage access.

To set up SSH, follow the steps below:

**Step 1: Get your VM IP address using your VM terminal**

```bash
ip addr show
```

**Step 2: Generate an SSH key on your local machine**

```bash
ssh-keygen -t ed25519
```

You’ll be asked to enter a name and some optional details. You can press Enter to accept the defaults.

**Step 3: Copy your public key to the VM**

```bash
ssh-copy-id yourusername@your.vm.ip.address
```

To confirm the key was copied successfully, run this command on your VM:

```bash
cat ~/.ssh/authorized_keys
```

**Step 4: Disable password authentication on your VM**

Use `vim` to open the SSH config file

    sudo vim /etc/ssh/sshd_config

Look for the line that says `PasswordAuthentication` and change its value to `no`. Remove the `#` and update the value if the line is commented out.

**Step 5: Restart the SSH service**

```bash
sudo systemctl restart ssh
```

**Step 6: Router Exposure and Port Forwarding**

If you want to access your VM from outside your local network, you’ll need to forward the SSH port (by default, that’s port 22) from your router to your VM. Every router is a bit different, so check your router’s manual or admin panel for how to set up port forwarding.

> **Tip:** You can change the default SSH port to something less common, like `2222`, to avoid drawing unnecessary attention from bots or scanners. Just remember to update the `Port` value in your SSH config file and restart the SSH service afterward.

## 4. Firewall Configuration and Traffic Management

Now that your VM is exposed to the internet, it’s important to lock it down. Without a firewall, it’s like leaving your house wide open with a sign that says "Come on in." You want to let in only what you need and block everything else.

The Uncomplicated Firewall (UFW) makes it easy to manage which traffic is allowed in and out of your VM.

**Step 1: Install and configure UFW**

Update your package list and install UFW.

```bash
sudo apt update && sudo apt install ufw
```

**Step 2: Allow essential services**

Let UFW know which ports you want open. For example:

```bash
sudo ufw allow 2222/tcp  # if you changed SSH port
sudo ufw allow 80/tcp    # for HTTP
sudo ufw allow 443/tcp   # for HTTPS
```

**Step 3: Enable the firewall**

Once your rules are in place, turn the firewall on.

```bash
sudo ufw enable
```

**Step 4: Check your firewall status**

```bash
sudo ufw status verbose
```

## 5. Installing and Configuring Nginx: Your First Web Server

To get a sense of what your VM can do, it’s a great idea to set up Nginx. Nginx is a lightweight, powerful web server that’s perfect for both testing and production environments. Setting it up will show that your VM is ready to serve web content and handle real-world traffic.

**Step 1: Install Nginx**

Start by updating your package list and installing Nginx.

```bash
sudo apt update && sudo apt install nginx
```

**Step 2: Start and enable the Nginx service**

Once installed, start the Nginx service and make sure it starts automatically on boot:

```bash
sudo systemctl start nginx
sudo systemctl enable nginx
```

**Step 3: Test your setup**

Open a browser and go to `http://your-vm-ip`. If everything is set up correctly, you’ll see the Nginx welcome page.

![Nginx welcome page](https://paper-attachments.dropboxusercontent.com/s_D8636135A0B0BB3900626336E0942EFDE38AEFE4AEBF27955658135C7F28048F_1743903396635_Image+06-04-2025+at+02.36.jpg)

## Wrapping Up

Setting up your Linux VM after installation might feel overwhelming at first, but each step is important for making sure your system is secure, efficient, and ready to handle your software deployment and hosting needs.

Keep in mind that system administration isn’t a one-time task. It’s an ongoing process. Make it a habit to review your security settings, keep your software updated, and be ready to adjust as your needs evolve.

Just installed a Linux VM and wondering what to do next? This guide walks you through everything you need to get your VM production-ready—from adding secure users and setting up SSH keys, to configuring firewalls, port forwarding, and even running your first Nginx web server.

Now that you've installed Linux VM, what's next?

I run k3s in my [homelab](https://fullstackwriter.dev/post/how-i-installed-kubernetes-on-my-homelab-using-k3s-on-proxmox-ubuntu?category=kubernetes), and I wanted to expose my apps to the internet without opening multiple ports or configuring complex router rules.

I had the options of NodePort or installing MetalLB. Although both approaches work, neither felt close to how things are done in production. NodePort would mean exposing a port on my router and mapping it to a node. That exposes my public IP and opens inbound traffic into my network. MetalLB is cleaner inside Kubernetes, but it still assumes you have public IP handling and proper routing at the edge.

What I wanted instead was something closer to how modern infrastructure works, that supports encrypted transport, no inbound ports, and clean DNS control.

That is why I chose Cloudflare Tunnel. The goal for me was to use my domain `demolamalomo.xyz` registered on AWS Route 53. I wanted something very close to production that caters to the following:

- Apps accessible at `*.demolamalomo.xyz` (e.g., `whoami.demolamalomo.xyz`, `grafana.demolamalomo.xyz`)
- No router port forwarding
- No public IP exposure
- Cloudflared running as Kubernetes pods, not installed on the host

## Step 1: Move the Domain to Cloudflare DNS

My first instinct was to delegate only a subdomain like `k3s.demolamalomo.xyz` to Cloudflare using NS records in Route 53. That way my root domain stays fully in Route 53 and only the Kubernetes subdomain gets managed by Cloudflare.

That didn’t work. Cloudflare's free plan does not support adding subdomain zones. You can only add root domains like `demolamalomo.yxz`. On testing, this didn’t work because anytime I dig through the nameserver, the nameservers returned are still the ones tied to Route 53.

So I moved the entire domain's DNS to Cloudflare by following the steps below:

- In the Cloudflare Dashboard, I clicked the + **Add** button, selected the **Connect a domain** option, and filled in the required field.

![Connect a domain](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774118683934_Screenshot+2026-03-04+at+01.27.15.png)
![Fill the form](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774118683991_Screenshot+2026-03-04+at+01.38.39.png)

- Selected the **Free** plan
- Cloudflare scanned and imported my existing Route 53 DNS records automatically
- Cloudflare assigned me two nameservers similar to `****.**.cloudflare.com` and `******.**.cloudflare.com`

Then I updated Route 53 to point to those nameservers.

**The First Mistake: Changing Nameservers in the Wrong Place**
Route 53 has two places where nameservers appear, and I changed the wrong one.

**Wrong place:** Route 53 >> Hosted zones >> demolamalomo.xyz >> NS record. This is just a DNS record inside the zone. Changing it does nothing to where the internet actually looks for your domain.
**Right place:** Route 53 >> **Registered domains** >> demolamalomo.xyz >> **Name servers**. This is what tells the TLD registry where to send DNS queries for your domain.

I spent time debugging why `dig clohea.net NS @1.1.1.1` still returned AWS nameservers before realizing I had changed the wrong thing. Once I updated the registered domain's nameservers, Cloudflare showed the domain as **Active**, and DNS started resolving correctly.

## Step 2: Create the Cloudflare Tunnel

In the Cloudflare Zero Trust [dashboard](https://one.dash.cloudflare.com), I created a tunnel following the steps below:

- Navigate to **Networks >> Connectors**
- Click **Create a tunnel**
- Select **Cloudflared** as the connector type

![Select Cloudflared](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774128112097_Screenshot+2026-03-21+at+21.39.42.png)

- Named it `k3s-tunnel`
- On the connector install page, copied the **tunnel token** starting with `eyJ...`
- Then, I installed Cloudflared on my machine with the token copied.
  sudo cloudflared service install eyJ......

Once Cloudflared is installed, the status shows `HEALTHY` in my dashboard.

Next, I proceeded to the next page to configure the subdomain and service (Traefik in my case)

![Configured subdomain](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774128103478_Screenshot+2026-03-21+at+22.21.07.png)

## Step 3: Deploy Cloudflared on Kubernetes

I chose to run **cloudflared** as Kubernetes pods rather than installing it on the host machine. This keeps everything inside the cluster and makes it easy to manage with standard Kubernetes tooling.

To do this, first, I created a namespace and stored the tunnel token I got earlier as a secret:

```bash
kubectl create namespace cloudflare

kubectl create secret generic cloudflared-token \
  --namespace cloudflare \
  --from-literal=token=<TUNNEL_TOKEN>
```

Then, I used the secret to create a deployment:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
    name: cloudflared
    namespace: cloudflare
spec:
    replicas: 2
    selector:
        matchLabels:
            app: cloudflared
    template:
        metadata:
            labels:
                app: cloudflared
        spec:
            containers:
                - name: cloudflared
                  image: cloudflare/cloudflared:latest
                  args:
                      - tunnel
                      - --no-autoupdate
                      - run
                  env:
                      - name: TUNNEL_TOKEN
                        valueFrom:
                            secretKeyRef:
                                name: cloudflared-token
                                key: token
                  resources:
                      requests:
                          memory: '128Mi'
                          cpu: '100m'
                      limits:
                          memory: '256Mi'
                          cpu: '200m'
```

This creates two replicas of the cloudflared instance, and the `TUNNEL_TOKEN` environment variable tells cloudflared to run in remotely-managed mode (the tunnel configuration lives in Cloudflare's dashboard, not in a local config file).

After applying, I verified the pods were connected:

```bash
kubectl logs -n cloudflare -l app=cloudflared --tail=20
```

I saw `Registered tunnel connection` messages with `protocol=quic`, confirming the tunnel was live.

## Step 4: Deploy a Test App

I deployed the `traefik/whoami` image as a simple test:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
    name: whoami
    namespace: default
spec:
    replicas: 1
    selector:
        matchLabels:
            app: whoami
    template:
        metadata:
            labels:
                app: whoami
        spec:
            containers:
                - name: whoami
                  image: traefik/whoami:latest
                  ports:
                      - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
    name: whoami
    namespace: default
spec:
    selector:
        app: whoami
    ports:
        - port: 80
          targetPort: 80
```

Also added the Ingress rule telling Traefik how to route based on the Host header:

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
    name: whoami-ingress
    namespace: default
spec:
    rules:
        - host: whoami.demolamalomo.xyz
          http:
              paths:
                  - path: /
                    pathType: Prefix
                    backend:
                        service:
                            name: whoami
                            port:
                                number: 80
```

**The 502 Problem**
At this point, I expected everything to work with the DNS resolved. The tunnel was connected. But when I hit `https://whoami.demolamalomo.xyz`, I got a **502 Bad Gateway** from Cloudflare.

This was the most confusing problem because:

- The cloudflared pods were healthy and connected
- The tunnel logs showed no request-level errors
- From a debug pod inside the cluster, curling `http://traefik.kube-system.svc.cluster.local:80` with the correct Host header returned a perfect 200 response

The issue was **DNS resolution inside the cloudflared container**. My tunnel config originally pointed to:

```bash
http://traefik.kube-system.svc.cluster.local:80
```

The cloudflared Docker image is built on scratch, an essentially empty container with nothing but the cloudflared binary, with no `shell`, `cat` , `curl` , or `/etc/resolv.conf`. When cloudflared tried to resolve `traefik.kube-system.svc.cluster.local`, it had no idea which DNS server to ask and failed silently. The request never reached Traefik, so Cloudflare returned a `502`.

I confirmed this by switching the tunnel config to use **Traefik's ClusterIP directly**:

```bash
http://10.43.250.137:80
```

Immediately, I ran curl `https://whoami.clohea.net` , and it returned the whoami response with Cloudflare headers confirming traffic was flowing through the tunnel.

**Why the ClusterIP Approach Is Fine**
In a homelab, the Traefik service ClusterIP is stable. It only changes if you delete and recreate the service. For a more robust solution, you could add explicit DNS config to the cloudflared deployment:

```yaml
spec:
    template:
        spec:
            dnsPolicy: 'ClusterFirst'
            dnsConfig:
                nameservers:
                    - 10.43.0.10 # kube-dns ClusterIP
                searches:
                    - kube-system.svc.cluster.local
                    - svc.cluster.local
                    - cluster.local
```

But for a homelab, the ClusterIP approach is simpler and works.

**Adding New Apps**
The best part about this setup is that adding a new app requires zero changes to Cloudflare. The wildcard `*.demolamalomo.xyz` already covers every subdomain. I just need to:

- A Deployment and Service for my app
- An Ingress with the right hostname

The architecture looks like this:

![](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774133078724_image.png)

## Lessons Learned

- **Route 53 has two places for nameservers.** The hosted zone NS record is not the same as the registered domain's nameserver setting. Change the wrong one and nothing happens. Change it under **Registered domains** >> **Name servers**.
- **The Cloudflared Docker image cannot resolve Kubernetes DNS names.** It is built on a scratch base image with no `/etc/resolv.conf`. Either use the target service's ClusterIP directly or add explicit `dnsConfig` to the pod spec.
- **Ingress host names must match exactly.** If your tunnel routes to `*.clohea.net` but your Ingress says `whoami.k3s.clohea.net`, Traefik will not match the request and you will get a 502.


I wanted production-style ingress for my k3s homelab without exposing ports or my public IP. What looked simple turned into debugging Route 53 nameservers, Kubernetes DNS resolution, and mysterious Cloudflare 502s before everything finally worked.

How I Exposed My Kubernetes Homelab to the Internet Using Cloudflare Tunnel

I recently set up Kubernetes in my homelab to build real operational experience, not just theoretical knowledge. My goal was to run something that behaves as close as possible to a production environment, similar to what you’d get with [Amazon Elastic Kubernetes Service](https://aws.amazon.com/eks/), while still keeping things lightweight and fully under my control.

To make that happen, I deployed a Kubernetes cluster using [k3s](https://k3s.io/) on an Ubuntu virtual machine hosted inside [Proxmox VE](https://www.proxmox.com/en/products/proxmox-virtual-environment/overview).

This article walks through exactly how I did it, why each step matters, and what the setup gives me moving forward.

## My Starting Point: Virtualized Infrastructure

Before installing Kubernetes, I already had a virtualization environment running on Proxmox. Inside it, I created a virtual machine running Ubuntu. That VM would serve as my Kubernetes control plane node.

Using virtualization mirrors how real production environments operate. Cloud providers run Kubernetes on virtual infrastructure, not bare laptops. My Proxmox VM plays the same role as a cloud compute instance, giving me isolation, resource control, and the ability to snapshot or rebuild when needed.

## Why I Chose k3s

I wanted a Kubernetes distribution that felt real but did not demand heavy hardware or complex setup.

k3s was the perfect fit because:

- It is fully Kubernetes compliant
- It runs with minimal resource usage
- Installation is extremely fast
- It removes operational overhead while preserving core behavior

In practice, it provides a learning environment that behaves like a production cluster without the operational overhead of a full upstream Kubernetes deployment.

## Phase 1: Preparing the Ubuntu Server

Before installing Kubernetes, the operating system had to be configured correctly. Kubernetes is sensitive to host configuration, especially networking and memory management.

**Static IP Configuration**

During Ubuntu installation, I manually assigned a static IP address for:

- Subnet
- Address
- Gateway
- DNS

This IP becomes the Kubernetes API endpoint. If it changes, cluster access breaks. Production clusters rely on stable control plane endpoints, so I treated my homelab the same way.

**Initial Server Setup**

After installation, I connected via SSH and updated the system packages. I installed common administrative tools such as curl, git, vim, htop, and networking utilities. These support everyday server operations like monitoring, editing configs, and downloading resources.

```bash
sudo apt update && sudo apt upgrade -y
sudo apt install -y curl wget git vim htop net-tools
```

**Proxmox Integration with QEMU Guest Agent**

I installed the QEMU guest agent to improve communication between the VM and Proxmox. This allows the hypervisor to detect IP addresses, perform graceful shutdowns, and monitor the VM’s state more reliably.

```bash
sudo apt install -y qemu-guest-agent
sudo systemctl enable qemu-guest-agent
sudo systemctl start qemu-guest-agent
```

**Network Validation**

Before going further, I verified:

- The static IP was assigned correctly
- External network access worked
- DNS resolution worked

Kubernetes depends heavily on networking. If connectivity is unstable, cluster components fail in unpredictable ways.

```bash
ip addr show ens18
# Should show your static IP
```

**Disabling Swap**

Kubernetes requires swap to be disabled because it interferes with resource scheduling and memory guarantees. This is a standard requirement across production clusters, so I disabled swap and made the change persistent.

```bash
sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstab

# Verify
free -h
# Swap should show 0
```

**Hostname Configuration**

I set the server hostname and mapped it in `/etc/hosts` so the system could resolve itself consistently. Stable node identity is important for certificates, networking, and cluster communication.

```bash
sudo hostnamectl set-hostname k3s-demo.local
```

Then modify `/etc/hosts` file to map the hostname:

```bash
# Update /etc/hosts
sudo vim /etc/hosts

# Add this line
<SERVER IP> k3s-demo.local k3s-demo
```

## Phase 2: Installing k3s

With the system prepared, I moved on to installing Kubernetes.

**Pre-Installation Checks**

Before running the installer, I confirmed:

- The required ports were free
- Enough RAM and CPU were available
- Sufficient disk space existed

This avoids conflicts and failed cluster startup.

```bash
# Check available ports
sudo ss -tulpn | grep -E ':(6443|10250)'

# Should return nothing (ports free)

# Check system resources
free -h  # Should show number of RAM
nproc    # Should show number of cores
df -h    # Should show plenty of disk space
```

**Installing k3s**

I installed k3s using the official script with three key options:

```bash
curl -sfL https://get.k3s.io | sh -s - \
  --write-kubeconfig-mode 644 \
  --disable traefik \
  --node-name k3s-demo
```

- `--write-kubeconfig-mode 644`: Makes kubectl work without sudo
- `--disable traefik`: Default Traefik ingress disabled so I can manage ingress myself later
- `--node-name`: Explicit node name for clarity and future scaling

The installation completed in under a minute.

**Verifying Cluster Health**

After installation, I confirmed:

- The K3s service was running
- The node reported Ready
- System pods in the kube-system namespace were healthy

If anything fails, system logs provide detailed troubleshooting information.

```bash
# Check k3s service
sudo systemctl status k3s

# Should see: active (running)

# Check node
kubectl get nodes

# Expected output:
# NAME          STATUS   ROLES                  AGE   VERSION
# k3s-demo    Ready    control-plane,master   1m    v1.28.x

# Check all system pods
kubectl get pods -A

# Should see pods in kube-system namespace running
```

![Running kubernetes](https://paper-attachments.dropboxusercontent.com/s_F3D46302D17DF57EA141A194AFED6667E114BCD5CC881287885039BD0D55EE8B_1772073514109_Image+26-02-2026+at+03.38.png)

## Does k3s Have Only a Control Plane? What About Worker Nodes?

When I ran `kubectl get nodes` to see the nodes in my Kubernetes cluster, I only saw a control plane node.

That worried me a bit because from what I understood about Kubernetes, the control plane, which acts as the brain, is separate from worker nodes, where applications actually run.

I expected my setup to look something like this:

```bash
NAME              STATUS   ROLES           AGE   VERSION
control-plane-1   Ready    control-plane   1h    v1.28.x
worker-1          Ready    <none>          1h    v1.28.x
worker-2          Ready    <none>          1h    v1.28.x
worker-3          Ready    <none>          1h    v1.28.x
```

After digging deeper into k3s architecture, I discovered this behavior is intentional. In k3s, a single provisioned node acts as both the control plane and the worker node. It manages the cluster and runs workloads at the same time.

This design is perfect for development, testing, and pre-production environments, which is exactly what most homelabs are built for.

Still, that setup was not enough for me. I wanted the real experience of managing a cluster with a separate control plane and worker nodes. Thankfully, k3s supports adding worker nodes, and that is exactly what I did.

**How to Add a Worker Node to k3s**

To add a worker node, the first step is to retrieve the control plane token. This token is required to join a new node to the cluster.

On the control plane VM, run:

```bash
sudo cat /var/lib/rancher/k3s/server/node-token
```

The output will look something like this:

```bash
K10abc123def456g.....
```

Next, I provisioned another VM, just like I did for the control plane. I disabled swap, installed the QEMU agent, and configured the necessary dependencies to prepare the VM for Kubernetes.

After that, I installed k3s in agent mode by running the following command on the new VM:

```bash
curl -sfL https://get.k3s.io | K3S_URL=https://<CONTROL_PLANE_IP> \
  K3S_TOKEN="K10abc123def456g......" \
  sh -
```

Here:

- `K3S_URL` is the control plane IP address
- `K3S_TOKEN` is the token retrieved earlier

Finally, I verified that the worker node successfully joined the cluster by running the following command on the control plane VM:

```bash
kubectl get nodes
```

![k3s with control plane and worker nodes](https://paper-attachments.dropboxusercontent.com/s_F3D46302D17DF57EA141A194AFED6667E114BCD5CC881287885039BD0D55EE8B_1772630050547_Image+04-03-2026+at+14.13.png)

## Final Thoughts

Setting up Kubernetes yourself changes how you understand it. Instead of interacting with a managed black box, you see every layer that makes the platform work.

Running k3s on a Proxmox-hosted Ubuntu server gives me a controlled, production-like environment where I can experiment, break things, and learn deeply without risk.

For anyone serious about mastering Kubernetes, building your own cluster is one of the most valuable steps you can take.


I built a Kubernetes cluster in my homelab using k3s on an Ubuntu VM running inside Proxmox. This guide walks through the full setup process, from server preparation to adding worker nodes.

#Homelab

Why One Node Pulled and the Other Didn't: Debugging an ECR Image Pull Failure in Kubernetes Homelab

Now that you've installed Linux VM, what's next?

How I Exposed My Kubernetes Homelab to the Internet Using Cloudflare Tunnel

How I Installed Kubernetes on My Homelab (Using k3s on Proxmox + Ubuntu)