I recently ran into an interesting issue in my cluster. One node could pull container images from AWS ECR without any problems, while another node refused to cooperate. Kubernetes kept throwing authentication errors even though everything looked correctly configured.

In this post, I will walk through:

- The problem I encountered
- The debugging process
- A few possible solutions
- The approach I eventually settled on

If you are running Kubernetes in a homelab and pulling images from Amazon Elastic Container Registry [(ECR)](https://aws.amazon.com/ecr/), this might save you some troubleshooting time.

## My Homelab Setup

My Kubernetes cluster runs on [k3s](https://fullstackwriter.dev/post/how-i-installed-kubernetes-on-my-homelab-using-k3s-on-proxmox-ubuntu?category=kubernetes) because it is lightweight and easy to operate in a small environment.

The cluster currently has two nodes:

| **Node**        | **Role**      |
| --------------- | ------------- |
| control-plane-1 | Control Plane |
| k3s-worker-1    | Worker        |

My application images are stored in AWS ECR and deployed to the cluster using a Kubernetes Deployment.

The deployment uses this image:

```bash
  <SERVICE_ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com/webappservice:webappservice-latest
```

## The Problem

I deployed the application with **two replicas**, expecting Kubernetes to schedule one pod on each node. That part worked, but the pods behaved differently depending on where they landed.

| **Node**      | **Pod Status**   |
| ------------- | ---------------- |
| Worker Node   | Running          |
| Control Plane | ImagePullBackOff |

The control plane pod kept failing with this error:

```bash
Back-off pulling image
ErrImagePull: failed to pull and unpack image
no basic auth credentials
```

This suggested the node could not authenticate with ECR.

## First Assumption: Missing Authentication

My first thought was that Kubernetes did not have credentials to access the registry. To fix that, I created a pull secret using the AWS CLI.

```bash
kubectl create secret docker-registry ecr-secret \
  --docker-server=<SERVICE_ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com \
  --docker-username=AWS \
  --docker-password=$(aws ecr get-login-password --region eu-west-2) \
  --namespace dev
```

Then I updated the deployment to reference the secret.

```bash
imagePullSecrets:
  - name: ecr-secret
```

However, the control plane node still failed to pull the image.

## Verifying the Node was the Problem

To confirm whether this was a Kubernetes issue or a node-level issue, I SSH'd into the control plane node and attempted to pull the image directly using containerd.

```bash
sudo ctr image pull <SERVICE_ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com/webappservice:latest
```

I got this error:

```bash
pull access denied
authorization failed: no basic auth credentials
```

This confirmed the issue was node authentication. Interestingly, the worker node worked fine.

At this point, I realized the worker node likely already had the image cached locally, which explained why its pod started successfully.

## Attempt 2: Node-Level Authentication

The next idea was configuring authentication at the node level.

One approach is using the [ECR credential helper](https://github.com/awslabs/amazon-ecr-credential-helper), which automatically fetches tokens from AWS whenever an image pull happens.

However, this turned out to be unreliable in my setup because k3s uses **containerd**, and containerd does not automatically read Docker credential helper configurations. This made the approach more complicated than necessary for a homelab cluster.

## The Simpler and More Reliable Solution

Instead of configuring authentication per node or per deployment, I decided to attach the ECR pull secret to the **default ServiceAccount in the namespace**.

This means any pod created in that namespace automatically inherits the image pull secret. To do this, I followed the steps below:

**Step 1: Create the ECR Pull Secret**

First, I recreated the secret.

```bash
kubectl create secret docker-registry ecr-secret \
  --docker-server=<SERVICE_ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com \
  --docker-username=AWS \
  --docker-password=$(aws ecr get-login-password --region eu-west-2) \
  --namespace dev
```

**Step 2: Attach the Secret to the Namespace ServiceAccount**

Next, I patched the default ServiceAccount in the `dev` namespace.

```bash
kubectl patch serviceaccount default \
  -n dev \
  -p '{"imagePullSecrets": [{"name": "ecr-secret"}]}'
```

Now every pod created in the namespace automatically uses the secret.

**Step 3: Restart the Failing Pod**

I deleted the failing pod, so Kubernetes would recreate it.

```bash
kubectl delete pod -n dev webappservice-hudjfi
```

The deployment created a new pod, and this time it started successfully.

**Verifying the Fix**

Running `kubectl describe pod` confirmed the container started correctly. Key parts of the output looked like this:

```bash
State: Running
Ready: True
```

The event logs also showed:

```bash
Container image already present on machine
Created container
Started container
```

At this point, both nodes were running the application successfully.

## Why I Chose the ServiceAccount Approach

There are a few ways to solve ECR authentication in Kubernetes.

| **Approach**                    | **Pros**                | **Cons**                     |
| ------------------------------- | ----------------------- | ---------------------------- |
| Node credential helper          | Automatic token refresh | Harder to configure with k3s |
| Static registry token           | Simple                  | Tokens expire after 12 hours |
| imagePullSecrets per deployment | Works                   | Repetitive configuration     |
| ServiceAccount imagePullSecret  | Simple and reusable     | Requires secret refresh      |

The **ServiceAccount approach** felt like the best tradeoff for a homelab environment because:

- It centralizes authentication
- It avoids repeating the configuration in every deployment
- It works consistently across nodes

Despite the flexibility the ServiceAccount approach brings, it still has the same problem of 12-hour expiry tokens that come with ECR tokens. Thankfully, this can be managed using a Kubernetes CronJob that refreshes the token periodically.

## Automatically Refreshing the ECR Secret

As mentioned earlier, authentication tokens from ECR expire every 12 hours. This does not affect running pods, but it can break new deployments or pods scheduled on fresh nodes if the secret has expired.

A simple way to solve this in a homelab cluster is to periodically refresh the secret using a Kubernetes CronJob.

![](https://paper-attachments.dropboxusercontent.com/s_88EBE4FB662784CC10B48EE40A4F16D609D3D7F24069A2CADFF36D32EB9F9E94_1772996213878_image.png)

This keeps the cluster working without manual intervention.

**Step 1: Create a ServiceAccount for the Job**

First, I created a ServiceAccount that the CronJob can use.

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
    name: ecr-secret-refresher
    namespace: dev
```

**Step 2: Allow the Job to Manage Secrets**

The job needs permission to update secrets in the namespace.

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
    name: ecr-secret-manager
    namespace: dev
rules:
    - apiGroups: ['']
      resources: ['secrets']
      verbs: ['get', 'create', 'delete']
```

Then bind the role to the ServiceAccount.

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
    name: ecr-secret-manager-binding
    namespace: dev
subjects:
    - kind: ServiceAccount
      name: ecr-secret-refresher
roleRef:
    kind: Role
    name: ecr-secret-manager
    apiGroup: rbac.authorization.k8s.io
```

**Step 3: Create the CronJob**

Before setting up the CronJob, the job container needs AWS credentials to call `ecr get-login-password`. In a managed cloud environment, I would typically handle this with an IAM role, but in a homelab the simplest approach is storing my AWS credentials in a Kubernetes secret and injecting them as environment variables.

```bash
kubectl create secret generic aws-credentials \
  --from-literal=AWS_ACCESS_KEY_ID=<your-access-key-id> \
  --from-literal=AWS_SECRET_ACCESS_KEY=<your-secret-access-key> \
  --namespace dev
```

Then, I created the CronJob that refreshes the secret every 11 hours.

```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
    name: refresh-ecr-secret
    namespace: dev
spec:
    schedule: '0 */11 * * *'
    jobTemplate:
        spec:
            template:
                spec:
                    serviceAccountName: ecr-secret-refresher
                    restartPolicy: OnFailure
                    containers:
                        - name: refresh-secret
                          image: amazon/aws-cli
                          env:
                              - name: AWS_REGION
                                value: eu-west-2
                          command:
                              - /bin/sh
                              - -c
                              - |
                                  TOKEN=$(aws ecr get-login-password --region $AWS_REGION)
                                  kubectl delete secret ecr-secret -n dev --ignore-not-found
                                  kubectl create secret docker-registry ecr-secret \
                                    --docker-server=<ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com \
                                    --docker-username=AWS \
                                    --docker-password=$TOKEN \
                                    -n dev
```

This job will:

1. Request a fresh login token from ECR
2. Delete the old secret
3. Recreate the secret with the new token

Since the namespace ServiceAccount already references `ecr-secret`, all new pods automatically use the refreshed credentials.

**Applying the Configuration**

I saved the resources in a file and apply them:

```bash
kubectl apply -f cronjob.yaml
```

Then I verified the CronJob with:

```bash
kubectl get cronjobs -n dev
```

![Cronjob](https://paper-attachments.dropboxusercontent.com/s_88EBE4FB662784CC10B48EE40A4F16D609D3D7F24069A2CADFF36D32EB9F9E94_1772997819523_Screenshot+2026-03-08+at+20.23.11.png)

## Final Thoughts

The most misleading part of this issue was that one pod was already running. It made the problem look like an intermittent auth failure when the real explanation was simpler: the worker node had the image cached, so it never needed to authenticate. The control plane node had no cache, no credentials, and nowhere to go.

Once that clicked, the fix was straightforward. Attaching the ECR pull secret to the namespace ServiceAccount means every pod in that namespace inherits it automatically, and the CronJob keeps the credentials fresh without any manual intervention.

If you're running a k3s homelab with images in ECR, this setup is worth the 10 minutes it takes to configure. It is less fragile than managing secrets per deployment and less complex than node-level credential helpers.


How to troubleshoot and pull images from Amazon Elastic Container Registry (ECR) into your Kubernetes

Why One Node Pulled and the Other Didn't: Debugging an ECR Image Pull Failure in Kubernetes Homelab

Running Kubernetes in a homelab is fun until everything starts living in the terminal.

The CLI is powerful. Tools like `kubectl` give you full control over your cluster. But after a while, constantly running commands just to inspect workloads or check logs can slow things down.

I wanted something closer to what many teams use in production. A visual interface where I could see workloads, inspect containers, and manage resources quickly.

That led me to [Portainer](https://www.portainer.io/).

In this article, I walk through how I installed Portainer on my Kubernetes homelab running on **Proxmox Server Solutions**, the issues I ran into along the way, and how I fixed them.

## My Homelab Setup

My infrastructure currently looks like this:

```bash
Proxmox host
    └── Virtual machines
        └── Kubernetes cluster (k3s)
                └── Container workloads
```

The Kubernetes cluster runs inside VMs managed by Proxmox, and most of my applications run as containers inside the cluster.

Portainer runs inside [Kubernetes](https://fullstackwriter.dev/post/how-i-installed-kubernetes-on-my-homelab-using-k3s-on-proxmox-ubuntu?category=kubernetes) as well, which allows it to interact directly with the Kubernetes API and manage cluster resources.

## Why I Wanted Portainer

Even though the CLI works perfectly, I wanted a management layer that offers:

- A visual view of workloads
- Faster troubleshooting
- Easier inspection of pods and logs
- Cluster-wide visibility
- A user interface similar to production environments

The goal was not to replace the CLI. It was to complement it.

In real infrastructure environments, engineers rarely rely on commands alone. Tools that visualize cluster state make day-to-day operations much easier.

## Step 1. Install Helm (If Not Already Installed)

[Helm](https://helm.sh/) is a package manager for Kubernetes (similar to NPM for Node.js and Cargo for Rust). Portainer provides an official Helm chart, so Helm is required for installation.

To check if Helm is installed:

```bash
helm version
```

If not installed, [install Helm](https://helm.sh/docs/intro/install) based on your operating system.

## Step 2. Add the Portainer Helm Repository

To do this, I ran the command below:

```bash
helm repo add portainer https://portainer.github.io/k8s/
helm repo update
```

This makes the Portainer chart available locally.

## Step 3. First Installation Attempt (And the Error I Hit)

When I first tried installing Portainer, I got this error:

```bash
cluster reachability check failed:
kubernetes cluster unreachable:
Get "http://localhost:8080/version":
connect: connection refused
```

It means Helm could not find my Kubernetes cluster configuration. It defaulted to trying `localhost:8080`, which does not exist.

This happened because my cluster runs [k3s](https://k3s.io/), which stores its kubeconfig in:

```bash
/etc/rancher/k3s/k3s.yaml
```

`kubectl` was automatically using this file, but Helm was not. So Helm had no cluster credentials.

## Step 4. Fixing the Cluster Reachability Error

I explicitly exported the kubeconfig:

```bash
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
```

After that, Helm could communicate with Kubernetes. I verified by running the command below

```bash
helm list
```

To make this changes permanent, I ran:

```bash
echo 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml' >> ~/.bashrc
source ~/.bashrc
```

## Step 5. Install Portainer in Kubernetes

Once Helm could talk to the cluster, installation was straightforward. I did so by running the command below:

```bash
helm install portainer portainer/portainer \
  --namespace portainer \
  --create-namespace \
  --set service.type=NodePort
```

This deploys:

- Portainer server
- Kubernetes integration components
- A service for web access

## Step 6. Verify Deployment

Check pods:

```bash
kubectl get pods -n portainer
```

I waited until the status showed `Running`:

![Portainer running on Kubernetes](https://paper-attachments.dropboxusercontent.com/s_05AD378A82578B8F940350A334CC24A83B9D9779BBD50D7F99799A62C18025AC_1772907054520_Screenshot+2026-03-07+at+19.10.10.png)

## Step 7. Access the Portainer UI

To access the UI, first, I need to find the service on which Portainer is exposed:

```bash
kubectl get svc -n portainer
```

Then looked for the NodePort value.

![Portainer NodePort value](https://paper-attachments.dropboxusercontent.com/s_05AD378A82578B8F940350A334CC24A83B9D9779BBD50D7F99799A62C18025AC_1772907432360_Screenshot+2026-03-07+at+19.13.20.png)


Access in browser:

```bash
https://<node-ip>:<nodeport>
```

## Step 8. Second Issue I Encountered (Security Timeout)

When I opened the UI, I saw:

![Timeout error](https://paper-attachments.dropboxusercontent.com/s_05AD378A82578B8F940350A334CC24A83B9D9779BBD50D7F99799A62C18025AC_1772907578117_screencapture-10-90-1-30-30777-timeout-html-2026-02-25-03_41_32.png)


This happens if you do not create the admin account quickly after the first launch, which I didn’t in my case. It’s a security measure with Portainer. It locks itself after some time.

## Step 9. Fixing the Timeout

To fix it, I restarted the  Portainer deployment:

```bash
kubectl rollout restart deployment portainer -n portainer
```

Then waited for the pod to become `Running`, then refreshed the browser.

This time, I created the admin account immediately.

![Portainer Admin Account](https://paper-attachments.dropboxusercontent.com/s_05AD378A82578B8F940350A334CC24A83B9D9779BBD50D7F99799A62C18025AC_1772907813899_screencapture-10-90-1-30-30777-2026-02-25-03_44_25.png)

## Final Result

After logging in, Portainer automatically detected my Kubernetes environment. From there, I could:

- View nodes and namespaces
- Inspect pods and logs
- Deploy applications
- Scale workloads
- Manage cluster resources visually

Everything worked out of the box.

![Portainer Admin](https://paper-attachments.dropboxusercontent.com/s_05AD378A82578B8F940350A334CC24A83B9D9779BBD50D7F99799A62C18025AC_1772907944407_screencapture-10-90-1-30-30777-2026-02-25-03_46_30.png)

## Final Thoughts

Running Kubernetes in a homelab is a great way to learn infrastructure, but tools like Portainer add an important layer of usability.

It provides a clear view of what is happening inside the cluster and makes management easier without removing the flexibility of the command line.

For anyone running Kubernetes at home, especially on virtualized infrastructure, adding a management interface is a worthwhile improvement.



A step-by-step guide to installing Portainer on a Kubernetes (k3s) homelab running on Proxmox. Covers Helm setup, fixing cluster reachability errors, resolving the security timeout, and accessing the Portainer UI.

How I Installed Portainer on My Kubernetes Homelab (Running on Proxmox)

I run k3s in my [homelab](https://fullstackwriter.dev/post/how-i-installed-kubernetes-on-my-homelab-using-k3s-on-proxmox-ubuntu?category=kubernetes), and I wanted to expose my apps to the internet without opening multiple ports or configuring complex router rules.

I had the options of NodePort or installing MetalLB. Although both approaches work, neither felt close to how things are done in production. NodePort would mean exposing a port on my router and mapping it to a node. That exposes my public IP and opens inbound traffic into my network. MetalLB is cleaner inside Kubernetes, but it still assumes you have public IP handling and proper routing at the edge.

What I wanted instead was something closer to how modern infrastructure works, that supports encrypted transport, no inbound ports, and clean DNS control.

That is why I chose Cloudflare Tunnel. The goal for me was to use my domain `demolamalomo.xyz` registered on AWS Route 53. I wanted something very close to production that caters to the following:

- Apps accessible at `*.demolamalomo.xyz` (e.g., `whoami.demolamalomo.xyz`, `grafana.demolamalomo.xyz`)
- No router port forwarding
- No public IP exposure
- Cloudflared running as Kubernetes pods, not installed on the host

## Step 1: Move the Domain to Cloudflare DNS

My first instinct was to delegate only a subdomain like `k3s.demolamalomo.xyz` to Cloudflare using NS records in Route 53. That way my root domain stays fully in Route 53 and only the Kubernetes subdomain gets managed by Cloudflare.

That didn’t work. Cloudflare's free plan does not support adding subdomain zones. You can only add root domains like `demolamalomo.yxz`. On testing, this didn’t work because anytime I dig through the nameserver, the nameservers returned are still the ones tied to Route 53.

So I moved the entire domain's DNS to Cloudflare by following the steps below:

- In the Cloudflare Dashboard, I clicked the + **Add** button, selected the **Connect a domain** option, and filled in the required field.

![Connect a domain](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774118683934_Screenshot+2026-03-04+at+01.27.15.png)
![Fill the form](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774118683991_Screenshot+2026-03-04+at+01.38.39.png)

- Selected the **Free** plan
- Cloudflare scanned and imported my existing Route 53 DNS records automatically
- Cloudflare assigned me two nameservers similar to `****.**.cloudflare.com` and `******.**.cloudflare.com`

Then I updated Route 53 to point to those nameservers.

**The First Mistake: Changing Nameservers in the Wrong Place**
Route 53 has two places where nameservers appear, and I changed the wrong one.

**Wrong place:** Route 53 >> Hosted zones >> demolamalomo.xyz >> NS record. This is just a DNS record inside the zone. Changing it does nothing to where the internet actually looks for your domain.
**Right place:** Route 53 >> **Registered domains** >> demolamalomo.xyz >> **Name servers**. This is what tells the TLD registry where to send DNS queries for your domain.

I spent time debugging why `dig clohea.net NS @1.1.1.1` still returned AWS nameservers before realizing I had changed the wrong thing. Once I updated the registered domain's nameservers, Cloudflare showed the domain as **Active**, and DNS started resolving correctly.

## Step 2: Create the Cloudflare Tunnel

In the Cloudflare Zero Trust [dashboard](https://one.dash.cloudflare.com), I created a tunnel following the steps below:

- Navigate to **Networks >> Connectors**
- Click **Create a tunnel**
- Select **Cloudflared** as the connector type

![Select Cloudflared](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774128112097_Screenshot+2026-03-21+at+21.39.42.png)

- Named it `k3s-tunnel`
- On the connector install page, copied the **tunnel token** starting with `eyJ...`
- Then, I installed Cloudflared on my machine with the token copied.
  sudo cloudflared service install eyJ......

Once Cloudflared is installed, the status shows `HEALTHY` in my dashboard.

Next, I proceeded to the next page to configure the subdomain and service (Traefik in my case)

![Configured subdomain](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774128103478_Screenshot+2026-03-21+at+22.21.07.png)

## Step 3: Deploy Cloudflared on Kubernetes

I chose to run **cloudflared** as Kubernetes pods rather than installing it on the host machine. This keeps everything inside the cluster and makes it easy to manage with standard Kubernetes tooling.

To do this, first, I created a namespace and stored the tunnel token I got earlier as a secret:

```bash
kubectl create namespace cloudflare

kubectl create secret generic cloudflared-token \
  --namespace cloudflare \
  --from-literal=token=<TUNNEL_TOKEN>
```

Then, I used the secret to create a deployment:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
    name: cloudflared
    namespace: cloudflare
spec:
    replicas: 2
    selector:
        matchLabels:
            app: cloudflared
    template:
        metadata:
            labels:
                app: cloudflared
        spec:
            containers:
                - name: cloudflared
                  image: cloudflare/cloudflared:latest
                  args:
                      - tunnel
                      - --no-autoupdate
                      - run
                  env:
                      - name: TUNNEL_TOKEN
                        valueFrom:
                            secretKeyRef:
                                name: cloudflared-token
                                key: token
                  resources:
                      requests:
                          memory: '128Mi'
                          cpu: '100m'
                      limits:
                          memory: '256Mi'
                          cpu: '200m'
```

This creates two replicas of the cloudflared instance, and the `TUNNEL_TOKEN` environment variable tells cloudflared to run in remotely-managed mode (the tunnel configuration lives in Cloudflare's dashboard, not in a local config file).

After applying, I verified the pods were connected:

```bash
kubectl logs -n cloudflare -l app=cloudflared --tail=20
```

I saw `Registered tunnel connection` messages with `protocol=quic`, confirming the tunnel was live.

## Step 4: Deploy a Test App

I deployed the `traefik/whoami` image as a simple test:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
    name: whoami
    namespace: default
spec:
    replicas: 1
    selector:
        matchLabels:
            app: whoami
    template:
        metadata:
            labels:
                app: whoami
        spec:
            containers:
                - name: whoami
                  image: traefik/whoami:latest
                  ports:
                      - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
    name: whoami
    namespace: default
spec:
    selector:
        app: whoami
    ports:
        - port: 80
          targetPort: 80
```

Also added the Ingress rule telling Traefik how to route based on the Host header:

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
    name: whoami-ingress
    namespace: default
spec:
    rules:
        - host: whoami.demolamalomo.xyz
          http:
              paths:
                  - path: /
                    pathType: Prefix
                    backend:
                        service:
                            name: whoami
                            port:
                                number: 80
```

**The 502 Problem**
At this point, I expected everything to work with the DNS resolved. The tunnel was connected. But when I hit `https://whoami.demolamalomo.xyz`, I got a **502 Bad Gateway** from Cloudflare.

This was the most confusing problem because:

- The cloudflared pods were healthy and connected
- The tunnel logs showed no request-level errors
- From a debug pod inside the cluster, curling `http://traefik.kube-system.svc.cluster.local:80` with the correct Host header returned a perfect 200 response

The issue was **DNS resolution inside the cloudflared container**. My tunnel config originally pointed to:

```bash
http://traefik.kube-system.svc.cluster.local:80
```

The cloudflared Docker image is built on scratch, an essentially empty container with nothing but the cloudflared binary, with no `shell`, `cat` , `curl` , or `/etc/resolv.conf`. When cloudflared tried to resolve `traefik.kube-system.svc.cluster.local`, it had no idea which DNS server to ask and failed silently. The request never reached Traefik, so Cloudflare returned a `502`.

I confirmed this by switching the tunnel config to use **Traefik's ClusterIP directly**:

```bash
http://10.43.250.137:80
```

Immediately, I ran curl `https://whoami.clohea.net` , and it returned the whoami response with Cloudflare headers confirming traffic was flowing through the tunnel.

**Why the ClusterIP Approach Is Fine**
In a homelab, the Traefik service ClusterIP is stable. It only changes if you delete and recreate the service. For a more robust solution, you could add explicit DNS config to the cloudflared deployment:

```yaml
spec:
    template:
        spec:
            dnsPolicy: 'ClusterFirst'
            dnsConfig:
                nameservers:
                    - 10.43.0.10 # kube-dns ClusterIP
                searches:
                    - kube-system.svc.cluster.local
                    - svc.cluster.local
                    - cluster.local
```

But for a homelab, the ClusterIP approach is simpler and works.

**Adding New Apps**
The best part about this setup is that adding a new app requires zero changes to Cloudflare. The wildcard `*.demolamalomo.xyz` already covers every subdomain. I just need to:

- A Deployment and Service for my app
- An Ingress with the right hostname

The architecture looks like this:

![](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774133078724_image.png)

## Lessons Learned

- **Route 53 has two places for nameservers.** The hosted zone NS record is not the same as the registered domain's nameserver setting. Change the wrong one and nothing happens. Change it under **Registered domains** >> **Name servers**.
- **The Cloudflared Docker image cannot resolve Kubernetes DNS names.** It is built on a scratch base image with no `/etc/resolv.conf`. Either use the target service's ClusterIP directly or add explicit `dnsConfig` to the pod spec.
- **Ingress host names must match exactly.** If your tunnel routes to `*.clohea.net` but your Ingress says `whoami.k3s.clohea.net`, Traefik will not match the request and you will get a 502.


I wanted production-style ingress for my k3s homelab without exposing ports or my public IP. What looked simple turned into debugging Route 53 nameservers, Kubernetes DNS resolution, and mysterious Cloudflare 502s before everything finally worked.

How I Exposed My Kubernetes Homelab to the Internet Using Cloudflare Tunnel

I recently set up Kubernetes in my homelab to build real operational experience, not just theoretical knowledge. My goal was to run something that behaves as close as possible to a production environment, similar to what you’d get with [Amazon Elastic Kubernetes Service](https://aws.amazon.com/eks/), while still keeping things lightweight and fully under my control.

To make that happen, I deployed a Kubernetes cluster using [k3s](https://k3s.io/) on an Ubuntu virtual machine hosted inside [Proxmox VE](https://www.proxmox.com/en/products/proxmox-virtual-environment/overview).

This article walks through exactly how I did it, why each step matters, and what the setup gives me moving forward.

## My Starting Point: Virtualized Infrastructure

Before installing Kubernetes, I already had a virtualization environment running on Proxmox. Inside it, I created a virtual machine running Ubuntu. That VM would serve as my Kubernetes control plane node.

Using virtualization mirrors how real production environments operate. Cloud providers run Kubernetes on virtual infrastructure, not bare laptops. My Proxmox VM plays the same role as a cloud compute instance, giving me isolation, resource control, and the ability to snapshot or rebuild when needed.

## Why I Chose k3s

I wanted a Kubernetes distribution that felt real but did not demand heavy hardware or complex setup.

k3s was the perfect fit because:

- It is fully Kubernetes compliant
- It runs with minimal resource usage
- Installation is extremely fast
- It removes operational overhead while preserving core behavior

In practice, it provides a learning environment that behaves like a production cluster without the operational overhead of a full upstream Kubernetes deployment.

## Phase 1: Preparing the Ubuntu Server

Before installing Kubernetes, the operating system had to be configured correctly. Kubernetes is sensitive to host configuration, especially networking and memory management.

**Static IP Configuration**

During Ubuntu installation, I manually assigned a static IP address for:

- Subnet
- Address
- Gateway
- DNS

This IP becomes the Kubernetes API endpoint. If it changes, cluster access breaks. Production clusters rely on stable control plane endpoints, so I treated my homelab the same way.

**Initial Server Setup**

After installation, I connected via SSH and updated the system packages. I installed common administrative tools such as curl, git, vim, htop, and networking utilities. These support everyday server operations like monitoring, editing configs, and downloading resources.

```bash
sudo apt update && sudo apt upgrade -y
sudo apt install -y curl wget git vim htop net-tools
```

**Proxmox Integration with QEMU Guest Agent**

I installed the QEMU guest agent to improve communication between the VM and Proxmox. This allows the hypervisor to detect IP addresses, perform graceful shutdowns, and monitor the VM’s state more reliably.

```bash
sudo apt install -y qemu-guest-agent
sudo systemctl enable qemu-guest-agent
sudo systemctl start qemu-guest-agent
```

**Network Validation**

Before going further, I verified:

- The static IP was assigned correctly
- External network access worked
- DNS resolution worked

Kubernetes depends heavily on networking. If connectivity is unstable, cluster components fail in unpredictable ways.

```bash
ip addr show ens18
# Should show your static IP
```

**Disabling Swap**

Kubernetes requires swap to be disabled because it interferes with resource scheduling and memory guarantees. This is a standard requirement across production clusters, so I disabled swap and made the change persistent.

```bash
sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstab

# Verify
free -h
# Swap should show 0
```

**Hostname Configuration**

I set the server hostname and mapped it in `/etc/hosts` so the system could resolve itself consistently. Stable node identity is important for certificates, networking, and cluster communication.

```bash
sudo hostnamectl set-hostname k3s-demo.local
```

Then modify `/etc/hosts` file to map the hostname:

```bash
# Update /etc/hosts
sudo vim /etc/hosts

# Add this line
<SERVER IP> k3s-demo.local k3s-demo
```

## Phase 2: Installing k3s

With the system prepared, I moved on to installing Kubernetes.

**Pre-Installation Checks**

Before running the installer, I confirmed:

- The required ports were free
- Enough RAM and CPU were available
- Sufficient disk space existed

This avoids conflicts and failed cluster startup.

```bash
# Check available ports
sudo ss -tulpn | grep -E ':(6443|10250)'

# Should return nothing (ports free)

# Check system resources
free -h  # Should show number of RAM
nproc    # Should show number of cores
df -h    # Should show plenty of disk space
```

**Installing k3s**

I installed k3s using the official script with three key options:

```bash
curl -sfL https://get.k3s.io | sh -s - \
  --write-kubeconfig-mode 644 \
  --disable traefik \
  --node-name k3s-demo
```

- `--write-kubeconfig-mode 644`: Makes kubectl work without sudo
- `--disable traefik`: Default Traefik ingress disabled so I can manage ingress myself later
- `--node-name`: Explicit node name for clarity and future scaling

The installation completed in under a minute.

**Verifying Cluster Health**

After installation, I confirmed:

- The K3s service was running
- The node reported Ready
- System pods in the kube-system namespace were healthy

If anything fails, system logs provide detailed troubleshooting information.

```bash
# Check k3s service
sudo systemctl status k3s

# Should see: active (running)

# Check node
kubectl get nodes

# Expected output:
# NAME          STATUS   ROLES                  AGE   VERSION
# k3s-demo    Ready    control-plane,master   1m    v1.28.x

# Check all system pods
kubectl get pods -A

# Should see pods in kube-system namespace running
```

![Running kubernetes](https://paper-attachments.dropboxusercontent.com/s_F3D46302D17DF57EA141A194AFED6667E114BCD5CC881287885039BD0D55EE8B_1772073514109_Image+26-02-2026+at+03.38.png)

## Does k3s Have Only a Control Plane? What About Worker Nodes?

When I ran `kubectl get nodes` to see the nodes in my Kubernetes cluster, I only saw a control plane node.

That worried me a bit because from what I understood about Kubernetes, the control plane, which acts as the brain, is separate from worker nodes, where applications actually run.

I expected my setup to look something like this:

```bash
NAME              STATUS   ROLES           AGE   VERSION
control-plane-1   Ready    control-plane   1h    v1.28.x
worker-1          Ready    <none>          1h    v1.28.x
worker-2          Ready    <none>          1h    v1.28.x
worker-3          Ready    <none>          1h    v1.28.x
```

After digging deeper into k3s architecture, I discovered this behavior is intentional. In k3s, a single provisioned node acts as both the control plane and the worker node. It manages the cluster and runs workloads at the same time.

This design is perfect for development, testing, and pre-production environments, which is exactly what most homelabs are built for.

Still, that setup was not enough for me. I wanted the real experience of managing a cluster with a separate control plane and worker nodes. Thankfully, k3s supports adding worker nodes, and that is exactly what I did.

**How to Add a Worker Node to k3s**

To add a worker node, the first step is to retrieve the control plane token. This token is required to join a new node to the cluster.

On the control plane VM, run:

```bash
sudo cat /var/lib/rancher/k3s/server/node-token
```

The output will look something like this:

```bash
K10abc123def456g.....
```

Next, I provisioned another VM, just like I did for the control plane. I disabled swap, installed the QEMU agent, and configured the necessary dependencies to prepare the VM for Kubernetes.

After that, I installed k3s in agent mode by running the following command on the new VM:

```bash
curl -sfL https://get.k3s.io | K3S_URL=https://<CONTROL_PLANE_IP> \
  K3S_TOKEN="K10abc123def456g......" \
  sh -
```

Here:

- `K3S_URL` is the control plane IP address
- `K3S_TOKEN` is the token retrieved earlier

Finally, I verified that the worker node successfully joined the cluster by running the following command on the control plane VM:

```bash
kubectl get nodes
```

![k3s with control plane and worker nodes](https://paper-attachments.dropboxusercontent.com/s_F3D46302D17DF57EA141A194AFED6667E114BCD5CC881287885039BD0D55EE8B_1772630050547_Image+04-03-2026+at+14.13.png)

## Final Thoughts

Setting up Kubernetes yourself changes how you understand it. Instead of interacting with a managed black box, you see every layer that makes the platform work.

Running k3s on a Proxmox-hosted Ubuntu server gives me a controlled, production-like environment where I can experiment, break things, and learn deeply without risk.

For anyone serious about mastering Kubernetes, building your own cluster is one of the most valuable steps you can take.


I built a Kubernetes cluster in my homelab using k3s on an Ubuntu VM running inside Proxmox. This guide walks through the full setup process, from server preparation to adding worker nodes.

#kubernetes

Why One Node Pulled and the Other Didn't: Debugging an ECR Image Pull Failure in Kubernetes Homelab

How I Installed Portainer on My Kubernetes Homelab (Running on Proxmox)

How I Exposed My Kubernetes Homelab to the Internet Using Cloudflare Tunnel

How I Installed Kubernetes on My Homelab (Using k3s on Proxmox + Ubuntu)