AI can write your Terraform. It can investigate your incidents. But should you let it touch production? As AI moves deeper into infrastructure, the hardest problem may not be making agents smarter, but making them safe to trust.

AI Has Reached Infrastructure. It Hasn't Reached Autonomy Yet.

I've been running a homelab Kubernetes cluster for a while now. It’s a simple two-node k3s setup with Traefik handling ingress and a Cloudflare Tunnel for external access. It’s been a great space to experiment and learn.

But deployment was still a pain.

Recently, I had a frontend app sitting in a GitLab repo, and every time I wanted to deploy, it turned into a small ritual. SSH into the server, pull the latest image, restart containers, and hope everything still worked. It got the job done, but it wasn’t something I trusted or enjoyed.

This post walks through how I moved from that setup to a full GitOps workflow using GitLab CI and ArgoCD. No prior ArgoCD experience, just figuring things out step by step until a simple `git push` became enough to deploy.

## The Starting Point

Going into this, my setup was pretty straightforward. I had a k3s cluster running on two Ubuntu nodes: one control plane and one worker, both on my local network. Traefik came bundled with k3s, so ingress was already handled. For external access, I was using a Cloudflare Tunnel running via `cloudflared` pods in a dedicated namespace, routing traffic from `*.demolamalomo.xyz` into the cluster.

The app itself was a frontend project. A Node.js app that builds into static files and gets served with nginx. The code lived in GitLab, and I was already using GitLab’s container registry to store images from earlier CI builds.

The CI pipeline, however, was doing too much. It built the image, then SSH-ed into the server to stop containers, remove them, pull new images, and restart everything. It worked most of the time, but it was fragile, hard to debug, and rolling back meant doing everything manually again.

## The Options I Considered

Throughout my deployment journey, I explored a couple of options before arriving at ArgoCD.

1. **Manual build and deploy** was basically what I was already doing. Build locally, push the image, run `kubectl apply`. The issue with this is that there’s no automation, no audit trail, and plenty of room for error.
2. **GitLab CI/CD handling everything** was the next step. The pipeline builds the image, pushes it, and deploys directly to the cluster using `kubectl`. This is where a lot of teams start, and honestly, it worked reasonably well for me too. But the CI was doing a lot of work, and it can get really messy with a lot of echoing and sshing commands. For context, below is a section of one of my CI build:

```bash
- echo "$ID_RSA_DEV_HOMELAB" > id_rsa_homelab
    - chmod 600 id_rsa_homelab
    - apk update && apk add openssh-client
    - echo "stop existing docker container & remove images"
    - echo "Port $STAGE_SVR_PORT_HOMELAB"
    - echo "User $DEV_SERVER_USER_HOMELAB"
    - echo "Host $STAGE_SVR_IP_HOMELAB"
    - echo "VM $STAGE_VM_IP_HOMELAB"
    - echo "Service $SERVICE_NAME_HOMELAB"
    - ssh -i id_rsa_homelab -p $STAGE_SVR_PORT_HOMELAB -o StrictHostKeyChecking=no $DEV_SERVER_USER_HOMELAB@$DEV_SVR_IP_HOMELAB "docker stop $SERVICE_NAME_HOMELAB || true"
......
```

3. **GitOps with ArgoCD** takes a different approach. CI still builds and pushes images, but it never talks to the cluster directly. Instead, it updates a Git repo that contains the Kubernetes manifests. ArgoCD, running inside the cluster, watches that repo and syncs any changes.The thing I liked most was that I could look at a repository and immediately know what should be running in the cluster.

That was the point where ArgoCD clicked for me. I liked the fact that I could open a repo and know exactly what was deployed. Rollbacks also become much simpler, by just reverting a commit. So I went with **ArgoCD**.

## Installing ArgoCD

I’d never used ArgoCD before, so this part was new. Thankfully, installation was straightforward:

```bash
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
```

After a minute or so, everything was running, and I could start exploring the dashboard.

To access the dashboard, I started with a simple port-forward:

```bash
kubectl port-forward svc/argocd-server -n argocd 8080:443
```

The default login is `admin`, and the initial password is stored in a Kubernetes secret:

```bash
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
```

I also installed the ArgoCD CLI, which made it easier to create applications and debug sync issues without constantly switching to the browser.

Later on, I exposed ArgoCD through Traefik and my Cloudflare Tunnel so I could access it remotely, but the port-forward was more than enough to get started.

![ArgoCD UI](https://paper-attachments.dropboxusercontent.com/s_2A415A66514362A902F36E27BA0662B25C280357D4A31FB4FA73DA4B634FE1E6_1775396935665_Screenshot+2026-04-05+at+14.48.16.png)

## Understanding the Two-Repo Pattern

One thing that took me a while to wrap my head around was the idea of maintaining two repositories. At first it felt unnecessary. Why split things up when everything belongs to the same application?

But after working with it for a bit, the separation started to make sense.

The **application repo** is where development happens. Code changes, Dockerfile, and CI configuration (`.gitlab-ci.yml`).

The **GitOps repo** is different. Its only job is to describe what's running in the cluster. Deployments, Services, and Ingress definitions. ArgoCD watches this repo and syncs the cluster whenever something changes.

The benefit became obvious pretty quickly. My application repo changes all the time with feature work, bug fixes, and random experiments, while the GitOps repo changes much less often because it only reflects what should be running in the cluster.

![ArgoCD workflow](https://paper-attachments.dropboxusercontent.com/s_2A415A66514362A902F36E27BA0662B25C280357D4A31FB4FA73DA4B634FE1E6_1775397646668_mermaid-diagram.png)

My GitOps repo (`homelab-gitops`) ended up looking like this:

```bash
homelab-gitops/
  clohea-admin/   # my application folder
    deployment.yaml
    service.yaml
    ingress.yaml
```

## Setting up the GitLab CI Pipeline

My existing `.gitlab-ci.yml` was doing too much. The new version only needed to do two things: build the image and update the GitOps repo.

The build stage pushes two tags:

- `:latest` for general use
- a commit hash tag like `:a1b2c3d`, which is what goes into the manifests

That commit-specific tag is important because it gives ArgoCD a concrete change to detect.

```yaml
docker-build-homelab-dev:
    image: docker:latest
    stage: build
    services:
        - docker:dind
    before_script:
        - echo $CI_JOB_TOKEN | docker login -u "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY
    script:
        - echo "$DEV_ENV" > $CI_PROJECT_DIR/.env
        - docker build -f ./Dockerfile --pull -t "$CI_REGISTRY_IMAGE:latest" -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA" .
        - docker push "$CI_REGISTRY_IMAGE:latest"
        - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA"
    only:
        - homelab-dev
```

The deploy stage updates the GitOps repo:

```yaml
update-manifest:
    stage: deploy
    image: alpine:latest
    before_script:
        - apk add --no-cache git sed
        - git config --global user.email "ci@gitlab.com"
        - git config --global user.name "GitLab CI"
    script:
        - git clone https://deploy-token:${GITOPS_TOKEN}@gitlab.com/clohea-fe/homelab-gitops.git
        - cd homelab-gitops
        - 'sed -i "s|image: .*|image: ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA}|" clohea-admin/deployment.yaml'
        - git add .
        - git diff --cached --quiet && echo "No changes" || git commit -m "Update frontend image to $CI_COMMIT_SHORT_SHA" && git push
    only:
        - homelab-dev
```

One thing that tripped me up: the `sed` command needs to use double quotes so that the shell expands the CI variables. My first attempt used single quotes, which meant `$CI_REGISTRY_IMAGE` was treated as literal text. The `sed` command found nothing to replace, `git commit` found nothing to commit, and the pipeline failed with `exit code 1`. A small thing, but the kind of mistake that costs you 30 minutes of staring at logs.

**A Note on the GITOPS_TOKEN**
The `update-manifest` job needs write access to the GitOps repo. I created a **Project Access Token** in the GitOps repo (Settings > Access Tokens) with `write_repository` scope, then stored it as a CI/CD variable called `GITOPS_TOKEN` in the app repo (Settings > CI/CD > Variables), with the "Mask variable" option checked so it doesn't leak into logs.

I could also have used a Personal Access Token, which would work across all my repositories, but a Project Access Token scoped to just the GitOps repo is cleaner from a security standpoint.

## The Kubernetes Manifests

The GitOps repo itself stayed pretty simple. For this application, I only needed three manifests:

- A Deployment
- A Service
- An Ingress

Nothing fancy. Just enough to get the application running and exposed through Traefik.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
    name: webapp-service
    namespace: frontend
spec:
    replicas: 2
    selector:
        matchLabels:
            app: webapp-service
    template:
        metadata:
            labels:
                app: webapp-service
        spec:
            imagePullSecrets:
                - name: gitlab-registry-cred
            containers:
                - name: webapp-service
                  image: registry.gitlab.com/clohea-fe/clohea-admin/webapp-service:INITIAL_TAG
                  ports:
                      - containerPort: 3000
```

Since the registry is private, I created an image pull secret:

```bash
kubectl create secret docker-registry gitlab-registry-cred \
  --namespace frontend \
  --docker-server=registry.gitlab.com \
  --docker-username=<deploy-token-username> \
  --docker-password=<deploy-token-value>
```

The service routes traffic internally:

```yaml
apiVersion: v1
kind: Service
metadata:
    name: webapp-service
    namespace: frontend
spec:
    selector:
        app: webapp-service
    ports:
        - port: 80
          targetPort: 3000
```

And the ingress exposes it via Traefik:

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
    name: webapp-service
    namespace: frontend
spec:
    ingressClassName: traefik
    rules:
        - host: admin-portal.demolamalomo.xyz
          http:
              paths:
                  - path: /
                    pathType: Prefix
                    backend:
                        service:
                            name: webapp-service
                            port:
                                number: 80
```

## Connecting ArgoCD to the GitOps Repo

For ArgoCD to watch the GitOps repo, it needs credentials since it's a private GitLab repo. I created a deploy token in the GitOps repo with `read_repository` scope, then added the repo to ArgoCD using the UI.

**Settings > Repositories > + CONNECT REPO**

![Add repository to ArgoCD](https://paper-attachments.dropboxusercontent.com/s_2A415A66514362A902F36E27BA0662B25C280357D4A31FB4FA73DA4B634FE1E6_1775418533445_Screenshot+2026-04-05+at+20.48.03.png)

Another option to do this is to use the CLI:

```bash
argocd repo add https://gitlab.com/clohea-fe/homelab-gitops.git \
  --username <deploy-token-username> \
  --password <deploy-token-value>
```

Then I created the application by entering the repository URL, the path to the application folder in the GitOps repo, and a few other required settings.

**Application > + NEW APP**

![New App](https://paper-attachments.dropboxusercontent.com/s_2A415A66514362A902F36E27BA0662B25C280357D4A31FB4FA73DA4B634FE1E6_1775418812700_Screenshot+2026-04-05+at+20.51.21.png)

I can also use the CLI by running the command below

```bash
argocd app create frontend \
  --repo https://gitlab.com/clohea-fe/homelab-gitops.git \
  --path clohea-admin \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace frontend \
  --sync-policy automated \
  --auto-prune \
  --self-heal
```

Quick breakdown of the flags:

- `--sync-policy automated`: syncs on every commit
- `--auto-prune`: removes deleted resources
- `--self-heal`: reverts manual cluster changes

## The Debugging Marathon

If I'm being honest, the setup didn't just work on the first try. Here's a condensed version of the issues I hit, because I think they're genuinely useful for anyone doing this for the first time.

**Image pull errors.** The first deployment failed with a `403 Forbidden` because Kubernetes couldn't authenticate with GitLab's private container registry. The fix was creating the `docker-registry` secret and adding `imagePullSecrets` to the deployment manifest. Then it failed again with `not found` — the image tag referenced in the manifest didn't exist in the registry because my build job hadn't pushed a commit-hash tag yet.

**Name mismatches.** My ingress was referencing a service called `frontend`, but the actual service was named `webapp-service`. A simple typo that resulted in a 502 from Traefik. `kubectl describe ingress` made this obvious once I thought to look.

**Port mismatch.** This one was subtle. The deployment had `containerPort: 80` and the service was targeting port 80, but nginx inside the container was actually listening on port 3000. The Docker image had a custom nginx config that bound to 3000 instead of the default 80. Running `kubectl exec` and checking `netstat -tlnp` inside the pod revealed the mismatch immediately.

## The End Result

Once everything was wired up and the bugs were resolved, the flow became exactly what I wanted:

- I push code to the `homelab-dev` branch.
- GitLab CI builds the Docker image and pushes it to the registry with both a `:latest` and a commit-hash tag.
- The `update-manifest` job clones the GitOps repo, updates the image tag in the deployment manifest, and pushes.
- ArgoCD detects the new commit within seconds.
- ArgoCD syncs the manifests to the cluster, Kubernetes pulls the new image, and the app is live.

The ArgoCD dashboard shows the full picture. Sync status, pod health, the Git commit that triggered the deployment. If something goes wrong, I can see exactly which commit caused it and revert.

![ArgoCD Dashboard](https://paper-attachments.dropboxusercontent.com/s_2A415A66514362A902F36E27BA0662B25C280357D4A31FB4FA73DA4B634FE1E6_1775420456749_Screenshot+2026-04-05+at+21.19.00.png)

The entire deployment chain is now traceable through Git commits. No more SSH-ing into servers or chained server commands. Just `git push`, and everything takes care of itself.


I got tired of SSH-ing into servers every time I needed to deploy a simple app. So I spent an evening setting up GitLab CI, ArgoCD, and a GitOps workflow in my homelab. Here's what worked, what broke, and what I learned along the way.

From Manual Deploys to GitOps: How I Automated my Homelab with ArgoCD and GitLab CI

I run k3s in my [homelab](https://fullstackwriter.dev/post/how-i-installed-kubernetes-on-my-homelab-using-k3s-on-proxmox-ubuntu?category=kubernetes), and I wanted to expose my apps to the internet without opening multiple ports or configuring complex router rules.

I had the options of NodePort or installing MetalLB. Although both approaches work, neither felt close to how things are done in production. NodePort would mean exposing a port on my router and mapping it to a node. That exposes my public IP and opens inbound traffic into my network. MetalLB is cleaner inside Kubernetes, but it still assumes you have public IP handling and proper routing at the edge.

What I wanted instead was something closer to how modern infrastructure works, that supports encrypted transport, no inbound ports, and clean DNS control.

That is why I chose Cloudflare Tunnel. The goal for me was to use my domain `demolamalomo.xyz` registered on AWS Route 53. I wanted something very close to production that caters to the following:

- Apps accessible at `*.demolamalomo.xyz` (e.g., `whoami.demolamalomo.xyz`, `grafana.demolamalomo.xyz`)
- No router port forwarding
- No public IP exposure
- Cloudflared running as Kubernetes pods, not installed on the host

## Step 1: Move the Domain to Cloudflare DNS

My first instinct was to delegate only a subdomain like `k3s.demolamalomo.xyz` to Cloudflare using NS records in Route 53. That way my root domain stays fully in Route 53 and only the Kubernetes subdomain gets managed by Cloudflare.

That didn’t work. Cloudflare's free plan does not support adding subdomain zones. You can only add root domains like `demolamalomo.yxz`. On testing, this didn’t work because anytime I dig through the nameserver, the nameservers returned are still the ones tied to Route 53.

So I moved the entire domain's DNS to Cloudflare by following the steps below:

- In the Cloudflare Dashboard, I clicked the + **Add** button, selected the **Connect a domain** option, and filled in the required field.

![Connect a domain](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774118683934_Screenshot+2026-03-04+at+01.27.15.png)
![Fill the form](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774118683991_Screenshot+2026-03-04+at+01.38.39.png)

- Selected the **Free** plan
- Cloudflare scanned and imported my existing Route 53 DNS records automatically
- Cloudflare assigned me two nameservers similar to `****.**.cloudflare.com` and `******.**.cloudflare.com`

Then I updated Route 53 to point to those nameservers.

**The First Mistake: Changing Nameservers in the Wrong Place**
Route 53 has two places where nameservers appear, and I changed the wrong one.

**Wrong place:** Route 53 >> Hosted zones >> demolamalomo.xyz >> NS record. This is just a DNS record inside the zone. Changing it does nothing to where the internet actually looks for your domain.
**Right place:** Route 53 >> **Registered domains** >> demolamalomo.xyz >> **Name servers**. This is what tells the TLD registry where to send DNS queries for your domain.

I spent time debugging why `dig clohea.net NS @1.1.1.1` still returned AWS nameservers before realizing I had changed the wrong thing. Once I updated the registered domain's nameservers, Cloudflare showed the domain as **Active**, and DNS started resolving correctly.

## Step 2: Create the Cloudflare Tunnel

In the Cloudflare Zero Trust [dashboard](https://one.dash.cloudflare.com), I created a tunnel following the steps below:

- Navigate to **Networks >> Connectors**
- Click **Create a tunnel**
- Select **Cloudflared** as the connector type

![Select Cloudflared](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774128112097_Screenshot+2026-03-21+at+21.39.42.png)

- Named it `k3s-tunnel`
- On the connector install page, copied the **tunnel token** starting with `eyJ...`
- Then, I installed Cloudflared on my machine with the token copied.
  sudo cloudflared service install eyJ......

Once Cloudflared is installed, the status shows `HEALTHY` in my dashboard.

Next, I proceeded to the next page to configure the subdomain and service (Traefik in my case)

![Configured subdomain](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774128103478_Screenshot+2026-03-21+at+22.21.07.png)

## Step 3: Deploy Cloudflared on Kubernetes

I chose to run **cloudflared** as Kubernetes pods rather than installing it on the host machine. This keeps everything inside the cluster and makes it easy to manage with standard Kubernetes tooling.

To do this, first, I created a namespace and stored the tunnel token I got earlier as a secret:

```bash
kubectl create namespace cloudflare

kubectl create secret generic cloudflared-token \
  --namespace cloudflare \
  --from-literal=token=<TUNNEL_TOKEN>
```

Then, I used the secret to create a deployment:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
    name: cloudflared
    namespace: cloudflare
spec:
    replicas: 2
    selector:
        matchLabels:
            app: cloudflared
    template:
        metadata:
            labels:
                app: cloudflared
        spec:
            containers:
                - name: cloudflared
                  image: cloudflare/cloudflared:latest
                  args:
                      - tunnel
                      - --no-autoupdate
                      - run
                  env:
                      - name: TUNNEL_TOKEN
                        valueFrom:
                            secretKeyRef:
                                name: cloudflared-token
                                key: token
                  resources:
                      requests:
                          memory: '128Mi'
                          cpu: '100m'
                      limits:
                          memory: '256Mi'
                          cpu: '200m'
```

This creates two replicas of the cloudflared instance, and the `TUNNEL_TOKEN` environment variable tells cloudflared to run in remotely-managed mode (the tunnel configuration lives in Cloudflare's dashboard, not in a local config file).

After applying, I verified the pods were connected:

```bash
kubectl logs -n cloudflare -l app=cloudflared --tail=20
```

I saw `Registered tunnel connection` messages with `protocol=quic`, confirming the tunnel was live.

## Step 4: Deploy a Test App

I deployed the `traefik/whoami` image as a simple test:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
    name: whoami
    namespace: default
spec:
    replicas: 1
    selector:
        matchLabels:
            app: whoami
    template:
        metadata:
            labels:
                app: whoami
        spec:
            containers:
                - name: whoami
                  image: traefik/whoami:latest
                  ports:
                      - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
    name: whoami
    namespace: default
spec:
    selector:
        app: whoami
    ports:
        - port: 80
          targetPort: 80
```

Also added the Ingress rule telling Traefik how to route based on the Host header:

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
    name: whoami-ingress
    namespace: default
spec:
    rules:
        - host: whoami.demolamalomo.xyz
          http:
              paths:
                  - path: /
                    pathType: Prefix
                    backend:
                        service:
                            name: whoami
                            port:
                                number: 80
```

**The 502 Problem**
At this point, I expected everything to work with the DNS resolved. The tunnel was connected. But when I hit `https://whoami.demolamalomo.xyz`, I got a **502 Bad Gateway** from Cloudflare.

This was the most confusing problem because:

- The cloudflared pods were healthy and connected
- The tunnel logs showed no request-level errors
- From a debug pod inside the cluster, curling `http://traefik.kube-system.svc.cluster.local:80` with the correct Host header returned a perfect 200 response

The issue was **DNS resolution inside the cloudflared container**. My tunnel config originally pointed to:

```bash
http://traefik.kube-system.svc.cluster.local:80
```

The cloudflared Docker image is built on scratch, an essentially empty container with nothing but the cloudflared binary, with no `shell`, `cat` , `curl` , or `/etc/resolv.conf`. When cloudflared tried to resolve `traefik.kube-system.svc.cluster.local`, it had no idea which DNS server to ask and failed silently. The request never reached Traefik, so Cloudflare returned a `502`.

I confirmed this by switching the tunnel config to use **Traefik's ClusterIP directly**:

```bash
http://10.43.250.137:80
```

Immediately, I ran curl `https://whoami.clohea.net` , and it returned the whoami response with Cloudflare headers confirming traffic was flowing through the tunnel.

**Why the ClusterIP Approach Is Fine**
In a homelab, the Traefik service ClusterIP is stable. It only changes if you delete and recreate the service. For a more robust solution, you could add explicit DNS config to the cloudflared deployment:

```yaml
spec:
    template:
        spec:
            dnsPolicy: 'ClusterFirst'
            dnsConfig:
                nameservers:
                    - 10.43.0.10 # kube-dns ClusterIP
                searches:
                    - kube-system.svc.cluster.local
                    - svc.cluster.local
                    - cluster.local
```

But for a homelab, the ClusterIP approach is simpler and works.

**Adding New Apps**
The best part about this setup is that adding a new app requires zero changes to Cloudflare. The wildcard `*.demolamalomo.xyz` already covers every subdomain. I just need to:

- A Deployment and Service for my app
- An Ingress with the right hostname

The architecture looks like this:

![](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774133078724_image.png)

## Lessons Learned

- **Route 53 has two places for nameservers.** The hosted zone NS record is not the same as the registered domain's nameserver setting. Change the wrong one and nothing happens. Change it under **Registered domains** >> **Name servers**.
- **The Cloudflared Docker image cannot resolve Kubernetes DNS names.** It is built on a scratch base image with no `/etc/resolv.conf`. Either use the target service's ClusterIP directly or add explicit `dnsConfig` to the pod spec.
- **Ingress host names must match exactly.** If your tunnel routes to `*.clohea.net` but your Ingress says `whoami.k3s.clohea.net`, Traefik will not match the request and you will get a 502.


I wanted production-style ingress for my k3s homelab without exposing ports or my public IP. What looked simple turned into debugging Route 53 nameservers, Kubernetes DNS resolution, and mysterious Cloudflare 502s before everything finally worked.

How I Exposed My Kubernetes Homelab to the Internet Using Cloudflare Tunnel

I recently ran into an interesting issue in my cluster. One node could pull container images from AWS ECR without any problems, while another node refused to cooperate. Kubernetes kept throwing authentication errors even though everything looked correctly configured.

In this post, I will walk through:

- The problem I encountered
- The debugging process
- A few possible solutions
- The approach I eventually settled on

If you are running Kubernetes in a homelab and pulling images from Amazon Elastic Container Registry [(ECR)](https://aws.amazon.com/ecr/), this might save you some troubleshooting time.

## My Homelab Setup

My Kubernetes cluster runs on [k3s](https://fullstackwriter.dev/post/how-i-installed-kubernetes-on-my-homelab-using-k3s-on-proxmox-ubuntu?category=kubernetes) because it is lightweight and easy to operate in a small environment.

The cluster currently has two nodes:

| **Node**        | **Role**      |
| --------------- | ------------- |
| control-plane-1 | Control Plane |
| k3s-worker-1    | Worker        |

My application images are stored in AWS ECR and deployed to the cluster using a Kubernetes Deployment.

The deployment uses this image:

```bash
  <SERVICE_ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com/webappservice:webappservice-latest
```

## The Problem

I deployed the application with **two replicas**, expecting Kubernetes to schedule one pod on each node. That part worked, but the pods behaved differently depending on where they landed.

| **Node**      | **Pod Status**   |
| ------------- | ---------------- |
| Worker Node   | Running          |
| Control Plane | ImagePullBackOff |

The control plane pod kept failing with this error:

```bash
Back-off pulling image
ErrImagePull: failed to pull and unpack image
no basic auth credentials
```

This suggested the node could not authenticate with ECR.

## First Assumption: Missing Authentication

My first thought was that Kubernetes did not have credentials to access the registry. To fix that, I created a pull secret using the AWS CLI.

```bash
kubectl create secret docker-registry ecr-secret \
  --docker-server=<SERVICE_ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com \
  --docker-username=AWS \
  --docker-password=$(aws ecr get-login-password --region eu-west-2) \
  --namespace dev
```

Then I updated the deployment to reference the secret.

```bash
imagePullSecrets:
  - name: ecr-secret
```

However, the control plane node still failed to pull the image.

## Verifying the Node was the Problem

To confirm whether this was a Kubernetes issue or a node-level issue, I SSH'd into the control plane node and attempted to pull the image directly using containerd.

```bash
sudo ctr image pull <SERVICE_ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com/webappservice:latest
```

I got this error:

```bash
pull access denied
authorization failed: no basic auth credentials
```

This confirmed the issue was node authentication. Interestingly, the worker node worked fine.

At this point, I realized the worker node likely already had the image cached locally, which explained why its pod started successfully.

## Attempt 2: Node-Level Authentication

The next idea was configuring authentication at the node level.

One approach is using the [ECR credential helper](https://github.com/awslabs/amazon-ecr-credential-helper), which automatically fetches tokens from AWS whenever an image pull happens.

However, this turned out to be unreliable in my setup because k3s uses **containerd**, and containerd does not automatically read Docker credential helper configurations. This made the approach more complicated than necessary for a homelab cluster.

## The Simpler and More Reliable Solution

Instead of configuring authentication per node or per deployment, I decided to attach the ECR pull secret to the **default ServiceAccount in the namespace**.

This means any pod created in that namespace automatically inherits the image pull secret. To do this, I followed the steps below:

**Step 1: Create the ECR Pull Secret**

First, I recreated the secret.

```bash
kubectl create secret docker-registry ecr-secret \
  --docker-server=<SERVICE_ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com \
  --docker-username=AWS \
  --docker-password=$(aws ecr get-login-password --region eu-west-2) \
  --namespace dev
```

**Step 2: Attach the Secret to the Namespace ServiceAccount**

Next, I patched the default ServiceAccount in the `dev` namespace.

```bash
kubectl patch serviceaccount default \
  -n dev \
  -p '{"imagePullSecrets": [{"name": "ecr-secret"}]}'
```

Now every pod created in the namespace automatically uses the secret.

**Step 3: Restart the Failing Pod**

I deleted the failing pod, so Kubernetes would recreate it.

```bash
kubectl delete pod -n dev webappservice-hudjfi
```

The deployment created a new pod, and this time it started successfully.

**Verifying the Fix**

Running `kubectl describe pod` confirmed the container started correctly. Key parts of the output looked like this:

```bash
State: Running
Ready: True
```

The event logs also showed:

```bash
Container image already present on machine
Created container
Started container
```

At this point, both nodes were running the application successfully.

## Why I Chose the ServiceAccount Approach

There are a few ways to solve ECR authentication in Kubernetes.

| **Approach**                    | **Pros**                | **Cons**                     |
| ------------------------------- | ----------------------- | ---------------------------- |
| Node credential helper          | Automatic token refresh | Harder to configure with k3s |
| Static registry token           | Simple                  | Tokens expire after 12 hours |
| imagePullSecrets per deployment | Works                   | Repetitive configuration     |
| ServiceAccount imagePullSecret  | Simple and reusable     | Requires secret refresh      |

The **ServiceAccount approach** felt like the best tradeoff for a homelab environment because:

- It centralizes authentication
- It avoids repeating the configuration in every deployment
- It works consistently across nodes

Despite the flexibility the ServiceAccount approach brings, it still has the same problem of 12-hour expiry tokens that come with ECR tokens. Thankfully, this can be managed using a Kubernetes CronJob that refreshes the token periodically.

## Automatically Refreshing the ECR Secret

As mentioned earlier, authentication tokens from ECR expire every 12 hours. This does not affect running pods, but it can break new deployments or pods scheduled on fresh nodes if the secret has expired.

A simple way to solve this in a homelab cluster is to periodically refresh the secret using a Kubernetes CronJob.

![](https://paper-attachments.dropboxusercontent.com/s_88EBE4FB662784CC10B48EE40A4F16D609D3D7F24069A2CADFF36D32EB9F9E94_1772996213878_image.png)

This keeps the cluster working without manual intervention.

**Step 1: Create a ServiceAccount for the Job**

First, I created a ServiceAccount that the CronJob can use.

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
    name: ecr-secret-refresher
    namespace: dev
```

**Step 2: Allow the Job to Manage Secrets**

The job needs permission to update secrets in the namespace.

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
    name: ecr-secret-manager
    namespace: dev
rules:
    - apiGroups: ['']
      resources: ['secrets']
      verbs: ['get', 'create', 'delete']
```

Then bind the role to the ServiceAccount.

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
    name: ecr-secret-manager-binding
    namespace: dev
subjects:
    - kind: ServiceAccount
      name: ecr-secret-refresher
roleRef:
    kind: Role
    name: ecr-secret-manager
    apiGroup: rbac.authorization.k8s.io
```

**Step 3: Create the CronJob**

Before setting up the CronJob, the job container needs AWS credentials to call `ecr get-login-password`. In a managed cloud environment, I would typically handle this with an IAM role, but in a homelab the simplest approach is storing my AWS credentials in a Kubernetes secret and injecting them as environment variables.

```bash
kubectl create secret generic aws-credentials \
  --from-literal=AWS_ACCESS_KEY_ID=<your-access-key-id> \
  --from-literal=AWS_SECRET_ACCESS_KEY=<your-secret-access-key> \
  --namespace dev
```

Then, I created the CronJob that refreshes the secret every 11 hours.

```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
    name: refresh-ecr-secret
    namespace: dev
spec:
    schedule: '0 */11 * * *'
    jobTemplate:
        spec:
            template:
                spec:
                    serviceAccountName: ecr-secret-refresher
                    restartPolicy: OnFailure
                    containers:
                        - name: refresh-secret
                          image: amazon/aws-cli
                          env:
                              - name: AWS_REGION
                                value: eu-west-2
                          command:
                              - /bin/sh
                              - -c
                              - |
                                  TOKEN=$(aws ecr get-login-password --region $AWS_REGION)
                                  kubectl delete secret ecr-secret -n dev --ignore-not-found
                                  kubectl create secret docker-registry ecr-secret \
                                    --docker-server=<ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com \
                                    --docker-username=AWS \
                                    --docker-password=$TOKEN \
                                    -n dev
```

This job will:

1. Request a fresh login token from ECR
2. Delete the old secret
3. Recreate the secret with the new token

Since the namespace ServiceAccount already references `ecr-secret`, all new pods automatically use the refreshed credentials.

**Applying the Configuration**

I saved the resources in a file and apply them:

```bash
kubectl apply -f cronjob.yaml
```

Then I verified the CronJob with:

```bash
kubectl get cronjobs -n dev
```

![Cronjob](https://paper-attachments.dropboxusercontent.com/s_88EBE4FB662784CC10B48EE40A4F16D609D3D7F24069A2CADFF36D32EB9F9E94_1772997819523_Screenshot+2026-03-08+at+20.23.11.png)

## Final Thoughts

The most misleading part of this issue was that one pod was already running. It made the problem look like an intermittent auth failure when the real explanation was simpler: the worker node had the image cached, so it never needed to authenticate. The control plane node had no cache, no credentials, and nowhere to go.

Once that clicked, the fix was straightforward. Attaching the ECR pull secret to the namespace ServiceAccount means every pod in that namespace inherits it automatically, and the CronJob keeps the credentials fresh without any manual intervention.

If you're running a k3s homelab with images in ECR, this setup is worth the 10 minutes it takes to configure. It is less fragile than managing secrets per deployment and less complex than node-level credential helpers.


How to troubleshoot and pull images from Amazon Elastic Container Registry (ECR) into your Kubernetes

Why One Node Pulled and the Other Didn't: Debugging an ECR Image Pull Failure in Kubernetes Homelab

Running Kubernetes in a homelab is fun until everything starts living in the terminal.

The CLI is powerful. Tools like `kubectl` give you full control over your cluster. But after a while, constantly running commands just to inspect workloads or check logs can slow things down.

I wanted something closer to what many teams use in production. A visual interface where I could see workloads, inspect containers, and manage resources quickly.

That led me to [Portainer](https://www.portainer.io/).

In this article, I walk through how I installed Portainer on my Kubernetes homelab running on **Proxmox Server Solutions**, the issues I ran into along the way, and how I fixed them.

## My Homelab Setup

My infrastructure currently looks like this:

```bash
Proxmox host
    └── Virtual machines
        └── Kubernetes cluster (k3s)
                └── Container workloads
```

The Kubernetes cluster runs inside VMs managed by Proxmox, and most of my applications run as containers inside the cluster.

Portainer runs inside [Kubernetes](https://fullstackwriter.dev/post/how-i-installed-kubernetes-on-my-homelab-using-k3s-on-proxmox-ubuntu?category=kubernetes) as well, which allows it to interact directly with the Kubernetes API and manage cluster resources.

## Why I Wanted Portainer

Even though the CLI works perfectly, I wanted a management layer that offers:

- A visual view of workloads
- Faster troubleshooting
- Easier inspection of pods and logs
- Cluster-wide visibility
- A user interface similar to production environments

The goal was not to replace the CLI. It was to complement it.

In real infrastructure environments, engineers rarely rely on commands alone. Tools that visualize cluster state make day-to-day operations much easier.

## Step 1. Install Helm (If Not Already Installed)

[Helm](https://helm.sh/) is a package manager for Kubernetes (similar to NPM for Node.js and Cargo for Rust). Portainer provides an official Helm chart, so Helm is required for installation.

To check if Helm is installed:

```bash
helm version
```

If not installed, [install Helm](https://helm.sh/docs/intro/install) based on your operating system.

## Step 2. Add the Portainer Helm Repository

To do this, I ran the command below:

```bash
helm repo add portainer https://portainer.github.io/k8s/
helm repo update
```

This makes the Portainer chart available locally.

## Step 3. First Installation Attempt (And the Error I Hit)

When I first tried installing Portainer, I got this error:

```bash
cluster reachability check failed:
kubernetes cluster unreachable:
Get "http://localhost:8080/version":
connect: connection refused
```

It means Helm could not find my Kubernetes cluster configuration. It defaulted to trying `localhost:8080`, which does not exist.

This happened because my cluster runs [k3s](https://k3s.io/), which stores its kubeconfig in:

```bash
/etc/rancher/k3s/k3s.yaml
```

`kubectl` was automatically using this file, but Helm was not. So Helm had no cluster credentials.

## Step 4. Fixing the Cluster Reachability Error

I explicitly exported the kubeconfig:

```bash
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
```

After that, Helm could communicate with Kubernetes. I verified by running the command below

```bash
helm list
```

To make this changes permanent, I ran:

```bash
echo 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml' >> ~/.bashrc
source ~/.bashrc
```

## Step 5. Install Portainer in Kubernetes

Once Helm could talk to the cluster, installation was straightforward. I did so by running the command below:

```bash
helm install portainer portainer/portainer \
  --namespace portainer \
  --create-namespace \
  --set service.type=NodePort
```

This deploys:

- Portainer server
- Kubernetes integration components
- A service for web access

## Step 6. Verify Deployment

Check pods:

```bash
kubectl get pods -n portainer
```

I waited until the status showed `Running`:

![Portainer running on Kubernetes](https://paper-attachments.dropboxusercontent.com/s_05AD378A82578B8F940350A334CC24A83B9D9779BBD50D7F99799A62C18025AC_1772907054520_Screenshot+2026-03-07+at+19.10.10.png)

## Step 7. Access the Portainer UI

To access the UI, first, I need to find the service on which Portainer is exposed:

```bash
kubectl get svc -n portainer
```

Then looked for the NodePort value.

![Portainer NodePort value](https://paper-attachments.dropboxusercontent.com/s_05AD378A82578B8F940350A334CC24A83B9D9779BBD50D7F99799A62C18025AC_1772907432360_Screenshot+2026-03-07+at+19.13.20.png)


Access in browser:

```bash
https://<node-ip>:<nodeport>
```

## Step 8. Second Issue I Encountered (Security Timeout)

When I opened the UI, I saw:

![Timeout error](https://paper-attachments.dropboxusercontent.com/s_05AD378A82578B8F940350A334CC24A83B9D9779BBD50D7F99799A62C18025AC_1772907578117_screencapture-10-90-1-30-30777-timeout-html-2026-02-25-03_41_32.png)


This happens if you do not create the admin account quickly after the first launch, which I didn’t in my case. It’s a security measure with Portainer. It locks itself after some time.

## Step 9. Fixing the Timeout

To fix it, I restarted the  Portainer deployment:

```bash
kubectl rollout restart deployment portainer -n portainer
```

Then waited for the pod to become `Running`, then refreshed the browser.

This time, I created the admin account immediately.

![Portainer Admin Account](https://paper-attachments.dropboxusercontent.com/s_05AD378A82578B8F940350A334CC24A83B9D9779BBD50D7F99799A62C18025AC_1772907813899_screencapture-10-90-1-30-30777-2026-02-25-03_44_25.png)

## Final Result

After logging in, Portainer automatically detected my Kubernetes environment. From there, I could:

- View nodes and namespaces
- Inspect pods and logs
- Deploy applications
- Scale workloads
- Manage cluster resources visually

Everything worked out of the box.

![Portainer Admin](https://paper-attachments.dropboxusercontent.com/s_05AD378A82578B8F940350A334CC24A83B9D9779BBD50D7F99799A62C18025AC_1772907944407_screencapture-10-90-1-30-30777-2026-02-25-03_46_30.png)

## Final Thoughts

Running Kubernetes in a homelab is a great way to learn infrastructure, but tools like Portainer add an important layer of usability.

It provides a clear view of what is happening inside the cluster and makes management easier without removing the flexibility of the command line.

For anyone running Kubernetes at home, especially on virtualized infrastructure, adding a management interface is a worthwhile improvement.



A step-by-step guide to installing Portainer on a Kubernetes (k3s) homelab running on Proxmox. Covers Helm setup, fixing cluster reachability errors, resolving the security timeout, and accessing the Portainer UI.

Read about

Featured

AI Has Reached Infrastructure. It Hasn't Reached Autonomy Yet.

Recent posts

Get in touch with FullstackWriter