I recently ran into an interesting issue in my cluster. One node could pull container images from AWS ECR without any problems, while another node refused to cooperate. Kubernetes kept throwing authentication errors even though everything looked correctly configured.

In this post, I will walk through:

- The problem I encountered
- The debugging process
- A few possible solutions
- The approach I eventually settled on

If you are running Kubernetes in a homelab and pulling images from Amazon Elastic Container Registry [(ECR)](https://aws.amazon.com/ecr/), this might save you some troubleshooting time.

## My Homelab Setup

My Kubernetes cluster runs on [k3s](https://fullstackwriter.dev/post/how-i-installed-kubernetes-on-my-homelab-using-k3s-on-proxmox-ubuntu?category=kubernetes) because it is lightweight and easy to operate in a small environment.

The cluster currently has two nodes:

| **Node**        | **Role**      |
| --------------- | ------------- |
| control-plane-1 | Control Plane |
| k3s-worker-1    | Worker        |

My application images are stored in AWS ECR and deployed to the cluster using a Kubernetes Deployment.

The deployment uses this image:

```bash
  <SERVICE_ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com/webappservice:webappservice-latest
```

## The Problem

I deployed the application with **two replicas**, expecting Kubernetes to schedule one pod on each node. That part worked, but the pods behaved differently depending on where they landed.

| **Node**      | **Pod Status**   |
| ------------- | ---------------- |
| Worker Node   | Running          |
| Control Plane | ImagePullBackOff |

The control plane pod kept failing with this error:

```bash
Back-off pulling image
ErrImagePull: failed to pull and unpack image
no basic auth credentials
```

This suggested the node could not authenticate with ECR.

## First Assumption: Missing Authentication

My first thought was that Kubernetes did not have credentials to access the registry. To fix that, I created a pull secret using the AWS CLI.

```bash
kubectl create secret docker-registry ecr-secret \
  --docker-server=<SERVICE_ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com \
  --docker-username=AWS \
  --docker-password=$(aws ecr get-login-password --region eu-west-2) \
  --namespace dev
```

Then I updated the deployment to reference the secret.

```bash
imagePullSecrets:
  - name: ecr-secret
```

However, the control plane node still failed to pull the image.

## Verifying the Node was the Problem

To confirm whether this was a Kubernetes issue or a node-level issue, I SSH'd into the control plane node and attempted to pull the image directly using containerd.

```bash
sudo ctr image pull <SERVICE_ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com/webappservice:latest
```

I got this error:

```bash
pull access denied
authorization failed: no basic auth credentials
```

This confirmed the issue was node authentication. Interestingly, the worker node worked fine.

At this point, I realized the worker node likely already had the image cached locally, which explained why its pod started successfully.

## Attempt 2: Node-Level Authentication

The next idea was configuring authentication at the node level.

One approach is using the [ECR credential helper](https://github.com/awslabs/amazon-ecr-credential-helper), which automatically fetches tokens from AWS whenever an image pull happens.

However, this turned out to be unreliable in my setup because k3s uses **containerd**, and containerd does not automatically read Docker credential helper configurations. This made the approach more complicated than necessary for a homelab cluster.

## The Simpler and More Reliable Solution

Instead of configuring authentication per node or per deployment, I decided to attach the ECR pull secret to the **default ServiceAccount in the namespace**.

This means any pod created in that namespace automatically inherits the image pull secret. To do this, I followed the steps below:

**Step 1: Create the ECR Pull Secret**

First, I recreated the secret.

```bash
kubectl create secret docker-registry ecr-secret \
  --docker-server=<SERVICE_ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com \
  --docker-username=AWS \
  --docker-password=$(aws ecr get-login-password --region eu-west-2) \
  --namespace dev
```

**Step 2: Attach the Secret to the Namespace ServiceAccount**

Next, I patched the default ServiceAccount in the `dev` namespace.

```bash
kubectl patch serviceaccount default \
  -n dev \
  -p '{"imagePullSecrets": [{"name": "ecr-secret"}]}'
```

Now every pod created in the namespace automatically uses the secret.

**Step 3: Restart the Failing Pod**

I deleted the failing pod, so Kubernetes would recreate it.

```bash
kubectl delete pod -n dev webappservice-hudjfi
```

The deployment created a new pod, and this time it started successfully.

**Verifying the Fix**

Running `kubectl describe pod` confirmed the container started correctly. Key parts of the output looked like this:

```bash
State: Running
Ready: True
```

The event logs also showed:

```bash
Container image already present on machine
Created container
Started container
```

At this point, both nodes were running the application successfully.

## Why I Chose the ServiceAccount Approach

There are a few ways to solve ECR authentication in Kubernetes.

| **Approach**                    | **Pros**                | **Cons**                     |
| ------------------------------- | ----------------------- | ---------------------------- |
| Node credential helper          | Automatic token refresh | Harder to configure with k3s |
| Static registry token           | Simple                  | Tokens expire after 12 hours |
| imagePullSecrets per deployment | Works                   | Repetitive configuration     |
| ServiceAccount imagePullSecret  | Simple and reusable     | Requires secret refresh      |

The **ServiceAccount approach** felt like the best tradeoff for a homelab environment because:

- It centralizes authentication
- It avoids repeating the configuration in every deployment
- It works consistently across nodes

Despite the flexibility the ServiceAccount approach brings, it still has the same problem of 12-hour expiry tokens that come with ECR tokens. Thankfully, this can be managed using a Kubernetes CronJob that refreshes the token periodically.

## Automatically Refreshing the ECR Secret

As mentioned earlier, authentication tokens from ECR expire every 12 hours. This does not affect running pods, but it can break new deployments or pods scheduled on fresh nodes if the secret has expired.

A simple way to solve this in a homelab cluster is to periodically refresh the secret using a Kubernetes CronJob.

![](https://paper-attachments.dropboxusercontent.com/s_88EBE4FB662784CC10B48EE40A4F16D609D3D7F24069A2CADFF36D32EB9F9E94_1772996213878_image.png)

This keeps the cluster working without manual intervention.

**Step 1: Create a ServiceAccount for the Job**

First, I created a ServiceAccount that the CronJob can use.

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
    name: ecr-secret-refresher
    namespace: dev
```

**Step 2: Allow the Job to Manage Secrets**

The job needs permission to update secrets in the namespace.

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
    name: ecr-secret-manager
    namespace: dev
rules:
    - apiGroups: ['']
      resources: ['secrets']
      verbs: ['get', 'create', 'delete']
```

Then bind the role to the ServiceAccount.

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
    name: ecr-secret-manager-binding
    namespace: dev
subjects:
    - kind: ServiceAccount
      name: ecr-secret-refresher
roleRef:
    kind: Role
    name: ecr-secret-manager
    apiGroup: rbac.authorization.k8s.io
```

**Step 3: Create the CronJob**

Before setting up the CronJob, the job container needs AWS credentials to call `ecr get-login-password`. In a managed cloud environment, I would typically handle this with an IAM role, but in a homelab the simplest approach is storing my AWS credentials in a Kubernetes secret and injecting them as environment variables.

```bash
kubectl create secret generic aws-credentials \
  --from-literal=AWS_ACCESS_KEY_ID=<your-access-key-id> \
  --from-literal=AWS_SECRET_ACCESS_KEY=<your-secret-access-key> \
  --namespace dev
```

Then, I created the CronJob that refreshes the secret every 11 hours.

```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
    name: refresh-ecr-secret
    namespace: dev
spec:
    schedule: '0 */11 * * *'
    jobTemplate:
        spec:
            template:
                spec:
                    serviceAccountName: ecr-secret-refresher
                    restartPolicy: OnFailure
                    containers:
                        - name: refresh-secret
                          image: amazon/aws-cli
                          env:
                              - name: AWS_REGION
                                value: eu-west-2
                          command:
                              - /bin/sh
                              - -c
                              - |
                                  TOKEN=$(aws ecr get-login-password --region $AWS_REGION)
                                  kubectl delete secret ecr-secret -n dev --ignore-not-found
                                  kubectl create secret docker-registry ecr-secret \
                                    --docker-server=<ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com \
                                    --docker-username=AWS \
                                    --docker-password=$TOKEN \
                                    -n dev
```

This job will:

1. Request a fresh login token from ECR
2. Delete the old secret
3. Recreate the secret with the new token

Since the namespace ServiceAccount already references `ecr-secret`, all new pods automatically use the refreshed credentials.

**Applying the Configuration**

I saved the resources in a file and apply them:

```bash
kubectl apply -f cronjob.yaml
```

Then I verified the CronJob with:

```bash
kubectl get cronjobs -n dev
```

![Cronjob](https://paper-attachments.dropboxusercontent.com/s_88EBE4FB662784CC10B48EE40A4F16D609D3D7F24069A2CADFF36D32EB9F9E94_1772997819523_Screenshot+2026-03-08+at+20.23.11.png)

## Final Thoughts

The most misleading part of this issue was that one pod was already running. It made the problem look like an intermittent auth failure when the real explanation was simpler: the worker node had the image cached, so it never needed to authenticate. The control plane node had no cache, no credentials, and nowhere to go.

Once that clicked, the fix was straightforward. Attaching the ECR pull secret to the namespace ServiceAccount means every pod in that namespace inherits it automatically, and the CronJob keeps the credentials fresh without any manual intervention.

If you're running a k3s homelab with images in ECR, this setup is worth the 10 minutes it takes to configure. It is less fragile than managing secrets per deployment and less complex than node-level credential helpers.


How to troubleshoot and pull images from Amazon Elastic Container Registry (ECR) into your Kubernetes

Why One Node Pulled and the Other Didn't: Debugging an ECR Image Pull Failure in Kubernetes Homelab

So you’ve installed a Linux virtual machine (VM). Congratulations! Now you’re staring at the terminal and thinking, _“What’s next?”_ _“How do I move forward?”_ _“What should I install to get the environment ready?”_ You’re not alone. We've all been there. I recently went through the same phase after setting up my Homelab. That’s a setup where you run your own servers, networks, or other infrastructure at home to learn, experiment, or support side projects. I remember wondering what to install and how to get everything running as a proper Homelab or even a lightweight production environment.

In this guide, I’ll walk you through the essential post-installation steps to get your Linux VM ready for deploying, hosting, and building software.

## 1. Initial System Updates and Package Management

Before you get into any complex setup, the first thing you should do is update your VM. This pulls in the latest security fixes, patches, and features. It might not seem exciting, but skipping this step is like leaving your front door unlocked. Outdated software is one of the easiest ways for things to go wrong.

```bash
sudo apt update
sudo apt upgrade -y
```

## 2. Install Essential Tools

With your Linux VM updated, the next step is to install some basic development tools. These will help you access the internet, manage code, and do everyday tasks more easily.

```bash
sudo apt install build-essential git curl wget vim -y
```

Here’s a quick breakdown of what each tool does:

-   `build-essential` is a package that includes tools like `gcc`, `g++`, and `make`, which are needed to compile and build software from source.
-   `git` is used to track changes in your code and collaborate with others on projects.
-   `curl` lets you fetch data from URLs directly from the terminal. It’s great for testing APIs or downloading files.
-   `wget` is another tool for downloading files from the internet. It works well for pulling entire files or even websites.
-   `vim` is a command-line text editor. It’s handy for quickly editing config files or writing code without needing a full-blown IDE.

## 3. User Management: Create a Non-Root User

When you first set up your Linux VM, you’ll usually have access as the root user. That’s convenient because you can do anything on the machine. However, it’s also risky. It's like driving without a seatbelt. The root account should be used only for core administrative tasks. You should create a regular user account for your daily work to stay safe and avoid mistakes.

```bash
# Create a new user
sudo adduser yourusername

# Add user to sudo group (for admin privileges when needed)
sudo usermod -aG sudo yourusername
```

## 3. Set Up SSH and Permission Keys for Remote Access

While using a username and password lets you access your VM, you probably don’t want to start sharing those credentials with applications that need to connect to it. For example, if you’ve set up a GitHub Action to deploy a Next.js app to your VM, using your username and password for remote access is not a good idea. Instead, you should use an SSH key. It gives you a secure way to connect to your VM without relying on a password.

Passwords are more vulnerable to brute force attacks. SSH keys help reduce that risk and give you a safer way to manage access.

To set up SSH, follow the steps below:

**Step 1: Get your VM IP address using your VM terminal**

```bash
ip addr show
```

**Step 2: Generate an SSH key on your local machine**

```bash
ssh-keygen -t ed25519
```

You’ll be asked to enter a name and some optional details. You can press Enter to accept the defaults.

**Step 3: Copy your public key to the VM**

```bash
ssh-copy-id yourusername@your.vm.ip.address
```

To confirm the key was copied successfully, run this command on your VM:

```bash
cat ~/.ssh/authorized_keys
```

**Step 4: Disable password authentication on your VM**

Use `vim` to open the SSH config file

    sudo vim /etc/ssh/sshd_config

Look for the line that says `PasswordAuthentication` and change its value to `no`. Remove the `#` and update the value if the line is commented out.

**Step 5: Restart the SSH service**

```bash
sudo systemctl restart ssh
```

**Step 6: Router Exposure and Port Forwarding**

If you want to access your VM from outside your local network, you’ll need to forward the SSH port (by default, that’s port 22) from your router to your VM. Every router is a bit different, so check your router’s manual or admin panel for how to set up port forwarding.

> **Tip:** You can change the default SSH port to something less common, like `2222`, to avoid drawing unnecessary attention from bots or scanners. Just remember to update the `Port` value in your SSH config file and restart the SSH service afterward.

## 4. Firewall Configuration and Traffic Management

Now that your VM is exposed to the internet, it’s important to lock it down. Without a firewall, it’s like leaving your house wide open with a sign that says "Come on in." You want to let in only what you need and block everything else.

The Uncomplicated Firewall (UFW) makes it easy to manage which traffic is allowed in and out of your VM.

**Step 1: Install and configure UFW**

Update your package list and install UFW.

```bash
sudo apt update && sudo apt install ufw
```

**Step 2: Allow essential services**

Let UFW know which ports you want open. For example:

```bash
sudo ufw allow 2222/tcp  # if you changed SSH port
sudo ufw allow 80/tcp    # for HTTP
sudo ufw allow 443/tcp   # for HTTPS
```

**Step 3: Enable the firewall**

Once your rules are in place, turn the firewall on.

```bash
sudo ufw enable
```

**Step 4: Check your firewall status**

```bash
sudo ufw status verbose
```

## 5. Installing and Configuring Nginx: Your First Web Server

To get a sense of what your VM can do, it’s a great idea to set up Nginx. Nginx is a lightweight, powerful web server that’s perfect for both testing and production environments. Setting it up will show that your VM is ready to serve web content and handle real-world traffic.

**Step 1: Install Nginx**

Start by updating your package list and installing Nginx.

```bash
sudo apt update && sudo apt install nginx
```

**Step 2: Start and enable the Nginx service**

Once installed, start the Nginx service and make sure it starts automatically on boot:

```bash
sudo systemctl start nginx
sudo systemctl enable nginx
```

**Step 3: Test your setup**

Open a browser and go to `http://your-vm-ip`. If everything is set up correctly, you’ll see the Nginx welcome page.

![Nginx welcome page](https://paper-attachments.dropboxusercontent.com/s_D8636135A0B0BB3900626336E0942EFDE38AEFE4AEBF27955658135C7F28048F_1743903396635_Image+06-04-2025+at+02.36.jpg)

## Wrapping Up

Setting up your Linux VM after installation might feel overwhelming at first, but each step is important for making sure your system is secure, efficient, and ready to handle your software deployment and hosting needs.

Keep in mind that system administration isn’t a one-time task. It’s an ongoing process. Make it a habit to review your security settings, keep your software updated, and be ready to adjust as your needs evolve.

Just installed a Linux VM and wondering what to do next? This guide walks you through everything you need to get your VM production-ready—from adding secure users and setting up SSH keys, to configuring firewalls, port forwarding, and even running your first Nginx web server.

Now that you've installed Linux VM, what's next?

I've been running a homelab Kubernetes cluster for a while now. It’s a simple two-node k3s setup with Traefik handling ingress and a Cloudflare Tunnel for external access. It’s been a great space to experiment and learn.

But deployment was still a pain.

Recently, I had a frontend app sitting in a GitLab repo, and every time I wanted to deploy, it turned into a small ritual. SSH into the server, pull the latest image, restart containers, and hope everything still worked. It got the job done, but it wasn’t something I trusted or enjoyed.

This post walks through how I moved from that setup to a full GitOps workflow using GitLab CI and ArgoCD. No prior ArgoCD experience, just figuring things out step by step until a simple `git push` became enough to deploy.

## The Starting Point

Going into this, my setup was pretty straightforward. I had a k3s cluster running on two Ubuntu nodes: one control plane and one worker, both on my local network. Traefik came bundled with k3s, so ingress was already handled. For external access, I was using a Cloudflare Tunnel running via `cloudflared` pods in a dedicated namespace, routing traffic from `*.demolamalomo.xyz` into the cluster.

The app itself was a frontend project. A Node.js app that builds into static files and gets served with nginx. The code lived in GitLab, and I was already using GitLab’s container registry to store images from earlier CI builds.

The CI pipeline, however, was doing too much. It built the image, then SSH-ed into the server to stop containers, remove them, pull new images, and restart everything. It worked most of the time, but it was fragile, hard to debug, and rolling back meant doing everything manually again.

## The Options I Considered

Throughout my deployment journey, I explored a couple of options before arriving at ArgoCD.

1. **Manual build and deploy** was basically what I was already doing. Build locally, push the image, run `kubectl apply`. The issue with this is that there’s no automation, no audit trail, and plenty of room for error.
2. **GitLab CI/CD handling everything** was the next step. The pipeline builds the image, pushes it, and deploys directly to the cluster using `kubectl`. This is where a lot of teams start, and honestly, it worked reasonably well for me too. But the CI was doing a lot of work, and it can get really messy with a lot of echoing and sshing commands. For context, below is a section of one of my CI build:

```bash
- echo "$ID_RSA_DEV_HOMELAB" > id_rsa_homelab
    - chmod 600 id_rsa_homelab
    - apk update && apk add openssh-client
    - echo "stop existing docker container & remove images"
    - echo "Port $STAGE_SVR_PORT_HOMELAB"
    - echo "User $DEV_SERVER_USER_HOMELAB"
    - echo "Host $STAGE_SVR_IP_HOMELAB"
    - echo "VM $STAGE_VM_IP_HOMELAB"
    - echo "Service $SERVICE_NAME_HOMELAB"
    - ssh -i id_rsa_homelab -p $STAGE_SVR_PORT_HOMELAB -o StrictHostKeyChecking=no $DEV_SERVER_USER_HOMELAB@$DEV_SVR_IP_HOMELAB "docker stop $SERVICE_NAME_HOMELAB || true"
......
```

3. **GitOps with ArgoCD** takes a different approach. CI still builds and pushes images, but it never talks to the cluster directly. Instead, it updates a Git repo that contains the Kubernetes manifests. ArgoCD, running inside the cluster, watches that repo and syncs any changes.The thing I liked most was that I could look at a repository and immediately know what should be running in the cluster.

That was the point where ArgoCD clicked for me. I liked the fact that I could open a repo and know exactly what was deployed. Rollbacks also become much simpler, by just reverting a commit. So I went with **ArgoCD**.

## Installing ArgoCD

I’d never used ArgoCD before, so this part was new. Thankfully, installation was straightforward:

```bash
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
```

After a minute or so, everything was running, and I could start exploring the dashboard.

To access the dashboard, I started with a simple port-forward:

```bash
kubectl port-forward svc/argocd-server -n argocd 8080:443
```

The default login is `admin`, and the initial password is stored in a Kubernetes secret:

```bash
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
```

I also installed the ArgoCD CLI, which made it easier to create applications and debug sync issues without constantly switching to the browser.

Later on, I exposed ArgoCD through Traefik and my Cloudflare Tunnel so I could access it remotely, but the port-forward was more than enough to get started.

![ArgoCD UI](https://paper-attachments.dropboxusercontent.com/s_2A415A66514362A902F36E27BA0662B25C280357D4A31FB4FA73DA4B634FE1E6_1775396935665_Screenshot+2026-04-05+at+14.48.16.png)

## Understanding the Two-Repo Pattern

One thing that took me a while to wrap my head around was the idea of maintaining two repositories. At first it felt unnecessary. Why split things up when everything belongs to the same application?

But after working with it for a bit, the separation started to make sense.

The **application repo** is where development happens. Code changes, Dockerfile, and CI configuration (`.gitlab-ci.yml`).

The **GitOps repo** is different. Its only job is to describe what's running in the cluster. Deployments, Services, and Ingress definitions. ArgoCD watches this repo and syncs the cluster whenever something changes.

The benefit became obvious pretty quickly. My application repo changes all the time with feature work, bug fixes, and random experiments, while the GitOps repo changes much less often because it only reflects what should be running in the cluster.

![ArgoCD workflow](https://paper-attachments.dropboxusercontent.com/s_2A415A66514362A902F36E27BA0662B25C280357D4A31FB4FA73DA4B634FE1E6_1775397646668_mermaid-diagram.png)

My GitOps repo (`homelab-gitops`) ended up looking like this:

```bash
homelab-gitops/
  clohea-admin/   # my application folder
    deployment.yaml
    service.yaml
    ingress.yaml
```

## Setting up the GitLab CI Pipeline

My existing `.gitlab-ci.yml` was doing too much. The new version only needed to do two things: build the image and update the GitOps repo.

The build stage pushes two tags:

- `:latest` for general use
- a commit hash tag like `:a1b2c3d`, which is what goes into the manifests

That commit-specific tag is important because it gives ArgoCD a concrete change to detect.

```yaml
docker-build-homelab-dev:
    image: docker:latest
    stage: build
    services:
        - docker:dind
    before_script:
        - echo $CI_JOB_TOKEN | docker login -u "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY
    script:
        - echo "$DEV_ENV" > $CI_PROJECT_DIR/.env
        - docker build -f ./Dockerfile --pull -t "$CI_REGISTRY_IMAGE:latest" -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA" .
        - docker push "$CI_REGISTRY_IMAGE:latest"
        - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA"
    only:
        - homelab-dev
```

The deploy stage updates the GitOps repo:

```yaml
update-manifest:
    stage: deploy
    image: alpine:latest
    before_script:
        - apk add --no-cache git sed
        - git config --global user.email "ci@gitlab.com"
        - git config --global user.name "GitLab CI"
    script:
        - git clone https://deploy-token:${GITOPS_TOKEN}@gitlab.com/clohea-fe/homelab-gitops.git
        - cd homelab-gitops
        - 'sed -i "s|image: .*|image: ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA}|" clohea-admin/deployment.yaml'
        - git add .
        - git diff --cached --quiet && echo "No changes" || git commit -m "Update frontend image to $CI_COMMIT_SHORT_SHA" && git push
    only:
        - homelab-dev
```

One thing that tripped me up: the `sed` command needs to use double quotes so that the shell expands the CI variables. My first attempt used single quotes, which meant `$CI_REGISTRY_IMAGE` was treated as literal text. The `sed` command found nothing to replace, `git commit` found nothing to commit, and the pipeline failed with `exit code 1`. A small thing, but the kind of mistake that costs you 30 minutes of staring at logs.

**A Note on the GITOPS_TOKEN**
The `update-manifest` job needs write access to the GitOps repo. I created a **Project Access Token** in the GitOps repo (Settings > Access Tokens) with `write_repository` scope, then stored it as a CI/CD variable called `GITOPS_TOKEN` in the app repo (Settings > CI/CD > Variables), with the "Mask variable" option checked so it doesn't leak into logs.

I could also have used a Personal Access Token, which would work across all my repositories, but a Project Access Token scoped to just the GitOps repo is cleaner from a security standpoint.

## The Kubernetes Manifests

The GitOps repo itself stayed pretty simple. For this application, I only needed three manifests:

- A Deployment
- A Service
- An Ingress

Nothing fancy. Just enough to get the application running and exposed through Traefik.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
    name: webapp-service
    namespace: frontend
spec:
    replicas: 2
    selector:
        matchLabels:
            app: webapp-service
    template:
        metadata:
            labels:
                app: webapp-service
        spec:
            imagePullSecrets:
                - name: gitlab-registry-cred
            containers:
                - name: webapp-service
                  image: registry.gitlab.com/clohea-fe/clohea-admin/webapp-service:INITIAL_TAG
                  ports:
                      - containerPort: 3000
```

Since the registry is private, I created an image pull secret:

```bash
kubectl create secret docker-registry gitlab-registry-cred \
  --namespace frontend \
  --docker-server=registry.gitlab.com \
  --docker-username=<deploy-token-username> \
  --docker-password=<deploy-token-value>
```

The service routes traffic internally:

```yaml
apiVersion: v1
kind: Service
metadata:
    name: webapp-service
    namespace: frontend
spec:
    selector:
        app: webapp-service
    ports:
        - port: 80
          targetPort: 3000
```

And the ingress exposes it via Traefik:

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
    name: webapp-service
    namespace: frontend
spec:
    ingressClassName: traefik
    rules:
        - host: admin-portal.demolamalomo.xyz
          http:
              paths:
                  - path: /
                    pathType: Prefix
                    backend:
                        service:
                            name: webapp-service
                            port:
                                number: 80
```

## Connecting ArgoCD to the GitOps Repo

For ArgoCD to watch the GitOps repo, it needs credentials since it's a private GitLab repo. I created a deploy token in the GitOps repo with `read_repository` scope, then added the repo to ArgoCD using the UI.

**Settings > Repositories > + CONNECT REPO**

![Add repository to ArgoCD](https://paper-attachments.dropboxusercontent.com/s_2A415A66514362A902F36E27BA0662B25C280357D4A31FB4FA73DA4B634FE1E6_1775418533445_Screenshot+2026-04-05+at+20.48.03.png)

Another option to do this is to use the CLI:

```bash
argocd repo add https://gitlab.com/clohea-fe/homelab-gitops.git \
  --username <deploy-token-username> \
  --password <deploy-token-value>
```

Then I created the application by entering the repository URL, the path to the application folder in the GitOps repo, and a few other required settings.

**Application > + NEW APP**

![New App](https://paper-attachments.dropboxusercontent.com/s_2A415A66514362A902F36E27BA0662B25C280357D4A31FB4FA73DA4B634FE1E6_1775418812700_Screenshot+2026-04-05+at+20.51.21.png)

I can also use the CLI by running the command below

```bash
argocd app create frontend \
  --repo https://gitlab.com/clohea-fe/homelab-gitops.git \
  --path clohea-admin \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace frontend \
  --sync-policy automated \
  --auto-prune \
  --self-heal
```

Quick breakdown of the flags:

- `--sync-policy automated`: syncs on every commit
- `--auto-prune`: removes deleted resources
- `--self-heal`: reverts manual cluster changes

## The Debugging Marathon

If I'm being honest, the setup didn't just work on the first try. Here's a condensed version of the issues I hit, because I think they're genuinely useful for anyone doing this for the first time.

**Image pull errors.** The first deployment failed with a `403 Forbidden` because Kubernetes couldn't authenticate with GitLab's private container registry. The fix was creating the `docker-registry` secret and adding `imagePullSecrets` to the deployment manifest. Then it failed again with `not found` — the image tag referenced in the manifest didn't exist in the registry because my build job hadn't pushed a commit-hash tag yet.

**Name mismatches.** My ingress was referencing a service called `frontend`, but the actual service was named `webapp-service`. A simple typo that resulted in a 502 from Traefik. `kubectl describe ingress` made this obvious once I thought to look.

**Port mismatch.** This one was subtle. The deployment had `containerPort: 80` and the service was targeting port 80, but nginx inside the container was actually listening on port 3000. The Docker image had a custom nginx config that bound to 3000 instead of the default 80. Running `kubectl exec` and checking `netstat -tlnp` inside the pod revealed the mismatch immediately.

## The End Result

Once everything was wired up and the bugs were resolved, the flow became exactly what I wanted:

- I push code to the `homelab-dev` branch.
- GitLab CI builds the Docker image and pushes it to the registry with both a `:latest` and a commit-hash tag.
- The `update-manifest` job clones the GitOps repo, updates the image tag in the deployment manifest, and pushes.
- ArgoCD detects the new commit within seconds.
- ArgoCD syncs the manifests to the cluster, Kubernetes pulls the new image, and the app is live.

The ArgoCD dashboard shows the full picture. Sync status, pod health, the Git commit that triggered the deployment. If something goes wrong, I can see exactly which commit caused it and revert.

![ArgoCD Dashboard](https://paper-attachments.dropboxusercontent.com/s_2A415A66514362A902F36E27BA0662B25C280357D4A31FB4FA73DA4B634FE1E6_1775420456749_Screenshot+2026-04-05+at+21.19.00.png)

The entire deployment chain is now traceable through Git commits. No more SSH-ing into servers or chained server commands. Just `git push`, and everything takes care of itself.


I got tired of SSH-ing into servers every time I needed to deploy a simple app. So I spent an evening setting up GitLab CI, ArgoCD, and a GitOps workflow in my homelab. Here's what worked, what broke, and what I learned along the way.

From Manual Deploys to GitOps: How I Automated my Homelab with ArgoCD and GitLab CI

I run k3s in my [homelab](https://fullstackwriter.dev/post/how-i-installed-kubernetes-on-my-homelab-using-k3s-on-proxmox-ubuntu?category=kubernetes), and I wanted to expose my apps to the internet without opening multiple ports or configuring complex router rules.

I had the options of NodePort or installing MetalLB. Although both approaches work, neither felt close to how things are done in production. NodePort would mean exposing a port on my router and mapping it to a node. That exposes my public IP and opens inbound traffic into my network. MetalLB is cleaner inside Kubernetes, but it still assumes you have public IP handling and proper routing at the edge.

What I wanted instead was something closer to how modern infrastructure works, that supports encrypted transport, no inbound ports, and clean DNS control.

That is why I chose Cloudflare Tunnel. The goal for me was to use my domain `demolamalomo.xyz` registered on AWS Route 53. I wanted something very close to production that caters to the following:

- Apps accessible at `*.demolamalomo.xyz` (e.g., `whoami.demolamalomo.xyz`, `grafana.demolamalomo.xyz`)
- No router port forwarding
- No public IP exposure
- Cloudflared running as Kubernetes pods, not installed on the host

## Step 1: Move the Domain to Cloudflare DNS

My first instinct was to delegate only a subdomain like `k3s.demolamalomo.xyz` to Cloudflare using NS records in Route 53. That way my root domain stays fully in Route 53 and only the Kubernetes subdomain gets managed by Cloudflare.

That didn’t work. Cloudflare's free plan does not support adding subdomain zones. You can only add root domains like `demolamalomo.yxz`. On testing, this didn’t work because anytime I dig through the nameserver, the nameservers returned are still the ones tied to Route 53.

So I moved the entire domain's DNS to Cloudflare by following the steps below:

- In the Cloudflare Dashboard, I clicked the + **Add** button, selected the **Connect a domain** option, and filled in the required field.

![Connect a domain](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774118683934_Screenshot+2026-03-04+at+01.27.15.png)
![Fill the form](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774118683991_Screenshot+2026-03-04+at+01.38.39.png)

- Selected the **Free** plan
- Cloudflare scanned and imported my existing Route 53 DNS records automatically
- Cloudflare assigned me two nameservers similar to `****.**.cloudflare.com` and `******.**.cloudflare.com`

Then I updated Route 53 to point to those nameservers.

**The First Mistake: Changing Nameservers in the Wrong Place**
Route 53 has two places where nameservers appear, and I changed the wrong one.

**Wrong place:** Route 53 >> Hosted zones >> demolamalomo.xyz >> NS record. This is just a DNS record inside the zone. Changing it does nothing to where the internet actually looks for your domain.
**Right place:** Route 53 >> **Registered domains** >> demolamalomo.xyz >> **Name servers**. This is what tells the TLD registry where to send DNS queries for your domain.

I spent time debugging why `dig clohea.net NS @1.1.1.1` still returned AWS nameservers before realizing I had changed the wrong thing. Once I updated the registered domain's nameservers, Cloudflare showed the domain as **Active**, and DNS started resolving correctly.

## Step 2: Create the Cloudflare Tunnel

In the Cloudflare Zero Trust [dashboard](https://one.dash.cloudflare.com), I created a tunnel following the steps below:

- Navigate to **Networks >> Connectors**
- Click **Create a tunnel**
- Select **Cloudflared** as the connector type

![Select Cloudflared](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774128112097_Screenshot+2026-03-21+at+21.39.42.png)

- Named it `k3s-tunnel`
- On the connector install page, copied the **tunnel token** starting with `eyJ...`
- Then, I installed Cloudflared on my machine with the token copied.
  sudo cloudflared service install eyJ......

Once Cloudflared is installed, the status shows `HEALTHY` in my dashboard.

Next, I proceeded to the next page to configure the subdomain and service (Traefik in my case)

![Configured subdomain](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774128103478_Screenshot+2026-03-21+at+22.21.07.png)

## Step 3: Deploy Cloudflared on Kubernetes

I chose to run **cloudflared** as Kubernetes pods rather than installing it on the host machine. This keeps everything inside the cluster and makes it easy to manage with standard Kubernetes tooling.

To do this, first, I created a namespace and stored the tunnel token I got earlier as a secret:

```bash
kubectl create namespace cloudflare

kubectl create secret generic cloudflared-token \
  --namespace cloudflare \
  --from-literal=token=<TUNNEL_TOKEN>
```

Then, I used the secret to create a deployment:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
    name: cloudflared
    namespace: cloudflare
spec:
    replicas: 2
    selector:
        matchLabels:
            app: cloudflared
    template:
        metadata:
            labels:
                app: cloudflared
        spec:
            containers:
                - name: cloudflared
                  image: cloudflare/cloudflared:latest
                  args:
                      - tunnel
                      - --no-autoupdate
                      - run
                  env:
                      - name: TUNNEL_TOKEN
                        valueFrom:
                            secretKeyRef:
                                name: cloudflared-token
                                key: token
                  resources:
                      requests:
                          memory: '128Mi'
                          cpu: '100m'
                      limits:
                          memory: '256Mi'
                          cpu: '200m'
```

This creates two replicas of the cloudflared instance, and the `TUNNEL_TOKEN` environment variable tells cloudflared to run in remotely-managed mode (the tunnel configuration lives in Cloudflare's dashboard, not in a local config file).

After applying, I verified the pods were connected:

```bash
kubectl logs -n cloudflare -l app=cloudflared --tail=20
```

I saw `Registered tunnel connection` messages with `protocol=quic`, confirming the tunnel was live.

## Step 4: Deploy a Test App

I deployed the `traefik/whoami` image as a simple test:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
    name: whoami
    namespace: default
spec:
    replicas: 1
    selector:
        matchLabels:
            app: whoami
    template:
        metadata:
            labels:
                app: whoami
        spec:
            containers:
                - name: whoami
                  image: traefik/whoami:latest
                  ports:
                      - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
    name: whoami
    namespace: default
spec:
    selector:
        app: whoami
    ports:
        - port: 80
          targetPort: 80
```

Also added the Ingress rule telling Traefik how to route based on the Host header:

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
    name: whoami-ingress
    namespace: default
spec:
    rules:
        - host: whoami.demolamalomo.xyz
          http:
              paths:
                  - path: /
                    pathType: Prefix
                    backend:
                        service:
                            name: whoami
                            port:
                                number: 80
```

**The 502 Problem**
At this point, I expected everything to work with the DNS resolved. The tunnel was connected. But when I hit `https://whoami.demolamalomo.xyz`, I got a **502 Bad Gateway** from Cloudflare.

This was the most confusing problem because:

- The cloudflared pods were healthy and connected
- The tunnel logs showed no request-level errors
- From a debug pod inside the cluster, curling `http://traefik.kube-system.svc.cluster.local:80` with the correct Host header returned a perfect 200 response

The issue was **DNS resolution inside the cloudflared container**. My tunnel config originally pointed to:

```bash
http://traefik.kube-system.svc.cluster.local:80
```

The cloudflared Docker image is built on scratch, an essentially empty container with nothing but the cloudflared binary, with no `shell`, `cat` , `curl` , or `/etc/resolv.conf`. When cloudflared tried to resolve `traefik.kube-system.svc.cluster.local`, it had no idea which DNS server to ask and failed silently. The request never reached Traefik, so Cloudflare returned a `502`.

I confirmed this by switching the tunnel config to use **Traefik's ClusterIP directly**:

```bash
http://10.43.250.137:80
```

Immediately, I ran curl `https://whoami.clohea.net` , and it returned the whoami response with Cloudflare headers confirming traffic was flowing through the tunnel.

**Why the ClusterIP Approach Is Fine**
In a homelab, the Traefik service ClusterIP is stable. It only changes if you delete and recreate the service. For a more robust solution, you could add explicit DNS config to the cloudflared deployment:

```yaml
spec:
    template:
        spec:
            dnsPolicy: 'ClusterFirst'
            dnsConfig:
                nameservers:
                    - 10.43.0.10 # kube-dns ClusterIP
                searches:
                    - kube-system.svc.cluster.local
                    - svc.cluster.local
                    - cluster.local
```

But for a homelab, the ClusterIP approach is simpler and works.

**Adding New Apps**
The best part about this setup is that adding a new app requires zero changes to Cloudflare. The wildcard `*.demolamalomo.xyz` already covers every subdomain. I just need to:

- A Deployment and Service for my app
- An Ingress with the right hostname

The architecture looks like this:

![](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774133078724_image.png)

## Lessons Learned

- **Route 53 has two places for nameservers.** The hosted zone NS record is not the same as the registered domain's nameserver setting. Change the wrong one and nothing happens. Change it under **Registered domains** >> **Name servers**.
- **The Cloudflared Docker image cannot resolve Kubernetes DNS names.** It is built on a scratch base image with no `/etc/resolv.conf`. Either use the target service's ClusterIP directly or add explicit `dnsConfig` to the pod spec.
- **Ingress host names must match exactly.** If your tunnel routes to `*.clohea.net` but your Ingress says `whoami.k3s.clohea.net`, Traefik will not match the request and you will get a 502.


I wanted production-style ingress for my k3s homelab without exposing ports or my public IP. What looked simple turned into debugging Route 53 nameservers, Kubernetes DNS resolution, and mysterious Cloudflare 502s before everything finally worked.

How I Exposed My Kubernetes Homelab to the Internet Using Cloudflare Tunnel

I recently set up Kubernetes in my homelab to build real operational experience, not just theoretical knowledge. My goal was to run something that behaves as close as possible to a production environment, similar to what you’d get with [Amazon Elastic Kubernetes Service](https://aws.amazon.com/eks/), while still keeping things lightweight and fully under my control.

To make that happen, I deployed a Kubernetes cluster using [k3s](https://k3s.io/) on an Ubuntu virtual machine hosted inside [Proxmox VE](https://www.proxmox.com/en/products/proxmox-virtual-environment/overview).

This article walks through exactly how I did it, why each step matters, and what the setup gives me moving forward.

## My Starting Point: Virtualized Infrastructure

Before installing Kubernetes, I already had a virtualization environment running on Proxmox. Inside it, I created a virtual machine running Ubuntu. That VM would serve as my Kubernetes control plane node.

Using virtualization mirrors how real production environments operate. Cloud providers run Kubernetes on virtual infrastructure, not bare laptops. My Proxmox VM plays the same role as a cloud compute instance, giving me isolation, resource control, and the ability to snapshot or rebuild when needed.

## Why I Chose k3s

I wanted a Kubernetes distribution that felt real but did not demand heavy hardware or complex setup.

k3s was the perfect fit because:

- It is fully Kubernetes compliant
- It runs with minimal resource usage
- Installation is extremely fast
- It removes operational overhead while preserving core behavior

In practice, it provides a learning environment that behaves like a production cluster without the operational overhead of a full upstream Kubernetes deployment.

## Phase 1: Preparing the Ubuntu Server

Before installing Kubernetes, the operating system had to be configured correctly. Kubernetes is sensitive to host configuration, especially networking and memory management.

**Static IP Configuration**

During Ubuntu installation, I manually assigned a static IP address for:

- Subnet
- Address
- Gateway
- DNS

This IP becomes the Kubernetes API endpoint. If it changes, cluster access breaks. Production clusters rely on stable control plane endpoints, so I treated my homelab the same way.

**Initial Server Setup**

After installation, I connected via SSH and updated the system packages. I installed common administrative tools such as curl, git, vim, htop, and networking utilities. These support everyday server operations like monitoring, editing configs, and downloading resources.

```bash
sudo apt update && sudo apt upgrade -y
sudo apt install -y curl wget git vim htop net-tools
```

**Proxmox Integration with QEMU Guest Agent**

I installed the QEMU guest agent to improve communication between the VM and Proxmox. This allows the hypervisor to detect IP addresses, perform graceful shutdowns, and monitor the VM’s state more reliably.

```bash
sudo apt install -y qemu-guest-agent
sudo systemctl enable qemu-guest-agent
sudo systemctl start qemu-guest-agent
```

**Network Validation**

Before going further, I verified:

- The static IP was assigned correctly
- External network access worked
- DNS resolution worked

Kubernetes depends heavily on networking. If connectivity is unstable, cluster components fail in unpredictable ways.

```bash
ip addr show ens18
# Should show your static IP
```

**Disabling Swap**

Kubernetes requires swap to be disabled because it interferes with resource scheduling and memory guarantees. This is a standard requirement across production clusters, so I disabled swap and made the change persistent.

```bash
sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstab

# Verify
free -h
# Swap should show 0
```

**Hostname Configuration**

I set the server hostname and mapped it in `/etc/hosts` so the system could resolve itself consistently. Stable node identity is important for certificates, networking, and cluster communication.

```bash
sudo hostnamectl set-hostname k3s-demo.local
```

Then modify `/etc/hosts` file to map the hostname:

```bash
# Update /etc/hosts
sudo vim /etc/hosts

# Add this line
<SERVER IP> k3s-demo.local k3s-demo
```

## Phase 2: Installing k3s

With the system prepared, I moved on to installing Kubernetes.

**Pre-Installation Checks**

Before running the installer, I confirmed:

- The required ports were free
- Enough RAM and CPU were available
- Sufficient disk space existed

This avoids conflicts and failed cluster startup.

```bash
# Check available ports
sudo ss -tulpn | grep -E ':(6443|10250)'

# Should return nothing (ports free)

# Check system resources
free -h  # Should show number of RAM
nproc    # Should show number of cores
df -h    # Should show plenty of disk space
```

**Installing k3s**

I installed k3s using the official script with three key options:

```bash
curl -sfL https://get.k3s.io | sh -s - \
  --write-kubeconfig-mode 644 \
  --disable traefik \
  --node-name k3s-demo
```

- `--write-kubeconfig-mode 644`: Makes kubectl work without sudo
- `--disable traefik`: Default Traefik ingress disabled so I can manage ingress myself later
- `--node-name`: Explicit node name for clarity and future scaling

The installation completed in under a minute.

**Verifying Cluster Health**

After installation, I confirmed:

- The K3s service was running
- The node reported Ready
- System pods in the kube-system namespace were healthy

If anything fails, system logs provide detailed troubleshooting information.

```bash
# Check k3s service
sudo systemctl status k3s

# Should see: active (running)

# Check node
kubectl get nodes

# Expected output:
# NAME          STATUS   ROLES                  AGE   VERSION
# k3s-demo    Ready    control-plane,master   1m    v1.28.x

# Check all system pods
kubectl get pods -A

# Should see pods in kube-system namespace running
```

![Running kubernetes](https://paper-attachments.dropboxusercontent.com/s_F3D46302D17DF57EA141A194AFED6667E114BCD5CC881287885039BD0D55EE8B_1772073514109_Image+26-02-2026+at+03.38.png)

## Does k3s Have Only a Control Plane? What About Worker Nodes?

When I ran `kubectl get nodes` to see the nodes in my Kubernetes cluster, I only saw a control plane node.

That worried me a bit because from what I understood about Kubernetes, the control plane, which acts as the brain, is separate from worker nodes, where applications actually run.

I expected my setup to look something like this:

```bash
NAME              STATUS   ROLES           AGE   VERSION
control-plane-1   Ready    control-plane   1h    v1.28.x
worker-1          Ready    <none>          1h    v1.28.x
worker-2          Ready    <none>          1h    v1.28.x
worker-3          Ready    <none>          1h    v1.28.x
```

After digging deeper into k3s architecture, I discovered this behavior is intentional. In k3s, a single provisioned node acts as both the control plane and the worker node. It manages the cluster and runs workloads at the same time.

This design is perfect for development, testing, and pre-production environments, which is exactly what most homelabs are built for.

Still, that setup was not enough for me. I wanted the real experience of managing a cluster with a separate control plane and worker nodes. Thankfully, k3s supports adding worker nodes, and that is exactly what I did.

**How to Add a Worker Node to k3s**

To add a worker node, the first step is to retrieve the control plane token. This token is required to join a new node to the cluster.

On the control plane VM, run:

```bash
sudo cat /var/lib/rancher/k3s/server/node-token
```

The output will look something like this:

```bash
K10abc123def456g.....
```

Next, I provisioned another VM, just like I did for the control plane. I disabled swap, installed the QEMU agent, and configured the necessary dependencies to prepare the VM for Kubernetes.

After that, I installed k3s in agent mode by running the following command on the new VM:

```bash
curl -sfL https://get.k3s.io | K3S_URL=https://<CONTROL_PLANE_IP> \
  K3S_TOKEN="K10abc123def456g......" \
  sh -
```

Here:

- `K3S_URL` is the control plane IP address
- `K3S_TOKEN` is the token retrieved earlier

Finally, I verified that the worker node successfully joined the cluster by running the following command on the control plane VM:

```bash
kubectl get nodes
```

![k3s with control plane and worker nodes](https://paper-attachments.dropboxusercontent.com/s_F3D46302D17DF57EA141A194AFED6667E114BCD5CC881287885039BD0D55EE8B_1772630050547_Image+04-03-2026+at+14.13.png)

## Final Thoughts

Setting up Kubernetes yourself changes how you understand it. Instead of interacting with a managed black box, you see every layer that makes the platform work.

Running k3s on a Proxmox-hosted Ubuntu server gives me a controlled, production-like environment where I can experiment, break things, and learn deeply without risk.

For anyone serious about mastering Kubernetes, building your own cluster is one of the most valuable steps you can take.


I built a Kubernetes cluster in my homelab using k3s on an Ubuntu VM running inside Proxmox. This guide walks through the full setup process, from server preparation to adding worker nodes.

#Homelab

Why One Node Pulled and the Other Didn't: Debugging an ECR Image Pull Failure in Kubernetes Homelab

Now that you've installed Linux VM, what's next?

From Manual Deploys to GitOps: How I Automated my Homelab with ArgoCD and GitLab CI

How I Exposed My Kubernetes Homelab to the Internet Using Cloudflare Tunnel

How I Installed Kubernetes on My Homelab (Using k3s on Proxmox + Ubuntu)