I recently ran into an interesting issue in my cluster. One node could pull container images from AWS ECR without any problems, while another node refused to cooperate. Kubernetes kept throwing authentication errors even though everything looked correctly configured.

In this post, I will walk through:

- The problem I encountered
- The debugging process
- A few possible solutions
- The approach I eventually settled on

If you are running Kubernetes in a homelab and pulling images from Amazon Elastic Container Registry [(ECR)](https://aws.amazon.com/ecr/), this might save you some troubleshooting time.

## My Homelab Setup

My Kubernetes cluster runs on [k3s](https://fullstackwriter.dev/post/how-i-installed-kubernetes-on-my-homelab-using-k3s-on-proxmox-ubuntu?category=kubernetes) because it is lightweight and easy to operate in a small environment.

The cluster currently has two nodes:

| **Node**        | **Role**      |
| --------------- | ------------- |
| control-plane-1 | Control Plane |
| k3s-worker-1    | Worker        |

My application images are stored in AWS ECR and deployed to the cluster using a Kubernetes Deployment.

The deployment uses this image:

```bash
  <SERVICE_ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com/webappservice:webappservice-latest
```

## The Problem

I deployed the application with **two replicas**, expecting Kubernetes to schedule one pod on each node. That part worked, but the pods behaved differently depending on where they landed.

| **Node**      | **Pod Status**   |
| ------------- | ---------------- |
| Worker Node   | Running          |
| Control Plane | ImagePullBackOff |

The control plane pod kept failing with this error:

```bash
Back-off pulling image
ErrImagePull: failed to pull and unpack image
no basic auth credentials
```

This suggested the node could not authenticate with ECR.

## First Assumption: Missing Authentication

My first thought was that Kubernetes did not have credentials to access the registry. To fix that, I created a pull secret using the AWS CLI.

```bash
kubectl create secret docker-registry ecr-secret \
  --docker-server=<SERVICE_ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com \
  --docker-username=AWS \
  --docker-password=$(aws ecr get-login-password --region eu-west-2) \
  --namespace dev
```

Then I updated the deployment to reference the secret.

```bash
imagePullSecrets:
  - name: ecr-secret
```

However, the control plane node still failed to pull the image.

## Verifying the Node was the Problem

To confirm whether this was a Kubernetes issue or a node-level issue, I SSH'd into the control plane node and attempted to pull the image directly using containerd.

```bash
sudo ctr image pull <SERVICE_ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com/webappservice:latest
```

I got this error:

```bash
pull access denied
authorization failed: no basic auth credentials
```

This confirmed the issue was node authentication. Interestingly, the worker node worked fine.

At this point, I realized the worker node likely already had the image cached locally, which explained why its pod started successfully.

## Attempt 2: Node-Level Authentication

The next idea was configuring authentication at the node level.

One approach is using the [ECR credential helper](https://github.com/awslabs/amazon-ecr-credential-helper), which automatically fetches tokens from AWS whenever an image pull happens.

However, this turned out to be unreliable in my setup because k3s uses **containerd**, and containerd does not automatically read Docker credential helper configurations. This made the approach more complicated than necessary for a homelab cluster.

## The Simpler and More Reliable Solution

Instead of configuring authentication per node or per deployment, I decided to attach the ECR pull secret to the **default ServiceAccount in the namespace**.

This means any pod created in that namespace automatically inherits the image pull secret. To do this, I followed the steps below:

**Step 1: Create the ECR Pull Secret**

First, I recreated the secret.

```bash
kubectl create secret docker-registry ecr-secret \
  --docker-server=<SERVICE_ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com \
  --docker-username=AWS \
  --docker-password=$(aws ecr get-login-password --region eu-west-2) \
  --namespace dev
```

**Step 2: Attach the Secret to the Namespace ServiceAccount**

Next, I patched the default ServiceAccount in the `dev` namespace.

```bash
kubectl patch serviceaccount default \
  -n dev \
  -p '{"imagePullSecrets": [{"name": "ecr-secret"}]}'
```

Now every pod created in the namespace automatically uses the secret.

**Step 3: Restart the Failing Pod**

I deleted the failing pod, so Kubernetes would recreate it.

```bash
kubectl delete pod -n dev webappservice-hudjfi
```

The deployment created a new pod, and this time it started successfully.

**Verifying the Fix**

Running `kubectl describe pod` confirmed the container started correctly. Key parts of the output looked like this:

```bash
State: Running
Ready: True
```

The event logs also showed:

```bash
Container image already present on machine
Created container
Started container
```

At this point, both nodes were running the application successfully.

## Why I Chose the ServiceAccount Approach

There are a few ways to solve ECR authentication in Kubernetes.

| **Approach**                    | **Pros**                | **Cons**                     |
| ------------------------------- | ----------------------- | ---------------------------- |
| Node credential helper          | Automatic token refresh | Harder to configure with k3s |
| Static registry token           | Simple                  | Tokens expire after 12 hours |
| imagePullSecrets per deployment | Works                   | Repetitive configuration     |
| ServiceAccount imagePullSecret  | Simple and reusable     | Requires secret refresh      |

The **ServiceAccount approach** felt like the best tradeoff for a homelab environment because:

- It centralizes authentication
- It avoids repeating the configuration in every deployment
- It works consistently across nodes

Despite the flexibility the ServiceAccount approach brings, it still has the same problem of 12-hour expiry tokens that come with ECR tokens. Thankfully, this can be managed using a Kubernetes CronJob that refreshes the token periodically.

## Automatically Refreshing the ECR Secret

As mentioned earlier, authentication tokens from ECR expire every 12 hours. This does not affect running pods, but it can break new deployments or pods scheduled on fresh nodes if the secret has expired.

A simple way to solve this in a homelab cluster is to periodically refresh the secret using a Kubernetes CronJob.

![](https://paper-attachments.dropboxusercontent.com/s_88EBE4FB662784CC10B48EE40A4F16D609D3D7F24069A2CADFF36D32EB9F9E94_1772996213878_image.png)

This keeps the cluster working without manual intervention.

**Step 1: Create a ServiceAccount for the Job**

First, I created a ServiceAccount that the CronJob can use.

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
    name: ecr-secret-refresher
    namespace: dev
```

**Step 2: Allow the Job to Manage Secrets**

The job needs permission to update secrets in the namespace.

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
    name: ecr-secret-manager
    namespace: dev
rules:
    - apiGroups: ['']
      resources: ['secrets']
      verbs: ['get', 'create', 'delete']
```

Then bind the role to the ServiceAccount.

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
    name: ecr-secret-manager-binding
    namespace: dev
subjects:
    - kind: ServiceAccount
      name: ecr-secret-refresher
roleRef:
    kind: Role
    name: ecr-secret-manager
    apiGroup: rbac.authorization.k8s.io
```

**Step 3: Create the CronJob**

Before setting up the CronJob, the job container needs AWS credentials to call `ecr get-login-password`. In a managed cloud environment, I would typically handle this with an IAM role, but in a homelab the simplest approach is storing my AWS credentials in a Kubernetes secret and injecting them as environment variables.

```bash
kubectl create secret generic aws-credentials \
  --from-literal=AWS_ACCESS_KEY_ID=<your-access-key-id> \
  --from-literal=AWS_SECRET_ACCESS_KEY=<your-secret-access-key> \
  --namespace dev
```

Then, I created the CronJob that refreshes the secret every 11 hours.

```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
    name: refresh-ecr-secret
    namespace: dev
spec:
    schedule: '0 */11 * * *'
    jobTemplate:
        spec:
            template:
                spec:
                    serviceAccountName: ecr-secret-refresher
                    restartPolicy: OnFailure
                    containers:
                        - name: refresh-secret
                          image: amazon/aws-cli
                          env:
                              - name: AWS_REGION
                                value: eu-west-2
                          command:
                              - /bin/sh
                              - -c
                              - |
                                  TOKEN=$(aws ecr get-login-password --region $AWS_REGION)
                                  kubectl delete secret ecr-secret -n dev --ignore-not-found
                                  kubectl create secret docker-registry ecr-secret \
                                    --docker-server=<ACCOUNT_ID>.dkr.ecr.eu-west-2.amazonaws.com \
                                    --docker-username=AWS \
                                    --docker-password=$TOKEN \
                                    -n dev
```

This job will:

1. Request a fresh login token from ECR
2. Delete the old secret
3. Recreate the secret with the new token

Since the namespace ServiceAccount already references `ecr-secret`, all new pods automatically use the refreshed credentials.

**Applying the Configuration**

I saved the resources in a file and apply them:

```bash
kubectl apply -f cronjob.yaml
```

Then I verified the CronJob with:

```bash
kubectl get cronjobs -n dev
```

![Cronjob](https://paper-attachments.dropboxusercontent.com/s_88EBE4FB662784CC10B48EE40A4F16D609D3D7F24069A2CADFF36D32EB9F9E94_1772997819523_Screenshot+2026-03-08+at+20.23.11.png)

## Final Thoughts

The most misleading part of this issue was that one pod was already running. It made the problem look like an intermittent auth failure when the real explanation was simpler: the worker node had the image cached, so it never needed to authenticate. The control plane node had no cache, no credentials, and nowhere to go.

Once that clicked, the fix was straightforward. Attaching the ECR pull secret to the namespace ServiceAccount means every pod in that namespace inherits it automatically, and the CronJob keeps the credentials fresh without any manual intervention.

If you're running a k3s homelab with images in ECR, this setup is worth the 10 minutes it takes to configure. It is less fragile than managing secrets per deployment and less complex than node-level credential helpers.


How to troubleshoot and pull images from Amazon Elastic Container Registry (ECR) into your Kubernetes

Why One Node Pulled and the Other Didn't: Debugging an ECR Image Pull Failure in Kubernetes Homelab

So you’ve installed a Linux virtual machine (VM). Congratulations! Now you’re staring at the terminal and thinking, _“What’s next?”_ _“How do I move forward?”_ _“What should I install to get the environment ready?”_ You’re not alone. We've all been there. I recently went through the same phase after setting up my Homelab. That’s a setup where you run your own servers, networks, or other infrastructure at home to learn, experiment, or support side projects. I remember wondering what to install and how to get everything running as a proper Homelab or even a lightweight production environment.

In this guide, I’ll walk you through the essential post-installation steps to get your Linux VM ready for deploying, hosting, and building software.

## 1. Initial System Updates and Package Management

Before you get into any complex setup, the first thing you should do is update your VM. This pulls in the latest security fixes, patches, and features. It might not seem exciting, but skipping this step is like leaving your front door unlocked. Outdated software is one of the easiest ways for things to go wrong.

```bash
sudo apt update
sudo apt upgrade -y
```

## 2. Install Essential Tools

With your Linux VM updated, the next step is to install some basic development tools. These will help you access the internet, manage code, and do everyday tasks more easily.

```bash
sudo apt install build-essential git curl wget vim -y
```

Here’s a quick breakdown of what each tool does:

-   `build-essential` is a package that includes tools like `gcc`, `g++`, and `make`, which are needed to compile and build software from source.
-   `git` is used to track changes in your code and collaborate with others on projects.
-   `curl` lets you fetch data from URLs directly from the terminal. It’s great for testing APIs or downloading files.
-   `wget` is another tool for downloading files from the internet. It works well for pulling entire files or even websites.
-   `vim` is a command-line text editor. It’s handy for quickly editing config files or writing code without needing a full-blown IDE.

## 3. User Management: Create a Non-Root User

When you first set up your Linux VM, you’ll usually have access as the root user. That’s convenient because you can do anything on the machine. However, it’s also risky. It's like driving without a seatbelt. The root account should be used only for core administrative tasks. You should create a regular user account for your daily work to stay safe and avoid mistakes.

```bash
# Create a new user
sudo adduser yourusername

# Add user to sudo group (for admin privileges when needed)
sudo usermod -aG sudo yourusername
```

## 3. Set Up SSH and Permission Keys for Remote Access

While using a username and password lets you access your VM, you probably don’t want to start sharing those credentials with applications that need to connect to it. For example, if you’ve set up a GitHub Action to deploy a Next.js app to your VM, using your username and password for remote access is not a good idea. Instead, you should use an SSH key. It gives you a secure way to connect to your VM without relying on a password.

Passwords are more vulnerable to brute force attacks. SSH keys help reduce that risk and give you a safer way to manage access.

To set up SSH, follow the steps below:

**Step 1: Get your VM IP address using your VM terminal**

```bash
ip addr show
```

**Step 2: Generate an SSH key on your local machine**

```bash
ssh-keygen -t ed25519
```

You’ll be asked to enter a name and some optional details. You can press Enter to accept the defaults.

**Step 3: Copy your public key to the VM**

```bash
ssh-copy-id yourusername@your.vm.ip.address
```

To confirm the key was copied successfully, run this command on your VM:

```bash
cat ~/.ssh/authorized_keys
```

**Step 4: Disable password authentication on your VM**

Use `vim` to open the SSH config file

    sudo vim /etc/ssh/sshd_config

Look for the line that says `PasswordAuthentication` and change its value to `no`. Remove the `#` and update the value if the line is commented out.

**Step 5: Restart the SSH service**

```bash
sudo systemctl restart ssh
```

**Step 6: Router Exposure and Port Forwarding**

If you want to access your VM from outside your local network, you’ll need to forward the SSH port (by default, that’s port 22) from your router to your VM. Every router is a bit different, so check your router’s manual or admin panel for how to set up port forwarding.

> **Tip:** You can change the default SSH port to something less common, like `2222`, to avoid drawing unnecessary attention from bots or scanners. Just remember to update the `Port` value in your SSH config file and restart the SSH service afterward.

## 4. Firewall Configuration and Traffic Management

Now that your VM is exposed to the internet, it’s important to lock it down. Without a firewall, it’s like leaving your house wide open with a sign that says "Come on in." You want to let in only what you need and block everything else.

The Uncomplicated Firewall (UFW) makes it easy to manage which traffic is allowed in and out of your VM.

**Step 1: Install and configure UFW**

Update your package list and install UFW.

```bash
sudo apt update && sudo apt install ufw
```

**Step 2: Allow essential services**

Let UFW know which ports you want open. For example:

```bash
sudo ufw allow 2222/tcp  # if you changed SSH port
sudo ufw allow 80/tcp    # for HTTP
sudo ufw allow 443/tcp   # for HTTPS
```

**Step 3: Enable the firewall**

Once your rules are in place, turn the firewall on.

```bash
sudo ufw enable
```

**Step 4: Check your firewall status**

```bash
sudo ufw status verbose
```

## 5. Installing and Configuring Nginx: Your First Web Server

To get a sense of what your VM can do, it’s a great idea to set up Nginx. Nginx is a lightweight, powerful web server that’s perfect for both testing and production environments. Setting it up will show that your VM is ready to serve web content and handle real-world traffic.

**Step 1: Install Nginx**

Start by updating your package list and installing Nginx.

```bash
sudo apt update && sudo apt install nginx
```

**Step 2: Start and enable the Nginx service**

Once installed, start the Nginx service and make sure it starts automatically on boot:

```bash
sudo systemctl start nginx
sudo systemctl enable nginx
```

**Step 3: Test your setup**

Open a browser and go to `http://your-vm-ip`. If everything is set up correctly, you’ll see the Nginx welcome page.

![Nginx welcome page](https://paper-attachments.dropboxusercontent.com/s_D8636135A0B0BB3900626336E0942EFDE38AEFE4AEBF27955658135C7F28048F_1743903396635_Image+06-04-2025+at+02.36.jpg)

## Wrapping Up

Setting up your Linux VM after installation might feel overwhelming at first, but each step is important for making sure your system is secure, efficient, and ready to handle your software deployment and hosting needs.

Keep in mind that system administration isn’t a one-time task. It’s an ongoing process. Make it a habit to review your security settings, keep your software updated, and be ready to adjust as your needs evolve.

Just installed a Linux VM and wondering what to do next? This guide walks you through everything you need to get your VM production-ready—from adding secure users and setting up SSH keys, to configuring firewalls, port forwarding, and even running your first Nginx web server.

Now that you've installed Linux VM, what's next?

At some point in your software development lifecycle (SDLC), you’ll need to work with a database, whether that’s in a development environment, a production environment, or both. Databases are a core part of most applications, and it’s important to host them close to your backend service to reduce latency and improve performance. It's also important to access them securely, both during development and after your application goes live.

In this guide, I’ll walk you through everything you need to get MySQL installed and secured on a Linux virtual machine (VM), and also set it up for remote access.

To get MySQL up and running on your Linux VM, follow the steps below:

## 1. Update VM and Install MySQL

Before anything else, the first thing to do is to update your VM to have the latest security fixes, patches, and features.

```bash
sudo apt update
sudo apt upgrade -y
```

Next, install the MySQL server:

```bash
sudo apt install mysql-server -y
```

## 2. Secure the MySQL Database

Even on a virtual machine, where the attack surface is relatively smaller, you shouldn't leave your database exposed. It's important to secure your installation from day one and reduce potential vulnerabilities by tightening access and applying security measures. To secure your database, run the command below:

```bash
sudo mysql_secure_installation
```

This script will prompt you to configure several security-related settings, like setting a root password, removing anonymous users, disallowing root login remotely, and dropping the test database.

## 3. Create a Database User

Since root login is disabled, you'll need to create a new user with the right permissions to access the database. Depending on your use case, the permissions can be limited to a specific database or granted at a super-user level with access to all available privileges. To do this, follow the steps below:

**Step 1: Access MySQL Shell and Log In**

```bash
sudo mysql -u root -p
```

**Step 2: Create a New User and Grant Permission**

```bash
CREATE USER 'remote_user'@'%' IDENTIFIED BY 'your_secure_password';
GRANT ALL PRIVILEGES ON *.* TO 'remote_user'@'%' WITH GRANT OPTION;
```

> Note that the `%` **wildcard** allows connections from any host. For stronger security, you can replace `%` with a specific IP address or a range. You can also adjust the **privileges** to apply only to a specific database or limit them to certain operations, depending on what the user needs access to.

**Step 3: Apply the Changes and Exit**

```bash
FLUSH PRIVILEGES;
EXIT;
```

## 4. Allow Remote Access to your Database

By default, MySQL only listens on the local network. That setup works fine if everything happens within the VM, but in most cases, you’ll be connecting through a workbench or a backend service. To allow remote access, follow the steps below:

**Step 1: Locate and Open the MySQL Config File**

```bash
sudo nano /etc/mysql/mysql.conf.d/mysqld.cnf
```

**Step 2: Bind Address**

Look for the line that says:

```bash
bind-address = 127.0.0.1
```

Change it to:

```bash
bind-address = 0.0.0.0
```

> Use **Ctrl O,** hit **Enter** to save, and **Ctrl X** to exit.

## 5. Configure Firewall for MySQL Access

By default, your VM will only allow traffic through ports that you've explicitly opened. Since MySQL uses port `3306`, you'll need to open this port so remote clients can connect to your database over the network.

```bash
sudo ufw allow 3306/tcp
sudo ufw reload
```

## 6. Connect to MySQL Remotely

With your MySQL server configured and the firewall set up, you can now connect to it remotely using any database client of your choice. To do this, do the following:

1. **Open a database client like MySQL Workbench on your local machine.**
2. **Create a New Connection:**
    - **Connection Name:** Give it a friendly name.
    - **Hostname:** Use your Linux VM’s public IP address. You can get it by running the command `hostname -I`.
    - **Port:** Ensure the port is set to 3306.
    - **Username:** Enter `remote_user` (or your chosen username).
    - **Password:** Click "Store in Vault" or similar, and input your secure password.
3. **Test the Connection:**
   Click the “Test Connection” button to make sure everything is working.

## Wrapping Up

This guide walks you through setting up a MySQL database on a Linux VM. Each step helps get your server up and running while keeping your data secure and accessible when you need it. Whether you're working on a simple project or deploying a production-ready database, following these best practices will help you use MySQL with confidence and efficiency.

If you're new to working with Linux VMs, you might find this [beginner’s guide on setting up a Linux virtual machine](https://fullstackwriter.dev/post/now-that-you-ve-installed-linux-vm-what-s-next?category=Linux) helpful before diving deeper into server and database management.


A step-by-step guide to installing MySQL on a Linux VM, securing it, and setting up remote access using a database client.

How to Install and Connect to MySQL on your Linux VM

I run k3s in my [homelab](https://fullstackwriter.dev/post/how-i-installed-kubernetes-on-my-homelab-using-k3s-on-proxmox-ubuntu?category=kubernetes), and I wanted to expose my apps to the internet without opening multiple ports or configuring complex router rules.

I had the options of NodePort or installing MetalLB. Although both approaches work, neither felt close to how things are done in production. NodePort would mean exposing a port on my router and mapping it to a node. That exposes my public IP and opens inbound traffic into my network. MetalLB is cleaner inside Kubernetes, but it still assumes you have public IP handling and proper routing at the edge.

What I wanted instead was something closer to how modern infrastructure works, that supports encrypted transport, no inbound ports, and clean DNS control.

That is why I chose Cloudflare Tunnel. The goal for me was to use my domain `demolamalomo.xyz` registered on AWS Route 53. I wanted something very close to production that caters to the following:

- Apps accessible at `*.demolamalomo.xyz` (e.g., `whoami.demolamalomo.xyz`, `grafana.demolamalomo.xyz`)
- No router port forwarding
- No public IP exposure
- Cloudflared running as Kubernetes pods, not installed on the host

## Step 1: Move the Domain to Cloudflare DNS

My first instinct was to delegate only a subdomain like `k3s.demolamalomo.xyz` to Cloudflare using NS records in Route 53. That way my root domain stays fully in Route 53 and only the Kubernetes subdomain gets managed by Cloudflare.

That didn’t work. Cloudflare's free plan does not support adding subdomain zones. You can only add root domains like `demolamalomo.yxz`. On testing, this didn’t work because anytime I dig through the nameserver, the nameservers returned are still the ones tied to Route 53.

So I moved the entire domain's DNS to Cloudflare by following the steps below:

- In the Cloudflare Dashboard, I clicked the + **Add** button, selected the **Connect a domain** option, and filled in the required field.

![Connect a domain](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774118683934_Screenshot+2026-03-04+at+01.27.15.png)
![Fill the form](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774118683991_Screenshot+2026-03-04+at+01.38.39.png)

- Selected the **Free** plan
- Cloudflare scanned and imported my existing Route 53 DNS records automatically
- Cloudflare assigned me two nameservers similar to `****.**.cloudflare.com` and `******.**.cloudflare.com`

Then I updated Route 53 to point to those nameservers.

**The First Mistake: Changing Nameservers in the Wrong Place**
Route 53 has two places where nameservers appear, and I changed the wrong one.

**Wrong place:** Route 53 >> Hosted zones >> demolamalomo.xyz >> NS record. This is just a DNS record inside the zone. Changing it does nothing to where the internet actually looks for your domain.
**Right place:** Route 53 >> **Registered domains** >> demolamalomo.xyz >> **Name servers**. This is what tells the TLD registry where to send DNS queries for your domain.

I spent time debugging why `dig clohea.net NS @1.1.1.1` still returned AWS nameservers before realizing I had changed the wrong thing. Once I updated the registered domain's nameservers, Cloudflare showed the domain as **Active**, and DNS started resolving correctly.

## Step 2: Create the Cloudflare Tunnel

In the Cloudflare Zero Trust [dashboard](https://one.dash.cloudflare.com), I created a tunnel following the steps below:

- Navigate to **Networks >> Connectors**
- Click **Create a tunnel**
- Select **Cloudflared** as the connector type

![Select Cloudflared](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774128112097_Screenshot+2026-03-21+at+21.39.42.png)

- Named it `k3s-tunnel`
- On the connector install page, copied the **tunnel token** starting with `eyJ...`
- Then, I installed Cloudflared on my machine with the token copied.
  sudo cloudflared service install eyJ......

Once Cloudflared is installed, the status shows `HEALTHY` in my dashboard.

Next, I proceeded to the next page to configure the subdomain and service (Traefik in my case)

![Configured subdomain](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774128103478_Screenshot+2026-03-21+at+22.21.07.png)

## Step 3: Deploy Cloudflared on Kubernetes

I chose to run **cloudflared** as Kubernetes pods rather than installing it on the host machine. This keeps everything inside the cluster and makes it easy to manage with standard Kubernetes tooling.

To do this, first, I created a namespace and stored the tunnel token I got earlier as a secret:

```bash
kubectl create namespace cloudflare

kubectl create secret generic cloudflared-token \
  --namespace cloudflare \
  --from-literal=token=<TUNNEL_TOKEN>
```

Then, I used the secret to create a deployment:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
    name: cloudflared
    namespace: cloudflare
spec:
    replicas: 2
    selector:
        matchLabels:
            app: cloudflared
    template:
        metadata:
            labels:
                app: cloudflared
        spec:
            containers:
                - name: cloudflared
                  image: cloudflare/cloudflared:latest
                  args:
                      - tunnel
                      - --no-autoupdate
                      - run
                  env:
                      - name: TUNNEL_TOKEN
                        valueFrom:
                            secretKeyRef:
                                name: cloudflared-token
                                key: token
                  resources:
                      requests:
                          memory: '128Mi'
                          cpu: '100m'
                      limits:
                          memory: '256Mi'
                          cpu: '200m'
```

This creates two replicas of the cloudflared instance, and the `TUNNEL_TOKEN` environment variable tells cloudflared to run in remotely-managed mode (the tunnel configuration lives in Cloudflare's dashboard, not in a local config file).

After applying, I verified the pods were connected:

```bash
kubectl logs -n cloudflare -l app=cloudflared --tail=20
```

I saw `Registered tunnel connection` messages with `protocol=quic`, confirming the tunnel was live.

## Step 4: Deploy a Test App

I deployed the `traefik/whoami` image as a simple test:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
    name: whoami
    namespace: default
spec:
    replicas: 1
    selector:
        matchLabels:
            app: whoami
    template:
        metadata:
            labels:
                app: whoami
        spec:
            containers:
                - name: whoami
                  image: traefik/whoami:latest
                  ports:
                      - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
    name: whoami
    namespace: default
spec:
    selector:
        app: whoami
    ports:
        - port: 80
          targetPort: 80
```

Also added the Ingress rule telling Traefik how to route based on the Host header:

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
    name: whoami-ingress
    namespace: default
spec:
    rules:
        - host: whoami.demolamalomo.xyz
          http:
              paths:
                  - path: /
                    pathType: Prefix
                    backend:
                        service:
                            name: whoami
                            port:
                                number: 80
```

**The 502 Problem**
At this point, I expected everything to work with the DNS resolved. The tunnel was connected. But when I hit `https://whoami.demolamalomo.xyz`, I got a **502 Bad Gateway** from Cloudflare.

This was the most confusing problem because:

- The cloudflared pods were healthy and connected
- The tunnel logs showed no request-level errors
- From a debug pod inside the cluster, curling `http://traefik.kube-system.svc.cluster.local:80` with the correct Host header returned a perfect 200 response

The issue was **DNS resolution inside the cloudflared container**. My tunnel config originally pointed to:

```bash
http://traefik.kube-system.svc.cluster.local:80
```

The cloudflared Docker image is built on scratch, an essentially empty container with nothing but the cloudflared binary, with no `shell`, `cat` , `curl` , or `/etc/resolv.conf`. When cloudflared tried to resolve `traefik.kube-system.svc.cluster.local`, it had no idea which DNS server to ask and failed silently. The request never reached Traefik, so Cloudflare returned a `502`.

I confirmed this by switching the tunnel config to use **Traefik's ClusterIP directly**:

```bash
http://10.43.250.137:80
```

Immediately, I ran curl `https://whoami.clohea.net` , and it returned the whoami response with Cloudflare headers confirming traffic was flowing through the tunnel.

**Why the ClusterIP Approach Is Fine**
In a homelab, the Traefik service ClusterIP is stable. It only changes if you delete and recreate the service. For a more robust solution, you could add explicit DNS config to the cloudflared deployment:

```yaml
spec:
    template:
        spec:
            dnsPolicy: 'ClusterFirst'
            dnsConfig:
                nameservers:
                    - 10.43.0.10 # kube-dns ClusterIP
                searches:
                    - kube-system.svc.cluster.local
                    - svc.cluster.local
                    - cluster.local
```

But for a homelab, the ClusterIP approach is simpler and works.

**Adding New Apps**
The best part about this setup is that adding a new app requires zero changes to Cloudflare. The wildcard `*.demolamalomo.xyz` already covers every subdomain. I just need to:

- A Deployment and Service for my app
- An Ingress with the right hostname

The architecture looks like this:

![](https://paper-attachments.dropboxusercontent.com/s_6C008F9FC02BD5FE8DB0288473F17FC153FDA96D1DFEDF46DA3E10796FF36870_1774133078724_image.png)

## Lessons Learned

- **Route 53 has two places for nameservers.** The hosted zone NS record is not the same as the registered domain's nameserver setting. Change the wrong one and nothing happens. Change it under **Registered domains** >> **Name servers**.
- **The Cloudflared Docker image cannot resolve Kubernetes DNS names.** It is built on a scratch base image with no `/etc/resolv.conf`. Either use the target service's ClusterIP directly or add explicit `dnsConfig` to the pod spec.
- **Ingress host names must match exactly.** If your tunnel routes to `*.clohea.net` but your Ingress says `whoami.k3s.clohea.net`, Traefik will not match the request and you will get a 502.


I wanted production-style ingress for my k3s homelab without exposing ports or my public IP. What looked simple turned into debugging Route 53 nameservers, Kubernetes DNS resolution, and mysterious Cloudflare 502s before everything finally worked.

How I Exposed My Kubernetes Homelab to the Internet Using Cloudflare Tunnel

If you work with Docker, you’ve probably run into this at some point.

You spin up a Postgres or MySQL container for a project. Everything is running fine. Then you want to look at the data. Maybe a migration behaved strangely, maybe you want to insert a quick test row, or maybe you just want to confirm what’s actually in the database.

That’s when things get annoying.

You either drop into the terminal and start typing `psql` commands, install a desktop tool like pgAdmin and wait for it to load, or spin up another heavy container just to get a UI. It always feels like more effort than it should be just to inspect your own data.

I kept running into this friction, and eventually I decided to stop working around it and build something that fits how I actually use Docker.

That’s how **DockAdmin** started.

## What is DockAdmin?

DockAdmin is a lightweight database admin UI that runs directly inside your Docker environment. You add it to your compose file, run your stack, and immediately get a clean browser interface for managing your database.

```yaml
services:
    postgres:
        image: postgres:16
        environment:
            POSTGRES_USER: admin
            POSTGRES_PASSWORD: admin
            POSTGRES_DB: myapp

    dockadmin:
        image: demlabz/dockadmin
        ports:
            - '3000:3000'
```

No extra configuration, no setup wizard, and no external tools to install. It’s just another container that gives you a database UI.

It supports PostgreSQL, MySQL, and SQLite, which covers most everyday development use cases.

## Why Build Another Database Admin Tool?

There are already plenty of tools out there, but most of them don’t fit naturally into a Docker-first workflow.

Desktop tools are powerful, but they live outside your container environment. That means installing them, configuring connections, and maintaining them separately. That feels excessive when you only need to inspect a few rows or quickly test something.

[Adminer](https://www.adminer.org/en/) comes much closer to what I wanted. It’s lightweight, runs in a container, and uses database credentials for authentication, which keeps things simple. But the interface feels dated, and the overall experience doesn’t match what modern developer tooling looks like today.

[phpMyAdmin](https://www.phpmyadmin.net/) is also container-friendly, but it’s MySQL-only and carries a lot of historical baggage.

What I really wanted was something simple like Adminer, but with a modern UI, proper multi-database support, and a design that feels native to Docker from the start. That combination didn’t really exist, so I built DockAdmin.

## Why Rust on the backend?

Choosing Rust was a practical decision.

The full DockAdmin Docker image is around 15 MB, which is smaller than many `node_modules` directories. Rust compiles to a single static binary, so the runtime container only needs Alpine Linux and the compiled executable. There’s no interpreter, no runtime dependency chain, and no garbage collector to manage.

The backend uses Axum as the web framework and SQLx for database access. Axum is fast and ergonomic, and SQLx provides compile-time query checking with async support across Postgres, MySQL, and SQLite.

Here’s the Dockerfile:

```Dockerfile
# Frontend Build
FROM node:22-alpine AS frontend
WORKDIR /ui
COPY ui/package*.json ./
RUN npm install
COPY ui .
RUN npm run build

# Backend Build
FROM rust:1-alpine AS backend
WORKDIR /backend
COPY backend/Cargo.toml backend/Cargo.lock ./
RUN mkdir src && echo "fn main() {}" > src/main.rs
RUN cargo build --release
COPY backend/src ./src
RUN touch src/main.rs
RUN cargo build --release
RUN strip target/release/backend

# Runtime
FROM alpine:latest
COPY --from=backend /backend/target/release/backend ./dockadmin
COPY --from=frontend /ui/dist ./ui
EXPOSE 3000
CMD ["./dockadmin"]
```

It’s a straightforward multi-stage build that compiles the frontend, compile the backend, and copy both into a minimal runtime container. Stripping debug symbols trims the binary further, and the final image starts quickly and uses very little memory.

## The Frontend Stack

The UI is built with **React 19**, **TypeScript**, and **shadcn/ui** components on top of **Radix UI** and **Tailwind CSS**. Routing is handled by **TanStack Router**, and the SQL editor uses **CodeMirror 6**.

The goal was to make the interface feel like a tool you’d actually enjoy using, not just something functional. shadcn/ui helped a lot here because it provides polished components without forcing a rigid design system.

The frontend compiles into static files that the Rust backend serves directly, so there’s no separate frontend server at runtime.

## What You Can Do With It

Here's a quick rundown of the features in the current release (v0.1.0):

**Connect to Your Database**

Open `localhost:3000`, pick your database type, enter your credentials, and you're in. When running inside Docker Compose, you use the container name as the host (e.g., `postgres` or `mysql`). That's the whole auth flow. Your database credentials are your login, the same approach Adminer uses, and it works really well.

![Auth page](https://paper-attachments.dropboxusercontent.com/s_F7A81FD18698A2A0DAA2E49AE47E9EE5806BC2A0C98ACE1794ED3FBA3C1217AC_1771168431303_auth.png)

**Browse and Edit Data**

Click on any table and you'll see your data in a clean, paginated grid. You can scroll through records, and get a feel for what's in there.

Need to change or add something? Use the CRUD buttons to add, edit, or delete a row.

![CRUD operation](https://paper-attachments.dropboxusercontent.com/s_F7A81FD18698A2A0DAA2E49AE47E9EE5806BC2A0C98ACE1794ED3FBA3C1217AC_1771168583152_add_row.png)

**Create Tables Visually**

You can create new tables without writing any SQL. The table builder lets you define columns, set data types, and toggle constraints like primary key, not null, and unique. You can also set up foreign key relationships to other tables, including ON DELETE and ON UPDATE cascade rules.

![Create Table](https://paper-attachments.dropboxusercontent.com/s_F7A81FD18698A2A0DAA2E49AE47E9EE5806BC2A0C98ACE1794ED3FBA3C1217AC_1771168634065_create_table.png)

**Explore Your Schema**

The schema viewer gives you a quick look at any table's structure. You can check the column names, data types, constraints, and foreign key relationships. It's handy when you need to understand how a database is set up without digging through migration files.

![Schema view](https://paper-attachments.dropboxusercontent.com/s_F7A81FD18698A2A0DAA2E49AE47E9EE5806BC2A0C98ACE1794ED3FBA3C1217AC_1771168698838_schema_view.png)

**Write Raw SQL**

For those times when a UI just won't cut it, there's a built-in SQL editor powered by CodeMirror. It has syntax highlighting, and you can run any query and see results right there. Great for quick debugging, running one-off scripts, or testing complex queries.

![SQL Editor](https://paper-attachments.dropboxusercontent.com/s_F7A81FD18698A2A0DAA2E49AE47E9EE5806BC2A0C98ACE1794ED3FBA3C1217AC_1771168745174_sql_editor.png)

## What I learned Building It

Supporting multiple databases consistently was harder than expected. Postgres, MySQL, and SQLite all expose schema information differently. Listing tables, reading constraints, handling relationships. Each database has its own quirks, and building one abstraction layer across all of them required more edge-case handling than I initially planned.

Docker image optimization was another learning curve. The first image I built was over 100 MB. Getting it down to around 15 MB involved multi-stage builds, Alpine base images, dependency caching, and stripping the binary. Each change helped a little, but together they made a huge difference.

I also learned how important open-source infrastructure is from the beginning. Issue templates, contribution guides, CI/CD pipelines, and automated workflows take time to set up, but they make collaboration much smoother once people start contributing.

## What’s Next

DockAdmin is currently in active development, and there’s a clear roadmap:

- Data export to SQL and CSV
- File import
- Full-text search
- Multi-database sessions

These are all areas where community feedback and contributions can shape what it becomes.

## Try It Out

You can run DockAdmin in seconds:

```bash
docker run -p 3000:3000 demlabz/dockadmin
```

Or add it to your Docker Compose setup:

```yaml
dockadmin:
    image: demlabz/dockadmin
    ports:
        - '3000:3000'
```

The project is open source under MIT and available on [GitHub](https://github.com/Mr-Malomz/dockadmin).

If you find it useful, star the [repo](https://github.com/Mr-Malomz/dockadmin). If something breaks, open an [issue](https://github.com/Mr-Malomz/dockadmin/issues). And if you want to contribute, check the [good-first-issue](https://github.com/Mr-Malomz/dockadmin/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22good%20first%20issue%22) label and the contributing guide.

The stack is Rust and React, so there’s plenty to explore on either side.

Thanks for reading.


DockAdmin is a lightweight, modern database admin UI that runs as a Docker container. It supports PostgreSQL, MySQL, and SQLite. The entire image is ~15MB.

DevOps

Why One Node Pulled and the Other Didn't: Debugging an ECR Image Pull Failure in Kubernetes Homelab

Now that you've installed Linux VM, what's next?

How to Install and Connect to MySQL on your Linux VM

How I Exposed My Kubernetes Homelab to the Internet Using Cloudflare Tunnel

I Built a Docker-Native Database Admin UI in Rust. Here’s Why.

blog

Contact