Mohammad Abu Mattar | DevTips

Terraform Workspaces vs. Directory-Based Environments: What Actually Scales

Mohammad Abu Mattar — Wed, 19 Aug 2026 13:50:21 GMT

Why this choice matters

Hey, want to stop sweating every prod apply?

The way you split dev, staging, and prod in Terraform decides how much damage a single mistake can do. Get it right and a bad apply is annoying. Get it wrong and the same command that fixes dev can wreck prod, because they share too much.

It is a decision you make early and live with

Most teams pick an environment strategy on day one, before they have a prod worth protecting, and never revisit it. By the time it hurts, there is real state to migrate. So it is worth understanding the trade-off now.

The problem with workspaces at scale

What's the issue?

Terraform workspaces share a single backend and split your state by name. You run terraform workspace select prod, and the same root config now points at the prod state. The config is identical across environments; only the state differs.

That is exactly what makes them risky. Every environment runs the same .tf files, so there is no room for prod to legitimately differ from dev, and switching environments is a single command with no folder to remind you where you are.

The real-world consequence

The classic incident is running terraform apply thinking you are in dev while the workspace is still set to prod. Nothing in the file tells you which one you are on. Add per-environment differences (a bigger instance in prod, an extra region) and you start bolting count and var.environment conditionals into shared code until it is a tangle nobody wants to touch.

A folder per environment

Here's how to fix it

Give each environment its own directory with its own backend and its own state. envs/dev/, envs/staging/, envs/prod/, each initialized separately. Now the blast radius stops at one folder. You cannot accidentally apply prod from the dev directory, and each environment can differ honestly without conditionals smeared through shared code.

Implementing it without copy-paste

The obvious objection is duplication. If every folder has its own config, are you not repeating yourself? That is what modules are for. The real resources live in modules/, and each environment folder is a thin wrapper that calls them with its own variables.

Tools and platforms

Terragrunt is the common answer for keeping the directory approach DRY. It generates the backend config per environment and lets each folder stay tiny, so you get isolated state without copy-pasting backend blocks. Plain Terraform with modules works too; Terragrunt just removes the last of the boilerplate.

Quick implementation steps

Quick takeaways

Workspaces: one backend, state split by name, config shared. Fine when small.
Directories: one folder and one state per environment. Scales because the blast radius is contained.
Migrate one environment at a time, dev first and prod last.
Trust the move only when terraform plan shows zero changes.

Migrating without breaking things

You do not have to do this all at once. Move one environment at a time, starting with dev, using state pull to grab the current state and state push (or targeted import) to load it into the new per-folder backend.

Benefits you feel quickly

Why it helps

The payoff is a smaller blast radius. When prod has its own folder and its own state, there is no single command that flips you into prod by accident, and a mistake in the dev folder simply cannot reach it.

Cleaner diffs and real differences

Because environments no longer share one config, per-environment differences become explicit and readable instead of hidden behind conditionals. Your prod folder says what prod is, in plain HCL, and a code review shows exactly which environment a change touches.

What's your approach?

Community discussion

What's your take? Are you still running everything through workspaces, or have you moved to a folder per environment? I am curious where the line is for other teams.

Share your experience

If you have migrated off workspaces, I would love to hear how the state pull / state push dance went and whether Terragrunt earned its place in your setup.

GitHub Actions Secrets and Environment Variables: Handle Config the Right Way

Mohammad Abu Mattar — Sat, 08 Aug 2026 13:50:18 GMT

Why secrets handling matters

Most CI leaks are config mistakes, not attacks

Hey, want to stop leaking credentials in your pipelines? Most secret leaks in CI are not the result of some clever attacker. They happen because a key got pasted into a plain environment variable, echoed into a log, or left sitting in repo settings for two years with no rotation. GitHub Actions gives you good tools to avoid all of that, but only if you use them on purpose. Handling config the right way is mostly about scoping secrets tightly and never letting them touch a log.

A workflow runs with real access

A workflow is a program that runs with production credentials. It talks to your cloud, your registry, and your deploy targets, often with credentials that can do real damage. Anyone who can open a pull request can trigger workflows, and anyone with repo access can read your logs and artifacts. That means the way you store and pass secrets is a security boundary, not a convenience setting.

The problem with sloppy config

What's the issue?

The usual pattern is to dump every secret into repository settings and reference them everywhere. Long-lived AWS keys, database passwords, and API tokens all live in one flat pile with no scope. Then someone echoes a variable to debug a failing step, or passes a secret to a job as a plain artifact, and now that value is sitting in the log output where it stays for as long as the run is retained.

Real-world consequences

Once a secret lands in a log or an unmasked output, treat it as compromised. Logs get shared in bug reports, artifacts get downloaded, and forks can sometimes see more than you expect. Long-lived credentials make it worse because a leaked key stays valid until someone remembers to rotate it, which is usually after the incident. A single careless echo can mean an emergency key rotation across every service that used it.

Scope, mask, and go short-lived

Here's how to fix it

Fixing this comes down to three habits. Scope secrets so each one is only visible where it is actually needed, mask any sensitive value so it never renders in a log, and replace long-lived cloud keys with OIDC so your workflow gets a short-lived token instead of a permanent credential. Do those three things and most of the ways a secret can escape are closed.

Implementing it

Start with scope. Repository secrets are for values shared across the whole repo, while environment secrets are tied to a specific environment like production and can sit behind required reviewers. For cloud access, use OIDC instead of stored keys.

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    permissions:
      id-token: write # required for OIDC
      contents: read
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/deploy
          aws-region: us-east-1

No AWS keys are stored anywhere. The workflow requests an OIDC token, AWS trusts it, and hands back short-lived credentials that expire when the job ends. If you do generate a secret at runtime, mask it right away with echo "::add-mask::$TOKEN" so it shows up as *** in the log.

Tools and platforms

The core tools are built into GitHub Actions: repository secrets, environment secrets with protection rules, and the ::add-mask:: workflow command. For cloud auth, the official aws-actions/configure-aws-credentials, google-github-actions/auth, and azure/login actions all support OIDC. For a deeper audit, tools like gitleaks or trufflehog can scan your history for secrets that already slipped through.

Quick implementation steps

Quick takeaways to lock down your config:

Use repository secrets for shared values and environment secrets for per-stage keys.
Put sensitive environments behind required reviewers for a manual gate.
Swap long-lived cloud keys for OIDC with a scoped IAM role.
Mask any runtime-generated secret with ::add-mask:: before using it.
Pass secrets between jobs through masked outputs, never plain artifacts.
Pass secrets into composite actions as explicit inputs.

Mind the composite action gap

Composite actions do not automatically inherit the secrets of the workflow that calls them. If your composite action needs a token, you have to pass it in as an input from the caller. Forgetting this leads to confusing empty values, and the fix is not to loosen anything, just to wire the secret through explicitly.

Never echo to debug

When a step fails, the temptation is to print the variable to see what it holds. Do not do that with anything sensitive. Use ::add-mask:: first, or check the length and a hash instead of the raw value. A masked value stays masked even if you accidentally print it later in the same run.

Benefits of doing it right

Why it helps

You shrink the blast radius of any single mistake. Scoped secrets mean a leaked value only affects one environment. OIDC means there is no permanent key to steal in the first place, since tokens expire in minutes. Masking means a careless log line does not turn into an incident. Each habit is small, but together they take most credential leaks off the table.

Less rotation, less panic

Long-lived keys are a standing liability that someone has to remember to rotate. OIDC removes that chore entirely for cloud access, because there is nothing stored to rotate. Environment protection rules add a human checkpoint before production secrets are ever used, so a bad change cannot quietly deploy itself. The result is fewer 2am rotations and a lot less guessing about who could have seen what.

What's your approach?

Community discussion

What's your take? Secrets handling is one of those things that feels fine until the day it very much is not. If you have moved a pipeline from stored cloud keys to OIDC, how did the rollout go, and did it simplify your rotation story as much as you hoped?

Share your experience

If you have a favorite pattern for scoping secrets across a lot of environments, or a tool that caught a leak before it shipped, I would love to hear it. Especially how you handle secrets in reusable and composite actions without turning every caller into boilerplate.

References

Docker Multi-Stage Builds: Smaller, Safer Images for Production

Mohammad Abu Mattar — Mon, 27 Jul 2026 16:00:07 GMT

Why multi-stage builds matter

Image size is really about what is inside

Hey, want to stop shipping a toolshed to production? If your Dockerfile builds and runs the app in one stage, your final image is carrying everything you used to build it: the compiler, the package manager caches, the dev dependencies, all of it. None of that runs in production, but all of it ships anyway. That means slower pulls, slower cold starts, and a lot more code that a scanner has to worry about. Multi-stage builds let you build with all your tools, then throw the tools away and keep only the finished app.

Build tools and runtime are different jobs

Building your app and running it need completely different things. The build needs compilers, headers, and dev packages. The runtime just needs your binary or bundled code and maybe a couple of shared libraries. Mixing the two into one image is the root of most bloated, insecure containers. Separating them is the whole idea here.

The problem with single-stage Dockerfiles

What's the issue?

A single-stage Dockerfile does everything in one place. You start from a full base image like node:20 or golang:1.22, install dependencies, compile or bundle, and that same fat image becomes what you deploy. So your production container includes gcc, npm, git, and every dev dependency you only needed for a few seconds during the build. You are shipping a build machine and calling it a runtime.

Real-world consequences

The costs pile up quietly. Images balloon to a gigabyte or more, which slows every deploy and every autoscale event. Worse, every extra package is another thing that can have a CVE. When your scanner flags fifty vulnerabilities, most of them are usually in build tools your app never even calls at runtime. You end up patching things that should not have been in the image in the first place.

The multi-stage fix

Here's how to fix it

A multi-stage build splits your Dockerfile into named stages using FROM ... AS. You do all the heavy work in a build stage, then start a fresh, minimal final stage and pull across only the artifact you need with COPY --from. Everything left behind in the build stage, the compilers and dev dependencies, never makes it into the image you ship.

Implementing it

Here is a Go example. The first stage compiles the binary, and the second stage starts from a tiny base and copies just that binary.

# Build stage: has the full Go toolchain
FROM golang:1.22 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /app ./cmd/server

# Production stage: minimal, no compiler
FROM gcr.io/distroless/static
COPY --from=build /app /app
ENTRYPOINT ["/app"]

The final image has no Go toolchain, no shell, and no package manager. It is just your binary on a distroless base. The same pattern works for Node: build and bundle in a node:20 stage, then copy the dist folder and production node_modules into a slim runtime stage.

Tools and platforms

BuildKit is the modern Docker builder and it runs independent stages in parallel and caches them well, so multi-stage builds stay fast. For final bases, reach for distroless images from Google or an alpine variant when you need a shell. Pair the build with a scanner like Trivy or Grype so you can see the vulnerability count drop after you strip the build tools out.

Quick implementation steps

Quick takeaways to slim down your images:

Split your Dockerfile into a build stage and a final stage with FROM ... AS.
Do all compiling and bundling in the build stage.
Start the final stage from a minimal base like distroless or alpine.
Copy only the finished artifact across with COPY --from.
Enable BuildKit so stages build in parallel and cache well.
Scan the final image and compare the size and CVE count before and after.

Copy only what runs

The trick is being strict about what crosses the COPY --from line. Bring over the compiled binary, the bundled assets, and production dependencies only. Leave source files, test files, and dev dependencies in the build stage where they belong. If you are not sure something is needed at runtime, it probably is not.

Measure the before and after

Run docker images to see the size difference, and run your scanner on both versions. It is not unusual to go from over a gigabyte down to under a hundred megabytes, with the vulnerability count dropping right along with it. That comparison is the easiest way to convince a team the change is worth it.

Benefits of multi-stage builds

Why it helps

You get a production image that only contains what it needs to run. That means faster pulls, faster cold starts, and a much smaller attack surface. With no compilers, package managers, or dev dependencies inside, there is simply less for an attacker to use and less for a scanner to flag.

Faster and safer deploys

Smaller images move faster through your whole pipeline. Registries store less, nodes pull quicker, and autoscaling responds sooner because there is less to download before a pod starts. On the security side, a distroless final stage with no shell makes a compromised container far harder to pivot from, since there is barely anything in there to run.

What's your approach?

Community discussion

What's your take? Multi-stage builds are one of those changes that feel small but pay off every single deploy. If you have converted an old single-stage Dockerfile, how much smaller did the image get, and did your scan results improve?

Share your experience

If you are running multi-stage builds in production, I would love to hear which final base you settled on. Distroless, alpine, or something custom, and how you handle the cases where you still need a shell for debugging.

References

ArgoCD GitOps: Sync Kubernetes Deployments Automatically from Git

Mohammad Abu Mattar — Wed, 22 Jul 2026 15:18:26 GMT

Why GitOps for Kubernetes?

From kubectl apply to Git as the source of truth

Hey, want to stop deploying to Kubernetes by hand? If your releases still come from someone running kubectl apply on their laptop, you already know the downsides. Nobody is quite sure what's actually running, changes are hard to audit, and a rollback means remembering what the old YAML looked like. GitOps flips that around. You keep all your manifests in Git, and a controller in the cluster continuously makes the live state match what's in the repo. Git becomes the one place that describes what should be running.

What GitOps actually means

The whole model rests on one habit. You never change the cluster directly. You change Git, and the cluster follows. That means every deploy is a commit, every rollback is a revert, and your audit trail is just the repository history. ArgoCD is one of the most popular tools that make this real for Kubernetes.

The problem with manual and push-based deploys

What's the issue?

Push-based pipelines hand your CI system cluster-admin credentials and let it run kubectl apply or helm upgrade from the outside. That works until it doesn't. The credentials live in your CI, which is a juicy target, and CI only touches the cluster when a pipeline runs. In between, nothing is watching whether the cluster still matches what you think you deployed.

Real-world consequences

The quiet killer is drift. Someone hotfixes a Deployment live during an incident, forgets to put it in Git, and now the repo and the cluster disagree. The next pipeline either clobbers the fix or silently leaves the difference in place. Multiply that across a few engineers and three environments and nobody can answer the basic question: what is actually running right now, and why.

The ArgoCD approach

Here's how to fix it

ArgoCD runs inside the cluster and pulls, rather than being pushed to. It watches a Git repo, compares the desired state there against the live state in the cluster, and reconciles the two. Because it lives in the cluster, it does not need to hand external systems admin credentials, and because it runs continuously, it notices drift instead of waiting for the next pipeline.

Implementing it

Install ArgoCD with Helm, then point it at a repo by creating an Application.

helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd --namespace argocd --create-namespace

An Application tells ArgoCD which repo and path to watch and where to deploy it.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/your-org/deploy.git
    targetRevision: main
    path: envs/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: web
  syncPolicy:
    automated:
      selfHeal: true # revert manual changes back to Git
      prune: true # delete resources removed from Git

Tools and platforms

ArgoCD is part of the Argo project alongside Argo Rollouts for progressive delivery. Flux is the other well-known GitOps controller if you prefer a more CLI-first model. Both pair naturally with Helm and Kustomize, so you keep the templating you already use.

Quick implementation steps

Quick takeaways to get GitOps running:

Put your manifests or Helm charts in a Git repo, one path per environment.
Install ArgoCD in the cluster with Helm.
Create an Application pointing at the repo path.
Turn on automated sync so merges deploy on their own.
Turn on selfHeal so live edits snap back to Git.
Turn on prune so deleting from Git deletes from the cluster.

Sync policies: manual vs automatic

Start in manual sync while you build trust. ArgoCD shows you the diff and you click sync. Once you trust it, switch on automated so a merge to main deploys itself. selfHeal is what closes the drift gap, because any change made directly to the cluster is reverted to match Git within minutes.

Multiple environments

Give each environment its own path (or its own repo) and its own Application. Promotion becomes a pull request that moves a change from envs/staging to envs/prod, so the same reviewed manifests flow through each stage.

Benefits of GitOps with ArgoCD

Why it helps

You get one honest answer to "what's running," because the answer is always "whatever is in Git." Rollbacks stop being scary, since reverting a commit reverts the deploy. And you shrink your attack surface, because the cluster pulls its own config instead of trusting an external pipeline with admin keys.

Confidence and recovery

The health and sync dashboard turns a fuzzy question into a clear status per app and per environment. When something drifts or degrades, you see it, and self-heal often fixes it before anyone notices.

What's your approach?

Community discussion

What's your take? GitOps is a real shift in how a team thinks about deploys, and the move from push to pull takes some getting used to. If you have made the switch, what finally sold you on it?

Share your experience

If you are running ArgoCD or Flux in production, I would love to hear how you handle promotion between environments and whether you let self-heal run everywhere or hold it back for some workloads.

References

Kubernetes Namespaces: Organize, Isolate, and Secure Multi-Team Clusters

Mohammad Abu Mattar — Thu, 28 May 2026 00:00:00 GMT

Why cluster isolation matters

The multi-tenant reality

If you're running a separate cluster for every environment and every dev team, you have already seen the bill and the amount of upgrade work that comes with it. Sharing a single cluster is a lot like staying in a hotel. Everyone gets their own secure, private room in the same building. You share the plumbing and foundation, but your space is entirely yours. You pack more workloads onto the same nodes, logging and monitoring live in one place, and there is one control plane to upgrade instead of thirty.

To get started with logical separation, you can declare a namespace with a few clear labels to keep things organized:

# namespace-basic.yaml
# A simple, labeled namespace to partition our cluster
apiVersion: v1
kind: Namespace
metadata:
  name: team-frontend
  labels:
    team: frontend
    environment: production
    managed-by: platform-team

Soft vs. hard multi-tenancy

Decide which model you are building before you write any manifests. Soft multi-tenancy is enough for trusted internal teams in the same company, where you only need logical boundaries. Hard multi-tenancy is what you need for untrusted external users or regulated workloads, and it costs more: dedicated node pools with taints and tolerations, and sometimes a sandboxed runtime like gVisor or Kata Containers so a host kernel exploit stays inside the sandbox.

The problem with default-allow clusters

The illusion of isolation

Most teams think that simply creating different namespaces for different teams keeps things isolated. It doesn't. Out of the box, Kubernetes is designed as an open, single-tenant system, and a namespace is really just a logical boundary for the API. Resource names can overlap across namespaces, which is handy for keeping things organized, but the scheduling and network layers stay wide open by default.

Real-world collateral damage

Without controls, one runaway batch job in a staging namespace can take all the memory on a shared node. That triggers the Out-of-Memory (OOM) killer, and the kernel does not care that the pod it picks belongs to production next door. Because pods allow all traffic by default, a compromised container in your frontend namespace can port-scan and query a database in your backend namespace. The worst of the three is a single tenant flooding the API server with thousands of Secrets or ConfigMaps until etcd runs out of storage and the control plane stops answering for everyone.

Three real failure modes from a default-allow cluster: an OOM kill that takes down a neighbor's production pods, a compromised frontend pod that pivots straight to a backend database, and one tenant exhausting etcd by creating thousands of Secrets. All preventable with the controls below.

Layered namespace isolation

Stacking the controls

No single setting protects a shared cluster. Isolation is something you build one control at a time, and it takes all five: namespaces, ResourceQuotas, LimitRanges, RBAC, and NetworkPolicies.

The table below shows how these controls work together for real defense-in-depth:

Control Type	Scope	Core Enforcement Mechanism	Mitigated Risk	Failure Mode if Omitted
Namespace	Logical / API	API Server name scoping	Naming collisions and basic management sprawl	Inability to separate administrative concerns
ResourceQuota	Namespace total	Admission controller validation	Cluster-wide resource starvation and etcd storage exhaustion	A single runaway tenant exhausts whole cluster capacity
LimitRange	Individual pod/container	Admission controller injection	Single container monopolizing namespace resources	Pods without resource declarations are rejected or run unbounded
NetworkPolicy	Pod network	Container Network Interface (CNI)	Lateral movement and cross-namespace port scanning	Full network reachability; compromised pods attack any internal target
RBAC Roles	Identity / Access	API authorization engine	Unauthorized credential exploit and cross-tenant tampering	Attackers exploit cluster-wide credentials to compromise all workloads

Resource control

To stop "noisy neighbors" from taking over your cluster, apply a ResourceQuota to every namespace. That sets a hard limit on the total CPU, memory, and object counts a team can use. There is a catch. Once a quota exists, the API server rejects any pod that doesn't explicitly state its own resource requests and limits, which breaks deploys for every team that hasn't retrofitted its manifests. A LimitRange fixes that by injecting default values at admission time when a developer forgets to set them.

A ResourceQuota on its own will reject any pod that doesn't declare its own requests/limits, which silently breaks deploys for any team that hasn't retrofitted their manifests. Pair every Quota with a LimitRange so missing values get sane defaults injected automatically.

Here's how to set up a ResourceQuota to keep resource usage in check:

# compute-quota.yaml
# Caps the total resources used by all pods in this namespace
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: team-frontend
spec:
  hard:
    requests.cpu: '4'
    requests.memory: 8Gi
    limits.cpu: '8'
    limits.memory: 16Gi

Pair that quota with a container-level LimitRange in the same namespace to establish default fallback values:

# container-limits.yaml
# Automatically injects resource defaults for containers that don't declare them
apiVersion: v1
kind: LimitRange
metadata:
  name: container-limits
  namespace: team-frontend
spec:
  limits:
    - type: Container
      default:
        cpu: 500m
        memory: 256Mi
      defaultRequest:
        cpu: 100m
        memory: 128Mi

Network and access paths

To block lateral movement, change the network default from "allow-all" to "deny-all". A default-deny NetworkPolicy that matches every pod shuts down unauthorized traffic immediately. Then you open only the paths you trust. Allow DNS resolution and traffic from your ingress controller explicitly, or your apps cannot resolve internal services and nothing reaches them from outside.

Here's your baseline default-deny policy to secure a namespace:

# default-deny-all.yaml
# Shuts down all incoming and outgoing network traffic by default
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: team-frontend
spec:
  podSelector: {} # An empty selector matches every pod in the namespace
  policyTypes:
    - Ingress
    - Egress

Once a default-deny Egress policy is active, pods can no longer reach CoreDNS, and every service lookup starts failing in confusing ways. Always pair the deny with an allow-DNS policy in the same change.

Once everything is blocked, add a policy to allow DNS resolution so your pods can find other services:

# allow-dns.yaml
# Selectively allows outbound DNS queries to CoreDNS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: team-frontend
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector: {} # Matches any namespace hosting the DNS pods
      ports:
        - protocol: UDP
          port: 53

For control plane access, keep roles scoped to namespaces with Role and RoleBinding rather than cluster-wide bindings. A team that can only see its own namespace cannot delete someone else's Deployment by pasting the wrong context.

Tools and platforms

You can automate this whole setup with Terraform and declare namespaces, quotas, and network policies alongside the rest of your infrastructure. If you need Layer 7 filtering or traffic you can actually watch flowing, a CNI like Cilium gives you both. For high-security workloads, gVisor or Kata Containers put a user-space kernel between the container and the host, so a breakout lands in the sandbox.

Quick implementation steps

Step-by-step hardening

Here's the checklist for hardening a shared environment:

Deploy a default-deny NetworkPolicy in all non-system namespaces to shut down unauthorized lateral traffic.
Set up a global LimitRange to automatically assign safe CPU and memory fallback defaults.
Enforce a ResourceQuota to cap total namespace consumption and keep your etcd storage from getting exhausted.
Keep developer access strictly within their designated namespace boundaries using group-based RBAC bindings.
Use minimal base images like Alpine or container sandboxes to shrink your host-level attack surface.

Namespaces in Terraform

Manage namespaces as code so every one of them ends up with the same controls. Here's a Terraform block that provisions a namespace and attaches its resource quota in the same apply:

# main.tf
# Provisions a team namespace and immediately pairs it with a resource quota

resource "kubernetes_namespace" "team_backend" {
  metadata {
    name = "team-backend"
    labels = {
      team        = "backend"
      environment = "production"
      managed-by  = "terraform"
    }
  }
}

resource "kubernetes_resource_quota" "backend_quota" {
  metadata {
    name      = "backend-quota"
    namespace = kubernetes_namespace.team_backend.metadata.name
  }
  spec {
    hard = {
      "requests.cpu"    = "4"
      "requests.memory" = "8Gi"
      "limits.cpu"      = "8"
      "limits.memory"   = "16Gi"
    }
  }
}

Automating default policies

Use Kustomize or a GitOps pipeline to roll out the default-deny policy and the quota with every new namespace. Otherwise the namespace somebody created by hand six months ago is still sitting there wide open.

Benefits of layered isolation

Predictable performance and security

Getting namespace isolation right pays off in two places. The bill goes down, because workloads consolidate onto fewer nodes and you no longer need a control plane per team. And your ops team monitors one cluster instead of thirty, which is the difference between an upgrade being a Tuesday and an upgrade being a quarter. The third benefit shows up later: once the defaults are codified, developers can create their own isolated staging namespace without waiting on an approval, because the guardrails come with it.

A smaller blast radius

A layered setup shrinks your blast radius if things go sideways. Even if a container gets compromised, the attacker is stuck inside a locked room. They can't access other tenants' data, query neighboring services, or starve the rest of the cluster of resources.

What's your approach?

Community discussion

The trade-off never fully goes away. Every control here makes the cluster safer and makes somebody's first deploy fail in a way they did not expect. Where have you landed on that?

Share your experience

Do you like managing your namespaces and quotas through Terraform, or do you rely on dynamic operators to do the heavy lifting? Have you ever run into a case where a default-deny network policy accidentally blocked something critical?

References

Helm Charts: Templating & Multi-Environment Kubernetes Deployments

Mohammad Abu Mattar — Mon, 30 Mar 2026 20:02:59 GMT

Why Helm matters

The Kubernetes manifest problem

Managing Kubernetes manifests at scale becomes a nightmare.

You have a deployment for dev, staging and production. Each one is 90% identical: same containers, same services, different replicas and resource limits and environment variables. Copy the YAML, edit a few lines, deploy. That works right up until someone edits dev and forgets staging, or a typo lands in production and nobody notices until users report the outage.

Helm packages a Kubernetes application as a chart with templates, values and a version number. The YAML you deploy is generated from one source instead of maintained in three places.

Where hand-written YAML breaks down

Environment Fragmentation:
prod-deployment.yaml  ← Different values hardcoded
staging-deployment.yaml  ← Needs 3 edits for prod
dev-deployment.yaml  ← Inconsistent structure

Result:
• Configurations drift
• Changes in one place aren't replicated
• Rollback = manually revert files
• New environments = copy-paste hell

The fix: Helm charts

What Helm is

Helm is package management for Kubernetes, like npm for Node.js or pip for Python.

Charts: Helm packages containing templated Kubernetes manifests
Values: Configuration that gets injected into templates (replicas, image tags, resources)
Releases: Deployed instances of charts, tracked with versions for easy rollback
Repos: Central repositories where teams share charts

What you get

Single chart, multiple environments: Use template variables instead of duplicating YAML
Templating system: {{ .Values.replicas }} → substituted with values from env-specific file
Dependency management: Charts can depend on other charts (PostgreSQL, Redis, etc.)
Versioning & rollback: Every deployment tracked, instant rollback to previous version
Validation: Helm validates charts before deployment
GitOps ready: Store charts in Git, deploy from Git

Chart structure

Directory layout

my-app/
├── Chart.yaml                 # Chart metadata (name, version, description)
├── values.yaml                # Default values
├── values-dev.yaml            # Dev environment overrides
├── values-staging.yaml        # Staging overrides
├── values-prod.yaml           # Production overrides
├── templates/
│   ├── deployment.yaml        # Templated Kubernetes Deployment
│   ├── service.yaml           # Service manifest
│   ├── configmap.yaml         # ConfigMap for config
│   ├── hpa.yaml               # Horizontal Pod Autoscaler (prod only)
│   └── _helpers.tpl           # Template helpers/functions
└── README.md                  # Chart documentation

Chart.yaml

apiVersion: v2
name: my-app
description: A Helm chart for my microservice
type: application
version: 1.0.0 # Chart version
appVersion: '1.2.3' # Application version
maintainers:
  - name: Your Team
    email: team@example.com

Templating

Basic values templating

# Default values for all environments
replicaCount: 1

image:
  repository: myregistry.azurecr.io/my-app
  tag: '1.2.3'
  pullPolicy: IfNotPresent

resources:
  requests:
    memory: '128Mi'
    cpu: '100m'
  limits:
    memory: '256Mi'
    cpu: '500m'

environment: 'dev'
debug: true

# Production overrides
replicaCount: 3 # More replicas for load

resources:
  requests:
    memory: '512Mi'
    cpu: '500m'
  limits:
    memory: '1Gi'
    cpu: '2000m'

environment: 'production'
debug: false # Disable debug logging

A templated deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{.Release.Name}}
  namespace: {{.Release.Namespace}}
spec:
  replicas: {{.Values.replicaCount}} # Injected from values
  selector:
    matchLabels:
      app: {{.Chart.Name}}
  template:
    metadata:
      labels:
        app: {{.Chart.Name}}
    spec:
      containers:
        - name: {{.Chart.Name}}
          image: '{{ .Values.image.repository }}:{{ .Values.image.tag }}'
          imagePullPolicy: {{.Values.image.pullPolicy}}
          env:
            - name: ENVIRONMENT
              value: {{.Values.environment}}
            - name: DEBUG
              value: '{{ .Values.debug }}'
          ports:
            - containerPort: 8080
          resources:
            requests:
              memory: {{.Values.resources.requests.memory | quote}}
              cpu: {{.Values.resources.requests.cpu | quote}}
            limits:
              memory: {{.Values.resources.limits.memory | quote}}
              cpu: {{.Values.resources.limits.cpu | quote}}

Conditional logic in templates

Environment-specific configuration

spec:
  {{- if eq .Values.environment "production" }}
  replicas: 3
  {{- else if eq .Values.environment "staging" }}
  replicas: 2
  {{- else }}
  replicas: 1
  {{- end }}

Resources that exist only in some environments

{{- if eq .Values.environment "production" }}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: {{ .Release.Name }}
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ .Release.Name }}
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
{{- end }}

Conditional security settings

spec:
  {{- if eq .Values.environment "production" }}
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    fsReadOnlyRootFilesystem: true
  {{- end }}

Managing multiple environments

Environment-specific values files

Deploy to dev:
helm install my-app . -f values.yaml -f values-dev.yaml

Deploy to staging:
helm install my-app . -f values.yaml -f values-staging.yaml

Deploy to prod:
helm install my-app . -f values.yaml -f values-prod.yaml

Values priority (the last one wins)

# Later files override earlier ones
helm install my-app \
  -f values.yaml \           # Base defaults
  -f values-prod.yaml \      # Override for prod
  --set environment=production # Override from CLI (highest priority)

Chart dependencies

Depending on PostgreSQL

dependencies:
  - name: postgresql
    version: '13.0.0'
    repository: https://charts.bitnami.com/bitnami

Installing dependencies

# Download dependencies
helm dependency update

# Then deploy (PostgreSQL chart auto-installs)
helm install my-app . -f values-prod.yaml

Values for a sub-chart

# My app values
replicaCount: 3

# PostgreSQL sub-chart values
postgresql:
  enabled: true
  auth:
    password: 'prod-secure-password'
  primary:
    persistence:
      size: 100Gi
  metrics:
    enabled: true

Release management

The basic commands

# Install a release
helm install my-app ./chart -f values-prod.yaml

# Upgrade to new version
helm upgrade my-app ./chart -f values-prod.yaml

# Check release history
helm history my-app

# Rollback to previous version (instant!)
helm rollback my-app 2

# Delete release
helm uninstall my-app

A deployment run, start to finish

# 1. Make changes to chart
# 2. Test locally
helm install test-release . -f values-dev.yaml

# 3. Test succeeded, upgrade
helm uninstall test-release

# 4. Deploy to production
helm upgrade my-app . -f values-prod.yaml

# 5. Verify
kubectl get pods -l app=my-app

# 6. If something's wrong, instant rollback
helm rollback my-app

GitOps integration

Storing charts in Git

git repository structure:
├── charts/
│   ├── my-app/
│   │   ├── Chart.yaml
│   │   ├── values.yaml
│   │   ├── values-dev.yaml
│   │   ├── values-prod.yaml
│   │   └── templates/
│   └── other-app/
├── .gitignore
└── README.md

GitOps with ArgoCD

ArgoCD watches your Git repo and automatically keeps your Kubernetes cluster in sync:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app-prod
spec:
  project: default
  source:
    repoURL: https://github.com/myteam/infrastructure
    targetRevision: main
    path: charts/my-app
    helm:
      values: |
        environment: production
        replicaCount: 3
  destination:
    server: https://kubernetes.default.svc
    namespace: prod
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Deploy workflow:

1. Developer updates Chart in Git
2. Commits and pushes
3. ArgoCD detects change
4. Automatically syncs to Kubernetes
5. Helm applies new deployment
6. Services updated with zero downtime

Going further with templates

Helper functions

{{- define "my-app.labels" -}}
helm.sh/chart: {{ include "my-app.chart" . }}
app.kubernetes.io/name: {{ include "my-app.name" . }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}

{{- define "my-app.name" -}}
{{ .Chart.Name }}
{{- end }}

# Reuse helper
metadata:
  labels: {{- include "my-app.labels" . | nindent 4}}

Loops

env:
{{- range $key, $value := .Values.env }}
- name: {{ $key }}
  value: {{ $value | quote }}
{{- end }}

Best practices

1. Version charts semantically

version: 2.1.0 # MAJOR.MINOR.PATCH
# MAJOR: Breaking changes
# MINOR: New features
# PATCH: Bug fixes

2. Use namespaces

# Deploy each environment to different namespace
helm install app . -n production -f values-prod.yaml
helm install app . -n staging -f values-staging.yaml
helm install app . -n dev -f values-dev.yaml

3. Lint before you deploy

# Validate chart syntax
helm lint ./chart

# Dry-run to see what will be deployed
helm install --dry-run my-app ./chart -f values-prod.yaml

4. Test charts

helm test my-app  # Run chart tests

5. Validate values against a schema

{
  '$schema': 'https://json-schema.org/draft-07/schema#',
  'type': 'object',
  'properties':
    {
      'replicaCount': {'type': 'integer', 'minimum': 1},
      'image': {'type': 'object', 'properties': {'tag': {'type': 'string'}}},
    },
}

A complete minimal example

Chart structure

my-service/
├── Chart.yaml
├── values.yaml
├── values-prod.yaml
└── templates/
    ├── deployment.yaml
    └── service.yaml

Deploying it

# Dev
helm install my-service . -f values.yaml -f values-dev.yaml

# Production
helm upgrade my-service . -f values.yaml -f values-prod.yaml --install

# Rollback if needed
helm rollback my-service

Wrapping up

Helm replaces four copies of a deployment with one chart and four values files.

Templated manifests, values per environment and numbered releases mean a rollback is one command instead of an archaeology session in Git history. Pair it with ArgoCD and the cluster follows the repo. The templating language is the price of admission, and it is a real cost, but it buys you environments that actually match.

Resources

Structured Logging & Log Aggregation with ELK Stack

Mohammad Abu Mattar — Sat, 21 Mar 2026 00:00:00 GMT

Why centralized logging matters

When services fail, where do you look first?

In a distributed system, logs scatter across servers, containers and regions. One request might touch five services. When it breaks, you're opening log files on several machines, without the context to connect them, and losing whatever the restarted container was holding.

Centralized logging puts all of it in one searchable index, with the fields you need to correlate one request across services.

What poor logging costs you

Slow debugging: 30+ minutes to find what went wrong 5 minutes ago
Lost logs: Container restarts = logs disappear and are never recovered
No correlation: Can't trace a request across multiple services
Manual hunting: SSH + grep through millions of lines
No alerting: You wake up to customer complaints, not alerts

The problem: distributed logs

Why per-server logs aren't enough

# Server 1: /var/log/app.log
2026-03-21 10:15:23 Error: Database connection refused

# Server 2: /var/log/app.log (you don't see this for 15 minutes)
2026-03-21 10:15:22 Error: Database connection refused

# Server 3: Combined, these tell a story, but:
# - They're on 3 different machines
# - You can't search them together
# - Container restart and logs are gone
# - You have no context (which user? which request?)

The fix: the ELK stack

What ELK is

Elasticsearch: Distributed search and analytics engine. Stores logs as searchable documents with full-text indexing.
Logstash: Log processing pipeline. Collects, parses, enriches, and routes logs to Elasticsearch.
Kibana: Visualization and exploration platform. Query logs with SQL-like syntax, build dashboards, set alerts.

What you get

Centralized: All logs in one place, searchable in milliseconds
Scalable: Handles billions of logs without slowdown
Structured: JSON-based searching and filtering
Correlated: Trace requests across multiple services
Persistent: No data loss when services restart
Alertable: Triggered notifications on patterns

How the pieces fit together

Services → Filebeat/Logstash → Elasticsearch ← Kibana (Query/Visualize)
 ↓           ↓                    ↓
App logs    Parse, enrich        Index, store, analyze
DB logs     Filter, route        Full-text search
System logs Add context          Real-time updates

Getting started with ELK

Docker Compose setup

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    container_name: elasticsearch
    environment:
      discovery.type: single-node
      xpack.security.enabled: false
      xpack.security.transport.ssl.enabled: false
    ports:
      - '9200:9200'
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data

  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.0
    container_name: kibana
    ports:
      - '5601:5601'
    environment:
      ELASTICSEARCH_HOSTS: http://elasticsearch:9200
    depends_on:
      - elasticsearch

  logstash:
    image: docker.elastic.co/logstash/logstash:8.11.0
    container_name: logstash
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    ports:
      - '5000:5000'
    environment:
      discovery.seed_hosts: elasticsearch
      LS_JAVA_OPTS: '-Xmx256m -Xms256m'
    depends_on:
      - elasticsearch

volumes:
  elasticsearch-data:

Start the stack:

docker-compose up -d
# Kibana available at http://localhost:5601
# Elasticsearch at http://localhost:9200

Structured logging with JSON

Why structure the log line

// Good: Structured (searchable, filterable)
{"timestamp": "2026-03-21T10:15:23Z", "service": "user-api", "level": "ERROR", "message": "Database connection failed", "user_id": 42, "request_id": "req-abc-123", "error_code": "DB_CONN_REFUSED", "retry_count": 3}

// Bad: Unstructured (exact string matching only)
"2026-03-21 10:15:23 ERROR [user-api] Database connection failed for user 42 in request req-abc-123"

Logging from your application

Python:

import json
import logging
from pythonjsonlogger import jsonlogger

# Configure JSON logging
logHandler = logging.StreamHandler()
formatter = jsonlogger.JsonFormatter()
logHandler.setFormatter(formatter)
logger = logging.getLogger()
logger.addHandler(logHandler)
logger.setLevel(logging.INFO)

# Use logging with context
logger.info("User login", extra={
    "user_id": 42,
    "request_id": "req-abc-123",
    "service": "user-api",
    "ip_address": "192.168.1.1"
})

logger.error("Database connection failed", extra={
    "user_id": 42,
    "request_id": "req-abc-123",
    "service": "user-api",
    "error_code": "DB_CONN_REFUSED",
    "retry_count": 3
})

Node.js:

import winston from 'winston';

const logger = winston.createLogger({
  format: winston.format.json(),
  defaultMeta: {service: 'api-gateway'},
  transports: [new winston.transports.Console()],
});

// Log with context
logger.info('User authenticated', {
  user_id: 42,
  request_id: 'req-abc-123',
  ip_address: '192.168.1.1',
});

logger.error('Database connection failed', {
  user_id: 42,
  request_id: 'req-abc-123',
  error_code: 'DB_CONN_REFUSED',
  retry_count: 3,
});

Logstash configuration

A basic pipeline

input {
  tcp {
    port => 5000
    codec => json
  }

  # Read from files
  file {
    path => "/var/log/app/*.log"
    codec => json
  }
}

filter {
  # Parse and enrich logs
  if [service] == "api-gateway" {
    mutate {
      add_field => { "service_tier" => "frontend" }
    }
  }

  # Extract request ID from logs for correlation
  grok {
    match => { "message" => "request_id=%{NOTSPACE:request_id}" }
  }

  # Add timestamp if missing
  date {
    match => [ "timestamp", "ISO8601" ]
    target => "@timestamp"
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "logs-%{+YYYY.MM.dd}"
  }

  # Also output to stdout for debugging
  stdout {
    codec => rubydebug
  }
}

Parsing logs from several services

input {
  tcp {
    port => 5000
    codec => json
  }
}

filter {
  # Normalize service names
  translate {
    field => "service"
    destination => "service_normalized"
    dictionary => {
      "user-api" => "user-service"
      "user_api" => "user-service"
      "users" => "user-service"
    }
  }

  # Add environment if not present
  if ![environment] {
    mutate {
      add_field => { "environment" => "production" }
    }
  }

  # Parse error stack traces
  if [level] == "ERROR" and [stack_trace] {
    mutate {
      split => { "stack_trace" => "\n" }
    }
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "logs-%{environment}-%{+YYYY.MM.dd}"
  }
}

Querying logs in Kibana

Creating index patterns

In Kibana:

Go to Stack Management → Index Patterns
Create pattern: logs-* (matches logs-2026.03.21, etc.)
Set timestamp field to @timestamp

Basic searches

# Find all ERROR logs
level: ERROR

# Errors in specific service
level: ERROR AND service: "user-api"

# Errors for specific user
level: ERROR AND user_id: 42

# Errors in time range (last 1 hour)
level: ERROR AND @timestamp: [now-1h TO now]

# Request tracing across services
request_id: "req-abc-123"

More of the Kibana Query Language (KQL)

# Multiple conditions
service: "user-api" AND level: "ERROR" AND response_time_ms > 1000

# Wildcard matching
service: "user-*" AND message: "*connection*"

# Range queries
http_status_code: [400 TO 599] AND @timestamp: [now-1d/d TO now]

# Logical operators
(service: "payment-api" OR service: "billing-api") AND level: "ERROR"

# Exists
error_trace:*

Building dashboards

A monitoring dashboard

Dashboard: "Microservices Health"

1. **Error Rate Panel** (Line chart)
   - Query: level: "ERROR"
   - Group by: service (X-axis), time (series)
   - Show: errors per minute

2. **Response Time Panel** (Bar chart)
   - Query: All logs
   - Metric: avg(response_time_ms)
   - Breakdown by: service

3. **Top Errors Panel** (Table)
   - Query: level: "ERROR"
   - Top 10: error_code

4. **Request Volume Panel** (Metric)
   - Query: All logs
   - Show: total request count

Setting up alerts

Alert: error rate spike

# In Kibana: Stack Management → Alerting → Create Rule

Condition:
  When: average(level: "ERROR") is greater than 100
  For: the last 5 minutes

Action:
  Webhook: POST to Slack channel
  Message: "Error rate spiked in production"

Alert: a specific error pattern

Condition:
  When: count(error_code: "DB_CONN_REFUSED") is greater than 10
  For: the last 2 minutes

Action:
  Send to PagerDuty
  Message: "Database connection failures detected"

Log retention

Index lifecycle management (ILM)

{
  "policy": "logs-policy",
  "phases": {
    "hot": {
      "min_age": "0d",
      "actions": {
        "rollover": {
          "max_primary_store_size": "50GB",
          "max_age": "1d"
        }
      }
    },
    "warm": {
      "min_age": "7d",
      "actions": {
        "set_replicas": {
          "number_of_replicas": 1
        }
      }
    },
    "cold": {
      "min_age": "30d",
      "actions": {
        "searchable_snapshot": {
          "snapshot_repository": "my_repository"
        }
      }
    },
    "delete": {
      "min_age": "90d",
      "actions": {
        "delete": {}
      }
    }
  }
}

Request tracing with correlation IDs

Adding a request ID

from fastapi import Request
import uuid
import logging

logger = logging.getLogger(__name__)

async def add_request_id(request: Request, call_next):
    # Generate or extract request ID
    request_id = request.headers.get("X-Request-ID") or str(uuid.uuid4())

    # Store in request state
    request.state.request_id = request_id

    # Log with correlation
    logger.info("Request started", extra={
        "request_id": request_id,
        "method": request.method,
        "path": request.url.path
    })

    response = await call_next(request)

    # Add to response headers for client
    response.headers["X-Request-ID"] = request_id

    return response

Passing the request ID between services

# When calling another service
import httpx

async def call_user_service(request):
    request_id = request.state.request_id

    async with httpx.AsyncClient() as client:
        response = await client.get(
            "http://user-api/users/42",
            headers={"X-Request-ID": request_id}  # Pass it along
        )

    return response.json()

Best practices

1. Log the right amount

# Good: Structured context without redundancy
logger.info("Payment processed", extra={
    "user_id": 42,
    "request_id": "req-abc",
    "amount": 99.99,
    "currency": "USD"
})

# Bad: Too verbose
logger.info(f"User with ID 42 has processed a payment of 99.99 USD via request req-abc at {timestamp}")

2. Use the same field names everywhere

// Across all services, use same field names
{
  "timestamp": "2026-03-21T10:15:23Z",
  "level": "ERROR",
  "service": "user-api",
  "user_id": 42,
  "request_id": "req-abc"
}

3. Add context to errors

try:
    result = db.query(...)
except Exception as e:
    logger.error("Database query failed", extra={
        "error_type": type(e).__name__,
        "error_message": str(e),
        "query": query,  # What failed?
        "user_id": user_id,  # Who was affected?
        "request_id": request_id  # Trace it
    })

4. Plan your indices

# Keep recent data hot (highly available)
# Archive old data (cost-effective)
# Delete after retention period

Daily indices: logs-2026.03.21, logs-2026.03.22
Retention: 90 days hot + searchable, 1 year archival, then delete

Wrapping up

One searchable index turns a half-hour of log hunting into a query.

The part that pays for itself is structured JSON with a request ID in every line. Do that before you touch Kibana, because a centralized pile of unstructured strings is still a pile. Add OpenTelemetry traces alongside it and you have both halves: logs for what happened inside a service, traces for the path between them.

Resources

Container Image Vulnerability Scanning in CI/CD with Trivy

Mohammad Abu Mattar — Tue, 10 Mar 2026 00:00:00 GMT

Why container security matters

Where the vulnerabilities hide

A container image is one of the largest pieces of untrusted code you ship.

Every image you build carries the base OS layer, runtime libraries, your dependencies and your application code. Any of those layers can hold known CVEs, and most of them you didn't write. Without scanning, that whole stack reaches production unread.

The numbers

80% of container images in production contain at least one known vulnerability
Supply chain attacks targeting container registries are increasing
Unpatched container vulnerabilities lead to data breaches and service disruptions

The challenge

Why manual review isn't enough

Nobody is going to read the dependency tree of every image on every build. Without automation, a vulnerable image ships, and you find out about it from a CVE feed or an incident rather than from the pipeline that built it.

Common vulnerabilities in containers

Outdated base images with unpatched OS vulnerabilities
Vulnerable dependencies pulled in from npm, pip or Maven
Exposed secrets accidentally included in image layers
Misconfigurations creating insecure defaults
Malware hidden in supply chain attacks

The fix: Trivy

What Trivy is

Trivy is a fast container vulnerability scanner from Aqua Security. It scans container images, filesystems and configuration files for known vulnerabilities, misconfigurations and secrets.

Why I reach for Trivy

Speed: Scans images in seconds, not minutes
Accuracy: Supports multiple vulnerability databases (NVD, GitHub Security, Aqua, Alpine)
Broad coverage: Detects OS vulnerabilities, application dependencies, and misconfigurations
Zero setup: Works out of the box without complex configuration
CI/CD ready: Integrates easily into GitHub Actions, GitLab CI, Jenkins
Open-source: Free, transparent, and community-driven

Installation and setup

Installing Trivy

# macOS
brew install trivy

# Linux (Ubuntu/Debian)
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | apt-key add -
echo "deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" | tee -a /etc/apt/sources.list.d/trivy.list
apt-get update
apt-get install trivy

# Docker
docker pull aquasec/trivy

Basic image scanning

# Scan a local image
trivy image my-app:latest

# Scan from registry
trivy image nginx:latest

# Scan with detailed output
trivy image --severity HIGH,CRITICAL my-app:latest

Setting severity thresholds

Scanning at specific severity levels

# Only show critical and high severity issues
trivy image --severity CRITICAL,HIGH my-app:latest

# Exit with error code if vulnerabilities found
trivy image --severity HIGH,CRITICAL --exit-code 1 my-app:latest

Output formats

# JSON output for parsing
trivy image --format json my-app:latest

# SARIF format for GitHub integration
trivy image --format sarif my-app:latest

# Table format (default)
trivy image --format table my-app:latest

CI/CD integration

GitHub Actions workflow

name: Container Vulnerability Scan

on:
  push:
    branches: [main]
    paths:
      - 'Dockerfile'
      - 'src/**'
  pull_request:
    branches: [main]

jobs:
  trivy-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: Build Docker image
        uses: docker/build-push-action@v4
        with:
          context: .
          file: ./Dockerfile
          push: false
          load: true
          tags: my-app:${{ github.sha }}

      - name: Run Trivy vulnerability scan
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: my-app:${{ github.sha }}
          format: 'sarif'
          output: 'trivy-results.sarif'
          severity: 'CRITICAL,HIGH'

      - name: Upload Trivy results to GitHub Security
        uses: github/codeql-action/upload-sarif@v2
        with:
          sarif_file: 'trivy-results.sarif'

      - name: Fail if critical vulnerabilities found
        run: |
          trivy image --severity CRITICAL my-app:${{ github.sha }} --exit-code 1

GitLab CI

stages:
  - build
  - scan

build:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

scan:
  stage: scan
  image: aquasec/trivy:latest
  script:
    - trivy image --severity HIGH,CRITICAL --exit-code 1 $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  allow_failure: false

Policy enforcement

Creating a Trivy policy

# Define what constitutes a vulnerability violation
severity: HIGH,CRITICAL

# Ignore specific CVEs for known/accepted risks
ignorefile: .trivyignore

# Policy for failing builds
exit-code: 1

# Require sign-off for medium severity
medium-requires-approval: true

Ignoring false positives

# Format: CVE-XXXX-XXXXX [optional: expiration date]

# Known false positive or acceptable risk (expires 2026-12-31)
CVE-2024-1234 2026-12-31

# Permanently ignore (use with caution)
CVE-2024-5678

Beyond images

Scanning filesystems

# Scan local directory
trivy fs .

# Scan with detailed output
trivy fs --severity HIGH,CRITICAL --format json . > fs-scan.json

Scanning configuration files

# Detect misconfigurations in Dockerfile
trivy config Dockerfile

# Scan Kubernetes manifests
trivy config k8s-manifests/

Generating a Software Bill of Materials (SBOM)

# Generate SBOM in CycloneDX format
trivy image --format cyclonedx my-app:latest > sbom.xml

# Generate SBOM in SPDX format
trivy image --format spdx my-app:latest > sbom.spdx

Remediation

When vulnerabilities are found

Smallest change: update the base image

# Before
FROM ubuntu:20.04

# After
FROM ubuntu:22.04

Targeted change: update the vulnerable dependency

FROM node:18-alpine

# Install with security patches
RUN npm install --no-save my-package@latest

Last resort: rebuild the image without cache

docker build --no-cache -t my-app:latest .

A scheduled scan across every image

name: Production Container Security

on:
  schedule:
    # Run daily scans
    - cron: '0 2 * * *'
  workflow_dispatch:

jobs:
  scan-all-images:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        image:
          - my-app:latest
          - api-gateway:latest
          - worker-service:latest

    steps:
      - uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ matrix.image }}
          format: 'json'
          output: 'trivy-${{ matrix.image }}.json'
          severity: 'CRITICAL,HIGH,MEDIUM'

      - name: Archive results
        uses: actions/upload-artifact@v3
        with:
          name: trivy-reports
          path: trivy-*.json

      - name: Notify security team
        if: failure()
        run: |
          curl -X POST -H 'Content-type: application/json' \
            --data '{"text":"Critical vulnerabilities found in ${{ matrix.image }}"}' \
            ${{ secrets.SLACK_WEBHOOK_URL }}

Monitoring and reporting

Storing results over time

# Generate timestamped reports
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
trivy image --format json my-app:latest > reports/scan-$TIMESTAMP.json

Tracking vulnerability trends

#!/bin/bash
# Count vulnerabilities by severity
trivy image --format json my-app:latest | \
  jq '[.Results[]?.Vulnerabilities[]?.Severity] | group_by(.) | map({severity: .[0], count: length})'

Best practices

1. Scan early and often

Scan during development (local images)
Scan in CI/CD pipeline (before merge)
Scan in registry (continuous monitoring)
Scan in production (runtime detection)

2. Use minimal base images

# Reduce attack surface
FROM alpine:3.18 as base
FROM gcr.io/distroless/base-debian11

3. Update dependencies regularly

# Update dependencies regularly
npm audit fix --force
python -m pip install --upgrade pip

4. Keep SBOMs

Generate and store SBOMs for supply chain transparency:

trivy image --format cyclonedx my-app:latest > sbom.json
git add sbom.json
git commit -m "Update SBOM for security tracking"

Registry integration

Push only images that passed

# Only push if scan passes
trivy image --severity CRITICAL,HIGH --exit-code 1 my-app:latest && \
  docker push my-registry/my-app:latest

Wrapping up

If you ship containers, something has to scan them before the registry does.

Trivy is the cheapest way I know to do that. Add it to the pipeline, pick the severity you'll fail the build on, and treat the .trivyignore file as something that gets reviewed rather than something that grows. Start at CRITICAL if HIGH would block every build on day one, then tighten it once the base images are current.

Resources

Policy-as-Code Governance with OPA/Rego

Mohammad Abu Mattar — Tue, 24 Feb 2026 00:00:00 GMT

Why policy-as-code matters

The governance problem

Managing infrastructure at scale gets complicated fast.

As your infrastructure grows, keeping it consistent and compliant gets harder. Manual reviews don't scale past a few teams, and configuration drift arrives on its own. What you need is enforcement that runs on every change without a human in the loop.

Common infrastructure issues

Developers accidentally making resources publicly accessible
Missing required tags on cloud resources
Non-compliant security group configurations
Kubernetes deployments without resource limits
Terraform modules bypassing organizational standards

The problem: manual governance

Why manual reviews fail

Code review and post-deployment checks depend on a tired human noticing one line in a 400-line diff. Standards get missed, security rules get worked around, and the issue surfaces after the apply. Guardrails have to run before the deploy to be worth anything.

What it costs

Unenforced policy shows up on the invoice and in the incident review:

Security breaches from misconfigured resources
Compliance violations leading to audits and fines
Cost overruns from unoptimized infrastructure
Operational chaos from inconsistent deployments

The fix: Open Policy Agent (OPA)

What OPA is

Open Policy Agent is a policy engine that keeps policy logic out of your application and infrastructure code. You write rules in Rego, a declarative query language, and OPA evaluates them against structured input: Terraform plans, Kubernetes manifests, CI/CD configuration, or any other JSON you hand it.

What you get

Unified enforcement across Terraform, Kubernetes, and custom tools
Declarative policies that are easy to understand and maintain
Pre-deployment validation to catch issues before they reach production
Audit trails for compliance and governance requirements
Organization-wide standards enforced consistently

Getting started with OPA and Rego

Installing OPA

# macOS
brew install opa

# Linux
curl -L -o opa https://openpolicyagent.org/downloads/latest/opa_linux_x86_64
chmod +x opa
sudo mv opa /usr/local/bin/

A basic Rego policy

# Deny public S3 buckets
package s3

deny[msg] {
    input.resource_type == "aws_s3_bucket"
    input.acl == "public-read"
    msg := sprintf("S3 bucket %s cannot be public", [input.name])
}

deny[msg] {
    input.resource_type == "aws_s3_bucket"
    input.acl == "public-read-write"
    msg := sprintf("S3 bucket %s cannot be public", [input.name])
}

Integrating OPA with Terraform

Using Conftest for Terraform

# Install conftest
brew install conftest

# Validate Terraform plan
terraform plan -json | conftest test -

Policy example: enforce tags

package terraform

deny[msg] {
    resource := input.resource_changes[_]
    resource.type in ["aws_instance", "aws_rds_cluster"]
    not resource.change.after.tags.Environment
    msg := sprintf("Resource %s must have Environment tag", [resource.address])
}

deny[msg] {
    resource := input.resource_changes[_]
    resource.type in ["aws_instance", "aws_rds_cluster"]
    not resource.change.after.tags.CostCenter
    msg := sprintf("Resource %s must have CostCenter tag", [resource.address])
}

Enforcing policies in Kubernetes

Installing OPA Gatekeeper

# Deploy Gatekeeper to your cluster
kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/release-3.14/deploy/gatekeeper.yaml

A Kubernetes ConstraintTemplate

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredResources
metadata:
  name: require-resource-limits
spec:
  match:
    kinds:
      - apiGroups: ['']
        kinds: ['Pod']
    excludedNamespaces: ['kube-system', 'gatekeeper-system']
---
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredresources
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredResources
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredresources

        violation[{"msg": msg}] {
            container := input.review.object.spec.containers[_]
            not container.resources.limits.cpu
            msg := sprintf("Container %s must have CPU limit", [container.name])
        }

        violation[{"msg": msg}] {
            container := input.review.object.spec.containers[_]
            not container.resources.limits.memory
            msg := sprintf("Container %s must have memory limit", [container.name])
        }

CI/CD pipeline integration

GitHub Actions example

name: Policy Validation

on: [pull_request]

jobs:
  policy-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Install OPA
        run: |
          curl -L -o opa https://openpolicyagent.org/downloads/latest/opa_linux_x86_64
          chmod +x opa
          sudo mv opa /usr/local/bin/

      - name: Run Policy Tests
        run: |
          opa test policies/ -v

      - name: Validate Terraform
        if: hashFiles('*.tf') != ''
        run: |
          terraform init
          terraform plan -json | opa eval -d policies/ 'data.terraform.deny' -f pretty

Best practices

1. Start at the organization level

Define policies at the organization level, not project level. Make them discoverable and documented.

# Start with clear policy namespaces
package org_policies.infrastructure.aws.security
package org_policies.kubernetes.workload

2. Fail safely

Distinguish between hard denials and warnings:

package my_policies

deny[msg] {
    # Hard deny: completely block this
    input.security_critical_violation == true
    msg := "This violates critical security policy"
}

warn[msg] {
    # Warning: recommend but allow with approval
    input.non_standard_naming == true
    msg := "Consider following naming standards"
}

3. Test your policies

# Test policy logic before deployment
opa test policies/ -v

4. Keep policies in version control

Keep policies in the same repo as infrastructure code, with proper code review processes.

Monitoring and auditing

Log policy violations

Capture and log every policy decision for audit trails:

package audit

log_decision[decision] {
    decision := {
        "action": "denied",
        "reason": input.violation_reason,
        "timestamp": input.timestamp,
        "resource": input.resource_id
    }
}

A rollout timeline

Phase	Focus	Timeline
Phase 1	Basic security policies (public access, required tags)	Week 1-2
Phase 2	Terraform integration in CI/CD	Week 3-4
Phase 3	Kubernetes Gatekeeper deployment	Week 5-6
Phase 4	Full auditing and monitoring rollout	Week 7-8

Wrapping up

Policy-as-code moves the check from after the deploy to before it.

With OPA and Rego, the same rules apply across Terraform, Kubernetes and whatever else you can feed JSON. Start with the security policies you'd be embarrassed to explain in an incident review, get those passing, then widen the net. Rolling out fifty policies at once mostly teaches your team how to add exceptions.

Resources

Setting Up GitHub Copilot Agent Skills in Your Repository

Mohammad Abu Mattar — Mon, 26 Jan 2026 00:00:00 GMT

Why teach Copilot new skills?

What a skill actually is

If you're ready to teach Copilot some new tricks, Agent Skills are your answer. Built on the agentskills.io open standard, a skill is a folder of instructions and tools that Copilot only opens when it's relevant to what you're working on. It's an assistant who knows which reference book to grab off the shelf, instead of one who dumps the whole library on the desk every time.

Loading only what's needed

Loading everything at once would slow Copilot down and fill its context with things that don't apply to the file you're in. Agent Skills use progressive disclosure to pull in targeted context only when it matters, so responses stay fast and stay on topic.

The problem: generic AI responses

One-size-fits-all answers

Out of the box, Copilot is helpful but generic. It doesn't know your team's workflows, your project's naming conventions, or the specialized tasks you run every week. So you type the same explanations over and over, session after session.

Context overload

Explaining everything upfront causes two problems. It slows every response down. And Copilot's answers get muddier, because the signal you care about is buried in the context you pasted.

Repeating yourself

Without skills, every new coding session starts from zero. You re-explain the same project patterns, and the explaining never turns into anything you can reuse.

The fix: repository-based Agent Skills

How Agent Skills are organized

Agent Skills are folders that live in your repo, or in your user profile. They hold instructions, scripts and tools that Copilot can reach for when needed. The format is portable across agents, including GitHub Copilot in VS Code, the CLI and the coding agent.

The three-level loading system (progressive disclosure)

To keep things efficient, Agent Skills use a three-level loading system:

Level 1: skill discovery (always on): Copilot reads the name and description from the YAML frontmatter of every available SKILL.md. This is lightweight and helps it decide if a skill is relevant.
Level 2: instructions loading: When a skill matches your prompt, Copilot loads the full body of the SKILL.md file.
Level 3: resource access: Copilot only accesses additional files (scripts, templates, examples) in the skill directory as needed.

Knowledge that stays put

Skills live in your repository, so the .github/skills directory travels with a clone and shows up in code review like any other change. Nobody has to configure anything to get them.

Setting up your first skill

Step 1: create the skills directory

First, pick where your skills folder lives. Recommended locations:

Project skills: .github/skills/ (recommended) or .claude/skills/ (for backward compatibility).
Personal skills: ~/.copilot/skills/ (recommended) or ~/.claude/skills/.

Say you pick .github/skills. Here's how you'd set it up:

# Create the main skills directory
mkdir -p .github/skills

# Create a specific skill folder
mkdir .github/skills/image-resizer

Inside that main folder, create a subfolder for each specific skill you want. Name them something clear like image-resizer or webapp-testing.

Step 2: write the SKILL.md file

Every skill needs a SKILL.md file (uppercase) with YAML frontmatter at the top. Spend your time on the description, because that is the only text level 1 discovery sees when it decides whether to load the skill at all.

Here's a real example:

---
name: 'Image Resizer'
description: 'Automatically resizes and optimizes images for web use. Handles batch processing, maintains aspect ratios, and generates responsive image sets. Use this when working with image assets that need multiple sizes or optimization.'
---

# Image Resizing Workflow

First, I'll check what you're starting with:

- Look at source image dimensions and format
- Figure out what target sizes you need
- Make sure the output directory exists

Next, I'll handle the actual work using the [optimization script](./scripts/optimize.js).

Finally, I'll verify the output quality and file sizes.

Notice how the description names concrete things: batch processing, aspect ratios, responsive sets. That specificity is what makes Copilot pick the skill up when you mention images or resizing.

Step 3: add scripts and resources

This is where skills get more useful than a prompt file. You're not limited to instructions. You can bundle JavaScript scripts, templates and configuration files alongside your SKILL.md, then reference them with relative paths.

Here's what your folder structure might look like:

.github/skills/image-resizer/
├── SKILL.md
├── scripts/
│   ├── resize-images.js
│   └── optimize.js
└── templates/
    └── config-template.json

Then in your SKILL.md, you can tell Copilot about these files:

## Tools Available

To resize images, run: `./scripts/resize-images.js`

For optimization, use: `./scripts/optimize.js`

Configuration template: `./templates/config-template.json`

Now when you ask Copilot to resize images, it points at the script that already exists and has been tested, instead of writing you a fresh one that has not.

Step 4: enable skills in VS Code

This feature is currently in preview, so you'll need VS Code Insiders to try it out. Here's how to get it working:

Open VS Code (or VS Code Insiders)
Open Settings (press Cmd+, on Mac or Ctrl+, on Windows/Linux)
Search for "Agent Skills"
Ensure the experimental skill support is enabled

You can also enable it via .vscode/settings.json if you prefer:

{
  "chat.useAgentSkills": true
}

Copilot's "Agent Mode" (Windows & Linux: Ctrl+I, Mac: Cmd+I) is highly optimized for using these skills autonomously. For the best experience, try invoking Copilot in Agent Mode when working with your custom skills.

Once it's enabled, test it. Open Copilot chat and ask "What skills do you have?" If the wiring is right, Copilot lists your new skill and summarises what it does. If it doesn't, your frontmatter is usually the problem.

Skill examples

Documentation generator

Say your team has a style guide for docs. A skill can hold those conventions so generated documentation comes out in your format rather than a generic one.

---
name: 'Documentation Generator'
description: "Generates API documentation following the team's style guide. Includes TypeScript examples, parameter descriptions, and usage patterns. Use when documenting new functions or API endpoints."
---

## Documentation Format

Each function should include:

1. Brief description (one sentence)
2. Parameter table with types and descriptions
3. Return value explanation
4. Code example showing typical usage
5. Common gotchas or edge cases

Example template is in `./templates/api-doc.md`

Test suite builder

Tired of writing the same boilerplate test code? Build a skill that knows your testing patterns and can scaffold a whole suite.

---
name: 'Test Suite Builder'
description: 'Generates comprehensive test suites using Jest and React Testing Library. Covers happy paths, edge cases, and error scenarios. Use when creating tests for new components or utilities.'
---

## Test Structure

For each component/function, generate:

- Setup and teardown blocks
- Happy path tests
- Edge case coverage
- Error handling tests
- Mock setup when needed

Refer to `./examples/sample-test.spec.ts` for the pattern.

Code review checklist

Your team's review standards can go into a skill too, so the checklist runs on the PR instead of living in a wiki page nobody opens.

---
name: 'Code Review Checklist'
description: 'Provides a comprehensive code review checklist based on team standards. Covers code quality, security, performance, and testing. Use when reviewing pull requests.'
---

## Review Criteria

### Code Quality

- [ ] Functions are under 50 lines
- [ ] No console.logs in production code
- [ ] Meaningful variable names
- [ ] Comments explain "why" not "what"

### Security

- [ ] No hardcoded credentials
- [ ] Input validation on all external data
- [ ] Proper error handling without exposing internals

### Testing

- [ ] Unit tests for new functions
- [ ] Integration tests for API endpoints
- [ ] Coverage above 80%

What you get out of it

Less repetition

The saving shows up in the explaining you stop doing. No typing out the same conventions every morning, no digging up the doc you wrote six months ago to paste into chat.

Tasks that used to take three or four rounds of correction land closer to first try, because the context Copilot needed was already loaded.

Knowledge the whole team can use

Skills move institutional knowledge out of people's heads. The pattern the senior dev always uses, the workaround for that one API quirk, the commit message format you keep correcting in review: put it in a skill and it applies to everyone.

New hires come up to speed faster because the knowledge is in the repo they just cloned.

Consistent output

Copilot follows the same patterns each time, so you stop getting one suggestion today and a different one tomorrow for the same kind of change.

Automation, not just advice

Bundling scripts with the instructions is the part I'd not give up. Copilot can run your image optimizer, generate the boilerplate, execute the tests. Advice you have to implement and a script that already works are not in the same category.

Quick Reference

Create Skills Directory: Choose a location (.github/skills/, .copilot/skills/, or .claude/skills/) and create a subfolder for your skill.
```
mkdir -p .github/skills/my-skill
```
Create SKILL.md: Inside your skill folder, create a SKILL.md file with YAML frontmatter including name and description.
```
---
name: 'My Skill'
description: 'Brief description of what this skill does and when to use it.'
---
```
Add Resources: Include any scripts, templates, or additional files your skill needs in the same folder.
```
mkdir .github/skills/my-skill/scripts
```
Reference Assets: Use relative paths in SKILL.md to point to your scripts or templates.
```
To run the script, use: `./scripts/my-script.js`
```
Enable in VS Code: Turn on Agent Skills in VS Code settings by enabling chat.useAgentSkills.
```
{
  "chat.useAgentSkills": true
}
```
Verify Setup: Open Copilot Chat and ask, "What skills do you have?" to confirm your skill is recognized.

Next steps

Don't try to write the perfect skill first. Start with one small task you do constantly. Generating test files in your team's format, say, or the checklist you run before a production deploy.

Get that working. Watch how Copilot uses it, because the first description you write is usually too vague to trigger reliably. Then add scripts and templates once the plain version earns its place.

As the library grows, Copilot stops behaving like autocomplete and starts behaving like someone who has read your codebase.

The write-up is a one-time cost, and the skill stays in the repo for whoever clones it next.

7 Reasons Learning the Linux Terminal is Worth It (Even for Beginners)

Mohammad Abu Mattar — Mon, 05 Jan 2026 00:00:00 GMT

Why learn the Linux terminal?

Why it still matters

Even with the graphical tools and AI assistants available now, the terminal is the most direct way to work with a Linux system. It's a core skill: real control over the machine, and the ability to automate almost anything you can do by hand.

The myth worth killing first

"The terminal is too hard for beginners"

GUIs look easier for the first hour. After that they hide the thing you need and cap what you can do. The terminal hands you the whole system, and it does it the same way every time.

Remembering commands is easier than you think

Commands vs menu hunting

New users worry about memorizing hundreds of commands. You don't. Commands are structured text with consistent rules, and there are maybe fifteen you'll type every day.

The learning curve myth

The structure makes sense once you see it: the command is the verb, then options, then arguments. sudo apt install -y curl is the same shape as sudo apt remove -y curl. Learn the shape and new commands stop looking new.

Short commands beat long instructions

Why text is more efficient

A GUI tutorial is eight screenshots. sudo apt upgrade -y is one line, and it tells you what it does. You can read it, paste it, and change one flag when your situation differs.

Copy-paste vs clicking through it again

Text commands can be copied, shared and scripted. GUI instructions have to be re-performed by hand each time, and step four is where people go wrong.

Commands are evergreen

GUI tutorials rot

A GUI tutorial breaks the next time someone moves a settings panel. Commands stay stable across distributions and across versions.

Knowledge with a long shelf life

The commands you learn today still work years from now, no matter how much the desktop environment changes around them.

Linux is built out of text

Everything is a file

Because configuration and logs are text, grep, sed and awk work on all of it. A GUI puts the same information behind a database or a format only that GUI reads.

Nothing is hidden from you

You can search /etc with ripgrep, pipe a log through awk, and count what you find. There is no GUI equivalent for that, because you'd need the vendor to have anticipated your exact question.

GUIs are training wheels

The abstraction gets in the way

A graphical interface sits between you and the system, spending screen space and attention on panels and buttons. Good while you're learning what things are called. Friction once you know.

Screen space and attention

Modern GUIs eat pixels and make you scan visually for what you already know the name of. Typing the name is faster than finding it.

GUIs are not scriptable

The automation ceiling

A graphical interface needs a person clicking it. That's the ceiling, and no amount of GUI polish raises it.

Scripts compose

Shell scripts chain together, run on a schedule and drop into a CI pipeline. GUI automation, when it exists, breaks when a button moves ten pixels.

GUIs don't teach you much

What you don't learn by clicking

Living only in GUIs keeps your model of the system shallow. The terminal pushes you toward scripting, automation and the parts of the system that actually explain the behaviour you're seeing.

It compounds

You start with ls and cd. A few months later you're writing a script that does in one line what used to be a ten-minute chore. That habit of composing small pieces makes you better at everything else too.

Making the terminal your primary interface

Start small

Start with package management (apt, dnf, pacman), file operations (ls, cd, cp, mv), and text processing (grep, cat, less). Each command you learn makes the next one cheaper.

Expect a rough week

The first week is genuinely annoying. You'll look up things you already know how to do with a mouse, and that feels like going backwards. Push through it, because the payoff starts the first time you script something you used to do by hand.

Where it leaves you

The terminal is the most direct and most reliable way to work with Linux, and it hasn't changed much in decades. That stability is the point: what you learn this month will still be true in ten years.

Docker Is Eating Your Disk Space (And How PruneMate Fixes It)

Mohammad Abu Mattar — Fri, 02 Jan 2026 00:00:00 GMT

The problem: Docker is eating your disk space

What it looks like when it happens

Your Docker host is running out of space. Again.

You've been spinning up containers, testing new services, building images. Everything's humming along nicely. Then your system starts throwing errors because the root filesystem is full. You check your disk usage and Docker has taken 200GB. Wait, what? How did this even happen?

Why Docker holds onto so much

What's actually going on

Docker is trying to be helpful. When you stop a container, you might want to restart it later with the same data, so Docker doesn't throw anything away. Volumes stick around after containers are gone. Build cache hangs out to make your next build faster. Old images stay put "just in case" you need them again.

This makes total sense for production. But in a home lab where you're constantly trying new stuff? It turns into a slow pile-up of junk. That database volume from three months ago? Still there. Build cache from a project you ditched? Yep, still hanging around. Images you pulled once for curiosity and never touched again? All of it adds up.

Checking what's using your space

Find out where it went. Run this:

docker system df

You'll probably see something like this:

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          47        12        23.5GB    15.2GB (64%)     # 15GB we could get back!
Containers      15        8         2.1GB     1.3GB (61%)
Local Volumes   89        24        45.8GB    38.2GB (83%)     # Ouch, 38GB of unused volumes
Build Cache     156       0         12.3GB    12.3GB (100%)    # 100% unused. All of it.

Look at that. 66GB sitting there doing nothing. And if your Docker volumes live on your root filesystem, which they probably do, the whole machine goes down with it. Databases refuse connections. Package managers throw errors. Containers can't write logs.

The risks of manual Docker cleanup

The built-in prune commands

Sure, you can manually prune stuff with Docker's built-in commands:

# These work, but they're scary to run blindly
docker image prune    # Removes unused images. Pretty safe.
docker volume prune   # Removes unused volumes. WAIT, ARE YOU SURE?
docker system prune   # Nuclear option. Add -a and you're in danger territory.

Why manual cleanup is risky

It always feels risky, though. Are those volumes actually unused? What if you delete something you needed? I made this mistake early on. I thought I'd be clever and deleted folders in /var/lib/docker/volumes by hand. I destroyed persistent data I cared about, because from the filesystem I couldn't tell which volume belonged to what.

The built-in prune commands are safer than my folder deletion, but they're still blunt. It's all or nothing. And if you've got multiple Docker hosts, you're SSHing into each one running the same commands over and over.

Enter PruneMate: actually sensible Docker cleanup

What makes PruneMate worth using

PruneMate is an open-source tool that fixes all this. It shows you what's eating your space, lets you pick exactly what to clean, and runs it all on a schedule so you don't have to think about it.

What makes it worth using:

Visual dashboard that shows where your space is going across all your Docker hosts
Granular control pick images, volumes, networks, containers, or build cache. Your choice.
Preview mode see exactly what'll get deleted before you pull the trigger
Scheduled cleanup jobs that run automatically while you sleep
Remote host support using Docker Socket Proxy (no SSH needed)
Notifications via Gotify, ntfy, Discord, or Telegram

Instead of SSHing around and guessing what's safe to delete, every host is in one interface.

Setting up PruneMate

Just deploy the container on one of your Docker hosts with Docker Compose. Here's all you need:

services:
  prunemate:
    image: anoniemerd/prunemate:latest
    container_name: prunemate
    ports:
      - '7676:8080' # Access the web UI on port 7676
    volumes:
      # Give PruneMate access to Docker on this host
      - /var/run/docker.sock:/var/run/docker.sock
      # Keep logs and config between restarts
      - ./prunemate/logs:/var/log
      - ./prunemate/config:/config
    environment:
      - PRUNEMATE_TZ=America/New_York # Change to your timezone
      - PRUNEMATE_TIME_24H=true # Or false if you prefer AM/PM
      # Optional: Add a password to protect the interface
      # - PRUNEMATE_AUTH_USER=admin
      # - PRUNEMATE_AUTH_PASSWORD_HASH=your_hash_here
    restart: unless-stopped

Bring it up:

docker compose up -d

Now just open http://your-host:7676 in your browser and you're good to go.

Managing multiple Docker hosts

If you want to manage other Docker hosts remotely, set up a Docker Socket Proxy on each one. PruneMate then connects through the proxy instead of getting full access to the Docker socket, which is effectively root on the host:

services:
  dockerproxy:
    image: ghcr.io/tecnativa/docker-socket-proxy:latest
    environment:
      # These control what PruneMate can do via the proxy
      - CONTAINERS=1 # Let it see and manage containers
      - IMAGES=1 # Let it manage images
      - NETWORKS=1 # Let it manage networks
      - VOLUMES=1 # Let it manage volumes
      - BUILD=1 # Needed for cleaning build cache
      - POST=1 # Needed for actually running prune commands
    ports:
      - '2375:2375' # Standard Docker API port
    volumes:
      # Read-only access to Docker socket. Much safer.
      - /var/run/docker.sock:/var/run/docker.sock:ro
    restart: unless-stopped

Deploy this on each remote host, then add them to PruneMate's interface with their hostname and port 2375. That's the whole home lab from one place.

Using PruneMate

The interface and cleanup options

The interface is a set of checkboxes for what to clean:

All unused containers
All unused images
All unused networks
All unused volumes
All build cache

By default, only unused images are checked, which is the right default. An image you delete by mistake is one docker pull away. A volume you delete by mistake is gone, so PruneMate leaves volumes alone until you say otherwise.

A typical cleanup

Here's what a typical cleanup looks like:

Check the boxes for what you want to clean
Hit "Preview & Run"
Look at what it's about to delete (with size estimates)
If it looks good, hit "Confirm & Execute"
Get a notification when it's done

That preview step is the reason I use it. You'll see something like "About to delete 50 unused volumes and free up 38GB" before anything happens, and if a number looks wrong you back out.

Setting up automated cleanup

For automated cleanup, I set up a schedule. Here's mine:

Images and containers: Weekly cleanup
Build cache: Weekly (I rebuild often, so cache gets stale fast)
Volumes: Manual only (way too risky to automate)

This keeps everything clean without me having to remember. The notifications tell me what got cleaned up, so I know what happened without watching it happen.

PruneMate vs manual cleanup

The trade-offs

	Manual Cleanup	PruneMate
Visibility	Run `docker system df` on each host manually	Dashboard shows all hosts at once
Safety	High risk with wrong flags (`-a`, `--volumes`)	Preview before deletion, focused on unused resources
Granularity	All-or-nothing (especially with `system prune`)	Pick exactly what to clean
Remote hosts	SSH to each one individually	Manage all hosts from one interface
Automation	Requires cron jobs or CI/CD pipelines	Built-in scheduler with notifications
Learning curve	Forces you to understand Docker maintenance	Hides complexity (good and bad)
Time investment	High upfront to script it properly	10 minutes to deploy and configure

Why this matters now

Manual cleanup teaches you how Docker actually works underneath, and that is worth learning once. But once you've learned it, running the same three commands on four hosts forever teaches you nothing new. PruneMate does the repetitive part and tells you what it did.

Component prices are also brutal right now. A 1TB NVMe that cost $80 last year is closer to $150. Thanks, AI boom. If you can't throw money at bigger drives, you have to get more out of the ones you have.

Docker disk space is one of those problems you ignore until it bites you. Then your system is already falling apart. Logs won't write. Databases run out of room. New deployments fail because there's no space for the image. By the time you notice, you're firefighting.

PruneMate keeps you ahead of that by cleaning up on a schedule. It's a boring tool for a boring problem, which is exactly what I want in something with delete permissions.

Bottom line

If you're running Docker anywhere, home lab or production, disk maintenance isn't optional. You can do it by hand with discipline and shell scripts, and plenty of people do. Or you can hand it to PruneMate and stop thinking about it.

Grab it here: PruneMate on GitHub

How are you handling Docker cleanup? Manual commands? Custom scripts? Already using some automation tool?

Understanding Kubernetes Services: ClusterIP vs NodePort vs LoadBalancer

Mohammad Abu Mattar — Tue, 23 Dec 2025 00:00:00 GMT

If you're working with Kubernetes, you've probably noticed that Pods come and go, and their IP addresses keep changing. That's where Services come in. They give you a stable way to keep your apps accessible and reliable. But picking the right type between ClusterIP, NodePort, and LoadBalancer? That can get confusing fast.

Why pods need a Service in front of them

Pods are temporary. They get created and destroyed, and their IPs change with them. Without a Service, one deployment's rollout breaks every caller that had cached an IP. Pick the wrong Service type and you either expose something internal to the internet or make your app unreachable, and both of those are worse to debug in production than to think about now.

What the choice affects

The Service type decides who can reach your workload and what it costs to run. It also decides how much of your cluster's edge you have to think about, which is why the default is the conservative one.

The three types

Kubernetes gives you three main Service types. Each one solves a different problem.

ClusterIP: internal traffic

How ClusterIP works

ClusterIP is the one you'll use most. It creates a stable virtual IP and DNS name inside the cluster that only other pods can reach. Backend services, databases, internal APIs: anything that has no business being reachable from outside. It's the default type, and leaving it as the default is usually the right answer.

When to use it

Backend services, databases, internal APIs and service-to-service calls inside the cluster. If nothing outside the cluster needs to connect, this is your type.

ClusterIP configuration

Here's what a basic ClusterIP Service looks like:

apiVersion: v1
kind: Service
metadata:
  name: backend-service
spec:
  type: ClusterIP # This is actually optional since it's the default
  selector:
    app: backend
  ports:
    - port: 8080 # Port the Service listens on
      targetPort: 3000 # Port your Pod listens on

NodePort: development and testing

How NodePort works

NodePort opens the same port (somewhere between 30000 and 32767) on every node in the cluster. Hit any node's IP on that port and kube-proxy forwards you to a pod, wherever it happens to be running. It takes one line of YAML, which is exactly why it's the right tool for development.

What it's good for

Quick external access in a dev cluster, without provisioning anything from your cloud provider or setting up an ingress controller.

Where it falls down

You're opening a port on every node, your clients need to know node IPs, and there's no health check in front of them. Nodes get replaced, and the IP your teammate bookmarked stops existing. Fine for testing, not something to hand to users.

NodePort example

Here's a NodePort example:

apiVersion: v1
kind: Service
metadata:
  name: test-service
spec:
  type: NodePort
  selector:
    app: webapp
  ports:
    - port: 8080
      targetPort: 3000
      nodePort: 30080 # Optional - K8s will assign one if you don't specify

Now you can access your app at http://<any-node-ip>:30080.

LoadBalancer: production external access

How LoadBalancer works

LoadBalancer is the production answer. Kubernetes asks your cloud provider for a real load balancer (AWS ELB, GCP Load Balancer, Azure Load Balancer) and gives you a public IP for it. Traffic spreads across healthy pods, and an unhealthy node drops out of rotation on its own.

Cloud provider integration

The cloud controller does the provisioning, so a type: LoadBalancer Service turns into an actual load balancer with a few lines of YAML and no console clicking.

What production gets from it

Health checks, failover when a node dies, and a stable IP you can point DNS at. The catch is that each LoadBalancer Service is a separate billed load balancer, which is why teams with many public services end up putting an ingress controller behind a single one instead.

LoadBalancer configuration

Here's how to set one up:

apiVersion: v1
kind: Service
metadata:
  name: frontend-service
spec:
  type: LoadBalancer
  selector:
    app: frontend
  ports:
    - port: 80 # External port
      targetPort: 8080 # Container port

Once it's deployed, Kubernetes talks to your cloud provider and sets everything up. You'll get an external IP that you can use in DNS records or share with users.

Comparing the three

They stack on each other

Use ClusterIP for internal services like databases, backend APIs, and microservice-to-microservice communication.
NodePort is handy for quick testing and development work.
LoadBalancer is what you need for production apps that face the internet.
These Service types actually build on each other. A LoadBalancer creates a NodePort, which creates a ClusterIP underneath.

Choosing one

Start from who needs to reach the workload. Nothing outside the cluster? ClusterIP. A teammate needs to poke at it this afternoon? NodePort. Real users on the internet? LoadBalancer, or an ingress controller sitting behind one.

Moving between types

Changing type is an edit to one field, and because the types nest, going from ClusterIP to LoadBalancer keeps the same in-cluster DNS name working. Going the other way removes the public IP, so anything pointing at it needs to move first.

Why the choice matters

Security

ClusterIP keeps internal traffic internal, and that is the whole security argument. Every time you promote a Service to NodePort or LoadBalancer, you're adding a door, so the question worth asking is whether that workload needed one.

Performance and scaling

A LoadBalancer distributes traffic and drops unhealthy backends. NodePort sends everything to whichever node the client picked, and if that node is busy, that's the client's problem.

Running it day to day

A cloud load balancer gives you metrics and health checks you'd otherwise build. NodePort gives you a port number to remember and nothing else.

What's your Kubernetes service strategy?

Community approaches

How are you exposing services in your Kubernetes clusters? Got any tips for managing external access?

Beyond Services

Most teams past a few public endpoints move to an ingress controller or a service mesh, and I'd like to hear where you drew that line and whether the mesh was worth its operational cost.

Managing Terraform at Scale with Terragrunt

Mohammad Abu Mattar — Sun, 21 Dec 2025 00:00:00 GMT

The problem with Terraform at scale

Duplicated code across environments

If you're managing infrastructure with Terraform across several environments or projects, you've probably hit the point where every new environment is a copy of the last one. That's what wrappers like Terragrunt exist for: they keep the shared parts in one file.

Maintenance overhead

With plain Terraform, a change to the backend block is a change in every environment directory. Miss one and that environment quietly runs on the old settings until someone notices the state file in the wrong bucket.

Why Terraform gets messy

Repetitive configuration

Plain Terraform is fine for one environment. Once you're running dev, staging and production, you're duplicating backend configuration, provider settings and variable files across directories. Every update means the same edit three times, and the third one is the one you forget.

Environment-specific boilerplate

Each environment needs almost the same configuration with two values changed. That "almost" is where copy-paste errors live, because a diff of two 40-line files is not something you read carefully at 5 PM.

State management

Every environment needs its own state file, its own lock table and its own key prefix. Setting that up by hand for each one is both tedious and the sort of thing that goes wrong silently.

The fix: the Terragrunt wrapper

How Terragrunt works

Terragrunt is a thin wrapper around Terraform that fills in the parts Terraform leaves to you. You define your backend config, provider settings and common variables once, and each environment inherits them. Your Terraform modules stay generic, and Terragrunt supplies the environment-specific values.

Configuration inheritance

The root terragrunt.hcl holds the shared settings. Each child config points at it with an include block, so the backend and provider definitions exist in exactly one file.

Environment-specific overrides

A child config adds only what differs: instance sizes, CIDR ranges, replica counts. The module never learns which environment it is running in, which is what makes it safe to reuse.

What it looks like in practice

Instead of duplicating the backend config in every environment, you define it once in the root terragrunt.hcl and let Terragrunt generate the per-environment state key from the directory path.

What Terragrunt gives you

DRY infrastructure code

Use Terragrunt to remove duplicate code across environments.
Define backend and provider configs once, reuse everywhere.
Keep your Terraform modules generic and environment-agnostic.
Let Terragrunt handle state management and dependencies between modules.

State management you don't write

Terragrunt derives each state key from the directory structure and creates the backend if it doesn't exist, so a new environment is a new folder rather than a checklist.

Consistent configuration

Every environment starts from the same base, and the differences are the handful of values you wrote down on purpose.

Why Terragrunt matters for teams

Scaling past a few environments

Terragrunt keeps a Terraform repo readable as it grows. Less time copying files, more time on the infrastructure itself. It does add a tool and a config language to learn, and that cost is real, though it's paid once rather than per environment.

Productivity

An update lands in one file and applies everywhere, so the "did we update staging too?" conversation stops happening.

Less to hold in your head

There's less code between a new hire and their first change, and most of what's left is the part that actually describes your infrastructure.

What's your Terraform strategy?

Community approaches

Do you use Terragrunt or another wrapper for Terraform? How do you keep your infrastructure code clean across environments?

Alternative tools

Terragrunt, Terraspace and hand-rolled wrapper scripts all solve this, and the shell script is genuinely the right call for some teams. I'd like to know which one you kept.

HashiCorp Pulls the Plug on CDKTF

Mohammad Abu Mattar — Mon, 15 Dec 2025 00:00:00 GMT

CDKTF is officially deprecated

The deprecation announcement

Well, it finally happened.

HashiCorp (now owned by IBM) officially deprecated the Cloud Development Kit for Terraform (CDKTF) on December 10, 2025. The repository is archived. No more features. No more fixes. If you chose CDKTF to write infrastructure code in TypeScript, Python, or Go instead of HCL, you're now being told to go back to the very thing you tried to avoid.

Impact on existing users

Your existing stacks keep working, because CDKTF only generates Terraform configuration and Terraform still runs it. What you lose is bindings for new providers, bug fixes, and anyone to report a security issue to after the archive date.

Why HashiCorp killed CDKTF

The business reason

Why did this happen?

According to HashiCorp, CDKTF "did not find product-market fit at scale." Translation: not enough enterprise customers using it to justify the investment. This is what happens when tools are owned by a single vendor focused on enterprise priorities over community needs.

What that says about priorities

CDKTF was never what enterprises were paying for. The paid tiers around Terraform are, and CDKTF meant maintaining a second toolchain that sold nothing on its own. Under IBM that arithmetic got less forgiving, not more.

Migration options after the deprecation

Your choices

What are your options?

You've got two paths forward:

Option 1: Go back to HCL (with OpenTofu)

HashiCorp suggests using cdktf synth --hcl to convert your code to raw HCL. But you don't have to run that HCL on HashiCorp's Terraform. OpenTofu is the truly open-source, Linux Foundation-backed alternative that won't change licenses on you or sunset tools you depend on.

Converting existing code

cdktf synth --hcl emits HCL rather than JSON, so you get files a reviewer can read. Budget time to clean the output up. Generated resource names are long, and every loop and conditional you wrote in TypeScript arrives fully expanded.

OpenTofu as the alternative

OpenTofu is a Linux Foundation fork of Terraform from before the licence change, so the state format, the provider protocol and most of the CLI are the same. In practice the move is a binary swap and a CI change rather than a rewrite.

Option 2: Switch to Pulumi

If you picked CDKTF because you wanted real programming languages with loops, variables, and proper abstractions, Pulumi is your best bet. Unlike CDKTF (which was always a translation layer), Pulumi is native infrastructure-as-software. You keep the power of TypeScript/Python/Go without the deprecation risk.

What Pulumi does differently

CDKTF generated Terraform configuration and handed it to Terraform. Pulumi talks to the providers itself, which mostly shows up in errors: a failure points at the line you wrote instead of at generated output you have to map back.

You keep the language

Loops, functions, real types and your existing unit tests all survive the move. What changes is the state backend, the CLI your pipeline runs, and the fact that you are now paying attention to a different company's roadmap.

The vendor lock-in lesson

Vendor risk

The bigger lesson here?

CDKTF is the small version of this. The Terraform licence change in 2023 was the large one. Both came out of one company deciding what happens to code other people had already shipped to production. Today it's CDKTF. Tomorrow it could be another tool you depend on.

Long-term strategy

The test I now apply is whether we could keep deploying if the vendor walked away tomorrow. An open state format counts. A provider protocol someone else has already implemented counts. A fork that exists and has commits in it counts. A tool only one company ships does not.

Plan your migration now

Start now

Your move:

If you're using CDKTF in production, start planning your migration now. Whether you go with OpenTofu for stability or Pulumi for programming power, don't wait until support runs out completely.

Migration timeline

Nothing breaks on a specific date, which sounds relaxed right up until a provider you use ships a change and nobody is left to regenerate the bindings. Treat it as a quarter of work spread across sprints, not a sprint.

Risk assessment

Sort your stacks by how often you change them. The ones you touch weekly hurt first, because those are the ones that will want a newer provider. A stack nobody has edited in a year can sit on archived CDKTF for a while, and being honest about that is what makes the rest of the plan fit.

Community discussion

Share your experience

What do you think?

Have you been using CDKTF? What's your migration plan? I would like to hear whether synth --hcl gave you output you were actually willing to keep, because that is the part I am least sure about.

Lessons learned

The thing I keep coming back to is that CDKTF was not a bad tool. It was a decent tool with no revenue attached to it, and that turns out to be the risk worth pricing when you pick anything from a single vendor.

Tracing Microservices with OpenTelemetry

Mohammad Abu Mattar — Mon, 23 Jun 2025 00:00:00 GMT

Why monitor your microservices?

The complexity of distributed systems

If you're juggling multiple services, it's hard to track how they work together. OpenTelemetry lets you follow one request end to end and see where it went wrong, which is closer to a map than to a dashboard.

What poor observability costs you

Without tracing, a microservice architecture is a black box with good uptime graphs. Every incident starts with the same twenty minutes of asking which service is at fault.

The challenge of microservices observability

Debugging distributed systems

With microservices, one slow or broken part drags the rest down with it. Without a clear view of what's happening, you're guessing at which service to look at, and you're guessing while the pager is going off.

The cascade effect

One failing service triggers timeouts in its callers, which trigger retries, which push load onto services that were fine a minute ago. By the time you look, five things are red and none of them is the cause.

Where logs and metrics stop helping

Metrics tell you latency went up. Logs tell you what one service did. Neither one connects the log line in your API to the log line in the database wrapper three hops later, and that connection is the thing you actually need.

The fix: OpenTelemetry distributed tracing

How distributed tracing works

OpenTelemetry is a free, open source toolkit that tracks requests as they move through your services. Each service records a span, the spans carry a shared trace ID, and the collector stitches them back into one timeline.

Instrumenting your services

Add the OpenTelemetry libraries to your code, then run a collector to receive the data and forward it to something like Jaeger or Zipkin. Auto-instrumentation for HTTP clients and database drivers gets you most of the picture before you write a single manual span.

Reading the traces

What you get is a waterfall: one bar per span, nested by caller. The slow hop is the wide bar, and the failed hop is the red one. That is usually the whole investigation.

Implementation steps

Adding the libraries

Add OpenTelemetry libraries to your services.
Set up a collector to feed data to a visualization tool.
Check traces regularly to spot and fix issues fast.

Configuring the collector

Point your services at one collector rather than at the backend directly. It handles batching, retries and sampling, and it means switching from Jaeger to a hosted platform later is a collector config change instead of a redeploy of every service.

Keeping up with it

Look at traces when nothing is broken, not only during incidents. Knowing what a healthy trace looks like is what makes an unhealthy one obvious, and it's how you notice the retry loop that has been quietly doubling your database load.

What tracing gives you

Faster resolution

You stop reading five log streams and start reading one timeline. The question changes from "which service is slow" to "why is this span slow", and that second question has an answer.

Fewer cascading failures

Catching the slow dependency early keeps it from turning into timeouts, retries and a system-wide incident.

A real picture of your dependencies

Traces show the calls your architecture diagram forgot. Every team I've watched turn tracing on has found at least one call nobody meant to make.

What's your monitoring strategy?

Community approaches

How do you keep tabs on your microservices? Got any favorite tools to share?

Tool comparisons

Jaeger, Zipkin and DataDog all pull this off, with different amounts of running your own storage. I'd like to hear which trade-off you took and whether you'd take it again.

Organizing Terraform with Modules

Mohammad Abu Mattar — Mon, 16 Jun 2025 00:00:00 GMT

Why organize your Terraform code?

Where the complexity comes from

If you're using Terraform to build out your infrastructure, you know how quickly things get complicated. Every new environment, every new account, every new team wants a slightly different version of the same VPC. That's where modules come in. Think of them as the thing that keeps your code reusable instead of copied.

What structure buys you

A module gives one VPC, one RDS instance or one IAM setup a single definition that every environment calls. When the definition changes, you change it once and bump a version tag instead of hunting through five directories.

The problem with messy Terraform code

Duplicated code

As your infrastructure gets bigger, your Terraform files turn into a real tangle. You copy and paste code, you end up with files nobody wants to open, and you lose track of which copy is current. Updates become a headache, and your whole team slows down with you.

Maintenance pain

Large, monolithic Terraform files are hard to read, hard to test and hard to change. A tweak to a security group in one area can break an unrelated part of your infrastructure, and terraform plan is the first place you find out.

Team collaboration problems

When several people work on the same large files, merge conflicts become frequent, and resolving a conflict in HCL that describes live infrastructure is exactly the kind of merge you don't want to get wrong.

The fix: Terraform modules

How a module is structured

Imagine modules as ready-to-go blueprints for parts of your infrastructure. You could create a module for a standard network setup or a database configuration. Then you reuse that blueprint across different projects or environments. Store these modules in a shared place, like a Git repository, so your team can call them up with a few lines of code whenever needed.

Reusing a module

Once a module exists, every project and environment calls the same source. Each caller passes its own variables, so dev and prod share the definition without sharing the sizing.

Version control and sharing

Pin the module source to a tag rather than a branch. A shared repository means everyone on the team can call the module with a few lines of code, and pinning means a change to main doesn't reach production until someone chooses it.

Module concepts

Breaking infrastructure into modules

Split your Terraform code into modules for the pieces you build more than once.
Put those modules in a shared repository so everyone on the team can reach them.
Use input variables to customize how a module behaves for different situations.

Input variables

Input variables are the only knobs a caller gets. Expose instance sizes, CIDR ranges and counts. Keep naming conventions and tagging inside the module, so callers can't drift from them by accident.

Module dependencies

Modules can reference each other's outputs, and that's how a network module feeds subnet IDs to a database module. It's also how you get an ordering problem, so keep the dependency chain shallow enough to explain out loud.

What modules give you

Code quality

Modules stop you writing the same configuration over and over. Fewer copies means fewer mistakes, and it means a review of one module covers every place that module is used.

Faster development

Reusable modules mean less time on boilerplate. Standing up a new environment turns into filling in variables rather than writing HCL from scratch.

Team productivity

Everyone works from the same definitions, so a discussion about the network is a discussion about one module instead of five copies that have quietly diverged.

What's your approach?

Community insights

How do you like to keep your Terraform projects organized? Any tips or tricks you've picked up along the way?

Sharing what works

Whether you're using the Terraform Registry, Git submodules or a private registry, the trade-offs are different, and hearing how you landed on yours is useful to the rest of us.

Securing CI/CD with IAM Roles

Mohammad Abu Mattar — Mon, 09 Jun 2025 00:00:00 GMT

Why secure your CI/CD pipeline?

Why pipeline security matters

Your pipeline holds credentials for every environment you deploy to, which makes it one of the most valuable targets you own. A separate IAM role per environment, each carrying only the permissions that environment needs, is the cheapest way to keep that target small.

Common security risks

Lots of CI/CD setups hand their tools far more access than they need. One long-lived key with broad permissions, shared by every job, because that was the fastest thing to wire up on a Friday.

The security problem

Over-privileged access

The usual shape is a single set of credentials with something close to admin. It works, so nobody revisits it. Then a test job with production write access runs a script someone edited in a hurry, and the blast radius is your whole account.

What it costs when it goes wrong

Compromised or misconfigured credentials let an attacker reach production systems, read sensitive data or push their own code through your deployment path. The outage is bad. The part where you cannot say which resources were touched is worse.

The fix: environment-specific IAM roles

Separating environments

Set up separate IAM roles for each stage: dev, staging and production. Give each role only the permissions it needs for its job. Your build tool might need to read a code repo, but it has no business touching production data. AWS IAM and GitHub Actions both make this straightforward to wire up.

Applying least privilege

Each environment gets its own IAM role with the minimum permissions it needs. Dev roles build and test. Staging roles deploy to test infrastructure. Production roles get the smallest set that still lets a deploy finish, and nothing else.

Tools and platforms

AWS IAM and GitHub Actions make this easy to set up. GitLab CI, Azure DevOps and Jenkins support the same role-based pattern, so the approach travels if you change platforms.

Implementation steps

Creating environment-specific roles

Create IAM roles for each environment in your pipeline.
Only give the exact permissions needed for each task.
Check roles regularly to keep access tight.

Mapping permissions

Write down what each pipeline stage actually does, then grant exactly that: repo read for builds, artifact bucket write for deployments, infrastructure permissions for provisioning. If you can't explain why a permission is there, take it away and see what breaks in dev.

Audits and updates

Permissions accumulate. Someone adds one to unblock a deploy at 6 PM and it stays for two years, so put a recurring review on the calendar and use your provider's access-analyzer output to find the ones nothing has used.

What least privilege gives you

A smaller blast radius

Scoped roles limit what a compromised job can reach. A leaked dev credential gets an attacker a dev environment, and that is the whole point.

Problems surface in dev

With roles separated, a job that tries to touch something it shouldn't fails in dev with a permission error rather than succeeding in production.

Shorter audits

Compliance work gets easier because the access pattern is already written down as policy. You point at the role definitions instead of reconstructing who could do what.

What's your approach?

Community discussion

How do you keep your pipelines locked down? Got any tips to share?

Share your experience

Whether you're on AWS, GitHub Actions or something else, I'm curious where you drew the line between safe and workable, because too strict and people start bypassing the pipeline entirely.