(Quick note: this post is probably best enjoyed in light mode: I’ve added a few fun doodles along the way and they show up much better there.)

Over the past few months, I’ve been trying to spend more time exploring different parts of the Kubernetes ecosystem as part of my open source learning journey. Sometimes that means contributing to a project, sometimes it’s digging through documentation, and other times it simply starts with building a small demo to understand how something works under the hood.

We would all agree that kubernetes has a lot of moving pieces, and I’ve found that the best way for me to truly understand something is to slow down, experiment a bit, and then write about it in a way that would have helped me when I first encountered the topic.

The Kubernetes Steering and Security Response Committees confirmed that Ingress NGINX will be retiring in March 2026, and that all security patches and bug fixes will end in March.

Considering that this project currently powers around 50% of cloud-native environments, this is quite a significant update for the ecosystem. Many teams are now evaluating different alternatives and approaches for handling ingress traffic moving forward, and Gateway API is one of the major directions the community has been exploring.

Seeing this made the topic feel even more relevant to learn about. So I decided to spend some time over the weekend digging into it, reading through the documentation to understand how things actually work.

And as often happens with Kubernetes, that small exploration quickly turned into a deeper learning exercise.

Because of that, I decided to turn this into a two-part blog.

In this first part, we’ll focus on the concepts and the “why” behind Gateway API. We’ll do a quick recap of what Kubernetes Ingress is, look at some of the limitations people ran into over time, and then understand how Gateway API tries to address those problems through its core resources.

In the second part, we’ll move from theory to practice and walk through a small Gateway API demo together so we can see how these pieces actually work inside a real Kubernetes cluster.

So before we jump into Gateway API, let’s take a step back and start with something familiar.

If Gateway API can be thought of as an evolution of Ingress, then the first step is simple:

Let’s quickly revisit what Kubernetes Ingress is and how it works.

Understanding Kubernetes Ingress

Before we talk about Gateway API, it helps to step back and look at something that has been part of Kubernetes networking for a long time: Ingress.

If you’ve worked with Kubernetes even a little, there’s a good chance you’ve come across it. Ingress is commonly used when we want to expose services inside a cluster to traffic coming from outside. Instead of directly exposing every service, Ingress gives us a way to define rules for how incoming traffic should be routed.

At its core, the Ingress object natively supports two kinds of routing:

• Host-based routing

• Path-based routing

This is something many of us have probably used without thinking too much about it. Inside the spec section of an Ingress resource, we usually define things like hosts and paths, which then determine how incoming requests should reach different services.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  namespace: mobileapp-ns
spec:
  rules:
  - host: example.com
    http:
      paths:
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: api-svc
            port:
              number: 80

For example, we might define that requests coming to example.com/api should go to one service, while requests to example.com/web should go to another. Kubernetes allows us to describe these routing rules in a structured way through the Ingress resource.

While the spec section defines the routing logic, a lot of the other important behavior, especially things related to how the load balancer should actually be configured, is often defined somewhere else: annotations.

The Hidden Complexity of Annotations in Ingress

In Kubernetes, annotations are essentially key–value metadata attached to objects. They were originally designed as a way to attach informational data to Kubernetes resources.

This information could then be read by other engineers or consumed by external tools.

metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:

For example, a team might use annotations to attach helpful information to a resource, or tools running inside the cluster might read those annotations and take some action based on them.

Ingress controllers use this mechanism heavily.

When we define an Ingress resource, the routing rules go into the spec section. But things like how the load balancer should behave are often defined in the annotations section. In many setups, this is where we specify configuration related to the load balancer that will actually handle the incoming traffic.

Under the hood, an Ingress controller is constantly watching Ingress resources in the cluster. When it detects a new or updated Ingress object, it reads the annotations attached to it and uses them to configure the load balancer accordingly.

Now here’s an interesting detail about how Kubernetes treats these annotations.

When we apply a manifest using kubectl, there are usually two levels of validation happening:

1. The kubectl client performs an initial verification.

2. The Kubernetes API server performs another round of validation before storing the object.

But annotations are treated differently.

Since annotations are meant to be informational metadata, Kubernetes itself does not validate their contents. As long as the YAML syntax is valid and the structure of the manifest is correct, Kubernetes will accept it.

This means that if there is a mistake in the annotation configuration, Kubernetes will not complain. The manifest will still be applied successfully.

However, the Ingress controller that relies on those annotations may not behave the way we expect.

If the annotation is misconfigured, the controller may configure the load balancer incorrectly, or sometimes not configure it at all.

Another thing that makes this more complex is that annotations are often vendor-specific.

If you are using different ingress controllers such as NGINX Ingress Controller, HAProxy Ingress Controller, AWS Load Balancer Controller, or Traefik’s ingress controller, the annotations required for configuration can be completely different.

So even though the Ingress resource itself looks standardized, the behavior behind it often depends heavily on the specific controller being used.

In short:

• The Ingress object natively supports host-based and path-based routing.

• Many advanced behaviors are implemented through controller-specific annotations.

• These annotations are not validated by Kubernetes.

• And they often vary depending on the ingress controller vendor.

This design worked for a long time, but as Kubernetes environments became more complex, these limitations started to become more visible.

What Was Missing in Ingress?

By now, we’ve seen that Kubernetes Ingress works well for basic routing. It allows us to define host-based and path-based rules, and with the help of controllers and annotations we can extend its behavior.

But as Kubernetes environments grew larger and more teams started running multiple applications inside the same cluster, some important limitations of Ingress began to appear. These limitations weren’t always obvious at first, but they became more visible as clusters started hosting many applications, multiple teams, and more complex networking requirements.

Let’s walk through some of the key things that were missing or difficult to achieve with Ingress.

1. Native Multi-Tenancy and Namespace Isolation

One of the first challenges comes from how Ingress resources are scoped.

An Ingress resource is namespaced, which means it belongs to a specific namespace in Kubernetes. At first glance, this seems perfectly normal because most Kubernetes resources are namespaced.

But things become tricky when we think about how applications are typically deployed in real environments.

In many organizations:

• Each application is deployed in its own namespace

• This helps maintain isolation between applications

• Different teams manage different namespaces

• Services belonging to each application live inside their own namespace

At the same time, exposing applications to the outside world usually involves a load balancer.

And load balancers are not something we want to create excessively, especially in cloud environments, because they incur cost. So a very common pattern is to use a single load balancer to expose multiple applications.

Now imagine we have 10 different applications, each running in its own namespace:

app1 -> namespace: payments
app2 -> namespace: orders
app3 -> namespace: users
...

Each of these applications has its own service inside its respective namespace.

Ideally, we would want a single external entry point (a load balancer) that routes traffic to all these applications.

But here’s where the problem appears.

Inside an Ingress configuration, when we define routing rules, we can only reference services that exist in the same namespace as the Ingress resource.

That means:

• If the Ingress object is created in namespace payments

• It can only reference services inside payments

• It cannot reference services from orders, users, or any other namespace

So if we wanted one Ingress resource to route traffic to services across multiple namespaces, Ingress does not allow that natively.

This creates an awkward situation.

On one hand:

• We want application isolation

• We want each application in its own namespace

But on the other hand:

• We want to reuse the same load balancer

• We want a single entry point for multiple applications

With the native Ingress resource, achieving this cleanly becomes difficult.

2. Limited Control Over Who Can Modify Routing Rules

Another challenge appears when multiple teams are involved.

Imagine again that we have multiple applications exposed through the same Ingress resource.

Maybe the routing rules look something like this conceptually:

/app1 -> service: app1
/app2 -> service: app2
/app3 -> service: app3

Now suppose each application is owned by a different development team.

Ideally, each team should only be able to modify their own routing configuration.

For example:

• The app1 team should manage /app1

• The app2 team should manage /app2

• The app3 team should manage /app3

But with a single Ingress resource, this becomes difficult.

Because the entire routing configuration lives inside one resource, anyone who needs to modify it usually needs access to the whole Ingress object.

That means:

• An app1 developer could accidentally change the configuration for app3

• Or modify something unrelated to their application

In other words, Ingress doesn’t provide a clean way to create clear boundaries between teams managing different routes.

3. Limited Protocol Support

Another limitation is related to protocol support.

The Kubernetes Ingress resource primarily focuses on HTTP and HTTPS traffic.

For many web applications this works perfectly fine. But modern systems often need to support additional protocols such as:

• TCP

• UDP

• gRPC

These protocols are not natively supported by the Ingress API itself. In some cases, ingress controllers may provide ways to support them, but again those solutions tend to be controller-specific extensions rather than part of the core API.

4. Important Features Hidden Inside Annotations

Earlier we discussed how annotations are heavily used to extend Ingress behavior.

Over time, many important features ended up being implemented through annotations rather than through the core API itself.

Some examples include:

Traffic splitting (Canary deployments)

Sometimes we want to gradually release a new version of an application. Instead of sending all traffic to the new version immediately, we may want to split traffic like this:

• 90% → old version

• 10% → new version

This type of weighted routing or canary deployment is not natively defined in the Ingress spec.

SSL Redirects

Sometimes we want to automatically redirect requests from HTTP to HTTPS.

For example:

<http://myapp.com>
→ redirect internally to
<https://myapp.com>

This type of behavior is typically implemented through controller-specific annotations.

URL Rewrites

Another common requirement is rewriting URLs.

For example, a request coming to:

/shop

might need to be internally rewritten to:

/store

Or redirected to another path.

Again, this behavior is usually configured using annotations instead of the native Ingress spec.

5. Vendor-Specific Annotations

And finally, we return to the issue we discussed earlier: annotations are vendor-specific.

Different ingress controllers use different annotations for configuring behavior.

For example, if today you are using:

• an AWS Load Balancer controller

and tomorrow you decide to migrate to another ingress controller, you may find that many of the annotations used in your manifests no longer work.

Because those annotations were specific to the previous controller.

This means that moving between controllers can require updating many of the Ingress manifests, simply because the configuration lives inside vendor-specific annotations.

What Is Gateway API?

After seeing some of the limitations of Ingress, the next natural question becomes: what exactly is Gateway API?

A simple way to think about it is this:

Gateway API is essentially the next generation of Ingress.

It was designed by the Kubernetes community to solve many of the problems that started appearing as clusters grew larger, applications became more complex, and more teams began sharing the same infrastructure.

While Ingress focused mainly on basic HTTP routing, Gateway API takes a broader approach to traffic management.

One of the first things that stands out is protocol support.

With Ingress, the native focus was mostly on HTTP and HTTPS traffic. Gateway API expands this significantly by supporting multiple protocols such as:

• HTTP / HTTPS

• TCP

• UDP

• gRPC

This makes it much more flexible for modern environments where different applications might communicate using different protocols.

Another important improvement is around role separation.

Earlier, we saw how a single Ingress resource could become difficult to manage when multiple teams needed to modify routing rules. Gateway API introduces a model that allows clear separation of responsibilities, so different teams can work on different parts of the configuration without interfering with each other.

In addition to that, Gateway API also focuses on portable traffic management. The goal is to reduce reliance on vendor-specific annotations and instead define many important traffic features directly as part of the API itself.

Some of these capabilities are treated as first-class features, meaning they are built into the API design rather than being added through controller-specific extensions.

So instead of relying heavily on annotations to enable these behaviors, Gateway API aims to provide structured ways to define them directly within the API model.

How Gateway API Works in Kubernetes

Another important thing to understand is that Gateway API is not built into Kubernetes by default.

Instead, it is introduced into a cluster through Custom Resource Definitions (CRDs).

If you’ve worked with CRDs before, you can think of them as templates that define new types of resources inside Kubernetes. Once these CRDs are installed in the cluster, we can start creating custom resources that follow those definitions.

In other words:

• The CRD defines the structure

• The custom resource follows that structure

Gateway API uses this mechanism to introduce new resource types that Kubernetes does not have out of the box.

But defining resources alone is not enough. Something also needs to watch those resources and make sure the desired behavior actually happens.

This is where the controller comes in.

In Kubernetes, controllers are responsible for making sure that the desired state matches the actual state. For example, when we create a Deployment, there is a controller that ensures the correct number of pods are always running.

Gateway API follows the same pattern.

Once the Gateway API CRDs are installed, a compatible controller must be running in the cluster. That controller continuously watches the Gateway API resources and ensures that whatever configuration we defined is actually implemented.

So the overall flow looks something like this:

1. Install the Gateway API CRDs into the cluster

2. Deploy a compatible controller that understands those resources

3. Create Gateway API resources that follow the CRD definitions

4. The controller watches those resources and configures the underlying infrastructure accordingly

Who Builds and Maintains Gateway API?

Gateway API is not maintained by a single vendor.

It is developed by the Kubernetes networking Special Interest Group (SIG), which is a community group responsible for networking-related features in Kubernetes.

Because it is developed under the Kubernetes SIG umbrella, the goal is to create a standardized and portable networking API that can work across different environments and implementations.

3 Stable API Kinds:

1. GatewayClass

When I first started looking into Gateway API, one thing that helped me a lot was thinking about it as a set of building blocks. Instead of one large resource like Ingress trying to do everything, Gateway API breaks things down into smaller, clearer pieces.

One of the first pieces you’ll come across is something called a GatewayClass.

At first glance, the name might sound a bit abstract, but the idea behind it is actually quite simple.

A GatewayClass is essentially a template that describes how gateways should behave.

You can think of it as a blueprint for gateways.

Just like in Kubernetes we have things like a StorageClass that defines how storage should be provisioned, a GatewayClass defines how a gateway should be implemented by a particular controller.

Let’s unpack that step by step.

Step 1: Connecting Gateway API to a Controller

Earlier we discussed that Gateway API works using controllers. A controller is responsible for watching resources and making sure the infrastructure behaves the way we defined.

GatewayClass is where we tell Kubernetes which controller should handle our gateway configuration.

For example:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: example-gatewayclass
spec:
  controllerName: example.com/gateway-controller
  description: GatewayClass with parameters
  parametersRef:
    group:""
    kind: ConfigMap
    name: gatewayclass-config
    namespace: default

The most important part here is:

controllerName: example.com/gateway-controller

What this line is saying is:

This GatewayClass should be managed by the controller called example.com/gateway-controller.

So whenever a gateway references this GatewayClass, that specific controller will take responsibility for configuring the underlying infrastructure, such as provisioning or managing a load balancer.

Step 2: GatewayClass Does Not Create Infrastructure

Another important thing to understand is that GatewayClass itself does not create anything at runtime.

It does not spin up a load balancer.

It does not route traffic.

Instead, it simply defines the template or policy that gateways will follow.

The actual runtime component comes later when we create a Gateway resource, which will reference this GatewayClass.

So the relationship looks something like this:

GatewayClass → defines how gateways should behave
Gateway → uses that definition to create actual networking infrastructure

In other words:

• GatewayClass defines the template

• Gateway consumes that template

Step 3: Provider-Specific Configuration

Another thing a GatewayClass can hold is provider-specific parameters and default policies.

Remember earlier how Ingress relied heavily on annotations that were often vendor-specific?

Gateway API tries to organize that configuration in a more structured way.

Instead of scattering configuration across annotations, a GatewayClass can reference a configuration object, such as a ConfigMap, which contains provider-specific settings.

In the example above, this part does that:

parametersRef:
  kind: ConfigMap
  name: gatewayclass-config

This allows the platform team to define default settings that should apply to gateways using this class.

Step 4: Who Usually Creates GatewayClass?

In most environments, GatewayClass is not something application developers create.

Instead, it is typically managed by the infrastructure or platform team.

Why?

Because GatewayClass defines things like:

• which controller is responsible

• default policies

• infrastructure-level configuration

These are decisions that usually affect the entire cluster, so they are typically standardized by the platform team to ensure consistency.

Application teams then simply reference the existing GatewayClass when creating their gateways.

2. The Gateway:

In the previous section, we looked at GatewayClass and saw that it acts like a template or blueprint for gateways. It tells Kubernetes which controller should handle the gateway and what kind of configuration it should follow.

But GatewayClass alone does not create anything in the cluster. It simply defines how gateways should behave.

The resource that actually results in something real being created, such as a load balancer, is the Gateway.

A Gateway is essentially the configuration that tells the controller:

“Create an entry point into the cluster that listens for traffic and routes it to applications.”

So if GatewayClass is the blueprint, the Gateway is the actual building plan that gets executed.

When a Gateway is created, the controller responsible for the referenced GatewayClass reads the configuration and provisions the infrastructure required to expose applications. In many environments, this results in a load balancer being created in the cloud.

Let’s look at an example Gateway resource.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: example-gateway
  namespace: ngf-gatewayapi-ns
spec:
  gatewayClassName: example-gatewayclass
  listeners:
  - name: http
    protocol: HTTP
    port: 80
    hostname: example.com
    allowedRoutes:
      namespaces:
        from: All

Let’s break down what’s happening here in a simple way.

Connecting the Gateway to Its GatewayClass

One of the first things you’ll notice in the specification is this field:

gatewayClassName: example-gatewayclass

This value matches the name of the GatewayClass we defined earlier.

What this means is:

This Gateway should follow the configuration defined in the example-gatewayclass, and the controller associated with that class should manage it.

Since the GatewayClass already tells us which controller is responsible, the controller now knows it should watch this Gateway resource and provision the required infrastructure.

Another interesting detail here is scope.

The GatewayClass is cluster-scoped, which means it does not belong to a specific namespace. That’s why when we defined it earlier, we did not include a namespace field.

However, Gateway resources themselves are namespaced.

In this example:

namespace: ngf-gatewayapi-ns

This means the Gateway exists within a specific namespace, even though it references a GatewayClass that exists at the cluster level.

Defining Listeners:

Another important part of the Gateway configuration is the listeners section.

listeners:
- name: http
  protocol: HTTP
  port: 80
  hostname: example.com

A listener defines how the gateway should accept incoming traffic.

You can think of a listener as a port on the load balancer that waits for incoming requests.

In this example, the configuration is saying:

• The gateway should listen for HTTP traffic

• It should listen on port 80

• It should accept requests for the hostname example.com

When the controller processes this configuration, it ensures that the resulting load balancer is configured to listen on port 80 for HTTP traffic directed at that host.

Allowed Routes: Solving a Major Ingress Limitation

One of the most interesting parts of this configuration is this section:

allowedRoutes:
  namespaces:
    from: All

Earlier when we were discussing Ingress, we saw that Ingress resources could only reference services from the same namespace. That made it difficult to expose applications across multiple namespaces using a single entry point.

Gateway API introduces a cleaner way to handle this.

With allowedRoutes, we can specify which namespaces are allowed to attach their routes to this gateway.

In this example, the configuration says:

Routes from all namespaces are allowed to attach to this gateway.

This means that the load balancer created by this Gateway can serve applications that live in multiple namespaces.

This directly addresses one of the key limitations we discussed earlier with Ingress.

Who Usually Creates Gateways?

In most environments, Gateways are typically created by cluster operators or platform teams.

Why?

Because a Gateway usually represents shared infrastructure, such as a load balancer that multiple applications might use.

Application teams then attach their routing rules to that gateway without needing to manage the infrastructure themselves.

3. Route Objects:

If you remember how Ingress worked, all the routing logic lived inside a single Ingress resource.

That Ingress manifest usually contained:

• The host or path rules

• The backend service

• The port to forward traffic to

For example, an Ingress rule might say:

If a request comes to /mobileapp, send it to the service mobileapp-svc.

That service is typically fronting a Deployment, and that deployment runs pods that serve the application. For example, it could be pods serving an application specifically designed for mobileapp users such as iPhone clients.

So in the Ingress world, all of this routing logic was bundled inside a single resource.

How Gateway API Handles Routing

Gateway API follows the same logical idea, but with a cleaner separation of responsibilities.

Instead of putting everything into one object, Gateway API splits the pieces into different resources with different responsibilities.

• GatewayClass → defines the gateway implementation and controller

• Gateway → creates the entry point (load balancer)

• Routes → define how traffic is routed to services

The Route objects are where the actual traffic routing rules live.

For HTTP traffic, the most common route resource is HTTPRoute.

An HTTPRoute defines rules like:

• What path or hostname to match

• Which backend service should receive the request

• Which port of the service should be used

Here is the example HTTPRoute.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: mobileapp-httproute
  namespace: mobileapp-ns
spec:
  parentRefs:
  - name: example-gateway
    namespace: ngf-gatewayapi-ns
  hostnames:
  - example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /mobileapp
    backendRefs:
    - name: mobileapp-svc
      port: 80

Let’s break this down step by step.

Connecting the Route to the Gateway

One important field in this configuration is parentRefs.

parentRefs:
- name: example-gateway
  namespace: ngf-gatewayapi-ns

This tells Kubernetes:

Attach this route to the Gateway called example-gateway that exists in the namespace ngf-gatewayapi-ns.

Defining the Routing Rule

Now let’s look at the actual routing rule.

matches:
- path:
    type: PathPrefix
    value: /mobileapp

This rule means:

If the request path starts with /mobileapp, this rule should match.

So if a request comes to:

example.com/mobileapp

or even

example.com/mobileapp/products

it will match this route.

Sending Traffic to the Backend Service

Once the request matches the rule, it is forwarded to the backend service defined here:

backendRefs:
- name: mobileapp-svc
  port: 80

This means the traffic will be sent to:

• Service: mobileapp-svc

• Port: 80

That service then forwards the traffic to the pods running the application.

Wrapping Up:

And that brings us to the end of the theory part of this post.

I realize that going through concepts like these purely through explanations can sometimes feel a bit overwhelming. Kubernetes documentation can be quite extensive, and when you're encountering something for the first time, it’s not always easy to connect all the pieces together immediately.

In this post, I tried to explain the ideas the same way I understood them while learning i.e. slowly breaking them down and focusing on the “why” behind the design decisions. My goal was simply to make the concepts a little easier to approach, especially if you’re seeing Gateway API for the first time.

That said, sometimes things only truly click once we see them working in practice.

So in the next part of this blog, we’ll move away from theory and walk through a hands-on demo, where we’ll actually set up these resources inside a Kubernetes cluster and see how they work together in a real environment.

And if some of the concepts in this post still feel a bit abstract right now, that’s completely fine, I’m fairly confident that once we go through the demo together, the pieces will start making a lot more sense.

To understand the integration, I needed to understand Cilium. And to understand Cilium, I needed to understand eBPF. And to understand eBPF… I had to go all the way down to the Linux kernel.

This series is not meant to be an expert guide. It’s not a replacement for official documentation. It’s simply a reflection of my learning process written in a way that I hope feels approachable, especially if you’re someone who is new to these concepts.

Sometimes when we read about networking, kernels, or Kubernetes internals, it can feel overwhelming. There are so many layers. So many abstractions.

If you’re curious about how modern Kubernetes networking works…and you’ve heard terms like Cilium or eBPF but never quite understood them…Or if you just enjoy learning by walking through someone else’s thought process…

Then I hope this journey resonates with you.

Let’s start at the beginning, with something called eBPF.

Before Cilium: Meeting eBPF for the First Time

What exactly is eBPF?

A simple way to think about it is this:

eBPF is a kernel technology that lets us inspect and manipulate calls made to the Linux kernel.

Let’s make it more concrete.

Imagine the Linux kernel as one giant API.

Every time you do something on your computer i.e. open a file, create a network socket, connect to the internet, change directories, create a file system, you’re indirectly making a call to that kernel API. Whether you’re inside a container or running something directly on the host, the kernel is always involved.

In a way, everything eventually talks to the kernel.

Now here’s where eBPF becomes interesting.

eBPF allows us to attach small programs to different points in that kernel API surface. When a specific event happens, like a network socket being opened, we can “catch” that moment. And once we catch it, we can decide:

• Do we just observe what’s happening?

• Do we collect context?

• Or do we influence what happens next?

And the most fascinating part is that this happens directly inside the kernel.

Why People Compare eBPF to JavaScript

eBPF is sometimes described as “JavaScript for the Linux kernel.”

Think about how JavaScript works in a browser.

When you open a web page, JavaScript is shipped along with it. That code runs inside the browser environment. It listens for events i.e. a button click, a page load, a form submission and when those events happen, it executes logic.

eBPF works in a somewhat similar way.

You write an eBPF program and decide where it should attach in the system stack. Once attached, whenever the relevant event occurs, the kernel runs that program.

So imagine this scenario:

A program maybe inside a container, maybe on the host, opens a network socket.
The kernel notices that event.
There’s an eBPF program attached to that type of event.
The kernel executes that eBPF program.
The result of that program helps determine what happens next.

It’s happening at the kernel layer itself.

Instead of adding more and more layers on top, eBPF works by embedding logic directly where events originate.

Cilium’s Journey With eBPF

What I found particularly interesting while learning about Cilium is that the project didn’t just use eBPF from day one in its current form.

In its early years, a significant focus was actually on helping eBPF mature to a place where something like Cilium could even be built on top of it.

In the beginning, eBPF wasn’t yet ready to support a full-fledged networking and security solution the way Cilium envisioned. There was groundwork to be done. Capabilities had to improve. The ecosystem needed to grow.

So part of Cilium’s journey was contributing to making eBPF production-ready for this kind of use case.

Because when you look at a polished networking solution in Kubernetes, you don’t always see the years of foundational work underneath. You don’t see the kernel-level innovation that made it possible.

But it’s there.

So, What Is Cilium?

Now that we’ve spent some time understanding eBPF, at least at an intuitive level we can finally ask the obvious question:

What is Cilium?

At its core, Cilium is a networking technology.

But that sentence alone doesn’t do it justice.

In the Kubernetes world, networking is about enabling pods to talk to other pods. It’s about allowing communication across nodes. Sometimes even across clusters. Sometimes across different cloud environments. And sometimes between cloud and physical or legacy environments.

In simpler terms, Cilium is focused on solving connectivity.

Connectivity between pods.
Connectivity between nodes.
Connectivity across clusters.
Connectivity across cloud and physical environments.

And not just connectivity, but doing it with a focus on security and observability as well.

Before understanding how Cilium does what it does, we had to understand something more foundational:

CNI.

Laying the Groundwork: What Happens When a Pod Is Created?

Let’s walk through something simple.

A pod is created in Kubernetes.

Behind the scenes, kubelet is responsible for making that pod actually come to life on a node. But part of that process is about giving that pod networking.

And this is where CNI comes in.

CNI (Container Network Interface) is the mechanism kubelet calls when it needs networking for a pod.

So imagine kubelet saying:

“Here’s a new pod. I need a networking stack for it.”

If Cilium is the configured CNI, kubelet calls Cilium at this moment.

What does that mean in practice?

It means Cilium needs to create the networking stack for that pod.

And at a very practical level, that networking stack involves creating something called a veth pair.

The veth Pair: Why You Only See One IP Inside a Pod

A veth pair is essentially two connected virtual network interfaces.

You can think of it like a virtual cable with two ends.

• One end lives inside the container’s network namespace.

• The other end lives in the host’s network stack (the underlying node).

So when a pod is created:

• One side of the veth pair is placed inside the pod’s network namespace.

• The other side remains in the host’s networking stack.

That’s why when you exec into a pod and run something like:

ip addr

You only see one interface (with one IP address). You don’t see all the host interfaces. You’re inside the pod’s own isolated network namespace.

That isolation is deliberate.

And Cilium is responsible for setting this up when kubelet makes the CNI call.

But Cilium doesn’t stop at just creating a veth pair.

Identity and eBPF; Where Cilium Becomes Different

When kubelet requests networking for a pod, Cilium:

1. Creates the veth pair

2. Places one end in the container

3. Places the other end in the host network namespace

But then it does something additional.

It generates an identity for that particular pod.

That identity is unique.

And then, this is the key part, Cilium attaches an eBPF program to the pod’s network namespace.

Now think back to what we discussed earlier about eBPF.

We said eBPF allows us to attach logic to kernel-level events.

So what Cilium is doing here is powerful:

Instead of handling networking decisions purely through traditional mechanisms, it attaches eBPF programs that operate at the kernel layer for that pod’s traffic.

So whenever traffic flows in or out of that pod, the eBPF logic can evaluate it.

So, now we are talking about:

• Enforcing identity-based decisions

• Observing traffic at the kernel layer

• Making networking decisions dynamically

All without adding heavy user-space components and this is where eBPF becomes the engine underneath Cilium’s networking model.

From Observing Traffic to Understanding It

So now we have something important in place.

When Cilium sets up a pod’s networking stack, it doesn’t just create a veth pair. It also attaches an eBPF program to that pod’s network namespace.

And because of that, we’ve already solved one big piece of the puzzle:

Observability.

Why?

Because now, anything that happens inside that pod’s network namespace can be seen.

Any new connection.
Any new packet.
Any network interaction.

All of it can be captured as an event stream.

Remember what we discussed about eBPF earlier? It allows us to “catch” kernel-level events. So when traffic flows in or out of that pod, the eBPF program can observe it directly at the kernel layer.

The first thing this gives us is visibility.

Whenever a network connection is made inside that namespace, a network event can be generated. Instead of guessing what’s happening or relying purely on external monitoring tools, we can see traffic at the source, right where it enters or leaves the pod.

But Cilium doesn’t stop at just observing.

From Observability to Control: Network Policies

In Kubernetes, there’s an important default behavior that often surprises beginners:

By default, every pod can talk to every other pod in the cluster.

There are no restrictions unless you explicitly define them.

This is where network policies come in.

Network policies are rules that restrict which pods can communicate with which other pods. They exist to enforce boundaries, because in real systems, not everything should be allowed to talk to everything else.

Cilium implements network policies using eBPF.

Just like we used eBPF to generate network events, we can extend that same mechanism to enforce policy.

So instead of only saying:

“Hey, I saw a connection happen.”

We can now say:

“I saw this connection attempt… should it be allowed?”

And because this logic runs in the kernel through eBPF, the evaluation happens at native kernel speed.

That’s an important detail.

We’re not sending traffic up to user space for evaluation. We’re not depending on an external proxy to make a decision. We’re evaluating policy directly inside the Linux kernel.

Once a network policy is defined, Cilium evaluates that policy for every packet that moves through that networking stack.

Every packet.

Which means the enforcement is extremely close to the traffic itself.

As Close to the Wire as Possible

If you think about where this enforcement is happening, it’s almost as close to the traffic as you can possibly get.

Because this logic is implemented at the kernel layer, you can think of it as allowing or denying traffic at the socket level i.e. even before it fully becomes just another packet moving through the system.

You’re affecting the traffic at its origin point.

We started with eBPF as a way to “inspect and manipulate kernel calls.”

Then we saw how Cilium uses it to attach logic to a pod’s network namespace.

First, that gave us observability.
Then, it gave us policy enforcement.

The Networking Side of eBPF: Reducing the Detour

So far, we’ve seen how eBPF helps with observability and policy enforcement.

But there’s another piece of the story and this one is about actual packet movement.

Let’s imagine something simple.

Two pods.
Same host.
They want to talk to each other.

You might assume that since they’re on the same node, communication would be straightforward and fast.

But traditionally, that’s not always the case.

Here’s what typically happens without something like eBPF handling things at the kernel layer:

1. A packet leaves the first pod.

2. It goes into the kernel.

3. It travels through user space mechanisms like iptables to evaluate network policies.

4. It gets routed appropriately.

5. It enters the network namespace of the second pod.

6. The second pod receives it.

7. And responses go back through the same path again.

Even though both pods are sitting on the same machine, the packet still goes through multiple layers of processing.

Each layer adds a little overhead.

And those small costs add up, especially at scale.

If two sockets on the same machine are trying to communicate, why should they take such a long path?

Short-Circuiting the Path With eBPF

This is where eBPF changes the game.

Because Cilium’s logic runs at the kernel layer, it understands what’s happening at the socket level.

If Pod A is trying to talk to Pod B, and both are on the same host, the eBPF program can see that.

It knows:

• Which sockets are involved.

• What identities are associated.

• What the network policy says.

So instead of letting the traffic travel down through multiple layers and come back up, we can evaluate the policy right there in the kernel and make a decision immediately.

If the policy allows it, the traffic can be short-circuited.

In other words, instead of sending packets through a longer processing path, we can allow those two sockets to communicate more directly.

This is often referred to as operating at the socket layer or even enabling socket-level load balancing.

We’re not waiting to process the packet after it has already traveled through multiple networking layers.

We’re making the decision as close to the origin of the traffic as possible.

And when this applies across many pods, many nodes, and potentially many clusters, the impact becomes significant.

And that’s the deeper story of how Cilium uses eBPF.

Hi everyone, I hope you’re doing well and finding a little time to breathe as the year comes to a close.

With only a handful of days left on the calendar, December always invites me to slow down a little. It’s my favorite month, it feels like the world slows down just enough to let us catch up with ourselves, space to rest after a long year, space to think about what we tried, what worked, and what didn’t.

Some people plan resolutions, some take breaks they’ve been postponing for months, and some simply allow themselves small joys in the form of good food, long conversations, or doing absolutely nothing without guilt.

I find myself reflecting a lot during this time. Thinking about where I started the year, how I felt back then, and how different or maybe similar I feel now.

So I decided to write this. To look back at 2025 and reflect on the moments that challenged me, the habits that slowly formed, and the growth that didn’t announce itself loudly but stayed with me anyway.

Before I talk about the “firsts” or the highlights, I wanted to begin here by saying that 2025, for me, wasn’t only about doing more or learning more. It was about becoming a little more aware, a little more consistent, and a little more confident than I was before.

Learning to Be Proud of Progress

As I look back, one of the things I feel most proud of this year is the progress I’ve made: not just professionally, but mentally and emotionally as well. It’s easy to overlook this kind of growth because it doesn’t come with certificates or announcements. But it changes how you show up every day, and that matters.

I wrote my very first blog on 16 June 2025.

Before 16 June 2025, if someone had told me I’d be writing blogs regularly, I probably would’ve laughed it off. I never saw myself as a writer. That label felt too big, too serious, like something meant for people who always knew what they were doing. I didn’t. And honestly, I was afraid of being judged. Afraid that what I wrote wouldn’t be good enough, or worse, wouldn’t be worth anyone’s time.

Still, the idea of writing kept coming back.

In the early days, the hardest part was this question that kept looping in my head: Why am I even doing this? There were moments when that doubt felt louder than any motivation. But I had already made a decision that I would keep showing up. No matter how the first few posts were received, or even if they weren’t received at all.

That decision mattered more than confidence ever could.

I started writing with the intention of learning in public. Not to teach, not to impress, just to document what I was figuring out as I went along. I held onto a quote from Techworld with Nana that felt uncomfortably close to home:

“Stop waiting to feel ready, stop waiting for the perfect moment: start messy, start scared, and start before you feel qualified.”

So I did. I wrote while unsure. I published while questioning myself.

Once I made that promise to myself, to keep publishing no matter what, the pressure eased a little. I wasn’t writing to prove anything anymore. I was writing because I had decided to show up.

In the beginning, consistency didn’t feel natural. Some days I’d sit in front of my screen wondering what to write, questioning whether my thoughts made sense, or whether anyone would relate to them. But instead of waiting for motivation, I leaned on routine. I wrote on days when I felt confident, and I wrote on days when I didn’t.

I noticed that I was clearer in how I understood new concepts, simply because I knew I’d try to explain them later. Learning felt more intentional. More honest.

Over time, the numbers added up: 48 blogs and still counting, by the end of the year. But what stayed with me was the discipline that formed in the background. The realization that consistency isn’t about pushing yourself endlessly; it’s about creating a rhythm that you can actually sustain. And without realizing it, that curiosity began nudging me toward something I had admired from afar for a long time.

It nudged me toward open source.

Taking the First Step Into Open Source

For a long time, open source felt like a world that existed somewhere far away from me.

Before this year, my understanding of open source was very limited. I knew Linux was open source, that much I had heard over and over again but beyond that, everything felt confusing. Big repositories, unfamiliar workflows, endless files, and conversations that seemed to assume you already knew how things worked. It felt vast, complex, and honestly, a little overwhelming. Too big to even know where to begin.

Out of curiosity, I started exploring what open source actually meant. I watched videos explaining the basics i.e. why open source exists, how communities work, and how people collaborate across the world. I spent time understanding what it means to contribute, learning that it’s not always about writing complex code. Sometimes it’s about documentation. Sometimes it’s about improving clarity. Sometimes it’s about fixing something small that helps someone else.

This journey led me to the Kubernetes project, the second largest open source project in the world after the Linux kernel. Even writing that sentence still feels a little unreal.

I didn’t choose it because it was easy. I chose it because it fascinated me. I watched KubeCon talks, beginner guides, and “how to get started with Kubernetes” videos, trying to make sense of this massive ecosystem. The more I learned, the more curious I became. And slowly, that curiosity replaced the fear of starting.

Eventually, I decided to try.

My first contribution wasn’t code-heavy. In fact, it was something very small: removing a duplicate paragraph from a blog.

You can check out my first contribution here.

On the surface, it might seem insignificant. But the moment my pull request was merged, it genuinely made my day.

It marked the beginning of my open source journey. It told me that I could participate. That I could contribute. That I didn’t need to be an expert to take the first step.

I never hesitated just because the contribution wasn’t code. By then, I had already understood that: no contribution is too small. Even showing up for yourself, choosing to be curious, choosing to learn, choosing to engage, is a contribution in its own way.

That first merged pull request gave me confidence, not in my skills alone, but in the idea that I belonged. That I could grow into this space.

And that’s what I hope anyone new to tech takes away from this: you don’t need to know everything to start. You don’t need to be perfect. Small contributions matter. Curiosity matters. And sometimes, all it takes is deciding to begin.

And while open source gave me a sense of belonging online, the next experience helped me feel that same connection in person, for the very first time.

Experiencing the Community in Person: My First CNCF Meetup

Up until this point, most of my learning and interactions had lived behind a screen. Talks, tutorials, discussions, everything happened online. I had spent hours watching KubeCon sessions, listening to people speak about communities, meetups, and the incredible work happening across the cloud native ecosystem. It always sounded exciting, but also distant, like something I was only meant to observe from afar.

That changed when I finally attended my first CNCF meetup in Kolkata.

When I entered the venue, the first thing I noticed was how quiet it was. There were only five or six people inside at that moment. I didn’t know anyone there, so I did the simplest thing I could i.e. I picked what felt like the best seat in the hall and sat down. (P.S. I’m the one in the red sweater)

Since this was my first in-person tech event, I went in without many expectations. I didn’t feel the need to prepare or prove anything. I just wanted to listen, learn, and experience what a community meetup actually felt like. I had a feeling the talks would be insightful and yes, I was also excited about the swag, but what stayed with me was so much more than that.

The sessions covered a wide range of topics, each offering a glimpse into different parts of the cloud native world. From intelligent workflows and chaos engineering to observability, security, and multi-architecture Kubernetes, every talk brought something new to the table. Even when some topics felt advanced, the way the speakers shared their experiences made them approachable and engaging.

There were moments of excitement too, like recognising tools I had only used in my local projects. I had used Grafana in my local Kubernetes projects before, and finally getting a sticker for something I had actually worked with felt surprisingly rewarding. And the tote bag, beautifully designed with Kolkata’s heritage, was a lovely touch too.

I went for the learnings and stayed for the swags but I left with something far more valuable.

For the first time, the community felt real. Not just names on slides or voices in videos, but people sitting next to me, asking questions, sharing experiences, and learning together. It reminded me that tech communities aren’t built only through code and conferences, they’re built through presence, conversations, and shared curiosity.

And just when I thought the day couldn’t get any more memorable, I had an experience that made the meetup truly unforgettable.

Meeting Someone I Admired: A Moment to Remember

One of the surreal moments of the meetup was getting the chance to meet Kunal Das in person. I had been following his work for quite some time, so seeing him there felt a little unreal.

A few days back, I attended an event where he spoke about migrating a live Minecraft server without any downtime. It was a fun and engaging talk, technical, yes, but explained in a way that made it easy to follow and genuinely enjoyable.

The idea of moving something live, something people are actively using, without disruption was fascinating, especially for someone still learning how large systems work behind the scenes.

What meant even more to me was getting the opportunity to have a short, casual conversation with him afterward. I thanked him for the work he’s doing and for sharing his knowledge so openly. We even managed to take a picture together! 👇

Encounters like this make you realize how accessible the community really is. The people you learn from online are often just as kind and approachable in person.

As I headed home that day, I found myself reflecting on how much had changed over the past year and how differently this December felt compared to the last.

Looking Back, and Looking Ahead

As I reflect on the year now, that contrast feels meaningful. December 2024 ended with me clearing my first AWS certification i.e. the AWS Cloud Practitioner. At the time, it felt like a big personal win, a sign that I was moving in the right direction.

And now, December 2025 ends with me attending my very first CNCF event, surrounded by people who are building, learning, and sharing in the open. Somewhere between those two Decembers, a lot changed.

This year has been about small beginnings turning into meaningful experiences.

Each “first” reminded me that growth doesn’t always arrive with dramatic fanfare, it often comes quietly, step by step, through curiosity, persistence, and the courage to show up.

I feel deeply grateful for everything I’ve experienced this year. For the curiosity that pushed me to start blogging. For the courage to start writing even when I wasn’t sure. For the confidence that came from small, consistent efforts. For the first open source contribution that showed me I belonged. And for a community that welcomed me, both online and in person.

I decided to write this blog because there are so many people out there doing incredible work: quietly, passionately and more people deserve to know about it. Communities like these need visibility. They need participation. They need students and working professionals alike to feel that they, too, have a place here.

And as I step into the next year, I carry that lesson with me: excited, grateful, and hopeful for all the learning and firsts still waiting ahead.

Here’s to a reminder that beginnings don’t need to be perfect, contributions don’t need to be massive, and showing up matters more than you think.

May 2026 be filled with more learning, more firsts, and more moments that shape us in ways we don’t yet imagine.

With that, I hope this season brings you rest, warmth, and joy.

Wishing you a Merry Christmas and a Happy New Year!

Hello everyone, and welcome back to Day 22.

If you’ve been following along on this learning journey with us, thank you for being here again. And if you’re just joining in, you’re very welcome too. This series has always been about learning together: without rushing through the fundamentals.

In the previous blog, we spent our time understanding AWS Policy and Governance using Terraform. We talked about how guardrails, permissions, and governance are not just “enterprise things,” but essential building blocks for any real system, no matter how small it starts. That foundation matters, because once we begin creating actual infrastructure, those decisions quietly shape everything that follows.

Today, we’re taking the next natural step.

Instead of talking only about policies and structure, we’re going to build something tangible i.e. a small but meaningful two-tier architecture on AWS using Terraform.

If a mini project carries this much complexity, just imagine what a real-world production setup looks like. This is also why challenges like our 30 days of learning exist, to expose us to these layers without overwhelming us all at once.

In this blog, we’ll walk through an architecture where:

• A web application runs on an EC2 instance

• A managed MySQL database lives securely in RDS

• Networking, security groups, and access boundaries are clearly defined

• Sensitive database credentials are handled safely, not hardcoded

We’ll move step by step, telling the story of why each piece exists, how it connects to the next, and how Terraform helps us keep everything organized through custom modules.

So take your time, read slowly, and don’t worry if some concepts feel new.

Before we dive in here’s a fact for today:

Fact #22: I really like having routines and a clear plan… but somehow, I rarely stick to them exactly as I imagine. Some days I’m on track, and other days I’m juggling a dozen things at once. That’s part of why I took up this #30daysofawsterraform challenge as it gave me a framework that I could follow along.

Understanding the Two-Tier Architecture We’re Building

Before we touch Terraform or look at any files, it’s important that we first picture the system in our minds. When things feel confusing later, and they will at some point, this mental picture is what helps everything fall back into place.

At a very high level, a two-tier architecture simply means we are separating responsibilities into two clear layers.

The first tier is what users interact with.
The second tier is where the data lives.

In our case, the first tier is a web server running on an EC2 instance, and the second tier is a MySQL database running on Amazon RDS. Both of these live inside AWS, but they live very different lives.

Let’s start from the user’s perspective.

A user sends an HTTP request from their browser. That request enters AWS through an Internet Gateway and reaches our web server, which is hosted on an EC2 instance inside a public subnet. This server runs a simple Flask application on Ubuntu, and it listens on port 80, just enough to serve requests and respond.

This web server is protected by a security group that allows:

• HTTP traffic on port 80 from the internet

• SSH access (something we should always restrict to our own IP)

This is the only part of our system that the outside world can see.

Now, here’s where the second tier comes in.

Behind the scenes, our Flask application needs a place to store data. For that, we use Amazon RDS running MySQL. This database does not sit in the public subnet. Instead, it lives inside a private subnet, completely shielded from direct internet access.

And this is a very intentional decision.

The database:

• Has no public access

• Accepts traffic only on port 3306

• Allows connections only from the web tier’s security group

In other words, even if someone knows the database exists, they cannot talk to it directly. The only trusted path to the database is through our web server.

There’s one more important piece that connects these two tiers in a secure way: AWS Secrets Manager.

Instead of writing database usernames and passwords inside our application or worse, inside Terraform files, we let Secrets Manager handle that responsibility. It generates the database password for us, stores it securely, and makes it available to the application when needed.

This is how our web server learns how to talk to the database without ever hardcoding sensitive information.

This is the essence of the two-tier architecture we’re building.

In the next section, we’ll start looking at how Terraform helps us organize all of this using custom modules, and how the root module ties everything together without becoming messy.

How We Organize This Infrastructure Using Terraform Modules

As soon as infrastructure grows beyond a few resources, things can get overwhelming very quickly. This is exactly why, in this project, we’re not placing everything into one large Terraform file. Instead, we’re using custom Terraform modules to keep each responsibility clearly separated.

If custom modules are still new to you, that’s completely okay. We’ve already explored them in detail earlier in this journey, especially in the blog where we implemented a real-time project using modules. The idea here is the same, we’re simply applying that structure to a two-tier architecture.

At the center of everything, we have the root module. This is the starting point.

When we open the root module, the first thing we define is the Terraform setup itself.

We specify the Terraform version we expect and the AWS provider version we want to use. This helps ensure consistency, especially when others pull the project and run it on their machines. We also configure the AWS provider using a region that comes from variables, keeping the setup flexible and reusable.

Once that foundation is in place, we begin calling our custom modules: one by one.

First module: Secrets module

Before we create servers or databases, we take a small but thoughtful pause to answer an important question:
Where should our database credentials live?

In many early projects, it’s common to see usernames and passwords written directly into code or configuration files. It works, but it also creates risk. In this setup, we deliberately choose a safer path by letting AWS Secrets Manager handle those sensitive details for us.

In the root module, the Secrets module is called very early.

Notice what we don’t pass into it, we never pass a password. The only database-related value we provide is the database username, along with the project name and environment. Everything else is handled automatically.

Inside the Secrets module, the first thing we do is generate a random password.

Terraform’s random_password resource allows us to define simple rules, such as:

• The password length

• Whether special characters are included

• Which special characters are allowed

This means the password is strong, unpredictable, and never manually created. Once generated, it exists only in Terraform’s state and in Secrets Manager, not in our code.

To avoid naming conflicts, we also generate a small random ID suffix.

This helps ensure that each secret has a unique name, even if we deploy the same project multiple times across environments.

With these pieces in place, we create an AWS Secrets Manager secret.

The name of the secret includes the project name, environment, and purpose, making it easy to identify later. We also add tags so the secret remains discoverable and organized inside AWS.

Next, we create a secret version, which is where the actual data lives.

Here, we store the credentials as a JSON structure. This JSON includes:

• The database username passed from the root module

• The randomly generated password

• The database engine (MySQL)

• A placeholder for the database host

Even though the database hasn’t been created yet, this is perfectly fine. Terraform allows us to define this structure now and connect the missing pieces later, once the database exists.

What’s important is the intent.
At no point do we hardcode secrets, and at no point does the application need to “know” where passwords come from. It simply consumes them securely when needed.

This approach gives us a few quiet but powerful benefits:

• Credentials are centrally managed

• Passwords can be rotated later without changing code

• Sensitive values are never exposed in plain text files

With secrets handled safely, we can now move forward with confidence.

Second module: VPC module.

This module is responsible for laying down the networking foundation. It creates:

• A VPC

• A public subnet for the web server

• Multiple private subnets for RDS, spread across availability zones

• An internet gateway

• Route tables and their associations

Because RDS is a managed service designed for high availability, it needs multiple private subnets.

That’s why we don’t just create one private subnet, we create more than one, ensuring resilience if an availability zone goes down.

It’s also important to remember that private subnets still need outbound access. This is where NAT gateways come into play.

They allow resources like RDS to communicate outward when required, without exposing them to inbound internet traffic.

Third module: Security groups

Here, responsibilities are clearly split:

• The web security group allows HTTP traffic on port 80 and SSH access (ideally restricted to our IP)

• The database security group allows MySQL traffic on port 3306, but only from the web security group

This setup ensures that the database trusts only the web tier and nothing else.

Fourth module: RDS module.

This module receives:

• Private subnet IDs from the VPC module

• The database security group ID

• Database name and username from variables

• The database password directly from the Secrets module

Notice how nothing is duplicated. Each module produces outputs, and the root module simply wires them together. This is how Terraform helps us build complex systems without losing clarity.

Final module: EC2 module

With the database securely running in the background, the final piece of our architecture is the web server. This is the part users interact with directly, and it’s created using the EC2 module.

When the EC2 module is called from the root module, it receives several important inputs from other parts of the system:

• The public subnet ID from the VPC module, so the instance knows where to live

• The web security group ID, so traffic rules are applied correctly

• The database endpoint from the RDS module

• The database credentials from the Secrets Manager module

Once again, nothing is duplicated. Each module provides exactly what it owns, and the EC2 module simply consumes what it needs.

Now, a natural question comes up here:
When does the application actually get deployed?

The answer is, at instance launch time.

Inside the EC2 module, we use a user data script. This script lives in a templates folder and is passed into the EC2 instance as part of its metadata. When the instance starts for the first time, this script runs automatically.

Within that user data script:

• Required system packages are installed

• Application dependencies are set up

• A simple Flask application is created

• Database connection details are injected into the application

This is how the application learns:

• Which database to connect to

• Which username and password to use

All of this happens without logging into the server manually.

The Flask application itself is intentionally simple. It has:

• A home page

• A health endpoint

• A database info endpoint

• Basic insert and read operations for messages

There are no delete or update operations here. The goal isn’t to build a full product, it’s to clearly demonstrate how a web tier and database tier communicate securely.

Once the EC2 instance is up and running, Terraform outputs the public DNS name of the server. This becomes the application URL that we can open in a browser.

At this stage, the entire flow is complete:

• Users access the application via the public DNS

• Requests hit the EC2 instance through the internet gateway

• The Flask app talks to RDS using secure credentials

• Data is stored safely in a private subnet

Everything we planned earlier is now working together.

Provisioning the Infrastructure and Testing the Application

At this point, all the pieces are defined. The modules are wired together, dependencies are clear, and Terraform understands the order in which everything needs to be created. Now comes the part where we simply let Terraform do its job.

From the project directory, we initialize Terraform. This step prepares the working directory, downloads the required provider, and gets everything ready for provisioning. Once that’s done, we run a Terraform plan.

The plan shows us exactly what Terraform is about to create. In our case, it lists all the resources i.e. VPC components, security groups, secrets, RDS, and the EC2 instance. Seeing this plan is reassuring. Nothing is hidden. Everything is explicit.

Once we’re comfortable with the plan, we apply it.

Terraform now starts creating resources in the correct sequence. You’ll notice that some resources come up quickly, while others, especially RDS, take a bit of time. This is expected. Managed database services don’t appear instantly, and Terraform patiently waits until they’re fully ready before moving on.

Eventually, the process completes, and all resources are provisioned.

Terraform then provides us with a few useful outputs:

• The RDS endpoint

• The public DNS name of the web server

• The application URL

This is our moment of truth.

We take the public DNS name and open it in a browser. The page loads, and we see the application running. This confirms a few important things all at once:

• The EC2 instance is reachable

• The Flask application started successfully

• Networking and security groups are working as expected

To test database connectivity, we try saving a message through the application. Once submitted, the message is stored in the MySQL database running in RDS.

And it works.

The message is saved, retrieved, and displayed, proving that:

• The web server can talk to the database

• Secrets were injected correctly

• The private subnet and security group setup is doing its job

The application also exposes a few helpful endpoints. There’s a health endpoint that confirms database connectivity, and a database info endpoint that shows details like the database name, host, and MySQL version. These endpoints help us verify that everything behind the scenes is wired together correctly.

What’s important here isn’t the complexity of the application. It’s the clarity of the architecture. We now have a fully functioning two-tier system:

• A public-facing web tier

• A private, secure database tier

• Clean separation of responsibilities

• Everything managed through Terraform

Before we wrap up, there’s one very important reminder.

Resources like RDS cost money. Once you’re done exploring and testing, make sure to destroy the infrastructure. Terraform makes this easy, and it’s a habit worth building early.

Conclusion

And with that, we’ve completed Day 22 of the 30 Days of Terraform Challenge. Today’s demo covered something really important: understanding how we can build a two-tier architecture step by step while keeping everything secure and modular.

Sometimes things just make more sense visually, and for that, here’s a helpful video by Piyush Sachdeva, which explains everything including the demo clearly, with some troubleshooting steps you can follow along.

We’ll continue our journey tomorrow, exploring the next steps together. Until then, happy terraforming!

Hello and welcome back to 30 Days of AWS Terraform.

In the previous part of this journey, we spent our time building Terraform custom modules for EKS. We focused on structure, reusability, and how to keep our infrastructure code clean and manageable as things start to grow. That work was important and it gave us a solid foundation and showed us how real-world teams avoid repeating themselves when managing cloud infrastructure.

But as our setup grows, something interesting starts to happen.

It’s no longer just about creating resources. It’s also about making sure everyone uses them in the right way.

This brings us to Day 21, where we take the next natural step in our real-world project: AWS Policy and Governance using Terraform.

Think of it this way, once multiple people are working in the same AWS environment, we need a shared understanding of what’s allowed and what’s not. We need some guardrails. Not to slow anyone down, but to protect the system, the data, and the team itself. This is where policy and governance step in and do their job.

In this blog, we’re building a real-world audit and compliance setup, the kind you’d actually see inside an organization. Step by step, we’ll look at how we can:

• Prevent certain risky actions before they happen

• Detect resources that don’t follow our standards

• Keep a clear, secure record of what’s happening in our AWS account

And we’ll do all of this using Terraform, so the entire setup is automated, repeatable, and easy to understand.

Let’s take this next step together and start putting some thoughtful rules and visibility around the infrastructure we’ve already built.

Why Policy and Governance Matter (and Why We Need Both)

Before we touch any Terraform files or AWS services, it’s important that we understand what we really mean by policy and governance. These words often sound heavy or intimidating, they’re actually very simple and very practical.

In any real-world setup, whether it’s a company, a team, or even a shared home, there are always some basic rules. These rules aren’t there to restrict creativity; they’re there to keep things safe, predictable, and manageable. The same idea applies to our AWS environment.

Policy and governance are two sides of that same idea.

Let’s start with policy.

When we talk about policy in AWS, we’re really talking about rules that are enforced upfront. These rules decide what actions are allowed and what actions should be blocked. If something breaks the rule, it simply doesn’t go through.

For example, imagine a rule that says:
“If a user does not have Multi-Factor Authentication (MFA) enabled, they should not be allowed to delete important resources.”

That’s a policy. The moment someone tries to perform that action without MFA, AWS checks the rule and immediately denies the request. Nothing happens. The system protects itself.

Another example could be around data security. Let’s say we want to make sure that whenever someone uploads a file to an S3 bucket, that data is secure while it’s being transferred. In simple terms, that means uploads must happen over HTTPS, not plain HTTP. If someone tries to upload data in an insecure way, the request is blocked right there.

We can even create policies around tags. For instance, we might decide that every resource created in AWS must have an Environment tag like dev, staging, or prod. If someone forgets to add it, the resource creation fails. This helps keep things organized as the environment grows.

All of these are examples of IAM policies. They act like strict gatekeepers. If the request doesn’t meet the rules, it doesn’t get through. Simple as that.

Now, here’s where governance comes in.

Governance doesn’t stop actions before they happen. Instead, it focuses on visibility and accountability. It helps us answer questions like:

• Are our resources following the rules?

• What changed, and when?

• Which resources are compliant, and which ones are not?

In our real-world project, we’ll handle governance using AWS Config.

AWS Config continuously looks at the resources in our account and checks them against a set of rules. If a resource doesn’t follow a rule, AWS Config doesn’t block it, but it records it as non-compliant.

For example, an S3 bucket might get created without encryption. AWS Config will allow it, but it will flag it as non-compliant. That way, we always know what needs attention. This is extremely important for audits, security reviews, and long-term maintenance.

So to put it simply:

• Policy prevents bad actions from happening

• Governance keeps track of what’s happening and what needs fixing

In this project, we’ll use both together. We’ll prevent risky behavior using IAM policies, and we’ll monitor and record compliance using AWS Config. Finally, we’ll store all of this audit and compliance data securely in an S3 bucket.

This combination is what makes the setup feel real and practical, exactly how things are done in real organizations.

Laying the Foundation: Setting Up a Secure Audit Bucket

Before we start enforcing rules or checking compliance, we need a safe place to store our records. In any real-world organization, audits and compliance data are treated very carefully. These logs tell the story of what happened in the environment, and losing them, or exposing them can be risky.

So in our project, the very first thing we do is create a dedicated S3 bucket that will store:

• AWS Config audit logs

• Compliance and non-compliance details

• Configuration history over time

This bucket becomes the backbone of our governance setup.

We define all of this inside our main.tf file.

We start by creating the S3 bucket itself. To make sure the bucket name is unique (since S3 bucket names are global), we attach a random suffix at the end. This way, we don’t have to worry about name conflicts, and the bucket still clearly belongs to our project.

Once the bucket is created, we immediately add tags to it. These tags help us identify the purpose of the bucket and keep it aligned with the rest of our infrastructure. Tagging might feel small right now, but it becomes extremely valuable as environments grow.

Next, we enable versioning on the bucket.

Versioning ensures that even if a file is overwritten or deleted, previous versions are still preserved. For audit data, this is especially important. We never want to lose historical information about what happened in the past.

After that, we enable server-side encryption on the bucket.

Here, we’re telling AWS that every object stored in this bucket must be encrypted at rest. We use a standard encryption algorithm provided by AWS, which means the data is protected automatically without any extra effort from users or services writing to the bucket.

Then comes one of the most important steps: blocking public access.

This is an audit bucket. It should never be exposed to the internet. So we explicitly block:

• Public ACLs

• Public bucket policies

• Any form of public access

By setting all of these to true, we make sure the bucket stays private by default. This aligns with best practices and prevents accidental exposure.

Now that the bucket itself is secure, we attach a bucket policy.

This policy serves two main purposes.

First, it allows the AWS Config service to interact with the bucket. AWS Config needs permissions to:

• Check the bucket’s ACL

• List the bucket

• Upload audit files

Without this policy, AWS Config wouldn’t be able to store its data.

Second, the bucket policy also denies any insecure traffic. If someone, or some service, tries to access the bucket over HTTP instead of HTTPS, the request is denied. This enforces encryption in transit and ensures data is always transferred securely.

At this point, we’ve done something very important for our real-world project.

We haven’t enforced any rules yet. We haven’t checked compliance yet.
But we’ve created a secure, private, versioned, and encrypted foundation where all governance-related data will live.

With this base in place, we’re now ready to move into the next part of the story i.e. using IAM policies to prevent risky actions before they happen.

Putting Guardrails in Place: Enforcing Rules with IAM Policies

Now that we have a secure audit bucket in place, we move to the next important layer of our real-world project: preventing risky actions before they happen.

This is where IAM policies come into the picture.

In simple terms, IAM policies act like strict rules. If a request doesn’t meet the rule, AWS doesn’t even allow the action to happen. There’s no warning, no log first, the request is just blocked. This makes IAM policies perfect for enforcing non-negotiable security requirements.

Let’s start with the first policy.

Enforcing MFA for Deleting S3 Objects

Deleting data from S3 is a sensitive action. In real environments, we never want this to happen casually. That’s why our first policy ensures that MFA must be enabled before any S3 object can be deleted.

Here’s the Terraform code for that policy:

Let’s slowly understand what’s happening here.

We’re creating a custom IAM policy. The effect is set to Deny, which is important, in AWS, an explicit deny always wins. The action we are targeting is s3:DeleteObject, and we apply it broadly to all resources.

The real heart of this policy is the condition.

AWS checks whether aws:MultiFactorAuthPresent is false. If the user does not have MFA enabled, the delete request is denied. If MFA is enabled, the policy does nothing and allows the request to proceed.

This single rule already removes a huge amount of risk from our environment.

Enforcing Encryption in Transit for S3 Uploads

Next, we focus on protecting data while it’s being transferred.

Whenever someone uploads an object to S3, that data travels over the network. We want to make sure it always travels securely using HTTPS.

Here’s the policy that enforces that:

This policy is very direct.

If someone tries to upload an object using insecure transport, meaning HTTP instead of HTTPS, AWS sets aws:SecureTransport to false. When that happens, the policy denies the request.

The user doesn’t have to remember anything. The rule enforces secure behavior automatically.

Requiring Tags When Creating EC2 Instances

As environments grow, untagged resources quickly turn into chaos. So in real-world projects, tagging is not optional, it’s enforced.

Here’s the policy that makes sure EC2 instances are created with the right tags:

This policy does two checks during EC2 creation.

First, it checks whether the Environment tag exists and whether its value is one of dev, staging, or prod. If not, the request is denied.

Second, it checks whether the Owner tag exists at all. If it’s missing, the request is denied.

Together, these rules ensure that every EC2 instance has context and accountability from day one.

Creating a Demo User to See This in Action

To demonstrate how these policies behave, we create a simple IAM user:

And then we attach the MFA delete policy to this user:

This helps us clearly see how the policy behaves when MFA is missing versus when it’s enabled.

At this point, we’ve done something powerful.

We’ve stopped certain bad actions from ever happening.
But we still don’t know whether all existing resources are compliant.

That visibility comes next, and that’s where AWS Config enters the story.

Keeping an Eye on Everything: Introducing AWS Config

So far in our real-world project, we’ve put strong guardrails in place using IAM policies. Certain risky actions are now blocked before they can even happen. That already gives us a safer environment.

But in real organizations, prevention alone is not enough.

We also need visibility.

We need to know:

• What resources exist in the account?

• Are they following our standards?

• If something drifts from what we expect, can we see it clearly?

This is where AWS Config becomes an essential part of governance.

AWS Config does not stop actions. Instead, it observes, records, and evaluates what’s happening in the account.

It continuously checks resources against predefined rules and tells us whether they are compliant or non-compliant.

To make AWS Config work, we need three things:

1. A role that AWS Config can assume

2. A recorder to capture configuration changes

3. A delivery channel to store the data

Let’s walk through each of these, one step at a time.

Giving AWS Config the Right Permissions

Even though we already allowed AWS Config access to our S3 bucket using a bucket policy, AWS Config itself still needs permission to act inside our account. For that, we create an IAM role that AWS Config can assume.

Here’s the role definition:

This role is very focused. We’re simply telling AWS:
“Allow the AWS Config service to assume this role.”

Next, we attach the AWS-managed policy that gives AWS Config its standard permissions:

This managed policy allows AWS Config to evaluate resources, record configuration changes, and perform its core functionality.

But we’re not done yet.

AWS Config also needs permission to write data into our audit S3 bucket. For that, we attach an additional inline policy:

This policy allows AWS Config to check bucket versioning and read and write objects inside the bucket. Together with the bucket policy we created earlier, this completes the permission setup cleanly and securely.

Turning On AWS Config Recording

Now that permissions are in place, we can tell AWS Config what to record.

We start by creating a configuration recorder:

Here, we’re asking AWS Config to record:

• All supported resource types

• Global resources as well (like IAM)

This ensures broad visibility across the account.

Next, we define a delivery channel, which tells AWS Config where to store the recorded data:

We point it to the secure audit bucket we created earlier. This connects everything together.

Finally, we enable the recorder:

This step is important. AWS Config does nothing until the recorder is enabled. We also make sure it only starts after the delivery channel exists, so data has somewhere to go.

At this point, AWS Config is alive and watching.

But it’s not evaluating anything yet.

That’s where Config Rules come in and that’s what we’ll build next.

Checking What’s Compliant: Defining AWS Config Rules

Now that AWS Config is up and running, recording everything and sending data to our secure audit bucket, it’s time to answer the most important governance question:

Are our resources actually following the rules?

This is where AWS Config rules come into play.

Config rules don’t block anything. Instead, they continuously evaluate resources and tell us whether they are compliant or non-compliant. This distinction is important. IAM policies prevent actions, while Config rules observe and report.

In real-world environments, this combination is powerful. We prevent critical mistakes upfront, and we detect everything else so it can be reviewed, fixed, and audited later.

Let’s walk through the rules we define in this project.

Making Sure S3 Buckets Don’t Allow Public Write Access

The first rule ensures that no S3 bucket allows public write access.

This is an AWS-managed rule, which means AWS already knows how to evaluate it. We simply reference it using the source identifier.

Once this rule is active, AWS Config continuously checks every S3 bucket. If any bucket allows public write access, it is immediately flagged as non-compliant.

Ensuring S3 Buckets Are Encrypted at Rest

Next, we make sure that all S3 buckets have server-side encryption enabled.

If a bucket is created without encryption, AWS Config doesn’t block it, but it records it as non-compliant. This gives us visibility and a clear action item to fix the issue.

Blocking Public Read Access on S3 Buckets

Public read access can expose sensitive data unintentionally. So we add another rule to detect that:

Together with the previous rule, this ensures that S3 buckets are neither publicly writable nor readable.

At this stage, something important has happened in our project.

• IAM policies are preventing risky actions

• AWS Config is recording everything

• Config rules are telling us what’s compliant and what isn’t

• All audit data is stored securely in our S3 bucket

This is exactly how policy and governance come together in real environments.

Conclusion

As we reach the end of Day 21 of our 30 Days of AWS Terraform journey, it’s worth taking a quiet moment to look back at what we’ve built together in this real-world project.

We designed a thoughtful policy and governance setup, the kind that real teams rely on every day to keep their cloud environments safe, visible, and manageable.

This was a slightly longer blog, and that’s completely okay. You don’t have to absorb everything in one go. I chose not to shorten it because policy and governance have many moving pieces, and writing them down step by step helps show how everything connects in a real setup. Even skimming through and coming back later is perfectly fine.

If you feel that things make more sense when you see them happening rather than just reading about them, there’s also a video by Piyush Sachdeva that walks through the entire setup, including the EKS demo, in a very clear and practical way. It’s a great companion if you want to follow along visually or understand some common troubleshooting steps.

That’s it for Day 21.

Thank you for learning along with us, for taking the time to understand not just the how, but the why. Tomorrow, we’ll be back with Day 22, where we’ll continue building more hands-on, real-world projects and deepen our understanding step by step.

Until then: keep learning, keep experimenting, and keep moving forward.

Welcome back to Day 20 of our 30 Days of AWS Terraform journey.

Last day, we explored how Terraform provisioners help us go beyond simply declaring infrastructure and how they allow us to bridge that small but important gap between creating resources and actually preparing them for real use.

With only 10 days left in our 30-day challenge, it feels like the perfect moment to start thinking about structure, organization, and reusability.

Up until now, most of our Terraform work lived comfortably inside a single folder, with files like main.tf, variables.tf, and outputs.tf sitting together and doing their job. That approach is perfectly fine when we’re learning or experimenting. But as our projects start to look more like something we’d run in the real world, a question naturally begins to surface: how do we keep this clean, reusable, and manageable as it grows?

This is where today’s topic enters the picture. In this blog, we will not include the full EKS cluster setup, as going through the entire cluster creation here would be overwhelming. Instead, we’ll focus on understanding modules and how they make complex Terraform projects manageable, using simplified examples that illustrate the concepts clearly.

We’ll understand why modules exist, what problem they solve, and how they help us structure Terraform code in a way that makes sense for real-world use. Think of today as the moment where everything we’ve learned so far starts to come together in a more organized, intentional way.

By the end of this blog, we’ll have infrastructure that’s thoughtfully designed, reusable, and easy to extend.

Before we dive in, a little about myself:

Fact #20: I love asking “why?” more than I probably should. I find myself questioning things constantly, why a process works a certain way, why a decision was made, why one approach is chosen over another. Sometimes it can be a bit much for people around me. :P

What are modules?

Before we talk about custom modules, it’s important that we slow down for a moment and understand something more fundamental: what Terraform modules actually are.

At its core, a module in Terraform is simply a reusable piece of code. That’s it. It’s just a way of grouping Terraform configuration so we can use it again without rewriting the same logic over and over.

Now, a natural question comes up at this point: how is this different from a function in a programming language? And that’s a very good question.

Terraform isn’t a fully-fledged programming language. It’s a configuration language, which means it gives us a limited set of built-in functions, but it doesn’t allow us to define our own functions the way we would in languages like Python or Java. So when we want reuse, not just reuse of values, but reuse of entire infrastructure logic, Terraform gives us modules instead.

Modules let us take a chunk of infrastructure logic, package it together, and reuse it wherever we need. They help us encapsulate complexity. Instead of looking at dozens of resources every time we open our configuration, we can hide that complexity behind a module and interact with it using just a few well-defined inputs and outputs.

This becomes incredibly important as soon as we move beyond small demos. Think about something like a VPC. A real VPC isn’t just one resource, it usually includes subnets, route tables, gateways, and more. Writing all of that repeatedly would be tiring, error-prone, and hard to maintain. A module allows us to define that complexity once and then simply use it wherever needed.

And this is where our journey today really begins. Since we’re building a production-grade EKS setup, we’ll have multiple moving parts i.e. networking, IAM, authentication, secrets, and the cluster itself. Each of these can live in its own module, neatly organized and clearly separated. Instead of one massive Terraform file, we’ll have a structure that feels intentional and easy to reason about.

So as we move forward, keep this simple idea in mind: a Terraform module is just a reusable container for Terraform code.

Types of Modules

Now that we’re comfortable with what a module is, let’s take the next small step and talk about the different types of modules we’ll come across in Terraform. This is an important part of the story, because it helps us understand why we’re choosing to build our own custom modules in this project instead of relying entirely on what already exists.

When we explore the Terraform ecosystem, we’ll quickly notice that not all modules are created or maintained in the same way. Broadly speaking, Terraform modules fall into three categories: public modules, partner modules, and custom modules.

Let’s start with public modules. These are modules that are typically published and maintained by Terraform providers themselves i.e. providers like AWS, Google Cloud, Azure, and others. We’ll find them listed in the Terraform Registry, and they’re designed to solve common infrastructure problems in a standardized way. For example, there are public modules for VPCs, EKS clusters, IAM configurations, and many other services. These modules are widely used, well-tested, and a great way to get started quickly.

Closely related to public modules are partner modules. These are created and maintained by organizations that have an official partnership with HashiCorp. When we browse the Terraform Registry and look at the available filters, we’ll notice provider filters such as AWS, Azure, Google, Helm, and others and within those, we’ll also see modules labeled as partner modules. These are jointly maintained and follow certain standards agreed upon with HashiCorp. In simple terms, partner modules sit somewhere between community contributions and official provider-backed modules.

Both public and partner modules are incredibly useful, especially when we want to move fast or follow widely accepted patterns. But this naturally leads to a question that many beginners ask at this point: if these modules already exist, why would we ever need to create our own?

This is where custom modules come into the picture.

A custom module is a module that’s not managed by a provider or a partner. It’s created and maintained by us, our team, or our organization. Anyone can create a custom module. The process is straightforward: we write the Terraform code, place it in its own folder or repository, and optionally publish it to a GitHub repository. From there, we can tag releases, manage versions, and evolve the module over time.

One of the biggest advantages of custom modules is control. With public or partner modules, the design decisions are made by someone else. If something breaks, or if a change doesn’t align with our requirements, we’re dependent on the maintainers to fix or update it. With a custom module, we own the entire lifecycle. We decide what goes in, what stays out, and who is allowed to modify it.

Custom modules also allow us to lock down certain values intentionally. If there are configurations that shouldn’t be changed casually, especially in production, we can hardcode or tightly control them inside the module itself. That way, anyone using the module interacts only with the inputs we expose, and any deeper changes require deliberate updates to the module code.

This is exactly why most real-world organizations rely heavily on custom modules. They provide consistency, enforce standards, and reduce accidental changes, all while keeping Terraform code clean and reusable.

How Terraform organises modules?

Now that we understand why custom modules matter, let’s shift our focus to how Terraform actually organizes modules. This part is especially important, because once the structure becomes clear, everything else starts to feel far less mysterious.

Whenever we write Terraform code, no matter how small or simple, it always lives inside a module. Even if we don’t explicitly create one, Terraform still treats our configuration as a module. The folder where Terraform execution begins is known as the root module.

Think back to what we’ve been doing so far in this series. We usually created a directory, and inside it we had files like main.tf, variables.tf, outputs.tf, maybe a backend.tf or a .tfvars file. All of that together formed our Terraform project. Even though we never called it a module, that entire folder was actually the root module.

So when we say “root module,” we’re simply referring to the top-level Terraform configuration, the place where terraform init, terraform plan, and terraform apply are executed. This is where Terraform starts reading and understanding what we want to build.

Now, when we move into more realistic projects, like the one we’re building today, we don’t want everything to live in that single folder. Instead, we break things down into smaller, focused pieces. And this is where custom modules come in.

Inside our root module, we create subfolders. Each of these subfolders represents a custom module, and each one is responsible for a specific part of the infrastructure. For example, in this Day 20 project, our root module lives at the top level, and inside it we have a modules directory. Within that directory, we create separate folders for things like VPC, EKS, IAM, and Secrets Manager.

(add image)

Each of these module folders looks very familiar. Just like the root module, they contain files such as main.tf, variables.tf, and outputs.tf. The difference is not in the file names, but in their role. The root module orchestrates everything, while the custom modules focus on doing one thing well.

So, at a high level, the structure looks something like this: we have a root module that serves as the entry point, and inside it, multiple custom modules that handle networking, security, and the Kubernetes cluster itself. The root module pulls these modules together and passes values to them, while the modules return outputs back to the root module.

This relationship is very similar to calling a function with parameters. The root module provides the inputs, the module performs its work, and then it returns outputs that other parts of the configuration can use.

First Module: VPC

We’ll start small and focused, because this is how custom modules truly make sense.

Imagine we’re at Day 20, and this is our project layout:

The code/ directory is our root module.
The modules/vpc/ directory is our first custom module.

Step 1: Writing the VPC logic inside the custom module

Let’s go inside modules/vpc/main.tf.

This file contains the actual infrastructure logic. In our case, we start with the VPC itself:

Now pause here for a second and notice something important.

We are not hardcoding values like the CIDR block or the name.
Instead, we’re using variables such as var.vpc_cidr and var.name_prefix.

This tells us something very clearly:

This module expects someone else to provide these values.

That “someone else” is the root module.

Step 2: Declaring what the module expects (variables)

Since we’re using var.vpc_cidr, we must define it.
That happens inside modules/vpc/variables.tf

This file is not assigning values.
It’s simply saying:

“If you want to use this VPC module, these are the values you must give me.”

This separation is one of the biggest reasons modules feel clean and professional.

Step 3: Defining values in the root module

Now we move back to the root module, inside code/variables.tf (check the friendly project layout above)

Here’s where the values actually come from:

This is the moment where things start to click.

The same variable names exist in both places:

• Declared in the module

• Defined in the root module

Terraform wires them together for us.

Step 4: Calling the custom module from the root module

Now comes the most important part: using the module.

Inside code/main.tf, we write:

Let’s slow down:

• module "vpc" → we are creating an instance of our custom module

• source = "./modules/vpc" → this tells Terraform where the module lives

• Everything below that → values being passed into the module

This is exactly like calling a function and passing arguments.

The module doesn’t care where the values came from.
The root module doesn’t care how the VPC is built internally.

Each has a single responsibility and that’s what makes this production-ready.

At this point, we’ve achieved something very important:

• The VPC logic is encapsulated

• The root module stays clean and readable

• We can reuse this VPC module anywhere

• We can control changes by modifying the module itself

Extending the VPC module: Subnets and Availability Zones

Let’s continue exactly where we left off.

Step 1: Deciding where logic should live

Before we write any code, we ask an important design question:

Should the VPC module decide which availability zones to use, or should the root module decide?

For production-grade projects, the answer is usually:
the root module decides, and the custom module consumes.

This keeps the module reusable across regions and environments.

Step 2: Fetching availability zones in the root module

Inside the root module (code/main.tf), we add a data source:

We don’t hardcode zone names. Instead, we ask AWS what’s available in the current region.

Now we take only the first three zones:

This locals block is just helping us keep things readable.
At this point, the root module knows which availability zones to use.

Step 3: Passing availability zones into the VPC module

We now pass these availability zones into the VPC module:

Notice something subtle but important.

azs, public_subnets, and private_subnets are not AWS resource arguments.
They are inputs to our custom module.

Terraform allows this because we define what these values mean inside the module.

Step 4: Declaring these variables inside the VPC module

Inside modules/vpc/variables.tf, we add:

Again, no values here, just expectations.

Step 5: Using these values inside the VPC module

Now comes the part where everything connects.

Inside modules/vpc/main.tf, we create public subnets:

Let’s slow this down.

• count lets us create multiple subnets

• Each subnet gets:

  • a CIDR from public_subnets

  • an AZ from azs

• The index ties them together

This is a very common real-world pattern, and it works beautifully with modules.

Step 6: Why this is such an important moment

At this point, something fundamental has happened:

• The root module controls decisions

• The custom module executes logic

We’ve built a clean contract between the two.

If tomorrow we want to change regions, availability zones, or subnet layouts, we don’t touch the module logic, we only change inputs.

That’s the real power of custom modules.

Conclusion

And with that, we’ve completed Day 20 of the 30 Days of Terraform Challenge!

Today’s demo covered something truly important: the idea of custom modules. While it might sound technical at first, modules are really just a way to simplify our lives, they let us encapsulate complexity, reuse code, and maintain control over our infrastructure.

We also explored how modules interact with root modules, passed values using variables, and saw some hands-on code examples that demonstrate the concept. I intentionally did not include the full EKS cluster setup in this blog, because going through the entire cluster creation here would have been overwhelming. But don’t worry, there’s a video by Piyush Sachdeva that explains everything, including the complete EKS demo, with helpful troubleshooting steps along the way.

So take your time to digest these concepts, play around with the examples we went over, and feel free to experiment. Tomorrow, we’ll continue our journey and explore the next exciting step in the challenge.

Happy Terraforming, and see you in the next session!

Hello and welcome back to 30 Days of AWS Terraform. This is Day 19 of the series.

If you’ve been following along from the early days, give yourself a quiet pat on the back, showing up consistently and learning something new every day is no small thing.

So far, we’ve covered quite a lot. We didn’t just talk about concepts in isolation; we slowly built them up and even worked through a few mini projects and a couple of real-world–style scenarios. Most recently, we spent time on a mini project around Image Processing using AWS Lambda, where we saw how serverless components can work together to process images efficiently. That project was an important step because it helped us move from “writing Terraform code” to actually thinking in Terraform, orchestrating multiple resources to achieve a real task.

Today, we’re stepping into another concept that often comes up once our infrastructure starts feeling more real. We can think of this as another mini project if you like, or simply as a focused learning exercise that adds one more useful tool to our Terraform toolbox. We’ll be talking about Terraform provisioners.

Provisioners tend to show up right at that moment when creating infrastructure alone is not enough. The reason we’re learning this topic is simple, sometimes creating resources is not enough. Once an EC2 instance, server, or other infrastructure exists, we often need to perform additional tasks automatically, like running commands, installing software, or copying files. Provisioners give us a safe, structured way to handle these tasks without manually logging into the server each time. i.e. maybe run a command, copy a file, or prepare a server just enough so that it’s ready for use.

By the end, provisioners would feel like another practical option we can reach for when the situation calls for it.

Fact #19: I usually avoid plain socks and prefer ones with small patterns, subtle designs, or little motifs. Maybe a tiny spaceship, a little Mickey Mouse logo, or something fun like that.

What is a Provisioner?

Let’s start with a very simple question.

What exactly is a provisioner in Terraform?

At its core, a provisioner is something that performs a task. That’s it. A task could be as small as running a single command, executing a script, copying a file, or doing some kind of operation at a particular moment in time. When Terraform creates or recreates a resource, a provisioner gives us a way to say, “Now that this thing exists, please do this extra step.”

This idea becomes easier to grasp if we think back to what we’ve already been doing. Until now, most of our Terraform work has focused on creating infrastructure i.e. VPCs, subnets, security groups, EC2 instances, and so on. Terraform is excellent at that. But sometimes, creating the resource is only part of the story. Sometimes we also want to prepare that resource in a small way.

That’s where provisioners come in.

Terraform gives us three types of provisioners, and each one is meant for a slightly different situation. Understanding which one to use and why is much more important than memorizing syntax. So we’ll take them one at a time and connect them back to real situations.

The first thing to keep in mind is that all provisioners exist to do some work, not to define infrastructure itself. They are helpers, not the foundation. With that mental model in place, the differences between them start to make sense.

The three provisioners we’ll be working with today are:

• local-exec

• remote-exec

• file

Let’s begin with the local-exec provisioner, because it’s the easiest place to start and helps set the stage for everything that comes after.

What is a Local Provisioner?

Let’s talk about the local-exec provisioner.

As the name suggests, local-exec is used when we want to run a command locally. And by locally, we mean on the machine where Terraform itself is running. In our case, throughout this series, we’ve been running Terraform from our own laptops. For example, if you’re following along on a MacBook or a personal system, that machine becomes the “local” environment for Terraform.

So when we use local-exec, Terraform is not reaching out to AWS, and it’s not logging into any EC2 instance. It’s simply running a command right there on our own system.

This is an important distinction to pause on.

If we think about how we’ve been working so far, every time we run terraform apply, those commands are already being executed locally. The local-exec provisioner just gives us a structured way to attach an extra local command to the lifecycle of a resource. It might be logging something, printing output, or triggering a small helper script.

To see this in action, let’s build on the infrastructure we already created.

We already have our provider block in place:

We also use a data source to fetch the Ubuntu AMI, because we don’t want to hardcode image IDs:

This gives us a clean and reliable way to always get the latest Ubuntu image. Nothing new here, we’ve already seen this pattern before.

Next, we create a security group that allows SSH access, simply so we can connect to the instance later:

With that in place, we move on to creating the EC2 instance itself:

Up to this point, everything should feel familiar. We’re pulling the AMI from the data source, reading values from variables, and attaching the security group we just created.

Now, before we add any provisioners, we need one more important piece: the connection block.

Even though local-exec itself doesn’t need SSH, this connection block becomes important as we move forward. It tells Terraform how to connect to this specific resource. The keyword self simply means “this resource right here.” So when we say self.public_ip, we’re referring to the public IP of this EC2 instance.

Now comes the local-exec provisioner itself:

Let’s slow down and really look at what’s happening here.

The command field is just a normal shell command. We’re using echo to print a message. Inside that message, we’re interpolating values from the EC2 instance itself, like the instance ID and its public IP. Even though the command runs locally, Terraform already knows about the resource, so it can safely substitute those values.

When this provisioner runs, the command is executed on our local machine, not on the EC2 instance. That’s the key idea to remember.

If we run terraform apply at this point, Terraform may say there are no changes, because provisioners don’t change the infrastructure itself. To force Terraform to recreate the resource and trigger the provisioner again, we can mark the instance as tainted:

This tells Terraform that the resource should be destroyed and recreated.

When we apply again, Terraform destroys the old instance, creates a new one, and during that creation, we see the local-exec provisioner run. In the output, we’ll see the echoed message showing the instance ID and IP address.

This is a small example, but it clearly shows what local-exec is good at: running local helper commands that react to infrastructure events.

With that foundation in place, we’re ready to move from “local” actions to running commands on the remote machine itself. That’s where the next provisioner comes in.

What is a Remote-exec provisioner?

Let’s now move on to the remote-exec provisioner.

Now that we’re comfortable with what local-exec does, the next step feels very natural.

Until now, every command we talked about was running on our own machine. But at some point, we usually want to go one step further. We don’t just want to know that an EC2 instance was created, we want to actually do something on that instance.

This is where the remote-exec provisioner comes in.

As the name suggests, remote-exec allows us to run commands remotely on a machine. In our case, that remote machine is the EC2 instance we just created using Terraform. Unlike local-exec, this provisioner does not run on our laptop. Instead, Terraform connects to the instance over SSH and executes the commands there.

This is an important shift, so let’s pause and absorb it.

For remote-exec to work, Terraform needs a way to log in to the server. That’s why the connection block we added earlier suddenly becomes very important. The SSH user, private key, and host information tell Terraform exactly how to reach the instance and authenticate itself. Without this, remote execution simply wouldn’t be possible.

Now, think about common real-life situations. When a server comes up for the first time, we often want to do some basic setup. Maybe we want to update packages, create a file, or install something simple. That initial preparation step is a very common use case for remote-exec.

Let’s see how this looks in our existing EC2 resource.

We add a new provisioner block, this time of type remote-exec:

Instead of a single command, we now use inline. This lets us provide a list of commands that Terraform will run one after another on the remote machine. These commands are executed over SSH, exactly as if we had logged into the instance ourselves and typed them manually.

The first command updates the package list. The second command creates a file inside the /tmp directory and writes a simple message into it. There’s nothing complex here and that’s intentional. The goal is not to do something fancy, but to clearly see the difference between local execution and remote execution.

At this point, if we run terraform apply again, Terraform may still say there are no changes. Just like before, provisioners don’t count as infrastructure changes. So once again, we mark the instance as tainted:

And then we apply the changes.

Terraform destroys the existing instance, creates a new one, and during that creation process, it connects to the EC2 instance over SSH and runs the remote-exec commands. This time, the work is happening inside the server, not on our local machine.

To really confirm this, we can log into the EC2 instance after it’s created. Once connected, we check the /tmp directory and see the file remote_exec.txt. When we open it, the content matches exactly what we defined in the Terraform configuration.

That small file is our proof.

It shows us that remote-exec truly runs commands on the remote machine, using the connection details we provided. This is the key difference between local-exec and remote-exec, and once that difference is clear, both concepts feel much easier to reason about.

Now that we’ve seen how to run commands locally and remotely, there’s one more very practical scenario left, copying files from our machine to the server.

That’s where the file provisioner comes into the picture, and that’s what we’ll explore next.

What is a File Provisioner?

Finally, let’s explore the file provisioner, which builds naturally on what we’ve learned so far.

Where remote-exec lets us run commands on the remote machine, the file provisioner is all about transferring files from our local environment to the instance. This is useful when we want scripts, configuration files, or other assets to be available on the machine right after it’s created.

Here’s a simple example based on our setup:

Let’s break this down:

• source points to a file in our local project directory, in this case, a script welcome.sh inside a scripts folder.

• destination specifies where that file should be placed on the EC2 instance. We’re putting it in /tmp/welcome.sh, but it could be any absolute path you like.

Once Terraform applies this provisioner, the file is copied over SSH to the remote machine. Notice that, just like remote-exec, the connection block we defined earlier is required for authentication.

After the file is transferred, we could even run it using another remote-exec provisioner if needed, for example, to configure the system or create additional files based on the script.

To see it in action, we taint the resource again:

After the instance is recreated, logging into it will show our welcome.sh script in the /tmp directory. Alongside the file created by remote-exec, we now have tangible evidence that the provisioners executed successfully.

The file provisioner is simple, yet powerful. It’s a straightforward way to move files where they’re needed immediately after creating a resource, no manual copying, no extra steps.

Conclusion

And with that, we’ve completed Day 19 of the 30 Days of Terraform Challenge! Today’s session introduced us to Terraform provisioners, which might seem like small helpers at first glance, but they are actually powerful tools that allow us to go beyond just creating infrastructure. They let us automate tasks that happen right after a resource is created, whether it’s running commands locally, configuring a server remotely, or copying important files over. In other words, provisioners help bridge the gap between infrastructure as code and infrastructure ready to use. And that’s why this topic is so important, it gives us the ability to make our resources immediately practical without manual intervention, saving time and reducing errors.

Compared to some of the previous blogs, today’s session was short and fun, and the concepts were very approachable. Even if you’re just starting out, you can follow along easily. The examples were simple enough to understand the mechanics, but also practical enough that you can apply them in real-world scenarios.

For those who like a step-by-step visual guide, there’s a helpful video by Piyush Sachdeva where he explains all the concepts covered today. The video also goes through some common troubleshooting steps, which can be a lifesaver if your first attempt at running provisioners doesn’t go exactly as expected.

So, that wraps up Day 19. Tomorrow, we’ll continue our journey, building on this foundation and exploring more exciting features of Terraform. Until then, take a moment to reflect on what you’ve learned, maybe even experiment a little with your own scripts and provisioners, and enjoy the process.

Happy Terraforming!

Hello, and welcome back to the blog!

This is Day 18 of our 30 Days of AWS Terraform journey, and if you’ve been following along, we’ve already covered quite a bit together. Last day, we explored a mini-project on AWS Terraform Blue-Green Deployment using Elastic Beanstalk. Today, we’re building on that learning, but we’re taking a slightly different turn.

Instead of focusing on networking or long-running infrastructure, we’re going to explore something lighter, more event-driven. Our focus for today is AWS Lambda, the star of the serverless world.

This blog remains Terraform-focused, just like the rest of the series. But instead of provisioning servers that must run all the time, we’ll see how Terraform can help us create something that runs only when it’s needed.

To make this practical, we’ll walk through an end-to-end image processing project i.e. from uploading a file to S3, to automatically processing it using a Lambda function, all orchestrated through Terraform.

Before touching any code or Terraform files, there’s one important step: understanding what AWS Lambda really is, how serverless works, and why this approach fits so well with event-driven workflows. Once we have that mental picture, everything else in the project will start to make a lot more sense.

Fact #18: I’m not really good at cooking, but if you put me in front of a pan and some rice, I can make omurice pretty well.

What Exactly Is AWS Lambda? And What Do We Mean by “Serverless”?

So, what exactly is Lambda?

At its core, AWS Lambda is a serverless function.

Now, that word serverless can sound a bit confusing at first. It’s not the same as someone saying, “There are no servers at all,” which isn’t really true. There are servers involved, but the difference is we don’t manage them.

Let’s break this down.

Whenever we build a traditional application, the usual flow looks something like this:

That server has to be running all the time, especially if it’s hosting a live application. Even if no one is using the app at that moment, the server is still there, still running, still costing money.

This is the model most of us are familiar with.

Now, serverless changes this mindset.

With Lambda, we don’t provision servers at all. Instead of thinking about machines, we think about functions. We simply write our application code, package it, and upload it as a Lambda function.

AWS takes care of everything else.

The servers still exist, but AWS manages them for us. We don’t worry about operating systems, scaling, or uptime. Our responsibility ends with the code.

And here’s the key difference:
a Lambda function does not run all the time.

A Lambda function runs only when something triggers it.

This trigger is called an event.

An event could be:

• A file being uploaded to an S3 bucket

• A scheduled time (for example, every Monday at 7 AM)

• A real-time system event

When the event happens:

That’s it.

There’s no server sitting idle in the background. The function exists until it’s needed, does its job, and then disappears again.

This makes Lambda especially useful for tasks that:

• Run for a short time

• React to events

• Don’t need to be running continuously

Typically, Lambda functions are used for executions that last a few seconds to a few minutes. AWS recommends not using Lambda if your code needs to run longer than 15 minutes, and that’s an important boundary to keep in mind.

Now that we have a clearer picture of what Lambda is and how serverless works, we’re ready to look at how this idea fits perfectly with the project we’re building today i.e. an image processing pipeline driven by events.

Understanding the Project Flow

Now that we have a clear understanding of what AWS Lambda is and how serverless works, it’s the perfect time to look at what we’re actually building in this project.

Before we touch Terraform files or Python code, let’s first form a simple mental picture. If the architecture makes sense in our head, the implementation will feel much easier later on.

At a high level, this project is an image processing pipeline. And the beauty of it is that once it’s set up, everything happens automatically.

Here’s the idea.

We’ll have two S3 buckets:

• One bucket where we upload the original image

• Another bucket where the processed images will be stored

The first bucket is our source bucket. Whenever we upload an image to this bucket, that upload creates an S3 event.

And remember what we discussed earlier, events are exactly what serverless functions like Lambda are waiting for.

So as soon as an image is uploaded, that S3 event will trigger our Lambda function.

This Lambda function is where all the image processing logic lives. It will take the original image and automatically generate:

• A JPEG image with 85% quality

• Another JPEG image with 60% quality

• A WebP version

• A PNG version

• And a thumbnail image resized to 200 by 200

All of this happens without us clicking any extra buttons or running any manual commands.

We simply upload a file using the AWS Console, AWS CLI, or even an API call and the rest of the workflow takes care of itself.

Once the processing is done, the Lambda function uploads all five generated images into the destination bucket. That’s where the final output lives.

Now, for this to work smoothly, permissions play a very important role.

The S3 bucket must be allowed to trigger the Lambda function.
The Lambda function must be allowed to:

• Read files from the source bucket

• Write processed files to the destination bucket

• Write logs so we can see what’s happening behind the scenes

These permissions are handled through IAM roles and policies, and we’ll be creating them using Terraform by giving the function exactly what it needs and nothing more.

This follows a principle we’ve touched on before: least privilege access. We don’t give broad permissions like full S3 access. We only allow what is strictly required.

There’s one more important piece in this setup.

For image manipulation, we’ll be using Pillow, which is a Python library designed for image processing. Since Lambda doesn’t include Pillow by default, we’ll package it as a Lambda layer and attach it to our function.

This keeps our Lambda function clean and makes dependencies easier to manage.

So to summarize the flow in simple terms:

1. We upload an image to the source S3 bucket

2. An S3 event is created

3. The event triggers the Lambda function

4. The Lambda function processes the image using Pillow

5. Processed images are saved to the destination S3 bucket

6. Logs are written to CloudWatch for visibility

All of this infrastructure i.e. buckets, permissions, Lambda, layers, and triggers will be created using Terraform.

Now that we understand the full picture, we’re ready to dive into the repository and start seeing how this architecture translates into Terraform code.

Getting Started: Cloning the Repository and Setting the Stage

Now that we understand what we’re building and how the pieces fit together, it’s time to get our hands a little closer to the actual project.

For this project, we’ll continue using the same GitHub repository that we’ve been working with throughout this series.

The first step is to clone the repository and move into the Day 18 directory.

The repository lives here

Once we clone it, navigate into the day-18 folder and then into the terraform directory. This is where all the Terraform files for today’s project live.

Before we create any real AWS resources, there’s one small but important thing we need to handle.

Making Resource Names Unique

Some AWS resources, like S3 buckets, must have globally unique names. That means even if the bucket name makes sense to us, AWS might reject it if someone else in the world is already using it.

To solve this, we use a small helper resource in Terraform i.e. a random suffix.

In our main.tf, the first thing we create looks like this:

What this does is simple. Terraform generates a short random value, and we attach that value to our resource names. This way, we don’t have to manually keep changing bucket names every time we deploy.

Next, we define some locals. Locals allow us to build names once and reuse them everywhere, which keeps the configuration clean and readable.

Here’s what’s happening:

• We build a common bucket prefix using the project name and environment

• We create two bucket names i.e. one for uploads and one for processed images

• We append the random suffix so the names stay unique

• We also define a clear name for our Lambda function

This approach might feel repetitive if you’ve been following the series, and that’s a good thing. It means the concepts are starting to feel familiar.

With naming out of the way, we’re now ready to create the first real AWS resources for this project i.e. S3 buckets.

Creating the Source S3 Bucket

Every story has a starting point, and in our case, it’s the place where the original image first arrives.

This project begins with an S3 bucket that acts as the source bucket. Any image we upload here will eventually kick off the entire image processing workflow.

We start by creating the bucket itself.

There’s nothing complicated happening here. We’re simply creating an S3 bucket and giving it the name we already prepared using locals. This bucket will hold the original image exactly as it’s uploaded.

But creating a bucket is only the first step. There are a few important decisions we need to make to ensure this bucket is safe, reliable, and production-friendly, even though this is a learning project.

Enabling Versioning

The next thing we do is enable bucket versioning.

Versioning helps us keep track of changes. If the same file name is uploaded again, S3 doesn’t overwrite the old object, it stores a new version instead.

This is especially useful when files are important or when multiple updates might happen. Even if we don’t strictly need it for this demo, it’s a good habit to build and reinforces real-world practices.

Adding Server-Side Encryption

Next, we enable server-side encryption.

This ensures that any image uploaded to the bucket is encrypted at rest using AES-256. We don’t need to manage encryption keys manually, AWS takes care of that for us.

Again, this is about forming good habits early. Even simple projects deserve sensible security defaults.

Blocking Public Access

Finally, we make sure the bucket is private.

For this project, there’s no need for the upload bucket to be publicly accessible. Images are uploaded intentionally, and access is controlled.

By blocking public ACLs and policies, we make sure the bucket isn’t accidentally exposed. In real production systems, public access is usually handled through controlled layers in front of S3, not directly on the bucket itself.

Now, with the source bucket in place, we now need somewhere to store the processed images.

Creating the Destination S3 Bucket

This second S3 bucket is our destination bucket. Every image generated by the Lambda function i.e. all five variants, will be stored here.

The setup for this bucket looks very similar to the source bucket, and that’s intentional. Consistency makes infrastructure easier to understand, maintain, and secure.

We start by creating the bucket itself

Just like before, we’re using a name generated from locals, which includes the project name, environment, and a random suffix to ensure uniqueness.

Enabling Versioning Again

Next, we enable versioning for the processed bucket as well.

Processed images may change over time, maybe we adjust compression levels, add new formats, or improve the logic later. Versioning ensures we don’t lose older outputs if something changes.

Encrypting the Processed Data

We then apply server-side encryption to the destination bucket.

This ensures that all generated images are encrypted at rest, just like the originals. Consistent security across buckets is always a good practice.

Keeping the Bucket Private

Finally, we block public access for the processed bucket as well.

There’s no reason for processed images to be publicly accessible in this setup. Everything is handled internally by AWS services, and access is controlled through IAM permissions.

At this stage, we now have:

• A source bucket for uploads

• A destination bucket for processed outputs

• Both buckets encrypted, versioned, and private

But these buckets don’t yet know how to talk to our Lambda function. And the Lambda function itself doesn’t exist yet either.

Before we can create the function, we need to solve an important question:
Who is allowed to do what?

That’s where IAM roles and policies come into play.

IAM Roles and Policies: Giving Lambda Just Enough Permission

So far, we’ve created two S3 buckets. They exist, they’re secure, and they’re ready. But right now, they’re just storage.

For our image processing workflow to actually work, we need a way for AWS Lambda to interact with these buckets, and to do that safely.

This is where IAM roles and policies come in.

Instead of hard-coding permissions or credentials, AWS uses roles to define what a service is allowed to do. In our case, we want the Lambda function to:

• Write logs so we can see what’s happening

• Read images from the source bucket

• Write processed images to the destination bucket

Nothing more, nothing less.

Creating the IAM Role for Lambda

We start by creating an IAM role that Lambda can assume.

This role doesn’t give any permissions yet. It simply says:

“This role can be assumed by AWS Lambda.”

We define this using a policy document. While the JSON might look a bit intimidating at first, it’s really just a structured way of expressing trust. AWS even provides a policy generator to help create these documents, which makes life easier when you’re starting out.

Once this role exists, we can start attaching actual permissions to it.

Defining the Permissions with an IAM Policy

Next, we create a policy that tells AWS exactly what this Lambda function is allowed to do.

The policy may look intimidating, but here’s what it does:

The first statement allows the Lambda function to create log groups, log streams, and write logs. Without this, we’d have no visibility into what the function is doing, especially if something goes wrong.

The second statement allows Lambda to read objects from the source bucket. This is how it gets access to the uploaded image.

The third statement allows Lambda to write objects to the destination bucket. This is where all the processed images will be stored.

Notice something important here?

We are not giving full S3 access. We’re not using broad permissions like “S3FullAccess.” Instead, we’re granting access only to:

• Specific actions

• Specific buckets

• Specific object paths

This is a perfect example of the principle of least privilege. We give the Lambda function exactly what it needs to work and nothing more.

With this IAM role and policy in place, our Lambda function will be able to:

• Read images

• Process them

• Store the results

• Write logs for us to inspect

Now that permissions are sorted, we’re ready to create something truly interesting, the Lambda function itself.

Lambda Layers: Bringing Image Processing Capabilities to Lambda

Now that our permissions are in place, we’re almost ready to create the Lambda function itself. But before we do that, there’s one important challenge we need to solve.

Our Lambda function is going to process images, and for that, we’ll be using Pillow, which is a Python library designed for image manipulation.

The problem is simple:
Pillow is not included by default in AWS Lambda.

So how do we use it?

This is where Lambda layers come into the picture.

A Lambda layer is a way to package external libraries and dependencies separately from our function code. Instead of bundling everything inside the function zip, we place shared or heavy dependencies into a layer and then attach that layer to the Lambda function.

This keeps the function code clean and makes dependencies easier to manage.

Creating the Lambda Layer Resource

In Terraform, we define the layer like this:

Here’s what’s happening:

• filename points to a zip file that contains the Pillow library

• layer_name gives the layer a clear, readable name

• compatible_runtimes ensures this layer works with Python 3.12

• The description tells us exactly what this layer is for

But this raises another question.

How do we create the pillow_layer.zip file in the first place?

Because AWS Lambda runs on Linux, the dependencies inside the layer must also be built for a Linux environment. This is important, especially if you’re working on macOS or Windows.

To solve this, we use some help from our friend Docker.

Building the Pillow Layer Using Docker

Inside the scripts folder, there’s a helper script that takes care of building the layer correctly.

This script runs a Linux-based Python container, installs Pillow inside the exact folder structure Lambda expects, and then packages it into a zip file.

The important thing to understand here isn’t every line of the script, it’s the idea behind it.

We’re ensuring that:

• Pillow is installed in a Linux environment

• The directory structure matches what Lambda expects

• The final output is a clean zip file we can reuse

Once this zip file is created, Terraform can reference it and upload it as a Lambda layer.

With the layer ready, we finally have everything needed to create the Lambda function itself:

• Storage (S3 buckets)

• Permissions (IAM role and policy)

• Dependencies (Pillow via a Lambda layer)

In the next section, we’ll create the Lambda function, attach the layer, configure memory and timeout, and pass environment variables that the function will use at runtime.

Creating the Lambda Function

With our storage, permissions, and dependencies ready, we can finally create the Lambda function that will do the actual image processing work.

But before Terraform can create the function, it needs one important thing:
the function code packaged as a zip file.

Packaging the Lambda Function Code

Our Lambda function is written in Python and lives inside the repository. To package it correctly, we use a Terraform data source.

This data source takes the Python file, compresses it into a zip archive, and makes it ready for deployment.

Even though we’re working with a local file here, Terraform treats this as data it needs to reference during deployment which is exactly what data sources are designed for.

Defining the Lambda Function

Now we define the Lambda function itself.

Let’s walk through this.

• filename points to the zip file created earlier

• function_name gives the Lambda function a clear identity

• role attaches the IAM role we created, allowing the function to access S3 and logs

• handler tells Lambda where execution begins in the Python file

• runtime specifies Python 3.12

• timeout is set to 60 seconds, which is more than enough for image processing

• memory_size is set to 1024 MB to give the function enough resources

We also attach the Pillow layer here. This is what enables image manipulation inside the function.

Finally, we pass in a couple of environment variables:

• The name of the processed images bucket

• The log level

This keeps our code flexible and avoids hard-coding values.

Creating the CloudWatch Log Group

To make sure logs are retained in a predictable way, we also create a CloudWatch log group.

This ensures that logs are stored for seven days and then automatically cleaned up.

At this point, the Lambda function exists, but it still doesn’t know when to run.

Remember, Lambda functions don’t run on their own. They need an event trigger.

In the next section, we’ll connect the source S3 bucket to the Lambda function so that every image upload automatically triggers the processing workflow.

Wiring the Event Trigger: Letting S3 Invoke Lambda Automatically

Our Lambda function now exists. It has the code, the permissions, and the image processing library attached. But right now, it’s still just waiting.

For Lambda to actually run, it needs an event.

In our project, that event is simple and very natural:
an image being uploaded to the source S3 bucket.

To make this work, we need to do two things:

1. Allow S3 to invoke the Lambda function

2. Tell the S3 bucket which events should trigger the function

Allowing S3 to Invoke the Lambda Function

First, we explicitly give S3 permission to invoke our Lambda function.

This resource acts like a handshake.

It tells AWS:

• S3 is allowed to invoke this Lambda function

• The permission applies only to our upload bucket

• The action allowed is strictly InvokeFunction

Without this permission, S3 events would never be able to trigger the function, even if everything else was configured correctly.

Creating the S3 Event Notification

Now that permission is in place, we configure the S3 bucket to send events to Lambda.

This tells the source bucket:

“Whenever an object is created, in any way, invoke this Lambda function.”

It doesn’t matter whether the file is uploaded through:

• The AWS Console

• The AWS CLI

• An API call

As long as an object is created in the bucket, the event fires.

The depends_on is important here. It ensures Terraform creates the permission first, so S3 is allowed to invoke the function before the notification is set up.

At this point, the entire flow is connected:

• Upload an image

• S3 generates an event

• Lambda is invoked

• Image processing happens automatically

Now all that’s left is deployment.

In the next section, we’ll look at the deployment scripts, how the Lambda layer is built automatically, and how everything is deployed using a single command.

Deploying Everything: Bringing the Project to Life

At this point, all the pieces of our architecture are defined. We’ve described the buckets, the permissions, the Lambda function, the layer, and the event trigger. Now comes the satisfying part, actually deploying everything.

To make this process smooth and beginner-friendly, we’ll not run long list of commands manually. Instead, the repository includes a small script that takes care of everything in the right order.

The deploy.sh Script

Inside the scripts folder, we’ll find a file called deploy.sh. This script is designed to guide the entire deployment process from start to finish.

When we run this script, here’s what it does.

First, it performs a few basic checks. It makes sure:

• AWS CLI is installed

• Terraform is installed

If either of these is missing, the script stops and tells us exactly what’s wrong. This saves time and avoids confusion later.

Next, the script builds the Lambda layer.

This is an important step. Remember, the Pillow library needs to be compiled in a Linux environment to work correctly with AWS Lambda. Instead of doing this manually, the script calls another helper script that uses Docker to:

• Spin up a Linux-based Python environment

• Install Pillow in the correct directory structure

• Package everything into a pillow_layer.zip file

Once that’s done, the script moves into the Terraform directory and runs the familiar commands:

• terraform init

• terraform plan

• terraform apply

All in the correct order.

Terraform then takes over and creates every AWS resource we discussed:

• Both S3 buckets

• IAM roles and policies

• Lambda layer

• Lambda function

• CloudWatch log group

• S3 event trigger

When the deployment finishes, the script prints out something very useful, the names of the buckets and the Lambda function that were just created.

This makes it easy to know exactly where to upload your image.

Testing the Workflow

Once deployment is complete, testing the project is beautifully simple.

We just upload an image to the upload bucket.

It can be any JPG or JPEG file. We can upload it using:

• The AWS Console

• The AWS CLI

• Any method that creates an object in the bucket

The moment the file is uploaded, the event is triggered.

Behind the scenes:

• Lambda starts

• The image is processed

• Five new images are generated

• All processed files appear in the destination bucket

If we open CloudWatch, we can also see the logs generated by the Lambda function, helpful for understanding what happened and for troubleshooting if something goes wrong.

And just like that, we’ve built an event-driven, serverless image processing system using Terraform.

Conclusion

And with that, we’ve completed Day 18 of the 30 Days of Terraform Challenge.

Today’s demo walked us through the serverless side of AWS, which is such a crucial piece when we talk about building truly cloud-native solutions. Services like Lambda, S3 event triggers, and IAM roles show us how powerful the cloud can be when we design systems that react to events instead of relying on always-running servers. This mindset shift is important, and today was a solid step in that direction.

It’s completely okay if the codebase felt a bit overwhelming at first glance. That feeling is part of the journey. As developers, we’re not expected to understand everything instantly. What really matters is how we approach it. We can lean on LLMs, documentation, blogs, and videos to slowly break things down and understand what each line of code is doing. Reading code carefully, line by line, is a skill in itself, and it plays a huge role in becoming a better, more confident programmer over time.

To make things even easier, there’s a great video by Piyush Sachdeva that explains this demo clearly and walks through the entire flow. It’s especially helpful if you want to follow along step by step or need guidance with troubleshooting, definitely worth checking out alongside the code.

We’ll stop here for today, take a moment to absorb what we’ve learned, and come back refreshed tomorrow to continue the journey.

Until then, happy terraforming, see you on Day 19

Welcome back to Day 17 of our 30 Days of AWS Terraform journey.
It nice to have you here again! Over the past few days, we’ve quietly built up little blocks of understanding together. Yesterday, we spent time with IAM authentication using Terraform, and even though IAM can feel a bit strict and serious, we made it through with patience.

Today, we’re opening a new chapter, something practical, something used by real teams every single day, yet something that can feel intimidating when we first hear its name**: blue-green deployment.**

If that phrase feels a bit heavy, that’s okay. Most things in tech sound complicated before someone explains them. And that’s exactly what we’re going to do today, breathe through it, and understand it like a story.

It’s a concept that might sound big but is actually just a simple idea wrapped in a slightly fancy name. By the time we reach the end, we’ll see how naturally all the pieces fit.

Our mini-project today uses AWS Elastic Beanstalk and Terraform to create two separate environments i.e. blue and green. We’ll package a small application, upload it to S3, create application versions, and watch how each environment gets its own version of the app. The most interesting part comes later, where with a simple “swap,” traffic can gently shift from one environment to the other, giving us a sense of how real deployments happen with almost no downtime.

So take a breath, settle in, and let’s begin our slow and steady walk into the world of blue-green deployments together.

Here’s a fact before we proceed:

Fact #17: During my school and college years, drawing little sketches and doodles was something I kept coming back to. It was a hobby I really enjoyed.

Understanding Blue Green Deployment

Before we dive into any Terraform blocks or AWS resources, let’s take a pause. I want you to imagine something simple.

Picture our application as a cozy little café.

Our customers walk in, take a seat, enjoy the calm atmosphere, and everything just… works.
This café is our blue environment i.e. the one everyone knows, trusts, and uses every day. It’s open, it’s running, and people depend on it.

Now, suppose we want to renovate a few things, maybe update the menu, repaint the walls, or try new lighting. But doing that while customers are sitting inside? That’s stressful, messy, and risky. We don’t want people slipping on wet paint or being disturbed by hammering noises.

So instead, we quietly build an identical café right next door. Same furniture, same setup, same layout, everything in the same place. This second café is our green environment. It looks just like the blue one, but no customers are sitting inside yet. It’s empty… peaceful… safe to experiment with.

And that is the beauty of it.

In the green café, we can try all our improvements without disturbing anyone. Paint the walls, test new recipes, adjust the lighting, whatever we want. If something goes wrong, no one sees it. No one is affected. We simply fix it and try again.

Only when everything feels perfect i.e. when the chairs are aligned, the lights are warm, and the menu is polished, we gently swap the entrances.

Suddenly, customers walk into the green café without even realizing it, because from the outside, the signboard (our DNS name) is exactly the same. All that changed was the door they’re stepping through.

The old blue café becomes the standby, still there, still safe, still intact. If we ever need to switch back because we found a problem later, we simply swap the doors again. Everything stays smooth.

In real applications, this is exactly how a blue-green deployment works.

• Blue is our production environment i.e. the live one.

• Green is our duplicate i.e. the safe testing ground.

• And the swap is the seamless transition our users never feel.

Maybe our blue environment started with version 1.0, and when we created green, it also mirrored 1.0 to stay identical. After checking everything, you upgraded green to 2.0, tested it, and once we felt confident, we swapped the environments. Traffic that once flowed to blue now flows to green, gracefully, without a single customer getting disturbed.

This is why teams love the blue-green approach: it keeps users happy, avoids those stressful downtimes, and gives us a cushion to experiment safely while ensuring our deployments feel smooth and effortless.

Preparing S3 and Application Packaging

Before we even talk about environments or deployments, I want to pause with you for a moment and slowly walk into the first stepping stone of this whole process.

Every application, no matter how small, needs a home where its packaged files can stay safely before being deployed. In our case, that home is an S3 bucket. Instead of imagining S3 as a complicated AWS service, picture it as a little storage room where we keep different versions of our application, each carefully labeled and tucked away.

So the very first thing we do is upload our application package to this bucket. The package is simply a zip file named app-v1.zip, we can think of it like wrapping your application in a clean, sealed envelope. Terraform picks up this envelope and stores it in the S3 room so that Elastic Beanstalk can come later, open it, and use it to create a version of our application.

This simple but important step looks like this:

Let’s walk through this:

• resource "aws_s3_object" "app_v1"
  This is Terraform saying, “I want to upload a file to S3, and I’ll refer to this upload as app_v1.”

• bucket = aws_s3_bucket.app_versions.id
  This tells Terraform which S3 bucket to place the file on. The bucket already exists somewhere in our setup.

• key = "app-v1.zip"
  Think of this as the label on the file once it’s inside the bucket. The name we want it to have.

• source = "${path.module}/app-v1/app-v1.zip"
  This is the real physical location of our zip file on our computer or project folder.
  Terraform looks here and picks up the file.

• etag = filemd5(...)
  This is just Terraform checking the file’s fingerprint to know if it has changed.
  We don’t need to worry about the details, just think of it as a “change detector.”

• tags = var.tags
  This attaches your usual tags so everything stays organized.

And that’s all it is.

A tiny resource, doing a tiny job, but it sets everything else in motion. Without this file sitting in S3, Elastic Beanstalk wouldn’t know where to pull the application version from.

Blue Environment Details

Now that our application package is safely stored in S3, it’s time to take the next step: creating our blue environment in Elastic Beanstalk. Think of the blue environment as the steady, reliable “home base” for our application, the environment that will first serve users and carry the version 1.0 of our app.

Before we actually deploy the environment, we need to define the application version. This tells Elastic Beanstalk exactly which package to deploy. Here’s the Terraform code that does that:

Let’s pause and understand this:

• name is just a label for this version. We’re calling it v1 to match our first release.

• application links this version to the Elastic Beanstalk application we created earlier.

• description is a friendly note to remind us this is version 1.0, the initial release.

• bucket and key tell Elastic Beanstalk where to find the zip file for this version. The bucket is our S3 bucket, and the key is the file name inside it (app-v1.zip).

• tags are simple labels we attach to this version for easy identification and organization.

Once the version is ready, we can create the blue environment itself:

Here’s what’s happening in this code:

• name gives our environment a friendly identifier: blue.

• application points back to the Elastic Beanstalk app we’re deploying.

• solution_stack_name tells Elastic Beanstalk what type of environment to run, like which operating system, web server, and platform to use.

• tier is set to WebServer because this environment will serve HTTP requests to our users.

• version_label tells the environment which version of the app to deploy, here, it is v1.

But a web application is not just a single instance; it has many moving parts. That’s why we add settings to configure the environment properly. These settings handle things like IAM roles, instance types, load balancing, scaling, health checks, and environment variables. Let’s take a few examples:

IAM instance profile (gives EC2 instances permission to access AWS resources):

Service role (allows Elastic Beanstalk itself to manage resources):

Instance type (defines the size of the virtual machine running our app):

Load balancer configuration (ensures traffic is distributed across instances):

Autoscaling (minimum number of instances):

Health reporting (helps monitor application status):

Environment variables (used by our app to know it’s running in blue):

Finally, we add tags to identify the environment:

These tags are helpful when you have multiple environments and want to quickly filter resources by environment or role.

When we put all these pieces together, we now have a fully configured blue environment, ready to serve our first users with version 1.0 of our application. Every setting ensures our application runs reliably, is secure, and can scale when traffic increases.

Green Environment Details

Now that our blue environment is up and running, serving version 1.0 of the application to users, it’s time to bring the green environment to life. Think of the green environment as the “practice stage” or the staging environment, it’s an exact twin of blue, but it will host the next version of our application, version 2.0.

The beauty of blue-green deployment is that we don’t touch the live production environment while preparing the next release. Instead, we work safely in the green environment, test everything thoroughly, and only then make it live.

IAM Roles and Profiles

Now that our blue and green environments are taking shape, let’s pause and explore one of the important parts of this journey: IAM roles and instance profiles. Think of IAM as the permission system in AWS, like a backstage pass that lets our application and the underlying servers do their jobs safely. Without proper roles, our Elastic Beanstalk environments wouldn’t be able to access S3, update themselves, or scale properly.

Let’s start with the EC2 role. Each EC2 instance in Elastic Beanstalk needs permission to perform certain actions on AWS services. Here’s how we define it in Terraform:

Let’s unpack this:

• name is simply the friendly name of this role. We’re using a variable so it stays dynamic.

• assume_role_policy is a JSON that tells AWS who can “assume” this role. In our case, the EC2 service itself (ec2.amazonaws.com) is allowed to assume it.

• tags help us organize and identify resources, which is especially useful in real-world projects.

Next, we attach AWS-managed policies to this role. These policies define what actions the EC2 instances can actually perform. For example, here’s the attachment for the web tier:

This is like giving your EC2 instances a pre-configured toolbox for web applications. We can also attach other policies for worker tiers or multi-container Docker if needed, these ensure that the EC2 instances can do everything required by the environment without manually assigning individual permissions.

Now that the role is ready, we create an instance profile, which essentially links the IAM role to the EC2 instances:

It carries the IAM role badge and gives it to the EC2 instances so they know what they’re allowed to do.

Finally, Elastic Beanstalk itself needs a role to perform operations on our behalf, like deploying new versions, reporting health, or performing managed updates. Here’s the service role:

• Here, the principal is elasticbeanstalk.amazonaws.com, this tells AWS that Elastic Beanstalk can assume this role.

• We attach policies for enhanced health reporting and managed updates, so the service can monitor and maintain the environments automatically.

All these pieces together i.e. EC2 role, policies, instance profile, and service role form the IAM foundation of our Elastic Beanstalk environments. Without them, nothing would work. This is one of those parts of the deployment that runs silently, allowing the environments to function reliably and securely.

App Versions, Deployment Flow and Swap

Now that we have both our blue and green environments set up, let’s take a moment to imagine how traffic flows between them. At this stage, each environment is running its own version of the application: blue is our stable production version, and green is our updated version waiting in the wings.

The magic of blue-green deployment lies in the swap. When we perform the swap, behind the scenes, the DNS entry is updated so that all traffic now flows to the green environment.

This approach is what makes blue-green deployment so powerful. It allows us to:

• Test safely: Any updates are first applied to the green environment without affecting users.

• Minimize downtime: The switch happens quickly; users barely notice.

• Roll back easily: If something goes wrong with the green environment, a simple swap can revert traffic back to blue, restoring the stable version instantly.

In short, the swap is the heart of this strategy. It lets us update our application confidently, keeping users happy and the system reliable, all without a stressful downtime.

Demo: Version 1.0 and Version 2.0

Now comes one of my favorite parts, the moment where the concepts we’ve been discussing start to feel real. Imagine we’ve carefully built your blue and green environments, uploaded your application packages, and set everything up. It’s time to see them in action.

First, we visit the blue environment. This is our steady, live production environment, the one that our users are currently interacting with. When we open the URL, you are greeted with a friendly message:

“Welcome to the blue-green deployment demo. This is version 1.0.”

Alongside this, we might see some metadata, like the server time and host name. These little details give us reassurance that our environment is alive and stable. The blue environment represents the foundation, it’s reliable, predictable, and serving users exactly as expected.

Next, we take a peek at the green environment. This is the twin environment, waiting in the wings. It mirrors the blue environment in every way, but now it carries version 2.0, the updated release. When we open the green URL, we’ll see a message like:

“It is in the status staging, and these are the new features released in version v2.0.”

Notice how the green environment isn’t yet live for users, it’s in staging. It’s a safe space to verify that everything works as intended. We can explore the new features, check performance, and make sure the updates behave correctly, all without touching the live blue environment.

So, at this stage, the setup is clear:

• Blue is live, serving version 1.0 to users.

• Green is staging, hosting version 2.0 and ready for testing.

This careful pacing ensures we can appreciate why blue-green deployment works so elegantly. The green environment is not rushing ahead, it’s waiting, tested and ready, until it is time to take the stage.

Rollback and Destroy

Now, imagine this: we’ve done the swap, and the green environment is live with the new version.

Everything seems fine, but what if something unexpected pops up? Maybe a small bug, or a feature isn’t behaving exactly as we hoped. Don’t worry, that’s the beauty of blue-green deployment.

Because the blue environment still exists, untouched, we have a safety net. Rolling back is as simple as performing another swap. The traffic will flow back to the stable blue environment, and our users won’t even notice the hiccup.

Once we’ve completed our testing, observed the new version, and are satisfied with how everything works, it’s time to tidy up. Terraform makes this easy. Running a terraform destroy will carefully remove all the resources we created for this demo.

Taking this approach not only keeps your AWS account clean but also reinforces a disciplined, thoughtful workflow, especially helpful for beginners. Step by step, you’re learning to manage deployments safely, predictably, and with confidence.

Conclusion

And just like that, we reach the gentle end of today’s little adventure together. I hope you’ve been able to follow along at a comfortable pace and felt the story of blue-green deployment unfold.

I truly hope that today’s journey has given us not only a clearer understanding of blue-green deployment but also a sense of confidence. And if at any point things feel overwhelming, that’s perfectly okay too. It simply means there’s something new asking for a little extra time and attention. With concepts like these, the only real way to feel confident is through gentle, steady hands-on practice.

So if you’d like to revisit today’s topic in a more visual way, there’s a wonderful video by Piyush Sachdeva that walks through the same idea slowly and clearly.

Tomorrow, we’ll continue our shared journey with another story, another mini-project, and more pieces of the AWS Terraform puzzle. Thank you for walking through today’s blog with me.

I’m grateful for your curiosity, your patience, and the steady effort you’re putting into understanding something new. Until then, take a moment to appreciate how far we’ve come and I’ll see you tomorrow.

Hello and welcome back to the blog! I hope you’re keeping warm in this chilly weather!

Yesterday, we wrapped up a small but meaningful mini-project around AWS VPC Peering using Terraform, a topic that can seem intimidating but we walked through it step by step, making it approachable and understandable.

Today, we’re easing into Day 16 of our 30 Days of AWS Terraform journey, and we’re tackling a question every cloud beginner eventually wonders about:
How do we manage users in AWS, especially when there are dozens of them?

Before jumping into code, imagine this: you start a new job and discover that thirty new employees are joining on the same day. Someone has to create their AWS accounts, assign access, place them into the right groups, and set temporary passwords so they can log in. Doing all that manually? Exhausting. Definitely a job made for automation.

Here’s what we’ll cover in today’s mini-project:

• Prepare a CSV file containing user details

• Convert that CSV into a list of maps

• Apply loops, conditionals, IAM groups, dynamic assignments, and login profiles

• Keep everything beginner-friendly while connecting each step to the bigger picture

We are not rushing into IAM resources just yet; we’ll set the stage first.

In the next section, we’ll start by creating our CSV file i.e. the heart of our bulk user-creation workflow and discuss why it’s important and how it will be used.

Before we proceed, a little personal fact for Day 16

Fact #16: If you’re not aware, I consider myself a bit of a movie connoisseur, and my favorite movie of 2025 was Sinners! So instead of looking it up, I’d suggest watching it without having any expectations!

Creating the CSV File: Laying the Foundation

Before we write a single line of Terraform code, let’s start with something simple and familiar: a CSV file. This small, plain text file will become the backbone of our entire mini-project. This file will hold all the user information we want Terraform to work with, allowing us to create many users at once, without having to type each one manually.

Let’s begin!

Instead of thinking of “user management” as a big, overwhelming AWS task, imagine it like organizing a classroom on the first day of school. We have a list of students, each with their own details, and our job is to place them in the right groups and give them what they need. In our case, the CSV file is that student list.

We’ll store information for around 30 users, and each row will contain just four simple fields:

• first_name

• last_name

• department

• job_title

We don’t want to complicate things. Terraform will use exactly these fields to create IAM users, tag them, add them to groups, and manage their login profiles.

Here’s a small preview of what the CSV might look like:

We actual file will have around 30 entries, but this small snippet is enough to understand the structure.

Once we save this file as users.csv inside our Terraform project directory, we’ll be ready for the next step: teaching Terraform how to read this file and convert it into something it can actually use.

Before moving ahead, take a moment to appreciate this:
Even something as simple as a CSV becomes powerful when Terraform interprets it. Instead of manually creating 30 IAM users, Terraform will handle everything for us.

In the next section, we’ll explore how to convert this CSV into a format Terraform understands using a very handy function: csvdecode().

Converting the CSV into a Terraform-Friendly Format: Understanding csvdecode()

Now that our CSV file is ready, the next step is to help Terraform understand it. At first glance, the file might seem simple i.e. just rows and columns of text but Terraform doesn’t see it the same way we do. To Terraform, it’s just raw text. It can’t yet loop through it, read values, or create users dynamically.

That’s where one of Terraform’s very helpful functions comes in: csvdecode().

Why do we need csvdecode()?

Terraform cannot directly perform operations on raw text files. If we tried to work with the CSV as-is, every step would require complicated parsing or messy manual loops which would defeat the purpose of using Terraform in the first place.

csvdecode() takes the entire CSV file and transforms it into a list of maps. Each row in our CSV becomes a map (think of it like a small dictionary), and each column in that row becomes a key-value pair.

This transformation makes our data predictable, easy to use, and ready for dynamic operations inside Terraform.

The Code

Here’s the local block we’ll use:

Let’s break this down:

• file("users.csv") → reads the CSV file as plain text

• csvdecode(...) → converts that plain text into structured data

• users → a variable that becomes a list of maps, where each map looks something like this:

We can almost imagine Terraform laying out all 30 users in front of it, each with their own labels and details.

Why is this conversion important?

Once the data becomes a list of maps, Terraform can:

• Loop through it easily

• Extract specific fields like user.first_name or user.department

• Create IAM users dynamically

• Use conditions to assign users to groups

• Tag resources properly

Without this conversion, we would be stuck writing repetitive logic or manually handling text, something Terraform is not designed for.

An Important Mindset Shift

Instead of thinking of the CSV as just an external file, consider it as part of Terraform’s data model. Once decoded, Terraform treats the CSV exactly like any other structured variable.

In the next section, we’ll add another small but crucial piece: retrieving our AWS account ID using a datasource. This helps Terraform understand where it’s creating these resources, keeping everything organized and consistent.

Retrieving Our AWS Account ID Using a Data Source

Now that Terraform understands our CSV and has neatly organized all the user data, the next step is helping Terraform understand where these resources will be created.
In AWS, every action i.e. creating users, policies, roles happens inside our unique AWS account.

Terraform can automatically fetch this account ID for us using a special feature called a data source.

What is a data source?

A data source in Terraform is like asking AWS a question and getting back an answer.
We’re not creating anything, we’re simply retrieving information that AWS already knows.

In this case, we want Terraform to ask AWS:

“What is my account ID?”

This ID is important because it helps Terraform correctly reference the AWS resources it creates, especially when generating ARNs.

The Code

Here’s the data source we’ll use:

When Terraform reads this, it contacts AWS and fetches three pieces of information:

• account_id

• arn

• user_id

But the one we care about most right now is:

Terraform will now know our account ID without us typing it anywhere, which keeps our code clean, safe, and reusable.

Why is this useful?

Once Terraform knows the account ID, we can:

• construct IAM ARNs dynamically

• avoid hardcoding sensitive values

• write modules that work in any AWS account

• reduce mistakes when referencing users or roles

• keep your configuration future-proof

Now that Terraform understands:

• our CSV users

• our AWS account context

We’re ready to write the IAM user creation logic, the part where Terraform creates users dynamically using loops.

Creating IAM Users Dynamically With for_each

Now that Terraform understands our CSV data and knows our AWS account ID, we can finally move to the exciting part i.e. creating IAM users automatically.

Instead of writing 30 separate IAM user blocks (which would be painful, repetitive, and easy to make mistakes), Terraform allows us to create all users in one clean, elegant block using for_each.

Why use for_each?

When we decoded our CSV using csvdecode, Terraform turned the plain text into a list of maps. Each map contains the user’s details like first_name, last_name, department, and job_title.

Now, we want Terraform to go through this list and create a corresponding IAM user for every single row. That’s exactly what for_each does, it loops through the list and handles each user automatically.

Each map in that list represents one user, so Terraform will:

• take the first row → create user 1

• take the second row → create user 2

• …and continue for every row in the CSV

This makes the whole configuration short, clean, and scalable, no matter how many users we have.

The Code

Here’s how we create all users from the CSV:

• for_each = { for user in local.users : user.first_name => user }

  • Here, Terraform is taking each user from the list we got from the CSV.

  • user.first_name is used as the key because Terraform needs a unique identifier for each item.

  • This loop ensures that Terraform creates a resource for every single user automatically.

• name = lower("${substr(each.value.first_name, 0, 1)}${each.value.last_name}")

  • This creates a username for AWS.

  • We take the first letter of the first name and combine it with the last name.

  • lower() converts the username to lowercase so it’s consistent.

  • Example: John Doe → jdoe

• tags = { ... }

  • Tags are like little labels attached to the user.

  • Here, we store the full name, department, and job title.

  • Tags help us organize users and make it easier to find them later in AWS.

Why this approach is so powerful

• We can create 10, 30, or even 10,000 users with the same code.

• Adding a new user to the CSV automatically creates that IAM user.

• Removing a user from the CSV tells Terraform to destroy that user safely.

• Every user receives tags that match your CSV data.

• The configuration remains readable, scalable, and beginner-friendly.

In short, Terraform does the heavy lifting, we only need to maintain the CSV, and everything else happens automatically.

Assigning IAM Users to Groups Based on Their Department

Now that we have our IAM users created from the CSV, it’s time to place them into the correct groups.
Why is this important? In AWS, groups help us manage permissions and access for multiple users at once. For example:

• Users in the Education department might need different access than users in Engineering.

• Instead of assigning permissions one by one, we can assign permissions to groups, and anyone in that group automatically inherits them.

In our mini-project, we’ll create three groups: Education, Managers, and Engineers. Then, we’ll assign users dynamically based on the Department tag from our CSV. This way, Terraform does all the heavy lifting, and we don’t have to manually add each user to a group.

Step 1: Create the Groups

Let’s first define our IAM groups in a new file, for example groups.tf. Here’s how it looks:

What’s happening here?

• name → the name of the IAM group in AWS.

• path → a folder-like structure for organizing groups (optional but recommended).

Now we have three groups ready. The next step is adding users to these groups dynamically.

Step 2: Add Users to a Group Dynamically

Instead of manually listing each user, Terraform lets us use a for loop to assign users automatically based on the Department tag.

Here’s an example for the Education group:

Let’s break this down:

• for user in aws_iam_user.users → loops through every IAM user we created earlier.

• user.name → grabs the username for each user.

• if user.tags.Department == "Education" → only includes users whose Department tag is “Education.”

This means Terraform automatically finds all users in the Education department and adds them to the Education group.

We don’t have to type each user manually. This is especially useful when dealing with 30 or more users.

Step 3: Repeat for Other Groups

The same logic can be applied to the Managers and Engineers groups:

This keeps everything consistent and dynamic. Adding a new user to the CSV automatically assigns them to the correct group without touching the Terraform code.

Step 4: How This Works Together

1. Terraform loops through all users created from the CSV.

2. Checks their Department tag.

3. Automatically adds them to the correct IAM group.

Once users are in their groups, the next step is to create IAM login profiles. This will allow each user to log in to the AWS console, set their own password, and start working.

Creating IAM Login Profiles So Users Can Sign In

Now that we have our IAM users ready, it’s time to give them a way to actually sign in to the AWS console.

Imagine this like handing each new team member a temporary key to the office. They can enter, take a look around, but they’ll set up their own permanent key (password) the first time they log in.

We’ll do the same with Terraform by creating login profiles for each user.

Why do we need login profiles?

Without a login profile, a user can exist in AWS but cannot log in to the console.
We want each user to:

• Access the console easily

• Receive a temporary password automatically

• Reset the password on their first login

Terraform will create these for every user automatically, even if there are 30 or more users.

The Code

Let’s walk through it

• for_each = aws_iam_user.users
  Think of this as Terraform going down the list of all users we created earlier.

• user = each.value.name
  This assigns the login profile to the correct user.

• password_reset_required = true
  This is like giving them a temporary key and saying:
  “Please change this key when you enter for the first time.”

• lifecycle { ignore_changes = [...] }
  This tells Terraform to leave certain settings alone, so the login profile doesn’t get accidentally changed in the future.

Seeing the login profiles (demo output)

For our demo, we can create an output to check that Terraform has created login profiles.

This does not show real passwords (AWS keeps them secure), but it gives us a confirmation message for each user.

sensitive = true ensures the messages don’t accidentally show up in log.

Conclusion

And with that, we’ve completed Day 16 of the 30 Days of Terraform Challenge.

I hope you found today’s blog a bit lighter and more approachable compared to the last two sessions. I say this because, having personally gone through the previous two blogs hands-on, I noticed how much more comfortable things start to feel once you get some practical experience under your belt. That’s one of the beautiful aspects of this journey: we slowly transition from simply learning theoretical concepts to actively applying them in real scenarios!

What I particularly enjoy about these hands-on sessions, which doesn’t always appear in the blog itself, is troubleshooting. Troubleshooting is often the hidden gem of learning, it’s where we really understand what’s happening behind the scenes. Sometimes Terraform throws an error, a loop doesn’t work as expected, or a resource isn’t created exactly how you anticipated. Figuring out why, experimenting with small changes, and eventually resolving the issue is where deep learning happens. These steps are usually messy and time-consuming, and the blogs don’t always show them, but trust me, that process is invaluable.

Personally, I like to dedicate uninterrupted time, often late at night, to complete a demo in one go.

I think I talked enough, the real learning comes from doing. Take the CSV file you prepared, apply the Terraform configuration, and observe how the automation unfolds. If you run into issues, take your time to explore the errors, check resource states, and practice fixing them.

To make this process easier, here’s a helpful video by Piyush Sachdeva that explains the workflow step by step and even walks through some common troubleshooting scenarios. Following along with such a resource can make the learning curve smoother and boost your confidence.

We’ll continue tomorrow with Day 17, diving deeper into Terraform’s capabilities, so take some time to practice today’s mini-project, reflect on the lessons learned, and enjoy the satisfaction of seeing your automation work flawlessly. Until then, happy terraforming!

Hello and welcome back to 30 Days of AWS Terraform. Today is Day 15, and if you’ve been following this journey from the start, congratulations, we’ve made it halfway!

Staying consistent, especially while learning new technology, takes real curiosity and effort. So truly… well done. Take a moment to appreciate yourself for showing up and sticking with it.

Over the past few days, we’ve been moving from understanding basic Terraform concepts to trying out mini projects. If you remember, our last hands-on adventure was hosting a static website on AWS S3. It was an example that showed that we can actually build something real in the cloud. Then, we made it even more exciting by integrating CloudFront to distribute our website globally. That small project taught us how Terraform can manage real AWS infrastructure, and how small changes can have a big impact.

Today, we’ll take another step forward. We’re going to explore AWS VPC Peering using Terraform. At first glance, networking concepts like VPC peering can feel intimidating. We’ll begin by understanding the “why” behind VPC peering, and slowly see how it fits naturally into the cloud architectures we’ve been creating.

A personal fact before we move on:

Fact #15: I’ve always been fascinated by Vietnam, its history, culture, and landscapes. I hope to visit someday and see it for myself; it’s been on my travel wish list for a long time.

What VPC Peering Really Is

Alright, let’s slow down for a moment and imagine this scenario.

Think of two separate VPCs as two neighborhoods in the vast city of AWS. Each neighborhood has its own streets, houses, and landmarks i.e. in AWS terms, this means each VPC has its own network space, subnets, and EC2 running services.

Now, sometimes these neighborhoods need to communicate. Perhaps one neighborhood has a web server, and the other has a backend database. Or maybe they’re sharing resources for an application.

Without VPC peering, these neighborhoods would have to shout across the public internet just to talk to each other. Not only is this less secure, it’s also slower and inefficient. We want them to talk privately, as if they were on the same internal street.

This is exactly what VPC peering does.

It creates a private bridge between two VPCs.

Some important things to keep in mind:

1. Bidirectional: The connection works both ways. Traffic can flow from A → B and from B → A. If either side isn’t properly connected, the peering won’t work.

2. Non-overlapping CIDR ranges: Imagine street addresses: two houses cannot have the same number in different neighborhoods. Our VPCs must have distinct IP ranges.

3. No transitive peering: A common beginner mistake is expecting A → B and B → C to automatically allow A → C. It doesn’t. Every pair of VPCs needs its own explicit peering connection.

A Simple Example

• Primary VPC (A): 10.0.0.0/16

• Secondary VPC (B): 10.1.0.0/16

We want instances in A to communicate with instances in B privately. Here’s what happens:

1. Create a peering request from A → B.

2. Accept the request on B → A.

Now, both neighborhoods can communicate securely and privately, without using the internet.

Transitive Peering: What Not to Do

Let’s imagine adding a third VPC (C):

• A is peered with B

• B is peered with C

Some might expect that A can automatically talk to C. But it won’t work, this is called transitive peering. If A needs to communicate with C, we must create a direct peering connection between A and C. Each pair of VPCs needs its own bridge.

Setting Up Terraform for Multi-Region VPC Peering

Before we build anything, let’s take a moment to think about what we are trying to achieve.

We want two separate networks (VPCs) to communicate securely with each other. One VPC will live in US East (us-east-1), and the other will live in US West (us-west-2).

Now, Terraform doesn’t automatically know where each VPC should go. Think of it like trying to organize two separate offices in different cities. If you don’t tell the movers which office belongs where, things could end up in the wrong place! That’s exactly what the provider does, it tells Terraform “this resource belongs to this region”.

Step 1: Introducing the Providers

To handle two regions, we need two providers. Each provider is like a workspace for a region.

Here’s how we define them:

Now, let’s break it down:

• region: This tells Terraform which AWS region the resources will live in.

• alias: This is a nickname so Terraform knows which provider to use for each resource.

Later, when we create a VPC in US East, we will tell Terraform:

“Use the primary workspace.”

And for the VPC in US West, we will say:

“Use the secondary workspace.”

This way, Terraform never gets confused, even though we are managing multiple regions in a single project.

Step 2: Making It Flexible with Variables

Hardcoding values like regions or IP ranges in multiple places can be messy. Instead, we define variables to make our configuration clean, readable, and reusable.

We set important parameters once, and Terraform reads them everywhere it needs to.

Here’s the basic setup:

Let’s understand each piece:

• Primary and Secondary Region: These are the regions where Terraform will create the VPCs.

• CIDR Blocks: Think of a CIDR block like an address range for our network. Each VPC needs a unique range so that devices in one network don’t conflict with the other.

  • Primary: 10.0.0.0/16

  • Secondary: 10.1.0.0/16

Using variables keeps your code tidy. If we ever need to change the region or IP range, we just change it in one place, and everything else updates automatically.

Step 3: Planning the Multi-Region Network

Now that Terraform knows where each VPC belongs and we have defined the CIDR ranges, we can visualize what we are building:

1. Primary VPC in us-east-1

   • Subnet

   • Internet gateway

   • Route table

   • Security group

2. Secondary VPC in us-west-2

   • Subnet

   • Internet gateway

   • Route table

   • Security group

3. VPC Peering Connection

   • Links the primary and secondary VPCs

   • Updates route tables so instances can talk privately

Think of it like building two houses in different cities, each with its own doors, streets, and security fences. Then, you build a private bridge between them so people inside can visit each other without going through public roads.

Creating Primary and Secondary VPCs

In this section, we’ll create two VPCs (Virtual Private Clouds) in AWS using Terraform:

• Primary VPC in us-east-1

• Secondary VPC in us-west-2

We’ll understand each piece, and focus on why it’s needed. By the end, you’ll understand how VPCs, subnets, internet gateways, and route tables work together.

Primary VPC

Here’s the Terraform code for the primary VPC:

1. provider = aws.primary

   • This tells Terraform which AWS account and region to create the VPC in.

   • We’ve set up a primary provider with us-east-1 as the region.

2. cidr_block = var.primary_vpc_cidr

   • Every VPC needs an IP range (like an address range).

   • We’re using a variable (10.0.0.0/16) so it’s easy to change later.

3. enable_dns_hostnames & enable_dns_support

   • These allow instances inside the VPC to have DNS names.

   • This means instead of remembering IP addresses, you can use friendly names.

4. tags

   • Tags are labels in AWS that help you organize and identify resources.

By defining the primary VPC like this, we’ve created the foundation for our network.

Primary Subnet

Next, we create a subnet inside the VPC:

1. vpc_id = aws_vpc.primary_vpc.id

   • This links the subnet to our primary VPC.

2. cidr_block = "10.0.1.0/24"

   • Each subnet has its own smaller IP range within the VPC.

3. availability_zone

   • AWS divides regions into availability zones (AZs).

   • We use a data source so Terraform automatically selects one, instead of hardcoding.

4. map_public_ip_on_launch = true

   • Instances launched in this subnet automatically get public IPs, so we can connect via SSH.

5. tags

   • Again, tags help us identify and organize resources in AWS.

Internet Gateway

The internet gateway allows traffic to flow in and out of the VPC:

• vpc_id: Attaches the gateway to our primary VPC.

• Once attached, instances in the VPC can communicate with the internet.

• Later, when we set up VPC peering, traffic between VPCs can flow privately, without using the internet.

Route Table

The route table tells AWS how network traffic should flow:

1. route { cidr_block = "0.0.0.0/0" }

   • This means all traffic going outside the VPC should use the internet gateway.

2. route_table_association

   • Links the route table to the subnet, so the rules actually apply to our subnet.

Secondary VPC

The secondary VPC in US West 2 is created the same way, but with a few differences:

• cidr_block = 10.1.0.0/16 (non-overlapping with primary)

• All resources (subnet, internet gateway, route table) are configured similarly.

• The provider points to aws.secondary instead of aws.primary.

This ensures we have two independent VPCs, ready to be connected using VPC Peering.

Connecting the Neighborhoods: VPC Peering

Imagine we have two separate VPCs, one in US East (Primary) and one in US West (Secondary). Right now, they are completely isolated. If a server in Primary wants to communicate with a server in Secondary, the traffic would have to travel over the public internet. That’s not ideal. We want the traffic to stay private and secure inside AWS.

We have discussed all of this in the above sections, now lets see them in action!

Step 1: Request a Peering Connection (Primary → Secondary)

Here’s the Terraform code:

• resource "aws_vpc_peering_connection" "primary_to_secondary" → We are creating a VPC Peering Connection resource.

• provider = aws.primary → This tells Terraform which AWS account/region to use. Here, it’s the Primary VPC in US East.

• vpc_id = aws_vpc.primary_vpc.id → This is the ID of the VPC we want to connect from (Primary).

• peer_vpc_id = aws_vpc.secondary_vpc.id → This is the VPC we want to connect to (Secondary).

• peer_region = var.secondary_region → Since the Secondary VPC is in a different region, we tell AWS where to find it.

• auto_accept = false → AWS won’t automatically connect; the Secondary VPC must accept this connection.

• tags = {...} → These are labels to help us identify the connection in AWS. Think of them as name tags for your resources.

Step 2: Accept the Peering Connection (Secondary → Primary)

Now, the Secondary VPC needs to accept the connection:

• resource "aws_vpc_peering_connection_accepter" → This resource accepts the peering request.

• provider = aws.secondary → We’re now working in the Secondary region/VPC.

• vpc_peering_connection_id = aws_vpc_peering_connection.primary_to_secondary.id → Refers to the connection request we created earlier.

• auto_accept = true → Secondary is saying “Yes, Primary, you can connect to me.”

• tags = {...} → Helpful labels to organize the resource in AWS.

Now the private connection is fully active.

Step 3: Update Route Tables

Even after creating the connection, AWS doesn’t yet know how to send traffic between the VPCs. We need to update the route tables in both VPCs.

• provider = aws.primary → Route is for Primary VPC.

• route_table_id = aws_route_table.primary_rt.id → Specifies which route table to update.

• destination_cidr_block = var.secondary_vpc_cidr → Any traffic destined for the Secondary VPC should follow this route.

• vpc_peering_connection_id = aws_vpc_peering_connection.primary_to_secondary.id → Use the peering connection we just created.

• depends_on = [...] → Ensures Terraform waits for the peering to be accepted before adding the route.

The secondary route works the same way:

• Now traffic from Secondary knows how to reach Primary, completing the private connection.

EC2 Instances & Security Groups

Now that our private bridge (VPC Peering) is ready, it’s time to launch EC2 instances

We want these instances to:

• Communicate privately across the peering connection.

• Be reachable for management tasks (like SSH).

• Serve a simple webpage to verify connectivity.

Step 1: Security Groups: Controlling Access

Security Groups act like virtual firewalls or “guards” around your instances. They control who can talk to your EC2 instances and what type of traffic is allowed.

Primary Security Group

• provider = aws.primary → Security group is for the Primary VPC.

• vpc_id = aws_vpc.primary_vpc.id → Associates this security group with the Primary VPC.

• Ingress Rules → Control incoming traffic:

  • SSH from anywhere (port 22) → Allows us to log in to the instance from our computer.

  • ICMP from Secondary VPC → Allows “ping” commands from the Secondary VPC to test connectivity.

  • All traffic from Secondary VPC → Lets applications like NGINX communicate freely between the two VPCs.

• Egress Rule → Outgoing traffic:

  • Allow all outbound traffic so the instance can reach the internet or other services if needed.

The Secondary VPC’s security group is a mirror of the Primary, allowing two-way communication.

Step 2: EC2 Instances

With guards in place, we can now launch the EC2 instances.

Primary VPC Instance

• provider = aws.primary → This instance is in the Primary VPC.

• ami → Specifies which operating system image to use.

• instance_type → Determines the size and power of the EC2 instance.

• subnet_id → Places the instance inside the Primary subnet.

• vpc_security_group_ids → Attaches the security group we created earlier.

• key_name → Name of the SSH key to connect to this instance.

• user_data → A script that runs on first boot to install Apache/Nginx and display a webpage.

• depends_on → Ensures the instance is created after the VPC peering is active.

The Secondary instance is configured the same way, just using aws.secondary, its subnet, and its security group.

Step 3: User Data

User Data is a script that runs when an EC2 instance starts. Here, we use it to install Apache and display a simple webpage showing which VPC/region the instance belongs to.

Explanation:

• Updates the instance and installs Apache web server.

• Starts and enables Apache so the server runs automatically on reboot.

• Creates a simple webpage showing the VPC region and private IP.

• When we access this page from the other VPC, we see the bridge (peering) working.

Testing Connectivity Between VPCs

Now that we have set up VPCs, subnets, route tables, security groups, and EC2 instances, it’s time to see if our private bridge is working.

1. Log into the EC2 Instances

   • One instance is in US East 1 (Primary VPC)

   • The other is in US West 2 (Secondary VPC)

   • We can connect via SSH using the key pairs we created earlier.

2. Test Private Connectivity with Ping

   • From the Primary VPC instance, ping the private IP of the Secondary instance:

     

     

   • From the Secondary VPC instance, ping the private IP of the Primary instance:

• If the pings are successful, it confirms that ICMP traffic works and the peering connection is active.

3. Test Web Connectivity with Nginx

   • Each instance runs a simple Nginx server serving a webpage with its region and private IP.

   • Use curl to access the other instance’s private IP:

• We should see the HTML page showing the region and private IP of the other instance.

• This confirms that TCP traffic (used by web servers) works across the peering connection.

4. Key Takeaways

   • The VPCs can now communicate privately across regions.

   • Both ping (ICMP) and web traffic (TCP) work.

   • Your VPC peering setup is successfully complete!

5. Clean Up After Testing

   • Remember to destroy your Terraform resources to avoid unnecessary AWS charges:

Conclusion

Congratulations! With today’s blog, we are officially halfway through the 30 Days of Terraform challenge.

I know this session had a lot of code blocks, and just by reading it, anyone might feel a little overwhelmed and that’s perfectly okay. Feeling overwhelmed is actually a good sign. It tells us that there is something new here, something worth putting in the effort to truly understand.

Remember: nothing builds confidence faster than hands-on practice. Reading alone is never enough. The real understanding comes when you try, make mistakes, troubleshoot, and see things work with your own eyes.

To help you along, here’s a video by Piyush Sachdeva that explains everything clearly, step by step. You can follow along, try the exercises, and even troubleshoot with the tips he provides.

So take your time, practice today’s VPC peering setup, explore the EC2 instances, and test your connections. Tomorrow, we’ll continue building on this knowledge and dive into something new.

Happy Terraforming!

Hello and welcome back!

If you’ve been following the series, you know that yesterday, on Day 13, we spent time exploring Terraform data sources. We saw how Terraform can fetch information from AWS and reuse it, helping us keep our configurations clean, organized, and dynamic.

And today, we finally begin that “something bigger”.

This is Day 14 of 30 Days of AWS Terraform, and until now, we’ve been learning concepts one by one. But from today onward, we’re stepping into real mini-projects.

To start this phase, we’ll tackle a project that many beginners find comforting: hosting a static website using an S3 bucket, with CloudFront in front of it. We may have tried this manually in the AWS console at some point i.e. uploading HTML files to a bucket, enabling static website hosting, maybe even connecting a domain. But doing all of this with Terraform is different, and it shows how we can automate and manage infrastructure efficiently.

Before we dive into code, it’s important to understand what we’re building and why it matters. CloudFront and S3 aren’t just two AWS services placed together, they solve real problems. For example, when our website has visitors from different parts of the world, these services work together to make the website secure, fast, and cost-efficient, all while keeping the S3 bucket private.

Before we dive in, here’s a personal tidbit:

Fact #14: What I love about working in tech is that there’s always something new to learn. Sometimes it can feel a bit intimidating with so much happening all the time, but that’s also what makes it exciting.

And by the end of today’s blog, you’ll have your first real Terraform-powered static website up and running.

Understanding S3 + CloudFront Together

Before we touch any Terraform code, let’s take a moment to understand why we need both S3 and CloudFront and how they work together to make our website fast, secure, and reliable.

Imagine our website has visitors scattered all around the world. Some might be in India, others in the United States, Australia, Latin America, or Europe. Now, if all our website files i.e. like HTML pages, CSS styles, JavaScript, images, and videos are stored in a single S3 bucket, every visitor is trying to reach that same bucket, no matter where they are.

If we serve the website directly from S3, a few challenges arise:

• Cost: Every time a visitor far from the bucket downloads a file, AWS charges for the data transfer. These charges can add up quickly if our website has many visitors.

• Performance: Visitors far away will notice slower loading times, because the files have to travel long distances across the internet.

• Security: To make the website publicly accessible, the S3 bucket must be open to everyone. This can expose our files to attacks and isn’t safe for production use.

This is where CloudFront comes in. We can think of CloudFront as a network of helpers, stationed all around the globe. These helpers are called edge locations. When someone requests a file from our website, CloudFront serves it from the nearest edge location instead of the original S3 bucket.

Here’s what this achieves:

• Faster delivery: Files are stored temporarily at the edge locations. When someone requests the same file nearby, it is served immediately without going all the way to S3.

• Lower cost: Since the data travels a shorter distance, transfer fees are smaller.

• Better security: Users never access the S3 bucket directly. Only CloudFront can fetch files, keeping our bucket private.

Each cached file has a TTL (time to live), usually 24 hours by default. This means that if another user nearby requests the same file within this period, CloudFront serves it from its cache. This reduces load on the S3 bucket and makes the website feel snappy.

By combining S3 and CloudFront in this way, we create a website that is secure, fast, and cost-efficient, no matter where our users are.

Keeping this in mind will help us understand why we create each Terraform resource: the private S3 bucket, origin access control, bucket policies, CloudFront distribution, and finally uploading our static files. Each step in Terraform has a purpose, and together they tell the story of a website that works smoothly for everyone.

Setting Up the S3 Bucket with Terraform

Now that we understand why we’re using S3 and CloudFront together, let’s move into the “how.” We’ll start with the S3 bucket i.e. the heart of our static website. This is a place where all our website files will live: your HTML, CSS, JavaScript, images, and any other static content.

In Terraform, creating this bucket is surprisingly straightforward, but before we rush to the code, let’s pause and understand what we are doing. We want the bucket to have a unique, reusable name so we don’t run into conflicts with other buckets in AWS. Terraform allows us to do this using a variable-based prefix, which is a way of letting us define part of the name dynamically. Here’s what that looks like:

Here’s what’s happening in beginner-friendly terms:

• aws_s3_bucket tells Terraform, “Hey, I want you to create a new S3 bucket.”

• bucket_prefix lets us build a unique bucket name by combining a fixed string (day-14-demo-) with whatever value we set in var.bucket_prefix.

• The magic of using a prefix is that if we run this code multiple times, Terraform can still create new, unique buckets without manual renaming.

Once we have the bucket, we need to make sure it’s private. Even though AWS buckets are private by default, explicitly blocking public access is like double-checking and making sure it is really secure. It prevents anyone from accidentally accessing our buckets. Here’s the Terraform block for that:

Let’s unpack this:

• block_public_acls = true → Any access control lists (ACLs) that are public will be blocked.

• block_public_policy = true → No public bucket policies can be applied.

• ignore_public_acls = true → Existing public ACLs are ignored to avoid accidental exposure.

• restrict_public_buckets = true → The bucket cannot be made public accidentally in the future.

With the bucket created and secured, we’ve now laid the foundation for a website that’s both safe and ready to serve content efficiently. The next step will be setting up Origin Access Control, which allows CloudFront to fetch files from this private bucket safely

Configuring Origin Access Control and Bucket Policy

After creating and securing our S3 bucket, the next step is to allow CloudFront to access the bucket safely. Because the bucket is private, users cannot directly access it. Only CloudFront should fetch the files. To achieve this, we use Origin Access Control (OAC) and a bucket policy.

1. Origin Access Control (OAC)

Origin Access Control is a configuration in CloudFront that ensures all requests from CloudFront to the S3 bucket are authenticated and secure. Without it, CloudFront cannot read from a private bucket. Terraform allows us to create OAC easily:

Detailed explanation of each field:

• name
  A descriptive name for the OAC. We can choose any name, but it’s best to keep it clear and consistent.

• description
  Provides additional context about the purpose of the OAC. Helpful when managing multiple distributions.

• origin_access_control_origin_type = "s3"
  Specifies that the origin CloudFront will access is an S3 bucket. This is required because the signing and permissions differ depending on the origin type.

• signing_behavior = "always"
  Ensures every request from CloudFront to the S3 bucket is signed. Signing is required for authentication, allowing CloudFront to access a private bucket.

• signing_protocol = "sigv4"
  Specifies the AWS signature protocol used to sign requests. SigV4 is the latest secure protocol that encrypts and validates the request.

This configuration guarantees that CloudFront can securely read from the private S3 bucket while keeping the bucket inaccessible to other users or services.

2. S3 Bucket Policy

While OAC enables secure communication, the bucket policy defines explicit permissions on the S3 bucket. It tells AWS: “This CloudFront distribution is allowed to read these files.”

Terraform configuration:

Detailed explanation:

• bucket
  Specifies the S3 bucket to which the policy applies.

• depends_on
  Ensures Terraform creates this bucket policy after the public access block and CloudFront distribution are ready. This prevents errors during deployment.

• policy = jsonencode({...})
  The policy itself, written in JSON, defines who can do what on the bucket.

  • Version
    AWS uses this to determine the policy syntax version.

  • Statement
    Contains the permission rules. Each statement describes one rule.

    • Sid: A unique identifier for the statement.

    • Effect = "Allow": Grants the permission.

    • Principal = { Service = "cloudfront.amazonaws.com" }: Grants access only to CloudFront. No other user or service can read the files.

    • Action = ["s3:GetObject"]: Specifies that CloudFront can read objects (but cannot modify or delete them).

    • Resource = bucket ARN + / *: Applies to all objects in the bucket.

    • Condition = { StringEquals = { "AWS:SourceArn" = CloudFront distribution ARN } }: Ensures only this specific CloudFront distribution can access the bucket.

Together, OAC and the bucket policy make the S3 bucket private, secure, and accessible only to CloudFront. Users will never access the S3 bucket directly, but CloudFront can fetch and cache files efficiently for global access.

Uploading Static Website Files to the S3 Bucket

Now that our S3 bucket is private and CloudFront has its secure key to access it, we can start placing our website files into the bucket.

Our website consists of static files: index.html, CSS for styling, JavaScript for interactivity, and maybe some images or videos. These are the files our users will see and interact with when they visit our site.

Terraform provides a very neat way to do this using the aws_s3_object resource. Here’s what it looks like in code:

Let’s understand this, step by step:

Let’s go slowly, so it’s easy to understand:

1. for_each = fileset("${path.module}/www", "/*")**

   • This tells Terraform to look in the www folder and find all files and subfolders.

   • Terraform will create an S3 object for each file it finds.

2. bucket = aws_s3_bucket.day_14_demo_bucket.id

   • This is the S3 bucket we created earlier. Every file will be uploaded here.

3. key = each.value

   • The “key” is the name or path of the file inside the S3 bucket. This is how CloudFront and users will reference the file.

4. source = "${path.module}/www/${each.value}"

   • This points to the actual file on our computer, so Terraform knows what to upload.

5. etag = filemd5("${path.module}/www/${each.value}")

   • Terraform uses this to check if a file has changed. If the file is updated locally, Terraform will upload the new version.

6. content_type = lookup(...)

   • This tells the browser what type of file it is, such as text/html for HTML, text/css for CSS, or image/png for PNG images.

   • If Terraform doesn’t recognize the file type, it uses a generic type (application/octet-stream).

By using this resource, we ensure that all website files are uploaded correctly, with the right type, and ready for CloudFront to serve.

Once the files are uploaded, our website is ready to be delivered through CloudFront, making it fast, secure, and globally accessible.

Once all the files are safely uploaded, we’ll move to the next step: creating the CloudFront distribution. This is where our website will start becoming fast, globally accessible, and secure.

Creating the CloudFront Distribution

Now that our S3 bucket is ready and all our website files are uploaded, the next step is to make the website fast, secure, and accessible from anywhere in the world. That’s what CloudFront does.

CloudFront is a service that delivers our website to users efficiently. It ensures that:

• Our private S3 bucket stays private.

• Files are delivered quickly to users, no matter where they are.

• Visitors connect securely using HTTPS.

We will use Terraform to create a CloudFront distribution, which is just a configuration that tells CloudFront how to serve our website. Here’s the code we’ll use:

Let’s carefully go through this line by line, so it’s easy to understand:

1. origin

   • This tells CloudFront which S3 bucket to use.

   • We also provide the Origin Access Control we created earlier. This is necessary so CloudFront can access the bucket safely, while keeping it private from everyone else.

2. enabled = true

   • This simply turns the CloudFront distribution on. Without this, it won’t serve any content.

3. is_ipv6_enabled = true

   • This ensures visitors using newer internet addresses (IPv6) can access your website.

4. default_root_object = "index.html"

   • When someone visits our main website address, CloudFront will serve the index.html page automatically.

5. default_cache_behavior

   • allowed_methods and cached_methods: Only GET and HEAD requests are allowed. For a static website, this is all we need.

   • forwarded_values: We are not sending cookies or query strings, which makes caching simpler and faster.

   • viewer_protocol_policy = "redirect-to-https": Ensures visitors connect securely using HTTPS.

   • min_ttl, default_ttl, max_ttl: These settings tell CloudFront how long to keep a copy of each file cached at the edge locations. Cached files load faster for users and reduce requests to your S3 bucket.

6. price_class = "PriceClass_100"

   • Limits which locations CloudFront uses to deliver our website, keeping costs reasonable.

7. viewer_certificate

   • CloudFront provides a default HTTPS certificate so our site is secure without extra setup.

Once this distribution is active, we can use Terraform outputs to get the website URL, distribution ID, and bucket name for easy reference.

Understanding Terraform Outputs

At this point, we have created:

• A private S3 bucket containing all our website files

• A CloudFront distribution to deliver the site globally

Everything is working behind the scenes. But after running Terraform, how do we quickly find the important information we need? For example:

• What is the website URL?

• What is the CloudFront distribution ID?

• What is the S3 bucket name?

We could look through the AWS console to find these, but that can be slow and confusing, especially for beginners.

This is where Terraform outputs help. Outputs are like a summary table that Terraform shows us after deployment. They make it easy to see and use the information you need, without searching.

Here’s how we define them:

1. Website URL

After deployment, this output gives the link to our live website.

• We can copy this link and open it in a browser.

• CloudFront serves our website files from here, so we don’t access the S3 bucket directly.

• This makes it fast, secure, and globally accessible.

2. CloudFront Distribution ID

Every CloudFront distribution has a unique identifier.

• We may need this ID later if we want to update the distribution or check its settings in AWS.

• Terraform shows it to us automatically, so we don’t need to search in the console.

For beginners, knowing this ID is helpful because it connects the configuration we wrote in Terraform to the actual CloudFront service in AWS.

3. S3 Bucket Name

This output shows the name of our private S3 bucket.

• We can use this name if we want to look at the files in the bucket, check permissions, or connect it with other AWS services.

• Even though the bucket is private, Terraform makes it easy to reference the exact bucket name whenever needed.

By defining these outputs, you have all the critical information in one place after deployment.

• We don’t need to hunt in the AWS console.

• We can quickly access the website, manage CloudFront, or check your S3 bucket.

Conclusion

With this, we have completed Day 14 of our 30-day AWS Terraform journey. Today was our first demo session, where we saw an S3 bucket and CloudFront distribution in action.

I know this blog contains many moving parts, and some sections, especially when we look at the code, might feel hard to understand at first. That’s completely normal, and it’s part of the learning process.

One suggestion that usually helps me make sense of things is this:

1. Read the explanation for a piece of code.

2. Then go to the actual line of code and try to connect it with the explanation.

3. Repeat this slowly for each section.

It does take time and effort, but this approach usually helps the concepts “click” and makes the learning much more meaningful.

If some parts still feel confusing, which is completely understandable, here is a video by Piyush Sachdeva that explains everything clearly and walks you through the demo step by step.

Remember, learning cloud infrastructure and Terraform is a gradual process, and every small effort adds up. Keep practicing, and soon these concepts will feel much more natural.

Happy learning!

Hello and welcome back to Day 13 of 30 Days of AWS Terraform.

If you’ve been following along, you know that yesterday we explored Terraform functions i.e. those helpers that make our configurations cleaner, smarter, and flexible. We took things one step at a time, carefully understanding not just what these functions do, but why they matter when we start building real environments.

Today, we’re stepping into something that becomes incredibly useful once our Terraform projects start touching real AWS environments: data sources.

Instead of treating them as just another “feature” to check off a list, think of them as Terraform’s way of observing what already exists in AWS, picking up ready-made details, and helping us avoid hardcoding values that might change over time.

By the end of today’s blog, we’ll walk through why data sources are needed, how they fit into a simple EC2 example, and how they help when working in shared environments, like a pre-existing VPC and subnets used by multiple teams.

Before we dive in, here’s a little personal note:

Fact #13: I love collecting interesting metaphors to explain tech concepts. They help ideas stick, spark curiosity, and make tricky topics easier to understand. Over time, these metaphors have become a small treasure I revisit when learning or teaching.

What Exactly Is a Data Source?

Imagine we want to provision a simple EC2 instance. Nothing fancy.
Just an instance that needs a few basic things: instance type, subnet, tags… and of course, an AMI.

If you’re new to AWS, an AMI is basically the machine image i.e. the operating system template our instance will use. And there isn’t just one AMI. There are multiple versions for Amazon Linux, Ubuntu, CentOS, and many others. These images get new releases over time, so the list keeps growing.

What’s interesting is that AMIs aren’t stored inside our AWS account.
They live outside, almost like a big open catalogue. And because there are so many of them, each release ends up having its own unique AMI ID.

Now here’s the challenge:
When we create an EC2 instance in Terraform, we need to provide that AMI ID. And yes, we can manually go to the AMI release page, copy the AMI ID, and paste it in our Terraform file… but that’s not something we want to rely on once we start automating things.

In an automated setup, the goal is simple:
no manual intervention, or as little as possible.

So hardcoding the AMI ID isn’t a good idea. It becomes outdated very quickly, and every update forces us to go hunting for the latest one.

This is where Terraform’s data sources step in.

A data source is like saying to Terraform,
“Hey, instead of me manually telling you the AMI ID, can you go and fetch the correct one for me?”

And Terraform says,
“Sure. Just tell me what type of AMI you want i.e. Amazon Linux 2, Ubuntu, anything, and I’ll go find the most recent one.”

With just that, Terraform fetches the latest Amazon Linux 2 image and uses that AMI ID when creating our EC2 instance.

That’s the core idea behind data sources.
They help Terraform pull in information that already exists, whether it’s an AMI, a VPC, a subnet, or anything else, so that we don’t have to create everything from scratch or maintain static values.

A Real-World Scenario: Using What Already Exists

Now that we understand why data sources are helpful, let’s step into a slightly more practical example. We’ll try to see how Terraform behaves when we’re working with resources that are already present in AWS.

Imagine an organization where multiple teams share the same VPC.
It’s pretty common: the DevOps team, the development team, QA… everyone uses this one shared VPC that is already created and maintained in the AWS environment. Inside this shared VPC, imagine there are two subnets i.e. Subnet 1 and Subnet 2, and they’re also already set up.

Now, for our task, we need to create a few EC2 instances:
two in Subnet 1 and two in Subnet 2.
Simple enough, but here’s the important part:

We’re are not creating a new VPC or new subnets.
We have to use the existing ones.

And this brings us back to the value of data sources.

Because if the VPC and subnets are already present in AWS, Terraform doesn’t know their IDs. It needs a way to look them up, just like it looked up the AMI earlier. Hardcoding these IDs is not ideal, especially when we’re working in environments where different teams might update or rename things over time.

So, just like we used a data source to fetch the latest Amazon Linux AMI, we’ll use data sources again, but this time for the VPC and the subnets.

This small setup i.e a shared VPC, shared subnets, and a few new EC2 instances, is the perfect place to learn how data sources help us connect Terraform to infrastructure that already exists. We’re still keeping things small, but this is the kind of approach enterprises use: slowly building on top of shared components without recreating everything from scratch.

Creating the Data Source for the VPC

Now that we understand why we need data sources, let’s start actually writing them. And just like always, we’ll take it one small step at a time.

Before this, whenever we looked up resources in Terraform documentation, we focused on the resource section, because we were creating things like EC2, VPC, security groups, and so on.
But now that we want to read something that already exists, we have to shift our attention to the data source section of the documentation instead.

So let’s begin with the first thing we need: the existing VPC.

In our AWS account, this shared VPC already exists and it has a tag named “day-13-vpc”.

To fetch this VPC, we use Terraform’s data source for AWS VPC:

This tiny block does something very powerful.
It tells Terraform:
“Look through all the VPCs in my AWS environment and find the one whose Name tag is set to ‘day-13-vpc’. Once you find it, make its details available to me.”

We’re not creating anything here.
We are simply filtering from what already exists.

And once Terraform finds this VPC, it will store all its details, like its ID, CIDR block, and more inside this data source. Later, when we create our EC2 instances, we can reference this VPC simply by calling:

It like asking Terraform,
“Can you show me the ID of that VPC we fetched earlier?”

This is the beauty of data sources.
They quietly connect Terraform to the real AWS environment without us having to hardcode anything or recreate resources that are already there.

Creating the Data Source for the Subnet

Now that Terraform knows how to fetch the existing VPC, the next step is to help it find the subnet inside that VPC.
Remember, both the VPC and the subnets already exist in our AWS environment, we’re simply asking Terraform to look them up.

So just like before, we go to the data source section in the Terraform documentation, but this time we search for AWS Subnet.

Let’s say the subnet we want has a tag called “day-13-subnet”.

To fetch this specific subnet, we can write the following:

Let’s pause and understand what’s happening here.

First, we apply a filter that searches for a subnet whose Name tag matches “day-13-subnet”.
This makes sure Terraform doesn’t return just any subnet but only the exact one we want.

Second, we link it to the VPC data source we created earlier:

With this small piece, Terraform now knows exactly which existing subnet to use when we later create our EC2 instance.

Nothing is being created here yet.
We’re still just gathering information and preparing Terraform for the final configuration.

In the next part, we’ll do the same thing for the Amazon Linux 2 AMI, because every EC2 instance we create needs a proper AMI. And again, we don’t want to hardcode it.

Creating the Data Source for the AMI (Amazon Linux 2)

Now that our VPC and subnet data sources are ready, there’s one more important piece we need before creating an EC2 instance: the AMI.

Just like we discussed earlier, an AMI (Amazon Machine Image) is basically the operating system template our EC2 instance will use. And there’s not just one AMI, there are multiple versions released over time. So instead of manually searching and copying AMI IDs, we’ll let Terraform fetch the latest one for us.

Terraform provides a data source just for this purpose.
Let’s build it.

Here’s how we define the Amazon Linux 2 AMI data source:

Let’s break this down:

• owners = ["amazon"] ensures we only pull AMIs officially published by Amazon, not community images or anything random.

• most_recent = true tells Terraform, “Always pick the latest Amazon Linux 2 image available.”

• The first filter looks for AMIs whose name pattern matches the Amazon Linux 2 naming format. The stars (*) allow Terraform to match any version.

• The second filter restricts the virtualization type to HVM, which is the standard for modern Amazon Linux AMIs.

We can add more filters if needed, like root device type, specific naming patterns, or architecture but for our example, this is more than enough.

With this, Terraform will always fetch the latest, correct Amazon Linux 2 AMI, without us ever needing to manually update anything. This tiny block brings a lot of reliability to our configurations, especially when AMIs get updated frequently.

Now we have all three data sources ready:

• VPC

• Subnet

• AMI

And in the next part, we'll finally use these data sources inside our aws_instance resource i.e. the part that actually creates the EC2 instance.

Creating the EC2 Instance Using Data Sources

Now that we have everything ready i.e. the VPC, the subnet, and the Amazon Linux 2 AMI, it’s time to bring it all together and create our EC2 instance.

Here’s what the resource block looks like in Terraform:

Let’s go through this:

• ami = data.aws_ami.linux2.id
  This tells Terraform to use the latest AMI we fetched using the data source. No hardcoding just the correct, up-to-date image.

• instance_type = "t3.micro"
  We are keeping it simple with a small instance type.

• subnet_id = data.aws_subnet.shared.id
  Terraform will place the EC2 instance in the subnet we fetched from the data source. Again, no manual ID lookup is needed.

• tags = var.tags
  Tags help us identify and organize our resources. We’re reusing the same tags variable as before.

We would be using the following values in var.tags:

Now, after writing this, you can save the file and run:

We’ll notice that Terraform recognizes one new resource i.e. the EC2 instance. It will use the existing VPC, the existing subnet, and the latest AMI.

We didn’t create any VPCs or subnets; Terraform simply referenced the ones we fetched using data sources.

Once you’re ready, running terraform apply will create the instance.

After the demo, make sure to destroy the resource to avoid unnecessary costs.

Conclusion

And that’s it for Day 13!

With today’s demo, we wrapped up our first hands-on example using Terraform data sources.

This being our first practical demo, I hope it gave you a feel for what a hands-on experience is like. There’s something truly valuable about seeing concepts in action i.e. how small, careful steps in Terraform build up into something real, and why practicing hands-on is so important for understanding and confidence.

Before we close, a small reminder: after you’ve applied this demo, don’t forget to destroy the instance when you’re done. This keeps costs under control while you continue experimenting and learning.

If some parts feel a little confusing, here’s a video by Piyush Sachdeva that explains everything clearly, step by step. Watching it can help reinforce what we covered today and give you another perspective on hands-on Terraform practice.

Looking ahead, Day 14 will be a mini project where we combine what we’ve learned so far into a slightly larger example. I’m excited to continue this journey with you.

Thank you for staying patient and curious today. I hope you feel encouraged by your first hands-on experience and I’ll see you tomorrow for Day 14!

Hello and welcome back, it’s good to see you here again.

We’re now on Day 12 of our 30-day AWS Terraform journey, and it really does feel like we’re continuing a conversation we paused only yesterday. That steady, day-by-day progress is what makes this whole thing manageable and, honestly, kind of enjoyable.

If you remember, on Day 11 we opened the door to Terraform functions: we saw what they are, ran a few demos, and began to get a feel for how Terraform “thinks.” That day was about getting comfortable with the ideas and seeing how small functions fit into a larger workflow. Today we pick up right where we left off, carrying the same slow, steady pace.

So settle in, make yourself comfortable, and let’s explore this next chapter together.

Personal tidbit time:

Fact #12: I’m not a big fan of mathematics, as a kid I used to dread it. And I think a big part of that was because I never really tried to understand it. Looking back, I actually feel a little sad for my younger self, if only I could explain things to that version of me in a simpler way, maybe it wouldn’t have felt so overwhelming.

Understanding Validation Functions

Before we jump into the Terraform code, let’s take a moment to understand why validation functions matter.

Imagine we’re asking someone to fill out a small form maybe choosing an instance type or providing a backup name. Now, people make mistakes. They might type something too short, too long, or in a completely different format. If Terraform went ahead without checking anything, we’d only discover the mistake much later, often after the infrastructure has already been created incorrectly.

That’s where validation functions step in.

And the best part? These validations happen before Terraform even starts planning anything.

Why validation goes inside the variable block

Unlike other checks we’ve done before, validation has to live inside the variable declaration itself.
That’s because we’re telling Terraform:

“Whenever someone provides a value for this variable, please check these rules first.”

Here’s the example:

Let’s break this down.

1. First validation: checking the length

Terraform first checks whether the instance type is at least 2 characters long and at most 20.

Why?
Because sometimes someone may type something extremely short like t or something very long accidentally. This rule keeps things sensible.

The validation says:

• if the length is 2–20, we’re good

• if not, Terraform stops you with the message:
  “instance type must be between 2 and 20 characters”

Simple and helpful.

2. Second validation: checking if it starts with t2 or t3

This one is more interesting.

Here’s the condition again:

Let’s break it into very easy language:

• regex()
  → checks whether a value matches a specific pattern

• the pattern ^t[2-3]\. means:
  “The value should start with t2. or t3.”

• can()
  → used so Terraform can safely test the regex without crashing

Think of the regex like a rulebook:

• ^ = start of the string

• t = must start with the letter "t"

• [2-3] = followed by either 2 or 3

• \. = then there must be a dot (.)

So valid examples are:

• t2.micro

• t2.small

• t3.large

Invalid ones would be:

• m5.large

• t4.micro

• t2micro (missing dot)

And if the input doesn’t follow the pattern, Terraform returns our custom error:

“Instance type must start with T2 or T3.”

This makes the error more human-friendly and less confusing.

Why can() is needed

If we used only regex(), Terraform might throw a harsh technical error if the pattern didn’t match.
But with can(), Terraform simply checks:

• If regex works → return true

• If regex fails → return false, but gracefully (no crashes)

An example to relate

Imagine we're checking someone’s ID:

• First, check if the ID length is reasonable (not too short, not too long).

• Then check if it starts with a certain prefix like "AB" or "BC".

If both checks pass, great, let them in.
If not, we politely say, “Sorry, this ID doesn’t look right.”

Terraform is doing exactly that for our variables.

A Practical Validation: Ensuring a Backup Name Ends With -backup

Now that we understand how validations work with instance_type, let’s look at another everyday example, something even simpler.

Imagine we’re creating a variable called backup name.
We want every backup to follow a proper naming pattern so it’s easy to identify later.

Let’s say the rule is:

“Every backup name must end with -backup.”

Why is this useful?
Because when our systems grow, naming patterns save us from chaos.
Backups become easy to filter, group, search, and automate around, all because they follow a familiar ending.

The Terraform validation

Here’s how it might look:

Let’s unpack this.

1. endswith() function

endswith(string, substring) simply checks whether a value ends with a specific text.

So:

• "mydata-backup" → valid ✔

• "backup-mydata" → invalid ❌

• "mydata-backup123" → invalid ❌

Terraform evaluates the condition, and if it fails, it shows you the custom message we provided.

We can enforce structure before Terraform even plans anything.

Understanding Sensitive Variables (Why Some Values Stay Hidden)

Some Terraform variables hold harmless information, like names or flags.
But others may contain things that should never appear in logs or screens i.e. things like passwords, tokens, keys, or database connection strings.

Terraform gives us a simple way to protect such values:

Mark the variable as sensitive = true.

This doesn’t encrypt the value, but it does hide it from output, logs, and accidental exposure during terraform plan or terraform apply.

Terraform example

Now, even if someone tries to print it:

Terraform will hide it.
You’ll see something like:

Why this matters

Sensitive variables protect us from:

• accidental screen-sharing leaks

• logs exposing secrets

• commit mistakes when values accidentally get echoed

• teammates seeing secrets they don’t need to

It’s not encryption, but it’s a safety layer that prevents embarrassment and accidents.

Validations Using Regex

So far, our validations checked simple conditions like:

• Does this value belong to a list?

• Does this value end with “-backup”?

But sometimes we need more powerful rules, rules that check patterns, formats, or structures.

This is where regex (regular expressions) comes in.

Regex sounds scary, but in Terraform, we’ll mostly use it for simple checks like:

• Does the value contain only letters and numbers?

• Does the name follow a specific pattern?

• Does it start with something specific?

• Does it avoid special characters?

A simple explanation of regex

Think of regex as a pattern detective.

Instead of checking a single condition, it looks for a shape in the text.

Example in plain English:

“The name must start with a letter and can only contain letters, numbers, and hyphens.”

Regex lets Terraform enforce that in one line.

Terraform regex example

Let’s say we want our environment names to follow this rule:

“Must start with a letter, and can contain letters, numbers, and hyphens.”

The regex for that (very common pattern) is:

We’ll understand what the above gibbersh means in a while.

Here’s how we apply it in Terraform:

Breaking this down in simple terms

1. ^ and $

These mean “start” and “end”.

Terraform checks the whole string, not just a part.

2. [a-zA-Z]

The name must start with a letter.

3. [a-zA-Z0-9-]*

After the first letter, we can have:

• letters

• numbers

• hyphens

• repeated any number of times (*)

Why regex matters in Terraform

When our project grows, naming mistakes create chaos:

• S3 buckets with weird characters

• IAM roles with spaces

• Environment names with typos

• Random uppercase/lowercase mixtures

Regex is our gatekeeper.

It stops wrong values before they spread through our infrastructure.

If this section is good, we’ll move to the next one:

Validation Rules for Lists

So far, we’ve seen validations for single values: an instance type, a backup name, or an environment name.
But what if our variable is a list of values? For example, a list of regions, instance types, or users.

Terraform lets us apply validations to every item in the list, ensuring consistency across the board.

Why this matters

Imagine we have a list of regions:

We want all regions to:

• Follow the proper format

• Only contain letters, numbers, and hyphens

• Avoid typos or accidental spaces

Without validation, Terraform would happily accept a wrong entry, and our deployments could fail later.

With validations, Terraform stops mistakes upfront.

Terraform example

Suppose we have a variable for allowed regions:

Breaking it down

• for r in var.allowed_regions
  Iterates over every region in the list.

• regex("^[a-z]{2}-[a-z]+-[0-9]$", r)
  Ensures each region follows AWS naming conventions i.e.

  • Accepted values: us-east-1

  • Rejected values: USEAST1, us-east

• can()
  Handles errors gracefully if the regex fails.

• alltrue([...])
  Ensures that every single item passes the condition. If even one fails, Terraform stops and shows our custom error.

Why list validation is useful

• Prevents subtle mistakes across multiple entries

• Makes infrastructure predictable

• Saves time debugging deployment errors

• Ensures human-friendly, consistent standards

Type Conversion Functions (Lists, Sets, and Unique Values)

In Terraform, sometimes we need to change the type of a variable to make it easier to work with.
This is especially common when dealing with lists and sets, or when we want to remove duplicate values.

Why this matters

Imagine we have a list of regions:

Notice how “us-east-1” appears twice.

• A list allows duplicates

• A set only keeps unique values

Converting a list to a set is a simple way to automatically remove duplicates, making our infrastructure cleaner and more predictable.

Terraform example: Converting a list to a set

Suppose we have two variables:

We can combine them with concat:

This produces:

Now, to get only unique locations, use toset:

Result:

Duplicates are automatically removed, no extra effort needed.

Why type conversion is useful

• Cleans up data automatically

• Makes further operations easier

• Prevents mistakes when working with lists that shouldn’t have duplicates

• Bridges differences between Terraform’s list and set types

Number Functions: Sum, Max, Min, Absolute, and Average

Terraform provides several handy number functions that help us perform calculations on variables, especially lists of numbers.'

These functions are useful for things like monthly costs, resource counts, or any scenario where we need math on our infrastructure data.

The scenario

Suppose we have a list of monthly costs (some are credits, so negative):

We want to:

1. Convert everything to positive numbers

2. Find the maximum cost

3. Find the minimum cost

4. Calculate the total cost

5. Calculate the average cost

Step 1: Convert to positive values

Terraform’s abs() function returns the absolute value of a number.
But because abs() doesn’t work on a list directly, we use a for loop:

Result:

Step 2: Find max and min

Terraform provides max() and min() functions.
Apply them on our positive list:

Result:

Step 3: Calculate total and average

To sum all costs:

Result:

Average is simply total divided by the number of elements:

Result:

Why this matters

• Handles negative/credit values easily

• Calculates statistics across any list of numbers

• Helps in budgeting, reporting, or monitoring infrastructure costs

• Makes our Terraform configurations smarter and more automated

Timestamp Functions: Current Time and Formatting

Terraform also provides timestamp functions to help us capture and format the current time.
This can be useful for logging, backups, or creating time-stamped resources automatically.

An example

Terraform’s timestamp() function doesn’t need any input:

This produces a value like:

This is the current UTC timestamp in ISO 8601 format.

Formatting the timestamp

Sometimes, we may want a more human-friendly format.
Terraform provides formatdate():

Result:

Now we can use this formatted timestamp in backup names, logs, or resource tags.

Why timestamp functions matter

• Helps with automation that relies on dates

• Makes logs and outputs easier to interpret

• Supports versioning and auditing in our Terraform deployments

File Handling Functions

Terraform provides file handling functions that let us read, decode, and use data from files directly in our configurations.

This is especially useful when working with JSON configuration files or other structured data.

The scenario

Suppose we have a JSON file config.json:

We want Terraform to:

• Check if the file exists

• Read its contents

• Decode the JSON into a usable map

Terraform example

Breaking it down

1. fileexists("config.json")
   Checks if the file exists. Returns true or false.

2. file("config.json")
   Reads the contents of the file.

3. jsondecode(...)
   Converts the JSON text into a Terraform map or object.

4. Conditional operator (? :)
   If the file exists → decode it
   If not → return an empty map {}

Output example

Now Terraform can use the database host, API endpoint, or any other value from the JSON file in our infrastructure resources.

Additional notes

• Terraform also provides jsonencode() to convert data structures back into JSON

• File handling functions let us reuse external data safely and efficiently

• Great for keeping secrets, configurations, or templates outside our main .tf files

Why this matters

• Makes our Terraform configurations more dynamic and flexible

• Keeps sensitive or large configuration data outside our main code

• Supports automation, templating, and scaling your deployments

Conclusion

And with this, we’ve reached the end of Day 12 in our 30-day Terraform journey.
Today we closed the loop on the remaining Terraform functions, everything we couldn’t finish in Part 1 finally found its place here in Part 2.

We explored validations, sensitive values, type conversions, numeric functions, timestamps, and even dipped our toes into file handling.
Each of these shape the kind of Terraform code that behaves reliably, predictably, and in a way that protects us from surprises.

And I just want to say this: if you’ve been following along till Day 12, that’s not a small thing.
Learning something every day, especially something as deep and layered as Terraform, is not easy.

We’ve come this far, and there’s still a long, exciting road ahead. We’ll tackle it together!

And hey, if you still have doubts or something felt a bit fuzzy, here’s a helpful video by Piyush Sachdeva that might clear things up.

Never hesitate to revisit, rewatch, or relearn.

With that thought, I’ll leave you for today. We’ll catch up again tomorrow with a new topic tomorrow.

Peace!

Hello and welcome back to our 30 Days of AWS Terraform journey! How’s everyone doing today? I hope life is treating you kindly and that you’re finding a little time to enjoy this learning adventure with me.

Last session, we dived into some fascinating pieces of Terraform i.e. conditional expressions, splat expressions, and dynamic blocks. Each of these little gems helped us see how Terraform makes infrastructure descriptions more flexible and expressive.

Today, we’re turning our attention to Terraform Functions: Part 1.

Now, if the word “function” makes you imagine complicated programming, don’t worry, that’s not what we’re doing here. Terraform isn’t a programming language. It’s a configuration language, which simply means we describe what we want our infrastructure to look like, rather than writing a program to compute it. And to make that description a bit smarter and more reusable, Terraform provides built-in helpers called functions.

Because there are so many functions, we’ll explore them in two parts. In Part 1, we’ll cover what a function really is, why Terraform only gives us inbuilt functions, and how these helpers can make our variables and values cleaner and easier to manage.

This journey isn’t about overwhelming us it’s about learning small, practical tools that help us express what we already know in a smarter, more organized way.

And here’s a little personal tidbit:

Fact #11: I prefer learning from YouTube tutorials instead of going through documentation. For me, I like to see things in action, that makes things click faster for me.

Understanding Functions and Terraform’s Inbuilt Functions

What is a Function, Really?

Before we dive into Terraform-specific functions, let’s take a moment to understand what a function really is, in the simplest way possible.

In plain language, a function is something that makes our life easier. How? By letting us reuse instructions instead of writing the same steps over and over again.

Imagine this: we want to add two numbers. In a typical programming language, we might write something like this:

Simple enough. But what if we had to do this 10 different times, with different numbers each time? Sure, we could copy and paste those four lines again and again, but that’s inefficient and tedious.

This is exactly where a function comes in. Instead of rewriting the same code multiple times, we wrap it once in a function, and then just call it whenever we need it, passing in the numbers we want to add. The logic is reused, our code stays clean, and our work becomes faster and more organized.

Terraform is Not a Programming Language

Now, we might already know this, but it’s worth repeating: Terraform isn’t a full-fledged programming language. It’s a configuration language, officially called HCL i.e. HashiCorp Configuration Language.

What does that mean?

• Terraform does not have classes, objects, or traditional OOP features.

• We cannot create our own custom functions.

Instead, Terraform provides a set of inbuilt functions that we can use anywhere in our configurations. These helpers cover a variety of tasks, including:

• String manipulation

• Numeric operations

• Collections (lists and maps)

• Type conversion

• Date and time handling

• Validation and lookup

Even though Terraform can’t do everything a programming language can, these inbuilt functions give us the power to write clean, reusable, and dynamic configurations for our infrastructure.

Categories of Terraform Functions

Here’s a bird’s-eye view of the function categories we’ll explore:

• String functions: manipulate and transform text

• Numeric functions: calculate max, min, absolute, etc.

• Collection functions: work with lists and maps

• Type conversion functions: convert between lists, sets, numbers, and strings

• Date and time functions:work with timestamps and formatted dates

• Validation and lookup function: check or select values dynamically

Terraform String Functions

Now that we understand what a function is and why Terraform provides inbuilt functions, let’s explore string functions, these are some of the most commonly used helpers in Terraform.

String functions let us manipulate text, making it easier to format, clean, or transform strings before using them in our infrastructure.

1. Upper and Lower Functions

• upper() converts a string to uppercase.

• lower() converts a string to lowercase.

Example:

Notice how we call the function and pass the string as an argument. This is exactly how functions let us reuse logic without rewriting it.

2. Trim Function

• trim(string, characters) removes unwanted characters (like spaces) from the beginning and end of a string.

Examples:

• The first argument is the string to clean.

• The second argument is the character to remove.

3. Replace Function

• replace(string, search, replace) replaces occurrences of a character or substring with another.

Example:

• The first argument is the original string.

• The second is the character to replace.

• The third is the new character.

4. Substring Function

• substr(string, start, length) extracts a portion of a string.

Example:

• Starts at index 0 (the first character) and goes up to length 3.

• Perfect for trimming or formatting strings in Terraform.

These functions are especially helpful when we need to clean inputs, format names, or prepare strings for resources like S3 bucket names, which have strict naming rules.

Numeric and Collection Functions in Terraform

1. Numeric Functions

Numeric functions help perform simple calculations or analyze numbers.

max(): returns the largest number in a set.

• min(): returns the smallest number.

  

  

• abs(): returns the absolute value (removes the negative sign).

  

  

These functions are helpful when working with variables where you might need to select limits, bounds, or ensure positive values.

2. Collection Functions

Collection functions let us manipulate lists and maps, which are fundamental in Terraform for storing multiple values.

• length(): returns the number of elements in a list.

• concat(): combines multiple lists into one.

• merge(): combines multiple maps (key-value pairs).

Collections are very common in Terraform. For example, when managing tags, security group rules, or multiple ports, these functions let us combine, iterate, and manipulate values cleanly and dynamically.

Type Conversion and Date/Time Functions in Terraform

In Terraform, sometimes we need to change the type of a value or work with time-based data. This is where type conversion and date/time functions come in handy.

1. Type Conversion Functions

Type conversion functions let us change one type of value into another, ensuring Terraform can handle it correctly.

• toset(): converts a list into a set.

• tonumber(): converts a string representing a number into an actual numeric value.

Note: In Terraform, anything in double quotes is treated as a string, even if it looks like a number. So we use tonumber() to ensure numeric operations work correctly.

2. Date and Time Functions

Terraform also allows us to work with timestamps or format dates:

• timestamp(): returns the current time in a predefined format.

• formatdate(): allows custom formatting of the timestamp.

These functions are useful when we need timestamps for logs, resources, or versioning, or when a specific format is required for our infrastructure configurations.

Using Terraform Functions in Practical AWS Examples

Let’s see how some of the Terraform functions we discussed can be applied in real configurations.

1. Lowercase and Replace for Variable Formatting

Imagine we have a variable for our project name:

We can use the lower() function to convert it to lowercase:

• Output will be: "mission is terraform"

Now, if we want to replace spaces with hyphens, we can combine functions:

• Output: "mission-is-terraform"

This ensures names are clean, lowercase, and safe for resources like S3 buckets.

2. Merge Function for Tags

Terraform often uses maps for tags. We might have default tags and environment-specific tags:

• merge() combines the two maps into one, ensuring all tags are applied without duplication.

• This is a practical way to manage resource metadata consistently.

3. Substring, Lower, Replace for Bucket Names

S3 bucket names have restrictions: no capitals, no spaces, max 64 characters.

We can enforce this using multiple functions together:

• lower() → converts to lowercase

• substr() → limits length to 64 characters

• replace() → replaces spaces with hyphens

Even if someone provides a bucket name with capitals or spaces, Terraform automatically formats it safely, preventing errors during terraform apply.

4. Split and For Expression for Multiple Ports

Suppose we have a variable with comma-separated ports:

• First, split the string into a list:

• Then, iterate over the list to create structured data:

This is a simple way to convert a string input into multiple security group rules, automating repetitive work.

5. Lookup Function for Environment-Specific Values

Sometimes, we want Terraform to pick a value automatically based on the environment we’re working in. For example, maybe we want smaller instances for development and larger instances for production.

Here’s a simple example:

Let’s break this down:

1. var.instance_sizes → This is a map that tells Terraform which instance to use for each environment.

   • dev → t2.micro

   • staging → t3.small

   • prod → t3.large

2. var.environment → This tells Terraform which environment we are currently working in. Here, the default is "dev".

3. lookup() → This function looks at the map, finds the key that matches the environment, and returns the corresponding value.

   • First argument: the map of values (var.instance_sizes)

   • Second argument: the key to look up (var.environment)

   • Third argument: the default value to use if the key is missing ("t2.micro")

So in this example:

• Terraform sees that the environment is dev.

• It looks in the map and finds dev = t2.micro.

• It sets local.instance_size to "t2.micro".

This approach makes your configuration flexible and safe, because you can switch environments without manually changing values everywhere.

Conclusion

And just like that, we’ve reached the end of Day 11 of our 30 Days of AWS Terraform journey.

Today, we explored Terraform Functions: Part 1. We saw how these functions can make our Terraform configurations cleaner, smarter, and more dynamic.

I know, at first glance, some of these functions might seem a little tricky. We might even wonder, “Do I really need all these extra steps?” Believe me, I’ve felt the same way.

But here’s the thing: if these functions exist, it’s because they are designed to make our lives easier. Instead of rewriting the same logic over and over, they help us reuse, automate, and avoid mistakes, so we can spend our energy on what truly matters i.e. building great infrastructure.

So, with that thought in mind, I’ll leave you here for today. Take a little time to practice what we covered, play around with the examples, and let the concepts sink in.

If any doubts remain, here’s a video by Piyush Sachdeva that explains these concepts in another friendly way, which might help clear up any lingering questions.

Tomorrow, we’ll continue with Terraform Functions Part 2, diving into more functions and practical examples to make your configurations even more powerful.

Until then, take care, stay curious, and peace out!

Hello and welcome back to our 30 Days of AWS Terraform journey. How are you doing today? I hope you’re well!

We’ve made it to Day 10!

Seriously, showing up, learning a little each day, is the part that builds real skill. So let’s take a moment to feel good about that.

Yesterday, in Day 9, we spent time with Terraform lifecycle rules. We learned how lifecycle settings let us protect, control, or shape how resources behave during changes. It reminded us that Terraform is not just about creating infrastructure, but it’s about guiding how things change over time.

Today we continue that same careful, thoughtful approach, but with a different kind of tool.

We’re stepping into the world of Terraform Expressions. Think of expressions as tiny helpers inside your configuration: small bits of logic that let Terraform make simple decisions, repeat useful blocks for us, and gather values without extra fuss. They’re not full functions (we’ll cover those later), but they make our files cleaner and our work gentler.

In this post we’ll explore three friendly expressions:

• Conditional Expressions: letting Terraform pick one value or another based on a simple condition.

• Dynamic Blocks: generating repeating nested blocks (like many ingress rules) without copy-paste.

• Splat Expressions: grabbing a single attribute from many resources in one tidy line.

As always, we’ll walk through demos so these ideas stop being abstract and start feeling practical. If any phrase sounds odd at first, that’s okay, it’ll click when we see it in action.

Here’s a little fun fact before we dive in:

Fact #10: My phone is filled with random screenshots of quotes I love, and I just can’t bring myself to delete any of them. Every single one feels important, even the silly ones..and btw, I’m out of space

Ready? Let’s begin with Conditional Expressions.

Conditional Expressions

Before we touch any code today, let’s take a moment to understand why we’re learning this.

When we build infrastructure, we often face tiny decisions:

• Should this environment use a small instance or a bigger one?

• Should this bucket be public or private?

• Should this feature be ON or OFF?

Doing this manually every single time becomes tiring, and honestly… it invites mistakes.

Terraform gives us a small tool to handle these decisions automatically i.e. conditional expressions.

And the good news?
If you’ve ever said “If this happens, do that,” or worked with if-else conditions, then you already understand the idea.

Why Do We Even Need Conditional Expressions?

Imagine this:

We’re working on three environments: dev, staging, and production.

Dev needs a tiny instance.
Staging needs something a bit stronger.
Production? Definitely more power.

If we hard-code the instance type each time, we’ll constantly have to remember:

“Wait, which size do we use for which environment?”

That’s stressful.
Terraform helps us remove the stress by letting it choose on our behalf.

A conditional expression simply tells Terraform:

“If this condition is true, pick value A. Otherwise, pick value B.”

Let’s Build It Together

Let’s start with a basic EC2 instance:

Right now, the instance type is fixed. But that’s the old way i.e. manual, repetitive, easy to forget.

Let’s guide Terraform to choose the instance type on its own.

The Conditional Pattern

Terraform uses a simple and readable structure:

Think of it like reading a sentence:

• “If this is true → pick this value.”

• “Else → pick that value.”

So let’s apply that to our environment.

Our First Real Example

If the environment is dev, we want a small instance:

• "t2.micro"

If it’s anything else (staging, prod, etc.), we want something stronger:

• "t3.micro"

Here’s how we express that:

Terraform reads this the same way we would:

“If environment equals dev, use t2.micro.
Otherwise, use t3.micro.”

Now that we’re comfortable with this, let’s step into something that looks bigger at first i.e. dynamic blocks.
I promise, with the right example, they’ll feel just as friendly.

Dynamic Blocks

When we start writing Terraform, most of us follow the same path:
we create a resource, it works, and we feel good.
But as we add more features, we often notice something frustrating:

“Why am I writing the same kind of block again and again?”

We’re not doing anything wrong, this is how everyone starts.
But after a few rules, a thought naturally appears:

“There has to be a cleaner way.”

And that thought is exactly what leads us to dynamic blocks.

The Real-Life Problem

Imagine we’re setting up a security group.
We need:

• One rule for HTTP

• One rule for HTTPS

• Maybe more later

And each rule looks almost identical, only one or two values change.

So we end up writing something like:

And after a few blocks, our file starts looking like a scroll.

And the moment we add or remove a rule, we have to rewrite or reorganize everything.

This is where terraform helps, using dynamic blocks.

A Way to Look at Dynamic Blocks

Think of dynamic blocks like a reusable mould.

If we were making chocolates, we wouldn’t shape each piece with your hands.
We’d use one mould and pour different fillings into it.
The mould stays the same.
The number of chocolates changes.

Dynamic blocks are exactly that:

• The mould is the block structure

• The fillings are the values in your list

• The result is however many blocks Terraform needs

This analogy helps many beginners understand that we don’t need to create blocks ourselves, terraform will do it for us!

Setting Up the Input

Let’s say we have a variable called ingress_rules.

This is simply our list of configurations which we need, the ports, protocols, and descriptions we want.

We don’t have to understand everything in this block right now.
Just see it as “a collection of rules.”

Let’s not worry about the details yet.
The important part is:

We now have a list of rules that look similar.

The Dynamic Block

Here is Terraform’s “dynamic block” or “the mould” if we go by the above analogy:

Let’s break it down, one tiny piece at a time.

1️⃣ dynamic "ingress"

This is where we tell Terraform:

“I want you to create ingress blocks for me,
based on the list I give you.”

We’re not creating the blocks manually anymore. Terraform is.

2️⃣ for_each = var.ingress_rules

This is Terraform’s way of saying:

“I’ll repeat this block once for each rule in the list.”

If there are 2 rules → it creates 2 blocks
If there are 5 rules → it creates 5 blocks
If we add a new rule tomorrow → Terraform adds one more automatically

3️⃣ content { ... }

This is the mould we talked about.

We’re simply describing what one ingress rule looks like.

Terraform will take this “shape” and fill it with:

• the first rule’s values

• then the second rule’s values

• and so on

We don’t write the block repeatedly, Terraform clones it for us.

Why This Matters

Dynamic blocks are helpful because they let us:

• Avoid writing repetitive blocks

• Make our file cleaner and shorter

• Add/remove rules by just editing the list

• Trust Terraform to do the repetitive work

• Reduce mistakes from copy-pasting

And that’s the whole spirit behind IaC, write clean, reusable, predictable code.

Splat Expressions

By the time we start working with multiple resources in Terraform, something interesting begins to happen:
we suddenly find yourself surrounded by lists.

Maybe we created 3 S3 buckets.
Or a few subnets.
Or several EC2 instances.

And at some point, a very simple need comes up:

• “Can I get all of their names?”

• “Can I collect all their IDs?”

• “Is there an easy way to gather all the ARNs?”

If you’ve ever asked yourself these questions, you’re not alone.
Everyone who uses Terraform eventually reaches this moment i.e. a moment where getting one value is easy, but getting all of them feels like extra work.

Terraform understands this pain, and it gives us a helper that makes things surprisingly easy:

The Splat Expression (*)

The splat expression is Terraform’s way of saying:

“Tell me the attribute you want…
and I’ll collect it from every resource for you.”

No loops.
No complex logic.
No extra locals.
Just a simple star.

A Simple Example

Imagine we created three S3 buckets:

Terraform now sees them like this:

Now imagine we want all their bucket names.

Without any shortcut, we’d probably think:

“Hmmm… do I need a loop?
Or a local variable?
Or do I write something long?”

Terraform’s answer is: “No… just use the star *”

And instantly, we get:

Using Splat Expressions With for_each

Now, suppose we created subnets using for_each:

If we now want all subnet IDs, we don’t have to write anything fancy.

Just use the splat:

Terraform gathers them and returns:

When Should You Use Splat Expressions?

They shine the most when we:

• Create multiple resources

• Want one specific attribute from all of them

• Prefer cleaner, shorter code

• Don’t want to create unnecessary loops or locals

• Want our Terraform files to stay simple and readable

Conclusion

Yaay! We’ve made it to the end of Day 10!

Today, we played with some of Terraform’s little helpers that make our life easier:

• Conditional expressions: so Terraform can decide the right value for us.

• Dynamic blocks: so we don’t have to rewrite the same structure over and over.

• Splat expressions: so we can collect multiple values in one simple line.

Each of these might seem small, but together they turn repetitive, boring code into something clean, readable, and intelligent. They’re the building blocks that help us think like Terraform, not just about Terraform.

I know sometimes it feels like there’s too much to remember, or we’re worried about making mistakes. That’s completely normal. Every learner feels that. The important thing is, we’re trying, experimenting, and actually building understanding step by step. That’s huge.

If you still have doubts or want a little visual walkthrough, here’s a video by Piyush Sachdeva that complements today’s blog beautifully and might make things click faster:

Look back at what we’ve done so far: 10 days of consistent learning!
And ahead? There’s so much more to explore, more Terraform, more ways to simplify our infrastructure.

Let’s keep going. We’ve got this.

Hello and welcome back!

We’re now on Day Nine of our 30 Days of AWS Terraform journey.

Over the past eight days, we’ve been building our understanding of Terraform step by step, from creating resources to seeing how Terraform keeps track of everything behind the scenes. Each day has added a new layer, and if you’ve been following along, you’ve already built a solid foundation.

Today, we’re taking a slightly deeper step. Our focus is on Terraform Lifecycle Rules.

Now, lifecycle rules may sound a little technical at first, but at their heart, they simply help Terraform decide how a resource should behave when it’s created, updated, or destroyed. Think of them as instructions we give Terraform so that our resources are handled safely, predictably, and without us turning into Joey!

In previous sessions, we’ve learned how Terraform provisions resources and manages their state. Today, we trace that thread forward:
How do we protect those resources? How do we avoid accidental deletions or unexpected downtime? How do we ensure updates happen the way we want?

That’s where lifecycle rules come in.

We’ll start with the basics, understand why they matter, and then explore each rule one by one, gently and clearly, just like we’ve done in earlier blogs.

Here’s a fact about me before we dive in:

Fact #9: I’m a big fan of horror and thriller TV shows, but somehow “The Office” landed in my top three TV shows!

What Are Terraform Lifecycle Rules?

Before we start looking at the lifecycle rules, let’s take a moment to understand what they really do.

Imagine we are using Terraform to manage AWS. We have an EC2 instance running our application, or an S3 bucket storing important files. Normally, Terraform knows what to do when we change something i.e. it will create, update, or delete resources. But sometimes, we need to tell it exactly how we want it to behave.

For example:

• Suppose we want to update the AMI of our EC2 instance. Terraform will need to create a new instance and remove the old one. But if it deletes the old one first, our app will go down.

• Or imagine we have an S3 bucket storing backups. We don’t want it to be deleted by accident.

• Maybe someone changed the desired capacity of our Auto Scaling Group outside of Terraform, and we don’t want Terraform to overwrite that change.

These are situations where lifecycle rules help us guide Terraform.

By using lifecycle rules, we can:

• Make updates safely with minimal downtime.

• Protect important AWS resources from accidental deletion.

• Respect changes made outside of Terraform when appropriate.

We’ll explore these rules one at a time, using real AWS examples, so we can see exactly how they work.

Let’s start slowly with the first rule: create_before_destroy, which helps ensure that updating an EC2 instance happens safely, without downtime.

Lifecycle Rule 1: create_before_destroy

Let’s begin with one of the most practical and meaningful lifecycle rules:
create_before_destroy

Now, at first glance, the name may look a bit long, but its purpose is beautifully straightforward:

Terraform should create the new resource before destroying the old one.

Why does this matter?

Imagine we have an EC2 instance running our application. One day, we decide to update it, maybe we want to use a new AMI. Terraform looks at this change and realizes it must recreate the resource.
But how should it do that?

• Should it destroy the old instance first and then create the new one?

• Or should it create the new instance first and only then remove the old one?

Well, if Terraform destroys the existing instance first, our application will go down until the new instance comes up. That’s downtime and we want to avoid that as much as we can.

And that’s exactly where create_before_destroy helps.

Adding It to Our Resource

Here’s the EC2 resource we’re working with:

Now, let’s add the lifecycle rule:

With this set to true, Terraform makes sure:

It’ll prepare our new instance first.
Only after it’s safely created, I’tll remove the old one.”

This reduces downtime and keeps things smooth.

What Happens If We Set it to False?

If we change it to:

create_before_destroy = false

Then Terraform behaves differently:

• It destroys the current resource immediately.

• Only then does it try to create the new one.

And here’s where things can go wrong.

Let’s say our new AMI isn’t authorized or there’s a misconfiguration. Terraform destroys the existing instance first, tries to create the new one… and fails.
Now our application is down, and nothing is running, all because the order got reversed.

This is why, in most real-world situations, create_before_destroy is kept true. It’s safer, more predictable, and much more friendly to applications that need high availability.ecycle Rule — prevent_destroy, written in the same warm, slow-paced, beginner-friendly style.

Lifecycle Rule 2: prevent_destroy

Now that we’ve seen how create_before_destroy can protect our resources from downtime, let’s move on to another very important lifecycle rule: prevent_destroy.

At its heart, prevent_destroy is about safeguarding resources from accidental deletion.

Imagine we have an S3 bucket that stores critical data. What if someone accidentally runs a terraform destroy or makes a change that would remove the bucket? That could be disastrous.

By setting prevent_destroy = true, we’re telling Terraform:

“No matter what, don’t destroy this resource unless I explicitly allow it.”

It’s like putting a safety lock on your resource. Even if a plan includes a destructive change, Terraform will block it and warn us first.

How to Use prevent_destroy

Here’s how it looks in your EC2 example:

Now, if someone tries to destroy the instance:

Terraform will first refresh the state and then throw an error, saying the resource cannot be destroyed.

Even changes that normally force a recreation (like changing the AMI) are blocked if prevent_destroy is enabled. To allow destruction, you would need to either set it to false or narrow the scope of the rule.

Why It Matters

Using prevent_destroy is a step that can save us from big mistakes. It ensures that important resources like databases, S3 buckets, or production EC2 instances are protected from accidental removal.

It’s a reminder that while Terraform gives us power to manage infrastructure, we also need safe guards to protect what matters most.dly style.

Lifecycle Rule 3: ignore_changes

We’ve seen how create_before_destroy helps us avoid downtime, and prevent_destroy keeps our important resources safe from accidental deletions.

Now, let’s take a step forward to ignore_changes, a rule that helps Terraform coexist peacefully with changes made outside of it.

At its heart, ignore_changes says:

“If certain properties of this resource change outside of Terraform, that’s okay. I won’t try to overwrite them.”

This is particularly useful in real-world environments, where resources aren’t always managed only by Terraform. Sometimes, changes happen manually, through scripts, or by other tools. Without ignore_changes, Terraform would notice the difference and try to “correct” it, even if the change was intentional.

An Example

Let’s imagine a small scenario:

We have an Auto Scaling Group (ASG) in AWS. Terraform is managing it, and the desired number of instances is set to 2 in our configuration:

One day, someone manually changes the desired capacity to 3 in the AWS console, maybe to handle extra load temporarily.

Without ignore_changes, if we run terraform apply, Terraform would notice that the desired capacity is 3 in the console, but 2 in your code, and would immediately try to change it back to 2.

This could be confusing, especially if manual scaling was needed for a short time. It might also cause unnecessary instance creation or termination, disrupting your application.

How ignore_changes Helps

By adding ignore_changes, you tell Terraform:

Now:

• Terraform will no longer touch the desired capacity if it changes externally.

• Other properties of the ASG are still fully managed by Terraform.

• Manual or external adjustments are respected, avoiding conflicts and unnecessary resource churn.

In other words, Terraform now trusts your intentional manual changes, while still keeping control of everything else.

Why This Matters

This small rule keeps harmony between:

• Terraform-managed configurations

• Real-world operational adjustments

Without it, Terraform might constantly fight against legitimate external changes, causing confusion, repeated updates, or even disruptions in your infrastructure.

With ignore_changes, our resources remain flexible, predictable, and safer to manage, while still respecting Terraform’s authority where it matters most.

Lifecycle Rule 4: replace_triggered_by

So far, we’ve seen how lifecycle rules can protect resources from downtime, accidental deletion, or external changes. Now, let’s explore a rule that helps resources stay in sync with related resources: replace_triggered_by.

Think of it like this:

Imagine we have an S3 bucket that stores logs, and we also have a logging configuration resource that sends logs to this bucket.
If the bucket changes, maybe we rename it or update its properties, the logging configuration might also need to be updated or recreated to keep sending logs correctly.

In Terraform, replace_triggered_by ensures that when one resource changes, dependent resources are automatically replaced, so everything continues to work smoothly.

AWS Example: S3 Bucket and Logging

Let’s say we have:

• An S3 bucket called log_bucket

• A bucket logging configuration called log_config that sends logs to log_bucket

If the S3 bucket changes (for example, the name or key policy is updated), you want the logging configuration to be recreated automatically so it still points to the correct bucket.

Here’s how we define it in Terraform:

The S3 bucket:

• resource "aws_s3_bucket" "log_bucket": This defines a new S3 bucket resource in Terraform named log_bucket.

• bucket = "my-app-logs": This is the actual name of the bucket in AWS.

• acl = "private": This sets the bucket’s access control to private, meaning only authorized users can access it.

In simple terms: This creates a bucket to store your application logs safely.

The S3 Bucket Logging Configuration

• resource "aws_s3_bucket_logging" "log_config": This defines a logging configuration resource. It tells AWS where logs should go.

• bucket = aws_s3_bucket.log_bucket.id: This is the bucket that will receive the logs (our log_bucket).

• target_bucket = aws_s3_bucket.log_bucket.id: This is the bucket where logs are stored — in this case, the same bucket.

• target_prefix = "logs/": All logs will be stored under the logs/ folder in the bucket.

In simple terms: This says, “Take the logs and put them into my-app-logs/logs/.”

The Lifecycle Rule: replace_triggered_by

• lifecycle { ... }: This block customizes how Terraform handles changes to the resource.

• replace_triggered_by = [aws_s3_bucket.log_bucket.id]: This tells Terraform:

“If the log_bucket changes in any way (like its name, ACL, or other important properties), automatically replace the logging configuration to keep it working properly.”

In simple terms: If the bucket changes, Terraform will recreate the logging setup automatically so logs keep flowing correctly. You don’t have to manually fix it.

Why It Matters

This lifecycle rule is helpful when resources are dependent on each other. It ensures:

• Dependent resources are updated automatically

• No broken connections between resources

• Your infrastructure remains consistent and predictable

Lifecycle Rules 5 & 6: Preconditions and Postconditions

We’ve already seen how lifecycle rules can protect resources from downtime, prevent accidental deletions, and handle external changes.

Now, let’s take a step into something a little different i.e. preconditions and postconditions.

Think of them as guides for our infrastructure. They don’t change how resources are created, updated, or destroyed, but they make sure things happen the way they should.

• Preconditions check before creating a resource: “Is everything okay to proceed?”

• Postconditions check after a resource is created: “Did everything turn out as expected?”

Preconditions

A precondition is like a pause before Terraform starts building a resource. It asks:

“Does this meet the rules we expect? If not, stop.”

Imagine this scenario: we are creating an EC2 instance, but our company only allows resources in the US East (N. Virginia) region.

Without a precondition, Terraform will happily create the instance anywhere, and we might break organizational rules.

With a precondition, we can tell Terraform to pause and warn us if the region is not allowed:

How It Works

• The condition checks if the EC2 instance’s availability_zone is in the list ["us-east-1", "us-west-2"].

• If the condition is true, Terraform continues and creates the resource.

• If the condition is false, Terraform stops and shows the error message.

Postconditions

A postcondition works after Terraform has created a resource. It’s like a double-check:

“Did it create the resource correctly? Are all the rules followed?”

For example, imagine we’re creating an S3 bucket for storing important logs, and your organization requires every bucket to have a ‘compliance’ tag.

We can add a postcondition to make sure this tag exists:

• If the bucket has the tag, all is good.

• If the tag is missing, Terraform stops and alerts you, giving you a chance to fix it.

Why They Matter

Preconditions and postconditions add peace of mind.

• Preconditions prevent mistakes before resources are created.

• Postconditions verify compliance and correctness after creation.

Together, they help you catch errors early, enforce rules automatically, and make Terraform safer and more predictable, all without adding stress or complexity.

Conclusion

Add with that, we’ve explored Terraform lifecycle rules together today!

From basics to understanding lifecycle rules that make our infrastructure safer and smarter, we’ve come a long way! And there’s still so much more to explore!

Each of these rules thatw e learned today might seem small on its own, but together, they give us control, confidence, and safety in managing your infrastructure.

I know learning all of this can feel overwhelming at times, and that’s perfectly okay.

If you still have doubts or want to see these lifecycle rules in action, here’s a helpful video by Piyush Sachdeva that walks through them clearly:

Keep going, stay curious, and I’ll see you tomorrow for the next exciting step in our Terraform journey.

Hello and welcome back! I hope you’re doing well today.

Can you believe it’s already 1st December? Time is flying, and here we are at Day Eight of our 30 Days of AWS Terraform journey. I’m genuinely happy you’ve taken the time to join me today.

Over the past week, we’ve been building our Terraform foundation, step by step. I know it can feel a little tricky at times and that’s completely normal! What matters most is that we’re comfortable, following along, and gradually gaining confidence with each concept.

Yesterday, we explored Terraform type constraints. We saw how lists, sets, and maps behave differently, and why that matters when we’re defining resources. It might have seemed a bit technical, but it’s going to be very useful for today’s topic, because meta arguments often interact with these types in subtle ways.

Today, we’re diving into Meta Arguments.

We’ll focus on three meta arguments: depends_on, count, and for_each. Each one plays a unique role, and by the end of today, we’ll see how they all fit together to make our Terraform code more predictable and easier to work with.

And, just to keep things personal as I like to do in my blogs:

Fact #8: I’m not really fond of sweets, chocolate and pastries rarely tempt me. Give me a spicy, flavourful dish any day, and I’m in my happy place.

What are Arguments vs Meta Arguments

Before we jump into the specific meta arguments, let’s take a moment to understand the difference between regular arguments and meta arguments in Terraform.

When we create a resource in Terraform, like an AWS S3 bucket we provide arguments. These arguments describe the resource’s properties, such as its name, tags, or region. Terraform’s documentation lists all these arguments, showing which are required, which are optional, and what type of values they expect.

But Terraform also gives us something a little extra: meta arguments. These aren’t defined by AWS or any other provider, they are built into Terraform itself. Think of them as small helpers that make our lives easier. They let us add logic directly into our Terraform configurations without needing to write extra scripts.

Meta arguments help guide Terraform on how to manage resources. For example, they can tell Terraform in what order to create resources, how many copies of a resource to make, or how to loop over a collection of items. They don’t change what the resource is, they just help Terraform handle it more efficiently and cleanly.

In the sections ahead, we’ll focus on three of the most useful meta arguments:

1. depends_on – to manage resource dependencies

2. count – to create multiple instances of a resource

3. for_each – to iterate over lists, sets, or maps

Understanding depends_on

Before we jump into the code, let’s pause for a moment and understand why this meta argument even exists.

When we write a Terraform file with multiple resources i.e. maybe a VPC, a subnet, an EC2 instance, and a security group, Terraform doesn’t automatically know the order in which you want things to be created. If all our resources are in a single file, Terraform simply sees a list of resources. Unless there’s a direct relationship between them, it can’t guess the sequence correctly.

We might think Terraform reads files from top to bottom, but it doesn’t.

How Terraform Decides the Order

• If we have multiple files, Terraform loads them alphabetically.

• If we have multiple resources in a single file, Terraform does not follow the order we wrote them in.

Instead, Terraform tries to detect implicit dependencies, which happen automatically when one resource references another (like aws_vpc.main.id). But if it can’t detect a relationship, it will create resources in whatever order it finds most efficient.

But sometimes, we need to make sure something is created first. For example:

• We want a VPC ready before creating an EC2 instance.

• We want a security group available before attaching it.

• We want an S3 bucket ready before something else references it.

Terraform won’t know this unless we tell it explicitly.

What is depends_on?

The depends_on meta argument is Terraform’s way of letting you say:

“Terraform, please create Resource A before Resource B.”

This is called an explicit dependency.

• Implicit dependency happens automatically when a resource references another.

• Explicit dependency is when we manually declare the relationship.

Terraform will then wait for Resource A to be fully provisioned before creating Resource B. This gives you complete control over the sequence, avoiding unexpected errors.

Example

Let’s take two S3 buckets as an example:

• aws_s3_bucket.day_07_bucket

• aws_s3_bucket.day_08_bucket

If we want the Day 08 bucket to be created only after the Day 07 bucket, we write:

This tells Terraform:

1. Create day_07_bucket first.

2. Only after that, create anything that depends on it.

Even if Terraform thinks it could create them in parallel, the depends_on argument ensures it waits.

Why This Matters

Using depends_on gives you predictable behavior.

• No resources being created out of order.

• No debugging issues because something wasn’t ready yet.

It’s a simple meta argument, but it can save you a lot of headaches and bring a lot of clarity to your Terraform workflow.

Understanding count

Now that we’ve understood depends_on, let’s move on to one of the most commonly used meta arguments in Terraform: count.

Why do we need count?

Imagine we’re creating an S3 bucket. We write our resource block, give it a bucket name, maybe some tags, and Terraform creates one bucket for us.

That’s easy enough. But what happens if we need multiple buckets with the same configuration?

Without count, we would have to copy and paste the resource block multiple times, changing the bucket name each time. Not only is that tedious, but it also becomes harder to maintain as our infrastructure grows.

This is where count comes in.

It lets you tell Terraform:

“Please create this resource N number of times.”

Just one clean resource block that repeats itself automatically based on the number we provide.

Creating One Bucket

Before we dive into count, let’s start with a simple S3 bucket, something familiar:

Here’s a quick recap:

• aws_s3_bucket → resource type

• "day_08_bucket" → Terraform’s internal name

• bucket → the actual bucket name in AWS

• tags → values passed from a variable

If we run terraform init and terraform plan, Terraform will show that it plans to create one bucket.

What if we want multiple buckets?

Let’s say we have a list of bucket names and we want Terraform to create all of them using one resource block.

First, we define a variable to hold those names:

Now we have a list of strings, each representing a bucket name.

Using count to create multiple buckets

Here’s how we can tell Terraform to loop through the list:

Let’s take this slowly, step by step:

✔ count = length(var.bucket_names)

• This tells Terraform:

“Create one bucket for each element in this list.”

• If the list has 3 names, Terraform creates 3 buckets.

• If the list grows to 10 names tomorrow, Terraform creates 10 buckets automatically, no changes needed in the resource block.

✔ bucket = var.bucket_names[count.index]

• count.index starts at 0, then moves to 1, 2, and so on…

• Terraform picks bucket names one by one:

  • index 0 → "my-unique-bucket-9876"

  • index 1 → "my-unique-bucket-3112"

  • index 2 → "my-unique-bucket-43242"

This is how a single resource block becomes multiple buckets effortlessly.

When count works best

Count works beautifully with lists, because:

• Lists have a fixed order

• Each element has an index (0, 1, 2…)

But here’s an important question:

What if the variable is not a list?
What if it’s a set?

Sets do not have indexes, and that’s when count.index cannot be used directly. We’ll see how to handle that next.

Understanding for_each

Now that we’ve explored count, let’s talk about for_each, which is a little different but very powerful. I know this one can feel tricky at first, so we’ll take it step by step.

We might remember that count works perfectly when we have a list of items and we want Terraform to create multiple copies of the same resource. But count has limitations:

• It works only with lists.

• It doesn’t work with sets (because sets are unordered).

• It doesn’t give us easy access to key-value pairs, which we’ll often need with maps.

That’s where for_each comes in. Think of for_each as a helper that goes through each item individually and lets us reference that item when creating resources.

It works with lists, sets, and maps, giving us much more flexibility than count.

Example 1: Using for_each with a set of bucket names

Let’s start with a simple set of bucket names.

Here’s what’s happening:

1. for_each loops through each element in the set. Think of it like taking each bucket name from the set, one at a time.

2. each.key gives us the current element during the iteration. In sets and lists, each.key and each.value are the same, because there’s no separate key-value structure.

3. Terraform will create one bucket for each element in the set, without us having to write multiple resource blocks.

So, if our set has three names, Terraform will create three buckets, one for each name.

Example 2: Using for_each with a map

Sets are simple i.e. they only have values. But sometimes, we want both a name and some extra information, like a description. That’s where maps come in.

Step by step explanation:

1. for_each loops over each key-value pair in the map.

   • Key = bucket name (dev-bucket or prod-bucket)

   • Value = description (Development environment bucket or Production environment bucket)

2. each.key → gives the bucket name

3. each.value → gives the description

4. Terraform creates one bucket for each key-value pair and assigns the description as a tag

This is something count cannot do easily, because count can only iterate over lists and doesn’t give you keys and values.

Why for_each is useful

• It works with sets and maps, not just lists.

• We can access keys and values separately, which makes it great for more complex configurations.

• It allows us to avoid repeating resource blocks and everything happens automatically for each element.

Conclusion

Here we are, at the end of another day in our 30 Days of AWS Terraform journey.

Sometimes, when I reflect on my own learning, I remember feeling a bit overwhelmed when I first encountered meta arguments like count and for_each. At first, the ideas seemed abstract, and I wasn’t sure how they would fit into my real projects. But slowly, by making mistakes, and observing what happened, it all started to make sense.

That’s exactly why I take the pace slow in these blogs, so we can see how these concepts actually play out, without feeling rushed or pressured.

We’ve come a long way already, but there’s still a lot ahead. Each day builds on the previous ones, and the journey is cumulative. The concepts we’ve learned now will be the foundation for more advanced features, like lifecycle rules, conditional creation, and modules. We’re developing intuition for how Terraform thinks, and that intuition is what will make your infrastructure work reliably.

If any of this still feels a bit fuzzy, here’s a video by Piyush Sachdeva that might help clarify things further:

Let’s take a moment to appreciate our progress. We’ve tackled eight days of learning, experiments, and new ideas and that’s no small feat.

Let’s keep going, stay curious, and remember: this is a journey, and every step forward matters.

Happy learning, and I can’t wait to continue exploring the next chapters with you!

Hello and welcome back to our little corner of the Terraform journey!

I’m truly happy to see you here on Day Seven of our 30 Days of AWS Terraform adventure.

Every time we take a step into Terraform, open a new concept, and learn something at our own pace, we’re quietly building something meaningful. So, let’s take a moment to appreciate ourself. We’ve made it this far, and that consistency matters more than we might realize.

Yesterday, we explored Terraform file structure, understanding how to organize our resources and variables so that everything feels neat, clear, and manageable. That was all about giving our project a proper home i.e. a foundation for what comes next.

And today… we’re going to build on that foundation and step into a topic that is both powerful and essential: Terraform type constraints.

Type constraints might sound technical at first, but think of them as a guide that tell Terraform, “This variable should behave in this specific way.” They help prevent mistakes, keep our configurations predictable, and make our code easier to understand, especially as projects grow larger.

And, as always, a little personal tidbit to keep our journey connected:

Fact #7: I was naturally introverted during school and college, but somehow my close friends never saw me that way. They always made me feel included, confident and curious :P

Now let’s ease into it together. In the next section, we’ll explore what type constraints are, why they matter, and how they help Terraform understand the kind of values each variable can hold.

Primitive Type: Numbers

Alright, let’s begin with something simple but very important: primitive types. These are the simplest kinds of variables, and they form the foundation for everything else we’ll build. There are three main primitive types: string, number, and boolean. Today, we’ll start with numbers first.

A number is exactly what it sounds like i.e. a numeric value like 10, 0, or 500. It seems simple, but defining it correctly in Terraform is important, because it tells Terraform what kind of value to expect.

Let’s imagine a simple scenario: we’re creating virtual machines for a small project. Maybe we want to start with two servers for our application. Instead of writing that number directly inside our resource block, we can create a variable. This makes our code flexible and easy to adjust later.

Here’s how we could define the variable in variables.tf:

Notice a few things:

• type = number tells Terraform that this variable should always hold a numeric value.

• default = 2 means if we don’t provide any other value, Terraform will create two servers.

Now, in our main.tf file, we can use this variable instead of hardcoding the number:

If tomorrow we decide your application needs five servers instead of two, we don’t touch the resource block, we just change the variable’s value. Terraform handles the rest.

Numbers are simple, yes but using them thoughtfully sets the stage for everything else in Terraform.

Primitive Types: Strings

Now that we’ve spent some time with numbers, let’s move to another very common primitive type in Terraform: strings.

A string is simply text. Anything inside quotes i.e. "hello", "production", "ap-south-1" is considered a string. It’s one of the most frequently used types in Terraform because so much of our infrastructure depends on names, IDs, regions, and descriptions… all of which are text.

Let’s understand this with a simple example:

Imagine we’re setting up an environment for a small project, and we want to store the environment name somewhere. Maybe today it’s "development", but later it might change to "staging" or "production". Instead of hunting for this word everywhere in your files, we can define it once as a variable.

Here’s how we might define a string variable in your variables.tf file:

A few things to notice:

• The value is wrapped in quotes, because strings always represent text.

• Terraform now knows this variable should always hold a text value.

• If we don’t provide anything else, "development" will be used by default.

Then we can use this variable in our main.tf to name or tag our resources:

Here’s what’s happening:

• The bucket name uses the string variable to stay consistent.

• The tag also uses the same variable, which means when we change "development" to "production", our bucket names and tags update automatically.

• No more searching and replacing text across files and Terraform takes care of it.

Strings may feel simple, but they quietly hold together so many parts of our configuration. Getting comfortable with them makes the rest of our Terraform journey feel much smoother.

Absolutely — let’s slow this down, soften the tone, and make it feel more welcoming for someone who might be completely new. I’ll also switch to a simpler, more relatable example so it doesn’t feel technical right from the start.

Primitive Types: Booleans

Now that we’ve explored numbers and strings, let’s take a moment to talk about something even simpler: booleans.

Booleans are one of those things that look almost too small to matter i.e. they can only ever be true or false. That’s it. Just two possibilities.

And yet, these tiny values quietly shape so many of the decisions our Terraform configuration makes.

If strings are like words and numbers are like quantities, then booleans are like the “yes” and “no” signals that help Terraform understand what we want.

Think of a boolean like a simple switch.
It’s either on or off.

To make this even more relatable, imagine we're defining Should the EC2 instance have a public IP?

Let’s define that in Terraform:

Here, we’re simply telling Terraform:

“By default, please don’t give this EC2 instance a public IP.”

Later, in our Terraform configuration, you might use that value like this:

Now his tiny boolean has a real impact.
If someone sets it to true, Terraform will happily give the instance a public IP.
If it stays false, the instance stays private and quiet in the VPC.

Booleans don’t look powerful at first glance. But they help us toggle features on and off, shape the behavior of our infrastructure, and make our Terraform code feel more flexible and human.

They may feel tiny and almost too simple, but they enable many everyday decisions in Terraform. As we move forward to complex types, we’ll notice how booleans work beautifully alongside numbers, strings, lists, and maps to help shape real-world infrastructure.

Of course — thank you for telling me.
Let’s slow it down, make it gentler, more welcoming, and use a completely different example so it feels fresh and easier for newcomers.

Complex Types: List

So far, we’ve talked about primitive types. i.e. numbers, strings, and booleans. These are simple because they hold just one value at a time. But as we start working with real-world Terraform configurations, we’ll notice that many things come in groups. And when we want to store multiple values together in a clean, predictable way, Terraform gives us something called complex types.

Let’s begin with one of the most familiar complex types: the list.

A list is exactly what it sounds like i.e. a small collection of items arranged in a specific order. Think of it like writing down your goals for the week. Maybe we write:

• exercise

• learn Terraform

• cook something new

Each item is written one after another, and the order matters because it helps us keep track of what comes first and what follows.

Terraform treats lists the same way. All the items inside a list must be of the same type i.e. all strings, or all numbers, or all booleans.

To make this feel simple, let’s look at a gentle example.

Imagine we’re defining the names of different availability zones we want to use in AWS. Instead of typing each one separately throughout your code, we might gather them into one tidy list in variables.tf:

Notice how the values sit inside square brackets, that’s Terraform’s way of saying “these belong together as a list.”

Later, in our main.tf, we can pick items from this list by using their position:

Here, [0] means “give me the first item,” because Terraform starts counting from zero.

Lists are especially helpful when we have values that naturally belong together and need to appear in order. Whether it’s zones, instance names, or allowed IPs, a list keeps everything neat and predictable.

We’ll look at sets next, they feel a bit like lists at first glance, but they bring their own small personality with them, especially around uniqueness.

Complex Types: Sets

Let’s take our time with this one, because sets can feel a little unusual when we first meet them.

Now that we’ve spent some time with lists, let’s move to something that looks similar at first glance, but behaves a bit differently: sets.

A good way to think about sets is to imagine a small bowl on your table.

We can drop a few marbles into it, maybe a red one, a blue one, and a green one. But here’s the special rule:
no matter how many times we try, the bowl will never let you put two identical marbles inside.

So even if we accidentally attempt to drop in another red marble, the bowl quietly ignores it.

That’s exactly how sets work in Terraform.
They hold multiple values, just like lists, but they always make sure each value is unique.

Here’s a simple example.
Let’s say we’re storing the names of different environments our team uses, like “dev”, “stage”, and “prod”. You want to make sure someone doesn’t add “dev” twice by mistake. A set is perfect for that:

Even if someone changes the variable later and repeats a value, like adding “dev” again, Terraform will clean it up. That’s one of the nice little protections sets give you.

But there’s something important to remember, and I want to say it slowly so it sticks:

Sets don’t care about order.
They don’t remember which value came first, second, or third.
Because of that…
we cannot use indexing like var.environments[0].

If, for any reason, we truly need to grab a value using an index, we can convert the set into a list first:

Now locals.environment_list behaves like a list, and indexing works as expected.

So, in short:

• Use a list when order matters or when duplicates are allowed.

• Use a set when order doesn’t matter and we want to protect ourself from duplicates.

Take your time with this concept. Sets might feel new, but once we start using them, we’ll appreciate how they keep things clean and consistent in our Terraform projects.

Complex Types: Maps

Up until now, we’ve been working with things like lists and sets i.e. places where values are stored one after another. Maps are a little different, and honestly, a lot friendlier once we get used to them.

Instead of remembering positions like “first value, second value…”, maps let us store things as key–value pairs.

If you’re completely new to Terraform or even programming in general, don’t worry, maps are actually one of the easiest ideas to understand once we see them in action.

Think of a map like a small notebook where we write things down in pairs. On the left side we write what the thing is, and on the right side we write its value. Something like:

• Favorite_color → Blue

• Country → India

• Editor → VS Code

This “something = something else” style is what a map is all about. It’s just a clean way to store information where the name of the value matters.

Let’s look at a Terraform example:

Imagine we are creating an S3 bucket, and we want to add a few tags to describe it, things like what the bucket is for, who owns it, and whether it’s used for dev or prod.

Instead of creating three separate variables, we can store all of these as a map:

Here’s what each part means:

• purpose is the name of the tag, and "store-logs" is its value

• owner is the tag name, "arnab" is the value

• env tells us this bucket belongs to the dev environment

Everything stays neat and grouped together.

Now, in our AWS S3 bucket resource, you can simply pass the entire map:

Just like that, all our tags are applied and no need to write them one by one.
It keeps your Terraform file cleaner, and it also makes updates easier. For example, if the environment changes from dev to prod, we only update the value in one place.

Maps shine when you have settings that naturally belong together.

Complex Types: Tupple

A tuple is simply a way of storing multiple values of different types, all together, in a specific order.

Maybe the easiest way to picture it is this:

Imagine we’re packing a small lunchbox for the day.
Inside it, we might have:

• a sandwich (text)

• a bottle of water (number for quantity or size)

• and a note someone wrote for you (text again)

Each item is different.
But they all belong together, in a certain order.
If we change the order or swap an item for something unexpected, the lunchbox suddenly doesn’t make sense the same way.

That’s exactly how tuples behave.

Here’s a simple Terraform example using a tuple.
Let’s say we want to describe a person in a very minimal way: their age, their name, and whether they like coffee.

Let’s pause here and break it down:

• The first item must be a number (the person’s age).

• The second item must be a string (their name).

• The third item must be a boolean (true or false about coffee).

If we change the order, Terraform will get confused, just like if you put the note at the bottom of your lunchbox and the sandwich on top of it.
The order matters because each position has a meaning.

Later, if we wanted to use these values inside our main.tf, it might look like this:

Now Terraform clearly knows:

• var.person_info[0] → 28

• var.person_info[1] → "Arnab"

• var.person_info[2] → true

Tuples shine in moments where we want to group different kinds of information together, not too many, just a handful and each item has a clear place and purpose.

Complex Types: Objects

We’ve reached the final type in our Terraform variables journey, and honestly… this one deserves a little smile.

Objects are powerful, yes but they don’t always have to feel serious or technical.
To make this easier to understand, let’s step away from servers, regions, and monitoring for a moment.

Imagine we’re standing at our favourite café.

We want to order a coffee.
Now, a coffee isn’t just one thing, it has several parts:

• the type (latte, cappuccino, espresso)

• the size (small, medium, large)

• whether we want it hot or iced

All these parts are different types of information…but they belong together as one coffee order.

That’s exactly what an object is in Terraform.

Breaking it down slowly:

• project → string (like the coffee type)

• environment → string (like the size: small/medium/large)

• owner → string (who the coffee is for)

• backup → boolean (like “iced or hot”)

Terraform will now know exactly what to expect for each field.
If a value is the wrong type, Terraform will alert us, keeping our infrastructure safe and predictable.

We can now use this object to tag our EC2 instance in AWS:

See how neat and organized that is? All the metadata is grouped together in one object, just like all parts of our coffee order are grouped in one cup.

This approach keeps your Terraform configuration clean, readable, and easy to maintain, while also helping you manage multiple values together in a single, logical package.

Conclusion

Wow… we’ve come a long way today!

I know this blog might feel a bit long, and that’s completely understandable as there were quite a few concepts to cover. Terraform type constraints are one of those ideas that take a little patience to fully grasp, but the effort is worth it.

Understanding type constraints is important because it gives structure and safety to our Terraform code. It helps prevent mistakes, keeps your configurations neat, and makes it easier to manage as your projects grow. The more comfortable we get with these types, the more confident we’ll feel building real-world infrastructure.

If something still feels a little unclear, here’s a helpful video by Piyush Sachdeva that explains these concepts in another way, which might help clear any remaining doubts.

Take a moment to appreciate how far we’ve come. Seven days in, and we’re already building real infrastructure in AWS with Terraform. There’s still a lot ahead, but each day, each example is helping us grow into a more confident and capable Terraform practitioner.

Let’s keep going! Tomorrow, we’ll continue our exploration, and I promise it will feel even more natural as we build on what we’ve learned so far.