# Troubleshooting Four AWS Networking Incidents

This article documents four networking incidents I resolved in the [Learn to Cloud networking lab](https://github.com/learntocloud/networking-lab), covering the investigation process, root causes, configuration changes, and verification steps.

The lab provides intentionally misconfigured AWS infrastructure, with issues involving private subnet internet access, internal DNS resolution, application connectivity, and network security rules.

I used the AWS CLI and Linux networking tools to investigate the deployed resources, identify the misconfigurations, apply fixes, and verify connectivity.

## 1\. Lab Environment

The infrastructure consisted of four EC2 instances deployed within an Amazon VPC:

| Component | Purpose |
| --- | --- |
| Bastion host | SSH access to the other instances |
| Web server | Public application entry point |
| API server | Private application service listening on TCP 8080 |
| Database server | PostgreSQL listening on TCP 5432 |

The application communication path was:

**Web server → API server → PostgreSQL database**

The environment also included:

- A NAT gateway providing outbound internet access for private instances.
- Amazon Route 53 with an `internal.test` private hosted zone for internal DNS.
- Security groups controlling instance-level inbound and outbound traffic.
- Network ACLs filtering traffic at subnet boundaries.

Terraform provisioned the infrastructure, while troubleshooting and configuration changes were performed against the deployed AWS resources using the AWS CLI.

The command examples below use variables instead of deployment-specific resource IDs and addresses.

## 2\. Incident 1: Private API Unable to Access the Internet

### Problem

The API instance was unable to establish outbound HTTPS connections. Requests to external services were timing out.

The instance was deployed in a private subnet and required outbound internet access through a NAT gateway.

### Investigation

I inspected the NAT gateway and the route table associated with the private subnet.

```bash
aws ec2 describe-nat-gateways \
  --nat-gateway-ids "$NAT_GATEWAY_ID"

aws ec2 describe-route-tables \
  --route-table-ids "$PRIVATE_ROUTE_TABLE_ID"
```

The NAT gateway existed in a public subnet, but the private subnet's route table did not contain a default route pointing to it.

### Root Cause

The private route table was missing the following route:

| Destination | Target |
| --- | --- |
| `0.0.0.0/0` | NAT gateway |

Without this route, internet-bound IPv4 traffic from the private subnet had no path through the NAT gateway.

### Resolution

I added the missing default route:

```bash
aws ec2 create-route \
  --route-table-id "$PRIVATE_ROUTE_TABLE_ID" \
  --destination-cidr-block 0.0.0.0/0 \
  --nat-gateway-id "$NAT_GATEWAY_ID"
```

### Verification

I repeated the HTTPS request from the API instance:

```bash
curl -4 --noproxy '*' -sS -o /dev/null \
  -w '%{http_code}\n' --max-time 10 https://example.com
```

**Result:** HTTP 200.

The API instance could establish outbound HTTPS connections through the NAT gateway while remaining in its private subnet.

Reference: [AWS VPC Route Table Documentation](https://docs.aws.amazon.com/vpc/latest/userguide/route-table-options.html)

## 3\. Incident 2: Internal DNS Resolution Failure

### Problem

The application could not resolve the following internal service names:

- `web.internal.test`
- `api.internal.test`
- `db.internal.test`

These hostnames were expected to resolve to the corresponding instances' private IPv4 addresses.

### Investigation

I inspected the Route 53 private hosted zone and its DNS records.

```bash
aws route53 get-hosted-zone \
  --id "$HOSTED_ZONE_ID"

aws route53 list-resource-record-sets \
  --hosted-zone-id "$HOSTED_ZONE_ID"
```

The investigation confirmed that:

- The `internal.test` private hosted zone existed.
- The hosted zone was associated with the lab VPC.
- The zone contained the default NS and SOA records.
- The required application A records were missing.

### Root Cause

The Route 53 private hosted zone did not contain A records mapping the application hostnames to their respective private IP addresses.

### Resolution

I created the missing A records using Route 53 UPSERT changes.

| DNS Record | Target |
| --- | --- |
| `web.internal.test` | Web server private IP |
| `api.internal.test` | API server private IP |
| `db.internal.test` | Database server private IP |

Each record was configured with a TTL of 60 seconds.

### Verification

I queried the VPC DNS resolver directly:

```bash
dig +short @169.254.169.253 web.internal.test A
dig +short @169.254.169.253 api.internal.test A
dig +short @169.254.169.253 db.internal.test A
```

All three queries returned their expected private IPv4 addresses.

I also checked DNS resolution through the instance's configured system resolver:

```bash
getent ahostsv4 api.internal.test
```

The command returned the API server's private address, confirming that internal name resolution was functioning.

Reference: [AWS Route 53 Private Hosted Zones](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zone-private-considerations.html)

## 4\. Incident 3: Application Connection Timeouts

### Problem

Two application connections were failing:

| Source | Destination | Port |
| --- | --- | --- |
| Web server | API server | TCP 8080 |
| API server | PostgreSQL | TCP 5432 |

The services were running, but remote connections between the instances were timing out.

### 4.1 Web Server to API Connectivity

#### Investigation

The API health endpoint responded successfully when accessed locally on the API instance.

However, requests from the web server could not reach the API.

I inspected the security-group rules controlling communication between the two instances.

The required permissions were:

| Security group | Direction | Traffic | Peer |
| --- | --- | --- | --- |
| Web | Outbound | TCP 8080 | API security group |
| API | Inbound | TCP 8080 | Web security group |

Security-group references have different meanings depending on rule direction. An inbound rule references an allowed source, while an outbound rule references an allowed destination.

#### Resolution

I corrected the application security-group rules to permit the required traffic from the web server to the API.

#### Verification

The API health endpoint was tested from the web server:

```bash
curl --noproxy '*' -fsS --max-time 5 \
  "http://${API_PRIVATE_IP}:8080/health"
```

The health check succeeded after the rules were corrected.

### 4.2 API Server to PostgreSQL Connectivity

#### Investigation

The API was also unable to establish a connection to PostgreSQL on TCP 5432.

An initial PostgreSQL readiness check returned no response.

I inspected the database service, listening sockets, host firewall, security groups, and subnet network ACL.

| Check | Finding |
| --- | --- |
| PostgreSQL service | Running |
| Listening socket | `0.0.0.0:5432` and `[::]:5432` |
| UFW host firewall | Inactive |
| Database security group | TCP 5432 permitted from API security group |
| Subnet network ACL | Inbound rule 100 explicitly denied TCP 5432 |

PostgreSQL was listening on the required port and was not restricted to the loopback interface.

The database security group also contained a rule permitting TCP 5432 from the API security group.

The subnet network ACL, however, contained an explicit deny rule for the same traffic.

### Root Cause

Network ACL rule 100 blocked inbound TCP 5432 traffic to the database subnet.

Unlike security groups, network ACLs are stateless and evaluate numbered rules in ascending order. An explicit matching deny prevents the traffic from reaching the instance, even when its security group permits the connection.

### Resolution

I removed the blocking network ACL rule and configured the database security group to permit PostgreSQL traffic specifically from the API security group.

Broad database access permissions were removed after the required application connection was restored.

### Verification

From the API instance, I tested TCP connectivity to the database:

```bash
nc -zvw3 "$DB_PRIVATE_IP" 5432
```

I also checked PostgreSQL readiness:

```bash
pg_isready -h "$DB_PRIVATE_IP" \
  -p 5432 -U labuser -d labdb -t 3
```

Both checks succeeded.

The required communication path between the web server, API, and PostgreSQL was restored.

Reference: [AWS Network ACL Documentation](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html)

## 5\. Incident 4: Restricting Network Access

### Problem

The lab infrastructure contained overly permissive network access rules.

The objective was to restrict access to the required sources and ports without interrupting application connectivity or administrative SSH access.

### Investigation

I reviewed the security-group permissions associated with:

- Administrative SSH access through the bastion host.
- SSH connections to private instances.
- Web server to API communication.
- API to PostgreSQL communication.
- ICMP access to the web server.

### Resolution

I configured the following access model:

| Destination | Traffic | Permitted source |
| --- | --- | --- |
| Bastion host | SSH, TCP 22 | My public IPv4 address (`/32`) |
| Web, API, and database | SSH, TCP 22 | Bastion security group |
| API server | TCP 8080 | Web security group |
| PostgreSQL | TCP 5432 | API security group |
| Web server | ICMP | Bastion security group |

The narrower rules were added and tested before the corresponding broad permissions were removed.

This sequence preserved the required connections while reducing unnecessary network access.

Security groups combine applicable allow rules. Adding a restrictive rule does not override an existing broader allow rule, so the unnecessary permissions also had to be removed.

### Verification

After updating the security groups, I reviewed the resulting permissions and tested the required connections.

The verification confirmed that:

- Administrative SSH access remained available through the bastion.
- Web-to-API traffic on TCP 8080 remained functional.
- API-to-database traffic on TCP 5432 remained functional.
- The intended network access restrictions were in place.

Reference: [AWS Security Group Rules](https://docs.aws.amazon.com/vpc/latest/userguide/security-group-rules.html)

## 6\. Summary of Findings

The four incidents involved different components of the AWS networking configuration.

| Incident | Root cause | Resolution |
| --- | --- | --- |
| Private API internet failure | Missing default route in private subnet route table | Added `0.0.0.0/0` route to NAT gateway |
| Internal DNS failure | Missing A records in Route 53 private hosted zone | Created records pointing to service private IPs |
| Application timeouts | Incorrect application traffic permissions and an explicit network ACL deny | Corrected security-group rules and removed blocking ACL rule |
| Overly permissive access | Security-group rules allowed unnecessary access | Restricted access to required sources and ports |

Each incident was investigated by identifying the affected source, destination, protocol, port, and relevant AWS networking components.

Configuration changes were followed by connectivity tests from the affected instances to verify that the original failures had been resolved.

The final configuration restored the required application communication paths and restricted network access to the intended sources.

---

**Project repository:** [Teedex1/networking-lab](https://github.com/Teedex1/networking-lab)

**Lab credit:** [Learn to Cloud Networking Lab](https://github.com/learntocloud/networking-lab). The project supplied the training infrastructure and incident scenarios. This article documents the troubleshooting and configuration changes performed in the deployed AWS environment.
