Skip to main content

Troubleshooting Four AWS Networking Incidents

Updated 9 min readMarkdown
Part of seriesLAB
Troubleshooting Four AWS Networking Incidents

This article documents four networking incidents I resolved in the Learn to Cloud networking lab, covering the investigation process, root causes, configuration changes, and verification steps.

The lab provides intentionally misconfigured AWS infrastructure, with issues involving private subnet internet access, internal DNS resolution, application connectivity, and network security rules.

I used the AWS CLI and Linux networking tools to investigate the deployed resources, identify the misconfigurations, apply fixes, and verify connectivity.

1. Lab Environment

The infrastructure consisted of four EC2 instances deployed within an Amazon VPC:

Component Purpose
Bastion host SSH access to the other instances
Web server Public application entry point
API server Private application service listening on TCP 8080
Database server PostgreSQL listening on TCP 5432

The application communication path was:

Web server → API server → PostgreSQL database

The environment also included:

  • A NAT gateway providing outbound internet access for private instances.
  • Amazon Route 53 with an internal.test private hosted zone for internal DNS.
  • Security groups controlling instance-level inbound and outbound traffic.
  • Network ACLs filtering traffic at subnet boundaries.

Terraform provisioned the infrastructure, while troubleshooting and configuration changes were performed against the deployed AWS resources using the AWS CLI.

The command examples below use variables instead of deployment-specific resource IDs and addresses.

2. Incident 1: Private API Unable to Access the Internet

Problem

The API instance was unable to establish outbound HTTPS connections. Requests to external services were timing out.

The instance was deployed in a private subnet and required outbound internet access through a NAT gateway.

Investigation

I inspected the NAT gateway and the route table associated with the private subnet.

aws ec2 describe-nat-gateways \
  --nat-gateway-ids "$NAT_GATEWAY_ID"

aws ec2 describe-route-tables \
  --route-table-ids "$PRIVATE_ROUTE_TABLE_ID"

The NAT gateway existed in a public subnet, but the private subnet's route table did not contain a default route pointing to it.

Root Cause

The private route table was missing the following route:

Destination Target
0.0.0.0/0 NAT gateway

Without this route, internet-bound IPv4 traffic from the private subnet had no path through the NAT gateway.

Resolution

I added the missing default route:

aws ec2 create-route \
  --route-table-id "$PRIVATE_ROUTE_TABLE_ID" \
  --destination-cidr-block 0.0.0.0/0 \
  --nat-gateway-id "$NAT_GATEWAY_ID"

Verification

I repeated the HTTPS request from the API instance:

curl -4 --noproxy '*' -sS -o /dev/null \
  -w '%{http_code}\n' --max-time 10 https://example.com

Result: HTTP 200.

The API instance could establish outbound HTTPS connections through the NAT gateway while remaining in its private subnet.

Reference: AWS VPC Route Table Documentation

3. Incident 2: Internal DNS Resolution Failure

Problem

The application could not resolve the following internal service names:

  • web.internal.test
  • api.internal.test
  • db.internal.test

These hostnames were expected to resolve to the corresponding instances' private IPv4 addresses.

Investigation

I inspected the Route 53 private hosted zone and its DNS records.

aws route53 get-hosted-zone \
  --id "$HOSTED_ZONE_ID"

aws route53 list-resource-record-sets \
  --hosted-zone-id "$HOSTED_ZONE_ID"

The investigation confirmed that:

  • The internal.test private hosted zone existed.
  • The hosted zone was associated with the lab VPC.
  • The zone contained the default NS and SOA records.
  • The required application A records were missing.

Root Cause

The Route 53 private hosted zone did not contain A records mapping the application hostnames to their respective private IP addresses.

Resolution

I created the missing A records using Route 53 UPSERT changes.

DNS Record Target
web.internal.test Web server private IP
api.internal.test API server private IP
db.internal.test Database server private IP

Each record was configured with a TTL of 60 seconds.

Verification

I queried the VPC DNS resolver directly:

dig +short @169.254.169.253 web.internal.test A
dig +short @169.254.169.253 api.internal.test A
dig +short @169.254.169.253 db.internal.test A

All three queries returned their expected private IPv4 addresses.

I also checked DNS resolution through the instance's configured system resolver:

getent ahostsv4 api.internal.test

The command returned the API server's private address, confirming that internal name resolution was functioning.

Reference: AWS Route 53 Private Hosted Zones

4. Incident 3: Application Connection Timeouts

Problem

Two application connections were failing:

Source Destination Port
Web server API server TCP 8080
API server PostgreSQL TCP 5432

The services were running, but remote connections between the instances were timing out.

4.1 Web Server to API Connectivity

Investigation

The API health endpoint responded successfully when accessed locally on the API instance.

However, requests from the web server could not reach the API.

I inspected the security-group rules controlling communication between the two instances.

The required permissions were:

Security group Direction Traffic Peer
Web Outbound TCP 8080 API security group
API Inbound TCP 8080 Web security group

Security-group references have different meanings depending on rule direction. An inbound rule references an allowed source, while an outbound rule references an allowed destination.

Resolution

I corrected the application security-group rules to permit the required traffic from the web server to the API.

Verification

The API health endpoint was tested from the web server:

curl --noproxy '*' -fsS --max-time 5 \
  "http://${API_PRIVATE_IP}:8080/health"

The health check succeeded after the rules were corrected.

4.2 API Server to PostgreSQL Connectivity

Investigation

The API was also unable to establish a connection to PostgreSQL on TCP 5432.

An initial PostgreSQL readiness check returned no response.

I inspected the database service, listening sockets, host firewall, security groups, and subnet network ACL.

Check Finding
PostgreSQL service Running
Listening socket 0.0.0.0:5432 and [::]:5432
UFW host firewall Inactive
Database security group TCP 5432 permitted from API security group
Subnet network ACL Inbound rule 100 explicitly denied TCP 5432

PostgreSQL was listening on the required port and was not restricted to the loopback interface.

The database security group also contained a rule permitting TCP 5432 from the API security group.

The subnet network ACL, however, contained an explicit deny rule for the same traffic.

Root Cause

Network ACL rule 100 blocked inbound TCP 5432 traffic to the database subnet.

Unlike security groups, network ACLs are stateless and evaluate numbered rules in ascending order. An explicit matching deny prevents the traffic from reaching the instance, even when its security group permits the connection.

Resolution

I removed the blocking network ACL rule and configured the database security group to permit PostgreSQL traffic specifically from the API security group.

Broad database access permissions were removed after the required application connection was restored.

Verification

From the API instance, I tested TCP connectivity to the database:

nc -zvw3 "$DB_PRIVATE_IP" 5432

I also checked PostgreSQL readiness:

pg_isready -h "$DB_PRIVATE_IP" \
  -p 5432 -U labuser -d labdb -t 3

Both checks succeeded.

The required communication path between the web server, API, and PostgreSQL was restored.

Reference: AWS Network ACL Documentation

5. Incident 4: Restricting Network Access

Problem

The lab infrastructure contained overly permissive network access rules.

The objective was to restrict access to the required sources and ports without interrupting application connectivity or administrative SSH access.

Investigation

I reviewed the security-group permissions associated with:

  • Administrative SSH access through the bastion host.
  • SSH connections to private instances.
  • Web server to API communication.
  • API to PostgreSQL communication.
  • ICMP access to the web server.

Resolution

I configured the following access model:

Destination Traffic Permitted source
Bastion host SSH, TCP 22 My public IPv4 address (/32)
Web, API, and database SSH, TCP 22 Bastion security group
API server TCP 8080 Web security group
PostgreSQL TCP 5432 API security group
Web server ICMP Bastion security group

The narrower rules were added and tested before the corresponding broad permissions were removed.

This sequence preserved the required connections while reducing unnecessary network access.

Security groups combine applicable allow rules. Adding a restrictive rule does not override an existing broader allow rule, so the unnecessary permissions also had to be removed.

Verification

After updating the security groups, I reviewed the resulting permissions and tested the required connections.

The verification confirmed that:

  • Administrative SSH access remained available through the bastion.
  • Web-to-API traffic on TCP 8080 remained functional.
  • API-to-database traffic on TCP 5432 remained functional.
  • The intended network access restrictions were in place.

Reference: AWS Security Group Rules

6. Summary of Findings

The four incidents involved different components of the AWS networking configuration.

Incident Root cause Resolution
Private API internet failure Missing default route in private subnet route table Added 0.0.0.0/0 route to NAT gateway
Internal DNS failure Missing A records in Route 53 private hosted zone Created records pointing to service private IPs
Application timeouts Incorrect application traffic permissions and an explicit network ACL deny Corrected security-group rules and removed blocking ACL rule
Overly permissive access Security-group rules allowed unnecessary access Restricted access to required sources and ports

Each incident was investigated by identifying the affected source, destination, protocol, port, and relevant AWS networking components.

Configuration changes were followed by connectivity tests from the affected instances to verify that the original failures had been resolved.

The final configuration restored the required application communication paths and restricted network access to the intended sources.


Project repository: Teedex1/networking-lab

Lab credit: Learn to Cloud Networking Lab. The project supplied the training infrastructure and incident scenarios. This article documents the troubleshooting and configuration changes performed in the deployed AWS environment.

About the author

  • Building in Cloud Engineering. Sharing practical projects and structured technical write-ups on Linux, networking, and DevOps.

No comments yet. Be the first to comment.

LAB

Part 1 of 1

Lab Experiments