Low-Latency Networking at Scale: When Every Hop Matters

There is a point where “low latency” stops meaning that the API feels fast.

It starts meaning that an extra network hop matters.

A cross-AZ transfer matters.

DNS behavior matters.

Where TLS terminates matters.

And sometimes the architecture that looks more highly available on a diagram is actually the architecture introducing unnecessary latency into every request.

So imagine a company operating a real-time B2B platform.

It is not some theoretical company processing a gazillion transactions per second.

It has a much more normal enterprise problem.

Customers have systems in New York, London and Frankfurt. They continuously send small requests to the platform — pricing requests, order validations, inventory reservations, risk checks, whatever the business happens to be.

The actual business operation might take 10–20 ms.

That means adding another 40 ms of network latency is not a small implementation detail anymore. It is most of the response time.

At the same time, this is an enterprise environment.

We have:

• production workloads
• internal services
• multiple AWS accounts
• private databases
• an on-premises environment
• security inspection
• development environments
• disaster recovery
• customers that sometimes require fixed IP allowlists

Now the networking becomes interesting.

The first version works

We start with one Region.

There is nothing particularly wrong with this.

For a normal web application, I would probably be completely happy with it.

But then the requirements say:

We whitelist IP addresses. Give us the IP addresses that your traffic will come through.

Now we have a problem.

An ALB does not give me a nice pair of static IP addresses that I can hand to the customer.

We could start engineering around that problem.

Or we can realize that this requirement has changed which networking component belongs at the edge.

The edge becomes an NLB

Now the architecture becomes:

The Network Load Balancer makes much more sense when I care about TCP-level behavior and static addresses.

But another customer appears in Europe.

And this is where I would avoid immediately reaching for Route 53 latency routing.

Route 53 can absolutely send a European user toward Europe and an American user toward the US.

But DNS resolution is not the same thing as optimizing the actual network path.

If this application really cares about latency, I want to think about where the client's traffic enters the AWS network.

That changes the architecture again.

Bring the user onto the AWS backbone earlier

I would put AWS Global Accelerator in front of the regional NLBs.

Now the customer gets two stable anycast IP addresses.

More importantly, traffic can enter the AWS global network relatively close to the customer instead of travelling across the public internet for the entire journey.

That distinction gets lost surprisingly often.

People sometimes think Route 53 latency routing and Global Accelerator are basically two ways of solving the same problem.

They are not.

For a normal website I might not care.

For a latency-sensitive TCP service with enterprise customers using fixed firewall rules, I care quite a lot.

Then we discover that the public side was the easy part

The application itself starts growing.

We now have:

Production account
Risk account
Data account
Shared services account
Security account
Development account
Analytics account

Each team creates VPCs.

Initially someone starts connecting them with VPC peering.

This works beautifully when there are three VPCs.

Then three becomes eight.

And the diagram starts looking like this:

This is the point where I would stop.

Not because VPC peering is bad.

Because I have accidentally made every new network another networking project.

Peering is also non-transitive.

If A ↔ B ↔ C, that does not mean A ↔ C.

So I would change the topology instead of adding more connections.

Create an actual network layer

I would create a dedicated networking account and put a Transit Gateway there.

This looks simpler.

But there is a dangerous mistake hiding here.

If I attach everything to one Transit Gateway and let everything propagate into the same route table, I have effectively created a giant routed network.

Development can potentially find production.

Analytics can potentially find systems it has no business talking to.

So the Transit Gateway is not the security architecture.

The route tables are.

I want the routes to describe trust

I would create different Transit Gateway route tables.

Production might know:

10.10.0.0/16   -> Production
10.50.0.0/16   -> Shared Services
172.16.0.0/12  -> On-prem

Development might know:

10.20.0.0/16   -> Development
10.50.0.0/16   -> Shared Services

And that is it.

There simply is no production route in the development routing domain.

That is an architecture I prefer.

Instead of saying:

Development technically has network connectivity to production, but don't worry because we configured 200 security groups correctly.

I can say:

Development doesn't have a route to production.

That is a much better starting position.

Now we hit the low-level problem

One of the application services becomes latency-sensitive.

It talks to another service in the same Region.

Someone decides that because we have three Availability Zones, every request should bounce through as many of them as possible for “high availability.”

We end up with something conceptually like:

Nothing here is technically absurd.

It might even survive failures very nicely.

But look at what we are doing to the normal request path.

We are repeatedly crossing Availability Zone boundaries.

For a normal enterprise application, that might be perfectly acceptable.

For this particular hot path, it might not be.

So I would start asking a different question:

Which failures am I protecting against, and am I forcing every successful request to pay for that protection?

That is where low-latency architecture becomes less about selecting AWS products and more about understanding traffic.

Maybe the application worker and its latency-sensitive dependency should prefer AZ-local communication.

Maybe I maintain capacity in every AZ but keep normal traffic local wherever possible.

Failure still matters.

But normal traffic does not need to tour the Region.

There is an important difference between:

Multi-AZ capability

and

randomly sending every packet across AZs because Multi-AZ sounds good.

NAT is another place where architecture quietly becomes expensive

Now suppose these application servers communicate heavily with S3.

The original route looks like this:

It works.

So nobody questions it.

But at enterprise scale, that traffic can become significant.

I am paying for a network component that I do not actually need in that path.

For S3 I can use a Gateway VPC Endpoint.

Now S3 traffic does not need the NAT Gateway.

I would do the same evaluation for every major flow.

Not:

How do I make my NAT architecture bigger?

But:

Why is this packet going through NAT at all?

That mentality matters enormously at scale.

For other AWS services I may use Interface Endpoints.

And if I own a service that needs to be exposed privately to other VPCs or customers, PrivateLink becomes even more interesting.

PrivateLink changes the problem completely

Imagine the pricing service lives in one VPC.

Twenty internal applications need to consume it.

One naive answer is:

Connect all twenty networks.

But maybe those applications do not need network connectivity to the entire pricing VPC.

They need one service.

That is a completely different requirement.

Instead:

Now I am exposing the service, not exposing the network.

That distinction is extremely powerful.

At enterprise scale, I want to reduce the number of networks that trust each other.

If application A only needs API B, giving A routing access to B's entire network is usually a much larger privilege than necessary.

Hybrid networking makes this even more important

We still have systems on-prem.

Initially the architecture uses a Site-to-Site VPN.

I would connect it to the Transit Gateway rather than creating independent VPN connections to every VPC.

Later the business needs more predictable bandwidth.

Now we add Direct Connect.

The important part is not simply that we upgraded VPN to Direct Connect.

The important part is that the VPC architecture barely changed.

That is what I mean by designing for scale.

A new connectivity method should plug into the network architecture.

It should not require a new architecture.

And I would still think about the failure path.

Direct Connect is not a magical cable that never fails.

I may keep Site-to-Site VPN available as a backup.

Now a physical connectivity problem does not necessarily mean a complete hybrid outage.

DNS can destroy an otherwise good failover design

Suppose we run:

US Region = primary
EU Region = standby

The database has a cross-Region replica.

The standby environment exists but normally runs minimal capacity.

Someone proposes:

Let's use weighted Route 53 records. Put US at 100 and Europe at 0.

That might be useful for certain deployment strategies.

But I would not confuse it with a proper active-passive failover design.

For DR, I care about health.

Normal condition:

Primary failure:

That is what a Route 53 failover policy is for.

But there is another uncomfortable detail.

DNS has caching.

Clients have resolvers.

Resolvers have TTLs.

So when someone tells me:

DNS failover means instantaneous failover.

I become suspicious.

The health decision might happen quickly.

That does not guarantee every client immediately forgets the DNS answer it already has.

For many systems that is perfectly acceptable.

For systems where connection recovery needs to be extremely fast, this is another reason something like Global Accelerator can become interesting.

Existing network connections and endpoint health are a different problem from DNS resolution.

Now the architecture looks different

At this point the design has layers.

But even that diagram hides one of the most important things.

Not every packet should traverse every layer.

A production service calling S3 should not travel:

Service → Transit Gateway → centralized NAT → internet path → S3

if:

Service → S3 Gateway Endpoint → S3

does the job.

A development workload does not need a production route just because both attach to the same Transit Gateway.

A consumer of one internal API does not necessarily need VPC peering with the service provider.

A European customer should not cross the Atlantic just because that was where we happened to deploy version one.

Once the architecture gets large, traffic locality becomes an architecture principle.

I would measure the network, not assume it

This is probably the part I would care about most in a real environment.

I would not optimize based entirely on what the AWS diagram looks like.

I want to know the actual path.

For a latency-sensitive request I would break the transaction down.

Now I have something useful.

If the business asks me to remove 10 ms, I can actually see where those 10 ms might exist.

Compare that with:

Maybe we should switch the database because databases are slow.

That is not engineering.

Maybe the database takes 3 ms.

Maybe 40 ms disappeared before the request even reached it.

I would look at:

• VPC Flow Logs
• application timing
• load balancer metrics
• connection establishment
• DNS resolution
• cross-AZ paths
• NAT usage
• packet loss
• retries
• TLS handshakes
• database timing

Especially retries.

A network that is normally 4 ms but occasionally loses a packet and triggers a retry can feel much worse than a stable 7 ms network.

Latency and consistency are not the same thing.

For some workloads I care more about the tail:

than I care about reducing p50 = 8 ms to p50 = 7 ms.

That p99 number is often where the interesting networking problem is hiding.

The architecture is not finished when the diagram looks clean

There are still uncomfortable questions I would ask.

What happens if one NAT Gateway disappears?

What happens if one AZ disappears?

What happens if Direct Connect disappears?

What happens if the Transit Gateway route table is changed incorrectly?

Can development resolve production DNS even if it cannot route there?

Can an internal service accidentally start sending terabytes through NAT?

Are we paying cross-AZ transfer because an application constantly talks to a dependency in another AZ?

Does the customer reconnect correctly when Global Accelerator changes endpoint health?

How long are clients caching DNS records during Route 53 failover?

Does the application create a new TLS connection for every request?

Do we have enough connection reuse?

These questions are much closer to the things that actually determine whether the architecture is good.

Conclusion

At small scale, networking is mostly connectivity.

Can A talk to B?

At enterprise scale, that question is nowhere near enough.

I want to know:

Should A be able to talk to B at all?

If yes:

What is the shortest sensible path between them?

Then:

What happens to that path when something fails?

And finally:

What does that path cost when we run it millions of times?

That is how I would approach low-latency networking at scale.

Not by throwing every networking service AWS has onto one architecture diagram.

Actually, almost the opposite.

I want fewer paths.

Fewer unnecessary relationships.

Fewer public hops.

Fewer cross-AZ trips.

Fewer things pretending to be highly available while quietly adding complexity to every request.

Transit Gateway gives me a scalable routing core.

Its route tables give me isolation.

PrivateLink lets me expose services without exposing entire networks.

VPC endpoints remove network paths that never needed to exist.

Direct Connect gives the hybrid environment a predictable path while VPN can provide another failure path.

Route 53 is excellent when the decision should happen at DNS.

Global Accelerator becomes interesting when the network connection itself is the thing I care about.

And the final architecture should have one property that I think is more important than looking impressive:

When the company creates VPC number 51, adds another customer in Frankfurt or deploys a third Region, it should feel like extending the architecture.

Not rebuilding it.

That is when I know the network was actually designed for scale.