Imagine the architecture below - very quickly it becomes a time and money sink that scales up uncontrollably.

In this article we will be diving exactly into that pattern - and how at a scale of a million calls per day, optimising inefficiencies allowed us to save years of compute time daily, and close to $2 million/year across our development and production environments combined.

The system

The system being optimised was responsible for modelling network behaviour and comparing network state before and after changes to physical entities (such as network routers).

The results of the calculations were critical to assert that changing the physical network of millions of interlinked devices would not cause network impact such as bottlenecks, degradation or packet loss for AWS customers.

Moreover, due to how mission critical this system’s results were to operations, the faster an answer could be given, the better.

The scale

As previously mentioned, this system received about 1 million calls/day - with most of those calls being replicated automatically to the development environment for pre-release validation. So in total, about 2 million calls per day.

As for calculation times and SLAs… it depends. Ask any network engineer and they will tell you about the seemingly endless ways to connect a physical network and make them behave in their own funky little ways. Hence modelling a network of millions of physical devices, with different protocols, physical interfaces, lags, seemingly all shapes and sizes - and it becomes tough to have a uniform calculation across them all.

For simplicity sake, imagine this system either replied in 5-10 seconds, or it would take nearly all of its 180-second SLA to finish a calculation.

Performance bottlenecks in a distributed world

Our synchronous Lambda fan-out pattern introduced two immediate structural flaws:

First, the parent Lambda idles until every single child invocation completes. This is the structural equivalent of grabbing a wad of cash and setting it on fire.

Second, your entire response time is hostage to the child Lambda’s p100. In an event-driven architecture, you can progress sequentially as individual chunks complete. With a synchronous implementation, Step B cannot start until the absolute entirety of Step A finishes, Step C is dependent on the p100 of Step B, and so on.

In systems design, this behaviour is known as the straggler problem. We can visualise it as such:

So, why?

While I wasn’t there for its inception, it is easy to understand the original engineering decisions.

The entire codebase was built in Python to run heavily CPU-bound tasks. Because of Python’s single-threaded nature, the team relied on architectural task-splitting (the Lambda fan-out) rather than in-process parallelism.

The rest comes down to velocity: the fastest way to build a new system is the way you already know. Why Python? Because the team knew it. Why a synchronous fan-out? Because it was operationally straightforward to reason about and easy to scale incrementally under delivery pressure.

When you are facing tight deadlines and intense stakeholder pressure, optimise-for-speed decisions make sense. You can always debate language and architecture choices, but ultimately this system has been operational for over half a decade now.

Optimising without rearchitecting

The entire optimisation effort was triggered by customer pain from transient timeouts. Because we needed immediate relief, a massive, high-risk architecture refactor was off the table. Our sole intention was to find tactical wins that would pull our worst-case tail latency safely under the 180-second SLA.

By targeting the hot paths, we managed to achieve a rock-solid, stable execution ceiling while simultaneously shaving 70% off our daily Lambda spend.

To understand how, you have to look at the math of our synchronous fan-out pattern. When you rely on synchronous layers, your end-to-end performance is entirely held hostage by your lowest, slowest leaf node. If a Lambda three levels deep chokes, it drags the entire parent infrastructure down with it.

At our scale, - 1 million daily top level Lambda invocations - getting dragged down by the Lambda 3 levels deep isn’t just a possibility, it’s a mathematical guarantee when your invocation ceiling is 640B invocations/day.

Targeting the multiplier effect

Because an outlier at deeply nested Lambdas is mathematically guaranteed to drag the entire system down, our primary target was flattening the tail latency (p100).

First, we employed simple profiling techniques to identify the hot execution paths. Then came our silver bullet: asking an AI agent to analyse those hot paths specifically and calculate their time complexity. This process flagged multiple algorithms that were ripe for optimisation, falling into two primary categories:

1. Replacing linear searches for constant-time lookups (O(n) → O(1)): In a single instance, replacing an iterative array loop with a direct hash map for data lookup produced a staggering 44x speedup.

2. Calculating shared data for parallel tasks once instead of re-doing it (O(N x C) → O(C)): Instead of forcing parallel workers to calculate overlapping elements in isolation, we cached the shared computations upfront. In this case, N refers to the number of elements in the underlying shared dataset, and C refers to the number of parallelised tasks.

Flipping the Statistical Game

The crucial takeaway here wasn’t just the sheer runtime gains made to specific snippets of code. The real victory was securing a guaranteed faster p100 by fundamentally altering our algorithmic time complexity.

For example, flattening a hot path from O(n) to O(1) means that specific chunk of logic is no longer sensitive to scaling. Whether it processes 10 items or 10,000, it takes the exact same fraction of a second.

By removing these variable multipliers from our deeply nested layers, we engineered a hard performance ceiling into our execution tree. We stripped the leaf nodes of their ability to scale out of control, ensuring that a catastrophically slow tail-end outlier can never drag down the runtime of the overall calculation again.

The welcomed side-effect - cost savings

Although cost reduction wasn’t our primary goal, flattening the leaf-node tail latencies had a massive impact on our infrastructure bill.

Once deployed, we observed a 70% decrease in our daily Lambda billed duration. For a system that was costing us over $3k/day/environment in Lambda compute alone, the savings scale to approximately $835k/year/environment.

But how?

AWS Lambda bills on a GB-second model (allocated memory x execution duration). Because of our synchronous fan-out pattern, a tiny runtime optimisation made several layers deep might only shave a few seconds off the end-to-end API response time, but it multiplies exponentially in aggregate.

By optimizing the hot paths, our API’s p99.9 runtime dropped by a modest 20 seconds (from 180s down to 160s to beat the SLA). But under the hood, shrinking those microscopic leaf-node durations eliminated ~9,600 hours of billable Lambda compute time every single day per environment.

The System

The system was a credit management platform for the tertiary education sector, designed to support application assessment, articulation rules, subject matching, and reporting workflows across institutions.

From an architectural perspective, it was already decomposed into independent domains. The system consisted of seven micro services alongside a shared infrastructure stack. Each service owned a specific area of the platform — users, courses, subjects, precedents, articulations, rules, and reporting — with roughly 15–20 Lambda functions per service.

The team consisted of eight engineers, and the system itself was functional. The focus was shifting from an MVP that answered ‘does it work?’ to a production-ready system that could be operated reliably.

That shift was the catalyst for unearthing multiple operational problems.

What Was Inherited

The infrastructure was built using the Serverless Framework, defined in YAML, and deployed via CloudFormation. The application code was written in JavaScript and bundled using Webpack.

Deployments were executed manually via the Serverless CLI, stack by stack, and the entire team worked in a single shared development environment.

These choices were reasonable for an MVP, but they didn’t scale well with the system or the team.

The Problems

Day-long deployments

Deployments took between six and eight hours.

They were run manually, in sequence, with each stack deployed one after the other. Even services that had no dependency on each other were forced into a linear deployment order.

In practice, a deployment looked like this: the lead engineer would trigger a CLI deploy, then sit and watch as each Lambda was built and packaged one by one. This would eventually roll into a CloudFormation stack update, where the deployment progressed resource by resource and required monitoring throughout. A failure partway through would trigger an automatic rollback, after which the entire process had to be run again.

Each stack took roughly 45–60 minutes to complete. Once one finished, the process was repeated for the next, and then the next, until all services had been deployed.

Because of the duration and the way deployments behaved end-to-end, supervision was required. In practice, this meant that once a week, the project’s lead engineer would spend an entire day running a production deployment and monitoring it throughout.

Slow bundling

The application was bundled using Webpack.

Each Lambda function triggered a full build of the entire project. There was no build caching in place, and builds were repeated across functions even when most of the code was shared.

This meant that a large part of the deployment time was not spent deploying — it was spent rebuilding the same codebase multiple times.

Bundling tools have improved significantly since, particularly with caching and incremental builds. But in this setup, the build process amplified the cost of every change.

Stepping on each other’s toes

All development and testing happened in a single shared environment.

Engineers deployed changes into the same stacks and tested on top of each other’s work. Even unrelated changes could interfere, simply because they shared the same runtime environment.

Testing became coupled to timing. Engineers had to coordinate deployments, wait for others to finish, and occasionally re-test work that had been affected by someone else’s changes.

No CI/CD

As previously mentioned in Day-long deployments, deployments were entirely manual.

There was no pipeline, no automation, and no abstraction over the deployment process. Every release required someone to be present, run commands, and monitor the outcome.

This increased both the time cost and the cognitive load of deployments.

Infrastructure as static templates

Infrastructure was defined as YAML templates.

While this worked for defining resources, it limited how infrastructure could be structured and evolved. Relationships between components were implicit, and changes required careful coordination across files.

A Visual Summary

On paper, the system was composed of micro-services. During deployment, it behaved as a monolith.

The rest of this article focuses on how we moved from the structure on the left to the one on the right.

The Operational Re-vamp

To summarise the decisions taken, we made a set of foundational changes to how the system was built and deployed:

• Migrated infrastructure from the Serverless Framework to AWS CDK

• Replaced Webpack with esbuild for faster, incremental builds

• Standardised on TypeScript across both infrastructure and application code

• Introduced an AWS CI/CD pipeline using CodePipeline and CodeDeploy to orchestrate and execute deployments

• Introduced a custom local deployment command to compile, package, and deploy individual Lambdas using esbuild and the AWS CLI, providing the same functionality as the Serverless deploy command with significantly reduced execution time

These changes established the baseline that allowed the deployment model to be restructured.

From day-long deployments to parallel waves

By leveraging AWS CodePipeline, deployments were restructured into two waves.

The first wave handled shared infrastructure. Once complete, all service stacks were deployed in parallel. Independent services were no longer forced into a sequential deployment order.

This removed a significant portion of idle waiting time during deployments.

From full rebuilds to incremental builds

Migrating from Serverless to AWS CDK allowed us to remove our dependency on Webpack and adopt a faster bundling approach using esbuild.

Builds became incremental, shared code was reused, and unchanged components were no longer recompiled on every deployment. Instead of rebuilding the entire system repeatedly, only what changed was rebuilt.

This reduced build times from hours to minutes.

It’s worth noting that Webpack itself was not the bottleneck here. While it had already introduced support for caching and incremental builds, our setup relied on a Serverless Framework plugin that had not yet adopted those capabilities. In practice, this meant full rebuilds were still occurring on every deployment.

From shared environment to sandboxed development

Sandbox environments were introduced for development.

Each engineer could deploy their own version of the Lambda functions while still pointing to shared infrastructure. This allowed changes to be tested in isolation without affecting others.

Development became parallel. Engineers no longer needed to coordinate to validate their work.

From manual execution to CI/CD

Deployments were moved into an automated pipeline orchestrated by AWS CodePipeline.

CodePipeline was configured with a source stage connected to our GitHub repository, triggering the pipeline on code changes. From there, it coordinated the flow through build and deployment stages, handing off to CodeDeploy for execution.

Releases could be initiated without manual intervention, and the need for supervision was removed. Roll outs became repeatable and consistent.

From templates to programmable infrastructure

Infrastructure was redefined using a programmatic approach.

This allowed explicit modelling of components, clearer structure, and reuse of logic. Infrastructure became part of the system, rather than a separate configuration layer.

The Result

• Deployments went from around eight hours down to approximately twenty minutes.

• Individual Lambda updates went from ~2 minutes using the Serverless CLI to under thirty seconds with a custom local deployment command.

• Full system deployments no longer required supervision.

• Engineers could test changes independently without interfering with each other.

The most important improvement was not just speed. It was the removal of unnecessary coupling — in builds, in deployments, and in how the team worked.

A Balanced Take

The original tooling was not incorrect. It was well-suited for getting the system to a working state quickly.

As the system grew, the requirements changed. More control was needed over how deployments were structured, how builds were executed, and how engineers interacted with the system.

Many of the limitations encountered here may be addressed differently today with newer tooling and improved workflows. But the underlying question remains:

Should the way a system is deployed reflect how it is structured?

The system itself did not change dramatically. The services were already there. The boundaries already existed. What changed was how the system was operated.

By the end, deployments were fast, automated, and no longer required supervision. Engineers could work independently without being blocked by the environment or each other.

The Trillion-Object Blind Spot

This article analyses a production system that accumulated 5.6 PB of data across 1.56 trillion objects in a single bucket. Within one year, monthly storage cost increased from approximately $100k to over $400k, with forecasts exceeding $1M per month just 12 months later.

The root cause was not data volume alone, but architectural fragmentation misaligned with S3’s pricing model. The architecture generated hundreds of small snapshot artifacts per request, causing object count to grow faster than volume. A consolidation experiment showed that by aligning object granularity with S3’s economic structure, equivalent logical data could have been stored at 37x lower monthly cost. This case demonstrates that cost modelling must be treated as a first-class architectural constraint.

Paying for an Index, Getting a Bucket

It’s easy in our day to day to simplify storage costs as purely volumetric - e.g., C≈V, where C = monthly cost and V = stored volume.

For AWS S3, an accurate representation requires considering many more facets. Most of them you can find on the official AWS Pricing calculator - hence I will skip the formula definitions here.

There is a crucial cost factor that most people skip though - and is also omitted in the official AWS cost calculator. Object cardinality. At small scale, object cardinality is negligible relative to volume but at larger scales it becomes a first-order variable.

The main variable that determines whether object count matters is average object size, with the following formula:

Where s = Average object size (GB), V = Total stored volume (GB) and N = total object count.

When average object size falls into the kilobyte range, per-object pricing dominates. The system no longer behaves like bulk storage, instead it’s more akin to a massively distributed index - except you are paying storage-layer economics for index-layer behaviour.

A Real-World Example

In the system analysed, our bucket composition was the following:

• 5.6 petabytes stored

• 1.56 trillion objects

• ~3.5 KB average object size

• $400k/month storage cost

• ~$50k/month in request charges

The architecture generated hundreds to thousands of small snapshot artifacts per back end request. Over time, fragmentation compounded. Object count grew faster than volume.

A consolidation experiment showed that the same data could instead have been stored in artifacts averaging ~2.5 MB.

This would have reduced storage cost from $457,000 to $11,900 per month for the same volume of data. This represents a 37× structural reduction.

This reduction was not due to compression or deletion of data. The total logical volume remained constant. Only object granularity changed.

The $7.2 Million Lifecycle "Ghost"

Another (nearly) expensive lesson came from trying to fix our exponentially increasing storage costs. Without permanently losing important insights into our service, the only cost saving alternative we believed to have was resorting to lifecycle transition policies.

At that point, the bucket that now holds over 1.56 trillion objects had 720 billion objects in it. We came up with a plan for all objects older than 6 months to transition automatically with Lifecycle Policies from Standard Access to Infrequent Access, then eventually into Glacier.

This solution could have had a seven figure cost, as lifecycle transitions are priced per 1,000 objects. Putting pen to paper, the lifecycle transitions calculation would have been:

What saved us

Approximately $7.2 million — for a single lifecycle rule on a single bucket. This excludes ongoing storage cost in the new tiers on top of that.

The transition did not execute because most objects were smaller than 128 KB, which do not transition by default. Hence the irony of the same fragmentation pattern that caused excessive steady-state cost also preventing an even larger transition bill.

Why Your Remediation Plan Might Bankrupt You

To prevent similar failures, object storage systems should be evaluated across three explicit budgets.

1. Volume Budget

Projected monthly storage cost.

2. Cardinality Budget

Total object count and average object size.

If average object size falls below a defined threshold (e.g. 1–10 MB for snapshot systems), object count becomes a risk indicator.

3. Remediation Budget

Cost of rewriting, transitioning, or migrating all objects.

Before implementing lifecycle rules or structural migrations, compute:

If remediation cost exceeds acceptable monthly spend, the architecture is already broken.

Object count must be monitored alongside stored bytes. Divergence between the two is architectural drift, not growth.

Conclusion

Our system scaled flawlessly. It simply became unaffordable, under:

• Multi-petabyte scale

• Trillion-object cardinality

• Monthly cost growth from $100k to $400k

• Forecast exceeding $1M/month

• A potential $7M lifecycle event

Object storage is not purely volumetric. It is priced across bytes, objects, and operations.

At extreme scale, pricing semantics become architectural constraints.

Cost modelling must be treated as a first-class design discipline.

If you want a more in-depth operational view on how to identify and tackle these AWS S3 storage issues, take a look at this article:

Disclaimer: Based on public AWS pricing and production experience. Not an official AWS statement.

In this article, we extend the engine in two fundamental ways. First, we introduce a data provider abstraction, decoupling traversal logic from the underlying data source. We begin with a local OpenFlights dataset, but the engine now depends only on a provider interface — allowing live API-backed implementations to be substituted without modifying search logic. The system becomes extensible through abstraction rather than conditionals.

Second, we enrich the domain model itself. Flights become timezone-aware events. Travel duration is derived from airport coordinates and aircraft cruise speeds. Layovers are validated against feasibility rules. Pricing becomes a structured responsibility of the provider rather than a hardcoded afterthought.

These changes transform the problem from structural traversal into constrained itinerary evaluation. The traversal strategy remains depth-aware, but the cost of evaluating each branch increases significantly. This performance regression is intentional — the natural consequence of introducing realism. In the next article, we will address how to make this richer system efficient.

Designing for Substitution - introducing data providers

In the previous iteration of the engine, traversal and data access were tightly coupled. The graph was constructed from a local dataset, and the search algorithm operated directly on that structure. While functional, this approach implicitly assumed that the data source was static and local.

Real flight search systems do not operate this way. They query distribution systems, APIs, and live availability services. If the engine is to evolve toward real-world integration, the abstraction boundary must shift.

We introduce a FlightDataProvider interface. The engine no longer knows how flights are stored or retrieved. It requests outbound flights for a given airport and departure time, and it requests a price for a completed route. That is the entirety of the contract.

Conceptually, the dependency direction changes:

Before, the engine depended directly on the graph and its concrete dataset. Now it depends on an abstraction. The concrete provider — whether local or API-backed — is selected at composition time.

The provider interface is intentionally narrow. It does not expose storage concerns or transport mechanisms. It exposes only the capabilities required for search. This ensures that substitution happens through implementation, not conditional branching.

Instead of embedding logic such as:

if provider_type == "local":
    ...
elif provider_type == "live":
    ...

We compose the engine with a concrete implementation:

provider = OpenFlightsProvider(data_dir)

routes = find_flight_routes(
    origin="JFK",
    destination="SYD",
    provider=provider,
    departure_time=departure_time,
    max_legs=3,
    max_routes=5,
)

Replacing the data source is as simple as instantiating a different provider:

provider = RealProvider(api_key=...)

The engine itself remains unchanged.

This design follows established principles:

• Liskov Substitution Principle — any provider must be replaceable without affecting correctness.

• Open/Closed Principle — the engine is open to extension, closed to modification.

• Dependency Inversion — high-level traversal depends on abstractions, not concrete data sources.

With this boundary in place, traversal logic, time modelling, feasibility constraints, and pricing can evolve independently of how data is sourced. That separation is the structural foundation for the remainder of this article.

Modelling Real-World Constraints

With the provider abstraction in place, we can now make the engine behave more like a real flight system.

Our OpenFlightsProvider is not a live scheduling backend. It is a deliberate approximation. We simulate daily departures at fixed times. We calculate travel duration from airport coordinates and aircraft cruise speeds. We attach structured pricing to completed routes. We enforce layover feasibility rules.

The goal is not to replicate airline reservation systems. The goal is to introduce the kinds of constraints that make flight search non-trivial — while remaining deterministic enough to run locally without relying on a live API.

We want something close enough to reality to reason about scheduling, cost, and optimisation. That is the purpose of this provider.

Time as a Constraint

Once we introduce departure and arrival times, routes stop being abstract paths.

Each flight now has:

• A departure datetime in the origin airport’s local timezone

• An arrival datetime in the destination timezone

• A calculated duration

We simulate a fixed daily departure time in the origin airport’s local timezone. Arrival time is derived from the computed travel duration. Internally, we normalise through UTC to ensure timezone transitions are handled correctly.

The important shift is this: connectivity is no longer sufficient. A connection must also be feasible in time.

Layover validation enforces this. The next flight must depart after arrival, and the layover must fall within acceptable bounds. At each recursive step, we carry forward not only the current airport, but the current arrival time.

The search state becomes richer: it is no longer just (airport), but (airport, time).

Physical Reality as a Constraint

In the structural engine, every edge was effectively weightless. Now, travel time is derived from physical distance.

We compute distance between airports using their latitude and longitude. That distance is combined with an aircraft-specific cruise speed to determine airborne time, and a small ground buffer is added to approximate taxi, climb, and descent.

Different aircraft types travel at different speeds. A long-haul wide-body does not behave like a regional turboprop. Even in a simplified model, this matters.

This introduces two important changes:

1. Duration is no longer constant or arbitrary.

2. Evaluating a route now requires real computation.

Each candidate route requires distance calculations, speed resolution, and datetime arithmetic. These operations accumulate quickly as the search deepens.

Cost as a Constraint

We also introduce pricing, and importantly, pricing belongs to the provider.

The search engine does not calculate fares itself. It requests a price for a completed FlightRoute. For the OpenFlightsProvider, we simulate pricing using a structured model:

• A base fare

• A distance-based component

• A layover penalty

The result is a structured Price object attached to the route.

This keeps the pricing logic aligned with the provider abstraction introduced earlier. A future live provider could replace this synthetic pricing model with real API-backed offers — without changing the search engine.

Cost is now part of the evaluation of a route, not an afterthought.

Feasibility as a Constraint

Taken together, these constraints change the nature of the problem.

Previously, we asked:

Is there a path between A and B within N hops?

Now we ask:

Is there a feasible, time-valid, economically evaluated itinerary between A and B within N hops?

The traversal algorithm itself has not changed. We still perform depth-first search with hop limits. What has changed is the weight of each expansion. Every step now carries temporal state, physical computation, and potential pricing. We have moved from connectivity to constrained feasibility.

With time, physical modelling, and pricing integrated, the engine now produces itineraries that reflect scheduling constraints rather than simple connectivity. A truncated example output is shown below.

./run_local.sh
Skymesh provider loaded
Airports loaded: 6072
Origins with outbound routes: 3241

============================================================
Searching routes: SYD -> MEL

SYD -> MEL (1 legs, total 1h 7m)
    Leg 1: SYD -> MEL
    Airline: Qantas
    Dep: 9:00 AM, 01-03-2026 (UTC+10:00)
    Arr: 10:07 AM, 01-03-2026 (UTC+10:00)
    Total Price: USD 134.65 (Base: 50.00, Distance: 84.65, Layover: 0.00)

============================================================
Searching routes: JFK -> SYD

JFK -> SYD (2 legs, total 29h 30m)
    Leg 1: JFK -> DXB
    Dep: 9:00 AM, 01-03-2026 (UTC-05:00)
    Arr: 7:16 AM, 02-03-2026 (UTC+04:00)

    Layover: 1h 43m

    Leg 2: DXB -> SYD
    Dep: 9:00 AM, 02-03-2026 (UTC+04:00)
    Arr: 5:30 AM, 03-03-2026 (UTC+10:00)

    Total Price: USD 2,891.30

Depth-Aware Ranking and Search

In Implementing a Flight Search Engine, we introduced Iterative Deepening Depth-First Search (IDDFS) as the traversal backbone of the system. There, we explored its mechanics in detail and justified the choice in terms of hop optimality, bounded memory behaviour, and predictable expansion.

[Article 3 here]

Introducing time, distance, and pricing does not require us to change that traversal strategy. What changes is the cost of evaluating each branch and the way we rank completed routes.

IDDFS remains the backbone. The difference now lies in how we order and constrain results within each depth.

Hop Count Remains the Primary Ordering

In practical flight search, direct routes are generally preferred over multi-stop itineraries. A two-hour direct flight should not be outranked by a slightly faster two-stop alternative.

For that reason, hop count remains the primary ordering constraint.

We search depth by depth:

• First, all 1-leg routes

• Then, 2-leg routes

• Then, 3-leg routes

Within each depth, we rank routes by total duration. But we never allow a deeper route to displace a shallower one purely because it is shorter.

This preserves structural preference while still allowing meaningful ranking within the same hop count.

Sorting Once Per Depth

A naive implementation might collect all routes and perform a global sort by duration or price. That approach breaks hop semantics and introduces unnecessary computational cost.

Instead, we:

1. Explore all routes for a given depth limit

2. Sort only the routes discovered at that depth

3. Select up to max_routes

4. Stop deepening once sufficient routes are found

Sorting happens once per depth — not on every append, and not globally.

This approach maintains depth priority while keeping ranking behaviour predictable and controlled.

Why Not Breadth-First Search?

Breadth-First Search (BFS) naturally finds shortest paths in terms of hop count. For purely structural traversal, BFS would be a reasonable alternative.

However, BFS maintains the entire frontier in memory. Once we introduce time-aware state — effectively (airport, time) pairs — the breadth of that frontier expands quickly.

IDDFS gives us the same hop-optimal guarantees as BFS, but with significantly lower memory overhead. We trade some re-computation for bounded memory growth. Given that each node evaluation now performs non-trivial computation, controlling memory pressure is important.

Why Not Dijkstra’s Shortest Path?

Dijkstra’s algorithm optimises for a weighted shortest path. If our objective were strictly “minimise total duration” or “minimise price,” Dijkstra would be a natural candidate.

But our problem is not purely weight-based.

We enforce layered constraints:

• Maximum hop count

• Layover feasibility

• Provider-defined pricing

• Depth-based preference (direct routes outrank multi-stop routes)

Dijkstra assumes a single monotonic weight function. Our engine applies structural limits first, then ranks within those limits.

Hop count is not just a weight — it is a boundary.

IDDFS allows us to enforce hop limits explicitly while layering duration-based ranking inside each depth. That alignment between algorithm and domain constraints is intentional.

The Backbone Remains the Same

The traversal strategy itself did not change when we introduced time, cost, and feasibility. What changed was the richness of the state propagated through recursion and the computational cost of evaluating each branch.

We did not replace the algorithm, we increased the weight of each node. That distinction becomes visible when we measure runtime.

Performance Regression: Measuring the Impact of Added Constraints

Up to this point, every change we introduced was about realism — not speed. We did not alter the traversal algorithm, increase max_legs, or expand the graph. Yet runtime increased significantly.

In the structural engine, a typical search completed in under one second. After introducing time modelling, aircraft-based duration calculations, layover validation, and provider-driven pricing, the same search now takes approximately seven to nine seconds.

Nothing about the traversal strategy changed. What changed was the cost of evaluating each node.

Search Runtime Comparison

What Became More Expensive?

Each completed route now requires:

• Distance calculation between airports

• Cruise speed resolution based on aircraft type

• Duration computation and timezone-aware datetime arithmetic

• Layover validation

• Provider-based pricing evaluation

These are not simple adjacency checks. They involve trigonometric functions, datetime conversions, and structured object construction. Individually small, collectively significant.

In the structural engine, evaluating a branch meant checking connectivity. In the constrained engine, evaluating a branch means simulating a flight.

Time also expands the search state. Previously, a node represented an airport identifier. Now it effectively represents (airport, arrival_time).

The same airport may appear multiple times in the search tree at different times of day, each carrying distinct feasibility implications. The graph has not grown, but the dimensionality of state has.

The regression does not stem from a more complex traversal algorithm. It stems from increased per-node computation and richer state propagation.

Having made the system more expressive and realistic, we now face the natural next question: how do we reduce this computational cost without compromising correctness? That will be the focus of the next article in this series, where we optimise this slower—but more accurate—engine.

The root cause was not storage volume. It was object cardinality.

This article is the operational companion: a runbook for diagnosing and fixing small-object explosions in production S3 systems.

The Operational Model

Operating S3 at scale follows a simple lifecycle:

Most large S3 cost failures occur because observation is missing.

Quick Triage

When S3 costs increase unexpectedly, start with one simple check.

If this number becomes too small, the system is accumulating fragmented artifacts.

Rule of thumb

> 10 MB     → Healthy
1–10 MB     → Acceptable
< 1 MB      → Fragmentation risk
< 128 KB    → CRITICAL

128 KB matters because:

• by default, lifecycle rules do not apply to objects sized ≥128KB

• some storage classes bill small objects as if they were 128 KB minimum

Once the average object size drops below this boundary, the system shifts into object-dominated pricing behaviour.

Diagnosis

Step 1 — Check bucket metrics

The quickest way to detect a cardinality issue is to compute average object size directly from CloudWatch metrics.

CloudWatch already exposes the required metrics:

• AWS/S3 BucketSizeBytes

• AWS/S3 NumberOfObjects

CloudWatch Dashboard Widget

Paste the following JSON into a CloudWatch dashboard (Source view) and replace the bucket name.

{
  “metrics”: [
    [ “AWS/S3”, “NumberOfObjects”, “BucketName”, “example-bucket”, “StorageType”, “AllStorageTypes”, { “id”: “m1”, “stat”: “Sum”, “label”: “objects”, “visible”: false } ],
    [ “.”, “BucketSizeBytes”, “.”, “.”, “.”, “StandardStorage”, { “id”: “m2”, “yAxis”: “right”, “label”: “size”, “visible”: false, “stat”: “Maximum” } ],
    [ { “expression”: “(m2/m1) / 1024”, “label”: “average object size (KB)”, “id”: “e1” } ]
  ],
  “view”: “timeSeries”,
  “stacked”: false,
  “region”: “us-east-1”,
  “period”: 86400,
  “stat”: “Average”
}

This widget will compute average object size and display it in kilobytes.

This is the single most useful metric for detecting S3 cost drift.

If the graph trends downward toward 128 KB, the bucket is likely accumulating fragmented artifacts faster than storage volume is growing.

Step 2 — Verify lifecycle eligibility

Average object size tells you that fragmentation exists.

The next step is determining whether lifecycle remediation will actually work. This requires analysing the distribution of object sizes.

Why this matters:

Many Glacier storage classes do not transition objects smaller than 128 KB by default (but can be explicitly configured to do so).

If most objects fall below this threshold, either lifecycle transitions won’t be triggered with default behaviour, or risk incurring large costs if configured for smaller object sizes.

Example Athena query

Using S3 Inventory:

SELECT
  CASE
    WHEN size < 32768 THEN ‘<32KB’
    WHEN size < 131072 THEN ‘32KB–128KB’
    WHEN size < 1048576 THEN ‘128KB–1MB’
    ELSE ‘>1MB’
  END AS size_bucket,
  COUNT(*) AS object_count
FROM s3_inventory_table
GROUP BY 1
ORDER BY 1;

Example output:

<32 KB        : 420B objects
32KB–128KB    : 230B objects
128KB–1MB     : 50B objects
>1MB          : 20B objects

Interpretation:

→ ~650B objects below 128 KB
→ lifecycle transitions will be largely ineffective

In this scenario, lifecycle transitions would generate massive transition costs while producing minimal storage savings.

Monitoring

Once the issue is diagnosed, implement continuous monitoring. The same average object size metric applies.

Tracking this value over time provides an early signal of fragmentation long before cost increases appear on billing dashboards.

Alerting

A practical alert should trigger when: AverageObjectSize < 128 KB for three consecutive days.

Requiring multiple evaluation periods avoids alerts caused by temporary ingestion spikes or batch jobs.

AWS CDK example

const bucketName = 'your-bucket-name';

// Core S3 metrics
const objects = new cloudwatch.Metric({
  namespace: 'AWS/S3',
  metricName: 'NumberOfObjects',
  dimensionsMap: { BucketName: bucketName, StorageType: 'AllStorageTypes' },
  statistic: 'Sum',
  period: Duration.days(1),
});

const bytes = new cloudwatch.Metric({
  namespace: 'AWS/S3',
  metricName: 'BucketSizeBytes',
  dimensionsMap: { BucketName: bucketName, StorageType: 'StandardStorage' },
  statistic: 'Maximum',
  period: Duration.days(1),
});

// Average object size = bytes / objects
const avgSize = new cloudwatch.MathExpression({
  expression: 'bytes / objects',
  usingMetrics: { bytes, objects },
  label: 'avg object size (bytes)',
});

// Alert if <128 KB for 3 days
new cloudwatch.Alarm(this, 'S3CardinalityAlarm', {
  metric: avgSize,
  threshold: 128 * 1024,
  evaluationPeriods: 3,
  datapointsToAlarm: 3,
  comparisonOperator: cloudwatch.ComparisonOperator.LESS_THAN_THRESHOLD,
});

Remediation

Once fragmentation is confirmed, remediation must be planned carefully. At large scale, fixing the dataset can itself be expensive.

Step 1 — Estimate lifecycle transition cost

Lifecycle transitions are priced per 1,000 objects. Estimate remediation cost before enabling transitions:

Example:

objects = 720_000_000_000
price_per_1000 = 0.01

transition_cost = (objects / 1000) * price_per_1000
print(f”${transition_cost:,.0f}”)

Output:

This calculation prevented a multi-million-dollar lifecycle event in the system analysed in the previous article.

Step 2 — Execute remediation safely

Avoid custom scripts for billion-object operations.

Instead use S3 Batch Operations, which provides:

• automatic parallelisation

• retry handling

• distributed execution across AWS infrastructure

Batch Operations is designed specifically for large-scale object-level changes.

Prevention

If the system is still evolving, implement patterns to prevent future cardinality explosions.

Aggregate small artifacts

Bad pattern: request → write 200 objects

Better pattern: request → buffer → write 1 aggregated object

If artifacts are generated in the KB range, introduce a buffering layer before writing to S3.

Design operational prefixes

Structure buckets around operational boundaries.

Example:

s3://bucket/
  service-a/
  service-b/
  telemetry/
  snapshots/

Benefits:

• targeted lifecycle rules

• efficient Athena queries

• easier remediation

Prefer metadata for read-time inspection

If object attributes must be inspected during reads: Use object metadata.

Use tags primarily for:

• lifecycle rules

• governance policies

Metadata avoids additional API calls when reading large numbers of objects.

Operational Checklist

When S3 costs spike unexpectedly

1️⃣ Check object count
2️⃣ Compute average object size
3️⃣ If <128 KB → fragmentation detected
4️⃣ Verify lifecycle eligibility (size distribution)
5️⃣ Estimate remediation cost
6️⃣ Use Batch Operations for large-scale fixes

Conclusion

The system analysed in the previous article scaled perfectly. It simply became economically unstable.

At petabyte scale, object storage is no longer purely a storage problem. It becomes a cardinality management problem.

Treat object count as a first-class operational metric.

Otherwise, a perfectly functioning system can quietly drift into a multi-million-dollar bill.

An alarm fires: SQS message delay > 5 minutes.

A batch job somewhere in the system just spiked the queue from 100k messages per minute to 1 million. Your workers are auto scaling, but it takes 10–15 minutes to catch up.

Nothing is broken. The processing rate is exactly what it was five minutes ago—stable, efficient, and maxed out. Nothing needs fixing. But someone once decided that a five-minute delay is worthy of waking another human being.

You acknowledge the alarm. You watch the queue drain. After all, you cannot make the auto-scaling go any faster, and messages will remain delayed until the ingestion rate catches up to the new queue size. You are there simply to witness the inevitable. Ten minutes later, the system heals itself.

You go back to sleep — or try to — and spend the next 45 minutes staring at the ceiling while the rest of tomorrow quietly collapses.

After enough nights like this, a certain mindset begins to form.

You start acknowledging alarms with less urgency. You observe the system more than you intervene in it. You learn to distinguish between problems that require action and problems that simply require time. Usually, the system corrects itself.

Eventually you arrive at a quiet realisation:

Many pages exist not because the system is failing, but because someone once mistook temporary discomfort for catastrophe.

This is what I call a nihilist on-call.

Not laziness. Not negligence. Just a gradual understanding that many pages do not correspond to real failures, and that intervention often changes little. You watch the system. You wait. And most of the time, it fixes itself.

The Heat Death of the Service

Spend enough time on-call and another thought begins to surface. Most services do not fail catastrophically. They decay.

Dependencies drift. Dashboards fall out of date. Runbooks turn into archaeological artifacts documenting systems that no longer exist. This is operational entropy, and given enough time, every sufficiently complex service approaches its own version of heat death.

This is where the nihilism takes root.

When you realize the service is in a state of slow-motion decay, the urgency of the pager starts to feel like a lie. You arrive at the nihilistic realization that you aren’t “saving” the system; you are merely performing an act of penance for a design you didn’t choose. Many alerts imply that the system is moments away from collapse, but the reality is a slow, gray fade into obsolescence.

But heat death is only inevitable if you accept the role of the bystander.

The sense of futility is a protective layer of scar tissue, but it is also a choice. We treat entropy as a law of nature, but in software, entropy is a choice of priority. Most pages are not preventing the heat death of the service because they focus on the symptoms of the decay rather than the decay itself. To move past this futility, we have to stop treating the pager as a death knell and start using it as a diagnostic tool for restoration.

Signal Dilution

There is another consequence of this dynamic that only becomes obvious at scale. When a system produces enough meaningless pages, the meaningful ones begin to dissolve into the noise.

The pager still rings. The alarm still says SEV-2. But psychologically it carries far less weight than it should. Not because engineers have become careless, but because the system itself has trained them to treat alerts with skepticism.

Alarm inflation is almost inevitable in large organizations. Every team adds alerts intended to protect their own services. Very few alarms are ever removed. Self-healing systems page simply because a metric briefly crossed an arbitrary threshold. Over time the system begins to generate a constant background hum of operational noise.

This is the real danger of alarm fatigue. Reliability is rarely destroyed by a single catastrophic failure. Much more often it is eroded gradually by a loss of trust in the signals meant to protect the system. The fastest way to make engineers ignore an alarm is to page them repeatedly for events that resolve themselves.

The Myth of the Pager

In the myth of Sisyphus, a man is condemned to push a boulder up a hill forever, only for it to roll back down each time he reaches the top.

On-call sometimes feels similar. The pager rings. You acknowledge the alarm. You click through dashboards. The system stabilizes. And somewhere in the background, another alert is already preparing to wake you tomorrow night.

But unlike Sisyphus, engineers are not actually condemned to this cycle. Most on-call pain is not inevitable. It is designed.

To cure the futility of the on-call, we should refuse absurd labor. We should stop engineering better ways to push the boulder and start questioning why it exists at all.

Practical Rules for On-Call

No Human Intervention, No Page.

If an alert fires and the resolution is simply “wait for it to clear” or “restart the service,” the rock has rolled back to the bottom.

The Rule: If a human doesn’t need to make a unique, creative decision to fix it, a computer should be doing it. Do not wake a human for a task a script could do.

Page on Symptoms, Not Causes.

We should let the system decay in silence if that decay doesn’t hurt the user.

The Rule: High CPU is a cause, not a symptom. If your system is still serving requests and users aren’t experiencing errors or noticeable latency, the pager should stay silent. Only wake someone up when the system is actually broken.

Delete the “Flappy” Alerts.

Every recurring alert that you “acknowledge and ignore” is a high-priority bug in your monitoring system.

The Rule: If an alert fires three times in a shift and requires no action, delete it. Don’t “tune” it. Kill it. If the system doesn’t break when the alert is gone, it was never an alert. It was noise.

Protect the Sleep of Others.

High-signal hygiene is a collective pact.

The Rule: Before you create an alarm, ask yourself: “Am I willing to wake up my best friend at 3:00 AM for this?” If the answer is no, it shouldn’t be pageable.

Conclusion

We should refuse to treat the pager as a tool of penance. The struggle itself toward the heights is enough to fill a man’s heart, but only if the heights actually exist.

If the system is going to reach its heat death anyway, we might as well get some sleep.

This article’s code specifically is available at https://github.com/adriangabardo/skymesh/releases/tag/v3.0.1

Overview

In previous articles in the series, we described the abstract idea of what we were trying to achieve, then laid the foundations for ingesting aviat…

Read more

This article’s code specifically is available at https://github.com/adriangabardo/skymesh/releases/tag/v2.0.0

Overview

The first article in The Graph Series framed the aviation industry as a graph problem: airports as nodes, flights as edges, and routing as constrai…

Read more

In this article, we introduce a project that models the global aviation industry as a graph-theory problem. Airports are represented as nodes and direct flight routes as edges, forming a large, sparse, and highly non-uniform network.

The objective of this project is to explore how common aviation questions - such as route reachability, optimal pa…

Read more