Imagine the architecture below - very quickly it becomes a time and money sink that scales up uncontrollably.

In this article we will be diving exactly into that pattern - and how at a scale of a million calls per day, optimising inefficiencies allowed us to save years of compute time daily, and close to $2 million/year across our development and production environments combined.

The system

The system being optimised was responsible for modelling network behaviour and comparing network state before and after changes to physical entities (such as network routers).

The results of the calculations were critical to assert that changing the physical network of millions of interlinked devices would not cause network impact such as bottlenecks, degradation or packet loss for AWS customers.

Moreover, due to how mission critical this system’s results were to operations, the faster an answer could be given, the better.

The scale

As previously mentioned, this system received about 1 million calls/day - with most of those calls being replicated automatically to the development environment for pre-release validation. So in total, about 2 million calls per day.

As for calculation times and SLAs… it depends. Ask any network engineer and they will tell you about the seemingly endless ways to connect a physical network and make them behave in their own funky little ways. Hence modelling a network of millions of physical devices, with different protocols, physical interfaces, lags, seemingly all shapes and sizes - and it becomes tough to have a uniform calculation across them all.

For simplicity sake, imagine this system either replied in 5-10 seconds, or it would take nearly all of its 180-second SLA to finish a calculation.

Performance bottlenecks in a distributed world

Our synchronous Lambda fan-out pattern introduced two immediate structural flaws:

First, the parent Lambda idles until every single child invocation completes. This is the structural equivalent of grabbing a wad of cash and setting it on fire.

Second, your entire response time is hostage to the child Lambda’s p100. In an event-driven architecture, you can progress sequentially as individual chunks complete. With a synchronous implementation, Step B cannot start until the absolute entirety of Step A finishes, Step C is dependent on the p100 of Step B, and so on.

In systems design, this behaviour is known as the straggler problem. We can visualise it as such:

So, why?

While I wasn’t there for its inception, it is easy to understand the original engineering decisions.

The entire codebase was built in Python to run heavily CPU-bound tasks. Because of Python’s single-threaded nature, the team relied on architectural task-splitting (the Lambda fan-out) rather than in-process parallelism.

The rest comes down to velocity: the fastest way to build a new system is the way you already know. Why Python? Because the team knew it. Why a synchronous fan-out? Because it was operationally straightforward to reason about and easy to scale incrementally under delivery pressure.

When you are facing tight deadlines and intense stakeholder pressure, optimise-for-speed decisions make sense. You can always debate language and architecture choices, but ultimately this system has been operational for over half a decade now.

Optimising without rearchitecting

The entire optimisation effort was triggered by customer pain from transient timeouts. Because we needed immediate relief, a massive, high-risk architecture refactor was off the table. Our sole intention was to find tactical wins that would pull our worst-case tail latency safely under the 180-second SLA.

By targeting the hot paths, we managed to achieve a rock-solid, stable execution ceiling while simultaneously shaving 70% off our daily Lambda spend.

To understand how, you have to look at the math of our synchronous fan-out pattern. When you rely on synchronous layers, your end-to-end performance is entirely held hostage by your lowest, slowest leaf node. If a Lambda three levels deep chokes, it drags the entire parent infrastructure down with it.

At our scale, - 1 million daily top level Lambda invocations - getting dragged down by the Lambda 3 levels deep isn’t just a possibility, it’s a mathematical guarantee when your invocation ceiling is 640B invocations/day.

Targeting the multiplier effect

Because an outlier at deeply nested Lambdas is mathematically guaranteed to drag the entire system down, our primary target was flattening the tail latency (p100).

First, we employed simple profiling techniques to identify the hot execution paths. Then came our silver bullet: asking an AI agent to analyse those hot paths specifically and calculate their time complexity. This process flagged multiple algorithms that were ripe for optimisation, falling into two primary categories:

1. Replacing linear searches for constant-time lookups (O(n) → O(1)): In a single instance, replacing an iterative array loop with a direct hash map for data lookup produced a staggering 44x speedup.

2. Calculating shared data for parallel tasks once instead of re-doing it (O(N x C) → O(C)): Instead of forcing parallel workers to calculate overlapping elements in isolation, we cached the shared computations upfront. In this case, N refers to the number of elements in the underlying shared dataset, and C refers to the number of parallelised tasks.

Flipping the Statistical Game

The crucial takeaway here wasn’t just the sheer runtime gains made to specific snippets of code. The real victory was securing a guaranteed faster p100 by fundamentally altering our algorithmic time complexity.

For example, flattening a hot path from O(n) to O(1) means that specific chunk of logic is no longer sensitive to scaling. Whether it processes 10 items or 10,000, it takes the exact same fraction of a second.

By removing these variable multipliers from our deeply nested layers, we engineered a hard performance ceiling into our execution tree. We stripped the leaf nodes of their ability to scale out of control, ensuring that a catastrophically slow tail-end outlier can never drag down the runtime of the overall calculation again.

The welcomed side-effect - cost savings

Although cost reduction wasn’t our primary goal, flattening the leaf-node tail latencies had a massive impact on our infrastructure bill.

Once deployed, we observed a 70% decrease in our daily Lambda billed duration. For a system that was costing us over $3k/day/environment in Lambda compute alone, the savings scale to approximately $835k/year/environment.

But how?

AWS Lambda bills on a GB-second model (allocated memory x execution duration). Because of our synchronous fan-out pattern, a tiny runtime optimisation made several layers deep might only shave a few seconds off the end-to-end API response time, but it multiplies exponentially in aggregate.

By optimizing the hot paths, our API’s p99.9 runtime dropped by a modest 20 seconds (from 180s down to 160s to beat the SLA). But under the hood, shrinking those microscopic leaf-node durations eliminated ~9,600 hours of billable Lambda compute time every single day per environment.

The System

The system was a credit management platform for the tertiary education sector, designed to support application assessment, articulation rules, subject matching, and reporting workflows across institutions.

From an architectural perspective, it was already decomposed into independent domains. The system consisted of seven micro services alongside a shared infrastructure stack. Each service owned a specific area of the platform — users, courses, subjects, precedents, articulations, rules, and reporting — with roughly 15–20 Lambda functions per service.

The team consisted of eight engineers, and the system itself was functional. The focus was shifting from an MVP that answered ‘does it work?’ to a production-ready system that could be operated reliably.

That shift was the catalyst for unearthing multiple operational problems.

What Was Inherited

The infrastructure was built using the Serverless Framework, defined in YAML, and deployed via CloudFormation. The application code was written in JavaScript and bundled using Webpack.

Deployments were executed manually via the Serverless CLI, stack by stack, and the entire team worked in a single shared development environment.

These choices were reasonable for an MVP, but they didn’t scale well with the system or the team.

The Problems

Day-long deployments

Deployments took between six and eight hours.

They were run manually, in sequence, with each stack deployed one after the other. Even services that had no dependency on each other were forced into a linear deployment order.

In practice, a deployment looked like this: the lead engineer would trigger a CLI deploy, then sit and watch as each Lambda was built and packaged one by one. This would eventually roll into a CloudFormation stack update, where the deployment progressed resource by resource and required monitoring throughout. A failure partway through would trigger an automatic rollback, after which the entire process had to be run again.

Each stack took roughly 45–60 minutes to complete. Once one finished, the process was repeated for the next, and then the next, until all services had been deployed.

Because of the duration and the way deployments behaved end-to-end, supervision was required. In practice, this meant that once a week, the project’s lead engineer would spend an entire day running a production deployment and monitoring it throughout.

Slow bundling

The application was bundled using Webpack.

Each Lambda function triggered a full build of the entire project. There was no build caching in place, and builds were repeated across functions even when most of the code was shared.

This meant that a large part of the deployment time was not spent deploying — it was spent rebuilding the same codebase multiple times.

Bundling tools have improved significantly since, particularly with caching and incremental builds. But in this setup, the build process amplified the cost of every change.

Stepping on each other’s toes

All development and testing happened in a single shared environment.

Engineers deployed changes into the same stacks and tested on top of each other’s work. Even unrelated changes could interfere, simply because they shared the same runtime environment.

Testing became coupled to timing. Engineers had to coordinate deployments, wait for others to finish, and occasionally re-test work that had been affected by someone else’s changes.

No CI/CD

As previously mentioned in Day-long deployments, deployments were entirely manual.

There was no pipeline, no automation, and no abstraction over the deployment process. Every release required someone to be present, run commands, and monitor the outcome.

This increased both the time cost and the cognitive load of deployments.

Infrastructure as static templates

Infrastructure was defined as YAML templates.

While this worked for defining resources, it limited how infrastructure could be structured and evolved. Relationships between components were implicit, and changes required careful coordination across files.

A Visual Summary

On paper, the system was composed of micro-services. During deployment, it behaved as a monolith.

The rest of this article focuses on how we moved from the structure on the left to the one on the right.

The Operational Re-vamp

To summarise the decisions taken, we made a set of foundational changes to how the system was built and deployed:

• Migrated infrastructure from the Serverless Framework to AWS CDK

• Replaced Webpack with esbuild for faster, incremental builds

• Standardised on TypeScript across both infrastructure and application code

• Introduced an AWS CI/CD pipeline using CodePipeline and CodeDeploy to orchestrate and execute deployments

• Introduced a custom local deployment command to compile, package, and deploy individual Lambdas using esbuild and the AWS CLI, providing the same functionality as the Serverless deploy command with significantly reduced execution time

These changes established the baseline that allowed the deployment model to be restructured.

From day-long deployments to parallel waves

By leveraging AWS CodePipeline, deployments were restructured into two waves.

The first wave handled shared infrastructure. Once complete, all service stacks were deployed in parallel. Independent services were no longer forced into a sequential deployment order.

This removed a significant portion of idle waiting time during deployments.

From full rebuilds to incremental builds

Migrating from Serverless to AWS CDK allowed us to remove our dependency on Webpack and adopt a faster bundling approach using esbuild.

Builds became incremental, shared code was reused, and unchanged components were no longer recompiled on every deployment. Instead of rebuilding the entire system repeatedly, only what changed was rebuilt.

This reduced build times from hours to minutes.

It’s worth noting that Webpack itself was not the bottleneck here. While it had already introduced support for caching and incremental builds, our setup relied on a Serverless Framework plugin that had not yet adopted those capabilities. In practice, this meant full rebuilds were still occurring on every deployment.

From shared environment to sandboxed development

Sandbox environments were introduced for development.

Each engineer could deploy their own version of the Lambda functions while still pointing to shared infrastructure. This allowed changes to be tested in isolation without affecting others.

Development became parallel. Engineers no longer needed to coordinate to validate their work.

From manual execution to CI/CD

Deployments were moved into an automated pipeline orchestrated by AWS CodePipeline.

CodePipeline was configured with a source stage connected to our GitHub repository, triggering the pipeline on code changes. From there, it coordinated the flow through build and deployment stages, handing off to CodeDeploy for execution.

Releases could be initiated without manual intervention, and the need for supervision was removed. Roll outs became repeatable and consistent.

From templates to programmable infrastructure

Infrastructure was redefined using a programmatic approach.

This allowed explicit modelling of components, clearer structure, and reuse of logic. Infrastructure became part of the system, rather than a separate configuration layer.

The Result

• Deployments went from around eight hours down to approximately twenty minutes.

• Individual Lambda updates went from ~2 minutes using the Serverless CLI to under thirty seconds with a custom local deployment command.

• Full system deployments no longer required supervision.

• Engineers could test changes independently without interfering with each other.

The most important improvement was not just speed. It was the removal of unnecessary coupling — in builds, in deployments, and in how the team worked.

A Balanced Take

The original tooling was not incorrect. It was well-suited for getting the system to a working state quickly.

As the system grew, the requirements changed. More control was needed over how deployments were structured, how builds were executed, and how engineers interacted with the system.

Many of the limitations encountered here may be addressed differently today with newer tooling and improved workflows. But the underlying question remains:

Should the way a system is deployed reflect how it is structured?

The system itself did not change dramatically. The services were already there. The boundaries already existed. What changed was how the system was operated.

By the end, deployments were fast, automated, and no longer required supervision. Engineers could work independently without being blocked by the environment or each other.

The Trillion-Object Blind Spot

This article analyses a production system that accumulated 5.6 PB of data across 1.56 trillion objects in a single bucket. Within one year, monthly storage cost increased from approximately $100k to over $400k, with forecasts exceeding $1M per month just 12 months later.

The root cause was not data volume alone, but architectural fragmentation misaligned with S3’s pricing model. The architecture generated hundreds of small snapshot artifacts per request, causing object count to grow faster than volume. A consolidation experiment showed that by aligning object granularity with S3’s economic structure, equivalent logical data could have been stored at 37x lower monthly cost. This case demonstrates that cost modelling must be treated as a first-class architectural constraint.

Paying for an Index, Getting a Bucket

It’s easy in our day to day to simplify storage costs as purely volumetric - e.g., C≈V, where C = monthly cost and V = stored volume.

For AWS S3, an accurate representation requires considering many more facets. Most of them you can find on the official AWS Pricing calculator - hence I will skip the formula definitions here.

There is a crucial cost factor that most people skip though - and is also omitted in the official AWS cost calculator. Object cardinality. At small scale, object cardinality is negligible relative to volume but at larger scales it becomes a first-order variable.

The main variable that determines whether object count matters is average object size, with the following formula:

Where s = Average object size (GB), V = Total stored volume (GB) and N = total object count.

When average object size falls into the kilobyte range, per-object pricing dominates. The system no longer behaves like bulk storage, instead it’s more akin to a massively distributed index - except you are paying storage-layer economics for index-layer behaviour.

A Real-World Example

In the system analysed, our bucket composition was the following:

• 5.6 petabytes stored

• 1.56 trillion objects

• ~3.5 KB average object size

• $400k/month storage cost

• ~$50k/month in request charges

The architecture generated hundreds to thousands of small snapshot artifacts per back end request. Over time, fragmentation compounded. Object count grew faster than volume.

A consolidation experiment showed that the same data could instead have been stored in artifacts averaging ~2.5 MB.

This would have reduced storage cost from $457,000 to $11,900 per month for the same volume of data. This represents a 37× structural reduction.

This reduction was not due to compression or deletion of data. The total logical volume remained constant. Only object granularity changed.

The $7.2 Million Lifecycle "Ghost"

Another (nearly) expensive lesson came from trying to fix our exponentially increasing storage costs. Without permanently losing important insights into our service, the only cost saving alternative we believed to have was resorting to lifecycle transition policies.

At that point, the bucket that now holds over 1.56 trillion objects had 720 billion objects in it. We came up with a plan for all objects older than 6 months to transition automatically with Lifecycle Policies from Standard Access to Infrequent Access, then eventually into Glacier.

This solution could have had a seven figure cost, as lifecycle transitions are priced per 1,000 objects. Putting pen to paper, the lifecycle transitions calculation would have been:

What saved us

Approximately $7.2 million — for a single lifecycle rule on a single bucket. This excludes ongoing storage cost in the new tiers on top of that.

The transition did not execute because most objects were smaller than 128 KB, which do not transition by default. Hence the irony of the same fragmentation pattern that caused excessive steady-state cost also preventing an even larger transition bill.

Why Your Remediation Plan Might Bankrupt You

To prevent similar failures, object storage systems should be evaluated across three explicit budgets.

1. Volume Budget

Projected monthly storage cost.

2. Cardinality Budget

Total object count and average object size.

If average object size falls below a defined threshold (e.g. 1–10 MB for snapshot systems), object count becomes a risk indicator.

3. Remediation Budget

Cost of rewriting, transitioning, or migrating all objects.

Before implementing lifecycle rules or structural migrations, compute:

If remediation cost exceeds acceptable monthly spend, the architecture is already broken.

Object count must be monitored alongside stored bytes. Divergence between the two is architectural drift, not growth.

Conclusion

Our system scaled flawlessly. It simply became unaffordable, under:

• Multi-petabyte scale

• Trillion-object cardinality

• Monthly cost growth from $100k to $400k

• Forecast exceeding $1M/month

• A potential $7M lifecycle event

Object storage is not purely volumetric. It is priced across bytes, objects, and operations.

At extreme scale, pricing semantics become architectural constraints.

Cost modelling must be treated as a first-class design discipline.

If you want a more in-depth operational view on how to identify and tackle these AWS S3 storage issues, take a look at this article:

Disclaimer: Based on public AWS pricing and production experience. Not an official AWS statement.

In this article, we extend the engine in two fundamental ways. First, we introduce a data provider abstraction, decoupling traversal logic from the underlying data source. We begin with a local OpenFlights dataset, but the engine now depends only on a provider interface — allowing live API-backed implementations to be substituted without modifying search logic. The system becomes extensible through abstraction rather than conditionals.

Second, we enrich the domain model itself. Flights become timezone-aware events. Travel duration is derived from airport coordinates and aircraft cruise speeds. Layovers are validated against feasibility rules. Pricing becomes a structured responsibility of the provider rather than a hardcoded afterthought.

These changes transform the problem from structural traversal into constrained itinerary evaluation. The traversal strategy remains depth-aware, but the cost of evaluating each branch increases significantly. This performance regression is intentional — the natural consequence of introducing realism. In the next article, we will address how to make this richer system efficient.

Designing for Substitution - introducing data providers

In the previous iteration of the engine, traversal and data access were tightly coupled. The graph was constructed from a local dataset, and the search algorithm operated directly on that structure. While functional, this approach implicitly assumed that the data source was static and local.

Real flight search systems do not operate this way. They query distribution systems, APIs, and live availability services. If the engine is to evolve toward real-world integration, the abstraction boundary must shift.

We introduce a FlightDataProvider interface. The engine no longer knows how flights are stored or retrieved. It requests outbound flights for a given airport and departure time, and it requests a price for a completed route. That is the entirety of the contract.

Conceptually, the dependency direction changes:

Before, the engine depended directly on the graph and its concrete dataset. Now it depends on an abstraction. The concrete provider — whether local or API-backed — is selected at composition time.

The provider interface is intentionally narrow. It does not expose storage concerns or transport mechanisms. It exposes only the capabilities required for search. This ensures that substitution happens through implementation, not conditional branching.

Instead of embedding logic such as:

if provider_type == "local":
    ...
elif provider_type == "live":
    ...

We compose the engine with a concrete implementation:

provider = OpenFlightsProvider(data_dir)

routes = find_flight_routes(
    origin="JFK",
    destination="SYD",
    provider=provider,
    departure_time=departure_time,
    max_legs=3,
    max_routes=5,
)

Replacing the data source is as simple as instantiating a different provider:

provider = RealProvider(api_key=...)

The engine itself remains unchanged.

This design follows established principles:

• Liskov Substitution Principle — any provider must be replaceable without affecting correctness.

• Open/Closed Principle — the engine is open to extension, closed to modification.

• Dependency Inversion — high-level traversal depends on abstractions, not concrete data sources.

With this boundary in place, traversal logic, time modelling, feasibility constraints, and pricing can evolve independently of how data is sourced. That separation is the structural foundation for the remainder of this article.

Modelling Real-World Constraints

With the provider abstraction in place, we can now make the engine behave more like a real flight system.

Our OpenFlightsProvider is not a live scheduling backend. It is a deliberate approximation. We simulate daily departures at fixed times. We calculate travel duration from airport coordinates and aircraft cruise speeds. We attach structured pricing to completed routes. We enforce layover feasibility rules.

The goal is not to replicate airline reservation systems. The goal is to introduce the kinds of constraints that make flight search non-trivial — while remaining deterministic enough to run locally without relying on a live API.

We want something close enough to reality to reason about scheduling, cost, and optimisation. That is the purpose of this provider.

Time as a Constraint

Once we introduce departure and arrival times, routes stop being abstract paths.

Each flight now has:

• A departure datetime in the origin airport’s local timezone

• An arrival datetime in the destination timezone

• A calculated duration

We simulate a fixed daily departure time in the origin airport’s local timezone. Arrival time is derived from the computed travel duration. Internally, we normalise through UTC to ensure timezone transitions are handled correctly.

The important shift is this: connectivity is no longer sufficient. A connection must also be feasible in time.

Layover validation enforces this. The next flight must depart after arrival, and the layover must fall within acceptable bounds. At each recursive step, we carry forward not only the current airport, but the current arrival time.

The search state becomes richer: it is no longer just (airport), but (airport, time).

Physical Reality as a Constraint

In the structural engine, every edge was effectively weightless. Now, travel time is derived from physical distance.

We compute distance between airports using their latitude and longitude. That distance is combined with an aircraft-specific cruise speed to determine airborne time, and a small ground buffer is added to approximate taxi, climb, and descent.

Different aircraft types travel at different speeds. A long-haul wide-body does not behave like a regional turboprop. Even in a simplified model, this matters.

This introduces two important changes:

1. Duration is no longer constant or arbitrary.

2. Evaluating a route now requires real computation.

Each candidate route requires distance calculations, speed resolution, and datetime arithmetic. These operations accumulate quickly as the search deepens.

Cost as a Constraint

We also introduce pricing, and importantly, pricing belongs to the provider.

The search engine does not calculate fares itself. It requests a price for a completed FlightRoute. For the OpenFlightsProvider, we simulate pricing using a structured model:

• A base fare

• A distance-based component

• A layover penalty

The result is a structured Price object attached to the route.

This keeps the pricing logic aligned with the provider abstraction introduced earlier. A future live provider could replace this synthetic pricing model with real API-backed offers — without changing the search engine.

Cost is now part of the evaluation of a route, not an afterthought.

Feasibility as a Constraint

Taken together, these constraints change the nature of the problem.

Previously, we asked:

Is there a path between A and B within N hops?

Now we ask:

Is there a feasible, time-valid, economically evaluated itinerary between A and B within N hops?

The traversal algorithm itself has not changed. We still perform depth-first search with hop limits. What has changed is the weight of each expansion. Every step now carries temporal state, physical computation, and potential pricing. We have moved from connectivity to constrained feasibility.

With time, physical modelling, and pricing integrated, the engine now produces itineraries that reflect scheduling constraints rather than simple connectivity. A truncated example output is shown below.

./run_local.sh
Skymesh provider loaded
Airports loaded: 6072
Origins with outbound routes: 3241

============================================================
Searching routes: SYD -> MEL

SYD -> MEL (1 legs, total 1h 7m)
    Leg 1: SYD -> MEL
    Airline: Qantas
    Dep: 9:00 AM, 01-03-2026 (UTC+10:00)
    Arr: 10:07 AM, 01-03-2026 (UTC+10:00)
    Total Price: USD 134.65 (Base: 50.00, Distance: 84.65, Layover: 0.00)

============================================================
Searching routes: JFK -> SYD

JFK -> SYD (2 legs, total 29h 30m)
    Leg 1: JFK -> DXB
    Dep: 9:00 AM, 01-03-2026 (UTC-05:00)
    Arr: 7:16 AM, 02-03-2026 (UTC+04:00)

    Layover: 1h 43m

    Leg 2: DXB -> SYD
    Dep: 9:00 AM, 02-03-2026 (UTC+04:00)
    Arr: 5:30 AM, 03-03-2026 (UTC+10:00)

    Total Price: USD 2,891.30

Depth-Aware Ranking and Search

In Implementing a Flight Search Engine, we introduced Iterative Deepening Depth-First Search (IDDFS) as the traversal backbone of the system. There, we explored its mechanics in detail and justified the choice in terms of hop optimality, bounded memory behaviour, and predictable expansion.

[Article 3 here]

Introducing time, distance, and pricing does not require us to change that traversal strategy. What changes is the cost of evaluating each branch and the way we rank completed routes.

IDDFS remains the backbone. The difference now lies in how we order and constrain results within each depth.

Hop Count Remains the Primary Ordering

In practical flight search, direct routes are generally preferred over multi-stop itineraries. A two-hour direct flight should not be outranked by a slightly faster two-stop alternative.

For that reason, hop count remains the primary ordering constraint.

We search depth by depth:

• First, all 1-leg routes

• Then, 2-leg routes

• Then, 3-leg routes

Within each depth, we rank routes by total duration. But we never allow a deeper route to displace a shallower one purely because it is shorter.

This preserves structural preference while still allowing meaningful ranking within the same hop count.

Sorting Once Per Depth

A naive implementation might collect all routes and perform a global sort by duration or price. That approach breaks hop semantics and introduces unnecessary computational cost.

Instead, we:

1. Explore all routes for a given depth limit

2. Sort only the routes discovered at that depth

3. Select up to max_routes

4. Stop deepening once sufficient routes are found

Sorting happens once per depth — not on every append, and not globally.

This approach maintains depth priority while keeping ranking behaviour predictable and controlled.

Why Not Breadth-First Search?

Breadth-First Search (BFS) naturally finds shortest paths in terms of hop count. For purely structural traversal, BFS would be a reasonable alternative.

However, BFS maintains the entire frontier in memory. Once we introduce time-aware state — effectively (airport, time) pairs — the breadth of that frontier expands quickly.

IDDFS gives us the same hop-optimal guarantees as BFS, but with significantly lower memory overhead. We trade some re-computation for bounded memory growth. Given that each node evaluation now performs non-trivial computation, controlling memory pressure is important.

Why Not Dijkstra’s Shortest Path?

Dijkstra’s algorithm optimises for a weighted shortest path. If our objective were strictly “minimise total duration” or “minimise price,” Dijkstra would be a natural candidate.

But our problem is not purely weight-based.

We enforce layered constraints:

• Maximum hop count

• Layover feasibility

• Provider-defined pricing

• Depth-based preference (direct routes outrank multi-stop routes)

Dijkstra assumes a single monotonic weight function. Our engine applies structural limits first, then ranks within those limits.

Hop count is not just a weight — it is a boundary.

IDDFS allows us to enforce hop limits explicitly while layering duration-based ranking inside each depth. That alignment between algorithm and domain constraints is intentional.

The Backbone Remains the Same

The traversal strategy itself did not change when we introduced time, cost, and feasibility. What changed was the richness of the state propagated through recursion and the computational cost of evaluating each branch.

We did not replace the algorithm, we increased the weight of each node. That distinction becomes visible when we measure runtime.

Performance Regression: Measuring the Impact of Added Constraints

Up to this point, every change we introduced was about realism — not speed. We did not alter the traversal algorithm, increase max_legs, or expand the graph. Yet runtime increased significantly.

In the structural engine, a typical search completed in under one second. After introducing time modelling, aircraft-based duration calculations, layover validation, and provider-driven pricing, the same search now takes approximately seven to nine seconds.

Nothing about the traversal strategy changed. What changed was the cost of evaluating each node.

Search Runtime Comparison

What Became More Expensive?

Each completed route now requires:

• Distance calculation between airports

• Cruise speed resolution based on aircraft type

• Duration computation and timezone-aware datetime arithmetic

• Layover validation

• Provider-based pricing evaluation

These are not simple adjacency checks. They involve trigonometric functions, datetime conversions, and structured object construction. Individually small, collectively significant.

In the structural engine, evaluating a branch meant checking connectivity. In the constrained engine, evaluating a branch means simulating a flight.

Time also expands the search state. Previously, a node represented an airport identifier. Now it effectively represents (airport, arrival_time).

The same airport may appear multiple times in the search tree at different times of day, each carrying distinct feasibility implications. The graph has not grown, but the dimensionality of state has.

The regression does not stem from a more complex traversal algorithm. It stems from increased per-node computation and richer state propagation.

Having made the system more expressive and realistic, we now face the natural next question: how do we reduce this computational cost without compromising correctness? That will be the focus of the next article in this series, where we optimise this slower—but more accurate—engine.

The root cause was not storage volume. It was object cardinality.

This article is the operational companion: a runbook for diagnosing and fixing small-object explosions in production S3 systems.

The Operational Model

Operating S3 at scale follows a simple lifecycle:

Most large S3 cost failures occur because observation is missing.

Quick Triage

When S3 costs increase unexpectedly, start with one simple check.

If this number becomes too small, the system is accumulating fragmented artifacts.

Rule of thumb

> 10 MB     → Healthy
1–10 MB     → Acceptable
< 1 MB      → Fragmentation risk
< 128 KB    → CRITICAL

128 KB matters because:

• by default, lifecycle rules do not apply to objects sized ≥128KB

• some storage classes bill small objects as if they were 128 KB minimum

Once the average object size drops below this boundary, the system shifts into object-dominated pricing behaviour.

Diagnosis

Step 1 — Check bucket metrics

The quickest way to detect a cardinality issue is to compute average object size directly from CloudWatch metrics.

CloudWatch already exposes the required metrics:

• AWS/S3 BucketSizeBytes

• AWS/S3 NumberOfObjects

CloudWatch Dashboard Widget

Paste the following JSON into a CloudWatch dashboard (Source view) and replace the bucket name.

{
  “metrics”: [
    [ “AWS/S3”, “NumberOfObjects”, “BucketName”, “example-bucket”, “StorageType”, “AllStorageTypes”, { “id”: “m1”, “stat”: “Sum”, “label”: “objects”, “visible”: false } ],
    [ “.”, “BucketSizeBytes”, “.”, “.”, “.”, “StandardStorage”, { “id”: “m2”, “yAxis”: “right”, “label”: “size”, “visible”: false, “stat”: “Maximum” } ],
    [ { “expression”: “(m2/m1) / 1024”, “label”: “average object size (KB)”, “id”: “e1” } ]
  ],
  “view”: “timeSeries”,
  “stacked”: false,
  “region”: “us-east-1”,
  “period”: 86400,
  “stat”: “Average”
}

This widget will compute average object size and display it in kilobytes.

This is the single most useful metric for detecting S3 cost drift.

If the graph trends downward toward 128 KB, the bucket is likely accumulating fragmented artifacts faster than storage volume is growing.

Step 2 — Verify lifecycle eligibility

Average object size tells you that fragmentation exists.

The next step is determining whether lifecycle remediation will actually work. This requires analysing the distribution of object sizes.

Why this matters:

Many Glacier storage classes do not transition objects smaller than 128 KB by default (but can be explicitly configured to do so).

If most objects fall below this threshold, either lifecycle transitions won’t be triggered with default behaviour, or risk incurring large costs if configured for smaller object sizes.

Example Athena query

Using S3 Inventory:

SELECT
  CASE
    WHEN size < 32768 THEN ‘<32KB’
    WHEN size < 131072 THEN ‘32KB–128KB’
    WHEN size < 1048576 THEN ‘128KB–1MB’
    ELSE ‘>1MB’
  END AS size_bucket,
  COUNT(*) AS object_count
FROM s3_inventory_table
GROUP BY 1
ORDER BY 1;

Example output:

<32 KB        : 420B objects
32KB–128KB    : 230B objects
128KB–1MB     : 50B objects
>1MB          : 20B objects

Interpretation:

→ ~650B objects below 128 KB
→ lifecycle transitions will be largely ineffective

In this scenario, lifecycle transitions would generate massive transition costs while producing minimal storage savings.

Monitoring

Once the issue is diagnosed, implement continuous monitoring. The same average object size metric applies.

Tracking this value over time provides an early signal of fragmentation long before cost increases appear on billing dashboards.

Alerting

A practical alert should trigger when: AverageObjectSize < 128 KB for three consecutive days.

Requiring multiple evaluation periods avoids alerts caused by temporary ingestion spikes or batch jobs.

AWS CDK example

const bucketName = 'your-bucket-name';

// Core S3 metrics
const objects = new cloudwatch.Metric({
  namespace: 'AWS/S3',
  metricName: 'NumberOfObjects',
  dimensionsMap: { BucketName: bucketName, StorageType: 'AllStorageTypes' },
  statistic: 'Sum',
  period: Duration.days(1),
});

const bytes = new cloudwatch.Metric({
  namespace: 'AWS/S3',
  metricName: 'BucketSizeBytes',
  dimensionsMap: { BucketName: bucketName, StorageType: 'StandardStorage' },
  statistic: 'Maximum',
  period: Duration.days(1),
});

// Average object size = bytes / objects
const avgSize = new cloudwatch.MathExpression({
  expression: 'bytes / objects',
  usingMetrics: { bytes, objects },
  label: 'avg object size (bytes)',
});

// Alert if <128 KB for 3 days
new cloudwatch.Alarm(this, 'S3CardinalityAlarm', {
  metric: avgSize,
  threshold: 128 * 1024,
  evaluationPeriods: 3,
  datapointsToAlarm: 3,
  comparisonOperator: cloudwatch.ComparisonOperator.LESS_THAN_THRESHOLD,
});

Remediation

Once fragmentation is confirmed, remediation must be planned carefully. At large scale, fixing the dataset can itself be expensive.

Step 1 — Estimate lifecycle transition cost

Lifecycle transitions are priced per 1,000 objects. Estimate remediation cost before enabling transitions:

Example:

objects = 720_000_000_000
price_per_1000 = 0.01

transition_cost = (objects / 1000) * price_per_1000
print(f”${transition_cost:,.0f}”)

Output:

This calculation prevented a multi-million-dollar lifecycle event in the system analysed in the previous article.

Step 2 — Execute remediation safely

Avoid custom scripts for billion-object operations.

Instead use S3 Batch Operations, which provides:

• automatic parallelisation

• retry handling

• distributed execution across AWS infrastructure

Batch Operations is designed specifically for large-scale object-level changes.

Prevention

If the system is still evolving, implement patterns to prevent future cardinality explosions.

Aggregate small artifacts

Bad pattern: request → write 200 objects

Better pattern: request → buffer → write 1 aggregated object

If artifacts are generated in the KB range, introduce a buffering layer before writing to S3.

Design operational prefixes

Structure buckets around operational boundaries.

Example:

s3://bucket/
  service-a/
  service-b/
  telemetry/
  snapshots/

Benefits:

• targeted lifecycle rules

• efficient Athena queries

• easier remediation

Prefer metadata for read-time inspection

If object attributes must be inspected during reads: Use object metadata.

Use tags primarily for:

• lifecycle rules

• governance policies

Metadata avoids additional API calls when reading large numbers of objects.

Operational Checklist

When S3 costs spike unexpectedly

1️⃣ Check object count
2️⃣ Compute average object size
3️⃣ If <128 KB → fragmentation detected
4️⃣ Verify lifecycle eligibility (size distribution)
5️⃣ Estimate remediation cost
6️⃣ Use Batch Operations for large-scale fixes

Conclusion

The system analysed in the previous article scaled perfectly. It simply became economically unstable.

At petabyte scale, object storage is no longer purely a storage problem. It becomes a cardinality management problem.

Treat object count as a first-class operational metric.

Otherwise, a perfectly functioning system can quietly drift into a multi-million-dollar bill.

An alarm fires: SQS message delay > 5 minutes.

A batch job somewhere in the system just spiked the queue from 100k messages per minute to 1 million. Your workers are auto scaling, but it takes 10–15 minutes to catch up.

Nothing is broken. The processing rate is exactly what it was five minutes ago—stable, efficient, and maxed out. Nothing needs fixing. But someone once decided that a five-minute delay is worthy of waking another human being.

You acknowledge the alarm. You watch the queue drain. After all, you cannot make the auto-scaling go any faster, and messages will remain delayed until the ingestion rate catches up to the new queue size. You are there simply to witness the inevitable. Ten minutes later, the system heals itself.

You go back to sleep — or try to — and spend the next 45 minutes staring at the ceiling while the rest of tomorrow quietly collapses.

After enough nights like this, a certain mindset begins to form.

You start acknowledging alarms with less urgency. You observe the system more than you intervene in it. You learn to distinguish between problems that require action and problems that simply require time. Usually, the system corrects itself.

Eventually you arrive at a quiet realisation:

Many pages exist not because the system is failing, but because someone once mistook temporary discomfort for catastrophe.

This is what I call a nihilist on-call.

Not laziness. Not negligence. Just a gradual understanding that many pages do not correspond to real failures, and that intervention often changes little. You watch the system. You wait. And most of the time, it fixes itself.

The Heat Death of the Service

Spend enough time on-call and another thought begins to surface. Most services do not fail catastrophically. They decay.

Dependencies drift. Dashboards fall out of date. Runbooks turn into archaeological artifacts documenting systems that no longer exist. This is operational entropy, and given enough time, every sufficiently complex service approaches its own version of heat death.

This is where the nihilism takes root.

When you realize the service is in a state of slow-motion decay, the urgency of the pager starts to feel like a lie. You arrive at the nihilistic realization that you aren’t “saving” the system; you are merely performing an act of penance for a design you didn’t choose. Many alerts imply that the system is moments away from collapse, but the reality is a slow, gray fade into obsolescence.

But heat death is only inevitable if you accept the role of the bystander.

The sense of futility is a protective layer of scar tissue, but it is also a choice. We treat entropy as a law of nature, but in software, entropy is a choice of priority. Most pages are not preventing the heat death of the service because they focus on the symptoms of the decay rather than the decay itself. To move past this futility, we have to stop treating the pager as a death knell and start using it as a diagnostic tool for restoration.

Signal Dilution

There is another consequence of this dynamic that only becomes obvious at scale. When a system produces enough meaningless pages, the meaningful ones begin to dissolve into the noise.

The pager still rings. The alarm still says SEV-2. But psychologically it carries far less weight than it should. Not because engineers have become careless, but because the system itself has trained them to treat alerts with skepticism.

Alarm inflation is almost inevitable in large organizations. Every team adds alerts intended to protect their own services. Very few alarms are ever removed. Self-healing systems page simply because a metric briefly crossed an arbitrary threshold. Over time the system begins to generate a constant background hum of operational noise.

This is the real danger of alarm fatigue. Reliability is rarely destroyed by a single catastrophic failure. Much more often it is eroded gradually by a loss of trust in the signals meant to protect the system. The fastest way to make engineers ignore an alarm is to page them repeatedly for events that resolve themselves.

The Myth of the Pager

In the myth of Sisyphus, a man is condemned to push a boulder up a hill forever, only for it to roll back down each time he reaches the top.

On-call sometimes feels similar. The pager rings. You acknowledge the alarm. You click through dashboards. The system stabilizes. And somewhere in the background, another alert is already preparing to wake you tomorrow night.

But unlike Sisyphus, engineers are not actually condemned to this cycle. Most on-call pain is not inevitable. It is designed.

To cure the futility of the on-call, we should refuse absurd labor. We should stop engineering better ways to push the boulder and start questioning why it exists at all.

Practical Rules for On-Call

No Human Intervention, No Page.

If an alert fires and the resolution is simply “wait for it to clear” or “restart the service,” the rock has rolled back to the bottom.

The Rule: If a human doesn’t need to make a unique, creative decision to fix it, a computer should be doing it. Do not wake a human for a task a script could do.

Page on Symptoms, Not Causes.

We should let the system decay in silence if that decay doesn’t hurt the user.

The Rule: High CPU is a cause, not a symptom. If your system is still serving requests and users aren’t experiencing errors or noticeable latency, the pager should stay silent. Only wake someone up when the system is actually broken.

Delete the “Flappy” Alerts.

Every recurring alert that you “acknowledge and ignore” is a high-priority bug in your monitoring system.

The Rule: If an alert fires three times in a shift and requires no action, delete it. Don’t “tune” it. Kill it. If the system doesn’t break when the alert is gone, it was never an alert. It was noise.

Protect the Sleep of Others.

High-signal hygiene is a collective pact.

The Rule: Before you create an alarm, ask yourself: “Am I willing to wake up my best friend at 3:00 AM for this?” If the answer is no, it shouldn’t be pageable.

Conclusion

We should refuse to treat the pager as a tool of penance. The struggle itself toward the heights is enough to fill a man’s heart, but only if the heights actually exist.

If the system is going to reach its heat death anyway, we might as well get some sleep.

This article’s code specifically is available at https://github.com/adriangabardo/skymesh/releases/tag/v3.0.1

Overview

In previous articles in the series, we described the abstract idea of what we were trying to achieve, then laid the foundations for ingesting aviation data (airports, airlines, planes) and representing it as a graph model.

The graph structure is a great starting point, but it is ultimately just a way to organise data. On its own, it does not do anything. In this article, we take the next step and give the graph a concrete, real-world use case: building a simple flight search engine.

The goal is deliberately modest. Given an origin airport X and a destination airport Y, we want to enumerate all reasonable flight routes that connect the two. Not the cheapest routes, not the fastest routes, and not even the “best” routes - just routes that make structural sense in the context of the network.

Later in the series, we will introduce additional constraints such as travel dates, maximum journey duration, and mock cost models. And ultimately, we will use live data for these API calls. For now, we focus on the most fundamental capability: turning a static graph into a system that can answer questions.

What “Flight Search” Means at This Stage

At this stage in the project, “flight search” has a very specific and deliberately limited meaning.

We are not trying to determine the cheapest, fastest, or best route. Instead, we are answering a simpler question:

Given an origin and a destination, which routes through the network are reasonable enough to consider at all?

In a large aviation graph, the number of possible paths grows extremely quickly. Without constraints, a traversal will happily return routes that are technically valid but completely unrealistic from a human perspective.

To keep the search grounded, the implementation applies a small set of explicit constraints.

The caller can control:

• the minimum number of routes to return

• the maximum number of routes to return

• the maximum number of legs per route

In addition, the system enforces a hard upper bound on the number of legs, regardless of user input. This prevents the search from blowing up in dense parts of the graph.

Together, these constraints ensure that direct routes are discovered first, multi-stop routes are explored gradually, and the result set stays small and usable.

Further constraints - such as minimum layover times and maximum total travel duration - are intentionally deferred and will be introduced later in the series, once this foundational search behaviour is in place.

The Final Version of the Search Algorithm

Without diving further into depth-first search (DFS), which is well documented by others, the figure below illustrates the order in which routes are explored and where branches are discarded.

When told to find up to 3 routes, of up to 2 legs away from the starting node, the traversal will do the following:

• Start in SYD, visit MEL, find SYD→MEL as a direct route

• Start in SYD, visit ADL, visit MEL, find SYD→ADL→MEL as a route within the constraints

• Start in SYD, visit AKL, then:

  • Visit ZQN, discard the route as we are already 2 legs away and haven’t reached the desired destination

  • Visit MEL, find SYD→AKL→MEL as a route within the constraints

The final version of the search algorithm is designed to produce sensible results without introducing any notion of optimisation or ranking.

It follows a few simple principles:

• routes are discovered in order of increasing number of legs

• the search stops as soon as enough routes are found

• the total number of routes returned is capped

• cycles are explicitly disallowed

The implementation lives in a single function, find_flight_routes(). Given an origin, a destination, and a small set of bounds, it returns a list of FlightRoute objects representing candidate routes through the network.

Rather than searching “up to N legs” in one pass, the algorithm iterates over leg counts and performs a constrained depth-first search for each one. Direct routes are explored first, followed by one-stop routes, then longer connections if required.

At a high level, the algorithm looks like this:

for legs in range(1, max_legs + 1):
    run DFS that only accepts routes with exactly `legs` hops
    collect routes that reach the destination

    if enough routes have been found:
        stop searching

    if hard maximum route count reached:
        stop immediately

This approach keeps the traversal predictable and prevents longer, lower-quality routes from overwhelming shorter and more obvious ones.

The search algorithm as presented in this article is available here: https://github.com/adriangabardo/skymesh/blob/v3.0.1/src/services/path_finder.py

Observing the Algorithm in Practice

With the implementation in place, the most useful thing to do is simply run it and inspect the output.

By running the search engine with different combinations of origin and destination airports, we can see how the algorithm behaves as the network changes. Direct routes appear first, followed by one-stop routes, then longer connections only when necessary.

$ ./run_local.sh
Skymesh graph loaded
Airports (nodes): 6072
Routes (edges): 37042

============================================================
Searching routes: SYD -> MEL
Search for flight routes complete.
SYD -> MEL (1 legs)
SYD -> ADL -> MEL (2 legs)
SYD -> AKL -> MEL (2 legs)


============================================================
Searching routes: YYC -> SYD
Search for flight routes complete.
YYC -> LAX -> SYD (2 legs)
YYC -> NRT -> SYD (2 legs)
YYC -> SFO -> SYD (2 legs)


============================================================
Searching routes: LHR -> JFK
Search for flight routes complete.
LHR -> JFK (1 legs)
LHR -> TXL -> JFK (2 legs)
LHR -> DEL -> JFK (2 legs)


============================================================
Searching routes: CDG -> DXB
Search for flight routes complete.
CDG -> DXB (1 legs)
CDG -> HAM -> DXB (2 legs)
CDG -> IST -> DXB (2 legs)

What’s Next

At this point, we have a working flight search engine in the most literal sense. Given an origin and a destination, the system can traverse the aviation graph and return a small, sensible set of candidate routes.

The next set of challenges are no longer about correctness, but about cost of computation.

So far, route discovery has been relatively cheap. Routes are purely structural, and each candidate can be evaluated with minimal work. That will change quickly as we start layering real-world constraints on top of the search.

In the next articles in the series, we will deliberately make the search engine heavier.

First, we’ll introduce temporal constraints. Routes will become time-aware paths with departure and arrival times, minimum connection windows, and an overall travel duration. Each candidate route will require additional validation, and many structurally valid routes will be rejected late in the process.

Next, we’ll introduce mock pricing and cost functions. Instead of simply enumerating routes, the system will start attaching weights to them, allowing us to reason about trade-offs between different options. At that point, route evaluation stops being trivial and becomes meaningfully expensive.

Only once this additional complexity is in place will we turn our attention to performance.

By first adding computationally heavy features and only then optimising, we get a clear before-and-after comparison. We can observe where the system slows down, which parts of the algorithm dominate runtime, and which optimisations actually matter.

That sets the stage for the next phase of the series: improving performance through caching, memoisation, and selective precomputation, without changing the core traversal logic.

Before moving forward, however, it’s worth reflecting on how this implementation evolved and why some of the earlier, more naive approaches fell short.

Lessons Along the Way

The Naive First Attempt

The first version of the search algorithm was a straightforward depth-first search with a maximum depth.

Starting from the origin airport, the DFS would explore outgoing edges, recursively expand paths, stop once a given number of legs was reached, and record any path that ended at the destination.

From a correctness standpoint, this worked. All valid paths up to the maximum number of legs were found.

In practice, however, the output quickly became unusable.

As a concrete example, querying a simple pair like SYD → MEL - which has plenty of direct connectivity - returned 45 different routes within a four-leg limit. Most of these routes were technically valid graph paths, but many of them went around the globe before eventually reaching Melbourne.

From the algorithm’s point of view, this behaviour was expected. A depth-first search with a depth limit will happily enumerate every path that fits the constraint. From a user’s point of view, however, the result was clearly not useful.

Rethinking the Search Strategy

The problem was not that DFS was the wrong tool, but that the search strategy was too permissive.

A depth limit alone does not meaningfully capture what “reasonable” means in the context of flight search. Treating all paths up to a given depth as equally interesting allows long, low-quality routes to drown out shorter and more obvious ones.

The key shift was to stop thinking in terms of “up to N legs” and instead search in order of increasing complexity. Shorter routes should always be discovered first, and longer routes should only be considered when necessary.

Bounding the Search in Practice

This change led naturally to the current implementation, which combines progressive deepening with a small set of hard bounds.

By enforcing:

• a capped maximum number of legs

• a minimum number of routes before early termination

• a hard maximum on returned routes

the search becomes predictable and tractable, even in dense parts of the network.

These bounds are not optimisations in the traditional sense. They are structural limits that define the shape of the search space and keep the system aligned with real-world expectations.

This article’s code specifically is available at https://github.com/adriangabardo/skymesh/releases/tag/v2.0.0

Overview

The first article in The Graph Series framed the aviation industry as a graph problem: airports as nodes, flights as edges, and routing as constrained path optimisation. This article turns that abstraction into something concrete.

Here we focus on project setup, data ingestion, and domain modelling - the unglamorous but decisive groundwork that determines whether graph algorithms remain elegant on paper or survive contact with real data. Before any shortest paths, cost functions, or optimisations can exist, the graph must be constructed correctly, consistently, and with an understanding of its limitations.

We will walk through how raw aviation datasets - airports, routes, schedules, and metadata - are transformed into a graph-ready representation. This includes decisions around node identity, edge directionality, temporal attributes, and how much of the real world to encode upfront versus defer to later computation. These choices directly affect correctness, performance, and extensibility in later stages of the system.

This article also introduces the data ingestion pipeline that underpins the rest of the series: how data is sourced, normalised, validated, and loaded in a way that supports iterative experimentation. The goal is not just to build a graph, but to build one that can evolve - supporting recalculation, enrichment, and re-modelling without collapsing under its own assumptions.

By the end of this article, we will have a working, queryable graph representation of the aviation network. It will be intentionally incomplete in terms of optimisation and routing intelligence - but structurally sound enough to support everything that follows: pathfinding algorithms, memoisation strategies, pre-computation, and dynamic updates.

This is the foundation. Every optimisation in later articles either benefits from, or is constrained by, the choices made here.

Project Structure and Separation of Concerns

At this point, we have started the implementation of the foundations of the project. I have given it a name, Skymesh, simply to make it easier to reference from here onwards. Right now, the project is intentionally small.

The goal at this stage is not to solve routing problems or optimise anything yet. It is to put a real system in place that we can build on incrementally. That means having a concrete codebase, real data, and something we can execute, inspect, and reason about.

What follows is a walkthrough of what has been implemented so far, starting from raw data acquisition and ending with a working, inspectable graph.

Data Gathering

Skymesh uses the OpenFlights dataset as its initial data source. Rather than pulling data dynamically or wrapping an API, the decision here is to work with static, versioned input files. This makes experimentation reproducible and keeps ingestion simple.

The OpenFlights data lives in a public GitHub repository and is provided as a set of flat .dat files. Each file represents a different part of the aviation domain, such as airports, routes, airlines, and aircraft.

The files are downloaded directly into a local data/ directory. The data sets have been downloaded with curl as follows:

$ curl -L -o airports.dat   https://raw.githubusercontent.com/jpatokal/openflights/master/data/airports.dat

$ curl -L -o routes.dat https://raw.githubusercontent.com/jpatokal/openflights/master/data/routes.dat

$ curl -L -o airlines.dat https://raw.githubusercontent.com/jpatokal/openflights/master/data/airlines.dat

$ curl -L -o planes.dat https://raw.githubusercontent.com/jpatokal/openflights/master/data/planes.dat

$ curl -L -o countries.dat https://raw.githubusercontent.com/jpatokal/openflights/master/data/countries.dat

Once downloaded, the directory looks roughly like this:

$ tree ./data/
./data/
├── airlines.dat
├── airports.dat
├── countries.dat
├── planes.dat
└── routes.dat

1 directory, 5 files

At this stage, no preprocessing or cleaning is performed. The data is consumed in its raw form so that modelling decisions remain explicit in the code rather than hidden in one-off scripts.

Project Layout and Separation of Concerns

With the data in place, the implementation itself lives under the src/ directory:

$ tree ./src/
./src/
├── graph_build.py
├── graph_viz.py
└── main.py

1 directory, 3 files

Each file has their own responsibility.

• graph_build.py contains all logic related to data ingestion and graph construction

• graph_viz.py contains utilities for inspecting the graph visually

• main.py acts as the entry point and orchestration layer

This separation is deliberate. Graph construction should not depend on visualisation, and visualisation should not be required for the graph to exist. Keeping these concerns isolated makes the code easier to reason about and easier to extend later.

At this stage, the structure may feel slightly heavier than necessary, but this pays off once optimisation, caching, or alternative graph backends are introduced.

Graph Initialisation

The core of the system lives in graph_build.py. This is where raw OpenFlights data is turned into a graph structure.

Graph construction begins by initialising a directed graph using NetworkX:

graph = nx.DiGraph()

Airports are ingested first. Each row in airports.dat is parsed, validated, and turned into a node in the graph. Only airports with a valid IATA code are included.

Routes are ingested next. Each route creates a directed edge from a source airport to a destination airport, but only if both airports already exist in the graph. This avoids implicit node creation and makes ingestion deterministic.

All of this logic is wrapped in a single function:

def build_graph() -> nx.DiGraph:
    graph = nx.DiGraph()
    load_airports(graph)
    load_routes(graph)
    return graph

Running the project at this point constructs the full aviation graph and prints some basic diagnostics:

$ python src/main.py
Skymesh graph loaded
Airports (nodes): 3366
Routes (edges): 67663

Sample airport:
GKA {
    "name": "Goroka Airport",
    "city": "Goroka",
    "country": "Papua New Guinea",
    "icao": "AYGA",
    "latitude": -6.081689834590001,
    "longitude": 145.391998291,
    "altitude": 5282,
    "timezone": "10"
}

Sample route:
('GKA', 'HGU') {
    "airline": "CG",
    "airline_id": "1308",
    "codeshare": false,
    "stops": 0,
    "equipment": [
        "DH8",
        "DHT"
    ]
}

Graph Visualisation

Attempting to visualise the entire graph immediately is neither practical nor especially helpful. We are working with thousands of nodes and tens of thousands of edges, and a naive render quickly turns into an unreadable cluster.

Instead, graph_viz.py provides a constrained visualisation focused on the most connected airports. We extract a hub-centric subgraph and project it directly onto real geographic coordinates. Because latitude and longitude were ingested as node attributes earlier, we can render the graph against an actual cartographic background rather than relying on an artificial layout algorithm.

With the rendering layered on top of a cartographic background, we can now visualise how our graph structure connects real-world airports based on the modelling decisions we made earlier. What was previously an abstract network of nodes and edges now maps directly onto the physical world. We can see transatlantic arcs forming naturally, dense European clusters emerging around major hubs, and the strong east–west connectivity across North America. This visual gives us confidence that the data modelling choices were sound.

We intentionally limit the visualisation to a subset of hub airports. Rendering the entire network would obscure structure rather than clarify it. At this stage, our goal is not completeness but coherence. We want to ensure that the foundation we have built is structurally correct before we begin asking more demanding questions of it.

Data Modelling Decisions

Now that we have a breakdown of the implementation so far, lets step back and talk about the modelling decisions that shaped the graph.

Node Identity

OpenFlights provides multiple identifiers for airports, including numeric IDs, ICAO codes, and IATA codes. Skymesh uses IATA codes as node identifiers.

This is a deliberate trade-off. IATA codes are human-readable, widely used, and make the graph much easier to inspect and debug. A path such as LHR → JFK → LAX is immediately meaningful.

The downside is that some airports do not have IATA codes and are therefore excluded. At this stage, Skymesh optimises for clarity and interoperability rather than exhaustive coverage.

Nodes as Data Carriers

Nodes in Skymesh are not just identifiers. Each airport node carries metadata such as geographic coordinates, country, and timezone.

Some of this information is not used immediately. It is ingested early to preserve optionality. Latitude and longitude, for example, will later enable distance calculations and spatial heuristics without requiring a second ingestion pass.

Directionality

Routes are modelled as directed edges. This reflects the reality of aviation networks, where routes are not necessarily symmetric. Treating the graph as undirected would simplify the structure, but it would also introduce incorrect assumptions that would surface later during routing and optimisation.

At this stage, edges are unweighted. Cost functions and constraints are intentionally deferred to the next article.

What’s Next

At this point, Skymesh has a structurally sound representation of the aviation network. We can ingest real data, construct a directed graph with meaningful identifiers, and perform basic inspection to verify that the model matches our expectations.

What we do not yet have is any notion of cost.

All routes are currently treated as equal. There is no concept of distance, time, price, feasibility, or optimisation beyond the existence of a path. This is intentional. Before introducing algorithms, it is important that the underlying graph is trustworthy and easy to reason about.

In the next article, the focus will shift from construction to computation. We will begin asking questions of the graph rather than just building it. That includes introducing pathfinding algorithms, defining cost functions, and exploring why naive shortest-path approaches quickly become insufficient in real-world networks.

In this article, we introduce a project that models the global aviation industry as a graph-theory problem. Airports are represented as nodes and direct flight routes as edges, forming a large, sparse, and highly non-uniform network.

The objective of this project is to explore how common aviation questions - such as route reachability, optimal paths between airports, and network-level efficiency - can be expressed as graph computations. As the project evolves, we will progressively introduce increasingly realistic constraints and cost functions, and examine how these affect both correctness and computational performance.

This article serves as the foundation for The Graph Series. Subsequent articles will build on this model to investigate optimisation techniques for graph traversal and path-finding, including algorithmic trade-offs, memoisation strategies, and performance improvements at scale.

The Aviation Industry

The aviation industry is broad and never still. On any given day, tens of thousands of commercial flights operate worldwide, connecting several thousand passenger airports and transporting millions of people across the globe. Beyond leisure travel, aviation underpins a wide range of other sectors — corporate travel, cargo and retail logistics, emergency services, and military operations, to name a few.

Despite this breadth, the scope of this series is intentionally narrow. For the purposes of this project, the focus is limited to leisure-oriented passenger flights, and specifically to the structure of the global flight network itself. Cargo operations, private aviation, and military routes are treated as out of scope.

Even within commercial passenger aviation, there is an overwhelming number of variables that could be modelled. Airlines operate different fleets with varying ranges and capacities, routes are constrained by aircraft performance and permitted airspace, and hub airports play an outsized role in shaping global connectivity. Additional factors such as weather, crew availability, regulatory constraints, and geopolitical considerations further complicate the picture.

This project does not attempt to model all of these factors upfront. Instead, it treats the aviation industry as a layered system: starting with the existence of routes between airports, then progressively introducing additional dimensions such as airline-specific routes, aircraft constraints, and — later in the series — temporal availability and scheduling. This allows individual modelling choices to be examined in isolation, while still grounding the work in a recognisably real-world system.

By clearly defining which parts of the aviation industry are being simulated, and which are intentionally ignored, we can focus on the graph problems themselves without losing sight of the domain they are inspired by.

Methodology

For this project, I have picked Python as the language of choice. The graph itself is implemented using NetworkX, and the underlying data is sourced from OpenFlights, which provides publicly available datasets covering airports, airlines, and direct flight routes.

The ingestion process focuses on building a clean and extensible representation of the aviation network. Airports are mapped to graph nodes, while direct routes between airports are represented as directed edges. At this stage, the emphasis is on establishing a structurally correct graph that can be easily extended with additional attributes and constraints in later iterations of the project.

Whilst the project is in active development, I plan to interact directly with the graph through a simple __init__.py entry point that exposes the graph as a first-class object. This allows for quick experimentation, ad-hoc inspection, and iterative refinement during development. Once the project reaches the benchmarking stages, interaction with the graph will shift to scripted, reproducible workflows designed to generate consistent and comparable performance measurements over time.

Alongside the core graph construction, a lightweight visualisation setup is introduced to generate visual representations of the network. These visualisations are not intended to be exhaustive or perfectly scaled, but rather to provide intuition around graph structure, connectivity, and the emergence of hubs within the aviation network. They also serve as a useful sanity check during development and a visual aid when discussing results later in the series.

This methodological foundation is intentionally kept simple. As the series progresses, the same setup will be reused to explore alternative cost functions, traversal strategies, and optimisation techniques, without changing the underlying data source or tooling.

Assumptions & Simplifications

It goes without saying that the aviation industry is far more complex than the features implemented in this project. This is a deliberately simplified, real-world-inspired example that allows us to explore graph representations, calculations, and optimisation techniques in a concrete setting.

The following assumptions and simplifications are made as part of this exercise:

• Direct flights only
  Only direct flight routes are represented. Multi-leg journeys are expressed implicitly through graph traversal.

• Single edge per route (initially)
  Routes between two airports are represented by a single directed edge, independent of airline or aircraft. Later in the project, this will be expanded into a multi-edge model to capture airline, aircraft, and other route-level properties.

• No temporal dimension (initially)
  The initial graph is static. Routes represent existence, not schedule or availability. Temporal constraints and availability will be introduced after the transition to multi-edge routes.

• Uniform edge behaviour
  All edges are treated equivalently at this stage. Attributes such as cost, duration, or reliability are deferred to later cost functions.

• No capacity or congestion modelling
  Airports and routes are assumed to have unlimited capacity. Operational constraints such as congestion or delays are out of scope.

• Airports as atomic nodes
  Each airport is represented as a single node, without modelling internal structure.

• Dataset limitations
  Route data is sourced from OpenFlights, which relies on a third-party provider that ceased updates in June 2014. As a result, the routes dataset is of historical value only. The other datasets (airports, airlines) appear to be maintained and are treated as current for the purposes of this project.
  As of June 2014, the routes dataset contains 67,663 routes connecting 3,321 airports across 548 airlines worldwide, which is sufficient for structural analysis and optimisation experiments.

These assumptions define the baseline model used in the early stages of the series and will be relaxed incrementally as additional complexity is introduced.

What’s next

Following on from this project abstract, we will look into the initial project’s setup including the necessary data ingestion and modelling, followed by implementing the essential algorithms we will be experimenting with.

Thanks for reading! Subscribe for free to receive new posts and support my work.