The root cause was not storage volume. It was object cardinality.

This article is the operational companion: a runbook for diagnosing and fixing small-object explosions in production S3 systems.

The Operational Model

Operating S3 at scale follows a simple lifecycle:

Most large S3 cost failures occur because observation is missing.

Quick Triage

When S3 costs increase unexpectedly, start with one simple check.

If this number becomes too small, the system is accumulating fragmented artifacts.

Rule of thumb

> 10 MB     → Healthy
1–10 MB     → Acceptable
< 1 MB      → Fragmentation risk
< 128 KB    → CRITICAL

128 KB matters because:

• by default, lifecycle rules do not apply to objects sized ≥128KB

• some storage classes bill small objects as if they were 128 KB minimum

Once the average object size drops below this boundary, the system shifts into object-dominated pricing behaviour.

Diagnosis

Step 1 — Check bucket metrics

The quickest way to detect a cardinality issue is to compute average object size directly from CloudWatch metrics.

CloudWatch already exposes the required metrics:

• AWS/S3 BucketSizeBytes

• AWS/S3 NumberOfObjects

CloudWatch Dashboard Widget

Paste the following JSON into a CloudWatch dashboard (Source view) and replace the bucket name.

{
  “metrics”: [
    [ “AWS/S3”, “NumberOfObjects”, “BucketName”, “example-bucket”, “StorageType”, “AllStorageTypes”, { “id”: “m1”, “stat”: “Sum”, “label”: “objects”, “visible”: false } ],
    [ “.”, “BucketSizeBytes”, “.”, “.”, “.”, “StandardStorage”, { “id”: “m2”, “yAxis”: “right”, “label”: “size”, “visible”: false, “stat”: “Maximum” } ],
    [ { “expression”: “(m2/m1) / 1024”, “label”: “average object size (KB)”, “id”: “e1” } ]
  ],
  “view”: “timeSeries”,
  “stacked”: false,
  “region”: “us-east-1”,
  “period”: 86400,
  “stat”: “Average”
}

This widget will compute average object size and display it in kilobytes.

This is the single most useful metric for detecting S3 cost drift.

If the graph trends downward toward 128 KB, the bucket is likely accumulating fragmented artifacts faster than storage volume is growing.

Step 2 — Verify lifecycle eligibility

Average object size tells you that fragmentation exists.

The next step is determining whether lifecycle remediation will actually work. This requires analysing the distribution of object sizes.

Why this matters:

Many Glacier storage classes do not transition objects smaller than 128 KB by default (but can be explicitly configured to do so).

If most objects fall below this threshold, either lifecycle transitions won’t be triggered with default behaviour, or risk incurring large costs if configured for smaller object sizes.

Example Athena query

Using S3 Inventory:

SELECT
  CASE
    WHEN size < 32768 THEN ‘<32KB’
    WHEN size < 131072 THEN ‘32KB–128KB’
    WHEN size < 1048576 THEN ‘128KB–1MB’
    ELSE ‘>1MB’
  END AS size_bucket,
  COUNT(*) AS object_count
FROM s3_inventory_table
GROUP BY 1
ORDER BY 1;

Example output:

<32 KB        : 420B objects
32KB–128KB    : 230B objects
128KB–1MB     : 50B objects
>1MB          : 20B objects

Interpretation:

→ ~650B objects below 128 KB
→ lifecycle transitions will be largely ineffective

In this scenario, lifecycle transitions would generate massive transition costs while producing minimal storage savings.

Monitoring

Once the issue is diagnosed, implement continuous monitoring. The same average object size metric applies.

Tracking this value over time provides an early signal of fragmentation long before cost increases appear on billing dashboards.

Alerting

A practical alert should trigger when: AverageObjectSize < 128 KB for three consecutive days.

Requiring multiple evaluation periods avoids alerts caused by temporary ingestion spikes or batch jobs.

AWS CDK example

const bucketName = 'your-bucket-name';

// Core S3 metrics
const objects = new cloudwatch.Metric({
  namespace: 'AWS/S3',
  metricName: 'NumberOfObjects',
  dimensionsMap: { BucketName: bucketName, StorageType: 'AllStorageTypes' },
  statistic: 'Sum',
  period: Duration.days(1),
});

const bytes = new cloudwatch.Metric({
  namespace: 'AWS/S3',
  metricName: 'BucketSizeBytes',
  dimensionsMap: { BucketName: bucketName, StorageType: 'StandardStorage' },
  statistic: 'Maximum',
  period: Duration.days(1),
});

// Average object size = bytes / objects
const avgSize = new cloudwatch.MathExpression({
  expression: 'bytes / objects',
  usingMetrics: { bytes, objects },
  label: 'avg object size (bytes)',
});

// Alert if <128 KB for 3 days
new cloudwatch.Alarm(this, 'S3CardinalityAlarm', {
  metric: avgSize,
  threshold: 128 * 1024,
  evaluationPeriods: 3,
  datapointsToAlarm: 3,
  comparisonOperator: cloudwatch.ComparisonOperator.LESS_THAN_THRESHOLD,
});

Remediation

Once fragmentation is confirmed, remediation must be planned carefully. At large scale, fixing the dataset can itself be expensive.

Step 1 — Estimate lifecycle transition cost

Lifecycle transitions are priced per 1,000 objects. Estimate remediation cost before enabling transitions:

Example:

objects = 720_000_000_000
price_per_1000 = 0.01

transition_cost = (objects / 1000) * price_per_1000
print(f”${transition_cost:,.0f}”)

Output:

This calculation prevented a multi-million-dollar lifecycle event in the system analysed in the previous article.

Step 2 — Execute remediation safely

Avoid custom scripts for billion-object operations.

Instead use S3 Batch Operations, which provides:

• automatic parallelisation

• retry handling

• distributed execution across AWS infrastructure

Batch Operations is designed specifically for large-scale object-level changes.

Prevention

If the system is still evolving, implement patterns to prevent future cardinality explosions.

Aggregate small artifacts

Bad pattern: request → write 200 objects

Better pattern: request → buffer → write 1 aggregated object

If artifacts are generated in the KB range, introduce a buffering layer before writing to S3.

Design operational prefixes

Structure buckets around operational boundaries.

Example:

s3://bucket/
  service-a/
  service-b/
  telemetry/
  snapshots/

Benefits:

• targeted lifecycle rules

• efficient Athena queries

• easier remediation

Prefer metadata for read-time inspection

If object attributes must be inspected during reads: Use object metadata.

Use tags primarily for:

• lifecycle rules

• governance policies

Metadata avoids additional API calls when reading large numbers of objects.

Operational Checklist

When S3 costs spike unexpectedly

1️⃣ Check object count
2️⃣ Compute average object size
3️⃣ If <128 KB → fragmentation detected
4️⃣ Verify lifecycle eligibility (size distribution)
5️⃣ Estimate remediation cost
6️⃣ Use Batch Operations for large-scale fixes

Conclusion

The system analysed in the previous article scaled perfectly. It simply became economically unstable.

At petabyte scale, object storage is no longer purely a storage problem. It becomes a cardinality management problem.

Treat object count as a first-class operational metric.

Otherwise, a perfectly functioning system can quietly drift into a multi-million-dollar bill.

This article analyses a production system that accumulated 5.6 PB of data across 1.56 trillion objects in a single bucket. Within one year, monthly storage cost increased from approximately $100k to over $400k, with forecasts exceeding $1M per month. The root cause was not data volume alone, but architectural fragmentation misaligned with S3’s pricing model.

By consolidating objects to align with S3’s economic structure, equivalent logical data could have been stored at up to 37× lower monthly cost. This case demonstrates that cost modelling must be treated as an architectural constraint.

A More Accurate AWS S3 Cost Model Representation

Object storage is frequently approximated as purely volumetric:

Where:

• C = monthly cost

• V = stored volume

This mental model is incomplete. A more accurate representation of S3 pricing is:

Where:

• C = Total monthly S3 cost

• V = Total stored volume (GB)

• Pgb​ = Price per GB-month of storage

• Nreq​​ = Total request count (PUT / GET / LIST)

• Preq​​ = Price per request

• sˉ = Average object size (GB)

• P1000​ = Price per 1,000 object-level operations (e.g., lifecycle transitions)

At small scale, object count is negligible relative to volume but at larger scales, it becomes a first-order variable. The structural quantity that determines whether object count matters is average object size:

Where:

• N = total object count

Inversely, we can express object count in terms of total volume and average object size:

This allows us to restate the object-driven components of the cost model directly in terms of architectural granularity. For a fixed total volume, reducing average object size necessarily increases object count — and therefore amplifies any per-object pricing terms.

We can now use the following cost equation to calculate total cost explicitly as a function of average object size:

This formulation makes object granularity a first-class variable in the cost model. Rather than treating object count as an opaque operational metric, it becomes a direct consequence of architectural design.

When sˉ falls into the kilobyte regime, per-object pricing dominates. The system no longer behaves like bulk storage, instead it’s more akin to a massively distributed index - except you are paying storage-layer economics for index-layer behavior.

A Real-World Example

In the system analyzed:

• 5.6 PB stored

• 1.56 trillion objects

• ~3.5 KB average object size

• $400k/month storage cost

• ~$50k/month in request charges

The architecture generated hundreds of small snapshot artifacts per back end request. Over time, fragmentation compounded. Object count grew faster than volume.

An empirical consolidation experiment showed that equivalent logical data could be stored in artifacts averaging ~2.5 MB.

This would have reduced storage cost from $457,000 to $11,900 per month for the same volume of data. This represents a 37× structural reduction.

This reduction was not due to compression or deletion of data. The total logical volume remained constant. Only object granularity changed.

The $7M Lifecycle Event

Another (nearly) expensive lesson came from trying to fix our exponentially increasing storage costs. Without permanently losing important insights into our service, the only cost saving alternative we believed to have was resorting to lifecycle transition policies.

This solution could have had a seven figure cost, as lifecycle transitions are priced per 1,000 objects. At one point, a single bucket contained approximately 720 billion objects. Meaning:

Approximately $7.2 million — for a single lifecycle rule on a single bucket.

This excludes ongoing storage cost and retrieval penalties.

The transition did not execute because most objects were smaller than 128 KB, which do not transition by default.

Ironically, the same fragmentation pattern that caused excessive steady-state cost also prevented an even larger transition bill.

A Cost Calculation Framework

To prevent similar failures, object storage systems should be evaluated across three explicit budgets.

1. Volume Budget

Projected monthly storage cost.

2. Cardinality Budget

Total object count and average object size.

If average object size falls below a defined threshold (e.g. 1–10 MB for snapshot systems), object count becomes a risk indicator.

3. Remediation Budget

Cost of rewriting, transitioning, or migrating all objects.

Before implementing lifecycle rules or structural migrations, compute:

If remediation cost exceeds acceptable monthly spend, the architecture is already broken.

Object count must be monitored alongside stored bytes. Divergence between the two is architectural drift, not growth.

Conclusion

Our system scaled flawlessly. It simply became unaffordable, under:

• Multi-petabyte scale

• Trillion-object cardinality

• Monthly cost growth from $100k to $400k

• Forecast exceeding $1M/month

• A potential $7M lifecycle event

Object storage is not purely volumetric. It is priced across bytes, objects, and operations.

At extreme scale, pricing semantics become architectural constraints.

Cost modelling must be treated as a first-class design discipline.

Disclaimer: Based on public AWS pricing and production experience. Not an official AWS statement.

An alarm fires: SQS message delay > 5 minutes.

A batch job somewhere in the system just spiked the queue from 100k messages per minute to 1 million. Your workers are auto scaling, but it takes 10–15 minutes to catch up.

Nothing is broken. The processing rate is exactly what it was five minutes ago—stable, efficient, and maxed out. Nothing needs fixing. But someone once decided that a five-minute delay is worthy of waking another human being.

You acknowledge the alarm. You watch the queue drain. After all, you cannot make the auto-scaling go any faster, and messages will remain delayed until the ingestion rate catches up to the new queue size. You are there simply to witness the inevitable. Ten minutes later, the system heals itself.

You go back to sleep — or try to — and spend the next 45 minutes staring at the ceiling while the rest of tomorrow quietly collapses.

After enough nights like this, a certain mindset begins to form.

You start acknowledging alarms with less urgency. You observe the system more than you intervene in it. You learn to distinguish between problems that require action and problems that simply require time. Usually, the system corrects itself.

Eventually you arrive at a quiet realisation:

Many pages exist not because the system is failing, but because someone once mistook temporary discomfort for catastrophe.

This is what I call a nihilist on-call.

Not laziness. Not negligence. Just a gradual understanding that many pages do not correspond to real failures, and that intervention often changes little. You watch the system. You wait. And most of the time, it fixes itself.

The Heat Death of the Service

Spend enough time on-call and another thought begins to surface. Most services do not fail catastrophically. They decay.

Dependencies drift. Dashboards fall out of date. Runbooks turn into archaeological artifacts documenting systems that no longer exist. This is operational entropy, and given enough time, every sufficiently complex service approaches its own version of heat death.

This is where the nihilism takes root.

When you realize the service is in a state of slow-motion decay, the urgency of the pager starts to feel like a lie. You arrive at the nihilistic realization that you aren’t “saving” the system; you are merely performing an act of penance for a design you didn’t choose. Many alerts imply that the system is moments away from collapse, but the reality is a slow, gray fade into obsolescence.

But heat death is only inevitable if you accept the role of the bystander.

The sense of futility is a protective layer of scar tissue, but it is also a choice. We treat entropy as a law of nature, but in software, entropy is a choice of priority. Most pages are not preventing the heat death of the service because they focus on the symptoms of the decay rather than the decay itself. To move past this futility, we have to stop treating the pager as a death knell and start using it as a diagnostic tool for restoration.

Signal Dilution

There is another consequence of this dynamic that only becomes obvious at scale. When a system produces enough meaningless pages, the meaningful ones begin to dissolve into the noise.

The pager still rings. The alarm still says SEV-2. But psychologically it carries far less weight than it should. Not because engineers have become careless, but because the system itself has trained them to treat alerts with skepticism.

Alarm inflation is almost inevitable in large organizations. Every team adds alerts intended to protect their own services. Very few alarms are ever removed. Self-healing systems page simply because a metric briefly crossed an arbitrary threshold. Over time the system begins to generate a constant background hum of operational noise.

This is the real danger of alarm fatigue. Reliability is rarely destroyed by a single catastrophic failure. Much more often it is eroded gradually by a loss of trust in the signals meant to protect the system. The fastest way to make engineers ignore an alarm is to page them repeatedly for events that resolve themselves.

The Myth of the Pager

In the myth of Sisyphus, a man is condemned to push a boulder up a hill forever, only for it to roll back down each time he reaches the top.

On-call sometimes feels similar. The pager rings. You acknowledge the alarm. You click through dashboards. The system stabilizes. And somewhere in the background, another alert is already preparing to wake you tomorrow night.

But unlike Sisyphus, engineers are not actually condemned to this cycle. Most on-call pain is not inevitable. It is designed.

To cure the futility of the on-call, we should refuse absurd labor. We should stop engineering better ways to push the boulder and start questioning why it exists at all.

Practical Rules for On-Call

No Human Intervention, No Page.

If an alert fires and the resolution is simply “wait for it to clear” or “restart the service,” the rock has rolled back to the bottom.

The Rule: If a human doesn’t need to make a unique, creative decision to fix it, a computer should be doing it. Do not wake a human for a task a script could do.

Page on Symptoms, Not Causes.

We should let the system decay in silence if that decay doesn’t hurt the user.

The Rule: High CPU is a cause, not a symptom. If your system is still serving requests and users aren’t experiencing errors or noticeable latency, the pager should stay silent. Only wake someone up when the system is actually broken.

Delete the “Flappy” Alerts.

Every recurring alert that you “acknowledge and ignore” is a high-priority bug in your monitoring system.

The Rule: If an alert fires three times in a shift and requires no action, delete it. Don’t “tune” it. Kill it. If the system doesn’t break when the alert is gone, it was never an alert. It was noise.

Protect the Sleep of Others.

High-signal hygiene is a collective pact.

The Rule: Before you create an alarm, ask yourself: “Am I willing to wake up my best friend at 3:00 AM for this?” If the answer is no, it shouldn’t be pageable.

Conclusion

We should refuse to treat the pager as a tool of penance. The struggle itself toward the heights is enough to fill a man’s heart, but only if the heights actually exist.

If the system is going to reach its heat death anyway, we might as well get some sleep.

This article’s code specifically is available at https://github.com/adriangabardo/skymesh/releases/tag/v3.0.1

Overview

In previous articles in the series, we described the abstract idea of what we were trying to achieve, then laid the foundations for ingesting aviation data (airports, airlines, planes) and representing it as a graph model.

The graph structure is a great starting point, but it is ultimately just a way to organise data. On its own, it does not do anything. In this article, we take the next step and give the graph a concrete, real-world use case: building a simple flight search engine.

The goal is deliberately modest. Given an origin airport X and a destination airport Y, we want to enumerate all reasonable flight routes that connect the two. Not the cheapest routes, not the fastest routes, and not even the “best” routes - just routes that make structural sense in the context of the network.

Later in the series, we will introduce additional constraints such as travel dates, maximum journey duration, and mock cost models. And ultimately, we will use live data for these API calls. For now, we focus on the most fundamental capability: turning a static graph into a system that can answer questions.

What “Flight Search” Means at This Stage

At this stage in the project, “flight search” has a very specific and deliberately limited meaning.

We are not trying to determine the cheapest, fastest, or best route. Instead, we are answering a simpler question:

Given an origin and a destination, which routes through the network are reasonable enough to consider at all?

In a large aviation graph, the number of possible paths grows extremely quickly. Without constraints, a traversal will happily return routes that are technically valid but completely unrealistic from a human perspective.

To keep the search grounded, the implementation applies a small set of explicit constraints.

The caller can control:

• the minimum number of routes to return

• the maximum number of routes to return

• the maximum number of legs per route

In addition, the system enforces a hard upper bound on the number of legs, regardless of user input. This prevents the search from blowing up in dense parts of the graph.

Together, these constraints ensure that direct routes are discovered first, multi-stop routes are explored gradually, and the result set stays small and usable.

Further constraints - such as minimum layover times and maximum total travel duration - are intentionally deferred and will be introduced later in the series, once this foundational search behaviour is in place.

The Final Version of the Search Algorithm

Without diving further into depth-first search (DFS), which is well documented by others, the figure below illustrates the order in which routes are explored and where branches are discarded.

When told to find up to 3 routes, of up to 2 legs away from the starting node, the traversal will do the following:

• Start in SYD, visit MEL, find SYD→MEL as a direct route

• Start in SYD, visit ADL, visit MEL, find SYD→ADL→MEL as a route within the constraints

• Start in SYD, visit AKL, then:

  • Visit ZQN, discard the route as we are already 2 legs away and haven’t reached the desired destination

  • Visit MEL, find SYD→AKL→MEL as a route within the constraints

The final version of the search algorithm is designed to produce sensible results without introducing any notion of optimisation or ranking.

It follows a few simple principles:

• routes are discovered in order of increasing number of legs

• the search stops as soon as enough routes are found

• the total number of routes returned is capped

• cycles are explicitly disallowed

The implementation lives in a single function, find_flight_routes(). Given an origin, a destination, and a small set of bounds, it returns a list of FlightRoute objects representing candidate routes through the network.

Rather than searching “up to N legs” in one pass, the algorithm iterates over leg counts and performs a constrained depth-first search for each one. Direct routes are explored first, followed by one-stop routes, then longer connections if required.

At a high level, the algorithm looks like this:

for legs in range(1, max_legs + 1):
    run DFS that only accepts routes with exactly `legs` hops
    collect routes that reach the destination

    if enough routes have been found:
        stop searching

    if hard maximum route count reached:
        stop immediately

This approach keeps the traversal predictable and prevents longer, lower-quality routes from overwhelming shorter and more obvious ones.

The search algorithm as presented in this article is available here: https://github.com/adriangabardo/skymesh/blob/v3.0.1/src/services/path_finder.py

Observing the Algorithm in Practice

With the implementation in place, the most useful thing to do is simply run it and inspect the output.

By running the search engine with different combinations of origin and destination airports, we can see how the algorithm behaves as the network changes. Direct routes appear first, followed by one-stop routes, then longer connections only when necessary.

$ ./run_local.sh
Skymesh graph loaded
Airports (nodes): 6072
Routes (edges): 37042

============================================================
Searching routes: SYD -> MEL
Search for flight routes complete.
SYD -> MEL (1 legs)
SYD -> ADL -> MEL (2 legs)
SYD -> AKL -> MEL (2 legs)


============================================================
Searching routes: YYC -> SYD
Search for flight routes complete.
YYC -> LAX -> SYD (2 legs)
YYC -> NRT -> SYD (2 legs)
YYC -> SFO -> SYD (2 legs)


============================================================
Searching routes: LHR -> JFK
Search for flight routes complete.
LHR -> JFK (1 legs)
LHR -> TXL -> JFK (2 legs)
LHR -> DEL -> JFK (2 legs)


============================================================
Searching routes: CDG -> DXB
Search for flight routes complete.
CDG -> DXB (1 legs)
CDG -> HAM -> DXB (2 legs)
CDG -> IST -> DXB (2 legs)

What’s Next

At this point, we have a working flight search engine in the most literal sense. Given an origin and a destination, the system can traverse the aviation graph and return a small, sensible set of candidate routes.

The next set of challenges are no longer about correctness, but about cost of computation.

So far, route discovery has been relatively cheap. Routes are purely structural, and each candidate can be evaluated with minimal work. That will change quickly as we start layering real-world constraints on top of the search.

In the next articles in the series, we will deliberately make the search engine heavier.

First, we’ll introduce temporal constraints. Routes will become time-aware paths with departure and arrival times, minimum connection windows, and an overall travel duration. Each candidate route will require additional validation, and many structurally valid routes will be rejected late in the process.

Next, we’ll introduce mock pricing and cost functions. Instead of simply enumerating routes, the system will start attaching weights to them, allowing us to reason about trade-offs between different options. At that point, route evaluation stops being trivial and becomes meaningfully expensive.

Only once this additional complexity is in place will we turn our attention to performance.

By first adding computationally heavy features and only then optimising, we get a clear before-and-after comparison. We can observe where the system slows down, which parts of the algorithm dominate runtime, and which optimisations actually matter.

That sets the stage for the next phase of the series: improving performance through caching, memoisation, and selective precomputation, without changing the core traversal logic.

Before moving forward, however, it’s worth reflecting on how this implementation evolved and why some of the earlier, more naive approaches fell short.

Lessons Along the Way

The Naive First Attempt

The first version of the search algorithm was a straightforward depth-first search with a maximum depth.

Starting from the origin airport, the DFS would explore outgoing edges, recursively expand paths, stop once a given number of legs was reached, and record any path that ended at the destination.

From a correctness standpoint, this worked. All valid paths up to the maximum number of legs were found.

In practice, however, the output quickly became unusable.

As a concrete example, querying a simple pair like SYD → MEL - which has plenty of direct connectivity - returned 45 different routes within a four-leg limit. Most of these routes were technically valid graph paths, but many of them went around the globe before eventually reaching Melbourne.

From the algorithm’s point of view, this behaviour was expected. A depth-first search with a depth limit will happily enumerate every path that fits the constraint. From a user’s point of view, however, the result was clearly not useful.

Rethinking the Search Strategy

The problem was not that DFS was the wrong tool, but that the search strategy was too permissive.

A depth limit alone does not meaningfully capture what “reasonable” means in the context of flight search. Treating all paths up to a given depth as equally interesting allows long, low-quality routes to drown out shorter and more obvious ones.

The key shift was to stop thinking in terms of “up to N legs” and instead search in order of increasing complexity. Shorter routes should always be discovered first, and longer routes should only be considered when necessary.

Bounding the Search in Practice

This change led naturally to the current implementation, which combines progressive deepening with a small set of hard bounds.

By enforcing:

• a capped maximum number of legs

• a minimum number of routes before early termination

• a hard maximum on returned routes

the search becomes predictable and tractable, even in dense parts of the network.

These bounds are not optimisations in the traditional sense. They are structural limits that define the shape of the search space and keep the system aligned with real-world expectations.

This article’s code specifically is available at https://github.com/adriangabardo/skymesh/releases/tag/v2.0.0

Overview

The first article in The Graph Series framed the aviation industry as a graph problem: airports as nodes, flights as edges, and routing as constrained path optimisation. This article turns that abstraction into something concrete.

Here we focus on project setup, data ingestion, and domain modelling - the unglamorous but decisive groundwork that determines whether graph algorithms remain elegant on paper or survive contact with real data. Before any shortest paths, cost functions, or optimisations can exist, the graph must be constructed correctly, consistently, and with an understanding of its limitations.

We will walk through how raw aviation datasets - airports, routes, schedules, and metadata - are transformed into a graph-ready representation. This includes decisions around node identity, edge directionality, temporal attributes, and how much of the real world to encode upfront versus defer to later computation. These choices directly affect correctness, performance, and extensibility in later stages of the system.

This article also introduces the data ingestion pipeline that underpins the rest of the series: how data is sourced, normalised, validated, and loaded in a way that supports iterative experimentation. The goal is not just to build a graph, but to build one that can evolve - supporting recalculation, enrichment, and re-modelling without collapsing under its own assumptions.

By the end of this article, we will have a working, queryable graph representation of the aviation network. It will be intentionally incomplete in terms of optimisation and routing intelligence - but structurally sound enough to support everything that follows: pathfinding algorithms, memoisation strategies, pre-computation, and dynamic updates.

This is the foundation. Every optimisation in later articles either benefits from, or is constrained by, the choices made here.

Project Structure and Separation of Concerns

At this point, we have started the implementation of the foundations of the project. I have given it a name, Skymesh, simply to make it easier to reference from here onwards. Right now, the project is intentionally small.

The goal at this stage is not to solve routing problems or optimise anything yet. It is to put a real system in place that we can build on incrementally. That means having a concrete codebase, real data, and something we can execute, inspect, and reason about.

What follows is a walkthrough of what has been implemented so far, starting from raw data acquisition and ending with a working, inspectable graph.

Data Gathering

Skymesh uses the OpenFlights dataset as its initial data source. Rather than pulling data dynamically or wrapping an API, the decision here is to work with static, versioned input files. This makes experimentation reproducible and keeps ingestion simple.

The OpenFlights data lives in a public GitHub repository and is provided as a set of flat .dat files. Each file represents a different part of the aviation domain, such as airports, routes, airlines, and aircraft.

The files are downloaded directly into a local data/ directory. The data sets have been downloaded with curl as follows:

$ curl -L -o airports.dat   https://raw.githubusercontent.com/jpatokal/openflights/master/data/airports.dat

$ curl -L -o routes.dat https://raw.githubusercontent.com/jpatokal/openflights/master/data/routes.dat

$ curl -L -o airlines.dat https://raw.githubusercontent.com/jpatokal/openflights/master/data/airlines.dat

$ curl -L -o planes.dat https://raw.githubusercontent.com/jpatokal/openflights/master/data/planes.dat

$ curl -L -o countries.dat https://raw.githubusercontent.com/jpatokal/openflights/master/data/countries.dat

Once downloaded, the directory looks roughly like this:

$ tree ./data/
./data/
├── airlines.dat
├── airports.dat
├── countries.dat
├── planes.dat
└── routes.dat

1 directory, 5 files

At this stage, no preprocessing or cleaning is performed. The data is consumed in its raw form so that modelling decisions remain explicit in the code rather than hidden in one-off scripts.

Project Layout and Separation of Concerns

With the data in place, the implementation itself lives under the src/ directory:

$ tree ./src/
./src/
├── graph_build.py
├── graph_viz.py
└── main.py

1 directory, 3 files

Each file has their own responsibility.

• graph_build.py contains all logic related to data ingestion and graph construction

• graph_viz.py contains utilities for inspecting the graph visually

• main.py acts as the entry point and orchestration layer

This separation is deliberate. Graph construction should not depend on visualisation, and visualisation should not be required for the graph to exist. Keeping these concerns isolated makes the code easier to reason about and easier to extend later.

At this stage, the structure may feel slightly heavier than necessary, but this pays off once optimisation, caching, or alternative graph backends are introduced.

Graph Initialisation

The core of the system lives in graph_build.py. This is where raw OpenFlights data is turned into a graph structure.

Graph construction begins by initialising a directed graph using NetworkX:

graph = nx.DiGraph()

Airports are ingested first. Each row in airports.dat is parsed, validated, and turned into a node in the graph. Only airports with a valid IATA code are included.

Routes are ingested next. Each route creates a directed edge from a source airport to a destination airport, but only if both airports already exist in the graph. This avoids implicit node creation and makes ingestion deterministic.

All of this logic is wrapped in a single function:

def build_graph() -> nx.DiGraph:
    graph = nx.DiGraph()
    load_airports(graph)
    load_routes(graph)
    return graph

Running the project at this point constructs the full aviation graph and prints some basic diagnostics:

$ python src/main.py
Skymesh graph loaded
Airports (nodes): 3366
Routes (edges): 67663

Sample airport:
GKA {
    "name": "Goroka Airport",
    "city": "Goroka",
    "country": "Papua New Guinea",
    "icao": "AYGA",
    "latitude": -6.081689834590001,
    "longitude": 145.391998291,
    "altitude": 5282,
    "timezone": "10"
}

Sample route:
('GKA', 'HGU') {
    "airline": "CG",
    "airline_id": "1308",
    "codeshare": false,
    "stops": 0,
    "equipment": [
        "DH8",
        "DHT"
    ]
}

Graph Visualisation

Attempting to visualise the entire graph immediately is neither practical nor especially helpful. We are working with thousands of nodes and tens of thousands of edges, and a naive render quickly turns into an unreadable cluster.

Instead, graph_viz.py provides a constrained visualisation focused on the most connected airports. We extract a hub-centric subgraph and project it directly onto real geographic coordinates. Because latitude and longitude were ingested as node attributes earlier, we can render the graph against an actual cartographic background rather than relying on an artificial layout algorithm.

With the rendering layered on top of a cartographic background, we can now visualise how our graph structure connects real-world airports based on the modelling decisions we made earlier. What was previously an abstract network of nodes and edges now maps directly onto the physical world. We can see transatlantic arcs forming naturally, dense European clusters emerging around major hubs, and the strong east–west connectivity across North America. This visual gives us confidence that the data modelling choices were sound.

We intentionally limit the visualisation to a subset of hub airports. Rendering the entire network would obscure structure rather than clarify it. At this stage, our goal is not completeness but coherence. We want to ensure that the foundation we have built is structurally correct before we begin asking more demanding questions of it.

Data Modelling Decisions

Now that we have a breakdown of the implementation so far, lets step back and talk about the modelling decisions that shaped the graph.

Node Identity

OpenFlights provides multiple identifiers for airports, including numeric IDs, ICAO codes, and IATA codes. Skymesh uses IATA codes as node identifiers.

This is a deliberate trade-off. IATA codes are human-readable, widely used, and make the graph much easier to inspect and debug. A path such as LHR → JFK → LAX is immediately meaningful.

The downside is that some airports do not have IATA codes and are therefore excluded. At this stage, Skymesh optimises for clarity and interoperability rather than exhaustive coverage.

Nodes as Data Carriers

Nodes in Skymesh are not just identifiers. Each airport node carries metadata such as geographic coordinates, country, and timezone.

Some of this information is not used immediately. It is ingested early to preserve optionality. Latitude and longitude, for example, will later enable distance calculations and spatial heuristics without requiring a second ingestion pass.

Directionality

Routes are modelled as directed edges. This reflects the reality of aviation networks, where routes are not necessarily symmetric. Treating the graph as undirected would simplify the structure, but it would also introduce incorrect assumptions that would surface later during routing and optimisation.

At this stage, edges are unweighted. Cost functions and constraints are intentionally deferred to the next article.

What’s Next

At this point, Skymesh has a structurally sound representation of the aviation network. We can ingest real data, construct a directed graph with meaningful identifiers, and perform basic inspection to verify that the model matches our expectations.

What we do not yet have is any notion of cost.

All routes are currently treated as equal. There is no concept of distance, time, price, feasibility, or optimisation beyond the existence of a path. This is intentional. Before introducing algorithms, it is important that the underlying graph is trustworthy and easy to reason about.

In the next article, the focus will shift from construction to computation. We will begin asking questions of the graph rather than just building it. That includes introducing pathfinding algorithms, defining cost functions, and exploring why naive shortest-path approaches quickly become insufficient in real-world networks.

In this article, we introduce a project that models the global aviation industry as a graph-theory problem. Airports are represented as nodes and direct flight routes as edges, forming a large, sparse, and highly non-uniform network.

The objective of this project is to explore how common aviation questions - such as route reachability, optimal paths between airports, and network-level efficiency - can be expressed as graph computations. As the project evolves, we will progressively introduce increasingly realistic constraints and cost functions, and examine how these affect both correctness and computational performance.

This article serves as the foundation for The Graph Series. Subsequent articles will build on this model to investigate optimisation techniques for graph traversal and path-finding, including algorithmic trade-offs, memoisation strategies, and performance improvements at scale.

The Aviation Industry

The aviation industry is broad and never still. On any given day, tens of thousands of commercial flights operate worldwide, connecting several thousand passenger airports and transporting millions of people across the globe. Beyond leisure travel, aviation underpins a wide range of other sectors — corporate travel, cargo and retail logistics, emergency services, and military operations, to name a few.

Despite this breadth, the scope of this series is intentionally narrow. For the purposes of this project, the focus is limited to leisure-oriented passenger flights, and specifically to the structure of the global flight network itself. Cargo operations, private aviation, and military routes are treated as out of scope.

Even within commercial passenger aviation, there is an overwhelming number of variables that could be modelled. Airlines operate different fleets with varying ranges and capacities, routes are constrained by aircraft performance and permitted airspace, and hub airports play an outsized role in shaping global connectivity. Additional factors such as weather, crew availability, regulatory constraints, and geopolitical considerations further complicate the picture.

This project does not attempt to model all of these factors upfront. Instead, it treats the aviation industry as a layered system: starting with the existence of routes between airports, then progressively introducing additional dimensions such as airline-specific routes, aircraft constraints, and — later in the series — temporal availability and scheduling. This allows individual modelling choices to be examined in isolation, while still grounding the work in a recognisably real-world system.

By clearly defining which parts of the aviation industry are being simulated, and which are intentionally ignored, we can focus on the graph problems themselves without losing sight of the domain they are inspired by.

Methodology

For this project, I have picked Python as the language of choice. The graph itself is implemented using NetworkX, and the underlying data is sourced from OpenFlights, which provides publicly available datasets covering airports, airlines, and direct flight routes.

The ingestion process focuses on building a clean and extensible representation of the aviation network. Airports are mapped to graph nodes, while direct routes between airports are represented as directed edges. At this stage, the emphasis is on establishing a structurally correct graph that can be easily extended with additional attributes and constraints in later iterations of the project.

Whilst the project is in active development, I plan to interact directly with the graph through a simple __init__.py entry point that exposes the graph as a first-class object. This allows for quick experimentation, ad-hoc inspection, and iterative refinement during development. Once the project reaches the benchmarking stages, interaction with the graph will shift to scripted, reproducible workflows designed to generate consistent and comparable performance measurements over time.

Alongside the core graph construction, a lightweight visualisation setup is introduced to generate visual representations of the network. These visualisations are not intended to be exhaustive or perfectly scaled, but rather to provide intuition around graph structure, connectivity, and the emergence of hubs within the aviation network. They also serve as a useful sanity check during development and a visual aid when discussing results later in the series.

This methodological foundation is intentionally kept simple. As the series progresses, the same setup will be reused to explore alternative cost functions, traversal strategies, and optimisation techniques, without changing the underlying data source or tooling.

Assumptions & Simplifications

It goes without saying that the aviation industry is far more complex than the features implemented in this project. This is a deliberately simplified, real-world-inspired example that allows us to explore graph representations, calculations, and optimisation techniques in a concrete setting.

The following assumptions and simplifications are made as part of this exercise:

• Direct flights only
  Only direct flight routes are represented. Multi-leg journeys are expressed implicitly through graph traversal.

• Single edge per route (initially)
  Routes between two airports are represented by a single directed edge, independent of airline or aircraft. Later in the project, this will be expanded into a multi-edge model to capture airline, aircraft, and other route-level properties.

• No temporal dimension (initially)
  The initial graph is static. Routes represent existence, not schedule or availability. Temporal constraints and availability will be introduced after the transition to multi-edge routes.

• Uniform edge behaviour
  All edges are treated equivalently at this stage. Attributes such as cost, duration, or reliability are deferred to later cost functions.

• No capacity or congestion modelling
  Airports and routes are assumed to have unlimited capacity. Operational constraints such as congestion or delays are out of scope.

• Airports as atomic nodes
  Each airport is represented as a single node, without modelling internal structure.

• Dataset limitations
  Route data is sourced from OpenFlights, which relies on a third-party provider that ceased updates in June 2014. As a result, the routes dataset is of historical value only. The other datasets (airports, airlines) appear to be maintained and are treated as current for the purposes of this project.
  As of June 2014, the routes dataset contains 67,663 routes connecting 3,321 airports across 548 airlines worldwide, which is sufficient for structural analysis and optimisation experiments.

These assumptions define the baseline model used in the early stages of the series and will be relaxed incrementally as additional complexity is introduced.

What’s next

Following on from this project abstract, we will look into the initial project’s setup including the necessary data ingestion and modelling, followed by implementing the essential algorithms we will be experimenting with.

Thanks for reading! Subscribe for free to receive new posts and support my work.