How We Reduced OpenTelemetry Memory Usage by 97%

During a routine memory profiling run, OpenTelemetry’s GraphQL instrumentation appeared near the top of the allocation report. A 56-line change reduced its allocations by 97%.

The problem

At GitHub, we use OpenTelemetry for distributed tracing. The opentelemetry-instrumentation-graphql gem instruments GraphQL queries, creating spans for field execution, type resolution, and authorisation checks.

I ran memory_profiler against a typical GraphQL request and saw this:

allocated memory by gem
-----------------------------------
   130,896  graphql/lib    <- the instrumentation

The instrumentation allocated more than 130KB per request.

The investigation

Span attributes were being rebuilt on every execute_field call. A GraphQL query can resolve dozens or hundreds of fields, so the same kinds of hashes were allocated repeatedly and then discarded.

The field name, parent type, and return type do not change during those calls.

The fix

GraphQL v2 instrumentation already cached these attributes when the schema loaded. I backported the same approach to v1.

In simplified form:

# Before: computed on every call
attributes['graphql.field.parent'] = data[:owner]&.graphql_name

# After: cached per field, computed once
attributes = @cached_attributes[key] ||= {
  'graphql.field.parent' => data[:owner]&.graphql_name
}

The patch added 56 lines and deleted 17.

The results

  allocated memory by gem
  -----------------------------------
-   130,896  graphql/lib
+     3,616  graphql/lib

Total allocated memory: 130,896 -> 3,616 bytes (97% reduction)
Total allocated objects: 786 -> 32 (96% reduction)
Overall bytes used: 2,084,536 -> 1,957,424 (6% reduction)

After the change, the instrumentation was barely visible in the allocation report.

The trade-off

This change wasn’t entirely free. By caching attributes at the interface level rather than the concrete type level, the graphql.field.parent attribute now reports the GraphQL interface name instead of the concrete class for execute_field spans.

For example, if you have a Node interface implemented by User and Repository, the parent would be reported as Node instead of User or Repository.

We accepted the loss of specificity in that attribute for v1. GraphQL v2 still reports the concrete class.

Profiling library code

Application profiles include the cost of libraries, middleware, ORMs, and instrumentation. In this case the useful change belonged upstream rather than in the application.

The fix was merged into OpenTelemetry Ruby contrib.

The tools I used:

memory_profiler for allocation analysis
stackprof for CPU profiling
A custom benchmark script to compare before/after in isolation