# 🌱 Cultivating Unique service.instance.id on NGINX Ingress with OpenTelemetry

A task to set a unique `service.instance.id` on NGINX Ingress Controller using the OpenTelemetry should be simple, right? But as it turned out, the ingress controller doesn't expose all of NGINX’s OTel knobs out of the box, so I had to roll my own tweak garden.

### Why should I care?

Without unique instance IDs, your tracing data looks like a tangled tangleweed—hard to trace, difficult to debug, and completely undermines what we’re trying to do with observability.

## The solution
### Part 1 - Nginx Ingress Controller - Helm Chart
To get `POD_UID` injected into each trace, here’s the minimal yet powerful config snippet I landed on:
- Set the POD_UID environment variable, once NGINX Ingress Controller set by default only the POD_NAME and POD_NAMESPACE.

```yaml
extraEnvs:
  - name: POD_UID
    valueFrom:
      fieldRef:
         fieldPath: metadata.uid
```

- Set the POD_UID as span-attribute for instance.

```yaml
controller:
  config:
    main-snippet: |
      env POD_UID;
    server-snippet: |
      set $pod_uid "unknown";
      access_by_lua_block {
        ngx.var.pod_uid = os.getenv("POD_UID") or "unknown"
      }
      opentelemetry_attribute service.instance.id $pod_uid;
```
### Part 2 - OpenTelemetry Collector Config
- Configure the [transform](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor) processor to set the resource-attribute. 

```yaml
    processors:
      transform:
        error_mode: ignore
        trace_statements:
          - set(resource.attributes["service.instance.id"], span.attributes["service.instance.id"])
          - delete_key(span.attributes, "service.instance.id")
```

### How it works:

  - Explicitly exposes the `POD_UID` environment variables so NGINX can see them.
  - Initializes a default `$pod_uid`—my fail-safe in case the env var goes missing.
  - Uses Lua to pull in the real `POD_UID`.
  - Finally, sets the OpenTelemetry attribute `service.instance.id` to match the actual UID.
  - Through OpenTelemetry Collector capture the `service.instance.id` from span-attributes and set it on resource-attributes.

### Benefits to your telemetry:

| Without unique `service.instance.id`                             | With unique `service.instance.id`          |
| ---------------------------------------------------------------- | ------------------------------------------ |
| Pods share the same instance ID, causing confusion in telemetry. | Each pod is individually identifiable.     |
| Difficult to isolate errors and debug effectively.               | Allows precise tracing to individual pods. |
| Limited visibility into pod-specific issues.                     | Enables accurate root-cause analysis.      |

### Final result

By implementing this solution, each NGINX ingress pod is clearly distinguishable in tracing data. This improves observability significantly by providing accurate, pod-specific telemetry, facilitating precise troubleshooting and diagnostics.