Fix entitlement chunk collision and active-lease truncation - #4
Merged
Conversation
- Label entitlement metrics with ems_entitlement_id / ems_product_key_id so same-product entitlements purchased in separate chunks emit distinct series instead of overwriting each other. - Export entitlement in-use and unassigned quantities, which were fetched but dropped. - Fetch active leases from /leases/all (with compressedClients support) instead of /leases, which truncates the client list for large orgs.
yummybomb
marked this pull request as ready for review
August 31, 2026 20:47
yummybomb
marked this pull request as draft
August 31, 2026 21:42
yummybomb
marked this pull request as ready for review
September 1, 2026 14:00
sjmiller609
approved these changes
Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
The exporter's numbers didn't match what the NVIDIA licensing portal shows. Three reasons:
1. Entitlements bought in separate orders overwrote each other.
nvidia_cls_entitlement_total_quantitywas labeled only by feature, product, version, and license type. If we bought the same product twice (say 72 seats, then 128 more), both chunks had identical labels and one silently replaced the other. The org total came out short.Fix: entitlement metrics now also carry
ems_entitlement_idandems_product_key_id, so each chunk is its own series. Sum across series to get the org total.2. In-use and unassigned seat counts were fetched but thrown away.
The API already returns them per entitlement feature. They're now exported as
nvidia_cls_entitlement_in_use_quantityandnvidia_cls_entitlement_unassigned_quantity, same labels as the total.3. Active lease counts capped at 100.
The
/leasesendpoint truncates the client list for large orgs. We saw lease counts flatline at exactly 100. The portal UI uses/leases/allinstead, which returns every client. For big payloads it sends them gzip+base64 encoded in acompressedClientsfield, which the client now decodes.Things to know
nvidia_cls_entitlement_total_quantity. Anything reading it as one series needs to sum across the new labels. Grouping by product or feature still works as before. The license-server metrics (nvidia_cls_license_server_feature_*) are unchanged./leases/allpath and theemsEntitlementId/emsProductKeyId/compressedClientsfield names come from the licensing portal's frontend bundle, which calls the same API. I didn't have a production key to hit it directly. If it fails on deploy, you'll seenvidia_cls_up=0with stale data, and reverting restores the old behavior.Testing
Unit tests cover the compressed-clients decode path, the per-chunk entitlement extraction, the
/leases/allHTTP contract, and the Prometheus and OTEL output.go test ./...passes.Note
Medium Risk
Entitlement metric label changes break existing dashboards and recording rules; the new leases API and compressed payload path were not verified against live CLS and can fail scrapes (
nvidia_cls_up=0).Overview
Fixes three gaps between CLS portal data and exported metrics.
Entitlement series now include
ems_entitlement_idandems_product_key_idonnvidia_cls_entitlement_total_quantity(and related series), so separate purchases of the same product no longer collide in Prometheus. Org-wide totals require summing across those labels.New gauges
nvidia_cls_entitlement_in_use_quantityandnvidia_cls_entitlement_unassigned_quantityexpose quantities the client already parsed but previously dropped; Prometheus and OTEL paths both emit them with the same label set.Active leases are loaded from
.../leases/allinstead of.../leases, with decoding of optionalcompressedClients(base64 + gzip JSON). That addresses truncated client lists that capped active-lease metrics for large orgs.README documents the entitlement labeling and new metrics. Unit tests cover compressed lease responses, the
/leases/allHTTP contract, per-chunk entitlement extraction, and collector/OTEL observation mapping.Reviewed by Cursor Bugbot for commit 33dfc89. Bugbot is set up for automated code reviews on this repo. Configure here.