THREESCALE-4556: Enable lifetime stats and utilization in usage API by Madnialihussain · Pull Request #4227 · 3scale/porta

Madnialihussain · 2026-02-17T13:54:43Z

What this PR does / why we need it:

Adds period=eternity support to the Stats/Usage API, allowing customers to retrieve lifetime usage data via a single eternity counter from Redis. Supported on application, service, and backend API endpoints. Only eternity granularity is accepted when period is eternity (non-eternity granularity is rejected).

Which issue(s) this PR fixes

https://issues.redhat.com/browse/THREESCALE-4556

Verification steps

Test eternity period via the application stats API:
GET /stats/applications/{APP_ID}/usage.json?provider_key={KEY}&metric_name=hits&period=eternity&timezone=UTC

Response should include a single element values array, correct total, and granularity: "eternity". No previous_total or change fields should be present.

Test that non-eternity granularity is rejected:
GET /stats/applications/{APP_ID}/usage.json?provider_key={KEY}&metric_name=hits&period=eternity&granularity=month&timezone=UTC

Should return 400 Bad Request.

Test that existing periods still work:
GET /stats/applications/{APP_ID}/usage.json?provider_key={KEY}&metric_name=hits&period=month&since=2026-01-01&timezone=UTC

Response should work as before. All tests should pass.

qltysh · 2026-02-17T13:57:01Z

❌ 12 blocking issues (13 total)

Tool	Category	Rule	Count
reek	Lint	Stats::Views::Usage#extract_range_and_granularity calls 'options[:granularity]' 8 times	3	❌
rubocop	Lint	Assignment Branch Condition size for `usage` is too high. [<15, 45, 7> 47.95/20]	2	❌
rubocop	Lint	Method has too many lines. [44/20]	2	❌
reek	Lint	Stats::Views::Usage#extract_range_and_granularity refers to 'options' more than self (maybe move it to another class?)	2	❌
rubocop	Lint	Module has too many lines. [161/100]	1	❌
rubocop	Lint	Cyclomatic complexity for `extract\_range\_and\_granularity` is too high. [14/7]	1	❌
rubocop	Lint	Perceived complexity for `extract\_range\_and\_granularity` is too high. [16/8]	1	❌
qlty	Structure	Function with high complexity (count = 5): usage	1

jlledom

I found an issue: If you send any granularity other than :year, it cashes. A few comments:

There's no reason to block granularity to :year. Other granularities are acceptable as well. We already implement some limits, so for instance, if you have more than 10 years of data and you send the :month period without since param, you'll get an error about being asking too much data.
Even if we wanted to block granularity, it should return a proper error message with a proper return code instead of crashing.

Essentially, as I mentioned in some other comment in my review, period = :eternity only requires minimal tweaks, because it's just a shortcut to an scenario this endpoint already supports: sending period = nil and setting since and until to service creation date and current time, respectively, so this PR cam be limited to just set those values when period is :eternity.

jlledom · 2026-02-18T13:47:41Z

Forgot to mention: we have an API Docs screen (/p/admin/api_docs) with a SwaggerUI interface for our endpoints. It would be good to update the OpenAPI definition at doc/active_docs/analytics_api.json To add the :eternity value to this selector:

jlledom

I added a few comments. Also, this PR adds some unit tests, but it lacks some integration tests. I found a few suites that seem to be related:

test/integration/stats/data/base_controller_test.rb
test/integration/stats/data/backend_apis_controller_test.rb
test/integration/stats/data/requests_to_api_test.rb
test/integration/stats/data/service_controller_test.rb
test/integration/stats/clients_test.rb

jlledom · 2026-02-25T09:53:47Z

+            range_since = to_time(options[:since].presence || source.first&.created_at, timezone).beginning_of_year
+            range_until = timezone.now.end_of_year


I'm not sure ranges should be enforced to beginning and end oy years, since we don't accept only years as granularity. It's a bit weird to ask for all data since say 2018-03-25 with granularity month, and receive the data since 2018-01-01.

IMO :eternity should be:

since: given date or service creation date

until: Now

@mayorova @akostadinov WDYT?

I'm somehow confused by this PR. I read the JIRA issue as if we have a separate counter

The "Hits per Eternity" metric seems to be an exception though and it's not purged periodically.

Which doesn't depend on date. I'd like to understand first whether we have a separate counter and whether it is a signular counter or some separate yearly counter. If we have an yearly counter, then specifying period in years makes sense. While if it is a true eternity counter, then no period makes sense ever. And the reported period should be the existence of the metric perhaps.

In general it makes sense to me to have the same endpoint return the eternity metrics, whatever it means.

@Madnialihussain , can you clarify the above points?

Re-reading JIRA, I see now key being stats/{service:id}/metric:id/eternity

So it looks like this is a singular counter and not something we record per year. So perhaps we can:

add this eternity period

ignore until, this will always be now

ignore granularity, or accept only a granularity of eternity because a singular counter cannot be reported per year

return the report period being from metric creation time until server current UTC time

Does this make sense?

Yes, I agree with @jlledom
Probably start should be the application creation date (not service). But I'm not sure - at the moment "since" is a required field, but indeed for "eternity" it might not make sense (or be possible) to start with an exact timestamp.

@akostadinov Yeah, I think what you're saying makes sense.
Also, depending on the endpoint the Redis key can also include app ID:
stats/{service:id}/cinstance:id/metric:id/eternity

Do you see yearly data saved in actual database?

Yes, I confirmed it locally. Redis has both yearly and eternity keys for application level metrics data

Yes, I confirmed it locally. Redis has both yearly and eternity keys for application level metrics data

From my tests, it seems that the "total" value is not actually taken from the "eternity" key in Redis? Looks like it still just gets data for each time range (as defined by granularity) and a sum is applied?

I just made a test where I bumped the actual /eternity key in redis (and added another /year one), but with the "per-month" granularity the "total" value seems to be the sum of the /month keys.

This approach can give wrong results specifically when old keys are cleared from redis, and the /eternity ones remain - we need to use the /eternity one for the "total", I think.

For reference, the keys that apisonator generates: https://github.com/3scale/apisonator/blob/master/docs/redis_keys.md#stats

From my tests, it seems that the "total" value is not actually taken from the "eternity" key in Redis? Looks like it still just gets data for each time range (as defined by granularity) and a sum is applied?

In the current state of the PR, this is true, see: #4227 (comment)

The total is fetched directly from the singular eternity counter in Redis and only eternity granularity is accepted.

jlledom · 2026-02-25T09:55:24Z

    options = slice_and_use_defaults(usage_params(parameter), parameter,
                                     :period, :since, :timezone, :granularity, :until, :skip_change)
    @data = @source.usage(options)
+    @data[:utilization] = @source.utilization if @source.respond_to?(:utilization)


When utilization is nil, this returns utilization: null. It should not return the attribute in this scenario.

On the other hand. About utilization, I'm not sure we should return utilization data in the usage endpoints. It's related to usage but not what we are directly asking.

@mayorova @akostadinov WDYT? Should we create a new endpoint for utilization data? make it optional in this endpoints? Ideas?

@Madnialihussain If we wanted to do it in another endpoint, that would be another jira issue and another PR.

Hm... not sure, to be honest.
I think having utilization (i.e. how much of limit has been used) is useful, for sure, but indeed, maybe it belongs to a different endpoint.

Currently, I guess Analytics API is meant to be an API representation of the Analytics graphs in the UI - the utilization in the UI appears as a widget on the application details info.
So, maybe it makes sense to split utilization in the API too.

how much of limit has been used)

Is this historic information or current state?

Is this historic information or current state?

Well, if we talk about "eternity" period - probably both 😬 for other periods it's current state.
Although it's also true that the per-eternity limits can also be removed, so it's probably also temporary.

If we allow per-eternity limits, then we surely should show utilization data. But I wouldn't make it available in the current end point unless we already have this data available.

So if at that point we have already loaded the utilization limit, adding to the endpoint is fine with me.

The utilization data isn't loaded at this point in the usage endpoint. it would require a separate call to apisonator. So I've removed utilization from this PR. We can handle it in a separate endpoint/PR (if needed).

Or alternatively we may show utilization if this is specifically requested with a param 🤔
Because with the information that customer wants to build a portal, they will perform double the calls if this is a separate endpoint.

Where did you see the customer requirement?

In Andreu's comment I see this request to add yearly aggregates that are not deleted, similar to eternity. Maybe that's why original version of the PR had these.

I wonder though, is the JIRA as it stands well defined or we need to work more on the definition? Because until now I was looking at the PR at face value and didn't try to understand what is the actual end goal (my bad).

If you see Gustavo comment

"To make sure we’re clear and all on the same page - we’re not asking for the daily usage history back to the start of the key. We know 3Scale purges that data, and that’s fine. We’re only trying to retrieve the “hits per eternity” value, which is shown on the GUI. Since it is on the GUI, we would expect it to be retrievable through the API. "

On the GUI we show the Utilization data, so from that i made that assumption.

On the GUI we show the Utilization data, so from that i made that assumption.

I understand. It's common for Jiras to have holes in the task definition, that we fill in with our assumptions. In this case I think we can in fact make utilization data available via API, but it can be in another endpoint, no need to be the same.

jlledom · 2026-02-25T09:58:23Z

            "schema": {
              "type": "string",
              "enum": [
+                "eternity",


This is fine.

On the other hand, not directly related: we shouldn't make since required in these endpoints, that's a mistake in the definition

jlledom · 2026-02-25T10:08:43Z

+      def utilization
+        return unless @cinstance
+
+        metrics = @cinstance.service.metrics
+        records = @cinstance.backend_object.utilization(metrics)
+        return if records.error? || records.empty?
+
+        records.map do |record|
+          {
+            metric_name: record.system_name,
+            friendly_name: record.friendly_name,
+            period: record.period,
+            current_value: record.current_value,
+            max_value: record.max_value,
+            percentage: record.percentage
+          }
+        end
+      end


I think we should add a few uni tests for this method

Madnialihussain · 2026-02-18T08:48:27Z

+      records = @cinstance.backend_object.utilization(metrics)
+      return if records.error? || records.empty?
+
+      result[:utilization] = records.map do |record|


Right now, on the application show page, there are two requests to Apisonator for utilization. One comes from the controller, which fetches utilization server-side to render the utilization widget partial. The other comes from the usage API JSON response (called by the inline chart JavaScript), which now also includes utilization via append_utilization.

We can either make the API utilization opt-in (only include it when include_utilization=true is passed), or update the frontend to use the utilization data from the JSON response instead of making its own separate
server-side call

I think UI is fine how it is, and the endpoint shouldn't return utilization data. That's out of this PR scope

Madnialihussain · 2026-03-09T10:42:35Z

+            range_since = to_time(options[:since].presence || source.first&.created_at, timezone).beginning_of_year
+            range_until = timezone.now.end_of_year


Yes, the eternity key (stats/{service:id}/metric:id/eternity) is a singular counter, it holds the total usage for the lifetime of the metric. The original customer requirement was to return this lifetime usage value, and returning the eternity key solves that.

However, there's a comment by Kevin on the Jira issue that suggests going further:

"But also we should add a year period to the different buckets queriable on the Analytics API... This would then give the customer the option to retrieve total traffic for eternity for any given metric + return that same value split into yearly buckets via the API or Analytics graph which is surely useful anyway given we have usage limits with year period configurable."

So I was trying to achieve that. We can support both (eternity counter and yearly breakdown) or just return the eternity counter. Happy to go with whatever the team decides.

Madnialihussain · 2026-03-09T11:02:38Z

    options = slice_and_use_defaults(usage_params(parameter), parameter,
                                     :period, :since, :timezone, :granularity, :until, :skip_change)
    @data = @source.usage(options)
+    @data[:utilization] = @source.utilization if @source.respond_to?(:utilization)


I saw the comments on the Jira, and the customer mentioned that they want to see the number from the GUI. On the GUI, we show the utilization data. There's also this comment from the customer:

"The business use case here is relatively straightforward. We've been trying to build out a client usage portal and need that value to accurately summarize our client's usage."

So returning the utilization data in the API would have achieved that. My original intent was to return the utilization data in the usage API response because that's what we can see on the GUI. But I agree that it makes more sense to do that through a different endpoint.

Madnialihussain · 2026-03-19T13:30:53Z

    options = slice_and_use_defaults(usage_params(parameter), parameter,
                                     :period, :since, :timezone, :granularity, :until, :skip_change)
    @data = @source.usage(options)
+    @data[:utilization] = @source.utilization if @source.respond_to?(:utilization)


The utilization data isn't loaded at this point in the usage endpoint. it would require a separate call to apisonator. So I've removed utilization from this PR. We can handle it in a separate endpoint/PR (if needed).

jlledom · 2026-03-20T12:25:53Z

I'm pretty confused trying to follow the state of this PR. I received an email in my inbox with date yesterday, which notifies me of a review by @Madnialihussain , but then all comments in that review are weeks old, certainly weird. So I don't what I'm answering really.

mayorova · 2026-03-26T11:59:02Z

-          range_until = (range_since + length - 1.second).end_of_minute # taking a second away means excluding the extra day in case of a month, etc
+
+          if period.to_sym == :eternity
+            range_since = to_time(options[:since].presence || source.first&.created_at, timezone).beginning_of_year


I think there are a few issues with this line:

source.first refers to the application's service, not to the application (at least in the "Application Traffic by Metric" call), so this doesn't make sense to make it a "since" date, as the application might be created much later than the service)

to_time(source.first&.created_at, timezone) doesn't actually convert the time zone to the specified one (I guess because of stuff.acts_like?(:time) check in to_time). If I run the endpoint with timezone=UTC, I still get these values in "period" in the response (given I am have +05:00 TZ configured elsewhere), which is confusing:

"period": { "name": "eternity", "since": "2024-01-01T00:00:00+05:00", "until": "2026-12-31T23:59:59Z", "timezone": "Etc/UTC", "granularity": "month" },

I think this was discussed below as well - but why are we forcing beginning_of_year ? And similarly, why are we forcing end_of_year for range_until ?...

And also, given that specifically for "eterntity" range since parameter is optional, I guess we need to make it non-required in the ActiveDocs. To ensure the correct usage, we probably need to mention in the field description that the field is required for all periods other than "eternity".

1. `source.first` refers to the application's service, not to the application (at least in the "Application Traffic by Metric" call), so this doesn't make sense to make it a "since" date, as the application might be created much later than the service)

You're right on this, but this module is reused also for backend and service usage endpoints. I think the problem here is :eternity shouldn't accept since or until, because the eternity key in Redis is just a counter which doesn't care about ranges at all.

2. `to_time(source.first&.created_at, timezone)` doesn't actually convert the time zone to the specified one (I guess because of `stuff.acts_like?(:time)` check in `to_time`). If I run the endpoint with `timezone=UTC`, I still get these values in "period" in the response (given I am have +05:00 TZ configured elsewhere), which is confusing:

"period": { "name": "eternity", "since": "2024-01-01T00:00:00+05:00", "until": "2026-12-31T23:59:59Z", "timezone": "Etc/UTC", "granularity": "month" },

OK, but this is probably a pre-existing bug, because this call is copied from the pre-existing call a few lines below.

I agree if you don't allow :eternity to not accept or disregard since.

wrt timezone conversion, all usage is stored in UTC! So no conversion should be applied for it will likely mess up aggregated usage counters. (although I haven't looked at how these work exactly).

The suggested approach to perform local time tracking of usage is to report the traffic in the local timezone. But this has never worked properly, not for many timezones and not for avoiding edge cases.

My understanding is that basically, you should always use UTC here. If in whatever way you reported your usages in a local timezone, you will get the usages according to that timezone.

Thanks for the detailed review. All fixed in the latest push.

The eternity branch now hardcodes since to UTC 1970-01-01 and until to utc.now.end_of_day using ActiveSupport::TimeZone['UTC'], so since/until params are ignored when period is eternity. In ActiveDocs, since is changed to required: false and eternity is added to the granularity enum.

jlledom

Good job! I added a few nitpicks, feel free to ignore them.

jlledom · 2026-04-01T11:04:35Z

          granularity = options[:granularity] || GRANULARITIES[period]
-          length = 1.send(period)

          timezone = extract_timezone(options)


This can be moved to the else clause since it's only used there

jlledom · 2026-04-01T11:06:42Z

+            utc = ActiveSupport::TimeZone['UTC']
+            range_since = utc.parse('1970-01-01')
+            range_until = utc.now.end_of_day


Maybe, to simplify:

Suggested change

utc = ActiveSupport::TimeZone['UTC']

range_since = utc.parse('1970-01-01')

range_until = utc.now.end_of_day

range_since = Time.utc(1970)

range_until = Time.current.utc.end_of_day

Tried this but Time.utc(1970) returns a plain Time, but 3scale_time_range gem requires TimeWithZone object.

jlledom

Thanks!

Support fetching lifetime (eternity) usage data via the stats API for application and service. Returns the single eternity counter from Redis. Rejects non-eternity granularity, ignores since/until parameters.

…rity

mayorova · 2026-04-23T14:03:06Z

I'm getting a response which is a bit weird, when providing since/until params.

since: 2024-01-12 12:00
until: 2026-01-12 12:00
granularity: eternity

(no period specified)

The result is:

{
  "metric": {
    "id": 41,
    "name": "product-method",
    "system_name": "product-method",
    "unit": "hit"
  },
  "period": {
    "name": null,
    "since": "1970-01-01T00:00:00Z",
    "until": "9999-12-31T23:59:59Z",
    "timezone": "Etc/UTC",
    "granularity": "eternity"
  },
  "total": 20,
  "values": [
    20
  ],
...
}

By the way, the value is correct, but the period in the response looks weird.

If I do provide period in the request, the period in the response looks like this (regardless of whether since/until are provided or not):

  "period": {
    "name": "eternity",
    "since": "1970-01-01T00:00:00Z",
    "until": "2026-04-23T23:59:59Z",
    "timezone": "Etc/UTC",
    "granularity": "eternity"
  },

(which I think looks better)

So, I am thinking - maybe granularity "eternity" should not be supported for anything else except for "eternity" period?
I see that it's already like that in the other direction, if "period: eternity", the only possible granularity is "eternity". Maybe it should be the other way around too? If granularity is "eternity", the only possible option is "period: eternity" (i.e. since and until do not matter).

Madnialihussain requested review from akostadinov, jlledom and mayorova February 17, 2026 13:54

Madnialihussain self-assigned this Feb 17, 2026

jlledom requested changes Feb 18, 2026

View reviewed changes

Madnialihussain force-pushed the eternity-granularity-and-utilization branch from 0102e2e to 16ba409 Compare February 24, 2026 13:30

Madnialihussain requested a review from jlledom February 24, 2026 14:23

jlledom reviewed Feb 25, 2026

View reviewed changes

Madnialihussain commented Mar 19, 2026

View reviewed changes

Madnialihussain requested a review from jlledom March 19, 2026 13:40

mayorova reviewed Mar 26, 2026

View reviewed changes

Madnialihussain force-pushed the eternity-granularity-and-utilization branch from 1a81458 to 7dfa26c Compare March 31, 2026 13:16

jlledom previously approved these changes Apr 1, 2026

View reviewed changes

Madnialihussain dismissed jlledom’s stale review via 107d8aa April 7, 2026 12:57

jlledom approved these changes Apr 10, 2026

View reviewed changes

Madnialihussain force-pushed the eternity-granularity-and-utilization branch from 7c3f699 to 25eb2c6 Compare April 15, 2026 07:49

Madnialihussain changed the title ~~THREESCALE-4556: Enable lifetime stats and utilization in usage API~~ THREESCALE-4556: Enable lifetime stats and utilization in usage API Apr 15, 2026

Madnialihussain requested a review from mayorova April 15, 2026 11:39

Madnialihussain added 8 commits April 21, 2026 17:20

Enable lifetime stats and utilization in usage API

791068f

simplify eternity, move utilization to views module

3bfacb7

minor stray colon fix

652e079

remove eternity from top_applications endpoint

22bff67

Remove utilization from usage API response

ec5efbe

Add period=eternity support for Stats Usage API

9d512df

Support fetching lifetime (eternity) usage data via the stats API for application and service. Returns the single eternity counter from Redis. Rejects non-eternity granularity, ignores since/until parameters.

simplify eternity branch style

7b2d864

Fix TimeWithZone requirement and update XSS test for eternity granula…

c6b4ae3

…rity

Madnialihussain force-pushed the eternity-granularity-and-utilization branch from 25eb2c6 to c6b4ae3 Compare April 21, 2026 16:20

		range_since = to_time(options[:since].presence \|\| source.first&.created_at, timezone).beginning_of_year
		range_until = timezone.now.end_of_year

Conversation

Madnialihussain commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qltysh Bot commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jlledom left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jlledom commented Feb 18, 2026

Uh oh!

jlledom left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Madnialihussain commented Feb 17, 2026 •

edited

Loading

qltysh Bot commented Feb 17, 2026 •

edited

Loading