-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Log to Metrics generates histogram buckets with invalid scientific notation for large values #11651
Copy link
Copy link
Open
Description
Bug Report
Describe the bug
When using the log_to_metrics filter to generate Prometheus histograms, bucket boundaries with values greater than or equal to 1,000,000 are emitted in scientific notation (e.g. 1e+06) but the format is not correct since it adds a decimal to this scientific notation. The error that you get in prometheus when you do a histogram_quantile is:
PromQL warning: bucket label "le" is missing or has a malformed value of "1e+08.0" for metric name "" (1:26)
As you can see the metrics that those buckets generate:
apache_http_histogram_request_size_bytes_bucket{le="100.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 0
apache_http_histogram_request_size_bytes_bucket{le="1000.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 3905
apache_http_histogram_request_size_bytes_bucket{le="10000.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 3959
apache_http_histogram_request_size_bytes_bucket{le="100000.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 3959
# these ones are not processed correctly
apache_http_histogram_request_size_bytes_bucket{le="1e+06.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 3959
apache_http_histogram_request_size_bytes_bucket{le="1e+07.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 3959
apache_http_histogram_request_size_bytes_bucket{le="1e+08.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 3959
apache_http_histogram_request_size_bytes_bucket{le="1e+09.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 3959 To Reproduce
- Configure a
log_to_metricsfilters with these buckets:
[FILTER]
Name log_to_metrics
Match httpd_access*
Tag metrics_httpd_access
Metric_Mode histogram
Metric_Name request_size_bytes
Metric_Description This metric shows the request size in bytes
Metric_Namespace apache_http
Value_Field requestsize
Label_Field _cluster
Label_Field _bundle
Label_Field _host
Label_Field proxy_type
Bucket 100
Bucket 1000
Bucket 10000
Bucket 100000
Bucket 1000000
Bucket 10000000
Bucket 100000000
Bucket 1000000000
Kubernetes_Mode off
Discard_Logs off
Flush_Interval_Sec 10
Flush_Interval_Nsec 0
Emitter_Mem_Buf_Limit 10M
[OUTPUT]
Name prometheus_exporter
Match metrics_httpd_access
Host 127.0.0.1
Port 9999
- Curl to
localhost:9999and you can see the wrong scientific notation in prometheus
Expected behavior
- I think the scientific notation is accepted, but not with the decimal since it's wrong scientific notation.
Screenshots
Your Environment
- Version used: 4.1.0
- Configuration:
- Environment name and version (e.g. Kubernetes? What version?):
- Server type and version:
- Operating System and version: Alma9
- Filters and plugins: log_to_metrics
Additional context
Reactions are currently unavailable