Skip to content

Log to Metrics generates histogram buckets with invalid scientific notation for large values #11651

@Pigueiras

Description

@Pigueiras

Bug Report

Describe the bug

When using the log_to_metrics filter to generate Prometheus histograms, bucket boundaries with values greater than or equal to 1,000,000 are emitted in scientific notation (e.g. 1e+06) but the format is not correct since it adds a decimal to this scientific notation. The error that you get in prometheus when you do a histogram_quantile is:

PromQL warning: bucket label "le" is missing or has a malformed value of "1e+08.0" for metric name "" (1:26)

As you can see the metrics that those buckets generate:

apache_http_histogram_request_size_bytes_bucket{le="100.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 0
apache_http_histogram_request_size_bytes_bucket{le="1000.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 3905
apache_http_histogram_request_size_bytes_bucket{le="10000.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 3959
apache_http_histogram_request_size_bytes_bucket{le="100000.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 3959
# these ones are not processed correctly 
apache_http_histogram_request_size_bytes_bucket{le="1e+06.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 3959
apache_http_histogram_request_size_bytes_bucket{le="1e+07.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 3959
apache_http_histogram_request_size_bytes_bucket{le="1e+08.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 3959
apache_http_histogram_request_size_bytes_bucket{le="1e+09.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 3959 

To Reproduce

  • Configure a log_to_metrics filters with these buckets:
[FILTER]
    Name  log_to_metrics
    Match httpd_access*
    Tag metrics_httpd_access
    Metric_Mode histogram
    Metric_Name request_size_bytes
    Metric_Description This metric shows the request size in bytes
    Metric_Namespace apache_http
    Value_Field requestsize
    Label_Field _cluster
    Label_Field _bundle
    Label_Field _host
    Label_Field proxy_type
    Bucket 100
    Bucket 1000
    Bucket 10000
    Bucket 100000
    Bucket 1000000
    Bucket 10000000
    Bucket 100000000
    Bucket 1000000000
    Kubernetes_Mode off
    Discard_Logs off
    Flush_Interval_Sec 10
    Flush_Interval_Nsec 0
    Emitter_Mem_Buf_Limit 10M

[OUTPUT]
    Name prometheus_exporter
    Match metrics_httpd_access
    Host 127.0.0.1
    Port 9999
  • Curl to localhost:9999 and you can see the wrong scientific notation in prometheus

Expected behavior

  • I think the scientific notation is accepted, but not with the decimal since it's wrong scientific notation.

Screenshots

Your Environment

  • Version used: 4.1.0
  • Configuration:
  • Environment name and version (e.g. Kubernetes? What version?):
  • Server type and version:
  • Operating System and version: Alma9
  • Filters and plugins: log_to_metrics

Additional context

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions