Store Gateway under pressure

[clcc2019]

On average, a prometheus instance saves 30d data. How much resources does it consume? We have monitored 30 clusters, and the thanos cluster has 6 machines with 128G memory, and the memory usage rate exceeds 60%

[fktkrt]

Could you please provide additional details?

• What is the load on each Prometheus instances (cardinality, ingestion), and what is the resource consumption there?
• You have 6 machines with a sum of 128G RAM? Is the sum usage rate >60%?
• What are the CPU and Memory request of your Store component? What is the utilization here?