CSPL-4513 Events for Index & Ingestion separation

[kasiakoziol]

Description

It adds events and enhances logs for IngestorCluster and enhances events for IndexerCluster.

Key Changes

• Refactoring existing event messages for IngestorCluster and IndexerCluster
• Introducing new events for IngestorCluster: ScaledUp, ScaledDown, QueueConfigUpdated and
  IngestorsRestarted
• Introducing new events for IndexerCluster: QueueConfigUpdated and
  IngestorsRestarted
• Introducing slog
• Automated tests for the above

Testing and Verification

Automated and manual tests

Related Issues

Jira: https://splunk.atlassian.net/browse/CSPL-4513

PR Checklist

• Code changes adhere to the project's coding standards.
• Relevant unit and integration tests are included.
• Documentation has been updated accordingly.
• All tests pass locally.
• The PR description follows the project's guidelines.

[github-actions]

CLA Assistant Lite bot CLA Assistant Lite bot All contributors have signed the COC ✍️ ✅

[github-actions]

CLA Assistant Lite bot:
Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.

---

I have read the CLA Document and I hereby sign the CLA

---

5 out of 6 committers have signed the CLA.
✅ @kasiakoziol
✅ @Igor-splunk
✅ @gabrielm-splunk
✅ @kubabuczak
✅ @minjieqiu
❌ @Nguyen Duc Trung
NGUYEN Duc Trung seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You can retrigger this bot by commenting recheck in this Pull Request}

[kubabuczak]

I'm not sure if we need all the events. We already are producing too much of them.
We need some rules to identify if this event is needed. You can ask Igor about exact ones but we should rather focus on:

1. Describe what happened (not what is)
2. Expose actionable events / errors
3. Expose very important events

We should also standardise event reason naming to UpperCamelCase + Past tense in the end of event reason

We should also remove error logging in event description - this should be logged, not exposed in event.

Other issue is that in some tests we are not calling code that we are testing, but we are reprosucing conditionals - this is wrong pattern. We should always call code that we are testing.

[nit] I prefer table testing

[kasiakoziol]

"I'm not sure if we need all the events. We already are producing too much of them." -> This will be reviewed by @Igor-splunk before we merge feature/slog-as-default-logger into develop, because that's a general SOK tendency.

"We should also standardise event reason naming to UpperCamelCase + Past tense in the end of event reason." and "We should also remove error logging in event description - this should be logged, not exposed in event." -> Already addressed

"Other issue is that in some tests we are not calling code that we are testing, but we are reprosucing conditionals - this is wrong pattern. We should always call code that we are testing." -> Looking into it.