Emit job usage data through IBM Cloud Event Streams instances

[ismaelRozasRamallal]

Summary

Fleets jobs now report their classical compute usage to downstream consumers in real time. From the moment a job starts running until it completes, consumers receive a continuous stream of usage events they can use to track how long the job has been executing and react when it finishes. This enables billing, monitoring, and any other business logic that depends on knowing a job's runtime to operate without polling the API.

One key point here is, Ray-based jobs are not affected by this change. Event publishing is scoped exclusively to Fleets (Code Engine) jobs.

🔨 related with #https://github.ibm.com/IBM-Q-Software/qiskit-serverless/issues/1507

Details and comments

• Add job usage event publishing to IBM Cloud Event Streams for Fleets (Code Engine) jobs.
• When a job transitions to RUNNING, a job_started event is published; subsequent scheduler iterations emit job_in_progress events with the elapsed classical time in nanoseconds; and when the job reaches a terminal state, a job_ended event is published.
• Events follow the CloudEvents 1.0 spec and are published to a topic named quantum.{ENVIRONMENT}.function-usage.v1. Publishing is gated by an EVENT_STREAMS_ENABLED feature flag — when disabled, a no-op client logs at DEBUG level instead.
• Publish failures block the DB status update and are retried on the next scheduler iteration.
• ⚠️ Consumers should handle duplicate events keyed on job_id + event_type, particularly for job_ended which can be re-emitted if the scheduler restarts between publish and DB commit. It should be that way anywhere since that's one principle for Event Driven architectures but just in case to take it into account
• There is a DB migration to add running_started_at field. We need to know when the job starts to run in order to keep it as base for the calculated delta we will be publishing as usage. As a consequence, each job will sabe a timestamp on when it changed to running status

More context

Still missing other events to be published. This PR just covers the Kafka integration and the very basic publishing, let's take it as reference for next iterations.

It also misses adding an instance in docker-compose to make acceptance tests. Didn't want to add it at first to avoid make the PR even bigger.

[marceloamaral]

Fleets will also need the compute profile (actually the allocated resources, number of CPU, RAM and GPUs) to calculate the cost. Even though this might be out of scope for this PR, it would be good to start thinking about how this information will be added later...

[ismaelRozasRamallal]

Fleets will also need the compute profile (actually the allocated resources, number of CPU, RAM and GPUs) to calculate the cost. Even though this might be out of scope for this PR, it would be good to start thinking about how this information will be added later...

Thanks for the comment Marcelo!! you're right, there is a potential metric there which can be good to use. The compute profile can be also took as metric for complexity, but we need to figure out if we can use since it's easy the "cheat" (since the profile is decided when adding a function, we need to avoid somebody deciding to pick up the highest compute profile to force users paying more on their function execution)