service.name and deployment.environment not being picked up

[davidjmemmett]

Describe the bug
According to the AWS documentation, the fields service.name and deployment.environment should be customisable in order to be able to distinguish logs from each other (service.name being a project name and deployment.environment being an environment name). This seems to work for a handful of logs, then reverts to the instance's hostname (or instance name tag?) and ec2:default.

Steps to reproduce

1. Specify service.name and deployment.environment in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
2. Run /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
3. Restart amazon-cloudwatch-agent.service
4. Write some logs to a location where AWS CloudWatch Agent is watching.

What did you expect to see?
In AWS CloudWatch Console -> Log Insights, using the query

fields @timestamp, @message,@entity.KeyAttributes.Name, @entity.KeyAttributes.Environment
| sort @timestamp desc
| limit 10000

All log entries should have the custom service.name and deployment.environment.

What did you see instead?
Custom service.name and deployment.environment are added to log entries for ~10 entries, then revert to the instance's hostname (or it's instance name tag) and ec2:default.

What version did you use?
Version 1.300067.0b1404-1

What config did you use?
Config: Custom JSON config file

Environment
OS: Ubuntu 24.04 LTS

Additional context
Ideally we'd be able to send rich, structured logs to AWS CloudWatch with custom attributes, instead only raw syslog is being sent. So we need to be able to distinguish projects/environments from each other.

[zhihonl]

Hi @davidjmemmett! Thanks for reporting this issue. We were able to reproduce the symptom locally and determined the root cause to something in our log pusher.

What's happening

When the agent sends a log batch via PutLogEvents, each request can carry an entity field that drives @entity.KeyAttributes.Name and @entity.KeyAttributes.Environment. Your service.name and deployment.environment config values flow into that entity through the entitystore extension and a per-source LogEntityProvider.

Root cause is in the per-stream queue in our log output. In newQueue the entityProvider parameter is used to build the first batch but is never stored on the queue struct itself:

amazon-cloudwatch-agent/plugins/outputs/cloudwatchlogs/internal/pusher/queue.go

Lines 49 to 75 in f722a9f

	func newQueue(
	logger telegraf.Logger,
	target Target,
	flushTimeout time.Duration,
	entityProvider logs.LogEntityProvider,
	sender Sender,
	wg *sync.WaitGroup,
	) Queue {
	q := &queue{
	target: target,
	logger: logger,
	converter: newConverter(logger, target),
	batch: newLogEventBatch(target, entityProvider),
	sender: sender,
	eventsCh: make(chan logs.LogEvent, 100),
	flushCh: make(chan struct{}),
	resetTimerCh: make(chan struct{}),
	flushTimer: time.NewTimer(flushTimeout),
	stopCh: make(chan struct{}),
	startNonBlockCh: make(chan struct{}),
	wg: wg,
	}
	q.flushTimeout.Store(flushTimeout)
	q.wg.Add(1)
	go q.start()
	return q
	}

The function parameter is just a local variable inside newQueue; once the function returns it's gone unless we explicitly store it somewhere with a longer lifetime. Pre-fix, the only place it was saved was inside the very first batch (line 61). The struct field q.entityProvider was left at its Go zero value, nil.

How that turns into "first ~10 lines work, then revert"

Batches are not reused. The queue holds one batch at a time, and after it's flushed it's thrown away and replaced. Every time a flush happens, q.send() builds the next batch by reading q.entityProvider from the struct, not the original function parameter, which has long since gone out of scope:

amazon-cloudwatch-agent/plugins/outputs/cloudwatchlogs/internal/pusher/queue.go

Lines 174 to 181 in f722a9f

	// send the current batch of events.
	func (q *queue) send() {
	if len(q.batch.events) > 0 {
	q.batch.addDoneCallback(q.onSuccessCallback(q.batch.bufferedSize))
	q.sender.Send(q.batch)
	q.batch = newLogEventBatch(q.target, q.entityProvider)
	}
	}

And batch.build() only attaches Entity when its provider is non-nil:

amazon-cloudwatch-agent/plugins/outputs/cloudwatchlogs/internal/pusher/batch.go

Lines 201 to 214 in f722a9f

	func (b *logEventBatch) build() *cloudwatchlogs.PutLogEventsInput {
	if b.needSort {
	sort.Stable(byTimestamp(b.events))
	}
	input := &cloudwatchlogs.PutLogEventsInput{
	LogGroupName: aws.String(b.Group),
	LogStreamName: aws.String(b.Stream),
	LogEvents: b.events,
	}
	if b.entityProvider != nil {
	input.Entity = b.entityProvider.Entity()
	}
	return input
	}

Verification

We reproduced this end-to-end locally. Across 30 lines written to a single file:

• Before fix: 5 / 30 events had the custom service.name / deployment.environment; the remaining 25 had no entity attribution.
• After fix: 30 / 30 events carried the custom values across multiple batches.

Fix

I think below fix should address the issue, one-line change in newQueue, store the provider on the queue struct so subsequent send() calls can read it back:

 q := &queue{
     ...
     batch:           newLogEventBatch(target, entityProvider),
+    entityProvider:  entityProvider,
     sender:          sender,
     ...
 }

If you want you can try building the agent with this change and test it on your end to see if it addresses the issue for you.

[davidjmemmett]

Is everything at AWS just bots now? This is clearly an LLM-generated response.