[Sailpoint ISC] Add new data streams for entity support

[chemamartinez]

Summary

The sailpoint_identity_sc integration currently collects only audit events from the SailPoint Identity Security Cloud Search API. It does not collect identity data and therefore cannot support ECS entity workflows. This issue tracks the addition of two new data streams (identities and machine_identities) to bring entity support to the integration.

Background

The existing events data stream collects audit logs via POST /v2025/search (indices: ["events"]). Identity records (who users are, what they own, who they manage, what permissions they hold) are not collected today. Adding entity support requires dedicated data streams that pull from the identity-focused endpoints of the SailPoint ISC API.

Scope

Two new data streams will be added:

Data Stream	ECS Entity Type	Description
identities	user	Human identities; employees, contractors, and external users governed by SailPoint
machine_identities	service	Non-human identities; service accounts, application accounts, and bots

The two data streams are kept separate because they use different API endpoints with different schemas, different OAuth scopes, different pagination mechanisms, and different ECS entity types. The machine identity API is also currently marked experimental by SailPoint, so isolating it avoids coupling its stability to the human identity stream.

Endpoint Strategy

identities data stream

Two SailPoint endpoints can serve human identity data:

Capability	GET /v2026/identities	POST /v2026/search (indices: identities)
Owned objects (roles, sources, apps)	Not included	Included (owns object)
Access items (entitlements, access profiles, roles)	Not included	Included (access array with type and source)
Manager reference	Basic (id, name)	Full (id, name, displayName)
Segment membership	Not included	Included (visibleSegments)
Connected applications	Not included	Included (apps array)
Max page size	250	10,000 (plus searchAfter cursor for datasets above 10K)
OAuth scope	idn:identity:read	sp:search:read
Already in use by integration	No	Yes, same endpoint used by events data stream

Decision: use POST /v2026/search with indices: ["identities"]. It provides the richest schema, is already authenticated, supports a 40× larger page size, and is the only endpoint that exposes the ownership, access, and segment data needed for entity field mapping.

For manager–report relationships (who a manager supervises), an additional targeted Search call is needed per manager identity: POST /v2026/search with query: "manager.id:<id>". This uses the same endpoint and scope and is gated on the isManager flag so it only fires for identities that manage others.

machine_identities data stream

Only one endpoint exists for machine identities: GET /v2026/machine-identities. It is marked experimental by SailPoint and requires the request header X-SailPoint-Experimental: true and a distinct OAuth scope (idn:mis-identity:read). Pagination uses limit/offset with a maximum page size of 250.

Endpoints

identities data stream

Endpoint	Method	Purpose
/{version}/search	POST	Primary collection of all human identity documents (indices: ["identities"])
/{version}/search	POST	Reverse-lookup of direct reports for manager identities (query: "manager.id:<id>")

OAuth scope required: sp:search:read

machine_identities data stream

Endpoint	Method	Purpose
/{version}/machine-identities	GET	Collection of all machine identity documents

OAuth scope required: idn:mis-identity:read
Required header: X-SailPoint-Experimental: true

Notes

• The machine_identities data stream depends on a SailPoint experimental API. The CEL program must handle 404/501 responses gracefully and the manifest should document this dependency clearly.
• The idn:mis-identity:read scope is not currently present in the integration manifest and will need to be added, either as a new default scope or as part of a separate optional policy template.
• The api_version variable already present in the manifest should be reused for both new data streams to remain consistent with the existing events data stream. Its default value could be updated from v2025 to v2026 as the most recent versions provide new features and ensures backward compatibility.

[infra-vault-gh-plugin-prod]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[github-actions]

tl;dr: this request should be handled as net-new SailPoint ISC data streams work, because the current package is hard-coded to a single events stream end-to-end.

Recommendation

Proceed with implementation of new data_stream/<entity> datasets (not just config tweaks), using the existing events stream as the template and updating docs/tests/changelog together in the same PR.

Findings

1. Current package scope is events-only:

   • packages/sailpoint_identity_sc/data_stream/events/manifest.yml:1 defines the only stream (Events).
   • packages/sailpoint_identity_sc/docs/README.md:7 lists only events under Data Streams.
   • packages/sailpoint_identity_sc/manifest.yml:30 policy template is explicitly “Sailpoint identity security cloud events”.

2. Collection logic is tied to events:

   • packages/sailpoint_identity_sc/data_stream/events/agent/stream/cel.yml.hbs:48 sends "indices": ["events"].
   • packages/sailpoint_identity_sc/data_stream/events/elasticsearch/ingest_pipeline/default.yml:23 parses into sailpoint_identity_sc.events.
   • packages/sailpoint_identity_sc/data_stream/events/elasticsearch/ingest_pipeline/default.yml:92 sets event.dataset: sailpoint_identity_sc.events.

3. Tests are stream-local and currently only cover events:

   • packages/sailpoint_identity_sc/data_stream/events/_dev/test/system/test-default-config.yml:17
   • packages/sailpoint_identity_sc/data_stream/events/_dev/test/pipeline/test-event.log-expected.json:12

4. Prior SailPoint PR history is also events-focused:

   • #12348 (initial integration) describes Events dataset/API only.
   • #14935 and #15359 are improvements/fixes for existing events stream behavior.

5. I attempted to read related issue threads (#18836, #12124) via MCP issue_read, but those resources were integrity-filtered in this workflow context.

Verification

I verified the local package layout and test surfaces with:

find packages/sailpoint_identity_sc/data_stream -mindepth 1 -maxdepth 1 -type d | sort
---
packages/sailpoint_identity_sc/data_stream/events

find packages/sailpoint_identity_sc/data_stream/events/_dev/test -type f | sort
packages/sailpoint_identity_sc/data_stream/events/_dev/test/pipeline/test-event.log
packages/sailpoint_identity_sc/data_stream/events/_dev/test/pipeline/test-event.log-expected.json
packages/sailpoint_identity_sc/data_stream/events/_dev/test/system/test-default-config.yml

This confirms only one stream exists today (events) and all test assets are under that stream.

Detailed Action Plan

1. Define target entities and stream names

   • Confirm the SailPoint entities/endpoints to support and map each to a dedicated dataset name (data_stream/<entity>).

2. Create new stream skeleton(s)

   • Copy packages/sailpoint_identity_sc/data_stream/events/* into each new stream directory.
   • Start by adapting:
     • data_stream/<entity>/manifest.yml
     • data_stream/<entity>/agent/stream/cel.yml.hbs
     • data_stream/<entity>/elasticsearch/ingest_pipeline/default.yml
     • data_stream/<entity>/fields/*.yml

3. Update CEL request/query per entity

   • In each new cel.yml.hbs, replace event-specific selector (indices: ["events"] in current file at .../events/agent/stream/cel.yml.hbs:48) with the entity-specific query/index selector required by SailPoint API.

4. Adjust ingestion mappings

   • Replace sailpoint_identity_sc.events object paths with entity-specific paths in pipeline + fields.
   • Ensure event.dataset is set to sailpoint_identity_sc.<entity> (current pattern shown at .../events/elasticsearch/ingest_pipeline/default.yml:92).

5. Add stream-local tests

   • Add _dev/test/system/test-default-config.yml and pipeline expected output fixtures for each new stream, mirroring current events test structure.

6. Wire package-level metadata/docs

   • Update packages/sailpoint_identity_sc/manifest.yml to expose the new stream(s) in policy templates/input text.
   • Update packages/sailpoint_identity_sc/docs/README.md Data Streams + Logs sections with new dataset filters/examples.

7. Record release note

   • Add an enhancement entry in packages/sailpoint_identity_sc/changelog.yml describing the new entity stream support.

Related Items

Type	Link	Relevance
Issue	#18836	Current triage target (“add new data streams for entity support”)
PR	#12348	Initial SailPoint ISC integration; events-only scope
PR	#14935	Events stream API/version handling improvements
PR	#15359	Events stream serialization/error-handling improvements
File	packages/sailpoint_identity_sc/data_stream/events/agent/stream/cel.yml.hbs:48	Hardcoded event index selector
File	packages/sailpoint_identity_sc/data_stream/events/elasticsearch/ingest_pipeline/default.yml:23	Event-specific ingestion target
File	packages/sailpoint_identity_sc/docs/README.md:7	Only events documented

Note

🔒 Integrity filter blocked 28 items

The following items were blocked because they don't meet the GitHub integrity level.

• #18836 issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #18836 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #15359 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #14935 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #14141 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #17560 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #12348 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #13712 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #13035 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #12997 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #14474 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #12919 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #14065 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #14063 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #14062 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #18509 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• ... and 12 more items

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

---

What is this? | From workflow: Issue Triage

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.