fix: Enhance blob management by adding clear_all_blobs method and updating file_id handling

[Pavan1-Microsoft]

Purpose

This pull request introduces improvements to file and blob management in the ingestion service, focusing on more robust cleanup of Azure storage and improved file ID handling. The key changes ensure that all blobs are properly cleared between scenarios, file IDs are sanitized for consistency, and document deletion aligns with the upload path.

Azure Storage Cleanup Enhancements:

• Added a new clear_all_blobs method to AzureStorageService to delete all blobs in the storage container, ensuring no stale files remain between scenarios. This method is now called in the service's clear method. [1] [2]

File ID Handling Improvements:

• Updated logic in multiple ingestion service methods to replace spaces with underscores in file_id generation, ensuring file IDs are consistent and safe for use as keys or identifiers. [1] [2] [3]

Blob Deletion Path Consistency:

• Modified the document-level blob deletion path to match the path used during upload, preventing orphaned blobs and ensuring correct cleanup.

Azure Pipeline Script Activation:

• Un-commented and activated the AI agent setup steps in both PowerShell and Bash sections of the Azure pipeline configuration, ensuring the agent is set up as part of the deployment process. [1] [2]

Does this introduce a breaking change?

• Yes
• No

Golden Path Validation

• I have tested the primary workflows (the "golden path") to ensure they function correctly without errors.

Deployment Validation

• I have validated the deployment process successfully and all services are running as expected with this change.