Revert STIG source file handling changes

[sketch0395]

This pull request refactors and simplifies the source file selection and batching logic in run-stig-assessment.py, with the goal of making file inclusion rules clearer and reducing the complexity of handling large files. The changes also tighten file size limits, update directory exclusion logic, and streamline how files are selected and presented for assessment.

File inclusion and exclusion logic:

• Simplified the SOURCE_EXTENSIONS and INCLUDE_FILENAMES sets to remove rarely used or redundant file types and combined related categories for easier maintenance. (scripts/shell/run-stig-assessment.py scripts/shell/run-stig-assessment.pyL85-R105)
• Refactored the _is_excluded_dir function to use a more concise check for hidden and excluded directories, removing the _ALLOWED_HIDDEN_DIRS variable and inlining its logic. (scripts/shell/run-stig-assessment.py scripts/shell/run-stig-assessment.pyL337-R321)

File size and batching policy:

• Reduced MAX_FILE_BYTES from 100,000 to 50,000, so only files up to 50 KB are included in the assessment; files above this size are now skipped entirely instead of being truncated. (scripts/shell/run-stig-assessment.py [1] [2]
• Updated the file collection logic to skip files exceeding MAX_FILE_BYTES instead of truncating them, simplifying content handling and reducing the risk of partial file analysis. (scripts/shell/run-stig-assessment.py scripts/shell/run-stig-assessment.pyL374-L396)

Batch construction and context management:

• Removed the excerpting logic for over-budget files in build_code_context; now, files are either fully included or fully skipped, and a note is added listing any files omitted due to context limits. (scripts/shell/run-stig-assessment.py scripts/shell/run-stig-assessment.pyL777-L823)
• Clarified comments and logic regarding the context budget, reflecting the latest model capabilities and simplifying the rationale for batch size and file inclusion. (scripts/shell/run-stig-assessment.py [1] [2]

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.