Feature: Add bulk URL processing and global installation support

[zazu-22]

Feature: Add bulk URL processing and global installation support

Problem

The current script only processes one URL at a time, making it inefficient for documentation sites with multiple pages. Users also need to manage dependencies manually and run the script from the project directory.

Solution

This PR adds bulk processing capabilities and global installation support to streamline documentation workflows.

Key improvements:

• Bulk processing: Process multiple URLs from a file with --urls-file
• Global installation: Use uv script metadata for dependency-free execution anywhere
• Smart error recovery: Auto-generate retry files for failed URLs
• Better file naming: Prevent overwrites with path-based, timestamped filenames

Example usage:

# Install globally
ln -sf "$(pwd)/generate-llmstxt.py" ~/.local/bin/generate-llmstxt

# Process multiple URLs
echo "https://docs.example.com/page1" >> urls.txt
echo "https://docs.example.com/page2" >> urls.txt
generate-llmstxt --urls-file urls.txt

# Auto-retry failures
generate-llmstxt --urls-file urls-failed.txt

Output changes:

• Before: docs.example.com-llms.txt (overwrites on same domain)
• After: docs_example_page1.txt, docs_example_page2.txt (unique files)
• Bulk mode: Individual files + docs_example_consolidated_index_20250103_143022.txt

Impact

• Backward compatible: All existing commands work unchanged
• Performance: Process entire documentation sites in one command
• Reliability: Failed URLs don't stop the entire batch
• Usability: Install once, run anywhere with automatic dependency management

Testing

Tested with various scenarios including API failures, mixed success/failure batches, and different URL patterns from sites like modelcontextprotocol.io.

Files changed:

• generate-llmstxt.py - Core enhancements
• README.md - Updated documentation
• CHANGELOG.md - Detailed change log
• pyproject.toml - uv dependencies
• sample-urls.txt - Example file