Add comprehensive examples demonstrating workqueue use cases

[Copilot]

This PR adds two comprehensive, runnable examples to the repository demonstrating practical use cases of the workqueue package.

Examples Added

1. URL Crawl Frontier (examples/url-crawler/)

Demonstrates how to use workqueue as a crawl frontier for a web crawler with:

• Priority-based fetching: Never-fetched URLs get highest priority (1000), previously fetched URLs get lower priority (500)
• Politeness delays: 1-hour delay between fetches of the same URL using DelayedUntil
• Automatic expiration: URLs expire and are removed after 24 hours using ExpiresAt
• Concurrent workers: 3 worker goroutines simulate realistic crawler architecture
• Queue lifecycle: Demonstrates adding URLs, fetching with proper delays, re-queueing, and expiration handling

// Example: Adding a URL with priority and expiration
item := workqueue.WorkItem[string, URLInfo]{
    Key:       "https://example.com",
    Value:     urlInfo,
    Priority:  1000,
    ExpiresAt: time.Now().Add(24 * time.Hour),
}
queue.Put(ctx, item)

2. Function Queue with Worker Pool (examples/function-queue/)

Demonstrates storing and executing functions as work items with:

• Function storage pattern: Shows how to store functions (which aren't comparable) by using metadata as the queue value and a map for function lookup
• Worker pool: 5 concurrent goroutines consuming from a shared queue
• Priority execution: Higher priority tasks execute first (priorities ranging from 40-100)
• Diverse task types: 8 different example tasks (database backup, email notifications, report generation, etc.)
• Graceful shutdown: Workers detect when queue is empty and shut down cleanly

// Example: Storing a function as a work item
taskFunctions[taskID] = func() error {
    // Task logic here
    return nil
}
queue.Put(ctx, workqueue.WorkItem[int, TaskMetadata]{
    Key:      taskID,
    Value:    metadata,
    Priority: priority,
})

Documentation

Each example includes:

• Complete, runnable main.go with detailed comments
• Comprehensive README.md explaining concepts, how to run, and production adaptations
• Clear demonstration of workqueue features (Priority, DelayedUntil, ExpiresAt, Take, Put)
• Educational code showing best practices for common patterns

Both examples can be run immediately with go run main.go and produce clear, educational output showing the queue in action.

Related

This addresses the need for practical examples showing how to use workqueue for real-world use cases like crawl frontiers and background job processing systems.

Original prompt

Create an examples/ directory in the repository with two comprehensive examples demonstrating different use cases of the workqueue:

Example 1: URL Crawl Frontier (examples/url-crawler/)

Create a simple URL crawl frontier example that demonstrates:

• A queue of URLs to be fetched
• Priority based on time of fetch (older fetches = higher priority, or URLs that haven't been fetched yet get highest priority)
• Add a 1-hour delay after each URL is fetched before it can be fetched again
• URLs expire and are removed from the queue after 24 hours
• Demonstrate adding URLs, fetching them with appropriate delays, and handling expiration

The example should include:

• A complete, runnable Go program (main.go)
• Clear comments explaining the workqueue configuration
• Demonstration of adding URLs to the queue
• Workers that fetch URLs (can simulate fetching with time.Sleep)
• Proper handling of the 1-hour delay between fetches
• 24-hour expiration logic
• Example output showing the queue in action

Example 2: Function Queue with Worker Pool (examples/function-queue/)

Create an example demonstrating storing functions as work items:

• Store functions/closures as the value of work items (e.g., func() { doSomething() })
• Create a pool of goroutines (workers) that pull from the queue
• Each worker executes the function it retrieves
• Demonstrate different types of work functions being queued and executed

The example should include:

• A complete, runnable Go program (main.go)
• Clear comments explaining how functions are stored and executed
• Multiple example work functions with different behaviors
• A worker pool (e.g., 3-5 goroutines) consuming from the queue
• Proper synchronization and graceful shutdown
• Example output showing which workers are executing which tasks

Both examples should:

• Include a README.md explaining what the example demonstrates and how to run it
• Be self-contained and runnable with go run main.go
• Use the workqueue package effectively
• Include appropriate error handling
• Have clear, educational code comments

This pull request was created as a result of the following prompt from Copilot chat.

Create an examples/ directory in the repository with two comprehensive examples demonstrating different use cases of the workqueue:

Example 1: URL Crawl Frontier (examples/url-crawler/)

Create a simple URL crawl frontier example that demonstrates:

• A queue of URLs to be fetched
• Priority based on time of fetch (older fetches = higher priority, or URLs that haven't been fetched yet get highest priority)
• Add a 1-hour delay after each URL is fetched before it can be fetched again
• URLs expire and are removed from the queue after 24 hours
• Demonstrate adding URLs, fetching them with appropriate delays, and handling expiration

The example should include:

• A complete, runnable Go program (main.go)
• Clear comments explaining the workqueue configuration
• Demonstration of adding URLs to the queue
• Workers that fetch URLs (can simulate fetching with time.Sleep)
• Proper handling of the 1-hour delay between fetches
• 24-hour expiration logic
• Example output showing the queue in action

Example 2: Function Queue with Worker Pool (examples/function-queue/)

Create an example demonstrating storing functions as work items:

• Store functions/closures as the value of work items (e.g., func() { doSomething() })
• Create a pool of goroutines (workers) that pull from the queue
• Each worker executes the function it retrieves
• Demonstrate different types of work functions being queued and executed

The example should include:

• A complete, runnable Go program (main.go)
• Clear comments explaining how functions are stored and executed
• Multiple example work functions with different behaviors
• A worker pool (e.g., 3-5 goroutines) consuming from the queue
• Proper synchronization and graceful shutdown
• Example output showing which workers are executing which tasks

Both examples should:

• Include a README.md explaining what the example demonstrates and how to run it
• Be self-contained and runnable with go run main.go
• Use the workqueue package effectively
• Include appropriate error handling
• Have clear, educational code comments

---

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.