Support large datasets with pagination across all server endpoints

[yedidyakfir]

Summary

Audit and update all server API endpoints to handle large amounts of data efficiently using pagination. Currently, endpoints may return unbounded result sets, which can cause performance degradation, timeouts, and high memory usage when data grows.

Features / Details

• Audit all existing server endpoints that return lists/collections of data
• Identify endpoints that lack pagination support
• Implement cursor-based or offset-based pagination for each endpoint
• Ensure consistent pagination patterns across the API (e.g., limit, offset, cursor, next_page)
• Add appropriate response metadata (total count, page info, has_next)
• Handle edge cases: empty results, last page, invalid page parameters

Use Cases

• Users with hundreds or thousands of tasks/workflows can browse data without performance issues
• Frontend can implement infinite scroll or paginated views
• API consumers can iterate through large datasets without timeouts or memory pressure

Implementation Considerations

• Choose a consistent pagination strategy (cursor-based preferred for scalability)
• Ensure database queries use proper LIMIT/OFFSET or cursor-based filtering
• Consider default and maximum page sizes
• Backwards compatibility: existing endpoints should still work without pagination params (with sensible defaults)
• Add pagination support to any relevant query/filter endpoints

List tasks

• Audit all server endpoints and document which ones return lists of data
• Design a consistent pagination interface (request params + response format)
• Implement pagination for each identified endpoint
• Update frontend to consume paginated responses
• Add tests for pagination edge cases (empty, single page, multi-page, invalid params)
• Update API documentation with pagination usage