Iterates through all pages in Sandboxes and hits rate limits

[arjunsatarkar]

When downloading a post in Sandboxes, this code (I think)

glowpub/src/main.rs

Line 223 in 6e1abc2

let board_posts = BoardPosts::get_all_cached(board.id, !use_cache)

causes glowpub to iterate through all 100+ pages of posts in that continuity just as part of deciding the file name. This often hits rate limits and cases multi-minute backoff after the post has already been fetched. It'd be nice to be able to skip this logic.

(After further thought, this might be cached and therefore only cause an issue on the first instance of downloading such a post?)

[Hate9]

Wow, it sure does look like it does that. How did that get in there?

[Hate9]

Oh dear, yeah I see - looks like it's just so we can have properly zero-padded post-section-orders? That's a way to do that but it sure is inefficient.

[Hate9]

Post already contains all the section info we need, so it looks like we're just using it for the number of posts in the section.

[Hate9]

I think the solution will turn out to be "don't worry about padding unless we're calling this from the download-whole-board code"

[QuartzLibrary]

Ah, classic. We care caching the response, but you need to opt into using it, and the setup is simple enough that there are a bunch of cases were that's not helpful (besides the initial wait).

We'll want the names to be exactly the same whether you are downloading a board or single post, and I am loath to give up correctly sorted posts, so the easy fix here is to just special-case sandboxes. Are there many boards of problematic size?

[Hate9]

Probably not? I do kinda hate that solution though.

[QuartzLibrary]

Special-casing seems correct to me. It's what essentially happens upstream, and we should match that behavior.
Separately, caching and execution could be improved, but that's more involved and the easy fix fixes this issue.