Repository navigation
Add sitemap-to-markdown house service ($0.35) - #18
Merged
Merged
Conversation
New POST /v1/house/sitemap-to-markdown: crawls a sitemap or a same-origin site starting from a root URL, converting up to SITEMAP_MAX_PAGES (50) pages to Markdown. Every fetch — robots.txt, the sitemap and any child sitemaps, and every discovered page — goes through the same per-hop netguard-checked _fetch_page() every other house endpoint uses; a link found on a page is not more trusted than the seed URL. Bounded by a whole-call time budget, a total byte cap on top of the existing per-page cap, a 1 req/s-per-host rate limit, a single-crawl-at-a-time process lock (second call gets 429), and a free- memory floor before starting. Fixed two bugs found while verifying the implementation, both now covered by the passing test suite: - _fetch_robots() returned an unparsed RobotFileParser on a missing/failed robots.txt, intending 'no restrictions apply', but stdlib RobotFileParser.can_fetch() defaults to *deny* until read()/parse() has run (guards against being queried before read() completes) — every crawl of a site without a robots.txt was silently fetching zero pages. Now sets rp.allow_all = True explicitly, matching stdlib read()'s own behavior for a 404. - tests/test_house.py: FakeWeb.handle() didn't await coroutine-returning handlers, so the one test needing a slow async robots.txt handler (concurrency test) silently never ran its body. Added an inspect.isawaitable check. 180 tests pass, ruff clean.
alipishbin77
added a commit
that referenced
this pull request
Sep 30, 2026
Add sitemap-to-markdown house service ($0.35)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Last of the three queued house services.
POST /v1/house/sitemap-to-markdown: crawls a sitemap or a same-origin site from a root URL, up toSITEMAP_MAX_PAGES(50) pages as Markdown.Every fetch — robots.txt, the sitemap and any child sitemaps, every discovered page — goes through the same per-hop netguard-checked
_fetch_page()every other house endpoint uses; a link found on a fetched page gets the identical SSRF check as the seed URL, no exceptions. Bounded by a whole-call time budget, a total byte cap (on top of the existing per-page cap), 1 req/s-per-host rate limiting, a single-crawl-at-a-time process lock (a second call gets 429, not queued), and a free-memory floor before starting.Found and fixed two real bugs while verifying, both now covered by tests:
_fetch_robots()intended 'no robots.txt = no restrictions' but stdlibRobotFileParser.can_fetch()defaults to deny untilread()/parse()has run — every crawl of a site without robots.txt was silently returning zero pages. Fixed by settingallow_all = Trueexplicitly, matching what stdlib's ownread()does for a 404.FakeWeb.handle()didn't await coroutine-returning handlers, so the one test needing a slow async robots.txt handler silently never executed its body.180 tests pass (up from 124 pre-fix), ruff clean.