diff --git a/agent-workspace/domain-skills/google-autocomplete/suggest-api.md b/agent-workspace/domain-skills/google-autocomplete/suggest-api.md new file mode 100644 index 00000000..636ee2af --- /dev/null +++ b/agent-workspace/domain-skills/google-autocomplete/suggest-api.md @@ -0,0 +1,34 @@ +# Google Suggest — autocomplete API for keyword expansion + +Field-tested 2026-07-24. This is the fastest way to get real (not invented) long-tail keyword variants for SEO/keyword-research tasks — no browser needed, plain `http_get`/`urllib`. + +## Endpoint + +``` +https://suggestqueries.google.com/complete/search?client=firefox&hl=en&q= +``` + +`client=firefox` returns clean JSON (`["", ["suggestion 1", "suggestion 2", ...]]`) with no JSONP wrapper to strip — simpler than `client=chrome` which wraps in extra metadata. Suggestions come back in Google's own ranked order (most-searched-adjacent first, roughly). + +```python +import json, urllib.request, urllib.parse + +def autocomplete(q, hl="en"): + url = "https://suggestqueries.google.com/complete/search?client=firefox&hl=" + hl + "&q=" + urllib.parse.quote(q) + req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"}) + with urllib.request.urlopen(req, timeout=10) as r: + return json.loads(r.read().decode())[1] +``` + +No auth, no rate-limit hit even across dozens of sequential calls in one session (unlike `pytrends`/Google Trends, which 429s fast — see `google-trends/explore.md`). A `time.sleep(0.3)` between calls is plenty polite; not strictly required. + +## Expansion technique + +1. Seed term alone → gets the "canonical" top-10 completions. +2. Seed + single letter (`"schulte table a"`, `"schulte table b"`, ...) → cycles through a different top-10 for each letter, effectively unlocking Google's full completion list beyond the first 10 (Google caps each response at ~10 regardless of query). +3. Seed + modifier (`free`, `online`, `app`, `vs`, `how`, `best`) → surfaces intent-specific long tail (transactional vs informational vs comparison). +4. Empty result (`[]`) is itself a signal — the seed+modifier combo has essentially no real search volume/pattern (e.g. `"focus training how"` → `[]`, `"aim trainer cognitive"` → `[]`). + +## No volume data + +This endpoint returns *which strings people type*, not *how often*. It's a discovery tool, not a sizing tool — pair with Google Trends (relative-index comparison) and manual SERP checks (who ranks = competition signal) to get a fuller picture. See `google-trends/explore.md` for the comparison-chart trick. diff --git a/agent-workspace/domain-skills/google-trends/explore.md b/agent-workspace/domain-skills/google-trends/explore.md new file mode 100644 index 00000000..13bb43d8 --- /dev/null +++ b/agent-workspace/domain-skills/google-trends/explore.md @@ -0,0 +1,39 @@ +# Google Trends (trends.google.com) — explore/compare for keyword research + +Field-tested 2026-07-24. Useful for getting *relative* search-interest signal and Rising-query breakouts when you don't have Ahrefs/SEMrush access. + +## `pytrends` (unofficial API) gets rate-limited fast + +`pip install pytrends` works, but `related_queries()` / `interest_over_time()` frequently return `TooManyRequestsError: 429` even on a single cold call from a fresh IP — Google is aggressive about blocking the raw HTTP client's fingerprint. **Prefer driving the real trends.google.com UI via browser-harness** — a real Chrome session with normal headers/cookies does not get blocked, and you get the "Rising" breakout data pytrends struggles to fetch reliably. + +## URL pattern — single term + +``` +https://trends.google.com/trends/explore?q=&geo=US&hl=en +``` + +## URL pattern — compare up to 5 terms at once (this is the valuable trick) + +Comma-separate terms in `q=`, no URL-encoding needed for the comma itself (spaces still need `%20` or `+`): + +``` +https://trends.google.com/trends/explore?date=today%2012-m&geo=US&q=term one,term two,term three,term four,term five&hl=en +``` + +This renders one shared "Interest over time" line chart with all terms **normalized to the same 0-100 scale** — this is the single fastest way to sanity-check relative demand between a candidate keyword cluster (e.g. is "schulte table" bigger or smaller than "reaction time test"?). Hover a point on the chart (or read the tooltip that appears) to get exact per-term index values for that week. + +Below the chart, each term gets its own "Related queries" panel (Top / Rising toggle, defaults vary — some panels default to Rising, some to Top, seemingly based on data availability) and its own "Related topics" panel. Scroll down — in 5-term compare mode this is a long page (~5x the single-term page height), one section per term, in the same order as entered in `q=`. + +## Rendering timing + +The page is an SPA — after `new_tab()` + `wait_for_load()`, `document.body.scrollHeight` is still small (initial ~1000px shell). Sleep ~2-3s for the charts/panels to hydrate before `window.scrollTo(0, document.body.scrollHeight)` — the page grows to its full height (thousands of px in compare mode) only after data loads. Re-check `page_info()['ph']` growth as a signal that content has rendered before screenshotting. + +## Reading "Rising" data — this is what actually matters for keyword research + +"Rising" related queries show a `+N%` badge instead of a 0-100 bar — this is a breakout/new-demand signal, exactly what you want for finding low-competition emerging keywords. "Top" queries show a 0-100 relative-popularity bar instead (no growth signal, just current relative volume within that term's related set). + +**Caveat — ambiguous/small-base terms produce noisy Rising lists.** If a seed term has very low absolute volume, Google Trends' Rising algorithm surfaces near-random breakout queries unrelated to your topic (e.g. for `attention training`, real observed Rising results included "walmart near me" and "dog training tips" — pure noise from a tiny/ambiguous base, not signal). Cross-check Rising queries against the base term's own relative interest level (from the compare chart) before trusting them — low-index terms (near 0 on the shared chart) have unreliable Related-queries panels. + +## Screenshot-driven reading beats DOM scraping here + +Trends' DOM uses obfuscated/generated class names with no stable selectors worth hardcoding — this page changes often. Screenshot + read is faster and more robust than trying to write a `querySelectorAll` scraper for chart tooltips or the related-queries table. diff --git a/agent-workspace/domain-skills/humanbenchmark/scraping.md b/agent-workspace/domain-skills/humanbenchmark/scraping.md new file mode 100644 index 00000000..93772796 --- /dev/null +++ b/agent-workspace/domain-skills/humanbenchmark/scraping.md @@ -0,0 +1,25 @@ +# humanbenchmark.com — product scope scraping + +Field-tested 2026-07-24. + +## Full test catalog lives on the homepage, not a `/tests` index + +There is no `/tests` listing route — `https://humanbenchmark.com/tests` is a 404. The complete list of tests is rendered as cards directly on the homepage (`https://humanbenchmark.com/`), no pagination, no lazy-load/infinite-scroll (a `querySelectorAll` scan on first paint already captures all cards — scrolling isn't required). + +Extraction (real DOM, works via `js()`): + +```js +Array.from(document.querySelectorAll("a")).filter(a => a.querySelector("h3, h2")).map(a => ({ + title: a.querySelector("h3, h2").innerText.trim(), + href: a.href, + desc: a.querySelector("p") ? a.querySelector("p").innerText.trim() : "", +})) +``` + +As of this scrape, Human Benchmark ships exactly **8 tests**: Reaction Time (`/tests/reactiontime`), Sequence Memory (`/tests/sequence`), Aim Trainer (`/tests/aim`), Number Memory (`/tests/number-memory`), Verbal Memory (`/tests/verbal-memory`), Chimp Test (`/tests/chimp`), Visual Memory (`/tests/memory`), Typing (`/tests/typing`). URL slugs are inconsistent (`reactiontime` no hyphen vs `number-memory` hyphenated) — don't guess slugs, read them off the cards. + +Notably absent (useful for competitive-gap analysis): no Schulte Table, no Visual Search test, no Multiple Object Tracking (MOT) test — these are classic cognitive-science paradigms Human Benchmark has never shipped, despite otherwise covering most "attention/reflex" benchmark territory. + +## Site is a fast static-ish SPA + +`wait_for_load()` after `new_tab()` is sufficient — no extra sleep needed for the homepage card grid to be queryable. diff --git a/agent-workspace/domain-skills/toolify/search.md b/agent-workspace/domain-skills/toolify/search.md new file mode 100644 index 00000000..ee897d5d --- /dev/null +++ b/agent-workspace/domain-skills/toolify/search.md @@ -0,0 +1,29 @@ +# toolify.ai — search & scope + +Field-tested 2026-07-24 (keyword research for a non-AI SaaS product). + +## Search URL pattern + +Path-based, not query-string. The homepage search box (`input[name="toolifySearch"]`) posts to: + +``` +https://www.toolify.ai/search/ +``` + +e.g. `https://www.toolify.ai/search/focus%20training`. `?q=` query-string form (`/search?q=...`) does **not** work — it silently redirects/500s to the homepage. Always use the `/search/` path form. + +Page title becomes `"The best AI websites & AI tools - Toolify"` when the search resolves correctly — a quick way to confirm you didn't get redirected. + +## Scope warning: this is an AI-tools-only directory + +Toolify indexes ~30k *AI* products/SaaS (its own homepage states the count, e.g. "29953 AIs"). It is **not** a general product/app directory. Searching for a non-AI-tools niche (games, cognitive-training tools, browser games, etc.) returns semantically-matched but functionally unrelated AI SaaS products — e.g. searching `focus training` and `attention training` returned an AI photo editor, an AI running coach, an AI ranking-tracker SaaS, etc. None were actually competitors. + +**Do not use toolify.ai for keyword/competitor research outside the AI-tools space.** Confirm the target niche is actually "AI tool"-shaped (LLM wrapper, AI-powered SaaS, GPT store entry) before spending time here. For cognitive-training / games / consumer-web-tool niches, Google autocomplete + Trends + direct competitor site scraping is a better source. + +## Extraction + +Card titles are inside `h2`/`h3`/`h4` wrapped by an ``; a plain `querySelectorAll("a")` scan filtering for a heading child works, e.g.: + +```js +Array.from(document.querySelectorAll("a")).filter(a => a.querySelector("h2,h3,h4")) +```