Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions agent-workspace/domain-skills/google-autocomplete/suggest-api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Google Suggest — autocomplete API for keyword expansion

Field-tested 2026-07-24. This is the fastest way to get real (not invented) long-tail keyword variants for SEO/keyword-research tasks — no browser needed, plain `http_get`/`urllib`.

## Endpoint

```
https://suggestqueries.google.com/complete/search?client=firefox&hl=en&q=<url-encoded query>
```

`client=firefox` returns clean JSON (`["<query>", ["suggestion 1", "suggestion 2", ...]]`) with no JSONP wrapper to strip — simpler than `client=chrome` which wraps in extra metadata. Suggestions come back in Google's own ranked order (most-searched-adjacent first, roughly).

```python
import json, urllib.request, urllib.parse

def autocomplete(q, hl="en"):
url = "https://suggestqueries.google.com/complete/search?client=firefox&hl=" + hl + "&q=" + urllib.parse.quote(q)
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
with urllib.request.urlopen(req, timeout=10) as r:
return json.loads(r.read().decode())[1]
```

No auth, no rate-limit hit even across dozens of sequential calls in one session (unlike `pytrends`/Google Trends, which 429s fast — see `google-trends/explore.md`). A `time.sleep(0.3)` between calls is plenty polite; not strictly required.

## Expansion technique

1. Seed term alone → gets the "canonical" top-10 completions.
2. Seed + single letter (`"schulte table a"`, `"schulte table b"`, ...) → cycles through a different top-10 for each letter, effectively unlocking Google's full completion list beyond the first 10 (Google caps each response at ~10 regardless of query).
3. Seed + modifier (`free`, `online`, `app`, `vs`, `how`, `best`) → surfaces intent-specific long tail (transactional vs informational vs comparison).
4. Empty result (`[]`) is itself a signal — the seed+modifier combo has essentially no real search volume/pattern (e.g. `"focus training how"` → `[]`, `"aim trainer cognitive"` → `[]`).

## No volume data

This endpoint returns *which strings people type*, not *how often*. It's a discovery tool, not a sizing tool — pair with Google Trends (relative-index comparison) and manual SERP checks (who ranks = competition signal) to get a fuller picture. See `google-trends/explore.md` for the comparison-chart trick.
39 changes: 39 additions & 0 deletions agent-workspace/domain-skills/google-trends/explore.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Google Trends (trends.google.com) — explore/compare for keyword research

Field-tested 2026-07-24. Useful for getting *relative* search-interest signal and Rising-query breakouts when you don't have Ahrefs/SEMrush access.

## `pytrends` (unofficial API) gets rate-limited fast

`pip install pytrends` works, but `related_queries()` / `interest_over_time()` frequently return `TooManyRequestsError: 429` even on a single cold call from a fresh IP — Google is aggressive about blocking the raw HTTP client's fingerprint. **Prefer driving the real trends.google.com UI via browser-harness** — a real Chrome session with normal headers/cookies does not get blocked, and you get the "Rising" breakout data pytrends struggles to fetch reliably.

## URL pattern — single term

```
https://trends.google.com/trends/explore?q=<term>&geo=US&hl=en
```

## URL pattern — compare up to 5 terms at once (this is the valuable trick)

Comma-separate terms in `q=`, no URL-encoding needed for the comma itself (spaces still need `%20` or `+`):

```
https://trends.google.com/trends/explore?date=today%2012-m&geo=US&q=term one,term two,term three,term four,term five&hl=en

@cubic-dev-ai cubic-dev-ai Bot Jul 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The example URL contains literal spaces in the q= parameter (term one,term two,...), but the preceding line explicitly states that spaces need %20 or + encoding. This contradiction will confuse readers and many HTTP clients will reject or re-encode the URL unpredictably. Encode the spaces to match your own guidance, e.g. q=term%20one,term%20two,term%20three,term%20four,term%20five.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At agent-workspace/domain-skills/google-trends/explore.md, line 20:

<comment>The example URL contains literal spaces in the `q=` parameter (`term one,term two,...`), but the preceding line explicitly states that spaces need `%20` or `+` encoding. This contradiction will confuse readers and many HTTP clients will reject or re-encode the URL unpredictably. Encode the spaces to match your own guidance, e.g. `q=term%20one,term%20two,term%20three,term%20four,term%20five`.</comment>

<file context>
@@ -0,0 +1,39 @@
+Comma-separate terms in `q=`, no URL-encoding needed for the comma itself (spaces still need `%20` or `+`):
+
+```
+https://trends.google.com/trends/explore?date=today%2012-m&geo=US&q=term one,term two,term three,term four,term five&hl=en
+```
+
</file context>
Suggested change
https://trends.google.com/trends/explore?date=today%2012-m&geo=US&q=term one,term two,term three,term four,term five&hl=en
https://trends.google.com/trends/explore?date=today%2012-m&geo=US&q=term%20one,term%20two,term%20three,term%20four,term%20five&hl=en
Fix with cubic

```

This renders one shared "Interest over time" line chart with all terms **normalized to the same 0-100 scale** — this is the single fastest way to sanity-check relative demand between a candidate keyword cluster (e.g. is "schulte table" bigger or smaller than "reaction time test"?). Hover a point on the chart (or read the tooltip that appears) to get exact per-term index values for that week.

Below the chart, each term gets its own "Related queries" panel (Top / Rising toggle, defaults vary — some panels default to Rising, some to Top, seemingly based on data availability) and its own "Related topics" panel. Scroll down — in 5-term compare mode this is a long page (~5x the single-term page height), one section per term, in the same order as entered in `q=`.

## Rendering timing

The page is an SPA — after `new_tab()` + `wait_for_load()`, `document.body.scrollHeight` is still small (initial ~1000px shell). Sleep ~2-3s for the charts/panels to hydrate before `window.scrollTo(0, document.body.scrollHeight)` — the page grows to its full height (thousands of px in compare mode) only after data loads. Re-check `page_info()['ph']` growth as a signal that content has rendered before screenshotting.

## Reading "Rising" data — this is what actually matters for keyword research

"Rising" related queries show a `+N%` badge instead of a 0-100 bar — this is a breakout/new-demand signal, exactly what you want for finding low-competition emerging keywords. "Top" queries show a 0-100 relative-popularity bar instead (no growth signal, just current relative volume within that term's related set).

**Caveat — ambiguous/small-base terms produce noisy Rising lists.** If a seed term has very low absolute volume, Google Trends' Rising algorithm surfaces near-random breakout queries unrelated to your topic (e.g. for `attention training`, real observed Rising results included "walmart near me" and "dog training tips" — pure noise from a tiny/ambiguous base, not signal). Cross-check Rising queries against the base term's own relative interest level (from the compare chart) before trusting them — low-index terms (near 0 on the shared chart) have unreliable Related-queries panels.

## Screenshot-driven reading beats DOM scraping here

Trends' DOM uses obfuscated/generated class names with no stable selectors worth hardcoding — this page changes often. Screenshot + read is faster and more robust than trying to write a `querySelectorAll` scraper for chart tooltips or the related-queries table.
25 changes: 25 additions & 0 deletions agent-workspace/domain-skills/humanbenchmark/scraping.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# humanbenchmark.com — product scope scraping

Field-tested 2026-07-24.

## Full test catalog lives on the homepage, not a `/tests` index

There is no `/tests` listing route — `https://humanbenchmark.com/tests` is a 404. The complete list of tests is rendered as cards directly on the homepage (`https://humanbenchmark.com/`), no pagination, no lazy-load/infinite-scroll (a `querySelectorAll` scan on first paint already captures all cards — scrolling isn't required).

Extraction (real DOM, works via `js()`):

```js
Array.from(document.querySelectorAll("a")).filter(a => a.querySelector("h3, h2")).map(a => ({
title: a.querySelector("h3, h2").innerText.trim(),
href: a.href,
desc: a.querySelector("p") ? a.querySelector("p").innerText.trim() : "",
}))
```

As of this scrape, Human Benchmark ships exactly **8 tests**: Reaction Time (`/tests/reactiontime`), Sequence Memory (`/tests/sequence`), Aim Trainer (`/tests/aim`), Number Memory (`/tests/number-memory`), Verbal Memory (`/tests/verbal-memory`), Chimp Test (`/tests/chimp`), Visual Memory (`/tests/memory`), Typing (`/tests/typing`). URL slugs are inconsistent (`reactiontime` no hyphen vs `number-memory` hyphenated) — don't guess slugs, read them off the cards.

Notably absent (useful for competitive-gap analysis): no Schulte Table, no Visual Search test, no Multiple Object Tracking (MOT) test — these are classic cognitive-science paradigms Human Benchmark has never shipped, despite otherwise covering most "attention/reflex" benchmark territory.

## Site is a fast static-ish SPA

`wait_for_load()` after `new_tab()` is sufficient — no extra sleep needed for the homepage card grid to be queryable.
29 changes: 29 additions & 0 deletions agent-workspace/domain-skills/toolify/search.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# toolify.ai — search & scope

Field-tested 2026-07-24 (keyword research for a non-AI SaaS product).

## Search URL pattern

Path-based, not query-string. The homepage search box (`input[name="toolifySearch"]`) posts to:

```
https://www.toolify.ai/search/<url-encoded query>
```

e.g. `https://www.toolify.ai/search/focus%20training`. `?q=` query-string form (`/search?q=...`) does **not** work — it silently redirects/500s to the homepage. Always use the `/search/<query>` path form.

Page title becomes `"The best <query> AI websites & AI tools - Toolify"` when the search resolves correctly — a quick way to confirm you didn't get redirected.

## Scope warning: this is an AI-tools-only directory

Toolify indexes ~30k *AI* products/SaaS (its own homepage states the count, e.g. "29953 AIs"). It is **not** a general product/app directory. Searching for a non-AI-tools niche (games, cognitive-training tools, browser games, etc.) returns semantically-matched but functionally unrelated AI SaaS products — e.g. searching `focus training` and `attention training` returned an AI photo editor, an AI running coach, an AI ranking-tracker SaaS, etc. None were actually competitors.

**Do not use toolify.ai for keyword/competitor research outside the AI-tools space.** Confirm the target niche is actually "AI tool"-shaped (LLM wrapper, AI-powered SaaS, GPT store entry) before spending time here. For cognitive-training / games / consumer-web-tool niches, Google autocomplete + Trends + direct competitor site scraping is a better source.

## Extraction

Card titles are inside `h2`/`h3`/`h4` wrapped by an `<a>`; a plain `querySelectorAll("a")` scan filtering for a heading child works, e.g.:

```js
Array.from(document.querySelectorAll("a")).filter(a => a.querySelector("h2,h3,h4"))

@cubic-dev-ai cubic-dev-ai Bot Jul 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The documented extraction snippet only returns matching <a> nodes, not the card titles it claims to extract. Consumers that use this result as a list of titles will receive DOM elements instead of strings; mapping each anchor to its heading text (and optionally its href) would make the example usable.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At agent-workspace/domain-skills/toolify/search.md, line 28:

<comment>The documented extraction snippet only returns matching `<a>` nodes, not the card titles it claims to extract. Consumers that use this result as a list of titles will receive DOM elements instead of strings; mapping each anchor to its heading text (and optionally its href) would make the example usable.</comment>

<file context>
@@ -0,0 +1,29 @@
+Card titles are inside `h2`/`h3`/`h4` wrapped by an `<a>`; a plain `querySelectorAll("a")` scan filtering for a heading child works, e.g.:
+
+```js
+Array.from(document.querySelectorAll("a")).filter(a => a.querySelector("h2,h3,h4"))
+```
</file context>
Fix with cubic

```