Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion specs/03-endpoints.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,7 +185,7 @@ refusal.
| `search_url(url)` | `ArtifactInstance.search_url` |
| `search_scans(hash_)` | `ArtifactInstance.list_scans` |
| `search_by_metadata(query, include=None, exclude=None, ips=None, urls=None, domains=None)` | `Metadata.get` |
| `iocs_by_hash(hash_type, hash_value, hide_known_good=False, beta=False)` | `IOC.iocs_by_hash` |
| `iocs_by_hash(hash_type, hash_value, hide_known_good=False, beta=False)` | `IOC.iocs_by_hash` — `hash_type` is `sha256`, `sha1` or `md5`, passed through unvalidated. The server resolves a sha1/md5 to one artifact (artifact-index specs/20-ioc-by-hash.md); one naming several is refused with a machine-readable `AMBIGUOUS_HASH` error (specs/05). Against a server predating that change a sha1/md5 is a plain 400 |
| `search_by_ioc(ip=None, domain=None, ttp=None, imphash=None, with_artifacts=False)` | `IOC.ioc_search` — the reverse IOC search. By default each item is an `IOC` whose `json` is a bare sha256 string, and the request carries no `with_artifacts`, so it is byte-compatible with the old contract. `with_artifacts=True` (4.7.0) sends `with_artifacts=1` (an int: the session renders a bool as `'True'`, which the server's boolean parser refuses) and yields `Metadata` resources instead — metadata-search rows trimmed to an include set the SERVER owns (today `artifact.*`; `scan.first_seen`, `scan.first_scan.created`, `scan.last_seen`, `scan.detections`, `scan.mimetype`; `scan.latest_scan.{polyscore,artifact_instance_id,created}`; `scan.filename`, `scan.url`; `polyunite.malware_family`; `hash.ssdeep`, `hash.tlsh`). So `first_seen`, `last_scanned`, the detection counts, mimetypes, filenames, `ssdeep` and `tlsh` are populated; attributes read from outside the set (the `strings.*` IOC lists, for instance) parse as `None` or empty. The SDK does not restate the set: a server-side change to it shows up here without an SDK release. Against a server that predates the parameter the flag is ignored upstream and the sha256 strings fail to parse as `Metadata` with a `TypeError` — so this SDK version must not be released ahead of that server. The SDK does not validate the terms. With `with_artifacts=True` and none of ip/domain/ttp/imphash the server answers 400, which surfaces as the usual typed exception; without the flag a bare call behaves exactly as before. |
| `check_known_hosts(ips=[], domains=[])` | `IOC.check_known_hosts` |
| `live_feed(since=None, …, livescan_id=None, max_results=None)` | `LiveHuntResult.list` — `livescan_id` scopes the feed to one live hunt (the hunt-page per-ruleset feed); `since` is in **SECONDS** (the server converts with `timedelta(seconds=since)`; the 3.x/4.x docstring said minutes and was wrong), and absent-or-`0` means no time filter at all — the server applies it on a truthiness test; `max_results` bounds how many results the generator yields — `None`/`0`/negative means no bound; it does not alter the request |
Expand Down
1 change: 1 addition & 0 deletions specs/04-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ How the test suite is organised. Three layers: pure unit tests (no HTTP at all
- `test/ioc_search_test.py` — `search_by_ioc(with_artifacts=)`: pure-unit request shape (the int flag, the unchanged default and bare call), pass-through on both client methods, and a dual-transport (`ClientTestCase`) respx parse of an artifact row — a stand-in until the e2e stack serves the parameter (see `99-open-questions.md`).
- `test/historical_window_test.py` — `historical_create(window_days=)` and `HistoricalHunt.window_days`: pure-unit request shape (the int in the body, the key absent for `None`, no client-side validation) and parse (int, explicit null, missing key — on `HistoricalHunt` and `HistoricalHuntList`), plus dual-transport (`ClientTestCase`) respx cases for the body each client sends — respx because VCR does not match bodies. What the server returns is pinned live by `test_historical_window_days` / `test_async_historical_window_days` (a hunt with a window and one without, read back on create, get and list).
- `test/ruleset_favorite_respx_test.py` — dual-transport (`ClientTestCase`) respx suite for the favorite toggle: the `FAVORITE_LIMIT` refusal envelope and the query/body split.
- `test/ioc_by_hash_respx_test.py` — dual-transport respx suite for the `AMBIGUOUS_HASH` refusal of `iocs_by_hash` on both the `ioc` and `ioc-beta` routes (two artifacts sharing an md5 cannot be produced on the shared stack).
- `test/sandbox_file_qrcode_respx_test.py` — dual-transport (`ClientTestCase`) respx suite pinning that a QR-code `sandbox_file` submission uploads the image's bytes (the S3 PUT body) under its unrefanged basename. Respx rather than e2e because the stack's sandbox providers cannot reasonably process a QR image.

## Three test layers
Expand Down
2 changes: 2 additions & 0 deletions specs/05-downstream-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,8 @@ class TimeoutException(PolyswarmException): ...

`FAVORITE_LIMIT` (a refused ruleset star: the team's favorite budget is spent) has **no typed exception** — deliberately, since no pre-existing consumer needs re-routing the way `NotFoundException` did. It surfaces as the generic 400 `RequestException`, and the machine-readable path is the raw envelope: `exc.request.errors == {'code': 'FAVORITE_LIMIT', 'favorites_used': N, 'favorites_limit': M}`. The counters are the same pair a successful toggle returns, so a caller can render the "budget full" state from either outcome. Pinned by the dual-transport respx suite `test/ruleset_favorite_respx_test.py`.

`AMBIGUOUS_HASH` (a refused `iocs_by_hash` sha1/md5 that names several artifacts) follows the same rule — no typed exception: `exc.request.errors == {'code': 'AMBIGUOUS_HASH', 'hash_type': …, 'sha256s': [≤10], 'truncated': bool}`, and the caller retries with one of the sha256s. Pinned by `test/ioc_by_hash_respx_test.py`.

**What "known-good" means on the server, as of artifact-index's two-predicate model** (its `specs/05`): the refusal fires on the server's *current understanding* — a catalogue entry exists for the sha256 **and** that entry's extension passes an executable allow-list — evaluated live on every request. Two consequences worth knowing as a consumer: the same download can start working again with no action on your part (the entry is deleted, or the policy narrows), and `ArtifactInstance.state` can report the new value **`NOT_STORED`** — a submission the server declined as known-good at the time whose hash is no longer currently known-good, so nothing was ever stored for it and a fresh submit of the same file works. `state` is a plain string here; the SDK does not enumerate it, so a new member needs no SDK release.

Each `RequestException` subclass carries a `.request` attribute holding the originating `PolyswarmRequest` (set by `RequestException.__init__`). Callers can read `exc.request.status_code`, `exc.request.json` (the parsed response body after execution), `exc.request.input_json` (the body that was sent), `exc.request.request_parameters` (the request kwargs that built the call), etc. `InvalidValueException` and `TimeoutException` are client-side errors and don't carry a request descriptor.
Expand Down
16 changes: 16 additions & 0 deletions specs/99-open-questions.md
Original file line number Diff line number Diff line change
Expand Up @@ -226,3 +226,19 @@ invariant 1 exists to close.
the live `test_search_by_ioc` pair, record both cassettes against a fresh stack, and delete
the respx `IocSearchWithArtifactsTestCase` along with this entry. The builder and
pass-through tests in that module stay: they pin request shape, which a cassette cannot.

## `iocs_by_hash` by sha1/md5 is not pinned against a live server

**Status:** gap, blocked on the server leg reaching the e2e stack.

The server now resolves a sha1/md5 on both the `ioc` and `ioc-beta` routes, but no test
here exercises that happy path: `test/ioc_by_hash_respx_test.py` pins only the
`AMBIGUOUS_HASH` refusal, against a fabricated envelope (two artifacts sharing one md5
cannot be produced on the shared stack). The sha256 path stays covered live by
`test_iocs_by_hash` / `test_async_iocs_by_hash`.

**Action:** once the e2e stack serves sha1/md5, extend that live pair to also call
`iocs_by_hash('sha1', …)` and `iocs_by_hash('md5', …)` with the same artifact's hashes, on
`beta=False` and `beta=True`, asserting the same `ips`/`ttps` as the sha256 lookup; delete
and re-record both cassettes against a fresh stack, update the `04-testing.md` line, and
remove this entry. The respx refusal test stays.
5 changes: 3 additions & 2 deletions src/polyswarm_api/aio/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -447,8 +447,9 @@ async def iocs_by_hash(self, hash_type, hash_value, hide_known_good=False, beta=
"""
Retrieve IOCs by artifact hash

:param hash_type: Hash type of the provided hash_
:param hash_value: A list of fields to be included in the result (.* wildcards are accepted)
:param hash_type: ``sha256``, ``sha1`` or ``md5``
:param hash_value: The hash of the artifact. A sha1/md5 naming several artifacts
raises RequestException with ``errors['code'] == 'AMBIGUOUS_HASH'``
:return: Generator of IOC resources
"""
logger.info('Getting IOCs by hash %s:%s', hash_type, hash_value)
Expand Down
5 changes: 3 additions & 2 deletions src/polyswarm_api/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -491,8 +491,9 @@ def iocs_by_hash(self, hash_type, hash_value, hide_known_good=False, beta=False)
"""
Retrieve IOCs by artifact hash

:param hash_type: Hash type of the provided hash_
:param hash_value: A list of fields to be included in the result (.* wildcards are accepted)
:param hash_type: ``sha256``, ``sha1`` or ``md5``
:param hash_value: The hash of the artifact. A sha1/md5 naming several artifacts
raises RequestException with ``errors['code'] == 'AMBIGUOUS_HASH'``
:return: Generator of IOC resources
"""
logger.info("Getting IOCs by hash %s:%s", hash_type, hash_value)
Expand Down
37 changes: 37 additions & 0 deletions test/ioc_by_hash_respx_test.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
"""``iocs_by_hash`` by md5/sha1, respx-mocked on BOTH clients (``ClientTestCase`` —
specs/04-testing.md invariant 5).

The ``AMBIGUOUS_HASH`` refusal (artifact-index specs/20-ioc-by-hash.md) lives here
because the e2e stack cannot produce it: it needs two artifacts sharing one md5.
"""
import re

import pytest

from polyswarm_api import exceptions

from test._client_harness import BASE_URL, ClientTestCase

MD5 = 'c' * 32
CANDIDATES = ['d' * 64, 'e' * 64]


class IocsByHashTestCase(ClientTestCase):
def test_ambiguous_md5_refusal_is_machine_readable_on_both_routes(self):
# No typed exception (specs/05): the contract is the raw envelope at
# ``exc.request.errors``, whose candidates let the caller retry by sha256.
envelope = {
'status': 'error',
'result': 'The md5 matches several artifacts; query by sha256.',
'errors': {'code': 'AMBIGUOUS_HASH', 'hash_type': 'md5',
'sha256s': CANDIDATES, 'truncated': False},
}
for path, beta in (('ioc', False), ('ioc-beta', True)):
self.mock.add('GET', re.compile(re.escape(f'{BASE_URL}/{path}/md5/{MD5}') + r'(\?.*)?$'),
json=envelope, status=400)
with pytest.raises(exceptions.RequestException) as excinfo:
list(self.api.iocs_by_hash('md5', MD5, beta=beta))
errors = excinfo.value.request.errors
assert errors['code'] == 'AMBIGUOUS_HASH', path
assert errors['sha256s'] == CANDIDATES, path
assert 'code=AMBIGUOUS_HASH' in str(excinfo.value), path
Loading