Repository navigation
Conversation
Canopy bills per request, so looking up every Amazon part of a large inventory up front is expensive, while most of them are never looked at again. A new option in the Canopy provider settings fetches a part's data only when somebody opens its info page, and only once. The page is rendered as usual and then asks the server for the data itself, so opening it is not slowed down by the request. It shows that the data is being fetched and reloads with the new data once it is there; if the daily limit is reached or Canopy fails, it says so instead. This applies to parts without an info provider reference that have an orderdetail linking to a product page of the configured Amazon marketplace. Only missing data is filled in (manufacturer, notes, pictures, parameters, a price if the part has none); nothing existing is changed, since nobody reviews the result. The part then gets a Canopy provider reference, which marks it as done and lets the normal info provider tools update it later. Costs are bounded by a second setting, the maximum number of such requests per 24 hours (default 100). A failed lookup is remembered for a day so a dead ASIN is not paid for on every view, and a short-lived lock keeps two viewers of the same part from both triggering a request.
Fetching data on the first view of a part only existed for Amazon parts via Canopy. The same reasoning applies to the web stores: looking up all parts of an inventory at once means hundreds of requests to the website of a small store, while most parts are never looked at again. So the mechanism is no longer tied to Canopy and can be switched on per provider. A new general info provider setting (or the PROVIDER_FETCH_ON_VIEW environment variable, a comma separated list of provider keys) selects the providers which do this. It is empty by default, so nothing changes unless it is set. Canopy keeps its own switch and its own daily limit in its settings and behaves as before; it is now just one user of the generic code. A part without a provider reference is recognized by its orderdetails: first by a product URL the provider handles, which gives the product ID directly; otherwise by a supplier named like the provider together with a supplier part number. In the second case the provider is searched for the number and a result is only used if its ID, part number or order number is exactly that number (ignoring case and a vendor prefix like ADA4062 or DEV-13975). Nobody reviews what is added, so a similar product is never taken: no exact result counts as a failed lookup. Deciding whether a fetch is needed never contacts a provider, as it runs on every page view. The safety properties stay the same: only missing data is filled in, the part gets the provider reference afterwards, a failed lookup (including a provider refusing or pausing its requests) is remembered for a day and reported on the page instead of raising an error, a lock prevents parallel lookups of one part, and the lookups per provider are capped per 24 hours (PROVIDER_FETCH_ON_VIEW_DAILY_LIMIT, default 100). The existing orderdetail the part was recognized by is completed with the product URL and prices instead of adding a second one for the same store. As stores pace their requests, a fetch can take much longer than a Canopy request: the session is released while it runs, so the user's other pages are not blocked, and the lock and the page wait longer.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1581 +/- ##
============================================
- Coverage 64.46% 64.13% -0.33%
- Complexity 10210 10342 +132
============================================
Files 762 766 +4
Lines 32711 32999 +288
============================================
+ Hits 21087 21164 +77
- Misses 11624 11835 +211 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Looking up every part of a large inventory at an info provider up front is slow, and for paid providers such as Canopy expensive, while most of those parts are never looked at again. This adds an opt-in mode where a part's data is fetched the first time somebody opens its info page, and only once.
How it behaves:
How a part is matched to a provider (the check done on every page view never contacts a provider):
URLHandlerInfoProviderInterface). Canopy is not a URL handler, so its ASIN extraction for the configured marketplace is handled as a special case.#, and a vendor prefix set apart by a separator or equal to the start of the provider's name). No exact result is a failed lookup; it never guesses. Several exact results are accepted only if they all carry the same MPN.Settings:
PROVIDER_FETCH_ON_VIEW=key1,key2) and "Max. lookups per day when viewing parts" (PROVIDER_FETCH_ON_VIEW_DAILY_LIMIT, default 100, 0 = unlimited; per provider, counting parts looked up). Empty by default, which is the stock behaviour.Implementation:
ProviderOnViewMatcher(which providers are enabled, matching, exact-result selection),ProviderOnViewFetcher(lock, daily cap, failure memo, filling missing data), a CSRF-protectedPOST /{id}/fetch_on_viewroute that requires read permission on the part (the administrator opts in with the setting, and only missing data is added), a small Stimulus controller and template for the notice, translations and a docs section. 37 unit tests cover the matching.The branch has two commits: the first adds the mode for Canopy only, the second generalises it to any provider.
Notes for review:
PartInfoRetrieveris final); it was exercised against real parts only.