updates for notebook server #32
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,26 +1,3 @@ | ||
| # [ArcGIS Notebook Server](https://enterprise.arcgis.com/en/notebook/) | ||
|
|
||
| This sample has moved to [Notebook](/Notebook) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,252 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## The following Python notebook represents code that would be called from a Notebook Server webhook receiver, triggered by an incoming webhook.\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "#### Run this cell to connect to your GIS and get started:" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 1, | ||
| "metadata": { | ||
| "trusted": true | ||
| }, | ||
| "outputs": [ | ||
| { | ||
| "name": "stderr", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "You are logged on as admin with an administrator role, proceed with caution.\n" | ||
| ] | ||
| } | ||
| ], | ||
| "source": [ | ||
| "from arcgis.gis import GIS\n", | ||
| "gis = GIS(\"home\")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "#### Functions to get the features and attributes from the /ExtractChanges API within the Feature Service" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": { | ||
| "trusted": true | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# When developing your notebook, use an example payload, one that would be similiar to a message that triggers the notebook\n", | ||
| "samplePayload = {\"serviceType\":\"FeatureServer\",\"changesUrl\":\"https://example.com/server/rest/services/Hosted/MyService/FeatureServer/extractChanges?serverGens=%5B1773230457695,1773244883393%5D\",\"name\":\"Posting\",\"id\":\"efab8ba4-6a0b-491e-8624-73a25e15ffb0\",\"folderName\":\"\",\"serviceName\":\"MyService\",\"events\":[{\"eventType\":\"FeaturesUpdated\",\"when\":1741189389713}]}\n", | ||
| "# After putting your notebook into production, this cell can be deleted" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": { | ||
| "trusted": true | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Functions to get the features and attributes from the /ExtractChanges API within the Feature Service\n", | ||
| "\n", | ||
| "from urllib.parse import urlparse\n", | ||
| "from urllib.parse import parse_qs\n", | ||
| "from urllib.parse import unquote\n", | ||
| "import time\n", | ||
| "\n", | ||
| "def parseResults(resultJSONURL, params=None):\n", | ||
| "\n", | ||
| " resultJSON = gis._con.post(resultJSONURL, params, files=None)\n", | ||
| " if \"edits\" in resultJSON:\n", | ||
| " print(\"Found {} layers with edits in the changes file\".format(len(resultJSON['edits'])))\n", | ||
| " else:\n", | ||
| " print(\"No edits found\")\n", | ||
| "\n", | ||
| " return resultJSON\n", | ||
| "\n", | ||
| "\n", | ||
| "def waitForStauts(statusURL, params=None):\n", | ||
| "\n", | ||
| " statusPayload = {\"status\":\"foo\"}\n", | ||
| " counter = 0 \n", | ||
| "\n", | ||
| " while statusPayload[\"status\"].upper() != \"COMPLETED\":\n", | ||
| " statusPayload = gis._con.post(statusURL, params)\n", | ||
| " counter += 1\n", | ||
| " time.sleep(1)\n", | ||
| " if counter == 20:\n", | ||
| " print(\"No results after 20 seconds, something is probably wrong\")\n", | ||
| " continue\n", | ||
| " \n", | ||
| " try:\n", | ||
| " return statusPayload['resultUrl']\n", | ||
| " except Exception:\n", | ||
| " print(\"Failed to get a resultURL\")\n", | ||
| "\n", | ||
| "\n", | ||
| "def getExtractChanges(changesURL):\n", | ||
| " \n", | ||
| " # Depending on the type of changes happening in your feature service,\n", | ||
| " # the following parameters can be updated to get back specific types\n", | ||
| " # of changes. Below is set to return Inserts (add) only.\n", | ||
| "\n", | ||
| " parse = urlparse(changesURL)\n", | ||
| " extract_params = {\n", | ||
| " \"serverGens\": parse_qs(parse.query)['serverGens'][0],\n", | ||
| " \"returnInserts\": \"true\",\n", | ||
| " \"returnUpdates\": \"false\",\n", | ||
| " \"returnDeletes\": \"false\",\n", | ||
| " \"returnAttachments\": \"false\",\n", | ||
| " \"returnAttachmentsDataByUrl\": \"false\"\n", | ||
| " }\n", | ||
| "\n", | ||
| " changePayload = gis._con.post(changesURL.split(\"?\")[0], extract_params)\n", | ||
| "\n", | ||
| " if \"statusUrl\" in changePayload:\n", | ||
| " return changePayload['statusUrl']\n", | ||
| " else:\n", | ||
| " print(\"No status URL found, cannot proceed\")\n", | ||
| "\n", | ||
| "def doExtractChanges(changesURL):\n", | ||
| "\n", | ||
| " statusURL = getExtractChanges(changesURL)\n", | ||
| "\n", | ||
| " resultJSONURL = waitForStauts(statusURL)\n", | ||
| "\n", | ||
| " changeResults = parseResults(resultJSONURL)\n", | ||
| "\n", | ||
| " return changeResults" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### Get the changeURL from the payload and use that to get the change JSON via Extract Change" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": { | ||
| "trusted": true | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Get the changeURL from the payload and use that to get the change JSON via Extract Changes\n", | ||
| "import json\n", | ||
| "\n", | ||
| "changeInfo = doExtractChanges(samplePayload['changesUrl'])\n", | ||
| "\n", | ||
| "# Comment out the above and uncomment this line to use the code in production, where the payload is coming from the \n", | ||
| "# trigger message instead of a sample payload\n", | ||
| "# 12.1+ example\n", | ||
| "#changeInfo = doExtractChanges(webhookPayload['changesUrl'])\n", | ||
| "# 11.5-12.0 example\n", | ||
| "#changeInfo = doExtractChanges(json.loads(webhookPayload['changesUrl']))\n", | ||
| "\n", | ||
| "print(changeInfo)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### Look through the edits to determine if further action is required" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 6, | ||
| "metadata": { | ||
| "trusted": true | ||
| }, | ||
| "outputs": [ | ||
| { | ||
| "name": "stdout", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "[{'attributes': {'rankcondition': 2, 'globalid': '{4F95E7F8-8065-419C-AB38-C2E1E37A70BF}', 'reviewed': 'Yes', 'collectorid': 2, 'objectid': 55}, 'geometry': {'x': 368120.28500000015, 'y': 5031795.9815, 'z': 0}}, {'attributes': {'rankcondition': 3, 'globalid': '{32CE8264-899B-4CB5-BFCE-3A73C351CDF2}', 'reviewed': 'Yes', 'collectorid': 2, 'objectid': 78}, 'geometry': {'x': 368093.27809999976, 'y': 5031804.243799999, 'z': 0}}, {'attributes': {'rankcondition': 3, 'globalid': '{7D86B33B-78C2-48A0-8686-D420780CA8EA}', 'reviewed': 'Yes', 'collectorid': 2, 'objectid': 85}, 'geometry': {'x': 368086.0548999999, 'y': 5031809.3959, 'z': 0}}]\n" | ||
| ] | ||
| } | ||
| ], | ||
| "source": [ | ||
| "updates = []\n", | ||
| "for e in changeInfo['edits']:\n", | ||
| " if 'updates' in e['features']:\n", | ||
| " updates = e['features']['updates']\n", | ||
| " break\n", | ||
| "\n", | ||
| "if not updates:\n", | ||
| " print(\"No new updates\")\n", | ||
| "else:\n", | ||
| " print(updates)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### Do something with the edits...." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": { | ||
| "trusted": true | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "if updates: \n", | ||
| " # Iterate through each new posted feature \n", | ||
| " for a in updates: \n", | ||
| " for u in updates:\n", | ||
| " # Eg. If attribute \"reviewed\" == yes, do something...\n", | ||
| " if u['attributes']['reviewed'] == 'Yes':\n", | ||
| " # do something...\n", | ||
| " pass" | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "esriNotebookRuntime": { | ||
| "notebookRuntimeName": "ArcGIS Notebook Python 3 Advanced", | ||
| "notebookRuntimeVersion": "14.0" | ||
| }, | ||
| "kernelspec": { | ||
| "display_name": "Python 3 (ipykernel)", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.13.10" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 4 | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||
|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,59 @@ | ||||||||
| # [ArcGIS Notebook Server](https://enterprise.arcgis.com/en/notebook/) | ||||||||
|
|
||||||||
| ## Organization and Service webhooks to Notebook Server | ||||||||
|
|
||||||||
| Both organization and service webhooks can be directed to ArcGIS Notebook Server. The incoming webhook message will ultimately trigger a Python notebook, in turn, running Python code. This pattern enjoys the benefits of leveraging the Notebook Server infrastructure with the flexibility of developer workflows, supported by the ArcGIS for Python API. | ||||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||||
|
|
||||||||
| ### Notebook Server webhook receiver | ||||||||
|
|
||||||||
| The release of ArcGIS Notebook Server 11.5 introduced a webhook receiver. Webhook receivers could be enabled, scoped with specific privileges to run and access specific Python code. These receivers provide a payload URL that can be used with ArcGIS webhooks. Prior to the 11.5 release, [Organization webhooks](https://enterprise.arcgis.com/en/notebook/latest/administer/windows/automate-notebook-workflows-with-webhook-events.htm#ESRI_SECTION1_E61B62B25F394778BCE814AA1298083C) could trigger a Python notebook directly (explanation in the [legacy section below](#legacy-pre-arcgis-enterprise-115)). While this workflow is still applicable, the preferred method, which also supports [feature and geoprocessing service](https://enterprise.arcgis.com/en/notebook/latest/administer/windows/automate-notebook-workflows-with-webhook-events.htm#ESRI_SECTION1_AA2CB1134C294DDD830C28BA007DDB8A) webhooks is to create a webhook receiver and use this webhook URL in the webhook creation. | ||||||||
|
|
||||||||
| [Consult the detailed documentation](https://enterprise.arcgis.com/en/notebook/latest/administer/windows/automate-notebook-workflows-with-webhook-events.htm#ESRI_SECTION1_AA2CB1134C294DDD830C28BA007DDB8A) to setup a webhook receiver in Notebook Server. For reference, the main requirements involve: | ||||||||
| - An existing Python Notebook (code which is run when the webhook is triggered) | ||||||||
| - Creating an API key scoped with Notebook privileges and permission to the Notebook item and any item the Python code references | ||||||||
| - Creating the receiver within Notebook Server with an exactly 32 character secret key. | ||||||||
| - Creation of the webhook, pointing the payload URL to the newly created webhook receiver and supplying the matching 32 character secret key | ||||||||
|
|
||||||||
| ### Python code as a webhook receiver | ||||||||
|
|
||||||||
| The Python code will be triggered when the webhook payload is received by the webhook receiver. In most instances, the incoming payload will be a necessary part of the workflow; the code will act upon the information inside the payload. This payload is automatically injected into the Python notebook as a variable, `webhookPayload`. | ||||||||
|
|
||||||||
| [!NOTE] 11.5 - 12.0 requires loading the payload as JSON: `json.loads(webhookPayload)`. 12.1 and onwards the `webhookPayload` is injected into the Notebook as a JSON object. | ||||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||||
|
|
||||||||
| The sample notebook, [Example_FS_ExtractChanges.ipynb](/Notebook/Example_FS_ExtractChanges.ipynb) demonstrates the use of the `webhookPayload` variable. In addition, the sample makes use of a `samplePayload`, a practice of capturing an example webhook payload and developing code against that. Prior to putting the Notebook code into production, the switch from `samplePayload` to `webhookPayload` is necessary. | ||||||||
|
|
||||||||
| The example code shows the process of using the `changesUrl` from a feature service payload to work through the [extract changes](https://developers.arcgis.com/rest/services-reference/enterprise/extract-changes-feature-service/) workflow. These steps get the actual change JSON from the feature service, allowing the code to interact with the actual changes that occurred which cause the webhook to trigger. The high-level workflow can be described as: | ||||||||
| - Making a call to the /extractChanges API of a feature service, supplying **serverGens** and return properties | ||||||||
| - This initial call returns a JobID. The job needs to be polled until it has **completed** | ||||||||
| - From the **results** of the job, use the supplied URL to fetch the `.json` file with the x/y and attributes of the changes. | ||||||||
|
|
||||||||
| ## Legacy (Pre ArcGIS Enterprise 11.5) | ||||||||
|
|
||||||||
| **Only use these steps if your ArcGIS Enterprise is older than 11.5** | ||||||||
|
|
||||||||
| Starting with ArcGIS Enterprise 10.9, ArcGIS Notebook Server can be configured as a webhook receiver for Portal webhooks. The webhook call will execute the code within a given notebook. Using Notebook Server is a great alternative to setting up an independent server as you can leverage many different Python modules, including the [ArcGIS API for Python](https://developers.arcgis.com/python/). | ||||||||
|
|
||||||||
| ### Setup | ||||||||
|
|
||||||||
| Prior to setting up the webhook, you must have an existing notebook (itemID). Create your Portal webhook by *omitting* the payload URL and using the following JSON for the **config** parameter with your ItemID. | ||||||||
|
|
||||||||
| `{ | ||||||||
| "deactivationPolicy": { | ||||||||
| "numberOfFailures": 5, | ||||||||
| "daysInPast": 5 | ||||||||
| }, | ||||||||
| "properties": { | ||||||||
| "federatedServer": { | ||||||||
| "itemId": "<Notebook item id to be executed>", | ||||||||
| "tokenTypeToSend": "owner", | ||||||||
| "tokenExpirationTimeMinutes": 10 | ||||||||
| } | ||||||||
| } | ||||||||
| } | ||||||||
| ` | ||||||||
|
|
||||||||
| Consult the [documentation](https://enterprise.arcgis.com/en/notebook/11.3/administer/windows/automate-notebook-execution.htm#ESRI_SECTION1_E61B62B25F394778BCE814AA1298083C) for detailed steps. | ||||||||
|
|
||||||||
| **Note**: Prior to ArcGIS Enterprise 11.5, Notebook Server is only available for Organization webhooks. It could not be the receiver for neither ArcGIS Enterprise or ArcGIS Online Hosted service based webhooks. | ||||||||
|
|
||||||||
|
|
||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.