diff --git a/modules/ROOT/pages/common/nav.adoc b/modules/ROOT/pages/common/nav.adoc index c1a182cb5..75f4b11a4 100644 --- a/modules/ROOT/pages/common/nav.adoc +++ b/modules/ROOT/pages/common/nav.adoc @@ -110,7 +110,7 @@ ***** link:{{navprefix}}/custom-action-payload[Callback response payload] ***** link:{{navprefix}}/edit-custom-action[Set the position of a custom action] ***** link:{{navprefix}}/add-action-viz[Add a local action to a visualization] -***** link:{{navprefix}}/add-action-worksheet[Add a local action to a worksheet] +***** link:{{navprefix}}/add-action-worksheet[Add a local action to a model] **** link:{{navprefix}}/code-based-custom-action[Code based custom actions] *** link:{{navprefix}}/customize-links[Customize links] *** link:{{navprefix}}/set-locale[Customize locale] @@ -236,16 +236,18 @@ include::generated/typedoc/CustomSideNav.adoc[] * link:{{navprefix}}/development-and-deployment[Deployment and integration] ** link:{{navprefix}}/development-and-deployment[Development and deployment] *** link:{{navprefix}}/thoughtspot-objects[ThoughtSpot objects overview] -*** link:{{navprefix}}/variables[Custom variables] -*** link:{{navprefix}}/git-integration[Deploy with Git] -**** link:{{navprefix}}/git-configuration[Configure Git integration] -**** link:{{navprefix}}/git-api[Version Control REST APIs] -**** link:{{navprefix}}/guid-mapping[GUID mapping] +*** link:{{navprefix}}/variables[Variables] +*** link:{{navprefix}}/parameterze-metdata[Parameterize metadata] *** link:{{navprefix}}/deploy-with-tml-apis[Deploy with TML APIs] +**** link:{{navprefix}}/git-provider-integration[Git provider integration] **** link:{{navprefix}}/modify-tml[TML modification] *** link:{{navprefix}}/publish-data-overview[Publish content to Orgs] -**** link:{{navprefix}}/parameterze-metdata[Parameterize metadata] **** link:{{navprefix}}/publish-to-orgs[Publish objects to Orgs] +*** link:{{navprefix}}/git-integration[Deploy with GitHub APIs (legacy)] +**** link:{{navprefix}}/git-configuration[Configure GitHub integration] +**** link:{{navprefix}}/git-api[GitHub REST APIs] +**** link:{{navprefix}}/guid-mapping[GUID mapping] + ** link:{{navprefix}}/multi-tenancy[Multi-tenancy] *** link:{{navprefix}}/orgs[Multi-tenancy with Orgs] diff --git a/modules/ROOT/pages/deploy-with-tml-apis.adoc b/modules/ROOT/pages/deploy-with-tml-apis.adoc index ffad32d04..e09eaf7d8 100644 --- a/modules/ROOT/pages/deploy-with-tml-apis.adoc +++ b/modules/ROOT/pages/deploy-with-tml-apis.adoc @@ -8,156 +8,258 @@ When deploying embedded analytics, each organization will have defined practices for development, testing, and deployment of content to ThoughtSpot. ThoughtSpot instances act as a constantly running service, so deployment only involves publishing *ThoughtSpot content*, in the form of link:https://docs.thoughtspot.com/cloud/latest/tml[ThoughtSpot Modeling Language (TML), window=_blank] files to a given ThoughtSpot instance. -ThoughtSpot has xref:version_control.adoc[Git integration] designed to automate most of the steps involved in the following process. Please see if the xref:version_control.adoc[Git integration capabilities] will work for your needs before building a process with the TML APIs directly. +The most typical pattern with ThoughtSpot involves a dev Org where content is modified by the team and then those objects are marked to form a "release" package, which then have their TML exported into a "release" branch in Git: -NOTE: Any example workflow you see within this document has been implemented and tested within the libraries available in the xref:development-and-deployment.adoc#relatedResources[Additional Resources]. We recommend that you start with these libraries and tools. +image::./images/multi_tenant_deployment.png[Multi-tenant Database Deployment SDLC Pattern] + +The TML files in the "release" branch are then moved into other "deployment branches" via pull/merge request. + +Other ThoughtSpot Orgs are "deployed to" by importing TML from their designated "deployment branch". + +The formalized release process is separate from xref:development-and-deployment.adoc#_version_control[version control], which is concerned with capturing all changes of all content in a given Org. == Overview The three steps to building an SDLC process with ThoughtSpot are: - . *xref:linkExportSource[Export source]*: Downloading ThoughtSpot objects as TML files into a source control system (for example, Git) - . *xref:linkBuildRelease[Build release]*: Altering copies of the TML files for the next stage / environment + . *xref:linkBuildRelease[Identify objects for release]*: Determine which objects in "dev Org" should become part of the "release" package + . *xref:linkExportSource[Export release TML into Git]*: Downloading ThoughtSpot objects as TML files into a source control system (for example, Git) . *xref:linkImportRelease[Import release]*: Importing the TML files into the new environment -Every object on a ThoughtSpot instance has a *GUID* as a unique reference. +== Pre-requisites +For TML files to be portable between any Org on any instance, the objects must have *obj_ids* and all Table and Connection objects must have been xref:metadata-parameterization.adoc[parameterized] with the appropriate xref:variables.adoc[variables] where necessary. -The *most essential* aspect of steps 2 and 3 is recording any newly created object GUIDs from the destination environment into a *xref:guidMapping[mapping file]* along with the GUID of the source object. +Once these pre-requisites are complete, your TML will be ready for TML Import across Orgs or publishing: -When publishing to a new environment on the same ThoughtSpot instance, you *must* swap out the GUIDs from the source environment with those of the equivalent objects in the destination environment within the TML files, so that only destination environment content is referenced. +[,YAML] +---- +obj_id: My_Connection__DATA_CHALLENGE__SALES +table: + name: SALES + db: ${dc_db} + schema: ${dc_schema} + db_table: SALES + connection: + name: My Connection + obj_id: My_Connection +---- -== Deployment scenarios +=== Parameterization with variables +In a single-tenanted database pattern, customer data is split into many separate logical databases or schemas, typically with identical table structures. -* xref:#_instance_to_instance_deployment[Instance-to-instance deployment] -* xref:#_multiple_environments_on_the_same_instance[Multiple environments on the same instance] +Queries for each customer can be identical by *xref:metadata-parameterization.adoc[parameterizing]* the attributes that vary. ThoughtSpot provides xref:variables.adoc[variables] that can be referenced in TML for this purpose. -=== Instance-to-instance deployment -The simplest deployment scenario is moving content from one ThoughtSpot instance to another separate instance. +Details about the fully-qualified table name are stored in each Table object in ThoughtSpot, rather than in the Connection object. The following can be parameterized in a Table TML file: -The TML Import process will use the `guid:` property of the imported TML files as the GUID for the new objects on the destination instance on all instances later than 9.0.0, which includes all ThoughtSpot Cloud deployments. +* database +* schema +* db_table_name -This means that a mapping file or swapping in and out GUIDs is not required. Make sure that all Connections have the same unique names on both instances and TML files should import without any modifications. +Properties of a Connection can also be parameterized. -If your instance is running a ThoughtSpot release version lower than 9.0.0.cl, refer to the xref:development-and-deployment.adoc#_notes_for_older_releases[Notes for older releases]. +Parameterization works with both TML import and xref:publishing-overview.adoc[publishing]. -=== Multiple environments on the same instance +=== obj_id +Before implementing any deployment process, you should ensure that `obj_id` is enabled on your ThoughtSpot instance. -Many ThoughtSpot customers have multiple "environments" on the same instance, either using xref:orgs.adoc[Orgs] or well-defined xref:multi-tenancy-best-practices.adoc[Access Control]. +[NOTE] +==== +Objects are only assigned automatic `obj_id` after the first change to the object once `obj_id` is enabled on an instance. +==== -In this scenario, you must track the equivalent GUIDS between source and destination environments, and swap them out within the TML files for your deployment process to work correctly. +`obj_id` for an object is available in the response from `/metadata/search` REST API as `metadata_obj_id` property: -The workflow for a very simple "dev" to "prod" flow on the same environment shown here, is the same pattern for any source-to-destination environment flow: +[,json] +---- +{ + "metadata_id":"c1e4043a-4524-4fcb-a20f-9e7aff4dc972", + "metadata_name":"Retail Sales RAD - KPIs", + "metadata_type":"LIVEBOARD", + "metadata_obj_id":"RetailSalesRAD-KPIs-c1e4043a" +} +---- +The format of `obj_id` above, with the `-{startOfGUID}` portion at the end, indicates it was an auto-generated obj_id. If the property value was `null`, then it is an object that has not been updated since the feature was enabled. -image::./images/development-deployment-process.png[Development and deployment workflow] +The `metadata/update-obj-id` REST API endpoint can change an object's `obj_id` property based on either the `metadata_id` (GUID) or the current `metadata_obj_id`: -[#guidMapping] -== GUID mapping file -As noted above, keeping a *mapping file* of GUIDs of source objects and their descendant objects in the destination environment is essential. The exact structure of the file will depend on the complexity of your deployment needs. +[,json] +---- +{ + "metadata": [ + { + "metadata_identifier": "c1e4043a-4524-4fcb-a20f-9e7aff4dc972", + "new_obj_id": "RetailSalesRAD-KPIs" + + } + ] +} +---- -The simplest pattern is to assume that releases are built exclusively from the "dev" environment, regardless of the destination environment. This pattern can be represented in the simple JSON structure: +or -[source,json] +[,json] ---- { - "test": { - "" : "" - }, - "uat": { - "" : "" - }, - "prod": { - "" : "" - } - ... + "metadata": [ + { + + "current_obj_id": "RetailSalesRAD-KPIs-c1e4043a", + "new_obj_id": "RetailSalesRAD-KPIs" + } + ] } ---- -You can use the same format for your mapping file as the xref:guid-mapping.adoc[Git deploy commits API] provided by ThoughtSpot. The most important aspect is recording the source GUID along with the GUID of the equivalent object in the destination after receiving the response from the `/metadata/tml/import` REST API call. +The request format for `metadata/update-obj-id` is an array, allowing for updating a large number of objects at once. -[#linkExportSource] -== Export source process -The process for exporting TML files into source control is: +==== Implementing obj_id in existing environments +Once you have decided on the appropriate `obj_id` for all the objects in your dev Org, you can use any existing GUID Mapping files to update the `obj_id` values for the equivalent objects in all your other Orgs. - . Use Metadata APIs (xref:metadata-api.adoc#metadata-list[/metadata/list] in v1 or link:{{navprefix}}/restV2-playground?apiResourceId=http/api-endpoints/metadata/search-metadata[/metadata/search] in v2.0) to get a filtered list of objects - . Use `/metadata/tml/export` endpoint in REST API v1 or v2.0 with `export_fqns=true` argument and `formmattype=YAML` to retrieve the TML of the object - . Save the TML response strings to disk in a Git-enabled directory using a consistent name format +You only have to do this process for existing objects *one time*, and from that point forward, you can update objects in any Org using TML with obj_ids rather than GUIDs. -You can use the link:https://thoughtspot.github.io/cs_tools/[CS Tools, window=_blank] package for a pre-built tool for programmatic exporting or build your own equivalent using the link:https://github.com/thoughtspot/thoughtspot_rest_api_v1_python[thoughtspot_rest_api_v1 Python library, window=_blank]. +[#linkBuildRelease] +== Identifying release package in dev Org +The set of objects on the *dev Org* that makes up the "deployed releases" should be identified in a way that allows easy identification of the objects to be exported into Git. -=== Best practices with TML export API -The `formattype` argument can be set to `YAML` or `JSON`. +The following mechanisms can be used to easily identify subsets of objects in an Org: -Export in YAML for saving to disk for source control or use with the `thoughtspot_tml` library. Export in JSON when you need details from TML within a web browser or just need to read values programmatically. +* Tags +* Content Author +* Collections (if enabled) -You can pass any number of GUIDs in the `export_ids` argument, although it is simpler to retrieve one at a time, particularly when processing the results obtained from the `export_associated=true` option. +The most typical practice is to add a *Tag*, something like *release* or *rc*, to any object that should be exported into the "release" branch. -The `export_associated` argument retrieves the TML objects for all related objects when used, including the GUID of each object within the headers. This is useful for dependency checking, and was valuable in versions lower than 8.9.0.cl to fill in `fqn` values. For more information, see xref:olderReleaseNotes[Notes for older releases]. +You can add multiple tags, if you'd like to add tags indicating versioning along with the release indicator. -[#linkBuildRelease] -== Build release process -To change the source environment TML files so that they can be imported into the destination environment, you need a process that correctly manipulates the TML files. +The `/metadata/search` REST API endpoint has parameters to filter on `tag_identifiers` and `created_by_user_identifiers`. The response from `/metadata/search` can be processed further to reduce down to the desired list of object IDs to pass to the `/metadata/tml/export` endpoint. -Common adjustments include: +[,json] +---- +{ + "metadata": [ + { + "type": "LIVEBOARD" + } + ], + "tag_identifiers": ["release"], + "sort_options": { + "field_name": order_field, + "order": "DESC" + }, + "record_size" : -1, + "record_offset": 0 +} +---- -* Switching connections at the Table level -* Changing database details within Table objects -* Adding or removing columns -* Renaming columns for translations +The `author` property of an object in the REST API is the value that the `created_by_user_identifiers` parameter in the `/metadata/search` endpoint filters on. `author` can be reassigned from the original creator via the link:https://developers.thoughtspot.com/docs/restV2-playground?apiResourceId=http%2Fapi-endpoints%2Fsecurity%2Fassign-change-author[/security/metadata/assign, window=_blank] endpoint. -For information about the specific TML changes to achieve these goals, see xref:modify-tml.adoc[Modify TML files]. There are also functioning code examples of many of these changes in the link:https://github.com/thoughtspot/thoughtspot_tml[thoughtspot_tml, window=_blank] repository. +[#linkExportSource] +== TML Export with obj_id and variables +The process for exporting TML files into source control is: -=== GUIDs in TML files determine create vs update operations + . Use Metadata Search REST API link:{{navprefix}}/restV2-playground?apiResourceId=http/api-endpoints/metadata/search-metadata[/metadata/search] to get a filtered list of objects for the release + . Use `/metadata/tml/export` endpoint with `formattype=YAML` to retrieve the TML of the object + . Save the TML response strings to disk in a Git-enabled directory using a consistent name format based on the `obj_id` -Objects of the same or different types can have the same display name in ThoughtSpot, so the GUID is necessary to identify the particular object. -In the REST APIs, `id` properties are the GUIDs. +=== Best practices with TML export API +The `/metadata/tml/export` endpoint has many options for controlling the format of the response. -In TML: +==== YAML or JSON +The `formattype` argument can be set to `YAML` or `JSON`. -* the `guid:` property will be at the top of the file -* `fqn:` properties are used to reference other connected objects (typically data sources) with a GUID +Export in YAML for saving to disk for use in Git or when using the `thoughtspot_tml` library, which is designed to handle the TML YAML format. -==== Rules for create vs. update operations -Object names are *never used* for determining an object to update, because object names are not unique within ThoughtSpot. +Export in JSON when you need details from TML within a web browser or just need to read values programmatically. -Whether an imported TML will create a new object or update an existing object depends on: +==== obj_id or guid +Once all your objects have defined `obj_id` properties, the ideal TML Export setting has the following options (you may have more or others as well): -* the presence/absence of the *guid:* property in the TML file -* whether that GUID matches an existing object on that ThoughtSpot instance -* the `force_create=true` parameter +[,json] +---- +{ + "edoc_format": "YAML", + "export_associated": false, + "export_fqn": false, + "export_options": { + "include_obj_id": true, + "include_obj_id_ref": true, + "include_guid": false + } +} +---- -Creation vs. update is determined by the following rules: +`"include_obj_id": true` specifies the `obj_id` at the top of the YAML file as the identifier for the object. - - *No GUID* in the TML file: always creates a new object with a new GUID - - *GUID in TML file*, where an object with the *same GUID already exists* in instance: update object - - *GUID in TML file*, where *no object with same GUID exists* in ThoughtSpot instance: creates a new object with the GUID from the TML file - - *Table objects* match on fully-qualified tables in the database (each Connection can only have one Table object per table in the database), not GUID: If a Table object representing the same database table is found, the GUID of the original object is maintained, but the updates are applied from the new TML file - - *force_create=true* parameter of the TML Import API is used: every uploaded TML file results in new objects being created +`"include_obj_id_ref": true` specifies that any related objects referenced in the TML file have an `obj_id` property as well. -[NOTE] -==== -In versions prior to 9.0.0.cl, ThoughtSpot did not consistently use the GUID provided in the TML file for a new object when that GUID was not already in use on that ThoughtSpot instance. -==== +The TML will end up looking like: + +[,YAML] +---- +obj_id: My_Connection__DATA_CHALLENGE__SALES +table: + name: SALES + db: ${dc_db} + schema: ${dc_schema} + db_table: SALES + connection: + name: My Connection + obj_id: My_Connection +---- + +If any of the objects do not have `obj_id` set, the TML will be exported with the `guid` property at the top, or the `fqn:` property for a reference, which will be the referenced object's GUID: + +[,YAML] +---- +guid: c1e4043a-4524-4fcb-a20f-9e7aff4dc972 +table: + name: SALES + db: ${dc_db} + schema: ${dc_schema} + db_table: SALES + connection: + name: My Connection + fqn: 75b717da-94b6-42ae-ab93-110c677703fb +---- + + +==== export_associated +The `export_associated` argument retrieves the TML objects for all related objects when used, including the GUID of each object within the headers. This is useful for dependency checking, but results in longer request times and bigger responses, and possibly exporting the same object over and over. -=== GUID mapping and swapping -Regardless of the other changes you make, building a release for an environment on the same instance will require swapping in the correct GUIDs. Because the presence of the *guid* property determines whether an individual TML file will cause a create or update action, you need to keep a *GUID mapping file* to determine how to adjust the TML files for upload to the new environment. +=== File and directory naming pattern +`obj_id` must be unique within an Org, which makes it the obvious value to use as part of the filename of the YAML file on disk. ThoughtSpot's default file name pattern also includes the object type in the filename, resulting in a suggested pattern: -The *guid mapping file* is referenced when creating the final TML files for publishing and then should be updated with any new object GUIDs after publishing: +`{obj_id}.{object_type}.tml` - . Check the *guid mapping file* - .. If no key-value pair exists for the *dev GUID* for the new environment: *remove the guid property from the TML file*. This will cause a *create* action - .. If a key-value pair exists: *swap* the TML file *guid* value from the *dev GUID* to the *destination environment GUID*. This will cause an *update* action - . When a new object is published for the first time, record the *dev GUID* as the key, and the *new object GUID* as the value - . Perform the same process for any *fqn* properties, which specify data object references. Remove the *fqn* property if the data object is being newly created, or swap it to the mapped GUID for that environment +You may also want to store various object types in their own directories on disk for ease of organization. -The link:https://github.com/thoughtspot/thoughtspot_tml[thoughtspot_tml library, window=_blank] provides a helper function called `disambiguate()` which implements the logic described above when provided with a Dict representing the GUID map. For information about how to use the library, see the README and examples or look at the source code if building an equivalent process yourself in another language. +The `obj_id` should be the first line of any TML file, and the object type should be the second, so you can easily build the filename from the `edoc` response from the TML Export REST API like: + +[,python] +---- +if 'edoc' in yaml_tml[0]: + lines = yaml_tml[0]['edoc'].splitlines() + + if lines[0].find('obj_id: ') != -1: + obj_id = lines[0].replace('obj_id: ', "") + else: + # Exception path of your choice + + obj_type = lines[1].replace(":", "") + + # Save the file with {obj_type}s/{obj_id}.{type}.{tml} + # Feel free to change directory naming structure to not have 's' at end + directory = "{}s".format(obj_type) + filename = "{}/{}.{}.tml".format(directory, obj_id, obj_type) +---- [#linkImportRelease] -== Import release process -The xref:tml-api#import[/metadata/tml/import] REST API endpoint is used to upload any number of TML files at one time. +== TML Import with obj_id and variables +The xref:tml-api.adoc#import[/metadata/tml/import] REST API endpoint is used to upload any number of TML files at one time. -All details of the objects to be created or modified are specified *within the uploaded TML file*, including the GUID which determines which existing object a given TML file will update. +All details of the objects to be created or modified are specified *within the uploaded TML file*, matching first on `obj_id` and then on `guid` if `obj_id` is not present. -The xref:development-and-deployment#linkBuildRelease[Build release process] section above describes the process for getting the TML files prepared for the import release process. The following describes the Import TML REST API call and what to do with the responses, which do feed back into the build release process in the form of the *GUID mapping file*. +If no match is found for `obj_id` or `guid`, a new object is created, with the `obj_id` specified in the TML file (`guid` will be auto-assigned). === TML import options and responses @@ -167,146 +269,52 @@ ThoughtSpot does not consider object display name for a TML file, but does use n All data objects are referenced as "tables" within TML, whether they are a ThoughtSpot Table, Model, View, SQL view, or any other data object type. The following heuristic is used to find matching objects by name within `tables` or `joins` sections: - + + . `obj_id` within other files in the same TML Import operation or existing objects in the Org . Data object names within the same TML Import operation: Must only be one single object with that name - . Searches the entire ThoughtSpot instance: Must be only one single object with that name + . Searches the entire ThoughtSpot Org: Must be only one single object with that name -The best practice is to create and upload "packages" of related objects together at once: +For this reason, we recommend importing Tables and Models from the same Connection in the same Import TML request, at least on the initial import. -* Give data objects within a package unique names, even though not enforced by ThoughtSpot -* All Table objects that use the same Connection object and all Models connected to those Tables should be uploaded together in a single TML Import -* If a data object already exists, swap out the *fqn* references to avoid the name matching heuristic +==== Rules for create vs. update operations +Object names are *never used* for determining an object to update, because object names are not unique within ThoughtSpot. -==== Storing new GUIDs in a mapping -To track relationships between objects in different environments, particularly on the same instance, you must store a *mapping* of the child object GUID to its source object GUID when you first publish the child object. +Whether an imported TML will create a new object or update an existing object depends on: -The xref:tml-api#import[import REST API endpoint] returns the GUID in the response after a successful import. The `object` key of the response to the import call contains an array, where each element has a `["response"]["header"]["id_guid"]` key providing the GUID. If you import multiple TML files at once, the response array will be in the same order as the request. This allows you to record a mapping of the originating GUID to the newly created GUIDs. +* the presence/absence of the *obj_id:* property in the TML file +* the presence/absence of the *guid:* property in the TML file +* whether that GUID matches an existing object on that ThoughtSpot instance +* the `force_create=true` parameter -[source,json] ----- -{ - "object": [ - { - "response": { - "status": { - "status_code": "OK" - }, - "header": { - "id_guid": "a09a3787-e546-42cb-888f-c17260dd1229", - "name": "Basic Answer 1", - "description": "This is basic answer with table and headline visualizations.", - "author_guid": "59481331-ee53-42be-a548-bd87be6ddd4a", - "owner_guid": "a09a3787-e546-42cb-888f-c17260dd1229", - "metadata_type": "QUESTION_ANSWER_BOOK" - } - } - } - ] -} ----- +Creation vs. update is determined by the following rules: -Update the *mapping file* with the new pair of source object GUID and destination environment object GUID, so that the release build process can do the appropriate swaps the next time the object needs to be updated. +*obj_id* is always considered first if present. ThoughtSpot looks at org_id + obj_id to find any existing object in that Org with a match, then creates a new object if one is not found +If only GUIDs are in the TML file, without obj_id, the rules are: -[#relatedResources] -== Additional Resources + - *No GUID* in the TML file: always creates a new object with a new GUID + - *GUID in TML file*, where an object with the *same GUID already exists* in instance: update object + - *GUID in TML file*, where *no object with same GUID exists* in ThoughtSpot instance: creates a new object with the GUID from the TML file + - *force_create=true* parameter of the TML Import API is used: every uploaded TML file results in new objects being created -* The link:https://github.com/thoughtspot/thoughtspot_tml[thoughtspot-tml module, window=_blank] is written in Python providing classes to work with the TML files as Python objects. You can install it via pip: +Table objects match on the real existence of the table in the particular database: -+ ----- -pip install thoughtspot_tml ----- + - *Table objects* match on fully-qualified tables in the database (each Connection can only have one Table object per table in the database), not GUID or obj_id: If a Table object representing the same database table is found, the obj_id/GUID of the original object is maintained, but the updates are applied from the new TML file -* The link:https://github.com/thoughtspot/thoughtspot_rest_api_v1_python[thoughtspot-rest-api-v1 module, window=_blank] is a Python module implementing the full ThoughtSpot V1 REST API. You can install it via pip: -+ ----- -pip install thoughtspot_rest_api_v1 ----- +[#relatedResources] +== Additional Resources -* The link:https://github.com/thoughtspot/ts_rest_api_and_tml_tools[ts_rest_api_and_tml_tools project, window=_blank] provides examples of workflows using the REST API and TML modification possible with the `thoughtspot_tml` and `thoughtspot_rest_api_v1` modules. This library is intended to provide working examples and is not maintained or supported by ThoughtSpot. +* The link:https://github.com/thoughtspot/thoughtspot_tml[thoughtspot_tml library, window=_blank] is written in Python providing classes to work with the TML files as Python objects. You can install it via pip: -* The link:https://github.com/thoughtspot/ts_rest_api_and_tml_tools/blob/main/examples/tml_and_sdlc/[examples/tml_and_sdlc/, window=_blank] directory includes many different example scripts for these TML-based workflows. + -//// -Within the examples directory, the link:https://github.com/thoughtspot/ts_rest_api_and_tml_tools/blob/main/examples/tml_and_sdlc/tml_download.py[tml_download.py, window=_blank] script is a simple example of exporting all TML objects to disk for use with Git or another source control system. -//// -* For command-line administration tools including many pre-built TML-based workflows, the link:https://github.com/thoughtspot/cs_tools[cs_tools project, window=_blank] is available. - -//// -== Notes for older releases (8.9.0.cl or earlier versions) - -[#olderReleaseNotes] -=== Add FQNs of associated objects in TML -Prior to ThoughtSpot 8.9.0.cl, TML files did not include the GUIDs of associated objects on export. However, you can use the `export_associated=true` argument to retrieve the GUIDs of the associated objects, then programmatically add the `fqn` property to the downloaded TML with the correct GUIDs. Including the GUIDs in the saved files on disk allows you to substitute the GUIDs for the equivalent objects in another environment. - -For example, in these earlier versions, the items in the `tables:` list of this example worksheet TML only include a `name:` property, representing the name of the ThoughtSpot *table* object (as opposed to the table's name in the data warehouse). - -If there are *table* objects with duplicate names, specify the GUID of the object using the `fqn:` property. This will distinguish the correct object when importing the TML back. - -When you set `export_associated=true` in the TML export command, the first item in the response will be the object you requested in the export: - -[source,yaml] ---- -guid: 0a0bb654-b0e8-482c-a6c8-9ed396d1cb92 -worksheet: - name: Markspot 2 Worksheet - tables: - - name: DIM_CUSTOMERS_2 - table_paths: - - id: DIM_CUSTOMERS_2_1 - table: DIM_CUSTOMERS_2 - join_path: - - {} -... ----- - -The overall response will be structured as a JSON array, with an `edoc` property representing the TML document itself and an `info` section providing basic metadata information, but more importantly the `name` and `id` properties. - -[source,json] ----- -{ - "object": [ - { - "edoc": "" - , - "info": { - "id": "", - "name": "", - ... - } - }, - ... - ] -} ----- - -Parse through this array and record a simple mapping of name to `guid`: - -.Python example of this process -[source,python] ----- -name_guid_map = {} - -for obj in objs: - name_guid_map[obj['info']['name']] = obj['info']['id'] +pip install thoughtspot_tml ---- -Because we know that these are the GUIDs that match the name values in this particular TML file, we can now use the map we created to add in the `fqn` properties, to result in the *worksheet* TML looking like this: +* The link:https://github.com/thoughtspot/thoughtspot_rest_api_python[thoughtspot_rest_api library, window=_blank] is a Python library implementing the full ThoughtSpot REST API. You can install it via pip: -[source,yaml] ++ ---- -guid: 0a0bb654-b0e8-482c-a6c8-9ed396d1cb92 -worksheet: - name: Markspot 2 Worksheet - tables: - - name: DIM_CUSTOMERS_2 - fqn: 3b87cea1-7767-4fd8-904f-23255d4ba7b3 - table_paths: - - id: DIM_CUSTOMERS_2_1 - table: DIM_CUSTOMERS_2 - join_path: - - {} +pip install thoughtspot_rest_api ---- -//// diff --git a/modules/ROOT/pages/deprecated-features.adoc b/modules/ROOT/pages/deprecated-features.adoc index 68581fbfb..9d3a9a1db 100644 --- a/modules/ROOT/pages/deprecated-features.adoc +++ b/modules/ROOT/pages/deprecated-features.adoc @@ -308,7 +308,7 @@ Use the new version of REST API v2.0 endpoints and SDK versions available for th ==== Documentation Starting from 9.0.0.cl, the API documentation for the REST API v2 [beta betaBackground]^Beta^ endpoints will not be accessible from the REST API Playground in ThoughtSpot. -For information about the REST API v2 [beta betaBackground]^Beta^ endpoints, see xref:rest-api-v2-reference-beta.adoc[REST API v2 ^Beta^ reference]. +//For information about the REST API v2 [beta betaBackground]^Beta^ endpoints, see xref:rest-api-v2-reference-beta.adoc[REST API v2 ^Beta^ reference]. Recommended action:: For information about REST API v2.0 endpoints, refer to the following articles and visit the link:{{navprefix}}/restV2-playground?apiResourceId=http%2Fgetting-started%2Fintroduction[REST API v2 Playground]. diff --git a/modules/ROOT/pages/development-and-deployment.adoc b/modules/ROOT/pages/development-and-deployment.adoc index 98932ab93..396ed312f 100644 --- a/modules/ROOT/pages/development-and-deployment.adoc +++ b/modules/ROOT/pages/development-and-deployment.adoc @@ -12,31 +12,99 @@ ThoughtSpot instances act as a constantly running service, so xref:development-a ThoughtSpot provides numerous tools for building a structured deployment process, built around the link:https://docs.thoughtspot.com/cloud/latest/tml[ThoughtSpot Modeling Language (TML), window=_blank] format for representing the xref:intro-thoughtspot-objects.adoc[objects within ThoughtSpot]. -== Best practices -The primary tool for structured development and deployment in ThoughtSpot is called xref:orgs.adoc[Orgs]. +== Overview +ThoughtSpot may provision your organization one or more separate *instances*, each with an individual URL. -Each Org in ThoughtSpot can be xref:version_control.adoc[paired] to a link:https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-branches[branch, target=_blank] in a link:https://docs.github.com/en/repositories/creating-and-managing-repositories/about-repositories[Git repository, target=_blank] as a single *environment*. +Within a ThoughtSpot instance, the administrator of the Primary Org can create any number of logical tenants called xref:orgs.adoc[Orgs]. xref:orgs.adoc[Orgs] are fully separated tenants on a single ThoughtSpot instance. For TSE customers and others who need a structured development and deployment process, Orgs should always be enabled. -Once your environments are configured, you can xref:version_control.adoc#moving-tml-changes-between-environments[move data models and content] from the initial "dev environment" to any other environment using the xref:git-rest-api-guide.adoc[Git integration REST APIs]. +image::./images/instances_and_orgs.png[Instances and Orgs] + +By setting the `xref:intro-thoughtspot-objects.adoc#object-identifiers[obj_id]` property of objects, objects in various Orgs that are related copies of one another will have the same `obj_id`, allowing for tracking related objects and updating them without concern for each object's unique GUID. [NOTE] ==== -ThoughtSpot does not recommend TML export and import across different versions of ThoughtSpot application because the TML syntax, supported features, and object schemas can vary between releases and can sometimes lead to compatibility issues and validation errors. +ThoughtSpot does not recommend TML export and import from a *newer version* of ThoughtSpot to an instance on a *previous version*, because the TML syntax, supported features, and object schemas can vary between releases and can sometimes lead to compatibility issues and validation errors. ==== +== Version Control +*Version control* is the process of tracking changes that occur to objects in ThoughtSpot. + +A single branch in Git can be used for version control of a single Org in ThoughtSpot. + +ThoughtSpot provides a GitHub-based link:https://docs.thoughtspot.com/cloud/10.15.0.cl/git-version-control[automated version control in the UI] for Liveboards and answers or a customized process can be built using the TML Export API to any Git provider. + +It is best to use separate branches or even repositories for the UI automated version control and direct REST API processes. + +When using the TML REST APIs and a Git provider, you can also implement version control branches, but they should be separate from the "deploy branches": + +image::./images/version-control-branches.png[version control branches diagram] + +It is important to have a version control branch for any "prod" Orgs with end user created content, which otherwise will not be archived in any way. + +== Deployment +*Deployment* is the process of *making copies of objects* from one Org to another Org. + +Deployment is used in the process of building a *release* from a *dev Org* and then deploying copies of the release objects via TML to *test*, *uat*, and eventual *prod* Orgs. + +The TML Export and Import APIs allow customizable release and deployment processes to integrate with any Git provider. + +The standard deployment pattern for a xref:multi-tenancy-intro.adoc[multi-tenanted prod database] is shown below. RLS rules will filter the shared data models on the "prod" Org so that standard LBs and Answers only show the right data for each end customer, who are all only added as users to the "prod" Org. + +image::./images/multi_tenant_deployment.png[Multi-tenant Database Deployment SDLC Pattern] + +In a scenario where xref:single-tenant-data-models.adoc[end customer databases are single-tenanted], an Org can be created programmatically matching with the level of tenant separation, so that there is an Org representing each separate logical database. + +There are two techniques for managing Orgs for single-tenanted databases: + + * Publishing [beta betaBackground]^Beta^ + * Deploy to each Org using TML Import + +== Publishing + +*xref:publishing-overview.adoc[Publishing]* makes objects available in other Orgs without making copies. + +Variables can be set at the Org level to override the Connection and Table object details for Publishing objects when they are accessed in a specific Org. Variables for connections and tables work with both Publishing and TML Import. + +image::./images/publishing_diagram.png[Publishing and Variables] + +With single-tenanted prod Orgs, the *Publishing* feature allows the final deployment step to import TML into the *publisher* Org, updating the *Published* objects, which instantly updates the objects in every Org the objects are published to. + +image::./images/single_tenant_publishing.png[Publishing to Single-Tenants] + +If there are structural differences within the various databases that make Publishing unviable, the TML Import process can be used to deploy unique copies of the release TML into each Org. This process may also include modifying the release TML to introduce variation into the objects that are deployed. + +image::./images/single_tenant_deployment.png[Individual Deployment to Single-Tenants] + + + + +//// +Each Org in ThoughtSpot can be xref:version_control.adoc[paired] to a link:https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-branches[branch, target=_blank] in a link:https://docs.github.com/en/repositories/creating-and-managing-repositories/about-repositories[Git repository, target=_blank] as a single *environment*. +//// + +//// +Once your environments are configured, you can xref:version_control.adoc#moving-tml-changes-between-environments[move data models and content] from the initial "dev environment" to any other environment using the xref:git-rest-api-guide.adoc[Git integration REST APIs]. +//// + + + //// If you have used ThoughtSpot for a long time and are enabling Orgs for the first time, please see xref:moving-to-orgs.adoc[moving to Orgs from single-tenant ThoughtSpot instance]. //// -== Standard structure of Orgs +//// +== Typical Orgs pattern for SDLC When Orgs are enabled, the *Primary Org* (`org_id: 0`) should be treated as the "root" or "super-admin" Org, and all "real content" should exist on Orgs other than the Primary. +Additionally, the *Primary Org* is the only Org where *Published* objects can originate, so *Primary* serves as the *Publisher* Org within SDLC flows that involve Publishing. + The most basic standard configuration for separated development and deployment, with a xref:multi-tenancy-best-practices.adoc[multi-tenanted "prod" environment], has the following environments: - *Primary*: only used for administration of instance and Orgs themselves -- *Dev*: ThoughtSpot customer internal development team builds data models and standard Liveboards and Answers -- *Prod*: Data models and standard content are published to this environment by service account. All end customers get READ-ONLY access to data models, standard Liveboards, and Answers. +- *dev*: ThoughtSpot customer internal development team builds data models and standard Liveboards and Answers +- *test*: +- *prod*: Data models and standard content are published to this environment by service account. All end customers get READ-ONLY access to data models, standard Liveboards, and Answers. image::./images/multi-tenanted_prod_deployment.png[Multi-tenanted prod deployment model] @@ -47,20 +115,25 @@ RLS filters the shared data models on the "prod" Org so that standard LBs and An If the production end customer data models are xref:single-tenant-data-models.adoc[single-tenanted (logical separation for each end customer)], such that a different connection is necessary), then the standard deployment involves a "release"/"pre-prod" Org and then one prod Org per end customer. image::./images/single-tenant_prod_per_customer.png[Single-tenant final deployment model] - +//// +//// === Additional "typical" Orgs - *Test*, *UAT*: Additional steps in the publishing process between Dev and Prod, for verification before changes are deployed into Prod - *Internal Org(s)*: Org(s) for ThoughtSpot customer's own business work, never to be shared with end customers - Separate *'dev' instance*: Some people want a fully separated environment for dev, especially if doing MAJOR changes, even to cluster config. Recommend there be a 'Test Org' on the 'Prod Instance' to push to first, before pushing to 'Prod' - Separate *regional instances*: prod per customer Orgs may extend to multiple ThoughtSpot instances in different regional cloud data centers +//// +//// [#deploy-between-environments] == Deploying content between environments -//// + The workflow for a very simple "dev" to "prod" flow on the same environment shown here, is the same pattern for any source-to-destination environment flow: image::./images/development-deployment-process.png[Development and deployment workflow] +//// + //// === Git deployment The xref:version_control.adoc[Git REST APIs] are designed for the best practice pattern of building on a Dev Org and then deploying to any number of Orgs through Prod(s). @@ -74,9 +147,12 @@ You will need a xref:guid-mapping.adoc[GUID Mapping file] that records the `orig == Multi-tenancy and data security The exact deployment pattern chosen will depend on the design of your data warehouse. Please see the xref:multi-tenancy-intro.adoc[full documentation on multi-tenancy within ThoughtSpot] to determine which deployment pattern best fits your needs. +//// +//// == Publishing content to Orgs within a ThoughtSpot instance Starting with the 10.10.0.cl release, ThoughtSpot provides the ability to parameterize object properties using variables for each Org and publish objects directly from the Primary Org to other Orgs on a multi-tenant instance. The publishing feature [beta betaBackground]^Beta^ enables administrators to create a single main object in the Primary Org and distribute it to other Orgs within the instance. -For more information, see xref:publishing-overview.adoc[Publishing content to Orgs]. \ No newline at end of file +For more information, see xref:publishing-overview.adoc[Publishing content to Orgs]. +//// diff --git a/modules/ROOT/pages/embed-authentication.adoc b/modules/ROOT/pages/embed-authentication.adoc index ec2fdd8dc..990b76ab3 100644 --- a/modules/ROOT/pages/embed-authentication.adoc +++ b/modules/ROOT/pages/embed-authentication.adoc @@ -71,6 +71,11 @@ a| Do not use this method if you don’t want the SDK to redirect your entire ap * This authentication will fail if multifactor authentication (MFA) is enabled on your ThoughtSpot instance. Contact https://community.thoughtspot.com/customers/s/login/?ec=302&startURL=%2Fcustomers%2Fs%2Fcontactsupport[ThoughtSpot Support] for assistance. |===== +[NOTE] +==== +xref:orgs.adoc#per-org-subdomain[Per Org Subdomain] can be enabled to allow Orgs with different IdPs to be identified properly within the authentication flows triggered by the Visual Embed SDK. +==== + == User accounts Many ThoughtSpot features are tied to individual user accounts with a valid email address. xref:just-in-time-provisioning.adoc[Just-In-Time Provisioning] and user management REST APIs make it easy to create and update user accounts as part of the SSO process. @@ -218,7 +223,7 @@ The `init` method returns an event emitter, which you can use to listen to `Auth [source,TypeScript] ---- -authStatus = init(embedConfig); authStatus.on(AuthStatus.FAILURE, (reason) => { +const authStatus = init(embedConfig); authStatus.on(AuthStatus.FAILURE, (reason) => { console.log('Authentication failed'); }); ---- @@ -267,7 +272,7 @@ import { logout } from "@thoughtspot/visual-embed-sdk"; logout(); ---- -The `logout` function returns a promise that resolves when the user is logged out of ThoughtSpot. When you call `logout`, the `autoLogin` attribute is set to `false` to prevent the SDK from automatically logging in the user again. If you do not want to disable `autoLogin`, set the `doNotDisableAutoLogin` parameter to `false`. +The `logout` function returns a promise that resolves when the user is logged out of ThoughtSpot. When you call `logout`, the `autoLogin` attribute is set to `false` to prevent the SDK from automatically logging in the user again. If you do not want to disable `autoLogin`, set the `doNotDisableAutoLogin` parameter to `true`. You can also call `init` again with the `autoLogin` property set to `true` to re-login a user. diff --git a/modules/ROOT/pages/git-configuration.adoc b/modules/ROOT/pages/git-configuration.adoc index 0840dd56e..dbbf1bf12 100644 --- a/modules/ROOT/pages/git-configuration.adoc +++ b/modules/ROOT/pages/git-configuration.adoc @@ -1,12 +1,17 @@ -= Configure Git integration += Configure GitHub integration :toc: true :toclevels: 2 -:page-title: Configure Git integration +:page-title: Configure GitHub integration :page-pageid: git-configuration -:page-description: Configuring the Git integration for a ThoughtSpot instance +:page-description: Configuring the GitHub integration for a ThoughtSpot instance -Git integration requires configuration within both ThoughtSpot, accomplished via the V2.0 REST APIs, and your Git provider (GitHub currently). +[NOTE] +==== +The legacy GitHub REST APIs referenced here are tied to GitHub exclusively. The newer xref:git-provider-integration.adoc[Git provider integration pattern] provides more flexibility and control and should be utilized if you are starting out or having issues with the GitHub APIs. +==== + +GitHub integration requires configuration within both ThoughtSpot, accomplished via the V2.0 REST APIs, and your Git provider (GitHub currently). == Configure Git repository @@ -44,7 +49,7 @@ tscli git-integration enable //// == Confirm permissions within ThoughtSpot Orgs -* To commit objects from Thoughtspot to a Git repository, your ThoughtSpot user account requires at least view permission for all objects that will be committed as part of the operation. +* To commit objects from ThoughtSpot to a Git repository, your ThoughtSpot user account requires at least view permission for all objects that will be committed as part of the operation. * To deploy or revert objects from a Git repository to ThoughtSpot, you require edit access to all objects that will be updated as part of the deployment. If the deployment contains Models, Views, or Tables, users require **Can manage data** (`DATAMANAGEMENT`) privilege for deploy, commit, and revert operations. [#guid-map-and-config-files] @@ -80,17 +85,16 @@ For more advanced Git development patterns, list multiple `branch_names` in the |=== |Parameter|Description |`repository_url`|__String__. The HTTPS URL of the Git repository; for example, `\https://github.com/user/repo.git`. -|`username` -|__String__. Username to authenticate to the Git repository. +|`username`|__String__. Username to authenticate to the Git repository. |`access_token`|__String__. Access token to authenticate to the Git repository. |`org_identifier`|__String__. ID of the Org. Define this parameter only if the Orgs feature is enabled on your ThoughtSpot cluster and separate Orgs are configured for development and production environments. |`branch_names`|__Array of strings__. List of Git branches to configure. -|`commit_branch_name` |__String__. Name of the remote branch where objects committed from this Thoughtspot instance will be versioned. Replaces `default_branch_name`, which is deprecated in 9.10.5.cl. +|`commit_branch_name` |__String__. Name of the remote branch where objects committed from this ThoughtSpot instance will be versioned. Replaces `default_branch_name`, which is deprecated in 9.10.5.cl. |`default_branch_name` + __Optional__|__String__. Deprecated in 9.10.5.cl. In earlier versions, this parameter was used to configure the name of the default Git branch to use for all operations on the cluster. |`enable_guid_mapping` |__Boolean__. Enables GUID mapping and generates a GUID mapping file. Starting from 9.7.0.cl, this attribute is set to `true` by default. To know more about GUID mapping, see xref:version_control.adoc#_guid_mapping_and_configuration_files[GUID mapping]. -|`configuration_branch_name` a|__String__. Name of the branch where the configuration files related to operations between Thoughtspot and the version control repository should be maintained. Replaces `guid_mapping_branch_name`, which is deprecated in 9.10.5.cl. +|`configuration_branch_name` a|__String__. Name of the branch where the configuration files related to operations between ThoughtSpot and the version control repository should be maintained. Replaces `guid_mapping_branch_name`, which is deprecated in 9.10.5.cl. [NOTE] ==== @@ -109,14 +113,14 @@ The following example shows the API request format for connecting ThoughtSpot to ---- curl -X POST \ --url 'https://{ThoughtSpot-Host-Dev}/api/rest/2.0/vcs/git/config/create' \ - -H 'Authorization: Bearer {Bearer_token} \ + -H 'Authorization: Bearer {Bearer_token}' \ -H 'Accept: application/json'\ -H 'Content-Type: application/json' \ --data-raw '{ "repository_url": "https://github.com/user/repo.git", "username": "ts-git-user", "access_token": "{ACCESS_TOKEN}", - "org_identifier": "dev" + "org_identifier": "dev", "branch_names": [ "dev", "main" @@ -128,7 +132,7 @@ curl -X POST \ If the API request is successful, the ThoughtSpot instance will be connected to the Git repository. -Go into each Org an issue the `config/create` call to link the appropriate branch to establish all environments. +Go into each Org and issue the `config/create` call to link the appropriate branch to establish all environments. The following example shows the API request parameters to connect a ThoughtSpot `Prod` Org to the Git repo. Note that GUID mapping is enabled in the API request. @@ -138,7 +142,7 @@ The `Bearer_token` value must be requested for the desired Org, specified throug ---- curl -X POST \ --url 'https://{ThoughtSpot-Host-Prod}/api/rest/2.0/vcs/git/config/create' \ - -H 'Authorization: Bearer {Bearer_token} \ + -H 'Authorization: Bearer {Bearer_token}' \ -H 'Accept: application/json'\ -H 'Content-Type: application/json' \ --data-raw '{ @@ -146,7 +150,7 @@ curl -X POST \ "username": "ts-git-user", "access_token": "{ACCESS_TOKEN}", "enable_guid_mapping": true, - "org_identifier": "prod" + "org_identifier": "prod", "branch_names": [ "prod" ], @@ -162,9 +166,9 @@ Version control and xref:guid-mapping.adoc[GUID mapping] utilize files stored in You can initialize these files by immediately using the xref:git-rest-api-guide.adoc#deploy-commits[deploy commits REST API] after configuring the Org for Git. -See the link:https://github.com/thoughtspot/thoughtspot_rest_api_v1_python/blob/main/examples_v2/create_orgs_with_linked_git_branch.py[create_orgs_with_linked_git_branch.py script] for an example of deploying a full Orgs or branches setup for structured development and deployment. +See the link:https://github.com/thoughtspot/thoughtspot_rest_api_v1_python/blob/main/examples_v2/create_orgs_with_linked_git_branch.py[create_orgs_with_linked_git_branch.py script, window=_blank] for an example of deploying a full Orgs or branches setup for structured development and deployment. -After the Orgs are set up, you can link:https://github.com/thoughtspot/thoughtspot_rest_api_v1_python/blob/main/examples_v2/create_connection_on_orgs.py[create connections across the orgs] and xref:guid-mapping.adoc#using-mapping-for-table-tml-properties[add any necessary entries to the GUID mapping files]. +After the Orgs are set up, you can link:https://github.com/thoughtspot/thoughtspot_rest_api_v1_python/blob/main/examples_v2/create_connection_on_orgs.py[create connections across Orgs, window=_blank] and xref:guid-mapping.adoc#using-mapping-for-table-tml-properties[add any necessary entries to the GUID mapping files]. [#update-git-config] diff --git a/modules/ROOT/pages/git-provider-integration.adoc b/modules/ROOT/pages/git-provider-integration.adoc new file mode 100644 index 000000000..9892e37d1 --- /dev/null +++ b/modules/ROOT/pages/git-provider-integration.adoc @@ -0,0 +1,197 @@ += Git provider integration +:toc: true +:toclevels: 2 + +:page-title: Git provider integration using TML APIs +:page-pageid: git-provider-integration +:page-description: Overview of using a Git provider to manage SDLC in ThoughtSpot + +Git providers such as GitHub, GitLab, Azure DevOps and BitBucket can run scripts that use the ThoughtSpot REST API and TML modification capabilities. + +This allows the Git provider to become the centralized control service for deployment processes to ThoughtSpot. + +The combination of Git commands and ThoughtSpot REST API calls allows for controlled synchronization of the Git repo and the various ThoughtSpot Orgs that represent environments within the software deployment lifecycle. + +[NOTE] +==== +The legacy GitHub REST APIs are used through ThoughtSpot and are tied to GitHub exclusively. The Git provider integration pattern provides more flexibility and control. +==== + +== Overview +The Git provider integration pattern is based on storing the necessary secrets in the Git provider to allow ThoughtSpot REST API commands to be run by the Git provider when accomplishing tasks on demand or on a schedule. + +All imports and exports of ThoughtSpot TML files to and from branches, along with the commits and merges are controlled entirely in git using the Git provider's capabilities. + +The two fundamental workflows are *xref:deploy-with-tml-apis.adoc#tml-export-with-obj_id-and-variables[TML Export] to branch from Org* and *xref:deploy-with-tml-apis.adoc#tml-import-with-obj_id-and-variables[TML Import] from branch to Org*: + +=== TML Export to branch from Org + + . `git checkout {branch-name}`: switch to existing remote branch or create local branch + . `git pull`: bring down latest from remote of branch + . Script to find desired objects then Export TML, saving TML files to disk with regular save command using name pattern `{obj_id}.{type}.tml` + . `git diff-index --quiet HEAD || git commit -a -m "{commit msg}"` + . `git push`: Send changes to remote + +[NOTE] +==== +If there are no changes to any files, `git commit` will throw an error. `git diff-index --quiet HEAD || git commit -a -m "{commit msg}"` avoids this issue. +==== + +The script that actually downloads the TML should also have capabilities for xref:deploy-with-tml-apis.adoc#linkBuildRelease[filtering items to just the release package]. The same download script can handle pure version control (downloading all items) as well as downloading only the items of the release package. + +==== Efficient exporting using modified timestamp +TML Export causes activity on a ThoughtSpot instance, as TML is generated dynamically at the time of request from the current state of the objects in ThoughtSpot. + +To make the process more quick and efficient, you can implement a method for only exporting objects that have changed since the last TML Export was run, using the `/metadata/search` REST API response, which contains a `modified` timestamp in the `metadata_header` section. + + +=== TML Import from branch to Org + + . `git checkout {branch-name/tag}`: switch to existing remote branch or tag + . `git pull`: bring down latest from remote of branch + . Script to find all desired TML files in the directory, then use TML Import REST API to import into the target Org + . Script to share newly created objects to Groups in the destination Org + + +== Setup +Each Git provider has its own mechanisms for doing the following, but the overall pattern described should be possible in any of them. + +Essentially, the Git provider will store the necessary details and secrets to make REST API calls to the ThoughtSpot instance(s). + +=== Secrets and variables +The following should be stored within the Git provider for use within any automated workflows: + +* TS_SERVER +* TS_SECRET_KEY +* TS_INSTANCE_ADMIN_USERNAME +* TS_DOWNLOAD_USERNAME +* TS_IMPORT_USERNAME + +Some of these, such a `TS_DOWNLOAD_USERNAME` and `TS_IMPORT_USERNAME` may vary at a branch / environment level. Even `TS_SERVER` and `TS_SECRET_KEY` may have variations per branch if you are using multiple ThoughtSpot instances. + +=== Branches +Due to the nature of ThoughtSpot TML, you'll want to separate deployment branches for the release package from any "pure version control" branches. + +image::./images/version-control-branches.png[version control branches diagram] + +Pull requests / merges should be possible smoothly from "release" -> "test_deploy" -> "uat_deploy" -> "prod_deploy", without needing to resolve any merge conflicts. + +The following is an example branch scheme, you are welcome to adjust the names to match your preferences: + +* main / master: Actions / workflows / other shared assets, but no TML files from any Org +* dev_vc: Version Control for all content on Dev Org +* release: Branch for Specific Content to go through deployment to other Orgs +* test_deploy: Import TML from 'release' and do other Actions to Test Org +* test_vc: Version control for all content on Test Org +* Optional UAT / etc.: +** uat_deploy: Import TML from 'release' and do other Actions to UAT Org +** uat_vc: Version control for all content on UAT Org +* prod_deploy Import TML from 'release' and do other Actions to Prod Org(s) + +Version control for prod Orgs: + +* prod_vc: if single Prod, version control of all Content +* customer_orgs(s): version control branch for each Single Tenant Org + +== Local branches, copies and edits +ThoughtSpot itself is the "living source of truth", with TML only a representation of the current state of the object in the system at the time of TML Export or the desired state to bring the system to via TML Import. + +A TML Import will cause the existing object in ThoughtSpot with the same identifier (`obj_id` or `guid` if no `obj_id` is present) to update to match the state described in the imported TML file. + +You can edit a TML file in a *local branch*, but using TML Import to see the result in the originating Org will cause the object to be updated. + +There are two options: + +* Create a new Org to represent the local branch, to test the new TML +* Create a copy object in the original org, with a different identifier + +The final step would be to change the identifier in the copy object's TML to match the original identifier, then TML Import the file back into the original Org. + +image::./images/copy_edit_change_obj_id.png[Copy/edit change obj_id pattern] + +Actions like this can be performed in git however your team feels most comfortable with. The resulting changes in the "dev Org" then become part of the release package files that are exported and moved through the deployment stages like normal. + +== Other sync actions + +=== Deleting objects +When an object is deleted in the dev Org, you also need to `git rm` the file within the git release branch, so that the file will be deleted in downstream branches as you do pull/merge requsts. + +The following is the git and REST API portion of an example GitHub workflow that cleans up files that do not have a matching object in the Org any longer. + +It looks at whether the filenames (matching obj_id) in the dev branch of the git directory continue to exist via `/metadata/search` REST API command. If not found, issue `git rm` to remove the TML file, then `git commit` and `git push` to update the origin/branch. + +The `files_with_no_obj_on_org.py` script is what performs the `/metadata/search` REST API to get the list of existing objects, then looks at all the files in the git branch to see which filenames do not have corresponding obj_ids: + +[,yaml] +---- +run: | + git config --local user.email "${{ github.actor_id }}+${{ github.triggering_actor }}@users.noreply.github.com" + git config --local user.name "${{ github.triggering_actor }}" + python .github/workflows/files_with_no_obj_on_org.py > files_to_remove.txt + if [ -s "files_to_remove.txt" ]; then + echo "Files found in Org: ${{ vars.TS_ORG_NAME }} for removal:" + cat files_to_remove.txt + if [ "${{ github.event.inputs.DELETE_FLAG }}" = "DELETE" ]; then + echo "Deleting the files" + cat files_to_remove.txt | xargs git rm + rm files_to_remove.txt + git add * + git diff-index --quiet HEAD || git commit -a -m "Run ID ${{ github.run_id }}/${{github.run_attempt}} cleanup files from Org ${{ vars.TS_ORG_NAME }}" + git push + fi + else + echo "No files found to remove" + rm files_to_remove.txt + fi +---- + +[NOTE] +==== +You cannot do the reverse and simply delete objects from an Org that do not have equivalent TML files in the git directory, because end users can create new objects in an Org. +==== + +You also must consider that attempts while `git rm` command on files in release branch will flow upstream through via merge commands, removing data objects in the linked Orgs may fail if any user-created content in those Orgs are connected to the data objects. + +Thus the full deletion / cleanup process may involve manual "clean-up" in Orgs where ThoughtSpot does not allow object deletion without first deleting or changing the data source of dependent objects. + +=== obj_id updates +The `obj_id` property is simple enough to change within an Org, however the files in the repo are typically named after the `obj_id` of the object at the time of export. There will also be objects in the other Orgs with the same `obj_id`. + +An `obj_id` update should thus be *initiated from the Git provider* to rename the existing file and perform the ThoughtSpot REST API commands to update the `obj_id` across all Orgs with a copy of the object. + +The following is the git and REST API portion of example of a workflow in GitHub Action syntax to handle the rename of an object in the dev Org and release branch: + +[,yaml] +---- + - name: Change obj_id and rename file + env: + TS_SERVER: ${{ secrets.TS_SERVER }} + TS_SECRET_KEY: ${{ secrets.TS_SECRET_KEY }} + TS_USERNAME: ${{ secrets.TS_INSTANCE_ADMIN_USERNAME }} + OLD_OBJ_ID: ${{ github.event.inputs.OLD_OBJ_ID }} + NEW_OBJ_ID : ${{ github.event.inputs.NEW_OBJ_ID }} + run: | + git config --local user.email "${{ github.actor_id }}+${{ github.triggering_actor }}@users.noreply.github.com" + git config --local user.name "${{ github.triggering_actor }}" + echo "Import to Org_ID $ORG_ID rename $OLD_OBJ_ID to: $NEW_OBJ_ID" + if find . -name "${{ github.event.inputs.OLD_OBJ_ID }}*" -print -quit | grep -q .; then + echo "file for ${{ github.event.inputs.OLD_OBJ_ID }} found" + echo "Updating obj_id in org from ${{ github.event.inputs.OLD_OBJ_ID }} to ${{ github.event.inputs.NEW_OBJ_ID }}" + python .github/workflows/update_obj_id.py + echo "obj_id updated, now renaming file in repo" + O_FILE=$(find . -name "${{ github.event.inputs.OLD_OBJ_ID }}*" -print) + N_FILE="${O_FILE//${{ github.event.inputs.OLD_OBJ_ID }}/${{ github.event.inputs.NEW_OBJ_ID }}}" + echo "$O_FILE rename to $N_FILE" + git mv $O_FILE $N_FILE + git diff-index --quiet HEAD || git commit -a -m "Run ID ${{ github.run_id }}/${{github.run_attempt}} changed obj_id" + git push + echo "File rename successful" + else + echo "no file found matching ${{ github.event.inputs.OLD_OBJ_ID }}, no action taken" + fi +---- + +== GitHub Actions example +A repo with link:https://github.com/bryanthowell-ts/git-test-2/tree/main/.github/workflows[example GitHub workflow files to power Actions is available, window=_blank]. + +The workflows are composed of a YAML workflow file and a Python script using the link:https://github.com/thoughtspot/thoughtspot_rest_api_python[thoughtspot_rest_api Python library, window=_blank]. Both components are useful starting points for implementing in a different Git provider. diff --git a/modules/ROOT/pages/git-rest-api-guide.adoc b/modules/ROOT/pages/git-rest-api-guide.adoc index dbf1d6e7f..499b6656b 100644 --- a/modules/ROOT/pages/git-rest-api-guide.adoc +++ b/modules/ROOT/pages/git-rest-api-guide.adoc @@ -1,10 +1,15 @@ -= Version Control REST APIs += GitHub REST APIs :toc: true :toclevels: 2 -:page-title: Git REST API guide +:page-title: GitHub REST API guide :page-pageid: git-api -:page-description: Guide to Git Integration REST APIs +:page-description: Guide to GitHub Integration REST APIs + +[NOTE] +==== +The legacy GitHub REST APIs referenced here are tied to GitHub exclusively. The newer xref:git-provider-integration.adoc[Git provider integration pattern] provides more flexibility and control and should be utilized if you are starting out or having issues with the GitHub APIs. +==== The `/vcs/git/` endpoints of the v2.0 REST API provide all the functionality necessary to xref:git-configuration.adoc[configure], perform version control, and deploy changes to different ThoughtSpot environments, other than xref:guid-mapping.adoc#connection_variations[configuring connections]. @@ -16,7 +21,7 @@ The xref:version_control.adoc#moving-tml-changes-between-environments[basic proc The following sections describe how to use the version control REST APIs after completing xref:git-configuration.adoc[Git configuration] for a ThoughtSpot instance. -There is a complete implementation of all Git REST APIs available in the link:https://thoughtspot.github.io/cs_tools/[CS Tools package, target=_blank] as well as link:https://github.com/thoughtspot/thoughtspot_rest_api_v1_python/blob/main/examples_v2/[example workflow scripts available in Python, target=_blank]. +There is a complete implementation of all Git REST APIs available in the link:https://thoughtspot.github.io/cs_tools/[CS Tools package, target=_blank] as well as link:https://github.com/thoughtspot/thoughtspot_rest_api_v1_python/blob/main/examples_v2/[example workflow scripts available in Python, window=_blank]. == Commit files The `/api/rest/2.0/vcs/git/branches/commit` API call exports the TML files for the requested ThoughtSpot objects directly to a Git branch and commits the changes. @@ -55,7 +60,7 @@ The following example shows the API request with Liveboard and Model objects to ---- curl -X POST \ --url 'https://{ThoughtSpot-Host}/api/rest/2.0/vcs/git/branches/commit' \ - -H 'Authorization: Bearer {Bearer_token}\ + -H 'Authorization: Bearer {Bearer_token}' \ -H 'Accept: application/json'\ -H 'Content-Type: application/json' \ --data-raw '{ @@ -189,13 +194,13 @@ Due to GUID mapping requirements in most destination environments, it is current === Request example -The following example shows the API request with Liveboard and Model objects to commit to Git. +The following example shows the API request to validate a merge from the dev branch to main. [source,cURL] ---- curl -X POST \ --url 'https://{ThoughtSpot-Host}/api/rest/2.0/vcs/git/branches/validate' \ - -H 'Authorization: Bearer {Bearer_token}\ + -H 'Authorization: Bearer {Bearer_token}' \ -H 'Accept: application/json'\ -H 'Content-Type: application/json' \ --data-raw '{ @@ -246,7 +251,7 @@ The following example shows the API request for reverting a commit. ---- curl -X POST \ --url 'https://{ThoughtSpot-Host}/api/rest/2.0/vcs/git/commits/afc0fea831558e30d7064ab019f49243b1f09552/revert' \ - -H 'Authorization: Bearer {Bearer_token}\\ + -H 'Authorization: Bearer {Bearer_token}' \ -H 'Accept: application/json'\ -H 'Content-Type: application/json' \ --data-raw '{ diff --git a/modules/ROOT/pages/just-in-time-provisioning.adoc b/modules/ROOT/pages/just-in-time-provisioning.adoc index 7a4d86e07..a5e80275f 100644 --- a/modules/ROOT/pages/just-in-time-provisioning.adoc +++ b/modules/ROOT/pages/just-in-time-provisioning.adoc @@ -1,53 +1,180 @@ = Just-in-time provisioning :toc: true -:toclevels: 1 +:toclevels: 2 :page-title: Just-in-time provisioning of users and groups :page-pageid: just-in-time-provisioning :page-description: Just-in-time provisioning using SSO -ThoughtSpot can use the SSO process to create users and assign them to ThoughtSpot groups. +Just-in-time (JIT) provisioning in ThoughtSpot refers to creating or updating authentication and authorization at the time a user signs in to ThoughtSpot. -The capabilities are the same between SAML, OIDC, and trusted authentication methods, but the implementations of each are slightly different. +Due to the variety of options in ThoughtSpot, there are often several ways to accomplish JIT provisioning. -== User creation -Just-in-time provisioning (JIT) allows creating a user if they do not already exist at the time of SSO sign-in. +The steps to JIT provisioning are conceptually: -Creating a new user requires the assertion containing the username, email address, display name, groups, and org details if your instance has Orgs. + . Does the user exist in ThoughtSpot? + .. If the user does not exist, create the user. + . Is the user in this Org? + .. If not in the Org, add the user to the Org. + . Update the user's current provisioning within the Org (ThoughtSpot Group membership). + . Update any variable values for the user via token request (token authentication only). -Users created via JIT provisioning are identical to users created manually or via the REST APIs, except they do not have passwords in Thoughtspot; they cannot access ThoughtSpot other than through the SSO method. You can assign a password to any user later through the UI or a REST API call. +Due to the nature of API capabilities and error responses, along with restrictions around Org separation, the actual order of efficient steps may not follow the order listed above. -== Group assignment -Users can be assigned to ThoughtSpot groups via JIT as well. The list of groups should be composed of `group_name` properties, rather than `display_name`. -* JIT group assignment is a full replacement of the user's set of groups. -* Groups that do not exist will be created during the provisioning process, but a ThoughtSpot group by default provides no access control or privileges. However, you can assign privileges to the new groups via REST API update requests. -* Groups created via JIT provisioning will have the same group name as the one used by the `ts_groups` property in any RLS rules. +== JIT REST API provisioning (trusted authentication) +All of the following may be considered for JIT provisioning, using the same REST API capabilities as provisioning ahead of time: -== JIT provisioning and authentication token generation via REST APIs -Both REST API V1 and V2 tokens support just-in-time provisioning of users. + * User creation or adding to Org + * Group assignment for users + ** Group creation and updates + * RLS variable values for user -=== REST API v2 (Recommended) -Both `/api/rest/2.0/auth/token/full' and `/api/rest/2.0/auth/token/object` API endpoints allow getting an authentication token for the user being provisioned just-in-time. +These can all be implemented as part of the *xref:trusted-auth-token-request-service.adoc[token request service]* in the trusted authentication pattern. + +The *token request service* will need access to both the `secret_key` for the token requests and a *service account* with the appropriate administrator privileges to run the other REST API commands. + +== User creation or adding to Org +For a login token to be requested, the user must exist in ThoughtSpot in the Org the token is being requested for. + +Creating a new user requires at minimum the username, email address, display name, and org IDs to create them in. + +=== Create user and update user REST APIs +If you need to update a user's details, including their group membership, the simplest method is to use: + +. `link:https://developers.thoughtspot.com/docs/restV2-playground?apiResourceId=http%2Fapi-endpoints%2Fusers%2Fupdate-user[/users/{user-identifier}/update]` in the target Org with the `operation: REPLACE` option. + +.. If you receive an error, retry the `/users/{user-identifier}/update` request with an Administrator service account in the Primary Org to add the user to the desired Org, with `operation: ADD`. +.. If you still receive an error, the user does not exist in ThoughtSpot. Use `/users/create` from the Primary Org to create the user and place in the desired Org(s). +.. Retry the original `/users/{user-identifier}/update` command in the desired Org. +. Request the preferred auth token for the user, without using any `autocreate=` options. + +You could instead check for the user's state with an Org via `/users/search`. and avoiding unnecessary updates and error handling steps. + +Any additional changes to the Groups themselves can be managed with other REST APIs described in the following section. + +=== autocreate using token request +There are several token request endpoints, each with an `autocreate=true` option for just-in-time provisioning: + +* `/api/rest/2.0/auth/token/full` +* `/api/rest/2.0/auth/token/custom` +* `/api/rest/2.0/auth/token/object` + +[NOTE] +==== +`autocreate=true` creates the user (and any new groups) with default options, which may not be preferred. Using the direct `users` and `groups` endpoints allows full control of their options. +==== + +The following details must be included in a request with `autocreate=true` to allow provisioning the user: * The `auto_create: true` parameter enables the token for the JIT provisioning of the user. * The `display_name` and `email` parameters are also required for JIT user creation. * If Orgs are enabled, specify the `org_id` parameter to direct ThoughtSpot to assign the user to the specified Org. * Specify the `group_identifiers` parameter only if you want to enable JIT group assignment. Passing `group_identifiers: []` will set the user to be assigned to *no groups*, while excluding the `group_identifiers` parameter altogether will leave the user assigned to their existing set of groups. -For more information, see xref:authentication.adoc#_just_in_time_provisioning[Just-in-time provisioning via REST v2 API]. +Users created via `autocreate=true` are identical to users created manually or via the REST APIs, except they do not have passwords in ThoughtSpot; they cannot access ThoughtSpot other than through the SSO method. You can assign a password to any user later through the UI or a REST API call. + +== Group assignment +Groups in ThoughtSpot are used to assign a variety of attributes such as content sharing, group-based RLS, roles, and column-level security. JIT provisioning often requires a combination of updates to both the user and the groups they belong to. + +=== Group assignment via Update User REST API +The `/users/{user-identifier}/update` API can update a user's Group membership within the Org, determined by the org_id of the auth token used for the REST API request: + +[,json] +---- +{ + "operation": "ADD", + "group_identifiers": [ + "New Group A" + ] +} +---- + +You can also choose `"operation" : "REPLACE"` to reset the entire set of Groups for a user. + +[NOTE] +==== +You cannot set a user's groups in an Org other than the REST API request's auth token. If you create a user from the Primary Org, any `group_identifiers` provided at that time will apply only within the Primary Org, even if you add the user to multiple Orgs in the `/users/create` request. +==== + +=== Group assignment via Update Group REST API +Because group membership is a relationship between a user and a group, you can also update the set of users within a group using the `link:https://developers.thoughtspot.com/docs/restV2-playground?apiResourceId=http%2Fapi-endpoints%2Fgroups%2Fupdate-user-group[/groups/{group_identifier}/update]` REST API endpoint. + +Similarly to the user update endpoint, you can choose between three operations: `ADD`, `REPLACE`, and `REMOVE` to update only the `user_identifiers` of a group without affecting other properties: + +[,json] +---- +{ + "operation": "ADD", + "user_identifiers": [ + "new.user@website.com" + ] +} +---- + +=== Group assignment via token request +The various token requests can set or update a user's group membership when `autocreate=true` is set. However, they have slightly different behaviors: + +* `/auth/token/full` and `/auth/token/object` do a *full replace* of the list of groups on every token request +* `/auth/token/custom` only assigns groups *if the user is created* + +The list of groups should be composed of `group_name` properties, rather than `display_name`. +If a group name is provided that does not match any existing group name, a new group will be created during the provisioning process. + +Groups created via `autocreate=true` will have identical `group_name` and `display_name` properties but will otherwise be a default ThoughtSpot group, granting no access control, privileges or roles. + +However, you can assign privileges or make any other adjustment to the new groups via REST API `link:https://developers.thoughtspot.com/docs/restV2-playground?apiResourceId=http%2Fapi-endpoints%2Fgroups%2Fupdate-user-group[/groups/{group_identifier}/update]` endpoint. + + +== Group privileges and access control via REST APIs +xref:api-user-management.adoc[Privileges] are either assigned directly to a group or part of roles if xref:roles.adoc[Role-based access control (RBAC)] is enabled on your instance. + +Roles must be created first prior to being assigned as part of the `/groups/create` or `/groups/{group_identifier}/update` REST API request. + +Access control in ThoughtSpot is based on *link:https://developers.thoughtspot.com/docs/restV2-playground?apiResourceId=http%2Fapi-endpoints%2Fsecurity%2Fshare-metadata[sharing content]* to groups or directly to users. + +Sharing requires that the group or user exist prior to calling the sharing REST API. + +== RLS variable values +RLS rules can use a user's *username*, *group membership* or *variable values* to define a filtering clause added to all queries on a table. + +If using `ts_groups` in a RLS Rule, the group names must match exactly with the values in a column in the data warehouse, so the name of the group itself serves as a __data entitlement__. + +Variable values are used as part of the xref:abac_rls-variables.adoc[ABAC via RLS variables] pattern, which allows assigning many values to multiple variables via the xref:abac_rls-variables.adoc#_create_an_abac_token_request_with_variable_attributes[Custom Token request]. + +There is also a direct xref:variables.adoc#_update_variable_values[variable values update REST API] for more complex or bulk updates. + + + + + +//// +=== JIT provisioning and authentication token generation via REST APIs +Both REST API V1 and V2 tokens support just-in-time provisioning of users. + +=== REST API v2 (Recommended) +//// + +//// === REST API v1 The `/tspublic/v1/session/auth/token` API endpoint can provision a new user by setting the `autocreate` property to `true`. For more information, see xref:session-api.adoc#session-authToken[Session API]. - +//// +//// == Org IDs If the Orgs feature is enabled on your instance, you do need to specify the Org ID when creating a user. Org IDs are integers that are created automatically when a cluster administrator creates an Org. Administrators can get the Org IDs configured on a ThoughtSpot instance via `/tspublic/v1/org/search` or `/api/rest/2.0/orgs/search` API endpoint. To know the Org ID of your current Org, send a `GET` request to the `/tspublic/v1/session/orgs` API endpoint within the browser. For more information about Org APIs, see xref:org-manage-api.adoc[Org administration]. +//// + +== IdP assertion provisioning +Due to the nature of assertions returned from an IdP, the JIT provisioning capabilities are more limited. The REST APIs available within the trusted authentication workflow described above can still be used for provisioning and updates, but must be made prior to IdP assertion. + +In general, the IdP assertion can create a user and add them to existing ThoughtSpot groups within existing ThoughtSpot Orgs. == SAML SSO authentication For SAML SSO users, you can link:https://docs.thoughtspot.com/cloud/latest/authentication-integration#_enable_saml_authentication[enable SAML authentication, window=_blank] and *Automatically add SAML users to ThoughtSpot upon first authentication*. @@ -61,11 +188,4 @@ OIDC SSO can be configured for JIT user creation, as the necessary properties sh JIT group assignment xref:configure-oidc.adoc#group-synchronization[can be enabled for OIDC via a support ticket]. -== Group privileges and access control via REST APIs - -JIT provisioning of groups does not provide the full configuration of groups or assign access control. You can use the REST APIs to create, edit, and synchronize groups with the embedding application. For more information about group privileges, see xref:api-user-management.adoc[user and group privileges]. - -REST API calls other than *token requests* must be performed using a ThoughtSpot user account with the appropriate level of administrative privileges. For example, you could use xref:rest-api-v2-reference.adoc#_groups[REST API v2.0] or the xref:rest-api-reference.adoc#_groups_and_privileges[REST API v1] group endpoints to implement ThoughtSpot groups that are intended for use in Row Level Security (RLS) rules. -For these groups, the group name must match exactly with a value in a column in the data warehouse, so the name of the group itself serves as a __data entitlement__. For more information, see xref:abac-user-parameters.adoc[Attribute-based access control]. -Similarly, if xref:roles.adoc[Role-based access control (RBAC)] is enabled on your instance, you must map role privileges to groups and then assign them to users. diff --git a/modules/ROOT/pages/modify-tml.adoc b/modules/ROOT/pages/modify-tml.adoc index 659a33093..28977bfe4 100644 --- a/modules/ROOT/pages/modify-tml.adoc +++ b/modules/ROOT/pages/modify-tml.adoc @@ -8,9 +8,9 @@ link:https://docs.thoughtspot.com/cloud/latest/tml[ThoughtSpot Modeling Language (TML), window=_blank] is slightly different for every object type, but all follow a general pattern that allows for programmatic editing. This article describes how various objects and their connections to each other are represented in TML. -The xref:version_control.adoc[Git integration features] within ThoughtSpot automatically handle the relevant aspects of a typical development and deployment process. +When doing development and deployment with TML, xref:variables.adoc[variables] and xref:deploy-with-tml-apis.adoc#obj_id[obj_id] make TML portable across different Orgs and ThoughtSpot instances. -Modifying TML programmatically is useful for more complex levels of transformations than the standard development and deployment practices handled by the Git integration capabilities, such as the following: +Modifying TML programmatically is useful for more complex levels of transformations, such as the following: - Re-mapping of table objects between different data warehouse types - Deploying dynamically generated data models from the data warehouse @@ -31,21 +31,6 @@ One best practice recommendation is to only connect *Answers* and *Liveboards* t image::./images/object_model_hierarchy.png[ThoughtSpot object model hierarchy and relationships] -=== Export with fqn: property -Always export TML files with `export_fqns=true`. -//(for versions lower than 9.0.0.cl, see xref:development-and-deployment.adoc#_notes_for_older_releases_8_9_0_cl_or_earlier_versions[Notes for older releases]). - -The `fqn:` property provides the GUID of associated objects, which can be swapped out when moving packages of TML files between environments. The following is an example of the `connection:` property within a Table TML: - -[source,YAML] ----- -connection: - name: Demo Connection - fqn: 2aa36dbd-dda6-4497-a6db-bc47e128862e ----- - -You can always *remove* an `fqn:` property when uploading a set of TML with the intention of creating all new objects. - === Name properties are identifiers in connected objects When editing entire packages of exported TML files to be deployed in a different environment, you *must consider changes to identifiers in other "downstream" TML files*. ThoughtSpot will do the necessary updates on an existing data model. @@ -218,7 +203,7 @@ The `model_tables` section is a list of table objects that exist on the ThoughtS [source,YAML] ---- - model_tables:: + model_tables: - name: id : fqn : @@ -314,9 +299,9 @@ model: join_progressive: [ true | false ] lesson_plans: - lesson_id: - lesson_plan_string + lesson_plan_string: - lesson_id: - lesson_plan_string + lesson_plan_string: parameters: - id: name: @@ -331,7 +316,7 @@ model: - id: name: data_type: - default_value: + default_value: list_config: list_choice: - value: @@ -357,7 +342,7 @@ Worksheets combine several *tables*, including *Views*, into a coherent data mod If you want to change the values for an existing object, the `tables`, `joins` and `table_paths` sections are the most important. -So you go from +For example, you go from [source,YAML] ---- tables: diff --git a/modules/ROOT/pages/orgs.adoc b/modules/ROOT/pages/orgs.adoc index ad659fbad..79fe7ce9b 100644 --- a/modules/ROOT/pages/orgs.adoc +++ b/modules/ROOT/pages/orgs.adoc @@ -4,7 +4,7 @@ :page-title: Multi-tenancy and orgs :page-pageid: orgs -:page-description: You can now configure your ThoughtSpot instance as a mult-tenant cluster with separate Org containers for your tenants. +:page-description: You can now configure your ThoughtSpot instance as a multi-tenant cluster with separate Org containers for your tenants. If your deployment requires logical separation of departments within your organization, or if it involves supporting many distinct organizations from a single application instance, use the *Orgs* feature. @@ -108,7 +108,7 @@ If a cluster administrator wants to perform a Create, Read, Update, or Delete (C If the per Org URL feature is enabled on your ThoughtSpot instance, you can access shared objects such as Liveboards and Answers from multiple Orgs in different browser tabs. For example, you can access a Liveboard from Org1 in one tab and open a shared object from Org2 in another browser tab. -To enable this feature on your instance, contact link:https://community.thoughtspot.com/customers/s/contactsupport[ThoughSpot Support, window=_blank]. When this feature is enabled, the Org ID will be shown in the ThoughtSpot object URL as a query parameter, as shown in the following example: +To enable this feature on your instance, contact link:https://community.thoughtspot.com/customers/s/contactsupport[ThoughtSpot Support, window=_blank]. When this feature is enabled, the Org ID will be shown in the ThoughtSpot object URL as a query parameter, as shown in the following example: `\https://{ThoughtSpot-Host}/orgId=0/pinboard/22946f4b-b4ce-4643-be50-66afcd5177` @@ -134,7 +134,7 @@ A user belonging to multiple Orgs can share ThoughtSpot objects such as Liveboar Starting with ThoughtSpot Cloud 10.5.0.cl release, developers embedding ThoughtSpot in their application will be able to edit their custom link settings for their application users to allow seamless access to content from another different Org. For example, a user has access to Org1, Org2 and Org3. While the user is logged in to Org1, they can access a Liveboard shared by another user in Org3 without using the Org switcher. -This feature is turned off by default. To enable this feature on your instance, contact link:https://community.thoughtspot.com/customers/s/contactsupport[ThoughSpot Support, window=_blank]. When this feature is enabled, the Org ID will be passed as an additional query parameter in the `{ts-query-param}` in the URL. +This feature is turned off by default. To enable this feature on your instance, contact link:https://community.thoughtspot.com/customers/s/contactsupport[ThoughtSpot Support, window=_blank]. When this feature is enabled, the Org ID will be passed as an additional query parameter in the `{ts-query-param}` in the URL. For example, if you have set the custom link as: `\https://www.mysite.com/liveboard/{object-id}?{ts-query-params}` @@ -154,6 +154,28 @@ The Org ID will be passed in the URL depending on the placement of `{ts-query-pa * The `overrideOrgId` parameter may not work properly with trusted authentication (`AuthType.TrustedAuthToken`) or cookieless authentication (`AuthType.TrustedAuthTokenCookieless`), if `tokenAuthPerOrg` is already enabled on your ThoughtSpot instance. ==== +==== Per Org Subdomain + +[earlyAccess eaBackground]#Early Access# + +"Per Org Subdomain" can be requested to be enabled via a support ticket. Once this feature is turned on, every Org that exists at the time will have a subdomain generated on the pattern: + + ..thoughtspot.cloud + +[NOTE] +==== +New Orgs generated after the initial request currently require a new request to have the subdomain generated. +==== + +Per Org Subdomain is used for identifying a specific Org to a login process for users who belong to multiple Orgs on the same instance, bypassing the Org selection UI. In particular, OIDC flows for MCP Server or instances with multiple IdPs per Org can benefit from using Per Org Subdomain. + +Auto-redirect to SSO IdP is a separate cluster level config that must be requested via ticket to ThoughtSpot support. + +[NOTE] +==== +Prior to requesting Per Org Subdomain, all org names need to be DNS-friendly; otherwise, ThoughtSpot will return errors. You should review your Org names prior to the request and make sure they don't have spaces or other strange characters. The subdomains are case-sensitive and will match the Org names' case at the time of the request. +==== + == Feature availability on a multi-tenant instance On an Orgs-enabled cluster, certain UI and API operations are allowed only at the cluster level. The following table lists the features and configuration operations allowed at the cluster or individual Org level. @@ -234,10 +256,10 @@ a|[tag greenBackground tick]#✓# == Authentication considerations for embedded apps //// -The Visual Embed SDK supports leveraging your IdP or OpenID provider setup to authenticate the embedded app users. To determine the authentication method that best suits your deployment, refer to the recommendations listed on the xref:embed-authentication.adoc[Authentication]. +The Visual Embed SDK supports leveraging your IdP or OpenID provider setup to authenticate the embedded app users. To determine the authentication method that best suits your deployment, refer to the recommendations listed on the xref:embed-authentication.adoc[Embed authentication]. //// -On a multi-tenant cluster with Orgs, ThoughtSpot supports local, SAML, and trusted authentication methods. If you are using Visual Embed SDK to embed ThoughtSpot in your app, use `AuthType.Basic` for local authentication, `AuthType.TrustedAuthToken` for trusted authentication, and `AuthType.EmbeddedSSO` or `AuthType.SAMLRedirect` for SAML SSO authentication. For more information, see xref:embed-authentication.adoc[Authentication]. +On a multi-tenant cluster with Orgs, ThoughtSpot supports local, SAML, and trusted authentication methods. If you are using Visual Embed SDK to embed ThoughtSpot in your app, use `AuthType.Basic` for local authentication, `AuthType.TrustedAuthToken` for trusted authentication, and `AuthType.EmbeddedSSO` or `AuthType.SAMLRedirect` for SAML SSO authentication. For more information, see xref:embed-authentication.adoc[Embed authentication]. === Trusted authentication diff --git a/modules/ROOT/pages/trusted-auth-token-request-service.adoc b/modules/ROOT/pages/trusted-auth-token-request-service.adoc index 2207eb2ff..cb21b86e9 100644 --- a/modules/ROOT/pages/trusted-auth-token-request-service.adoc +++ b/modules/ROOT/pages/trusted-auth-token-request-service.adoc @@ -2,20 +2,36 @@ :toc: true :toclevels: 2 -:page-title: trusted authentication +:page-title: Token request service :page-pageid: trusted-auth-token-request-service :page-description: You can configure support for token-based authentication service on ThoughtSpot. == Overview The *token request service* is the portion of the web application with access to the ThoughtSpot `xref:trusted-auth-secret-key.adoc[secret_key]` that issues the request to ThoughtSpot to generate tokens. -In most cases, the *token request service* is a REST API endpoint added to the web application. Providing a REST API service allows the Visual Embed SDK to request new tokens whenever they are needed to start a new ThoughtSpot session. +image::./images/trusted-auth-new-1.png[Trusted Authentication Workflow] + +In most cases, the *token request service* is a REST API endpoint added to the web application, which then calls the ThoughtSpot REST API. + +Providing a REST API service within the app allows the Visual Embed SDK to request new tokens whenever they are needed to start or continue a new ThoughtSpot session. The `init()` function calls the *token request service* endpoint, which returns the *token* to the web browser, triggering the Visual Embed SDK to handle the rest of the login flow. The token request can instead happen on the back-end of the web app and the token can be embedded into the returned web page directly, but this will require a page reload if there is a need for a new token. There are no requirements for how the *token request service* is built or hosted, other than being able to issue REST API commands to the ThoughtSpot instance and having access to the `secret_key`. When using a ThoughtSpot cloud instance, the authenticator service will need outbound request access to the hosted ThoughtSpot cloud instance. -Trusted authentication tokens can be requested in a way that creates users and assigns them at the time of login. Please see the full documentation of xref:just-in-time-provisioning.adoc[just-in-time provisioning] for implementation details. +== Orgs +Tokens are scoped to one given Org at a time. + +The *token request service* will need to be aware of the `org_id` of the matching ThoughtSpot org for a given user at token request time. If there are single-tenanted Orgs rather than single "prod" Org, you'll need to build a mechanism for the token request service to know the appropriate `org_id` for a given user. + +The `/orgs/search` endpoint can be run by an administrator level account in the Primary Org (Org 0) to retrieve a full list of Orgs including their names and IDs. + +== Just-in-time (JIT) provisioning +Trusted authentication tokens can be requested in a way that creates users and assigns them to groups at the time of login, but this is only one aspect of just-in-time (JIT) provisioning. + +Please see the full documentation of xref:just-in-time-provisioning.adoc[just-in-time provisioning] for the full set of token request and other REST APIs used to achieve your desired JIT provisioning workflow. + +JIT provisioning will use a combination of the `secret_key` and one or more *service accounts* with administrator privileges to issue all of the necessary REST API commands. == Quick code example Every web application framework has its own way of handling users, sessions, and making REST API requests to other services. @@ -98,7 +114,7 @@ If you are using ThoughtSpot Cloud, the best practice is to use xref:authenticat Access control in ThoughtSpot (called Sharing) prohibits a signed-in user from loading any content to which they don't have access. Access control (sharing) can be granted during the login token request process by adding the user to the appropriate ThoughtSpot groups. -Tokens obtained via REST API v2 endpoints tokens can be used for cookie-based or cookieless trusted authentication. REST API v1 login tokens only work for cookie-based trusted authentication. +Tokens obtained via REST API v2 endpoints can be used for cookie-based or cookieless trusted authentication. REST API v1 login tokens only work for cookie-based trusted authentication. The `/session/token/login` REST API v1 endpoint used by the Visual Embed SDK can accept the token obtained via REST API v1 or v2 endpoint to establish a ThoughtSpot session. @@ -144,9 +160,6 @@ V1 tokens stay valid for a length of time based on the following rules: To set a consistent five-minute expiration time, you can generate a second token to start the expiration clock for the previous login token that is sent to the user's browser. -== Org-enabled clusters -On multi-tenant clusters with xref:orgs.adoc[Orgs] enabled, tokens are scoped to one given Org at a time. The *token request service* will also need to be aware of the `org_id` of the matching ThoughtSpot org for a given user at token request time. - == Revoking a token There is a xref:authentication.adoc#revoke-a-token[REST API for revoking a V2 Token], which could be incorporated into an additional endpoint of the *token request service* if you have concerns about the longer-lived tokens existing beyond the web application's session lifespan. @@ -194,7 +207,7 @@ public class TSFullTokenResponse { public string token {get; set;} public int creation_time_in_millis {get; set;} - public int expiration_time_in_milis {get; set;} + public int expiration_time_in_millis {get; set;} public TSTokenScope scope {get; set;} public string valid_for_user_id {get; set;} public string valid_for_username {get; set;} @@ -207,7 +220,7 @@ var tsHost = 'https://{}.thoughtspot.cloud'; var tsTokenEndpoint = '/api/rest/2.0/auth/token/full'; var fullEndpoint = tsHost + tsTokenEndpoint; -// secret_key stored in the Azure Key Valut +// secret_key stored in the Azure Key Vault var azureServiceTokenProvider = new AzureServiceTokenProvider(); KeyVaultClient kvc = new KeyVaultClient(azureServiceTokenProvider.KeyVaultTokenCallback); SecretBundle tsSecretKey = kvc.GetSecretAsync(baseUrl, "tsSecretKey").Result; diff --git a/modules/ROOT/pages/trusted-authentication.adoc b/modules/ROOT/pages/trusted-authentication.adoc index 763f21d13..a3c6fbf29 100644 --- a/modules/ROOT/pages/trusted-authentication.adoc +++ b/modules/ROOT/pages/trusted-authentication.adoc @@ -2,7 +2,7 @@ :toc: true :toclevels: 1 -:page-title: trusted authentication +:page-title: Trusted authentication :page-pageid: trusted-auth :page-description: You can configure support for token-based authentication service on ThoughtSpot. @@ -12,13 +12,13 @@ It is the most seamless method of single sign-on (SSO) available to embed Though Trusted authentication can also be used for back-end REST API processes that need to *impersonate* an individual user to retrieve a filtered data response. In those scenarios, you implement the token retrieval and sign-in calls directly without the browser portion. -image::./images/trusted-auth-workflow.png[Trusted Authentication Workflow] +image::./images/trusted-auth-new-1.png[Trusted Authentication Workflow] == Overview of implementation The trusted authentication implementation method includes the following steps: . xref:trusted-auth-secret-key.adoc[Enable Trusted authentication on ThoughtSpot] in the **Develop** > **Customizations** > **Security Settings** page. Copy the `secret_key` and place where the *token request service* can xref:trusted-auth-secret-key.adoc[securely access it]. -. Create the *xref:trusted-auth-token-request-service.adoc[token request service]*, typically a REST API endpoint in the embedding application. This service returns a login token for the user signed in by the web application. +. Create the *xref:trusted-auth-token-request-service.adoc[token request service]*, typically a REST API endpoint in the embedding application. This service returns a login token for the user signed in by the web application. The token request service may also xref:just-in-time-provisioning.adoc[run other ThoughtSpot REST APIs for just-in-time provisioning]. . Include the xref:trusted-auth-sdk.adoc[Visual Embed SDK] into the embedding web application. The authentication type is defined in the `init` function. You can configure cookie-based (`AuthType.TrustedAuthToken`) or cookieless authentication (`AuthType.TrustedAuthTokenCookieless`) as per your deployment needs. + . When `init()` is called, the SDK checks if there is an existing ThoughtSpot session for the instance in the browser. If not, it will request a *login token* from either the `authEndpoint` URL specified in the SDK or the `getAuthToken` callback function. For `authEndPoint`, specify the authentication endpoint URL from which you want to obtain the authentication token. If using `getAuthToken`, call the `getAuthToken` function to invoke your login endpoint. The login endpoint then returns a `Promise` string that resolves to an authentication token. @@ -55,7 +55,7 @@ With access to the `xref:trusted-auth-secret-key.adoc[secret_key]`, back-end RES * a login token using `session/login` to create a long-lived session as that user * a xref:rest-api-v2-getstarted.adoc#_log_in_to_thoughtspot[bearer token] for all subsequent REST API calls -This is useful for xref:rest-api-v2-reference.adoc#_data[data] and xref:rest-api-v2-reference#_reports[reports] endpoints that have row-level security or other filtering applied based on the user account itself. +This is useful for xref:rest-api-v2-reference.adoc#_data[data] and xref:rest-api-v2-reference.adoc#_reports[reports] endpoints that have row-level security or other filtering applied based on the user account itself. == Additional resources * link:https://codesandbox.io/s/big-tse-react-demo-i4g9xi[Big React Demo, window=_blank] + diff --git a/modules/ROOT/pages/variables.adoc b/modules/ROOT/pages/variables.adoc index dafd73e9c..49d2feb49 100644 --- a/modules/ROOT/pages/variables.adoc +++ b/modules/ROOT/pages/variables.adoc @@ -1,4 +1,4 @@ -= Custom variables += Variables :toc: true :toclevels: 2 @@ -102,7 +102,7 @@ In ThoughtSpot, the DOUBLE data type is used for columns that require floating-p * `DATE` + Date in the epoch format. If you are creating a formula variable for a specific start date, set the data type as `DATE` and configure the date value in epoch format. For example, `1711933200` for 1 April 2024 (GMT). * `DATE_TIME` + -Date with time stamp. If you are creating a formula variable for specific timestamp, specify the data type as `DATE_TIME` and assign a timestanp in epoch format as variable value. For example, `1711933200000` for 1 April 2024 01:00:00 (GMT). +Date with time stamp. If you are creating a formula variable for specific timestamp, specify the data type as `DATE_TIME` and assign a timestamp in epoch format as variable value. For example, `1711933200000` for 1 April 2024 01:00:00 (GMT). + [NOTE] The `data_type` is required only for formula variables and is not supported for other variable types. @@ -143,7 +143,7 @@ curl -X POST \ --data-raw '{ "type": "TABLE_MAPPING", "name": "schema_var", - "is_sensitive": true, + "is_sensitive": true }' ---- @@ -169,14 +169,14 @@ The following example shows the request body for the connection property variabl [source,cURL] ---- curl -X POST \ - --url 'https://{ThoughtSpot-Host}/]api/rest/2.0/template/variables/create' \ + --url 'https://{ThoughtSpot-Host}/api/rest/2.0/template/variables/create' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer {AUTH_TOKEN}' \ --data-raw '{ "type": "CONNECTION_PROPERTY", "name": "accountNameVar", - "is_sensitive": false, + "is_sensitive": false }' ---- @@ -311,7 +311,7 @@ Adds new values. Use this operation type to assign values to the variable. Replaces the existing attributes with new values. * `REMOVE` + Removes the values assigned to the variable. For example, you can remove the values assigned to a variable configured for an Org. -* `RESET + +* `RESET` + Resets all values at the variable level. For example, if a variable is assigned to multiple entities such as Org, user, or user group, the reset operation clears the values assigned to the variable for all entities. .5+|`variable_value_scope` 2+| Set the scope for variable values. These properties determine the entity level, such as Org, user, or user-group, for which the values will apply. @@ -405,7 +405,7 @@ curl -X POST \ "analyst" ], "operation": "ADD" - }, + } ], "variable_value_scope": [ { @@ -470,7 +470,7 @@ curl -X POST \ "1743469200" ], "operation": "ADD" - }, + } ], "variable_value_scope": [ { @@ -482,7 +482,7 @@ curl -X POST \ }, { "org_identifier": "Org_Sales", - "model_identifier": "cd252e5c-b552-49a8-821d-3eadaa049cca" + "model_identifier": "cd252e5c-b552-49a8-821d-3eadaa049cca", "principal_type": "USER", "principal_identifier": "tsuser" } @@ -531,7 +531,7 @@ curl -X POST \ -H 'Authorization: Bearer {AUTH_TOKEN}' \ --data-raw '{ "record_offset": 0, - "record_size": -10, + "record_size": -1, "output_format": "METADATA", "variable_details": [ { @@ -567,7 +567,7 @@ If the request is successful, the API returns the variable data in the response: == Delete a variable -To delete a variable, send a `POST` request to the `/api/rest/2.0/template/variables/{identifier}/delete` API endpoint, with the variable ID in the path parameter. +To delete a variable, send a `POST` request to the `/api/rest/2.0/template/variables/{identifier}/delete` API endpoint, with the variable ID in the path parameter. Note that you can delete only one variable at a time. diff --git a/modules/ROOT/pages/whats-new.adoc b/modules/ROOT/pages/whats-new.adoc index 9268e772e..fd41f2823 100644 --- a/modules/ROOT/pages/whats-new.adoc +++ b/modules/ROOT/pages/whats-new.adoc @@ -1282,7 +1282,7 @@ For more information, see xref:version_control.adoc[Git integration and version ==== The Visual Embed SDK provides a new JavaScript library to embed the Search page with AI-powered features such as natural language search and AI-suggested answers. To view the AI-suggested answers, make sure the AI search support is enabled on the data source or worksheet used for searching data. -ThoughtSpot does not display AI-suggested search responses if the xref:search-assist-tse.adoc[Search Assist] feature is enabled. +//ThoughtSpot does not display AI-suggested search responses if the xref:search-assist-tse.adoc[Search Assist] feature is enabled. For more information, see xref:SageEmbed.adoc[SageEmbed SDK reference]. ==== @@ -1803,7 +1803,7 @@ For information about REST API enhancements, see xref:rest-apiv1-changelog.adoc[ ==== Your application users can now access sample search walkthrough lessons created using Search Assist on embedded ThoughtSpot instances. If the Search Assist feature is enabled in the SDK, and the Search Assist lessons are created on the Worksheet, users can view sample search questions and follow the actions in the walkthrough to get answers. -For more information, see xref:search-assist-tse.adoc[Enable Search Assist, window=_blank]. +//For more information, see xref:search-assist-tse.adoc[Enable Search Assist, window=_blank]. ==== .Visual Embed SDK 1.13.0 diff --git a/static/doc-images/images/copy_edit_change_obj_id.png b/static/doc-images/images/copy_edit_change_obj_id.png new file mode 100644 index 000000000..6c49fd153 Binary files /dev/null and b/static/doc-images/images/copy_edit_change_obj_id.png differ diff --git a/static/doc-images/images/instances_and_orgs.png b/static/doc-images/images/instances_and_orgs.png new file mode 100644 index 000000000..bba727723 Binary files /dev/null and b/static/doc-images/images/instances_and_orgs.png differ diff --git a/static/doc-images/images/multi_tenant_deployment.png b/static/doc-images/images/multi_tenant_deployment.png new file mode 100644 index 000000000..ea6740467 Binary files /dev/null and b/static/doc-images/images/multi_tenant_deployment.png differ diff --git a/static/doc-images/images/publishing_diagram.png b/static/doc-images/images/publishing_diagram.png new file mode 100644 index 000000000..526ba164c Binary files /dev/null and b/static/doc-images/images/publishing_diagram.png differ diff --git a/static/doc-images/images/single_tenant_deployment.png b/static/doc-images/images/single_tenant_deployment.png new file mode 100644 index 000000000..effe7d8e8 Binary files /dev/null and b/static/doc-images/images/single_tenant_deployment.png differ diff --git a/static/doc-images/images/single_tenant_publishing.png b/static/doc-images/images/single_tenant_publishing.png new file mode 100644 index 000000000..d8de9f49d Binary files /dev/null and b/static/doc-images/images/single_tenant_publishing.png differ diff --git a/static/doc-images/images/trusted-auth-new-1.png b/static/doc-images/images/trusted-auth-new-1.png new file mode 100644 index 000000000..69b66cd0a Binary files /dev/null and b/static/doc-images/images/trusted-auth-new-1.png differ diff --git a/static/doc-images/images/version-control-branches.png b/static/doc-images/images/version-control-branches.png new file mode 100644 index 000000000..11f5a8585 Binary files /dev/null and b/static/doc-images/images/version-control-branches.png differ