fix: missing $schema is explicitely Data Package v1

CHANGELOG.md

@@ -2,11 +2,17 @@
 
 Here described only the breaking and most significant changes. The full changelog and documentation for all released versions could be found in nicely formatted [commit history](https://github.com/frictionlessdata/frictionless-py/commits/main).
 
-## Next
+## v5.20.1
 
 - feat: Data Package v2, support `exclusiveMinimum`, `exclusiveMaximum` 
   properties 
   ([#1840](https://github.com/frictionlessdata/frictionless-py/pull/1840))
+- feat: Data Package v2, relax constraints on `name` property
+- doc: new page "Advanced Topics" > "Data Package version" that details how 
+  frictionless-py handles different versions of datapackage.
+- fix: missing `$schema` property is properly interpreted as datapackage 
+  version 1 ; with custom `$schema` profile, the validation is lenient and 
+  handles a superset of v1 and v2 features
 
 ## v5.20.0
 

data/custom-profile.json

@@ -0,0 +1,3 @@
+{
+    "type": "object"
+}

docs/advanced/datapackage-versions.md

@@ -0,0 +1,87 @@
+# Data Package versions
+
+This page documents how the framework resolves and handles the Data Package
+standard versions (v1 and v2).
+
+## Declaring a version
+
+A descriptor declares its version through `$schema`, a JSON Schema profile URL:
+
+```python
+package = Package({
+    "$schema": "https://datapackage.org/profiles/2.0/datapackage.json",
+    "name": "package",
+    # ... v2 features are accepted from here on
+})
+```
+
+The standard profiles are hosted at `https://datapackage.org/profiles/{1.0,2.0}/{datapackage,dataresource,tableschema,tabledialect}.json`.
+
+## Resolution rules
+
+The version of a descriptor is resolved once, top-down, at
+descriptor-validation time. In accordance with the specification, only the
+root object is expected to have a `$schema` property; all child objects
+inherit the resolved version of the root.
+
+The resolved version depends on the descriptor's own declaration:
+   - if its `$schema` is a standard `datapackage.org` profile URL, the
+     version is resolved to the version in its URL (v1 or v2);
+   - if its `$schema` is a custom profile URL, the version resolves to
+     `"unknown"`: validation will be lenient and accept a **superset of v1
+     and v2** (instead of guessing a standard version);
+   - an absent `$schema` is an explicit v1, as mandated by the spec.
+
+| Descriptor state                     | Resolved version        |
+| ---                                  | ---                     |
+| Under a v1/v2/unknown ancestor       | the ancestor's version  |
+| `$schema` standard profile URL       | v1 or v2 (from the URL) |
+| `$schema` custom URL                 | unknown (superset)      |
+| `$schema` absent                     | v1                      |
+
+### Version gates
+
+v2-specific features apply on explicit v2 version or when version is unknown.
+These features may raise errors or be ignored however when validating with
+unambiguous v1 version. See table below for examples of behavior changes.
+
+| Feature | v1 | v2 / unknown |
+| --- | --- | --- |
+| `name` restricted to the v1 pattern | enforced | any string |
+| `missingValues` object form `{value, label?}` | rejected | accepted |
+| `exclusiveMinimum` / `exclusiveMaximum` | rejected | accepted |
+
+## Detached children keep the inherited version
+
+`Resource` and `Schema` store the version they resolved at import time (a
+private `_datapackage_version`). This stored version is threaded top-down at
+import, and seeded back into every detached validation, so that:
+
+- `package.validate()` (which validates each resource standalone) agrees with `package.validate_descriptor()`;
+- a resource extracted from a package, a `to_copy()` copy, or a transform pipeline step keeps the semantics of its tree instead of re-resolving positionally;
+- a path-referenced schema lazily loaded by a resource inherits the resource's version.
+
+Exports stay canonical: by default, exporting the descriptor of children and
+reimporting them will change the behavior. The absent `$schema` property of the
+children is interpreted as v1 at re-import. Exporting with `upgrade=True`
+produces however a standalone descriptor of a child (see below).
+
+## Upgrading to v2
+
+`upgrade()` returns a new instance (the original is untouched) whose descriptor tree is transformed into the v2 form — also available as an export option: `to_descriptor(upgrade=True)`.
+
+- the root gets `$schema` (as its first property) set to the canonical v2 profile of the class (`Package` → `datapackage.json`, `Resource` → `dataresource.json`, `Schema` → `tableschema.json`, `Dialect` → `tabledialect.json`);
+- `$schema` is root-only: a child `$schema` is removed (it would be ignored by the inheritance rules anyway);
+- the legacy `profile` property is migrated: a custom profile URL is retained
+  in `$schema`; `"tabular-data-resource"` becomes `"type": "table"`; other
+  values are dropped.
+
+A `$schema` or `profile` pointing to a **custom profile URL is kept as-is**:
+an automatic upgrade of a custom profile is currently not supported. Its
+upgrade is up to its author. As a consequence, an upgraded custom profile
+resolves to `"unknown"` (permissive).
+
+```python
+package = Package("package-v1.json")
+v2_package = package.upgrade()  # a new, v2 instance
+```

frictionless/dialect/dialect.py

@@ -251,6 +251,10 @@ def blank_filter(cells: List[Any]):
     # Metadata
 
     metadata_type = "dialect"
+    metadata_v2_profile = "https://datapackage.org/profiles/2.0/tabledialect.json"
+    """
+    Canonical Data Package v2 profile URL (used by `upgrade`).
+    """
     metadata_Error = errors.DialectError
     metadata_profile = {
         "type": "object",

frictionless/formats/sql/__spec__/test_adapter.py

@@ -176,10 +176,15 @@ def test_sql_adapter_constraints_not_valid_error(sqlite_url, field_name, cell):
 
 @pytest.mark.parametrize("cell, valid", [(5, True), (4, False), (8, False)])
 def test_sql_adapter_exclusive_constraints(sqlite_url, cell, valid):
+    # The exclusive bounds being a v2 addition, the v2 `$schema` is declared
+    # both on the schema (imported standalone: an absent `$schema` is an
+    # explicit v1) and on the resource (which imposes its version top-down on
+    # the schema and its fields)
     resource = TableResource(
         data=[["value"], [cell]],
         schema=Schema.from_descriptor(
             {
+                "$schema": "https://datapackage.org/profiles/2.0/tableschema.json",
                 "fields": [
                     {
                         "name": "value",
@@ -189,6 +194,7 @@ def test_sql_adapter_exclusive_constraints(sqlite_url, cell, valid):
                 ]
             }
         ),
+        schema_profile="https://datapackage.org/profiles/2.0/dataresource.json",
     )
     control = formats.SqlControl(table="table")
     if valid:

frictionless/metadata/__spec__/test_custom_profile.py

@@ -0,0 +1,201 @@
+from frictionless import Package, Resource, Schema
+
+# A custom `$schema` URL — the v2 way to point at a profile — makes the Data
+# Package version known to nobody but the profile's author: it resolves to
+# "unknown", and version gates then accept a superset of v1 and v2 (instead
+# of guessing a standard version). Only `$schema` plays in the resolution:
+# a legacy `profile` is ignored when a `$schema` is declared (full backward
+# compatibility), and resolves to v1 without one (absent `$schema`:
+# explicit v1).
+
+CUSTOM = "https://example.com/my-profile.json"
+
+V1_PROFILE = "https://datapackage.org/profiles/1.0/datapackage.json"
+V2_PROFILE = "https://datapackage.org/profiles/2.0/datapackage.json"
+
+
+# Version resolution
+
+
+def test_custom_profile_resolves_to_unknown_version():
+    # a `$schema` pointing outside the datapackage.org profiles
+    assert Package.effective_datapackage_version({"$schema": CUSTOM}, None) == "unknown"
+
+    # a legacy `profile` (standards v1) without a `$schema` resolves to
+    # v1 (absent `$schema`: explicit v1), whatever the profile value
+    assert Package.effective_datapackage_version({"profile": "data-package"}, None) == "v1"
+    assert (
+        Package.effective_datapackage_version({"profile": "tabular-data-package"}, None)
+        == "v1"
+    )
+    assert (
+        Resource.effective_datapackage_version({"profile": "data-resource"}, None) == "v1"
+    )
+    assert (
+        Package.effective_datapackage_version({"profile": CUSTOM}, None) == "v1"
+    )
+    assert (
+        Package.effective_datapackage_version(
+            {"profile": "data/custom-profile.json"}, None
+        )
+        == "v1"
+    )
+
+    # standard profiles and an absent `$schema` are unaffected
+    assert Package.effective_datapackage_version({"$schema": V1_PROFILE}, None) == "v1"
+    assert Package.effective_datapackage_version({"$schema": V2_PROFILE}, None) == "v2"
+    assert Package.effective_datapackage_version({}, None) == "v1"
+
+    # a legacy `profile` coexisting with a `$schema` is ignored: the
+    # version is the `$schema`'s (declaring both behaves exactly as
+    # `$schema` alone — full backward compatibility)
+    assert (
+        Package.effective_datapackage_version(
+            {"$schema": V2_PROFILE, "profile": "data-package"}, None
+        )
+        == "v2"
+    )
+    assert (
+        Package.effective_datapackage_version(
+            {"$schema": V1_PROFILE, "profile": "data-package"}, None
+        )
+        == "v1"
+    )
+
+
+def test_unknown_version_is_inherited_top_down():
+    # an ancestor's resolved version wins (a descendant cannot escape it):
+    # "unknown" is imposed on the children like any other version
+    assert (
+        Resource.effective_datapackage_version({"profile": CUSTOM}, "v1") == "v1"
+    )
+    assert (
+        Resource.effective_datapackage_version({"$schema": V2_PROFILE}, "unknown")
+        == "unknown"
+    )
+
+
+# Superset of v1 and v2
+
+SUPERSET_DESCRIPTOR = {
+    "$schema": CUSTOM,
+    "name": "My Package",  # invalid per the v1 pattern
+    "resources": [
+        {
+            "name": "My Table",  # invalid per the v1 pattern
+            "data": [["name"], [1], [2]],
+            "schema": {
+                "fields": [
+                    # `exclusiveMinimum` is a v2-only constraint
+                    {"name": "name", "type": "integer", "constraints": {"exclusiveMinimum": 0}},
+                ],
+                # the object form is a v2-only feature
+                "missingValues": [{"value": "-99", "label": "REFUSED"}],
+            },
+        }
+    ],
+}
+
+
+def test_custom_profile_accepts_superset():
+    report = Package.validate_descriptor(SUPERSET_DESCRIPTOR)
+    assert report.flatten(["type", "note"]) == []
+    assert report.valid
+
+    package = Package(SUPERSET_DESCRIPTOR)
+    resource = package.get_table_resource("My Table")
+    assert resource._datapackage_version == "unknown"
+    assert resource.schema._datapackage_version == "unknown"
+
+    # the detached child keeps the unknown (permissive) version
+    assert resource.validate().valid
+    assert resource.to_copy()._datapackage_version == "unknown"
+
+
+def test_legacy_profile_property_is_explicit_v1():
+    # a custom profile referenced through the legacy `profile` property
+    # (fetched for validation — hence a real file) is a v1-era profile: the
+    # descriptor stays explicitly v1, and the version gates stay strict
+    descriptor = {
+        "profile": "data/custom-profile.json",
+        "name": "p",
+        "resources": [
+            {
+                "name": "r",
+                "data": [["name"], ["a"]],
+                "schema": {
+                    "fields": [{"name": "name", "type": "string"}],
+                    "missingValues": [{"value": "-99", "label": "REFUSED"}],
+                },
+            }
+        ],
+    }
+    report = Package.validate_descriptor(descriptor)
+    assert report.flatten(["type", "note"]) == [
+        ["schema-error", "missing values in object form require datapackage v2"]
+    ]
+
+    # the import refuses the invalid descriptor; importing despite the
+    # errors confirms the explicit v1 resolution
+    package = Package.from_descriptor(descriptor, allow_invalid=True)
+    assert package.get_table_resource("r")._datapackage_version == "v1"
+
+
+def test_child_cannot_escape_v1_with_a_custom_profile():
+    # a v1 package (an absent `$schema`) imposes v1 on a resource whose own
+    # profile is custom: the version gates stay strict
+    descriptor = {
+        "name": "p",
+        "resources": [
+            {
+                "name": "r",
+                "profile": CUSTOM,
+                "path": "data/table.csv",
+                "schema": {
+                    "fields": [{"name": "f", "type": "string"}],
+                    "missingValues": [{"value": "-99"}],
+                },
+            }
+        ],
+    }
+    report = Package.validate_descriptor(descriptor)
+    assert report.flatten(["type", "note"]) == [
+        ["schema-error", "missing values in object form require datapackage v2"]
+    ]
+
+
+def test_standalone_custom_schema_is_permissive():
+    schema = Schema(
+        {
+            "$schema": CUSTOM,
+            "fields": [
+                {"name": "n", "type": "integer", "constraints": {"exclusiveMinimum": 0}}
+            ],
+        }
+    )
+    assert schema._datapackage_version == "unknown"
+    assert schema.to_descriptor(validate=True)
+
+    resource = Resource.from_descriptor(
+        {
+            "$schema": CUSTOM,
+            "name": "My Resource",
+            "data": [["f"], [1]],
+            "schema": {
+                "fields": [
+                    {"name": "f", "type": "integer", "constraints": {"exclusiveMinimum": 0}}
+                ],
+            },
+        }
+    )
+    assert resource._datapackage_version == "unknown"
+    assert resource.validate().valid
+
+
+def test_upgrade_keeps_custom_profile_version_unknown():
+    upgraded = Package(SUPERSET_DESCRIPTOR).upgrade()
+    # the custom `$schema` is kept as-is (an automatic upgrade of a custom
+    # profile is not possible), so the version stays unknown (permissive)
+    assert upgraded._schema_profile == CUSTOM
+    assert upgraded.resources[0]._datapackage_version == "unknown"
+    assert upgraded.validate().valid