diff --git a/CHANGELOG.md b/CHANGELOG.md index c413e816a1..a22e7bfe59 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,11 +2,17 @@ Here described only the breaking and most significant changes. The full changelog and documentation for all released versions could be found in nicely formatted [commit history](https://github.com/frictionlessdata/frictionless-py/commits/main). -## Next +## v5.20.1 - feat: Data Package v2, support `exclusiveMinimum`, `exclusiveMaximum` properties ([#1840](https://github.com/frictionlessdata/frictionless-py/pull/1840)) +- feat: Data Package v2, relax constraints on `name` property +- doc: new page "Advanced Topics" > "Data Package version" that details how + frictionless-py handles different versions of datapackage. +- fix: missing `$schema` property is properly interpreted as datapackage + version 1 ; with custom `$schema` profile, the validation is lenient and + handles a superset of v1 and v2 features ## v5.20.0 diff --git a/data/custom-profile.json b/data/custom-profile.json new file mode 100644 index 0000000000..50c7cdcdc9 --- /dev/null +++ b/data/custom-profile.json @@ -0,0 +1,3 @@ +{ + "type": "object" +} diff --git a/docs/advanced/datapackage-versions.md b/docs/advanced/datapackage-versions.md new file mode 100644 index 0000000000..d6b7c1057e --- /dev/null +++ b/docs/advanced/datapackage-versions.md @@ -0,0 +1,87 @@ +# Data Package versions + +This page documents how the framework resolves and handles the Data Package +standard versions (v1 and v2). + +## Declaring a version + +A descriptor declares its version through `$schema`, a JSON Schema profile URL: + +```python +package = Package({ + "$schema": "https://datapackage.org/profiles/2.0/datapackage.json", + "name": "package", + # ... v2 features are accepted from here on +}) +``` + +The standard profiles are hosted at `https://datapackage.org/profiles/{1.0,2.0}/{datapackage,dataresource,tableschema,tabledialect}.json`. + +## Resolution rules + +The version of a descriptor is resolved once, top-down, at +descriptor-validation time. In accordance with the specification, only the +root object is expected to have a `$schema` property; all child objects +inherit the resolved version of the root. + +The resolved version depends on the descriptor's own declaration: + - if its `$schema` is a standard `datapackage.org` profile URL, the + version is resolved to the version in its URL (v1 or v2); + - if its `$schema` is a custom profile URL, the version resolves to + `"unknown"`: validation will be lenient and accept a **superset of v1 + and v2** (instead of guessing a standard version); + - an absent `$schema` is an explicit v1, as mandated by the spec. + +| Descriptor state | Resolved version | +| --- | --- | +| Under a v1/v2/unknown ancestor | the ancestor's version | +| `$schema` standard profile URL | v1 or v2 (from the URL) | +| `$schema` custom URL | unknown (superset) | +| `$schema` absent | v1 | + +### Version gates + +v2-specific features apply on explicit v2 version or when version is unknown. +These features may raise errors or be ignored however when validating with +unambiguous v1 version. See table below for examples of behavior changes. + +| Feature | v1 | v2 / unknown | +| --- | --- | --- | +| `name` restricted to the v1 pattern | enforced | any string | +| `missingValues` object form `{value, label?}` | rejected | accepted | +| `exclusiveMinimum` / `exclusiveMaximum` | rejected | accepted | + +## Detached children keep the inherited version + +`Resource` and `Schema` store the version they resolved at import time (a +private `_datapackage_version`). This stored version is threaded top-down at +import, and seeded back into every detached validation, so that: + +- `package.validate()` (which validates each resource standalone) agrees with `package.validate_descriptor()`; +- a resource extracted from a package, a `to_copy()` copy, or a transform pipeline step keeps the semantics of its tree instead of re-resolving positionally; +- a path-referenced schema lazily loaded by a resource inherits the resource's version. + +Exports stay canonical: by default, exporting the descriptor of children and +reimporting them will change the behavior. The absent `$schema` property of the +children is interpreted as v1 at re-import. Exporting with `upgrade=True` +produces however a standalone descriptor of a child (see below). + +## Upgrading to v2 + +`upgrade()` returns a new instance (the original is untouched) whose descriptor tree is transformed into the v2 form — also available as an export option: `to_descriptor(upgrade=True)`. + +- the root gets `$schema` (as its first property) set to the canonical v2 profile of the class (`Package` → `datapackage.json`, `Resource` → `dataresource.json`, `Schema` → `tableschema.json`, `Dialect` → `tabledialect.json`); +- `$schema` is root-only: a child `$schema` is removed (it would be ignored by the inheritance rules anyway); +- the legacy `profile` property is migrated: a custom profile URL is retained + in `$schema`; `"tabular-data-resource"` becomes `"type": "table"`; other + values are dropped. + +A `$schema` or `profile` pointing to a **custom profile URL is kept as-is**: +an automatic upgrade of a custom profile is currently not supported. Its +upgrade is up to its author. As a consequence, an upgraded custom profile +resolves to `"unknown"` (permissive). + +```python +package = Package("package-v1.json") +v2_package = package.upgrade() # a new, v2 instance +``` diff --git a/frictionless/dialect/dialect.py b/frictionless/dialect/dialect.py index 21532ff502..9bb0208196 100644 --- a/frictionless/dialect/dialect.py +++ b/frictionless/dialect/dialect.py @@ -251,6 +251,10 @@ def blank_filter(cells: List[Any]): # Metadata metadata_type = "dialect" + metadata_v2_profile = "https://datapackage.org/profiles/2.0/tabledialect.json" + """ + Canonical Data Package v2 profile URL (used by `upgrade`). + """ metadata_Error = errors.DialectError metadata_profile = { "type": "object", diff --git a/frictionless/formats/sql/__spec__/test_adapter.py b/frictionless/formats/sql/__spec__/test_adapter.py index e9d4c47d27..613234f0b8 100644 --- a/frictionless/formats/sql/__spec__/test_adapter.py +++ b/frictionless/formats/sql/__spec__/test_adapter.py @@ -176,10 +176,15 @@ def test_sql_adapter_constraints_not_valid_error(sqlite_url, field_name, cell): @pytest.mark.parametrize("cell, valid", [(5, True), (4, False), (8, False)]) def test_sql_adapter_exclusive_constraints(sqlite_url, cell, valid): + # The exclusive bounds being a v2 addition, the v2 `$schema` is declared + # both on the schema (imported standalone: an absent `$schema` is an + # explicit v1) and on the resource (which imposes its version top-down on + # the schema and its fields) resource = TableResource( data=[["value"], [cell]], schema=Schema.from_descriptor( { + "$schema": "https://datapackage.org/profiles/2.0/tableschema.json", "fields": [ { "name": "value", @@ -189,6 +194,7 @@ def test_sql_adapter_exclusive_constraints(sqlite_url, cell, valid): ] } ), + schema_profile="https://datapackage.org/profiles/2.0/dataresource.json", ) control = formats.SqlControl(table="table") if valid: diff --git a/frictionless/metadata/__spec__/test_custom_profile.py b/frictionless/metadata/__spec__/test_custom_profile.py new file mode 100644 index 0000000000..a7cc69054f --- /dev/null +++ b/frictionless/metadata/__spec__/test_custom_profile.py @@ -0,0 +1,201 @@ +from frictionless import Package, Resource, Schema + +# A custom `$schema` URL — the v2 way to point at a profile — makes the Data +# Package version known to nobody but the profile's author: it resolves to +# "unknown", and version gates then accept a superset of v1 and v2 (instead +# of guessing a standard version). Only `$schema` plays in the resolution: +# a legacy `profile` is ignored when a `$schema` is declared (full backward +# compatibility), and resolves to v1 without one (absent `$schema`: +# explicit v1). + +CUSTOM = "https://example.com/my-profile.json" + +V1_PROFILE = "https://datapackage.org/profiles/1.0/datapackage.json" +V2_PROFILE = "https://datapackage.org/profiles/2.0/datapackage.json" + + +# Version resolution + + +def test_custom_profile_resolves_to_unknown_version(): + # a `$schema` pointing outside the datapackage.org profiles + assert Package.effective_datapackage_version({"$schema": CUSTOM}, None) == "unknown" + + # a legacy `profile` (standards v1) without a `$schema` resolves to + # v1 (absent `$schema`: explicit v1), whatever the profile value + assert Package.effective_datapackage_version({"profile": "data-package"}, None) == "v1" + assert ( + Package.effective_datapackage_version({"profile": "tabular-data-package"}, None) + == "v1" + ) + assert ( + Resource.effective_datapackage_version({"profile": "data-resource"}, None) == "v1" + ) + assert ( + Package.effective_datapackage_version({"profile": CUSTOM}, None) == "v1" + ) + assert ( + Package.effective_datapackage_version( + {"profile": "data/custom-profile.json"}, None + ) + == "v1" + ) + + # standard profiles and an absent `$schema` are unaffected + assert Package.effective_datapackage_version({"$schema": V1_PROFILE}, None) == "v1" + assert Package.effective_datapackage_version({"$schema": V2_PROFILE}, None) == "v2" + assert Package.effective_datapackage_version({}, None) == "v1" + + # a legacy `profile` coexisting with a `$schema` is ignored: the + # version is the `$schema`'s (declaring both behaves exactly as + # `$schema` alone — full backward compatibility) + assert ( + Package.effective_datapackage_version( + {"$schema": V2_PROFILE, "profile": "data-package"}, None + ) + == "v2" + ) + assert ( + Package.effective_datapackage_version( + {"$schema": V1_PROFILE, "profile": "data-package"}, None + ) + == "v1" + ) + + +def test_unknown_version_is_inherited_top_down(): + # an ancestor's resolved version wins (a descendant cannot escape it): + # "unknown" is imposed on the children like any other version + assert ( + Resource.effective_datapackage_version({"profile": CUSTOM}, "v1") == "v1" + ) + assert ( + Resource.effective_datapackage_version({"$schema": V2_PROFILE}, "unknown") + == "unknown" + ) + + +# Superset of v1 and v2 + +SUPERSET_DESCRIPTOR = { + "$schema": CUSTOM, + "name": "My Package", # invalid per the v1 pattern + "resources": [ + { + "name": "My Table", # invalid per the v1 pattern + "data": [["name"], [1], [2]], + "schema": { + "fields": [ + # `exclusiveMinimum` is a v2-only constraint + {"name": "name", "type": "integer", "constraints": {"exclusiveMinimum": 0}}, + ], + # the object form is a v2-only feature + "missingValues": [{"value": "-99", "label": "REFUSED"}], + }, + } + ], +} + + +def test_custom_profile_accepts_superset(): + report = Package.validate_descriptor(SUPERSET_DESCRIPTOR) + assert report.flatten(["type", "note"]) == [] + assert report.valid + + package = Package(SUPERSET_DESCRIPTOR) + resource = package.get_table_resource("My Table") + assert resource._datapackage_version == "unknown" + assert resource.schema._datapackage_version == "unknown" + + # the detached child keeps the unknown (permissive) version + assert resource.validate().valid + assert resource.to_copy()._datapackage_version == "unknown" + + +def test_legacy_profile_property_is_explicit_v1(): + # a custom profile referenced through the legacy `profile` property + # (fetched for validation — hence a real file) is a v1-era profile: the + # descriptor stays explicitly v1, and the version gates stay strict + descriptor = { + "profile": "data/custom-profile.json", + "name": "p", + "resources": [ + { + "name": "r", + "data": [["name"], ["a"]], + "schema": { + "fields": [{"name": "name", "type": "string"}], + "missingValues": [{"value": "-99", "label": "REFUSED"}], + }, + } + ], + } + report = Package.validate_descriptor(descriptor) + assert report.flatten(["type", "note"]) == [ + ["schema-error", "missing values in object form require datapackage v2"] + ] + + # the import refuses the invalid descriptor; importing despite the + # errors confirms the explicit v1 resolution + package = Package.from_descriptor(descriptor, allow_invalid=True) + assert package.get_table_resource("r")._datapackage_version == "v1" + + +def test_child_cannot_escape_v1_with_a_custom_profile(): + # a v1 package (an absent `$schema`) imposes v1 on a resource whose own + # profile is custom: the version gates stay strict + descriptor = { + "name": "p", + "resources": [ + { + "name": "r", + "profile": CUSTOM, + "path": "data/table.csv", + "schema": { + "fields": [{"name": "f", "type": "string"}], + "missingValues": [{"value": "-99"}], + }, + } + ], + } + report = Package.validate_descriptor(descriptor) + assert report.flatten(["type", "note"]) == [ + ["schema-error", "missing values in object form require datapackage v2"] + ] + + +def test_standalone_custom_schema_is_permissive(): + schema = Schema( + { + "$schema": CUSTOM, + "fields": [ + {"name": "n", "type": "integer", "constraints": {"exclusiveMinimum": 0}} + ], + } + ) + assert schema._datapackage_version == "unknown" + assert schema.to_descriptor(validate=True) + + resource = Resource.from_descriptor( + { + "$schema": CUSTOM, + "name": "My Resource", + "data": [["f"], [1]], + "schema": { + "fields": [ + {"name": "f", "type": "integer", "constraints": {"exclusiveMinimum": 0}} + ], + }, + } + ) + assert resource._datapackage_version == "unknown" + assert resource.validate().valid + + +def test_upgrade_keeps_custom_profile_version_unknown(): + upgraded = Package(SUPERSET_DESCRIPTOR).upgrade() + # the custom `$schema` is kept as-is (an automatic upgrade of a custom + # profile is not possible), so the version stays unknown (permissive) + assert upgraded._schema_profile == CUSTOM + assert upgraded.resources[0]._datapackage_version == "unknown" + assert upgraded.validate().valid diff --git a/frictionless/metadata/__spec__/test_upgrade.py b/frictionless/metadata/__spec__/test_upgrade.py new file mode 100644 index 0000000000..2b240616da --- /dev/null +++ b/frictionless/metadata/__spec__/test_upgrade.py @@ -0,0 +1,163 @@ +from frictionless import Dialect, Package, Schema +from frictionless.resources.table import TableResource + +DP_V2 = "https://datapackage.org/profiles/2.0/datapackage.json" +DR_V2 = "https://datapackage.org/profiles/2.0/dataresource.json" +TS_V2 = "https://datapackage.org/profiles/2.0/tableschema.json" +TD_V2 = "https://datapackage.org/profiles/2.0/tabledialect.json" + + +# Descriptor-level transformation + + +def test_upgrade_descriptor_package(): + descriptor = { + "profile": "tabular-data-package", + "name": "package", + "resources": [ + { + "name": "table", + "profile": "tabular-data-resource", + "path": "table.csv", + "schema": { + # a `$schema` may be provided at the root only + "$schema": "https://datapackage.org/profiles/1.0/tableschema.json", + "fields": [{"name": "name", "type": "string"}], + }, + }, + {"name": "other", "profile": "data-resource", "path": "other.csv"}, + ], + } + upgraded = Package._upgrade_descriptor(descriptor) + + # `$schema` as the first property, set to the canonical v2 profile + assert list(upgraded)[0] == "$schema" + assert upgraded["$schema"] == DP_V2 + assert upgraded.pop("$schema") == DP_V2 + assert upgraded == { + "name": "package", + "resources": [ + { + "name": "table", + "path": "table.csv", + "type": "table", + "schema": {"fields": [{"name": "name", "type": "string"}]}, + }, + {"name": "other", "path": "other.csv"}, + ], + } + + +def test_upgrade_descriptor_custom_profile_urls(): + # a custom profile URL cannot be upgraded automatically: it is kept + # (in `$schema` at the root, in `profile` on a child, where `$schema` + # is not allowed) + descriptor = { + "$schema": "https://example.com/custom-package.json", + "name": "package", + "resources": [ + {"name": "other", "profile": "https://example.com/custom-resource.json"}, + ], + } + upgraded = Package._upgrade_descriptor(descriptor) + assert upgraded["$schema"] == "https://example.com/custom-package.json" + assert upgraded["resources"][0]["profile"] == "https://example.com/custom-resource.json" + assert "$schema" not in upgraded["resources"][0] + + # the same, provided through the legacy `profile` property + descriptor = { + "profile": "https://example.com/custom-package.json", + "name": "package", + "resources": [], + } + upgraded = Package._upgrade_descriptor(descriptor) + assert upgraded["$schema"] == "https://example.com/custom-package.json" + + +def test_upgrade_descriptor_standalone_classes(): + assert Package._upgrade_descriptor({"name": "p"})["$schema"] == DP_V2 + assert TableResource._upgrade_descriptor({"name": "t"})["$schema"] == DR_V2 + assert Schema._upgrade_descriptor({"fields": []})["$schema"] == TS_V2 + assert Dialect._upgrade_descriptor({})["$schema"] == TD_V2 + + +# Instance upgrade + + +def test_upgrade_returns_new_instance(): + descriptor = { + "name": "package", + "resources": [ + { + "name": "table", + "profile": "tabular-data-resource", + "path": "data/table.csv", + "schema": {"fields": [{"name": "name", "type": "string"}]}, + }, + ], + } + package = Package(descriptor) + upgraded = package.upgrade() + + # a new instance: the original is untouched (still v1) + assert upgraded is not package + assert package.to_descriptor().get("$schema") is None + assert package.resources[0].profile == "tabular-data-resource" + + # the upgraded tree is v2, root-only `$schema` + assert upgraded._schema_profile == DP_V2 + assert upgraded.resources[0].type == "table" + assert upgraded.resources[0]._schema_profile is None + assert upgraded.resources[0].schema._datapackage_version == "v2" + assert upgraded.to_descriptor(validate=True) + + # idempotent + assert upgraded.upgrade().to_descriptor() == upgraded.to_descriptor() + + +def test_upgrade_fixes_v1_invalid_names(): + # a name invalid per the v1 pattern ("My Package") is valid in v2 + package = Package.from_descriptor( + {"name": "My Package", "resources": [{"name": "t", "path": "data/table.csv"}]}, + allow_invalid=True, + ) + upgraded = package.upgrade() + assert upgraded.name == "My Package" + assert upgraded.to_descriptor(validate=True) + + +def test_upgrade_standalone_objects(): + schema = Schema({"fields": [{"name": "name", "type": "string"}]}) + upgraded = schema.upgrade() + assert upgraded._schema_profile == TS_V2 + assert upgraded.to_descriptor()["$schema"] == TS_V2 + + dialect = Dialect({"csv": {"delimiter": ";"}}) + assert dialect.upgrade()._schema_profile == TD_V2 + + +# Export option + + +def test_to_descriptor_upgrade_option(): + descriptor = { + "name": "package", + "resources": [ + { + "name": "table", + "path": "data/table.csv", + "schema": {"fields": [{"name": "name", "type": "string"}]}, + }, + ], + } + package = Package(descriptor) + exported = package.to_descriptor(upgrade=True) + + # the instance is not mutated + assert package.to_descriptor().get("$schema") is None + + # the exported tree is v2 and standalone-valid + assert list(exported)[0] == "$schema" + assert exported["$schema"] == DP_V2 + assert Package.validate_descriptor(exported).valid + assert package.to_descriptor(validate=True, upgrade=True) == exported diff --git a/frictionless/metadata/context.py b/frictionless/metadata/context.py index 7c29ea90ac..12f8ba8114 100644 --- a/frictionless/metadata/context.py +++ b/frictionless/metadata/context.py @@ -8,7 +8,7 @@ if TYPE_CHECKING: from referencing import Resource - from ..types import IStandards + from ..types import IResolvedStandards @attrs.frozen @@ -19,10 +19,13 @@ class ValidationContext: one (see `attrs.evolve`). """ - datapackage_version: Optional[IStandards] = None + datapackage_version: Optional[IResolvedStandards] = None """ - Data Package standard version imposed by an ancestor's `$schema` - (top-down inheritance). `None` when no ancestor declares one. + Data Package standard version imposed by an ancestor (top-down + inheritance: an ancestor's resolved version always wins over the + descriptor's own `$schema`). `None` when no ancestor has resolved one yet + (a fresh context); "unknown" when a custom `$schema` URL makes the + version known to nobody but its author. """ json_schema_cache: Dict[str, Resource] = attrs.field(factory=dict) diff --git a/frictionless/metadata/metadata.py b/frictionless/metadata/metadata.py index c829058452..2c95c607a9 100644 --- a/frictionless/metadata/metadata.py +++ b/frictionless/metadata/metadata.py @@ -78,6 +78,14 @@ class Metadata: `$schema` is not a valid Python identifier. """ + metadata_v2_profile = None + """ + Canonical Data Package v2 profile URL of this class, used by `upgrade` to + set the root `$schema` (e.g. "https://datapackage.org/profiles/2.0/ + datapackage.json" for a package). `None` when the class has no versioned + profile (e.g. a field). + """ + def __new__(cls, *args: Any, **kwargs: Any): obj = super().__new__(cls) obj.custom = obj.custom.copy() @@ -239,8 +247,13 @@ def from_descriptor( If `allow_invalid = True`, the class creation will try to continue despite the descriptor having errors. + + The optional "datapackage_version" option is the Data Package standard + version imposed by an ancestor (top-down inheritance); it seeds the + validation and the import, overriding the descriptor's own `$schema`. """ descriptor_path = None + inherited_version = options.pop("datapackage_version", None) if isinstance(descriptor, str): descriptor_path = descriptor @@ -268,20 +281,27 @@ def from_descriptor( Error = Class.metadata_Error or platform.frictionless_errors.MetadataError Class.metadata_transform(descriptor) - errors = list(Class.metadata_validate(descriptor)) + context = ValidationContext(datapackage_version=inherited_version) + errors = list(Class.metadata_validate(descriptor, context=context)) if not allow_invalid: if errors: error = Error(note="descriptor is not valid") raise FrictionlessException(error, reasons=errors) - metadata = Class.metadata_import(descriptor, **helpers.remove_non_values(options)) + metadata = Class.metadata_import( + descriptor, + datapackage_version=inherited_version, + **helpers.remove_non_values(options), + ) if descriptor_path: metadata.metadata_descriptor_path = descriptor_path metadata.metadata_descriptor_initial = metadata.to_descriptor() return metadata # type: ignore - def to_descriptor(self, *, validate: bool = False) -> types.IDescriptor: + def to_descriptor( + self, *, validate: bool = False, upgrade: bool = False + ) -> types.IDescriptor: """Export the instance as a descriptor. The descriptor is a canonical form of the metadata: it preserves all @@ -292,11 +312,25 @@ def to_descriptor(self, *, validate: bool = False) -> types.IDescriptor: descriptor. If `validate = True`, the descriptor will additionally be validated. + + If `upgrade = True`, the descriptor (and its children) is first + transformed into the Data Package v2 form — see `upgrade`. """ descriptor = self.metadata_export() + if upgrade: + descriptor = type(self)._upgrade_descriptor(descriptor) if validate: Error = self.metadata_Error or platform.frictionless_errors.MetadataError - errors = list(self.metadata_validate(descriptor)) + # Seed the context with the version inherited at import time (if + # any): a detached instance keeps the semantics of its tree. + # An upgraded descriptor is self-contained (its version is + # declared at the root): it is validated positionally + context = ValidationContext( + datapackage_version=None + if upgrade + else getattr(self, "_datapackage_version", None) + ) + errors = list(self.metadata_validate(descriptor, context=context)) if errors: error = Error(note="descriptor is not valid") raise FrictionlessException(error, reasons=errors) @@ -311,9 +345,35 @@ def to_descriptor_source(self) -> Union[types.IDescriptor, str]: return descriptor def to_copy(self, **options: Any) -> Self: - """Create a copy of the metadata""" + """Create a copy of the metadata + + The copy keeps the Data Package version the instance inherited from + its ancestors (if any), so that a detached copy validates like it did + within its tree. + """ + options.setdefault( + "datapackage_version", getattr(self, "_datapackage_version", None) + ) return type(self).from_descriptor(self.to_descriptor(), **options) + def upgrade(self) -> Self: + """Upgrade this metadata to the Data Package v2 standard + + Returns a new instance (this one is left untouched) whose descriptor + tree — this object and its children — is transformed into the v2 + form: the root gets `$schema` (as its first property) set to the + canonical v2 profile of the class, the legacy `profile` property is + migrated (a custom URL is retained in `$schema`; a + "tabular-data-resource" profile becomes "type": "table"), and + `$schema` never appears on children (it is removed: a `$schema` may + be provided at the root only). + + A `$schema` or `profile` pointing to a custom profile URL is kept + as-is: an automatic upgrade of a custom profile is not possible (its + upgrade path is up to its author). + """ + return type(self).from_descriptor(self.to_descriptor(upgrade=True)) + def to_dict(self) -> types.IDescriptor: """Export metadata as dictionary (alias for "to_descriptor")""" return self.to_descriptor() @@ -645,11 +705,13 @@ def metadata_validate( otherwise it defaults to the class profile. "context" carries the information shared down the recursion, such as - the Data Package standard version imposed by an ancestor's `$schema` - (top-down inheritance). When no version is imposed, the descriptor may - declare its own `$schema`; the resulting version is propagated to all - children. Subclasses read it (via `effective_datapackage_version`) - to gate version-specific properties. + the Data Package standard version resolved from an ancestor's `$schema` + (top-down inheritance). An absent `$schema` resolves to an explicit + "v1" (as mandated by the spec), a custom profile URL to "unknown" + (gates accept a superset of v1 and v2); the resulting version is + imposed on all children, whatever `$schema` they declare themselves. + Subclasses read it (via `effective_datapackage_version`) to gate + version-specific properties. """ context = context or ValidationContext() Error = error_class @@ -738,44 +800,137 @@ def _accepts_schema_profile(cls) -> bool: @staticmethod def _resolve_datapackage_version( schema_profile: Optional[str], - ) -> types.IStandards: - """Resolve the Data Package standard version from a `$schema` value. + ) -> types.IResolvedStandards: + """Resolve the Data Package standard version from a profile URL. - - a missing `$schema` defaults to "v1" (as mandated by the spec); - a standard datapackage.org profile uses the version from its URL; - - any custom profile falls back to the latest known standard. + - any custom profile URL resolves to "unknown": the version is known + to nobody but the profile's author, so version gates accept a + superset of v1 and v2. """ if not schema_profile: return "v1" url = urlparse(schema_profile) - if url.netloc == "datapackage.org": + if Metadata._is_standard_profile(schema_profile): match = re.search(r"/profiles/(\d+)\.\d+/", url.path) if match: if match.group(1) == "1": return "v1" if match.group(1) == "2": return "v2" - return settings.DEFAULT_STANDARDS + return "unknown" + + @staticmethod + def _is_standard_profile(schema_profile: str) -> bool: + """Whether a `$schema` value is a datapackage.org standard profile + URL (i.e. a profile this library knows how to upgrade)""" + url = urlparse(schema_profile) + return url.netloc == "datapackage.org" and re.search( + r"/profiles/\d+\.\d+/", url.path + ) is not None + + @classmethod + def _upgrade_descriptor( + cls, descriptor: types.IDescriptor, *, root: bool = True + ) -> types.IDescriptor: + """Transform a descriptor tree into the Data Package v2 form + + Root (only the root can declare `$schema`): + - the legacy `profile` property is migrated: a custom profile URL is + retained in `$schema` instead of the canonical v2 URL; + "tabular-data-resource" becomes "type": "table"; other values + (e.g. "data-package"/"data-resource") are dropped; + - a `$schema` pointing to a standard profile is replaced by the + canonical v2 profile of the class; a custom `$schema` URL is kept + as-is (an automatic upgrade of a custom profile is not possible); + - otherwise `$schema` is set (as the first property) to the canonical + v2 profile of the class. + + Children: `$schema` is removed (root-only) and the same legacy + `profile` handling is applied (a custom profile URL is kept in + place: it cannot be expressed as a child `$schema`). + """ + descriptor = dict(descriptor) + + # Children first: the same transformation as a root, but `$schema` + # is root-only (removed, never added) + profile = cls.metadata_ensure_profile() + for name in profile.get("properties", {}): + if name in ("$schema", "profile"): + continue + value = descriptor.get(name) + if not isinstance(value, (dict, list)): + continue + ItemClass = cls.metadata_select_property_class(name) + if not ItemClass: + continue + if isinstance(value, list): + for ix, item in enumerate(value): # type: ignore + if isinstance(item, dict): + TypedItemClass = ItemClass.metadata_select_class( + item.get("type") # type: ignore + ) + value[ix] = TypedItemClass._upgrade_descriptor( # type: ignore + item, root=False # type: ignore + ) + else: + descriptor[name] = ItemClass._upgrade_descriptor(value, root=False) # type: ignore + + # Legacy `profile` property (standards v1) + profile_url = None + profile_value = descriptor.pop("profile", None) + if profile_value == settings.DEFAULT_TABULAR_RESOURCE_PROFILE: + descriptor["type"] = "table" + elif profile_value and urlparse(profile_value).netloc: + # custom profile URL: retained (see the docstring) + profile_url = profile_value + + if not root: + descriptor.pop("$schema", None) # a `$schema` is root-only + if profile_url: + descriptor["profile"] = profile_url + return descriptor + + # `$schema` (first property) — only for the classes with a versioned + # profile (a field, for instance, has nothing to declare) + if cls.metadata_v2_profile: + schema_profile = descriptor.pop("$schema", None) + if profile_url: + schema_profile = profile_url + elif not schema_profile or cls._is_standard_profile(schema_profile): + schema_profile = cls.metadata_v2_profile + # else: a custom `$schema` URL is kept (see the docstring) + descriptor = {"$schema": schema_profile, **descriptor} + return descriptor @classmethod def effective_datapackage_version( cls, descriptor: types.IDescriptor, - inherited: Optional[types.IStandards], - ) -> Optional[types.IStandards]: + inherited: Optional[types.IResolvedStandards], + ) -> types.IResolvedStandards: """Version imposed top-down on this descriptor and its descendants. - An ancestor's `$schema` wins (a descendant cannot escape it); otherwise - the descriptor's own `$schema`, if any, sets the version. Returns `None` - when no `$schema` is declared anywhere in the chain (undeclared version, - which validation treats leniently). + An ancestor's resolved version wins (a descendant cannot escape it, + whatever `$schema` it declares); otherwise the descriptor's own + `$schema` sets the version. An absent `$schema` resolves to an + explicit "v1" (as mandated by the spec): only `$schema` plays — + a legacy `profile` coexisting with a `$schema` is ignored, so + declaring both behaves exactly as `$schema` alone (full backward + compatibility). A custom `$schema` URL resolves to "unknown" (its + version is known to nobody but its author, so version gates accept + a superset of v1 and v2). """ if inherited is not None: return inherited schema_profile = descriptor.get("$schema") if schema_profile: return cls._resolve_datapackage_version(schema_profile) - return None + # An absent `$schema` is an explicit "v1" (as mandated by the spec) + # for the classes that can declare one. A field never declares + # `$schema`: a standalone field carries no version of its own and + # only inherits one from its ancestors ("unknown": lax). + return "v1" if cls._accepts_schema_profile() else "unknown" @classmethod def metadata_import( @@ -788,10 +943,19 @@ def metadata_import( """Deserialization of a descriptor to a class instance The deserialization and serialization must be lossless. + + The optional "datapackage_version" option is the version imposed by + an ancestor (top-down inheritance); by default the version is + resolved positionally from this descriptor. It is threaded to the + children like in `metadata_validate`, and stored by the classes + declaring the `_datapackage_version` field so that a detached + validation keeps the tree semantics. """ merged_options = {} profile = cls.metadata_ensure_profile() basepath = options.pop("basepath", None) + inherited_version = options.pop("datapackage_version", None) + version = inherited_version or cls.effective_datapackage_version(descriptor, None) # Special handling of `$schema` # Only the subclasses declaring the `schema_profile` field @@ -815,17 +979,29 @@ def metadata_import( if isinstance(item, dict): type = item.get("type") # type: ignore ItemClass = Class.metadata_select_class(type) # type: ignore - value[ix] = ItemClass.metadata_import(item, basepath=basepath) # type: ignore + value[ix] = ItemClass.metadata_import( + item, # type: ignore + basepath=basepath, + datapackage_version=version, + ) elif isinstance(item, str): - value[ix] = Class.from_descriptor(item, basepath=basepath) + value[ix] = Class.from_descriptor( + item, basepath=basepath, datapackage_version=version + ) elif isinstance(value, dict): - value = Class.metadata_import(value, basepath=basepath) # type: ignore + value = Class.metadata_import( + value, # type: ignore + basepath=basepath, + datapackage_version=version, + ) merged_options.setdefault(stringcase.snakecase(name), value) # type: ignore merged_options.update(options) # type: ignore + if attrs.has(cls) and "_datapackage_version" in attrs.fields_dict(cls): + merged_options["datapackage_version"] = version if with_basepath: merged_options["basepath"] = basepath metadata = cls(**merged_options) - metadata.custom = descriptor + metadata.custom = descriptor # type: ignore return metadata def metadata_export(self, *, exclude: List[str] = []) -> types.IDescriptor: @@ -856,8 +1032,9 @@ def metadata_export(self, *, exclude: List[str] = []) -> types.IDescriptor: descriptor[name] = value # Special handling of `$schema` + # The spec recommends `$schema` as the first property if self._schema_profile is not None: - descriptor["$schema"] = self._schema_profile + descriptor = {"$schema": self._schema_profile, **descriptor} descriptor.update(self.custom) # type: ignore return descriptor # type: ignore diff --git a/frictionless/metadata/names.py b/frictionless/metadata/names.py index 055e034403..aa3077a697 100644 --- a/frictionless/metadata/names.py +++ b/frictionless/metadata/names.py @@ -6,15 +6,16 @@ from .. import settings if TYPE_CHECKING: - from ..types import IStandards + from ..types import IResolvedStandards -def validate_name(name: str, version: Optional[IStandards]) -> Optional[str]: +def validate_name(name: str, version: IResolvedStandards) -> Optional[str]: """Note when a package or resource name breaks the v1 pattern Datapackage v2 allows any string as the `name` of a package or a resource, - while v1 restricts it to `settings.NAME_PATTERN`. The pattern is only - checked when the version is explicitly resolved to v1. + while v1 restricts it to `settings.NAME_PATTERN` (v1 is also the version + of an absent `$schema`). An "unknown" version (custom profile) accepts + the superset: the pattern is not enforced. """ if version != "v1" or re.search(settings.NAME_PATTERN, name): return None diff --git a/frictionless/package/__spec__/test_validate.py b/frictionless/package/__spec__/test_validate.py index 7a54d2e1fd..0fd11f0ea8 100644 --- a/frictionless/package/__spec__/test_validate.py +++ b/frictionless/package/__spec__/test_validate.py @@ -763,9 +763,10 @@ def test_package_validate_missing_values_version_gate_inheritance( # # Datapackage v2 allows any string as a package or resource name, while v1 # restricts it to lowercase alphanumerics plus ".", "-", "_" and "/". As for -# missing values, the v1 pattern is only checked under an explicitly declared -# v1: a package `$schema` is imposed on its resources (top-down), otherwise the -# resource's own `$schema` sets the version, and an undeclared version is lenient. +# missing values, the v1 pattern is only checked under v1: an absent package +# `$schema` is an explicit v1, imposed on its resources (top-down) whatever +# `$schema` they declare; otherwise the resource's own `$schema` sets the +# version. DR_V1_PROFILE = "https://datapackage.org/profiles/1.0/dataresource.json" DR_V2_PROFILE = "https://datapackage.org/profiles/2.0/dataresource.json" @@ -776,9 +777,15 @@ def test_package_validate_missing_values_version_gate_inheritance( ) RESOURCE_NAME_VERSION_GATE_CASES = [ - ("undeclared", None, None, []), - ("resource-v1", None, DR_V1_PROFILE, [["resource-error", NAME_NOTE]]), - ("resource-v2", None, DR_V2_PROFILE, []), + # an absent package `$schema` is an explicit v1... + ("absent-package-is-v1", None, None, [["resource-error", NAME_NOTE]]), + # ...which the resource cannot escape by declaring its own `$schema` + ( + "resource-cannot-escape-absent-v1", + None, + DR_V2_PROFILE, + [["resource-error", NAME_NOTE]], + ), ("package-v2", DP_V2_PROFILE, None, []), ("package-v2-resource-v2", DP_V2_PROFILE, DR_V2_PROFILE, []), # a resource cannot escape the version imposed by the package @@ -817,11 +824,12 @@ def test_package_validate_resource_name_version_gate( @pytest.mark.parametrize( "package_profile, expected", [ - (None, []), + # an absent package `$schema` is an explicit v1 + (None, [["package-error", PACKAGE_NAME_NOTE]]), (DP_V1_PROFILE, [["package-error", PACKAGE_NAME_NOTE]]), (DP_V2_PROFILE, []), ], - ids=["undeclared", "package-v1", "package-v2"], + ids=["absent-is-v1", "package-v1", "package-v2"], ) def test_package_validate_package_name_version_gate(package_profile, expected): descriptor = { @@ -832,3 +840,53 @@ def test_package_validate_package_name_version_gate(package_profile, expected): descriptor["$schema"] = package_profile report = Package.validate_descriptor(descriptor) assert report.flatten(["type", "note"]) == expected + + +# Inherited version kept by detached children +# +# A resource without its own `$schema` inherits the package's version. The +# resolved version is stored on the instance at import time, so a detached +# child (per-resource validation inside `package.validate()`, `to_copy()`, +# transform pipelines) keeps the tree semantics instead of re-resolving +# positionally (an absent `$schema` being an explicit v1). + + +def test_package_validate_v2_inherited_by_detached_resource(): + descriptor = { + "$schema": DP_V2_PROFILE, + "resources": [ + { + "name": "table", + "data": [["name", "note"], ["a", "-99"], ["b", "ok"]], + "schema": { + "fields": [ + {"name": "name", "type": "string"}, + {"name": "note", "type": "string"}, + ], + # object form: a v2-only feature, inherited from the package + "missingValues": [{"value": "-99", "label": "REFUSED"}], + }, + } + ], + } + + # metadata validation of the whole tree + assert Package.validate_descriptor(descriptor).valid + + # full validation (per-resource detached validation inside) + package = Package(descriptor) + assert package.validate().valid + + # the extracted resource validates on its own (keeps the inherited v2) + resource = package.get_table_resource("table") + assert resource.validate().valid + + # a copy keeps the inherited v2 and the schema feature it enables + copy = resource.to_copy() + assert copy.validate().valid + assert copy.schema.missing_values == ["-99"] + + # the stored version is private: the exports stay canonical + # (no `$schema` injected anywhere) + assert "$schema" not in resource.to_descriptor() + assert "$schema" not in resource.schema.to_descriptor() diff --git a/frictionless/package/package.py b/frictionless/package/package.py index 4ce211f85c..b1006ea464 100644 --- a/frictionless/package/package.py +++ b/frictionless/package/package.py @@ -577,6 +577,10 @@ def to_er_diagram(self, path: Optional[str] = None) -> str: # Metadata metadata_type = "package" + metadata_v2_profile = "https://datapackage.org/profiles/2.0/datapackage.json" + """ + Canonical Data Package v2 profile URL (used by `upgrade`). + """ metadata_Error = errors.PackageError metadata_profile = { "type": "object", @@ -676,8 +680,9 @@ def metadata_validate( # type: ignore return # Name version gate - # The version is the one of this package's own `$schema`; `None` (undeclared) - # stays lenient. The profile above ensures that "name" is a string if present. + # The version is the one of this package's own `$schema` (an absent + # `$schema` resolves to an explicit "v1"), imposed on all children. + # The profile above ensures that "name" is a string if present. name = descriptor.get("name") if name is not None: version = cls.effective_datapackage_version( diff --git a/frictionless/resource/__spec__/test_validate.py b/frictionless/resource/__spec__/test_validate.py index de75b9ae93..37cd47b406 100644 --- a/frictionless/resource/__spec__/test_validate.py +++ b/frictionless/resource/__spec__/test_validate.py @@ -675,8 +675,8 @@ def test_resource_validate_invalid_resource_standards_v2_strict(): # Name (version gate) # -# Datapackage v2 allows any string as a resource name; the v1 pattern is only -# checked when the resource explicitly declares a v1 `$schema`. +# Datapackage v2 allows any string as a resource name; the v1 pattern is +# checked under v1 — which is also the version of an absent `$schema`. NAME_NOTE = "'My Table' does not match '^([-a-z0-9._/])+$' at property 'name'" @@ -684,7 +684,8 @@ def test_resource_validate_invalid_resource_standards_v2_strict(): @pytest.mark.parametrize( "schema, expected", [ - (None, []), + # an absent `$schema` is an explicit v1 + (None, [["resource-error", NAME_NOTE]]), ( "https://datapackage.org/profiles/1.0/dataresource.json", [["resource-error", NAME_NOTE]], diff --git a/frictionless/resource/__spec__/test_validate_schema.py b/frictionless/resource/__spec__/test_validate_schema.py index 4bffa67b1a..92bdfcdcf8 100644 --- a/frictionless/resource/__spec__/test_validate_schema.py +++ b/frictionless/resource/__spec__/test_validate_schema.py @@ -121,8 +121,13 @@ def test_resource_validate_schema_minimum_constraint(): def test_resource_validate_schema_exclusive_minimum_and_maximum_constraints(): source = [["row", "score"], [2, 1], [3, 2], [4, 3], [5, 4], [6]] + # The exclusive bounds being a v2 addition, the v2 `$schema` is declared + # both on the schema (imported standalone: an absent `$schema` is an + # explicit v1) and on the resource (which imposes its version top-down on + # the schema and its fields) schema = Schema.from_descriptor( { + "$schema": "https://datapackage.org/profiles/2.0/tableschema.json", "fields": [ {"name": "row", "type": "integer"}, { @@ -133,7 +138,11 @@ def test_resource_validate_schema_exclusive_minimum_and_maximum_constraints(): ] } ) - resource = TableResource(data=source, schema=schema) + resource = TableResource( + data=source, + schema=schema, + schema_profile="https://datapackage.org/profiles/2.0/dataresource.json", + ) checklist = Checklist(pick_errors=["constraint-error"]) report = resource.validate(checklist) assert report.flatten(["rowNumber", "fieldNumber", "type"]) == [ diff --git a/frictionless/resource/resource.py b/frictionless/resource/resource.py index b38f9ce386..68b740990a 100644 --- a/frictionless/resource/resource.py +++ b/frictionless/resource/resource.py @@ -210,6 +210,17 @@ class Resource(Metadata, metaclass=Factory): # type: ignore See `Metadata._schema_profile`. """ + _datapackage_version: Optional[types.IResolvedStandards] = attrs.field( + default=None, alias="datapackage_version" + ) + """ + Data Package standard version resolved at import time: the one imposed + by an ancestor (top-down inheritance), or this resource's own `$schema` + ("unknown" when a custom profile URL is involved). Private (not + exported): a detached validation keeps the tree semantics. See + `Metadata.effective_datapackage_version`. + """ + detector: Detector = attrs.field(factory=Detector) """ File/table detector. @@ -336,7 +347,13 @@ def dialect(self, value: Union[Dialect, str]): @property def schema(self) -> Schema: if isinstance(self._schema, str): - self._schema = Schema.from_descriptor(self._schema, basepath=self.basepath) + # The path-referenced schema is under this resource: it inherits + # the resource's resolved version (top-down) + self._schema = Schema.from_descriptor( + self._schema, + basepath=self.basepath, + datapackage_version=self._datapackage_version, + ) return self._schema @schema.setter @@ -748,6 +765,10 @@ def to_copy(self, **options: Any) -> Self: # Metadata metadata_type = "resource" + metadata_v2_profile = "https://datapackage.org/profiles/2.0/dataresource.json" + """ + Canonical Data Package v2 profile URL (used by `upgrade`). + """ metadata_Error = errors.ResourceError metadata_profile = { "type": "object", @@ -913,8 +934,9 @@ def metadata_validate( # type: ignore return # Name version gate - # The version is the one imposed top-down by an ancestor's `$schema`, or - # this resource's own `$schema` otherwise; `None` (undeclared) stays lenient. + # The version is the one imposed top-down by an ancestor (whatever + # `$schema` this resource declares), or resolved from this resource's + # own `$schema` otherwise (absent: explicit "v1"). # The profile above ensures that "name" is present and is a string. version = cls.effective_datapackage_version( descriptor, context.datapackage_version diff --git a/frictionless/schema/__spec__/test_convert.py b/frictionless/schema/__spec__/test_convert.py index f3e9fb0fd1..ac535a3ff4 100644 --- a/frictionless/schema/__spec__/test_convert.py +++ b/frictionless/schema/__spec__/test_convert.py @@ -61,6 +61,8 @@ def test_schema_to_copy(): # is preserved, the notation is not. Object entries are emitted only when at # least one label exists; otherwise the entries canonicalize to plain strings. +V2_PROFILE = "https://datapackage.org/profiles/2.0/tableschema.json" + MISSING_VALUES_EXPORT_CASES = [ ("strings", ["", "NA"], ["", "NA"]), ( @@ -78,7 +80,10 @@ def test_schema_to_copy(): ids=[case[0] for case in MISSING_VALUES_EXPORT_CASES], ) def test_schema_to_descriptor_missing_values_canonical_form(name, source, expected): + # The object form being a v2 addition, the descriptors declare a v2 + # `$schema` (an absent `$schema` is an explicit v1, which gates it out) descriptor = { + "$schema": V2_PROFILE, "fields": [{"name": "name", "type": "string"}], "missingValues": source, } @@ -86,7 +91,11 @@ def test_schema_to_descriptor_missing_values_canonical_form(name, source, expect assert exported == expected # the canonical form is a fixed point: re-importing it exports identically reimported = Schema.from_descriptor( - {"fields": [{"name": "name", "type": "string"}], "missingValues": exported} + { + "$schema": V2_PROFILE, + "fields": [{"name": "name", "type": "string"}], + "missingValues": exported, + } ) assert reimported.to_descriptor()["missingValues"] == exported diff --git a/frictionless/schema/__spec__/test_general.py b/frictionless/schema/__spec__/test_general.py index 43e4997be0..55732ac5fd 100644 --- a/frictionless/schema/__spec__/test_general.py +++ b/frictionless/schema/__spec__/test_general.py @@ -88,8 +88,11 @@ def test_schema_read_cells_null_values(): def test_schema_read_cells_object_missing_values(): + # The object form being a v2 addition, the descriptor declares a v2 + # `$schema` (an absent `$schema` is an explicit v1, which gates it out) schema = Schema.from_descriptor( { + "$schema": "https://datapackage.org/profiles/2.0/tableschema.json", "fields": [ {"name": "name", "type": "string"}, {"name": "age", "type": "integer"}, diff --git a/frictionless/schema/__spec__/test_validate.py b/frictionless/schema/__spec__/test_validate.py index 51e2ba1cd9..c3fb01255b 100644 --- a/frictionless/schema/__spec__/test_validate.py +++ b/frictionless/schema/__spec__/test_validate.py @@ -35,6 +35,9 @@ def test_validate_required_invalid(): # Object entries must have a unique value and an optional unique label; # absent labels do not collide with each other. Plain string entries keep # the lax v1 behavior (duplicates allowed). +# +# The object form being a v2 addition, these descriptors declare a v2 `$schema` +# (an absent `$schema` is an explicit v1, which gates the object form out). MISSING_VALUES_UNIQUENESS_CASES = [ ("valid-strings", ["", "NA"], []), @@ -73,7 +76,11 @@ def test_validate_required_invalid(): ) def test_validate_missing_values_uniqueness(name, source, expected): report = Schema.validate_descriptor( - {"fields": [{"name": "name", "type": "string"}], "missingValues": source} + { + "$schema": V2_PROFILE, + "fields": [{"name": "name", "type": "string"}], + "missingValues": source, + } ) assert report.flatten(["type", "note"]) == expected @@ -81,6 +88,7 @@ def test_validate_missing_values_uniqueness(name, source, expected): def test_validate_field_missing_values_uniqueness(): report = Schema.validate_descriptor( { + "$schema": V2_PROFILE, "fields": [ { "name": "name", @@ -90,7 +98,7 @@ def test_validate_field_missing_values_uniqueness(): {"value": "-99", "label": "OMITTED"}, ], } - ] + ], } ) assert report.flatten(["type", "note"]) == [ @@ -100,10 +108,10 @@ def test_validate_field_missing_values_uniqueness(): # Missing values (version gate) # -# The object form `{value, label?}` is a datapackage v2 addition. -# It is rejected only under an explicitly declared v1 `$schema`. -# An absent `$schema` (undeclared version) keeps accepting the v1 union v2 -# superset, and a custom (unrecognized) profile is treated as v2 by default. +# The object form `{value, label?}` is a datapackage v2 addition, rejected +# under v1 — which is also the version of an absent `$schema`. A custom +# (unrecognized) profile resolves to an "unknown" version: gates accept a +# superset of v1 and v2. V1_PROFILE = "https://datapackage.org/profiles/1.0/tableschema.json" V2_PROFILE = "https://datapackage.org/profiles/2.0/tableschema.json" @@ -122,7 +130,13 @@ def test_validate_field_missing_values_uniqueness(): ("v1-strings-valid", V1_PROFILE, ["", "NA"], []), ("v2-objects-valid", V2_PROFILE, OBJECT_MISSING_VALUES, []), ("custom-objects-valid", CUSTOM_PROFILE, OBJECT_MISSING_VALUES, []), - ("no-schema-objects-valid", None, OBJECT_MISSING_VALUES, []), + # an absent `$schema` is an explicit v1 + ( + "no-schema-objects-rejected", + None, + OBJECT_MISSING_VALUES, + [["schema-error", GATE_NOTE]], + ), ] @@ -144,10 +158,10 @@ def test_validate_missing_values_version_gate(name, schema, source, expected): # Exclusive bounds (version gate) # -# `exclusiveMinimum`/`exclusiveMaximum` are a datapackage v2 addition. -# They are rejected only under an explicitly declared v1 `$schema`. -# An absent `$schema` (undeclared version) keeps accepting the v1 union v2 -# superset, and a custom (unrecognized) profile is treated as v2 by default. +# `exclusiveMinimum`/`exclusiveMaximum` are a datapackage v2 addition, +# rejected under v1 — which is also the version of an absent `$schema`. A +# custom (unrecognized) profile resolves to an "unknown" version: gates +# accept a superset of v1 and v2. EXCLUSIVE_BOUNDS = {"exclusiveMinimum": 1, "exclusiveMaximum": 10} EXCLUSIVE_GATE_NOTES = [ @@ -165,7 +179,13 @@ def test_validate_missing_values_version_gate(name, schema, source, expected): ("v1-absent-valid", V1_PROFILE, {}, []), ("v2-valid", V2_PROFILE, EXCLUSIVE_BOUNDS, []), ("custom-profile-valid", CUSTOM_PROFILE, EXCLUSIVE_BOUNDS, []), - ("no-schema-valid", None, EXCLUSIVE_BOUNDS, []), + # an absent `$schema` is an explicit v1 + ( + "no-schema-rejected", + None, + EXCLUSIVE_BOUNDS, + [["field-error", note] for note in EXCLUSIVE_GATE_NOTES], + ), ] @@ -192,7 +212,8 @@ def test_validate_exclusive_bounds_version_gate(name, schema, constraints, expec MISSING_VALUES_FIELD_INHERITS_SCHEMA_CASES = [ ("v1-schema-rejects-field-objects", V1_PROFILE, [["field-error", GATE_NOTE]]), ("v2-schema-accepts-field-objects", V2_PROFILE, []), - ("no-schema-accepts-field-objects", None, []), + # an absent `$schema` is an explicit v1 + ("no-schema-rejects-field-objects", None, [["field-error", GATE_NOTE]]), ] diff --git a/frictionless/schema/field.py b/frictionless/schema/field.py index 08c9547583..c67061f381 100644 --- a/frictionless/schema/field.py +++ b/frictionless/schema/field.py @@ -287,7 +287,7 @@ def metadata_validate( # type: ignore # Missing Values version gate # A field never declares its own `$schema`; the version is the one - # imposed top-down by an ancestor (`None` undeclared stays lenient). + # imposed top-down by an ancestor. version = cls.effective_datapackage_version( descriptor, context.datapackage_version ) @@ -295,9 +295,9 @@ def metadata_validate( # type: ignore yield errors.FieldError(note=note) # Exclusive bounds version gate - # `exclusiveMinimum`/`exclusiveMaximum` are a datapackage v2 addition. - # They are rejected only when the version is explicitly resolved to v1; - # an undeclared version (`None`) stays lenient and accepts them. + # `exclusiveMinimum`/`exclusiveMaximum` are a datapackage v2 addition, + # rejected under unambiguous v1 (an "unknown" version, from a + # custom profile, accepts the superset). if version == "v1": for name in ("exclusiveMinimum", "exclusiveMaximum"): if name in descriptor.get("constraints", {}): diff --git a/frictionless/schema/missing_values.py b/frictionless/schema/missing_values.py index c430eb8cdf..3858f68069 100644 --- a/frictionless/schema/missing_values.py +++ b/frictionless/schema/missing_values.py @@ -9,10 +9,10 @@ from __future__ import annotations -from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union +from typing import TYPE_CHECKING, Any, Dict, List, Tuple, Union if TYPE_CHECKING: - from ..types import IStandards + from ..types import IResolvedStandards IEntries = List[Union[str, Dict[str, str]]] @@ -71,14 +71,12 @@ def validation_notes(entries: IEntries) -> List[str]: return notes -def version_gate_notes( - entries: IEntries, version: Optional[IStandards] -) -> List[str]: - """Notes when object entries appear under an explicitly declared v1 schema +def version_gate_notes(entries: IEntries, version: IResolvedStandards) -> List[str]: + """Notes when object entries appear under a v1 schema - The object form `{value, label?}` is a datapackage v2 addition. It is - rejected only when the version is explicitly resolved to v1; an undeclared - version (`None`) stays lenient and accepts the v1 union v2 superset. + The object form `{value, label?}` is a datapackage v2 addition, rejected + when the resolved version is unambiguously v1 (an "unknown" version, + from a custom profile, accepts the superset). """ if version != "v1": return [] diff --git a/frictionless/schema/schema.py b/frictionless/schema/schema.py index ce66d25cac..cf73300a34 100644 --- a/frictionless/schema/schema.py +++ b/frictionless/schema/schema.py @@ -41,6 +41,17 @@ class Schema(Metadata, metaclass=Factory): See `Metadata._schema_profile`. """ + _datapackage_version: Optional[types.IResolvedStandards] = attrs.field( + default=None, alias="datapackage_version" + ) + """ + Data Package standard version resolved at import time: the one imposed + by an ancestor (top-down inheritance), or this schema's own `$schema` + ("unknown" when a custom profile URL is involved). Private (not + exported): a detached validation keeps the tree semantics. See + `Metadata.effective_datapackage_version`. + """ + # TODO: why it's optional?? name: Optional[str] = None """ @@ -295,6 +306,10 @@ def to_summary(self) -> str: # Metadata metadata_type = "schema" + metadata_v2_profile = "https://datapackage.org/profiles/2.0/tableschema.json" + """ + Canonical Data Package v2 profile URL (used by `upgrade`). + """ metadata_Error = errors.SchemaError metadata_profile = { "type": "object", @@ -417,8 +432,9 @@ def metadata_validate( # type: ignore yield errors.SchemaError(note=note) # Missing Values version gate - # The version is the one imposed top-down by an ancestor's `$schema`, or - # this schema's own `$schema` otherwise; `None` (undeclared) stays lenient. + # The version is the one imposed top-down by an ancestor (whatever + # `$schema` this schema declares), or resolved from this schema's own + # `$schema` otherwise (absent: explicit "v1"). version = cls.effective_datapackage_version(descriptor, context.datapackage_version) for note in missing_values_module.version_gate_notes(missing_values, version): yield errors.SchemaError(note=note) diff --git a/frictionless/types.py b/frictionless/types.py index ac9bff6d49..d8a7d91cb1 100644 --- a/frictionless/types.py +++ b/frictionless/types.py @@ -24,6 +24,10 @@ IStandards = Literal["v1", "v2"] +IResolvedStandards = Literal["v1", "v2", "unknown"] +"""Data Package standard version resolved from a descriptor tree: "unknown" +when a custom `$schema`/`profile` URL makes the version unambiguous to +nobody but its author (gates then accept a superset of v1 and v2).""" IDescriptor = Dict[str, Any] IByteStream = BinaryIO ITextStream = TextIO diff --git a/livemark.yaml b/livemark.yaml index dd37b9abb6..5ffcad3a7b 100644 --- a/livemark.yaml +++ b/livemark.yaml @@ -82,6 +82,7 @@ pages: items: - path: docs/advanced/design - path: docs/advanced/system + - path: docs/advanced/datapackage-versions - path: docs/advanced/security - path: docs/advanced/extending - name: Data Resources