Adding a dataframe backend

A backend connects one or more dataframe classes and one native schema class to Data Joinery. The DataFrameBackend[Frame, NativeSchema] protocol makes those types part of the interface: create_dataframe and coerce_dataframe return Frame, while compile_schema returns NativeSchema. Register one backend instance with register_backend.

Import the protocol and registration/lookup functions from data_joinery.backends. The protocol is defined in backends/protocol.py; registry state, registration validation, lookup, and lazy built-in discovery live in backends/registry.py. Importing these modules does not load Polars or PySpark. Lookup discovers installed built-in backends when no registered backend matches

What to implement

Member	What it does
`name`	Gives the backend a unique name.
`dataframe_types`	Lists the frame classes accepted and returned by this backend.
`schema_type`	Identifies the native schema class returned by `compile_schema`.
`compile_schema(schema)`	Maps each `ModelSchema` field and annotation to a native schema. Handle nested and backend-specific field types here.
`create_dataframe(rows, schema, frame_type, **kwargs)`	Checks the model rows, builds the requested frame class with the compiled schema, and handles backend-specific options. Reject unknown options.
`coerce_dataframe(frame, schema, mode)`	Checks or reshapes a frame according to the requested mode. Return the same frame class or raise `SchemaCoercionError` for schema mismatches.

Use ModelSchema[T].serialize_rows(rows) to validate model instances before creating a frame. It returns list[dict[str, Any]]: dataclass rows become dictionaries through dataclasses.asdict(), and Pydantic rows through model_dump(mode="python"). Nested models are converted to dictionaries as well.

The four coercion modes have distinct promises:

Mode	Required behavior
`strict`	Check field names and types without removing or casting fields. Extra and missing fields fail.
`project_top_level`	Remove extra top-level fields. Check nested fields and types without changing them.
`project`	Remove extra fields, including nested fields. Check types without casting.
`project_cast`	Project fields and cast types where the native library allows it. Report casts that cannot be made.

Every mode rejects missing fields. See Coercing Schemas for the user-facing behavior. Use SchemaCoercionError and SchemaDifference to describe missing, additional, and mismatched fields.

A small example

This toy frame holds only column names, so its native schema is a tuple of names. It shows the registration and method shapes; a real backend must also handle values, field types, and nested fields.

from collections.abc import Sequence
from dataclasses import dataclass
from typing import Any

from data_joinery import Schema
from data_joinery.backends import DataFrameBackend, register_backend
from data_joinery.schemas.coercion_error import (
    CoercionMode,
    SchemaCoercionError,
    SchemaDifference,
)
from data_joinery.schemas.model_schema import ModelSchema


@dataclass(frozen=True)
class ExampleDataFrame:
    columns: tuple[str, ...]


class ExampleBackend(DataFrameBackend[ExampleDataFrame, tuple[str, ...]]):
    name = "example"
    dataframe_types = (ExampleDataFrame,)
    schema_type = tuple

    def compile_schema(self, schema: ModelSchema[Any]) -> tuple[str, ...]:
        return tuple(field.name for field in schema.fields)

    def coerce_dataframe(
        self,
        dataframe: ExampleDataFrame,
        schema: ModelSchema[Any],
        mode: CoercionMode,
    ) -> ExampleDataFrame:
        expected = self.compile_schema(schema)
        missing = tuple(
            SchemaDifference("missing", name, None, name)
            for name in expected
            if name not in dataframe.columns
        )
        additional = tuple(
            SchemaDifference("additional", name, name, None)
            for name in dataframe.columns
            if name not in expected
        )
        violations = missing + (additional if mode == "strict" else ())
        if violations:
            raise SchemaCoercionError(mode, violations)
        if mode == "strict":
            return dataframe
        return ExampleDataFrame(expected)

    def create_dataframe(
        self,
        rows: Sequence[object],
        schema: ModelSchema[Any],
        frame_type: type[ExampleDataFrame],
        **kwargs: object,
    ) -> ExampleDataFrame:
        if kwargs:
            raise TypeError(f"Unexpected options: {', '.join(kwargs)}")
        schema.serialize_rows(rows)
        return frame_type(self.compile_schema(schema))


backend: DataFrameBackend[ExampleDataFrame, tuple[str, ...]] = ExampleBackend()
register_backend(backend)


@dataclass
class Order:
    order_id: int


print(Schema(Order).native_schema(tuple))
print(Schema(Order).create_dataframe([Order(1)], ExampleDataFrame))

Outputs:

('order_id',)
ExampleDataFrame(columns=('order_id',))

dataframe_types is a tuple of classes, while schema_type takes one class. Subclasses are recognized when finding a backend; when creating a frame or native schema, the returned object must also match the class requested by the caller. Registration rejects duplicate names and overlapping frame or schema classes across backends, so each type has one clear owner.

Connecting it to the rest of the library

Register the backend before using its frame or schema class. Schema.native_schema() finds it by schema_type; Schema.create_dataframe() and frame coercion find it by the classes in dataframe_types. Transform annotations use the same lookup.

Backends written against the earlier protocol need to replace dataframe_type with dataframe_types = (FrameClass,) and accept frame_type in create_dataframe().

Put the backend's native type mapping, row conversion, schema comparison, projection, and casting rules in its implementation. The built-in Polars and PySpark backends are complete examples in src/data_joinery/backends/polars/ and src/data_joinery/backends/spark/.