Skip to content

Plugin API

Plugins are the site-specific half of Ladon. A plugin bundles a Source (discovers top-level refs), one or more Expanders (fan out through the URL tree), and a Sink (fetches each leaf and returns a record). All protocols are structural (PEP 544) — no inheritance from Ladon is required.

Ladon ships two parallel protocol hierarchies: sync and async.

Running a plugin

Use run_plugin() for the normal whole-plugin path: it calls plugin.source.discover(client) once, runs every discovered root in source order, and returns a PluginRunResult. The aggregate exposes total leaf counts and errors while retaining a RunResult per root in results. RunConfig.leaf_limit applies separately to every discovered root. If a later root raises a globally fatal error, earlier roots may already have run their on_leaf callback; make that callback idempotent when retrying a whole plugin run.

run_crawl(top_ref, ...) remains available when the caller intentionally owns root discovery or needs to process one known root. The async equivalents are async_run_plugin() and async_run_crawl(); async whole-plugin runs keep root processing ordered, while each root's leaf work uses async_concurrency.

Sync protocols

typing.Protocol definitions for Ladon crawl plugins.

Adapters implement these protocols by structural subtyping — no inheritance from this module is required. This keeps third-party plugins decoupled from Ladon internals.

All adapters receive an object satisfying SyncHttpClientProtocol: either the native HttpClient or curl-cffi CurlHttpClient implementation. They must not construct their own HTTP sessions or import requests directly.

The three-layer pipeline is:

Source  →  [Expander, ...]  →  Sink

Source[RefT] produces top-level refs. Each Expander[RefT, RecordT, ChildRawT] takes a ref and returns an Expansion (record + child refs). Sink[RefT, RecordT] takes a leaf ref and returns a final record. CrawlPlugin[TopRefT, LeafRefT, LeafRecordT] bundles all three.

Source

Bases: Protocol[SourceRefT_co]

Discover top-level refs from an external source.

Source code in src/ladon/plugins/protocol.py 
53
54
55
56
57
58
59
60
61
@runtime_checkable
class Source(Protocol[SourceRefT_co]):
    """Discover top-level refs from an external source."""

    def discover(
        self, client: SyncHttpClientProtocol
    ) -> Sequence[SourceRefT_co]:
        """Return all discoverable top-level references."""
        ...

discover(client)

Return all discoverable top-level references.

Source code in src/ladon/plugins/protocol.py 
57
58
59
60
61
def discover(
    self, client: SyncHttpClientProtocol
) -> Sequence[SourceRefT_co]:
    """Return all discoverable top-level references."""
    ...

Expander

Bases: Protocol[ExpanderRefT_contra, ExpanderRecordT_co, ExpanderChildRawT_co]

Expand one ref into a record plus child refs.

Source code in src/ladon/plugins/protocol.py 
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
@runtime_checkable
class Expander(
    Protocol[
        ExpanderRefT_contra,
        ExpanderRecordT_co,
        ExpanderChildRawT_co,
    ]
):
    """Expand one ref into a record plus child refs."""

    def expand(
        self, ref: ExpanderRefT_contra, client: SyncHttpClientProtocol
    ) -> Expansion[ExpanderRecordT_co, ExpanderChildRawT_co]:
        """Fetch ref, return its record and the child refs to process next.

        Raises:
            ExpansionNotReadyError: ref is not yet ready to be expanded.
            PartialExpansionError: child list is incomplete.
            ChildListUnavailableError: child list could not be retrieved.
        """
        ...

expand(ref, client)

Fetch ref, return its record and the child refs to process next.

Raises:

Type Description
ExpansionNotReadyError

ref is not yet ready to be expanded.
PartialExpansionError

child list is incomplete.
ChildListUnavailableError

child list could not be retrieved.
Source code in src/ladon/plugins/protocol.py 
74
75
76
77
78
79
80
81
82
83
84
def expand(
    self, ref: ExpanderRefT_contra, client: SyncHttpClientProtocol
) -> Expansion[ExpanderRecordT_co, ExpanderChildRawT_co]:
    """Fetch ref, return its record and the child refs to process next.

    Raises:
        ExpansionNotReadyError: ref is not yet ready to be expanded.
        PartialExpansionError: child list is incomplete.
        ChildListUnavailableError: child list could not be retrieved.
    """
    ...

Sink

Bases: Protocol[SinkRefT_contra, SinkRecordT_co]

Consume a leaf ref and return its final record.

Source code in src/ladon/plugins/protocol.py 
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
@runtime_checkable
class Sink(Protocol[SinkRefT_contra, SinkRecordT_co]):
    """Consume a leaf ref and return its final record."""

    def consume(
        self, ref: SinkRefT_contra, client: SyncHttpClientProtocol
    ) -> SinkRecordT_co:
        """Fetch and parse one leaf ref, returning a complete record.

        Context for the leaf (e.g. parent data) flows through
        ``ref.raw`` — no parent-record parameter is needed here.

        Raises:
            LeafUnavailableError: ref could not be fetched or parsed.
        """
        ...

consume(ref, client)

Fetch and parse one leaf ref, returning a complete record.

Context for the leaf (e.g. parent data) flows through ref.raw — no parent-record parameter is needed here.

Raises:

Type Description
LeafUnavailableError

ref could not be fetched or parsed.
Source code in src/ladon/plugins/protocol.py 
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
def consume(
    self, ref: SinkRefT_contra, client: SyncHttpClientProtocol
) -> SinkRecordT_co:
    """Fetch and parse one leaf ref, returning a complete record.

    Context for the leaf (e.g. parent data) flows through
    ``ref.raw`` — no parent-record parameter is needed here.

    Raises:
        LeafUnavailableError: ref could not be fetched or parsed.
    """
    ...

CrawlPlugin

Bases: Protocol[PluginTopRefT_co, PluginLeafRefT_contra, PluginLeafRecordT_co]

Bundle of all adapters for one crawl domain.

name is a short identifier used in log lines and error messages (e.g. "acme_shop", "widgetco"). source produces top-level refs. expanders is an ordered list of expansion steps (one per tree level above the leaves). sink consumes the leaf refs produced by the last expander.

The heterogeneous chain interior deliberately uses Any because a single homogeneous sequence cannot express per-stage types; see ADR-015.

CLI convention

When loaded via ladon run --plugin module:Class, the CLI instantiates the plugin as plugin_cls(client=client). Adapters intended for CLI use must accept client as a keyword argument in __init__. This constraint is not enforced by the Protocol check — it is a CLI convention only and not part of this protocol.

Source code in src/ladon/plugins/protocol.py 
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
@runtime_checkable
class CrawlPlugin(
    Protocol[
        PluginTopRefT_co,
        PluginLeafRefT_contra,
        PluginLeafRecordT_co,
    ]
):
    """Bundle of all adapters for one crawl domain.

    ``name`` is a short identifier used in log lines and error messages
    (e.g. ``"acme_shop"``, ``"widgetco"``). ``source`` produces
    top-level refs. ``expanders`` is an ordered list of expansion steps
    (one per tree level above the leaves). ``sink`` consumes the leaf
    refs produced by the last expander.

    The heterogeneous chain interior deliberately uses ``Any`` because a
    single homogeneous sequence cannot express per-stage types; see ADR-015.

    CLI convention
    --------------
    When loaded via ``ladon run --plugin module:Class``, the CLI
    instantiates the plugin as ``plugin_cls(client=client)``.  Adapters
    intended for CLI use **must** accept ``client`` as a keyword argument
    in ``__init__``.  This constraint is not enforced by the Protocol
    check — it is a CLI convention only and not part of this protocol.
    """

    @property
    def name(self) -> str: ...

    @property
    def source(self) -> Source[PluginTopRefT_co]: ...

    @property
    def expanders(self) -> Sequence[Expander[Any, Any, Any]]: ...

    @property
    def sink(self) -> Sink[PluginLeafRefT_contra, PluginLeafRecordT_co]: ...

Async protocols

The async protocols mirror the sync ones exactly but use async def methods and accept AsyncHttpClientProtocol instead of SyncHttpClientProtocol.

typing.Protocol definitions for async Ladon crawl plugins.

Async adapters implement these protocols by structural subtyping — no inheritance from this module is required.

All async adapters receive an object satisfying AsyncHttpClientProtocol: either the native AsyncHttpClient or curl-cffi AsyncCurlHttpClient implementation. They must not construct their own HTTP sessions or import httpx directly.

The three-layer pipeline is:

AsyncSource  →  [AsyncExpander, ...]  →  AsyncSink

AsyncSource[RefT] discovers top-level refs. Each AsyncExpander[RefT, RecordT, ChildRawT] awaits a ref and returns an Expansion. AsyncSink[RefT, RecordT] awaits a leaf ref and returns a final record. AsyncCrawlPlugin[TopRefT, LeafRefT, LeafRecordT] bundles all three.

AsyncSource

Bases: Protocol[AsyncSourceRefT_co]

Discover top-level refs from an external source, asynchronously.

Source code in src/ladon/plugins/async_protocol.py 
61
62
63
64
65
66
67
68
69
@runtime_checkable
class AsyncSource(Protocol[AsyncSourceRefT_co]):
    """Discover top-level refs from an external source, asynchronously."""

    async def discover(
        self, client: AsyncHttpClientProtocol
    ) -> Sequence[AsyncSourceRefT_co]:
        """Return all discoverable top-level references."""
        ...

discover(client) async

Return all discoverable top-level references.

Source code in src/ladon/plugins/async_protocol.py 
65
66
67
68
69
async def discover(
    self, client: AsyncHttpClientProtocol
) -> Sequence[AsyncSourceRefT_co]:
    """Return all discoverable top-level references."""
    ...

AsyncExpander

Bases: Protocol[AsyncExpanderRefT_contra, AsyncExpanderRecordT_co, AsyncExpanderChildRawT_co]

Expand one ref into a record plus child refs, asynchronously.

Source code in src/ladon/plugins/async_protocol.py 
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
@runtime_checkable
class AsyncExpander(
    Protocol[
        AsyncExpanderRefT_contra,
        AsyncExpanderRecordT_co,
        AsyncExpanderChildRawT_co,
    ]
):
    """Expand one ref into a record plus child refs, asynchronously."""

    async def expand(
        self,
        ref: AsyncExpanderRefT_contra,
        client: AsyncHttpClientProtocol,
    ) -> Expansion[AsyncExpanderRecordT_co, AsyncExpanderChildRawT_co]:
        """Fetch ref, return its record and the child refs to process next.

        Raises:
            ExpansionNotReadyError: ref is not yet ready to be expanded.
            PartialExpansionError: child list is incomplete.
            ChildListUnavailableError: child list could not be retrieved.
        """
        ...

expand(ref, client) async

Fetch ref, return its record and the child refs to process next.

Raises:

Type Description
ExpansionNotReadyError

ref is not yet ready to be expanded.
PartialExpansionError

child list is incomplete.
ChildListUnavailableError

child list could not be retrieved.
Source code in src/ladon/plugins/async_protocol.py 
82
83
84
85
86
87
88
89
90
91
92
93
94
async def expand(
    self,
    ref: AsyncExpanderRefT_contra,
    client: AsyncHttpClientProtocol,
) -> Expansion[AsyncExpanderRecordT_co, AsyncExpanderChildRawT_co]:
    """Fetch ref, return its record and the child refs to process next.

    Raises:
        ExpansionNotReadyError: ref is not yet ready to be expanded.
        PartialExpansionError: child list is incomplete.
        ChildListUnavailableError: child list could not be retrieved.
    """
    ...

AsyncSink

Bases: Protocol[AsyncSinkRefT_contra, AsyncSinkRecordT_co]

Consume a leaf ref and return its final record, asynchronously.

Source code in src/ladon/plugins/async_protocol.py 
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
@runtime_checkable
class AsyncSink(Protocol[AsyncSinkRefT_contra, AsyncSinkRecordT_co]):
    """Consume a leaf ref and return its final record, asynchronously."""

    async def consume(
        self, ref: AsyncSinkRefT_contra, client: AsyncHttpClientProtocol
    ) -> AsyncSinkRecordT_co:
        """Fetch and parse one leaf ref, returning a complete record.

        Context for the leaf flows through ``ref.raw`` — no parent-record
        parameter is needed here.

        Raises:
            LeafUnavailableError: ref could not be fetched or parsed.
        """
        ...

consume(ref, client) async

Fetch and parse one leaf ref, returning a complete record.

Context for the leaf flows through ref.raw — no parent-record parameter is needed here.

Raises:

Type Description
LeafUnavailableError

ref could not be fetched or parsed.
Source code in src/ladon/plugins/async_protocol.py 
101
102
103
104
105
106
107
108
109
110
111
112
async def consume(
    self, ref: AsyncSinkRefT_contra, client: AsyncHttpClientProtocol
) -> AsyncSinkRecordT_co:
    """Fetch and parse one leaf ref, returning a complete record.

    Context for the leaf flows through ``ref.raw`` — no parent-record
    parameter is needed here.

    Raises:
        LeafUnavailableError: ref could not be fetched or parsed.
    """
    ...

AsyncCrawlPlugin

Bases: Protocol[AsyncPluginTopRefT_co, AsyncPluginLeafRefT_contra, AsyncPluginLeafRecordT_co]

Bundle of all async adapters for one crawl domain.

name is a short identifier used in log lines and error messages. source discovers top-level refs. expanders is an ordered list of async expansion steps. sink consumes the leaf refs produced by the last expander.

The heterogeneous chain interior deliberately uses Any because a single homogeneous sequence cannot express per-stage types; see ADR-015.

Source code in src/ladon/plugins/async_protocol.py 
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
@runtime_checkable
class AsyncCrawlPlugin(
    Protocol[
        AsyncPluginTopRefT_co,
        AsyncPluginLeafRefT_contra,
        AsyncPluginLeafRecordT_co,
    ]
):
    """Bundle of all async adapters for one crawl domain.

    ``name`` is a short identifier used in log lines and error messages.
    ``source`` discovers top-level refs. ``expanders`` is an ordered list
    of async expansion steps. ``sink`` consumes the leaf refs produced by
    the last expander.

    The heterogeneous chain interior deliberately uses ``Any`` because a
    single homogeneous sequence cannot express per-stage types; see ADR-015.
    """

    @property
    def name(self) -> str: ...

    @property
    def source(self) -> AsyncSource[AsyncPluginTopRefT_co]: ...

    @property
    def expanders(self) -> Sequence[AsyncExpander[Any, Any, Any]]: ...

    @property
    def sink(
        self,
    ) -> AsyncSink[AsyncPluginLeafRefT_contra, AsyncPluginLeafRecordT_co]: ...

Data models

Immutable data models for Ladon plugin adapters.

All models are frozen dataclasses. Adapters produce them; the runner consumes them. The raw field on Ref carries house-specific data that does not fit the shared schema.

Ref[RawT] preserves adapter-specific raw context. Expansion[RecordT, ChildRawT] carries the current record and typed child refs to the next stage.

Ref dataclass

Bases: Generic[RawT]

Generic reference to any crawlable resource.

url is the canonical URL of the resource. raw carries any house-specific data discovered alongside the URL (e.g. an ID or code needed by the expander). Omitting raw selects the default Mapping[str, object] specialization and supplies an empty dict.

Source code in src/ladon/plugins/models.py 
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
@dataclass(frozen=True, init=False)
class Ref(Generic[RawT]):
    """Generic reference to any crawlable resource.

    ``url`` is the canonical URL of the resource. ``raw`` carries any
    house-specific data discovered alongside the URL (e.g. an ID or code
    needed by the expander). Omitting ``raw`` selects the default
    ``Mapping[str, object]`` specialization and supplies an empty dict.
    """

    url: str
    # The constructor overloads constrain omission to the default specialization.
    raw: RawT = field(default_factory=_empty_raw)  # type: ignore[assignment]

    @overload
    def __init__(self: Ref[Mapping[str, object]], url: str) -> None: ...

    @overload
    def __init__(self, url: str, raw: RawT) -> None: ...

    def __init__(self, url: str, raw: object = _MISSING_RAW) -> None:
        """Initialize a reference while preserving frozen-dataclass semantics."""
        object.__setattr__(self, "url", url)
        object.__setattr__(
            self, "raw", _empty_raw() if raw is _MISSING_RAW else raw
        )

Expansion dataclass

Bases: Generic[RecordT_co, ChildRawT_co]

Result of an Expander.expand() call.

Carries the record for the expanded node plus the child refs to be processed next (either expanded further or consumed by a Sink).

Source code in src/ladon/plugins/models.py 
59
60
61
62
63
64
65
66
67
68
@dataclass(frozen=True)
class Expansion(Generic[RecordT_co, ChildRawT_co]):
    """Result of an Expander.expand() call.

    Carries the record for the expanded node plus the child refs to be
    processed next (either expanded further or consumed by a Sink).
    """

    record: RecordT_co
    child_refs: Sequence[Ref[ChildRawT_co]]

Errors

Error taxonomy for Ladon house plugins.

Each exception maps to a specific runner behaviour. Expansion signals remain typed because they describe tree completeness; non-fatal Phase-3 exceptions are recorded independently so one bad leaf does not discard the rest of a run.

PluginError

Bases: Exception

Base class for all plugin-level errors.

Source code in src/ladon/plugins/errors.py 
 9
10
class PluginError(Exception):
    """Base class for all plugin-level errors."""

ExpansionNotReadyError

Bases: PluginError

The ref is not yet ready to be expanded (e.g. content not live).

The runner should skip this ref without writing to DB or disk. Do not retry during the same run; the ref will be discovered again on the next scheduled run.

Source code in src/ladon/plugins/errors.py 
13
14
15
16
17
18
19
class ExpansionNotReadyError(PluginError):
    """The ref is not yet ready to be expanded (e.g. content not live).

    The runner should skip this ref without writing to DB or disk.
    Do not retry during the same run; the ref will be discovered again
    on the next scheduled run.
    """

PartialExpansionError

Bases: PluginError

The child list was fetched but is incomplete (e.g. a paginated response returned fewer items than the declared total).

Runner behaviour: non-fatal for non-first expanders — the affected branch is isolated and recorded in RunResult.errors. Propagates unchanged from the first expander. Always fatal from a Sink because Phase 3 has no branch to isolate.

Raise this instead of ChildListUnavailableError when the HTTP response was valid but the payload signals an incomplete result. Raise ChildListUnavailableError when the response could not be parsed or the request itself failed.

Source code in src/ladon/plugins/errors.py 
22
23
24
25
26
27
28
29
30
31
32
33
34
35
class PartialExpansionError(PluginError):
    """The child list was fetched but is incomplete (e.g. a paginated
    response returned fewer items than the declared total).

    Runner behaviour: non-fatal for non-first expanders — the affected
    branch is isolated and recorded in ``RunResult.errors``. Propagates
    unchanged from the first expander. Always fatal from a Sink because
    Phase 3 has no branch to isolate.

    Raise this instead of ``ChildListUnavailableError`` when the HTTP
    response was valid but the payload signals an incomplete result.
    Raise ``ChildListUnavailableError`` when the response could not be
    parsed or the request itself failed.
    """

ChildListUnavailableError

Bases: PluginError

The child list could not be retrieved.

Fatal for this ref's run. Raised when the network request succeeded but the response cannot be parsed into a usable child list. Always fatal from a Sink because Phase 3 has no branch to isolate.

Source code in src/ladon/plugins/errors.py 
38
39
40
41
42
43
44
class ChildListUnavailableError(PluginError):
    """The child list could not be retrieved.

    Fatal for this ref's run. Raised when the network request succeeded
    but the response cannot be parsed into a usable child list.
    Always fatal from a Sink because Phase 3 has no branch to isolate.
    """

LeafUnavailableError

Bases: PluginError

A single leaf ref could not be fetched or parsed.

Non-fatal. The runner logs the failure, increments leaves_failed, and continues to the next leaf.

Source code in src/ladon/plugins/errors.py 
47
48
49
50
51
52
class LeafUnavailableError(PluginError):
    """A single leaf ref could not be fetched or parsed.

    Non-fatal. The runner logs the failure, increments leaves_failed,
    and continues to the next leaf.
    """

AssetDownloadError

Bases: PluginError

An asset download failed.

Not recovered from by the runner — propagates as a fatal error that aborts the run. Plugins requiring non-fatal handling must catch this exception internally before returning from the Sink or Expander.

Source code in src/ladon/plugins/errors.py 
65
66
67
68
69
70
71
class AssetDownloadError(PluginError):
    """An asset download failed.

    **Not recovered from by the runner** — propagates as a fatal error that
    aborts the run. Plugins requiring non-fatal handling must catch this
    exception internally before returning from the Sink or Expander.
    """