Skip to content

feat: index contract definitions by asset id when resolving the catalog - #6163

Draft
wolf4ood wants to merge 1 commit into
eclipse-edc:mainfrom
wolf4ood:feat/improve_catalog_perf
Draft

wolf4ood wants to merge 1 commit into
eclipse-edc:mainfrom
wolf4ood:feat/improve_catalog_perf

Conversation

@wolf4ood

Copy link
Copy Markdown
Contributor

What this PR changes

Two changes to how the catalog resolves datasets in DatasetResolverImpl. The returned catalog stays the same.

  • New ContractDefinitionMatcher, built once per catalog request.
    • It builds each contract definition's assets-selector predicate once, instead of once per asset.
    • It indexes the definitions whose selector is a single id = <asset id> or id in [<asset ids>] criterion by
      those ids. Both id and the deprecated https://w3id.org/edc/v0.0.1/ns/id operand are covered.
    • An asset is tested only against the definitions indexed under its id, plus the ones that could not be indexed.
    • Candidates are still tested with their real predicates, and matches come back in the original definition order.
      over the asset id in the property lookup.
  • Datasets are built only for the requested page. For each asset, the resolver now works out the offers first and
    applies offset/limit before building the dataset. Distributions and offer policies are no longer built for
    assets that skip throws away.

Why it does that

Before this change, every catalog request tested each asset against every contract definition. It did so for all the
assets before the requested page as well, and built a full dataset for each of them before skip/limit. With one
contract definition per asset, the usual setup when access is granted per asset, the request did
assets × contract definitions selector checks, and the response time grew linearly with the offset.

Measured on a local deployment with 10k assets and 10k contract definitions per participant (one definition per
asset, id = selector), one page of 50 datasets, medians of 3 runs:

Provider offset 0 2,500 5,000 9,950
unconstrained policies, before 2.37 s 6.33 s 9.26 s 16.04 s
unconstrained policies, after 2.32 s 2.77 s 2.37 s 2.84 s
Membership access policy per CD, after 7.62 s 7.23 s 7.87 s 7.24 s

Every page now costs about the same as the first. Reading the whole catalog page by page goes from about 30 minutes
to about 8 minutes on the unconstrained provider.

What remains is the cost every request pays before any paging: loading and evaluating all contract definitions and
their access policies in ContractDefinitionResolverImpl. That is left for follow-ups.

Further notes

  • No API or SPI changes. ContractDefinitionMatcher is package-private.
  • Behaviour changes, limited to work whose result was thrown away:
    • DistributionResolver is no longer called for assets before the requested offset.
    • The warning about a catalog asset without endpointURL is only logged when that asset is on the returned page.
  • Selector predicates are now built when the request starts. A selector with an unsupported operator fails the request
    even when no asset is streamed. Before, it failed when the first asset was tested.
  • Other selectors are not indexed and are tested against every asset, as before: property-based ones, selectors with
    several criteria, and not in.
  • Follow-up candidates found with the same benchmark:
    • cache the access policy evaluation per policy id within a request;
    • load the policies in one query instead of one findById per contract definition;
    • profile the policy evaluation, which takes about 0.5 ms each.

Who will sponsor this feature?

Please @-mention the committer that will sponsor your feature.

Linked Issue(s)

Closes # <-- insert Issue number if one exists

Please be sure to take a look at the contributing guidelines and our etiquette for pull requests.

@wolf4ood wolf4ood added the enhancement New feature or request label Oct 26, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant