Catalog requests get slow with one contract definition per asset (30k assets → seconds per request)

[agala-stratio]

Hi all,

we run EDC as the provider connector of a data marketplace, and with ~30k published datasets the catalog became the
slowest part of the connector by far. We found two hot spots in control-plane-catalog (unchanged in v1.0.0-RC4),
fixed them in an extension on our side, and would like to propose the fixes upstream. Before opening issues/PRs we'd
like to know whether the approach fits.

Our setup

Each data product becomes one asset, one policy definition and one contract definition. The contract
definition's assetsSelector is https://w3id.org/edc/v0.0.1/ns/id = <assetId>, and the policy (used as access and
contract policy) carries per-product terms: an availability window (dateTime gteq / lteq) and, for restricted
products, a tenant constraint.

Why not share contract definitions? The terms are different for each product, so the contract policy is
different, and with it the contract definition. Moving the dates out of the policy would remove them from the
agreement. As far as we can see this is a normal DSP shape (each dataset with its own offers), and #4971 looks like
the same cost with "a few thousand assets".

What we measured

One control plane on PostgreSQL, 30,000 assets / policy definitions / contract definitions, all accessible, DSP
2025-1 catalog requests. Warm timings, first request after boot in brackets:

Request	EDC 0.18.1	(a) only	(a) + (b) + cache
Catalog, limit=1	4.4 – 4.8 s (6.2 s)	0.75 – 1.5 s (2.3 s)	0.11 – 0.18 s (1.8 s)
Catalog, limit=50	8.7 – 8.9 s	4.6 – 5.4 s	0.23 – 0.46 s
Catalog, limit=50, offset=15000	> 300 s (timed out)	> 300 s (timed out)	1.7 s
Catalog, limit=1000	96 s	81 s	2.1 s
Single dataset	4.5 – 5.0 s	0.66 – 0.69 s	0.07 – 0.09 s

All three return the same JSON (same datasets, offers and order) apart from the random UUIDs in the catalog @id and
in the offer ids. We haven't measured (a) + (b) without the cache.

Where the time goes

1. One PolicyDefinitionStore.findById per contract definition, on every request, in
   ContractDefinitionResolverImpl.resolveFor.
   computeIfAbsent deduplicates shared policies, but with distinct ones it is 30k queries per catalog or dataset
   request, even for limit=1.
2. Every definition's assetsSelector is tested against every asset in
   DatasetResolverImpl.toDataset,
   and pagination happens after that
   (L84-L88),
   so a page costs assets scanned × definitions predicates: 1.5M for the first page of 50, ~450M for the page at
   offset 15000.

The proposal

• (a) Load the access policies in bulk in ContractDefinitionResolverImpl: one PolicyDefinitionStore.findAll
  with id in [...] (chunked) instead of one findById per definition. No caching and no behaviour change; policies
  are still evaluated on every request.
• (b) Index id selectors in DatasetResolverImpl: definitions whose selector is only id = X / id in [...]
  are looked up by asset id, everything else is tested per asset as today. Matches are sorted by their original
  position, so the output doesn't change. This is what makes large and deep pages usable.
• (c) Open question: caching. Our extension also caches definitions and policies per participant context, with
  a TTL and invalidation on ContractDefinitionEvent / PolicyDefinitionEvent. It brings requests under 100 ms, but
  with several control-plane replicas a change made on another replica only shows after the TTL (negotiation always
  reads the stores, so only the catalog is affected). Would something like this be welcome as an opt-in, or is it
  better kept out of core?

We'd be happy to contribute (a) and (b) as PRs against main, with tests. Feedback, or pointers if this is already
being worked on, are very welcome.

Thanks!

[paullatzelsperger]

General comments

declaring one Asset, one PolicyDefinition and one ContractDefinition for 30k data assets is an anti-pattern. no wonder it gets sluggish. no level of optimization will completely eradicate that.

As you already stated, policies are loaded from the database, and a findAll would materialize them all in memory, which is not a viable option. Similarly, doing a id in... would still materialize a potentially very large amount of policies in memory, which may not be desirable either. Also, with the way we implemented streaming from the DB it may not even be possible, although i'm not 100% on that.

Indexing the ID selectors: if you are referring to the SQL schema, then this is a job for you as adopter. we don't (nor will we ever) publish a production-grade SQL schema, only the one we use for testing. You have to activate the DDL for it explicitly, at which point you are on your own. The reasoning is, that publishing the schema would require us to also publish migration scripts etc., and that gets complicated quickly, so we chose not to do it.

Caching: there is no general approach to caching that will satisfy everyone, nor will everyone need it. So if you already have something, best keep it in your code base.

Recommendation

Instead of optimizing the code for what I think is an anti-pattern (very high amounts of assets), i would strongly urge you to reconsider you data architecture: you said you have lots of products, each with a different timing policy attached to it. I don't know exactly what the policy is, but lets say it's something like a "must only use in this time window", then this policy is un-enforcible anyways (how do you prove not to do something). If it's more like a "must have done X within time frame Y" then its a different story, but needing this different for every one of the 30k assets still sounds fishy to me.

In many use cases similar to yours, we have suggested creating just one EDC asset - the retrieval API for all data products, with query parameters. Assuming any claims like "I did X in timeframe Y" is present as verifiable credential, it is checked and verified upon contract negotiation. Any client how has successfully negotiated a contract has already proven its claims, so the provider can just trust them. So later, on the API call, you can take that claim at face value. The decision, whether a user is able to retrieve an item is then made by the data plane, e.g. based on an authorization token that contains those claims.

take this suggestion with a grain of salt, since i don't know your exact use case

[agala-stratio]

Thanks Paul, that was the right call. The dates were really a publication window, not a usage term: our side already only publishes a product inside it and removes it when it expires, so we took them out of the policy. Products now share one policy and one contract definition per access rule (one for public products, one per tenant for restricted ones), selected by a private asset property.

That's also why we had put the dates in the policy: we wanted the connector itself to enforce the window. A function of our own for odrl:dateTime made the provider refuse a negotiation outside it. But we had only bound it to the negotiation scope, so once an agreement existed, nothing checked the dates on transfer. So it only ever controlled when a contract could be concluded, never the use of the data, which is your point. We also looked at inForceDate, which is checked on transfer, but with a fixed end date per product it would have brought back one policy per asset.

We didn't go for the single "retrieval API" asset, though: our consumers discover and contract each data product individually through the DSP catalog, so we need one asset per product.

Same 30k assets, stock EDC resolvers, no caching:

Request	before (30k definitions)	now (1 definition)
Catalog, limit=1	4.4 – 4.8 s	0.03 s
Catalog, limit=50	8.7 – 8.9 s	0.15 s
Catalog, limit=50, offset=15000	> 300 s	1.7 s
Single dataset	4.5 – 5.0 s	0.02 s

So we're dropping the proposal, and our custom resolver and cache with it. Just to clear up (b) for the record: we didn't mean the SQL schema. It was an in-memory map, built on each request inside DatasetResolverImpl from the contract definitions already resolved, from asset id to the definitions whose selector is only id = X / id in [...]. Each asset would look up its definitions there instead of testing every selector. Nothing was persisted, and the database was not touched. With one definition per access rule it's moot anyway.

In case it helps someone else with the same setup: a selector on privateProperties.'<iri>' works both when the catalog is built and when the negotiation is validated (SQL asset index), so it's a handy way to group assets into a shared contract definition without exposing the grouping key in the catalog.

A side note, not a proposal, just so you know what we're doing on our side: once the definitions were few, the next cost on deep pages in our Kubernetes environment (PostgreSQL behind pgbouncer) was the distributions. DatasetResolverImpl resolves them for every asset it scans, before paginating, and the flow controller queries the data plane registry each time (DataPlaneSignalingFlowController.transferTypes in v1.0.0-RC4, selectorClient.getAll() in 0.18.1, which is what we run). So the last page of 30k assets meant ~30k registry queries. We wrap the default DistributionResolver in our own extension. It keeps the result for 10 s, keyed by what the result depends on: protocol, participant context, source type, response channel type and data plane profiles. Catalog assets are not cached, and transfers still select their data plane without it. That page went from 27 s to 1.4 s, with the same output.

Thanks again for the quick and clear answer!