Repository navigation
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR changes
Two changes to how the catalog resolves datasets in
DatasetResolverImpl. The returned catalog stays the same.ContractDefinitionMatcher, built once per catalog request.id = <asset id>orid in [<asset ids>]criterion bythose ids. Both
idand the deprecatedhttps://w3id.org/edc/v0.0.1/ns/idoperand are covered.over the asset id in the property lookup.
applies
offset/limitbefore building the dataset. Distributions and offer policies are no longer built forassets that
skipthrows away.Why it does that
Before this change, every catalog request tested each asset against every contract definition. It did so for all the
assets before the requested page as well, and built a full dataset for each of them before
skip/limit. With onecontract definition per asset, the usual setup when access is granted per asset, the request did
assets × contract definitionsselector checks, and the response time grew linearly with theoffset.Measured on a local deployment with 10k assets and 10k contract definitions per participant (one definition per
asset,
id =selector), one page of 50 datasets, medians of 3 runs:Every page now costs about the same as the first. Reading the whole catalog page by page goes from about 30 minutes
to about 8 minutes on the unconstrained provider.
What remains is the cost every request pays before any paging: loading and evaluating all contract definitions and
their access policies in
ContractDefinitionResolverImpl. That is left for follow-ups.Further notes
ContractDefinitionMatcheris package-private.DistributionResolveris no longer called for assets before the requested offset.endpointURLis only logged when that asset is on the returned page.even when no asset is streamed. Before, it failed when the first asset was tested.
several criteria, and
not in.findByIdper contract definition;Who will sponsor this feature?
Please @-mention the committer that will sponsor your feature.
Linked Issue(s)
Closes # <-- insert Issue number if one exists
Please be sure to take a look at the contributing guidelines and our etiquette for pull requests.