Skip to content

fix: [stable-8.0][OCISDEV-1685] remove stale group shares in jsoncs3 - #813

Open
2403905 wants to merge 22 commits into
stable-8.0from
fix/stable-8.0-OCISDEV-1685
Open

2403905 wants to merge 22 commits into
stable-8.0from
fix/stable-8.0-OCISDEV-1685

Conversation

@2403905

@2403905 2403905 commented Nov 12, 2026 •

Copy link
Copy Markdown

Description

The jsoncs3 share manager uploaded received.json again on each listing of the received shares when a user had a stale group share.

A stale group share is a share ID in the group cache that has no share in the provider cache.
ListReceivedShares found the stale ID and removed it from received.json.
The ID is not in that file, so the removal changed nothing, but the file was uploaded anyway.
The group cache kept the ID, so the next listing did the same again.
In production, this causes a large volume of share not found. cleaning up warnings and one full document upload per stale share for each listing.

Stale group shares come from partial write failures.
Share and removeShare wrote the provider cache, created.json and the grantee entry at the same time, with no rollback.
A failed write also cancelled the shared context, so the group cache update sometimes did not run.
purgeSpace ignored its errors, and it logged them to a disabled logger.

A loop that removed many shares, for example in a purge or in a listing that finds expired shares, uploaded each touched document once for each share.

sharecache.Cache.Remove panicked when the user or group had no entry for the space of the share, or when the share ID had no delimiter. The panic ran in an errgroup goroutine and stopped the process. Production saw it in CreatedCache.Remove during the cleanup of orphaned shares, and the same path runs in each unshare.

Changes

Read side:

  • receivedsharecache.Cache.Remove and sharecache.Cache.Remove do not upload when the share is not in the cache. When the share is not in memory, they sync first and check again.
  • sharecache.Cache.Remove does not panic when the user or group has no entry for the space of the share, or when the share ID has no delimiter.
  • sharecache RemoveMany deletes the space entry when it removes the last share of the space. Before, each listing of a group member read the space file of a purged space again.
  • providercache.Cache.Remove does not upload when the share is not in the space, and it checks the result of Spaces.Load.
  • ListReceivedShares records which groups give each share ID. When a share is missing, it removes the ID from each group that gave the ID. It sends to the user received states only the IDs that are in received.json of the user.
  • The cleanup runs only when the space file was read. When the download of the space file gets NotFound, the listing runs a Stat. Only a second NotFound proves that the file is missing. Otherwise, the share manager keeps all entries.
  • The cleanup reads each share from the same ListSpace snapshot. Before, a parallel listing could reset the live cache between two reads, and then the cleanup removed valid shares.
  • The cleanup removes the stale IDs of a space with one upload for each user or group document.

Grace period for the stale cleanup:

  • An older version writes the grantee entry and the share at the same time. During a rolling upgrade, a listing on a new pod can find the grantee entry before the share is on storage.
  • A listing removes a stale entry only after StaleShareGracePeriod. The default is 5 minutes. Zero removes the entry at the first listing. The rule applies to user entries and to group entries.
  • Each pod counts the grace period itself. A pod forgets a stale entry only when no listing found it for 1 hour.
  • A pod looks for the entries to forget at most once a minute. Thus a listing that finds many new stale shares does not hold the lock for a long time.
  • A listing that finds the share again stops the grace period.

Write side:

  • Share writes the provider cache and created.json first. It adds the grantee entry only after both writes succeed.
  • When a write fails, Share rolls back. It removes the grantee entry first, and then the share and the created.json entry at the same time. Thus a new Share does not fail with AlreadyExists, and no stale grantee entry stays.
  • The rollback uses a context without the cancellation of the request. Each removal has its own timeout of 30 seconds. A failed Share waits for at most two timeouts.
  • removeShare removes the grantee entry first. It removes the share also when that fails, because the gateway removes the storage grant in any case. The next listing of a grantee removes a stale grantee entry.
  • removeShares returns an error only when it cannot remove the shares.
  • The writes run with no shared context, so a failed write cannot cancel another write.

Failed uploads:

  • Add and RemoveMany in all caches change memory before the upload. Before, a failed upload that is not a CAS conflict kept the changed memory. The etag did not change, so a sync got NotModified and kept that state.
  • Now dropOnFailedUpload drops the cached document after such a failure, so the next call reads the file again.
  • Before this fix, a retry of a failed removal found nothing to remove and reported success. A failed add showed a share or a share state that was not in the file.

Space delete:

  • ProcessEvents puts its logger into the context. purgeSpace returns all errors with the share ID, and ProcessEvents logs them.
  • In a purge, removeShare removes the grantee entry and the created.json entry at the same time. A failure of one does not stop the other.
  • The logger in the ProcessEvents context writes only warnings and errors, because it does not use the configured log level.

Batch removal:

  • The three caches get RemoveMany. It removes many IDs with one upload, and does no upload when no ID is present. Remove calls RemoveMany with one ID.
  • removeShares groups the shares for each document across spaces and uploads each document once. purgeSpace calls it once for all shares of the space.
  • removeShares runs at most MaxConcurrency removals in parallel. When MaxConcurrency is not set, the limit is 10.
  • listSharesByIDs, listCreatedShares and ListReceivedShares collect the expired shares of all spaces. They remove them together after the listing workers stop. One helper, expireShares, replaces the five copies of the expiry code.
Case Uploads before Uploads after
Purge of a space with 20 group shares 41 3
One listing that finds 20 expired shares 60 3
Listings that find 40 stale group IDs in 2 groups 40 on each listing 0 in the grace period, then 2 one time, then 0

The numbers come from the same test on stable-8.0 and on this branch. The 20 expired shares give the same numbers for ListShares, ListShares with a resource filter and ListReceivedShares.

ShareExpired event:

  • providercache RemoveMany returns the IDs that its upload removed. The CAS upload makes sure that only one caller removes a share, also across pods.
  • expireShares publishes ShareExpired only for these IDs. The event always has the ShareID.
  • In a test with 20 parallel listings of one expired share, stable-8.0 sent 1–3 events. This branch sends exactly 1 event. A failed removal sends no event.

Logs:

  • Each log line of a share listing has the fields of one share only. Before, the listing added the fields of each share to the same logger, and a log search found the wrong share.

Tests:

  • New tests in pkg/share/manager/jsoncs3/jsoncs3_test.go, receivedsharecache/receivedsharecache_test.go, sharecache/sharecache_test.go, providercache/providercache_test.go, purgespace_internal_test.go, batchremove_internal_test.go, rollback_internal_test.go, stalegrace_internal_test.go, expire_internal_test.go and listsharesbyids_internal_test.go.
  • The regression tests for the sharecache.Cache.Remove panic fail with a panic on the sharecache.go of stable-8.0.
  • New test helpers in pkg/share/manager/jsoncs3/testhelpers/fakes.go: CountingUploadStorage, AlwaysAbortedUploadStorage and NotModifiedStorage. NotModifiedStorage returns NotModified for a matching etag, like the CS3 metadata storage. metadata.Disk ignores IfNoneMatch.
  • Three existing receivedsharecache tests are adapted: a Remove of an unknown share does not upload now, so the bootstrap test uses Add, and the two resync tests warm the cache first.

Changelog: changelog/unreleased/fix-jsoncs3-stale-group-shares.md.

Checks

  • All pkg/share/manager/jsoncs3/... packages pass with -race. go vet and gofmt report nothing.
  • go build ./... passes. ./pkg/share/..., the share provider services and the gateway pass.
  • The ocis sharing, storage-users and ocis packages build against this branch.

Notes for the reviewers

  • Share and Unshare now do two write steps one after the other. This adds one storage round trip of latency.
  • Because removeShare now removes the grantee entry before the share, an expired share stays visible a little longer to a parallel listing.
  • A stale entry stays visible for up to StaleShareGracePeriod on each pod. Each pod counts the grace period itself, so the cleanup can need one more listing on each pod.
  • Warnings and errors from purgeSpace and the caches below it are now visible. Before, they went to a disabled logger.
  • A space delete event with an ID that is not <storageid>$<spaceid> purges nothing. That is a different bug and is not part of this pull request.

🤖 Generated with Claude Code

A stale group share is a share ID in the group cache that has no share in
the provider cache. ListReceivedShares found the stale ID on each listing.
It removed the ID from received.json, but the ID is not in that file. So
each listing uploaded received.json again with the same content, and the
stale ID stayed forever.

Read side:
- receivedsharecache and sharecache: Remove does not upload when the share
  is not in the cache. When the share is not in memory, Remove syncs first
  and checks again.
- sharecache: Remove does not panic when the user or group has no entry for
  the space of the share.
- ListReceivedShares records which groups give each share ID. Before the
  cleanup, it reads the share again from storage. When the share is still
  missing, it removes the ID from the user received states and from each
  group that gave the ID.

Write side:
- Share writes the provider cache and created.json first. It adds the
  grantee entry only after both writes succeed.
- removeShare removes the grantee entry first. When that fails, Unshare
  returns the error and the share stays, so the caller can try again.
- The grantee step runs alone, so a failed sibling write cannot cancel it.

Space delete:
- ProcessEvents puts its logger into the context. Before, purgeSpace logged
  to a disabled logger.
- purgeSpace returns all errors with the share ID, and ProcessEvents logs
  them.
- In a purge, removeShare continues after a grantee error and removes the
  created.json entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kw-security

kw-security commented Nov 12, 2026 •

Copy link
Copy Markdown

✅ Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
✅ Open Source Security 0 0 0 0 0 issues
✅ Licenses 0 0 0 0 0 issues
✅ Code Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@2403905
2403905 requested a review from a team November 12, 2026 18:37
2403905 and others added 4 commits November 12, 2026 19:47
… share fix

- ListReceivedShares cleans up stale entries only when it read the space
  file. A NotFound for the space file can be temporary. Before, the cleanup
  then removed valid group shares for all members of the group.
- Remove the second provider read before the cleanup. It used the same sync
  path, so it gave no protection.
- Unshare removes the share also when the grantee entry cannot be removed.
  The gateway removes the storage grant in this case too. Before, a broken
  grantee file blocked the unshare and left a share without a grant.
- Share removes the entries that it wrote when a later write fails. Before,
  the share stayed, and a new Share failed with AlreadyExists.
- The writes in Share and removeShare do not cancel each other.
- In a purge, the grantee removal and the created.json removal run at the
  same time again.
- The logger in the ProcessEvents context writes only warnings and errors,
  because it does not use the configured log level.
- Fix the Unshare test. It checked nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Send a SpaceDeleted event to ProcessEvents, as in production, and check the
result of the purge goroutine.

- A normal delete removes the group shares. The listings after the delete
  do no uploads.
- When the group file cannot be written during the purge, the group entry
  stays. The first listing of a member removes it with one upload. The
  listings after that do no uploads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…of shares

A loop that removes many shares uploaded each touched document once for each
share. Each upload sends the whole document. A purge of 20 group shares did
41 uploads, and a listing that finds 20 expired shares did 60.

These tests fail now:

- providercache, sharecache, receivedsharecache: RemoveMany removes many
  share IDs with one upload, and does no upload when no ID is present.
- purgeSpace, listCreatedShares, listSharesByIDs and ListReceivedShares
  upload each touched document once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r document

A loop that removed many shares called removeShare for each share. Each
removal uploaded every touched document again, with the full content. A
purge of 20 group shares did 41 uploads, and a listing that found 20
expired shares did 60 uploads. The bytes grew with the square of the share
count, and each upload held the lock of the space or the user.

- providercache, sharecache and receivedsharecache get RemoveMany. It
  removes many IDs with one upload, and does no upload when no ID is
  present. Remove calls RemoveMany with one ID.
- providercache.Cache.Remove now checks the result of Spaces.Load and does
  not upload when the share is not in the space.
- removeShares groups the shares per document and uploads each document
  once. removeShare calls it with one share.
- purgeSpace removes all shares of the space with one removeShares call.
- listSharesByIDs, listCreatedShares and ListReceivedShares collect the
  expired shares of each space and remove them together. One helper,
  expireShares, replaces the five copies of the expiry code. The event now
  always has the ShareID.
- The stale cleanup in ListReceivedShares removes the stale IDs of a space
  with one upload per user or group document.

A purge of 20 group shares now does 3 uploads, and a listing that finds 20
expired shares also does 3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…che Remove panic

sharecache.Cache.Remove read us.UserShares[ssid].IDs with no nil check. When
the user had no entry for the space of the share, the entry was nil and
Remove panicked. The panic ran in an errgroup goroutine, so it stopped the
process. Production saw it in CreatedCache.Remove during the cleanup of
orphaned shares, and the same path runs in each unshare.

The nil guard is already in this pull request. Add the two cases from the
report:

- a share in space-a, then a Remove of a share in space-b, on a warm cache
- a share ID with no delimiter, so the space key is ":"

On the sharecache.go of stable-8.0, both tests and the existing nil test
fail with a panic.

Correct the changelog: the panic was in the created cache too, not only in
the group cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2403905 added a commit to owncloud/ocis that referenced this pull request Nov 16, 2026
…e manager

Bump reva to v2.0.0-20261112224757-ebbb52833e5f, the head of
owncloud/reva#813.

A stale group share is a share ID in the group cache that has no share. On
each listing of the received shares, the jsoncs3 share manager uploaded the
received.json file of the user again with the same content, and the stale ID
stayed.

The new reva:
- removes stale group shares from the group cache, only when it read the
  share file of the space
- does not upload a file when a Remove changes nothing
- writes a share before the grantee entry, and rolls back after a failure
- removes the share in an unshare also when the grantee entry fails
- removes many shares with one upload for each document
- logs the share cleanup errors after a space delete

The bump also brings owncloud/reva#796 into
stable-8.0. The previous reva pin did not have it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Roman Perekhod <2403905@gmail.com>
2403905 and others added 3 commits November 16, 2026 13:44
…he workers stop

The listing workers in listCreatedShares and ListReceivedShares send the
expired shares to a channel. The main goroutine reads this channel and the
results channel in one select loop. After all workers stop, the manager
calls expireShares one time with the parent context.

listSharesByIDs collects the expired shares of all spaces and calls
expireShares one time after the loop.

removeShares groups the shares by document across spaces. Thus expired
shares of one creator in different spaces now cause one upload of
created.json.

The errgroup context is now gctx. errgroup cancels this context when
Wait returns, so the expiration uses the parent context.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and the stale cleanup

The stale cleanup in ListReceivedShares took the etag from the ListSpace
snapshot, but it read each share from the live provider cache. A parallel
listing that got a temporary NotFound reset the live cache between the two
reads. Then all shares looked stale, and the cleanup removed valid shares
from the group cache. The cleanup now reads each share from the snapshot.

The rollback of a failed Share used the request context. A cancelled
request is a common cause of the failure, and then each rollback write
failed at once. The share stayed, and a new Share failed with
AlreadyExists. The rollback now uses a context without the cancellation,
with a timeout.

A failed grantee write keeps the share ID in memory. The rollback did not
remove it, so the next write of the grantee document uploaded the ID
without a share behind it. The rollback now removes the grantee entry
first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…share cleanup

An older version writes the grantee entry and the share at the same time.
During a rolling upgrade, a listing on a new pod can find the grantee entry
before the share is on storage. The listing removed this entry at once, and
no group member saw the share.

A listing now notes the time when this pod first finds a stale share. It
removes the entry only on a listing after StaleShareGracePeriod. The
default is 5 minutes. Zero removes the entry at the first listing. The
rule applies to user entries and to group entries.

The tests of the cleanup set the grace period to zero. The changelog
describes the fixes of the last commits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2403905 added a commit to owncloud/ocis that referenced this pull request Nov 16, 2026
…stale share fix

Bump reva to v2.0.0-20261116153307-8f53b9a6148f, the head of
owncloud/reva#813. The new reva commits change only the jsoncs3 share
manager:

- The stale cleanup reads each share from the same copy of the space
  file. A parallel listing can no longer make valid shares look stale.
- The cleanup removes a stale entry only after a grace period of 5
  minutes. During a rolling upgrade, an older version can write the
  grantee entry before the share.
- The rollback of a failed share also removes the grantee entry, and it
  runs also when the request was cancelled.
- A listing removes the expired shares of all spaces together.

The changelog describes these fixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2403905 added a commit to owncloud/ocis that referenced this pull request Nov 16, 2026
…e manager

Bump reva to v2.0.0-20261112224757-ebbb52833e5f, the head of
owncloud/reva#813.

A stale group share is a share ID in the group cache that has no share. On
each listing of the received shares, the jsoncs3 share manager uploaded the
received.json file of the user again with the same content, and the stale ID
stayed.

The new reva:
- removes stale group shares from the group cache, only when it read the
  share file of the space
- does not upload a file when a Remove changes nothing
- writes a share before the grantee entry, and rolls back after a failure
- removes the share in an unshare also when the grantee entry fails
- removes many shares with one upload for each document
- logs the share cleanup errors after a space delete

The bump also brings owncloud/reva#796 into
stable-8.0. The previous reva pin did not have it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Roman Perekhod <2403905@gmail.com>
2403905 added a commit to owncloud/ocis that referenced this pull request Nov 16, 2026
…stale share fix

Bump reva to v2.0.0-20261116153307-8f53b9a6148f, the head of
owncloud/reva#813. The new reva commits change only the jsoncs3 share
manager:

- The stale cleanup reads each share from the same copy of the space
  file. A parallel listing can no longer make valid shares look stale.
- The cleanup removes a stale entry only after a grace period of 5
  minutes. During a rolling upgrade, an older version can write the
  grantee entry before the share.
- The rollback of a failed share also removes the grantee entry, and it
  runs also when the request was cancelled.
- A listing removes the expired shares of all spaces together.

The changelog describes these fixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2403905 and others added 8 commits November 16, 2026 22:46
…timeout and the stale cleanup

- removeShares returns an error only when it cannot remove the shares.
  When the shares are gone, a retry gets NotFound. The next listing
  removes a stale grantee entry, and ListShares skips a stale created entry.
- Each rollback removal in Share gets its own timeout. A slow removal
  does not use the time of the next one.
- When the download of the space file gets NotFound, the listing runs a
  Stat. A second NotFound proves that the file is missing, and the stale
  entry is removed after the grace period.
- A listing that finds a share again restarts its grace period.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hare cache

sharecache RemoveMany kept the space entry after it removed the last share
of the space. Each listing of a group member then read the space file of
a purged space again. RemoveMany now deletes the empty space entry, the
same as receivedsharecache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tale group entries

The stale cleanup sent the stale IDs of a group also to the received.json
of the user. These IDs are usually not in that file, so the user cache
downloaded the file again only to find nothing to remove. The cleanup now
sends only the IDs that are in received.json of the user.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
removeShares started one goroutine for each grantee and creator document,
with no limit. A purge of a space with many grantees then sent hundreds of
parallel writes to the metadata storage. Now MaxConcurrency limits the
parallel removals. When MaxConcurrency is not set, the limit is 10.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ch expired share

Each listing that found an expired share published a ShareExpired event,
also when another listing removed the share first or when the removal
failed. Each event can send a notification. providercache RemoveMany now
returns the IDs that its upload removed. The CAS upload makes sure that
only one caller removes a share, also across pods. expireShares publishes
the event only for these shares.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gofmt reported four files of the jsoncs3 share manager. Sort the imports
of jsoncs3.go, split a long closure in providercache.go and remove an
empty line in providercache_test.go. Move a comment in
batchremove_internal_test.go to its own line, so that gofmt does not
align it with the next line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Roman Perekhod <2403905@gmail.com>
…moval upload

RemoveMany in the provider cache, the created and group cache and the
user cache deletes the shares from memory before the upload. When the
upload failed with an error that is not a CAS conflict, the memory state
was different from the file. The etag did not change, so a sync got
NotModified and kept that state. A retry then found nothing to remove,
did not upload and reported success. The share stayed in the file, and
an expired share got no ShareExpired event. In the user cache, this also
occurred inside one RemoveMany after a transient upload error, because
the resync before the next attempt got NotModified.

Now a failed upload that is not a CAS conflict drops the cached document
from memory, so the next call reads the file again. The entries are not
put back: a failed Add leaves an extra entry in memory, and the rollback
of Share relies on Remove to delete it.

The new test helper NotModifiedStorage returns NotModified for a matching
etag, like the CS3 metadata storage. metadata.Disk ignores IfNoneMatch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Roman Perekhod <2403905@gmail.com>
ListReceivedShares added a shareid field to the space logger for each
share that it found, and listSharesByIDs added the storageid, spaceid
and opaqueid fields to the listing logger for each share. A later log
line then had the fields of all earlier shares. Log parsers usually keep
the first key, so a log search found the wrong share. The logger also
got bigger with each share.

ListReceivedShares now keeps the space logger unchanged, because each
log call adds its own shareid. listSharesByIDs now makes a new logger
for each share.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Roman Perekhod <2403905@gmail.com>
2403905 added a commit to owncloud/ocis that referenced this pull request Nov 17, 2026
…e manager

Bump reva to v2.0.0-20261112224757-ebbb52833e5f, the head of
owncloud/reva#813.

A stale group share is a share ID in the group cache that has no share. On
each listing of the received shares, the jsoncs3 share manager uploaded the
received.json file of the user again with the same content, and the stale ID
stayed.

The new reva:
- removes stale group shares from the group cache, only when it read the
  share file of the space
- does not upload a file when a Remove changes nothing
- writes a share before the grantee entry, and rolls back after a failure
- removes the share in an unshare also when the grantee entry fails
- removes many shares with one upload for each document
- logs the share cleanup errors after a space delete

The bump also brings owncloud/reva#796 into
stable-8.0. The previous reva pin did not have it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Roman Perekhod <2403905@gmail.com>
2403905 added a commit to owncloud/ocis that referenced this pull request Nov 17, 2026
…stale share fix

Bump reva to v2.0.0-20261116153307-8f53b9a6148f, the head of
owncloud/reva#813. The new reva commits change only the jsoncs3 share
manager:

- The stale cleanup reads each share from the same copy of the space
  file. A parallel listing can no longer make valid shares look stale.
- The cleanup removes a stale entry only after a grace period of 5
  minutes. During a rolling upgrade, an older version can write the
  grantee entry before the share.
- The rollback of a failed share also removes the grantee entry, and it
  runs also when the request was cancelled.
- A listing removes the expired shares of all spaces together.

The changelog describes these fixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Roman Perekhod <2403905@gmail.com>
2403905 added a commit to owncloud/ocis that referenced this pull request Nov 17, 2026
…g fixes of the stale share cleanup

Bump reva to deffdab6d, the head of owncloud/reva#813. The new commits
change only the jsoncs3 share manager:

- A failed removal upload drops the cached document, so a retry reads
  the file again and removes the share.
- Each log line of a share listing has the fields of one share only.
- gofmt formats the jsoncs3 files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Roman Perekhod <2403905@gmail.com>
2403905 and others added 2 commits November 17, 2026 15:00
…d upload

Add in the provider cache, the created and group cache and the user
cache puts the entry in memory before the upload. When the upload failed
with an error that is not a CAS conflict, the entry stayed in memory. The
etag did not change, so a sync got NotModified and kept it. The pod then
listed a share or a share state that was not in the file, and the next
write for the same document uploaded it. For example, a client got an
error when it accepted a share, but this pod showed the share as
accepted.

Each cache now has dropOnFailedUpload, which RemoveMany and Add use. It
drops the cached document after a failed upload that is not a CAS
conflict, so the next call reads the file again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Roman Perekhod <2403905@gmail.com>
…listing that found it

staleGracePeriodOver dropped each entry that was older than ten grace
periods (50 minutes), counted from the first time that the pod found the
share. Each pod counts the grace period itself. With many pods, one pod
can find the same stale share less often than every 50 minutes. When
other new stale shares arrived in between, the pod dropped the entry and
started the grace period again on each listing, so the stale entry
stayed.

Now each entry holds the first time, for the grace period, and the last
time that a listing found the share. The pod forgets an entry only when
no listing found it for 24 hours.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Roman Perekhod <2403905@gmail.com>
…ntry at the same time

The rollback of a failed Share removed the grantee entry, the share in
the provider cache and the entry in the created cache one after the
other. Each removal has its own timeout of 30 seconds. A slow metadata
storage is a usual cause of the first failure, so a failed Share could
block the request for about 90 seconds.

The removal of the share and the removal of the created entry do not
depend on each other. They now run at the same time, after the removal
of the grantee entry. A failed Share now waits for at most two timeouts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Roman Perekhod <2403905@gmail.com>
2403905 added a commit to owncloud/ocis that referenced this pull request Nov 17, 2026
…eriod and rollback fixes

Bump reva to 6b741a384, the head of owncloud/reva#813. The new commits
change only the jsoncs3 share manager:

- A failed add upload drops the cached document, so the pod does not
  show an entry that is not in the file.
- A pod remembers a stale share for 24 hours after the last listing that
  found it, so the grace period also ends on a pod that finds the share
  only rarely.
- The rollback of a failed share removes the share and the created entry
  at the same time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Roman Perekhod <2403905@gmail.com>
2403905 added a commit to owncloud/ocis that referenced this pull request Nov 17, 2026
…e share memory

Bump reva to a31eb537b, the head of owncloud/reva#813. The new commit
changes only the jsoncs3 share manager:

- A pod forgets a stale share 1 hour after the last listing that found
  it. Before, it was 24 hours.
- A pod looks for the stale shares to forget at most once a minute, so a
  listing that finds many new stale shares does not hold the lock for a
  long time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Roman Perekhod <2403905@gmail.com>
2403905 added a commit to owncloud/ocis that referenced this pull request Nov 17, 2026
…e manager

Bump reva to a31eb537b, the head of owncloud/reva#813.

A stale group share is a share ID in the group cache that has no share. On
each listing of the received shares, the jsoncs3 share manager uploaded the
received.json file of the user again with the same content, and the stale ID
stayed.

The new reva:
- removes stale group shares from the group cache, only when it read the
  space file or a Stat proved that the file is missing
- removes a stale entry only after a grace period of 5 minutes, and forgets
  a stale entry 1 hour after the last listing that found it
- does not upload a file when a Remove changes nothing, and reads the file
  again after a failed Add or Remove upload
- does not panic in sharecache Remove when the space entry is missing
- writes a share before the grantee entry, and rolls back after a failure
- removes the share in an unshare also when the grantee entry fails
- removes many shares with one upload for each document, with at most
  MaxConcurrency removals in parallel
- publishes one ShareExpired event for each expired share
- logs the share cleanup errors after a space delete

The default of SHARING_USER_JSONCS3_MAX_CONCURRENCY changes from 1 to 5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Roman Perekhod <2403905@gmail.com>
…e updates

This commit joins 5 commits.

Stale shares:
- A pod forgot a stale share 24 hours after the last listing that found
  it. It now forgets the share after 1 hour.
- sweepStale ran a full sweep of staleSince for each new stale ID. It
  now runs at most once in each staleForgetAfter/60.
- The listing forgets a stale share only after all removals of it
  succeed. A failed removal no longer starts a new grace period.

Update under the space lock:
- UpdateShare read the share and then released the space lock. A
  parallel Add or RemoveMany with a failed upload could drop the space
  from memory. The client got the changed share, but the file kept the
  old share.
- The new Cache.Update reads the share under the space lock. It changes
  a copy and uploads the space with a CAS retry. It returns NotFound
  when a parallel unshare removed the share. A retry keeps the
  expiration.
- listCreatedShares reads each share from the ListSpace snapshot and
  not from the live cache.
- The listings collect the expired shares in a slice with a mutex.

Cleanup:
- CleanupStaleShares removes the orphaned shares in one removeShares
  call. 20 orphaned shares in one space now do 3 uploads, not 60.
- The loops do not change the cache maps that they read.
- After a Stat error or a gateway selector error, the cleanup logs the
  error. It does not stop with a nil pointer panic.

Parallel removals:
- removeShares used MaxConcurrency as the limit. The ocis default is 1,
  so the removals ran one after the other. The limit is now at least 3.
  A higher MaxConcurrency still works. The default for a value of 0 or
  less is still 10.

Sync of the space:
- RemoveMany and Update synced the space only when the space was not in
  memory. A listing with a temporary NotFound stored an empty space with
  no etag. RemoveMany then found no share, did not upload and reported
  success. The share stayed in the space file.
- RemoveMany and Update now sync first when a share is not in memory.
  With a known etag, the sync gets NotModified.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Roman Perekhod <2403905@gmail.com>
@2403905
2403905 force-pushed the fix/stable-8.0-OCISDEV-1685 branch from 38d3324 to c600f66 Compare November 17, 2026 23:53
deyankiteworks
deyankiteworks previously approved these changes Nov 18, 2026
delete(us.UserShares[ssid].IDs, shareID)
// An empty space entry makes each listing read the space file again.
if len(us.UserShares[ssid].IDs) == 0 {
delete(us.UserShares, ssid)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question about rolling upgrades:
The old version has a bug, that it panics, when sharecache.Remove is called, but the space entry was already deleted. The new version fixes that bug, but now deletes these space entries more often. That means that when both versions run at the same time, these panics are a lot more likely. Do you think thats acceptable for rolling upgrades? Otherwise, we could release this delete call in a subsequent version.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't fix it in the old binary. It is a slim chance of getting a panic during the rolling upgrades. Here it is safe because before it checked if c.has(..

@LarsJurgensen

Copy link
Copy Markdown

Very nice!
I tested the main flows locally with old and new version and everything looks good now:

  • The stale group shares are not created anymore
  • When a group share is stale, it does not trigger a reupload anymore
  • When a group share is stale, the group share is automatically fixed
  • Normal flows are uneffected
  • The panic is resolved

LarsJurgensen
LarsJurgensen previously approved these changes Nov 18, 2026
PurgeSpace stored a space with a nil Shares map in memory.
Add does not sync a space that is in memory.
Thus an Add to the purged space on the same pod wrote to the nil map and panicked.
PurgeSpace now stores an empty map.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@2403905
2403905 dismissed stale reviews from LarsJurgensen and deyankiteworks via 444cdda November 18, 2026 16:09
2403905 added a commit to owncloud/ocis that referenced this pull request Nov 18, 2026
…e manager

Bump reva to a31eb537b, the head of owncloud/reva#813.

A stale group share is a share ID in the group cache that has no share. On
each listing of the received shares, the jsoncs3 share manager uploaded the
received.json file of the user again with the same content, and the stale ID
stayed.

The new reva:
- removes stale group shares from the group cache, only when it read the
  space file or a Stat proved that the file is missing
- removes a stale entry only after a grace period of 5 minutes, and forgets
  a stale entry 1 hour after the last listing that found it
- does not upload a file when a Remove changes nothing, and reads the file
  again after a failed Add or Remove upload
- does not panic in sharecache Remove when the space entry is missing
- writes a share before the grantee entry, and rolls back after a failure
- removes the share in an unshare also when the grantee entry fails
- removes many shares with one upload for each document, with at most
  MaxConcurrency removals in parallel
- publishes one ShareExpired event for each expired share
- logs the share cleanup errors after a space delete

The default of SHARING_USER_JSONCS3_MAX_CONCURRENCY changes from 1 to 5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Roman Perekhod <2403905@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants