Skip to content

fix: [stable-8.0.10] fix a panic when a grpc server stops before it serves - #815

Open
2403905 wants to merge 1 commit into
stable-8.0.10-fixfrom
fix/stable-8.0.10-rgrpc-stop-before-start
Open

2403905 wants to merge 1 commit into
stable-8.0.10-fixfrom
fix/stable-8.0.10-rgrpc-stop-before-start

Conversation

@2403905

@2403905 2403905 commented Nov 13, 2026

Copy link
Copy Markdown

Description

On school-91003, two sharing pods crashed on 2026-11-12. Each crash was a nil pointer panic in grpc.(*Server).stop(0x0, 0x1) during the shutdown:

panic: runtime error: invalid memory address or nil pointer dereference
google.golang.org/grpc.(*Server).stop(0x0, 0x1)
google.golang.org/grpc.(*Server).GracefulStop(...)
github.com/owncloud/reva/v2/pkg/rgrpc.(*Server).GracefulStop
    pkg/rgrpc/rgrpc.go:306
github.com/owncloud/reva/v2/cmd/revad/runtime.(*server).Stop.func1()
    cmd/revad/runtime/drivenserver.go:106

Cause

  • pkg/rgrpc/rgrpc.go creates the *grpc.Server only at the end of registerServices, after all service constructors return. Start calls registerServices.
  • Some constructors are slow. NewDefault in pkg/share/manager/jsoncs3/jsoncs3.go connects to NATS. Nats in pkg/events/stream/nats.go retries this connection for up to 15 minutes.
  • A SIGTERM during this time causes a call to Stop or GracefulStop. These functions call the nil s.s and panic.
  • The panic replaces the normal shutdown with a crash. It also hides the real reason for the shutdown.
  • Start writes s.s in one goroutine, and Stop reads it in another goroutine. No synchronization protects it, so this is also a data race.

Fix

All changes are in pkg/rgrpc/rgrpc.go.

  • registerServices returns the grpc server and the services. It does not store them.
  • Start stores them under a mutex. When a stop came first, Start closes the services and returns nil. It does not serve.
  • Stop and GracefulStop mark the server as stopped. They call the grpc server only when it exists.
  • cleanupServices closes each service one time only, also when Stop runs after GracefulStop.
  • Start treats grpc.ErrServerStopped as a normal end. Serve returns this error when the stop comes after s.s is set but before Serve starts.

Service constructors still get a nil *grpc.Server, as before. s.s was nil at that point in the old code too.

Tests

pkg/rgrpc/rgrpc_test.go is new. It has 6 tests:

  1. Stop before Start.
  2. GracefulStop before Start.
  3. Start after GracefulStop. Start must return without serving.
  4. GracefulStop while a service constructor blocks. The service must be closed one time.
  5. Stop while a service constructor blocks. The service must be closed one time.
  6. GracefulStop after Start.

On the old pkg/rgrpc/rgrpc.go, tests 1–5 panic with the production error, and test 6 shows a data race. With the fix, all 6 tests pass 10 times with -race. go build ./... passes.

Related

🤖 Generated with Claude Code

…erves

pkg/rgrpc created the grpc server only at the end of registerServices,
after all service constructors returned. The sharing service builds the
jsoncs3 share manager there, and that constructor retries the NATS
connection for up to 15 minutes. A SIGTERM during this time made
Stop and GracefulStop call the nil *grpc.Server and panic. This
happened on school-91003 on 2026-11-12.

- registerServices returns the grpc server and the services. It does
  not store them.
- Start stores them under a mutex. When a stop came first, Start closes
  the services and returns nil. It does not serve.
- Stop and GracefulStop mark the server as stopped. They call the grpc
  server only when it exists.
- cleanupServices closes each service one time only.
- Start treats grpc.ErrServerStopped as a normal end.

The mutex also removes the data race on s.s between Start and Stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kw-security

kw-security commented Nov 13, 2026 •

Copy link
Copy Markdown

✅ Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
✅ Open Source Security 0 0 0 0 0 issues
✅ Licenses 0 0 0 0 0 issues
✅ Code Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

2403905 added a commit to owncloud/ocis that referenced this pull request Nov 13, 2026
…ncloud/reva#814

owncloud/reva#814 is merged as d74730112. Point reva to that commit.
The merge does not contain the rgrpc stop-before-start fix. That fix
is in owncloud/reva#815. Thus this commit also removes the vendored
rgrpc change and its changelog entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants