Rewrite reference counting with complete Perl-compatible ownership semantics #1649

Proposal and size estimate

Replace PerlOnJava's selective reference-counting overlay and corrective reachability decisions with an explicit, comprehensive Perl ownership model. Keep JVM allocation and physical memory reclamation, but make Perl-visible reference counts, weak-reference clearing, destruction, resurrection, and resource release follow standard Perl.

This is a major runtime/compiler project: provisionally 20–36 engineer-weeks for design, migration, both execution backends, and production acceptance (roughly 5–9 months for one experienced full-time maintainer). Two experienced contributors may shorten elapsed time, but the ownership model, migration barriers, and integration gates remain sequential dependencies. This is an engineering estimate from a source/design review, not a measured implementation schedule. Re-estimate after the ownership audit; native extension compatibility or changes to runtime value identity could expand it substantially.

Explicit goals: resolve #1618 and #1642, with permanent regression coverage and complete acceptance evidence. Keep those issues open until their individual acceptance criteria pass. This ticket proposes implementation; no rewrite or new validation has been performed as part of filing it.

Why this is larger than changing a counter

Reviewed baseline: master c38ef8adff2b8067c4ff51fa929ec491f89baa8e (2026-10-05).

Current code distributes lifetime decisions across RuntimeBase, scalar stores and proxies, container operations, call-frame aliases, captured pads, generated scope cleanup, interpreter registers, deferred releases, weak-owner registries, global destruction, thread graph cloning, and native/resource wrappers. RuntimeBase.refCount also encodes lifecycle sentinels, while other fields record ownership absent from that counter. MortalList can preserve a zero-count referent after root/capture/cycle checks. Internals::SvREFCNT contains calling-convention and named-binding adjustments. The existing design deliberately retains selective accounting and reachability fallback; this proposal requires a stronger invariant.

Java references are implementation pointers, not automatically Perl owners. Counting every Java field/temporary would also be incorrect. The rewrite must represent Perl scalar identity, RV-to-referent ownership, aggregate element ownership, aliases, pad ownership, and temporary ownership distinctly. Standard Perl's conventional borrowed argument stack and its optional reference-counted stack also differ; aliasing does not imply that every internal stack policy has identical numeric counts.

Evidence motivating the project:

• HTML::Tree 5.07 refloop test retains objects after releasing references #1618: HTML::Tree 5.07 t/refloop.t fails 4/8 assertions; expected releases leave observed counts of 2, 3, and 5, and destruction fails. Excess ownership or delayed release is a hypothesis, not yet an established root cause.
• Track independent reachability/lifecycle fixes and repeated Catalyst root walks #1642: its independent lifecycle candidate reports 4,830 of 4,836 JFR samples containing ReachabilityWalker.isReachableFromRoots during Catalyst prerequisite/setup. Repeated per-aggregate root walks are an identified cost. Those samples do not establish an infinite loop or quantify end-to-end speedup. Historical combined-parent passes do not validate the independent candidate.

Meaning of “exactly like standard Perl”

Use the repository's Perl 5.44 compatibility target as the intended oracle. Record the precise Perl version, build configuration (including argument-stack policy), relevant platform, and probe calling convention. Older installed Perl can provide preliminary evidence; record differences and obtain the target oracle before claiming exact parity.

Required observable behavior:

1. Strong ownership creation, transfer, replacement and release match Perl across scalar cells, RV referents, arrays, hashes, globs, code/pads and applicable runtime objects. A borrowed alias creates no independent referent owner merely because a Java wrapper exists.
2. Counts reflect the selected Perl oracle at controlled B::svref_2object(...)->REFCNT and Internals::SvREFCNT(...) observation points, including probe-induced temporary owners and API-specific semantics. Cover both reads and supported writes; assess artificial refcount changes explicitly rather than silently normalizing them. Do not compute the answer by forcing JVM GC or reconstructing reachability.
3. Releasing the last actual owner triggers the appropriate weak-reference and destruction behavior at Perl's cleanup boundary. Mortal temporary releases remain deferred according to Perl semantics; “exact” does not mean every decrement must occur at Java assignment time.
4. Correct DESTROY reentrancy, resurrection, reblessing, exception/warning behavior and preservation of relevant special variables; global destruction follows Perl's documented behavior and oracle-observable constraints. Do not impose a total destructor order where Perl leaves it unspecified.
5. Strong cycles behave as Perl cycles. Standard Perl does not generally collect an unreachable strong cycle; weak observers must not disappear solely because JVM GC runs. Design semantic retention for positive-count cyclic objects, scoped to the owning runtime, with defined global teardown. An opt-in diagnostic sweep must not silently change default lifetime semantics.
6. Filehandles, accepted sockets, pipes, closures, ithreads and shared aggregates retain their existing documented behavior under the new ownership model.

JVM heap addresses, allocation timing and allocator internals need not match Perl's C implementation. Full native XS/C-API support is a separate project (#1586); ownership exposed by existing Java/bundled providers is in scope.

References: Perl ownership and mortality, Perl circular references. Correct the existing architecture table's “cycle collector” claim when updating documentation. Also replace stale statements about selective tracking, argument copies and reachability with verified current behavior.

Proposed implementation contract

• Define an ownership table for every value-producing/consuming operation: owned result, borrowed cell/alias, ownership transfer, or mortal temporary. Distinguish scalar-cell and referent counts.
• Centralize retain/release/transfer in a small documented API. Prefer explicit ownership tokens or equivalently auditable slot/frame contracts; prototype before choosing the representation. Do not assume every Java object maps one-to-one to a Perl SV.
• Keep lifecycle state separate from a nonnegative semantic count. Define ordinary zero transitions, destructor entry/exit, weak-reference handling, resurrection, immortal constants, and externally adjusted counts.
• Stores acquire or transfer the incoming owner before releasing the outgoing owner when needed to preserve self-assignment and reentrant destructor safety. Container mutations retain Perl-specific sequencing and alias behavior.
• Generate cleanup for all exits: fall-through, return, exceptions/eval/try, loop exits, goto/tail calls and backend fallback. Release escaping results only after ownership is transferred to the caller; captured pads survive their declaring frame.
• Use explicit runtime roots for ownership and cycle retention. Runtime reset/thread teardown must release its own registry and avoid holding other runtimes alive. Ordinary local accounting should not require global atomic counters; cross-thread shared storage needs a defined synchronized lifetime protocol.
• Graph walkers may remain diagnostic or transitional tools. In fully migrated default paths, ordinary release, weak clearing, count queries and destructor dispatch must not require corrective whole-root scans, JVM GC, or fabricated owner counts.
• A migration flag may enable comparison of legacy and new accounting, but only one implementation may dispatch destructors or close resources. Define explicit bridges at migrated/unmigrated boundaries; never count the same owner twice.

Phases: complete, enabled ownership paths

Organize implementation around complete ownership paths so every phase can be fully merged, tested, and measured. A path includes its producers, storage, consumers, cleanup on both backends, and any weak-reference, destruction, introspection, or resource behavior needed to make its lifetime correct. Core API work belongs inside the first complete path; a dormant API alone does not complete an implementation phase.

Effort remains provisionally 20–36 engineer-weeks overall, including approximately 2–3 weeks for phase 0. The previous estimates by technical layer do not directly map to these complete paths. Re-estimate individual phases after the audit establishes their dependencies; moving work into earlier phases does not by itself reduce total effort.

The audit must establish whether the current representation permits these boundaries. Pull prerequisites such as scalar identity, destruction or call cleanup into the earliest phase that needs them. If a family cannot be isolated safely, expand that phase or combine adjacent phases before implementation. Split phase 4 into separately numbered delivery phases if its families can be isolated. Do not declare a phase complete with a dormant implementation or an incomplete default lifetime path.

Required completion evidence for every phase

• The phase's PRs are merged and its documented checks pass on the final integrated source commit. Phase 0 establishes measurements; every implementation phase delivers an enabled result with a named observable improvement. Improvements may be correctness, supported behavior or reduced cleanup cost; do not imply every phase must make every benchmark faster.
• Every migrated object has exactly one authoritative lifetime implementation. All producers, storage slots, consumers and exits for its path are covered; transfers across the remaining legacy boundary are explicit and tested, including reentrancy, escaping values and prevention of double-release. New accounting stays non-dispatching until the complete path is ready to enable.
• New Perl-level regressions are validated against the recorded oracle, reproduce the failure on the unfixed parent, and pass on JVM and interpreter after the fix. Primitive tests cover count overflow/underflow, lifecycle transitions, cycle retention and reentrancy as applicable. All previously delivered phase regressions remain green.
• nice -n 19 make, relevant bundled gates and relevant unchanged upstream suites pass from immutable source barriers. Baseline known failures must have an explicit diagnostic/expected-failure mechanism so phase 0 can merge without breaking required gates; promote them to required passing coverage when fixed. Do not alter or delete existing tests to obtain a pass.
• Record before/after passing cases, retain/releases, root queries, roots seeded, edges visited, snapshot builds, runtime, allocation and retained heap where relevant. Compare against both the original baseline and the previous phase using the same workloads/configurations and the phase 0 tolerances. Fully migrated ordinary cleanup/count-query paths must require zero corrective whole-root scans; report remaining legacy costs separately.
• Update the design document with the merged commit, enabled value/owner families, remaining legacy boundaries, commands and full evidence, observed improvement, next phase and blockers. Keep a documented rollback mechanism during stabilization; a rollback must select a coherent lifetime implementation and must never switch the ownership authority of live objects unsafely.

Acceptance and evidence

• Project-owned minimal regressions for HTML::Tree 5.07 refloop test retains objects after releasing references #1618: standard Perl passes; unfixed PerlOnJava fails; new JVM and interpreter pass. Unmodified HTML::Tree 5.07 t/refloop.t passes 8/8, including final destruction and expected counts.
• Track independent reachability/lifecycle fixes and repeated Catalyst root walks #1642 retains all DOM/weak-parent, resurrection, accepted-socket and process-pipe ownership regressions. Review its independent candidate and sibling PR fix(parser): resolve five high impact parser issues #1623 before porting; neither previous green runs nor a snapshot-cache change proves this rewrite correct.
• Deterministic cost tests exercise repeated cleanup through MortalList or its replacement. Count retain/releases, root queries, roots seeded, edges visited and snapshot builds. Fully migrated ordinary cleanup and count queries perform zero corrective whole-root scans; unrelated roots do not multiply cleanup work. Recursive release work may scale with the owned subgraph actually released.
• Catalyst prerequisite/setup and its actual unchanged suite complete normally in an isolated CPAN home. Separate setup failures from suite outcomes. Run fresh end-to-end make test-cpan-release-acceptance and preserve the existing exclusions from Track independent reachability/lifecycle fixes and repeated Catalyst root walks #1642; add no exclusions to conceal lifetime defects.
• Unchanged Mojolicious dom.t and lifecycle-sensitive ecosystem suites (Moo/Moose, DBIx::Class, Sub::Quote/Sub::Defer, Resource::Silo, Dist::Zilla, Algorithm::SlidingWindow, relevant AnyEvent cases and Math::Decimal) pass against recorded versions; baseline independent failures are reported separately.
• Differential tests cover alias versus copy, replace/delete/splice/clear, temporaries, nested calls, closures, tied containers, scalar refs, globs/code/pad constants, cycles with weak observers, all scope exits, resurrection/reblessing and global destruction. Supported refcount introspection APIs match the selected oracle at explicit checkpoints.
• Thread clone/shared ownership, runtime isolation/reset and resource-release suites pass; repeated runtime teardown leaves no retained registries or orphan resources.
• nice -n 19 make and relevant bundled gates pass from immutable source barriers. Validate new Perl-level tests on system Perl first; capture full output and use timeouts for all jperl/jcpan/prove executions. Never alter/delete existing tests or mutate a checkout while a gate runs.
• Record CPU/allocation/heap effects against baseline on no-DESTROY workloads, closure calls, DOM construction and repeated cleanup. Set regression tolerances from baseline variance in phase 0; deterministic complexity checks accompany timings. A larger scalar/container metadata footprint is a material risk.
• Update a dedicated dev/design/ document after each phase, reconcile existing architecture/alignment docs, add a terse changelog entry under Work in progress, and document per-phase enablement, rollback and legacy-removal criteria.

Risks and design decisions to resolve in phase 0

Exact count compatibility may require separating wrapper identity from Perl SV identity and broadening tracking to all relevant cells. Thread-local versus shared lifetimes, reentrant ties/destructors, constant caches, probe-induced owners and explicitly writable counts are difficult cases. Preserving Perl strong-cycle semantics may intentionally retain memory that JVM tracing would otherwise reclaim. The migration must not trade delayed destruction for premature destruction. If phase 0 finds that the current value representation cannot support this contract, publish the architectural change and revised estimate before continuing.

Related existing plans are historical context, not implementation-complete evidence: dev/architecture/weaken-destroy.md, dev/design/refcount_alignment_plan.md, dev/design/refcount_alignment_progress.md, dev/design/refcount_alignment_52leaks_plan.md. This project supersedes the selective-accounting-plus-reachability strategy only as each path is migrated and validated.

Merge strategy and reviewable PR boundaries

A phase is a delivery milestone with a complete, enabled ownership path and recorded acceptance evidence. It may contain several independently reviewable PRs. Supporting PRs may introduce dormant primitives or diagnostics, but the phase is complete only after its behavior is enabled by default and its final integrated commit passes all phase gates. Every PR that changes runtime behavior must pass nice -n 19 make, focused standard-Perl/JVM/interpreter regressions and relevant unchanged upstream tests before merge. Keep source immutable while a gate runs.

1. Phase 0 PRs — Baselines and migration design. Add project-owned, system-Perl-validated reproducers for HTML::Tree 5.07 refloop test retains objects after releasing references #1618 and the Track independent reachability/lifecycle fixes and repeated Catalyst root walks #1642 repeated-cleanup cost case, deterministic diagnostic counters, and the ownership inventory/design. Record failing parent evidence on both backends where applicable. The cost test must exercise real deferred cleanup and report root queries, roots seeded, edges visited and snapshots built. Preserve passing required gates while recording known baseline failures. Merge the baseline and audit deliverables; keep HTML::Tree 5.07 refloop test retains objects after releasing references #1618 and Track independent reachability/lifecycle fixes and repeated Catalyst root walks #1642 open.
2. Phase 1 PRs — Core contracts plus the first working path. Develop documented owner kinds/transitions and focused primitive tests as part of the selected path. Include every required producer, storage slot, consumer, backend cleanup edge and lifecycle action. Test overflow/underflow, reentrancy and migration bridges. Enable the complete path by default in the phase's delivery PR and record its observable improvement. A dormant core API PR is supporting work, not the phase exit criterion.
3. Phases 2–4 PRs — Expand complete default-enabled paths. Each delivery PR migrates a coherent set of ownership edges and enables that path after its boundary and acceptance tests pass. Exactly one implementation dispatches destruction or releases resources for each object; aliases and cross-boundary references must not create a second authority. Keep incomplete new accounting non-dispatching. Pull required lifecycle/compiler integration forward rather than deferring it to a nominal later layer. Never mix in unrelated ownership changes without renewing acceptance evidence. Finish each phase with gates on the merged integrated commit and before/after results.
4. Phase 5 PRs — Final qualification and stabilization. After the inventory is complete, qualify the already default-enabled model with release acceptance, Catalyst, Mojolicious, lifecycle-sensitive CPAN suites, runtime-isolation/thread/resource tests, and performance/complexity checks on the exact candidate commit. Require HTML::Tree 5.07 refloop test retains objects after releasing references #1618 acceptance and removal of Track independent reachability/lifecycle fixes and repeated Catalyst root walks #1642's deterministic repeated-walk cause or a demonstrated bounded-cost replacement consistent with the zero corrective whole-root scan contract for fully migrated ordinary paths. Preserve a documented rollback mechanism for a stabilization window whose exit criteria are defined in the design.
5. Phase 5 completion PR — Remove legacy accounting and recovery. After stabilization evidence shows no supported path depends on legacy ownership, delete old selective-counter corrections, duplicate ownership fields, walker gates and migration bridges. Preserve useful semantically inert diagnostic graph tools. Re-run the complete final acceptance gate after removal. Phase 5 and this parent ticket remain incomplete until that evidence is recorded.

Each PR must update the design document's progress table with its source commit and, after merge, the merged commit; migrated value/owner families; remaining legacy boundary; tests/evidence; next PR; and blockers. Close #1618 or #1642 only when that issue's individual acceptance criteria are satisfied, which may precede completion of this parent project. Close #1649 only after all phases, including legacy removal and its final acceptance, are complete.

Sequencing constraint: define and revise phase boundaries around complete ownership paths. If the audit finds that identity or cleanup cannot be isolated safely, expand or combine the affected phases before proceeding. An always-on incomplete counter, a feature flag left off, or primitive tests alone do not constitute a completed implementation phase.