feat: utility to verify sync behaviour correctness

agent/outbound.go

@@ -100,7 +100,10 @@ func (a *Agent) addAppUpdateToQueue(_ *v1alpha1.Application, new *v1alpha1.Appli
 	// because for managed-agent all changes should be done through principal
 	if reverted := a.appManager.RevertManagedAppChanges(a.context, new, a.sourceCache.Application); reverted {
 		logCtx.Debugf("Modifications done in application: %s are reverted", new.Name)
-		return
+
+		// We continue here, rather than return:
+		// - It's possible that the reason this addAppUpdateToQueue function was called was due to .status of Application being updated (which would be unaffected by revert of spec, above)
+		// - We thus need to send that .status event to ensure principal receives it
 	}
 
 	q := a.queues.SendQ(defaultQueueName)

hack/sync-consistency-util/.gitignore

@@ -0,0 +1,2 @@
+sync-consistency-util
+vendor

hack/sync-consistency-util/README.md

@@ -0,0 +1,103 @@
+
+# Application Synchronization Simulation and Verification Utility
+
+The primary goal of this utility is to verify the correctness of Application synchronization behaviour between agent and principal. 
+
+This utility simulates Application resource events:
+- Randomly creates/modifies/deletes Argo CD Application resources on control plane cluster
+    - Verifies that the corresponding creation/modification/delete was made on managed-agent, in order and in a timely fashion (via K8s watch API)
+    - Each time Application is modified, the `.spec.source.repoURL` field is modified with a unique value, to track `.spec` synchronization to agent
+- Randomly modifies `.status` field of Argo CD Applications on workload cluster
+    - Verifies that the corresponding `.status` update is broadcast back to Application on control plane cluster
+    - Each time Application status is modified, the `.status.sync.revision` field is modified with a unique value, allowing us to track `.status` synchronization back to principal
+- `.spec` writer and `.status` writer run concurrently, from a single OS process
+- `.spec` watcher and `.status` watcher (responsible for observing events before verification) likewise run concurrently
+
+However, the implementation details of this utility are slightly more complex, due to Argo CD agent de-duplication behaviour:
+- Event writer (`event_writer.go`) will de-dupe '.status update' and '.spec update' events 
+- Event writer will also de-dupe delete events: if an application is deleted, all previously unsent events (.spec modifications, etc) will be discarded (as they are necessarily stale)
+- If informer event buffer (`informer_event_buffer.go`) feature is enabled, updates to Application `.spec` that occur within X seconds will be de-duplicated
+
+This complex behaviour requires that this utility uses an algorithm that checks for eventual consistency, rather than an algorithm that looks for exact match between source/destination events.
+
+
+## How to run
+
+
+#### Setup
+
+Test utility currently assumes standard vcluster dev-env.
+
+```
+make setup-e2e
+```
+
+#### From window A: start agent processes
+Start principal/agents on vcluster dev-env, running locally:
+```
+make start-e2e
+
+# If you want to write the screen output to a file for debugging purposes, use `script (filename)`.
+
+# or, if debugging an eventual consistency problem, you can enable full logging on agent via:
+ARGOCD_AGENT_FULL_DETAIL=all ARGOCD_AGENT_LOG_LEVEL=trace  ARGOCD_PRINCIPAL_FULL_DETAIL=all ARGOCD_PRINCIPAL_LOG_LEVEL=trace make start-e2e
+```
+
+#### From window B: start utility
+
+In a separate window, run the utility:
+```
+cd hack/sync-consistency-util
+
+# Modify constants in `main.go` to fit your needs, or just run default.
+
+go run .
+
+# Or, if you want to keep running it over and over until it fails (useful for finding bugs), see the 'until-fail.sh' utility (from https://gist.github.com/jgwest/7048a765d398519837f990120cf3fdd0) and use:
+
+until-fail.sh go run .
+```
+
+Note:
+- Running this utility will delete all Applications on all vclusters at startup.
+- This will set replicas to 0 on application controller on managed agent (to avoid generating extra events).
+
+
+## Further details
+
+See source code for implementation details of eventual consistency checking.
+
+Principles of eventual consistency:
+- There is a non-trivial delay between when a create/update/delete is made on source (of truth), to when it is transmitted, processed, and applied on destination peer
+    - Within the utility codebase we refer to this non-trivial delay as propagation delay. 
+    - The more overloaded the K8s API server/etcd, and agents, the longer this propagation delay can take.
+    - We thus allow configuration of a 'max propagation delay' constant, which is the max expected value before we assert an event has been incorrectly, permanently lost
+- Add/delete events (lifecycle events) must always be transmitted from source/destination, as there should be no-dedupe of these:
+    - With one exception: if an application is created then quickly deleted, e.g. the outbox will contain: '1) create Application event 2) delete Application event'
+    - In this case, when the outbox still contains both, neither will be sent. (There is no point to sending a create if you already know it will be deleted immediately)
+- However, not all update events will be transmitted from source to destination: some deduplication occurs within agent code at various points
+    - BUT, even though we are de-duplicating spec-update and status-updates, we should never permanently MISS an Application resource update: 
+        - If
+            - Event 'A' occurs on source of truth (SoT) and is processed on peer (non-SoT) at time t=X
+            - And, one (or more) event 'B'-'Z' occur (on SoT) at T=X+Y, for some Y > 0
+        - Then
+            - At least one of the events from 'B'-'Z' should be processed on peer by X+'PD' (where PD is expected propagation, e.g. 15 seconds).
+- We should never receive events out of the order the event occurred on source of truth:
+    - A delete sent before a create
+    - Updates received in a different order on peer, from the order they were sent from source of truth
+
+## Current Limitations
+
+These are not design limitations: these are just features not yet implemented.
+
+- `Application` resources only (no other resource types are validated via create/modify/delete)
+- Non-destination-based-mapping case only (consistent with `make setup-e2e` default, as of this writing)
+- Assumes/requires that we are running on local dev-env/vcluster configuration (via `make setup-e2e`)
+- Only 1 writer goroutine per writer type (one for spec, one for status) -- spec-writes and status-writes run concurrently, but there are no parallel writers of the same type
+- Currently only verifies principal <-> managed-agent. Does not verify principal <-> autonomous-agent.
+- This tool may also be useful for benchmarking, BUT, at present we are blocked on vcluster etcd CPU usage 
+- Simulate creation of a new `Application` with name of a previously deleted `Application:` The mechanism I am using to create Applications will never create an application which had the name of a previously deleted application.
+- Currently this utility works end to end: a single OS process uses client-go to both watch Application CR and also create/update/delete Application CR. Instead, this could be split into two separate utilities, with a third utility to check the results. This would be useful for large scale multi-cluster (not just vcluster) checking.
+
+## Outstanding improvements to Argo CD Agent
+- As of this writing, boundedQueue [will silently drop events](https://github.com/argoproj-labs/argocd-agent/blob/8241768ae7a0ed5d110187af83d40b2fbf988f81/internal/queue/queue.go#L71)

hack/sync-consistency-util/go.mod

@@ -0,0 +1,198 @@
+module github.com/argoproj-labs/argocd-agent/hack/sync-consistency-util
+
+go 1.26.0
+
+require (
+	github.com/argoproj/argo-cd/v3 v3.4.3
+	k8s.io/api v0.34.0
+	k8s.io/apimachinery v0.34.0
+	k8s.io/client-go v0.34.0
+	sigs.k8s.io/controller-runtime v0.21.0
+)
+
+require (
+	cloud.google.com/go/compute/metadata v0.9.0 // indirect
+	cyphar.com/go-pathrs v0.2.1 // indirect
+	dario.cat/mergo v1.0.2 // indirect
+	github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.0 // indirect
+	github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.13.1 // indirect
+	github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.2 // indirect
+	github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect
+	github.com/AzureAD/microsoft-authentication-library-for-go v1.6.0 // indirect
+	github.com/MakeNowJust/heredoc v1.0.0 // indirect
+	github.com/Masterminds/semver/v3 v3.4.0 // indirect
+	github.com/Microsoft/go-winio v0.6.2 // indirect
+	github.com/ProtonMail/go-crypto v1.1.6 // indirect
+	github.com/argoproj/argo-cd/gitops-engine v0.7.1-0.20250908182407-97ad5b59a627 // indirect
+	github.com/argoproj/pkg v0.13.6 // indirect
+	github.com/argoproj/pkg/v2 v2.0.1 // indirect
+	github.com/beorn7/perks v1.0.1 // indirect
+	github.com/blang/semver/v4 v4.0.0 // indirect
+	github.com/bmatcuk/doublestar/v4 v4.10.0 // indirect
+	github.com/bombsimon/logrusr/v4 v4.1.0 // indirect
+	github.com/bradleyfalzon/ghinstallation/v2 v2.18.0 // indirect
+	github.com/casbin/casbin/v2 v2.135.0 // indirect
+	github.com/casbin/govaluate v1.10.0 // indirect
+	github.com/cenkalti/backoff/v5 v5.0.3 // indirect
+	github.com/cespare/xxhash/v2 v2.3.0 // indirect
+	github.com/chai2010/gettext-go v1.0.3 // indirect
+	github.com/chainguard-dev/git-urls v1.0.2 // indirect
+	github.com/cloudflare/circl v1.6.3 // indirect
+	github.com/cyphar/filepath-securejoin v0.6.1 // indirect
+	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
+	github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
+	github.com/distribution/reference v0.6.0 // indirect
+	github.com/dlclark/regexp2 v1.11.5 // indirect
+	github.com/emicklei/go-restful/v3 v3.12.2 // indirect
+	github.com/emirpasic/gods v1.18.1 // indirect
+	github.com/evanphx/json-patch/v5 v5.9.11 // indirect
+	github.com/exponent-io/jsonpath v0.0.0-20210407135951-1de76d718b3f // indirect
+	github.com/fatih/camelcase v1.0.0 // indirect
+	github.com/fxamacker/cbor/v2 v2.9.0 // indirect
+	github.com/go-errors/errors v1.5.1 // indirect
+	github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
+	github.com/go-git/go-billy/v5 v5.6.2 // indirect
+	github.com/go-git/go-git/v5 v5.14.0 // indirect
+	github.com/go-logr/logr v1.4.3 // indirect
+	github.com/go-openapi/jsonpointer v0.22.5 // indirect
+	github.com/go-openapi/jsonreference v0.21.5 // indirect
+	github.com/go-openapi/swag v0.23.1 // indirect
+	github.com/go-openapi/swag/jsonname v0.25.5 // indirect
+	github.com/go-redis/cache/v9 v9.0.0 // indirect
+	github.com/gobwas/glob v0.2.3 // indirect
+	github.com/gogo/protobuf v1.3.2 // indirect
+	github.com/golang-jwt/jwt/v4 v4.5.2 // indirect
+	github.com/golang-jwt/jwt/v5 v5.3.1 // indirect
+	github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect
+	github.com/google/btree v1.1.3 // indirect
+	github.com/google/gnostic-models v0.7.0 // indirect
+	github.com/google/go-cmp v0.7.0 // indirect
+	github.com/google/go-github/v69 v69.2.0 // indirect
+	github.com/google/go-github/v84 v84.0.0 // indirect
+	github.com/google/go-querystring v1.2.0 // indirect
+	github.com/google/uuid v1.6.1-0.20241114170450-2d3c2a9cc518 // indirect
+	github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674 // indirect
+	github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79 // indirect
+	github.com/inconshreveable/mousetrap v1.1.0 // indirect
+	github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
+	github.com/jonboulle/clockwork v0.5.0 // indirect
+	github.com/josharian/intern v1.0.0 // indirect
+	github.com/json-iterator/go v1.1.12 // indirect
+	github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
+	github.com/kevinburke/ssh_config v1.2.0 // indirect
+	github.com/klauspost/compress v1.18.0 // indirect
+	github.com/kylelemons/godebug v1.1.0 // indirect
+	github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de // indirect
+	github.com/mailru/easyjson v0.9.0 // indirect
+	github.com/mitchellh/go-wordwrap v1.0.1 // indirect
+	github.com/moby/spdystream v0.5.1 // indirect
+	github.com/moby/term v0.5.2 // indirect
+	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
+	github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect
+	github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 // indirect
+	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
+	github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f // indirect
+	github.com/opencontainers/go-digest v1.0.0 // indirect
+	github.com/opencontainers/image-spec v1.1.1 // indirect
+	github.com/patrickmn/go-cache v2.1.1-0.20191004192108-46f407853014+incompatible // indirect
+	github.com/peterbourgon/diskv v2.0.1+incompatible // indirect
+	github.com/pjbgf/sha1cd v0.3.2 // indirect
+	github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
+	github.com/pkg/errors v0.9.1 // indirect
+	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
+	github.com/prometheus/client_golang v1.23.2 // indirect
+	github.com/prometheus/client_model v0.6.2 // indirect
+	github.com/prometheus/common v0.66.1 // indirect
+	github.com/prometheus/procfs v0.16.1 // indirect
+	github.com/redis/go-redis/v9 v9.18.0 // indirect
+	github.com/robfig/cron/v3 v3.0.2-0.20210106135023-bc59245fe10e // indirect
+	github.com/russross/blackfriday/v2 v2.1.0 // indirect
+	github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3 // indirect
+	github.com/sirupsen/logrus v1.9.4 // indirect
+	github.com/skeema/knownhosts v1.3.1 // indirect
+	github.com/spf13/cobra v1.10.2 // indirect
+	github.com/spf13/pflag v1.0.10 // indirect
+	github.com/vmihailenco/go-tinylfu v0.2.2 // indirect
+	github.com/vmihailenco/msgpack/v5 v5.4.1 // indirect
+	github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
+	github.com/x448/float16 v0.8.4 // indirect
+	github.com/xanzy/ssh-agent v0.3.3 // indirect
+	github.com/xlab/treeprint v1.2.0 // indirect
+	go.opentelemetry.io/otel v1.43.0 // indirect
+	go.opentelemetry.io/otel/trace v1.43.0 // indirect
+	go.uber.org/atomic v1.11.0 // indirect
+	go.yaml.in/yaml/v2 v2.4.2 // indirect
+	go.yaml.in/yaml/v3 v3.0.4 // indirect
+	golang.org/x/crypto v0.49.0 // indirect
+	golang.org/x/net v0.52.0 // indirect
+	golang.org/x/oauth2 v0.36.0 // indirect
+	golang.org/x/sync v0.20.0 // indirect
+	golang.org/x/sys v0.42.0 // indirect
+	golang.org/x/term v0.41.0 // indirect
+	golang.org/x/text v0.35.0 // indirect
+	golang.org/x/time v0.15.0 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20260226221140-a57be14db171 // indirect
+	google.golang.org/grpc v1.79.3 // indirect
+	google.golang.org/protobuf v1.36.11 // indirect
+	gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
+	gopkg.in/inf.v0 v0.9.1 // indirect
+	gopkg.in/warnings.v0 v0.1.2 // indirect
+	gopkg.in/yaml.v2 v2.4.0 // indirect
+	gopkg.in/yaml.v3 v3.0.1 // indirect
+	k8s.io/apiextensions-apiserver v0.34.0 // indirect
+	k8s.io/apiserver v0.34.0 // indirect
+	k8s.io/cli-runtime v0.34.0 // indirect
+	k8s.io/component-base v0.34.0 // indirect
+	k8s.io/component-helpers v0.34.0 // indirect
+	k8s.io/controller-manager v0.34.0 // indirect
+	k8s.io/klog/v2 v2.130.1 // indirect
+	k8s.io/kube-aggregator v0.34.0 // indirect
+	k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b // indirect
+	k8s.io/kubectl v0.34.0 // indirect
+	k8s.io/kubernetes v1.34.2 // indirect
+	k8s.io/utils v0.0.0-20250604170112-4c0f3b243397 // indirect
+	oras.land/oras-go/v2 v2.6.0 // indirect
+	sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 // indirect
+	sigs.k8s.io/kustomize/api v0.20.1 // indirect
+	sigs.k8s.io/kustomize/kyaml v0.20.1 // indirect
+	sigs.k8s.io/randfill v1.0.0 // indirect
+	sigs.k8s.io/structured-merge-diff/v6 v6.3.2 // indirect
+	sigs.k8s.io/yaml v1.6.0 // indirect
+)
+
+replace (
+	github.com/argoproj/argo-cd/gitops-engine v0.7.1-0.20250908182407-97ad5b59a627 => github.com/argoproj/argo-cd/gitops-engine v0.0.0-20260527205206-ce55c857b5e5
+	github.com/golang/glog => github.com/golang/glog v1.2.4
+	k8s.io/api => k8s.io/api v0.34.0
+	k8s.io/apiextensions-apiserver => k8s.io/apiextensions-apiserver v0.34.0
+	k8s.io/apimachinery => k8s.io/apimachinery v0.34.0
+	k8s.io/apiserver => k8s.io/apiserver v0.34.0
+	k8s.io/cli-runtime => k8s.io/cli-runtime v0.34.0
+	k8s.io/client-go => k8s.io/client-go v0.34.0
+	k8s.io/cloud-provider => k8s.io/cloud-provider v0.34.0
+	k8s.io/cluster-bootstrap => k8s.io/cluster-bootstrap v0.34.0
+	k8s.io/code-generator => k8s.io/code-generator v0.34.0
+	k8s.io/component-base => k8s.io/component-base v0.34.0
+	k8s.io/component-helpers => k8s.io/component-helpers v0.34.0
+	k8s.io/controller-manager => k8s.io/controller-manager v0.34.0
+	k8s.io/cri-api => k8s.io/cri-api v0.34.0
+	k8s.io/cri-client => k8s.io/cri-client v0.34.0
+	k8s.io/csi-translation-lib => k8s.io/csi-translation-lib v0.34.0
+	k8s.io/dynamic-resource-allocation => k8s.io/dynamic-resource-allocation v0.34.0
+	k8s.io/endpointslice => k8s.io/endpointslice v0.34.4
+	k8s.io/externaljwt => k8s.io/externaljwt v0.34.4
+	k8s.io/kms => k8s.io/kms v0.34.0
+	k8s.io/kube-aggregator => k8s.io/kube-aggregator v0.34.0
+	k8s.io/kube-controller-manager => k8s.io/kube-controller-manager v0.34.0
+	k8s.io/kube-proxy => k8s.io/kube-proxy v0.34.0
+	k8s.io/kube-scheduler => k8s.io/kube-scheduler v0.34.0
+	k8s.io/kubectl => k8s.io/kubectl v0.34.0
+	k8s.io/kubelet => k8s.io/kubelet v0.34.0
+	k8s.io/legacy-cloud-providers => k8s.io/legacy-cloud-providers v0.34.0
+	k8s.io/metrics => k8s.io/metrics v0.34.0
+	k8s.io/mount-utils => k8s.io/mount-utils v0.34.0
+	k8s.io/pod-security-admission => k8s.io/pod-security-admission v0.34.0
+	k8s.io/sample-apiserver => k8s.io/sample-apiserver v0.34.0
+	k8s.io/sample-cli-plugin => k8s.io/sample-cli-plugin v0.34.0
+	k8s.io/sample-controller => k8s.io/sample-controller v0.34.0
+)