[gagantrivedi]

Specs the three experimentation methods from the issue, as a new @events capability.

What the spec now covers

Feature file	When step	SDK method
features/get_experiment_flag.feature	the application gets the experiment flag "{feature}" for identity "{identifier}" with traits:	getExperimentFlag
features/track_event.feature	the application tracks event "{event}" for identity "{identifier}" with value {value} and metadata:	trackEvent
features/track_exposure_event.feature	the application tracks an exposure of feature "{feature}" for identity "{identifier}" with value {value} and metadata:	trackExposureEvent

getExperimentFlag gets two scenarios: it resolves the feature for the identity and
persists its traits, and it records no exposure for an identity outside a running
experiment — the behaviour that separates it from getIdentityFlags and the one the
reference implementations go out of their way to get right.

The events path

compose.yml now brings up the experimentation services alongside Flagsmith, mirroring
Flagsmith's own docker-compose.yml, so @events scenarios run against the real path
rather than a stand-in for it:

SDK ─► events (flagsmith/events-ingestion-api) ─► kafka ─► warehouse-delivery ─► clickhouse ─► runner

• Flagsmith is pinned to 2.287.0, up from 2.276.0. That is the first release shipping
  experimentation/warehouses/clickhouse.py, the experimentation_environment_keys and
  experimentation_delivery_connections db views that ingestion and delivery read, and
  experimentation/clickhouse/events.sql. EXPERIMENTATION_ENABLED is on, and
  WAREHOUSE_CREDENTIALS_SECRET is shared with warehouse-delivery.
• Given the SDK sends events to Flagsmith connects a ClickHouse warehouse to the
  scenario's environment (POST /environments/<key>/warehouse-connections/) and asks for a
  client with events enabled. Both are needed: events are off by default in every SDK, and
  the events API only accepts an environment that has a warehouse connection.
• The runner reads delivered events out of ClickHouse (runner/sdk_spec/warehouse.py),
  keyed by the environment's API key — the same key Flagsmith reads them by in
  WarehouseConnectionViewSet.events.

Because the events client is a separate step, Create the SDK client with events enabled,
an SDK that has not implemented it reports it missing and the scenarios behave like any
other new step: xfail in the compatibility matrix, red in SDK repositories until they add
it. Every harness's marker regex was broadened from exactly Create the SDK client to any
marker starting with it; existing steps files are unaffected, and harness/rust and
harness/swift already infer "this step creates a client" from the absence of a client
parameter, so both creation steps work there unchanged.

What a reviewer should check

• The ClickHouse column mapping and the waits. An event crosses an SDK's send timer,
  Kafka, and a delivery batch, so Flagsmith has received the following events: polls for
  60s and Flagsmith has received no events waits 30s. warehouse-delivery runs with
  BATCH_MAX_WAIT_MS: 1000 to keep that bounded. The negative can only produce false
  passes, never false failures.
• value is asserted as text because all three reference SDKs stringify it, and
  metadata is matched on the entries the table names, because SDKs add their own and the
  ingestion server lifts sdk_version into its own column.
• clickhouse-init fetches events.sql from the raw URL pinned to v2.287.0, the way
  Flagsmith's compose does from main. Vendoring the schema here instead is an easy swap.
• EVENTS_SDK_URL defaults to http://events:8080, the compose service, rather than
  being derived from another URL. That keeps run-spec.yml unchanged, so SDK repositories
  pick events up by bumping their pinned ref. Running a harness on the host needs
  EVENTS_SDK_URL=http://localhost:8080, which the README says.

Additive for SDKs, so this is a minor version: new scenarios, a new tag, new When steps
and a new client-creating step, with no existing step reworded. The pinned Flagsmith
version moves, as a spec release's pin is expected to.

Testing

Docker does not run in the environment this was written in, so the stack could not be
brought up locally. The runner was exercised end to end against stubs shaped like the real
services — ClickHouse's HTTP interface returning JSONEachRow with traits/metadata as
JSON text, and an ingestion endpoint that rejects an environment with no warehouse
connection. All six scenarios pass; mutating the fake SDK to send a wrong value, or to send
an exposure outside an experiment, fails the right scenario with a readable message;
removing Create the SDK client with events enabled leaves the four @events scenarios
xfail and the @core ones passing. The stubs are not part of this change.

CI on this PR confirms the stack comes up — all eight containers healthy, kafka-init and
clickhouse-init completed — and that the existing scenarios still pass against Flagsmith
2.287.0. It cannot exercise the @events assertions, because every SDK still xfails the
events client. The first real run of them will be whichever SDK implements that step first.

Closes #13

This pull request was opened by an AI agent (OpenHands).

[khvn26]

We're bringing up live Flagsmith in the spec testing workflow. I expect these endpoints to not be mocked.

[gagantrivedi]

Agreed, and fixed in 3342bac — events-api/ is deleted.

I had looked for an events endpoint on the Flagsmith image and concluded there wasn't one. I'd missed that the ingestion service is a separate published image, and that your docker-compose.yml wires up the whole path. compose.yml now mirrors it: kafka + kafka-init, clickhouse + clickhouse-init, flagsmith/events-ingestion-api:0.5.1 as events, and flagsmith/warehouse-delivery:0.4.0. The SDK posts to the real ingestion server and the runner reads the delivered rows out of ClickHouse.

Three things that came with it, which are worth a look:

• Pinned Flagsmith 2.276.0 → 2.287.0. 2.276.0 has neither experimentation/warehouses/clickhouse.py nor the experimentation_environment_keys / experimentation_delivery_connections db views the ingestion server and delivery read. 2.287.0 is the first release carrying all of them plus experimentation/clickhouse/events.sql. EXPERIMENTATION_ENABLED and WAREHOUSE_CREDENTIALS_SECRET are set on the flagsmith service, the latter shared with warehouse-delivery.
• Given the SDK sends events to Flagsmith now connects a warehouse to the scenario's environment via POST /environments/<key>/warehouse-connections/. experimentation_environment_keys only lists keys whose environment has a connection, so without it the ingestion server rejects the batch.
• clickhouse-init fetches events.sql from the raw URL pinned to v2.287.0, the way your compose does from main. Say the word if you'd rather vendor the schema into this repo.

One honest caveat on verification: Docker does not run in the environment I work in, so I could not bring this stack up. I exercised the runner end to end against stubs shaped like the real thing — ClickHouse's HTTP interface returning JSONEachRow with traits/metadata as JSON text, an ingestion endpoint that rejects an environment with no warehouse connection — and the six scenarios pass, with deliberate mutations failing the right ones. What CI proves on this PR is that the stack comes up and the existing scenarios still pass; it cannot exercise the assertions, because every SDK still xfails Create the SDK client with events enabled. The first real run of these scenarios will be whichever SDK implements that step first, so please treat the timings (60s arrival, 30s settle) and the ClickHouse column mapping as the parts most worth a second pair of eyes.

— OpenHands agent

[khvn26]

Suggested change

	An SDK resolves a feature for an identity and records the exposure, in one call, so that
	an application can show a variant and have it measured. Only an identity enrolled in a
	running experiment is exposed: every other evaluation is left out of the data, so that
	an experiment's results count the identities it actually reached.
	An SDK resolves a feature for an identity and records the exposure.

[gagantrivedi]

Applied in 3342bac.

I took the same cut to track_exposure_event.feature, which you didn't comment on but reads the same way, so the three @events files are consistent: An SDK records which variant of a feature an identity was shown, as a "$flag_exposure" event. Happy to revert that one if you meant the edit narrowly.

— OpenHands agent

[khvn26]

Suggested change

	An SDK sends an application's tracking events to Flagsmith, so that an experiment can
	measure what the identities it exposed went on to do. Values reach Flagsmith as text,
	whatever type the application tracked them as.
	An SDK sends an application's tracking events to Flagsmith.

[gagantrivedi]

Applied in 3342bac.

— OpenHands agent

[khvn26]

Same as https://github.com/Flagsmith/sdk-spec/pull/17/changes#r4340296883 — ideally, we'd like the harness to use the live API we offer via docker compose.

[gagantrivedi]

Done in 3342bac. runner/sdk_spec/events_api.py is gone, replaced by runner/sdk_spec/warehouse.py, which queries the ClickHouse that compose.yml connects to each @events environment:

SELECT event, feature_name, identifier, value, traits, metadata
FROM events
WHERE environment_key = {environment_key:String}
ORDER BY timestamp, event

So the runner now reads the far end of the real delivery path — ingestion → Kafka → warehouse delivery → warehouse — rather than a recording of what the SDK posted.

Two details that changed with it:

• Events are read by the environment's API key, not the server-side key the SDK authenticates with. That matches how Flagsmith reads them itself: WarehouseConnectionViewSet.events passes self.kwargs["environment_api_key"] as environment_key.
• traits and metadata arrive as JSON text and are decoded before comparison. metadata is still matched on just the entries the table names, which now also covers the ingestion server lifting sdk_version into its own column.

The waits grew to match the longer path: 60s for an event to arrive, 30s before concluding one was never sent. warehouse-delivery runs with BATCH_MAX_WAIT_MS: 1000 to keep that bounded.

— OpenHands agent