Repository navigation
Make the simulator hold up under a real client: scheduler, simulated nodes, and a Raft-backed store - #7
Merged
Conversation
`kubectl describe pod` said "State: Terminated (exit 0)" for a container that was running. Nothing wrote that: exitCode and signal inside the terminated state defaulted to 0 in the schema, so mongoose materialised a terminated state for every container status it stored. The defaults are gone, and a container that starts now records `state.running.startedAt`, so describe agrees with the pod phase. `kubectl get pods -o wide` printed metadata.generateName — the container-name prefix — in the NODE column, so every pod appeared to be on its own node named after itself. It now reports spec.nodeName. Empty cells use `<none>` rather than `<None>`, which is what kubectl's own printers emit, and READY counts the ready containers against the pod's container count instead of assuming one.
A pod that finished reported phase Succeeded while its container status still said running, so `kubectl describe` on a completed pod described a running container. The exit watcher now writes a terminated state per container with the real exit code, a Completed/Error reason and a finish time. `immutable: true` was stored and never enforced: an immutable ConfigMap accepted a data patch and answered 200, which is the one thing the flag exists to prevent. Patches and replacements that touch anything outside metadata on an immutable object are now rejected 422 Invalid, for merge, strategic, JSON Patch and apply bodies alike.
`kubectl expose` produced a Service whose TYPE column was blank: Kubernetes defaults an unset spec.type to ClusterIP and the schema didn't. IngressClass's table was the Service table copied wholesale, so `kubectl get ingressclasses` printed TYPE / CLUSTER-IP / EXTERNAL-IP / PORT(S) / SELECTOR against an object that has none of those fields — five empty columns and no sign of the controller, which is the only thing anyone looks at. It now shows NAME, CONTROLLER, PARAMETERS and AGE. Empty table cells now read `<none>` rather than `<None>`, matching what kubectl's own printers emit.
STATUS showed the raw phase, so a finished pod read "Succeeded" where kubectl shows "Completed" and a pod being deleted never showed "Terminating" at all. kubectl's printer treats the phase as the fallback: a deletion timestamp wins, then a container's waiting or terminated reason, then the exit code. Same order here now, which the terminated states written on container exit make possible.
`kubectl get events` listed event names and ages — the two things nobody looks at — and nothing about the event itself. The name is a generated string, so the output was a column of noise. Now LAST SEEN, TYPE, REASON, OBJECT and MESSAGE, which is what kubectl's event printer shows, with the name demoted to a wide-only column.
`kubectl apply --dry-run=server` printed "(server dry run)" and created the object; `kubectl delete --dry-run=server` deleted it. `?dryRun=All` was ignored on every write path, so the one guarantee a dry run makes was the one thing it didn't do. Creates, replaces, patches and both delete paths now return what would have happened without touching storage. A delete carries the flag in a DeleteOptions body rather than the query string, which is why checking only the query still deleted the object. `kubectl get all` failed with "the server doesn't have a resource type all" — it's a category resolved from discovery, and no resource advertised one. Pods, services, replication controllers, deployments, replicasets, statefulsets, daemonsets, jobs, cronjobs and HPAs now declare the `all` category. test:wire creates and deletes with dryRun=All and fails if either took effect.
`kubectl get <kind> -A` asks for the collection without a namespace segment —
/apis/batch/v1/jobs, not /apis/batch/v1/namespaces/{ns}/jobs. Eleven kinds
404'd on that: most route files did register a cluster-wide alias, but under
/api/v1/<plural>, which is the core group and not where any of these resources
live (one of them was spelled "csistoragecapacitys"). The path is now derived
from the namespaced route by dropping the namespace segment, so it always
matches the one discovery advertises.
Only bindings and selfsubjectreviews still refuse a list, which is correct:
they are create-only in Kubernetes too.
test:wire lists every namespaced kind through its cluster-wide path.
Watch streams ignored their own scope. Live events arrive on a process-wide per-kind bus that knows nothing about the request, and nothing re-checked them, so a watch on one namespace received every object of that kind in the cluster — `kubectl get cm -n default -w` showed ConfigMaps created in other namespaces, and a label-selected watch showed objects that didn't match. Events are now filtered against the same query the equivalent list would run. Only the live path was affected; the initial state was always scoped. The scale subresource was the one write path that never checked dryRun, so `kubectl scale --dry-run=server` scaled the workload for real. Immutability was enforced by listing the fields a body mentions rather than comparing what the write would leave behind, so re-applying an immutable object's own contents was rejected 422 where Kubernetes returns 200 — any loop that re-applies unchanged manifests failed permanently. A PUT is compared as a replacement (an omitted field is a removal) and a patch as a merge. listByReq passed its empty queryOptions through as findAllSortedByReq's sortOptions, overriding that function's default, so the list path ran with no sort at all — and the _id tie-break added to make offset paging stable never applied on the one path that pages. Two pages either side of an update could repeat or skip an object. PUT serialised the raw mongoose document instead of going through asApiObject like the patch path, so `kubectl replace` answered with a body carrying `_id`. A delete held up by a finalizer answered Status: Success while the object was still there with only deletionTimestamp set. It now returns the object, as Kubernetes does while a deletion is pending; an ordinary delete still returns Success. Also fixes create() calling K8Object.applyGenerateName instead of this.applyGenerateName, which left `this.kind` undefined so the nameless review kinds never matched for callers that bypass the save middleware. test:wire opens a watch per namespaced kind and fails if an object created in another namespace arrives — verified to catch the original bug for 19 kinds when the filter is removed.
A patch whose update document contains no mongo operator had a bare `metadata` field merged into it to carry the new resourceVersion. Mongo applies that as a whole-subdocument set, so an update that only meant to change `spec` dropped name, namespace and uid and left the object unreachable. The version now goes in by dot-path when the caller supplied no metadata of its own, and into the caller's metadata when it did — the two cases have to be separate, because a dot-path alongside a whole-object set is a conflict. A deployment found its ReplicationControllers with the regex `^<name>`, which also matched every other deployment whose name starts with this one's. Creating `web` alongside an existing `web-cache` numbered web's first controller web-2, and deleting `web` cascaded into web-cache's controller and pods. The lookup is anchored on the generation suffix now, and the name is escaped rather than interpolated raw into a regex. Verifying that cascade turned up a third bug: deleting a deployment left its pods and their containers running. Two causes — deletePods bailed out when the in-memory replica counter said zero, which it often does, and the pod lookup passed its sort/limit into find()'s projection argument, so the pods came back with no metadata and nothing matched. test:workload now asserts the pods are gone after the deployment is.
Two more from a second review pass over the same branch. The finalizer guard read "has finalizers AND no deletionTimestamp", so only the first delete was held: the second fell through and removed the object with its finalizer still set. kubectl retries deletes while it waits for the object to disappear, so in practice the finalizer bought nothing. The guard is the finalizers alone now, and a repeat delete on an already-marked object is a no-op that returns it unchanged. A ReplicationController selected its pods with the regex `^<name>-`, which also matched the pods of any controller whose name extends its own: a standalone ReplicationController named `app` and a Deployment named `app` (whose controller is `app-1`) share that prefix, so deleting the standalone one deleted the deployment's pods and stopped their containers. Pods now record an ownerReference to the controller that made them — which is what the ReplicaSet controller already did — and cleanup asks that instead of guessing from the name. The anchored name pattern stays as a fallback for pods written before the reference existed. test:wire deletes each kind twice behind a finalizer and fails if the object vanishes, then clears the finalizer and fails if it doesn't.
The owner lookup added in the previous commit asked for `ownerReferences.uid: <controller uid>`. When the controller has no uid that becomes a query for pods whose owner uid is null — which is every pod that has no owner at all — so deleting such a controller would delete unrelated pods, including standalone ones a user created. Controllers written through the raw-insert fallback in create() bypass the schema defaults and have exactly that shape, so it is reachable rather than theoretical. The owner clause is now only added when there is an owner to ask about; otherwise the anchored name pattern stands alone. test:workload creates a pod nobody owns alongside the deployment and fails if the deployment's cleanup takes it.
Three of the pod features the README advertises were broken, each in a way that
answered 201 and then produced a pod that never ran or ran without its data.
`envFrom` failed whenever an entry set only a configMapRef. `secretRef` is a
nested schema path, so mongoose materialises it as `{}` — truthy, with no name
— and the branch that handles secrets ran anyway, looked up Secret
"undefined", threw, and left the pod in Failed. The entries are keyed on the
name now rather than on the presence of the wrapper object.
ConfigMap volumes never mounted, for two independent reasons. The pod schema
declared volumes as `{name, volumeSource}`, a wrapper Kubernetes doesn't have —
manifests put `configMap` inline next to `name`, so mongoose dropped the source
and stored `{name: 'cfg'}` alone. And the mount options were built by spreading
a mongoose subdocument, which copies its internals rather than its fields, so
mountPath came out undefined and the bind-mount construction threw. The pod
then went to Failed with a TypeError for a status message.
`stringData` on a Secret was stored verbatim and echoed back. It is write-only
in Kubernetes: the server base64s it into `data` and never returns it. Secrets
written the way most manifests write them came back with no `data` at all, so
anything consuming them got nothing.
Also: the pod log endpoint required `?container=`, answering 404 "pod not
found" for a plain GET, where Kubernetes defaults to the only container and
asks for a name only when there are several. And reading logs from a container
that never started surfaced docker's error as a 500 rather than an empty body.
test:pods covers all of it end to end.
Reviewing the previous commit turned up two problems with its own changes.
Folding stringData into data only matched a whole `stringData` object, but a
merge patch arrives flattened to dot-paths — `stringData.TOKEN`, not
`stringData` — so `kubectl patch secret --patch '{"stringData":…}'` still stored
the plaintext under stringData and produced no `data`. Both shapes are handled
now.
Declaring the pod's volume sources by spreading volumeSchema pulled in every
sub-schema's defaults, so each volume came back carrying ten empty source
objects (awsElasticBlockStore, cephfs, fc, gcePersistentDisk, iscsi, rbd,
downwardAPI, projected, secret) that the client never set — noise in every
`kubectl get pod -o yaml`, and a round trip through apply would send them all
back. The sources are declared as Mixed instead: what the client sent is stored
verbatim and nothing else materialises.
test:pods asserts a volume carries only the source that was set, and that a
patched stringData is converted and not stored.
An exec probe never ran what it was asked to run. `exec.command` is argv, but runProbe joined it into a string and execInContainer wraps a string in `sh -c` — so the canonical probe `['sh','-c','exit 1']` executed as `sh -c "sh -c exit 1"`, where "1" is $0 rather than an argument. It exited 0. Every exec probe passed regardless of what it checked, including the liveness probes that are supposed to restart a container. The readiness result was then discarded anyway: scheduleProbes computed it and only acted on liveness, so a container whose readiness probe was failing still reported ready: true and the pod still reported Ready — the single fact the probe exists to establish. Readiness now updates the container status and the Ready / ContainersReady conditions, and both probes honour failureThreshold rather than acting on the first failed check. Probes were also scheduled for every container. The schema defaults a readinessProbe object onto all of them, so testing the field rather than its contents armed a timer per container with nothing to check. And the timers outlived their pod: `delete()` stops the pod through a new Pod instance whose own interval list is empty, so the original timers kept running `docker exec` against a container that no longer existed. They are tracked per pod now and cleared on delete and on exit. test:pods asserts a failing probe leaves the container not ready and the pod not Ready, and that a passing one doesn't.
Reviewing the previous commit found two problems with its own changes.
`isConfiguredProbe` looked for `probe.httpGet` and `probe.tcpSocket`, but the
schema nests both inside `exec` — a shape Kubernetes doesn't have. So a probe
declaring httpGet had it dropped on save, and the guard I added then refused to
schedule it at all. Previously it was scheduled and silently answered true, so
neither version ever ran an httpGet or tcpSocket check. They are siblings of
exec now, declared as Mixed so a probe doesn't come back carrying an empty
`httpGet: {httpHeaders: []}` it never asked for, and both the guard and runProbe
read either shape.
Readiness was written on every probe tick rather than on a transition, so a
one-second probe bumped the resourceVersion and pushed a MODIFIED event to
every watcher once a second for a status that hadn't changed. It writes only
when the answer flips now.
test:pods covers an httpGet probe end to end and asserts a settled readiness
state stops rewriting the object.
Labels, annotations, nodeSelector, matchLabels and ConfigMap/Secret data
were `{type: Map, of: String}`. Mongo rejects dots in map keys, and
mongoose responded by dropping the *entire* map rather than erroring, so
`app.kubernetes.io/name` -- the label half of Kubernetes uses -- took the
whole label set with it. Everything reported success; the data was just
gone by the time it came back out.
They are Mixed now, which stores dotted keys as written. Label selectors
had to change with them: an equality clause can no longer be a plain
`metadata.labels.k` path, because a dotted key makes that path ambiguous,
so selectors match through $getField with the key as a $literal.
Merge patches needed the same care. A merge patch is flattened to dot
paths so it doesn't replace whole subdocuments, but the maps above are
opaque values, not structure -- flattening a label key that contains dots
invents nesting that was never there. Those paths stop flattening and
merge as values.
A controller's pod template is stored through the same metadata schema as a real object, so saving a ReplicationController stamped a uid onto spec.template.metadata. createPods deep-copied that template for each replica, and every pod came out with the *same* uid. Since a pod's status writes are keyed on uid, one replica's updates landed on its sibling: a pod whose container was running reported Pending indefinitely because its phase write had gone to the other document. Identity is stripped from the copied template now -- a controller carries labels and annotations out of a template, not uid, resourceVersion or creationTimestamp. Conditions are keyed by type, but every writer used $push, so a pod that transitioned twice carried two entries of the same type and a client reading "the" PodScheduled condition got whichever it happened to find first. setConditions() removes by type before pushing. The container-start path also stopped restating PodScheduled, which the scheduler owns.
Pods used to be placed by whatever created them, which meant nodeName was
decoration: nodeSelector, taints, cordons and resource requests had no
effect, and a pod that could not run anywhere still claimed a node. An
operator author testing against this saw a cluster that never says no.
There is a real scheduler now, and nodes for it to schedule onto. The
fleet is created at boot from SIM_NODES/SIM_NODE_CPU/SIM_NODE_MEMORY/
SIM_NODE_PODS/SIM_NODE_ZONES, each node carrying capacity, allocatable
and the topology labels (hostname, zone, region, os, arch) that real
scheduling constraints are written against.
Placement is filter-then-score, as upstream: readiness, cordoning,
NoSchedule/NoExecute taints against the pod's tolerations, nodeSelector,
required node affinity, and cpu/memory/pod-count fit computed from the
requests already on the node. Survivors are scored least-allocated, plus
preferred-affinity weight, so replicas spread instead of stacking.
What matters as much as the placement is the refusal. A pod that fits
nowhere stays Pending with PodScheduled=False, reason Unschedulable, and
a message in the form kubectl already knows how to read:
0/3 nodes are available: 1 node(s) had untolerated taint {dedicated:
batch}, 2 node(s) didn't match Pod's node affinity/selector.
Scheduled and FailedScheduling events are emitted alongside, and the
scheduler retries every 5s so a pod parked behind a full node moves once
room appears.
Resource quantities are parsed properly on the way in: "250m" is a
quarter core and "64Mi" is bytes, and anything unparseable is NaN rather
than a silent zero that would let an oversized pod fit anywhere.
test/scheduling.js covers this end to end -- 27 assertions, most of them
on the unhappy paths, since those are what a controller under test
actually has to handle.
resourceVersion was a counter in a database. It produced increasing
numbers, which is enough for a client to tell newer from older, but it
had no opinion about what "newer" meant: two writes racing on unrelated
objects were ordered by whichever transaction the database committed
first, and nothing recorded that decision anywhere a second process could
see it.
This adds the thing a real API server keeps underneath itself. store/ is
an MVCC keyspace with etcd's semantics -- a global revision, per-key
version chains, reads at a past revision, compaction that refuses stale
reads with code 11 instead of quietly serving the nearest surviving one,
leases, and watches that replay from a revision before going live --
sitting on a Raft log that decides the order writes happened in.
The API server's resourceVersion is now that revision, the way a real
one's is etcd's, so the numbers a client sees are positions in a
replicated log rather than a local tally. A store starting on a keyspace
whose versions came from the old counter adopts the highest number
already issued before serving any, because resuming below it would
reissue versions clients have seen and a watch resuming from one would
skip everything in between.
Raft here is not about uptime -- nobody runs three replicas of a test
fixture. It is what makes the ordering a fact: a leader assigns log
indices, followers forward writes rather than inventing an order, and a
committed entry survives losing the leader that accepted it.
The store speaks etcd's v3 JSON API on 2379, so it can be checked from
outside the process rather than taken on faith:
curl 127.0.0.1:2379/v3/kv/range -d '{"key":"L3JlZ2lzdHJ5"}'
That is the JSON gateway, not gRPC, so etcdctl won't attach to it yet.
What it does implement follows protobuf's JSON mapping exactly, including
the two rules that look cosmetic and aren't: bytes are base64, and int64
is a string, because a revision parsed as a JSON number stops being the
revision somewhere north of 2^53.
STORE_PEERS turns one node into a cluster; the defaults need no
configuration and keep everything in .store/. STORE=off falls back to the
database counter, and the log says which one is in use, because "what is
ordering my writes" is not a good thing to have to guess.
test/store.js covers it in 83 assertions, including a three node cluster
that elects a leader, replicates a write, forwards one sent to a
follower, and keeps both after the leader is killed.
…bjects A Binding is an instruction, not a record: POSTing one tells the API server to put a named pod on a named node, and nothing is stored. This kept it as an object -- so a custom scheduler could POST a binding, get a 201, and find its pod still Pending, which makes the one endpoint an out-of-tree scheduler needs the one endpoint that did nothing. It also meant a second binding failed on name uniqueness rather than on the pod already being assigned, and left rows behind that no client ever asks for. Both binding paths now do the write they claim to: /bindings and the pods/binding subresource go through the same bind the built-in scheduler uses, so a custom scheduler's placement is indistinguishable from ours -- same nodeName, same PodScheduled condition, same Scheduled event. A pod naming another scheduler is left alone until that scheduler binds it, binding an assigned pod is a 409 that says which node it is on, and binding a pod that doesn't exist is a 404. SelfSubjectReview was stored for the same reason and is a question too: `kubectl auth whoami` asks who it is and reads the answer. The events those paths emit rendered as "(x0 over 8s)" in kubectl, because every event carried a `series` -- and kubectl prefers series over count whenever the field is present, so an empty one reports that the event happened zero times. A series now appears only when there is actually a series, and an event that happened once counts as one.
Compaction removes the revisions *before* the compact point, not the point itself, so a read at exactly that revision is valid -- refusing it made the boundary a different number than the one the client was told about. Same for a watch starting there. A put naming a lease that doesn't exist threw after it had already appended the entry and detached the previous lease, and the batch rollback can't see into lease bookkeeping to undo that half. The check moves ahead of every mutation. A leader that stepped down left every uncommitted proposal waiting on a promise that would never settle -- a hung API request rather than a retry. Stepping down now rejects them, and the caller retries against whoever leads next. That retry is ambiguous by nature, because the original entry may still commit under the new leader; it is the same ambiguity an etcd client handles, and the worst case is a compare-and-swap that correctly reports the world moved. Documents the scheduler, the simulated fleet and the store in the README, including what they still don't do: etcdctl speaks gRPC and won't attach to the JSON endpoint, the object classes still write through Mongoose, and the simulated nodes have no kubelet.
A Service with a selector had no Endpoints object at all, and a Service
that points at nothing looks exactly like one whose pods haven't started
yet. Anything that waits for a backend -- an operator waiting for its
workload to become reachable, an ingress controller building a backend
list, a person running `kubectl get endpoints` to check that a selector
matches what they think it does -- waited forever and was told nothing.
controllers/endpoints.js keeps them in step: matching pods split into
ready and not-ready addresses, each carrying the pod IP, its node and a
targetRef back to the pod, with `targetPort` resolved against the
container's ports so a service naming a port by name reports a number a
client can dial. It is level-triggered on purpose -- it recomputes the
set from the pods that exist rather than patching it incrementally, which
is where real clusters grow their subtlest endpoint bugs -- and it skips
selectorless Services, since those are the documented way to point a
Service at addresses you manage yourself.
Building it turned up that Service.spec.selector was still a mongoose
Map, so a selector keyed on `app.kubernetes.io/name` -- the label half of
Kubernetes uses -- was dropped on write and matched nothing. Every
remaining `{type: Map, of: ...}` went the same way: resource `limits`,
`requests`, `capacity`, `allocatable`, ResourceQuota `hard` and `used`
(whose keys are literally `requests.cpu`), StorageClass `parameters`,
`nodeSelector`, `volumeAttributes`. All of them hold dotted keys in
ordinary use, and all of them silently held nothing.
`kubectl get endpoints` printed NAME/SUBSETS, which is not a header
anyone has seen; it is NAME/ENDPOINTS/AGE. Services rendered
"80/undefined" in PORT(S) whenever the protocol was left out.
Init containers were started without their `command` or `args`, so every one of them ran its image's default entrypoint, exited 0, and reported success without doing the work. An init container written to fail couldn't. Nothing recorded that they had run either -- status.initContainerStatuses stayed empty, so `kubectl describe` showed no Init Containers section and a controller waiting on its init step had nothing to read. They now report waiting → running → terminated with the real exit code, and a non-zero one fails the pod without starting the main containers. Containers also started the moment a pod was created, whether or not anything had scheduled it. A pod the scheduler had refused -- no node satisfied its selector, its resources fit nowhere -- still ran, went Running, took an IP, and turned up in a Service's endpoints: refused by the API and serving traffic at the same time. Startup now waits for spec.nodeName, which is the only moment a real kubelet would hear about the pod, and it doesn't care whether the built-in scheduler or somebody else's Binding did the assigning.
There was no way to ask "is this repo working?" -- `npm test` runs the external conformance harness and everything else was nine scripts you had to remember, which is eight scripts that don't get run. `npm run test:all` runs them all, each in its own process so one suite's leftover connections and timers can't decide another's result, and prints a line per suite plus the failing assertions of any that broke. `npm run test:all -- store rv` runs a subset, and it says up front when the server isn't running rather than letting six suites discover that one at a time. It reads each suite's own summary line rather than trusting exit codes: a suite that crashed before printing anything shouldn't look like one that passed. The README's test section is now that table -- what each suite covers and what it needs -- instead of nine near-identical code blocks, and the rest of it catches up with the scheduler, the store, endpoints and init containers, including what they still don't do.
Endpoints told a client which pods back a Service. EndpointSlice is what
anything written since Kubernetes 1.21 actually reads, and a controller
that asks for slices and finds none concludes the Service has no backends
-- it does not fall back to Endpoints. Offering only the older shape sent
half of this simulator's likely users looking for a bug in their own
code.
Each Service with a selector now gets both. The slice carries what
Endpoints has nowhere to put: per-endpoint conditions that separate
`serving` from `ready` and mark a pod `terminating`, the node and
topology zone each address sits in (which is what topology-aware routing
decides on), a targetRef back to the pod, and the
`kubernetes.io/service-name` label clients select on. One slice per
service -- sharding exists for a scale this will never see, and a client
is told to expect one or many either way.
The slice name is derived from the service rather than random, so
reconciling doesn't leave a trail of orphaned slices each claiming to
describe the same Service, and deleting a Service now takes its slices
with it.
Two things had to be fixed to make any of it work. The schema pointed
`endpoints[].targetRef` at the Endpoints address shape ({ip, nodeName,
targetRef}), so a targetRef written as an ObjectReference matched nothing
and was dropped -- the slice listed an address with no way to tell whose
it was. And the cascade query looked up the service-name label through a
dotted Mongo path, which means "nested field" and matched nothing now
that labels hold dotted keys literally.
`kubectl get endpointslices` prints NAME/ADDRESSTYPE/PORTS/ENDPOINTS/AGE;
the table returned name and age, so every slice in a list looked alike.
Two jobs, split by what they need. The store suite needs neither the API server nor a database, so it answers in seconds while the other job is still pulling images -- and it is the one most likely to catch a real logic error rather than an environment one. The second brings up MongoDB as a service container, starts the API server, and runs `npm run test:all` against the runner's own Docker daemon, which plays exactly the part a laptop's does: pods are sibling containers on it. The details that decide whether a red build means anything: mongo has a health check, so the server can't start against a database that isn't listening yet and fail every suite for a reason unrelated to the change; busybox is pulled up front, so a slow registry looks like a slow pull rather than a pod that wouldn't start; and on failure the server log and `docker ps -a` are printed, because when a suite fails for a reason its assertion can't see, that is where the answer is.
Every response body was logged in full. That was fine when the server answered a handful of requests by hand, but one `kubectl get pods` writes an entire PodList to the log, and a full test run buried whatever mattered under thirty thousand lines -- in CI it would be the whole build output. It is one line per request now (method, path, status, duration), with DEBUG_BODIES=1 to get the headers and bodies back when that is what you are actually looking for. A test run went from ~30k lines to 1.8k. The five one-off migration scripts in k82JSHelpfulScripts are gone: their output is committed in objects/ and routes/, they rewrite code shapes that no longer exist, and running one now would corrupt the files it was written to fix. The generators that can still be re-run stay. kubeconfig is written by scripts/setup.sh, so committing it tracked a generated file that every setup would dirty. It is ignored instead, and the README says where it comes from. Also dropped exports nothing imports -- the scheduler's internals were exported "for tests" that go through HTTP instead, and endpoints and mvcc exposed helpers only they use -- plus a dead variable left in the Raft append path.
CI found this on its first run, which is the whole reason to have it: objects/index.js required './csiDriver.js' and the file is csidriver.js. macOS doesn't care, Linux does, and the difference is not a subtle bug -- the server threw MODULE_NOT_FOUND at startup and never listened. Every CI runner and every container is Linux; every machine this was written on is not. Five requires were wrong that way (csiDriver twice, daemonSet, replicaSet, and csidriver in namespace.js), and the filename itself was inconsistent between objects/ and routes/. The file is csiDriver.js now, matching the camelCase every neighbouring file uses. test/portability.js walks every relative require in the repo and resolves it the way Node does minus the filesystem's help, so this fails on the machine that introduces it rather than eight minutes later in CI. 500 requires checked, and it needs nothing to run -- it joins the store in the standalone CI job.
master renamed the project to "Kubernetes in Node.js" and gave it a tagline; this branch had rewritten the README around a different name. The owner's naming wins -- the title and tagline come from master, the body from here, since it describes what the code now does. Three things master's README said that this one had dropped come back with it: which Kubernetes version this tracks (v1.29), that you can point DB_URL at your own MongoDB instead of the one setup starts, and the invitation to open an issue or a PR. The stated Node floor moves from 18 to 20, which is what master documented and what CI actually runs; scripts/setup.sh was still gating on 18 and now agrees.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The claim in the README is that you can point a real
kubectlat this. This branch is about the places where that claim held for the request and broke somewhere the client could see — plus the three pieces the simulator was missing to be useful for testing controllers against: somewhere to schedule pods, something to schedule them onto, and something that orders writes.A scheduler, and nodes to schedule onto
Placement used to be whatever created the pod set
nodeNameto, which madenodeSelector, taints, cordons and resource requests decorative — an operator author saw a cluster that never says no.There is a fleet now (
sim-node-1..N, with capacity, allocatable and topology labels) and a scheduler that runs kube-scheduler's shape: filter (readiness, cordons,NoSchedule/NoExecutetaints against tolerations,nodeSelector, required node affinity, cpu/memory/pod fit) then score (least-allocated, plus preferred-affinity weight).The refusals matter as much as the placements:
Scheduled/FailedSchedulingevents,PodScheduled=Falsewith reasonUnschedulable, and a 5s retry so a parked pod moves when room appears.Bindingused to be stored as an object, so a custom scheduler could POST one, get a 201, and find its pod still Pending. Both binding paths now perform the write, sharing the built-in scheduler's bind — a pod naming anotherschedulerNameis left alone until that scheduler claims it.An etcd-compatible store, on a Raft log
resourceVersionwas a counter in a database: increasing numbers with no opinion about what "newer" meant.store/is an MVCC keyspace with etcd's semantics — global revision, reads at a past revision, compaction that refuses stale reads with code 11 rather than serving the nearest survivor, leases, watches that replay from a revision — on a Raft log with a fsynced WAL, snapshots and leader failover.resourceVersionis now that revision, the way a real API server's is etcd's, and it adopts the old counter's high-water mark at boot so no version is ever reissued. The store serves etcd's v3 JSON API on:2379:curl 127.0.0.1:2379/v3/kv/range \ -d "{\"key\":\"$(printf '/registry/' | base64)\",\"range_end\":\"$(printf '/registry0' | base64)\"}"That is the JSON gateway, not gRPC —
etcdctlwill not attach to it. The object classes still read and write through Mongoose; the store hands out revisions and is durable and replicated, but it is not yet the object store.Services that have backends
A Service with a selector had no
Endpointsat all, which looks exactly like a Service whose pods haven't started. Now bothEndpointsandEndpointSliceare kept in step with the pods a selector matches — ready/not-ready split,targetPortresolved against the container's ports, and per-endpointready/serving/terminating, node and zone on the slice.Bugs this turned up
Pendingforever.app.kubernetes.io/namelabels, Service selectors, ConfigMap data keys, ResourceQuotahard(whose keys are literallyrequests.cpu), and resourcelimits/requests.command, so each one ran its image's default entrypoint, exited 0, and reported success without doing the work.(x0 over 8s)beside every event, because each carried an emptyseriesand kubectl prefers series over count when the field is present.Tests
npm run test:allruns every suite in its own process:CI runs the same thing on every push (
.github/workflows/tests.yml): one job for the store, which needs neither the server nor a database, and one that brings up MongoDB, starts the API server and runs the lot against the runner's own Docker daemon. Verified againstkubectlv1.34.1 throughout.Removed
The response-body logging (one
kubectl get podswrote a whole PodList to the log; a test run was ~30k lines, now 1.8k, withDEBUG_BODIES=1to get it back), the five one-off migration scripts whose output is already committed and which would corrupt the files they were written to fix, and the generatedkubeconfig, now ignored.