Repository navigation
Add apple_fm integration for Apple's on-device model - #100
Open
piotrekno1 wants to merge 11 commits into
Open
piotrekno1 wants to merge 11 commits into
piotrekno1 wants to merge 11 commits into
Conversation
Coverage report
Top uncovered files
|
Track LanguageModelSession.respond() and stream_response() from apple-fm-sdk: duration, time to first token for streams, prompt size, generation options, and the SDK's error names on failure. Sessions share one model per use case, versioned by the macOS release. Token counts stay empty: the SDK reports no usage, and counting would cost an extra model call per request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Apple's on-device model never returns an HTTP status, so the server had nothing to derive a code from and stored every failure as UNKNOWN. The integration now sends a specific code for each SDK error and keeps the SDK's name in provider_error_code. New codes MODEL_UNAVAILABLE, CONTEXT_LENGTH_EXCEEDED and CONTENT_FILTERED cover failures the existing codes don't. RATE_LIMITED and BAD_REQUEST match the names the server already derives for remote APIs. The server keeps a specific code as sent. docs/manual-tracking.md now lists every code.
An app that checks SystemLanguageModel.is_available() and falls back when the model can't be used left no trace. is_available() now records a model load. When the model is unavailable the load fails with the SDK's reason as its error code: APPLE_INTELLIGENCE_NOT_ENABLED, DEVICE_NOT_ELIGIBLE or MODEL_NOT_READY. Apps poll is_available() while the model downloads, so an event is sent only when the answer changes, per process and model.
The example printed the bare enum value, such as "unavailable: 2".
Both were recorded as UNKNOWN. DecodingFailureError means the model's output couldn't be parsed into the requested structure, so it is now INVALID_OUTPUT. ToolCallError wraps an exception from the app's own tool, so it is now TOOL_CALL_FAILED and tells app failures apart from model failures. Both codes fit other integrations too.
piotrekno1
force-pushed
the
feat/apple-fm-integration
branch
from
October 24, 2026 22:46
2795bc3 to
d478a1d
Compare
The test had only run where the model was unavailable. With the model ready, the content tagging model can't finish a reply in 8 tokens, and the SDK raises DecodingFailureError. 64 tokens is enough.
apple-fm-sdk reports no token usage. SystemLanguageModel.token_count() counts tokens without generating, at about 40 ms per call on an M2 Mac. With the variable set, respond() and stream_response() count the prompt and the reply in a background task and send the inference event with tokens in, tokens out and tokens per second. The call returns as soon as the model has answered. macOS queues counts behind generation, so a request that follows right after waits about 80 ms, which is why counting is opt-in. A done callback sends the event whether counting finished, failed or was cancelled before it ran, as when asyncio.run() closes the loop. The setting is an env var, not an init() parameter, so wildedge run can use it too. docs/configuration.md lists it under integration settings.
The docs, changelog and module docstring opened with "the SDK doesn't report token usage". SystemLanguageModel.token_count() counts them, and the integration uses it when WILDEDGE_APPLE_FM_COUNT_TOKENS is set.
The integration reported coreml because Apple doesn't say where the system model runs. powermetrics on an M2 settles it: while the model generates, Neural Engine power rises from 0 to about 5 W, GPU power stays near idle (38 to 67 mW) and CPU power rises to 1.5 W for the app and the model service. npu is a new accelerator name. The Swift and Android SDKs already use it.
LanguageModelSession.from_transcript(transcript, model=...) creates the session as cls(_ptr=...), so the patched __init__ never saw the model. A resumed content tagging session was recorded as the general model, and token counts used the default model. from_transcript() is now patched too, and registers the session for the model passed to it. A compat test resumes a content tagging session on the real SDK, and fails without the fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an
apple_fmintegration for apple-fm-sdk, Apple's Python SDK for the on-device model behind Apple Intelligence.With
wildedge.init(integrations=["apple_fm"]):LanguageModelSession.respond()andstream_response()each record one inference event with duration, prompt character/word counts andGenerationOptions(temperature, max tokens, sampling top-k/threshold/seed). Streams also record time to first token. Stream snapshots are cumulative and are passed through unchanged.Token counts are opt-in. With
WILDEDGE_APPLE_FM_COUNT_TOKENS=1,SystemLanguageModel.token_count()counts the prompt and the reply in a background task, and the event carries tokens in, tokens out and tokens per second. Each count takes about 40 ms on an M2 Mac. The call isn't slowed, but macOS queues counts behind generation, so a request sent right after waits about 80 ms. That's why counting is off by default.SystemLanguageModel.is_available()records a model load. When the model can't be used, the load fails with the SDK's reason as its error code (APPLE_INTELLIGENCE_NOT_ENABLED,DEVICE_NOT_ELIGIBLE,MODEL_NOT_READY). An event is sent only when the answer changes, since apps poll while the model downloads.Failures record an error event with a specific error code, plus the SDK's error name in
provider_error_code:assets_unavailableMODEL_UNAVAILABLEexceeded_context_window_sizeCONTEXT_LENGTH_EXCEEDEDguardrail_violation,refusalCONTENT_FILTEREDrate_limited,concurrent_requestsRATE_LIMITEDunsupported_guide,unsupported_language_or_locale,invalid_schemaBAD_REQUESTdecoding_failureINVALID_OUTPUTtool_callTOOL_CALL_FAILEDRATE_LIMITEDandBAD_REQUESTmatch the codes the server already derives for remote APIs (analytics/error_codes.py). The server keeps a specific code as sent, so no server change is needed. The other five are newErrorCodevalues, listed indocs/manual-tracking.md.Sessions share one handle per use case:
apple-foundation-modelorapple-foundation-model-content-tagging. The model version is the macOS version.from_transcript()is patched too, because the SDK creates that session ascls(_ptr=...)and__init__never sees the model.The accelerator is
npu, a new name inwildedge.acceleratorsthat the Swift and Android SDKs already use. Apple doesn't document where the model runs, so I measured it withpowermetricson an M2: while the model generates, Neural Engine power goes from 0 to about 4.9 W, GPU from 38 to 67 mW, and CPU from 51 mW to 1.5 W (the app and the model service).Docs:
docs/apple_fm.md, an error code table indocs/manual-tracking.md, an integration settings section indocs/configuration.md, a README row, an example, a note indocs/providers.mdonfm serve(macOS 27) via theopenaiintegration, and changelog entries.Known gaps
asyncio.run()cancels the pending count. The event is still sent, without tokens.Testing
tests/test_integrations_apple_fm.py) run against a fakeapple_fm_sdkthat copies 0.2.1's signatures. Full suite: 949 passed.Mac14,9), with events sent to a dev project:MODEL_NOT_READY, andAssetsUnavailableErrorfrom a call made anyway.respond()andstream_response()inference events with duration and time to first token. With counting on,respond()recorded 5 tokens in and 37 out at 23.9 tokens/s.from_transcript(), and it fails without the fix. Running it with the model available showed that the content tagging model can't finish a reply in 8 tokens, so the test now allows 64.apple_fm(apple-fm-sdk==0.2.1, Python 3.12/3.13) on amacos-26runner. Unverified: I haven't confirmed that the runner label exists or that the SDK builds there. CI can't enable Apple Intelligence, so there the compat test covers the signatures the patches rely on, the failed load, and the failed call.🤖 Generated with Claude Code