Skip to content

Add a workflow canvas for chaining models - #429

Merged
Priyanshu-u07 merged 8 commits into
InferiaAI:mainfrom
Dhirenderchoudhary:feat/workflow-canvas
Oct 24, 2026
Merged

Priyanshu-u07 merged 8 commits into
InferiaAI:mainfrom
Dhirenderchoudhary:feat/workflow-canvas

Conversation

@Dhirenderchoudhary

@Dhirenderchoudhary Dhirenderchoudhary commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Adds a Workflows page to the dashboard (under Build, behind deployment:list like Sandbox). You drag nodes onto a canvas, connect them, pick a deployed model for each, and run the whole pipeline. Each node shows its status, time in ms and output.

Banking voice agent workflow after a real run: STT 576 ms, LLM 1174 ms, TTS 4823 ms

Nodes: audio input (record or upload), text input, speech to text, LLM (with a system prompt), text to speech, text output, audio output. Two built-in templates:

  • Banking voice agent: mic -> STT -> LLM -> TTS -> audio out, with a demo bank system prompt and fake customer data.
  • Text chat: text -> LLM -> text.

How it works:

  • src/lib/workflow/nodeKinds.ts is one registry table. Each node kind declares its input and output type (audio or text), its config fields and a run function, so adding a node type is one entry.
  • engine.ts rejects bad connections (type mismatch, self loop, second input, cycle), runs nodes in topological order and stops at the first error.
  • Calls go to the inference service with the user's token and x-sandbox: true, the same as Sandbox. STT/TTS use /v1/audio/transcriptions and /v1/audio/speech from Add speech-to-text and text-to-speech endpoints #428.
  • Workflows save to localStorage. There's no backend storage yet.
  • The ready-deployments query moved out of Sandbox.tsx into useReadyDeployments so both pages share it. Sandbox behaves the same.

Needs #428 to run audio nodes (and for the model dropdowns to tell STT/TTS from LLMs).

Testing:

  • 13 new vitest tests for the registry, connection rules, run order, the voice template run (with mocked fetch checking all three requests and that outputs flow node to node) and storage. Full dashboard suite: 432 passed.
  • npm run build passes. npm run lint and tsc already fail on main (156 lint errors, 89 tsc lines). This adds none; the new files lint clean.
  • Ran it in Chromium against a local stack with Add speech-to-text and text-to-speech endpoints #428, speaches and Ollama, nothing mocked: logged in, loaded the voice template, uploaded a recorded question. STT 576 ms -> "What is the balance on my checking account?", LLM 1.2 s -> "Your checking balance is $4,820.55.", TTS 4.8 s on CPU -> playable audio.

Live latency per node (ticking while a node runs, then p50/p95 over recent runs, plus total and time to first audio):

Mid-run: LLM node timer ticking
After 3 runs: per-node p50/p95

@Priyanshu-u07

Copy link
Copy Markdown
Collaborator

Tested locally: the 440 dashboard tests and the build pass, and the API calls match #428. Two fixes before merge:

"Stop and run" uses the state from when Record was clicked (recorder.onstop keeps that render's setAudio). Reproduced with a test that renders the Workflows page with a fake mic and inference server:

  • Record while a run is in flight, let it finish, then Stop and run: the recording is stored but never runs. Easy to hit in a live demo when asking the next question while the previous answer is still processing.
  • Change the LLM model while recording: the old model is sent.

Calling setAudio through a ref fixes both (verified with the same test). Please also replace the silent if (running) return; in run() with a toast, so stopping during a run isn't silent.

TTS autofill can pick the STT model: the TTS pattern matches bare speech, so deployments ["Speech Recognition", "Kokoro"] put both nodes on "Speech Recognition". Tightening both patterns and skipping deployments already given to another node fixes it. Default speaches names aren't affected.

Stopping a recording called the setAudio from the render where Record was clicked, so a run in flight at that moment swallowed the recording and settings changed while recording were ignored. It now calls the current one, and a run started while another is going shows a toast instead of doing nothing. The TTS name match included 'speech', so an STT model named 'Speech Recognition' was picked for TTS too; the patterns are tighter and autofill no longer gives one deployment to two nodes.

Signed-off-by: Priyanshu-u07 <connect.priyanshu8271@gmail.com>
@Priyanshu-u07

Copy link
Copy Markdown
Collaborator

Pushed the fixes for both bugs above. Looks good to merge now.

@Priyanshu-u07
Priyanshu-u07 merged commit ec41a8a into InferiaAI:main Oct 24, 2026
2 checks passed
Priyanshu-u07 added a commit to Dhirenderchoudhary/InferiaLLM that referenced this pull request Oct 24, 2026
Resolve conflicts with the InferiaAI#429 fix commit: keep his autofill as written; carry over main's recorder fix, run-in-progress toast and tighter STT/TTS name hints. Add readiness and deploy to the node card test's context.

Signed-off-by: Priyanshu-u07 <connect.priyanshu8271@gmail.com>
Priyanshu-u07 added a commit to Dhirenderchoudhary/InferiaLLM that referenced this pull request Oct 24, 2026
Resolve conflicts with InferiaAI#429/InferiaAI#430: keep the setInput/inputs rename; carry over main's recorder fix and run-in-progress toast. Rename the node card test to setInput/inputs.

Signed-off-by: Priyanshu-u07 <connect.priyanshu8271@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants