Skip to content

Add image and video nodes to workflows - #432

Merged
Priyanshu-u07 merged 4 commits into
InferiaAI:mainfrom
Dhirenderchoudhary:feat/image-video-nodes
Oct 24, 2026
Merged

Priyanshu-u07 merged 4 commits into
InferiaAI:mainfrom
Dhirenderchoudhary:feat/image-video-nodes

Conversation

@Dhirenderchoudhary

Copy link
Copy Markdown
Collaborator

Builds on #430 (and #429). Adds image and video to the Workflows canvas so a flow can mix text, audio, image and video models.

New nodes:

  • Image input and Video input. A video is turned into N evenly spaced frames in the browser (default 4), shown as thumbnails.
  • Image generation (text -> image) through /v1/images/generations, model_type = image_generation.
  • Vision (image or frames -> text) through /v1/chat/completions with OpenAI image content, for vision models such as moondream or llava.
  • Image output.

New templates: "Image from a description", "Describe a video", "Spoken image request" (mic -> STT -> image). The plain-English planner picks the new nodes up from the registry, the keyword fallback knows image/video requests, and the readiness catalog recommends sd-turbo and moondream. Vision models now rank after text models in the LLM dropdowns so a chat node doesn't default to moondream.

Testing:

  • 66 workflow tests (new: port types, image edges, request bodies, frame timing, planner), full dashboard suite 485 passed, build, eslint and tsc clean on changed files.
  • Ran all three templates in Chromium against a local stack (no GPU): sd-turbo behind a small OpenAI-compatible image server and moondream on Ollama, both registered as deployments. All calls 200.
  • On laptop CPU, image generation took ~46 s warm while another model was using the CPU, and moondream's description of 4 frames was weak. These are CPU numbers; on a GPU sd-turbo is a single step.

Image from a description
Describe a video
Spoken image request

Dhirenderchoudhary and others added 4 commits October 24, 2026 08:10
Resolve conflicts with InferiaAI#429/InferiaAI#430: keep the setInput/inputs rename; carry over main's recorder fix and run-in-progress toast. Rename the node card test to setInput/inputs.

Signed-off-by: Priyanshu-u07 <connect.priyanshu8271@gmail.com>
@Priyanshu-u07

Copy link
Copy Markdown
Collaborator

Merged main into this branch (b8a6418) to resolve the conflicts after #430. Your design is unchanged; only main's two fixes are carried over: the recorder fix (now on setInput) and the run-in-progress toast. Full dashboard suite: 488 passed, build passes. Please pull before pushing again.

Two follow-ups for you, not blocking:

  • Image generation Deploy uses engine: "generic", which neither worker pools nor Kubernetes can run. Added to Workflow Deploy offers engines the pools can't run (speaches, generic) #433.
  • Planner keyword rules need word boundaries: "voice assistant that helps customers withdraw cash" builds mic → image generation ("withdraw" contains "draw"), and "speech recognition" goes to the vision pipeline.

@Priyanshu-u07
Priyanshu-u07 merged commit abeea64 into InferiaAI:main Oct 24, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants