Skip to content

About

No description, website, or topics provided.

Resources

Contributing

Stars

4 stars

Watchers

0 watching

Forks

Latest commit

 

History

72 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 

Repository files navigation

一份精心策划的交互模型资源列表,包括文本、语音、图像和视频交互领域的顶级文章和开源项目。

Awesome

目录


Survey (综述)

Date Title Paper Code Comment
2024-01 Agent AI: Surveying the Horizons of Multimodal Interaction Paper - 系统性综述Agent AI领域,定义多模态交互式智能体系统,涵盖视觉、语言等环境感知与具身动作生成

Benchmark (评测基准)

Date Title Paper Code Comment
2026-03 MIBench: Evaluating LMMs on Multimodal Interaction Paper - 面向多模态交互能力的综合评测基准,10000+ 视觉-文本上下文对,覆盖32个任务,从视觉/文本信息溯源和联合协同三个维度分级评估LMM的识别、理解与推理能力
2025-03 ViSpeak: Visual Instruction Feedback in Streaming Videos Paper - 提出视觉指令反馈任务,使模型从视觉内容中感知手势等指令并主动发起交互;定义 7 个关键子任务,发布 ViSpeak-Instruct 训练集与 ViSpeak-Bench 评测集,并推出达到 GPT-4o 级性能的流式视频理解模型
2024-11 StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding Paper - 首个面向 MLLM 流式视频理解能力的综合评测基准,从实时视觉理解、全源理解和上下文理解三个维度,覆盖 18 个任务、900 视频与 4500 条人工标注问答;实验表明 Gemini 1.5 Pro 和 GPT-4o 等最先进模型仍显著低于人类水平

Speech Interaction (语音交互)

Date Title Paper Code Comment
2026-07 Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs Paper - 提出 Lychee-FD 层次化声学-语义建模框架,首次揭示全双工 SLM 中模态干扰的根源(深层梯度冲突),通过参数分离与语义对齐通道实现高性能全双工交互,Spoken QA +7.4%,FullDuplexBench 1.5 +28.5%
2026-07 DuplexChat: Constructing Speaker-Separated Full-Duplex Dialogue Speech at Scale for Spoken Dialogue Language Modeling Paper Code 提出 DuplexChat-Pipe,从公共播客自动构建说话人分离的双通道全双工对话语料;发布约 41.5 万小时的 DuplexChat 数据集,覆盖 282,634 小时英语和 132,723 小时日语,并保留轮次切换、附和与重叠语音等自然对话动态
2026-07 Multimodal Voice Activity Projection for Turn-Taking in Social Robots with Voice-Activity-Related Pretrained Encoders Paper Code 将 VAP 从纯音频扩展到同步音视频输入,利用与语音活动相关的预训练编码器和跨说话人注意力预测未来轮次动态,并面向社交机器人交互进行验证
2026-06 Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models Paper - 通过强化学习直接优化停顿处理、轮次切换、附和反馈和用户打断四类交互行为,并通过语义质量奖励避免回复内容退化
2026-06 Adaptive Turn-Taking for Real-time Multi-Party Voice Agents Paper - 提出 ModeratorLM,在多方语音对话中根据助手承担的角色动态决定何时以及是否介入,通过流式 Speech LLM 联合建模轮次决策与回复生成,将 turn-taking 从固定触发规则扩展为角色条件下的主动参与策略
2026-05 Thinking-while-speaking: A Controlled, Interleaved Reasoning Method for Real-Time Speech Generation Paper - 提出 InterRS 受控交错推理方法,在实时语音生成过程中插入与表达进度对齐的推理步骤,结合交错监督微调与强化学习,在保持即时响应和自然流畅表达的同时提升数学与逻辑推理能力
2026-05 DuplexSLA: A Full-Duplex Spoken Language Model with Synchronized Speech, Language, and Action Paper Code 提出全双工 Speech-Language-Action 模型,在统一的 160 ms 时间轴上联合建模用户语音、助手语音和结构化行动,使监听、说话、规划与工具调用能够同步进行,并发布 DuplexSLA-Bench 评测轮次控制和对话中工具调用能力
2026-05 How Should LLMs Listen While Speaking? A Study of User-Stream Routing in Full-Duplex Spoken Dialogue Paper Demo 系统研究全双工语音模型中的用户流路由问题,对比通道融合与交叉注意力两种方案;前者具有更强的语义整合能力,后者能更好地保护生成上下文,并在用户打断和语音重叠时保持稳健
2026-04 Full-Duplex Interaction in Spoken Dialogue Systems: A Comprehensive Study from the ICASSP 2026 HumDial Challenge Paper Code 发布基于真实人类双通道录音的 HumDial-FDBench,覆盖用户打断、重叠语音、附和、停顿、第三方语音与动态轮次协商,并从行为正确性、对话连续性和响应延迟等维度评测全双工语音系统
2026-01 PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models Paper Code 面向实时 speech-to-speech 交互的全双工语音模型,在 Moshi 基础上加入了角色与声音控制
2025-11 Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents Paper Code 提出 conversational infill 方法,小模型即时生成响应隐藏大模型推理延迟,并流式融入推理结果,实现毫秒级首 token 延迟同时逼近前沿模型性能
2025-07 Full-Duplex-Bench v1.5: Evaluating Overlap Handling for Full-Duplex Speech Models Paper Code Full-Duplex-Bench 的扩展版本,关注重叠语音处理,覆盖用户打断、听者附和、旁路对话和环境语音等场景
2025-07 Triadic Multi-party Voice Activity Projection for Turn-taking in Spoken Dialogue Systems Paper - 首次将 VAP 从双人对话扩展到三人交互,通过联合预测多位参与者未来的语音活动实现多方轮次管理
2025-06 NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction Paper Code 提出 next-token-pair prediction,用双通道语音数据建模对话双方的同步交互
2025-05 SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model Paper Code 直接建模用户语音输入与智能体语音输出双流,支持 speech-to-speech 的全双工交互
2025-05 SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation Paper Code 提出无需向 LLM 注入音频 codec token 的独立全双工语音模型,通过流式语音编码器、统一 LLM 与流式语音合成器实现同时监听和说话,并利用显式 thinking 状态预测轮次切换、附和、回声和上下文相关打断
2025-03 Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities Paper Code 面向全双工语音对话模型的交互评测基准,关注停顿处理、附和反馈、轮次管理和用户打断等能力
2025-02 FlexDuo: A Pluggable System for Enabling Full-Duplex Capabilities in Speech Dialogue Systems Paper - 提出可插拔的全双工控制模块,将 duplex control 从语音对话系统中解耦出来
2024-11 Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM Paper Code 用冻结的文本 LLM 连接语音输入和语音输出,在保持智能性的同时实现低延迟 speech-to-speech 对话
2024-10 Moshi: a speech-text foundation model for real-time dialogue Paper Code 提出面向实时对话的 full-duplex 语音对话模型,直接建模语音到语音交互,支持低延迟响应、用户打断、重叠语音与自然轮次管理
2024-10 OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation Paper Page 通过 progressive flattening 将模态对齐、半双工学习和全双工学习统一为标准 GPT 序列建模,无需修改语言模型主干架构
2024-10 Yeah, Un, Oh: Continuous and Real-time Backchannel Prediction with Fine-tuning of Voice Activity Projection Paper Code 在连续真实对话流中预测附和反馈的时机和类型,将 backchannel 与普通轮次切换和用户打断区分开来
2024-09 Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents Paper Page 提出 SyncLLM,将现实时间信息注入自回归语言模型,使模型能够同步处理双方语流,并建模快速轮次切换、重叠语音和附和反馈
2024-08 VITA: Towards Open-Source Interactive Omni Multimodal LLM Paper Code 开源交互式 omni multimodal LLM,支持视频、图像、文本和音频的统一理解与交互
2024-08 Language Model Can Listen While Speaking Paper - 提出 listening-while-speaking language model,让模型在生成语音时同时监听用户输入
2024-05 A Full-duplex Speech Dialogue Scheme Based On Large Language Models Paper - 较早探索基于 LLM 的全双工语音对话方案,通过控制 token 决定等待、响应或打断
2024-03 Multilingual Turn-taking Prediction Using Voice Activity Projection Paper Code 将 VAP 扩展到英语、中文和日语,表明单语模型难以直接跨语言迁移,而联合训练的多语言模型可以保持稳定的轮次预测性能
2024-01 Real-time and Continuous Turn-taking Prediction Using Voice Activity Projection Paper Code 将 VAP 实现为实时连续轮次预测系统,直接从双通道对话音频预测未来语音活动,并可在 CPU 环境中实时运行
2023-05 Response-conditioned Turn-taking Prediction Paper - 将轮次预测与候选回复联合建模,系统是否接管话轮不仅取决于用户是否说完,也取决于系统准备说出的回复是否适合当前位置
2022-09 How Much Does Prosody Help Turn-taking? Investigations using Voice Activity Projection Models Paper Page 通过操纵基频、强度和频谱信息分析 VAP 的决策依据,说明模型能够隐式利用韵律线索预测轮次切换和附和位置
2022-05 Voice Activity Projection: Self-supervised Learning of Turn-taking Events Paper Code 将轮次管理建模为未来双方语音活动预测,通过自监督目标统一支持 turn shift、hold、短回应和 backchannel 等事件预测
2022-03 Generative Spoken Dialogue Language Modeling Paper Page 提出 dGSLM,在双通道原始对话语音上进行无文本建模,可联合生成双方语音、笑声和副语言信号,是双流语音对话建模的重要前身
2020-11 TurnGPT: a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialog Paper Code 利用语言上下文增量预测 turn shift,将语义和句法完整性引入轮次判断,代表从静音阈值向语义轮次预测的发展
2019-09 A Study for Improving Device-Directed Speech Detection Toward Frictionless Human-Machine Interaction Paper - Interspeech 2019,设备定向语音检测的早期研究,结合声学与文本特征提升唤醒检测精度

Video Interaction (视频交互)

Date Title Paper Code Comment
2026-08 AVA-Encoder: Towards Agent-Native Video Representation Learning Paper - 面向 Agent 的视频表征学习,通过 Agentic Video Auto-Encoder 将视频转换为知识图谱(KG)并重构回视频,使创意智能体能理解、查询和编辑结构化电影内容;较最强基线提升 20.7 个百分点,并发布电影级 KG 数据集与重建评测基准
2026-08 StreamFlow: Dynamic Memory Flows for Streaming Video Understanding Paper - 提出高效的视觉记忆框架,通过轻量的动态感知中期记忆在视觉编码前过滤时序冗余,并结合潜在长期记忆将历史视频压缩为可访问的视觉隐变量;生成时以注意力引导的检索机制按需注入相关视觉证据,StreamingBench 达 67.73%,VAS 提升 59.1%,端到端延迟与峰值内存分别降低 50.4% 和 21.1%
2026-08 StreamArena: Toward Continuous, Interactive, and Long-Horizon Agentic Streaming Video Understanding Paper - 面向小时级、交互式流式视频理解的评测基准,含 243 个全长视频(平均 88.8 分钟)与 3,646 条人工标注开放式问答,覆盖实时感知、历史回溯、主动交互与多模态工具使用;并提出两层架构 StreamMind,前端独立调度低延迟交互与主动监测,后端异步构建持久多模态记忆并执行历史回忆与外部检索
2026-07 Open-AoE: An Open Egocentric Manipulation Dataset and Toolchain for Embodied Learning Paper Code 开源以自我为中心操控数据集与工具链,含2000+小时操控视频,500+位贡献者采集,提供手部姿态、相机轨迹与原子动作标注,覆盖从手机采集到VLA/世界模型训练的全流程
2026-07 Vidu S1: A Real-Time Interactive Video Generation Model Paper Demo 实时交互视频生成,支持语音控制数字角色,540p下最高42FPS无限长度生成,消费级GPU即可运行
2026-06 LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory Paper Code 层次化记忆的主动流式视频理解,在连续视频流中自主决策何时发声
2026-06 JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence Paper Code 视觉驱动的VL交互模型,自主决策静默/响应/委托,实现实时视觉-语言交互智能
2026-06 Streaming Interventions: Can Video LLMs Correct Mistakes as They Occur? (Ego-CoMist) Paper Page 评估视频LLM在流式场景中实时纠正错误的能力,支持主动干预与交互修正
2026-06 Harnessing Streaming Video in the Wild (Streaming Harness) Paper - 利用12小时长时记忆进行自然场景流式视频理解
2026-06 LyraV (Don't Pause): Streaming Video-Language Synchrony Paper - 流式视频-语言同步,无需暂停的实时视频对话交互
2026-06 X-Stream: Multi-Stream Understanding as Multiplexers Paper Page 将多流视频理解建模为多路复用器,同时处理多路视频输入
2026-05 IPIBench: Interactive Proactive Intelligence under Continuous Streams Paper - 连续流下的交互式主动智能评测基准
2026-05 OmniInteract: Real-World Streaming Interaction Benchmark Paper Code 真实世界流式交互评测基准,考察多模态代理的主动交互能力
2026-05 OmniPro: Omni-Proactive Streaming Video Understanding Benchmark Paper - 全模态主动流式视频理解,2700样本,9子任务,84%样本需音频信号
2026-05 Omni-DuplexEval: Evaluating Real-time Duplex Omni-modal Interaction Paper - 全双工多模态实时交互评测,660视频9任务
2026-05 StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Video Paper - 将流式视频理解从被动感知推向主动决策,提出 CB-Stream Loss 缓解静默/响应监督失衡,并以多粒度奖励的 GRPO 联合优化响应正确性与时机;在 StreamPro-Bench 上达 41.5,远超此前最优 10.4,且在 StreamingBench-RTVU 上达 78.9
2026-05 VideoSeeker: Agentic Tool Invocation for Instance-level Video Paper Page 面向实例级视频的智能体工具调用,实现更精细的视频交互检索
2026-05 EgoPro-Bench: Personalized Proactive Interaction in Egocentric Video Paper - 以自我为中心视频中的个性化主动交互评测基准
2026-04 VSAS-Bench: Real-Time Evaluation of Visual Streaming Assistant Models Paper Code Apple出品,CVPR 2026 Findings,18000+标注
2026-04 AURA: Always-On Real-Time Video Understanding & Assistance Paper - 始终在线的实时视频理解与辅助助手,持续感知并主动响应用户需求
2026-03 STRIDE: Diffusion-based When-to-Speak for Streaming Video Paper Page 基于扩散模型的流式视频何时发声决策,实现自然的交互时机预测
2026-03 VST: Video Streaming Thinking — VideoLLMs Can Watch and Think Simultaneously Paper Code 边看边思考范式,相比Video-R1快15.7倍
2026-03 RIVER Bench: Real-Time Video Interaction Benchmark Paper Code 实时视频交互评测基准,评估模型在流式视频中的交互表现
2026-03 Proact-VL: Proactive VideoLLM for Real-Time AI Companions (ICML 2026) Paper - 面向实时AI伴侣的主动视频语言模型,被ICML 2026接收
2026-01 ROMA: Real-time Omni-Multimodal Assistant Paper Page 实时全模态助手,同时处理音频与视频流实现多模态交互
2025-12 OmniAgent: Audio-Guided Active Perception Agent for Omnimodal Audio-Video Understanding Paper Code 首个音频引导的主动感知代理,动态调度工具实现细粒度音视频推理
2025-12 MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learning Paper Code 通过多轮强化学习提升视频多模态大模型主动交互能力
2025-11 LiveCook: Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance? Paper - NeurIPS 2025,实时任务指导
2025-10 StreamingVLM: Real-Time Understanding for Infinite Video Streams Paper - 面向无限视频流的实时理解模型,通过复用注意力 sink、近期视觉 token 与文本 token 维护紧凑 KV 缓存,并以短重叠视频块的 SFT 对齐训练与流式推理;ICLR 2026,在 Inf-Streams-Eval 上对 GPT-4o mini 达 66.18% 胜率,单卡 H100 达 8 FPS
2025-09 StreamForest: Efficient Online Video Understanding with Persistent Event Memory Paper - 面向流式视频理解的高效架构,通过持久事件记忆森林将视频帧自适应组织为事件级树结构,结合细粒度时空窗口实现实时感知与未来预测;NeurIPS 2025 Spotlight,StreamingBench 达 77.3%,1024 token 极端压缩下仍保留 96.8% 平均精度
2025-07 ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Models Paper Code 评估视频大语言模型主动交互能力的综合基准
2025-05 StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant Paper Code Apple出品,将离线Video-LLM无缝转换为流式主动助手,memory buffer+round-decayed压缩策略支持长上下文多轮交互,解耦轻量activation模型实现持续主动响应,NeurIPS 2025
2024-11 VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Format Paper Code 提出Video-Text Duet互动格式,让VideoLLM感知说话时机以提升时序视频理解

Text Interaction (文本交互)

Date Title Paper Code Comment
2025-07 Teaching Language Models To Gather Information Proactively Paper Code 训练LLM主动提问收集信息,从被动文本生成器提升为协作思考伙伴
2025-04 Proactive Conversational Agents with Inner Thoughts Paper Code 提出 Inner Thoughts 框架,让对话代理在多方交流中形成内部想法,并在合适时机主动参与对话
2025-02 CollabLLM: From Passive Responders to Active Collaborators Paper Code 训练语言模型在多轮任务中主动理解用户意图并提出建议,从被动回答者转向主动协作者
2025-01 Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs Paper - 研究模型何时需要向用户澄清、该问什么问题,以及如何利用用户补充信息完成任务
2024-10 Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions Paper - 通过建模后续对话结果训练模型在面对歧义请求时主动提出澄清问题
2024-10 Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance Paper Code 将 LLM 代理从被动响应转向主动辅助的框架/方法
2024-09 ClarQ-LLM: A Benchmark for Models Clarifying and Requesting Information in Task-Oriented Dialog Paper Code 面向任务型对话的澄清问题评测基准,考察模型通过提问补全关键信息的能力
2024-07 Towards Human-centered Proactive Conversational Agents Paper - 从人本交互角度讨论主动对话代理,强调主动性需要兼顾用户需求、适应性与不过度打扰
2024-03 STaR-GATE: Teaching Language Models to Ask Clarifying Questions Paper Code 通过自我改进训练模型提出更有用的澄清问题,从而生成更符合用户偏好的回复
2023-10 Eliciting Human Preferences with Language Models Paper Code 提出 GATE 框架,让语言模型通过自由文本交互主动引出用户偏好和任务意图

Image Interaction (图像交互)

Date Title Paper Code Comment
2025-07 Teaching Vision-Language Models to Ask: Resolving Ambiguity in Visual Questions Paper Code 训练视觉语言模型主动提问以解决视觉问答中的歧义问题
2023-11 GLaMM: Pixel Grounding Large Multimodal Model Paper Code 将多模态对话与像素级 grounding 结合,使自然语言回答中的目标可对应到图像中像素级 mask
2023-10 Ferret: Refer and Ground Anything Anywhere at Any Granularity Paper Code 支持点、框、自由形状等任意区域输入,可围绕图中具体区域进行指代和定位
2023-08 LISA: Reasoning Segmentation via Large Language Model Paper Code 提出 reasoning segmentation任务,使模型先根据复杂或隐含的自然语言指令推理用户意图,再分割目标
2023-04 Visual Instruction Tuning Paper Code 提出 LLaVA(Large Language and Visual Assistant),通过视觉指令微调构建通用图像-文本对话助手
2023-04 Segment Anything Paper Code 提出 promptable segmentation,允许用户通过点、框、mask 等提示告诉模型要分割的目标
2023-03 Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models Paper Code 连接 ChatGPT 与视觉基础模型,支持图像问答、绘制、编辑和反馈式交互
2016-11 Visual Dialog Paper Code 提出围绕图像进行多轮问答的视觉对话任务,是图像交互研究的早期代表工作

Multimodal Interaction (多模态交互)

Date Title Paper Code Comment
2025-03 Qwen2.5-Omni Technical Report Paper Code 统一处理文本、图像、音频与视频输入,并支持文本和自然语音流式输出,推动实时多模态交互
2024-08 VITA: Towards Open-Source Interactive Omni Multimodal LLM Paper Code 面向图像、视频、文本和音频的开源交互式全模态模型,强调自然的人机多模态交互
2024-02 AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling Paper Code 用离散序列统一建模语音、文本、图像和音乐,支持多轮任意模态对话

Embodied Interaction (具身交互)

Date Title Paper Code Comment
2026-07 Open-AoE: An Open Egocentric Manipulation Dataset and Toolchain for Embodied Learning Paper Code 开源大规模第一人称操作数据集(~2000小时),500+贡献者、400+手机采集,提供MANO手部姿态、相机轨迹与原子动作标注,配套完整的可视化、跨具身重定向与VLA/World Model训练工具链

Contributing

欢迎贡献!请参阅 CONTRIBUTING.md 了解详细信息。

快速贡献步骤

  1. Fork 这个仓库
  2. 创建你的特性分支 (git checkout -b add-new-resource)
  3. 在表格中添加新资源(按时间倒序排列)
  4. 提交你的改变 (git commit -m 'Add: [Resource Title]')
  5. 推送到分支 (git push origin add-new-resource)
  6. 开启一个 Pull Request

表格字段说明

  • Date: 论文发表日期或代码发布日期 (YYYY-MM 格式)
  • Title: 论文或项目的标题
  • Paper: 论文链接 (如无则留空)
  • Code: 开源代码链接 (如无则留空)
  • Comment: 简要描述资源的主要贡献或特点

License

CC0

本项目采用 CC0 1.0 Universal License 许可证。

About

No description, website, or topics provided.

Resources

Contributing

Stars

4 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors