Skip to content

feat(rabitq): upgrade RaBitQ-Library to v0.3.8 and enable Windows x86_64 - #780

Open
egolearner wants to merge 4 commits into
mainfrom
feat/upgrade-rabitq-windows-support
Open

egolearner wants to merge 4 commits into
mainfrom
feat/upgrade-rabitq-windows-support

Conversation

@egolearner

Copy link
Copy Markdown
Collaborator

Summary

  • 将 vendored RaBitQ-Library 升级到 v0.3.8(0e87c6c)。此前 RaBitQ-Library-0.1
    目录里的 checkout 实际停在 v0.1.1 之前的快照。
  • 上游 v0.3.8 新增 MSVC 支持,因此放开 RABITQ_SUPPORTED 平台门槛,除 Linux x86_64
    外也接受 Windows x86_64:MSVC 分支用 /arch:AVX2、/arch:AVX512 逐文件下发指令集,
    并编入上游新增的 MASM FHT 源文件(enable_language(ASM_MASM) 必须在顶层目录调用)。
    ARM/Android/iOS/32 位仍走 stub,Windows ARM64 CI 不受影响。
  • 删除 rabitq.defer_simd_errors.patch:已被上游 VectorDB-NTU/RaBitQ-Library@9b59bf0
    收录,且是 v0.3.8 的祖先提交。
  • 将 rabitq_lut_reuse.patch 重新移植到新的 query / lookup-table API
    (Lut<T>::reset() + SplitBatchQuery<T>::reset()),ivf_rabitq_reformer.cc 的
    LUT 复用路径依赖它。
  • 把 12 处重复的平台不支持文案收敛为 rabitq_params.h 中的单个常量
    kRabitqUnsupportedPlatform;LOG_ERROR 是 printf 风格宏,故统一用 "%s" 形式传参
    以避开 -Wformat-security。
  • zvec 侧算法代码无需改动:v0.3.8 保留了所有在用的签名。

Test plan

  • ninja -C build.release 全量构建通过(Linux x86_64,GCC)

  • RABITQ_SUPPORTED=0 分支用 compile_commands.json 提取命令 + -fsyntax-only
    逐个 TU 校验(本机该分支是死代码)

  • C++ 单测:rabitq_dispatch / hnsw_rabitq_streamer / ivf_rabitq_streamer /
    index_interface / collection / schema / segment_helper 全绿

  • Python:test_collection_hnsw_rabitq test_collection_ivf_rabitq
    test_refine_query_params 46 passed

  • clang-format 无 diff;clang-tidy(新 pre-push 钩子同款参数
    --warnings-as-errors=*)对 4 个改动的 .cc exit 0、零诊断

  • gist-100k(960 维,100 查询)端到端建索引 + 召回,GT 由 Flat 精确索引暴力算出
    (Flat 自校验 100%):

    | 索引 | 参数 | R@1 | R@10 | R@100 |
    |---|---|---|---|---|
    | HNSW-RaBitQ 7-bit | ef_search=300 | 99 | 98.3 | 98.62 |
    | HNSW-RaBitQ 7-bit | ef_search=800 | 99 | 98.4 | 99.13 |
    | HNSW-RaBitQ 4-bit | ef_search=300 | 97 | 93.2 | 96.14 |
    | IVF-RaBitQ 7-bit | nprobe=32 | 96 | 97.5 | 95.19 |
    | IVF-RaBitQ 7-bit | nprobe=128 | 98 | 99.7 | 99.28 |
    
    召回随 ef_search / nprobe 单调上升,4-bit 明显低于 7-bit,说明 ex_bits 量化路径
    生效、距离核排序正确。
    
  • Windows x86_64 (MSVC) 实际编译未验证 —— 本机无 Windows 工具链,依赖
    .github/workflows/05-windows-build.yml

  • iOS / Android CI(应走 stub 分支,未实际编译)

Upstream v0.3.8 adds MSVC support, so the RABITQ_SUPPORTED gate now also
accepts Windows x86_64, driving the AVX2/AVX-512 kernels with /arch flags and
the MASM FHT source. The out-of-tree patch deferring SIMD feature errors was
upstreamed, leaving only the LUT-reuse patch, re-ported to the new query and
lookup-table API.

The unsupported-platform message becomes a single constant in rabitq_params.h
rather than 12 duplicated literals.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant