problem-framing(extract-worker): document embedding

[ClemDoum]

Precisely define/scope semantic vectors usages inside DS

• similar doc search (MLT vector)
• hybrid search
• RAG

Inputs

• document in markdown formats
• embedding configuration

Outputs

• document embedded as chunks stored in a vector DB

Success metrics (TBD)

Element to help us define success:

• what storage and vector size can we afford ?
• review embedding leaderboard: https://huggingface.co/spaces/mteb/leaderboard
• where do we want to store vectors (storage ES ?) ?

[ClemDoum]

Potential priority for sprint 50: P1