Embedding Pipeline & Vector Store
Build a reusable embedding pipeline in backend/app/services/embedding.py that all four HerConnect capabilities share.
EMBEDDING SERVICE:
- Use the Nomic nomic-embed-text-v1.5 model via sentence-transformers for local embeddings (768 dimensions), with Azure OpenAI text-embedding-3-large (3072 dims) as a production alternative
- Create embed_single(text: str) -> list[float] that generates a single embedding vector
- Create embed_batch(texts: list[str]) -> list[list[float]] for bulk operations
- Create build_employee_text(employee: dict) -> str that concatenates: name, role, level, team, skills (comma-separated), aspirations, and bio into a single string for embedding
MILVUS COLLECTIONS — Define 7 collections in backend/app/db/milvus.py:
1. employee_profiles: {employee_id (VARCHAR primary key), embedding (FLOAT_VECTOR dim=768), name, role, team, level} — HNSW index, cosine metric
2. role_descriptions: {role_id, embedding, title, department}
3. documents: {doc_id, embedding, source, chunk_text, metadata} — for RAG in leave handover
4. org_updates: {update_id, embedding, update_type, content, timestamp}
5. learning_content: {content_id, embedding, title, category, duration}
6. pod_contexts: {employee_id, embedding, life_stage, challenges, interests}
7. resources: {resource_id, embedding, title, description, category}
Create a MilvusManager class with methods: ensure_collections(), upsert(collection, data), search(collection, query_vector, top_k, filter_expr), get(collection, ids). Each collection uses HNSW indexing with M=16, efConstruction=256 for high recall.
Write a populate_embeddings() function that reads all 50 employees from Neo4j, builds their text representation, embeds them, and upserts into the employee_profiles collection.