feat(rag): Implement complete RAG pipeline with reranking

[Yoo-SH]

Description

Implement a complete RAG pipeline in the Spring backend that handles embedding, indexing, search, and reranking when files are uploaded.

Architecture

File Upload → Parse → Chunk → Embed → Index → Search → Rerank → Return

Current State

• Basic embedding and indexing implemented
• Search uses hybrid BM25 + vector similarity
• No reranking stage

Required Implementation

1. Embedding Service

• Ollama integration for embeddings
• Batch embedding for performance
• Error handling and retry logic
• Embedding cache for repeated content

2. Indexing Service

• Elasticsearch indexing
• Optimize index settings for Korean content
• Bulk indexing for large documents
• Index health monitoring

3. Search Service

• Hybrid search (BM25 + vector)
• Configurable search weights
• Query expansion for better recall
• Filtering by document metadata

4. Reranking Service (NEW)

• Implement reranking algorithm
• Options to consider:
  • Cross-encoder reranking
  • LLM-based reranking
  • Custom scoring based on metadata
• Configurable reranking parameters
• Performance optimization

Implementation Tasks

• Design reranking service interface
• Choose reranking strategy
• Implement RerankerService
• Integrate reranking into search pipeline
• Add configuration options
• Performance testing and optimization
• Add monitoring and metrics
• Update API documentation

Configuration

opencontext:
  rag:
    embedding:
      batch-size: 10
      model: dengcao/Qwen3-Embedding-0.6B:F16
    search:
      top-k: 20  # Initial retrieval
      bm25-weight: 0.3
      vector-weight: 0.7
    reranking:
      enabled: true
      top-k: 5  # Final results after reranking
      strategy: cross-encoder  # or 'llm' or 'custom'

Testing Requirements

• Unit tests for each RAG component
• Integration tests for full pipeline
• Performance benchmarks
• Quality evaluation with test queries

Related Files

• core/src/main/java/com/opencontext/service/EmbeddingService.java
• core/src/main/java/com/opencontext/service/IndexingService.java
• core/src/main/java/com/opencontext/service/SearchService.java
• core/src/main/java/com/opencontext/service/RerankerService.java (new)

References

• LangChain4j RAG documentation
• Reranking strategies for RAG systems