Senior Software Engineer – Retrieval-Augmented Generation (RAG) System
We are seeking an engineer to work with a team to build and support a healthcare centered production-scale RAG system that combines document retrieval with response generation to deliver accurate, context-aware answers. This engineer we be expected to design, implement, and operate end-to-end RAG pipelines— LLM interaction, API creation, and high-performance, secure delivery of knowledge-grounded capabilities. You will collaborate with data engineers, platform teams, and product partners to ship reliable, scalable, and observable systems.
Role and responsibilities
- Architect, implement, test, and operate end-to-end RAG workflows:
- Ingest and normalize documents from diverse sources
- Generate and manage embeddings; index and query vector databases
- Retrieve relevant passages, apply reranking or fusion strategies, and feed prompts to LLMs
- Build scalable, low-latency services and APIs (Python preferred; other languages acceptable) and ensure production-grade reliability (monitoring, tracing, alerting)
- Integrate with vector databases and embedding pipelines and optimize for latency, throughput, and cost
- Design and implement ML Ops workflows: model/version management, experiments, feature stores, CI/CD for ML-enabled services, rollback plans
- Develop robust data pipelines and governance around ingestion, provenance, quality checks, and access controls
- Collaborate with data engineers to improve retrieval quality (embedding strategies, reranking, cross-encoder models, prompt engineering) and implement evaluation metrics (precision/recall, MRR, QA accuracy, user-centric metrics)
- Implement monitoring and observability for RAG components (latency, success rate, cache hit rate, retrieval quality, data drift)
- Ensure security, privacy, and compliance (authentication, authorization, data masking, PII handling, audit logging)
- Optimize for scalability and reliability in cloud environments (AWS/GCP/Azure) and containerized deployments (Docker, Kubernetes)
- Contribute to architecture decisions, drive technical debt reduction, and mentor junior engineers
- Collaborate with product, design, and data teams to translate requirements into robust software solutions
- Document APIs, runbooks, and architectural decisions; participate in code reviews and design reviews
Required qualifications
- 5+ years of professional software engineering experience designing and delivering production systems
- Strong programming skills (Python required; NodeJs a plus)
- Deep understanding of retrieval-augmented or application-scale NLP systems and practical experience building RAG-like pipelines
- Hands-on experience with ML workflow tooling and MLOps concepts (model serving, versioning, experiments, feature stores, reproducibility)
- Proficiency with cloud infrastructure and modern software practices (AWS/GCP/Azure; Docker; Kubernetes; CI/CD)
- Strong problem-solving skills, excellent communication, and ability to work with cross-functional teams
- Familiarity with data governance, privacy, and security best practices
Preferred qualifications
- Experience with agentic workflow tools (LangGraph) and familiarity with prompt engineering for LLMs
- Exposure to working with and evaluating different LLMs
- Knowledge of evaluation methodologies for retrieval and QA systems and the ability to set up A/B tests and dashboards
- Experience with data processing frameworks (SQL, Pandas, Spark) and working with large-scale data pipelines
- Background in performance optimization for low-latency AI services (MLflow)
- Experience with monitoring and logging via New Relic, K9s, Portkey, etc
- Experience with minimizing token usage and cost optimization
- Comfortable with design and implementation of security controls for data-intensive AI systems
