Observability Engineer – Grafana, Prometheus, Thanos

Roles and Responsibilities

• Design and implement observability frameworks for ML/LLM applications
• Build telemetry for AI models including latency, token usage, throughput, error rates, and SLOs
• Develop and maintain self-service observability dashboards
• Monitor model performance, data drift, reliability, and cost metrics
• Create dashboards using Grafana and Tableau for operational insights
• Implement monitoring using Prometheus and Thanos for scalable metrics collection
• Analyze time-series data and build actionable visualizations.
• Partner with ML engineers and platform teams to improve system reliability
• Define and track SLOs for AI endpoints and services
• Enable end-to-end observability using metrics, logs, and traces

Required Skills

• Strong experience in Data Analysis and Visualization
• Hands-on experience with Grafana dashboard creation
• Experience with Tableau, Grafana, Prometheus, and Thanos stack
• Strong knowledge of SQL and time-series data
• Experience with PromQL
• Working knowledge of Linux environments
• Expertise in building telemetry dashboards
• Understanding of different visualization graphs and charts
• Experience monitoring production systems and observability pipelines

Preferred Qualifications

• Experience with ML/AI observability
• Knowledge of LLM monitoring metrics (tokens, latency, hallucination tracking, etc.)
• Experience defining SLOs/SLIs
• Familiarity with distributed tracing and logging frameworks
• Experience with large-scale observability platforms

Posted By