What is OpenLIT?
OpenTelemetry-native Observability and Evaluations for LLMs and GPUs
OpenLIT is an OpenTelemetry-native tool designed to help developers gain insights into the performance of their LLM applications in production. It automatically collects LLM input and output metadata, and monitors GPU performance for self-hosted LLMs.
OpenLIT makes integrating observability into GenAI projects effortless with just a single line of code. Whether you’re working with popular LLM providers such as OpenAI and HuggingFace, or leveraging vector databases like ChromaDB, OpenLIT ensures your applications are monitored seamlessly, providing critical insights including GPU performance stats for self-hosted LLMs to improve performance and reliability.
This project proudly follows the Semantic Conventions of the OpenTelemetry community, consistently updating to align with the latest standards in observability.
What is LIT?
LIT stands for Learning and Inference Tool, which is a visual and interactive tool designed for understanding AI models and visualizing data.
The term LIT was introduced by Google.
⚡ Features
- Advanced Monitoring of LLM and VectorDB Performance: OpenLIT offers automatic instrumentation that generates traces and metrics, providing insights into the performance and costs of your LLM and VectorDB usage. This helps you analyze how your applications perform in different environments, such as production, enabling you to optimize resource ussage and scale efficiently.
- Cost Tracking for Custom and Fine-Tuned Models: OpenLIT enables you to tailor cost tracking for specific models by using a custom JSON file. This feature allows for precise budgeting and ensures that cost estimations are perfectly aligned with your project needs.
- OpenTelemetry-native & vendor-neutral SDKs: OpenLIT is built with native support for OpenTelemetry, making it blend seamlessly with your projects. This vendor-neutral approach reduces barriers to integration, making OpenLIT an intuitive part of your software stack rather than an additional complexity.
Getting Started
Select from the following guides to learn more about how to use OpenLIT:
Was this page helpful?