Introduction
Features
Integrations
- Overview
- LLMs
- VectorDBs
- Frameworks
- GPUs
Connections
API Reference
- Introduction
- Prompt Hub
- Vault
Privacy
Connections
New Relic
LLM Observability with New Relic and OpenLIT
1
Get your New Relic Credentials
- Go to the New Relic API Keys UI Page
- Click Create a Key Button
- In the Create an API Key Wizard:
- Set Key Type as
Ingest - License - Give the API Key Name, for example
openlitand Optionally add any Notes - Click on Create a Key
- Set Key Type as
- Copy the key by clicking on the three dots (…).
2
Add the following two lines to your application code:
Copy
import openlit
openlit.init(
otlp_endpoint="https://otlp.nr-data.net:443",
otlp_headers="api-key=YOUR_NEWRELIC_LICENSE_KEY"
)
YOUR_NEWRELIC_LICENSE_KEYwith the API Key value you copied in Step 1.- Example -
"api-key=acxxxxxxxx"
- Example -
3
Import the pre-built Dashboard
- Go to one.newrelic.com > Dashboards.
- In the top-right corner, click Import dashboard.
- Copy the dashboard JSON provided in the accordion named
Dashboardbelow. - Paste the dashboard JSON text directly into the text area.
- Choose the account and permission settings for the dashboard. You can’t change the account once you’ve set it, but you can change the permissions at any time.
- Click Import Dashboard.
Dashboard
Dashboard
Copy
{
"name": "GenAI Observability",
"description": null,
"permissions": "PUBLIC_READ_WRITE",
"pages": [
{
"name": "GenAI Observability",
"description": null,
"widgets": [
{
"title": "",
"layout": {
"column": 1,
"row": 1,
"width": 4,
"height": 2
},
"linkedEntityGuids": null,
"visualization": {
"id": "viz.markdown"
},
"rawConfiguration": {
"text": "# GenAI Observability Dashboard\nThis dashboard displays the usage of Large Language Models (LLM) and Vector Databases, tracking OpenTelemetry Traces and Metrics sent using [OpenLIT](https://github.com/openlit/openlit)."
}
},
{
"title": "",
"layout": {
"column": 5,
"row": 1,
"width": 4,
"height": 2
},
"linkedEntityGuids": null,
"visualization": {
"id": "viz.billboard"
},
"rawConfiguration": {
"facet": {
"showOtherSeries": false
},
"nrqlQueries": [
{
"accountIds": [
4060909
],
"query": "SELECT count(`gen_ai.total.requests`) AS 'Total Successful LLM Requests' FROM Metric WHERE `telemetry.sdk.name` = 'openlit'"
}
],
"platformOptions": {
"ignoreTimeRange": false
}
}
},
{
"title": "",
"layout": {
"column": 9,
"row": 1,
"width": 4,
"height": 2
},
"linkedEntityGuids": null,
"visualization": {
"id": "viz.billboard"
},
"rawConfiguration": {
"facet": {
"showOtherSeries": false
},
"nrqlQueries": [
{
"accountIds": [
4060909
],
"query": "SELECT count(`db.total.requests`) AS 'Total Successful VectorDB Requests' FROM Metric WHERE `telemetry.sdk.name` = 'openlit'"
}
],
"platformOptions": {
"ignoreTimeRange": false
}
}
},
{
"title": "",
"layout": {
"column": 1,
"row": 3,
"width": 4,
"height": 2
},
"linkedEntityGuids": null,
"visualization": {
"id": "viz.billboard"
},
"rawConfiguration": {
"dataFormatters": [],
"facet": {
"showOtherSeries": false
},
"nrqlQueries": [
{
"accountIds": [
4060909
],
"query": "SELECT average(gen_ai.usage.cost AS 'Usage Cost') FROM Span WHERE gen_ai.operation.name != 'vectordb' And gen_ai.operation.name != 'framework'"
}
],
"platformOptions": {
"ignoreTimeRange": false
}
}
},
{
"title": "",
"layout": {
"column": 5,
"row": 3,
"width": 4,
"height": 2
},
"linkedEntityGuids": null,
"visualization": {
"id": "viz.billboard"
},
"rawConfiguration": {
"dataFormatters": [],
"facet": {
"showOtherSeries": false
},
"nrqlQueries": [
{
"accountIds": [
4060909
],
"query": "SELECT sum(gen_ai.usage.cost AS 'Usage Cost') FROM Span WHERE gen_ai.operation.name != 'vectordb' And gen_ai.operation.name != 'framework'"
}
],
"platformOptions": {
"ignoreTimeRange": false
}
}
},
{
"title": "",
"layout": {
"column": 9,
"row": 3,
"width": 4,
"height": 2
},
"linkedEntityGuids": null,
"visualization": {
"id": "viz.billboard"
},
"rawConfiguration": {
"dataFormatters": [],
"facet": {
"showOtherSeries": false
},
"nrqlQueries": [
{
"accountIds": [
4060909
],
"query": "SELECT average(gen_ai.usage.total_tokens AS 'Total Tokens') FROM Span WHERE gen_ai.operation.name != 'vectordb' And gen_ai.operation.name != 'framework'"
}
],
"platformOptions": {
"ignoreTimeRange": false
}
}
},
{
"title": "Request Duration Dirstribution",
"layout": {
"column": 1,
"row": 5,
"width": 12,
"height": 3
},
"linkedEntityGuids": null,
"visualization": {
"id": "viz.histogram"
},
"rawConfiguration": {
"legend": {
"enabled": true
},
"nrqlQueries": [
{
"accountIds": [
4060909
],
"query": "SELECT histogram(duration.ms / 1000, 1, 10) AS 'Duration' FROM Span"
}
],
"platformOptions": {
"ignoreTimeRange": false
},
"yAxisLeft": {
"zero": true
}
}
},
{
"title": "Top GenAI Models by Usage",
"layout": {
"column": 1,
"row": 8,
"width": 6,
"height": 6
},
"linkedEntityGuids": null,
"visualization": {
"id": "viz.bar"
},
"rawConfiguration": {
"facet": {
"showOtherSeries": false
},
"nrqlQueries": [
{
"accountIds": [
4060909
],
"query": "SELECT count(`gen_ai.total.requests`) FROM Metric FACET `gen_ai.request.model` WHERE `telemetry.sdk.name` = 'openlit' LIMIT 10"
}
],
"platformOptions": {
"ignoreTimeRange": false
}
}
},
{
"title": "GenAI Requests by Platform",
"layout": {
"column": 7,
"row": 8,
"width": 3,
"height": 3
},
"linkedEntityGuids": null,
"visualization": {
"id": "viz.pie"
},
"rawConfiguration": {
"facet": {
"showOtherSeries": true
},
"legend": {
"enabled": true
},
"nrqlQueries": [
{
"accountIds": [
4060909
],
"query": "SELECT count(`gen_ai.total.requests`) AS 'GenAI Requests by Platform' FROM Metric FACET `gen_ai.system` WHERE `telemetry.sdk.name` = 'openlit' LIMIT 5"
}
],
"platformOptions": {
"ignoreTimeRange": false
}
}
},
{
"title": "GenAI Requests by Type",
"layout": {
"column": 10,
"row": 8,
"width": 3,
"height": 3
},
"linkedEntityGuids": null,
"visualization": {
"id": "viz.pie"
},
"rawConfiguration": {
"facet": {
"showOtherSeries": true
},
"legend": {
"enabled": true
},
"nrqlQueries": [
{
"accountIds": [
4060909
],
"query": "SELECT count(`gen_ai.total.requests`) AS 'GenAI Requests by Type' FROM Metric FACET `gen_ai.operation.name` WHERE `telemetry.sdk.name` = 'openlit' LIMIT 5"
}
],
"platformOptions": {
"ignoreTimeRange": false
}
}
},
{
"title": "GenAI Requests by Environment",
"layout": {
"column": 7,
"row": 11,
"width": 3,
"height": 3
},
"linkedEntityGuids": null,
"visualization": {
"id": "viz.pie"
},
"rawConfiguration": {
"facet": {
"showOtherSeries": true
},
"legend": {
"enabled": true
},
"nrqlQueries": [
{
"accountIds": [
4060909
],
"query": "SELECT count(`gen_ai.total.requests`) AS 'GenAI Requests by Environment' FROM Metric FACET `gen_ai.environment` WHERE `telemetry.sdk.name` = 'openlit' LIMIT 5"
}
],
"platformOptions": {
"ignoreTimeRange": false
}
}
},
{
"title": "GenAI Requests by Application",
"layout": {
"column": 10,
"row": 11,
"width": 3,
"height": 3
},
"linkedEntityGuids": null,
"visualization": {
"id": "viz.pie"
},
"rawConfiguration": {
"facet": {
"showOtherSeries": true
},
"legend": {
"enabled": true
},
"nrqlQueries": [
{
"accountIds": [
4060909
],
"query": "SELECT count(`gen_ai.total.requests`) AS 'GenAI Requests by Application' FROM Metric FACET `gen_ai.application_name` WHERE `telemetry.sdk.name` = 'openlit' LIMIT 5"
}
],
"platformOptions": {
"ignoreTimeRange": false
}
}
},
{
"title": "Prompt Tokens vs Completion Tokens vs Usage Cost",
"layout": {
"column": 1,
"row": 14,
"width": 12,
"height": 4
},
"linkedEntityGuids": null,
"visualization": {
"id": "viz.area"
},
"rawConfiguration": {
"facet": {
"showOtherSeries": false
},
"legend": {
"enabled": true
},
"nrqlQueries": [
{
"accountIds": [
4060909
],
"query": "SELECT average(`gen_ai.usage.input_tokens`) AS 'Avg Input Tokens', average(`gen_ai.usage.output_tokens`) AS 'Avg Output Tokens', average(`gen_ai.usage.cost`) AS 'Avg Usage Cost' FROM Span WHERE `telemetry.sdk.name` = 'openlit' TIMESERIES "
}
],
"platformOptions": {
"ignoreTimeRange": false
}
}
},
{
"title": "GenAI Requests",
"layout": {
"column": 1,
"row": 18,
"width": 12,
"height": 6
},
"linkedEntityGuids": null,
"visualization": {
"id": "viz.table"
},
"rawConfiguration": {
"facet": {
"showOtherSeries": false
},
"initialSorting": {
"direction": "desc",
"name": "timestamp"
},
"nrqlQueries": [
{
"accountIds": [
4060909
],
"query": "SELECT timestamp, trace.id, gen_ai.operation.name AS 'Category', gen_ai.environment AS 'Environment', gen_ai.application_name AS 'Application', gen_ai.request.model AS 'Model', gen_ai.usage.cost AS 'Usage Cost', duration.ms AS 'Duration' FROM Span WHERE gen_ai.operation.name != 'vectordb' And gen_ai.operation.name != 'framework' And telemetry.sdk.name = 'openlit'"
}
],
"platformOptions": {
"ignoreTimeRange": false
}
}
},
{
"title": "Top VectorDB Operations by Usage",
"layout": {
"column": 1,
"row": 24,
"width": 6,
"height": 3
},
"linkedEntityGuids": null,
"visualization": {
"id": "viz.bar"
},
"rawConfiguration": {
"facet": {
"showOtherSeries": false
},
"nrqlQueries": [
{
"accountIds": [
4060909
],
"query": "SELECT count(`db.total.requests`) FROM Metric FACET `db.operation` WHERE `telemetry.sdk.name` = 'openlit'"
}
],
"platformOptions": {
"ignoreTimeRange": false
}
}
},
{
"title": "DB Requests by System",
"layout": {
"column": 7,
"row": 24,
"width": 2,
"height": 3
},
"linkedEntityGuids": null,
"visualization": {
"id": "viz.pie"
},
"rawConfiguration": {
"facet": {
"showOtherSeries": true
},
"legend": {
"enabled": true
},
"nrqlQueries": [
{
"accountIds": [
4060909
],
"query": "SELECT count(`db.total.requests`) AS 'DB Requests by System' FROM Metric FACET `db.system` WHERE `telemetry.sdk.name` = 'openlit' LIMIT 5"
}
],
"platformOptions": {
"ignoreTimeRange": false
}
}
},
{
"title": "DB Requests by Environment",
"layout": {
"column": 9,
"row": 24,
"width": 2,
"height": 3
},
"linkedEntityGuids": null,
"visualization": {
"id": "viz.pie"
},
"rawConfiguration": {
"facet": {
"showOtherSeries": true
},
"legend": {
"enabled": true
},
"nrqlQueries": [
{
"accountIds": [
4060909
],
"query": "SELECT count(`db.total.requests`) AS 'DB Requests by Environment' FROM Metric FACET `gen_ai.environment` WHERE `telemetry.sdk.name` = 'openlit' LIMIT 5"
}
],
"platformOptions": {
"ignoreTimeRange": false
}
}
},
{
"title": "DB Requests by Application",
"layout": {
"column": 11,
"row": 24,
"width": 2,
"height": 3
},
"linkedEntityGuids": null,
"visualization": {
"id": "viz.pie"
},
"rawConfiguration": {
"facet": {
"showOtherSeries": true
},
"legend": {
"enabled": true
},
"nrqlQueries": [
{
"accountIds": [
4060909
],
"query": "SELECT count(`db.total.requests`) AS 'DB Requests by Application' FROM Metric FACET `gen_ai.application_name` WHERE `telemetry.sdk.name` = 'openlit' LIMIT 5"
}
],
"platformOptions": {
"ignoreTimeRange": false
}
}
},
{
"title": "VectorDB Requests",
"layout": {
"column": 1,
"row": 27,
"width": 12,
"height": 6
},
"linkedEntityGuids": null,
"visualization": {
"id": "viz.table"
},
"rawConfiguration": {
"dataFormatters": [],
"facet": {
"showOtherSeries": false
},
"initialSorting": {
"direction": "desc",
"name": "timestamp"
},
"nrqlQueries": [
{
"accountIds": [
4060909
],
"query": "SELECT timestamp, trace.id, gen_ai.environment AS 'Environment', gen_ai.application_name AS 'Application', db.system AS 'System', db.operation AS 'Operation', duration.ms AS 'Duration' FROM Span SINCE 24 hours ago WHERE gen_ai.operation.name = 'vectordb'"
}
],
"platformOptions": {
"ignoreTimeRange": false
}
}
}
]
}
],
"variables": []
}
Was this page helpful?
Assistant
Responses are generated using AI and may contain mistakes.