1. Get your Credentials
- Sign in to your Elastc account.
- Open the side navigation and click on APM under Observability.
- Make sure the APM Server is running
- In the APM Agents section, Select OpenTelemetry and directly jump to Step 5 (Configure OpenTelemetry in your application):
- Copy and save the configuration value for
OTEL_EXPORTER_OTLP_ENDPOINTandOTEL_EXPORTER_OTLP_HEADERSReplace the space afterBearerwith%20: OTEL_EXPORTER_OTLP_HEADERS=“Authorization=Bearer%20[APIKEY]“
2. Instrument your application
- SDK
- CLI
For direct integration into your Python applications:Replace:Set these environment variables:Replace:Refer to the OpenLIT Python SDK repository for more advanced configurations and use cases.
- Function Arguments
- Environment Variables
import openlit
openlit.init(
otlp_endpoint="YOUR_ELASTIC_OTEL_URL",
otlp_headers="YOUR_ELASTIC_OTEL_AUTH"
)
YOUR_ELASTIC_OTEL_URLwith theOTEL_EXPORTER_OTLP_ENDPOINTvalue you copied in Step 1.- Example:
https://38xxx.apm.us-central1.gcp.cloud.es.io:443
- Example:
YOUR_ELASTIC_OTEL_AUTHwith theOTEL_EXPORTER_OTLP_HEADERSvalue you copied in Step 1.- Example:
Authorization=Bearer%20<APIKEY>
- Example:
import openlit
openlit.init()
export OTEL_EXPORTER_OTLP_ENDPOINT="YOUR_ELASTIC_OTEL_URL"
export OTEL_EXPORTER_OTLP_HEADERS="YOUR_ELASTIC_OTEL_AUTH"
YOUR_ELASTIC_OTEL_URLwith theOTEL_EXPORTER_OTLP_ENDPOINTvalue you copied in Step 1.- Example:
https://38xxx.apm.us-central1.gcp.cloud.es.io:443
- Example:
YOUR_ELASTIC_OTEL_AUTHwith theOTEL_EXPORTER_OTLP_HEADERSvalue you copied in Step 1.- Example:
Authorization=Bearer%20<APIKEY>
- Example:
For zero-code auto-instrumentation via command line:Replace:Replace:Refer to the OpenLIT Python SDK repository for more advanced configurations and use cases.
- CLI Arguments
- Environment Variables
# Using CLI arguments
openlit-instrument \
--otlp-endpoint "YOUR_ELASTIC_OTEL_URL" \
--otlp-headers "YOUR_ELASTIC_OTEL_AUTH" \
--service-name "my-ai-service" \
--deployment-environment "production" \
python app.py
YOUR_ELASTIC_OTEL_URLwith theOTEL_EXPORTER_OTLP_ENDPOINTvalue you copied in Step 1.- Example:
https://38xxx.apm.us-central1.gcp.cloud.es.io:443
- Example:
YOUR_ELASTIC_OTEL_AUTHwith theOTEL_EXPORTER_OTLP_HEADERSvalue you copied in Step 1.- Example:
Authorization=Bearer%20<APIKEY>
- Example:
# Set environment variables (takes precedence over CLI args)
export OTEL_EXPORTER_OTLP_ENDPOINT="YOUR_ELASTIC_OTEL_URL"
export OTEL_EXPORTER_OTLP_HEADERS="YOUR_ELASTIC_OTEL_AUTH"
export OTEL_SERVICE_NAME="my-ai-service"
export OTEL_DEPLOYMENT_ENVIRONMENT="production"
# Run your application
openlit-instrument python app.py
YOUR_ELASTIC_OTEL_URLwith theOTEL_EXPORTER_OTLP_ENDPOINTvalue you copied in Step 1.- Example:
https://38xxx.apm.us-central1.gcp.cloud.es.io:443
- Example:
YOUR_ELASTIC_OTEL_AUTHwith theOTEL_EXPORTER_OTLP_HEADERSvalue you copied in Step 1.- Example:
Authorization=Bearer%20<APIKEY>
- Example:
3. Import the pre-built Dashboard
- Copy the dashboard NDJSON provided in the accordion named
Dashboardbelow and save it in a file (.ndjson). - Log into your Elastic Instance.
- Go to Stack Management > Saved Objects.
- Click Import and upload your file containing the dashboard NDJSON.
- Click Import and you should have the dashboard available.
Dashboard
Dashboard
{"attributes":{"allowHidden":false,"allowNoIndex":true,"fieldAttrs":"{}","fieldFormatMap":"{\"trace.id\":{\"id\":\"url\",\"params\":{\"urlTemplate\":\"apm/link-to/trace/{{value}}\",\"labelTemplate\":\"{{value}}\"}},\"transaction.id\":{\"id\":\"url\",\"params\":{\"urlTemplate\":\"apm/link-to/transaction/{{value}}\",\"labelTemplate\":\"{{value}}\"}},\"transaction.duration.us\":{\"id\":\"duration\",\"params\":{\"inputFormat\":\"microseconds\",\"outputFormat\":\"asMilliseconds\",\"showSuffix\":true,\"useShortSuffix\":true,\"outputPrecision\":2,\"includeSpaceWithSuffix\":true}}}","fields":"[]","name":"APM","runtimeFieldMap":"{}","sourceFilters":"[]","timeFieldName":"@timestamp","title":"traces-apm*,apm-*,logs-apm*,apm-*,metrics-apm*,apm-*"},"coreMigrationVersion":"8.8.0","created_at":"2024-06-02T10:27:43.786Z","id":"apm_static_data_view_id_default","managed":false,"references":[],"type":"index-pattern","typeMigrationVersion":"8.0.0","updated_at":"2024-06-02T10:27:43.786Z","version":"WzUsMV0="}
{"attributes":{"color":"#a6a6fb","description":"","name":"genai"},"coreMigrationVersion":"8.8.0","created_at":"2024-06-02T11:00:42.608Z","id":"dd5a07f8-eb90-41fa-92e1-7e0535e92881","managed":false,"references":[],"type":"tag","typeMigrationVersion":"8.0.0","updated_at":"2024-06-02T11:00:42.608Z","version":"WzU5MiwxXQ=="}
{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"optionsJSON":"{\"useMargins\":true,\"syncColors\":false,\"syncCursor\":true,\"syncTooltips\":false,\"hidePanelTitles\":false}","panelsJSON":"[{\"type\":\"visualization\",\"gridData\":{\"x\":0,\"y\":0,\"w\":48,\"h\":9,\"i\":\"0ab1b11b-153c-4a27-a342-940e2253dacb\"},\"panelIndex\":\"0ab1b11b-153c-4a27-a342-940e2253dacb\",\"embeddableConfig\":{\"savedVis\":{\"id\":\"\",\"title\":\"\",\"description\":\"\",\"type\":\"markdown\",\"params\":{\"fontSize\":12,\"openLinksInNewTab\":false,\"markdown\":\"---\\n# GenAI Observability\\n\\nThis dashboard displays the usage of Large Language Models (LLM) and Vector Databases, tracking OpenTelemetry Traces and Metrics sent using [OpenLIT](https://github.com/openlit/openlit).\\n\\n---\"},\"uiState\":{},\"data\":{\"aggs\":[],\"searchSource\":{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}}},\"hidePanelTitles\":true,\"description\":\"\",\"enhancements\":{}},\"title\":\"\"},{\"type\":\"lens\",\"gridData\":{\"x\":0,\"y\":9,\"w\":10,\"h\":7,\"i\":\"fc9f8451-fa92-46b2-a47a-94c45a7338aa\"},\"panelIndex\":\"fc9f8451-fa92-46b2-a47a-94c45a7338aa\",\"embeddableConfig\":{\"attributes\":{\"title\":\"\",\"visualizationType\":\"lnsLegacyMetric\",\"type\":\"lens\",\"references\":[{\"type\":\"index-pattern\",\"id\":\"apm_static_data_view_id_default\",\"name\":\"indexpattern-datasource-layer-b7f3b353-d163-4371-9c4b-ae3c1a76712f\"}],\"state\":{\"visualization\":{\"layerId\":\"b7f3b353-d163-4371-9c4b-ae3c1a76712f\",\"accessor\":\"d3a9a186-1a69-4130-a4e5-d863954e517c\",\"layerType\":\"data\",\"colorMode\":\"Background\",\"palette\":{\"type\":\"palette\",\"name\":\"status\",\"params\":{\"name\":\"status\",\"reverse\":false,\"rangeType\":\"number\",\"rangeMin\":0,\"rangeMax\":1.5,\"progression\":\"fixed\",\"stops\":[{\"color\":\"#209280\",\"stop\":0},{\"color\":\"#d6bf57\",\"stop\":0.66},{\"color\":\"#cc5642\",\"stop\":1.33}],\"steps\":3,\"continuity\":\"all\",\"maxSteps\":5}}},\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filters\":[],\"datasourceStates\":{\"formBased\":{\"layers\":{\"b7f3b353-d163-4371-9c4b-ae3c1a76712f\":{\"columns\":{\"d3a9a186-1a69-4130-a4e5-d863954e517c\":{\"label\":\"Total Successful GenAI Requests\",\"dataType\":\"number\",\"operationType\":\"count\",\"isBucketed\":false,\"scale\":\"ratio\",\"sourceField\":\"gen_ai.total.requests\",\"params\":{\"format\":{\"id\":\"number\",\"params\":{\"decimals\":0}},\"emptyAsNull\":true},\"customLabel\":true}},\"columnOrder\":[\"d3a9a186-1a69-4130-a4e5-d863954e517c\"],\"incompleteColumns\":{},\"sampling\":1,\"indexPatternId\":\"apm_static_data_view_id_default\"}},\"currentIndexPatternId\":\"apm_static_data_view_id_default\"},\"indexpattern\":{\"layers\":{}},\"textBased\":{\"layers\":{}}},\"internalReferences\":[],\"adHocDataViews\":{}}},\"hidePanelTitles\":true,\"enhancements\":{}},\"title\":\"Total Successful GenAI Requests\"},{\"type\":\"lens\",\"gridData\":{\"x\":10,\"y\":9,\"w\":11,\"h\":7,\"i\":\"4230457e-9ccc-4fcd-8ae6-5942d0e86360\"},\"panelIndex\":\"4230457e-9ccc-4fcd-8ae6-5942d0e86360\",\"embeddableConfig\":{\"attributes\":{\"title\":\"\",\"visualizationType\":\"lnsLegacyMetric\",\"type\":\"lens\",\"references\":[{\"type\":\"index-pattern\",\"id\":\"apm_static_data_view_id_default\",\"name\":\"indexpattern-datasource-layer-b7f3b353-d163-4371-9c4b-ae3c1a76712f\"}],\"state\":{\"visualization\":{\"layerId\":\"b7f3b353-d163-4371-9c4b-ae3c1a76712f\",\"accessor\":\"d3a9a186-1a69-4130-a4e5-d863954e517c\",\"layerType\":\"data\",\"colorMode\":\"Background\",\"palette\":{\"type\":\"palette\",\"name\":\"status\",\"params\":{\"name\":\"status\",\"reverse\":false,\"rangeType\":\"number\",\"rangeMin\":0,\"rangeMax\":1.5,\"progression\":\"fixed\",\"stops\":[{\"color\":\"#209280\",\"stop\":0},{\"color\":\"#d6bf57\",\"stop\":0.66},{\"color\":\"#cc5642\",\"stop\":1.33}],\"steps\":3,\"continuity\":\"all\",\"maxSteps\":5}}},\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filters\":[],\"datasourceStates\":{\"formBased\":{\"layers\":{\"b7f3b353-d163-4371-9c4b-ae3c1a76712f\":{\"columns\":{\"d3a9a186-1a69-4130-a4e5-d863954e517c\":{\"label\":\"Total Successful VectorDB Requests\",\"dataType\":\"number\",\"operationType\":\"count\",\"isBucketed\":false,\"scale\":\"ratio\",\"sourceField\":\"db.total.requests\",\"params\":{\"format\":{\"id\":\"number\",\"params\":{\"decimals\":0}},\"emptyAsNull\":true},\"customLabel\":true}},\"columnOrder\":[\"d3a9a186-1a69-4130-a4e5-d863954e517c\"],\"incompleteColumns\":{},\"sampling\":1,\"indexPatternId\":\"apm_static_data_view_id_default\"}},\"currentIndexPatternId\":\"apm_static_data_view_id_default\"},\"indexpattern\":{\"layers\":{}},\"textBased\":{\"layers\":{}}},\"internalReferences\":[],\"adHocDataViews\":{}}},\"hidePanelTitles\":true,\"enhancements\":{}},\"title\":\"Total Successful GenAI Requests\"},{\"type\":\"lens\",\"gridData\":{\"x\":21,\"y\":9,\"w\":11,\"h\":7,\"i\":\"921e9588-c4f9-48f2-8842-fb94fa871789\"},\"panelIndex\":\"921e9588-c4f9-48f2-8842-fb94fa871789\",\"embeddableConfig\":{\"attributes\":{\"title\":\"\",\"visualizationType\":\"lnsLegacyMetric\",\"type\":\"lens\",\"references\":[{\"type\":\"index-pattern\",\"id\":\"apm_static_data_view_id_default\",\"name\":\"indexpattern-datasource-layer-b7f3b353-d163-4371-9c4b-ae3c1a76712f\"}],\"state\":{\"visualization\":{\"layerId\":\"b7f3b353-d163-4371-9c4b-ae3c1a76712f\",\"accessor\":\"d3a9a186-1a69-4130-a4e5-d863954e517c\",\"layerType\":\"data\",\"colorMode\":\"Background\",\"palette\":{\"type\":\"palette\",\"name\":\"status\",\"params\":{\"name\":\"status\",\"reverse\":false,\"rangeType\":\"number\",\"rangeMin\":0,\"rangeMax\":1.5,\"progression\":\"fixed\",\"stops\":[{\"color\":\"#209280\",\"stop\":0},{\"color\":\"#d6bf57\",\"stop\":0.66},{\"color\":\"#cc5642\",\"stop\":1.33}],\"steps\":3,\"continuity\":\"all\",\"maxSteps\":5}}},\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filters\":[],\"datasourceStates\":{\"formBased\":{\"layers\":{\"b7f3b353-d163-4371-9c4b-ae3c1a76712f\":{\"columns\":{\"d3a9a186-1a69-4130-a4e5-d863954e517c\":{\"label\":\"Total Usage Tokens\",\"dataType\":\"number\",\"operationType\":\"sum\",\"sourceField\":\"gen_ai.usage.total_tokens\",\"isBucketed\":false,\"scale\":\"ratio\",\"params\":{\"format\":{\"id\":\"number\",\"params\":{\"decimals\":0}},\"emptyAsNull\":true},\"customLabel\":true}},\"columnOrder\":[\"d3a9a186-1a69-4130-a4e5-d863954e517c\"],\"incompleteColumns\":{},\"sampling\":1,\"indexPatternId\":\"apm_static_data_view_id_default\"}},\"currentIndexPatternId\":\"apm_static_data_view_id_default\"},\"indexpattern\":{\"layers\":{}},\"textBased\":{\"layers\":{}}},\"internalReferences\":[],\"adHocDataViews\":{}}},\"hidePanelTitles\":true,\"enhancements\":{}},\"title\":\"Total Successful GenAI Requests\"},{\"type\":\"lens\",\"gridData\":{\"x\":32,\"y\":9,\"w\":16,\"h\":7,\"i\":\"4fc53e67-90b4-45a2-9eb2-7c7daadcff6e\"},\"panelIndex\":\"4fc53e67-90b4-45a2-9eb2-7c7daadcff6e\",\"embeddableConfig\":{\"attributes\":{\"title\":\"\",\"visualizationType\":\"lnsGauge\",\"type\":\"lens\",\"references\":[{\"type\":\"index-pattern\",\"id\":\"apm_static_data_view_id_default\",\"name\":\"indexpattern-datasource-layer-cedecfe6-b308-4a3b-8d5d-b3856f68bc5c\"}],\"state\":{\"visualization\":{\"shape\":\"horizontalBullet\",\"layerId\":\"cedecfe6-b308-4a3b-8d5d-b3856f68bc5c\",\"layerType\":\"data\",\"ticksPosition\":\"auto\",\"labelMajorMode\":\"auto\",\"metricAccessor\":\"13325455-54c7-4511-97bc-b40379cde51c\",\"palette\":{\"name\":\"custom\",\"type\":\"palette\",\"params\":{\"steps\":3,\"name\":\"custom\",\"reverse\":false,\"rangeType\":\"percent\",\"rangeMin\":null,\"rangeMax\":null,\"progression\":\"fixed\",\"stops\":[{\"color\":\"#00ff0080\",\"stop\":33.33},{\"color\":\"#eaff0080\",\"stop\":66.66},{\"color\":\"#E7664C\",\"stop\":100}],\"colorStops\":[{\"color\":\"#00ff0080\",\"stop\":null},{\"color\":\"#eaff0080\",\"stop\":33.33},{\"color\":\"#E7664C\",\"stop\":66.66}],\"continuity\":\"all\",\"maxSteps\":5}},\"colorMode\":\"palette\"},\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filters\":[],\"datasourceStates\":{\"formBased\":{\"layers\":{\"cedecfe6-b308-4a3b-8d5d-b3856f68bc5c\":{\"columns\":{\"13325455-54c7-4511-97bc-b40379cde51c\":{\"label\":\"Avg Usage Cost\",\"dataType\":\"number\",\"operationType\":\"average\",\"sourceField\":\"gen_ai.usage.cost\",\"isBucketed\":false,\"scale\":\"ratio\",\"params\":{\"emptyAsNull\":true,\"format\":{\"id\":\"number\",\"params\":{\"decimals\":1,\"suffix\":\"$\",\"compact\":false}}},\"customLabel\":true}},\"columnOrder\":[\"13325455-54c7-4511-97bc-b40379cde51c\"],\"incompleteColumns\":{},\"sampling\":1,\"indexPatternId\":\"apm_static_data_view_id_default\"}},\"currentIndexPatternId\":\"apm_static_data_view_id_default\"},\"indexpattern\":{\"layers\":{}},\"textBased\":{\"layers\":{}}},\"internalReferences\":[],\"adHocDataViews\":{}}},\"hidePanelTitles\":true,\"enhancements\":{}},\"title\":\"\"}]","timeRestore":false,"title":"GenAI Observability","version":1},"coreMigrationVersion":"8.8.0","created_at":"2024-06-03T04:45:39.460Z","id":"e6384f67-5e58-43f2-b2a2-44c16e207dbd","managed":false,"references":[{"id":"apm_static_data_view_id_default","name":"fc9f8451-fa92-46b2-a47a-94c45a7338aa:indexpattern-datasource-layer-b7f3b353-d163-4371-9c4b-ae3c1a76712f","type":"index-pattern"},{"id":"apm_static_data_view_id_default","name":"4230457e-9ccc-4fcd-8ae6-5942d0e86360:indexpattern-datasource-layer-b7f3b353-d163-4371-9c4b-ae3c1a76712f","type":"index-pattern"},{"id":"apm_static_data_view_id_default","name":"921e9588-c4f9-48f2-8842-fb94fa871789:indexpattern-datasource-layer-b7f3b353-d163-4371-9c4b-ae3c1a76712f","type":"index-pattern"},{"id":"apm_static_data_view_id_default","name":"4fc53e67-90b4-45a2-9eb2-7c7daadcff6e:indexpattern-datasource-layer-cedecfe6-b308-4a3b-8d5d-b3856f68bc5c","type":"index-pattern"},{"id":"dd5a07f8-eb90-41fa-92e1-7e0535e92881","name":"tag-ref-dd5a07f8-eb90-41fa-92e1-7e0535e92881","type":"tag"}],"type":"dashboard","typeMigrationVersion":"8.9.0","updated_at":"2024-06-03T04:45:39.460Z","version":"WzE3MSwxXQ=="}
{"excludedObjects":[],"excludedObjectsCount":0,"exportedCount":3,"missingRefCount":0,"missingReferences":[]}