Elastic Cloud Managed OTLP Endpoint (mOTLP)

Serverless Observability Serverless Security Stack Planned ECH

The Elastic Cloud Managed OTLP Endpoint allows you to send OpenTelemetry data directly to Elastic Cloud using the OTLP protocol, with Elastic handling scaling, data processing, and storage. The Managed OTLP endpoint can act like a Gateway Collector, so that you can point your OpenTelemetry SDKs or Collectors to it.

This diagram shows data ingest using Elastic Distribution of OpenTelemetry and the Elastic Cloud Managed OTLP Endpoint:

×

Telemetry is stored in Elastic in OTLP format, preserving resource attributes and original semantic conventions. If no specific dataset or namespace is provided, the data streams are: traces-generic.otel-default, metrics-generic.otel-default, and logs-generic.otel-default.

For a detailed comparison of how OTel data streams differ from classic Elastic APM data streams, refer to OTel data streams compared to classic APM.

To use the Elastic Cloud Elastic Cloud Managed OTLP Endpoint you need the following:

• An Elastic Cloud Serverless project or an Elastic Cloud Hosted (ECH) deployment.
• An OTLP-compliant shipper capable of forwarding logs, metrics, or traces in OTLP format. This can include:
  • OpenTelemetry Collector (EDOT, Contrib, or other distributions)
  • OpenTelemetry SDKs (EDOT, upstream, or other distributions)
  • EDOT Cloud Forwarder
  • Any other forwarder that supports the OTLP protocol.

To send data to Elastic through the Elastic Cloud Managed OTLP Endpoint, follow the Send data to the Elastic Cloud Managed OTLP Endpoint quickstart.

To retrieve your Elastic Cloud Managed OTLP Endpoint endpoint address, follow these steps:

To configure OpenTelemetry SDKs to send data directly to the Elastic Cloud Managed OTLP Endpoint, set the OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS environment variable.

For example:

export OTEL_EXPORTER_OTLP_ENDPOINT="https://<motlp-endpoint>"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey <key>"
		

You can route logs to dedicated datasets by setting the data_stream.dataset attribute to the log record. This attribute is used to route the log to the corresponding dataset.

For example, if you want to route the EDOT Cloud Forwarder logs to custom datasets, you can add the following attributes to the log records:

processors:
  transform:
    log_statements:
      - set(log.attributes["data_stream.dataset"], "aws.cloudtrail") where log.attributes["aws.cloudtrail.event_id"] != nil
		

You can also set the OTEL_RESOURCE_ATTRIBUTES environment variable to set the data_stream.dataset attribute for all logs. For example:

export OTEL_RESOURCE_ATTRIBUTES="data_stream.dataset=app.orders"
		

The Elastic Cloud Managed OTLP Endpoint endpoint is designed to be highly available and resilient. However, there are some scenarios where data might be lost or not sent completely. The Failure store is a mechanism that allows you to recover from these scenarios.

The Failure store is always enabled for Elastic Cloud Managed OTLP Endpoint data streams. This prevents ingest pipeline exceptions and conflicts with data stream mappings. Failed documents are stored in a separate index. You can view the failed documents from the Data Set Quality page. Refer to Data set quality.

The following limitations apply when using the Elastic Cloud Managed OTLP Endpoint:

• Tail-based sampling (TBS) is not available.
• Universal Profiling is not available.
• Only supports histograms with delta temporality. Cumulative histograms are dropped.
• Latency distributions based on histogram values have limited precision due to the fixed boundaries of explicit bucket histograms.

For more information on billing, refer to Elastic Cloud pricing.

Requests to the Elastic Cloud Managed OTLP Endpoint are subject to rate limiting and throttling. If you exceed your Elasticsearch capacity in Elastic Cloud Hosted, or send data at a rate that exceeds the limits, your requests might be rejected.

The following rate limits and burst limits apply:

As long as your data ingestion rate stays at or below the rate limit and burst limit, your requests are accepted.

If you send data that exceeds the available limits, the Elastic Cloud Managed OTLP Endpoint responds with an HTTP 429 Too Many Requests status code. A log message similar to this appears in the OpenTelemetry Collector's output:

{
  "code": 8,
  "message": "error exporting items, request to <ingest endpoint> responded with HTTP Status Code 429"
}
		

After your sending rate goes back to the allowed limit, the system automatically begins accepting requests again.

Depending on the reason for the rate limiting, you can either increase your Elasticsearch capacity or request higher limits.

If data intake exceeds the capacity of Elasticsearch in your Elastic Cloud Hosted deployment, you might get rate limiting errors. To solve this issue, scale or resize your deployment:

• Scaling considerations
• Resize deployment
• Autoscaling in ECE and ECH

If rate limiting is not caused by Elasticsearch capacity or you're on Elastic Cloud Serverless, you can either decrease data volume or request higher limits.

To increase the rate limit, reach out to Elastic Support.