﻿---
title: Prometheus Integration
description: This integration can collect metrics from: Prometheus Exporters (Collectors), Prometheus Server Remote-Write, Prometheus Queries (PromQL). The Prometheus...
url: https://www.elastic.co/docs/reference/integrations/prometheus
---

# Prometheus Integration


|                                                                                                                                     |                                         |
|-------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|
| **Version**Version                                                                                                                  | 1.24.2 ([View all](#changelog)View all) |
| **Subscription level**Subscription level[What's this?](https://www.elastic.co/subscriptions)What's this?                            | Basic                                   |
| **Level of support**Level of support[What's this?](https://www.elastic.co/docs/reference/integrations/level-of-support)What's this? | Elastic                                 |
| **Ingestion method(s)**Ingestion method(s)                                                                                          | Prometheus                              |

This integration can collect metrics from:
- *Prometheus Exporters (Collectors)*
- *Prometheus Server Remote-Write*
- *Prometheus Queries (PromQL)*


## Metrics


### Prometheus Exporters (Collectors)

The Prometheus integration `collector` dataset connects to the Prometheus server and pulls metrics using either the `/metrics` endpoint or the [Prometheus Federation API](https://prometheus.io/docs/prometheus/latest/federation/).

#### Scraping from a Prometheus exporter

To scrape metrics from a Prometheus exporter, configure the `hosts` setting to it. The path
to retrieve the metrics from (`/metrics` by default) can be configured with Metrics Path.

#### Histograms and types

`Use Types` parameter (default: true) enables a different layout for metrics storage, leveraging Elasticsearch
types, including [histograms](https://www.elastic.co/guide/en/elasticsearch/reference/current/histogram.html).
`Rate Counters` parameter (default: true) enables calculating a rate out of Prometheus counters. When enabled, Elastic Agent stores
the counter increment since the last collection. This metric should make some aggregations easier and with better
performance. This parameter can only be enabled in combination with `Use Types`.
When `Use Types` and `Rate Counters` are enabled, metrics are stored like this:
```json
{
  "_index": ".ds-metrics-prometheus.collector-default-000001",
  "_id": "JlK9AHMBeyDc0b9rCwVA",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2020-06-29T15:40:55.028Z",
    "prometheus": {
      "labels": {
        "slice": "inner_eval",
        "instance": "localhost:9090",
        "job": "prometheus"
      },
      "prometheus_engine_query_duration_seconds_sum": {
        "counter": 0.002697546,
        "rate": 0.00006945900000000001
      },
      "prometheus_engine_query_duration_seconds_count": {
        "rate": 1,
        "counter": 37
      }
    },
    "dataset": {
      "type": "metrics",
      "name": "prometheus.collector",
      "namespace": "default"
    },
    "agent": {
      "ephemeral_id": "98420e91-ee6d-4883-8ad3-02fa8d47f5c1",
      "id": "9fc3e975-6789-4738-a11a-ba7108b0a92c",
      "name": "minikube",
      "type": "metricbeat",
      "version": "8.0.0"
    },
    "ecs": {
      "version": "1.5.0"
    },
    "event": {
      "module": "prometheus",
      "duration": 15397122,
      "dataset": "prometheus.collector"
    },
    "metricset": {
      "period": 10000,
      "name": "collector"
    },
    "service": {
      "address": "localhost:9090",
      "type": "prometheus"
    },
    "stream": {
      "namespace": "default",
      "type": "metrics",
      "dataset": "prometheus.collector"
    },
    "host": {},
  },
  "fields": {
    "@timestamp": [
      "2020-06-29T15:40:55.028Z"
    ]
  },
  "highlight": {
    "event.dataset": [
      "@kibana-highlighted-field@prometheus.collector@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1593445255028
  ]
}
```


#### Metrics count

The Prometheus integration's `collector` dataset provides a `Metrics Count` parameter, which is disabled by default. When enabled, it counts the total number of Prometheus metrics within each Elasticsearch document. This count is stored in a field called `metrics_count` and its value is calculated prior to any enrichments by Ingest Pipelines or Agent Processors, ensuring consistency. This field name is reserved for internal use and must not be altered using Agent Processors or Ingest Pipelines.

#### Scraping all metrics from a Prometheus server

We recommend using the Remote Write dataset for this, and make Prometheus push metrics to Agent.

#### Filtering metrics

In order to filter out/in metrics one can make use of `Metrics Filters Include`, `Metrics Filters Exclude` settings:
```yml
Metrics Filters Include: ["node_filesystem_*"]
Metrics Filters Exclude: ["node_filesystem_device_*"]
```

The configuration above will include only metrics that match `node_filesystem_*` pattern and do not match `node_filesystem_device_*`.
To keep only specific metrics, anchor the start and the end of the regexp of each metric:
- the caret ^ matches the beginning of a text or line,
- the dollar sign $ matches the end of a text.

```yml
Metrics Filters Include: ["^node_network_net_dev_group$", "^node_network_up$"]
```

<dropdown title="Example">

  ```json
  {
      "@timestamp": "2024-08-20T08:38:11.185Z",
      "agent": {
          "ephemeral_id": "b9fad797-a22c-47be-b2f4-44c0a89b6c25",
          "id": "9822f27e-ae7c-4cee-98af-094356f8bf91",
          "name": "elastic-agent-35087",
          "type": "metricbeat",
          "version": "8.14.0"
      },
      "data_stream": {
          "dataset": "prometheus.collector",
          "namespace": "52976",
          "type": "metrics"
      },
      "ecs": {
          "version": "8.0.0"
      },
      "elastic_agent": {
          "id": "9822f27e-ae7c-4cee-98af-094356f8bf91",
          "snapshot": false,
          "version": "8.14.0"
      },
      "event": {
          "agent_id_status": "verified",
          "dataset": "prometheus.collector",
          "duration": 1958134070,
          "ingested": "2024-08-20T08:38:13Z",
          "module": "prometheus"
      },
      "host": {
          "architecture": "x86_64",
          "containerized": true,
          "hostname": "elastic-agent-35087",
          "id": "345c85cf1fe945e2b19719b370c09a48",
          "ip": [
              "192.168.241.8",
              "192.168.242.2"
          ],
          "mac": [
              "02-42-C0-A8-F1-08",
              "02-42-C0-A8-F2-02"
          ],
          "name": "elastic-agent-35087",
          "os": {
              "codename": "focal",
              "family": "debian",
              "kernel": "5.4.0-189-generic",
              "name": "Ubuntu",
              "platform": "ubuntu",
              "type": "linux",
              "version": "20.04.6 LTS (Focal Fossa)"
          }
      },
      "metricset": {
          "name": "collector",
          "period": 10000
      },
      "prometheus": {
          "labels": {
              "dialer_name": "alertmanager",
              "instance": "svc-prometheus:9090",
              "job": "prometheus"
          },
          "labels_fingerprint": "jn10I8M3W8CSQq1v0nbhVyegvgQ=",
          "net_conntrack_dialer_conn_attempted_total": {
              "counter": 0,
              "rate": 0
          },
          "net_conntrack_dialer_conn_closed_total": {
              "counter": 0,
              "rate": 0
          },
          "net_conntrack_dialer_conn_established_total": {
              "counter": 0,
              "rate": 0
          }
      },
      "service": {
          "address": "http://svc-prometheus:9090/metrics",
          "type": "prometheus"
      }
  }
  ```
</dropdown>

The fields reported are:
**ECS Field Reference**
Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.
<dropdown title="Exported fields">


  | Field                         | Description                                                                                                                                                                                                                                                                                            | Type             | Metric Type |
  |-------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|-------------|
  | @timestamp                    | Event timestamp.                                                                                                                                                                                                                                                                                       | date             |             |
  | agent.id                      | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.                                                                                                                                                                                                             | keyword          |             |
  | cloud.account.id              | The cloud account or organization id used to identify different entities in a multi-tenant environment.  Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.                                                                                                                    | keyword          |             |
  | cloud.availability_zone       | Availability zone in which this host is running.                                                                                                                                                                                                                                                       | keyword          |             |
  | cloud.image.id                | Image ID for the cloud instance.                                                                                                                                                                                                                                                                       | keyword          |             |
  | cloud.instance.id             | Instance ID of the host machine.                                                                                                                                                                                                                                                                       | keyword          |             |
  | cloud.provider                | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.                                                                                                                                                                                                                       | keyword          |             |
  | cloud.region                  | Region in which this host is running.                                                                                                                                                                                                                                                                  | keyword          |             |
  | container.id                  | Unique container id.                                                                                                                                                                                                                                                                                   | keyword          |             |
  | data_stream.dataset           | Data stream dataset.                                                                                                                                                                                                                                                                                   | constant_keyword |             |
  | data_stream.namespace         | Data stream namespace.                                                                                                                                                                                                                                                                                 | constant_keyword |             |
  | data_stream.type              | Data stream type.                                                                                                                                                                                                                                                                                      | constant_keyword |             |
  | event.module                  | Event module.                                                                                                                                                                                                                                                                                          | constant_keyword |             |
  | host.containerized            | If the host is a container.                                                                                                                                                                                                                                                                            | boolean          |             |
  | host.name                     | Name of the host.  It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.                                                                                                                    | keyword          |             |
  | host.os.build                 | OS build information.                                                                                                                                                                                                                                                                                  | keyword          |             |
  | host.os.codename              | OS codename, if any.                                                                                                                                                                                                                                                                                   | keyword          |             |
  | metrics_count                 | Total count of Prometheus metrics within the Elasticsearch document. This value is calculated prior to any enrichments by Ingest Pipelines or Agent Processors, ensuring consistency. This field name is reserved for internal use and must not be altered using Agent Processors or Ingest Pipelines. | long             |             |
  | prometheus.*.counter          | Prometheus counter metric                                                                                                                                                                                                                                                                              | object           | counter     |
  | prometheus.*.histogram        | Prometheus histogram metric                                                                                                                                                                                                                                                                            | object           |             |
  | prometheus.*.rate             | Prometheus rated counter metric                                                                                                                                                                                                                                                                        | object           | gauge       |
  | prometheus.*.value            | Prometheus gauge metric                                                                                                                                                                                                                                                                                | object           | gauge       |
  | prometheus.labels.*           | Prometheus metric labels.                                                                                                                                                                                                                                                                              | keyword          |             |
  | prometheus.labels_fingerprint | Autogenerated ID representing the fingerprint of labels object and includes query name.                                                                                                                                                                                                                | keyword          |             |
  | prometheus.metrics.*          | Prometheus metric                                                                                                                                                                                                                                                                                      | object           | gauge       |
  | service.address               | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets).                                                                                                                                               | keyword          |             |
</dropdown>


### Prometheus Server Remote-Write

The Prometheus `remote_write` can receive metrics from a Prometheus server that has configured [remote_write](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote%5Fwrite)
setting accordingly, for instance:
```yml
remote_write:
  - url: "http://localhost:9201/write"
```

In Kubernetes, a Service resource should also be created:
```yml
---
apiVersion: v1
kind: Service
metadata:
  name: elastic-agent
  namespace: kube-system
  labels:
    app: elastic-agent
spec:
  ports:
    - port: 9201
      protocol: TCP
      targetPort: 9201
  selector:
    app: elastic-agent
  sessionAffinity: None
  type: ClusterIP
```

This Service can be used as a `remote_write.url` in Prometheus configuration:
```yml
remote_write:
  - url: "http://elastic-agent.kube-system:9201/write"
```

<tip>
  In order to assure the health of the whole queue, the following configuration [parameters](https://prometheus.io/docs/practices/remote%5Fwrite/#parameters) should be considered:
</tip>

- `max_shards`: Sets the maximum number of parallelism with which Prometheus will try to send samples to Elastic Agent.
  It is recommended that this setting should be equal to the number of cores of the machine where Elastic Agent runs.
  Elastic Agent can handle connections in parallel and hence setting `max_shards` to the number of parallelism that
  Elastic Agent can actually achieve is the optimal queue configuration.
- `max_samples_per_send`: Sets the number of samples to batch together for each send. Recommended values are
  between 100 (default) and 1000. Having a bigger batch can lead to improved throughput and in more efficient
  storage since Elastic Agent groups metrics with the same labels into same event documents.
  However this will increase the memory usage of Elastic Agent.
- `capacity`: It is recommended to set capacity to 3-5 times `max_samples_per_send`.
  Capacity sets the number of samples that are queued in memory per shard, and hence capacity should be high enough so as to
  be able to cover `max_samples_per_send`.

<tip>
  To limit amount of samples that are sent by the Prometheus Server can be used [`write_relabel_configs`](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote%5Fwrite)
  configuration. It is a relabeling, that applies to samples before sending them to the remote endpoint. Example:
</tip>

```
remote_write:
  - url: "http://localhost:9201/write"
    write_relabel_configs:
      - source_labels: [job]
        regex: 'prometheus'
        action: keep
```

Metrics sent to the http endpoint will be put by default under the `prometheus.` prefix with their labels under `prometheus.labels`.
A basic configuration would look like:
```yml
host: "localhost"
port: "9201"
```

Also consider using secure settings for the server, configuring the module with TLS/SSL as shown:
```yml
host: "localhost"
ssl.certificate: "/etc/pki/server/cert.pem"
ssl.key: "/etc/pki/server/cert.key"
port: "9201"
```

and on Prometheus side:
```yml
remote_write:
  - url: "https://localhost:9201/write"
    tls_config:
        cert_file: "/etc/prometheus/my_key.pem"
        key_file: "/etc/prometheus/my_key.key"
        # Disable validation of the server certificate.
        #insecure_skip_verify: true
```

<dropdown title="Example">

  ```json
  {
      "@timestamp": "2022-09-22T12:23:35.757Z",
      "agent": {
          "ephemeral_id": "5c3d912b-9bf3-4747-b784-1f7c275a5979",
          "id": "af0df4c2-33b7-41fd-8eb5-573376996db2",
          "name": "kind-control-plane",
          "type": "metricbeat",
          "version": "8.4.0"
      },
      "data_stream": {
          "dataset": "prometheus.remote_write",
          "namespace": "default",
          "type": "metrics"
      },
      "ecs": {
          "version": "8.11.0"
      },
      "elastic_agent": {
          "id": "af0df4c2-33b7-41fd-8eb5-573376996db2",
          "snapshot": true,
          "version": "8.4.0"
      },
      "event": {
          "agent_id_status": "verified",
          "dataset": "prometheus.remote_write",
          "ingested": "2022-09-22T12:24:16Z",
          "module": "prometheus"
      },
      "host": {},
      "metricset": {
          "name": "remote_write"
      },
      "prometheus": {
          "labels": {
              "app": "prometheus",
              "app_kubernetes_io_managed_by": "Helm",
              "chart": "prometheus-15.10.1",
              "component": "node-exporter",
              "cpu": "5",
              "heritage": "Helm",
              "instance": "172.19.0.2:9100",
              "job": "kubernetes-service-endpoints",
              "mode": "user",
              "namespace": "kube-system",
              "node": "kind-control-plane",
              "release": "prometheus-server",
              "service": "prometheus-server-node-exporter"
          },
          "node_cpu_guest_seconds_total": {
              "counter": 0,
              "rate": 0
          },
          "node_cpu_seconds_total": {
              "counter": 2284.68,
              "rate": 0
          }
      },
      "service": {
          "type": "prometheus"
      }
  }
  ```
</dropdown>

The fields reported are:
**ECS Field Reference**
Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.
<dropdown title="Exported fields">


  | Field                                | Description                                                                                                                                                                                                                                                                                            | Type             | Metric Type |
  |--------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|-------------|
  | @timestamp                           | Event timestamp.                                                                                                                                                                                                                                                                                       | date             |             |
  | agent.id                             | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.                                                                                                                                                                                                             | keyword          |             |
  | cloud.account.id                     | The cloud account or organization id used to identify different entities in a multi-tenant environment.  Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.                                                                                                                    | keyword          |             |
  | cloud.availability_zone              | Availability zone in which this host is running.                                                                                                                                                                                                                                                       | keyword          |             |
  | cloud.image.id                       | Image ID for the cloud instance.                                                                                                                                                                                                                                                                       | keyword          |             |
  | cloud.instance.id                    | Instance ID of the host machine.                                                                                                                                                                                                                                                                       | keyword          |             |
  | cloud.provider                       | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.                                                                                                                                                                                                                       | keyword          |             |
  | cloud.region                         | Region in which this host is running.                                                                                                                                                                                                                                                                  | keyword          |             |
  | container.id                         | Unique container id.                                                                                                                                                                                                                                                                                   | keyword          |             |
  | data_stream.dataset                  | Data stream dataset.                                                                                                                                                                                                                                                                                   | constant_keyword |             |
  | data_stream.namespace                | Data stream namespace.                                                                                                                                                                                                                                                                                 | constant_keyword |             |
  | data_stream.type                     | Data stream type.                                                                                                                                                                                                                                                                                      | constant_keyword |             |
  | event.module                         | Event module.                                                                                                                                                                                                                                                                                          | constant_keyword |             |
  | host.containerized                   | If the host is a container.                                                                                                                                                                                                                                                                            | boolean          |             |
  | host.name                            | Name of the host.  It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.                                                                                                                    | keyword          |             |
  | host.os.build                        | OS build information.                                                                                                                                                                                                                                                                                  | keyword          |             |
  | host.os.codename                     | OS codename, if any.                                                                                                                                                                                                                                                                                   | keyword          |             |
  | metrics_count                        | Total count of Prometheus metrics within the Elasticsearch document. This value is calculated prior to any enrichments by Ingest Pipelines or Agent Processors, ensuring consistency. This field name is reserved for internal use and must not be altered using Agent Processors or Ingest Pipelines. | long             |             |
  | prometheus.*.counter                 | Prometheus counter metric                                                                                                                                                                                                                                                                              | object           | counter     |
  | prometheus.*.histogram               | Prometheus histogram metric                                                                                                                                                                                                                                                                            | object           |             |
  | prometheus.*.rate                    | Prometheus rated counter metric                                                                                                                                                                                                                                                                        | object           | gauge       |
  | prometheus.*.value                   | Prometheus gauge metric                                                                                                                                                                                                                                                                                | object           | gauge       |
  | prometheus.labels.*                  | Prometheus metric labels.                                                                                                                                                                                                                                                                              | keyword          |             |
  | prometheus.labels_fingerprint        | Autogenerated ID representing the fingerprint of labels object and includes query name.                                                                                                                                                                                                                | keyword          |             |
  | prometheus.metrics.*                 | Prometheus metric                                                                                                                                                                                                                                                                                      | object           | gauge       |
  | prometheus.metrics_names_fingerprint | Autogenerated ID representing the fingerprint of the list of metrics names                                                                                                                                                                                                                             | keyword          |             |
</dropdown>


#### Histograms and types

`use_types` parameter (default: true) enables a different layout for metrics storage, leveraging Elasticsearch
types, including [histograms](https://www.elastic.co/guide/en/elasticsearch/reference/current/histogram.html).
`rate_counters` parameter (default: true) enables calculating a rate out of Prometheus counters. When enabled, Elastic Agent stores
the counter increment since the last collection. This metric should make some aggregations easier and with better
performance. This parameter can only be enabled in combination with `use_types`.
`period` parameter (default: 60s) configures the timeout of internal cache, which stores counter values in order to calculate rates between consecutive fetches. The parameter will be validated and all values lower than 60sec will be reset to the default value.
Note that by default prometheus pushes data with the interval of 60s (in remote write). In case that prometheus push rate is changed, the `period` parameter needs to be configured accordingly.
When `use_types` and `rate_counters` are enabled, metrics are stored like this:
```json
{
    "prometheus": {
        "labels": {
            "instance": "172.27.0.2:9090",
            "job": "prometheus"
        },
        "prometheus_target_interval_length_seconds_count": {
            "counter": 1,
            "rate": 0
        },
        "prometheus_target_interval_length_seconds_sum": {
            "counter": 15.000401344,
            "rate": 0
        }
        "prometheus_tsdb_compaction_chunk_range_seconds_bucket": {
            "histogram": {
                "values": [50, 300, 1000, 4000, 16000],
                "counts": [10, 2, 34, 7]
            }
        }
    },
}
```


#### Types' patterns

Unlike `collector` metricset, `remote_write` receives metrics in raw format from the prometheus server.
In this, the module has to internally use a heuristic in order to identify efficiently the type of each raw metric.
For these purpose some name patterns are used in order to identify the type of each metric.
The default patterns are the following:
. `_total` suffix: the metric is of Counter type
. `_sum` suffix: the metric is of Counter type
. `_count` suffix: the metric is of Counter type
. `_bucket` suffix and `le` in labels: the metric is of Histogram type
Everything else is handled as a Gauge. In addition there is no special handling for Summaries so it is expected that
Summary's quantiles are handled as Gauges and Summary's sum and count as Counters.
Users have the flexibility to add their own patterns using the following configuration:
```yml
types_patterns:
    counter_patterns: ["_my_counter_suffix"]
    histogram_patterns: ["_my_histogram_suffix"]
```

The configuration above will consider metrics with names that match `_my_counter_suffix` as Counters
and those that match `_my_histogram_suffix` (and have `le` in their labels) as Histograms.
To match only specific metrics, anchor the start and the end of the regexp of each metric:
- the caret `^` matches the beginning of a text or line,
- the dollar sign `$` matches the end of a text.

```yml
types_patterns:
    histogram_patterns: ["^my_histogram_metric$"]
```

Note that when using `types_patterns`, the provided patterns have higher priority than the default patterns.
For instance if `_histogram_total` is a defined histogram pattern, then a metric like `network_bytes_histogram_total`
will be handled as a histogram, even if it has the suffix `_total` which is a default pattern for counters.

#### Metrics count

The Prometheus integration's `remote_write` dataset provides a `Metrics Count` parameter, which is disabled by default. When enabled, it counts the total number of Prometheus metrics within each Elasticsearch document. This count is stored in a field called `metrics_count` and its value is calculated prior to any enrichments by Ingest Pipelines or Agent Processors, ensuring consistency. This field name is reserved for internal use and must not be altered using Agent Processors or Ingest Pipelines.

### Prometheus Queries (PromQL)

The Prometheus `query` dataset executes specific Prometheus queries against [Promethes Query API](https://prometheus.io/docs/prometheus/latest/querying/api/#expression-queries).

#### Instant queries

The following configuration performs an instant query for `up` metric at a single point in time:
```yml
queries:
- name: 'up'
  path: '/api/v1/query'
  params:
    query: "up"
```

More complex PromQL expressions can also be used like the following one which calculates the per-second rate of HTTP
requests as measured over the last 5 minutes.
```yml
queries:
- name: "rate_http_requests_total"
  path: "/api/v1/query"
  params:
    query: "rate(prometheus_http_requests_total[5m])"
```


#### Range queries

The following example evaluates the expression `up` over a 30-second range with a query resolution of 15 seconds:
```yml
queries:
- name: "up_master"
  path: "/api/v1/query_range"
  params:
    query: "up{node='master01'}"
    start: "2019-12-20T23:30:30.000Z"
    end: "2019-12-21T23:31:00.000Z"
    step: 15s
```

<dropdown title="Example">

  ```json
  {
      "@timestamp": "2024-08-20T08:39:07.000Z",
      "agent": {
          "ephemeral_id": "cc18c40d-dcb8-4192-aede-e988d68c376c",
          "id": "842b000b-c1bd-4608-bbd8-2a1849afc2f5",
          "name": "elastic-agent-31805",
          "type": "metricbeat",
          "version": "8.14.0"
      },
      "data_stream": {
          "dataset": "prometheus.query",
          "namespace": "54564",
          "type": "metrics"
      },
      "ecs": {
          "version": "8.0.0"
      },
      "elastic_agent": {
          "id": "842b000b-c1bd-4608-bbd8-2a1849afc2f5",
          "snapshot": false,
          "version": "8.14.0"
      },
      "event": {
          "agent_id_status": "verified",
          "dataset": "prometheus.query",
          "duration": 6078736,
          "ingested": "2024-08-20T08:39:10Z",
          "module": "prometheus"
      },
      "host": {
          "architecture": "x86_64",
          "containerized": true,
          "hostname": "elastic-agent-31805",
          "id": "345c85cf1fe945e2b19719b370c09a48",
          "ip": [
              "192.168.241.8",
              "192.168.242.2"
          ],
          "mac": [
              "02-42-C0-A8-F1-08",
              "02-42-C0-A8-F2-02"
          ],
          "name": "elastic-agent-31805",
          "os": {
              "codename": "focal",
              "family": "debian",
              "kernel": "5.4.0-189-generic",
              "name": "Ubuntu",
              "platform": "ubuntu",
              "type": "linux",
              "version": "20.04.6 LTS (Focal Fossa)"
          }
      },
      "metricset": {
          "name": "query",
          "period": 10000
      },
      "prometheus": {
          "labels": {
              "query_name": "scalar"
          },
          "labels_fingerprint": "uE8iX47vrW1H38mLYMD73p8/CcA=",
          "query": {
              "scalar": 100
          }
      },
      "service": {
          "address": "http://svc-prometheus:9090",
          "type": "prometheus"
      }
  }
  ```
</dropdown>

The fields reported are:
**ECS Field Reference**
Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.
<dropdown title="Exported fields">


  | Field                         | Description                                                                                                                                                                         | Type             | Metric Type |
  |-------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|-------------|
  | @timestamp                    | Event timestamp.                                                                                                                                                                    | date             |             |
  | agent.id                      | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.                                                                                          | keyword          |             |
  | cloud.account.id              | The cloud account or organization id used to identify different entities in a multi-tenant environment.  Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword          |             |
  | cloud.availability_zone       | Availability zone in which this host is running.                                                                                                                                    | keyword          |             |
  | cloud.image.id                | Image ID for the cloud instance.                                                                                                                                                    | keyword          |             |
  | cloud.instance.id             | Instance ID of the host machine.                                                                                                                                                    | keyword          |             |
  | cloud.provider                | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.                                                                                                    | keyword          |             |
  | cloud.region                  | Region in which this host is running.                                                                                                                                               | keyword          |             |
  | container.id                  | Unique container id.                                                                                                                                                                | keyword          |             |
  | data_stream.dataset           | Data stream dataset.                                                                                                                                                                | constant_keyword |             |
  | data_stream.namespace         | Data stream namespace.                                                                                                                                                              | constant_keyword |             |
  | data_stream.type              | Data stream type.                                                                                                                                                                   | constant_keyword |             |
  | event.module                  | Event module.                                                                                                                                                                       | constant_keyword |             |
  | host.containerized            | If the host is a container.                                                                                                                                                         | boolean          |             |
  | host.name                     | Name of the host.  It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword          |             |
  | host.os.build                 | OS build information.                                                                                                                                                               | keyword          |             |
  | host.os.codename              | OS codename, if any.                                                                                                                                                                | keyword          |             |
  | prometheus.labels.*           | Prometheus metric labels.                                                                                                                                                           | keyword          |             |
  | prometheus.labels_fingerprint | Autogenerated ID representing the fingerprint of labels object and includes query name.                                                                                             | keyword          |             |
  | prometheus.query.*            | Prometheus value resulted from PromQL                                                                                                                                               | object           | gauge       |
  | service.address               | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets).                            | keyword          |             |
</dropdown>


## Dashboard

Prometheus integration is shipped including default overview dashboard.
Default dashboard works only for `remote_write` datastream and `collector` datastream, if metrics are scraped from the Prometheus server metrics endpoint.

## Screenshots

This integration includes one or more Kibana dashboards that visualizes the data collected by the integration. The screenshots below illustrate how the ingested data is displayed.
<carousel>

  ![Metricbeat Prometheus Overview](https://epr.elastic.co/package/prometheus/1.24.2/img/prometheus-server-overview.png)
</carousel>


## Changelog

<dropdown title="Changelog">


  | Version          | Details                                                                                                                                                                                                                                                                                                                                                                                   | Kibana version(s)               |
  |------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------|
  | **1.24.2**1.24.2 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/14694)View pull request)Fix a typo in the Prometheus integration documentation.                                                                                                                                                                                                              | 8.16.0 or higher9.0.0 or higher |
  | **1.24.1**1.24.1 | **Bug fix**Bug fix ([View pull request](https://github.com/elastic/integrations/pull/14526)View pull request)Fixed ssl-verification-mode not respected, when no bearer token is set.                                                                                                                                                                                                      | 8.16.0 or higher9.0.0 or higher |
  | **1.24.0**1.24.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/13969)View pull request)Add username, password, and SSL related fields for query dataset.                                                                                                                                                                                                    | 8.16.0 or higher9.0.0 or higher |
  | **1.23.1**1.23.1 | **Bug fix**Bug fix ([View pull request](https://github.com/elastic/integrations/pull/13010)View pull request)Added description to ssl nodes in datastream manifest.yml files to including links to documentation.                                                                                                                                                                         | 8.16.0 or higher9.0.0 or higher |
  | **1.23.0**1.23.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/12475)View pull request)Add support for Kibana `9.0.0`.                                                                                                                                                                                                                                      | 8.16.0 or higher9.0.0 or higher |
  | **1.22.0**1.22.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/11996)View pull request)Added custom configuration option to all data streams.                                                                                                                                                                                                               | 8.16.0 or higher                |
  | **1.21.0**1.21.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/11988)View pull request)Upgrade the package format_version to 3.3.0.                                                                                                                                                                                                                         | 8.16.0 or higher                |
  | **1.20.1**1.20.1 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/11876)View pull request)Improve documentation to replace references to Metricbeat with Elastic Agent.                                                                                                                                                                                        | 8.16.0 or higher                |
  | **1.20.0**1.20.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/11447)View pull request)Add support for `metrics_count` to count number of metrics per Elasticsearch document.                                                                                                                                                                               | 8.16.0 or higher                |
  | **1.19.0**1.19.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/10592)View pull request)Allow automatic rerouting in promethues ingest-pipelines. E.g. for rerouting of events in Kubernetes environments.                                                                                                                                                   | 8.14.0 or higher                |
  | **1.18.2**1.18.2 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/10840)View pull request)Add missing SSL parameters to the collector data stream.                                                                                                                                                                                                             | 8.14.0 or higher                |
  | **1.18.1**1.18.1 | **Bug fix**Bug fix ([View pull request](https://github.com/elastic/integrations/pull/10766)View pull request)Add a dimension field with a fingerprint value applied over Prometheus label values to avoid document drop.                                                                                                                                                                  | 8.14.0 or higher                |
  | **1.18.0**1.18.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/10171)View pull request)ECS version updated to 8.11.0. Update the kibana constraint to ^8.14.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.                                                                                   | 8.14.0 or higher                |
  | **1.17.0**1.17.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/9785)View pull request)Remove labels fingerprint, leading to better TSDB performance and storage reduction.                                                                                                                                                                                  | 8.14.0 or higher                |
  | **1.16.0**1.16.0 | **Bug fix**Bug fix ([View pull request](https://github.com/elastic/integrations/pull/9995)View pull request)Add global filter on data_stream.dataset to improve performance.                                                                                                                                                                                                              | 8.12.1 or higher                |
  | **1.15.3**1.15.3 | **Bug fix**Bug fix ([View pull request](https://github.com/elastic/integrations/pull/10088)View pull request)Adding ssl verfication mode and removing default username for collector                                                                                                                                                                                                      | 8.12.1 or higher                |
  | **1.15.2**1.15.2 | **Bug fix**Bug fix ([View pull request](https://github.com/elastic/integrations/pull/7914)View pull request)Fix typo - Replace "darastream" with "datastream"                                                                                                                                                                                                                             | 8.12.1 or higher                |
  | **1.15.1**1.15.1 | **Bug fix**Bug fix ([View pull request](https://github.com/elastic/integrations/pull/9424)View pull request)Add a "period" parameter to the counter cache.                                                                                                                                                                                                                                | 8.12.1 or higher                |
  | **1.15.0**1.15.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/9321)View pull request)Enable secrets for sensitive fields. For more details, refer https://www.elastic.co/guide/en/fleet/current/agent-policy.html#agent-policy-secret-values                                                                                                               | 8.12.1 or higher                |
  | **1.14.2**1.14.2 | **Bug fix**Bug fix ([View pull request](https://github.com/elastic/integrations/pull/9279)View pull request)Disable secrets for older stack versions due to errors.                                                                                                                                                                                                                       | 8.9.0 or higher                 |
  | **1.14.1**1.14.1 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/9264)View pull request)Add Certificate Authorities for Remote Write                                                                                                                                                                                                                          | 8.9.0 or higher                 |
  | **1.14.0**1.14.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/9009)View pull request)Enable 'secret' for the sensitive fields, supported from 8.12.                                                                                                                                                                                                        | 8.9.0 or higher                 |
  | **1.13.1**1.13.1 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/8016)View pull request)Migrate Prometheus Server Overview dashboard to lens.                                                                                                                                                                                                                 | 8.9.0 or higher                 |
  | **1.13.0**1.13.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/7082)View pull request)Ensure event.kind is correctly set for pipeline errors.                                                                                                                                                                                                               | 8.9.0 or higher                 |
  | **1.12.1**1.12.1 | **Bug fix**Bug fix ([View pull request](https://github.com/elastic/integrations/pull/7882)View pull request)Fix remote_write ingest pipeline to include both metric path 'prometheus.*' and 'prometheus.metrics.*' and 'prometheus.metrics.' to fingerprint calculation                                                                                                                   | 8.9.0 or higher                 |
  | **1.12.0**1.12.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/7667)View pull request)Use ecs definition of the 'event.dataset' field                                                                                                                                                                                                                       | 8.9.0 or higher                 |
  | **1.11.0**1.11.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/7598)View pull request)Enable TSDB by default for remote_write datastreams. This improves storage usage and query performance. For more details, see https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html                                                               | 8.9.0 or higher                 |
  | **1.10.0**1.10.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/7594)View pull request)Align fingerprint field name across all datastreams, add handling of pipeline failures to the collector and query datastreams                                                                                                                                         | 8.9.0 or higher                 |
  | **1.9.0**1.9.0   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/7565)View pull request)Add dimension and metric_type fields to remote_write datastream                                                                                                                                                                                                       | 8.9.0 or higher                 |
  | **1.8.0**1.8.0   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/7261)View pull request)Enable TSDB by default for collector and query metrics data streams. This improves storage usage and query performance. For more details, see https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html. Still TSDB is not supported for remote_write | 8.9.0 or higher                 |
  | **1.7.0**1.7.0   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/7324)View pull request)Revert metrics field definition to the format used before introducing metric_type                                                                                                                                                                                     | 8.9.0 or higher                 |
  | **1.6.0**1.6.0   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/6981)View pull request)Add metric_type fields to collector and query datastreams for TSDB support                                                                                                                                                                                            | 8.9.0 or higher                 |
  | **1.5.0**1.5.0   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/5974)View pull request)Add dimension fields to collector and query datastreams for TSDB support                                                                                                                                                                                              | 8.4.0 or higher                 |
  | **1.4.0**1.4.0   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/6536)View pull request)Add processor configuration                                                                                                                                                                                                                                           | 8.4.0 or higher                 |
  | **1.3.2**1.3.2   | **Bug fix**Bug fix ([View pull request](https://github.com/elastic/integrations/pull/6140)View pull request)Fix timeout and connect_timeout parameter parsing issue                                                                                                                                                                                                                       | 8.4.0 or higher                 |
  | **1.3.1**1.3.1   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/5123)View pull request)Added categories and/or subcategories.                                                                                                                                                                                                                                | 8.4.0 or higher                 |
  | **1.3.0**1.3.0   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/5336)View pull request)Support data_stream.dataset name                                                                                                                                                                                                                                      | 8.4.0 or higher                 |
  | **1.2.0**1.2.0   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/5317)View pull request)Enable setting condition on Prometheus Collector                                                                                                                                                                                                                      | 8.4.0 or higher                 |
  | **1.1.0**1.1.0   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/4983)View pull request)Remove "integration" from the package name                                                                                                                                                                                                                            | 8.4.0 or higher                 |
  | **1.0.1**1.0.1   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/4545)View pull request)Removing x-pack references and updating default values                                                                                                                                                                                                                | 8.4.0 or higher                 |
  | **1.0.0**1.0.0   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/4315)View pull request)Promote integration to GA                                                                                                                                                                                                                                             | 8.4.0 or higher                 |
  | **0.14.0**0.14.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/4275)View pull request)Update default dashboard                                                                                                                                                                                                                                              |                                 |
  | **0.13.0**0.13.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/4060)View pull request)Revert index mapping changes for histogram type                                                                                                                                                                                                                       |                                 |
  | **0.12.0**0.12.0 | **Bug fix**Bug fix ([View pull request](https://github.com/elastic/integrations/pull/4034)View pull request)Disable leader election by default, remove bearer token file default value                                                                                                                                                                                                    |                                 |
  | **0.11.0**0.11.0 | **Bug fix**Bug fix ([View pull request](https://github.com/elastic/integrations/pull/3891)View pull request)Fix histogram type in collector data_stream                                                                                                                                                                                                                                   |                                 |
  | **0.10.0**0.10.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/3662)View pull request)Hide some configuration for remote_write data_stream; Add leader election for collector and query data_streams                                                                                                                                                        |                                 |
  | **0.9.2**0.9.2   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/2916)View pull request)reworded a link to related documentation in the Readme so it is clearer                                                                                                                                                                                               |                                 |
  | **0.9.1**0.9.1   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/2916)View pull request)Add documentation for multi-fields                                                                                                                                                                                                                                    |                                 |
  | **0.9.0**0.9.0   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/2632)View pull request)Add standard HTTP options to the package                                                                                                                                                                                                                              |                                 |
  | **0.8.0**0.8.0   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/2619)View pull request)Improve default datastream enablement                                                                                                                                                                                                                                 |                                 |
  | **0.7.0**0.7.0   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/2189)View pull request)Release prometheus package for v8.0.0                                                                                                                                                                                                                                 |                                 |
  | **0.6.1**0.6.1   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/2050)View pull request)Uniform with guidelines                                                                                                                                                                                                                                               |                                 |
  | **0.6.0**0.6.0   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/1705)View pull request)Update to ECS 1.12.0                                                                                                                                                                                                                                                  |                                 |
  | **0.5.1**0.5.1   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/1405)View pull request)Escape special characters in docs                                                                                                                                                                                                                                     |                                 |
  | **0.5.0**0.5.0   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/1364)View pull request)Update integration description                                                                                                                                                                                                                                        |                                 |
  | **0.4.1**0.4.1   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/1323)View pull request)Fix yml code blocks                                                                                                                                                                                                                                                   |                                 |
  | **0.4.0**0.4.0   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/1249)View pull request)Set `event.module` and `event.dataset`                                                                                                                                                                                                                                |                                 |
  | **0.3.5**0.3.5   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/766)View pull request)Updating package owner                                                                                                                                                                                                                                                 |                                 |
  | **0.3.4**0.3.4   | **Bug fix**Bug fix ([View pull request](https://github.com/elastic/integrations/pull/754)View pull request)Correct sample event file.                                                                                                                                                                                                                                                     |                                 |
  | **0.3.3**0.3.3   | **Bug fix**Bug fix ([View pull request](https://github.com/elastic/integrations/pull/749)View pull request)Change kibana.version constraint to be more conservative.                                                                                                                                                                                                                      |                                 |
  | **0.1.0**0.1.0   | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/114)View pull request)initial release                                                                                                                                                                                                                                                        |                                 |
</dropdown>
