﻿---
title: Hadoop
description: This integration is used to collect Hadoop metrics as follows: Application Metrics, Cluster Metrics, DataNode Metrics, NameNode Metrics, NodeManager Metrics...
url: https://www.elastic.co/docs/reference/integrations/hadoop
---

# Hadoop


|                                                                                                                                     |                                        |
|-------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------|
| **Version**Version                                                                                                                  | 1.9.0 ([View all](#changelog)View all) |
| **Subscription level**Subscription level[What's this?](https://www.elastic.co/subscriptions)What's this?                            | Basic                                  |
| **Level of support**Level of support[What's this?](https://www.elastic.co/docs/reference/integrations/level-of-support)What's this? | Elastic                                |
| **Ingestion method(s)**Ingestion method(s)                                                                                          | API                                    |

This integration is used to collect [Hadoop](https://hadoop.apache.org/) metrics as follows:
- Application Metrics
- Cluster Metrics
- DataNode Metrics
- NameNode Metrics
- NodeManager Metrics

This integration uses Resource Manager API and JMX API to collect above metrics.

## Compatibility

This integration has been tested against Hadoop version `3.3.6`.

### Troubleshooting

If host.ip is shown conflicted under `logs-*` data view, then this issue can be solved by [reindexing](https://www.elastic.co/guide/en/elasticsearch/reference/current/use-a-data-stream.html#reindex-with-a-data-stream) the `Application` data stream's indices.
If host.ip is shown conflicted under `metrics-*` data view, then this issue can be solved by [reindexing](https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds-reindex.html) the `Cluster`, `Datanode`, `Namenode` and `Node Manager` data stream's indices.

## application

This data stream collects Application metrics.
<dropdown title="Example">

  ```json
  {
      "@timestamp": "2023-02-02T12:03:41.178Z",
      "agent": {
          "ephemeral_id": "71297f22-c3ed-49b3-a8a9-a9d2086f8df2",
          "id": "2d054344-10a6-40d9-90c1-ea017fecfda3",
          "name": "docker-fleet-agent",
          "type": "filebeat",
          "version": "8.5.1"
      },
      "data_stream": {
          "dataset": "hadoop.application",
          "namespace": "ep",
          "type": "logs"
      },
      "ecs": {
          "version": "8.11.0"
      },
      "elastic_agent": {
          "id": "2d054344-10a6-40d9-90c1-ea017fecfda3",
          "snapshot": false,
          "version": "8.5.1"
      },
      "event": {
          "agent_id_status": "verified",
          "category": [
              "database"
          ],
          "created": "2023-02-02T12:03:41.178Z",
          "dataset": "hadoop.application",
          "ingested": "2023-02-02T12:03:42Z",
          "kind": "metric",
          "module": "httpjson",
          "type": [
              "info"
          ]
      },
      "hadoop": {
          "application": {
              "allocated": {
                  "mb": 2048,
                  "v_cores": 1
              },
              "id": "application_1675339401983_0001",
              "memory_seconds": 12185,
              "progress": 0,
              "running_containers": 1,
              "time": {
                  "elapsed": 7453,
                  "finished": "1970-01-01T00:00:00.000Z",
                  "started": "2023-02-02T12:03:33.662Z"
              },
              "vcore_seconds": 5
          }
      },
      "input": {
          "type": "httpjson"
      },
      "tags": [
          "forwarded"
      ]
  }
  ```
</dropdown>

**ECS Field Reference**
Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.
<dropdown title="Exported fields">


  | Field                                 | Description                                                                         | Type             |
  |---------------------------------------|-------------------------------------------------------------------------------------|------------------|
  | @timestamp                            | Event timestamp.                                                                    | date             |
  | data_stream.dataset                   | Data stream dataset.                                                                | constant_keyword |
  | data_stream.namespace                 | Data stream namespace.                                                              | constant_keyword |
  | data_stream.type                      | Data stream type.                                                                   | constant_keyword |
  | hadoop.application.allocated.mb       | Total memory allocated to the application's running containers (Mb)                 | long             |
  | hadoop.application.allocated.v_cores  | The total number of virtual cores allocated to the application's running containers | long             |
  | hadoop.application.id                 | Application ID                                                                      | keyword          |
  | hadoop.application.memory_seconds     | The amount of memory the application has allocated                                  | long             |
  | hadoop.application.progress           | Application progress (%)                                                            | long             |
  | hadoop.application.running_containers | Number of containers currently running for the application                          | long             |
  | hadoop.application.time.elapsed       | The elapsed time since application started (ms)                                     | long             |
  | hadoop.application.time.finished      | Application finished time                                                           | date             |
  | hadoop.application.time.started       | Application start time                                                              | date             |
  | hadoop.application.vcore_seconds      | The amount of CPU resources the application has allocated                           | long             |
  | input.type                            | Type of Filebeat input.                                                             | keyword          |
</dropdown>


## cluster

This data stream collects Cluster metrics.
<dropdown title="Example">

  ```json
  {
      "@timestamp": "2022-04-04T17:22:22.255Z",
      "agent": {
          "ephemeral_id": "a8157f06-f6b6-4eae-b67f-4ad08fa7c170",
          "id": "abf8f8c1-f293-4e16-a8f8-8cf48014d040",
          "name": "docker-fleet-agent",
          "type": "metricbeat",
          "version": "8.1.0"
      },
      "data_stream": {
          "dataset": "hadoop.cluster",
          "namespace": "ep",
          "type": "metrics"
      },
      "ecs": {
          "version": "8.11.0"
      },
      "elastic_agent": {
          "id": "abf8f8c1-f293-4e16-a8f8-8cf48014d040",
          "snapshot": false,
          "version": "8.1.0"
      },
      "event": {
          "agent_id_status": "verified",
          "category": [
              "database"
          ],
          "dataset": "hadoop.cluster",
          "duration": 50350559,
          "ingested": "2022-04-04T17:22:25Z",
          "kind": "metric",
          "module": "http",
          "type": [
              "info"
          ]
      },
      "hadoop": {
          "cluster": {
              "application_main": {
                  "launch_delay_avg_time": 2115,
                  "launch_delay_num_ops": 1,
                  "register_delay_avg_time": 0,
                  "register_delay_num_ops": 0
              },
              "node_managers": {
                  "num_active": 1,
                  "num_decommissioned": 0,
                  "num_lost": 0,
                  "num_rebooted": 0,
                  "num_unhealthy": 0
              }
          }
      },
      "host": {
          "architecture": "x86_64",
          "containerized": true,
          "hostname": "docker-fleet-agent",
          "ip": [
              "172.27.0.7"
          ],
          "mac": [
              "02-42-AC-1F-00-07"
          ],
          "name": "docker-fleet-agent",
          "os": {
              "codename": "focal",
              "family": "debian",
              "kernel": "3.10.0-1160.53.1.el7.x86_64",
              "name": "Ubuntu",
              "platform": "ubuntu",
              "type": "linux",
              "version": "20.04.3 LTS (Focal Fossa)"
          }
      },
      "metricset": {
          "name": "json",
          "period": 60000
      },
      "service": {
          "address": "http://elastic-package-service_hadoop_1:8088/jmx?qry=Hadoop%3Aservice%3DResourceManager%2Cname%3DClusterMetrics",
          "type": "http"
      },
      "tags": [
          "hadoop-cluster"
      ]
  }
  ```
</dropdown>

**ECS Field Reference**
Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.
<dropdown title="Exported fields">


  | Field                                                   | Description                                                                                                                                                                                               | Type             | Metric Type |
  |---------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|-------------|
  | @timestamp                                              | Event timestamp.                                                                                                                                                                                          | date             |             |
  | agent.id                                                | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.                                                                                                                | keyword          |             |
  | cloud.account.id                                        | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.                        | keyword          |             |
  | cloud.availability_zone                                 | Availability zone in which this host, resource, or service is located.                                                                                                                                    | keyword          |             |
  | cloud.instance.id                                       | Instance ID of the host machine.                                                                                                                                                                          | keyword          |             |
  | cloud.provider                                          | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.                                                                                                                          | keyword          |             |
  | cloud.region                                            | Region in which this host, resource, or service is located.                                                                                                                                               | keyword          |             |
  | container.id                                            | Unique container id.                                                                                                                                                                                      | keyword          |             |
  | data_stream.dataset                                     | Data stream dataset.                                                                                                                                                                                      | constant_keyword |             |
  | data_stream.namespace                                   | Data stream namespace.                                                                                                                                                                                    | constant_keyword |             |
  | data_stream.type                                        | Data stream type.                                                                                                                                                                                         | constant_keyword |             |
  | hadoop.cluster.application_main.launch_delay_avg_time   | Application Main Launch Delay Average Time (Milliseconds)                                                                                                                                                 | long             | gauge       |
  | hadoop.cluster.application_main.launch_delay_num_ops    | Application Main Launch Delay Operations (Number of Operations)                                                                                                                                           | long             | gauge       |
  | hadoop.cluster.application_main.register_delay_avg_time | Application Main Register Delay Average Time (Milliseconds)                                                                                                                                               | long             | gauge       |
  | hadoop.cluster.application_main.register_delay_num_ops  | Application Main Register Delay Operations (Number of Operations)                                                                                                                                         | long             | gauge       |
  | hadoop.cluster.applications.completed                   | The number of applications completed                                                                                                                                                                      | long             | counter     |
  | hadoop.cluster.applications.failed                      | The number of applications failed                                                                                                                                                                         | long             | counter     |
  | hadoop.cluster.applications.killed                      | The number of applications killed                                                                                                                                                                         | long             | counter     |
  | hadoop.cluster.applications.pending                     | The number of applications pending                                                                                                                                                                        | long             | gauge       |
  | hadoop.cluster.applications.running                     | The number of applications running                                                                                                                                                                        | long             | gauge       |
  | hadoop.cluster.applications.submitted                   | The number of applications submitted                                                                                                                                                                      | long             | counter     |
  | hadoop.cluster.containers.allocated                     | The number of containers allocated                                                                                                                                                                        | long             | gauge       |
  | hadoop.cluster.containers.pending                       | The number of containers pending                                                                                                                                                                          | long             | gauge       |
  | hadoop.cluster.containers.reserved                      | The number of containers reserved                                                                                                                                                                         | long             | gauge       |
  | hadoop.cluster.memory.allocated                         | The amount of memory allocated in MB                                                                                                                                                                      | long             | gauge       |
  | hadoop.cluster.memory.available                         | The amount of memory available in MB                                                                                                                                                                      | long             | gauge       |
  | hadoop.cluster.memory.reserved                          | The amount of memory reserved in MB                                                                                                                                                                       | long             | gauge       |
  | hadoop.cluster.memory.total                             | The amount of total memory in MB                                                                                                                                                                          | long             | gauge       |
  | hadoop.cluster.node_managers.num_active                 | Number of Node Managers Active                                                                                                                                                                            | long             | gauge       |
  | hadoop.cluster.node_managers.num_decommissioned         | Number of Node Managers Decommissioned                                                                                                                                                                    | long             | gauge       |
  | hadoop.cluster.node_managers.num_lost                   | Number of Node Managers Lost                                                                                                                                                                              | long             | gauge       |
  | hadoop.cluster.node_managers.num_rebooted               | Number of Node Managers Rebooted                                                                                                                                                                          | long             | gauge       |
  | hadoop.cluster.node_managers.num_unhealthy              | Number of Node Managers Unhealthy                                                                                                                                                                         | long             | gauge       |
  | hadoop.cluster.nodes.active                             | The number of active nodes                                                                                                                                                                                | long             | gauge       |
  | hadoop.cluster.nodes.decommissioned                     | The number of nodes decommissioned                                                                                                                                                                        | long             | gauge       |
  | hadoop.cluster.nodes.decommissioning                    | The number of nodes being decommissioned                                                                                                                                                                  | long             | gauge       |
  | hadoop.cluster.nodes.lost                               | The number of lost nodes                                                                                                                                                                                  | long             | gauge       |
  | hadoop.cluster.nodes.rebooted                           | The number of nodes rebooted                                                                                                                                                                              | long             | gauge       |
  | hadoop.cluster.nodes.shutdown                           | The number of nodes shut down                                                                                                                                                                             | long             | gauge       |
  | hadoop.cluster.nodes.total                              | The total number of nodes                                                                                                                                                                                 | long             | gauge       |
  | hadoop.cluster.nodes.unhealthy                          | The number of unhealthy nodes                                                                                                                                                                             | long             | gauge       |
  | hadoop.cluster.virtual_cores.allocated                  | The number of allocated virtual cores                                                                                                                                                                     | long             | gauge       |
  | hadoop.cluster.virtual_cores.available                  | The number of available virtual cores                                                                                                                                                                     | long             | gauge       |
  | hadoop.cluster.virtual_cores.reserved                   | The number of reserved virtual cores                                                                                                                                                                      | long             | gauge       |
  | hadoop.cluster.virtual_cores.total                      | The total number of virtual cores                                                                                                                                                                         | long             | gauge       |
  | host.name                                               | Name of the host. It can contain what hostname returns on Unix systems, the fully qualified domain name (FQDN), or a name specified by the user. The recommended value is the lowercase FQDN of the host. | keyword          |             |
  | service.address                                         | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets).                                                  | keyword          |             |
</dropdown>


## datanode

This data stream collects Datanode metrics.
<dropdown title="Example">

  ```json
  {
      "@timestamp": "2023-02-02T12:05:04.266Z",
      "agent": {
          "ephemeral_id": "2f5c3354-b1d6-4f1b-b12e-2824ae65dd02",
          "id": "2d054344-10a6-40d9-90c1-ea017fecfda3",
          "name": "docker-fleet-agent",
          "type": "metricbeat",
          "version": "8.5.1"
      },
      "data_stream": {
          "dataset": "hadoop.datanode",
          "namespace": "ep",
          "type": "metrics"
      },
      "ecs": {
          "version": "8.11.0"
      },
      "elastic_agent": {
          "id": "2d054344-10a6-40d9-90c1-ea017fecfda3",
          "snapshot": false,
          "version": "8.5.1"
      },
      "event": {
          "agent_id_status": "verified",
          "category": [
              "database"
          ],
          "dataset": "hadoop.datanode",
          "duration": 241651987,
          "ingested": "2023-02-02T12:05:05Z",
          "kind": "metric",
          "module": "http",
          "type": [
              "info"
          ]
      },
      "hadoop": {
          "datanode": {
              "blocks": {
                  "cached": 0,
                  "failed": {
                      "to_cache": 0,
                      "to_uncache": 0
                  }
              },
              "cache": {
                  "capacity": 0,
                  "used": 0
              },
              "dfs_used": 804585,
              "disk_space": {
                  "capacity": 80637005824,
                  "remaining": 56384421888
              },
              "estimated_capacity_lost_total": 0,
              "last_volume_failure_date": "1970-01-01T00:00:00.000Z",
              "volumes": {
                  "failed": 0
              }
          }
      },
      "host": {
          "architecture": "x86_64",
          "containerized": true,
          "hostname": "docker-fleet-agent",
          "id": "75e38940166b4dbc90b6f5610e8e9c39",
          "ip": [
              "172.28.0.5"
          ],
          "mac": [
              "02-42-AC-1C-00-05"
          ],
          "name": "docker-fleet-agent",
          "os": {
              "codename": "focal",
              "family": "debian",
              "kernel": "3.10.0-1160.80.1.el7.x86_64",
              "name": "Ubuntu",
              "platform": "ubuntu",
              "type": "linux",
              "version": "20.04.5 LTS (Focal Fossa)"
          }
      },
      "metricset": {
          "name": "json",
          "period": 60000
      },
      "service": {
          "address": "http://elastic-package-service_hadoop_1:9864/jmx?qry=Hadoop%3Aname%3DDataNodeActivity%2A%2Cservice%3DDataNode",
          "type": "http"
      },
      "tags": [
          "hadoop-datanode"
      ]
  }
  ```
</dropdown>

**ECS Field Reference**
Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.
<dropdown title="Exported fields">


  | Field                                         | Description                                                                                                                                                                                               | Type             | Metric Type |
  |-----------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|-------------|
  | @timestamp                                    | Event timestamp.                                                                                                                                                                                          | date             |             |
  | agent.id                                      | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.                                                                                                                | keyword          |             |
  | cloud.account.id                              | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.                        | keyword          |             |
  | cloud.availability_zone                       | Availability zone in which this host, resource, or service is located.                                                                                                                                    | keyword          |             |
  | cloud.instance.id                             | Instance ID of the host machine.                                                                                                                                                                          | keyword          |             |
  | cloud.provider                                | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.                                                                                                                          | keyword          |             |
  | cloud.region                                  | Region in which this host, resource, or service is located.                                                                                                                                               | keyword          |             |
  | container.id                                  | Unique container id.                                                                                                                                                                                      | keyword          |             |
  | data_stream.dataset                           | Data stream dataset.                                                                                                                                                                                      | constant_keyword |             |
  | data_stream.namespace                         | Data stream namespace.                                                                                                                                                                                    | constant_keyword |             |
  | data_stream.type                              | Data stream type.                                                                                                                                                                                         | constant_keyword |             |
  | hadoop.datanode.blocks.cached                 | The number of blocks cached                                                                                                                                                                               | long             | gauge       |
  | hadoop.datanode.blocks.failed.to_cache        | The number of blocks that failed to cache                                                                                                                                                                 | long             | gauge       |
  | hadoop.datanode.blocks.failed.to_uncache      | The number of failed blocks to remove from cache                                                                                                                                                          | long             | gauge       |
  | hadoop.datanode.bytes.read                    | Data read                                                                                                                                                                                                 | long             | counter     |
  | hadoop.datanode.bytes.written                 | Data written                                                                                                                                                                                              | long             | counter     |
  | hadoop.datanode.cache.capacity                | Cache capacity in bytes                                                                                                                                                                                   | long             | gauge       |
  | hadoop.datanode.cache.used                    | Cache used in bytes                                                                                                                                                                                       | long             | gauge       |
  | hadoop.datanode.dfs_used                      | Distributed File System Used                                                                                                                                                                              | long             | gauge       |
  | hadoop.datanode.disk_space.capacity           | Disk capacity in bytes                                                                                                                                                                                    | long             | gauge       |
  | hadoop.datanode.disk_space.remaining          | The remaining disk space left in bytes                                                                                                                                                                    | long             | gauge       |
  | hadoop.datanode.estimated_capacity_lost_total | The estimated capacity lost in bytes                                                                                                                                                                      | long             | gauge       |
  | hadoop.datanode.last_volume_failure_date      | The date/time of the last volume failure in milliseconds since epoch                                                                                                                                      | date             |             |
  | hadoop.datanode.volumes.failed                | Number of failed volumes                                                                                                                                                                                  | long             | gauge       |
  | host.name                                     | Name of the host. It can contain what hostname returns on Unix systems, the fully qualified domain name (FQDN), or a name specified by the user. The recommended value is the lowercase FQDN of the host. | keyword          |             |
  | service.address                               | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets).                                                  | keyword          |             |
</dropdown>


## namenode

This data stream collects Namenode metrics.
<dropdown title="Example">

  ```json
  {
      "@timestamp": "2022-03-28T11:36:07.166Z",
      "agent": {
          "ephemeral_id": "1304d85b-b3ba-45a5-ad59-c9ec7df3a49f",
          "id": "adf6847a-3726-4fe6-a202-147021ff3cbc",
          "name": "docker-fleet-agent",
          "type": "metricbeat",
          "version": "8.1.0"
      },
      "data_stream": {
          "dataset": "hadoop.namenode",
          "namespace": "ep",
          "type": "metrics"
      },
      "ecs": {
          "version": "8.11.0"
      },
      "elastic_agent": {
          "id": "adf6847a-3726-4fe6-a202-147021ff3cbc",
          "snapshot": false,
          "version": "8.1.0"
      },
      "event": {
          "agent_id_status": "verified",
          "category": [
              "database"
          ],
          "dataset": "hadoop.namenode",
          "duration": 341259289,
          "ingested": "2022-03-28T11:36:09Z",
          "kind": "metric",
          "module": "http",
          "type": [
              "info"
          ]
      },
      "hadoop": {
          "namenode": {
              "blocks": {
                  "corrupt": 0,
                  "missing_repl_one": 0,
                  "pending_deletion": 0,
                  "pending_replication": 0,
                  "scheduled_replication": 0,
                  "total": 0,
                  "under_replicated": 0
              },
              "capacity": {
                  "remaining": 11986817024,
                  "total": 48420556800,
                  "used": 4096
              },
              "estimated_capacity_lost_total": 0,
              "files_total": 9,
              "lock_queue_length": 0,
              "nodes": {
                  "num_dead_data": 0,
                  "num_decom_dead_data": 0,
                  "num_decom_live_data": 0,
                  "num_decommissioning_data": 0,
                  "num_live_data": 1
              },
              "num_stale_storages": 0,
              "stale_data_nodes": 0,
              "total_load": 0,
              "volume_failures_total": 0
          }
      },
      "host": {
          "architecture": "x86_64",
          "containerized": true,
          "hostname": "docker-fleet-agent",
          "ip": [
              "192.168.160.7"
          ],
          "mac": [
              "02-42-AC-1F-00-07"
          ],
          "name": "docker-fleet-agent",
          "os": {
              "codename": "focal",
              "family": "debian",
              "kernel": "3.10.0-1160.45.1.el7.x86_64",
              "name": "Ubuntu",
              "platform": "ubuntu",
              "type": "linux",
              "version": "20.04.3 LTS (Focal Fossa)"
          }
      },
      "metricset": {
          "name": "json",
          "period": 60000
      },
      "service": {
          "address": "http://elastic-package-service_hadoop_1:9870/jmx?qry=Hadoop%3Aname%3DFSNamesystem%2Cservice%3DNameNode",
          "type": "http"
      },
      "tags": [
          "hadoop-namenode"
      ]
  }
  ```
</dropdown>

**ECS Field Reference**
Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.
<dropdown title="Exported fields">


  | Field                                          | Description                                                                                                                                                                                               | Type             | Metric Type |
  |------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|-------------|
  | @timestamp                                     | Event timestamp.                                                                                                                                                                                          | date             |             |
  | agent.id                                       | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.                                                                                                                | keyword          |             |
  | cloud.account.id                               | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.                        | keyword          |             |
  | cloud.availability_zone                        | Availability zone in which this host, resource, or service is located.                                                                                                                                    | keyword          |             |
  | cloud.instance.id                              | Instance ID of the host machine.                                                                                                                                                                          | keyword          |             |
  | cloud.provider                                 | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.                                                                                                                          | keyword          |             |
  | cloud.region                                   | Region in which this host, resource, or service is located.                                                                                                                                               | keyword          |             |
  | container.id                                   | Unique container id.                                                                                                                                                                                      | keyword          |             |
  | data_stream.dataset                            | Data stream dataset.                                                                                                                                                                                      | constant_keyword |             |
  | data_stream.namespace                          | Data stream namespace.                                                                                                                                                                                    | constant_keyword |             |
  | data_stream.type                               | Data stream type.                                                                                                                                                                                         | constant_keyword |             |
  | hadoop.namenode.blocks.corrupt                 | Current number of blocks with corrupt replicas.                                                                                                                                                           | long             | gauge       |
  | hadoop.namenode.blocks.missing_repl_one        | Current number of missing blocks with replication factor 1                                                                                                                                                | long             | gauge       |
  | hadoop.namenode.blocks.pending_deletion        | Current number of blocks pending deletion                                                                                                                                                                 | long             | gauge       |
  | hadoop.namenode.blocks.pending_replication     | Current number of blocks pending to be replicated                                                                                                                                                         | long             | gauge       |
  | hadoop.namenode.blocks.scheduled_replication   | Current number of blocks scheduled for replications                                                                                                                                                       | long             | gauge       |
  | hadoop.namenode.blocks.total                   | Current number of allocated blocks in the system                                                                                                                                                          | long             | gauge       |
  | hadoop.namenode.blocks.under_replicated        | Current number of blocks under replicated                                                                                                                                                                 | long             | gauge       |
  | hadoop.namenode.capacity.remaining             | Current remaining capacity in bytes                                                                                                                                                                       | long             | gauge       |
  | hadoop.namenode.capacity.total                 | Current raw capacity of DataNodes in bytes                                                                                                                                                                | long             | gauge       |
  | hadoop.namenode.capacity.used                  | Current used capacity across all DataNodes in bytes                                                                                                                                                       | long             | gauge       |
  | hadoop.namenode.estimated_capacity_lost_total  | An estimate of the total capacity lost due to volume failures                                                                                                                                             | long             | gauge       |
  | hadoop.namenode.files_total                    | Current number of files and directories                                                                                                                                                                   | long             | gauge       |
  | hadoop.namenode.lock_queue_length              | Number of threads waiting to acquire FSNameSystem lock                                                                                                                                                    | long             | gauge       |
  | hadoop.namenode.nodes.num_dead_data            | Number of datanodes which are currently dead                                                                                                                                                              | long             | gauge       |
  | hadoop.namenode.nodes.num_decom_dead_data      | Number of datanodes which have been decommissioned and are now dead                                                                                                                                       | long             | gauge       |
  | hadoop.namenode.nodes.num_decom_live_data      | Number of datanodes which have been decommissioned and are now live                                                                                                                                       | long             | gauge       |
  | hadoop.namenode.nodes.num_decommissioning_data | Number of datanodes in decommissioning state                                                                                                                                                              | long             | gauge       |
  | hadoop.namenode.nodes.num_live_data            | Number of datanodes which are currently live                                                                                                                                                              | long             | gauge       |
  | hadoop.namenode.num_stale_storages             | Number of storages marked as content stale                                                                                                                                                                | long             | gauge       |
  | hadoop.namenode.stale_data_nodes               | Current number of DataNodes marked stale due to delayed heartbeat                                                                                                                                         | long             | gauge       |
  | hadoop.namenode.total_load                     | Current number of connections                                                                                                                                                                             | long             | gauge       |
  | hadoop.namenode.volume_failures_total          | Total number of volume failures across all Datanodes                                                                                                                                                      | long             | gauge       |
  | host.name                                      | Name of the host. It can contain what hostname returns on Unix systems, the fully qualified domain name (FQDN), or a name specified by the user. The recommended value is the lowercase FQDN of the host. | keyword          |             |
  | service.address                                | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets).                                                  | keyword          |             |
</dropdown>


## node_manager

This data stream collects Node Manager metrics.
<dropdown title="Example">

  ```json
  {
      "@timestamp": "2022-03-28T11:54:32.506Z",
      "agent": {
          "ephemeral_id": "9948a37a-5732-4d7d-9218-0e9cf30c035a",
          "id": "adf6847a-3726-4fe6-a202-147021ff3cbc",
          "name": "docker-fleet-agent",
          "type": "metricbeat",
          "version": "8.1.0"
      },
      "data_stream": {
          "dataset": "hadoop.node_manager",
          "namespace": "ep",
          "type": "metrics"
      },
      "ecs": {
          "version": "8.11.0"
      },
      "elastic_agent": {
          "id": "adf6847a-3726-4fe6-a202-147021ff3cbc",
          "snapshot": false,
          "version": "8.1.0"
      },
      "event": {
          "agent_id_status": "verified",
          "category": [
              "database"
          ],
          "dataset": "hadoop.node_manager",
          "duration": 12930711,
          "ingested": "2022-03-28T11:54:35Z",
          "kind": "metric",
          "module": "http",
          "type": [
              "info"
          ]
      },
      "hadoop": {
          "node_manager": {
              "allocated_containers": 0,
              "container_launch_duration_avg_time": 169,
              "container_launch_duration_num_ops": 2,
              "containers": {
                  "completed": 0,
                  "failed": 2,
                  "initing": 0,
                  "killed": 0,
                  "launched": 2,
                  "running": 0
              }
          }
      },
      "host": {
          "architecture": "x86_64",
          "containerized": true,
          "hostname": "docker-fleet-agent",
          "ip": [
              "192.168.160.7"
          ],
          "mac": [
              "02-42-AC-1F-00-07"
          ],
          "name": "docker-fleet-agent",
          "os": {
              "codename": "focal",
              "family": "debian",
              "kernel": "3.10.0-1160.45.1.el7.x86_64",
              "name": "Ubuntu",
              "platform": "ubuntu",
              "type": "linux",
              "version": "20.04.3 LTS (Focal Fossa)"
          }
      },
      "metricset": {
          "name": "json",
          "period": 60000
      },
      "service": {
          "address": "http://elastic-package-service_hadoop_1:8042/jmx?qry=Hadoop%3Aservice%3DNodeManager%2Cname%3DNodeManagerMetrics",
          "type": "http"
      },
      "tags": [
          "hadoop-node_manager"
      ]
  }
  ```
</dropdown>

**ECS Field Reference**
Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.
<dropdown title="Exported fields">


  | Field                                                  | Description                                                                                                                                                                                               | Type             | Metric Type |
  |--------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|-------------|
  | @timestamp                                             | Event timestamp.                                                                                                                                                                                          | date             |             |
  | agent.id                                               | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.                                                                                                                | keyword          |             |
  | cloud.account.id                                       | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.                        | keyword          |             |
  | cloud.availability_zone                                | Availability zone in which this host, resource, or service is located.                                                                                                                                    | keyword          |             |
  | cloud.instance.id                                      | Instance ID of the host machine.                                                                                                                                                                          | keyword          |             |
  | cloud.provider                                         | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.                                                                                                                          | keyword          |             |
  | cloud.region                                           | Region in which this host, resource, or service is located.                                                                                                                                               | keyword          |             |
  | container.id                                           | Unique container id.                                                                                                                                                                                      | keyword          |             |
  | data_stream.dataset                                    | Data stream dataset.                                                                                                                                                                                      | constant_keyword |             |
  | data_stream.namespace                                  | Data stream namespace.                                                                                                                                                                                    | constant_keyword |             |
  | data_stream.type                                       | Data stream type.                                                                                                                                                                                         | constant_keyword |             |
  | hadoop.node_manager.allocated_containers               | Containers Allocated                                                                                                                                                                                      | long             | gauge       |
  | hadoop.node_manager.container_launch_duration_avg_time | Container Launch Duration Average Time (Seconds)                                                                                                                                                          | long             | gauge       |
  | hadoop.node_manager.container_launch_duration_num_ops  | Container Launch Duration Operations (Operations)                                                                                                                                                         | long             | counter     |
  | hadoop.node_manager.containers.completed               | Containers Completed                                                                                                                                                                                      | long             | counter     |
  | hadoop.node_manager.containers.failed                  | Containers Failed                                                                                                                                                                                         | long             | counter     |
  | hadoop.node_manager.containers.initing                 | Containers Initializing                                                                                                                                                                                   | long             | gauge       |
  | hadoop.node_manager.containers.killed                  | Containers Killed                                                                                                                                                                                         | long             | counter     |
  | hadoop.node_manager.containers.launched                | Containers Launched                                                                                                                                                                                       | long             | counter     |
  | hadoop.node_manager.containers.running                 | Containers Running                                                                                                                                                                                        | long             | gauge       |
  | host.name                                              | Name of the host. It can contain what hostname returns on Unix systems, the fully qualified domain name (FQDN), or a name specified by the user. The recommended value is the lowercase FQDN of the host. | keyword          |             |
  | service.address                                        | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets).                                                  | keyword          |             |
</dropdown>


## Screenshots

This integration includes one or more Kibana dashboards that visualizes the data collected by the integration. The screenshots below illustrate how the ingested data is displayed.
<carousel>

  ![Application dashboard screenshot](https://epr.elastic.co/package/hadoop/1.9.0/img/hadoop-application-dashboard-screenshot.png)

  ![Cluster Overview dashboard screenshot](https://epr.elastic.co/package/hadoop/1.9.0/img/hadoop-cluster-overview-dashboard-screenshot.png)

  ![Data node dashboard screenshot](https://epr.elastic.co/package/hadoop/1.9.0/img/hadoop-data-node-dashboard-screenshot.png)

  ![Node Manager dashboard screenshot](https://epr.elastic.co/package/hadoop/1.9.0/img/hadoop-node-manager-dashboard-screenshot.png)
</carousel>


## Changelog

<dropdown title="Changelog">


  | Version        | Details                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | Kibana version(s)               |
  |----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------|
  | **1.9.0**1.9.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/13521)View pull request)Allow @custom pipeline access to event.original without setting preserve_original_event.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | 8.13.0 or higher9.0.0 or higher |
  | **1.8.1**1.8.1 | **Bug fix**Bug fix ([View pull request](https://github.com/elastic/integrations/pull/12727)View pull request)Updated SSL description to be uniform and to include links to documentation.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | 8.13.0 or higher9.0.0 or higher |
  | **1.8.0**1.8.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/12468)View pull request)Add support for Kibana `9.0.0`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | 8.13.0 or higher9.0.0 or higher |
  | **1.7.0**1.7.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/10407)View pull request)ECS version updated to 8.11.0. Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | 8.13.0 or higher                |
  | **1.6.0**1.6.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/9768)View pull request)Add global filter on data_stream.dataset to improve performance.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | 8.10.2 or higher                |
  | **1.5.2**1.5.2 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/8423)View pull request)Inline "by reference" visualizations                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | 8.10.2 or higher                |
  | **1.5.1**1.5.1 | **Bug fix**Bug fix ([View pull request](https://github.com/elastic/integrations/pull/9021)View pull request)Update the link to the correct reindexing procedure.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | 8.10.2 or higher                |
  | **1.5.0**1.5.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/8489)View pull request)Limit request tracer log count to five.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | 8.10.2 or higher                |
  | **1.4.0**1.4.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/8172)View pull request)Update value format of "Average elapsed time" metric visualization.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | 8.10.2 or higher                |
  | **1.3.0**1.3.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/8171)View pull request)Update the package format_version to 3.0.0.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | 8.8.0 or higher                 |
  | **1.2.1**1.2.1 | **Bug fix**Bug fix ([View pull request](https://github.com/elastic/integrations/pull/7827)View pull request)Remove forwarded tag from metrics data streams.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | 8.8.0 or higher                 |
  | **1.2.0**1.2.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/7769)View pull request)Enable time series data streams for the service metrics dataset. This dramatically reduces storage for metrics and is expected to progressively improve query [performance](https://www.elastic.co/blog/70-percent-storage-savings-for-metrics-with-elastic-observability)performance. For more details, see [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html)here.                                                                                                                                                                                                                                                                                                                            | 8.8.0 or higher                 |
  | **1.1.8**1.1.8 | **Bug fix**Bug fix ([View pull request](https://github.com/elastic/integrations/pull/7845)View pull request)Add null check and ignore_missing check to the rename processor                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | 8.3.0 or higher                 |
  | **1.1.7**1.1.7 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/7722)View pull request)Add metric_type mapping for `namenode` datastream.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | 8.3.0 or higher                 |
  | **1.1.6**1.1.6 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/7720)View pull request)Add dimension mapping for `namenode` datastream.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | 8.3.0 or higher                 |
  | **1.1.5**1.1.5 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/7717)View pull request)Add metric_type mapping for `datanode` datastream.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | 8.3.0 or higher                 |
  | **1.1.4**1.1.4 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/7676)View pull request)Add dimension mapping for `datanode` datastream.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | 8.3.0 or higher                 |
  | **1.1.3**1.1.3 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/7644)View pull request)Add metric_type mapping for `node_manager` datastream.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | 8.3.0 or higher                 |
  | **1.1.2**1.1.2 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/7642)View pull request)Add dimension mapping for `node_manager` datastream.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | 8.3.0 or higher                 |
  | **1.1.1**1.1.1 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/7632)View pull request)Add metric_type mapping for `cluster` datastream.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | 8.3.0 or higher                 |
  | **1.1.0**1.1.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/7631)View pull request)Add dimension mapping for `cluster` datastream.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | 8.3.0 or higher                 |
  | **1.0.0**1.0.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/7673)View pull request)Make Hadoop GA.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | 8.3.0 or higher                 |
  | **0.9.1**0.9.1 | **Bug fix**Bug fix ([View pull request](https://github.com/elastic/integrations/pull/7564)View pull request)Resolve host.ip field conflict.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |                                 |
  | **0.9.0**0.9.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/7343)View pull request)Add support for HTTP request trace logging in application data stream.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |                                 |
  | **0.8.0**0.8.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/6298)View pull request)Rename ownership from obs-service-integrations to obs-infraobs-integrations                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |                                 |
  | **0.7.1**0.7.1 | **Bug fix**Bug fix ([View pull request](https://github.com/elastic/integrations/pull/5731)View pull request)Remove `(s)` from the Applications dashboard visualizations.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |                                 |
  | **0.7.0**0.7.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/5606)View pull request)Migrate `Cluster overview` dashboard visualizations to lens.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |                                 |
  | **0.6.0**0.6.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/5607)View pull request)Migrate `Applications` dashboard visualizations to lens.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |                                 |
  | **0.5.0**0.5.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/5605)View pull request)Migrate `Data nodes` and `Node Manager` dashboard visualizations to lens.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |                                 |
  | **0.4.2**0.4.2 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/5123)View pull request)Added categories and/or subcategories.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |                                 |
  | **0.4.1**0.4.1 | **Bug fix**Bug fix ([View pull request](https://github.com/elastic/integrations/pull/5098)View pull request)Replace format of date processor from epoch_millis to UNIX_MS                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |                                 |
  | **0.4.0**0.4.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/5057)View pull request)Update ECS version to 8.5.1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |                                 |
  | **0.3.0**0.3.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/4485)View pull request)Added infrastructure category.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |                                 |
  | **0.2.0**0.2.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/3008)View pull request)Add dashboard and visualizations for Hadoop.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |                                 |
  | **0.1.1**0.1.1 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/3080)View pull request)Update SSL and proxy configuration for application data stream.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |                                 |
  | **0.1.0**0.1.0 | **Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/2955)View pull request)Add namenode data stream for Hadoop.**Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/2964)View pull request)Add node_manager data stream, visualizations and Dashboards for Hadoop.**Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/2954)View pull request)Add datanode data stream for Hadoop.**Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/2953)View pull request)Add cluster metrics data stream for Hadoop.**Enhancement**Enhancement ([View pull request](https://github.com/elastic/integrations/pull/2952)View pull request)Add Hadoop integration with application data stream. |                                 |
</dropdown>
