Loading

Google Cloud Platform vertexai_logs metricset

Stack Planned

The vertexai_logs metricset is designed to collect Vertex AI prompt-response logs from GCP BigQuery. BigQuery is a fully-managed, serverless data warehouse that stores detailed logs of interactions with Vertex AI models.

Vertex AI logs export to BigQuery enables you to export detailed Google Cloud Vertex AI interaction data (such as prompts, responses, model usage, and metadata) automatically to a BigQuery dataset that you specify. Then you can access your Vertex AI logs from BigQuery for detailed analysis and monitoring using Metricbeat. This enables comprehensive tracking of AI model usage, performance monitoring, and cost analysis.

The logs include detailed information about:

  • API endpoints and deployed models
  • Request and response payloads
  • Model versions and API methods used
  • Request metadata and timing information

Metricset-specific configuration notes

  • table_id: (Required) Full table identifier in the format project_id.dataset_id.table_name that contains the Vertex AI logs data. You can copy this from the "Details" tab when viewing your table in the BigQuery web console, under the "Table ID" field.

Configuration example

- module: gcp
  metricsets:
    - vertexai_logs
  period: 10m
  project_id: "your project id"
  credentials_file_path: "your JSON credentials file path"
  table_id: "your_project.your_dataset.your_vertex_ai_logs_table"

Here is a sample event for vertexai_logs:

{
  "@timestamp": "2023-12-01T10:30:45.000Z",
  "cloud": {
    "provider": "gcp",
    "project": {
      "id": "my-gcp-project"
    }
  },
  "gcp": {
    "vertexai_logs": {
      "endpoint": "https://us-central1-aiplatform.googleapis.com",
      "deployed_model_id": "1234567890123456789",
      "logging_time": "2023-12-01T10:30:45.000Z",
      "request_id": 98765432101234567,
      "request_payload": ["What is machine learning?"],
      "response_payload": ["Machine learning is a subset of artificial intelligence..."],
      "model": "gemini-2.5-pro",
      "model_version": "1.0",
      "api_method": "generateContent",
      "full_request": {
        "inputs": ["What is machine learning?"],
        "parameters": {
          "temperature": 0.7
        }
      },
      "full_response": {
        "outputs": ["Machine learning is a subset of artificial intelligence..."],
        "usage": {
          "input_tokens": 5,
          "output_tokens": 50
        }
      },
      "metadata": {
        "user_id": "user123",
        "session_id": "session456"
      }
    }
  }
}

Fields

For a description of each field in the metricset, see the exported fields section.

Here is an example document generated by this metricset:

{
  "@timestamp": "2025-09-02T10:14:50.313Z",
  "agent": {
    "hostname": "metricbeat-host",
    "name": "metricbeat-host"
  },
  "cloud": {
    "account": {
      "id": "elastic-beats"
    },
    "provider": "gcp",
    "project": {
      "id": "elastic-beats"
    }
  },
  "event": {
    "dataset": "gcp.vertexai_logs",
    "duration": 123456789,
    "module": "gcp"
  },
  "gcp": {
    "vertexai_logs": {
      "endpoint": "https://us-central1-aiplatform.googleapis.com/v1/projects/elastic-beats/locations/us-central1/endpoints/123456789",
      "deployed_model_id": "model-deployment-123",
      "logging_time": "2025-09-02T10:14:50.313Z",
      "request_id": 98765432101234567,
      "model": "gemini-2.5-pro",
      "model_version": "001",
      "api_method": "generateContent",
      "request_payload": [
        "What is the weather like today?"
      ],
      "response_payload": [
        "I don't have access to real-time weather information. Please check a weather service or app for current conditions."
      ],
      "full_request": {
        "contents": [
          {
            "parts": [
              {
                "text": "What is the weather like today?"
              }
            ],
            "role": "user"
          }
        ],
        "generationConfig": {
          "temperature": 0.7,
          "maxOutputTokens": 1024
        }
      },
      "full_response": {
        "candidates": [
          {
            "content": {
              "parts": [
                {
                  "text": "I don't have access to real-time weather information. Please check a weather service or app for current conditions."
                }
              ],
              "role": "model"
            },
            "finishReason": "STOP",
            "safetyRatings": [
              {
                "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
                "probability": "NEGLIGIBLE"
              }
            ]
          }
        ],
        "usageMetadata": {
          "promptTokenCount": 8,
          "candidatesTokenCount": 24,
          "totalTokenCount": 32
        }
      },
      "metadata": {
        "region": "us-central1",
        "zone": "us-central1-a"
      }
    }
  },
  "metricset": {
    "name": "vertexai_logs",
    "period": 300000
  },
  "service": {
    "type": "gcp"
  }
}