Loading

Google Threat Intelligence

Version 0.6.1 beta:[] (View all)
Compatible Kibana version(s) 8.16.0 or higher
9.0.0 or higher
Supported Serverless project types
What's this?
Security
Subscription level
What's this?
Basic
Level of support
What's this?
Partner

Google Threat Intelligence is a security solution that helps organizations detect, analyze, and mitigate threats. It leverages Google's global telemetry, advanced analytics, and vast infrastructure to provide actionable insights. Key features include threat detection, malware and phishing analysis, and real-time threat alerts.

Google Threat Intelligence integration offers support for two APIs:

  1. Threat List API to deliver hourly data chunks. The Threat Lists feature allows customers to consume Indicators of Compromise (IOCs) categorized by various threat types.
  2. IOC Stream API to deliver various types of Indicators of Compromise (IOCs) originating from multiple sources. Depending on the source of the notification, different context-specific attributes are added to enrich the IOCs.

The Threat List API provides the following types of threat feeds:

  • Cryptominers
  • Daily Top Trending
  • First Stage Delivery Vectors
  • Infostealers
  • Internet of Things (IoT)
  • Linux
  • Malicious Network Infrastructure
  • Malware
  • Mobile
  • OS X
  • Phishing
  • Ransomware
  • Threat Actor
  • Vulnerability Weaponization

Customers can access a subset of the available threat lists based on their Google Threat Intelligence (GTI) tier:

  • GTI Standard: Ransomware, Malicious Network Infrastructure
  • GTI Enterprise: Ransomware, Malicious Network Infrastructure, Malware, Threat Actor, Daily Top Trending
  • GTI Enterprise+: Access to all available threat lists

Data collection is available for all threat feeds and IOC Stream, each with a separate data stream. By default, Ransomware and Malicious Network Infrastructure is enabled. Users can enable additional data streams based on their GTI subscription tier. If a user enables data collection for a data stream they do not have access to, it will result in an error log on the Discover page.

Agentless integrations allow you to collect data without having to manage Elastic Agent in your cloud. They make manual agent deployment unnecessary, so you can focus on your data instead of the agent that collects it. For more information, refer to Agentless integrations and the Agentless integrations FAQ.

Agentless deployments are only supported in Elastic Serverless and Elastic Cloud environments. This functionality is in beta and is subject to change. Beta features are not subject to the support SLA of official GA features.

Elastic Agent must be installed. For more details, check the Elastic Agent installation instructions.

  • VirusTotal URL will work as the base URL for this integration: https://www.virustotal.com
  • An API key will be used to authenticate your request.
  • Time Selection of Initial Interval and Interval:
    • Users need to specify the initial interval and interval in an hourly format, such as 2h, 3h, etc. Note: Please make sure both initial interval and interval are in hours and greater than 1 hour.
  1. In Kibana, go to Management > Integrations.
  2. In the "Search for integrations" search bar, type Google Threat Intelligence.
  3. Click on the Google Threat Intelligence integration from the search results.
  4. Click on the Add Google Threat Intelligence button to add the integration.
  5. While adding the integration, to collect logs via REST API, provide the following details:
    • Enable the type of data stream you have access to.
    • Access Token
    • Initial Interval
    • Interval
    • (Optional) Query to add custom query filtering on relationship, GTI score, and positives. (not applicable to IOC Stream)
  6. Click on Save and Continue to save the integration. Note: Please make only the threat feed types you have the privilege to access are enabled.

To keep the collected data up to date, Transforms are used.

Users can view the transforms by navigating to Management > Stack Management > Transforms.

Follow Steps to enable transforms to enable transforms and populate Threat Feed Overview and IOC Stream Overview dashboard.

Here, users can see continuously running transforms and also view the latest transformed GTI data in the Discover section.

The labels.is_transform_source field indicates log origin:

  • False for transformed index
  • True for source index

Currently, four transforms are available across all 14 data streams.

The following are four transforms along with their associated pipelines:

Transform Name Description
IP Transform (ID: logs-ti_google_threat_intelligence.ip_ioc, Pipeline: ti_google_threat_intelligence-latest_ip_ioc-transform-pipeline) Keeps IP entity type data up to date.
URL Transform (ID: logs-ti_google_threat_intelligence.url_ioc, Pipeline: ti_google_threat_intelligence-latest_url_ioc-transform-pipeline) Keeps URL entity type data up to date.
Domain Transform (ID: logs-ti_google_threat_intelligence.domain_ioc, Pipeline: ti_google_threat_intelligence-latest_domain_ioc-transform-pipeline) Keeps Domain entity type data up to date.
File Transform (ID: logs-ti_google_threat_intelligence.file_ioc, Pipeline: ti_google_threat_intelligence-latest_file_ioc-transform-pipeline) Keeps File entity type data up to date.
IP IOC Stream Transform (ID: logs-ti_google_threat_intelligence.ip_ioc_st, Pipeline: ti_google_threat_intelligence-latest_ip_ioc_st-transform-pipeline) Keeps IP entity type data up to date for IOC Stream.
URL IOC Stream Transform (ID: logs-ti_google_threat_intelligence.url_ioc_st, Pipeline: ti_google_threat_intelligence-latest_url_ioc_st-transform-pipeline) Keeps URL entity type data up to date for IOC Stream.
Domain IOC Stream Transform (ID: logs-ti_google_threat_intelligence.domain_ioc_st, Pipeline: ti_google_threat_intelligence-latest_domain_ioc_st-transform-pipeline) Keeps Domain entity type data up to date for IOC Stream.
File IOC Stream Transform (ID: logs-ti_google_threat_intelligence.file_ioc_st, Pipeline: ti_google_threat_intelligence-latest_file_ioc_st-transform-pipeline) Keeps File entity type data up to date for IOC Stream.

For example:

  • The query event.module: ti_google_threat_intelligence and labels.is_transform_source: true indicates that the logs originate from the source index.
  • The query event.module: ti_google_threat_intelligence and labels.is_transform_source: false indicates that the logs originate from the transformed index.

A retention policy is used to remove data older than the default retention period. For more details, refer to the Retention Policy Documentation.

In this integration, all data streams have a retention period of 30 days.

Detection Rules match the user's Elastic environment data with GTI data, generating an alert if a match is found. To access detection rules:

  1. Navigate to Security > Rules > Detection Rules and click on Add Elastic Rules.
  2. Search for Google Threat Intelligence to find prebuilt Elastic detection rules.
  3. Four detection rules are available for IP, URL, File, and Domain. Users can install one or more rules as needed.

To tailor a rule based on Elastic environment:

  1. Click the three dots on the right side of any detection rule.
  2. Select Duplicate Rule.
  3. Modify the duplicated rule to tailor it to your Elastic environment:
    • Index Pattern: Add the index pattern relevant to your data. Keeping this specific ensures optimal performance.
    • Custom Query: Further refine rule conditions.
    • Indicator Mapping: Map relevant fields from your Elastic environment to GTI fields. Do not modify the indicator index field.
    • Schedule Rules:
      • Set Runs Every - Defines how frequently the rule runs.
      • Additional Lookback Time - Specifies how far back to check for matches.

Once saved, successfully executed rules will generate alerts. Users can view these alerts in the Alerts section.

Note

Two transforms are available to filter relevant data from alerts. The data_stream.dataset: ti_google_threat_intelligence.enriched_ioc and data_stream.dataset: ti_google_threat_intelligence.enriched_ioc_stream field represents logs for enriched threat intelligence data, which can be analyzed in the Discover section.

The following are the names of the eight sample rules:

Sample Rule Name Description
Google Threat Intelligence URL IOC Correlation Detects and alerts on matches between URL IOCs collected by GTI data with user's selected Elastic environment data.
Google Threat Intelligence Domain IOC Correlation Detects and alerts on matches between Domain IOCs collected by GTI data with user's selected Elastic environment data.
Google Threat Intelligence File IOC Correlation Detects and alerts on matches between File IOCs collected by GTI data with user's selected Elastic environment data.
Google Threat Intelligence IP Address IOC Correlation Detects and alerts on matches between IP Address IOCs collected by GTI data with user's selected Elastic environment data.
Google Threat Intelligence URL IOC Stream Correlation Detects and alerts on matches between URL IOCs collected by GTI IOC Stream data with user's selected Elastic environment data.
Google Threat Intelligence Domain IOC Stream Correlation Detects and alerts on matches between Domain IOCs collected by GTI IOC Stream data with user's selected Elastic environment data.
Google Threat Intelligence File IOC Stream Correlation Detects and alerts on matches between File IOCs collected by GTI IOC Stream data with user's selected Elastic environment data.
Google Threat Intelligence IP Address IOC Stream Correlation Detects and alerts on matches between IP Address IOCs collected by GTI IOC Stream data with user's selected Elastic environment data.

The following are two transforms along with their associated pipelines to filter relevant data from alerts. Follow Steps to enable transforms to enable these transforms and populate Threat Intelligence, Adversary Intelligence and IOC Stream Threat Intelligence dashboards.

Transform Name Description
Detected IOC Transform (ID: logs-ti_google_threat_intelligence.rule, Pipeline: ti_google_threat_intelligence-correlation_detection_rule-pipeline) Filters and extracts necessary information from Detected IOCs from threat feed.
Detected IOC from IOC stream Transform (ID: logs-ti_google_threat_intelligence.rule_ioc_st, Pipeline: ti_google_threat_intelligence-correlation_detection_rule_ioc_st-pipeline) Filters and extracts necessary information from Detected IOCs from IOC stream.
  1. Navigate to Stack Management > Transforms in Kibana.
  2. Locate the transform you want to enable by searching for its Transform ID.
  3. Click the three dots next to the transform, then select Edit.
  4. Under the Destination configuration section, set the Ingest Pipeline:
    • Each transform in the Google Threat Intelligence integration has a corresponding ingest pipeline.
    • Refer to the Transforms table above for the appropriate pipeline name associated with transform.
    • Prefix the pipeline name with the integration version. For example:
      {package_version}-ti_google_threat_intelligence-latest_ip_ioc_st-transform-pipeline
      
    • Click Update to save the changes.
  5. Click the three dots again next to the transform and select Start to activate it.
Note

After updating the integration, make sure to update the pipeline prefix accordingly.

  1. If an event contains multiple matching mappings (e.g., two file hash fields within the same event match GTI data), only one alert per detection rule will be generated for that event.
  2. If an IOC from the user's Elasticsearch index is enriched with GTI information, and the GTI information is updated later, the changes are not reflected in the dashboards because Elastic detection rules only run on live data.
  1. If you see an error like Package 2025031310 is not available until 2025-03-13 at 11:00 UTC because of privacy policy., ensure that your initial interval and interval are set in hours and are greater than one hour.
  2. If events are not appearing in the transformed index, check if transforms are running without errors. If you encounter issues, refer to Troubleshooting transforms.
  3. If detection rules take longer to run, ensure you have specified index patterns and applied queries to make your source events more specific. Note: More events in index patterns mean more time needed for detection rules to run.
  4. Ensure that relevant fields are correctly mapped in the Indicator Mapping section. Verify that fields in the specified index pattern are properly mapped, and ensure entity-specific fields (e.g., IP fields to IP fields, keyword fields like file hash SHA256 to corresponding file hash SHA256 fields) are accurately configured.
  5. If any transform is not in a Healthy state, try resetting it:
    • Click the three dots next to the transform, then select Reset.
    • After resetting, follow the Steps to enable transforms above to reconfigure and restart the transform.

This is the Cryptominer dataset.

This is the First Stage Delivery Vectors dataset.

This is the Infostealers dataset.

This is the IOC Stream dataset.

This is the Internet of Things dataset.

This is the Linux dataset.

This is the Malicious Network Infrastructure dataset.

This is the Malware dataset.

This is the Mobile dataset.

This is the OS X dataset.

This is the Phishing dataset.

This is the Ransomware dataset.

This is the Threat Actor dataset.

This is the Daily Top trending dataset.

This is the Vulnerability Weaponization dataset.

This integration includes one or more Kibana dashboards that visualizes the data collected by the integration. The screenshots below illustrate how the ingested data is displayed.