Skip to content

[axonius][gateway] Add Axonius Gateway datastream #16169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
muskan-agarwal26 wants to merge 2 commits into
base: feature/axonius-0.1.0
Choose a base branch
from
Draft

[axonius][gateway] Add Axonius Gateway datastream #16169

muskan-agarwal26 wants to merge 2 commits into from

Conversation

@muskan-agarwal26
Copy link
Contributor

@muskan-agarwal26 muskan-agarwal26 commented Dec 1, 2025 •
edited
Loading

Proposed commit message

The initial release includes gateway data stream and associated dashboard.

Axonius fields are mapped to their corresponding ECS fields where possible.

Test samples were derived from live data samples, which were subsequently
sanitized.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

To test the axonius package:

  • Clone integrations repo.
  • Install elastic package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/axonius directory.
  • Run the following command to run tests.

elastic-package test

elastic-package test -v
2025/12/01 11:46:54 DEBUG Enable verbose logging
2025/12/01 11:46:54 DEBUG latest version (cached): v0.117.1. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.117.1 (Timestamp 2025-12-01 11:41:44.316264165 +0530 IST)
Run asset tests for the package
2025/12/01 11:46:54 DEBUG Connecting with Kibana host from current profile (profile: default, host: "https://127.0.0.1:5601")
2025/12/01 11:46:54 DEBUG Running tests sequentially
2025/12/01 11:46:54 DEBUG installing package...
2025/12/01 11:46:54 DEBUG Build directory: /home/devuser/integrations/build/packages/axonius/0.1.0
2025/12/01 11:46:54 DEBUG Clear target directory (path: /home/devuser/integrations/build/packages/axonius/0.1.0)
2025/12/01 11:46:54 DEBUG Copy package content (source: /home/devuser/integrations/packages/axonius)
2025/12/01 11:46:55 DEBUG Copy license file if needed
2025/12/01 11:46:55 DEBUG No license text file is included in package
2025/12/01 11:46:55 DEBUG Include linked files
2025/12/01 11:46:55 DEBUG Encode dashboards
2025/12/01 11:46:55 DEBUG Resolve external fields
2025/12/01 11:46:55 DEBUG Package has external dependencies defined
2025/12/01 11:46:55 DEBUG data_stream/gateway/fields/base-fields.yml: source file has been changed
2025/12/01 11:46:55 DEBUG data_stream/gateway/fields/ecs.yml: source file has been changed
2025/12/01 11:46:55 DEBUG Package doesn't have to import ECS mappings
2025/12/01 11:46:55 DEBUG Build zipped package
2025/12/01 11:46:55 DEBUG Compress using archives.Zip (destination: /home/devuser/integrations/build/packages/axonius-0.1.0.zip)
2025/12/01 11:46:55 DEBUG Skip validation of the built .zip package
2025/12/01 11:46:56 DEBUG removing package...
--- Test results for package: axonius - START ---
╭─────────┬─────────────┬───────────┬──────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                                                        │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼──────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ axonius │             │ asset     │ dashboard axonius-428b4f4d-be94-4840-8dc2-8f03a38f97db is loaded │ PASS   │      1.171µs │
│ axonius │             │ asset     │ search axonius-df393688-9361-4015-b5b7-a93420fa0429 is loaded    │ PASS   │        321ns │
│ axonius │             │ asset     │ search axonius-e83a650f-80cc-430f-be10-dd6e0ff11855 is loaded    │ PASS   │        373ns │
│ axonius │ gateway     │ asset     │ index_template logs-axonius.gateway is loaded                    │ PASS   │        265ns │
│ axonius │ gateway     │ asset     │ ingest_pipeline logs-axonius.gateway-0.1.0 is loaded             │ PASS   │        260ns │
╰─────────┴─────────────┴───────────┴──────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: axonius - END   ---
Done
Run pipeline tests for the package
2025/12/01 11:46:58 DEBUG Connecting with Elasticsearch host from current profile (profile: default, host: "https://127.0.0.1:9200")
2025/12/01 11:46:58 DEBUG Running tests sequentially
2025/12/01 11:46:59 DEBUG Imported ECS fields definition from external schema for validation (embedded in package: false, stack uses ecs@mappings template: true)
2025/12/01 11:46:59 DEBUG Dump Elastic stack data
2025/12/01 11:46:59 DEBUG Dump stack logs for elasticsearch
--- Test results for package: axonius - START ---
╭─────────┬─────────────┬───────────┬─────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                                   │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼─────────────────────────────────────────────┼────────┼──────────────┤
│ axonius │ gateway     │ pipeline  │ (ingest pipeline warnings test-gateway.log) │ PASS   │ 339.733202ms │
│ axonius │ gateway     │ pipeline  │ test-gateway.log                            │ PASS   │ 168.189528ms │
╰─────────┴─────────────┴───────────┴─────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: axonius - END   ---
Done
Run policy tests for the package
2025/12/01 11:46:59 DEBUG Connecting with Kibana host from current profile (profile: default, host: "https://127.0.0.1:5601")
--- Test results for package: axonius - START ---
No test results
--- Test results for package: axonius - END   ---
Done
Run script tests for the package
--- Test results for package: axonius - START ---
PKG axonius
[no test files]
--- Test results for package: axonius - END ---
Done
Run static tests for the package
2025/12/01 11:46:59 DEBUG Running tests sequentially
2025/12/01 11:46:59 DEBUG Imported ECS fields definition from external schema for validation (embedded in package: false, stack uses ecs@mappings template: true)
--- Test results for package: axonius - START ---
╭─────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ axonius │ gateway     │ static    │ Verify sample_event.json │ PASS   │ 133.970037ms │
╰─────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: axonius - END   ---
Done
Run system tests for the package
2025/12/01 11:46:59 DEBUG Connecting with Kibana host from current profile (profile: default, host: "https://127.0.0.1:5601")
2025/12/01 11:46:59 DEBUG Connecting with Elasticsearch host from current profile (profile: default, host: "https://127.0.0.1:9200")
2025/12/01 11:46:59 DEBUG Running suite...
2025/12/01 11:46:59 DEBUG Running system tests for data stream "gateway"
2025/12/01 11:46:59 DEBUG System runner: data stream "gateway" config file "test-default-config.yml" variant ""
2025/12/01 11:46:59  INFO Installing package...
2025/12/01 11:46:59 DEBUG Build directory: /home/devuser/integrations/build/packages/axonius/0.1.0
2025/12/01 11:46:59 DEBUG Clear target directory (path: /home/devuser/integrations/build/packages/axonius/0.1.0)
2025/12/01 11:46:59 DEBUG Copy package content (source: /home/devuser/integrations/packages/axonius)
2025/12/01 11:46:59 DEBUG Copy license file if needed
2025/12/01 11:46:59 DEBUG No license text file is included in package
2025/12/01 11:46:59 DEBUG Include linked files
2025/12/01 11:46:59 DEBUG Encode dashboards
2025/12/01 11:46:59 DEBUG Resolve external fields
2025/12/01 11:46:59 DEBUG Package has external dependencies defined
2025/12/01 11:47:00 DEBUG data_stream/gateway/fields/base-fields.yml: source file has been changed
2025/12/01 11:47:00 DEBUG data_stream/gateway/fields/ecs.yml: source file has been changed
2025/12/01 11:47:00 DEBUG Package doesn't have to import ECS mappings
2025/12/01 11:47:00 DEBUG Build zipped package
2025/12/01 11:47:00 DEBUG Compress using archives.Zip (destination: /home/devuser/integrations/build/packages/axonius-0.1.0.zip)
2025/12/01 11:47:00 DEBUG Skip validation of the built .zip package
2025/12/01 11:47:11 DEBUG Running tests sequentially
2025/12/01 11:47:11 DEBUG Using config: "default"
2025/12/01 11:47:11  INFO Running test for data_stream "gateway" with configuration 'default'
2025/12/01 11:47:11 DEBUG creating enroll policy...
2025/12/01 11:47:15 DEBUG creating test policy...
2025/12/01 11:47:19  INFO Setting up independent Elastic Agent...
2025/12/01 11:47:19 DEBUG setting up agent using Docker Compose agent deployer
[+] Running 2/2
 ✔ Network elastic-package-agent-axonius-gateway-12550_default            Created                                                                                 0.1s 
 ✔ Container elastic-package-agent-axonius-gateway-12550-elastic-agent-1  Started                                                                                 0.3s 
2025/12/01 11:47:20 DEBUG Wait for healthy containers: be9e0e665fb8
2025/12/01 11:47:20 DEBUG Container elastic-agent (be9e0e665fb8) status: unhealthy
2025/12/01 11:47:21 DEBUG Container elastic-agent (be9e0e665fb8) status: unhealthy
2025/12/01 11:47:22 DEBUG Container elastic-agent (be9e0e665fb8) status: unhealthy
2025/12/01 11:47:23 DEBUG Container elastic-agent (be9e0e665fb8) status: unhealthy
2025/12/01 11:47:24 DEBUG Container elastic-agent (be9e0e665fb8) status: unhealthy
2025/12/01 11:47:25 DEBUG Container elastic-agent (be9e0e665fb8) status: unhealthy
2025/12/01 11:47:26 DEBUG Container elastic-agent (be9e0e665fb8) status: running (health: healthy)
2025/12/01 11:47:26 DEBUG adding service container elastic-package-agent-axonius-gateway-12550-elastic-agent-1 internal ports to context
2025/12/01 11:47:26 DEBUG found 0 enrolled agent(s)
2025/12/01 11:47:27 DEBUG found 0 enrolled agent(s)
2025/12/01 11:47:28 DEBUG found 0 enrolled agent(s)
2025/12/01 11:47:29 DEBUG found 0 enrolled agent(s)
2025/12/01 11:47:30 DEBUG found 0 enrolled agent(s)
2025/12/01 11:47:31 DEBUG found 0 enrolled agent(s)
2025/12/01 11:47:32 DEBUG found 0 enrolled agent(s)
2025/12/01 11:47:33 DEBUG found 1 enrolled agent(s)
2025/12/01 11:47:33 DEBUG Selected enrolled agent "22725dc2-af9f-449f-b292-4000608e286a"
2025/12/01 11:47:33  INFO Setting up service...
2025/12/01 11:47:33 DEBUG setting up service using Docker Compose service deployer
WARN[0000] /home/devuser/integrations/packages/axonius/_dev/deploy/docker/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
[+] Running 2/2
 ✔ Network elastic-package-service-96184_default      Created                                                                                                     0.1s 
 ✔ Container elastic-package-service-96184-axonius-1  Started                                                                                                     0.3s 
WARN[0000] /home/devuser/integrations/packages/axonius/_dev/deploy/docker/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
2025/12/01 11:47:34 DEBUG Wait for healthy containers: 8cecb6ac56b6
2025/12/01 11:47:34 DEBUG Container axonius (8cecb6ac56b6) status: running (no health status)
2025/12/01 11:47:34 DEBUG adding service container elastic-package-service-96184-axonius-1 internal ports to context
WARN[0000] /home/devuser/integrations/packages/axonius/_dev/deploy/docker/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
2025/12/01 11:47:35 DEBUG adding package data stream to test policy...
2025/12/01 11:47:38 DEBUG found 1 enrolled agent(s)
2025/12/01 11:47:38 DEBUG Selected enrolled agent "22725dc2-af9f-449f-b292-4000608e286a"
2025/12/01 11:47:38 DEBUG Set Debug log level to agent
2025/12/01 11:47:40 DEBUG assigning package data stream to agent...
2025/12/01 11:47:41 DEBUG Wait until the policy (ID: 95941176-8943-4a96-8018-19251c44f55a, revision: 2) is assigned to the agent (ID: 22725dc2-af9f-449f-b292-4000608e286a)...
2025/12/01 11:47:41 DEBUG Agent 22725dc2-af9f-449f-b292-4000608e286a (Host: elastic-agent-12550): Policy ID 95941176-8943-4a96-8018-19251c44f55a LogLevel: info Status: updating
2025/12/01 11:47:43 DEBUG Agent 22725dc2-af9f-449f-b292-4000608e286a (Host: elastic-agent-12550): Policy ID 95941176-8943-4a96-8018-19251c44f55a LogLevel: info Status: updating
2025/12/01 11:47:45 DEBUG Agent 22725dc2-af9f-449f-b292-4000608e286a (Host: elastic-agent-12550): Policy ID 95941176-8943-4a96-8018-19251c44f55a LogLevel: debug Status: updating
2025/12/01 11:47:47 DEBUG Agent 22725dc2-af9f-449f-b292-4000608e286a (Host: elastic-agent-12550): Policy ID 95941176-8943-4a96-8018-19251c44f55a LogLevel: debug Status: updating
2025/12/01 11:47:49 DEBUG Agent 22725dc2-af9f-449f-b292-4000608e286a (Host: elastic-agent-12550): Policy ID 95941176-8943-4a96-8018-19251c44f55a LogLevel: debug Status: online
2025/12/01 11:47:49 DEBUG Policy revision assigned to the agent (ID: 22725dc2-af9f-449f-b292-4000608e286a)...
2025/12/01 11:47:49 DEBUG checking for expected data in data stream (10m0s)...
2025/12/01 11:47:49 DEBUG found 0 hits in logs-axonius.gateway-44931 data stream
2025/12/01 11:47:50 DEBUG found 0 hits in logs-axonius.gateway-44931 data stream
2025/12/01 11:47:51 DEBUG found 0 hits in logs-axonius.gateway-44931 data stream
2025/12/01 11:47:52 DEBUG found 2 hits in logs-axonius.gateway-44931 data stream
2025/12/01 11:47:56 DEBUG found 2 hits in logs-axonius.gateway-44931 data stream
WARN[0000] /home/devuser/integrations/packages/axonius/_dev/deploy/docker/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
2025/12/01 11:47:56 DEBUG Found 0 deprecation warnings for data stream logs-axonius.gateway-44931
2025/12/01 11:47:56 DEBUG Check whether or not synthetic source mode is enabled (data stream logs-axonius.gateway-44931)...
2025/12/01 11:47:56 DEBUG Data stream logs-axonius.gateway-44931 has synthetic source mode enabled: false
2025/12/01 11:47:56  INFO Validating test case...
2025/12/01 11:47:56 DEBUG Imported ECS fields definition from external schema for validation (embedded in package: false, stack uses ecs@mappings template: true)
2025/12/01 11:47:56 DEBUG Performing validation based on mappings
2025/12/01 11:47:56 DEBUG Get Mappings from data stream (logs-axonius.gateway-44931)
2025/12/01 11:47:56 DEBUG Simulate Index Template (logs-axonius.gateway)
2025/12/01 11:47:56 DEBUG assert hit count expected 2, observed 2
2025/12/01 11:47:56  INFO Tearing down service...
2025/12/01 11:47:56 DEBUG tearing down service using Docker Compose runner
WARN[0000] /home/devuser/integrations/packages/axonius/_dev/deploy/docker/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
[+] Stopping 1/1
 ✔ Container elastic-package-service-96184-axonius-1  Stopped                                                                                                     0.4s 
WARN[0000] /home/devuser/integrations/packages/axonius/_dev/deploy/docker/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
2025/12/01 11:47:57  INFO Write container logs to file: /home/devuser/integrations/build/container-logs/axonius-1764569877426022181.log
WARN[0000] /home/devuser/integrations/packages/axonius/_dev/deploy/docker/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
[+] Running 2/2
 ✔ Container elastic-package-service-96184-axonius-1  Removed                                                                                                     0.0s 
 ✔ Network elastic-package-service-96184_default      Removed                                                                                                     0.3s 
2025/12/01 11:47:57 DEBUG Deleting data stream for testing logs-axonius.gateway-44931
2025/12/01 11:47:57 DEBUG removing agent...
2025/12/01 11:48:00  INFO Tearing down agent...
2025/12/01 11:48:00 DEBUG tearing down agent using Docker Compose runner
2025/12/01 11:48:00  INFO Write container logs to file: /home/devuser/integrations/build/container-logs/elastic-agent-1764569880313961631.log
[+] Running 2/2
 ✔ Container elastic-package-agent-axonius-gateway-12550-elastic-agent-1  Removed                                                                                 1.5s 
 ✔ Network elastic-package-agent-axonius-gateway-12550_default            Removed                                                                                 0.3s 
2025/12/01 11:48:02 DEBUG deleting test policies...
2025/12/01 11:48:08 DEBUG Dump Elastic stack data (location: /tmp/test-system-874887537)
2025/12/01 11:48:08 DEBUG Dump stack logs for elastic-agent
2025/12/01 11:48:09 DEBUG Dump stack logs for fleet-server
2025/12/01 11:48:09 DEBUG Dump stack logs for kibana
2025/12/01 11:48:09 DEBUG Dump stack logs for package-registry
2025/12/01 11:48:10 DEBUG Dump stack logs for elasticsearch
2025/12/01 11:48:10  INFO Uninstalling package...
--- Test results for package: axonius - START ---
╭─────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤
│ axonius │ gateway     │ system    │ default   │ PASS   │ 44.932970426s │
╰─────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: axonius - END   ---
Done

Screenshots

image (5) image (6)

Go Code for Ingest Pipeline Generation

The gateway data stream pipeline is generated using Go code built on top of the Dispear library.
Below is the code used for generating the pipeline logic:

package main

import (
	"fmt"
	"strings"

	. "github.com/efd6/dispear"
)

const (
	ECSVersion = "9.2.0"
	PkgRoot    = "axonius.gateway"
)
const errorFormat = "Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}"

// removeErrorHandler generates a series of Renders that first remove the given field
// from the document and then append a custom error message to the 'error.message' field.
// This is typically used to handle errors in an Ingest Pipeline by removing a field that
// caused an issue and appending a formatted error message to the document.
func removeErrorHandler(f string) []Renderer {
	return []Renderer{
		REMOVE(f),
		APPEND("error.message", errorFormat),
	}
}

// safeNavigateAndCheck converts a dot-separated field path to a safe navigation string.
//
// Example:
// "parent.child.grandchild" -> "ctx?.parent?.child?.grandchild"
func safeNavigateAndCheck(field string) string {
	parts := strings.Split(field, ".")
	condition := "ctx"
	for i, part := range parts {
		if i > 0 { // Skip the first part which is already included in the condition
			condition += fmt.Sprintf("?.%s", part)
		} else {
			condition += fmt.Sprintf(".%s", part)
		}
	}
	return condition
}

func main() {

	// Initial processors of pipeline

	DESCRIPTION("Pipeline for processing gateway logs.")

	SET("ecs.version").VALUE(ECSVersion)

	TERMINATE("data collection error").
		IF("ctx.error?.message != null && ctx.message == null && ctx.event?.original == null").
		DESCRIPTION("error message set and no data to process.")

	BLANK()
	BLANK().COMMENT("remove agentless metadata")

	REMOVE(
		"organization",
		"division",
		"team",
	).
		IF("ctx.organization instanceof String && ctx.division instanceof String && ctx.team instanceof String").
		IGNORE_MISSING(true).
		TAG("remove_agentless_tags").
		DESCRIPTION("Removes the fields added by Agentless as metadata, as they can collide with ECS fields.")

	BLANK()
	BLANK().COMMENT("parse the event JSON")

	RENAME("message", "event.original").
		IF("ctx.event?.original == null").
		DESCRIPTION("Renames the original `message` field to `event.original` to store a copy of the original message. The `event.original` field is not touched if the document already has one; it may happen when Logstash sends the document.").
		IGNORE_MISSING(true)

	REMOVE("message").
		TAG("remove_message").
		IF("ctx.event?.original != null").
		DESCRIPTION("The `message` field is no longer required if the document has an `event.original` field.").
		IGNORE_MISSING(true)

	JSON(PkgRoot, "event.original")

	// Add fingerprint

	BLANK()
	BLANK().COMMENT("Add fingerprint")

	FINGERPRINT("_id", "axonius.gateway.id").IGNORE_MISSING(true)

	// Setting event.* and observer.* fields

	BLANK()
	BLANK().COMMENT("Set event.* and observer.* fields")

	SET("event.kind").VALUE("event")
	SET("observer.vendor").VALUE("Axonius")
	SET("observer.version").VALUE("7.0")

	// Convert to boolean

	BLANK()
	BLANK().COMMENT("Convert to Boolean Processors")

	for _, field := range []string{
		"axonius.gateway.default",
		"axonius.gateway.email_when_connected",
		"axonius.gateway.email_when_disconnected",
		"axonius.gateway.tunnel_proxy_settings.enabled",
	} {
		CONVERT("", field, "boolean").
			IGNORE_MISSING(true).
			ON_FAILURE(removeErrorHandler(field)...)
	}

	// Convert to String

	BLANK()
	BLANK().COMMENT("Convert to String Processors")

	for _, field := range []string{
		"axonius.gateway.tunnel_proxy_settings.tunnel_proxy_port",
	} {
		CONVERT("", field, "string").
			IGNORE_MISSING(true)
	}

	// Convert to IP

	BLANK()
	BLANK().COMMENT("Convert to IP")

	for _, field := range []string{
		"axonius.gateway.dns_server",
	} {
		CONVERT("", field, "ip").
			IGNORE_MISSING(true).
			IF(safeNavigateAndCheck(field) + " != ''").
			ON_FAILURE(removeErrorHandler(field)...)
	}
	// Set ECS Mapping

	BLANK()
	BLANK().COMMENT("Map custom fields to corresponding ECS and related fields.")

	// Map ECS mapping for top-level fields

	for _, mapping := range []struct {
		ecsField, customField string
	}{
		{ecsField: "event.id", customField: "axonius.gateway.id"},
	} {
		SET(mapping.ecsField).COPY_FROM(mapping.customField).IGNORE_EMPTY(true)
	}

	for _, mapping := range []struct {
		ecsField, customField string
	}{
		{ecsField: "dns.resolved_ip", customField: "axonius.gateway.dns_server"},
	} {
		APPEND(mapping.ecsField, "{{{"+mapping.customField+"}}}").
			IF(safeNavigateAndCheck(mapping.customField) + " != null").
			ALLOW_DUPLICATES(false)
	}
	// Map ECS mapping for array fields

	FOREACH("axonius.gateway.email_recipients",
		APPEND("email.to.address", "{{{_ingest._value}}}").
			ALLOW_DUPLICATES(false),
	).IF("ctx.axonius?.gateway?.email_recipients instanceof List")

	SCRIPT().
		TAG("script_to_map_event_outcome").
		DESCRIPTION("Map event outcome based on value of field status.").
		LANG("painless").
		IF("ctx.axonius?.gateway?.status != null").
		SOURCE(`
			ctx.event.put("outcome", params.get(ctx.axonius.gateway.status.toLowerCase()));
		`).
		PARAMS(map[string]any{
			"success": "success",
			"error":   "failure",
		})

	// Map related mapppings
	for _, mapping := range []struct {
		ecsField, customField string
	}{
		{ecsField: "related.user", customField: "axonius.gateway.tunnel_proxy_settings.tunnel_proxy_user"},
		{ecsField: "related.ip", customField: "axonius.gateway.dns_server"},
	} {
		APPEND(mapping.ecsField, "{{{"+mapping.customField+"}}}").
			IF(safeNavigateAndCheck(mapping.customField) + " != null").
			ALLOW_DUPLICATES(false)
	}

	// Remove Duplicate Fields.

	BLANK()
	BLANK().COMMENT("Remove Duplicate Custom Field if preserve_duplicate_custom_fields are not enabled")

	REMOVE(
		"axonius.gateway.dns_server",
		"axonius.gateway.email_recipients",
		"axonius.gateway.id",
	).
		IF("ctx.tags == null || !ctx.tags.contains('preserve_duplicate_custom_fields')").
		TAG("remove_custom_duplicate_fields").
		IGNORE_MISSING(true)

	// Clean up script

	BLANK()
	BLANK().COMMENT("Cleanup")

	SCRIPT().
		TAG("script_to_drop_null_values").
		DESCRIPTION("This script processor iterates over the whole document to remove fields with null values.").
		LANG("painless").
		SOURCE(`
		void handleMap(Map map) {
		map.values().removeIf(v -> {
			if (v instanceof Map) {
			handleMap(v);
			} else if (v instanceof List) {
			handleList(v);
			}
			return v == null || v == '' || (v instanceof Map && v.size() == 0) || (v instanceof List && v.size() == 0)
		});
		}
		void handleList(List list) {
		list.removeIf(v -> {
			if (v instanceof Map) {
			handleMap(v);
			} else if (v instanceof List) {
			handleList(v);
			}
			return v == null || v == '' || (v instanceof Map && v.size() == 0) || (v instanceof List && v.size() == 0)
		});
		}
		handleMap(ctx);
		`)

	// Set and Append processor on last

	SET("event.kind").VALUE("pipeline_error").IF("ctx.error?.message != null")
	APPEND("tags", "preserve_original_event").
		IF("ctx.error?.message != null").
		ALLOW_DUPLICATES(false)

	// Global on failure processor

	ON_FAILURE(
		APPEND("error.message", errorFormat),
		SET("event.kind").VALUE("pipeline_error"),
		APPEND("tags", "preserve_original_event").
			ALLOW_DUPLICATES(false),
	)

	// Generate the pipeline

	Generate()
}

@muskan-agarwal26 muskan-agarwal26 requested a review from a team as a code owner December 1, 2025 05:12
@muskan-agarwal26 muskan-agarwal26 marked this pull request as draft December 1, 2025 05:17
@muskan-agarwal26 muskan-agarwal26 changed the base branch from main to feature/axonius-0.1.0 December 1, 2025 08:23
@muskan-agarwal26 muskan-agarwal26 changed the title Initial release of Gateway datastream in Axonius [axonius][gateway] Add Axonius Gateway datastream Dec 1, 2025
@andrewkroh andrewkroh added Crest Contributions from Crest developement team. New Integration Issue or pull request for creating a new integration package. dashboard Relates to a Kibana dashboard bug, enhancement, or modification. documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:axonius [Integration not found in source] labels Dec 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

Crest Contributions from Crest developement team. dashboard Relates to a Kibana dashboard bug, enhancement, or modification. documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:axonius [Integration not found in source] New Integration Issue or pull request for creating a new integration package.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@muskan-agarwal26 @andrewkroh @muskan-crest