From d28810606557c8253f4fca5fbd05ff76960b8914 Mon Sep 17 00:00:00 2001 From: Ryan Haney Date: Wed, 13 Aug 2025 17:01:15 -0500 Subject: [PATCH 1/3] Staging VPC Flow Sensor v0.3.0 docs --- terraform/aws-flow-sensor/README.md | 109 ++++++++++++++++++++++++---- 1 file changed, 96 insertions(+), 13 deletions(-) diff --git a/terraform/aws-flow-sensor/README.md b/terraform/aws-flow-sensor/README.md index 40130d9..297e295 100644 --- a/terraform/aws-flow-sensor/README.md +++ b/terraform/aws-flow-sensor/README.md @@ -5,35 +5,66 @@ This directory provides Terraform code for deploying Corelight's AWS Flow Sensor ## Overview This example uses the [terraform-aws-single-sensor](https://github.com/corelight/terraform-aws-single-sensor) module -to simplify the deployment of the Flow sensor and includes example resources for authorizing it to the VPC Flow s3 bucket. +to simplify the deployment of the Flow Sensor and includes example resources for authorizing it to the VPC Flow s3 bucket. ## Requirements & Considerations -* A Flow Sensor must be deployed in each AWS account +* A Flow Sensor has a 1:1 association with an S3 bucket +* Many Accounts can feed flows to the single S3 bucket +* Any account sending flows to the S3 bucket will need a cross account role deployed * The sensor should be deployed similarly to a traditional sensor with a separate management and monitoring subnet * VPC Flow Logs will only be processed for VPCs with flow log configurations matching the following criteria: * Log Destination Target is `s3` * AWS Default (v2) Log Format * `plain-text` File Format * `Per Hour Partition` and `Hive Compatible Partitions` are disabled +* Only flow log configuration S3 destinations with one level of "folder" (prefix) are supported + * supported: `arn:aws:s3:::bucket` + * supported: `arn:aws:s3:::bucket/production` + * not supported: `arn:aws:s3:::bucket/not/this` ## Configuration Once connected to Fleet, configure the AWS VPC Flow feature (Private Preview) under `Advanced` as follows * Enable the feature by switching on `cloud_vpc_flow.enable` * All configurations below begin with `cloud_vpc_flow.` -| Configuration | Required | Type | Default Region | Purpose | Example | -|---------------------|----------|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------|-------------------------| -| `start_date` | YES | string | N/A | Date to begin processing
flow logs in AWS format | `2025/06/01` | -| `log_level` | NO | string | `info` | The log level of the service | `debug` to troubleshoot | -| `monitored_vpcs` | NO | string | `null` | | `vpc-12345,vpc-54321` | -| `monitored_regions` | NO | string | `us-east-1`
`us-east-2`
`us-west-1`
`us-west-2`
`ap-south-1`
`ap-northeast-1`
`ap-northeast-2`
`ap-northeast-3`
`ap-southeast-1`
`ap-southeast-2`
`ca-central-1`
`eu-central-1`
`eu-west-2`
`eu-west-3`
`eu-north-1`
`sa-east-1` | Regions to enumerate
for compatible
configurations | `us-east-1,us-east-2` | -| `s3_bucket_prefix` | NO | string | `AWSLogs` | VPC flow log s3 object prefix | `AWSLogs` | +| Configuration | Required | Type | Default Region | Purpose | Example | +|---------------------------|----------|--------|---------------------------|-----------------------------------------------------------------------------------|-----------------------------------------| +| `start_date` | YES | string | N/A | Date to begin processing flows in AWS format | `2025/06/01` | +| `bucket_name` | YES | string | N/A | VPC flow log s3 bucket name | `vpc-flow-logs` | +| `bucket_region` | YES | string | N/A | VPC flow log bucket region | `us-east-1` | +| `log_level` | NO | string | `info` | The log level of the service | `debug` | +| `monitored_accounts` | NO | string | `nil` | Filters which account(s) the Flow Sensor will process logs | `111111111111,222222222222` | +| `monitored_vpcs` | NO | string | `nil` | Filters which VPC(s) the Flow Sensor will process logs | `vpc-12345,vpc-54321` | +| `monitored_regions` | NO | string | Default Region List Below | Filters which region(s) the Flow Sensor will process logs | `us-east-1,us-east-2` | +| `cross_account_role_name` | NO | string | `nil` | Name of the cross account role the Flow Sensor should assume into in each account | `corelight-vpc-flow-cross-account-role` | -## Limitations -* While a Flow Sensor can read from a s3 bucket that includes VPC Flow logs for multiple accounts, it will only process - logs for the account in which it is deployed. +### Default AWS Region List +* `us-east-1` +* `us-east-2` +* `us-west-1` +* `us-west-2` +* `ap-south-1` +* `ap-northeast-1` +* `ap-northeast-2` +* `ap-northeast-3` +* `ap-southeast-1` +* `ap-southeast-2` +* `ca-central-1` +* `eu-central-1` +* `eu-west-1` +* `eu-west-2` +* `eu-west-3` +* `eu-north-1` +* `sa-east-1` -## IAM Policy JSON + +## Flow Sensor IAM + +### Ec2 Instance Profile Role +The EC2 instance will need permissions to read and list objects in the VPC Flow S3 bucket as well as enumerate VPCs and +VPC Flow Log configurations associated with VPCs in the account it is deployed. If the Flow Sensor does not need to +process any flows in the account it is deployed then the `ec2:DescribeVPCs` and `ec2:DescribeFlowLogs` permissions +can be removed. ```json { "Statement": [ @@ -59,4 +90,56 @@ Once connected to Fleet, configure the AWS VPC Flow feature (Private Preview) un ], "Version": "2012-10-17" } +``` + +## Processing Flow Logs From Other Accounts +VPC Flow S3 buckets can contain flows from several other accounts. The Flow Sensor requires a cross account role +in any account sending logs to the bucket it is paired with to process them. Accounts found in the bucket where a cross +account role is not provisioned will be ignored. + +### Example +If the VPC Flow Sensor is deployed in `111111111111` paired with `vpc-flow-bucket` then it would need a cross account role +to enumerate VPC Flow configuration information in `222222222222` and `333333333333` +``` +vpc-flow-bucket/ + -> AWSLogs/111111111111/ + -> AWSLogs/222222222222/ + -> folder/AWSLogs/333333333333/ +``` + +### Cross Account Trust Policy +Grant the IAM Role associated with your Flow Sensor EC2 instance profile access to enumerate VPCs and Flow log +configurations in the other account(s) +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "AWS": [ + "arn:aws:iam::111111111111:role/corelight-vpc-flow-role" + ] + }, + "Action": "sts:AssumeRole" + } + ] +} +``` + +### Cross Account Role Policy +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "ec2:DescribeVpcs", + "ec2:DescribeFlowLogs" + ], + "Resource": "*" + } + ] +} ``` \ No newline at end of file From 63337610be60ee087a782998c8040631f9d18cb9 Mon Sep 17 00:00:00 2001 From: Ryan Haney Date: Mon, 25 Aug 2025 16:29:24 -0500 Subject: [PATCH 2/3] updating the docs --- terraform/aws-flow-sensor/README.md | 82 +++++++++++++++++------------ 1 file changed, 49 insertions(+), 33 deletions(-) diff --git a/terraform/aws-flow-sensor/README.md b/terraform/aws-flow-sensor/README.md index 297e295..5506316 100644 --- a/terraform/aws-flow-sensor/README.md +++ b/terraform/aws-flow-sensor/README.md @@ -9,34 +9,34 @@ to simplify the deployment of the Flow Sensor and includes example resources for ## Requirements & Considerations * A Flow Sensor has a 1:1 association with an S3 bucket -* Many Accounts can feed flows to the single S3 bucket -* Any account sending flows to the S3 bucket will need a cross account role deployed +* Many Accounts can send flows to the single S3 bucket +* Any account sending flows to the S3 bucket will need a cross account role deployed and the flow sensor will need + permission to assume role * The sensor should be deployed similarly to a traditional sensor with a separate management and monitoring subnet * VPC Flow Logs will only be processed for VPCs with flow log configurations matching the following criteria: * Log Destination Target is `s3` * AWS Default (v2) Log Format * `plain-text` File Format * `Per Hour Partition` and `Hive Compatible Partitions` are disabled -* Only flow log configuration S3 destinations with one level of "folder" (prefix) are supported - * supported: `arn:aws:s3:::bucket` - * supported: `arn:aws:s3:::bucket/production` - * not supported: `arn:aws:s3:::bucket/not/this` - -## Configuration -Once connected to Fleet, configure the AWS VPC Flow feature (Private Preview) under `Advanced` as follows -* Enable the feature by switching on `cloud_vpc_flow.enable` -* All configurations below begin with `cloud_vpc_flow.` - -| Configuration | Required | Type | Default Region | Purpose | Example | -|---------------------------|----------|--------|---------------------------|-----------------------------------------------------------------------------------|-----------------------------------------| -| `start_date` | YES | string | N/A | Date to begin processing flows in AWS format | `2025/06/01` | -| `bucket_name` | YES | string | N/A | VPC flow log s3 bucket name | `vpc-flow-logs` | -| `bucket_region` | YES | string | N/A | VPC flow log bucket region | `us-east-1` | -| `log_level` | NO | string | `info` | The log level of the service | `debug` | -| `monitored_accounts` | NO | string | `nil` | Filters which account(s) the Flow Sensor will process logs | `111111111111,222222222222` | -| `monitored_vpcs` | NO | string | `nil` | Filters which VPC(s) the Flow Sensor will process logs | `vpc-12345,vpc-54321` | -| `monitored_regions` | NO | string | Default Region List Below | Filters which region(s) the Flow Sensor will process logs | `us-east-1,us-east-2` | -| `cross_account_role_name` | NO | string | `nil` | Name of the cross account role the Flow Sensor should assume into in each account | `corelight-vpc-flow-cross-account-role` | + * The S3 bucket destination for VPC Flow Logs must be structured with a single prefix level. The flow sensor will not + be able to process logs from S3 buckets with nested prefixes. + * Supported: `arn:aws:s3:::bucket-name/prefix` + * Not Supported: `arn:aws:s3:::bucket-name/prefix/another-prefix` + +## Configuration +Once paired with Fleet, configure the AWS VPC Flow feature (Private Preview) under `Advanced` as follows: + +| Configuration | Required | Default | Purpose | Example | +|------------------------------------|----------|---------------------------|-----------------------------------------------------------------------------------|-----------------------------------------| +| `vpc_flow.enable` | YES | N/A | Enables the service | Toggle On | +| `vpc_flow.bucket_name` | YES | N/A | VPC flow log s3 bucket name | `vpc-flow-logs` | +| `vpc_flow.bucket_region` | YES | N/A | VPC flow log bucket region | `us-east-1` | +| `vpc_flow.start_date` | YES | N/A | Date to begin processing flows in AWS format | `2025/06/01` | +| `vpc_flow.log_level` | NO | `info` | The log level of the service | `debug` | +| `vpc_flow.monitored_accounts` | NO | `nil` | Filters which account(s) the Flow Sensor will process logs | `111111111111,222222222222` | +| `vpc_flow.monitored_vpcs` | NO | `nil` | Filters which VPC(s) the Flow Sensor will process logs | `vpc-12345,vpc-54321` | +| `vpc_flow.monitored_regions` | NO | Default Region List Below | Filters which region(s) the Flow Sensor will process logs | `us-east-1,us-east-2` | +| `vpc_flow.cross_account_role_name` | NO | `nil` | Name of the cross account role the Flow Sensor should assume into in each account | `corelight-vpc-flow-cross-account-role` | ### Default AWS Region List * `us-east-1` @@ -60,11 +60,13 @@ Once connected to Fleet, configure the AWS VPC Flow feature (Private Preview) un ## Flow Sensor IAM -### Ec2 Instance Profile Role +### EC2 Instance Profile Role The EC2 instance will need permissions to read and list objects in the VPC Flow S3 bucket as well as enumerate VPCs and -VPC Flow Log configurations associated with VPCs in the account it is deployed. If the Flow Sensor does not need to -process any flows in the account it is deployed then the `ec2:DescribeVPCs` and `ec2:DescribeFlowLogs` permissions +VPC Flow Log configurations associated with VPCs in the account it is deployed. If the Flow Sensor does not need to +process any flows in the account it is deployed then the `ec2:DescribeVPCs` and `ec2:DescribeFlowLogs` permissions can be removed. + +**Note:** Replace `` and `` with your specific values. ```json { "Statement": [ @@ -86,6 +88,11 @@ can be removed. ], "Effect": "Allow", "Resource": "*" + }, + { + "Effect": "Allow", + "Action": "sts:AssumeRole", + "Resource": "arn:aws:iam::*:role/" } ], "Version": "2012-10-17" @@ -94,22 +101,31 @@ can be removed. ## Processing Flow Logs From Other Accounts VPC Flow S3 buckets can contain flows from several other accounts. The Flow Sensor requires a cross account role -in any account sending logs to the bucket it is paired with to process them. Accounts found in the bucket where a cross -account role is not provisioned will be ignored. +in any account sending logs to the bucket it is paired with to process those logs. The Flow sensor will attempt to +assume into accounts found in the bucket with the configured cross account role and will ignore any that are inaccessible. ### Example -If the VPC Flow Sensor is deployed in `111111111111` paired with `vpc-flow-bucket` then it would need a cross account role -to enumerate VPC Flow configuration information in `222222222222` and `333333333333` +For example, if the VPC Flow Sensor is deployed in account `111111111111` and is paired with the `vpc-flow-bucket`, the +bucket may contain logs from other accounts. The following directory structure shows that logs are being sent from +accounts `222222222222` and `333333333333`. To process these logs, the Flow Sensor will need a cross-account role in +each of these accounts. + ``` vpc-flow-bucket/ - -> AWSLogs/111111111111/ - -> AWSLogs/222222222222/ - -> folder/AWSLogs/333333333333/ +|-- AWSLogs/ +| |-- 111111111111/ +| |-- 222222222222/ +|-- folder/ +| |-- AWSLogs/ +| |-- 333333333333/ ``` ### Cross Account Trust Policy Grant the IAM Role associated with your Flow Sensor EC2 instance profile access to enumerate VPCs and Flow log configurations in the other account(s) + +**Note:** Replace `` with the name of the IAM role associated with your Flow Sensor EC2 +instance profile. ```json { "Version": "2012-10-17", @@ -118,7 +134,7 @@ configurations in the other account(s) "Effect": "Allow", "Principal": { "AWS": [ - "arn:aws:iam::111111111111:role/corelight-vpc-flow-role" + "arn:aws:iam::111111111111:role/" ] }, "Action": "sts:AssumeRole" From 6acd97a4fb51bc00866fd5c3507b72e9bad14102 Mon Sep 17 00:00:00 2001 From: Ryan Haney Date: Wed, 27 Aug 2025 12:40:24 -0500 Subject: [PATCH 3/3] updating with PR recommendations and other changes --- terraform/aws-flow-sensor/README.md | 57 +++++++++++++++++------------ terraform/aws-flow-sensor/main.tf | 20 +++++++--- 2 files changed, 47 insertions(+), 30 deletions(-) diff --git a/terraform/aws-flow-sensor/README.md b/terraform/aws-flow-sensor/README.md index 5506316..8853acd 100644 --- a/terraform/aws-flow-sensor/README.md +++ b/terraform/aws-flow-sensor/README.md @@ -8,35 +8,39 @@ This example uses the [terraform-aws-single-sensor](https://github.com/corelight to simplify the deployment of the Flow Sensor and includes example resources for authorizing it to the VPC Flow s3 bucket. ## Requirements & Considerations -* A Flow Sensor has a 1:1 association with an S3 bucket -* Many Accounts can send flows to the single S3 bucket -* Any account sending flows to the S3 bucket will need a cross account role deployed and the flow sensor will need - permission to assume role -* The sensor should be deployed similarly to a traditional sensor with a separate management and monitoring subnet -* VPC Flow Logs will only be processed for VPCs with flow log configurations matching the following criteria: +* A Flow Sensor is paired (1:1) with an S3 bucket and should be deployed in the same region as the bucket +* Flow logs in the S3 bucket can originate from many regions, VPCs, and AWS accounts. Accounts that send flows to the S3 bucket will need a cross account role deployed and the Flow Sensor role will need + permission to assume into the cross account role + * Deploy the role with [Terraform](https://github.com/corelight/terraform-aws-single-sensor/tree/main/modules/vpc_flow_assume_role) +* Multiple Flow Sensors can monitor a single bucket by configuring specific regions, VPCs, and AWS account filters to handle + large scale VPC Flow log implementations +* The Flow Sensor should be deployed with a separate management and monitoring subnet + +## Supported Flow Log Configuration +Flow Logs will only be processed for VPCs with flow log configurations matching the following criteria: * Log Destination Target is `s3` * AWS Default (v2) Log Format * `plain-text` File Format * `Per Hour Partition` and `Hive Compatible Partitions` are disabled - * The S3 bucket destination for VPC Flow Logs must be structured with a single prefix level. The flow sensor will not - be able to process logs from S3 buckets with nested prefixes. - * Supported: `arn:aws:s3:::bucket-name/prefix` - * Not Supported: `arn:aws:s3:::bucket-name/prefix/another-prefix` + * The S3 bucket destination must be structured with, at most, a single prefix level. The flow sensor will not + be able to process logs from S3 buckets with nested prefixes. + * Supported: `arn:aws:s3:::bucket-name/prefix` + * Not Supported: `arn:aws:s3:::bucket-name/prefix/another-prefix` ## Configuration Once paired with Fleet, configure the AWS VPC Flow feature (Private Preview) under `Advanced` as follows: -| Configuration | Required | Default | Purpose | Example | -|------------------------------------|----------|---------------------------|-----------------------------------------------------------------------------------|-----------------------------------------| -| `vpc_flow.enable` | YES | N/A | Enables the service | Toggle On | -| `vpc_flow.bucket_name` | YES | N/A | VPC flow log s3 bucket name | `vpc-flow-logs` | -| `vpc_flow.bucket_region` | YES | N/A | VPC flow log bucket region | `us-east-1` | -| `vpc_flow.start_date` | YES | N/A | Date to begin processing flows in AWS format | `2025/06/01` | -| `vpc_flow.log_level` | NO | `info` | The log level of the service | `debug` | -| `vpc_flow.monitored_accounts` | NO | `nil` | Filters which account(s) the Flow Sensor will process logs | `111111111111,222222222222` | -| `vpc_flow.monitored_vpcs` | NO | `nil` | Filters which VPC(s) the Flow Sensor will process logs | `vpc-12345,vpc-54321` | -| `vpc_flow.monitored_regions` | NO | Default Region List Below | Filters which region(s) the Flow Sensor will process logs | `us-east-1,us-east-2` | -| `vpc_flow.cross_account_role_name` | NO | `nil` | Name of the cross account role the Flow Sensor should assume into in each account | `corelight-vpc-flow-cross-account-role` | +| Configuration | Required | Default | Purpose | Example | +|------------------------------------|----------|----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------| +| `vpc_flow.enable` | YES | N/A | Enables the service | Toggle On | +| `vpc_flow.bucket_name` | YES | N/A | VPC flow log s3 bucket name | `vpc-flow-logs` | +| `vpc_flow.bucket_region` | YES | N/A | VPC flow log bucket region | `us-east-1` | +| `vpc_flow.start_date` | YES | N/A | Date to begin processing flows in AWS format (YYYY/MM/DD) | `2025/06/01` | +| `vpc_flow.log_level` | NO | `info` | The log level of the service | `debug` | +| `vpc_flow.monitored_accounts` | NO | `nil` | Filters which account(s) the Flow Sensor will process logs. Attempts to process all accounts found in the bucket if not configured | `111111111111,222222222222` | +| `vpc_flow.monitored_vpcs` | NO | `nil` | Filters which VPC(s) the Flow Sensor will process logs. Attempts to process all VPCs with supported flow configurations found in each account if not configured | `vpc-12345,vpc-54321` | +| `vpc_flow.monitored_regions` | NO | `Default AWS Region List` | Filters which region(s) the Flow Sensor will process logs. Will enumerate VPCs in the `Default AWS Region List` if not configured | `us-east-1,us-east-2` | +| `vpc_flow.cross_account_role_name` | NO | `nil` | Name of the cross account role the Flow Sensor should assume into in each account. Will ignore any account that is not the account the Flow Sensor is deployed in if not configured. | `corelight-vpc-flow-cross-account-role` | ### Default AWS Region List * `us-east-1` @@ -100,15 +104,15 @@ can be removed. ``` ## Processing Flow Logs From Other Accounts -VPC Flow S3 buckets can contain flows from several other accounts. The Flow Sensor requires a cross account role +VPC Flow S3 buckets can contain flows from several accounts. The Flow Sensor requires a cross account role in any account sending logs to the bucket it is paired with to process those logs. The Flow sensor will attempt to assume into accounts found in the bucket with the configured cross account role and will ignore any that are inaccessible. ### Example For example, if the VPC Flow Sensor is deployed in account `111111111111` and is paired with the `vpc-flow-bucket`, the bucket may contain logs from other accounts. The following directory structure shows that logs are being sent from -accounts `222222222222` and `333333333333`. To process these logs, the Flow Sensor will need a cross-account role in -each of these accounts. +accounts `222222222222` and `333333333333`. The Flow Sensor will need a cross-account role in each of these accounts to +process the flow logs. ``` vpc-flow-bucket/ @@ -120,6 +124,11 @@ vpc-flow-bucket/ | |-- 333333333333/ ``` + +## Cross Account IAM +The cross account policy and role can be created manually or through our +[Terraform](https://github.com/corelight/terraform-aws-single-sensor/tree/main/modules/vpc_flow_assume_role) submodule. + ### Cross Account Trust Policy Grant the IAM Role associated with your Flow Sensor EC2 instance profile access to enumerate VPCs and Flow log configurations in the other account(s) diff --git a/terraform/aws-flow-sensor/main.tf b/terraform/aws-flow-sensor/main.tf index 629823e..267f523 100644 --- a/terraform/aws-flow-sensor/main.tf +++ b/terraform/aws-flow-sensor/main.tf @@ -41,8 +41,8 @@ data "aws_iam_policy_document" "flow_policy_data" { ] resources = [ // provide the flow sensor access to read from the flow log bucket - "arn:aws:s3:::/*", - "arn:aws:s3:::", + "arn:aws:s3:::/*", + "arn:aws:s3:::", ] } statement { @@ -53,6 +53,16 @@ data "aws_iam_policy_document" "flow_policy_data" { ] resources = ["*"] } + // Add if flows originate from other accounts to grant the ability to assume into other roles + statement { + effect = "Allow" + actions = [ + "sts:AssumeRole" + ] + resources = [ + "arn:aws:iam::*:role/" + ] + } } data "aws_iam_policy_document" "ec2_assume_policy" { @@ -67,14 +77,12 @@ data "aws_iam_policy_document" "ec2_assume_policy" { } resource "aws_iam_policy" "flow_policy" { - // Name the IAM policy - name = "" + name = "corelight-vpc-flow-sensor-policy" policy = data.aws_iam_policy_document.flow_policy_data.json } resource "aws_iam_role" "flow_role" { - // Name the flow sensor IAM role - name = "" + name = "corelight-vpc-flow-sensor-role" assume_role_policy = data.aws_iam_policy_document.ec2_assume_policy.json }