AWS S3 Splunk CII Integration

Overview

To establish an integration between Splunk and Cisco Identity Intelligence, please refer to the setup guide section in this document. This integration method utilizes the Splunk Add-on for AWS to ingest log data from your AWS S3 bucket. This method is frequently employed for batch processing of historical or high-volume log data. It requires setting up AWS credentials and the accurate specification of the SQS queue region and URL.

Prerequisites

Before you begin, please ensure you have the following:

1. Administrative access to your Cisco Identity Intelligence

2. Administrative access to your Splunk Enterprise or Splunk Cloud

3. The Cisco Security Cloud application installed from Splunkbase

1. The Splunk Add-on for AWS application installed from Splunkbase

2. Appropriate permissions to manage AWS S3 buckets and Splunk data inputs

Setup Guide

Configure AWS S3

1. Create an S3 Bucket

   a. Go to the Amazon S3 console

   b. Click Create bucket

   c. Provide a name for the bucket, such as "splunk-cii-demo-set-up"for this example. Keep this name handy, it will be used throughout the setup process.

   d. Click Create bucket to complete the process

2. Create an IAM Policy

   a. Navigate to IAM > Policies > Create Policy

b. Select JSON in the policy editor and paste the following JSON:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::${BUCKET_NAME}/*"
    }
  ]
}

c. Click Next and provide the policy with a name (In this example, we will use S3WritePolicy)

d. Click Create policy

1. Create an IAM Role

   a. Navigate to IAM > Roles > Create Role

b. Select Custom trust policy and paste the following JSON:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::${CII_ACCOUNT_ID}:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "ForAnyValue:StringEquals": {
          "sts:ExternalId": "${UNIQUE_EXTERNAL_ID}"
        }
      }
    }
  ]
}

c. Click Next.

d. Attach the (S3WritePolicy) policy (created earlier)

e. Provide the role with the name CrossAccountS3WriteRole (This name is required and must not be changed)

f. Click Create role

1. Update the S3 Bucket Policy

   a. Go to the Amazon S3 console and select the splunk-cii-demo-set-up bucket.

   b. Navigate to the Permissions tab > Bucket Policy > Edit

c. Paste the following JSON into the bucket policy editor:

Notes:

• Replace ${S3_ACCOUNT_ID} with your AWS account ID

• Replace ${ROLE_NAME} with CrossAccountS3WriteRole

• Replace ${BUCKET_NAME} with your bucket name, e.g. splunk-cii-set-up

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::${S3_ACCOUNT_ID}:role/${ROLE_NAME}"
      },
      "Action": [
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::${BUCKET_NAME}/*"
    }
  ]
}

d. Save changes

1. Create an SNS Topic

   a. Go to the Amazon SNS console > Create Topic

b. Provide a name: SplunkSNS

c. In the Access Policy, select Advanced and paste the following JSON: Note: Replace ${BUCKET_NAME} with your bucket name, e.g. splunk-cii-set-up

{
  "Version": "2008-10-17",
  "Id": "PolicyForS3Access",
  "Statement": [
    {
      "Sid": "AllowS3Publish",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "SNS:Publish",
      "Resource": "*",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "arn:aws:s3:::${S3 BUCKET NAME}"
        }
      }
    }
  ]
}

d. Click Create topic

6. Configure S3 Event Notifications for the SNS Topic

• Go to the Amazon S3 console and select your splunk-cii-set-up bucket.

• Navigate to the Properties tab > Event Notifications

• Click Create event notification

• Configure the event notification:

  • Event name: Provide any name.

  • Prefix (optional): Use if you want to trigger events for specific folders.

  • Event types: Select Put and Post.

  • Destination: Select SNS Topic and choose SplunkSNS.

• Save changes

7. Create Two SQS Queues

• Create Dead-Letter Queue (DLQ):

  • Go to the Amazon SQS console > Create Queue

  • Provide the name: SplunkDLQ

  • Set Visibility Timeout to 300 seconds

  • Click Create Queue

1. Create Main Queue:

• Go to the Amazon SQS console > Create Queue

• Provide the name: SplunkMain

• Set Visibility Timeout to 300 seconds

• Under Dead-letter queue, select Enable and choose SplunkDLQ

• Click Create Queue

Important: Configure the SQS visibility timeout to prevent multiple inputs from receiving and processing messages in a queue more than once. Set your SQS visibility timeout to 5 minutes or longer. If the visibility timeout for a message is reached before the message is fully processed by the SQS-based S3 input, the message reappears in the queue and is retrieved and processed again, resulting in duplicate data)

8. Create an SNS Subscription for SQS

• Go to the Amazon SNS console > Subscriptions > Create Subscription.

• Configure the subscription:

  • Protocol: Select SplunkSNS (created earlier)

  • Protocol: Select Amazon SQS

  • Endpoint: Select SplunkMain

• Click Create Subscription.

9. Generate Access Key and Secret Key

• Go to IAM > Users and select your user.

• Navigate to the Security Credentials tab.

• Under Access Keys, click Create Access Key.

• Select Command Line Interface (CLI) and follow the instructions.

• Download the .csv file containing the Access Key ID and Secret Access Key.

Configure Splunk

• AWS Access Key ID: From the .csv file.

• AWS Secret Access Key: From the .csv file.

• SQS Queue URL: Go to the Amazon SQS console, select SplunkMain, and copy the queue URL.

• External ID: Use the any-autogenerated-id-value (UNIQUE_EXTERNAL_ID) specified earlier.

• S3 Bucket URL:

• S3 Bucket Region: The AWS region where your S3 bucket is hosted (e.g., us-east-1).

Troubleshooting

In Splunk:

1. Navigate to Apps in Splunk and open the Splunk Add-on for AWS.

2. Go to the Inputs tab and verify if a new input named test_splunk_demo has been created.

1. Select the Account tab and find the entry for test_splunk_demo.

   a. Ensure the account details are correct and match the credentials used during the integration setup.

In AWS S3:

1. Check the S3 Bucket:

   a. Log in to your AWS console and go to the S3 service.

   b. Locate the bucket named splunk-cii-demo-set-up (or the bucket name used during the integration setup).

2. Verify Data in the S3 Bucket:

   a. Check if data appears in the bucket after clicking Test Connectivity in Cisco Identity Intelligence.