Azure Event Hub Log Streaming for Microsoft Entra ID

Overview

The Cisco identity security platform integrates with Azure AD tenants via the Graph API to collect user account information, device information, and sign-on and application activity.

The Microsoft Graph API is rate limited, meaning that platforms such as CII can only pull a certain amount of data in a given interval before the API stops responding.

In some larger environments, Graph API does not provide enough bandwidth for the initial capture of historical user activity data (prior 30 days) or even the on-going user event data.

The solution to this limitation is to convert the sign-on activity data to an Azure Event Hub streaming model. Then the CII platform can subscribe to that Hub and capture the events in that way. This also provides the benefit of enabling hourly notifications for event or behavioral based identity threats.

What is an Azure Event Hub?

Per Microsoft documentation -

“Azure Event Hubs is a Big Data streaming platform and event ingestion service that can receive and process millions of events per second. Event Hubs can process and store events, data, or telemetry produced by distributed software and devices. Data sent to an event hub can be transformed and stored using any real-time analytics provider or batching/storage adapters.”

From the Microsoft Event Hubs overview page, the diagram below depicts the high-level architecture and data flow.

In this diagram, the Event Producer (green) on the left side would be the Azure AD tenant and the Event Receiver on the right side would be your unique CII cloud tenant.

Deploy Event Hub Streaming from Azure Marketplace (Recommended)

Prerequisites

• A completed and functioning Microsoft Entra ID (Azure AD) Data Integration in your CII tenant

• Azure admin role / permissions sufficient to create the objects below.

• An existing EMPTY resource group to deploy this Azure Marketplace application to (in case of any resource group-level policies that could cause issues or conflicts).

Event Hub Tier and Scaling

The Marketplace app will create the Event Hub object with the Standard Tier. Standard tier provides a maximum of 40 throughput units. For very, very large enterprises, if you feel Standard tier is not sufficient and a higher tier like the Premium tier is required, follow the Azure Event Hub Log Streaming for Microsoft Entra IDbelow to manually create the Event Hub Namespace and Event Hub object with that their.

For Azure Event Hub pricing, please contact your Microsoft representative or see this article.

Install Azure Marketplace Application

When pre-requisites steps are in place, proceed to install the Entra ID integration package to Azure.

1. In Azure, click on Create a Resource.

2. Search for Cisco Identity Intelligence and select Log Streaming

3. Select the Free Plan option and Create it

4. Enter all the details into the input boxes as per the table and example screenshot below. NOTE: As mentioned above, the Azure Resource group specified here to deploy Marketplace offer MUST be empty. (It cannot have other existing resources already contained within it.)

1. Click Create

2. The following objects will have been created in the Azure environment:

   1. Event Hub Namespace

   2. Event Hub

   3. Consumer Group

Configure Event Hub Settings

The Marketplace app will create the Event Hub object with the Standard Tier with 1 Throughput Unit.

Auto-infate is not enabled by default, but it should be, as one TU will not be sufficient for most mid to large organizations. See Event Hub Scalability and Automatically Scale-up Event Hub for more information.

To do so:

1. Navigate to the Event Hub Namespace, expand the Settings menu, and select the Scale pane from the left hand menu.

2. Under Auto-Inflate, check the Enable box and set the max throughput units.

3. Click Save.

4. Optional: When you apply the auto inflate configuration to increase throughput units, the Event Hubs service emits diagnostic logs that give you information about why and when the throughput increased. To enable diagnostic logging for an event hub, select Diagnostic settings on the left menu on the Event Hub page in the Azure portal. For more information, see Set up diagnostic logs for an Azure event hub.

Connect Entra ID logs to the Event Hub

Connect Entra ID logs to the Event Hub using the Microsoft instructions in this link.

High-level steps and important notes:

1. Select the following Event types to stream (screenshot below) and select the Event Hub namespace and Event Hub created in the steps above.

2. Also, use the RootManageSharedAccessKey or another key with write permissions to the event hub namespace

1. Save and confirm that events after visible in the Event Hub after ~15 min

Proceed to the Create a shared access key policy section to finish the setup Event Hub setup.

Manual Azure Event Hub Setup Process

To configure an Azure Event Hub and integrate it with Azure AD involves the following high-level steps.

Step 1 - Create an Event Hub in Azure

Open and follow these Microsoft instructions - Event Hub setup - to complete initial the initial setup of the Event Hub namespace and event hub object.

High-level steps and notes:

1. Create a Resource Group in Azure. A Resource Group is associated with a subscription in Azure. You may already have a Resource Group associated with Azure AD or identity security that you want to use for this purpose.

2. Create an Event Hub namespace. Within the Event Hub namespace create, you will choose the Resource Group and Subscription. You also need to choose the Pricing Tier and Throughput units. For test environments and smaller production environments, Basic will likely suffice. For larger production environments, Standard should be selected, with the Enable Auto-Inflate selected. See Event Hub Scalability and Automatically Scale-up Event Hub for more information. For Azure Event Hub pricing, please contact your Microsoft representative or see this article.

   
3. Create the Event Hub object. The default number of partitions will suffice. For the Retention period, set the time to a higher value than the default 1 hr, up to 24 hrs for Basic tier event hub namespaces.

   
4. Create a Consumer Group for the event hub. For Basic tier event hubs, the $Default consumer group is created automatically and can be used. For Standard tier, different consumer groups can be created if desired.

   

   

Step 2 - Connect Azure AD logs to the Event Hub

Connect Azure AD logs to the Event Hub using the Microsoft instructions in this link.

High-level steps and important notes:

1. Configure Azure AD event logs to export to your new Event Hub

2. Select the following Event types to stream (screenshot below) and select the Event Hub namespace and Event Hub created in the steps above. Also, use the RootManageSharedAccessKey or another key with write permissions to the event hub namespace

1. Save and confirm that events after visible in the Event Hub after ~15 min

Create a shared access key policy

Note that Shared Access Policies can be created at the Event Hub namespace level or the Event Hub object itself. In this example, we'll create a policy at the namespace level.

1. Within the Event Hub namespace, select Shared Access Policies and then click Add.

   
2. Give the policy a name, select Listen permissions and click Create.

   
3. Click on the new policy and note the Primary key for use in the next steps.

   

CII Tenant Configuration for Azure Event Hub

The next step is to connect the Event Hub to the CII tenant.

1. In the CII console, under the existing Entra ID integration, select Edit Settings and navigate to the Event Streaming tab.

   
2. Toggle the slider button to Use EventHub for Logs Streaming to ON.

3. Enter the required information -

   1. EventHub Name - NOTE: this field is NOT just a display name. This name needs to match the name of the Event Hub object created under your Event Hub Namespace in Azure. In the screenshot below, you would enter dev-event-hub if that was the specific object you had created and configured.

1. Consumer Group (if not using $Default group)

2. Endpoint FQDN - this is typically of the form <namespace>.servicebus.windows.net

3. Shared Access Policy Name

4. Shared Access Primary Key Value

Click Save to save the configuration.

Troubleshooting

In the event that the following error is received when trying to collect data from the Event Hub:

 [ERROR] CNT: Failed to receive event hub data MessagingError: The supplied sequence number '0' is invalid. The last sequence number in the system is '-1'

The quickest solution to at this time is to simply delete the Event Hub object (not the entire Event Hub namespace) and recreate it. We would suggest altering the Event Hub name slightly and updating the event streaming configuration in the CII Entra ID integration, just for easy identification between the previous and current hubs.