Azure Event Hub Log Streaming for Microsoft Entra ID (Azure AD)

08/2024

Overview

The Cisco identity security platform integrates with Azure AD tenants via the Graph API to collect user account information, device information, and sign-on and application activity.

The Microsoft Graph API is rate limited, meaning that platforms such as CII can only pull a certain amount of data in a given interval before the API stops responding.

In some larger environments, Graph API does not provide enough bandwidth for the initial capture of historical user activity data (prior 30 days) or even the on-going user event data.

The solution to this limitation is to convert the sign-on activity data to an Azure Event Hub streaming model. Then the CII platform can subscribe to that Hub and capture the events in that way. This also provides the benefit of enabling hourly notifications for event or behavioral based identity threats.

Azure Event Hub Overview

Per Microsoft documentation -

“Azure Event Hubs is a Big Data streaming platform and event ingestion service that can receive and process millions of events per second. Event Hubs can process and store events, data, or telemetry produced by distributed software and devices. Data sent to an event hub can be transformed and stored using any real-time analytics provider or batching/storage adapters.”

From the Microsoft Event Hubs overview page, the diagram below depicts the high-level architecture and data flow.

In this diagram, the Event Producer (green) on the left side would be the Azure AD tenant and the Event Receiver on the right side would be your unique CII cloud tenant.

Azure Event Hub - High-level Setup Process

Estimated time to configure in Azure: under 30 min

To configure an Azure Event Hub and integrate it with Azure AD involves the following high-level steps.

Step 1 - Create an Event Hub in Azure

Open and follow these Microsoft instructions - Event Hub setup - to complete initial the initial setup of the Event Hub namespace and event hub object.

High-level steps and notes:

1. Create a Resource Group in Azure. A Resource Group is associated with a subscription in Azure. You may already have a Resource Group associated with Azure AD or identity security that you want to use for this purpose.

2. Create an Event Hub namespace. Within the Event Hub namespace create, you will choose the Resource Group and Subscription. You also need to choose the Pricing Tier and Throughput units. For test environments and smaller production environments, Basic will likely suffice. For larger production environments, Standard should be selected, with the Enable Auto-Inflate selected. See Event Hub Scalability and Automatically Scale-up Event Hub for more information. For Azure Event Hub pricing, please contact your Microsoft representative or see this article.

   
3. Create the Event Hub object. The default number of partitions will suffice. For the Retention period, set the time to a higher value than the default 1 hr, up to 24 hrs for Basic tier event hub namespaces.

   
4. Create a Consumer Group for the event hub. For Basic tier event hubs, the $Default consumer group is created automatically and can be used. For Standard tier, different consumer groups can be created if desired.

   

   

Step 2 - Connect Azure AD logs to the Event Hub

Connect Azure AD logs to the Event Hub using the Microsoft instructions in this link.

High-level steps and important notes:

1. Configure Azure AD event logs to export to your new Event Hub

2. Select the following Event types to stream (screenshot below) and select the Event Hub namespace and Event Hub created in the steps above. Also, use the RootManageSharedAccessKey or another key with write permissions to the event hub namespace

1. Save and confirm that events after visible in the Event Hub after ~15 min

Step 3 - Create a shared access key policy

Note that Shared Access Policies can be created at the Event Hub namespace level or the Event Hub object itself. In this example, we'll create a policy at the namespace level.

1. Within the Event Hub namespace, select Shared Access Policies and then click Add.

   
2. Give the policy a name, select Listen permissions and click Create.

   
3. Click on the new policy and note the Primary key for use in the next steps.

   

Step 4 - CII Tenant Configuration for Azure Event Hub

The next step is to connect the Event Hub to the CII tenant.

Note - this article assumes that the Azure AD data integration is already in place for your tenant. If it is not, complete that configuration first.

1. In the CII console, under the existing Azure AD integration object, select Edit Settings and navigate to the Event Streaming tab.

   
2. Toggle the slider button to Use EventHub for Logs Streaming to ON.

3. Enter the required information -

   1. EventHub Name - NOTE: this field is NOT just a display name. This name needs to match the name of the Event Hub object created under your Event Hub Namespace in Azure. In the screenshot below, you would enter dev-event-hub if that was the specific object you had created and configured.

1. Consumer Group (if not using $Default group)

2. Endpoint FQDN - this is typically of the form <namespace>.servicebus.windows.net

3. Shared Access Policy Name

4. Shared Access Primary Key Value

Click Save to save the configuration.

Troubleshooting

In the event that the following error is received when trying to collect data from the Event Hub:

 [ERROR] CNT: Failed to receive event hub data MessagingError: The supplied sequence number '0' is invalid. The last sequence number in the system is '-1'

The quickest solution to at this time is to simply delete the Event Hub object (not the entire Event Hub namespace) and recreate it. We would suggest altering the Event Hub name slightly and updating the event streaming configuration in the CII Entra ID integration, just for easy identification between the previous and current hubs.