Azure Event Hubs data connection

Azure Event Hubs is a big data streaming platform and event ingestion service. Azure Data Explorer offers continuous ingestion from customer-managed Event Hubs.

The Event Hubs ingestion pipeline transfers events to Azure Data Explorer in several steps. First you create an event hub in the Azure portal. Then create a target table in Azure Data Explorer into which the data in a particular format, is ingested using the provided ingestion properties. The Event Hubs connection needs to be aware of events routing. Data may be embedded with selected properties according to the event system properties. Create a connection to Event Hubs to create an event hub and send events. This process can be managed through the Azure portal, programmatically with C# or Python, or with the Azure Resource Manager template.

For general information about data ingestion in Azure Data Explorer, see Azure Data Explorer data ingestion overview.

Azure Data Explorer data connection authentication options

• Managed Identity based data connection (recommended): Using a managed identity-based data connection is the most secure way to connect to data sources. It provides full control over the ability to fetch data from a data source. Setup of a data connection using managed identity requires the following steps:

  1. Add a managed identity to your cluster.
  2. Grant permissions to the managed identity on the data source. To fetch data from Azure Event Hubs, the managed identity must have Azure Event Hubs Data Receiver permissions.
  3. Set a managed identity policy on the target databases.
  4. Create a data connection using the managed identity authentication to fetch data.

  Caution

  If the managed identity permissions are removed from the data source, the data connection will no longer work and will be unable to fetch data from the data source.

• Key-based data connection: If a managed identity authentication is not specified for the data connection, the connection automatically defaults to key-based authentication. Key-based connections fetch data using a resource connection string, such as the Azure Event Hubs connection string. Azure Data Explorer gets the resource connection string for the specified resource and securely saves it. The connection string is then used to fetch data from the data source.

  Caution

  If the key is rotated, the data connection will no longer work and will be unable to fetch data from the data source. To fix the issue, update or recreate the data connection.

Data format

• Data is read from the event hub in form of EventData objects.
• See supported formats.

Event Hubs properties

Azure Data Explorer supports the following Event Hubs properties:

• A closed set of ingestion properties, which helps to route the event to the relevant table.
• A closed set of event system properties, which can be embedded in the data based on a given mapping.

Ingestion properties

Ingestion properties instruct the ingestion process, where to route the data, and how to process it. You can specify ingestion properties of the events ingestion using the EventData.Properties. You can set the following properties:

Events routing

When you create a data connection to your cluster, you can specify the routing for where to send ingested data. The default routing is to the target table specified in the connection string that is associated with the target database. The default routing for your data is also referred to as static routing. You can specify an alternative routing and processing options for your data by setting one or more of event data properties mentioned in the previous paragraph.

Route event data to an alternate database

Routing data to an alternate database is off by default. To send the data to a different database, you must first set the connection as a multi-database connection. This feature can be enabled in the Azure portal Azure portal, with C# or Python management SDKs, or with an ARM template. The user, group, service principal, or managed identity used to allow database routing must at least have the contributor role and write permissions on the cluster.

To specify an alternate database, set the Database ingestion property.

Route event data to an alternate table

To specify an alternate table for each event, set the Table, Format, Compression, and mapping ingestion properties. The connection dynamically routes the ingested data as specified in the EventData.Properties, overriding the static properties for this event.

The following example shows you how to set the event hub details and send weather metric data to alternate database (MetricsDB) and table (WeatherMetrics). The data is in JSON format and mapping1 is pre-defined on table WeatherMetrics.

// This sample uses Azure.Messaging.EventHubs which is a .Net Framework library.
await using var producerClient = new EventHubProducerClient("<eventHubConnectionString>");
// Create the event and add optional "dynamic routing" properties
var eventData = new EventData(Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(
    new { TimeStamp = DateTime.UtcNow, MetricName = "Temperature", Value = 32 }
)));
eventData.Properties.Add("Database", "MetricsDB");
eventData.Properties.Add("Table", "WeatherMetrics");
eventData.Properties.Add("Format", "json");
eventData.Properties.Add("IngestionMappingReference", "mapping1");
eventData.Properties.Add("Tags", "['myDataTag']");
var events = new[] { eventData };
// Send events
await producerClient.SendAsync(events);

Event Hubs system properties mapping

System properties are fields set by the Event Hubs service, at the time the event is enqueued. Azuer Data Explorer Event Hubs data connection can embed a predefined set of system properties into the data ingested into a table based on a given mapping.

Event Hubs service exposes the following system properties:

When you work with IoT Central event hubs, you can also embed IoT Hub system properties in the payload. For the complete list, see IoT Hub system properties.

If you selected Event system properties in the Data Source section of the table, you must include the properties in the table schema and mapping.

Schema mapping examples

Table schema mapping example

If your data includes three columns (TimeStamp, MetricName, and Value) and the properties you include are x-opt-enqueued-time and x-opt-offset, create or alter the table schema by using this command:

    .create-merge table TestTable (TimeStamp: datetime, MetricName: string, Value: int, EventHubEnqueuedTime:datetime, EventHubOffset:string)

CSV mapping example

Run the following commands to add data to the beginning of the record. Note ordinal values.

    .create table TestTable ingestion csv mapping "CsvMapping1"
    '['
    '   { "column" : "TimeStamp", "Properties":{"Ordinal":"2"}},'
    '   { "column" : "MetricName", "Properties":{"Ordinal":"3"}},'
    '   { "column" : "Value", "Properties":{"Ordinal":"4"}},'
    '   { "column" : "EventHubEnqueuedTime", "Properties":{"Ordinal":"0"}},'
    '   { "column" : "EventHubOffset", "Properties":{"Ordinal":"1"}}'
    ']'

JSON mapping example

Data is added by using the system properties mapping. Run these commands:

    .create table TestTable ingestion json mapping "JsonMapping1"
    '['
    '    { "column" : "TimeStamp", "Properties":{"Path":"$.TimeStamp"}},'
    '    { "column" : "MetricName", "Properties":{"Path":"$.MetricName"}},'
    '    { "column" : "Value", "Properties":{"Path":"$.Value"}},'
    '    { "column" : "EventHubEnqueuedTime", "Properties":{"Path":"$.x-opt-enqueued-time"}},'
    '    { "column" : "EventHubOffset", "Properties":{"Path":"$.x-opt-offset"}}'
    ']'

Schema mapping for Event Hubs Capture Avro files

One way to consume Event Hubs data is to capture events through Azure Event Hubs in Azure Blob Storage or Azure Data Lake Storage. You can then ingest the capture files as they are written using an Event Grid Data Connection in Azure Data Explorer.

The schema of the capture files is different from the schema of the original event sent to Event Hubs. You should design the destination table schema with this difference in mind. Specifically, the event payload is represented in the capture file as a byte array, and this array isn't automatically decoded by the Event Grid Azure Data Explorer data connection. For more information on the file schema for Event Hubs Avro capture data, see Exploring captured Avro files in Azure Event Hubs.

To correctly decode the event payload:

1. Map the Body field of the captured event to a column of type dynamic in the destination table.
2. Apply an update policy that converts the byte array into a readable string using the unicode_codepoints_to_string() function.

Ingest custom properties

When ingesting events from Event Hubs, data is taken from the body section of the event data object. However, Event Hubs custom properties are defined in the properties section of the object and are not ingested. To ingest customer properties, you must embed them into the data in body section of the object.

The following example compares the events data object containing custom property customProperty as defined by Event Hubs (left) with the embedded property required for ingestion (right).

{
"body":{
"value": 42
},
"properties":{
"customProperty": "123456789"
}
}

{
"body":{
"value": 42,
"customProperty": "123456789"
}
}

You can use one of the following methods to embed custom properties into the data in body section of the event data object:

• In Event Hubs, when creating the event data object, embed the custom properties as part of the data in the body section of the object.
• Use Azure Stream Analytics to process events from the event hub and embed the custom properties in the event data. From Azure Stream Analytics you can ingest the data natively using the Azure Data Explorer output connector, or route the data into another event hub and from there into your cluster.
• Use Azure Functions to add the custom properties and then ingest the data.

Cross-region Event Hubs data connection

For best performance, create all the following resources in the same region as the cluster. If there is no other alternative, consider using Premium or Dedicated Event Hubs tiers. Event Hubs tiers comparison can be found here.

Create an event hub

If you don't already have one, Create an event hub. Connecting to event hub can be managed through the Azure portal, programmatically with C# or Python, or with the Azure Resource Manager template.

Send events

See the sample app that generates data and sends it to an event hub.

Set up Geo-disaster recovery solution

Event hub offers a Geo-disaster recovery solution. Azure Data Explorer doesn't support Alias event hub namespaces. To implement the Geo-disaster recovery in your solution, create two event hub data connections: one for the primary namespace and one for the secondary namespace. Azure Data Explorer listens to both event hub connections.

Related content

• Ingest data from event hub into Azure Data Explorer
• Create an event hub data connection for Azure Data Explorer using C#
• Create an event hub data connection for Azure Data Explorer using Python
• Create an event hub data connection for Azure Data Explorer using Azure Resource Manager template
• Ingest and query Azure Monitor logs with Azure Data Explorer