Add natural language understanding to your bot

After each processing of user input, DialogBot saves the current state of both UserState and ConversationState. Once all the required information has been gathered, the coding sample creates a demo flight booking reservation. In this article, we'll be covering the LUIS aspects of this sample. However, the general flow of the sample is:

• OnMembersAddedAsync is called when a new user is connected and displays a welcome card.
• OnMessageActivityAsync is called for each user input received.

The OnMessageActivityAsync module runs the appropriate dialog through the Run dialog extension method. Then the main dialog calls the LUIS helper to find the top scoring user intent. If the top intent for the user input returns "BookFlight", the helper fills out information from the user that LUIS returned. After that, the main dialog starts the BookingDialog, which acquires additional information as needed from the user such as:

• Origin the originating city
• TravelDate the date to book the flight
• Destination the destination city

After each processing of user input, dialogBot saves the current state of both userState and conversationState. Once all the required information has been gathered, the coding sample creates a demo flight booking reservation. In this article, we'll be covering the LUIS aspects of this sample. However, the general flow of the sample is:

• onMembersAdded is called when a new user is connected and displays a welcome card.
• OnMessage is called for each user input received.

The onMessage module runs the mainDialog, which gathers user input. Then the main dialog calls the LUIS helper FlightBookingRecognizer to find the top scoring user intent. If the top intent for the user input returns "BookFlight", the helper fills out information from the user that LUIS returned. Upon the response back, mainDialog preserves information for the user returned by LUIS and starts bookingDialog. bookingDialog acquires additional information as needed from the user such as

• destination the destination city.
• origin the originating city.
• travelDate the date to book the flight.

1. After each processing of user input, DialogBot saves the current state of both UserState and ConversationState.
2. Once all the required information has been gathered, the coding sample creates a demo flight booking reservation.
3. In this article, we'll be covering the LUIS aspects of this sample. However, the general flow of the sample is:

• onMembersAdded is called when a new user is connected and displays a welcome card.
• onMessageActivity is called for each user input received.

The onMessageActivity module runs the appropriate dialog through the run dialog extension method. Then the main dialog calls the LUIS helper to find the top scoring user intent. If the top intent for the user input returns "BookFlight", the helper fills out information from the user that LUIS returned. After that, the main dialog starts the BookingDialog, which acquires additional information as needed from the user such as:

• Origin the originating city
• TravelDate the date to book the flight
• Destination the destination city

After each processing of user input, DialogBot saves the current state of both user_state and conversation_state. Once all the required information has been gathered, the coding sample creates a demo flight booking reservation. In this article, we'll be covering the LUIS aspects of this sample. However, the general flow of the sample is:

• on_members_added_activity is called when a new user is connected and displays a welcome card.
• on_message_activity is called for each user input received.

The on_message_activity module runs the appropriate dialog through the run_dialog dialog extension method. Then the main dialog calls LuisHelper to find the top scoring user intent. If the top intent for the user input returns "BookFlight", the helper function fills out information from the user that LUIS returned. After that, the main dialog starts the BookingDialog, which acquires additional information as needed from the user such as:

• destination the destination city.
• origin the originating city.
• travel_date the date to book the flight.

Add the information required to access your LUIS app including application ID, authoring key, and region into the appsettings.json file. In the previous step, you retrieved these values from your published LUIS app. The API host name should be in the format <your region>.api.cognitive.azure.cn.

appsetting.json

{
  "MicrosoftAppType": "",
  "MicrosoftAppId": "",
  "MicrosoftAppPassword": "",
  "MicrosoftAppTenantId": "",
  "LuisAppId": "",
  "LuisAPIKey": "",
  "LuisAPIHostName": ""
}

Add the information required to access your LUIS app including application ID, authoring key, and region into the .env file. In the previous step, you retrieved these values from your published LUIS app. The API host name should be in the format <your region>.api.cognitive.azure.cn.

.env

MicrosoftAppType=
MicrosoftAppId=
MicrosoftAppPassword=
MicrosoftAppTenantId=
LuisAppId=
LuisAPIKey=
LuisAPIHostName=

Add the information required to access your LUIS app including application ID, authoring key, and region into the application.properties file. In the previous step, you retrieved these values from your published LUIS app. The API host name should be in the format <your region>.api.cognitive.azure.cn.

application.properties

MicrosoftAppId=
MicrosoftAppPassword=
LuisAppId=
LuisAPIKey=
LuisAPIHostName=
server.port=3978

Add the information required to access your LUIS app including application ID, authoring key, and region into the config.py file. In the previous step, you retrieved these values from your published LUIS app. The API host name should be in the format <your region>.api.cognitive.azure.cn.

config.py

APP_ID = os.environ.get("MicrosoftAppId", "")
APP_PASSWORD = os.environ.get("MicrosoftAppPassword", "")
APP_TYPE = os.environ.get("MicrosoftAppType", "MultiTenant")
APP_TENANTID = os.environ.get("MicrosoftAppTenantId", "")
LUIS_APP_ID = os.environ.get("LuisAppId", "")
LUIS_API_KEY = os.environ.get("LuisAPIKey", "")

Be sure that the Microsoft.Bot.Builder.AI.Luis NuGet package is installed for your project.

To connect to the LUIS service, the bot pulls the information you added to the appsetting.json file. The FlightBookingRecognizer class contains code with your settings from the appsetting.json file and queries the LUIS service by calling RecognizeAsync method.

FlightBookingRecognizer.cs

public class FlightBookingRecognizer : IRecognizer
{
    private readonly LuisRecognizer _recognizer;

    public FlightBookingRecognizer(IConfiguration configuration)
    {
        var luisIsConfigured = !string.IsNullOrEmpty(configuration["LuisAppId"]) && !string.IsNullOrEmpty(configuration["LuisAPIKey"]) && !string.IsNullOrEmpty(configuration["LuisAPIHostName"]);
        if (luisIsConfigured)
        {
            var luisApplication = new LuisApplication(
                configuration["LuisAppId"],
                configuration["LuisAPIKey"],
                "https://" + configuration["LuisAPIHostName"]);
            // Set the recognizer options depending on which endpoint version you want to use.
            // More details can be found in https://docs.microsoft.com/en-gb/azure/cognitive-services/luis/luis-migration-api-v3
            var recognizerOptions = new LuisRecognizerOptionsV3(luisApplication)
            {
                PredictionOptions = new Bot.Builder.AI.LuisV3.LuisPredictionOptions
                {
                    IncludeInstanceData = true,
                }
            };

            _recognizer = new LuisRecognizer(recognizerOptions);
        }
    }

    // Returns true if luis is configured in the appsettings.json and initialized.
    public virtual bool IsConfigured => _recognizer != null;

    public virtual async Task<RecognizerResult> RecognizeAsync(ITurnContext turnContext, CancellationToken cancellationToken)
        => await _recognizer.RecognizeAsync(turnContext, cancellationToken);

    public virtual async Task<T> RecognizeAsync<T>(ITurnContext turnContext, CancellationToken cancellationToken)
        where T : IRecognizerConvert, new()
        => await _recognizer.RecognizeAsync<T>(turnContext, cancellationToken);
}

The FlightBookingEx.cs contains the logic to extract From, To and TravelDate; it extends the partial class FlightBooking.cs used to store LUIS results when calling FlightBookingRecognizer.RecognizeAsync<FlightBooking> from the MainDialog.cs.

CognitiveModels\FlightBookingEx.cs

// Extends the partial FlightBooking class with methods and properties that simplify accessing entities in the luis results
public partial class FlightBooking
{
    public (string From, string Airport) FromEntities
    {
        get
        {
            var fromValue = Entities?._instance?.From?.FirstOrDefault()?.Text;
            var fromAirportValue = Entities?.From?.FirstOrDefault()?.Airport?.FirstOrDefault()?.FirstOrDefault();
            return (fromValue, fromAirportValue);
        }
    }

    public (string To, string Airport) ToEntities
    {
        get
        {
            var toValue = Entities?._instance?.To?.FirstOrDefault()?.Text;
            var toAirportValue = Entities?.To?.FirstOrDefault()?.Airport?.FirstOrDefault()?.FirstOrDefault();
            return (toValue, toAirportValue);
        }
    }

    // This value will be a TIMEX. And we are only interested in a Date so grab the first result and drop the Time part.
    // TIMEX is a format that represents DateTime expressions that include some ambiguity. e.g. missing a Year.
    public string TravelDate
        => Entities.datetime?.FirstOrDefault()?.Expressions.FirstOrDefault()?.Split('T')[0];
}

To use LUIS, your project needs to install the botbuilder-ai npm package.

To connect to the LUIS service, the bot uses the information you added to the .env file. The flightBookingRecognizer.js class contains the code that imports your settings from the .env file and queries the LUIS service by calling recognize() method.

dialogs/flightBookingRecognizer.js

class FlightBookingRecognizer {
    constructor(config) {
        const luisIsConfigured = config && config.applicationId && config.endpointKey && config.endpoint;
        if (luisIsConfigured) {
            // Set the recognizer options depending on which endpoint version you want to use e.g v2 or v3.
            // More details can be found in https://docs.microsoft.com/en-gb/azure/cognitive-services/luis/luis-migration-api-v3
            const recognizerOptions = {
                apiVersion: 'v3'
            };

            this.recognizer = new LuisRecognizer(config, recognizerOptions);
        }
    }

    get isConfigured() {
        return (this.recognizer !== undefined);
    }

    /**
     * Returns an object with preformatted LUIS results for the bot's dialogs to consume.
     * @param {TurnContext} context
     */
    async executeLuisQuery(context) {
        return await this.recognizer.recognize(context);
    }

    getFromEntities(result) {
        let fromValue, fromAirportValue;
        if (result.entities.$instance.From) {
            fromValue = result.entities.$instance.From[0].text;
        }
        if (fromValue && result.entities.From[0].Airport) {
            fromAirportValue = result.entities.From[0].Airport[0][0];
        }

        return { from: fromValue, airport: fromAirportValue };
    }

    getToEntities(result) {
        let toValue, toAirportValue;
        if (result.entities.$instance.To) {
            toValue = result.entities.$instance.To[0].text;
        }
        if (toValue && result.entities.To[0].Airport) {
            toAirportValue = result.entities.To[0].Airport[0][0];
        }

        return { to: toValue, airport: toAirportValue };
    }

    /**
     * This value will be a TIMEX. And we are only interested in a Date so grab the first result and drop the Time part.
     * TIMEX is a format that represents DateTime expressions that include some ambiguity. e.g. missing a Year.
     */
    getTravelDate(result) {
        const datetimeEntity = result.entities.datetime;
        if (!datetimeEntity || !datetimeEntity[0]) return undefined;

        const timex = datetimeEntity[0].timex;
        if (!timex || !timex[0]) return undefined;

        const datetime = timex[0].split('T')[0];
        return datetime;
    }
}

The logic to extract From, To and TravelDate is implemented as helper methods inside flightBookingRecognizer.js. These methods are used after calling flightBookingRecognizer.executeLuisQuery() from mainDialog.js

Be sure that the com.microsoft.bot.bot-ai-luis-v3 package is added to your pom.xml file.

<dependency>
  <groupId>com.microsoft.bot</groupId>
  <artifactId>bot-ai-luis-v3</artifactId>
  <version>4.14.1</version>
</dependency>

To connect to the LUIS service, the bot pulls the information you added to the application.properties file. The FlightBookingRecognizer class contains code with your settings from the application.properties file and queries the LUIS service by calling recognize method.

FlightBookingRecognizer.java

/**
 * The constructor of the FlightBookingRecognizer class.
 *
 * @param configuration The Configuration object to use.
 */
public FlightBookingRecognizer(Configuration configuration) {
    Boolean luisIsConfigured = StringUtils.isNotBlank(configuration.getProperty("LuisAppId"))
        && StringUtils.isNotBlank(configuration.getProperty("LuisAPIKey"))
        && StringUtils.isNotBlank(configuration.getProperty("LuisAPIHostName"));
    if (luisIsConfigured) {
        LuisApplication luisApplication = new LuisApplication(
            configuration.getProperty("LuisAppId"),
            configuration.getProperty("LuisAPIKey"),
            String.format("https://%s", configuration.getProperty("LuisAPIHostName"))
        );
        // Set the recognizer options depending on which endpoint version you want to use.
        // More details can be found in
        // https://docs.microsoft.com/en-gb/azure/cognitive-services/luis/luis-migration-api-v3
        LuisRecognizerOptionsV3 recognizerOptions = new LuisRecognizerOptionsV3(luisApplication);
        recognizerOptions.setIncludeInstanceData(true);

        this.recognizer = new LuisRecognizer(recognizerOptions);
    }
}

/**
 * Runs an utterance through a recognizer and returns a generic recognizer result.
 *
 * @param turnContext Turn context.
 * @return Analysis of utterance.
 */
@Override
public CompletableFuture<RecognizerResult> recognize(TurnContext turnContext) {
    return this.recognizer.recognize(turnContext);
}

The FlightBookingRecognizer.cs contains the logic to extract From, To and TravelDate; and is called from the MainDialog.java to decode the results of the Luis query result.

FlightBookingRecognizer.java

/**
 * Gets the From data from the entities which is part of the result.
 *
 * @param result The recognizer result.
 * @return The object node representing the From data.
 */
public ObjectNode getFromEntities(RecognizerResult result) {
    String fromValue = "", fromAirportValue = "";
    if (result.getEntities().get("$instance").get("From") != null) {
        fromValue = result.getEntities().get("$instance").get("From").get(0).get("text")
            .asText();
    }
    if (!fromValue.isEmpty()
        && result.getEntities().get("From").get(0).get("Airport") != null) {
        fromAirportValue = result.getEntities().get("From").get(0).get("Airport").get(0).get(0)
            .asText();
    }

    ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();
    ObjectNode entitiesNode = mapper.createObjectNode();
    entitiesNode.put("from", fromValue);
    entitiesNode.put("airport", fromAirportValue);
    return entitiesNode;
}

/**
 * Gets the To data from the entities which is part of the result.
 *
 * @param result The recognizer result.
 * @return The object node representing the To data.
 */
public ObjectNode getToEntities(RecognizerResult result) {
    String toValue = "", toAirportValue = "";
    if (result.getEntities().get("$instance").get("To") != null) {
        toValue = result.getEntities().get("$instance").get("To").get(0).get("text").asText();
    }
    if (!toValue.isEmpty() && result.getEntities().get("To").get(0).get("Airport") != null) {
        toAirportValue = result.getEntities().get("To").get(0).get("Airport").get(0).get(0)
            .asText();
    }

    ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();
    ObjectNode entitiesNode = mapper.createObjectNode();
    entitiesNode.put("to", toValue);
    entitiesNode.put("airport", toAirportValue);
    return entitiesNode;
}

/**
 * This value will be a TIMEX. And we are only interested in a Date so grab the first result and
 * drop the Time part. TIMEX is a format that represents DateTime expressions that include some
 * ambiguity. e.g. missing a Year.
 *
 * @param result A {link RecognizerResult}
 * @return The Timex value without the Time model
 */
public String getTravelDate(RecognizerResult result) {
    JsonNode datetimeEntity = result.getEntities().get("datetime");
    if (datetimeEntity == null || datetimeEntity.get(0) == null) {
        return null;
    }

    JsonNode timex = datetimeEntity.get(0).get("timex");
    if (timex == null || timex.get(0) == null) {
        return null;
    }

    String datetime = timex.get(0).asText().split("T")[0];
    return datetime;
}

Be sure that the botbuilder-ai PyPI package is installed for your project.

To connect to the LUIS service, the bot uses the information you added to the config.py file. The FlightBookingRecognizer class contains the code that imports your settings from the config.py file and queries the LUIS service by calling recognize() method.

flight_booking_recognizer.py

class FlightBookingRecognizer(Recognizer):
    def __init__(self, configuration: DefaultConfig):
        self._recognizer = None

        luis_is_configured = (
            configuration.LUIS_APP_ID
            and configuration.LUIS_API_KEY
            and configuration.LUIS_API_HOST_NAME
        )
        if luis_is_configured:
            # Set the recognizer options depending on which endpoint version you want to use e.g v2 or v3.
            # More details can be found in https://docs.microsoft.com/azure/cognitive-services/luis/luis-migration-api-v3
            luis_application = LuisApplication(
                configuration.LUIS_APP_ID,
                configuration.LUIS_API_KEY,
                "https://" + configuration.LUIS_API_HOST_NAME,
            )

            self._recognizer = LuisRecognizer(luis_application)

    @property
    def is_configured(self) -> bool:
        # Returns true if luis is configured in the config.py and initialized.
        return self._recognizer is not None

    async def recognize(self, turn_context: TurnContext) -> RecognizerResult:
        return await self._recognizer.recognize(turn_context)

The logic to extract From, To and travel_date is implemented as helper methods from the LuisHelper class inside luis_helper.py. These methods are used after calling LuisHelper.execute_luis_query() from main_dialog.py

helpers/luis_helper.py

class LuisHelper:
    @staticmethod
    async def execute_luis_query(
        luis_recognizer: LuisRecognizer, turn_context: TurnContext
    ) -> (Intent, object):
        """
        Returns an object with preformatted LUIS results for the bot's dialogs to consume.
        """
        result = None
        intent = None

        try:
            recognizer_result = await luis_recognizer.recognize(turn_context)

            intent = (
                sorted(
                    recognizer_result.intents,
                    key=recognizer_result.intents.get,
                    reverse=True,
                )[:1][0]
                if recognizer_result.intents
                else None
            )

            if intent == Intent.BOOK_FLIGHT.value:
                result = BookingDetails()

                # We need to get the result from the LUIS JSON which at every level returns an array.
                to_entities = recognizer_result.entities.get("$instance", {}).get(
                    "To", []
                )
                if len(to_entities) > 0:
                    if recognizer_result.entities.get("To", [{"$instance": {}}])[0][
                        "$instance"
                    ]:
                        result.destination = to_entities[0]["text"].capitalize()
                    else:
                        result.unsupported_airports.append(
                            to_entities[0]["text"].capitalize()
                        )

                from_entities = recognizer_result.entities.get("$instance", {}).get(
                    "From", []
                )
                if len(from_entities) > 0:
                    if recognizer_result.entities.get("From", [{"$instance": {}}])[0][
                        "$instance"
                    ]:
                        result.origin = from_entities[0]["text"].capitalize()
                    else:
                        result.unsupported_airports.append(
                            from_entities[0]["text"].capitalize()
                        )

                # This value will be a TIMEX. And we are only interested in a Date so grab the first result and drop
                # the Time part. TIMEX is a format that represents DateTime expressions that include some ambiguity.
                # e.g. missing a Year.
                date_entities = recognizer_result.entities.get("datetime", [])
                if date_entities:
                    timex = date_entities[0]["timex"]

                    if timex:
                        datetime = timex[0].split("T")[0]

                        result.travel_date = datetime

                else:
                    result.travel_date = None

        except Exception as exception:
            print(exception)

        return intent, result