What is a Suite?

A Suite is used to group related Tests, like a folder in a filesystem. Full Suites can be run with a single click in both the Chrome Extension and the main application.

If you had a customer service chatbot, you might create a Suite for general product-related questions, and a separate Suite for order inquiries.

What is an Environment?

An Environment is a way to organize testing of a Bot across multiple development environments (such as Development, Staging, Production, etc).

Environments are defined by the URL of the chatbot website, and it is a requirement that the HTML layout between environments is the same.

What is a Test?

A Test is a single specific scenario you want your chatbot to be able to handle correctly. A Test can be thought of as the semantic meaning behind a request or series of prompts. A Test is more than a sequence of specific prompts, since there are practically unlimited ways to say the same thing!

For example, a single Test might be asking your chatbot how to initiate a refund, and then having follow-up questions regarding the specifics of an order. There are many syntactically different ways to ask each of those prompts with the same semantic meaning, but they would all fit under a single Test.

What is a Variant?

A Variant is a specific formulation of a Test. Because there are many slightly different ways that a user can ask the same question, Variants allow you to ensure that your Bot correctly answers when it is presented with different phrasing.

For example, if your Test contained only one question, you might have the following Variants:
User: How do I get a refund for my order?
User: Can I get a refund on my order?
User: What are the steps to get a refund?

You can automatically generate new Variants based on the original Variant recorded for the Test.

What is a Baseline?

A Baseline is a valid/optimal set of responses given by the chatbot in a conversation. During the Test evaluation process, each response from the chatbot will be compared to the set of responses stored as the Baseline to determine whether the Test passed or failed.

You can setup more than one Baselines for a single Test if there are multiple distinct but correct ways for a chatbot to respond to the Test inquiry.

For example, if you had a support chatbot to help with customer issues, responses to a question about how to fix a connection issue might either be to 1. Restart the device, or 2. Check the internet connection. Since followup questions such as asking for the relevant documentation will bring the user down distinct conversation paths, you might decide to setup two Baselines for the Test.

What is an Evaluation?

An Evaluation is a single determination by the bottest.ai Test Evaluator whether the responses from the Bot matches at least one Baseline defined for that Test.

The number of Evaluations that will run for a specific Test depends on two factors:

• The number of Variants defined for the Test
• The number of iterations defined in the Test (the number of times each Variant will be ran). Multiple iterations are performed of each Variant to measure the consistency in responses provided by the Bot.