Regression Testing

Thoroughly evaluate every aspect of your Bot's performance after every change.

Regression testing performed by our LLM-based Test Evaluator gives you the peace of mind that changes in your Bot or its underlying language model have not adversary impacted Bot performance.

Customizable Test Evaluation Criteria

Using our LLM-based Test Evaluator, each response from your chatbot is carefully considered to determine whether the Test should be failed.

Modify the LLM-based evaluation criteria for an individual Test or an entire Suite using custom instructions.

This ensures that evaluations are performed with the necessary nuance to properly measure response accuracy and confidently determine whether a Test passed or failed.

Check Response Consistency

Chatbots will often provide different responses when prompted with the same question. bottest.ai ensures you ship a product that delivers consistently accurate answers to your users.

Our Test Evaluator will run multiple iterations of each Test to check the consistency of responses. Customize the number of iterations on a Suite level, or configure them on each Test individually, to ensure quality for all of your users.

Allow Multiple Correct Answers

Some user inquiries with your chatbot may only have one answer that is deemed valid or correct, such as when a user is attempting to jailbreak a system (see our protection against this with Adversarial Testing).

However, some inquiries may have multiple correct or valid answers that should be considered, for example responding with a summary of a resource or a direct link to it.

Our Test Evaluator gives you the flexibility to define multiple Baselines for each Test to ensure that all evaluations are accurate and appropriate. During the evaluation process, we check that the chatbot’s response passes to at least one Baseline.

Identify Root Cause of Failed Tests

Understanding the reason for a Test failure is just as important as knowing that a Test failed.

bottest.ai provides a detailed description of all Test failures. After each run, a comprehensive report summarizing overall failure statistics and providing failure analysis on a granular level is sent to you for review.

Evaluation with Multiple Correct Answers

Some inquiries may have multiple correct or valid answers that should be considered, for example responding with a summary of a resource or a direct link to it. Our Test Evaluator gives you the flexibility to define multiple Baselines for each Test to ensure that all evaluations are accurate and appropriate. During the evaluation process, we check that the chatbot’s response passes to at least one Baseline.

Consistency Across Multiple Responses

Chatbots will often provide different responses when prompted with the same question. It’s important that the product you ship is consistent in quality for each of your users. Our Test Evaluator will run multiple iterations of each Test to check the consistency of responses. Customize the number of iterations on a Suite level, or configure them on each Test individually, to ensure quality for all of your users.

Linguistic Analysis with Customized Criteria

Our Test Evaluator allows the user to customize evaluation instructions and criteria to follow when determining a result, both on a Suite and an individual Test level. This ensures that even nuanced and complex Test results are evaluated with utmost accuracy and context on the specific requirements of your application.

Root Cause Identification of Failed Tests

Understanding why a Test is failing is just as important as knowing that a Test fails in the first place. When an evaluation fails, a detailed description of exactly why will be provided to the user. Upon running full Suite Runs, a comprehensive report is generated to both summarize failures from a glance as well as give insight on a granular level.