​

Introduction

Brightbot is a technologically advanced AI system designed to enhance productivity and improve the overall user experience. However, like any other system, its performance can be measured and improved upon. In order to do this, we need to understand the various levers that can be used to quantify and enhance its performance.

​

Measuring Performance

There are four main factors that are used to measure the performance of BrightBot:

​

Usability

This involves measuring the response time and API reliability of the system.

​

Accuracy

The percentage of correct answers against benchmarking tasks and the retrieval of the most relevant information from within the warehouse are key indicators of accuracy.

​

Recall

This refers to the ability of the system to retrieve all relevant tables or documents based on a query.

​

Creativity

This is a subjective measure of the ability of the bots to create novel ideas and insights for users.

​

Improving Performance

Once the performance of Brightbot has been measured, there are a number of ways in which it can be improved:

​

System Engineering

This involves improving API performance and reliability.

​

Prompt Engineering

Utilizing advanced prompting techniques such as the chain of thought and MEDPrompt can greatly enhance the performance of the system.

​

RAG refinement

This involves better chunking, indexing, reranking, and multi-part context enhancement to improve system performance.

​

Function and Skill Augmentation

Adding new function calls and skills to the workflow to offload logic and reasoning from the Language Models (LLMs).

​

Changing LLMs

Different tasks may require different LLMs, and swapping which agents use which LLMs for which tasks can improve performance.

​

Fine-tuning

This involves making minor adjustments to the system to improve its overall performance.