Problem Statement

The client was designing a new chatbot feature on their teaching platform to help users (primarily teachers) find information they were looking for more efficiently.

Key Research Questions:

Did users notice the new chatbot?

Given that the users know about the chatbot, would the users choose to use it?

When using the chatbot, were the users able to find the relevant information or be guided toward the correct part of the website?

What were users' opinions of the chatbot?

Recruitment

The platform was primarily available to teachers and department heads. Due to this, the client already had preselected a series of potential candidates, half of whom had experience using the platform (without the new functionality), and the other half had no experience of the platform prior to evaluation.

Moderated Remote Usability Study

Due to the COVID-19 pandemic, the evaluation needed to be performed remotely, being recorded over Zoom. In all parts of the usability study users were provided tasks to find information while following a think out loud protocol. They were prompted for their thoughts and attitude several times throughout and after the tasks.
Firstly, users were given a scenario where they had to imagine to be coming up with a research question and answers from the website, which they would then have to present to their department. The platform contained a large amount of data that was open to interpretation.
For the second part of the study, users were first informed about the existence of the chatbot. Users asked to find additional information related to the research question they came up with in the first part, and were again reminded about the chatbot prior to starting.
In the final part of the study, users were instructed to specifically use the chatbot to find the some information.

Survey

After the usability study, participants were emailed surveys to gather demographic data, including prior website usage. The survey was also used to identify discrepancies from the study, clarify trends, and gain any additional thoughts they may have only had after finishing the tasks.

Data Analysis

Interviews were transcribed, and then analyzed to determine several dimensions along with the survey responses.
For each dimension, data was collated for metrics, such as shown in the below table.

Dimension	Positive remarks	Neutral remarks	Negative remarks
Chatbot satisfaction	2	1	3
Chatbot functionality	4	0	2

From the survey data we were able to confirm the categorization of "expert" and "novice" users based on their prior experience of the platform, which helped us further evaluate the differences in responses across these groups.

Key Findings

Users were generally unaware of the chatbot unless initially directed toward it.

Novice users were more likely to use the chatbot, and found it more useful.

Expert users of the platform were more likely to be frustrated with the chatbot and be hindered by being forced into use.

Specific areas of frustration and confusion with chatbot functionality were identified with recommendations based on stated expected outcomes.

What if I had the chance to do it again?

If I had a chance to perform this evaluation again, I would like to spend more time focused on the chatbot itself, giving participants more scenarios in which they had to use the chatbot. The chatbot functionality was also only compared to the absence of a chatbot, whereas I think it would be useful to also have a comparison to other functionality such as a search bar. For methodology, I would add clickstream analysis by having a piece of software collect this data, though this would be easier if the evaluation could have been in-person.

Chatbot usability evaluation