Dashboard Focus group

Problem Statement

The client was developing a personalized dashboard for university faculty members to show them performance in their classes along with information about how these statistics could be improved. The data included was fake data, not personalized for the users of the focus group, but participants were asked to act as if it were. The same data was used for all participants.
Key Research Questions:

1. What were users' reactions and behavior to the data being provided to them?
2. What were users' reactions to seeing negative results from their teaching?
3. Can users understand the way the data is presented such that it could be used to improve teaching?

Methodology

This was performed during the COVID-19 pandemic, and therefore there was a requirement for all testing to be performed remotely. Due to the busy schedule of participants, sessions could not last longer than 60 minutes.
In this evaluation we opted for a series of focus groups with a walkthrough analysis of a prototype of the website. After the focus group, participants were also emailed a survey for demographic data and to include additional information in case they wished not to disagree with others in person.

Recruitment

The personalized dashboard was being designed for a series of universities, and was intended to be viewed by professors and department heads to review data. Therefore recruitment was limited to current (or recently retired) professors that were (or had been) working within the associated group of universities. As a professor himself, our supervisor reached out to colleagues within those universities to gather participants for the study.

Challenges and Solutions

Challenge: Schedules

Due to professors' busy schedules we were only able to have focus groups lasting 60 minutes. As such we had to be very careful with our question selection. Several dry-runs of the questions were performed with colleagues to make the questions less vague, prevent straying topic, and questions were combined where possible, or ommitted when less relevant. Due to the busy schedules it was also difficult to get a day where all participants were available at the same time, therefore we ended up having a series of focus smaller focus groups. Smaller focus groups meant that there was more time for answers for each participant.

Challenge: Remote Focus Groups

We identified several potential problems that may arise from remote focus groups along with solutions. For example:

Problem	Solution
Distractions	Close all other computer applications, and enable "do not disturb" functionality if applicable. Turn off cell phones.
Mic/Speaker/Camera problems	Try to get everyone on the call early to confirm no problems. 2 interviewers were available so that 1 could take over if technological problems with the other.
Ease of interruptions	Remote meetings make it very possible to easily talk over each other. We tried to have each person speak one at a time and use "hand raise" features available.
Straying answers	Prompts created to keep participants focused on the question at hand. Where possible, questions were re-worded to prevent straying.

Data Analysis

Focus groups were transcribed, and then analyzed to determine several dimensions along with the survey responses.
Dimensions such as "Interface" and "Clarity" were identified, and comments were grouped together as either positive, negative, or neutral comments within these dimensions.
Areas of agreements and disagreements between participants within the focus groups were also identified in case they affected other results.

Key Findings

• Participants from a more scientific background were confused by the negative results reported, often disagreeing with the results, and wanted more of a breakdown as to why the results were negative, preferably by having raw data available.
• Participants from a less scientific background seemed more accepting of the results and were focused on how to improve their performance.
• All participants felt that the resources provided were insufficient to improve their situation, but with varying reasons why.
• There was also a concern that this could be used as an evaluation tool when there may be other circumstances not considered in this dataset.

Outcomes

• Actionable insights were added to pages to make it clearer how this data could be used to improve in the future.
• Specific data deficiencies were resolved prior to going live that made those graphs and datasets more useful to professors.
• The platform was able to show how raw data could be obtained.

What if I had the chance to do it again?

If I had the opportunity again, I would have loved to have had much more time to go through the prototype, and if it was further in development, be able to perform a usability evaluation where the participants could go through it themselves. The participants that wanted to see more data really wanted to be able to dig down further to see more information, but those pages were not available, or may have been incomplete. Although the focus group helped understand this request for additional data, these participants may have been more useful with a more complete prototype.
I think it is also difficult to accurately act as if negative data is being displayed about your performance. Some participants had difficulty relating to the data provided (for example, class sizes were too big, or demographic information was very different from their typical classes), which meant their reactions to negative data were likely harder to imitate, with the data being so far from their reality. A more accurate representation of emotion may be possible if participants were actually looking at their personalized data, but it had been modified to show more negative results than reality.