Enhancing the experience Ruby GenAI chatbot
I led a UX evaluation of Ruby GenAI's beta feature for generating Help Center summaries to prepare it for broader release. Using a 5-day diary study with 6 participants and 17 interaction responses, followed by in-depth interviews, I uncovered 6 key pain points related to transparency, explainability, and personalization. To address these, I designed conversational improvements that enhanced the chatbot, boosting the KPI by 30.8%. Additionally, I identified 4 new use cases for Ruby AI integration, influencing the product strategy for the next two quarters. I designed an AI experience to automate and streamline support ticket creation, significantly improving the user experience.
DURATION
3 months
MY ROLE
UX Intern at Rubrik
METHODS
Diary Studies, Semi-Structured Interviews, Qualitative Analysis, Conversational Design, UX Design
Problem Statement
Through diary studies and user interviews with end users, I identified six key pain points, synthesized as follows:
Users perceived Ruby GenAI's response, "Sorry, I do not have sufficient information…" as a failure of the system to address their queries, creating an undesirable experience. This response discouraged users from continuing the conversation or providing additional context to refine results.
Additionally, I uncovered four use cases where integrating AI into user workflows could provide significant value. I focused on one use case, synthesized as follows:
Users faced frustration and inefficiency when creating a support case, as they had to manually re-enter contextual information already shared during troubleshooting with Ruby. This redundancy not only prolonged the process but also contributed to a poor overall user experience.
Goal
Solution
I redesigned the conversational flow of the Ruby GenAI chatbot to address key pain points and enhance the user experience. Additionally, I designed AI-driven solutions to simplify troubleshooting and automate support ticket creation, improving user experience.
Designed to automate & simplify support ticket creation process
Through an iterative design process, I developed a streamlined flow for automating support ticket creation, enhancing both efficiency and user experience.
15 seconds walkthrough of the solution
Design for transparency and explainability in Ruby’s response
Previously, Ruby responded with, "Sorry, I don’t have sufficient information," for the following use cases. I redesigned the conversation to improve clarity and user experience. Here's the updated version:
Requesting Information Ruby Can’t Access
Clearly communicate that Ruby cannot access personal information, reassuring users of their privacy and building trust.
Requesting a Feature, available in a
Higher Version
Highlight that the requested feature is unavailable in the current version but accessible in a higher version. Provide a clear CTA to explore upgrade options, aligning with business goals.
Requesting a Non-Existent Feature
Inform users that the feature does not exist, and offer the option to share their request with the product team. Assure them their feedback is shared only with their consent, reinforcing trust.
Research
Objectives
Through all the 1:1, I synthesized following as the key objectives of my research.
Evaluate the UX of Ruby GenAI Chatbot
Assess users’ experiences and satisfaction with Ruby’s new beta feature in their daily workflows. The goal is to ensure the feature effectively supports tasks and delivers meaningful value before its wider release.
Identify use cases for AI Integration
Discover and prioritize additional use cases where AI integration in the product can add value for users. These insights will guide future development and enhance Ruby’s overall utility for Rubrik users.
Research Methodology
The evaluative research was structured into three key phases: Opening Interviews, Diary Studies, and Closing Interviews
Analysis
I used Dovetail to transcribe the interviews, which I then coded and analyzed to derive key findings. For the diary studies, I exported the data into Excel, where I conducted an in-depth analysis and developed follow-up questions for each participant's closing interview.
Findings
After synthesizing the data and generating findings and recommendations, I presented the insights to key stakeholders. The findings included:
Final Design
Leveraging research insights from the 'support ticket creation' use case, I iteratively refined and crafted the final design. Below is a comparison of the user flow before and after implementing the proposed solution.
User Control Over Data Access
When enabling Ruby, users can manage the metadata Ruby accesses, with clear explanations of why each data point is required.
Automatic Contextual Information Gathering
Clicking the Ruby icon automatically collects metadata to generate a personalized response, displaying it in a sidebar for users to follow along while continuing their tasks seamlessly.
Content References for Transparency
To build trust and establish authenticity, Ruby includes references for its generated content, linking each suggestion to the respective source document—similar to citations in research papers
Automatic Support Ticket Drafting
If Ruby’s suggested solution doesn’t resolve the issue, it prompts users to submit a support ticket by auto-generating an editable draft for quick submission
Reflections
At the start of the project, I spent considerable time refining how to ask a specific question during user interviews: What is their perception of Ruby? I proposed using images for participants who struggled to articulate their thoughts verbally. After making a case for this approach, we tested it during the interviews. This taught me the value of advocating for experimental techniques and iterating based on their effectiveness.
I also learned that the interview question list should serve as a guiding framework, not a rigid script. Adapting questions based on a participant’s responses allows for deeper insights and a more organic flow to the conversation.
While designing an AI experience, I recognized the importance of carefully balancing the level of control given to users. This helped me make intentional design decisions that aligned with user needs and expectations.
Additionally, I realized the significance of keeping both user intentions and business goals in mind throughout the design process. This perspective enabled me to explore various iterations thoughtfully, evaluate them critically, and choose the most effective solutions.
What I would do differently
In retrospect, while I held individual 1:1 meetings with stakeholders to uncover known information, assumptions, and areas for discovery, I realized that some pain points discussed during the share-out were already known. To improve this process, I would have conducted a collaborative workshop with all stakeholders at the outset. This would provide a more comprehensive understanding of what is known and what needs to be learned, ensuring that the research plan is even more targeted and efficient.
Snapshots of my internship memories
OTHER PROJECTS
Out of all the pixels in all the screens, you ended up here. Grateful for your visit!
2024 Ravi Jangir. All Rights Reserved.
Let’s connect