Project Overview
Deliverable
User Research
Key Outcome
AI/ML Prototype
Partnered with data scientist to creating working prototype for user study
14 Interviews
Conducted 14 interviews and translated user needs to product requirements
24% Improvement
In a study, participants identified 24% more risks, benefits, and considerations of a fictional project using our tool
Hypothesis Definition
Recent advances in generative AI have led to greater accessibility
ChatGPT has popularized the use of Large Language Models (LLMs) like never before. This led us to wonder: if, how, and why will product developers (i.e. product managers and designers) use LLMs?
In organizations, the volume of documents is too much for any one person to synthesize
We spoke to 14 diverse subject matter experts across industries. We learned it's important to incorporate insights from past documents into new product ideas to avoid repeating past mistakes. The expansive volume of files is an obstacle.
Product developers benefit the most from past documents during brainstorming
We asked our users: when would product developers benefit the most from utilizing institutional knowledge? They told us: the earlier the better.
How might we amplify brainstorming using generative agents?
Creatives use thinking frameworks because they help to think through a problem completely. What if we could make Edward de Bono's Six Thinking Hats into generative agents, so you could have a conversation with them? Would this lead to more idea generation?
Hypothesis
The use of generative agents, modeled after a popular thinking framework, by product developers during a brainstorming session will lead to a greater number of ideas compared to using a generic LLM.
Experiment #1
We built a tool to investigate if, and how, product developers would use a generic LLM
We created a chatbot using the OpenAI API connected to a folder of supporting materials.
.gif)
Given the option, product developers use the LLM
All participants used our tool more than the files directly. The study included 7 experienced product developers. Participants brainstormed on a hypothetical product for 30 minutes. Using a folder of relevant files and our generic LLM tool, they brainstormed risks, benefits, and other considerations. They 'thought aloud' during this task.
Thematic analysis of 'think aloud' results
We conducted a thematic analysis, grouping quotes into themes, and prioritizing themes based on prevalence.
Experiment #2
Would brainstorming with multiple generative agents lead to more ideas?
We updated the tool so that users could chat with a group of generative agents, each representing one of Edward de Bono's Six Thinking Hats. We modified the generative agent architecture by Joon Sung Park (2023) to add a knowledge base of files and different system prompts. Would brainstorming with multiple generative agents enable product developers to identify more risks, benefits, and considerations compared to the generic LLM tool?
Head to head: Generic LLM vs. Multiple Generative Agents
Again, participants brainstormed on a hypothetical product for 30 minutes. Using a folder of relevant files and the Generic LLM or Multiple Generative Agents tool, they brainstormed risks, benefits, and other considerations. We tested with 10 participants (5 test, 5 control) and again asked them to 'think aloud.'
Result: 24% more ideas by those using Multiple Generative Agents
The test group (Multiple Generative Agents) was able to generate 24% more risks, benefits, and considerations.
Note: Risks, benefits, and considerations were coded to prevent double counting.
Improved brainstorming and sensemaking
For product developers who need to improve their brainstorming and sensemaking ability on a large corpus of files, chat with multiple generative agents connected to the corpus can be effective.
Looks-Like Prototype
Introducing Perspective
For product managers and designers, we propose a tool to chat with multiple generative agents simultaneously, to improve sensemaking on a large corpus of files and make creative brainstorming more complete. Relevant files are surfaced to the user as they come up in conversation.
Multiple agents in one chat
Integrates with files
Customize chat partners
Auto Chat
Footnotes
Through this Independent Design Engineering Project, I was able to attend the UIST 2023 Conference, visit Google Cambridge, present at IDEO, read countless HCI papers, and deepen my knowledge about generative AI. I'm extremely grateful to our advisors and instructors for this time of growth.