Notes On AI & UXR

Researchers: I am here to tell you that AI has not made you irrelevant — but you’re doing yourself a disservice by ignoring it.

As with so many disciplines, the explosion of access to powerful language models has fundamentally changed our roles. We often bemoan the labor that’s nearly invisible to our stakeholders: the planning, documentation, negotiation, and coordination that distracts us from the meaningful work of learning. Tools like ChatGPT can immediately erase a surprising amount of this toil.

And yet, our skeptical researcher brains also know that LLMs are, in many ways, *less* capable than they appear at first glance. They retain limited context, their output is often dry and repetitive, and they’re highly confident even when they’re completely fabricating information. As professionals, can we safely rely on such constrained tools? (That one infamous lawyer who recently got in hot water for submitting a brief with completely fabricated ChatGPT-generated sources might argue we shouldn’t!)

As ever, the answer is “it depends.”

In order to figure out how to best use LLMs in your workflow, it’s helpful to think about how these tools work. While the industry debates whether LLMs are “stochastic parrots” that simply stitch together algorithmically-likely sentences or whether they approach actual artificial intelligence… let me assure you, they aren’t able to replace the artistry and expertise of a pro researcher.

LLMs are amazing at creating templates and outlines. They’re also great at remixing data into new formats. But they are NOT so great at knowing what your business should build next or how to best meet your users’ needs.

I recently experimented by ‘pairing’ with gpt4 on a round of messaging research. It did a decent job of translating research questions into a discussion guide draft, and an excellent job at ministerial tasks like generating e-mail comms.

It shined the most at maximizing our speed to learning. By feeding it transcripts from session recordings, I was able to get instant summaries, including takeaways and quotes. I could then offer these to my partners within minutes, an impossible turnaround time for a mere mortal. 

Of course, those summaries lacked all the implicit signals that we know to look for. That’s where you come in. (Though I can imagine adapting my technique to ‘prime’ my transcripts for AI-assisted analysis in the future, much like radio hosts describe visuals for the benefit of their listeners.)

And it may surprise nobody to learn that when I asked gpt4 to generate recommendations, they were thin, shallow, lacking context, and hardly actionable. None of them made it off the cutting-room floor.

As you explore what’s possible, one thought I’d like to leave you with: as ever, the synthesis IS THE WORK. It’s hard, but we can’t automate all of it away. Without forcing our brains to simmer, we can’t bring the creative spark that ignites product work. Instead, WE risk becoming the stochastic parrots.