Building Excellent UX in the Age of AI

Flightcrew is an AI tool that catches and fixes thousands of cloud infrastructure issues by shifting-left into native engineering workflows. This means we rely on LLMs to convince a skeptical and stressed engineer that they should make a change to their cloud infrastructure.

AI interactions complicate traditional Product and Design loops: context is key, product experience is probabilistic, and intent is no longer channeled but inferred.

Here's what we've learned from building Flightcrew.

Adapt to your Product Surfaces(s)

Building an AI Product can feel like a whole new world but the good news is that most of the old usability rules still apply to the surfaces of your product.

If your AI product is text-centric, many smart people have thought how to make text scannable and branded. Similarly if you are building a chat interaction with Slack, then you'll find plenty of literature on chatbot usability.

The difference today might be that much of your usability gains are achieved at the prompt level.

For example Snyk has a pretty good case study of how they improved the UX of their VSCode extension and it seems their biggest usability wins were through better prompts.

It's a reminder that usability is a team objective. Designers know usability best practices, and Engineers hold many of the keys to success.

Make Portability your Ally

One of the unsung benefits of LLMs is the portability it unlocks for your product - our users can access Flightcrew's intelligence through GitHub, Slack, Jira, etc.

The usability challenge is that GitHub, Slack, Jira all use different presentation layers, and users have different associations with each of these platforms. For example GitHub uses a variation of markdown, and aside from abbreviations like 'SGTM' it's a pretty formal and prescriptive place. On the other hand Slack uses Block Kit with mrkdwn and a lot of context/sentiment is expressed through emojis. Much less formal! Preserving usability gets tricky when a workflow crosses surfaces with different UI patterns and user expectations.

What's worked for us?

1. Enforcing consistent naming conventions and information model across surfaces means we can reuse data models and prompts. It's the only scalable way to preserve usability.
2. We implement a few UI patterns across surfaces to bring a common thread of authorship. Within an agent context, the user needs to recognize source just as much as object.

In short, we've built a minimalist design system to carry us across surfaces. It might feel like overkill to some startups, but the alternative is far worse.

Use the concept of Task Fidelity to set the bar for releases

The unstructured and probabilistic nature of 'Chat' can make it difficult to align on a Definition of Done for a feature or release. We’ve borrowed the concept of Task Fidelity to give us a common language for product usability across features and surfaces.

We use this framework to calibrate our usability standards per feature. To be specific with

Engineers ask a lot of Low Fidelity questions in Slack which are usually 'search with typos'. Person-person and Person-Agent communications are usually Low Fidelity

If you look at the PagerDuty app you can infer that Incidents and Incidents are Medium Fidelity. Content is precise and formatting encourages scannability, but tone and content are pretty utilitarian. Anything that is compared, scanned or intermediate' is probably Medium Fidelity.

So what’s High Fidelity? In our view it's artifacts with social or quantifiable stakes. A Flightcrew generated GitHub PR is high fidelity because it is human reviewed, public to a team, and will likely fix (or break something).

Aligning on Task Fidelity means a clear definition of done. Do your team a favor and decide this at the start of a sprint.

Educate with Spare Citations

When we started looking at session analytics we had trouble understanding the link between engagement (session depth) and value (PR approval rate). Many of our analytics benchmarks for a good or bad session didn't apply to the iterative and open-ended nature of prompt interactions.

What became obvious in usability testing was that many short, failed sessions were caused by users verifying Flightcrew information through cloud console, Datadog, GitHub, etc.

At first we compensated by porting this data into our surfaces (ex: summarizing Datadog in our slackbot), but quickly found that this caused other problems. What worked best was to to link and describe the data we had access to while largely skipping summarization. This 'Spare Citation' gives the user the confidence (or excuse) to skip immediate verification of Flightcrew's context.

I'll quote Öznur Özkurt via on the Google Design Blog

“We found that people don’t necessarily need to understand the math behind the algorithm to trust it; the algorithm can show the user what it’s thinking by outlining what it sees,“

I'll add that in a B2B context the majority of your users won't have seen your marketing and are not educated on your product's footprint and capabilities. Your product has to educate the user and sell itself.

Working in Public on a Work in Progress

Usability for AI products is a new field and we're excited to see the best minds from Product, Design and Research align on a few set of standards.

Ironically it's never been more difficult to find and reference good research so send us a note if you've got something to share.