(seed planted — october 2024)
I began following Maggie Appleton’s work after coming across Elicit, the company she led design at, through the MLOps and scientific research community. Not only has Maggie’s digital garden inspired the inception of my own, but I am deeply interested in her philosophy and framing around designing LLM products. The most recent interviews she’s done for Dive Deep and Local First tie so many principles that are shaping up in the UX landscape of LLMs. I’ve pulled out a few of her ideas here to expand on.
1. “Condensing, synthesizing, sense-making AI is way more important than generative AI”
This is a direct quote from Maggie and the most pertinent problems across AI today directly reflect this. There is a shift in LLM product thinking which is pulling away from sole generative capabilities and more emphasis on the ways that we can transform our workflows. How can our agents work in consortium with one another? Our interactions with LLMs remind us that these are powerful reasoning engines. We should treat them as critical reasoning partners, and our UX experiences should highlight this.
First, we want to be able to do complex tasks faster. Agentic workflows can accomplish tasks where ambiguity and reasoning are capabilities required to serve a more complex outcome of a task. We want to think more in systems and less in pure generative capabilities. How can we use and think of LLMs to integrate in and design certain experiences well?
I like how Maggie talks of how we have grew with language models vs. directly built them. They’ve sprouted out of some organic, possibility that has been planted and designed out. There’s lots of latent possibility around what these language models can do. We are only now cooking with them in ways where we are wrapping and harnessing capabilities into our daily experiences and flows.
The chatbot as an interface is far too open-ended gives too much onus to the user to figure out what it is they should be doing and interacting with the product. We have to be critical of this assumption. It is extremely user dependent and assumes high comfort level and understanding of the power of prompting and how doing this well can accomplish many tasks. This layer is extremely powerful in that it abstracts out multiple processes behind the ways the workflows are operating in the background.
However, without a goal (even with in-line suggestions and buttons) this direct experience is too open-ended. There’s a hedged bet you make with the user, relying on them to be in the steering wheel that drives the menu and cooking of the end goal. Ultimately AI services as highly advanced search. When the user is very clear of the direct output of the task they are trying to perform in an application it becomes a powerful co-pilot experience. Guardrails are often required to give the user the best experience without going into tutorial mode for too long.
Contrastive to an open-ended horizontal product experience such as a chatbot, comes the power of vertical interfaces. In this evolution of how to meaningfully integrate AI, there is a new focus on efficiency and workflows. When building out a system, we can lean on already existing software design principles which break down largely integrated and complex processes into their constituent individualized and specialized parts such as classes and tasks. For example, we want our agents to do a very specific thing really well. We want to defined super specific goals with pre-defined workflows. Handling data at the right time, handing it off to subsequent operators to then go ahead and do other tasks, how can you identify patterns within masses of data you’re trying to understand?
2. Cognitive architectures
Frameworks such as LangChain are innovating in the developer tooling open-source local development of this space, enabling ML engineers and enthusiasts to adopt cognitive architecture frameworks to build human-agent and multi-agent systems. Now that multi agentic frameworks are being explored creatively and built integrated into existing systems as their own layers, the goal can transition into making LLMs more truthful, reliable, and accurate.
Keeping the user informed of all the varied processes the LLM is making has become the newest UX problem that builders are trying to resolve. How do we integrate in and simplify complex decision making loops, searches, and executed processes the LLM system is engaged in. Besides machine learning architectural challenges involved in latency and scale, how do we expose this “thinking” layer so it is not overwhelming to someone, but exposes the procedural steps a system operates? Builds trust, builds exciting analytical experiences where user can participate in the transformation that is occurring. I see it as a magic layer. It’s almost GUID. We’re almost taking an orchestration task and processing it live for the user.
AI agent architectures ultimately “think thoughts” — the new UI/UX question is how do we expose the thinking layer in such a way that it is simple and beautiful? Reference and integrate in Sequoia blog post to include in diagrams that break down the cognitive processing layer here. Good cognitive layers must exist and sit on top of core strong and reliable infrastructure that is already in place. We still need fundamental software basics to be able to do this well. If you have a proper codebase in place, you put AI o top to train and you’re winning in the market. It also showcases missing gaps that may exist in terms of connectivity and processes within your codebase. We cannot create net new API actions. We are simply strengthening the connections that exist already in the architecture. I see this akin to neural plasticity in that: we are not necessarily growing brand new neurons (pure scale). We are strengthening an existing amalgamation of neuronal networks or creating net new paths that can be accessed in new, varied, and important ways.
- Elicit’s research on Factored Cognition have really solidified this principle of thinking up the actions an LLM can perform as ingredients (tasks), and in combination these can build up recipes (workflows).