A Utopian Future for Graphical User Interfaces

Can we imagine a better future for creating graphical user interfaces? 

Let’s scope “creating” to: visual design, writing code, managing a team, or authoring content & text. For each of these disciplines, and especially the collaborative interfaces between them, certainly the future will be better than today. But how so?  And why? How do we get there?

Let’s start by painting a Star-Trek-like utopian future — free from the constraints of the present — as it relates to creating GUIs.  Here are some brush strokes to get the canvas started.

The future is multi-modal

That is, you can create UIs using different modes — code is just one such mode.  Following are some other creative modes of the future, some of which we can already glimpse in the present.

Creative mode: drawing, painting, and visual building

In addition to code, you will be able to create user interfaces visually, with graphical tools akin to modern vector & raster design tools. 

We have already seen a glimpse of this with visual builders like Webflow, Squarespace, Framer, and Wix — and a vivid illustration of it back in the ‘90s and ‘00s with Flash’s visual design & development environment — but these tools have plateaued at a local maximum over the last two decades. 

Trying to move an image into the same column as some text in Framer. Why doesn’t it do what I expect it to do?

There is a steep learning curve to every one of these tools, and the “visual editing” process in fact requires understanding the underlying developer platform, HTML and CSS, and their complex, technical systems of styling and layout.  In other words, to be effective with these tools, you must first be a developer — which limits the utility of these tools as an alternative to writing code.

Can we move this heading to the top-left corner of its container in Webflow? Modern website builders are “technical & constrained.”

One day, after we break past this plateau of constrained, technical visual editors, creating app GUIs & websites will be ultimately be as intuitive and accessible as drawing, painting, or modeling clay.

Creative mode: natural language & AI

Natural language via LLMs and their successors will be another mode of creating user interfaces.  This facet of the future is already at our doorstep, at least early iterations of it, with tools like Durable.co and Framer’s AI.  

As an extension of LLMs transforming explicit requirements into GUIs, you can also imagine autonomous agents creating on their own, or some degree separated from an initial prompt.  Instead of a human turning the crank on each iteration, agents will be able to run with and satisfactorily complete, zero-shot, high-level tasks.

Finally, we will soon be able to input graphical inspiration and generate something that looks or feels like something similar.  A common tool for visual designers is the mood board — a collage of images that channel the “mood” of whatever is being designed, as a tool to get the creative juices flowing toward a yet-inchoate idea.  

A mood board that we put together while ideating the branding for Pax. You can imagine LLMs in the near future “channeling visual inspiration” like this into changes in a codebase.

LLMs will be able to accept prompts from us in the form of "this picture I took of the sunrise," or "this collection of brands assets or illustrations or graphics" — and channel that inspiration into a coat of digital paint on our apps & websites.

Creative mode: code

Is code here to stay?  To formulate an opinion on this, it’s worth examining why code exists in the first place. 

Code is a tool for unambiguously encoding data or sequences of instructions (algorithms) that we can reason about logically as humans, which also have meaning to computers. 

When we use the word “code,” we are generally describing either programming languages like C++ and JavaScript, or data-encoding languages like JSON and YAML.  In both scenarios, you can think of code as a common language between computers and humans.  With LLMs at our doorstep, we're finally breaking into the territory of natural language like English being a common language as well.  

So: will we create all of our apps, websites, and digital graphical interfaces entirely with English & other natural languages in the future? 

While English & other natural languages will increasingly become portals to computation, “code” as we think of it today will stick around for at least two reasons: 

(1) Ambiguity — Natural language is generally more awkward to channel into unambiguous instructions than it is to channel a programming language into unambiguous instructions.  As long as we speak organic languages like English, rife with both structural and lexical ambiguity, we will benefit from computer languages designed to avoid these pitfalls.

(2) Visuospatial information density — well paraphrased with the idiom "a picture is worth 1000 words." Describing graphical or spatial content — like a picture, or a GUI, or hierarchical information like relational data, or spatially related data like the values in a spreadsheet — will remain best served by code and data-encoding languages. This boils down to the fact that visuospatial information can be quite dense, while natural language is inherently sequential. For an illustration of this phenomenon, think about the last time you had to wait for a call center’s automated telephone recording to read out numeric options for you — and how you felt while you navigated this system — vs. how quickly you can find options when you have them all laid out in front of you on a screen.

The future is more collaborative

As humans, we tend to develop specialized skills.  Some people are proficient with code; some are proficient with drawing & painting; some are proficient writing natural language.  Some folks are dealt a lucky hand and are proficient across several different skills — but usually, we work professionally on diverse teams where different contributors have different talents, and this diversity becomes a strength of the team as a whole.

Now the problem today is, these modes of building software are mutually exclusive.  Either you create a GUI using code, or you use a visual builder like Webflow.  Either you draw an idealized GUI in a design tool like Figma, or you “design in the final medium,” — but you don’t do these things together, not in the same language.  Where we must collaborate across these modes, we do so in the form of “hand-offs” — of tasking a human being to translate, for example, a drawing of a user interface (for example, in Figma) into a “real” user interface (for example, in HTML/CSS, or Webflow, or developer platforms like SwiftUI / Jetpack Compose.)

In the future, we won’t have hand-offs.  Tools specialized for each creative mode will be able to affect the substrate itself.  We will be able to collaborate seamlessly across different modes, and each contribution will be additive and interleaved, instead of siloed and waterfalled.  The ability to create collaboratively around the same source of truth will not only make creation far more efficient, but more creative as well.  Imagine artists adding art to a codebase while engineers add code, and marketers add copy and managers add tickets & comments.

The future is democratized 

You may have already identified this in the negative space of what we’ve envisioned above — but by lowering the technical bar to creating interactive experiences, more and more humans will be able to create interactive experiences — apps, websites, interactive cartoons, games, data visualizations, 2D and 3D experiences, and augmented & virtual reality.  

As a reference point, consider the humble spreadsheet.  Anyone, with very limited technical training, can pick up & use a spreadsheet.  It’s so visual — point, click, type, enter.  Spreadsheets are also the sterling example of a “low-code” tool, where writing programmatic expressions has an extremely low technical bar, with immediate feedback, and without all of the ancillary technical hurdles of most programming environments.

This may help explain why Excel spreadsheet formulae are the world’s most widely used programming language  — and why as many as 2B people use spreadsheets today. That’s roughly 1/4 of all living humans.

In our utopian future, anyone who can make a spreadsheet can make an app or a website. Any mom & pop selling handmade goods in a developing economy will be able to use their smartphone to publish an interactive e-commerce site.  The prosperity enabled by digital connectivity and commerce will be extended to the entire globe, as more people can create interactive experiences, rather than just consume them.

How do we get there?

We face a certain truth today: we don’t yet live in our utopian future. 

“Truth is not what you want it to be; it is what it is. And you must bend to its power or live a lie.”

- Miyamoto Musashi

Our design & developer tools live across a chasm of hand-offs.  Visual builders are creatively limited and locked into proprietary platforms.  Not to mention, there’s tremendous momentum & commercial interest  behind existing technologies & tools, and these existing technologies aren’t going to unify themselves.  The truth is, we live in a fractured digital creative world.

But if we can imagine this future, we can build it, right?  This is the magic of invention after all: with enough audacity & grit, history has shown time & again that a third option can be forced into Musashi-sama’s dichotomy.  You can bend to the power of truth, you can live a lie — or you can bend truth itself, by building something that changes the rules.

“You never change things by fighting the existing reality. To change something, build a new model that makes the existing model obsolete.”

- Buckminster Fuller

When you think about it this way, a more creative, more efficient, more connected future seems well within reach. It will require blood, sweat, and tears — but a better future for GUIs is ours for the building.