When you’re trying to fall asleep it’s usually a terrible idea to start thinking about interesting problems. But I might get a blog post out of it, so it’s all good.

It all began with media.

We rarely notice them but they are absolutely fundamental to everything we do. This sentence originated in the medium of neural excitations, was subsequently converted to the medium of text, and, after having crossed too many other media to mention, is being reassembled in your brain now.

Text

There is only a handful of media we use for creative purposes and arguably none is more important than text. Which is weird, if you think about it, because it’s an absolutely awful medium to express most any idea. Why do we do it then? I believe there are mainly two reasons.

First, the medium is extremely versatile. Sure, it might take several pages to describe the beautiful scenery you could snap with a camera instead, but at least it’s possible. Which is great when you don’t have a camera on you. Or when the scenery only exists in your head.

The other reason is our modern times gave rise to devices which do not let us express ourselves properly. We are essentially reduced to smart phones and PCs. The former provides audio and video capture but there is very little you can do to express novel ideas without typing. So not even did the new medium (a touch display) failed to provide new means of interaction, it kept the old one and made it worse.

You do have more options on a PC but the keyboard is still the ultimate gatekeeper. Everybody knows that that’s how you do computers, so nobody asks for anything better, and therefore nobody delivers anything better. And since we don’t have anything better than keyboards, all the software we use relies on this single method of input and disproportionately favors textual representations.

Worlds collide

Our world already enables us to manipulate objects inside of it. It’s so intuitive we never even think about complex motions that are executed when grasping a glass of water to take a sip. Imagine if we could recruit this fastastic machinery in our hands to reveal our thoughts and play with them. But as great as our world is, it is very rigid and expensive to modify. We cannot conjure up stars in our room, configure their parameters, and watch them move. But that’s exactly what computers simulations are good at.

Unfortunately, to express this idea as a computer program requires a lot of typing in a particular programming language, which has two fatal flaws: it excludes the part of the population that cannot code, a big majority; and even for programmers it would take so much time to code it up that many of them wouldn’t pursue it or would give up in the process. Most of the ideas are forever locked in our heads and will never be tried out. And thus the only option we usually have is to search for images that already exist and link to them, like I did above.

We need to be able to combine the ease of object manipulation in the real world with the endless possibilities of computation in the digital one. To that end we need to completely replace our software and hardware stacks. I am not going to talk about hardware since I don’t know much about it and there are already pioneers forging ahead, such as Bret Victor ranting about how we should use the full capabilities of our hands and Saku Panditharatne on combining AI with AR/VR to obtain the kind of tools you can see in Tony Stark’s house.

Instead, let’s talk about the software since that is both much easier to fix than replacing billions of keyboards with novel controllers and at the same time still offers a lot potential to improve on our text-oriented applications.

The elephant in the room

The lack of hardware devices is not at all the only problem we face when getting rid of text. After all, we can also use the mouse to enable a much wider range of expression. And we can use the keyboard to produce non-textual interfaces, such as the widely loved Control-C Control-V combo. In fact, we could do worse than learn from graphical editors, such as Photoshop, which have long embraced this mode of expression (and also marched ahead in the hardware sector with drawing tablets). They provide a rich interface for editing pictures using layers, effects, edit history and many other tools.

Compared to visual media, programming is stuck in the middle ages. A computer program is still a piece of text. If you have a typo, the program won’t run; or worse, will produce rare and terrible effects and you won’t know why. Even if we could prevent typos, you often cannot know what your program will do until you run it, often a matter of hours when doing big changes in a codebase. There has been liters of ink spillt on how best to alleviate these problems with better language features. But as far as I know, rarely do people acknowledge the elephant in the room that is the opaque textual representation; and only very few people research alternative methods for producing and interacting with programs.

In fact, this is such a big problem, that it plagues many other fields besides programming. Like mathematics. The subject could be presented in a very intuitive and fun way, with beautiful interactive visualizations rather than ugly symbols manipulations nobody really cares about and eventually learns to hate.

But I intend to help with some of these issues, in my own small way.

Stay tuned.