We Need a Revolution in Software Interaction Design

We need a revolution in software interaction design. Apple and Google will not provide it. They are too big. They are not the solution. They are the problem.

This revolution will probably come from some unsuspecting source, like the Maker Movement, or an independent group of people or company that is manufacturing physical goods. Here’s why: as computers increasingly inhabit physical objects, as the “internet of things” grows, as more and more computing makes its way into cars, clothing, and houses, there will come new modalities of interacting with software. And it will be dictated by properties of the physical things themselves. Not by the whims and follies of interface designers, whose entire universe consists of a rectangle of pixels and the touch of a user.

Let me explain what I mean when I say that software interaction design needs a major paradigm shift.

Affordance

I hate having to use the word “affordance”. It’s not a very attractive or colorful word. But it’s the best I’ve got. If you’ve read my other blogs posts or my book, Virtual Body Language, you have heard me use it before. The word was given higher currency in the user interface design world thanks to Donald Norman, whose book, The Design of EveryDay Things, I highly recommend. (He was forced to change the name from the “Psychology of Everyday things”. I like his original title better).

Affordance, originally used in J.J. Gibson’s theory of ecological psychology, refers to the possible ways an animal or human can interact with an object (which can be another animal). We often use it in reference to the ways that one interprets visual, tactile and sonic features of a thing, be it an egg-beater, a frightened dog, or a new version of iMovie.

A “natural affordance” is a property that elicits an understanding or response that does not have to be learned  – it’s instinctual. In reference to industrial design: a knob affords twisting, and perhaps pushing, while a cord affords pulling. The term has more recently been used in relation to UI design to indicate the easy discoverability of possible actions.

UPDATE…

Bruce Tognazzini pointed out to me that Donald Norman has more recently been using the term “Perceived Signifier”. And this article explains some of the new semantic parsing going on regarding the word “affordance”. Personally, I would be happy if all of this got subsumed into the language of semiotics.

I believe we have WAY TOO MANY artificial affordances in our software interfaces. I will repeat the call of many wise and learned designers: we need to build tools with natural affordances in mind. Easier said than done, I realize. Consider a common modern interface, such as the typical drop-down dialog of the Apple that allows one to download a file:

As a general rule, I like to download files to my desktop instead of specifying the location using this dialog. Once a file is there, I then move it to the appropriate place. Even though it takes me a bit longer, I like the feeling of putting it there myself. My muscles and my brain prefer this.

Question: have you ever downloaded a file to a specific, somewhat obscure folder, and then later download another file, thinking it was to your desktop, and then not being able to find it? Well, you probably didn’t think to check the dialog box settings. You just hit SAVE, like you usually do, right? It’s automatic. You probably forgot that you had previously set the dialog box to that obscure, hard-to-remember folder, right? Files can get lost easily, right? Here’s the reason:

Software files are abstract concepts. They have no physical location, no mass, no weight. All the properties that we associate with files are virtual. The computer interface is just a bundle of physical metaphors (primarily desktop metaphors) that provide us with affordances so we can think about them as if they were actual things with properties.

The dialog I showed you doesn’t visually express “in” in a natural way. The sensation of the action is not like putting a flower in a vase or drawing a dot in a circle.

Herein lies a fundamental problem of software interface and interaction design. Everything is entirely arbitrary. Natural laws do not apply.

Does the natural world present the same kind of problem as we have when we lose files? Sometimes, but not so often. That’s because the natural world is full of affordances. Our memories are decorated with sensations, associations, and connections, related to our actions. If I physically put a rubber band in an obscure bowl on the top shelf, I have reason to remember this action. Muscle-memory plays a major part in this. With downloading files on a computer, you may not know where you put the file. In fact, sometimes you can’t know!

In fact, it’s not fair to use the word “put”, since “putting” is a deliberate, conscious act. An accidental fumble on the keyboard can cause a keyboard shortcut command that deletes a file or opens up a new window. When this happens, the illusion breaks down completely: this is not a real desktop.

Am I getting too esoteric? Okay, I’ll get more down-to-earth and gritty…

My Deteriorating Relationship with Apple

Bruce Tognazzini says: “While Apple is doing a bang-up job of catering to buyers, they have a serious disconnect at the point at which the buyer becomes a user.” 

My recent experiences with Apple software interfaces have left me worse than disenchanted. I am angry. Apple has changed the interface and interaction of one too many of its products, causing my productivity in some applications (like iMovie) to come to a screeching halt.

At the end of the day, I’d rather keep my old computer with my trusty collection of tools than to have a shiny new, sexy, super-thin Macbook that replaces my trusty old tools. There are years of muscle memory that I have built up in learning and using these tools. My career depends on this muscle memory. When Apple changes these interactions with no clear reason, I become very angry. And so should all its customers.

And then…there is iTunes.

Let’s not talk about iTunes.

Apple’s latest operating system disabled many of the interactions that I (and many others) have been using for decades.

One annoyed user said: “I wish Apple wouldn’t change the fundamental functions of the Finder like this. First, they reversed the scroll button directions then they changed the default double-click to open a folder in a new window function. I’m just glad that they don’t make cars otherwise we would have the brake and accelerator pedal functions reversed (with a pull on the wiper lever to revert to the original)!”

And from the same thread… “If a company cares about users, it doesn’t make a change just for the sake of change that wipes out thirty years of muscle memory.”

Am I saying that a company like Apple should never change its interfaces? Of course not. But when they do, they should do it carefully, for valid reasons, and gracefully. And they should ALWAYS give their customers the choice of if, when and how to adapt to these changes. Easier said than done, I know. Apple is like most other companies. They feel they have to constantly make NEW products. And that’s because our capitalist system emphasizes growth over sustainability.

Will the Maker Revolution Cure Us of Arbitrary, Ephemeral Design?

This is why I believe we need to return to natural affordances that are as intuitive as putting a spoon in a bowl or carving the bark off a stick. The more natural the affordance, the less arbitrary the design. Designers will have to be less cocky, more reverent to human nature and physical nature.

When real physical things start dictating how we interact with software, the playing field will be different. And software interaction designers will have to fully understand natural affordances, and design for them. That’s a revolution I can get behind.

.

Advertisement

If Tablets Become Any Thinner, I’ll Get a Paper Cut

The trend in miniaturizing electronics has reached a point at which our lumbering human bodies cannot keep up.

Taylor Martin, in a blog post, says: “Apple and Samsung are already at each other’s necks…. proving who can make the thinnest and lightest device seems to be their main ambition and I fear they’re going to take it too far at some point.”

Natalie P, in another blog post about the Apple Nano, complains:  “My headphone cord is literally heavier than the entire Nano device…”

Human Factors and Ergonomics

I prefer using the older, heavier tablets and smartphones than the newer, super-thin variety. My hands were just not made for things that thin and that light.

The human body and mind evolved to manipulate hand-held objects of a certain range of size and weight. Originally these were sticks and stones, then later, manufactured tools, and…books. The book remains a form factor that fits comfortably in the hand…and the mind. For this reason, I believe that the book will never die as a standard form factor.

When I hold one of those new super-thin tablets – without any protective covering, I feel as if it could easily slip out of my fingers. Maybe this is why the protective covers are so popular.

Now…check out this blog post which is aptly named iPad as Weapon.

Apparently this boy and his sister had an argument involving Minecraft, which led to an injury.

Never mind the fact that their argument turned into a fight.

What worries me it the weapon involved.

Riffing Deacon, Dehaene, Rotman, and Tegmark: More on the Nature of Mathematics

Is mathematics an invention of the human mind, or is it a universal “eternal truth” that we humans discover?

I once wrote a blog post about Brian Rotman’s book, Ad Infinitum, in which he advocates “taking God out of mathematics and putting the body back in”.

Since writing that blog post, I have thought a lot more about the reality of math. And one reason is that I have since written a book about a set of fractal curves that I “discovered”. Or, did I “design” them? (read on to find the answer :)

Brain Rotman recently wrote a review he wrote for the Guardian on the book Our Mathematical Universe, by Max Tegmark.

Rotman’s review is quite critical. Tegmark claims that mathematics is the very foundation of the structure of the universe. He goes as far as to say that “Our reality isn’t just described by mathematics – it is mathematics”.

Rotman would be one of a growing number of mathematician/philosophers who take issue with this kind of claim. They would counter that mathematics is a human invention. My first encounter of this idea was through Lakoff an Nunez in their book, Where Mathematics Comes From – How the Embodied Mind Brings Mathematics into Being. This book explores mathematics from the point of view of cognition and linguistics.

All very well and good. Math comes from human brains. But the more I engage in mathematical activities, the more sympathetic I become with mathematicians who believe that they are “uncovering” great truths rather than creating ideas out of thin air. While exploring mathematical ideas, the feeling that I am discovering something outside of myself is very strong. What is the reason for this feeling? What is happening in my brain that makes this feeling so strong?

Why I am Becoming a Mathematical Agnostic

I am just about finished reading The Number Sense by Stanislas Dehaene, who is working on the cutting edge of brain imaging and identifying cerebral structures associated with abstract number and calculation. 

Dehaene believes that math is a human invention.

However, he acknowledges the long reality that precedes us. According to Dehaene, the notion of number is perhaps the most primal mathematical construct that our brains perceive, and we share a rudimentary number sense with other animals.

No surprise here: the universe – at least on the scale of mammals and birds, has clumps of matter and events that we perceive and that are meaningful to us. And of course the biosphere is doing its part to continue clumpifying matter and events, including me writing this blog post.

“Number” could be seen as an emergent property of the biosphere. Animals have evolved behaviors associated with number (as well as symmetry, and other attributes associated with math). In turn they impose their emergent representations back onto the environment as part of an ongoing feedback loop of complexification. Humans have simply taken this feedback loop to a conscious level.

A few years back I read Incomplete Nature, How Mind Emerged from Matter by Terrence Deacon. Although his writing is painful, the ideas in the book are very thought-provoking. What I learned from Deacon makes me pause before heading straight to mathematical atheism. I recommend this book. Deacon points out the many levels of emergence that forged the structures of our universe, and ultimately, our minds.

My conclusion: to say that math is strictly a human invention is taking it too far. Math is not arbitrary. We are made out of structured matter that exists throughout the universe. Our bodies, brains, and minds are tuned-in to that structure. We evolved with it and in it. Our language, as well as our genes, are “about” the biosphere, which is “about” the universe.

On Discovering vs. Designing

In Brainfilling Curves, I chide Mandelbrot for saying he “designed” certain fractal curves. I can’t say that I blame him, considering how clever the snowflake sweep is, for instance. But I prefer to say he discovered it.

And I claim that many of the fractal curves in my book were discovered by me. But it gets a little fuzzy at times. Geometrical objects range in primality from the circle (which no one would claim to have designed) to rare, high-order self-avoiding space-filling curves, many of which I have spent years uncovering. The more rare, the more unlikely the geometrical object, and the more information required to describe it, the more like design it becomes. I see it as a continuum. At the far extreme of this continuum would be a painting by Kandinsky.

I believe that we feel the sensation of mathematical discovery because of the evolution of our brains, language, and culture, from which we cannot escape.

The evolution of our brains, language, and culture are a continuation of the evolution of the structure of the universe. Thus math, while it may not originate from the universe, is a language we invented which is finely tuned to it.

And since we are made of the stuff that our math describes, our math feels…

perfect.