C. Applied Semiotics: HCI as the result of semiotic processes.

1. HCI is an example of Peircean semiotics at work

The sign being the unity between what is represented (the object), how it is represented (the representamen), and the interpretation, let us now look at the signs involved in HCI. In other words, how do we design interaction with semiotic awareness?

2. One computer—many interpretations

In human-computer interaction, we can consider the computer as the object (what is represented); the operating system as the representamen (the desktop metaphor is an example of a representamen), and a specific use as one possible interpretation. Word processing is such an interpretation. Obviously, in such an interpretation (one application from many), a lot is left out. But after all, for a typist, the computer is a not the computer with its many functions and components, but a typewriter with more possibilities.

The same computer offers its interpretation as a data-processing device, as a database management tool, and/or as a multimedia console based on the specific application a user is concerned with.

By the same token, the object can be an application: PhotoShop (the metaphor of the darkroom carried over to the digital realm), database, text processing, visualization, e-commerce, among other applications. In such a case, the representamen is the "representation" of the "language" one must command in order to achieve the desired performance. And the interpretation is the performance actually achieved.

3. One sign—three functions

The unity of what is represented, how or through what that representation takes place (medium), and interpretation (the operation desired or actually performed, i.e., I want to process an image, write a letter, buy a car online, etc.) constitutes the sign. The 3 functions of the sign—representation, communication, and signification—can be understood only together (i.e., as an ensemble).

4. Representation

4.1. A caveat!

An unfortunate, simplified model of Peirce’s semiotics (due mainly to Charles Morris) popularized 3 forms of representation: iconic, indexical, symbolic.

Removed from the context of sign definition, these forms are mistakenly called signs. Why do I say mistakenly? The error is evident: You cannot speak of volume, for instance, without acknowledging its 3 dimensions in the measurement. Accordingly, if it is wrong to simply say that the volume of a room is 5 yards (meters for non-USA readers), or even 5 square yards, you cannot characterize a sign only by how it represents the object without relating to the other 2 aspects: the kind of representation and the kind of interpretant.

You have to relate characteristics of the object (WHAT is represented) to HOW these are represented and to their interpretation. Are we looking at an object’s qualities (e.g., softness, color)? Are we looking at its necessary condition? (For example, water is necessarily the combination of hydrogen and oxygen; gravity will cause objects heavier than air to fall to earth.) Are we focusing on its singular (i.e., unique, such as the uniqueness of each individual) nature?

4.2. What is represented?

From among the 3 characteristics of any object or action: uniqueness, formal qualities, and necessary character, one is chosen as representative.

4.3. Types of signs

4.3.1. Sin-sign, exemplified through Signature

A Signature—think of a password or of a fingerprint—is an object of a singular nature. It can be imitated: you look at a signature and try to match the handwriting and the type of pen and ink used. When dealing with a signature, what you probably want is to make possible only one interpretation. For example, if someone wants to cash a check, the banker has to be sure that the signature of the endorser belongs to the appropriate person. If someone wants to access a file, that person should be entitled, and the HCI characteristic of the validation should be designed to make this clear to everyone: "Don’t even try if you are not entitled!"

4.3.2. Quali-sign

A certain quality (e.g., softness, friendliness, pinkness) of an object or an action might stand for the entire object. Let’s say, the Smiley, : ). It suggests that the object or action semiotically identified through this quality support an interaction that is friendly. The design of such an element involves understanding how from many characteristics of a sign one can be selected to stand for the entire object or action it represents.

4.3.3. Legi-sign

It says that something has to take place. If you triggered a shutdown procedure (on a PC, on a UNIX machine, or on a Macintosh), the semiotics of the process has to be simple and direct: No more and no less heavy than the semiotics of a switch (ON/OFF). Still today, this almost trivial operation is only rarely supported by an interface that is both precise and expressive.

5. How do we represent? Indexically, iconically, symbolically.

Semiotic awareness involves understanding the different characteristics of the object or action represented. It also involves understanding the types of representation: indexical, iconic, symbolic. Please take note: These are types of representation, not types of signs!

5.1. Indexical: the marks left by an object (fingerprint, wind arrow)

 

5.2. Iconic: resemblance to an object (the famous garbage can icon)

 

5.3. Symbolic: as in the convention of numbers (Roman, Arabic) standing for quantities

1 2 3 4...
I II III IV...

6. Interpretant process or, the closure of the sign.

We can now combine what is represented and how it is represented.

Example: garbage can.
What: the place where we dispose of what we do not need or desire. What we represent is neither a unique characteristic, nor a singular characteristic.

How: It is most commonly represented iconically. It looks like a garbage can. But it can be presented symbolically, in the action called Erase/Throwaway/Discard.

However, the semiotics of the interaction makes sense only in the context of its specific use.

6.1. Interpreting the interpretant

Semiotic awareness necessitates defining a desired interpretation: the "Aha!" the taking notice of something, the awareness of the consequences of our actions (such as with the ERASE function), and the possibility of individualizing available possibilities (customization as an advanced interpretation).

As opposed to the reactive model of usability tests (addressing "which user?"; "the statistical average?"; "the focus group?"), semiotics suggests the possibility of achieving semiotic adequacy. This is the interplay among various kinds of signs (visual, verbal, tactile, etc.), that is, the sign process conceived with a clear cognitive goal and evaluated in cognitive terms—proactive as opposed to the reactive usability measurements.

6.1.1. Rhema:

a realization (the Aha! effect of something we realize spontaneously), such as the iconic representation of the garbage can. Once in use, things get a bit more complicated. On the Desktop of the Macintosh, one can take the icon of a floppy disk and place it in the garbage can! The result is the Eject function, semiotically inadequate, but which, through use, became part of the Macintosh "language."

6.1.2. Dicent:

the calculator as an "icon" on the iconic interface. You use the representation of buttons as you would work with real buttons. This second-third-fourth level of iconic representation is a description of a description, etc. Semiotically, it is a primitive concept. But once computers without keyboards emerged, it proved to be quite an efficient means for HCI on pocketsize gadgets.

6.1.3. Argument:

the level at which computations result from their own knowledge domain (visualization, simulation, etc.). Computational sciences are not about computers, but in computational form. The HCI of this domain no longer supports applications, but becomes part of the computation. Here HCI is dynamic, growing with the computational inquiry, and becomes part of the result.

7. Semiotic adequacy

Semiotic means of all kinds are integrated in the process we call HCI. In order to evaluate the result of our choices (the kind of means we use) and the effectiveness of the semiosis we designed, we need to "run" the HCI "program," not unlike the way we test various software solutions. Adequacy is a qualitative measurement. Semiotic adequacy is established through the basic semiotic operations already mentioned:

Substitution, i.e., variation of the representamen:

 

Insertion, i.e., an addition of representamina until the object is adequately represented:

 

Omission, i.e., leaving aside or removing sign interpretations that obscure the semiosis:

 

As opposed to the reactive model of measuring user performance, semiotic adequacy is a method of fine tuning the semiotic elements involved in HCI.

8. Aspects of Communication

8.1. As I have already pointed out, there are many levels at which communication aspects become critical to HCI (See: Interface design: A semiotic paradigm). Let us examine the sequence of communication diagrams corresponding to the variety of situations occurring in computation:

  1. the user and the computer in a problem-solving interaction;
  2. the computer and the user in an interaction focused on what is computable and what is not;
  3. communication of results.

8.2. Semantics vs. pragmatics

Communication is a theme of semantics; syntax refers to representation. Pragmatics integrates expression, representation, and communication and results in knowledge.

8.2.1. When Sickenius and her group refer to the specific aspects of how well designed some communicative aspects of a pull-down menu are, she makes us aware of the fact that semantics plays an important role.

8.2.2. However, semiotics becomes relevant only if the perspective is pragmatic: Why do we enter into interaction with a computer? Based on this assessment, we can define the semantics of HCI and design based on a syntax that allows for a clear "language" of interactions.

9. Examples

What follows are, for the time being, only images referring to projects carried out on the basis of semiotic principles at the Computational Design Program. These projects are:

  1. Soft tools for designing interfaces and for determining their semiotic adequacy:

    Lowfidelity Prototyper (Clemens Langowsky)

  2. The Associative Encyclopedia

  3. Webos

  4. Knowledge Community

Within the months to come, I shall explain some of the work here only illustratively represented. Moreover, I shall add a new section on the methodology that our group uses.