The Sentograph: Input Devices and the Communication of Bodily Expression

Roel Vertegaal
Department of Ergonomics
University of Twente
P.O. Box 217
7500 AE Enschede, The Netherlands

R.Vertegaal@wmw.utwente.nl

 Tamas Ungvary
Dept. of Speech Communication & Music Acoustics
Royal Institute of Technology
Box 700 14
100 44 Stockholm, Sweden

ungvary@kacor.kth.se

ABSTRACT: In this paper, we present a qualitative movement analysis of the Sentograph, a 3D isometric input device. We believe the essence of this device is its capability to directly transduce rapidly accelerating and decelerating curves of muscular tension, naturally associated with the expression of emotion, into a form in which it can be communicated musically. The direct correspondence between the controlled parameters and the primary sensory feedback parameters plays an important role in this process.

Introduction

Musical instruments are excellent examples of user interfaces where refined control is more important than ease of use. Musicians often need years of intensive training in order to be able to produce a desired effect. In a broad sense, the desired effect in this particular case can be seen as the successful communication of emotions to the audience, for instance, those abstractly represented in a musical score. In this process, the musical instrument can be seen as a transducer of physical parameters controlled by the musician. Music is produced by the interaction of all components of this man-instrument system, including sound itself.

This process of interaction is bewilderingly complex by nature. Researchers in the field of Human-Computer Interaction often decompose interaction into two main components: input and output. This way, one can study the representation of a parameter independent from its manipulation. Although this approach is certainly viable in circumstances where generic utilization is an important factor [Vertegaal et al.], it is not very appropriate in circumstances where a strong binding between input and output are necessary to produce the desired effect. Consider the process of moulding a clay sculpture: what is input, and what is output? In this process, tactile feedback represents both the amount of muscular force exerted as well as the resulting shape. Seen from this perspective, we present a qualitative movement analysis of the Sentograph , a touch-sensitive input device with three degrees of freedom.

The Sentograph was originally developed as a two-dimensional touch transducer for biocybernetic measurements by Manfred Clynes in the late sixties [Clynes]. Its original construction comprised two sets of strain gauges mounted on a cantilevered arm of square cross-section, with the cantilever placed at right angles to the horizontal (x) and vertical (z) directions of measurement. A finger rest is mounted on top of the free end of the cantilever, which the user can press, thereby slightly bending the cantilever. The frequency range of measurement of the device is from 0-300 Hz, and the output 0 to 5 volts, corresponding to 8 to 1000 grams of weight on top of the finger rest. The resolution of the force measurement is better than 0,05 N, and the deflection of the cantilever is less than .04 mm / N. The original design was improved upon at the University of Uppsala, Sweden, to incorporate the measurement of back and forth finger pressure (y). Also, the shape of the finger rest was made concave in order to obtain a better fit to the shape of the finger tip. In its current configuration, its three analog output signals are converted into MIDI controller messages by a modified Fadermaster with a resolution of 128 discrete steps in all directions [Ungvary & Lunden]. After exerting pressure, the input parameters jump back to their rest value. The x and y dimensions of the device have a rest value of about 64, while the z component has a rest value of about 0.

Clynes used his 2D Sentograph to measure essentic form, that is, the shape expressive actions may have in time during the expression of a particular emotional state, such as love, hate or anger [Clynes]. During these experiments, muscle activity in the fore- and upper arm, shoulder and back was also recorded through measurement of the electrical activity produced in the muscular joints. Subjects were asked to repetitively express a particular emotion by means of finger pressure. Results indicated that in this way, given an emotional state, a corresponding and unique essentic shape might be recorded. Although the absolute position of the pressure components varied between trials and subjects, the shape of the pressure curves remains consistent between trials and even between subjects. Based on the results of these experiments, Clynes constructed a morphological language for the expression of emotional states.

In order to obtain a better insight into the nature of the Sentograph as a musical instrument, we did some informal experimentation. We recorded a number of gestural phrases repeated over time. This in order to get a picture of the underlying form of the phrase and an impression of the accuracy with which these phrases could be repeated over time. The next sections will describe the experimentation and results, after which we will discuss the essence of interacting with this type of transducer. We conclude this paper by presenting a view on the investigation of musical instrument design in which the musician is seen as an integral part of a system of input-output transducers.

Experimentation

During our experimentation, Max was used to record the three directional parameters of pressure from a Sentograph as controller messages with time stamps. Each parameter had a resolution of 128 discrete steps. The minimum of the x parameter corresponded to maximum pressure towards the left, while the maximum of this parameter corresponded to maximum pressure towards the right. The minimum of the y parameter corresponded to maximum pressure away from the body, while the maximum of this parameter corresponded to maximum pressure towards the body. The minimum of the z parameter corresponded to maximum downward pressure, while the maximum of this parameter corresponded to minimal downward pressure. In the first trial, a short circular gesture of about one second was repeated over one minute. The resulting recording was separated into 6 files of approximately 10 seconds length. In the second trial, a more complex movement pattern was recorded in the same way. The resulting files were then processed in Max to produce two different types of movement diagrams. The first type shows the three cross-sections of movement through the three-dimensional space constituted by the three pressure parameters, independent of time (Fig. 1a and 1b). The second type shows the relationship between the value of each pressure parameter and the velocity with which this value changes, again independent of time (Fig 1c and 1d). This way, the underlying form of each pressure component can be visualized, providing a means of comparing the shape of each pressure components between trials in which this same shape is expressed. Results led to an analysis of the overall finger pressure pattern by putting plastocine on top of the finger rest.

Results

Figure 1a shows cross-sections of the movement of the three pressure components from an excerpt of 10 seconds length at the beginning of the first trial (the time series of this excerpt is shown in figure 2). Figure 1b shows an excerpt of movement of equal lenght at the end of the first trial. If we compare these images, we see that although a significant amount of jitter in absolute positioning can be observed, the shapes of the movements are quite similar. Figure 1c shows the dynamic behaviour of each pressure component. It shows that most of the differences between fig. 1a and 1b can be explained by subtle differences in the dynamic behaviour of the individual components, particularly the z component. The x and y components are quite stable in both shape and scale. The z component shows a similar stability only in shape. The dynamical range of the x and z components is much greater than that of the y component. Closer examination of the overall finger pressure pattern in plastocine revealed that when moving on the x, the finger is rolled slightly horizontally. When moving on the y, however, the range in which this can be done without affecting the z parameter is much smaller. This difference is probably due to differences of the pivoting point of the finger tip between the x and y rotation of the finger tip. On the x, this pivoting point lies in the centre of the finger tip, while at the y, it lies towards the end of the finger tip. Figure 1e shows the overall movement pattern during the first trial. Figure 1f shows the overall movement pattern during the second trial, in which a more complex movement was made. If we compare the ranges of the components, we see that the range of y is consistently smaller than that of x and z. On the x and y dimensions, movement is attracted towards the centre, and on the z dimension, movement is attracted towards the minimum. This relates to the rest states of both the finger and device.

Discussion

What is the essence of the Sentograph as a computer music controller? Our most important observation is that although the exact position of the individual pressure parameters at any given moment in time may not be controlled very accurately, the shape of the underlying phrase can be expressed quite accurately and consistently through time. We believe this capacity to accurately express the underlying shape of a musical phrase, without exact repetition of the physical parameters that constitute it, is a very important characteristic that gives the Sentograph distinct qualities as a computer music controller.

From our experience with the Sentograph, it seems rapidly accelerating and decelerating movements on the dimensions of pressure seem the most natural. An important explaining factor for this lies in the self-centering nature of the device, which corresponds to the rest state of the finger tip. For the x and y components, the interaction of the curved finger tip with the concavity of the finger rest might play a role. However, for the z component this is not a satisfactory explanation. A more speculative, physiological explanation is given by Clynes [Clynes]. He suggests that passionate states of emotion have underlying shapes of expression which feature rapidly accelerating and decelerating curves, which often correspond to patterns of muscular tension and relaxation. It is the force exerted by this muscular tension which is transduced by the Sentograph. If we look at the nature of musical expression, we can indeed find some evidence for this notion. Highly curved acceleration and deceleration patterns are indeed predominant parameters for the communication of tension and relaxation in music. We believe this capacity to produce rapidly accelerating and decelerating movements of musical parameters is another important characteristic of the Sentograph as a computer music controller.

The strong, almost synesthetic, sensation of reinforcement experienced while playing the Sentograph might also be partly explained by this direct correspondence between the muscular tension that represents the emotion that needs expressing, and the expression itself. However, the nature of the transducer also plays a predominant role in this process. Particularly for the z component, there is a direct correspondence between the state of the input parameter and the state of the primary sensory feedback parameters: the experienced muscular tension and the experienced pressure on the tip of the finger.

With the Sentograph, the subtleness of expression can be influenced by refined control of the balance between the transfer of weight and the transfer of muscular force onto the device. This balancing of weight and muscular force to control subtleness of expression is not without precedent in musical instruments.

When we look at musical instruments in terms of the transducers that constitute a particular effect, we see that subtle modulation of pitch and loudness is often achieved by means of pressure control with haptic and muscular tension feedback. When examining the violin, for instance, we might identify at least 7 transducers: 4 absolute position transducers for pitch (finger positioning on strings), a position transducer for timbre (position of the bow relative to the bridge), a movement transducer, affecting timbre and whether the instrument sounds or not (bow movement), and a pressure transducer affecting dynamics (bow pressure) [Pressing]. Most of these transducers can be freely combined in order to produce complex effects. However, another transducer might be added to the model. Although we are aware that the process of movement constituting vibrato on a violin is a complex one, which should be studied in greater detail before making any assumptions on the underlying principles, we believe that the subtle changing of the length of the vibrating part of the string in this process is not modelled satisfactorily by a change in absolute position of the finger on the string. During vibrato, the tip of the finger firmly presses the string against the fingerboard. Pressure is also exerted on the finger tip in the direction of the bridge. However, this does not result into an actual displacement of the finger. Our hypothesis is that as a descriptive component which adds to the traditional view of this kind of pitch modulation, the function of the pressure or force exerted on the finger tip in the direction of the bridge could be considered. Clearer examples of subtle expression of emotion by means of pressure exerted with fine motor control might be the subtle raising of pitch of a guitar string by exerting upward finger pressure onto the string, or the use of embouchure in wind instruments.

Future Directions

From our discussion, it seems clear that in order to develop a design rationale for computer music controllers, we need to closely examine different musical instruments in terms of the nature of their transducers and the feedback mechanisms used in their control. This way, we might develop a better understanding of the relation between instrument control and musical function. Some interesting attributes that could be scored for each transducer component are:

Conclusion

We believe the capacity of the Sentograph to accurately express and reproduce the shape of musical phrases, without the exact replication of the physical parameters that constitute them is a key to it qualities as a computer music controller. Furthermore, its capability to directly transduce rapidly accelerating and decelerating curves of muscular tension, naturally associated with the expression of emotion, into a form in which it can be communicated musically, is essential. The direct correspondence between the state of the input parameters and the state of the primary sensory feedback parameters: the experienced muscular tension and the experienced pressure on the tip of the finger plays an important role in the process.

Acknowledgements

We would like to thank Michael Kieslinger and Jeroen Beysens for their assistance.

References

Clynes, M. (1970). "Toward a View of Man." in Clynes & Milsum, ed. Biomedical Engineering Systems. McGraw-Hill.

Pressing, J. (1990). "Cybernetic Issues in Interactive Performance Systems." Computer Music Journal 14(1).

Ungvary, T. and Lunden, P. (1993) "Notes on a prototype of human-computer interface for artistic interaction: Sentograffito" . Proceedings of the Stockholm Music Acoustics Conference, Royal Academy of Music, no. 79.

Vertegaal, R., Eaglestone, B., et al. (1994). "An Evaluation of Input Devices for Use in the ISEE Human-Synthesizer Interface." Proceedings of the ICMC '94, Aarhus, Denmark. International Computer Music Association. Available in Postscript (119 K)