Part I: in dialogue with Mark Coniglio

Part II: comments from Jean-Baptiste Barrière, Jem Finer, Armando Menicacci, Giorgio Olivero, Steina Vasulka

[short biographies and reference URLs are at the end]

Edited by Scott deLahunta

15 September 2002

INTRODUCTION:

The field of real time multi-media software tools created by artists for artists emerged from the hardware and software developments of the early to mid 1980s. Fifteen plus years later hundreds of individuals and groups are working consistently with these tools in their own creative practice, often to some degree customising them; creating tools within tools and sharing these with other practitioners. However, there are a handful of key artist programmers who have devoted themselves to developing major software contributions to this field; amongst them Tom Demeyer, David Rokeby, Netochka Nezvanova and Miller Puckette. This article focuses on one of the latest of these contributions, Isadora, which is programmed by Mark Coniglio. It is intended to provide some insight for those who may be relatively new to this area of software tool development for artists by artists; and in particular the development of multi-media tools to be used in the context of live performance practices.

BACKGROUND/ BIOGRAPHY:

Mark Coniglio is an artist whose work crosses the disciplines of music, dance, theater and interactive media. Dubbed an "interactive performance pioneer" by the New York Times, his work has been performed nationally and internationally primarily with Troika Ranch, a New York City based performance company committed to creating multidisciplinary works of which he is Co-director with choreographer Dawn Stoppiello. A native of Nebraska, Mark studied at the California Institute of the Arts with electronic music pioneer Morton Subotnick and received his degree in music composition in 1989. He was on the staff of the Center for Experiments in Art, Information and Technology at CalArts teaching courses in Interactive Music from 1990 to 1994. In 1993, they started Troika Ranch while Dawn was dancing for Bella Lewitzky, and in 1994 they moved with the company to New York City. Besides the rehearsals, performances, symposia appearances and residencies that make up the bulk of their creative contribution to the field, Troika Ranch regularly conducts their popular Live Interactive (Live-I) workshop which gives participants the opportunity to explore the use of interactive computer technology in the creation of live performance artworks. Participants also have a chance to learn to use the custom hardware and software Mark has created.

While the building of new interactive instruments has a robust tradition within the electronic music field dating to the early 20th century, Mark is one of a handful of artists who have specialised in creating these instruments to monitor the movements of dancers and use this information to control other media events in real time. The MidiDancer, a wearable device Mark first built in 1989, measures the angular change at several joints on the dancer's body. This information is then sent over a wireless link to a computer where the data (input) is used to control a variety of media events (output), e.g. sonic, video, lighting and/ or robotic. A software programmer for as long as he has been a composer, Mark has written two programs to map this data flow; the first was Interactor and the second is Isadora. Interactor was made available for others to use, however it required time to learn (made easier with some knowledge of software like Max, a graphical programming environment for music and multimedia). Isadora (named after the renowned early 20th century modern dance pioneer) has been designed to be used by a non-programmer after only the briefest of introductions; placing the control of the instrument in the hands of the dancers themselves. Another key development is that while Interactor was focussed on handling MIDI data (a communications protocol that allows electronic musical instruments to interact with each other); Isadora, while it can also work with MIDI, is designed primarily for the manipulation of video.

PART I: DIALOGUE:

Scott deLahunta: You have been programming actually longer than you have been composing. So do you consider your programming a part of your artistic practice?

Mark Coniglio: I definitely do, but it does feel a bit secondary to my composition and/or media art making in that I see it as more of a "support" activity. The software I program allows the creation of the artwork, whether sonic and/ or visual in some combination with live performance, that I envision. I always seem to resort to musical metaphors for things like this. The artwork is like a musical composition; the software tools are like the instruments in the orchestra. If you can develop a more advanced instrument, you can create more advanced music. The French Horn is a good example; early versions required removing a piece of tubing from the instrument and replacing it with another piece of a different length to change key. About 1815 the modern version of the instrument with valves was invented. This technological innovation meant that you could now include the horn in passages where there were rapid changes of key. This was immediately taken advantage of by composers, notably by Beethoven in his symphonies. The difference between Beethoven and me is that he was not driving the innovation directly. I am both artist and instrument inventor. My artistic ideas drive the development of software and hardware I need to realise them; while simultaneously the programming I do expands my world of expressive possibilities.

Scott: I understand the analogy with the French Horn, but software 'instruments' are comprised of another type of material altogether. Instead of metals being forged into shapes; one could speak of algorithms, formal abstractions and language. As another way to relate programming and art making, I am curious about the creative process of writing code compared to working with other materials. It seems that most programming is driven by the assumption that the software will have a purpose (more like the notion of the instrument). Once that purpose is understood and defined so there is a goal, only then does the programmer get down to the work of coding. If this is true then programming is a very different creative process than, say, how the choreographer might work with movement material.

Mark: When writing software it is useful to have a clear goal in mind, and this may not be true with other types of art making, as you suggest. Having a goal that is too specific can be detrimental to the process of making a dance for instance or composing music. I don't think I understood that early on, but over the last 3 or 4 years I found myself exploring much more organic, open-ended approaches to art making. Being involved in dance in particular, which relies on improvisation as a primary source of generating material, has profoundly influenced my way of working. Now, my approach is a much more, "try everything and follow your nose." By this I mean, try not to preconceive as much, make lots of stuff and follow through with the material that seems interesting and let the material begin to tell you what it is about. Now, it's pretty difficult to program that way, a kind of "goalless" coding. The architecture of the microprocessor, from which all programming languages derive, is actually antithetical to such behaviour. However, this "follow your nose" approach has definitely influenced the way I program. I still have a goal, but I don't often plan out the algorithms. I simply write towards my goal, improvising my approach to solving the problem.

Scott: There is quite a big discussion about 'software as art' these days in Europe and I'm sure it's going on in North America as well. Besides its utility, its usefulness in helping to support your art works, do you consider Isadora a work of art by itself?

Mark: In a word, I don't - not in and of itself anyway. It's bit like asking if the telephone system is a work of art. Does the creation of the technology to support that system display an incredibly high level of inventive thought and uber-craftsmanship? Definitely. I have the utmost respect for the creative people that designed and implemented such a robust, complicated, and reliable system. But that particular technology is dormant until you pick up the receiver, ring your lover, tell her that you no longer desire to see her, and a heated conversation ensues.

Scott: It seems to me that the telephone system is the collective result, over time, of a multiplicity of individuals and institutions labouring together, and it's difficult to locate individual contributions in that, or at least I don't use the telephone system and sense that I am in contact with one of its creators. But when someone opens up Isadora and begins to build a patch that will map an arm motion to the speed of a video clip, do they not encounter your presence directly, as the primary creator, in the look and feel of the interface?

Mark: Well, I guess the only way that I WOULD consider Isadora to be an artwork is the personal stamp that I have on its design and functionality. To take that a bit further, in a broad sense I could say that I collaborate with each Isadora user as they use the program. Because I can't totally erase myself from the software I create, they have to embrace some of my predilections to make use of the program; which is what happens whenever you choose to collaborate with another artist. It's just a question of how apparent the influence of the software's creator is. That's where software designers and artists who make software may differ, I think. A typical software designer does everything he or she can to filter out their personality and create something that is useful in a general way.

Scott: Maybe we could say that the extent software like Isadora is an artwork is dependent on the degree to which the maker tries to remove him or herself from it? I suppose Isadora then is more of an artwork than say, Photoshop, but is maybe less of an artwork than something like Auto-Illustrator by Adrian Ward (which won the Software Art prize at the Berlin 2001 Transmediale Festival). Auto-Illustrator looks like a vector graphics program, but it doesn't act like one. It misbehaves frequently because it has seemingly autonomous behaviours built into it that take over for you. Adrian creates these behaviours using certain algorithms when he does the coding, so as such his authorship is revealed every time the software does something you did not expect it to which is frequently.

But let's talk a little bit about the background of Isadora as a graphic programming environment for real time manipulation of media. You made something similar with Interactor first didn't you?

Mark: Here's a bit of history. In 1986 my soon-to-be mentor and Interactor collaborator Mort Subotnick had just come from a residency at MIT where he was using a program called Hookup created by a student there named David Levitt. Hookup was the first program I know of that used the "patch-cord" metaphor, i.e., modules that manipulate data are linked by virtual wires, the connection of which is determined by the user. For those in the world of early analog, patch-cord programmed synthesizers, this was a familiar interface. Mort was using David's program to do tempo following of MIDI instruments -- this allowed him to lock hardware MIDI sequences to the tempo of the live performers. I was a composition student at CalArts at the time, and word had gotten around that I was a good programmer. So Mort contacted me to see if I could hardcode some of the ideas he had implemented in Hookup on a Mac, so that he could use them in his next performance. That program (used in Mort's 1987 multimedia work "Hungers") would eventually become Interactor. Mort designed the functionality of the early versions, but I became more influential in the design as time went on.

Scott: I guess the hardware and software development of the early to mid 1980s where we saw the advent of the personal computer and more importantly the graphical user interface (marked by Apple's introduction of the Macintosh to the consumer market in 1984) created a context out of which the 'computer' could emerge as a creative instrument or tool. The electronic music field was already well advanced in analysing and exploring the formal and physical properties of music as part of compositional and performing practice, so moving to programming real time graphic interfaces for this seems like a rather natural progression.

Mark: Yes, that's true and importantly a kind of creative intuition was creeping back in through the development of these new visual interface possibilities for software. Part of the thing I reacted to in Hookup was the way you could easily drop modules into the program and try things; a lot like you could do with the patch-cord synthesizers. I may not have realized it explicitly then, but this ability to program improvisationally allowed for that kind of artful playfulness that is so important. So I set out to make a similar user interface for Interactor. The creation of Isadora was a natural outgrowth of Interactor. In 1996 Troika Ranch had a two-week residency at STEIM, where I first saw Tom Demeyer's real-time video processing program Image/ine. I first started using Image/ine in concert with Interactor, because Image/ine didn't allow the kind of complicated interactive decision making that I was used to having in Interactor. So, Interactor would process the MIDI data from my interactive sensors, and then tell Image/ine what to do. By 1998 I was using Image/ine in a major way in my performances with Troika Ranch.

But, while I loved what Image/ine did, I wasn't fond of its table-based interface. And there were problems with crashing during performance, which is unacceptable when there is an audience. Furthermore, we were teaching our Live-I (Live-Interactive) workshops using Interactor and Image/ine, and the students (especially the ones with weak computer backgrounds) found learning both programs, and figuring out how to have them communicate with each other daunting at best. I wanted Isadora to take the qualities that made Interactor and Image/ine great, and put them together in one package that was easy to learn but still offered enough depth to satisfy "power" users. And, I wanted it to be more affordable to members of my community, which I consider to be choreographers, because of my involvement with Troika Ranch.

Scott: How does Isadora compare to Max, which is probably the most successful and widely used graphic programming environment for controlling and mapping data flow?

Mark: Isadora and Max both inherit the modules linked by the patch-cord metaphor from Hookup. But unlike Max, each Isadora module shows the parameter names and current values for all of its inputs and outputs, and many modules give real-time graphic feedback about their operation. This is important from the perspective of helping new users understand what's going on right away. But perhaps the biggest difference is that Max is a very powerful, open-ended programming language in which you could solve any number of problems. Isadora isn't that. It is a lot like Interactor in that each module is essentially a macro that accomplishes some specific function. This approach helps people who are just beginning to do this kind of work, as it means that useful functionality is already embodied for you and it's very easy to start doing things and getting interesting results quickly (like with Image/ine). Max allows the most flexibility, but may be somewhat more difficult to program because more things have to be built up from scratch. Isadora offers somewhat less flexibility, but is still open-ended enough for the user to imprint his or her aesthetic on the result.

Scott: You have told me that your most important Isadora user is yourself. How do you use Isadora in Troika Ranch's performance work and in particular in connection with the MidiDancer which is the sensor system you built to get data input from a moving body.

Mark: I first created MidiDancer in 1989 while I was still a student at CalArts. While it is now much smaller and more reliable, the basic functionality has not changed much since then. Basically, it is a set of up to eight sensors that measure the flexion of joints on a dancer's body. Thirty times a second this information is sent over a wireless, radio link and a receiver up to 150' away decodes the information, reporting the angle of each joint in the form of a MIDI (Musical Instrument Digital Interface) message. Any computer with a MIDI interface can accept and process these messages as desired.

The problem with MidiDancer is that, to really play it, and for the audience to see that the dancers are playing, you need to move like a musician. What I mean is that the movement of the dancer needs to be in service of the sound or image that they are generating or controlling. We have worked hard to find ways to make this work choreographically, but it is quite difficult to do. My basic instinct in putting the sensors on the "gross" joints (elbows, knees, hips, wrists) was correct, in that these angular changes can be clearly seen by the audience. But, I have really been seeing lately that this is not the gestalt that we perceive when we watch a dancer move. We really see energy -- that's a bit vague, but it's the best word I can think of to describe it. We're not looking at the individual angles of the joints, but the way that the dancer moves through space and the overall articulation of the movement.

Scott: Well, electronic musicians have been building sophisticated playable interfaces for a long time, but these tend towards either the hyper-instrument (extending an existing musical instruments capabilities) or a few unique hand orientated interfaces. But, I've always thought that one would need to think quite differently to develop an appropriate system for a trained mover or dancer. I think more research is needed with various sensor input devices and maybe not always towards the aim of live stage performance, but maybe just experimenting much longer with what might emerge from the kind of feedback conditions for the senses interactive systems can generate.

Mark: That's why we'll be using a residency we have next year to make some changes to the MidiDancer. I want to start working with accelerometers in addition to the flexion sensors. The act of turning, or stopping suddenly, or shaking the whole body, now becomes something that can be measured. My instinct is that using this information to manipulate the media will be a more natural linkage between what the audience sees in the dancer and the resulting sonic or visual manipulation. I can then use the position of the limbs to allow the performer to enhance their level of control -- but I suspect that being able to sense the impulse of movement may become the primary source of manipulation. I think that, not being a dancer or choreographer myself, I have been slow to let go of the notion of being a musician. In fact, I have often described the MidiDancer as allowing a dancer to be "both musician and dancer." I now think that is incorrect. I need a device that allows a dancer to be a dancer, with the media taking its cue from what it sees the dancer doing.

Scott: Isadora is just about finished with its public beta testing phase during which I know you have been working with several artists who have been trying out the software for different projects and giving you feedback and suggestions for additional functions, etc. I have invited some input from some of these individuals [Part II], but first I just wanted to ask you to say something about how long you have been working on Isadora and your decision to sell the software instead of using an Open Source approach (in which software code is released for free into a collaborative development environment).

Mark: As I think I've indicated Isadora has grown out of a need Troika Ranch had for a reliable relatively easy to use but also sophisticated software for both workshops as well as performance. The end result is that I have taken two plus years to develop Isadora in my spare time so it has grown quite organically. In regard to deciding to sell Isadora, I don't have much interest in starting a real business, so I am feeling out my progress quite slowly. But in the U.S., arts funding is very difficult to come by, and discovering ways to supplement what we can get from the government or foundations is essential. I have always had to hold a day job, so I wanted to see if I could make a tool that would be: 1) useful to me; 2) useful to others and; 3) perhaps generate enough income to help me spend more of my time making my artwork. Obviously, an Open Source model would not generate any income, and thus wouldn't help to support my artistic pursuits. As I have mentioned, it is important to me that the Isadora is affordable to those in the dance community, so I figure, for what the program does (and will do as it grows) the single license fee is a reasonable amount. I have yet to make a final determination about site licenses for schools etc, but they will definitely get a break if they purchase a 5-seat license for example. I have no idea at this point if this whole plan will work, and if the burden of supporting a growing user base will actually be more work than the day job, but I am hopeful.

Scott: Thanks Mark. Now, what I would like to do is to invite some comments from some of the artists who have been working with Isadora and are familiar with the history and evolution of artist software tools to reflect on some of our dialogue and to talk about how it is they use or may use Isadora in their work.

Go to Part II, back to main page or to the top of this page