SENSORIUM is a multi-modal arts, research and community project exploring fundamental questions of what it means to have voice, what it means to be fully and essentially human, and how our relationships as parents, as children, and as families deepen our own sense of voice and identity.
The project is centered around an opera, Sensorium Ex – a multi-sensory narrative woven together at the intersections of disability and artificial intelligence. The OPERA serves as the core anchor in an integrated web of collaborations and creative endeavors: COMMUNITY IMPACT, SCIENTIFIC RESEARCH, and a DOCUMENTARY FILM.
At a deeper level, SENSORIUM is also an experiment and a pilot platform for developing new methods of collaboration and impact between and across the arts, sciences, and larger communities.
Components
Artwork by Dana Jaye Cadman
The SENSORIUM project is guided by two central questions and explorations: What is the nature of voice beyond language, and how can artificial intelligence offer new means of expression, particularly to those with non-verbal patterns of communication?
Throughout 2022 - 2023, we will develop community-driven platforms and spaces where each of these diverse project threads can explore these questions through their respective lenses and tools. For example: developing a creative collaboration which uses the crowd-sourcing of diverse voices and stories from across the world and AI technology to build a collective voice and identity for Sophia - the robot narrator of the Sensorium Ex opera. Questions in the libretto around nonverbal communication and voice beyond language will also be explored, and integrated into the building of the operatic work. The Opera and documentary film will premiere in 2024/2025, and we’ll build impact initiatives/campaigns across the world to accompany these premieres.
"The kind of experimentation Prestini has lent to her work will help shape what masterpieces come out of the next 50 years." – LA Times
ACTIVATE AI
ACTIVATE AI is an international research, culture and technology project, with the overall aim of democratizing the development of inclusive voice recognition AI, with a specific focus on strengthening the voices of people with disabilities, speech impairments and atypical speech patterns – both in terms of physical voice and expression, as well as social and democratic inclusion.
AI Systems are based on the data that they are given, and so if a group isn't represented in that data, then that exclusion will be reproduced and amplified by the AI system. This is particularly in the case of voice recognition AI, where people with atypical patterns of speech and vocal expression (e.g. speech impairments caused by stroke, ALS, Parkinson’s Disease and Cerebral Palsy) are excluded from using existing voice recognition technologies.
The ACTIVATE AI project will be driven by a set of guiding questions about how we might activate the current AI-development within voice recognition.
- How might we activate the current AI-development within voice recognition to include people with disabilities around the world in a meaningful way?
- How might the activation of voice recognition AI help expand the possibilities for voice and expression for those with atypical patterns of speech and voice?
- How might we active AI to explore and investigate the relationships between voice, agency, and expression, identity, and memory?
The ACTIVATE AI project will answer these research and technology questions as part of the larger project SENSORIUM, which provides an ideal opportunity for the ACTIVATE AI project, since it is an international platform that explores key questions of voice, disability and AI through multiple lenses – social, technological, and artistic.
Using the larger Sensorium platform, the ACTIVATE AI project will work with four focus-areas:
- Voice Research: Investigate and explore the experience and meaning of voice for people with atypical speech patterns.
- Tech Improvement: Improve current voice recognition AI for atypical patterns of speech by developing new approaches to community-driven data gathering across the world.
- Cross-cultural Access: Ensure that the developed voice recognition AI are accessible and meaningful for people with disability around the world.
- Social Impact: Leverage the power of voice recognition AI to help people with atypical speech patterns to develop their own unique sense of voice and vocal expression.
The ACTIVATE AI project will work with these four focus-areas within these work-packages:
1. Cross-cultural Data Generation: working as part of the SENSORIUM project, the ACTIVATE AI project will leverage the power of arts/culture platforms to design a process that is deeply co-creative and works with community partners around the world (potentially South Africa, Denmark, Greece and USA). The aim is to create a cross-cultural data generation process that improves the voice recognition AI and demonstrates how voice recognition AI can be meaningfully developed and implemented around the world.
The arts/culture platforms from which this process is driven are twofold: 1) It will be part of developing Sophia, an AI-driven robot that will play a role in the Sensorium Ex opera. Sophia will be developed through the cross-cultural data generation process and given an artistic expression on the stage of the opera. 2) An interactive art-installation (working title Phone Booth) will be developed and installed in theatres where the opera is performed and public spaces around the world, so that people with atypical speech patterns can contribute with their specific voices and data.
2. International Implementation: Working with a tech company and the projects’ community partners, we will roll-out enhanced voice-driven AI tools and technologies and co-create impact campaigns around the intersections of voice, expression and AI to deepen community engagement. The installation that will help aggregate voices and will premiere in NYC April 2023.
In addition, the Opera Sensorium Ex will premiere in 2024/2025 and provide a platform for showcasing the voice of Sophia everywhere the opera premieres. These Opera premieres will also create further spaces for community engagement initiatives which allow people to test and explore the tools and technologies developed through the process.
The Sensorium Ex Opera, commissioned by Atlanta Opera and Beth Morrison Projects, will form the core of an integrated web of emergent artistic, scientific and community-driven collaborations which take the key ideas and themes of the Opera from stage to society. Its central themes center around the following questions: What is the nature of voice beyond language, and how can we creatively explore and express forms of non-verbal or non-typical patterns of speech and voice? How might artificial intelligence expand the possibilities for voice and expression - particularly those with non-verbal or non-typical patterns of speech and voice?
Kitsune is a young boy with a disability who is non-verbal - and much of the Opera centers around his relationship with his mother Mem, as they seek to explore their own forms of expression, listening and exchange beyond the conventions of spoken language. The creation of Kit’s musical and sensory voice will be guided by in-depth interviews, focus groups and creative workshops with this international co-creation community.
Some of the core questions guiding Kitsune’s development include:
- If a foundational aspect to being human is to be able to communicate, express oneself, and be heard by others - how does a character like Kitsune, who is non-verbal, become understood? How do we facilitate communication so as many people as possible can understand each other - regardless of their forms of expression, (non-verbal/verbal/etc.)?
- In the opera, how can we show this process of new understandings developing on stage — in choreography, music, staging, voice, narrative, characters?
Sophia is the robot narrator of the story, an amalgamation of the memories and lived experiences of many of the opera’s main characters. Her musical voice will be developed through a co-creative process which seeks to build new, inclusive AI datasets around non-normative patterns of voice and speech. This AI will be leveraged both to create Sophia’s voice on-stage, as well as be leveraged to make AI tools and technologies related to voice more accessible and inclusive for all. The co-creation process will be scaffolded around a series of 3 workshops and convenings throughout the course of 2022-2023.
Kitsune’s role will be co-created with the person cast to play the role and informed by the team, and our choreographer in terms of movement. -In a culminating scene of the opera called “the escape”- traditional notation will be replaced by medieval chant notation-neumes and a mixture of sound and language and direction and color. Like Benedectine monks, the ensemble will improvise on a series of calls and responses within which Mem and Kitsune are held. Improvisation can mean that there is an embodied and shared sense of space and purpose. In this instance, like the monks, there is a devotional and safe and healing space created for this decision where Kitsune’s agency is made clear. A “Community commission” will be given to create a sense of place in each country the opera goes to, and will be performed by the choir in the opera.
Sophia is the AI in the opera, and her voice will be developed in conjunction with NYU with the brilliant R. Luke Dubois and Kristian Martiny. She is an amalgamation of all characters central to the escape: Kitsune, Mycelia and Mem. By the very end of our process, she is an aggregation of all the people who have participated in our workshops: she grows more complex as she learns and absorbs the data sets. The AI data set amalgamation will aggregate in the international locations we develop the opera: the US, Denmark, and potentially South Africa and Greece to create Sophia’s voice, and to help create an eventual app for assistive technology for varied voices and expressions.
SENSORIUM is a hybrid documentary about finding voice beyond language, and through our relationships as parents, as children, and as families. The sci-fi, fantasy universe of a new opera, Sensorium Ex, serves as an imaginative stage to bring together the worlds and personal narratives of an international ensemble of characters - as they each explore their own relationships with voice, disability, identity and family.
The film is co-directed by Christina Moltke Martiny and Daniel Oxenhandler and is being produced by two-time Oscar nominated producer, Signe Byrge of Final Cut For Real.
We believe the questions that the film asks are of paramount importance in this current moment, as communities the world over seek to deepen their sense of voice - socially, culturally and democratically.
SYNOPSIS
Brenda is a mother and a poet, crafting worlds through her words. Her son, Cal - full of life and expression all his own - was born with Cerebral Palsy and is non-verbal. She is now reimagining her relationships with voice, language and parenthood through the creation of a new opera, Sensorium Ex - which she is developing with composer, Paola Prestini. Across the ocean in Denmark, Jacob - a disability activist and artist - has recently become a father. He was born with cerebral palsy and didn’t speak until the age of eight, and lives with a speech impediment. As he raises his son Magnus, his relationship to voice takes on a whole new significance, as a father.
The narrative is grounded in the daily lives of Brenda and Jacob, but tactfully opens up into the realm of the imaginary - working with the fictional universe of the opera Sensorium Ex as a creative platform for Brenda and Jacob to more deeply embody and experiment with their own relationships to voice, disability and parenthood.
THE SENSORIUM TEAM
Paola Prestini | Composer
Through an illustrious career being equal parts creator and connector, composer Paola Prestini is known both for her “otherworldly...outright gorgeous” music (The New York Times), as well as the “visionary-in-chief” (Time Out New York) and Co-Founder/Artistic Director of the non-profit music organization National Sawdust, based in Brooklyn, NY. She has been named an “Innovator” on the list of Top 30 Professionals of the Year by Musical America, the country’s oldest classical music magazine; she is on Brooklyn Magazine’s latest list of “influencers of Brooklyn culture...in perpetuity” alongside such household names as Chuck Schumer and Spike Lee; she is one of the “Top 35 Female Composers in Classical Music” (The Washington Post); and on the “Top 100 Composers in the World” list by NPR. Prestini’s music and works have been commissioned by and performed at the Brooklyn Academy of Music, The Brooklyn Youth Chorus, the Barbican Centre, The Cannes Film Festival, Carnegie Hall, the Kennedy Center, The Los Angeles Philharmonic, The Los Angeles Opera, The New York Philharmonic, Roomful of Teeth, the Choir of Trinity Wall Street, and the Young People’s Chorus, among others.
Brenda Shaughnessy | Poet, Librettist
Brenda Shaughnessy is the author of four poetry collections, most recently The Octopus Museum, So Much Synth and Our Andromeda, which was a New York Times’ 100 Notable Book, a finalist for the Grifn International Prize and the Kingsley Tufts Prize. Her second book, Human Dark with Sugar, was a finalist for the National Book Critics Circle Award and was the recipient of the James Laughlin Prize of the Academy of American Poets. She received a 2018 Literature Award from the American Academy of Arts and Letters and a 2013 Guggenheim Foundation Fellowship, was a Bunting Fellow of the Radclife Institute for Advanced Study at Harvard University and a Japan/U.S. Friendship Commission Fellow of the NEA. Her poems have appeared in Best American Poetry, Harpers, The Nation, The New Yorker, Paris Review, Poetry, The Yale Review, and anthologized in The Penguin Book of 20th Century Poetry and O Magazine’s Little Book of...series. She has given readings and talks on poetry at TedX Harvard, the U.S. Library of Congress, the Chicago Humanities Festival, and many universities, libraries, festivals, conferences, and high schools. She is Associate Professor of English and Creative Writing at Rutgers University- Newark and lives in Verona, New Jersey.
Jerron Herman | Choreographer
Jerron Herman is an interdisciplinary artist creating through dance, text, and visual storytelling. He's based in New York City. Jerron was born and grew up in the San Francisco Bay Area where he began his career pursuing performance and playwriting. In 2009 he moved to New York City to study Dramatic Writing at the Tisch School of the Arts at NYU. Continuing his dream of crafting stories to be performed, he then studied Media, Culture and the Arts, with an emphasis in Art History and Playwriting at The King's College where he graduated in 2013. While in school he was "discovered" by a choreographer who led him to audition for Heidi Latsky, quickly becoming a key member of her company, Heidi Latsky Dance. Jerron has performed at venues like Lincoln Center and The Whitney Museum of Art, resulting in the New York Times calling him, “the inexhaustible Mr. Herman.” As a strong advocate for disabled athletes and performers, he has been a featured model for both Tommy Hilfiger and Nike. His performances have begun to shed light on an often overlooked niche of performance. He currently sits on the Board of Trustees at Dance/USA . As a writer, Jerron was a finalist for the inaugural Lark Play Development Lab/Apothetae Playwriting Fellowship, 2017. He is a 2020 Disability Futures Fellow, a joint initiative of the Andrew W. Mellon and Ford Foundation, respectively.
Kristian Martiny | Head of Research, Enactlab
Kristian is the Co-Founder and Head of Research & Action of the EnactLab - a lab for change working to break down prejudices and transform stigmatizing social narratives through the intersection of scientific research, community engagement and the arts. He works in the space where cognitive science - which includes philosophy, psychology and neuroscience - meets technology, practice and art. He conducts research by openly applying this combined approach in many different contexts - including developing rehabilitation strategies and technologies, documentary films, art installations, theatre plays, and collaborative arts, science and community projects. Through his work, he explores ways in which cognitive science can be applied and communicated through different media, as well as in cross-cutting collaborations, in order to improve the spectrum of possibilities for the communities with whom he collaborates. Kristian holds a Ph.D. in Philosophy and Neuroscience from the Center for Subjectivity Research, University of Copenhagen, and the Elsass Institute.
Jacob Nossell | Disability Activist, Journalist
Jacob works at the crossover where science, culture and society meet, in order to drive positive change and action around complex social challenges - such as disability and mental health. He was the Co-founder and Head of Communication for the Enactlab - a lab for change based in Copenhagen - where he worked to ensure that the first-person lived experiences of individuals and communities are deeply integrated into every aspect of the creative and research process; he also leds all media and communications efforts for the organization. He has extensive experience creating award-winning arts and media projects, that provide a more deeply embodied and nuanced understanding of disability - including journalism; documentary films – The Red Chapel (Sundance Award Winner - 2009) and Natural Disorder (2015); theater plays – Human Liquidation (2016); podcasts – My Damn Voice (2018); comedy, talks, and campaigns – It’s not a handicap (2013).
Christina Martiny Moltke | Documentary Filmmaker & Cinematographer
Christina is a documentary filmmaker and cinematographer interested in the development of projects built on co-creation: uniquely collaborative productions which bring the lived experience and wisdom of the film’s protagonists into all aspects of the creative process. This approach sees film as a deep process - in which all collaborators contribute to a collective vision, and build a space for collective inquiry, exploration and evolution. She is a graduate of the Media Studies Masters program at Aarhus University in Denmark and has a practically oriented art education from Slade School of Fine Art, University College London. She has extensive experience in documentary film production, working as a cinematographer and line producer on films and documentary series, primarily for Danish television stations DR and TV2.
Daniel Oxenhandler | Filmmaker/Cultural Producer | Head of Culture, Enactlab
Daniel Oxenhandler is a filmmaker, and interdisciplinary cultural producer. His work focuses on developing interdisciplinary arts projects which bring together unique intersections of film and media, arts and culture, academic research, and community, in order to co-create new modes and methods of exploring complex social issues across cultures and geographies. As a filmmaker, he has directed + produced a variety of independent film and video projects, ranging from feature-length documentaries and short narrative films, to short music films and travelogues. His most recent documentary film, The Open Window, saw its premiere at the CPH:DOX Film Festival 2019. As an interdisciplinary producer, he has played a leading role in designing, curating and organizing international arts, culture, and innovation festivals and projects (theater, music, etc) in India, Mexico, Denmark, the US, and across the world. He is currently based in Copenhagen, Denmark and has previously lived, learned and worked around the globe - in Brazil, Mexico, India, Spain and the US.
R. Luke Dubois | Integrated Design & Media
R. Luke DuBois is a composer, artist, and performer who explores the temporal, verbal, and visual structures of cultural and personal ephemera. He holds a doctorate in music composition from Columbia University, and has lectured and taught worldwide on interactive sound and video performance. An active visual and musical collaborator, DuBois is the co-author of Jitter, a software suite for the real-time manipulation of matrix data developed by San Francisco-based software company Cycling'74.
He is the co-chair of the department of Technology, Culture, and Society and research director of the programs in Integrated Design & Media at the NYU Tandon School of Engineering, and a founding co-director of the NYU Ability Project, where his research focuses on integrative systems for equity, ranging from open source software projects for signal processing to tele-present communication systems for motion capture to citizen science for noise pollution to design for disability. He is on the Board of Directors of the ISSUE Project Room and Eyebeam.
ENACTLAB | Enact Lab Impact Producers
Enactlab S/I was founded in 2018 by Kristian Moltke Martiny and Jacob Nossell as a global lab for change –based in Copenhagen, Denmark. Enactlab is a team of researchers, media and communication specialists, community leaders, artists and global collaborators. Enactlab strives for new ways of enacting and working with change – new ways of solving societal problems and challenges. Solutions where citizens, scientists, artists and professionals – people and organizations – join forces, openly engage and wholeheartedly collaborate vis-a-vis the Enact Model. The Enact Model is a scientifically proven model which combines different modes of knowledge and sense-making – from the lived experience of individuals, to the rigor of scientific research – in order to enact change and develop solutions to complex problems and challenges.
Beth Morrison Projects | Commissioner and Producer, Sensorium EX
Beth Morrison Projects (BMP) is one of the foremost creators and producers of new opera-theatre and music theatre, with a fierce commitment to leading the industry into the future, cultivating a new generation of talent, and telling the stories of our time Founded by “contemporary opera mastermind” (LA Times) Beth Morrison, who was honored as one of Musical America’s Artists of the Year/Agents of Change in 2020, BMP has grown into “a driving force behind America’s thriving opera scene” (Financial Times), with Opera News declaring that the company, “more than any other… has helped propel the art form into the twenty-first century.” Operating across the US and internationally, with offices in Brooklyn and Los Angeles, BMP’s unique model offers living composers the support, guidance, and freedom to experiment, allowing them to create singularly innovative and impactful projects. Since forming in 2006, the company has commissioned, developed, produced and toured over 50 works in 14 countries around the world, including the Pulitzer Prize-winning chamber operas Angel’s Bone and p r i s m.
Atlanta Opera | Commissioner and Presenter, Sensorium Ex
The Atlanta Opera’s mission is to build the major international opera company Atlanta deserves, with a vision to reimagine opera. Founded in 1979, The Atlanta Opera celebrates its 40th anniversary in the 2019-20 season. The Opera works with world-renowned singers, conductors, directors, and designers who seek to enhance the art form. Under the leadership of internationally recognized stage director and Carl W. Knobloch, Jr. General & Artistic Director Tomer Zvulun, The Atlanta Opera expanded from three to four mainstage productions at Cobb Energy Centre and launched the acclaimed Discoveries series. In recent years, the company was recognized by The Atlanta Journal-Constitution as part of its “Best of 2015” awards; it was nominated for an International Opera Award in 2016; and it won ArtsATL’s 2019 Luminary Award for Community Engagement, recognizing its successful Veterans Program in partnership with The Home Depot Foundation. In addition, The Opera was featured in a 2018 Harvard Business School case study about successful organizational growth, and Zvulun presented a TEDx Talk at Emory University entitled “The Ambidextrous Opera Company, or Opera in the Age of iPhones.”
VisionIntoArt | Commissioner
VisionIntoArt is a non-profit new music & interdisciplinary arts production company in New York City. Since Paola Prestini co-founded the company in 1999 at the Juilliard School, VIA has created and performed over twenty five original works as well as many album releases through their self titled record label. VIA was started by a team of artists at Juilliard in 1999 that included composers Paola Prestini, Nora Kroll-Rosenbaum and Nico Muhly, pianist Stephen Gosling, and conductor Paul Haas. In 2006, dreams parted ways, and the last VIA productions and its touring years were directed by Paola Prestini, with a brief hiatus while Prestini co-founded the arts incubator and micro-institution, National Sawdust.
VisionIntoArt, “always intriguing and frequently beguiling”, is a multimedia production company that “facilitates flamboyant, confounding and enticing collaborations” (New York Times). VIA creates and commissions works that involve various disciplines, presented around the world for the general audience, and forged from the most exciting emerging and established artists living today as well as interdisciplinary experts including scientists and conservationists. VIA works often bridge impact, community building and scientific inquiry, with the belief that collaboration sustains artistic innovation and promotes a healthier society.
VIA’s works range from the Hubble Cantata-the largest communal VR operatic event, to multiplatform works like The Colorado, now viewable on PBS. VIA productions have been seen at Lincoln Center, the Barbican Centre, HIFA in Zimbabwe, The Brooklyn Academy of Music, and the Kennedy Center, along with residencies at MASS MoCA and The Park Avenue Armory. They have also toured to colleges and universities in the US, and to international festivals such as Apertif in Concerto at Teatro Manzoni, Etna Fest in Italy, and BEMUS in Belgrade, Serbia.
Ras Dia | Producer, VisionIntoArt
Ras Dia is a Brooklyn-born, Harlem-bred creative working at the intersection of inspiration and empowerment in the arts. His work has been described as “stirring” (Washington Post), “bracing, compelling, and heartbreaking” (Musical America), and “grippingly produced” (The Boston Globe). Recent projects include Heartbeat Opera’s BREATHING FREE: a visual album, the Frederick R. Koch Foundation’s Townhouse Series, San Francisco Symphony’s MTT25: An American Icon, San Francisco Opera’s In Song, and the inaugural season of Little Island, a public park and arts organization developed by the Diller-von Furstenberg Family Foundation.
Ras previously served as the Assistant Producer of the Metropolitan Opera’s Peabody- and Emmy award-winning Live in HD series, and as the Managing Director of the New York City Master Chorale, in addition to marketing, development, production, and administrative roles with the National Children’s Chorus, Sadler’s Wells Theatre, Carnegie Hall, The New School, where he supported programs for immigrant, refugee, and survivor communities across New York City, and National Sawdust, where he produced the Artists-in-Residence program, and co-created SAUCE, a series of artist sessions.
He is a 2021-22 Artist Scholar at the Manhattan School of Music (MSM), and a graduate of the Guildhall School of Music & Drama, Purchase College (SUNY), as well as the Boy’s Choir of Harlem, and has appeared as a guest speaker for the National YoungArts Foundation, Amherst College, the Black Artists Fund, and MSM, in addition to serving as a grant panelist for OPERA America.
