Auditory Analytics is the scientific method augmented by data sonification and analytical listening. For example:
Preprints
“Auditory Analytics for pattern discovery in protein folding dynamics,” accepted by the Journal of Chemometrics | CONTRIBUTORS: Carla Scaletti*; Kurt J. Hebel; Martin Gruebele*
Pre-peer reviewed version
Publications
“Water-mediated hydrogen bonds and local side-chain interactions in the cooperative collapse and expansion of PNIPAM oligomers,”(DOI: 10.1073/pnas.2523755123) Proceedings of the National Academy of Sciences February 3, 2026 | vol. 123 | no. 5 | by Wanlin Chen, Martin Gruebele, Martina Havenith, Kurt J. Hebel, Carla Scaletti. Supporting Information (includes links to sonification/animations on YouTube)
Poly(N-isopropylacrylamide) (PNIPAM), a thermoresponsive homopolymer, is a well-established model for investigating coil-to-globule transitions. Here, we combine molecular dynamics (MD) simulations, data sonification, and graph-theory analysis to elucidate the roles of intramolecular and PNIPAM–solvent hydrogen-bond (H-bond) patterns in PNIPAM compaction dynamics. Our analysis separates the driving forces for compaction into two contributions: the entropic gain from the loss of hydration water around hydrophobic patches, and the enthalpic stabilization from water H-bonded to PNIPAM. We find that the role of the solvent in polymer compaction is more active and complex than has been previously assumed. Our observations indicate that direct, intra-chain hydrogen bonds between amide groups (N-H···O=C) are not the primary stabilizing force. Instead, the collapsed globule contains an N..HN network of local side chain interactions and is stabilized by a dynamic network of persistent, long-distance water bridges, where single water molecules form hydrogen bonds with multiple parts of the polymer chain.
“Hydrogen bonding heterogeneity correlates with protein folding transition state passage time as revealed by data sonification,” Proceedings of the National Academy of Sciences 2024-05-28 | Journal article DOI: 10.1073/pnas.2319094121 | CONTRIBUTORS: Carla Scaletti*; Premila P. Samuel Russell; Kurt J. Hebel; Meredith M. Rickard; Mayank Boob; Franz Danksagmüller; Stephen A. Taylor; Taras V. Pogorelov; Martin Gruebele*
Protein–protein and protein–water hydrogen bonding interactions play essential roles in the way a protein passes through the transition state during folding or unfolding, but the large number of these interactions in molecular dynamics (MD) simulations makes them difficult to analyze. Here, we introduce a state space representation and associated “rarity” measure to identify and quantify transition state passage (transit) events. Applying this representation to a long MD simulation trajectory that captured multiple folding and unfolding events of the GTT WW domain, a small protein often used as a model for the folding process, we identified three transition categories: Highway (faster), Meander (slower), and Ambiguous (intermediate). We developed data sonification and visualization tools to analyze hydrogen bond dynamics before, during, and after these transition events. By means of these tools, we were able to identify characteristic hydrogen bonding patterns associated with “Highway” versus “Meander” versus “Ambiguous” transitions and to design algorithms that can identify these same folding pathways and critical protein–water interactions directly from the data. Highly cooperative hydrogen bonding can either slow down or speed up transit. Furthermore, an analysis of protein–water hydrogen bond dynamics at the surface of WW domain shows an increase in hydrogen bond lifetime from folded to unfolded conformations with Ambiguous transitions as an outlier. In summary, hydrogen bond dynamics provide a direct window into the heterogeneity of transits, which can vary widely in duration (by a factor of 10) due to a complex energy landscape.
“Sonification-Enhanced Lattice Model Animations for Teaching the Protein Folding Reaction,” Journal of Chemical Education, J. Chem. Educ. 2022, 99, 3, 1220–1230. Carla Scaletti*, Meredith M. Rickard, Kurt J. Hebel, Taras V. Pogorelov, Stephen A. Taylor, and Martin Gruebele*, February 16, 2022. | Journal article DOI: 10.1021/acs.jchemed.1c00857 Accepted Manuscript (AM) PDF. Supporting Information (includes links to sonification/animations on YouTube)
The protein folding reaction is one of the most important chemical reactions in the human body. Yet, despite its importance, it is sometimes omitted from undergraduate courses due to the challenging nature of some of the underlying concepts. To help make key concepts of the protein folding reaction accessible to our undergraduate students, we implemented three, simplified 2D lattice models of various amino acid chains, and we used these models to generate sound-enhanced animations that allow students to see and hear the dynamics of protein folding in action. In spring of 2021, we used these videos in remote learning biophysics and music courses to introduce four key concepts of the folding reaction: solvation and hydrophobicity; energy and conformational entropy; funneled energy landscape; and frustration and traps. Our lattice model animations and sonifications helped provide insight into protein folding dynamics for undergraduate and graduate biophysical chemistry students, undergraduate musicians, and even authors who are experts in this field. We plan to incorporate these and additional animations, along with enhancements to the 2D lattice models, in our future courses.
“Sonification ≠ music” in The Oxford Handbook of Algorithmic Composition, edited by Alex McLean and Roger Dean, 363–386. New York: Oxford University Press, 2018.
Data sonification is a mapping from data generated by a model, captured in an experiment, or otherwise gathered through observation to one or more parameters of an audio signal or sound synthesis model for the purpose of better understanding, communicating or reasoning about the original model, experiment or system. Although data sonification shares techniques and materials with data-driven music, it is in the interests of the practitioners of both sound art and data sonification to maintain a distinction between the two fields.
Sonification of the world (2014 essay)
We don’t live on the earth. We are the Earth. An essay written for Joel Chadabe’s Ear to the Earth on how we can listen to the music of our sphere.
LHCSound (2013)
A website with sound examples and descriptions of work with Lily Asquith on sonifying data from the Large Hadron Collider.
Sound Synthesis Methods for Auditory Data Representation (1992)
An invited talk presented at the first International Conference on Auditory Display (ICAD) at the Santa Fe Institute, proceedings published as Auditory Display: Sonification, audification, and auditory interfaces”, Gregory Kramer, ed., Santa Fe Institute Studies in the Science of Complexity, 1994.
Using Sound to Extract Meaning from Complex Data (1991)
A video produced with Alan Craig at the National Center for Supercomputing Applications to demonstrate some of the ways in which data-driven sound can enhance and extend data-driven visualizations
Winner of the NICOGRAPH International 1991 Multimedia award
Using Sound to Extract Meaning from Complex Data (1991) Scaletti, C. & A. Craig.
A talk presented at the 1991 SPIE Conference in San Jose and printed in Extracting Meaning From Complex Data: Processing, Display, Interaction II Volume 1459, Edward J. Farell Chair/Editor, SPIE-The International Society for Optical Engineering, San Jose, February 1991.
Presentations
Keynote lecture for the 25th anniversary of the International Conference of Auditory Display: ICAD 2017 — Sound in Learning at Pennsylvania State University June 2017
Mapping Numbers to Sound—From Scientific Exploration to Immersive Musical Experience
Lecture / demonstration with biophysicist Martin Gruebele, composer/software developer Carla Scaletti, composer David Rosenboom at Center for Advanced Study, University of Illinois
Podcasts
The Science of Sound (2023). Can sound help us understand the complex patterns in our universe? This question leads Nate to Symbolic Sound in Champaign, Illinois, where composer Carla Scaletti guides him on a journey where sound, music, and data intertwine in captivating and thought-provoking ways…
2022 Science We Missed (2022). Maura Armstrong & Bobby Frankenberger’s All Around Science podcast (starting at timecode: 35:43).
The Sounds of Data (2024) by Julie Gould, Physics Magazine (This is Physics Podcast), July 11, 2024. Physics Magazine speaks with scientists who are relying on senses other than their sight, such as hearing and touch, to interpret data
Press
Sounds of science: Why just look at your data when you could listen to it? (2023) by Sumeet Kulkarni, Los Angeles Times, Science and Medicine, February 3, 2023
Biochemist Martin Gruebele… uses a software program called Kyma to add a specific sound to each of the numerous bonds that occur as the protein folds. When played back, the sound brings order to the chaos by highlighting which particular interactions dominate.
“You have to think of that sound in the same way that you think about a graph as opposed to a painting,” Gruebele said.
… Scaletti agreed that sound has the power to convey a lot of meaning… That’s why she’s carving a new niche in the human soundscape for science.
The Science is Sound (2025) by Pallab Roygupta, Shaastra, Vol. 04, Issue 06, July 2025
“Scaletti’s Kyma captured the sounds of hydrogen bonds forming and breaking as proteins folded and unfolded, while her team created 3D animations, with each bond being assigned a specific pitch. ‘… I could listen to hydrogen bonds forming and breaking in the loops and recognise a pattern, (whereas) looking at the video did not show me anything visibly coherent,’ says Gruebele. ‘That’s when I realised (that) sound can be more effective than sight for the right things.’”
From Symphony to Structure: Listening to Proteins Fold (2024) by Rohini Subrahmanyam, The Scientist, August 21, 2024
“This is a nice example of using sonification for discovery,” said Roseanne Ford, a chemical engineer at the University of Virginia, who was not involved in the research. “Visually, there are simultaneous things going in your field of view—with multiple hydrogen bonds in different locations of the protein—and your eyes can’t track all of that. But you can hear multiple tones or multiple pitches at a time, so you can get a sense of the temporal changes in the hydrogen bonding that are harder to get at visually,” said Ford.
Hearing proteins fold (2024) by Pallab Roygupta, Shaastra, Vol. 03, Issue 05, June 2024
“Scaletti took inspiration from computer games, where a metallic sound plays when two swords collide. Applying a similar approach in their simulation, the researchers triggered a specific sound whenever a pair of amino acids were close enough to form a hydrogen bond. ‘So, we could hear when bonds were forming and we could also distinguish bonds from one another and hear when they formed cooperatively, meaning near-simultaneously,’ said the researchers.”
Scientists are turning the process of protein folding into sound (2024) by Payal Dhar, C&EN Chemical & Engineering News, Vol. 102, Issue 17, June 1, 2024
By tuning into the sounds of proteins as they fold, researchers are developing new insights into the process.
Illinois musicians, chemists use sound to better understand science (2022) by Jodi Heckel, Illinois News Bureau, University of Illinois
Musicians join scientists to explore data through sound (2017) by Carolyn Beans, Science Writer for the Proceedings of the National Academy of Sciences of the United States of America (PNAS vol. 114 no. 18 Carolyn Beans, 4563–4565, doi: 10.1073/pnas.1705325114).
Carolyn Beans’ Front Matter article in PNAS gives an overview of how musicians and researchers are working with data sonification, translating data into sound with the end goal of developing “deep insights into data revealed through sound”.
For composer and data sonifier Carla Scaletti, data sonification and music have different goals. Data sonification aims to “discover something about the original phenomenon that produced the data,” she says. “It’s almost like you don’t care that it was conveyed by sound. You’re trying to hear that underlying structure; whereas for music, you do want people to be aware of the sound.”
Scaletti likens aesthetic choices in data sonification to graphic design choices when preparing a chart for a scientific paper. “You choose colors and you choose a font, but all your choices are guided by the goal of wanting to make the data very clear.” When Scaletti isn’t working on scientific projects, she sometimes uses data in compositions, but she calls those works data-driven music, or just music.
Courses and tutorials
Hear Your Data: A Hands-On Data Sonification Workshop (2025) C. Scaletti, K. Hebel
By mapping data to sound, you can perceive complex patterns in a new dimension, making it a powerful method for discovery and analysis — a unique opportunity for researchers to explore an innovative analytical technique.
Sounds of Busan (2019) C. Scaletti, K. Lee, H. Park
Workshop presented at the KISS2019 conference in Busan, South Korea. In this hands-on session, we will take time series data related to the city of Busan (datasets from the 2017 Pohang earthquake and Busan tidal levels) and map the data to sound. Can we hear patterns in data that we might not otherwise detect?
Sonification Tools in Kyma (2011)
Tools for mapping data space to sound space, presented at the Kyma International Sound Symposium (KISS2011) in Porto.
An Introduction to Data Sonification (1993) Evans, B., R. Bargar & C. Scaletti.
Course Notes for Tutorial 81 of the SIGGRAPH 20th International Conference on Computer Graphics and Interactive Techniques. New York: Association for Computing Machinery.
Awards
2025 Sonification Award in the data analysis category
Advisory panels
“Sonification: a tool for research, outreach and inclusion in space sciences” in support of full and equal participation of persons with disabilities in the space sector,“ United Nations Office for Outer Space Affairs, April 2023.
“Accessible Oceans: Exploring Ocean Data Through Sound: Building knowledge about effective design and use of auditory display for inclusive inquiry in ocean science“ NSF Award Amy Bower (PI)