With this tool, researchers can listen to cells changing

Modern biomedicine increasingly relies on enormous amounts of image data. Although computers can do part of the analyzation work, biologists often still need to look through the images themselves. This takes time.

“Therefore, we asked ourselves whether we could turn images into sound, so-called image sonification. This could prove useful in detecting patterns that are difficult to see with the naked eye,” Bálint Laczkó says.

In close collaboration with colleagues at the University of Oslo, he has developed Pixasonics, a tool that can turn image data into sound. Its underlying method is freely available.

Published as a Python library

Today, various kinds of biological and medical research apply image sonification, for instance in efforts to improve diagnosis. While some of the existing tools are research-oriented, others are mobile apps where you can upload an image and an audio representation in return.

Pixasonics uses the programming language Python and was created in collaboration with biologists in the project Autorhythm. The primary intended users are researchers in the life sciences, who can integrate it into programs they already use. Beyond that, it can be of use both to other researchers, to sound designers, and musicians, according to Bálint Laczkó.

“It is a toolbox published as an open Python library. This means that it is a set of ready-made building blocks: When you write Python code, you can plug these blocks in, so that you do not have to build everything yourself,” he says.

However, if you wish to create something more unique, you can easily customize the blocks or build your own.

– In this way, image sonification can be seamlessly integrated into your personal workflow.

Working to understand cells’ recycling system

In the Autorhythm project, researchers are working towards a better understanding of the cells’ own recycling system, called autophagy: The cells degrade and recycle protein clumps and structures that no longer function properly.

The older humans get, the more these processes weaken. This leads to a gradual destruction of the brain and other organs.

Research has indicated that strengthening the processes can be life-extending and that it may also open possibilities for better treatment of diseases – especially diseases in which nerve cells gradually deteriorate, such as Alzheimer’s and Parkinson’s. For this to happen, however, researchers must fully understand the autophagy process.

This is where Pixasonics comes in, since the work involves analysis of thousands of detailed images: Cell components are colored with so-called fluorescent proteins, which emit light, before biologists take a long series of high-resolution images, so-called timelapses. In this way they see what happens in the cells over time.

Sound reveals changes in an image series

“This is painstaking work, which can both give you sore eyes and involve a risk that important details are not detected,” Bálint Laczkó explains.

With Pixasonics, biologists can upload the images they want to analyze and assign a specific task to the tool. The pixels in the image translate into sound, which plays in real time while the images display on the screen.

“For example, a bright pixel can give a high, beeping tone and a dark pixel can give a tone that is deeper and rumbling. If you move the mouse pointer around, bright areas will make the pitch rise, while dark areas will make it fall.”

Using the tool on an image series, you can hear how the subject changes over time. In this way, researchers will not have to jump back and forth between images to see what is happening in a particular area. The sound reveals how the image changes.

Can help find the needle in the haystack

“Our vision and hearing have evolved with different strengths. While vision is best at detecting spatial patterns, such as shapes, distance and position, hearing is best at perceiving patterns that unfold over time, for example rhythms, changes, and sequences. Thus, a combination of eyes and ears can make large volumes of images easier to explore,” Laczkó says.

Researchers, musicians, and technologists have all contributed input to Pixasonics. Now he hopes to continue developing the tool, continue holding workshops and even conduct formal user studies.

“In addition, my colleagues and I have several ideas to how Pixasonics can become particularly useful in researching the cells’ cleaning processes. For example, we believe that it can help find the needle in the haystack in cases where a tiny but important part of the autophagy process is buried in the sea of cells that are visible under the microscope,” Bálint Laczkó says.