How Sound Affect Perception
Capstone Project for Music Tech BM degree
For the Capstone Project for my bachelor degree for Music Technology at NYU, I used a comparative examine method to challenge audiences' perception about visions.
Introduction
Different from other film projects that focus more on the picture and coloring, this project was aimed to focus on how sound plays its role in story-telling. I was experimenting to see if the story can be altered without touching the image. In order to achieve that goal, I chose two documentary clips and sound-designed each of them twice with different sound element choices. In the end, I found that without changing any images, the use of sound alone can change the course of the entire story plot, or even fabricate a 'story' out of thin air.
Background
The idea of this project is inspired by the movie A Quiet Place. In this movie, sound and music play a very important role and are the key messages that drive the entire plot of the movie. The film's viewers are "forced" to pay more attention to the information they receive in their ears. There have been many studies on "how music images film" and most of them point out that “Film music has powerful aesthetic effects on the perception and understanding of screen content” (Hoeckner). Composer Bernard Herrmann commented when discussing the effect of music in a film, that “[music] can invest a scene with terror, grandeur, gaiety or misery. It can propel narrative swiftly forward or slow it down. It often lifts mere dialogue into the realm of poetry. Finally, it is the communicating link between the screen and the audience, reaching out and enveloping all into one single experience” (Hermann). Almost clearly, people have understood that upbeat tunes can make the whole movie bright and lively, while heavy music can make the whole plot clouded with gloomy colors. However, I believe that sound effects themselves can do the same thing without the help of music and achieve the same effect. Walter Murch once said, “Images come in through the front door, but sound comes in through the back door” (Murch). Ethan Van der Ryn, the sound editor/designer of A Quiet Place said that “it's so important for screenwriters when they're writing to actually be thinking about what is happening sonically in the world” (Ethan). Similarly, Scott Beck once said, “you have to envision where there might be sound design moments” (Beck). And these statements influenced the order of my process when completing the project.
Project Description
Finding a Clip
The first step was to find the right film to work on. My initial choice was a student film of about 15 minutes about a man trapped in a cave in a desperate situation and his innermost struggle. There is a lot of inner monologues in this film, and the quiet environment gives the film a lot of room for creativity. I almost had a draft in my head for how the different voices would be arranged. However, because the director was too busy towards the end of the semester, he was not able to send me the audio files I needed in time, thus, unfortunately, I did not have the opportunity to work on this piece due to time constraints. Then I found two student documentaries I had done before, one about a factory and one documenting a trip to a hot air balloon flight. After getting permission from both producers, I chose these two clips as my capstone project.
Project Description
First Clip----Factory
After selecting the film, I first started working on the first clip of ----Factory. The content of this film is very empty, without any plotline, just some simple switching between scenes of factory exterior, interior, machines, etc. For the first production I chose "Plain Style", which means that everything was done in the same way as it was presented in the film. If the picture is a machine, then I put in the sound of the machine, if it's a long shot, then I would be putting in factory sound and bird chirping in the ambiance and background. The first time was easier because I just had to fill in the corresponding sounds as they were in the picture. For the second time, I needed to use more imagination and creativity.
In the end, the film ends with a white abstract flower fire flashing out on a black background. In this scene in the first "plain" I simply put some fireworks sound up. But in the second time, I added a lot of other elements. In addition to fireworks, I also added the sound of distant explosions and some sirens. In all the sounds together to enhance a certain degree when suddenly a plunge, in addition to the sound of the alarm all the sounds are reduced to an almost untraceable degree. This effect works well to bring a dramatic enhancement to the end of the film.
The film is relatively relaxed at the beginning, I correspondingly chose some simple and relaxed distant factory sound and some light wind sound for the background ambiance. And in the next color-shift, I also made a huge shift in sound. With a warning alarm sounded, the whole story was brought to the wartime. In this scene, I added a lot of war sound elements. For example, Morse code, air defense sirens, soldiers' footsteps, bombs, gunfire, etc.
In the later closeup shots of the machine, in addition to the sound I designed the machine itself would make, I also added some people crying in the background. This way, listeners can't help but think about whether the soldiers threw the captured people into these machines, squeezed them into minced meat, or cooked them into some thick soup.
Near the end of the film, the last scene where something concrete appears is a girl standing in front of a huge factory fan, her hair fluttering dramatically with the wind. I thought this scene could have a lot of room for imagination, so I turned it into a girl's farewell to the world. I added helicopters, footsteps, cries, bombs, etc. to achieve this effect.
.png)
Project Description
Second Clip----Adventure
My second clip documents a group of people together on a hot air balloon flight journey. Because this film introduces the whole team from preparation, departure, take-off, flight, and finally return in a relatively short period of time (2 minutes and 30 seconds), there is not much room left for me to manipulate the storyline. So, like Factory, I finished a "Plain Version" first. This film has more content and a lot of transitions between scenes. It also contained less-common elements such as hot air balloons and ignition, which took me a lot of effort even in the first easier version. Especially the element that makes up a large part, the sound of flames in a hot air balloon. I had to dig through Soundsnap.com for a long time to find a few more appropriate sounds for ignition and combustion. Due to the change of scene and the different states of the balloon, I could only use the few sound clips I had extracted through different designs and mixing to achieve the effect I wanted.
When brainstorming how the second sound approach should be presented, it was because of the complexity of the content that I struggled with, it was almost very hard to present the creative power while presenting the content in the video at the same time. Because I firmly believe that the basis for indulging in creation must be to do the sound's job ---- to help the visuals tell the story together. This has led to an extreme lack of space for me to go wild.
After much deliberation, I decided to make a big shift in the ending of this documentary. In the end, the film shows people simply driving home. However, before the balloon landed, while the scene was still in the sky, I added some sharp metallic chattering sounds, accompanied by the beeping of the machine reporting errors, depicting a feeling of a hot air balloon malfunctioning and shaking violently. Here I also used the same technique as in the first clip ---- All the sounds are enhanced together and suddenly disappear at a certain moment, leaving the audience with dead silence. Then, suddenly there is a huge falling explosion sound. Followed closely by the sound of an ambulance and the background sounds of a woman and child crying. Through this short shift of a few dozen seconds, I managed to give these people who arrived home safely a tragic ending.
Future Work
Although I managed to change the audience's understanding and interpretation of the story through my control of sound in these two clips, I know that there are still some shortcomings and areas for improvement. For example, in the second clip, the clip itself has a lot of visual material, leaving very little room for the sound to play freely. In my future studies, I would like to spend more time considering how I can make the sound play more of a role in this already dominant visual situation, or if I actually need to do so. At the same time, during the presentation, one of the teachers in the teacher panel gave a very good suggestion ---- to turn this 2D work into 3D and to utilize more use of immersive audio. At the same time, in future research, I want to take into account the influence of music on the film as well. I will look deeper into how different combinations of music and sound have wonderful chemistry that can have a deeper impact on the audience.
Future Work
Although I managed to change the audience's understanding and interpretation of the story through my control of sound in these two clips, I know that there are still some shortcomings and areas for improvement. For example, in the second clip, the clip itself has a lot of visual material, leaving very little room for the sound to play freely. In my future studies, I would like to spend more time considering how I can make the sound play more of a role in this already dominant visual situation, or if I actually need to do so. At the same time, during the presentation, one of the teachers in the teacher panel gave a very good suggestion ---- to turn this 2D work into 3D and to utilize more use of immersive audio. At the same time, in future research, I want to take into account the influence of music on the film as well. I will look deeper into how different combinations of music and sound have wonderful chemistry that can have a deeper impact on the audience.
Reference Used
Hoeckner, Berthold. “Film Music Influences How Viewers Relate to Movie Characters.” American Psychological Association, https://psycnet.apa.org/fulltext/2011-01009-001.html.
Khaled, Fatma. “The Underlying Effect of Music in Film and the Art of Film Scoring.” EgyptToday, 4 Apr. 2018, https://www.egypttoday.com/Article/15/46906/The-underlying-effect-of-music-in-film-and-the-art.
Lessons from the Screenplay. “A Quiet Place - How to Write Sound into a Screenplay.” YouTube, YouTube, 31 Aug. 2018, https://www.youtube.com/watch?v=T-s81-Wl2v0&t=3s.