The Music Tapes started life as a creative outlet for the 16 year old Julian Koster (later of Neutral Milk Hotel), who would create tapes of songs and stories as means of escapism, which he would share with his friends, culminating in 'The American Phoam Rubber Co.

Symphony Orchestra Proudly Presents the 2nd Silly Putty Symphony for Edison Wax Cylander', which was essentially tape collage. In 1989, Koster formed the band Chocolate USA, a more conventional indie-pop band, which also included Bill Doss and Eric Harris, later (along with Koster, though somewhat unofficially) members of the Olivia Tremor Control.

The band released two full albums, All Jets Are Gonna Fall Today and Smoke Machine, before splitting in 1994, after which Koster chose to focus his creative efforts on the Music Tapes once more.In 1999 Julian released the album First Imaginary Symphony For Nomad, featuring a slew of musicians helping him, most notably Robbie Cucchiaro and Eric Harris.

Ten years later Music Tapes for Clouds and Tornadoes was released, featuring the usual bevy of artists, most notably an eight foot tall metronome. On september 4th, 2012, their last album, Mary's Voice, was released by Merge Records.

In the lead-up to the album, Julian Koster held a Kickstarter campaign for a subsequent tour, with rewards including signed copies/master copies of the album, personalized messages and images, and Koster's banjo used on previous albums, most notably Neutral Milk Hotel's In The Aeroplane Over The Sea.