Generated Media: New york is better than here Video

How much meaning do words carry for an algorithm? Using a text to image generator I explore how a machine learning program interprets a love song written by my partner, who misses me very much.

New York is Better Than Here, 2021 / Acoustic Recording and Images Generated with Machine Learning, 4:17 / by Maya Williams and Nicholas Luna

New York is Better Than Here is an original song written and performed by Nicholas Luna (@nickluna.music). For this video, I used Runway ML’s Text to Image generator (AttnGAN) to create images based on lyrics from the song. These images were then used as a backdrop for a video of the artist live performing the song. The idea behind this was to allow this algorithm to create a sense of tone and mood through abstract generated visuals to accompany the organic performance. Below are each of the images and the corresponding lyric that created them.

Drivin away

Past the interstate

The sounds and the lights

Of the city I left start to fade

And already I’m thinking

Of when im coming back

I vacuumed the dishes

and washed all the carpet

But dust bunnies still found a home on the hardwood floors

I guess we’ll call it bargaining

But I don’t really  wanna bargain anymore

than here

has you

There’s something about

This cold combination

Of distorted guitars.

And the wind blowing at my face

Puts me in my place

I guess we’ll call it bargaining a

And I’m bargaining again

That was always your chore

New York

Yes New York is better than here

It’s a little unfair

I haven’t even left yet

Here on the couch

Back in my apartment

Scribbling down

a pros and a cons list, abhorred

It’s a little unfair

I haven’t even settled

And already I’m dreaming

Of walking out the door

is better

Cause New York

The lyrics were broken up largely at my discretion based on what I felt worked best with the recorded performance. If someone was to take the same song, they could break it up differently and end up with a completely different set of visuals. As I was inputting text, it was interesting to see that the generator would drastically change the image even based on the inclusion of spaces and line breaks. Taking some artistic license I alternated between including and excluding the line breaks based on how much I liked the visual it was producing.

With the more descriptive lyrics, the algorithm seems quite good at conjuring up an image that feels reminiscent of the associations being made in my head. There are a couple of sections that demonstrate this really well:

• “Driving away past the interstate the sounds and the lights of the city I left start to fade” truly looks like a lit-up city

• “Here on the couch back in my apartment scribbling down a pros and a cons list, abhorred” definitely looks like an apartment with some dark chunky couches on each side of the room

• “I vacuumed the dishes and washed all the carpet but dust bunnies still found a home on the hardwood floors” while still being very abstract has several textures that look like hardwood, carpet, and dust bunnies

For less visually descriptive sections of the song, the results seem a bit more random. One interesting result was that the lyrics “New York” and “Cause New York” both generated completely black backgrounds. This result certainly doesn’t align with what I think would come to mind for most people when they picture New York, but the songwriter really liked the stylistic impact of the cut to black at those parts of the song.

Looking to the future, I think it would be interesting to try this with a speech-to-image generator especially one that was coded to read emotional cues and see if the generated images could reflect the emotionality of the performance.

This last image was obtained when the entire song was input into the image generator. I’m encouraging the artist to use it as album art.