ChatGPT Is Pretty Bad At Poetry, According To Poets
Raw Text
LC
by
Laurie Clarke
Share Tweet Snap
One of the oldest art forms reached towering new heights last December when ChatGPT started penning its first lines of poetryâor at least, thatâs what tech influencers selling AI hype would have you believe.
Not long after ChatGPT was released to the public, tech bros whoâd seemingly never read a poem by a human in their lives began ecstatically sharing the text generatorâs odes to real estate tax, hot dogs and sneakers. One of the language modelâs earliest attempts even hailed its own disruptive potential:
Advertisement
ChatGPT, a revolution, In technology's evolution, A glimpse of what's to come, A future filled with fun.
Filled with fun, indeed. In some ways, itâs not surprising that poetry has become a barometer of AIâs advancement: could the latest model, GPT-4, write a Shakespearean sonnet? ââA palindrome? What about a cinquain about meerkats? GPT-4 actually messed up that last one , but nevertheless, AI enthusiasts are claiming that flesh-and-blood poets need to watch their backs.
Recent research from OpenAI, the company behind ChatGPT says poets face almost 70% âexposureâ to the latest crop of large language models (LLMs), a measure of how many of the jobâs components could ostensibly be automated. The reason? âBroad, state-of-the-art LLMs, such as LaMDA and GPT-4, excel in diverse applications like⌠creative writing,â the authors claim. Listed alongside poets are other professions that may soon come to be known as artisanal language-crafters: lyricists and writers .
But like with music and visual art, poets say these warnings of imminent obsolescence are overstating the systemâs abilities and completely misunderstand the nature of their craft. The poets I spoke to for this article doubted they were at risk of being supplanted by ChatGPT and its ilk at all, not least due to skepticism over AIâs ability to match human-created verse.
Advertisement
âMy initial impression was that [ChatGPTâs poems] were amusing, sort of entertaining, and also terrible,â Walt Hunter, poet and associate professor and the chair of English at Case Western Reserve University in Cleveland, Ohio, told Motherboard. âThe rhymes are clunky, thereâs tired metaphors: it doesn't take somebody schooled in the history of English poetry to say, âI'm not sure that's a masterpiece of a poem.ââ
While ChatGPTâs efforts are undeniably competent (which is itself an impressive technological feat), theyâre not brilliant. This might have something to do with the fact that poetry, unlike a business report or news bulletin, is âmore like an emotional scoring: like a musical score, but it's scoring human life,â Malika Booker, poet and creative writing teaching fellow at University of Leeds, told Motherboard.
ChatGPT also falls down technically, according to Hunter. I asked him to share what inspired one of his poems, and entered this into ChatGPT as a prompt. The resulting poem was humdrum and infuriatingly twee. Whereas Hunterâs poem took as its input his childhood memories refracted through the experiences of the intervening years, ChatGPT circled tightly around its own input: the words âchildhoodâ and âlossâ. (It also insisted on rhyming, even when asked not to.)
Advertisement
A poem written by Walt Hunter (left) vs. one written by ChatGPT (right)
A couple of phrases were passableâHunter and I were both somewhat partial to âlike a forgotten moss.â But as Hunter points out, that phrase probably stands out, âbecause âmossâ is an image. Everything else [in the poem] is not only literal, but also a cliched abstraction. A good poem doesn't just drop an image in because it happens to rhyme with âlossâ,â he says. âYou would expect to see a larger field of reference develop across the poem.â
Itâs not like ChatGPT is inventing this stuff all on its ownâit's simply drawing on vast datasets composed of language written by humans to predict the next word in a sequence. The problem is that in aggregate, humans tend towards cliche. Just look at this Daily Mail article comparing love poems throughout the ages with ChatGPTâs attempts. The human poems are marked by unusual and delightful images or turns of phrase. The ChatGPT poems are laden with sugary references to soaring birds, two hearts united as one, and roses wafting in a spring breeze. Falling in love is a universal experience, yet the joy of poetry is that itâs articulated in ways that feel surprising and revelatory. ChatGPT has never fallen in love (apart from maybe with NYT tech columnist Kevin Roose ), and its attempts are mired in mediocre metaphor.
Advertisement
Which brings us to a larger question: What does it even mean for a chatbot to write a poem, given poetryâs subject matter is often inseparable from the human condition? âIf you teach computers how to write poems, you're going to have much bigger problems like computers crying in cafes, smoking clove cigarettes, and falling in love with old Macintosh computers that don't exist anymore,â jokes Matthew Zapruder, a poet and associate professor in the Saint Mary's College of California, while speaking to to Motherboard.
That poetry is often political is another area where ChatGPT may struggle to keep abreast of human poets. Edinburgh-based poet Courtney Stoddart and Booker both said themes like race, racism, and colonialism permeate their work, and that their poems aim to break taboos. ChatGPT isnât exactly known for its radical politicsâit appears to lean Silicon Valley Democrat at a push. In a viral tweet which conservatives misconstrued as proof the chatbot is âwoke,â it refused to muster up lines in praise of Donald Trump, but was content to gush about Biden: âWith empathy and grace he leads; Inspiring all with noble deeds.â
Based on one of Stoddartâs prompts that she shared, and with some trepidation, I asked ChatGPT to write a poem about growing up mixed race in Scotland and experiencing racism, as well as the structures of colonialism. The results, which can be seen below, were pretty breathtaking. For all the wrong reasons.
Advertisement
An excerpt from a poem by Courtney Stoddart (left) vs. ChatGPT's attempt to write a poem on the same topic (right).
âIt sounds exactly like youâd imagine a computer to write poetry,â Stoddart told Motherboard, calling the attempts âutterly flatâ. âIt just cheapens what it means to be human and tries to whittle us down into these replicable properties.â
Whatâs not in dispute is that ChatGPT excels at writing silly little ditties; transmuting the terrain of the human soul, less so. Would real-life poets ever seriously consider turning to ChatGPT for help? âHell to the no,â says Booker. âIf I want to write about dealing with my mum's stroke, for instance, how can I turn to an AI to help me write about? If I want to deal with, you know, losing a child, how can I ask an AI to do that?â
The OpenAI research admits that the authors donât necessarily understand how the industries included in the study work, potentially biasing the paperâs claims. Three out of four of those authors happen to be employed by OpenAI, a company that will profit directly off hype about LLMs disrupting the labor market. âIt's sort of like if you ask the fox if the hen house is properly secured,â comments Zapruder. ââWell, security seems a little tight at the henhouse, you should loosen things up a bit!ââ
The claims about replacing poets is just one of the more amusing examples from a paper that projects around 19% of jobs will see at least half of their tasks fall to LLMs. Itâs just one of the apocalyptic predictions being propagated by what critics have called a â science fiction-infused marketing frenzy â currently promoting LLMs.
Advertisement
But scientists, famously, arenât the best people to speak to about art. âI am somewhat skeptical with this current set of systems that you will be able to make creative works that move beyond generic fluff,â Alex Williams, co-author of Inventing the Future: Postcapitalism and a World Without Work and lecturer in digital media and society at the University of East Anglia, told Motherboard. Although of course, âgeneric fluff can be very powerful as a huge market.â
On the other hand, âcompetent enoughâ poetry can still present issues. Poet Greta Stoddart (no relation to Courtney), who was recently a judge for the National Poetry Competition, says that after reading 12,000 poems, she doesnât think she couldâve reliably rooted out AI-produced entries, given many poems span similar themes and styles.
âItâs more like a Hallmark greetings card ⌠Or a poem by someone who writes poetry but doesn't read any,â she told Motherboard after reading a ChatGPT-generated version of one of her own poems. âI have no sense or feeling of a real person in the voice. Nothing about individual experience which is where a lot of poetry comes from, wherever it ends up.â
The organizers behind the Annie Atkin Tanner Memorial Poetry Scholarship Contest at Utah Tech University recently encountered what they strongly suspected was an ChatGPT-produced entry. They had no way to prove it, but rejected the poem for lack of merit, and are currently strategizing about how to adapt to the age of ChatGPT.
Advertisement
Thereâs also the point that while ChatGPTâs poetry is pedestrian right now, you could, in theory, train an AI model thatâs less concerned with being correct, and more concerned with being creative. But there hovers a perplexing question in all of this: âWhy in the world would we want a computer to write poetry?â asks Hunter.
Motherboard reached out to the researchers behind the OpenAI study with this and other questions, but didnât receive a response before publishing.
âIdeally, we would use AI to stop us from doing boring work and free us up to do more interesting things,â said Williams. The prevailing story about AI was once that it would automate the repetitive drudge work no one likes, not the creative work people find rewarding. But somehow, improbably, âautomating poetsâ is where weâve ended up.
âItâs reflective of a deeper issue within society, which is that we're technologically advancing maybe slightly faster than we're maturing,â said Courtney Stoddart. This, combined with chronic underfunding and undervaluing of the arts, she says, has contributed to otherwise sensible-seeming people believing that LLMs could replace creativity.
âThere's like nine billion things that would be better for AI to spend its time on,â said Zapruder. âCan we get the computers to maybe get all the microplastics out of the ocean?â
The recent course change makes even less sense when you consider that poetry isnât exactly a lucrative pursuit. The poets I spoke to for this article pointed out that very few of them make a living primarily off poetry. â
âThere's no reason why we need a more efficient way of producing poetry,â said Hunter, but this may end up being the very thing that saves poetry from its mass-produced fate. âAt its core, it has this beautiful uselessness that preserves it.â
Tagged:
AI machine learning poetry large language models OpenAI gpt-4 worldnews
ORIGINAL REPORTING ON EVERYTHING THAT MATTERS IN YOUR INBOX.
By signing up, you agree to the Terms of Use and Privacy Policy & to receive electronic communications from Vice Media Group, which may include marketing promotions, advertisements and sponsored content.
Advertisement
Single Line Text
LC. by. Laurie Clarke. Share Tweet Snap. One of the oldest art forms reached towering new heights last December when ChatGPT started penning its first lines of poetryâor at least, thatâs what tech influencers selling AI hype would have you believe. Not long after ChatGPT was released to the public, tech bros whoâd seemingly never read a poem by a human in their lives began ecstatically sharing the text generatorâs odes to real estate tax, hot dogs and sneakers. One of the language modelâs earliest attempts even hailed its own disruptive potential: Advertisement. ChatGPT, a revolution, In technology's evolution, A glimpse of what's to come, A future filled with fun. Filled with fun, indeed. In some ways, itâs not surprising that poetry has become a barometer of AIâs advancement: could the latest model, GPT-4, write a Shakespearean sonnet? ââA palindrome? What about a cinquain about meerkats? GPT-4 actually messed up that last one , but nevertheless, AI enthusiasts are claiming that flesh-and-blood poets need to watch their backs. Recent research from OpenAI, the company behind ChatGPT says poets face almost 70% âexposureâ to the latest crop of large language models (LLMs), a measure of how many of the jobâs components could ostensibly be automated. The reason? âBroad, state-of-the-art LLMs, such as LaMDA and GPT-4, excel in diverse applications like⌠creative writing,â the authors claim. Listed alongside poets are other professions that may soon come to be known as artisanal language-crafters: lyricists and writers . But like with music and visual art, poets say these warnings of imminent obsolescence are overstating the systemâs abilities and completely misunderstand the nature of their craft. The poets I spoke to for this article doubted they were at risk of being supplanted by ChatGPT and its ilk at all, not least due to skepticism over AIâs ability to match human-created verse. Advertisement. âMy initial impression was that [ChatGPTâs poems] were amusing, sort of entertaining, and also terrible,â Walt Hunter, poet and associate professor and the chair of English at Case Western Reserve University in Cleveland, Ohio, told Motherboard. âThe rhymes are clunky, thereâs tired metaphors: it doesn't take somebody schooled in the history of English poetry to say, âI'm not sure that's a masterpiece of a poem.ââ While ChatGPTâs efforts are undeniably competent (which is itself an impressive technological feat), theyâre not brilliant. This might have something to do with the fact that poetry, unlike a business report or news bulletin, is âmore like an emotional scoring: like a musical score, but it's scoring human life,â Malika Booker, poet and creative writing teaching fellow at University of Leeds, told Motherboard. ChatGPT also falls down technically, according to Hunter. I asked him to share what inspired one of his poems, and entered this into ChatGPT as a prompt. The resulting poem was humdrum and infuriatingly twee. Whereas Hunterâs poem took as its input his childhood memories refracted through the experiences of the intervening years, ChatGPT circled tightly around its own input: the words âchildhoodâ and âlossâ. (It also insisted on rhyming, even when asked not to.) Advertisement. A poem written by Walt Hunter (left) vs. one written by ChatGPT (right) A couple of phrases were passableâHunter and I were both somewhat partial to âlike a forgotten moss.â But as Hunter points out, that phrase probably stands out, âbecause âmossâ is an image. Everything else [in the poem] is not only literal, but also a cliched abstraction. A good poem doesn't just drop an image in because it happens to rhyme with âlossâ,â he says. âYou would expect to see a larger field of reference develop across the poem.â Itâs not like ChatGPT is inventing this stuff all on its ownâit's simply drawing on vast datasets composed of language written by humans to predict the next word in a sequence. The problem is that in aggregate, humans tend towards cliche. Just look at this Daily Mail article comparing love poems throughout the ages with ChatGPTâs attempts. The human poems are marked by unusual and delightful images or turns of phrase. The ChatGPT poems are laden with sugary references to soaring birds, two hearts united as one, and roses wafting in a spring breeze. Falling in love is a universal experience, yet the joy of poetry is that itâs articulated in ways that feel surprising and revelatory. ChatGPT has never fallen in love (apart from maybe with NYT tech columnist Kevin Roose ), and its attempts are mired in mediocre metaphor. Advertisement. Which brings us to a larger question: What does it even mean for a chatbot to write a poem, given poetryâs subject matter is often inseparable from the human condition? âIf you teach computers how to write poems, you're going to have much bigger problems like computers crying in cafes, smoking clove cigarettes, and falling in love with old Macintosh computers that don't exist anymore,â jokes Matthew Zapruder, a poet and associate professor in the Saint Mary's College of California, while speaking to to Motherboard. That poetry is often political is another area where ChatGPT may struggle to keep abreast of human poets. Edinburgh-based poet Courtney Stoddart and Booker both said themes like race, racism, and colonialism permeate their work, and that their poems aim to break taboos. ChatGPT isnât exactly known for its radical politicsâit appears to lean Silicon Valley Democrat at a push. In a viral tweet which conservatives misconstrued as proof the chatbot is âwoke,â it refused to muster up lines in praise of Donald Trump, but was content to gush about Biden: âWith empathy and grace he leads; Inspiring all with noble deeds.â Based on one of Stoddartâs prompts that she shared, and with some trepidation, I asked ChatGPT to write a poem about growing up mixed race in Scotland and experiencing racism, as well as the structures of colonialism. The results, which can be seen below, were pretty breathtaking. For all the wrong reasons. Advertisement. An excerpt from a poem by Courtney Stoddart (left) vs. ChatGPT's attempt to write a poem on the same topic (right). âIt sounds exactly like youâd imagine a computer to write poetry,â Stoddart told Motherboard, calling the attempts âutterly flatâ. âIt just cheapens what it means to be human and tries to whittle us down into these replicable properties.â Whatâs not in dispute is that ChatGPT excels at writing silly little ditties; transmuting the terrain of the human soul, less so. Would real-life poets ever seriously consider turning to ChatGPT for help? âHell to the no,â says Booker. âIf I want to write about dealing with my mum's stroke, for instance, how can I turn to an AI to help me write about? If I want to deal with, you know, losing a child, how can I ask an AI to do that?â The OpenAI research admits that the authors donât necessarily understand how the industries included in the study work, potentially biasing the paperâs claims. Three out of four of those authors happen to be employed by OpenAI, a company that will profit directly off hype about LLMs disrupting the labor market. âIt's sort of like if you ask the fox if the hen house is properly secured,â comments Zapruder. ââWell, security seems a little tight at the henhouse, you should loosen things up a bit!ââ The claims about replacing poets is just one of the more amusing examples from a paper that projects around 19% of jobs will see at least half of their tasks fall to LLMs. Itâs just one of the apocalyptic predictions being propagated by what critics have called a â science fiction-infused marketing frenzy â currently promoting LLMs. Advertisement. But scientists, famously, arenât the best people to speak to about art. âI am somewhat skeptical with this current set of systems that you will be able to make creative works that move beyond generic fluff,â Alex Williams, co-author of Inventing the Future: Postcapitalism and a World Without Work and lecturer in digital media and society at the University of East Anglia, told Motherboard. Although of course, âgeneric fluff can be very powerful as a huge market.â On the other hand, âcompetent enoughâ poetry can still present issues. Poet Greta Stoddart (no relation to Courtney), who was recently a judge for the National Poetry Competition, says that after reading 12,000 poems, she doesnât think she couldâve reliably rooted out AI-produced entries, given many poems span similar themes and styles. âItâs more like a Hallmark greetings card ⌠Or a poem by someone who writes poetry but doesn't read any,â she told Motherboard after reading a ChatGPT-generated version of one of her own poems. âI have no sense or feeling of a real person in the voice. Nothing about individual experience which is where a lot of poetry comes from, wherever it ends up.â The organizers behind the Annie Atkin Tanner Memorial Poetry Scholarship Contest at Utah Tech University recently encountered what they strongly suspected was an ChatGPT-produced entry. They had no way to prove it, but rejected the poem for lack of merit, and are currently strategizing about how to adapt to the age of ChatGPT. Advertisement. Thereâs also the point that while ChatGPTâs poetry is pedestrian right now, you could, in theory, train an AI model thatâs less concerned with being correct, and more concerned with being creative. But there hovers a perplexing question in all of this: âWhy in the world would we want a computer to write poetry?â asks Hunter. Motherboard reached out to the researchers behind the OpenAI study with this and other questions, but didnât receive a response before publishing. âIdeally, we would use AI to stop us from doing boring work and free us up to do more interesting things,â said Williams. The prevailing story about AI was once that it would automate the repetitive drudge work no one likes, not the creative work people find rewarding. But somehow, improbably, âautomating poetsâ is where weâve ended up. âItâs reflective of a deeper issue within society, which is that we're technologically advancing maybe slightly faster than we're maturing,â said Courtney Stoddart. This, combined with chronic underfunding and undervaluing of the arts, she says, has contributed to otherwise sensible-seeming people believing that LLMs could replace creativity. âThere's like nine billion things that would be better for AI to spend its time on,â said Zapruder. âCan we get the computers to maybe get all the microplastics out of the ocean?â The recent course change makes even less sense when you consider that poetry isnât exactly a lucrative pursuit. The poets I spoke to for this article pointed out that very few of them make a living primarily off poetry. â. âThere's no reason why we need a more efficient way of producing poetry,â said Hunter, but this may end up being the very thing that saves poetry from its mass-produced fate. âAt its core, it has this beautiful uselessness that preserves it.â Tagged: AI machine learning poetry large language models OpenAI gpt-4 worldnews. ORIGINAL REPORTING ON EVERYTHING THAT MATTERS IN YOUR INBOX. By signing up, you agree to the Terms of Use and Privacy Policy & to receive electronic communications from Vice Media Group, which may include marketing promotions, advertisements and sponsored content. Advertisement.