Posts and Updates – www.talklitmining.ugent.be

GSA Montreal (Panel: Digital Humanities)

We were invited by the DH chapter of the German Studies Association to contribute to “Integrating Digital Humanities and German Studies”. So we participated in a DH-linked panel “Digital Humanities: Literature, Translation, Methods” during GSA Montreal in October 2023. Interesting fundamental discussions with Andrew Piper, hosted by Fabian Offert (USCB) and Thorsten Ries (UTexas at Austin).

IGEL 2023 (Monopoli, Italy)

We participated in IGEL 2023, you can find the prerecorded “short talk” here:

Link to the conference website: “Comparing Professional and Platform-driven Critics: A Digital and Sociological Analysis of Literary Gatekeeping and Gatewatching“

CLIN33 Conference (22/09/2023): Poster on “Aspect-Based Sentiment Analysis for Literary Criticism: Experts vs. Social Critics on Literary Prizes”

On the 22nd of September, Prof. Gunther Martens discussed our research during the CLIN33 conference at the University of Antwerp. CLIN33 represents the 33rd Meeting of Computational Linguistics in The Netherlands, which is organised by the Centre for Computational Linguistics, Psycholinguistics and Sociolinguistics (CLiPS) and University of Antwerp.

You can find the abstract as well as our poster below. For a more detailed look at the poster in better resolution, please consult this: CliN_2023_Poster.

Abstract

Generative pre-trained transformers continue to be the talk of the town, but have been met with far more limited enthusiasm within NLP research. Bracketing the admittedly vexed question whether it is feasible and/or ethical to use sparsely documented proprietary models, this contribution aims to assess the relative merits and drawbacks of LLM by pitching ChatGPT’s API against our in-house Transformers-based architecture developed for the specific task of ABSA (Aspect-Based Sentiment Analysis) in the domain of literary criticism. (De Greve et al 2021)

The increase in the number of book reviews created by users has presented unparalleled prospects for empirical investigations on books, reading habits, and audience participation, jumpstarting various recent rule-based and transformer-based approaches in empirical literary studies and cultural analytics. (Boot 2023; Salgaro 2023) Despite this flurry of attention, the process of domain adaptation continues to be time-consuming and challenging. Recently various studies have aimed to explore whether large language models (LLM) may provide zero-shot or few-shot alternatives. They have arrived at very mixed results, ranging from abysmal (Chen et al. 2023) to an assessment of performing on average 25% worse than SOTA models (Kocoń et al 2023). Specifically with regard to emotion and sentiment extraction, others have been more hopeful (Rathje 2023). While on “subtasks of Relation Extraction and Event Extraction GPT models may “rarely exceed 30% of SOTA”, Han et al (2023) have noticed that “almost all sub-tasks of ABSA can reach more than 50% of SOTA” via plain zero-shot prompting. While we cannot evaluate performance on the extended range of tasks employed in the cited research, we will be able to compare results generated by ChatGPT’s 3.5 Turbo API with our BERT-based annotations and with our manual annotations for various subtasks of ABSA. In a second step, it will be explored how few-shot prompting affects performance and to what extent our architecture may profit from the ability to generate fully annotated synthetic training data. Finally, it will be considered to what extent the generative model’s potential for steadily growing context windows, for “chaining” subtasks and for multimodality opens up venues for potential future research.

Cited references

Boot, Peter, ‘“A Pretty Sublime Mix of WTF and OMG”. Four Explorations into the Practice of Evaluation on Online Book Reviewing Platforms’, Journal of Cultural Analytics, 7.2 (2023) <https://doi.org/10.22148/001c.68086>

Chen, Xuanting, Junjie Ye, Can Zu, Nuo Xu, Rui Zheng, Minlong Peng, and others, ‘How Robust Is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Tasks’ (arXiv, 2023) <https://doi.org/10.48550/arXiv.2303.00293>

De Greve, Lore, Pranaydeep Singh, Cynthia Van Hee, Els Lefever, and Gunther Martens, ‘Aspect-Based Sentiment Analysis for German : Analyzing “talk of Literature” Surrounding Literary Prizes on Social Media’, COMPUTATIONAL LINGUISTICS IN THE NETHERLANDS JOURNAL, 11 (2021), 85–104

Han, Ridong, Tao Peng, Chaohao Yang, Benyou Wang, Lu Liu, and Xiang Wan, ‘Is Information Extraction Solved by ChatGPT? An Analysis of Performance, Evaluation Criteria, Robustness and Errors’ (arXiv, 2023) <https://doi.org/10.48550/arXiv.2305.14450>

Kocoń, Jan, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, and others, ‘ChatGPT: Jack of All Trades, Master of None’ (arXiv, 2023) <http://arxiv.org/abs/2302.10724> [accessed 19 May 2023]

Rathje, Steve, Dan-Mircea Mirea, Ilia Sucholutsky, Raja Marjieh, Claire Robertson, and Jay J. Van Bavel, ‘GPT Is an Effective Tool for Multilingual Psychological Text Analysis’ (PsyArXiv, 2023) <https://doi.org/10.31234/osf.io/sekf5>

Salgaro, Massimo, Stylistics, Stylometry and Sentiment Analysis in German Studies: The Operationalization of Literary Values (Göttingen: V&R unipress, 2023)

Zhu, Yiming, Peixian Zhang, Ehsan-Ul Haq, Pan Hui, and Gareth Tyson, ‘Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks’ (arXiv, 2023) <http://arxiv.org/abs/2304.10145> [accessed 19 May 2023]

By The Book 8 Conference (21-22/06/2023): Conference Paper on “‘So much for literary democracy’: Journalistic Capital and the Purpose of Scandal Surrounding Literary Prizes on Social Media”

This year, I presented at the By The Book Conference for the first time on the 21st of June. The conference was co-organised by the University of Porto, where it took place, and the Oxford Brookes University. It’s an initiative of the European Publishing Studies Association (EuroPub) which aims to “foster knowledge exchange around the contemporary book trade”. The conference contributions focus on the book publishing industry and the field of publishing studies in general.

My conference contribution focused on the influence and impact of scandal on literary prizes dependent on journalistic capital, examining the 2012 scandal surrounding the Not The Booker Prize and the Twitter-discourse.

For more information on the conference and the program, please consult the conference website.

“So much for literary democracy”: Journalistic Capital and the The Influence and Reception of Scandal Surrounding Literary Prizes on Social Media

Literary prizes, increasingly topic of research (e.g Moser 2004, English 2009, Leinen 2010, Ducas 2013, Röhricht 2016, Sapiro 2016, Auguscik 2017, Childress et al. 2017, Kennedy-Karpat and Sandberg 2017, Rahmann 2017, Mosely 2019), function as traditional literary gatekeepers and are responsible for the institutionalised consecration of literary texts. Contemplating Bourdieu’s field of cultural production (1983) literary prizes are frequently discussed in terms of economic and symbolic capital, but they can also bestow visibility. James English thus proposes journalistic capital, which may accelerate the accumulation of symbolic capital (2002, p. 115) and represents “the marketability of the award and its winners” (Squires 2013, p. 293).

Scandal is hardly unusual (English 2002, p. 113), though valuable, established and prestigious prizes (e.g. Booker Prize) are generally able to weather public opinion. But what challenge does bad publicity pose for a prize which relies almost exclusively on journalistic capital? As a case study, I shall examine the 2012 edition of the Not The Booker Prize (NTBP), when the behaviour and back-room negotiations of a contender scandalised organisers and audience alike. The NTBP was created by left-wing The Guardian in 2009 as a democratic literary counter-prize. Officially, winners receive nothing but a “Guardian”-mug. Although relatively low in both symbolic and economic capital, because the newspaper is able to use its own platform, the prize generates a lot of media attention and therefore predominantly awards journalistic capital. Besides nominating and voting in the newspaper’s comment section, the audience also discusses the prize on Twitter. Within this context, I aim to study the figure of the layperson critic as an additional literary gatekeeper on social media platforms. For this, I will analyse the tweets created in the immediate wake of the scandal to gain insight into the content of the user-generated literary criticism.

Bibliography:

Auguscik, Anna. Prizing Debate: The Fourth Decade of the Booker Prize and the Contemporary Novel in the UK. Transcript, 2017.

Bourdieu, Pierre. ‘The Field of Cultural Production, or: The Economic World Reversed’. Poetics, vol. 12, no. 4, 1983, pp. 311–56.

Childress, Clayton, et al. “Publishers, Authors, and Texts: The Process of Cultural Consecration in Prize Evaluation.” Poetics, vol. 60, Feb. 2017, pp. 48–61.

Ducas, Sylvie. La littérature à quel(s) prix? Histoire des prix littéraires. La Découverte, 2013.

English, James F. “Winning the Culture Game: Prizes, Awards, and the Rules of Art.” New Literary History, vol. 33, no. 1, Johns Hopkins University Press, 2002, pp. 109–135.

English, James F. The Economy of Prestige: Prizes, Awards, and the Circulation of Cultural Value. Harvard University Press, 2009.

Kennedy-Karpat, Colleen, and Eric Sandberg. Adaptation, Awards Culture, and the Value of Prestige. Springer Berlin Heidelberg, 2017.

Leinen, Angela. Wie man den Bachmannpreis gewinnt. Gebrauchsanweisung zum Lesen und Schreiben. Heyne, 2010.

Moseley, Merritt. ‘How the Booker Prize Won the Prize’. American, British and Canadian Studies, vol. 33, Dec. 2019, pp. 206–21.

Moser, Doris. Der Ingeborg-Bachmann-Preis. Börse, Show, Event. Böhlau Verlag, 2004.

Pickford, Susan. “The Booker Prize and the Prix Goncourt. A Case Study of Award-Winning Novels in Translation.” Book History, vol. 14, 2011, pp. 221–240.

Rahmann, Kathrin. Von der Wirkung zur Wertung. Formal-ästhetische Werte in den Diskussionen des Ingeborg-Bachmann-Wettbewerbs 1999-2009. 2017. Georg-August-Universität Göttingen, PhD dissertation.

Röhricht, Karin. Wettlesen um den Ingeborg-Bachmann-Preis. Korpusanalyse der Anthologie Klagenfurter Texte (1977-2011). Studienverlag, 2016.

Sapiro, Gisèle. “The Metamorphosis of Modes of Consecration in the Literary Field: Academies, Literary Prizes, Festivals.” Poetics, vol. 59, Dec. 2016, pp. 5–19.

Squires, Claire. ‘Literary Prizes and Awards’. A Companion to Creative Writing, edited by Graeme Harper, John Wiley & Sons, Ltd, 2013, pp. 291–303.

Vernetzungskonferenz “Brückenräume” (08-09/06/2023): Research Project Presentation

On the 8th and 9th of June, the Department of German Literature participated in a conference at the University of Regensburg, which aimed at strengthening the ties between the departments of German Literature at the University of Regensburg, Ghent and Brno as well as to create the possibility of future research collaborations. The name of the conference was “Brückenräume: aktuelle Forschungsperspektiven der In- und Auslandsgermanistik in Brünn, Gent und Regensburg” and I presented and discussed my research project and approach.

DH Benelux (31/05-2/06/2023): Presentation “Large Language Models in the Humanities: magic shortcut or just beating about the bots? #nocode #alldata”

This summer, we presented at the 10th edition of the DH Benelux conference, which took place at the Royal Library of Belgium (KBR) from the 31st of May until the 2nd of June. During our presentation we discussed the results of our semi-supervised learning system, which was trained ofr aspect-based analysis tasks in German concerning literary criticism on social media and compared them to the possibilities provided by new developments such as ChatGPT.

For more information, please consult the conference website.

Large Language Models in the Humanities: magic shortcut or just beating about the bots? #nocode #alldata

Following up on our earlier contributions on doing Aspect-Category-Opinion-Sentiment Quadruple Extraction, in this talk, we aim to weigh some of the tried and trusted methods of doing NLP against the new kid on the block, namely LLM (Large Language Models) trained for various purposes and languages. Few-shot and zero-shot approaches may seem adventurous, and they indeed may not live up to the state-of-the-art achieved with regard to our specific use-case, i.e. aspect-based sentiment analyses (ABSA) in specific domains such as laptops and restaurants. But what if e.g. your target vocabulary is out of domain (OOD) by principle? What if your data is largely unstructured full-text? Compared to the time-consuming task of generating annotations, using GPT-3 turns out to yield surprisingly good results for a variety of tasks like NER and ABSA. We will show how to “escape” the chatbox and feed data in bulk to the model. We aim to discuss whether this is a viable solution, especially for the aspiring DH researcher with limited programming skills, limited GPU access, and/or without persistent support from teams dedicated to humanities-style qualitative research questions. We will present some of the tasks that have become possible over the course of conducting our study of multilingual sentiment in a specific multimodal setting, such as: automatic transcription of oral sources, alignment of oral and written data, speaker diarization, automatic generation of training data with annotations, etc. We will illustrate the major advantages of API-based solutions for classification, such as the small amount of training data needed to fine-tune the model, and the ease of access. All in all, this type of modular pipeline is a less prohibitive pathway for aspiring researchers, compared to setting up a demanding and complicated environment that is typically required in traditional NLP methods. In addition, allowing people to “chat with your data” will inevitably become a new way of disseminating research results in a non-directive way, also in domains where access to the actual artefacts might be marred by copyright. This workshop is not about presenting results or final conclusions, but aims to provide an opportunity to discuss what some consider to be a watershed moment, while others remain sceptical.

CIMRH 2022 Symposium (15/09/2022): Regular Project Presentation

In August, I was invited by Dr. Anna Klamet (London College of Communication) to participate in the online CIMRH Symposium on “Innovation in Publishing”. At this Symposium, which took place on the 15th of September (2022), I presented a brief overview of my FWO-funded research project. I outlined its premesis, summarised the employed methodology and highlighted several results. My presentation thus focused mostly on the reception of a literary work by reader-reviewers after publication and the influence and role of social media in this process.

For more information on the symposium and the participants, please consult the symposium website.

SHARP 2022 Conference (11-15/07/2022): Panel Paper on “‘Scandal sells books’: An Analysis of the Reception and Perception of Scandals Surrounding Literary Prizes on Social Media.”

In July, I once again participated in the SHARP Conference (https://sharp2022.nl/) and this year’s focal point was the “power of the written word”. I presented a paper on scandals in the context of two literary prizes, namely the (Man) Booker Prize as well as the Not the Booker Prize and the reception of these scandals on Twitter. The title of this individual paper was “‘Scandal sells books’: An Analysis of the Reception and Perception of Scandals Surrounding Literary Prizes on Social Media.”

This paper was originally scheduled to be part of the panel “The public power of the book”, in which Henriët Graafland was to present her research as well. However, due to organisational challenges, the panels had to be split up. I was then invited to join the panel on “Literary Institutions”, which was chaired by Prof. Laura Dietz. This granted me the wonderful opportunity to present my research alongside Prof. Simone Murray and Lisa Gitelman, who discussed “How Literary Studies Came to Ignore Books: Choices in Institutionalising English” and “Citation and Mediation: A Reading of MLA Style” respectively.

Abstract: “Scandal sells books”: An Analysis of the Reception and Perception of Scandals Surrounding Literary Prizes on Social Media

Due to their role as traditional gatekeepers in the field of literary criticism, literary prizes are responsible for the institutionalised consecration of literary texts. They are increasingly topic of research (e.g Moser 2004, English 2005, Leinen 2010, Ducas 2013, Röhricht 2016, Sapiro 2016, Auguscik 2017, Childress et al. 2017, Kennedy-Karpat and Sandberg 2017, Rahmann 2017). The general discourse surrounding prizes, however, tends to be negative (cf. English 2005, p. 187) and they are frequently subject of scandals. Nevertheless, scandal is “the instrument par excellence of symbolic action” (Bourdieu p. 84) and English argues that “scandal […] is constitutive of prizes” (English 2002, p. 113). Scandal is consequently inherent to literary prizes.

In this presentation, I shall examine two scandals surrounding the Booker Prize and the Not the Booker Prize, respectively. The Booker Prize is no stranger to scandal, in fact its success relies on it (English 2002, p. 113) and Drabble states that the prize “has failed in its task of stimulating public interest” if it is not subjected to “bitter controversy and scandal” (p. 250). The case study in question is the 2013 scandal concerning the expanded nomination criteria, which allow any novel written in English and published in the UK or Ireland to qualify. Although its counter-prize is hardly as well-known and despite its goal to challenge the Booker Prize’s obscure nomination and judging practices, the Not the Booker has also known controversy. I shall take a closer look at the 2012 edition and the scandal pertaining to Ewan Morrison’s leaked e-mail, in which he lied about the competition and criticized the voting system, his refusal to withdraw, as well as the reactions of the organisers. Within this context, I thus aim to study the emerging and evolving figure of the layperson critic as an additional literary gatekeeper on social media platforms. For this, I will analyse the tweets that were created in the immediate wake of these scandals in order to gain insight into the content of the user-generated literary criticism created by new layperson gatekeepers regarding traditional gatekeepers.

Bibliography:

Auguscik, Anna. Prizing Debate: The Fourth Decade of the Booker Prize and the Contemporary Novel in the UK. Transcript, 2017.

Bourdieu, Pierre, and Hans Haacke. Free Exchange. Polity Press, 1995.

Childress, Clayton, et al. “Publishers, Authors, and Texts: The Process of Cultural Consecration in Prize Evaluation.” Poetics, vol. 60, Feb. 2017, pp. 48–61.

Drabble, Margaret. “Pleasure and Prestige. A Writer Reflects on the Prize System.” British Book News, April 1989, pp. 250-251.