System disruptions
There are recurring service disruptions on some of the search portals due to high traffic. We are working to resolve the issue, you may temporarily encounter an error message.
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • harvard-cite-them-right
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Beyond the gold standard: evaluating AI for subject indexing of Swedish LGBTQ + fiction
Linneaus University Faculty of Arts and Humanities, , ,.ORCID iD: 0000-0003-4169-4777
University of Borås, Faculty of Librarianship, Information, Education and IT.ORCID iD: 0000-0001-8184-8051
Gothenburg Research Infrastructure in Digital Humanities (GRIDH), University of Gothenburg , ,.ORCID iD: 0000-0003-0968-2725
2026 (English)In: Journal of Documentation, ISSN 0022-0418, E-ISSN 1758-7379, Vol. 82, no 7, p. 526-543Article in journal (Refereed) Published
Abstract [en]

Purpose This study evaluates the potential of large language models (LLMs) for subject indexing of Swedish LGBTQ + fiction using the dedicated QLIT thesaurus, based on Homosaurus. It examines both the quantitative performance and qualitative characteristics of AI-generated subject terms, with particular attention to thesaurus compliance and indexing policy. Design/methodology/approach Four general-purpose AI tools (ChatGPT, Claude, Gemini, and DeepSeek) were evaluated on a collection of 14 historical LGBTQ + literary works represented by 16 texts. Subject terms were generated from both metadata records and full texts and compared with existing indexing in the Queerlit database of Swedish LGBTQ + fiction. The metadata subset produced 518 AI-generated subject terms in total, while the full-text subset produced 590 terms. The analysis was based on a coding scheme, which was refined through several rounds of analysis until complete agreement was achieved. Findings Performance was substantially higher for metadata than for full texts, with average F1 scores of 44% and 12%, respectively. Across both datasets, fully incorrect terms constituted the largest category of generated outputs (37.5% for metadata and 56.1% for full texts), while fully correct terms accounted for only 30.0 and 8.3%, respectively. Qualitative analysis revealed recurring problems, including the assignment of broader concepts rather than, or in addition to, more specific concepts, failure to follow thesaurus definitions and indexing policies, confusion between general themes and LGBTQ + -specific themes, and the application of contemporary LGBTQ + concepts to historical texts. Poetry proved particularly challenging because themes were often implicit and open to interpretation. Although a small proportion of generated terms were judged potentially useful despite not appearing in existing indexing, no AI-generated terms were classified as fully correct additions beyond the current Queerlit metadata. Research limitations/implications The study focuses on a relatively specific collection of Swedish historical LGBTQ + fiction and a specialized controlled vocabulary. The findings nevertheless demonstrate the importance of complementing quantitative measures with qualitative evaluation and suggest that assessment of automated subject indexing should consider thesaurus definitions, indexing policies, and interpretive aspects of literary aboutness in addition to agreement with existing metadata. Practical implications The results indicate that current general-purpose LLMs are unlikely to be suitable as semi-automated indexing tools for historical LGBTQ + fiction without substantial human oversight. While AI systems may assist in identifying candidate concepts and additional access points, effective indexing continues to depend on expert knowledge of controlled vocabularies and indexing policy. Originality/value This study contributes to research on AI-assisted subject indexing by combining quantitative and qualitative evaluation of LLM-generated subject terms in a specialized LGBTQ + knowledge organization context. It demonstrates the limitations of gold-standard-based evaluation for fiction indexing and highlights the role of human expertise in applying controlled vocabularies to historically and culturally complex literary materials.

Place, publisher, year, edition, pages
Emerald Group Publishing Limited, 2026. Vol. 82, no 7, p. 526-543
Keywords [en]
Fiction, AI, LGBTQ plus, Automated subject indexing, Large language models (LLMs), QLIT, Queerlit
National Category
Information Studies
Research subject
Library and Information Science
Identifiers
URN: urn:nbn:se:hb:diva-36307DOI: 10.1108/jd-06-2026-0363ISI: 001879563200001Scopus ID: 2-s2.0-105050581510OAI: oai:DiVA.org:hb-36307DiVA, id: diva2:2107196
Funder
Linnaeus University, 2026
Note

Funding: This study was partially funded by the Faculty of Arts and Humanities at Linnaeus University through the 2026 Excellence Funding (Excellensmedel), as part of the project Queer Metadata in the Age of AI.

Available from: 2026-10-02 Created: 2026-10-02 Last updated: 2026-10-02Bibliographically approved

Open Access in DiVA

fulltext(553 kB)21 downloads
File information
File name FULLTEXT01.pdfFile size 553 kBChecksum SHA-512
fee513636c806c70f257217ec19359c68cfa5dd6d138b4950526b7752147c6d0bdc162282873fab6a5a2f5693198f4bc45a73f13fe5acbe8900d1f80f3242454
Type fulltextMimetype application/pdf

Other links

Publisher's full textScopus

Authority records

Falk, Olof

Search in DiVA

By author/editor
Golub, KoraljkaFalk, OlofHumlesjö, Siska
By organisation
Faculty of Librarianship, Information, Education and IT
In the same journal
Journal of Documentation
Information Studies

Search outside of DiVA

GoogleGoogle Scholar
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 406 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • harvard-cite-them-right
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf