What are you looking for?
Birds of a Feather (BoF) Session April 21, 2025

Documenting LLM Interactions throughout the Research Data lifecycle

Plenary: RDA 25th Plenary Meeting [part of International Data Week 2025]

Submitted by
Brian Ballsun-Stanton
Recording
Slide

Meeting objectives

Collaborative notes https://docs.google.com/document/d/1CfH9Dl

 

This Birds of a Feather session aims to establish frameworks for documenting, sharing, and standardizing Large Language Model (LLM) interactions within the research data lifecycle. As LLMs become integral to research processes across disciplines, the prompts and interactions themselves constitute valuable data artifacts that warrant standardization and preservation. Our objectives are to:

  1. Frame LLM prompting part of documentation of research methodologies, thereby requiring documentation standards comparable to code, protocols, and other methodology components, enabling transparency and reproducibility
  2. Address the current lack of norms, formats, and tools for documenting AI use in research by establishing foundational principles for an “LLM interaction standard” that aligns with FAIR data principles
  3. Explore methods for using AI to create, improve, and annotate existing datasets in reliable and reproducible ways.
  4. Develop preliminary specifications for formats and tools (such as bookmarklets or browser extensions) that could capture LLM interactions as structured, shareable research objects
  5. Explore opportunities to extend existing frameworks (like RO-Crate) to incorporate LLM interaction documentation, and identify potential repository solutions for a community-maintained “grimoire” of research prompts
  6. Initiate cross-disciplinary discussions that will form the basis for potential working groups focused on technical standards, ethical norms, and disciplinary best practices for LLM use throughout the research data lifecycle

This BoF distinguishes itself from existing RDA groups by specifically addressing the emerging need to treat researcher-LLM interactions as data artifacts worthy of standardization, documentation, and sharing within the open science ecosystem. We would be delighted to merge with any other BoFs on related AI topics.

Meeting presenters

Brian Ballsun-Stanton

Meeting agenda

  1. Introduction and Problem Statement (10 minutes)

    • Framing prompting as methodology
    • Current fragmented landscape of AI use disclosure
    • The concept of “research grimoires” in the scholarly context
  2. Current Challenges in Documenting LLM Research Use (15 minutes)

    • Missing standards for prompt documentation
    • Inconsistent publisher/journal requirements
    • Technical barriers to capturing interactions
    • Ethical considerations in sharing prompts
  3. Interactive Discussion: Requirements Gathering (20 minutes)

    • What elements of LLM interactions need standardization?
    • Which existing standards might be extended?
    • What tools would facilitate documentation?
  4. Formation of Potential Working Themes (30 minutes)

    • Breakout discussions on key areas of interest
    • Identifying champions for potential working groups
    • Documenting shared priorities across themes
  5. Next Steps and Action Items (15 minutes)

    • Synthesizing outcomes
    • Establishing communication channels
    • Timeline for follow-up activities

Have you presented a session on the same topic at any previous plenaries?

No

Additional links to informative material

Meeting takeaways

Describe what your session was about in three sentences maximum:

  • As researchers integrate LLMs into their workflows through prompting, fine-tuning, and other interactions, there are no established standards for documenting this work to ensure transparency and reproducibility. 
  • Participants broke into groups to discuss challenges around standards, publisher requirements, ethics, and technical implementation. 
  • Discussions covered both researcher-facing documentation practices and technical infrastructure needs around engaging with and producing methodological process documentation.
Highlight a maximum of five key outcomes/actions/takeaways of your session:

  1. We identified people who wanted to meet during the conference and establish the terms of reference for an interest group.
  2. Established initial frameworks including the three P’s (Prompt, Process, Provenance) and treating LLM interactions as methods documentation
  3. Surfaced cultural differences across disciplines about what constitutes legitimate AI use and requires documentation
  4. Identified tension between comprehensive documentation needs and researcher burden when AI use is iterative (hundreds/thousands of prompts per paper)
  5. Recognized that AI is now embedded in many systems (spell-checkers to research tools), making it challenging to define documentation thresholds across disciplines

References

 

Multiple models were used by Ballsun-Stanton and Ross in the production of this.

Estimate of the required venue room capacity

30-50

Applicable Pathways

AI meets data: exploring use cases, applications and innovation

Please indicate at least (3) three breakout slots that would suit your meeting.

Breakout 1. Monday, 13 October 2025, 01:30-03:00 UTC
Breakout 2. Monday, 13 October 2025, 06:00-07:30 UTC
Breakout 3. Wednesday, 15 October 2025, 01:30-03:00 UTC