13 FEB 2024
Submitted by Francis P. Crawley
Meeting objectives:
The main objective of this working meeting is to examine the need for a transformation on how we value science and measure its contribution to research, education, and society generally. In February of 2024 the Coalition for the Advancement of Research Assessment (CoARA) announced a new Working Group on Ethics and Research Integrity Policy in Responsible Research Assessment for Data and Artificial Intelligence (ERIP) promoted by the European Commission, ALLEA and leading European and international research institutions, including RDA and RDA-Europe. This opens up the opportunity for greater RDA involvement in ERIP and research assessment generally as it relates to data and AI.
This session builds on the AIDV-WG’s ELSI and governance outputs and relates them to the need for the integration of research ethics and research integrity into digital tools for the establishment of policy and governance in the evaluation of scientific research. In the framework of ERIP, the session examines three key trajectories regarding the implementation of machine learning methods and artificial intelligence models for expanding traditional understandings of participation in, and contributions to, scientific outputs and communication. In particular, it looks at the need to develop the following areas:
- methods and tools to ensure the research ethics and integrity of scientific methods and outputs with the advancing use of data and the impact of AI;
- methods and tools to evaluate digital contributions to science/knowledge in research programs and assessment procedures; and
- innovative methodologies for employing data ecosystems and AI models for research assessment in digital environments with a focus on open science infrastructures.
The session will demonstrate how data and AI governance, policy, and guidance can be integrated into digital tools for advancing research assessment that promote the role of, and define the ethical and integrity characteristics of, a responsible culture for the assessment of data and AI in research, fostering responsibility, transparency, and societal benefit. It will look at the relationship between research assessment policy and data and AI tools for developing new indicators and metrics in evaluating the contributions of science to the academic and research communities as well as society as a whole.
Meeting agenda:
Draft Meeting Agenda, speakers are identified and will be later confirmed.
| Time | Title and Speaker(s) |
| 00:00 | Opening Remarks from the Session ChairpersonFrancis P. Crawley Co-chair, CoARA’s Working Group on Ethics and Research Integrity Policy in Responsible Research Assessment for Data and Artificial Intelligence (ERIP) |
| 00:05 | Perspectives on the need for revising research evaluationTwo early career researchers A global perspective from Europe A local perspective from Africa |
| 00:15 | Developing responsible ethics and research integrity policy in research evaluationA perspectives from a national endeavor to transform research in Turkey A perspectives from an international endeavor to transform research in Latin America |
| 00:25 | A first interactive discussion with the audience on the need for revising research assessment with regard to data and AI |
| 00:40 | Various proposals on tools for reforming research assessment From an open science approach From an industry approach From a cross-disciplinary approach |
| 00:55 | A second interactive discussion with the audience on pathways for integrating AI governance models into research assessment reform |
| 00:20 | Summary of the Session |
| 01:30 | Close of the Working Session |
Type of Meeting:
Working meeting
Short introduction describing any previous activities:
Higher education institutions, academic/scientific journals, and generally all institutions involved in scientific research as well as the academics/scientists themselves share responsibility for ensuring adequate scientific and ethical standards in academic authorship, scientific integrity, and the production of knowledge.
The CoARA ERIP Working Group was developed in the context of the RDA Plenary 21 in Strasbourg through informal meetings and interactions with RDA members. It applies the deliverables of the AIDV-WG to ELSI, policy, and governance models where decisions regarding the use of data and AI in scientific research and its outputs, including publications, are based on well defined roles, uses, and attributions of these new technologies to the development of the sciences, their uses, and their communication. Decisions to employ new AI technologies must be supported by an adequate understanding of their impact on the scientific method, scientific processes, and the results generated. In this context, questions can be posed concerning the use of AI in scientific research, its outputs, and its publications. ERIP examines the primary and fundamental values for the use of AI in the scientific research and publication process with a focus on transparency, honesty, and diligent care. These three values and the principles derived from them provide the necessary ethical framework for the use of AI in the scientific field. ERIP will work with the RDA community as well as other leading scientific and academic groups, and European as well as UN and other (inter-)governmental organisations to develop policy as well as digital tools that can realise that policy for the reliable scientific evaluation of data and AI.
BoF applicant serving as contact person:
Additional links to informative material:
- A Pathway towards Multidimensional Academic Careers: A LERU Framework for the Assessment of Researchers (Prof. Bert Overlaet, LERU position paper, January 2022)
- A Science|Business Special Report. Confidence in Science: How to ensure sustainable and trustworthy channels of scientific information? (July 2023) https://sciencebusiness.net/system/files/reports/SB-TRUSTWORTHY_0.pdf
- ACOLA Report. Research Assessment in Australia: Evidence for Modernisation. Australian Council of Learned Academies (2023)
- AI in Education: Enhancing Learning or Diminishing Reliability?
- AI, Machine Learning & Big Data Laws and Regulations 2023 | India
- Alberts B., Kirschner M. W., Tilghman S., Varmus H. (2014). Rescuing US biomedical research from its systemic flaws. Proceedings of the National Academy of Sciences of the United States of America, 111(16), 5773–5777.
- Alberts, Bruce, Marc W. Kirschner, Shirley Tilghman, and Harold Varmus. 2014. “Rescuing US biomedical research from its systemic flaws.” Proceedings of the National Academy of Sciences 111(16): 5773–77. (April 16, 2014).
- ALLEA European Code of Conduct for Research Integrity (revised edition 2023)
- Ancion, Zoé, Borrell-Damián, Lidia, Mounier, Pierre, Rooryck, Johan, and Saenen, Bregt. ‘Action Plan for Diamond Open Access’ (March 2022)
- Are numerical scores important for grant proposals’ evaluation? A cross sectional study [version 1; peer review: awaiting peer review]
- Assessing Ethics Education in Science and Engineering, Special Collection. Science and Engineering Ethics. Forthcoming.
- “Australian Council of Learned Academies (ACOLA). Research Assessment in Australia:
- Evidence for Modernisation (November 2023).”
- Bakiner, O. What do academics say about artificial intelligence ethics? An overview of the scholarship. AI Ethics 3, 513–525 (2023). https://doi.org/10.1007/s43681-022-00182-4
- Banks, G.C., Rogelberg, S.G., Woznyj, H.M. et al. Editorial: Evidence on Questionable Research Practices: The Good, the Bad, and the Ugly. J Bus Psychol 31, 323–338 (2016). https://doi.org/10.1007/s10869-016-9456-7
- Brey, P., Dainow, B. Ethics by design for artificial intelligence. AI Ethics (2023). https://doi.org/10.1007/s43681-023-00330-4
- BRIDGE2HE H2020 Project 101005071. Guiding notes to use the TRL self-assessment tool. No date.
- Bringula, R. What do academics have to say about ChatGPT? A text mining analytics on the discussions regarding ChatGPT on research writing. AI Ethics (2023). https://doi.org/10.1007/s43681-023-00354-w
- Capetown Statement Working Group. The Cape Town Statement on Fostering Research Integrity through Fairness and Equity (May 2019)
- CESAER: Conference of European Schools for Advanced Engineering Education and Research: ‘Keeping science open? Current challenges in the day-to-day reality of universities (White paper 18 October 2023)
- CHAI: Center for Human-Compatible AI ‘Coordinated global action on AI safety research and governance is critical to prevent uncontrolled frontier AI development from posing unacceptable risks to humanity’ (Ditchley Park, United Kingdom, 31 October 2023)
- Charisi, V., Chaudron, S., Di Gioia, R., Vuorikari, R., Escobar Planas, M., Sanchez Martin, J.I. and Gomez Gutierrez, E., Artificial Intelligence and the Rights of the Child : Towards an Integrated Agenda for Research and Policy, EUR 31048 EN, Publications Office of the European Union, Luxembourg, 2022, ISBN 978-92-76-51837-2, doi:10.2760/012329, JRC127564.
- “ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations
- https://www.frontiersin.org/articles/10.3389/frai.2023.1169595/full“
- Chaudhry, M.A., Kazim, E. Artificial Intelligence in Education (AIEd): a high-level academic and industry note 2021. AI Ethics 2, 157–165 (2022). https://doi.org/10.1007/s43681-021-00074-z
- Chaudhry, M.A., Kazim, E. Artificial Intelligence in Education (AIEd): a high-level academic and industry note 2021. AI Ethics 2, 157–165 (2022). https://doi.org/10.1007/s43681-021-00074-z
- Checco A., Bracciale L., Loreti P., Pinfield S., Bianchi G. (2021). AI-assisted peer review. Humanities and Social Sciences Communications, 8(1). https://doi.org/10.1057/s41599-020-00703-8
- Chiarelli, A., Johnson, R., & Loffreda, L. (2022). Discussion Document – Indicators of Research Integrity: An initial exploration of the landscape, opportunities and challenges. Zenodo. https://doi.org/10.5281/zenodo.7115562
- COAR & SPARC. Good Practice Principles for Scholarly Communication Services (2019).
- CoARA: Agreement on Reforming Research Assessment (20 July 2022)
Meeting presenters:
Experts in research evaluation, data and AI ethics and governance, and leaders in local and global perspectives related to research evaluation; the session will engage the audience in questions of policy and tools needed for the future of research evaluat
Avoid conflict with the following group (1):
Avoid conflict with the following group (2):
Contact for group (email):
Applicable Pathways:
Data Infrastructures – Organisational to Environments
Driven by RDA Organisational Member:
No
Please indicate at least (3) three breakout slots that would suit your meeting.:
- Breakout 4
- Breakout 7
- Breakout 17
Please indicate a minimum of (3) three breakout slot (s) that would suit your repeat session in a different time zone.:
- Breakout 4
- Breakout 7
- Breakout 17
Are you willing to host a second, repeat session in a different time zone?:
Yes
Have you previously held a session at plenaries?:
Yes