Abstract
Prompt learning is a new paradigm in the Natural Language Processing (NLP) field which has shown impressive performance on a number of natural language tasks with common benchmarking text datasets in full, few-shot, and zero-shot train-evaluation setups. Recently, it has even been observed that large but frozen pre-trained language models (PLMs) with prompt learning outperform smaller but fine-tuned models. However, as with many recent NLP trends, the performance of even the largest PLMs such as GPT-3 do not perform well on specialized domains (e.g. medical text), and the common practice to achieve State of the Art (SoTA) results still consists of pre-training and fine-tuning the PLMs on downstream tasks. The reliance on fine-tuning large PLMs is problematic in clinical settings where data is often held in non-GPU environments, and more resource efficient methods of training specialized domain models is crucial. We investigated the viability of prompt learning on clinically meaningful decision tasks and directly compared with more traditional fine-tuning methods. Results are partially in line with the prompt learning literature, with prompt learning able to match or improve on traditional fine-tuning with substantially fewer trainable parameters and requiring less training data. We argue that prompt learning therefore provides lower computational resource costs applicable to clinical settings, that can serve as an alternative to fine-tuning ever increasing in size PLMs.
Niall Taylor
DPhil Student
Niall is a DPhil candidate at the Big Data Institute at the University of Oxford. He contributes his expertise with NLP for clinical data to the chronosig project.
Dan W Joyce
Clinical Research Fellow
My research explores how computational methods can be used to improve personalisation of care for patients with mental illness
Alejo J Nevado-Holgado
Associate Professor
I am an Associate Professor of the Department of Psychiatry and the Big Data Institute, and part of Dementia Research Oxford. I am very glad to supervise the AI team in the TNDR, formed by 10 excellent machine learners and bioinformaticians. Our focus is on the applications of machine learning and bioinformatics to mental health care. In addition, I also hold a position at the Big Data Institute, where we collaborate in the application of machine learning to genomics and target discovery. I am also consultant to a number of AI companies.
Andrey Kormilitzin
Senior Researcher
My research is centred around translating advances in mathematics, statistical machine learning and deep learning to address challenges involved in learning, inference and ethical decision making using complex biomedical and health data.