#AUTOPROMPT SKIP PROMPT HOW TO#
Furthermore, for each task, we optimize a mixture of prompts, learning which prompts are most effective and how to ensemble them. Our prompts consist of “soft words,” i.e., continuous vectors that are not necessarily word type embeddings from the language model. However, where does this prompt come from? We explore the idea of learning prompts by gradient descent-either fine-tuning prompts taken from previous work, or starting from random initialization. For example, language models retain factual knowledge from their training corpora that can be extracted by asking them to “fill in the blank” in a sentential prompt. On a diverse set of tasks, this contextual calibration procedure substantially improves GPT-3 and GPT-2's average accuracy (up to 30.0% absolute) and reduces variance across different choices of the prompt.read more read lessĪbstract: Natural-language prompts have recently been used to coax pretrained language models into performing other AI tasks, using a fill-in-the-blank paradigm (Petroni et al., 2019) or a few-shot extrapolation paradigm (Brown et al., 2020). We then fit calibration parameters that cause the prediction for this input to be uniform across answers. To mitigate this, we first estimate the model's bias towards each answer by asking for its prediction when given the training prompt and a content-free test input such as "N/A". We demonstrate that this instability arises from the bias of language models towards predicting certain answers, e.g., those that are placed near the end of the prompt or are common in the pre-training data. We show that this type of few-shot learning can be unstable: the choice of prompt format, training examples, and even the order of the training examples can cause accuracy to vary from near chance to near state-of-the-art.
These results demonstrate that automatically generated prompts are a viable parameter-free alternative to existing probing methods, and as pretrained LMs become more sophisticated and capable, potentially a replacement for finetuning.read more read lessĪbstract: GPT-3 can perform numerous tasks when provided a natural language prompt that contains a few training examples. We also show that our prompts elicit more accurate factual knowledge from MLMs than the manually created prompts on the LAMA benchmark, and that MLMs can be used as relation extractors more effectively than supervised relation extraction models. Using AutoPrompt, we show that masked language models (MLMs) have an inherent capability to perform sentiment analysis and natural language inference without additional parameters or finetuning, sometimes achieving performance on par with recent state-of-the-art supervised models. To address this, we develop AutoPrompt, an automated method to create prompts for a diverse set of tasks, based on a gradient-guided search.
#AUTOPROMPT SKIP PROMPT MANUAL#
Reformulating tasks as fill-in-the-blanks problems (e.g., cloze tests) is a natural approach for gauging such knowledge, however, its usage is limited by the manual effort and guesswork required to write suitable prompts. Abstract: The remarkable success of pretrained language models has motivated the study of what kinds of knowledge these models learn during pretraining.