What Is RLHF? How Human Feedback Shapes AI Chatbots

Reinforcement learning from human feedback, or RLHF, is a training method in which people rank a language model's answers and the model is then adjusted to produce answers that rank higher. It is one of the steps that turns a raw text-predicting model into an assistant that follows instructions. This article explains the steps described by Hugging Face and the research paper behind InstructGPT.

Why a plain language model is not enough

A model trained only to predict the next word learns patterns from huge amounts of text. That does not make it good at doing what a person asks. Asking it a question may just produce more questions or an unhelpful continuation. RLHF adds a signal about what people actually want.

Step 1: start with a pretrained model

Hugging Face explains that the process begins with a language model that is already pretrained on general text and can respond to varied instructions. Optionally it is fine-tuned first on carefully written examples. The authors say no single starting model is clearly the best choice.

Step 2: build a reward model

Next, people rank several model outputs for the same prompt. Hugging Face notes that ranking is used instead of giving each answer a score because raw scores are noisy and not calibrated between people. The rankings are converted into a reward model, a second model that reads a piece of text and returns a single number showing how well people would likely rate it. Its size is typically similar to the language model it judges.

Step 3: tune with reinforcement learning

A copy of the original model becomes the policy, the model being improved. For each prompt it writes text, the reward model scores it, and an algorithm called Proximal Policy Optimization nudges the policy toward higher scores. A penalty measures how far the policy's word probabilities drift from the original model. Without it, Hugging Face says, the model can produce nonsense that still fools the reward model.

• Policy writes an answer to a prompt
• Reward model scores the answer
• Penalty keeps the policy close to the original
• The policy is updated, and the cycle repeats over many prompts

What InstructGPT showed

The InstructGPT paper from OpenAI researchers, posted on arXiv, fine-tuned GPT-3 in two stages: supervised learning on writers' demonstrations, using prompts written by labelers and prompts submitted through the OpenAI API, then RLHF using human rankings. In human evaluations on their prompts, raters preferred outputs from the 1.3 billion parameter InstructGPT over those of the 175 billion parameter GPT-3, though the first model is about 100 times smaller. The authors also report gains in truthfulness, less toxic output and only minimal drops on public benchmarks.

Limits

The paper's own abstract concedes that the models still make simple mistakes. Hugging Face lists further challenges:

• Collecting human preference data is expensive, and results depend on how well people annotate
• Annotators often disagree with each other, and there is no ground truth to settle it
• Reward models can be exploited, so a model learns to please the scorer rather than the person
• The reinforcement learning step is costly because every generated output has to be scored
• There is no clear point at which a model counts as fully aligned

The method also shows why feedback matters more than size alone. A smaller model tuned on human preferences can feel more useful than a much larger one that has not been tuned, because usefulness depends on following the request and keeping a suitable tone, and these are things people judge directly. At the same time, the people giving feedback are a limited group, so the preferences the model learns reflect who was asked and what instructions they were given. Researchers therefore publish the guidelines that labelers follow, so that readers can judge what the training rewarded.

Where it fits

RLHF is one stage among several in building a chatbot, alongside large-scale pretraining and instruction fine-tuning. Companies do not always publish the exact recipe of their products, so the details above describe the published methods rather than any specific current chatbot.

Sources

• Illustrating Reinforcement Learning from Human Feedback (RLHF), Hugging Face
• Training language models to follow instructions with human feedback, arXiv