Download
lm-human-preferences is the official OpenAI codebase that implements the method from the paper Fine-Tuning Language Models from Human Preferences. Its purpose is to show how to align language models with human judgments by training a reward model from human comparisons and then fine-tuning a policy model using that reward signal. The repository includes scripts to train the reward model (learning to rank or score pairs of outputs), and to fine-tune a policy (a language model) with reinforcement learning (or related techniques) guided by that reward model. The code is provided “as is” and explicitly says it may no longer run out-of-the-box due to dependencies or dataset migrations. It was tested on the smallest GPT-2 (124M parameters) under a specific environment (TensorFlow 1.x, specific CUDA / cuDNN combinations). It includes utilities for launching experiments, sampling from policies, and simple experiment orchestration.
Features
- Training a reward model from human preference comparisons
- Fine-tuning a policy (language model) guided by the reward model
- Sampling / inference utilities to generate outputs from the trained policy
- Experiment orchestration (launch.py) to combine stages (reward + policy)
- Label handling and mapping from human comparisons to scalar reward signals
- Support for small GPT-2 (124M) model as reference environment
Project Samples
