Skip to contents

Preference optimization with DPO or ORPO

Source: R/prefer.R
dragon_prefer.Rd

The second stage of post-training. Where dragon_train() teaches a model what a good reply looks like, this teaches it which of two replies is better, from a dataset mapped with dragon_map_pairs().

Usage

dragon_prefer(
  dataset,
  model,
  method = c("dpo", "orpo"),
  beta = 0.1,
  lora = dragon_lora(),
  args = dragon_train_args(learning_rate = 5e-05, epochs = 2),
  hardware = dragon_hardware(),
  name = NULL,
  run_dir = NULL,
  runs_dir = dragon_runs_dir(),
  n_samples = 10,
  revision = NULL,
  trust_remote_code = FALSE,
  wait = FALSE
)

Arguments

dataset

A dataset mapped with dragon_map_pairs().

model

A Hugging Face model id such as "HuggingFaceTB/SmolLM2-135M-Instruct", a local model directory, or a finished dragon_run to continue from. In the last case the earlier run's adapters are folded into the weights before this run adds its own, so stages chain: fine-tune, then dragon_prefer(), and so on. See dragon_presets() for model ids.

method

"dpo" or "orpo".

beta

Preference strength. Typical values are 0.05 to 0.5.

lora

LoRA settings from dragon_lora().

args

Training settings. The defaults use a lower learning rate than dragon_train(), which preference methods need.

hardware

Hardware settings from dragon_hardware().

name

Short label used in the run id. Defaults to the model name.

run_dir

Exact directory to use. Defaults to a timestamped directory under runs_dir.

runs_dir

Parent directory for runs. See dragon_runs_dir().

n_samples

Number of held-out rows to generate sample replies for at the end of training.

revision

Model revision (branch, tag, or commit) on the Hub.

trust_remote_code

Allow the model repository to run custom code.

wait

Block until training finishes.

Value

A dragon_run object.

Details

Two methods are available:

  • "dpo", Direct Preference Optimization. Pushes the model's implicit reward for the chosen reply above the rejected one, relative to a reference model. With LoRA the reference is the same model with the adapter switched off, so it costs no extra memory.

  • "orpo", Odds Ratio Preference Optimization. Combines the supervised loss on the chosen reply with an odds-ratio penalty on the rejected one. Needs no reference model and works from a base model that has not been fine-tuned yet.

beta controls how hard the model is pushed: the KL strength for DPO, the odds-ratio weight for ORPO. 0.1 is a sensible start for both.

Pass a finished run as model to continue from it. Its adapters are folded into the weights before this stage adds its own, which is the usual sequence: dragon_train() first, then dragon_prefer() on top.

Examples

if (interactive()) {
sft <- dragon_dataset("tickets.csv") |>
  dragon_map(prompt = "question", response = "answer") |>
  dragon_train("Qwen/Qwen2.5-0.5B-Instruct", wait = TRUE)

dpo <- dragon_dataset("preferences.csv") |>
  dragon_map_pairs(prompt = "question", chosen = "better", rejected = "worse") |>
  dragon_prefer(sft, method = "dpo", beta = 0.1, wait = TRUE)

dragon_evaluate(dpo)
dragon_generate(dpo, "My thermostat keeps dropping off Wi-Fi.")
}