This is the default route through dragonfarm: drop in documents, ask them questions straight away, have a model write training questions from them, review those, and fine-tune. Nothing here needs a table of prompts and replies. If you already have one, the table route is one link away in the app and is described in the quickstart.
Everything below also happens in the app: dragon_app()
opens on the Documents tab, and the Teach card at the bottom is the
review step.
1. Read the documents
PDF, HTML, Markdown, plain text, Word, spreadsheets, whole folders, and web pages all work. The text is split into chunks of about 170 words that never cross a heading, and each chunk remembers its file, section, and (for PDFs) page so answers can say where they came from.
library(dragonfarm)
options(dragonfarm.runs_dir = "~/dragonfarm_runs") # keep sets and runs after R closes
docs <- dragon_documents(c("policies/", "https://example.com/help"), name = "support")
docs
#> Document set support
#> • returns.pdf: 9 chunks, 1,320 words
#> • warranty.docx: 4 chunks, 610 words
#> • https://example.com/help: 11 chunks, 1,540 words
#> 24 chunks of about 170 words; index: BAAI/bge-small-en-v1.5Two things happen quietly. Quoted passages and code blocks are marked
on their chunks, and files that look like a changelog
(NEWS, CHANGELOG, or mostly version headings)
are flagged and down-weighted. Both came out of testing: a quoted
example reply in a README turned into confident, wrong training
questions about a product that does not exist, and changelog entries
kept crowding out the page that answered the question. Reading the same
set again re-reads only files whose contents changed.
2. Ask, before training anything
ans <- dragon_ask(docs, "How long is the return window?")
ans
#> You can return any device within 30 days of delivery, opened or unopened [1].
#>
#> [1]* returns.pdf, p. 2 › Returns
#> [2] returns.pdf, p. 3 › Refunds
#> [3] https://example.com/help › Orders
#> [4] warranty.docx › Coverage
#> * cited in the answer. Excerpt text is in $sources$text.The k most relevant chunks, found by combining the
meaning of the question with its exact words and figures, go in front of
a model as numbered excerpts with an instruction to answer only from
them and cite them. The model is whatever dragon_check() recommends for
your GPU memory: a 1.5B model under 6 GB, a 3B at 6 to 8 GB, a 7B loaded
in 4-bit at 8 GB and up. Pass model = to use a run, another
id, or a server backend. Models of 3B and up answer in full, with the
figures, steps, or code the excerpts hold; smaller ones answer in a
sentence or two, because asked for more they made things up. In the app
the answer appears as it is written (on_token = does the
same in code).
This step is worth doing first for two reasons. It shows you whether
the documents came through (if the answer cites the wrong file, look at
dragon_chunks(docs)), and for many uses it is already
enough: swap the documents, ask again, no training.
A conversation works the same way, with the history kept plain:
chat <- dragon_chat(documents = docs)
chat$say("How long is the return window?")
chat$say("And if I already set the device up?")
chat$sources() # the excerpts behind the last reply3. Write training questions, then review them
qa <- dragon_questions(docs) # the recommended local model, in the background
#> 72 questions written, 41 kept by the filters (57%). Review them before training.
#> Rejected: answer not grounded (19), names not in the source (7), question length (3), duplicate question (2)For each chunk a model writes three self-contained questions and
answers them from that chunk alone: by default a mix of a “what”, a
“why” or “how”, and, where the passage compares two things, a “how do
they differ” (mix = FALSE asks for any three). Simple
deterministic filters then mark pairs whose answer is not grounded in
the chunk, that name things the chunk does not, that refer to “the
passage”, or that duplicate another question. Nothing is thrown away:
every pair comes back with keep and a
reason.
Review them. This is the step that decides whether fine-tuning helps. On the package’s own documentation, a 7B model’s pairs were about 60 percent usable, a 3B’s about 33 percent, a 1.5B’s about 20 percent, and roughly a fifth of what passed the filters was still wrong or trivial. A model fine-tuned on unreviewed pairs learned to sound sure when it was wrong.
qa[!qa$keep, c("question", "reason")] # read the rejections
qa$keep[qa$id == "p0017"] <- TRUE # rescue one
qa$keep[qa$id == "p0030"] <- FALSE # drop one
dragon_questions_save(qa) # decisions are kept with the setA stronger generator gives a higher yield. Any teacher works:
qa <- dragon_questions(docs, generator = dragon_llm_anthropic()) # needs ANTHROPIC_API_KEY4. Fine-tune with the excerpts in the prompt
ds <- dragon_questions_dataset(qa, docs) # kept pairs, excerpts retrieved into every prompt
run <- dragon_train(ds, "Qwen/Qwen2.5-0.5B-Instruct", wait = TRUE)Each training prompt is the retrieved excerpts followed by the
question, retrieved by the question rather than taken from the chunk it
was written from, so training matches what happens at answer time; the
chunk the pair came from is always among them, and each answer ends with
that excerpt’s number, so the tuned model keeps citing the way
dragon_ask() expects (on R’s own manuals a model trained on
uncited pairs stopped citing altogether). This matters more than it
sounds. In the prototypes, a model trained this way recalled 89 percent
of the facts in a held-out test against 68 percent for a plain
fine-tune, and when the documents were edited after training it followed
the new text in 88 percent of answers while the plain fine-tune kept
repeating the old policy. The model learns the style and how to use
excerpts; the documents stay the source of truth, and they can change
without retraining.
5. Use it
dragon_ask(docs, "How long is the return window?", model = run)
dragon_chat(run, documents = docs)$say("A customer wants to return an opened device.")
dragon_merge(run, "models/support-0.5b") # a standalone model that loads with plain transformersWhat to expect, honestly
- Retrieval does most of the work. Without it a small model answered none of our test questions correctly and invented function names; with it, the same model answered most of the questions whose answer had been retrieved.
- Model size matters for answering, mostly in how a model fails. On 55 questions with known answers (an annual report and R’s manuals), the 7B in 4-bit was right on all of them; the 3B and 1.5B were wrong or empty four and six times, the 0.5B 14 times. On 20 questions the documents do not cover, the 7B, 3B, and 1.5B said so every time, the 0.5B 16 times. The 4-bit 7B uses about the memory of a full-precision 3B. All the numbers, including fine-tuning and a fixed-format task, are in Choosing a model size.
- Fine-tuning on a few dozen unreviewed machine-written pairs barely helped a 0.5B and made its mistakes confident. Review, or use a stronger generator, before you train.
- Know what retrieval cannot do. It answers from the four or so
excerpts it finds, so a question that has to look at every row of a
spreadsheet or every year of a table (“which state had the highest
murder rate?”, “in which year did margins peak?”) is out of reach; ask
those in R on the data itself. Tables are flattened into lines of text;
where a table puts the years in a header row, each figure is labelled
with its year (“Net sales (2024: $249,625; 2023: $237,710; …)”), which
took questions about a specific year’s figure in an annual report from 5
of 10 right to 9 (PDF) and 10 (Word). Tables without such a header can
still be misread. Scanned PDFs have no text to read: they are reported
by name, and need OCR (for example
ocrmypdf) first. A PDF that needs a password to open says so; add an unlocked copy. Numbers inside charts and maps are usually lost. - On real PDFs it holds up. R’s own manuals (An Introduction to R as PDF plus the R FAQ as HTML, 425 chunks) read and indexed in 41 seconds; the 7B in 4-bit answered nine of ten questions from the right section and said “the documents do not cover this” on the tenth; the generator wrote 796 pairs in 21 minutes of which 78 percent passed the filters. A 0.5B trained on the kept pairs learned to cite and to answer in one sentence, and named the right function more often (27 of 32 R questions against 22 untrained), but it also made up an answer, with a citation, to every question the manuals do not cover. The model size you can run matters more than the training.
- Fine-tuning a 7B is possible on 8 GB with
dragon_hardware(load_in_4bit = TRUE)(QLoRA), but only just: batch 1, 512 tokens, gradient checkpointing, about 22 minutes for three epochs on 190 short tickets, 7.9 GB peak. It is worth it when it fits: on held-out tickets the three-epoch 7B reproduced every policy the 0.5B had got wrong or blended. One epoch was not enough. Document prompts of about a thousand tokens also fit now, slowly: three epochs on 616 pairs from R’s manuals took 4.3 hours (batch 1 x 16, 1,536 tokens, gradient checkpointing). The tuned 7B was as accurate as the untrained one on ten test questions (nine of ten), answered more briefly, cited an excerpt every time, and was never confidently wrong; a tuned 0.5B on the same pairs was wrong on about half of those ten. For speed, a cloud GPU still does this in a fraction of the time. - PDFs come through as text, not layout. Where a table puts its years in a header row, each figure is labelled with its year; other tables can still be misread. Markdown, HTML, Word, and spreadsheets come through cleanly.
- The first question in a session loads the model. For a 7B in 4-bit that is about a minute; later questions take a few seconds.
The numbers above come from one corpus (this package’s own
documentation), 35 hand-written questions, and one run each. They are
measured, not promised; measure on your own documents. The scripts are
in scripts/doc_prototype/ and
scripts/rag_prototype/ in the repository.