Project Baracter: How a Language Model Gets Made

A model that invented an author

Given the start of a sentence, "Critical thinking is," one version of the model finished it this way:

Critical thinking is the rational question (Baracter, 1986).

No author named Baracter appears anywhere in the text the model was trained on. The model had seen hundreds of citations, each with a name, a comma, a year, and parentheses, and it learned to produce that pattern. It has no way to check whether a source exists. Large commercial AI models invent references in the same way. This one is small enough that the cause can be traced, and the project is named after the author it made up.

A language model does one thing: it predicts the next piece of text. Everything it knows comes from the text it was given. The rest of this page shows how that works, one step at a time.

The five steps

The model at a glance

Built by

Dr. Reginald V. Finley Sr., with Claude (Anthropic) as a guide

Type

GPT-style transformer, written from scratch in Python with PyTorch

Size

11,532,288 weights (adjustable numbers)

Vocabulary

2,048 word pieces, built from the training text itself

Memory span

Looks back up to 256 word pieces, about 800 characters

Layers

6 blocks, each with 6 attention heads

Hardware

Dr. Finley's laptop, with an NVIDIA RTX 2060 graphics card

Training time

About 4 to 5 minutes for a 3,000-lesson run; 36 minutes for the 24,000-lesson Wikipedia round

Largest training set

About 196 million word pieces (Round F, mostly Wikipedia)

For scale: GPT-2 small, released in 2019, had 124 million weights. GPT-3 trained on about 300 billion word pieces, roughly 1,500 times more text than Baracter's largest round. Today's large models have hundreds of billions of weights.

The model was built to learn how the process works. It and its training text stay private. The code is public, set up to run on free books, in section 09.

Collect the text, then clean it

We will follow one sentence through every step: "The heart pumps blood through the body."

Where the text came from

All of it came from folders on Dr. Finley's computer. It is his own writing plus course materials and textbooks from his degree programs:

• His graduate and undergraduate papers from eight programs, including his doctoral thesis from UNISA
• 63 of his LinkedIn articles (35 written before 2023, and 28 from 2023 onward that he edited by hand first)
• 2,980 of his Facebook posts
• Course readings, instructor slides, and classmates' papers from those programs
• Two full textbooks: Microbiology Demystified (McGraw-Hill) and Teaching About Evolution and the Nature of Science (National Academy of Sciences)

That was all of the training text in Part 1 of the project. In Part 2, he added WikiText-103, a public research collection of 28,475 Wikipedia articles (Merity et al., 2016), to see what a much larger pile of general English would do. Section 07 compares the two parts.

Email addresses and phone numbers were replaced with placeholders, so the model cannot repeat them. Student work was left out.

Step 1: Pull out the words

Word files, PDFs, and slide decks store their words inside formatting code. A script opens each file and keeps only the words. It skips files under 400 characters, and it drops near-duplicate drafts by comparing overlapping 8-word chunks, so a paper saved five times is read once.

Step 2: Clean it

The cleaning rules were added one at a time. Each was added after the model started copying a specific kind of junk:

• Web addresses and reference lists, after the model started writing web addresses.
• Slide leftovers: picture file names, figure captions, slide numbers, and the dotted lines in tables of contents.
• Two-column PDFs, read one column at a time. The first version read straight across the page and glued together sentences from two different paragraphs.

Whatever is left in the text, the model learns to write. Section 07 shows each kind of junk turning up in its output.

Turn text into numbers

Here is how Baracter's vocabulary splits "The cell membrane":

The cell mem b r ane

"Cell" appeared often enough in the training text to become a single piece. "Membrane" appeared less often, so it is built from four smaller pieces. Each piece has an ID number, and the model only ever sees those numbers. Computers only do arithmetic, so this step comes before anything else.

The pieces, called tokens, were built from the training text itself:

1. Start with single letters and symbols.
2. Find the most common pair of neighbors in all the text, such as "t" followed by "h", and glue it into a new piece, "th".
3. Repeat. "th" plus "e" becomes "the". Common words end up as single pieces.
4. Stop at 2,048 pieces.

This method is called Byte Pair Encoding. It produced one side effect: Dr. Finley's own writing contains no em dashes, so the em dash never became a piece, and this model cannot write one.

Build the network

Before training, every one of the 11.5 million weights is a random number. Asked to continue "Critical thinking is," the untrained model wrote:

Critical thinking is sh correl your shumm first±ool assignysis ways bestcept ask provide...

Those are real pieces from the vocabulary, picked at random. Training turns that noise into sentences.

What happens to one word inside

Follow " pumps" in "The heart pumps":

1. Embedding. The model looks up 384 numbers that describe " pumps." After training, words used in similar ways, such as " heart" and " lung," end up with similar numbers.
2. Position. It adds numbers for where the word sits, since "dog bites man" and "man bites dog" use the same words.
3. Attention. " pumps" looks back at the earlier words to work out which ones help predict what comes next. It can learn to focus on " heart." It looks only backward, because when the model writes, later words do not exist yet.
4. Repeat. Six stacked blocks refine the result.
5. Output. The model gives a score to each of its 2,048 pieces. The highest scores are its likeliest next pieces.

Training

One lesson looks like this. The model reads "The heart pumps" and gives every one of its 2,048 pieces a probability of coming next. If it gives the real answer, " blood," a low probability, its loss is high. The model then works out which direction each weight should move to make " blood" more likely, and moves every weight a small step that way.

1. Take 32 random stretches of text, 256 pieces each.
2. At every position, guess the next piece.
3. Measure the loss.
4. Work backward through the network to find which way each weight should move. This is called backpropagation.
5. Move every weight a small step that way. The rule that does this is called the optimizer.

Most training runs repeat this 3,000 times.

Two-stage training: read widely, then specialize

In Round G, training started from the finished Round F model, which had already read about 196 million word pieces, mostly Wikipedia. It then trained for 2,000 more lessons on Dr. Finley's text alone, with smaller nudges so it would adjust what it already knew and keep most of it. That took under three minutes.

The first stage is called pretraining, and the second is called fine-tuning. Large models such as ChatGPT and Claude are built in the same order: pretraining on a very large collection of general text, then further training for specific purposes.

The held-back test

Part of the text is kept out of training and used as a test, the way a teacher gives a new exam with questions students have not practiced. If the model scores well on its training text and poorly on the test, it is memorizing. The project saves the version that scores best on the test.

Writing and testing

To write, the model takes a starting phrase, predicts one piece, adds it to the end, and predicts again. Two settings control its choices. Temperature (set to 0.8) controls how adventurous each pick is. Top-k (set to 50) limits each pick to the 50 likeliest pieces.

To compare versions fairly, every version was scored on the same locked test text and given the same eight starting phrases with the same random seed. The only thing that changed between versions was the training text.

How to read the score

The score is shown as options per guess: how many equally likely choices the model is torn between for each next piece. Lower is better.

• A model guessing at random: about 2,048 options, the whole vocabulary
• Baracter's first trained version: about 72
• Baracter's best version from its own text (Part 1): about 44
• Baracter's best version after adding Wikipedia (Part 2): about 27

For technically minded readers: options per guess is e raised to the cross-entropy loss. The losses ran from 4.28 down to 3.31.

Results: two parts

The project ran in two parts, and the difference between them is the main result.

• Part 1 (November 2, 2026): text from Dr. Finley's computer only. Seven versions, trained on his own writing plus the course materials and textbooks from his degree programs. Several versions were trained more than once to check the results.
• Part 2 (November 3, 2026): Wikipedia added. Round F added 28,475 Wikipedia articles from WikiText-103, a public research collection (Merity et al., 2016), about 49 times more text than Part 1's largest round. Round G then took the Round F model and trained it again on Dr. Finley's text alone.

Every version in both parts was scored on the same locked test, which is Dr. Finley's own writing, so the numbers can be compared across parts.

Part 1: text from Dr. Finley's computer

Each Part 1 round trained for 3,000 lessons, about 4 to 5 minutes. The best Part 1 score was 44.

Part 2: Wikipedia added

Rounds F and G were each trained once. They have not yet been repeated with different random starting points the way Rounds D, D2, and E were.

What it wrote: Part 1

Each sample starts with the phrase the model was given (in bold). The rest is the model's own output.

What it wrote: Part 2

Same prompts, same random seed. Compare these with Part 1.

Citations the model invented

None of these exist. From Part 1: Choy & San Oo 2017, Salpern 2016, Maslow 2013, GMarine 2001, Phys & Ya 2014, Slex et al. 1996, Sham 2021, Baracter 1986, Bellikren 2012, and Berlera. From Part 2: Scannann et al. 2018.

Two are near-misses of real sources in the training text. Dr. Finley's thesis cites Choy & San Oo from 2012 and 2014, and the model produced a 2017 version. "Salpern, 2016" blends the real "Halpern, 2016." Adding Wikipedia did not stop the model from inventing sources.

What the project showed

These are findings from one small model. They are consistent with what researchers report about large models. This project tested only one small model.

From Part 1: text from his computer

1. The model writes what it read. Every odd output traced back to a source. Web addresses came from reference lists, file names and "(a) (b)" labels came from slides and federal regulations, and the copyright text came from a National Academies book.
2. More text helped the most. Going from about 600,000 to about 3.8 million word pieces dropped the score from 72 to 49, the largest improvement of any change in Part 1.
3. Cleaning alone scored worse at first. Round A removed the junk along with 17% of the text, and its score rose from 72 to 93. Part of the reason is that the locked test text was frozen before cleaning, so it still contains reference lists. The later slide cleanup in Round C improved the score slightly.
4. Fluent writing can be false. The model produced citations in correct form for papers that do not exist. It has no way to check a source, so a citation that looks right still has to be looked up.
5. Run it more than once. On a single run, adding the hand-edited LinkedIn articles (D2) looked like an improvement over D. Each version was then trained three times with different random starting points. The averages reversed the result, and the difference between starting points was larger than the difference between D and D2.

From Part 2: adding Wikipedia

1. Much more general text lowered the score on his writing. Round F scored 37, down from Part 1's best of 44, even though nearly all of the new text was Wikipedia and the test is Dr. Finley's writing.
2. More text needed more lessons. Round F trained for 24,000 lessons, eight times as many as Part 1. The training log shows that after 3,000 lessons, the number Part 1 used, Round F scored about 69, worse than Round E. More text helped only when the model had enough lessons to read it. This matches the scaling laws AI researchers have reported: predictions improve in a predictable way as training data, model size, and computing power grow together.
3. It wrote like what it read most. Round F's writing turned into encyclopedia entries about invented people and places. Round G, trained again on Dr. Finley's text, returned to his subjects.
4. Reading widely, then specializing, scored best. Round G reached 27, the best of any round, after under three minutes of extra training. Large models are built in this same order (see section 05). Rounds F and G have each been run once, so these two results still need repeat runs.

Try it yourself, for free

The scripts behind Project Baracter are public. The tutorial version trains on seven free science books from Project Gutenberg, including Darwin's On the Origin of Species and Faraday's The Chemical History of a Candle, so anyone can run every step. It explains each script in plain words and runs in a free Google Colab notebook or on your own computer. A version sized for ordinary laptops is included for computers without a graphics card.

Open the tutorial on GitHub

What comes next

Transcripts from Dr. Finley's radio program and interviews he has conducted are planned as the next training text. They are not in the model yet. This page will be updated with the results.

Questions about the project, or interested in bringing it to a classroom? Email finleyrobotics@gmail.com.

Glossary and sources

Language model

A program that predicts the next piece of text.

Token (word piece)

The unit the model reads and writes. It can be a whole word, part of a word, or punctuation.

Weight

One adjustable number inside the model. Training changes the weights.

Training

Showing the model text, measuring its mistakes, and adjusting its weights to make fewer.

Loss

A score for how wrong the model's guesses were. Lower is better.

Held-back test text

Text kept out of training, used to check whether the model learned patterns or memorized.

Memorizing (overfitting)

Doing well on training text and worse on new text.

Seed

The number that sets the random starting point. Different seeds give slightly different models from the same text.

Attention

The part of the model that decides which earlier words help predict the next one.

Pretraining

Training a model from random weights on a large, general collection of text.

Fine-tuning

Further training of an already-trained model on a smaller, targeted collection of text.

Scaling laws

The finding that model predictions improve in a predictable way as data, model size, and computing power increase.

Sources

• Kaplan, J., et al. (2020). Scaling laws for neural language models. arxiv.org/abs/2001.08361
• Hoffmann, J., et al. (2022). Training compute-optimal large language models. arxiv.org/abs/2203.15556
• Radford, A., et al. (2019). Language models are unsupervised multitask learners (GPT-2). OpenAI
• Merity, S., et al. (2016). Pointer sentinel mixture models (introduces WikiText-103). arxiv.org/abs/1609.07843
• Brown, T., et al. (2020). Language models are few-shot learners (GPT-3). arxiv.org/abs/2005.14165

Credit: Dr. Finley ran the training on his laptop, reviewed the results, and decided what data to use. Claude (Anthropic) wrote and explained the scripts and prepared some of the data.