An Overview of Machine Learning Model Training
Machine learning training involves initializing models, predicting, calculating loss, and updating.
Contributions
Every accepted correction to this page is recorded with the exact change, so readers can see how the page improved over time.
-
This technical article on machine learning model training was unusually well sourced, with all of its foundational research citations, including several of the most famous papers in the field, checking out precisely against the real published record. Two smaller issues were found: a training-cost estimate for GPT-3 that did not match the standard cited figure, and a LoRA efficiency statistic presented as a general property when it is actually specific to one benchmark in the original paper.
What the page claimedThe article gave GPT-3's training cost as a range of $4 to $12 million, when the standard, widely cited independent estimate is approximately $4.6 million, with no credible source supporting a figure as high as $12 million for this particular model. It also presented LoRA's reported 10,000 times reduction in trainable parameters and 3 times reduction in GPU memory as a general property of the technique, when these specific figures come from one particular benchmark in the original paper, fine-tuning GPT-3 at 175 billion parameters.
What was correctedThe article now cites the standard $4.6 million estimate for GPT-3's training cost, while noting that OpenAI never officially disclosed the exact figure, and clarifies that LoRA's reported efficiency figures are specific to the GPT-3 175B benchmark used in the original paper rather than a fixed ratio that applies at every model size.
Why: Web searches confirmed that all seven foundational research citations in this article, spanning backpropagation, LSTMs, weight initialization, residual networks, the Adam optimizer, cyclical learning rates, and neural scaling laws, matched the real published record precisely, along with the great majority of the specific figures in the article's real-world case studies. Only the GPT-3 cost estimate and the scope of the LoRA efficiency figures needed correction.
View the full record →