An Overview of Machine Learning Model Training
Machine learning training involves initializing models, predicting, calculating loss, and updating.
What was corrected
The article gave GPT-3's training cost as a range of $4 to $12 million, when the standard, widely cited independent estimate is approximately $4.6 million, with no credible source supporting a figure as high as $12 million for this particular model. It also presented LoRA's reported 10,000 times reduction in trainable parameters and 3 times reduction in GPU memory as a general property of the technique, when these specific figures come from one particular benchmark in the original paper, fine-tuning GPT-3 at 175 billion parameters.
The article now cites the standard $4.6 million estimate for GPT-3's training cost, while noting that OpenAI never officially disclosed the exact figure, and clarifies that LoRA's reported efficiency figures are specific to the GPT-3 175B benchmark used in the original paper rather than a fixed ratio that applies at every model size.
Why this is better
Web searches confirmed that all seven foundational research citations in this article, spanning backpropagation, LSTMs, weight initialization, residual networks, the Adam optimizer, cyclical learning rates, and neural scaling laws, matched the real published record precisely, along with the great majority of the specific figures in the article's real-world case studies. Only the GPT-3 cost estimate and the scope of the LoRA efficiency figures needed correction.
More by Emir Baycan in Explainers
- Correction How Version Control Systems Work
- Correction What Near-Death Experiences Reveal About Consciousness
- Correction How to Conduct a Premortem: Identify Project Risks
- Correction How Epigenetics Functions: Gene Regulation and Experience