Elon Musk has recently agreed with a viewpoint held among AI researchers that there’s a finite amount of authentic data available for training AI models. As per Musk’s insight, expressed during a real-time chat session with the chairman of Stagwell, Mark Penn, we have essentially used up the entirety of human knowledge to train AI, a milestone he claims we achieved last year. Musk, the owner of xAI, aligns his understanding with Ilya Sutskever, former chief scientist at OpenAI. In a talk at NeurIPS, an influential machine learning conference, Sutskever also signaled that we could be reaching a ‘data saturation point’ within the AI field.
Looking to the future, both Musk and Sutskever suggest a shift in AI development practices is impending. The calculated guess predicts a move away from traditional learning models due to the apparent data shortage. However, Musk presents an innovative solution to the impending data deficiency problem: synthetic data. He suggests that synthetic materials, produced by AI systems themselves, hold substantial promise.
Musk conveyed his belief on how the introduction of synthetic data will change the AI landscape. He foresees AI systems using their self-generated data for training, offering an alternative to real-world data. By harnessing synthetic data, he suggests, AI systems could engage in a type of self-supervised learning where they continually evaluate their performance and adapt accordingly.
Several high-profile tech firms have already begun using synthetic data in the development of their flagship AI models. Names include industry leaders such as Microsoft, Meta, OpenAI, and a firm named Anthropic. Predictions made by Gartner further indicate the growing acceptance of synthetic data. It estimates that by 2024, nearly 60% of the data used in AI and analytics projects will be synthetically produced.
Notably, Microsoft recently contributed to the emerging synthetic data trend. The tech giant’s ‘Phi-4’ model was trained using a combination of synthetic and authentic data. Microsoft’s approach to developing Phi-4 attests to the potential viability of synthetic data in AI training, perhaps signifying a shift in industry norms.
Google is another notable tech firm leveraging synthetic data. Recent reports reveal that Google trained its ‘Gemma’ models on synthetic data. These instances show us how synthetic data fitting alongside real-world data is laying the groundwork for next-generation AI models.
Anthropic is a further example of a company harnessing synthetic data to push the boundaries of what’s possible. For one of its most efficient systems, the ‘Claude 3.5 Sonnet,’ synthetic data was a crucial development component. It exemplifies the importance of synthetic data usage in the creation of advanced systems.
Meta, another tech behemoth, has used AI-generated synthetic data to finetune its newest ‘Llama’ series of models. As such, influential companies like Meta illustrate that synthetic data use is not confined to a niche corner of the industry; instead, it’s gaining traction across the board.
Training AI models on synthetic data has its benefits beyond data availability. A financial advantage of such practice has been reported by AI startup Writer. The company stated that it was able to develop its ‘Palmyra X 004’ model, which heavily utilized synthetic data, for around $700,000, comparing to an estimated $4.6 million for a similar-sized model by OpenAI.
However, despite the possible benefits, the industry’s increasing reliance on synthetic data is not without potential pitfalls. Some experts suggest that training AI models on synthetic data could, in fact, lead to a phenomenon known as ‘model collapse.’ Under this circumstance, a model’s creativity might be stifled over time, culminating in more biased outputs and a compromise in functionality.
The core concern here is that while synthetic data production is controlled by the AI models themselves, these models, in turn, are trained on already existing data. Consequently, if the original training data contains inherent biases or limitations, these flaws could potentially be replicated, and even magnified, in the synthetic data produced, thereby tainting the model’s outputs.