What are foundation models?

Foundation models are arguably the most important technology to arise out of the deep learning revolution of the 2010s. These massive machine learning models are characterised by the use of unsupervised data, allowing them to learn generalisable concepts in a range of domains. We call this data unsupervised because there is no manual labelling of each data point - fo example, Large Language Models (LLMs) like ChatGPT simply learn to predict the next word ina sequence, that is they just train on a sequence minus the next word - no human annotation required.

One of the most important features of foundation models are their impressive capabilities in transfer learning. Transfer learning is the process of taking knowledge learned from one task (e.g. image classification) and applying it to another (e.g. product recommendation). In the context of foundation models, this typically involves training the model with a specific task in mind, for our LLM example this would be next word prediction, then the model is adapted to excel at a different and often more specific task, such as acting as an assistant or chatbot.

Why task specific models don't scale sustainably

Before foundation models, any task to be solved with machine learning required a task-specific model. These models would use human-labelled data and be built from the ground up to solve one particular task. This approach created significant environmental overhead - not only did these systems require expensive annotation pipelines and greater engineering resources to maintain and deploy, but each new application meant starting the entire training process from scratch. Especially for smaller organisations, this approach is often prohibitively costly in terms of both time and resources, while also creating unnecessary computational waste.

Social and environmental impact of adaptation

Yes, training a foundation model from scratch costs significantly more than building a traditional task-specific model, hence their bad rep. But here's the key insight: that upfront investment pays dividends when the model gets reused across many different applications. Think of it like building infrastructure. A highway is expensive to construct, but once it's there, thousands of people can use it daily. Foundation models work similarly - the heavy computational lifting happens once, then countless teams can adapt the model for their specific needs at a fraction of the original cost.

This reusability creates two major advantages. First, it democratises access to cutting-edge AI. Many foundation models are released as open source, meaning smaller organisations like Moonsift can build sophisticated systems without the massive resources typically required. Instead of AI capabilities being concentrated among a few tech giants, open-source foundation models level the playing field and spark innovation across the entire ecosystem.

Second, there's a significant environmental upside. While training a foundation model initially requires substantial energy, this cost gets spread across all its future uses. Every time someone adapts an existing model instead of training from scratch, we collectively reduce the computational footprint of AI development. It's a more sustainable approach that supports both innovation and environmental responsibility.

Putting sustainable AI to work at Moonsift

At Moonsift, we've adapted various open-source foundation models to provide product recommendations. Rather than training a recommendation system from scratch - which would have required massive computational resources and energy consumption - we utilised established models' understanding of a range of concepts from synonyms in descriptions to identifying materials of items in photos to predict what users might want next. This approach not only saved us significant development time and resources, but also represents a more sustainable path to AI innovation, giving us access to sophisticated fashion knowledge that would have taken years and substantial energy to build independently.

Foundation models are also powerful tools for data analysis, with their "understanding" of general concepts allowing for broader insights than traditional analysis would allow. In this vein, we're also using FashionCLIP as a data analysis tool to understand how users' fashion preferences evolve over time. By tracking the "directionality of user movement" through the model's embedding space, we can literally map out style journeys and see patterns in how tastes develop and shift.

This dual use of a single foundation model - both for recommendations and deep user analysis - exemplifies sustainable AI practices in e-commerce an advocates for further innovation in open source foundation model adaption. It's efficient, it's powerful, and it opens up entirely new ways of understanding fashion behaviour without the environmental and social impact of training multiple specialised systems.