Engineers at UC San Diego have unveiled a new technique that enables large language models to learn new tasks with significantly less data and computing power, potentially democratizing AI usage.
Researchers from the University of California San Diego have unveiled a pioneering approach to customize large language models (LLMs) — the engines behind chatbots and protein sequencing tools. This breakthrough method significantly reduces the data and computing power needed for these adaptations, potentially making advanced AI more accessible.
LLMs consist of billions of parameters that shape how they interpret and process information.
Traditionally, fine-tuning these models involves adjusting all these parameters, a process that is both costly and prone to overfitting. Overfitting occurs when a model memorizes specific patterns rather than understanding them, leading to poor performance on new examples.
The innovative method from UC San Diego engineers streamlines this process by updating only the most essential parts of an LLM, bypassing the need to retrain the entire model from scratch.
This approach not only cuts costs but also enhances the model’s flexibility and ability to generalize learning across different tasks.
“With our method, even small labs and startups without huge budgets, supercomputer-level resources or large datasets can adapt large AI models for their own needs,” Pengtao Xie, a professor in the Department of Electrical and Computer Engineering at the UC San Diego Jacobs School of Engineering, said in a news release. “This work represents a step toward democratizing AI.”
The team demonstrated their method on protein language models, crucial tools for studying and predicting protein properties.
For instance, the new technique achieved higher accuracy in predicting whether peptides can cross the blood-brain barrier while using 326 times fewer parameters than conventional methods. Similarly, it matched the performance of full fine-tuning in predicting protein thermostability while using 408 times fewer parameters.
The method, published in Transactions on Machine Learning Research, holds promise for a variety of applications beyond protein modeling, encompassing any field that leverages LLMs.