ETH Zurich researchers have developed SIFT, a new algorithm that enhances AI response accuracy and efficiency by selecting the most relevant data. This innovation has the potential to revolutionize specialized AI applications across various fields.
In a significant leap forward for artificial intelligence, researchers at ETH Zurich’s Institute for Machine Learning have developed an innovative algorithm to improve the reliability and efficiency of AI language models. The new method, known as SIFT (Selecting Informative data for Fine-Tuning), enhances AI’s ability to generate accurate responses by reducing uncertainty, a persistent issue in current AI models.
The research, led by Jonas Hübotter from the Learning & Adaptive Systems Group as part of his doctoral studies, has significant implications for the future of specialized AI applications. By incorporating additional data from the relevant subject areas, the SIFT algorithm precisely selects the information that is most likely to generate correct answers.
“Our algorithm can enrich the general language model of the AI with additional data from the relevant subject area of a question. In combination with the specific question, we can then extract from the depths of the model and from the enrichment data precisely those connections that are most likely to generate a correct answer,” Hübotter said in a news release.
Enhancing AI With Targeted Data
Designed to address the needs of companies, scientists and other users operating in specialized fields, SIFT allows users to integrate their locally stored data into large language models (LLMs).
Traditional methods often rely on the nearest neighbor approach, which can lead to redundant information and missed critical data points.
In contrast, SIFT evaluates the angles between vectors in a multidimensional space to identify and select complementary data, leading to more coherent and reliable responses.
“The method is particularly suitable for companies, scientists or other users who want to use general AI in a specialized field that is only covered partially or not at all by the AI training data,” added Andreas Krause, head of the research group and director of the ETH AI Centre.
Revolutionizing Response Accuracy and Computational Efficiency
One of the key advantages of the SIFT algorithm is its ability to improve response quality while significantly reducing the computational power needed.
By indirectly measuring uncertainty, the model dynamically determines the amount of data required for reliable answers. This adaptability not only enhances AI performance but also allows smaller models to achieve comparable results to much larger counterparts.
“In tests with standard data sets, we used SIFT tuning to outperform even the best current AI models with models up to 40 times smaller,” Hübotter added.
Broader Implications and Future Applications
Beyond improving AI responses, SIFT’s capabilities extend to data evaluation in various sectors, including medicine. By identifying which pieces of data are most relevant to specific inquiries, the algorithm could assist in pinpointing significant laboratory analyses or measurements for diagnostics.
“We can track which enrichment data SIFT selects. They are closely related to the question and therefore particularly relevant to this subject area. This could be used in medicine, for example, to investigate which laboratory analyses or measurement values are significant for a specific diagnosis and which less so,” Krause added.
Recognition
Hübotter is currently presenting his innovative approach at the International Conference on Learning Representations (ICLR) in Singapore.
The significance of their breakthrough was also recognized at the NeurIPS Annual Conference on Neural Information Processing Systems, where their scientific article won the Best Scientific Article award in the “Fine-tuning in Modern Machine Learning” workshop.
Source: ETH Zurich