GELATO: The Frozen Towers Approach to Multimodal Embeddings
English summary
GELATO investigates extending a strong pre-trained text embedding model to handle multimodal data rather than training a new model from scratch. The text encoder remains frozen (the 'text tower') while separate modality-specific encoders are trained to align images, audio, or other modalities into the same embedding space. This 'frozen towers' strategy leverages existing text understanding and avoids retraining the core model. The blog post outlines the method and its motivation for efficient multimodal representation learning.
Chinese summary
GELATO 探索将强大的预训练文本嵌入模型扩展到多模态场景，而非从头训练新模型。其文本编码器保持冻结（“文本塔”），同时训练独立的其他模态编码器，将图像等数据对齐到相同的嵌入空间。这种“冻结塔”策略利用了已有文本理解能力，避免重新训练核心模型。文章介绍了该方法及其高效多模态表示学习的动机。
Key points
GELATO extends a pre-trained text embedding model for multimodal use without retraining the text encoder.
GELATO 在不重新训练文本编码器的情况下，将预训练文本嵌入模型扩展用于多模态。
The text model (text tower) is frozen, and new encoders for other modalities are trained to align with its representation space.
文本模型（文本塔）被冻结，其他模态的新编码器被训练以对齐其表示空间。
The approach avoids costly training from scratch and reuses strong text embeddings.
该方法避免昂贵的从头训练，并重用强大的文本嵌入。