> OpenAI's CLIP model already embeds images and text into the same embedding space.
Not really. Embeddings from images occupy a different region of the space than embeddings from text. A picture of cat and the text "cat" do not resolve to the same embedding.
This is why DALL-E has a model learning to translate CLIP text embeddings into CLIP image embeddings before decoding the embedding.
Not really. Embeddings from images occupy a different region of the space than embeddings from text. A picture of cat and the text "cat" do not resolve to the same embedding.
This is why DALL-E has a model learning to translate CLIP text embeddings into CLIP image embeddings before decoding the embedding.