Contrastive fashions like CLIP have been proven to be taught sturdy representations of photographs that seize each semantics and magnificence. To leverage these representations for picture technology, we suggest a two-stage mannequin: a previous that generates a CLIP picture embedding given a textual content caption, and a decoder that generates a picture conditioned on the picture embedding. We present that explicitly producing picture representations improves picture range with minimal loss in photorealism and caption similarity. Our decoders conditioned on picture representations also can produce variations of a picture that protect each its semantics and magnificence, whereas various the non-essential particulars absent from the picture illustration. Furthermore, the joint embedding house of CLIP permits language-guided picture manipulations in a zero-shot style. We use diffusion fashions for the decoder and experiment with each autoregressive and diffusion fashions for the prior, discovering that the latter are computationally extra environment friendly and produce higher-quality samples.
