Multimodal Models
Pre-trained models for vision, audio, and cross-modal tasks.
Model Overview
| Model | Modality | Task |
|---|---|---|
| CLIP | Image + Text | Zero-shot classification, similarity |
| Whisper | Audio → Text | Transcription, translation |
| Stable Diffusion | Text → Image | Image generation, editing |
CLIP (Vision-Language)
Zero-shot image classification without training on specific labels.
CLIP Use Cases
| Task | How |
|---|---|
| Zero-shot classification | Compare image to text label embeddings |
| Image search | Find images matching text query |
| Content moderation | Classify against safety categories |
| Image similarity | Compare image embeddings |
CLIP Models
| Model | Parameters | Trade-off |
|---|---|---|
| ViT-B/32 | 151M | Recommended balance |
| ViT-L/14 | 428M | Best quality, slower |
| RN50 | 102M | Fastest, lower quality |
CLIP Concepts
| Concept | Description |
|---|---|
| Dual encoder | Separate encoders for image and text |
| Contrastive learning | Trained to match image-text pairs |
| Normalization | Always normalize embeddings before similarity |
| Descriptive labels | Better labels = better zero-shot accuracy |
Key concept: CLIP embeds images and text in same space. Classification = find nearest text embedding.
CLIP Limitations
- Not for fine-grained classification
- No spatial understanding (whole image only)
- May reflect training data biases
Whisper (Speech Recognition)
Robust multilingual transcription supporting 99 languages.
Whisper Use Cases
| Task | Configuration |
|---|---|
| Transcription | Default transcribe task |
| Translation to English | task="translate" |
| Subtitles | Output format SRT/VTT |
| Word timestamps | word_timestamps=True |
Whisper Models
| Model | Size | Speed | Recommendation |
|---|---|---|---|
| turbo | 809M | Fast | Recommended |
| large | 1550M | Slow | Maximum quality |
| small | 244M | Medium | Good balance |
| base | 74M | Fast | Quick tests |
| tiny | 39M | Fastest | Prototyping only |
Whisper Concepts
| Concept | Description |
|---|---|
| Language detection | Auto-detects, or specify for speed |
| Initial prompt | Improves technical terms accuracy |
| Timestamps | Segment-level or word-level |
| faster-whisper | 4× faster alternative implementation |
Key concept: Specify language when known—auto-detection adds latency.
Whisper Limitations
- May hallucinate on silence/noise
- No speaker diarization (who said what)
- Accuracy degrades on >30 min audio
- Not suitable for real-time captioning
Stable Diffusion (Image Generation)
Text-to-image generation with various control methods.
SD Use Cases
| Task | Pipeline |
|---|---|
| Text-to-image | DiffusionPipeline |
| Style transfer | Image2Image |
| Fill regions | Inpainting |
| Guided generation | ControlNet |
| Custom styles | LoRA adapters |
SD Models
| Model | Resolution | Quality |
|---|---|---|
| SDXL | 1024×1024 | Best |
| SD 1.5 | 512×512 | Good, faster |
| SD 2.1 | 768×768 | Middle ground |
Key Parameters
| Parameter | Effect | Typical Value |
|---|---|---|
| num_inference_steps | Quality vs speed | 20-50 |
| guidance_scale | Prompt adherence | 7-12 |
| negative_prompt | Avoid artifacts | "blurry, low quality" |
| strength (img2img) | How much to change | 0.5-0.8 |
| seed | Reproducibility | Fixed number |
Control Methods
| Method | Input | Use Case |
|---|---|---|
| ControlNet | Edge/depth/pose | Structural guidance |
| LoRA | Trained weights | Custom styles |
| Img2Img | Source image | Style transfer |
| Inpainting | Image + mask | Fill regions |
Memory Optimization
| Technique | Effect |
|---|---|
| CPU offload | Reduces VRAM usage |
| Attention slicing | Trades speed for memory |
| VAE tiling | Large image support |
| xFormers | Faster attention |
| DPM scheduler | Fewer steps needed |
Key concept: Use SDXL for quality, SD 1.5 for speed. Always use negative prompts.
SD Limitations
- GPU strongly recommended (CPU very slow)
- Large VRAM requirements for SDXL
- May generate anatomical errors
- Prompt engineering matters
Common Patterns
Embedding and Similarity
All three models use embeddings:
- CLIP: Image/text embeddings for similarity
- Whisper: Audio embeddings for transcription
- SD: Text embeddings for image conditioning
GPU Acceleration
| Model | VRAM Needed |
|---|---|
| CLIP ViT-B/32 | ~2 GB |
| Whisper turbo | ~6 GB |
| SD 1.5 | ~6 GB |
| SDXL | ~10 GB |
Best Practices
| Practice | Why |
|---|---|
| Use recommended model sizes | Best quality/speed balance |
| Cache embeddings (CLIP) | Expensive to recompute |
| Specify language (Whisper) | Faster than auto-detect |
| Use negative prompts (SD) | Avoid common artifacts |
| Set seeds for reproducibility | Consistent results |
Resources
- CLIP: https://github.com/openai/CLIP
- Whisper: https://github.com/openai/whisper
- Diffusers: https://huggingface.co/docs/diffusers