# MUSIQ — Multi-Scale Image Quality Transformer

## Model Info

- **Paper:** ICCV 2021 — "MUSIQ: Multi-Scale Image Quality Transformer" (Ke et al.)
- **Source:** Google Research (Pengchuan Zhang, Xiujun Li, Ping Luo, Kai Wang, Yu-Xiong Wang)
- **Official Google Repo:** <https://github.com/google-research/google-research/tree/master/musiq>
- **Google Blog:** <https://ai.googleblog.com/2021/07/musiq-assessing-image-aesthetic-and.html>
- **TensorFlow Hub:** Models available at `tfhub.dev` (official hosted checkpoints)
- **PyPI (IQA-PyTorch toolbox):** `pip install pyiqa` -> `pyiqa create_metric 'musiq'`
- **PyTorch unofficial impl:** <https://github.com/anse3832/MUSIQ> (PyTorch, works on KonIQ-10k)
- **License:** Apache 2.0 (via IQA-PyTorch reimplementation)
- **Status:** Actively maintained in IQA-PyTorch; official TensorFlow code archived but functional

## Architecture

- **Type:** Patch-based Vision Transformer (ViT) with multi-scale input
- **Backbone:** ResNet50 (ImageNet-pretrained) for feature extraction -> Transformer encoder for aggregation
- **Multi-scale processing:** Processes image at 3 scales simultaneously — native resolution, 224x224, and 384x384
- **Input:** Full-resolution images (no fixed-size constraint) — handles any aspect ratio natively
- **Output:** Single float score 0-100 (technical + aesthetic combined)
- **Training checkpoints:**
  - `ava_ckpt.npz` — trained on AVA dataset (aesthetic quality)
  - `koniq_ckpt.npz` — trained on KonIQ-10k (technical quality)
  - `paq2piq_ckpt.npz` — trained on PaQ-2-PiQ (technical quality)
  - `spaq_ckpt.npz` — trained on SPAQ (technical quality)
  - `imagenet_pretrain.npz` — ImageNet pretraining only
- **Key innovation:** Patch-based design bypasses CNN fixed-size constraint; hash-based 2D spatial embedding + scale embedding for positional encoding
- **MUSIQ-single variant:** Processes only native resolution (faster, slightly lower accuracy)

## Installation

```bash
# Option 1: IQA-PyTorch toolbox (recommended — easiest, supports all metrics)
pip install pyiqa

# Option 2: Official TensorFlow code
git clone https://github.com/google-research/google-research.git
cd google-research/musiq
pip install -r requirements.txt

# Option 3: Unofficial PyTorch implementation
git clone https://github.com/anse3832/MUSIQ.git
cd MUSIQ
pip install torch torchvision einops scipy tqdm
```

## Python Usage

```python
# Via IQA-PyTorch (recommended — single API for all metrics)
import pyiqa
import torch

device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")

# Create MUSIQ metric (auto-downloads checkpoint on first use)
musiq = pyiqa.create_metric('musiq', device=device)

# Single image
score = musiq('./photo.jpg')
print(f"MUSIQ score: {score:.2f} / 100")

# Directory of images
import glob
scores = []
for path in glob.glob('./wedding-photos/*.jpg'):
    scores.append(musiq(path))
```

```python
# Via official TensorFlow code
cd google-research/musiq
python3 -m musiq.run_predict_image \
  --ckpt_path=/path/to/spaq_ckpt.npz \
  --image_path=/path/to/photo.jpg
```

## Performance Benchmarks (from IQA-PyTorch docs & Google Research)

| Dataset | SRCC (rank correlation) | PLCC (Pearson correlation) |
|---------|------------------------|---------------------------|
| KonIQ-10k | ~0.90 | ~0.92 |
| PaQ-2-PiQ | ~0.85 | ~0.87 |
| SPAQ | ~0.88 | ~0.90 |
| AVA | ~0.75 | ~0.78 (lower — aesthetic-only dataset) |

## Hardware Requirements

- **VRAM:** ~3-4 GB (FP16), ~6-8 GB (FP32)
- **Batch size:** ~8-16 on 24 GB GPU (RTX 4090)
- **Speed per image:** ~50-150ms on GPU (FP16), ~200-500ms on CPU
- **Estimated time for 4,000 images:** ~15-25 minutes on GPU (FP16, batch 16)
- **Native resolution handling:** No resizing needed — processes full-resolution images directly

## Score Interpretation for Wedding Photos

| Score Range | Interpretation | Recommended Action |
|-------------|---------------|-------------------|
| 85-100 | Excellent quality | Auto-keep |
| 70-84 | Good quality | Keep, manual review if borderline |
| 50-69 | Acceptable | Review — may have minor issues |
| 30-49 | Poor quality | Likely reject |
| 0-29 | Very poor | Auto-reject |

**Recommended threshold for wedding culling:** Score >= 50 (i.e., not in the bottom third). This typically reduces 4,000 images to ~800-1,200 for manual review.

## What It's Good At

- **Technical quality detection:** Better than LAION predictors at detecting blur, noise, compression artifacts, over/under exposure
- **Full-resolution input:** No resizing artifacts — important for wedding photos with varying aspect ratios (portrait, landscape, square)
- **Combined aesthetic + technical scoring:** Not just "pretty" but also "technically sound" — single model for both dimensions
- **Google-backed research:** Well-documented, peer-reviewed, actively maintained in IQA-PyTorch
- **Apache 2.0 license:** Fully permissive for any use
- **Multiple training checkpoints:** Can choose checkpoint based on use case (aesthetic vs. technical focus)
- **Handles diverse aspect ratios:** Native resolution processing means no distortion from forced resizing

## What It's Bad At

- **Still no genre awareness:** General-purpose model, not wedding-specific
- **No context understanding:** Same as LAION — cannot distinguish emotional moments from empty frames
- **Slower than CLIP-based:** Transformer architecture is heavier (~50-150ms vs ~10-30ms for LAION V2)
- **Score range 0-100:** Different from LAION's 0-10 scale (requires threshold adjustment if combining)
- **Limited community adoption:** Less community testing and discussion compared to LAION predictors
- **Checkpoint selection matters:** Different checkpoints optimized for different datasets — no single "best" checkpoint for all use cases
- **TF/HF ecosystem split:** Official code is TensorFlow; PyTorch users need IQA-PyTorch or unofficial impl