diff --git a/README.md b/README.md index 0f24ecb..95b2e4f 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,14 @@ Reference: official I-JEPA README https://github.com/facebookresearch/ijepa/blob - Supervised training on WILDS-iWildCam labeled dataset: https://arxiv.org/abs/2012.07421 (WILDS: A Benchmark of in-the-Wild Distribution Shifts) - Supervised learning supports full fine-tuning or freezing the encoder - +

+ I-JEPA masking on an iWildCam camera-trap image +

+ +

+ I-JEPA masking on a real iWildCam camera-trap image: the context block (blue) is + encoded to predict the representations of several target blocks (green/red/orange). +

## Models @@ -16,7 +23,49 @@ Reference: official I-JEPA README https://github.com/facebookresearch/ijepa/blob - ViT-H, 16x16 patches, 448x448 resolution (trained) - Plan: add a graph comparing models with the WILDS leaderboard https://wilds.stanford.edu/leaderboard/#with-unlabeled-data-1 - +## Results + +Linear-probing evaluation on iWildCam2020-WILDS with a **frozen** I-JEPA encoder and a +single trained linear head. All values are the mean over 5 seeds (±std). The headline +metric is F1-Macro on the Target Out-of-Distribution (OOD) split; the gap +Δ = F1(ID) − F1(OOD) measures the drop under distribution shift. Only the +ImageNet-pretrained checkpoints are reported here. + +| Model | ID F1-Macro | OOD F1-Macro | Gap ΔF1 | +|---|---|---|---| +| ViT-H/14 (224, IN-1K) | 0.338 ±0.015 | 0.214 ±0.006 | 0.124 | +| ViT-H/16 (448, IN-1K) | 0.385 ±0.005 | 0.247 ±0.003 | 0.138 | +| **ViT-H/14 (224, IN-22K)** | 0.348 ±0.003 | **0.260 ±0.003** | 0.088 | +| ViT-g/16 (224, IN-22K) | 0.371 ±0.002 | 0.255 ±0.003 | 0.116 | + +Key takeaways: + +- The best frozen probe (**ViT-H/14, IN-22K**) reaches **0.260 OOD F1-Macro**, ranking + **#16** on the iWildCam2020-WILDS leaderboard — despite training only a linear head + rather than fine-tuning the full backbone. +- Its ID→OOD generalization gap (ΔF1 = 0.088) is comparable to the full-fine-tuning + CLIP leaders on the leaderboard (FLYP ΔF1 = 0.139, AutoFT ΔF1 = 0.115). +- Absolute F1-Macro scales with pretraining data (IN-22K > IN-1K) and input resolution + (the higher-resolution ViT-H/16 448 is the strongest IN-1K checkpoint). + +### Label efficiency + +Because I-JEPA pretrains without labels, the representations stay useful when labeled +data is scarce. OOD Target F1-Macro when the linear probe trains on 1%, 10%, 50%, and +100% of the labeled Source split (mean over 5 seeds): + +| Model | 1% | 10% | 50% | 100% | +|---|---|---|---|---| +| ViT-H/14 (224, IN-22K) | 0.193 ±0.009 | 0.236 ±0.011 | 0.261 ±0.009 | 0.260 ±0.003 | +| ViT-g/16 (224, IN-22K) | 0.204 ±0.009 | 0.243 ±0.007 | 0.259 ±0.014 | 0.255 ±0.003 | +| ViT-H/14 (224, IN-1K) | 0.120 ±0.036 | 0.190 ±0.010 | 0.220 ±0.012 | 0.214 ±0.006 | +| ViT-H/16 (448, IN-1K) | 0.146 ±0.022 | 0.214 ±0.012 | 0.230 ±0.011 | 0.247 ±0.003 | + +A small labeled subset already recovers most of the full-data performance +(diminishing returns), with the IN-22K backbones degrading most gracefully — a +practical advantage for wildlife monitoring where labeled camera-trap data is expensive. + +These results are from the accompanying master thesis evaluating I-JEPA on iWildCam2020-WILDS. ## Repo layout diff --git a/assets/ijepa_masking_turkey.jpg b/assets/ijepa_masking_turkey.jpg new file mode 100644 index 0000000..c3dc6c2 Binary files /dev/null and b/assets/ijepa_masking_turkey.jpg differ