Visual anomaly classification (AC) and anomaly segmentation (AS) classify and localize defects in industrial manufacturing, respectively.

Samples of MVTec AD dataset

Samples of VisA (Visual Anomaly) dataset
Challenge of one-class and unsupervised anomaly machines Existing work (including ours) have focused on one-class or unsupervised anomaly detection, which only requires normal images. These methods fit a model to the normal images and treat any deviations from it as anomalous. However,
Challenge with zero-shot AC and AS

Example anomalous circuit board
Works build on top of visual-language models (VLMs) like CLIP, benefiting from:
Compositional prompt ensemble

Using the pre-trained CLIP model, WinCLIP efficiently extracts and aggregates multi-scale spatial features aligned with language for zero-shot AS

WinCLIP Pipeline
WinCLIP+ combines the language-guided and vision-only methods for few-normal-shot anomaly recognition, using a reference association method
WinCLIP and WinCLIP+ results on MVTec-AD and VisA
Given an input $x$ and a closed set of free-form texts $S = \set{s_1, …, s_k}$, CLIP can perform zero-shot classification $p(s=s_i | x; s \in S) := \frac{exp(\langle f(x), g(s_i) \rangle / \tau)}{\sum_{s \in S}{exp(\langle f(x), g(s_i) \rangle / \tau)}}$
normal [o]''}$ vs. $\text{[c] - anomalous [o]''}$
$\text{[o]}$ is an object-level label
Also a one-class design $s := \text{normal [o]''}$ to define anomaly score as $- \langle f(x), g(s\_) \rangle''$
two-class design outperforms one-class design significantlyCompositional prompt ensemble (CPE) combinations of pre-defined lists of a) state words per label and b) text templates compute the average of text embeddings per label to represent the normal and anomalous classes CPEs are more aligned with images in CLIP’s joint embedding space for anomaly tasks, than CLIP prompt ensemble that does not explain object labels Denote the zero-shot scoring model with CPE as $ascore_0 : \R_d \to [0,1]$ for an image embedding $f(x)$
WINCLIP for zero-shot AS Given an image $x \in \mathcal{X}$, of size $h \times w$ and an image encoder $f$, WINCLIP obtains a map of $d$-dim feature map $F^W \in \R^{h \times w \times d}$ as follows,

WinCLIP multi-scale feature extraction through CLIP encoder
Dense features $F^W$ from WinCLIP are more aligned with language than penultimate feature map, the last feature map before pooling $F^P$
For each local window, $M^W_{0,ij}$ is similarity between the window feature $F^W_{ij}$ and text embeddings from CPE. This score is distributed to every pixel of the local window. At each pixel, multiple scores from all overlapping windows are aggregated to improve segmentation by harmonic averaging, weighing more on scores towards normality prediction (zero) $\bar{M}^{W}{0,ij} := (\frac{1}{\sum{u,v}{(w_{uv}){ij}}} \sum{u,v}{\frac{(w_{uv}){ij}}{M^W{0,uv}}})^{-1}$ $k$ is the kernel size that corresponds to the amount of surrounding context for each location in computing WinCLIP features. Multi-sacle features are aggregated: (a) small 32x32 pixels (b) mid 48x48 pixels (c) image-scale features (ViT class token)
WinCLIP+ and reference association
