LLMs for Anomaly Detection

Introduction

  1. its generalizability, as the text-image pairs capture broad concepts
  2. without additional fine-tuning, text prompts can be used to extract knowledge from VLMs for zero/few shot transfer to downstream tasks
  1. helps clarify context-dependent states
  2. additional information to distinguish defects from acceptable deviation from normality

WinCLIP (Window-based CLIP)

Given an input $x$ and a closed set of free-form texts $S = \set{s_1, …, s_k}$, CLIP can perform zero-shot classification $p(s=s_i | x; s \in S) := \frac{exp(\langle f(x), g(s_i) \rangle / \tau)}{\sum_{s \in S}{exp(\langle f(x), g(s_i) \rangle / \tau)}}$

Compositional prompt ensemble (CPE) combinations of pre-defined lists of a) state words per label and b) text templates compute the average of text embeddings per label to represent the normal and anomalous classes CPEs are more aligned with images in CLIP’s joint embedding space for anomaly tasks, than CLIP prompt ensemble that does not explain object labels Denote the zero-shot scoring model with CPE as $ascore_0 : \R_d \to [0,1]$ for an image embedding $f(x)$

Untitled