Today, we will look at the basic approaches to anomaly detection.
Anomaly detection is the process of identifying rare observations or patterns that are significantly different from majority of data. Examples could be unusual log behavior of social media account (probably hacking), credit card fraud, or weird signals from machines. Two important features of anomaly detection tasks are :
-
Class Imbalance : there is much less anomalies compared to normal data so class imbalance is a serious issue when using classifiers
-
Anomalies are significantly different from normal data and defining this level of discrepency is difficult
To handle these problems, supervised learning (that requires labels of all samples) has evolved into semi-supervised where only normal samples are used. This is called One-Class Classification and we will return to this later. Semi-supervised approach currently is the most-researched one.
Unsupervised learning is also applicable, where no labels are required. It learns how to describe data by itself and eventually divides it into abnormals and normals. In this area, Autoencoder(AE) is most frequently used.
Upcoming :
- [Paper Review] Deep Learning for Anomaly Detection: A Survey (Chalapathy, 2019) Link