SlideServe
Télécharger Obtenir Pro
Menu
Accueil Pro
Préférences
ApparenceLight Aide et commentaires
© SlideServe · v2026

Anomaly Detection in Network Security: Understanding Data Regularity and Its Impact

cian avatar
cian
27 diapositives · 405 Views · Jul 13, 2014

This paper explores the impact of data regularity on anomaly detection systems within network security. It discusses a framework for generating training and testing datasets, injecting anomalies, and the significance of data structure. Key findings emphasize that different environments necessitate tailored anomaly detection methodologies due to varying levels of data regularity. The study presents diverse types of anomalies and analyzes how background noise, regularity metrics, and detection thresholds influence scoring and effectiveness. Ultimately, it highlights the need for intelligent detection systems to adapt to structural data changes.

Anomaly Detection in Network Security: Understanding Data Regularity and Its Impact - slide 1
01 / 27Anomaly Detection in Network Security: Understanding Data Regularity and Its Impact
1 / 17

Télécharger la présentation

Anomaly Detection in Network Security: Understanding Data Regularity and Its Impact
Une image / un lien ci-dessous est fourni (tel quel) pour télécharger la présentation

Politique de téléchargement : le contenu du site vous est fourni TEL QUEL pour votre information et votre usage personnel ; il ne peut être vendu, concédé sous licence ni partagé sur d’autres sites sans l’accord de son auteur.Le contenu vous est fourni TEL QUEL, pour votre information et votre usage personnel uniquement. Téléchargez la présentation en cliquant sur ce lien. Lors du téléchargement, si vous ne parvenez pas à télécharger une présentation, l’éditeur a peut-être supprimé le fichier de son serveur.Pendant le téléchargement, si vous ne parvenez pas à obtenir une présentation, le fichier a peut-être été supprimé par l’éditeur.

F I N

Plus de contenu associé

Transcription de la présentation

  1. Benchmarking Anomaly-based Detection Systems Ashish Gupta Network Security May 2004

  2. Overview • The Motivation for this paper • Waldo example • The approach • Structure in data • Generating the data and anomalies • Injecting anomalies • Results • Training and Testing: the method • Scoring • Presentation • The ROC curves: somewhat obvious

  3. Motivation Does anomaly detection depend on regularity/randomness of data ?

  4. Where’s Waldo !

  5. Where’s Waldo !

  6. Where’s Waldo !

  7. The aim • Hypothesis: • Differences in data regularity affect anomaly detection • Different environments  different regularity • Regularity • Highly redundant or random ? • Example of environment’s affect 010101010101010101010101 Or 0100011000101000100100101

  8. Consequences One IDS : Different False Alarm Rates Need custom system/training for each environment ? Temporal affects: Regularity may vary over time ?

  9. Structure in data Measuring randomness

  10. 010101010101010101010101 Or 0100011000101000100100101 Measuring Randomness + Relative Entropy Sequential Dependence Conditional Relative Entropy

  11. The benchmark datasets • Three types: • Training data ( the background data) • Anomalies • Testing data ( background + anomalies ) • Generating the sequences • 5 sets, each set  11 files ( for increasing regularity) • Each set  different alphabet size • Alphabet size  decides complexity

  12. Anomaly Generation • What’s a surprise ? • Different from the expected probability • Types: • Juxta-positional : different arrangements of data • 001001001001001001111 • Temporal • Unexpected periodicities • Other types ?

  13. Types in this paper • Foreign symbol • AAABABBBABABCBBABABBA • Foreign n-gram • AAABABAABAABAAABBBBA • Rare n-gram • AABBBABBBABBBABBBABBBABBAA

  14. Injecting anomalies • Make sure not more than 0.24 %

  15. The experiments The Hypothesis is true

  16. The hypothesis: • Nature of “normal” background noise affects signal detection • The anomaly detector • To detect anomalous subsequences • Learning phase  n-gram probability table • Unexpected event  anomaly ! • Anomaly threshold decides level of surprise

  17. Example of anomaly detection AAC  ANOMALY !

  18. Scoring • Event outcomes • Hits • Misses • False alarms • Threshold • Decides level of surprise • 0  completely unsurprising, 1  astonishing • Need to calibrate

  19. Presentation of results • Presents two aspects: • % correct detections • % false detections • Detector operates through a range of sensitivities • Higher sensitivity  ? • Need the right sensitivity

  20. Interpretation • Nothing overlaps  regularity affects detection !

  21. What does this mean ? • Detection metrics are data dependent • Cannot say: • My XYZ product will flag down 75% percent anomalies with 10% false hit rate ! • Sir, are you sure ?

  22. Real world data • Regularity index for system calls for different users

  23. Is this surprising ? • What about network traffic ?

  24. Conclusions Anomaly Detection Effectiveness Data Structure Evaluation is data dependent

  25. Conclusions Different system Or Change the parameters Change in regularity

  26. Quirks ? • Assumes rather naïve detection systems • “Simple retraining will not suffice” • An intelligent detection can take this into account. • What is really an anomaly ? • If data is highly irregular, won’t randomness produce some anomalies by itself • Anomaly is a relative term • Here anomalies are generated independently