Yıl: 2021 Cilt: 29 Sayı: 5 Sayfa Aralığı: 2486 - 2501 Metin Dili: İngilizce DOI: 10.3906/elk-2102-89 İndeks Tarihi: 24-06-2022

A new distributed anomaly detection approach for log IDS management based on deep learning

Öz:
Today, with the rapid increase of data, the security of big data has become more important than ever for managers. However, traditional infrastructure systems cannot cope with increasingly big data that is created like an avalanche. In addition, as the existing database systems increase licensing costs per transaction, organizations using information technologies are shifting to free and open source solutions. For this reason, we propose an anomaly attack detection model on Apache Hadoop distributed file system (HDFS), which stands out in open source big data analytics, and Apache Spark, which stands out with its speed performance in analysis to reduce the costs of organizations. This model consists of four stages: the first of which is to store instant data on HDFS in a distributed manner. In the second stage, the log data generated in the network traffic are analyzed by taking the data on Apache Spark and including the log data created by HDFS. In the third stage, the data preprocessing stage and with the CUDA parallel programming in the TensorFlow library, we apply our deep learning (cuDNN) method to the distributed anomaly detection with the computational support of GPUs. In the last stage, the generated alarms are recorded on HDFS again. We conducted comparative experiments with the approach we propose to detect cyberattack anomalies in log data management with the classification methods used in machine learning. The results obtained in these experiments appear to provide a promising gain in performance evaluation metrics compared to the other available methods.
Anahtar Kelime:

Belge Türü: Makale Makale Türü: Araştırma Makalesi Erişim Türü: Erişime Açık
  • [1] Baykara M, Das R. A novel hybrid approach for detection of web-based attacks in intrusion detection systems. International Journal of Computer Networks and Applications 2017; 4 (2): 62-76. doi: 10.22247/ijcna/2017/48968
  • [2] Baykara M and Das R. SoftSwitch: a centralized honeypot-based security approach using software-defined switching for secure management of VLAN networks. Turkish Journal of Electrical Engineering & Computer Sciences 2019; 27 (5): 3309-3325. doi: 10.3906/elk-1812-86
  • [3] Zuech R, Khoshgoftaar TM, Wald R. Intrusion detection and big heterogeneous data: a survey. Journal of Big Data 2015; 2 (1): 1-41. doi: 10.1186/s40537-015-0013-4
  • [4] White T. Hadoop: The Definitive Guide. Massachusetts, USA: O’Reilly Media Incorporated Company, 2012.
  • [5] Salloum S, Dautov R, Chen X, Peng PX, Huang JZ. Big data analytics on Apache Spark. International Journal of Data Science and Analytics 2016; 1 (3): 145-64. doi: 10.1007/s41060-016-0027-9
  • [6] Chetlur S, Woolley C, Vandermersch P, Cohen J, Tran J et al. Cudnn: Efficient primitives for deep learning. Arxiv preprint 2014; arxiv: 1410.0759.
  • [7] Lunt TF. A survey of intrusion detection techniques. Computers & Security 1993; 12 (4): 405-418.
  • [8] Tan J, Pan X, Kavulya S, Gandhi R, Narasimhan P. SALSA: analyzing logs as state machines. Carnegie Mellon University Research Centers and Institutes, Pittsburgh, USA, 2008.
  • [9] Xu W, Huang L, Fox A, Patterson D, Jordan MI. Detecting large-scale system problems by mining console logs. In: Proceedings of the 27 th International Conference on Machine Learning; Haifa, Israel; 2010. pp. 117-132.
  • [10] Lee Y, Lee Y. Toward scalable internet traffic measurement and analysis with hadoop. ACM SIGCOMM Computer Communication Review 2012; 43 (1): 5-13. doi: 10.1145/2427036.14 2427038
  • [11] Suthaharan S. Big data classification: problems and challenges in network intrusion prediction with machine learning. ACM SIGMETRICS Performance Evaluation Review 2014; 41 (4): 70-73. doi: 10.1145/2627534.2627557
  • [12] Desai AS, Gaikwad DP. Real time hybrid intrusion detection system using signature matching algorithm and fuzzy-GA. In: IEEE international conference on advances in electronics communication and computer technology (ICAECCT); Pune, India; 2016. pp. 291-294.
  • [13] Marir N, Wang H, Feng G, Li B, Jia M. Distributed abnormal behavior detection approach based on deep belief network and ensemble svm using spark. IEEE Access 2018; 6: 59657-59671. doi: 10.1109/ACCESS.2018.2875045
  • [14] Karatas G, Demir O, Sahingoz OK. Deep learning in intrusion detection systems. In: International Congress on Big Data Deep Learning and Fighting Cyber Terrorism (IBIGDELFT); Ankara, Turkey; 2018. pp. 113-116.
  • [15] Lovick C. The BSD syslog Protocol. Network Working Group, Grossrinderfeld, DE, 2001.
  • [16] Stallings W. SNMP and SNMPv2: the infrastructure for network management. IEEE Communications Magazine 1998; 36 (3): 37-43. doi: 10.1109/35.663326
  • [17] Estan C, Keys K, Moore D, Varghese G. Building a better NetFlow. ACM SIGCOMM Computer Communication Review 2004; 34 (4): 245-256. doi: 10.1145/1030194.1015495
  • [18] Suriadi S, Andrews R, ter Hofstede AH, Wynn MT. Event log imperfection patterns for process mining: Towards a systematic approach to cleaning event logs. Information Systems 2017; 64: 132-150. doi: 10.1016/j.is.2016.07.011
  • [19] Rabkin A, Katz R. Chukwa: a system for reliable large-scale log collection. In: LISA’10 24th Large Installation System Administration Conference; San Diego, CA, USA; 2010. pp. 163-177.
  • [20] Groeneveld RA, Meeden G. Measuring skewness and kurtosis. Journal of the Royal Statistical Society: Series D (The Statistician) 1984; 33 (4): 391-399.
  • [21] Mardia KV. Measures of multivariate skewness and kurtosis with applications. Biometrika 1970; 57 (3): 519-530. doi: 10.1093/biomet/57.3.519
  • [22] Li C, Ren J, Huang H, Wang B, Zhu Y, Hu H. PCA and deep learning based myoelectric grasping control of a prosthetic hand. Biomedical Engineering 2018; 17 (1): 1-8. doi: 10.1186/s12938-018-0539-8
  • [23] Gulli A, Pal S. Deep learning with Keras. Birmingham, UK: Packt Publishing Ltd, 2017.
  • [24] Gardner MW, Dorling SR. Artificial neural networks (the multilayer perceptron)—a review of applications in the atmospheric sciences. Atmospheric Environment 1998; 32 (14-15): 2627-2636. doi: 10.1016/S1352-2310(97)00447-0
  • [25] Eckle K, Schmidt-Hieber J. A comparison of deep networks with ReLU activation function and linear spline-type methods. Neural Networks 2019; 110: 232-242. doi: 10.1016/j.neunet.2018.11.005
  • [26] Tavallaee M, Bagheri E, Lu W, Ghorbani AA. A detailed analysis of the KDD CUP 99 data set. In: IEEE symposium on computational intelligence for security and defense applications; Ottawa, ON, Canada; 2009. pp. 1-6.
  • [27] Sharafaldin I, Lashkari AH, Ghorbani AA. Toward generating a new intrusion detection dataset and intrusion traffic characterization. In: Proceedings of the 4th International Conference on Information Systems Security and Privacy (ICISSP 2018); Fredericton, Canada; 2018. pp. 108-116.
  • [28] Sharafaldin I, Gharib A, Lashkari AH, Ghorbani AA. Towards a reliable intrusion detection benchmark dataset. Software Networking 2018; 2018 (1): 177-200. doi:10.13052/jsn2445-9739.2017.009
  • [29] Noble WS. What is a support vector machine? Nature Biotechnology 2006; 24 (12): 1565-1567. doi: 10.1038/nbt1206-1565
  • [30] Le Roux N, Bengio Y. Representational power of restricted Boltzmann machines and deep belief networks. Neural Computation 2008; 20 (6): 1631-1649. doi: 10.1162/neco.2008.04-07-510
APA Koca M, AYDIN M, Sertbas A, Zaim A (2021). A new distributed anomaly detection approach for log IDS management based on deep learning. , 2486 - 2501. 10.3906/elk-2102-89
Chicago Koca Murat,AYDIN M.Ali,Sertbas Ahmet,Zaim Abdul Halim A new distributed anomaly detection approach for log IDS management based on deep learning. (2021): 2486 - 2501. 10.3906/elk-2102-89
MLA Koca Murat,AYDIN M.Ali,Sertbas Ahmet,Zaim Abdul Halim A new distributed anomaly detection approach for log IDS management based on deep learning. , 2021, ss.2486 - 2501. 10.3906/elk-2102-89
AMA Koca M,AYDIN M,Sertbas A,Zaim A A new distributed anomaly detection approach for log IDS management based on deep learning. . 2021; 2486 - 2501. 10.3906/elk-2102-89
Vancouver Koca M,AYDIN M,Sertbas A,Zaim A A new distributed anomaly detection approach for log IDS management based on deep learning. . 2021; 2486 - 2501. 10.3906/elk-2102-89
IEEE Koca M,AYDIN M,Sertbas A,Zaim A "A new distributed anomaly detection approach for log IDS management based on deep learning." , ss.2486 - 2501, 2021. 10.3906/elk-2102-89
ISNAD Koca, Murat vd. "A new distributed anomaly detection approach for log IDS management based on deep learning". (2021), 2486-2501. https://doi.org/10.3906/elk-2102-89
APA Koca M, AYDIN M, Sertbas A, Zaim A (2021). A new distributed anomaly detection approach for log IDS management based on deep learning. Turkish Journal of Electrical Engineering and Computer Sciences, 29(5), 2486 - 2501. 10.3906/elk-2102-89
Chicago Koca Murat,AYDIN M.Ali,Sertbas Ahmet,Zaim Abdul Halim A new distributed anomaly detection approach for log IDS management based on deep learning. Turkish Journal of Electrical Engineering and Computer Sciences 29, no.5 (2021): 2486 - 2501. 10.3906/elk-2102-89
MLA Koca Murat,AYDIN M.Ali,Sertbas Ahmet,Zaim Abdul Halim A new distributed anomaly detection approach for log IDS management based on deep learning. Turkish Journal of Electrical Engineering and Computer Sciences, vol.29, no.5, 2021, ss.2486 - 2501. 10.3906/elk-2102-89
AMA Koca M,AYDIN M,Sertbas A,Zaim A A new distributed anomaly detection approach for log IDS management based on deep learning. Turkish Journal of Electrical Engineering and Computer Sciences. 2021; 29(5): 2486 - 2501. 10.3906/elk-2102-89
Vancouver Koca M,AYDIN M,Sertbas A,Zaim A A new distributed anomaly detection approach for log IDS management based on deep learning. Turkish Journal of Electrical Engineering and Computer Sciences. 2021; 29(5): 2486 - 2501. 10.3906/elk-2102-89
IEEE Koca M,AYDIN M,Sertbas A,Zaim A "A new distributed anomaly detection approach for log IDS management based on deep learning." Turkish Journal of Electrical Engineering and Computer Sciences, 29, ss.2486 - 2501, 2021. 10.3906/elk-2102-89
ISNAD Koca, Murat vd. "A new distributed anomaly detection approach for log IDS management based on deep learning". Turkish Journal of Electrical Engineering and Computer Sciences 29/5 (2021), 2486-2501. https://doi.org/10.3906/elk-2102-89