Olivia A. Gallucci

Can a simple decision tree catch network attacks as well as a neural network? To find out, group 4 (Derek Chan and I, under Dr. Leon Reznik) built an intrusion detection system (IDS) two ways for RIT’s Introduction to Intelligent Security Systems course: once with decision trees and baseline thresholds (part I), and once with a neural network (part II). Both versions used the CIC IDS 2017 dataset and had two modes: misuse detection, which matches known attacks, and anomaly detection, which flags anything that strays from normal traffic.

IDS specification

Part I

The goal is to distinguish regular network connections from potential attacks in the data from the Canadian Institute for Cybersecurity (CIC) IDS 2017 Dataset. The techniques below could be generalized to an Intrusion Detection and Prevention System (IDPS), but this report focuses on the provided dataset.

The group’s anomaly-based IDS uses the training data to establish a baseline for normal network traffic. The program flags a connection as anomalous based on how far it deviates from this baseline. The misuse-based IDS uses a decision tree to assess the attributes in the data and decide whether a connection is anomalous.

To assess the effectiveness of the filtering, the program measures its accuracy and misclassification rates. The goal: catch as many attacks as possible (high true positive rate) with few false alarms (low false positive rate).

Part II

The group trained a neural network using the CIC IDS 2017 Dataset. With the neural network, the group developed a misuse detection system to independently identify five unique attack types. This process resulted in five models; each one specialized in detecting its respective attack. Then, the group used the neural network to determine multiple attack types with one training set. Unlike part I, a model was generated for each attack type, so that the IDS could detect all five attacks separately. Lastly, the group built an anomaly detection system to recognize benign and anomalous traffic patterns. The goal was to train it so that it could recognize novel malicious traffic.

Methods and techniques

Data preparation

Design

During data preparation, the program fixes inconsistent column names, duplicate columns, and missing values. It also applies min-max scaling so all features have the same scale. It filters out the ‘Infiltration’ and ‘Heartbleed’ attack types, which lack enough data for classification; these attacks are given to the IDS as unseen data to gauge how well it detects malicious attacks.

Operation

The program reads the raw CSVs, applies the cleanup and min-max scaling above, and relabels the data based on ‘idsType’: either all attacks as abnormal or each attack type as its own label. It then splits the data into training and testing sets, with a balanced mix of benign and attack data in the training set, and converts both to ARFF, the format Weka reads.

Misuse IDS

Design

For the misuse IDS, the program creates five ‘seen’ attack labels and a ‘BENIGN’ label containing normal packet data and unseen attack packet data. Then, it groups each datapoint into the correct label.

For example, all the attacks originally labeled ‘Web Attack – Brute Force,’ ‘Web Attack – XSS,’ or ‘Web Attack – Sql Injection’ are grouped into the ‘WebAttack’ label. Simplifying attack labels and separating them from benign data makes the dataset suitable for supervised machine learning, which helps detect well-defined attack patterns but not novel threats; flagging those is the anomaly IDS’s job.

Operation

With uniform labels, the model learns each known attack. During operation, it flags traffic that matches those learned patterns as malicious. The group tested it with the filtered-out attacks as ‘unseen’ data.

Anomaly IDS

Design

Regarding anomaly IDS, the program treats all attacks as abnormal behavior, categorizing them as ‘abnormal,’ while benign activities are labeled BENIGN. Anomaly IDS focuses on identifying deviations from normal behavior. Combining attack types into a single ‘anomaly’ category simplifies detection to finding unusual patterns or outliers in the data. This helps uncover novel attacks that are not well-defined in advance.

Operation

During operation, the anomaly IDS flags any traffic that strays from the learned baseline, including patterns it never saw in training.

Neural network

The group used a multilayer perceptron (MLP) in Weka—which provides tools for training and using MLPs—to tackle a machine learning IDS.

An MLP is a feedforward neural network for supervised learning: an input layer, one or more hidden layers, and an output layer, with every node connected to every node in the next layer. Training adjusts the weights on those connections.

Depending on the test and objective, the group’s MLP used a neural network with various hidden layers and nodes in each layer. Here is one of the configurations we used within Weka:

weka.classifiers.functions.MultilayerPerceptron -L 0.3 -M 0.2 
-N 500 -V 0 -S 0 -E 20 -H a

Hyperparameters and options

The hyperparameters and options work as follows:

•  -L 0.3: Sets the learning rate to 0.3. The learning rate is a hyperparameter that controls the step size during the weight update process in training the neural network. It affects the convergence and stability of training.
•  -M 0.2: Sets the momentum to 0.2. Momentum is a hyperparameter that helps the optimization process by adding a fraction of the previous weight update to the current weight update. It can improve the convergence of the training process.
•  -N 500: Sets the number of training epochs to 500. An epoch is one complete pass through the training data. Training for a fixed number of epochs can help prevent overfitting, as the model stops learning after a certain point.
•  -V 0: Sets the validation set size to 0. The validation set is used for early stopping, which can prevent overfitting.
•  -S 0: Sets the seed for the random number generator to 0. Setting the seed ensures the training process is reproducible, as using the same seed will produce the same random weights initialization and data shuffling.
•  -E 20: This is the consecutive number of errors allowed for validation testing before the network terminates.
•  -H a: Determines the number of neurons in the hidden layers. The a specifies the number of neurons based on the number of attributes and the number of classes. The group has this information embedded into each ARFF file.

In short, the command configures the MultilayerPerceptron classifier with a specific learning rate, momentum, number of training epochs, and a hidden layer architecture determined automatically by the algorithm. It uses a seed of 0 for reproducibility and no separate validation set. The algorithm determines the architecture of the hidden layers based on each dataset the group used.

Parameter summary

Aspect	Effect	Impact on FP and FN rates
Number of Neurons in Hidden Layers	Increasing improves capacity but may cause overfitting	May reduce false negatives but increase false positives (overfitting risk)
Learning Rate	Determines step size in weight updates; affects convergence	Too high may lead to overshooting, increasing false positives/negatives; too low may cause slow convergence
Number of Hidden Layers	Deeper architectures capture hierarchical features; risk of overfitting	May reduce false negatives but increase false positives (overfitting risk)
Activation Functions	Choice influences information flow; ReLU mitigates vanishing gradients	Can influence the model’s ability to capture complex patterns
Regularization Techniques (e.g., Dropout)	Prevents overfitting by dropping out neurons	Can reduce overfitting, potentially improving generalization and reducing false positives
Batch Size	Determines training examples in each iteration; affects convergence	Small batches may introduce noise, affecting convergence and potentially increasing false positives/negatives; larger batches may provide stable updates but slower convergence
Weight Initialization	Proper initialization aids convergence and avoids vanishing/exploding gradients	Well-initialized weights contribute to stable training, potentially reducing the risk of false positives/negatives

Table: Aspects to MLP and their corresponding effects on FP and FN rates.

Impact of MLP parameters on false positive and false negative ratios

More neurons in the hidden layers improve the model’s capacity to discern complex patterns, potentially reducing false negatives. However, they also risk overfitting, which might increase false positives.

The learning rate sets the step size during weight updates. A learning rate that is too high may speed up convergence but overshoot, increasing false positives and negatives. A rate that is too low can slow convergence and hurt model performance.

Deeper architectures, with more hidden layers, can capture hierarchical features and reduce false negatives. However, vanishing and exploding gradients and overfitting may increase false positives.

Regularization techniques, such as dropout, prevent overfitting by randomly dropping neurons during training. Weka’s MultilayerPerceptron has no dropout option, so the group’s runs didn’t use it.

Batch size determines the number of training examples per iteration. Small batch sizes may introduce noise, while larger batch sizes offer stability but might result in slower convergence. The group kept Weka’s default.

Finally, weight initialization affects the MLP’s outcomes. Well-initialized weights contribute to stable training and more efficient convergence.

Implementation

The Python script cleans the CSVs and writes ARFF files for Weka. It uses pandas, numpy, sklearn, and concurrent.futures for data manipulation and parallel processing.

The first thing the program does is concurrently read in raw data from multiple CSV files using pandas read_csv function. This function parses the first row of the CSV as column names and the rest of the CSV as data corresponding to those columns. The result of this read function is a pandas data frame object representing the data in the CSV.

The next step is the filtering and normalization of data. Using the pandas rename function, this program removes all the extra whitespace from the column names. Then, using the pandas drop function, this program removes any duplicate and unnecessary data features. Using the pandas min and max function, the program normalizes the data between the values of 0 and 1 based on the minimum and maximum value of each column. After the min-max normalization, the program results with NaNs if the minimum and maximum values in the column are the same; to fix this, it uses the pandas fillna function, which replaces any NaNs in the data with a 0.

Data grouping

After the filtering and normalization of the data, the way the program groups the data is based on whether it is creating an anomaly dataset or a misuse dataset. In either case, the program uses the pandas apply function. The difference comes when the program marks all attacks as abnormal in the anomaly case or groups similar attacks together in the misuse case. After that, the program creates a dictionary that groups by type of attack label, including benign. This is done through the pandas groupby function, which automatically groups data points by different values given a column name.

With this dictionary of grouped data points, the program uses sklearn’s train_test_split function to randomly select data points without replacement. The number of datapoints selected is provided by the user. In Group 4’s program, eight-thousand benign data points, and two-hundred of each attack datapoint was specified. The test set is a quarter the size of the training set (an 80/20 split). The data points selected for training and testing are mutually exclusive.

ARFF generation

Finally, the program generates the testing and training ARFF files. First, the headers are written with the specifications provided by the University of Waikato. Then the data is shuffled using pandas sample function and written to a file using numpy’s savetxt function. Each line of comma-separated values represents an individual data point.

The program lacks a user interface; it is intended to be run from the command line, taking command-line arguments for customization. Users need to specify parameters such as the type of intrusion detection system (anomaly or misuse), the maximum number of threads for parallel processing, the directory containing CSV files, and quantities for training data.

To run the script, navigate to the location of the script and run:

python3 preprocessing.py [--idsType IDSTYPE] [--attackType [ATTACKTYPE]] [--unprocessedDataPath UNPROCESSEDDATAPATH] [--benignTrainingQty BENIGNTRAININGQTY] [--eachAttackTrainingQty EACHATTACKTRAININGQTY] [--maxThreads MAXTHREADS]

The attackType parameter is optional and multiple attacks can be specified like attack1_attack2_attack3. For example, the results for

python preprocessing.py -idst misuse -a Bot_DDoS_DoSHulk_WebAttack_PortScan -t 8 -dp ./data/unprocessedCSVs -bq 8000 -eaq 200

will be in data/processedCSVs with the base directory as the same as the directory of the script.

Limitations of the software include its command-line interface, which might not be user-friendly for non-technical users. It also assumes a basic understanding of the dataset structure and preprocessing requirements. It also lacks extensive error handling for edge cases.

Software and hardware requirements include a Python environment with the required libraries installed, and access to the CIC IDS 2017 Dataset in CSV format.

Weka user guide

Data analysis

Additional module in part I

1. Loading data
   1. Click on the “Explorer” button in the Weka GUI Chooser.
   2. In the “Preprocess” tab, click the “Open file” button to load your dataset.
   3. Weka supports various data formats, including ARFF, CSV, and more.
2. Data preprocessing
   1. After loading your data, you can preprocess it using the “Preprocess” tab.
   2. Explore options for data cleaning, transformation, and attribute selection.
3. Building a Machine Learning Model
   1. Go to the “Classify” tab to build a machine learning model.
   2. Select a classification algorithm from the left panel (e.g., J48, Random Forest, Naive Bayes).
   3. Configure algorithm parameters and choose the target class attribute.
   4. Click the “Start” button to build the model.
4. Model evaluation
   1. After the model is built, evaluate its performance on your dataset.
   2. Go to the “Classify” tab and use the “Supplied test set” option to load your test data (if available).
   3. Click the “Start” button to evaluate the model’s accuracy and other metrics.

Artificial neural networks

Additional module in part II.

1. Loading Data
   1. Click on the “Explorer” button in the Weka GUI Chooser.
   2.  In the “Preprocess” tab, click the “Open file” button to load your dataset.
   3. Weka supports various data formats, including ARFF, CSV, and more.
2. Attribute Selection
   1. After loading your data, you can preprocess it using the “Preprocess” tab.
   2. Explore options for data cleaning, transformation, and attribute selection. If your dataset contains many attributes, you may want to perform attribute selection or dimensionality reduction before training your multilayer perceptron.
3. Select, Train, and Test the Multilayer Perceptron Algorithm & Model
   1. In the Weka Explorer, go to the “Classify” tab. Note that you may also use it in Java code if you are using Weka programmatically.
   2. Select the MultilayerPerceptron classifier for training a multilayer perceptron.
   3. Configure the hyperparameters, such as the number of hidden layers, the number of nodes in each hidden layer, learning rate, and momentum.
   4. Click the “Supplied test set” button and select your testing data.
   5. Click the “Start” button to build, train, and test the model.

Tests

The group used the CIC IDS 2017 IDS Dataset to test the IDS systems. The threshold for the anomaly-based detection system was determined through trial and error using an ROC curve, which returns the TP rate over the FP rate.

Then, the group generated a receiver-operating characteristic curve for the anomaly detection system, determining the optimal threshold. Lastly, the group used a separate dataset to validate and test the MLP neural network. First, the group trained models to distinguish a specific attack from other traffic and computed a confusion matrix. The group repeated these procedures for five attacks.

For every step of testing, the group cross-checked the program’s results by verifying with a test set of data unseen by the classifier. True positives plus false negatives matched the number of attacks in the test set, so every attack was counted.

DDoS

( n = 2050 )	Predicted Normal	Predicted Anomalous
Actually Normal	1995	5
Actually Anomalous	1	49

• Model Accuracy: 99.71%
• Average Training Time: 250.24 seconds
• Memory Footprint: 2.22 GB

DoSHulk

( n = 2050 )	Predicted Normal	Predicted Anomalous
Actually Normal	2000	0
Actually Anomalous	22	28

• Model Accuracy: 98.93%
• Average Training Time: 248.57 seconds
• Memory Footprint: 2.34 GB

Web Attack

( n = 2050 )	Predicted Normal	Predicted Anomalous
Actually Normal	1997	3
Actually Anomalous	10	40

• Model Accuracy: 99.37%
• Average Training Time: 245.68 seconds
• Memory Footprint: 2.15 GB

FTP-Patator

( n = 2050 )	Predicted Normal	Predicted Anomalous
Actually Normal	1983	17
Actually Anomalous	1	49

• Model Accuracy: 99.12%
• Average Training Time: 251.13 seconds
• Memory Footprint: 2.21 GB

Port Scan

( n = 2050 )	Predicted Normal	Predicted Anomalous
Actually Normal	1974	26
Actually Anomalous	0	50

• Model Accuracy: 98.73%
• Average Training Time: 248.83 seconds
• Memory Footprint: 2.24 GB

Parameter effects using group 4’s anomalous dataset

The group also experimented with changing the layers within the MLP neural network.

Layers	False Negative Rate	False Positive Rate
20	0.500	0.000
50	0.380	0.001
100	0.380	0.001

Table 1: Layer amount in comparison to False Negatives and False Positives

IDS results

Part I

The program normalizes, preprocesses, and balances the dataset. The misuse IDS matches known attack patterns, and the anomaly IDS flags deviations from established baselines, so it can catch unseen threats.

Accuracy =

Misclassification Rate =

Precision =

True Positive Rate =

False Positive Rate =

Summarized results

( n = 2050 )	Predicted Normal	Predicted Anomalous
Actually Normal	1990	10
Actually Anomalous	8	42

Part II

Using the MLP neural network and the methods described above, the program’s updated statistics are as follows:

Accuracy =

Misclassification Rate =

Precision =

True Positive Rate =

False Positive Rate =

Summarized results

( n = 2050 )	Predicted Normal	Predicted Anomalous
Actually Normal	1998	2
Actually Anomalous	19	31

Work distribution

The group divided the research, Python programming (including data preprocessing, misuse IDS, and anomaly IDS), and documentation. The group debugged together and checked in every three to four days to discuss progress and challenges. For version control, the group used GitHub, with repository settings that required pull requests to prevent accidental pushes to the main branch.

In total, it took the group around fifteen hours for research, forty-two hours for development, and twenty-two hours for documentation.

Notes 

Note that this post was made for the Introduction to Intelligent Security Systems (CSCI 532) course at the Rochester Institute of Technology. 

The course will introduce students into the current state of an application of intelligent methodologies in computer security and information assurance systems design. It will review different application areas such as intrusion detection and monitoring systems, access control and biological authentication, firewall structure and design. The students will be required to implement a course project on design of a particular security tool with an application of an artificial intelligence methodology and to undertake its performance analysis. 

If you enjoyed this post on IDS security, consider reading Fine-Tuning LLMs: Pre-trained Transformers with Python or my other research.