Create a Machine Learning Model
Create a machine learning model in Cortex XSOAR 6.13 for phishing classification.
A machine learning model enables Cortex XSOAR to predict the classification of phishing incidents. For example, whether the incident should be classified as legitimate, malicious, or Spam. You can use these models in conjunction with your default investigation playbooks, or run commands separately in the War Room. It is usually used for training a model to predict the classification of a phishing incident. The main goal of the machine learning model is leveraging past phishing incidents to assist with the investigation of future incidents.
Select Settings → Advanced → ML Models → New Model.
Define the Incidents Training Set Scope.
In the Model name field, type the name of the model that you want to create.
(Optional) In the Description field, type a meaningful description for the model.
To choose which incidents are to be used for training the model, in the Incident type field, from the dropdown list, select the type of incident for training, such as Phishing.
Select the date range from which incidents will be used for the training set. The more incidents, the better the expected results. It is recommended to use a longer period.
Select the field for which you want the model to learn to predict.
In the Incident field from the dropdown list, select the relevant field.
The Incident Field (classification field), stores the classification of the incident. This is a single select field, where the classification or the closed reason of incidents are stored. The out of the box fields are “Email Classification” or “Close Reason”, but you can use any other custom field.
After selecting the Incident field in the Field Values field, you can see the different values of classifications and the number of values across the selected incidents scope of incidents.
ml-step3.png
Set the final classification values.
In the Verdict columns, define the names of the verdicts for mapping your existing classification values.
This stage allows you to control which incidents’ classifications will be used in the training, and also merge multiple classifications into a single category. Verdict is a group of classifications values, for which each verdict includes one classification or more. The trained model predicts each new incident as one of those verdicts.
Map your data by associating the verdict with your defined classification values by dragging and dropping the Field Values into the respective Verdict fields.
ml-step4.png Where values remain in the Field Values column, their corresponding incidents are not involved in the training. You may want to leave classifications such as Undetermined, Internal Phishing Test, or any other classifications that you do not want to participate in the training. For example:
ml-step4.2.png It is possible to drag multiple classifications values into a single verdict. If so, the model treats all the classification values under the same verdict as if they had the same classification. This allows you to better define the prediction task of the model and merge some smaller groups into a single group.
This might be helpful if you have different sub-types of classifications. For example, if you have classification values of Spear Phishing, Malware, and Ransomware, you may want to map them all into a single verdict called Phishing. If you want to have a model which distinguishes between one classification and the rest (for instance, if you want to train a model which distinguishes between phishing and the rest of the classifications, you can map all other classifications other than phishing into a single verdict called “Non-Phishing”). In the following example we have 2 verdicts, one has phishing, the other has everything other than phishing:
ml-step4.3.png You can have 2-3 different verdicts, where each verdict needs a minimum of 50 incidents for each. For an example, see Machine Learning Model Example.
(Optional) Change any of the Advanced training configuration settings.
In the Query incidents to include in training field, add a query to include specific incidents in the training model.
For example, if you use the phishing classifier to close incidents automatically without a manual review, you can train the model to include only those incidents that were not closed automatically. If you define the query as
closenotes: x, only incidents where the Close Notes field equalsxwill be used for the training model. You may also want to train a classifier from a playbook include specific incidents based on a query.Use the Incident page to test your query and then copy it to this field.
If your environment contains phishing incidents with multiple languages, and you want to train a machine learning model using incidents of one specific language, you can select a Language. Incidents recognized as being in different languages will be filtered out of training.
Select a Training algorithm.
From Scratch. Train a new model based only on your incidents.
Fine Tune. Use your incidents to better adjust a model that is already partially trained.
Auto. One of the two aforementioned algorithm options is automatically selected based on the number of incidents provided.
In general, From Scratch is designed for use with a larger number of incidents, while Fine Tune is designed for use with a smaller number of incidents. If you’re not sure which option to choose, you can leave the default value of Auto selected, or you can train different models using different algorithms to test which model achieves better results.
In the Maximum number of incidents to test field, type the number of incidents that will be used to train the model.
Reduce the number only if the number of incidents is too large and causes performance problems. Use a higher number if you have more samples in your environment. Default is 3000.
In the Argument Mapping select the equivalent fields for Email body, Email HTML and Email subject.
By default, training is done based on the Email body, Email HTML, and Email subject.
Train the model by clicking Start Training.
You will be redirected back to the Machine Learning Models page, and the training process takes several minutes (it is possible to close the page).
If training is completed successfully, the percentage scores appear, which reflect the precision of the model of the different verdicts.
(Optional) View detailed performance information of the model.
Expand the results information by clicking + to the model name.
View a detailed evaluation, by clicking Evaluation of model performance.
A window opens showing a detailed evaluation of the model, which enables you to decide whether and how to use the trained model. You can see a detailed breakdown, showing what is the expected performance of the model for each class, displaying different metrics, such as precision, coverage, suggestions for applying a confidence threshold, etc.
ml-report.png If using the phishing incident type, you can now use model in the machine learning or War Room window or in the playbook. For more information, see Machine Learning Models.
Machine Learning Model Example
In this example, we want to create a machine learning model for Phishing using a customlabelfield. The customlabelfield manually classifies data as phishing, spam, or legit.
From the New ML Model window, in the Incident type field, select Phishing.
In the Incident field, select the field which contains the label you want the model to train. Assume that the field which contains the type of the mail is called customlabelfield.
In the Field Values column, the following data is returned:
ml-model.png Drag and drop the data from Field Values column into the relevant Verdict columns.
ml_field_map2.png In the Argument mapping field keep the arguments as the default. We want the machine learning model to train on Email body, HTML and Subject.
Click Start Training.
The model starts training and when successful returns the following information:
ml_results.png
The returned data shows that it found 3 categories together with the percentage scores, which reflect the precision of the results.
You can now use the machine learning model in the Phishing - Generic v3 playbook, in the Machine Learning page or in the War Room. For examples of how to use it in the War Room, see Phishing Command Examples Using a Machine Learning Model.
Phishing Command Examples Using a Machine Learning Model
In this example, we have created a machine learning model, called “demoModel” that predicts the following:
For an example how to create the machine learning model, see Machine Learning Model Example.
After running the command, Cortex XSOAR returns the following information:
TextTokensHighlighted: The text of the email message with the highlighted positive words (if found).
Label: The predicted label found by the model.
Probability: The prediction probability.
PositiveWords: Words that encouraged the model to make the prediction.
NegativeWords: Words that are in general not correlated with the predicted class and reduced the model’s confidence in its prediction.
In the War Room, run the following commands:
!DBotPredictPhishingWords modelName="demoModel" emailBody=”Your email account was LOGIN today by Unknown IP address: 10.240.180.228, click on UPDATE <http://helpd.moonfruit.com/> to validate and verify your email account now to avoid Outlook Web App been disabled for user”
!DBotPredictPhishingWords modelName="demoModel" emailBody=“Your Outlook Exceeded its storage limit Click here <https://docs.google.com/forms/d/e/1FAIpQLSckF75SUgErVFmTEfHhhFkiX2-4V2tgC0nssDvpkqZnPz4pkQ/viewform> fill and SUBMIT for more space or you wont be able to send Mail.”
!DBotPredictPhishingWords modelName="demoModel" emailBody=“Dear member, the credit card we have on file for your PayPal service was declined when we attempted to bill you for your most recent service fees. For this reason, your service could be suspended. You must update your billing information immediately in order to avoid any interruption to your services”
DBotPredictPhishingWords modelName="demoModel" emailBody=“lose 22.5lbs in 3 weeks! flush fat away forever! free 30-day supply **http://www.adclick.ws/p.cfm?o=423&s=pk19.** to unsubscribe, click below: http://u2.azoogle.com/?z=93-1090346-62llc4”
Last updated
Was this helpful?