~ By: Michael Ades ~ Sep 11, 2025
Humans vs. Machines in Malware Classification (Topic 1.1)
Resources Used
Group 1 Slides: https://www.canva.com/design/DAGxSWoXAvs/_KXzYXFBF1uDmLWWye-AuA/edit
Topic 1.1 PDF, Slides, and Presentation Video: https://www.usenix.org/conference/usenixsecurity23/presentation/aonzo
Topics
Here’s a list of topics required knowledge of in this chapter. malware goodware machine learning random forests convolutional neural networks static analysis dynamic analysis
Abstract
The paper classifies the differences between “human and machine intelligence in malware analysis.” An experimental study was performed to identify how human novices and experts would compare when identifying the important features of malware. The results showed that both groups picked almost identical features when looking at different malware samples.
Problem
The sheer mass of samples that an antivirus analyzes per day is “over 350,000 samples.”
- “41% of all malware families” were entirely novel as well → This presents an issue for us: Humans just cannot feasibly analyze all of these malware samples individually and must rely on machine learning
Solution
Use of machine learning to automate classification of malware.
- “Some models report 96% accuracy” → But we must identify what features the machine learning algorithms use to detect the difference between goodware and malware.
During the classification of malware samples, we can split it up into two groups: static analysis and dynamic analysis
Results
The results showed a lot of overlap between human intuition and ML models.
- Humans pointed to imports, strings, and PE section metadata.
- The ML models highlighted very similar features when classifying samples.
- Both static and dynamic features (like registry keys, DLL loads, and processes) were seen as strong indicators of malicious behavior.
In terms of the performance and differences between humans and ML:
- Accuracy: random forests and convolutional neural networks models achieved over 90% accuracy which was above human classification consistency.
- Explanations: Humans were better at articulating why a feature mattered, while machines excelled at scaling up analysis across thousands of samples.
My Opinion
I think machines and humans should complement each other rather than compete. While ML is superior with its speed or scale when it comes to thousands of samples, it is vulnerable to adversarial evasion (for example, obfuscation of imports or packing techniques).
This aligns with the authors argument on the paper that we should let ML models triage massive datasets while humans validate and investigate high-value or ambiguous cases.
We should employ a tiered system in my opinion where ML handles bulk classification for samples but people will review edge cases and novel samples. This would improve the accuracy of identification of unique malware that would otherwise slip past the ML models.
Malware Detection on Highly Imbalanced Data through Sequence Modeling (Topic 1.2)
Resources Used
Group 2 Slides: https://canvas.tamu.edu/courses/405410# Topic 1.2 Paper: https://dl.acm.org/doi/10.1145/3338501.3357374
Topics
Here’s a list of topics you should be familiar with for this paper. Many overlap with Topic 1.1.
malware
goodware
machine learning
dynamic analysis
sequence modeling
imbalanced datasets
deep learning
LSTM
Abstract
This paper studies Android malware detection under highly imbalanced datasets through sequence modeling. The authors analyze the sequences of app activity specifically dynamic and apply deep learning methods that preserve ordering like LSTM to detect malware when malware samples are a small fraction of all apps. They show that sequence models perform better than simpler baselines, especially when the malware percentage is very low.
Problem
- In realistic Android datasets, malware is very rare only about 0.01-2% in apps.
- Most ML models do poorly when analyzing malware samples vs goodware: they may get high accuracy simply by labeling everything as benign. False negatives (missed malware) cause a lot of issues with the model.
- Need for methods that can detect malware reliably even when there is severe class imbalance. → Basically there is just not enough training data for malware.
Solution
- Use dynamic analysis to extract activity sequences of Android apps, the order and type of activities the app performs, rather than just static features.
- Apply sequence modeling with deep learning using LSTM, or Transformer models like BERT that can take into account context and ordering in the app behavior.
- Use sampling strategies and training methods that handle imbalanced data well (sampling majority class, over-sampling minority class, possibly cost-sensitive losses.
Results
- Sequence models (LSTM or Transformer‐based) outperform non‐sequence baselines in detecting malware, especially when malware is rare.
- Even when malware is <0.5% of the dataset, they get good detection rates (good recall) while keeping false positives not too bad.
- Preserving order of actions for the behavior of the malware is very important with sequence modeling and just looking at features will miss the overall context of the sample.
My Opinion
Preserving the sequence in dynamic behavior gives an advantage as opposed to just looking at static features. However, dynamic analysis is more resource-intensive and we would need to simulate the execution. I think this approach fills an important gap compared to what we learned in Topic 1.1 in Humans vs Machines most of the focus was given to static features and comparing human‐picked features vs ML, this paper focus more on dynamic, temporal behavior, and models that can pick up patterns over time.
We need to focus on properly splitting training data and using deep learning. We need to have a balance between malware and goodware while training to minimize false positives.
Thoughts
- Sequence modeling helps for malware that tries to hide malicious behavior until certain conditions (triggered behaviors) for example checking a debugging environment. Static features may be defeated by obfuscating the malware sample.
- But dynamic analysis + sequence models will likely have tradeoffs in speed, cost, and maybe adversarial evasion (hiding malicious behavior or delaying it)
Connection with Topic 1.1
- Topic 1.1 studied which features static/dynamic human experts, novices, and ML pick as a priority. This paper builds on that by focusing not just which features but how features happen over time basically the sequence of these samples.
- Both papers focus primarily on dynamic features, but this one shows dynamic + ordering matters!!! Topic 1.1 mentioned features like processes, registry, DLLs.
- Also, the importance of understanding what features ML uses from Topic 1.1 connects here sequence modeling makes features harder to interpret, so we might need interpretability tools for sequence models too.