~ By: Michael Ades
~ Sep 22, 2025
ML-in-Security: Streams, Drift & Real-World Pitfalls (Topic 2.1)
Resources Used
Ceschin et al., “Machine Learning (In) Security: A Stream of Problems,” DTRAP 2023 (arXiv:2010.16045).
Topics
machine learning data streams concept drift temporal consistency labeling class imbalance evaluation metrics
Abstract
This paper focused on the challenges of applying machine learning (ML) to cybersecurity that contains major problems like concept driftand adversarial attacks. The authors look at the entire ML pipeline (data collection to evaluation), and view cybersecurity data as a continuous data stream. They showed how many current ML-based security solutions are not viable in the real world because they fail to account for the dynamic and adversarial conditions. The paper’s main contribution is a discussion of the pitfalls, along with proposed mitigations and a checklist to guide the development of more robust / practical ML solutions for cyber.
Problem
- While ML is considered as a very good solution for many cybersecurity problems, it is very difficult to evaluate how good these solutions are because of challenges unique to security.
- The primary challenge is concept drift, where attackers evolve their methods to overcome defenses. For example the paper mentions, 68% of phishing emails blocked by Gmail are new day-to-day, and 30% of new Windows malware may belong to previously unknown families, requiring models to be constantly updated
- Other significant issues include delayed labels (it can take 20-30 days for antivirus labels to stabilize), extreme class imbalance (in the AndroZoo dataset, only about 18% of apps are malicious), and the constant threat of adversarial ML.
Solution
- They advocate for modeling security problems as data streams, which accounts for the time-based nature of threats and the need for continuous model updates
- The paper structures its analysis around a proposed ML pipeline for cyber as such:
Data Collection → Attribute Extraction → Feature Extraction → Model Training & Updating → Evaluation- Specific mitigations are suggested, such as using temporal information during data resampling to avoid leakage, including the feature extractor in the incremental learning process alongside the classifier, and evaluating models under realistic label delay conditions
- The primary deliverable is a checklist designed to help researchers and practitioners develop and review ML solutions, ensuring these common problems are addressed
Results
- Temporal Inconsistency (Data Leakage): Typical evaluation methods like k-fold cross-validation are not appropriate for security data because they train the model on “future” threats, leading to inflated accuracy.
- Concept Drift Detectors: In experiments simulating realistic scenarios with delayed labels, concept drift detectors like ADWIN performed worse than models without them. Their performance dropped a lot after just a one-day delay, as the model learns from outdated concepts.
- Feature Extraction: The paper showed that retraining the feature extractor along with the classifier when drift occurs significantly improves performance. In one experiment, this approach improved the F1-score by nearly 10 percentage points (from ~65% to 75%) on the AndroZoo dataset.
- Dataset Bias: Interestingly, models showed strong regional characteristics. A model trained on a Brazilian malware dataset performed poorly when tested on the EMBER dataset from another region, showing a much higher false-negative rate
My Opinion
I found that this paper provided a lot of context into how we should treat security data as a dynamic and evolving stream. It exposed a lot of the flaws in static and lab-based evaluations for ML in security. Especially, with how a pipeline-based approach was used to identify problems. However, while it did excel at problem identification I felt as though some of the proposed solutions are more high-level guidelines that concrete, implementable techniques. For example, the authors say new drift detection strategies can handle delayed labels but did not propose one themselves. But the creation of the checklist is very practical and could serve as a standard for a peer review, and potentially will elevate the quality and real-world application of future research in the field.
Dos & Don’ts of ML in Computer Security (Topic 2.2)
Resources Used
Arp et al., “Dos and Don’ts of Machine Learning in Computer Security,” USENIX Security ’22.
Topics
pitfalls sampling bias label noise data snooping spurious correlations hyperparameters baselinesmetrics base rate fallacy deployment threat model
Abstract
This paper analyzes 10 common issues in the design, implementation, and evaluation of learning-based security systems. The researches did a study of the 30 top-tier security papers published (last decade), and found that these issues happened within every paper and they had at least 3 of the issues. As a solution they suggest “dos and don’ts” and the use of an empirical impact analysis across four security topics to show how these issues lead to unrealistic performance and flawed conclusions.
Problem
- The application of ML in security is vulnerable to subtle issues that affect its performance and lead to over-optimistic results, and pose challenges in a real deployment.
- Sampling bias (P1) was found in 90% of papers, and data snooping (P3) in 73%.
- This widespread issue creates a “false sense of achievement” that slows the adoption of genuine research advances in both academia and industry.
Solution
The authors define the pitfalls as:
1. Data Collection: Sampling Bias, Label Inaccuracy.
2. System Design: Data Snooping, Spurious Correlations, Biased Parameter Selection.
3. Evaluation: Inappropriate Baselines, Inappropriate Performance Measures, Base Rate Fallacy.
4. Deployment: Lab-Only Evaluation, Inappropriate Threat Model.
- For each pitfall (“don’t”), the paper proposes (“dos”) to help researchers avoid them.
- The authors validate their claims with two main studies: a prevalence analysis on 30 papers to show how common the pitfalls are, and an impact analysis to empirically demonstrate how correcting these pitfalls changes results in four domains (mobile malware detection, vulnerability discovery, code attribution, and intrusion detection)
Results
- Mobile Malware Detection: Correcting for sampling bias (P1) and spurious correlations (P4) by training on data from a single source (Google Play) instead of mixed sources caused a classifier’s recall to drop by over 12%. The model had learned to distinguish app markets, not malware.
- Vulnerability Discovery: A deep learning model was found to be relying on spurious correlations (P4), such as specific buffer sizes unique to non-vulnerable code. A simple SVM baseline outperformed the complex neural network, focused on the need for appropriate baselines (P6).
- Network Intrusion Detection: A deep learning system was shown to be way to complex for its evaluation dataset (P9). A far simpler baseline (a boxplot on packet frequency) achieved superior performance because the synthetic attack data was overly conspicuous.
- Prevalence Study: Every one of the 30 analyzed papers from top security conferences was affected by at least 3 pitfalls
My Opinion
I found that the paper provided a critical look into the state of ML in security. The strength of the paper was having a multi-stage methodology and identifying the pitfalls in top research papers and showing their impacts. The impact analysis was very useful as it showed concrete examples of how the results could be misleading. Furthermore, the do’s and don’ts framework of the paper made it actionable for researchers and reviewers. Even though the paper acknowledges it’s difficult to entirely eliminate sampling bias raising the awareness was important to prevent future papers from going down the same road. I feel as though this paper should be required to be read for people doing further research in the field.
Connection Between 2.1 and 2.2
Both papers serve as important surveys on the application of ML in cyber and argue that a significant gap exists between academic research and real-world practice. They both identify a common set of problems which include flawed data handling (Ceschin et al. calls this “data leakage” and Arp et al.’s call this “temporal snooping”), biased datasets, and improper evaluation techniques.
Both papers approach it in a similar way:
- Ceschin et al. does a deep dive into the challenges of dynamic and evolving systems, along with using the data stream paradigm as a conceptual framework to address issues liek concept drift and delayed labs. Their focus is on the inherent nature of security data over time.
- Arp et al. focuses on a broad critique of research methodology. They focus on the ten pitfalls and back their claims by going through a bunch of top-tier security papers and doing an assessment on their practices.
To contrast them Ceschin et al. explain the technical reasons as to why a model might fail in a real environment (due to concept drift) while Arp et al. explains the methodology as to why a research paper might report misleading, optimistic results in the firist place (evaluation was done on a biased, static dataset). In combination they form a guide to achieving a methodological approach to producing ML security research that matters.