mike.exe

4. Report 4

9 min read

~ By: Michael Ades ~ Oct 8, 2025

Fast & Furious: On the modelling of malware detection as an evolving data stream (Topic 4.1)

Resources Used Ceschin et al., “Fast & Furious: On the modelling of malware detection as an evolving data stream,” Expert Systems With Applications 212 (2023) 118590.

Topics Malware Detection Concept Drift Data Streams Feature Extractor Update Android.

Abstract 

This paper evaluates the impact of concept drift, the constant evolution of malware samples that changes the data distribution, on machine learning classifiers. The authors analyzed two Android datasets (DREBIN and a subset of AndroZoo). They propose a data stream pipeline, Fast & Furious (F&F), that updates not only the classifier but also the feature extractor when concept drift is detected. The goal is to determine the best approach for long-term detection in real environments

Problem

  • Malware is a major threat causing significant challenges to cybersecurity, causing millions of dollars in losses.
  • Malware constantly changes features to bypass detection, leading to concept drift.
  • This evolution causes AntiViruses (AVs) to quickly become outdated mechanisms with decreased detection rates over time.
  • Most prior work focuses on achieving high accuracy rates for a temporally localized dataset, and overlooks long-term detection challenges imposed by malware evolution.
  • Predicting future threats based only on past data is the worst-case scenario for AV companies, resulting in a significant drop in F1-score and recall.

Solution 

The solution is a data stream learning pipeline that incorporates the feature extractor into the adaptation loop. This pipeline is triggered by the detection of concept drift and uses classifiers suitable for evolving data streams, such as the Adaptive Random Forest (ARF) and Stochastic Gradient Descent (SGD) classifier.

Five steps when a new sample is obtained from the raw data stream:

  1. Obtain .
  2. Extract features using a feature extractor (), trained with previous data.
  3. Predict the class of using the classifier ().
  4. Update the drift detector () to check the drift level (using detectors like DDM, EDDM, ADWIN, or KSWIN).
  5. Based on the drift level, follow one of three paths:
    • Normal or Warning: Keep updating incrementally with (Warning level also adds to a buffer).
    • Drift: Retrain both and  using only the data collected during the warning level (from the buffer).

We need to update the feature extractor because new features emerge over time (new API calls, permissions, URLs)

Results

  • The data stream pipeline that updates both and significantly outperformed traditional data stream solutions.
  • The pipeline achieved an improvement of 22.05% of F1Score in the DREBIN dataset and 8.77% in the AndroZoo dataset.
  • The Incremental Windowed Classifier (IWC), showed unstable and declining recall rates, especially on the AndroZoo dataset, which means it loses the ability to detect new malware over time.
  • Concept drift is a generalized phenomenon in Android malware, not specific to a particular dataset.
  • ADWIN with retraining generally performed best across initial comparisons, while KSWIN with retraining was recommended based on multiple evals.
  • Analysis showed malware evolution is driven by Android platform changes (changes to permissions or deprecation of APIs like SMS sending features), forcing malware authors to adapt, which results in feature drift and concept drift.

My Opinion 

This paper shows that malware detection is a data stream problem challenged by concept drift. Malware detection is a feature drift problem as well as a concept drift problem. The proposal to retrain the feature extractor alongside the classifier is a necessary step for longterm detection accuracy.

DroidEvolver: Self-Evolving Android Malware Detection System (Topic 4.2)

Resources Used Xu et al., “DroidEvolver: Self-Evolving Android Malware Detection System,” 2019 IEEE European Symposium on Security and Privacy (EuroS&P).

Topics Self-Evolving System Online Learning Pseudo Labels Model Pool Feature Set Evolvement.

Abstract 

DroidEvolver is a malware detection system designed to automatically and continually update itself to handle the constant evolution of Android malware and the changing framework. DroidEvolver does not need continous retraining with true labels. DroidEvolver makes necessary and lightweight updates using online learning techniques, and uses an evolving feature set and pseudo labels. It maintains a model pool initialized with online learning algorithms and uses a Juvenilization Indicator (JI) to identify aging models and drifting applications.

Problem

  • Existing Android detection systems rapidly age due to continuous evolution of both the Android framework and malicious applications.
  • Traditional retraining solutions have issues with determining the optimal frequency for retraining (too frequent → wastes resources; too infrequent → leads to vulns).
  • Retraining requires manual labeling of new applications, which is resource intensive and constrains update frequency.
  • Retraining with cumulative datasets is expensive and unscalable as the volume of new applications grows too fast.

Solution 

DroidEvolver operates in two phases:

  • Initialization: The system is trained on a set of known applications with true labels. It extracts Android API calls as features and constructs an initial Model Pool using five different linear online learning algorithms (Passive Aggressive, Online Gradient Descent, AROW, RDA, Ada-FOBOS).
  • **Detection: ** For an incoming unknown application (A), DroidEvolver performs three steps:
  1. Finds drift: Computes a Juvenilization Indicator (JI) per model M. If a model’s JI is outside the range <τ0,τ1>, that model is marked aging and A​ is a drifting sample.
  2. Labels it: Classifies A​ by weighted voting among the non aging models. The vote becomes A’s pseudo label.
  3. Updates itself: Expands features with any new API calls from A, then updates the aging models using A​ and its pseudo label.

Approach avoids periodic batch retraining with cumulative datasets.

Results

  • DroidEvolver achieved a high initial detection F-measure (95.27%).
  • Its F-measure went down by 1.06% on average per year over the next five years, which is slow aging.
  • It outperformed the model MAMADROID, achieving an overall F-measure 2.11x over time.
  • It takes 1.37s on average to process an unknown application in the detection phase, making it 28.58 times faster than MAMADROID.
  • The system correctly classifies 98.22% of identified drifting applications, mitigating the risk of incorrect pseudo labels misleading aging models.
  • Evolving feature set grows continuously, and adds new features resulting from application evolvement and Android framework changes.

My Opinion 

DroidEvolver has a good auto and scalable malware detector by eliminating the need for timely ground-truth labels. The use of a model pool and the Juvenilization Indicator provides a effective mechanism for automatic self-correction. Its fast which makes it useable for detection environments with lots of samples. They also have a sanitization module to prevent poisoning attacks on the model.

Transcending TRANSCEND: Revisiting Malware Classification in the Presence of Concept Drift (Topic 4.3)

Resources Used Barbero et al., “Transcending TRANSCEND: Revisiting Malware Classification in the Presence of Concept Drift,”. Jordaney et al., “Transcend: Detecting Concept Drift in Malware Classification Models,” USENIX Security Symposium 2017.

Topics Classification with Rejection Concept Drift Detection Conformal Prediction p-values Computational Efficiency.

Abstract 

This paper addresses the performance degradation of malware classifiers due to concept drift by employing Classification with Rejection. It builds on Transcend, which uses prediction measures to detect aging models during deployment. Their framework TRANSCENDENT, effectiveness stems from detecting violations of the i.i.d. assumption (drift) via p-values. TRANSCENDENT introduces evaluators (ICE and CCE) to overcome the overhead of the original Transductive Conformal Evaluator (TCE).

Problem

  • Malware classification assumes the training and test data are drawn independently and identically distributed (i.i.d.), which is violated in adversarial security settings by concept drift.
  • When drift occurs, the classifier’s performance degrades, but relying on traditional metrics (like precision or recall) only provides a backdated view of poor performance.
  • Using probabilities for rejection (fixed thresholds) is unreliable because probabilities are constrained to sum to 1.0, leading to artificially skewed confidence even for previously unseen objects.
  • The original Transcend (TCE) method was computationally (impossible) for large datasets, requiring retraining the underlying classifier for nearly every training point (1.9 CPU years for one experiment).

Solution

Conformal Evaluator (CE):

  • Measure oddness: Compute an algorithm-specific nonconformity score (NCM) showing how unlike the training data the new sample is.
  • Convert to a p-value: Turn that score into a credibility p-value. low p means the sample likely doesn’t fit the training distribution (possible drift).
  • Set a threshold: Choose a cutoff τ during calibration to separate typically correct (high p) from incorrect (low p) cases. TRANSCENDENT picks τ efficiently via random search.
  • Reject when unsafe: At test time, if (p < τ), reject or quarantine the prediction flagging drift without needing labels.

To solve the computational feasibility issue of TCE:

  • Inductive Conformal Evaluator (ICE): Splits the training set once into a proper training set and a calibration set (fastest method).
  • Cross-Conformal Evaluator (CCE): Partitions the training set into folds, using each fold each time for calibration.

Results

  • TRANSCENDENT significantly improved the performance of the degraded baseline classifier. For kept elements, it was able to maintain high performance (F1 > 0.8) for up to two years, which doubled the lifespan of the model
  • Approx-TCE and ICE achieved F1, Precision, and Recall of 0 for rejected elements, meaning all rejected predictions would have been wrong if accepted.
  • Statistical p-values are superior to probabilities: using p-value thresholds made reliable predictions that were around 24% more correct than those derived from probability thresholds.
  • The proposed random search threshold optimization found similarly effective thresholds as a full grid search but required two orders of magnitude fewer trials.
  • The framework generalizes well across different domains (Android, Windows PE, PDF malware) and classifiers (linear SVM, GBDT, Random Forest).

My Opinion 

This work provides a missing component from the original Transcend paper and improves its practical applicability through the ICE and CCE. P-values derived from conformal evaluation provide a much better, algorithm-agnostic mechanism to quantify prediction quality and flag drif which solves the problem caused by delayed labels. Also rising rejection rates will act as a signal to trigger retraining or other remediation strategies is also good guidance.

Comparisons

4.1 Fast & Furious and 4.2 DroidEvolver

  • F&F uses explicit detectors. DroidEvolver infers drift with a Juvenilization Indicator (model aging).
  • F&F does full retraining of classifier and feature extractor. DroidEvolver does incremental updates to aging models and grows the feature set.
  • F&F needs true labels for buffered data. DroidEvolver uses pseudo labels after initialization.
  • F&F targets max accuracy with clean labels. DroidEvolver prioritizes scalability and speed when labels lag.

4.2 DroidEvolver and 4.3 TRANSCENDENT

  • DroidEvolver adapts to drift (learns from it). TRANSCENDENT rejects low-credibility predictions to avoid errors.
  • DroidEvolver uses JI and ensemble scores. TRANSCENDENT uses conformal p-values with statistical guarantees.
  • DroidEvolver updates models using pseudo labels. TRANSCENDENT quarantines the sample and defers handling.
  • TRANSCENDENT’s p-value–based rejection is more solid than probability or score evals, especially when labels are unavailable.

My takeway is to use TRANSCENDENT to detect and gate drifted inputs, then choose F&F (when I can obtain labels and want peak accuracy) or DroidEvolver (when labels are delayed and I need fast adaptation that is continous).

Graph View

Backlinks