Skip Navigation
Search

Anthony Kelly

Data Scientist
Northrop Grumman Mission Systems

Anthony (Tony) Kelly is a Data Scientist supporting the Northrop Grumman Corporation Mission Systems Sector Analytics and Insights organization, where he is responsible for providing advanced analytical support to various homerooms across the Northrop Grumman enterprise. He holds an Honors B.S. in Information Technology from La Salle University and is a forthcoming graduate of Johns Hopkins University, completing his M.S. degree in Computer Science with specializations in Artificial Intelligence and Software Engineering in August of 2020. Tony has worked in various roles at Northrop Grumman including as a data analyst, systems engineer, and cyber software analytics engineer. He has been responsible for leading executive-level data science projects and is the lead inventor on 7 NG IP Trade Secret technologies.

ABSTRACT

Determining Machine Learning Model Maturity with Probability Intervals

The evaluation and interpretability of machine learning models continues to be a field evolving with different techniques, visualizations, and approaches to understanding the mechanics of black box models. One of the primary challenges faced by industry applications of machine learning is the understanding of these black box models and when to know they are ready for production deployment. Gauging the maturity and strength of a machine learning model outside of prediction accuracy is often an overlooked, but critical task for the development, deployment, and integration of industry-best machine learning models. Optimization techniques like genetic algorithms, gridsearch, and randomsearch are common in attempting to optimize the hyperparameters of a model, but there exists no way to consistently test whether a model has reached peak performance, and when optimization can stop. Recent applied industry practice has indicated a novel way to determine candidacy for optimization. This technique applies the use of probabilistic confidence intervals. By determining the probabilities associated with the predictions of a machine learning model, there is an observable confidence and inherent risk, furthering an understanding of how “confident” a model is in its own predictions, thus encouraging (or discouraging) future optimization. This understanding of a model’s general confidence can be generalized to overall model maturity; once a model has reached a high confidence in all of its predictions, its performance can no longer be improved. An example of this using two popular gradient boosting methodologies will be discussed.