Understanding Black-box Predictions via Influence Functions - YouTube AboutPressCopyrightContact usCreatorsAdvertiseDevelopersTermsPrivacyPolicy & SafetyHow YouTube worksTest new features 2022. Linearization is one of our most important tools for understanding nonlinear systems. We have two ways of measuring influence: Our first option is to delete the instance from the training data, retrain the model on the reduced training dataset and observe the difference in the model parameters or predictions (either individually or over the complete dataset). LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Understanding black-box predictions via influence functions We look at three algorithmic features which have become staples of neural net training. Understanding Black-box Predictions via Influence Functions (2017) 1. Check if you have access through your login credentials or your institution to get full access on this article. Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Understanding Black-box Predictions via Influence Functions - ResearchGate Kelvin Wong, Siva Manivasagam, and Amanjit Singh Kainth. Implicit Regularization and Bayesian Inference [Slides]. GitHub - kohpangwei/influence-release Applications - Understanding model behavior Inuence functions reveal insights about how models rely on and extrapolate from the training data. We see how to approximate the second-order updates using conjugate gradient or Kronecker-factored approximations. PVANet: Lightweight Deep Neural Networks for Real-time Object Detection. Adaptive Gradient Methods, Normalization, and Weight Decay [Slides]. Using machine teaching to identify optimal training-set attacks on machine learners. In many cases, the distance between two neural nets can be more profitably defined in terms of the distance between the functions they represent, rather than the distance between weight vectors. We show that even on non-convex and non-differentiable models where the theory breaks down, approximations to influence functions can still provide valuable information. Understanding Blackbox Prediction via Influence Functions - SlideShare Which optimization techniques are useful at which batch sizes? With the rapid adoption of machine learning systems in sensitive applications, there is an increasing need to make black-box models explainable. PDF Understanding Black-box Predictions via Influence Functions - GitHub Pages The list One would have expected this success to require overcoming significant obstacles that had been theorized to exist. Understanding Black-box Predictions via Influence Functions On the importance of initialization and momentum in deep learning, A mathematical theory of semantic development in deep neural networks. In this paper, we use influence functions a classic technique from robust statistics to trace a . nimarb/pytorch_influence_functions - Github , . In order to have any hope of understanding the solutions it comes up with, we need to understand the problems. To scale up influence functions to modern machine learning This could be because we explicitly build optimization into the architecture, as in MAML or Deep Equilibrium Models. In, Metsis, V., Androutsopoulos, I., and Paliouras, G. Spam filtering with naive Bayes - which naive Bayes? influences. the algorithm will then calculate the influence functions for all images by Wojnowicz, M., Cruz, B., Zhao, X., Wallace, B., Wolff, M., Luan, J., and Crable, C. "Influence sketching": Finding influential samples in large-scale regressions. % Neural tangent kernel: Convergence and generalization in neural networks. In. On linear models and convolutional neural networks, we demonstrate that influence functions are useful for multiple purposes: understanding model behavior, debugging models, detecting dataset errors, and even creating visually-indistinguishable training-set attacks. In this paper, we use influence functions a classic technique from robust statistics to trace a models prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. In many cases, they have far more than enough parameters to memorize the data, so why do they generalize well? Interpreting black box predictions using Fisher kernels. calculated. While influence estimates align well with leave-one-out. Differentiable Games (Lecture by Guodong Zhang) [Slides]. Google Scholar Digital Library; Josua Krause, Adam Perer, and Kenney Ng. CSC2541 Winter 2021 - Department of Computer Science, University of Toronto Huang, L., Joseph, A. D., Nelson, B., Rubinstein, B. I., and Tygar, J. Adversarial machine learning. Thus, we can see that different models learn more from different images. Understanding Black-box Predictions via Influence Functions However, as stated when calculating the influence of that single image. All information about attending virtual lectures, tutorials, and office hours will be sent to enrolled students through Quercus. A sign-up sheet will be distributed via email. Please try again. Overview Neural nets have achieved amazing results over the past decade in domains as broad as vision, speech, language understanding, medicine, robotics, and game playing. In, Mei, S. and Zhu, X. which can of course be changed. Gradient-based hyperparameter optimization through reversible learning. Github >> In. With the rapid adoption of machine learning systems in sensitive applications, there is an increasing need to make black-box models explainable. Highly overparameterized models can behave very differently from more traditional underparameterized ones. In Proceedings of the international conference on machine learning (ICML). The canonical example in machine learning is hyperparameter optimization. prediction outcome of the processed test samples. We have 3 hours scheduled for lecture and/or tutorial. Are you sure you want to create this branch? Understanding Black-box Predictions via Influence Functions Background information ICML 2017 best paper Stanford Pang Wei Koh CourseraStanfordNIPS 2019influence function Percy Liang11Michael Jordan Abstract Understanding black-box predictions via influence functions. In, Cadamuro, G., Gilad-Bachrach, R., and Zhu, X. Debugging machine learning models. samples for each test data sample. Understanding Black-box Predictions via Influence Functions - SlideShare We have a reproducible, executable, and Dockerized version of these scripts on Codalab. The model was ResNet-110. Visualised, the output can look like this: The test image on the top left is test image for which the influences were Understanding black-box predictions via influence functions Computing methodologies Machine learning Recommendations On second-order group influence functions for black-box predictions With the rapid adoption of machine learning systems in sensitive applications, there is an increasing need to make black-box models explainable. Model selection in kernel based regression using the influence function. The most barebones way of getting the code to run is like this: Here, config contains default values for the influence function calculation Copyright 2023 ACM, Inc. Understanding black-box predictions via influence functions. Debruyne, M., Hubert, M., and Suykens, J. ; Liang, Percy. Biggio, B., Nelson, B., and Laskov, P. Support vector machines under adversarial label noise. Understanding Black-box Predictions via Influence Functions Chatterjee, S. and Hadi, A. S. Influential observations, high leverage points, and outliers in linear regression. Borys Bryndak, Sergio Casas, and Sean Segal. Students are encouraged to attend class each week. So far, we've assumed gradient descent optimization, but we can get faster convergence by considering more general dynamics, in particular momentum. most harmful. Understanding Black-box Predictions via Influence Functions by Pang Wei Koh and Percy Liang. The power of interpolation: Understanding the effectiveness of SGD in modern over-parameterized learning. Koh P, Liang P, 2017. and even creating visually-indistinguishable training-set attacks. We'll see how to efficiently compute with them using Jacobian-vector products. This code replicates the experiments from the following paper: Understanding Black-box Predictions via Influence Functions. Delta-STN: Efficient bilevel optimization of neural networks using structured response Jacobians. In. Natural gradient works efficiently in learning. You signed in with another tab or window. In this paper, we use influence functions --- a classic technique from robust statistics --- to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. values s_test and grad_z for each training image are computed on the fly Goodman, B. and Flaxman, S. European union regulations on algorithmic decision-making and a "right to explanation". Google Scholar (a) train loss, Hessian, train_loss + Hessian . More details can be found in the project handout. /Filter /FlateDecode Inception-V3 vs RBF SVM(use SmoothHinge) The inception networks(DNN) picked up on the distinctive characteristics of the fish. This alert has been successfully added and will be sent to: You will be notified whenever a record that you have chosen has been cited. Pang Wei Koh, Percy Liang; Proceedings of the 34th International Conference on Machine Learning, . Fortunately, influence functions give us an efficient approximation. This paper applies influence functions to ANNs taking advantage of the accessibility of their gradients. Understanding the Representation and Computation of Multilayer Perceptrons: A Case Study in Speech Recognition. Limitations of the empirical Fisher approximation for natural gradient descent. NIPS, p.1097-1105. Influence functions can of course also be used for data other than images, can take significant amounts of disk space (100s of GBs) but with a fast SSD S. McCandish, J. Kaplan, D. Amodei, and the OpenAI Dota Team. Assignments for the course include one problem set, a paper presentation, and a final project. This code replicates the experiments from the following paper: Pang Wei Koh and Percy Liang Understanding Black-box Predictions via Influence Functions International Conference on Machine Learning (ICML), 2017. Understanding Black-box Predictions via Influence Functions understanding model behavior, debugging models, detecting dataset errors, Proc 34th Int Conf on Machine Learning, p.1885-1894. Programming languages & software engineering, Programming languages and software engineering, Designing AI Systems with Steerable Long-Term Dynamics, Using platform models responsibly: Developer tools with human-AI partnership at the center, [ICSE'22] TOGA: A Neural Method for Test Oracle Generation, Characterizing and Predicting Engagement of Blind and Low-Vision People with an Audio-Based Navigation App [Pre-recorded CHI 2022 presentation], Provably correct, asymptotically efficient, higher-order reverse-mode automatic differentiation [video], Closing remarks: Empowering software developers and mathematicians with next-generation AI, Research talks: AI for software development, MDETR: Modulated Detection for End-to-End Multi-Modal Understanding, Introducing Retiarii: A deep learning exploratory-training framework on NNI, Platform for Situated Intelligence Workshop | Day 2. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. Theano D. Team. Understanding Black-box Predictions via Influence Functions International Conference on Machine Learning (ICML), 2017. SVM , . I am grateful to my supervisor Tasnim Azad Abir sir, for his . I. Sutskever, J. Martens, G. Dahl, and G. Hinton. Thus, you can easily find mislabeled images in your dataset, or ICML'17: Proceedings of the 34th International Conference on Machine Learning - Volume 70. We show that even on non-convex and non-differentiable models where the theory breaks down, approximations to influence functions can still provide valuable information. The dict structure looks similiar to this: Harmful is a list of numbers, which are the IDs of the training data samples Understanding Black-box Predictions via Influence Functions Unofficial implementation of the paper "Understanding Black-box Preditions via Influence Functions", which got ICML best paper award, in Chainer. This leads to an important optimization tool called the natural gradient. 10 0 obj In. While these topics had consumed much of the machine learning research community's attention when it came to simpler models, the attitude of the neural nets community was to train first and ask questions later. D. Maclaurin, D. Duvenaud, and R. P. Adams. /Length 5088 To run the tests, further requirements are: You can either install this package directly through pip: Calculating the influence of the individual samples of your training dataset How can we explain the predictions of a black-box model? 2018. For details and examples, look here. Understanding Black-box Predictions via Influence Functions In this lecture, we consider the behavior of neural nets in the infinite width limit. training time, and reduce memory requirements. Understanding Black-box Predictions via Influence Functions # do someting with influences/harmful/helpful. When testing for a single test image, you can then . S. Arora, S. Du, W. Hu, Z. Li, and R. Wang. on to the next image. International Conference on Machine Learning (ICML), 2017. ordered by harmfulness. The details of the assignment are here. Existing influence functions tackle this problem by using first-order approximations of the effect of removing a sample from the training set on model . Requirements Installation Usage Background and Documentation config Misc parameters The ACM Digital Library is published by the Association for Computing Machinery. In. approximations to influence functions can still provide valuable information. calculations even if we could reuse them for all subsequent s_test more recursions when approximating the influence. sample. Besides just getting your networks to train better, another important reason to study neural net training dynamics is that many of our modern architectures are themselves powerful enough to do optimization. We look at what additional failures can arise in the multi-agent setting, such as rotation dynamics, and ways to deal with them. Cook, R. D. Detection of influential observation in linear regression. can speed up the calculation significantly as no duplicate calculations take Apparently this worked. Influence functions are a classic technique from robust statistics to identify the training points most responsible for a given prediction. to trace a model's prediction through the learning algorithm and back to its training data, If Influence Functions are the Answer, Then What is the Question? In this paper, we use influence functions -- a classic technique from robust statistics -- to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. Validations 4. On the limited memory BFGS method for large scale optimization. Influence functions help you to debug the results of your deep learning model 10.5 Influential Instances | Interpretable Machine Learning - GitHub Pages Ribeiro, M. T., Singh, S., and Guestrin, C. "why should I trust you? multilayer perceptrons), you can use straight-up JAX so that you understand everything that's going on. calculations, which could potentially be 10s of thousands. vector to calculate the influence. We are given training points z 1;:::;z n, where z i= (x i;y i) 2 XY . We'll cover first-order Taylor approximations (gradients, directional derivatives) and second-order approximations (Hessian) for neural nets. Koh, Pang Wei. S. L. Smith, B. Dherin, D. Barrett, and S. De. The infinitesimal jackknife. We have a reproducible, executable, and Dockerized version of these scripts on Codalab. Influence functions efficiently estimate the effect of removing a single training data point on a model's learned parameters. Understanding black-box predictions via influence functions. This class is about developing the conceptual tools to understand what happens when a neural net trains. On the Accuracy of Influence Functions for Measuring - ResearchGate Understanding short-horizon bias in stochastic meta-optimization. Systems often become easier to analyze in the limit. Please download or close your previous search result export first before starting a new bulk export. Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W. A theory of learning from different domains. Are you sure you want to create this branch? A tag already exists with the provided branch name. numbers above the images show the actual influence value which was calculated. To scale up influence functions to modern machine learning settings, we develop a simple, efficient implementation that requires only oracle access to gradients and Hessian-vector products. 2017. For modern neural nets, the analysis is more often descriptive: taking the procedures practitioners are already using, and figuring out why they (seem to) work. For more details please see Li, J., Monroe, W., and Jurafsky, D. Understanding neural networks through representation erasure. This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. ? That can increase prediction accuracy, reduce On linear models and convolutional neural networks, we demonstrate that influence functions are useful for multiple purposes: understanding model behavior, debugging models, detecting dataset errors, and even creating visually-indistinguishable training-set attacks. A. We'll use the Hessian to diagnose slow convergence and interpret the dependence of a network's predictions on the training data. G. Zhang, S. Sun, D. Duvenaud, and R. Grosse. stream Pang Wei Koh - Google Scholar kept in RAM than calculating them on-the-fly. Then, it'll calculate all s_test values and save those to disk. The second mode is called calc_all_grad_then_test and Jaeckel, L. A. %PDF-1.5 Second-Order Group Influence Functions for Black-Box Predictions With the rapid adoption of machine learning systems in sensitive applications, there is an increasing need to make black-box models explainable. Thus, in the calc_img_wise mode, we throw away all grad_z Often we want to identify an influential group of training samples in a particular test prediction. We show that even on non-convex and non-differentiable models Understanding black-box predictions via influence functions. . 2016. https://dl.acm.org/doi/10.5555/3305381.3305576. The final report is due April 7. For these The degree of influence of a single training sample z on all model parameters is calculated as: Where is the weight of sample z relative to other training samples. In. A classic result tells us that the influence of upweighting z on the parameters ^ is given by. influence-instance. The algorithm moves then In this paper, we use influence functions a classic technique from robust statistics to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. Hopefully this understanding will let us improve the algorithms. prediction outcome of the processed test samples. No description, website, or topics provided. Noisy natural gradient as variational inference. We'll consider two models of stochastic optimization which make vastly different predictions about convergence behavior: the noisy quadratic model, and the interpolation regime. PDF Understanding Black-box Predictions via Influence Functions lehman2019inferringE. While this class draws upon ideas from optimization, it's not an optimization class. affecting everything else. In. Despite its simplicity, linear regression provides a surprising amount of insight into neural net training. To scale up influence functions to modern [] calculates the grad_z values for all images first and saves them to disk. Uses cases Roadmap 2 Reviving an "old technique" from Robust statistics: Influence function , . we develop a simple, efficient implementation that requires only oracle access to gradients The marking scheme is as follows: The problem set will give you a chance to practice the content of the first three lectures, and will be due on Feb 10. We'll mostly focus on minimax optimization, or zero-sum games. M. MacKay, P. Vicol, J. Lorraine, D. Duvenaud, and R. Grosse. Stochastic Optimization and Scaling [Slides]. Deep inside convolutional networks: Visualising image classification models and saliency maps. In this paper, we use influence functions a classic technique from robust statistics to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. In this paper, we use influence functions --- a classic technique from robust statistics --- Jianxin Ma, Peng Cui, Kun Kuang, Xin Wang, and Wenwu Zhu. Some JAX code examples for algorithms covered in this course will be available here. thereby identifying training points most responsible for a given prediction. An evaluation of the human-interpretability of explanation. This will naturally lead into next week's topic, which applies similar ideas to a different but related dynamical system. Understanding Black-box Predictions via Influence Functions Acknowledgements The authors of the conference paper 'Understanding Black-box Predictions via Influence Functions' Pang Wei Koh et al. It is individual work. , . Therefore, this course will finish with bilevel optimziation, drawing upon everything covered up to that point in the course. Biggio, B., Nelson, B., and Laskov, P. Poisoning attacks against support vector machines. How can we explain the predictions of a black-box model? On linear models and convolutional neural networks, we demonstrate that influence functions are useful for multiple purposes: understanding model behavior, debugging models, detecting dataset errors, and even creating visually-indistinguishable training-set attacks. This is a tentative schedule, which will likely change as the course goes on. Reference Understanding Black-box Predictions via Influence Functions logistic regression p (y|x)=\sigma (y \theta^Tx) \sigma . Loss , . the training dataset were the most helpful, whereas the Harmful images were the He, M. Narayanan, S. Gershman, B. Kim, and F. Doshi-Velez. Your search export query has expired. Appendix: Understanding Black-box Predictions via Inuence Functions Pang Wei Koh1Percy Liang1 Deriving the inuence functionIup,params For completeness, we provide a standard derivation of theinuence functionIup,params in the context of loss minimiza-tion (M-estimation). Understanding Black-box Predictions via Influence Functions Fast exact multiplication by the hessian. As a result, the practical success of neural nets has outpaced our ability to understand how they work. Google Scholar Krizhevsky A, Sutskever I, Hinton GE, 2012. Yuwen Xiong, Andrew Liao, and Jingkang Wang. A unified analysis of extra-gradient and optimistic gradient methods for saddle point problems: Proximal point approach. Abstract. Imagenet classification with deep convolutional neural networks. 7 1 . and Hessian-vector products. Understanding Black-box Predictions via Influence Functions --- Pang [ICML] Understanding Black-box Predictions via Influence Functions Online delivery. Pang Wei Koh and Percy Liang. Theano: A Python framework for fast computation of mathematical expressions. Depending what you're trying to do, you have several options: You are welcome to use whatever language and framework you like for the final project. How can we explain the predictions of a black-box model? Understanding Black-box Predictions via Influence Functions A. In contrast with TensorFlow and PyTorch, JAX has a clean NumPy-like interface which makes it easy to use things like directional derivatives, higher-order derivatives, and differentiating through an optimization procedure. The reference implementation can be found here: link. place. A. Mokhtari, A. Ozdaglar, and S. Pattathil. Understanding Black-box Predictions via Influence Functions A tag already exists with the provided branch name. x\Y#7r~_}2;4,>Fvv,ZduwYTUQP }#&uD,spdv9#?Kft&e&LS 5[^od7Z5qg(]}{__+3"Bej,wofUl)u*l$m}FX6S/7?wfYwoF4{Hmf83%TF#}{c}w( kMf*bLQ?C}?J2l1jy)>$"^4Rtg+$4Ld{}Q8k|iaL_@8v Tasha Nagamine, . A. S. Benjamin, D. Rolnick, and K. P. Kording. Lage, E. Chen, J. Frenay, B. and Verleysen, M. Classification in the presence of label noise: a survey. Deep learning via Hessian-free optimization. Understanding Black-box Predictions via Influence Functions Proceedings of the 34th International Conference on Machine Learning . We show that even on non-convex and non-differentiable models where the theory breaks down, approximations to influence functions can still provide valuable information. To scale up influence functions to modern machine learning settings, we develop a simple, efficient implementation that requires only oracle access to gradients and Hessian-vector products. Optimizing neural networks with Kronecker-factored approximate curvature. This isn't the sort of applied class that will give you a recipe for achieving state-of-the-art performance on ImageNet. as long as you have a supervised learning problem. In this paper, we use influence functions a classic technique from robust statistics to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction.
