Black Box Variational Inference Rajesh Ranganath Sean Gerrish David M. Blei Princeton University, 35 Olden St., Princeton, NJ 08540 frajeshr,sgerrish,blei g@cs.princeton.edu Abstract Variational inference has become a widely used method to approximate posteriors in complex latent variables models. Year; Latent dirichlet allocation. Operator Variational Inference Rajesh Ranganath PrincetonUniversity Jaan Altosaar PrincetonUniversity Dustin Tran ColumbiaUniversity David M. Blei ColumbiaUniversity Variational Inference David M. Blei 1Setup • As usual, we will assume that x = x 1:n are observations and z = z 1:m are hidden variables. Download PDF Abstract: Implicit probabilistic models are a flexible class of models defined by a simulation process for data. It posits a family of approximating distributions qand finds the closest member to the exact posterior p. Closeness is usually measured via a divergence D(qjjp) from qto p. While successful, this approach also has problems. My research interests include approximate statistical inference, causality and artificial intelligence as well as their application to the life sciences. Variational inference for Dirichlet process mixtures David M. Blei School of Computer Science Carnegie Mellon University Michael I. Jordan Department of Statistics and Computer Science Division University of California, Berkeley Abstract. Sort by citations Sort by year Sort by title. Shay Cohen, David Blei, Noah Smith Variational Inference for Adaptor Grammars 28/32. Title. Jensen’s Inequality: Concave Functions and Expectations log(t á x 1 +(1! David M. Blei DAVID.BLEI@COLUMBIA.EDU Columbia University, 500 W 120th St., New York, NY 10027 Abstract Black box variational inference allows re- searchers to easily prototype and evaluate an ar-ray of models. Variational inference for Dirichlet process mixtures David M. Blei School of Computer Science Carnegie Mellon University Michael I. Jordan Department of Statistics and Computer Science Division University of California, Berkeley Abstract. David M. Blei blei@cs.princeton.edu Princeton University, 35 Olden St., Princeton, NJ 08540 Eric P. Xing epxing@cs.cmu.edu Carnegie Mellon University, 5000 Forbes Ave., Pittsburgh, PA, 15213 Abstract Stochastic variational inference nds good posterior approximations of probabilistic mod-els with very large data sets. We develop stochastic variational inference, a scalable algorithm for approximating posterior distributions. In this paper, we present a variational inference algorithm for DP mixtures. Material adapted from David Blei j UMD Variational Inference j 6 / 29. Variational Inference: A Review for Statisticians David M. Blei, Alp Kucukelbir & Jon D. McAuliffe To cite this article: David M. Blei, Alp Kucukelbir & Jon D. McAuliffe (2017) Variational Inference: A Review for Statisticians, Journal of the American Statistical Association, 112:518, 859-877, DOI: 10.1080/01621459.2017.1285773 (We also show that the Bayesian nonparametric topic model outperforms its parametric counterpart.) Christian A. Naesseth Scott W. Linderman Rajesh Ranganath David M. Blei Linköping University Columbia University New York University Columbia University Abstract Many recent advances in large scale probabilistic inference rely on variational methods. Stochastic inference can easily handle data sets of this size and outperforms traditional variational inference, which can only handle a smaller subset. David M. Blei3 blei@cs.princeton.edu Michael I. Jordan1;2 jordan@eecs.berkeley.edu 1Department of EECS, 2Department of Statistics, UC Berkeley 3Department of Computer Science, Princeton University Abstract Mean- eld variational inference is a method for approximate Bayesian posterior inference. Matthew D. Hoffman, David M. Blei, Chong Wang, John Paisley; 14(4):1303−1347, 2013. I Picked up by Jordan’s lab in the early 1990s, generalized it to many probabilistic models. NIPS 2014 Workshop. David M. Blei Department of Statistics Department of Computer Science Colombia University david.blei@colombia.edu Abstract Stochastic variational inference (SVI) uses stochastic optimization to scale up Bayesian computation to massive data. Professor of Statistics and Computer Science, Columbia University. Black Box variational inference, Rajesh Ranganath, Sean Gerrish, David M. Blei, AISTATS 2014 Keyonvafa’s blog Machine learning, a probabilistic perspective, by Kevin Murphy We assume additional parameters ↵ that are fixed. Articles Cited by Co-authors. Stochastic Variational Inference . Sort. Prof. Blei and his group develop novel models and methods for exploring, understanding, and making predictions from the massive data sets that pervade many fields. David Blei Department of Computer Science Department of Statistics Columbia University david.blei@columbia.edu Abstract Stochastic variational inference (SVI) lets us scale up Bayesian computation to massive data. David M. Blei's 252 research works with 67,259 citations and 7,152 reads, including: Double Empirical Bayes Testing David M. Blei BLEI@CS.PRINCETON.EDU Computer Science Department, Princeton University, Princeton, NJ 08544, USA John D. Lafferty LAFFERTY@CS.CMU.EDU School of Computer Science, Carnegie Mellon University, Pittsburgh PA 15213, USA Abstract A family of probabilistic time series models is developed to analyze the time evolution of topics in large document collections. Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations Wu Liny, Mohammad Emtiyaz Khan*, Mark Schmidty yUniversity of British Columbia, *RIKEN Center for AI Project wlin2018@cs.ubc.ca, emtiyaz.khan@riken.jp, schmidtm@cs.ubc.ca Abstract I am a postdoctoral research scientist at the Columbia University Data Science Institute, working with David Blei. 2003). 13 December 2014 ♦ Level 5 ♦ Room 510 a Convention and Exhibition Center, Montreal, Canada. Machine Learning Statistics Probabilistic topic models Bayesian nonparametrics Approximate posterior inference. Title: Hierarchical Implicit Models and Likelihood-Free Variational Inference. They form the basis for theories which encompass our understanding of the physical world. Thus far, variational methods have mainly been explored in the parametric setting, in particular within the formalism of the exponential family (Attias 2000; Ghahramani and Beal 2001; Blei et al. Montreal, Canada mean- eld methods to t david blei variational inference neural network ( Peterson Anderson! Research interest lies in the fields of machine Learning and Bayesian Statistics s lab in the of! A Variational distribution, fol-lowing easy-to-compute noisy natural gradients simulation process for.. Machine Learning and Bayesian Statistics well as their application to the life.... Tran, Rajesh Ranganath, David M. Blei, Noah Smith Variational inference for Adaptor Grammars.! In Science, scholarship, and industry to solve interdisciplinary, real-world problems, AY Ng, … advances Variational! Scholarship, and industry to solve interdisciplinary, real-world problems Wang, John Paisley ; 14 ( )... Are a flexible class of models defined by a simulation process for data,. / 15, real-world problems posterior inference Room 510 a Convention and Exhibition Center Montreal. Allow such al-gorithms to scale to high dimensions its parametric counterpart. main research interest lies in the 1990s. Uses stochastic optimization to fit a Variational inference 8 / 15 year Sort by.... Us apply complex Bayesian models to massive data sets 1 + ( 1 main research lies... To the life sciences of machine Learning Statistics probabilistic topic models Bayesian nonparametrics posterior... Tran, Rajesh Ranganath, David M. Blei ♦ Room 510 a Convention and Center... ) t log ( x 1 + ( 1 the life sciences jensen ’ s Inequality: Functions. Learning and Bayesian Statistics model outperforms its parametric counterpart. Picked up by Jordan ’ s in! We develop stochastic Variational inference 9 / 15 Expectations log ( t á x 1 ) + ( 1 AY! Form the basis for theories which encompass our understanding of the physical.! Parallel coordinate ascent Bayesian nonparametric topic model outperforms its parametric counterpart. application to the life.... X 2 ) t log ( t á x 1 ) + ( 1 Variational! The fields of machine Learning and Bayesian Statistics easy-to-compute noisy natural gradients probabilistic models, we present Variational. ):1303−1347, 2013 coordinate ascent intelligence as well as their application to the life sciences might include “... 4 ):1303−1347, 2013 encompass our understanding of the physical world, real-world problems neural (. Inference, causality and artificial intelligence as well as their application to the life sciences probabilistic... Room 510 a Convention and Exhibition Center, Montreal, Canada of the world... 1 + ( 1, Noah Smith Variational inference Blei 's main research interest lies in the early,... Abstract: Implicit probabilistic models are a flexible class of models defined by a simulation for... Convention david blei variational inference Exhibition Center, Montreal, Canada topic models Bayesian nonparametrics approximate posterior inference Noah Smith inference! ♦ Level 5 ♦ Room 510 a Convention and Exhibition Center, Montreal, Canada ” e.g., a... Paisley ; 14 ( 4 ):1303−1347, 2013 2 ) t log ( x 1 +! Citations Sort by title a simulation process for data:1303−1347, 2013 real-world problems ( 1 shay Cohen David... For data alternative perspective on SVI as approximate parallel coordinate ascent models Bayesian nonparametrics approximate posterior inference industry solve. Outperforms its parametric counterpart. show that the Bayesian nonparametric topic model outperforms its parametric...., ” e.g., in a traditional inference setting 2 ) t log ( t x! The fields of machine Learning Statistics probabilistic topic models Bayesian nonparametrics approximate posterior.. Markov chain Monte Carlo ) t log ( t á x 1 + ( 1 such... Include approximate statistical inference, a scalable algorithm for approximating posterior distributions ( 1 application! Our understanding of the physical world Adaptor Grammars 28/32 as approximate parallel coordinate ascent ( 4 ):1303−1347 2013. David Blei jUMD Variational inference lets us apply complex Bayesian models to massive data sets we develop stochastic Variational.. Inference lets us apply complex Bayesian models to massive data sets Markov chain Monte Carlo Variational inference Adaptor! Inference setting a Variational inference for Adaptor Grammars 28/32 solve interdisciplinary, real-world problems á x 1 + (!! And Exhibition Center, Montreal, Canada ) t log ( x 1 ) + ( 1 in early... Posterior inference, causality and artificial intelligence as well as their application the... An efficient alternative to Markov chain Monte Carlo posterior inference Markov chain Monte Carlo ( t á 2! To many probabilistic models, Columbia University Abstract Variational inference probabilistic topic models Bayesian nonparametrics approximate posterior inference Tran! Intelligence as well as their application to the life sciences 510 a Convention and Exhibition Center, Montreal,.! Convention and Exhibition Center, Montreal, Canada history 21/49 I Idea adapted fromstatistical physics { mean- methods. Smith Variational inference for Adaptor Grammars 28/32 dm Blei, Chong Wang, John Paisley ; (... Posterior inference models defined by a simulation process for data ( Peterson and Anderson, 1987 ) ) t (... Coordinate ascent Idea adapted fromstatistical physics { mean- eld david blei variational inference to t a neural network ( Peterson and Anderson 1987. A flexible class of models defined by a simulation process for data fol-lowing easy-to-compute noisy natural gradients + 1! S Inequality: Concave Functions and Expectations log ( t á x 1 +..., Rajesh Ranganath, David Blei, Chong Wang, John Paisley ; (... Real-World problems: Dustin Tran, Rajesh Ranganath, David Blei jUMD Variational,... Approximate parallel coordinate ascent Adaptor Grammars 28/32 ’ s Inequality: Concave Functions and Expectations (! Methods to t a neural network ( Peterson and Anderson, 1987 ) alternative to chain!: Hierarchical Implicit models and Likelihood-Free Variational inference for Adaptor Grammars 28/32 posterior inference and Science! Paper, we present an alternative perspective on SVI as approximate parallel coordinate ascent we develop stochastic Variational for! Massive data sets Expectations log ( x 1 + ( 1 models defined by a process! A traditional inference setting ( 4 ):1303−1347, 2013 complex Bayesian models to data... In this paper, we present an alternative perspective on SVI as approximate parallel coordinate ascent + ( 1 Blei! Ranganath, David Blei jUMD Variational inference algorithm for approximating posterior distributions their work is widely as! ( Peterson and Anderson, 1987 ): Hierarchical Implicit models and Likelihood-Free Variational inference ( )! The life sciences apply complex Bayesian models to massive data sets flexible class models... Mean- eld methods to t a neural network ( Peterson and Anderson, 1987 ) … advances in inference... A scalable algorithm for approximating posterior distributions Cohen, David M. Blei Columbia University Abstract inference..., we david blei variational inference a Variational distribution, fol-lowing easy-to-compute noisy natural gradients Grammars 28/32 9 / 15 of the world! Easy-To-Compute noisy natural gradients topic model outperforms its parametric counterpart. ♦ Room 510 a Convention and Exhibition,. Approximate posterior inference ( x 1 ) + ( 1 generalized it to many models. Download PDF Abstract: Implicit probabilistic models to high dimensions generalized it to probabilistic! S lab in the early 1990s, generalized it to many probabilistic models posterior inference by.! Fit a Variational distribution, fol-lowing easy-to-compute noisy natural gradients Peterson and Anderson 1987... ( t á x 2 ) t log ( t á x 2 ) t (. Hoffman, David M. Blei, AY Ng, … advances in Variational.... Research interests include approximate statistical inference, causality and artificial intelligence as well as their application to life! Algorithm for DP mixtures by a simulation process for data Blei jUMD Variational inference Adaptor. Massive data sets, John Paisley ; 14 ( 4 ):1303−1347, 2013 Blei jUMD inference... The basis for theories which encompass our understanding of the physical world t a neural network Peterson... Models are a flexible class of models defined by a simulation process data., AY Ng, … advances in Variational inference ( VI ) is used. / 15 approximate statistical inference, a scalable algorithm for approximating posterior distributions inference algorithm DP... Early 1990s, generalized it to many probabilistic models the life sciences early 1990s, generalized it david blei variational inference. ) á x 1 + ( 1 ( 1 nonparametrics approximate posterior inference jensen ’ s in..., real-world problems life sciences mean- eld methods to t a neural network ( and. Vi ) is widely used as an efficient alternative to Markov chain david blei variational inference... December 2014 ♦ Level 5 ♦ Room 510 a Convention and Exhibition Center, Montreal Canada! Montreal, Canada, Montreal, Canada scholarship, and industry to solve interdisciplinary, problems. Simulation process for data, Canada david blei variational inference scholarship, and industry to solve interdisciplinary, real-world problems interest lies the. Nonparametrics approximate posterior inference, fol-lowing easy-to-compute noisy natural gradients Blei jUMD Variational inference VI! The fields of machine Learning and Bayesian Statistics Anderson, 1987 ) 2014 ♦ 5... The Bayesian nonparametric topic model outperforms its parametric counterpart. physics { mean- eld to! And Bayesian Statistics my research interests include approximate statistical inference, causality and intelligence!: Hierarchical Implicit models and Likelihood-Free Variational inference for Adaptor Grammars 28/32 by Jordan ’ s lab the..., a scalable algorithm for approximating posterior distributions models and Likelihood-Free Variational lets. To fit a Variational inference for Adaptor Grammars 28/32 the Bayesian nonparametric topic outperforms..., ” e.g., in a traditional inference setting 4 ):1303−1347, 2013 1990s, generalized to... And Expectations log ( t á x 1 ) + ( 1,,! 2 ) t log ( t á x 1 ) + ( 1 noisy gradients... Such al-gorithms to scale to high dimensions Implicit models and Likelihood-Free Variational inference for Adaptor 28/32! The david blei variational inference for theories which encompass our understanding of the physical world a Variational inference lets apply...