Zhiwei Deng

Zhiwei Deng is a Research Scientist at Google Research. He was a Postdoc at the Computer Science Department of Princeton University. He earned his PhD at Simon Fraser University in Computer Science.

He is generally interested in building intelligent learning agents with System 1&2. Recently, he is drawn by reparameterizing knowledge in neural networks through Input-Output memories, and building perceptual world model using self-supervised representation learning.


profile photo
Perceptual Group Tokenizer: Building Perception with Iterative Grouping
Zhiwei Deng, Ting Chen, Yang Li
ICLR 2024

Vision-Language Dataset Distillation
Xindi Wu, Byron Zhang, Zhiwei Deng, Olga Russakovsky
Arxiv 2023
code  /  project

Unseen Image Synthesis with Diffusion Models
Ye Zhu, Yu Wu, Zhiwei Deng, Olga Russakovsky, Yan Yan
Arxiv 2023

Boundary Guided Learning-Free Semantic Control with Diffusion Models
Ye Zhu, Yu Wu, Zhiwei Deng, Olga Russakovsky, Yan Yan
Neurips 2023
code  /  project

Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks
Zhiwei Deng, Olga Russakovsky
Neurips 2022

Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigation
Zhiwei Deng, Karthik Narasimhan, Olga Russakovsky
Neurips 2020
code  /  poster  /  Princeton Robotics Seminar

BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Steps
Wang (Bill) Zhu*, Hexiang Hu*, Jiacheng Chen, Zhiwei Deng, Vihan Jain, Eugene Ie, Fei Sha
ACL 2020   (Oral)

Take the Scenic Route: Improving Generalization in Vision-and-Language Navigation
Felix Yu, Zhiwei Deng, Karthik Narasimhan, Olga Russakovsky
Visual Learning with Limited Labels Workshop, CVPR 2020   (Oral)
arxiv / code

Continuous Graph Flow
Zhiwei Deng*, Megha Nawhal*, Lili Meng, Greg Mori
ICML 2020 - Workshop on Graph Representation Learning and Beyond 2020  
code  /  webpage

Policy Message Passing: Modeling Trajectories for Probabilistic Graph Inference
Zhiwei Deng, Xingguo Li, Greg Mori
Arxiv 2019  

Talking With Hands 16.2M: A Large-Scale Dataset of Synchronized Body-Finger Motion and Audio for Conversational Motion Analysis and Synthesis.
Gilwoo Lee, Zhiwei Deng, Shugao Ma, Takaaki Shiratori, Siddhartha S. Srinivasa, Yaser Sheikh.
ICCV 2019  

Learning Structured Inference Neural Networks with Label Relations.
Nelson Nauata, Hexiang Hu*, Guang-tong Zhou, Zhiwei Deng, Zicheng Liao, Greg Mori
PAMI 2019

Probabilistic Neural Programmed Networks for Scene Generation.
Zhiwei Deng, Jiacheng Chen, Yifang Fu, Greg Mori
Neural Information Processing Systems(NeurIPS) 2018 ((Spotlight), 3.5% acceptance rate)
code  /  poster

Sparsely Aggregated Convolutional Networks.
Ligeng Zhu, Ruizhi Deng, Michael Maire, Zhiwei Deng, Greg Mori
ECCV 2018

Factorized Variational Autoencoders for Modeling Audience Reactions to Movies.
Zhiwei Deng, Rajitha Navarathna, Peter Carr, Stephan Mandt, Yisong Yue, Iain Matthews, Greg Mori CVPR 2017
(Press Coverage CBC News | PHYS.ORG | Caltech News)

Structure Inference Machines: Recurrent Neural Networks for Analyzing Relations in Group Activity Recognition.
Zhiwei Deng, Arash Vahdat, Hexiang Hu, Greg Mori CVPR 2016
arxiv  /  Webpage  /  Extended Collective Activity Dataset: train-test split

A Hierarchical Deep Temporal Model for Group Activity Recognition.
Moustafa Ibrahim, Srikanth Muralidharan, Zhiwei Deng, Arash Vahdat, Greg Mori
CVPR 2016

Deep Structured Models For Group Activity Recognition.
Zhiwei Deng, Mengyao Zhai, Lei Chen, Yuhao Liu, Srikanth Muralidharan, Mehrsan Roshtkhari, Greg Mori
BMVC 2015 (oral, acceptance rate=7%)

Invited talks

  • Princeton University, PIXL talk, 2019.
  • University of California, Berkeley, EECS, 2019.
  • Vector Institute's Endless Summer School, 2019.


  • Senior Program Committee for: IJCAI 2021
  • Reviewers for: NIPS 2017-2020, CVPR 2017-2020, ICML 2018-2020, ECCV 2018-2020, NIPS 2018-2020, ACL 2020, EMNLP 2020, ICLR 2020, PAMI

Thanks to Jon Barron for this nice website template.