Andrew (YuXuan) Liu

I am a Tech Lead Manager at Covariant, leading the ML team for autonomous robots. While working at Covariant, I started and completed my PhD at UC Berkeley with Pieter Abbeel on Perception for Real-World Robotic Applications.

Previously, I did my Masters and Undergraduate studies at UC Berkeley, also with Pieter Abbeel. During my undergrad, I interned at Quora on language classification and Dropbox on iOS. Before Berkeley, I wrote the first iOS swipe keyboard and a dynamic theming platform for the jailbroken iOS platform.

 /  CV  /  Github  /  Linkedin  /  Twitter

profile photo
Self-Supervised Instance Segmentation by Grasping
YuXuan Liu, Xi Chen, Pieter Abbeel
International Conference on Intelligent Robots and Systems (IROS), 2023

We use before and after grasp images to augment instance segmentation supervision in a self-supervised manner, resulting in 10x data efficiency.

Distributional Instance Segmentation: Modeling Uncertainty and High Confidence Predictions with Latent-MaskRCNN
YuXuan Liu, Nikhil Mishra, Pieter Abbeel, Xi Chen
International Conference on Robotics and Automation (ICRA), 2023
patent / project page / dataset / arXiv / code

We augment instance segmentation models with a latent code in a VAE manner to model uncertainty and propose a confidence mask method that can significantly reduce robotic picking errors.

Autoregressive Uncertainty Modeling for 3D Bounding Box Prediction
YuXuan Liu, Nikhil Mishra, Maximilian Sieb, Yide Shentu, Pieter Abbeel, Xi Chen
European Conference on Computer Vision (ECCV), 2022
patent / project page / dataset / arXiv / code / blog

We augment instance segmentation models with a latent code in a VAE manner to model uncertainty and propose a confidence mask method that can significantly reduce robotic picking errors.

Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation
YuXuan Liu*, Abhishek Gupta*, Pieter Abbeel, Sergey Levine
International Conference on Robotics and Automation (ICRA), 2018
project page / arXiv / code

We propose an imitation learning method from video demonstractions by translating task representations between viewpoints.

Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
John D. Co-Reyes*, YuXuan Liu*, Abhishek Gupta*, Ben Eysenbach, Pieter Abbeel, Sergey Levine
International Conference on Machine Learning (ICML), 2018
project page / video / arXiv / code

We encode trajectories into a latent space which can be decoded for hierarchical planning and executed as a policy.

Meta-Reinforcement Learning of Structured Exploration Strategies
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, Sergey Levine
Neural Information Processing Systems (NeurIPS), 2018
arXiv / code

We meta-learn coherent exploration strategies and use this learned representation space for more meaningful exploration.

Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning
Abhishek Gupta*, Coline Devin*, YuXuan Liu, Pieter Abbeel, Sergey Levine
International Conference on Learning Representations (ICLR), 2017
video / arXiv

We learn an invariant representation space to transfer policies between different robots.

iOS Projects
Robo Call Screener - Phone AI

Call screening with live transcription for iOS.

Available on App Store, 2021

Download on the App Store

iSwipe Keyboard

First gesture-based keyboard for jailbroken iOS (pre-iOS 8.0) using a dynamic programming algorithm to match words with gesture angle differences.

code / released 2011


Powerful iOS theming platform with interactive widgets, custom script parser, and over 11 million downloads in Cydia.

code / released 2011

Technical Reports
Information Maximizing Preference Learning
IEOR290 Machine Learning for Operations with Humans, 2018

We model the reward as a Bayesian neural network and maximize information gain to efficiently learn from human preferences.

Joint Embedding Spaces for Language and Behavior
INFO259 Natural Language Processing, 2017

We learn an embeddings space where trajectories are points and language feedback are vectors that modulate between different behaviors in latent trajectory space.

Information Maximizing Exploration and Control
EE229A Information Theory, 2017

We maximize state entropy for exploration and mutual information with a latent variable for controllability.

Temporal Convolutional Policy Networks
YuXuan Liu, Tony Duan, Wesley Hsieh
CS294 Deep Neural Networks, 2016

We show that temporal convolutional policies can be effective for partially observed tasks including classical control and pong.

Hackathon Projects
HackRice 2015. 1st Place Winner. Best use of Microsoft Product
project page / video

We designed an audio transmission protocol to send messages, links, and pictures using the Fast-Fourier Transform.

TreeHacks 2015. Best Microsoft Hack. Best use of Vertical Response
project page / video

We use EEG states to encode brain activity and transmit messages using Huffman encoding.

Hack The North, 2014. Best Myo Hack
We built an iOS fitness tracker that collects real-time motion data and classifies activities using dynamic time warping.

HackTech, 2016. Best Synaptics Hack - Touch.
We learn handwriting styles with a neural network and built an authentication platform based on handwriting.

HackRice, 2016. 3rd Place Winner.
We automatically generate Instagram tags using image classification and real-time Instagram data.

Meme Generator
Cal Hacks 3.0, 2016. Best ML Hack.
We trained an image captioning model to generate meme captions conditioned on input images.

HackTX, 2015.
We applied speech recognition on lecture videos to match lectures with corresponding textbook material using tf-idf.

HackSC, 2014.
We implemented a pose classification algorithm with the Kinect for learning Yoga.

HackTech, 2017. Best Use of iRobot
We hacked a Roomba to follow you with real-time face detection.

SoHacks, 2014. Pebble Prize
We built a smartwatch app that alerts you when your phone is out of a certain range.



CS188 Introduction to Artificial Intelligence: Spring 2017
EE16A Design of Information Devices and Systems: Spring 2016, Fall 2016, Fall 2017
Outstanding Graduate Student Instructor Award: May 2017



Website template from Jon Barron.

*indicates equal contribution