publications

For the up-to-date publication list, please see Google Scholar or Semantic Scholar pages.

For coverage of research in press, please see News

Papers (Journals & Conferences)

1. 

   ProgPrompt: Generating Situated Robot Task Plans using Large Language Models. Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, Animesh Garg. IEEE International Conference on Robotics and Automation (ICRA) 2023 We present a programmatic LLM prompt structure that enables plan generation functional across situated environments, robot capabilities, and tasks. Our key insight is to prompt the LLM with program-like specifications of the available actions and objects in an environment, as well as with example programs that can be executed. arXiv  project 
2. 

   MVTrans: Multi-View Perception of Transparent Objects. Yi Ru Wang, Yuchi Zhao, Haoping XU, Sagi Eppel, Alan Aspuru-Guzik, Florian Shkurti, Animesh Garg. IEEE International Conference on Robotics and Automation (ICRA) 2023 MVTrans is an end-to-end multi-view architecture with multiple perception capabilities, including depth estimation, segmentation, and pose estimation. We also present a procedural photo-realistic large-scale transparent object detection dataset, Syn-TODD. arXiv  project  video 
3. 

   DexGrasp-1M: Dexterous Multi-finger Grasp Generation Through Differentiable Simulation. Dylan Turpin, Tao Zhong, Shutong Zhang, Guanglei Zhu, Eric Heiden, Miles Macklin, Stavros Tsogkas, Sven Dickinson, Animesh Garg. IEEE International Conference on Robotics and Automation (ICRA) 2023 DexGrasp-1M: a large-scale dataset for multi-finger robotic grasping synthesized with FastGrasp’D, a novel diffferentiable grasping simulator. DexGrasp1M contains one million training examples for three (three, four and five-fingered) robotic hands, each with multimodal visual inputs (RGB+depth+segmentation, available in mono and stereo). project  video 
4. 

   Self-Supervised Learning of Action Affordances as Interaction Modes. Liquan Wang, Nikita Dvornik, Rafael Dubeau, Mayank Mittal, Animesh Garg. IEEE International Conference on Robotics and Automation (ICRA) 2023 We learn interaction modes as priors of useful interactions with articulated objects. In contrast to the prior art, we only use perception data from sensors in both training ad testing. project 
5. 

   nerf2nerf: Pairwise Registration of Neural Radiance Fields. Lily Goli, Daniel Rebain, Sara Sabour, Animesh Garg, Andrea Tagliasacchi. IEEE International Conference on Robotics and Automation (ICRA) 2023 We introduce a technique for pairwise registration of neural fields that extends classical optimization-based local registration (i.e. ICP) to operate on Neural Radiance Fields (NeRF) – neural 3D scene representations trained from collections of calibrated images. arXiv  project  code 
6. 

   SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models. Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, Animesh Garg. International Conference on Learning Representations (ICLR) 2023 A Transformer-based autoregressive model operating on learned object-centric representations. Evalaute on video prediction, Visual Question Answering (VQA), and goal-conditioned planning. arXiv  project 
7. 

   Learning Achievement Structure for Structured Exploration in Domains with Sparse Reward. Zihan Zhou, Animesh Garg. International Conference on Learning Representations (ICLR) 2023 Structured Exploration with Achievements (SEA), a multi-stage reinforcement learning algorithm that learns the environment structure with offline data and uses the learned structure to learn different skills and improve overall exploration with online environment interactions in a particular type of environment that has an internal achievement system. project 
8. 

   DiSECt: A Differentiable Simulator for Parameter Inference and Control in Robotic Cutting. Eric Heiden, Miles Macklin, Yashraj Narang, Dieter Fox, Animesh Garg, Fabio Ramos. Autonomous Robots (AURO) 2023 Differenctiable Cutting made easier. Now on real robots! arXiv  project  code  blog  video 
9. 

   ORBIT: A Unified Simulation Framework for Interactive Robot Learning Environments. Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Pooria Poorsarvi Tehrani, Ritvik Singh, Yunrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck Babich, Gavriel State, Marco Hutter, Animesh Garg. IEEE Robotics and Automation Letters (RA-L) and ICRA 2023 Orbit is a unified and modular framework for robotics and robot learning. Supports different types of physics, robot types and sensors. Supports differen Robot Learning workflows: RL, imitation, and motion planning. arXiv  project  code 
10. 

    RoboTube: Learning Household Manipulation from Human Videos with Simulated Twin Environments. Haoyu Xiong, Haoyuan Fu, Jieyi Zhang, Chen Bao, Qiang Zhang, Yongxi Huang, Wenqiang Xu, Animesh Garg, Cewu Lu. Conference on Robot Learning (CoRL) 2022 Reproducible and democratized benchmark for learning household robotic manipulation from 5000+ human videos. Also, a simulated twin environment RT-sim with matching assets for learning from these demonstrations. Oral Presentation pdf  project  video 
11. 

    Bayesian Object Models for Robotic Interaction with Differentiable Probabilistic Programming. Krishna Murthy Jatavallabhula, Miles Macklin, Dieter Fox, Animesh Garg, Fabio Ramos. Conference on Robot Learning (CoRL) 2022 We present a differentiable probabilistic program that helps robots build mental representations of complex everyday objects. pdf  project 
12. 

    SMPL: Simulated Industrial Manufacturing and Process Control Learning Environments. Mohan Zhang, Xiaozhou Wang, Benjamin Decardi-Nelson, Bo Song, An Zhang, Jinfeng Liu, Sile Tao, Jiayi Cheng, Xiaohong Liu, Dengdeng Yu, Matthew Poon, Animesh Garg. Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks 2022 An easy-to-use library that includes five high-fidelity simulation environments for manufacturing processes and an extensive Offline and online RL Benchmark arXiv  project  code 
13. 

    Breaking Bad: A Dataset for Geometric Fracture and Reassembly. Silvia Sellán, Yun-Chun Chen, Ziyi Wu, Animesh Garg, Alec Jacobson. Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks 2022 A large-scale dataset (1M+) of fractured objects to enable the study of fractured object reassembly and presents new challenges for geometric shape understanding. Featured Paper Presentation arXiv  pdf  project 
14. 

    MoCoDA: Model-based Counterfactual Data Augmentation. Silviu Pitis, Elliot Creager, Ajay Mandlekar, Animesh Garg. Advances in Neural Information Processing Systems (NeurIPS) 2022 MOCODA uses a locally factored model to create out-of-distribution data, and enable otherwise unsolvable tasks in offline RL. arXiv 
15. 

    Monotonic Quantile Network for Worst-Case Offline Reinforcement Learning. Chenjia Bai, Ting Xiao, Zhoufan Zhu, Lingxiao Wang, Fan Zhou, Animesh Garg, Bin He, Peng Liu, Zhaoran Wang. IEEE Transactions on Neural Networks and Learning Systems 2022 Learning a distributional value function in offline RL and optimizing a worst-case criterion of returns. We propose monotonic quantile network (MQN) with conservative quantile regression (CQR) for risk-averse policy learning. pdf 
16. 

    InfoCNF: An Efficient Conditional Continuous Normalizing Flow with Adaptive Solvers. Tan M. Nguyen, Animesh Garg, Richard G. Baraniuk, Anima Anandkumar. IEEE Asilomar Conference on Signals, Systems, and Computers 2022 InfoCNF is an efficient conditional CNF that partitions the latent space into a class-specific supervised code and an unsupervised code that shared among all classes for efficient use of labeled information arXiv 
17. 

    Grasp’D: Differentiable Contact-rich Grasp Synthesis for Multi-fingered Hands. Dylan Turpin, Liquan Wang, Eric Heiden, Yun-Chun Chen, Miles Macklin, Stavros Tsogkas, Sven Dickinson, Animesh Garg. European Conference on Computer Vision (ECCV) 2022 Differentiable Contact Simulation enables more realistic denser contact multifinger grasping than grasp computation with analytical metrics. Oral (Top 2.7%) arXiv  project  code  video 
18. 

    Continuous-Time Fitted Value Iteration for Robust Policies. Michael Lutter, Boris Belousov, Shie Mannor, Dieter Fox, Animesh Garg, Jan Peters. Transactions on Pattern Analysis and Machine Intelligence (T-PAMI) 2022 Solving HJB differential equation and its extension the Hamilton-Jacobi-Isaacs equation yields a robust optimal policy that achieves the maximum reward on a give task. We propose continuous and robust fitted value iteration that leverage the non-linear control-affine dynamics and separable state & action reward in continuous control to derive the optimal policy and optimal adversary in closed form. arXiv  project 
19. 

    Articulated Object Interaction in Unknown Scenes with Whole-Body Mobile Manipulation. Mayank Mittal, David Hoeller, Farbod Farshidian, Marco Hutter, Animesh Garg. IEEE International Conference on Intelligent Robots and Systems (IROS) 2022 Predict expected keyframes of operating an articulated object, then plan for close-loop dyanmically-feasible whole-body motion to match predicted object trajectory. arXiv  project 
20. 

    Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFinger. Arthur Allshire, Mayank Mittal, Varun Lodaya, Viktor Makoviychuk, Denys Makoviichuk, Felix Widmaier, Manuel Wüthrich, Stefan Bauer, Ankur Handa, Animesh Garg. IEEE International Conference on Intelligent Robots and Systems (IROS) 2022 A framework for learning a challenging dexterous manipulation task. The systems builds on IsaacGym for large scale simulation, keypoint based state representation and Cross-Atlantic remote sim2real to demonstrate the viability to scalability of robot learning. arXiv  project 
21. 

    Koopman Q-learning: Offline Reinforcement Learning via Symmetries of Dynamics. Matthias Weissenbacher, Samarth Sinha, Animesh Garg, Yoshinobu Kawahara. International Conference on Machine Learning (ICML) 2022 Koopman Forward (Conservative) Q-learning (KFC): a model-free RL algorithm which uses the symmetries in the dynamics of the environment to guide data augmentation in Offline RL. arXiv 
22. 

    Neural Shape Mating: Self-Supervised Object Assembly with Adversarial Shape Priors. Yun-Chun Chen, Haoda Li, Dylan Turpin, Alec Jacobson, Animesh Garg. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2022 Pairwise 3D geometric shape mating as general framework for part to part 3D pose matching for shape assembly. project 
23. 

    Modular Action Concept Grounding in Semantic Video Prediction. Wei Yu, Wenxin Chen, Songheng Yin, Steve Easterbrook, Animesh Garg. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2022 Object-Oriented semantic manipulation of scenes with unsupervised capsule networks which learn grounding of both objects and actions. arXiv  project 
24. 

    X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrieval. Satya Krishna Gorti, Noel Vouitsis, Junwei Ma, Keyvan Golestan, Maksims Volkovs, Animesh Garg, Guangwei Yu. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2022 Text to video retrieval with a scaled dot product attention for a text to attend to its most semantically similar frames. arXiv  project  code 
25. 

    GLiDE: Generalizable Quadrupedal Locomotion in Diverse Environments with a Centroidal Model. Zhaoming Xie, Xingye Da, Buck Babich, Animesh Garg, Michiel van de Panne. Workshop on Algorithmic Foundations of Robotics (WAFR) 2022 Model-Free RL in centroidal model for desired body accelerations with subsequent computation of ground reaction forces using a robot model. arXiv  project 
26. 

    Experience Replay with Likelihood-free Importance Weights. Samarth Sinha, Jiaming Song, Animesh Garg, Stefano Ermon. Learning for Dynamics and Control (L4DC) 2022 A likelihood-free density ratio estimator to reweight experiences based on their likelihood under the stationary distribution of the current policy. Best Paper Finalist arXiv 
27. 

    Value Gradient weighted Model-Based Reinforcement Learning. Claas A Voelcker, Victor Liao, Animesh Garg, Amir-massoud Farahmand. International Conference on Learning Representations (ICLR) 2022 Value aware model learning to fix Objective Mismatch in Model-based RL. The gradient of the empirical value function as a measure of the sensitivity of the RL algorithm to model errors arXiv  project 
28. 

    Accelerated Policy Learning with Parallel Differentiable Simulation. Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojciech Matusik, Animesh Garg, Miles Macklin. International Conference on Learning Representations (ICLR) 2022 A high-performance differentiable simulator and a new policy learning algorithm (SHAC) that can effectively leverage simulation gradients, even in the presence of non-smoothness pdf  project 
29. 

    PlaTe: Visually-Grounded Planning with Transformers in Procedural Tasks. Jiankai Sun, De-An Huang, Bo Lu, Yun-Hui Liu, Bolei Zhou, Animesh Garg. IEEE Robotics and Automation Letters (RA-L) and ICRA 2022 Planning Transformer to learn structured and plannable state and action spaces directly from unstructured videos. The model learns both action-conditional video prediction and goal conditioned planning. arXiv  project  code 
30. 

    Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning. Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, Zhaoran Wang. International Conference on Learning Representations (ICLR) 2022 Generalization beyond dataset in offline RL: Uncertainty quantification via the disagreement of bootstrapped Q-functions, and pessimistic updates by penalizing the value function based on the estimated uncertainty arXiv 
31. 

    Integration of Reinforcement Learning in a Virtual Robotic Surgical Simulation. Alexandra T. Bourdillon, Animesh Garg, Hanjay Wang, Y. Joseph Woo, Marco Pavone, Jack Boyd. Journal of Surgical Innovation 2022
32. 

    Centralized Model and Exploration Policy for Multi-Agent RL. Qizhen Zhang, Chris Lu, Animesh Garg, Jakob Foerster. International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS) 2022 Fully cooperative multi-agent settings (Dec-POMDPs) are fiendlishly hard. MARCO builds on the insight that using just a polynomial number of samples, it can learn a centralized model that generalizes across different policies. Oral Presentation arXiv 
33. Convergence and Optimality of Policy Gradient Methods in Weakly Smooth Settings. Matthew Shunshi Zhang, Murat Erdogdu, Animesh Garg. Confernce on Artificial Intelligence (AAAI) 2022 Convergence analysis in RL relies on non-intuitive, impractical and often opaque conditions such as strict smoothness and bounded function approximation. In this work, we establish explicit convergence rates of policy gradient methods without relying on these conditions, instead extending the convergence regime to weakly smooth policy classes with L2 integrable gradient. arXiv 
34. 

    Drop-DTW: Aligning Common Signal Between Sequences While Dropping Outliers. Nikita Dvornik, Isma Hadji, Konstantinos G. Derpanis, Animesh Garg, Allan D. Jepson. Advances in Neural Information Processing Systems (NeurIPS) 2021 Drop-DTW efficiently computes the optimal alignment between two variable-length sequences while automatically dropping the outlier elements from the matching. arXiv 
35. 

    Neural Hybrid Automata: Learning Dynamics with Multiple Modes and Stochastic Transitions. Michael Poli, Stefano Massaroli, Luca Scimeca, Seong Joon Oh, Sanghyuk Chun, Atsushi Yamashita, Hajime Asama, Jinkyoo Park, Animesh Garg. Advances in Neural Information Processing Systems (NeurIPS) 2021 A recipe for learning SHS dynamics without a priori knowledge on the number of modes and inter-modal transition dynamics. Method leverages Normalizing Flows and Stochastic ODEs. arXiv 
36. 

    Dynamic Bottleneck for Robust Self-Supervised Exploration. Chenjia Bai, Lingxiao Wang, Lei Han, Animesh Garg, Jianye Hao, Peng Liu, Zhaoran Wang. Advances in Neural Information Processing Systems (NeurIPS) 2021 Robust exploration via dynamic bottleneck-based representation and UCB-based bonus. arXiv 
37. 

    Real Robot Challenge: A Robotics Competition in the Cloud. Stefan Bauer, Manuel Wüthrich, Felix Widmaier, Annika Buchholz, Sebastian Stark, Anirudh Goyal, Thomas Steinbrenner, Joel Akpo, Shruti Joshi, Vincent Berenz, Vaibhav Agrawal, Niklas Funk, Julen Urain De Jesus, Jan Peters, Joe Watson, Claire Chen, Krishnan Srinivasan, Junwu Zhang, Jeffrey Zhang, Matthew Walter, Rishabh Madan, Takuma Yoneda, Denis Yarats, Arthur Allshire, Ethan Gordon, Tapomayukh Bhattacharjee, Siddhartha Srinivasa, Animesh Garg, Takahiro Maeda, Harshit Sikchi, Jilong Wang, Qingfeng Yao, Shuyu Yang, Robert McCarthy, Francisco Sanchez, Qiang Wang, David Bulens, Kevin McGuinness, Noel O’Connor, Redmond Stephen, Bernhard Schölkopf. Neural Information Processing Systems (NeurIPS) Competitions and Demonstrations Track 2021 A framework for democratizing multi-finger manipulation using a common hardware and software benchmark. arXiv  pdf 
38. 

    A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution. Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, Yoav Artzi. Conference on Robot Learning (CoRL) 2021 Data-augmentation with simple perturbations improve robustness, generalization, and OOD performance in Offline RL arXiv  project  code  poster 
39. 

    Seeing Glass: Joint Point-Cloud and Depth Completion for Transparent Objects. Haoping Xu, Yi Ru Wang, Sagi Eppel, Alan Aspuru-Guzik, Florian Shkurti, Animesh Garg. Conference on Robot Learning (CoRL) 2021 TraspareNet is a joint point cloud and depth completion method to recover learned depth of transparent objects in cluttered and complex scenes, even with partially filled fluid contents within the vessels Oral Presentation arXiv  project  code  talk 
40. 

    S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning. Samarth Sinha, Ajay Mandlekar, Animesh Garg. Conference on Robot Learning (CoRL) 2021 Data-augmentation with simple perturbations improve robustness, generalization, and OOD performance in Offline RL arXiv 
41. 

    Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos. Haoyu Xiong, Quanzhou Li, Yun-Chun Chen, Homanga Bharadhwaj, Samarth Sinha, Animesh Garg. IEEE International Conference on Intelligent Robots and Systems (IROS) 2021 Style transfer human videos to robot perspective, then sparse unsupervised keypoints for reward estimation, use RL for model-free task completion. arXiv  project  video 
42. 

    Learning Latent Actions to Control Assistive Robots. Dylan P. Losey, Hong Jun Jeon, Mengxi Li, Krishnan Srinivasan, Ajay Mandlekar, Animesh Garg, Jeannette Bohg, Dorsa Sadigh. Autonomous Robots (AURO) 2021 arXiv 
43. 

    Robust Value Iteration for Continuous Control Tasks. Michael Lutter, Shie Mannor, Jan Peters, Dieter Fox, Animesh Garg. Robotics: Systems and Science (RSS) 2021 Robustness to Sim2Real via Dynamic Programming based Value Iteration in Continuous time RL. arXiv  project 
44. 

    GIFT: Generalizable Interaction-aware Functional Tool Affordances without Labels. Dylan Turpin, Liquan Wang, Stavros Tsogkas, Sven Dickinson, Animesh Garg. Robotics: Systems and Science (RSS) 2021 Interaction-aware affordance mapping to unsupervised keypoints for tool-use in different scenarios. arXiv  blog  video 
45. 

    DiSeCT: A Differentiable Simulation Engine for Autonomous Robotic Cutting. Eric Heiden, Miles Macklin, Yashraj Narang, Dieter Fox, Animesh Garg, Fabio Ramos. Robotics: Systems and Science (RSS) 2021 Differenctiable Cutting made easy. Best Student Paper Award arXiv  project  code  blog  video 
46. 

    Value Iteration in Continuous Actions, States and Time. Michael Lutter, Shie Mannor, Jan Peters, Dieter Fox, Animesh Garg. International Conference on Machine Learning (ICML) 2021 RL in continuous states and actions can be solved with a closed-form extention of value iteration in cases of non-linear control-affine dynamics, resulting in a practical alternative to policy search. arXiv  project 
47. 

    Coach-Player Multi-agent Reinforcement Learning for Dynamic Team Composition. Bo Liu, Qiang Liu, Lei Han, Peter Stone, Animesh Garg, Yuke Zhu, Anima Anandkumar. International Conference on Machine Learning (ICML) 2021 Coordinating teams with time-varying composition and roles requires oversight from coach who can help with low-frequency updates to role assignments and team strategy. Long Talk arXiv 
48. 

    Principled Exploration via Optimistic Bootstrapping and Backward Induction. Chenjia Bai, Lingxiao Wang, Lei Han, Jianye Hao, Animesh Garg, Peng Liu, Zhaoran Wang. International Conference on Machine Learning (ICML) 2021 Improving exploration in RL through Optimistic Bootstrapping using UCB-bonus to capture epistemic uncertainty. Time-consistent uncertainty propagation through backward induction. arXiv  code 
49. 

    Tesseract: Tensorised Actors for Multi-Agent Reinforcement Learning. Anuj Mahajan, Mikayel Samvelyan, Lei Mao, Viktor Makoviychuk, Animesh Garg, Jean Kossaifi, Shimon Whiteson, Yuke Zhu, Anima Anandkumar. International Conference on Machine Learning (ICML) 2021 Tensorised formulation of the Bellman equation in Cooperative multi-agent RL is an effective solution to exponential blowup of the action space with the number of agents. arXiv 
50. 

    Dynamics Randomization Revisited:A Case Study for Quadrupedal Locomotion. Zhaoming Xie, Xingye Da, Michiel van de Panne, Buck Babich, Animesh Garg. IEEE International Conference on Robotics and Automation (ICRA) 2021 Dynamics randomization is neither necessary nor sufficient for sim-to-real transfer of learning robust locomotion policies. arXiv  project 
51. 

    LASER: Learning a Latent Action Space for Efficient Reinforcement Learning. Arthur Allshire, Roberto Martin-Martin, Charles Lin, Shawn Mendes, Silvio Savarese, Animesh Garg. IEEE International Conference on Robotics and Automation (ICRA) 2021 Learn a lower dimensional action-space that results in efficient exploration in similar tasks. arXiv  project  video 
52. 

    LEAF: Latent Exploration Along the Frontier. Homanga Bharadhwaj, Animesh Garg, Florian Shkurti. IEEE International Conference on Robotics and Automation (ICRA) 2021 Learn a dynamics aware manifold of reachable states, and then use this for guided exploration in hard continuous control tasks with RL. arXiv  project 
53. 

    Emergent Hand Morphology and Control from Optimizing Robust Grasps of Diverse Objects. Xinlei Pan, Animesh Garg, Animashree Anandkumar, Yuke Zhu. IEEE International Conference on Robotics and Automation (ICRA) 2021 A data-driven bayesian optimization approach to jointly optimize hand-design along with policy for grasping diverse objects in multiple modes. arXiv  project 
54. 

    C-Learning: Horizon-Aware Cumulative Accessibility Estimation. Panteha Naderian, Gabriel Loaiza-Ganem, Harry J. Braviner, Anthony L. Caterini, Jesse C. Cresswell, Tong Li, Animesh Garg. International Conference on Learning Representations (ICLR) 2021 Horizon-Aware policies trade off safety and performance while encoding multimodal solutions. Insight is to learn cumulative accessibility C(s,a,h) with time horizon h instead of the usual Q-function Q(s,a). arXiv  project  talk 
55. 

    Conservative Safety Critics for Exploration. Homanga Bharadhwaj, Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, Animesh Garg. International Conference on Learning Representations (ICLR) 2021 We need to guarantee safety during training in RL. Instead of unintuitive specification of state based safety, we can learn safety as a separate value function, and can jointly optimize for task performance with safety value as a constraint. arXiv  project  talk 
56. 

    Latent Skill Planning for Exploration and Transfer. Kevin Xie, Homanga Bharadhwaj, Danijar Hafner, Animesh Garg, Florian Shkurti. International Conference on Learning Representations (ICLR) 2021 Combine benefits of learned world-model with a set of modular skills for faster online test-time adaptation. Use learned skills during planning stage improves both speed and data efficiency arXiv  project  talk 
57. 

    DIBS: Diversity inducing Information Bottleneck in Model Ensembles. Samarth Sinha, Homanga Bharadhwaj, Anirudh Goyal, Hugo Larochelle, Animesh Garg, Florian Shkurti. Confernce on Artificial Intelligence (AAAI) 2021 Ensembles of deep nets to model uncertianty in modeling multi-modal data by encouraging diversity in prediction through adversarial loss for learning the stochastic latent variables arXiv 
58. 

    Unsupervised Disentanglement of Pose, Appearance and Background from Images and Videos. Aysegul Dundar, Kevin J. Shih, Animesh Garg, Robert Pottorf, Andrew Tao, Bryan Catanzaro. Transactions on Pattern Analysis and Machine Intelligence (T-PAMI) 2021 arXiv  code 
59. 

    Causal Discovery in Physical Systems from Videos. Yunzhu Li, Antonio Torralba, Animashree Anandkumar, Dieter Fox, Animesh Garg. Advances in Neural Information Processing Systems 33 (NeurIPS) 2020 Learn the underlying generative model as a causal graph with a few frames of observation. Generalize across variable latent dynamics (both graph connectivity and parameters). arXiv  project  talk 
60. 

    Counterfactual Data Augmentation using Locally Factored Dynamics. Silviu Pitis, Elliot Creager, Animesh Garg. Advances in Neural Information Processing Systems 33 (NeurIPS) 2020 Outstanding Paper Award at Object-Oriented Learning Workshop, ICML 2020 arXiv  code  talk 
61. 

    Curriculum By Smoothing. Samarth Sinha, Animesh Garg, Hugo Larochelle. Advances in Neural Information Processing Systems 33 (NeurIPS) 2020 Curriculum deisgn to improve representation learning in CNN by restricting access to high frequency information until later in the training Spotlight Talk arXiv  talk 
62. 

    Learning a Contact-Adaptive Controller for Robust, Efficient Legged Locomotion. Xingye Da, Zhaoming Xie, David Hoeller, Byron Boots, Animashree Anandkumar, Yuke Zhu, Buck Babich, Animesh Garg. Conference on Robot Learning (CoRL) 2020 arXiv  blog  video 
63. 

    Visuomotor Mechanical Search: Learning to Retrieve Target Objects in Clutter. Andrey Kurenkov, Joseph Taglic, Rohun Kulkarni, Marcus Dominguez-Kuhne, Animesh Garg, Roberto Martín-Martín, Silvio Saverese. IEEE International Conference on Intelligent Robots and Systems (IROS) 2020 arXiv  project  talk 
64. 

    A Programmable Approach to Neural Network Compression. Vinu Joseph, Ganesh Gopalakrishnan, Saurav Muralidharan, Michael Garland, Animesh Garg. IEEE Micro 2020 Demystifying network compression: user inputs pretrained model, compression scheme, objective and constraints. Condensa uses bayesian optimization to infer optimal sparsity ratio and corresponding compressed model. arXiv  pdf  project  code 
65. 

    Ocean: Online Task Inference for Compositional Tasks with Context Adaptation. Hongyu Ren, Yuke Zhu, Jure Leskovec, Anima Anandkumar, Animesh Garg. Conference on Uncertainty in Artificial Intelligence (UAI) 2020 A hierarchical latent variable prior that goes beyond vanilla gaussians to capture global and local context in sequential decision making for Meta-RL. arXiv  pdf  supp  code  talk 
66. 

    Semi-Supervised StyleGAN for Disentanglement Learning. Weili Nie, Tero Karras, Animesh Garg, Shoubhik Debhath, Anjul Patney, Ankit B. Patel, Anima Anandkumar. International Conference on Machine Learning (ICML) 2020 Dientanglement in GANs with 0.25-2.5% labelled dataset for high-resolution fine-grained control over image generation. arXiv  project  code  slides  talk 
67. 

    Angular Visual Hardness. Beidi Chen, Weiyang Liu, Animesh Garg, Zhiding Yu, Anshumali Shrivastava, Jan Kautz, Anima Anandkumar. International Conference on Machine Learning (ICML) 2020 Normalized angular distance between the sample feature embedding and the target classifier to measure sample hardness. arXiv  project  slides  talk 
68. 

    IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data. Ajay Mandlekar, Fabio Ramos, Byron Boots, Silvio Savarese, Li Fei-Fei, Animesh Garg, Dieter Fox. IEEE International Conference on Robotics and Automation (ICRA) 2020 Offline demonstrations are both suboptimal and multimodal. Use two-stage model-learning: a high leven generative model to fit multi-modal state density, and a low-level imitation model for near optimal control. arXiv  project  video  talk 
69. 

    Controlling Assistive Robots with Learned Latent Actions. Dylan P. Losey, Krishnan Srinivasan, Ajay Mandlekar, Animesh Garg, Dorsa Sadigh. IEEE International Conference on Robotics and Automation (ICRA) 2020 Learn a action space encoding from expert demonstrations, align the encoding with lower-dimension controller to enable efficient teleoperation. arXiv  blog  video  talk 
70. 

    Motion Reasoning for Goal-Based Imitation Learning. De-An Huang, Yu-Wei Chao, Chris Paxton, Xinke Deng, Li Fei-Fei, Juan Carlos Niebles, Animesh Garg, Dieter Fox. IEEE International Conference on Robotics and Automation (ICRA) 2020 Combine task & motion planning to disambiguate the true intention of the demonstrator from video where they performed multiple subtasks but only a subset was relevant to true objective, others were constraint satisfaction. arXiv  video 
71. 

    Guided Uncertainty-Aware Policy Optimization: Combining Learning and Model-Based Strategies for Sample-Efficient Policy Learning. Michelle A. Lee, Carlos Florensa, Jonathan Tremblay, Nathan Ratliff, Animesh Garg, Fabio Ramos, Dieter Fox. IEEE International Conference on Robotics and Automation (ICRA) 2020 Best Paper Award at 2019 Neurips Workshop on Robot Learning arXiv  video  talk 
72. 

    Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks. Michelle A. Lee, Yuke Zhu, Peter Zachares, Matthew Tan, Krishnan Srinivasan, Silvio Savarese, Li Fei-Fei, Animesh Garg, Jeannette Bohg. IEEE Transactions on Robotics (T-RO) 2020 arXiv  project 
73. Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks. Roberto Martı́n-Martı́n, Michelle A Lee, Rachel Gardner, Silvio Savarese, Jeannette Bohg, Animesh Garg. IEEE International Conference on Intelligent Robots and Systems (IROS) 2019 arXiv  project 
74. 

    Scaling Robot Supervision to Hundreds of Hours with RoboTurk: Robotic Manipulation Dataset through Human Reasoning and Dexterity. Ajay Mandlekar, Jonathan Booher, Max Spero, Albert Tung, Anchit Gupta, Yuke Zhu, Animesh Garg, Silvio Savarese, Li Fei-Fei. IEEE International Conference on Intelligent Robots and Systems (IROS) 2019 IROS Best Cognitive Robotics Paper Finalist arXiv  project  code  blog 
75. Continuous Relaxation of Symbolic Planner for One-Shot Imitation Learning. De-An Huang, Danfei Xu, Yuke Zhu, Animesh Garg, Silvio Savarese, Li Fei-Fei, Juan Carlos Carlos. IEEE International Conference on Intelligent Robots and Systems (IROS) 2019 arXiv  video 
76. 

    Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation. Kuan Fang, Yuke Zhu, Animesh Garg, Silvio Savarese, Li Fei-Fei. Conference on Robot Learning (CoRL) 2019 Oral Presentation arXiv  project 
77. 

    AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teachers. Andrey Kurenkov, Ajay Mandlekar, Roberto Martín-Martín, Silvio Savarese, Animesh Garg. Conference on Robot Learning (CoRL) 2019 arXiv  project  code  blog 
78. Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervision. Kuan Fang, Yuke Zhu, Animesh Garg, Andrey Kurenkov, Viraj Mehta, Li Fei-Fei, Silvio Savarese. International Journal of Robotics Research (IJRR) 2019 pdf  project  video 
79. 

    Neural Task Graphs: Generalizing to unseen tasks from a single video demonstration. De-An Huang, Suraj Nair, Danfei Xu, Yuke Zhu, Animesh Garg, Li Fei-Fei, Silvio Savarese, Juan Carlos Niebles. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2019 Oral Presentation arXiv  video 
80. 

    Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter. Mike Danielczuk, Andrey Kurenkov, Ashwin Balakrishna, Matthew Matl, Roberto Martín-Martín, Animesh Garg, Silvio Savarese, Ken Goldberg. IEEE International Conference on Robotics and Automation (ICRA) 2019 abstract  arXiv  project  video 

    When operating in unstructured environments such as warehouses, homes, and retail centers, robots are frequently required to interactively search for and retrieve specific objects from cluttered bins, shelves, or tables. Mechanical Search describes the class of tasks where the goal is to locate and extract a known target object. In this paper, we formalize Mechanical Search and study a version where distractor objects are heaped over the target object in a bin. The robot uses an RGBD perception system and control policies to iteratively select, parameterize, and perform one of 3 actions – push, suction, grasp – until the target object is extracted, or either a time limit is exceeded, or no high confidence push or grasp is available. We present a study of 5 algorithmic policies for mechanical search, with 15,000 simulated trials and 300 physical trials for heaps ranging from 10 to 20 objects. Results suggest that success can be achieved in this long-horizon task with algorithmic policies in over 95% of instances and that the number of actions required scales approximately linearly with the size of the heap.

81. 

    Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks. Michelle A Lee*, Yuke Zhu*, Krishnan Srinivasan, Parth Shah, Silvio Savarese, Li Fei-Fei, Animesh Garg, Jeannette Bohg (* equal contribution). IEEE International Conference on Robotics and Automation (ICRA) 2019 ICRA Best Paper Award and Finalist: Best Cognitive Robotics Paper arXiv  project  video 
82. ROBOTURK: A Crowdsourcing Platform for Robotic Skill Learning through Imitation. Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, Silvio Savarese, Li Fei-Fei. Conference on Robot Learning (CoRL) 2018 arXiv  pdf  project  code  blog  talk 
83. 

    Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervision. Kuan Fang, Yuke Zhu, Animesh Garg, Andrey Kurenkov, Viraj Mehta, Li Fei-Fei, Silvio Savarese. Robotics: Systems and Science (RSS) 2018 arXiv  project  video  talk 
84. SWIRL: A Sequential Windowed Inverse Reinforcement Learning Algorithm for Robot Tasks With Delayed Rewards. Sanjay Krishnan, Animesh Garg, Richard Liaw, Brijen Thananjeyan, Lauren Miller, Florian T Pokorny, Ken Goldberg. International Journal of Robotics Research (IJRR) 2018 pdf  project 
85. 

    Finding It: Weakly-Supervised Reference-Aware Visual Grounding in Instructional Video. De-An Huang, Shyamal Buch, Lucio Dery, Animesh Garg, Li Fei-Fei, Juan Carlos Niebles. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2018 Oral Presentation pdf  supp  project  poster  talk 
86. 

    Neural Task Programming: Learning to generalize across hierarchical tasks. Danfei Xu, Suraj Nair, Yuke Zhu, Julian Gao, Animesh Garg, Li Fei-Fei, Silvio Savarese. IEEE International Conference on Robotics and Automation (ICRA) 2018 arXiv  project  video  talk 
87. DeformNet: Free-Form Deformation Network for 3D Shape Reconstruction from a Single Image. Andrey Kurenkov*, Jingwei Ji*, Animesh Garg, Viraj Mehta, JunYoung Gwak, Christopher Choy, Silvio Savarese (* equal contribution). IEEE Winter Conference on Applications of Computer Vision (WACV) 2018 arXiv  project  talk 
88. Transition State Clustering: Unsupervised surgical trajectory segmentation for robot learning. Sanjay Krishnan*, Animesh Garg*, Sachin Patil, Colin Lea, Gregory Hager, Pieter Abbeel, Ken Goldberg (* equal contribution). International Journal of Robotics Research (IJRR) 2017 pdf  code 
89. AdaPT: Zero-Shot Adaptive Policy Transfer for Stochastic Dynamical Systems. James Harrison*, Animesh Garg*, Boris Ivanovic, Yuke Zhu, Silvio Savarese, Li Fei-Fei, Marco Pavone (* equal contribution). International Symposium on Robotics Research (ISRR) 2017 arXiv 
90. Weakly supervised 3D Reconstruction with Adversarial Constraint. JunYoung Gwak*, Christopher B Choy*, Animesh Garg, Manmohan Chandraker, Silvio Savarese (* equal contribution). IEEE Conference on 3D Vision (3DV) 2017 arXiv  code 
91. Adversarially Robust Policy Learning: Active Construction of Physically-Plausible Perturbations. Ajay Mandlekar*, Yuke Zhu*, Animesh Garg*, Li Fei-Fei, Silvio Savarese (* equal contribution). IEEE International Conference on Intelligent Robots and Systems (IROS) 2017 pdf  project  video 
92. Multilateral Surgical Pattern Cutting in 2D Orthotropic Gauze with Deep Reinforcement Learning Policies for Tensioning. Brijen Thananjeyan, Animesh Garg, Sanjay Krishnan, Carolyn Chen, Lauren Miller, Ken Goldberg. IEEE International Conference on Robotics and Automation (ICRA) 2017 pdf  video 
93. SWIRL: A Sequential Windowed Inverse Reinforcement Learning Algorithm for Robot Tasks With Delayed Rewards. Sanjay Krishnan, Animesh Garg, Richard Liaw, Brijen Thananjeyan, Lauren Miller, Florian T Pokorny, Ken Goldberg. Workshop on Algorithmic Foundations of Robotics (WAFR) 2016 pdf  talk 
94. 

    Interchangeable Surgical Instrument System with Application to Supervised Automation of Multilateral Tumor Resection. . Stephen McKinley, Animesh Garg, Siddarth Sen, David V. Gealy, Jonathan McKinley, Yiming Jen, Menglung Guo, Doug Boyd, Ken Goldberg. IEEE International Conference on Automation Science & Engg. (CASE) 2016 Best Video Award at 2015 Hamlyn Symposium project  video 
95. 

    Automating Multi-Throw Multilateral Surgical Suturing with a Mechanical Needle Guide and Sequential Convex Optimization. Siddarth Sen*, Animesh Garg*, David Gealy, Stephen McKinley, Yiming Jen, Ken Goldberg (* equal contribution). IEEE International Conference on Robotics & Automation (ICRA) 2016 project  video 
96. TSC-DL: Unsupervised Trajectory Segmentation of Multi-Modal Surgical Demonstrations with Deep Learning . Adithyavairavan Murali*, Animesh Garg*, Sanjay Krishnan*, Florian Pokorny, Pieter Abbeel, Trevor Darrell, Ken Goldberg (* equal contribution). IEEE International Conference on Robotics & Automation (ICRA) 2016 project  code  video 
97. 

    Tumor localization using automated palpation with Gaussian Process Adaptive Sampling. Animesh Garg, Siddarth Sen, Rishi Kapadia, Yiming Jen, Stephen McKinley, Lauren Miller, Ken Goldberg. IEEE International Conference on Automation Science & Engg. (CASE) 2016 pdf  project 
98. 

    A Single-Use Haptic Palpation Probe for Locating Subcutaneous Blood Vessels in Robot-Assisted Minimally Invasive Surgery. Stephen McKinley, Animesh Garg, Siddarth Sen, Rishi Kapadia, Adithyavairavan Murali, Kirk Nichols, Susan Lim, Sachin Patil, Pieter Abbeel, Allison M. Okamura, Ken Goldberg. IEEE International Conference on Automation Science & Engg. (CASE) 2015 Best Poster/Demo Award at ICRA 2015 Workshop on Shared Frameworks for Medical Robotics abstract  pdf  project 

    We present the design and evaluation of a novel low-cost palpation probe for Robot assisted Minimally Invasive Surgery (RMIS) for localizing subcutaneous blood vessels. It measures probe tip deflection using a Hall Effect sensor as the spherical tip is moved tangentially across a surface under automated control. The probe is intended to be single-use and disposable, built from 3D printed parts and commercially available electronics. The prototype has a cross-section of less than 15mm×10mm and fits on the end of an 8mm diameter needle driver in the Intuitive Surgical da Vinci ® Research Kit (dVRK). We report experiments for quasi-static sliding palpation with silicone based tissue phantoms with embedded cylinders as subcutaneous blood vessel phantoms. We analyzed signal-to-noise ratios with multiple diameters of silicone cylinders (1.58-4.75 mm) at varying subcutaneous depths (1-5 mm) with a range of indentation depths (0-8 mm) and sliding speeds (0.5-21 mm/s). Results suggest that the probe can detect subcutaneous structures in phantoms of diameter 2.25 mm at a depth of up to 5mm below the tissue surface.

99. 

    Learning by Observation for Surgical Subtasks: Multilateral Cutting of 3D Viscoelastic and 2D Orthotropic Tissue Phantoms. Adithyavairavan Murali, Siddarth Sen, Ben Kehoe, Animesh Garg, Seth McFarland, Sachin Patil, W Douglas Boyd, Susan Lim, Pieter Abbeel, Ken Goldberg. IEEE International Conference on Robotics & Automation (ICRA) 2015 Finalist: Best Paper, Student Paper, and Medical Robotics Paper Award abstract  project  video 

    Automating repetitive surgical subtasks such as suturing, cutting and debridement can reduce surgeon fatigue and procedure times and facilitate supervised tele-surgery. Programming is difficult because human tissue is deformable and highly specular. Using the da Vinci Research Kit (DVRK) robotic surgical assistant, we explore a “Learning By Observation” (LBO) approach where we identify, segment, and parameterize motion sequences and sensor conditions to build a finite state machine (FSM) for each subtask. The robot then executes the FSM repeatedly to tune parameters and if necessary update the FSM structure. We evaluate the approach on two surgical subtasks: debridement of 3D Viscoelastic Tissue Phantoms (3d-DVTP), in which small target fragments are removed from a 3D viscoelastic tissue phantom; and Pattern Cutting of 2D Orthotropic Tissue Phantoms (2d-PCOTP), a step in the standard Fundamentals of Laparoscopic Surgery training suite, in which a specified circular area must be cut from a sheet of orthotropic tissue phantom. We describe the approach and physical experiments with repeatability of 96% for 50 trials of the 3d-DVTP subtask and 70% for 20 trials of the 2d-PCOTP subtask.

100. Transition State Clustering: Unsupervised Surgical Trajectory Segmentation For Robot Learning. Sanjay Krishnan*, Animesh Garg*, Sachin Patil, Colin Lea, Gregory Hager, Pieter Abbeel, Ken Goldberg (* equal contribution). International Symposium on Robotics Research (ISRR) 2015 code 
101. Evaluation of PC‐ISO for customized, 3D printed, gynecologic 192Ir HDR brachytherapy applicators. Katherine Mellis, Timmy Siauw, Atchar Sudhyadhom, Rajni Sethi, I-Chow Hsu, Jean Pouliot, Animesh Garg, Ken Goldberg. Journal of Applied Clinical Medical Physics (JACMP) 2015 project 
102. Exact Reachability Analysis for Planning Skew-Line Needle Arrangements for Automated Brachytherapy. Animesh Garg, Timmy Siauw, Guang Yang, Sachin Patil, J Adam M Cunha, I-Chow Hsu, Jean Pouliot, Alper Atamtürk, Ken Goldberg. IEEE International Conference on Automation Science & Engg. (CASE) 2014 abstract  project 

     When planning skew-line needle arrangements for automated brachytherapy, one objective is to identify a set of candidate needles that enter from a specified entry region, avoid specified organs-at-risk and sufficiently cover the target (tumor) volume. Existing methods use uniform or random sampling to generate a set of candidate needles, which may not adequately cover the target volume. In this paper we present an exact reachability analysis that can be used to guide the selection of candidate needles and to identify which subset of the target volume may not be reachable. Assuming linear needles, convex polyhedral representations of entry zone, organs-at-risk and target volume, we give an exact polynomial time algorithm for checking existence and calculation of the non-reachable set in the target volume. We perform experiments using patient data from 18 brachytherapy cases and found that 11 cases had non-empty occluded volume inside the target ranging from 0.01% to 4.3% of target volume. We also report a sensitivity study showing the change in the occluded volume with dilation of the avoidance volume and entry zone.

103. Customized Needle Guides for Inserting Non-Parallel Needle Arrangements in Prostate HDR Brachytherapy: A Phantom Study. Timmy Siauw, J. Adam M. Cunha, Animesh Garg, Ken Goldberg, I Hsu, Jean Pouliot. Brachytherapy 2014 project 
104. Robot-Guided Open-Loop Insertion of Skew-Line Needle Arrangements for High Dose Rate Brachytherapy. Animesh Garg, Timmy Siauw, Dmitry Berenson, J Adam M Cunha, I-C Hsu, Jean Pouliot, Dan Stoianovici, Ken Goldberg. IEEE Transactions on Automation Science and Engineering (T-ASE) 2013 abstract  pdf 

     We present a study in human-centered automation that has potential to reduce patient side effects from high dose rate brachytherapy (HDR-BT). To efficiently deliver radiation to the prostate while minimizing trauma to sensitive structures such as the penile bulb, we modified the Acubot-RND 7-axis robot to guide insertion of diamond-tip needles into desired skew-line geometric arrangements. We extend and integrate two algorithms: Needle Planning with Integer Programming (NPIP) and Inverse Planning with Integer Programming (IPIP) to compute skew-line needle and dose plans. We performed three physical experiments with anatomically correct phantom models to study performance: two with the robot and one control experiment with an expert human physician (coauthor Hsu) without the robot. All were able to achieve needle arrangements that meet the RTOG-0321 clinical dose objectives with zero trauma to the penile bulb. We analyze systematic and random errors in needle placement; total RMS error for the robot system operating without feedback ranged from 2.6 to 4.3 mm, which is comparable to the RMS error of 2.7 mm obtained in an earlier study for PPI-BT treatment using a robot with 3D ultrasound feedback.

105. An Algorithm for Computing Customized 3D Printed Implants with Curvature Constrained Channels for Enhancing Intracavitary Brachytherapy Radiation Delivery. Animesh Garg, Sachin Patil, Timmy Siauw, J Adam M Cunha, I Hsu, Pieter Abbeel, Jean Pouliot, Ken Goldberg. IEEE International Conference on Automation Science & Engg. (CASE) 2013 abstract  pdf  project 

     Brachytherapy is a widely-used treatment modality for cancer in many sites in the body. In brachytherapy, small radioactive sources are positioned proximal to cancerous tumors. An ongoing challenge is to accurately place sources on a set of dwell positions to sufficiently irradiate the tumors while limiting radiation damage to healthy organs and tissues. In current practice, standardized applicators with internal channels are inserted into body cavities to guide the sources. These standardized implants are one-size-fits-all and are prone to shifting inside the body, resulting in suboptimal dosages. We propose a new approach that builds on recent results in 3D printing and steerable needle motion planning to create customized implants containing customized curvature-constrained internal channels that fit securely, minimize air gaps, and precisely guide radioactive sources through printed channels. When compared with standardized implants, customized implants also have the potential to provide better coverage: more potential source dwell positions proximal to tumors. We present an algorithm for computing curvature-constrained channels based on rapidly-expanding randomized trees (RRT). We consider a prototypical case of OB/GYN cervical and vaginal cancer with three treatment options: standardized ring implant (current practice), customized implant with linear channels, and customized implant with curved channels. Results with a two-parameter coverage metric suggest that customized implants with curved channels can offer significant improvement over current practice.

106. 

     Initial experiments toward automated robotic implantation of skew-line needle arrangements for HDR brachytherapy. A. Garg, T. Siauw, D. Berenson, A. Cunha, I-Chow Hsu, J. Pouliot, D. Stoianovici, K. Goldberg. IEEE International Conference on Automation Science & Engg. (CASE) 2012 IEEE CASE Best Application Paper Award abstract  project  video  talk 

     Automation seeks to improve the reliability and quality of processes. This study aims to automate high dose rate brachytherapy (HDR-BT), a radiation therapy that places radioactive sources at the site of the tumor using needles. Although HDR-BT has a high rate of clinical success in curing prostate cancer, it also has several side effects related to needle and dose trauma. A new planning algorithm from previous work optimizes needle arrangements using skew-lines (non-parallel, non-intersecting lines). This paper presents initial experiments towards an automated system for implanting skew-line needle arrangements computed from a planning system. We describe the interface, calibration and integration of the robotic hardware with the planning system, and present experiments using our robotic system to implant needles into anatomically-correct tissue phantoms. Results suggest that this system can achieve HDR-BT treatment objectives with reduced trauma to organs and low demands on operator skill, thus making the procedure more reliable and repeatable. In the future, we believe that robotic HDR-BT will improve overall treatment quality with reduced dependence on physician skill.

107. Robotic Brachytherapy Demonstration: Implant of HDR Brachytherapy Needle Configuration Computer-Optimized to Avoid Critical Structures Near the Bulb of the Penis. JA Cunha, T Siauw, A Garg, N Zhang, K Goldberg, D Stoianovici, M Roach III, I-C Hsu, J Pouliot. Medical Physics 2012 pdf 
108. Robot-guided Delivery of Brachytherapy Needles along Non-Parallel Paths to Avoid Penile Bulb Puncture. JAM Cunha, A Garg, T Siauw, N Zhang, Y Zuo, K Goldberg, D Stoianovici, M Roach III, J Pouliot. Radiotherapy and Oncology 2012 pdf 
109. Low-Cost Teleoperation of Remotely Located Actuators Based on Dual Tone Multi-Frequency Data Transfer. Sahil Thakkar, Animesh Garg, Adesh Midha, Prerna Gaur. MEMS, NANO and Smart Systems 2012 abstract  pdf 

     DTMF (Dual Tone Multi Frequency) is a system of signal tones used for telecommunications. DTMF uses two tones to represent each key on the touch pad. DTMF data transfer technique has advantages such as high reliability, constant speed, and high signal to noise ratio, low cost and optimal utilization of existing resources. These features make DTMF an attractive Data Transfer Technique. It finds application in home and car security systems, robot control, SMS and voice call controlled industrial applications. In this paper, we discuss the use of DTMF data transfer in a voice call to control a toy car. Cell phone 1(CP 1), which is at user end, makes a call to cell phone 2(CP 2) at the machine end and establishes a connection. A key is pressed at the user side. Two tones corresponding to one key are encoded and sent through the cell phone network. Both tones are tapped through the earphone jack of cell phone at machine end and are decoded. The output is fed to a Micro-controller. The Micro-controller is connected to the remote control unit of the toy car, which in turn controls the motion of the car. The car moves in various directions according to the key pressed user. The electronic circuit is divided into 2 parts. The transmitter side consisting of Cell phone 1 with an inbuilt encoder and the receiver side consisting of Cell Phone 2, 8870 DTMF decoder and Atmega 16 micro-controller. The programming has been carried out on AVR Bascom®.

110. The Autotrix: Design and Implementation of an Autonomous Urban Driving System. Animesh Garg, Anju Toor, Sahil Thakkar, Shiwangi Goel, Sachin Maheshwari, Satish Chand. MEMS, NANO and Smart Systems 2012 abstract  pdf 

     The Autotrix is an interactive, intelligent, Autonomous Guided Vehicle (AGV) designed to serve in urban environments. Autonomous ground vehicle navigation requires the integration of many technologies such as path planning, odometry, control, obstacle avoidance and situational awareness. The objective of this project is for this prototype to navigate autonomously in an urban environment and reach its destination while detecting and avoiding obstacles on the path .This will be achieved by extracting information from multiple sources of real-time data including digital camera, GPS &ultra sonic sensors, collecting data from this extracted information, processing this data and send controlling instructions to our platform (Autotrix). The significance of this work is in presenting the methods needed for real time navigation; GPS based continuous mapping and obstacle avoidance for intelligent autonomous driving systems.

111. Object Identification and Mapping using Monocular Vision in an Autonomous Urban Driving System. Animesh Garg, Anju Toor, Sahil Thakkar, Shiwangi Goel, Sachin Maheshwari, Satish Chand. International Conference of Machine Vision 2010

Preprints

1. ∆-Networks for Efficient Model Patching. Chaitanya Devaguptapu, Samarth Sinha, K J Joseph, Vineeth N Balasubramanian, Animesh Garg. preprint 2023 arXiv 
2. 

   Errors are Useful Prompts: Instruction Guided Task Programming with Verifier-Assisted Iterative Prompting. Marta Skreta, Naruki Yoshikawa, Sebastian Arellano-Rubach, Zhi Ji, Lasse Bjørn Kristensen, Kourosh Darvish, Alán Aspuru-Guzik, Florian Shkurti, Animesh Garg. preprint 2023 CLAIRify is a novel approach that combines automatic iterative prompting in LLMs with program verification to ensure programs written in domain-specific languages are syntactically valid and incorporate environment constraints. arXiv  project  video 
3. 

   An Adaptive Robotics Framework for Chemistry Lab Automation. Naruki Yoshikawa, Andrew Zou Li, Kourosh Darvish, Yuchi Zhao, Haoping Xu, Alan Aspuru-Guzik, Animesh Garg, Florian Shkurti. preprint 2022 We propose a framework for robots to assist chemists by performing lab experiments autonomously. The solution allows a general-purpose robot to perform diverse chemistry experiments and efficiently make use of available lab tools. arXiv  project  video 

Peer-Reviewed Workshops

1. Uniform Priors for Data-Efficient Transfer. Samarth Sinha, Karsten Roth, Anirudh Goyal, Marzyeh Ghassemi, Hugo Larochelle, Animesh Garg. Workshop on Learning with Limited Labelled Data for Image and Video Understanding at CVPR 2022 Features that are most transferable have high uniformity in the embedding space and propose a uniformity regularization scheme that encourages better transfer and feature reuse. arXiv  talk 
2. 

   Neural Motion Fields: Encoding Grasp Trajectories as Implicit Value Functions. Yun-Chun Chen, Adithyavairavan Murali, Balakumar Sundaralingam, Wei Yang, Animesh Garg, Dieter Fox. Workshop on Implicit Representations for Robotic Manipulation at RSS 2022 Neural Motion Fields learns a value function that can be queried to generate reactive grasping. arXiv 
3. 

   D2RL: Deep Dense Architectures in Reinforcement Learning. Samarth Sinha, Homanga Bharadhwaj, Aravind Srinivas, Animesh Garg. Workshop on Deep Reinforcement Learning (DRL) at Neurips 2020 Architectures in RL do not need to be simple MLPs! Dense connections in both actor and critic improve representation learning and hence performance. arXiv  project  code  talk 
4. Counterfactual Data Augmentation using Locally Factored Dynamics. Silviu Pitis, Elliot Creager, Animesh Garg. Workshop on Object-Oriented Learning (OOL) at ICML 2020 Outstanding Paper Award at Object-Oriented Learning Workshop, ICML 2020 code  poster  talk 
5. Offline Policy Optimization in RL with Variance Regularizaton. Riashat Islam, Samarth Sinha, Homanga Bharadhwaj, Samin Yeasar Arnob, Zhuoran Yang, Animesh Garg, Zhaoran Wang, Lihong Li, Doina Precup. Workshop on Offline Reinforcement Learning at NeurIPS 2020 arXiv 
6. Combining Model-Free and Model-Based Strategies for Sample-Efficient Reinforcement Learning. Michelle A. Lee, Carlos Florensa, Jonathan Tremblay, Nathan Ratliff, Animesh Garg, Fabio Ramos, Dieter Fox. Workshop on Robot Learning at NeurIPS 2019 Best paper award at the workshop pdf 
7. Turbulence forecasting via Neural ODE. Gavin Portwood, Peetak Mitra, Mateus Dias Ribeiro, Tan Minh Nguyen, Balasubramanya Nadiga, Juan Saenz, Micheal Chertkov, Animesh Garg, Anima Anandkumar, Andreas Dengel, Richard Baraniuk, David Schmidt. Workshop on Machine Learning and the Physical Sciences at NeurIPS 2019 pdf 
8. Towards Grasp Transfer using Shape Deformation.. Andrey Kuyenkov*, Viraj Mehta*, Jingwei Ji, Animesh Garg, Silvio Savarese (* equal contribution). Conference on Robot Learning (CoRL), workshop track 2017 pdf  poster 
9. Hierarchical Task Generalization with Neural Programs. Danfei Xu, Yuke Zhu, Julian Gao, Animesh Garg, Li Fei-Fei, Silvio Savarese. Conference on Robot Learning (CoRL), workshop track 2017 talk 
10. Hierarchical Task Generalization with Neural Programs. Danfei Xu, Yuke Zhu, Julian Gao, Animesh Garg, Li Fei-Fei, Silvio Savarese. R:SS Workshop on New Frontiers for Deep Learning in Robotics 2017 poster 
11. Adversarially Robust Policy Learning through Active Construction of Physically-Plausible Perturbations. Ajay Mandlekar*, Yuke Zhu*, Animesh Garg*, Li Fei-Fei, Silvio Savarese (* equal contribution). Multidisciplinary Conference on Reinforcement Learning and Decision Making (RLDM) 2017 pdf 
12. HIRL: Hierarchical inverse reinforcement learning for long-horizon tasks with delayed rewards. Sanjay Krishnan, Animesh Garg, Richard Liaw, Lauren Miller, Florian T Pokorny, Ken Goldberg. 2016 arXiv 
13. On Visual Feature Representations for Transition State Learning in Robotic Task Demonstrations. Animesh Garg*, Sanjay Krishnan*, Adithyavairavan Murali, Trevor Darrell, Pieter Abbeel, Ken Goldberg (* equal contribution). NIPS Workshop on Feature Extraction 2015 pdf  project 
14. Autonomous Tumor Localization and Extraction: Palpation, Incision, Debridement and Adhesive Closure with the da Vinci Research Kit. Stephen McKinley, Siddarth Sen, Animesh Garg, Yiming Jen, David Gealy, Pieter Abbeel, Ken Goldberg. Hamlyn Surgical Robotics Conference, London 2015 Best Video Award video 
15. Automated Delivery Instrument for Stem Cell Treatment Using the daVinci Robotic Surgical System. Stephen McKinley, Animesh Garg, Susan Lim, Sachin Patil, Ken Goldberg. Annual Meeting of the International Society for Stem Cell Research. Stockholm, Sweden. 2015 poster 

Theses

1. Autonomous Palpation for Tumor Localization: Design of a Palpation Probe and Gaussian Process Adaptive Sampling. MS Thesis, EECS Department, University of California, Berkeley, 2016 project 
2. Optimization and Design for Automation of Brachytherapy Delivery and Learning Robot-Assisted Surgical Sub-Tasks. PhD Thesis, University of California, Berkeley, 2016 project 

Patents

1. Precision injector/extractor for robot-assisted minimally-invasive surgery. Stephen Mckinley, Animesh Garg, Sachin Patil, Susan ML Lim, Ken Goldberg. U.S. Provisional. PCT International Application No.: PCT/US2016/039,026, June, 2016 project 
2. Patient-Specific Temporary Implants For Accurately Guiding Local Means of Tumor Control Along Patient-Specific Internal Channels to Treat Cancers. Jean Pouliot, I-Chow Ken Goldberg, J. Adam Cunha, Animesh Garg, Sachin Patil, Pieter Abbeel, Timmy Siauw. US Patent 10,286,197, May, 2019 project 

Technical Reports

1. 

   See, Plan, Predict: Language-guided Cognitive Planning with Video Prediction. Maria Attarian, Advaya Gupta, Ziyi Zhou, Wei Yu, Igor Gilitschenski, Animesh Garg. 2022 A new video prediction architecture using pre-trained transformers for language conditioned goal prediction and planning. arXiv  project 
2. 

   Auditing AI models for Verified Deployment under Semantic Specifications. Homanga Bharadhwaj, De-An Huang, Chaowei Xiao, Animashree Anandkumar, Animesh Garg. 2021 How can we design a similarly motivated auditing scheme for deep learning models? We propose a sequence of semantically aligned unit tests each to verify a predefined specification. arXiv  project  blog 
3. Making Sense of the Robotized Pandemic Response: A Comparison of Global and Canadian Robot Deployments and Success Factors. Tim Barfoot, Jessica Burgner-Kahrs, Eric Diller, Animesh Garg, Andrew Goldenberg, Jonathan Kelly, Xinyu Liu, Hani E. Naguib, Goldie Nejat, Angela P. Schoellig, Florian Shkurti, Hallie Siegel, Yu Sun, Steven Waslander. 2020 arXiv  blog 
4. De-anonymization of authors through arXiv submissions during double-blind review. Homanga Bharadhwaj, Dylan Turpin, Animesh Garg, Ashton Anderson. 2020 Doubt-blind reviewing process may tilt the scales in favor of eminent authors who de-anonymize through concurrent arxiv release. arXiv  blog 
5. Video Interpolation and Prediction with Unsupervised Landmarks. Kevin J. Shih, Aysegul Dundar, Animesh Garg, Robert Pottorf, Andrew Tao, Bryan Catanzaro. 2019 arXiv 
6. Composing Meta-Policies for Autonomous Driving Using Hierarchical Deep Reinforcement Learning. Richard Liaw, Sanjay Krishnan, Animesh Garg, Daniel Crankshaw, Joseph E Gonzalez, Ken Goldberg. 2017 arXiv 
7. Autonomous localization and navigation using 2D laser scanners. Manohar Paluri, Animesh Garg, Henrik Christensen. 2010