Task-driven Perceptual Representations: Sensing, Planning and Control under Resource Constraints

ICRA 2016 workshop

Organizers: Andrea Censi (MIT), Stefano Soatto (UCLA), Panagiotis Tsiotras (GATech)

Technical Committees endorsements:

This workshop has taken place on Monday, May 16, in Room 27.

Please use this link to download all the presentations as they are uploaded.


Textbook robotics relies on the duality of “inference” vs “control”, or “perception” vs “planning”. These are usually considered distinct problems that can be tackled separately, using the “belief” of the agent as the interface between the two. However, the two become entangled again when computational resources are constrained; this happens either in the regime where the available on-board computational resources are limited (small UAVs, robotic insects), as well as in the regime where environment and sensor data are complex.

When computation, memory, or sensing bandwidth are constrained or are associated to a cost, many classical notions must be revised. The most efficient implementation of a behaviorally “optimal” agent does not estimate a belief over the state, but rather it estimates the “minimal representation”, which is the smallest statistic of the observations that is sufficient to perform the task, and is typically much smaller than the full belief. The best sensor is not the one that provides the most bits about the environment, but rather the one whose bits are most “informative” for the task at hand given available resources. If computation has a cost, the best agent aims to achieve “bounded rationality” or “rational inattention”.

The goal of this workshop is to bring together the researchers in robotics who have been working from many complementary angles on the general issue of designing optimal agents under resources constraints. We would like to understand together which of the competing formalizations are expressive enough to model the resource constraints of a realistic robotic system; which lead to tractable design problems; and whether there is an intersection between the two sets. We also would like to attract researchers in the neighboring fields of computer vision/machine learning and control/identification theory who work on largely equivalent problems.


9:00am-9:15amOpening Remarks
9:15am-10:00amByron Boots, GaTech - Predictive State Inference Machines
10:00am-10:45amDieter Fox, UW - Model-based and Learning-based Approaches to Perception and Control
10:45am-11:30amGeorge Konidaris, Duke University - Robots, Skills, and Symbols
11:30m-12:15amTakashi Tanaka, KTH - LQG Control with Minimal Information: A Semidefinite Programming Approach
1:30pm-2:15pmDanica Kragic - Task-driven Representations for Grasping and Object Manipulation
2:15pm-3:00pmLaurent Itti, USC - Attention Strategies for Robotics
3:00pm-3:45pmTim Genewein, Max Planck institute for Intelligent Systems, Tübingen - Information-theoretic bounded rationality in perception-actionsystems
3:45pm-4:30pmEvangelos TheodorouGrady Williams, GaTech - Fast Decision Making Using Parallel Stochastic Optimal Control and Inference
4:30pm-too tiredClosing Remarks and Open Q&A Forum


Danica Kragic (KTH)

Task-driven representations for grasping and object manipulation

Grasping and manipulation of objects in regular, unstructured environments is an important ability of a service robot. The robot needs to reason about objects, scenes and task requirements and also ground these in sensorimotor information. We present our work on probabilistic models for scene/object representation and task based reasoning. One of the approaches consists of Gaussian mixture models for generic data discretisation and Bayesian networks for encoding task-relevant variables. including object and action features as well as constraints. Another approach uses Predictive State Representation (PSR) that allow for modelling of dynamical systems directly in observables. We overview how PSRs can be extended using prior information to learn representations which are suitable for planning and task interpretation.

Byron Boots (GaTech)

Predictive State Inference Machines

Latent state space models are a fundamental and widely used tool for modeling dynamical systems. However, they are difficult to learn from data and learned models often lack performance guarantees on tasks such as filtering and prediction. In this talk, I will introduce Predictive State Inference Machines (PSIM), a framework that considers the inference procedure on the dynamical system as a composition of predictors. The key idea behind PSIM is that rather than focusing on learning a latent state space model, which could then be used for inference, one should develop a learning algorithm that considers the inference task directly. PSIM accomplishes this by learning predictors for inference in predictive state space, where sufficient features of observations can be used for supervision. We provide theoretical guarantees for our approach, in both realizable and agnostic settings, and showcase practical performance on a variety of simulated and real world robotics benchmarks.

Dieter Fox (UW)

Model-based and learning-based approaches to perception and control

Over the last years, deep learning techniques trained on large sets of labeled data have resulted in significant improvements on various perception tasks such as visual recognition, detection, and tracking of objects, and control tasks such as playing Go, Nintendo games, or pushing an object with a robot manipulator. However, successes in robot control have mostly been in relatively narrow settings. In this talk, I will present some thoughts on the potential benefits of learning-based approaches to robot perception and control, and how they relate to more established, model-based techniques.

George Konidaris (Duke)

Robots, Skills, and Symbols

Robots are increasingly becoming a part of our daily lives, from the automated vacuum cleaners in our homes to the rovers exploring Mars. However, while recent years have seen dramatic progress in the development of affordable, general-purpose robot hardware, the capabilities of that hardware far exceed our ability to write software to adequately control. The key challenge here is one of abstraction. Generally capable behavior requires high-level reasoning and planning, but perception and actuation must ultimately be performed using noisy, high-bandwidth, low-level sensors and effectors. I will describe recent research that uses hierarchical reinforcement learning as a basis for constructing robot control hierarchies through the use of learned motor controllers, or skills. I will present new results establishing a link between the skills available to a robot and the abstract representations it should use to plan with them. I will then show that this representation acquisition phase can be combined with skill acquisition to build true action hierarchies for reinforcement learning problems.

Takashi Tanaka (KTH)

LQG Control with Minimal Information: A Semidefinite Programming Approach

Real-time decision-making procedures in general require continuous acquisition of information from the environment. In this talk, we revisit one of the most fundamental questions in real-time decision-making theory: what is the minimal information acquisition rate to achieve sequential decision-making with desired accuracy? We tackle this question using basic tools from control theory, information theory, and convex optimization theory. Specifically, we consider a Linear-Quadratic-Gaussian (LQG) control problem where Massey's directed information from the state sequence to the control sequence is taken into account. We show that the most "information-frugal" decision-making policy achieving desired LQG control performance admits an attractive three-stage separation structure comprised of (1) a linear sensor with additive Gaussian noise, (2) Kalman filter, and (3) a certainty equivalence controller. We also show that an optimal policy can be synthesized using a numerically efficient algorithm based on semidefinite programming (SDP).

Laurent Itti (USC)

Attention strategies for robotics

Visual attention allows primates to rapidly detect potential predators, prey or mates in the environment. Attention thus acts as a rapid heuristic to solving the complex problem of finding these potentially relevant and important items under strong time pressure. Many computational models have been developed to endow machines with a similar heuristic mechanism of attention. These algorithms rely on a rapid and shallow analysis of the incoming sensory data, which, by nature, often yields false positives, but also demonstrates a high hit rate for behaviorally relevant targets, as tested through comparisons of model outputs with eye tracking records from humans and monkeys. Here I will review attention theories and computational models, with a special emphasis on systems-level developments that use attention as part of broader processing pipelines for automated target detection and tracking, robot localization, and autonomous robot navigation.

Sertac Karaman (MIT)

Sparsity and Compression for Robot Planning and Perception

Sparsity and compression has attracted a tremendous amount of attention in signal processing during the last decade. It was shown that, signals that are sparse, can be reconstructed with very few measurements, when compared to the (worst-case) fundamental limits established by the Nyquist sampling theorem. In this talk, we apply similar ideas to robot motion planning/control and perception. First, we present a novel computational framework based on compression. Specifically, the new algorithms utilize a novel continuous version of the widely-used tensor decomposition methods to "compress" value functions and efficiently work on their compressed versions. The resulting algorithms efficiently compute exact (arbitrarily good) solutions to stochastic optimal control, estimation, inference, and uncertainty quantification problems. Their run time scales linearly with dimension and polynomially with the rank of the optimal cost-to-go function! In other words, we obtain polynomial time algorithms for "low-rank" problems. These problems suffer from the curse of dimensionality: all known exact algorithms run in time that is exponential with increasing dimensionality of the state space of the system, in the worst case. Second, we present novel algorithms for depth reconstruction with sensors that provide sparse (e.g., very low resolution) measurements. We characterize the conditions under which depth can be recovered from relatively few measurements using l1 minimization techniques. We demonstrate the new algorithms on simulated robots with 10-beam laser range finders for 2D mapping and stereo reconstruction with sparse measurements for 3D sensing.

Tim Genewein (Max Planck institute for Intelligent Systems, Tübingen)

Information-theoretic bounded rationality in perception-action systems

The ability to form abstractions and to generalize well from few samples are hallmarks of human and animal intelligence underlying the unrivaled flexibility of behavior in biological systems. Achieving such flexibility in artificial systems is challenging, particularly because the underlying computational principles are not fully understood. This talk introduces an information-theoretic framework for bounded rational decision-making, that is optimal decision-making under limited computational resources. One consequence of acting optimally under computational limitations is the emergence of natural abstractions which allow for more efficient processing of information. The consequent application of the theoretical framework to perception-action systems results in an interesting optimality principle that leads to a tight coupling between perception and action. As a result, the objective of bounded-optimal perception is not to represent a sensory state as faithfully as possible, but rather to extract the most relevant information for bounded-optimal acting.

Evangelos Theodorou (GaTech)

Fast decision making using parallel stochastic optimal control and inference

For autonomous systems to operate in stochastic environments, they have to be equipped with fast decision-making processes to reason about the best possible action. Grounded on first principles in stochastic optimal control theory and statistical physics, the path integral control framework provides a mathematically sound methodology for decision making under uncertainty. It also creates opportunities for the development of novel sampling-based planning and control algorithms that are highly parallelizable. In this talk, I will present results in the area of sampling-based adaptive stochastic control that go beyond classical formulations and show applications to robotics and autonomous systems for tasks such as high-speed navigation and multi-agent control. In addition to sampling-based stochastic trajectory optimization, alternative stochastic control methods that rely on concepts drawn from stochastic mechanics and their implications will be presented.