Deep Context Maps

Systems and methods for predicting the trajectory of an object are disclosed herein. One embodiment receives sensor data that includes a location of the object in an environment of the object; accesses a location-specific latent map, the location-specific latent map having been learned together with a neural-network-based trajectory predictor during a training phase, wherein the neural-network-based trajectory predictor is deployed in a robot; inputs, to the neural-network-based trajectory predictor, the location of the object and the location-specific latent map, the location-specific latent map providing, to the neural-network-based trajectory predictor, a set of location-specific biases regarding the environment of the object; and outputs, from the neural-network-based trajectory predictor, a predicted trajectory of the object.