The emergence of perceptual concepts through situated communicative interactions

Concepts play a crucial role in human cognition as they enable us to think, make sense of our sensory experiences, and share these experiences with others. As the meaning of a concept is deeply rooted in one's individual sensorimotor experiences, language becomes an indispensable tool for constructing meaningful symbolic abstractions over these continuous experiences. Current approaches to concept learning in artificial agents that bridge the gap between the continuous and symbolic domain make use of either supervised or unsupervised learning regimes. However, supervised approaches rely on large amounts of labelled examples, while unsupervised approaches require specifying the number of concepts to be learned in advance. Both approaches thereby lack the ability to dynamically expand the conceptual inventory of an artificial agent, making agents unable to adapt to a changing environment. Recently, Nevens et al. (2020) proposed a method by which concepts are formed by discriminative combinations of prototypical values on human-interpretable feature channels. While this method enables a learning agent to dynamically expand its conceptual inventory, it is still limited by its reliance on a tutor with an existing set of concepts.

In this talk, we present how a population of autonomous agents can self-organise a dynamic inventory of concepts in an incremental and data-efficient manner through a series of situated communicative interactions. Concretely, we extend the approach by Nevens et al. (2020) by equipping agents in the population with novel mechanisms for inventing, adopting, and aligning their conceptual representations. Through these mechanisms, the agents construct personal and dynamic inventories of concepts, which allow them to solve a communicative task. We validate our methodology using the benchmark CLEVR dataset (Johnson et al., 2017). In this experiment, a population of ten agents participates in pairwise communicative interactions. In each interaction, a speaker wants to draw the attention of a listener to an object through a concept describing its shape, size, material, or colour. We demonstrate that the agents construct an inventory of concepts from the ground up, achieving 100% communicative success after 10,000 interactions. By eliminating the need for a tutor, the agents are given the freedom to select relevant feature channels for each concept. They thereby construct concepts that are optimally adapted for their environment and the task at hand. Furthermore, the emergent conceptual system can be used to both comprehend utterances and describe novel instances. Lastly, concepts are constructed incrementally, enabling agents to use their concepts effectively even after a single interaction.

References

Nevens, J., Van Eecke, P., & Beuls, K. (2020). From continuous observations to symbolic concepts: A discrimination‐based strategy for grounded concept learning. Frontiers in Robotics and AI, 7, 84. https://doi.org/10.3389/frobt.2020.00084

Johnson, J., Hariharan, B., van der Maaten, L., Fei‐Fei, L., Lawrence Zitnick, C., & Girshick, R. (2017). CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In R. Chellappa, Z. Zhang, & A. Hoogs (Eds.), Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2901–2910).