Object-based visual attention for computer vision. (English) Zbl 1082.68839

Sun, Yaoru; Fisher, Robert

Artif. Intell. 146, No. 1, 77-123 (2003).

Summary: A novel model of object-based visual attention extending Duncan’s Integrated Competition Hypothesis [Philos. Trans. R. Soc. Lond., Ser. B 353, 1307–1317 (1998)] is presented. In contrast to the attention mechanisms used in most previous machine vision systems which drive attention based on the spatial location hypothesis, the mechanisms which direct visual attention in our system are object-driven as well as feature-driven. The competition to gain visual attention occurs not only within an object but also between objects. For this purpose, two new mechanisms in the proposed model are described and analyzed in detail. The first mechanism computes the visual salience of objects and groupings; the second one implements the hierarchical selectivity of attentional shifts. The results of the new approach on synthetic and natural images are reported.

MSC:

68T45

Machine vision and scene understanding

Keywords:

Visual attention; Object-based visual attention; Integrated competition; Grouping salience; Hierarchical selectivity

References:

[1]	Ahrns, I.; Neumann, H., Space-variant dynamic neural fields for visual attention, (Proc. IEEE Computer Vision and Pattern Recognition, Fort Collins, CO (1999)), 313-318
[2]	Baluja, S.; Pomerleau, D., Dynamic relevance: Vision-based focus of attention using artificial neural networks, Artificial Intelligence, 97, 381-395 (1997) · Zbl 0904.68140
[3]	Baluja, S.; Pomerleau, D., Expectation-based selective attention for visual monitoring and control of a robot vehicle, Robotics and Autonomous Systems, 22, 329-344 (1997)
[4]	Behrmann, M.; Zemel, R. S.; Mozer, M. C., Occlusion, symmetry, and object-based attention: Reply to Saiki (2000), J. Exp. Psych.: Hum. Percept. Perf., 26, 4, 1497-1505 (2000)
[5]	Bundesen, C., A computational theory of visual attention, Phil. Trans. R. Soc. London B, 353, 1271-1281 (1998)
[6]	Burt, P., Attention mechanisms for vision in a dynamic world, (Proc. Ninth International Conference on Pattern Recognition, Beijing, China (1988)), 977-987
[7]	Carpenter, G.; Grossberg, S.; Lesher, G., The representation of visual salience in monkey parietal cortex, Nature, 391, 481-484 (1998)
[8]	Clark, J. J.; Ferrier, N., Modal control of an attention vision system, (Proc. IEEE Internat. Conf. Computer Vision, Tarpon Springs, FL (1988)), 514-523
[9]	Clark, J. J., Spatial attention and latencies of saccadic eye movements, Vision Res., 39, 3, 583-600 (1998)
[10]	Concepcion, V.; Wechesler, H., Detection and localization of objects in time-varying imagery using attention, representation and memory pyramids, Pattern Recognition, 29, 9, 1543-1557 (1996)
[11]	Cowan, W., Evolving conceptions of memory storage, selective attention and their mutual constraints within the human information-processing system, Psychol. Bull., 104, 163-191 (1988)
[12]	Crick, F.; Koch, C., Towards a neurobiological theory of consciousness, Seminars in the Neurosciences, 2, 263-275 (1990)
[14]	Desimone, R.; Duncan, J., Neural mechanisms of selective visual attention, Ann. Rev. Neurosci., 18, 193-222 (1995)
[15]	Desimone, R., Visual attention mediated by biased competition in extrastriate visual cortex, Phil. Trans. R. Soc. London B, 353, 1245-1255 (1998)
[16]	Driver, J.; Baylis, G. C., Attention and visual object segmentation, (Parasuraman, R., The Attentive Brain (1998), MIT Press: MIT Press Cambridge, MA), 299-325
[17]	Driver, J.; Davis, G.; Russell, C.; Turatto, M.; Freeman, E., Segmentation, attention and phenomenal visual objects, Cognition, 80, 61-95 (2001)
[18]	Duncan, J., Selective attention and the organization of visual information, J. Exp. Psychol., 113, 501-517 (1984)
[19]	Duncan, J.; Humphreys, G. W., Visual search and stimulus similarity, Psychological Rev., 96, 433-458 (1989)
[20]	Duncan, J., Target and non-target grouping in visual search, Perception and Psychophysics, 57, 1, 117-120 (1995)
[21]	Duncan, J., Coordinated brain systems in selective perception and action, (Iaui, T.; McClelland, J. L., Attention and Performance XVI (1996), MIT Press: MIT Press Cambridge, MA), 549-578
[22]	Duncan, J., Integrated mechanisms of selective attention, Curr. Opin. Biol., 7, 255-261 (1997)
[23]	Duncan, J., Converging levels of analysis in the cognitive neuroscience of visual attention, Phil. Trans. R. Soc. London B, 353, 1307-1317 (1998)
[24]	Egeth, H. E.; Yantis, S., Visual attention: Control, representation, and time course, Ann. Rev. Psychol., 48, 269-297 (1997)
[25]	Egly, R., Shifting visual attention between object and locations: Evidence from normal and parietal lesion subjects, J. Exp. Psychol. Hum. Percept., 123, 161-177 (1994)
[26]	Engle, S.; Zhang, X.; Wandell, B. A., Colour tuning in human visual cortex measured with functional magnetic resonance imaging, Nature, 388, 6637, 68-71 (1997)
[27]	Eriksen, C. W.; Yeh, Y. Y., Allocation of attention in the visual field, J. Exp. Psychol.: Hum. Percept. Perf., 11, 5, 583-597 (1985)
[28]	Eriksen, C. W.; James, J. D.St, Visual attention within and around the field of focal attention: A zoom lens model, Perception and Psychophysics, 40, 4, 225-240 (1986)
[29]	Exel, S.; Pessoa, L., Attention visual recognition, International Conference on Pattern Recognition, Brisbane, Australia (1998)
[30]	Farah, M. J., “What” and “where” in visual attention: Evidence from the neglect syndrome, (Unilateral Neglect: Clinical and Experimental (1993)), 123-138
[31]	Ferrara, V.; Lisberger, S., Attention and target selection for smooth pursuit eye movements, J. Neurosci., 15, 11, 7472-7484 (1995)
[32]	Fink, G. R., Space-based and object-based visual attention: Shared and specific neural domains, Brain, 120, 2013-2028 (1997)
[33]	Folk, C. H.; Remington, W. R.; Wright, J. H., The structure of attentional control: contingent attentional capture by apparent motion, abrupt onset, and color, J. Exp. Psychol.: Hum. Percept. Perf., 20, 2, 317-329 (1994)
[34]	Gottlieb, J. P., The representation of visual salience in monkey parietal cortex, Nature, 391, 6666, 481-484 (1998)
[35]	Greenspan, H.; Belongie, S.; Goodman, R.; Persona, P.; Rakshit, S.; Anderson, C. H., Overcomplete steerable pyramid filters and rotation invariance, (Proc. IEEE Computer Vision and Pattern Recognition, Seattle, WA (1994)), 222-228
[36]	Grimson, W. E.L, An active visual attention system to play “Where”s Waldo”, (Proc. Conference on Computer Vision and Pattern Recognition, Seattle, WA (1994)), 85-90
[37]	Grossberg, S., A neural theory of attentive visual search: interactions of boundary, surface, spatial and object representations, Psychological Rev., 10, 3, 470-489 (1994)
[38]	Grossberg, S., How does the cerebral cortex work? Learning, attention, and grouping by the laminar circuits of visual cortex, Spatial Vision, 12, 2, 13-185 (1999)
[39]	Grossberg, S.; Raizada, R., Contrast-sensitive perceptual grouping and object-based attention in the laminar circuits of primary visual cortex, Vision Res., 40, 1413-1432 (2000)
[40]	Grove, T. D.; Fisher, R. B., Attention in iconic object matching, (Proc. BMVC96, Edinburgh (1996)), 293-302
[41]	Heinke, D.; Humphreys, G. W., SAIM: A model of visual attention and neglect, (Proc. International Conference on Artificial Neural Networks, New York (1997)), 913-918
[42]	Humphreys, G. W., SEarch via recursive rejection (SERR): A connectionist model of visual search, Cognitive Psychology, 25, 43-110 (1993)
[43]	Humphreys, G. W., Neural representation of objects in space: A dual coding account, Phil. Trans. R. Soc. London B, 353, 1341-1351 (1998)
[44]	Hoffman, J. E., Visual attention and eye movements, (Pashler, H., Attention (1998), Psychology Press), 119-154
[45]	Itti, L.; Koch, C.; Niebur, E., A model of saliency-based visual attention for rapid scene analysis, IEEE Trans. Pattern Anal. Machine Intel., 20, 11, 1254-1259 (1998)
[46]	Itti, L.; Koch, C., A saliency-based search mechanism for overt and covert shifts of visual attention, Vision Res., 40, 10-12, 1489-1506 (2000)
[47]	Johnson, E. N.; Hawken, M. J.; Shapley, R., The spatial transformation of color in the primary visual cortex of the macaque monkey, Nature, 4, 4, 409-416 (2001)
[48]	Kahneman, D.; Henik, A., Perceptual organization and attention, (Kubovy, M.; Pomerantz, J. R., Perceptual Organization (1984), Erdbaum: Erdbaum Hillsdale, NJ), 181-211
[49]	Kaiser, P.; Boynton, R. M., Human Color Vision (1996), Optical Society of America
[50]	Kastner, S.; Ungerleider, L. G., Mechanisms of visual attention in the human cortex, Ann. Rev. Neurosci., 23, 315-341 (2000)
[51]	Kazanovich, Y. B.; Borisyuk, R. M., Dynamics of neural networks with a central element, Neural Networks, 12, 441-454 (1999)
[52]	Kowler, E., The role of attention in the programming of saccades, Vision Res., 35, 13, 1897-1916 (1995)
[53]	Koch, C.; Ullman, S., Shifts in selective visual attention: towards the underlying neural circuitry, Human Neurobiology, 4, 219-227 (1985)
[54]	Kramer, A. F.; Jacobson, A., Perceptual organization and focused attention: The role of objects and proximity in visual processing, Perception and Psychophysics, 50, 267-284 (1991)
[55]	Kryukov, V. I., An attention model based on the principle of dominanta, (Holden, A. V.; Kryukov, V. I., Neurocomputers and Attention I: Neurobiology, Synchronization and Chaos (1991), Manchester University Press: Manchester University Press Manchester), 319
[56]	LaBerge, D., Attentional Processing: The Brain’s Art of Mindfulness (1995), Harvard University Press: Harvard University Press Harvard
[57]	Lavie, N., Perceptual load as a necessary condition for selective attention, J. Exp. Psychol.: Hum. Percept. Perf., 21, 451-468 (1995)
[58]	Lavie, N.; Driver, J., On the spatial extent of attention in object-based selection, Perception and Psychophysics, 58, 1238-1251 (1996)
[59]	Logan, G. D., The CODE theory of visual attention: An integration of space-based and object-based attention, Psychological Rev., 103, 4, 603-649 (1996)
[60]	Luck, S. J., Neurophysiology of selective attention, (Pashler, H., Attention (1998), Psychology Press), 257-295
[61]	McPeek, R. M., Saccades require focal attention and are facilitated by a short-term memory system, Vision Res., 39, 1555-1566 (1999)
[62]	Nemcsics, A., Color Dynamics (1993), Akademiai Kiad: Akademiai Kiad Budapest
[63]	Niebur, E., An oscillation based model for the neuronal basis of attention, Vision Res., 33, 2789-2802 (1993)
[64]	Niebur, E.; Koch, C., A model for the neuronal implementation of selective visual attention based on temporal correlation among neurons, J. Neurosci., 1, 141-158 (1994)
[65]	Nothdurft, H. C., The conspicuousness of orientation and motion contrast, Spatial Vision, 7, 4, 341-363 (1993)
[66]	Olshausen, B. A., A neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information, J. Neurosci., 13, 11, 4700-4719 (1993)
[67]	Palmer, S. E., Vision Science-Photons to Phenomenology (1999), MIT Press: MIT Press Cambridge, MA
[68]	Pashler, H., The Psychology of Attention (1998), MIT Press: MIT Press Cambridge, MA
[69]	Patel, G. A.; Sathian, K., Visual search: bottom-up or top-down?, Frontiers in Bioscience, 5, 169-193 (2001)
[70]	Posner, M. E., Orienting of attention, Q. J. Exp. Psychol., 32, 3-25 (1980)
[71]	Postma, E. O., SCAN: A scalable model of attentional selection, Neural Networks, 10, 993-1015 (1997)
[74]	Robinson, D. J.; Peterson, S. E., The pulvinar and visual salience, Trends in Neuroscience, 15, 4, 127-132 (1992)
[75]	Rybak, I. A., A model of attention-guided visual perception and recognition, Vision Res., 38, 2387-2400 (1998)
[76]	Scholl, B. J., Objects and attention: The state of the art, Cognition, 80, 1-46 (2001)
[77]	Shokoufandeh, A., View-based object recognition using saliency maps, Image and Computing, 17, 445-460 (1999)
[78]	Sillito, A. M., Visual cortex mechanisms detecting focal orientation discontinuities, Nature, 378, 492-496 (1995)
[79]	Singer, W.; Gray, C. W., Visual feature integration and the temporal correlation hypothesis, Ann. Rev. Neurosci., 18, 555-586 (1995)
[80]	Sela, G.; Levine, M. D., Real-time attention from robotic vision, Real-Time Imaging, 3, 173-194 (1997)
[82]	Takacs, B.; Wechsler, H., A dynamic and multiresolution model of visual attention and its application to facial landmark detection, Computer Vision and Image Understanding, 70, 1, 63-73 (1998)
[83]	Treisman, A.; Gelade, G., A feature integration theory of attention, Cognition Psychology, 12, 97-136 (1980)
[84]	Treisman, A., Features and objects: The fourteenth Bartlett Memorial lecture, Q. J. Experimental Psychology, 40A, 201-237 (1988)
[85]	Treisman, A., The perception of features and objects, (Baddeley, A.; Weiskrantz, L., Attention: Selection, Awareness, and Control (1993), Uarendon Press: Uarendon Press Oxford), 5-35
[86]	Tsotsos, J. K., Modelling visual attention via selective tuning, Artificial Intelligence, 78, 507-545 (1995)
[87]	Usher, M.; Donnelly, N., Visual synchrony affects binding and segmentation in perception, Nature, 394, 179-182 (1998)
[88]	Chichilnisky, E. J.; Wandell, B. A., Trichromatic opponent color classification, Vision Res., 39, 20, 3444-3458 (1999)
[89]	Wandell, B. A., Computational neuroimaging: color representations and processing, (Gazzaniga, M. S., New Cognitive Neuroscience (1999), MIT Press: MIT Press Cambridge, MA)
[90]	Westin, C. F., Attention control for robot vision, (Proc. IEEE Computer Vision and Pattern Recognition, San Francisco, CA (1996)), 18-20
[91]	Wolfe, J. W., Guided Search 2.0: A revised model of visual search, Psychonomic Bulletin and Review, 1, 202-238 (1994)
[92]	Wolfe, J. W., Visual search, (Pashler, H., Attention (1998), Psychology Press), 13-73
[93]	Wyszechi, G.; Stiles, W. S.; Wyszecki, G.; Wyszecki, G., Color Science: Concepts and Methods, Quantitative Data and Formulae (2000), Wiley: Wiley New York
[94]	Yantis, S., Control of visual attention, (Pashler, H., Attention (1998), Psychology Press), 223-256