As mentioned already, there are two "streams" of visual processing in the human brain, a ventral stream related to "what" an object is, and a dorsal stream related to "where" it is. However in the context of a neural timeline mapping brain electrical activity, we can note something important about the concept of an "object", in that it's invariant with respect to time (whereas its position and orientation and velocity may not be). The concept of an object as an invariant abstraction one can label and attach a name to, has important ramifications in relation to a timeline mapping, because the invariances have to be specifically separated from the other information, which requires specialized neural processing.

From a machine learning standpoint, we get some of this for free in convolutional networks, which are very good at extracting invariances, and indeed there are portions of the visual cortex that look very much like a convolutional network. However as mentioned in the earlier section on neurons, these CNN's are highly non-biological for the most part, and don't exhibit the complex dynamics seen in the human brain.

The architecture of the visual streams tells us other important things too. For one, it is important to understand the many ways they come together, and for what reasons. Our brains do not simply store all the information they get, that would be a lot of bits, even for trillions of synapses. Instead, our brains extract the relevant information, and then reconstruct scenes in such a way that they contain only the relevant information. This process is essential to short term memory, and it is well worth studying the circuitry in and around the hippocampus to see how it works.

Anatomy of the Visual Streams

From the primary visual cortex, the visual signal feeds forward into V2, the secondary visual cortex. V2 is like a shell that surrounds V1. It has very specific architecture, that treats inputs from ocular dominance columns, orientation columns, and blobs differently.

From V2, the visual signal splits into multiple streams. There is a dorsal stream that goes up along the top of the brain into the parietal lobe and from there to the frontal eye fields, and there is a ventral stream that goes down into the temporal lobe and eventually feeds the scene mapping circuitry in the hippocampus. Along the way there are cortical visual areas specific for color, motion, faces, object recognition, and 3d spatial reconstruction.

Early Visual Processing

V1 itself has multiple channels, some of which are color specific and motion specific, however when this information is forwarded to subsequent stages it is organized in different ways. The connection from V1 to V2 is very specific.

There seems to be a separation of motion and position related information for objects, as distinct from object locations. These two pathways are the ventral and dorsal visual streams. The ventral stream goes into the temporal lobe, innervating areas like the inferotemporal cortex that are involved in object identification and labeling ("semantics", and the object processing areas are in fact located very close to Wernicke's area). The dorsal stream goes into the parietal cortex, around an area called LIP in humans and primates (near the lateral intraparietal sulcus, behind the splenium of the corpus callosum around the retrosplenial area). Area LIP is involved in visual attention, directing the eyes to move to objects of interest and areas of interest within objects.

There are also massive projections from almost all areas of the visual cortex to the pulvinar area of the thalamus, and the frontal eye fields. The pulvinar is thought to be related to attention in various modalities, while the frontal eye fields initiate voluntary saccades even in the absence of visual stimuli. The pulvinar receives direct retinotopic projections from the superior colliculus, which we'll discuss in the next section on eye movements.

Humans can "pay attention" to visual stimuli, or not. Sometimes, paying attention involves eye movements. There are visuall driven eye movements that seem to be mostly organized in the parietal lobe, and voluntary movements that seem to be mostly organized in the frontal lobe. These two areas talk to each other, they have direct reciprocal connections and they light up like Christmas trees in a functional MRI. The pulvinar portion of the attention system is the visually driven part. Whereas, the frontal lobe attention systems are considerably more complex, involving the anterior cingulate cortex in addition to the scene mapping circuitry around the hippocampus.

Object Identification

Object identification is undertaken by the "what" pathway, although there are notably visual object cells in areas like the claustrum. Objects are not just visual entities, they also have semantic meaning (especially insofar as they are "labeled" by words). Central to object identification and labeling is the inferior temporal cortex, in area called IT which is very close to and even overlaps Wernicke's area for spoken and written speech. The figure shows a map of the human temporal lobe with the numbered Brodmann areas. Wernicke's area is closest to area 22, while the visual IT is closest to area 20, overlapping areas 21, 37, and parts of 19.

Object Localization and Characterization

Objects are invariant, but their properties are not. Real objects that appear in the visual field are "examples" of abstract objects that are stored in memory. While an object is present in the visual field, it's always the same object, but its properties change, its shape depends on the viewing angle, and its textures depend on lighting and shadows.

The "where" stream tracks the variant properties of objects in the visual field. One of the important processing stations in this dorsal stream isarea MT in the middle temporal lobe, which processes motion. From area MT, fibers enter the spatial processing areas in the parietal lobe, including area 7 in the superior parietal lobule and the retrosplenial area which includes portions of the area called IPL in the inferior parietal lobule (area IPL is important for eye movements, we'll look at it more extensively in the next section).

The parietal lobe is involved in the spatial orientation of the organism within its environment, and that includes the position of the body as well as its parts (head, neck, eyes, and so on).

The dorsal and ventral visual streams join in the area around the hippocampus, which handles scene reconstruction and short term memory. Breaking up the visual information into two parts is a clever strategy on the part of the visual system, because it saves a lot of computation time. The general strategy in the visual system is break-up-and-reconstruct, there are at least 20 retinotopic maps in the visual system that are kept in register all the way through the hippocampal area. Computations are performed in parallel, and the results recombined as needed.

Scene Reconstruction

The reconstruction of visual scenes and their transfer into short term (working) memory is a very complicated subject, and we can only touch on it in the most rudimentary way on these pages. Please refer to the bibliography for a wealth of information on scene reconstruction in the hippocampus.

To build a scene, the brain combines information from the "where" pathways and the "what" pathways. Relative to episodic memory, an episode can be considered as a collection of scenes. A scene could be something navigable, like a maze, where the objects all stay in one place but different views create different visual configurations. Or, it could be something relatively static like a room in one's own house, where one can sometimes find things without even looking.

The functional connectivity of the visual pathways into the hippocampal area is shown in the figures below, as determined by magnetoencephalography and diffusion tractography.