The visual system is the largest sensory system in the brain. It is sophisticated and complex. The number of visual neurons expands from about a million retinal ganglion cells to about 8 billion cells in the cerebral cortex. fMRI studies reveal two streams of information in the visual cortex, a ventral stream related mostly to "what" an object is, and a dorsal stream related to "where" it is. But even in the retina the visual signal is split up into multiple channels that convey real-time information which would be difficult to compute sequentially from an image in the allotted time.

The human visual field is shown in the figure below, it's about 200 degrees horizontally and about 130 degrees vertically. The fields from each eye overlap in the central 120 degrees or so, and the high resolution foveal area is actually quite small, covering about 5 degrees of horizontal arc.

The human retina emits a little north of a million fibers from its ganglion cells. The ones we're discussing now, travel from the retina through the thalamus and into the occipital lobe of the cerebral cortex. (Others connect with the superior colliculus, which we discuss elsewhere, and other structures like the suprachiasmatic nucleus which is involved in light-related circadian rhythms, and the olivary pretectal nucleus which is involved with light-driven pupil constriction). The figure shows the general plan of the early visual system, specifically the pathway from the retina to the primary visual cortex.

The nerve fibers from each retina cross in the optic chiasm. The nasal hemifield from one is combined with the temporal hemifield from the other, and the result is that the left side of the brain processes the right side of the visual field, and vice versa. The retina "knows" its orientation and its subdivisions early in development, based on chemical gradients. Later in development there is also oscillatory neural activity that helps to self-organize the connections.

The retina is a layered structure, and it is inverted with respect to light - in other words the photoreceptors are on the inside, nearest the brain. Light has to travel through the ganglion cells to get to the photoreceptors. This figure shows the general orientation of the neurons within the retina.

The number of layers in the retina keeps increasing as our experimental methods get better, Ramon y Cajal said 5 but newer results bring the number closer to 10. Some layers contain cells, some layers contain fibers and synapses, and some contain both. The general connection pattern of the retina is photoreceptors => bipolar cells => ganglion cells. Laterally between the photoreceptors and the bipolar cells are the horizontal cells, and laterally between the bipolar cells and the ganglion cells are the amacrine cells. This pattern is schematized in the figure.

There are four main types of photoreceptors, a dozen kinds of bipolar cells, and about 20 different varieties of ganglion cells.

The electrical activity from the retina associated with an optical stimulus is shown in the figure below. Historically it was detected as early as the mid-1860's by placing electrodes on the cornea. This "electroretinogram" is also visible in other places (it can be detected with a special contact lens, and it is also visible using ordinary electrodes on the forehead and temple). The ERG is subject to volume conductor effects from the vitreous humor and other parts of the eye and nearby tissue.

Photoreceptors

The right eye is shown schematically in the figure below. The typical human eye is about 2.4 cm from front to back, it is approximately spherical but slightly wider than it is deep. The retina is in back of the eye, light has to travel through the vitreous humor to get to the photoreceptors. Not only that, but the retina is inverted with respect to light, the photoreceptors are in back along the choroid wall and the nerve fibers from the ganglion cells exit in the front and travel laterally to enter the optic nerve at the blind spot.

Humans have about 126 million photoreceptors, of which 120 million are rods and 6 million are cones. The distribution of photoreceptors in a human retina is shown in the figure.

The retinal photoreceptors hyperpolarize in response to light, they have a baseline membrane potential that goes negative as a result of ion influx. This process is shown in the figures below.

There are four main types of photoreceptors, their characteristics are shown below. The density of small color-sensitive photoreceptors is highest in the foveal region of the eye, while the density of wider field contrastive cells is highest in the periphery. The cones are called S, M, and L for blue, green, and red respectively, based on their preferred wavelength of light. They are arranged in a mosaic pattern within the retina.

The various photoreceptors form a mosaic pattern on the retina. The mosaic is organized in a radial manner, with a very high concentration of cones at the fovea, that tapers off in the periphery. The figure shows the mosaic pattern of color-sensitive cones in a normal retina (left) and a retina with protanopsia (in this case, no red cones). The foveal area is outlined.

Photoreceptors of all kinds synapse on bipolar cells, using glutamate as a neurotransmitter. In addition, they form gap junctions and triadic synapses with horizontal cells, of which there are several different kinds. Generally speaking only one or a few photoreceptors connect to a bipolar cell in the high resolution area of the fovea, whereas the number may be slightly higher in the periphery.

There are gap junctions between rods, between cones, and between rods and cones. These gap junctions are associated with various connexin proteins, that allow ions and small molecules to move between cells. Electrically, the movement of sodium, calcium, potassium, and chloride between cells creates a syncytium, whose properties can be modeled as one large membrane. Some of the gap junctions are rectifying, they only allow movement in one direction. Gap junctions trade off a small amount of visual acuity for better contrast enhancement. In the cone syncytium, the extent of voltage penetration away from a cone is only about 0.5 cone widths, indicating that a single cone at most affects its nearest neighbors via gap junctions (DeVries et al 2002).

Bipolar Cells and Horizontal Cells

Photoreceptors feed forward to bipolar cells. There are two kinds of bipolar cells, those that respond to increased illumination, and those that respond to decreased illumination. These two responses are provided by two different kinds of neurotransmitter receptors that process the glutamate released by photoreceptors. The sign-preserving (off-center) synapses use the ionotropic kainate-type GRIK-1 glutamate receptor, whereas the sign-reversing (on-center) synapses use the metabotropic mGlu-R6 (GRM6) receptor. These two response types are sometimes called "ON" and "OFF", but that's really a misnomer at this level. An ON bipolar cell will respond whenever the level of illumination in the center exceeds that in the periphery, and an OFF bipolar cell will respond when the center is darker than the periphery. Thus the primary axis of differentiation is spatial rather than temporal.

Between the photoreceptors and the bipolar cells are the horizontal cells, which connect laterally in the outer plexiform layer. In humans there are three different kinds of horizontal cells, called H1, H2, and H3. The H1 type sometimes has an axon that travels laterally in a preferred direction, whereas the H2 and H3 types are axonless. Photoreceptors excite horizontal cells via the neurotransmitter glutamate, and horizontal cells inhibit photoreceptors via the neurotransmitter GABA.

The surround field can be loosely modeled as a "sum of Gaussians", where the middle peak is determined by the viewing field of the photoreceptor and the edges are determined by the extent of the horizontal cell. In real life the situation is a bit more complicated, because of both of these patterns are dynamic, they undergo adaptation and other forms of plasticity that help the eye adjust to changing light conditions.

Horizontal cells generate an inhibitory surround field for the bipolar cells. Neighboring horizontal cells are coupled by gap junctions that are much more permeable than those between photoreceptors. A typical horizontal cell receptive field can be up to 20 times larger than the extent of its dendritic arborization. The surround does not depend on the bipolar cells, it is generated by a cooperative process between the photoreceptors and the horizontal cells.

In humans and trichromatic primates, the H1 class of horizontal cell shapes M and L cone output (red and green), and the H2 class shapes S cone output (blue) and contributes to S-M/L opponency (Chapot et al 1997).

The surround is not necessarily color-specific, for example the red and green cones both contact the H1 horizontal cells without regard to color specificity. There is no "green horizontal cell", the same horizontal cell generates the surrounds for both red and green cones. In humans, the axons from the H1 cells terminate mainly on nearby rods, and can be strongly directional.

Though there are only two main types of horizontal cells, there are at least 12 different kinds of bipolar cells in humans. One of them is a specialized rod bipolar cell that only contacts AII amacrine cells (20-80/cell) and does not connect directly with ganglion cells. The others feed forward to retinal ganglion cells.

The inhibitory surrounds of bipolar cells are generated by specialized synapses between the receptors, horizontal cells, and bipolar cells. These surrounds are generated when activity in one photoreceptor activates the connected horizontal cell, which then inhibits nearby receptors. In real life during changing light conditions, the horizontal cell synapses are part of the circuitry related to adaptive gain control (Scholl et al 2012, VanLeeuwen et al 2009). Bipolar cells also participate in adaptive gain control (Beaudoin et al 2007).

The synaptic web between the photoreceptors and horizontal cells sends signals with a very small delay, in a similar range to synaptic delays, around or under 1 msec (Trenholm & Awatramani 2025, Gollisch & Meister 2010). Therefore the inhibitory surrounds of bipolar cells activate without much delay, after an optical stimulus. Most of the early processing in the retina is done by graded transmission in the absence of action potentials. Photoreceptors and horizontal cells do not generate action potentials at all, and of the 12 types of bipolar cells only the three fastest (transient) types generate action potentials. In the retina, it is mainly the amacrine and ganglion cells that generate action potentials (amacrine cells may or may not, and ganglion cells always do). In this context, the organization of gap junctions and the creation of electrical syncytia within the retinal neural network is important to understand.

Bipolar cells excite ganglion cells, they use glutamate as a neurotransmitter. In humans 12 types of bipolar cells have been identified, 5 are on-type cone bipolars, one is an on-type rod bipolar, and 6 are off-type cone bipolars. The different types of bipolar cells segregate local information into at least 20 independent "channels" that exit the retina at the level of the ganglion cells. Each of these channels is topographically consistent, and the result is an overlay of various "views" of the retinal input. Each view is filtered in a different way, some carry color information, some carry motion information, and some carry general luminance. The channels remain segregated on their way to the visual cortex. This is a great resource for visualizing cells in the retina.

Neither retinal photoreceptors nor horizontal cells generate action potentials. Instead they use graded potentials, and the overall shape of response is mostly sigmoidal with a nearly linear region in the middle. Bipolar cells, on the other hand, can use both graded and action potentials, as can certain kinds of amacrine cells. As far as we know right now, retinal ganglion cells communicate with the LGN exclusively via action potentials. On and off type bipolar cell axons ramify in specific layers within the retina's inner plexiform layer (IPL). This arrangement is shown in one of the figures below. The fastest connections are towards the middle of the IPL, and these are where the bipolar cell action potentials can be found, in at least 3 of the 12 bipolar cell types (Baden et al 2013).

Ganglion Cells and Amacrine Cells

The bipolar cells send outputs to a bewildering array of amacrine cells and ganglion cells, and the amacrine cells in turn connect back with the ganglion cells, both synaptically and electrically. Over 20 different kinds of retinal ganglion cells have been identified in humans and primates, and many of these obtain their properties (both static and dynamic) from the amacrine cells. Each different combination of ganglion and amacrine cells performs a local filtering operation on the input from bipolar cells. The filter acts in both space and time, as many of the inhibitory effects from amacrine cells are synaptic and have time constants on the order of milliseconds.

The inner plexiform layer is where the retinal output acquires its dynamics. Many of the retinal ganglion cells (especially the M-type cells) respond transiently to visual stimuli. These transient characteristics are introduced at the level of the amacrine cell, and research to date has shown that these amacrine-induced dynamics are more significant than the differences in glutamate receptor response times in the bipolar cells, in determining ganglion cell output. The figure shows the stratification of the inner plexiform layer. Off cells are on top (closest to the photoreceptors), on cells are on the bottom (closest to the ganglion cells). In between the sustained types are layers with transient response. These differing responses come from various combinations of bipolar and amacrine cell input.

The figure below shows some of the varieties of amacrine cells. Particular varieties can use GABA, glycine, acetylcholine, dopamine and other neurotransmitters. One of the well studied amacrine cells in the inner nuclear layer of the retina is the A-II cell, which gets inputs from rod bipolars and distributes them to cone bipolars (Kolb, 1997). The figure shows the stratification of some of the 40+ different kinds of retinal amacrine cells. Even though this figure is from a rabbit, you can appreciate the specificity of the wiring in the inner plexiform layer.

Amacrine cells participate in retinal gain control, and they help report the direction of movement to ganglion cells. In addition, the rapid response properties of certain kinds of amacrine cells throttle the visual output during ballistic eye movements, helping to avoid blurred images.

The stratification of the inner plexiform layer is relevant for the behavior of the ganglion cells. Most of the synapses between bipolar cells, amacrine cells, and ganglion cells are carefully organized with this layer. The IPL is further subdivided into at least 5 distinct strata, each of which contains a particular type of connection, or multiple types. Kolb reviews at least 40 different kinds of amacrine cells here. As an example of IPL organization, the high-level stratification at the level of on and off responses is shown in the figure.

The information from the photoreceptors is combined in different ways into the retinal ganglion cells. Some examples are shown in the figure. There are over 20 different kinds of retinal ganglion cells that pass filtered versions of the retinal image into the brain. Many of these pathways are kept segregated in the central areas, and it is not entirely clear whether and how they may be eventually combined into a unified visual perception. Most likely, the human brain performs correlations between the information in the various retinal channels, to extract the statistical properties of the visual input.

There are only a very few centrifugal fibers that enter the retina from the brainstem and the central nervous system. Most of the ganglion cells send projections into the optic chiasm, with a few of them traveling to the superior colliculus. The ganglion cells are organized into "channels" that convey different kinds of information. Many are static color sensitive or color insensitive cells that respond mainly to luminance and luminance contrast. Others are highly dynamic and may be both motion and direction sensitive.

The richness of dynamic behavior in the retina is important to understand. It is reflected not only in the dynamic behaviors of individual neurons, but also in the form of traveling waves that augment precision in the population and become very important during development. Many of the dynamics are provided by the amacrine cell layer(s).

Finally, there is a small subset of intrinsically photoreceptive retinal ganglion cells, that contain the photopigment melanopsin.

Retinal ganglion cells in primates are broadly classified into three categories: P cells, M cells, and K cells. These correspond approximately, to the X, Y, and W cells in cats. The P (parvocellular) cells exhibit mostly tonic responses to light, whereas the M (magnocellular) neurons respond transiently to onset and offset. The K (koniocellular) neurons have distinct targets, they project to different areas of the LGN and superior colliculus, and the LGN pathways have different endpoints in the primary visual cortex.

Center-Surround Organization

With respect to light stimuli, the ganglion cells in the retina act like filters. The filters have a "Mexican hat" shape, they have a tall central area that responds to input with excitation, and smaller peripheral lobes that respond to input with inhibition. These filter shapes translate directly to response shapes in the visual field. This relationship is called the "receptive field" of the neuron.

While the retinal photoreceptors respond to luminance (light intensity), the center-surround filters are well suited for processing contrast. It is the contrast that is most meaningful, it has to be extracted in both low light and bright light. (Object definition should not depend on light intensity). When studying the center-surround organization of retinal receptive fields, it is important to understand that they are not static entities. Stimulation of a single spot on the retina creates a multiplicity of responses in retinal ganglion cells, some when the light turns on, some when it turns off, and some during the entire duration of the stimulus. Additionally, some retinal ganglion cells are direction sensitive, and some of the surrounds are too. Both the center and the surround take time to develop, and generally speaking the surrounds involve more synapses so they take longer (Turner et al 2018, Wienbar & Schwartz 2018).

The filters in the retina pre-process the visual signal before it ever gets to the brain. The output of the retina is already gain-adapted, and subject to the centrally controlled effects of pupil constriction and lens accommodation. Generally speaking the filters cause ganglion cells to respond well to edges and transients, as depicted in these drawings.

Any or all of the P/M/K ganglion cells can be color sensitive and direction sensitive. Color sensitivity is conferred by a particular combination of retinal cones, and direction sensitivity can be generated by geometric specificity of both axon and dendrite branching patterns.

In addition to the time-varying elements of receptive fields, there is also local and global adaptation mediated at the synaptic level.

By the time the visual signal leaves the retina, it has been considerably modified, although its original character is still recognizable. Filtering by the retinal circuitry guarantees that the image leaving the retinal ganglion cells is not the same as the pattern of light that created it. So how then, does the brain reconstruct the original visual input from multiple channels that distort its original character?

To answer this question, we have to keep going and look at what happens to the visual information as it gets transmitted centrally from the eye. The first stop along the way to the cerebral cortex is the lateral geniculate nucleus of the thalamus, which receives inputs from retinal ganglion cells that feed the optic nerve.