While individual neurons have substantial computational power, neurons in populations can display actual intelligence. There is enormous computational power in populations of neurons, and even more when we can combine them in meaningful ways. In previous sections we looked at some of the behaviors that neurons in populations can accomplish, like precision extraction of a peak from a collection of imprecise samples, and the creation of brain waves from a collection of simple excitatory and inhibitory connections. Now we can consider population activity in the larger context of structured geometry. This includes layers, topography, and modular organization.

Layered Structures

There are many conveniently layered structures in the human brain. The figure shows the layering of the early visual pathway, from the thalamus to the cerebral cortex. Other layered structures in the brain include the hippocampus and the cerebellum, and we'll look at the retina and the brainstem in the next section. From such layered structures, we can often pick up electrical activity externally, because the currents tend to align.

Ions in neurons will tend to flow in the direction of electric fields. If there is a difference in potential between one part of a neuron and another, positively charged ions (sodium, potassium) will tend to migrate towards regions of negative potential, and vice versa. These flows create currents both inside and outside the cells, and these currents are what is measured when we place electrodes on the scalp for an EEG. Layered structures are very convenient because they tend to create measurable field potentials.

Scientists look for layered structures in the brain, because the currents line up. If all the currents are in different directions they will cancel out at long distances, and nothing will be measured on the scalp. However if they all line up then detectable scalp potentials are measurable, even though the signals diffuse considerably because of the conductive nature of brain tissue and the interference of the skull and meninges. Fortunately many of the important structures in the human brain are conveniently layered, including the cerebral and cerebellar cortex, the hippocampus, and sensorimotor structures like the retina and superior colliculus. The figure shows some typical patterns of current flow in layered brain structures.

In the cerebral cortex, there are typically six identifiable layers (as shown in the figure), and there are large pyramidal neurons that are oriented vertically through the layers. The apical (vertical) dendrites from these neurons are aligned and generate large externally measurable currents. Within the cortex, such pyramidal cells cluster in layers 5 and 2/3, so the current measured externally is the sum of these activities.

Here you can see a layered arrangement in the hippocampus, and its neighbor the entorhinal cortex. The hippocampus is considered allocortex and so has three layers, while the entorhinal cortex is neocortex and so has six layers.

Topographic Organization

Sensorimotor wiring patters are often topographic, which means the topography of the environmental interface is maintained through the neural pathways. For example many visual areas are "retinotopic". Topography means that every point in the input space can be identified by its coordinates, and that the output space aligns with the input space and preserves the topography. The network below is not topographic, because everything is connected to everything else, and therefore the topography is lost between layers.

However the network below is topographic, because the coordinate system of the input space is maintained through the layers.

The network above is a "convolutional neural network" (CNN). The topographic mapping doesn't always have to be exactly point-to-point, it can be convolutional, which can entail point-to-region or region-to-point. Magnifications and distortions may also occur. For example the mapping of the retina into the primary visual cortex V1 is topographically tight but changes shape, from the smoothly curved retina it becomes approximately complex logarithmic in the cortex, the foveal area is magnified considerably relative to its size on the retina.

When specifying the topographic wiring, two of the key principles are the divergence and convergence of the connections, which determine the extent of any convolutional filtering. These are sometimes called "fan-out" and "fan-in" respectively, after the digital circuitry. These concepts are diagrammed below.

The convergence and divergence of neuronal projections can sometimes be adequately abstracted as "projection fields" from individual neurons. This intuition was suggested earlier in the sum-of-Gaussians model for retinal receptive fields. Sometimes such an abstraction can be computationally useful, especially when the time courses of the individual components are difficult to specify. On the other hand, the time courses matter, and so in more detailed models it becomes important to describe them individually.

These concepts of convergence and divergence are vitally important when studying neural embeddings. We're almost ready to revisit this topic, but first we'll take a quick trip through a couple of example systems in the brain, to establish a comfort level and a model for additional study. But while we're on the topic, here is the overall architecture of a convolutional network from a machine learning standpoint. The left hand side of the diagram labeled "feature extraction" is where the progressively convergent convolutions occur, and the area on the right labeled "classification" is a different kind of network that gives us a one-hot code for the most likely object represented in the input. We'll talk more about machine learning later.

Modular Organization

In many brain areas the construction is "modular". This is true for example in just about the entire visual system. The modularity takes on different forms. In visual cortex V1, there are the well known orientation columns and ocular dominance columns, and also some color-related structures fondly known as "blobs" that become important in the connections to V2. While the modules may look a little different from one area to the next, the modular organization is ubiquitous in neural networks, across both brain areas and species.

The concept of cerebral modularity goes back to Vernon Mountcastle in the early 70's, who proposed the organization of the neocortex into columns, minicolumns, and hypercolums, on the basis of both anatomy and electrophysiology. At that time the work of Hubel and Wiesel had already been published, there were already existing examples of columnar organization.

One of the first clues to the modular cortical organization is that incoming axons from the thalamus branch considerably more widely than many cortical neurons. Pyramidal dendrites may extend for several hundred microns, whereas incoming axons may branch over several millimeters.

In spite of the complexity, a characteristic wiring pattern becomes clearly evident. The neurons in different layers have distinct branching patterns.

The pyramidal cells from layer 5 generally project subcortically, to the basal ganglia, to the superior colliculus, spinal cord, and so on. Whereas those in the upper layers project preferentially to other areas of cortex, both near and distant. In layer 6 are found cells that project back into the thalamus, so there is a loop from the thalamo-cortical relay cells that project to layer 4, through these layer 6 neurons and back to the thalamus.

Processing modules in the cortex can have varying sizes depending on their function. For example in the visual system (discussed in the next section), the "mini-columns" are very small, they're only about 50 microns wide and contain around 100 neurons. Whereas, a processing "column" is usually definable on the order of 500 microns, containing 50 to 100 mini-columns. Depending on the vocabulary used by an author, a "hypercolumn" is either equated with a column, or is a higher order collection of columns. In the visual system, a hypercolumn is considered a collection of columns, it's about 1-2 mm wide, which is much closer to the branching size of an incoming axon from the thalamus. The actual organization of a cortical area is shown in the figure, this particular one is from the vibrissal cortex, which contains large easily accessible columns that have been modeled computationally.

Cerebral modularity is evident at many levels. It is usually more pronounced in the sensory areas, and less so in the frontal lobes and upper motor areas, although the lower motor areas are definitely topographic in most regards. It is entirely unclear how the different levels of architecture work together to perform computations and to support brain waves. Many models exist, but they are mostly speculative. And there is the additional complication that as we move centrally, the "receptive fields" of cortical neurons become more complicated, and it is sometimes difficult to know what to look for. For example the discovery of grid cells in the entorhinal cortex was made by pure chance (more or less), and now it turns out that their relationships to place cells in the hippocampus is flexible - that relationship can be reprogrammed on the fly by bidirectional synapses in the hippocampus (Milstein et al 2021).

Cortical connectivity is flexible. The same basic architecture lends itself to multiple wiring patterns, in different regions of cortex.

In many cases the circuitry can be tied directly to computational requirements. In the figure below, the symbols in green are related to predictive coding and Bayesian inference, which we'll discuss in an upcoming section.

Oscillatory Activity

Modular organization raises some interesting questions. For example, what exactly constitutes an oscillator? You can create an oscillator with a single neuron, or a pair of neurons, or a small population. What defines the boundaries of an oscillator, and why are oscillations sometimes brain-wide, to the point where they can be picked up all over the EEG?

Oscillatory activity is detectable in just about every central structure in the brain, and many of the peripheral structures too. Historically it has been rather arbitarily divided into alpha, beta, gamma, theta, and delta components on the basis of frequency, for example the theta rhythm is usually cited as being in the 4-7 Hz range, although it is more specific in particular brain structures (for example in humans the frontal midline theta associated with intense focal concentration comes in around 6.5 Hz, but even that varies). Given the very few neurons needed to generate an oscillator, it is somewhat surprising that large areas of the brain can sometimes be identified as oscillating in synchrony, and that large anatomical areas are sometimes dedicated to populations of oscillators (like the medial septal nucleus, which drives theta in the hippocampus).

Generally speaking the population activity detectable on the scalp is an average of the neural activities inside the brain. There are other and sometimes more precise ways of looking at the generators of brain potentials and brain currents, for example the figure shows frontal midline theta generators detected with "synthetic aperture magnetometry" (Ishii et al 2014).

Oscillations occur at many different frequencies at once. One of the popular theories of brain function is the "dynamic assembly" theory, that posits cortical modules engage other modules with similar frequencies. The frequency content of a brain wave is frequently assessed through its power spectrum. There is considerable evidence of energy transfer between different frequencies in the power spectra of connected cerebral modules.

The prototypical model of brain "waves" is the Wilson-Cowan model, which we've discussed because of its illuminating behaviors in the phase plane. In its original form, this model was simply a collection of omniconnected excitatory and inhibitory neurons, and depending on the synaptic weights one could see damped ringing in response to an input, or network-wide oscillations that resemble brain waves. However a newer version of this model (Harris and Ermentrout, 2018) is topographic, and in the topographic version the oscillations don't have to involve the whole network (because it's not omni-connected), instead they may have a range of connectivity within which they operate, and the oscillators can be narrow (just a few neurons) or wide (many neurons) depending on input and synaptic strength. They can create traveling waves, as well as standing waves with the appropriate boundary conditions. Furthermore, the region that comprises an oscillator can shrink or grow, depending on the local dynamics.

This concept of "the boundary of an oscillator" is our introduction to "phase transitions". One can call the oscillating region a "phase" of activity, and here the usage is distinctly different from the angle associated with a sine wave. The usage here is more like "phase of matter", like ice-water-steam. When the network is in a down state, it's icy. When it's in an up state, it's liquid, and when it's oscillating and bursting it's steamy. At the boundary between a steamy place and a liquid place, is a "phase transition", an area which has very special properties. Strange things occur within the phase transitions, they acquire special dynamics and special computational properties.

The delay associated with the refractory period is sometimes enough to trigger oscillations in a network, as shown by yet another variation of the Wilson-Cowan model (Meijer and Coombes, 2013). In general, introducing delays into either the synapses or the neurons themselves will alter the network's dynamic behavior, and modifying these delays even slightly is sometimes sufficient to create noticeable differences in population behavior. In many parts of the brain, even though neural behavior is stochastic, the time constants and transmission times are kept within narrow ranges, and often these are optimized through evolution to be "just right" for important computational functions. One such example is the gain adaptation in the retina, which has to work with topographic mapping in the superior colliculus while it's still developing (when an infant first opens its eyes, when it's receiving its first visual input). The point being, that plasticity has the capability to dramatically alter the dynamic behavior of neural networks, which in turn affects their computational abilities.

The purpose of the Wilson-Cowan model is to demonstrate network dynamics, and for this reason it is not inherently plastic. But it can be made that way, with some simple modifications to the synapses. However the resulting behavior is somewhat uninteresting as long as the network is omni-connected, because in this case there is no way to distinguish the synapses with data. Nevertheless, when slightly modified, simple dynamic models can be combined with synaptic plasticity to yield intriguing behaviors, for example the figure below shows self-organized phase realignment due to spike timing dependent plasticity (STDP) in a network of Morris-Lecar neurons.

An important form of self-organization occurs in networks of coupled oscillators. Such networks are described by the Kuramoto model and its extensions, and we'll look at that more carefully on the next page. Fundamentally Kuramoto describes the behavior of oscillators of the same frequency but at different phases, much akin to pendulums of the same length coupled by a mutual base (a rod, or string). The model has been solved exactly for bimodal frequency distributions, and characterized approximately for multimodal distributions.

Another very important form of self-organized dynamics is self-organized criticality. We'll take a look at chaos and criticality in a moment, for now the heads-up is that there are dynamic systems that spontaneously evolve towards criticality. The study of this phenomenon is part of the larger consideration of how to program neural networks to adhere to certain kinds of desirable dynamics. Part of this has to do with designing filters at the single-neuron level, and part of it has to do with network wiring, and part of it has to do with adaptation and plasticity.

State Transitions

According to the dynamic assembly theory, cerebral modules (which we tentatively and loosely equate with columnar organization) go in and out of functional and computational assemblies. What causes a group of neurons to engage with another group of neurons? This becomes a considerably complicated question in an environment full of oscillations with a myriad of different ways to synchronize them, couple and uncouple them, and regulate their presence or absence and their character.

With a bit of study, we become interested not only in the states the network can support, but also in the transitions between states and how they begin and end. Related to the dynamic assembly theory is another theory called the "critical brain theory", which posits that much of the brain operates near criticality, in other words near one or more phase transitions.

We'll review the concepts of state transitions and criticality extensively in the upcoming pages. For now note that there are dynamic local and global "states" in the network, and these states are both controlled by the brain and control the brain. We've already seen an example of network states in the Hopfield machine, which is a thermodynamic model in which the number of possible states can be conveniently calculated. We need to spend some more time talking about the Hopfield model, not just because it's the Nobel prize winning foundation for most of modern AI, but also because of its variations, which over the past 40 years include complex dynamic behaviors like coexisting firing patterns, bursting oscillations, coexisting hidden attractors, multi-structure chaotic attractors, and network chimeras (Li et al 2024).

The basic idea of the Hopfield network is different from that of a traditional topographic network layer. A Hopfield network is an example of a "thermodynamic" model in that it relies on an energy function, which is repetitively minimized to find a final state. The network works by "gradient descent", like a ball rolling down the energy surface. As already mentioned, the secret to this network is that neural updates are asynchronous. Only one neuron at a time is allowed to be updated. Which neuron is chosen is important, network behaviors can be induced with particular selection algorithms. Generally the Monte Carlo method is used to ensure random selection (or selection from a desired distribution using the Metropolis-Hastings algorithm). Thermodynamic networks are very good at solving combinatorial optimization problems. The initial conditions are provided by the inputs, and the constraints are programmed into the synaptic matrix. The network simply runs until it finds an energy minimum. Sometimes it gets trapped in a local minimum, and in these cases a little bit of noise or a kick can cause it to resume its search, and there are also stochastic descent methods and other methods that tend to avoid local minima. Some of these networks are guaranteed to converge to a global minimum (depending on the energy function and the connectivity), but some aren't, and in some case the behavior can be adjusted by the judicious addition of additional neurons. The Hopfield network stores "memories" (bit patterns), not very many, but the separability is good, although it is possible to confuse the network by engineering certain kinds of inputs. There are also so-called "autonomous" Hopfield networks, which find applications in systems that don't require real-time learning. They are interesting from a theoretical standpoint because they can be used to study the stochastic dynamics that lead to chaotic behaviors and critical network states. There are many important variations on the Hopfield network, including the Boltzmann machine that uses a statistical mechanical formulation. We'll get into information theory on the next page.

Communication

The communication between neurons can be considerably different in various network states. The way a neuron responds to a synaptic input can be totally different when the population (or local dynamic) is in an up state or a down state. A single synapse can change its character depending on the postsynaptic membrane potential. These are highly complex and nonlinear relationships. Ultimately they are regulated by electrical and synaptic character of the neural environment (including specifically the glial environment).

The volume and the efficiency of communication between neurons depends on their individual properties as well as their wiring into the network. And, synaptic properties can directly affect the ability to communicate, for example STP plasticity can make communication more efficient. There is extensive evidence that neural communication as well as information storage can be both more voluminous and more efficient in critical or nearly critical states. Additionally, critical states usually entail long range coupling of the kind typically found between cerebral modules, and we'll review these concepts in considerable detail in the section on information theory.

Now it's time to investigate the issue of criticality more closely. To begin with, we can look at chaotic behaviors. In the Hopfield network just discussed, chaos can be induced by simply adding self-recurrent connections. The reasons for chaos are many, and it has to be carefully controlled to make it computationally useful. What is interesting is that chaotic regions tend to cluster, much like oscillators. There is useful behavior in the regions between clusters, which are often critical or super-critical in nature, have fractal geometric structure, and obey power law dynamics.