Traditional neural simulators merge connectionist geometry with cable models to arrive at very realistic looking spike trains. We saw some of the popular neuron abstractions a few pages ago. There are many excellent simulators to choose from. NEURON is an industry standard, Brian2 is a popular choice, Nest works great with Docker, and Nengo is a fascinating variant that's definitely worth investigating.

If you're just playing around and you want to see some neurons in action, start simple with something like Nengo, where everything has defaults and you can create your network with a single command. To do anything useful with NEURON you'll need to specify your h and m variables, which requires you to have an understanding of the differential equations, and in Brian2 you actually have to write the equations. The workflow of neural modeling involves "placing ion channels on neurons", and another way of saying it is a marriage between morphology and electrophysiology.

If you're serious about neural modeling and you don't mind differential equations, a heads-up is the new end-to-end system from the Open Brain Institute. It does exactly what we just said, marries morphology with electrophysiology, and then sends the results to the Neuron simulator, all in the cloud 100% in real time. This system is the first of its kind, it's the first one to fully support a modeling workflow end to end, it's like a little virtual laboratory and on that basis alone it gets a huge thumbs up. Many people have had this vision, but these guys are the first to implement it, and it works, and it scales, and if you're running big simulations all you pay is the cloud costs. This is a superb choice if you're a doc or post-doc needing to get some hands-on experience with Neuron, you can bypass all the installation garbage and cut straight to the chase. I can't say enough good things about this, we need these kinds of tools. Kudos to the people who created this.

Historically, Warren McCulloch and Walter Pitts were the first to implement a working neural model. Neither were biologists, both were computer scientists and McCulloch worked in psychiatry. At the time the McCulloch-Pitts neuron was created, Norbert Wiener was at MIT inventing the field of cybernetics. McCulloch and Pitts were logicians interested in the possibilities of threshold logic. What they came up with, was incredibly simple - a binary neuron that sums its inputs and decides whether to change state. Amazingly enough, John Hopfield used this same formulation 40 years later to win a Nobel Prize. The difference is, that Hopfield's network uses a Hamiltonian and feeds back to itself, whereas the first simulations were strictly feed-forward layers.

Frank Rosenblatt took this concept to the next level by building the Perceptron, which was essentially a simulation in hardware - what today might be called a "neuromorphic device". The Perceptron is still the foundation of all of modern AI. The new thing about the Perceptron was that it learns. It has a learning rule, called the "Perceptron learning rule". What it learns specifically, is to partition a dataset by linear discrimination. It will find the line that best partitions the dataset, according to a supervised signal provided by the instructor (which is usually a human). It does so by "curve fitting", that is to say, back-propagating errors through the network to the synapses. Variations exist using least-squares and different kinds of fitting algorithms. Basically the machine just iterates until it converges, and then the solution is read out from the synaptic weights. Many (most) of the early neural network simulators were exactly like the Perceptron. Some of them introduced new ideas, but most of them were just variations on the integration function or threshold function.

When Wilfrid Rall introduced the cable model in the early 60's, neuroscientists had to take a step back and inhale slowly. Because this was actual physics, it was no longer just algebra. Rall's model provided for the first time a direct link between observable electrical activity and the underlying physics - a link that's computationally accessible and even tractable on small geometries. (At the time there was no such thing as a PC, and it was before the time of the small personal minicomputer like the Data General Nova or DEC PDP-11). Cybernetics was an esoteric field at the time, there were only a handful of people in the whole world who understood it. But when Rall's model came out, cybernetics went public. It took a while before the results started appearing in print, and a lot of it can be traced to the availability of the IBM-360 computer, which became a staple at educational institutions. One has to understand that the IBM-360 had a whopping 8 megabytes of memory. That was the biggest one - memory started at 4 kb, and it was very expensive because it was core memory, each slot had a tiny magnet at the intersection of two wires.

Subsequently, neural network modeling became an academic undertaking, largely a curiosity because there were no specific applications for it. A few people had visions for robotics and artificial networks, but the technology wasn't ready yet and the only way these people could test their ideas was with math or in software on a tiny scale. In the mid-70's two Japanese visionaries set the field on a new footing. One of them is Kunihiko Fukushima, who was working at the NHK laboratories at the time. He created the first convolutional network and demonstrated its applicability for rotation and translation invariant pattern recognition. The second key player was Shun-Ichi Amari, the godfather of information geometry. But at the time, he hadn't invented that yet - at the time, his work was in the creation of the first mathematical model for statistical neurodynamics. Dr Amari was the first to expand the Wilson-Cowan population model stochastically, and he used it to prove the existence of "catastrophes" in the neural dynamics, which today we would simply call attractors. This was an enormous advance because it directly linked the EEG with information processing.

Then came the PC, and the internet, and AI, and suddenly there was an explosion of interest in neural networks. In all fairness, machine learning has contributed tremendously to our understanding of neuroscience, but at the same time, it's not neuroscience, it's something different. However the people doing that work, encounter a lot of the same issues as neuroscientists doing simulations. For one thing, they have to store their results somewhere, the simulations create huge amounts of output and it has to be processed somehow, it has to be turned into something someone can look at and say "ah-ha!". So now we have to add another essential job skill to the list: computer graphics.

Anyone can look at a neuron, it's not hard. Just pull down one of the SWC tracings from neuromorpho.org and import it into Blender. (Blender is free software, you can download it from the internet, it's been around and field-tested for 20 years, and it's expandable with Python scripts - it's still an industry standard and everyone eventually ends up with a copy of it on their computer). But you've seen the new AI-enhanced images of neural networks that were created from serially sectioned electron micrographs, they're very pretty! The visualization goes in two different directions - one way, we're interested in morphology and connectomics and we like to look at pretty pictures of network, and the other way, we're interested in synapses and astrocyte leaflets and we'll probably want to look at those on a mesh. A mesh is a computational object, as much as it is a graphical object - but the geometry is indeed very graphical, we get vertices, edges, and faces, and we require certain things from these elements to get our computations to work.

The visualization of brains on a large scale began in the laboratory of Dr. Robert Livingston at UCSD, when he helped produce the Nova film called "The Brain". I had the unmitigated honor of working in his lab, I was just an undergraduate at the time and couldn't even spell Unix, but it was the first time I saw a mesh. I was very impressed! We used an Evans and Sutherland Picture System to visualize the graphics. Every micrograph section was traced by hand on a Talos tablet (the computer system you see in the video is the exact system I worked on). The purpose was to visualize the macroscopic structure of the brain, but today, we can do that at any level of resolution. We don't have to trace anymore, the AI will do that for us. All we have to do is tell it what kind of mesh to create. The difficulty is, that graphics is computationally intensive. If you're trying to visualize results at the same time simulations are running, they'll conflict and everything will slow down. Graphics in a client-server or cloud environment involves the transmission of a lot of information across the network. You can see this effect in an otherwise blazingly fast cloud application, when you're trying to rotate a mesh on the screen. The frames stutter, and they arrive slowly, and that's because the internet is highly variable and you're competing with others for bandwidth. So, many scientists prefer to work at home, as it were, on their own computers without being connected to the network. There's a lot to be said for that, and there's also a big down side. Ideally, we could freely move things back and forth, but that's still a tall order. Some limited amount of interoperability exists, but it's nowhere near an "ecosystem".

An "ecosystem" implies that the parts work together symbiotically. And that, unfortunately, is not the case for most end-to-end workflows in neuroscience. A software "ecosystem" implies that one can seamlessly move data from one toolset to another, for whatever scientific purpose happens to present itself that day. The reality is more like, moving data between systems is pushing a rock up a hill, the minute you get something working you discover there's something it doesn't work on. My favorite example is the simple concept of placing organelles inside a cell. How does one do that? One can use a CAD/CAM program to locate objects in a scene, but then one has to arrange them hierarchically and name each organelle, and one can then export the result in a common format like OBJ, only to discover that the vast majority of programs in the so-called "ecosystem" won't read the file correctly, some will even try to turn separate objects into one big gigantic mesh.

One discovers also, that there are tools from completely different fields of research, that can be very helpful for neuroscientists. An example is machine learning tools like PyTorch and TensorFlow. Everyone engaged in neural network modeling should understand these tools, because even though they lack certain functionalities, they're very strong in others. And it goes without saying that some knowledge of LLM's is helpful too. The scale of brains gets overwhelming, it's too hard to do things by hand. Today's technology is so good it generates mountains of data, and sometimes we have to post-process it in various ways, we may even have to become data scientists (so add that to the list of useful job skills). Pandas is an essential tool for neuroscientists, it allows the quick and easy (and powerful) manipulation of data frames interactively in Python.

Is TensorFlow a neural network simulator? No, not really. It's machine learning software, its purpose is mainly commercial. What are its shortcomings for neuroscience? Well, it doesn't generate brain waves, for one thing. It's an algebraic tool, it's not really for dynamics. One "can" create some dynamics with it, but they're tempered by the learning algorithms that require back-and-forth sweeps of the network. There are other issues with this type of software. In the original Hopfield network the nodes were updated one at a time, using a Monte Carlo method. In the interest of computation time, some of these tools alter the math underlying the network, instead of one at a time it becomes four at a time, and then eight. Pretty soon people are doing synchronous updates and they're wondering why they're getting spurious oscillations in Neuron that don't show up in TensorFlow. It is definitely a good idea to try one's model on different tool sets in different environments, and it's also a good idea to be aware of the reasons for the differences in behavior.

One of the interesting issues with traditional simulators, is they seem to have a very hard time with connection maps, and even solid body geometry. The go-to starting point for many simulations is the layer or "sheet" of neurons, and it usually begins life as a square or rectangular grid. (It's rare to see any other arrangement, once in a great while one sees a spherical layout but mostly they're lattices). In a topographic point to point connection such as the one between the retina and the thalamus, we'd like the retinal axons to terminate in small clusters with characteristic synaptic arrangements. For connectionist models, the exact layout of the synapses matters, so we have to specify shapes in terms of densities. It so happens a Gaussian arrangement works pretty well in this network, but it wouldn't work in the cerebellum with Purkinje cells that are laid out like a crossbar. The connection map tells the presynaptic layer which neurons to connect to in the postsynaptic layer. In a real brain this function would be carried out by axonal growth cones in combination with chemical markers, so the connection map is a pretty significant abstraction, and it only really works in a strictly connectionist context.

Now let's consider this same connection, in the context of volume conduction. We have a retina, and we have a thalamus, and we're going to put them inside a sphere, and arrange them so they look somewhat like a real brain, that is to say, they have realistic geometry. To do this, we note that the retina is not a flat sheet of neurons, it's actually curved. And so is the lateral geniculate nucleus. So we have one geometric transformation when the axons leave the retina, and another when they enter the thalamus. (We're ignoring the optic chiasm in this example). The thing is, the axis of the retina is not aligned with the axis of the lateral geniculate nucleus. The LGN is "bent" (which is where it gets its name), and if we're really trying to be realistic about the geometry we have to position it properly and introduce the axons in a realistic way - that is to say, at an angle and so they can contact multiple layers at the same topographic point. Connectionists wouldn't care about geometric details like this, but anyone interested in volume conduction certainly would. Without going into a long litany about the interaction between the intralaminar thalamus and the thalamic reticular nucleus, suffice it to say that the orientation of the LGN matters. The LGN helps generate alpha waves that perfuse the visual input. These waves are detectable on the scalp because of the synchronous activity of layer 5 pyramidal cells in the cortex which are driven by the thalamus. If we reorient either of these structures we'll get a different interaction between the fields they generate, which in turn will impact the observed synchrony. We can measure how much that matters, by testing small increments of rotation. We can repeat the experiment discussed on the previous page, in a simulation, which will show clearly and unequivocally that rotation matters. (We already knew this, all the way back in 1979, but now we know it again).

Let's say we have an existing Neuron model of a brain structure, let's say the hippocampus because there are many such models. Now let's say we want to take that model to the next level, by introducing astrocytes. And, since we know a bit about astrocytes, we're going to have to account for the effects of fields on extracellular calcium, which Neuron doesn't really do. What are our options? Well, at this point we can either start from scratch on a different simulator, or we can try to "port" the Neuron code to save ourselves some time. There is software that translates Neuron code to other languages, and there is software that translates other languages to Neuron. How much this software will work, depends on how many special features are in the model. If you're starting from a connection algebra in Brian2, no amount of tweaking is going to get that into Neuron automagically. In that case your best bet is to translate all those weird symbols into a connection map, and then translate that into Neuron-speak.

For neural networks, there are some intermediate file formats that claim to be common, like NeuroML, the trouble is not much of the other software uses it. And even though it's editable by humans, XML is a horrible language, it's hard to read and harder to repair. Another alternative is JSON, which is slightly more readable and is also editable by humans. These formats lay out neurons and connections, but they don't describe internal or external geometries, only the boundaries. One of the big issues with translating simulations is the coordinate systems. In some software, Z is up and Y is forward, whereas in other software it's the other way around. So the translator has to be aware of which geometries are in play. Generally though, the more advanced scene-based representations will carry some metadata along with the component information. This may include for example, a description of the scene in ascii followed by a long list of vertices for each constituent mesh. XML endeavors to keep all the information in one big file, and so does hdf5, whereas glTF has the ability to separate the meshes from the metadata. In practice this affects our ability to extract information. For instance let's say someone created a big hdf5 file a few years ago, containing all the evidence for their entire workflow, including the tracings, the meshings, the simulations, and the visualizations. Now here we are several years later, and we've identified one specific mesh that we're interested in, and we'd like to extract that from the dataset. How do we do that? Do we have to download the entire dataset? And once we have it, how do we extract the mesh? And once we have that, is it going to be in the form we need? There's no answer. These interoperability issues can be very difficult. Reality is, you just have to try it and find out. If you encounter something that doesn't work, scream loudly and hope someone hears you.

At the end of the day, what do we have in a traditional simulation? Well, if we've done our job well, Neuron will generate both spike trains and extracellular field potentials. The one thing it can't do, is reflect those fields back onto the spike trains. The abstraction needed to do that is so intense that the whole model becomes unrealistic and unusable. Traditional simulators are fundamentally R-C models, which means you could do the same thing in SPICE. In the next section we'll look a more sophisticated and realistic method, that lets us apply fields to the compartments of a cell. Why do we care about this? The Hodgkin-Huxley equation isn't good enough? Well, frankly... no. It's not good enough. It doesn't account for fields, at all. It's an electrochemical model, it came out of the theory of electrolytes. If we want to look at fields, we need Maxwell's equations. Finite elements are the minimal model that will handle fields, without making deal-breaking assumptions. Isotropy and homogeneity are not required for finite elements (although many times, thankfully, they exist anyway).