Effectively Simulating a Neuromorphic System

As the system stands currently, each neuron is located in a large array, referenced by its index. Similarly, all the synapses occupy their own spot in an array. These two structures alone are nothing surprising, as each of these objects needs to exist in memory in some form. We can then reference neurons via their index, with each neuron also storing the indices of its synapses.

The unique part of this system is an event queue to store spikes as they occur. Each network has some upper bound of delay (associated with each synapse) that must be representable within the simulation. You can think of this as the longest possible time in the future that an event may need to fire. As these networks are run continuously the time steps will eventually wrap back around, reusing space as the simulation moves along.

Each index in this array has the capacity to store a fixed number of events, which again comes from the characteristics of the network. This means that at any given time we need to have the capacity for every neuron to fire, and propagate a spike to their downstream neurons.

One of the major benefits of a neuromorphic system is that it operates on events, and subsequently produces more events. An event-based simulation lends itself to a lazy approach, where only the “active” parts of a network require computation. A common implementation uses an event queue (or priority queue), where individual spike propagations are enqueued, which are then applied to the network at a later time. This approach allows for sparse networks to save on simulation cost when little activity is present.

Due to the delay of any given synapse having some fixed upper bound in networks, there is no need to maintain a dynamic priority queue, instead each discrete time step can have its own pool of events. By providing a fixed location for events to be stored we can effectively eliminate the need for dynamic memory allocation, while maintaining the same advantages of a priority queue. The issue now comes when considering the memory footprint of such a solution.

Each possible time step needs to have the capacity to hold that maximum number of events possible in the network. Therefore, even though this solution is optimized to perform the smallest amount of computation under low network activity conditions, it always consumes more memory.

Instead of storing sparse events in a global container, we can instead store the “upcoming” state of each neuron internally. Our approach takes the form of a ring buffer within each neuron to store the effective charge at each offset from the current time step. Under this approach each time a neuron fires it propagates a spike down each of its synapses, adding its weight at some offset in the downstream neurons ring buffer. This ring buffer is then reused in much the same way as the event queue described previously.

This has the benefit of integrating events into their final sum at the time of firing, reducing the number of tracked states drastically, thus reducing memory used. On processors or platforms with limited cache, reducing the overall memory footprint of a simulator can have large benefits on the simulation speed. This is especially important on microcontrollers where limited or no cache is available to speed up execution.

Another benefit of this approach is the improved spatial locality. Instead of interacting with a large global container storing all activity, we are directly writing/reading from a single neuron at a time. Large arrays are likely to not be entirely resident in cache, meaning we will take the penalty of a cache miss more frequently as we jump around to different time steps. In contrast, the embedded ring buffer approach not only lowers the overall size in memory, but it also reduces the number of possible memory location we can write to.

However, due to the shift in how our network is represented in memory our computation will also need to change. We can no longer act on only active neurons, which is a tradeoff we must make in order to support densely connected networks with high activity. Now we iterate through each neuron, checking for to see if it should fire, continuing on if not. If the neuron should fire we begin iterating through each of its synapses to apply a charge to any downstream neurons. With this computation model, leak can be applied within the same loop as charges are propagated, further increases time savings.

Again, microcontrollers are less sophisticated and thus have much shallower pipelines, leading to less overall cost to branching. Therefore, we can pay the cost of checking if each neuron should fire and still achieve a speedup over the event-based simulation due to the much lower pressure placed on memory.

In an effort to further reduce our program’s memory footprint, we can also eliminate the global array of synapses. To allow our system to support a reasonable number of neurons/synapses they are referenced with a 2 byte index (65536 possible indexes). The issue is that a synapse itself is only 4 bytes in total, therefore a 2 byte index is a 50% overhead.

Each synapse represents a relationship between exactly 2 neurons (or a recurrent connection between a neuron and itself), and they are never shared, meaning that a synapse can be embedded within the upstream neuron. This change eliminates the need for the 2 byte index, and further improves spatial locality by bundling neurons with their associated synapses.

It should be noted that this approach requires each neuron to have the capacity for the same number of synapses. This would likely be a downside for a sparsely connected network, but it proves to be beneficial in networks with dense connectivity.

Thus far, we have focused on optimizations for a lower power, potentially single-core processor, yet our system is trying to simulate parallel operation. If hardware is present that enables parallel processing the current model would not scale well to many units of execution. This is due to the fact that we would have contention around “firing” events. If we imagine each neuron executing independently, two separate neurons may fire, resulting in two spikes propagating to a downstream neuron at the same time. This is the classical i = i+1 problem in computer science.

Instead, if we think about this problem from the perspective of the downstream neuron, we all but eliminate this issue. Instead of having each neuron propagate a charge downstream, we can have neurons look at their upstream neurons to determine if they fired. This is a bit of backwards way to think about things, but it enables potentially massive parallelism.

There are some potential downsides to this parallel approach. First, the system now requires 2 full iterations over all neurons, one to collect charges from all neurons which have fired on this time step, and a second to apply leak to the network. Second, our memory access pattern is much more unpredictable, however, with sufficient parallelism this benefits will outweigh the costs.