Sparse Distributed Representations (SDRs, Pentti Kanerva ) are indeed amongst the most powerful discoveries of neuroscience that best exploit the properties of statistics to both improve semantic representations as well as improve the process of inference (the process to compare any two or more inputs and identify their degree of association to each other). Rapid inference that is computationally efficient is one aspect of their advantages. But the ability to still identify close association of inputs despite high variability in their exact encoding, is perhaps the most important aspect of SDRs. This is because, this statistical property of sparsity gives SDRs a higher tolerance for small errors in our sensory input and also a higher tolerance for the high situational variability of our perception of reality. They enable a robust process for our recognition of the world that surrounds us, despite high perceptual variability.
If you are not yet familiar with SDRs, you should spend a few moments to understand them. The beauty of SDRs is both their simplicity and computational power. (See video links to Jeff Hawkins in the section above for a great introduction).
Sparse Distributed Representations Revisited
One of the areas that I am convinced poses the highest potential for both Machine Intelligence (MI) and Artificial General Intelligence (AGI), meaning non-specialized narrow AIs, instead, general and universal intelligence across areas of application, is the proper use of SDRs.
For this reason, I am proposing an expanded concept for SDRs which I am coining “Multi-Dimensional Semantic Space” (MDSS). Within this extended conceptual framework of MDSS, individual SDRs each possess an additional property of semantic direction, like a vector in n-dimensional space, also possesses. So any SDR existing within an MDSS, I am coining as “Sparse Multi-Dimensional Semantic Representation” (SMDSR).
These individual SMDSRs do not necessarily contain multiple dimensions, though this is not excluded. But it can be said, that an SMDSR does intersect at least one or more dimensions within the MDSS.
At this point, you may be wondering, what determines how many semantic dimensions does an MDSS (semantic space) need to have? And how does an MDSS get created? These are very important questions to explore and attempt to answer well, if we are ever to obtain AGI and understand the fundamental principles of cognition. I call these questions, the principle of Semantic Space Genesis (SSG).
If you are wondering, I have not yet identified any perfect SSG principle, nor have found any convincing results on this subject from other researchers. However, I do have some proposals and I also have the conviction that this is the area which holds the most promising prospects for future breakthroughs in both the science of neuro-cognition and machine intelligence.
If we would strictly adhere to the principle of seeking AI by reverse-engineering the human brain, we would probably need to wait for a very long time, before our wet-labs around the globe can collect such highly detailed information. Observing such a process of semantic space genesis (SSG) would require a very advanced knowledge about both the computational processes in the neocortex across hierarchical levels as well as, detailed observations of all developmental stages of the brain. While the hope of obtaining such insights from neuroscience is not a dead proposal, it does seem wise to proceed with the creation of hypothetical models for SSG that may someday be coroborated by neuroscience.