AI development should be done by neuroscientists

I disagree with this idea on so many levels.

Firstly, there is a big misconception that because neural networks were inspired by the model of the brain, they must also behave like a brain, which is simply not the case. In NNs, features are fed forward through nodes which collect a weighted sum of all inputs and output based on how the sum compares to some threshold. For training the model, the result is compared to the actual value and the weights are corrected according to gradient descent which propagates backward through the network. There is no evidence that neurons apply the same sum and output behavior that NNs follow and last I heard (I'm trying to find the source again) brains almost certainly don't use back-propagation to "train" themselves. The biological complexity of a brain also adds several layers of abstraction on what NNs refer to as "weighting" connections and even how connections get made or destroyed. Just because some high level features of the two are similar, does not mean the act or can be used in the same way.

Furthermore, using brains as the basis for future AI developments is extremely limiting. While evolution is good at optimizing over billions of years, it's unreasonable to assume that a brain is the pinnacle of learning capabilities in either physical or computational models. We shouldn't be trying to copy nature explicitly by rather we should be trying to realize novel ways of modeling and rationalizing about the world. To use neuroscience as a basis for everything about learning is a handicap.

Also, the brain is still very much a mystery. There have been many advancements recently, but we still know more about what the brain doesn't do than what it does do or how it fundamentally works. There's little advancements that neuroscientists could bring to the table in terms of modeling an actual brain. It's also very difficult to translate all the complexities of a biological organ into a simple machine; the correct level of abstraction simply doesn't exist.

(First, my thoughts on that paper)

I realize they were just applying techniques naïvely, but that paper really should have been 10 times as long. It felt like in a lot of cases they were stopping short of where there would be obvious next steps to search for more clues. It would be really interesting to see how these techniques could be combined and iterated to try to find more patterns or ways of modeling the system.

But also, huge props to the authors of that paper. It takes a lot of courage to admit that your field of study is falling short of what it intends to do.

(Okay, now back on topic)

This actually made me realize a completely new reason I wouldn't want a neuroscientist to do AI development (or at least not ML).

A neruoscientist is, just that, a scientist. A scientist seeks to understand some system by a repetitive hypothesis-experiment cycle. This is very good for trying to understand an existing system, but when you need to design a new system, it may not provide enough insights into how to modify the system to produce better results. In other words, the post-experiment analysis will provide several new hypotheses to test instead of narrowing the range of possible hypothesis.

The hypothesis-experiment cycle can still work, though. For small or relatively simple systems, it's proven to be quite effective. For example, the top rated chess engine, Stockfish, still uses Alpha-Beta Tree Search with a manually tuned evaluation function.

But with modern AI development (that is, the Machine Learning kind), we're rarely interested in simple or well-defined systems. Instead, we want a program which is capable of making decision in high-dimensional space quickly and which can accept and interpret new kinds of information readily.

These kinds of problems aren't well suited for science, but they do seem to fit right in with engineering.