Learning vs Awesomeness by Analysis

A couple years ago I began (and stopped, now restarting) writing a system whose intended goal was to be able to perform novel deductions based on the natural language text it was given. I could give it English language rules such as (verbatim):

• a function is at an infinimum when its derivative is equal to zero
• Birds and mammals are animals
• Birds and platypus lay eggs
• pigeons and crows are birds
• mammals breastfeed their young
• Elephants, Horses and cars are big animals and chairs have legs
• Elephants, Horses, platypus and Mice are mammals

And ask: "This has legs"; "This lays eggs" to which it would answer with: "birds, crows, pigeons and platypus". If I were to instead ask: "This lays eggs and these breastfeed their young", then it would reply with "platypus". It could figure out things it was never told. This was to serve as a prototype for a more robust system able to learn less rigidly from text (although as was, it was already more flexible, language wise, than prolog). But before I can talk more about that, it's worth spending a little time on how modern AIs traded away the ability to think on their feet for the ability to learn.

Sometimes you'll hear someone, with supreme indignation, say something like: "But that's not intelligence..splutter..it's, it's just...brute force!". There are two ways to respond to that kind of comment. One is to acknowledge the possibility that the search was indeed exhaustive and the next is to point out that those sorts of problems, where exhaustive search actually works, are extremely rare. Thus, if the algorithm is arriving at a good enough solution with high probability, then it almost certainly wasn't just brute forcing anything.

Exhaustive Search

Should the fact that the system is quick enough to consider all the possibilities be held against it? Consider this scenario: there are two individuals, one can multiply two digit numbers in their head while the other resorts to memorized short cuts they'd been taught earlier. To most people, the fact that the first individual does not need to resort to any tricks to solve such problems is impressive. Their mind is so quick, they don't need to waste time learning tricks. But suddenly, if it's a computer that's the quick one, it's no longer impressive. It is this sort of double standard that Turing was against so long ago.

Brute Force

But focusing on exhaustive search is not productive anyways, since really, the number of interesting problems where brute forcing is a possibility are small. For these, if the system really were brute forcing, i.e. random search, then it would not be able to solve any non-trivial problems before the heat death of the universe. Various heuristics are applied in order to allow the algorithm to exploit structure in problems such that it can get to good enough answers most of the time. It is in that ability to exploit the search space that the, say Chess AI's, intelligence, lies. These systems are so quick, they don't need to learn anything because they can just figure it all out as they go along.

Learning

Learning is also search but at the end you end up with a product that should have achieved some compression of the data it was exposed to. Afterwards, it can quickly solve problems from the learned space efficiently. Kind of similar to how decompression is much faster than compression. Learning and pattern recognition is preferable to figuring out things as you go along exactly because it is so much more efficient. You don't have to keep on deducing things from scratch.

Awesomeness by Analysis

So you can think of it this way, in modern machine learning and in humans (mostly), intelligence is essentially backwards looking; you learn a pattern, achieving a compression of the data then matching instances to patterns quickly as you ran into problems. On the other hand, a lot of so called "brute force" methods are forwards looking, they do not bother with compressing patterns, preferring instead, to search out solutions (not so different from the learning process and ultimately) figuring things as they go along. The Chess AI is not just dumb brute force, it intelligently traverses the space in order to make a seemingly intractable problem solvable. All without spending any time learning from experience; instead preferring to work off a very simple theory of Chess. The TV tropes page on Awesomeness by Analysis precisely describes these sort of AIs:

Some people learn by flipping pages. Some people must gain knowledge through pain. Some people study by television. And then there are those who just have to observe [think]...

Need to make a million-to-one shot to stop the Doomsday Device from exploding the world, but have never even fired a gun? Just run off some mental calculations about your gun's firing speed, friction, gravity and the slightly off-kilter scope...

As an example, it takes a lot of training and experience for a human stunt driver to pull off some of the amazing parking tricks they do. On the other hand, we can imagine a supreme deducer who--utilizing only knowledge of gravity, it's own plus the car's mass, friction, aerodynamics, the material make up of the gears and engines etc--deduces what sequence of actions and timings are required to perform such tricks after only seeing it done once.

Limits to Deduction

But if deduction is so awesome, why did Good Old Fashioned AI fail so spectacularly? There are a number of reasons for this. One of these is that input was brittle, requiring not just experts to provide the core theory but also, someone trained in a logic programming to input. This simply did not scale and so was quickly deemed untenable. Modern supervised learners (including deep learning) also need reams and reams of data, but the upside is that one does not need to be an expert in formal logics as well as in some esoteric subject to prepare the data.

This matters a lot when you consider something like vision, not only is it much more complex than something like Chess, but also, how it's done is not consciously available to us so there are essentially no experts in vision. Further on that, trying to write down as a human, a formal theory of vision, was a no go (computers can finally do this today).

Another and actually, the key limitation is that deducing everything from some base theory is (and will always be) extremely inefficient--especially for machines of the 1980s!

Deep vs Shallow Reasoners

In the following, I use deep reasoners not in the sense of the architecture of certain projective functions but instead to talk about the ability to perform long chains of deduction vs. abstraction and learning. Everything but GOFAI, (even neural nets and humans) are shallow reasoners.

You, as a human trying to solve a problem in linear algebra or topolgy, do not have to deduce everything from the axioms of ZF-set theory. However, having learnt say algebra, you can perform only shallow deductions but still get to deep artifacts off the back of all the abstractions you're leveraging. This very productive middle ground (actually it's more like not to so extremely induction only as Machine Learners) is what we humans occupy.

Humans do Both Deductive and Inductive (actually abductive) learning

Perhaps you have heard of Google's Deepmind's new game playing robot? Although it was impressively able to learn how to play some Atari games based only on Reinforcement learning signals, it could only reason about situations it had run into before.

(as an aside they also talked about a trick the system was able to employ after playing breakout for some time. this simple scenario is also an instructive one to look at as a way to compare evolution to learning. we can just as well say that instead of learning over time, and other system through natural selection across the better plays evolved such a strategy.)

Supervised, unsupervised, transfer, reinforcement learning and deduction.