Case-based reasoning as main technique for implementing AI?

In the hope to improve “Learning from demonstration”, I found a new term in Artificial Intelligence: Case based reasoning. As far as i understand from the documentation, the idea is, that an human expert demonstrates an task and from that demonstration a database is created. If the robot must solve the task for its own he request the database. So LfD and CBR have much in common. The difference is, that LfD lets the retrieval process open while CBR is the more general approach.

But let us simplify the general idea. At first, we are programming a system in which RC-car2 must follow RC-car1. That means, car 1 is controlled by a human-operator, and car2 works like a linefollower robot, which means it is a simple tracking controller.

In the second step, the system is modified a bit. Both cars are controlled by humans. That means, car1 is driven by a human expert, and car2 too. The second car has the same goal like before, he should follow the other car. The question is now, how to write the tracking controller for car2? And here comes case-based reasoning into the game. The idea is to store the gameplay into a .csv file. That means, we get a list of positions of car1 and car2, so it is visible who the human expert2 is reacting to car1. This .csv file must converted into a case database. This is used for generate the tracking controller.


How big is the Artificial Intelligence market?

Surprisingly small. According to different sources around 10 billion US$ per year worldwide are invested into so called AI-startups. That are companies how are researching Artificial Intelligence and robotics. If the salary of an employee in the sector is around 50000 US$ per year, than the spending is equal to keep 200k people running. Most of the employees are working in the US, but some of them in China and Europe. A deeper look into the list of AI-startup companies shows us, that most of them are classical tech-companies which are programming software and call themself AI-company because it sounds great. Under the assumption that the market grows in future it remains a minority of employees who are involved in Artificial Intelligence research.

Limits of neural networks

Sometimes, neural networks and deeplearning is recognized as voodoo magic. In the following blog post I want to bring a bit clearness into the game. At first it is important to ask what is nessary to tranform a neural network into a truing-capable machine. The simplest form of implementing a computer is not a turing-machine itself, but a more basic structure, called logicgates. Logicgates are used in a truthtable and can only have AND, OR, NOT operation. The interesting aspect of logicgates is, that the can do everything what a turing machine can do.

On Youtube there is a video available which shows a primenumber generator build from logicgates only. I would guess, that the diagram was created with a compiler from a highlevel C-program. Most important, the shown logicgate diagram is very huge. There at least 40 logicgates visible, perhaps more.

The reason why I’m explaining the logicgate is because logicgates can be trained like a neural network. They can be seen as a real neural turing machine. Not as an external tape, which is used together with a neural network, and not as an LSTM network, but as a truing-capable computer. So what is necessary to train a logicgate network? I have absolutely no idea but i posted it as a question to stackoverflow

I would guess, that it is simply not possible to train a logicgate network. In theory perhaps, but in reality the state space is too huge. So we have a truing-capable neural network but no idea how to adjust the weight. And this gives us the answer what the limits of neural networks are. A normal 3 layer neural network is not truing-capapable. Larger deeplearning networks which are based on LSTM neurons and deepmind NTM are perhaps turing-ready, but they are not more powerful than a logicgate network. The problem is, that for normal neural networks, LSTM machines, logicgates or whetever kind of apparatus no efficient learning algorithm is known. That is the real bottleneck.

Learning algorithm like the delta rule, backpropagation or quickprop as not very elegant form of searching in the error diagram for a minimum. They are not able to find a minimal solution. And they won’t even find a simple weight combination for calculating prime numbers. Using a neural network as a turing-machine is a deadend.

But i wouldn’t call the deeplearning movement in general a fail. Only that part of the community who is trying to move neural networks in the direction of a computing device is a failed-project. Even with the fastest nvidia cards this is not possible. Another aspect of deeplearning is in contrast very attractive. That is called big data and means to use neural networks for storing images.

What is the difference? A neural network can be seen twofold: at first as a turing-like device with the aim to search for a program for converting input into output. That is equal to a neural turing machine or to a logicgate network. The other option is to see a neural network as a propabalistic database. Here is the aim to store for example, 10 gb of images and than search the database for similarity. That is a technology that works. It means, that it is possible to use it practical.

I think the deeplearning community should resign from the idea, that their 20 layer network is some kind of trainable computer. For realizing even a simple primenumber algoirthm a way more neurons (logicgates are needed) and training such computer is not possible with current hardware. But what makes sense is to see a neural network as similarity search algorithm for retrieving images. The simplest form is to store 1 Mio dog photos as compressed jpeg file on harddrive, and use a convolutional filter for searching if a given image is similar to one of the photos. If yes, we can label the image with the name “dog”.

This special topic is currently not very well researched and it makes sense to investigate it deeper. It think we should use other vocabulary to make clear what we are doing. Instead of talking about “training a neural network” the aim is to build a similarity search algorithm which has access to an image database.

Neural turing machines

In some papers the so called “Neural turing machine” is presented. Often, these papers are very complicated and contains much mathematical formulas. In reality, a neural turing machine is simply a device which has logicgates and the exact configuration is driven by a learning algorithm. This kind of neural network is called “McCulloch Pitts neuron” and it is truing-capable. It is a trainable turing-machine. But it can’t be used for any purpose. Because it is unclear how to train the logicgates, that means to decide if the gate #23 is an AND gate and if yes which other two neurons are the input signal.

The reason why today no complex logicgates are used in computers, but instead the von-neuman-architecture has to do because on a von-neuman-machine a program can be executed on a tape. And creating such program can be done with higher language. Such programs can be also converted into logicgates, but that is equal to build a different computer for a different algorithm, which is not very useful.

How deeplearning works
Deeplearning has nothing to do with neural networks. Instead the algorithm can be described as a similarity search with Convolutional Neural Networks. That means, the CNN is used as a metric to determine if two images are equal. And the input image is compared with a database of known images. The accuracy is higher, if the database is bigger.

The misunderstanding is, that most tutorials suggests that a neural networks works like a computer, and after training a certain program is found. But in reality, there is a huge difference between a turing-machine and a neural network. Instead it makes sense, to call deeplearning a sort of filter-generation for determine if two images are equal.

Perhaps a small example who OCR works in reality. At first, we need a database of .svg files. The filesize should be 10 gigabyte or more. In that database every kind of characters are stored from every possible font. And now we take a new .svg file. We are searching inside the database for a similar image. The similarity index is calculated with a Convolutional Neural Network, the request goes very fast. If we found an image, we know that on the picture is the character “w”, for example. With deeplearning or neural networks this has nothing to do. Also not with a logicgate or with a neural turing machine. Instead the accuracy depends on two factors:

– size of the database
– similarity filter

It is false to store image data into a neural network. The images can be stored in a normal database. And it also false to search for an algorithm. Instead a given algorithm is used to generate the image filter and this results into the optical character recognition.

Monitoring AI at stackoverflow again

A while ago I made a small list about the activities on Stackoverflow which have the tag “Artificial intelligence” in mind: I recently watched again on the topic and like in the past the situation is not very good. The number of questions is small and most questions get their first downvote before the punctuation point was made … But let us investigate the traffic of the last days:

Question 1: -4

Question 2: -4

Question 3: -1

Question 5: -1

Question 7: -4 (put on hold)

Question 10: -3 (put on hold)

Question 11: -5 (put on hold)

Question 12: -1 (put on hold)

Question 13: -3

Question 14: -7 (put on hold)

Question 15: -1

Question 16: -3

What goes on the stackoverflow forum? At first, I do not think that it is a problem of the moderators, because the questions are really bad and before the admin puts a question on hold, many other users have pressed the downvote button to express that the question itself is not well enough. And if we are counting the numbers of how often the downvote button was pressed it is more the result of collective activities, which means the overall stackoverflow community comes to the conclusion that they don’t like the AI tag very much.

So what is best-practice in answering AI questions? At first, it is important to give the question in every case an upvote, even if the OP shows a lack of understanding of the subject. And secondly it is important to increase the overall traffic, that means for posting more questions and more answers, because this results into a more efficient community.

Why are pressing so much user the downvote-button? According to the software, every user can press only one time the button. So all the 35 votes in the above list come from different users. Their understanding is, that the Stackoverflow community is an evaluation platform for judge about content. In a sense, that somebody is writing a paragraph and another user is reviewing the content and makes an evaluation about it. In reality, the vote button has another idea in mind. The idea is, that the upvote button supports a question. It means, that if somebody also wants to know how the Tensorflow algorithm works, he should press the upvote button, because this increases the priority of the question.

Evaluating the content and judging about postings is done outside of Stackoverflow, for example in an academic paper or in a hardcore robotics wiki, in which topics are criticized and papers can be called as not helpful. Instead, the Stackoverflow community is a place of divergent thinking, that means the aim is to increase the traffic and encourage people to post their ideas.

Artificial Intelligence is a non-topic

Let us investigate the most famous Tech-website in the universe: Stackoverflow. According to Alexa, the website rank in the U.S. is #41, and according to the stats yesterday, they have 9.9 million visits per day. That is absolutely amazing, because my own blog has 80 visits/day. It seems, that the stackoverflow website is the most important tech-forum, gains a high traffic and is the number one place for programmers. But suppose, we are not interested in C++, NTFS or installing Javascript in C#, is SO also good in the topic “Artificial Intelligence”? The good news is, that the website has a dedicated tag for searching only posts from that subject. The bad news is, that this seems to be a low traffic tag. A search results into 4611 hits overall. And that was the search worldwide in the language English. So it seems, that the developers on SO are not interested in programming intelligent machines.

But perhaps we have searched on the wrong place? If were are browsing through the Stackexchange network, we will find a dedicated forum only for AI. The website has also a very low traffic. On a daily basis it generates not more than 750 visits, and the number of overall questions is 1300. If we are sum up the original stackoverflow with the AI forum than the result is around 5900 questions which are AI related. Everybody can calculate for its own, how many people are interested in the subject. If the ordinary question teller has posted 3 questions, than the number of people worldwide who are interested in getting information about AI is not more than 2000. And again, Stackoverflow is not a small website or a blog which is driven by amateurs. It collects the manpower of all programmers from US, Asia, Europe and India.

The raw number can be only called a joke. We have on the one side the most important technology ever invented which is understand by nearly nobody, and on the other hand a collective lack of interest especially by programmers and computer experts.

Even reddit is more interested in AI, they have around 250k postings with the term “Artificial Intelligence”. And reddit is not a programmer website, it is an entertainment forum which contains mostly jokes and stories.

Posting questions

Only for getting the numbers right. If somebody posts 13 simple questions about “Artificial Intelligence” in the AI.stackexchange forum, he will increase by this simple actions, the total count of all worldwide questions from 1300 to 1313. That are +1.0%. Like I mentioned above, this can only be called a joke, because even a single person can post 13 question over a while, and then he was responsible for 1% of the total traffic of that subject?

AI in a RSS feed

The above screenshot shows a RSS feedreader with two subscriptions. The first is the AI.stackexchange forum with around 3 new questions per day, the second is the AI section of which is also available as RSS feed and produces around 30 new papers per day. The difference is, that the AI.stackexchange forum has light-weight-content. That means, it is possible to read all questions. The stream of arxiv is instead a bit difficult to read, because every item is referencing to a complete paper with around 30 pages and in most cases the content is very complicated to understand. Reading the incoming Arxiv paper in realtime is not a good idea, instead I’m prefering to do a fulltext search on the content and retrieve only the needed information.

The open question is, why is the science community able to generate 30 new AI papers every day, but in the AI.stackexchange forum are only 3 new questions per day? Normally, asking a question is easier than writing a paper, so what we see here is a gap between amateur AI and professional AI. The arxiv papers are written usually by professors and science-teams in companies which are doing so their half life.

Getting real numbers about the authors is difficult. I would guess, that the ordinary Arxiv paper was written by 3 authors, and that the average author was subauthor in 10 papers per year. Every year the Arxiv AI section gets 10950 new papers. Now the calculation:

– one author writes 3.3 papers per year alone
– 10950 papers / 3.3 = 3318 authors

The result is, that 3000 Arxiv authors are producing 10000 papers per year in the AI section.