tl&dr: we don't understand how deep networks learn
Ah, but does it matter? If the goal is science, then "yes". If the goal is practical solutions, then "no".
Humans seem perfectly comfortable using their brains despite having no clue how they work.
This is not wrong. But it does not address the core issue.
Even for practical methods, you need to know what the underlying principles are I suppose. Avoids nasty surprises. Even Hinton is reviewing his backpropagation algorithm.
I would prefer to meet a panda than a gibbon in a dark alleyway on a Saturday night. Improving in the next evolutionary version is too late.
Are all sheep in Scotland black or is that a sheep in the field, one of whose sides is black?
Seriously, you need to know for which class of problems a given method is suitable. The original Perceptron broke down because the corresponding mathematical/minimisation problem did not have a solution. Seems to me that this should be the first problem to address..