It's not surprising that TF is the slowest in many cases. It has been widely, sometimes harshly, criticized in the past for that reason. On the other hand, despite its speed TF appears to be the only tool that doesn't have to sit out any of the tests due to incompatibilities or lack of features.

Other tools like MXNet deserve a shoutout as well, and it would be interesting to see how a wider group compares. MXNet also integrates seamlessly into R, something of a rarity in deep learning tools (excepting the also excellent h2o package).

When properly configured, most of these libraries use NVidia's CuDNN package under the hood. The only thing you're really measuring here is overhead, not the actual computation.

No Theano comparison?

CNTK is great at scaling out beyond a simple machine. The paper didn't benchmark that but only tested one single box performance.

I wonder why Torch is so slow. But, the authors did not provide the configuration of each tool in the paper or on the web.