长期记忆神经图灵机
Long-Term Memory Neural Turing Machines
摘要: 可读写的外部记忆模块可以在事实的记忆和基于记忆的推理上扩充神经网络的能力。神经图灵机利用注意力机制设计了一种对内存模块的读写机制，并以循环神经网络作为控制器，实现了排序、复制等算法。为了在更广泛的应用(例如自然语言处理)中缩短训练时间或加快收敛速度，我们在神经图灵机的基础上设计了一种基于全局内存的读写机制，利用卷积运算提取全局内存特征。对于一些较长的序列任务，训练速度相对于神经图灵机提高了6倍，收敛速度也有所提升，在bAbi数据集中取得了更好的推理分类结果。
Abstract: The readable and writable external memory module can improve the ability of the neural network which is based on factual memory and memory-based reasoning. Neural Turing Machines use attention mechanism to design a reading and writing mechanism for the memory module. It also realized sorting, copying and other algorithms by using a recurrent neural network as a controller. To shorten the training time and speed up convergence in a wider range of applications (such as natural language processing), we design a kind of read-write mechanism based on the global memory which is applied in Turing machines. This kind of read-write mechanism does convolution operations to extract the global memory features, whose training speed is 6 times better than Neural Turing Machines; the convergence rate and reasoning results in the bAbi dataset are also better.
文章引用：解笑, 史有群. 长期记忆神经图灵机[J]. 计算机科学与应用, 2018, 8(1): 49-58. https://doi.org/10.12677/CSA.2018.81008

参考文献

[1] Zeiler, M.D. and Fergus, R. (2014) Visualizing and Understanding Convolutional Networks. European Conference on Computer Vision, September 6 2014, 818-833. [Google Scholar] [CrossRef] 
[2] Krizhevsky, A., Sutskever, I. and Hinton, G.E. (2012) Imagenet Classification with Deep Convolutional Neural Networks. Advances in Neural Information Processing Systems, 1097-1105.
[3] Goodfellow, I., Bengio, Y. and Courville, A. (2016) Deep Learning. The MIT Press, Cambridge, 418-422.
[4] Hinton, G.E. (1990) Mapping Part-Whole Hierarchies into Connectionist Networks. Artificial Intelligence, 46, 47-75. [Google Scholar] [CrossRef] 
[5] Hochreiter, S. and Schmidhuber, J. (1997) Long Short-Term Memory. Neural Computation, 9, 1735-1780. [Google Scholar] [CrossRef] [PubMed]
[6] Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D. and Lillicrap, T. (2016) One-Shot Learning with Memory- Augmented Neural Networks. arXiv preprint arXiv: 1605.06065
[7] Weston, J., Chopra, S. and Bordes, A. (2014) Memory Networks. arXiv preprint arXiv:1410.3916
[8] Sukhbaatar, S., Weston, J. and Fergus, R. (2015) End-to-End Memory Networks. Advances in Neural Information Pro- cessing Systems, 2440-2448.
[9] Graves, A., Wayne, G. and Danihelka, I. (2014) Neural Turing Machines. arXiv preprint arXiv:1410.5401
[10] Siegelmann, H.T. and Sontag, E.D. (1995) On the Computational Power of Neural Nets. Journal of Computer and System Sciences, 50, 132-150. [Google Scholar] [CrossRef] 
[11] Kurach, K., Andrychowicz, M. and Sutskever, I. (2015) Neural Random-Access Machines. arXiv preprint arXiv:1511.06392
[12] Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwińska, A., Colmenarejo, S.G., Grefenstette, E., Ramalho, T., Agapiou, J. and Badia, A.P. (2016) Hybrid Computing Using a Neural Network with Dynamic External Memory. Nature, 538, 471-476. [Google Scholar] [CrossRef] [PubMed]
[13] Bahdanau, D., Cho, K. and Bengio, Y. (2014) Neural Machine Translation by Jointly Learning to Align and Translate. arXiv preprint arXiv:1409.0473
[14] Mnih, V., Heess, N. and Graves, A. (2014) Recurrent Models of Visual Attention. Advances in Neural Information Processing Systems, 2204-2212.
[15] LeCun, Y. and Bengio, Y. (1995) Convolutional Networks for Images, Speech, and Time Series. The Handbook of Brain Theory and Neural Networks, 3361, 1995.
[16] Li, Y. (2017) Deep Reinforcement Learning: An Overview. arXiv preprint arXiv:1701.07274
[17] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G. and Petersen, S. (2015) Human-Level Control through Deep Reinforcement Learning. Nature, 518, 529-533. [Google Scholar] [CrossRef] [PubMed]
[18] Reed, S. and De Freitas, N. (2015) Neural Programmer-Interpreters. arXiv preprint arXiv:1511.06279
[19] Weston, J., Bordes, A., Chopra, S., Rush, A.M., van Merriënboer, B., Joulin, A. and Mikolov, T. (2015) Towards Ai- Complete Question Answering: A Set of Prerequisite Toy Tasks. arXiv preprint arXiv:1502.05698
[20] Stroh, E. and Mathur, P. (2016) Question Answering Using Deep Learning.