A Stochastic Computing Method For Generating Activation Functions in Multilayer Feedforward Neural Networks

Erkmen, Burcu; ERSOY, Durmuş

doi:10.5152/electr.2021.21043

A Stochastic Computing Method For Generating Activation Functions in Multilayer Feedforward Neural Networks

Durmuş ERSOY, (İstanbul Esenyurt Üniversitesi, Elektrik ve Elektronik Mühendisliği Bölümü, İstanbul, Türkiye)

Burcu ERKMEN (Yıldız Teknik Üniversitesi, Elektronik ve Haberleşme Mühendisliği Bölümü, İstanbul, Türkiye)

Electrica

7 2

Yıl: 2021 Cilt: 21 Sayı: 3 Sayfa Aralığı: 376 - 387 Metin Dili: İngilizce DOI: 10.5152/electr.2021.21043 İndeks Tarihi: 29-01-2022

A Stochastic Computing Method For Generating Activation Functions in Multilayer Feedforward Neural Networks

Öz:

Stochastic computing using basic arithmetic logic elements based on stochastic bit sequences provides very beneficial solutions in terms of speed and hardware cost, relative to deterministic calculation. Studies for the realization of tangent hyperbolic and exponential functions used in the development of activation functions in Artificial Neural Networks by stochastic methods exist in the literature. The techniques presented using state transitions on finite state machines were constructed on the basis of two different forms of finite state machines, one-dimensional (Linear) and two-dimensional. In this analysis, in terms of both error rate and circuit cost, the advantageous two-dimensional finite state machines-based stochastic computing approach for tangent hyperbolic and exponential functions is presented. The presented approach is implemented on Field Programmable Gate Array and the results are given for hardware simulation. The dataset used for the classification process in a decentralized smart grid control has been applied to the multilayer feedforward neural network and deterministic computing, for the stability classification which is carried out separately with the linear finite state machines-based stochastic computing and the proposed 2D finite state machines-based stochastic computing methods.

Anahtar Kelime:

Belge Türü: Makale Makale Türü: Araştırma Makalesi Erişim Türü: Erişime Açık

1. J. Li, A. Ren, Z. Li, C. Ding, B. Yuan, Q. Qiu, and Y. Wang. “Towards acceleration of deep convolutional neural networks using stochastic computing,” 22nd Asia and South Pacific Design Automation Conference (ASP-DAC), pp. 115–120, Jan. 2017.
2. B. R. Gaines, “Stochastic computing systems,” Advances in Information Systems Science. Tou J.T. (eds) Boston, MA: Springer, 1969, pp. 37–172.
3. P. Li and D. J. Lilja, “Using stochastic computing to implement digital image processing algorithms” 29th International Conference on Computer Design (ICCD), pp. 154–161, October 2011.
4. P. Li, D. J. Lilja, W. Qian, K. Bazargan and M. D. Riedel, “Computation on stochastic bit streams digital image processing case studies,” IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 22, no. 3, pp. 449–462, April 2013.
5. X. Zeng and J. Wang, “A parallel hybrid electric vehicle energy management strategy using stochastic model predictive control with road grade preview,” IEEE Transactions on Control Systems Technology, vol. 23, no. 6, pp. 2416–2423, Nov. 2015.
6. J. Zheng, Y. Cai, Y. Wu and X. Shen, “Dynamic computation offloading for mobile cloud computing: A stochastic game-theoretic approach,” IEEE Transactions on Mobile Computing, vol. 18, no. 4, pp. 771–786, June 2018.
7. G. Arslan and S. Yüksel, “Decentralized Q-learning for stochastic teams and games,” IEEE Transactions on Automatic Control, vol. 62, no. 4, pp. 1545–1558, Aug. 2016.
8. B. Li, Y. Qin, B. Yuan and D. J. Lilja, “Neural network classifiers using stochastic computing with a hardware-oriented approximate activation function,” IEEE International Conference on Computer Design (ICCD), vol. 66, no. 7, pp. 97 –104, Nov. 2017.
9. S. Liu, H. Jiang, L. Liu and J. Han, “Gradient descent using stochastic circuits for efficient training of learning machines,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 37, no. 11, pp. 2530–2541, Oct. 2018.
10. Y. Liu, S. Liu, Y. Wang, F. Lombardi and J. Han, “A stochastic computational multi-layer perceptron with backward propagation,” IEEE Transactions on Computers, vol. 67, no. 9, pp. 1273–1286, March 2018.
11. J. Yu, K. Kim, J. Lee and K. Choi, “Accurate and efficient stochastic computing hardware for convolutional neural networks,” IEEE International Conference on Computer Design (ICCD), pp. 105–112, Nov. 2017.
12. Z. Li, J. Li, A. Ren, C. Ding, J. Draper, Q. Qiu, B. Yuan, and Y. Wang, “Towards budget-driven hardware optimization for deep convolutional neural networks using stochastic computing,” IEEE Computer Society Annual Symposium on VLSI (ISVLSI), pp. 28–33, July 2018.
13. Y. Ji, F. Ran, C. Ma and D. J. Lilja, “A hardware implementation of a radial basis function neural network using stochastic logic,” Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 880–883, Mar. 2015.
14. B. D. Brown and H. C. Card, “Stochastic neural computation. I. Computational elements,” IEEE Transactions on Computers, vol. 50, no. 9, pp. 891–905, Sept. 2001.
15. P. Li, D. J. Lilja, W. Qian, M. D. Riedel and K. Bazargan, “Logical computation on stochastic bit streams with linear finite-state machines,” IEEE Transactions on Computers, vol. 63, no. 6, pp. 1474–1486, June 2014.
16. P. S. Ting and J. P. Hayes, “Stochastic logic realization of matrix operations,”17th Euromicro Conference on Digital System Design, pp. 356–364, Oct. 2014.
17. K. Kim, J. Kim, J. Yu, J. Seo, J. Lee and K. Choi, “Dynamic energyaccuracy trade-off using stochastic computing in deep neural networks, ” Proceedings of the 53rd Annual Design Automation Conference, pp. 1–6, June 2016.
18. A. Ardakani, F. Leduc-Primeau, N. Onizawa, T. Hanyu and W. J. Gross, “VLSI implementation of deep neural network using integral stochastic computing,” IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 25, no. 10, pp. 2688–2699, October 2017.
19. P. Li, D. J. Lilja, W. Qian, K. Bazargan and M. Riedel, “The synthesis of complex arithmetic computation on stochastic bit streams using sequential logic,” Proceedings of the International Conference on Computer-Aided Design, pp. 480–487, Nov. 2012.
20. D. Ersoy and B. Erkmen, “Stochastic Gaussian function for RBF network,” IEEE Transactions on Bio-Medical Engineering International Conference on Electrical, Communication, and Computer Engineering (ICECCE), pp. 1–3, June 2020.
21. S. C. Smithson, K. Boga, A. Ardakani, B. H. Meyer and W. J. Gross, “Stochastic computing can improve upon digital spiking neural networks,” IEEE International Workshop on Signal Processing Systems (SiPS), Dallas, TX, USA, pp. 309–314, Oct. 2016.
22. J. Von Neumann, “Probabilistic logics and the synthesis of reliable organisms from unreliable components.”, In C. Shannon and J. McCarthy,editors, Automata Studies, Vol. 34, pp. 43–98, Princeton University Press, 1956.
23. W. J. Poppelbaum, C. Afuso and J. W. Esch, “Stochastic computing elements and systems,” Proceedings of the November 14-16, 1967, Fall Joint Computer Conference, pp. 635–644, Nov. 1967.
24. Md. A. Abeed, and S. Bandyopadhyay, “Sensitivity of the power spectra of thermal magnetization fluctuations in low barrier nanomagnets proposed for stochastic computing to in-plane barrier height variations and structural defects,” Spin, vol. 10, no. 1, Oct. 2020.
25. T. Hirtzlin, B. Penkovsky, M. Bocquet, J. Klein, J. Portal and D. Querlioz, “Stochastic computing for hardware implementation of binarized neural networks,” IEEE Access, vol. 7, pp. 76394–76403, June 2019.
26. O. Camps, S. G. Stavrinides and R. Picos, “Stochastic computing implementation of chaotic systems,” Mathematics, vol. 9, no. 4, p. 375, Feb. 2021.
27. V. C. Gaudet and A. C. Rapley, “Iterative decoding using stochastic computation,” Electronics Letters, vol. 39, no. 3, pp. 299–301, Feb. 2003.
28. S. I. Chu, C. E. Hsieh and Y. J. Huang, “Design of FSM-based function with reduced number of states in integral stochastic computing,” IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 27, no. 6, pp. 1475–1479, Jan. 2019.
29. Y. Liu and K. K. Parhi, “Architectures for recursive digital filters using stochastic computing,” IEEE Transactions on Signal Processing, vol. 64, no. 14, pp. 3705–3718, April 2016.
30. A. A. Markov , “Extension of the limit theorems of probability theory to a sum of variables connected in a chain.” The Notes of the Imperial Academy of Sciences of St. Petersburg VIII Series, PhysioMathematical College, vol. XXII, no. 9, May 1907.
31. V. Arzamasov, K. Böhm, P. Jochem, U. Machine and Learning Repository, May. 2020. Available: https://archive.ics.uci.edu/ml/ datasets/Electrical+Grid+Stability+Simulated+Data+. [Accessed].

APA	ERSOY D, Erkmen B (2021). A Stochastic Computing Method For Generating Activation Functions in Multilayer Feedforward Neural Networks. , 376 - 387. 10.5152/electr.2021.21043
Chicago	ERSOY Durmuş,Erkmen Burcu A Stochastic Computing Method For Generating Activation Functions in Multilayer Feedforward Neural Networks. (2021): 376 - 387. 10.5152/electr.2021.21043
MLA	ERSOY Durmuş,Erkmen Burcu A Stochastic Computing Method For Generating Activation Functions in Multilayer Feedforward Neural Networks. , 2021, ss.376 - 387. 10.5152/electr.2021.21043
AMA	ERSOY D,Erkmen B A Stochastic Computing Method For Generating Activation Functions in Multilayer Feedforward Neural Networks. . 2021; 376 - 387. 10.5152/electr.2021.21043
Vancouver	ERSOY D,Erkmen B A Stochastic Computing Method For Generating Activation Functions in Multilayer Feedforward Neural Networks. . 2021; 376 - 387. 10.5152/electr.2021.21043
IEEE	ERSOY D,Erkmen B "A Stochastic Computing Method For Generating Activation Functions in Multilayer Feedforward Neural Networks." , ss.376 - 387, 2021. 10.5152/electr.2021.21043
ISNAD	ERSOY, Durmuş - Erkmen, Burcu. "A Stochastic Computing Method For Generating Activation Functions in Multilayer Feedforward Neural Networks". (2021), 376-387. https://doi.org/10.5152/electr.2021.21043

APA	ERSOY D, Erkmen B (2021). A Stochastic Computing Method For Generating Activation Functions in Multilayer Feedforward Neural Networks. Electrica, 21(3), 376 - 387. 10.5152/electr.2021.21043
Chicago	ERSOY Durmuş,Erkmen Burcu A Stochastic Computing Method For Generating Activation Functions in Multilayer Feedforward Neural Networks. Electrica 21, no.3 (2021): 376 - 387. 10.5152/electr.2021.21043
MLA	ERSOY Durmuş,Erkmen Burcu A Stochastic Computing Method For Generating Activation Functions in Multilayer Feedforward Neural Networks. Electrica, vol.21, no.3, 2021, ss.376 - 387. 10.5152/electr.2021.21043
AMA	ERSOY D,Erkmen B A Stochastic Computing Method For Generating Activation Functions in Multilayer Feedforward Neural Networks. Electrica. 2021; 21(3): 376 - 387. 10.5152/electr.2021.21043
Vancouver	ERSOY D,Erkmen B A Stochastic Computing Method For Generating Activation Functions in Multilayer Feedforward Neural Networks. Electrica. 2021; 21(3): 376 - 387. 10.5152/electr.2021.21043
IEEE	ERSOY D,Erkmen B "A Stochastic Computing Method For Generating Activation Functions in Multilayer Feedforward Neural Networks." Electrica, 21, ss.376 - 387, 2021. 10.5152/electr.2021.21043
ISNAD	ERSOY, Durmuş - Erkmen, Burcu. "A Stochastic Computing Method For Generating Activation Functions in Multilayer Feedforward Neural Networks". Electrica 21/3 (2021), 376-387. https://doi.org/10.5152/electr.2021.21043