Synaptic 1/f noise injection for overfitting suppression in hardware neural networks

Du, Y, Shao, W, Chai, Z ORCID: 0000-0003-3446-7138, Zhao, H, Diao, Q, Gao, Y, Yuan, X, Wang, Q, Li, T ORCID: 0000-0002-3337-6202, Zhang, WD ORCID: 0000-0003-4600-7382, Zhang, JF and Min, T (2022) Synaptic 1/f noise injection for overfitting suppression in hardware neural networks. Neuromorphic Computing and Engineering, 2.

Preview

Text Synaptic 1 f noise injection for overfitting suppression in hardware neural networks.pdf - Published Version Available under License Creative Commons Attribution. Download (3MB) | Preview

Abstract

Overfitting is a common and critical challenge for neural networks trained with limited dataset. The conventional solution is software-based regularization algorithms such as Gaussian noise injection. Semiconductor noise, such as 1/f noise, in artificial neuron/synapse devices, which is often regarded as undesirable disturbance to the hardware neural networks (HNNs), could also play a useful role in suppressing overfitting, but that is as yet unexplored. In this work, we proposed the idea of using 1/f noise injection to suppress overfitting in different neural networks, and demonstrated that: (i) 1/f noise could suppress the overfitting in Multilayer Perceptron (MLP) and long short-term memory (LSTM); (ii) 1/f noise and Gaussian noise performs similarly for the MLP but differently for the LSTM; (iii) The superior performance of 1/f noise on LSTM can be attributed to its intrinsic long range dependence. This work reveals that 1/f noise, which is common in semiconductor devices, can be a useful solution to suppress the overfitting in HNNs, and more importantly, further evidents that the imperfectness of semiconductor devices is a rich mine of solutions to boost the development of brain-inspired hardware technologies in the AI era.

Item Type:	Article
Subjects:	T Technology > TA Engineering (General). Civil engineering (General)
Divisions:	Engineering
Publisher:	IOP Publishing
Date of acceptance:	5 May 2022
Date of first compliant Open Access:	10 May 2022
Date Deposited:	10 May 2022 11:56
Last Modified:	03 Jul 2025 12:29
DOI or ID number:	10.1088/2634-4386/ac6d05
URI:	https://researchonline.ljmu.ac.uk/id/eprint/16797

View Item