by Researchers at the Ecole Polytechnique Federale de Lausanne (EPFL) have developed a groundbreaking algorithm that allows for the training of analog neural networks with the same accuracy as their digital counterparts.
Deep neural networks, such as Chat-GPT, have revolutionized data processing with their ability to learn through algorithms rather than traditional programming. However, as these systems have grown in size and complexity, their energy consumption has also increased significantly.
In search of a solution, EPFL researcher Romain Fleury and his team focused on physical alternatives to digital deep neural networks. Their paper, published in the journal Science, describes an algorithm for training physical systems that offer improved speed, enhanced robustness, and reduced power consumption compared to existing methods.
The team successfully tested their training algorithm on three wave-based physical systems that utilize sound waves, light waves, and microwaves to carry information. However, the algorithm is versatile and can be applied to train any physical system, according to Ali Momeni, the first author and researcher of the Laboratory of Wave Engineering at EPFL.
Neural network training involves teaching systems how to generate optimal parameter values for tasks like image or speech recognition. Traditionally, this process includes a forward pass to calculate an error function based on the output and a backward pass, known as backpropagation (BP), to calculate the gradient of the error function with respect to network parameters.
BP, while effective in digital systems, is energy-intensive and ill-suited for physical systems. Usually, the training of physical systems requires a digital twin for the BP step, which is inefficient and may lead to discrepancies between the digital model and the physical system.
To overcome this issue, the EPFL scientists proposed replacing the BP step with a second forward pass through the physical system. This approach allows for local updates in each network layer, reducing power consumption and eliminating the need for a digital twin. Additionally, this method aligns better with human learning processes.
Momeni explains that while neural network structures are inspired by the brain, it is unlikely that the brain learns through backpropagation. By training each physical layer locally, the researchers can use the actual physical system instead of relying on a digital model, resulting in a more biologically plausible approach.
The researchers successfully applied their physical local learning algorithm (PhyLL) to train experimental acoustic and microwave systems, as well as a modeled optical system, for tasks such as classifying vowel sounds and images. The accuracy achieved by PhyLL was comparable to that of BP-based training methods and showcased robustness and adaptability, even in the face of unpredictable external perturbations.
Although the EPFL’s approach is the first to enable BP-free training of deep physical neural networks, some digital parameter updates are still necessary. Nonetheless, the team aims to minimize digital computation as much as possible in their hybrid training approach.
Moving forward, the researchers plan to implement their algorithm on a small-scale optical system to tackle network scalability. While they successfully trained neural networks with up to 10 layers in their experiments, the next challenge lies in extending this approach to networks with 100 layers and billions of parameters. Overcoming the technical limitations of physical systems will be crucial in achieving this goal.
Note:
- Source: Coherent Market Insights, Public sources, Desk research
- We have leveraged AI tools to mine information and compile it
