Efficient Low-precision Training for Deep Learning Accelerators

Training Deep Neural Networks (DNNs) is computationally intensive, driving interest in low-precision arithmetic to improve efficiency. This thesis explores novel approaches to enable efficient low-precision training for deep learning accelerators. First, we introduce MPTorch-FPGA, an extension of the MPTorch framework designed to facilitate mixed and low-precision DNN training across CPU, GPU, and FPGA environments. MPTorch-FPGA can generate a model-specific accelerator for DNN training, with customizable sizes and arithmetic implementations, providing bit-level accuracy with respect to emulated low-precision DNN training on GPUs or CPUs. An offline matching algorithm selects one of several pre-generated (static) FPGA configurations using a performance model to estimate latency. Second, we propose an optimized stochastic rounding-enabled multiply-accumulate (MAC) unit tailored for low-precision training. Our design employs an FP8 multiplier with FP12 accumulation and relies on a novel implementation of stochastic rounding within floating point adders, significantly reducing area, power consumption and delay compared to conventional implementations. Together, these contributions highlight the potential of customized arithmetic and mixed-precision training to enhance deep learning accelerator performance while preserving model accuracy.