WNCG student Zhiwen Fan and his teammate Rishov Sarkar (Georgia Tech) were among the winners of this year's Qualcomm Innovation Fellowships for North America. The pair's winning project studied "Real-Time Visual Processing for Autonomous Driving via Video Transformer with Data-Model-Accelerator Tri-Design."
"Multi-task learning (MTL) encapsulates multiple learned tasks in a single model and has become successful, and often essential, for autonomous driving and indoor robots," Fan explained. "However, optimizing all tasks simultaneously is often difficult due to gradient conflicts across tasks[.] Most real systems demand only one or two tasks at each moment, while flexibly switching between tasks per need."
"In our QIF project, we proposed M³ViT, which customizes mixture-of-experts (MoE) layers into a vision transformer (ViT) backbone for MTL, coupled with hardware-level innovation. M³ViT reduces the memory requirement by 2.40×, and achieves energy efficiency (as the product of latency and power) up to 9.23× higher than a comparable FPGA baseline, while significantly reducing 88% inference FLOPs than the MTL baseline."
The Qualcomm Innovation Fellowship (QIF) program fosters research by recognizing and rewarding innovation, execution, and teamwork in doctoral student-led projects. The program provides an opportunity for Qualcomm engineers to mentor and support the students in attaining their research goals.
Fan and Sarkar were among only 19 winning teams across North America.
The team's win marks the third consecutive year WNCG students have received Qualcomm Innovation Fellowships. Akash Doshi and Manan Gupta were selected in 2021 for their project "Federated Generative Learning for Channel Estimation in mmWave and THz," and Khurram Mazher and Andrew Graff won in 2020 for the project "Radar-to-radar Interference: System Level Analysis and Solutions."
After this year's win, Fan is excited about the road ahead.
"The QIF supports my team [in] ambitious research projects, networking opportunities, and mentorship from the industry," Fan stated. "I always believe that innovation and passion are powerful forces for research, particularly as my interest [is] in pursuing frontier research with a strong product impact."
Zhiwen Fan is a Ph.D. student at the University of Texas at Austin, advised by Prof. Atlas Wang. Rishov Sarkar is a Ph.D. student at Georgia Tech advised by Prof. Callie Hao.
More information
https://www.qualcomm.com/research/university-relations/innovation-fellowship/2022-north-america