Saptarshi Mitra

selected publications

1. BMVC

   DeepliteRT: Computer Vision at the Edge

   Ashfaq, Saad, Hoffman, Alexander,  Mitra, Saptarshi, Sah, Sudhakar, AskariHemmat, MohammadHossein, and Saboori, Ehsan

   2023

   Abs Bib PDF

   The proliferation of edge devices has unlocked unprecedented opportunities for deep learning model deployment in computer vision applications. However, these complex models require considerable power, memory and compute resources that are typically not available on edge platforms. Ultra low-bit quantization presents an attractive solution to this problem by scaling down the model weights and activations from 32-bit to less than 8-bit. We implement highly optimized ultra low-bit convolution operators for ARM-based targets that outperform existing methods by up to 4.34x. Our operator is implemented within Deeplite Runtime (DeepliteRT), an end-to-end solution for the compilation, tuning, and inference of ultra low-bit models on ARM devices. Compiler passes in DeepliteRT automatically convert a fake-quantized model in full precision to a compact ultra low-bit representation, easing the process of quantized model deployment on commodity hardware. We analyze the performance of DeepliteRT on classification and detection models against optimized 32-bit floating-point, 8-bit integer, and 2-bit baselines, achieving significant speedups of up to 2.20x, 2.33x and 2.17x, respectively.

   @misc{ashfaq2023deeplitert,
     title = {DeepliteRT: Computer Vision at the Edge},
     author = {Ashfaq, Saad and Hoffman, Alexander and Mitra, Saptarshi and Sah, Sudhakar and AskariHemmat, MohammadHossein and Saboori, Ehsan},
     year = {2023},
     eprint = {2309.10878},
     archiveprefix = {arXiv},
     primaryclass = {cs.LG},
   }

2. ICCVW

   YOLOBench: Benchmarking Efficient Object Detectors on Embedded Systems

   Lazarevich, Ivan, Grimaldi, Matteo, Kumar, Ravish,  Mitra, Saptarshi, Khan, Shahrukh, and Sah, Sudhakar

   In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops Oct 2023

   Abs Bib PDF Code

   We present YOLOBench, a benchmark comprised of 550+ YOLO-based object detection models on 4 different datasets and 4 different embedded hardware platforms (x86 CPU, ARM CPU, Nvidia GPU, NPU). We collect accuracy and latency numbers for a variety of YOLO-based one-stage detectors at different model scales by performing a fair, controlled comparison of these detectors with a fixed training environment (code and training hyperparameters). Pareto-optimality analysis of the collected data reveals that, if modern detection heads and training techniques are incorporated into the learning process, multiple architectures of the YOLO series achieve a good accuracy-latency trade-off, including older models like YOLOv3 and YOLOv4. We also evaluate training-free accuracy estimators used in neural architecture search on YOLOBench and demonstrate that, while most state-of-the-art zero-cost accuracy estimators are outperformed by a simple baseline like MAC count, some of them can be effectively used to predict Pareto-optimal detection models. We showcase that by using a zero-cost proxy to identify a YOLO architecture competitive against a state-of-the-art YOLOv8 model on a Raspberry Pi 4 CPU. The code and data are available.

   @inproceedings{Lazarevich_2023_ICCV,
     author = {Lazarevich, Ivan and Grimaldi, Matteo and Kumar, Ravish and Mitra, Saptarshi and Khan, Shahrukh and Sah, Sudhakar},
     title = {YOLOBench: Benchmarking Efficient Object Detectors on Embedded Systems},
     booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops},
     month = oct,
     year = {2023},
     pages = {1169-1178}
   }

3. CVPRW

   DeepGEMM: Accelerated Ultra Low-Precision Inference on CPU Architectures Using Lookup Tables

   Ganji, Darshan C., Ashfaq, Saad, Saboori, Ehsan, Sah, Sudhakar,  Mitra, Saptarshi, AskariHemmat, MohammadHossein, Hoffman, Alexander, Hassanien, Ahmed, and Léonardon, Mathieu

   In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops Jun 2023

   Abs Bib PDF

   A lot of recent progress has been made in ultra low-bit quantization, promising significant improvements in latency, memory footprint and energy consumption on edge devices. Quantization methods such as Learned Step Size Quantization can achieve model accuracy that is comparable to full-precision floating-point baselines even with sub-byte quantization. However, it is extremely challenging to deploy these ultra low-bit quantized models on mainstream CPU devices because commodity SIMD (Single Instruction, Multiple Data) hardware typically supports no less than 8-bit precision. To overcome this limitation, we propose DeepGEMM, a lookup table based approach for the execution of ultra low-precision convolutional neural networks on SIMD hardware. The proposed method precomputes all possible products of weights and activations, stores them in a lookup table, and efficiently accesses them at inference time to avoid costly multiply-accumulate operations. Our 2-bit implementation outperforms corresponding 8-bit integer kernels in the QNNPACK framework by up to 1.74x on x86 platforms.

   @inproceedings{Ganji_2023_CVPR,
     author = {Ganji, Darshan C. and Ashfaq, Saad and Saboori, Ehsan and Sah, Sudhakar and Mitra, Saptarshi and AskariHemmat, MohammadHossein and Hoffman, Alexander and Hassanien, Ahmed and L\'eonardon, Mathieu},
     title = {DeepGEMM: Accelerated Ultra Low-Precision Inference on CPU Architectures Using Lookup Tables},
     booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
     month = jun,
     year = {2023},
     pages = {4655-4663},
   }

4. DAC

   Accelerating and Pruning CNNs for Semantic Segmentation on FPGA

   Morı̀, Pierpaolo, Vemparala, Manoj-Rohit, Fasfous, Nael,  Mitra, Saptarshi, Sarkar, Sreetama, Frickenstein, Alexander, Frickenstein, Lukas, Helms, Domenik, Nagaraja, Naveen Shankar, Stechele, Walter, and Passerone, Claudio

   In Proceedings of the 59th ACM/IEEE Design Automation Conference Jun 2022

   Abs Bib PDF Code

   Semantic segmentation is one of the popular tasks in computer vision, providing pixel-wise annotations for scene understanding. However, segmentation-based convolutional neural networks require tremendous computational power. In this work, a fully-pipelined hardware accelerator with support for dilated convolution is introduced, which cuts down the redundant zero multiplications. Furthermore, we propose a genetic algorithm based automated channel pruning technique to jointly optimize computational complexity and model accuracy. Finally, hardware heuristics and an accurate model of the custom accelerator design enable a hardware-aware pruning framework. We achieve 2.44X lower latency with minimal degradation in semantic prediction quality (−1.98 pp lower mean intersection over union) compared to the baseline DeepLabV3+ model, evaluated on an Arria-10 FPGA. The binary files of the FPGA design, baseline and pruned models can be found in github.com/pierpaolomori/SemanticSegmentationFPGA

   @inproceedings{10.1145/3489517.3530424,
     author = {Mor\`{\i}, Pierpaolo and Vemparala, Manoj-Rohit and Fasfous, Nael and Mitra, Saptarshi and Sarkar, Sreetama and Frickenstein, Alexander and Frickenstein, Lukas and Helms, Domenik and Nagaraja, Naveen Shankar and Stechele, Walter and Passerone, Claudio},
     title = {Accelerating and Pruning CNNs for Semantic Segmentation on FPGA},
     year = {2022},
     isbn = {9781450391429},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3489517.3530424},
     doi = {10.1145/3489517.3530424},
     booktitle = {Proceedings of the 59th ACM/IEEE Design Automation Conference},
     pages = {145–150},
     numpages = {6},
     location = {San Francisco, California},
     series = {DAC '22},
   }