Low Resource, Post-processed Lecture Recording from 4K Video Streams

21 pages•Published: July 18, 2022

Charles Fitzhenry, Tanweer Khatieb, Patrick Marais and Stephen Marquard

Abstract

Many universities are using lecture recording technology to expand the reach of their teaching programs, and to continue instruction when face to face lectures are not possi- ble. Increasingly, high-resolution 4K cameras are used, since they allow for easy reading of board/screen context. Unfortunately, while 4K cameras are now quite affordable, the back- end computing infrastructure to process and distribute a multitude of recorded 4K streams can be costly. Furthermore, the bandwidth requirements for a 4K stream are exorbitant - running to over 2GB for a 45-60 minute lecture. These factors mitigate against the use of such technology in a low-resource environment, and motivated our investigation into meth- ods to reduce resource requirements for both the institution and students. We describe the design and implementation of a low resource 4K lecture recording solution, which addresses these problems through a computationally efficient video processing pipeline. The pipeline consists of a front-end, which segments presenter motion and writing/board surfaces from the stream and a back-end, which serves as a virtual cinematographer (VC), combining this contextual information to draw attention to the lecturer and relevant content. The bandwidth saving is realized by defining a smaller fixed-size, context-sensitive ‘cropping window’ and generating a new video from the crop regions. The front-end utilises compu- tationally cheap temporal frame differencing at its core: this does not require expensive GPU hardware and also limits the memory required for processing. The VC receives a small set of motion/content bounding boxes and applies established framing heuristics to determine which region to extract from the full 4K frame. Performance results coupled to a user survey show that the system is fit for purpose: it is able to produce good presenter framing/context, over a range of challenging lecture venue layouts and lighting conditions within a time that is acceptable for lecture video processing.

Keyphrases: 4k video post processing, automated lecture recording systems, presenter tracking, video segmentation, virtual cameraman

In: Aurona Gerber (editor). Proceedings of 43rd Conference of the South African Institute of Computer Scientists and Information Technologists, vol 85, pages 15-35.

Links:	https://easychair.org/publications/paper/V8j2
	https://doi.org/10.29007/vfb6

BibTeX entry

@inproceedings{SAICSIT2022:Low_Resource_Post_processed,
  author    = {Charles Fitzhenry and Tanweer Khatieb and Patrick Marais and Stephen Marquard},
  title     = {Low Resource, Post-processed Lecture Recording from 4K Video Streams},
  booktitle = {Proceedings of 43rd Conference of the South African Institute of Computer Scientists and Information Technologists},
  editor    = {Aurona Gerber},
  series    = {EPiC Series in Computing},
  volume    = {85},
  publisher = {EasyChair},
  bibsource = {EasyChair, https://easychair.org},
  issn      = {2398-7340},
  url       = {/publications/paper/V8j2},
  doi       = {10.29007/vfb6},
  pages     = {15-35},
  year      = {2022}}

Download PDF Open PDF in browser