Demuxed 2022: A Journey from Manual to Fully Automatic Video Player Testing

In this talk at the Demuxed 2022 conference for video engineers, I explain how my team and I transformed the testing process of the Bitmovin Video Player and scaled test executions from hundreds to millions.

After introducing the initial manual testing process, I explain why we decided to develop a custom testing framework, rather than relying on existing tools like Selenium. This approach enabled us to parallelize and execute tests more efficiently, manage flaky tests, and triage failures. Another important aspect of this strategy is the use of a custom embedded domain-specific language (EDSL), which allows us to write and maintain concise and robust test cases.

Additionally, we automate a vast array of consumer devices to run these tests. This automation extended across a wide range of platforms, including various browsers on Windows, macOS, and Linux computers, as well as on Android and iOS smartphones. Our testing scope also covers smart TVs from brands like Samsung, LG, Panasonic, and Hisense, gaming consoles such as PlayStation and Xbox, and streaming sticks like Chromecast. I highlight the challenges we faced in automating these devices, and how it even led us to develop our own automation hardware. This proved invaluable in allowing remote device use from all of our global offices and helped our operations remain uninterrupted during the Corona pandemic.

In conclusion, I introduce Stream Lab, a new product built upon this testing platform that offers our customers the ability to test their streams across our extensive device farm.