Protyposis

Travelling through time & space in real-time...

AUTHOR: Mario

Smartifying and automating a cheap fog machine

The cheapest fog machines typically come with a wired remote control with an indicator light and a button. The indicator light signals when the machine is at its operating temperature and ready to emit fog, and the button makes it emit fog while pressed. However, this turned out inconvenient once I found myself repeatedly running to the machine to trigger it. And even if it were wireless, it would still need manual handling. So this calls for automation, and it’s quite easily possible without DIY electronics, with a Wi-Fi Shelly switch and a relay. Watch the demo video below and read on for build instructions and a Home Assistant automation.

Visual smart home notifications with WLED

A smart home sometimes needs to notify its users of certain events or states. Instead of installing dedicated notification devices, why not use the smart lights which are present in most smart homes anyway? This turned out to be much more difficult than it sounds. However, I found that WLED, a great firmware for LED light strip controllers, offers an interface almost predestined to implement visual notification effects, and so I wrote and published a Node-RED node which does exactly that. Visual notification effects, simple to use, without side effects. It’s open source and now available on GitHub, npm, and the Node-RED library. Watch the demo video below and read on for more details.

Automatic subtitle synchronization for edited TV productions

Two years ago I was approached by someone from a public TV broadcaster in Germany with the following problem: Given multiple video files with differently cut versions of the same production, is it possible to use the technology from AudioAlign/Aurio to automatically generate edit decision lists (EDL/XML) and use them to transfer subtitles from a reference version to the different cuts? The answer is “yes”, and that’s just one of many use-cases. This article describes the challenges and how the Aurio technology solves them almost magically in a successful prototype developed for the TV station.

Aurio adds support for realtime live fingerprinting and cross-plattform cloud deployments with .NET Core 2.0

A collaboration with eyecandylab, a company developing products for augmenting TV programs, recently gave me the opportunity to implement great new features into Aurio. The most recent version released today extends the architecture to support processing of realtime audio streams with infinite lengths, which means that live streams can now be fingerprinted on the fly with minimal latency. Additionally, the Aurio core library has been ported to .NET Standard 2.0 and will run with the .NET Core 2.0 framework on Windows, Linux and MacOS, enabling building microservices in containerized environments like Docker.

An example application named Aurio.Test.RealtimeFingerprinting has been added to demonstrate how realtime live fingerprinting can be implemented with only a few lines of code. As part of supporting .NET Core, resampler and FFT dependencies building on native code have been moved from the core into optional add-on packages, and the WDL resampler ported to purely managed .NET code by NAudio has been integrated as an alternative for deployments where native dependencies are undesired. This means that now there are purely managed implementations available for both FFT (Exocortex.DSP with Aurio.Exocortex) and resampling (NAudioWdlResampler within the Aurio core).

Of course Aurio continues to support the good old WPF GUI applications, but the framework requirement had to be increased from .NET Framework 4.0 to 4.6.2. AudioAlign has also been updated to the latest Aurio version.

Aurio on GitHub
AudioAlign on GitHub

Demuxed 2017: Ambisonic Audio Using the WebAudio API

This October I talked about spherical audio for 360°/VR videos at Demuxed, a conference for video engineers. I explain the principles of the Ambisonics surround sound technique from recording to playback through a speaker array or headphones with binaural playback, and how it can be used in web applications and video players through the Web Audio API.

The slides edited into the video are slightly incomplete and miss a few important details, and I made a mistake talking about “headphones” instead of “microphones” on the HRTF slide. Other than that, I think the video can be helpful to everyone trying to understand what the Ambisonics technique is. There’s also a short recap on the talk, conference, and the Ambisonics integration into the Bitmovin video player on the company blog called 360° VR Audio with the Bitmovin Player.

Native immersive 360° VR video playback on Android with Spectaculum

Playback of immersive 360° video on Android is usually done in a WebView with an HTML5 video player. This tutorial demonstrates how to display 360° video in a native view widget to save the overhead of a whole browser stack. This is done by using the versatile Spectaculum view widget for video rendering and the popular ExoPlayer for video decoding. Both of these libraries are open source under the Apache 2.0 license and available on GitHub and the JCenter repository.

Spectaculum – Android OpenGL ES Accelerated View for Picture and Video Content

Spectaculum is a view widget for Android to display visual content in a GLES accelerated context, providing zooming and panning functionality, parameterized shader effects, and frame grabbing. It comes with additional views that save developers a lot of time and implement all functionality for displaying bitmap images, camera preview, and videos through the Android MediaPlayer, MediaPlayer Extended, and ExoPlayer. The packaged shader effects range from simple color filters to immersive 360° VR video rendering.

The Spectaculum view can be used with all sources that can write to a surface or surface texture, which is essentially every visual content source, but I recommend using one of the many available modules if applicable. Example use-cases are photo galleries and picture viewers with zooming/panning support and optional picture effects through shaders (e.g. contrast adjustment, color correction), video players with live image adjustments through shaders and 3D/360°/immersive/VR playback, and camera previews with live effects. An extensive documentation on functionality, API, usage, and modules is available on GitHub. The library is also available from JCenter’s Maven repository, and a demo app that showcases various views and shader effects is available on the Play Store.

Spectaculum on GitHub
Spectaculum Demo
Spectaculum Demo
Developer:
Price: Free

Painful Bugs: Android’s Matrix.setRotateEulerM(…) Rotation Around Y-Axis

Android’s Matrix.setRotateEulerM(…) function contains a bug and returns a wrong matrix result when rotating around the Y-axis. Googling android setRotateEulerM bug returns just 15 hits out of which only the first one is relevant: a neglected AOSP issue, marked as obsolete, that luckily contains the fix in comment #3. This bug has been reported in 2010 for Android 2.2 and still exists today in Android 6.0.1.

Left: rotation gone wrong. Right: rotation with fixed function.

Can it really be the case that almost nobody is using this function although there are thousands of OpenGL apps and games around? Why is it not discussed more frequently? Why is it marked as obsolete? Is this not a bug, but a case of incorrect usage?

Publishing a JavaScript Library – for Dummies

Publishing a JavaScript library can be overwhelming these days. You want to reach a crowd as broad as possible, but there are dozens of different approaches existing and choices to be made. This article is a quick summary of current best practices, which are developing with npm, and publishing via npm and Bower package managers as well as a universal standalone distributable file. To follow, you need to be familiar with npm.

In the good old times when Prototype and jQuery were competing for the default JS framework, publishing a JS library was easy. Provide a download of the .js file somewhere, optionally add a fancy minified version, and specify the dependencies on other libraries that had to be present. Potential users would manually download the file (CDNs were just starting to be used), add it to their project, take care that the dependencies were met, and hopefully get it to work. Today, there are dependency managers that just require a single install command and take care about (almost) everything else. This of course requires some legwork to be done by the developer, which isn’t too straightforward as I recently found out. In an experiment on publishing a simple JS library called gallerygrid.js, it took me quite an amount of research time to figure out how to publish it, almost more than the actual time needed to develop the library itself. So this is a guide that sums up the current best practice and required steps to publish a JavaScript library.