In one of our monthly innodays, where we try out new technologies and different approaches to old problems, we had the idea to collaborate with another company. Slowsoft is a provider of text to speech (TTS) solutions. To my knowledge they are the only ones who are able to generate Swiss German speech synthesis in various Swiss accents. We thought it would be a cool idea to combine it with our existing automatic speech recognition (ASR) expertise and build a cooking assistant that you can operate completely hands free. So no more touching your phone with your dirty fingers only to check again how many eggs you need for that cake. We decided that it would be great to go with some recipes from a famous swiss cookbook provider.

Overview

Generally there are quite a few text to speech solutions out there on the market. In the first out of two blog posts would like to give you a short overview of the available options. In the second blog post I will then describe at which insights we arrived in the UX workshop and how we then combined wit.ai with the solution from slowsoft in a quick and dirty web-app prototype built on socket.io and flask.

But first let us get an overview over existing text to speech (TTS) solutions. To showcase the performance of existing SaaS solutions I've chosen a random recipe from Betty Bossi and had it read by them:

Ofen auf 220 Grad vorheizen. Broccoli mit dem Strunk in ca. 1 1/2 cm dicke Scheiben schneiden, auf einem mit Backpapier belegten Blech verteilen. Öl darüberträufeln, salzen.
Backen: ca. 15 Min. in der Mitte des Ofens.
Essig, Öl und Dattelsirup verrühren, Schnittlauch grob schneiden, beigeben, Vinaigrette würzen.
Broccoli aus dem Ofen nehmen. Einige Chips mit den Edamame auf dem Broccoli verteilen. Vinaigrette darüberträufeln. Restliche Chips dazu servieren. 

But first: How does TTS work?

The classical way works like this: You have to record at least dozens of hours of raw speaker material in a professional studio. Depending on the task, the material can range from navigation instructions to jokes, depending on your use case. The next trick is called "unit-selection", where recorded speech is sliced into a high number (10k - 500k) of elementary components called phones, in order to be able to recombine those into new words, that the speaker has never recorded. The recombination of these components is not an easy task because the characteristics depend on the neighboring phonemes and the accentuation or prosody. These depend on a lot on the context. The problem is to find the right combination of these units that satisfy the input text and the accentuation and which can be joined together without generating glitches. The raw input text is first translated into a phonetic transcription which then serves as the input to selecting the right units from the database that are then concatenated into a waveform. Below is a great example from Apple's Siri engineering team showing how the slicing takes place.

Using an algorithm called Viterbi the units are then concatenated in such a way that they create the lowest "cost", in cost resulting from selecting the right unit and concatenating two units together. Below is a great conceptual graphic from Apple's engineering blog showing this cost estimation.

Now in contrast to the classical way of TTS new methods based on deep learning have emerged. Here deep learning networks are used to predict the unit selection. If you are interested how the new systems work in detail, I highly recommend the engineering blog entry describing how Apple crated the Siri voice. As a final note I'd like to add that there is also a format called speech synthetisis markup language, that allows users to manually specify the prosody for TTS systems, this can be used for example to put an emphasis on certain words, which is quite handy. So enough with the boring theory, let's have a look at the available solutions.

SaaS / Commercial

Google TTS

When thinking about SaaS solutions, the first thing that comes to mind these days, is obviously Google's TTS solution which they used to showcase Google's virtual assistant capabilities on this years Google IO conference. Have a look here if you haven't been wowed today yet. When you go to their website I highly encourage you to try out their demo with a German text of your choice. It really works well - the only downside for us was that it's not really Swiss German. I doubt that they will offer it for such a small user group - but who knows. I've taken a recipe and had it read by Google and frankly liked the output.

Azure Cognitive Services

Microsoft also offers TTS as part of their Azure cognitive services (ASR, Intent detection, TTS). Similar to Google, having ASR and TTS from one provider, definitely has the benefit of saving us one roundtrip since normally you would need to perform the following trips:

1. Send audio data from client to server,
2. Get response to client (dispatch the message on the client)
3. Send our text to be transformed to speech (TTS) from client to server
4. Get the response on client. Play it to the user.

Having ASR and TTS in one place reduces it to:

1. ASR From client to server. Process it on the server.
2. TTS response to client. Play it to the user.

Judging the speech synthesis quality, I personally I think that Microsoft's solution didn't sound as great as Googles synthesis. But have a look for yourself.

Amazon Polly

Amazon - having placed their bets on Alexa - of course has a sophisticated TTS solution, which they call Polly. I love the name :). To be where they are now, they have acquired a startup called Ivona already back in 2013, which were back then producing state of the art TTS solutions. Having tried it I liked the soft tone and the fluency of the results. Have a check yourself:

Apple Siri

Apple offers TTS as part of their iOS SDK in the name of SikiKit. I haven’t had the chance yet to play in depth with it. Wanting to try it out I made the error to think that apples TTS solution on the Desktop is the same as SiriKit. Yet SiriKit is nothing like the built in TTS on the MacOS. To have a bit of a laugh on your Macbook you can do a really poor TTS in the command line you can simply use a command:

say -v fred "Ofen auf 220 Grad vorheizen. Broccoli mit dem Strunk in ca. 1 1/2 cm dicke Scheiben schneiden, auf einem mit Backpapier belegten Blech verteilen. Öl darüberträufeln, salzen.
Backen: ca. 15 Min. in der Mitte des Ofens."

While the output sounds awful, below is the same text read by Siri on the newest iOS 11.3. That shows you how far TTS systems have evolved in the last years. Sorry for the bad quality but somehow it seems impossible to turn off the external microphone when recording on an IPhone.

IBM Watson

In this arms race IBM also offers a TTS system, with a way to also define the prosody manually, using the SSML markup language standard. I didn't like their output in comparison to the presented alternatives, since it sounded quite artificial in comparison. But give it a try for yourself.

Other commercial solutions

Finally there are also competitors beyond the obvious ones such as Nuance (formerly Scansoft - originating from Xerox research). Despite their page promising a lot, I found the quality of the TTS in German to be a bit lacking.

Facebook doesn't offer a TTS solution, yet - maybe they have rather put their bets on Virtual Reality instead. Other notable solutions are Acapella, Innoetics, TomWeber Software, Aristech and Slowsoft for Swiss TTS.

OpenSource

Instead of providing the same kind of overview for the open source area, I think it's easier to list a few projects and provide a sample of the synthesis. Many of these projects are academic in nature, and often don't give you all the bells and whistles and fancy APIs like the commercial products, but with some dedication could definitely work if you put your mind to it.

• Espeak. sample - My personal favorite.
• Festival a project from the CMU university, focused on portability. No sample.
• Mary. From the german "Forschungszentrum für Künstliche Intelligenz" DKFI. sample
• Mbrola from the University of Mons sample
• Simple4All - a EU funded Project. sample
• Mycroft. More of an open source assistant, but runs on the Raspberry Pi.
• Mimic. Only the TTS from the Mycroft project. No sample available.
• Mozilla has published over 500 hours of material in their common voice project. Based on this data they offer a deep learning ASR project Deep Speech. Hopefully they will offer TTS based on this data too someday.
• Char2Wav from the University of Montreal (who btw. maintain the theano library). sample

Overall my feeling is that unfortunately most of the open source systems have not yet caught up with the commercial versions. I can only speculate about the reasons, as it might take a significant amount of good raw audio data to produce comparable results and a lot of fine tuning on the final model for each language. For an elaborate overview of all TTS systems, especially the ones that work in German, I highly recommend to check out the extensive list that Felix Burkhardt from the Technical University of Berlin has compiled.

That sums up the market overview of commercial and open source solutions. Overall I was quite amazed how fluent some of these solutions sounded and think the technology is ready to really change how we interact with computers. Stay tuned for the next blog post where I will explain how we put one of these solutions to use to create a hands free recipe reading assistant.

These days there are a myriad of possible IoT devices, among the most popular are the Arduino, Raspberry Pi, ESP8266 or the Spark/Particle (see this blogpost which gives you a nice overview of the different models). For me the Particle Photon was a great choice because it is cheap (19 USD), has an online IDE (see screenshot below), and works mostly out of the box. The sensor for the project can be bought online at the maker-show.ch for about CHF 19. If you buy them in bulk from a chinese retailer, you can cut the cost probably down to a few bucks per sensor.

Step 1: Making the IoT Device

When you bought one of these starter kits, those Particle Photons come with such a wiring board where you can plug in the cables easily. If not you can buy one anywhere for a CHF 1-3. I followed this blog post about connecting the sensor. That worked great, only don't make the same mistake as I did to put two wires at the same “height” – they should be set off one pin because otherwise you will create a short circuit. (Btw. from my experience a short circuit apparently luckily doesn't kill the photon, but forces it to turn itsself off.)

What I did additionally is I've connected it to a normal USB powerbank and I've put the whole thing it into a little lunch box where I have drilled out the holes for the sensor so I can put it somewhere and it doesn't break that easily.

Step 2: Writing the Firmware

Ususally those IoT devices work in such a way that you write a piece of program that runs on the device. This piece of software or simply firmware is responsible for handling the input form the sensor and sending it somewhere over the internet. The photon devices come with a so called tinker firmware which lets you use your mobile phone to turn certain bits on and off. Its nice to start with it, but for this sensor we can't just turn it on and then see something change on the phone. It is because this sensor needs to constantly send out a signal (echo) every couple of microseconds and then listen for it. So we are going to replace this firmware with our own firmware that is going to take the signal of the HC-SR04 sensor process it and send it to the server.

Now luckily, on the blog post above the author has also already provided some nice firmware code that works well with this sensor. All firmware code is written in C++, but don't worry its quite easy to understand. Lets have a look a some points:

The actual ping

So what we are doing here is sending a signal every 10 ms on the trigger_pin and then are listening on the echo_pin for the signal to come back. It basically works like a echolot bat, send out something and the faster the signal comes back, the closer you are to the sensor.

    digitalWriteFast(trig_pin, HIGH);
    delayMicroseconds(10);
    digitalWriteFast(trig_pin, LOW);
    duration = pulseIn(echo_pin, HIGH);

The setup

What is nice about the particle devices is that you can connect those to your computer via USB and then download the offline ATOM IDE. In this IDE you can select the USB port to listen to (see screenshot below). You can then run your little firmware and make it output stuff to the console. Like this you can debug your tiny device by sending signals to the console and having them displayed on your computer – thats why we need to open this serial port there in setup.

    Serial.begin(115200);
    Particle.variable("cm", cm);
    Particle.variable("human_count", &human_count, INT)

The REST API

The second cool thing about the particle is that any variable that you use in the code can be made accessible online via a REST API. This makes it insanely easy to access data from your device; might it be for debugging reasons or for actually interacting with the device. For this you need to define which variables you want to make accessible in the setup method. So for example the variable above “human_count” can be made accessible online via https://api.spark.io/v1/devices/240034001147343339383037/human_count?access_token=abcde1234. (Notice that this is just a dummy link, I've replaced the access token with random chars)

The main loop

The main loop basically does the following, it sends out a signal every 10ms and once the measured distance is below a certain threshold, we say we see a human. As long as this distance stays under that threshold we say we are still tracking the same person. This might occur when someone is standing in front of the device. We don't want to count this person multiple times but just once. Additionally we have some timeouts, e.g. saying how long should a minimal distance be, between seeing two humans. In the end the sensor at in the loop says ok lets count this person. We output some nice debugging information, set up human_count +1 and most important of all publish this variable as a stream to the net.

          human_count++;
          Serial.printf("Humans %2d:", human_count);
          Particle.publish("humancount", String(human_count));

The stream

Now the last point is the most interesting one. In order to have a nice real time application we could potentially simply query the REST API constantly and ask how many persons we have counted so far. This is doable, but its slow, not actually in realtime and very expensive for the server. Instead we will create a stream of events that is published to the net and consumed by our server. The server can listen to this event, and whenever a new person comes along we can update our widget. You can actually curl this stream via for example: curl “https://api.spark.io/v1/devices/240034001147343339383037/human_count?access_token=abcde1234”

and see those events fireing in realtime. Or you can go the really nice particle dashboard that the particle people have already build and have a look there. It looks like this:

So bascially you are done. You have a little device that is counting, when a person walks by, and it is both making the total count accessible via REST API and is producing a stream of events that can be consumed by our server. The only thing left is to build tiny little dashboard. So we did that too.

Step 3: The dashboard

The dashboard is a tiny node application with a d3.js widget in the frontend. The backend is served by the node application and in the frontend we have an event listener that is bound to our stream of events that we are receiving from the photon. Now normally you woundn't want to build it this way, because you would want to have some sort of database or buffer in between. This buffer would log these events and make it easy to aggregate and query those. For that you might use google pubsub or the opensource kafka alternative. Both are fine, but for the sake of brevity we won't go into details how to publish and subscribe to those services and how to save it in the database. Lets save it for another blogbost.

function connect() {
    console.log('connecting')
    var deviceID = "12345";
    var accessToken = "abcdefg";
    var eventSource = new EventSource("https://api.spark.io/v1/devices/" + deviceID + "/events/?access_token=" + accessToken);

    eventSource.addEventListener('open', (e) => {
        console.log("Opened!"); },false)

    eventSource.addEventListener('error', (e) => {
        console.log("Errored!"); },false)

    return eventSource
}
...
eventSource.addEventListener('humancount', (e) => {
    let parsedData = JSON.parse(e.data)
    console.log('Received data', parsedData)
    pplCount.innerHTML = parsedData.data
    displayChart.newPoint(0.3)
}, false)

So what we see above, is that we simply create an event source that connects to our stream and an event listener that adds another datapoint to our widget once someone walks by. Thats basically it. You migh want to checkout the project code on github and leave a little star if you like it. Below you can see our little widget in action. Notice the small little red spikes that occur when actual people walking by.

Our widget in action. Notice the red bars represent people walking by in real-time.

Where to go from here?

Well for starters if you were to aggregate this data over minutes, weeks or hours, you would actually get a nice chart of how frequented our steps are and on what weekdays people are walking them the most. For that we would want to save our events to a database and query it with a kibana dashboard or use a prebuilt IoT infrastructure for it like ubidots project.

We could improve our battery life by just collecting the datapoints and not constantly sending them over wifi, because this is draining our little battery pack quite fast. But none the less our experiments have shown that with a small lipstick battery pack this device can run for up to 24 hours. So this might be enough in order to deploy it in a one time measurement scenario, like in a shop to measure how many customers are walking by certain isles for example.

On the other hand you might want to deploy those devices with a fixed power source and monitor data constantly to acualy learn something about the seasonality of the data, or use it in a completely differen way.

We were thinking to connect those devices to PIR sensors and placing them into our meeting rooms. Like this we could have smart meeting rooms that actually know if there are persons in them or not. Based on that we might discover that often a meeting room looks booked in the calendar but is actually empty. But that is material for another project.

There is btw. a great ressource of photon tutorials out there, if you want to build more things.

I hope you enjoyed this little IoT experiment and have found some motivation to get started yourself.

Cheers

Thomas Ebermann and Lukasz Gintowt

Forget about “new”

The first thing that confuses developer that really start developing with javascript is the new keyword. “ah ok, like in PHP or Java, I can create instances of Car with new Car(), easy”. I won't dive into [history details] here, but you have to know that new was only introduced to let javascript gain more popularity. new is syntactic sugar, but hides the real prototypal nature of the language.

Check the following constructor pattern in javascript:

function Rectangle(width, height) {
  this.height = height;
  this.width = width;
}

Rectangle.prototype.area = function () {
  return this.width * this.height;
};

var rect = new Rectangle(5, 10);
alert(rect.area());

It looks not so bad. Well I have to use this prototype keyword there, but ok, if it works like that. Let's define a square now, which is kind of a rectangle, so it extends our Rectangle class:

function Square(side) {
  return Rectangle.call(this, side, side);
}

Square.prototype = new Rectangle(); // yes, no arguments
Square.prototype.constructor = Square; // fix, otherwise it points to Rectangle

var sq = new Square(5);
alert(sq.area());

It become quite complicated to explain what we are doing here, we have to play a lot with this 'prototype' property, and this is not intuitive at all.

'Object.create' is your friend

Forget the constructor stuff, we have to think differently, with objects, and you will eventually find out that there is not much to know to build whatever you want: in javascript, objects inherit from other objects, and that's what you do when you want a new instance inheriting from another: create a new object 'B' and tell it to inherit this other object 'A'. All the properties of 'A' will be available in the new object, and you can override them if you want to.
Let's take our previous example with Rectangle and Square and rewrite it:

var Rectangle = { // just an object
  width: 0,
  height: 0,
  area: function() {
    return this.width * this.height;
  }
};

var rect1 = Object.create(Rectangle); // create a new object, extend it from Rectangle
rect1.width = 10;
rect1.height = 5;
alert(rect1.area());

Isn't that much clearer? Object.create is a native javascript function in ECMAScript 5, compatible in all major browsers (there is a polyfill for old browsers). The argument it takes is the object we want to extend (it can take a second one but I won't talk about it in this post). The way to define width and height is not so nice though, let's create kind of a constructor:

var Rectangle = {
  create: function(width, height) {
    var newObj = Object.create(this); // create a new object based on itself
    newObj.width = width;
    newObj.height = height;

    return newObj;
  },
  area: function() {
    return this.width * this.height;
  }
};

var rect1 = Rectangle.create(10, 5);
alert(rect1.area());

create is kind of a factory (avoid the term constructor now), and what's nice is that it's integrated in the object itself.

Let's create the square now:

var Square = Object.create(Rectangle); // extend Rectangle
Square.create = function(side) {
  // take the create function of Rectangle, or do something totally different
  return Rectangle.create(side, side);
}

var sq = Square.create(10);
alert(sq.area());

This is so more intuitiv: Square is a base object, once defined you don't want to touch it (except if you want to modify all square instances). Then you create an instance of it by calling Square.create. Note that we didn't have to use the prototype keyword to achieve inheritance.

What's behind

Let's have a look at the sq object in the console:

We can see that the properties of Rectangle are in __proto__, and width and height are own properties of the object. If the object doesn't redefine its properties, the javascript engine will look for them in __proto__, and if it doesn't find them, will dive deeper into __proto__ of __proto__, until it finds it (or an exception is thrown). So by nature, javascript has this inheritance chaining built-in, and there lies all the power of the language.

By the way you maybe also noticed that the very last __proto__ contains some basic functions: all objects in javascript inherit from Object and therefore have these native functions like valueOf or toString.

Beware of references

Because the objects extend other objects, you have to keep in mind that some properties of your object are in __proto__, and therefore were not copied when you created your instance. This can lead to unexpected behaviour if you forget about it, let's see an example:

var CarOption = {
  create: function(name, price) {
    var newOption = Object.create(this);
    newOption.name = name;
    newOption.price = price;

    return newOption;
  }
};

var Car = {
  type: 'car',
  options: [],
  addOption: function(option) {
    this.options.push(option);
  }
};

var leatherOption = CarOption.create('leather', 200);
var metallicOption = CarOption.create('metallic paint', 600);

var luxuriousCar = Object.create(Car);
luxuriousCar.addOption(leatherOption);
luxuriousCar.addOption(metallicOption);
console.log(luxuriousCar.options.length); // 2 options, ok

var poorCar = Object.create(Car);
console.log(poorCar.options.length); // 2 options too!

poorCar has 2 options too because both instances have their options property pointing to the base Car.options property. To fix it, you have to create an options array whenever you create a car instance. Let's add a create function to do this:

var Car = {
  type: 'car',
  create: function() {
    var newCar = Object.create(this);
    newCar.options = []; // give it its own options array

    return newCar;
  },
  addOption: function(option) {
    this.options.push(option);
  }
};

var luxuriousCar = Car.create(Car); // use our new factory
luxuriousCar.addOption(leatherOption);
console.log(luxuriousCar.options.length); // 1 option, ok

var poorCar = Car.create();
console.log(poorCar.options.length); // 0 options, ok

Having properties in common can be very useful though, if I change the type property of Car, all the instances will get the new type.

Conclusion

Javascript requires another way of thinking, where Objects are first-citizen and can construct or be constructed easily, without the need for a new keyword.

Create a base object, extend from it, extend from others, change its behaviour, the language let you do nearly everything, leading to a lot of flexibility, but also to multiple way of doing things (which is not always good for developer, often looking for the right way to do what they intend to do).

Of course new has sometimes legitimate uses, and does not really cause problems if you don't need inheritance. But are you really wanting to use it now that you read this?