At Liip we have the great opportunity to experiment in new technologies when working on innovation projects. Within the last two years we developed Houston: a VOIP communication system for buses. The system includes an Android app which takes VOIP calls and collects sensor information like the GPS position. When I heard about the release of Android Things, I thought it’d be interesting to investigate if we could use it to make the app run on IoT devices like the Raspberry PI. Therefore I experimented with the platform for a few days. And I discovered an amazing operating system bringing all the benefits of Android to IoT devices.

How to start

The best thing about Android Things is that it is built for Android developers and comes with standard development tools, the ones you are used to. You will develop inside Android Studio. And being able to debug code using adb, like for a normal Android device, is great. Android Things extends the core Android framework with additional APIs provided by the Things Support Library (reference, platform differences). That means it will be easy to use the Android API to develop a user interface, communicate with a server, etc. It is also possible to use Android libraries like Google Firebase (real time datababe, push notifications, ...) or Tensorflow (machine learning).

To start developing for Android Things, you need a development board. Google supports different hardware among the well-known Raspberry Pi 3 Model B. In 2017, more boards where supported, such as the Intel Edison, but it seems they stopped supporting them (at least officially, list of supported platforms).

If you don’t have a Raspberry Pi 3 already, you can buy a starter kit like this one or this one. You will get a Raspberry Pi 3 with a power supply with the first kit. Furthermore there will be a SD card with the Android Things OS flashed and the Rainbow Hat. The hat is easy to plug on the Raspberry and contains a buffet of sensors (temperature, pressure, etc.), LEDs, alphanumeric display, etc.. To explore the platform, I wanted to be able to do my own electronic schemes. Therefore I bought the Raspberry itself and this kit that contains a breadboard (thin plastic board used to hold electronic components) with cables, resistors, LEDs and few sensors. Something you shouldn’t forget is buying a cable which allows you to connect the breadboard to the GPIO connector on the Raspberry (like this one).

If you want to use Android Things with a specific sensor or peripheral, be aware that usage without a driver is impossible. Therefore it is better to look at the list of available drivers first (available here, here and here). It is also possible to write a driver yourself. You’ll find a tutorial here. Once you have a Raspberry PI, it is easy to get Android Things running on it. You can follow this tutorial to flash the OS on a Micro SD card and connect the hardware.

Prototype 1: Weather station

The first prototype I built is a small weather station. The temperature in the office was high and my idea was to measure the temperature of the room. Furthermore I wanted to light up a led if the temperature reaches a given threshold. A next step would be to switch on a fan for example .

To build the electronic circuit on the development breadboard, you’ll need a bit of electronic knowledge. This feels like becoming an electronic engineer, which is great! Reading a tutorial is great as a memory refresher. This one was very helpful to me.

One import thing to remember is Ohm’s Law U = RI , to know which resistance you have to put between the Raspberry PI and the LED. The resistance needs to respect the intensity of the LED that switches on automatically . It was also not easy to understand how the Peripheral I/O interface works at the beginning (where is the ground (-), power (+), functionality of the different pins, etc.). I printed the scheme here to have it next to me at all times.

General Purpose Input/Output (GPIO) provides a programmable interface to read the state of a binary input device or control the on/off state of a binary output device (such as an LED). The red LED was connected on pin BCM6 and the green LED was connected on pin BMC 5, for example. To get the temperature via the sensor and display it on the alphanumeric display, I used BCM2 and BMC3 (inter-integrated circuit bus).

The steps to create the Android Things app are the following:

1. Create a Android project with a blank Activity in Android Studio and add the Android Things library and drivers in build.gradle

provided ‘com.google.android.things:androidthings:<version>'
implementation 'com.google.android.things.contrib:driver-bmx280:<version>'
implementation 'com.google.android.things.contrib:driver-ht16k33:<version>'

2. Add the following line in the AndroidManifest.xml under the application tag

<uses-library android:name="com.google.android.things"/>

3. In OnCreate of the MainActivity, register the peripherals

val service = PeripheralManagerService()
mSensorManager = getSystemService(Context.SENSOR_SERVICE) as SensorManager

//Temperature sensor
try {
    mEnvironmentalSensorDriver = Bmx280SensorDriver("I2C1")
    mSensorManager.registerDynamicSensorCallback(mDynamicSensorCallback)
    mEnvironmentalSensorDriver.registerTemperatureSensor()
    mEnvironmentalSensorDriver.registerPressureSensor()
    Log.d(TAG, "Initialized I2C BMP280")
} catch (e: IOException) {
    throw RuntimeException("Error initializing BMP280", e)
}

//Alphanumeric display
try {
    mDisplay = AlphanumericDisplay("I2C2")
    mDisplay.setEnabled(true)
    mDisplay.clear()
    Log.d(TAG, "Initialized I2C Display")
} catch (e: IOException) {
    Log.e(TAG, "Error initializing display", e)
    Log.d(TAG, "Display disabled")
    mDisplay = null
}

//Green LED
try {
    mGreenLedGpio = service.openGpio("BCM5")
    mGreenLedGpio.setDirection(Gpio.DIRECTION_OUT_INITIALLY_LOW)

} catch (e: IOException) {
    throw RuntimeException("Problem connecting to IO Port", e)
}

//Red LED
try {
    mRedLedGpio = service.openGpio("BCM6")
    mRedLedGpio.setDirection(Gpio.DIRECTION_OUT_INITIALLY_LOW)
} catch (e: IOException) {
    throw RuntimeException("Problem connecting to IO Port", e)
}

4. And finally register for sensor events:

private val mTemperatureListener = object : SensorEventListener {
    override fun onSensorChanged(event: SensorEvent) {

        mTempCount++
        if (mTempCount % 100 !== 0) {
            return
        }

        val temperature = Temperature(System.currentTimeMillis(), event.values[0])
        Log.i(FragmentActivity.TAG, "Temperature: " + temperature.getTemperature())

        mTemperatureTexView.setText(String.valueOf(temperature.getTemperature()))
        updateDisplay(temperature.getTemperature())

        try {
            if (temperature.getTemperature() >= TEMPERATURE_THRESHOLD) {
                mRedLedGpio.setValue(true)
                mGreenLedGpio.setValue(false)
            } else {
                mRedLedGpio.setValue(false)
                mGreenLedGpio.setValue(true)
            }
        } catch (e: IOException) {
            Log.e(TAG, "Error", e)
        }
    }

    override fun onAccuracyChanged(sensor: Sensor, accuracy: Int) {
        Log.d(TAG, "accuracy changed: $accuracy")
    }
}

private fun updateDisplay(value: Float) {
    if (mDisplay != null) {
        try {
            mDisplay.display(value)
        } catch (e: IOException) {
            Log.e(TAG, "Error setting display", e)
        }
    }
}

In the onSensorChanged callback for the temperature sensor, we filter events (otherwise we receive too many temperature changes) and display the value on the display with updateDisplay() . Once the value reached a certain threshold (TEMPERATURE_THRESHOLD), we switch the red or the green LED with setValue(true)/setValue(false).

And that’s pretty much it. Thanks to the Android framework and the implemented drivers it is very simple to communicate with the Raspberry PI using simple interfaces and without the need for low level programming.

Prototype 2: GPS tracker

In the next prototype, I wanted to test Android Things with a GPS antenna. If we come back to the Houston project, the idea would be to have an embedded device that is able to track the positions of the buses.

It was laborious to find out which GPS sensor to buy. There are many models so I decided to use the Grove GPS with GrovePi+. GrovePi+ offers a plug-and-play module for the Raspberry PI. You can connect Grove sensors without cables and breadboards.

It is possible to use the Grove Pi+ with Android Things, but I am not sure if it will work with all Grove sensors. I guess it depends on the connection interface you have (I2C, UART, …) and if there is a driver available for it. In my case, I needed a UART connection and it was enough to connect the Grove GPS to the RPISER port of the GrovePi+.

To create the Android Things app, the steps are the following:

1. Create a Android project with a blank Activity in Android Studio and add the Android Things library and drivers in build.gradle

provided ‘com.google.android.things:androidthings:<version>'
implementation 'com.google.android.things.contrib:driver-gps:<version>'

2. Add the following line in the AndroidManifest.xml under the application tag

<uses-library android:name="com.google.android.things"/>

3. In onCreate of the MainActivity, register the GPS sensor

// Callback when SensorManager delivers temperature data.
val I2C_BUS = "I2C1"
val UART_BUS = "UART0"
val UART_BAUD = 9600
val ACCURACY = 2.5f // From GPS datasheet

...

mLocationManager = getSystemService(Context.LOCATION_SERVICE) as LocationManager

// We need permission to get location updates
if (checkSelfPermission(Manifest.permission.ACCESS_FINE_LOCATION) != PackageManager.PERMISSION_GRANTED) {
    // A problem occurred auto-granting the permission
    Log.e(TAG, "No ACCESS_FINE_LOCATION permission")
    return
}

try {
    // Register the GPS driver
    mGpsDriver = NmeaGpsDriver(this, UART_BUS, UART_BAUD, ACCURACY)
    mGpsDriver.register()

} catch (e: Exception) {
    Log.e(TAG, "Unable to open GPS UART", e)
}

4. Then, register the listener to receive location updates, just like for a standard Android code base.

 ...

// Register for location updates
mLocationManager.requestLocationUpdates(LocationManager.GPS_PROVIDER, 2, 0, mLocationListener)

...

val mLocationListener = object : LocationListener {
    override fun onLocationChanged(location: Location) {
        Log.v(TAG, "Location update: $location")
    }

    override fun onStatusChanged(provider: String, status: Int, extras: Bundle) {}
    override fun onProviderEnabled(provider: String) {}
    override fun onProviderDisabled(provider: String) {}
}

When you are running the app, you will see the current locations displayed in logcat. We can imagine to collect the locations in a Firebase Firestore database or display them on a map. We have seen, that it is very simple to build an Android Things prototype. As long as you have a bit of Android development experience and the correct driver and hardware setup, you’ll be okay.

Next steps

The goal of this innovation project was to experiment with the Android Things platform and discover if we could use it in our projects. It was great fun to build the electronic circuits and interact with the hardware via the Android framework. The setup is simple and the framework allows us to easily build a lot of amazing apps for IoT devices. You’ll find a lot of interesting use cases on the Web such as controlling you home switches remotely. Or even funnier examples like building an high five machine. Google insists on their website that the platform is not meant just for quick prototyping, but also for real products. Android Things is definitely a platform that was worth a try. The next step is to start integrating some of the Android code bases we have developed for Houston or other projects. We have a lot of ideas already!

The major components

So how's Houston working? Here is an overview of the major components. Houston is a distributed system composed of:

• Two Mobile Apps that fulfill the specific need of bus drivers and team leaders
• A Web Client used by the operators to create any type of call
• A Cloud Communication Platform to establish Voice over IP (VoIP) and standard phone call over the Public Switch Telephone Network (PSTN) without having to deal with the complexity of building and maintaining a communication infrastructure. The cloud platform used in this system is Twilio (more details on that later)
• A Public API : implemented by a backend server used as a bridge between mobile apps, web clients and Twilio. It is connected to a database that stores user identities and bus planning. The database also keeps traces of all performed calls.

A deeper look inside each component of the system and how each of them interacts together will help to understand why Houston was a huge challenge.

Mobile Apps

Each of the two mobile apps provide specific features for busses and team leaders. There is no specificities in how the app is built. The apps are a set of views that allow to perform calls by communicating with the public API using standard HTTPS requests and JSON formatted messages.

The real challenges were the following:

• How to allow drivers to use an app without losing eyesight of the road?
• How to integrate the app with the existing busses microphone and loudspeaker?
• How to manage and upgrade a set of more than 200 devices remotely, as they are installed in busses dispatched in different locations?

How to tackle the usability problem?

To tackle the usability problem, we organised workshops with the end-users (e.g. drivers) to find out the best way to interact with the app without having to look at the screen. The solution found, was to implement a carousel screen with voice feedback at each screen change.

The integration with the existing microphone and loudspeaker has been solved with the installation of a bluetooth amplifier. The big problematic here was to ensure that the app was automatically connected with the amplifier at any time.

Finally, we decided to use Android4Work and MobileIron to manage the 200 devices distantly.

About the importance of user testing

We started with a prototype running on a server and on real devices. It was working like a perfectly in our offices. However, we should have installed the busses applications in real busses and let final users try it in their daily business. Even though we made many workshops with end users, a sketch on paper doesn't replace the real experience.

Deploying the prototype in the real environment would have save a lot of time! In fact, we ended up implementing some features that have not been used as much as we initially thought.

Another important lesson learned by our development team of mobile developers is that working with physical devices such as microphones and loud speakers is not trivial. We underestimated the impact of the environment on the device. Indeed, we had quite a bad surprise when we discovered after months of development that the volume in the busses was fare too low. Again, making a prototype working in a real environment would have helped.

Web Client

The web client is a single page application built using backbone JS. The challenge of the web client development was less in its technical aspects than in the definition of the features needed by the operators. In fact, it was difficult to make a system working similarly as a radio in a web browser.

To help operators understand what was happening as they created calls, we decided to display as much information about the state of the call as possible. For example, operators know – from the start of the call – which users are included in it as well as their current state.

Twilio

Twilio is a cloud communication platform offering an elegant way to perform any type of call such as a telephone conference. It offers a RESTful API that completely hides the complexity of the infrastructure behind. Thanks to Twilio, the Houston public API offers the possibility to create conferences between more than 50 users in only one request.

Twilio API offers tons of features. The most used one in Houston are the followings:

• Getting real time information about ongoing and past calls.
• Muting / Unmuting users
• Adding / Removing users to an ongoing conference

But this is not all, Twilio can register calls and offers a nice administration web client allowing to retrieve all records as well as the detailed logs of all performed calls. In addition to that API, Twilio offers libraries for web clients and mobile applications. This helped us a lot to handle incoming/outgoing calls.

Heroku

The public API offers a standardised set of endpoints to serve requests from the mobile applications and from the web client. We hosted it on Heroku in order to get rid of the hosting complexity.

Heroku allowed us to easily scale when the amount of user of the system raised. With Heroku It has been a piece of cake to integrate our system with Twilio and It solved lots of security issues.

Conclusion

Lessons learned during this project are numerous. The most important to remember is that innovation requires to start with a simple feature and to make it work in the final environment. User workshops and controlled environment testing is not enough. It is essential to test prototypes in a real environment with real users.

Finally, I think that what had most impact on the success of the project is the way we worked hand in hand with the client. From the beginning, we collaborated as one team. We were able to share challenges and issues. We did not hide problems, but talked about them openly to find the best solution together.

What's next

Hundreds of people use daily Houston. Of course there is still many improvements to bring. Today, we are still working on the project by analysing its use in order to improve it by bringing additional or improved features to make this system even easier and powerful to the end users.

Once the system will be stable enough, the door to many developments opportunities will open. Other sectors might be interested to implement similar systems. The good thing is that our team has become very experienced with these topics.

Finally, our team was delighted to be awarded 4 times in 2016 for this awesome project : 2 awards at the Best of Swiss Apps 2016 (“Silver” in Enterprise category and “Gold” in Innovation category) as well as 2 award at the Meilleur du Web 2016 (Winner in UX and in Technology Categories). These awards will just continue to fuel our appetite for more innovation and success! :)

With the project Houston, we took the challenge of replacing the old radio network of the Transports Publics Fribourgeois (TPF), a swiss public transportation company by a system using existing data network and running on mobile applications. This solution solved the problem of maintaining a dedicated radio network. It also improved both the global quality of the communication and the availability of the system.

Initial situation: communication based on radio system

Since decades, employees of the Transports Publics Fribourgeois (TPF) have been using standard radio to communicate between them. The radio system is meant to cover the needs of the users. It is spread over more than 200 busses, 30 team leaders and the operation center). There are three types of users, with specific needs:

• The operators – working in the operation center – use the radio to speak to a specific bus driver, or to broadcast messages to all or part of the running busses.
• The team leaders are dispatched at different locations. They use the radio to manage daily events – such as the replacement of a driver – or to inform many drivers of a change in the network – for example in case of an accident.
• The bus drivers use the bus radio as the main means of communication while driving. They can call other busses, the team leaders or the operation center.

The project: replace the old radio system

The radio system used by the employees of the TPF had some drawbacks :

• The poor quality of the communication,
• The low coverage of the network (about 60% of the territory. This is because it is too hard to cover the all territory since busses drive all over the countryside),
• The exploitation and maintenance cost of the a radio network were expensive (antennas were installed in strategic locations, all over the canton).

The goal of the project Houston was to improve the former system by replacing the radio communication system.

The challenge was to provide at least all already existing features and possibly add some new one enabled by the geo localisation of the busses. The possibility to call all busses currently in a given region was one of the new requested feature. 

Screenshot of the app Houston

The solution: a mobile application

A mobile app would allow the system to modern technologies. With a mobile app, we also expected to drastically reduce the exploitation cost. Indeed, with a mobile app, it is not necessary to install and maintain radio antennas all over the Fribourg region.

We decided to create a mobile app, because they are easy to develop in comparison with dedicated embedded devices. The cost of 200 smartphones is more than justified by the flexibility and development cost that this solution offers.

We started by building trust

The way we worked hand in hand with the client had a huge impact on the success of the project. From the beginning, we asked all stakeholders and end users to explain us how they worked together with the former system. We asked what the strengths and the weaknesses of the system were. We asked them how they would like to use it.

As a result, we had a good idea of what the main features to implement were. Most importantly, we also gained the trust of the end users.

The fact that there was such a level of trust, was a good basis to develop the project. We were not hiding problems, but we were exposing them directly to find the best solution possible together. This collaboration mindset helped a lot during the development of the project.

Additional Information

Best of Swiss App 2016: Houston wins Silver in the «Enterprise» category and Gold in the «Innovation» category (read here)

HOUSTON: The first VoIP Communication System for Swiss Public Transportation (read here)

Meilleur du Web 2016: HOUSTON awarded in the category UX and Technology (read here)

First of all, it is important to carefully chose between a web and a mobile application. Mobile apps are useful when they use one of the device's native capabilities (e.g. Bluetooth, GPS, camera, etc.), and when they enhance the experience provided to the user (compared to what a web application could do).

Industrial challenges

The challenges industries face are often very good candidates for mobile apps as they leverage all their potential. Therefore, I have identified five recurring issues that industries are facing nowadays. These pain points can be easily relieved by mobile applications.

Challenge 1: Productivity issues due to technical limitations

In some industries, technology gets old and sometimes stands in the way of employees' productivity. Our project with Schindler is a good example: technicians had to go inside the elevator shaft, detect the default, take note of the spare part reference, go back to his vehicle or office to check the catalog, to finally be able to identify the issue. Now, technicians can access the catalog directly in front of the problematic spare part, and shorten its identification by 30%. It increases process speed, elevators are fixed quicker. As a result, the users are happier, and it is profitable for the business.

Time for an industrial digitalization?

Challenge 2: Reduce existing operating costs

In the recent project implementation of HOUSTON, we saw how mobile technology could help to reduce drastically the operating costs of a client's existing radio infrastructure. With the use of new technologies such as VoIP (via the  Twilio service) coupled with mobile apps, we managed to decrease their total maintenance costs by two.

This kind of Return On Investment can auto-finance the project thanks to the realized savings. The funniest in this case is that the true ROI comes afterwards when the employees enjoy a better User Experience.

Challenge 3: Inflexibility due to stationary machines

Similar situations occurred for several industrial audits we performed. For instance, our watchmaker production line, where we implemented a web application based on an API in order to satisfy the brand's IT and security regulations.

We are convinced that the next step of their digital transformation is: go mobile. Drop the static machines, allow people to prepare their procedures and actions on the mobile app, and let them connect to the manufacturing production line via Bluetooth to execute the commands needed there. This way, employees wouldn't be tied anymore to a location. Way more convenient and flexible.

When a production line goes mobile, employees will save on-site time, as all their supply chain procedures are prepared in advance on their mobile app.

Challenge 4: Slow reaction time to alerts that generate high risks

Most of the industries need real time monitoring to ensure that supply chain risk levels are kept to their minimum. This implies costly equipments as well as human resources to track possible failures and malfunctions. Thanks to push notifications and other real time capabilities, mobile apps can both increase reaction time with their timely alerts, and decrease monitoring costs.

Challenge 5: Information overflow that leads to misinformed decisions

In older systems, information is often spread without the capacity to target users based on their profile or location. As people are spammed, they take decisions that are misinformed as they can't keep up with the load of data received.

Back to the HOUSTON example: mobile app built in GPS capability enables buses' agents to be geolocated. Thus, messages can be geo-targeted to provide the relevant information to the right employee. No complex locating system needs to be setup.

To conclude: Carefully choose how to invest

Mobile apps can solve specific industrial problems. By experience, we have seen that the best mobile app solutions are the ones who answer one need or problem, by leveraging the native devices capabilities. This kind of app that solve a problem in the employees' everyday life is what make them worth investing in.

It is necessary to take time to analyze your issues in order to define the best possible options to reach your goals. Appropriate workshops will ensure that you make the right investment.

We are happy to hear your existing challenges in the comments section below (or simply contact us and let's have a coffee), so that we exchange feedbacks on your struggles.

But when we sat down recently with our team, I realized that we indeed use a myriad of different tools, frameworks, and SaaS solutions. I thought it would be useful to organize them in a meaningful data stack. I have not only included the tools we are using, but I sat down and started researching. It turned out into an extensive list aka. the data stack PDF.  This poster will:

• provide an overview of solutions available in the 5 layers (Sources, Processing, Storage, Analysis, Visualization)
• offer you a way to discover new tools and
• offer orientation in a very densely populated area

So without further ado, here is my data stack overview Click to open PDF. Feel free to share it with your friends too.

Liip data stack version 1.0

Click here to get notified by email when I release version 2.0 of the data stack.

Let me lay out some of the questions that guided me in researching each area and throw in my 5 cents while researching each one of them:

• Data Sources: Where does our data usually come from? For us, it's websites with sophisticated event tracking. But for some projects the data has to be scraped, comes from social media outlets or comes from IoT devices.
• Data Processing: How can we initially clean or transform the data? How and where can we store the logs that those events create? Also from where do we also take additional valuable data?
• Database: What options are out there to store the data? How can we search through it? How can we access big data sources efficiently?
• Analysis: Which stats packages are available to analyze the data? Which frameworks are out there to do machine learning, deep learning, computer vision, natural language processing?
• Visualization, Dashboards, and Applications: What happens with the results? What options do we have to visually communicate them? How do we turn those visualizations into dashboards or whole applications? Which additional ways of communicating with the user beside reports/emails are out there?
• Business Intelligence:

What solutions are out there that try to integrate the data sourcing, data storage, analysis and visualization in one package? What solutions BI solutions are out there for big data? Are there platforms/solutions that offer more of a flexible data-scientist approach?

My observations when compiling the list:

Data Sources

• For scrapers, there are actually quite a lot of open source projects out there that work really well, probably because those are used mostly by developers.
• While there is quite a few software as a service solutions with slightly different focus, capturing website data in most cases is done via google analytics, although Piwik offers a nice on-premise alternative.
• We have been experimenting quite a bit with IoT devices and analytics, and it turns out that there seems to be quite a few integrated data-collection and analysis software as a service solutions out there, although you are always able to to use your own (see later) solutions.
• For social media data, the data comes either from the platforms themselves via an API (which is probably the default for most projects), but there are some convenient data providers out there that allow you to ingest social media data across all platforms.

Data Processing

• While there are excellent open source logging services like graylog or logstash, it can sometimes save a lot of time to use those pricey saas solutions because people have solved all the quirks and tiny problems that open source solutions sometimes have.
• While there are some quite old and mature open source solutions (e.g. RabbitMQ or Kafka) in the message queues or streams category, it turned out that there a lot of new open source  stream analytics solutions (Impala, Flink or Flume) in the market and almost all of the big four (Microsoft, Google, Facebook, Amazon) offer their own approaches.
• The data cleansing or transformation category is quite a mixed bag. While on one hand there are a number of very mature industry standard solutions (e.g. Talend), there are also alternatives for end users that allow them simply to clean their data without any programming knowledge (e.g. Trifacta or Open Refine)

Databases

• Databases: If you haven't followed the development in the databases area closely like me, you might think that solutions will fall either in the SQL (e.g. MySQL) or the NoSQL (e.g. MongoDB) bucket. But apparently a LOT has been going on here, probably among the most notable are the graph based databases (e.g. Neo4J) and the Column Oriented databases (e.g. Hana or Monet DB) that offer a much better performance for BI tasks. There are also some recent experimental highly promising solutions like databases in the GPU (e.g. Mapd) or ones that only sample (e.g. BlinkDB) the whole dataset.
• The distributed big data ecosystem: It is mostly populated by mature projects from the Apache foundation that integrate quite well in the Hadoop ecosystem. Worth mentioning are of course the distributed machine learning solutions for large scale processing like Spark or Mahout, that are really handy. There are also a lot of mature options like Cloudera or Hortonworks that offer out of the box integrations.
• In Memory Databases or Search: Of course you the first thing that comes to mind is elastic(search) that proved over the years to be a reliable solution. Overall the area is populated by quite a lot of stable open source projects (e.g. Lucene or Solr) while on the other hand, you can now directly tap into search as a service (e.g. AzureSearch or Cloudsearch) from the major vendors. The most interesting projects I will try follow are the fastest in-memory database Exasol and its “competitor” VoltDB.

Analysis / ML Frameworks

• Deep Learning Frameworks: Obviously, on one hand, you will find the kind of low-level frameworks like Tensorflow, Torch, and Theano in there. But on the other hand, there are also high-level alternatives that build up upon those like TFlearn (that has been integrated into Tensorflow now) or Keras, which allow you to make progress faster with less coding but also without being able to control all the details. Finally, there are also alternatives to hosting these solutions yourself, in services like the Google ML platform.
• Statistic software packages: While maybe a long time ago you could only choose from commercial solutions like SPSS, Matlab or SAS, nowadays there is really a myriad of open source solutions out there. Whole ecosystems have developed around those languages (python, R, Julia etc.). But also even without programming, you can analyze data quite efficiently with tools like Rapidminer, Orange or Rattle. For me, nothing beats the combination of pandas and an ipython notebook.
• General ML libraries: I put the focus here on mainly the python ecosystem, although the other ones are probably as diverse as this one. With scipy, numpy or scikit-learn we've got a one-stop shop for all your ML needs, but nowadays there are also libraries that take care of the hyperparameter optimization (e.g. REP) or model selection (AutoML). So again here you can also choose your level of immersion yourself.
• Computer vision: While you will find a lot of open source libraries that rely on OpenCV somehow a myriad of awesome SaaS solutions (e.g. Google CV, Microsoft CV) from big vendors have popped up in the last years. These will probably beat everything you might hastily build over the weekend but are going to cost you a bit. The deep learning movement has really made computer vision, object detection etc.. really accessible for anyone.
• Natural language processing: Here I noticed a similar movement. We used NLP libraries to process social media data (e.g. sentimentanalysis) and found that there are really great open source projects or libraries out there. While there are various options for text processing (e.g. natural for node.js, or nltk for python or coreNLP from Stanford), it is deep learning and the SaaS products built upon it that have really made natural language processing available for anyone. I am very impressed with the results of these tools, although I doubt that we will come anywhere close in the next years to computers really understanding us. After all its the holy grail of AI.

Dashboards / Visualization

• Visualization: I was really surprised how many js libraries are out there, that allow you to do the fanciest data visualizations in the browser. I mean its great to have those solid libraries like ggplot or matplotlib, or the fancy ones like bokeh or seaborn but if you want to communicate your results to the user in a periodic way, you will need to go through the mobile / browser. I guess we have to thank the strong D3 community for the great developments in this area, but also there are a lot of awesome: SaaS and open source solutions that go way beyond just visualization like Shiny for R or Redash that feel more like a business intelligence solution.
• Dashboards: I am personally a big fan of dashing.io because it is simply free and it's in ruby, but plotly has really surprised me as a very useful tool to just create a dashboard without a hassle. There is a myriad of SaaS solutions out there that I stumbled upon when researching this field, which I will have to try. I am not sure if they will all hold up to the shiny expectations, that those websites sell.
• Bot Frameworks: Although I think of bots or agents more of a way of interacting with a user, I have put them into the visualization area because they didn't fit in anywhere else. P-Brain.ai and Wit.ai or botpress turn out to be a really fast way to get started here when you just want to build a (slack)-bot. I am however not sure if chatbots will be able to deliver the right results, given the hype with those.

Business Intelligence

• Business Intelligence: I thought I knew more or less the alternatives that are out there. But having researched a bit, boy was I surprised to find how much is actually out there. Basically, every vendor of the big four has a very mature solution out there. Yet I found it really hard to distinguish between the different SaaS solutions out there, maybe it's because of the marketing talk, or maybe because they just all do the same thing. It's interesting to compare how potentially business intelligence solutions are offering the capabilities of the before mentioned data stack, but given the variety of different solutions in each layer, I think more and more people will be tempted to pick and chose instead of buying the expensive all in one solution. There are however open source alternatives, of which some feel quite mature (e.g. Kibana or Metabase) while others are quite small but really useful (e.g. Blazer). Also don't judge me too hard, if I put Tableau in there, some may say it's just a visualization tool, others perceive it as a BI solution – I think the boundaries are really blurry in this terrain.
• BI on Hadoop: I had to introduce this category because I discovered that a lot of solutions are particularly tailored to working on the Hadoop stack. It's great to see that there are options out there and I am eager to explore this terrain in the future.
• Data Science Platforms: What I noticed too is that somehow that data scientists are becoming a target group of integrated business intelligence solutions or data science platforms. I had some experience with BigML and Snowplow before, but it turns out that there is a lot of different platforms popping up, that might make your life much easier. For example, when it comes to deploying your models (like Yhat) or having a totally automated way of learning models (e.g. Datarobot). I am really excited to see what things will pop up here in the future.

What I realized that this task of creating an overview of the different tools and solutions in the data-centric area will never be complete. Even when writing this blog post I had to add 14 more tools to the list. And I am aware of the fact that I might have missed some major tools out there, simply because it's hard to be unbiased when researching.

That is why I created a little email list that you can sign up to, and I will send you the updated version of this stack somewhere this year. So sign up to stay up to date (I promise I will not spam you) and write me a comment to let me know of new solutions or to let me know in the comments how you would have segmented this field or what your favorite tools are.

Click here to get notified by email when I release version 2.0 of the data stack.

These days there are a myriad of possible IoT devices, among the most popular are the Arduino, Raspberry Pi, ESP8266 or the Spark/Particle (see this blogpost which gives you a nice overview of the different models). For me the Particle Photon was a great choice because it is cheap (19 USD), has an online IDE (see screenshot below), and works mostly out of the box. The sensor for the project can be bought online at the maker-show.ch for about CHF 19. If you buy them in bulk from a chinese retailer, you can cut the cost probably down to a few bucks per sensor.

Step 1: Making the IoT Device

When you bought one of these starter kits, those Particle Photons come with such a wiring board where you can plug in the cables easily. If not you can buy one anywhere for a CHF 1-3. I followed this blog post about connecting the sensor. That worked great, only don't make the same mistake as I did to put two wires at the same “height” – they should be set off one pin because otherwise you will create a short circuit. (Btw. from my experience a short circuit apparently luckily doesn't kill the photon, but forces it to turn itsself off.)

What I did additionally is I've connected it to a normal USB powerbank and I've put the whole thing it into a little lunch box where I have drilled out the holes for the sensor so I can put it somewhere and it doesn't break that easily.

Step 2: Writing the Firmware

Ususally those IoT devices work in such a way that you write a piece of program that runs on the device. This piece of software or simply firmware is responsible for handling the input form the sensor and sending it somewhere over the internet. The photon devices come with a so called tinker firmware which lets you use your mobile phone to turn certain bits on and off. Its nice to start with it, but for this sensor we can't just turn it on and then see something change on the phone. It is because this sensor needs to constantly send out a signal (echo) every couple of microseconds and then listen for it. So we are going to replace this firmware with our own firmware that is going to take the signal of the HC-SR04 sensor process it and send it to the server.

Now luckily, on the blog post above the author has also already provided some nice firmware code that works well with this sensor. All firmware code is written in C++, but don't worry its quite easy to understand. Lets have a look a some points:

The actual ping

So what we are doing here is sending a signal every 10 ms on the trigger_pin and then are listening on the echo_pin for the signal to come back. It basically works like a echolot bat, send out something and the faster the signal comes back, the closer you are to the sensor.

    digitalWriteFast(trig_pin, HIGH);
    delayMicroseconds(10);
    digitalWriteFast(trig_pin, LOW);
    duration = pulseIn(echo_pin, HIGH);

The setup

What is nice about the particle devices is that you can connect those to your computer via USB and then download the offline ATOM IDE. In this IDE you can select the USB port to listen to (see screenshot below). You can then run your little firmware and make it output stuff to the console. Like this you can debug your tiny device by sending signals to the console and having them displayed on your computer – thats why we need to open this serial port there in setup.

    Serial.begin(115200);
    Particle.variable("cm", cm);
    Particle.variable("human_count", &human_count, INT)

The REST API

The second cool thing about the particle is that any variable that you use in the code can be made accessible online via a REST API. This makes it insanely easy to access data from your device; might it be for debugging reasons or for actually interacting with the device. For this you need to define which variables you want to make accessible in the setup method. So for example the variable above “human_count” can be made accessible online via https://api.spark.io/v1/devices/240034001147343339383037/human_count?access_token=abcde1234. (Notice that this is just a dummy link, I've replaced the access token with random chars)

The main loop

The main loop basically does the following, it sends out a signal every 10ms and once the measured distance is below a certain threshold, we say we see a human. As long as this distance stays under that threshold we say we are still tracking the same person. This might occur when someone is standing in front of the device. We don't want to count this person multiple times but just once. Additionally we have some timeouts, e.g. saying how long should a minimal distance be, between seeing two humans. In the end the sensor at in the loop says ok lets count this person. We output some nice debugging information, set up human_count +1 and most important of all publish this variable as a stream to the net.

          human_count++;
          Serial.printf("Humans %2d:", human_count);
          Particle.publish("humancount", String(human_count));

The stream

Now the last point is the most interesting one. In order to have a nice real time application we could potentially simply query the REST API constantly and ask how many persons we have counted so far. This is doable, but its slow, not actually in realtime and very expensive for the server. Instead we will create a stream of events that is published to the net and consumed by our server. The server can listen to this event, and whenever a new person comes along we can update our widget. You can actually curl this stream via for example: curl “https://api.spark.io/v1/devices/240034001147343339383037/human_count?access_token=abcde1234”

and see those events fireing in realtime. Or you can go the really nice particle dashboard that the particle people have already build and have a look there. It looks like this:

So bascially you are done. You have a little device that is counting, when a person walks by, and it is both making the total count accessible via REST API and is producing a stream of events that can be consumed by our server. The only thing left is to build tiny little dashboard. So we did that too.

Step 3: The dashboard

The dashboard is a tiny node application with a d3.js widget in the frontend. The backend is served by the node application and in the frontend we have an event listener that is bound to our stream of events that we are receiving from the photon. Now normally you woundn't want to build it this way, because you would want to have some sort of database or buffer in between. This buffer would log these events and make it easy to aggregate and query those. For that you might use google pubsub or the opensource kafka alternative. Both are fine, but for the sake of brevity we won't go into details how to publish and subscribe to those services and how to save it in the database. Lets save it for another blogbost.

function connect() {
    console.log('connecting')
    var deviceID = "12345";
    var accessToken = "abcdefg";
    var eventSource = new EventSource("https://api.spark.io/v1/devices/" + deviceID + "/events/?access_token=" + accessToken);

    eventSource.addEventListener('open', (e) => {
        console.log("Opened!"); },false)

    eventSource.addEventListener('error', (e) => {
        console.log("Errored!"); },false)

    return eventSource
}
...
eventSource.addEventListener('humancount', (e) => {
    let parsedData = JSON.parse(e.data)
    console.log('Received data', parsedData)
    pplCount.innerHTML = parsedData.data
    displayChart.newPoint(0.3)
}, false)

So what we see above, is that we simply create an event source that connects to our stream and an event listener that adds another datapoint to our widget once someone walks by. Thats basically it. You migh want to checkout the project code on github and leave a little star if you like it. Below you can see our little widget in action. Notice the small little red spikes that occur when actual people walking by.

Our widget in action. Notice the red bars represent people walking by in real-time.

Where to go from here?

Well for starters if you were to aggregate this data over minutes, weeks or hours, you would actually get a nice chart of how frequented our steps are and on what weekdays people are walking them the most. For that we would want to save our events to a database and query it with a kibana dashboard or use a prebuilt IoT infrastructure for it like ubidots project.

We could improve our battery life by just collecting the datapoints and not constantly sending them over wifi, because this is draining our little battery pack quite fast. But none the less our experiments have shown that with a small lipstick battery pack this device can run for up to 24 hours. So this might be enough in order to deploy it in a one time measurement scenario, like in a shop to measure how many customers are walking by certain isles for example.

On the other hand you might want to deploy those devices with a fixed power source and monitor data constantly to acualy learn something about the seasonality of the data, or use it in a completely differen way.

We were thinking to connect those devices to PIR sensors and placing them into our meeting rooms. Like this we could have smart meeting rooms that actually know if there are persons in them or not. Based on that we might discover that often a meeting room looks booked in the calendar but is actually empty. But that is material for another project.

There is btw. a great ressource of photon tutorials out there, if you want to build more things.

I hope you enjoyed this little IoT experiment and have found some motivation to get started yourself.

Cheers

Thomas Ebermann and Lukasz Gintowt