The good thing about it was, that every tool I had in mind could be found there somewhere, and like a map I could instantly see to which category it belonged. As a bonus I was able to identify my personal white spots on the map. The bad thing about it was, that as soon as I have compiled the list, it was out of date. So I transferred the collection into a google sheet and whenever a new tool emerged on my horizon I added it there. Since then - in almost a year - I have added over 102 tools to it.

From PDF to Data Science Stack website

While it would be OK to release another PDF of the stack year after year, I thought that might be a better idea to turn this into website, where everybody can add tools to it.
So without further ado I present you the http://datasciencestack.liip.ch page. Its goal is still to provide an orientation like the PDF, but eventually never become stale.

Adding Tools: Adding tools to my google sheet felt a bit lonesome, so I asked others internally to add tools whenever they find new ones too. Finally when moving away from the old google sheet and opening our collection process to everybody I have added a little button on the website that allows everybody to add tools by themselves to the appropriate category. Just send us the name, link and a quick description and we will add it there after a quick sanity check. The goal is to gather user generated input too! The I am thinking also about turning the website into a “github awesome” repository, so that adding tools can be done more in a programmer friendly way.

Search: When entering new tools, I realized that I was not sure if that tool already exists on the page, and since tools are hidden away after the first five the CTRL+F approach didn’t really work. That's why the website now has a little search box to search if a tool is already in our list. If not just add it to the appropriate category.

Mailing List: If you are a busy person and want to stay on top of things, I would not expect you to regularly check back and search for changed entries. This is why I decided to send out a quarterly mailing that contains the new tools we have added since our last data science stack update. This helps you to quickly reconnect to this important topic and maybe also to discover a data science gem you have not heard of yet.

JSON download: Some people asked me for the raw data of the PDF and at that time I was not able to give it to them quickly enough. That's why I added a json route that allows you to simply download the whole collection as a json file and create your own visualizations / maps or stacks with the tools that we have collected. Maybe something cool is going to come out of this.

Communication: Scanning through such a big list of options can sometimes feel a bit overwhelming, especially since we don’t really provide any additional info or orientation on the site. That’s why I added multiple ways of contacting us, in case you are just right now searching for a solution for your business. I took the liberty to also link our blog posts that are tagged with machine learning at the bottom of the page, because often we make use of the tools in these.

Zebra integration: Although it's nowhere visible on the website, I have hooked up the data science stack to our internal “technology database” system, called Zebra (actually Zebra does a lot more, but for us the technology part is relevant). Whenever someone enters a new technology into our technology db, it is automatically added for review to the data science stack. Like this we are basically tapping into the collective knowledge of all of our employees our company. A screenshot below gives a glimpse of our tech db on zebra capturing not only the tool itself but also the common feelings towards it.

Insights from collecting tools for one more year

Furthermore, I would like to provide you with the questions that guided me in researching each area and the insights that I gathered in the year of maintaining this list. Below you see a little chart showing to which categories I have added the most tools in the last year.

Data Sources

One of the remaining questions, for us is what tools do offer good and legally compliant ways to capture user interaction? Instead of Google Analytics being the norm, we are always on the lookout for new and fresh solutions in this area. Despite Heatmap Analytics, another new category I added is «Tag Management˚ Regarding the classic website analytics solutions, I was quite surprised that there are still quite a lot of new solutions popping up. I added a whole lot of solutions, and entirely new categories like mobile analytics and app store analytics after discovering that great github awesome list of analytics solutions here.

Data Processing

How can we initially clean or transform the data? How and where can we store logs that are created by these transformation events? And where do we also take additional valuable data? Here I’ve added quite a few of tools in the ETL area and in the message queue category. It looks like eventually I will need to split up the “message queue” category into multiple ones, because it feels like this one drawer in the kitchen where everything ends up in a big mess.

Database

What options are out there to store the data? How can we search through it? How can we access data sources efficiently? Here I mainly added a few specialized solutions, such as databases focused on storing mainly time series or graph/network data. I might either have missed something, but I feel that since there is no new paradigm shift on the horizon right now (like graph oriented, or nosql, column oriented or newsql dbs). It is probably in the area of big-data where most of the new tools emerged. An awesome list that goes beyond our collection can be found here.

Analysis

Which stats packages are available to analyze the data? What frameworks are out there to do machine learning, deep learning, computer vision, natural language processing? Obviously, due to the high momentum of deep learning leads to many new entries in this category. In the “general” category I’ve added quite a few entries, showing that there is still a huge momentum in the various areas of machine learning beyond only deep learning. Interestingly I did not find any new stats software packages, probably hinting that the paradigm of these one size fits all solutions is over. The party is probably taking place in the cloud, where the big five have constantly added more and more specialized machine learning solutions. For example for text, speech, image, video or chatbot/assistant related tasks, just to name a few. At least those were the areas where I added most of the new tools. Going beyond the focus on python there is the awesome list that covers solutions for almost every programming language.

Visualization, Dashboards, and Applications

What happens with the results? What options do we have to visually communicate them? How do we turn those visualizations into dashboards or entire applications? Which additional ways of to communicate with user beside reports/emails are out there? Surprisingly I’ve only added a few new entries here, may it be due to the fact that I accidentally have been quite thorough at research this area last year, or simply because of the fact that somehow the time of js visualizations popping up left and right has cooled off a bit and the existing solutions are rather maturing. Yet this awesome list shows that development in this area is still far from cooling off.

Business Intelligence

What solutions do exist that try to integrate data sourcing, data storage, analysis and visualization in one package? What BI solutions are out there for big data? Are there platforms/solutions that offer more of a flexible data-scientist approach (e.g. free choice of methods, models, transformations)? Here I have added solutions that were platforms in the cloud, it seems that it is only logical to offer less and less of desktop oriented BI solutions, due to the restrained computational power or due to the high complexity of maintaining BI systems on premise. Although business intelligence solutions are less community and open source driven as the other stacks, there are also awsome lists where people curate those solutions.

You might have noticed that I tried to slip in an awsome list on github into almost every category to encourage you to look more in depth into each area. If you want to spend days of your life discovering awesome things, I strongly suggest you to check out this collection of awesome lists here or here.

Conclusion or what's next?

I realized that keeping the list up to date in some areas seems almost impossible, while others gradually mature over time and the amount of new tools in those areas is easy to oversee. I also had to recognize that maintaining an exhaustive and always up to date list in those 5 broad categories seems quite a challenge. That's why I went out to get help. I’ve looked for people in our company interested particularly in one of these areas and nominated them technology ambassadors of this part of the stack. Their task will be to add new tools whenever they pop up on their horizon.

I have also come to the conclusion that the stack is quite useful when offering customers a bit of an overview at the beginning of a journey. It adds value to just know what popular solutions are out there and start digging around yourself. Yet separating more mature tools from the experimental ones or knowing which open source solutions have a good community behind it, is quite a hard task for somebody without experience. Somehow it would be great to highlight “the pareto principle” in this stack by pointing out to only a handful of solutions and saying you will be fine when you use those. Yet I also have to acknowledge that this will not replace a good consultation in the long run.

Already looking towards the improvement of this collection, I think that each tool needs some sort of scoring: While there could be plain vanilla tools that are mature and do the job, there are also the highly specialized very experimental tools that offers help in very niche area only. While this information is somewhat buried in my head, it would be good to make it explicit on the website. Here I am highly recommending what Thoughtworks has come up with in their technology radar. Although their radar goes well beyond our little domain of data services, it offers a great idea to differentiate tools. Namely into four categories:

• Adopt: We feel strongly that the industry should be adopting these items. We see them when appropriate on our projects.
• Trial: Worth pursuing. It is important to understand how to build up this capability. Enterprises should try this technology on a project that can handle the risk.
• Asses: Worth exploring with the goal of understanding how it will affect your enterprise.
• Hold: Proceed with caution.

Assessing tools according to these criteria is no easy task - thoughtworks is doing it by nominating a high profile jury that vote regularly on these tools. With 4500 employees, I am sure that their assessment is a representative sample of the industry. For us and our stack, a first start would be to adopt this differentiation, fill it out myself and then get other liipers to vote on these categories. To a certain degree we have already started this task internally in our tech db, where each employee assessed a common feeling towards a tool.

Concluding this blogpost, I realized that the simple task of “just” having a list with relevant tools for each area seemed quite easy at the start. The more I think about it, and the more experience I collect in maintaining this list, the more realize that eventually such a list is growing into a knowledge and technology management system. While such systems have their benefits (e.g. in onboarding or quickly finding experts in an area) I feel that turning this list into one will be walking down this rabbit hole of which I might never re-emerge. Let’s see what the next year will bring.

The February 2nd 2017, I have presented a talk entitled Hello, Rust! — An overview about the Rust language. This language describes itself as safe, concurrent, and practical. The goal of this presentation was to give an overview of several features brought by the language, such as its strong safety guarantees, or speed and memory performances.

The slides are available online

First slide from the Hello, Rust! talk

This talk has been presented during the first TupperRust meetup event in Lyon (France) . This is a serie of meetups focusing on Rust. The interaction was exceptional: The audience has been a great actor of this talk, and we even had a live-coding session on projects made by someones in the room. It was a great moment to talk about concrete problem, memory safety, performance etc.

It was also an opportunity to present a project that I have started here at Liip, called Tagua VM, which is an experimental PHP Virtual Machine that guarantees safety and quality by removing large classes of vulnerabilities thanks to the Rust language and the LLVM Compiler Infrastructure.

If you have any question, feel free to ask anything!

Some people travel to visit the CERN, whereas I had never been there. It is not an impressive building lost in the middle of a green field in the countryside of Geneva like I pictured it. It is lost in a suburban area and the building is not high. Rather, it has long, never-ending corridors filled with doors leading to little offices. It's very quiet, people whispers there. It looks nothing like the big open space that I am used to. However the people I listened too, have the same conviction about their projects.

Technology & research: for better or worse?

Technology already controls me. I am doing what it tells me to do. See, it tells me where to go and when (well that's my google calendar). And we are developing robots, they are clever machines. One day It will replace me at my work, then I will wake up and they will take over the world. And all babies will be perfect and born in incubators.

I admit, sometimes, I am lead by emotions and fear technology.

At the CERN, for one afternoon, I listened – mostly open-mouthed. Each of these scientists was amazing, not only because they were brilliant, but also because they were filled with conviction. Each of them develops or researches something which is beyond what we imagine on a daily basis.

Software, ADN, Medical Testing, Artificial Intelligence, each of these researches could revolutionize our world. For better or worse?

Samira Hayat – beyond her fear

Where she comes from, drones are killing machines. As a telecommunication engineer, she overcame her fear and started developing software for drone. It enables drones to work together as a single unit.

Drones are killing machine

Hayat started by sharing that where she comes from, kids say that they like better cloudy days than sunny days. Because on cloudy days, drones do not fly, so they do not kill. In other words, on cloudy days, kids can play in the streets, just like any normal kids.

As a telecommunication engineer, Hayat got a job at the University of Klagenfurt. She doubted, then accepted the job, because she is convinced that technology is what we make of it.

Drone can save lifes!

Hayat is currently developing software that enables groups of drones to work together as a single unit. The autonomous drone system developed by Lakeside Labs and U Klagenfurt, where she is a PhD student is one of «15 novel ideas for 2015» featured by the WIRED magazine.

As an example, the image below is an overview generated during a project work for a firefighter drill.

“The red line in the image shows the flight path of the drones and indicates where the drones took pictures. The multiple images taken by the drones are then stitched together for a final overview image”, explains Hayat.

Hayat and her team develops UAV systems which could be used in emergency situations or accidents or for automatic deliveries.

Her work can have application to larger global issues, such as unmanned searches and rescue missions or urgent medical deliveries, and construction in remote areas.

Technology is a tool

A tool, just like a knife. You can use it to cut someone's throat, save someone's life during a surgical operation or cut your bread. It is still the same tool. What changes is the hand holding the tool and the intention behind it.

What matters here, is trust and conviction. When I develop something, I am convinced that I am doing something good, and I see the positive use of it. I cannot ever predict or chose how other people are going to use it. I just trust, that they will be caring. The danger does not come from the technology, but from the person manipulating it.

To conclude: we should not be led by fear

We can transform something that makes us fear into something that serves our well-being. People with bad intention should never prevent us from working on good things. We should not be lead by fear but let our conviction lead us.

It sounds idealistic and soppy. However, sometimes, I am overwhelmed by what happens around me (let's say elections) and I forget it. What I remember from this afternoon at the CERN is that my actions should be lead by my beliefs and not by feelings of fear. If I think that something is good, I should just carry on.

Spread the word and carry on!

Thank you Samira for the picture!

If you are interested in reading more about Samira Hayat, clic here (in German) and if you want to read more about her work clic here.

The Era of Specialization

During his talk, Johnny Lee was mentioning that we are facing a change of the technological paradigm. Firstly we had an «Era of Conversion» when we tried to combine multiple functions into one device, like for example our mobile phones. But now with the cost drop of multiple devices people can afford to buy the best product for each use case. Let's say people prefer to buy a GoPro camera than to film with their mobile phone, they prefer to read a book through Kindle than through a regular tablet and so on. We are entering into an «Era of Specialization». Products are there to serve only a few use cases, though these have to have a good hardware, software and User Experience Design, otherwise they won't succeed in the market.

The Foggy Cloud

The talk of Johnny Lee made me think about my personal experience with «new» technologies which lack a good User Experience and don't match society's current needs. Then I thought about the first time I came across iCloud. Although I consider myself almost a digital native, I never trusted this «cloudy» invention. Maybe because I still remember writing letters to my friends, which means I'm not that much of a digital native. Or could it be because I wasn't properly introduced to it? I felt like digging deeper into this subject and began to research about other users' opinions, until I realized that I wasn't the only one feeling lost when getting in touch with iCloud for the first time – there was for sure an overall User Experience problem.

If I remember well I suddenly was asked to register to this iCloud and to upload all my personal data to «Apple's Heaven». The «Apple Angels» would then synchronize all my Apple devices and I would never have to worry again. Of course I felt insecure and dissatisfied! What the hell? I want to be able to control what I want to synchronize and to backup. Like I did before with iTunes. But they didn't let me stay with the old way. There were always some warnings telling me to login to iCloud and that if I didn't do so something bad would happen. At least this was how I and several other users felt.

Here is one of many examples which prove the overall users dissatisfaction – « Why is iCloud Contact Sync such a terrible mess?» by Cfacc, Apple Communities. If you click on the link you will see the comment underneath about a user complaint. It perfectly illustrates how I felt the first time I had to deal with iCloud.

«I don't use iCloud and using it for contact and calendar sync is something I will not do. Mavericks has messed up my whole workflow and I'm extremely ** off and there is no way in **** I'm ever buying anything from apple again if this is the road they want to force us users down.

My iPhone is now a piece of worthless junk without calendar syncing. I want my money back… to buy hardware and software that works and doesn't blackmail me into your foggy cloud with an open backdoor to anyone who feels like they need to take a peek into other people's lives.

An extremely ** off user of your overpriced junk which is progressively degenerating with every update.»

I found this comment extremely useful for my further research and of course I couldn't help but laugh, even knowing how frustrating it was for me as well when I first got in touch with iCloud.

Moreover, analyzing this User Experience problem through Don Norman's, director of The Design Lab at University of California and co-founder of Nielsen Norman Group, Human Action Cycle, made me realize what went wrong. The interaction process missed the first three phases: «Goal Formation», «Intent to Act» and «Planning the Act». Instead the interaction starts immediately with the «Execution of the Act». Well, this only could go wrong, because the user has no control nor freedom.

Human-machine interaction steps according to the Norman's theory.

Golden Rules for a Good User Experience Design

Besides not following the whole Human Action Cycle, it doesn't accomplish several golden rules for a good User Experience Design. Going back to Johnny Lee's talk – a product only succeeds if it has a good software, hardware and a good User Experience. In my opinion iCloud lacks the last one. At least seven of Jakob Nielsen's ,web usability consultant and co-founder of the usability consulting company Nielsen Norman Group, ten Usability Heuristics weren't followed.

Missing Usability Heuristics:

Usability Heuristic 1 – Visibility of system status

Users weren't being informed about what was going on, for example what, when and where something was being backed up or synchronized.

Usability Heuristic 2: Match between system and the real world There was never a clear and easy explanation about how Cloud Computing works. This may be obvious for a programmer, but for a non-specialist person it's something obscure and scary.

Usability Heuristic 3: User control and freedom This best practice was completely ignored because there was no introduction and users were more or less forced to use it. Besides that I never had the feeling I was in control, for example: I often felt like if I press «that» button there is no way back and that I could never get rid of the reminders and warnings about me not being a «well behaved» iCloud user.

Usability Heuristic 4: Consistency and standards The wording wasn't clear. For example I was never sure if «iCloud Drive» was the same as «iCloud» – the icon was the same but the name different. There are also different ways to access iCloud throughout the various devices and some visual inconsistency, which makes everything even more confusing.   

Usability Heuristic 5 and 9: Error prevention and help users recognize, diagnose, and recover from errors Due to unclear wording and lack of explanations, accidents like the one described on the user's complaint above occurred. Personally, as I have no patience to get informed about something that I was obliged to use, I just never dared to truly use iCloud's full potential. I didn't want to risk losing data or have it stolen – there is indeed only one password which prevents a hacker from stealing all my information.

Usability Heuristic 7: Flexibility and efficiency of use Using iCloud requires reading many warnings and going through steps which are confusing. Like using email accounts which I never created: xxx@me.com, xxx@icloud.com or xxx@mac.com. Even my Facebook account and my Apple-ID are somehow involved, no clue how. Furthermore I never recall how I have to access iCloud: Is it through the website www.icloud.com? Or through my phone? But on my phone it's integrated under «utilities» and it has a different interface design, etc.

Graphical representation of the Usability Heuristics relevance and relation in iCloud.

The User Need First

The iCloud example is just one of many new technologies which do not achieve their full potential and popularity, because they aren't aligned with the user need and habits yet. The same happened for example with the Google Glass and the Apple Watch, which turned out to be just gadgets for a minority of technology enthusiasts. These kinds of business failures surely result from a partial or even total lack of user research. That's why Human Computer Interaction Design plays a crucial role in the process of creating a new product.

Apple should have made a much smoother introduction and transition to iCloud, e.g. through:

• Doing honest user research – more research about the user need before launch ;
• Launching one feature at a time and testing its acceptance – more user testing (before and after launching);
• Making sure that people understand what iCloud is, so that they can rely on this technology – more user support ;
• Giving users the option to decide whether they want to use iTunes, iCloud or a mix of both – more user control and freedom .

In my opinion, these are the four most relevant aspects which Apple failed to achieve.

In conclusion, users should always come before the aspect of technical innovation. There is no benefit for users if an innovative solution or product exceeds their needs. An innovation only gains real value if it is employed in a useful way. That's why there are in fact many «Interface Technologies That Have Not Yet Left the Lab». In fact, the majority of new consumer tech is «old tech» for which the time is NOW ripe.