So at liip we have a slack channel for everything, for #machine-learning (for topics related to machine learning), for #zh-staff (where Zürich staff announcments are made), for #lambda (my team slack channel) and so on. Everybody can create a Slack channel, invite people, and discuss interactively there. What I always found a little bit hard was «How do I know which channels to join?», since we have over 700 of those nowadays.

Wouldn't it be cool if I had a tool that tells me, well if you like machine-learning why don't you join our #bi (Business Intelligence) channel? Since Slack does not have this built in, I thought lets build it and show you guys how to integrate the Slack-API, Pandas (a multipurpose data scientist tool), Flask (a tiny python web server) and Heroku (a place to host your apps).

These days there are myriads of ways to build a recommender system and there are lots of commercial and open source solutions out there already. There are recommenders that recommend based on contents of items (e.g. read this book because it is about snakes and planes) or there are solutions that recommend based purely on user behavior data (e.g. people who have bought this book have also bought this book). The latest ones are called collaborative filtering recommenders (fancy name) and we will build one of these since they are easy to build and work reasonably well.

Collaborative Filtering approaches can be divided into two main approaches: user-item filtering and item-item filtering. A user-item filtering will take a particular user, find users that are similar to that user, based on similarity of ratings, and recommend items that those similar users liked. In contrast, item-item filtering will take an item, find users who liked that item and find other items that those users or similar users also liked. It takes items and outputs other items as recommendations.

What does this mean for Slack for us? Well in an item-item approach we might say give me “channels that are similar to machine-learning”. The algorithm will then look who joined this channel and look if there are channels that have been joined by exactly the same people. If that is the case then those channels might be somewhat related. In a user-user approach, we try to recommend a channel to a particular user. So we look through all users and look for the ones that have joined the same channels as our user joined. Then look if there are some channels among those that our user has not joined yet. These ones might be interesting for him.

There are myriads or better blog articles that do a great job explaining in detail on «How to build a recommender in Python» – I think I should at least list a couple of those ( Deconstructing Recommender Systems by Spectrum, Implementing your own recommender in Python by online Cambridge coding, Collaborative Filtering with python by Salem Marafi). There are also a couple of nice Ipython notebooks that you can download and play along ( Playing with recommender systems). If you want to know what the people are doing and who is creating state-of-the-art recommender systems you should visit the Recsys conference. You can also download tons of academic papers on how different recommender systems perform and why some are better than others in different scenarios. There is even a great coursera course on recommenders by Joseph Konstan that I highly recommend. But for the sake of brevity, we can not go into all the details here, we just want to get you started. Right?

What it boils down to is that in our case for Slack you need to construct a matrix of users x channels. If you have a matrix that has users x items, you have a setup that is called “item-item collaborative filtering”, if you rotate this matrix 90 degrees and have a matrix that has items x users you have an user-user collaborative filtering. It is as easy as that. I apologize for stealing those two great images from this great blog post of online Cambridge coding, but they show nicely how things work.

Step 1: Getting the data

So in order to generate the needed data, just create a Slack-client give it some key that you will get from Slack and let it go through all the channels and note down which person is in which channel. Furthermore write down a 1 if a person is in a channel, write down a 0 when not.

Once we have our data we can start working. We will first read in the data into pandas and display it. It is basically two lines of code and you will see the nice matrix unfolding. Notice that we are really working on a tiny dataset here, that's why it is perfectly ok to do everything in memory. But once you have a matrix consisting of millions of row things start to become interesting.

Step2: Computing the distances

In order to know which channels are similar to which other channels, we have to look at those user vectors for each channel, and see if there is another channel for which the same kind of users have also joined. Those channels must have something in common. Now there are myriads of different metrics on «how to compute a similarity between two vectors?», like pearson-correlation, jaccard-distance, manhattan-metric, or cosine similarity. Just pick one. They all have their strengths and weaknesses. For this experiment I went for cosine-similarity. Doing this in Pandas is really easy. Only 4 lines of code and you will have this:

Isn't that great?! We now have a matrix that shows how similar each channel is to the other ones. To make a recommendation we have to look up for a certain channel which other channels have the highest number in that row and we are done.

Step3: Making a recommender

So, to make a recommendation we look for the row matching the channel name and then sort the columns by value and remove one entry. Namely the channel itself, because we know it is similar to itself :). We then output, for example, the 5 highest numbers. Again, this can be done with 4 lines of code in Pandas. You just have to love this thing! See below:

Well, there we have it: apparently if I go to the channel #machine-learning I should also visit #bi and #beekeping. Our company has a weird humor when it comes to naming things. :) I would have never guessed a second, going simply by name. So what is next you ask? Well, it is time to take those results and make them work for the company. At this point we will skip the part where I show you «how to recommend channels for users» or «how to compute users that are similar to each other» or «how to evaluate the results» (In real life, problem evaluations of recommenders is a serious problem and should be definitely part of your approach).

Step4: Build a service out of it

What normally happens it that our precious models and results end up rotting away on our laptop, because we have no time, budget, or priority to turn them into actionable artifacts. But I think this is actually the most important step for a data scientist.

Make sure that your results are understandable and communicate them to people that might find them useful. Make sure you are actually going to act on those results, because that is where you actually helping people. So I thought we must turn this tiny recommender into a website in our company where everybody can get their own recommendations.

That's where F lask helps us. The good thing about Flask is that our recommender code is written in Python and the web server is in Python too, so we can simply cut the code from our notebook and paste it into our web server. This makes our life very easy because normally at this point data scientists struggle because other engineers have to implement their shiny and fancy solution in another programming language. That takes time, lots of swearing from the developers and creates lots of headaches for the data scientist. So my approach uses the same programming language across the whole stack. Then you should be fine. Well, indeed you can create an endpoint in Flask that takes a channel_id and runs your recommender code with it and returns the results as a JSON back to the frontend. You should recognize the same code above in the ipython notebook and here below:

Step 5: Make it look fancy

The last remaining step is to create a nice looking GUI for the recommender where the users can select either a channel or a username. I have also implemented a goofy friend finder, that lists persons that seem to have the similar interests on Slack, based on what channels they use. So given the sea of messages, data and communication it seems to be useful having a tool that can help us out a bit. Now without further ado: I present to you the Liip Slack channel recommender:

A screenshot of the tool

The only thing left is to push it to Heroku or Cloudfoundry or the provider of your choice. It really makes your life easier if you control the deployment process because this way you can iterate on your model easily without asking your sysops for permission. (Of course, if a big system depends on your algorithm you should get serious about testing, fault tolerance, SLAs and all of that grow up stuff).

If you feel like you have liked the solution drop me a line, and we can send you the code to create this solution for yourself or just talk to us and we will think of something on «how implement recommenders or similar for your slack channels or your people directory». After all, I think these days it is time to not only build good looking, stable, fast and easy to use – but also smart data-driven websites.

If you are a Liiper reading this: click here to login into the recommender. PW/Login are available on #announcements. :)

After updating my system to the newest versions of pandas, I had to find a new connector to fetch Google Analytics data, and found an advantageous replacement in the google2pandas module from Panalysis.

This blogpost walks you through the setup of Pandas and google2pandas, and breifly introduces you to fetching and getting Google Analytics data into Pandas dataframes, for further exploration with Pandas.

Installation

The following instructions are aimed at OSX and Linux system users.

1. Download and install Anaconda by following the official instructions. Although your machine might already have a Python copy pre-installed, I recommend you install Anaconda, a bundle of 300+ data science tools that comes with its own package manager. When it comes to data analysis, one tool never fits all needs and with Anaconda, you get in the good company of so many of them. If you do not want to install the whole Anaconda, follow installation instructions for Pandas.

2. Install the connector module and its dependencies:

pip install git+https://github.com/panalysis/Google2Pandas

3. Test both installations on an interactive python environment, the following statement should run quietly and not throw any ImportError:

import pandas
import google2pandas

Setting up the connection

Now the tricky part: setting up the authorized connection to Google Analytics API.

1. Follow the step 1 in the official instructions for installed apps .
2. Create ‘ga-creds' directory in your data analysis workspace/directory, and move in it the client_secrets.json you downloaded in the previous step. An alternative would bet to put those credentials in the google2pandas installation folder. I however recommend you save those credentials in a private project folder and always specify their path when establishing the connection (see how below). That will allow you to reinstall modules without loosing credentials and manage multiple credentials.
3. Set correct permissions :

chmod 644 ga-creds/client_secrets.json

4. Save the following script in a file (ga-conn-init.py):

from google2pandas import *
conn = GoogleAnalyticsQuery(secrets='./ga-creds/client_secrets.json', token_file_name='./ga-creds/analytics.dat')

5. Run python ga-conn-init.py.

A browser window should open, with a usual Google Account authentication process, plus asking permission to View your Google Analytics data. Please allow. Then the browser window redirected to a barebone page displaying: The authentication flow has completed. Were you to face a _no data received (ERR_EMPTYRESPONSE) page, try python ga-conn-init.py --noauth_local_webserver. If that also does not work, rerun the same command in another terminal window while the previous one is still running, this has worked for me.

6. Check that it created the ga-creds/analytics.dat file .
7. Adjust its permissions :

chmod 664 ga-creds/analytics.dat

Take a deep breath. This authentication process done, you shouldn't need to do it again in a near future. Just remember to always point to both files when establishing connection.

Fetching Google Analytics data

Basic query

1. Go to Google Analytics , choose a view from which you want to import data, click on the admin tab and note down the view ID :
2. Replace your VIEW-ID in the following lines and run them . That should print a tabular listing with 10 rows and data for the following columns: pageviews, data, pagePath.

from google2pandas import *

conn = GoogleAnalyticsQuery(secrets='./ga-creds/client_secrets.json', token_file_name='./ga-creds/analytics.dat')

query = {\
  'ids' : 'VIEW-ID',
  'metrics' : 'pageviews',
  'dimensions' : ['date', 'pagePath'],
  'start_date' : '8daysAgo',
  'max_results' : 10
}

df, metadata = conn.execute_query(**query)
print(df)

Further queries

Both the official Google Analytics Core Reporting API and the Google2Pandas module documentation and code can be useful ressources for expressing (complex) queries:

Google2pandas module:

• module query parameters

Google Analytics Core Reporting API Documentation:

• API query parameters
• Dimensions & Metrics Explorer
• Using filter operators

Last year's DjangoCon Europe took place on its own island, and this time we were welcomed by the beautiful Cardiff University. One of the goals the organizers immediately set was to promote diversity and embrace minorities, encouraging them to participate to this event. I remember when I bought my ticket and I read “all dietary requirements can be met” and I thought “they're probably trying to set the bar too high”. This is a not so unusual statement, but when confronted to reality you usually end up with dishes not adapted to your dietary requirements, and, as a minority, having to ask for something special can really make you feel excluded. This time was different, as the organizers meticulously took care of that.

I talked about dietary requirements because it's an important topic for me, but they also made sure to have a code of conduct, a financial assistance for people who want but can't afford to attend, a wellbeing service, a subscription system for meals to avoid wasting food, reusable aluminum bottles instead of plastic water bottles, live transcription of talks (for hard of hearing or non-english people), and even a crèche.

All of this was not just buzzwords but was carefully handled by the organizers. In the end, it made everyone feel welcome and put a very warm atmosphere to the event. Talking with people was very interesting and rewarding because of the incredible diversity of backgrounds. A few days after the conference the Django Software Foundation even released a diversity statement to make sure everyone feels welcome in the Django community.

The choice of talks was about 50/50 between technical talks and community talks. I found the community talks to be very inspiring such as Ola Sendecka's “ Into the rabbit hole” which reminded me the benefits of the Pomodoro technique and pair programming. Also Adrienne Lowe's talk “Coding with knives” and Russell Keith-Magee's touching talk about burnout debunked the myth of the rock-star programmer, and how important it is to help people to get started on things they're not comfortable with.

I can't list all the talks I loved because there are way too many but among those I could find the slides for were Ana Balica's “ Demystifying mixins with Django“, Erik Romijn's “ A sincere tale of Django, developers and security“, Aaron Bassett's “ Effortless real time apps in Django“, James Bennett's “ The net is dark and full of terrors“, and of course Rae Knowler's “ Randomised testing for Django with Hypothesis“.

The 3 days of talks were followed by 2 days of sprints and this was a very good opportunity to dive into Django's code and start fixing things. The core developers did a good job at helping people setup a development environment and start hacking on the code. This was a very rewarding part of the conference as this allowed me to do my first pull requests on Django and Django CMS.

Thanks to the DjangoCon Europe team who did an amazing job at organizing this event. See you next year in Budapest!

Recently we decided to explore the edX platform. It's an open source Learning Management System (LMS) and authoring tool. EdX is one of the most popular MOOC platforms alongside Coursera and Udacity, but unlike the two others their code is open source and you can host your own MOOC platform. EdX has been created by MIT and Harvard and is listing other well known universities as contributors. Also, in September 2013, Google, who was previously experimenting its own LMS (Google Course Builder), decided to join the open edX platform as a contributor too. The platform is implemented mostly in Python and they created their own component architecture called XBlock (https://xblock.readthedocs.org/en/latest/). On paper edX sounds great, but how does it compare to Moodle or Totara? Does it fulfill the criteria our clients are targeting?

First steps

Installing a local instance is fairly easy since they provide a vagrant box and the documentation is good. But on the other hand, you will need to run three different servers with three different port forwardings. One for the LMS, one for the CMS (authoring tool) and the last one for the forums (this one is written in Ruby). Moodle or Totara use the same tool for standard users and administrative users which can lead to a bad user experience. So having different tools can have advantages and disadvantages.

Managing users

One key feature Moodle and Totara are offering is the user management. This is also one of the first requests we usually receive when dealing with offers. With Moodle and Totara there are lots of ways to authenticate a user and also to enroll them to a course out of the box. You can also easily manage roles and create groups (even dynamic groups with Totara).

In edX a user can authenticate if he creates first an account on the platform but edX is built on Django so any authentication method from this framework should be available on edX too (e.g. LDAP authentication).

It was not so easy to find how you can restrict the access to a course, this seems to be feasible depending on different settings (they were not explained in the user documentation, but understandable from the code documentation). You can create groups but it's not clear yet if you can have groups platform-wide too.

Inside a course it's easy to manage your staff team, define your teaching assistants or other admin users. You can even add beta testers so that some students can test your course before its official public release. You can also have discussion moderators for your courses.

Multilanguage

Having the platform in several languages is possible, and courses in different languages too. But you cannot have the same course in different languages (which is the same for Moodle and Totara, you simply need to create a course for each language). A good and bad thing with the current state of internationalization is that it's a work in progress. It's bad because if you want a german edX platform, the translations are not finished yet. But it's good because there are working on it and plan to finish it for the end of second quarter 2014 and they have a translation tool where anyone can contribute (https://www.transifex.com/organization/open-edx). By default, it would use the browser language, but if you're logged in you can also have a user language (at least it's configured that way on www.edx.org).

Hosting

Configuring manually your server to host an edX platform is really difficult, there are a lot of requirements and dependencies to install. For Debian some of them don't even have a package. But they provide a way to install it using Ansible requirements which seems quite easy to do. They also provide some script to execute update which is nice.

Interaction between users

Like on Moodle and Totara, you can have discussion forums for a course. There is a nice WYSIWYG editor with the possibility to preview your post first.

A nice feature of edX is the ability to comment any exercise or theory page and allowing others to upvote/report those. It's a pity though that the comments aren't ordered by upvotes by default.

Students can find study buddies using Meetup communities. EdX can also integrate Google Hangouts.

Reporting

EdX comes with basic reporting tools. One can generate for example the list of enrolled students and the grade reports. Compared with Totara's custom reports feature, edX has still plenty of room for improvement.

User Interface

Generally the interface looks very nice. The UX-UI concept is very strong and has a clear consistent interaction level. You can also implement custom themes as Stanford did for its own instance. Without going into details, it seems edX has a nice base theme and state-of-the-art Interaction Patterns but they don't have a responsive theme – only the home page is responsive.

Learning experience

The learning experience is appealing and has the same feel as new online platforms (team-treehouse, codeacadamy, …). Courses offer a lot of options out of the box: video integration (via Youtube), multiple question forms (radio buttons, checkboxes, dropdowns, text, maths inputs, …), Google Hangouts, discussions, text, announcements, integrating an Iframe, full-screen images, zoom images, imaged-mapped input, drag and drop answers, electrical schemas, LTI integration, … but there are bad things too. For example, the grading system is not suitable for the usual Swiss grading system (grades from 1 to 6), it's apparently made for the American grading system (from F to A).

Conclusion

EdX does less things than Moodle but does them nicely. It's also better suited for open education which can be achieved in Moodle and Totara too but the user and learning experience seems to be better on edX. Sever configuration, user management, access restriction, internationalization and reporting, all those things are easier to set and ready to use out of the box with Moodle (reporting especially on Totara) and cannot be neglected. Some might say that Moodle does too many things but from a business point of view it's easier to adapt client's requests that way. Also the community around Moodle is bigger which is a great thing from a developer point of view. But to end on a positive note, edX provides a roadmap on what they are working or will work on, thus we will keep an eye on this innovative platform and check if their improvements could fit future client requests.