Roomba Recon - A musichackday brain dump

So this past weekend I attended the 2nd (annual?) London Music Hackday at the Guardian's offices at King's Cross. For the hack I created an algorithm that generates playlists between arbitrary start and end songs on soundcloud. It does this with almost no pre-indexing, allowing for playlists to cover the entire network and always use an up-to-date graph. It's (mostly) running live if you'd like to play with it.

Briefly, it performs a sort of bastardized A* search, bilaterially from both the start and the end song to form the playlist. There's a parameter to limit the length of the two playlist segments, by default this is 4 so the max playlistlength is 10 (2*4+2 for the end points).

The search algorithm collects social links of the artist corresponding to the given song. For each of these connections (you know, 'friends' or in soundcloud jargon 'followings') a determination of the cost of adding that song is calculated in the following way (for the half built from the start song):

where is the cost to add song m to list after song n, is some measure of distance from song n to song m and is the same measure of distance from song m to song e. Song e is the end song for the whole playlist. So basically the idea is that the cost of moving to a node is a ratio of how far away it is from where you were to how far it is to where you're trying to get. The whole thing is reversed for the other half, so the cost function makes it cheap to move toward the start song. If you simply want to randomly traverse social links the cost can be set to an arbitrary equal value (I used 1) for all links.

This leaves the matter of distance.

Starting with what I know best, I decided to try a content-based distance first. I should say that from the onset I figured this would be insanely slow, but none the less, I gave it a go. I implemented (available directly as well) a little object that will grab the echonest timbre features for any two soundcloud songs, summarize the features into a single multidimensional gaussian (mean and std) then take the cosine distance between the two tracks (other distance metrics could be computed as well, but cosine seemed reasonable). That takes something on the order of 45 seconds to do for every pair of tracks. When using it in the above playlister the whole thing would take maybe 4 hours (I think, I never actually let it complete). Clearly way too slow.

So taking inspiration from my about to be published work at WOMRAD, I thought some NLP could save the day. So the other distance measure I implemented (no direct access yet) is based on a tracks tags and comments. First I tokenize the comments and combine them with the tags to create a vector space model of a track's descriptive text. I then weighted everything using tfidf (the idf was populated with a random sample of tracks from across soundcloud that I gathered over the weekend, about 41,000 tracks in total. This is the only indexing that is done in advance). From the tfidf weighted terms in a vector space, I took the cosine distance. This is both quite quick and gives pretty good results.

Everything was built in python, the app is running in cherrypy, using numpy and scipy for the data handling and gensim for the tfidf related bits. Soundcloud and echonest interaction is all via their respective python wrappers. Also there's a more terse write up over at the musichackday wiki. I'll stick the code on my repository on github once it's cleaned up a bit (though that might be a little while as I seem to be rather busy with something at the moment...)

Right. Back to writing my thesis.

MusicHackday: Stockholm

So in a touch more than 48hrs I'll be hoping on a plane to go the Stockholm MusicHackday. It should be excellent, if the last one I went to is any judge. I'll be joined by fellow ISMS member Mike Jewell. The hack is being formulated, but may involve The World Bank's api and some yet to be determined sources of listener statistics. Also, somehow the echonest's api will be involved because I need to leave stockholm with one of these. We may need some further assistance to get something done in 24hrs, so if you're going to be at the hack and are looking for some folk to hack with drop a line in the comments...

musichackday

So I spent the weekend holed up in the Guardian offices at the musichackday. I went in with some perhaps overly ambition plans to generate playlists across the SoundCloud user graph, with song selection optimization done with features via theechonest. This might have been barely possible if I had been working with a couple other people of similar background, but circumstances led to me hacking mostly solo at this particular event.

In the end I spent a substantial amount of time beating the SoundCloud python wrapper into being more helpful for what I wanted it to do (which is perhaps not what it's envisioned use was, but hey, that's what hacks are for), namely walking the user (artist) space and creating a Complex Network so I can move the playlist generation tools we've created around myspace crawls over to SoundCloud.

So, to that end, I've created some bits of python that walk through the user graph on the SoundCloud and build a graph using iGraph. This code base is living over at a new github repository I've created called pySomethingClever. Included over there are diff files documenting the changes I made to official SoundCloud-api-wrapper, which will enable any willing victims to grab and run the hacky bits of code I have up.

Once I got the api wrapper in a place where it could do a bit of what I wanted I fired off a crawl. I got through about 4,000 users (of a complete user network of about 170k nodes for ~2.3% of the network) in SoundCloud's network before the presentations started on Sunday. To clarify slightly, the network contains all the users of SoundCloud, but only the outlinks (users a given user follows) from 4,000 nodes. This is to say I had a (mostly) complete vertex list and a very incomplete edge list. With the super great help of kurtjx this sampled network was pushed through the lanet k-core decomposition visualization to draw out some of the community structure and related forms of the sample graph. Here's that graph:

The size of each node is tied to the number of links (either direction) touching that node. The color and placement have to do with how critical the node is to the rest of the network maintaining its current state of connectedness.

Since the hack I've continued gather edges toward a complete representation of SoundCloud. I currently have the out link edges from more than 17,000 SoundCloud users (about 10% of the user base) and should have a full capture in the next few days. Below you can see the same visualization with the edges from 16,000 users (the graph is set to write every 2k):

As the crawl continues, my guess is the middle bits will continue to fill in, which would be expected if the SoundCloud behaves in the usual Power Law fashion (as most of The Internet's networks, social or otherwise, tend to).

It should be noted that these visualizations, while very interesting, are just the beginning of what is possible once the whole user network is captured. I'm going to be building some playlist generators and recommenders around this in the coming weeks. If things look good (and from here I'm quite excited) I'll push some of it to the ISMIR late breaking demos and possibly to AdMIRe. More to come!