With that in mind, here some the basic starting questions followed by my [brief and biased] thinking on them:
- What is a playlist?
From wikipedia:
In its most general form, a playlist is simply a list of songs. The term has several specialized meanings in the realms of radio broadcasting and personal computers.
The term originally came about in the early days of top 40 radio formats when stations would devise (and, eventually, publish) a limited list of songs to be played. ...
As music storage and playback using personal computers became common, the term playlist was adopted by various media player software programs intended to organize and control music on a PC such as Nexus from NexTune. ...
Some websites allow categorization, editing, and listening of playlists online, such as Project Playlist, Plurn, imeem and Webjay. ...
This is a sensible starting place from a broad definitional stand, especially when we acknowledge that the exact MIREX will obviously be a subset within the idea of a playlist (more on that later...) How do we quantitatively evaluate an automatically generated playlist?
Upon a brief reflection this is clearly the sticky bit. As such this question is being posed prior to any sort attempt to specify an exact task. Functionally speaking, if the evaluation is properly specified the ask description will practically fall out of it.
There seem to be a few approach to this, though many tend to revolve around an idea of mining some form of ground truth out of crowd wisdom (here and here for instance)And of course, what exactly should the task be?
Again, this is of course all very draftish, but my prime thought on this is to have a highly specific task of one (or all) of the following forms: query with song A B C algorithm generates song D (and perhaps song E, where order matters), the simple deviation of that form given A B E, generate C D. The other possibility is something alog the lines of give a query song and provide the next song that should occur in the list.
So anyway, those are my initial thoughts on this potential task for next years MIREX. Please comment if you have thoughts...
Posts
1 comment:
For me, a playlist is a set of songs that have something in common (let's call it theme), played in a particular order.
First of all, I think, the order of the songs should not be important in a MIREX task, since retrieval of additional songs from a large database (having a query, whatever this will look like) is hard enough. Further, I don't have the slightest clue how to rate playlists that only differ in the ordering of the songs (if all the songs are somehow similar).
So if the ordering is skipped, the task would be something like a „song set enrichment“.
Songs in a playlist do have something in common. Allthough I know that many music playlist generation systems start with a single song as a query, I would favour more than one song to formulate a query, because I don't see how a system could be able to find out about the theme of a song set with using just one song. So „having something in common“ basically means to have a query (as starting point in the song space), and further a definition of what similarity (the thing in common) means.
A possible MIREX scenario could be:
People can submit song sets of 10 songs in advance to the contest, in addition to the theme, they were thinking of when compiling the set. Each set is split into two halves. One half is used as query, the other half is put into the retrieval database with thousands of other songs. Each system then has to add 5 more songs from the retrieval database to the given set of 5 songs – which means that they have to find out, what those 5 songs from the query have in common, and then have to retrieve 5 more songs from the database. The evaluation would be related to former music similarity MIREX evaluations. The person that contributed the set, has to rate all the retrieved songs (according to the theme he submitted). Just looking how many songs of the second half of the contributions have been retrieved by each algorithm might not be enough, since there can be more songs in the database that fit into the set.
There are many more things that are important for a good playlist (order, novelty, ...) but a system that is able to find songs that share the same theme than the set of query songs would be already valuable.
Post a Comment