It’s a quite challenging and repetitive job to calculate and analyze artist similarities as well as make the results visible using data visualization techniques.

Since I’m working on a web-based visualization toolkit, which is dedicated to data analysis and visualization, I just decided to use this specific task of calculating artist similarities as a real-world use-case for the toolkit I’m working on.

The approach described in this post tries to generalize the problem domain and aims to make the calculation, analyzation and visualization of artist similarities a uniform and repeatable process.

Unveil.js

Before I start explaining my approach on the exercise, I want to give you a short introduction on Unveil.js, a data-driven visualization toolkit. It basically provides a simple abstraction layer for visualizations to ease the process of creating re-usable charts. To accomplish this, a data-driven methodology is used.

Data-driven means, that the appearance of the result is determined by the underlying data rather than by user defined plotting options. Visualizations directly access data trough a well defined data interface, a Collection, so there’s no longer a gap between domain data and data used by the visualization engine.

Such visualization can be re-used in terms of putting in arbitrary data in, as long as the data is a valid Collection and satisfies the visualization specification (some visualizations exclusively use numbers as their input, others use dates (e.g. Timeline plots), and so on…).

You can track the project’s progress at Github.

The toolkit’s data layer also provides means to implement arbitrary transformations that operate on raw data. So the challenge was to take input data, a set of user generated playlists, and perform similarity-measurement algorithms on it. The result is a new collection of artists (that occurred in the original data-set) that feature a set of similar artists. This set is based on a similarity score, which is determined by the algorithm.

Data Retrieval

In the first place you are faced with the problem of extracting data from different sources where various methods of information retrieval (MIR) need to be applied. So this task needs to be done mostly by hand and depends on how the data source is organized. The result of this task is usually an intermediate proprietary data format, which is used to perform processing and analysis on the data.

If you would agree on a uniform data exchange format you would be able to generically apply common operations like co-occurrence analysis on it.

Therefore, here’s a proposal of a lightweight Collection format represented as a JSON string that holds the raw data as well as some meta information about the data’s structure.

For my task (analyzation of playlists) I’ve built a simple data aggregation service that pulls user generated playlists from Last.fm and translates them into a uniform collection exchange format.

I’ve used the following strategy to get some suitable playlists:

1. Load a set of seed Artists (those relationships you’re finally interested in)
2. Collect top fans (users) for all seed artists
3. For every user pull its playlists
4. The resulting playlists contain only those artists and songs that are listed in the set of seed artists

Using this approach I’m most likely getting playlists that contain artists of the set of seed artists and eliminate the buzz. So once you have this format of data it’s quite easy to operate on it.

Here’s how a resulting collection of playlists looks like:

{
  "properties": {
    "name": {
      "name": "Playlist Name",
      "type": "string",
      "unique": true
    },
    "artists": {
      "name": "Artists",
      "type": "string",
      "unique": false
    },
    "songs": {
      "name": "Songs",
      "type": "string",
      "unique": false
    },
    "song_count": {
      "name": "Song count",
      "type": "number",
      "unique": false
    }
  }
  "items": {
    "4976660": {
      "name": "JD_Turk's.Vol.2",
      "artists": [
        "Beck",
        "Michael Jackson",
        "Green Day",
        "Madonna"
      ],
      "songs": [
        "Beck - Think I'm In Love",
        "Michael Jackson - Remember the Time",
        "Green Day - Last Night On Earth",
        "Madonna - Forbidden Love"
      ],
      "song_count": 8
    },
    "1380177": {
      "name": "Random!!!",
      "artists": [
        "Daft Punk",
        "Depeche Mode",
        "Moloko",
        "Radiohead"
      ],
      "songs": [
        "Daft Punk - Something About Us",
        "Depeche Mode - Precious",
        "Moloko - Cannot Contain This",
        "Radiohead - Nude"
      ],
      "song_count": 4
    },
  }
}

See the full collection here.

You can read more about Collections at the Unveil.js documentation page.

Gaining Similarity Measures

In order to perform similarity measurement on that data, I’ve implemented two different algorithms that calculate a similarity score based on co-occurrences. Those algorithms have been described in detail by Schedl, Knees, 2009.

Below you can see the essential parts of the transformation functions I’ve implemented.

Normalized Co-Occurrences described by Pachet et. al, 2001:

This approach just counts co-occurrences and normalize the result by using a symmetric function.

function coOccurrences(v1, v2) {
  var items1 = v1.list('items'),
      items2 = v2.list('items');
  return items1.intersect(items2).length;
};

function similarity(v1, v2) {
  return 0.5* (coOccurrences(v1, v2) / coOccurrences(v1, v1)
        + coOccurrences(v2, v1) / coOccurrences(v2, v2));
};

See the full implementation here.

Weighted co-occurrences on different distances, described by Baccigalupo et. al.:

function coOccurencesAtDistance(v1, v2, d) {
  var items1 = v1.list('items'),
      items2 = v2.list('items'),
      playlists = items1.intersect(items2);
  return playlists.select(function(p) {
    return checkDistance(p, v1, v2, d);
  }).length;
};

function similarity(v1, v2) {
  return 1*coOccurencesAtDistance(v1, v2, 0) +
         0.8* coOccurencesAtDistance(v1, v2, 1) +
         0.64* coOccurencesAtDistance(v1, v2, 2);
};

See the full implementation here

Visualization of the results

In order to visualize the result I decided to implement an interactive visualization that uses a number of small barcharts (showing similar artists for a certain artist).

Visual representation of the result

See the result in action

Conclusio

The algorithms implemented do not claim to be state-of-the art similarity measurement methods. However, the fact, that you can just swap-in any other implementation makes similarity analysis a repeatable process. In case you’re considering using this framework for your own data analysis tasks, just let me know. I’d be glad to get you started.

References:

• Automatically Extracting, Analyzing, and Visualizing Information on Music Artists (M. Schedl, 2008)
• Context-based Music Similarity Estimation (M. Schedl, P. Knees, 2009)

UPDATE: I uploaded an extended version of the patch described below, which also shows some parts from the previous article. You can download it here: walkingboxes.rar.

Some skinning theory

Skinning describes the process of binding a 3D geometry to a skeleton. The skeleton acts as some kind of abstraction layer over the 3D geometry: instead of moving single vertices, the animator can move joints - the 3D vertices are moved accordingly.

Besides X,Y,Z coordinates, every 3D vertex can have further data assigned, e.g. bind indices and skin weights. Those measures describe, to which extent a vertex is influenced by a certain joint. The bind index is the index of the influencing joint, and the skin weight is the amount of influence, the joint has on a vertex.

If each vertex of a model is influenced by only one joint to full extent, we call this rigid skinning. If multiple joints influence one vertex, we speak of soft skinning or smooth skinning. As you might have guessed, the results here are softer. And smoother, as well. If you are working with vertex buffers and hardware skinning, the number of influencing joints per vertex is limited to 4.

Building some character

We already talked about creating skeletons, so for this, we take a humanoid skeleton as granted. The image below shows the skeletal structure, and the global joint positions represented by quads in the renderer.

A humanoid skeleton created inside a subpatch. The joint positions in world space are computed by the GetJointTransform (Skeleton) node.

We can use the joint positions to create some fancy 3D geometry. For this example, I’m going to build one from some boxes. I’m not going into detail on how I did this, because it’s not looking that awesome anyway, so we will take this as granted as well

Some fancy 3D geometry dynamically built based on that given skeleton. The computed vertices are passed into a VertexBuffer, which flows into a Phong shader effect.

Skin it

The only way to create perfect skin weights still is manually doing it in a modelling package. This would be the way to go, if we created a model e.g. in Maya and imported it into VVVV using the COLLADA format. Since we built our geometry in VVVV, we need a way to create skin weights in VVVV as well. There are algorithms which automatically calculate skin weights. The most trivial of all is fully assigning a vertex to its nearest joint, which of course results in a rigid skinned mesh. Exaclty this is done by the node AutoSkinWeights (Skeleton). It takes vertex coordinates and a skeleton, and computes simple skin weights. Of course, there are smarter algorithms, like using heat maps, which aren’t implemented yet.

Using the AutoSkinWeights node to automatically determine skin weights for the created geometry. It simply assignes every vertex to its nearest joint, which results in a rigid skinned mesh. Skin weights and bind indices computed by the AutoSkinWeights node flow into the VertexBuffer node (you have to enable those pins in the inspector!)

Deform it

Since our mesh is now officially skinned, we can go on and actually deform it. The image above already contains the Skinning.fx shader effect. This effect does the calculation of the deformed vertices based on the joint transformations of the skeleton. All we have to do, is using GetJointTransform (Skeleton) on a posing skeleton, to get all the joints’ compiled transformations, and pass it on to the Skinning effect. The image below shows, how this would look like.

The SetJoint (Skeleton) nodes rotate the character's hip and knee. The resulting, possing Skeleton object flows into the GetJointTransform (Skeleton) node, which calculates the combined joint transformations. The Skinning shader effect deforms the geometry based on those joint transformations.

The inverse bind pose transformation

There’s still one thing in the image above, that has to be clarified: the 2nd input of GetJointTransform (Skeleton), which gets the inverse of the initial, unanimated joint transformations. This is, what in literature is referred to as the Inverse Bind Pose Transformation (IBP). The IBP holds information about the bind pose - the skeleton’s pose in the moment, the skin weights of a geometry are defined.

In our example, the skeleton’s initial pose, and the model’s initial pose match up, so the IPB is just the inverse of the “still standing” skeleton. Sometimes, this is not the case, e.g. if you use the same skeleton for different meshes. If you are using a mesh you imported from an external source, the IBP should be delivered by the importer.

A word on software skinning

Although using a shader effect for deforming geometry is definitly the common way to go, in certain cases you have to calculate the vertex positions on the CPU. This is, e.g. if your character rig exceeds the current hardware limits of 60 joints or 4 influences, or if you need the transformed vertex positions for further processing. In that case you can use the SkinDeformer (Skeleton), which basically does the same calculation as the Skinning vertex shader effect. However, instead of GPU calculation, SkinDeformer (Skeleton) works on the CPU, and returns the deformed vertices as vector spread, which then can run into a vertex buffer, or can be used however you want.

What to do in the future

Besides being very slow, AutoSkinWeights (Skeleton) only performs rigid skinning, which results in un-smooth deformation in joint regions. This might be ok for prototyping and in the cases, the result should look noisy anyway. But for more advanced projects, AutoSkinWeights (Skeleton) should also provide a soft skinning algorithm. Using heat maps as proposed in section 4 of Automatic Rigging and Animation of 3D characters looks promising.

The nodes have been developed using VVVV version 40beta23, and are therefore only tested with that version. First, download the nodes here: Skeleton.zip. Just place the files contained in the zip into the plugins-directory of you VVVV folder. Note, that you need Microsoft .NET Framework 2.0 and SlimDX installed, to use the nodes.

Alternatively, you can checkout the latest version of the sourcecode from the VVVV plugin repository on Sourceforge:

http://vvvv.svn.sourceforge.net/svnroot/vvvv/plugins/c#/Skeleton/SkeletonNodes/trunk
and
http://vvvv.svn.sourceforge.net/svnroot/vvvv/plugins/c#/Skeleton/SkeletonInterfaces/trunk

That should be it, let’s go, create some skeletons.

But before we start …

… let’s talk about some terms we will use from now on, quickly. For character animation in computer graphics, the most commonly used metaphor is binding skeletons to a geometry. A skeleton consists of joints, which can be animated. Animating joints ultimatly leads to animating the bound geometry accordingly.

Joints are arranged in a tree structure, beginning with the root joint (which might be somewhere around a characters belly, for example). Every joint can have child joints (e.g. fingers are children of the wrist), and most of the joints have one parent joint (e.g. the elbow’s parent is the shoulder). The position of a specific joint relative to its parent joint in “unanimated” pose is defined by the Base Transformation. The transformation, which describes the movement of a joint (e.g. raising the arm) is refered to as Animation Transformation. Both terms, “Base Transformation” and “Animation Transformation” are not commonly used in computer graphics, but only have been introduced here, to distinguish between a) describing a skeleton’s shape and b) defining a skeleton’s animation, respectivly. There may be other, more suitable terms for those two transformations.

One last thing …

In the screenshots below, a certain window is used to display skeletons and visualize how the nodes work. Don’t get confused, if you haven’t seen it yet in VVVV, because it comes from the SelectJoint node, which is described below. It just comes very handy for debugging and viewing skeletons.

Oh, i forgot: the Skeleton type

To be able to pass skeletal information from one VVVV node to another, the “type” Skeleton has been introduced. Most of the nodes described below will have such a Skeleton pin as in- or output.

I guess we can start: creating skeletons

One way of using character rigs in VVVV of course is importing them from an external 3D modelling package like Maya. This can be done easily by using the Collada Importer provided by VVVV. Another approach is creating the skeleton data inside VVVV. There are two ways of doing that: a spread-based and a graph-based way, both having advantages and disadvantages.

Using the node CreateJoint (Skeleton) is the way to go, if you wish to create skeletons based on spreads - for example if you want a dynamic skeleton, with an arbitrary number of joints (which you actually might not even know), and arbitrary topology. It takes spreads of data, one slice for each joint, and outputs a Skeleton-object based on this. This is an example of how you would use CreateJoint (Skeleton):

The use of CreateJoint to dynamically create a straight skeleton. Joints are defined by its joint names, which have to be unique. The topology of the skeleton is defined by every joint's parent joint. The parent of 'joint1' is 'joint0', the parent of 'joint2' is 'joint1', and so on. Every joint is translated 2 units and rotated slightly to its parent joint.

If you’d like to create some skeleton, which is more specific and static in terms of its underlying structure, e.g. a human skeleton, you might prefer the graph-based approach using the node Joint (Skeleton Join). In contrast to its spread-based brother, this node doesn’t take spreads as input, but only one value per input. In the most simple case, the output of Joint (Skeleton Join) is a Skeleton-object, which contains one single joint. Because this alone isn’t any fun at all, such a Skeleton-object can flow into another Joint (Skeleton Join)-node, which makes it the child of another joint. This way, the skeleton’s topology is defined by the vvvv graph. Take a look at this example to finally understand, what I’m trying to say:

Creating a skeleton using the Joint (Skeleton Join) node. Every node creates a sub-skeleton, which again is assigned to other ones. The bottom most Joint node outputs the complete skeleton of the hand. Here, the skeleton's topology is defined by the VVVV graph topology.

Moving single joints

Alright, now that we have created our skeleton, animating it would make sense. This is simply done by passing our brandnew Skeleton-object through a SetJoint (Skeleton) node. Although typical setter-nodes in VVVV usually manipulate single slices of a spread, SetJoint (Skeleton) works in a similar way. It manipulates single joints of a skeleton, selected through a selector pin.

Using this node, you can set most of a joint’s properties - you can even change its parent joint (Which is not tested very well yet, by the way). But the most common way of using SetJoint (Skeleton) is setting the Animation Transform pin, which ultimatly leads to a movement of a specific joint.

Animating joints using the SetJoint (Skeleton) node. The 'elbow' joint is selected through the selector pin. The joint is rotated by setting its animation transformation property. Note, how there come up multiple different 'poses' inside the patch.

Selecting joints

As you can see in the previous image, you need to know a joint’s name to be able to edit it with the SetJoint (Skeleton) node. This might be ok, if you get the joint name from somewhere programatically anyway. If you want to explicitly select a specific joint, e.g. your character’s shoulder, you don’t want to mess with the correct spelling, and prefer a visual way of selecting instead. This is, where the SelectJoint (Skeleton) node comes into play. It provides a window, which displays a skeleton in its current pose, and enables you to select specific joints by clicking them. SelectJoint (Skeleton) outputs the names of the selected joints. Besides selecting, this node is also very useful for debugging, and viewing animations - just as in the screenshots above.

Selecting Joints using the SelectJoint (Skeleton) node. The names of the joints, which are selected by clicking them inside the editor window, are returned by the node.

Animating on a higher level

Ok, moving single joints might be enough for waving a character’s arm or make it nod. But for more complex animations, or in your overall program logic, you don’t want to deal with single joints all the time. Instead, you want to talk about “poses” and “animations”, which should be triggerd in certain situations. To be able to work in such kind of way, the nodes InputMorph (Skeleton) and MixPose (Skeleton) are very useful.

InputMorph (Skeleton) works similar to the InputMorph (Value) node. It is used to interpolate through a series of poses by modulating an input value. This is very useful to morph from one pose to another - and therefore can be used to work in a way oftenr refered to as “pose-to-pose animation”.

Interpolating through a series of poses. The SetJoint (Skeleton) nodes define four key poses of a simple walk cycle. The InputMorph (Skeleton) nodes create intermediate poses, based on the input value on the left most side.

MixPose (Skeleton) like InputMorph (Skeleton) interpolates multiple poses and returns a result pose. In contrast to InputMorph (Skeleton) it doesn’t have one modulating value, but one for each incoming pose. This modulating value defines, to which extend the according input pose is expressed in the final result pose.

Combining two poses or animations using the MixPose (Skeleton) node. For each input pose, the amount of expression in the resulting pose can be defined.

InputMorph (Skeleton) works like a cross fader, MixPose (Skeleton) like line faders between single poses, if you want to.

Applying animations

So far we only viewed the results of our animations via SelectJoint (Skeleton)’s editor window, but didn’t output anything in any kind of renderer. First thing you will need are the global, compiled transformations of each joint. All transformations we dealt with until now were local transformations inside the joint’s space. However, to get a joint’s global transformation in world space, we need something, that “sums up” all local transformations in a joint chain. This is done by the node GetJointTransform (Skeleton). It takes an animated skeleton as an input, and outputs the compiled transformations of the joints, which are selected by the selector pin (leave the selector pin empty, to get all joints).

A humanoid skeleton created inside a subpatch. The joint positions in world space are computed by the GetJointTransform (Skeleton) node.

Having the global transformations, you can use them e.g. to kind of “assign” objects to joints, say, if your character consists of boxes. Also, you can use the information of the joint’s world position to accomplish some kind of collision detection, and so on. Finally, the compiled transformations are necessary for skinning, which will be explained in the next article.

Known Issues

There are still a lot of issues with the nodes described above. I set up quick bug tracker at sagishi.zive.at/bugs, where there are listed upcoming tasks to do. It’s public, so feel free to add issues, if you want to give feedback. However, some of the major issues are:

• Constraints (or joint limits): In the initial implementation there has been a way of setting constraints for each joint, to define, how far, and aroun which axes a joint can move. However, this has been in combination with saving joint rotations as euler angles. In the meantime, those euler angles have been replaced with quaternions. I haven’t stumbled upon a simple way of defining limits for quaternion rotations - any hints very much appreciated.
• Performance: There’s definitly a lot of potential in improving performance. When several InputMorph (Skeleton) nodes are connected in serial, framerate drops. This is caused by the fact, that the whole chain of nodes has to recalculate complete skeletons, as soon es only one joint changes. There is need of a smarter way to handle this.
• Inverse kinematcis would be awesome.

]]>