Blueprint for the skills machine

User generated content – everyone wants it. Getting users to generate content is an age old problem, and one we are encountering in the skillclouds project. 

From user studies we have been told by students they would like to know what transferable (and sometimes subject specific) skills they learn during a degree course. This necessitates that the users designing the course are aware of and able to define the skills students learn during their course, and tag the course with this metadata – not an easy thing to do! 

An alternative approach to this generation of institutional metadata is to let a machine do it

 

Let’s take a look at the course description for the Sussex University course Approaches to meaning in English.

Course description

Course outline

In this course, exploration of word meaning introduces you to general linguistic concepts, terminology, methods and resources, while developing skills in linguistic analysis, research and argumentation. You will investigate meaning from psychological, social, historical, theoretical, and descriptive perspectives. Questions that may be considered include: what do you know when you know a word? Where is meaning located (in the word, society, or the mind)? How many meanings can a word have? How do meanings change? You will explore such questions in small, individual research projects.

Learning outcomes

By the end of the course, a successful student should be able to demonstrate:
1) an understanding of distinct levels of linguistic description (sound, meaning, grammar, etc.);
2) an understanding of basic concepts relating to English words and meaning (lexicon, semantics/pragmatics, reference, denotation/connotation, prototype, compositionality, lexicalisation, lexicography, necessary & sufficient conditions, etc.);
3) an understanding of some of the applications of linguistic analysis (social, historical, psychological, pedagogical, lexicographical);
4) discipline-specific skills in linguistic definition and analysis, the use of linguistic reference tools (dictionaries, etc.), finding linguistic resources in the library (beyond the reading list), accessing linguistic data resources and collecting linguistic data, and representing linguistic data in writing.

 

This all sounds like good stuff, but i want it displayed in a tag cloud.

So there are various tagcloud generators (like http://tagcrowd.com/ ) out their which go though a site or some text and based on the frequency of a word produce you a nice tag cloud. 

 

Word frequency based tagcloud

 

This is a nice quick way to get a first impression, but it’s not really highlighting my great skills i’m developing in ‘linguistic analysis’ ‘linguistic research’ and ‘linguistic argumentation’, it’s just not skills specific. 

 

So how does a machine know what skills are gained during a university course? 

 

i know what your thinking

 

Reading the excellent book Programming Collective Intelligence gives us some strategies for building a machine learning system to do this from scratch – but hang on, don’t i have a tool that does this already ? 

On another other wordpress blog i write for their is a handy plugin i installed called Tag suggest thing which suggests  tags for my posts based on the content in a much more seemingly intelligent way than a the word frequency based one above.

Giving it a try with the same course description gave the following output

 

tag suggest thing, suggested tags

 

This is obviously much more like the kind of metadata a user designing the course would be likely to produce, and probably more useful to students. It’s still not exactly what we want for skill specific information, but it’s gives us a steer as to how to make machine generated content if the users are not going to generate it.  

Lets take a look under the hood of Tag suggest thing.

‘Tag Suggest Thing uses the Yahoo! term extraction API to find tags to suggest.’

So what we are looking at here is what Yahoo’s term extraction tool sees as the relevant search terms for which it would display the page containing the Course Description text – same process as ( the currently not working) Tagcloud.  I find this interesting as someone who regularly engages in a bit of tagclouding to check the SEO of my content. 

 

Looking around i found another wordpress tag suggestion plugin Tagmahal.

Tagmahal suggests that (besides the fact that the author has a terrible sense of humor) our Course Outcomes text tags are.

 

tagmahal suggestions

 

At first glance, not as good as Yahoo!, but it is what’s under the artificial intelligence hood in Tagmahal that makes it relevant to us.

‘The algorithm will guess some tags based on a set of training documents tagged by humans.’

Tagmahal can be trained, it learns to recognise and suggest tags, it acts as a Bayesian Classifier. It maybe what we need – this HAL of Tags.

Built by Flaptor who also have an online limited demo Flaptor Autotagger Flaptor’s blog post suggests Autotagger was trained by looking through lots of (wordpress?) blog posts and seeing what their text content contained, compared to what humans had tagged the post. The picture below shows how autotagger arrived at the tag research.

 

Why autotagger suggested the tag reaserch

 

 

The highlighted words are those which have also apeared in other blog posts tagged research. The table below shows the contributions of each unique word, towards the decision of autotagger to tag our text with research. 

 

autotagger percentages for research tag

 

 

So we have a new hope. Get our own version of an autotagger, show it a largeish sample set of Course Descriptions which are specifically tagged with skills specific metadata. Let it run across the all the Sussex courses. In theory at least – it’s a winner. 

Even without user generated content we may have another method of getting the students the information they want –  as long as us humans don’t sabotage it

Rss strain

I used to blog about things people had made/done on the web.

Now most of these things are announced via a blog post – everyone has a blog.

It’s easy for me to see new things via rss and be inspired. It seems more difficult (or just less automated!) for me to find the things i used to blog about before rss was so well delivered. It’s much easier to blog something delivered to your door than to go out onto the big wide scary internet and find original content that inspires you enough to blog it – and even then, the chances are someone beat you to it.

So basically these days the chances are most bloggers are writing a blog post about a blog post. It’s a change, you deal with it, and originality becomes more and more difficult.

If there is a perspective to add to the original post, thats good and becomes part of the community of interlinked bloggers we all know. If your a big blog or blog in a different geo/specialist area giving exposure to a smaller blog it’s probably a v.good thing.

As 20jfg we were part of a digital music panel hosted by musically at the great escape festival in brighton and all of the other panellists spoke of the web media overload/saturation they experience. Just how many rss feeds can you keep track of in your netvibes? How often do you actually check your delicious for that thing you bookmarked and must look at in further detail when you have time? And how can you blog it all ?

On those days when you really just want to give a heads without much more ? You can have your blog, of blogs.

http://www.google.com/reader/

mine with the catchy url

http://www.google.com/reader/shared/03770040989831834500

I think the icon says it best –

del.icio.us bug

del.icio.us allows javascript calls which return the Javascript Object Notation (JSON) output of a users tags. The JSON format is a stripped bare XML for easy use by Javascript. Unlike the tag rolls JSON tag feeds contains no style information and can be skinned much more easily to the hosting site style.

The following output is all based on the sussex.skills delicious page.

The lists below use the browser default style.

Delicious also allows javascript calls which return the JSON output of a users bookmarked urls in the JSON format. 

You can also allows you to limit the bookmarks to particular tag. 

Or pick the union of two tags

Which may sometimes result in the empty set e.g.

all good!

But then we found a bug…

A Bug – Pattern matching and the JSON post feed….

You can filter the links that are loaded by tag. To only get bookmarks tagged with both ‘art’ and ‘history’:

http://del.icio.us/feeds/json/sussex.skills/art+history

 

The sussex.skills cloud contains many hyphenated term such as ‘data-collection’. 6 links are tagged ‘data-collection’.

An intersection of a set such as ‘data-collection’ and ‘analyisis’ which contains only 1 tagged link should produce a maximum of 1 result.

The code for this query would be

http://del.icio.us/feeds/json/sussex.skills/data-collection+analysis

For our dataset the expected intersection is none – the one url tagged as ‘analysis’ has not also been tagged ‘data-collection’.

The actual outcome is:

Supposition.

None of our test data is tagged ‘data’ so the result should be the empty set, and a search for ‘data’ delivers the empty set. However a search for the tags ‘data’ AND ‘data’ revealed the answer.

 

The Delicious search algorithm is doing partial pattern matching on any term on either side of the ‘+’ sign in the javascript arguments – a greedy search.

This renders it a little useless for our needs of displaying courses tagged with a specific match of skills

I wrote to delicious about a month ago now, but have heard nothing back so far. Writing a reminder now.

Anyone else found this or any ideas about a work around?