Blueprint for the skills machine

User generated content – everyone wants it. Getting users to generate content is an age old problem, and one we are encountering in the skillclouds project. 

From user studies we have been told by students they would like to know what transferable (and sometimes subject specific) skills they learn during a degree course. This necessitates that the users designing the course are aware of and able to define the skills students learn during their course, and tag the course with this metadata – not an easy thing to do! 

An alternative approach to this generation of institutional metadata is to let a machine do it

 

Let’s take a look at the course description for the Sussex University course Approaches to meaning in English.

Course description

Course outline

In this course, exploration of word meaning introduces you to general linguistic concepts, terminology, methods and resources, while developing skills in linguistic analysis, research and argumentation. You will investigate meaning from psychological, social, historical, theoretical, and descriptive perspectives. Questions that may be considered include: what do you know when you know a word? Where is meaning located (in the word, society, or the mind)? How many meanings can a word have? How do meanings change? You will explore such questions in small, individual research projects.

Learning outcomes

By the end of the course, a successful student should be able to demonstrate:
1) an understanding of distinct levels of linguistic description (sound, meaning, grammar, etc.);
2) an understanding of basic concepts relating to English words and meaning (lexicon, semantics/pragmatics, reference, denotation/connotation, prototype, compositionality, lexicalisation, lexicography, necessary & sufficient conditions, etc.);
3) an understanding of some of the applications of linguistic analysis (social, historical, psychological, pedagogical, lexicographical);
4) discipline-specific skills in linguistic definition and analysis, the use of linguistic reference tools (dictionaries, etc.), finding linguistic resources in the library (beyond the reading list), accessing linguistic data resources and collecting linguistic data, and representing linguistic data in writing.

 

This all sounds like good stuff, but i want it displayed in a tag cloud.

So there are various tagcloud generators (like http://tagcrowd.com/ ) out their which go though a site or some text and based on the frequency of a word produce you a nice tag cloud. 

 

Word frequency based tagcloud

 

This is a nice quick way to get a first impression, but it’s not really highlighting my great skills i’m developing in ‘linguistic analysis’ ‘linguistic research’ and ‘linguistic argumentation’, it’s just not skills specific. 

 

So how does a machine know what skills are gained during a university course? 

 

i know what your thinking

 

Reading the excellent book Programming Collective Intelligence gives us some strategies for building a machine learning system to do this from scratch – but hang on, don’t i have a tool that does this already ? 

On another other wordpress blog i write for their is a handy plugin i installed called Tag suggest thing which suggests  tags for my posts based on the content in a much more seemingly intelligent way than a the word frequency based one above.

Giving it a try with the same course description gave the following output

 

tag suggest thing, suggested tags

 

This is obviously much more like the kind of metadata a user designing the course would be likely to produce, and probably more useful to students. It’s still not exactly what we want for skill specific information, but it’s gives us a steer as to how to make machine generated content if the users are not going to generate it.  

Lets take a look under the hood of Tag suggest thing.

‘Tag Suggest Thing uses the Yahoo! term extraction API to find tags to suggest.’

So what we are looking at here is what Yahoo’s term extraction tool sees as the relevant search terms for which it would display the page containing the Course Description text – same process as ( the currently not working) Tagcloud.  I find this interesting as someone who regularly engages in a bit of tagclouding to check the SEO of my content. 

 

Looking around i found another wordpress tag suggestion plugin Tagmahal.

Tagmahal suggests that (besides the fact that the author has a terrible sense of humor) our Course Outcomes text tags are.

 

tagmahal suggestions

 

At first glance, not as good as Yahoo!, but it is what’s under the artificial intelligence hood in Tagmahal that makes it relevant to us.

‘The algorithm will guess some tags based on a set of training documents tagged by humans.’

Tagmahal can be trained, it learns to recognise and suggest tags, it acts as a Bayesian Classifier. It maybe what we need – this HAL of Tags.

Built by Flaptor who also have an online limited demo Flaptor Autotagger Flaptor’s blog post suggests Autotagger was trained by looking through lots of (wordpress?) blog posts and seeing what their text content contained, compared to what humans had tagged the post. The picture below shows how autotagger arrived at the tag research.

 

Why autotagger suggested the tag reaserch

 

 

The highlighted words are those which have also apeared in other blog posts tagged research. The table below shows the contributions of each unique word, towards the decision of autotagger to tag our text with research. 

 

autotagger percentages for research tag

 

 

So we have a new hope. Get our own version of an autotagger, show it a largeish sample set of Course Descriptions which are specifically tagged with skills specific metadata. Let it run across the all the Sussex courses. In theory at least – it’s a winner. 

Even without user generated content we may have another method of getting the students the information they want –  as long as us humans don’t sabotage it

why are skills important ?

napoleon dynamite says it all.

User testing recruitment – social network style

How to recruit students for user testing at sussex university, the Stuart method.

 

1. go to facebook

2. look at a friend

3. look at their ‘Networks with the most friends’

4. click on the one called ’sussex.uni’ (if they have it) – this will show you all their friends that go to / have gone to or work at sussex uni

 

The most difficult bit is telling if you know or might have met the student from their tiny photo, but we got a really good response.

Epoch

As part of the skillclouds user research session we discovered there is a very specific time when the majority of students will actually want to access information on their skills, course details and grades – mainly motivated by filling in a CV or application for further study. 

This time of readiness is different in all students, be it for a work placement in the first year, a summer holiday job in the second year, or a to assess which parts of a course to take next. This makes it very difficult to predict.

Bombarding students with information before this can actually prove detrimental. 

My old colleague Pete Gale from Cogapp gave a recent Sussex HCT group presentation showing information from some projects we had been involved with in the health care sector and the aftermath brought up some interesting parallels. 

The readiness to change ruler

Healthcare patients information motivation can be measured on a readiness-to-change ruler, with information being delivered at the point of motivational ‘epoch’.  

After the peak is often too late.

Delivering information before users are ready leads to overexposure and a numbness which reduces the chances of the information having any impact.

 

The same ideas prove true in student motivation. It’s a very difficult thing to predict, so we have an interesting question ;

When are students ready to receive skills information ? 

del.icio.us bug

del.icio.us allows javascript calls which return the Javascript Object Notation (JSON) output of a users tags. The JSON format is a stripped bare XML for easy use by Javascript. Unlike the tag rolls JSON tag feeds contains no style information and can be skinned much more easily to the hosting site style.

The following output is all based on the sussex.skills delicious page.

The lists below use the browser default style.

Delicious also allows javascript calls which return the JSON output of a users bookmarked urls in the JSON format. 

You can also allows you to limit the bookmarks to particular tag. 

Or pick the union of two tags

Which may sometimes result in the empty set e.g.

all good!

But then we found a bug…

A Bug – Pattern matching and the JSON post feed….

You can filter the links that are loaded by tag. To only get bookmarks tagged with both ‘art’ and ‘history’:

http://del.icio.us/feeds/json/sussex.skills/art+history

 

The sussex.skills cloud contains many hyphenated term such as ‘data-collection’. 6 links are tagged ‘data-collection’.

An intersection of a set such as ‘data-collection’ and ‘analyisis’ which contains only 1 tagged link should produce a maximum of 1 result.

The code for this query would be

http://del.icio.us/feeds/json/sussex.skills/data-collection+analysis

For our dataset the expected intersection is none – the one url tagged as ‘analysis’ has not also been tagged ‘data-collection’.

The actual outcome is:

Supposition.

None of our test data is tagged ‘data’ so the result should be the empty set, and a search for ‘data’ delivers the empty set. However a search for the tags ‘data’ AND ‘data’ revealed the answer.

 

The Delicious search algorithm is doing partial pattern matching on any term on either side of the ‘+’ sign in the javascript arguments – a greedy search.

This renders it a little useless for our needs of displaying courses tagged with a specific match of skills

I wrote to delicious about a month ago now, but have heard nothing back so far. Writing a reminder now.

Anyone else found this or any ideas about a work around?  

43things

As a user of 43things, the goal based social network, i make lists of things i want to do. From pipe dreams to the more practical I can check out others who have achieved my goals, who are working towards them and even those who failed.

Once i complete a goal it’s there for me to see and forms part of my profile or DNA. I find the social network aspects motivating and encouraging within 43things and my tagcloud of things i have achieved makes me proud.

One day i’ll be able to add time travel to it, but for the moment i’m happy with my cloud of using wordpress as a full cms, playing a kraftwerk song on the keyboard, learn jive dancing and stop dressing like a teenager.

skillclouds

I’m currently working on a jisc project at sussex university called skillclouds.

Part of this is an investigation into presenting information, specifically ‘skills’, to students in the form of a tag cloud – hence the name skillclouds.

The current stage is looking at the project and user goal of allowing students to identify skills they have gained from a university course – not as easy as it may seem!

Student’s who have previously written CVs, applied for or had jobs find this relatively easy. Those without any experience find it a challenge recognise how the skills they have gained during their study program can be identified, expressed in a CV and applied to an application or job. The skills employers ask for are there, with examples of usage throughout a study program and extracurricular activity, the challenge is for students to acknowledge they have these demonstratable skills.