Monday, August 17, 2009

Classic Vector Space Model in C#

Vector Space Model (VSM) is widely used in Information Retrieval systems. The model creates a space in which both documents and queries are represented by vectors. VSM treats the query as a document as well and then it determines similarity between query and other documents by calculating the deviation of angles between each document vector and the original query vector. It then uses the deviation to rank documents.

I implemented the model in C# using the example described in Mi Islita. The program will display elements of a dictionary object that hold the name of each document and its similarity value.

You can use any sorting algorithm to sort the dictionary by the similarity values. The program also assumes that there is a string array "docs" which contains the query's (at zero index) and the documents' text.

Here is the code:


using System;
using System.Collections.Generic;
using System.Text;
using System.Text.RegularExpressions;
using System.Collections;
using System.Linq;
namespace VectorSpaceModel
{
class Program
{
static Hashtable DTVector = new Hashtable(); //Hashtable to hold Document Term Vector
static List wordlist = new List(); //List of terms found in documents
static Dictionary sortedList = new Dictionary(); //Documents ranked by VSM with angle value
static string[] docs ={"gold silver truck", //Query
"shipment of gold damaged in a fire",//Doc 1
"delivery of silver arrived in a silver truck", //Doc 2
"shipment of gold arrived in a truck"}; //Doc 3

static void Main(string[] args)
{
createWordList();
createVector();
classify();

var dict = sortedList;
foreach (var x in dict.Reverse())
{
Console.WriteLine("{0} -> Doc{1}", x.Key, x.Value);
}


Console.ReadLine();
}


public static void createWordList()
{
foreach (string doc in docs)
{
wordlist = getWordList(wordlist, doc);
}
}

public static List getWordList(List wordlist, string query)
{
Regex exp = new Regex("\\w+", RegexOptions.IgnoreCase);
MatchCollection MCollection = exp.Matches(query);

foreach (Match match in MCollection)
{
if (!wordlist.Contains(match.Value))
{
wordlist.Add(match.Value);
}
}

return wordlist;
}

public static void createVector()
{
double[] queryvector;

for (int j = 0; j < docs.Length; j++)
{
queryvector = new double[wordlist.Count];

for (int i = 0; i < wordlist.Count; i++)
{

double tfIDF = getTF(docs[j], wordlist[i]) * getIDF(wordlist[i]);
queryvector[i] = tfIDF;
}

if (j == 0) //is it a query?
{
DTVector.Add("Query", queryvector);
}
else
{
DTVector.Add(j.ToString(), queryvector);
}
}
}

public static void classify()
{
double temp = 0.0;

IDictionaryEnumerator _enumerator = DTVector.GetEnumerator();

double[] queryvector = new double[wordlist.Count];

Array.Copy((double[])DTVector["Query"], queryvector, wordlist.Count);

while (_enumerator.MoveNext())
{
if (_enumerator.Key.ToString() != "Query")
{
temp = cosinetheta(queryvector, (double[])_enumerator.Value);

sortedList.Add(temp, _enumerator.Key.ToString());

}
}
}

public static double dotproduct(double[] v1, double[] v2)
{
double product = 0.0;
if (v1.Length == v2.Length)
{
for (int i = 0; i < v1.Length; i++)
{
product += v1[i] * v2[i];
}
}
return product;
}

public static double vectorlength(double[] vector)
{
double length = 0.0;
for (int i = 0; i < vector.Length; i++)
{
length += Math.Pow(vector[i], 2);
}

return Math.Sqrt(length);
}
private static double getTF(string document, string term)
{
string[] queryTerms = Regex.Split(document, "\\s");
double count = 0;


foreach (string t in queryTerms)
{
if (t == term)
{
count++;
}
}
return count;

}

private static double getIDF(string term)
{
double df = 0.0;
//get term frequency of all of the sentences except for the query
for (int i = 1; i < docs.Length; i++)
{
if (docs[i].Contains(term))
{
df++;
}
}

//Get sentence count
double D = docs.Length - 1; //excluding the query

double IDF = 0.0;

if (df > 0)
{
IDF = Math.Log(D / df);
}

return IDF;
}

public static double cosinetheta(double[] v1, double[] v2)
{
double lengthV1 = vectorlength(v1);
double lengthV2 = vectorlength(v2);

double dotprod = dotproduct(v1, v2);

return dotprod / (lengthV1 * lengthV2);

}
}
}

24 comments:

Anonymous said...

Hey There. I discovered your weblog using msn. That is a very smartly written article.

I'll make sure to bookmark it and return to learn more of your helpful info. Thanks for the post. I'll definitely return.


Also visit my web site :: ideal waist to hip ratio

Anonymous said...

' Vegetables ѕhould make uρ a large part of yоur diet
if yߋu are trүing tο lose weight. The scientific community fօund that Garcinia Cambogia Һad ɑll thesе fat-busting qualities.
Sߋme drugs prevent absorption οf fat and glucose fгom intestine,
ѕo that some fat is excreted.

Мy blog ... วิธีลดน้ําหนักเร่งด่วน

Anonymous said...

People do not do enough to retain their hair healthy. Nonetheless, they're close enough to chickens and a number
of the images appear to be chickens. You'll have it as a topic
for the birthday party.

Review my web blog :: coloring pages xmas tree

Anonymous said...

The news breaking bad supporters have been waiting on for months
has finally been uncovered. We would like benefit; great effects, such as for instance, great storytelling, and identity advancement.


my homepage; http://movie2k-movie4k.webstarts.com/

Anonymous said...

This is the perfect wallet for "clubbing" or hiking when you don't want to hassle with a purse.

The Apple boot emblem will translate into a Skull emblem
during boot. It can be downloaded by opening Cydia and lookup for the Winterboard.



Also visit my page: ifunbox ()

Anonymous said...

Making money is not guaranteed with any opportunity, but it is
possible. Brainstorm ways to make your life experience work for you.

When it comes to ways to earn extra money
online you best believe that there are ways out there.


My site: ways to make extra money

Anonymous said...

That way you can choose the ones that can't get to you when you need
them. Quite often they are manufactured by the original manufacturer.
Other times, body shop mechanics will ensure you that they only
use OEM body parts, but you still need to be diligent and examine the final work.


Feel free to surf to my web blog; advance auto parts

Anonymous said...

eхcellent issues altogether, you smply received a brand nnew readеr.

What might you recommend in egards to your submit that you mde some days in the past?

Any certain?

Here iѕ my paցe :: Pain relief

Anonymous said...

Its lіke you read my mind! You ѕeеm tto know sso much аbout thiѕ, like you wrote
the book in it oг something.I think that you can do with
a few pics to drivе the meѕsаge home a little bit, but instead of that,
this is excellent blog. A great read. I wipl certainly be
back.

Review my homepage; living with Rheumatoid Arthritis

Anonymous said...

Heya! I just wanted to ask if you ever have any trouble with hackers?

My last blog (wordpress) was hacked and I ended up losing many
months of hard work due to no backup. Do you have any
solutions to prevent hackers?

Here is my web site locksmith reading

Anonymous said...

You might probably mow two of your neighbors grass or shovel a couple of driveways to have enough cash raised for the Minecraft accounts,
you could jump over the insane level of hoops how the plethora involving internet con artists might have you bouncing through, only to
determine that right now there aren’t virtually any working rules available, or you could
simply utilize our generator and obtain a working code in less than one minute of your time.
The single-player story method features company new people who appear to be meant for future Combat on the internet games, especially two Delta Force middle management
codenamed Ice and Sandman. The most recent version of Minecraft involves seventeen diverse varieties of creatures.


My blog :: Minecraft Cracked Exe 1.5.2

Anonymous said...

Somebody essentially assist to make critically articles I might
state. This is the very first time I frequented your web page
and up to now? I surprised with the research you made
to make this particular put up amazing. Wonderful job!


my homepage - Erotic Massage London ()

Anonymous said...

Chanel Handbags Burberry Outlet Mac Cosmetics Wholesale Nike Air Max Oakley Sunglasses Outlet Gucci Outlet CHI Flat Iron Website Jimmy Choo Shoes Online Polo Ralph Lauren Gucci Outlet Celine Bag Air Max Oakley Sunglasses Wholesale Celine Outlet Hermes Outlet Celine Bags Giuseppe Zanotti Sneakers On Sale Hermes Outlet Christian Louboutin Outlet Online Toms Outlet Prada Handbags Outlet Nike Air Max Polo Ralph Lauren Outlet Kevin Durant Shoes Kate Spade Outlet Gucci Handbags
Lebron James Shoes For Sale
Polo Ralph Lauren Outlet Prada Handbags Outlet Prada Handbags

Anonymous said...

I have read so many articles on the topic of the blogger lovers
except this paragraph is genuinely a nice paragraph,
keep it up.

Here is my webpage życzenia na walentynki (soto.comunitac.ipbackup.irontec.com)

Anonymous said...

are a teenaged line socio-economic class, do not lack to be
cognisant of. forever prevent or exposure out a new period.
role player a aggregation of habiliment pants with a lithesome visualize, but if you bequeath the steadying thoughts joint
in this substance. Before fashioning a buy out online, Cheap Jerseys Wholesale Jerseys From China Paypal Jerseys China Cheap Jerseys Free Shipping Cheap NFL Jerseys
Cheap MLB Jerseys with Free Shipping Wholesale Jerseys China China Wholesale Jerseys
Cheap NFL Jerseys Online Cheap Jerseys Wholesale NFL Jerseys China Paypal Jerseys China World Cup Jerseys
Cheap MLB Jerseys Cheap NFL Jerseys China $15 ones is not the clause
preceding, property has to psychoanalyse
a condition your contract amount of money for checkup costs,
although it should be avoided. This makes you happen to
be sect aground with no restrictions. Now title
see to it or with a pendant that matches yourmatches appurtenance.
This is because rely cards

Coach Factory said...

new balance pas cher, nike air max, vans shoes, north face, michael kors, ralph lauren pas cher, vanessa bruno, michael kors, sac guess, nike free, hollister, sac louis vuitton, nike blazer, michael kors, hollister, mulberry, barbour, oakley pas cher, longchamps, lululemon, nike roshe run, nike roshe, timberland, karen millen, nike trainers, louis vuitton, vans pas cher, hollister, ray ban sunglasses, hermes pas cher, ralph lauren, abercrombie and fitch, sac louis vuitton, north face, converse pas cher, nike free pas cher, longchamp, air jordan, hogan outlet, longchamp, louboutin, nike huarache, tn pas cher, lacoste, ray ban pas cher, louis vuitton uk, air max, montre homme, burberry, air force

Coach Factory said...

tiffany and co, burberry outlet online, louboutin outlet, ray ban sunglasses, michael kors outlet online, longchamp handbags, jordan shoes, true religion jeans, christian louboutin shoes, coach outlet, polo ralph lauren outlet, air max, prada handbags, kate spade outlet, nike shoes, louis vuitton outlet online, louboutin, coach outlet, michael kors outlet, michael kors outlet, kate spade handbags, louis vuitton handbags, louis vuitton outlet stores, true religion outlet, louis vuitton outlet, michael kors outlet, nike free, coach purses, nike air max, longchamp handbags, coach factory outlet, oakley sunglasses, polo ralph lauren outlet online, louboutin shoes, burberry outlet online, longchamp outlet, tory burch outlet, louis vuitton, chanel handbags, gucci outlet, oakley sunglasses, michael kors outlet online, michael kors outlet, ray ban sunglasses, prada outlet, coach outlet store online, cheap oakley sunglasses

Coach Factory said...

tiffany and co, oakley sunglasses, prada outlet, polo ralph lauren outlet online, oakley sunglasses, coach outlet, louis vuitton handbags, louis vuitton outlet stores, louboutin, gucci outlet, longchamp handbags, louis vuitton, louboutin outlet, cheap oakley sunglasses, louis vuitton outlet, michael kors outlet, nike air max, tory burch outlet, air max, longchamp outlet, michael kors outlet, michael kors outlet, michael kors outlet online, christian louboutin shoes, prada handbags, louis vuitton outlet online, kate spade outlet, michael kors outlet, chanel handbags, ray ban sunglasses, nike free, longchamp handbags, ray ban sunglasses, jordan shoes, louboutin shoes, kate spade handbags, burberry outlet online, coach purses, burberry outlet online, nike shoes, true religion outlet, coach outlet, coach factory outlet, coach outlet store online, true religion jeans, polo ralph lauren outlet, michael kors outlet online

Anonymous said...

GHG new automatic Hair curler
Does your blog have a contact page? I'm having trouble locating it but, I'd like to send you an e-mail.
I've got some creative ideas for your blog you might be interested in hearing.
Either way, great site and I look forward to seeing it improve over time.

Venu Gopalan said...

cleaning services providers in mumbai | cctv camera dealers | security guard agency in mumbai | printed bag manufacturers | tiffin services for offices | best hair care products | online dry fruits | skin care products | deodorant for women |

oakleyses said...

louis vuitton outlet, polo outlet, longchamp outlet, tiffany jewelry, uggs on sale, louboutin pas cher, chanel handbags, gucci handbags, longchamp pas cher, louis vuitton outlet, nike free, ugg boots, burberry pas cher, louis vuitton, polo ralph lauren outlet online, nike roshe, michael kors pas cher, christian louboutin shoes, prada outlet, cheap oakley sunglasses, oakley sunglasses, replica watches, ray ban sunglasses, replica watches, christian louboutin outlet, jordan shoes, prada handbags, sac longchamp pas cher, longchamp outlet, oakley sunglasses, louis vuitton, ray ban sunglasses, nike free run, air max, nike air max, christian louboutin, oakley sunglasses, christian louboutin uk, nike outlet, oakley sunglasses wholesale, kate spade outlet, ray ban sunglasses, longchamp outlet, tory burch outlet, jordan pas cher, ugg boots, nike air max, polo ralph lauren, louis vuitton outlet

oakleyses said...

hollister pas cher, nike free uk, kate spade, guess pas cher, sac hermes, nike air max uk, michael kors outlet, vans pas cher, nike roshe run uk, michael kors outlet online, sac vanessa bruno, michael kors outlet, abercrombie and fitch uk, coach outlet, hogan outlet, uggs outlet, michael kors, michael kors outlet online, oakley pas cher, nike air max, mulberry uk, nike air max uk, true religion jeans, coach purses, hollister uk, burberry outlet, michael kors outlet online, new balance, nike tn, ralph lauren uk, ray ban uk, converse pas cher, true religion outlet, michael kors outlet online, north face, true religion outlet, polo lacoste, michael kors, michael kors outlet, true religion outlet, timberland pas cher, uggs outlet, coach outlet store online, north face uk, nike air force, burberry handbags, ray ban pas cher, lululemon canada, nike blazer pas cher

oakleyses said...

giuseppe zanotti outlet, ray ban, celine handbags, ghd hair, asics running shoes, gucci, babyliss, bottega veneta, converse, lululemon, hollister clothing, instyler, soccer shoes, mac cosmetics, nfl jerseys, ralph lauren, hollister, mont blanc pens, baseball bats, wedding dresses, mcm handbags, jimmy choo outlet, hermes belt, oakley, iphone cases, north face outlet, ferragamo shoes, longchamp uk, p90x workout, valentino shoes, vans outlet, vans, north face outlet, nike air max, herve leger, nike air max, hollister, new balance shoes, nike huaraches, beats by dre, chi flat iron, nike trainers uk, insanity workout, soccer jerseys, abercrombie and fitch, reebok outlet, timberland boots, converse outlet, louboutin, nike roshe run

oakleyses said...

pandora jewelry, canada goose outlet, juicy couture outlet, canada goose, moncler, moncler, juicy couture outlet, canada goose outlet, montre pas cher, pandora jewelry, swarovski crystal, marc jacobs, pandora charms, moncler, canada goose, ugg, swarovski, doudoune moncler, barbour uk, ugg,ugg australia,ugg italia, toms shoes, moncler uk, hollister, canada goose outlet, louis vuitton, wedding dresses, moncler, canada goose, ugg pas cher, louis vuitton, moncler outlet, canada goose jackets, replica watches, links of london, moncler outlet, canada goose uk, karen millen uk, louis vuitton, lancel, ugg uk, louis vuitton, louis vuitton, pandora uk, supra shoes, barbour, coach outlet, ugg,uggs,uggs canada, thomas sabo