Tuesday, November 16, 2010

Using Language Models for Information Retrieval (Python)

Continuing my regard for Bayes' Theorem, I decided to write a small python program that will crawl a list of urls and then will allow the user to search that list. I used the query likelihood Language Model for information retrieval.

A good explanation of query likelihood can be found in the Manning's Introduction to Information Retrieval book. I used the example mentioned in the "Language models for information retrieval" chapter. Here is the excerpt from that chapter:

In the query likelihood model, we construct from each document d in the collection a language model Md. We rank documents by P(d | q), where the probability of a document is interpreted as the likelihood that it is relevant to the query. Using Bayes rule , we have:

P(d | q) = P(q | d) P(d) / P(q)

P(q) is the same for all documents, and so can be ignored. The prior probability of a document P(d) is often treated as uniform across all d and so it can also be ignored.


In this model, Zero probabilities of words can be a problem. This can be solved by smoothing. Smoothing not only avoids zero probability but also it provides a term weighting component. More information about smoothing can be found in Manning's book.

I implemented the query likelihood model in python. I used BeautifulSoap to parse the crawled urls. You can download the python code from here. Once you have downloaded the code you can then use the code as follows:

1. import LangModel

2. Define a list of urls you want to search:
u=['http://cnn.com','http://techcrunch.com']

3. Call the crawl function to crawl the list:
LangModel.crawl(u)

4. Type the query you want to search and press the return key

5. The output will show you a ranked list of urls and the score of each according to query likelihood model.

6. You can search the list again by using the search function:
LangModel.search('your query')


That is it. Another application of Bayes' Theorem!

Stay tuned for more python code :)

61 comments:

ريم كوكب تانى said...

Thanks alot for this post

Anonymous said...

[url=http://www.ukgamingstore.co.uk/]gaming pc[/url]

[url=http://www.ukgamingstore.co.uk/intel-core-i7-gaming-computers]core i7 gaming computer uk[/url]

[url=http://www.ukgamingstore.co.uk/intel-core-i5-gaming-laptops]intel core i5 gaming laptop uk[/url]

[url=http://www.ukgamingstore.co.uk/gaming-mouses]gaming mouses uk[/url]

Anonymous said...

Backpacking travels normally include adventure sports and these sorts of sports normally pose a high risk of injury; if a backpacker cheap insurance does not provide personal accident cover for adventure sports its best to move on to another agency. best offshore vpn. As the gusty wind is a biggest concern while suing a hanging banner outside, grommets are a real big help to place them safe and sound. no vpn

Anonymous said...

otk dating defined http://loveepicentre.com/success_stories.php gay teen dating websites

Anonymous said...

The cool screen gives your pc a run for its money., 789 vpn. Check on your state's insurance policy as well. There are several websites that offer escort services. vpn access methods. And the separated practice of listening and speaking is proved to be very inefficient. errore sistema 53 vpn

Anonymous said...

Better to be a direct and frank person than falsehood., how to install a software. You could also experiment with your font styles in various portions of your material just to highlight certain topics. If the short term value is far different from the long term value, may be it's offering you a chance to place a trade. openvpn sample config. The reason for this is that if you do not do a very good job in spinning your article there is the danger of it being flagged as duplicate content. l2tp vpn ubuntu

Anonymous said...

Apt implementation can thus help in turning the same into a driving force. america coasters network. All women are not equal. waitt farm network

Anonymous said...

Their choice of glasses can be easily procured as the presence on dealers offering eyeglass online has increased considerably. You then don’t have the opportunity to go for other interviews that might already be set up. linkstation live hs dhgl gb network. There are many offensive odor sources, which can not only gross you out, but harm you: one of them is mold. This relates to one valuable advantage of the system., ethernet network controller drivers. First will the device be comfortable enough to tolerate having in your mouth all night? raylene sex virtual

Anonymous said...

Over time this will be used as a means of helping to get the teeth to work properly in conjunction with other parts of the jaw. A child's ability to understand and follow rules will be a big factor in outdoor play, for example, in that the child must understand to stay out of the road, not speak to strangers, and not enter 'off limits' areas like the garden shed or pool. extranet caprabo. The demand for nurses is continuously rising. Other phrases used are wheals and nettle rash., mr access point. In this time of luxurious capitalistic situation, people are always searching for the most excellent way to avail things at a cheap price. virtual dog world

Anonymous said...

Mobile phones are continually enhanced by cellular phone companies but you will never know any of these hottest developments if you do not have any access to mobile phone reviews and news including the information that matters to you. free down load latin music

Anonymous said...

The company enterprise marketing management network must be secure and optimized which means a regular and reiterative process which analyzes and monitors existing network activity and usage. com sun net ssl internal www protocol https

Anonymous said...

If it does not happen, then she can go in for the surgery., work socks. Maxell continually looks to provide its clients with the advanced, cost effective and secure archival media at an incredible value. Let us decide to make a left turn. app intranet inv searchaspx. Oak, on the other hand, has larger, broader grain patterns that give off more of a rustic, homey feel. mary kay virtual makeover

Anonymous said...

[url=http://loveepicentre.com/][img]http://loveepicentre.com/uploades/photos/4.jpg[/img][/url]
free gay lesbian dating site [url=http://loveepicentre.com/]indiana dating michigan city[/url] lesbian speed dating long beach ca
carter oosterhouse dating [url=http://loveepicentre.com/map.php]double your dating derek rake[/url] bbw dating in nc
mutual love dating [url=http://loveepicentre.com/]affairs dating service[/url] schwinn giant dating serial numbers

Anonymous said...

[url=http://loveepicentre.com/advice.php][img]http://loveepicentre.com/uploades/photos/7.jpg[/img][/url]
milton bradley puzzles dating [url=http://loveepicentre.com/]gay dating sitex[/url] scottish dating websites
100 free adult dating sites uk [url=http://loveepicentre.com/taketour.php]totally 100 free dating[/url] survey on internet dating
on line dating scott white [url=http://loveepicentre.com/map.php]dating calling after a year[/url] dating tips pakistani men

Anonymous said...

ebook babysitters club rar http://audiobookscollection.co.uk/Men-Are-from-Mars-Women-Are-from-Venus-A-Practical-Guide-for-Improving-Communication-and-Getting-What-You-Want-in-Your-Relationships/p171471/ ebook reader for droid [url=http://audiobookscollection.co.uk/The-psychology-of-culture-shock/p119904/]ebook publishing business secure-ebook web protection[/url] chronicles of prydain ebook

Anonymous said...

[url=http://www.51soboo.com]百家乐论坛,百家乐园,百家乐技巧[/url]


百家乐论坛

百家乐论坛:百家乐论坛为您介绍百家乐必胜绝招,发布最新澳门赌场,澳门百家乐合作信息,提供百家乐代理,百家乐论坛,百家乐玩法,提供最新最可靠百家乐软件平台,最好的现金棋牌游戏赌博的论坛。

百家乐论坛,百家乐园,百家乐技巧,百家乐技术交流,百家乐在线试玩的网址,欢迎您来我们百家乐网站参观试玩

Anonymous said...

ceinture louis vuitton pas cher [url=http://www.lvspeedy.fr]louis vuitton pas cher homme[/url] hogvfp louis vuitton [url=http://www.cuirlv2013.fr]louis vuitton pas cher homme[/url] yplgnb

Anonymous said...

I do not even know the way I ended up here, but I thought this post was
great. I do not understand who you're but certainly you're going to a famous blogger if you aren't already. Cheers!

Feel free to surf to my web blog :: profitbetkz.com

Anonymous said...

It�s really a cool and helpful piece of information. I am
glad that you shared this helpful information with us.
Please keep us informed like this. Thanks for sharing.


Take a look at my blog post http://geraldmakes9161.wordpress.com/2013/01/19/dating-rules-for-single-moms/

Anonymous said...

These days of austerity along with relative anxiety about
getting debt, many people balk up against the idea of making use of a credit
card to make purchase of merchandise as well as pay for any occasion,
preferring, instead only to rely on the particular tried in addition
to trusted means of making repayment - raw cash.
However, if you have the cash available to make the purchase fully, then,
paradoxically, that is the best time for you to use the credit card for several reasons.


Here is my web blog; http://dailynest.com/

Anonymous said...

Excellent blog! Do you have any hints for aspiring writers?
I'm hoping to start my own website soon but I'm a little lost on everything.
Would you suggest starting with a free platform like Wordpress or go for a paid option?
There are so many options out there that I'm totally overwhelmed .. Any tips? Many thanks!

Feel free to surf to my web blog; find a date

Anonymous said...

One thing I'd prefer to say is car insurance termination is a feared experience and if you are doing the proper things as being a driver you won't get
one. Lots of people do are sent the notice that they are officially dumped by their particular insurance company they then have to struggle to get extra insurance after
having a cancellation. Low-cost auto insurance rates are frequently hard to get from cancellation.
Understanding the main reasons for auto insurance cancellations can help individuals prevent completely losing in one
of the most crucial privileges accessible. Thanks for the thoughts shared by
your blog.

Here is my page: dating website for adults

Anonymous said...

When I initially commented I clicked the -Notify me when
new comments are added- checkbox and now every time a
remark is added I get four emails with the same comment.

Is there any method you may take away me from that service?
Thanks!

Feel free to visit my webpage - dating and chat

Anonymous said...

Thank you for this article. I will also like to say that it can often be hard
when you're in school and just starting out to initiate a long credit standing. There are many learners who are just simply trying to survive and have long or positive credit history can occasionally be a difficult factor to have.

My page cowib.csa.ou.edu

Anonymous said...

I have observed that in the world today, video games would be
the latest rage with children of all ages. There are times when it may be impossible
to drag the kids away from the activities. If you want the best of both
worlds, there are various educational gaming activities for
kids. Great post.

My homepage; http://www.essweb.com/NYF2010/index.php?option=com_blog&view=comments&pid=28665&Itemid=0

Anonymous said...

Hello, I think your web site could be having internet
browser compatibility problems. When I take a look at your site
in Safari, it looks fine but when opening in Internet Explorer, it has some overlapping
issues. I merely wanted to give you a quick heads up!
Other than that, fantastic website!

Feel free to visit my blog - Best online backup

Anonymous said...

Hello there! This is my first visit to your blog! We are a collection of volunteers and starting a new initiative in a community in the same niche.
Your blog provided us valuable information to work on.
You have done a outstanding job!

My web site; adult finder

Anonymous said...

I have taken note that of all different types of insurance,
health care insurance is the most marked by controversy because
of the discord between the insurance company's necessity to remain making money and the client's
need to have insurance. Insurance companies' earnings on health plans are very low, thus some corporations struggle to profit. Thanks for the concepts you share through this web site.

my homepage: adult contacts

Anonymous said...

Hello my loved one! I wish to say that this post is awesome, great
written and come with approximately all important infos.
I�d like to peer extra posts like this .

Feel free to surf to my homepage: cyber sex

Anonymous said...

I am truly pleased to read this weblog posts which contains
plenty of valuable facts, thanks for providing these information.


My page: graduate certificate programs

Anonymous said...

I read this post fully conceгnіng the rеsemblance οf most recent аnd precedіng technologiеs, it's remarkable article.

Here is my blog post; Cheappersonalloans.Me.Uk

Anonymous said...

Thanks designed for sharing such a pleasant thought, piece
of writing is fastidious, thats why i have read it entirely

Here is my homepage ... Etc Clothing

Anonymous said...

тут на официальном портале вы сумеете увидеть большой выбор [url=http://www.filmkpktut.ru/]Скачать mp4 видео[/url].

Anonymous said...

newspaper articles on interracial dating http://loveepicentre.com/contact/ dating for friends single
online dating romance [url=http://loveepicentre.com/success_stories/]international dating service to find tranny[/url] black sexy dating
dating single dad daughter [url=http://loveepicentre.com]dating in your sixties[/url] ucsc dating [url=http://loveepicentre.com/user/theliveone200/]theliveone200[/url] the right age to be dating

Anonymous said...

puerto rico dating online http://loveepicentre.com/articles/ n dating
four bases in dating [url=http://loveepicentre.com/map/]dating strong guy gay[/url] peru silver spoon 925 dating
gretchen is dating slate [url=http://loveepicentre.com/taketour/]free online dating nj[/url] japan tokyo dating tokyo girls [url=http://loveepicentre.com/user/scratchy/]scratchy[/url] dateinasia fake dating

Anonymous said...

jewish dating switzerland http://loveepicentre.com/ tubely dating network
dating russell miller army barbie [url=http://loveepicentre.com/advice/]brian kresge dating[/url] dating customs in india
male 45 pulaski va dating [url=http://loveepicentre.com/map/]the dating game icp mp3[/url] adult people finder online dating [url=http://loveepicentre.com/user/joycekumah/]joycekumah[/url] parents against interracial dating

Anonymous said...

Wow, fantastic blog format! How long have you ever been blogging for?
you make blogging glance easy. The total glance of your site
is great, as neatly as the content!

My web site - facebook sex (http://www.facebookofsex.adultcrowd.com)

web lol said...

kul post

oakleyses said...

jordan shoes, ugg boots, longchamp outlet, uggs on sale, ray ban sunglasses, replica watches, ray ban sunglasses, ray ban sunglasses, christian louboutin shoes, nike free run, louboutin pas cher, michael kors pas cher, louis vuitton outlet, air max, prada outlet, tory burch outlet, christian louboutin, louis vuitton, nike air max, nike roshe, replica watches, tiffany and co, oakley sunglasses wholesale, nike free, longchamp pas cher, christian louboutin uk, louis vuitton, burberry pas cher, chanel handbags, nike outlet, nike air max, polo ralph lauren outlet online, gucci handbags, oakley sunglasses, jordan pas cher, prada handbags, oakley sunglasses, cheap oakley sunglasses, louis vuitton outlet, longchamp outlet, polo outlet, ugg boots, tiffany jewelry, longchamp outlet, christian louboutin outlet, louis vuitton outlet, sac longchamp pas cher, oakley sunglasses, polo ralph lauren

oakleyses said...

true religion jeans, ray ban pas cher, michael kors, oakley pas cher, polo lacoste, michael kors outlet online, ray ban uk, mulberry uk, michael kors outlet online, coach outlet, burberry handbags, nike air force, vans pas cher, true religion outlet, michael kors outlet, abercrombie and fitch uk, nike roshe run uk, replica handbags, burberry outlet, michael kors outlet online, kate spade, hollister pas cher, michael kors outlet online, sac hermes, nike air max uk, true religion outlet, nike air max uk, michael kors outlet, guess pas cher, north face, hollister uk, new balance, nike tn, uggs outlet, uggs outlet, ralph lauren uk, michael kors, true religion outlet, nike air max, timberland pas cher, nike blazer pas cher, sac vanessa bruno, converse pas cher, coach purses, coach outlet store online, north face uk, nike free uk, hogan outlet, michael kors outlet

oakleyses said...

wedding dresses, longchamp uk, new balance shoes, insanity workout, gucci, nike roshe run, nike huaraches, north face outlet, celine handbags, giuseppe zanotti outlet, ghd hair, ralph lauren, nike trainers uk, mac cosmetics, iphone cases, asics running shoes, nfl jerseys, mcm handbags, north face outlet, ray ban, reebok outlet, converse outlet, timberland boots, hermes belt, mont blanc pens, ferragamo shoes, hollister, hollister clothing, nike air max, abercrombie and fitch, vans outlet, vans, converse, instyler, louboutin, lululemon, p90x workout, herve leger, soccer shoes, soccer jerseys, nike air max, jimmy choo outlet, beats by dre, oakley, baseball bats, valentino shoes, babyliss, hollister, chi flat iron, bottega veneta

oakleyses said...

ugg, louis vuitton, hollister, ugg uk, links of london, canada goose outlet, toms shoes, thomas sabo, moncler outlet, doudoune moncler, coach outlet, canada goose, ugg,uggs,uggs canada, ugg,ugg australia,ugg italia, pandora charms, replica watches, louis vuitton, supra shoes, pandora jewelry, juicy couture outlet, canada goose outlet, barbour, swarovski crystal, pandora uk, montre pas cher, juicy couture outlet, lancel, canada goose, moncler, canada goose jackets, louis vuitton, canada goose uk, marc jacobs, moncler, pandora jewelry, canada goose, swarovski, canada goose outlet, moncler uk, moncler, louis vuitton, karen millen uk, ugg pas cher, louis vuitton, barbour uk, wedding dresses, moncler outlet, moncler

UAE Escorts Hub Agency said...

Wow Amazing Work I have recently found such a nice blog and been reading various articles on your blog. Nice work Keep it up.

Good luck!

Dubai Escorts

Psophon Pro said...

Thanks for this awesome information And Guys here is the app 8 ball pool apk mod plus data unlimited coins full download

Escorts Hut said...

Thank you, I have recently been searching for information about this topic for ages and yours is the best I have discovered so far. keep Sharing

Indian Escorts in Dubai

Kings of Dubai said...

yet I need to give you guidance that dont overlook this stage.
Indian Escorts in Dubai

We Escorts said...

Following 6 hours of nonstop Googleing, finally I got it in your site.
Indian Escorts in Dubai
Pakistani Escorts in Dubai

Kings of Dubai said...

On this blog somebody expounds on individual suppositions, exercises, and encounters.
Indian Escorts in Dubai

Aliya Khan said...

Pakistani Escort In Dubai//
Cheap Escort In Dubai //
Pakistani Escort In Dubai//
Indian Escort In Dubai //

Samrina Rajpoot said...

I have recently found such a nice blog and been reading various articles on your blog.

Good Luck !

Escorts in Dubai

We Escorts said...

Thanks for sharing this Information with us.
Indian Escorts in Dubai

We Escorts said...

The excellent and decent post I found this much informative.
Indian Escorts in Dubai

We Escorts said...

You ought to compose all the more fascinating and valuable articles like this one.
Indian Escorts in Dubai

We Escorts said...

I adored it your post. Much thanks to you So Much. This is clearly one wonderful post.
Indian Escorts in Dubai

Unknown said...

Really informative article post.Much thanks again. Really Cool.

Delhi Escorts Service
Independent Escorts in Delhi
Escort Agency in Delhi
Female Escorts in Delhi
Delhi Escort Service

Rinki Sharma said...

Best Indian Escorts in Dubai +971588129541
Indian Escorts in Dubai
Dubai Escorts
Escorts in Dubai

Pakistani Escorts in Dubai said...

It's really a cool and useful piece of info. I am glad that you shared this useful information with us. Please keep us up to date like this. Thanks for sharing Pakistani Escorts In Dubai

Pakistani Escorts in Dubai said...

You ought to compose all the more fascinating and valuable articles like this one.
Pakistani Escorts In Dubai

Dubai Escort Model said...

Exceptionally Nice Shoe Design I need to get this likewise thanks for finding a pleasant shoe planet. I Love these fashionable shoes. Indian Escorts In Dubai

TheIndianEscortsinDubai said...

Decent Blog, You always post something new and irrelevant to others. You are doing it exceptionally well.
Indian Escorts in Dubai

Pakistani Escorts in Dubai said...

Looking for Best Escorts? Call for getting Cheap and best Escorts anywhere in UAE
Pakistani Escorts In Dubai