Friday, July 3, 2009

Extract Hyperlinks Using WebBrowser Object in C#

In my previous post, I mentioned using regular expressions to extract links from a web page. While I like using regular expressions, there is a much easier way of doing this in C#.net. All that you need is a 'WebBrowser' object to read a webpage and extracting all of the hyperlinks present in it. However, you need to create Windows Forms applications to use the WebBrowser object and then add the following code (on a button click event).

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;

namespace WebbrowserTest
{
public partial class WebBrowserSample : Form
{

public WebBrowserSample()
{
InitializeComponent();
}

private void button1_Click(object sender, EventArgs e)
{
WebBrowser web = new WebBrowser();
web.NewWindow += new CancelEventHandler(web_NewWindow);
web.Navigate("http://www.cnn.com");

//wait for the browser object to load the page
while (web.ReadyState != WebBrowserReadyState.Complete)
{
Application.DoEvents();
}

HtmlElementCollection ht = web.Document.Links;
List urls = new List();
foreach (HtmlElement h in ht)
{
try
{
if ((h.GetAttribute("href") != null) )
{
urls.Add(h.GetAttribute("href"));
}
}
catch
{ }
}

foreach (string url in urls)
{
Console.WriteLine(url);
}



}

void web_NewWindow(object sender, CancelEventArgs e)
{
e.Cancel = true;
}
}
}



The 'NewWindow' event is used to discard popups which may open while navigating a web page. The 'Document.Links' property of the WebBrowser is used to get the HtmlCollection of all of the links present in a webpage. Once you have the HtmlCollection of links, the links could be processes/viewed using 'href' attribute of each 'HtmlElement' in the 'HtmlCollection'.

Even though this is a pretty straight forward code, I found it a little slower. I think the reason is that you have to make sure that the webbrowser object has read the entire webpage before you can start using it (achieved by the while '(web.ReadyState != WebBrowserReadyState.Complete)' condition).

Also if you are thinking about link processing in a non-Window Forms application, then this program is not for you. For all the Window Forms application developers, enjoy the code :)

19 comments:

Jim said...

Thanks, I hadn't come across either the links collection or the new window code. Could be useful.

Syed Rizvi said...

@Jim, let me know if you find it of some use. It is always good to know how your code can be tweaked to achieve something more useful :)

Nick said...

Thanks for the idea. I just want to add that it doesn't take much effort to utilize the WebBrowser object for this purpose in a console (non winforms) application. Simply do the following 2 things:

1) Add a project reference to System.Windows.Forms

2) Mark your application entry point as single threaded apartment. (put [STAThread] above your Main function)

For a quick and dirty app I really don't see a problem with doing it this way. I imagine it's much less error prone that using regex or other methods.

Cheers.

Syed Rizvi said...

@Nick, wow! thanks for the suggestion. Will definitely try it out. Best.

Anonymous said...

seattle hindu dating http://loveepicentre.com/ gardening gifts singles dating

Anonymous said...

Hi all
I'm Mary, 26 years old :)
I am looking for good man. If he exists...
I'm love sport and... Hope you understand

[img]http://loveepicentre.com/uploades/photos/30375.jpg[/img]

www.blogger.com is very interesting forum for me, so I think we have the same hobbies
So lets chat...
My e-mail is marypopovaks@gmail.com, Skype *
Send me your foto!

__
Added later:

This photography is corrupted, sorry!

Oh, and all my photos here:
http://loveepicentre.com/user/Popova

Anonymous said...

bitter freedom ebook http://audiobookscollection.co.uk/it/Moab-Is-My-Washpot/p223922/ test ebook reader [url=http://audiobookscollection.co.uk/de/Radoslaw-Pytlak/m103893/]free english to spanish ebook[/url] sourav ganguly the fire within ebook

Anonymous said...

warez trout fishing tips ebook http://audiobooksworld.co.uk/es/authors/?letter=Ih oracle9i programming ebook [url=http://audiobooksworld.co.uk/Zoubin-Ghahramani/m135821/]ebook e-book pmp[/url] advanced masking dvd ebook free download

Anonymous said...

pandigital software http://buyoem.co.uk/product-37095/Mediachance-Dynamic-Auto-Painter-2-5-Portable bluetooth software modem [url=http://buyoem.co.uk/fr/product-37383/Crossword-Express-7-7]zetron zbase software[/url] system analyzer software
[url=http://buyoem.co.uk/product-17985/Transmit-3-5-Mac]Transmit 3.5 [Mac] - Download OEM, Software Sale, OEM Software[/url] accounting software tools
[url=http://buyoem.co.uk/fr/product-37293/TweakNow-RegCleaner-Pro-4-6-Portable][img]http://buyoem.co.uk/image/1.gif[/img][/url]

Anonymous said...

taxwise tax software http://buyoemsoftware.co.uk/de/product-15987/Goldfish-2-1-Mac wm6 audio software [url=http://buyoemsoftware.co.uk/es/product-37380/Chily-Registry-Cleaner-7-1]yearly schedule software[/url] win iso software free
[url=http://buyoemsoftware.co.uk/product-20130/AutoCAD-LT-2010-English-x32]AutoCAD LT 2010 English x32 - Software Store[/url] palm software gprs
[url=http://buyoemsoftware.co.uk/es/product-14239/Synchronize-Pro-X-5-0-Mac][img]http://buyoem.co.uk/image/5.gif[/img][/url]

Anonymous said...

best site for technical software http://buysoftwareonline.co.uk/de/product-18667/iPOD-Music-Liberator-5-2-Mac outsourcing software business [url=http://buysoftwareonline.co.uk/es/product-34189/Unified-Color-32-Float-1-0-for-Adobe-Photoshop-CS3-x32]gaurdian software for internet[/url] with instructional software
[url=http://buysoftwareonline.co.uk/es/product-24957/Adobe-Master-Collection-CS4-Mac]Adobe Master Collection CS4 [Mac] - Cheap Legal OEM Software, Software Sale, Download OEM[/url] showing point real estate software
[url=http://buysoftwareonline.co.uk/fr/manufacturer-9/Autodesk][img]http://buyoem.co.uk/image/4.gif[/img][/url]

Anonymous said...

weight lifting personal software http://buysoftwareonline.co.uk/de/product-14632/AppleWorks-6-2-Mac beyond trust software [url=http://buysoftwareonline.co.uk/product-37099/Splash-Pro-Ex-1-1]talking atomic clock software[/url] origins home embroidery auto-digitizing software
[url=http://buysoftwareonline.co.uk/product-14120/TextMate-1-5-Mac]TextMate 1.5 [Mac] - Cheap Legal OEM Software, Software Sale, Download OEM[/url] sign software
[url=http://buysoftwareonline.co.uk/product-36150/Primasoft-Member-Organizer-Deluxe-3-7][img]http://buyoem.co.uk/image/5.gif[/img][/url]

Anonymous said...

super anti virus software free http://buysoftwareonline.co.uk/de/category-1/Betriebssysteme golf cart maintenance log software [url=http://buysoftwareonline.co.uk/de/product-37297/UltimateDefrag-1-7]ses provider software[/url] what is intelligence software
[url=http://buysoftwareonline.co.uk/es/category-13/Programas-de-m-sica]Programas de musica - Software Store[/url] tanning bed management software
[url=http://buysoftwareonline.co.uk/it/product-35464/NETGATE-Spy-Emergency-9-0][img]http://buyoem.co.uk/image/4.gif[/img][/url]

Anonymous said...

[url=http://onlinemedistore.com/products/vasotec.htm][img]http://onlinemedistore.com/4.jpg[/img][/url]
organic pharmacy http://onlinemedistore.com/products/emsam.htm walmart pharmacy rancho santa fe avondale [url=http://onlinemedistore.com/categories/blood-pressure.htm]pharmacy week 2007[/url]
kirklin clinic pharmacy http://onlinemedistore.com/products/cleocin-gel.htm levitra lipitor online pharmacy pharmacy zoloft phentermine [url=http://onlinemedistore.com/products/zyrtec.htm]zyrtec[/url]
cvs pharmacy mecedon ny http://onlinemedistore.com/products/bactroban.htm online pharmacy no prescription needed [url=http://onlinemedistore.com/products/levitra-professional.htm]pharmacy school on line[/url]
online pharmacy with no prescriptions http://onlinemedistore.com/catalogue/f.htm buy link pharmacy net top ultram [url=http://onlinemedistore.com/products/prazosin.htm]prazosin[/url]

Anonymous said...

buen dato! no sabia que se podia sacar el listado de links con un metodo del objeto.

OFF: ese anonimo culiao que se meta el antivirus por la raja!

oakleyses said...

jordan shoes, ugg boots, longchamp outlet, uggs on sale, ray ban sunglasses, replica watches, ray ban sunglasses, ray ban sunglasses, christian louboutin shoes, nike free run, louboutin pas cher, michael kors pas cher, louis vuitton outlet, air max, prada outlet, tory burch outlet, christian louboutin, louis vuitton, nike air max, nike roshe, replica watches, tiffany and co, oakley sunglasses wholesale, nike free, longchamp pas cher, christian louboutin uk, louis vuitton, burberry pas cher, chanel handbags, nike outlet, nike air max, polo ralph lauren outlet online, gucci handbags, oakley sunglasses, jordan pas cher, prada handbags, oakley sunglasses, cheap oakley sunglasses, louis vuitton outlet, longchamp outlet, polo outlet, ugg boots, tiffany jewelry, longchamp outlet, christian louboutin outlet, louis vuitton outlet, sac longchamp pas cher, oakley sunglasses, polo ralph lauren

oakleyses said...

true religion jeans, ray ban pas cher, michael kors, oakley pas cher, polo lacoste, michael kors outlet online, ray ban uk, mulberry uk, michael kors outlet online, coach outlet, burberry handbags, nike air force, vans pas cher, true religion outlet, michael kors outlet, abercrombie and fitch uk, nike roshe run uk, replica handbags, burberry outlet, michael kors outlet online, kate spade, hollister pas cher, michael kors outlet online, sac hermes, nike air max uk, true religion outlet, nike air max uk, michael kors outlet, guess pas cher, north face, hollister uk, new balance, nike tn, uggs outlet, uggs outlet, ralph lauren uk, michael kors, true religion outlet, nike air max, timberland pas cher, nike blazer pas cher, sac vanessa bruno, converse pas cher, coach purses, coach outlet store online, north face uk, nike free uk, hogan outlet, michael kors outlet

oakleyses said...

wedding dresses, longchamp uk, new balance shoes, insanity workout, gucci, nike roshe run, nike huaraches, north face outlet, celine handbags, giuseppe zanotti outlet, ghd hair, ralph lauren, nike trainers uk, mac cosmetics, iphone cases, asics running shoes, nfl jerseys, mcm handbags, north face outlet, ray ban, reebok outlet, converse outlet, timberland boots, hermes belt, mont blanc pens, ferragamo shoes, hollister, hollister clothing, nike air max, abercrombie and fitch, vans outlet, vans, converse, instyler, louboutin, lululemon, p90x workout, herve leger, soccer shoes, soccer jerseys, nike air max, jimmy choo outlet, beats by dre, oakley, baseball bats, valentino shoes, babyliss, hollister, chi flat iron, bottega veneta

oakleyses said...

ugg, louis vuitton, hollister, ugg uk, links of london, canada goose outlet, toms shoes, thomas sabo, moncler outlet, doudoune moncler, coach outlet, canada goose, ugg,uggs,uggs canada, ugg,ugg australia,ugg italia, pandora charms, replica watches, louis vuitton, supra shoes, pandora jewelry, juicy couture outlet, canada goose outlet, barbour, swarovski crystal, pandora uk, montre pas cher, juicy couture outlet, lancel, canada goose, moncler, canada goose jackets, louis vuitton, canada goose uk, marc jacobs, moncler, pandora jewelry, canada goose, swarovski, canada goose outlet, moncler uk, moncler, louis vuitton, karen millen uk, ugg pas cher, louis vuitton, barbour uk, wedding dresses, moncler outlet, moncler