[Tutorial] Get All URLs on a Page

hacks · 05-23-2008, 06:06 AM

In this article, I show a class that can be used to find and display all of the urls on a web page. What for you may ask? Well, in my experience as a web developer, I have found a class like this to be very useful. Sometimes, you may want to use this class a a basis for a more complex application that crawls your site checking for bad or broken links. In other cases, you may simply want to check an individual page to make sure your links are formatted correctly, or don’t contain any obsolete pages. You could also easily change this class to look for other items within your page, like specific text or tags. Who knows, this may be the start of a specialized spider that crawls sites on the internet looking for something specific.

I think you get the picture. Of course, to make this class do all those wonderful things, you would have to expand on what I am presenting here. However, I believe this is a good start. The class has one public method - RetrieveUrls. The method calls two private methods. The RetrieveContents method will issue a request to the web page, and retreive the contents. The GetAllUrls method will use a regular expression to find all of the urls on the page. This method writes the matches to the screen, as well as saving them in a log file. Of course, if you prefer, you could modify the method to save the matches somewhere else, like an array or a database table.

Using the code

Code:

</p>
<p>GetUrls urls = new GetUrls();</p>
<p>urls.RetrieveUrls(”http://www.microsoft.com”);</p>
<p>

The class is listed below. Have fun!

Code:

<br />
using System;<br />
using System.Collections.Generic;<br />
using System.Text;<br />
using System.Net;<br />
using System.IO;<br />
using System.Text.RegularExpressions;</p>
<p>namespace FindAllUrls<br />
{<br />
class GetUrls<br />
{</p>
<p>//public method called from your application<br />
public void RetrieveUrls( string webPage )<br />
{<br />
GetAllUrls(RetrieveContent(webPage));<br />
}</p>
<p>//get the content of the web page passed in<br />
private string RetrieveContent(string webPage)<br />
{<br />
HttpWebResponse response = null;//used to get response<br />
StreamReader respStream = null;//used to read response into string<br />
try<br />
{<br />
//create a request object using the url passed in<br />
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(webPage);<br />
request.Timeout = 10000;</p>
<p>//go get a response from the page<br />
response = (HttpWebResponse)request.GetResponse();</p>
<p>//create a streamreader object from the response<br />
respStream = new StreamReader(response.GetResponseStream());</p>
<p>//get the contents of the page as a string and return it<br />
return respStream.ReadToEnd();<br />
}<br />
catch (Exception ex)//houston we have a problem!<br />
{<br />
throw ex;<br />
}<br />
finally<br />
{<br />
//close it down, we’re going home!<br />
response.Close();<br />
respStream.Close();<br />
}<br />
}</p>
<p>//using a regular expression, find all of the href or urls<br />
//in the content of the page<br />
private void GetAllUrls( string content )<br />
{<br />
//regular expression<br />
string pattern = @”(?:href\s*=)(?:[\s""']*)(?!#|mailto|location.|javascript|.*css|.*this\.)(?<br />
.*?)(?:[\s>""'])”;</p>
<p>//Set up regex object<br />
Regex RegExpr = new Regex(pattern, RegexOptions.IgnoreCase);</p>
<p>//get the first match<br />
Match match = RegExpr.Match(content);</p>
<p>//loop through matches<br />
while (match.Success)<br />
{</p>
<p>//output the match info<br />
Console.WriteLine(”href match: ” + match.Groups[0].Value);<br />
WriteToLog(”C:\matchlog.txt”, “href match: ” + match.Groups[0].Value + “\r\n”);</p>
<p>Console.WriteLine(”Url match: ” + match.Groups[1].Value);<br />
WriteToLog(”C:\matchlog.txt”, “Url | Location | mailto match: ” + match.Groups[1].Value + “\r\n”);</p>
<p>//get next match<br />
match = match.NextMatch();<br />
}<br />
}</p>
<p>//Write to a log file<br />
private void WriteToLog(string file, string message)<br />
{<br />
using (StreamWriter w = File.AppendText(file))<br />
{<br />
w.WriteLine(DateTime.Now.ToString() + “: ” + message); w.Close();<br />
}<br />
}<br />
}<br />
}<br />

chan001 · 09-03-2008, 10:03 AM

what this stand for? can give a more clarify works infos?

Thanks and advance i wan to know this...