Crawling web page with c#

Here's an example of how to crawl a web page using HTTP in C#:

using System;
using System.Net;
using System.IO;
using System.Text;
 
namespace WebCrawler
{
    class Program
    {
        static void Main(string[] args)
        {
            string url = "https://www.example.com";
            string html = GetHtml(url);
            Console.WriteLine(html);
        }
 
        static string GetHtml(string url)
        {
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
            HttpWebResponse response = (HttpWebResponse)request.GetResponse();
            StreamReader reader = new StreamReader(response.GetResponseStream(), Encoding.UTF8);
            string html = reader.ReadToEnd();
            return html;
        }
    }
}

In this example, we first define a URL that we want to crawl (https://www.example.com). We then call the GetHtml method, which takes a URL as input and returns the HTML content of the page as a string.

In the GetHtml method, we create an HttpWebRequest object using the URL as the input, and then call the GetResponse method to send the request and get a response. We then read the response stream using a StreamReader object, and return the HTML content as a string.

You can modify this example to crawl multiple web pages by creating a list of URLs and calling the GetHtml method for each URL in a loop. You can also add additional code to parse the HTML content and extract information from the web pages.