Using HttpWebRequest to Login to Site
I know there are many similar questions here, but I have tried almost nothing.
Basically, I have never used the method before POST
and I struggle to log into any site and maintain cookies to use in next requests.
Here's my code (I'm only using Facebook for testing):
Login method:
protected static CookieContainer Login()
{
string userName = "username";
string password = "password";
ASCIIEncoding encoding = new ASCIIEncoding();
string postData = "email=" + userName + "&pass=" + password;
byte[] postDataBytes = encoding.GetBytes(postData);
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create("https://www.facebook.com/login.php");
httpWebRequest.Method = "POST";
httpWebRequest.ContentType = "application/x-www-form-urlencoded";
httpWebRequest.ContentLength = postDataBytes.Length;
httpWebRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36";
httpWebRequest.Proxy = null;
httpWebRequest.AllowAutoRedirect = false;
using (var stream = httpWebRequest.GetRequestStream())
{
stream.Write(postDataBytes, 0, postDataBytes.Length);
stream.Close();
}
var cookieContainer = new CookieContainer();
using (var httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse())
{
using (var streamReader = new StreamReader(httpWebResponse.GetResponseStream()))
{
foreach (Cookie cookie in httpWebResponse.Cookies)
{
cookieContainer.Add(cookie);
}
}
}
return cookieContainer;
}
Get home page after login:
void Retrieve()
{
var req =(HttpWebRequest)WebRequest.Create("https://www.facebook.com");
req.Proxy = null;
req.KeepAlive = true;
req.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36";
req.CookieContainer = Login();
string src = "";
using (var res = (HttpWebResponse)req.GetResponse())
{
src = new StreamReader(res.GetResponseStream()).ReadToEnd();
}
txt.Text = src;
wb.DocumentText = src;
}
I also tried to create first CookieContainer
and then assign it to both requests, but still no luck.
I would appreciate it if someone can tell me what I am missing.
Note. As mentioned, I am using Facebook here for testing purposes only. If necessary, I can publish the original website I am trying to login to.
Update:
The actual site I'm trying to automate the login process is: carrierpoint.com .
I have to say that I don't have much experience with web requests / responses, authentication, etc. I have never worked with ASP.NET or any web application, so please excuse my limited knowledge in this area.
So, for more information, this is an example of request and response headers for a successful login to this site:
If it's not just about cookies, I would appreciate if someone can help me figure out what other elements are required to send the correct request and maintain a login session.
It would also be great if someone could suggest some resources as a good start for me to generally understand the correct ways to use the method POST
and login and authenticate the website (I mean the general concept to be able to use on other sites).
source to share
I don't see any different methods at once how you do it. Logging in isn't that straightforward for us because we don't have the credentials to conduct proper testing, and to be honest, we can't just go overboard with that. I would point out two things:
You are ASCII encoding. You must encode your data as UTF8.
System.Text.Encoding.UTF8.GetBytes
...
You need url encode your data.
postdata = "loginId=" & WebUtility.UrlEncode("myloginid") & "&password=" & WebUtility.UrlEncode("mypassword")
You may need Import System.Web
.
I would honestly confirm that Carrierpoint doesn't have some kind of developer API for you. Many large companies have some kind of API that they have been forced to develop for their operations.
You don't need to learn HttpWebRequest
, but for practice I would try using the PayPal or Stripe API. They are both REST APIs for performing CRUD operations on their respective accounts in the online payment world. They are very easy to use and they both use the OAuth and Basic Authentication forms of authentication.
In the case of PayPal, you'll send a basic authorization request to receive the OAuth token:
Dim _req As HttpWebRequest = HttpWebRequest.Create(Base_URL & "/api/v1/token")
_req.Method = "POST"
_req.Headers.Add("Authorization", "Basic " & Convert.ToBase64String(System.Text.Encoding.Default.GetBytes(ClientID & ":" & SecretKey))) 'Convert both credentials into a base 64 string
_req.AuthenticationLevel = System.Net.Security.AuthenticationLevel.MutualAuthRequested 'Sends authorization header immediately on first request
In these scripts, you will see response lines, which are often in the format JSON
:
{
"scope": "https://api.paypal.com/v1/payments/.* https://api.paypal.com/v1/vault/credit-card https://api.paypal.com/v1/vault/credit-card/.*",
"access_token": "EEwJ6tF9x5WCIZDYzyZGaz6Khbw7raYRIBV_WxVvgmsG",
"token_type": "Bearer",
"app_id": "APP-6XR95014BA15863X",
"expires_in": 28800
}
After you have received your token by parsing this JSON response, you will send it with whatever request you need to make to the api. An example of using the PayPal Payments API to get all payments in a list
Dim _req As HttpWebRequest = HttpWebRequest.Create(Base_URL & "/v1/payments/payment")
_req.Method = Method.ToUpper
_req.Headers.Add("Authorization", "Bearer " & Token) 'The access_token from the previous authentication request
_req.Headers.Add("Accept_Language", "en_US")
_req.Accept = "application/json"
Getting answers is always the same as you. The only thing you will need to do is analyze the data and make sure you know what format the data is in. If you need to parse JSON
, use Newtonsoft
NugetPackage. It is very simple for us and has a load of documentation. RestSharp is another Nuget package that contains dozens of helper methods for making RESTful requests for cases like PayPal and Stripe.
Resources
https://www.nuget.org/packages/RestSharp/
https://www.nuget.org/packages/newtonsoft.json/
https://developer.yahoo.com/dotnet/howto-rest_cs.html
If you need more help, don't be afraid to ask.
source to share
You must set the cookie container for the login request. In addition, the response handling code can be simplified, so the rest of the methods Login
look like this:
...
httpWebRequest.CookieContainer = new CookieContainer();
using (var stream = httpWebRequest.GetRequestStream())
{
stream.Write(postDataBytes, 0, postDataBytes.Length);
stream.Close();
}
using (var httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse())
{
var cookieContainer = new CookieContainer();
cookieContainer.Add(httpWebResponse.Cookies);
return cookieContainer;
}
source to share
HttpWebRequest is tricky and not web-friendly IMO. You might want to write your own helpers or use nugets.
I can successfully authenticate to facebook with the below code (based on this )
class Program
{
static void Main(string[] args)
{
// Main logic: Login && GetHomePage
var client = new FacebookClient();
if (client.Login("userMail", "userPassword"))
client.GetHomePage();
}
}
// Specific client
public class FacebookClient : CookieAwareWebClient
{
public bool Login(string email, string password)
{
var loginResult = this.Login("https://www.facebook.com/login.php",
new NameValueCollection
{
{ "email", email },
{ "pass", password }
});
return loginResult;
}
public void GetHomePage()
{
// Here the magic.. Cookies are injected via an overriden
var webRequest = this.GetWebRequest(new Uri("https://www.facebook.com"));
string src = "";
using (var webResponse = (HttpWebResponse)webRequest.GetResponse())
{
src = new StreamReader(webResponse.GetResponseStream()).ReadToEnd();
}
}
}
Here's the code from the linked post, I just added the UserAgent .. line.
// General client
public class CookieAwareWebClient : WebClient
{
public CookieContainer CookieContainer { get; private set; }
public CookieAwareWebClient()
: this(new CookieContainer())
{ }
public CookieAwareWebClient(CookieContainer container)
{
CookieContainer = container;
}
public bool Login(string loginPageAddress, NameValueCollection loginData)
{
var request = (HttpWebRequest)WebRequest.Create(loginPageAddress);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36";
var parameters = new StringBuilder();
foreach (string key in loginData.Keys)
{
parameters.AppendFormat("{0}={1}&",
HttpUtility.UrlEncode(key),
HttpUtility.UrlEncode(loginData[key]));
}
parameters.Length -= 1;
var buffer = Encoding.ASCII.GetBytes(parameters.ToString());
request.ContentLength = buffer.Length;
var requestStream = request.GetRequestStream();
requestStream.Write(buffer, 0, buffer.Length);
requestStream.Close();
request.CookieContainer = new CookieContainer();
var response = (HttpWebResponse)request.GetResponse();
response.Close();
CookieContainer = request.CookieContainer;
return response.StatusCode == HttpStatusCode.OK;
}
// Add cookies to WebRequest
protected override WebRequest GetWebRequest(Uri address)
{
var request = (HttpWebRequest)base.GetWebRequest(address);
request.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36";
request.CookieContainer = CookieContainer;
return request;
}
}
Hope this helps!
source to share