Custom HTTPS POST queries in Ruby
Complementing both the manual for the Net::HTTP class and Rail’s guide to rendering proxied pages (from the Rails wiki) here is a short script, which makes use of a custom made HTTPS POST query (btw. HTTP POST is basically the same, you just have to remove the ssl declaration and change the destination port).
First, the script makes a GET query to the target host (in the example, it is the login page for Wirtualna Polska webmail). In the next step it reads which cookies the website wants to set. After that, we can make our POST query. We send both filled in POST parameters (and hidden inputs too) and all of the cookies, which would normally be set in the browser. After that the script just prints out response headers and data (if there’s any).
The solution is simple, short and clean, although it took me few hours to find out that POST parameters must be a &-separated string and that the headers is a hash array. After that, it was a piece of cake.
Anyway, here’s the script:
#!/usr/bin/ruby
require 'net/http'
require 'net/https'
http = Net::HTTP.new('profil.wp.pl', 443)
http.use_ssl = true
path = '/login.html'
# GET request -> so the host can set his cookies
resp, data = http.get(path, nil)
cookie = resp.response['set-cookie']
# POST request -> logging in
data = ’serwis=wp.pl&url=profil.html&tryLogin=1&countTest=1&logowaniessl=1&login_username=blah&login_password=blah’
headers = {
‘Cookie’ => cookie,
‘Referer’ => ‘http://profil.wp.pl/login.html’,
‘Content-Type’ => ‘application/x-www-form-urlencoded’
}
resp, data = http.post(path, data, headers)
# Output on the screen -> we should get either a 302 redirect (after a successful login) or an error page
puts ‘Code = ‘ + resp.code
puts ‘Message = ‘ + resp.message
resp.each {|key, val| puts key + ‘ = ‘ + val}
puts data

June 3rd, 2006 at 10:08 am
Hey, thanks for this, it’s really useful! Was wondering if you knew of any conveneint way to follow redirects?
June 3rd, 2006 at 11:08 am
It sould be fairly easy to combine my code with the redirect implementation from ruby-doc (scroll down one screen).
June 6th, 2006 at 2:41 am
Was playing with that for awhile and then I just discovered WWW::Mechanize. Have you seen this thing? It’s totally amazing. It was so easy, I took the liberty of rewriting your script using it like so:
require ‘rubygems’
require ‘mechanize’
agent = WWW::Mechanize.new
agent.user_agent_alias = ‘Mac Safari’
page = agent.get(’http://profil.wp.pl/login.html’)
form = page.forms[0]
form.fields.find{|f| f.name == ‘login_username’}.value = “blah”
form.fields.find{|f| f.name == ‘login_password’}.value = “blah”
page = agent.submit(form)
puts page.body
So easy! You can also turn on verbose logging by changing that first line to:
agent = WWW::Mechanize.new {|a| a.log = Logger.new(STDERR) }. Plus it does link-clicking and all kinds of other fancy stuff. It’s the best thing since sliced-bread.
June 6th, 2006 at 7:00 pm
Sounds very interesting. I must try it out.
October 27th, 2006 at 2:32 am
[...] Custom HTTPS POST queries in Ruby (tags: ruby programming) [...]
July 10th, 2007 at 3:39 pm
Wow, this turns out to be _extremely_ helpful!!! I WWW::Mechanize takes care of all the dirty work for you. I was able to use it to construct a pseudo-API to a site that doesnt have one
January 4th, 2008 at 3:07 pm
Hi, I am trying to post form data from a jsp page which is using http method GET . How can I use it and get response .