Screen Scrape Experienced? - fresh eyes needed

Mod, Add-on & other Development discussion
Post Reply
Martin
Site Admin
Site Admin
Posts: 1854
Joined: Wed Jun 17, 2009 6:30 pm
Location: South Yorkshire UK
Contact:

Screen Scrape Experienced? - fresh eyes needed

Post by Martin »

I've got a custom bit of software that I've been writing for myself to deal with the complete lack of API that almost all UK couriers and parcel delivery services use but I've hit something of a dead end.

My code scrapes a particular delivery service form which has dynamically assigned field names but constant id values then uses the correct field names along with a curl lib to submit the data I need to provide to get a quote... (eg: weight, country, postcode, etc..) but for some reason the CURL library doesn't appear to be submitting the correct information or is including some kind of tell tale because instead of submitting the form properly it just reverts back to the initial form all over again as if no values have been submitted.

Now, I'm pretty sure a lot of work went into making the form/page I'm trying to work with, very difficult to do this sort of thing with so if anyone has a good grounding in screen scraping and CURL in particular I would be very grateful for any tips and a fresh eye...

I hasten to add this code is intended for my own personal use so, if you don't have the skills, please don't ask for a copy...

Martin
Site Admin
Site Admin
Posts: 1854
Joined: Wed Jun 17, 2009 6:30 pm
Location: South Yorkshire UK
Contact:

Re: Screen Scrape Experienced? - fresh eyes needed

Post by Martin »

Mulling things over to myself so bear with me...

Up until now I've been using the standard CURL implementation and not thought about the trail the system would leave...

I'm beginning to wonder if the problem might be partially because the referrer information being passed when submitting the form is google.com and not the URL of the form itself..

Martin
Site Admin
Site Admin
Posts: 1854
Joined: Wed Jun 17, 2009 6:30 pm
Location: South Yorkshire UK
Contact:

Re: Screen Scrape Experienced? - fresh eyes needed

Post by Martin »

Hmm... nope that didn't seem to make any difference...

I've a feeling I'm missing something blindingly obvious but whatever it is, it's annoying...

It'll work fine in a browser so I wonder if there's any way to track what's going on with the browser interaction that will allow me to examine the log and try to duplicate it with CURL...

Firebug may be useful but I don't know how to use it for this... Hmmm again... :|

Post Reply