App Engine's URLFetch: http GET works locally but not when deployed to GAE on a specific URL
get, google-app-engine, http, java
Solution
We had something similar here a few months ago. In the end the mystery was that the site redirected to itself and expected to see some cookie it set back. But the urlfetch redirect handling doesn't send any cookies it receives. It's possible that the urlfetch emulation when running locally does something different with cookies.
If you can't get this to work, you may want to turn off redirect following in urlfetch and manage redirects and cookies yourself.
Problem
The Problem: Locally my application works fine. My HTTP GET returns code 200 and a final URL of: http://vow.mlspin.com/clients/index.aspx? The rest of my application works great too. When I deploy my code to GAE servers (I use the Eclipse plugin to do my deployment) my application ceases to work because the wrong html page is returned! the return code is still 200, but the URL is now (final URL): http://vow.mlspin.com/clients/signin.aspx?id= My Question is: Is there a problem with redirects? Are the google app engine servers somehow blacklisted? What am I doing wrong here? Has anyone encountered this before? The closest question I've found was this one: Http GET from GAE Java I've implemented this suggestion but it hasn't worked for me so far. Thank you all in advance! Additional Info -> below are HTTPResponse headers from the same exact GET request, one from the local deployment and the other from the GAE deployment. Local HTTP Response Headers ``` Date :: Tue, 24 Apr 2012 04:12:32 GMT Server :: Microsoft-IIS/6.0 X-Powered-By :: ASP.NET X-AspNet-Version :: 2.0.50727 P3P :: CP="NOI DSP COR NID ADMa OPTa OUR NOR" Cache-Control :: no-cache Pragma :: no-cache Expires :: -1 Content-Type :: text/html; charset=utf-8 Content-Length :: 133704 ``` Deployed HTTP Response Headers ``` date :: Tue, 24 Apr 2012 04:11:19 GMT server :: Microsoft-IIS/6.0 x-powered-by :: ASP.NET x-aspnet-version :: 2.0.50727 p3p :: CP="NOI DSP COR NID ADMa OPTa OUR NOR" cache-control :: private content-type :: text/html; charset=utf-8 content-length :: 4991 x-google-cache-control :: remote-fetch via :: HTTP/1.1 GWA ``` How I'm crafting my requests: First I tried the easy way ``` Document doc = Jsoup.connect(baseMLSURL).get(); ``` then I tried to go low level and just use java.net ``` private String getHttpFromServer(String url) throws IOException log.severe("getting http from: "+ url); StringBuilder sb = new StringBuilder(); URL yahoo = new URL(url); URLConnection yc = yahoo.openConnection(); yc.setRequestProperty("Host", "vow.mlspin.com"); yc.setRequestProperty("User-Agent", "Mozilla/5.0 (X11; Linux i686; rv:7.0.1) Gecko/20100101 Firefox/7.0.1"); BufferedReader in = new BufferedReader( new InputStreamReader( yc.getInputStream())); String inputLine; while ((inputLine = in.readLine()) != null) { sb.append(inputLine.replaceAll(" ", " ")+"\r\n"); } in.close(); return sb.toString(); } ``` Finally I've also tried to use Google's URLFetcher ``` private String getHttpUsingFetchService(String url) throws MalformedURLException, IOException { URLFetchService fetchService = URLFetchServiceFactory.getURLFetchService(); HTTPResponse targetResponse = fetchService.fetch(new URL(url)); // Error log.severe("Code returned from request: "+targetResponse.getResponseCode()); log.severe("final URL: "+targetResponse.getFinalUrl()); String result = new String(targetResponse.getContent()); return result.replaceAll(" ", " "); } ```