Parallelizing many GET requests

Question

Is there any efficient way to parallelize a large numer of GET requests in Java. I have a file with 200,000 lines, each one needing a GET request from Wikimedia. And then I have to write a part of the response to a common file. I've pasted the main part of my code below as reference. ``` while ((line = br.readLine()) != null) { count++; if ((count % 1000) == 0) { System.out.println(count + " tags parsed"); fbw.flush(); bw.flush(); } //System.out.println(line); String target = new String(line); if (target.startsWith("\"") && (target.endsWith("\""))) { target = target.replaceAll("\"", ""); } String url = "http://en.wikipedia.org/w/api.php?action=query&prop=revisions&format=xml&rvprop=timestamp&rvlimit=1&rvdir=newer&titles="; url = url + URLEncoder.encode(target, "UTF-8"); URL obj = new URL(url); HttpURLConnection con = (HttpURLConnection) obj.openConnection(); // optional default is GET con.setRequestMethod("GET"); //add request header //con.setRequestProperty("User-Agent", USER_AGENT);…

Accepted Answer

If you really, really need to do it via GET requests, I recommend that you use a ThreadPoolExecutor with a small thread pool (2 or 3) to avoid overloading the wikipedia servers. That will avoid a lot of coding ...

Also consider using the Apache HttpClient libraries (with persistent connections!).

But a much better idea to use the database download option. Depending on what you are doing, you may be able to choose one of the smaller downloads. This page discusses the various options.

Note: that Wikipedia prefers people to download the database dumps (etcetera) rather than pounding on their web servers.

Parallelizing many GET requests

Solution

Problem