Getting Interlanguage Links from a Wiki dump
api, dump, hyperlink, wikidata, wikipedia
Solution
A really excellent library for easily dealing with Wikidata dumps is Wikidata Toolkit, which abstract away a lot details for you. In the latest release `0.3` there is a growing collection of example scripts that help with basic tasks like yours. In the examples readme we find `SitelinksExample.java`:
This program shows how to get information about the site links that are used in Wikidata dumps. The links to Wikimedia projects use keys like "enwiki" for English Wikipedia or "hewikivoyage" for Hebrew WikiVoyage. To find out the meaning of these codes, and to create URLs for the articles on these projects, Wikidata Toolkit includes some simple functions that download and process the site links information for a given project. This example shows how to use this functionality.
Problem
I am trying to extract the interlanguage links from Wikipedia dumps. It seems that these links were moved to WikiData Project, and the access is provided only via API. This branch explains how to deal with the issue and suggests to switch to the API: Retrieving the Interlanguage links from an exported Wikipedia article? However, the scope of my research seems to be too large to use the web API (millions of queries). Does anyone know whether it is possible to extract these links from anywhere except API? Parsing the dump of any size is more preferrable than querying the API. Wikipedia dumps I used: http://dumps.wikimedia.org/backup-index.html WikiData dump I used: http://dumps.wikimedia.org/wikidatawiki/latest/