Python and Incapsula
python
Solution
For this I would suggest Selenium. It allows you to hook into a browser and pull everything from it. The browser is responsible for rendering the page, including any javascript, and you can then access the dom and pull what you need.
http://docs.seleniumhq.org/
Problem
I have been scraping a website and it works, but suddenly it does not work anymore which I found out incapsula protection in their website. I read the new TOS and it mentions that as long as it is within human can do limit, it is acceptable to get information from their site. But the problems now is my script encounter incapsula javascript that look like ``` var z="";var b="7472797B766172207868723B76617220743D6E6577204461746528292E67657454696D6528293B766172207374617475733D227374617274223B7661722074696D696E673D6E65772041727261792833293B77696E646F772E6F6E756E6C6F61643D66756E6374696F6E28297B74696D696E675B325D3D22723A222B286E6577204461746528292E67657454696D6528292D74293B646F63756D656E742E637265617465456C656D656E742822696D6722292E7372633D222F5F496E63617073756C615F5265736F757263653F4553324C555243543D363726743D373826643D222B656E636F6465555249436F6D706F6E656E74287374617475732B222028222B74696D696E672E6A6F696E28292B222922297D3B69662877696E646F772E584D4C4874747052657175657374297B7868723D6E657720584D4C48747470526571756573747D656C73657B7868723D6E657720416374697665584F626A65637428224D6963726F736F66742E584D4C4854545022297D7868722E6F6E726561647973746174656368616E67653D66756E6374696F6E28297B737769746368287868722E72656164795374617465297B6361736520303A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2072657175657374206E6F7420696E697469616C697A656420223B627265616B3B6361736520313A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2073657276657220636F6E6E656374696F6E2065737461626C6973686564223B627265616B3B6361736520323A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2072657175657374207265636569766564223B627265616B3B6361736520333A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2070726F63657373696E672072657175657374223B627265616B3B6361736520343A7374617475733D22636F6D706C657465223B74696D696E675B315D3D22633A222B286E6577204461746528292E67657454696D6528292D74293B6966287868722E7374617475733D3D323030297B706172656E742E6C6F636174696F6E2E72656C6F616428297D627265616B7D7D3B74696D696E675B305D3D22733A222B286E6577204461746528292E67657454696D6528292D74293B7868722E6F70656E2822474554222C222F5F496E63617073756C615F5265736F757263653F535748414E45444C3D323736303537313932393030303739323830342C31363537303433373735303338313336383533322C353931373335333737373039353633353432312C313735363339222C66616C7365293B7868722E73656E64286E756C6C297D63617463682863297B7374617475732B3D6E6577204461746528292E67657454696D6528292D742B2220696E6361705F6578633A20222B633B646F63756D656E742E637265617465456C656D656E742822696D6722292E7372633D222F5F496E63617073756C615F5265736F757263653F4553324C555243543D363726743D373826643D222B656E636F6465555249436F6D706F6E656E74287374617475732B222028222B74696D696E672E6A6F696E28292B222922297D3B";for (var i=0;i<b.length;i+=2){z=z+parseInt(b.substring(i, i+2), 16)+",";}z = z.substring(0,z.length-1); eval(eval('String.fromCharCode('+z+')')); ``` and when I decode it, i found ``` eval( 'try{var xhr;var t=new Date().getTime();var status="start";var timing=new Array(3);window.onunload=function(){timing[2]="r:"+(new Date().getTime()-t);myvar="/_Incapsula_Resource?ES2LURCT=67&t=78&d="+encodeURIComponent(status+" ("+timing.join()+")")};if(window.XMLHttpRequest){xhr=new XMLHttpRequest}else{xhr=new ActiveXObject("Microsoft.XMLHTTP")}xhr.onreadystatechange=function(){switch(xhr.readyState){case 0:status=new Date().getTime()-t+": request not initialized ";break;case 1:status=new Date().getTime()-t+": server connection established";break;case 2:status=new Date().getTime()-t+": request received";break;case 3:status=new Date().getTime()-t+": processing request";break;case 4:status="complete";timing[1]="c:"+(new Date().getTime()-t);if(xhr.status==200){parent.location.reload()}break}};timing[0]="s:"+(new Date().getTime()-t);xhr.open("GET","/_Incapsula_Resource?SWHANEDL=2760571929000792804,16570437750381368532,5917353777095635421,175639",false);xhr.send(null)}catch(c){status+=new Date().getTime()-t+" incap_exc: "+c;myvar="/_Incapsula_Resource?ES2LURCT=67&t=78&d="+encodeURIComponent(status+" ("+timing.join()+")")};' ); ``` however evaluating the statement resulting in error using pyV8 ``` ReferenceError: ReferenceError: document is not defined ( @ 1 : 68 ) -> send(null)}catch(c){status+=new Date().getTime()-t+" incap_exc: "+c;document.c ``` It seems that PyV8 does not create document object so it fails, when I replace the document.createElement("img").src to something like a variable, the variable will have string ``` myvar = "/_Incapsula_Resource?ES2LURCT=67&t=78&d=start0%20incap_exc%3A%20ReferenceError%3A%20window%20is%20not%20defined%20(%2C%2C)" ``` it seems that it knows that there is no window exist, how do I scrape a site that uses incapsula like this?