Google Trends Quota Limit

google-api, limit, quota

Solution

I'm struggling with the same issue! From your question I can't figure out what stage have you achieved... But here is the solution that I've found:

- You should emulate browser with cookies. I think the best way to do it is to use Mechanize library.

- At first your program should "login" using GET request to "https://accounts.google.com/Login?hl=en"

- Immediately after that you can access some other personal resources, but not google trends!

- After some significant time you can successfully get google trends data as CSV.

- I still have not discovered the exact time period, but it is more than 10 minutes and less than several hours :). That is why saving your cookies for latter use is a good idea!

Few more tips:

If you are developing using python / ruby under Windows do not forget to set up CA ROOT certificates package for OpenSSL library. Otherwise HTTPS connection will fail and you won't login! See Getting the `certificate verify failed (OpenSSL::SSL::SSLError)` erro with Mechanize object

I recommend you to save cookies to external file at program shutdown. And restoring them at startup.

Do not forget to allow redirects, because Google is using redirects all the time.

Ruby code example:

require 'mechanize'
require 'logger'
begin
  agent = Mechanize.new { |a|
    a.user_agent = 'Opera/9.80 (Windows NT 5.1) Presto/2.12.388 Version/12.16'

    cert_store = OpenSSL::X509::Store.new
    cert_store.add_file 'cacert.pem'
    a.cert_store = cert_store

    a.log = Logger.new('mech.log')

    if File.file?('mech.cookies')
      cookies = Mechanize::CookieJar.new
      cookies.load('mech.cookies')
      a.cookie_jar = cookies
    end

    a.open_timeout = 5
    a.read_timeout = 6
    a.keep_alive   = true
    a.redirect_ok  = true
  }

  LOGIN_URL = "https://accounts.google.com/Login?hl=en&continue=http://www.google.com/trends/"
  login_page = agent.get(LOGIN_URL)
  login_form = login_page.forms.first
  login_form.Email = *
  login_form.Passwd = *
  login_response_page = agent.submit(login_form)

  page = agent.get(url)

  # DO SOME TRENDS REQUESTS AFTER SIGNIFICANT PERIOD OF TIME

ensure
  if agent
    agent.cookie_jar.save('mech.cookies')
  end
end

Problem

Original source