Python requests to access OAUTH website content - SNL Finance

oauth, python, python-requests

Solution

I ended up figuring this out. I underestimated Python's requests library.

It appears to be as easy as this:

# prep token request data
request_data = {
    "client_id": client_id,
    "client_secret": client_secret,
    "scope": "https://www.snl.com",
    "grant_type": "refresh_token",
    "refresh_token": new_refresh_token
}

# post to token url with token credentials
# the request object stores the token response cookies
r1 = requests.post(token_url, data=request_data)

# post to protected url by setting cookies arg as cookies from previous response
r2 = requests.post(protected_url, cookies=r1.cookies)

Problem

I've been banging my head up against the wall trying to retrieve content from the news source "SNL Finance". I have valid credentials, so in theory I should be able to programmatically access their news content. In short, I've tried executing the below script but with no success: ``` s = requests.Session() client_id = "..." client_secret = "..." token_url = "https://www.snl.com/SNL.Services.Security.Service/oauth/token" protected_url = "https://www.snl.com/web/client?auth=inherit#news/article?id=40666532&KeyProductLinkType=14" request_data = { "client_id": client_id, "client_secret": client_secret, "scope": "https://www.snl.com", "grant_type": "refresh_token", "refresh_token": refresh_token } token_response = s.post(token_url, data=request_data) ### token response is in jwt format, including token_type, expires_in, scope, etc. ### token = json.loads(token_response.text)["access_token"].split('>')[1].split('<')[0] request_data["token"] = token article = s.post(protected_url, headers=request_data) ``` Sadly, this fails. I end up with a 200 response, but it appears to just be the login page (honestly not entirely sure what I'm looking at). For more background, I've included browser information as it populates throughout the authentication process: Attempt to visit protected url, redirected to the below url (omitted snl base): ``` /web/client?auth=inherit&contextType=external&username=string&enablePersistentLogin=true&OverrideRetryLimit=0&SwitchGetToPostLimit=50000&contextValue=%2Foam&password=secure_string&challenge_url=https%3A%2F%2Fwww.snl.com%2Fweb%2Fclient%3Fauth%3Dinherit&request_id=-6149669210818920852&authn_try_count=0&locale=en_US&resource_url=https%253A%252F%252Fwww.snl.com%252FInteractiveX%252FDefault.aspx%253Ftarget%253Dnews%25252Farticle%25253Fid%25253D40666532%252526KeyProductLinkType%25253D14%2526SNL3%253D1 ``` Request headers are shown here. - Upon entering login / password, a token is retrieved and the protected page loads. Request cookies are as shown here. Also, I'm a bit confused at to why the token value of `SNL_OAUTH_TOKEN` in the above link (second link) differs from what is shown in the jwt token response I receive from my script. Any guidance here will be hugely appreciated. I'm also happy to send any other non-personal information that proves useful. Thank you!

Original source