Sending "User-agent" using Requests library in Python

python, python-requests, web-crawler

Solution

The `user-agent` should be specified as a field in the header.

Here is a list of HTTP header fields, and you'd probably be interested in request-specific fields, which includes `User-Agent`.

If you're using requests v2.13 and newer

The simplest way to do what you want is to create a dictionary and specify your headers directly, like so:

import requests

url = 'SOME URL'

headers = {
    'User-Agent': 'My User Agent 1.0',
    'From': 'youremail@domain.example'  # This is another valid field
}

response = requests.get(url, headers=headers)

If you're using requests v2.12.x and older

Older versions of `requests` clobbered default headers, so you'd want to do the following to preserve default headers and then add your own to them.

import requests

url = 'SOME URL'

# Get a copy of the default headers that requests would use
headers = requests.utils.default_headers()

# Update the headers with your custom ones
# You don't have to worry about case-sensitivity with
# the dictionary keys, because default_headers uses a custom
# CaseInsensitiveDict implementation within requests' source code.
headers.update(
    {
        'User-Agent': 'My User Agent 1.0',
    }
)

response = requests.get(url, headers=headers)

Problem

I want to send a value for `"User-agent"` while requesting a webpage using Python Requests. I am not sure is if it is okay to send this as a part of the header, as in the code below: ``` debug = {'verbose': sys.stderr} user_agent = {'User-agent': 'Mozilla/5.0'} response = requests.get(url, headers = user_agent, config=debug) ``` The debug information isn't showing the headers being sent during the request. Is it acceptable to send this information in the header? If not, how can I send it?

Original source

Related problems