Utilizing Session Objects in Python Requests Library - Part 1

Intro

When it comes to making web requests in Python, the requests library is my go to option. It’s especially handy for working with APIs. Some APIs might require certain parameters to remain consistent across multiple requests. For instance, you might need to use the same User-Agent string for a specialized tool you’re developing or add a custom header to every request your Python application sends out.

To handle these scenarios efficiently, the Sessions object is invaluable. It allows you to persist parameters across requests and maintain cookies within the Session instance. Additionally, if you’re making multiple requests to the same host, the underlying TCP connection will be reused, boosting performance significantly.

The Session object comes with all the methods of the main Requests API, making it a seamless switch for enhancing your scripts. I’ve recently incorporated Session objects into my Python projects, and here’s how you can do it too.

Why use Session?

Let’s call a URL https://httpbin.org/cookies/set/myCustomCookie/abcd1234 that sets a cookie. This will set the cookie myCustomCookie with a value of abcd1234.

Calling the https://httpbin.org/cookies URL will display the cookies sent by the browser.

Let’s do this using the requests library.

import requests

cookie_setting_url = 'https://httpbin.org/cookies/set/myCustomCookie/abcd1234'

# set cookie
resp = requests.get(cookie_setting_url, allow_redirects=False)

We can see the cookies that were sent by the server.

# print headers
print(resp.headers)

{
   "Date":"Thu, 16 May 2024 20:38:25 GMT",
   "Content-Type":"text/html; charset=utf-8",
   "Content-Length":"223",
   "Connection":"keep-alive",
   "Server":"gunicorn/19.9.0",
   "Location":"/cookies",
   "Set-Cookie":"myCustomCookie=abcd1234; Path=/",
   "Access-Control-Allow-Origin":"*",
   "Access-Control-Allow-Credentials":"true"
}

We can also verify the cookies in our CookieJar.

# cookiejar
print(resp.cookies)
<RequestsCookieJar[<Cookie myCustomCookie=abcd1234 for httpbin.org/>]>

# access the cookie
print(resp.cookies.get('myCustomCookie'))
'abcd1234'

Now, let’s make a request to an endpoint that displays the set cookies and print the response.

cookie_printing_url = 'https://httpbin.org/cookies'

resp = requests.get(cookie_printing_url)

print(resp.json())

{
   "cookies":{}
}

We get an empty {}. What happened? Although the cookies were set by the server, we need to manage them manually when using the requests library without the Session object . The Session object can help us manage these cookies.

Let’s retry this again using the Session object.

from requests import Session

cookie_setting_url = 'https://httpbin.org/cookies/set/myCustomCookie/abcd1234'

my_session = Session()

resp = my_session.get(cookie_setting_url)

Again, we confirm the cookies that were sent by the server.

# print headers
print(resp.headers)

{
   "Date":"Thu, 16 May 2024 21:01:52 GMT",
   "Content-Type":"text/html; charset=utf-8",
   "Content-Length":"223",
   "Connection":"keep-alive",
   "Server":"gunicorn/19.9.0",
   "Location":"/cookies",
   "Set-Cookie":"myCustomCookie=abcd1234; Path=/",
   "Access-Control-Allow-Origin":"*",
   "Access-Control-Allow-Credentials":"true"
}

We can also verify the cookies in our CookieJar.

# cookiejar
print(resp.cookies)
<RequestsCookieJar[<Cookie myCustomCookie=abcd1234 for httpbin.org/>]>

# access the cookie
print(resp.cookies.get('myCustomCookie'))
'abcd1234'

Now, let’s retry the request that displays the set cookies and print the response.

cookie_printing_url = 'https://httpbin.org/cookies'

resp = my_session.get(cookie_printing_url)

print(resp.json())

{
   "cookies":{
      "myCustomCookie":"abcd1234"
   }
}

To summarize, our session my_session saved the cookies set by the server and then used those cookies to make further requests to the same server.

Use case: Set custom headers

This is how you can use the Session object to set custom authentication headers to your web requests.

from requests import Session

# Custom headers
headers = {
    "User-Agent": "Mozilla/999.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0",
    "Accept": "application/json, text/plain, */*",
    "My-Custom-Header": "berserk_turing"
}

# Create a Session object
romantic_galileo_session = Session()

# Update headers
romantic_galileo_session.headers.update(headers)

headers_printing_url = 'https://httpbin.org/headers'

resp = romantic_galileo_session.get(headers_printing_url)

print(resp.json())

{
   "headers":{
      "Accept":"application/json, text/plain, */*",
      "Accept-Encoding":"gzip, deflate",
      "Host":"httpbin.org",
      "My-Custom-Header":"berserk_turing",
      "User-Agent":"Mozilla/999.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0",
      "X-Amzn-Trace-Id":"Root=1-664677bb-6a1e47654f884f662f4c603c"
   }
}

Use case: Set custom cookies

from requests import Session

# Custom cookies
cookies = {
    "sessionCookie1": "123456789",
    "sessionCookie2": "987654321",
    "myCustomCookie": "happy-hippo"
}

# Create a Session object
sharp_payne_session = Session()

# Update cookies
sharp_payne_session.cookies.update(cookies)

Use case: Set custom proxies

from requests import Session

# Custom proxies
proxies = {
  'http': 'http://10.10.1.10:3128',
  'https': 'http://10.10.1.10:1080',
}

# Create a Session object
drunk_williams_session = Session()

# Update proxies
drunk_williams_session.proxies.update(proxies)

In the next part of Session objects, we will explore session mounting and its uses.

References

Python Requests - Session Objects

Intro#

Why use Session?#

Use case: Set custom headers#

Use case: Set custom cookies#

Use case: Set custom proxies#

References#