Working with Proxies in Python Requests

Proxies act as intermediaries between your application and the target server, often used to enhance security, bypass restrictions, or balance network traffic. The Requests library in Python provides robust support for working with proxies.

Configuring Proxies in Requests

You can configure proxies in Requests using the proxies parameter. The proxies dictionary maps protocols (e.g., http, https) to the proxy URLs.

Example: Configuring a Proxy

import requests
 
# Define proxy URLs
proxies = {
    "http": "http://10.10.1.10:3128",
    "https": "https://10.10.1.11:1080"
}
 
# Send a request using proxies
response = requests.get("http://example.com", proxies=proxies)
print(response.status_code)

Authenticating with a Proxy

If the proxy requires authentication, include the credentials in the proxy URL:

proxies = {
    "http": "http://user:password@10.10.1.10:3128",
    "https": "https://user:password@10.10.1.11:1080"
}

Using Environment Variables for Proxies

Requests can automatically use proxies specified in your system environment variables. Supported environment variables include HTTP_PROXY, HTTPS_PROXY, and NO_PROXY.

Example: Setting Environment Variables

Set the variables in your terminal or script:

Linux/Mac:

export HTTP_PROXY="http://10.10.1.10:3128"
export HTTPS_PROXY="https://10.10.1.11:1080"
export NO_PROXY="localhost,127.0.0.1,.example.com"

Windows (Command Prompt):

set HTTP_PROXY=http://10.10.1.10:3128
set HTTPS_PROXY=https://10.10.1.11:1080
set NO_PROXY=localhost,127.0.0.1,.example.com

Example: Using Environment Variables in Code

When environment variables are set, Requests automatically uses them:

import requests
 
# Send a request without explicitly specifying proxies
response = requests.get("http://example.com")
print(response.status_code)

Tips for Working with Proxies

• Testing Connectivity: Test the proxy server independently to ensure it is functioning before integrating it into your application.
• Exception Handling: Wrap your requests in try-except blocks to handle potential connectivity issues with the proxy.
• Proxy Rotation: For scraping or automation tasks, consider rotating proxies to avoid detection or throttling.

By effectively configuring and managing proxies, you can enhance your application’s networking capabilities, improve security, and optimize resource usage.