Streaming Responses in Requests

When working with large files or long-running requests, streaming responses can help manage memory usage efficiently. By using the stream=True parameter in Requests, you can process data incrementally without loading the entire content into memory.

Handling Large Files with Streaming

Streaming allows you to download and process large files in chunks rather than loading them entirely into memory.

Example: Downloading a Large File

import requests
 
url = "https://example.com/largefile.zip"
 
# Stream the response
response = requests.get(url, stream=True)
 
if response.status_code == 200:
    for chunk in response.iter_content(chunk_size=1024):
        print(f"Downloaded {len(chunk)} bytes")

Writing Streamed Responses to a File

To save a streamed response to a file, you can write chunks of data to the file incrementally.

Example: Saving a File

import requests
 
url = "https://example.com/largefile.zip"
 
# Stream the response
response = requests.get(url, stream=True)
 
if response.status_code == 200:
    with open("largefile.zip", "wb") as file:
        for chunk in response.iter_content(chunk_size=1024):
            file.write(chunk)
    print("File saved successfully.")
else:
    print("Failed to download the file.")

Best Practices for Streaming

• Chunk Size: Adjust chunk_size based on the size of your file and available memory. Larger chunk sizes may improve performance but use more memory.
• Response Validation: Always check response.status_code before processing.
• Timeouts: Use a timeout to prevent hanging requests.

Try It Yourself

Problem 1: Stream and Save a Text File

Download and save the file from https://httpbin.org/stream/20 to output.txt.

import requests
 
url = "https://httpbin.org/stream/20"
response = requests.get(url, stream=True)
 
if response.status_code == 200:
    with open("output.txt", "w") as file:
        for line in response.iter_lines():
            file.write(line.decode("utf-8") + "\n")
    print("File saved successfully.")
else:
    print("Failed to download the file.")

Problem 2: Download an Image File

Stream and save an image from https://httpbin.org/image/png to image.png.

import requests
 
url = "https://httpbin.org/image/png"
response = requests.get(url, stream=True)
 
if response.status_code == 200:
    with open("image.png", "wb") as file:
        for chunk in response.iter_content(chunk_size=1024):
            file.write(chunk)
    print("Image saved successfully.")
else:
    print("Failed to download the image.")