Consuming APIs with Requests
HTTP for Humans. How to make your Python scripts talk to the rest of the internet.
1. The Big Idea (ELI5)
👶 Explain Like I'm 10: The Universal Browser
Your Python script is stuck inside your computer. To get data from Instagram, Google, or ChatGPT, it needs to go online.
- The Browser: When you open Chrome and type
google.com, you are sending a GET Request. The server sends back HTML. - Requests Library: This library lets your Python script act like a Browser. It can "visit" web pages, "fill out" forms (POST), and download files, all without a screen.
2. The Basics: GET and POST
99% of the web is just these two verbs.
PYTHON
import requests
# 1. GET: "Give me data"
# We ask GitHub for info about a user
resp = requests.get("https://api.github.com/users/rohit")
if resp.status_code == 200:
data = resp.json() # Parse JSON automatically
print(f"User found: {data['name']}")
# 2. POST: "Here is new data"
# We send a dictionary (JSON) to a server
payload = {"username": "rohit", "message": "hello world"}
resp = requests.post("https://httpbin.org/post", json=payload)
print(resp.json()) # Server confirms it received our data3. Headers: The ID Card
Some websites block scripts. To pretend to be a real browser, you need to send a User-Agent Header.
PYTHON
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)...",
"Authorization": "Bearer my-secret-api-token"
}
requests.get("https://api.netflix.com/movies", headers=headers)4. Deep Dive: Performance (Session Objects)
Beginners use requests.get() directly. Professionals use Sessions.
Why? Creating a connection (TCP Handshake + SSL) is slow. A Session keeps the connection open (Keep-Alive) for subsequent requests. It is 5x-10x faster for bulk operations.
PYTHON
# ⌠SLOW: Opens and closes 10 separate connections
for i in range(10):
requests.get("https://api.my-app.com/data")
# ✅ FAST: Reuses 1 connection
session = requests.Session()
for i in range(10):
session.get("https://api.my-app.com/data")5. Production Safety: Timeouts & Retries
The internet is unreliable. Without a Timeout, your script might hang FOREVER if a server stops responding.
PYTHON
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
# 1. Define Retry Logic
# "Try 3 times. Wait 1s, then 2s, then 4s (Backoff)"
retry_strategy = Retry(
total=3,
backoff_factor=1,
status_forcelist=[500, 502, 503, 504]
)
# 2. Mount it to a Session
session = requests.Session()
adapter = HTTPAdapter(max_retries=retry_strategy)
session.mount("https://", adapter)
# 3. Use it with a Timeout!
try:
# If server takes > 5s, it raises a Timeout Error
session.get("https://api.slow-site.com", timeout=5)
except requests.exceptions.Timeout:
print("Server is too slow!")