Scrapy proxy setup: Use meta proxy for unique IPs per request

How does Scrapy handle proxies?

Scrapy enables HttpProxyMiddleware by default. Add a proxy address to the request's meta dictionary, and the middleware converts username/password to Proxy-Authorization headers. HTTPS uses CONNECT tunnels.

Scrapy's default download handlers support HTTP proxies. For SOCKS5 (requiring custom handlers), we recommend using the same IP's HTTP port - target sites see the same IP.

The example uses IP 198.51.100.23, HTTP port 8000, username spx_user, and password spx_pass.

pip install scrapy
scrapy startproject shop
cd shop

How to add proxy to a single request?

Add proxy to meta when creating a Request in start_requests. Use the same method when requesting the next page in the callback. To maintain the same session, pass the proxy value from the previous response’s meta to keep the same IP.

import scrapy

PROXY = "http://spx_user:spx_pass@198.51.100.23:8000"

class IpCheckSpider(scrapy.Spider):
    name = "ip_check"

    def start_requests(self):
        yield scrapy.Request("https://api.ipify.org?format=json", meta={"proxy": PROXY})

    def parse(self, response):
        self.logger.info("seen as %s", response.json()["ip"])

How to rotate IPs per request?

Create a downloader middleware to rotate proxies for all requests without modifying spider code. The middleware below fetches the proxy list via API at startup and retains existing proxies in meta to support session persistence and rotation.

Store API keys in settings.py or read from environment variables. Register middleware with a priority number lower than HttpProxyMiddleware(750) so it processes addresses before the default middleware.

import itertools
import requests

class RotatingProxyMiddleware:
    def __init__(self, key):
        text = requests.get(
            "https://socks5proxy.io/api/v1/proxies/download",
            params={"format": "user:pass@ip:port", "protocol": "http"},
            headers={"Authorization": "Bearer " + key},
            timeout=30,
        ).text
        self.pool = itertools.cycle(["http://" + l.strip() for l in text.splitlines() if l.strip()])

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.get("SPX_API_KEY"))

    def process_request(self, request, spider):
        if "proxy" not in request.meta:  # keep a proxy that was set on purpose (sticky sessions)
            request.meta["proxy"] = next(self.pool)

SPX_API_KEY = "YOUR_API_KEY"

DOWNLOADER_MIDDLEWARES = {
    "shop.middlewares.RotatingProxyMiddleware": 740,
}

CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 16
DOWNLOAD_DELAY = 0.25
AUTOTHROTTLE_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 522, 524, 408]
DOWNLOAD_TIMEOUT = 30

How to balance speed and retries?

CONCURRENT_REQUESTS_PER_DOMAIN sets simultaneous requests per domain. More proxy IPs let you increase this without overloading single IPs (e.g., 50 concurrent requests across 500 IPs = ~10 requests per IP).

AUTOTHROTTLE auto-adjusts request intervals. For frequent 429s, use RETRY_HTTP_CODES and clear meta['proxy'] to rotate IPs on retries.

Adding a pipeline step to verify that collected results are actual pages, not confirmation screens, improves data quality. Responses missing required elements can be retried with a different IP.

• Adjust concurrent requests per domain based on IP count
• AUTOTHROTTLE: Adjust intervals based on response speed
• Retry with different IP for 429 and 5xx errors
• Filter CAPTCHAs via response inspection

What changes when connecting via IP authorization?

Register your Scrapy server's IPv4 in the dashboard to use HTTP proxies like http://198.51.100.23:8000 without credentials. You can also download lists in ip:port format. If running spiders across multiple servers, register all server IPs (up to 10).

How to group session-dependent requests under one IP?

Cookie-based flows like logins or shopping carts require consistent IPs. Inject the proxy directly into the initial request's meta, then pass response.meta['proxy'] unchanged in callbacks to prevent rotation middleware from switching IPs. Combine with Scrapy's cookiejar meta key for session isolation.

Pair one proxy with one cookie set per session to prevent cross-session contamination during parallel operations.

import scrapy

PROXIES = [
    "http://spx_user:spx_pass@198.51.100.23:8000",
    "http://spx_user:spx_pass@198.51.100.24:8000",
]

class SessionSpider(scrapy.Spider):
    name = "sessions"

    def start_requests(self):
        for i, proxy in enumerate(PROXIES):
            yield scrapy.Request("https://example.com/", meta={"proxy": proxy, "cookiejar": i}, dont_filter=True)

    def parse(self, response):
        # follow-up request keeps the same proxy and the same cookie jar
        yield response.follow("/account", meta={"proxy": response.meta["proxy"], "cookiejar": response.meta["cookiejar"]}, callback=self.parse_account)

    def parse_account(self, response):
        self.logger.info("status %s via %s", response.status, response.meta["proxy"])

How to temporarily exclude failed proxies?

For 403/429 errors in middleware's process_response, pause the proxy for minutes and retry with another. Copy the request, clear meta['proxy'], enable dont_filter, and resend.

Managing a list of resting proxies and reuse times with a dictionary simplifies things. Avoiding immediate reuse of the same IP gives target sites time to lift restrictions.

How to save bandwidth in Scrapy?

Scrapy defaults to HTML-only responses, using far less bandwidth than browser tools. With HttpCompressionMiddleware enabled by default, shared plan monthly bandwidth handles extensive scraping.

Enable HTTPCACHE_ENABLED during development to avoid duplicate requests. In production, use dont_filter sparingly for essential repeats only.

Narrow link-following scope to avoid unnecessary subpages. Clear crawl boundaries save time, bandwidth, and reduce target site load.

How do I resolve common issues?

For 407 errors, verify the username/password in the address. Percent-encode special characters in passwords. TunnelError means the proxy rejected CONNECT - ensure you're using the HTTP port.

If all requests use the same IP, check if middleware order places HttpProxyMiddleware too late or if the spider directly sets meta proxy. Review settings.py and spider code.