How to Use Proxies in Geotargeting Web Scraping With Python | by Lynn Kwong | May, 2022

Get correct knowledge in your software with proxies

Picture by kreatikar in Pixabay

A proxy is one thing sitting between you and the goal web site. While you scrape a web site with a proxy, it’s the IP of the proxy that’s used to scrape the goal web site, not yours.

In case your software depends closely on internet scraping, you’d wish to use proxies attributable to numerous causes. One in every of which is geotargeting, which implies we wish to scrape the goal web site from a selected area or nation, and never from our native IPs. That is particularly essential for on-line retailers which usually show completely different content material for various areas.

On this article, we’ll introduce alternative ways to make use of proxies for internet scraping in Python with the requests and Scrapy libraries. Each free proxies for testing and devoted ones for manufacturing can be launched.

Free proxies are good for testing, and will usually solely be used for testing. It is because free proxies are usually shared by many customers. Due to this fact, the efficiency is far decrease than that of devoted ones and you will get banned simply. In addition to, there may be safety points if a nasty proxy is used.

Then again, devoted proxies are all business and may be fairly pricey, relying on the plans. Nonetheless, they’re much safer and steady and will usually be used for manufacturing. Nonetheless, don’t fear, those launched on this publish all have a free trial plan and also you don’t should pay something for studying functions.

Firstly, we have to set up the packages for internet scraping. We’ll set up the requests and Scapy libraries for this text. It’s endorsed to create a virtual environment and set up the packages there so that they gained’t mess up system libraries.

For simplicity, we’ll use conda to create the digital setting. In addition to, we’ll set up iPython with the intention to run Python code interactively and extra conveniently.

Now we are able to begin to write some take a look at code in Python. For testing with the requests library, we are able to run the code straight iPython. Nonetheless, for the Scrapy internet scraping framework, it’s higher to write down the code in some IDE, reminiscent of VS Code.

Let’s first attempt to do primary scraping with out utilizing a proxy. We’ll scrape and get our IP. Run the next code in iPython or run it as a script straight:

To make use of the Scrapy framework for scraping, we have to create a scraping challenge and a spider first:

With the Scrapy framework, the code can also be fairly easy. Add the next code within the spider file proxy_tutorial/spiders/ Be aware that the start_urls needs to be up to date as specified to get the IP handle.

Now run the spider to get your IP handle:

Be aware that the -L possibility is used to mute the logging of the scraping course of. The identical result’s obtained as with the requests module. Now let’s use a proxy and see what magical issues will occur.

Shared/Free proxies

There are some shared proxy providers accessible on-line totally free. A really useful one is Webshare, which is a famend proxy supplier and thus secure to make use of. It has a free plan together with 10 proxies in several nations and is ideal for testing functions. In addition to, you possibly can improve your plan to paid personal ones or devoted ones in order for you higher proxy providers.

While you log in to Webshare, you will note your free proxy listed as follows:

Use shared proxies for the requests library

Now let’s use the shared proxies in internet scraping. Substitute the Username, Password, Proxy Tackle, and Port with your personal as proven within the image above, then you will note the IP of the proxy, relatively than yours as proven above:

Be aware that within the proxies, we have to have one for HTTP and HTTPS requests, respectively. Nonetheless, their values are the identical, and each begin with http. It’s the URL for the proxy with authentication data. This syntax is required by the requests library, however not the Scrapy framework, as we’ll see quickly.

You may examine the geolocation of the IP of your proxy at or different comparable IP providers accessible on-line.

Use shared proxies for the Scrapy framework

With the Scrapy framework, we are able to add the proxy in two methods. First, we are able to add it with the meta request parameter:

Alternatively, we are able to add the proxy in customized middleware. In our easy Scrapy challenge folder, there’s a file referred to as which has a variety of boilerplate code. Take away all of them and add the next code:

To really make the proxy specified within the middleware work, remark out or take away the start_requests operate added above. Then add the next code to For simplicity, you can too take away the boilerplate code in and simply maintain what’s proven right here:

Be aware that the orders of the proxy middlewares matter, the customized middleware needs to be put earlier than the built-in HttpProxyMiddleware, in any other case, it gained’t work correctly.

After we run the spider once more, we see it exhibits the IP of the proxy as effectively:

With the free or shared proxies as proven above, we’ll all the time scrape the web site from the identical IP of the proxy and due to this fact may simply get blocked. To resolve this drawback, you need to have a lot of proxies and replace/rotate them frequently.

That is very demanding and time-consuming work. It may be enjoyable for the primary time, however not enjoyable in any respect if you need to do it on daily basis. Due to this fact, for manufacturing utilization, we should always usually search a business answer with good assist so we don’t want to fret about proxy updates and rotations.

Now let’s examine some devoted proxy providers. There are fairly just a few proxy suppliers which give devoted proxies and likewise handle IP rotations routinely and neatly for you. To call just a few, there are Zyte, ScraperAPI, ProxyCrawl, and so forth. These three suppliers all have a free trial plan and you may mess around for a number of days for studying functions. For an exhaustive listing, you possibly can examine this article or different comparable ones on-line. The rankings of the proxy suppliers are fairly subjective, and you need to examine and evaluate fastidiously and select the one which most accurately fits your wants and likewise your funds.

We’ll introduce Zyte and ScraperAPI on this publish as a result of they’re comparatively reasonably priced and have superb documentation.

Zyte is developed by the identical group of builders of Scrapy, the favored scraping framework launched above. Due to this fact, the combination of Zyte and Scrapy is native and quite simple to do. In case you use Scrapy in your challenge, Zyte may be the primary proxy supplier to think about.

Really, Zyte is an unbiased proxy service and might work independently of Scrapy. Due to this fact, we are able to additionally use Zyte with the requests library. The utilization is similar to that of the shared proxy above:

To make use of the Zyte Sensible Proxy Supervisor, you should create an account at Zyte and add your bank card to get the 14 days trial. You gained’t be charged till the trial interval ends otherwise you overuse your quota. So keep watch over your utilization and keep in mind to cancel the subscription after you will have completed testing.

After you have completed registration, you’ll land on the “Getting began with Sensible Proxy Supervisor” web page, the place you possibly can set some “Configuration Settings”. To open this web page later, you possibly can click on the “API Entry” tab below the “Sensible Proxy Supervisor” on the left navigation bar.

For geotargeted scraping, you should create an account for every area. For demonstration right here, two accounts can be created, one for the US and the opposite for the UK. An API key can be generated routinely for every account. Now let’s use the API key within the requests and Scrapy framework, respectively.

Use Zyte proxy for the requests library

For the requests library, we have to obtain the certificates for scraping HTTPS webpages. Obtain the certificates named zyte-proxy-ca.crt and put it aside in your pc.

We’ll scrape once more to see the IP of the proxy:

It needs to be famous that we have to specify the confirm parameter after we make HTTPS requests with the requests library. The worth is the trail to the certificates file zyte-proxy-ca.crt you simply downloaded.

Use Zyte proxy for the Scrapy framework

Since each Scrapy and Zyte are developed by the identical firm, it is rather simple to make use of Zyte contained in the Scrapy framework. Nonetheless, first, you would wish to put in the Zyte Sensible Proxy Supervisor middleware in your digital setting:

$ pip set up scrapy-zyte-smartproxy

Then you should add the Zyte Sensible Proxy Supervisor middleware to the DOWNLOADER_MIDDLEWARES in your file:

Be aware that you should take away the customized proxy middleware added above. The order worth 610 is offered by the official document of Zyte.

After that, you possibly can both allow Zyte good proxy in globally or simply allow it for a selected spider.

Allow Zyte in

Allow Zyte for a selected spider:

Each methods would work, and you may select the one which fits your want. After we run the spider now, we are able to see the IP of the proxy as effectively.

Be aware that while you run the spider a number of instances, you’ll get a unique IP every time, which is rotated neatly by Zyte.


The second devoted proxy supplier I wish to introduce is ScraperAPI. It may be a possible various for Zyte as a result of it’s relatively cheap for interest utilization and has very good documentation. In addition to, it’s handy to scrape JavaScript webpages with a single argument. The drawback is that isn’t low cost to have the geotargeting function. Nonetheless, in case your scraping challenge doesn’t have a requirement for geotargeting, it may be a possible selection. And after you have discovered use free and devoted proxies as launched on this article, you will get began with any proxy suppliers very simply.

To check ScraperAPI, you possibly can click on “Free Trial” on the home page of ScraperAPI. There you will get began with 5,000 free API Credit. No bank card is required for the free trial, in distinction to Zyte.

After you will have registered and logged in, you’ll land on the ScraperAPI dashboard the place you possibly can see the API key and a few pattern code:

Use ScraperAPI proxy for the requests library

There are various ways to make use of the ScraperAPI, we’ll follow the proxy mode as a result of it’s generic and thus comparable for various proxy suppliers. The code for utilizing the requests library with ScraperAPI is similar to that with Zyte Sensible Proxy proven above, with some provider-specific settings:

Keynotes right here:

  • To allow the geotargeting function, we have to specify the country_code within the proxy. To examine different customized parameters, please examine the official documentation.
  • We should specify confirm=False for the requests library when ScraperAPI is used. This may occasionally or might not be a problem in your case, relying on what websites you’re scraping.

Use ScraperAPI proxy for the Scrapy framework

To make use of ScraperAPI with Scrapy, we can’t allow it in the identical approach as for Zyte Sensible Proxy, however have to do it in the identical approach as for the free proxy of Webshare. It is because Zyte is native for Scrapy, different proxy suppliers are all third-party for Scrapy.

We are able to add the proxies within the start_requests methodology, or by way of a customized proxy middleware as proven above. For simplicity, we’ll add the proxy to the start_requests methodology right here:

Be aware that, much like the Webshare free proxy, we solely want so as to add one HTTP proxy for Scrapy. After we run this spider now, we are able to get the IP of the proxy:

We’ve got specified the nation code to be France simply to ensure the proxy works as anticipated.

On this article, we’ve got launched alternative ways to make use of proxies in internet scraping. You should utilize free proxies for studying and testing functions. Nonetheless, when it goes to manufacturing utilization, you would wish to think about using a devoted one that may present good IP rotation and prevent a variety of time for upkeep.

There are a lot of proxy suppliers out there. The key variations between them are the geolocation protection, the scale of the IP pool, and velocity, which collectively decide the value. You would wish to examine and evaluate them fastidiously your self and select the one which most accurately fits your wants and funds.

In case you occur to make use of the Scrapy framework for internet scraping, the Zyte Sensible Proxy is usually a good selection as a result of it’s straightforward to make use of and has a comparatively financial plan. And by the best way, don’t neglect to cancel the subscription to Zyte Sensible Proxy should you simply wish to do some assessments, in any other case, you can be charged after the trial interval.

More Posts