{ "version": "https:\/\/jsonfeed.org\/version\/1", "title": "LEFT JOIN: blog on analytics, visualisation & data science, posts tagged: selenium", "home_page_url": "https:\/\/en.leftjoin.ru\/tags\/selenium\/", "feed_url": "https:\/\/en.leftjoin.ru\/tags\/selenium\/json\/", "icon": "https:\/\/en.leftjoin.ru\/user\/userpic@2x.jpg", "author": { "name": "Nikolay Valiotti", "url": "https:\/\/en.leftjoin.ru\/", "avatar": "https:\/\/en.leftjoin.ru\/user\/userpic@2x.jpg" }, "items": [ { "id": "36", "url": "https:\/\/en.leftjoin.ru\/all\/gathering-fresh-proxies-with-python-for-free\/", "title": "Gathering fresh proxies with Python for Free", "content_html": "

Sometimes, when we try to parse a website with Selenium our IP address might get blacklisted. That’s why it’s better to use a proxy. Today we are going to write a script that would scrape new proxies, do checking and, in case of success, pass them to Selenium<\/p>\n

Scraping fresh proxies<\/h2>\n

Let’s start by importing libraries, we will need modules for sending requests, scraping and storing data.<\/p>\n

import requests_html\r\nfrom bs4 import BeautifulSoup\r\nimport pickle\r\nimport requests<\/code><\/pre>
All proxies wiil be stored in the px_list<\/span> and written to  proxis.pickle<\/span>. Data will be loaded from this file if it’s not empty.<\/p>\n
px_list = set()\r\ntry:\r\n    with open('proxis.pickle', 'rb') as f:\r\n            px_list = pickle.load(f)\r\nexcept:\r\n    pass<\/code><\/pre>
The  scrap_proxy()<\/span> function will navigate to free-proxy-list.net and gather the latest 20 proxies, which are updated every minute on the site. Here’s the field we are interested in:<\/p>\n
\n\"\"\n<\/div>\n
We will need to extract an IP Address and Port. Let’s inspect elements on this page:<\/p>\n
\n\"\"\n<\/div>\n
The data we need to gather is represented as table cells. We will take the first 20 cells with a for loop, and request an IP-address and Port with xpath<\/span>.  Finally, the function will send fresh proxies to the pickle file and return them as a list.<\/p>\n
def scrap_proxy():  \r\n    global px_list\r\n    px_list = set()\r\n\r\n    session = requests_html.HTMLSession()\r\n    r = session.get('https:\/\/free-proxy-list.net\/')\r\n    r.html.render()\r\n    for i in range(1, 21):\r\n        add=r.html.xpath('\/html\/body\/section[1]\/div\/div[2]\/div\/div[2]\/div\/table\/tbody\/tr[{}]\/td[1]\/text()'.format(i))[0]\r\n        port=r.html.xpath('\/html\/body\/section[1]\/div\/div[2]\/div\/div[2]\/div\/table\/tbody\/tr[{}]\/td[2]\/text()'.format(i))[0]\r\n        px_list.add(':'.join([add, port]))\r\n\r\n    print("---New proxy scraped, left: " + str(len(px_list)))\r\n    with open('proxis.pickle', 'wb') as f:\r\n        pickle.dump(px_list, f)\r\n    return px_list<\/code><\/pre>
Checking new proxies<\/h2>\n
Oftentimes gathered proxies might be not  working, so we need to write a function that would check them by sending a GET request to Google and, if there is an error, it will return False<\/span>. In case the proxy is working, it will return  True<\/span>.<\/p>\n
def check_proxy(px):\r\n    try:\r\n        requests.get("https:\/\/www.google.com\/", proxies = {"https": "https:\/\/" + px}, timeout = 3)\r\n    except Exception as x:\r\n        print('--'+px + ' is dead: '+ x.__class__.__name__)\r\n        return False\r\n    return True<\/code><\/pre>
Main function<\/h2>\n
We will pass to our main function the scap parameter, which is False<\/span> by default. New proxies will be gathered if the following conditions are met: scrap == True<\/span>  or len(px_list)<6. Then we gather new proxies using a while loop<\/span> , take the last one to check, if  check_proxy<\/span> returns True<\/span> , other proxies will be sent to the pickle file and the function return the working IP address and Port.<\/p>\n
def get_proxy(scrap = False):\r\n    global px_list\r\n    if scrap or len(px_list) < 6:\r\n            px_list = scrap_proxy()\r\n    while True:\r\n        if len(px_list) < 6:\r\n            px_list = scrap_proxy()\r\n        px = px_list.pop()\r\n        if check_proxy(px):\r\n            break\r\n    print('-'+px+' is alive. ({} left)'.format(str(len(px_list))))\r\n    with open('proxis.pickle', 'wb') as f:\r\n            pickle.dump(px_list, f)\r\n    return px<\/code><\/pre>
Changing proxies in Selenium<\/h2>\n
Сheck out our previous articles on Selenium about handling website buttons<\/a> and scraping an online store catalog<\/a><\/p>\n
Import the get_proxy<\/span> function to configure proxies in Selenium and run a while loop. The PROXY<\/span>  variable will accept our freshly-grabbed proxy and be added to the browser options. Now we can create a new webdriver instance with updated options and let’s try to access the website, add some cookies, and if everything works fine the while loop will be break<\/span>. Otherwise, the function will run until there’s a working proxy found.<\/p>\n
from px_scrap import get_proxy\r\n\r\nwhile True:\r\n    PROXY = get_proxy(scrap=True)\r\n    options.add_argument('--proxy-server=%s' % PROXY)\r\n    driver = webdriver.Chrome(chrome_options=options, executable_path=os.path.abspath("chromedriver"))\r\n    try:\r\n        driver.get('https:\/\/google.com')\r\n        driver.add_cookie(cookies)\r\n    except:\r\n        print('Captcha!')<\/code><\/pre>",
            "date_published": "2020-07-10T15:11:47+03:00",
            "date_modified": "2020-08-07T14:32:31+03:00",
            "image": "https:\/\/en.leftjoin.ru\/pictures\/1-10.png",
            "_date_published_rfc2822": "Fri, 10 Jul 2020 15:11:47 +0300",
            "_rss_guid_is_permalink": "false",
            "_rss_guid": "36",
            "_e2_data": {
                "is_favourite": false,
                "links_required": [
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css"
                ],
                "og_images": [
                    "https:\/\/en.leftjoin.ru\/pictures\/1-10.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/2-10.png"
                ]
            }
        }
    ],
    "_e2_version": 3386,
    "_e2_ua_string": "E2 (v3386; Aegea)"
}