I'm trying to scrape employement offers on this very webpage using selenium : .
But it returns an issue. However, when I try with another page (such as for example), it works fine and I can get the html normally.
So I don't understand why this particular webpage blocks the whole process.
Here's my code :
from bs4 import BeautifulSoup import requests import csv import pandas as pd from selenium import webdriver from selenium.webdriver.chrome.service import Service options = webdriver.ChromeOptions() options.add_argument('--headless') options.add_argument('--no-sandbox') options.add_argument('--disable-dev-shm-usage') site = ' wd = webdriver.Chrome('chromedriver', options=options) wd.get(site) And here is the error message :
WebDriverException Traceback (most recent call last) <ipython-input-45-d232051ff849> in <module>() 1 wd = webdriver.Chrome('chromedriver', options=options) ----> 2 wd.get(site) 2 frames /usr/local/lib/python3.7/dist-packages/selenium/webdriver/remote/webdriver.py in get(self, url) 428 Loads a web page in the current browser session. 429 """ --> 430 self.execute(Command.GET, {'url': url}) 431 432 @property /usr/local/lib/python3.7/dist-packages/selenium/webdriver/remote/webdriver.py in execute(self, driver_command, params) 416 response = self.command_executor.execute(driver_command, params) 417 if response: --> 418 self.error_handler.check_response(response) 419 response['value'] = self._unwrap_value( 420 response.get('value', None)) /usr/local/lib/python3.7/dist-packages/selenium/webdriver/remote/errorhandler.py in check_response(self, response) 241 alert_text = value['alert'].get('text') 242 raise exception_class(message, screen, stacktrace, alert_text) # type: ignore[call-arg] # mypy is not smart enough here --> 243 raise exception_class(message, screen, stacktrace) 244 245 def _value_or_default(self, obj: Mapping[_KT, _VT], key: _KT, default: _VT) -> _VT: WebDriverException: Message: unknown error: net::ERR_CONNECTION_RESET (Session info: headless chrome=94.0.4606.71) Stacktrace: #0 0x563e4590b6e3 <unknown> #1 0x563e4561c9f3 <unknown> #2 0x563e45619be6 <unknown> #3 0x563e4560c0a5 <unknown> #4 0x563e4560cfd5 <unknown> #5 0x563e4560c366 <unknown> #6 0x563e4560b945 <unknown> #7 0x563e4560a86c <unknown> #8 0x563e4560ab21 <unknown> #9 0x563e4561e50c <unknown> #10 0x563e456840c3 <unknown> #11 0x563e4566f212 <unknown> #12 0x563e456839bc <unknown> #13 0x563e4566f643 <unknown> #14 0x563e456468ec <unknown> #15 0x563e45647db5 <unknown> #16 0x563e4592fa14 <unknown> #17 0x563e4593f15d <unknown> #18 0x563e4593ee7b <unknown> #19 0x563e4593f7a2 <unknown> #20 0x563e4597852b <unknown> #21 0x563e4593fa01 <unknown> #22 0x563e459248f1 <unknown> #23 0x563e45948408 <unknown> #24 0x563e4594859a <unknown> #25 0x563e4596232f <unknown> #26 0x7f08e1bb86db start_thread Related questions 8 Python, error with web driver (Selenium) 1 WebDriverException in Selenium using python 3 1 WebDriverException thrown in Python for Selenium driver for Chrome Related questions 8 Python, error with web driver (Selenium) 1 WebDriverException in Selenium using python 3 1 WebDriverException thrown in Python for Selenium driver for Chrome 0 Python Selenium: Chrome 57 version - WebDriverException: Message: Unknown Error 1 Selenium NoSuchElement Exception with specific website 0 WebDriver not able to run on Python 3 Python - Selenium: selenium.common.exceptions.WebDriverException 0 ChromeDriver Selenium exception 0 Chrome WebDriverException using selenium 1 selenium.common.exceptions.WebDriverException: Message: in Python selenium 4 Load 7 more related questions Show fewer related questions
Reset to default
