Webscraping with selenium, click() line only works after x number of tries

Hi I have a script that scrapes a website based on filters, this was working fine up until the latest chrome update and chromedriver update and then one of the filters was failing to click.

I have a try and except loop that repeats for 10 tries, sometimes this is enough sometimes and other times I need > 15 clicks of the same line of code for it to eventually work.

Here is my code to initiate the webpage:

from selenium import webdriver

url='https://www.marketscreener.com/stock-exchange/calendar/finance/'
options=webdriver.ChromeOptions()
chrome_prefs = {}
options.experimental_options["prefs"]=chrome_prefs
chrome_prefs["profile.defaul_content_settings"] = {"popups":1}
options.add_argument('--disable-browser-side-navigation')
options.add_argument('--disable-infobars')
options.add_argument('--disable-extensions')
options.add_argument('--disable-popup-blocking')
options.add_argument('--disable-notifications')
driver = webdriver.Chrome(executable_path=r'foo.exe',options=options)

after I log in using my credentials I try this line of code:

driver.find_element_by_id("selCountry").click()

which opens up the country filter drop down menu. The majority of times it takes an arbitrary number >10 tries for it to eventually work.

I am using chrome version 87.0.4280.141 and its chromedriver with python 3.7.

Can anyone provide a better solution to my problem or maybe an explanation on why it suceedes after x number of tries?



Read more here: https://stackoverflow.com/questions/66456287/webscraping-with-selenium-click-line-only-works-after-x-number-of-tries

Content Attribution

This content was originally published by matthewr at Recent Questions - Stack Overflow, and is syndicated here via their RSS feed. You can read the original post over there.

%d bloggers like this: