I’ve used Beautiful Soup way, way back in the day. But now I can see how coupling it with a controllable browser is sort of … amazing via this article. But I found that instructions were missing a few things.
First of all, one thing I always get wrong is setting up the environment correctly. So be sure to:
python -m venv env source env/bin/activate
Correction. I found that pipenv is much better (at least for me) and it seems to use virtualenv.
pipenv install pipenv shell
You want to have the right libraries installed so the errors don’t shock you:
source env/bin/activate (turn on the env in your directory) pip3 install beautifulsoup4 (or pipenv install ) pip3 install pandas pip3 install tabulate pip3 install lxml
You’ll need to update the find_element call:
from selenium.webdriver.common.by import By ... python_button = driver.find_element(By.ID, 'MainContent_uxLevel1_Agencies_uxAgencyBtn_33')