I’ve used Beautiful Soup way, way back in the day. But now I can see how coupling it with a controllable browser is sort of … amazing via this article. But I found that instructions were missing a few things.
First of all, one thing I always get wrong is setting up the environment correctly. So be sure to:
python -m venv env
source env/bin/activate
Correction. I found that pipenv is much better (at least for me) and it seems to use virtualenv.
pipenv install
pipenv shell
Libraries
You want to have the right libraries installed so the errors don’t shock you:
source env/bin/activate (turn on the env in your directory)
pip3 install beautifulsoup4 (or pipenv install )
pip3 install pandas
pip3 install tabulate
pip3 install lxml
Minor Edits
You’ll need to update the find_element call:
from selenium.webdriver.common.by import By
...
python_button = driver.find_element(By.ID, 'MainContent_uxLevel1_Agencies_uxAgencyBtn_33')