Web Scraping With Selenium

I’ve used Beautiful Soup way, way back in the day. But now I can see how coupling it with a controllable browser is sort of … amazing via this article. But I found that instructions were missing a few things.

First of all, one thing I always get wrong is setting up the environment correctly. So be sure to:

python -m venv env
source env/bin/activate

Correction. I found that pipenv is much better (at least for me) and it seems to use virtualenv.

pipenv install
pipenv shell

Libraries

You want to have the right libraries installed so the errors don’t shock you:

source env/bin/activate (turn on the env in your directory) 
pip3 install beautifulsoup4 (or pipenv install )
pip3 install pandas
pip3 install tabulate
pip3 install lxml

Minor Edits

You’ll need to update the find_element call:

from selenium.webdriver.common.by import By

...

python_button = driver.find_element(By.ID, 'MainContent_uxLevel1_Agencies_uxAgencyBtn_33')

Related Posts