install selenium ide addon in chrome and excel

This poses the question: Is there an alternative way in which we could leverage something as useful as Selenium for web scraping, while at the same time overcome the aforementioned issues? However, this approach is not in accordance with the developer mindset as we want to automate tasks as much as possible.Īn alternative would be to set up Python on their machine, send the script and explain how to execute it. That means you’d have to set a respectable amount of time aside (especially if you have to do it multiple times), or even worse, your colleagues/friends might refuse to set up Python/learn what they have to do in order to get the data when they want. The other issue that comes to mind is sharing the data with colleagues/friends that are not developers.Īn obvious solution would be to send a new copy of the data every time you run the script. However, if we have to scrape multiple complex sites (pagination, AJAX, the data needs a lot of work to format it according to our liking) suddenly we lose a lot of time just by validating the results from the generated file(s). This isn’t an overwhelming issue if we have to scrape only one website with simple markup where there isn’t much room for mistake when writing the script. Every time we run our script we have to open the generated output file to see if it matches our demands. Validating the output of our Python script can prove to be a time-consuming task.

There are a couple of issues that come with the Python approach.

As a consequence we can accomplish our goal without writing a lot of code.Outputting the results – csv, pandas or xlsxwriter.Getting the data – Requests for static websites, Selenium with Python for dynamic websites (preferred).There are a lot of well-documented libraries (or even frameworks -> Scrapy) to help us out with the subtasks.

Outputting the results (most commonly to an Excel spreadsheet or CSV file)īased on the subtasks, the most obvious choice would be to use Python for web scraping because:.

So, how do we tackle web scraping when the time comes?įirst, let’s split the scraping task into three subtasks: Web scraping is something that will come up sooner or later in a developer’s career – be it for a business requirement or for personal use.

Install selenium ide addon in chrome and excel