Building your own database from internet data goldmines
Data is gold to anyone doing business online and offline.
Google is a huge source of data having indexed 99% of an internet that gets bigger by the second.
For example, let's say you would like a spreadsheet with all webpages on Time Magazine's website. You can do a Google advanced search "site:time.com" and with a web scraper pull all information into your own database and mine it as you wish.
But what if you had a list of 10 or 100 or 1000, or 10,000 websites that you want in your database? With a little customization your web scraper can loop through that list of websites and pull every indexed webpage from all those sites and write it to your database.
Those are two examples of big data scraping that happens every day online. There's thousands of people that do nothing but scrape data, collecting various bits and pieces, assembling it into a massive unique and valuable databases, then reselling it.
I know because I ran a popular web scraper for internet marketers for 10 years. The crazy thing is I coded it while on vacation on a laptop as the same day the idea hit me.
Since then I have written countless little programs to pull information from hundreds of websites and data sources.
These programs are fast to develop, small and very focused applications that I've been building for 13 years.
It's a big data world and I've found that the types of databases that can be built are only limited by our imagination.
If you have an idea for something that would help you in your business or workplace, I'd love to hear about it. Email me at CoreyBornmann@gmail.com.
~ Corey Bornmann