In my latest project, PriceMind I wanted to make the database more scalable. I had one database used by Flask. Inside that one db I had the tables a user needs to reach. Also, I had the users table inside this db. But it was…
Author: Attila Toth
How To Write Scrapy Spiders Quickly And Effectively
This is something new. I’ve just started out the ScrapingAuthority Youtube channel. On this channel you will find videos about web scraping, data processing, data mining, big data and some other stuff. Also, I’m gonna share my progress with PriceMind. As always I appreciate your…
Building a Web Scraping Based SaaS Business, part 4
Data flow in PriceMind As I talked about it earlier, I’m developing a price intelligence platform for ecommerce companies. If you don’t know, this kind of stuff is heavily relied on data. The most important function I have to focus on is what actionable insights…
How I Write Scrapy Spiders in Minutes
In the last post of my web scraping business blog post series I mentioned that I have a spider-creating system. This system makes me able to build scrapy spiders literally in minutes. With this system my only goal is to be able to produce new…
Building a Web Scraping Based SaaS Business, part 3
Hey I’m back again with a new business documentation post. The last time we talked about how I validated my idea without starting to code. Now in this one, I’m gonna go (sort of) deep on the technical side. What programming languages I use for…
Building a Web Scraping Based SaaS Business, part 2
In the opening post of this series about building my business I gave you a quick overall view what this blog post series will be about. Now in this one I want to be as specific and detailed as possible how I validated the idea…
Building a Web Scraping Based SaaS Business, part 1
This is not a tutorial on how to scrape the web. This is something new I’m trying out. I’ve decided to document the whole process of creating my new “business”. And as you would think from the post’s title it is based on online gathered…
Crawling with Scrapy – AJAX Forms and Infinite Scrolling
AJAX stands for Asynchronous JavaScript And XML (nowadays JSON instead). With AJAX websites can send and receive data from the server in the background, without reloading the whole page. This technique became really popular because it makes it easier to load data from the server…
Crawling with Scrapy – Javascript Generated Content
It’s really hard to find a modern website which doesn’t use javascript technology. It just makes it easier to create dynamic and fancy websites. When you want to scrape javascript generated content from a website you will realize that Scrapy or other web scraping libraries…
Crawling with Scrapy – Download Images
One of the most useful features of Scrapy is that it can download and process images. For example in the ecommerce world, retail companies use web scraping technology to make use of online data of products. Scraping images is necessary in order to match competitors’…