sqlalchemy multiple db

Setting Up SqlAlchemy To Use Multiple Databases

In my latest project, PriceMind I wanted to make the database more scalable. I had one database used by Flask. Inside that one db I had the tables a user needs to reach. Also, I had the users table inside this db. But it was…

spiders quickly

How To Write Scrapy Spiders Quickly And Effectively

This is something new. I’ve just started out the ScrapingAuthority Youtube channel. On this channel you will find videos about web scraping, data processing, data mining, big data and some other stuff. Also, I’m gonna share my progress with PriceMind. As always I appreciate your…

developing pricemind

Building a Web Scraping Based SaaS Business, part 4

Data flow in PriceMind As I talked about it earlier, I’m developing a price intelligence platform for ecommerce companies. If you don’t know, this kind of stuff is heavily relied on data. The most important function I have to focus on is what actionable insights…

scrapy spider

How I Write Scrapy Spiders in Minutes

In the last post of my web scraping business blog post series I mentioned that I have a spider-creating system. This system makes me able to build scrapy spiders literally in minutes. With this system my only goal is to be able to produce new…

programming stack

Building a Web Scraping Based SaaS Business, part 3

Hey I’m back again with a new business documentation post. The last time we talked about how I validated my idea without starting to code. Now in this one, I’m gonna go (sort of) deep on the technical side. What programming languages I use for…

validate idea
web scraping business

Building a Web Scraping Based SaaS Business, part 1

This is not a tutorial on how to scrape the web. This is something new I’m trying out. I’ve decided to document the whole process of creating my new “business”.  And as you would think from the post’s title it is based on online gathered…

scrapy ajax

Crawling with Scrapy – AJAX Forms and Infinite Scrolling

AJAX stands for Asynchronous JavaScript And XML (nowadays JSON instead). With AJAX websites can send and receive data from the server in the background, without reloading the whole page. This technique became really popular because it makes it easier to load data from the server…

scrapy javascript

Crawling with Scrapy – Javascript Generated Content

It’s really hard to find a modern website which doesn’t use javascript technology. It just makes it easier to create dynamic and fancy websites. When you want to scrape javascript generated content from a website you will realize that Scrapy or other web scraping libraries…

Crawling with Scrapy – Download Images

One of the most useful features of Scrapy is that it can download and process images. For example in the ecommerce world, retail companies use web scraping technology to make use of online data of products. Scraping images is necessary in order to match competitors’…