
Description: Scrapy is an open-source web crawling framework used for scraping, parsing, and storing data from websites. It is written in Python and allows users to extract data quickly and efficiently, handling tasks like crawling, data extraction, and more automatically.
Type: Open Source Test Automation Framework
Founded: 2011
Primary Use: Mobile app testing automation
Supported Platforms: iOS, Android, Windows

Description: StormCrawler is an open source web crawler designed to crawl large websites efficiently by scaling horizontally through Apache Storm. It is fault-tolerant and allows integration with other Storm components like machine learning pipelines.
Type: Cloud-based Test Automation Platform
Founded: 2015
Primary Use: Web, mobile, and API testing
Supported Platforms: Web, iOS, Android, API