Images are a critical part of the modern web, used for everything from product photos on e-commerce sites to visual content across social media. For many applications, being able to automatically download images from websites at scale can be extremely valuable. This is where image crawlers come in.
An image crawler (also known as an image scraper) is a bot that systematically browses webpages, finds images, and downloads them. Traditionally, building a crawler required coding skills. But today, there are powerful tools that allow you to create your own image crawlers without writing a single line of code.
In this guide, we‘ll walk through everything you need to know to build robust image crawlers using no-code tools. While we‘ll focus on Octoparse as our example platform, the same concepts and techniques can be applied using other web scraping tools and services.
Why Build an Image Crawler?
There are many reasons you might need to extract images from websites in bulk:
- Populating an image dataset for machine learning
- Monitoring a site‘s visual content over time
- Analyzing how products are visually presented online
- Archiving or backing up images from a site
- Enabling image search capabilities on a site or app
Manually downloading images simply doesn‘t scale when you need hundreds, thousands, or even millions of images. An automated crawler is essential for any kind of large-scale image extraction.
No-Code Web Scraping Tools
Traditionally, building a web crawler required programming skills in languages like Python, JavaScript, or PHP. Not only did you need to write the code to crawl pages and extract data, but you also had to handle challenges like rotating proxy IPs, avoiding blocking, and dealing with CAPTCHAs.
Fortunately, a new generation of no-code web scraping tools has emerged to make building crawlers much more accessible. With a visual interface, pre-built extraction templates, and managed infrastructure, these services drastically lower the technical bar to entry.
Some of the most popular no-code web scraping tools include:
- Octoparse
- ParseHub
- Mozenda
- Webhose.io
- Apify
For this guide, we‘ll use Octoparse as our example platform. Octoparse is a powerful yet user-friendly tool for building all kinds of web scrapers without coding, including image crawlers. Its point-and-click workflow builder, built-in data transformation functions, and cloud-based extraction make it an ideal choice for beginners and advanced users alike.
How to Build an Image Crawler with Octoparse
Let‘s walk through the steps to build an image crawler using Octoparse. We‘ll cover three common use cases:
- Fetching images directly from a webpage
- Scraping full-sized images instead of thumbnails
- Extracting full-size image URLs from thumbnail URLs
Example 1: Fetching Images from a Webpage
In this first example, we‘ll fetch all the images directly from a search results page. We‘ll use Pixabay, a popular source for free stock photos, as our example site.
Step 1: Create a new task
In the Octoparse dashboard, click "Advanced Mode" and then "+ Task" to create a new scraping job. Enter the URL for the webpage you want to crawl, e.g. https://pixabay.com/images/search/dogs/
Step 2: Select the images
On the webpage, click one of the image thumbnails to select it. In the Action Tips panel, you should see "Image selected, X similar images found". Click "Select all" and then "Extract image URL in the loop" to capture the URLs for all the found images.
Step 3: Paginate through results
If the results span multiple pages, we‘ll need to teach the crawler how to navigate through them. Scroll down to the bottom of the page and click the "Next page" button. In the Action Tips panel, select "Loop click the selected link". This will instruct the bot to keep clicking "Next" until it reaches the final page.
Step 4: Configure auto-scroll
Many webpages use lazy loading, where images are only loaded as the user scrolls down the page. To ensure our crawler captures all images, we need to enable auto-scrolling.
In the workflow, click the "Go to Web Page" step. On the panel on the right, find "Advanced options" and check the box for "Scroll down to the bottom of the page when finish loading". Configure the scroll times, speed and adjust as needed. Apply the same auto-scroll settings to the pagination step as well.
Step 5: Run the crawler
Click "Start Extraction" and select "Local Extraction" to run the crawler on your own machine. Octoparse will open the target webpage and start automatically scrolling, paginating, and extracting image URLs according to the workflow. You can view the extracted results in the "Data" panel.
Example 2: Scraping Full-Sized Images
Many sites display thumbnail images on search/browse pages to improve loading speed. In this example, we‘ll try to extract the full-resolution versions of those thumbnails.
The basic process is the same as Example 1, with one key difference. Instead of directly selecting the thumbnail, we‘ll first need to click into each image‘s details page to locate the full-sized image.
After initially selecting all the thumbnails, choose "Loop click each image" instead of extracting the URLs immediately. This will take us to the individual details pages for each image. From there, we can select the full-resolution image and extract its URL.
We‘ll also need to add a "Go back to the previous page" step to the workflow so the crawler returns to the main results page after grabbing each full-sized URL. Apply auto-scrolling and pagination as described in Example 1 to complete the crawler.
Example 3: Extracting Full-Size URLs via RegEx
Sometimes thumbnail and full-sized image URLs share a predictable pattern, usually only differing by a sizing parameter. In these cases, we can extract just the thumbnail URLs and use Octoparse‘s built-in tools to transform them into full-size URLs.
Step 1: Select thumbnails and extract URLs
Select the thumbnail images and choose "Extract the elements‘ outer HTML" instead of their URLs. The outer HTML will contain the thumbnail URLs within it.
Step 2: Define RegEx and test
In the workflow designer, click the "Customize" icon for the extraction step. Choose "Refine extracted data" and add a "Match with regular expression" step. Use the built-in RegEx tool to define the pattern for matching thumbnail URLs within the HTML. Test your expression with "Match" and "Match all" to ensure it captures the URLs correctly.
Step 3: Replace sizing parameter
Examine the thumbnail and full-size URLs to determine the sizing difference. Usually this will be a width/height parameter like "w=100" that needs to be changed to something like "w=1000".
Add a "Replace" step after the RegEx matching. Specify the thumbnail size to search for and the full-size to replace it with. Preview with "Evaluate" to make sure the URLs are transformed correctly.
Advanced Image Crawling Techniques
The examples above cover the basics of building an image crawler without code. However, there are a few other techniques worth knowing to build production-grade image crawlers:
File Naming
By default, Octoparse will download images with their original filenames from the site. To customize the naming, use the "Customize output filename" option in the extraction settings. You can specify a naming pattern using extracted data like the image title, date, dimensions, etc.
Deduplication
Many websites display the same image thumbnails on multiple pages, which can lead to duplicate downloads. To avoid this, add a "Remove duplicates" step to your image URL extraction workflow. You can deduplicate based on the URL or a hashed version of the image file.
Cloud Extraction
Running a crawler from your local machine is fine for small jobs, but large crawls can take a long time and put a lot of strain on your computer and network. Octoparse offers cloud-based extraction that allows you to offload the crawling to their scalable servers. This can dramatically speed up large image crawls.
API Access
If you‘re building an app or want to programmatically start image crawls, you can use Octoparse‘s API. The API allows you to manage crawlers, initiate and monitor crawls, and retrieve extracted data, all via HTTP requests. This enables powerful integrations and automations with other tools.
Limitations of No-Code Crawlers
As powerful as tools like Octoparse are, there are limitations to what you can achieve without custom coding. Some of the challenges that may require a programmatic approach include:
- Sites using advanced anti-bot techniques
- JavaScript-heavy pages that are hard to auto-scroll
- Complex page interactions not handled by the visual workflow designer
- Processing and analyzing images after downloading
For simpler sites and standard crawling patterns, no-code tools can get you pretty far. But the most advanced crawling and scraping tasks will likely still require some coding, whether it‘s customizing your no-code bot or building the crawler from scratch.
The Future of Image Crawling
As we move into 2023 and beyond, image crawling capabilities continue to evolve at a rapid pace. Some key areas of innovation include:
Computer Vision AI
The latest image crawlers are beginning to incorporate computer vision AI that can "see" and understand the contents of the images as they are extracted. This enables powerful capabilities like visual search, object detection, facial recognition, and more. Expect to see no-code tools begin to offer plug-and-play computer vision add-ons.
Rendering JavaScript Pages
An increasing number of sites rely heavily on JavaScript frameworks that make them challenging to crawl using standard HTML parsing. The next generation of web scraping tools are beginning to offer full JavaScript rendering, allowing them to crawl sites like single-page apps (SPAs) as easily as server-rendered pages.
Headless Browser Automation
For the most human-like crawling that can handle even the most bot-sensitive sites, some tools now offer full headless browser automation. This runs the crawler through an actual instance of Chrome or Firefox, perfectly replicating human behavior. While more resource-intensive, browser automation can often succeed where traditional crawling fails.
Learning More
Building an image crawler is just the beginning when it comes to the exciting field of web scraping and data extraction. To further your learning, check out some of these resources:
- Octoparse Tutorials: https://www.octoparse.com/tutorial
- ParseHub Documentation: https://www.parsehub.com/docs/
- Automate the Boring Stuff with Python (Web Scraping Chapter): https://automatetheboringstuff.com/2e/chapter12/
- Web Scraping Subreddit: https://www.reddit.com/r/webscraping/
Whether you‘re a marketer, data scientist, developer, or entrepreneur, learning to build your own web scrapers is an incredibly valuable skill. Visual, no-code tools like Octoparse make it possible to get started without a heavy coding background.
So choose a site, open up your scraping tool of choice, and start building your first image crawler today. You‘ll be amazed at how quickly you can start extracting value from the visual data across the web.