Tutorial-Scrape Load More Pages

Overview

This tutorial app is designed for list pages where content loads dynamically via a “Load More” button, which makes it difficult to collect all items at once. It solves this by repeatedly clicking the “Load More” button until all product items are visible, then loops through each item, enters its detail page, extracts the product subtitle, and returns to the list to continue the process.

It’s ideal for scraping product listings, articles, or any pageless content feed that requires user interaction to fully load. You can open the workflow in the Octoparse AI editor to see exactly how the automation is built and customize it for your own needs.

This demo uses the following sample page:https://www.crutchfield.com/S-VTBO1dSE69f/shopsearch/TV.html?&fa=1&pg=23#&nvpair=FFCategory|[rank260050]TVs

How to Use

1. Download the app from Octoparse AI app store
2. Double-click the app in your list.

   1. Variable Description

       To assign a value to a variable, double-click its name in the right sidebar and open the editing interface. For example:

1. Results: This data table stores the scraped data.

   2. Click "Run"

Notes

• This app was developed using the Octoparse AI Client 2.0 Beta. To run it successfully, please download and install the 2.0 beta version from our official site.
• Do not use the mouse or keyboard while the app is running to avoid interruptions.
• Octoparse AI is only compatible with Windows.
• How to Identify the Last Page

To determine whether you've reached the last page, open both the last page and the second-to-last page, then scroll to the location of the “Load More Products” button. Right-click and inspect the surrounding HTML elements on both pages.

You’ll notice that the “Load More Products” button exists on both the last and second-to-last pages, so it can't be used alone to detect the end. However, by continuing to compare the surrounding HTML, you may find a difference—an extra attribute that appears only on the last page (as shown in the screenshot). This unique feature can be used as a flag (e.g., flagLastPage element) to stop the automation from clicking the button.

Troubleshooting

Encounter obstacles but don't know how to resolve them when executing the app?

Please contact our support team at [email protected] to find the way out!

To help us better understand your issue, follow the steps below to export your running logs: