1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Scraping data from Magento without privileged access or trust

Discussion in 'Programming/Internet' started by danpalmer, Oct 8, 2018.

  1. danpalmer

    danpalmer Guest

    There appear to be a number of ways of scraping product data from a Magento site, but all seem to have their upsides and downsides.

    We deal with sites who have little to no technical resource, but who have given us permission to scrape their product catalog. There appear to be 3 different ways of doing this, none of which really work:

    • Manual web scraping - developer intensive, requires updating when the theme changes.
    • Magento Web API - requires setting up an API user, too technical for many users.
    • Magento Plugin - too technical for many users, exposes sensitive business data so many companies won't do this.

    Are we missing something? Is there a better alternative, or are there ways of changing any of the above 3 to be better for scraping?

    For example, is it possible to provide a link to a 'one-click-setup' like process for API access? Shopify do this in a nice way using OAuth and permission scopes, so we can give our partners a link that will give us read only access to just their product catalog, in a way that non-technical users can use.

    Login To add answer/comment
     

Share This Page