Help with SimpleScraper source_url

App Development
1

Hey all,

I set up simple scraper to scrape this company snapshot site: https://safer.fmcsa.dot.gov/query.asp?searchtype=ANY&query_type=queryCarrierSnapshot&query_param=USDOT&query_string=(INSERT DOT HERE)

I would like to add the “query string” parameter based on my form input of the dot number and then the api call will get data from SimpleScraper.

For some reason, when I add this link as a “source_url” parameter to SimpleScraper, it runs the call and then returns me a result of the fmcsa site with just the first parameter “searchtype=Any” and it never goes through.

APIs are new to me so I’m still learning and I appreciate and insights you might have on this.

Does anyone has experience with other scrapers maybe? Am I appending too many parameters in my source url maybe?

Thank you so much for your help

2

I just did a quick search and are you referring to SimpleScraper that requires a Chrome Extension to run? I’m not familiar with scrapers, but I’m sure this shouldn’t be too hard if they have an API. If they don’t and it requires a chrome extension, that seems a bit more complex.

3

Moe,

So it’s a chrome extension just to set up the scraping elements and then it turns that into a API URL you can run to retrieve that data.

Screen Shot 2021-11-20 at 10.10.39 PM|690x111Screen Shot 2021-11-20 at 10.10.47 PM|614x344Screen Shot 2021-11-20 at 10.10.22 PM|690x499

The problem is when I send a “source_url” to simplescraper, it does not go through with all the parameters. This is more of a simplescraper issue maybe?

4

I’ll check it out and let you know soon.

5

@simplyvlad

Here’s how I (sort of) got mine to work.

  1. Create a new pipe and set the URL of the pipe to the URL from the SimpleScraper site:

https://simplescraper.io/api/meAdMWHDda0hd0O2nE7qf?apikey=YOUR_API_KEY&limit=100&run_now={run_now}&source_url={source_url}

(Make sure to update the API key and the section after the /api/ to your details.

  1. Next, create 2 new parameters, like so:
    image
    image1064×266 19 KB

You should now be abe to test the scraper by passing a unique URL in the source_url parameter.

  1. Last part is to create the Response to be used in the app. The Scraper returns an array, so we can only work with one value at a time in the pipe. You therefore need to structure it like so:

data.0.NAME <- The name is the property name you named it inside of simpleScraper.

image
image1495×185 18.4 KB

I couldn’t get it to work on my side since SimpeScraper always kept giving me selector errors or access denied errors.

If you share with me your app (moe at tadabase do io) I’d be happy to take a look. Or if you have an API you recommend that can work for this, I can also help further with a video or more on how to implement this pipe.

1 Like
6

@moe

Thank you for the help!

So I actually got it to work - all the API setup on Pipes I did correctly, but SimpleScraper needed the URL to be sent in as an Encoded URL. So I just sent an encoded url and everything works great now!

1 Like
7

Awesome! Happy to hear that!

1 Like