Welcome, Guest
News and Announcements

TOPIC: Comic Vine Scraper

Comic Vine Scraper 3 years 5 months ago #39302

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1318
  • Thank you received: 503
  • Karma: 181
As far as I know, the default Comic Vine Scraper API key was only banned for a couple hours over the weekend, then it started working again. As of right now, it is currently working (I just tested it.)

If you're having problems, the first thing you should do is make sure you've updated to the latest version of the scraper. Pretty much all of the older versions no longer work properly, due to changes that Comic Vine made to their servers.

And yes, if you're concerned that the default API key might get banned again, you can easily use your own API key instead.
The administrator has disabled public write access.

Comic Vine Scraper 3 years 4 months ago #39422

  • actioncomics
  • actioncomics's Avatar
  • Offline
  • Senior Boarder
  • Posts: 43
  • Thank you received: 6
  • Karma: 2
Getting the following error message after 30-40 books scrapped.


Caught _ComicVineError: code 107: "Rate limit exceeded. Slow down cowboy.

this is using my own API. anyone know what is going on?
The administrator has disabled public write access.

Comic Vine Scraper 3 years 4 months ago #39424

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1318
  • Thank you received: 503
  • Karma: 181
Sounds like Comic Vine may be introducing some kind of rate limit for how many comics you can scrape in one session, or how rapidly you can access their database, or something like that. The limitation may not be based on your API key.

If you start scraping again right after getting this error, does it work for another 30 or 40 comics or do you have to wait a while before it lets you keep scraping? How long?

What happens if you switch to the default API key?

Is anyone else having this problem?

I haven't heard anything from Comic Vine lately, so any details you guys can give me will help me figure out if I need to make any changes to the scraper so it can handle this situation better.
Last Edit: 3 years 4 months ago by cbanack.
The administrator has disabled public write access.

Comic Vine Scraper 3 years 4 months ago #39428

  • actioncomics
  • actioncomics's Avatar
  • Offline
  • Senior Boarder
  • Posts: 43
  • Thank you received: 6
  • Karma: 2
Switched back to the default api and didn't even get one issue scrapped. restarting the scrap (same api) didn't help any it just stopped with the first issue. I did think it was my api at first so i created a new account just to test the api theory, after 30-40 books it stopped again on the new api.

Sorry didn't get an exact count of the issues, that is roughly where it seems to stop.
The administrator has disabled public write access.

Comic Vine Scraper 3 years 4 months ago #39429

  • shaba21
  • shaba21's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 4
  • Thank you received: 1
  • Karma: 0
I also encountered the same block and error code using the default api. Hadn't scraped anything today. 30 issues between Wed-Fri.
The administrator has disabled public write access.

Comic Vine Scraper 3 years 4 months ago #39430

  • Nomadtla
  • Nomadtla's Avatar
  • Offline
  • Junior Boarder
  • Posts: 22
  • Thank you received: 10
  • Karma: 5
Just scraped about 87 issues on my own API and got no block or warning. It was mostly from 2-3 series so they scraped pretty easily after it located the series. This was after about 42 issues scraped Wed and a few yesterday with my normal 0-day stuff. Just my input if it helps.
The administrator has disabled public write access.

Comic Vine Scraper 3 years 4 months ago #39431

  • Schlodz
  • Schlodz's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 3
  • Karma: 0
Got the same Problem. Looks like they introduced this this night, because yesterday i could scrape >200 comics in one session without a problem and today it won't work for a single one.
The administrator has disabled public write access.

Comic Vine Scraper 3 years 4 months ago #39432

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1318
  • Thank you received: 503
  • Karma: 181
Yeah, I just tried (re)scraping some comics and also got the same error code after a few dozen.

I couldn't find anything in the Comic Vine forum about what was going on, so I posted my own question:

www.comicvine.com/forums/api-developers-...down-cowboy-1571089/

We'll see what they say. Maybe I'll just have to slow the scraper down, so we don't all keep exceeding our rate limits.

Has anyone "recovered" from this problem? As in, you got the error, stopped scraping, and then a while later you could scrape again?
The administrator has disabled public write access.

Comic Vine Scraper 3 years 4 months ago #39433

  • Schlodz
  • Schlodz's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 3
  • Karma: 0
It hasn't recovered yet for me, but it's been "only" an hour so far.
The administrator has disabled public write access.

Comic Vine Scraper 3 years 4 months ago #39434

  • Gemris
  • Gemris's Avatar
  • Offline
  • Junior Boarder
  • Posts: 35
  • Thank you received: 12
  • Karma: 6
The error code I've been getting is completely different. Is says:

"An unexpected error occurred. Would you like to save a log file with more details about this problem?"

Occasionally it will let me scrape 3 or 4 books, but then this error pops up. I've noticed it since last night.
The administrator has disabled public write access.
Time to create page: 0.215 seconds

Who's Online

We have 181 guests and 4 members online