Welcome, Guest
News and Announcements

TOPIC: Comic Vine Scraper

Comic Vine Scraper 3 years 4 months ago #39453

  • Ulvirfaust
  • Ulvirfaust's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 8
  • Karma: -1
Just adding my voice to the chorus, it seems.. I'm having the same problems with the comic vine scraper - getting the "slow down cowboy..." message in the logfile... ugh... :(
The administrator has disabled public write access.

Comic Vine Scraper 3 years 4 months ago #39454

  • simian
  • simian's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 4
  • Karma: 0
simian wrote:
Just tried with the basic comic vine API Key and only got one book done before getting the error.

Added my own API Key which I hadn't used in a few months and got through 36 books before the error occurred.

Then I waited 10 minutes and it allowed me to do the next 24 issues I had (which was all I was needing to scan today)

So if your having troubles use your own API and try batches of less than 36 ten or so minutes apart
The administrator has disabled public write access.

Comic Vine Scraper 3 years 4 months ago #39455

  • fieldhouse
  • fieldhouse's Avatar
  • Offline
  • Expert Boarder
  • Posts: 88
  • Thank you received: 9
  • Karma: 1
looks like the current limit is set to 200 API calls per 15 minutes. That'll slow things down a little :D Sounds reasonable.
The administrator has disabled public write access.

Comic Vine Scraper 3 years 4 months ago #39456

My install was hosed and I have like 2k worth of books still to scrape.

How do I get my own API Key and how do I install it after I get it?


UPDATE:
*******************************************************
I totally found the part about the API Key and how to change it.

Thanks to cbanack and sorry if I had caused any problems with his API Key!

You are awesome brah!
Last Edit: 3 years 4 months ago by Xpistos Ekimogloy. Reason: Update
The administrator has disabled public write access.

Comic Vine Scraper 3 years 4 months ago #39459

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1318
  • Thank you received: 503
  • Karma: 181
Looks like for the time being, Comic Vine will restrict us to scraping about 30-40 comics per API key per 15 minutes. Those may not be the final numbers--I've been talking to them about that.

For now the best way to get things working is to grab your own API key (set it in the Advanced Settings dialog, as described here) and then try not to scrape more than about 30 comics every 15 minutes.

Since there are many people trying to use the default API key, I wouldn't expect that to be working very often over the next few days. That's why getting your own key is the best solution for now. Depending on how Comic Vine wants to do things, I may end up creating a new version of the scraper that requires you to provide your own API key.
Last Edit: 3 years 4 months ago by cbanack.
The administrator has disabled public write access.

Comic Vine Scraper 3 years 4 months ago #39460

  • tino55
  • tino55's Avatar
  • Offline
  • Senior Boarder
  • Posts: 43
  • Thank you received: 14
  • Karma: 7
cbanack wrote:
Looks like for the time being, Comic Vine will restrict us to scraping about 30-40 comics per API key per 15 minutes. Those may not be the final numbers--I've been talking to them about that.

this morning i have been able to scrape exactly 198 issues before the error and after a 5 minute break i can do another 198.
The administrator has disabled public write access.

Comic Vine Scraper 3 years 4 months ago #39462

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1318
  • Thank you received: 503
  • Karma: 181
@tino55: were you "rescraping" files that had previously been scraped, or were you scraping new files for the first time?

By my calculations, their current limit will allow 30-40 "new" scrapes every 15 minutes (per API key) and about 100 "rescrapes" every 15 minutes. Perhaps they've changed the limit somewhat...
The administrator has disabled public write access.
The following user(s) said Thank You: Xpistos Ekimogloy

Comic Vine Scraper 3 years 4 months ago #39463

  • Hagi
  • Hagi's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 5
  • Karma: 0
Interesting, I can't even do one without that error message appearing.

EDIT: After getting my own API key, that made things much easier. Thanks for the help guys
Last Edit: 3 years 4 months ago by Hagi.
The administrator has disabled public write access.

Comic Vine Scraper 3 years 4 months ago #39464

  • Gemris
  • Gemris's Avatar
  • Offline
  • Junior Boarder
  • Posts: 35
  • Thank you received: 12
  • Karma: 6
If you go to the forums in CV, it says right there that they are allowing 200 scrapes per API every fifteen minutes. In doing so, they are encouraging everyone to sign up for their own API.
The administrator has disabled public write access.

Comic Vine Scraper 3 years 4 months ago #39465

Under normal circumstances 200 every 15 minutes would be fine, but I don't think those numbers are right. I am trying to about 80 and I can't get through that full amount in one shot. But it is better after getting my own API Key than none ... so.
The administrator has disabled public write access.
Time to create page: 0.292 seconds

Who's Online

We have 180 guests and 6 members online