Welcome, Guest
Python Scripts for ComicRack

TOPIC: ComicVine (Scraper) current limit

ComicVine (Scraper) current limit 1 year 11 months ago #44152

  • pweasel
  • pweasel's Avatar
  • Offline
  • Expert Boarder
  • Posts: 123
  • Thank you received: 18
  • Karma: 8
i set it to 20 and had no problems with the remaining 50 comics
CRW 0.9.178 x64 on Win10
CRA 1.80 on Nexus 10
The administrator has disabled public write access.

ComicVine (Scraper) current limit 1 year 11 months ago #44153

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1327
  • Thank you received: 508
  • Karma: 182
Hi guys,

I've had a couple patches submitted (via pull request on github) that should resolve this issue by adding a short second delay between every API request.

Since no one has created a working fork of the scraper yet, I will review these changes and build them into a new release when I have a few spare moments (probably on the weekend.)
The administrator has disabled public write access.
The following user(s) said Thank You: actioncomics, duckpuppy, pweasel, fieldhouse, jkthemac

ComicVine (Scraper) current limit 1 year 11 months ago #44154

  • duckpuppy
  • duckpuppy's Avatar
  • Offline
  • Junior Boarder
  • Posts: 39
  • Thank you received: 3
  • Karma: 1
I made a pull request to cbanack anyway, just to be complete. I'll try to get a new thread going with my plugin.

EDIT: I'll wait until I see if cbanack accepts my PR (or at least releases a working fix) before starting my own thread.
Last Edit: 1 year 11 months ago by duckpuppy.
The administrator has disabled public write access.

ComicVine (Scraper) current limit 1 year 11 months ago #44167

  • fieldhouse
  • fieldhouse's Avatar
  • Offline
  • Expert Boarder
  • Posts: 89
  • Thank you received: 10
  • Karma: 1
pweasel wrote:
i set it to 20 and had no problems with the remaining 50 comics
I had it set to 20 and was able to scrape 395 / 1088 before it barfed and told me I needed to wait.
I'm just happy that they didn't lower the ban hammer on my IP like last time. :|
The administrator has disabled public write access.

ComicVine (Scraper) current limit 1 year 11 months ago #44171

  • solidus0079
  • solidus0079's Avatar
  • Offline
  • Senior Boarder
  • Posts: 75
  • Thank you received: 3
  • Karma: 1
I wish they'd offer a darned paid option for full speed scraping, these limits just get lower and lower.
The administrator has disabled public write access.

ComicVine (Scraper) current limit 1 year 11 months ago #44173

  • jkthemac
  • jkthemac's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 766
  • Thank you received: 253
  • Karma: 55
I dont think it woud work. the database would be just as fragile and they would be forced to act more responsibly with the contents of the database. I would only be prepared to pay if I could trust the data.
The administrator has disabled public write access.

ComicVine (Scraper) current limit 1 year 11 months ago #44241

  • ArKay
  • ArKay's Avatar
  • Offline
  • Senior Boarder
  • Posts: 70
  • Thank you received: 2
  • Karma: 0
Ooook, doesn't look like this is going anywhere and I don't feel like using the plugin with a 1 minute delay between files. So which sources do we have to modify in order to add the 1s delay?
The administrator has disabled public write access.

ComicVine (Scraper) current limit 1 year 11 months ago #44242

  • Teryx
  • Teryx's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 5
  • Karma: 0
cbanack updated the plugin and posted about it in the thread about it in News and Announcements. There was a bug on ComicVine's side that was supposed to have been fixed today, not sure if that's the case, though.
The administrator has disabled public write access.

ComicVine (Scraper) current limit 1 year 11 months ago #44243

  • ArKay
  • ArKay's Avatar
  • Offline
  • Senior Boarder
  • Posts: 70
  • Thank you received: 2
  • Karma: 0
I see, had been watching this thread! :) Thanks for the heads up! Not fixed, 11 files were all I could do.
Last Edit: 1 year 11 months ago by ArKay.
The administrator has disabled public write access.

ComicVine (Scraper) current limit 1 year 11 months ago #44245

  • solidus0079
  • solidus0079's Avatar
  • Offline
  • Senior Boarder
  • Posts: 75
  • Thank you received: 3
  • Karma: 1
Update from Edgework the CV API guy:
"Uh, the fix didn't make it into today's release but it is for sure queued up for tomorrow. Please refrain from stalking me over this..."
www.comicvine.com/forums/api-developers-...#js-message-15878311
(Posting this in two places since the convo is split here)
The administrator has disabled public write access.
Time to create page: 0.217 seconds

Who's Online

We have 283 guests and 4 members online