Welcome, Guest
Python Scripts for ComicRack

TOPIC: ComicVine (Scraper) current limit

ComicVine (Scraper) current limit 2 years 9 months ago #41496

  • romsnesrom
  • romsnesrom's Avatar
  • Offline
  • Expert Boarder
  • Posts: 120
  • Thank you received: 12
  • Karma: 3
Does anybody know what are the current limits?
The administrator has disabled public write access.

ComicVine (Scraper) current limit 2 years 9 months ago #41501

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1318
  • Thank you received: 503
  • Karma: 181
I haven't heard anything new about limits changing for a long time, so I am also curious to know what Comic Vine is allowing people to get away with these days...
The administrator has disabled public write access.
The following user(s) said Thank You: romsnesrom

ComicVine (Scraper) current limit 2 years 9 months ago #41507

  • DoctorBlasphemy
  • DoctorBlasphemy's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 2
  • Thank you received: 1
  • Karma: 0
A few months ago I tagged the bulk of my collection and I could tag ~140-160 issues in a row before reaching "database limits". If I immediately started again I'd get another ~40 issues before getting cut off once more.
Last Edit: 2 years 9 months ago by DoctorBlasphemy. Reason: Words
The administrator has disabled public write access.
The following user(s) said Thank You: romsnesrom

ComicVine (Scraper) current limit 2 years 9 months ago #41512

  • burnshroom
  • burnshroom's Avatar
  • Offline
  • Junior Boarder
  • Posts: 25
  • Thank you received: 2
  • Karma: 2
150 last time I tested it without a delay. A couple of weeks ago. Full scrape.

With the 30 second delay, I would say limitless.
The administrator has disabled public write access.
The following user(s) said Thank You: romsnesrom

ComicVine (Scraper) current limit 2 years 9 months ago #41526

  • romsnesrom
  • romsnesrom's Avatar
  • Offline
  • Expert Boarder
  • Posts: 120
  • Thank you received: 12
  • Karma: 3
I asked because I am not getting a consistent behavior AFAICS. Sometimes I get a lot of scrapes and then increments of 200, and sometimes I start with 200 scrapes and get smaller increments (I assume 50 with the search overhead).
The administrator has disabled public write access.

ComicVine (Scraper) current limit 2 years 9 months ago #41547

  • Corwin
  • Corwin's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 13
  • Thank you received: 1
  • Karma: 0
I'm pretty certain it has to do with the amount of data you're transferring. So for example The New 52 Future's End never comes right up. The scraper has to scan and bring up a long list of possible titles and then once I pick the right title it brings up the issue numbers and covers. This uses up more data.
The administrator has disabled public write access.
The following user(s) said Thank You: romsnesrom

ComicVine (Scraper) current limit 2 years 8 months ago #41548

  • romsnesrom
  • romsnesrom's Avatar
  • Offline
  • Expert Boarder
  • Posts: 120
  • Thank you received: 12
  • Karma: 3
That would be the case if I was getting the same amount or less scrapes than everybody else. The thing is I am getting sometimes way more, and sometimes what burnshroom and DoctorBlasphemy reported and I could not form a pattern (day of week, time, etc).
Last Edit: 2 years 8 months ago by romsnesrom.
The administrator has disabled public write access.

ComicVine (Scraper) current limit 2 years 8 months ago #41595

  • boshuda
  • boshuda's Avatar
  • Online
  • Gold Boarder
  • Posts: 285
  • Thank you received: 63
  • Karma: 7
I also can't discern any pattern. Sometimes it blocks me pretty quickly, others it chugs along without interruption. It makes me think they have some sort of internal logic dictating an individual's access based on any number of factors that can change at pretty much any time.
The administrator has disabled public write access.
The following user(s) said Thank You: romsnesrom

ComicVine (Scraper) current limit 2 years 7 months ago #41808

  • Drybonz
  • Drybonz's Avatar
  • Offline
  • Gold Boarder
  • Posts: 296
  • Thank you received: 1
  • Karma: 9
I'm a bit out of the loop because I haven't had time to tag any comics lately... it used to be you could tag as many as you needed to without interruptions. I tried today and it is stopping after about 150 comics.

I'm wondering how long I need to wait to start tagging again. I have a run of Amazing Spiderman that I'm doing right now and it's going to take a while at this rate.

Thanks for any info.
The administrator has disabled public write access.

ComicVine (Scraper) current limit 2 years 7 months ago #41809

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1318
  • Thank you received: 503
  • Karma: 181
Drybonz wrote:
I'm a bit out of the loop because I haven't had time to tag any comics lately... it used to be you could tag as many as you needed to without interruptions. I tried today and it is stopping after about 150 comics.

I'm wondering how long I need to wait to start tagging again. I have a run of Amazing Spiderman that I'm doing right now and it's going to take a while at this rate.

Thanks for any info.

I don't usually run up against the scrape limit, so I don't remember for sure but I think it was an hour or two. It might also be that if you hit the limit too many times in a day, you might get banned for longer--really, this is an attempt by ComicVine to stop people from abusing their service, so they are in control of the limits and they can change them whenever they want.

One thing that seems to work for people who have a lot of comics to scrape is to use the SCRAPE_DELAY feature in the scraper.

I described it here:

comicrack.cyolito.com/forum/32-news-and-...aper?start=470#39770
Last Edit: 2 years 7 months ago by cbanack.
The administrator has disabled public write access.
Time to create page: 0.428 seconds

Who's Online

We have 176 guests and 6 members online