Welcome, Guest
News and Announcements

TOPIC: Comic Vine Scraper

Comic Vine Scraper 2 years 2 months ago #42689

  • sensei6375
  • sensei6375's Avatar
  • Offline
  • Senior Boarder
  • Posts: 46
  • Thank you received: 15
  • Karma: 4
They have said it is 400 requests per 15 minutes and 500 searches per 24 hours.
The administrator has disabled public write access.

Comic Vine Scraper 2 years 2 months ago #42690

  • sensei6375
  • sensei6375's Avatar
  • Offline
  • Senior Boarder
  • Posts: 46
  • Thank you received: 15
  • Karma: 4
If you are sure the volume is correct for the comics you are scraping then you can do this. You don't need the 4050 just after the -. Copy that number into the custom tab, on the line comicvine_volume. If doing a lot you will probably still need a scrape delay of 2-3 seconds, at least that amount used to work. With the new limits I'm not sure if 2-3 seconds won't still run into the error limit. Trial and error at this point.
The administrator has disabled public write access.

Comic Vine Scraper 2 years 2 months ago #42691

  • tglass1976
  • tglass1976's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 8
  • Karma: 0
sensei6375 wrote:
If you are sure the volume is correct for the comics you are scraping then you can do this. You don't need the 4050 just after the -. Copy that number into the custom tab, on the line comicvine_volume. If doing a lot you will probably still need a scrape delay of 2-3 seconds, at least that amount used to work. With the new limits I'm not sure if 2-3 seconds won't still run into the error limit. Trial and error at this point.
Didn't have the Custom tab enabled previously, thanks for bringing that to my attention.

I entered the volume number for a couple comics but when I scrape it still shows searching first. Is this expected behavior? I was hoping that by specifying the volume it would skip the search step and just update the data.
The administrator has disabled public write access.

Comic Vine Scraper 2 years 2 months ago #42692

  • tglass1976
  • tglass1976's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 8
  • Karma: 0
Ah, had to check the option to use previous value when re-scraping. Looks like that got it.
The administrator has disabled public write access.

Comic Vine Scraper 2 years 2 months ago #42693

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1318
  • Thank you received: 503
  • Karma: 181
hyperspacerebel wrote:
Hey, cbanack. In light of the stricter comicvine api limits, I'm wondering if you've ever given any thought to extending your plugin to work with local copies of GCD's database dump (www.comics.org/download/)? I was considering hacking something together myself, but my Python is getting pretty rusty :P

That's a good idea, and the data even looks like it would be a decent fit...but it's also a LOT of work, which I just don't have time for these days. I'm committed to keeping things running with Comic Vine, at least until they reduce the API limits below 1 comic per day. :lol: But that's all I can do right now.
The administrator has disabled public write access.

Comic Vine Scraper 2 years 2 months ago #42694

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1318
  • Thank you received: 503
  • Karma: 181
Deleted double post. :ohmy:
Last Edit: 2 years 2 months ago by cbanack.
The administrator has disabled public write access.

Comic Vine Scraper 2 years 2 months ago #42695

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1318
  • Thank you received: 503
  • Karma: 181
tglass1976 wrote:
Ah, had to check the option to use previous value when re-scraping. Looks like that got it.

You can also use CVINFO files if you are scraping a lot of comics from the same series. Bear in mind that eliminating your search requests does not eliminate all the requests associated with scraping a comic, though it will help significantly.
The administrator has disabled public write access.

Comic Vine Scraper 2 years 2 months ago #42697

  • krandor
  • krandor's Avatar
  • Offline
  • Gold Boarder
  • Posts: 190
  • Thank you received: 16
  • Karma: 2
sensei6375 wrote:
They have said it is 400 requests per 15 minutes and 500 searches per 24 hours.

I don't think that is accurate anymore. I used to be able to scrape about 100 books before I hit the 400/15 limit. Now I'm lucky to do 40-50. It can't be the 500/24h limit I'm hitting because if I wait a few minutes and try again it works.

I'm not sure what the real limits are, but I don't think they are the above anymore.
The administrator has disabled public write access.

Comic Vine Scraper 2 years 2 months ago #42698

  • Nomadtla
  • Nomadtla's Avatar
  • Offline
  • Junior Boarder
  • Posts: 22
  • Thank you received: 10
  • Karma: 5
This may seem silly or may be more work then it is worth but would it help to put a search counter into the CVS GUI somehow. I think if people could see how their habits added up to search numbers it may help them adjust terms or file names prior to scraping to put less workload on the scraper.
The administrator has disabled public write access.

Comic Vine Scraper 2 years 2 months ago #42699

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1318
  • Thank you received: 503
  • Karma: 181
Nomadtla wrote:
This may seem silly or may be more work then it is worth but would it help to put a search counter into the CVS GUI somehow. I think if people could see how their habits added up to search numbers it may help them adjust terms or file names prior to scraping to put less workload on the scraper.

That's an interesting idea. I've added it to the issue tracker so I don't forget about it the next time I'm working on the scraper.
The administrator has disabled public write access.
Time to create page: 0.492 seconds

Who's Online

We have 228 guests and 8 members online