Welcome, Guest
News and Announcements

TOPIC: Comic Vine Scraper

Comic Vine Scraper 3 years 4 months ago #39698

  • forkicks
  • forkicks's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 869
  • Thank you received: 108
  • Karma: 37
cbanack wrote:
There are some advantages for them, though; each API key is tied to a specific Comic Vine account, which means they can track how much each user is using the API, and they can adjust your API limit based on details about your account. For example, if you're a regular contributor to Comic Vine, your API key might be given a higher limit...or if you pay them money, or join the Comic Vine "premium" subscription.

Or, going on an orwellian limb here, they can track which comics you scrape and target content (read ads) when you visit the sites in their network.

fK
The administrator has disabled public write access.

Comic Vine Scraper 3 years 4 months ago #39699

  • mcpierce
  • mcpierce's Avatar
  • Offline
  • Senior Boarder
  • Posts: 46
  • Thank you received: 2
  • Karma: 1
Well, only if they can correlate your API usage to your web browser usage. Something I would think difficult at best.
Check out The Comic Book Update podcast!
www.comicbookupdate.com/
The administrator has disabled public write access.

Comic Vine Scraper 3 years 4 months ago #39700

  • mcpierce
  • mcpierce's Avatar
  • Offline
  • Senior Boarder
  • Posts: 46
  • Thank you received: 2
  • Karma: 1
Well, only if they can correlate your API usage to your web browser usage. Something I would think difficult at best since, at least for me, I browse anonymously.
Check out The Comic Book Update podcast!
www.comicbookupdate.com/
The administrator has disabled public write access.

Comic Vine Scraper 3 years 4 months ago #39701

  • forkicks
  • forkicks's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 869
  • Thank you received: 108
  • Karma: 37
I was half-joking above, but you do need to register an account on the website to request the api key. It's quite easy to do that. Of course you can be more private or less, but the average joe will simply create an account, get an api key, and bring home a boat load of tracking cookies.

fK
The administrator has disabled public write access.

Comic Vine Scraper 3 years 3 months ago #39770

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1318
  • Thank you received: 503
  • Karma: 181
Hey guys, I've changed my mind about how to deal with Comic Vine's rate limits, so here's a new version of Comic Vine Scraper! ;)

This version has a new SCRAPE_DELAY option in the advanced settings. You can use it to make the scraper wait for a certain number of seconds between each comic that it scrapes, for example:

SCRAPE_DELAY=30

That'll make the scraper wait 30 seconds between each comic. If you assume that Comic Vine only lets you do 30 scrapes every 15 minutes (or 120 per hour) then a 30 second delay between each scrape may allow you to scrape continuously for as long as you need. You'll be able to scrape about 2500 comics every 24 hours at that rate.

Of course, this feature really slows scraping down a lot, and it also makes the Comic Vine Scraper's user interface seem very unresponsive, so you won't want to do this all the time. But maybe it can save you some tedium if you need to work through a large number of scrapes over a few days or weeks.


Also, before someone asks:

Sorry, no, there's no easy way for me to make Comic Vine Scraper just "wait for a while and then keep going" once you hit your Comic Vine rate limit. That would be a nicer solution than SCRAPE_DELAY, but it would also require a major overhaul to how Comic Vine Scraper's error handling code works, among other things. I don't have time to do that right now!

Plus, Comic Vine has specifically asked me NOT to write the scraper in such a way that it keeps trying and trying once it has encountered an API error.
Last Edit: 3 years 3 months ago by cbanack.
The administrator has disabled public write access.
The following user(s) said Thank You: forkicks, 600WPMPO, actioncomics, oraclexview, fieldhouse, gdison

Comic Vine Scraper 3 years 3 months ago #39771

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1318
  • Thank you received: 503
  • Karma: 181
Also, if you happen to be fluent in a language other than English, I could really use your help translating a few new phrases that I added to the latest versions of Comic Vine Scraper.

See here for details. Thanks!
The administrator has disabled public write access.
The following user(s) said Thank You: abellnz

Comic Vine Scraper 3 years 3 months ago #39787

  • fieldhouse
  • fieldhouse's Avatar
  • Offline
  • Expert Boarder
  • Posts: 88
  • Thank you received: 9
  • Karma: 1
you are my hero. I added a couple of time delays in the previous build which worked for the most part but I definitely didn't take the time to figure out all the code and parameterize it.
The administrator has disabled public write access.

Comic Vine Scraper 3 years 3 months ago #39789

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1318
  • Thank you received: 503
  • Karma: 181
Glad to hear it's working. Personally, I've found a scrape delay of 30 seconds seems to work well when scraping new comics, and 10 seconds works when "re-scraping".

I've got a better version of the scraper coming soon, where the user interface doesn't completely freeze up during the SCRAPE_DELAY and some of the other networking stuff is a bit more solid. I'm just waiting to see if anyone else gives me some more language translations so I can roll all of that stuff into a single update in the next day or so.

(I still need translations for French, Dutch (Belgian), and Polish!)
Last Edit: 3 years 3 months ago by cbanack.
The administrator has disabled public write access.

Comic Vine Scraper 3 years 3 months ago #39822

  • bodhikun
  • bodhikun's Avatar
  • Offline
  • Junior Boarder
  • Posts: 28
  • Karma: -1
Hi cbanak,

please read my comments on Belgian Dutch in that thread. ;)
The administrator has disabled public write access.

Comic Vine Scraper 3 years 3 months ago #39824

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1318
  • Thank you received: 503
  • Karma: 181
bodhikun wrote:
Hi cbanak,

please read my comments on Belgian Dutch in that thread. ;)

Thanks, I have, and I responded there.

All I need now is a French translation! I know I'm supposed to be able to do that myself because I'm Canadian, but I'm afraid my French is not good enough. :blush:
The administrator has disabled public write access.
Time to create page: 0.228 seconds

Who's Online

We have 261 guests and 7 members online