Welcome, Guest
Submit ComicRack bugs to get them squashed

TOPIC: Comic Vine Scraper

Comic Vine Scraper 3 years 5 months ago #39291

  • sykoone
  • sykoone's Avatar
  • Offline
  • Expert Boarder
  • Posts: 153
  • Thank you received: 16
  • Karma: 5
If the API key got banned again, it doesn't matter what version you're using. However, upgrading to 1.77 will allow you to easily add your own personal API key from comicvine. It's free to anyone with a membership, and you can simply add the key to the scraper by using the advanced options as cbanack described above.
The administrator has disabled public write access.

Comic Vine Scraper 3 years 5 months ago #39337

Hey, I'm a Comic Vine user. I've come to your site because your API scrapers have crashed Comic Vine many times. That's why your keys keep getting banned. Yesterday the site was offline for over an hour because of Comic Rack's users scrapers.

As you can imagine this is extremely frustrating. If you all could slow down a little bit it wouldn't put so much stress on our server and cause us to crash.

Thanks :)
The administrator has disabled public write access.

Comic Vine Scraper 3 years 5 months ago #39339

  • boshuda
  • boshuda's Avatar
  • Offline
  • Gold Boarder
  • Posts: 296
  • Thank you received: 65
  • Karma: 8
This appears to be the thread she came from: www.comicvine.com/forums/bug-reporting-2...and-more-1561706/#19. True or not, we've become their scapegoat for all site problems.

It seems to me it might be time to retire your API, cbanack, and require people to use their own API from now on. That way there would at least be traceability to one particular user and would instantly invalidate the older versions that could be part of the problem.

It might also be helpful to put in a sleep/delay at least when re-scraping. It seems the rescraping is the biggest problem as it goes faster than general scraping and could potentially involve tens of thousands of titles in one go.
The administrator has disabled public write access.

Comic Vine Scraper 3 years 5 months ago #39347

  • Shinrai
  • Shinrai's Avatar
  • Offline
  • Platinum Boarder
  • With great power comes great W/T.
  • Posts: 885
  • Thank you received: 81
  • Karma: 33
Well FWIW there is a site admin in that thread claiming it's overscraping, not just random assumptions from users. This is absolutely the first time I've heard of the scraper causing THIS KIND of problem, though.

I would love to know how a few thousand additional page requests in an hour can bring down a site that makes as much revenue as ComicVine, though...
Last Edit: 3 years 5 months ago by Shinrai.
The administrator has disabled public write access.

Comic Vine Scraper 3 years 5 months ago #39348

  • kenjio
  • kenjio's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 597
  • Thank you received: 127
  • Karma: 32
I also think it may be time to retire the "public" key.
I understand how some users may not want to register on Comic Vine just to use the scraper, but if the risk is that one or two users f*** it up for the others, then it has to be done.
ComicRack has already faced situations where the actions of a few users have given the program (and the community) a bad name, and I'd hate to see it happen on Comic Vine, too.
I'm baaaaaaaaaaaaaaack!!
The administrator has disabled public write access.

Comic Vine Scraper 3 years 5 months ago #39349

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
Hi guys,

Just thought I'd better jump in here and clear up a few things:

1) The "default" API key has not been banned. If our default API key was causing significant problems for ComicVine, it would be banned immediately. But I've used it a little bit every day this week without any problems, including a few minutes ago.

2) It does sound like there have been a couple of very heavy users who have had their personal API keys banned over the past few weeks.

3) For those saying I should force everyone to get their own API key: I can and I will do this if Comic Vine asks me to, but I doubt they will since it wouldn't really solve their problem:

Comic Vine Scraper users (as a whole) want to access the freely available Comic Vine data more often than Comic Vine can support.

Giving everyone their own API key won't change this. Sure, it would allow Comic Vine to ban the heaviest users, but there's nothing to stop those users from getting a new API key or switching back to the "default" key.

4) If you're worried that heavy users will get the default API key banned, just get your own API key and don't abuse it. Then you shouldn't have a problem, regardless of what happens to the default key.

5) Comic Vine is offering a free service; as with all free services, most people are just going to take as much as they want. Unfortunately, there's only so much that individual app developers like me can do to stop this from happening. On the other hand, Comic Vine has plenty of options to limit the amount of "free service" they hand out. In the end, I'm pretty sure that's how this problem will be solved. They'll just implement some kind of throttling on the API, or limit the number of accesses per day per person, or something like that.
Last Edit: 3 years 5 months ago by cbanack.
The administrator has disabled public write access.

Comic Vine Scraper 3 years 5 months ago #39350

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
boshuda wrote:
It might also be helpful to put in a sleep/delay at least when re-scraping. It seems the rescraping is the biggest problem as it goes faster than general scraping and could potentially involve tens of thousands of titles in one go.

Yes, this I might do.

I'm waiting to hear back from someone I've contacted on the Comic Vine team. It's possible that they already have their own throttling solution in place (or in the works)...but if they don't, I could add something in myself.
The administrator has disabled public write access.

Comic Vine Scraper 3 years 5 months ago #39361

  • boshuda
  • boshuda's Avatar
  • Offline
  • Gold Boarder
  • Posts: 296
  • Thank you received: 65
  • Karma: 8
"We're working on a new API, that will have rate/limit information available for users."

www.comicvine.com/forums/bug-reporting-2...1706/?page=1#22-mark
The administrator has disabled public write access.

Comic Vine Scraper 2 years 4 months ago #42611

  • cabridges
  • cabridges's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 8
  • Karma: 0
Probably a dumb question but...

Got a new computer, reinstalled Comic rack, added the scripts back in (latest versions), restarted. CVS won't work for me, and I suspect it's because it wants the API, but I don't see where to put it and I'm not prompted for it. It's been years since I set it up the first time and I don't remember what I did then. Where's the API number go?
The administrator has disabled public write access.

Comic Vine Scraper 2 years 4 months ago #42612

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
cabridges wrote:
Probably a dumb question but...

Got a new computer, reinstalled Comic rack, added the scripts back in (latest versions), restarted. CVS won't work for me, and I suspect it's because it wants the API, but I don't see where to put it and I'm not prompted for it. It's been years since I set it up the first time and I don't remember what I did then. Where's the API number go?

What version are you trying to install? Perhaps you're installing an old version by mistake.

Be sure you are using a recent version of Comic Rack, and the latest version of Comic Vine Scraper (1.0.89) which is available here: cvs.xanatos.ca/

The API key can be entered in the first (leftmost) tab of the scraper's settings dialog.
Last Edit: 2 years 4 months ago by cbanack.
The administrator has disabled public write access.
Time to create page: 0.225 seconds

Who's Online

We have 225 guests and 2 members online