Welcome, Guest
News and Announcements

TOPIC: Comic Vine Scraper

Comic Vine Scraper 3 years 8 months ago #38765

  • mrpibb
  • mrpibb's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 3
  • Thank you received: 3
  • Karma: 0
Hey dudes
This is mrpibb from comicvine. Just a quick heads up that the main CV scraper key has been blocked again (we saw extreme usage over 2-3 hours almost as if people were trying to rebuild their collections). Those who generated their own keys will be ok for now. I'm working w/ cbanack to see what we can do to mitigate this, but I can't have the site go down from 'rogue' scraping, especially during winter soldier weekend.
Best
Last Edit: 3 years 8 months ago by mrpibb.
The administrator has disabled public write access.
The following user(s) said Thank You: 600WPMPO, kenjio, KnobblySavage

Comic Vine Scraper 3 years 8 months ago #38767

  • BobaCox
  • BobaCox's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 6
  • Thank you received: 2
  • Karma: 1
I'm probably also at fault. I've scraped proably 10,000 comics in the last month and a half. I just changed to my own api. Sorry.

Boba Cox
The administrator has disabled public write access.

Comic Vine Scraper 3 years 8 months ago #38768

  • Thales
  • Thales's Avatar
  • Offline
  • Junior Boarder
  • Posts: 38
  • Thank you received: 1
  • Karma: 0
Real quick Thank You to perezmu and cbanack and anyone/everyone else for being on top of things as usual.

Been away from CR some time (real life and all that). Fired up and couldn't scrape.

Updated and followed the instructions for obtaining API and I'm working again.

Now if there was just a way to magically catch up for oh let's say 6+ months or more....

NO, I will not be doing all that at once just a bit at a time (when I have the time).

You folks dedication to this program is always amazing.
The administrator has disabled public write access.

Comic Vine Scraper 3 years 8 months ago #38769

  • pweasel
  • pweasel's Avatar
  • Offline
  • Expert Boarder
  • Posts: 124
  • Thank you received: 18
  • Karma: 8
so... what do we civilians do?
CRW 0.9.178 x64 on Win10
CRA 1.80 on Nexus 10
The administrator has disabled public write access.

Comic Vine Scraper 3 years 8 months ago #38772

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
pweasel wrote:
so... what do we civilians do?
Just give me and mrpibb a day or two to figure out the best way to move forward. For now, if you're desperate to scrape comics right away, you can do the trick that perezmu described where you manually change the API key in your copy of the scraper's cvconnection.py file.
The administrator has disabled public write access.
The following user(s) said Thank You: 600WPMPO, pweasel

Comic Vine Scraper 3 years 8 months ago #38773

  • pweasel
  • pweasel's Avatar
  • Offline
  • Expert Boarder
  • Posts: 124
  • Thank you received: 18
  • Karma: 8
done, thanks
CRW 0.9.178 x64 on Win10
CRA 1.80 on Nexus 10
The administrator has disabled public write access.

Comic Vine Scraper 3 years 8 months ago #38774

  • perezmu
  • perezmu's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1114
  • Thank you received: 64
  • Karma: 51
cbanack wrote:
pweasel wrote:
so... what do we civilians do?
Just give me and mrpibb a day or two to figure out the best way to move forward. For now, if you're desperate to scrape comics right away, you can do the trick that perezmu described where you manually change the API key in your copy of the scraper's cvconnection.py file.

This last 'incident' makes me thing there might be something fishy here... Don't sweat it Cory, we have a solution for the time being, take your time! I PM'd you, BTW.

I've been trying for eons to figure out how to simplify all this, and it all comes to the same point: if packs out there were pre-scrapped, but so far no luck!...
The administrator has disabled public write access.

Comic Vine Scraper 3 years 8 months ago #38775

  • SiPfan
  • SiPfan's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 14
  • Thank you received: 2
  • Karma: 1
This has been driving me nuts.

I am NOT a heavy user, but I do scrape a handful per day most days. It's what I do when I'm watching Cops or Walking Dead. Louis C.K. right now. Thanks Comedy Central uncensored! I go off on tangents.

A lot.

I feel like I got punished for other people's bandwidth issues, and for the last hour while I tried to understand what in YHWH's name an API is (I still have zero idea) and how to input my own API (Thanks Mr. Perez.) and just staring at the screen more often than not going "WHAT THE #@$%, OVER!?" I finally found Perez's delightfully specific instructions. (Posted HERE, for the easily frustrated... like me. )

Actually when I opened the .py file and found it crammed with comments explaining what was going on in the file, I was rather impressed with the specificity of what was being done. It became a simple matter of pasting my billion letter string over Cory's and restarting CR. Ta-Da! Also noticed scraping has sped up significantly for me, particularly over what it had been the last few days.

So, apart from the very existence of CR, a tremendous program; the work of Srs. Banack and Perez was well-communicated enough to allow me to get back on my horse.

Let's all have our own API, huh? Once I got what I needed to do, the fix was just a few moments. Normally, I'm a Collectivist/Socialist, but if there's abuse, I'd prefer not to have my usage destroyed by others. Also, it seems like ComicVine might prefer adding legitimate users who could bring in additional revenue...

Anyway, well done on many people's parts here, ya buncha shameless nerds. Hope to see you at Comic-Con.
The martyrs and madmen I learned of in school will remember my name.

-Kevin Gilbert
1966-1996
The administrator has disabled public write access.
The following user(s) said Thank You: Couverdude

Comic Vine Scraper 3 years 8 months ago #38776

  • RevQuixo
  • RevQuixo's Avatar
  • Offline
  • Gold Boarder
  • Posts: 280
  • Thank you received: 26
  • Karma: 12
I'm not sure it is a solution, but the thought I had was to have "Scraper Packs" available on say a weekly basis with the most up-to date version of CV data. Not sure how large an aggregate pull of the accumulated database would look like though. We could then store it locally (and grab it via torrent or some such).

This would work even better if CV had the ability to provide only deltas since the last pack was generated..that way the entire DB doesn't need to be hit when it is updated.
The administrator has disabled public write access.

Comic Vine Scraper 3 years 8 months ago #38777

  • Couverdude
  • Couverdude's Avatar
  • Offline
  • Junior Boarder
  • Posts: 22
  • Karma: 0
perezmu wrote:
HOW TO GET YOUR OWN API TO WORK WITH CVS:

Thanks. I was already a CV member and I didn't know this was even an option. I switched.

600WPMPO wrote:
Only doubt: If, someday, ComicVine bans an individual's key, how would he scrape? :unsure:

He would just need to re-register with a new email address and get a new key.
Last Edit: 3 years 8 months ago by Couverdude.
The administrator has disabled public write access.
Time to create page: 0.235 seconds

Who's Online

We have 232 guests and one member online