Welcome, Guest
News and Announcements

TOPIC: Comic Vine Scraper

Comic Vine Scraper 1 week 4 days ago #48523

  • krandor
  • krandor's Avatar
  • Offline
  • Gold Boarder
  • Posts: 204
  • Thank you received: 21
  • Karma: 4
Jothay wrote:
Also showing banned here. Do we think it's specific to ComicRack users that scrape?

Based on previous experiences with comicvine I bet they turned sonethjvgon and didn’t realize it would have this effect on api users.
The administrator has disabled public write access.

Comic Vine Scraper 1 week 4 days ago #48524

  • forkicks
  • forkicks's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 871
  • Thank you received: 109
  • Karma: 37
#MeToo

Worked wonders on wednesday and stopped yesterday. Strangely, after the first couple of times it failed through the scraper, i could still access the web site. It wasn't until much later i stopped being able to reach it too.
The administrator has disabled public write access.
The following user(s) said Thank You: frater

Comic Vine Scraper 1 week 4 days ago #48525

  • frater
  • frater's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 3
  • Karma: 0
Thanks a lot for this guys, I was pulling my hair out thinking it was just me - or my ISP - having issues. The idea that the site might be banning my IP range never even occurred to me, but now you've said it I see you're right.

It's definitely something they changed last night or the night before, and interestingly only happens when you use the scraper to search for an issue - the way you'd generally use it.

There is a tedious way around it (if you're not already banned) that I used to get around the cyborg problem (that was seperate to this particular issue, but I had that one as well). Basically if you copy the "notes" value out of one of your already scraped issues (looks like: Scraped metadata from ComicVine [CVDB636407].) and paste it into one of your unscraped one.

Then if you manually search for the issue through the website, you can find that number (the number after CVDB there) in the URL of the issue you're looking for. For instance:

comicvine.gamespot.com/3-floyds-alpha-king-5/4000-636407/

The number after the 4000- value there is the number to replace.

Once that's in there, if you have the scraper set up to "rescrape" automatically if it's already been scraped, then it will do just that - scrape the details of the issue without whatever is causing the ban.

Obviously it's ridiculously tedious to do this for more than one or two - doing it 200 times would do your head in - but it's interesting in that it points to an issue in the search API itself - or in some security system monitoring the search API, that isn't necessarily monitoring the "issue" api.

Hope this is just a mistake and not the start of a "ban comicrack users" plan.
The administrator has disabled public write access.

Comic Vine Scraper 1 week 4 days ago #48526

  • forkicks
  • forkicks's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 871
  • Thank you received: 109
  • Karma: 37
It's going to be an "accident", like the other times this happened. Someone placed a 10 instead of a 1 somwhere and it broke the auto ban setting.

Hopefuly they'll fix it. Has anyone opened a thread on their forum? It will get more visibility to the problem than here.
The administrator has disabled public write access.

Comic Vine Scraper 1 week 4 days ago #48527

  • frater
  • frater's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 3
  • Karma: 0
It looks like no-one has, i'm not 100% sure where I would. API developers forum? Bug forum?

I have sent an email through to their support email with what we've seen so hopefully that helps them. From going over it again, I think the issue likely lies within the cover API. When I hit scrape, it seems to return the list of possibles, but not the cover, so the first call to /search appears to work but call to /cover triggers the ban. That's just a guess though.
The administrator has disabled public write access.

Comic Vine Scraper 1 week 4 days ago #48528

  • Reason
  • Reason's Avatar
  • Offline
  • Junior Boarder
  • Posts: 32
  • Thank you received: 1
  • Karma: -1
as of 620am est I am now working again
The administrator has disabled public write access.

Comic Vine Scraper 1 week 4 days ago #48529

  • frater
  • frater's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 3
  • Karma: 0
Just tested and yes, it appears to be fixed now. Three cheers for comicvine devs :P
The administrator has disabled public write access.

Comic Vine Scraper 1 week 4 days ago #48531

frater wrote:
Just tested and yes, it appears to be fixed now. Three cheers for comicvine devs :P

Agreed! It's working here again too. Those guys must be working 24/7 to have gotten that sorted so quickly. Hooray for the ComicVine guys!
The administrator has disabled public write access.

Comic Vine Scraper 1 week 4 days ago #48533

  • FPMPlay
  • FPMPlay's Avatar
  • Offline
  • Senior Boarder
  • Posts: 58
  • Thank you received: 12
  • Karma: 3
I got 9 out of 44 scraped and then it errored out. Still not working for me. :(

I checked my usage and it was changing so it's definitely connecting:
Current API Usage

You have used 26 requests in the last hour for API Path '/search' (reset in 53 minutes)
You have used 4 requests in the last hour for API Path '/issues' (reset in 53 minutes)
Your request rate is fine

I'll try again later today.
Last Edit: 1 week 4 days ago by FPMPlay.
The administrator has disabled public write access.

Comic Vine Scraper 1 week 4 days ago #48534

  • boshuda
  • boshuda's Avatar
  • Offline
  • Gold Boarder
  • Posts: 296
  • Thank you received: 65
  • Karma: 8
I'm scraping okay, but it's slower than usual. Cover images, for example, are taking a tremendous amount of time. It's slowly cranking through a bunch of pre-volumed comics nicely. If you're not already use Xelloss auto-complete volume scripts. That helps a lot.
The administrator has disabled public write access.
Time to create page: 0.288 seconds

Who's Online

We have 252 guests and 3 members online