Welcome, Guest
News and Announcements

TOPIC: Comic Vine Scraper

Comic Vine Scraper 8 months 5 days ago #47151

  • Martell86
  • Martell86's Avatar
  • Offline
  • Junior Boarder
  • Posts: 27
  • Karma: 0
cbanack wrote:
Thanks, that's very nice of you to say. :)

I'm always happy to see that so many people are still using the Scraper (the download numbers suggest that people are using it more than ever, actually). And for what it's worth, I haven't completely abandoned the project. I'm not building new features these days, but I am still maintaining it (fixing bugs, etc.) and I do keep an eye on this forum thread.

It's been an immeasurable source of help to me over the last few years, and your input/feedback on this forum has always been super helpful. I actually look forward to my weekly scraping with the CVine Scraper.
The administrator has disabled public write access.

Comic Vine Scraper 8 months 3 days ago #47162

  • whinkle
  • whinkle's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 18
  • Thank you received: 3
  • Karma: 1
I agree. I don't comment a lot here, but I am a heavy user of the CVS, and I pay back by being one of the most prolific contributors of wiki info at Comic Vine. The scraper is a wonderful contribution to the CVS community.

I would say to those who use it, especially if you are collecting older comics, pitch in to the wiki on Comic Vine. In particular, Archie comics are missing many months of publication.

The more we ComicRack users support the ComicVine wiki, the more CVS helps us.
The administrator has disabled public write access.

Comic Vine Scraper 7 months 3 weeks ago #47201

  • DWoodhouse
  • DWoodhouse's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 1
  • Karma: 0
never mind it's back up
Last Edit: 7 months 3 weeks ago by DWoodhouse.
The administrator has disabled public write access.

Comic Vine Scraper 7 months 2 weeks ago #47225

Is there any way for the automatic mode to try the next item in the list when the first one doesn't match?

Like 90% of the comics the automatic mode doesn't get detects the wrong series in the first field, (in manual mode) but mostly the second "guess" is right, if the automatic mode could just match every cover from the results, or just the first 5 series it finds, it would be much more effective. Would that be a hard feature to add? I know cbanack isn't working on new features any more though.
The administrator has disabled public write access.

Comic Vine Scraper 7 months 2 weeks ago #47234

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
This kind of thing is certainly possible, but I deliberately chose not to do it because downloading multiple images from Comic Vine's servers is too resource intensive, especially when everyone is doing it all at the same time on new comic book day.

Currently, the Scraper focuses on trying to get a match based on the comic's filename. It only ever downloads a single cover image once it's pretty sure that it has already found the right match. And that downloaded cover is only used to confirm that the match is correct.

But downloading multiple covers to use as part of the search can cause problems. It works fine if the next cover image is a match, but whenever the search fails to find any matches at all, we end up rapidly and automatically downloading a bunch of non-matching covers. That's not good behaviour, because sometimes there can be a lot of failed matches. The whole scraping process would be slowed down considerably, but more importantly this would not really be 'playing nice' with Comic Vine's servers. And believe me, they do notice stuff like that.
The administrator has disabled public write access.

Comic Vine Scraper 7 months 2 weeks ago #47238

  • boshuda
  • boshuda's Avatar
  • Offline
  • Gold Boarder
  • Posts: 296
  • Thank you received: 65
  • Karma: 8
cbanack wrote:
The whole scraping process would be slowed down considerably, but more importantly this would not really be 'playing nice' with Comic Vine's servers. And believe me, they do notice stuff like that.

What? No. Their API is super robust and able to deal with whatever we throw at it :lol:

For all the issues and differences of opinions on naming/numbering conventions, I dread the day they shut down their API. I still try to regularly update information on their site and greatly appreciate what's there.
Last Edit: 7 months 2 weeks ago by boshuda.
The administrator has disabled public write access.

Comic Vine Scraper 7 months 2 weeks ago #47240

  • boshuda
  • boshuda's Avatar
  • Offline
  • Gold Boarder
  • Posts: 296
  • Thank you received: 65
  • Karma: 8
.
Last Edit: 7 months 2 weeks ago by boshuda.
The administrator has disabled public write access.

Comic Vine Scraper 7 months 2 weeks ago #47241

  • Xelloss
  • Xelloss's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 455
  • Thank you received: 117
  • Karma: 24
What about this:

Everytime a manual search es done, the server image is loaded all the same... So, instead of showing it, compare the covers and in case the covers don't match let it be delayed till the end... in case they matched, do the scrap automatically... this way we load the same amount of covers and we can do first all the mateched covers and let the "complicated ones" to the end...

Another thing I think could be done, is instead of everytime a manual search is done, instead of loading the sometimes hundreds of matches... just load the first 5 ones... as what it takes a lot of time are comics such as Batman (when usually it will be the first o second comic in the list). In case it isn't in the list, a button "show more matched" to load all...

All the same since I have been using my Autocomplete values script, only about of 10% of my comics scraps are done manually every week, so no complain there XD
Last Edit: 7 months 2 weeks ago by Xelloss.
The administrator has disabled public write access.

Comic Vine Scraper 7 months 2 weeks ago #47244

  • jkthemac
  • jkthemac's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 766
  • Thank you received: 253
  • Karma: 55
Xelloss wrote:
All the same since I have been using my Autocomplete values script, only about of 10% of my comics scraps are done manually every week, so no complain there XD

I can second this. If you are using a weekly system to add to your database Xelloss's script performed before the scrape will mean hardly any books for which you previously have entries will ask for manual intervention, because it tells the scraper which volume to look for when the option 'use your previous choice' is ticked. It also has the benefit of reducing the load on Comicvine and speeding up your scrapes.
Last Edit: 7 months 2 weeks ago by jkthemac.
The administrator has disabled public write access.

Comic Vine Scraper 7 months 12 hours ago #47329

  • Drybonz
  • Drybonz's Avatar
  • Offline
  • Gold Boarder
  • Posts: 296
  • Thank you received: 1
  • Karma: 9
Hey guys... is there a fix for the comic cover not showing for the current comic you are scraping?
The administrator has disabled public write access.
Time to create page: 0.284 seconds

Who's Online

We have 159 guests and 2 members online