Welcome, Guest
News and Announcements

TOPIC: Comic Vine Scraper 1.0.59-64

Comic Vine Scraper 1.0.59-64 4 years 8 months ago #31679

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182


Comic Vine Scraper 1.0.64 is now available for download from the Comic Vine Scraper website. It is identical to version 1.0.63, except that it also includes fixes for bugs 316 and 317.

This is a 'stable' release, which means it should be relatively bug free.


Comic Vine Scraper 1.0.63 is now available for download from the Comic Vine Scraper website. It is identical to version 1.0.61/62, except that it also includes a fix for bug 307.


Comic Vine Scraper 1.0.61 is now available for download from the Comic Vine Scraper website. It is identical to version 1.0.60, except that it also includes fixes for bugs 301, 302, 303, 304, and 306.


Comic Vine Scraper 1.0.60 is now available for download from the Comic Vine Scraper website. It is identical to version 1.0.59, except that it also includes a fix for issue 298.


Comic Vine Scraper 1.0.59 is now available for download from the Comic Vine Scraper website. This release includes a large number of "under-the-hood" changes, which means that it probably also includes a few bugs. A more stable (and mostly bug free) release will be available within the next week or two. Watch this forum thread or the project feeds for updates.

As many of you are aware, Comic Vine's public comic book database was recently overhauled. This caused a lot of trouble for Comic Vine Scraper, but the problems have all been fixed now. In particular, this latest release addresses issues 286, 289, 290, 291, and 297.

Also, Comic Vine Scraper 1.0.59 now scrapes the release date (i.e. the "In Stores" date) for your comic books. To see this new data, you'll need to turn off the ComicRack setting called "Show Catalog fields only for fileless Books".

As always, if you find any bugs in this release please report them in the issue tracker. You can log into it with your google/gmail account. Or if you prefer, you can also post a description of your problem (or any other feedback or questions) in this forum thread.

Enjoy! :)
Last Edit: 4 years 7 months ago by cbanack.
The topic has been locked.
The following user(s) said Thank You: Just Bill, docdoom, Mammut, Wolverutto

Re: Comic Vine Scraper 1.0.59 4 years 8 months ago #31680

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
A couple of notes on the new Comic Vine API and problems that it causes for the scraper:

1) Teams and Alternate Series (Crossovers) are NOT being scraped right now. The ComicVine API only returns blank values when the scraper tries to access those details, so there is currently no way to scrape them. I've reported this as a bug, however, and apparently it is going to be fixed by ComicVine in the next day or two. Once it is fixed, I expect that the current release of Comic Vine Scraper should start picking up those details again without any further complications.

2) Regarding the Published Date (that's the cover date that appears in the Year/Month/Day field of the details tab): right now, the scraper will only scrape the 'year' and 'month' of the published date, but never the 'day', even if the 'day' is available in ComicVine (which is fairly rare anyway.) It works like this because of a shortcoming in the ComicVine API, which I've described here if anyone is interested. If ComicVine adopts the API changes that I've suggested, I'll update the scraper so that it can scrape the published 'day', too.
Last Edit: 4 years 8 months ago by cbanack.
The topic has been locked.
The following user(s) said Thank You: sorrentmutie, T3KN0Gh057

Re: Comic Vine Scraper 1.0.59 4 years 8 months ago #31684

  • T3KN0Gh057
  • T3KN0Gh057's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 407
  • Thank you received: 114
  • Karma: 27
cbanack wrote:
A couple of notes on the new Comic Vine API and problems that it causes for the scraper:

1) Teams and Alternate Series (Crossovers) are NOT being scraped right now. The ComicVine API only returns blank values when the scraper tries to access those details, so there is currently no way to scrape them. I've reported this as a bug, however, and apparently it is going to be fixed by ComicVine in the next day or two. Once it is fixed, I expect that the current release of Comic Vine Scraper should start picking up those details again without any further complications.

2) Regarding the Published Date (that's the cover date that appears in the Year/Month/Day field of the details tab): right now, the scraper will only scrape the 'year' and 'month' of the published date, but never the 'day', even if the 'day' is available in ComicVine (which is fairly rare anyway.) It works like this because of a shortcoming in the ComicVine API, which I've described here if anyone is interested. If ComicVine adopts the API changes that I've suggested, I'll update the scraper so that it can scrape the published 'day', too.

You da man!

I'm making a list of Imprint / Rarent Publishers mappings that are missing from the scraper though, should i send those your way or work it into the script personally?
Last Edit: 4 years 8 months ago by T3KN0Gh057.
The topic has been locked.

Re: Comic Vine Scraper 1.0.59 4 years 8 months ago #31685

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
T3KN0Gh057 wrote:
I'm making a list of Imprint / Rarent Publishers mappings that are missing from the scraper though, should i send those your way or work it into the script personally?
Well, I use ComicVine to decide what is an imprint and what isn't.

So if an imprint is listed on ComicVine (go to the publisher page and see if it says something like "xxx is an imprint of yyy") then I will gladly add it to the scraper. Otherwise, you should just add it to your own Advanced Settings.

Of course, you are free to correct the publisher page on ComicVine if they're missing imprint information for a publisher...as long as your correction is accepted by ComicVine, I will use it. (Just don't forget to tell me about it; I don't have any automatic way to detect newly added imprints.)

In fact, I'm pretty sure this is exactly why ComicVine lets us use their data for free--they are hoping some of the people who use the data will want to correct errors or add new information. :)
Last Edit: 4 years 8 months ago by cbanack.
The topic has been locked.

Re: Comic Vine Scraper 1.0.59 4 years 8 months ago #31686

  • T3KN0Gh057
  • T3KN0Gh057's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 407
  • Thank you received: 114
  • Karma: 27
you know i haven't the slightest idea how to use advanced settings to do anything let alone map imprints.



But I'd like to use that until i finish compiling my list (some obvious ones are missing for instance Top Cow Being an imprint of Image... I'm aware there was a moment when they weren't but they are again sporting the Image "i" on their covers
Last Edit: 4 years 8 months ago by T3KN0Gh057.
The topic has been locked.

Re: Comic Vine Scraper 1.0.59 4 years 8 months ago #31687

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
Advanced Settings aren't too hard, here's the documentation, just look for "IMPRINTS". There's a couple of examples there.

EDIT: a couple users around here already have some pretty extensive lists of additional imprint mappings that maybe you could use as a starting point.
Last Edit: 4 years 8 months ago by cbanack.
The topic has been locked.

Re: Comic Vine Scraper 1.0.59 4 years 8 months ago #31688

  • cYo
  • cYo's Avatar
  • Offline
  • Moderator
  • Posts: 3476
  • Thank you received: 676
  • Karma: 182
Great work as usual :)

But one question: As there's quite a support for community ratings in ComicRack (all platforms), shouldn't the option to turn it on be in the main UI?
The topic has been locked.

Re: Comic Vine Scraper 1.0.59 4 years 8 months ago #31689

  • T3KN0Gh057
  • T3KN0Gh057's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 407
  • Thank you received: 114
  • Karma: 27
cbanack wrote:
Advanced Settings aren't too hard, here's the documentation, just look for "IMPRINTS". There's a couple of examples there.

EDIT: a couple users around here already have some pretty extensive lists of additional imprint mappings that maybe you could use as a starting point.

Can almost guarantee 600 is one of them!
Last Edit: 4 years 8 months ago by T3KN0Gh057.
The topic has been locked.

Re: Comic Vine Scraper 1.0.59 4 years 8 months ago #31690

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
cYo wrote:
Great work as usual :)

But one question: As there's quite a support for community ratings in ComicRack (all platforms), shouldn't the option to turn it on be in the main UI?
Unfortunately, "Community Rating" is not part of the ComicVine web API; it requires me to actually "scrape" the data off the live webpage for each comic. I don't like doing this because a) it's very fragile, b) it makes scraping slow, and most importantly, c) ComicVine asked me not to--it messes up their page visit statistics.

But I know some people love Community Ratings...at least if it is "hidden" in the Advanced Settings, most casual users won't find it. :P

Of course, I've asked ComicVine to add Rating into the regular web API. If they do that, I will definitely put it into the main UI, turned on by default.
Last Edit: 4 years 8 months ago by cbanack.
The topic has been locked.

Re: Comic Vine Scraper 1.0.59 4 years 8 months ago #31691

  • Madmatx
  • Madmatx's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 457
  • Thank you received: 63
  • Karma: 19
Something I never noticed before is they have the user ratings (which is what gets scraped) and "Comicvine rating" which is the rating from a staff reviewer... I think.

Is there a way to scrape that as well and maybe choose to have it put in the "My Rating" field?
The topic has been locked.
Time to create page: 0.238 seconds

Who's Online

We have 248 guests and 2 members online