Welcome, Guest
News and Announcements

TOPIC: Comic Vine Scraper

Comic Vine Scraper 4 years 5 months ago #33534

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1318
  • Thank you received: 503
  • Karma: 181

Welcome to the Comic Vine Scraper thread!

The latest version of Comic Vine Scraper is always available on the Comic Vine Scraper website.

To report bugs in Comic Vine Scraper, please go here, or leave a comment in this thread.

To see the scraper's change history, or to download previous versions, go here.

Happy Scraping! :)
Last Edit: 2 years 4 months ago by cbanack.
The administrator has disabled public write access.
The following user(s) said Thank You: cYo, perezmu, forkicks, 600WPMPO, oraclexview, James Spaceman, alext41, T3KN0Gh057, wojosama, Couverdude and this user have 2 others thankyou

Re: Comic Vine Scraper 1.0.65 4 years 5 months ago #33536

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1318
  • Thank you received: 503
  • Karma: 181
Ok, so here's a bit more detail about 1.0.65 for those that are interested:

Scraping automatically: the scraper "guesses" a series for your comic, just like it always has. If you do not have the new "choose series automatically" feature turned on, then the scraper makes you confirm that its guess is correct, just like it always has. But if you turn on the "choose series automatically" feature, the scraper does something different. It will use a fancy image matching algorithm to compare the cover of the ComicVine entry with the cover of your comic book--if the two images are the same, it will accept its own guess automatically and move on to the next comic. And if the covers are different, it still falls back to making you choose the right series.

When I originally started working on this feature, some people expressed concern because sometimes trade paperback issues have the same cover art as the first issue of the comics that they contain. That means there is a possibility that the scraper might get confused and guess the wrong one, and then automatically accept it because its cover looks right. To put it simply, I think I've taken care of this problem. If you are curious about how, I described my solution in the google code issue for this feature.

What's up with Notes and Tags and CVDB?: basically, I've changed the scraper to use ComicRack's new Custom Book Fields (as I described earlier) to store the Comic Vine issue ID and volume (series) ID for each comic. The issue ID is actually pretty much the same thing as the CVDB value that many of you are used to seeing in the Notes and Tags fields of your comics. But now that this information is stored in the Custom Fields, it doesn't need to be stored in Notes and Tags as well (the scraper will find it no matter where it is.) However, the option to store CVDB values in Notes and Tags is still available, and the scraper will still use them as it always has.

In fact, it's probably a good idea to keep storing the CVDB value in your Notes field, since this is the only place you can put it where it will actually be written out to your comic book file (if you have that option turned on in ComicRack.)

Speaking of storing CVDB values in the Notes of your comics, you'll notice that I've changed the scraper so that it no longer includes the scrape date in the Notes. Instead of looking like this:
Scraped metadata from ComicVine [CVDBXXXXXXX] on 2010.10.0 at 09:34:22.
the notes field will look like this:
Scraped metadata from ComicVine [CVDBXXXXXXX].
The reason for this change is to eliminate spurious updates to your comic books. If you are allowing scraped data to be written into your comic files, then updating the scrape date in Notes will cause your comic files to be modified every single time they are scraped (even if nothing else changed!)

This can be frustratingly slow if your files are stored on a networked drive, and it can cause a LOT of unnecessary backup activity if you are using a backup program to backup your collection!
Last Edit: 4 years 5 months ago by cbanack.
The administrator has disabled public write access.

Re: Comic Vine Scraper 1.0.65 4 years 5 months ago #33537

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1318
  • Thank you received: 503
  • Karma: 181
Oh yeah, one other thing.

Once the bugs are fixed for this release, I am going on a vacation from coding for the rest of the summer. I'll continue to fix new bugs and offer support for Comic Vine Scraper in these forums, but I won't be working on any new features or changes to the scraper until September or October.

It might also take me a little bit longer than usual to answer forum questions, since I don't plan to be in front of my computer quite so often in the coming months. :)
Last Edit: 4 years 5 months ago by cbanack.
The administrator has disabled public write access.

Re: Comic Vine Scraper 1.0.65 4 years 5 months ago #33540

  • T3KN0Gh057
  • T3KN0Gh057's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 407
  • Thank you received: 114
  • Karma: 27
cbanack wrote:
If you choose to keep adding "CVDB" values to your comic book's Notes (you don't actually need to anymore)

That's a matter of opinion, in fact I'd rather the 2 new custom values be in notes as well (as this is a field that actually gets saved to the comic files, and any custom values are lost without comicrack.

In the instance the comicrack database fails or is lost, you have to manually scrape all over again if this information isn't read from the notes field. It's bad enough in my opinion that cYo deems all the info that one has to spend precious time collecting (Book Price, and all the other cataloguing info) not worth saving to the book files themselves, but the comicvine id is now the only maintainable reference to get back half the info one stands to lose if comicrack goes tits-up. So please tell me you still read this info from Notes, because otherwise i just lost my last reliable time-saver.

Edit: just read above, thank you for still allowing them to be read from notes. Have to admit that i panicked when reading the initial post. But I'm damn tired of my precious info getting lost whenever for some given reason (a power surge or ComicRack just dying, or a re-install) My database is lost. I like ComicRack, just not the ignorance of refusing to store valuable metadata information as part of the file.

It's easier to delete info you don't want than to find info you don't have.

Edit 2: (and a slightly self-promotional dialogue) I'd rather you have the option to save the two new id fields to Notes, however if not i will be using CR Data Manager to copy them to the Notes field, as soon as we get the new version out!
Last Edit: 4 years 5 months ago by T3KN0Gh057.
The administrator has disabled public write access.

Re: Comic Vine Scraper 1.0.65 4 years 5 months ago #33542

  • 600WPMPO
  • 600WPMPO's Avatar
  • Offline
  • Moderator
  • Posts: 3788
  • Thank you received: 557
  • Karma: 232
Many thanks for the new 'automatic' feature. Can't wait to test it..

And, as always, a +1 karma for your unwavering dedication to the scraper project. :-)
cbanack wrote:
Hi guys, a new version of Comic Vine Scraper is now available.

You see, forkicks and perezmu? I told you it wouldn't be much longer!
Hope your spoiled kids get satisfied with this. :laugh:
Now Playing: The ComicRack Manual (Online)

See my new comics & gadgets on: Tumblr!
The administrator has disabled public write access.

Re: Comic Vine Scraper 1.0.65 4 years 5 months ago #33543

  • T3KN0Gh057
  • T3KN0Gh057's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 407
  • Thank you received: 114
  • Karma: 27
600WPMPO wrote:
Many thanks for the new 'automatic' feature. Can't wait to test it..

And, as always, a +1 karma for your unwavering dedication to the scraper project. :-)
cbanack wrote:
Hi guys, a new version of Comic Vine Scraper is now available.

You see, forkicks and perezmu? I told you it wouldn't be much longer!
Hope your spoiled kids get satisfied with this. :laugh:

You know what I like about you 600? You're always Jimmy on the spot with the positivity. Me I tend to go straight to thinking "What can go wrong next..." And your avatar looks sooo happy!

Teach me the ways of the force
Last Edit: 4 years 5 months ago by T3KN0Gh057.
The administrator has disabled public write access.
The following user(s) said Thank You: 600WPMPO

Re: Comic Vine Scraper 1.0.65 4 years 5 months ago #33547

  • perezmu
  • perezmu's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1114
  • Thank you received: 64
  • Karma: 51
600WPMPO wrote:
cbanack wrote:
You see, forkicks and perezmu? I told you it wouldn't be much longer!
Hope your spoiled kids get satisfied with this. :laugh:

:-):-):-):-):-):-):-)
The administrator has disabled public write access.

Re: Comic Vine Scraper 1.0.65 4 years 5 months ago #33552

  • forkicks
  • forkicks's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 869
  • Thank you received: 108
  • Karma: 37
600WPMPO wrote:
Many thanks for the new 'automatic' feature. Can't wait to test it..

And, as always, a +1 karma for your unwavering dedication to the scraper project. :-)
cbanack wrote:
Hi guys, a new version of Comic Vine Scraper is now available.

You see, forkicks and perezmu? I told you it wouldn't be much longer!
Hope your spoiled kids get satisfied with this. :laugh:

Like you're not :)

fK
The administrator has disabled public write access.

Re: Comic Vine Scraper 1.0.65 4 years 5 months ago #33573

  • RevQuixo
  • RevQuixo's Avatar
  • Offline
  • Gold Boarder
  • Posts: 279
  • Thank you received: 25
  • Karma: 12
What?! Now i can walk away from the Scraper and interact with family?! You've overstepped your bounds Mr. Banack!

Incidentally everything worked pretty darn well so far..a few pesky marvel titles that fail because of Panini equivalents, titles with letters in their numbering "15AU" not being picked up, and of course spelling issues "2000AD" is never found until you rename, same with "Buffy Season 9".
The administrator has disabled public write access.

Re: Comic Vine Scraper 1.0.65 4 years 5 months ago #33574

  • 600WPMPO
  • 600WPMPO's Avatar
  • Offline
  • Moderator
  • Posts: 3788
  • Thank you received: 557
  • Karma: 232
RevQuixo wrote:
Incidentally everything worked pretty darn well so far..
Yes, exactly! I made a screencast for the non-believers! :cheer:

Now Playing: The ComicRack Manual (Online)

See my new comics & gadgets on: Tumblr!
The administrator has disabled public write access.
Time to create page: 0.308 seconds

Who's Online

We have 199 guests and 11 members online