Welcome, Guest
Python Scripts for ComicRack

TOPIC: Proposed Changes to Comic Vine Scraper Settings

Proposed Changes to Comic Vine Scraper Settings 4 years 7 months ago #32505

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
Hi guys,

I'm working on the summer release of Comic Vine Scraper, and I've come up with a couple of ideas to change the scraper's settings. I don't think these changes are going to cause any big problems for anyone, but I thought I'd give you guys a chance to protest/talk me out of it, in case there's something I'm not thinking of.

So take a look at this. It's the way the scraper settings' "Behaviour" tab currently looks:



Notice those two options that I have circled in red? I'm planning on REMOVING both of them, and REPLACING them with the following two options:

1) "Choose each comic's series automatically, when possible"
2) "Confirm each issue choice before scraping it"

So what does this mean? Setting #1 will enable a new "automatic scraping" feature that I've been working on--it uses an image-matching algorithm to compare the cover for each comic with the cover of the scraper's "best guess" on Comic Vine, and if the two are the same, it just assigns that issue to your comic book automatically, without making you confirm it. If this feature is NOT turned on, you will be required to confirm the series for each comic book, just like before. And of course, you'll still have to manually deal with any comics that the automatic scraping feature can't figure out...

Setting #2 will force the scraper to show you which issue it is about to scrape just before it actually scrapes that issue. This is for people who prefer to manual confirm every issue as they go. This feature still works even if you let the scraper choose the series automatically (i.e. the previous setting); you'll still have to confirm the right issue number for each book.

The only functionality that will be lost here is the ability to "Confirm each series name before searching for it." But I don't think people really use this setting anyway. And even without it, you can still click "Search Again" if you don't like the scraper's automatic search.

I'm still a few weeks away from making these changes, but I'd like to hear if anyone has any objections or concerns with these changes. :)
The administrator has disabled public write access.

Re: Proposed Changes to Comic Vine Scraper Settings 4 years 7 months ago #32507

  • Brennok
  • Brennok's Avatar
  • Offline
  • Senior Boarder
  • Posts: 60
  • Thank you received: 2
  • Karma: 0
How will the new option 1 work with multiple covers? Also what about when issue 1 and the TPB have nearly identical covers?
The administrator has disabled public write access.

Re: Proposed Changes to Comic Vine Scraper Settings 4 years 7 months ago #32508

  • jkthemac
  • jkthemac's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 768
  • Thank you received: 253
  • Karma: 55
Having been using the scraper solidly for the last three weeks I have been wishing for an automatic image matching version every day, so I am totally in agreement.
The administrator has disabled public write access.

Re: Proposed Changes to Comic Vine Scraper Settings 4 years 7 months ago #32512

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
Brennok wrote:
How will the new option 1 work with multiple covers?
I'll make it compare the first page of your comic to all of the covers for the ComicVine issue.
Also what about when issue 1 and the TPB have nearly identical covers?
The scraper will take its "best guess" (i.e. the series that appears at the top of the list in the Series Window) and then see if that cover matches your comic book. If it does not, you will still have to manually choose the right series.

So I suppose if the scraper's "best guess" is wrong (e.g. it chooses a TPB instead of a regular issue) AND the covers are the same, then it could automatically assign the wrong issue. If this starts happening much, I guess I'll have to try to improve the scraper's series guessing algorithm.

I'm planning to make the autoscrape feature behave like a little helper that compares your cover image with ComicVine and clicks OK for you if they match. It is NOT going to be an image search on the ComicVine database--it's just a reasonably reliable way to automatically confirm the choices that the scraper is already suggesting.
Last Edit: 4 years 7 months ago by cbanack.
The administrator has disabled public write access.

Re: Proposed Changes to Comic Vine Scraper Settings 4 years 7 months ago #32513

  • docdoom
  • docdoom's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 320
  • Thank you received: 89
  • Karma: 31
Brennok wrote:
Also what about when issue 1 and the TPB have nearly identical covers?

Hopefully the image-mapping will handle that better than I did manually. It helped a lot that over the last time (don't know if I have to thank the Scraper or the CV API for that) that TPBs were mostly listed as second choice (and not top of the list) in the series selector dialog of the Scraper.

Nevertheless, I look forward to the new image-mapping algorithm. If it will be working fine (and I'm pretty sure that Cory will do a great job!) it will save a lot of time and be much more fool-proof.
Author of the CR Data Manager. Download and manual at google code - please post feature requests and bugs here
Last Edit: 4 years 7 months ago by docdoom.
The administrator has disabled public write access.

Re: Proposed Changes to Comic Vine Scraper Settings 4 years 7 months ago #32515

  • Shinrai
  • Shinrai's Avatar
  • Offline
  • Platinum Boarder
  • With great power comes great W/T.
  • Posts: 885
  • Thank you received: 81
  • Karma: 33
cbanack wrote:
Also what about when issue 1 and the TPB have nearly identical covers?
The scraper will take its "best guess" (i.e. the series that appears at the top of the list in the Series Window) and then see if that cover matches your comic book. If it does not, you will still have to manually choose the right series.

So I suppose if the scraper's "best guess" is wrong (e.g. it chooses a TPB instead of a regular issue) AND the covers are the same, then it could automatically assign the wrong issue. If this starts happening much, I guess I'll have to try to improve the scraper's series guessing algorithm.

Depending on how exacting the image matching it's doing is, I could certainly see it happening, so I think it's worth testing extensively. (Also possibly issues with homage-type covers, which can sometimes be veeery similar.)
Last Edit: 4 years 7 months ago by Shinrai.
The administrator has disabled public write access.

Re: Proposed Changes to Comic Vine Scraper Settings 4 years 7 months ago #32516

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
Well, no algorithm is perfect, and certainly this one won't be. Those who are not comfortable with he possibility of it making an error should simply continue using the scraper without automatic matching--the feature won't be forced on anyone, and in fact it won't even be turned on by default.
The administrator has disabled public write access.

Re: Proposed Changes to Comic Vine Scraper Settings 4 years 7 months ago #32527

  • Madmatx
  • Madmatx's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 457
  • Thank you received: 63
  • Karma: 19
If you are scraping "Random_Comic 12 (2013)" but the file is accidentally named "Random_Comic 02 (2013)" will the "auto scrape" feature

A. check issue #2 and then tell you covers don't match?
B. find the match for issue #12 but ask you to confirm it?
C. scrape it as issue #12?

I guessing it would be the first option.
Last Edit: 4 years 7 months ago by Madmatx.
The administrator has disabled public write access.

Re: Proposed Changes to Comic Vine Scraper Settings 4 years 7 months ago #32528

  • Mammut
  • Mammut's Avatar
  • Offline
  • Gold Boarder
  • Posts: 196
  • Thank you received: 25
  • Karma: 7
I agree with Edition Edtion, you don't need to throw away an already working setting, there can be 3.

About the same cover for issue and tpb problem. Is there any flag which tells you which series have the same cover? That way the Scraper can ask which one to use.
I guess it would be much more time to compare the covers of the issues and the tpb first all the time.
Last Edit: 4 years 7 months ago by Mammut.
The administrator has disabled public write access.

Re: Proposed Changes to Comic Vine Scraper Settings 4 years 7 months ago #32536

  • Mammut
  • Mammut's Avatar
  • Offline
  • Gold Boarder
  • Posts: 196
  • Thank you received: 25
  • Karma: 7
Edtion wrote:
Edtion

Edtion, sorry.
The administrator has disabled public write access.
The following user(s) said Thank You: Edtion
Time to create page: 0.220 seconds

Who's Online

We have 221 guests and 4 members online