Welcome, Guest
News and Announcements

TOPIC: Comic Vine Scraper 1.0.33 - 1.0.38

Re: Comic Vine Scraper 1.0.33 - 1.0.37 6 years 10 months ago #11691

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1327
  • Thank you received: 508
  • Karma: 182
Thanks for the suggestions, pescuma!

A tray icon would look cool, but there's two problems I see with the idea:

1) plugins/addons don't usually put stuff in the tray--that's more the province of full-blown applications, like ComicRack.

2) I'm not sure what benefit the tray icon would be right now, since the scrape doesn't really run in the background. I guess if someone started a large re-scrape and then minimized the window to do something else for a while, it would be kind of handy. But in that case, I think something like this would be even better!

The image idea is pretty cool. I've thought about doing image recognition in CVS--it's actually not that hard, for our purposes anyway. All you have to do is divide your images up into chunks (imagine a 6x6 grid, for example) and compare the color content of each chunk. So if chunk one is 25% red, 25% blue, and 50% green in both images, then the chunks are the same. Any image with approximately the same chunks throughout will be considered "identical". The neat thing is that this technique works even if the images are not the same size.

My plan was to try to make it so that the cover that appears in the "Issue Dialog" is automatically the one that most closely matches your comic book (i.e. in cases where there are multiple covers available from ComicVine.) I dropped the idea, though, because it would mean downloading ALL the images before displaying them, which means it could take up to 20 or 30 seconds before you'd see the Comic Vine cover image for the selected comic. :( (It's also kind of an abuse of the Comic Vine servers...)

Using the images to help with automatic identification of comics is a good idea, though, and one I hadn't thought of! :laugh:

The biggest obstacle to implementing this is actually the ComicVine API itself--ComicVine won't let me query for a single comic using just its series and issue number (which is really dumb, since that's the most obvious way for a program to identify a comic book.) This is the reason why there's sometimes that long delay (i.e. the progress bar) when you scrape a series for the first time--in order to find a single comic book, I have to download information about ALL the comic books in the series (and then cache it). This is also why I always show the cover image of the first issue in each series in the Series Dialog, and not the cover of the specific Issue that would be scraped if you chose that series.

This API flaw is also why I can't use your idea to use images to identify comics--at least not right now. It would take too long for the scraper to find the right image to compare, not to mention too much bandwidth. But I will keep this idea in mind (along with about 6 other major improvements to make) for the day when ComicVine listens to me and finally updates their API so I don't have this limitation. :lol:
Last Edit: 6 years 10 months ago by cbanack.
The topic has been locked.

Re: Comic Vine Scraper 1.0.33 - 1.0.37 6 years 10 months ago #11741

  • 600WPMPO
  • 600WPMPO's Avatar
  • Offline
  • Moderator
  • Posts: 3788
  • Thank you received: 557
  • Karma: 232
[Feature Request]

I often have to skip a lot of files while scraping, as ComicVine lacks their info.



I was thinking of an option of adding a tag to these skipped files as soon as the skip button is pressed; a tag called something like 'CVS Negative'.

Such a 'CVS Negative' tag would help exclude such comics from the 'To be Scraped' smart list, and hence we would not have to skip them everytime, and would not have to manually add such a tag to them.

What say cbanack. ? :unsure:
Now Playing: The ComicRack Manual (Online)

See my new comics & gadgets on: Tumblr!
Last Edit: 6 years 10 months ago by 600WPMPO.
The topic has been locked.

Re: Comic Vine Scraper 1.0.33 - 1.0.37 6 years 10 months ago #11744

  • Shinrai
  • Shinrai's Avatar
  • Offline
  • Platinum Boarder
  • With great power comes great W/T.
  • Posts: 885
  • Thank you received: 81
  • Karma: 33
I usually manually add a CVDB-BLANK tag to them but something standardized (that the script would recognize) would be nice. The issue with skipping everytime is that sometimes they get updated eventually! (Of course I never scrape my entire library anyway)
The topic has been locked.

Re: Comic Vine Scraper 1.0.33 - 1.0.37 6 years 10 months ago #11745

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1327
  • Thank you received: 508
  • Karma: 182
Well, I think having both a "CVDBNEGATIVE" and a "CVDBSKIP" tag would be a little be redundant, since, if I understand you right, they would both do the same thing...? Or do you mean that the "Skip" button would automatically add a different tag to your files--one that you could use for your smartlists, etc, but it wouldn't have any effect on the scraper's behaviour?

(For those who don't know, if you manually add the tag "CVDBSKIP" to any comic book, the scraper will always automatically skip over it, at least until you remove that tag.)
The topic has been locked.

Re: Comic Vine Scraper 1.0.33 - 1.0.37 6 years 10 months ago #11747

  • 600WPMPO
  • 600WPMPO's Avatar
  • Offline
  • Moderator
  • Posts: 3788
  • Thank you received: 557
  • Karma: 232
cbanack wrote:
..do you mean that the "Skip" button would automatically add a different tag to your files--one that you could use for your smartlists, etc, but it wouldn't have any effect on the scraper's behaviour?
No, I meant that the scraper automatically adds a CVDBSKIP tag when we skip a comic (this may be given as an option in settings), so that the next time the skipped file doesn't come up again & we don' have to manually add the CVDBSKIP tag.
Now Playing: The ComicRack Manual (Online)

See my new comics & gadgets on: Tumblr!
The topic has been locked.

Re: Comic Vine Scraper 1.0.33 - 1.0.37 6 years 10 months ago #11748

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1327
  • Thank you received: 508
  • Karma: 182
Well, I'll have to think about that one for a while, and see if I can come up with a way to do it that won't have new users mistakenly thinking that their previously skipped comic books are now "unscrapable". (I can just see the bug reports already...) :)
The topic has been locked.
The following user(s) said Thank You: 600WPMPO

Re: Comic Vine Scraper 1.0.33 - 1.0.37 6 years 9 months ago #11778

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1327
  • Thank you received: 508
  • Karma: 182
Hey guys, it's been a while since I had a bug report with Comic Vine Scraper, so I'm declaring version 1.0.37 to be "stable". That version has been out for over a week now, so many of you probably already have it. For those who don't, now's the time to upgrade! :cheer:
The topic has been locked.

Re: Comic Vine Scraper 1.0.33 - 1.0.37 6 years 9 months ago #11779

  • forkicks
  • forkicks's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 871
  • Thank you received: 109
  • Karma: 37
Is it just me, or has comicvine been awfully slow these past 2 days, to the point of being unusable? I can't get anything to scrape, takes ages to show resuls list, then another awful amount of time to get the details.
The topic has been locked.

Re: Comic Vine Scraper 1.0.33 - 1.0.37 6 years 9 months ago #11780

  • 600WPMPO
  • 600WPMPO's Avatar
  • Offline
  • Moderator
  • Posts: 3788
  • Thank you received: 557
  • Karma: 232
forkicks wrote:
Is it just me, or has comicvine been awfully slow these past 2 days, to the point of being unusable? I can't get anything to scrape, takes ages to show resuls list, then another awful amount of time to get the details.
I just scraped a few with the now declared 'stable' ;) 1.0.37 and everything was fine..!
Now Playing: The ComicRack Manual (Online)

See my new comics & gadgets on: Tumblr!
The topic has been locked.

Re: Comic Vine Scraper 1.0.33 - 1.0.37 6 years 9 months ago #11781

  • forkicks
  • forkicks's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 871
  • Thank you received: 109
  • Karma: 37
600WPMPO wrote:
I just scraped a few with the now declared 'stable' ;) 1.0.37 and everything was fine..!

Ok, thanks. I'll try on one of the other machines to see if i get something different then.
fK

[Edit: Its my ISP thats doing something weird. Going through a vpn connection through my workplace and no problem anymore. I'll blame this on ISP gremlins.]

[Edit #2: -OR- it just occurred to me, perphaps comicvine didn't like me scraping my collection on a weekly basis and is throttling my ip address? Has anyone had problems like these before?]
Last Edit: 6 years 9 months ago by forkicks.
The topic has been locked.
Time to create page: 0.214 seconds

Who's Online

We have 258 guests and 3 members online