Welcome, Guest
News and Announcements

TOPIC: Comic Vine Scraper

Re: Comic Vine Scraper 1.0.65 4 years 5 months ago #33625

  • perezmu
  • perezmu's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1114
  • Thank you received: 64
  • Karma: 51
And now a suggestion (that probably should wait till Autumn since it is actually kind of a request):

When a comic is not automatically detected, skip it, proceed to the next and finish the whole batch, and only at the end present to the user those that were not detected, so the batch does not require user interaction until the end... Say 500 comics, and the 54th one is not detected... the process now is halted waiting for user input... how about going through the 500, match those possible, and then at the end request input for the missing ones.

Anyway, THANKS THANKS THANKS... I have progressed more in a night than in the previous ten days!
The administrator has disabled public write access.

Re: Comic Vine Scraper 1.0.65 4 years 5 months ago #33626

  • 600WPMPO
  • 600WPMPO's Avatar
  • Offline
  • Moderator
  • Posts: 3788
  • Thank you received: 557
  • Karma: 232
@perezmu: If you look at the screencast I posted in this thread, it does exactly what you are asking. When it finished the automatically scrapable ones, only then does it ask for user interaction.
Now Playing: The ComicRack Manual (Online)

See my new comics & gadgets on: Tumblr!
The administrator has disabled public write access.

Re: Comic Vine Scraper 1.0.65 4 years 5 months ago #33627

  • perezmu
  • perezmu's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1114
  • Thank you received: 64
  • Karma: 51
600WPMPO wrote:
@perezmu: If you look at the screencast I posted in this thread, it does exactly what you are asking. When it finished the automatically scrapable ones, only then does it ask for user interaction.

Yeah, but no! I mean, it looks like it does so, but from time to time (for me at least) it stops waiting for my input and then goes on!
The administrator has disabled public write access.

Re: Comic Vine Scraper 1.0.65 4 years 5 months ago #33628

  • perezmu
  • perezmu's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1114
  • Thank you received: 64
  • Karma: 51
perezmu wrote:
Strange bug here...

In particular this happens for "Captain America and the Hawkeye 629", or "Captain America and Iron Man 635" for which CVS suggests the TPB... I need to actually skip it to keep going, "Go Back" doesn't work.

Ok, let's add a little weirdness to this:

- If I try to scrape "Captain America and the Hawkeye" 629-630-631, the first one is matched to the TPB (identical cover), and I need to skip it ("Go back" does nothing) and then 630 and 631 are also matched to the TPB, even though the cover is completely different.

- BUT, if I scrape 629 alone, even though the cover matches that of the TPB, this comic alone makes a good match to the regular series!

So it seems that it is the conjunction of equal covers + several issues what makes it come up with the TPB...?
The administrator has disabled public write access.

Re: Comic Vine Scraper 1.0.65 4 years 5 months ago #33655

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1327
  • Thank you received: 508
  • Karma: 182
Hey guys, I just uploaded version 1.0.66. Give it a try; it should work better now, and should address all of the bugs you've mentioned recently.

It was always my intention to make it so that all of the "manual" scrapes happen at the end--the scraper should finish ALL of the comics that it can do automatically before it starts making you choose manually. It wasn't always doign this in 1.0.65, but I think I've fixed that up in 1.0.66.

@perezmu: using page counts to identify TPBs is a great idea, but unfortunately ComicVine doesn't provide me with this info in their web API.

@forkicks: the bug you reported should be fixed, but unfortunately "Arak Son of Thunder Annual" still doesn't scrape because it isn't available at ComicVine. :(

@revQuixo: I think you've run into a ComicRack bug that I reported quite a while ago (except that now I can't find where I posted it!) Basically, if you run "Clear Data" on any comic, the scraper will not be able to display any of the comic's pages until you restart ComicRack. This is because the ComicBook object (from ComicRack) suddenly starts reporting that it has zero pages. You can try it--just click "Clear Data" on any comic book, then try to scrape it. You will see the same weird behaviour you described in your last comment.

I'm not sure if cYo is interested in changing this behaviour in ComicRack, but unless he does, there's unfortunately no way I can fix this problem.
Last Edit: 4 years 5 months ago by cbanack.
The administrator has disabled public write access.

Re: Comic Vine Scraper 1.0.65 4 years 5 months ago #33657

  • cYo
  • cYo's Avatar
  • Offline
  • Moderator
  • Posts: 3476
  • Thank you received: 676
  • Karma: 181
I look into it.
The administrator has disabled public write access.

Re: Comic Vine Scraper 1.0.65 4 years 5 months ago #33661

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1327
  • Thank you received: 508
  • Karma: 182
cYo wrote:
I look into it.
Thanks cYo!

There's another bug I've noticed--it's been around for a while, but I only just figured out how to duplicate it reliably:

1) Start ComicRack, and open Comic Vine Scraper
2) Change some settings in the scraper so they are NOT at the default settings
3) Run Comic Vine Scraper. Scrape something, or cancel. Whatever, just make sure it runs.
4) Reinstall Comic Vine Scraper (the same version, or a newer version.)

After step 4, 95% of the time the toolbar icon for the scraper will be missing, and the scraper will be "half" uninstalled in the ComicRack preferences dialog. You can then install the scraper a second time, and that works OK, EXCEPT now all your settings are reverted back to default. (You can verify this; check the changes you made in step 2--they are gone!)

If you skip step 3 (that is, if you install a scraper .crplugin file WITHOUT first running the scraper), then everything works fine and you don't have to re-install the scraper a second time or lose your settings.

I've been trying to figure out what causes this (in case it's the scraper!) but I haven't come up with anything yet.
The administrator has disabled public write access.

Re: Comic Vine Scraper 1.0.65 4 years 5 months ago #33664

  • T3KN0Gh057
  • T3KN0Gh057's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 407
  • Thank you received: 114
  • Karma: 27
docdoom also reported on this bug as well... if the script is the last to have run, installing any version of the script will fail a good deal of the time (not all the time).

I think some of it is "locked" into memory. Maybe a filestream not getting closed or something.
The administrator has disabled public write access.

Re: Comic Vine Scraper 1.0.65 4 years 5 months ago #33666

  • perezmu
  • perezmu's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1114
  • Thank you received: 64
  • Karma: 51
cbanack wrote:
@perezmu: using page counts to identify TPBs is a great idea, but unfortunately ComicVine doesn't provide me with this info in their web API.

Yes, Cory, I guessed so, so I was thinking more about using the number of pages in the file being scrapped... I think it is a pretty safe bet to say that all files with, say, less than 65 pages is standard, and beyond that is likely a TPB... Just an idea.

Also, 1.66 is awesome. Really. Also, what surprises me most is how you have managed to factor in the cover comparison algorithm without any noticeable reduction on scrapping speed!!!!!!!!
The administrator has disabled public write access.

Re: Comic Vine Scraper 1.0.65 4 years 5 months ago #33741

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1327
  • Thank you received: 508
  • Karma: 182
Edtion wrote:
Many thanks for the update, I just ran the scraper on a lot of different series, fell asleep, and I only had less than 1/4 of them to scrape manually.

While I was doing this, I noticed that half the cover is displayed for wide/double-paged covers (TPB in my case), could the scraper remove the left side from it's search as it does the display?

Yup, it already does this.
Last Edit: 4 years 5 months ago by cbanack.
The administrator has disabled public write access.
Time to create page: 0.302 seconds

Who's Online

We have 255 guests and 5 members online