Welcome, Guest
News and Announcements

TOPIC: Comic Vine Scraper

Comic Vine Scraper 5 months 4 weeks ago #47750

  • Drybonz
  • Drybonz's Avatar
  • Offline
  • Gold Boarder
  • Posts: 296
  • Thank you received: 2
  • Karma: 10
I was wondering if there was a way to have the scraper run in the background, or minimized, so that we can open books, etc while it is working. If not, is this a feature that could be added? Thank you.
The administrator has disabled public write access.

Comic Vine Scraper 5 months 4 weeks ago #47754

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
Drybonz wrote:
I was wondering if there was a way to have the scraper run in the background, or minimized, so that we can open books, etc while it is working. If not, is this a feature that could be added? Thank you.

Unfortunately, I suspect something like that would be too much work. I don't really think ComicRack is set up to run plugins in the background, and the Scraper certainly doesn't work in the background anyway. Though it is a good idea; if I were writing the whole thing over from scratch, I'd try to make it more thread aware (i.e. more able to run in the background.)

But these days I am only supporting the scraping the the form that it currently exists, not adding new features.
The administrator has disabled public write access.

Comic Vine Scraper 5 months 4 weeks ago #47755

  • Drybonz
  • Drybonz's Avatar
  • Offline
  • Gold Boarder
  • Posts: 296
  • Thank you received: 2
  • Karma: 10
Ok... thanks for the reply. I figured that might be the case with the state of ComicRack as it is... but worth a shot. Thanks again.
The administrator has disabled public write access.

Comic Vine Scraper 3 months 3 weeks ago #48077

  • Crave
  • Crave's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 3
  • Karma: 0
Is there a way to scrape the user ratings from Comic Vine again? I really miss that feature.
The administrator has disabled public write access.

Comic Vine Scraper 3 months 1 week ago #48119

  • n8thagr8
  • n8thagr8's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 11
  • Karma: 0
Hey

I noticed lately that the scraper keeps adding this (or some variation) to the plot summary of each book:

"List of covers and their creators:CoverNameCreator(s)Sidebar LocationACover AJamie McKelvie & Matthew Wilson1BCover BMeredith McClaren2CJonathan Hickman Month Variant Cover CJonathan Hickman3"

That's at the end of the plot summary i scraped for Wicked + Divine 30, for another example, this is the full plot summary metadata for Secret Empire #10:

"Can there be any redemption for Captain America as the SECRET EMPIRE starts to crumble?List of covers and their creators:CoverNameCreator(s)Sidebar LocationRegRegular CoverMark Brooks1VarVariant CoverJ. Scott Campbell2VarVillain Variant CoverDan Mora & Edgar Delgado3VarCivil Warrior Variant CoverGabriele Dell'Otto4VarAction Figure Variant CoverJohn Tyler Christopher5VarShield Variant CoverSteve McNiven6VarHydra Heroes Variant CoverAndrea Sorrentino7VarJack Kirby's 100th Birthday Variant CoverJack Kirby, Dick Ayers, Paul Mounts & Joe Frontirre8VarGenerations Variant CoverAlex Ross9"

Anyone know what causes this or how to fix it? It does not do this for every comic I scrape (probably more than half though) and I have no idea what is different about the ones it does and doesn't do this for.

Thanks
The administrator has disabled public write access.

Comic Vine Scraper 3 months 1 week ago #48120

  • pweasel
  • pweasel's Avatar
  • Offline
  • Expert Boarder
  • Posts: 124
  • Thank you received: 18
  • Karma: 8
I checked my W+D 30 and the plot summary is ok (the correct for the issue) + the alt-covers info that's badly formatted.
CRW 0.9.178 x64 on Win10
CRA 1.80 on Nexus 10
The administrator has disabled public write access.

Comic Vine Scraper 3 months 1 week ago #48121

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
The problem here is that comicvine allows html formatted text (I.e. web page programming) in its plot summaries, whereas ComicRack does not (at least it didn't the last time I looked).

It used to be the case that very few comics on comicvine actually used html in their plot summaries, so the text came over in the scraper without a problem. But that doesn't seem to be the case anymore.

Converting html to plain text isn't a simple process...if someone wants to try it, they could submit a patch to the Comic Vine Scraper code on GitHub. Otherwise, this would be one of the first problems I'll look at if I'm ever actively doing development on the scraper again.
The administrator has disabled public write access.
The following user(s) said Thank You: n8thagr8

Comic Vine Scraper 3 months 1 week ago #48122

  • boshuda
  • boshuda's Avatar
  • Offline
  • Gold Boarder
  • Posts: 296
  • Thank you received: 65
  • Karma: 8
There's a relatively easy way to clean it up using the (I believe) data manager. It's written about on here somewhere, not sure where. My CR is currently a little jacked up due to computer problems (yay creators update) or I would look. But basically any tool that allows you to modify the extra text added by ComicVine indicating the alternate covers will work.

BTW, if someone figures it out it would be helpful if it was added to the Wiki.
The administrator has disabled public write access.

Comic Vine Scraper 3 months 1 week ago #48123

  • Xelloss
  • Xelloss's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 463
  • Thank you received: 118
  • Karma: 24
It can be fixed easily with a script, I did one for fixing this...
The administrator has disabled public write access.

Comic Vine Scraper 3 months 1 week ago #48124

  • n8thagr8
  • n8thagr8's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 11
  • Karma: 0
Care to share? Lol
The administrator has disabled public write access.
Time to create page: 0.285 seconds

Who's Online

We have 254 guests and no members online