Welcome, Guest
News and Announcements

TOPIC: Comic Vine Scraper

Comic Vine Scraper 3 months 6 days ago #48127

  • Xelloss
  • Xelloss's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 463
  • Thank you received: 118
  • Karma: 24
What my script do is search for "List of covers and their creator" and everything that comes after that and delete it, I am working in one that add this info but in a correct way. When it is ready I will release it in this forum :)
The administrator has disabled public write access.

Comic Vine Scraper 3 months 6 days ago #48128

  • Xelloss
  • Xelloss's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 463
  • Thank you received: 118
  • Karma: 24
If what you want to do is just delete the extra text, use this:

File Attachment:

File Name: DeleteCoverData.zip
File Size:1 KB


It os what I am using now... Remember it will delete all the text from the alternate covers... (you will have to rescrap the books for recovering this in the future)

It also counts the covers and save the result in a custom value called "Covers", before deleting the text.
The administrator has disabled public write access.
The following user(s) said Thank You: romsnesrom, n8thagr8

Comic Vine Scraper 3 months 6 days ago #48130

  • n8thagr8
  • n8thagr8's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 11
  • Karma: 0
You're amazing, thank you!
The administrator has disabled public write access.

Comic Vine Scraper 3 months 5 days ago #48131

  • Xelloss
  • Xelloss's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 463
  • Thank you received: 118
  • Karma: 24
No, thank you and I must apoligise for posting a script not well tested...

Thanks to this topic I revised this script (it was never made to be published in the forum, and I shouldn't have published it without testing it accordingly) and I improve it a lot... Here is the final version with all the fixes... please replace the other one with this.

Modifications:

* It now has a decent performance according with the simple task it makes... it should work with a lot of comics in less than a second...
* It recognises the data to read even if the person who wrote it used Capitals or not... and even if it uses a few different words than the most commons (based in all the books in my library)
* It will now ignore books who were already edited by the script before, and will not overwrite the "Covers" custom value with 1 the second time it runs
* Better number of covers recognition

Again, sorry for publishing a script with such a bad quility...

New version:

File Attachment:

File Name: DeleteCoverData-2.zip
File Size:1 KB
Last Edit: 3 months 5 days ago by Xelloss.
The administrator has disabled public write access.
The following user(s) said Thank You: romsnesrom, n8thagr8

Comic Vine Scraper 2 months 3 weeks ago #48178

  • boshuda
  • boshuda's Avatar
  • Offline
  • Gold Boarder
  • Posts: 296
  • Thank you received: 65
  • Karma: 8
cbanack wrote:
Drybonz wrote:
I was wondering if there was a way to have the scraper run in the background, or minimized, so that we can open books, etc while it is working. If not, is this a feature that could be added? Thank you.

Unfortunately, I suspect something like that would be too much work. I don't really think ComicRack is set up to run plugins in the background, and the Scraper certainly doesn't work in the background anyway. Though it is a good idea; if I were writing the whole thing over from scratch, I'd try to make it more thread aware (i.e. more able to run in the background.)

But these days I am only supporting the scraping the the form that it currently exists, not adding new features.

I believe ComicRack explicitly prevents the spawning of new threads from scripts anyway. I haven't confirmed it in any way, but I ran into something long ago that lead me to that conclusion.
The administrator has disabled public write access.

Comic Vine Scraper 2 months 3 weeks ago #48179

  • boshuda
  • boshuda's Avatar
  • Offline
  • Gold Boarder
  • Posts: 296
  • Thank you received: 65
  • Karma: 8
n8thagr8 wrote:
Hey

I noticed lately that the scraper keeps adding this (or some variation) to the plot summary of each book:

"List of covers and their creators:CoverNameCreator(s)Sidebar LocationACover AJamie McKelvie & Matthew Wilson1BCover BMeredith McClaren2CJonathan Hickman Month Variant Cover CJonathan Hickman3"

That's at the end of the plot summary i scraped for Wicked + Divine 30, for another example, this is the full plot summary metadata for Secret Empire #10:

"Can there be any redemption for Captain America as the SECRET EMPIRE starts to crumble?List of covers and their creators:CoverNameCreator(s)Sidebar LocationRegRegular CoverMark Brooks1VarVariant CoverJ. Scott Campbell2VarVillain Variant CoverDan Mora & Edgar Delgado3VarCivil Warrior Variant CoverGabriele Dell'Otto4VarAction Figure Variant CoverJohn Tyler Christopher5VarShield Variant CoverSteve McNiven6VarHydra Heroes Variant CoverAndrea Sorrentino7VarJack Kirby's 100th Birthday Variant CoverJack Kirby, Dick Ayers, Paul Mounts & Joe Frontirre8VarGenerations Variant CoverAlex Ross9"

Anyone know what causes this or how to fix it? It does not do this for every comic I scrape (probably more than half though) and I have no idea what is different about the ones it does and doesn't do this for.

Thanks

I take care of it with the Data Manager, which I always run as part of my normal workflow. To add the rule I use, create a new rule and copy the following into the Text line of your new rule. Then click on the little Update icon to the right of the text box.

<<Summary.ContainsAnyOf:List of covers||CoverNameCreator>> => <<Summary.RegexReplace:List of covers.*||>> <<Summary.RegexReplace:CoverNameCreator.*||>>
The administrator has disabled public write access.

Comic Vine Scraper 1 month 1 week ago #48471

  • Oberon1464
  • Oberon1464's Avatar
  • Offline
  • Expert Boarder
  • Posts: 82
  • Thank you received: 5
  • Karma: 1
Hi guys,

Is the scraper not working properly?
Im still in the process of pre-organizing my collection and only scrape now and then for testing purposes.
Yesterday I noticed that while scraping and matching a specific issue I'm not getting a picture of the cover with the "show Covers" option..

a few weeks back this was working without any problems.

Anyone else noticed this?
The administrator has disabled public write access.

Comic Vine Scraper 1 month 1 week ago #48472

  • boshuda
  • boshuda's Avatar
  • Offline
  • Gold Boarder
  • Posts: 296
  • Thank you received: 65
  • Karma: 8
If you're talking about the thing to browse extra covers that's been removed for ages due to ComicVine restrictions.

The regular cover display thing can be flaky. I suspect it's related to timeouts downloading the larger images from ComicVine. If you try to browse an issue on their site with a lot of covers it can take a really, really long time to get all of the covers as you scroll down the page.
The administrator has disabled public write access.

Comic Vine Scraper 1 month 1 week ago #48473

  • Oberon1464
  • Oberon1464's Avatar
  • Offline
  • Expert Boarder
  • Posts: 82
  • Thank you received: 5
  • Karma: 1
Yes, I was aware about the removal due to the restrictions.

Guess its because of the timeouts you're reffering to...

Thanks for you're quick reply Boshuda :)
The administrator has disabled public write access.

Comic Vine Scraper 1 month 6 days ago #48487

  • Pachilles
  • Pachilles's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 12
  • Karma: 0
I am having a strange problem. Scraper has been great, except for one comic.
The title matches, and I have the comicvine page with the matching title, but it refuses to find it in the Db.
"All-New Wolverine - Marvel Legacy Primer Pages"

Is there any way to force Comic Vine Scraper to see it, like maybe with the code in the webpage (4000-623734)?

Pachilles
The administrator has disabled public write access.
Time to create page: 0.387 seconds

Who's Online

We have 250 guests and no members online