Welcome, Guest
Python Scripts for ComicRack

TOPIC: Import comic info from ComicVine

Import comic info from ComicVine 8 years 3 months ago #3860

  • perezmu
  • perezmu's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1114
  • Thank you received: 64
  • Karma: 51
Version 0.3.5 available here:
http://comicrack.cyolito.com/index.php?option=com_kunena&Itemid=9&func=view&catid=13&id=3860&limit=6&limitstart=54#4008
No bug fixes - I had interchanged debug and normal versions :blush:

Please, be sure to read THIS post for configuration instructions before using!!

Intro: After much lurking, finally I am able to contribute to something interesting to the community!!! This script will import info for the selected comics from the ComicVine database. This database is well maintained, supports and encourages external requests through a (somewhat incomplete but functional) API. This is version 0.1. I hope to improve and maintain it as discussed below and according from your user experience.


Credits: Note that this script is heavily based on a previous one by wadegiles. Thanks to the API available for ComicVine, most of my work has been to replace his works on regexpressions to XML tags management. Credit must be his. Also, I am using two external libraries for xml access coded by DevHawk (details in the source code)


Installation: As usual, you must extract the files included in the attached zip to the Comirack\\Scripts directory. *Note of caution*, for the script to work, you must be sure that the files "imprints.py", "ipypulldom.py" and "xml2py.py" are exactly under Comicrack\\Scripts and not under further subdirectories!!!!!!


Usage: As usual, select the ecomics you want to get info for, rightclick, select Automation -> ImportcomicVineInfo, and let the magic be done!

*Note of caution* The script is reasonably fast, but the larger the number of issues in the series that is being consulted, the slower it gets... for long series (Action Comics, Batman, Fantastic Four... and the like) the script is extremely slow... it can take 10 minutes to find your issue info! (subsequent issues in the sames series will be fast though. My suggestion is to run the script on several issues of these series at the same time, so your first wait period will not be repeated for each issue. My priority is to fix this for next versions... technical explanation on why this is so is included in a separate post below.

Differences to other scripts:
  1. Comicbookdb and Chronowerks do Annuals as special issues, while comicvine provides separate series for them... if you are getting info for an annual, make sure to select the appropriate series in the list shown.

  2. ComicVine does not provide volume numbers, only the start year of a volume is available to identify (thus, you have Fantastic Four (1963) and Fantastic Four (1996), not FF v1 and FF v2). So, the option to use start year as volume id present in previous scripts is not available anymore, and this approach is thus forced!

  3. ComicVine does not consider Imprints (such as Vertigo), but considers them Publishers. Since I think this is not correct, I provide an option to override this behavior and (from a list I provide in the file imprints.py) when an Imprint is found in the Publisher field, it is moved to the Imprint field and the proper Publisher filled. Note this is an option in the first dialog.
Troubleshooting:
To help us troubleshoot any further problems, I am including two files:
ImportComicVineInfo.py and ImportComicVineInfo.Debug.py. They are identical, except for the fact that the second is overly verbose on the console to help me track any errors (thus is much slower)

If you find something and need to report some error, it would be a great help if you could do the following:
  • Exit ComicRack (make sure it is not in your system tray!)
  • Rename ImportComicVineInfo.Debug.py to ImportComicVineInfo.py (make sure you back up the original first) overwriting the original
  • Start ComicRack in "debug" mode
    To make it simpler debugging your script you should start ComicRack with the following command line (or add those options to the shortcut in your desktop or programs menu):

    ComicRack.exe -dso -ssc

    The meaning of these switches is:

    -dso is Disable Script Optimization
    Scripts are always reloaded when executed. Otherwise they are compiled once during startup.
    -ssc is Show Script Console
    With this you get a look at all your script output (good for debugging messages).
    Now you should have the normal ComicRack window and a little text window were the scripts will spit their output
  • Run the script on the comics you found to give problems
  • Go to the Script Console, copy the log and send it to me using pastebin.com
  • Finally, in your bug post, please be sure to include the following:
    1. ComicRack Version
    2. Windows Version (inc. 32,64 bits)
    3. Link to your debug log pasted at pastebin.com
Final Words and Request: I hope this script serves you well. The ComicVine database is reasonably well populated. I suggest not to use the script over too many issues at the time until you are sure it works for you. Please, if you use this script, and find any bug, post here... provide me any info you can (error, if running in debug mode, and exact comic you were looking for -the series + issue number provided by comicrack, not the filename).

Cheers! and Enjoy

File Attachment:

File Name: ImportComi...fo_0.zip
File Size:20 KB
Last Edit: 8 years 2 months ago by perezmu.
The administrator has disabled public write access.

Re:Import comic info from ComicVine 8 years 3 months ago #3861

  • perezmu
  • perezmu's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1114
  • Thank you received: 64
  • Karma: 51
Why is your script so #$#!@# slow???

I know. But I am no programmer... I write programs that work but are far from optimized.. if you can do it faster, be my guest ;). Seriously, I think optimizing code is an art in itself, gift and craft I do not hold!

Now to the technical issues:

- There is a slow initiation reading a large dictionary from scripts

- Due to the nature of the objects created by the xml2py I heavily use exceptions to identify the pressence or not of an element... I am not sure, but I suspect this makes the soft very slow.

- The worst part: for series with large number of issues (>250), the script is very, very slow. This, as far as I have been able to investigate is caused by a limitation of the ComicVine API. If you search for a given series (try in an XML capable browser -IE- : http://api.comicvine.com/search/?api_key=bf7d30d5fa29cade917c2ec4b431ee78e79d81d3&resources=volume&query=daredevil&field_list=name,id,start_year&sort=start_year&format=xml), and then get the info for the corresponding series, for example http://api.comicvine.com/volume/2190/?api_key=bf7d30d5fa29cade917c2ec4b431ee78e79d81d3&format=xml&field_list=issues, you can see that for each issue it provides the internal database id and its title?!?!?! but not each name...!!!! so, next I need to stablish a connection and a query to the server for EVERY AND EACH issue id in order to get its issue_number (see http://api.comicvine.com/issue/136975/?api_key=bf7d30d5fa29cade917c2ec4b431ee78e79d81d3&format=xml&... so this can mean 300, 500 or 800 connections before getting to the correct issue!!!!

For some series, there seems to be a sequential order of the id's and the issue numbers, but this, I fear, needs not be so. If that is the case for ALL series, and that is what I am investigating, then I could, at least, bound the range of ids which might contain the issue and take an educated guess, getting less issues before reaching the needed one. But this is yet a guess, and needs to be done.
    So, if by any means, you can think of some approach to solve this procedural problem, please, let me know or have a go at it yourself with all my blessings
:P
Last Edit: 8 years 3 months ago by perezmu.
The administrator has disabled public write access.

Re:Import comic info from ComicVine 8 years 3 months ago #3865

  • perezmu
  • perezmu's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1114
  • Thank you received: 64
  • Karma: 51
Known bugs in version 0.1

Minor bugs identified so far (do not hinder proper use):

  • Spurious spaces appearing at the begining of some "titles" (fixed in next version)
  • Some imprints are not correctly identified, for example "Wildstorm", becuase different capitalization spelling in the imprints.py file (fixed in next version)
  • Some problems with non-standard characters is credits and summary
  • Last Edit: 8 years 3 months ago by perezmu.
    The administrator has disabled public write access.

    Re:Import comic info from ComicVine 8 years 3 months ago #3867

    • chaoshunter
    • chaoshunter's Avatar
    • Offline
    • Junior Boarder
    • Posts: 29
    • Karma: 0
    Just used the script, works pretty great.

    Just one suggestion though:

    Can you add an option whether or not to import the year of the comic into the volume area? I always add the volume number manually anyways, and it's a pain to go back and fix it.

    Otherwise, it's working great so far.

    Thanks a lot for the script. Speed is great too.
    The administrator has disabled public write access.

    Re:Import comic info from ComicVine 8 years 3 months ago #3868

    • perezmu
    • perezmu's Avatar
    • Offline
    • Platinum Boarder
    • Posts: 1114
    • Thank you received: 64
    • Karma: 51
    Good to hear someone is using it!

    I will honor your request, of course! Expect a new version tomorrow!
    The administrator has disabled public write access.

    Re:Import comic info from ComicVine 8 years 3 months ago #3869

    • chaoshunter
    • chaoshunter's Avatar
    • Offline
    • Junior Boarder
    • Posts: 29
    • Karma: 0
    Well, that's great to hear. I'm currently going through my library with the script and it works great. I have to be honest though, the feature which allows you to choose which info. is added to the comic or not is GREAT! I didn't realize it would be so helpful in library organization. It's almost a breeze now to organize (and update missing info.) in comics.
    The administrator has disabled public write access.

    Re:Import comic info from ComicVine 8 years 3 months ago #3871

    • perezmu
    • perezmu's Avatar
    • Offline
    • Platinum Boarder
    • Posts: 1114
    • Thank you received: 64
    • Karma: 51
    Good, it is there where I plan to add the option for volume update!
    The administrator has disabled public write access.

    Re:Import comic info from ComicVine 8 years 3 months ago #3873

    • Alan Scott
    • Alan Scott's Avatar
    • Offline
    • Gold Boarder
    • Posts: 237
    • Thank you received: 10
    • Karma: 10
    @ Perezmu - Thanks a ton! Nice to have a new option, especially one that is approved by the site. Their data does look pretty deep. Here's some notes for ya...

    I have indeed noticed some speed issues. Not every time, but here and there, it will slow or stop. It doesn't seem to matter the amount of issues; I've had it time out with as little as seven issues. Since there are a couple coders hanging around here I'm sure you'll get some help with this eventually.

    I notice that I am not able to import characters; I would really like to, so if are able to code this it would very much be appreciated.

    I am able to get summaries :woohoo: but they're having some text problems. Here's an example from Avengers Classic:

    OverviewIn the tradition of CLASSIC X-MEN, we’re re-presenting the original AVENGERS #1 digitally re-mastered with THX â„¢ surround sound and a brand spankin’ new cover by ARTHUR ADAMS! But that’s not all!!! We have not one, but TWO brand new short stories as well! DWAYNE McDUFFIE and painter MICHAEL AVON OEMING bring you a tale of the Avengers first tumultuous meeting and STAN “THE MAN” LEE (who started it all!) and artist extraordinaire KEVIN MAGUIRE bring you a compelling tale about how the Avengers REALLY formed! Face it, this book is HOT! 

    Thank you for changing the volume option in the next iteration.

    I see all this other forms of data on their site that I would love have, Teams, Locations, Concepts, Objects. What do you think are the chances you could coordinate with cYo to have these added to CR? Shoot, I almost wouldn't need Wiki anymore, which is what I covet. I want to be able to just search the comic itself for the data I need.
    ... The failure to appreciate... is perfectly understandable, because the readership never evaluates old material in the context of the cultural climate in which it was created, or the state of the art at the time it was created.
    Marty Pasko
    The administrator has disabled public write access.

    Re:Import comic info from ComicVine 8 years 3 months ago #3874

    • perezmu
    • perezmu's Avatar
    • Offline
    • Platinum Boarder
    • Posts: 1114
    • Thank you received: 64
    • Karma: 51
    Hey Alan!

    Thanks for the feedback... I appreciate it!

    Let's see your points:

    - I did not do characters at the first version because I wanted to know if someone (I don't) use them... Definitely I will include that, but I think first I need to address the two other problems you mention...

    - Yep, I am aware of the problems in the summaries text... they come up with non ascii characters, and this is one of my priorities now... it also happens in credits (like George Pérez).

    - Regarding other data... with the xml provided by the query at comicvine, once the issue is found, adding any info is straightforward, as long as CR supports it... the thing is, I guess cYo is watching us all, but he is nowhere to be seen! I would love to have him include as many fields as possible. Also, another interesting piece of info in Comicvine are Story-Arcs, which I plan to convert to ComicRack reading lists!

    - Finally, the speed issues... maybe I did not explain myself... the speed is not related to how many books you try to update in one go, but to other thing: if you try just one issue of a Series which has many issues in Comicvine's database, let's say Action Comics #734 (or #1 for that matter), due to the API structure, I need to grab all 800-something issues one by one and then do the search... doing this 800 plus queries to comicvine is what slows it so much.... on the contrary if you try to fetch issues from series with few issues (4,5,10...) it is very fast, since it only takes 4, 5 or 10 queries to get the whole list of issues for the series.

    So, I will try to solve the grabbled text issue, and next the characters info. :)

    cheers!
    The administrator has disabled public write access.

    Re:Import comic info from ComicVine 8 years 3 months ago #3875

    • perezmu
    • perezmu's Avatar
    • Offline
    • Platinum Boarder
    • Posts: 1114
    • Thank you received: 64
    • Karma: 51
    Alan, and everyone :)

    Here is my suggestion for you, if you want to be able to update the info real fast with new versions of the script: Like previous scripts (wadegile's), this one writes the Comicvine id for each issue in the tags field... CVDBXXXXX... When asked to search for info, the script first checks if this info exists in the tags fields, and if it does, it goes directly to the issue id'ed so grabbing more info (like characters when available) is very swift...

    Now, what happens if the db is corrupted (as happened to you), or you download the info in a computer (let's say at work :blush: ) and then you want to get this new data... well the tag CVDBXXXXX is gone! and the search must be done again (slow!)... This is what I do, and the reason for the other little scripts called toolbox:

    - I convert my comics to cbz
    - I get info from comicvine
    - I use the SaveTagstoNotes script, so a copy of the CVDBXXXXX tag is saved to the Notes field (note: a bug in the script overwrites all other info in the notes, so use with care!)
    - I save the comic with all this info

    Now if I move the comic somewhere else, or my database was corrupted, I use RetrieveTagsfromNotes, to restore the CVDBXXXXX tags, and updating the comic info is now a matter of seconds!

    This seems to be an obsession by me, nobody else seems to worry about this, but just in case :silly:
    The administrator has disabled public write access.
    Time to create page: 0.235 seconds

    Who's Online

    We have 198 guests and 4 members online