Welcome, Guest
Python Scripts for ComicRack
  • Page:
  • 1
  • 2

TOPIC: ComicVine Scraper - also write ComicBookInfo data (for ComicBookLover)

ComicVine Scraper - also write ComicBookInfo data (for ComicBookLover) 5 years 10 months ago #20190

  • superjoe
  • superjoe's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 12
  • Karma: 0
Can ComicVine Scraper be made to write the ComicBookInfo to a .cbz that ComicBookLover for iPad looks for, when it creates the .xml for ComicRack?

Or does ComicRack itself do the .xml writing in the first place?
The administrator has disabled public write access.

Re: ComicVine Scraper - also write ComicBookInfo data (for ComicBookLover) 5 years 10 months ago #20197

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
In fact, this IS something in my medium-to-long range plans. But it will require an enormous amount of changes to the existing scraper. Right now, as you say, the scraper plugin doesn't directly write to the comic files, ComicRack does. And it's not really safe to change that--when ComicRack is running, other programs shouldn't mess with the comic files directly. That means I will probably need to create a separate, standalone version of the scraper in order to do what you're suggesting.

Still, I've been slowly chipping away at the changes need to do that kind of thing, but I'm not going to be finished soon (i.e. not in the next release of the scraper.) I just noticed today that ComicVine seems to have improved their API format, which should allow me to greatly speed up how fast the scraper runs...so that's going to be my project for the next little while and for the next release of the scraper.
Last Edit: 5 years 10 months ago by cbanack.
The administrator has disabled public write access.

Re: ComicVine Scraper - also write ComicBookInfo data (for ComicBookLover) 5 years 10 months ago #20203

  • 600WPMPO
  • 600WPMPO's Avatar
  • Offline
  • Moderator
  • Posts: 3788
  • Thank you received: 557
  • Karma: 233
cbanack wrote:
...I will probably need to create a separate, standalone version of the scraper in order to do what you're suggesting.

......

I just noticed today that ComicVine seems to have improved their API format, which should allow me to greatly speed up how fast the scraper runs...so that's going to be my project for the next little while and for the next release of the scraper.
Eagerly awaiting both of these! :)
Now Playing: The ComicRack Manual (Online)

See my new comics & gadgets on: Tumblr!
The administrator has disabled public write access.

Re: ComicVine Scraper - also write ComicBookInfo data (for ComicBookLover) 5 years 10 months ago #20210

  • superjoe
  • superjoe's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 12
  • Karma: 0
That would be terrific, I really appreciate all of your hard work!
The administrator has disabled public write access.

Re: ComicVine Scraper - also write ComicBookInfo data (for ComicBookLover) 5 years 10 months ago #20240

  • perezmu
  • perezmu's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1114
  • Thank you received: 64
  • Karma: 51
cbanack wrote:
Still, I've been slowly chipping away at the changes need to do that kind of thing, but I'm not going to be finished soon (i.e. not in the next release of the scraper.) I just noticed today that ComicVine seems to have improved their API format, which should allow me to greatly speed up how fast the scraper runs...so that's going to be my project for the next little while and for the next release of the scraper.

Cory, I keep lurking your google code page and get notified to the changes that you make to the script. It is long past the time when I could follow what you did, but I can see you keep adding here and there, polishing and the like... and now you added all those issues related to the changes to the API (I have not checked those yet), but I guess this means we are in for a great new version when it is ready... Not pushing :), just wanted to acknowledge your great work on this script once more!!!! (and try to get you to tell us about your plans... :P )...
The administrator has disabled public write access.

Re: ComicVine Scraper - also write ComicBookInfo data (for ComicBookLover) 5 years 10 months ago #20247

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
Hmmm I thought I saw you sneaking around the google code page... :lol:

As for my plans, the next release is going to focus on making use of the improved ComicVine API, so many of the changes will be "under the hood" -- the Comic Vine connection code will get a lot cleaner and easier to read, but the main thing most people will notice is that it runs faster. Especially the first time you scrape a series--no more "loading issues" dialog.

The improved API also gives me fast access to the title of each issue (when Comic Vine has one), so I've already changed the "Show Issues" dialog to display those titles, instead of the series name repeated over and over (which was kind of useless anyway):



I've got a few bugs to fix, of course, and obviously the story arcs should get scraped into the new Story Arc metadata field that cYo added (right now they get scraped into the Alternate Series metadata field, but that's not really the right place anymore.)

Hmmm...what else? I've already changed the scraper so that it extracts the series names and issue numbers directly from the comic's filename when scraping a comic for the first time (instead of relying on ComicRack's "shadow values" to give it that information.) Most people won't notice much of a difference, it just gives me more flexibility when it comes to dealing with weirdly named series like "2000AD."

Also, I'd like to try to make it so that the series search results tend to "prefer" series that you've chosen in the past--so if there are 5 series called "Batman", the one you picked last time will be more likely to appear at the top of the list next time. It's sort of like your choices will "train" the scraper to make better guesses about which series you are collecting and scraping each week.

Oh yeah, and I'm going to add this "cvinfo" file feature. That one will be tremendously useful for people (like me!) who sort their comics into directories based on series.

I've got lots of other small (and bigger) ideas, but they'll have to wait. I'm not even sure I'll get to all of this stuff for the next release, but I'll try. I just don't want to push the release back too far, since it's already been a long time since we've had an update, and I know a lot of people are waiting for certain features, like the Arc metadata thing. B)
Last Edit: 5 years 10 months ago by cbanack.
The administrator has disabled public write access.
The following user(s) said Thank You: 600WPMPO

Re: ComicVine Scraper - also write ComicBookInfo data (for ComicBookLover) 5 years 10 months ago #20251

  • perezmu
  • perezmu's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1114
  • Thank you received: 64
  • Karma: 51
Just WOW...!!!!
The administrator has disabled public write access.

Re: ComicVine Scraper - also write ComicBookInfo data (for ComicBookLover) 5 years 10 months ago #20254

  • oraclexview
  • oraclexview's Avatar
  • Offline
  • Moderator
  • aka SoundWave
  • Posts: 906
  • Thank you received: 182
  • Karma: 37
perezmu wrote:
Just WOW...!!!!
I have to agree. cbanack, your work is beyond astounding. In fact, your great work makes you, in my opinion 2nd in command of the Astonishing & Uncanny CR-Men! Right behind cYo of course...cYo is the great Professor C after all! lol
cbanack wrote:
The improved API also gives me fast access to the title of each issue (when Comic Vine has one), so I've already changed the "Show Issues" dialog to display those titles, instead of the series name repeated over and over (which was kind of useless anyway):

I never really noticed how redundant the comic series being repeated over and over in that list was until you just pointed it out. Placing the title of the book there does make the column way more meaningful. Nice touch.
cbanack wrote:
Also, I'd like to try to make it so that the series search results tend to "prefer" series that you've chosen in the past--so if there are 5 series called "Batman", the one you picked last time will be more likely to appear at the top of the list next time. It's sort of like your choices will "train" the scraper to make better guesses about which series you are collecting and scraping each week.
You pull this off, and you are a programming genius with Python. I may be bugging you for personal lessons soon...be on the look out!
cbanack wrote:
Oh yeah, and I'm going to add this "cvinfo" file feature. That one will be tremendously useful for people (like me!) who sort their comics into directories based on series.
Another great addition to the "House of Cbanack Ideas" once you complete this feature. The XBMC media player has a lot of great script features written for it so I know for sure this is going to be an awesome function.
cbanack wrote:
I just don't want to push the release back too far, since it's already been a long time since we've had an update
My friend, with all the great work you've put into this, I don't think anyone can really complain about your time frame on releasing the next version. Please, take your time, and keep putting out the top quality script that is only matched by cYo's work on the ComicRack itself, both for Windows and for Android! Great job...you get madd kudos!
Last Edit: 5 years 10 months ago by oraclexview.
The administrator has disabled public write access.

Re: ComicVine Scraper - also write ComicBookInfo data (for ComicBookLover) 5 years 10 months ago #20256

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
oraclexview wrote:
I have to agree. cbanack, your work is beyond astounding. In fact, your great work makes you, in my opinion 2nd in command of the Astonishing & Uncanny CR-Men! Right behind cYo of course...cYo is the great Professor C after all! lol
Hopefully this means I'll get some Uncanny Superpowers! :woohoo:

But I could never compete with Professor C--his superpower must be the ability to code new features with his mind while he's asleep. Otherwise, I don't know how one person can write so much software so fast... :blink:
The administrator has disabled public write access.

Re: ComicVine Scraper - also write ComicBookInfo data (for ComicBookLover) 5 years 10 months ago #20263

  • 600WPMPO
  • 600WPMPO's Avatar
  • Offline
  • Moderator
  • Posts: 3788
  • Thank you received: 557
  • Karma: 233
cbanack wrote:
oraclexview wrote:
I have to agree. cbanack, your work is beyond astounding. In fact, your great work makes you, in my opinion 2nd in command of the Astonishing & Uncanny CR-Men! Right behind cYo of course...cYo is the great Professor C after all! lol
Hopefully this means I'll get some Uncanny Superpowers!

But I could never compete with Professor C--his superpower must be the ability to code new features with his mind while he's asleep. Otherwise, I don't know how one person can write so much software so fast...
All this is giving me ideas to have an X-Man (or Supe) assigned to each of the main ComicRack members! :laugh:

Now that cYo is Professor Xavier, I think cbanack will be Cyclops, Stonepaw will be Cyborg, and the evil {Oo} will be Deadpool.

What will you be oraclexview? :)
cbanack wrote:
I've got a few bugs to fix, of course, and obviously the story arcs should get scraped into the new Story Arc metadata field that cYo added (right now they get scraped into the Alternate Series metadata field, but that's not really the right place anymore.
I would wish to have your (and other members') opinions as to how to 'officially' (to be written in-the-manual) use the Alternate Series & the Story Arc fields.

I have made a new thread for this discussion here: Story Arc vs Alternate Series. Please contribute your thoughts there.
Now Playing: The ComicRack Manual (Online)

See my new comics & gadgets on: Tumblr!
Last Edit: 5 years 10 months ago by 600WPMPO.
The administrator has disabled public write access.
  • Page:
  • 1
  • 2
Time to create page: 0.410 seconds

Who's Online

We have 285 guests and one member online