Welcome, Guest
Python Scripts for ComicRack

TOPIC: [Script] Find Missing Comics using CV Offline (Updated: 20-Sep-2017, NEW FORMAT)

[Script] Find Missing Comics using Comicvine Offline (Updated: 11-Sep-2017) 1 month 4 days ago #48195

  • Xelloss
  • Xelloss's Avatar
  • Online
  • Platinum Boarder
  • Posts: 387
  • Thank you received: 101
  • Karma: 21
bferrell wrote:
I'm really loving this script--you're awesome! As a way to say thank you, I wrote a small python program to update the mcl file. I haven't fully tested it, but for "good" inputs it seems to work just fine. This way you won't be too burdened with us waiting on you to provide updated files :)

Usage is pretty straightforward.
python update_missing.py <in_file> <out_file> <api_key> <start_date> <end_date>

An example command based on the current database version (fill in API_KEY with your own):
python update_missing.py missing.mcl missing_20170917.mcl API_KEY 2017-09-11 2017-09-17

There is more documentation in the py file itself. Feel free to copy/modify to your heart's desire.

Tested and it works PERFECTLY! It is awesome how well it updates it so easily and so fast!

Btw, I have to install Python (I use Iron Python in Windows) and the requests module for it to work...

note for other users: Ones you install python, just run: "python.exe -m pip install requests" for the module you need
Last Edit: 1 month 4 days ago by Xelloss.
The administrator has disabled public write access.

[Script] Find Missing Comics using Comicvine Offline (Updated: 11-Sep-2017) 1 month 3 days ago #48201

  • bferrell
  • bferrell's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 2
  • Thank you received: 2
  • Karma: 1
Glad I could help! You're not alone about not knowing how exactly to query ComicVine by date. I spent a (frustrated) night searching for it, couldn't find it, put it down, and eventually found it buried on another page the next morning.

I think the reason the update script works so well is I just started off by seeing if I could simply read in and parse the mcl file and then write it out to a new file. From there I compared the two and found those little discrepancies. Just kept at it until the output was identical :) I hadn't really programmed in Python before (typically a C++ guy), so this whole thing was a learning experience lol. Is IronPython the standard for ComicRack developers?

I like your proposal for the comma issue, but I may have a more robust solution if you're open to it. How about we use the csv module? It's built to handle the comma dilemma by wrapping offending entries in double quotes. Wrote a toy program (csv_split.py) to demonstrate this. Just run 'python csv_split.py'

Looking at the updated version of your script I believe all you'd need to modify is line 184 to be something similar to what I have on line 14 in the toy program (and adding the two needed imports). I've 'cleaned up' the mcl file so the issue nums adhere to the CSV specification. Also, you shouldn't need the fix code (lines 170-180 ish) anymore :)

Hooray for parsing!

Cleaned up MCL file
Just exit out of the popup if it asks you to sign up.

Attached:
1. csv_split.py : toy program using CSV parsing
2. update_missing.py : update script using CSV parsing
Attachments:
The administrator has disabled public write access.
The following user(s) said Thank You: Xelloss

[Script] Find Missing Comics using Comicvine Offline (Updated: 11-Sep-2017) 1 month 3 days ago #48204

  • Xelloss
  • Xelloss's Avatar
  • Online
  • Platinum Boarder
  • Posts: 387
  • Thank you received: 101
  • Karma: 21
ok... I came here to post something, and now I have more things to read and learn XD

Well, first of all, I tinkered a bit with your script, and add a few "fixes" for unicode characters in the API content (I tried to make a database with your script from 0 and it crashed becuase of this XD). As unicode is my arch-enemy, I don't know if the solution I implemented is the correct one, but at least it doesn't crash anymore XD (you can have a look at it if you want to)

I also changed the "," and ";" in the numbers for "." to avoid problems when splitting them XD

As I have deleted the "s in my own mcl, I also deleted the line where you added the "s in yours, I will now read what you write above and see if that is coherent with that (I read it quickly, I will read it again after posting this XD)

About programming knowledge, I also "learned" python with Comicrack... (and by learn I mean tinker with already made scripts until I learned how it worked). I had also worked (a bit) with C++ before... But to tell you the truth, I think I know more python now than what I remembered from C++ (and not because I know Python XD)

Every time I open other people scripts, I learned something new about Python... In your case for example, the use of [:-1] instead of [0,len(string)-1] is something I had been looking for XD. Also the interkeys() is something interesting.

As someone totally noob in programming, feel free to correct me in any noob mistake/horror I make... My script are 90% trial and error, hahaha

About IronPython, I don't know... I think I read someone using it here in the forum and I downloaded it (I had no idea where to start with python). To tell you the truth, for years I made my scripts only using comicrack as my compilator :P (I even used to run scripts for other things from comicrack to avoid having to install anything else, hahaha). I downloaded IronPython less than a year ago, when a project I have in datamining data from the marvel wikia grew too big and needed modules comicrack didn't have... (and I realised even when it had started being about comicrack, it ended up being something completely apart from it, at least for now)

File Attachment:

File Name: update_missing.zip
File Size:2 KB


ps: Excuse my horrible English, not my mother tongue and not much practice in my daily life apart from comics
Last Edit: 1 month 3 days ago by Xelloss.
The administrator has disabled public write access.

[Script] Find Missing Comics using Comicvine Offline (Updated: 11-Sep-2017) 1 month 3 days ago #48205

  • Xelloss
  • Xelloss's Avatar
  • Online
  • Platinum Boarder
  • Posts: 387
  • Thank you received: 101
  • Karma: 21
I read your solution, and I think it is indeed very professional... but my question is, is the csv module part of comicrack? I try not to use modules not in comicrack in my scripts, for users to install them easily... (that is if the idea is to use it inside my script).

I use in Comicrack the modules from my IronPyhton library, but most users use the defaults that come with CR

If the problem is only the ","s, I think to use CSV parsing is a bit of and overkill... we could change it for another symbol (or symbols) without any problem inside the mcl file and then change them in the script output again for a ",". All this without having to use and extra module in the script... On the other hand if you think this will help avoiding other problems of the same type, that would be a plus.

Also I don't know how the csv module behave with unicode... (comicvine use unicode), and that could be also a problem...

All in all, I think you have much more experience than me with this. If you think using the csv module is the best, I will change my script (always only if it is a module already present in CR) and the way I store the data in the mcl files...
Last Edit: 1 month 3 days ago by Xelloss.
The administrator has disabled public write access.

[Script] Find Missing Comics using Comicvine Offline (Updated: 11-Sep-2017) 1 month 3 days ago #48206

  • Xelloss
  • Xelloss's Avatar
  • Online
  • Platinum Boarder
  • Posts: 387
  • Thank you received: 101
  • Karma: 21
How about this solution:

File Attachment:

File Name: update_missing-2.zip
File Size:2 KB


File Attachment:

File Name: FindMissin...-2-3.zip
File Size:4 KB


This will load (from the API) "," as ".&@1" and ";" as".&@2" (and then replace them in the output to correct symbol)

Also the update_missing will not only work with old mcl files, but will update them to the new standard

The findmissing still works with old and new mcl files too...

I am making a new mcl file from 0 using this version of your script, to fix any mistake or problem that could be in my mcl file, when finished I will upload it :)
Last Edit: 1 month 3 days ago by Xelloss.
The administrator has disabled public write access.

[Script] Find Missing Comics using Comicvine Offline (Updated: 11-Sep-2017) 1 month 1 day ago #48222

  • Xelloss
  • Xelloss's Avatar
  • Online
  • Platinum Boarder
  • Posts: 387
  • Thank you received: 101
  • Karma: 21
Well, I have been playing a bit with the script (update script) and I have built a new mcl file with it from 0 (fixing a few errors and inconsistencies in my usual mcl file)

This one is completely updated, and as far I can tell, 100% complete (minus one comic :P)

I use this opportunity to test the update script, and it works perfectly with all the comicvine database. I found only one error, and it was because of some error in the comicvine database:

If you run a full update from 0, from the beggning of time, the script will crash on offset: "358000" (it took me a bit to find where the bug was)

This is because for some reason the api itself crash when you ask for a specific comic, the one between id:407157 and id:407158 (let's call it 407157.5). Any comic query that contain that comic, crashed... So I just modify the script a bit (temporarily) to "avoid" this comic, and include all comics in the gap before and after this one... in the 358000 offset... and now I think the mcl is complete (I found about 300 comics that for some reason my old mcl file didn't include)

I have tested the file with my library (that constains about 50.000 comic entries) and the only ones the script didn't recognised where the ones outdated (I scraped them again, and they are recognised now, probably because of changes in the comicvine site since the last time I scraped them), which were only 3 or 4 comics...

So it is probable that if a comic was recognised with my old mcl file and now it isn't, it is because it is outdated. Just scrap them again and they will work ok.


The new mcl file is built according to new specifications discused here in the last days (dealing better with , and ; in numbers), so I strongly recommend to download it and delete the old ones... That said, the last version of my script deal better with the old ones than before, but still I recommend using the new one.

For those interested, you can now update the mcl file easily (you need python installed) with the GREAT bferrell script (I recommend to use the modified version I posted yesterday for the current version of my script, but the original version should work ok)

If bferrell gives his permission, I would like to add this script to the main page of this topic... with a brief tutorial in how to use it :)

All the same, now that it is A LOT easier thanks to this script, more than ever, I will continue updating the mcl file in the main page as usual once in a while

The updated file, and the one I will use from now on to update it (update date: 20/9/17)

mega.nz/#!RYsFzJjY!M2cxuyUZopDxlVj0KoxtawJCZ0auuEgiQ9Slx7YnfBE
Last Edit: 1 month 1 day ago by Xelloss.
The administrator has disabled public write access.
The following user(s) said Thank You: rmagere, romsnesrom

[Script] Find Missing Comics using CV Offline (Updated: 20-Sep-2017, NEW FORMAT) 3 weeks 8 hours ago #48239

  • rmagere
  • rmagere's Avatar
  • Offline
  • Gold Boarder
  • Posts: 219
  • Thank you received: 23
  • Karma: 6
Very minor comment: is it possible to add an icon so that it can be run from the toolbar rather than just from right-click automation?
The administrator has disabled public write access.
The following user(s) said Thank You: Xelloss

[Script] Find Missing Comics using CV Offline (Updated: 20-Sep-2017, NEW FORMAT) 3 weeks 3 hours ago #48240

  • Xelloss
  • Xelloss's Avatar
  • Online
  • Platinum Boarder
  • Posts: 387
  • Thank you received: 101
  • Karma: 21
Something like this?

File Attachment:

File Name: FindMissin...crplugin
File Size:5 KB


(just look for a "magnifying glass" icon in the toolbar, or in files automation)

If someone has a better icon, just tell me XD
Last Edit: 3 weeks 3 hours ago by Xelloss.
The administrator has disabled public write access.
The following user(s) said Thank You: rmagere

[Script] Find Missing Comics using CV Offline (Updated: 20-Sep-2017, NEW FORMAT) 3 weeks 3 hours ago #48241

  • Xelloss
  • Xelloss's Avatar
  • Online
  • Platinum Boarder
  • Posts: 387
  • Thank you received: 101
  • Karma: 21
note to myself: I must clean my script folder of all old and lost projects XD



There some things there that I don't even remember what they were for XD (if they work at all XD)
Last Edit: 3 weeks 2 hours ago by Xelloss.
The administrator has disabled public write access.

[Script] Find Missing Comics using CV Offline (Updated: 20-Sep-2017, NEW FORMAT) 2 weeks 6 days ago #48242

  • rmagere
  • rmagere's Avatar
  • Offline
  • Gold Boarder
  • Posts: 219
  • Thank you received: 23
  • Karma: 6
Xelloss wrote:
Something like this?

Perfect! :laugh:
The administrator has disabled public write access.
Time to create page: 0.257 seconds

Who's Online

We have 172 guests and 6 members online