Welcome, Guest
Python Scripts for ComicRack

TOPIC: Missing Issues using ComicVine (New Version 06-DEC-2014)

Re: Missing Issues using ComicVine (Now available for download) 6 years 1 month ago #17581

  • Samael69
  • Samael69's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 381
  • Thank you received: 47
  • Karma: 21
I don't need you're entire library yet. :) Check your PM, I sent you a link for a beta copy of the new release. It has all the memory optimizations in it, but more importantly for you, it has the code to "hopefully" skip the situation you're encountering AND place a message in your output indicating the offending record(s) so we can figure out what the problem is.
Last Edit: 6 years 1 month ago by Samael69.
The administrator has disabled public write access.

Re: Missing Issues using ComicVine (Now available for download) 6 years 1 month ago #17588

  • haywire
  • haywire's Avatar
  • Offline
  • Junior Boarder
  • Posts: 28
  • Thank you received: 2
  • Karma: 0
thanks, i'll check it out.

I did grab netbeans, and make a few tweaks to get me over the hurdle. Still don't know what the volume was causing the issue, but i'll try your beta copy and go from there.

I'd be happy to help wrap any issues you have, if you still have work to be done. Your app is probably tied the 2nd most important part of my import process at this point. (After the ComicVine scraper, which is a prerquisite of sorts, and tied with the Library Organizer)
The administrator has disabled public write access.

Re: Missing Issues using ComicVine (Now available for download) 6 years 1 month ago #17590

  • Samael69
  • Samael69's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 381
  • Thank you received: 47
  • Karma: 21
Functionally, there's not a lot I can think of. I've really been building it to suit my own needs and hope others can use it. It's kind of a narcissistic approach, but there you have it. :P There's a lot of new stuff coming in the next release, along with some of the feature requests, so I'd hold off before making a bunch of changes. I'll post the source code at the same time I post the app. One thing I'd like to resolve is threading. I'd love to eliminate the need of the DOS Window to see progress, but just don't have time to learn Java multi-threading. It really doesn't affect functionality...only aesthetics, so it's on my list of desired things to resolve, but not high on my list. The blacklist is a little clunky and could use some streamlining but, again, it works.

The next thing I'll probably start working on is a cache cleaner, so when one does library maintenance, there's no need to rebuild the entire cache to get rid of strays.

The CVS is not a "prerequisite of sorts", my app would not function at all without it...so huge props to Cory.
The administrator has disabled public write access.

Re: Missing Issues using ComicVine (Now available for download) 6 years 1 month ago #17591

  • haywire
  • haywire's Avatar
  • Offline
  • Junior Boarder
  • Posts: 28
  • Thank you received: 2
  • Karma: 0
I was actually thinking about threading out the requests to ComicVine. They seem to chug along pretty slowly, but I'd also be cautious about pounding the hell out of their API.

I've got the beta running through now. Seems to be stalling Processing Volumes. (or it could be that there's less console spam than the prior version).

I do see the process taking 13% cpu, which is likely 100% of 1 of my 8 cores. Memory usage is stuck at just over 300mb.
The administrator has disabled public write access.

Re: Missing Issues using ComicVine (Now available for download) 6 years 1 month ago #17593

  • Samael69
  • Samael69's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 381
  • Thank you received: 47
  • Karma: 21
No. One request at a time. This application already sends 10's of thousands of requests through in a short time, I don't want to flood their servers. It currently doesn't take THAT long.

You were probably using about the same amount of PCU before, but closer to 1.5 - 2G of memory. Correct?

I've added new functionality tonight. At the top of the report, it will now identify those books where the issue ID is no longer valid. For example...

ERROR: ISSUE ID IS INVALID PLEASE RESCRAPE - Turok, Son of Stone #1 v1956 (133111)

I've discovered I have nearly 40 such books. :woohoo:
Last Edit: 6 years 1 month ago by Samael69.
The administrator has disabled public write access.

Re: Missing Issues using ComicVine (Now available for download) 6 years 1 month ago #17598

  • haywire
  • haywire's Avatar
  • Offline
  • Junior Boarder
  • Posts: 28
  • Thank you received: 2
  • Karma: 0
Ok, so I've found the culprit, with your fix:
ERROR: REMOTE VOLUME NOT FOUND - Punisher: Empty Quarter, The(27640)

which is odd, as it surely does exist:
www.comicvine.com/the-punisher-empty-quarter/49-27640/

I'm re-scraping it and will run through everything again to see if helps anything
The administrator has disabled public write access.

Re: Missing Issues using ComicVine (Now available for download) 6 years 1 month ago #17600

  • Samael69
  • Samael69's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 381
  • Thank you received: 47
  • Karma: 21
That truly is peculiar. If recraping doesn't help, perhaps I'll get you to send me your cache files and DB through sendspace so I can look at the records. Glad to see the "work-around" worked though. :laugh:
Last Edit: 6 years 1 month ago by Samael69.
The administrator has disabled public write access.

Re: Missing Issues using ComicVine (New Version 23-OCT-2011) 6 years 1 month ago #17616

  • Samael69
  • Samael69's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 381
  • Thank you received: 47
  • Karma: 21
New version is up. See OP. This has not been tested extensively, please let me know if there are any issues.

Sorry for taking so long, but summer was crazy.

Cheers.
Last Edit: 6 years 1 month ago by Samael69.
The administrator has disabled public write access.
The following user(s) said Thank You: RevQuixo

Re: Missing Issues using ComicVine (New Version 23-OCT-2011) 6 years 1 month ago #17618

  • RevQuixo
  • RevQuixo's Avatar
  • Offline
  • Gold Boarder
  • Posts: 280
  • Thank you received: 26
  • Karma: 12
Running it now..super excited to see the output sorted by series.
The administrator has disabled public write access.

Re: Missing Issues using ComicVine (Now available for download) 6 years 1 month ago #17634

  • Samael69
  • Samael69's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 381
  • Thank you received: 47
  • Karma: 21
haywire wrote:
Ok, so I've found the culprit, with your fix:
ERROR: REMOTE VOLUME NOT FOUND - Punisher: Empty Quarter, The(27640)

which is odd, as it surely does exist:
www.comicvine.com/the-punisher-empty-quarter/49-27640/

I'm re-scraping it and will run through everything again to see if helps anything

No issue number on ComicVine...that's the problem.
<issue>
	<api_detail_url>http://api.comicvine.com/issue/169028/</api_detail_url>
	<id>169028</id>
	<issue_number/>
	<name>Empty Quarter</name>
</issue>
I mistakingly assumed issue number was a required field. It'll just skip such errors now. In the future, I'll look at at handling this properly, perhaps by assigning an internal number to such books. I suspect it should only affect TPBs and one-offs, that would never show on the final output to begin with, since these are the only ones that make sense to not have a number.
Last Edit: 6 years 1 month ago by Samael69.
The administrator has disabled public write access.
Time to create page: 0.385 seconds

Who's Online

We have 200 guests and 2 members online