Welcome, Guest
News and Announcements

TOPIC: Comic Vine Scraper 1.0.53-58

Re: Comic Vine Scraper 1.0.53-57 4 years 8 months ago #31390

  • T3KN0Gh057
  • T3KN0Gh057's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 407
  • Thank you received: 114
  • Karma: 27
@600WPMPO I know both you and me have some STRONG feelings about what should and shouldn't be in in ComicRack. But since we aren't the programmers *shrugs*

however.... I'm working on a program now that will read the ComicRack Library and embed ALL tags it can find into cbz files, though it's taking a back seat for the moment to my RegExp field swap script.
Last Edit: 4 years 8 months ago by T3KN0Gh057.
The topic has been locked.
The following user(s) said Thank You: 600WPMPO

Re: Comic Vine Scraper 1.0.53-57 4 years 8 months ago #31391

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
T3KN0Gh057 wrote:
you'll have that option 600 btw anyone know how to access System.Text.RegularExpressions in ComicRack scrips or will i have to make my own dll for my script?

This should work:
import clr
clr.AddReference('System.Text.RegularExpressions')
from System.Text.RegularExpressions import Regex

r = Regex( r'\s*Hi*' )
...

Note the 'r' before the regular expression. That's Python's raw string notation, which makes writing regular expressions much easier.

I've never compiled any code like the example above, though, because python actually has a very elegant and powerful regular expression engine of it's own. I use this heavily in ComicVineScraper:

docs.python.org/2/library/re.html

EDIT: Oh yeah, and that book.ReleasedTime code did work, just as you said. Clearly I need to get more sleep.
Last Edit: 4 years 8 months ago by cbanack.
The topic has been locked.

Re: Comic Vine Scraper 1.0.53-57 4 years 8 months ago #31392

  • T3KN0Gh057
  • T3KN0Gh057's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 407
  • Thank you received: 114
  • Karma: 27
cbanack wrote:
I've never compiled any code like the example above, though, because python actually has a very elegant and powerful regular expression engine of it's own. I use this heavily in ComicVineScraper:

docs.python.org/2/library/re.html

i've tried working with re
i don't doubt it's powerful, in fact its just plain too powerful.

I'm just to used to c# now, and .net Regular expressions are powerful AND intuitive (easy for dummies like me)

that's one of the reason i had to step away from C/C++ (but mean to go back soon now that i have C# down). C++ it's easy to get the basics down but the learning curve shoots straight through the roof after you learn variables built in keywords loops and classes (especially if you're self-educating). pythons regular expression engine is just a bit too much like that for me right now.
Last Edit: 4 years 8 months ago by T3KN0Gh057.
The topic has been locked.

Re: Comic Vine Scraper 1.0.53-57 4 years 8 months ago #31393

  • T3KN0Gh057
  • T3KN0Gh057's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 407
  • Thank you received: 114
  • Karma: 27
cbanack wrote:
This should work:
import clr
clr.AddReference('System.Text.RegularExpressions')
from System.Text.RegularExpressions import Regex
r = Regex( r'\s*Hi*' )
...


tried that and i get a nice greeting




I think the problem is with text encoding ... but i may be wrong
Last Edit: 4 years 8 months ago by T3KN0Gh057.
The topic has been locked.

Re: Comic Vine Scraper 1.0.53-57 4 years 8 months ago #31394

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
@T3KN0Gh057: I just tried copy-pasting that code into the scraper code and it works fine for me. Maybe there's something missing from your IronPython environment or something? What that code snippet is doing (adding a reference to a .NET Assembly and then importing it) is a *very* common thing to do in IronPython--it's the only way to access .NET libraries. So probably something fundamental about your IronPython environment/runtime is not set right.

If you can get a stack trace or more detailed error message for this error, it might give you more of a clue. (Post it in a new thread in the Developers section of the forum -- more people looking at it will increase the odds that someone will know what's wrong.)
The topic has been locked.

Re: Comic Vine Scraper 1.0.53-57 4 years 8 months ago #31395

  • T3KN0Gh057
  • T3KN0Gh057's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 407
  • Thank you received: 114
  • Karma: 27
cbanack wrote:
@T3KN0Gh057: I just tried copy-pasting that code into the scraper code and it works fine for me. Maybe there's something missing from your IronPython environment or something? What that code snippet is doing (adding a reference to a .NET Assembly and then importing it) is a *very* common thing to do in IronPython--it's the only way to access .NET libraries. So probably something fundamental about your IronPython environment/runtime is not set right.

If you can get a stack trace or more detailed error message for this error, it might give you more of a clue. (Post it in a new thread in the Developers section of the forum -- more people looking at it will increase the odds that someone will know what's wrong.)

(after pausing on handled exceptions) I got this:
System.IO.FileNotFoundException: Could not load file or assembly 'System' or one of its dependencies. The system cannot find the file specified.
   at static RuntimeAssembly System.Reflection.RuntimeAssembly.nLoad(System.Reflection.AssemblyName fileName, System.String codeBase, System.Security.Policy.Evidence assemblySecurity, System.Reflection.RuntimeAssembly locationHint, System.Threading.StackCrawlMark stackMark, System.Boolean throwOnFileNotFound, System.Boolean forIntrospection, System.Boolean suppressSecurityChecks)
   at static RuntimeAssembly System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(System.Reflection.AssemblyName assemblyRef, System.Security.Policy.Evidence assemblySecurity, System.Threading.StackCrawlMark stackMark, System.Boolean forIntrospection, System.Boolean suppressSecurityChecks)
   at static RuntimeAssembly System.Reflection.RuntimeAssembly.InternalLoad(System.String assemblyString, System.Security.Policy.Evidence assemblySecurity, System.Threading.StackCrawlMark stackMark, System.Boolean forIntrospection)
   at static Assembly System.Reflection.Assembly.Load(System.String assemblyString)
   at Assembly Microsoft.Scripting.PlatformAdaptationLayer.LoadAssembly(System.String name)
   at static Assembly IronPython.Runtime.ClrModule.LoadAssemblyByName(IronPython.Runtime.CodeContext context, System.String name)
   at static System.Void IronPython.Runtime.ClrModule.AddReference(IronPython.Runtime.CodeContext context, System.String name)
   at static System.Void IronPython.Runtime.ClrModule.AddReference(IronPython.Runtime.CodeContext context, System.Object reference)
   at static System.Void IronPython.Runtime.ClrModule.AddReference(IronPython.Runtime.CodeContext context, System.Object[] references)
   at Int32 Microsoft.Scripting.Interpreter.ActionCallInstruction`2[IronPython.Runtime.CodeContext,System.Object[]].Run(Microsoft.Scripting.Interpreter.InterpretedFrame frame)
   at System.Void Microsoft.Scripting.Interpreter.Interpreter.Run(Microsoft.Scripting.Interpreter.InterpretedFrame frame)
   at Object Microsoft.Scripting.Interpreter.LightLambda.Run4(System.Object arg0, System.Object arg1, System.Object arg2, System.Object arg3)
   at static Object System.Dynamic.UpdateDelegates.UpdateAndExecute3(System.Runtime.CompilerServices.CallSite site, System.Object arg0, System.Object arg1, System.Object arg2)
   at static Object <unnamed>$2.<unnamed>(IronPython.Runtime.CodeContext $globalContext, IronPython.Runtime.FunctionCode $functionCode) in C:\Users\T3KN0GH057\AppData\Roaming\cYo\ComicRack\Scripts\RegExSwap\regexswap.py:line 3
   at Object IronPython.Compiler.PythonScriptCode.RunWorker(IronPython.Runtime.CodeContext ctx)
   at Object IronPython.Compiler.PythonScriptCode.Run(Microsoft.Scripting.Runtime.Scope scope)
   at Object IronPython.Compiler.RuntimeScriptCode.InvokeTarget(Microsoft.Scripting.Runtime.Scope scope)
   at Object IronPython.Compiler.RuntimeScriptCode.Run(Microsoft.Scripting.Runtime.Scope scope)
   at Object Microsoft.Scripting.SourceUnit.Execute(Microsoft.Scripting.Runtime.Scope scope, Microsoft.Scripting.ErrorSink errorSink)
   at Object Microsoft.Scripting.Hosting.ScriptSource.Execute(Microsoft.Scripting.Hosting.ScriptScope scope)
   at System.Void cYo.Projects.ComicRack.Plugins.PythonCommand.OnPreCompile(System.Boolean handleException)
   at Object cYo.Projects.ComicRack.Plugins.PythonCommand.OnInvoke(System.Object[] data)
   at Object cYo.Projects.ComicRack.Plugins.Command.Invoke(System.Object[] data, System.Boolean catchErrors)
   at static System.Void cYo.Projects.ComicRack.Viewer.ScriptUtility.CreateBookCode(System.Windows.Forms.Control parent, cYo.Projects.ComicRack.Plugins.Command command, System.Func`1[System.Collections.Generic.IEnumerable`1[cYo.Projects.ComicRack.Engine.ComicBook]] books)
   at System.Void cYo.Projects.ComicRack.Viewer.ScriptUtility+<>c__DisplayClassb`1[System.__Canon].<CreateToolItem>b__8(System.Object s, System.EventArgs e)
   at System.Void System.Windows.Forms.ToolStripItem.RaiseEvent(System.Object key, System.EventArgs e)
   at System.Void System.Windows.Forms.ToolStripButton.OnClick(System.EventArgs e)
   at System.Void System.Windows.Forms.ToolStripItem.HandleClick(System.EventArgs e)
   at System.Void System.Windows.Forms.ToolStripItem.HandleMouseUp(System.Windows.Forms.MouseEventArgs e)
   at System.Void System.Windows.Forms.ToolStripItem.FireEventInteractive(System.EventArgs e, System.Windows.Forms.ToolStripItemEventType met)
   at System.Void System.Windows.Forms.ToolStripItem.FireEvent(System.EventArgs e, System.Windows.Forms.ToolStripItemEventType met)
   at System.Void System.Windows.Forms.ToolStrip.OnMouseUp(System.Windows.Forms.MouseEventArgs mea)
   at System.Void System.Windows.Forms.Control.WmMouseUp(System.Windows.Forms.Message m, System.Windows.Forms.MouseButtons button, System.Int32 clicks)
   at System.Void System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message m)
   at System.Void System.Windows.Forms.ScrollableControl.WndProc(System.Windows.Forms.Message m)
   at System.Void System.Windows.Forms.ToolStrip.WndProc(System.Windows.Forms.Message m)
   at System.Void System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message m)
   at System.Void System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message m)
   at IntPtr System.Windows.Forms.NativeWindow.Callback(System.IntPtr hWnd, System.Int32 msg, System.IntPtr wparam, System.IntPtr lparam)
   at Boolean System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(System.IntPtr dwComponentID, System.Int32 reason, System.Int32 pvLoopData)
   at System.Void System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(System.Int32 reason, System.Windows.Forms.ApplicationContext context)
   at System.Void System.Windows.Forms.Application+ThreadContext.RunMessageLoop(System.Int32 reason, System.Windows.Forms.ApplicationContext context)
   at static System.Void System.Windows.Forms.Application.Run(System.Windows.Forms.Form mainForm)
   at static System.Void cYo.Projects.ComicRack.Viewer.Program.StartNew(System.String[] args)
   at System.Void cYo.Common.Runtime.SingleInstance.Run(System.String[] args)
   at static Int32 cYo.Projects.ComicRack.Viewer.Program.Main(System.String[] args)


which i can't make heads or tails of.

The first line says it can't find "System"
but i load that fine when i take out System.Text.RegularExpressions reference

actually i got a whole host of errors the last of which was this
System.IO.IOException: Could not add reference to assembly System.Text.RegularExpressions
   at Object cYo.Projects.ComicRack.Plugins.Command.Invoke(System.Object[] data, System.Boolean catchErrors)
   at static System.Void cYo.Projects.ComicRack.Viewer.ScriptUtility.CreateBookCode(System.Windows.Forms.Control parent, cYo.Projects.ComicRack.Plugins.Command command, System.Func`1[System.Collections.Generic.IEnumerable`1[cYo.Projects.ComicRack.Engine.ComicBook]] books)
   at System.Void cYo.Projects.ComicRack.Viewer.ScriptUtility+<>c__DisplayClassb`1[System.__Canon].<CreateToolItem>b__8(System.Object s, System.EventArgs e)
   at System.Void System.Windows.Forms.ToolStripItem.RaiseEvent(System.Object key, System.EventArgs e)
   at System.Void System.Windows.Forms.ToolStripButton.OnClick(System.EventArgs e)
   at System.Void System.Windows.Forms.ToolStripItem.HandleClick(System.EventArgs e)
   at System.Void System.Windows.Forms.ToolStripItem.HandleMouseUp(System.Windows.Forms.MouseEventArgs e)
   at System.Void System.Windows.Forms.ToolStripItem.FireEventInteractive(System.EventArgs e, System.Windows.Forms.ToolStripItemEventType met)
   at System.Void System.Windows.Forms.ToolStripItem.FireEvent(System.EventArgs e, System.Windows.Forms.ToolStripItemEventType met)
   at System.Void System.Windows.Forms.ToolStrip.OnMouseUp(System.Windows.Forms.MouseEventArgs mea)
   at System.Void System.Windows.Forms.Control.WmMouseUp(System.Windows.Forms.Message m, System.Windows.Forms.MouseButtons button, System.Int32 clicks)
   at System.Void System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message m)
   at System.Void System.Windows.Forms.ScrollableControl.WndProc(System.Windows.Forms.Message m)
   at System.Void System.Windows.Forms.ToolStrip.WndProc(System.Windows.Forms.Message m)
   at System.Void System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message m)
   at System.Void System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message m)
   at IntPtr System.Windows.Forms.NativeWindow.Callback(System.IntPtr hWnd, System.Int32 msg, System.IntPtr wparam, System.IntPtr lparam)
   at Boolean System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(System.IntPtr dwComponentID, System.Int32 reason, System.Int32 pvLoopData)
   at System.Void System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(System.Int32 reason, System.Windows.Forms.ApplicationContext context)
   at System.Void System.Windows.Forms.Application+ThreadContext.RunMessageLoop(System.Int32 reason, System.Windows.Forms.ApplicationContext context)
   at static System.Void System.Windows.Forms.Application.Run(System.Windows.Forms.Form mainForm)
   at static System.Void cYo.Projects.ComicRack.Viewer.Program.StartNew(System.String[] args)
   at System.Void cYo.Common.Runtime.SingleInstance.Run(System.String[] args)
   at static Int32 cYo.Projects.ComicRack.Viewer.Program.Main(System.String[] args)
Last Edit: 4 years 8 months ago by T3KN0Gh057.
The topic has been locked.

Re: Comic Vine Scraper 1.0.53-57 4 years 8 months ago #31396

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
Well, it looks to me like when it tries to find the assembly "System.Text.RegularExpressions", for some reason it can't. I can't really explain why not, though, it just works for me... :dry:

But let's not hijack the Comic Vine Scraper thread into a discussion about IronPython. You should post your problem (and that stack trace) here; someone else might know what's wrong.
The topic has been locked.

Re: Comic Vine Scraper 1.0.53-57 4 years 8 months ago #31397

  • T3KN0Gh057
  • T3KN0Gh057's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 407
  • Thank you received: 114
  • Karma: 27
i found out why.

I was using a variable that i had yet to define a value for. it never hit it until i added the reference to the file that i was adding the reference to Regular Expressions Assembly

loads up just fine now
Last Edit: 4 years 8 months ago by T3KN0Gh057.
The topic has been locked.

Re: Comic Vine Scraper 1.0.53-57 4 years 8 months ago #31420

  • T3KN0Gh057
  • T3KN0Gh057's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 407
  • Thank you received: 114
  • Karma: 27
cYo wrote:
I will not change any behavior of ComicRack depending if some 3rd party API is complete/implemented/available.

Why is released/in store date on the catalog field? Because it makes not much sense for eComics. And of course, opposite to the published date, it does not have to be universally the same (countries, cities etc.).

Published date is quite important btw, as it is used to sort story lines.

So why not put the released date into the published date if no published date is available. Maybe an option.

I really don't mean to be a dick, (I really don't) but if the book is released in a different country, on a different release date (and consequently with a different ISBN) then it is a completely different book. One given book will ALWAYS have the same release date (and ISBN)...

Also there are lots of comics without an Editor (self-published) yet that field is allowed permanence in book files. Lots of comics without an Colorist (Black & White comics) also allowed permanence I could go on... so by your reckoning (just because webcomics don't have a release date, release date they belong in catalog and does not deserve permanence... though to make a point I actually use release dates for my web comics that are split into chapters to indicate the day they finish the chapter) then maybe every field should be in the catalog tab :P

but as i always point out it's your program cYo :)
Last Edit: 4 years 8 months ago by T3KN0Gh057.
The topic has been locked.

Re: Comic Vine Scraper 1.0.53-57 4 years 8 months ago #31421

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
T3KN0Gh057 wrote:
600WPMPO wrote:
T3KN0Gh057 wrote:
you'll have that option 600..
No, actually the 'non-purists' should have that option (to scrape the released date into the published date). :P

you never know the API add it so both exist that before he puts out another version :D
In fact, this is exactly what has happenned. :)

I'm now in touch with a developer at ComicVine, and he's been really helpful. He's fixed all of the API bugs that I reported, and he added API access to both the cover_date and the store_date. Unfortunately, he changed some more stuff around--I spent most of today fixing up new problems that have cropped up (e.g. the latest scraper can't find ANY kind of date to scrape anymore).

I also made a suggestion for some small changes to the API that will let me optimize the scraper (so it will run better on slow internet connections). He said he'd implement the changes next week. That's good; it gives me a bit of time to deal with all the other little problems. And maybe sleep, too. :blink:

So at this point it looks like I'll be able to release a fully functional version of the scraper (early?) next week.
Last Edit: 4 years 8 months ago by cbanack.
The topic has been locked.
Time to create page: 0.206 seconds

Who's Online

We have 259 guests and one member online