Welcome, Guest
News and Announcements

TOPIC: Comic Vine Scraper 1.0.39-43

Re: Comic Vine Scraper 1.0.39-41 6 years 5 months ago #14554

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1318
  • Thank you received: 503
  • Karma: 181
Perhaps something is going wrong with ComicRack updating its database? Since ComicVineScraper doesn't actually write any files directly (though I am considering trying to change this), the part where your changes are saved is handled by ComicRack.

If you exit ComicRack too quickly, maybe the database isn't finished writing the changes out? Or maybe if your Comics are stored on an external hard drive, you have to give it more time to write the changes to the .cbz files? Normally after I scrape, I wait for a while to make sure everything has had a chance to get saved properly to my external device...I don't know if that's really necessary, but it might explain why I haven't encountered this problem.
The topic has been locked.

Re: Comic Vine Scraper 1.0.39-41 6 years 5 months ago #14570

  • Prandi
  • Prandi's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 9
  • Karma: 0
My comics are in an extenal HD in fact; but I wait for Comicrack to close everithing before shut down my Laptop and this evenience happens even fi i close CR after many hours.


For 600WPMPO: thanks, I have tryed to change the "Proposed Value" label, but it's unselectable and fixed on Yes
The topic has been locked.

Re: Comic Vine Scraper 1.0.39-41 6 years 5 months ago #14571

  • perezmu
  • perezmu's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1114
  • Thank you received: 64
  • Karma: 51
cbanack wrote:
ComicVineScraper doesn't actually write any files directly (though I am considering trying to change this)

Uuuuh?!?!?
The topic has been locked.

Re: Comic Vine Scraper 1.0.39-41 6 years 5 months ago #14572

  • cYo
  • cYo's Avatar
  • Offline
  • Moderator
  • Posts: 3476
  • Thank you received: 675
  • Karma: 181
cbanack wrote:
Perhaps something is going wrong with ComicRack updating its database? Since ComicVineScraper doesn't actually write any files directly (though I am considering trying to change this), the part where your changes are saved is handled by ComicRack.

What would be the benefit?

I advise against writing directly to files. ComicRack handles a load scenarios (cbz,cbt,cb7, alternate streams, fileless entries etc.).
The topic has been locked.

Re: Comic Vine Scraper 1.0.39-41 6 years 5 months ago #14586

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1318
  • Thank you received: 503
  • Karma: 181
Well, I don't have any specific plans right now, but I've been thinking about trying to make the scraper able to run in a "stand-alone" mode (in addition to running as a ComicRack plugin.)

When running as a plugin, CVS will still rely on ComicRack to store data (in the program db or the cbz files, as users prefer)...but obviously, if it is running outside of ComicRack, it will have to write scraped data into files directly.
The topic has been locked.

Re: Comic Vine Scraper 1.0.39-41 6 years 4 months ago #15055

  • Rancoras
  • Rancoras's Avatar
  • Offline
  • Junior Boarder
  • Posts: 37
  • Thank you received: 1
  • Karma: 2
Is the speed of the plug-in based on the speed of the CBV site and database? I find myself waiting quite a while for scrapes to complete. It's still faster than typing the information myself, though. Thanks for a great plug-in!
The topic has been locked.

Re: Comic Vine Scraper 1.0.39-41 6 years 4 months ago #15056

  • Der Internet
  • Der Internet's Avatar
  • Offline
  • Gold Boarder
  • Posts: 176
  • Thank you received: 5
  • Karma: 1
no, the scaper can scrape faster then the speed the of the CBC database! it's called MAGIC!
The topic has been locked.

Re: Comic Vine Scraper 1.0.39-41 6 years 4 months ago #15058

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1318
  • Thank you received: 503
  • Karma: 181
Yeah, the limiting factor is definitely the speed of the ComicVine website/database (or even your internet connection, if it's quite slow). Sometimes when ComicVine is under heavy load or they're doing maintenance on their servers, scraping can be MUCH slower than usual. B)
The topic has been locked.

Re: Comic Vine Scraper 1.0.39-41 6 years 4 months ago #15065

  • Rancoras
  • Rancoras's Avatar
  • Offline
  • Junior Boarder
  • Posts: 37
  • Thank you received: 1
  • Karma: 2
It seemed that each successive issue was taking longer and longer to scrape, so I did a bit of testing. On a fresh start of ComicRack, it seems that I can scrape about 20 issues before the time gets noticeably longer. For the first approximately 20 issues, the time for each issue is about 5 to 7 seconds. The number of issues varies a bit. It hasn't been less than 20, but hasn't been more than 25. Soon after the 20th issue, the time jumps to over 2 minutes per issue and gets worse with each successive issue. I canceled the scrape when it got to over 4 minutes. If I select another group without restarting ComicRack, the time will still be very long for each issue. Oddly, the last issue in the selected group will only take about 2 or 3 seconds to complete. If I restart ComicRack, everything is fast again for around 20 to 25 issues. I have a group running as I type this that is taking a over 3 minutes per issue and again, the last issue completed in about 2 to 4 seconds. The current memory usage for ComicRack is 177,096K. After a fresh restart of ComicRack, the memory usage is 91,688K.

Is there anything else I can do to troubleshoot this problem? Does any of this help you?
The topic has been locked.

Re: Comic Vine Scraper 1.0.39-41 6 years 4 months ago #15066

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1318
  • Thank you received: 503
  • Karma: 181
A couple questions come to mind:

1) is anyone else seeing this problem? (I just scraped 100 comics, and each one took approximately 5-7 seconds, no noticeable slowdown...so I'm not seeing it.)

2) are you able to duplicate this problem i.e. if you try it again, perhaps using some different comics, do you consistently see the same slowing down effect? Is it possible that you just caught the Comic Vine server on a bad day?

There are a number of people on this board who regularly (re)scrape hundreds or even thousands of comics at a time, so I'm guessing that this isn't a universal problem...otherwise I'd have heard about it by now! If it is something that you can reliably reproduce, however, then the next step is to figure out what's different in your situation...
The topic has been locked.
Time to create page: 0.242 seconds

Who's Online

We have 206 guests and 4 members online