Welcome, Guest
News and Announcements

TOPIC: Comic Vine Scraper 1.0.39-43

Re: Comic Vine Scraper 1.0.39-41 6 years 5 months ago #15067

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1327
  • Thank you received: 508
  • Karma: 182
Wait...here's an interesting thought...are you running ComicRack with the debug console window open? That would cause symptoms like you are talking about--Comic Vine Scraper generates a lot of output to that debug console, and for whatever reason, ComicRack's console gets very, very inefficient once a certain amount of output has been sent to it. But if you don't have the console open (i.e. almost everyone) then it's no problem.

(And if you don't know what the console window is, then don't worry, you don't have it open.)
Last Edit: 6 years 5 months ago by cbanack.
The topic has been locked.

Re: Comic Vine Scraper 1.0.39-41 6 years 5 months ago #15068

  • Rancoras
  • Rancoras's Avatar
  • Offline
  • Junior Boarder
  • Posts: 37
  • Thank you received: 1
  • Karma: 2
Nope, no debug console. It was reproducible through 4 different restarts of ComicRack last night. Started fresh just now, same thing: I started a scrape of 30 comics. It did 25 comics at 5 to 7 seconds each, the 26th one jumped to just over 2 minutes and got progressively longer. The 29th one was 3 minutes and the 30th was about 2 seconds. This was a batch of comics with no metadata, that had never been scraped before. This was the same for all previous tests.

This led me to check comics that had been scraped before. I re-scraped the same 30 comics and they all took the normal 5 to 7 seconds. I, then, re-scraped the 1994 run of Aquaman, 77 issues. The initial search took a while, and once I selected the series, every issue was taking several minutes. I canceled that batch after the 3rd issue. I restarted ComicRack and started the re-scrape of Aquaman again. This time 50 issues scraped at 5 to 7 seconds. The 51st issue was about a minute and got longer from there. I canceled that batch after the 53rd issue. I'm not sure what to make of this.

My system is Windows 7 Home Premium 64bit. Intel Core i7-2600 @ 3.4 GHz, 8 GB RAM. The comics are stored on a hard drive attached to my airport extreme. I am connected to the airport extreme with a 1 Gb wired connection. I have to leave the house right now, but when I return, I will attach the comic hard drive directly to my machine and see if that has any effect.
The topic has been locked.

Re: Comic Vine Scraper 1.0.39-41 6 years 5 months ago #15069

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1327
  • Thank you received: 508
  • Karma: 182
Hmm, so it seems like when you scrape the 25th previously unscraped comic, things slow down. And this effect is reproducible. Is it 25 different comic book series (i.e. does it search 25 times, and show you the Series chooser window 25 times?) Or are some of the comics from the same series? Is it *exactly* 25 each time? More generally, it would interesting to know how many times you've seen the search progress bar, series chooser window, and issue chooser window before the problem starts to manifest.

You're using ComicRack version 0.9.142 (latest) and CVS 1.0.42 (latest) I presume?
I'm not sure what to make of this.
Does a similar amount of RAM get used when you scrape, say 30 previous-unscraped comics (i.e. you perform the bug) vs. when you rescrape 30 already-scraped comics (an the bug doesn't happen?) Are you able to monitor your CPU activity (or even just the behaviour of other apps)? I'm trying to rule out the idea that CVS or ComicRack is slowly eating up all your RAM or CPU somehow. I suspect this problem is either that, or something network related.

You haven't said anything about actual errors occurring, so I'm assuming that the comics are all still being scraped properly--it's just taking way too long for some reason.

I have to go help a friend move, but once I've heard back from you again, I'll try to copy what you are doing as closely as possible, and see if I can duplicate the problem. We appear to have exactly the same computer and OS, so at least we can rule out hardware (except maybe network or your airport...though my comics are on a drobo, so even that's pretty similar.)
The topic has been locked.

Re: Comic Vine Scraper 1.0.39-41 6 years 5 months ago #15070

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1327
  • Thank you received: 508
  • Karma: 182
Also, is there anyone else out there who's encountered a slowdown problem like what Rancoras is seeing? i.e. the scraper gets slower and slower as you scrape comics (more than 25) and the only way to get it running quickly again is to restart ComicRack?
The topic has been locked.

Re: Comic Vine Scraper 1.0.39-41 6 years 5 months ago #15074

  • Rancoras
  • Rancoras's Avatar
  • Offline
  • Junior Boarder
  • Posts: 37
  • Thank you received: 1
  • Karma: 2
The scraping process is barely making a dent in the CPU usage (it hasn't gone above 3%). Memory usage creeps up a little during scraping, but I haven't seen it go above 178,000K. The memory usage for a re-scraped and unscraped batch is the same. I'm doing a "fresh" scrape of 30 comics now and the slowness started on #24. So it doesn't seem to be a specific number of comics. It's always been at least 20 and no more than 25. It's very strange to me that the last comic is always about 2 seconds.

It seems that removing the network as a potential factor may be more trouble than it's worth. I'm still a very new CR user and I'm not sure how to "re-map" the comic files with the change in location so I'll leave the hard drive attached to the airport. Since you are using a drobo, we are very similar anyway. Maybe we can get together in gtalk or something to try some further testing and see what we can find. I'll be around the house checking in from time to time for the rest of the day.

Edit: I just started a batch of Batman after a fresh restart of CR. The first one went fine at 5 to 7 seconds. The slowness started with the second one this time. Could it have something to do with the file names? The only other thing I saw different from these files was that the scanner tag was the first image in the archive instead of the cover. That wasn't the case with my earlier tests and there isn't really anything odd about the way these batman files are named.
Last Edit: 6 years 5 months ago by Rancoras.
The topic has been locked.

Re: Comic Vine Scraper 1.0.39-41 6 years 5 months ago #15075

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1327
  • Thank you received: 508
  • Karma: 182
Well, the filename should only affect how quickly the search portion of the scrape is, and in a very predictable way (nothing like what you're seeing here.)

One thing you could try is to make a copy of your comics to a local drive, and then add the local directory to your comicrack library (in preferences) and see if the newly added (duplicate) comics scrape any differently (then you just remove that local directory from your library afterwards to clean up.)

I'm wondering if it could have something to do with the airport--do those things do some kind of backup/versioning when files get changed?
The topic has been locked.

Re: Comic Vine Scraper 1.0.39-41 6 years 5 months ago #15077

  • Rancoras
  • Rancoras's Avatar
  • Offline
  • Junior Boarder
  • Posts: 37
  • Thank you received: 1
  • Karma: 2
We have success! I copied 274 issues of Hellblazer to the local machine and added them to the library. They had never been scraped before. CVS scraped all 274 issues in less than 9 minutes. It has to be something to do with the drive being on the airport. I know how to proceed now. Thanks so much for your help cbanack! I appreciate it very much.

Edit: Where is the scraped metadata stored? If not in the archive, then is there a way to get it there? It would be nice not to have to re-scrape everything if I ever have to re-import the library for whatever reason.
Last Edit: 6 years 5 months ago by Rancoras.
The topic has been locked.

Re: Comic Vine Scraper 1.0.39-41 6 years 5 months ago #15124

  • Taro
  • Taro's Avatar
  • Offline
  • Senior Boarder
  • Posts: 60
  • Thank you received: 2
  • Karma: 2
Sniff, I wanted to download the latest version but it seems the Google code page is down. Is anyone else experiencing the same problem or is it just me? :pinch:
The topic has been locked.

Re: Comic Vine Scraper 1.0.39-41 6 years 5 months ago #15128

  • 600WPMPO
  • 600WPMPO's Avatar
  • Offline
  • Moderator
  • Posts: 3788
  • Thank you received: 557
  • Karma: 232
Google code page is opening fine for me..
Now Playing: The ComicRack Manual (Online)

See my new comics & gadgets on: Tumblr!
The topic has been locked.

Re: Comic Vine Scraper 1.0.39-41 6 years 5 months ago #15129

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1327
  • Thank you received: 508
  • Karma: 182
Yup, it's working fine for me. Are you sure you're logged in properly? Might be worth restarting your browser.
The topic has been locked.
Time to create page: 0.219 seconds

Who's Online

We have 392 guests and 7 members online