Welcome, Guest
News and Announcements

TOPIC: Comic Vine Scraper 1.0.59-64

Re: Comic Vine Scraper 1.0.59-64 4 years 6 months ago #33337

  • mal
  • mal's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 19
  • Thank you received: 2
  • Karma: -1
I have a question. I have a large collection and I use the comic vine scraper. I also have my files in cbr format, and I write the changes to the file (command - update all book files). I also backup my files to a cloud backup service.

I want to be able to re-scrape my files from time to pick up any changes that might occur on comicvine. The problem is the only way I know how to do this is to re-scrape every file. The problem is this takes alot of time, and when I re-scrape them, comicrack marks all the files as changed and needing an update. I believe it then changes all the time stamps on all the records, and I need to re-copy them all to the cloud. This is a waste of bandwidth and time as the majority of the records don't change.

So, am I doing the correct thing by re-scaping using comicvine to find changes ? and if so is there a way to have the scraper/comicrack only flag the ones that have actually been changed/updated ?
The topic has been locked.

Re: Comic Vine Scraper 1.0.59-64 4 years 6 months ago #33338

  • cYo
  • cYo's Avatar
  • Offline
  • Moderator
  • Posts: 3476
  • Thank you received: 676
  • Karma: 181
If you make sure to backup your database file there is actually no need to also write the changes to the files.
The topic has been locked.

Re: Comic Vine Scraper 1.0.59-64 4 years 6 months ago #33347

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
mal wrote:
So, am I doing the correct thing by re-scaping using comicvine to find changes ? and if so is there a way to have the scraper/comicrack only flag the ones that have actually been changed/updated ?
Well, if you don't write out the metadata to your comic files, your comic files will never get changed, so there'll be no extra uploads to your online backup. You must make sure your ComicRack database is being backed up, though, because that's really where all the metadata is being stored.

On the other hand, having the metadata written out to the files themselves could save you a lot of headache if your ComicRack database is ever corrupted or lost, or you move to another computer and you have trouble getting the database moved over (especially if you comic files move to a different location). Having the metadata in the files guarantees that you can always re-add those files to a fresh install of ComicRack, and re-scraping them will be easy and automatic.


The reason that ALL your comics are being changed every time you scrape is because the "Notes" field for the comic book is being updated with the most recent scrape date. This causes the file to be updated every time you scrape. If you turn off the "Save that choice in 'Notes'" checkbox in the Scraper settings, then your comic files will only change when there actually is a real change in the scraped metadata. Be sure to leave "Save that choice in 'Tags'" turned on, though, or rescraping will stop working...

Another thing to keep in mind: your online backup probably uses "incremental backup", meaning that only the changed portion of each comic file actually gets uploaded. So the bandwidth cost of rescraping might not be as bad as you think.
The topic has been locked.

Re: Comic Vine Scraper 1.0.59-64 4 years 6 months ago #33350

  • forkicks
  • forkicks's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 871
  • Thank you received: 109
  • Karma: 37
cbanack wrote:
Another thing to keep in mind: your online backup probably uses "incremental backup", meaning that only the changed portion of each comic file actually gets uploaded. So the bandwidth cost of rescraping might not be as bad as you think.

Compressed files (like the CBRs and CBZs) are not very fond of incremental changes. I'm not sure if this affects all compression levels, but i'm pretty sure a change to a single bit in a file will cause the whole thing to be recompressed and hence be completely different, not just a portion. Perphaps the compression level 0 (ie, store), won't do this, but i'm pretty sure all the others do. It changes the dictionaries used in the compression schemes, and that forces a rebuild of all the blocks.

For our files, since most of them are simply jpges which won't compress to begin with, simply using compression level 0 is feasible and won't sacrifice a significant amount of space, but for most other purposes it defeats the whole idea of compression.

fK
The topic has been locked.

Re: Comic Vine Scraper 1.0.59-64 4 years 6 months ago #33353

  • mal
  • mal's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 19
  • Thank you received: 2
  • Karma: -1
Thanks for the input. I'll try not updating the actual files.
The topic has been locked.

Re: Comic Vine Scraper 1.0.59-64 4 years 6 months ago #33354

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
forkicks wrote:
Compressed files (like the CBRs and CBZs) are not very fond of incremental changes. I'm not sure if this affects all compression levels, but i'm pretty sure a change to a single bit in a file will cause the whole thing to be recompressed and hence be completely different, not just a portion.
Good point, I never thought of that.

I'm actually thinking about removing the date stamp part from the details that are added to the "Notes" field, so that it is possible to re-scrape a comic without creating any change in the file if no metadata has changed...
Last Edit: 4 years 6 months ago by cbanack.
The topic has been locked.

Re: Comic Vine Scraper 1.0.59-64 4 years 6 months ago #33355

  • perezmu
  • perezmu's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1114
  • Thank you received: 64
  • Karma: 51
cbanack wrote:
I'm actually thinking about removing the date stamp part from the details that are added to the "Notes" field, so that it is possible to re-scrape a comic without creating any change in the file if no metadata has changed...

+1 for this...

Now that I am preparing to massively scan tons of books I had to go and peek at the code... how is .65 coming along ;) ? :blush: :blush: :blush:
The topic has been locked.

Re: Comic Vine Scraper 1.0.59-64 4 years 6 months ago #33356

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1328
  • Thank you received: 508
  • Karma: 182
perezmu wrote:
cbanack wrote:
I'm actually thinking about removing the date stamp part from the details that are added to the "Notes" field, so that it is possible to re-scrape a comic without creating any change in the file if no metadata has changed...

+1 for this...

Now that I am preparing to massively scan tons of books I had to go and peek at the code... how is .65 coming along ;) ? :blush: :blush: :blush:
Funny you should ask, I was just working on it this afternoon. :) All the hard stuff is done, and I'm just fixing a few smaller problems now. I'm motivated to get it finished soon, because once it's done, I'm taking the rest of the summer off from coding (except for bug fixes, of course.)
The topic has been locked.

Re: Comic Vine Scraper 1.0.59-64 4 years 6 months ago #33357

  • perezmu
  • perezmu's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1114
  • Thank you received: 64
  • Karma: 51
:woohoo: :woohoo: :woohoo: :woohoo:
The topic has been locked.

Re: Comic Vine Scraper 1.0.59-64 4 years 6 months ago #33359

  • Madmatx
  • Madmatx's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 457
  • Thank you received: 63
  • Karma: 19
Pre-congrats on the vacation and, yeah, no need for date stamp in notes. I mean, does anyone care when they last scanned a comic as long as the can see when they added it?
The topic has been locked.
Time to create page: 0.227 seconds

Who's Online

We have 340 guests and 2 members online