Welcome, Guest
Python Scripts for ComicRack

TOPIC: Scrape from Text - A script to get comic data from .CSV (NEW Version !!! 1.6)

Scrape from Text - A script to get comic data from .CSV (NEW Version !!! 1.6) 6 years 10 months ago #11472

  • mizio66
  • mizio66's Avatar
  • Offline
  • Platinum Boarder
  • Started reading comics at 4... and still counting!
  • Posts: 451
  • Thank you received: 143
  • Karma: 67
My second script. Also in this case I made because I needed it: am I egoistic? This one can *surely* be improved, but already as it is it helped me a lot.

Imagine you have a number of comics you can't scrape as they are nowhere in ComicVine or in BDGest. But you have (or you extract) the data from Wiki, website, etc... Well, with this script, once you saved the data to a .CSV file, you can use the data to fill the fields of your favourite program !

i.e. I used for my ****** comics (hidden on purpose): went to the site, with Excel I pulled all the data (dates, inkers, writers, teams...), saved them in a CSV, run the script and in 5" I had my e-collection of ****** perfectly scraped !

I suggest the use of this to expert CR users: no time/will to write detailed WI, but you can ask questions here. The logic/concept is easy to understand...
To start, save the CSV with the same name as the series into C:\ (I know, it is all Windows related... sorry) and run the script. You can anyhow change the folder for the data files. Define the sequence of the .CSV fields,like series;number;title; etc. Then click OK and smile...

Some known limitation:
- it uses the TITLE or the Number of the comic to retrieve the data...
- it overwrites the fields, no question asked.. be careful

Again,suggestion or enhancements are welcome !

mizio66
Last Edit: 4 years 1 month ago by mizio66.
The administrator has disabled public write access.
The following user(s) said Thank You: perezmu, Alan Scott, lg5

Re: Scrape from Text - A script to get comic data from .CSV 6 years 10 months ago #11533

  • lg5
  • lg5's Avatar
  • Offline
  • Junior Boarder
  • Posts: 35
  • Karma: -1
I've loved the fact that you've developed this script but I have some questions:

1) I need to put the .CSV file in the root dir of C: even if my comics files are in another disk?

2) I haven tried yet to run this script but it supports all the description fields available on ComicRack or only some?

3) In wich place that I need to tell the script my sequence of fields?

Many thanks for developing this script!
The administrator has disabled public write access.

Re: Scrape from Text - A script to get comic data from .CSV 6 years 10 months ago #11534

  • mizio66
  • mizio66's Avatar
  • Offline
  • Platinum Boarder
  • Started reading comics at 4... and still counting!
  • Posts: 451
  • Thank you received: 143
  • Karma: 67
lg5 wrote:
I've loved the fact that you've developed this script but I have some questions:

1) I need to put the .CSV file in the root dir of C: even if my comics files are in another disk?

The CSV can be placed anywhere you want. You will select the folder at the moment you scrape, via the browse button. The script will remember the position for each series.
lg5 wrote:
2) I haven tried yet to run this script but it supports all the description fields available on ComicRack or only some?
They should be all... at least those I found ! if you run the script you will see the list (you can run and not apply the changes. Before you can apply changes you have to define some parameters in the screen, so no danger!)
lg5 wrote:
3) In wich place that I need to tell the script my sequence of fields?
In the script parameter window that will appear as you run it. You will see how it is easy: just click the buttons and the sequence will be built...

Ciao,

M
The administrator has disabled public write access.

Re: Scrape from Text - A script to get comic data from .CSV 6 years 10 months ago #11557

  • Alan Scott
  • Alan Scott's Avatar
  • Offline
  • Gold Boarder
  • Posts: 264
  • Thank you received: 20
  • Karma: 10
I've got the gist of what this does, but I'm not certain how to properly save data to a .csv to use it. Is there anyone who would be kind enough to give a quick lesson using a site that allows scraping, like Comicvine or, even better, Grand Comic Book Database to help me and others like me through this? Thanks a ton in advance.
... The failure to appreciate... is perfectly understandable, because the readership never evaluates old material in the context of the cultural climate in which it was created, or the state of the art at the time it was created.
Marty Pasko
The administrator has disabled public write access.

Re: Scrape from Text - A script to get comic data from .CSV 6 years 10 months ago #11560

  • mizio66
  • mizio66's Avatar
  • Offline
  • Platinum Boarder
  • Started reading comics at 4... and still counting!
  • Posts: 451
  • Thank you received: 143
  • Karma: 67
Alan Scott wrote:
I've got the gist of what this does, but I'm not certain how to properly save data to a .csv to use it. Is there anyone who would be kind enough to give a quick lesson using a site that allows scraping, like Comicvine or, even better, Grand Comic Book Database to help me and others like me through this? Thanks a ton in advance.

Alan, the creation of the CSV file itself is normally not a big kerfuffle :cheer: although there could be some problems due to the regional setting of Excel that we cannot discuss here as it would take a book and half. I suggest to create manually a first CSV entering some data manually.
Do not forget a CSV is just another geeky way to calla text file, with the addition of using separators between one value and the other.
Imagine a CSV like a database in a text file... you can start your Notepad or whateever editor you like and start typing some values for a series you feel like experimenting with...
You will have, if you are consistent,something like:
Seriesname;1;1990;11;Title1;writer;penciller
Seriesname;2;1990;12;Title2;writer;penciller
Seriesname;3;1991;1;Title3;writer;penciller
And so on,,, now save it as seriesname.csv and most of the work is done.

Run the script after having selected 1 or 2 comics of the same series (the series name and the file name must correspond), select the folder where you save the file, define the structure of the file using the buttons (in this case, let's imagine you click series then number then year then month then title then writer then penciller. If you i.e. don't want writer, click ignore instead).
Click OK and the data will be scraped and written. Please note that the TITLE will be used as key, so if the title has no correspondence, nothing will be written.

Back to the CSV, i created some using Excel and importing the data as tables from the web... I used some VBA to go around some small issues there, but for this... I can't help more than a good VBA book...
Just to help, let me know a series in GCDB and I will see if I can get that in a CSV format... not guaranteed, but I will give it a try.

ciao,

M
The administrator has disabled public write access.

Re: Scrape from Text - A script to get comic data from .CSV 6 years 10 months ago #11578

  • Alan Scott
  • Alan Scott's Avatar
  • Offline
  • Gold Boarder
  • Posts: 264
  • Thank you received: 20
  • Karma: 10
Thanks for the help. It's not creating the .csv so much as it is that I'm not certain what data and in what fashion I should save to that. I'm completely unskilled at this, hence my issues :blush: . Also, I'm using Open Office, though that works as well as Excel, just something to keep in mind. If I can see one properly saved .csv, I would know then what to do.

Here's a series from GCBD: The Man Of Steel. Thanks for giving this a shot. I thank you very much for all your help and your patience.
... The failure to appreciate... is perfectly understandable, because the readership never evaluates old material in the context of the cultural climate in which it was created, or the state of the art at the time it was created.
Marty Pasko
The administrator has disabled public write access.

Re: Scrape from Text - A script to get comic data from .CSV 6 years 10 months ago #11596

  • mizio66
  • mizio66's Avatar
  • Offline
  • Platinum Boarder
  • Started reading comics at 4... and still counting!
  • Posts: 451
  • Thank you received: 143
  • Karma: 67
Ok,will take a look. Getting the data from tables in the GCD is quite tough... format is not really respected 100%..

I will give a try for a scrape script, it will be a good exercise for me. Hopefully, you're not in a hurry though!!!!!!!!

B)

ciao,

M
Last Edit: 6 years 10 months ago by mizio66.
The administrator has disabled public write access.

Re: Scrape from Text - A script to get comic data from .CSV 6 years 10 months ago #11796

  • mizio66
  • mizio66's Avatar
  • Offline
  • Platinum Boarder
  • Started reading comics at 4... and still counting!
  • Posts: 451
  • Thank you received: 143
  • Karma: 67
Alan Scott wrote:
Here's a series from GCBD: The Man Of Steel. Thanks for giving this a shot. I thank you very much for all your help and your patience.

Alan, sent you a PM for this scraping script, check incoming mail...
The administrator has disabled public write access.

Re: Scrape from Text - A script to get comic data from .CSV 6 years 8 months ago #12587

  • mizio66
  • mizio66's Avatar
  • Offline
  • Platinum Boarder
  • Started reading comics at 4... and still counting!
  • Posts: 451
  • Thank you received: 143
  • Karma: 67
:laugh: *** A new release V1.2 is out *** :silly:

Changes:
- Added possibility to scrape by Number or by Title
- Improved the sequence management: remove, cycle elements...
- Improved/Fixed general logic
- Added more error management
- Improved debug info when in debug mode (need to change the flag in the code)

There is a caveat though... it will reset the configurations saved, so better to first save them and then install. ScrapeTxt.dat is in the script folder. Save it under another name and after installing the new release, edit them with a text editor: you'll notice a 4th line added in the new Default configuration. Just add the same value below those you have and save it with the proper name.
Sorry for this, better doing it manual...

Anyhow, the new release in the first post. Please redownload, the wrong version was attached...

Enjoy,

M
Last Edit: 6 years 8 months ago by mizio66.
The administrator has disabled public write access.
The following user(s) said Thank You: MoDy

Re: Scrape from Text - A script to get comic data from .CSV (NEW Version 1.3) 6 years 8 months ago #12648

  • mizio66
  • mizio66's Avatar
  • Offline
  • Platinum Boarder
  • Started reading comics at 4... and still counting!
  • Posts: 451
  • Thank you received: 143
  • Karma: 67
:laugh: A new release V1.3 is out *** :silly:

Changes: some bug fixed - improved error management.

Downlaod from firts post.

enjoy,

M
Last Edit: 6 years 8 months ago by mizio66.
The administrator has disabled public write access.
Time to create page: 0.227 seconds

Who's Online

We have 225 guests and 4 members online