Welcome, Guest
General discussion about ComicRack
  • Page:
  • 1
  • 2

TOPIC: Comicvine complete volumes info "dump"

Comicvine complete volumes info "dump" 11 months 3 weeks ago #46719

  • Xelloss
  • Xelloss's Avatar
  • Online
  • Platinum Boarder
  • Posts: 464
  • Thank you received: 118
  • Karma: 24
For a project I am working on for a script, I am downloading the complete list of the first number of any comic volume stored in comicvine using the scrapper.

As this is about a 90.000 comics list, it is taking a lot of time to download (a comic at a time) but I have it already at 90% complete

So I thought perhaps someone would like a copy of this data in a single file... (I use it or example for doing checklists of comcs by year, publisher, etc, and in the future I plan to use it for my "wikia sync" script for sync the wikias databases with the comicvine comics offline)

I will upload it as soon as I have it completed, but I was thinking about uploading it in XML format... what do you think? Json would be better?

My original plan was a COMPLETE comic database dump, but it would take it forever to do that... so I am doing first this, and then I will download a complete one for only DC and Marvel Comics (with the volume id list of each one already downloaded)
Last Edit: 11 months 3 weeks ago by Xelloss.
The administrator has disabled public write access.

Comicvine complete volumes info "dump" 11 months 3 weeks ago #46720

  • Targg
  • Targg's Avatar
  • Offline
  • Senior Boarder
  • Posts: 40
  • Thank you received: 6
  • Karma: 1
That sounds like a great idea! I have always wondered about a shared comic database file that we as a community could keep updated...
The administrator has disabled public write access.

Comicvine complete volumes info "dump" 11 months 3 weeks ago #46721

  • Xelloss
  • Xelloss's Avatar
  • Online
  • Platinum Boarder
  • Posts: 464
  • Thank you received: 118
  • Karma: 24
The thing is it already exist and it is called comicvine XD (but it has to be imported to tje CR)
The administrator has disabled public write access.

Comicvine complete volumes info "dump" 11 months 3 weeks ago #46722

  • Targg
  • Targg's Avatar
  • Offline
  • Senior Boarder
  • Posts: 40
  • Thank you received: 6
  • Karma: 1
I understand and use CV via the scraper daily. :)

I meant a local file that could be queried like CV that could help scrape comics without even needing to go beyond a local network.
The administrator has disabled public write access.

Comicvine complete volumes info "dump" 11 months 3 weeks ago #46724

  • Xelloss
  • Xelloss's Avatar
  • Online
  • Platinum Boarder
  • Posts: 464
  • Thank you received: 118
  • Karma: 24
I know I know, I meant that we don't need to do it from scratch, the database is already done there... what we need is a way to have it offline easily :)

I also want to merge info from other databases, as the wikias, which for example have characters divided by Earthes... but that will be a bit more complicated XD
Last Edit: 11 months 3 weeks ago by Xelloss.
The administrator has disabled public write access.

Comicvine complete volumes info "dump" 11 months 3 weeks ago #46729

  • Xelloss
  • Xelloss's Avatar
  • Online
  • Platinum Boarder
  • Posts: 464
  • Thank you received: 118
  • Karma: 24
Well, Here it is the file...

I opted for a csv file, for easy use

The format is: comicvine_volume, Series Name, Volume Year

mega.nz/#!9ItX0YDT

I have been thinking in a project for making a Format database with this and help from whoever want to colaborate... (and a pair script to automate it) but for now this is all

If you want me to add any value to the csv ask for it and I will do it (remember I only have the data of a comic per volume, usually, but not always the first one)

It has volumes in comicvine data till the end of November of 2016 (I wil continue updating it from time to time)
Last Edit: 11 months 3 weeks ago by Xelloss.
The administrator has disabled public write access.

Comicvine complete volumes info "dump" 11 months 3 weeks ago #46732

  • rmagere
  • rmagere's Avatar
  • Offline
  • Gold Boarder
  • Posts: 223
  • Thank you received: 24
  • Karma: 7
Xelloss wrote: It asks for a Decryption Key to download - not sure if that is intentional :)
The administrator has disabled public write access.

Comicvine complete volumes info "dump" 11 months 3 weeks ago #46734

  • Xelloss
  • Xelloss's Avatar
  • Online
  • Platinum Boarder
  • Posts: 464
  • Thank you received: 118
  • Karma: 24
rmagere wrote:
Xelloss wrote: It asks for a Decryption Key to download - not sure if that is intentional :)

It shouldn't! D:

mega.nz/#!9ItX0YDT!PgLvCLJ7IPibOMRYTRg5khNtyPwANY11OFuqQFJwyNc

I never use mega, I read without key and I thought it was to download without key, but it was the link without key XD
The administrator has disabled public write access.

Comicvine complete volumes info "dump" 11 months 3 weeks ago #46739

  • boshuda
  • boshuda's Avatar
  • Offline
  • Gold Boarder
  • Posts: 296
  • Thank you received: 65
  • Karma: 8
I've kind of been working on something like this also. I've been working on pulling their data into a local sqlite database. I got it to pull the publisher data but I haven't formatted it to splat into my database yet. I figured if I didn't have to query their server whenever I added replacement books that would really make me happy. Plus, I dread the day they just block API access altogether. I was also hoping to use it to power an open-source, cross-platform, sql-based ComicRack replacement. I made some strides, but life constantly gets in the way. Especially around the holidays. And I have a huuuuge learning curve on it. Man, I wish cYo would release ComicRack source code.

The other one, comicdb.org (?) has a full mysql dump in an sql script, also.
The administrator has disabled public write access.

Comicvine complete volumes info "dump" 11 months 3 weeks ago #46740

  • Xelloss
  • Xelloss's Avatar
  • Online
  • Platinum Boarder
  • Posts: 464
  • Thank you received: 118
  • Karma: 24
Comicdb has a database dump now?? I remember downloading the entire site in html and data mining all the data from the raw text some years ago XD (then I discovered the wikias has a much better character database and became my new obsesion XD)

Btw, it took me a huge multi proxy strategy to avoid ban downloading the site then XD

The Marvel and DC wikias has the best comic database I have seen (for this two companies), and they do have updated dumps. But 1) it is not parsed, it is pure raw human written text data and it is full of formatting errors, which make it a nightnare to data mine (I didn't give up yet though) and 2) the dump system is somehow brocken for wikias so huge (I have reported the error, anf they are working in a fix now) which make it they only download part of the database
Last Edit: 11 months 3 weeks ago by Xelloss.
The administrator has disabled public write access.
  • Page:
  • 1
  • 2
Time to create page: 0.201 seconds

Who's Online

We have 266 guests and 3 members online