Welcome, Guest
Python Scripts for ComicRack

TOPIC: [Request]Myanimelist Scraper for manga

Re: [Request]Myanimelist Scraper for manga 6 years 1 week ago #18718

  • quidam
  • quidam's Avatar
  • Offline
  • Platinum Boarder
  • not for nom
  • Posts: 448
  • Thank you received: 23
  • Karma: 30
Good to see something finally going on with this. I'm waiting for manga scrapper few years already (since 0.9.27 when CR got scripting available). I hope that in few weeks/months we'll enjoy decent scrapper.

I strongly suggest working on mangaupdates.com scrapper than MAL because of much more bigger database. Also, MU provides some information about latest released chapters that could come handy in later versions of scrapper when basics are done.

As for search engine - if that's so complicated to delay you let's start with semi-manual way: let user input manga ID/link or read it from 'Web' metadata field (I usually store there links to MU webpage) and just scrap info about the series. That should be the first thing to get done. Automatic searching and matching series to search result - make it the next step. In meantime above should be more than enough.

If it's not easy to recognize and assign chapter to the volume - just skip scrapping this for now or let user input it manually.
The administrator has disabled public write access.
The following user(s) said Thank You: Joentjuh

Re: [Request]Myanimelist Scraper for manga 6 years 1 week ago #18740

  • rdu90
  • rdu90's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 16
  • Thank you received: 1
  • Karma: 0
Maybe the volume scrape should be a separate thing? Scrape for information, THEN, scrape to assign volume images from the volume image db...
Last Edit: 6 years 1 week ago by rdu90. Reason: I'm a BIG idiot
The administrator has disabled public write access.

Re: [Request]Myanimelist Scraper for manga 6 years 1 week ago #18954

  • rdu90
  • rdu90's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 16
  • Thank you received: 1
  • Karma: 0
Hey everyone

I did a little work. I'm stuck at the point where I talk REST to myanimelist

But I expect to work that out soon

I just wanted to let everyone know I haven't disappeared yet

and I wanted to keep myself motivated :blush:

aaron
The administrator has disabled public write access.

Re: [Request]Myanimelist Scraper for manga 6 years 1 week ago #18964

  • O.Ren
  • O.Ren's Avatar
  • Offline
  • Expert Boarder
  • Ahoy !
  • Posts: 146
  • Thank you received: 32
  • Karma: 14
Thank you so much for doing this and for updating us ! I can tell there are a lots of mangas readers lurking in the forum by the number of views these manga related threads get :) So even if you don;t always see them, people who will use this script are cheering :laugh:

How about we post a screenshot of how good looking our mangas collection looks after applying your script to it ? Would that make you feel good ? :)

Keep it up, and don't hesitate to ask for help, I'll be more than happy to test stuff if you want, and we can also help with the design of the GUI later if needed.
ComicRackers, ASSEMBLE !
Join us on IRC, don't be shy :) Server:irc.rizon.net Channel:#comicrack
More info here
The administrator has disabled public write access.

Re: [Request]Myanimelist Scraper for manga 6 years 1 week ago #18969

  • arathon
  • arathon's Avatar
  • Offline
  • Junior Boarder
  • Posts: 27
  • Thank you received: 1
  • Karma: 1
thanks for the update!!!!For all manga lover you're like our God, if you need a tester contact me....


P.S for the manga cover database, can i upload covers of series that are not in the site? 'cause i have found almost all the covers of my collection about 3500 volumes and i wanted to contribute (have all covers of Shijou Saikyou no Deshi Kenichi for example). there is an extra part of the cover that you bent inside the book, hope you understand sorry but i'm not english so don't speak so well, do i upload it or not?
The administrator has disabled public write access.

Re: [Request]Myanimelist Scraper for manga 6 years 1 week ago #18970

  • Joentjuh
  • Joentjuh's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 17
  • Thank you received: 1
  • Karma: 1
arathon wrote:
thanks for the update!!!!For all manga lover you're like our God, if you need a tester contact me....


P.S for the manga cover database, can i upload covers of series that are not in the site? 'cause i have found almost all the covers of my collection about 3500 volumes and i wanted to contribute (have all covers of Shijou Saikyou no Deshi Kenichi for example). there is an extra part of the cover that you bent inside the book, hope you understand sorry but i'm not english so don't speak so well, do i upload it or not?

Any contribution is more than welcome, I've underestimated the amount of work a bit when I started this, but luckily people are willing to contribute (which leaves me with more time to improve the website and add features).
Use the contact form (or forum).
The administrator has disabled public write access.

Re: [Request]Myanimelist Scraper for manga 6 years 1 week ago #18982

  • rdu90
  • rdu90's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 16
  • Thank you received: 1
  • Karma: 0
heh heh, yes, when this is over, screenshots would be cool to see.

I got the pointer I needed from the developers and I have some reading to do. So when I can get back to it, I should be in good shape.

As for UI, it is going to be a script, so it will just update your comics if it can find data. If we're lucky, if I get the scrape/apply portion correct, then maybe I can integrate it into something like ComicVine that already has an excellent UI otherwise version 2!!!

anyway, thanks for the encouragement everyone
The administrator has disabled public write access.

Re: [Request]Myanimelist Scraper for manga 6 years 3 days ago #19175

If you need someone to test, I've got a huge stack of manga to test out ;)
The administrator has disabled public write access.

Re: [Request]Myanimelist Scraper for manga 6 years 2 days ago #19203

  • Joentjuh
  • Joentjuh's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 17
  • Thank you received: 1
  • Karma: 1
I've been tinkering a bit the past week, and added some nice new features. I'm now able to offer the following for most of the volumes on my site (not all is displayed by default):
Manga:
- MegaUpdates ID
- MU title
- MyAnimeList ID
- MAL title
- Year (of first release)
- Type (manga/manwha/manhua)
- Completed (y/n)
- Nr. of volumes (if original print had 10 volumes and reprint has 4, 10 is returned)
- Alternative titles from both MAL and MU (for better searching)
Volume:
- Front cover
- Spine cover
- Back cover
- Full cover (full dustcover as included in the RAWs, uncropped and unsliced)
- Amazon.co.jp info, Kinokuniya as fallback:
- Volume title
- ISBN (ean-10 and ean-13)
- Nr. of pages
- Book dimensions
- Release date (year + month)
- Publisher (in Kanji, English translation available for some)
- Author (also in Kanji)
- Thumbnail path
- Direct link to item
I might have more information stored in the cache, but I don't plan on putting everything in the database. That's what the MU/MAL links are for.

At the top of my list is now building the API, I hope to have a simple form available soon with which you can request data in JSON format.

I.e. separate input fields for: MU id, MAL id, and Title.
Result is displayed in array format (in case of partial title matches), with a limit of 10? results.
Volumes are going to be a bit trickier. Would it be better to include all volumes in the manga search... which would add to the payload considerably, or add separate fields for volume lookup (via ISBN or Vol. nr)...
The administrator has disabled public write access.

Re: [Request]Myanimelist Scraper for manga 6 years 1 day ago #19220

  • rdu90
  • rdu90's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 16
  • Thank you received: 1
  • Karma: 0
That sounds really pretty great.

I've finally stopped procrastinating and am putting my time in

fingers crossed
The administrator has disabled public write access.
Time to create page: 0.400 seconds

Who's Online

We have 244 guests and 3 members online