Welcome, Guest
Python Scripts for ComicRack

TOPIC: [Request]Myanimelist Scraper for manga

Re: [Request]Myanimelist Scraper for manga 5 years 4 months ago #23454

  • rdu90
  • rdu90's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 16
  • Thank you received: 1
  • Karma: 0
there doesn't seem to be

when i looked it seemed like you could find one site that had the information you would want for a couple manga, but not everything (for example jump), but if you wanted something else, you'd have to go to another site

and no api, just a web page to scrape

so, in the end, it would be a lot of scrapers

that's why I think it needs to be a community effort
The administrator has disabled public write access.

Re: [Request]Myanimelist Scraper for manga 5 years 4 months ago #23457

  • arathon
  • arathon's Avatar
  • Offline
  • Junior Boarder
  • Posts: 27
  • Thank you received: 1
  • Karma: 1
i thought that this project was dead. the only site that have almost all the information si mangaupdate (excluding doujinshi in which) but has no api even though the problem is searching the series' title.
There is another one Myanimelist that has minor database (not existent for doujinshi) but has unofficial api, i've linked them before maybe even in the first post.

The main problem for me is searchin the series in whatever database 'cause the variation of the name depending if it is in japanese or in latin letters or translated.

P.S i had some time ago an idea: first we add all of our series manually on Myanimelist then with a scaper for MAL we scan our CR library and search the series directly from our MANGA LIST in our account. i don't know if you had understood but with this it decrease the processing behind the scraping and i think even the errors.
The administrator has disabled public write access.

Re: [Request]Myanimelist Scraper for manga 5 years 4 months ago #23458

  • Joentjuh
  • Joentjuh's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 17
  • Thank you received: 1
  • Karma: 1
If it's search you want, I might be able to simplify things a bit.
To make things a bit easier for me, I've recently put together a search engine that scours known sources for a specific title. Currently it looks on MangaUpdates (every title that is encountered is cached in a local database, even the irrelevant ones) and my own MCD database, which contains a collection of titles from MyAnimeList, MangaUpdates, AnimeNewsNetwork, Kinokyuniya, and Amazon (hopefully soon also Anime-Planet).

It's a preliminary prototype, so it's not really ready for public use (sometimes titles are not found for some odd reason I've yet to identitify). Only results with a match on MangaUpdates are returned (titles aren't interlinked, a few database changes are needed for that).

Search: link

Fairly simple actually, it's just a big lookup-table that grows with each consecutive search.
Note that it's a development version, so not guaranteed to work at all times.

---

I agree with rdu90, ultimately this plug-in could really use some sort of collaborative database, but that's something for the future.
What's needed (the way I see it) are the following things:
1. Search
- try to identify the series by either title or ISBN
2. Scrape
- should be done via either a local or a third party cache, especially when scraping multiple pages (i.e. author and company)
- a third party cache would also simply the plug-in quite a bit, having to only contact one source with merged metadata from multiple sources, instead of having to do this itself. (in the future, maybe with the option to push metadata back to the database?)
3. Covers
- give the user a list of all found volumes (with the guessed one as focus)
4. Verify
- overview of scraped metadata
5. Mutate
- Add to/Update ComicRack database


P.S.
AAARGH @ THIS FORUM, this is the second time I've lost a post because of the stupid session expiration.
Last Edit: 5 years 4 months ago by Joentjuh.
The administrator has disabled public write access.

Re: [Request]Myanimelist Scraper for manga 4 years 4 months ago #34427

  • arathon
  • arathon's Avatar
  • Offline
  • Junior Boarder
  • Posts: 27
  • Thank you received: 1
  • Karma: 1
any news?? i'm really waiting for someone who can make the magical scraper, unfortunatly i don't know how to code but i can beta test.
The administrator has disabled public write access.

Re: [Request]Myanimelist Scraper for manga 4 years 2 months ago #35596

  • lg5
  • lg5's Avatar
  • Offline
  • Junior Boarder
  • Posts: 35
  • Karma: -1
The administrator has disabled public write access.
Time to create page: 0.212 seconds

Who's Online

We have 178 guests and 5 members online