Welcome, Guest
News and Announcements

TOPIC: Comic Vine Scraper

Comic Vine Scraper 6 months 2 weeks ago #48986

  • krandor
  • krandor's Avatar
  • Offline
  • Gold Boarder
  • Posts: 310
  • Thank you received: 32
  • Karma: 5
If you are still seeing issues let me know and I'll collect them and give them to the guys on GB. I don't mind being a liason here. Hopefully though they got things fixed.
The administrator has disabled public write access.
The following user(s) said Thank You: cbanack

Comic Vine Scraper 6 months 2 weeks ago #48987

  • krandor
  • krandor's Avatar
  • Offline
  • Gold Boarder
  • Posts: 310
  • Thank you received: 32
  • Karma: 5
Scrapped a few hundred tonight and looks much better. In anything, search is returning MORE results then needed and slowing things down. I had a scrape for all-star winners that came back with over 4000 matches. Too many matches is on them IMO since the script does a great job of picking the right selection. I'm liking what I'm seeing tonight,
The administrator has disabled public write access.

Comic Vine Scraper 6 months 2 weeks ago #48988

  • garcunning
  • garcunning's Avatar
  • Offline
  • Fresh Boarder
  • Posts: 7
  • Karma: 0
It's too slow to really be useful if you have a lot of comics to scrape at the moment. It needs to be refined.

"Ice Cream Man" a pretty unique name that should be quick enough to find if you search for the whole thing came in with over 2000 results because it searches for each individual word and gives you those results too.

"Deadpool vs. Old Man Logan" returned over 4000 results but again should be quick to find in an exact search.

There needs to be an option to only enable single word searches if it can't find anything doing an exact search.

Don't know if this is a script problem or a comicvine problem.
The administrator has disabled public write access.

Comic Vine Scraper 6 months 2 weeks ago #48989

  • kino13
  • kino13's Avatar
  • Offline
  • Senior Boarder
  • Posts: 58
  • Thank you received: 6
  • Karma: 0
Same as people is commenting already, there are so many results that the search becomes useless.

Is there any option or some modification we can do on the scrapper to accept only the first 200 results?
with no power comes no responsibility. except that wasn't true
The administrator has disabled public write access.

Comic Vine Scraper 6 months 2 weeks ago #48990

  • krandor
  • krandor's Avatar
  • Offline
  • Gold Boarder
  • Posts: 310
  • Thank you received: 32
  • Karma: 5
kino13 wrote:
Same as people is commenting already, there are so many results that the search becomes useless.

Is there any option or some modification we can do on the scrapper to accept only the first 200 results?

Definitely a lot of results. Almost like it is doing as OR operation instead of an AND.

We'll have to wait on cbanak to see if anything can be modified on the scraper or if we have to go to the GB devs.
The administrator has disabled public write access.

Comic Vine Scraper 6 months 2 weeks ago #48991

  • krandor
  • krandor's Avatar
  • Offline
  • Gold Boarder
  • Posts: 310
  • Thank you received: 32
  • Karma: 5
garcunning wrote:
Don't know if this is a script problem or a comicvine problem.

Right now it is a CV problem because it didn't used to do this. Can it be patched in the script to account for the CV change is the question?
The administrator has disabled public write access.

Comic Vine Scraper 6 months 2 weeks ago #48992

  • Scuttle
  • Scuttle's Avatar
  • Offline
  • Junior Boarder
  • Posts: 28
  • Thank you received: 15
  • Karma: 6
krandor wrote:
Definitely a lot of results. Almost like it is doing as OR operation instead of an AND.

I think this is exactly what is happening. I dug around in the API, and there doesn't seem to be a way to use AND/OR using the API right now, so the problem doesn't lie within the scraper, it's in the search engine itself

I tried this URL, and it returns 487 results.
www.comicvine.com/api/search/?api_key=XX..._list=name&limit=100
The relevant ones are the first two ones, then it just devolves into a list of titles that has one or more of the words in the title
Last Edit: 6 months 2 weeks ago by Scuttle.
The administrator has disabled public write access.
The following user(s) said Thank You: cbanack

Comic Vine Scraper 6 months 2 weeks ago #48993

  • krandor
  • krandor's Avatar
  • Offline
  • Gold Boarder
  • Posts: 310
  • Thank you received: 32
  • Karma: 5
Scuttle wrote:
krandor wrote:
Definitely a lot of results. Almost like it is doing as OR operation instead of an AND.

I think this is exactly what is happening. I dug around in the API, and there doesn't seem to be a way to use AND/OR using the API right now, so the problem doesn't lie within the scraper, it's in the search engine itself

I tried this URL, and it returns 487 results.
www.comicvine.com/api/search/?api_key=XX..._list=name&limit=100
The relevant ones are the first two ones, then it just devolves into a list of titles that has one or more of the words in the title

Thanks for the example URL. I'm going to go post that on their forums right now.
The administrator has disabled public write access.

Comic Vine Scraper 6 months 2 weeks ago #48995

  • cbanack
  • cbanack's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1367
  • Thank you received: 550
  • Karma: 186
Hi guys, sorry I've been quiet lately, but I've still been following this conversation carefully.

The search results from the scraper are definitely still broken, because it is ORing the search terms instead of ANDing them, like krandor says.

This is either because:

a) Comic Vine / Gamespot still hasn't fully sorted their problems out with their search engine, or

b) Their search API has changed, and this is how it works now.

If the problem is A, we just need to keep posting to their forums to let them know whats up, and wait for them to fix it.

If the problem is B, we need to find out how to do "AND" based searches with their API. Scuttle has looked at the API and not found a documented way to do this, and I've fiddled around a bit and haven't yet found a way either. But if and when we DO find a way, I will of course update the Scraper to fix the problem.
The administrator has disabled public write access.
The following user(s) said Thank You: Scuttle

Comic Vine Scraper 6 months 2 weeks ago #48996

  • krandor
  • krandor's Avatar
  • Offline
  • Gold Boarder
  • Posts: 310
  • Thank you received: 32
  • Karma: 5
It is definitely the new search because if you search on the actual CV website it does the exact same ting. The question though is if that is the way the GB people want it to be working or if this is a bug.
The administrator has disabled public write access.
Time to create page: 0.273 seconds

Who's Online

We have 212 guests and one member online