Welcome, Guest
News and Announcements

TOPIC: Comic Vine Scraper

Comic Vine Scraper 9 months 3 weeks ago #48756

  • kino13
  • kino13's Avatar
  • Offline
  • Senior Boarder
  • Posts: 58
  • Thank you received: 6
  • Karma: 0
Yes, it's back online.

However, the results are erratic at best.

Covers take about a minute to appear. Scrapping stops randomly for no reason.

I have tested in three different systems.

It improved a bit on sequential scrapping (automatic, same series), after I manually removed all the cache files from the disk, but still it fails constantly.
with no power comes no responsibility. except that wasn't true
The administrator has disabled public write access.

Comic Vine Scraper 9 months 3 weeks ago #48757

Interesting. I've experienced two outages in the last hour. In each case it came back within a few minutes and then all was back to normal for me. I haven't experienced any performance issues at all over the course of the last few hours.
The administrator has disabled public write access.

Comic Vine Scraper 9 months 3 weeks ago #48759

  • kino13
  • kino13's Avatar
  • Offline
  • Senior Boarder
  • Posts: 58
  • Thank you received: 6
  • Karma: 0
Well, to make it more funny... It is working perfectly at this moment. :(

Whatever's the issue it is not related to my system.

I am starting to think there may be something wrong on an isp level, but I have no way to check if there are lost packets against the server...
with no power comes no responsibility. except that wasn't true
The administrator has disabled public write access.

Comic Vine Scraper 9 months 3 weeks ago #48760

  • boshuda
  • boshuda's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 335
  • Thank you received: 86
  • Karma: 10
kino13 wrote:
Well, to make it more funny... It is working perfectly at this moment. :(

Whatever's the issue it is not related to my system.

I am starting to think there may be something wrong on an isp level, but I have no way to check if there are lost packets against the server...
I doubt it's ISP unless you're experiencing similar issues on other sites/services. People more knowledgeable about this stuff than I am have written that the comicvine API is less than optimal and we've all had weird intermittent issues with it over the years. So the problem is probably on their end. You could try to report it with as much detail as possible on their forum. But it's a free service so it pretty much is what it is.
The administrator has disabled public write access.

Comic Vine Scraper 9 months 3 weeks ago #48761

  • kino13
  • kino13's Avatar
  • Offline
  • Senior Boarder
  • Posts: 58
  • Thank you received: 6
  • Karma: 0
Devil knows what the hell are they doing on the server side...

And yes, it may be a free service, but it was created by the community.

I still remember downloading the imdb locally with my friends 20 years ago. That was also made by the community, so yes, I get quite angry when they don't offer other options like replicating the fucking system somewhere else.

Sorry, I am just venting, it is just so frustrating that they have no consideration for the people using the frigging db...
with no power comes no responsibility. except that wasn't true
The administrator has disabled public write access.

Comic Vine Scraper 9 months 3 weeks ago #48762

Yeah, they have ICMP disabled on their hosting server. Not so simple to check for packet loss against a host that doesn't respond to pinging. I've been scraping since 7:00AM or so (I have a bad back and this beats staring at the walls on Christmas waiting for my wife to get up) I've had only one connection issue this morning so far. Since the disconnect issues I've experienced over the last few weeks have always resolved themselves within a few minutes I'm guessing these issues might be load related. I'm no nginx wizard so I couldn't begin to guess why these load issues are occurring.
I have my monitoring system set up to check HTTP response times on TCP/443 on the off chance that proves useful to the ComicVine guys at some point. It will graph how quickly the API responds to HTTPS requests and therefore show outages as well. That's not much help in diagnosing connectivity issues though.
Last Edit: 9 months 3 weeks ago by Cyber-Wizard.
The administrator has disabled public write access.

Comic Vine Scraper 9 months 3 weeks ago #48763

  • kino13
  • kino13's Avatar
  • Offline
  • Senior Boarder
  • Posts: 58
  • Thank you received: 6
  • Karma: 0
Yesterday it worked fine all day. I scrapped a lot of delayed comics I had around.

Today is back to shit. It is completely unusable. It made 80 comics in an hour.

Performance is completely random, from 7 seconds to 3 minutes for a comic, with constant interruptions of the whole process.

I will stop reporting this shit here, I believe the problem is not located on my side (or the scrapper).

Regards
with no power comes no responsibility. except that wasn't true
The administrator has disabled public write access.

Comic Vine Scraper 9 months 2 weeks ago #48768

  • kino13
  • kino13's Avatar
  • Offline
  • Senior Boarder
  • Posts: 58
  • Thank you received: 6
  • Karma: 0
Yeah, I just said yesterday I would not report this shit anymore... anyway.

I am starting to see a pattern here, this morning I have been scrapping with no issues whatsoever, faster than hell. Cover response just a couple seconds.

Until 15h GMT, then it slowly when back to shit. This is exactly the same that happened the other day.

At this moment it is completely broken again. It has been trying to scrape 15 avengers comics for the last 40 minutes.
with no power comes no responsibility. except that wasn't true
The administrator has disabled public write access.

Comic Vine Scraper 9 months 1 week ago #48805

  • perezmu
  • perezmu's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 1135
  • Thank you received: 64
  • Karma: 51
Hello,

I have been absent from CR for months. Now I am scraping lots of books - fairly 8000 with no problems from the server.

However, I have found a couple of hip-cups that AFAIK were not there... any thoughts?

- The symbol "-" is not recognized: so, if i search "Spider-Man", nothing comes out, I need to use "SpiderMan"
- The word "presents" is not recognized... in particular "Marvel Comics Presents", see comicvine.gamespot.com/marvel-comics-presents/4050-4058/ cannot be found by the scrapper, no matter how I search for it (searching "Marvel Comics" only does not make it appear)

Finally, I still find frustrating that the scraper cannot separate different volumes of the same series when all issues are scraped together... but this is old ;)

Cheers,

Arturo
The administrator has disabled public write access.

Comic Vine Scraper 9 months 1 week ago #48806

  • oraclexview
  • oraclexview's Avatar
  • Offline
  • Moderator
  • aka SoundWave
  • Posts: 919
  • Thank you received: 189
  • Karma: 38
perezmu wrote:
Finally, I still find frustrating that the scraper cannot separate different volumes of the same series when all issues are scraped together... but this is old ;)

The only time I've had a problem with the scraper always recognizing the wrong volume of a series is when a series had two different volumes that began in the same year. Another example would be if the file name itself was too simple or a complete mess. In any case those scenarios are not the fault of the scraper. Yet, definitely let me know if I'm wrong on those conclusions.
The administrator has disabled public write access.
Time to create page: 0.303 seconds

Who's Online

We have 114 guests and no members online