Welcome, Guest
News and Announcements

TOPIC: Comic Vine Scraper

Comic Vine Scraper 4 months 1 day ago #48731

  • Martell86
  • Martell86's Avatar
  • Offline
  • Junior Boarder
  • Posts: 29
  • Karma: 0
kino13 wrote:
And now it back to slow...

There's definitely an issue there, my bet is on the server side.

I am located in Europe, and it stopped working just around 15:00 gmt, which is just the morning in the US.

Hey, I'm using it right now, no issues. I just did a run of 59 comics, it took no time at all. As before, this is likely an issue on your side. If it were on the comicvine side, you'd see a whole lot more discussion right here.
The administrator has disabled public write access.

Comic Vine Scraper 4 months 1 day ago #48732

  • kino13
  • kino13's Avatar
  • Offline
  • Senior Boarder
  • Posts: 58
  • Thank you received: 6
  • Karma: 0
Well,

I took the time to measure the response:
-For the cover to appear, among 5 samples, it took from 39 to 72 seconds.
-From the moment the comic is selected, until the information is downloaded: from 52 to 84 seconds.

That's more than 2 minutes for a scrap. This morning I have scrapped a few comics and it was working correctly.

My internet connection is a 300Mb symmetric fiber.

Before anyone asks, my API page shows "You have used 7 requests in the last hour for API Path '/volume' (reset in 23 minutes)
Your request rate is fine ".

I am creating a virtual machine at the moment to make a clean install of comicrack to compare.
with no power comes no responsibility. except that wasn't true
The administrator has disabled public write access.

Comic Vine Scraper 4 months 1 day ago #48733

  • boshuda
  • boshuda's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 333
  • Thank you received: 84
  • Karma: 9
kino13 wrote:
My internet connection is a 300Mb symmetric fiber.
Every once in a while I'm reminded of how much the States is falling behind the rest of the world. :(


Can you do a simple ComicVine api request w/o using ComicRack? A bunch of programming languages have libraries for it, and I think there are even some 3rd party tools that are generically for testing web apis. I think some are even completely online (although that would remove you from the equation, which could be useful). IDK for sure, I'm not a web developer and haven't accessed the CV API directly in a long, long time.
The administrator has disabled public write access.

Comic Vine Scraper 4 months 1 day ago #48734

I just did a quick scrape of 20 comics and even did a few cover lookups. My scraper is set to pause 5 seconds between lookups but I didn't notice any problems. It returned data just as quickly as ever.

The size of the Internet connection is no guarantee of performance in every situation. I'm in Canada on a 25Mbps cable connection. After a little QoS configuration and a few firewall tweaks I get great performance across the board. I have a buddy with a 250Mbps cable pipe who is plagued with performance issues, especially when streaming video. I have a customer with a 300Mbps duplex fiber line who had terrible web traffic issues. A couple of users complained loudly about one particular website causing issues before we found out that the issue was the same with most other servers. It turned out their ISP poorly manages their DNS servers and every lookup was taking twice as long as it should.

It is also reasonable to think that the ComicVine server is subject to load issues. We don't know what else they may have running on that particular host or how heavily it's used by others. There are a lot of people using it for various things, some are using it correctly while others aren't. While it may be troublesome for us if it's intermittently slow, it's unlikely that they would put a great deal of effort into making a free service more consistent in it's performance.
The administrator has disabled public write access.

Comic Vine Scraper 4 months 1 day ago #48735

  • oraclexview
  • oraclexview's Avatar
  • Offline
  • Moderator
  • aka SoundWave
  • Posts: 915
  • Thank you received: 183
  • Karma: 38
I just did a quick scrape of 38 books (which 12 of them were automatically found and updated) in less than 1 and a half minutes.
The administrator has disabled public write access.

Comic Vine Scraper 4 months 1 day ago #48738

  • kino13
  • kino13's Avatar
  • Offline
  • Senior Boarder
  • Posts: 58
  • Thank you received: 6
  • Karma: 0
Thank you all for your interest, and for checking on your own.

I will have some spare time this weekend, so I will make as many tests as possible (clean reinstall, clean OS, different connection, different computer, etc).

I will report back with anything I find. Will try to check the address for the server, as well as if it is possible to manually make requests to it (telnet maybe?).

By the way, when it works correctly (and this morning did), it scraps quite fast.
with no power comes no responsibility. except that wasn't true
The administrator has disabled public write access.

Comic Vine Scraper 4 months 20 hours ago #48739

It does look like they are having some intermittent issues. I just had a scrape fail. When I tried going to the API web page I got that "Whoops, something went wrong" page. A few minutes later it was fine again.
The administrator has disabled public write access.

Comic Vine Scraper 4 months 8 hours ago #48746

  • boshuda
  • boshuda's Avatar
  • Offline
  • Platinum Boarder
  • Posts: 333
  • Thank you received: 84
  • Karma: 9
Sending API requests is really as simple as formatting your HTML address. I was playing with an Android app to test out Web APIs and I had requests coming back in about 10 minutes total, after creating a new account with a new API-key (in case I did something stupid, I didn't want to mess up my real key).

I would recommend before going through the hassle of new machines, fresh install, etc that you find or script a tool to make regular hits to the API and record their performance. Especially given the post right before this one from Cyber-Wizard:
It does look like they are having some intermittent issues. I just had a scrape fail. When I tried going to the API web page I got that "Whoops, something went wrong" page. A few minutes later it was fine again.
The administrator has disabled public write access.

Comic Vine Scraper 3 months 3 weeks ago #48754

  • kino13
  • kino13's Avatar
  • Offline
  • Senior Boarder
  • Posts: 58
  • Thank you received: 6
  • Karma: 0
Well, comicvine is down. So much for my testings...
with no power comes no responsibility. except that wasn't true
The administrator has disabled public write access.

Comic Vine Scraper 3 months 3 weeks ago #48755

I've been scraping on and off all morning. It went down a while ago but came back within a few minutes.
The administrator has disabled public write access.
Time to create page: 0.222 seconds

Who's Online

We have 156 guests and 3 members online