Ticket #147 (closed bug: worksforme)

Opened 7 months ago

Last modified 7 months ago

Drone Database - Inserting URLs Causes Web Service to Slow Down

Reported by: kindlund Assigned to: mbriggs
Priority: low Milestone: 1.2
Component: Drone Version: 1.02
Severity: major Keywords: drone, crawl, insert, slow, performance
Cc: dbryson

Description

When inserting a large amount of URLs into the Drone web service, I've noticed that the rest of the web service slows down completely.

I suspect this is because the Ruby web service is running in some sort of single-thread mode, where new requests will not get processed until every older request is serviced.

So, if we feed the Drone webservice 50 URLs, then the service will have to perform a DNS lookup on each hostname (which takes significant time). During that time, the web service appears to hang/pause. Once the insert is finished, the web service completes.

Dave, is there any way we could multi-thread this portion of the code, in order to handle simultaneous requests properly? Perhaps this is just an artifact of using WEBrick as opposed to Apache? Let me know your thoughts on this.

Attachments

Change History

03/26/08 19:48:59 changed by kindlund

  • status changed from new to closed.
  • resolution set to worksforme.

I've switched the Drone web service to using Apache FastCGI for now. The performance benefit so far is huge. I'll look into incorporating the use of Mongrel at a later date.

Add/Change #147 (Drone Database - Inserting URLs Causes Web Service to Slow Down)




Change Properties
Action