Been bughunting the last week or so on this import job that is suddenly running so slowly that it takes more than 24h and is restarting on top of itself.

It used to run in anywhere from 4 to 8 hours, which was bad enough. You see, it ran on timer scheduled by our main site. So our deployment window was determined by when this job finished, and if it didn't finish during work hours, then no deployment that day.

So we got the idea to move it to a separate service to eliminate that deployment window bottleneck. And now, seemingly unrelated, it is just running slow as shit.

There is a lot of bad design in the code, and we know we want to build a completely new solution. But we also absolutely need this import to run every day until a better solution can take over.

We've taken care of some of the most obvious problems that could cause the poor performance, but it's unclear whether it's going to be enough. And with a runtime of about a day and wild variations of the most atomic partial imports, it's extremely tedious to test as well...

  • 1
    Is it on a server with less resources?
  • 1
    @steaksauce Same server. Good question though :-)
  • 4
    Have you compared versions and configs?
    If it's php, a different php.ini can cause all sorts of issues.
    How is it connecting to your db? Wrong shard?
    What is its network connection like? If it's in a colo, inquire about its vlan, as that could potentially decrease network speed. Could also be a dying switch; I've had that happen before with Hetzner. (Remedied quickly to their credit.)
    How about niceness? If your process is too nice, it would run slower. Especially if there are other resource-intense processes on the same box. I've had that issue with Heroku: the company's vm was hosted on a server with other vm's using too many resources, causing caching issues and other random bottlenecks.

    Could also be coincidence, and the issues actually stem from your db, e.g. a missing index, a table hitting a threshold, etc.
Add Comment