Our Tender system is, for all intents and purposes, completely down
Hello,
We have been experiencing dramatic slowdowns in our Tender since about 3:15 Central time. We have ruled out internal network issues. I am able to load pages very slowly about 20% of the time now, the other 80% results in the error "We're sorry, but something went wrong. We've been notified about this issue and we'll take a look at it shortly."
We are at an extremely busy time so we are asked some of our staff to go home early (they weren't able to do any work) and check back from home tonight in the hope that Tender is usable again.
We had a short period (I'd say 3 minutes) a little bit ago where things seemed normal, but my dashboard has been (again) trying to load for about - ok, it took about 2 minutes to load the dashboard.
Dawn
Mariner Software
Discussions are closed to public comments.
If you need help with Tender please
start a new discussion.
2 Posted by Will on 08 Mar, 2011 11:35 PM
The server is running a hotter than normal, but for the most part everything looks to be up and running, just a little slower than normal.
We've been crunching through some processes on the server and that may be related. Average response time has been slower, but errors shouldn't be occurring either way.
3 Posted by dawn on 09 Mar, 2011 12:51 AM
I'm trying to catch up on our support load from home and I'm continuing to get very slow page loads and 500 errors.
I just did a speed test on my internet connection, I have 15.36Mbps down and 8.32Mbps up.
When I am in a discussion and I click on Admin, it is taking around 45 seconds for the dashboard to load. My last two attempts to do this have resulted in 500 errors.
My dashboard view is currently filtering/sorting by queue and category.
4 Posted by Will on 09 Mar, 2011 08:16 AM
As of 5:00 PM PST everything has been completely idle across the servers. All delayed items have been processed. We'll be keeping a close eye on the server in the morning when traffic picks back up, but right now we don't expect the kind of slowness we saw today.
5 Posted by David Coallier on 09 Mar, 2011 10:52 AM
We've been noticing some of the slowness as well. Are you guys running some sort of backup or analytics job around our European morning? :) Currently in GMT and around 10 it's been very slow for the past few days.
6 Posted by Will on 09 Mar, 2011 11:24 AM
Yesterday was unrelated. But yes, Amazon EC2 backs up at that time and we're seeing elevated response times. Tonight EC2 was having issues for approximately 22 minutes during that backup window where the servers respond slow.
We're working on placing measures to prevent the backup from causing the site to runs low.
Will closed this discussion on 09 Mar, 2011 11:24 AM.
dawn re-opened this discussion on 09 Mar, 2011 02:32 PM
7 Posted by dawn on 09 Mar, 2011 02:32 PM
I started my morning by viewing unqueued discussions in the In Box. My process: click on a discussion in the unqueued list, assign it to a queue, and click the Admin link to return to the dashboard. The first time I did this, it took maybe 30 seconds for the dashboard to load after viewing a specific discussion. The second time: 500 error.
It occurred to me that I've only recently started using the category/queue filtering extensively, and thought perhaps that may have something to do with the performance and errors, Sure enough - when I filter by category=In Box, queue=unqueued, the dashboard loading problems are persistent. When my filters reflect category=In Box, queue=All, the problem doesn't happen.
So it appears that I had two things going on yesterday, your server slowdowns + what appears to be a performance issue related to filtering by queue, which I suspect may be related to:
https://help.tenderapp.com/discussions/problems/3326-list-of-unqueu...
Filtering by queue could be a massive time saver for us but it seems to be problematic enough that we can't really use the feature.
Dawn
Support Staff 8 Posted by Courtenay on 09 Mar, 2011 03:21 PM
Got it. We are tuning the queries and hope to vastly improve the
performance on these specific actions in the near future. That query really hammers our database.
Will closed this discussion on 17 Mar, 2011 11:04 PM.