The NEW Build Your Own Arcade Controls

Main => Forum/Website Discussion => Topic started by: saint on April 03, 2016, 02:47:33 pm

Title: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: saint on April 03, 2016, 02:47:33 pm
*** SERVER ISSUES *** - we are currently experiencing a heavy load on the server that we are attempting to resolve. Slow loading may occur until we have this puzzled out.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: Malenko on April 03, 2016, 07:05:06 pm
everything seems back to normal meow.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: WakiMiko on April 24, 2016, 10:16:03 am
I'm getting the following pretty much every day now when I try to visit the forums. It seems to depend on the time of day, the site is usually fine after 15:00 CEST or so.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: Malenko on April 25, 2016, 09:46:25 am
Our top guys are on it.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: yotsuya on April 25, 2016, 10:36:42 am
Why hasn't the ScottBot just jacked into the server with his KADE neural interface and diagnosed the issue?
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: vwalbridge on April 25, 2016, 10:47:45 am
I'll create a GUI interface using Visual Basic...see if I can track an IP address.

http://www.youtube.com/watch?v=hkDD03yeLnU (http://www.youtube.com/watch?v=hkDD03yeLnU)
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: lilshawn on April 25, 2016, 03:16:07 pm
have to say anything user specific related is slow (new replies, my messages, etc) are all slow. the site itself is fine. (pulling up a specific post or opening the main trunk thread.)

something in the user database seems to be boggin things down.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: acvieluf on April 25, 2016, 06:26:15 pm
I'll create a GUI interface using Visual Basic...see if I can track an IP address.

http://www.youtube.com/watch?v=hkDD03yeLnU (http://www.youtube.com/watch?v=hkDD03yeLnU)

This is the greatest thing I've watched on YouTube in a while! Thank you.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: dmckean on April 27, 2016, 08:18:25 pm
Any news?

Nothing has improved.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: lilshawn on May 03, 2016, 11:26:10 am
boy she is sloooow today. took me several minutes to get to this thread and reply...and that was after i spent 5 and gave up on reading a message in my inbox.

is it time to nuke and start over?

http://i.imgur.com/j7wfDMu.jpg (http://i.imgur.com/j7wfDMu.jpg)
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: ed12 on May 03, 2016, 11:41:58 am
well for me at exactley 12:am my time est. the server goes dead to the world,
has been going on for a few weeks,just like someone through the power switch.?
i agree time to nuke it and start over ?

ed
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: dmckean on May 03, 2016, 12:07:03 pm
It's ridiculous now.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: harveybirdman on May 03, 2016, 12:09:52 pm
I hate to jump on the complaint train but dang Saint.... Let's figure it out, make everyone pitch in a dollar for posting privileges or something but just get it fixed.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: harveybirdman on May 03, 2016, 12:24:25 pm
Did this thread get post helled? Nice..
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: yotsuya on May 03, 2016, 12:35:16 pm
Did this thread get post helled? Nice..
I can see it. Woah.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: yotsuya on May 03, 2016, 12:35:54 pm
(http://uploads.tapatalk-cdn.com/20160503/9d1c487286271561fddc0f5371f6e275.jpg)

Last night on the iPad...
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: dmckean on May 03, 2016, 12:39:59 pm
It does that every night now between 10 and 11 PST.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: vwalbridge on May 03, 2016, 02:15:23 pm
It's been particularly slow the last 48 hours. In some cases, the forum simply never loads.

I'm not going to complain about something not working when it is FREE to the community.

However, I vote for the KLOV model:

Make everyone donate $1 or $2 to post. Not only does this generate money to keep the site hosted, but it will also help keep the spammers out. In turn, it might make the mods life easier too. They would only have to approve accounts that have already donated. Instead of approving every new post that comes in.

Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: Token on May 03, 2016, 02:38:15 pm
However, I vote for the KLOV model:

Make everyone donate $1 or $2 to post.

Someone start a gofundme for pbj.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: dmckean on May 03, 2016, 02:52:31 pm
Most forums I'm a member of just periodically ask for donations to help pay for hosting and forum software updates.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: lilshawn on May 03, 2016, 03:06:19 pm
while I was waiting for my reply page to load...2 replies had posted

when did we update SMF to 2.0.11? is there something in that update that broke the forum?

some pages load fine *snap* like that, others..... ticktockticktock if at all.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: Malenko on May 03, 2016, 03:39:11 pm
best guess, its MySQL related and probably more specifically, and issue with the database.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: pbj on May 03, 2016, 03:47:45 pm
I'm getting logged out fairly frequently.  Some fine 'network engineering' going on around here.  :-P

Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: ed12 on May 03, 2016, 04:09:10 pm
well what i gat at 12:am on the dot is the pic yout's posted
error 500 internal server error ??
some one is on the switch!! fire them

ed
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: Nephasth on May 03, 2016, 06:18:58 pm
(http://i.onionstatic.com/avclub/3977/98/16x9/960.jpg)
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: Vigo on May 03, 2016, 06:56:38 pm
Saint, anything we can do to help with the recent issues?
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: shock_ on May 04, 2016, 12:37:13 am
I just saw a slowdown at about 3-4 mins past the hour, but it's come good again. It's just now a little past midnight US Eastern. In the next couple of hours if history is a guide, the site is going to slow again and then throw 500 errors.

Given the symptom (500 errors from a server running SMF) and the timing (daily, just after "midnight", in someone's US timezone at least) I'm willing to guess:


Things I'd do

Check for active queries while the site is slow.
Assuming there's direct access to the database (e.g. command line or even PHPMyAdmin), send this query to the site database as the site database's user:

SHOW FULL PROCESSLIST\G

If the database user has the necessary privileges this will braindump the active db queries. Things to look for are the lines with "Time:" and "State:" -- if there are heaps of rows and they're slow, well, yeah, database bottleneck. The queries themselves might give a clue exactly WHAT it's busy with, which might lead to some ideas about shifting that activity to some other time etc.

Check for any spikes in site traffic.

Count rows per minute in the web server's access.log file, look for any spiky patterns for IP addresses or User Agents. Could be as simple as something set to crawl the site at the same time each night.

Make sure someone with SMF knowledge is involved

I'm talking very generally here, a lot of the true fixes are going to require knowledge of how to tune SMF to avoid being wasteful. Perhaps there was a spamfest a few months ago that led to hundreds of thousands of dummy registrations in the database, now slowing down daily admin functions due to sheer size. Maybe a feature was enabled that runs some daily background processing and nobody remembered to turn it off. I don't know, I'm a Drupal guy, not an SMF guy, but all commodity web apps have their quirks and all can be abused from time to time :)
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: RayB on May 04, 2016, 02:10:19 pm
man it's SOOOO slow.
Speaking of donations, I agree with this idea. Saint, set up a Patreon. I'd pay a buck or two a month to keep this place running.

https://www.patreon.com/ (https://www.patreon.com/)

Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: Token on May 04, 2016, 09:13:38 pm
I'm all for throwing money at the problem, but that does no good if saint's TOP MEN don't know what the problem is.



It took five minutes of page refreshing to post this comment.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: Nephasth on May 04, 2016, 09:47:40 pm
saint's TOP MEN...


 :laugh2:
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: Vigo on May 05, 2016, 11:05:40 am
(https://cdn.meme.am/instances/43262206.jpg)
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: vwalbridge on May 10, 2016, 10:33:14 am
(http://pixel.nymag.com/content/dam/daily/other/2014/06/30/beuller.gif)
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: yotsuya on May 10, 2016, 10:48:11 am
Reminds me of a former IT director we had who swore unequivocally that if there were any issues, "we'd have boots on the ground in 15 minutes." Lots of minutes went by with no boots ever.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: Phreakwars on May 10, 2016, 02:37:18 pm
I'm gonna guess that the problem is in the SQL database. I had this issue when I was using velcom. I switched my server to Liquidweb and the problem went away. Also velcom did not update their php myadmin panel often enough and I was compromised by a mid eastern hacker.

I'd seriously consider changing servers. It seems the issue is constantly happening.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: ark_ader on May 10, 2016, 07:16:07 pm
I would seriously look at the Burning Board software.  It is not free but they have excellent technical support.  I had their services when I ran CS gaming boards.

https://www.woltlab.com/wbb/ (https://www.woltlab.com/wbb/)

I would donate $10 or $20 a year for a new server platform.

You have to look at the benefits BYOAC has to offer, and what we have been receiving for free over the many years.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: shock_ on May 10, 2016, 07:27:14 pm
Migrating an application of this size and age off Simple Machines Forum to another application like Burning Board is likely a thousand times more work and risk and cost, than getting the hosting stabilised. Saint's running the latest stable version of SMF which demonstrates someone at least is doing the right thing and keeping it up to date.

I'd suggest passing the hat around is not a bad idea, with the goal of engaging the support required to diagnose, then repair or migrate the hosting depending on what's actually failing.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: Phreakwars on May 11, 2016, 03:55:00 am
If your worried about moving to another platform, just do like I did and install XAMPP on your local drive then copy the entire forum and database and convert from there. I know there are tons of converter tools for PhPBB, vBulletin, SMF, UBB, IPB, MyBB, XenForo, etc, etc.  SMF is nice because of the hook system when installing MODS, but it tends to be pretty taxing on the SQL. My personal preference would be PhPBB because it is so easy to work with. I've had no problem at all creating my own MOD's for it, even have a couple of them listed in the PhPBB MOD database. Not to mention there are tons of theme's available for it, and creating theme's is a breeze too.

One of the forums I owned had 500 plus sub forums (newsucks500.com now under new ownership and new platform), which, on most any other software, can really overheat those SQL database tables. PhPBB was able to handle it all just fine with no slowdown or crashing. Something I never would have attempted with SMF. In fact, the largest forum on the internet (gaiaonline.com), is nothing more then a heavily modified version of PhPBB.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: saint on May 11, 2016, 07:07:03 am
Hey everyone - I appreciate it, and I'm sorry I haven't been responsive here.

Thank you for the offers of financial donations. It is within my means to be able to afford it if needed, so your intentions are gratefully acknowledged but respectfully declined :)

We're not on a hosted platform, I lease the entire server.

I've asked for assistance from someone with a level of expertise in the OS/config on the server. If this person is able to help then we'll be in good shape. If not, my intent is to lease a new server (this one is now about 3 years old) and migrate everything over to a fresh installation. I plan to stick with SMF unless it turns out the issue is SMF in some fashion. I don't believe it is, but obviously if I knew what it was it'd be resolved by now :)

It's going to take some more time. The person I've asked for help has their own life/work/family responsibilities, so time on fixing the server has to compete with that. If they aren't able to assist, in a couple of weeks I will start working 4x10 work weeks with 3 day weekends, and that will give me the extra time to purchase and start configuring the new server.

I appreciate everyone's patience and concern. If all else fails I will throw up the white flag and ask for additional help from the forum. For security's sake I am cautious about who I give access to the server directly, because it has your information on it in some fashion, but the option is on the table if needed.

I did finally figure out why my script to restart sql/web-server was failing, and as of this morning it automatically restarted things rather than having to wait for me to manually log on and restart them. This means when I sleep in on the weekend the forum won't be down until 9-10 AM :)

Thanks everyone!
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: saint on May 11, 2016, 07:10:47 am
I just saw a slowdown at about 3-4 mins past the hour, but it's come good again. It's just now a little past midnight US Eastern. In the next couple of hours if history is a guide, the site is going to slow again and then throw 500 errors.

Given the symptom (500 errors from a server running SMF) and the timing (daily, just after "midnight", in someone's US timezone at least) I'm willing to guess:

  • SMF has too many scheduled tasks set to fire at 00:00 that don't need to be running at exactly that time - see http://wiki.simplemachines.org/smf/SMF2.0:Scheduled_Tasks (http://wiki.simplemachines.org/smf/SMF2.0:Scheduled_Tasks)
  • Those tasks are possibly creating a queue on MySQL.
  • PHP is throwing "MySQL has gone away" errors into an appropriate log once calls out to the DB exceed its timeout
  • Web server is slow to respond while it still trusts MySQL might eventually have an answer for it, then starts sending quick 500s once it's sure it won't be back

Things I'd do

Check for active queries while the site is slow.
Assuming there's direct access to the database (e.g. command line or even PHPMyAdmin), send this query to the site database as the site database's user:

SHOW FULL PROCESSLIST\G

If the database user has the necessary privileges this will braindump the active db queries. Things to look for are the lines with "Time:" and "State:" -- if there are heaps of rows and they're slow, well, yeah, database bottleneck. The queries themselves might give a clue exactly WHAT it's busy with, which might lead to some ideas about shifting that activity to some other time etc.

Check for any spikes in site traffic.

Count rows per minute in the web server's access.log file, look for any spiky patterns for IP addresses or User Agents. Could be as simple as something set to crawl the site at the same time each night.

Make sure someone with SMF knowledge is involved

I'm talking very generally here, a lot of the true fixes are going to require knowledge of how to tune SMF to avoid being wasteful. Perhaps there was a spamfest a few months ago that led to hundreds of thousands of dummy registrations in the database, now slowing down daily admin functions due to sheer size. Maybe a feature was enabled that runs some daily background processing and nobody remembered to turn it off. I don't know, I'm a Drupal guy, not an SMF guy, but all commodity web apps have their quirks and all can be abused from time to time :)

Thank you Shock, I'll dig into these. While it's slow all day, it *is* consistent on when it finally gives up the ghost each day.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: saint on May 11, 2016, 07:17:28 am
For any Linux (particularly Ubuntu) gurus out there, here is what a random sampling of iotop shows while the forum is running. The various bits that are showing up in the top change about 1-2 times per second.  In a few minutes, I'll down the web server and mysql, and capture a few iotops while they aren't running for comparison. Unfortunately, I don't have a baseline to know what is and isn't normal.

Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: saint on May 11, 2016, 07:21:56 am
Some additional info for consideration. In addition to the forum (relying on mysql) exhibiting slowness, I also run Squirrel mail (no reliance on mysql) on the server.

When the forum is slow, access to squirrel mail is slow. This makes me think it's not a mysql problem, or better said, not just a mysql problem, but rather whatever the problem is is also bogging down the server, as the iotop shows.

OK, killing the server now to get some non-mysql/web-server iotops for comparison.

Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: saint on May 11, 2016, 08:15:37 am
Additional observations:

Mysql takes a long time to stop or to restart. Minutes.

When mysql is down but the web server is running, Squirrel mail runs fast and snappy, no lag whatsoever.

Once I restart sql, it takes a good 10-15 minutes or so before the forum can connect to the sql server
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: saint on May 11, 2016, 08:24:21 am
iotop snapshots when mysql was stopped but everything else was still running:

Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: saint on May 11, 2016, 08:25:13 am
A couple more from when mysql was stopped:

Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: pbj on May 11, 2016, 09:20:02 am
But do you need money?  Apparently your users are gagging to give you money.  Please tell them you need money.

Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: Phreakwars on May 11, 2016, 03:23:21 pm
Are there other domains on the server that you are sharing, or is it a dedicated server? You can experience problems of the like when you are sharing with several domains. That's just kind of the nature of it when it comes to some of these storage space companies. You could try asking to be switched to a different rack that isn't being utilized as much by others.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: saint on May 11, 2016, 03:55:17 pm
Are there other domains on the server that you are sharing, or is it a dedicated server? You can experience problems of the like when you are sharing with several domains. That's just kind of the nature of it when it comes to some of these storage space companies. You could try asking to be switched to a different rack that isn't being utilized as much by others.

Dedicated server.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: shock_ on May 11, 2016, 04:00:52 pm
Saint that does sound encouraging. I thought maybe there was some manual breakfast service restarts going on, that does fit the pattern of downtime :)

One other really good thing to do is install New Relic PHP extensions. Not too hard to get going, and the application monitoring will show you exactly what is going on over time, and allow for some deep diving into slow transactions for the first couple of weeks while they give you the full, expensive report package. Their free tier is still worth leaving up though too, once the trial expires. They also have a server health metrics thing that will graph things like your iotop and more.

The reporting in New Relic shouldn't have any personal info exposed and if you later feel like more public discussion is useful, well, we all love graphs :)
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: Malenko on May 12, 2016, 03:55:53 am
Install atop

Run it. Press "i" to set the interval, I like 3 seconds.

See if the disks are actually busy and what the request latency is.  See if there's a bunch of CPU iowait. Are the CPUs busy? Is the box actively swapping to disk?

If the disk latency is higher than low single digit milliseconds but the disks don't look busy, check the SMART stats with smartctl. Look for a high number of reallocated sectors. This would indicate the drive is failing.

If all this looks okay and the drives are mostly idle, the box isn't swapping, the CPUs aren't busy and there's no high iowait, look at mysql.

Enable the slow query log: set global slow_query_log=on;
See where mysql is writing the slow log: show variables; look for slow_query_log_location or similar.
Find a query or queries that are logged slow, and run explain on it in mysql: explain <paste query here>;
Look for things that aren't using indexes, anything that says "null null all" is doing a full table scan and is going to be hella slow.

Let me know how all this crap looks and we'll go from there.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: Phreakwars on May 12, 2016, 06:26:46 am
Would also like to add, try asking one of the guys on The Admin Zone.

https://theadminzone.com/

Been a member there for years, they have some of the best guys who can also help solve the issue. You'll be able to find people who know the in's and out's of every aspect of running a forum and also know all about the SMF software. The SMF developers themselve's also frequent the forum.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: shock_ on May 12, 2016, 07:10:27 am
Make sense of your MySQL slow query log file by installing percona toolkit and running pt-query-digest on the mysql-slow.log.  pt-query-digest takes --since= and --until= so you can limit it to slow queries that occurred at an interval you care about, e.g. midnight to 1am.  If a part of the problem is indeed iowait (which stands to reason if other services are slow during the worst of it, your CPU load is probably enormous) it's presumably going to either be a lot of table writes or perhaps large temp tables on disk due to subqueries. pt-query-digest will show you that plain enough.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: Le Chuck on May 12, 2016, 12:08:28 pm
I bet if we could like threads and had +fu this wouldn't have happened. 
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: pbj on May 12, 2016, 12:40:27 pm
Have you tried turning it off and back on?

Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: Slippyblade on May 12, 2016, 01:14:32 pm
Have you tried turning it off and back on?

For great justice!  Seriously though...  have you?
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: lilshawn on May 12, 2016, 03:28:23 pm
Have you tried turning it off and back on?

For great justice!  Seriously though...  have you?

have you tried forcing an unexpected reboot?
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: ed12 on May 13, 2016, 11:58:37 pm
let me see
glue it down once like felt it sticks
then yank it back out and fource it back in ?
is that about right ???
it is a simple rule,if it is not broke DO NOT FIX IT...
so with that in hand we jumped to a new ?? rev we can not fix ? and need to wait ? huh ?

ed
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: ed12 on May 14, 2016, 12:32:45 am
these errors right here point to the weakness of ubt...

    SMF has too many scheduled tasks set to fire at 00:00 that don't need to be running at exactly that time - see http://wiki.simplemachines.org/smf/SMF2.0:Scheduled_Tasks (http://wiki.simplemachines.org/smf/SMF2.0:Scheduled_Tasks)
    Those tasks are possibly creating a queue on MySQL.
    PHP is throwing "MySQL has gone away" errors into an appropriate log once calls out to the DB exceed its timeout
    Web server is slow to respond while it still trusts MySQL might eventually have an answer for it, then starts sending quick 500s once it's sure it won't be back

it cannot deal with the load..never was intended for that..
and that is a period
time to move to debain again and run some nice ice

ed
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: Malenko on May 23, 2016, 01:10:52 pm
Been a couple weeks, do anything I posted? Got any results?

Server was down around 2am with the error 500 in case you didn't know.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: ark_ader on June 02, 2016, 04:52:05 pm
I don't bother visiting this board that often now, with all these issues.  I'm sure the majority of lurkers have left too.

It is a shame really, but until someone gets a handle on this only the die-hards will frequent this site. 

Probably a good thing considering.

Hope you get this fixed soon Saint.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: UEDan on June 02, 2016, 08:06:59 pm
You're running all that on 1 server? Dude....
What are the specs of that machine?
Why are you still hosting your own mail server when Gapps is low as $5/user/month. Or $4/user/month with AWS.
Why not look into a cloud compute platform? Are the operating costs that low at your current provider?

C'mon man! Lets do some migrations! I'm sure there are plenty of people here willing to help. *hint"hint*
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: ed12 on June 02, 2016, 10:18:20 pm
well 1 has to admit after 6 weeks of this, some is asleep at the switch.

ed
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: Malenko on June 03, 2016, 11:27:42 am
well 1 has to admit after 6 weeks of this, some is asleep at the switch.

ed

thread is 2 months old, issue has been going on for a couple of weeks longer (probably about 11 weeks total?). I tired to help but have gotten no response.
So much of us at Zapcon were going to post a ton of content but it wasnt worth the hassle of the site throwing out 500 errors when posting. I started a new project that I havent posted about because Im not ready for the headache/timesink of uploading pictures
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: lilshawn on June 03, 2016, 11:57:06 am
i would often check in a half dozen or more times a day....

since this started happening... maybe once every 2 days.

not good.

Archive the site, and nuke it, and start over.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: ed12 on June 03, 2016, 01:14:08 pm
Malenko  well put
shawn    same thoughts

time to either fix it saint or walk away..

ed
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: saint on June 05, 2016, 08:52:31 pm
OK, first, sorry for the lack of communication everyone. That's on me.

This evening Sirwoogie was able to look at the server and has an inkling as to what may be going on and how to resolve it. Over the next few days, if you see the site isn't available (as compared to being slow or throwing 500 errors) it's being worked on. I'll give an update when I can.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: saint on June 05, 2016, 11:59:58 pm
Looks like one main problem is a failing drive in the raid. Will be working with support to get it swapped out.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: sirwoogie on June 06, 2016, 12:11:12 am
Yep, in all my time working on Linux hosts, I've not seen a server behave like this. What we end up getting is high service wait times on the drives. But, there is not enough load going on to cause that behavior. We've run this way for nearly 4 years without issue. I can't track this back to a particular upgrade or change made, so I'm somewhat stumped.

We're going to look at the drive replacement, but also doing some OS upgrade and see if that kicks out the bugs.

The worst case scenario is moving to another box. But that takes a lot of time of course, and that would be the last ditch option.

We'll try to keep the updates coming.
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: ed12 on June 06, 2016, 12:27:00 am
what ever u were doing around 10:12 pm (my) time worked flawless
for 2mths have not been able to log in with (reg) screen.. has been walking around that now.?
if this is any help ?

ed
Title: Re: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server
Post by: saint on June 07, 2016, 11:56:22 am
Please see update here: forum.arcadecontrols.com/index.php/topic,150884