Main Restorations Software Audio/Jukebox/MP3 Everything Else Buy/Sell/Trade
Project Announcements Monitor/Video GroovyMAME Merit/JVL Touchscreen Meet Up Retail Vendors
Driving & Racing Woodworking Software Support Forums Consoles Project Arcade Reviews
Automated Projects Artwork Frontend Support Forums Pinball Forum Discussion Old Boards
Raspberry Pi & Dev Board controls.dat Linux Miscellaneous Arcade Wiki Discussion Old Archives
Site News

Unread posts | New Replies | Recent posts | Rules | Chatroom | Wiki | File Repository | RSS | Submit news


  

Author Topic: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server  (Read 3518 times)

0 Members and 1 Guest are viewing this topic.

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 5915
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Some additional info for consideration. In addition to the forum (relying on mysql) exhibiting slowness, I also run Squirrel mail (no reliance on mysql) on the server.

When the forum is slow, access to squirrel mail is slow. This makes me think it's not a mysql problem, or better said, not just a mysql problem, but rather whatever the problem is is also bogging down the server, as the iotop shows.

OK, killing the server now to get some non-mysql/web-server iotops for comparison.

--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade.com/
     saint@arcadecontrols.com

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 5915
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Additional observations:

Mysql takes a long time to stop or to restart. Minutes.

When mysql is down but the web server is running, Squirrel mail runs fast and snappy, no lag whatsoever.

Once I restart sql, it takes a good 10-15 minutes or so before the forum can connect to the sql server
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade.com/
     saint@arcadecontrols.com

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 5915
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
iotop snapshots when mysql was stopped but everything else was still running:

--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade.com/
     saint@arcadecontrols.com

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 5915
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
A couple more from when mysql was stopped:

--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade.com/
     saint@arcadecontrols.com

pbj

  • Trade Count: (+3)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 5898
  • Obey.
    • The Chris Burke Band
But do you need money?  Apparently your users are gagging to give you money.  Please tell them you need money.


Phreakwars

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 280
  • Waka-Waka
    • forum.arcadecontrols.com/index.php/topic,146325.0.html
    • My Facebook Page
Are there other domains on the server that you are sharing, or is it a dedicated server? You can experience problems of the like when you are sharing with several domains. That's just kind of the nature of it when it comes to some of these storage space companies. You could try asking to be switched to a different rack that isn't being utilized as much by others.

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 5915
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Are there other domains on the server that you are sharing, or is it a dedicated server? You can experience problems of the like when you are sharing with several domains. That's just kind of the nature of it when it comes to some of these storage space companies. You could try asking to be switched to a different rack that isn't being utilized as much by others.

Dedicated server.
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade.com/
     saint@arcadecontrols.com

shock_

  • Wiki Contributor
  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 354
  • In Brisbane, it's GMT+10 all the time.
Saint that does sound encouraging. I thought maybe there was some manual breakfast service restarts going on, that does fit the pattern of downtime :)

One other really good thing to do is install New Relic PHP extensions. Not too hard to get going, and the application monitoring will show you exactly what is going on over time, and allow for some deep diving into slow transactions for the first couple of weeks while they give you the full, expensive report package. Their free tier is still worth leaving up though too, once the trial expires. They also have a server health metrics thing that will graph things like your iotop and more.

The reporting in New Relic shouldn't have any personal info exposed and if you later feel like more public discussion is useful, well, we all love graphs :)

Malenko

  • KNEEL BEFORE ZODlenko!
  • Trade Count: (+58)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 11977
  • 說好話 做好事 存好心
    • forum.arcadecontrols.com/index.php/topic,142404.msg1475162.html
Install atop

Run it. Press "i" to set the interval, I like 3 seconds.

See if the disks are actually busy and what the request latency is.  See if there's a bunch of CPU iowait. Are the CPUs busy? Is the box actively swapping to disk?

If the disk latency is higher than low single digit milliseconds but the disks don't look busy, check the SMART stats with smartctl. Look for a high number of reallocated sectors. This would indicate the drive is failing.

If all this looks okay and the drives are mostly idle, the box isn't swapping, the CPUs aren't busy and there's no high iowait, look at mysql.

Enable the slow query log: set global slow_query_log=on;
See where mysql is writing the slow log: show variables; look for slow_query_log_location or similar.
Find a query or queries that are logged slow, and run explain on it in mysql: explain <paste query here>;
Look for things that aren't using indexes, anything that says "null null all" is doing a full table scan and is going to be hella slow.

Let me know how all this crap looks and we'll go from there.
2014 Standup UCA Runner Up
2015 Restoration UCA Winner
2016 thomas_surles choice award

Latest project:
http://forum.arcadecontrols.com/index.php/topic,150902.0.html
RedheadKingPinball aka GingerBalls

Latest Restore?
http://forum.arcadecontrols.com/index.php/topic,152600.0.html
Killer Instinct Cab

Phreakwars

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 280
  • Waka-Waka
    • forum.arcadecontrols.com/index.php/topic,146325.0.html
    • My Facebook Page
Would also like to add, try asking one of the guys on The Admin Zone.

https://theadminzone.com/

Been a member there for years, they have some of the best guys who can also help solve the issue. You'll be able to find people who know the in's and out's of every aspect of running a forum and also know all about the SMF software. The SMF developers themselve's also frequent the forum.

shock_

  • Wiki Contributor
  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 354
  • In Brisbane, it's GMT+10 all the time.
Make sense of your MySQL slow query log file by installing percona toolkit and running pt-query-digest on the mysql-slow.log.  pt-query-digest takes --since= and --until= so you can limit it to slow queries that occurred at an interval you care about, e.g. midnight to 1am.  If a part of the problem is indeed iowait (which stands to reason if other services are slow during the worst of it, your CPU load is probably enormous) it's presumably going to either be a lot of table writes or perhaps large temp tables on disk due to subqueries. pt-query-digest will show you that plain enough.

Le Chuck

  • I want a new quote!
  • Wiki Contributor
  • Trade Count: (+6)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 5297
  • I may have forgotten to whom I am talking.
I bet if we could like threads and had +fu this wouldn't have happened. 

pbj

  • Trade Count: (+3)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 5898
  • Obey.
    • The Chris Burke Band
Have you tried turning it off and back on?


Slippyblade

  • Trade Count: (+2)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 2954
  • And to the death god we say, "Not today!"
Have you tried turning it off and back on?

For great justice!  Seriously though...  have you?

lilshawn

  • Trade Count: (+3)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 5632
  • I break stuff...then fix it...sometimes
Have you tried turning it off and back on?

For great justice!  Seriously though...  have you?

have you tried forcing an unexpected reboot?

ed12

  • Trade Count: (-1)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 3886
  • it is what it is..."Nobody Said It Was Easy"....
let me see
glue it down once like felt it sticks
then yank it back out and fource it back in ?
is that about right ???
it is a simple rule,if it is not broke DO NOT FIX IT...
so with that in hand we jumped to a new ?? rev we can not fix ? and need to wait ? huh ?

ed
Shipping something from the U.S. to Canada for repair/exchange?  Please use USPS to avoid (additional?/excessive?) shipping charges.  PM me if you have any questions.

ed12

  • Trade Count: (-1)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 3886
  • it is what it is..."Nobody Said It Was Easy"....
these errors right here point to the weakness of ubt...

    SMF has too many scheduled tasks set to fire at 00:00 that don't need to be running at exactly that time - see http://wiki.simplemachines.org/smf/SMF2.0:Scheduled_Tasks
    Those tasks are possibly creating a queue on MySQL.
    PHP is throwing "MySQL has gone away" errors into an appropriate log once calls out to the DB exceed its timeout
    Web server is slow to respond while it still trusts MySQL might eventually have an answer for it, then starts sending quick 500s once it's sure it won't be back

it cannot deal with the load..never was intended for that..
and that is a period
time to move to debain again and run some nice ice

ed
Shipping something from the U.S. to Canada for repair/exchange?  Please use USPS to avoid (additional?/excessive?) shipping charges.  PM me if you have any questions.

Malenko

  • KNEEL BEFORE ZODlenko!
  • Trade Count: (+58)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 11977
  • 說好話 做好事 存好心
    • forum.arcadecontrols.com/index.php/topic,142404.msg1475162.html
Been a couple weeks, do anything I posted? Got any results?

Server was down around 2am with the error 500 in case you didn't know.
2014 Standup UCA Runner Up
2015 Restoration UCA Winner
2016 thomas_surles choice award

Latest project:
http://forum.arcadecontrols.com/index.php/topic,150902.0.html
RedheadKingPinball aka GingerBalls

Latest Restore?
http://forum.arcadecontrols.com/index.php/topic,152600.0.html
Killer Instinct Cab

ark_ader

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 5071
  • Present for you Grandfather...
I don't bother visiting this board that often now, with all these issues.  I'm sure the majority of lurkers have left too.

It is a shame really, but until someone gets a handle on this only the die-hards will frequent this site. 

Probably a good thing considering.

Hope you get this fixed soon Saint.
"I really do feel pity for those who go out of their way to perform a personal slight, than taking the effort in joining the conversation and creating some useful dialogue which would benefit the persons' perusal of this topic.  Yet where would we be without the persistant antagonist?"

UEDan

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 76
  • I want to build my own arcade controls!
You're running all that on 1 server? Dude....
What are the specs of that machine?
Why are you still hosting your own mail server when Gapps is low as $5/user/month. Or $4/user/month with AWS.
Why not look into a cloud compute platform? Are the operating costs that low at your current provider?

C'mon man! Lets do some migrations! I'm sure there are plenty of people here willing to help. *hint"hint*

ed12

  • Trade Count: (-1)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 3886
  • it is what it is..."Nobody Said It Was Easy"....
well 1 has to admit after 6 weeks of this, some is asleep at the switch.

ed
Shipping something from the U.S. to Canada for repair/exchange?  Please use USPS to avoid (additional?/excessive?) shipping charges.  PM me if you have any questions.

Malenko

  • KNEEL BEFORE ZODlenko!
  • Trade Count: (+58)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 11977
  • 說好話 做好事 存好心
    • forum.arcadecontrols.com/index.php/topic,142404.msg1475162.html
well 1 has to admit after 6 weeks of this, some is asleep at the switch.

ed

thread is 2 months old, issue has been going on for a couple of weeks longer (probably about 11 weeks total?). I tired to help but have gotten no response.
So much of us at Zapcon were going to post a ton of content but it wasnt worth the hassle of the site throwing out 500 errors when posting. I started a new project that I havent posted about because Im not ready for the headache/timesink of uploading pictures
2014 Standup UCA Runner Up
2015 Restoration UCA Winner
2016 thomas_surles choice award

Latest project:
http://forum.arcadecontrols.com/index.php/topic,150902.0.html
RedheadKingPinball aka GingerBalls

Latest Restore?
http://forum.arcadecontrols.com/index.php/topic,152600.0.html
Killer Instinct Cab

lilshawn

  • Trade Count: (+3)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 5632
  • I break stuff...then fix it...sometimes
i would often check in a half dozen or more times a day....

since this started happening... maybe once every 2 days.

not good.

Archive the site, and nuke it, and start over.

ed12

  • Trade Count: (-1)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 3886
  • it is what it is..."Nobody Said It Was Easy"....
Malenko  well put
shawn    same thoughts

time to either fix it saint or walk away..

ed
Shipping something from the U.S. to Canada for repair/exchange?  Please use USPS to avoid (additional?/excessive?) shipping charges.  PM me if you have any questions.

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 5915
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
OK, first, sorry for the lack of communication everyone. That's on me.

This evening Sirwoogie was able to look at the server and has an inkling as to what may be going on and how to resolve it. Over the next few days, if you see the site isn't available (as compared to being slow or throwing 500 errors) it's being worked on. I'll give an update when I can.
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade.com/
     saint@arcadecontrols.com

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 5915
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Looks like one main problem is a failing drive in the raid. Will be working with support to get it swapped out.
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade.com/
     saint@arcadecontrols.com

sirwoogie

  • Supreme Chancellor
  • Trade Count: (0)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 753
  • It's Hollyfeld in a Winnebago!
    • Build Your Own Arcade Controls
Yep, in all my time working on Linux hosts, I've not seen a server behave like this. What we end up getting is high service wait times on the drives. But, there is not enough load going on to cause that behavior. We've run this way for nearly 4 years without issue. I can't track this back to a particular upgrade or change made, so I'm somewhat stumped.

We're going to look at the drive replacement, but also doing some OS upgrade and see if that kicks out the bugs.

The worst case scenario is moving to another box. But that takes a lot of time of course, and that would be the last ditch option.

We'll try to keep the updates coming.

ed12

  • Trade Count: (-1)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 3886
  • it is what it is..."Nobody Said It Was Easy"....
what ever u were doing around 10:12 pm (my) time worked flawless
for 2mths have not been able to log in with (reg) screen.. has been walking around that now.?
if this is any help ?

ed
Shipping something from the U.S. to Canada for repair/exchange?  Please use USPS to avoid (additional?/excessive?) shipping charges.  PM me if you have any questions.

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 5915
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Please see update here: forum.arcadecontrols.com/index.php/topic,150884
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade.com/
     saint@arcadecontrols.com

  
 

Sitemap 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31