Main Restorations Software Audio/Jukebox/MP3 Everything Else Buy/Sell/Trade
Project Announcements Monitor/Video GroovyMAME Merit/JVL Touchscreen Meet Up Retail Vendors
Driving & Racing Woodworking Software Support Forums Consoles Project Arcade Reviews
Automated Projects Artwork Frontend Support Forums Pinball Forum Discussion Old Boards
Raspberry Pi & Dev Board controls.dat Linux Miscellaneous Arcade Wiki Discussion Old Archives
Lightguns Arcade1Up Try the site in https mode Site News

Unread posts | New Replies | Recent posts | Rules | Chatroom | Wiki | File Repository | RSS | Submit news

  

Author Topic: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server  (Read 14831 times)

0 Members and 1 Guest are viewing this topic.

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Some additional info for consideration. In addition to the forum (relying on mysql) exhibiting slowness, I also run Squirrel mail (no reliance on mysql) on the server.

When the forum is slow, access to squirrel mail is slow. This makes me think it's not a mysql problem, or better said, not just a mysql problem, but rather whatever the problem is is also bogging down the server, as the iotop shows.

OK, killing the server now to get some non-mysql/web-server iotops for comparison.

--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Additional observations:

Mysql takes a long time to stop or to restart. Minutes.

When mysql is down but the web server is running, Squirrel mail runs fast and snappy, no lag whatsoever.

Once I restart sql, it takes a good 10-15 minutes or so before the forum can connect to the sql server
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
iotop snapshots when mysql was stopped but everything else was still running:

--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
A couple more from when mysql was stopped:

--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

pbj

  • Trade Count: (+4)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 10871
  • Last login:Today at 02:51:51 pm
  • Obey.
    • The Chris Burke Band
But do you need money?  Apparently your users are gagging to give you money.  Please tell them you need money.


Phreakwars

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 591
  • Last login:March 17, 2024, 01:52:39 pm
  • Waka-Waka
    • forum.arcadecontrols.com/index.php/topic,146325.0.html
    • My Facebook Page
Are there other domains on the server that you are sharing, or is it a dedicated server? You can experience problems of the like when you are sharing with several domains. That's just kind of the nature of it when it comes to some of these storage space companies. You could try asking to be switched to a different rack that isn't being utilized as much by others.

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Are there other domains on the server that you are sharing, or is it a dedicated server? You can experience problems of the like when you are sharing with several domains. That's just kind of the nature of it when it comes to some of these storage space companies. You could try asking to be switched to a different rack that isn't being utilized as much by others.

Dedicated server.
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

shock_

  • Wiki Contributor
  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 354
  • Last login:January 30, 2020, 06:23:29 pm
  • In Brisbane, it's GMT+10 all the time.
Saint that does sound encouraging. I thought maybe there was some manual breakfast service restarts going on, that does fit the pattern of downtime :)

One other really good thing to do is install New Relic PHP extensions. Not too hard to get going, and the application monitoring will show you exactly what is going on over time, and allow for some deep diving into slow transactions for the first couple of weeks while they give you the full, expensive report package. Their free tier is still worth leaving up though too, once the trial expires. They also have a server health metrics thing that will graph things like your iotop and more.

The reporting in New Relic shouldn't have any personal info exposed and if you later feel like more public discussion is useful, well, we all love graphs :)

Malenko

  • KNEEL BEFORE ZODlenko!
  • Trade Count: (+58)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 13999
  • Last login:March 26, 2024, 07:29:48 pm
  • Have you played with my GingerBalls?
    • forum.arcadecontrols.com/index.php/topic,142404.msg1475162.html
Install atop

Run it. Press "i" to set the interval, I like 3 seconds.

See if the disks are actually busy and what the request latency is.  See if there's a bunch of CPU iowait. Are the CPUs busy? Is the box actively swapping to disk?

If the disk latency is higher than low single digit milliseconds but the disks don't look busy, check the SMART stats with smartctl. Look for a high number of reallocated sectors. This would indicate the drive is failing.

If all this looks okay and the drives are mostly idle, the box isn't swapping, the CPUs aren't busy and there's no high iowait, look at mysql.

Enable the slow query log: set global slow_query_log=on;
See where mysql is writing the slow log: show variables; look for slow_query_log_location or similar.
Find a query or queries that are logged slow, and run explain on it in mysql: explain <paste query here>;
Look for things that aren't using indexes, anything that says "null null all" is doing a full table scan and is going to be hella slow.

Let me know how all this crap looks and we'll go from there.
If you're replying to a troll you are part of the problem.
I also need to follow this advice. Ignore or report, don't reply.

Phreakwars

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 591
  • Last login:March 17, 2024, 01:52:39 pm
  • Waka-Waka
    • forum.arcadecontrols.com/index.php/topic,146325.0.html
    • My Facebook Page
Would also like to add, try asking one of the guys on The Admin Zone.

https://theadminzone.com/

Been a member there for years, they have some of the best guys who can also help solve the issue. You'll be able to find people who know the in's and out's of every aspect of running a forum and also know all about the SMF software. The SMF developers themselve's also frequent the forum.

shock_

  • Wiki Contributor
  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 354
  • Last login:January 30, 2020, 06:23:29 pm
  • In Brisbane, it's GMT+10 all the time.
Make sense of your MySQL slow query log file by installing percona toolkit and running pt-query-digest on the mysql-slow.log.  pt-query-digest takes --since= and --until= so you can limit it to slow queries that occurred at an interval you care about, e.g. midnight to 1am.  If a part of the problem is indeed iowait (which stands to reason if other services are slow during the worst of it, your CPU load is probably enormous) it's presumably going to either be a lot of table writes or perhaps large temp tables on disk due to subqueries. pt-query-digest will show you that plain enough.

Le Chuck

  • Saint, make a poll!
  • Wiki Contributor
  • Trade Count: (+6)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 5509
  • Last login:March 26, 2024, 08:00:17 pm
  • <insert personal text here>
I bet if we could like threads and had +fu this wouldn't have happened. 

pbj

  • Trade Count: (+4)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 10871
  • Last login:Today at 02:51:51 pm
  • Obey.
    • The Chris Burke Band
Have you tried turning it off and back on?


Slippyblade

  • Trade Count: (+2)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 3167
  • Last login:November 16, 2023, 11:39:51 am
  • And to the death god we say, "Not today!"
Have you tried turning it off and back on?

For great justice!  Seriously though...  have you?

lilshawn

  • Trade Count: (+3)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 7377
  • Last login:Today at 02:55:20 pm
  • I break stuff...then fix it...sometimes
Have you tried turning it off and back on?

For great justice!  Seriously though...  have you?

have you tried forcing an unexpected reboot?

ed12

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 3972
  • Last login:March 31, 2018, 03:44:39 pm
  • it is what it is..."Nobody Said It Was Easy"....
let me see
glue it down once like felt it sticks
then yank it back out and fource it back in ?
is that about right ???
it is a simple rule,if it is not broke DO NOT FIX IT...
so with that in hand we jumped to a new ?? rev we can not fix ? and need to wait ? huh ?

ed
Shipping something from the U.S. to Canada for repair/exchange?  Please use USPS to avoid (additional?/excessive?) shipping charges.  PM me if you have any questions.

ed12

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 3972
  • Last login:March 31, 2018, 03:44:39 pm
  • it is what it is..."Nobody Said It Was Easy"....
these errors right here point to the weakness of ubt...

    SMF has too many scheduled tasks set to fire at 00:00 that don't need to be running at exactly that time - see http://wiki.simplemachines.org/smf/SMF2.0:Scheduled_Tasks
    Those tasks are possibly creating a queue on MySQL.
    PHP is throwing "MySQL has gone away" errors into an appropriate log once calls out to the DB exceed its timeout
    Web server is slow to respond while it still trusts MySQL might eventually have an answer for it, then starts sending quick 500s once it's sure it won't be back

it cannot deal with the load..never was intended for that..
and that is a period
time to move to debain again and run some nice ice

ed
Shipping something from the U.S. to Canada for repair/exchange?  Please use USPS to avoid (additional?/excessive?) shipping charges.  PM me if you have any questions.

Malenko

  • KNEEL BEFORE ZODlenko!
  • Trade Count: (+58)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 13999
  • Last login:March 26, 2024, 07:29:48 pm
  • Have you played with my GingerBalls?
    • forum.arcadecontrols.com/index.php/topic,142404.msg1475162.html
Been a couple weeks, do anything I posted? Got any results?

Server was down around 2am with the error 500 in case you didn't know.
If you're replying to a troll you are part of the problem.
I also need to follow this advice. Ignore or report, don't reply.

ark_ader

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 5645
  • Last login:March 02, 2019, 07:35:34 pm
  • I glow in the dark.
I don't bother visiting this board that often now, with all these issues.  I'm sure the majority of lurkers have left too.

It is a shame really, but until someone gets a handle on this only the die-hards will frequent this site. 

Probably a good thing considering.

Hope you get this fixed soon Saint.
If I had only one wish, it would be for three more wishes.

UEDan

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 104
  • Last login:November 10, 2019, 05:54:08 am
  • I want to build my own arcade controls!
You're running all that on 1 server? Dude....
What are the specs of that machine?
Why are you still hosting your own mail server when Gapps is low as $5/user/month. Or $4/user/month with AWS.
Why not look into a cloud compute platform? Are the operating costs that low at your current provider?

C'mon man! Lets do some migrations! I'm sure there are plenty of people here willing to help. *hint"hint*

ed12

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 3972
  • Last login:March 31, 2018, 03:44:39 pm
  • it is what it is..."Nobody Said It Was Easy"....
well 1 has to admit after 6 weeks of this, some is asleep at the switch.

ed
Shipping something from the U.S. to Canada for repair/exchange?  Please use USPS to avoid (additional?/excessive?) shipping charges.  PM me if you have any questions.

Malenko

  • KNEEL BEFORE ZODlenko!
  • Trade Count: (+58)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 13999
  • Last login:March 26, 2024, 07:29:48 pm
  • Have you played with my GingerBalls?
    • forum.arcadecontrols.com/index.php/topic,142404.msg1475162.html
well 1 has to admit after 6 weeks of this, some is asleep at the switch.

ed

thread is 2 months old, issue has been going on for a couple of weeks longer (probably about 11 weeks total?). I tired to help but have gotten no response.
So much of us at Zapcon were going to post a ton of content but it wasnt worth the hassle of the site throwing out 500 errors when posting. I started a new project that I havent posted about because Im not ready for the headache/timesink of uploading pictures
If you're replying to a troll you are part of the problem.
I also need to follow this advice. Ignore or report, don't reply.

lilshawn

  • Trade Count: (+3)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 7377
  • Last login:Today at 02:55:20 pm
  • I break stuff...then fix it...sometimes
i would often check in a half dozen or more times a day....

since this started happening... maybe once every 2 days.

not good.

Archive the site, and nuke it, and start over.

ed12

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 3972
  • Last login:March 31, 2018, 03:44:39 pm
  • it is what it is..."Nobody Said It Was Easy"....
Malenko  well put
shawn    same thoughts

time to either fix it saint or walk away..

ed
Shipping something from the U.S. to Canada for repair/exchange?  Please use USPS to avoid (additional?/excessive?) shipping charges.  PM me if you have any questions.

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
OK, first, sorry for the lack of communication everyone. That's on me.

This evening Sirwoogie was able to look at the server and has an inkling as to what may be going on and how to resolve it. Over the next few days, if you see the site isn't available (as compared to being slow or throwing 500 errors) it's being worked on. I'll give an update when I can.
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Looks like one main problem is a failing drive in the raid. Will be working with support to get it swapped out.
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

sirwoogie

  • Supreme Chancellor
  • Trade Count: (0)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 753
  • Last login:August 16, 2022, 12:37:22 pm
  • It's Hollyfeld in a Winnebago!
    • Build Your Own Arcade Controls
Yep, in all my time working on Linux hosts, I've not seen a server behave like this. What we end up getting is high service wait times on the drives. But, there is not enough load going on to cause that behavior. We've run this way for nearly 4 years without issue. I can't track this back to a particular upgrade or change made, so I'm somewhat stumped.

We're going to look at the drive replacement, but also doing some OS upgrade and see if that kicks out the bugs.

The worst case scenario is moving to another box. But that takes a lot of time of course, and that would be the last ditch option.

We'll try to keep the updates coming.

ed12

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 3972
  • Last login:March 31, 2018, 03:44:39 pm
  • it is what it is..."Nobody Said It Was Easy"....
what ever u were doing around 10:12 pm (my) time worked flawless
for 2mths have not been able to log in with (reg) screen.. has been walking around that now.?
if this is any help ?

ed
Shipping something from the U.S. to Canada for repair/exchange?  Please use USPS to avoid (additional?/excessive?) shipping charges.  PM me if you have any questions.

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Please see update here: forum.arcadecontrols.com/index.php/topic,150884
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com