Main Restorations Software Audio/Jukebox/MP3 Everything Else Buy/Sell/Trade
Project Announcements Monitor/Video GroovyMAME Merit/JVL Touchscreen Meet Up Retail Vendors
Driving & Racing Woodworking Software Support Forums Consoles Project Arcade Reviews
Automated Projects Artwork Frontend Support Forums Pinball Forum Discussion Old Boards
Raspberry Pi & Dev Board controls.dat Linux Miscellaneous Arcade Wiki Discussion Old Archives
Lightguns Arcade1Up Try the site in https mode Site News

Unread posts | New Replies | Recent posts | Rules | Chatroom | Wiki | File Repository | RSS | Submit news

  

Author Topic: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server  (Read 14823 times)

0 Members and 1 Guest are viewing this topic.

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
*** SERVER ISSUES *** - we are currently experiencing a heavy load on the server that we are attempting to resolve. Slow loading may occur until we have this puzzled out.
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

Malenko

  • KNEEL BEFORE ZODlenko!
  • Trade Count: (+58)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 13999
  • Last login:March 26, 2024, 07:29:48 pm
  • Have you played with my GingerBalls?
    • forum.arcadecontrols.com/index.php/topic,142404.msg1475162.html
everything seems back to normal meow.
If you're replying to a troll you are part of the problem.
I also need to follow this advice. Ignore or report, don't reply.

WakiMiko

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 317
  • Last login:January 04, 2019, 03:17:46 pm
I'm getting the following pretty much every day now when I try to visit the forums. It seems to depend on the time of day, the site is usually fine after 15:00 CEST or so.

Malenko

  • KNEEL BEFORE ZODlenko!
  • Trade Count: (+58)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 13999
  • Last login:March 26, 2024, 07:29:48 pm
  • Have you played with my GingerBalls?
    • forum.arcadecontrols.com/index.php/topic,142404.msg1475162.html
Our top guys are on it.
If you're replying to a troll you are part of the problem.
I also need to follow this advice. Ignore or report, don't reply.

yotsuya

  • Trade Count: (+21)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 19955
  • Last login:Yesterday at 10:19:17 pm
  • 2014 UCA Winner, 2014, 2015, 2016 ZapCon Winner
    • forum.arcadecontrols.com/index.php/topic,137636.msg1420628.html
Why hasn't the ScottBot just jacked into the server with his KADE neural interface and diagnosed the issue?
***Build what you dig, bro. Build what you dig.***

vwalbridge

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 2082
  • Last login:July 31, 2021, 12:21:09 pm
  • Don't half-ass two things, whole-ass one thing.
I'll create a GUI interface using Visual Basic...see if I can track an IP address.

If you can read this, it means Photobucket's money grab ruined my signature photos.

lilshawn

  • Trade Count: (+3)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 7377
  • Last login:Yesterday at 11:41:53 pm
  • I break stuff...then fix it...sometimes
have to say anything user specific related is slow (new replies, my messages, etc) are all slow. the site itself is fine. (pulling up a specific post or opening the main trunk thread.)

something in the user database seems to be boggin things down.

acvieluf

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 956
  • Last login:March 13, 2024, 03:07:12 pm
  • I want to build my own arcade controls!
    • forum.arcadecontrols.com/index.php/topic,147793.0.html
I'll create a GUI interface using Visual Basic...see if I can track an IP address.



This is the greatest thing I've watched on YouTube in a while! Thank you.

dmckean

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 729
  • Last login:January 13, 2024, 08:50:41 pm
Any news?

Nothing has improved.

lilshawn

  • Trade Count: (+3)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 7377
  • Last login:Yesterday at 11:41:53 pm
  • I break stuff...then fix it...sometimes
boy she is sloooow today. took me several minutes to get to this thread and reply...and that was after i spent 5 and gave up on reading a message in my inbox.

is it time to nuke and start over?

« Last Edit: May 03, 2016, 11:41:20 am by lilshawn »

ed12

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 3972
  • Last login:March 31, 2018, 03:44:39 pm
  • it is what it is..."Nobody Said It Was Easy"....
well for me at exactley 12:am my time est. the server goes dead to the world,
has been going on for a few weeks,just like someone through the power switch.?
i agree time to nuke it and start over ?

ed
Shipping something from the U.S. to Canada for repair/exchange?  Please use USPS to avoid (additional?/excessive?) shipping charges.  PM me if you have any questions.

dmckean

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 729
  • Last login:January 13, 2024, 08:50:41 pm
It's ridiculous now.

harveybirdman

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 2538
  • Last login:February 19, 2024, 12:40:25 pm
  • SHMUP'EM
I hate to jump on the complaint train but dang Saint.... Let's figure it out, make everyone pitch in a dollar for posting privileges or something but just get it fixed.

harveybirdman

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 2538
  • Last login:February 19, 2024, 12:40:25 pm
  • SHMUP'EM
Did this thread get post helled? Nice..

yotsuya

  • Trade Count: (+21)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 19955
  • Last login:Yesterday at 10:19:17 pm
  • 2014 UCA Winner, 2014, 2015, 2016 ZapCon Winner
    • forum.arcadecontrols.com/index.php/topic,137636.msg1420628.html
Did this thread get post helled? Nice..
I can see it. Woah.
***Build what you dig, bro. Build what you dig.***

yotsuya

  • Trade Count: (+21)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 19955
  • Last login:Yesterday at 10:19:17 pm
  • 2014 UCA Winner, 2014, 2015, 2016 ZapCon Winner
    • forum.arcadecontrols.com/index.php/topic,137636.msg1420628.html


Last night on the iPad...
***Build what you dig, bro. Build what you dig.***

dmckean

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 729
  • Last login:January 13, 2024, 08:50:41 pm
It does that every night now between 10 and 11 PST.

vwalbridge

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 2082
  • Last login:July 31, 2021, 12:21:09 pm
  • Don't half-ass two things, whole-ass one thing.
It's been particularly slow the last 48 hours. In some cases, the forum simply never loads.

I'm not going to complain about something not working when it is FREE to the community.

However, I vote for the KLOV model:

Make everyone donate $1 or $2 to post. Not only does this generate money to keep the site hosted, but it will also help keep the spammers out. In turn, it might make the mods life easier too. They would only have to approve accounts that have already donated. Instead of approving every new post that comes in.

If you can read this, it means Photobucket's money grab ruined my signature photos.

Token

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 490
  • Last login:September 20, 2021, 03:45:43 pm
However, I vote for the KLOV model:

Make everyone donate $1 or $2 to post.

Someone start a gofundme for pbj.

dmckean

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 729
  • Last login:January 13, 2024, 08:50:41 pm
Most forums I'm a member of just periodically ask for donations to help pay for hosting and forum software updates.

lilshawn

  • Trade Count: (+3)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 7377
  • Last login:Yesterday at 11:41:53 pm
  • I break stuff...then fix it...sometimes
while I was waiting for my reply page to load...2 replies had posted

when did we update SMF to 2.0.11? is there something in that update that broke the forum?

some pages load fine *snap* like that, others..... ticktockticktock if at all.

Malenko

  • KNEEL BEFORE ZODlenko!
  • Trade Count: (+58)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 13999
  • Last login:March 26, 2024, 07:29:48 pm
  • Have you played with my GingerBalls?
    • forum.arcadecontrols.com/index.php/topic,142404.msg1475162.html
best guess, its MySQL related and probably more specifically, and issue with the database.
If you're replying to a troll you are part of the problem.
I also need to follow this advice. Ignore or report, don't reply.

pbj

  • Trade Count: (+4)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 10871
  • Last login:Today at 09:00:13 am
  • Obey.
    • The Chris Burke Band
I'm getting logged out fairly frequently.  Some fine 'network engineering' going on around here.  :-P


ed12

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 3972
  • Last login:March 31, 2018, 03:44:39 pm
  • it is what it is..."Nobody Said It Was Easy"....
well what i gat at 12:am on the dot is the pic yout's posted
error 500 internal server error ??
some one is on the switch!! fire them

ed
Shipping something from the U.S. to Canada for repair/exchange?  Please use USPS to avoid (additional?/excessive?) shipping charges.  PM me if you have any questions.

Nephasth

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 1376
  • Last login:March 11, 2024, 11:02:52 pm
  • Bitches love Centipede
%Bartop

Vigo

  • the Scourage of Carpathia
  • Global Moderator
  • Trade Count: (+24)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6417
  • Last login:March 21, 2024, 08:20:28 am
Saint, anything we can do to help with the recent issues?

shock_

  • Wiki Contributor
  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 354
  • Last login:January 30, 2020, 06:23:29 pm
  • In Brisbane, it's GMT+10 all the time.
I just saw a slowdown at about 3-4 mins past the hour, but it's come good again. It's just now a little past midnight US Eastern. In the next couple of hours if history is a guide, the site is going to slow again and then throw 500 errors.

Given the symptom (500 errors from a server running SMF) and the timing (daily, just after "midnight", in someone's US timezone at least) I'm willing to guess:

  • SMF has too many scheduled tasks set to fire at 00:00 that don't need to be running at exactly that time - see http://wiki.simplemachines.org/smf/SMF2.0:Scheduled_Tasks
  • Those tasks are possibly creating a queue on MySQL.
  • PHP is throwing "MySQL has gone away" errors into an appropriate log once calls out to the DB exceed its timeout
  • Web server is slow to respond while it still trusts MySQL might eventually have an answer for it, then starts sending quick 500s once it's sure it won't be back

Things I'd do

Check for active queries while the site is slow.
Assuming there's direct access to the database (e.g. command line or even PHPMyAdmin), send this query to the site database as the site database's user:

SHOW FULL PROCESSLIST\G

If the database user has the necessary privileges this will braindump the active db queries. Things to look for are the lines with "Time:" and "State:" -- if there are heaps of rows and they're slow, well, yeah, database bottleneck. The queries themselves might give a clue exactly WHAT it's busy with, which might lead to some ideas about shifting that activity to some other time etc.

Check for any spikes in site traffic.

Count rows per minute in the web server's access.log file, look for any spiky patterns for IP addresses or User Agents. Could be as simple as something set to crawl the site at the same time each night.

Make sure someone with SMF knowledge is involved

I'm talking very generally here, a lot of the true fixes are going to require knowledge of how to tune SMF to avoid being wasteful. Perhaps there was a spamfest a few months ago that led to hundreds of thousands of dummy registrations in the database, now slowing down daily admin functions due to sheer size. Maybe a feature was enabled that runs some daily background processing and nobody remembered to turn it off. I don't know, I'm a Drupal guy, not an SMF guy, but all commodity web apps have their quirks and all can be abused from time to time :)

RayB

  • I'm not wearing pants! HA!
  • Trade Count: (+4)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 11279
  • Last login:February 24, 2024, 12:00:13 am
  • There's my post
    • RayB.com
man it's SOOOO slow.
Speaking of donations, I agree with this idea. Saint, set up a Patreon. I'd pay a buck or two a month to keep this place running.

https://www.patreon.com/

NO MORE!!

Token

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 490
  • Last login:September 20, 2021, 03:45:43 pm
I'm all for throwing money at the problem, but that does no good if saint's TOP MEN don't know what the problem is.



It took five minutes of page refreshing to post this comment.

Nephasth

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 1376
  • Last login:March 11, 2024, 11:02:52 pm
  • Bitches love Centipede
%Bartop

Vigo

  • the Scourage of Carpathia
  • Global Moderator
  • Trade Count: (+24)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6417
  • Last login:March 21, 2024, 08:20:28 am

vwalbridge

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 2082
  • Last login:July 31, 2021, 12:21:09 pm
  • Don't half-ass two things, whole-ass one thing.
If you can read this, it means Photobucket's money grab ruined my signature photos.

yotsuya

  • Trade Count: (+21)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 19955
  • Last login:Yesterday at 10:19:17 pm
  • 2014 UCA Winner, 2014, 2015, 2016 ZapCon Winner
    • forum.arcadecontrols.com/index.php/topic,137636.msg1420628.html
Reminds me of a former IT director we had who swore unequivocally that if there were any issues, "we'd have boots on the ground in 15 minutes." Lots of minutes went by with no boots ever.
***Build what you dig, bro. Build what you dig.***

Phreakwars

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 591
  • Last login:March 17, 2024, 01:52:39 pm
  • Waka-Waka
    • forum.arcadecontrols.com/index.php/topic,146325.0.html
    • My Facebook Page
I'm gonna guess that the problem is in the SQL database. I had this issue when I was using velcom. I switched my server to Liquidweb and the problem went away. Also velcom did not update their php myadmin panel often enough and I was compromised by a mid eastern hacker.

I'd seriously consider changing servers. It seems the issue is constantly happening.

ark_ader

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 5645
  • Last login:March 02, 2019, 07:35:34 pm
  • I glow in the dark.
I would seriously look at the Burning Board software.  It is not free but they have excellent technical support.  I had their services when I ran CS gaming boards.

https://www.woltlab.com/wbb/

I would donate $10 or $20 a year for a new server platform.

You have to look at the benefits BYOAC has to offer, and what we have been receiving for free over the many years.
If I had only one wish, it would be for three more wishes.

shock_

  • Wiki Contributor
  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 354
  • Last login:January 30, 2020, 06:23:29 pm
  • In Brisbane, it's GMT+10 all the time.
Migrating an application of this size and age off Simple Machines Forum to another application like Burning Board is likely a thousand times more work and risk and cost, than getting the hosting stabilised. Saint's running the latest stable version of SMF which demonstrates someone at least is doing the right thing and keeping it up to date.

I'd suggest passing the hat around is not a bad idea, with the goal of engaging the support required to diagnose, then repair or migrate the hosting depending on what's actually failing.

Phreakwars

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 591
  • Last login:March 17, 2024, 01:52:39 pm
  • Waka-Waka
    • forum.arcadecontrols.com/index.php/topic,146325.0.html
    • My Facebook Page
If your worried about moving to another platform, just do like I did and install XAMPP on your local drive then copy the entire forum and database and convert from there. I know there are tons of converter tools for PhPBB, vBulletin, SMF, UBB, IPB, MyBB, XenForo, etc, etc.  SMF is nice because of the hook system when installing MODS, but it tends to be pretty taxing on the SQL. My personal preference would be PhPBB because it is so easy to work with. I've had no problem at all creating my own MOD's for it, even have a couple of them listed in the PhPBB MOD database. Not to mention there are tons of theme's available for it, and creating theme's is a breeze too.

One of the forums I owned had 500 plus sub forums (newsucks500.com now under new ownership and new platform), which, on most any other software, can really overheat those SQL database tables. PhPBB was able to handle it all just fine with no slowdown or crashing. Something I never would have attempted with SMF. In fact, the largest forum on the internet (gaiaonline.com), is nothing more then a heavily modified version of PhPBB.
« Last Edit: May 11, 2016, 03:57:22 am by Phreakwars »

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Hey everyone - I appreciate it, and I'm sorry I haven't been responsive here.

Thank you for the offers of financial donations. It is within my means to be able to afford it if needed, so your intentions are gratefully acknowledged but respectfully declined :)

We're not on a hosted platform, I lease the entire server.

I've asked for assistance from someone with a level of expertise in the OS/config on the server. If this person is able to help then we'll be in good shape. If not, my intent is to lease a new server (this one is now about 3 years old) and migrate everything over to a fresh installation. I plan to stick with SMF unless it turns out the issue is SMF in some fashion. I don't believe it is, but obviously if I knew what it was it'd be resolved by now :)

It's going to take some more time. The person I've asked for help has their own life/work/family responsibilities, so time on fixing the server has to compete with that. If they aren't able to assist, in a couple of weeks I will start working 4x10 work weeks with 3 day weekends, and that will give me the extra time to purchase and start configuring the new server.

I appreciate everyone's patience and concern. If all else fails I will throw up the white flag and ask for additional help from the forum. For security's sake I am cautious about who I give access to the server directly, because it has your information on it in some fashion, but the option is on the table if needed.

I did finally figure out why my script to restart sql/web-server was failing, and as of this morning it automatically restarted things rather than having to wait for me to manually log on and restart them. This means when I sleep in on the weekend the forum won't be down until 9-10 AM :)

Thanks everyone!
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
I just saw a slowdown at about 3-4 mins past the hour, but it's come good again. It's just now a little past midnight US Eastern. In the next couple of hours if history is a guide, the site is going to slow again and then throw 500 errors.

Given the symptom (500 errors from a server running SMF) and the timing (daily, just after "midnight", in someone's US timezone at least) I'm willing to guess:

  • SMF has too many scheduled tasks set to fire at 00:00 that don't need to be running at exactly that time - see http://wiki.simplemachines.org/smf/SMF2.0:Scheduled_Tasks
  • Those tasks are possibly creating a queue on MySQL.
  • PHP is throwing "MySQL has gone away" errors into an appropriate log once calls out to the DB exceed its timeout
  • Web server is slow to respond while it still trusts MySQL might eventually have an answer for it, then starts sending quick 500s once it's sure it won't be back

Things I'd do

Check for active queries while the site is slow.
Assuming there's direct access to the database (e.g. command line or even PHPMyAdmin), send this query to the site database as the site database's user:

SHOW FULL PROCESSLIST\G

If the database user has the necessary privileges this will braindump the active db queries. Things to look for are the lines with "Time:" and "State:" -- if there are heaps of rows and they're slow, well, yeah, database bottleneck. The queries themselves might give a clue exactly WHAT it's busy with, which might lead to some ideas about shifting that activity to some other time etc.

Check for any spikes in site traffic.

Count rows per minute in the web server's access.log file, look for any spiky patterns for IP addresses or User Agents. Could be as simple as something set to crawl the site at the same time each night.

Make sure someone with SMF knowledge is involved

I'm talking very generally here, a lot of the true fixes are going to require knowledge of how to tune SMF to avoid being wasteful. Perhaps there was a spamfest a few months ago that led to hundreds of thousands of dummy registrations in the database, now slowing down daily admin functions due to sheer size. Maybe a feature was enabled that runs some daily background processing and nobody remembered to turn it off. I don't know, I'm a Drupal guy, not an SMF guy, but all commodity web apps have their quirks and all can be abused from time to time :)

Thank you Shock, I'll dig into these. While it's slow all day, it *is* consistent on when it finally gives up the ghost each day.
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
For any Linux (particularly Ubuntu) gurus out there, here is what a random sampling of iotop shows while the forum is running. The various bits that are showing up in the top change about 1-2 times per second.  In a few minutes, I'll down the web server and mysql, and capture a few iotops while they aren't running for comparison. Unfortunately, I don't have a baseline to know what is and isn't normal.

--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Some additional info for consideration. In addition to the forum (relying on mysql) exhibiting slowness, I also run Squirrel mail (no reliance on mysql) on the server.

When the forum is slow, access to squirrel mail is slow. This makes me think it's not a mysql problem, or better said, not just a mysql problem, but rather whatever the problem is is also bogging down the server, as the iotop shows.

OK, killing the server now to get some non-mysql/web-server iotops for comparison.

--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Additional observations:

Mysql takes a long time to stop or to restart. Minutes.

When mysql is down but the web server is running, Squirrel mail runs fast and snappy, no lag whatsoever.

Once I restart sql, it takes a good 10-15 minutes or so before the forum can connect to the sql server
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
iotop snapshots when mysql was stopped but everything else was still running:

--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
A couple more from when mysql was stopped:

--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

pbj

  • Trade Count: (+4)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 10871
  • Last login:Today at 09:00:13 am
  • Obey.
    • The Chris Burke Band
But do you need money?  Apparently your users are gagging to give you money.  Please tell them you need money.


Phreakwars

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 591
  • Last login:March 17, 2024, 01:52:39 pm
  • Waka-Waka
    • forum.arcadecontrols.com/index.php/topic,146325.0.html
    • My Facebook Page
Are there other domains on the server that you are sharing, or is it a dedicated server? You can experience problems of the like when you are sharing with several domains. That's just kind of the nature of it when it comes to some of these storage space companies. You could try asking to be switched to a different rack that isn't being utilized as much by others.

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Are there other domains on the server that you are sharing, or is it a dedicated server? You can experience problems of the like when you are sharing with several domains. That's just kind of the nature of it when it comes to some of these storage space companies. You could try asking to be switched to a different rack that isn't being utilized as much by others.

Dedicated server.
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

shock_

  • Wiki Contributor
  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 354
  • Last login:January 30, 2020, 06:23:29 pm
  • In Brisbane, it's GMT+10 all the time.
Saint that does sound encouraging. I thought maybe there was some manual breakfast service restarts going on, that does fit the pattern of downtime :)

One other really good thing to do is install New Relic PHP extensions. Not too hard to get going, and the application monitoring will show you exactly what is going on over time, and allow for some deep diving into slow transactions for the first couple of weeks while they give you the full, expensive report package. Their free tier is still worth leaving up though too, once the trial expires. They also have a server health metrics thing that will graph things like your iotop and more.

The reporting in New Relic shouldn't have any personal info exposed and if you later feel like more public discussion is useful, well, we all love graphs :)

Malenko

  • KNEEL BEFORE ZODlenko!
  • Trade Count: (+58)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 13999
  • Last login:March 26, 2024, 07:29:48 pm
  • Have you played with my GingerBalls?
    • forum.arcadecontrols.com/index.php/topic,142404.msg1475162.html
Install atop

Run it. Press "i" to set the interval, I like 3 seconds.

See if the disks are actually busy and what the request latency is.  See if there's a bunch of CPU iowait. Are the CPUs busy? Is the box actively swapping to disk?

If the disk latency is higher than low single digit milliseconds but the disks don't look busy, check the SMART stats with smartctl. Look for a high number of reallocated sectors. This would indicate the drive is failing.

If all this looks okay and the drives are mostly idle, the box isn't swapping, the CPUs aren't busy and there's no high iowait, look at mysql.

Enable the slow query log: set global slow_query_log=on;
See where mysql is writing the slow log: show variables; look for slow_query_log_location or similar.
Find a query or queries that are logged slow, and run explain on it in mysql: explain <paste query here>;
Look for things that aren't using indexes, anything that says "null null all" is doing a full table scan and is going to be hella slow.

Let me know how all this crap looks and we'll go from there.
If you're replying to a troll you are part of the problem.
I also need to follow this advice. Ignore or report, don't reply.

Phreakwars

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 591
  • Last login:March 17, 2024, 01:52:39 pm
  • Waka-Waka
    • forum.arcadecontrols.com/index.php/topic,146325.0.html
    • My Facebook Page
Would also like to add, try asking one of the guys on The Admin Zone.

https://theadminzone.com/

Been a member there for years, they have some of the best guys who can also help solve the issue. You'll be able to find people who know the in's and out's of every aspect of running a forum and also know all about the SMF software. The SMF developers themselve's also frequent the forum.

shock_

  • Wiki Contributor
  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 354
  • Last login:January 30, 2020, 06:23:29 pm
  • In Brisbane, it's GMT+10 all the time.
Make sense of your MySQL slow query log file by installing percona toolkit and running pt-query-digest on the mysql-slow.log.  pt-query-digest takes --since= and --until= so you can limit it to slow queries that occurred at an interval you care about, e.g. midnight to 1am.  If a part of the problem is indeed iowait (which stands to reason if other services are slow during the worst of it, your CPU load is probably enormous) it's presumably going to either be a lot of table writes or perhaps large temp tables on disk due to subqueries. pt-query-digest will show you that plain enough.

Le Chuck

  • Saint, make a poll!
  • Wiki Contributor
  • Trade Count: (+6)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 5509
  • Last login:March 26, 2024, 08:00:17 pm
  • <insert personal text here>
I bet if we could like threads and had +fu this wouldn't have happened. 

pbj

  • Trade Count: (+4)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 10871
  • Last login:Today at 09:00:13 am
  • Obey.
    • The Chris Burke Band
Have you tried turning it off and back on?


Slippyblade

  • Trade Count: (+2)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 3167
  • Last login:November 16, 2023, 11:39:51 am
  • And to the death god we say, "Not today!"
Have you tried turning it off and back on?

For great justice!  Seriously though...  have you?

lilshawn

  • Trade Count: (+3)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 7377
  • Last login:Yesterday at 11:41:53 pm
  • I break stuff...then fix it...sometimes
Have you tried turning it off and back on?

For great justice!  Seriously though...  have you?

have you tried forcing an unexpected reboot?

ed12

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 3972
  • Last login:March 31, 2018, 03:44:39 pm
  • it is what it is..."Nobody Said It Was Easy"....
let me see
glue it down once like felt it sticks
then yank it back out and fource it back in ?
is that about right ???
it is a simple rule,if it is not broke DO NOT FIX IT...
so with that in hand we jumped to a new ?? rev we can not fix ? and need to wait ? huh ?

ed
Shipping something from the U.S. to Canada for repair/exchange?  Please use USPS to avoid (additional?/excessive?) shipping charges.  PM me if you have any questions.

ed12

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 3972
  • Last login:March 31, 2018, 03:44:39 pm
  • it is what it is..."Nobody Said It Was Easy"....
these errors right here point to the weakness of ubt...

    SMF has too many scheduled tasks set to fire at 00:00 that don't need to be running at exactly that time - see http://wiki.simplemachines.org/smf/SMF2.0:Scheduled_Tasks
    Those tasks are possibly creating a queue on MySQL.
    PHP is throwing "MySQL has gone away" errors into an appropriate log once calls out to the DB exceed its timeout
    Web server is slow to respond while it still trusts MySQL might eventually have an answer for it, then starts sending quick 500s once it's sure it won't be back

it cannot deal with the load..never was intended for that..
and that is a period
time to move to debain again and run some nice ice

ed
Shipping something from the U.S. to Canada for repair/exchange?  Please use USPS to avoid (additional?/excessive?) shipping charges.  PM me if you have any questions.

Malenko

  • KNEEL BEFORE ZODlenko!
  • Trade Count: (+58)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 13999
  • Last login:March 26, 2024, 07:29:48 pm
  • Have you played with my GingerBalls?
    • forum.arcadecontrols.com/index.php/topic,142404.msg1475162.html
Been a couple weeks, do anything I posted? Got any results?

Server was down around 2am with the error 500 in case you didn't know.
If you're replying to a troll you are part of the problem.
I also need to follow this advice. Ignore or report, don't reply.

ark_ader

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 5645
  • Last login:March 02, 2019, 07:35:34 pm
  • I glow in the dark.
I don't bother visiting this board that often now, with all these issues.  I'm sure the majority of lurkers have left too.

It is a shame really, but until someone gets a handle on this only the die-hards will frequent this site. 

Probably a good thing considering.

Hope you get this fixed soon Saint.
If I had only one wish, it would be for three more wishes.

UEDan

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 104
  • Last login:November 10, 2019, 05:54:08 am
  • I want to build my own arcade controls!
You're running all that on 1 server? Dude....
What are the specs of that machine?
Why are you still hosting your own mail server when Gapps is low as $5/user/month. Or $4/user/month with AWS.
Why not look into a cloud compute platform? Are the operating costs that low at your current provider?

C'mon man! Lets do some migrations! I'm sure there are plenty of people here willing to help. *hint"hint*

ed12

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 3972
  • Last login:March 31, 2018, 03:44:39 pm
  • it is what it is..."Nobody Said It Was Easy"....
well 1 has to admit after 6 weeks of this, some is asleep at the switch.

ed
Shipping something from the U.S. to Canada for repair/exchange?  Please use USPS to avoid (additional?/excessive?) shipping charges.  PM me if you have any questions.

Malenko

  • KNEEL BEFORE ZODlenko!
  • Trade Count: (+58)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 13999
  • Last login:March 26, 2024, 07:29:48 pm
  • Have you played with my GingerBalls?
    • forum.arcadecontrols.com/index.php/topic,142404.msg1475162.html
well 1 has to admit after 6 weeks of this, some is asleep at the switch.

ed

thread is 2 months old, issue has been going on for a couple of weeks longer (probably about 11 weeks total?). I tired to help but have gotten no response.
So much of us at Zapcon were going to post a ton of content but it wasnt worth the hassle of the site throwing out 500 errors when posting. I started a new project that I havent posted about because Im not ready for the headache/timesink of uploading pictures
If you're replying to a troll you are part of the problem.
I also need to follow this advice. Ignore or report, don't reply.

lilshawn

  • Trade Count: (+3)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 7377
  • Last login:Yesterday at 11:41:53 pm
  • I break stuff...then fix it...sometimes
i would often check in a half dozen or more times a day....

since this started happening... maybe once every 2 days.

not good.

Archive the site, and nuke it, and start over.

ed12

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 3972
  • Last login:March 31, 2018, 03:44:39 pm
  • it is what it is..."Nobody Said It Was Easy"....
Malenko  well put
shawn    same thoughts

time to either fix it saint or walk away..

ed
Shipping something from the U.S. to Canada for repair/exchange?  Please use USPS to avoid (additional?/excessive?) shipping charges.  PM me if you have any questions.

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
OK, first, sorry for the lack of communication everyone. That's on me.

This evening Sirwoogie was able to look at the server and has an inkling as to what may be going on and how to resolve it. Over the next few days, if you see the site isn't available (as compared to being slow or throwing 500 errors) it's being worked on. I'll give an update when I can.
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Looks like one main problem is a failing drive in the raid. Will be working with support to get it swapped out.
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

sirwoogie

  • Supreme Chancellor
  • Trade Count: (0)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 753
  • Last login:August 16, 2022, 12:37:22 pm
  • It's Hollyfeld in a Winnebago!
    • Build Your Own Arcade Controls
Yep, in all my time working on Linux hosts, I've not seen a server behave like this. What we end up getting is high service wait times on the drives. But, there is not enough load going on to cause that behavior. We've run this way for nearly 4 years without issue. I can't track this back to a particular upgrade or change made, so I'm somewhat stumped.

We're going to look at the drive replacement, but also doing some OS upgrade and see if that kicks out the bugs.

The worst case scenario is moving to another box. But that takes a lot of time of course, and that would be the last ditch option.

We'll try to keep the updates coming.

ed12

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 3972
  • Last login:March 31, 2018, 03:44:39 pm
  • it is what it is..."Nobody Said It Was Easy"....
what ever u were doing around 10:12 pm (my) time worked flawless
for 2mths have not been able to log in with (reg) screen.. has been walking around that now.?
if this is any help ?

ed
Shipping something from the U.S. to Canada for repair/exchange?  Please use USPS to avoid (additional?/excessive?) shipping charges.  PM me if you have any questions.

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Please see update here: forum.arcadecontrols.com/index.php/topic,150884
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com