Main Restorations Software Audio/Jukebox/MP3 Everything Else Buy/Sell/Trade
Project Announcements Monitor/Video GroovyMAME Merit/JVL Touchscreen Meet Up Retail Vendors
Driving & Racing Woodworking Software Support Forums Consoles Project Arcade Reviews
Automated Projects Artwork Frontend Support Forums Pinball Forum Discussion Old Boards
Raspberry Pi & Dev Board controls.dat Linux Miscellaneous Arcade Wiki Discussion Old Archives
Site News

Unread posts | New Replies | Recent posts | Rules | Chatroom | Wiki | File Repository | RSS | Submit news


  

Author Topic: *** SERVER ISSUES *** - we are currently experiencing a heavy load on the server  (Read 3509 times)

0 Members and 1 Guest are viewing this topic.

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 5915
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
*** SERVER ISSUES *** - we are currently experiencing a heavy load on the server that we are attempting to resolve. Slow loading may occur until we have this puzzled out.
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade.com/
     saint@arcadecontrols.com

Malenko

  • KNEEL BEFORE ZODlenko!
  • Trade Count: (+58)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 11977
  • 說好話 做好事 存好心
    • forum.arcadecontrols.com/index.php/topic,142404.msg1475162.html
everything seems back to normal meow.
2014 Standup UCA Runner Up
2015 Restoration UCA Winner
2016 thomas_surles choice award

Latest project:
http://forum.arcadecontrols.com/index.php/topic,150902.0.html
RedheadKingPinball aka GingerBalls

Latest Restore?
http://forum.arcadecontrols.com/index.php/topic,152600.0.html
Killer Instinct Cab

WakiMiko

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 317
I'm getting the following pretty much every day now when I try to visit the forums. It seems to depend on the time of day, the site is usually fine after 15:00 CEST or so.

Malenko

  • KNEEL BEFORE ZODlenko!
  • Trade Count: (+58)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 11977
  • 說好話 做好事 存好心
    • forum.arcadecontrols.com/index.php/topic,142404.msg1475162.html
Our top guys are on it.
2014 Standup UCA Runner Up
2015 Restoration UCA Winner
2016 thomas_surles choice award

Latest project:
http://forum.arcadecontrols.com/index.php/topic,150902.0.html
RedheadKingPinball aka GingerBalls

Latest Restore?
http://forum.arcadecontrols.com/index.php/topic,152600.0.html
Killer Instinct Cab

yotsuya

  • Trade Count: (+21)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 16014
  • 2014 UCA Winner, 2014, 2015, 2016 ZapCon Winner
    • forum.arcadecontrols.com/index.php/topic,137636.msg1420628.html
Why hasn't the ScottBot just jacked into the server with his KADE neural interface and diagnosed the issue?
***Build what you dig, bro. Build what you dig.***

vwalbridge

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 1920
  • Don't half-ass two things, whole-ass one thing.
I'll create a GUI interface using Visual Basic...see if I can track an IP address.


lilshawn

  • Trade Count: (+3)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 5625
  • I break stuff...then fix it...sometimes
have to say anything user specific related is slow (new replies, my messages, etc) are all slow. the site itself is fine. (pulling up a specific post or opening the main trunk thread.)

something in the user database seems to be boggin things down.

acvieluf

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 535
  • I want to build my own arcade controls!
    • forum.arcadecontrols.com/index.php/topic,147793.0.html
I'll create a GUI interface using Visual Basic...see if I can track an IP address.



This is the greatest thing I've watched on YouTube in a while! Thank you.

dmckean

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 555
Any news?

Nothing has improved.

lilshawn

  • Trade Count: (+3)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 5625
  • I break stuff...then fix it...sometimes
boy she is sloooow today. took me several minutes to get to this thread and reply...and that was after i spent 5 and gave up on reading a message in my inbox.

is it time to nuke and start over?

http://i.imgur.com/j7wfDMu.jpg
« Last Edit: May 03, 2016, 11:41:20 am by lilshawn »

ed12

  • Trade Count: (-1)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 3884
  • it is what it is..."Nobody Said It Was Easy"....
well for me at exactley 12:am my time est. the server goes dead to the world,
has been going on for a few weeks,just like someone through the power switch.?
i agree time to nuke it and start over ?

ed
Shipping something from the U.S. to Canada for repair/exchange?  Please use USPS to avoid (additional?/excessive?) shipping charges.  PM me if you have any questions.

dmckean

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 555
It's ridiculous now.

harveybirdman

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 2384
  • SHMUP'EM
I hate to jump on the complaint train but dang Saint.... Let's figure it out, make everyone pitch in a dollar for posting privileges or something but just get it fixed.

harveybirdman

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 2384
  • SHMUP'EM
Did this thread get post helled? Nice..

yotsuya

  • Trade Count: (+21)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 16014
  • 2014 UCA Winner, 2014, 2015, 2016 ZapCon Winner
    • forum.arcadecontrols.com/index.php/topic,137636.msg1420628.html
Did this thread get post helled? Nice..
I can see it. Woah.
***Build what you dig, bro. Build what you dig.***

yotsuya

  • Trade Count: (+21)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 16014
  • 2014 UCA Winner, 2014, 2015, 2016 ZapCon Winner
    • forum.arcadecontrols.com/index.php/topic,137636.msg1420628.html


Last night on the iPad...
***Build what you dig, bro. Build what you dig.***

dmckean

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 555
It does that every night now between 10 and 11 PST.

vwalbridge

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 1920
  • Don't half-ass two things, whole-ass one thing.
It's been particularly slow the last 48 hours. In some cases, the forum simply never loads.

I'm not going to complain about something not working when it is FREE to the community.

However, I vote for the KLOV model:

Make everyone donate $1 or $2 to post. Not only does this generate money to keep the site hosted, but it will also help keep the spammers out. In turn, it might make the mods life easier too. They would only have to approve accounts that have already donated. Instead of approving every new post that comes in.


Token

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 490
However, I vote for the KLOV model:

Make everyone donate $1 or $2 to post.

Someone start a gofundme for pbj.

dmckean

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 555
Most forums I'm a member of just periodically ask for donations to help pay for hosting and forum software updates.

lilshawn

  • Trade Count: (+3)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 5625
  • I break stuff...then fix it...sometimes
while I was waiting for my reply page to load...2 replies had posted

when did we update SMF to 2.0.11? is there something in that update that broke the forum?

some pages load fine *snap* like that, others..... ticktockticktock if at all.

Malenko

  • KNEEL BEFORE ZODlenko!
  • Trade Count: (+58)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 11977
  • 說好話 做好事 存好心
    • forum.arcadecontrols.com/index.php/topic,142404.msg1475162.html
best guess, its MySQL related and probably more specifically, and issue with the database.
2014 Standup UCA Runner Up
2015 Restoration UCA Winner
2016 thomas_surles choice award

Latest project:
http://forum.arcadecontrols.com/index.php/topic,150902.0.html
RedheadKingPinball aka GingerBalls

Latest Restore?
http://forum.arcadecontrols.com/index.php/topic,152600.0.html
Killer Instinct Cab

pbj

  • Trade Count: (+3)
  • Full Member
  • ***
  • Online Online
  • Posts: 5890
  • Obey.
    • The Chris Burke Band
I'm getting logged out fairly frequently.  Some fine 'network engineering' going on around here.  :-P


ed12

  • Trade Count: (-1)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 3884
  • it is what it is..."Nobody Said It Was Easy"....
well what i gat at 12:am on the dot is the pic yout's posted
error 500 internal server error ??
some one is on the switch!! fire them

ed
Shipping something from the U.S. to Canada for repair/exchange?  Please use USPS to avoid (additional?/excessive?) shipping charges.  PM me if you have any questions.

Nephasth

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 384
  • Bitches love Centipede
%Bartop

Vigo

  • the Scourage of Carpathia
  • Trade Count: (+24)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 5666
  • The Soap Water of Moldavia
Saint, anything we can do to help with the recent issues?

shock_

  • Wiki Contributor
  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 354
  • In Brisbane, it's GMT+10 all the time.
I just saw a slowdown at about 3-4 mins past the hour, but it's come good again. It's just now a little past midnight US Eastern. In the next couple of hours if history is a guide, the site is going to slow again and then throw 500 errors.

Given the symptom (500 errors from a server running SMF) and the timing (daily, just after "midnight", in someone's US timezone at least) I'm willing to guess:

  • SMF has too many scheduled tasks set to fire at 00:00 that don't need to be running at exactly that time - see http://wiki.simplemachines.org/smf/SMF2.0:Scheduled_Tasks
  • Those tasks are possibly creating a queue on MySQL.
  • PHP is throwing "MySQL has gone away" errors into an appropriate log once calls out to the DB exceed its timeout
  • Web server is slow to respond while it still trusts MySQL might eventually have an answer for it, then starts sending quick 500s once it's sure it won't be back

Things I'd do

Check for active queries while the site is slow.
Assuming there's direct access to the database (e.g. command line or even PHPMyAdmin), send this query to the site database as the site database's user:

SHOW FULL PROCESSLIST\G

If the database user has the necessary privileges this will braindump the active db queries. Things to look for are the lines with "Time:" and "State:" -- if there are heaps of rows and they're slow, well, yeah, database bottleneck. The queries themselves might give a clue exactly WHAT it's busy with, which might lead to some ideas about shifting that activity to some other time etc.

Check for any spikes in site traffic.

Count rows per minute in the web server's access.log file, look for any spiky patterns for IP addresses or User Agents. Could be as simple as something set to crawl the site at the same time each night.

Make sure someone with SMF knowledge is involved

I'm talking very generally here, a lot of the true fixes are going to require knowledge of how to tune SMF to avoid being wasteful. Perhaps there was a spamfest a few months ago that led to hundreds of thousands of dummy registrations in the database, now slowing down daily admin functions due to sheer size. Maybe a feature was enabled that runs some daily background processing and nobody remembered to turn it off. I don't know, I'm a Drupal guy, not an SMF guy, but all commodity web apps have their quirks and all can be abused from time to time :)

RayB

  • I'm not wearing pants! HA!
  • Trade Count: (+4)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 11242
  • There's my post
    • RayB.com
man it's SOOOO slow.
Speaking of donations, I agree with this idea. Saint, set up a Patreon. I'd pay a buck or two a month to keep this place running.

https://www.patreon.com/

NO MORE!!

Token

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 490
I'm all for throwing money at the problem, but that does no good if saint's TOP MEN don't know what the problem is.



It took five minutes of page refreshing to post this comment.

Nephasth

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 384
  • Bitches love Centipede
%Bartop

Vigo

  • the Scourage of Carpathia
  • Trade Count: (+24)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 5666
  • The Soap Water of Moldavia

vwalbridge

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 1920
  • Don't half-ass two things, whole-ass one thing.

yotsuya

  • Trade Count: (+21)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 16014
  • 2014 UCA Winner, 2014, 2015, 2016 ZapCon Winner
    • forum.arcadecontrols.com/index.php/topic,137636.msg1420628.html
Reminds me of a former IT director we had who swore unequivocally that if there were any issues, "we'd have boots on the ground in 15 minutes." Lots of minutes went by with no boots ever.
***Build what you dig, bro. Build what you dig.***

Phreakwars

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 280
  • Waka-Waka
    • forum.arcadecontrols.com/index.php/topic,146325.0.html
    • My Facebook Page
I'm gonna guess that the problem is in the SQL database. I had this issue when I was using velcom. I switched my server to Liquidweb and the problem went away. Also velcom did not update their php myadmin panel often enough and I was compromised by a mid eastern hacker.

I'd seriously consider changing servers. It seems the issue is constantly happening.

ark_ader

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 5063
  • Present for you Grandfather...
I would seriously look at the Burning Board software.  It is not free but they have excellent technical support.  I had their services when I ran CS gaming boards.

https://www.woltlab.com/wbb/

I would donate $10 or $20 a year for a new server platform.

You have to look at the benefits BYOAC has to offer, and what we have been receiving for free over the many years.
Thank God I am me.

shock_

  • Wiki Contributor
  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 354
  • In Brisbane, it's GMT+10 all the time.
Migrating an application of this size and age off Simple Machines Forum to another application like Burning Board is likely a thousand times more work and risk and cost, than getting the hosting stabilised. Saint's running the latest stable version of SMF which demonstrates someone at least is doing the right thing and keeping it up to date.

I'd suggest passing the hat around is not a bad idea, with the goal of engaging the support required to diagnose, then repair or migrate the hosting depending on what's actually failing.

Phreakwars

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 280
  • Waka-Waka
    • forum.arcadecontrols.com/index.php/topic,146325.0.html
    • My Facebook Page
If your worried about moving to another platform, just do like I did and install XAMPP on your local drive then copy the entire forum and database and convert from there. I know there are tons of converter tools for PhPBB, vBulletin, SMF, UBB, IPB, MyBB, XenForo, etc, etc.  SMF is nice because of the hook system when installing MODS, but it tends to be pretty taxing on the SQL. My personal preference would be PhPBB because it is so easy to work with. I've had no problem at all creating my own MOD's for it, even have a couple of them listed in the PhPBB MOD database. Not to mention there are tons of theme's available for it, and creating theme's is a breeze too.

One of the forums I owned had 500 plus sub forums (newsucks500.com now under new ownership and new platform), which, on most any other software, can really overheat those SQL database tables. PhPBB was able to handle it all just fine with no slowdown or crashing. Something I never would have attempted with SMF. In fact, the largest forum on the internet (gaiaonline.com), is nothing more then a heavily modified version of PhPBB.
« Last Edit: May 11, 2016, 03:57:22 am by Phreakwars »

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 5915
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Hey everyone - I appreciate it, and I'm sorry I haven't been responsive here.

Thank you for the offers of financial donations. It is within my means to be able to afford it if needed, so your intentions are gratefully acknowledged but respectfully declined :)

We're not on a hosted platform, I lease the entire server.

I've asked for assistance from someone with a level of expertise in the OS/config on the server. If this person is able to help then we'll be in good shape. If not, my intent is to lease a new server (this one is now about 3 years old) and migrate everything over to a fresh installation. I plan to stick with SMF unless it turns out the issue is SMF in some fashion. I don't believe it is, but obviously if I knew what it was it'd be resolved by now :)

It's going to take some more time. The person I've asked for help has their own life/work/family responsibilities, so time on fixing the server has to compete with that. If they aren't able to assist, in a couple of weeks I will start working 4x10 work weeks with 3 day weekends, and that will give me the extra time to purchase and start configuring the new server.

I appreciate everyone's patience and concern. If all else fails I will throw up the white flag and ask for additional help from the forum. For security's sake I am cautious about who I give access to the server directly, because it has your information on it in some fashion, but the option is on the table if needed.

I did finally figure out why my script to restart sql/web-server was failing, and as of this morning it automatically restarted things rather than having to wait for me to manually log on and restart them. This means when I sleep in on the weekend the forum won't be down until 9-10 AM :)

Thanks everyone!
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade.com/
     saint@arcadecontrols.com

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 5915
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
I just saw a slowdown at about 3-4 mins past the hour, but it's come good again. It's just now a little past midnight US Eastern. In the next couple of hours if history is a guide, the site is going to slow again and then throw 500 errors.

Given the symptom (500 errors from a server running SMF) and the timing (daily, just after "midnight", in someone's US timezone at least) I'm willing to guess:

  • SMF has too many scheduled tasks set to fire at 00:00 that don't need to be running at exactly that time - see http://wiki.simplemachines.org/smf/SMF2.0:Scheduled_Tasks
  • Those tasks are possibly creating a queue on MySQL.
  • PHP is throwing "MySQL has gone away" errors into an appropriate log once calls out to the DB exceed its timeout
  • Web server is slow to respond while it still trusts MySQL might eventually have an answer for it, then starts sending quick 500s once it's sure it won't be back

Things I'd do

Check for active queries while the site is slow.
Assuming there's direct access to the database (e.g. command line or even PHPMyAdmin), send this query to the site database as the site database's user:

SHOW FULL PROCESSLIST\G

If the database user has the necessary privileges this will braindump the active db queries. Things to look for are the lines with "Time:" and "State:" -- if there are heaps of rows and they're slow, well, yeah, database bottleneck. The queries themselves might give a clue exactly WHAT it's busy with, which might lead to some ideas about shifting that activity to some other time etc.

Check for any spikes in site traffic.

Count rows per minute in the web server's access.log file, look for any spiky patterns for IP addresses or User Agents. Could be as simple as something set to crawl the site at the same time each night.

Make sure someone with SMF knowledge is involved

I'm talking very generally here, a lot of the true fixes are going to require knowledge of how to tune SMF to avoid being wasteful. Perhaps there was a spamfest a few months ago that led to hundreds of thousands of dummy registrations in the database, now slowing down daily admin functions due to sheer size. Maybe a feature was enabled that runs some daily background processing and nobody remembered to turn it off. I don't know, I'm a Drupal guy, not an SMF guy, but all commodity web apps have their quirks and all can be abused from time to time :)

Thank you Shock, I'll dig into these. While it's slow all day, it *is* consistent on when it finally gives up the ghost each day.
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade.com/
     saint@arcadecontrols.com

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 5915
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
For any Linux (particularly Ubuntu) gurus out there, here is what a random sampling of iotop shows while the forum is running. The various bits that are showing up in the top change about 1-2 times per second.  In a few minutes, I'll down the web server and mysql, and capture a few iotops while they aren't running for comparison. Unfortunately, I don't have a baseline to know what is and isn't normal.

--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade.com/
     saint@arcadecontrols.com

  
 

Sitemap 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31