Main Restorations Software Audio/Jukebox/MP3 Everything Else Buy/Sell/Trade
Project Announcements Monitor/Video GroovyMAME Merit/JVL Touchscreen Meet Up Retail Vendors
Driving & Racing Woodworking Software Support Forums Consoles Project Arcade Reviews
Automated Projects Artwork Frontend Support Forums Pinball Forum Discussion Old Boards
Raspberry Pi & Dev Board controls.dat Linux Miscellaneous Arcade Wiki Discussion Old Archives
Lightguns Arcade1Up Try the site in https mode Site News

Unread posts | New Replies | Recent posts | Rules | Chatroom | Wiki | File Repository | RSS | Submit news

  

Author Topic: What's up with the server and BYOAC?  (Read 8396 times)

0 Members and 1 Guest are viewing this topic.

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
What's up with the server and BYOAC?
« on: June 07, 2016, 11:31:27 am »
Short version:

We have a hard drive in the RAID on the server that was failing. It's always the simplest things in the end, but at first it really looked like something gone haywire with journaling on the drives, or mysql going bonkers, or something similar.

We've since taken the bad drive out of the RAID, and spent the last 24 hours or so backing up several hundred gigs of data off-server as a precaution. I've had the forum offline during this time.

We still have more to look at, and need to get the bad drive replaced, so there will be additional downtime in the near future.

Also, while we suspect the RAID was the alpha and omega of the problems, only time will tell if other issues need attention. The server seems responsive enough right now, but I'm still seeing occasional spikes of high CPU usage with corresponding server lag, so I'm not holding my breath just yet.

More updates when I have them. Thanks!

--- saint
« Last Edit: June 07, 2016, 11:35:35 am by saint »
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Re: What's up with the server and BYOAC?
« Reply #1 on: June 07, 2016, 11:55:38 am »
Trivia:

The forum alone takes up 36 gigs of drive space, not including the SQL database back end.

There are 236,849 attachments stored in the forum as of this writing.
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

pbj

  • Trade Count: (+4)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 10871
  • Last login:Yesterday at 11:58:22 pm
  • Obey.
    • The Chris Burke Band
Re: What's up with the server and BYOAC?
« Reply #2 on: June 07, 2016, 12:16:12 pm »
Failing hard drive is just about as embarrassing as "problem went away after hours of trouble shooting when I rebooted."

;)



Howard_Casto

  • Idiot Police
  • Trade Count: (+1)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 19399
  • Last login:March 16, 2024, 05:59:16 pm
  • Your Post's Soul is MINE!!! .......Again??
    • The Dragon King
Re: What's up with the server and BYOAC?
« Reply #3 on: June 07, 2016, 12:23:56 pm »
Good work Saint.  We appreciate you man. 

yotsuya

  • Trade Count: (+21)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 19955
  • Last login:Yesterday at 10:19:17 pm
  • 2014 UCA Winner, 2014, 2015, 2016 ZapCon Winner
    • forum.arcadecontrols.com/index.php/topic,137636.msg1420628.html
Re: What's up with the server and BYOAC?
« Reply #4 on: June 07, 2016, 12:35:13 pm »
Seems snappy now!

Sent from my SM-G386T using Tapatalk

***Build what you dig, bro. Build what you dig.***

smalltownguy

  • Trade Count: (+1)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 924
  • Last login:February 13, 2023, 10:48:26 am
Re: What's up with the server and BYOAC?
« Reply #5 on: June 07, 2016, 12:39:48 pm »
Thanks for getting things back up and running. I don't post much here these days, but I do visit frequently.

More than willing to pop a few bucks for a new drive if you need it. LMK where to "donate" money  :laugh2:
Man, will my cab EVER be finished?

Vidiot

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 303
  • Last login:February 15, 2024, 10:00:17 am
  • John 3:16
Re: What's up with the server and BYOAC?
« Reply #6 on: June 07, 2016, 12:45:51 pm »
Fingers crossed you have the problem identified. Thanks for all you do Saint.  :applaud:


saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Re: What's up with the server and BYOAC?
« Reply #7 on: June 07, 2016, 12:47:46 pm »
Thanks guys :) The credit is all sirwoogie's for figuring out what was going on and doing the heavy lifting.

Appreciate the offer of a donation :) BYOAC is my way of paying it forward and the costs of keeping it running are within my means. Back in the day we took donations, but these days if you're so inclined, please find a charity that appeals to you and donate to them instead. Thanks for the offer :)
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

wp34

  • Trade Count: (+3)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 4794
  • Last login:April 10, 2022, 09:48:19 pm
Re: What's up with the server and BYOAC?
« Reply #8 on: June 07, 2016, 01:32:49 pm »
Thanks Saint and sirwoogie!   :cheers:

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Re: What's up with the server and BYOAC?
« Reply #9 on: June 07, 2016, 09:23:24 pm »
They did *not* get the hard drive replaced today. Sigh... So, some time Wednesday we will be going down again to hopefully get the drive replaced.
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

sean_sk

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 144
  • Last login:August 06, 2019, 10:27:48 am
  • If quizzes are quizzical, then what are tests?
Re: What's up with the server and BYOAC?
« Reply #10 on: June 08, 2016, 12:26:05 am »
This forum ranks as being my favorite and most visited, so saint I wanted to take the time thank yourself, sirwoogie and others that may be involved in all their hard work. I hope it hasn't been too stressful.

EssexMame

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 471
  • Last login:September 21, 2023, 07:43:50 am
  • Mame Weekender
Re: What's up with the server and BYOAC?
« Reply #11 on: June 08, 2016, 06:34:01 am »
This forum ranks as being my favorite and most visited, so saint I wanted to take the time thank yourself, sirwoogie and others that may be involved in all their hard work. I hope it hasn't been too stressful.

I second that. Indeed, its the only forum I visit...

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Re: What's up with the server and BYOAC?
« Reply #12 on: June 08, 2016, 03:53:03 pm »
Thanks, nice to hear :)

Status: The failing drive has been replaced. Next step is to re-establish the mirror and wait for the RAID to rebuild.

Then the bad news is apparently the first drive is also showing signs of failure, though it's not nearly as bad as the second was (144 errors vs. 16,000 errors).

So - once we have the mirror rebuilt and the RAID re-established, we're going to do this all again and replace the first drive.

We'll keep the server up until then though :)

Thanks for your patience everyone!
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

Howard_Casto

  • Idiot Police
  • Trade Count: (+1)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 19399
  • Last login:March 16, 2024, 05:59:16 pm
  • Your Post's Soul is MINE!!! .......Again??
    • The Dragon King
Re: What's up with the server and BYOAC?
« Reply #13 on: June 08, 2016, 04:02:44 pm »
Man, when your redundancies are failing it's pretty bad.  We must be hammering this poor forum.

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Re: What's up with the server and BYOAC?
« Reply #14 on: June 08, 2016, 04:05:54 pm »
This particular incarnation of the box is pushing 4 years old I think. We were rather startled when we looked at it a few weeks ago to see that its uptime was over 3 years without a reboot.
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

DeLuSioNal29

  • Global Moderator
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 4778
  • Last login:October 20, 2023, 11:39:06 pm
  • Build the impossible -"There is no Spoon"
    • DeLuSioNaL's YouTube Videos
Re: What's up with the server and BYOAC?
« Reply #15 on: June 08, 2016, 04:27:36 pm »
Glad to hear about progress!   :applaud:

DeL
Stop by my Youtube channel and leave a comment:

behrmr

  • Trade Count: (+1)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 459
  • Last login:April 23, 2021, 09:17:13 am
Re: What's up with the server and BYOAC?
« Reply #16 on: June 09, 2016, 11:05:06 am »
Man, when your redundancies are failing it's pretty bad.  We must be hammering this poor forum.

It's not uncommon for drives from the same manufacturing run to fail at the same time or within a short period of time.  Particularly those within a RAID configuration where access times are similar.  I've had this happen more than once in my [previous] career managing server farms.

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Re: What's up with the server and BYOAC?
« Reply #17 on: June 09, 2016, 12:35:24 pm »
Yeah, anytime I have a raid failure on a single drive I start eyeballing the other drives.

We are rebuilding the raid now. So far the forum still seems responsive but if you notice slowdowns over the next several to many hours today, that's likely the cause.
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

Token

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 490
  • Last login:September 20, 2021, 03:45:43 pm
Re: What's up with the server and BYOAC?
« Reply #18 on: June 09, 2016, 01:40:40 pm »
 :cheers:

I had pretty much given up. Glad things are looking better for BYAOAC.

stigzler

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 977
  • Last login:February 11, 2022, 03:01:48 pm
  • It seemed like a good idea at the time....
    • Mago Arcade
Re: What's up with the server and BYOAC?
« Reply #19 on: June 10, 2016, 08:12:06 am »
Thanks to you guys for all your efforts and hosting.

harveybirdman

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 2538
  • Last login:February 19, 2024, 12:40:25 pm
  • SHMUP'EM
Re: What's up with the server and BYOAC?
« Reply #20 on: June 15, 2016, 09:24:55 am »
Internal Server Error last night... Related to the other drive? maintenance?

PL1

  • Global Moderator
  • Trade Count: (+1)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 9393
  • Last login:Today at 04:47:42 am
  • Designated spam hunter
Re: What's up with the server and BYOAC?
« Reply #21 on: June 15, 2016, 09:57:42 am »
Internal Server Error last night... Related to the other drive? maintenance?
Nothing new last night.

There were the usual two daily down-times.

  1.)  1 A.M. Eastern -- Down for about 1-2 minutes

  2.)  Shortly after 4 A.M. Eastern -- Down for about 30 minutes

The site followed that pattern before the drive went bad and is back to following it again.


Scott
« Last Edit: July 02, 2016, 01:55:05 am by PL1 »

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Re: What's up with the server and BYOAC?
« Reply #22 on: June 22, 2016, 09:24:46 am »
Today I'm doing database maintenance if you see slow downs. I have various jobs running backups overnight, once I get done with this bit of maintenance I'll look at them to see if there's anything I can do about the nightly outage.
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

Howard_Casto

  • Idiot Police
  • Trade Count: (+1)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 19399
  • Last login:March 16, 2024, 05:59:16 pm
  • Your Post's Soul is MINE!!! .......Again??
    • The Dragon King
Re: What's up with the server and BYOAC?
« Reply #23 on: June 25, 2016, 12:26:39 pm »
Yeah, because I'm apparently a vampire or something I'm rarely asleep when it goes down.  It had been doing that for quite some time before the server issues, but I thought it was nightly maintenance so I never mentioned it. 

artyfarty

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 72
  • Last login:February 24, 2024, 03:35:15 am
  • I want to build my own arcade controls!
Re: What's up with the server and BYOAC?
« Reply #24 on: June 26, 2016, 10:22:39 am »
Don't forget your overseas members - overnight isn't the same everywhere!


Sent from my iPhone using Tapatalk

Jakobud

  • Trade Count: (+1)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 1940
  • Last login:March 11, 2024, 04:45:38 am
Re: What's up with the server and BYOAC?
« Reply #25 on: July 03, 2016, 07:41:48 pm »
Have you considered moving thing site over to AWS? Way cheaper and easier than trying to maintain hardware someplace.

You could even do a hybrid solution. Host the website and forum on your own server (or even a VPS somewhere) and then host the database in AWS RDS and all the attachments in S3. Saint if you were ever interested in exploring solutions like this, let me know. I could help you set some things up.

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:March 17, 2024, 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Re: What's up with the server and BYOAC?
« Reply #26 on: July 06, 2016, 10:03:07 pm »
I haven't, but I'll take a look. Other than this drive failure (which, on a server in constant action for 4 years, isn't surprising), service and speeds have been good with this hosting solution I think. I'll compare costs.
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com