Main Restorations Software Audio/Jukebox/MP3 Everything Else Buy/Sell/Trade
Project Announcements Monitor/Video GroovyMAME Merit/JVL Touchscreen Meet Up Retail Vendors
Driving & Racing Woodworking Software Support Forums Consoles Project Arcade Reviews
Automated Projects Artwork Frontend Support Forums Pinball Forum Discussion Old Boards
Raspberry Pi & Dev Board controls.dat Linux Miscellaneous Arcade Wiki Discussion Old Archives
Lightguns Arcade1Up Try the site in https mode Site News

Unread posts | New Replies | Recent posts | Rules | Chatroom | Wiki | File Repository | RSS | Submit news

  

Author Topic: BYOAC GOING DOWN FOR MAINTENANCE  (Read 12785 times)

0 Members and 1 Guest are viewing this topic.

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:Yesterday at 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
BYOAC GOING DOWN FOR MAINTENANCE
« on: January 19, 2018, 06:38:18 am »
Sometime between now and Sunday BYOAC will be going offline for maintenance in the server center. While they're doing that we'll be tackling a failed hard drive on the server as well. I don't know when or what duration it'll be down during this time.

Afterwards I hope to start a thread about the current and future status of the site. Short version is site's not going anywhere as long as people keep visiting, no major changes (assuming we don't blow up during maintenance!). However, I need help. I need a server administrator with strong Linux skills to help update the site and resolve some of the broken issues (wiki, some issues with the forum database). I have the financial means to maintain the site, I don't have the time, and while I dabble pretty well in Linux, the two issues facing the site are beyond me. SirWoogie no longer has the time to devote here either, and I am extremely grateful for the time he's given us (you don't know how many late night hours he's put in here after a panicked text or phone call).

That's a discussion for post-maintenance however.

--- saint
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

Titchgamer

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 4222
  • Last login:December 17, 2023, 08:05:48 am
  • I have a gaming addiction.....
Re: BYOAC GOING DOWN FOR MAINTENANCE
« Reply #1 on: January 19, 2018, 06:43:50 am »
Thanks Saint for your hard work.
Does this mean we can get a light gun section now to ;)

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:Yesterday at 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Re: BYOAC GOING DOWN FOR MAINTENANCE
« Reply #2 on: January 19, 2018, 06:46:03 am »
That's one of the problems in the database right now. I can't add new forums .... The maintenance this weekend won't fix that. I'm considering paying the SMF forum folks to come take a look at the database to see if they can puzzle out the problem.

Thanks Saint for your hard work.
Does this mean we can get a light gun section now to ;)
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

Titchgamer

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 4222
  • Last login:December 17, 2023, 08:05:48 am
  • I have a gaming addiction.....
Re: BYOAC GOING DOWN FOR MAINTENANCE
« Reply #3 on: January 19, 2018, 06:47:07 am »
That's one of the problems in the database right now. I can't add new forums .... The maintenance this weekend won't fix that. I'm considering paying the SMF forum folks to come take a look at the database to see if they can puzzle out the problem.

Thanks Saint for your hard work.
Does this mean we can get a light gun section now to ;)

Great stuff!

Wish I had the skills to help out alas Linux is alien to me!

Malenko

  • KNEEL BEFORE ZODlenko!
  • Trade Count: (+58)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 13999
  • Last login:Yesterday at 01:15:07 pm
  • Have you played with my GingerBalls?
    • forum.arcadecontrols.com/index.php/topic,142404.msg1475162.html
Re: BYOAC GOING DOWN FOR MAINTENANCE
« Reply #4 on: January 19, 2018, 07:56:00 am »
I've been offering to fix this place for like TWO YEARS now.

Is your server at least on a RAID10? You could hot swap a drive without issue with no downtime.
If you're replying to a troll you are part of the problem.
I also need to follow this advice. Ignore or report, don't reply.

Nephasth

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 1376
  • Last login:March 11, 2024, 11:02:52 pm
  • Bitches love Centipede
Re: BYOAC GOING DOWN FOR MAINTENANCE
« Reply #5 on: January 19, 2018, 11:18:11 am »
YAS! We finally get forum karma!!!
%Bartop

vwalbridge

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 2082
  • Last login:July 31, 2021, 12:21:09 pm
  • Don't half-ass two things, whole-ass one thing.
Re: BYOAC GOING DOWN FOR MAINTENANCE
« Reply #6 on: January 19, 2018, 11:25:29 am »
YAS! We finally get forum karma!!!
One last fake rep for you!
If you can read this, it means Photobucket's money grab ruined my signature photos.

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:Yesterday at 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Re: BYOAC GOING DOWN FOR MAINTENANCE
« Reply #7 on: January 19, 2018, 11:42:07 am »
You have and I appreciate it and I will be contacting you offline to discuss after 1and1 does their maintenance this weekend.

The server is using a RAID-1 software mirror. Drive 1 is dead and we're operating on drive 2, as per design. However, in the implementation of Linux/Grub we're using, there was a bug and GRUB doesn't actually get installed on both drives in the mirror, just the first drive. Hence, we're now operational but if we reboot we're bricked. I'd have thought fixing that to be easy enough but I'm either missing something basic or it's more complicated than I expected. I have a ticket in to 1and1 support to fix that now and am waiting to hear back from them.

I've been offering to fix this place for like TWO YEARS now.

Is your server at least on a RAID10? You could hot swap a drive without issue with no downtime.
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

opt2not

  • Trade Count: (+15)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 6173
  • Last login:February 15, 2024, 07:31:21 pm
Re: BYOAC GOING DOWN FOR MAINTENANCE
« Reply #8 on: January 19, 2018, 12:18:00 pm »
Wow, Saint’s alive!  :o

wutno

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 22
  • Last login:June 03, 2023, 07:33:51 pm
  • I want to build my own arcade controls!
Re: BYOAC GOING DOWN FOR MAINTENANCE
« Reply #9 on: January 19, 2018, 01:03:52 pm »
Please make sure you have off-site backups!

Malenko

  • KNEEL BEFORE ZODlenko!
  • Trade Count: (+58)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 13999
  • Last login:Yesterday at 01:15:07 pm
  • Have you played with my GingerBalls?
    • forum.arcadecontrols.com/index.php/topic,142404.msg1475162.html
Re: BYOAC GOING DOWN FOR MAINTENANCE
« Reply #10 on: January 19, 2018, 02:36:54 pm »
The server is using a RAID-1 software mirror.
Strike 1! lol   Software mirrors, generally speaking, are ---steaming pile of meadow muffin---.

Please make sure you have off-site backups!
Well offsite backups are more for like a building burning down or some other sort of catastrophic event; if Saint's crib went kaput I don't think restoring BYOAC would be very high on the priority list. It's not bad advice though! At minimum he should be dumping backups to an external drive. Its an arcade forum, not medical records ;)

Where I work, I have the VMs replicating between hosts for fail over and redundancy. Hosts are on RAID 10 (mirror of stripes), and that data is backed up to a Unitrends appliance with its own RAID10 array. Unitrends does incremental daily back ups, with synthetic fulls at the end of the week, with natural full back up once a month. That data is mirrored to the cloud nightly after CRC checks.  Test restores are performed monthly as well. In the unlikely event all of our physical hosts fail we can spin up VMs on the unitrends box. If that also somehow magically died too, we can spin up VMs in the cloud.

I wrote out the entire DR plan and it was a giant pain in the butt and hopefully we never have to use it. I'm pretty savvy with backups, but my focus is still cyber security.
If you're replying to a troll you are part of the problem.
I also need to follow this advice. Ignore or report, don't reply.

opt2not

  • Trade Count: (+15)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 6173
  • Last login:February 15, 2024, 07:31:21 pm
Re: BYOAC GOING DOWN FOR MAINTENANCE
« Reply #11 on: January 19, 2018, 02:57:27 pm »
^ Get this man on the case, Saint! The guy does this stuff for a living!

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:Yesterday at 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Re: BYOAC GOING DOWN FOR MAINTENANCE
« Reply #12 on: January 19, 2018, 03:57:54 pm »
The server is hosted somewhere out in cyberspace, wherever 1and1.com is physically located. In hindsight I'm not a fan of software RAID :( We use hardware RAID 5 or 10 in my data center but I don't have physical access to the box BYOAC is on so software raid it was.

Every night the various databases on BYOAC are backed up and rotated, so at any one point I have 6 copies of the forum, wiki, main site, and files databases, not including the live one being used in production. I have pulled a copy of all of these off-site to my computer. I also have backups of /etc, /boot,  the html/css/etc files for sites hosted on the server, etc - all pulled off to my computer as well.

So if everything went to heck it could be rebuilt, it would just be painful.
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

Le Chuck

  • Saint, make a poll!
  • Wiki Contributor
  • Trade Count: (+6)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 5509
  • Last login:March 14, 2024, 06:07:15 pm
  • <insert personal text here>
Re: BYOAC GOING DOWN FOR MAINTENANCE
« Reply #13 on: January 19, 2018, 04:07:13 pm »
1) thanks for the heads up
B) we need to get post likes and features like what pinside has asap or this place will become the ghost town it’s slowly devolving into. User interaction is a huge component of how we consume media now.

vwalbridge

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 2082
  • Last login:July 31, 2021, 12:21:09 pm
  • Don't half-ass two things, whole-ass one thing.
Re: BYOAC GOING DOWN FOR MAINTENANCE
« Reply #14 on: January 19, 2018, 04:07:49 pm »
All this effort to keep data redundant and yet Photobucket still wrecked 1/2 the build threads.

#PourOneOut
If you can read this, it means Photobucket's money grab ruined my signature photos.

Howard_Casto

  • Idiot Police
  • Trade Count: (+1)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 19399
  • Last login:March 16, 2024, 05:59:16 pm
  • Your Post's Soul is MINE!!! .......Again??
    • The Dragon King
Re: BYOAC GOING DOWN FOR MAINTENANCE
« Reply #15 on: January 19, 2018, 04:46:04 pm »
^ Get this man on the case, Saint! The guy does this stuff for a living!

^^This^^

Also stop in more... even if it's just to say "hi"

Titchgamer

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 4222
  • Last login:December 17, 2023, 08:05:48 am
  • I have a gaming addiction.....
Re: BYOAC GOING DOWN FOR MAINTENANCE
« Reply #16 on: January 19, 2018, 04:47:16 pm »
All this effort to keep data redundant and yet Photobucket still wrecked 1/2 the build threads.

#PourOneOut

---fudgesicle--- photobucket in its cyber ass hole!

That is all :D

vwalbridge

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 2082
  • Last login:July 31, 2021, 12:21:09 pm
  • Don't half-ass two things, whole-ass one thing.
Re: BYOAC GOING DOWN FOR MAINTENANCE
« Reply #17 on: January 19, 2018, 04:54:59 pm »
Where I work, I have the VMs replicating between hosts for fail over and redundancy. Hosts are on RAID 10 (mirror of stripes), and that data is backed up to a Unitrends appliance with its own RAID10 array. Unitrends does incremental daily back ups, with synthetic fulls at the end of the week, with natural full back up once a month. That data is mirrored to the cloud nightly after CRC checks.  Test restores are performed monthly as well. In the unlikely event all of our physical hosts fail we can spin up VMs on the unitrends box. If that also somehow magically died too, we can spin up VMs in the cloud.

I wrote out the entire DR plan and it was a giant pain in the butt and hopefully we never have to use it. I'm pretty savvy with backups, but my focus is still cyber security.



...but where I work we just let AWS do the dirty work.  ;)
If you can read this, it means Photobucket's money grab ruined my signature photos.

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:Yesterday at 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Re: BYOAC GOING DOWN FOR MAINTENANCE
« Reply #18 on: January 20, 2018, 01:25:09 pm »
STATUS:

Problem with GRUB not installed on second drive in RAID-mirror set solved. Test reboot at 1:15 PM today successful.

Next: Server will be shut down at about 9:00 PM Eastern tonight in preparation for the data center shutdown. They will be working on power in their data center throughout the night. Sometime in the morning the server will be brought back online. I'm not clear on if they'll power everything up or if I'll need to bring it back up via remote management tool. I'm usually up between 7 and 9 AM on Sundays so I'll start trying to get things back online then if not already.

After: Replacing the bad drive in the mirror, rebuild the mirror, then bring someone on to help with the things the OS/forum need.
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

saint

  • turned to the Dark Side
  • Supreme Chancellor
  • Trade Count: (+6)
  • Full Member
  • *****
  • Offline Offline
  • Posts: 6144
  • Last login:Yesterday at 07:49:54 am
  • I only work in cyberspace...
    • Build Your Own Arcade Controls
Re: BYOAC GOING DOWN FOR MAINTENANCE
« Reply #19 on: January 21, 2018, 09:28:34 am »
Data center maintenance complete. Back up and running. Tomorrow will get them to swap out the failed hard drive.
--- John St.Clair
     Build Your Own Arcade Controls FAQ
     http://www.arcadecontrols.com/
     Project Arcade 2!
     http://www.projectarcade2.com/
     saint@arcadecontrols.com

wp34

  • Trade Count: (+3)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 4794
  • Last login:April 10, 2022, 09:48:19 pm
Re: BYOAC GOING DOWN FOR MAINTENANCE
« Reply #20 on: January 21, 2018, 10:25:42 am »
Data center maintenance complete. Back up and running. Tomorrow will get them to swap out the failed hard drive.

 :applaud:  Thanks Saint.