The NEW Build Your Own Arcade Controls
Main => Forum/Website Discussion => Topic started by: saint on January 19, 2018, 06:38:18 am
-
Sometime between now and Sunday BYOAC will be going offline for maintenance in the server center. While they're doing that we'll be tackling a failed hard drive on the server as well. I don't know when or what duration it'll be down during this time.
Afterwards I hope to start a thread about the current and future status of the site. Short version is site's not going anywhere as long as people keep visiting, no major changes (assuming we don't blow up during maintenance!). However, I need help. I need a server administrator with strong Linux skills to help update the site and resolve some of the broken issues (wiki, some issues with the forum database). I have the financial means to maintain the site, I don't have the time, and while I dabble pretty well in Linux, the two issues facing the site are beyond me. SirWoogie no longer has the time to devote here either, and I am extremely grateful for the time he's given us (you don't know how many late night hours he's put in here after a panicked text or phone call).
That's a discussion for post-maintenance however.
--- saint
-
Thanks Saint for your hard work.
Does this mean we can get a light gun section now to ;)
-
That's one of the problems in the database right now. I can't add new forums .... The maintenance this weekend won't fix that. I'm considering paying the SMF forum folks to come take a look at the database to see if they can puzzle out the problem.
Thanks Saint for your hard work.
Does this mean we can get a light gun section now to ;)
-
That's one of the problems in the database right now. I can't add new forums .... The maintenance this weekend won't fix that. I'm considering paying the SMF forum folks to come take a look at the database to see if they can puzzle out the problem.
Thanks Saint for your hard work.
Does this mean we can get a light gun section now to ;)
Great stuff!
Wish I had the skills to help out alas Linux is alien to me!
-
I've been offering to fix this place for like TWO YEARS now.
Is your server at least on a RAID10? You could hot swap a drive without issue with no downtime.
-
YAS! We finally get forum karma!!!
-
YAS! We finally get forum karma!!!
One last fake rep for you!
-
You have and I appreciate it and I will be contacting you offline to discuss after 1and1 does their maintenance this weekend.
The server is using a RAID-1 software mirror. Drive 1 is dead and we're operating on drive 2, as per design. However, in the implementation of Linux/Grub we're using, there was a bug and GRUB doesn't actually get installed on both drives in the mirror, just the first drive. Hence, we're now operational but if we reboot we're bricked. I'd have thought fixing that to be easy enough but I'm either missing something basic or it's more complicated than I expected. I have a ticket in to 1and1 support to fix that now and am waiting to hear back from them.
I've been offering to fix this place for like TWO YEARS now.
Is your server at least on a RAID10? You could hot swap a drive without issue with no downtime.
-
Wow, Saint’s alive! :o
-
Please make sure you have off-site backups!
-
The server is using a RAID-1 software mirror.
Strike 1! lol Software mirrors, generally speaking, are ---steaming pile of meadow muffin---.
Please make sure you have off-site backups!
Well offsite backups are more for like a building burning down or some other sort of catastrophic event; if Saint's crib went kaput I don't think restoring BYOAC would be very high on the priority list. It's not bad advice though! At minimum he should be dumping backups to an external drive. Its an arcade forum, not medical records ;)
Where I work, I have the VMs replicating between hosts for fail over and redundancy. Hosts are on RAID 10 (mirror of stripes), and that data is backed up to a Unitrends appliance with its own RAID10 array. Unitrends does incremental daily back ups, with synthetic fulls at the end of the week, with natural full back up once a month. That data is mirrored to the cloud nightly after CRC checks. Test restores are performed monthly as well. In the unlikely event all of our physical hosts fail we can spin up VMs on the unitrends box. If that also somehow magically died too, we can spin up VMs in the cloud.
I wrote out the entire DR plan and it was a giant pain in the butt and hopefully we never have to use it. I'm pretty savvy with backups, but my focus is still cyber security.
-
^ Get this man on the case, Saint! The guy does this stuff for a living!
-
The server is hosted somewhere out in cyberspace, wherever 1and1.com is physically located. In hindsight I'm not a fan of software RAID :( We use hardware RAID 5 or 10 in my data center but I don't have physical access to the box BYOAC is on so software raid it was.
Every night the various databases on BYOAC are backed up and rotated, so at any one point I have 6 copies of the forum, wiki, main site, and files databases, not including the live one being used in production. I have pulled a copy of all of these off-site to my computer. I also have backups of /etc, /boot, the html/css/etc files for sites hosted on the server, etc - all pulled off to my computer as well.
So if everything went to heck it could be rebuilt, it would just be painful.
-
1) thanks for the heads up
B) we need to get post likes and features like what pinside has asap or this place will become the ghost town it’s slowly devolving into. User interaction is a huge component of how we consume media now.
-
All this effort to keep data redundant and yet Photobucket still wrecked 1/2 the build threads.
#PourOneOut
-
^ Get this man on the case, Saint! The guy does this stuff for a living!
^^This^^
Also stop in more... even if it's just to say "hi"
-
All this effort to keep data redundant and yet Photobucket still wrecked 1/2 the build threads.
#PourOneOut
---fudgesicle--- photobucket in its cyber ass hole!
That is all :D
-
Where I work, I have the VMs replicating between hosts for fail over and redundancy. Hosts are on RAID 10 (mirror of stripes), and that data is backed up to a Unitrends appliance with its own RAID10 array. Unitrends does incremental daily back ups, with synthetic fulls at the end of the week, with natural full back up once a month. That data is mirrored to the cloud nightly after CRC checks. Test restores are performed monthly as well. In the unlikely event all of our physical hosts fail we can spin up VMs on the unitrends box. If that also somehow magically died too, we can spin up VMs in the cloud.
I wrote out the entire DR plan and it was a giant pain in the butt and hopefully we never have to use it. I'm pretty savvy with backups, but my focus is still cyber security.
(https://ci.memecdn.com/9250600.jpg)
...but where I work we just let AWS do the dirty work. ;)
-
STATUS:
Problem with GRUB not installed on second drive in RAID-mirror set solved. Test reboot at 1:15 PM today successful.
Next: Server will be shut down at about 9:00 PM Eastern tonight in preparation for the data center shutdown. They will be working on power in their data center throughout the night. Sometime in the morning the server will be brought back online. I'm not clear on if they'll power everything up or if I'll need to bring it back up via remote management tool. I'm usually up between 7 and 9 AM on Sundays so I'll start trying to get things back online then if not already.
After: Replacing the bad drive in the mirror, rebuild the mirror, then bring someone on to help with the things the OS/forum need.
-
Data center maintenance complete. Back up and running. Tomorrow will get them to swap out the failed hard drive.
-
Data center maintenance complete. Back up and running. Tomorrow will get them to swap out the failed hard drive.
:applaud: Thanks Saint.