1. DLP Flash Christmas Competition + Writing Marathon 2024!

    Competition topic: Magical New Year!

    Marathon goal? Crank out words!

    Check the marathon thread or competition thread for details.

    Dismiss Notice

  2. Hi there, Guest

    Only registered users can really experience what DLP has to offer. Many forums are only accessible if you have an account. Why don't you register?
    Dismiss Notice
  3. Introducing for your Perusing Pleasure

    New Thread Thursday
    +
    Shit Post Sunday

    READ ME
    Dismiss Notice
DLP Story Competition Guest User Registration Nag SPS + NTT

Downtime 7/23/14

Discussion in 'Announcements' started by Lord Ravenclaw, Jul 23, 2014.

4.3333334922791/5, 3 votes
Page 1 of 3
  1. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin DLP Supporter

    Joined:
    Apr 2, 2005
    Messages:
    4,372
    Location:
    Denver, CO
    Jul 23, 2014 #1
    Hello DLPers,

    I'm sad to announce that we had about 1 hour of downtime today.

    We had previously been running for 1426 days straight before restarting our server. An honorable record.

    A cron job (a periodic task) had begun to stall on the DLP server. Unfortunately, it wasn't just one task, it was hundreds of them over time. They exhausted the DLP server of much-needed RAM, at which point the operating system began killing processes, including HTTP servers, the DLP API, Scryer, IRC, and (most importantly) SSHd used to give me management access to the server. At this point, I had to force a restart into a recovery mode to run filesystem checks (thankfully, no disk failures or we'd potentially still be down).

    This same issue has been causing the occasional response timeouts, site slowdowns, and 500 errors.

    We're back up now. Please reply if you see any downed services, bringing up a server after 1426 days of constant uptime leaves a lot of be forgotten.

    Future outages will be reported on at https://twitter.com/darklordpotter
     
    Midknight, Xiph0, Jibril, Jheph, Warlocke, Iztiak, Vira, Riley, Fenraellis, Irene, Oz, KrzaQ, Ched, Alindrome, H_A_Greene, Klackerz, Hachi, Aerylife, Nauro
  2. Aerylife

    Aerylife Not Equal

    Joined:
    Jan 28, 2011
    Messages:
    141
    Location:
    Everywhere
    High Score:
    1,828
    Jul 23, 2014 #2
    I still think this is somehow Jon's fault, but I'm happy you were able to deal with it so quickly.
     
  3. Innomine

    Innomine Alchemist ~ Prestige ~ DLP Supporter

    Joined:
    Nov 27, 2007
    Messages:
    2,333
    Gender:
    Male
    Location:
    New Zealand
    High Score:
    4,500
    Jul 24, 2014 #3
    I woke up this morning to see that DLP wasn't working. You never know how much you'll miss something until you don't have it!
     
    Rahkesh Asmodaeus, Aekiel, afrojack, Sully, Sophie, H_A_Greene, TheWiseTomato, CrackedMind, AlbusPHolmes, Meerkats
  4. H_A_Greene

    H_A_Greene Unspeakable –§ Prestigious §– DLP Supporter

    Joined:
    Aug 30, 2009
    Messages:
    715
    High Score:
    4,492
    Jul 24, 2014 #4
    Hit another 500 Internal Server Error about three minutes ago(~3AM even EST).
     
  5. fire

    fire Order Member

    Joined:
    Dec 25, 2011
    Messages:
    808
    Aug 2, 2014 #5
    Experienced a 500 ISE about a few hours ago. Not sure if it was the site, or just me.
     
  6. H_A_Greene

    H_A_Greene Unspeakable –§ Prestigious §– DLP Supporter

    Joined:
    Aug 30, 2009
    Messages:
    715
    High Score:
    4,492
    Aug 2, 2014 #6
    The site was definitely offline starting around 5 AM EST until about the last hour.
     
    Mage, Anme, ScottPress
  7. ScottPress

    ScottPress The Horny Sovereign –§ Prestigious §– DLP Supporter

    Joined:
    Apr 22, 2013
    Messages:
    110
    Gender:
    Male
    Location:
    The Holy Moose Empire
    High Score:
    6900
    Aug 2, 2014 #7
    Same thing happened for me. Some kind of database error, it said. Forgot to take a screenshot.
     
  8. Oz

    Oz For Zombie. Moderator DLP Supporter

    Joined:
    Jan 31, 2008
    Messages:
    9,027
    Gender:
    Female
    Location:
    Baile Átha Cliath
    Aug 2, 2014 #8
    12:46:25 PM <~Raven> Back up for now
    12:46:35 PM <~Raven> I have to go, but it may well die while I'm gone
     
  9. ScottPress

    ScottPress The Horny Sovereign –§ Prestigious §– DLP Supporter

    Joined:
    Apr 22, 2013
    Messages:
    110
    Gender:
    Male
    Location:
    The Holy Moose Empire
    High Score:
    6900
    Aug 2, 2014 #9
    ^ so DLP is up only as long as Raven's brain is hooked up to the Internet?

    edit: inb4wellduh?
     
  10. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin DLP Supporter

    Joined:
    Apr 2, 2005
    Messages:
    4,372
    Location:
    Denver, CO
    Aug 2, 2014 #10
    Database server ran out of disk space overnight. Investigating the source of the data.
     
    Gizmore, H_A_Greene, A.K.$J6-J5
  11. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin DLP Supporter

    Joined:
    Apr 2, 2005
    Messages:
    4,372
    Location:
    Denver, CO
    Aug 5, 2014 #11
    Update 8/5/14: Mysterious server hardlock. Had to reboot from remote management.

    Server memory caches are rewarmed, performance recovering.

    Moving to new hardware soon.

    Next step: Temporary VPS. (update: 90% there, database is now running on VPS)

    Not sure we'll need colocation after seeing how this VPS is performing.
     
    H_A_Greene
  12. Oz

    Oz For Zombie. Moderator DLP Supporter

    Joined:
    Jan 31, 2008
    Messages:
    9,027
    Gender:
    Female
    Location:
    Baile Átha Cliath
    Aug 6, 2014 #12
    Step 4: complete global takeover.
     
    afrojack, KrzaQ, mknote, Another Empty Frame
  13. afrojack

    afrojack Chief Warlock DLP Supporter

    Joined:
    Dec 29, 2006
    Messages:
    1,592
    Gender:
    Male
    Location:
    Southron California
    Aug 6, 2014 #13
    DLPlanet

    /tenchar
     
    Another Empty Frame
  14. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin DLP Supporter

    Joined:
    Apr 2, 2005
    Messages:
    4,372
    Location:
    Denver, CO
    Aug 6, 2014 #14
    DLP is in-progress moving to a newer, better, cheaper server. The database & forums are already hosted on that server, it's now about moving auxiliary services such as IRC, the DLP API, Scryer, and such.

    We'll be maintaining overlapping servers for one month, and then dropping the old one.

    You should already be feeling the speed increase over the old server if your DNS has propagated. Better hardware and SPDY-enabled.
     
    Gizmore, Hero of Stupidity
  15. ScottPress

    ScottPress The Horny Sovereign –§ Prestigious §– DLP Supporter

    Joined:
    Apr 22, 2013
    Messages:
    110
    Gender:
    Male
    Location:
    The Holy Moose Empire
    High Score:
    6900
    Aug 6, 2014 #15
    Does server-moving have something to do with some features in the post-reply box not working?
     
  16. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin DLP Supporter

    Joined:
    Apr 2, 2005
    Messages:
    4,372
    Location:
    Denver, CO
    Aug 6, 2014 #16
    Be specific. I haven't had any issues.
     
  17. ScottPress

    ScottPress The Horny Sovereign –§ Prestigious §– DLP Supporter

    Joined:
    Apr 22, 2013
    Messages:
    110
    Gender:
    Male
    Location:
    The Holy Moose Empire
    High Score:
    6900
    Aug 6, 2014 #17
    The button to cancel post editing wasn't working and I couldn't recolor or resize the text, but it must have been something temporary on my end. Everything is working fine now.
     
  18. Lord Ravenclaw

    Lord Ravenclaw DLP Overlord Admin DLP Supporter

    Joined:
    Apr 2, 2005
    Messages:
    4,372
    Location:
    Denver, CO
    Aug 6, 2014 #18
    You're right though, it's looking like some of the JS features aren't working. Will investigate.
     
  19. H_A_Greene

    H_A_Greene Unspeakable –§ Prestigious §– DLP Supporter

    Joined:
    Aug 30, 2009
    Messages:
    715
    High Score:
    4,492
    Aug 6, 2014 #19
    Trying to rate this thread brings up a database error. Specifically the below;


    EDIT: Trying to add a new rating to any thread seems to bring this up. I just tested it out on this and it gave me the same result.
     
    Last edited: Aug 6, 2014
  20. ScottPress

    ScottPress The Horny Sovereign –§ Prestigious §– DLP Supporter

    Joined:
    Apr 22, 2013
    Messages:
    110
    Gender:
    Male
    Location:
    The Holy Moose Empire
    High Score:
    6900
    Aug 6, 2014 #20
    I checked myself out of curiosity on two random threads and it happened for me too.
     
Page 1 of 3
Loading...
Similar Threads
  1. Lord Ravenclaw

    Scheduled Downtime: Saturday, July 31st

    Lord Ravenclaw, Jul 29, 2010, in forum: Announcements
    Replies:
    11
    Views:
    6,649
    Antivash
    Aug 1, 2010
  2. Lord Ravenclaw

    Unplanned Downtime

    Lord Ravenclaw, Jun 20, 2009, in forum: Announcements
    Replies:
    10
    Views:
    4,807
    Innomine
    Jun 21, 2009
  3. Lord Ravenclaw

    Downtime Saturday, February 9th at 9:00AM EST (2:00PM GMT)

    Lord Ravenclaw, Feb 6, 2008, in forum: Announcements
    Replies:
    12
    Views:
    5,365
    Lord Ravenclaw
    Feb 9, 2008
  4. Lord Ravenclaw

    Downtime Issues Solved

    Lord Ravenclaw, Jan 28, 2007, in forum: Announcements
    Replies:
    8
    Views:
    3,746
    That Wench
    Jan 28, 2007
  5. Lord Ravenclaw

    Downtime

    Lord Ravenclaw, Jul 29, 2006, in forum: Announcements
    Replies:
    0
    Views:
    2,605
    Lord Ravenclaw
    Jul 29, 2006