Data Loss

Discussion in 'Announcements' started by Reaper, Apr 17, 2015.

  1. Reaper

    Reaper Administrator Staff Member Administrator

    Time for a not-so-positive update. I was merging some forums together, and the forum software we use released a new version which lets me drag forums around to move them around, which makes life a lot easier. So I went ahead and moved all of the posts to a forum and moved the forum out, then deleted its previous parent category.

    That's an illustration of how it looks like. To me, it looked like it's on the after position. So when I deleted the category, I wasn't expecting the add-on to fail and also remove the category with the forum with all of the messages in. Unfortunately, it was actually the forum with all of our gaming discussions in.

    I've tried data recovery software but I can't find it. I have an old backup from February, which has a lot of the data. I tried importing the posts but threads and posts get mixed up and it makes no sense weirdly. I'll probably find a solution to recover the data from the backup which has a lot of the posts, but until then I really don't know. During the full day the board was on maintenance mode I've been trying to recover the data, and had a backup to revert the database to in a worst case scenario. Since the backup restoration didn't work I tried to restore the database backup, and guess what it didn't work, the file was empty. So we're on my experimental site. I removed most of my mess but there are probably a few issues, please point them out if you notice any. I'm not sure what to do about them. There are a few ideas I have, one of them is to pull posts out of an RSS feed from the old board being put live, that should get the posts back but not with the authors of the content. Merging databases didn't work either.

    I'm really pissed off about all of this, hopefully I can find a way to get a lot of the data back. There is a new General forum, and I'd really appreciate if we could make up for lost content. I've made a script to make backups every 24 hours and email any errors to me, backups are taken off-site onto a local hard drive in my house every week, where they are kept for 6 months. I had the script for this a while ago as well actually, but, guess what, it also failed. I'd appreciate your patience through this.

    There is an archived copy of our content you can find at web.archive.org (http://web.archive.org/web/20150315040344/https://www.gamingforums.net/#gaming.15 (the entire category)). Sadly it doesn't do much good if we can't pull it off there with the post owners there respectively. We lost about 300-400 posts, not including the staff forums which also had drafts of content, which took pretty damn long to write, so some really long 50 or so posts too. My message count personally went from 200 to 88.
     
    Last edited: Apr 17, 2015
  2. Darth_Marques

    Darth_Marques Administrator Staff Member Administrator Premium

    Lol. Shit happens man, I'll see if I can't dreg up some of the old posts and repost them this weekend. I can't remember if xF let you swap authors of threads or not. I remember being able to do it on forum software of some sort (from Admin CP) but which one is beyond me. The only other option is to manually swap the thread author ID number in the database. Extremely tedious, but for preservation of posts and content the option is there.

    At least, I think that'll work. I thought about it and realized I'd never attempted this xF before lol. Just be sure to have a tested backup prior to changing anything.
     
  3. Darth_Marques

    Darth_Marques Administrator Staff Member Administrator Premium

    Also, are the threads still in existence in the database? (Not likely if you did a restore already.) But if they were you could run a query to restore the missing ones.
     
  4. Reaper

    Reaper Administrator Staff Member Administrator

    Not in the admin control panel.

    Quite easy actually. Tedious in PhpMyAdmin but easy in a simple MySQL console. UPDATE xf.posts SET user_id=5 WHERE post_id=2. Something like that should work. Probably need to update thread authors in xf.thread too.

    Yeah, fixed why the backup script wasn't working. Wrong MySQL password o_O

    Now, nope. Removing the forum runs DELETE FROM queries. Thing is, we have bin logs on, so every single MySQL queries is recorded to my knowledge. The hard part? Searching for every one of the 300 posts and recovering them by hand. I can't find an application to parse the file into human readable form, tick the queries I want to run again and get a .sql file to run the queries from. We also have .ibd files right after the event. I just have no clue how to use them in a way I haven't tried already. I have an old backup with most of them but again, it won't work. lol.

    Just be lucky we didn't have tons of posts. I might make an add-on that asks for my password before doing irreversible damage - or the smart way is just to make backups daily, that's checked off my list)
     
  5. Lara

    Lara Guest

    Stuff happens :p I didn't lose too many posts, didn't have many in the first place :rolleyes:
     
  6. Creaky

    Creaky Captain

    0%
    Does your host not have a databae backup you could use?
     
  7. Reaper

    Reaper Administrator Staff Member Administrator

    We use VPS hosting, self managed. Backups system failed. I have a backup with the posts we need but I can't seem to 'merge' the databases. I'll find a way, hopefully.
     
  8. Darth_Marques

    Darth_Marques Administrator Staff Member Administrator Premium

    Merging databases is risky... Still have issues with the posts being out of order. I can't find anything from xF that helps. Content can be replaced, much easier than an entire database.
     
  9. Reaper

    Reaper Administrator Staff Member Administrator

    Seems like threads, authors and posts are muddled up. Going to restore posts (xf.threads, xf.node, xf.posts). Backup ofc in case it fails. I'll try this tomorrow, 2PM GMT.
     
  10. Reaper

    Reaper Administrator Staff Member Administrator

    Complete and utter failure. We're back to where we were. XenForo really doesn't make it easy to merge two database together. Anyway, whatever. Some thread OPs are mixed up but we'll live. I'm really pissed off about this. I'm going to open a ticket with XF.
     
  11. Darth_Marques

    Darth_Marques Administrator Staff Member Administrator Premium

    Hmm. I'm not sure what else to try. At this point, this as much xF's problem as it is the add-on dev's problem. I don't suppose it had a log for errors when everything went to shit did it? I'd submit every thing I had to everyone involved.
     
  12. Reaper

    Reaper Administrator Staff Member Administrator

    We can just live with it I guess. XF refuse to provide assistance in 'manipulating the database' as they call it. I call it lazy to provide support. I just asked for the table names I'd need to replace to go back in time just on the posts and threads.
     
  13. Darth_Marques

    Darth_Marques Administrator Staff Member Administrator Premium

    In other words, they don't know what happened lol. Much less how to fix it. It'll be fine, we can recover some of the content at least. I'm just sad my post on X1 vs PS4 is gone lol.
     
  14. Reaper

    Reaper Administrator Staff Member Administrator

    Should be on web.archive.org, easy reposting it because all styles are kept if you copy and paste it :p. It's basically a copy paste job. If not, I think I have a backup with it in.
     
  15. Reaper

    Reaper Administrator Staff Member Administrator

    [19/04/2015 15:24:31] Robust: https://www.gamingforums.net/threads/hi.121/
    [19/04/2015 15:24:33] Robust: like here
    [19/04/2015 15:24:37] Robust: someones intro thread
    [19/04/2015 15:24:39] Robust: got mixed into
    [19/04/2015 15:24:42] Robust: lani’s ***** ****
    [19/04/2015 15:25:11] Robust: https://www.gamingforums.net/threads/hello-everybody.73/
    [19/04/2015 15:25:16] Robust: suggestion mixed up with introduction
    [19/04/2015 15:26:08] person2: wtf
    [19/04/2015 15:26:20] person2: thats like a big problem

    If it was just data loss it'd be alright, but we've got threads all mixed up as well.
     
  16. Darth_Marques

    Darth_Marques Administrator Staff Member Administrator Premium

    Holy crap, that's insane.
     
    • Agree Agree x 1
  17. Nova

    Nova Colonel

    100%
    This is old, but I was like "forum merge? What?"
     
  18. Reaper

    Reaper Administrator Staff Member Administrator

    Yeah, I meant to merge data from an old backup but I'd have to merge multiple database tables and account for huge changes we had during that time period, so I ended up just leaving it.
     

Share This Page