TWC Downtime

[GrnEyedDvl]

Certainly a very rare turn of events. At least you're getting a head start on the "get fitter" new years resolution with the generator olympics.

I definitely should quit smoking. This has been kicking my ass even though I wouldn't say I was in terrible shape to begin with.

Hey guys.

Not sure why it went out again earlier but as far as I know GED knew immediately which is why it seemed to be restored so relatively quickly.

I knew within 5 min and sent Gen Chris an email that I was heading down there. Something glitched and the server didnt come back up properly and was throwing 500 errors. I was terrified it would be a database corruption error but it wasnt.

I spent the night there last night and ended up moving some client machines. Our stuff is still there running on generator, and not all of our stuff is up. The Vault and database backup server are still down. I will do a manual database backup later tonight, probably have to shut the site down for that. I should go post an announcement I guess.

Just when posting this I got a database error / timeout and on repost forum quoted it is a repost. Hope that everything is already running in sync.

Bad timing for you probably. When it first boots its got a LOT of stuff to cache in ram and it always takes a bit.

The same thing occurred to me. If the first fuse blowing was a highly unusual event, the owner/manager of the building may have owed you and your clients a duty to investigate the cause. You may want to look at your insurance policy, and hire an atty to look at the buildings policies.

He is getting a bill from me that would give most people a heart attack. He has earned it though, and they still haven't brought in even a temp solution. We are running on "portable" generators still. Two of them belong to me.

I have already been told the building and our data provider both have insurance to cover this stuff. Considering the hours I have put in, the equipment of mine they are using, lost downtime for all of my stuff even beyond TWC and likely some stuff I havent thought of yet, it will be a big bill. And I am charging him my full rate for all of this. Lots of people I give a break or dont count every single hour, not this time.

[Dresden]

I get a database error when I post or reply, but the post still goes through. Also, in the forums, new replies are not shown as being in the thread in the forum listing and new replies also do not bump up to the top.

[GrnEyedDvl]

Some weird things going on, something in the search core is screwed up. Will try and figure that out.

[Dresden]

sorry double post in this database error

Since yesterday, new replies are not bumping posts up and do not show up as replies in any forum thread listings. Also, as you said, I get a db error everytime I post or reply, but usually they go through.

[GrnEyedDvl]

I think the database errors are fixed.

[Rabbit55821]

Thank you very much for your hard work, and I wish you the best of luck! As others have said, I got really concerned over the long downtime, but thank you for getting things back up and running!

[GrnEyedDvl]

Ok hopefully that solves our other issues.

[GrnEyedDvl]

I think latest posts are showing up properly now, and a few other things that were happening seem to be ok. Site might be a bit slow for a few.

[Veteraan]

Awesome job thus far, GED, getting this mess sorted. Future plans sound good to me too.

I did see something while perusing the TATW forum that still seems to be going wrong. I can imagine there might be other similar things going on in the forum.

As Local Moderator, I am of course subscribed to the Third Age Total War forum. On my "My account" page, I see this as being the most recent post made on the forum:

Spoiler Alert, click show to read: 

Subscribed Forums

• Third Age: Total War
  
  
  Third Age: Total War Third Age: Total War is a total conversion modification for Medieval II: Total War that brings you into the world of Middle Earth
  
  • Unsubscribe from this Forum
  
  
  
  • Threads: 12,810
    
  • Posts: 507,802
  
  Divide and Conquer -...
  by Hrothgar Heavenlight
  
  Yesterday, 07:47 PM
  
  
  

When I try to go to that post however, I am dropped on page 122 of the thread. The post is not on that page, everything there is from longer ago. There is a page 123 and 124, but clicking on either of these numbers gets me back to page 122. Here is a link to page 124 of the thread:

http://www.twcenter.net/forums/showt...on-1-1/page124

Clicking on the post that is in the spoiler, results in the same outcome.

[Dresden]

Sorry about the previous double posts, was during the db issues.

[Frunk]

The same issue Vet described affects this thread, too (page 26 unreachable).

EDIT: I clicked Reply and Go Advanced, and the page 26 post (only one existed) was visible in the preview section which appears at the bottom of the page. I then submitted a reply and this worked; I can now access page 26 correctly.

EDIT: I just noticed that in forum view that thread says it has 500 replies, but my test post is #502 (i.e. it should be 501 replies).

EDIT: Another bizarre thing. The user in this thread has at least 2 posts, but a post count of 0. Resolved since scheduled maintenance.

EDIT: This thread has a reply made yesterday, but in forum view, it does not appear at all. Instead, the second most recent reply info appears, and since that post was made November 30, it's further down the list. I'm unsure if this thread would have been marked as unread in forum view, since I clicked on the link that was on the sub-forum, but I doubt it would have been. Therefore, I assume there might be other threads which appear to still be old and inactive, even if they have a reply in the past day or so. If there is more than one such affected thread in a forum or sub-forum, presumably, there is no easy way to find it.

Not sure if anything can be done about these things anyway, but I'll keep updating this as I go (until someone else replies here).

[GrnEyedDvl]

Part of what you guys are reporting is intentional and has been set that way for at least 3-4 years. Dunno why its just being noticed though. From the admincp:




Some of the other stuff could be related to mismatched indexes or caching issues. I purged the vBulletin cache, and the CloudFlare cache and I think most of the tables are optimized now.

[Frunk]

That screenshot mentions thread views, but not thread replies.

Of the above bugs I mentioned, the user's post count one is resolved now. The others still exist: a 502-post thread saying it only has 500 replies (+ 1 OP) in forum view, and; a recently replied to thread not having its reply details appear in forum view, thus appearing inactive since November.

[Veteraan]

Part of what you guys are reporting is intentional and has been set that way for at least 3-4 years. Dunno why its just being noticed though. From the admincp:




Some of the other stuff could be related to mismatched indexes or caching issues. I purged the vBulletin cache, and the CloudFlare cache and I think most of the tables are optimized now.

Edit: Just checked one more time and now it works as it should. I noticed there have been new replies added to the thread (not by me) since I last checked. Maybe that solved it, based on what Frunk reported earlier about adding a post to a thread. Checked the two examples Frunk provided in the above post and they seem to be solved also.

At this moment the situation with the thread I mentioned in my first post about this subject is still the same. Perhaps some optimizing is indeed still going on?
I did not try the workaround Frunk mentioned earlier, which is posting in the thread itself. If he mentions it solves the problem (temporary?) I'm quite confident that is the case.

I must admit that I didn't know about any "update thread views once every hour" system being active. I can't remember ever having noticed, or being troubled before by what I described earlier.
Originally I clicked on the "report" button of the last post in the thread I could view and explained the situation in a report. I was in a hurry, as I had to leave for the New Year celebration and thought that a report could be seen by quite a few people. There are many moderators for the TATW forum, including two Hex guys. Sadly, the report failed because of a database error. When I got back from the New Year festivities, I did a quick check if the problem was still there, noticed it was and wrote a quick reply in this thread, before collapsing (I did make it to my bed). Well, I guess that is something you can relate to going by what you wrote earlier.

[alQamar]

Some weird things going on, something in the search core is screwed up. Will try and figure that out.

Bad timing for you probably. When it first boots its got a LOT of stuff to cache in ram and it always takes a bit.

Have you been successful to find the culprit?

[Noif de Bodemloze]

Finally I can breath...! Yesterday It was a nightmare when I tried to do new threads for picture of the week content and then I got database error like hell. Today afternoon I managed to do new threads normally. Thanks GED to fix this

[GrnEyedDvl]

Have you been successful to find the culprit?

I hope so.

I copied the entire database exactly as it was, then ran the built in mysqlcheck on everything. That came back clean so I committed all the transaction logs, shut down mysql and dumped all the temp files, flushed all the caches, fired mysql back up then did a proper mysqldump. One all that ran I fired the site back up and went to bed.

[alQamar]

I hope so.

I copied the entire database exactly as it was, then ran the built in mysqlcheck on everything. That came back clean so I committed all the transaction logs, shut down mysql and dumped all the temp files, flushed all the caches, fired mysql back up then did a proper mysqldump. One all that ran I fired the site back up and went to bed.

You cannot be thanked enough man. Thanks for standing strong in all this christmas nightmare, also kudos to the unnamed that helped you with the rotation.

[Lifthrasir]

^^ I second that

[GrnEyedDvl]

You cannot be thanked enough man. Thanks for standing strong in all this christmas nightmare, also kudos to the unnamed that helped you with the rotation.

Yeah I owe a couple of guys big time.