Factomize Forum Issue

Secured
#1
The Factomize forum had a serious issue and I want to update everyone on the situation. At approximately 12:07pm yesterday, August 25th I used administrative powers on the Factomize forum to batch delete spam threads and posts in the ANO forum. I have done this multiple times in the past on this specific forum and thousands of times over the years with no issues. Late last night while I was offline, I was messaged by a member telling me some of their threads were missing. Upon investigating this morning, it became apparent that not only were the spam threads removed, but quite a few valid threads and their posts were removed as well. These were permanently deleted from the database. Our last database snapshot prior to the removal of the threads was from 2018-08-25 03:40 UTC which is not long before the deletion. That database has been restored. If we caught this early, that would normally be the end of it but the problem is, we didn't realize this issue until almost a day later and posts have been made since then which the DB snapshop from the 25th does not take into account. We've dumped the new DB with those posts so have them but our dev is not available on this short notice to combine the two DBs and Guides need the forum to discuss the grading of applicants ASAP. As such, we've decided to bring the forum up with the 33 hour old DB. Screenshots of the posts that were lost were made and will be posted in each thread (there's five such posts in the public forums).

There is no evidence whatsoever of a hack. The admin log shows the valid post removal happened at the exact same time as the spam removal. As such, this was either a bug (possible) or user error (more likely as much as I hate to admit it though admittedly I have no idea how I could have screwed up so bad as it's not rocket science). We will be delving deeper into exactly what happened in case it was a bug so it can be reported to the software devs and fixed. In the meantime, the ability for even admin to permanently delete posts has been removed to ensure this does not happen again. Additional precautions and processes will be discussed and implemented as we catch our breath. On the bright side of things, this goes to show how valuable Factom integration for content publishing and communication platforms like our forum can be. With additional functionality, it would be possible to see everything from whether the restoration in fact did restore all posts, if the admin log had been tampered with, and much more. The reality is, user errors, hardware errors, software bugs, and hacks happen. The world's data needs to be better secured and auditable and the protocol we're all working on is the solution. We sincerely apologize for the trouble. We're embarrassed and frustrated and will take action to make sure it doesn't happen again.
 
Secured
#6
What I'm asking is if there are any existing add-ons that would make it easier for forum users to be able to independently verify post hashes by downloading posts and threads. The one I linked above looks like it would be helpful but I don't know anything about running a Xenforo forum so don't know if this would be hard/infeasible to deploy.
I understand. I say we'd need to develop (or modify) the tool as we'd need to ensure that the thread/post content was parsed exactly like how we parse it when we Factomize it otherwise the hash would be different. For example, the raw message content of your first post with bb code is:

Code:
Are there any Xenforo add ons that would make it easy for users to be able to download threads and posts locally so we could independently compare message hashes to the factomized ones?

Edit: Something like this, perhaps? [URL]https://xenforo.com/community/resources/export-thread.3814/[/URL]
If the tool exported even one character differently (for example, they stripped the url tag), the hash would not match.

For the record, that tool was developed for 1.x versions of Xenforo while we run 2.x which renders it incompatible. They may have created one for 2.x, I haven't checked.
 
Secured
#7
I didn't catch the 1.x versus 2.x version, sorry about that. I haven't been able to find a similar tool for 2.x so maybe we're out of luck on that. I do not think that the tool I posted would have to be modified, if it worked for the Xenforo version we're running. All we need is the plain bb text. I tested several posts on the command line, and as long as you don't add anything to the text and you hash it with the right hashing function you'll get the same hash as what's in the Factom entry.

For example:

Code:
echo "Are there any Xenforo add ons that would make it easy for users to be able to download threads and posts locally so we could independently compare message hashes to the factomized ones?

Edit: Something like this, perhaps? [URL]https://xenforo.com/community/resources/export-thread.3814/[/URL]" | sha512sum
returns the value:

Code:
a1ea661d87d524bf7335518f40c0d9a51dc4d612f479ffa20ec0f9979d4374f785372ddcd8178a754a07c7e0e118cd2aa19ea84d6b8b5901444b9a55f248236b
which matches the message hash in the Factom entry for that post.