![[info]](http://l-stat.livejournal.com/img/userinfo.gif){kind=small}
Derik in Minnesota
Oct. 5th, 2008
10:53 am - The last helicopter out of Wikia ( I believe in Google )
Just a brief update. (I'll do more proper status report at some future date.)
20 days into the move, the last of the major tech hurdles has been ironed out (our server didn't have ImageMagic installed!,) the site is clicking along like an engine and we're stopped looking back. It feels like we've been gone for a month.
Wikia meanwhile has redesigned its look, briefly gone login-only, and changed its name- dropping the "Teletraan 1" moniker only after we decided to dump it ourselves. There are now TWO sites called "Transformers Wiki." (Oy veh!)
"If it turns out we drive any anonymous people away [by going login-only], well, we still have all the power of Google directing people here, so we won't be starving for a constant influx of new users no matter what happens."
— Shawn, a Wikia Helper, argues that it doesn't matter if you piss your users off, there's more where they came from.
Now, 20 days in, October 5th... we start to get an idea how things are going to shake out. These are my observations;
- Wikia has had 500 edits in the last 8 days. TFWiki.net has had 500 edits in the last 24 hours.
- With all the old sysops gone, Wikia admins and community helpers have been admining the Wikia. Of course they don't actually know enough about Transformers to judge what information is good and bad, and the result has been reverted edits and a demand for citation. (This was more noticable in the initial flury of activity of the changeover. It's less noticeable now because, well, there's less editing going on.)
- 6 days ago I googled "Transformers wiki," and didn't get a hit off TFWiki.net site until the 7th page of search results. 2 days ago I did it again-- page 3. Today, page 2. (That first hit was Overkill, a page which we have edited 27 times since moving (and considerably expanded) and which Wikia has edited once, adding a single sentence about an upcoming toy release.
I believe in Google. People talk about it like an awful uncaring demon with no rhyme or reason... but Google's search algorithm can see that one site has stopped growing and the other continues to. That we've increased out number of articles by 6% in the last month and Wikia hasn't. That a good portion of the new links it notices are to us rather than Wikia.
I designed a website for a client 3 years ago... not a splashy or showy thing- but I included a content management system so he could edit the pages and add new ones. He's the #2 search result in Google for some VERY generic terms related to the direct-mailing industry. A couple of months ago... he basically asked me why, since he knows other guys who kill themselves over search terms and optimization and blah blah blah...
What I told him was this;
The Google algorithm is smart enough to tell the difference between a "living" site that has modest, regular updates, and a dead one that looks beautiful but never changes.
I believe in Google. When TFWiki.net announced we were leaving Wikia, I had people tell me flat-out that I was crazy to believe we'd ever be able to surpass Wikia in the Google index. But suddenly... it doesn't seem so crazy anymore.
Just. Be. Awesome. Google will notice.
(It really is that simple.)
Update: 6 hours later, TFWiki.net is now the #8 match on the first page. Progressing to #5 should be relatively easy- after that it gets hard.
![[mood icon]](http://l-stat.livejournal.com/img/mood/classic/energetic.gif){kind=small}
bouncySep. 14th, 2008
06:43 pm - The last helicopter out of Wikia (Moving Day) UPDATED
Third in my ongoing series on departing from Wikia... is Moving Day.
Teletraan 1: The Transformers Wiki is moving from Wikia, and will (by Monday,) be relocated at tfwiki.net on a dedicated server. (At the moment that URL directs back to wikia.)
Never ones to leave a cheap opportunity for drama unexploited, our dedicated server is in Houston, Texas, and is currently being pounded by Hurricane Ike. The tech-guy manually patching the rolling DNA problems this is causing is actually in the hurricane.
(I'm sure that's all very dramatic, but this is real life, not a movie of the week. The tech-girl actually performing the import is in Canada, safe and smug from everything but lying politicians, and she's bypassing the DNS problem entirely by using the server's dedicated IP address.)
Wikia coughed up an updated image archive with a minimum of problems. (We had to bounce the request off a few different people- but that's just because the person we'd made arrangements with to do so was on vacation.)
The database has now processed ~5100 article out of ~27,000. Assuming that it processed Namespace:0 first... that means it's about 3/4 done. (Annoyingly the progress counter does not give you a readout on the number of actually revisions processed, which would remove guessing.)
Total time, all told, looks like it's gonna be ~48 hours. Basically we put up a notice and told people "Refrain from editing," and they've mostly done so. (Annoying that it fell on a weekend... but that's life.) The move becaun on September 14th, approximately 2 weeks after the arbitrary date we were originally shooting for. Into the slippage fit several personal and work emergencies, shipping problems, and a hurricane. All told, I think it turned out alright.
Still to come- is the promised server-requirements rundown and recap of the import process. And maybe a discussion of ads...
As a community the TF Wiki decided some time ago we were NOT going to be sabotaging the wiki on our way out. (It being fairly pointless to do so anyway... too simple to revert.)
Still, having butted heads with Wikia over our ideas of the way things should be done on far too many occasions (we thought the experience of not-logged-in-users mattered, they thought we should stop complaining as long as we didn't have to see layout-destroying ads,) TT1 came to a collective agreement that while we'd be leaving the wiki intact, the stuff that represented our community was goign with us. And so starting at around 7PM CST, began the mass deletion of user pages that their users have been flagged, community portal pages, policy pages, user notice templates, any anything tied to TT1's unique way of doing business. Nothing on the main site was affected.
Now, I knew that was coming, but I have to admit, I got a dark, dark chuckle when I saw </i>this</i> 10 minutes ago...
Jul. 14th, 2008
10:41 pm - The last helicopter out of Wikia (usernames)
First in my (presumed) series about Decamping from Wikia... is the question of usernames.
Someone else has set up a dedicated mediawiki install, and downloaded a copy of the (full) wikiaarticle database and image archive, (about 700mb each in the Transformers wikia's case,) and done an import- taking ~10 hours. (It re-does every single edit ever done to an article in order.)
The problem: On Wikia (Server A) I'm User:Derik, and there is a User:Derik page on the new site (Server B), having been imported... but there ISN'T a user named Derik. Or rather- the Wikia export included all the pages- but not the user accounts. So now anyone can sign up for the name Derik on the new server. Potentially quite confusing, especially with a number of persistent trolls we have who might pass themselves off as existing users, or later claim an 'abandoned' username.
My solution: To register a username on Server B that also exists on Sever A (Wikia,) you must provide the same password used for this account on Server A.
Details
Code changes live in:
=====CODE CHANGES======
includes/SpecialUserlogin.php
languages/messages/MessagesEn.php
=====NEW FILES=====
transition/usernames.txt
transition/snoopy.class.phpStarting from the bottom (because where else would one start? Well- it is the logical order anyway...)
•snoopy.class.php
When someone tries to register an account on our new server (that also exists on our new one,) we will use Snoopy to see if the username/password combo they provided can be used to log in to the old server. If it can, then we know they're actually the "owner" of this account, and we allow the registration to continue.
Snoopy is used in a completely out-of-the-box fashion, you won't need to understand it to use it.
•usernames.txt
has less than 5000 users... or a ~65kb text file. (You can manually edit the 'limit' number in the URL to make all the users list on one page.)
When someone attempts to register a new account, this file will be loaded into memory and read line-by-line (the line is trimmed so if you miss some whitespace at the end of a line, this will catch it,) and the name compared with the username the person is trying to register. If there is a match, it does the post-to-wikia trick using Snoopy. (above.) This is a computationally 'lossy' procedure, but it only occurs when someone is registering an account and the memory overhead isn't terrible so... who cares?
Known issue: If you've got usernames that have Unicode characters in them, you'll have to use a different text format and adjust your code. (Or just let them take their chances while you protect the other user accounts.)
•"transition" folder.
I could have tucked them someplace else, but I wanted to be neat. This folder (depending on how your server is configured) may be listable. postable-to, etc... but who cares? The Snoopy class file ignores posts, and a list of (publicly-available) usernames isn't exactly a state secret. You can stick 'em someplace else if it makes you feel better but they're perfectly safe in a standard location.
•languages/messages/MessagesEn.php
This file contains all the user messages used by the Mediawiki software. It was easier to add one 'properly' than to try to hack it.
~line 865 I added a new error message named 'wikiauser-password'. I think you can add it anywhere, but I stuck it right after 'wrongpassword', a similar error message. The line I added reads like this;
'wikiauser-password' => 'The username "$1" is active on the <a href="http://transformers.wikia.com">T
The "$1" bit will be replaced by username. Notice that you can use HTML here, just be careful of nested quotes.
•includes/SpecialUserlogin.php
Here's the big one. Fortunately, it's a 'clean' addition. (You just need to customize a couple URL's.)
Within the function "addNewAccountInternal()", ~line 213, after the first if-block:
// If the user passes an invalid domain, something is fishy
if( !$wgAuth->validDomain( $this->mDomain ) ) {
$this->mainLoginForm( wfMsg( 'wrongpassword' ) );
return false;
}...but before the 'local authentication' block marked by this comment...
// If we are not allowing users to login locally, we should
// be checking to see if the user is actually able to
// authenticate to the authentication server before they
// create an account (otherwise, they can create a local account
// and login as any domain user). We only need to check this for
// domains that aren't local.You add... this!
/* ==== BEGIN WIKIA "TRANSITION" ADDITIONS=== */
$filename = "transition/usernames.txt";
$lines = file($filename); //Turn the file into an array, each element containing 1 line of the file.
for($i = 0; $i < count($lines); $i++){
$nameTaken = '';
if ( !strcasecmp(trim($lines[$i]), $this->mName) ){
//Provided username matches a reserved 'un.
$nameTaken = trim($lines[$i]);
$username = $this->mName; // Provided username
$passwd = $this->mPassword; // Provided password
require_once("transition/Snoopy.class.ph
$snoopy = new Snoopy();
$snoopy->agent = "Mozilla/5.0 Snoopy/1.2.3"; //We're pretending to be Firefox.
// Log into wikipedia (sets cookies for writing use):
$login_url = "http://transformers.wikia.com/index.ph
$login_vars = array(
"wpName" => $username,
"wpPassword" => $passwd,
"wpRemember" => "0",
"wpLoginattempt" => "Log in");
$snoopy->submit($login_url, $login_vars);
$result = $snoopy->results;
$result = stripos($result, '<h1 class="firstHeading">Login successful</h1>'); //True if it finds this text on wiki after trying to log in. (Markup is the same in Monaco and Monobook. May differ if you're running some weird-ass custom skin.)
if ($result){
//Login successful! Do nuthin.
}else{
//Login unsuccessful!
$this->mainLoginForm( wfMsg( 'wikiauser-password',$nameTaken) ); //Displays the custom error message we created.
return false; //Stops the login proscess prematurely and displays errors.
} //Error out if Wikia login fails
} //ifMatch
} // foreach
/* ==== END WIKIA "TRANSITION" ADDITIONS=== */That's about it. It's pretty straightforward. Uh- this is probably PHP5 only, though I don't imagine it'd take much work to adapt it for PHP4.
Difficulty: low-to-moderate. You want someone who's done PHP before executing this... but just so they know to avoid the basic pitfalls.
If you did it right... try to sign up using someone else's account and you'll end up with a result like this!
![[mood icon]](http://l-stat.livejournal.com/img/mood/classic/angry.gif){kind=small}
irritated09:08 pm - The last helicopter out of Wikia (intro)
"For all things change, making way for each other."
—Euripides, Heracles
Wikia is the for-profit sister/cousin project from Wikipedia. 6000-odd dedicated-subject wikis- mostly dedicated to sci-fi and fantasy such as Star Wars, Star Trek, Marvel, DC and Warcraft. (To name some of the biggest.)
Unlike the not-for-profit Wikipedia, Wikia supports its endeavors with ads. This June there was a push to monetize Wikia's initial investment. That can be good or bad, but since they've managed to piss me off I'm going to compare it to a similar push at ShopNBC a couple years ago; which at one point had their phone reps constantly harassing customers with "Could I also interest you in..."'s because investors got tired of 'slowly building customer loyalty' and mandated that every every possible cent be wrung out each purchase... and to hell with the customer experience.
A new layout that removes functionality from the program in favor of making it 'friendlier to new users,' ads at the top of articles instead out outside of them- then ads inside articles screwing up their layout, then ads appearing randomly inside articles.
Basically, Wikia is looking more-and-more like a spamblog (complete with broken content,) and yesterday when I discovered that the newest version of their ad-script hadn't been checked against Safari and was vomiting HTML across articles if you used that browser, I dropped my opposition to the movie.
It's kinda emblematic of Wikia's handlig of their userbase. Their june layout stopped working if tyou didn't have Javascript turned on- but their development team didnt' consider that a problem because "our research shows only 1% don't have Javascript turned on."
Basically, it's shit, and getting shittier, and the attitude towards the people generating the content they're making money off of has been to lie, pacify, misdirect or condescend to them.
The Transformers Wikia has decided to leave.
I'm terribly irritated about this- the entire reason I wanted to stay with them is because it meant THEY were providing the logistical backdone and I didn't have to dedicate time I'd rather be spending actually editing articles managing shit like the site's 2 gigabyte sql database.
But it's done, so there's no use crying over it. I've decided to document the work I'm going in the hopes that it's useful to others who are decamping from Wikia-- so THEY can spend their time editing articles instead of coding.
Should be fun...
Chopper #1 - Usernames
Chopper #2 - Removing export-spam
Chopper #3 - Moving Day